How can I stream file uploads directly into Mongo's GridFS using Formidable?

Question

How can I stream file uploads directly into Mongo's GridFS using Formidable?

I am trying to stream an upload from Formidable directly into Mongo's GridFS.

The GridStore needs to be opened before any data can be written however in the time it takes to open the store, too much data is already unparsed and it fails.

How can I prepare the GridStore then handle the incoming upload?

function upload (request, response, next, options) {
    var form = new Formidable.IncomingForm();

    var store = new Mongo.GridStore(options.mongoose.connection.db, new ObjectID, 'w+', {
        root: 'store',
        chunk_size: 1024 * 64
    } );

    form.onPart = function (part) {
        if(!part.filename){
            form.handlePart(part);
            return;
        }

        part.on('data', function(buffer){
            store.write(buffer);
        });

        part.on('end', function() {
            store.close();
        });
    };

    store.open( function (error, store) {
        form.parse(request);
    });

    response.send();
}

Also the store instantiation and open should probably be put inside onPart somehow.

Alex

node.js
mongodb
file-upload
stream

Answer 1

Since opening a GridStore file is async and formidable is not, you'll need to do some manual buffering of formidables incoming stream while waiting for the GridStore file to open. Since GridFS cannot be piped to you'll then need to manually write each buffered chunk to the GridStore after it opens.

Just released gridform which should do what you need.

https://github.com/aheckmann/gridform

Answer 2

You could move your form's handling code into the store.open callback, for example :

function upload (request, response, next, options) {

    var store = new Mongo.GridStore(options.mongoose.connection.db, new ObjectID, 'w+', {
        root: 'store',
        chunk_size: 1024 * 64
    } );


    store.open( function (error, store) {
        var form = new Formidable.IncomingForm();
        form.onPart = function (part) {
            if(!part.filename){
                form.handlePart(part);
                return;
            }

            part.on('data', function(buffer){
                store.write(buffer);
            });

            part.on('end', function() {
                store.close();
            });
        };
        form.parse(request);
    });

    response.send();
}