Model.create() vs Model.collection.insert(): a faster approach

Model.create() is a bad way to do inserts if you are dealing with a very large bulk. It will be very slow. In that case you should use Model.collection.insert, which performs much better. Depending on the size of the bulk, Model.create() will even crash! Tried with a million documents, no luck. Using Model.collection.insert it took just a few seconds.

Model.collection.insert(docs, options, callback)

• docs is the array of documents to be inserted;
• options is an optional configuration object - see the docs
• callback(err, docs) will be called after all documents get saved or an error occurs. On success, docs is the array of persisted documents.

As Mongoose's author points out here, this method will bypass any validation procedures and access the Mongo driver directly. It's a trade-off you have to make since you're handling a large amount of data, otherwise you wouldn't be able to insert it to your database at all (remember we're talking hundreds of thousands of documents here).

A simple example

var Potato = mongoose.model('Potato', PotatoSchema);

var potatoBag = [/* a humongous amount of potato objects */];

Potato.collection.insert(potatoBag, onInsert);

function onInsert(err, docs) {
    if (err) {
        // TODO: handle error
    } else {
        console.info('%d potatoes were successfully stored.', docs.length);
    }
}

Reference

• Mongo documentation
• Aaron Heckman on Google Groups discussing bulk inserts

Indeed, you can use the "create" method of Mongoose, it can contain an array of documents, see this example:

Candy.create({ candy: 'jelly bean' }, { candy: 'snickers' }, function (err, jellybean, snickers) {
});

The callback function contains the inserted documents. You do not always know how many items has to be inserted (fixed argument length like above) so you can loop through them:

var insertedDocs = [];
for (var i=1; i<arguments.length; ++i) {
    insertedDocs.push(arguments[i]);
}

Update: A better solution

A better solution would to use Candy.collection.insert() instead of Candy.create() - used in the example above - because it's faster (create() is calling Model.save() on each item so it's slower).

See the Mongo documentation for more information: http://docs.mongodb.org/manual/reference/method/db.collection.insert/

(thanks to arcseldon for pointing this out)

You can perform bulk insert using mongoDB shell using inserting the values in an array.

db.collection.insert([{values},{values},{values},{values}]);

You can perform bulk insert using mongoose, as the highest score answer. But the example cannot work, it should be:

/* a humongous amount of potatos */
var potatoBag = [{name:'potato1'}, {name:'potato2'}];

var Potato = mongoose.model('Potato', PotatoSchema);
Potato.collection.insert(potatoBag, onInsert);

function onInsert(err, docs) {
    if (err) {
        // TODO: handle error
    } else {
        console.info('%d potatoes were successfully stored.', docs.length);
    }
}

Don't use a schema instance for the bulk insert, you should use a plain map object.

It seems that using mongoose there is a limit of more than 1000 documents, when using

Potato.collection.insert(potatoBag, onInsert);

You can use:

var bulk = Model.collection.initializeOrderedBulkOp();

async.each(users, function (user, callback) {
    bulk.insert(hash);
}, function (err) {
    var bulkStart = Date.now();
    bulk.execute(function(err, res){
        if (err) console.log (" gameResult.js > err " , err);
        console.log (" gameResult.js > BULK TIME  " , Date.now() - bulkStart );
        console.log (" gameResult.js > BULK INSERT " , res.nInserted)
      });
});

But this is almost twice as fast when testing with 10000 documents:

function fastInsert(arrOfResults) {
var startTime = Date.now();
    var count = 0;
    var c = Math.round( arrOfResults.length / 990);

    var fakeArr = [];
    fakeArr.length = c;
    var docsSaved = 0

    async.each(fakeArr, function (item, callback) {

            var sliced = arrOfResults.slice(count, count+999);
            sliced.length)
            count = count +999;
            if(sliced.length != 0 ){
                    GameResultModel.collection.insert(sliced, function (err, docs) {
                            docsSaved += docs.ops.length
                            callback();
                    });
            }else {
                    callback()
            }
    }, function (err) {
            console.log (" gameResult.js > BULK INSERT AMOUNT: ", arrOfResults.length, "docsSaved  " , docsSaved, " DIFF TIME:",Date.now() - startTime);
    });
}