Itterate through Array, update/create Objects asynchronously, when everything is done call callback

I have a problem, but I have no idea how would one go around this.

I'm using loopback, but I think I would've face the same problem in mongodb sooner or later. Let me explain what am I doing:

  • I fetch entries from another REST services, then I prepare entries for my API response (entries are not ready yet, because they don't have id from my database)
  • Before I send response I want to check if entry exist in database, if it doesn't:
    • Create it, if it does (determined by source_id):
    • Use it & update it to newer version
  • Send response with entries (entries now have database ids assigned to them)

This seems okay, and easy to implement but it's not as far as my knowledge goes. I will try to explain further in code:

//This will not work since there are many async call, and fixedResults will be empty at the end
var fixedResults = [];
//results is array of entries
results.forEach(function(item) {
    Entry.findOne({where: {source_id: item.source_id}}, functioN(err, res) {
        //Did we find it in database?
        if(res === null) { 
            //Create object, another async call here
            fixedResults.push(newObj);
        } else {
            //Update object, another async call here
            fixedResults.push(updatedObj);
        }
    });
});
callback(null, fixedResults);

Note: I left some of the code out, but I think its pretty self explanatory if you read through it.

So I want to itterate through all objects, create or update them in database, then when all are updated/created, use them. How would I do this?

You can use promises. They are callbacks that will be invoked after some other condition has completed. Here's an example of chaining together promises https://coderwall.com/p/ijy61g.

The q library is a good one - https://github.com/kriskowal/q

This question how to use q.js promises to work with multiple asynchronous operations gives a nice code example of how you might build these up.

This pattern is generically called an 'async map'

var fixedResults = []; var outstanding = 0; //results is array of entries results.forEach(function(item, i) { Entry.findOne({where: {source_id: item.source_id}}, functioN(err, res) { outstanding++; //Did we find it in database? if(res === null) { //Create object, another async call here DoCreateObject(function (err, result) { if (err) callback(err); fixedResults[i] = result; if (--outstanding === 0) callback (null, fixedResults); }); } else { //Update object, another async call here DoOtherCall(function (err, result) { if(err) callback(err); fixedResults[i] = result; if (--outstanding === 0) callback (null, fixedResults); }); } }); }); callback(null, fixedResults);

You could use async.map for this. For each element in the array, run the array iterator function doing what you want to do to each element, then run the callback with the result (instead of fixedResults.push), triggering the map callback when all are done. Each iteration ad database call would then be run in parallel.

Mongo has a function called upsert.

http://docs.mongodb.org/manual/reference/method/db.collection.update/

It does exactly what you ask for without needing the checks. You can fire all three requests asnc and just validate the result comes back as true. No need for additional processing.