How to overcome asynchronous non-blocking NOT returning values in times?

I am creating an array of JSON objects which is then stored in mongodb. Each JSON object contains a number of fields - each being populated before I save the object to mongodb.

Some of the Objects attributes are populated by making API calls to other websites such as last.fm but the returned value is not quick enough to populate the attribute before the object is saved to mongodb.

How can I wait for all attributes of an object to be populated before saving it? I did try async.waterfall but it still falls through without waiting and I end up with a database filled with documents with empty fields..

Any help would be greatly appreciated. Thanks :)

You have a few options for controlling asynchrony in JavaScript:

  1. Callback pattern: (http://npmjs.org/async) async.all([...], function (err) {

  2. Promises: (http://npmjs.org/q) Q.all([...]).then(function () {

  3. Streams: (http://npmjs.org/concat-stream) see also https://github.com/substack/stream-handbook

Since you say you are making multiple API calls to other websites, you may want to try:

async.each(api_requests,
    function(api_request, cb) {
        request(api_request, function (error, response, body) {
            /* code */
            /* add to model for Mongo */
            cb();
        });
    },
    function(err) {
        // continue execution after all cbs are received
        /* code */
        /* save to Mongo, etc.. */
    }
);

The above example is most applicable when you are making numerous requests following the same format. Please review the documentation for Waterfall (https://github.com/caolan/async#waterfall) if the input into your next step depends on the output of the previous step or Parallel (https://github.com/caolan/async#parallel) if you have a bunch of unrelated tasks that don't rely on each other. The great thing about async is that you can nest and string all the functions together to support what you're trying to do.

You'll either want to use promises or some sort of callback mechanism. Here's an example of the promise method with jPromise:

var jPromise = require('jPromise');
var promises = [];

for(var i = 0; i < 10; i++) {
    promises.push(someAsyncApiCall(i));
}

jPromise.when(promises).then(function() {
    saveThingsToTheDb();
});

Similarly, without the promise library:

var finished = 0;
var toDo = 10;

function allDone() {
    saveThingsToTheDb();
}

for(var i = 0; i < toDo.length; i++) {
    someAsyncApiCall(function() {
        finished++;

        if(finished === toDo) {
            allDone();
        }
    });
}

Personally, I prefer the promise method, but that will only that well if the API you're calling returns some sort of a promise. If it doesn't, you'll be SOL and wrap the callback API with promises somehow (Q does this pretty well).