Patterns for async list processing in node.js

Question

Patterns for async list processing in node.js

I have an application where a database query returns a number of rows (typically, less than 100). For each row, I need to make an http call to get supplemental data. I'd like to fire off all of the requests, and then when the last callback completes, move on to rendering the result page.

So far, the answers to similar questions I've looked at have either been chaining the requests by making request #2 in the callback for request #1 (advantages: simple, avoids burying the server in multiple requests), or by firing all of the requests with no tracking of whether all of the requests have completed (works well in the browser where the callback updates the UI).

My current plan is to keep a counter of requests made and have the callback decrement the counter; if it reaches zero, I can call the render function. I may also need to handle the case where responses come in faster than requests are being made (not likely, but a possible edge case).

Are there are other useful patterns for this type of problem?

node.js
design-patterns
asynchronous

Answer 1

When using async code could roughly look like this:

var async = require('async');

results = [];

var queue = async.queue(function(row, callback) {
    http.fetchResultForRow(row, function(data){
        result.push(data);
        callback();      
    });
}, 1);

queue.drain = function() {
    console.log("All results loaded");
    renderEverything(results);
}

database.fetch(function(rows) {
    for (var i=0; i < rows.length; i++) {
        queue.push(rows[i]);
    }
});

If the order does not matter you also could use: map

Look around in the documenation of async, there are a lot of useful patterns.

Answer 2

You can implement this quite nicely with promises using the when library. Though if you want to rate limit the calls to getting the extra info you will need to do a little more work than in the async approach of TheHippo I think.

Here's an example:

when = require('when')

// This is the function that gets the extra info.
// I've added a setTimeout to show how it is async.    
function get_extra_info_for_row(x, callback) {
   setTimeout( function(){ return callback(null, x+10); }, 1 );
};

rows = [1,2,3,4,5];

row_promises = rows.map(
   function(x) {
      var defered = when.defer()
      get_extra_info_for_row(x, function(err,extra_info) {
         if(err) return defered.reject(err);
         defered.resolve([x,extra_info]);
      });
      return defered.promise;
   })


when.all( row_promises )
   .then(
      function(augmented_rows) { console.log( augmented_rows ); },
      function(err) { console.log("Error", err ); }
      );

This outputs

[ [ 1, 11 ], [ 2, 12 ], [ 3, 13 ], [ 4, 14 ], [ 5, 15 ] ]