What's the right way to find out when a series of callbacks (fired from a loop) have all executed?

Question

What's the right way to find out when a series of callbacks (fired from a loop) have all executed?

I'm new to Node.js and am curious what the prescribed methodology is for running a loop on a process (repeatedly) where at the end of the execution some next step is to take place, but ONLY after all the iterations' callbacks have fired.

Specifically I'm making SQL calls and I need to close the sql connection after making a bunch of inserts and updates, but since they're all asynchronous, I have no way of knowing when all of them have in fact completed, so that I can call end() on the session.

Obviously this is a problem that extends far beyond this particular example, so, I'm not looking for the specific solution regarding sql, but more the general practice, which so far, I'm kind of stumped by.

What I'm doing now is actually setting a global counter to the length of the loop object and decrementing from it in each callback to see when it reaches zero, but that feels REALLY klugy, and I'm hoping theres a more elegant (and Javascript-centric) way to achieve this monitoring.

TIA

node.js
loops

Answer 1

There are a bunch of flow-control libraries available that apply patterns to help with this kind of thing. My favorite is async. If you wanted to run a bunch of SQL queries one after another in order, for instance, you might use series:

async.series([
  function(cb) { sql.exec("SOME SQL", cb) },
  function(cb) { sql.exec("SOME MORE SQL", cb) },
  function(cb) { sql.exec("SOME OTHER SQL", cb) }
], function(err, results) {
  // Here, one of two things are true:
  // (1) one of the async functions passed in an error to its callback
  //     so async immediately calls this callback with a non-null "err" value
  // (2) all of the async code is done, and "results" is
  //     an array of each of the results passed to the callbacks
});

Answer 2

I wrote my own queue library to do this (I'll publish it one of these days), basically push queries onto a queue (an array basically) execute each one as it's removed, have a callback take place when the array is empty.

It doesn't take much to do it.

*edit. I've added this example code. It isn't what I've used before and I haven't tried it in practice, but it should give you a starting point. There's a lot more you can do with the pattern.

One thing to note. Queueing effectively makes your actions synchronous, they happen one after another. I wrote my mysql queue script so I could execute queries on multiple tables asynchronously but on any one table in synch, so that inserts and selects happened in the order they were requested.

var queue = function() {
    this.queue = [];

    /**
     * Allows you to pass a callback to run, which is executed at the end
     * This example uses a pattern where errors are returned from the 
     * functions added to the queue and then are passed to the callback 
     * for handling.
     */
    this.run = function(callback){
        var i = 0;
        var errors = [];

        while (this.queue.length > 0) {
            errors[errors.length] = this.queue[i]();
            delete this.queue[i];
            i++;
        }
        callback(errors);
    }

    this.addToQueue = function(callback){
        this.queue[this.queue.length] = callback;
    }
}

use:

var q = new queue();
q.addToQueue(function(){
    setTimeout(function(){alert('1');}, 100);
});

q.addToQueue(function(){
    setTimeout(function(){alert('2');}, 50);
});

q.run();