Node.js - multiple rounds of parallel io

Question

Node.js - multiple rounds of parallel io

I was at a node.js meetup recently and the guy giving the talk said something along the lines of:
"You need generators for being able to do multiple rounds of parallel io. Callbacks won't cut it"

I'm new to node.js and have no idea what this means. Could someone explain?

Edit: As asked in the comments, here is a more detailed version: I was at a meetup for introduction to node.js. Someone in the audience asked the speaker what he thought was the biggest shortcoming of node.js. He said that until node.js gets generators it doesn't have a good solution for multiple rounds of parallel I/O. Any large-scale web app will have to do this. Hitting memcache in parallel in multiple rounds is one example, databases and third-party APIs are others. Anyone coming from languages like Python, Ruby or Go which have support for generators, threads or microthreads would have a hard time accepting that a platform could rely exclusively on callbacks.

javascript
node.js
io

Answer 1

He could have been referring to a sequence helper, running one task in the callback of the next would mean the chain would be running synchronously.

This is an example of a generator I use for inserting huge amounts of data into a Mongo collection. This generates sequences of operations to be performed in parallel. Performing a million inserts by chaining them using callback methods wouldn't be very practical in this instance. So that's why I use a generator / sequence helper like the one below.

var Inserter = function (collection) {
    this.collection = collection;
    this.data = [];
    this.maxThreads = 30;
    this.currentThreads = 0;
    this.batchSize = 1000;
    this.queue = 0;
    this.inserted = 0;
    this.startTime = Date.now();
};

Inserter.prototype.add = function(data) {
    this.data.push(data);
};

Inserter.prototype.insert = function(force) {
    var that = this;
    if (this.data.length >= this.batchSize || force) {
        if (this.currentThreads >= this.maxThreads) {
            this.queue++;
            return;
        }
        this.currentThreads++;
        console.log('Threads: ' + this.currentThreads);
        this.collection.insert(this.data.splice(0, this.batchSize), {safe:true}, function() {
            that.inserted += that.batchSize;
            var currentTime = Date.now();
            var workTime = Math.round((currentTime - that.startTime) / 1000)
            console.log('Speed: ' + that.inserted / workTime + ' per sec');
            that.currentThreads--;
            if (that.queue > 0) {
                that.queue--;
                that.insert();
            }
        });
    }
};