NodeJs/expressjs : Run lengthy code in a callback

Question

NodeJs/expressjs : Run lengthy code in a callback

Callbacks are asynchronous , So does that mean that if I run a lengthy computation in a callback it wont affect my main thread ?

For example:

function compute(req,res){    // this is called in an expressjs route.
    db.collection.find({'key':aString}).toArray(function(err, items) {
        for(var i=0;i<items.length;i++){      // items length may be in thousands.
            // Heavy/lengthy computation here, Which may take 5 seconds.
        }
        res.send("Done");
    });
}

So, the call to database is ascnchronous. Does that mean the for loop inside the callback will NOT block the main thread ?

And if it is blocking, How may I perform such things in an async way?

javascript
multithreading
node.js

Answer 1

For the most part, node.js runs in a single thread. However, node.js allows you to make calls that execute low-level operations (file reads, network requests, etc.) which are handled by separate threads. As such, your database call most likely happens on a separate thread. But, when the database call returns, we return back to the main thread and your code will run in the main thread (blocking it).

The way to get around this is to spin up a new thread. You can use cluster to do this. See:

http://nodejs.org/api/cluster.html

Your main program will make the database call
When the database call finishes, it will call fork() and spin up a new thread that runs your-calculations.js and sends an event to it with any input data
your-calculations.js will listen for an event and do the necessary processing when it handles the event
your-calculations.js will then send an event back to the main thread when it has finished processing (it can send any output data back)
If the main thread needs the output data, it can listen for the event that your-calculations.js emits

Answer 2

If you can't do, or don't want to use a thread, you can split up the long computation with setImmediates. e.g. (writing quickly on my tablet so may be sloppy)

function compute(startIndex, max, array, partialResult, callback) {
   var done = false;
   var err = null;
   var stop = startIndex+100;  // or some reasonable amount of calcs...
   if (stop >= max) {
      stop = max;
      done = true;
   }

   // do calc from startIndex to stop, using partialResult as input
   if (done)
      callback(err, result);
   else 
      process.setImmediate ( go look this part up or I'll edit tomorrow)...
       But the idea is you call youself again with start += 100.
}

In between every 100 calculations node will have time to process other requests, handle other callbacks, etc. Of course, if they trigger another huge calculation evedntually things will grind to a halt.