How to write through to a db one by one from evented input that arrives too fast in node.js

Question

How to write through to a db one by one from evented input that arrives too fast in node.js

I receive input from a MS SQL SELECT query using the tedious driver.

I have attached a listener to the "row" event of the reader:

request.on('row', function(columns) {
     insert_row_other_db(columns);
});

I am writing the results to another database in the insert_row_other_db function.

But the rows arrive much faster than they can be written, and I want to open only one connection. What is a good way to de-asyncronyze the writes to the other db? I would like to write the rows one after the other.

node.js
asynchronous
io

Answer 1

Assuming you are able to receive a callback when insert_row_other_db completes, you can use the Async library to create a queue that you can use to schedule your inserts. The queue has a concurrency setting that allows you to limit the number of tasks that can run at once.

var async = require("async");

var insertQueue = async.queue(function (columns, callback) {
  insert_row_other_db(columns, callback);
}, 1);

request.on("row", function (columns) {
  insertQueue.push(columns);
});