Limiting requests with the async and request modules

Question

Limiting requests with the async and request modules

I'm combining the async and request modules to make api requests asynchronously and with rate limiting.

Here is my code

var requestApi = function(data){

    request(data.url, function (error, response, body) {

        console.log(body);

    });
};


async.forEachLimit(data, 5, requestApi, function(err){
      // do some error handling.
});

Data contains all the urls I make request to. Am limiting the number of concurrent request to 5 using forEachLimit method. This code makes the first 5 request then stops.

In the async docs it says "The iterator is passed a callback which must be called once it has completed". But I don't understand this, what should I be doing to signal that the request has completed?

node.js
asynchronous
callback
request

Answer 1

First, you shall add callback to your iterator function:

var requestApi = function(data, next){
    request(data.url, function (error, response, body) {
        console.log(body);
        next(error);
    });
};

next(); or next(null); tells Async that all processing is done. next(error); indicates an error (if error not null).

After processing all requests Async calls its callback function with err == null:

async.forEachLimit(data, 5, requestApi, function(err){
    // err contains the first error or null
    if (err) throw err;
    console.log('All requests processed!');
});

Async calls its callback immediately after receiving the first error or after all requests completed succesfully.