Rate limit GET requests

Question

Rate limit GET requests

I have an external API that rate-limits API requests to up to 25 requests per second. I want to insert parts of the results into a MongoDB database.

How can I rate limit the request function so that I don't miss any of API results for all of the array?

MongoClient.connect('mongodb://127.0.0.1:27017/test', function (err, db) {
    if (err) {
        throw err;
    } else {
        for (var i = 0; i < arr.length; i++) {
            //need to rate limit the following function, without missing any value in the arr array
            request( {
                method: 'GET',
                url: 'https://SOME_API/json?address='+arr[i]
            },
            function (error, response, body) {
                //doing computation, including inserting to mongo
            }
            )
        };
    }
});

node.js
mongodb

Answer 1

You need to combine 2 things.

A throttling mechanism. I suggest _.throttle from the lodash project. This can do the rate limiting for you.
You also need an async control flow mechanism to make sure the requests run in series (don't start second one until first one is done). For that I suggest async.eachSeries

Both of these changes will be cleaner if you refactor your code to this signature:

function scrape(address, callback) {
   //code to fetch a single address, do computation, and save to mongo here
   //invoke the callback with (error, result) when done
}