Currently I am developing an app that fire off hundreds of concurrent requests to external API service (like instagram for example) using ajax on client side. Response time is very fast.
However, I am migrating the request handling part to node backend using request + jsonstream but always get socket hang up error due to concurrency > 5 requests (even after changing maxsockets to higher values). Overall it is much much slower than doing API requests directly on client side using ajax.
My question is how can I make the proxy server faster/more responsive? Or maybe using ajax similar to when doing on client side but on node?
Server side: when client hits endpoint /fetchmedia/, node directs to this function.
var fetchInstagram = function(user_id, max_id, min_timestamp, max_timestamp, array, callback) {
http.request({
host: 'endpoint',
path: 'endpoint',
method: 'get'
}, function(res) {
var body = '';
res.on('data', function(chunk) {
body += chunk;
});
res.on('end', function() {
var data = JSON.parse(body);
array = array.concat(data.data);
if (data.pagination.next_max_id != undefined) {
fetchInstagram(user_id, data.pagination.next_max_id, min_timestamp, max_timestamp, array, callback);
} else {
callback(array);
}
});
}).on('error', function(e) {
console.log("Got error: ", e);
}).end();
Client-side: Backbone sends hundreds of requests (/fetchmedia) at the same time, which calls many fetchinstagram functions. The way I was doing before was sending ajax, which also hundreds concurrently but it handles very well. Node hangs up even with 20 users while ajax handles 1000+ users
Thanks