302 Redirect prevents callback in Request

Question

302 Redirect prevents callback in Request

I've noticed some websites don't close 302 redirects which causes the callback not to fire.

Anyone know how to remedy this?

var request = require('request');

request({url:'http://craigslist.org' }, function (error, response, body)
{
    console.log("Callback Never Runs. ");
});

node.js

Answer 1

Sorry, upon further investigation this suggestion also fails. But when trying multiple times with your code that will work occasionally too. The best I've come up with is using a timeout to retry. timeout fires the complete event where you can check if the request really finished and if not retry.

var request = require('request');

function doRequest(location) {
    request({
      timeout: 5000, // five seconds
      url:location
    }, function (error, response, body) {
        console.log("Callback Never Runs. ");
    }).on("complete", function(response) {
      if(!response.complete) setImmediate(function() {
         doRequest(location);
      });
    });
}

doRequest('http://craigslist.org');

You can expand on this further to only retry X number of times before throwing an error.

----original----

This could be a bug in request as in this case craigslist.org sends all 302 errors. But if you add followAllRedirects: true to you options it works.

var request = require('request');

request({
  followAllRedirects: true,
  url:'http://craigslist.org'
}, function (error, response, body) {
    console.log("Callback Never Runs. ");
});

Answer 2

It should work if you include a User-Agent (such as your browser's):

request({
    url: 'http://craigslist.com',
    headers: {
        'User-Agent': 'Mozilla/5.0 (...) ...'
    }
}, function (err, res, body) {
    // ...
});

Without that, Craigslist seems to leave the connection open, so it doesn't 'end' or 'complete' and the callback isn't called.