How to stream an http response to a file, retrying on 503

Question

How to stream an http response to a file, retrying on 503

Piping the response of an http request to a file is pretty easy:

http.get(url, function (res) {
  var file = fs.createWriteStream(filename)
  res.pipe(file)
  file.on('finish', function () {
    file.close()
  })
})

But when I try to set up a retry system, things get complicated. I can decide whether to retry based on res.statusCode, but then if I decide to retry, this means not piping the response to the writable stream, so the response just stays open. This considerably slows down execution when I do many retries. A solution to this is to (uselessly) listen to the data and end events just to get the response to close, but then if I decide not to retry, I can not longer pipe to the writeStream, as the response is now closed.

Anyways, I'm surely not the only one wanting to stream the response of an http request to a file, retrying when I get a 503, but I can't seem to find any code "out there" that does this.

javascript
node.js
http
stream

Answer 1

Solved. The slowness happens when a lot of responses are left open (unconsumed). The solution was to response.resume() them, letting them "spew in nothingness" when a retry is necessary. So in pseudo code:

http.get(url, function (response) {
  if (response.statusCode !== 200) {
    response.resume()
    retry()
  } else {
    response.pipe(file)
  }
})

My original problem was that I was checking wheter to retry or not too late, forcing me to "spew into nothingness" before having decided, making it impossible to decide not to retry because the data from the response "stream" was already consumed.