Writing hex digest to file in Node.js

Question

Writing hex digest to file in Node.js

I have a bot that makes a request to open a page and obtain image links and downloads them. However, I noticed that when I tried to download the image to my current working directory, I receive an empty image file. I've provided a scenario below.

Edit:

I realized that what was causing this is checking to see if the image count is equal to the max. For some reason it's not performing the final write process when making the request causing it to write a blank image. How can I resolve this?

Code Scenario:

var request = require('request');
var crypto = require('crypto');
var fs = require('fs');

var md5 = function (pic) {
  var hash = crypto.createHash('md5');
  hash.update(pic);
  return hash.digest('hex');
};

var pics = ['http://37.media.tumblr.com/eea272630e7c0749241b1594b1b911b4/tumblr_n1t63oOzF81rt268so1_1280.jpg',
           'http://38.media.tumblr.com/9b40422b9129745292b174ffd683e8f3/tumblr_mvhhcuOQn41rt268so1_1280.jpg',    
           'http://37.media.tumblr.com/7f96f676610efa9beb2f322c4f6eeb36/tumblr_mudrsmtKtl1r4lu5ao1_1280.jpg'
];

var downloadCount = 0, max = 2;

for (var i = 0; i < pics.length; i++) {
  var hexDigest = md5(pics[i]) + pics[i].slice(-4);
  request(pics[i]).pipe(fs.createWriteStream(process.cwd() + '/' + hexDigest));
  downloadCount++;
  if (downloadCount === max) {
    process.exit(0);
  }
}

javascript
node.js

Answer 1

When you're looping through the array of images and doing the requests, you're creating asynchronous streams. The following line of code only starts the download, it does not wait for the download to complete:

request(pics[i]).pipe(fs.createWriteStream(process.cwd() + '/' + hexDigest));

When you're checking the download count and exiting the process once it has reached max, the downloads are interrupted before they have even gotten the chance to start.

A possible solution is to move the download counter to an end event listener on the stream, like so:

request(pics[i]).pipe(fs.createWriteStream(process.cwd() + '/' + hexDigest))
  .on('end', function () {
    downloadCount++;
    if (downloadCount === max) {
      process.exit(0);
    }
  });

It can also be solved using a library such as async.

Also, the NodeJS runtime keeps track of the number of callbacks (= streams) that are active, an will exit the process once they are all completed. So unless something else is forcing you to stop the program with process.exit you don't need to track the download count at all.