I've just started playing with web development, and node.js + express helped me get a simple server up quickly on my not-so-powerful embedded system. When trying to serve a 270MB directory as a tarfile, however, the performance is not good at all with my solution.
After getting some help on irc I tried serving the file using fstream and node-tar, like this:
fstream.Reader({ type: "Directory" , path: mydir }).pipe(tar.Pack({})).on("error", function(){ res.statusCode = 500; res.end('Unable to create tarball'); }).pipe(res)
It works, but it takes ~20 minutes to download the 270MB file to my PC. As an experiment I tried tar'ing the folder on target, and then scp'ed it down to my PC. This took 1m12s + 2m17s, which makes quite a big difference.
(I tried serving the pre-tar'ed file using node-static, but ran out of memory on my embedded system)
Any idea what could make the express+node.js-solution so slow?
Update with some benchmarking:
I suspect that using another webserver would be better for static file serving, but I don't think it can explain the big difference here.
Benchmarking done on a folder resulting in a 64MB tar
Manual:
46 seconds
# on server:
tar -cf foo.rar <folder>
# on client
scp root@<ip>/path/to/foo.rar .
fstream + node-tar: directly
4 min 25 seconds
fstream.Reader({type: "Directory", path: dir}).pipe(tar.Pack({})).on("error", function(){res.statusCode = 500; res.end('Unable to create tarball');}).pipe(res);
fstream + node.tar: first streaming to file, then serving file
1 min 11 seconds
fstream.Reader({type: "Directory", path: dir}).pipe(tar.Pack({})).on("error", function(){res.statusCode = 500; res.end('Unable to create tarball');}).pipe(fstream.Writer('/tmp/test.rar').on("close", function() {
var rs = fs.createReadStream('/tmp/test.rar');
rs.on('data', function(data) {
res.write(data);
});
rs.on('end', function() {
res.end();
});
}));