I am having a really hard time wrapping my head around how to stream data back to my client when using Nodejs/Expressjs.
I am grabbing a lot of data from my database and I am doing it in chunks, I would like to stream that back to the client as I get the data such that I do not have to store the entire dataset in memory as a json object before sending it back.
I would like the data to stream back as a file, ie I want the browser to ask my users what to do with the file on download. I was previously creating a file system write stream and stream the contents of my data to the file system, then when done I would send the file back to the client. I would like to eliminate the middle man (creating tmp file on file system) and just stream data to client.
app.get(
'/api/export',
function(req, res, next) {
var notDone = true;
while (notDone) {
var partialData = // grab partial data from database (maybe first 1000 records);
// stream this partial data as a string to res???
if (checkIfDone) notDone = false;
}
}
);
I can call res.write("some string data"), then call res.end() when I am done. However I am not 100% sure that this is actually streaming the response to the client as I write. Seems like expressjs is storing all the data until I call end and then sending the response. Is that true?
What is the proper way to stream strings chunks of data to a response using expressjs?
The response object is already a writable stream. Express handles sending chunked data automatically, so you won't need to do anything extra but:
response.send(data)
You may also want to check out the built-in pipe method, http://nodejs.org/api/stream.html#stream_event_pipe.
You can do this by setting the appropriate headers and then just writing to the response object. Example:
res.writeHead(200, {
'Content-Type': 'text/plain',
'Content-Disposition': contentDisposition('foo.data')
});
var c = 0;
var interval = setInterval(function() {
res.write(JSON.stringify({ foo: Math.random() * 100, count: ++c }) + '\n');
if (c === 10) {
clearInterval(interval);
res.end();
}
}, 1000);
// extracted from Express, used by `res.download()`
function contentDisposition(filename) {
var ret = 'attachment';
if (filename) {
filename = basename(filename);
// if filename contains non-ascii characters, add a utf-8 version ala RFC 5987
ret = /[^\040-\176]/.test(filename)
? 'attachment; filename="' + encodeURI(filename) + '"; filename*=UTF-8\'\'' + encodeURI(filename)
: 'attachment; filename="' + filename + '"';
}
return ret;
}
Also, Express/node does not buffer data written to a socket unless the socket is paused (either explicitly or implicitly due to backpressure). Data buffered by node when in this paused state may or may not be combined with other data chunks that already buffered. You can check the return value of res.write() to determine if you should continue writing to the socket. If it returns false, then listen for the 'drain' event and then continue writing again.