nodejs pipe stdin into zlib inflate without waiting for EOF

Question

nodejs pipe stdin into zlib inflate without waiting for EOF

I having trouble with realtime decompressing of data streaming from stdin (using nodejs). The requirement is to handle the decompressed stream as soon as it arrives to stdin (a few milliseconds latency at most). The problem is that piping stdin to zlib seams to wait for the stream to close.

The following prints 12345

$ echo 12345 | node deflate.js | node inflate.js
12345

However the following commandline does not since it does not receive EOF:

$ node generator.js | node deflate.js | node inflate.js

This relates to the question if zlib deflate can internally handle partial input, or should it be added into the stream (such as size of chunk before each stream chunk).

deflate.js:

process.stdin
.pipe(require("zlib").createDeflate())
.pipe(process.stdout);

inflate.js

process.stdin
.pipe(require('zlib').createInflate())
.pipe(process.stdout)

generator.js:

var i = 0
setInterval(function () {
    process.stdout.write(i.toString())
    i++
},1000)

node.js
zlib

Answer 1

The problem was that Z_SYNC_FLUSH flag was not set:

If flush is Z_SYNC_FLUSH, deflate() shall flush all pending output to next_out and align the output to a byte boundary. A synchronization point is generated in the output.

deflate.js:

var zlib = require("zlib");
var deflate = zlib.createDeflate({flush: zlib.Z_SYNC_FLUSH});
process.stdin.pipe(deflate).pipe(process.stdout);

inflate.js:

var zlib = require('zlib')
process.stdin
.pipe(zlib.createInflate({flush: zlib.Z_SYNC_FLUSH}))
.pipe(process.stdout)