I am using ReadLine to read a very large file line by line so as to be able of detecting places where I can split it into meaningful pieces into several other files. I use a WriteStream to write to those files. When a regular expression detects the place for the cut, I end() the currently open WriteStream and open a new one for the line just read. The problem is that all those files end up truncated as if the end() call didn't do its job.
I checked for the success of write() and pause the input stream when the buffer is full. This is the relevant part of my code:
var rl = readline.createInterface({
input: fs.createReadStream('basesystem.js',{encoding:'utf8'}),
output: process.stdout
});
rl.on('line', function (line) {
var match = rexp.exec(line);
if (match) {
out.end();
console.log(path.join(PATH, match[2],'.js'));
out = fs.createWriteStream(path.join(PATH,match[2]) + '.js', {encoding:'utf8'});
}
if (!out.write(line + '\n')) {
console.log('***** paused **** ');
rl.pause();
out.once('drain', function () {
console.log('***** resume **** ');
rl.resume();
});
}
});
When I find the write() fails, I puase the input stream on ReadLine and listen for the 'drain' event when I resume it. I tried leaving the listener for 'drain' permanent, but it seems that the lines from ReadLine keep coming. All the files end up truncated. ReadLine doesn't seem to stop sending 'line' events even when paused.
The initial WriteStream (out) is created before this sample and I listen to the 'close' event on the ReadLine stream to end() the final file, but anyway, all files, not just the last one, get truncated.
The display shows all the ** pause * signs on the console along the lines being read but no resume . All the resume ** lines come all at once at the end.
Thanks in advance
I found out, it is ReadLine's fault. When ReadLine receives a pause(), it pauses its own input stream but if it has a buffer already split in lines, it will keep emitting 'line' events for each of the lines already split from the previous buffer. It will only stop when that array of lines is finished. I've filed the bug report on Node's site.