Let's say we want to read in a 1GB file to be uploaded to a network endpoint. The only issue is, before making the network upload request, we have to make another network request to get a token to allow the upload to go through.
So, if someone's code looked like...
fs.createReadStream('1-gb-file').pipe(myWritableStream);
...should I be worried about data buffering up in memory before my writestream says it's ready (after having fetched a token)?
What I've found is a fs.createReadStream
readable has a highWaterMark
of 64kb. Does this mean the max in-buffer limit be 64kb, before more data is sucked out of the readable?
Will data start being read (beyond the highWaterMark) from the readable only after I call this.emit('writable')
(or some other trigger)? Edit: maybe the trigger is simply calling the callback to the first call that _write receives?
I'm quite new to the streams api, so help is appreciated. I've searched around for similar questions, and couldn't find any. Sad face.
Thanks!