I'm trying to write a program that downloads a large zip file, unzips it in memory, and then pushes the contents (a series of CSV files) to MongoDB. However, I keep hitting a point where the program halts and prints
FATAL ERROR: CodeRange::GetNextAllocationBlock Allocation failed - process out of memory
I've been setting buffers to null when they're no longer in use, setting records to null once they're in Mongo, and preventing more than one file from processing at a time. None of this has helped. Are there any more tricks to releasing memory?
Somethings I would consider would be(not sure if it will all work out as desired):
Make sure references are gone:
Performing GC manually and increase v8 heap size:
Spawning(and killing) child process from parent process to do work:
That way I think the OS will reclaim memory for child process even if it is not returning memory.
Ram as filesystem:
That way you can treat filesystem as memory(let operating system put it into memory).
Use freelist:
node.js has a hard 1.4Gb limit for memory allocations. But I dont think that downloading large files to memory is good idea anyway, I suggest writing file to disk, unzip to disk, and then push to MongDB. This way you are always will be operating just small chunks of data in memory