Nodejs - How to maintain a global datastructure

So I have a backend implementation in node.js which mainly contains a global array of JSON objects. The JSON objects are populated by user requests (POSTS). So the size of the global array increases proportionally with the number of users. The JSON objects inside the array are not identical. This is a really bad architecture to begin with. But I just went with what I knew and decided to learn on the fly.

I'm running this on a AWS micro instance with 6GB RAM.

How to purge this global array before it explodes?

Options that I have thought of:

  1. At a periodic interval write the global array to a file and purge. Disadvantage here is that if there are any clients in the middle of a transaction, that transaction state is lost.
  2. Restart the server every day and write the global array into a file at that time. Same disadvantage as above.
  3. Follow 1 or 2, and for every incoming request - if the global array is empty look for the corresponding JSON object in the file. This seems absolutely absurd and stupid.

Somehow I can't think of any other solution without having to completely rewrite the nodejs application. Can you guys think of any .. ? Will greatly appreciate any discussion on this.

I see that you are using memory as a storage. If that is the case and your code is synchronous (you don't seem to use database, so it might), then actually solution 1. is correct. This is because JavaScript is single-threaded, which means that when one code is running the other cannot run. There is no concurrency in JavaScript. This is only a illusion, because Node.js is sooooo fast.

So your cleaning code won't fire until the transaction is over. This is of course assuming that your code is synchronous (and from what I see it might be).

But still there are like 150 reasons for not doing that. The most important is that you are reinventing the wheel! Let the database do the hard work for you. Using proper database will save you all the trouble in the future. There are many possibilites: MySQL, PostgreSQL, MongoDB (my favourite), CouchDB and many many other. It shouldn't matter at this point which one. Just pick one.

I would suggest that you start saving your JSON to a non-relational DB like http://www.couchbase.com/.

Couchbase is extremely easy to setup and use even in a cluster. It uses a simple key-value design so saving data is as simple as:

couchbaseClient.set("someKey", "yourJSON")

then to retrieve your data:

data = couchbaseClient.set("someKey")

The system is also extremely fast and is used by OMGPOP for Draw Something. http://blog.couchbase.com/preparing-massive-growth-revisited