I would like to start using Node.js for a project I'll be working on.
I have streaming position information in an electric car, and I'd like to store this locally on a small database in the car and synchronize it over a (fragile) 3G connection, when this connection is up, to a server with a large DB. So:
Car Os 3G Server OS
Data stream ---> ------ ---> ---------
Node.js Node.js
MongoDB MongoDB
Small DB Large DB
The part that I'm worried about is how to sync the local DB to the server DB over a non-continuous 3G connection. What Node.js module is perfect for this job?
Thank you for your advice!
I believe CouchDB would be a good tool for the job. It has built in synchronization capabilities and is often use to synchronize offline/online web apps and mobile apps. I'm not sure it supports the "big db"/"small db" architecture out of the box though.
From Wikipedia:
CouchDB was designed with bi-direction replication (or synchronization) and off-line operation in mind. That means multiple replicas can have their own copies of the same data, modify it, and then sync those changes at a later time.
Hope this can help you a little.
MongoDB as well has built-in replication between databases.
As with any database when one member goes offline for a long period of time, you'll need to make sure that data (which was recorded in the no-signal time) will be able to synchronize.
In MongoDB uses an internal collection capped called oplog which stores historical data of logical writes.
By default MongoDB will allocate 5% of the available free disk space to the oplog [1]. This means you'll have this amount of oplog 'time' for the replication to occur between the time of the last operation on the Server DB and the first on the Car DB that booth servers where sync on.
Of course if you want to send a lot of data or have many cars, you can change the size of the oplog.
So, this option would mean that as long as your mongodb servers can connect to each other you'll just need to insert data to them and they'll do the syncronization for you.
Depending on your requirements you might don't want to leave the Server Database port open to the world. This means authentication will come into place or restrictions to hostnames (using Dynamic DNS).
In terms of Node modules that would help here, i won't be able to help but do try using the mongooplog command.
And please do let us know how your project progresses as it seems very interesting.