We currently have a NodeJS program collecting data from a socket that gets sent realtime information from one our other services.
This data is then stored in a Database by the afore mentioned instance. This process simply listens for the stream. Parses the objects out and then updates the database with the information.
Another service can then simply read the data from the database. Which is what our users poll for the info.
My question is:
How would I make sure 2 instances of this code is running and kept running in 2 separate data centres ? So if one instance goes down the other simply starts updating the database. While we fix the other process.
Here what I was thinking:
Was thinking of having 2 PosgresSQL servers running, one in each data centre. Running in Master and Standby mode. This will enabled the web services in both data centres to read the data from the database but not update it while the master is still alive. Giving me the only one update that I want.
But the other problem is what if the NodeJS instances dies. I then have no way of knowing ? I could implement a heartbeat system but there has to be a better ?
Also considered running a Rabbit MQ service with a queues, but that gives me a single point of failure.
Been thinking around this for a while but can't seem to find a architecture that would work. Any advice ?
Basically one way or another you need to have some sort of Shoot The Other Node In The Head solution (affectionately called STONITH). This can be launched from some sort of heartbeat program or manually. Ideally you have two aspects to it:
Shoot the other node in the head. Take it out of the picture. Make it stop.
Direct traffic to running node.
If I was trying to do this, I would look at several layers. In certain types of failures (let's call them fatal failures) your nodejs instance would start the failover process and then exit. If it was verified to stop responding, heartbeat would start the failover and kill the process.