I've inherited legacy code which is causing issues lately. I was wondering if the approach I need to take is the right one before updating the source code.
Currently, we have 11 different services that need to listen to 11 different ports. As it's implemented at the moment, there are 11 separate nodejs instances running on one server, where each one uses express to listen to a different port. Each nodejs instance is using the cluster module, having one master and three workers. So in total, there are 11 "master" nodejs processes, and 33 "worker" nodejs processes running on the same server at once.
As an entry point, we have nginx listening to port 80, and redirects the traffic to the specific port based on the server name.
One of the service receives about 10,000 requests per minute during peak, and some of the requests are timing out (not being served). The requests which are being served have an average response time of about 10 ms. There are no errors or exceptions in the logs.
I've tried several different approaches such as: using a larger instance, tweaking the number of worker processes, tuning the ipv4 stack, etc.. and still have the same issue.
From what I've read, the proper way to scale nodejs involves having only one nodejs instance using cluster, and have it listen to all 11 ports.
Has anyone encountered something similar or has any tips/ideas on fixing this issue? My approach of having only one nodejs instance would require a bit of code change, which could introduce bugs. So, if I can avoid changing source code, I would rather fix the current implementation.
Thanks