Image processing server using NodeJS

I have a NodeJS server that is serving out web requests, users can also upload images to my server. I am currently processing these images on the same server. This was really a stop gap solution until I could do something better as the image processing eats up a lot of CPU, and I don't want to do it on my web server.

Type of processing is very simple:

  1. Auto orient the image, POST the image to the processing server and have it return an oriented image.
  2. Create thumbnails, POST the image to the processing server and have it return a thumbnail.

I would like to be able to use this image processing server for many other servers to hit. I also want it to run with little memory/cpu. I.E. if I have more CPU/memory available then great but I don't want it to crash if deployed on a server with limited resources.

The way I currently handle this is I have a config file the specifies how many images I can process concurrently. If I have very limited resources I will only process 1 image at a time while queuing the rest. This works fine right now cause the web server and image processing server is one in the same.

My standalone image processing server will have to 'queue' requests such that it can limit the amount of images it is processing at a time.

I don't really want to store the images on the processing server. Mainly because my web server knows the data structure (which images belong with which content). So I really want to just make a request to a server and get back the processed image.

Some questions.

  1. Is a NodeJS implementation a good solution?
  2. Should this be a RESTful http api. I am wondering about this because I am not sure I can pull it off given the requirement of only processing a certain amount of images at a time. If I get 15 requests at the same time I am not sure what to do here, I don't want to process the first while the others wait for a response (processed image).
  3. Could use sockets to connect my web server with my image processing server to avoid question #2. This isn't as 'featureful' as it may be harder to use with other servers. But does get me out of the request/response problem I have in #2 above. IE web server could push image to be processed on socket and processing server could just answer back on the socket whenever it is done.
  4. Is there some opensource solution that will run on linux/unix that will suit my needs. This seems like a pretty common problem, should be fun to implement but would hate to reinvent the wheel if I could use/contribute to another product. I have seen Apace::ImageMagick, but I cannot POST images they have to reside on the server already. There are also some windows only solutions that I cannot use.

Please help...

Why not break down the process into discrete steps.

  1. User uploads image. Gets some kind of confirmation message that image has been successfully uploaded and is being processed.

  2. Place request for image processing in queue.

  3. Perform image processing.

  4. Inform user that image processing is complete.

This fits your requirements in that users can upload many images in a short period of time without overloading the system (they simply go into the queue), image processing can happen on a separate server without moving the files)