Distributed queue wrapped with https request/response

Looking for some advice on how to do the following:

  1. Receive request from website for certain long running process (~10-30seconds)
  2. Website backend schedules a job and puts onto distributed queue .. could be SQS/Kue/resque
  3. A worker takes the job off the queue and processes it. Stores result somewhere.
  4. Website backend subscribes to job complete event and gets the result of processed job.
  5. Website backend closes request to website with result of the task.

1,2 and 3 are fine. I am just finding it tricky to pass the result of a queued task back to the backend so that it can close the request.

Polling from the website isnt an option - the request has to stay open for however long the task takes to be processed. I'm using nodejs.

2 - 4 are all happening on the server side. There is nothing stopping you from polling the expected result location (on the server side) for the result and then returning the result when it finally appears.

  1. Client sends requests
  2. Server starts job and begins polling for the result
  3. The result comes back so the poll loop on the server side ends
  4. Server sends result back to client
  5. The client-server connection is finally severed

You could get even more efficient code going if the job can execute a url when it finishes. In this case your service would have two endpoints... one for the client to start the process, and another that your job queue can call.

  1. Client sends requests

  2. Server starts job... saves the response callback in a global object so that it is not closed (I'm assuming something like express here)

    openJobs.push({ id: 12345, res: res }); jobQueue.execute({ id: 12345, data: {...}});

  3. When the job finishes and saves the result, call the service url with the id

  4. You can check that the job has actually finished and remove the job from the openJobs list

  5. Finish the original response

    openJob.res.send(data);

  6. This will send the data and close the original client-server connection.

The overall result is that you have no polling at all... which is cool.

Of course... In either of these scenarios you are screwed if your server shuts down in the middle of a batch... This is why I would recommend something like socket.io in this scenario. You would queue the results of jobs somewhere and socket.io would poll/wait for callbacks on the list and push to the client when there are new items. This is better because if the server crashes no biggie - the client will re-connect once the server comes back up.