mongodb: building a job queue

Question

mongodb: building a job queue

I am building a job queue (Nodejs+Mongodb)which has this kind of structure :

{
    "_id" : ObjectId("53e09abd23bbaeea776598e1"),
    "createDate" : ISODate("2014-08-05T08:50:05.643Z"),
    "status" : "new",
    ......
}

I have defined 4 statuses in my system as : new, waiting, working, failed, done

Requirement: I need to pick the most recent job in the job queue with status as "new".

Currently, I am doing this by using

jobQueue.findAndModify(
        {
          status: 'new',
        },
        {date: 1},
        {$set: { ... } },
        {remove: false},
        function (err, job) {......

which works fine when the job documents are less. But there have been situations when document nos. rises to 50k. Then the locking period (using mongostat) goes beyond 100%(at times reaches 200% ). Hence my server is not able to deliver the new jobs in such scenario.

Can anybody suggest me a more efficient way to this query??

Thanks in advance :)

node.js
mongodb
job-queue

Answer 1

you can update them in batches

Can you tell the exact error that you are getting This is a pseudo code in coffeescript

mongoskin = require('mongoskin')

db = mongoskin.db("localhost:27017/your_db",  {journal: false})

db.collection("your_collection").find({"status" : "new"}).sort({"createDate" : -1}).toArray (err, doc) ->
    if not err? and doc? and doc.length
        console.log doc.length
        #doc is a sorted list of objects with latest createDate 
        #do your job here
        #I am assuming if you want to take the latest job and mark it done after processing the job.
        #you can use async module to complete each job and update the whole batch.
db.close()

PS: https://github.com/caolan/async