MongoDB refusing connections on high volume ( 10000 ) inserts / writes

We are inserting data through a Node.js application and connecting to mongodb using the node-mongodb-native library. Mongo DB is sharded with a total of 5 shards. The node and mongos instance is contained on a quad core AWS instance with 14980MB of RAM. Two shards and the config server are contained in a single core instance and the remaining three shards are on a different instance with a single core processor.

We can effectively insert 1000 records concurrently in 9 seconds using a Node.js test script.

For 10000 concurrent inserts, about 2000 of them fail and return the following error message:

    [Error: failed to connect to [ip address of mongos]]

During this 10000 insert process, the first 5000 or 6000 inserts are successful with no error. During this time, the CPU usage remains relatively low (13% usage). Then the CPU on one core jumps to about 75% and MongoDB throws the error message refusing connections. After some failures to insert, some writes are intermittently succesful.

We've set ulimit -n to 20000 on linux. We have passed the poolSize=5 parameter within the insert function in our Node.js api.

var responseHandlers = require('./responseHandlers')
    ,schemaValidation = require('./schemaValidation')
    ,mongoDb = require('mongodb')
    ,md5 = require('MD5')
    ,ObjectID = require('mongodb').ObjectID;

function insert(data, timestamp, response) {
  // validating the data to be inserted
  schemaValidation.validate(data, function(err) {
    if(err) {
      console.log(err);
      responseHandlers.invalidRequest(response, 2); 
    } else {
      //console.log("opening db..");
      server = new mongoDb.Server(mongoConfig.host,mongoConfig.port,{'auto_reconnect': true, 'poolSize': 5});
      db = new mongoDb.Db(mongoConfig.database, server, {w: 1});
      db.open(function(err, db) {
        if(err) { 
          console.log(err);
          responseHandlers.invalidRequest(response, 2); 
        } else {
          db.collection(mongoConfig.collection, function(err, collection) {
            if(err) {
              console.log(err);
              responseHandlers.invalidRequest(response, 2); 
            } else {
              //going to instantiate document fields
              var time = new Date().getTime(),
              oid = new ObjectID(),
              hash = md5(oid.toHexString()),
                            obj = {'data_utc' : timestamp, 'server_utc' : time, '_id' : oid,    'hash' : hash}; 
                            obj.data = data;
              //inserting..
              collection.insert(obj, {w:1}, function(err, result) {
                if(err) {
                  console.log(err);
                  responseHandlers.invalidRequest(response, 2); 
                } else {
                  console.log('Insert successful');
                  responseHandlers.validRequest(response, false, result);
                }   
              db.close();
              }); 
            }   
          }); 
        }   
      });   
    }   
  }); 
}   
exports.insert = insert;

Can anyone help with why it would be refusing connections, failing to insert? Why does it work for a portion of the inserts then fail about halfway through?

To troubleshoot this issue you should observe the number of connections that are open in the mongod console. Based on the code sample you provided you are probably opening a new mongodb connection on each insert. To correct this try moving the instantiation of your 'server' outside the scope of the control loop that you're using to invoke this function.