Is there any better way to handle the nested callbacks in Node.js/Expressjs?

Maybe a more elegant way would just be to put all of your various possible functions into a single object and call whichever one you need based on the state you're in. Kind of like

var States = {
   1: checkuser,
   2: resendEmail,
   3: getSysIds,
   4: createUser,
   5: whateverIsNext 
}

Then you just have to have one function that does everything

function exec(req,res,state,_this,callback){
   States[state].apply(this, arguments) 
}

And each of your functions would just set the state argument to whatever it needed to be, then recall exec, for example

function checkUser(req,res,state,_this,callback){
    var doneRight = do_whatever_you_need;
    state = doneRight? state++: state; //just set state to whatever number it should be
    exec(req,res,state,_this,callback);
}

This way you've got your list of steps nice and readable/changeable. You've got your one function that executes everything, and each step just has to manage what the next step should be. If you wanted, you could also replace the numbered steps with named, which would make it more readable, but you'd lose the ability to increment/decrement your step variable.

You are overengineering here mixing middlewares, database layer and route handlers. Let's make it simpler and more reusable.

First of all, lets make sure, that our middlewares and route handlers can communicate with each other. That could be accomplished with session middleware:

// Somewhere inside configuration:
app.use(express.cookieParser()
app.use(express.session({secret: 'cookie monster'));

// If you don't want sessions, change with this:
app.use(function (req, res, next) {
  req.session = {};
  next();
});

What you want now is to have user info everywhere during request handling. For now, it's just email and whether it registered or not. That would be first stand-alone middleware. Let's assume for brevity that you detect whether user registred or not by querying db with email from query string.

// Fills `req.session.user` with user info (email, registration). 
function userInfo (req, res, next) {
  var user = req.session.user = req.session.user || {};
  user.email = req.query.email;  // I'll skip validation.

  // I'm not sure which mongo driver you are using, so this is more of a pseudo-code.
  db.users.findOne({email: user.email}, function(err, bson) {
    if (err) {
      return next(err);  // Express 3 will figure that error occured
                         // and will pass control to error handler:
                         // http://expressjs.com/guide.html#error-handling
    }

    // Could remember information from DB, but we'll just set `registred` field.
    user.registred = !!bson;
    next();
  });
}

After determining registration status you should either send email or create new user, that will be request handler (last param for app.get). Prior to executing handler we want our userInfo to run (can pass just userInfo (not array), but I personally like this way more):

app.get('/register', [userInfo], function (req, res) {
  var user = req.session.user;
  var fn = user.registred ? sendActivationEmail
                          : registerUser;

  // If signatures differs or you need different reply for cases...
  // Well, you'll figure.
  fn(user.email, function (err) {
    return res.send(err ? 500 : 200);
  });
});

Request handler should not be bothered with what's going on inside this two functions (how many db queries, who is sending emails and how). All it knows is that first argument of provided callback is not null if error happens.

Now about creating new user. Write registerUser function which will query table with IDs, prepare and save user info in DB. For how to synchronize this two operations, see async and other SO questions. When creation is done or errored, call callback with results. This function should be part of DB layer. At least, create db.js module and export important stuff like registerUser; querying user information by email (inside first middleware) should also live inside DB layer.

Similar applies to sendActivationEmail.

Side Note: you can put whatever you want inside _id in mongo documet, not just ObjectId.