ref with String in mongoose

I have two models

 var Passenger = mongoose.model('passengers', new Schema({
        username: {
            type: String,
            ref: 'users'
        },
        company: String,
        baggage: String,
        note: String,
        owner: String
    }));


    var User = mongoose.model('users', new Schema({
        username: String,
        first: String,
        last: String,
        email: String,
        password: String
    }));

I really would like to get a full user instead of a passenger with a username. I thought ref and populate would be the way to do this.

Trip.find({
        request: true
    }).populate('flights.users').exec(function (err, trips) {

});

The top level document is Trip and Flights is an array under Trips and Passengers is an array under Flights.

Is this not possible. Would I have to give my users all ID's instead?

Thanks for the help.