I have collection with around 2.5million records. Currently I'm using nodejs with mongoose. I'm open to any suggestions which would improve efficiency of current search.
Artist collection schema is something like that:
var Artist = new mongoose.Schema({
realname: String,
names: Array
});
User passes string for example: Michael Jackson was contacted by police or Michael Jackson-Jillie bean. Now I have to find artist/singer/person so the only logical thing I could find was: Iterate through all documents in collection, check if any of names is in given string if YES we have match -> stop loop.
But this is very memory inefficient since it has to load whole collection to memory so nodejs can iterate through and check all records.
It seems that retrieving whole collection takes most of time. Any way to speed this up? In mongoshell db.collection.find() is pretty fast. But using mongoose in nodejs takes waaay too long.
What you need is a full-text search. Possible options:
text index. (new in 2.4)