Please look at the following image, from http://mongoexplorer.com/:
I've been trying to work through GridFS, referencing https://github.com/jamescarr/nodejs-mongodb-streaming. The files I uploaded, come back nicely and the stream that comes back via the following get function looks right.
var gridfs = (function () {
function gridfs() { }
gridfs.get = function (id, fn) {
var db, store;
db = mongoose.connection.db;
id = new ObjectID(id);
store = new GridStore(db, id, "r", {
root: "fs"
});
return store.open(function (err, store) {
if (err) {
return fn(err);
}
return fn(null, store);
});
};
return gridfs;
})();
Using http://mongoexplorer.com/ I uploaded files into GridFS to test with, but they seem broken when I use the node code above to retrieve them.
That is when I noticed the filename / fileName thing. Looking here /node_modules/mongodb/lib/mongodb/gridfs/gridstore.js I saw the reference to filename with a lowercase 'N', but in my GridFS, it's fileName with a capital 'N'.
OK, so just for kicks, I changed it to lowercase in GridFS, but I still get some corruption in the stream (node code above) when retrieving files uploaded with http://mongoexplorer.com/. Clicking Save as... in http://mongoexplorer.com/, however brings back my fine just perfectly.
To get back to my question, (since my tests didn't seem to prove anything,) I am wondering which is it: filename with a lowercase 'N', or fileName with 'N' in caps?
Please use the latest mongodb native driver as there are a ton of fixes for GridFS, there is a ton of examples in the github directory for the driver under the tests for usage of GridFS as streams as well.
Docs are at
http://mongodb.github.com/node-mongodb-native
In general I would say that if you use the core functionalities stick to the driver as the one you are using it using a driver that' way of of date which explains you corruption issues.
Another Windows tool nl. MongoVue also looks for filename instead of fileName. I'd say the answer is more likely filename instead of fileName.
With retrieving the small Windows file from GridStore, I found a bug, but I don't know how to fix it. I guess there must be some value like Chunk.CurrentSize or the like, but looking at the chunk.js file in the native node mongo driver https://github.com/mongodb/node-mongodb-native/blob/master/lib/mongodb/gridfs/chunk.js, I did the following...
I found this:
Chunk.prototype.readSlice = function(length) {
if ((this.length() - this.internalPosition + 1) >= length) {
var data = null;
if (this.data.buffer != null) { //Pure BSON
data = this.data.buffer.slice(this.internalPosition, this.internalPosition + length);
} else { //Native BSON
data = new Buffer(length);
length = this.data.readInto(data, this.internalPosition);
}
this.internalPosition = this.internalPosition + length;
return data;
} else {
return null;
}
};
and moved the following
data = this.data.buffer.slice(this.internalPosition, this.internalPosition + length);
into the this if statement (1024 * 256 is the value from Chunk.DEFAULT_CHUNK_SIZE = 1024 * 256;)
if (this.data.buffer != null) { //Pure BSON
if (this.data.buffer.length > 1024 * 256) {
// move to here
}
else
{
data = this.data.buffer;
}
like so:
Chunk.prototype.readSlice = function(length) {
if ((this.length() - this.internalPosition + 1) >= length) {
var data = null;
if (this.data.buffer != null) { //Pure BSON
if (this.data.buffer.length > 1024 * 256) {
data = this.data.buffer.slice(this.internalPosition, this.internalPosition + length);
}
else
{
data = this.data.buffer;
}
} else { //Native BSON
data = new Buffer(length);
length = this.data.readInto(data, this.internalPosition);
}
this.internalPosition = this.internalPosition + length;
return data;
} else {
return null;
}
};
The issue with windows files smaller than the chunk size is solved, but this isn't the most elegant solution. I'd like to propose this as the answer, but I realize using the default chunk size hard coded isn't the dynamic value which would make this less of a workaround ;-)