Duplicate chunks on JSON response

Question

Duplicate chunks on JSON response

I've got a fairly standard MEAN project setup with the angular-fullstack generator using yeoman.

What I'm finding is that when GETting a largish (over 65536 bytes) json result, it is encoded using gzip and chunked, but the json returned is not valid viewed either in chrome or consumed by my angular client $resource because it contains TWO responses! e.g {name:'hi'}{name:'hi'} for a single id or [{..},{..}][{..},{..}] for a array.

The server api endpoint was autogenerated from the angular-fullstack generator and looks something like:

// Get list of worlds
exports.index = function(req, res) {
  World.find(function (err, worlds) {
    if(err) { return handleError(res, err); }
    res.json(200, worlds);
  });
};

If i slice the data so it's not chunked, then the json is well formed. I've checked the mongo db and the data is ok there too and debugging the worlds variable, I can JSON.stringify and get the expected string result without any duplicates. but the moment it's sent, I'm getting a doubling up of json result on the response.

Update for comment

angular-fullstack 2.0.4

the schema looks like:

'use strict';

var mongoose = require('mongoose'),
    Schema = mongoose.Schema;

var WorldSchema = new Schema({
  name: String,
  info: String,
  active: Boolean,
  tiles: [Schema.Types.Mixed]
});

module.exports = mongoose.model('World', WorldSchema);

seeded with:

 var newWorld = new WorldModel({
                    _id: planet._objectId,
                    name: "SimDD World",
                    tiles : seed()
                });
                newWorld.save();

...

var seed = function () {
    var data = [];
    for (var i = 0; i < planet.HEIGHT; i++) {
        for (var j = 0; j < planet.WIDTH; j++) {
            data.push({
                coords:{
                    x:i,
                    y:j
                },
                type:'.'
            });
        }
    }
    return data;
}

javascript
json
node.js
angularjs
chunked-encoding

Answer 1

Looks like this is being caused by the compression middleware, removing app.use(compression()); from the express config seems to fix this.

Answer 2

Does this work for you? I don't see a reason why it shouldn't.
I assume you have a planet object that has:
HEIGHT, WIDTH and _objectId properties.

Remember if you modify a mixed type you need to tell mongoose that the value changed and subsequently save it.
http://mongoosejs.com/docs/schematypes.html#mixed

var WorldModel = require('../api/world/world.model');
var planet = require('planetSeedData');

var seed = function() {
  var data = [];
  for (var i = 0; i < planet.HEIGHT; i++) {
    for (var j = 0; j < planet.WIDTH; j++) {
      data.push({
        coords: {x:i, y:j},
        type: '.'
      });
    }
  }
  return data;
};

var myPlanet = {
  _id: Mongoose.Types.ObjectId(planet._objectId),
  name: "SimDD World",
  tiles : seed()
};               

WorldModel.create(myPlanet);

// if modified, you would do something like:
// WorldModel.markModified('tiles');
// WorldModel.save();

Answer 3

The issue is seen in browsers and not in postman. I checked the HTTP request headers and when I add 'Accept' Header as html in postman the same problem is seen in postman as well. So I believe the browsers are handling differently with Accept type with html.

Answer 4

// app.use(require('connect-livereload')());

I came across the same problem when building my angular-fullstack app (thanks, DaftMonk), after some extensive debugging using node-inspector, turns out the JSON data gets passed to the livereload module and gets duplicated when it comes out. Disabling this middleware eliminated the problem for me.