Backbone.relational, real-time and handling large data

Question

Backbone.relational, real-time and handling large data

I'm building a real-time feed application using Backbone.js, node.js and socket.io.

My Feed is a collection of Update models. Displaying these, overriding Backbone.sync for integration with socket.io works fine.

The complication comes in that each Update has a set of comments associated with it. When I show each Update in the Feed view, I want to show a summary of the associated comments (number of comments and a single 'most poular' comment), and also have the ability to click through to a different view to display each Update on its own with a paginated list of comments with further data.

I'm using backbone-relational to model the relationship between the Update model and Comment model, as follows:

Feed (collection) -> Update (model) -(has many)-> Comment (model)

I've been following this backbone-relational tutorial, but it seems to assume that I'd want to have all related data in memory at once in my Feed view, which I don't as there are potentially thousands of comments updating in real-time:

http://antoviaque.org/docs/tutorials/backbone-relational-tutorial/

My questions are:

How can I bring in summary data for comments to each Update in my Feed view without loading all comment data, and also maintain the ability to show paginated full data in my Update view?
I'm using backbone.layoutmanager for rendering my views. How best should I break my views up to accomplish the above?

node.js
backbone.js
socket.io
backbone-relational

Answer 1

For Q1:

I'm assuming you're using something like ioSync to use socket.io in Backbone.sync instead of REST API, or a similar solution.
Include metadata (such as # of comments) as an attribute on Update. If your Update object is heavy weight in itself, you could update the count using ioBind and custom server-side socket.io events instead of sending the whole object every time.
Include an attribute topComment as an additional one-to-one relation in Update. When initially loading Update from the server, include topComment in the response, but not the other comments.
Lazy-load the rest of the comments using custom socket.io events. You will likely want a server-side handler that takes as parameters updateId, startIndex, maxComments, which returns a list of comments for the given Update starting at the given index. If the result is sent to the client as JSON, then it's easy to do something like this on the client:
```
// Assume `model` is an instance of `Update`.
socket.emit('get_comments_page', {
  updateId: model.get('id'),
  startIndex: 1,
  maxComments: 10
}, function(err, data) {
  if (err) {
    alert('Unable to fetch comments: ', err);
  } else {
    model.get('messages').reset(data)
  }
});
```
Avoid sending ID for all comments when fetching Update then trying to use fetchRelated to resolve them. I learned this one the hard way :O/
You could also store the comments collection directly on the view without associating it as relationship of Update

For Q2:

I don't have any experience with layoutmanager as I use Backbone.Marionette for managing my views. Marionette has an async extension (disclaimer: I'm a co-maintainer). I encourage to see how Marionette.async does the delayed rendering, waiting for the data to arrive from the server.

The main idea is to use jquery's Deferred objects that resolve when the data comes back from the server. Extending the above example with deferred:

var MyView = Backbone.View.extend({
  // ... normal stuff that views need ...
  initialize: function() {
    var deferred = $.Deferred();
    // Assume `model` is an instance of `Update`.
    var that = this;
    socket.emit('get_comments_page', {
      updateId: that.model.get('id'),
      startIndex: that.options.pageNumber,
      maxComments: 10
    }, function(err, data) {
      if (err) {
        alert('Unable to fetch comments: ', err);
      } else {
        that.model.get('messages').reset(data)
      }
      deferred.resolve();
    });
    this.promise = deferred.promise();
  },

  render: function() {
    var that = this;
    this.promise.done(function() {
      // Do your normal rendering code here, for instance:
      $(that.el).html(that.template(that.model.toJSON()));
    });
    return this;
  }
});

Note: the code snippets above are not tested as is.