Implementing general purpose long polling

I've been trying to implement a simple long polling service for use in my own projects and maybe release it as a SAAS if I succeed. These are the two approaches I've tried so far, both using Node.js (polling PostgreSQL in the back).

1. Periodically check all the clients in the same interval

Every new connection is pushed onto a queue of connections, which is being walked through in an interval.

var queue = [];

function acceptConnection(req, res) {
  res.setTimeout(5000);
  queue.push({ req: req, res: res });
}

function checkAll() {
  queue.forEach(function(client) {
    // respond if there is something new for the client
  });
}

// this could be replaced with a timeout after all the clients are served
setInterval(checkAll, 500);

2. Check each client at a separate interval

Every client gets his own ticker which checks for new data

function acceptConnection(req, res) {
  // something which periodically checks data for the client
  // and responds if there is anything new
  new Ticker(req, res);
}

While this keeps the minimum latency for each client lower, it also introduces overhead by setting a lot of timeouts.

Conclusion

Both of these approaches solve the problem quite easily, but I don't feel that this will scale up easily to something like 10 million open connections, especially since I'm polling the database on every check for every client.

I thought about doing this without the database and just immediately broadcast new messages to all open connections, but that will fail if a client's connection dies for a few seconds while the broadcast is happening, because it is not persistent. Which means I basically need to be able to look up messages in history when the client polls for the first time.

I guess one step up here would be to have a data source where I can subscribe to new data coming in (CouchDB change notifications?), but maybe I'm missing something in the big picture here?

What is the usual approach for doing highly scalable long polling? I'm not specifically bound to Node.js, I'd actually prefer any other suggestion with a reasoning why.