Adding couchdb persistence to a socketio json feed in node

Question

Adding couchdb persistence to a socketio json feed in node

I'm currently researching how to add persistence to a realtime twitter json feed in node. I've got my stream setup, it's broadcasting to the client, but how do i go about storing this data in a json database such as couchdb, so i can access the stores json when the client first visits the page?

I can't seem to get my head around couchdb.

var array = { 
    "tweet_id": tweet.id,
    "screen_name": tweet.user.screen_name,
    "text" : tweet.text, 
    "profile_image_url" : tweet.user.profile_image_url 
};

db.saveDoc('tweet', strencode(array), function(er, ok) {
    if (er) throw new Error(JSON.stringify(er));
    util.puts('Saved my first doc to the couch!');
});

db.allDocs(function(er, doc) {
    if (er) throw new Error(JSON.stringify(er));
    //client.send(JSON.stringify(doc));
    console.log(JSON.stringify(doc));
    util.puts('Fetched my new doc from couch:');
});

These are the two snippets i'm using to try and save / retrieve tweet data. The array is one individual tweet, and needs to be saved to couch each time a new tweet is received.

I don't understand the id part of saveDoc - when i make it unique, db.allDocs only lists ID's and not the content of each doc in the database - and when it's not unique, it fails after the first db entry.

Can someone kindly explain the correct way to save and retrieve this type of json data to couchdb? I basically want to to load the entire database when the client first views the page. (The database will have less than 100 entries)

Cheers.

javascript
json
node.js
couchdb

Answer 1

You need to insert the documents in the database. You can do this by inserting the JSON that comes from the twitter API or you can insert one status at a time (for loop)
You should create a view that exposes that information. If you saved the JSON directly from Twitter you are going to need to emit several times in your map function
There operations (ingestion and querying) are not the same thing, so you should really do them at the different times in your program.
You should consider running a bg process (maybe in something as simple as a setInterval) that updates your database. Or you can use something like clarinet (http://github.com/dscape/clarinet) to parse the Twitter streaming API directly.

I'm the author of nano, and here is one of the tests that does most of what you need:

https://github.com/dscape/nano/blob/master/tests/view/query.js

For the actual query semantics and for you learn a bit more of how CouchDB works I would suggest you read:

http://guide.couchdb.org/editions/1/en/index.html

I you find it useful I would suggest you buy the book :)

Answer 2

If you want to use a module to interact with CouchDB I would suggest cradle or nano.

You can also use the default http module you find in Node.js to make requests to CouchDB. The down-side is that the default http module tends to be a little verbose. There are alternatives that give you an better API to deal with http requests. The request is really popular.

To get data you need to make a GET request to a view you can find more information here. If you want to create a document you have to use PUT request to your database.