We have been developing social media based Rails application which digs data of various services like twitter, facebook, linkedin and many more. We are facing a serious performance issue to consume these services.
For example, We are accessing Twitter user_timeline to prepare some statistics. As twitter has imposed limitation on number of tweets can be fetched, we have to use pagination to fetch all tweets which results in multiple API calls and blocking IO. Hence the user has to wait for many seconds which would not be acceptable UX.
We are new to Ruby/Rails and need suggestions to architect/design consuming such API(Facebook, LinkedIn) rightly. Please help us.
Update We have been exploring ruby(delayed_job, eventmachine) and non-ruby(node.js) solutions.
Normally In rails world, login running processes such as you explained above are done through background services.
background services are like threads in Java. you the workflow would be
and there are several background process gems delayed_job, or sidekiq