I have a strange problem-
Because of the strange way that this site presents time data, I wanted to write a small parser.
I was testing my code on one specific url -
http://www.sfweekly.com/search/results/?keyword=*&type=events#type:events/page:57/
Note that when you visit the url, the page first loads a bunch of entries and then changes those entries. What's happening there is that it's going to the first page and then re-directing. How do I get around that?
To scrape I'm using
jsdom.env({
html: url,
scripts:['http://code.jquery.com/jquery.js'],
done: function(errors,window){
//doSomething
I originally thought I could get around this with a pause, but that's not the case. Is there some way I can 'listen' for a redirect and wait until the real page has been loaded? I also have a feeling that the new entries may be entered with a jquery replace, but I'm not sure how to test that theory.
Scraping ajax-y sites like this can be a real pain. In this case, it seems there is a way around it, because you can snoop around in the developer tools in your browser of choice and discover the ajax endpoint, and use that directly:
http://www.sfweekly.com/search/ajaxsearch/type%3aevents/page:57/
In some scenarios, javascript-y sites that intentionally try to foil scrapers, you have to use some kind of headless or automated browser situation. That's slow and annoying, avoid it if you can.