I am trying to fetch the youtube videos for the data I scraped from officialcharts.com. I am able to scrape the data and get the youtube url's for them . But when I try pushing the data into the array only the title and the artist name is pushed. The variable youtubeUrl is not assigned the value. Why is the variable youtubeUrl not in scope ?
I tried multiple ways of doing it. I tried declaring youtubeUrl inside the search function and then outside the request but I get them same issue.
I also tried moving the json.tracks.push inside the search function but then json is not in scope so nothing is added to tracks
I am using https://www.npmjs.org/package/youtube-search for searching the videos.
// Scraping charts
app.get('/scrape', function(req, res){
url = 'http://www.officialcharts.com/music-charts/';
request(url, function(error, response, html){
if(!error){
var $ = cheerio.load(html);
var title, artist, youtubeUrl;
var json = {tracks:[]};
$('div .infoHolder').each(function(){
var data = $(this);
var opts = {
maxResults: 1,
startIndex: 1
};
// Scrapped title and artist name
title = data.children('h3').text();
artist = data.children('h4').text();
// Searching for youtube videos
search(title, opts, function (err, results) {
if (err) {
return console.log(err);
}
youtubeUrl = results[0].url;
})
console.log(youtubeUrl); // THIS IS RETURNED UNDEFINED
// youtubeUrl IS NOT PUSHED TO THE ARRAY
// ONLY TITLE AND ARTIST PUSHED
json.tracks.push({title:title,artist:artist,url: youtubeUrl});
})
}
res.send(json);
})
})
My guess is that you are defining the youtubeUrl variable outside of the .each() and then try to reach it from within your search() function.
Try declaring the youtubeUrl variable inside the .each() since it doesn't seem to need to hold data for too long. That should do the trick.
Edit: Forget what I just wrote, because I completely misread your code. Check what you get back in the search function. It seems to be asynchronous and thus the youtubeUrl var is set after you try to get its value. You can try and move your data collection to an external function that gets called after the search succeeded.
// Scraping charts
app.get('/scrape', function(req, res){
url = 'http://www.officialcharts.com/music-charts/';
function pushInfo(res,title, artist, youtubeUrl){
res.send({title:title,artist:artist,url: youtubeUrl});
};
request(url, function(error, response, html){
if(!error){
var $ = cheerio.load(html);
var title, artist, youtubeUrl;
var json = {tracks:[]};
$('div .infoHolder').each(function(){
var data = $(this);
var opts = {
maxResults: 1,
startIndex: 1
};
// Scrapped title and artist name
title = data.children('h3').text();
artist = data.children('h4').text();
// Searching for youtube videos
search(title, opts, function (err, results) {
if (err) {
return console.log(err);
}
//youtubeUrl = results[0].url;
pushInfo(res, title, artist, results[0].url);
});
});
}
});
});
This is completely untested since I don't actually know what the search function does but at least all info should be available to you once the search function completes with this method.