I've 80 urls in an array ( The number can go upto 200 urls also ) from which I want to scrap images and save the url for the image whose width and height is more than 200.
I've 2 functions which are maily performing the job. The first function LoadNextImage() which increments the index of array and send the next url to the ImageExtracter function along with the index in order to get the image from that url.
Problem :
The problem is that it never loads all the pictures. Although I've called LoadNextImage() in the on error events also so that if some error occurs we start loading the next image. But the processing stops and no more images are loaded. The behaviour is unexpected due to which I am unable to identify to source of the problem. Sometimes it loads 5 images and stops, sometimes 10 images and stops.
As you can see in the screenshot. It has loaded 3 images after which it stopped unexpectedly without any error or warning whatsoever.
Kindly help me out.
I've given the defination for variables after the code in order to make it more clear.
// arg2: index will help us remember on which index to insert the image url in compared array.
ImageExtracter:function(url,index)
{
var self = this;
//will scrap images from the page.
this.requests(url, function(err, resp, body){
if (!err && resp.statusCode == 200) {
$ = self.cheerio.load(body);
$.totalimagecount = $('img').length;
$.imagecounteach = 0;
$.imagecountsizeof = 0;
if($.totalimagecount == 0)
{
self.LoadNextImage();
return false;
}
$('img').each(function (i,img)
{
$.imagecounteach++;
/* if(self.nonImgPosts[index].postimage!=null &&
self.nonImgPosts[index].postimage!==undefined){
// get out we got our postimage.
if(!self.nonImgPosts[index].gotImage)
{
self.nonImgPosts[index].gotImage = true;
self.LoadNextImage();
return false;
}
}
if(img.attribs.width!==undefined)
{
if(img.attribs.width > 200 && img.attribs.height > 200)
{
if(!self.nonImgPosts[index].gotImage)
{
self.nonImgPosts[index].postimage = img.attribs.src;
self.nonImgPosts[index].gotImage = true;
self.LoadNextImage();
return false;
}
}
else if($.imagecount == $.totalimagecount
&& self.nonImgPosts[index].gotImage == false)
{
self.LoadNextImage();
}
}
else
{
*/ if(img.attribs.src.trim().substr(0,5) != "https")
{
var request = self.http.get(img.attribs.src,
function (response) {
self.sizeOf(response, function (err, result) {
console.log("finding size...");
$.imagecountsizeof++;
if(err!="invalid")
{
if(result.width > 200 && result.height >200 && result.format!="gif")
{
if(!self.nonImgPosts[index].gotImage)
{
self.nonImgPosts[index].postimage = img.attribs.src;
self.nonImgPosts[index].gotImage = true;
// self.LoadNextImage();
request.abort();
return;
}
}
}
if($.imagecountsizeof >= $.totalimagecount
// && self.nonImgPosts[index].gotImage == false
&& self.nonImgPosts[index].called == false)
{
self.nonImgPosts[index].called = true;
// if all the images are looped and still
// we couldn't found the useful image i.e
// the gotImage property is false. Than
// call the next image since LoadNextImage function
// is not called due to gotImage = false.
// same condition is applied above.
request.abort();
self.LoadNextImage();
return;
}
});
}).on('error',function(e){
$.imagecountsizeof++;
if($.imagecountsizeof >= $.totalimagecount
// && self.nonImgPosts[index].gotImage == false
&& self.nonImgPosts[index].called == false)
{
self.nonImgPosts[index].called = true;
self.LoadNextImage();
request.abort();
}
});
// }
}
});
}
}).on('error',function(e){
console.log(e);
self.LoadNextImage();
// request.abort();
});
},
LoadNextImage:function()
{
if(this.finishcalled)
return false;
console.log("Image Number = "+this.imageloadindex);
// once we'll exceed the length of the array. No more post images to load. FInish up.
if((this.nonImgPosts.length-1) < this.imageloadindex)
{
this.finishCuration();
this.finishcalled = true;
return false;
}
this.ImageExtracter(this.nonImgPosts[this.imageloadindex].posturl,this.imageloadindex);
this.imageloadindex++;
},
this.nonImgPosts : The array containing the urls from which we've to scrap.
this.imageloadindex : The index of the array. This will be incremented each time the
LoadNextImage() is called such that we start loading the images from the next url. This helps us to load images from urls one by one and reduces the load on the server.
self.sizeof : refers to the this npm package. It is used by first loading image from url using
http package. Than finding the size of the response given by http using the sizeof package. As shown in the imagesize package link.