I have 99 urls in the queue. But scraper finishes crawl...

At a glance

The community member is experiencing an issue with a scraper that finishes crawling after a few URLs, even though they have 99 URLs added to the queue. The comments suggest that the issue may be due to non-unique URLs, and a community member provides a solution to use a different uniqueKey for each request. Additionally, a community member asks if they need to re-initialize the crawler to scrape the same URL again, and another community member responds that they need to set a different uniqueKey for each request.

Useful resources

NNisthar

The scraper finishes crawl after a few ones everytime. I have 99 urls added to the queue.

This is my config:

Attachments

9 comments

NNisthar

I add urls like this:

Plain Text

 const playRequestQ = await RequestQueue.open('q1');
    const urlsToAdd: { url: string; }[] = []
    nodeEvts.on('links', onLinks)

    async function onLinks(link: any;) {
        urlsToAdd.push({ url: `${link}` })
        if (urlsToAdd.length % 99 === 0) {
            await playRequestQ.addRequests(urlsToAdd)
            crawler.run();
        }
    }

AApifyBot

just advanced to level 2! Thanks for your contributions! 🎉

NNisthar

I can see the urls added inside the queue

HHonzaS

maybe urls are not unique

NNisthar

Ok that was the issue i think

NNisthar

Thanks 👍

ccurioussoul

what if I want to scrap the same url again and again ? do I need to re-initialize the crawler ?

HHonzaS

Then you need to have different uniqueKey for each request. By default uniqueKey is the same as url but you can set your own.
https://crawlee.dev/api/core/class/Request#uniqueKey

ccurioussoul

Thankyou very much

Add a reply

Apify Discord Mirror

I have 99 urls in the queue. But scraper finishes crawl after a few urls, why?