Apify Discord Mirror

Updated 2 weeks ago

Routing issue

At a glance

The community member has a listing website and is using enqueueLinks to add links (case studies) to the crawler. However, when the crawler adds the links with a new label, nothing happens. When using only the case study page, the data is scraped and working. The community member is unsure what to do next and how to test it further. They also wonder if the queue system waits to complete adding all links before starting the scraping.

In the comments, another community member suggests providing a minimal reproducible example and sample URLs to help investigate the issue. They also mention that enqueueLink usually adds new links to the end of the queue, and the community member can use forefront: true to change this behavior, as discussed in the linked GitHub issue.

Another community member confirms that the forefront setting was indeed the solution.

Useful resources
I have a listing website as INPUT and enqueueLinks of it. These links (case studies) at the time has also multiple pages. When the cralwer adds the links with the new label attached, it's not happening anything. When using only case study page, it's scrapping the data and working. Not sure, what to do next and how to test it more. Does the Queue System waits to complete to add all links to start scrapping?
Marked as solution
Hi @Scai would it be possible to put together some minimal reproducible example, when it happens? And probably an example of such a URLs. enqueueLink usually put new link on the end of the queue. you may use forefront: true to change it https://github.com/apify/crawlee/issues/389 .
View full solution
P
S
2 comments
Hi @Scai would it be possible to put together some minimal reproducible example, when it happens? And probably an example of such a URLs. enqueueLink usually put new link on the end of the queue. you may use forefront: true to change it https://github.com/apify/crawlee/issues/389 .
Yeah, from what I see, it was indeed forefront setting.
Add a reply
Sign up and join the conversation on Discord