Apify Discord Mirror

Updated 5 months ago

per-site interval between requests?

At a glance

The community member is using Crawlee (PlaywrightCrawler) and has a request queue containing URLs from two or more sites. They would like to configure Crawlee to have a per-site interval between requests, such as 20 seconds between requests for example.com and 60 seconds between requests for another-site.com. A community member suggests adding a wait function in the preNavigationHooks to achieve this.

nnew_in_town

Imagine the request queue of Crawlee (PlaywrightCrawler) containing URLs of two (or more) sites:

example.com/url1
another-site.com/url2
example.com/url3
another-site.com/url4
...

I would like to configure Crawlee to have per-site interval between requests. For the above example it means:

example.com: 20 sec (or more) between requests
another-site.com: 60 sec (or more) between requests

How to do this with Crawlee?

1 comment

HHonzaS

just add put wait function in preNavigationHooks

Add a reply