Apify

Apify and Crawlee Official Forum

b
F
A
J
A

per-site interval between requests?

Imagine the request queue of Crawlee (PlaywrightCrawler) containing URLs of two (or more) sites:

example.com/url1
another-site.com/url2
example.com/url3
another-site.com/url4
...

I would like to configure Crawlee to have per-site interval between requests. For the above example it means:

example.com: 20 sec (or more) between requests
another-site.com: 60 sec (or more) between requests

How to do this with Crawlee?
H
1 comment
just add put wait function in preNavigationHooks
Add a reply
Sign up and join the conversation on Discord
Join