Apify and Crawlee Official Forum
b
F
A
J
A
1,148
Apify Platform Forum
Crawlee JavaScript Forum
Crawlee for Python Forum
Sign up for Apify Platform here
Star Crawlee on GitHub
Star Crawlee for Python on GitHub
Powered by
Hall
Apify Platform Forum
0
Join
on Discord
per-site interval between requests?
Apify Platform Forum
per-site interval between requests?
0
Join
on Discord
n
new_in_town
2 years ago
Imagine the request queue of Crawlee (PlaywrightCrawler) containing URLs of two (or more) sites:
example.com/url1
another-site.com/url2
example.com/url3
another-site.com/url4
...
I would like to configure Crawlee to have per-site interval between requests. For the above example it means:
example.com: 20 sec (or more) between requests
another-site.com: 60 sec (or more) between requests
How to do this with Crawlee?
H
1 comment
Share
H
HonzaS
2 years ago
just add put wait function in
preNavigationHooks
Add a reply
Sign up and join the conversation on Discord
Join
on Discord