Apify

Apify and Crawlee Official Forum

Updated last year

Rate limit based on key

TTay
·
Is it possible to rate limit based on a key? So basically it would only process 1 URL at a time per key.
g
H
2 comments
ggtry
you can try maxConcurrency option but it wouldn't be actual rate limiting though
HHamza
There is no native way at the moment.

An idea is to start with one URL for each key, and pass the rest of the URLs to the request.userData object, then after each URL is processed, add another one from request.userData to the queue.
Add a reply
Sign up and join the conversation on Discord