:perfecto: Hey!
Firstly I just want to thank you for creating such an amazing product ❤️ !
Question itself:Regarding documentation (
https://crawlee.dev/docs/guides/scaling-crawlers) we can set
maxRequestsPerMinute
limitation for global crawler process.
But for some cases,
maxRequestsPerMinute
should be set by session itself.
For example:Website
iammastrongwebsite.com
have request limit per session (5rpm).
I have a bunch of proxies, and setting
maxRequestsPerMinute
to
5
not ideal, since all of my imaginative hundreds of proxies will be waiting without providing any payload
Of course, we can remember simple math, and make maxRequestsPerMinute = [SessionLimit] * 5
, but it's actually could be worse, since this approach full breaks a purpose of autoscaling
Suggestion:Add ability to set
maxRequestsPerMinute
in sessions itself.
Thanks again 😊