:perfecto: Hey!
Firstly I just want to thank you for creating such an amazing product ❤️ !
Question itself:Regarding documentation (
https://crawlee.dev/docs/guides/scaling-crawlers) we can set
maxRequestsPerMinute limitation for global crawler process.
But for some cases,
maxRequestsPerMinute should be set by session itself.
For example:Website
iammastrongwebsite.com have request limit per session (5rpm).
I have a bunch of proxies, and setting
maxRequestsPerMinute to
5 not ideal, since all of my imaginative hundreds of proxies will be waiting without providing any payload
Of course, we can remember simple math, and make maxRequestsPerMinute = [SessionLimit] * 5, but it's actually could be worse, since this approach full breaks a purpose of autoscaling
Suggestion:Add ability to set
maxRequestsPerMinute in sessions itself.
Thanks again 😊
