Apify and Crawlee Official Forum

Updated 3 weeks ago

`maxRequestsPerMinute` But for session

:perfecto: Hey!

Firstly I just want to thank you for creating such an amazing product ❤️ !

Question itself:

Regarding documentation (https://crawlee.dev/docs/guides/scaling-crawlers) we can set maxRequestsPerMinute limitation for global crawler process.
But for some cases, maxRequestsPerMinute should be set by session itself.

For example:
Website iammastrongwebsite.com have request limit per session (5rpm).

I have a bunch of proxies, and setting maxRequestsPerMinute to 5 not ideal, since all of my imaginative hundreds of proxies will be waiting without providing any payload

Of course, we can remember simple math, and make maxRequestsPerMinute = [SessionLimit] * 5, but it's actually could be worse, since this approach full breaks a purpose of autoscaling


Suggestion:
Add ability to set maxRequestsPerMinute in sessions itself.

Thanks again 😊
E
1 comment
Hi, you can try to use rotate proxy
Add a reply
Sign up and join the conversation on Discord