Apify and Crawlee Official Forum

Home
Members
Mr. Big Bua
M
Mr. Big Bua
Offline, last seen last week
Joined November 23, 2024
:perfecto: Hey!

Firstly I just want to thank you for creating such an amazing product ❤️ !

Question itself:

Regarding documentation (https://crawlee.dev/docs/guides/scaling-crawlers) we can set maxRequestsPerMinute limitation for global crawler process.
But for some cases, maxRequestsPerMinute should be set by session itself.

For example:
Website iammastrongwebsite.com have request limit per session (5rpm).

I have a bunch of proxies, and setting maxRequestsPerMinute to 5 not ideal, since all of my imaginative hundreds of proxies will be waiting without providing any payload

Of course, we can remember simple math, and make maxRequestsPerMinute = [SessionLimit] * 5, but it's actually could be worse, since this approach full breaks a purpose of autoscaling


Suggestion:
Add ability to set maxRequestsPerMinute in sessions itself.

Thanks again 😊
3 comments
E
M