Apify and Crawlee Official Forum

Updated 2 months ago

Trying to get rid out of 429 errors

Hello, do you have some nice tips how to get rid out of 429? I am not exactly how parrarellism working here, but I am afraid, even I am putting in the sleep the process, the other parallel request as considered as request and they can lead to 429, is there any nice tip/best practice how can I defend against it ? πŸ˜„
M
1 comment
Hi @tomasj97

You can limit the maximum parallelism. And the number of tasks per minute max_concurrency=100, max_tasks_per_minute=10 - https://crawlee.dev/python/docs/guides/scaling-crawlers
Add a reply
Sign up and join the conversation on Discord