Apify and Crawlee Official Forum

Updated 3 months ago

Memory usage is spiking, even with CRAWLEE_AVAILABLE_MEMORY_RATIO set to .7

Any ideas on how to solve this?
O
L
b
4 comments
Can You please provide any logs / links to your run / examples of mentioned issue?
Limit maxConcurrency or the supplied memory. You can also slow down the scaling up but it is kinda trying stuff out
I have the container giving 8gb, do I have to tell it that it only has 8bb? I thought available_memory_ratio would handle.
That controls the concurrency so it should slow down while reaching CPU/memory/event loop utilization. But spikes can still happen if you have bad luck.

First look at the logs, is concurrency rapidly increasing before you see the spike? We need more insight into what those spikes are
Add a reply
Sign up and join the conversation on Discord