Apify and Crawlee Official Forum

Updated 2 years ago

How to increase max memory?

I am running a script that needs concurrency. I have 64 GB of RAM available and I want to use it to the max. I am running my script on a server so there is not much else running. The problem is, at around 15GB I always get memory overloaded error.

I have tried:

Plain Text

config.set('memoryMbytes', 50_000)
config.set('availableMemoryRatio', 0.95)

Nothing seems to change this behavior. Anything else I can try?

1 comment

PPepa J

Hello Jeno, based on https://sdk.apify.com/api/apify/interface/ConfigurationOptions#memoryMbytes it seems to me, that in your case, there is still the default memory limit being used.

Have you tried to run the script with CRAWLEE_MEMORY_MBYTES environmental variable being set?

Add a reply