probably there is a designed process to do that. But on a bigger scale, is a big difference on ram usage when closing page manually and auto handling this. For spiders between 5-10k pages where I tested, the difference in ram memory and speed is huge
The memory usage should scale with concurrency. If the memory is increasing but the concurrency is decreasing that can mean memory leak either in your code or in Crawlee. We need a reproduction in that case. Generally, you should be able to run crawler for days without problems
this is completely not what's happening -- i'm getting "memInfo":{"isOverloaded":true,"limitRatio":0.2,"actualRatio":1} even without running anything. Snapshotter is always leaking. I don't know how to reproduce.
I'm on holidays, I'll get back to my laptop later tonight. I have hundreds of Crawlee spiders running daily on Alpine (mostly Puppeteer and Cheerio) that will try to scale infinitely. If I don't limit max concurrency, AWS batch is killing the jobs automatically . I think the easiest way to reproduce, is to take the demo spider for the crawlee page and just replace the Dockerfile with an Alpine one. Then run the container with limited resources something like 2vcpu and 4gb of ram. It will be very easy to see that autoscailing fails
By default Crawlee is saying that if I not mention maxConcurrency it will scale up to available resources and I'm telling you that on many OSs it is not able to read the resources. I have containers killed with OOM if I don't use maxConcurrency based on previous benchmarking that I've done myself.
On Linux max requests per minute is never working. It's outputting a bunch of logs and it gets stuck (I'll provide an example in approx 8-10 hours)
Also not related to this, on Linux for the moment neither the request locking works properly.
Thank you for your explanation. If you would be able to provide minimal PoC repository, where this problem occur and we may investigate this, that would help a lot.
