The community member is using Crawlee in Kubernetes and wants to make their jobs resumable if a pod gets evicted. They have set up a persistent volume (PV) for storage, but this poses an issue if they have multiple pods running at once. To solve this, the community member wants to change the storage directory programmatically when instantiating Crawlee, rather than using environment variables.
The comments suggest that the community member can define storageClientOptions in the Configuration constructor, where they can use the MemoryStorageOptions. Then, they can pass the Configuration object to the crawler constructor. This appears to be the answer to the community member's question.
I want to use crawlee in kubernetes. I want my jobs to be resumable if a pod gets evicted so I have set up a PV for storage. This however poses an issue if I have multiple pods running at once. To solve this I want to change the storage dir programaticly when instanciating crawlee. I know I can do this through ENV vars however id prefer a more programatic solution if possible?
I have looked at the constructors for the (Playwright) Crawler and the Configuration class but I dont seem to be able to set it.
You can define storageClientOptions in the Configuration constructor, where you can use the MemoryStorageOptions, see the links above. Then, you can pass the Configuration object to the crawler constructor.
