How to use MemoryStorage (mainly for RequestQueue) on t...

sstrajk

My actor runs typically use Cheerio, take <20m and have around 1k requests.
For this scenario, costs for RequestQueue writes/reads are often higher than compute units. I wanted to experiment with using in-memory storage to optimize costs (i think I understand the associated risks and I'm ok with them).
I've tried setting storage: new MemoryStorage() in Actor.main second argument as noted in docs & TS definitions, but actor runs on platform still seems to use "platform RQ", not "in-memory one". Any pointers?

https://console.apify.com/actors/64sLcqgxq4IB5hZrI/runs/QIkqpBDa846Ftn5xK#storage

Attachment

3 comments

LLukas Krivka

Good question, asking the Crawlee gods

LLukas Krivka

So you should be able to do it if you call this after Apify.init()
Configuration.getGlobalConfig().useStorageClient(new MemoryStorage())

sstrajk

Thanks both to Discord god and Crawlee gods :yay:
I managed the cut the costs to half – it's just a cents, but multiplied with number of actors and schedule, it will some me some money I can spend on buying bikes 😄

Attachment

Add a reply

Apify and Crawlee Official Forum

How to use MemoryStorage (mainly for RequestQueue) on the platform?