Apify

Apify and Crawlee Official Forum

b
F
A
J
A

How to use MemoryStorage (mainly for RequestQueue) on the platform?

My actor runs typically use Cheerio, take <20m and have around 1k requests.
For this scenario, costs for RequestQueue writes/reads are often higher than compute units. I wanted to experiment with using in-memory storage to optimize costs (i think I understand the associated risks and I'm ok with them).
I've tried setting storage: new MemoryStorage() in Actor.main second argument as noted in docs & TS definitions, but actor runs on platform still seems to use "platform RQ", not "in-memory one". Any pointers?

https://console.apify.com/actors/64sLcqgxq4IB5hZrI/runs/QIkqpBDa846Ftn5xK#storage
Attachment
unknown.png
L
s
3 comments
Good question, asking the Crawlee gods
So you should be able to do it if you call this after Apify.init()
Configuration.getGlobalConfig().useStorageClient(new MemoryStorage())
Thanks both to Discord god and Crawlee gods :yay:
I managed the cut the costs to half – it's just a cents, but multiplied with number of actors and schedule, it will some me some money I can spend on buying bikes πŸ˜„
Attachment
image.png
Add a reply
Sign up and join the conversation on Discord
Join