Apify Discord Mirror

Updated 5 months ago

requestQueue write costs

At a glance

The post indicates that the running cost of an actor is now predominantly determined by writing to the request queue, rather than the amount of CU it takes. Community members have questions about the cost of writing to the request queue, such as whether batching requests can reduce the cost, and whether using the new RequestQueueV2 (which is still experimental) could be more efficient and cheaper. Some community members suggest exploring alternative approaches like using an in-memory queue to store results. The community members are trying to come up with ideas to make the request queue writes more cost-effective.

Useful resources
Hi there,
was there a change how much it costs to write to the requestQueue?
Now running cost of the actor is determined predominantly by writing to the request queue and not how much CU it takes.
Attachment
image.png
2
H
A
S
9 comments
I am using crawler.addRequests shouldn't this be like one batch write? Or it just iterates over the requests and inserts one after the another under the hood?
can somebody reply also here please?
I have the same question!
It's crazy how much you need to pay for writting into a q. I can introduce my memory Q and store result there, but this is crazy.
can you please help here?
Its billing question, from dev point of view does not matter
In general usage, RQ is a costly operation.

Besides, what You see in "Usage" table under "Request queue writes", includes all operation with your RQ. "write" => is not only adding new req. to the queue. Could be operation with some already existing request (mark it as "processed"/ "failed", update retryCount value, etc..)
It's all "write to RQ".

So it's mainly about how efficient the logic in your actor is. Maybe there is a space for some improvement?

Also, Maybe try to use RQ v2
https://crawlee.dev/api/next/core/class/RequestQueueV2

It's still experimental, but worth checking. Could be not just faster, but a bit cheaper ?
We will try the new version.

Im trying to come up with an idea how to make it cheaper:

Batching made it cheaper a bit, not using the router for all cases but await function in one route.
Add a reply
Sign up and join the conversation on Discord