Apify and Crawlee Official Forum

Updated 3 months ago

requestQueue write costs

Hi there,
was there a change how much it costs to write to the requestQueue?
Now running cost of the actor is determined predominantly by writing to the request queue and not how much CU it takes.
Attachment
image.png
2
H
A
S
9 comments
I am using crawler.addRequests shouldn't this be like one batch write? Or it just iterates over the requests and inserts one after the another under the hood?
can somebody reply also here please?
I have the same question!
It's crazy how much you need to pay for writting into a q. I can introduce my memory Q and store result there, but this is crazy.
can you please help here?
Its billing question, from dev point of view does not matter
In general usage, RQ is a costly operation.

Besides, what You see in "Usage" table under "Request queue writes", includes all operation with your RQ. "write" => is not only adding new req. to the queue. Could be operation with some already existing request (mark it as "processed"/ "failed", update retryCount value, etc..)
It's all "write to RQ".

So it's mainly about how efficient the logic in your actor is. Maybe there is a space for some improvement?

Also, Maybe try to use RQ v2
https://crawlee.dev/api/next/core/class/RequestQueueV2

It's still experimental, but worth checking. Could be not just faster, but a bit cheaper ?
We will try the new version.

Im trying to come up with an idea how to make it cheaper:

Batching made it cheaper a bit, not using the router for all cases but await function in one route.
Add a reply
Sign up and join the conversation on Discord