Using userData in the queue from the API

JJerome

Not using JS or python so I have to interact with the Apify API directly.
I need to store arbitrary data in request items in the queue, I've seen I can use the userData field when posting a request, but when getting a request from the head (https://docs.apify.com/api/v2#/reference/request-queues/queue-head/get-head) the response does not contain this userData field. Instead i get the request id from this response, and have to make a second API call to get the details for a specific request (https://docs.apify.com/api/v2#/reference/request-queues/request/get-request) based on the request ID.
That's 2 API calls for 1 item from the queue, is there a better way?
Why doesn't the Get head endpoint return the complete request (including userData)?
I probably missed something there, thanks for your help.

4 comments

vvojtechmaslan

Hello ,

you can't do this in one request, but you can fetch multiple request IDs using the get head call, which might help to optimize your solution.

If you'd be interested, here is the request fetching logic from crawlee:
https://github.com/apify/crawlee/blob/master/packages/core/src/storages/request_queue_v2.ts#L100

JJerome

Hi , thanks for your answer.
Yes I read the python version of that in the apify SDK.
I'm curious at to why the row is filtered to return only some values?

vvojtechmaslan

I will try to get some info from the platform team.

vvojtechmaslan

Hello , I apologize for the delayed answer. This is intentional because userData field can be very big in size. There is currently no plan to change this.

Add a reply

Apify and Crawlee Official Forum

Using userData in the queue from the API