Using userData in the queue from the API

At a glance

The community member is trying to interact with the Apify API directly, without using JavaScript or Python. They need to store arbitrary data in request items in the queue, and have seen they can use the userData field when posting a request. However, when getting a request from the head of the queue, the response does not contain the userData field. Instead, they get the request ID and have to make a second API call to get the details for a specific request. The community member is wondering if there is a better way to do this, and why the "Get head" endpoint doesn't return the complete request (including userData).

In the comments, another community member suggests that the community member can't do this in one request, but they can fetch multiple request IDs using the "Get head" call, which might help to optimize the solution. The community member is curious about why the row is filtered to return only some values.

Another community member says they will try to get some information from the platform team. Finally, a community member from the platform team apologizes for the delayed answer and explains that the userData field is intentionally not returned in the "Get head" response because it can be very large in size, and there are currently no plans to change this.

Useful resources

JJerome

Not using JS or python so I have to interact with the Apify API directly.
I need to store arbitrary data in request items in the queue, I've seen I can use the userData field when posting a request, but when getting a request from the head (https://docs.apify.com/api/v2#/reference/request-queues/queue-head/get-head) the response does not contain this userData field. Instead i get the request id from this response, and have to make a second API call to get the details for a specific request (https://docs.apify.com/api/v2#/reference/request-queues/request/get-request) based on the request ID.
That's 2 API calls for 1 item from the queue, is there a better way?
Why doesn't the Get head endpoint return the complete request (including userData)?
I probably missed something there, thanks for your help.

4 comments

vvojtechmaslan

Hello ,

you can't do this in one request, but you can fetch multiple request IDs using the get head call, which might help to optimize your solution.

If you'd be interested, here is the request fetching logic from crawlee:
https://github.com/apify/crawlee/blob/master/packages/core/src/storages/request_queue_v2.ts#L100

JJerome

Hi , thanks for your answer.
Yes I read the python version of that in the apify SDK.
I'm curious at to why the row is filtered to return only some values?

vvojtechmaslan

I will try to get some info from the platform team.

vvojtechmaslan

Hello , I apologize for the delayed answer. This is intentional because userData field can be very big in size. There is currently no plan to change this.

Add a reply

Apify Discord Mirror

Using userData in the queue from the API