request queue data

Stewart McKinlay (CAL) · 2024-08-06T20:30:54.875Z

I'd like to access the request queue data that's stored in memory and being written to the storage directory. I want to add data to my output such as the url, handledAt, etc. How would I go about that? Thanks.

hey, the request data is stored in crawlingContext.request, which is accessible from your requestHandler:

Plain Text

const crawler = new PuppeteerCrawler({
    requestHandler: async (crawlingContext) => {
        const { url } = crawlingContext.request;
        ...
        await Dataset.pushData({ ...data, url });
    },
};

According to the docs (https://crawlee.dev/api/3.10/core/class/Request#handledAt), handledAt is set after the request is processed, so it'll be always undefined in your requestHandler. However, you can just use new Date().toISOString() to create your own timestamp.

Apify and Crawlee Official Forum

request queue data