Using `transformRequestFunction` in `enqueueLinks` over...

I am using enqueueLinks to pass some URLs to another handler. This works fine when I do:

Plain Text

await enqueueLinks({
    label: "LIST_PLACES",
    urls: searchSquareLinks,
    strategy: "same-domain",
    userData,
})

but as soon as I add a transformRequestFunction the label is overridden and it queues the links back to the handler from which is is being queued:

Plain Text

await enqueueLinks({
    label: "LIST_PLACES",
    urls: searchSquareLinks,
    transformRequestFunction: (request) => {
        request.userData = {
            ...userData,
            zoomTarget: request.url.match(zoomRegex)?.[1],
        }
        return request
    },
    strategy: "same-domain",
    userData,
})

Why is the label being overridden when the only property of request being changed is the zoomTarget?

5 comments

LLukas Krivka

I assume the userData is the current userData? That might be a bug in Crawlee since the label is piped theough userData. Can you please copy this to https://github.com/apify/crawlee/issues?

ccdslash

Thanks, have submitted here: https://github.com/apify/crawlee/issues/2247

ccdslash

This was resolved - for anyone else coming across this, request.label is a shortcut for request.userData.label so the transform function was overwriting the label, but I couldn't see it because I thought label was a separate property.

bbmax

Nice!

bbmax

I'm actually experiencing another issue with same-domain strategy not working as expected and I wonder if it's becuase I'm also overriding w/ transformRequestFunction

Add a reply

Apify and Crawlee Official Forum

Using `transformRequestFunction` in `enqueueLinks` overrides `label`