Apify and Crawlee Official Forum

Updated last month

Max redirects

I am getting this error message, how to best deal with it?
Reclaiming failed request back to the list or queue. Redirected 10 times. Aborting.
Can I increase the max number of redirects for my CheerioCrawler?
O
1 comment
You don’t need to do anything with it. This usually means the enqueued URL redirects somewhere else (likely a different domain), so you wouldn’t get the desired data anyway. Crawlee automatically stops processing such URLs.

Also, just in case, there’s an option called followRedirect that might be helpful depending on your use case. It allows you to control whether redirects should be followed or not:
https://crawlee.dev/api/next/core/interface/HttpRequest#followRedirect

usage example:

Plain Text
const response = await gotScraping({
                url: 'https://www.example.com',
                method: 'POST',
                proxyUrl,
                body: `some_payload`,
                followRedirect: false
            }
Add a reply
Sign up and join the conversation on Discord