Apify

Apify and Crawlee Official Forum

b
F
A
J
A

custom logic for status codes

Is it possible to do some custom logic based on status codes in the Puppeteer Crawler ? If yes, how ?
R
N
m
3 comments
You can check the status code in the response object from the PuppeteerCrawlingContext and do some logic based on that: https://crawlee.dev/api/puppeteer-crawler/interface/PuppeteerCrawlingContext#response
For example if the status code is 403 it gets retried before I have the chance to check this response object
In the session pool options, you can specify what status codes should be treated as blocked requests. I think 403 is there by default, so passing an empty array will allow you full control here. Look this up - https://crawlee.dev/api/core/interface/SessionPoolOptions#blockedStatusCodes
Add a reply
Sign up and join the conversation on Discord
Join