Apify and Crawlee Official Forum
b
F
A
J
A
1,148
Apify Platform Forum
Crawlee JavaScript Forum
Crawlee for Python Forum
Sign up for Apify Platform here
Star Crawlee on GitHub
Star Crawlee for Python on GitHub
Powered by
Hall
Crawlee JavaScript Forum
0
Join
on Discord
custom logic for status codes
Crawlee JavaScript Forum
custom logic for status codes
0
Join
on Discord
N
NeoNomade
5 months ago
Is it possible to do some custom logic based on status codes in the Puppeteer Crawler ? If yes, how ?
R
N
m
3 comments
Share
R
Rado Ch.
5 months ago
You can check the status code in the
response
object from the
PuppeteerCrawlingContext
and do some logic based on that:
https://crawlee.dev/api/puppeteer-crawler/interface/PuppeteerCrawlingContext#response
N
NeoNomade
5 months ago
For example if the status code is 403 it gets retried before I have the chance to check this response object
m
microworlds
5 months ago
In the session pool options, you can specify what status codes should be treated as blocked requests. I think 403 is there by default, so passing an empty array will allow you full control here. Look this up -
https://crawlee.dev/api/core/interface/SessionPoolOptions#blockedStatusCodes
Add a reply
Sign up and join the conversation on Discord
Join
on Discord