Sign up for Apify Platform here
Star Crawlee on GitHub
Star Crawlee for Python on GitHub
Apify and Crawlee Official Forum
New post
View all posts
Related posts
Did this answer your question?
๐
๐
๐
Powered by
Hall
Inactive
Updated 3 months ago
0
Follow
custom logic for status codes
custom logic for status codes
Inactive
0
Follow
N
NeoNomade
6 months ago
ยท
Is it possible to do some custom logic based on status codes in the Puppeteer Crawler ? If yes, how ?
R
N
m
3 comments
Share
Open in Discord
R
Rado Ch.
6 months ago
You can check the status code in the
response
object from the
PuppeteerCrawlingContext
and do some logic based on that:
https://crawlee.dev/api/puppeteer-crawler/interface/PuppeteerCrawlingContext#response
N
NeoNomade
6 months ago
For example if the status code is 403 it gets retried before I have the chance to check this response object
m
microworlds
6 months ago
In the session pool options, you can specify what status codes should be treated as blocked requests. I think 403 is there by default, so passing an empty array will allow you full control here. Look this up -
https://crawlee.dev/api/core/interface/SessionPoolOptions#blockedStatusCodes
Add a reply
Sign up and join the conversation on Discord
Join on Discord