Apify

Apify and Crawlee Official Forum

b
F
A
J
A

got many 429 status code when crawled the target site,even though proxies. How to optimise my code?

Hi,guys. I am a python coder but not good at nodejs. I make a crawler to bulk check the infomation by the crawlee.

This is my option :
useSessionPool: true,
useIncognitoPages: true,

and I using the Residential Proxies .
But I found that the ip of some pages as same as the others.
I want launch diffrent browser context in each target url . But I don't know how to do that.
Anybody could help me ?

Sorry for my poor English .
Thanks
Attachment
2023-08-19_202459.webp
N
h
2 comments
https://crawlee.dev/api/core/interface/SessionOptions

Set maxErrorScore to 1 and a new proxy from your proxy config will be assigned after each error
Thanks for your help. I will test your method.
Add a reply
Sign up and join the conversation on Discord
Join