Apify and Crawlee Official Forum

Updated last month

Something is off when running from Apify

At a glance
Hey,

So, I developed the NewEgg scraper actor, and I’m having some trouble, particularly because I think NewEgg upped their anti-bot protection during holiday season.

However, something weird is happening, when I run the code locally using your residential proxies, it sometimes fails, but generally it doesn’t, specially after I implemented UA Rotation.

But, from Apify (and from your maintenance runs), it always fails, this kinda indicates to me that it’s not being able to properly connect to your residential proxies via the platform.
A
R
H
10 comments
@Rorisjack just advanced to level 2! Thanks for your contributions! 🎉
Actually just tested it, proxy usage goes up via the platform,
Nonetheless the exact same code is failing when running locally vs the platform?
are you using automated browser?
I’m merely scraping using httpx and then BS4 to parse the HTML
Ok, so that is weird. Pure request should be the same.
right? it’s very weird
i suppose it should be temporary, i bet the extra anti-bot / anti-scraping measures are a “before christmas” thing at newegg
but still, it’s weird
I have tried this url https://www.newegg.com/asrock-rx6600-cld-8g-amd-radeon-rx-6600-8gb-gddr6/p/N82E16814930066 and it works even with simple request from insomnia
Add a reply
Sign up and join the conversation on Discord