Apify Discord Mirror

Updated 2 months ago

Something is off when running from Apify

At a glance

A community member developed a NewEgg scraper actor and is having trouble, as they believe NewEgg has increased its anti-bot protection during the holiday season. The code works locally using residential proxies, but fails when running on the Apify platform. The community members discuss this issue, noting that the same code behaves differently when run locally versus on the platform. They speculate that the extra anti-bot measures may be temporary and related to the holiday season. One community member mentions that they were able to successfully scrape a specific NewEgg URL using a simple request, indicating that the issue may be temporary or specific to the community member's setup.

Useful resources
Hey,

So, I developed the NewEgg scraper actor, and I’m having some trouble, particularly because I think NewEgg upped their anti-bot protection during holiday season.

However, something weird is happening, when I run the code locally using your residential proxies, it sometimes fails, but generally it doesn’t, specially after I implemented UA Rotation.

But, from Apify (and from your maintenance runs), it always fails, this kinda indicates to me that it’s not being able to properly connect to your residential proxies via the platform.
A
R
H
10 comments
@Rorisjack just advanced to level 2! Thanks for your contributions! 🎉
Actually just tested it, proxy usage goes up via the platform,
Nonetheless the exact same code is failing when running locally vs the platform?
are you using automated browser?
I’m merely scraping using httpx and then BS4 to parse the HTML
Ok, so that is weird. Pure request should be the same.
right? it’s very weird
i suppose it should be temporary, i bet the extra anti-bot / anti-scraping measures are a “before christmas” thing at newegg
but still, it’s weird
I have tried this url https://www.newegg.com/asrock-rx6600-cld-8g-amd-radeon-rx-6600-8gb-gddr6/p/N82E16814930066 and it works even with simple request from insomnia
Add a reply
Sign up and join the conversation on Discord