Apify

Apify and Crawlee Official Forum

Updated 3 months ago

Crawler only working in headed mode.

NNeoNomade
·
I have a Puppeteer Crawler that works almost flawless in headed mode, but if I go headless all the requests are getting 403 errors.
I was thinking that xvfb should fix this but unfortunately it doesn't. Any other ideas ?
1
A
N
c
10 comments
AAlexey Udovydchenko
Please share run, I guess its site specific, they using blocking of some kind
NNeoNomade
hipercor.es/supermercado
ccurioussoul
did you find any solution ?
OOleg V.

Can't find your actor via "hipercor.es/supermercado"

Some suggestions:
  1. try to test different proxy groups
  2. try to modify (change some) headers (as in original requests in the browser)
  3. Try to use Playwright + firefox. It's more stealthy (btw, why are You using pptr?)
NNeoNomade
  1. fixed it, I fixed it for a short time with xvfb
  2. will try
  3. will also try, I'm using pptr because I find it easier to code
OOleg V.
  1. Why do You think so?)
I would say, that Playwright is improved pptr. And for sure it's easier to bypass anti-bot protection with Playwright.
NNeoNomade
will try with that one too
NNeoNomade
coded playwright in the past but found easier the workflow in puppeteer :))
ccurioussoul
how did you fix it with XVFB ?
NNeoNomade
Didn’t fix it, it worked with playwright and firefox
Add a reply
Sign up and join the conversation on Discord