Apify

Apify and Crawlee Official Forum

b
F
A
J
A
Join

Crawlee not working(?) on a page with shadow dom

MMichal
Hey, I've encountered a website using shadow dom, where crawlee isn't able to find elements (for a good reason).
https://developer.mozilla.org/en-US/docs/Web/Web_Components/Using_shadow_DOM

I was wondering since there is no mentions of shadow dom if anyone knows what to look at to make it work?
M
A
8 comments
MMichal
It looks like this might be the answer https://stackoverflow.com/questions/64368789/puppeteer-clicking-button-in-shadowroot, going to try this with crawlee
MMichal
Okay, that doesn't work. Chromiu has the "Copy JS path" option disabled for that element.
MMichal
official Puppeteer discussion https://github.com/puppeteer/puppeteer/issues/858
MMichal
Looks like as long as the shadow root is attached with the closed option, it will be inaccessible from the page's JS, neat.
AApifyBot
just advanced to level 1! Thanks for your contributions! 🎉
MMichal
Since the page is opening a consent dialog in the shadow dom, I can just ignore it for now. But if the whole page was encased int it that might be a problem.
MMichal
One way to get through to the shadow dom would be to use Chrome DevTools https://chromedevtools.github.io/devtools-protocol/tot/DOM/
MMichal
Someone even used the DevTools protocol to intercept traffic, very cool https://jsoverson.medium.com/using-chrome-devtools-protocol-with-puppeteer-737a1300bac0
Add a reply
Sign up and join the conversation on Discord
Join