I have some code using PlaywrightCrawler. I added "playwright-extra" with "stealthPlugin" to this code. Exactly as in documentation [1]
I added to my code only this:
Plain Text
import { firefox } from 'playwright-extra';
import stealthPlugin from 'puppeteer-extra-plugin-stealth';
firefox.use(stealthPlugin());
The rest of program remains the same as before. And I have useFingerprints: true and launcher: firefox in code.
Well, the code works. Bot detection sites report that my crawler has 3 plugins and supports 4 mime types, so something changed. But! I got this is the stdout:
Plain Text
INFO PlaywrightCrawler: Starting the crawler.
An error occured while executing "onPageCreated" in plugin "stealth/evasions/user-agent-override": TypeError: Cannot read properties of undefined (reading 'userAgent')
at Proxy.<anonymous> (.../node_modules/playwright-extra/src/puppeteer-compatiblity-shim/index.ts:217:23)
at runNextTicks (node:internal/process/task_queues:61:5)
at processImmediate (node:internal/timers:437:9)
at process.topLevelDomainCallback (node:domain:161:15)
at process.callbackTrampoline (node:internal/async_hooks:128:24)
at async Plugin.onPageCreated (.../node_modules/puppeteer-extra-plugin-stealth/evasions/user-agent-override/index.js:69:8)
It looks like I have to use the stealth plugin in order to bypass cloudflare captchas using firefox. I thought crawlee already implemented all of the features from the stealth plugin but it wont bypass cloudflare cpatchas on firefox without it. What does the stealth plugin have that crawlee doesnt?