I am using PlaywrightCrawler with Firefox. When accessing
wellfound.com
and see this error:
DEBUG PlaywrightCrawler:SessionPool:Session: Could not set cookies. {"errorMessages":["Cookie not in this host's domain. Cookie:prod.website-files.com Request:wellfound.com"]}
It might be that this cookie is something important: I'm navigating to another page on this site and get HTTP 403 and captcha...
How to fix this error?
Have these settings in code:
useSessionPool: true,
persistCookiesPerSession: true,
sessionPoolOptions: {
maxPoolSize: 300,
sessionOptions:{
maxAgeSecs: 70,
maxUsageCount: 2,
},
},
launchContext: {
...
launchOptions: {
bypassCSP: true,
acceptDownloads: true,