Target page, context or browser has been closed
. It appears at the ~ hour mark is when this first presents itself and then incrementally increases in frequency until I'm getting more failed records than successful (at which point, I kill the cluster or restart it).browserPoolOptions
like retireBrowserAfterPageCount: 100
and closeInactiveBrowserAfterSecs: 200
await crawler.teardown();
in hopes that this would clear and sort of cache/memory that could be stacking upcrawlee 3.9.2
mcr.microsoft.com/playwright:v1.42.1-amd64
CRAWLEE_MEMORY_MBYTES=32768
proxyConfiguration
with about 25 proxies.maxRequestRetries: 3, maxConcurrency: 15, maxRequestsPerMinute: 150, maxRequestsPerCrawl: 35000, requestHandlerTimeoutSecs: 180,
const crawler = new PlaywrightCrawler()
inside our loop (ie. it's not a single crawler that stays alive for the entire thread). Is that your approach too?Do you store fairly consistent amounts of data in each request? If so, crashing at vastly different points would point me away from memory/storage issues and almost more towards... site-specific errors?
When you start to see the error does it only happen once or does it plague all the threads eventually until the process is basically useless?
Also, we're calling const crawler = new PlaywrightCrawler() inside our loop (ie. it's not a single crawler that stays alive for the entire thread). Is that your approach too?
I'm trying to understand a bit more about how Crawlee initiates browsers / clears storage. You'd think if the page was no longer available, just that request would fail and the next one would open a fresh browser and be just fine.
postNavigationHooks
and/or preNavigationHooks
? We're using both. It seems to me that during one of these, the browser is closed - I can't seem to pinpoint the origin trigger for this error.requestHandler
I might say...because we recently updated our crawler logic for thoroughness. Previously, we were not doing any page.evaluate()
, page.waitForTimeout(3000)
, or page scrolling in the requestHandler
- so it might be there then. π€true
, not using any chrome flags