Apify and Crawlee Official Forum

Updated 4 months ago

Per request timeout

I can set an overall timeout for a crawler with requestHandlerTimeoutSecs like so:

Plain Text

const crawler = new PlaywrightCrawler({
    requestHandler: router,
    browserPoolOptions: {
        maxOpenPagesPerBrowser: 4,
    },
    requestHandlerTimeoutSecs: 3600,
})

Is there a way to set a timeout per page, rather than an overall timeout? E.g. if there is an exception being caught and the crawl then hangs, I was to timeout after 30 seconds on that page, but have a much longer timeout for the run overall

1 comment

HHamza

requestHandlerTimeoutSecs applies to each page, not the overall crawl. It's the timeout for the requestHandler function.

https://crawlee.dev/api/3.5/basic-crawler/interface/BasicCrawlerOptions#requestHandlerTimeoutSecs

Add a reply