net::ERR_ABORTED when scraping any type of pdf file, so, the only way I've been able to figure out how to handle this is in a preNavigationHook, since, I can't catch the error in the routerhandler. async (crawlingContext, gotoOptions) => {
gotoOptions.waitUntil = 'networkidle'
const page = crawlingContext.page
const request = crawlingContext.request
await page.route('**/*.pdf', async route => {
request.noRetry = true
console.log('running pdf', request.url)
const crawler_request = new CrawleeRequest({ url: request.url, userData: request.userData })
await pdf_crawler.addRequests([crawler_request])
})
page.on('download', async (download: Download) => {
request.noRetry = true
console.log('running download', request.url)
const crawler_request = new CrawleeRequest({ url: request.url, userData: request.userData })
await pdf_crawler.addRequests([crawler_request])
})
},got-scraping .maxRequestsPerCrawl per request queue and then create a new requestqueue every time I have a "spearate crawl" / is there a way to open a new queue and set it on a specific crawler??is there a way to make maxRequestsPerCrawl per requestit is Crawler option so it has to be set on Crawler.
create a new requestqueue every time I have a "spearate crawl"Yes You may create new RequestQueue whenever you want
await Actor.openRequestQueue("my-nw-request-queue-1")is there a way to open a new queue and set it on a specific crawler??You need to pass the RequestQueue to the Crawler options (
requestQueue option).