What you can do is create named named queues for each crawler or requests array and keep pushing the requests into it, once the Playwright crawler finishes, you can create the Cheerio crawler and pass the queue/requests array to it and that should solve the issue.
Note that if you want to use queues, you need to create separate named queues for both crawlers
Something like this:
const playwrightQueue = await Actor.openRequestQueue('playwright_queue');
const cheerioQueue = await Actor.openRequestQueue('cheerio_queue');
// Add initital requests to the queue
playwrightQueue.addRequests(initialRequests);
// const cheerioRequests = [];
const playwrightCrawler = new PlaywrightCrawler({
proxyConfiguration,
requestQueue: playwrightQueue,
requestHandler: async () => {
// handle request...
// Push the requests
cheerioQueue.addRequests(
// ...
);
// OR
// cheerioRequests.push(
// ...
// );
},
});
// Run playwright crawler
await playwrightCrawler.run();
// Once it's done, run cheerio crawler
const cheerioCrawler = new CheerioCrawler({
proxyConfiguration,
// Pass the cheerio queue
requestQueue: cheerioQueue,
requestHandler: async () => { },
});
await cheerioCrawler.run();
// Or
// Pass the generated requests to cheerio
// await cheerioCrawler.run(cheerioRequests);