How to close the crawler from a RequestHandler?

Instead of await crawler.run() just crawler.run() and then teardown when you condition or event will be handled by your own code outside of crawler

issue is the conditions are triggered in specific routes of a site

for eg. we have a resume function in our selenium scrapers which checks for duplicates and if some n number appear in a row we stop scraping assuming the rest of the data will have been scraped already too

plus we have a couple of other such conditions, it would be helpful if something like this was present inside the request handlers

similar question, how do I stop the request handler flow if some condition is satisfied? e.g. if some element is not present, stop the function right there. Will a simple return; suffice? since we anwyays dont return anything and just enqueue links.

can someone help wth this? could really use this functionality to avoid duplicate/redundant scrapes etc

or is there a way we can empty out request queue? I think this might work since the crawler will stop as soon as it sees there's nothing to scrape

just advanced to level 4! Thanks for your contributions! 🎉

For reference answered here https://discord.com/channels/801163717915574323/1075487274424352888/1179490876850974810

If you want to stop the request handler itself, you need to have a condition at that point, JS doesn't allow cancelling functions/promises from the outside.

yeah, that should not be an issue since we can control the number of requests via maxConcurrency

thanks!