Hello, I’m scraping a website where I have to choose the location (it’s an e commerce store and I have to choose the shop) . The issue is , I’ve created a post navigation hook that is checking for the location and in case it’s not correct it’s starting the process of selecting the location again, saving the cookies and retry the request with the new cookies. With this workflow I can only achieve concurrency 1. Otherwise I’ll have multiple selections of the location at the same time and 99% of time it fails. How could I somehow pause the entire process, do the selection, and retry all the unhandled requests in the queue .
I am afraid there is no premade solution for such a case. You might set crawler.maxConcurrency = 1 when dealing with selecting the location and then put it back once you finish and set the cookies. But this still doesn't close/pause the other browser tabs and you need to handle it on yourself.
