Apify Discord Mirror

Updated last week

How to retry only failed requests after the crawler has finished ?

I finished the crawler with around 1.7M, and got around 100k failed requests. Is there a way to retry just the failed requests ?
R
V
A
4 comments
hey, that's not currently supported, I would recommend creating a dataset/kv store for failed requests and push to it from failed request handler
Retrying it would necessitate writing a new scraper since the crawler consists of multiple routes handlers....

I thought I could just run the scraper again but increasing the retry count.
@Vi just advanced to level 3! Thanks for your contributions! πŸŽ‰
Add a reply
Sign up and join the conversation on Discord