Apify
Crawlee
GitHub
Apify Discord Mirror
View all posts
Related posts
Was this helpful?
π
π
π
Powered by
Hall
Active
Updated last week
0
Follow
How to retry only failed requests after the crawler has finished ?
Crawlee
How to retry only failed requests after the crawler has finished ?
Active
0
Follow
V
Vi
2 weeks ago
Β·
I finished the crawler with around 1.7M, and got around 100k failed requests. Is there a way to retry just the failed requests ?
R
V
A
4 comments
Share
Open in Discord
R
Rado Ch.
last week
hey, that's not currently supported, I would recommend creating a dataset/kv store for failed requests and push to it from failed request handler
V
Vi
last week
Retrying it would necessitate writing a new scraper since the crawler consists of multiple routes handlers....
I thought I could just run the scraper again but increasing the retry count.
A
ApifyBot
last week
@Vi just advanced to level 3! Thanks for your contributions! π
R
Rado Ch.
edited last week
you can use failed request handler to handle all failed request in one place
https://crawlee.dev/api/playwright-crawler/interface/PlaywrightCrawlerOptions#failedRequestHandler
Add a reply
Sign up and join the conversation on Discord
Join on Discord