Apify Discord Mirror

C
Chris
Offline, last seen 2 months ago
Joined December 4, 2024
I'm a data reporter at CBS News using crawlee to archive web pages. Currently, when I finish a crawl, the next crawl continues to crawl pages enqueued in the previous crawl.

Is there an easy fix to this? I've looked at the docs, specifically the persist_storage and purge_on_start parameters, but it's unclear from the documentation what exactly those do.

Happy to provide a code sample if helpful
9 comments
C
M
A