Gracefully closing the crawler with keepalive flag true

mmakditeam

I'm using puppeteer crawler with keepAlive as true and crawler.run() (without await).

This runs the crawler infinitely and if I insert new requests to the requests queue, they get processed.

(I'm using non persisted request queue)

What I want is to gracefully close the crawler, as in If I get a signal to close, I want to process all the pending requests in the requests queue first and then kill the crawler.

Right now If I do crawler.teardown(), it abruptly closes the crawler instances without processing the pending requests.

2 comments

PPepa J

Hi :
Maybe you want to define isFinishedFunction instead? https://docs.apify.com/sdk/js/docs/2.3/typedefs/autoscaled-pool-options#isfinishedfunction

Plain Text

// ...
    autoscaledPoolOptions: {
        isFinishedFunction: () => {
           return wasAbortedAndAllRequestsHandled()
        },
    },
// ...

LLukas Krivka

You can check there if the queue is empty

Add a reply

Join on Discord

Apify and Crawlee Official Forum

Gracefully closing the crawler with keepalive flag true