crawler process not exiting after teardown is called.

AAltairSama2

I am running a Playwright crawler and I call await crawler.teardown() after some condition is met,

Issue with this approach is that the crawlee process hangs indefinitely and doesn't exit even after a while. I have to force stop it using CTRL + C, on searching github, it looks like its an issue with some timeouts but I havent set one anywhere.

here's how I am initializing my crawler:

Plain Text

export const crawler = new PlaywrightCrawler({
    requestHandler:router,
    maxRequestsPerMinute:10,
})

and I have a playwright config where I have set the default timeout

Plain Text

import { defineConfig } from '@playwright/test';

export default defineConfig({
    timeout: 3000,
});

3 comments

AAltairSama2

it just logs this at the end and then hangs indefinitely

Plain Text

2023-12-06T03:29:29.063Z info: Finished! Total 7 requests: 7 succeeded, 0 failed. {
  "terminal": true
}

AApifyBot

just advanced to level 5! Thanks for your contributions! 🎉

AAltairSama2

I fixed it by removing the return statement in my route handler, I guess it was causing issues with teardown's promise, I'm not sure why though

Plain Text

if (state.duplicate_count <= 0 || state.page_count > state.num_pages) {
            log.debug(
                `10 duplicate entries encountered in a row, Shutting down....`
            );
            await crawler.teardown();
            return; //removing this
        }

Add a reply

Join on Discord

Apify and Crawlee Official Forum

crawler process not exiting after teardown is called.