Blocking network requests with crawlee PuppeteerCrawler

kkennysmithnanic

I'm trying to block network requests from specific domains within PuppeteerCrawler but can't get it to work.

I'd like to run something like this:

Plain Text

page.on('request', (req) => {
                // If the URL doesn't include our keyword, ignore it
                if (req.url().includes('bouncex')) {
                    req.abort();
                    return;
                };
                req.continue();
            });

But it has to be initiated before page.goto.

I tried adding it to preNavigationHooks like so:

Plain Text

preNavigationHooks: [
        async ({ page }, goToOptions) => {
            goToOptions!.waitUntil = "networkidle2";
            goToOptions!.timeout = 3600000;
            await blocker.enableBlockingInPage(page);
            page.on('request', (req) => {
                // If the URL doesn't include our keyword, ignore it
                if (req.url().includes('bouncex')) {
                    req.abort();
                    return;
                };
                req.continue();
            });
            await page.setViewport(viewportConfig);
        },
    ],

But this returns Error: Request is already handled!

Is there a way to do this with PuppeteerCrawler?

3 comments

oondro_k

Hey, when you're using multiple Intercept Handlers, you need to check if a request has already been handled: if (interceptedRequest.isInterceptResolutionHandled()) return; . Take a look at this: https://pptr.dev/guides/network-interception#multiple-intercept-handlers-and-asynchronous-resolutions.

LLukas Krivka

Just be aware that request interception disables cache which makes large crawls much worse performance wise

PPepa J

Also you can check blockRequest method from PuppeteerCrawlerContext:

Plain Text

preNavigationHooks: [
    async ({ blockRequests }) => {
        await blockRequests({
            rlPatterns: [
                'yandex.ru',
                'google-analytics.com',
            ]
        });
    }
]

Add a reply

Apify and Crawlee Official Forum

Blocking network requests with crawlee PuppeteerCrawler