Apify and Crawlee Official Forum

How to prevent following redirects to other domains?

I see there is a way to prevent this once page loads with something like this:

Plain Text

    await page.setRequestInterception(true);

    page.on('request', async (request) => {
      if (ok) {
        await request.continue();
      } else {
        await request.abort();
      }
    }

But how about when I have a url in queue, is there a nice way to have the request interception setup in the crawler options instead and have it defined globally?

4 comments

MMichal

I guess I'm looking for something like a global beforeNavigation callback.

MMichal

Yay, it's preNavigationHooks, I wasn't able to find it in the API docs though. Found it here in fulltext and then used github all code search for examples.

PPepa J

Yes, you need to put this to the prenavigationHooks, did this solve your issue? 🙂

MMichal

Yep I got it working, although I didn't end up blocking them in the end.

Add a reply

Join on Discord