Apify and Crawlee Official Forum

EnqueueLInks

It is possible to add delay's in enqueueLinks? like before to crawl a certain URL I want to add some delay.

5 comments

You can delay any request in the preNavigationHooks. Just keep in mind that Crawler is scaling concurrency based on the CPU utilization so it will not influence maxConcurrency

PPepa J

Hi There is basic example to the Lukas's suggestion:

Plain Text

import { utils } from 'crawlee';

// ...

new PlaywrightCrawler({
  // ....
  preNavigationHooks: [
    async (context) => {
        await utils.sleep(10_000);
    },
  ],
  // ...
});

BBanul;

hmm does cheerioocrawler has a preNavigationHooks?

LLukas Krivka

Yes. You can always find that in the docs https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlerOptions

BBanul;

Oh I didn't see this. Thanks for this I thought preNavigationHooks option is only for PlaywrightCrawler. Again Thank you !

Add a reply

Join on Discord