which browser is the best to crawl

As title said

I’m using chromium currently but it is cpu heavy in usage

Killing browser do not kill the process and because of that it’s easy to get 100% cpu usage pretty quickly

(I’m crawling thousands of websites where on each I’m looking for different data) I already try to load pure html without css, images and other assets, that helped a lot but issue is still there

2 comments

LLukas Celnar

Hi @h
I recommend also blocking unnecessary network requests. with the blockRequests
Make sure that are running it in headless mode.
Also you could try using cheerio if the use-case allows it.

Regarding your question about the browser:
Firefox tends to be lighter on CPU usage.

yes I already do that

Plain Text

   const launchContext: PlaywrightLaunchContext = {
      launcher: firefox,
      launchOptions: {
        headless: false,
        args: [
          '--no-sandbox',
          '--disable-setuid-sandbox',
          '--disable-dev-shm-usage',
        ],
      },
      useChrome: false, // Use Chromium instead of Chrome for better performance
      userAgent: userAgents[Math.floor(Math.random() * userAgents.length)],
    }
...
     launchContext,
      preNavigationHooks: [
        async ({ page }) => {
          await playwrightUtils.blockRequests(page, {
            urlPatterns: [
              '.png',
              '.jpg',
              '.jpeg',
              '.gif',
              '.svg',
              '.ico',
              '.woff',
              '.woff2',
              'adsbygoogle.js',
            ],
            extraUrlPatterns: ['adsbygoogle.js'],
          })

          await playwrightUtils.closeCookieModals(page)
        },
      ],

unfortunetly I recive: WARN Playwright Utils: blockRequests() helper is incompatible with non-Chromium browsers.

I didn't know that 😄

Add a reply

Join on Discord

Apify and Crawlee Official Forum

which browser is the best to crawl