Cheerio Fingerprint

TTay

Is there a way to use fingerprints with the Cheerio crawler? I need it to send Firefox headers. It's currently sending chromium ones

Plain Text

Host: localhost:8000
Connection: keep-alive
Content-Length: 0
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 Edg/124.0.0.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Sec-Fetch-Site: same-site
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Ch-Ua: "Chromium";v="124", "Microsoft Edge";v="124", "Not-A.Brand";v="99"

6 comments

TTay

or can i remove some headers?

vvojtechmaslan

Hi , to use only Firefox headers, you can modify gotScraping headerGeneratorOptions in preNavigationHooks:

Plain Text

    ...
    preNavigationHooks: [
        (_, gotOptions) => {
            gotOptions.headerGeneratorOptions = {
                browsers: ['firefox'],
            };
        },
    ],
    ...

https://github.com/apify/got-scraping?tab=readme-ov-file#headergeneratoroptions

TTay

Thanks, is it possible to remove a header after generation?

AApifyBot

just advanced to level 1! Thanks for your contributions! 🎉

TTay

i can add extra headers by appending to request.headers in preNavigationHooks but not remove it

OOleg V.

Did you try to make it undefined ? e.q. request.headers['my-header'] = undefined

Add a reply

Apify and Crawlee Official Forum

Cheerio Fingerprint