save HTML file using crawlee

NNyanmaru

Has anybody tried downloading the HTML file of the URL using Crawlee? Was wondering if Crawlee has a capacity of downloading the HTML file of the URL since I've just been using Crawlee and really loving the experience.

6 comments

EExp

You can download HTML content of a webpages using Crawlee

MMarco

It depends on which crawler you are using:

Cheerio: https://cheerio.js.org/docs/api/classes/Cheerio#html
Playwright: https://playwright.dev/docs/api/class-page#page-content
Puppeteer: https://pptr.dev/api/puppeteer.page.content

NNyanmaru

Thanks for this awesome answer! Was wondering if Crawlee has examples on how to save it to a file?

MMarco

You can use the KeyValueStore: https://crawlee.dev/api/core/class/KeyValueStore. E.g., with Cheerio:

Plain Text

await store.setValue('my-html', $.html('html'), { contentType: 'text/html' });

CCrafty

everything you save to a crawlee store is saved to disk, and can be accessed through crawlee or otherwise. Something my company is using is saving page.content() to a cloud storage bucket during the request handler which has worked quite well for us since it offloads the data nice and quickly.

sshanky

I want to retrieve html for some specific table, say I want to retrieve html for "item-list" div in the code below (not the data inside each element). How to do this?

Plain Text

<body>
    ...
    ...
    <div class="item-list">
        <div class="item">
            <div class="product-label"></div>
            <div class="product-image"></div>
            <div class="product-cost"></div>
        </div>
        <div class="item">
            <div class="product-label"></div>
            <div class="product-image"></div>
            <div class="product-cost"></div>
        </div>
        ...
        ...
    </div> <!-- end item-list -->
    ...
    ...
    <div class="testimonials">
        ...
        ...
        ...
    <div>
</body>

I do not want resultant html to be saved in a file/disk, I want to return my desired result to my api for further processing. How to do this?

Add a reply

Join on Discord

Apify and Crawlee Official Forum

save HTML file using crawlee