Apify and Crawlee Official Forum

Updated 5 months ago

File size upload to KeyValueStore

At a glance

I created actor which downloads and uploads videos to KeyValueStore, some of videos are pretty big and getting this error:

2024-05-18T15:54:15.572Z Failed to upload video: RangeError [ERR_FS_FILE_TOO_LARGE]: File size (2183982088) is greater than 2 GiB

so anyway to get around this and what are best practices on apify for downloading and uploading of big files

1 comment

oondro_k

Hi, 2GB is the node's limit for reading files. You can avoid it by using streams:

Plain Text

import got from 'got'
import stream from 'node:stream'

const headers = { 'Content-Type': contentType }
await streamPipeline(
    got.stream.get(DOWNLOAD_URL, { headers }),
    new stream.PassThrough(),
    got.stream.put(UPLOAD_URL, { headers }),
)

Add a reply