Apify and Crawlee Official Forum

Home
Members
darkprince
d
darkprince
Offline, last seen 2 weeks ago
Joined December 18, 2024
I'm trying to run Crawlee for production use and try to scale where we can have a cluster of worker nodes who will be ready for crawling pages based on the request. How can achieve this.

The RequestQueue is basically writing requests to files and not utilizing any queueing system. I couldn't find doc that said how i can utilise Redis queue or something similar.
7 comments
d
M
A