scraping at scale

The community member is asking how to structure a crawler when scraping possibly hundreds of different websites with varying structures, while handling multiple requests at once using Crawlee. Another community member suggests using an external message queue, such as beanstalkd, to manage the requests. They also recommend creating a configuration file (e.g., YAML or JSON) that describes where to find specific content on each website. There is no explicitly marked answer in the comments.