How should I structure my crawler when scraping possibly 100s of different sites with different structures, handling multiple requests at once in Crawlee
Well, I am implementing something similar... 30-40 sites but with SIMILAR structure (if the structure of your sites is different -> you are implementing something like google/bing - king of generic web crawler)
You might use something like an external message queue, we discussed it here and in few other places:
