We are just testing Apify's Web Crawling API right now in a free account and will subscribe shortly. Right now it is taking 57 seconds for 11 pages using Cheerio. My question is, will this be faster when we subscribe and get higher memory? Also are there any other suggestions to increase speed? What is the fastest method you have seen and how long should it take? This is a tad slow for us in production without some user experience changes.
You need big batch of URLs, in performance terms scaling only possible as concurrent requests, so approx 50 per minute under 4Gb of RAM should be possible for most of the sites.
As Alexey mentioned, the problem is not your plan but the stricture of your scrape. You need ot run as many URLs at once as possible to maximaly utilize the CPU. Cheerio can be very fast.
