Efficiency / Cost management of multiple smaller actors vs. one large actor
Efficiency / Cost management of multiple smaller actors vs. one large actor
At a glance
The community member is using the Clockworks TikTok scraper and is unsure whether to use 8 actors with 4GB RAM each, spreading out the pages to scrape evenly across the actors, or 1 large 32GB actor for all the pages. The community member found that it is cheaper to spread out the jobs across 8 actors ($10-12) compared to using 1 large actor ($3-4), though the speed is not much faster. The community member was advised by their team to try the 1 large actor, but their findings seem different. The community member is asking if they are doing something wrong or if the scrape is better with batching the jobs across multiple actors.
A comment from another community member suggests that the actual actor and input really determine the best approach, and the community member's tests make sense.
I'm using the clockworks tiktok scraper and had dilemna on whether to use 8 actors with 4gb RAM each, spreading out pages to scrape evenly accross the actors vs. 1 large 32gb actor for all the pages.
It seems like from trying out the scrape its far cheaper (not much faster though), to spread out the jobs accross 8 actors (we're looking at $10-12 vs. $3-4). I was adviced from the team to try one large actor since it should scale okay, but seems like my findings are different?
Am I doing something wrong here, or is it just that this scrape is better with me batching the jobs.
