working with requestQueue/ catching apify Errors when using Apify API Client for Javascript
working with requestQueue/ catching apify Errors when using Apify API Client for Javascript
At a glance
The community member is trying to run multiple actor calls simultaneously, but is running into memory issues when exceeding the 32GB limit. They are interested in using a request queue to manage the calls, but are having trouble understanding how to add actors to the queue using the API. If they cannot figure out how to use the request queue, they would like to create their own custom queue implementation.
In the comments, another community member explains that the request queue is just a storage for requests, and that the community member would need to use a crawler like BasicCrawler to actually evaluate the items in the request queue. The community member suggests setting the maxConcurrency option on the crawler to limit the number of concurrent runs based on the available memory. Alternatively, the community member could try to fill the unused memory space with runs that use different amounts of RAM. However, the community member notes that in this more complex scenario, the community member would need to implement their own priority-based queue.
The original community member acknowledges the suggestion and states that they will implement their own queue solution.
Hey everyone. I'm trying to run multiple actor calls simultaneously, and that works fine, unless I go over the allotted memory (32GB). I would like to use a requestQueue, which I assume would run calls in order whenever there is available memory? But I have a hard time understanding how to add actors to the queue with the API. e.g. client.requestQueue("...").addRequest() accepts some unclear parameters that don't seem to contain an actor or the data to pass to the actor. This very much might be because I'm a javascript noob that has a bit of a hard time understanding the documentation XD
If I completely misunderstood the idea of a requestQueue, I would like to create my own. For that, I would like to be able to try/catch a normal client.actor(...).call() but specify the error, except I can't seem to find a way to specify the specific error you would get when requesting the use of too much memory (e.g. some error object to import from apify and use).
Hi , RequestQueue is just a storage that hold the Requests. You need to use a Crawler like BasicCrawler and to actually evaluate the items from EequestQueue.
Simple situation could be that:
You may set the maxConcurrency to the Crawler options. If your runs are each taking 1 GB and you have only 5 GB availaable on your account set the maxCocncurency to 5 GB.
More complex situation would be to try always fill the unused memory space with a new run that uses different amount of RAM than the other runs. I am afraid that in this case, you would nee to implement the sort of Priorify based Queue on yourself.
