Get Memory Size From Input?

At a glance

The community member is looking for a way to automatically determine the user's memory usage without having them manually input it. They are using multithreading in their program and want to pick the optimal number of workers/threads based on the memory. They tried using multiprocessing.cpu_count() but it always returned 16. The community member also mentioned a message from Lukas Krivka about using an API to get the current memory value, but this seems to be more about handling it on the user side rather than the developer side.

In the comments, another community member provided a solution to get the memory limit by reading from the /sys/fs/cgroup/memory/memory.limit_in_bytes or /sys/fs/cgroup/memory.max files. Another community member suggested checking the APIFY_MEMORY_MBYTES environment variable.

There is no explicitly marked answer in the post or comments.

Useful resources

JJonathan Larson

Currently, is there a way to determine the user's memory used without having them manually type it in? My program using multithreading (works better than async for my use case) and I want to automatically pick the optimal number of workers/threads based on the memory inputted. I tried multiprocessing.cpu_count(), but it seems to just respond with 16 every time.

I viewed https://docs.apify.com/platform/actors/running/usage-and-resources but didn't seem to see it there

I also noticed Lukas Krivka's message on 01/09/2023, but it seems to be more about handling it on the user side rather than developer:

"Generally, if you know your total memory, you can just manage it locally completely but if you don't, you can use this API to get current value: https://api.apify.com/v2/users/me/limits?token=TOKEN. Just keep in mind that meanwhile some other task could have started if you don't have sole control over the account."

3 comments

JJonathan Larson

If anyone else has the same issue, I got it with this: (returning bytes)

def get_memory_limit():
try:
with open('/sys/fs/cgroup/memory/memory.limit_in_bytes') as f:
return int(f.read())
except FileNotFoundError:
with open('/sys/fs/cgroup/memory.max') as f:
return int(f.read())

JJonathan Larson

If anyone sees this though, please message if you know of a way to use the or operator for required?

(I want user to be able to either enter usernames or search terms but have the flag raised without wasting compute it both are missing)

LLukas Krivka

Just check the APIFY_MEMORY_MBYTES env var

Add a reply

Apify Discord Mirror

Get Memory Size From Input?