Apify Discord Mirror

Updated 5 months ago

Error handling/Best Practices Python SDK

At a glance

The community member is using pre-built actors in their application to create a dataset. They are interested in improving the error handling of this approach and are wondering what types of errors or issues they could encounter, such as if the actor breaks or if there are errors in the dataset. The community member is looking for best practices and recommendations on how to handle these situations.

In the comments, another community member suggests that the original poster should re-post the question to the Python forum, as it may be more suitable for this type of question. The original poster then asks for a link to the Python forum, and another community member provides a link to a channel on the same Discord server.

There is no explicitly marked answer in the provided information.

Useful resources
Hello,
I am using pre-built actors in my application. I use them like this to create the dataset:
Plain Text
    client = ApifyClientAsync(token=settings.APIFY_API_TOKEN)
    run = await client.actor(actor.value).start(run_input=run_input)
    processed = 0
    while True:
        await asyncio.sleep(2)
        data = client.dataset(run["defaultDatasetId"]).iterate_items(offset=processed)
        async for item in data:
            dataset.append(item)
            processed += 1
            logger.info(f"processing item: {item.get('url')}")
        run_status = await client.run(run["id"]).get()
        if run_status.get("status", None) == "RUNNING":
            logger.info("Run is still running")
            continue
        else:
            logger.info("Run is finished.")
            break

I want to improve the error handling of this approach. I am wondering which types of errors or issues I could encounter and what the best practices are. Example: What happens if the actor breaks (memory/cpu/other issue) or I get an exception (which types)? What if there are errors in the dataset (400 status code, crawler blocked, etc.). Does anyone have recommendations here? Thank you!!
c
v
5 comments
Any ideas? Thanks
Hi , can you please re-post this question to the Python forum? It should be more suitable for this question and it will be easier for you to get an answer.
Can you please link the python forum? Is it another channel/server?
I didn't see this channel at all. Thanks. Posted it there.
Add a reply
Sign up and join the conversation on Discord