Apify and Crawlee Official Forum

Updated 3 months ago

Crawler not very Efficient

I'm using the apify actor that crawls website contents, but it hasn't been very efficient some website weren't crawled at all. I need help
O
t
b
5 comments
Hi,
Sorry to hear that. But I need more info, if You want me to help.

Could you please provide more details on the issues you're facing? Specifically:
The URLs of the websites that weren't crawled.
Any error messages or logs you received.
Your current scraper settings and configuration.

Link to your run on the platform would be the best
Thanks , the run has been deleted, I used a family link " www.hive.com/blogs" withe. Glob setting to crawls the blog contents for all the blogs articles in that domain, the result returned were only a few, It left out a lot of there blogs in this url "" www.hive.com/blogs" today I used the puppeteer actors based off a recommendation with hopes that it will handles this, it did really great getting the url's and blog titles around 600+ which the previous couldn't but it didn't crawl the blogs contents, I can only share the new run link , but privately
It seems there might be an issue with your scraping logic . Also, try to double-check if You are using correct selectors. I'd recommend checking out this beginner-friendly article: https://docs.apify.com/academy/apify-scrapers

Additionally, I noticed that hive.com/blogs returns a "404 Page Not Found" error. It's possible you have an incorrect URL, or the target site might be experiencing some performance issues.
I created actor in python on apify, Monitized as paid per result.. i want to restrict for using free user..any idea how i can do this.. right now free user use my actor and produce result but i did not getting anything..please suggest me how to restrict apify free user to use actor

Did you check out this tutorial?
https://docs.apify.com/academy/get-most-of-actors/monetizing-your-actor

Maybe you missed something? Please double-check it.
Add a reply
Sign up and join the conversation on Discord