Apify

Apify and Crawlee Official Forum

b
F
A
J
A

Do you know a dashboard for Crawleee?

I want to monitor all Crawlee crawlers. To do this I looked for a dashboard to control the crawlers.
I only found this distributed web crawler management platform: Crawlab (https://github.com/crawlab-team/crawlab).

Do you know of any others?
t
L
A
9 comments
Have you tried the Apify platform? If you deploy your crawlers to the platform, you can run them and monitor them right from the dashboard there!

https://apify.com

The great thing about this solution is that Crawlee was built with the Apify platform in mind, so there is seamless support
Thanks for your answer, but using sensitive data, I cannot use a Cloud solution. That's why I'm looking for an open source solution and I use Crawlee.
So, your crawlers will only be run locally?
Yes, I have a cluster of server where crawlee crawler run.
How are these servers being hosted?
These are Raspberry pi's at my house
just advanced to level 1! Thanks for your contributions! πŸŽ‰
In order to make a dashboard, you'd need a centralized location where you could store data about each crawler. A SQL or Mongo database would be great for this.

You could have a server hosted on one of your rasp pis that serves as an API for storing data about crawlers. When a new crawler is started, a new row could be added to the "crawlers" table by sending a request from the crawler to your API.

Then, you could create some sort of UI service with Electron or React Native that uses your API.
With this custom solution, you could make the dashboard as minimalistic or as robust as you like. You'd have full control. And everything would be hosted locally.
Add a reply
Sign up and join the conversation on Discord
Join