Adapting apify for Crawling

rraymondbags

Hi I need some assistance adapting some of the actors for my Crawler

Details: my company has a Crawler Java application that crawls 70-100 sites for review data and currently we need to outsource this application functionality to a 3rd party. This engine handles http, api and standard html download , I want actors that can handle these 3 usecase scenarios. My application handles orchestration, download and parsing , we can still handle the parsing phase of the collected chunks and extract the target data internally but we need the orchestration and download phases handled externally. Is this something you can help with ? If so, I have more questions and I need to speak to some one from Apify implementation before going any further

7 comments

SSaurav Jain

Can you explain a little bit more?

mmemo23

sent you dm

rraymondbags

I have provided context in the post above. I hope to hear from you soon

OOleg V.

Hi,

Absolutely, everything you mentioned is achievable with Apify and Crawlee. If you have specific questions, feel free to ask. Just keep in mind that we don't support Java. Instead, you'll use our Python or JavaScript SDKs to create orchestrators. For handling download phases, you can utilize Apify's schedulers or implement custom logic. Check out our documentation for more details:

Apify SDK (JavaScript/Python):

JavaScript: https://docs.apify.com/sdk/js/
Python: https://docs.apify.com/sdk/python/
Apify's Scheduler:
https://docs.apify.com/platform/schedules

For general Apify functionality and Crawlee docs:

Apify: https://docs.apify.com/platform
Crawlee: https://crawlee.dev/docs/3.6/quick-start

rraymondbags

Hi , love it, and thanks for the quick response. I thinkJS /Python orchestrators will be just fine. If we wanted your team to fully integrate our site catalogue (about 70-100 -review sites ) , what would that take financially ( I mean all hands off from my company’s perspective and all the initial setup and integration is implemented from your side - of course with mentainance supported by you. ) is there a base subscription arrangement. When you get time, let’s talk about timeline and more sensitive details . Not sure if this chat is safe enough to share sensitive info, of course I’ll invite my product manager as well to the initial dialogue

OOleg V.

for your specific inquiry, it's best to reach out to our sales team directly through this link: https://apify.com/enterprise
They'll be able to provide you with the most efficient assistance and address all your questions.

rraymondbags

Appreciate it man. Thanks. Let me do that right away.

Add a reply

Apify and Crawlee Official Forum

Adapting apify for Crawling