Apify and Crawlee Official Forum

Updated 4 months ago

Web Page Finder

New to Apify. Need advice on best Actor to complete desired outcome.

Goal: Find all websites using a set of specific root urls.
Use case: A known company supplies it clients with dedicated webpages using a root url that doesn't itself display information. To get to a page with information, you need to have a complete url which always follows this format:
http://company-url.com/client-identifier
Where company-url is the creator/host of the webpages and client-identifier is the client name or some version of it.

Usually we can manually put in google search "inurl:http://company-url.com/"*" and manually find each result.

Can this be automated in Apify? Whic Actor would complete this? What settings do I need to be aware of for optimal results?

Thanks!
m
H
2 comments
Add a reply
Sign up and join the conversation on Discord