Apify Discord Mirror

Updated 5 months ago

Hello Friends,

At a glance

The community member is a new Apify user and has some questions: a) Can Apify be used to scrape websites that require a username and password, or is it limited to public websites? b) How can the community member better control the proxy settings to get country-specific IP addresses and see results on the website for those?

In the comments, another community member responds that: a) There are no limits, but the community member should be aware of website terms of service and anti-bot protection. There is a tutorial on logging into websites. b) The community member can find information on rotating proxies and limiting proxy location.

The second community member finds the responses helpful.

Useful resources
I am a new user and loving what apify can do. Have run some actors already and results are ok. Had a question:

a) If I want to write a custom scrapper or host an existing code here then does apify limit scrapping to only public websites or can it scrap those with username and password (which we will provide of course in our scrapper)?
b) How do I better control the proxy settings as I want to get country specific IP and see results on the website for those.
A
p
2 comments
a) No limits, just be aware that its your responsibility how to deal with website ToS and anti-bot protection in each given case. In tech terms there is even tutorial https://docs.apify.com/academy/puppeteer-playwright/common-use-cases/logging-into-a-website
b) https://docs.apify.com/academy/expert-scraping-with-apify/solutions/rotating-proxies#limiting-proxy-location
Add a reply
Sign up and join the conversation on Discord