Apify and Crawlee Official Forum

Updated 5 months ago

Scraping Threads Timeline. But Not able to get entire data.

At a glance

I am making a scrapper to scrap threads.net using playwright and python and willing to host on Apify.

But It I am not able to scrap entire timeline. It is only giving some data and when scrolling down, I am not able to scrap the new data.

Any help or suggestion is highly appreciated.

3 comments

AAlexey Udovydchenko

Try https://docs.apify.com/academy/puppeteer-playwright/common-use-cases/paginating-through-results#auto-scrolling or find site-specific controls

LLukas Krivka

I recommend storing screenshots often, you will see exactly where you got stuck and why. More here https://docs.apify.com/academy/node-js/analyzing-pages-and-fixing-errors

SSoul_Denied

Thank you guys. It helped.

Add a reply