Apify Discord Mirror

Updated 5 months ago

Scraping Threads Timeline. But Not able to get entire data.

At a glance

The community member is creating a scraper using Playwright and Python to scrape threads.net, but is unable to scrape the entire timeline as the new data is not being scraped when scrolling down. The comments suggest trying auto-scrolling or finding site-specific controls, as well as storing screenshots to analyze where the issue is occurring. One community member indicates that the suggestions helped.

Useful resources
I am making a scrapper to scrap threads.net using playwright and python and willing to host on Apify.

But It I am not able to scrap entire timeline. It is only giving some data and when scrolling down, I am not able to scrap the new data.

Any help or suggestion is highly appreciated.
A
L
S
3 comments
I recommend storing screenshots often, you will see exactly where you got stuck and why. More here https://docs.apify.com/academy/node-js/analyzing-pages-and-fixing-errors
Thank you guys. It helped.
Add a reply
Sign up and join the conversation on Discord