Hi,
I have a puppeteer scrapper, which worked just fine until this Monday. Nothing is changed, but scrapper stopped working.
The HTML markup of the page is not changed, a[data-testid="search-listing-title"] this element is still there. Apify run logs says it is failing to find this HTML element:
TimeoutError: waiting for selector a[data-testid="search-listing-title"] failed: timeout 30000ms exceededI have tried to launch scrapper from local machine and it did work but does not work on Apify platform. I guess something has to do with proxy.
This is part of my code:
//...
const proxyConfiguration = await Apify.createProxyConfiguration();
const launchContext = {
useChrome: true,
stealth: true,
launchOptions: {
headless: true,
},
};
const crawler = new Apify.PuppeteerCrawler({
requestList,
requestQueue,
proxyConfiguration,
launchContext: launchContext as any,
maxRequestRetries: 5,
handlePageTimeoutSecs: 180,
navigationTimeoutSecs: 180,
async handlePageFunction({ page, request }): Promise<void> {
await utils.puppeteer.saveSnapshot(page, { key: 'beforescrap', saveHtml: false });
const cheerio = load(await page.content());
const html = cheerio.html();
await Apify.setValue('htmlstring', html, { contentType: 'text/html' });
await page.waitForSelector('a[data-testid="search-listing-title"]');
//...I have tried to take a screenshot to see what the page looks like and it gives blank white page.
I have also tried to change proxy settings to use residential servers and change the country - also did not work.
How can I debug this?
Logs screenshot is also attached.
