my crawler with PlaywrightCrawler works just fine but I have issue when adding proxy !!!
this is the code
import { PlaywrightCrawler, ProxyConfiguration } from "crawlee";
const startUrls = ['http://quotes.toscrape.com/js/'];
const crawler = new PlaywrightCrawler({
requestHandler: async ({ page, parseWithCheerio }) => {
await page.waitForSelector("div.quote span.text", { "timeout": 60000 });
const $ = await parseWithCheerio()
const quotes = $("div.quote span.text")
quotes.each((_, element) => { console.log($(element).text()) });
},
});
await crawler.run(startUrls);however when I add my proxy port I always get timeout erros !!!
const proxyConfiguration = new ProxyConfiguration({
proxyUrls: ["url-to-proxy-port-im-using"]
})
// and the add it to crawler
const crawler = new PlaywrightCrawler({
proxyConfiguration,
...and also the same code with the proxy configuration works with CheerioCrawler !!!!
can anyone help with this issue !?
