APIFY PROXY ECONNREFUSED 127.0.0.1:80

JJohn

Hello, I'm in trouble with Apify Proxy (on a free plan) I have a connexion refused. With and witout cloudscraper.

Any ideas please?

let proxyConfiguration;
if (useApifyProxy) {
proxyConfiguration = await Actor.createProxyConfiguration({ useApifyProxy: true });
} else if (proxyUrls.length > 0) {
proxyConfiguration = await Actor.createProxyConfiguration({ proxyUrls });
}

console.log(proxyConfiguration, proxyConfiguration.newUrl());

const crawler = new PlaywrightCrawler({
requestQueue,
launchContext: {
launchOptions: {
proxy: proxyConfiguration ? proxyConfiguration.newUrl() : undefined,
},
},
requestHandler: async ({ page, request }) => {

try {
// Use cloudscraper to fetch the page content
const html = await cloudscraper.get({
uri: request.url,
proxy: proxyConfiguration ? proxyConfiguration.newUrl() : undefined,
// Add other cloudscraper options as needed
});

// Process the HTML content as needed
const title = await page.title();
console.log("Page Title: " + title);

(...)

8 comments

JJohn

Failed to fetch data: Error: tunneling socket could not be established, cause=connect ECONNREFUSED 127.0.0.1:80

MMarco

Hello! Is the error coming from Apify or cloudscraper?

JJohn

Hi! Directly from the crawler apparently : Failed to fetch data: Error: tunneling socket could not be established, cause=connect ECONNREFUSED 127.0.0.1:80
2024-06-14T07:30:41.048Z WARN PlaywrightCrawler: Reclaiming failed request back to the list or queue. Error: tunneling socket could not be established, cause=connect ECONNREFUSED 127.0.0.1:80
2024-06-14T07:30:41.050Z

JJohn

groups: [],
2024-06-14T07:30:36.531Z countryCode: undefined,
2024-06-14T07:30:36.532Z password: '*',
2024-06-14T07:30:36.534Z hostname: '10.0.44.225',
2024-06-14T07:30:36.535Z port: 8011,
2024-06-14T07:30:36.537Z usesApifyProxy: true

MMarco

The error is caused by the fact that proxyConfiguration.newUrl() returns a promise, so you must await it, like this:

Plain Text

const html = await cloudscraper.get({
    uri: request.url,
    proxy: proxyConfiguration ? await proxyConfiguration.newUrl() : undefined,
});

Moreover, when you configure the PlaywrightCrawler, there is no need to put your proxy into launchContext.launchOptions: you can just pass the proxyConfiguration object, like this:

Plain Text

const crawler = new PlaywrightCrawler({
    requestQueue,
    proxyConfiguration,
    ...

Hope this helps! 🙂

JJohn

Thank you Marco, it works but I get a 403 from Cloudflare (Attention Required! | Cloudflare) with : const crawler = new PlaywrightCrawler({
requestQueue,
proxyConfiguration,
// launchContext: {
// launchOptions: {
// proxy: proxyConfiguration ? await proxyConfiguration.newUrl() : undefined,
// },
// },
requestHandler: async ({ page, request }) => {
if (dev) await page.setViewportSize({ width: 1200, height: 800 });

console.log(Processing ${request.url}...);

try {
// Use cloudscraper to fetch the page content
const html = await cloudscraper.get({
uri: request.url,
proxy: proxyConfiguration ? await proxyConfiguration.newUrl() : undefined,
// Add other cloudscraper options as needed
});

AApifyBot

just advanced to level 1! Thanks for your contributions! 🎉

MMarco

That depends from the anti-scraping policies the website you're interested in may have put in place, and it's a complicated matter. I suggest you to take a look at this: https://docs.apify.com/academy/anti-scraping

Add a reply

Apify and Crawlee Official Forum

APIFY PROXY ECONNREFUSED 127.0.0.1:80