Proxy fails on SSL secured(httpS) websites

At a glance

The community member is trying different proxy providers and encountering issues where the requests fail with 407 (Proxy Authentication Required) or 422 responses, even though the same proxy string works fine with curl. The community members discuss potential causes, such as the presence of a semicolon in the username and/or password, and suggest trying to escape it. However, they are unable to find a solution, and one community member notes that they have encountered a similar issue before but cannot remember how they solved it.

Useful resources

TThePhantom

Hey!

I'm trying different proxy providers and I've noticed the issue in the title.

I'm setting the proxy in

Plain Text

proxyUrls

in the following format:

Plain Text

http://user:pass@host:port

as I usually do. But with the current providers I'm testing, the request will fail with either 407 (Proxy Authentication Required) or 422 responses.

Strangely enough, if tried with

Plain Text

curl -x 'proxy string from the same providers, in the same format' https://example.com

- it works.

Any idea what could be causing it?

10 comments

LLukas Krivka

Can you provide the target URL? The syntax looks correct

TThePhantom

example.com or amazon

LLukas Krivka

Can you share more of the code? No one ever reported this so it is unlikely it would be a bug in Crawlee

TThePhantom

Same proxy string works perfectly fine with curl, as mentioned above. Not sure why it "asks for authentication". Let me grab a screenshot with the resi proxies as well

Attachment

TThePhantom

There you go

Attachment

TThePhantom

Ahh, I think I know why it happens. There's a semicolon in the username and/or password in both cases. Let me see how should I escape it

AApifyBot

just advanced to level 2! Thanks for your contributions! 🎉

TThePhantom

Can't seem to do so, though. Should not need to escape semicolon as it's not really a special character, but Crawlee is not happy with it :\

And that's with Cheerio. Same formatting works just fine with Playwright

LLukas Krivka

I think I remember this issue but don't remember how I solved it 😄

TThePhantom

Haha, happens to me too. That's why I started to "write things down"(thanks note taking apps and Github gists).

Let me know if you remember the solution. It'll be helpful for others as well, I'm sure 🙂

Add a reply

Apify Discord Mirror

Proxy fails on SSL secured(httpS) websites