Issue with Residential Proxies

Question

Hi there. Whenever I try to use residential proxies ('HTTP://groups-RESIDENTIAL:/...') I run into this error:

httpx.ConnectError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129)

The 'auto' group seems to work fine. Can anyone tell me what I'm doing wrong here?

Thanks!

Mantisus · Accepted Answer

HeyTry disabling certificate validation from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext
from crawlee.http_clients import HttpxHttpClient async def main() -> None: http_client = HttpxHttpClient(verify=False) crawler = BeautifulSoupCrawler(http_client=http_client)

uberpea5000 · Answer

Hmm. The automated response may have directed me to an answer: I'm crawling a .gov URL.

Mantisus · Answer

this work from version 0.5.0

uberpea5000 · Answer

Ah ha! Yes. That seems to do it. Much appreciated. (It alsoimplies the solution to my next problem: extending the httpx client timeout period.)

Mantisus · Answer

Note that if httpx will be very slow (more than a minute), you may also need to increase the timeout for the handler - https://crawlee.dev/python/api/class/BasicCrawlerOptions#request_handler_timeout

uberpea5000 · Answer

Thanks. Yes. Just had to do that as well 🙂

uberpea5000 · Answer

Got another question, but I'll start a new thread for it. Thanks again.

Apify and Crawlee Official Forum

Issue with Residential Proxies