Crawlee+PlaywrightCrawler+proxy - original IP leaking through WebRTC

What type of proxy are you using? Is it an 'Elite' proxy?

well, I do not know what an "Elite proxy" is :-)

i am using proxies from smartproxy.com
these are "residential", "rotating" proxies.
The US proxies availiable under "us.smartproxy.com:<port>" (they have proxies for almost every country and the global pool)

I think for $12.5 per GB, "pay as you go" - they are not bad.

Anyway, I am open to alternatives

P.S. lets discuss different proxy services here: https://discord.com/channels/801163717915574323/1060179502392684594

Apify has the same price for residentials and they aggregate multiple providers, you can try on free plan. Will check the original issue

Check the Web RTC leak

by the way, this: https://browserleaks.com/webrtc might be helpfull.

and, in the Firefox/Linux I use every day for browsing - all fields on that page are either "n/a" or "false" - and I do not even see that green START button (twicked my Firefox to prevent fingerprinting + "Privacy Badger" extension + "Canvas Blocker" extension)

I think all features related to WebRTC could be just disabled

To disable WebRTC:
media.peerconnection.enabled (WebRTC) = false

Taken from this article
https://restoreprivacy.com/firefox-privacy/

I think, it make sense to change configuration of headless Firefox. To set privacy.resistFingerprinting and some other flags described there.

I did it in my desktop browser - no WebRTC leak more.

So, who knows how to apply this to headless Firefox?

UPDATE:
https://stackoverflow.com/questions/53017490/how-can-i-disable-webrtc-local-ip-leak-with-puppeteer
looks like solution for Puppeteer...

I believe that Firefox and Chrome have this setting disabled by default. I remember this was a huge security issue because you could access the IP addresses of other devices on the same network as the computer. I tested it on my browsers and have no leaks when using VPN or proxy. This might be because your proxy is sending x-forwarded headers. Let me know if you have a repro. I am curious to find out what is happening.

Firefox and Chrome have this setting disabled by default.

this is not true for Firefox! I changed the media.peerconnection.enabled according to above-mentioned article -- these settings was enabled by default.

This might be because your proxy is sending x-forwarded headers.

Who knows what the hell my proxy is doing...

- how to check for x-forwarded headers?
Just make request through my proxy to this address https://httpbin.org/get and look fo "x-forwarded" ?

What is your Firefox version?

Yes the httpbin should work.

firefox used to run these tests:
/home/<my-user-name>/.cache/ms-playwright/firefox-1357/
(thats on a machine in datacenter)

firefox I use on desktop (to read this forum for example ) is: 108.02 (64bit)

this screenshot made in headless mode, httpbin.org/get was accessed through proxy.

and - the same program accessing pixelscan.net - it shows the WebRTC leak

This is interesting. I believe that this is connected to the DC proxy. However, you are right that this leak should be prevented in the crawlee. I will test it and process this change. Thanks a lot for bringing this up!

So the only way how to do it programmatically (http://kb.mozillazine.org/User.js_file). It is necessary to create a user.js file and add it to your docker build flow.

the only way how to do it programmatically (http://kb.mozillazine.org/User.js_file).

Well, in the firefox I use for browsing every day - I have no WebRTC leak and I do not use additional User.js !


I just changed the media.peerconnection.enabled and few other settings, thats it.
Is it possible to pass this and few other settings to Firefox used by Playwright/Crawlee?
Some command-line flags?

...to create a user.js file and add it to your docker build flow.

I do not using Docker at all. I just installed Crawlee on a machine in datacener and thats it
I see that Firefox is here:
~/.cache/ms-playwright/firefox-1357/

Is there any news on this (WebRTC detection/leak) ?

Hi There is currently PR https://github.com/apify/fingerprint-suite/pull/141 for this.

Great news! So, in the next release this will be availiable, right?

It is not merged yet, so not sure it would be part of next release, if you need it now, you can pull the got branch and build and use it locally.