Open-source tests of web browser privacy(privacytests.org)
privacytests.org
Open-source tests of web browser privacy
https://privacytests.org/
33 comments
To me, it seemed that if a browser had passed every test, the user might not be able to use many web sites. Perhaps the current status is a conscious decision by developers to keep users from hating their browser. Security vs usability.
If enough users used a privacy friendly browser web devs would be forced to make their sites compatible or they would lose traffic.
Catering to how companies want to run their sites is not the answer. The problem about privacy is that, left in the hands of those providing websites (and advertising networks via data brokers and surveillance), nothing would improve. They have to be forced to improve the state of end user privacy.
This reminds me so much of the ACID tests[1] from the late 90's, early 2000s. I wonder if it will have a similar effect, i.e. to drive people away from Chrome in the same way it eventually drove people away from IE.
[1] https://en.wikipedia.org/wiki/Acid1
[1] https://en.wikipedia.org/wiki/Acid1
I don't think the Acid tests had anything to do with Chrome's rise. That thing was pushed and promoted in ways that reminded everyone with half a brain (and knowledge) of malware. As difficult to remove, too.
Firefox with these settings from Librewolf looks very much equivalent: https://gitlab.com/librewolf-community/settings/-/blob/maste...
I would disagree. Most of the fingerprinting protection is enabled by the user.js, of which LibreWolf inherits the efforts made by arkenfox[1][2]. Many of the most significant preferences themselves made it into Firefox by the Tor uplift project[3].
[1] https://github.com/arkenfox/user.js
[2] https://librewolf.net/license-disclaimers
[3] https://wiki.mozilla.org/Security/Tor_Uplift
[1] https://github.com/arkenfox/user.js
[2] https://librewolf.net/license-disclaimers
[3] https://wiki.mozilla.org/Security/Tor_Uplift
"Most of the fingerprinting protection is enabled by the user.js."
Right, it seems to me that in many ways it'd make sense if we had the option of restricting the JavaScript engine per se.
As users, if we could determine what functions JavaScript could actually perform then many of the security risks could be avoided. For example, if say we could change or randomize data in response to certain website queries which then resulted in them receiving garbage in ways they could not detect then users would be in much more control. Exactly how JavaScript would respond could be set in the browser's settings and changed as necessary.
I've long wondered why privacy browsers don't go to the heart of the problem and use modified JS engines. I'd love to know what those who're involved with browser development think about such a tactic and why it hasn't previously developed any traction.
Right, it seems to me that in many ways it'd make sense if we had the option of restricting the JavaScript engine per se.
As users, if we could determine what functions JavaScript could actually perform then many of the security risks could be avoided. For example, if say we could change or randomize data in response to certain website queries which then resulted in them receiving garbage in ways they could not detect then users would be in much more control. Exactly how JavaScript would respond could be set in the browser's settings and changed as necessary.
I've long wondered why privacy browsers don't go to the heart of the problem and use modified JS engines. I'd love to know what those who're involved with browser development think about such a tactic and why it hasn't previously developed any traction.
Where would I put this file on my Firefox install ?
The file's first line points to a README with details.
(On my machine it's under "C:\Program Files\Mozilla Firefox\distribution".)
(On my machine it's under "C:\Program Files\Mozilla Firefox\distribution".)
Wow. Glad that I switched to Brave. I've also looked at ungoogled-chromium and other similar forks, but also concerned about tracking the upstream for fixes fast enough; some of the smaller forks take too long. Brave works really well and is a great experience overall (once crypto ads are disabled).
How is the stability of Brave on Linux? When I was using it 2 years ago on macOS it had frequent crashes and was forced to use nightly builds with lesser crashes.
It would be tough to change from Firefox to Brave on Linux, FF feels native to Linux after all those years of being default browser on several distros.
Also I wonder what would be the test results if FF was run with uBlock Origin; As FF is the best browser to use uBlock with due to lesser API level restrictions.
It would be tough to change from Firefox to Brave on Linux, FF feels native to Linux after all those years of being default browser on several distros.
Also I wonder what would be the test results if FF was run with uBlock Origin; As FF is the best browser to use uBlock with due to lesser API level restrictions.
brave iz zhe beztezt!
Very useful table. But it would be very helpful if the table of test results were furnished with a glossary that:
(a) explains the function and workings of each item under test, and,
(b) also explain how said item is a threat to one's privacy, etc. (how it leaks one's info).
This ought to be important, for many users wouldn't have a clue what some or even many of those functions do and it wouldn't be expected as such that they do (given that the internal working of browsers is a specialized business).
Second, I tried to send feedback to this effect but could not as I'm not a Twitter user (it seemingly being the only of sending feedback to PrivacyTests.org).
It is highly annoying that such groups don't have alternative more neutral methods of feeding back such information, especially so in this instance given PrivacyTests.org's stance on privacy (that being its primary reason for being).
Perhaps someone with a Twitter account who thinks similarly could send them my feedback and or send them this link.
Edit: Re results, I was rather surprised to see how poorly Tor featured in the tracking department, it failing every test. I'd have thought tracking failures would be deemed a significant security risk.
Would someone like to comment on this? (BTW, I've little need of Tor's security, although I've used it on occasions, thus I can't claim I'm very familiar with it.)
(a) explains the function and workings of each item under test, and,
(b) also explain how said item is a threat to one's privacy, etc. (how it leaks one's info).
This ought to be important, for many users wouldn't have a clue what some or even many of those functions do and it wouldn't be expected as such that they do (given that the internal working of browsers is a specialized business).
Second, I tried to send feedback to this effect but could not as I'm not a Twitter user (it seemingly being the only of sending feedback to PrivacyTests.org).
It is highly annoying that such groups don't have alternative more neutral methods of feeding back such information, especially so in this instance given PrivacyTests.org's stance on privacy (that being its primary reason for being).
Perhaps someone with a Twitter account who thinks similarly could send them my feedback and or send them this link.
Edit: Re results, I was rather surprised to see how poorly Tor featured in the tracking department, it failing every test. I'd have thought tracking failures would be deemed a significant security risk.
Would someone like to comment on this? (BTW, I've little need of Tor's security, although I've used it on occasions, thus I can't claim I'm very familiar with it.)
Hi -- thanks for the comments!
> it would be very helpful if the table of test results were furnished with a glossary that:
There is some explanation for each test -- to see these explanations, you need to click on category titles, test titles, or test results themselves. But I take your point that these annotations need to be expanded and easier to find.
> I tried to send feedback to this effect but could not as I'm not a Twitter user (it seemingly being the only of sending feedback to PrivacyTests.org).
Actually in the upper-right corner of each page there is a link to an email address for feedback ([email protected]). And github issues are also welcome. I gather these links could be made more clear. :)
> I was rather surprised to see how poorly Tor featured in the tracking department, it failing every test.
The tracking content section results actually stem from Tor Browser's approach: Tor Browser currently does not block any third-party content from loading in a page. Rather it prevents tracking by various policies, including always using the Tor network, providing full state partitioning, and providing strong fingerprinting resistance. Generally speaking, third-party trackers are prevented from tracking users by these measures. However, I think blocking of trackers could offer defense in depth in Tor Browser, in case any of the other measures fails.
> it would be very helpful if the table of test results were furnished with a glossary that:
There is some explanation for each test -- to see these explanations, you need to click on category titles, test titles, or test results themselves. But I take your point that these annotations need to be expanded and easier to find.
> I tried to send feedback to this effect but could not as I'm not a Twitter user (it seemingly being the only of sending feedback to PrivacyTests.org).
Actually in the upper-right corner of each page there is a link to an email address for feedback ([email protected]). And github issues are also welcome. I gather these links could be made more clear. :)
> I was rather surprised to see how poorly Tor featured in the tracking department, it failing every test.
The tracking content section results actually stem from Tor Browser's approach: Tor Browser currently does not block any third-party content from loading in a page. Rather it prevents tracking by various policies, including always using the Tor network, providing full state partitioning, and providing strong fingerprinting resistance. Generally speaking, third-party trackers are prevented from tracking users by these measures. However, I think blocking of trackers could offer defense in depth in Tor Browser, in case any of the other measures fails.
Thank you for your reply on all points.
On the matter of testing browser exploits etc. it seems to me there's no decent comprehensive list of exploitable browser functions that's easily comprehensible to normal users as well as being easily accessible.
I reckon that a link on your homepage to a well organized table etc. that lists browser function names, their description or explanation thereof together with their various exploits (description, modus operandi, notes, links to more info etc.) in an easily readable format would be very worthwhile as would also draw users to your site for that reason alone. When there, they'd also find the browser tests.
(There's any amount of stuff on the web about browsers, exploits etc. but I've not seen one that's comprehensive in that it brings all three aspects into one place.)
About Tor, I would certainly agreed with you. Bootstrapping with fallback security makes sense. If there's any argument then make it optional at the click of a button, etc. In a way, your test results have acted as a review and I reckon that's a good thing.
Frankly, I'm horribly disenchanted with most browsers and we need a site that reviews most of them in an objective and comprehensive way. Perhaps in the future you might consider doing this by reviewing browsers in the light of your tests.
On the matter of testing browser exploits etc. it seems to me there's no decent comprehensive list of exploitable browser functions that's easily comprehensible to normal users as well as being easily accessible.
I reckon that a link on your homepage to a well organized table etc. that lists browser function names, their description or explanation thereof together with their various exploits (description, modus operandi, notes, links to more info etc.) in an easily readable format would be very worthwhile as would also draw users to your site for that reason alone. When there, they'd also find the browser tests.
(There's any amount of stuff on the web about browsers, exploits etc. but I've not seen one that's comprehensive in that it brings all three aspects into one place.)
About Tor, I would certainly agreed with you. Bootstrapping with fallback security makes sense. If there's any argument then make it optional at the click of a button, etc. In a way, your test results have acted as a review and I reckon that's a good thing.
Frankly, I'm horribly disenchanted with most browsers and we need a site that reviews most of them in an objective and comprehensive way. Perhaps in the future you might consider doing this by reviewing browsers in the light of your tests.
More useful would be showing what can be enabled (if it's not on by default), how useful it actually is (not every checkmark is created equally heh), and what can be added (especially on the tracking side) by simply installing μblock origin (which everyone would/should do in every browser anyway).
>simply installing μblock origin (which everyone would/should do in every browser anyway).
If only I could install it in Safari I might be able to browse and watch videos on ios...
If only I could install it in Safari I might be able to browse and watch videos on ios...
Agreed -- in the future I'm hoping to have a page showing results with browsers with various privacy-helpful extensions installed.
Very interesting! I’m positively surprised by the iOS situation, also good to see Tor perform well. Never tried librewolf but I might give it a go.
I’m curious what it would look like with some extensions installed.
I’m curious what it would look like with some extensions installed.
Surprising to see librewolf there; Last time i tried any firefox forks they were all frankly terrible;My go-to privacy browser for work-related stuff or anything serious is ungoogled chromium for years now, any other chromium-based forks like Brave seem private only as a facade.Vivaldi has the best gesture system of any browser hands-down, but my issue in the past with it was performance(especially HW accell).
Also ff-based forks have the problem of availability of packages & ease of installation.
Also ff-based forks have the problem of availability of packages & ease of installation.
This is under the assumption that each browser is being used with the default settings, firefox can do much better with customizable settings.
> This is under the assumption that each browser is being used with the default settings
That is the only valid assumption. The absolute vast majority of users will use any and all software under default settings.
That is the only valid assumption. The absolute vast majority of users will use any and all software under default settings.
The entire last two sections are completely arbitrary and cherry-picked, and simply amount to "does the browser ship uBlock and ClearURLs by default with these specific filters", which isn't very informative nor useful a privacy feature, as easy as it is to circumvent by simply using different URL tokens or telemetry providers.
Thank you for the feedback!
Granted, blocklists (lists of tracking domains or URL query parameters) can be circumvented by a determined attacker. Indeed, I agree that blocklists aren't sufficient on their own for a browser to provide solid privacy protection. In my view it's critical, primarily, to have policies that enforce privacy, including such protections as state partitioning and fingerprinting resistance. That's exactly why I included tests for such policies.
However: I do think blocklists provide substantial, though incomplete, privacy protection in practice. And, importantly, blocklists are enforced by a number of popular browsers (Brave, DuckDuckGo, Firefox Private Mode, Firefox Focus) and popular browser extensions and other services (uBlock, ClearURLs, DuckDuckGo Privacy Essentials, Disconnect, etc.). These blocklists seem to work pretty well, at least judging by the ad-free experience they provide. So I felt that to give a more complete picture I should test for blocking.
I tried to avoid cherry picking query parameters or blockers. Here's how I arrived at the current selections for these two sections:
* Tracking query parameter tests: I tried to gather all the query parameters I could find; the list on the page was my full list at the time. (If there are suggestions for more parameters, I will be happy to add them.)
* Tracker content blocking tests: I used the list of the top 20 tracking entities from https://whotracks.me. These are, roughly speaking, 20 of the most widespread third-party tracking domains on the web -- they should be a high priority for any browser respecting privacy, in my opinion. I hope testing for blocking of these 20 serves to gives a sense of each browser's approach to third-party tracking scripts and pixels.
Granted, blocklists (lists of tracking domains or URL query parameters) can be circumvented by a determined attacker. Indeed, I agree that blocklists aren't sufficient on their own for a browser to provide solid privacy protection. In my view it's critical, primarily, to have policies that enforce privacy, including such protections as state partitioning and fingerprinting resistance. That's exactly why I included tests for such policies.
However: I do think blocklists provide substantial, though incomplete, privacy protection in practice. And, importantly, blocklists are enforced by a number of popular browsers (Brave, DuckDuckGo, Firefox Private Mode, Firefox Focus) and popular browser extensions and other services (uBlock, ClearURLs, DuckDuckGo Privacy Essentials, Disconnect, etc.). These blocklists seem to work pretty well, at least judging by the ad-free experience they provide. So I felt that to give a more complete picture I should test for blocking.
I tried to avoid cherry picking query parameters or blockers. Here's how I arrived at the current selections for these two sections:
* Tracking query parameter tests: I tried to gather all the query parameters I could find; the list on the page was my full list at the time. (If there are suggestions for more parameters, I will be happy to add them.)
* Tracker content blocking tests: I used the list of the top 20 tracking entities from https://whotracks.me. These are, roughly speaking, 20 of the most widespread third-party tracking domains on the web -- they should be a high priority for any browser respecting privacy, in my opinion. I hope testing for blocking of these 20 serves to gives a sense of each browser's approach to third-party tracking scripts and pixels.
The tests are neat, Thank you.
Can you update the steps to run the tests locally to test against browser with uBlock Origin vs default browser?
Can you update the steps to run the tests locally to test against browser with uBlock Origin vs default browser?
These really need context or better explanations.
For instance I clicked on the Blob line and the code looks to fetch and URL with a Blob encoded and fetches it again? There is so little context to say what is really wrong... or if there is anything really wrong.
I looked up Blobs myself and read through the specs on MDN and I just dont see a problem.
For instance I clicked on the Blob line and the code looks to fetch and URL with a Blob encoded and fetches it again? There is so little context to say what is really wrong... or if there is anything really wrong.
I looked up Blobs myself and read through the specs on MDN and I just dont see a problem.
Thank you for the feedback -- I agree more context and explanation is needed for each of these tests.
In the Blob case: the test code is storing a unique string in a Blob URL under one website (first party), and then attempting to read back that string under a second, different website. (See "result, different first party".) If the string is accessible under a different first party, then it is possible to use a Blob URL to track a user between two different websites.
In the Blob case: the test code is storing a unique string in a Blob URL under one website (first party), and then attempting to read back that string under a second, different website. (See "result, different first party".) If the string is accessible under a different first party, then it is possible to use a Blob URL to track a user between two different websites.
Oh ok. That does make sense. Hopefully you read my comment as feedback and not super negative.
Just some verbiage on each test would be wonderful.
You have clearly worked on it. It is a really good resource.
Just some verbiage on each test would be wonderful.
You have clearly worked on it. It is a really good resource.
> Oh ok. That does make sense. Hopefully you read my comment as feedback and not super negative.
It was helpful feedback. I value all critiques because they help me make the site better.
> Just some verbiage on each test would be wonderful.
There is some explanation for each test, if you click on the test name. But it's clear I need to expand those explanations and also make them easier to find.
It was helpful feedback. I value all critiques because they help me make the site better.
> Just some verbiage on each test would be wonderful.
There is some explanation for each test, if you click on the test name. But it's clear I need to expand those explanations and also make them easier to find.
Brave rulez