WireGuard Servers Running from RAM(mullvad.net)
mullvad.net
WireGuard Servers Running from RAM
https://mullvad.net/tr/help/wireguard-servers-running-from-ram/
141 comments
gigatexal(6)
Not if the whole os is a VM in ram, still though, network logging can happen by middleware.
Even without network logging, the hypervisor can still access the memory of the VM, parsing the kernel process and paging structures to read process memory, which can store transient logging information.
Things like AMD SEV/SEV-ES are designed to mitigate that aspect, the implementation of which can be proven to end-users of the VM by remote attestation.
Things like AMD SEV/SEV-ES are designed to mitigate that aspect, the implementation of which can be proven to end-users of the VM by remote attestation.
That's not necessarily the case, vendors have VMs that can spin up with encrypted memory so that only the person spinning them up can see what is in the memory. All AMD and Intel processors have capabilities for this. Just in case you didn't know.
Sure you can. For example, with rsyslog you can just send all logs over the network to a logging server (which has disks).
That's the network, someone looking at the logs via ssh is also a leak by that logic. Of course!
[deleted]
https://github.com/mullvad/system-transparency seems to include the stboot mentioned in the article.
Would be pretty bad transparency if the code providing it wasn't transparent haha.
I get the immediate appeal of “open source servers that run from RAM”… but it’s the same thing as always, you just have to believe your VPN provider.
There is just no saying that they don’t swap gear out for the auditors or for the disk less servers or whatever it is.
I like that they say the right things. But I think the current models of Facebook and Google show just how much money there is in data, so it’s just faith that the VPN provider won’t fall to temptation.
There is just no saying that they don’t swap gear out for the auditors or for the disk less servers or whatever it is.
I like that they say the right things. But I think the current models of Facebook and Google show just how much money there is in data, so it’s just faith that the VPN provider won’t fall to temptation.
You are right.
But.
Mullvad is more trustworthy that most, and possibly more trustworthy than all other public VPN providers. They have a good history, and are regularly audited: https://mullvad.net/en/blog/tag/audits/
Don't get me wrong, you still have to believe them. But they're easier to believe than others.
But.
Mullvad is more trustworthy that most, and possibly more trustworthy than all other public VPN providers. They have a good history, and are regularly audited: https://mullvad.net/en/blog/tag/audits/
Don't get me wrong, you still have to believe them. But they're easier to believe than others.
Well, I have a blog post regarding audits: https://patrakov.blogspot.com/2020/01/vpn-privacy-policies-a...
In short: auditors audit providers against their own privacy policies. If there is a loophole, the result, with some auditors, might still be a legitimate "pass" - e.g. because exfiltration is not logging, and the privacy policy says only "we don't log".
P.S. Mullvad's privacy policy falls into the same bucket: "we never store any activity logs of any kind". Sending data to a third party who stores it would be 100% compliant. Mullvad, please fix the wording.
In short: auditors audit providers against their own privacy policies. If there is a loophole, the result, with some auditors, might still be a legitimate "pass" - e.g. because exfiltration is not logging, and the privacy policy says only "we don't log".
P.S. Mullvad's privacy policy falls into the same bucket: "we never store any activity logs of any kind". Sending data to a third party who stores it would be 100% compliant. Mullvad, please fix the wording.
Their most recent audit contains this quote:
> In regards to information leakage and logging of customer data the configuration is sound
I cannot imagine this could ever mean that Mullvad sends data to third parties.
> In regards to information leakage and logging of customer data the configuration is sound
I cannot imagine this could ever mean that Mullvad sends data to third parties.
We are talking about different documents. I mention the privacy policy, which contains a "leaky" wording, likely unintentionally. You mention the audit report that plugs the leak and confirms that Mullvad protects customer data even better than written in their privacy policy.
Mullvad is pretty much the only VPN provider saying the right things and doing the right things. They actually publish open source code and articles backing up their statements.
Ontop of that many of us here in Sweden know people at Mullvad and can attest to them being conscious of privacy issues.
Ontop of that many of us here in Sweden know people at Mullvad and can attest to them being conscious of privacy issues.
> many of us here in Sweden know people at Mullvad and can attest to them being conscious of privacy issues
Me for example, I can vouch for them if that matters to anyone, but it's the same problem isn't it, if people can't trust Mullvad on their word, why should they trust two random HN users they've never met or conversed with before?
Trust in IT systems is super complicated.
Me for example, I can vouch for them if that matters to anyone, but it's the same problem isn't it, if people can't trust Mullvad on their word, why should they trust two random HN users they've never met or conversed with before?
Trust in IT systems is super complicated.
Yup. But you can use crypto or mail cash to Mullvad without your name attached, and then just multihop to them as an endpoint, and it's about as secure as a VPN can be.
I like the idea. But it’s pointless if the connections do log - or leak data in some way.
You are mostly connecting to the VPN with a same or similar IP. So right there; you are back to having to trust they aren’t logging or leaking.
If they are, it doesn’t matter Mull is missing your name, you are already correlated from ISP to endpoint in this case.
Likewise, if they aren’t logging, what does it matter if they have your name and account info?
You are mostly connecting to the VPN with a same or similar IP. So right there; you are back to having to trust they aren’t logging or leaking.
If they are, it doesn’t matter Mull is missing your name, you are already correlated from ISP to endpoint in this case.
Likewise, if they aren’t logging, what does it matter if they have your name and account info?
It is not pointless, it is a huge practical win. That means that you don't have to trust your VPN to have a reasonable guarantee that adding VPN won't reduce your privacy (depending on your goals).
As many VPNs are sketchy (and even the best one might get bought out or otherwise get compromised) relying on one might seriously reduce your privacy. Especially considering that in many jurisdictions your ISP has much higher legal requirements on integrity and privacy.
But if the VPN don't know who you are the absolute worst case is that you fall back to your ISP. Whether that is good enough for you is something you have to check against your threat model. But in a world where adding a VPN in many cases threatens to reduce your privacy it is massive win.
As many VPNs are sketchy (and even the best one might get bought out or otherwise get compromised) relying on one might seriously reduce your privacy. Especially considering that in many jurisdictions your ISP has much higher legal requirements on integrity and privacy.
But if the VPN don't know who you are the absolute worst case is that you fall back to your ISP. Whether that is good enough for you is something you have to check against your threat model. But in a world where adding a VPN in many cases threatens to reduce your privacy it is massive win.
"fall to temptation" implies they were legitimate actors in the first place.
Most VPNs are run by small and relatively non-transparent private companies. It would be trivial and in fact the obvious thing to do for law enforcement and similar agencies to setup VPN companies and advertise them to their targets.
ISPs are actually more transparent and their business model does not fundamentally relies on exploiting customer data.
Most VPNs are run by small and relatively non-transparent private companies. It would be trivial and in fact the obvious thing to do for law enforcement and similar agencies to setup VPN companies and advertise them to their targets.
ISPs are actually more transparent and their business model does not fundamentally relies on exploiting customer data.
> ISPs are actually more transparent and their business model does not fundamentally relies on exploiting customer data.
And yet they do, largely because many people _can't_ vote with their dollars. I can move which VPN provider I use. Also, VPN's are a big benefit on filtered wifi if that's something you have to deal with.
And yet they do, largely because many people _can't_ vote with their dollars. I can move which VPN provider I use. Also, VPN's are a big benefit on filtered wifi if that's something you have to deal with.
> ISPs are actually more transparent and their business model does not fundamentally relies on exploiting customer data.
To be fair this applies to VPNs too. Their customers give them money, and that should make their business sustainable without needing to spy on people.
To be honest I trust Mullvad way more than I trust my ISP not to sell my DNS queries.
To be fair this applies to VPNs too. Their customers give them money, and that should make their business sustainable without needing to spy on people.
To be honest I trust Mullvad way more than I trust my ISP not to sell my DNS queries.
gruez(1)
I was going to say that theoretically this was something that trusted computing can solve, but then I realized even if you could guarantee that the server was behaving as advertised (ie. diskless, no log), there's nothing preventing the provider from putting a middlebox in front of the server to log all packet flows. From there it's fairly trivial to deanonymize every connection through traffic analysis.
The provider doesn’t have to do anything. Their upstream transit provider probably has a span port plugged directly into the NSA. Which is what was happening with Room 641A over a decade ago.
On one hand, they won’t be able intercept encrypted VPN traffic, but on the other, they are definitely collecting IP addresses, timestamps and traffic heuristics. With enough resources to pick out patterns and bulk collection on both sides, encryption and logging might not even matter.
On one hand, they won’t be able intercept encrypted VPN traffic, but on the other, they are definitely collecting IP addresses, timestamps and traffic heuristics. With enough resources to pick out patterns and bulk collection on both sides, encryption and logging might not even matter.
I bet some kind of blockchain based trust can solve some issues ,for eg , hardware ownership delegated with blockchain trust .It will add p2p ownership/rent of hardware
I may be wrong. But OVPN is the only one I know that has actually been proven in court not to store any user activity.
https://www.ovpn.com/en/blog/ovpn-wins-court-order
https://www.ovpn.com/en/blog/ovpn-wins-court-order
PIA (Privateinternetaccess) had a similar case: https://torrentfreak.com/private-internet-access-no-logging-...
This was before they were bought by some shady company though, so I wouldn't rely on this policy being unchanged anymore.
This was before they were bought by some shady company though, so I wouldn't rely on this policy being unchanged anymore.
The bad things is, the past does not predict the future ( would be awesome ^^ )
They did not log, but maybe they are now. I don’t say that ovpn logs, but the 3 letter agencies can infiltrate a company.
They did not log, but maybe they are now. I don’t say that ovpn logs, but the 3 letter agencies can infiltrate a company.
Or as the catchphrase of the scientist character of Welcome to Night Vale goes: Past performance is not an indicator of future results.
Wouldn't there be a clear performance difference between a RAM based and disk based (even if it's NVMe SSD) machine? If so, isn't that something you can routinely/programmatically authenticate?
I doubt there would be any detectable difference. Logging to disk can easily be done out of the hot loop, and I doubt any service with disks has any relevant memory in swap/page file.
The point of the RAM-only server is not that it ensures that everything is operating from RAM. Everything is operating from RAM already even on servers that have attached disks. The point is that RAM-only means there is no intentional or even unintentional logging to non-volatile memory or storage. Think of it as a physics enforced capability system (no disk is physically connected).
The point of the RAM-only server is not that it ensures that everything is operating from RAM. Everything is operating from RAM already even on servers that have attached disks. The point is that RAM-only means there is no intentional or even unintentional logging to non-volatile memory or storage. Think of it as a physics enforced capability system (no disk is physically connected).
Fair enough on there not being a performance difference. I suppose you could run an IPMI or redfish query (assuming they expose it to you) to get hardware specs on the server to see if any storage is physically connected?
I guess there's a larger question - is it possible to construct a completely transparent architecture for customers who are trustless in you as a service provider?
I guess there's a larger question - is it possible to construct a completely transparent architecture for customers who are trustless in you as a service provider?
You just described https://en.m.wikipedia.org/wiki/Software_Guard_Extensions
The CPU essentially signs running code and API responses using a key that only the CPU manufacturer knows. That way, you can verify that your cloud services are running the binaries you told them to run.
Note the long list of vulnerabilities on that page and the removal of this feature from desktop CPUs. (Let’s be real, its only use case on desktop is DRM)
The CPU essentially signs running code and API responses using a key that only the CPU manufacturer knows. That way, you can verify that your cloud services are running the binaries you told them to run.
Note the long list of vulnerabilities on that page and the removal of this feature from desktop CPUs. (Let’s be real, its only use case on desktop is DRM)
No.
I mean, you need some kind of trust, somewhere. Maybe you don't have to trust the service provider, if it provides some type of TPM attestation traced to the key of someone you do trust.
On the other hand, they have physical access. Even with efforts at remote attestation, etc, the game is lost.
I mean, you need some kind of trust, somewhere. Maybe you don't have to trust the service provider, if it provides some type of TPM attestation traced to the key of someone you do trust.
On the other hand, they have physical access. Even with efforts at remote attestation, etc, the game is lost.
> RAM-only means there is no intentional or even unintentional logging to non-volatile memory or storage
Let’s say worst case, there is an unintentional leak to a another machine, pretty likely that machine has a disk. These are very obviously highly connected machines. Sorry, but it can never be anything but faith - which is fine if you have or, or chose to.
Let’s say worst case, there is an unintentional leak to a another machine, pretty likely that machine has a disk. These are very obviously highly connected machines. Sorry, but it can never be anything but faith - which is fine if you have or, or chose to.
I don't have a lot of faith in such things, at base, you always have to trust the provider as people or an organization. I'm just describing what this technical measure is and what it is not. It's also not a silver bullet that on its own means you can trust their service enough for your needs -- no purely technical measure can do that.
Forwarding packets doesn't need much if any disk i/o, so the server will be running mostly from RAM either way, even if the hard drive isn't physically unplugged.
Would it be clear from 100km+ away?
I think that low latency can act as a proof of being close and fast, but once the apparent network latency goes up how do you know that server times are low if the server can claim the time was spent on the network?
I think that low latency can act as a proof of being close and fast, but once the apparent network latency goes up how do you know that server times are low if the server can claim the time was spent on the network?
> you just have to believe your VPN provider
Do you have a better plan...?
Do you have a better plan...?
I think the only thing that would give more trust is a remote attestation setup similar to GrapheneOS[1] since it's the only way to prove that servers are running the software they publicly say they are.
The configuration would have to be run at the hypervisor level, so that logging couldn't simply occur by nesting the server in a VM then logging that.
[1] https://attestation.app
The configuration would have to be run at the hypervisor level, so that logging couldn't simply occur by nesting the server in a VM then logging that.
[1] https://attestation.app
Reminds me of old days when ed2k servers are entirely running on RAM to avoid raids.
I read this differently than intended. For the life of me, I couldn’t figure out why RAIDs were so troublesome. I mean, yes, they can be a little slow, so running in RAM would be much faster.
Then I got it. Raid. Not RAID. Made so much more sense.
Then I got it. Raid. Not RAID. Made so much more sense.
Good old times.
Running from memory is no big deal. You gain all and everything like a normal system and can still log persistent to some lokal disk or a remote central log service as it is quite common.
For example, all Ceph storage nodes bootet with croit.io run in RAM and have no OS installed. But you still have all logs and everything available right out of the box.
For example, all Ceph storage nodes bootet with croit.io run in RAM and have no OS installed. But you still have all logs and everything available right out of the box.
Read what they are actually doing, its not just running from ram.
A trick I've used in the past for secrets on cloud providers: store them in "/dev/shm". Requires init after boot from a trusted source of course.
It's absolutely not foolproof but reduces the odds of them ending up on a SAN somewhere where they might be found by someone scanning free space or gaining direct access to an iSCSI bus or similar.
It's absolutely not foolproof but reduces the odds of them ending up on a SAN somewhere where they might be found by someone scanning free space or gaining direct access to an iSCSI bus or similar.
Adding to this for completeness sake, if using tmpfs such as the default /dev/shm mount, one must ensure that swap is either encrypted or disabled as tmpfs is swap backed and data not being actively accessed/written can end up in swap on disk.
Lets say, hypothetically, that there did exist some trustworthy VPN provider, one that behaved in a way that you would approve (where, if you had unlimited time and access to the internal operation of the provider to evaluate it, then you would eventually be convinced that it's "good", as good as reasonably, humanly possible). So assuming it exists, what would it look like? What would you see that would be positive or negative signs?
[deleted]
I love that Mullvad is doing these public-facing experiments.
Sweden is going to join NATO soon. So do not expect VPNs operating from their teritory and under their law to be any different then the rest of Europe, as it was before.
[deleted]
Meh, I generally agree with the laws of my country and of the EU, so I don’t really care if they can obtain a warrant to wiretap me. I prefer to live in a society where law enforcement can be effective (with checks!) than one where bad things can happen with zero negative consequences.
How would you expect Sweden joining NATO to change privacy laws for that country?
Sweden is a 14 eyes country already.
Running entirely in RAM is same as running from an encrypted disk, with keys available only to whoever can dump the RAM.
A better solution would be something like Apple’s private relay.
Also, either you trust the provider or you don’t.
A better solution would be something like Apple’s private relay.
Also, either you trust the provider or you don’t.
Apple’s private relay is just CloudFlare WARP with some additional IP blocks + locality built in, not?
I wouldn’t bet on it being super secure in case law enforcement comes after you, for example.
I wouldn’t bet on it being super secure in case law enforcement comes after you, for example.
Here’s the whitepaper: https://www.apple.com/privacy/docs/iCloud_Private_Relay_Over...
They do two hops, first to an Apple-controlled server, then to the “second relay” which is operated by Cloudflare in a lot of cases. Encryption is terminated at the second relay.
So Cloudflare sees the content (or whatever is visible in a TLS stream), and Apple sees your real IP, but neither can know both without collaboration.
They do two hops, first to an Apple-controlled server, then to the “second relay” which is operated by Cloudflare in a lot of cases. Encryption is terminated at the second relay.
So Cloudflare sees the content (or whatever is visible in a TLS stream), and Apple sees your real IP, but neither can know both without collaboration.
Not exactly. WARP hides your content and IP from anyone but the destination website. The website can still track the user. Apple mixed it with own relay such that neither Apple nor Cloudflare can track the IP and content simultaneously.
It’s mostly an anti-tracking feature. But also now government needs cooperation from two companies.
On another note, WARP is a VPN. But Mullvad is preferred to WARP, because Cloudflare most likely logs connections for some time.
It’s mostly an anti-tracking feature. But also now government needs cooperation from two companies.
On another note, WARP is a VPN. But Mullvad is preferred to WARP, because Cloudflare most likely logs connections for some time.
You seem to be contradicting yourself.
> A better solution would be something like Apple’s private relay.
then
> On another note, WARP is a VPN. But Mullvad is preferred to WARP
> A better solution would be something like Apple’s private relay.
then
> On another note, WARP is a VPN. But Mullvad is preferred to WARP
Private relay (WARP + relay) > Mullvad > WARP
But keep in mind that private relay applies only to Safari traffic. For applications, use Mulvad.
But keep in mind that private relay applies only to Safari traffic. For applications, use Mulvad.
Not really equivalent. There are possible attacks based on: key generation process, stored data correlation, access patterns, etc. You're much safer if you don't store anything in the first place.
I would trust real TOR much more than Apple's pseudo-TOR. They control all the entry and exit nodes so correlation attacks are quite viable.
Might as well chain two VPNs if you want a TOR-like experience without slowdowns. Bonus feature is that you can rotate providers.
Of course, but Tor speeds are KB/s or low Mb/s in my experience.
> Running the system in RAM does not prevent the possibility of logging. It does however minimise the risk of accidentally storing something that can later be retrieved.
This doesn’t mean you can’t be logged. Running in RAM just means that any system level logging is transient and largely accidental. But if there were a need to specific logging, data could always be sent to a different node with disks. From Mullvad’s point of view, there is a reliability benefit to having diskless nodes. But from a privacy point of view, your access could still be logged, if required. But it would probably require more “active” monitoring than “passive”/accidental logging.
https://mullvad.net/en/blog/2022/1/12/diskless-infrastructur...