Rsync.net Technical Notes – Q4 2021(rsync.net)
rsync.net
Rsync.net Technical Notes – Q4 2021
https://www.rsync.net/resources/notes/2021-q4-rsync.net_technotes.html
49 comments
> The state of userspace sandboxing across all operating systems I'm aware of is dismal. As a developer it should be trivial to spawn a process with a subset of the privileges of the parent process.
It's a breeze on OpenBSD, at least as an application developer. The key is realizing that you can't effectively sandbox a process without the process cooperating. That often means refactoring the source code to reorganize how and when it acquires resources. If you're refactoring the source code, then in many cases it's easier and most effective for the process to sandbox itself, thus pledge(2) (https://man.openbsd.org/pledge.2) and unveil(2) (https://man.openbsd.org/unveil.2). Afterall, the process and its authors are the final and best arbiters of the resources it will need to initialize and acquire during runtime.
All the pain on Linux comes from people struggling to sandbox uncooperative processes (i.e. processes you can't modify and refactor) by trying to construct a fake environment. That's ultimately a losing battle. And quite ironic that people instinctively seek that route even in the context of open source applications where source code is readily available and upstreaming patches and improvements extremely common. OpenBSD has managed to sandbox to some extent almost every program, including command-line utilities, that ship with the OS. And they continue to incrementally improve the those sandboxes over time--IOW, pledge and unveil are amenable to iterative, steady development of capability management, rather than requiring herculean effort up front, potentially resulting brittle architectures that make feature additions excessively difficult.
While there's nothing quite as simple as pledge on Linux (seccomp is close but has several problematic caveats), something very similar to unveil was upstreamed into Linux last year: Landlock. See https://docs.kernel.org/security/landlock.html Unfortunately, AFAICT, Landlock, like seccomp, is mandatorily inherited on fork (no option to disable), which means it can be difficult to integrate into existing applications that might fork or exec other processes--they might be completely unrelated and require a privilege set that isn't a subset of the invoker. Reordering the sequence of process spawning to before the privilege drop can be impractical--effectively impossible if it means basically rewriting the entire program, for example with a read-eval loop (e.g. procmailrc). Spawning processes indirectly through, e.g., systemd or pkgexec, simply shifts the problem and can even increase exposure and bug risk. But for daemon services which are only leaves on the process tree, that's not a concern; Landlock and to a lesser extent seccomp can better approximate unveil and pledge, respectively.
It's a breeze on OpenBSD, at least as an application developer. The key is realizing that you can't effectively sandbox a process without the process cooperating. That often means refactoring the source code to reorganize how and when it acquires resources. If you're refactoring the source code, then in many cases it's easier and most effective for the process to sandbox itself, thus pledge(2) (https://man.openbsd.org/pledge.2) and unveil(2) (https://man.openbsd.org/unveil.2). Afterall, the process and its authors are the final and best arbiters of the resources it will need to initialize and acquire during runtime.
All the pain on Linux comes from people struggling to sandbox uncooperative processes (i.e. processes you can't modify and refactor) by trying to construct a fake environment. That's ultimately a losing battle. And quite ironic that people instinctively seek that route even in the context of open source applications where source code is readily available and upstreaming patches and improvements extremely common. OpenBSD has managed to sandbox to some extent almost every program, including command-line utilities, that ship with the OS. And they continue to incrementally improve the those sandboxes over time--IOW, pledge and unveil are amenable to iterative, steady development of capability management, rather than requiring herculean effort up front, potentially resulting brittle architectures that make feature additions excessively difficult.
While there's nothing quite as simple as pledge on Linux (seccomp is close but has several problematic caveats), something very similar to unveil was upstreamed into Linux last year: Landlock. See https://docs.kernel.org/security/landlock.html Unfortunately, AFAICT, Landlock, like seccomp, is mandatorily inherited on fork (no option to disable), which means it can be difficult to integrate into existing applications that might fork or exec other processes--they might be completely unrelated and require a privilege set that isn't a subset of the invoker. Reordering the sequence of process spawning to before the privilege drop can be impractical--effectively impossible if it means basically rewriting the entire program, for example with a read-eval loop (e.g. procmailrc). Spawning processes indirectly through, e.g., systemd or pkgexec, simply shifts the problem and can even increase exposure and bug risk. But for daemon services which are only leaves on the process tree, that's not a concern; Landlock and to a lesser extent seccomp can better approximate unveil and pledge, respectively.
Thanks for the info. I'll look into Landlock.
I'm totally fine changing the way I write my applications to make them more secure. I just don't feel like the APIs are there.
Personally I think capability-based security models is the best idea I've seen so far. Set up the file descriptors and network sockets your program needs at startup, then drop any other privileges. The problem is file descriptors don't cover things like directory trees AFAIK.
If performance isn't critical, what are your thoughts on using QEMU to wrap an application for security purposes? That might even work on Windows.
I'm totally fine changing the way I write my applications to make them more secure. I just don't feel like the APIs are there.
Personally I think capability-based security models is the best idea I've seen so far. Set up the file descriptors and network sockets your program needs at startup, then drop any other privileges. The problem is file descriptors don't cover things like directory trees AFAIK.
If performance isn't critical, what are your thoughts on using QEMU to wrap an application for security purposes? That might even work on Windows.
I don’t know for sure about the memory and cpu rationing but doesn’t systemd expose the rest of those limits for service units?
Not sure. Even if it does, it's going to be built using the APIs I already mentioned, and therefore not available in userspace. It doesn't make any sense for a process to need special privileges in order to limit a child process to strictly lesser privileges.
> Someone
Why not you?
Why not you?
Because I'm not a security expert. Gaining the knowledge necessary to do it well would probably require a specialization change and years of prep, which would be an inefficient division of labor. Plus, it's an interesting problem, but not one I'm particularly drawn to. I'm currently focused on solving other problems[0] higher up the stack.
[0]: https://boringproxy.io/
[0]: https://boringproxy.io/
Happy, as always, to answer any questions or discuss any comments here.
Thanks again to Allan Jude of Klara Systems and Luke Young of bored.engineer for their very interesting comments and explanations.
(these tech notes were published in March of 2022 - they are backward looking to the events of Q4 2021)
Thanks again to Allan Jude of Klara Systems and Luke Young of bored.engineer for their very interesting comments and explanations.
(these tech notes were published in March of 2022 - they are backward looking to the events of Q4 2021)
[deleted]
Any news on adding the mkfifo command? I requested it a while back and it sounded like it was doable but I have not heard any updates.
Thanks! mkfifo was included in this new command addition/whitelisting and we're going to write up an entire section on it next quarter.
Can you email ? I actually searched for our conversation for a while and could not find it ... bottom line is that 'mkfifo' is in place and I can't wait to craft up some fascinating remote ssh examples with it ...
Can you email ? I actually searched for our conversation for a while and could not find it ... bottom line is that 'mkfifo' is in place and I can't wait to craft up some fascinating remote ssh examples with it ...
Bumped our email thread, thanks!
For everyone else, the mkfifo command lets you transfer arbitrary streams of data between two systems while using rsync.net as an intermediary.
For everyone else, the mkfifo command lets you transfer arbitrary streams of data between two systems while using rsync.net as an intermediary.
I’m slightly confused on why that would be useful within the rsync.net context. Is bouncing traffic a new intended use case?
Hi, the rsync.net website seems to be offline right now. Is there any other link where we can find the technical notes?
I had trouble loading it, too, but managed to get an archive.is capture: https://archive.ph/YEzR5
I can't get archive.ph to load but rsync.net loads fine...
Site loading a bit lumpily for me for a minute there.
Otherwise interesting write-up and pruning unused features I do think helps with security generally.
Otherwise interesting write-up and pruning unused features I do think helps with security generally.
Is native encryption available on the send-capable accounts?
Yes. ZFS encryption and raw send.
Not technical per se, but I'm interested to know how the "unlimited plan for life" experiment worked out. Did many folks take you up on the offer? I didn't, but I strongly considered it (if I'm being honest I just procrastinated the decision until I realized the deadline had passed).
Shhh ... :)
It was, indeed, a very limited test promotion to some small subset of customers and I am still undecided on whether it makes sense to pursue or not.
Generally speaking, I dislike "lifetime" offers and there is some bad history with those being badly abused - specifically by cloud providers. Was it Joyent that dishonored the strongspace lifetime offers ?
I forget ...
It was, indeed, a very limited test promotion to some small subset of customers and I am still undecided on whether it makes sense to pursue or not.
Generally speaking, I dislike "lifetime" offers and there is some bad history with those being badly abused - specifically by cloud providers. Was it Joyent that dishonored the strongspace lifetime offers ?
I forget ...
Yes, Joyent was the one that totally screwed the pooch on that.
I had a few lifetime plans with them. One was with TextDrive, before the Joyent acquisition/merger and that was their shared hosting plan back when they were doing everything on FreeBSD. Then after the merger they did a sort of pre-VPS type plan where it was using Solaris Containers. Those were really cool little boxes.
I also had one of their Strongspace lifetime plans, and that still exists as Expandrive bought them and continues offering the lifetime plans.
Edit: oh, and for how it was ended, Joyent was purchased by Samsung (I think it was) and to get the lifetime stuff off the books they basically forced everyone onto their VPS-like solution but only for ~2 years I think it was. So everything disappeared.
I had a few lifetime plans with them. One was with TextDrive, before the Joyent acquisition/merger and that was their shared hosting plan back when they were doing everything on FreeBSD. Then after the merger they did a sort of pre-VPS type plan where it was using Solaris Containers. Those were really cool little boxes.
I also had one of their Strongspace lifetime plans, and that still exists as Expandrive bought them and continues offering the lifetime plans.
Edit: oh, and for how it was ended, Joyent was purchased by Samsung (I think it was) and to get the lifetime stuff off the books they basically forced everyone onto their VPS-like solution but only for ~2 years I think it was. So everything disappeared.
Generally speaking, I dislike "lifetime" offers and there is some bad history with those being badly abused
Back when Sirius Satellite Radio started, it offered "lifetime" subscriptions. People eventually learned that Sirius meant the lifetime of the radio as long as it tied to the purchaser's account, not the human being.
Many people got Sirius with their new cars. When they started trading them in three or four years later, they learned that their "lifetime" subscription not only didn't go with them to their new car, but the buyer of their old car also didn't get the old subscription.
Today, it wouldn't be as big a deal because you can stream Sirius ($10/month), or some other radio service over the internet. But this was back before even RealPlayer, and the mass adoption of cell phones.
Back when Sirius Satellite Radio started, it offered "lifetime" subscriptions. People eventually learned that Sirius meant the lifetime of the radio as long as it tied to the purchaser's account, not the human being.
Many people got Sirius with their new cars. When they started trading them in three or four years later, they learned that their "lifetime" subscription not only didn't go with them to their new car, but the buyer of their old car also didn't get the old subscription.
Today, it wouldn't be as big a deal because you can stream Sirius ($10/month), or some other radio service over the internet. But this was back before even RealPlayer, and the mass adoption of cell phones.
I remember this.
This is why we’re hesitant - the bad behavior in this area of pricing and promotion makes any offer like this seem scammy …
This is why we’re hesitant - the bad behavior in this area of pricing and promotion makes any offer like this seem scammy …
Honestly, when I'm buying backups, I want to be a paying customer; that aligns my providers incentives with mine. Being a will-never-pay-again customer means that they want to minimize the costs of serving me, and are happy to see me go. No thanks.
This is a great way of thinking about it. Same goes for "unlimited" plans: the more I use, the lower their profit margins. I'd rather agree on terms and pay for what I use.
I think Tivo did a similar thing too, back in the day. (I wasn’t a Tivo customer but I had friends who were.)
Sadly, these things never seem to end well.
Sadly, these things never seem to end well.
Sirius was early 2000s. RealPlayer was 5+ years before that?
Sirius was 1990. RealPlayer was 1995.
I’m not sure that’s true. Wikipedia suggests otherwise.
I got those numbers from Wikipedia.
FWIW, I got my first Sirius radio in 1998.
FWIW, I got my first Sirius radio in 1998.
The timeline for this seems all over the place.
“The first confirmed music transmission from satellite to radio receiver for Sirius was September 1, 2000 in NYC at the current HQ of SiriusXM.”
So what was on in 1998? A precursor to the Sirius service?
“The first confirmed music transmission from satellite to radio receiver for Sirius was September 1, 2000 in NYC at the current HQ of SiriusXM.”
So what was on in 1998? A precursor to the Sirius service?
I got the offer and really considered it as well! It was the fear of an acquisition that convinced me not to take the offer. I was a TextDrive customer before they were acquired by Joyent, I was a LastPass customer before they were acquired by LogMeIn, and it was CrashPlan's cancellation of CrashPlan for Home that drove me to find rsync.net.
None of those other companies are rsync.net, so it's not really fair of me to consider them when making this decision -- but it made me think it's safer to just pay you guys regularly for a good service. Worst case, I spend a little more over the life of my account, but that money is going to a company I really like, so it's not much of a downside.
None of those other companies are rsync.net, so it's not really fair of me to consider them when making this decision -- but it made me think it's safer to just pay you guys regularly for a good service. Worst case, I spend a little more over the life of my account, but that money is going to a company I really like, so it's not much of a downside.
I haven't forgotten how Joyent treated the customer who had lifetime offers and I'll never do business with Joyent again.
I have two accounts in my name, one for me personally and one for work. I got the offer on behalf of my employer, and was like; what would this even mean for this account? I'm pretty sure I would take the offer had it been for my personal account.
Your technical notes really illustrate that there is a science to running good backups. Thanks so much!
> However, if your workload consists of one or more zfs datasets that store, and subsequently access, many small files, you can actually store the files themselves on the device - not just the metadata.
> For instance, if you have a pool named TANK and a filesystem named fs1 and you want to store all files smaller than, or equal to, 8K in the special vdev, you would run: ...
This sounds like the files themselves must be of size 8K or less in order to be allocated on the metadata vdev. But doesn't that include all individual file blocks of size 8K or less, not just entire files?
> For instance, if you have a pool named TANK and a filesystem named fs1 and you want to store all files smaller than, or equal to, 8K in the special vdev, you would run: ...
This sounds like the files themselves must be of size 8K or less in order to be allocated on the metadata vdev. But doesn't that include all individual file blocks of size 8K or less, not just entire files?
Now that borg 1.2 is out, are there any plans on supporting that?
Rsync.net is an awesome service. Ive been using their 100GB for 18 USD offer for the last few years to host my RESTIC backups. Have not had a single problem.
Not related to this technical note, but I was surprised to learn that the rsync service has scheduled downtime, like for hours at a time.
Is it possible to make your service highly available? Seems that is table stakes for other cloud storage services today.
Is it possible to make your service highly available? Seems that is table stakes for other cloud storage services today.
Not sure what you mean… generally speaking we keep the storage servers up continuously - as you’d expect.
I see several with 80-100 day uptimes right now …
I see several with 80-100 day uptimes right now …
Then how do you explain this email sent by yourself?
We are announcing an extended, off-peak maintenance window for your backup account.
Beginning Saturday March 5th at 9:00a PT (UTC -8) the server will go offline for approx. 6 hours as we perform data integrity checks and rebalancing.
...That's a single, one-off maintenance planned for Sat. March 5.
That is not a regular, every Saturday maintenance ... perhaps the word "Beginning" made that confusing ?
Let me check with engineering folks and see if they plan on repeating it - but even if they do, it is not a regular maintenance and you should expect your account to have long, uninterrupted uptimes ...
That is not a regular, every Saturday maintenance ... perhaps the word "Beginning" made that confusing ?
Let me check with engineering folks and see if they plan on repeating it - but even if they do, it is not a regular maintenance and you should expect your account to have long, uninterrupted uptimes ...
The state of userspace sandboxing across all operating systems I'm aware of is dismal. As a developer it should be trivial to spawn a process with a subset of the privileges of the parent process.
Linux is better than most in terms of options, but you have to understand seccomp, namespaces, cgroups, containers, capabilities, etc etc. It's too confusing. Why can't I just say "spawn this process, limit it to reading files in this directory, writing files in this directory, this socket, max 10GB disk space, max 5% CPU, max 10% memory"? I'd even settle for just being able to limit filesystem and network access.
Even if you do manage to make it work, since the APIs require SETUID executables you might actually be opening yourself up to a bigger risk.
Someone needs to pull a WireGuard and make a small, simple API for userspace sandboxing. And please somehow miraculously make it so it is likely to be implemented by Windows.