How to Use User Mode Linux(christine.website)
christine.website
How to Use User Mode Linux
https://christine.website/blog/howto-usermode-linux-2019-07-07
62 comments
I’ve recently been using proot [1] as a way to have a fully controlled rootfs for jobs on a HPC cluster/grid service where I am a very regular unprivileged user.
Is it theoretically possible I could run an entire linux kernel instead with this? Is there any setup on the host system that is required which needs administrative privileges?
[1] https://proot-me.github.io/
Is it theoretically possible I could run an entire linux kernel instead with this? Is there any setup on the host system that is required which needs administrative privileges?
[1] https://proot-me.github.io/
I do not have much info on UML, but you may be interested in https://rootlesscontaine.rs/
Unless you're interested in OCI (?) containers, you're probably better off with straight proot, or charlecloud if you're allowed user namespaces.
https://github.com/hpc/charliecloud
Yes, Proot is useful, but limited. It will be somewhat slow in some cases and not do the right thing in others, like interpret shebangs as expected, as namespaces will.
In most cases you can run HPC stuff by installing packages under proot and setting paths in the environment appropriately to use the resulting root. (I don't know if that's the case here, or if it means running under proot.)
No, you certainly can't run your own kernel that way.
The HPC usage scenario is exactly the reason why I played with UML a few years ago. Many HPC clusters nowadays completely lock down the network and only allow in/outbound SSH connections to whitelisted IP networks. On the other hand, users are only given an unpriviledged account. Typically, users set up proxies on a per-application level (i.e. for git, svn, to sync their codes with the outer world). I found this very frustrating and wanted to use any usual Linux command line tools. My idea was a UML based virtual machine (ie. running a Linux distribution within UML) and have it connected to the outer world by tunneling a single slirp and VDE connection. VDE is quite amazing because it marries unix pipes with ethernet-level networking. It even allows to run ethernet switches as processes and all that. My codes based on https://github.com/jpetazzo/sekexe
Turned out: It basically worked, but it was a bit painful to setup and to manage.
Turned out: It basically worked, but it was a bit painful to setup and to manage.
I'd have thought it would be relatively unusual not to be able to use HTTP into a cluster login node, but I'm puzzled why you can't "sync code" with ssh otherwise. If you're prevented from doing things you need to work, it sounds like an institutional politics problem; subverting controls is likely to cause more friction as well as presumably wasting resources due to inefficiency.
That is very helpful to know it has been done, thank you. I need to try UML out it seems. The SMP issue and any overhead are other big questions.
Udocker also use Proot: https://github.com/indigo-dc/udocker
I used it on a server where I was not root to be able to launch docker containers.
I used it on a server where I was not root to be able to launch docker containers.
It was probably the computer security course, assuming we took many of the same courses.
It had students exploit a setuid binary to get root for an assignment. Done on UML so we could run the thing as our regular users.
It had students exploit a setuid binary to get root for an assignment. Done on UML so we could run the thing as our regular users.
Yes! That was indeed it. You motivated me to go look into my archives - I took that course in 2010 and almost entirely forgot I had :)
Many many years ago, when I was working at a large 3-letter-starting-with-S Unix company (no, not the graphics guys, nor the suing-Linux-for-Microsoft guys), we were planning a Linux compatibility layer... At the time, there were a couple of options out there - lxrun and uml. Anyone remember lxrun? I tested them both and did a little write-up, basically saying that neither of them ran reliably or fast enough to be worth adopting
What we decided was to go with option 3 - we implemented a Linux system call interface within our own kernel to give us 'native' support. This, of course, is exactly what the first version of WSL did, many years later. Plus ça change...
What we decided was to go with option 3 - we implemented a Linux system call interface within our own kernel to give us 'native' support. This, of course, is exactly what the first version of WSL did, many years later. Plus ça change...
Didn't FreeBSD do this before Solaris?
> What we decided was to go with option 3 - we implemented a Linux system call interface within our own kernel to give us 'native' support. This, of course, is exactly what the first version of WSL did, many years later. Plus ça change...
Joyent as well for SmartOS.
Joyent as well for SmartOS.
I wonder if that was before SunOS binaries ran on SPARC Linux; I must have been doing that at least 15 years ago.
>at a large 3-letter-starting-with-S Unix company (no, not the graphics guys, nor the suing-Linux-for-Microsoft guys)
Is it Sun? Why use more than 20 words to (probably) mean "for Sun"?
Is it Sun? Why use more than 20 words to (probably) mean "for Sun"?
[deleted]
Presumably they think this is a defense that will hold up in court as plausible deniability if they are ever sued for defamation relating to this comment.
You guys are weird. It was just a bit of wordplay :D
I didn't really have an issue with what you wrote. It made me realize there are several three-letter, starts-with-S, Unix companies.
No fun allowed here on HN
I hope you don't code the way you talk ;)
Seriously though, it's bad writing. Not a crime as bad as starting an off-topic subthread, but still...
Seriously though, it's bad writing. Not a crime as bad as starting an off-topic subthread, but still...
Sun doesn't exist anymore though. Is getting sued even a possibility?
In the hypothetical scenario that this commenter is implying that Solaris contains (or contained) a load of IP which is non-clean-roomed Linux copyright (is that the implication??), then the current owners of Solaris would be the ones getting in trouble with - the FSF? That would be Oracle.
https://www.oracle.com/solaris/solaris11/
And it would be Oracle chasing the commenter for an NDA breach, I suppose.
https://www.oracle.com/solaris/solaris11/
And it would be Oracle chasing the commenter for an NDA breach, I suppose.
I have often thought back to how great UML was. I had filed it under "distant history" when I thought about it last week...
Back in 2003 or 2004, I threw together a "virtual hosting" platform using UML. It had a web form accepting payment, and a few minutes later you'd have a fully running virtual machine. I put this together in a couple weeks leading up to PyCon, where we launched it. But we didn't really have the business savvy to take it anywhere, we never had more than a few hundred VMs.
We eventually moved to other technologies: VMWare, Xen, Proxmox (KVM). But UML was my favorite. (Xen was my least favorite :-)
Back in 2003 or 2004, I threw together a "virtual hosting" platform using UML. It had a web form accepting payment, and a few minutes later you'd have a fully running virtual machine. I put this together in a couple weeks leading up to PyCon, where we launched it. But we didn't really have the business savvy to take it anywhere, we never had more than a few hundred VMs.
We eventually moved to other technologies: VMWare, Xen, Proxmox (KVM). But UML was my favorite. (Xen was my least favorite :-)
https://lwn.net/2001/features/OLS/pdf/pdf/uml.pdf
In 2001 this mentions SMP support is possible. First thing tomorrow I need to check if it was ever implemented. Perhaps it’s late and I’m tired but somehow this seems to be much better than containers for a number of problems, or maybe the performance hit is too much.
In 2001 this mentions SMP support is possible. First thing tomorrow I need to check if it was ever implemented. Perhaps it’s late and I’m tired but somehow this seems to be much better than containers for a number of problems, or maybe the performance hit is too much.
Off topic, but here it goes: I really like the styling of this website. I shouldn't like it, because I don't like non-proportional fonts being used for prose, but it works great in this case.
Also loads super fast, which is an exception these days.
Also loads super fast, which is an exception these days.
I like it as well. It looks like one of those generated static sites from hugo or haunt. I've been seeing it more often lately and I think it's a good trend.
My site isn't actually a static site. The code is in https://github.com/Xe/site
weberlars/diuid https://github.com/weber-software/diuid runs a UML in a docker container, which then runs a Docker daemon of a different version inside. Clever.
It would be nice to combine UML with an encrypted root filesystem so a nosy admin on a VPS will have a harder time seeing what you are doing on your VPS.
Still I wonder what the performance will be like. Is the entire UML process running single threaded on a single CPU core?
Each process in the guest has its own host thread, but yeah the way I have it set up there's only one logical "CPU core" being exposed to the guest.
Great work!
I am trying to run this on the latest Ubuntu but getting the below error:
root@kit:~/uml/furry-happiness# docker run --rm -it xena/docker-uml [WARN tini (7)] Tini is not running as PID 1 and isn't registered as a child subreaper. Zombie processes will not be re-parented to Tini, so zombie reaping won't work. To fix the problem, use the -s option or set the environment variable TINI_SUBREAPER to register Tini as a child subreaper, or run Tini as PID 1. Core dump limits : soft - NONE hard - NONE Checking that ptrace can change system call numbers...ptrace: Operation not permitted check_ptrace : expected SIGSTOP, got status = 9
I am trying to run this on the latest Ubuntu but getting the below error:
root@kit:~/uml/furry-happiness# docker run --rm -it xena/docker-uml [WARN tini (7)] Tini is not running as PID 1 and isn't registered as a child subreaper. Zombie processes will not be re-parented to Tini, so zombie reaping won't work. To fix the problem, use the -s option or set the environment variable TINI_SUBREAPER to register Tini as a child subreaper, or run Tini as PID 1. Core dump limits : soft - NONE hard - NONE Checking that ptrace can change system call numbers...ptrace: Operation not permitted check_ptrace : expected SIGSTOP, got status = 9
When I tried User Mode Linux I found that there was very little in the way of ready-made, up to date tooling.
Very little "just worked".
I think there's huge potential there, however, as an architectural building block, just like containers are.
Very little "just worked".
I think there's huge potential there, however, as an architectural building block, just like containers are.
It's the kind of stuff that drives you insane a bit. You have to hack together things from software last released in 2006. You have to dig through kernel source code to really understand what is going on in the user mode Linux drivers.
The article states that processes in User Mode Linux are completely isolated from the host, and cannot be seen.
Could someone clarify this for me - there must be some way of pulling data from UML back into the host system?
Could someone clarify this for me - there must be some way of pulling data from UML back into the host system?
Actually, very much on the contrary, the processes running inside UML are completely visible as separate threads within the UML process, and you can go poking their `proc` files at-will. Also, UML uses a read/write disk image as the rootfs, so any writes to the filesystem will be persisted in the disk image.
> processes running inside UML are completely visible as separate threads within the UML process
Does this mean the UML system can utilize many cores well and SMP was eventually implied?
Does this mean the UML system can utilize many cores well and SMP was eventually implied?
I thought as much! Thanks.
I had no idea this was possible. Excellent article - thank you for opening my eyes to yet another insanely cool aspect of Linux.
The next logical thing to do for more fun is to run User Mode Linux under the Windows Subsystem for Linux.
After that, it would make sense to try WINE.
After that, it would make sense to try WINE.
There used to be a version of uml that ran on windows. It was very like the modern windows subsystem for Linux. http://colinux.org/
A Windows app running on WINE inside UML inside WSL of a Win10 which is itself a VM running on e.g. macOS would be funny to see.
Does WSL support ptrace now? UML can't run without it.
Especially if the app is Electron based.
I remember that at some point there was wine port to Windows, so you could run windows while running windows...
Very nice. Now I'm tempted to try and run X11 and firefox in UML somehow.
Brings back memories. My first VPS back in 2003 used UML and it worked great. You could also run it inside Windows as well. Could even get GUI apps running by running Xming on your Windows desktop using CoLinux (http://www.colinux.org/?section=screenshots).
Colinux was freaking great, I could run an OpenSSH server on port 22 and sometimes ending up on a Linux box, sometimes on a Windows box. Weird.
How does this compare to other methods of isolation / running untrusted code? What are the pros and cons?
https://www.linode.com/ used to run off UML, though they switched quite a number of years ago. It might be worth hunting out their documentation and/or forum archives from around the time of the transition. I recall a lot of detailed discussion (useful discussion too: not just people bashing the technology they don't prefer).
I used to run it myself, way back when. For isolation, it is going to be better than a container or jail, but not as good as a full VM, though I was using it for testing personal stuff so wasn't really concerned with isolation from a security point of view. One key con is that it can perform pretty badly in some respects, particularly for loads that ned to talk to the kernel much at all, including processes that perform much IO. It "felt" less cumbersome than full VMs (IIRC I was playing with a mix of VirtualBox and VMWare' Player & Server for that sort of thing at the time), in terms of my automation and to some extent in terms of host resource use (though I didn't test that with any scientific rigour, and if I did any good tests back then the results would not be relevant today anyway as they were pre VT-x and other CPU support which UML can't use but a modern hypervisor can and will).
I used to run it myself, way back when. For isolation, it is going to be better than a container or jail, but not as good as a full VM, though I was using it for testing personal stuff so wasn't really concerned with isolation from a security point of view. One key con is that it can perform pretty badly in some respects, particularly for loads that ned to talk to the kernel much at all, including processes that perform much IO. It "felt" less cumbersome than full VMs (IIRC I was playing with a mix of VirtualBox and VMWare' Player & Server for that sort of thing at the time), in terms of my automation and to some extent in terms of host resource use (though I didn't test that with any scientific rigour, and if I did any good tests back then the results would not be relevant today anyway as they were pre VT-x and other CPU support which UML can't use but a modern hypervisor can and will).
I worked for a hosting company for 10ish years, and they started out their hosting by providing people with UML-based virtual machines for £10/month.
In terms of isolation it is "pretty good". The guest has their own kernel, and you can run it as an ordinary user. Add a chroot jail and you can have 50-150 different UML instances running on a single host, each under their own UID. They can't talk to each other, and if they do manage to exploit the kernel they're only gonna get the access to the host which you restrict with chroot, and standard things.
It's less of a risk to run isolated things than this than have lots of containers share the same kernel. An escape there means container-A can view/poke container-B AND the host.
Of course there is overhead, and most people moved on from UML to Xen/KVM when they became available - both have better performance thanks to virtio, and both have more flexibility for networking, block devices & etc.
In terms of isolation it is "pretty good". The guest has their own kernel, and you can run it as an ordinary user. Add a chroot jail and you can have 50-150 different UML instances running on a single host, each under their own UID. They can't talk to each other, and if they do manage to exploit the kernel they're only gonna get the access to the host which you restrict with chroot, and standard things.
It's less of a risk to run isolated things than this than have lots of containers share the same kernel. An escape there means container-A can view/poke container-B AND the host.
Of course there is overhead, and most people moved on from UML to Xen/KVM when they became available - both have better performance thanks to virtio, and both have more flexibility for networking, block devices & etc.
I've used for developing on a community mesh network - you can boot OpenWrt with a fake wifi driver module and you can plug in in networking as you will and run multiple processes. It's far simpler to configure and does not need compatibily fixes for running in containers.
I couldn't find any documentation about the intended security model of UML, but from what I could tell, it's not resistant against malicious code running as root in the guest.
It’s got great documentation. See the kernel.org documentation here: https://www.kernel.org/doc/Documentation/virtual/uml/UserMod... and a write up from the creator Jeff Dike (which includes a lot of technical detail on the inner workings) here: https://lwn.net/2001/features/OLS/pdf/pdf/uml.pdf (from 2001!)
slirp is the only _purely usermode_ way to do networking, but UML also supports TUN/TAP for something a little more sane. UML does a bunch of pretty crazy things in order to work, including a dedicated thread whose job it is to ptrace every other process for syscall and signal emulation (and, as a bonus, it even ptraces external debuggers to make them think they’ve successfully ptraced a process that’s already being traced!), and a full emulation of physical/virtual memory translation with paging (done by giving every virtual process under UML its own virtual address space).
All in all, a super impressive project, and definitely one worth trying out.