Escaping privileged containers for fun(pwning.systems)
pwning.systems
Escaping privileged containers for fun
https://pwning.systems/posts/escaping-containers-for-fun/
41 comments
> Podman rootless
This is the way. Docker rootless and podman rootless. A lot less attack surface than running containers as root.
I haven't tried a dedicated user for this though. I'm sure that mounting volumes would get messed up.
This is the way. Docker rootless and podman rootless. A lot less attack surface than running containers as root.
I haven't tried a dedicated user for this though. I'm sure that mounting volumes would get messed up.
I run services like Plex or DNS adblocker in this manner. Each service gets its own user to run rootless under and inside the container the app doesn't run as root.
Security is layers.
Security is layers.
Thanks very good suggestion! I'll take a look if Hugo has a feature for that otherwise I 'll edit the templates :D
Containers are not a security mechanism.
Containers are not a security mechanism.
Containers are not a security mechanism.
...
Every engineer making a foray into Docker or any similar tech for the first time should be made to write this line x1000 before proceeding.
Containers are not a security mechanism.
Containers are not a security mechanism.
...
Every engineer making a foray into Docker or any similar tech for the first time should be made to write this line x1000 before proceeding.
Containers are a security mechanism. We can say that now. For a long time they weren't, and then it wasn't really clear, but at this point they are. How good are they? Questionable - there are footguns (running privileged) and they rely on the Linux kernel (garbage), but they are a barrier.
If you put a process into a container, even a fairly default config'd container, it will require an additional vulnerability for an attacker to escape that container. That makes it a security boundary, period. There is no universal "ask to leave the container" by default.
This post is attacking a non-default configuration ie: you have to explicitly pass in "--privileged". Don't do that (the post says this repeatedly).
The fact that people are using docker and we're basically getting a half decent sandbox for free, everywhere, is an incredible security win.
I wouldn't rely on them for multi-tenant workloads, but as a nice way to restrict attackers given RCE, yeah, sure.
If you put a process into a container, even a fairly default config'd container, it will require an additional vulnerability for an attacker to escape that container. That makes it a security boundary, period. There is no universal "ask to leave the container" by default.
This post is attacking a non-default configuration ie: you have to explicitly pass in "--privileged". Don't do that (the post says this repeatedly).
The fact that people are using docker and we're basically getting a half decent sandbox for free, everywhere, is an incredible security win.
I wouldn't rely on them for multi-tenant workloads, but as a nice way to restrict attackers given RCE, yeah, sure.
Containers increase the effort for the attacker to put into escaping that container — provided the developers actually run code that doesn't make that easy right of the bat. I have seen to much code where you can really feel the thought of "let's just use docker, then we don't have to think about complicated linux things like priviledges and permissions".
Using containers as an extra layer of security is perfectly fine. Using it as an excuse to not do any other security measures is not.
Using containers as an extra layer of security is perfectly fine. Using it as an excuse to not do any other security measures is not.
> "let's just use docker, then we don't have to think about complicated linux things like priviledges and permissions"
Honestly, that's fine. What more do you want? For developers to roll their own SELinux and Apparmor profiles? For them to manually implement seccomp filtering? No one does that.
The reality is that no one is choosing between "containers or some other approach", they're choosing between "containers or nothing".
Honestly, that's fine. What more do you want? For developers to roll their own SELinux and Apparmor profiles? For them to manually implement seccomp filtering? No one does that.
The reality is that no one is choosing between "containers or some other approach", they're choosing between "containers or nothing".
[deleted]
Hopefully they aren't choosing containers between "containers + insecure app" or "no containers + secure app", thinking containers will secure the app.
Who does that? I have specific apps to run, the choice is to deploy them bare or in containers.
As a user yes, as a developer many people who would have to think long and hard about when to use which linux user with which priviledges, when to drop them outside of an container will just use root for everything within the container out of an false sense of security.
Software security always works in layers and if your only layer is the container, good luck with that.
Software security always works in layers and if your only layer is the container, good luck with that.
Unless you have a secure mechanism to forbid containers from running with `--privileged` (which does not come with container runtimes that I'm aware of), or unless your container runtime launches a VM to run the containers in, then the conception of containers being a security mechanism is a pretty thin one, indeed.
Containers can be part of a secure application runtime environment, but much like sugary cereal can be a part of a balanced breakfast, more is required.
Containers can be part of a secure application runtime environment, but much like sugary cereal can be a part of a balanced breakfast, more is required.
Attackers have no control over the --privileged flag, so what you're basically asking for is a lint that greps for "--privileged" and fails CI if it finds it.
Some things may make this harder, like Kubernetes presumably, but idk it seems relatively straightforward and also not exactly the fault of containers.
Some things may make this harder, like Kubernetes presumably, but idk it seems relatively straightforward and also not exactly the fault of containers.
> Attackers have no control over the --privileged flag
Maybe, maybe not. It really depends on the environment. If one's orchestration system allows an attacker to supply arbitrary arguments to a runtime, then it's quite possible that a container could be run in privileged mode without explicit authorization.
Good CI hygiene is part of the solution, but again, a comprehensive approach is needed.
The thing that makes `--privileged` an attractive nuisance is that the container runtime usually runs as root, and so it's literally a mechanism for trivial privilege escalation if the agent trusts the request being given to it, and container agents are rarely configured to require authentication of any sort. (The typical default is to run without TCP enabled, and where the Docker socket has some sort of group membership required.) Compare with the traditional UNIX privilege model, in which non-root users don't get privileges, and escalation happens through an agent like sudo(1) which typically requires re-authentication unless you're explicitly adding `NOPASSWD` to the sudoers(5) file.
Maybe, maybe not. It really depends on the environment. If one's orchestration system allows an attacker to supply arbitrary arguments to a runtime, then it's quite possible that a container could be run in privileged mode without explicit authorization.
Good CI hygiene is part of the solution, but again, a comprehensive approach is needed.
The thing that makes `--privileged` an attractive nuisance is that the container runtime usually runs as root, and so it's literally a mechanism for trivial privilege escalation if the agent trusts the request being given to it, and container agents are rarely configured to require authentication of any sort. (The typical default is to run without TCP enabled, and where the Docker socket has some sort of group membership required.) Compare with the traditional UNIX privilege model, in which non-root users don't get privileges, and escalation happens through an agent like sudo(1) which typically requires re-authentication unless you're explicitly adding `NOPASSWD` to the sudoers(5) file.
Oh, you're talking about an attacker who's outside of the container? That's a completely different threat model.
Why is the Linux kernel garbage? I thought Linux had decent security chops?
Written in C, huge codebase, decades of alienating security researchers, etc.
Is this "Linux is garbage" opinion shared broadly by security researchers? I did some Googling around about opinions, and I didn't find anything that suggested that this view is widely shared by security researchers (I found some claims that security researchers held Linux in high regard, but those claims themselves didn't seem particularly authoritative). Also, what is "alienating"? (I don't have much of a view into interactions between kernel maintainers and security researchers) As for "written in C", what production-grade kernels aren't written in C?
> Is this "Linux is garbage" opinion shared broadly by security researchers?
Everyone I know would agree, yeah. I think the infosec world is probably more or less aligned on this, as much as anything, although that's a low bar and there is no single infosec community.
> what is "alienating"
Decades of Linus and Greg insulting or attacking security researchers in various ways. It's become pretty antagonistic on both sides I think.
> As for "written in C", what production-grade kernels aren't written in C?
None afaik. It just all jumbles together to paint the 'garbage' picture.
Everyone I know would agree, yeah. I think the infosec world is probably more or less aligned on this, as much as anything, although that's a low bar and there is no single infosec community.
> what is "alienating"
Decades of Linus and Greg insulting or attacking security researchers in various ways. It's become pretty antagonistic on both sides I think.
> As for "written in C", what production-grade kernels aren't written in C?
None afaik. It just all jumbles together to paint the 'garbage' picture.
Fair enough. While I understand that there may not be any body which perfectly represents the infosec community, I would be curious if there are any surveys or even statements by somewhat authoritative bodies (something like NIST?).
No doubt that Linus is antagonistic. I haven't heard anything that suggests security researchers as a collective have taken it personally (I would expect they, like most people, understand that this is a personality quirk).
> None afaik. It just all jumbles together to paint the 'garbage' picture.
It seems pretty meaningless and unhelpful to categorize all C software as "garbage". Programming language isn't the only factor in the number or severity of vulnerabilities, and indeed you could have a Rust (or whatever) kernel which has more numerous and/or severe vulnerabilities than the Linux kernel. To deserve the "garbage" designation, I would expect some supporting evidence that the Linux kernel development process lacks rigor.
Moreover, "secure" is relative, so if you're going to call the Linux kernel "garbage" then it seems like you should have some kernel which is not "garbage" (and personally my minimum qualification for a "non-garbage" kernel is "production grade" i.e., not a research project).
No doubt that Linus is antagonistic. I haven't heard anything that suggests security researchers as a collective have taken it personally (I would expect they, like most people, understand that this is a personality quirk).
> None afaik. It just all jumbles together to paint the 'garbage' picture.
It seems pretty meaningless and unhelpful to categorize all C software as "garbage". Programming language isn't the only factor in the number or severity of vulnerabilities, and indeed you could have a Rust (or whatever) kernel which has more numerous and/or severe vulnerabilities than the Linux kernel. To deserve the "garbage" designation, I would expect some supporting evidence that the Linux kernel development process lacks rigor.
Moreover, "secure" is relative, so if you're going to call the Linux kernel "garbage" then it seems like you should have some kernel which is not "garbage" (and personally my minimum qualification for a "non-garbage" kernel is "production grade" i.e., not a research project).
> I would be curious if there are any surveys or even statements by somewhat authoritative bodies (something like NIST?).
Hmmm, I'm not sure if anything like that exists, certainly nothing from a standards body. The closest might be recommendations to not rely on things like Linux containers, or to be wary of kernel vulnerabilities. But usually official bodies don't make comments like "this software is bad".
> I haven't heard anything that suggests security researchers as a collective have taken it personally
I assure you that they do, I know many people who have been active in Linux kernel security for decades.
> It seems pretty meaningless and unhelpful to categorize all C software as "garbage".
Oh yeah, what I meant is that all of these things are factors. I'm sort of hand-wavingly justifying the "garbage" description. There's a lot of problems that are all interwoven. C is one of those problems.
> To deserve the "garbage" designation, I would expect some supporting evidence that the Linux kernel development process lacks rigor.
That's fair. I would probably point to the lack of testing/ coverage, the lack of code review, the huge LOC:maintainer ratio.
> Moreover, "secure" is relative, so if you're going to call the Linux kernel "garbage" then it seems like you should have some kernel which is not "garbage"
I'd have to disagree here. I don't believe I need to have an example of something better in order to point out that there's something bad.
Hmmm, I'm not sure if anything like that exists, certainly nothing from a standards body. The closest might be recommendations to not rely on things like Linux containers, or to be wary of kernel vulnerabilities. But usually official bodies don't make comments like "this software is bad".
> I haven't heard anything that suggests security researchers as a collective have taken it personally
I assure you that they do, I know many people who have been active in Linux kernel security for decades.
> It seems pretty meaningless and unhelpful to categorize all C software as "garbage".
Oh yeah, what I meant is that all of these things are factors. I'm sort of hand-wavingly justifying the "garbage" description. There's a lot of problems that are all interwoven. C is one of those problems.
> To deserve the "garbage" designation, I would expect some supporting evidence that the Linux kernel development process lacks rigor.
That's fair. I would probably point to the lack of testing/ coverage, the lack of code review, the huge LOC:maintainer ratio.
> Moreover, "secure" is relative, so if you're going to call the Linux kernel "garbage" then it seems like you should have some kernel which is not "garbage"
I'd have to disagree here. I don't believe I need to have an example of something better in order to point out that there's something bad.
Containers provide a level of isolation relative to standard Linux processes. Docker style containers were not designed specifically as a security sandbox (IMO) but instead re-use a set of Linux primitives to provide a level of isolation.
One of the good parts about this is that as they use Linux primitives, the level of isolation can be increased relatively easily. Things like --cap-drop=ALL and --no-new-privileges, combined with ensuring the contained process(es) don't run as root can make things better.
It's also important to ensure that the isolation put in place isn't removed. The major case where this happens is that Kubernetes disables Docker's seccomp filter which has left containers running in k8s clusters vulnerable to issues that stock Docker containers were not (CVE-2022-0185 for a recent example).
If I was running truly untrusted processes from unknown people, I probably would not use Docker containers, instead I'd use gVisor or Firecracker.
One of the good parts about this is that as they use Linux primitives, the level of isolation can be increased relatively easily. Things like --cap-drop=ALL and --no-new-privileges, combined with ensuring the contained process(es) don't run as root can make things better.
It's also important to ensure that the isolation put in place isn't removed. The major case where this happens is that Kubernetes disables Docker's seccomp filter which has left containers running in k8s clusters vulnerable to issues that stock Docker containers were not (CVE-2022-0185 for a recent example).
If I was running truly untrusted processes from unknown people, I probably would not use Docker containers, instead I'd use gVisor or Firecracker.
I wonder how hard is it to escape a unprivileged container as a non-root user in a container. It seems that many CTF organizers already host their RCE challenge this way.
FWIW, most of the untrusted code execution "platforms" these days use Docker with some hardening (dropped capabilities, syscall filtering, readonly fs etc).
Eg. Repl.it and Rust Playground.
These days, Docker doesn't care about the underlying container runtime. You can use Docker to run containers in VMs with Firecracker, gVisor, Kata, etc. Security and multitenancy are their raison d'être.
You might have been right in 2014. In 2022, containers are most definitely a security mechanism.
You might have been right in 2014. In 2022, containers are most definitely a security mechanism.
How about virtual machines?
There are _some_ containers that are security mechanism.
Docker is definitely not one of them. But LXD for example uses unprivileged containers by default and can be hardened with additional software for syscall filtering
Docker is definitely not one of them. But LXD for example uses unprivileged containers by default and can be hardened with additional software for syscall filtering
> uses unprivileged containers by default and can be hardened with additional software for syscall filtering
You also described docker.
I agree that docker's main focus isn't to be a security oriented sandbox, but perhaps you need to find a different wording of what exactly makes LXD (or others) so different.
You also described docker.
I agree that docker's main focus isn't to be a security oriented sandbox, but perhaps you need to find a different wording of what exactly makes LXD (or others) so different.
> You also described docker.
Can docker do secure multiuser?
https://m.youtube.com/watch?v=6O0q3rSWr8A
Can docker do secure multiuser?
https://m.youtube.com/watch?v=6O0q3rSWr8A
If LXD also uses cgroups and Linux namespaces it seems like the differences would be pretty small? Docker also allows custom syscall filtering, unprivileged containers, etc.
I think the reality is just that containers will always be imperfect, and you should understand that risk when building systems.
I think the reality is just that containers will always be imperfect, and you should understand that risk when building systems.
I was under the understanding that docker didn't use user namespaces by default -- so "root" in the container is the same user as "root" outside the container -- but that LXD does. Has that changed?
It doesn't but TBH user namespaces have a mixed record when it comes to security. For example CVE-2022-0185 wasn't exploitable from a standard container unless user namespaces were available allowing additional capabilities to be gained, inside that user namespace (https://blog.aquasec.com/cve-2022-0185-linux-kernel-containe... has more details.
My biggest concern is how docker's default behavior is to run roughshod over traditional unix security. Usually, adding a user to a group grants them some additional files or devices. The expectation is that adding a user to the "docker" group would allow that user to interact with the docker daemon, and that the docker daemon would check user-level privileges internally. Instead, adding a user to the "docker" group allows a user to have passwordless root-level privileges to access or modify any file on the system, regardless of file ownership (e.g. `docker run --volume /:/mnt ubuntu cat /mnt/etc/shadow`).
There's some documentation on how only trusted users should be given access to the docker daemon, but I do not consider it sufficient. At no point in the installation or getting started documents [0,1] is it mentioned that it drives a gaping hole through existing security measures. Instead, the mention was on the security page, three sections down in a discussion about the attack surface [2]. This is the sort of issue that should be in big bold blinking letters at the top of every tutorial, that access to docker is
I know that rootless docker exists and improves this situation, but it is neither the default behavior, nor the introductory example in official documentation. I know that docker's primary role is dependency management, and that it only considers escalation coming from within the container as security issues. But there's a world of difference between "I'm not a locksmith, so I don't sell locks." and "I'm not a locksmith, so I break in your windows." One is ambivalent to existing security measures, while the other, like Docker, actively subverts them.
[0] https://docs.docker.com/get-started/
[1] https://docs.docker.com/engine/install/
[2] https://docs.docker.com/engine/security/#docker-daemon-attac...
Edit: When I was initially researching this, because I was absolutely floored that this security flaw was even a possibility in something as widely used as docker, I came across this reddit post [3], which explains the issue quite well. It is 4 years old and predates rootless docker, but is still accurate to the default and most widely used behavior.
[3] https://www.reddit.com/r/docker/comments/7y2yp2/why_is_singu...
There's some documentation on how only trusted users should be given access to the docker daemon, but I do not consider it sufficient. At no point in the installation or getting started documents [0,1] is it mentioned that it drives a gaping hole through existing security measures. Instead, the mention was on the security page, three sections down in a discussion about the attack surface [2]. This is the sort of issue that should be in big bold blinking letters at the top of every tutorial, that access to docker is
I know that rootless docker exists and improves this situation, but it is neither the default behavior, nor the introductory example in official documentation. I know that docker's primary role is dependency management, and that it only considers escalation coming from within the container as security issues. But there's a world of difference between "I'm not a locksmith, so I don't sell locks." and "I'm not a locksmith, so I break in your windows." One is ambivalent to existing security measures, while the other, like Docker, actively subverts them.
[0] https://docs.docker.com/get-started/
[1] https://docs.docker.com/engine/install/
[2] https://docs.docker.com/engine/security/#docker-daemon-attac...
Edit: When I was initially researching this, because I was absolutely floored that this security flaw was even a possibility in something as widely used as docker, I came across this reddit post [3], which explains the issue quite well. It is 4 years old and predates rootless docker, but is still accurate to the default and most widely used behavior.
[3] https://www.reddit.com/r/docker/comments/7y2yp2/why_is_singu...
One-liner to trigger core dumping:
$ sh -c 'kill -ABRT $$'
Aborted (core dumped)$$ would give the pid of the inner sh command right? Because the single quote would disable interpolation in the outer shell. So why would this work?
It works by killing the inner shell. The outer shell remains alive, so that it can print the "core dumped" message.
If you don't mind losing your current shell and don't care about the message, you can use just:
If you don't mind losing your current shell and don't care about the message, you can use just:
$ kill -ABRT $$[deleted]
Traditional advice was you analyze potentially malicious binaries in a VM. I guess its more prudent now to do it on air-gapped systems?
Well, at the very least don't do it in a privileged docker container.
Fun read! Maybe put an RSS button somewhere so it's easier to subscribe to?