Systemd-coredump: CVE-2022-4415: local information leak(openwall.com)
openwall.com
Systemd-coredump: CVE-2022-4415: local information leak
https://www.openwall.com/lists/oss-security/2022/12/21/3
51 comments
Quite disappointing someone would enable that kernel option without fully considering the ramifications :/.
Do some distributions use systemd-coredump by default? I not just learned about systemd-coredump and I'm wondering if it could be actually useful for develoment purposes; apport doesn't seem to be.
Do some distributions use systemd-coredump by default? I not just learned about systemd-coredump and I'm wondering if it could be actually useful for develoment purposes; apport doesn't seem to be.
systemd-coredump is honestly incredibly useful. Any time an application randomly crashes in the background, you can easily pop it in a debugger to look through the backtrace without having to re-run it. Very handy for when the crashes are not easily reproducible, or you didn't expect something to fail and are not sure how to make it fail again.
It's also incredibly annoying, because it changes longstanding cross-platform common behavior regarding this exact functionality. "A coredump is written to the crashing process's current working directory, named 'core' or 'core.<pid>'" - that used to be true on Linux and all the BSDs.
Several setups of ours rely on that fact, and since we're developing software that is supposed to work on Linux and {Free,Open,Net}BSD, we were happy to be able to reuse a bunch of crash-capture logic.
systemd kneecapped us out of the blue, it took us some time to figure out where the coredumps were going. Now we disable systemd-coredump on all systems. Which works, except when something reenables it or we add a new platform and someone forgets to disable it. Sigh.
Yes, we can deal with it, by disabling it. But crash captures went from being reliable to "did systemd-coredump creep back in?" - a definite regression.
Several setups of ours rely on that fact, and since we're developing software that is supposed to work on Linux and {Free,Open,Net}BSD, we were happy to be able to reuse a bunch of crash-capture logic.
systemd kneecapped us out of the blue, it took us some time to figure out where the coredumps were going. Now we disable systemd-coredump on all systems. Which works, except when something reenables it or we add a new platform and someone forgets to disable it. Sigh.
Yes, we can deal with it, by disabling it. But crash captures went from being reliable to "did systemd-coredump creep back in?" - a definite regression.
It's not like it was ever a safe assumption to blindly dump core to a process' CWD. It was just a convenient option in a developer-centric UNIX mindset, in lieu of anything better.
Sometimes daemons run with a CWD changed somewhere rarely if ever written to at runtime, and presumed to primarily contain information intended for sharing. It's not hard to imagine a security-conscious daemon that's done both chroot() and chdir() into a path intended for publishing information out of, and not generally intended for the daemon to write into except by privileged users/publishers.
When that daemon spuriously dumps core into its CWD now you have potentially sensitive information sitting in the published tree, e.g. maybe the credentials of the privileged users are in that core file.
Sometimes daemons run with a CWD changed somewhere rarely if ever written to at runtime, and presumed to primarily contain information intended for sharing. It's not hard to imagine a security-conscious daemon that's done both chroot() and chdir() into a path intended for publishing information out of, and not generally intended for the daemon to write into except by privileged users/publishers.
When that daemon spuriously dumps core into its CWD now you have potentially sensitive information sitting in the published tree, e.g. maybe the credentials of the privileged users are in that core file.
I really don’t get why people are so quick to jump on “it changed so it’s bad.” systemd-coredump has to work in every possible case and is dealing with every unhappy path you can imagine. The kernel has a laundry list of situations where core files can’t be saved and systemd made the choice to set it up so some of the failure modes can’t happen.
If they just tried to write to cwd there would be a blogpost about “systemd loses coredumps if your app directory is read-only” which isn’t some theoretical problem, it’s containers.
There was no way to really win here.
If they just tried to write to cwd there would be a blogpost about “systemd loses coredumps if your app directory is read-only” which isn’t some theoretical problem, it’s containers.
There was no way to really win here.
> I really don’t get why people are so quick to jump on “it changed so it’s bad.”
That's not actually the issue here. Rather: "it changed so now it's inconsistent across platforms"
> There was no way to really win here.
There is:
- if you're changing a multi-platform standard, include the other platforms (e.g. create a coredumpctl tool for *BSD)
- do the changes slowly/incrementally in a backwards-compatible way (e.g. apply the new coredump locations first to read-only containers and service daemons running as systemd units)
That's not actually the issue here. Rather: "it changed so now it's inconsistent across platforms"
> There was no way to really win here.
There is:
- if you're changing a multi-platform standard, include the other platforms (e.g. create a coredumpctl tool for *BSD)
- do the changes slowly/incrementally in a backwards-compatible way (e.g. apply the new coredump locations first to read-only containers and service daemons running as systemd units)
- It's not a multi-platform standard more a happy accident of history.
- The BSD are irrelevant. Why hamper Linux development to cater for users who don't exist?
- The BSD are irrelevant. Why hamper Linux development to cater for users who don't exist?
Forgetting about Aix, HP-UX,....?
It isn't as if it worked the same everywhere anyway.
It isn't as if it worked the same everywhere anyway.
That default unix way is very annoying. I want all my core dumps collected in one place so I can easily monitor for new core dumps and easily find them. I do not want to have too search for them. The tool does not have to be systemd-coredump but it is good enough for the job.
Well, the used-to-be-true logic also resulted in scripts to find where there were core files, to remove them :).
At least in Debian and Ubuntu systemd-coredump is a separate package (and wasn't installed by default for me atleast), so I expect you can just remove it and it doesn't get automatically enabled in any way.
At least in Debian and Ubuntu systemd-coredump is a separate package (and wasn't installed by default for me atleast), so I expect you can just remove it and it doesn't get automatically enabled in any way.
> except when something reenables it
Have you tried masking it? (https://www.freedesktop.org/software/systemd/man/systemctl.h...)
Have you tried masking it? (https://www.freedesktop.org/software/systemd/man/systemctl.h...)
As far as I can tell all the magic you're describing is provided by the kernel's core dumper and not systemd. You can achieve the same effect by creating /tmp/cores and setting a meaningful pattern in /proc/sys/kerne/core_pattern.
> systemd-coredump is honestly incredibly useful. Any time an application randomly crashes in the background, you can easily pop it in a debugger to look through the backtrace without having to re-run it.
coredumps are of course incredibly useful, as they have been for decades. Couldn't live without them.
Is systemd adding any value here other than making things more complex and brittle?
coredumps are of course incredibly useful, as they have been for decades. Couldn't live without them.
Is systemd adding any value here other than making things more complex and brittle?
coredumping crashed applications (or the kernel) was not invented by systemd
systemd-coredump is vuln, systemd-oomd kills the wrong stuff, and you don't even learn that these programs exist (and are running on your system) until you find out from a CVE or a blog post detailing how to mitigate the problem[1].
[1]: https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdOomdNo...
And then, there are very few alternatives, because more and more software requires systemd (and even if most things still don't, so much stuff requires glibc so Alpine or OpenBSD aren't always very good as alternatives).
[1]: https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdOomdNo...
And then, there are very few alternatives, because more and more software requires systemd (and even if most things still don't, so much stuff requires glibc so Alpine or OpenBSD aren't always very good as alternatives).
You can turn the situation around: Pick some non-systemd OS, stick to it, and avoid software that does not run on it. You will end up with the old and boring tech, but it will be robust and run more reliable in the long term.
I've been running Alpine Linux in production for 4 years now.
I've been running Alpine Linux in production for 4 years now.
You will end up with the old and boring tech
Not always. I have sway and pipewire running smooth as can be on my systemd-free workstation. As well as recent releases of Linux, glibc, Mesa, and Firefox. All the essentials are there and up to date for me!
Not always. I have sway and pipewire running smooth as can be on my systemd-free workstation. As well as recent releases of Linux, glibc, Mesa, and Firefox. All the essentials are there and up to date for me!
That's true.
I am using Gentoo with no systemd and with the latest versions of all programs that I am interested in.
I am using Gentoo with no systemd and with the latest versions of all programs that I am interested in.
Unfortunately systemd & glibc (or even macOS) are usually the more pragmatic choice. I'd love to have the freedom to ignore them, but even outside of work, there is software (like games) that I want to run that doesn't play nice with anything more sane.
The glibc 2.26 breakages (including the ucontext thing) were what made me stick to musl/Alpine. This was in 2017, around the same time poettering did the "i don't consider this much of a problem"-things on github.
Note that musl libc in Alpine Linux has significantly smaller default thread stack size than glibc (128 kB vs 8 MB), therefore software developed and tested with glibc can easily crash with stack overflow.
Alpine's great. My entire homelab runs on it. Only complaint I have is its IPv6 support sucks. I have to hack together a bunch of scripts to get a reasonable network configuration.
I had no trouble with IPv6. For static config i add it to /etc/network/interfaces, for dynamic config i use dhcpcd without issues.
The systemd-oomd post discussed on HN:
https://news.ycombinator.com/item?id=33894469 (2 weeks ago, 134 comments)
https://news.ycombinator.com/item?id=33894469 (2 weeks ago, 134 comments)
I use systemd-coredump all the time on Fedora, it's pretty convenient. When a program crashes, run "coredumpctl gdb" to launch gdb on the latest coredump.
systemd-coredump is on my arch and i dont remember setting it up, anyway its rather cool, it even collects core dumps from podman
Fedora has it bundled by default. I've used it once, recently, to report a pipewire crash with bluetooth headphones.
The normal way I use machines these days is that anyone with login access has some ability to become root.
But anyone who is hosting shellboxes (including "bastion" hosts in ISO27001 environments): this is pretty serious.
But anyone who is hosting shellboxes (including "bastion" hosts in ISO27001 environments): this is pretty serious.
> The normal way I use machines these days is that anyone with login access has some ability to become root.
That is good, but you don't necessarily need login access to exploit this. You could exploit it via an RCE vulnerability in some other program, like the HTTP server or NTP client.
That is good, but you don't necessarily need login access to exploit this. You could exploit it via an RCE vulnerability in some other program, like the HTTP server or NTP client.
I'll bet you don't. The natural extension of that logic is that you don't bother with UID separation for running processes (since you don't believe it provides any protection), so for convenience all your software runs as root and all your ssh logins are directly to root, etc... No one does that, and I'll bet anything you don't either.
In fact the Unix uid/gid does provide real security value in practice. It is not absolute, and relative to other barriers commonly used (LSM, cgroup containers, VMs, firewalls, BPF syscall filtering, etc...) it's comparatively weak and porous. But it's not something to dismiss either.
The nature of "defense in depth" security analysis is that you have to embrace that depth and not throw bits out. UID separation has the distinct advantage of being dirt simple to understand and extremely cheap to deploy. Use it.
In fact the Unix uid/gid does provide real security value in practice. It is not absolute, and relative to other barriers commonly used (LSM, cgroup containers, VMs, firewalls, BPF syscall filtering, etc...) it's comparatively weak and porous. But it's not something to dismiss either.
The nature of "defense in depth" security analysis is that you have to embrace that depth and not throw bits out. UID separation has the distinct advantage of being dirt simple to understand and extremely cheap to deploy. Use it.
>> The normal way I use machines these days is that anyone with login access has some ability to become root.
> I'll bet you don't
I'll... bet they do? You realize that description most trivially matches "I give people sudo access", right?
So they say they do something and you promptly say that you don't think they do that, because "nobody" does a more extreme version of the same idea that they never suggested? That's a classic straw man.
> I'll bet you don't
I'll... bet they do? You realize that description most trivially matches "I give people sudo access", right?
So they say they do something and you promptly say that you don't think they do that, because "nobody" does a more extreme version of the same idea that they never suggested? That's a classic straw man.
That seems like a maximally uncharitable way of interpreting what I thought was a pretty uncontroversial point. The point was that when deploying software for use, best practice says "don't run it as root", and not "assume all software has root access". Do you really disagree? Because the grandparent poster definitely seems to, and I think that's pretty clearly wrong.
Like your maximally uncharitable interpretation of the original comment.
That still seems senselessly combative. I wasn't being mean, at least re-reading my comment it doesn't seem so to me. What's the deal here? Who did I offend, and how?
To repeat a third time: don't run stuff as root simply because you assume root elevation exploits exist. I stand by that, and don't really think anyone here disagrees.
To repeat a third time: don't run stuff as root simply because you assume root elevation exploits exist. I stand by that, and don't really think anyone here disagrees.
> I wasn't being mean, at least re-reading my comment it doesn't seem so to me.
I don't have a dog in this race, but you started your comment with "I bet you don't", which, to me, is unnecessarily combative and/or snarky. Right out of the gate I was predisposed to think ill of whatever you had to say.
I don't have a dog in this race, but you started your comment with "I bet you don't", which, to me, is unnecessarily combative and/or snarky. Right out of the gate I was predisposed to think ill of whatever you had to say.
FWIW: the grandparent has editted the comment to remove the framing I was responding to (something along the lines of "root access is inevitable"), which explains why everyone is being so outrageously touchy.
I guess I should have quoted. My apologies for seeming "snarky", but the conversation you read isn't the one I was engaged in.
But let me repeat for a fourth time that you shouldn't be running processes as root, whether or not you have worries about undiscovered vulnerabilities and especially regardless of whether you are "predisposed to think ill" of the people telling you that.
UID separation isn't a perfect model, but it's a good one, and you should use it.
I guess I should have quoted. My apologies for seeming "snarky", but the conversation you read isn't the one I was engaged in.
But let me repeat for a fourth time that you shouldn't be running processes as root, whether or not you have worries about undiscovered vulnerabilities and especially regardless of whether you are "predisposed to think ill" of the people telling you that.
UID separation isn't a perfect model, but it's a good one, and you should use it.
I don't think you were being mean, but you are being dense. This is the OP's statement:
> anyone with login access has some ability to become root
If you take that to mean "I run all my software as root", then you're being maximally uncharitable. And when people point out to you that your interpretation isn't the only one available (nor the most logical one), you accuse others of being senselessly combative?
> anyone with login access has some ability to become root
If you take that to mean "I run all my software as root", then you're being maximally uncharitable. And when people point out to you that your interpretation isn't the only one available (nor the most logical one), you accuse others of being senselessly combative?
Shouldn't a shell box just disable core dumps altogether?
Oh, that's nasty, using extended ACL have good chance someone will not notice it when looking at glance
Another day, another issue w/ core_pattern. Custom crash handlers have led to many local privilege escalations in the past. Don't use them!
When will RedHat learn that too many knobs and buttons, especially global settings, result in leaks and exploits? I don't care if the customers demand these.
I'm quite happy to enable/disable core dumps in the shell. Make Linux great again!
I'm quite happy to enable/disable core dumps in the shell. Make Linux great again!
How it started: [blog post from Lennart about the woeful state of Linux security and how systemd-newthingd fixes it]
How it's going: CVE-2022-4415
How it's going: CVE-2022-4415
Wow, this is pretty awful. Willfully defeating a security feature of the kernel.
Maybe it's time to switch to Devuan, which is a systemd-free fork of Debian that works really well.
I wonder if it would be generally better for su/PAM to mark this data as non-dumpable (MADV_DONTDUMP), just for defence in depth. You could also imagine coredumps being uploaded in bug reports, unintentionally revealing sensitive data.