I've now disabled systemd-oomd on my Fedora desktops(utcc.utoronto.ca)
utcc.utoronto.ca
I've now disabled systemd-oomd on my Fedora desktops
https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdOomdNowDisabled
126 comments
When are we going to stop replacing battle-tested components of our important computers with new and broken systemd functionality?
> When are we going to stop replacing battle-tested components of our important computers with new and broken systemd functionality?
We aren't. Our time's up. The new generation is on now, and we didn't teach them hardly anything we learned because we were so invested in making fun of them for enjoying ubuntu.
Best way we can move past it is to take the next curmudgeonly step and move to OpenBSD and Plan9, as they're the only things sufficiently dense and opaque to be as yet out of reach of the current generation.
We aren't. Our time's up. The new generation is on now, and we didn't teach them hardly anything we learned because we were so invested in making fun of them for enjoying ubuntu.
Best way we can move past it is to take the next curmudgeonly step and move to OpenBSD and Plan9, as they're the only things sufficiently dense and opaque to be as yet out of reach of the current generation.
I’m a certified curmudgeon then. If a workload can run on OpenBSD then it does when I’m working. If I’m ever speaking the programmers language and cursing a computer then it is rarely an OpenBSD box. They just work.
[deleted]
Seems to me like in this case we've just gone from a broken system to another broken system
Sure, but the new system is even more broken than the old one. We're moving in the wrong direction.
what? reading your comments here you seem to have a deep and abiding irrational hatred for all things systemd.
It's not systemd-oomd that is the main problem here, it's Fedora's implementation/application of it to all the [email protected]. systemd-oomd does suck at logging, but that seems to me like a silly deal-breaker to throw out the baby with the bathwater. It can (and will) be fixed.
It's not systemd-oomd that is the main problem here, it's Fedora's implementation/application of it to all the [email protected]. systemd-oomd does suck at logging, but that seems to me like a silly deal-breaker to throw out the baby with the bathwater. It can (and will) be fixed.
Quite. Can we not phase out overcommit? Windows doesn't do it.
I think fork() tends to be implemented using COW memory. Would it help to deprecate fork+exec in favour of posix_spawn()?
I think fork() tends to be implemented using COW memory. Would it help to deprecate fork+exec in favour of posix_spawn()?
What would we do about all of the processes that fork but don't ever exec? And isn't posix_spawn implemented in terms of vfork, which a lot of people like even less?
> What would we do about all of the processes that fork but don't ever exec?
I speculate it isn't common, or at least it isn't required for ordinary desktop apps. Windows survives without fork().
> And isn't posix_spawn implemented in terms of vfork, which a lot of people like even less?
That sounds like a glibc implementation detail, not something required by the standard. I'm not an expert however. Simply, again I come back to: Windows can spawn processes without using it as an excuse to overcommit.
I speculate it isn't common, or at least it isn't required for ordinary desktop apps. Windows survives without fork().
> And isn't posix_spawn implemented in terms of vfork, which a lot of people like even less?
That sounds like a glibc implementation detail, not something required by the standard. I'm not an expert however. Simply, again I come back to: Windows can spawn processes without using it as an excuse to overcommit.
> I speculate [fork] isn't common, or at least it isn't required for ordinary desktop apps.
From a science perspective, the ability to fork is really convenient for parallel processing. Load some data and then fork a pool of worker processes, and they can all read almost free copies of that data. This is much easier than setting up shared memory. It's a bigger deal on an HPC machine with 128 cores than a laptop with 8 cores, but even on a laptop it's a significant point in favour of Linux.
Getting rid of it in Linux would also be going against Linus' rule that the kernel never breaks userspace.
From a science perspective, the ability to fork is really convenient for parallel processing. Load some data and then fork a pool of worker processes, and they can all read almost free copies of that data. This is much easier than setting up shared memory. It's a bigger deal on an HPC machine with 128 cores than a laptop with 8 cores, but even on a laptop it's a significant point in favour of Linux.
Getting rid of it in Linux would also be going against Linus' rule that the kernel never breaks userspace.
Windows is a terrible OS, especially for servers, so pointing out that it doesn't have something isn't really a great argument for that thing not being very important.
Windows is proprietary and until recently, not very scriptable. None of that is relevant. We're talking about desktop users having their session killed.
What's this whole open-source thing worth if we can't fix those to do it a better way?
Alternatively, just let programs opt-out of overcommit and also OOM killer. Then only badly behaving programs will be shot in the head randomly.
Alternatively, just let programs opt-out of overcommit and also OOM killer. Then only badly behaving programs will be shot in the head randomly.
> Then only badly behaving programs will be shot in the head randomly.
So, all of them then?
So, all of them then?
Too bad it’s basically impossible to deal with the possibility that you request memory and don’t get it, apart from crashing.
There are lots of non-fatal ways to handle an out of memory condition (including doing your own paging) depending on the use-case of the allocation. Programmers suffer from learned helplessness because they're writing for systems that lie to them about memory and languages that make handling errors a pain.
That sounds obvious until you realize that you can’t get away with just fixing the obvious parts of allocating large optional caches, you have to defend for every single allocation you do to fail. In every library you use, for every string you copy, for every structure you create. That’s a lot harder than handwaving about how someone else might be able to fix some cases.
Yes, and? The difficulty of this is overstated. Except maybe that library thing, but my general guiding principle is to use as few dependencies as possible already anyway so that mitigates it somewhat.
If there are some situations where there is no logical course of action? Then yeah, crash I guess. Assuming your software isn't responsible for people's lives. It's still better than the OS randomly killing you because some other program fucked up.
I mean this is just crazy. The OS lies to you about memory, you ignore it anyway since it always lies to you, and then we randomly kill programs when we're out of memory. This is not a reasonable way to do things.
The problem, as I see it, is that by now programmers are so used to not paying attention to memory (or any other resource in most cases) that there's a ton of inertia behind the "nothing can be done" sentiment.
If there are some situations where there is no logical course of action? Then yeah, crash I guess. Assuming your software isn't responsible for people's lives. It's still better than the OS randomly killing you because some other program fucked up.
I mean this is just crazy. The OS lies to you about memory, you ignore it anyway since it always lies to you, and then we randomly kill programs when we're out of memory. This is not a reasonable way to do things.
The problem, as I see it, is that by now programmers are so used to not paying attention to memory (or any other resource in most cases) that there's a ton of inertia behind the "nothing can be done" sentiment.
I suggest you pick up a project and show us how easily this can be done.
Leave that aside for a sec. Are you honestly saying that because it isn't easy to do this, the best course of action is to make it impossible by lying to the program?
If it bothers you, you can just turn off overcommit. It’s not hard:
I didn’t claim that it is not easy. I claimed that it is practically impossible. You claimed it was easy. So please, prove me wrong and just show us how easy it is.
$ sysctl vm.overcommit_memory=2 # policy number 2
$ sysctl vm.overcommit_ratio=0 # ratio = 0%
I didn’t claim that it is not easy. I claimed that it is practically impossible. You claimed it was easy. So please, prove me wrong and just show us how easy it is.
That depends on what you mean by ‘crashing,’ doesn’t it?
One could at least fail gracefully: at start-up, request enough memory for a buffer reading ‘request to allocate memory failed; exiting’ and then print it to standard output, then exit.
One might initiate a garbage collection cycle.
One might even allocate enough buffers at start-up to cleanly process outstanding work, write results and then terminate.
It requires a little thought and engineering, but there are approaches which don’t just dump core.
One could at least fail gracefully: at start-up, request enough memory for a buffer reading ‘request to allocate memory failed; exiting’ and then print it to standard output, then exit.
One might initiate a garbage collection cycle.
One might even allocate enough buffers at start-up to cleanly process outstanding work, write results and then terminate.
It requires a little thought and engineering, but there are approaches which don’t just dump core.
For very large values of ‘a little’. Just like that other poster with the ‘easy’ solutions, I challenge you to actually implement a demonstration of those easy ideas.
You can actually take some ideas from the Linux kernel if you like because it has what you propose, a way to allocate memory that never fails and never blocks. And there you can tell how easy that is.
You can actually take some ideas from the Linux kernel if you like because it has what you propose, a way to allocate memory that never fails and never blocks. And there you can tell how easy that is.
You could hang and hope someone else free()'s enough for you to continue, but the system will lock up if too many processes try that without giving up after a timeout
Discussions about overcommit are red herring for OOM situations. You could be fully within commit limit for allocated unreclaimable memory, and still be in effective OOM situation, if working set is larger than RAM.
And even if we ignore issues with too large working sets and only focus on memory allocations, then if some badly written program eats all memory, then running some OOM killer (who hopefully kills the offender) is better than keeping it running and just failing all consequent allocations, as such system is unusable anyways.
Nope, and nope.
Doing this wont give you the fix you think it will.
Doing this wont give you the fix you think it will.
Linux has never forced you to use memory overcommit. You can argue about the default, but the default seems pretty sane for most systems/use cases.
https://www.kernel.org/doc/Documentation/vm/overcommit-accou...
https://www.kernel.org/doc/Documentation/vm/overcommit-accou...
Good to know it's not required, but I'd certainly say it's a bad default. I can understand an app that is denied a malloc crashing.
But an app which thinks it has successfully been allocated memory, being chosen at random to crash? Not ideal, just like in OP's example.
But an app which thinks it has successfully been allocated memory, being chosen at random to crash? Not ideal, just like in OP's example.
If you read the article, you will see that the issue is that Fedora configures it really poorly for a single user system not that systemd-oomd doesn't work. They enroll all the user processes into it.
The problem in this case is deeper. The Linux kernel overcommits memory.
OOMD is a workaround for this behaviour.
And it can't be disabled. I disabled the kernel memory overcommit and then neither Chrome nor Edge were running.
I can't use my computer without my browser of choice. Therefore, overcommit can't be disabled.
OOMD is a workaround for this behaviour.
And it can't be disabled. I disabled the kernel memory overcommit and then neither Chrome nor Edge were running.
I can't use my computer without my browser of choice. Therefore, overcommit can't be disabled.
Chrome and Edge not functioning without overcommit is not the fault of the kernel though.
The browsers will allocate more memory than is physically available in the system but often not "touch" it. Without over-commit this causes an OOM situation and the kernel kills the browser, but that's what it's supposed to do when over-commit is disabled!
I believe some of these massive allocations are used for security purposes or somehow related to the chromium sandbox but I can't remember exactly what's going on. Some sanitizers like ASAN do something similar for detecting invalid memory accesses (and hence can't be used without over-commit either)!
The browsers will allocate more memory than is physically available in the system but often not "touch" it. Without over-commit this causes an OOM situation and the kernel kills the browser, but that's what it's supposed to do when over-commit is disabled!
I believe some of these massive allocations are used for security purposes or somehow related to the chromium sandbox but I can't remember exactly what's going on. Some sanitizers like ASAN do something similar for detecting invalid memory accesses (and hence can't be used without over-commit either)!
Maybe the problem is with the bloated apps that don't free memory? You can't blame the kernel for that. I routinely see Firefox consuming 10GB of RAM on my Linux desktop. It's nuts.
Do you seriously consider the kernel OOM killer to be "battle-tested" and (implied) sufficient?
I'm not saying the kernel one is perfect. I'm just saying it's better than this one.
Can I ship a badly configured Linux kernel in my distro? Yes.
Is that the fault of the kernel developers? No.
I thought systemd is open source and if you don't like something about it, you can just submit a patch or even fork it or just not use it.
Replacing system components is hard and the early days are going to suck. Just because there is a lot of pain and suffering with Wayland does not mean that it was a mistake and we should have just stuck with X11.
Computing has changed since 1970, some things in our unix-like systems will also need to change to accommodate the current world and some of it is going to be painful, that's just how it is, we made fun of Vista, now it's our turn.
I thought systemd is open source and if you don't like something about it, you can just submit a patch or even fork it or just not use it.
Replacing system components is hard and the early days are going to suck. Just because there is a lot of pain and suffering with Wayland does not mean that it was a mistake and we should have just stuck with X11.
Computing has changed since 1970, some things in our unix-like systems will also need to change to accommodate the current world and some of it is going to be painful, that's just how it is, we made fun of Vista, now it's our turn.
> Can I ship a badly configured Linux kernel in my distro? Yes. Is that the fault of the kernel developers? No.
If the default configuration were good and you changed it to be bad, that's entirely your fault. But if the default configuration were bad and you just didn't fix it, that's mostly if not entirely upstream's fault.
> I thought systemd is open source and if you don't like something about it, you can just submit a patch or even fork it or just not use it.
Yes, we can do that. The point is that we shouldn't have to.
> Replacing system components is hard and the early days are going to suck.
Nobody's complaining that things are getting replaced. We're complaining that the replacement is worse than the old thing.
> Computing has changed since 1970, some things in our unix-like systems will also need to change to accommodate the current world and some of it is going to be painful, that's just how it is, we made fun of Vista, now it's our turn.
Systemd didn't exist until 2010, so any changes from 1970 to 2010 clearly can't have made it necessary. And the BSDs are all still good, functional operating systems, despite not being able to run it, so it's clearly not necessary even for what's changed since 2010.
If the default configuration were good and you changed it to be bad, that's entirely your fault. But if the default configuration were bad and you just didn't fix it, that's mostly if not entirely upstream's fault.
> I thought systemd is open source and if you don't like something about it, you can just submit a patch or even fork it or just not use it.
Yes, we can do that. The point is that we shouldn't have to.
> Replacing system components is hard and the early days are going to suck.
Nobody's complaining that things are getting replaced. We're complaining that the replacement is worse than the old thing.
> Computing has changed since 1970, some things in our unix-like systems will also need to change to accommodate the current world and some of it is going to be painful, that's just how it is, we made fun of Vista, now it's our turn.
Systemd didn't exist until 2010, so any changes from 1970 to 2010 clearly can't have made it necessary. And the BSDs are all still good, functional operating systems, despite not being able to run it, so it's clearly not necessary even for what's changed since 2010.
Probably around the same time as people stop posting poorly informed comments about systemd on Hacker News (i.e. approximately never)
You do realize that oomd has as much to do with systemd as your bash script that has a service file written for it?
What are you talking about? Systemd-oomd is literally part of systemd. Its source code is maintained in the same Git repo, bugs are tracked in the same project, etc. Are you getting it mixed up with Facebook's oomd?
I never started, I went right from Debian 5 to OpenBSD.
Same here; been disabling it since it was introduced to Fedora... a release or two ago.
My desktops have 64GB+ of memory typically, they're virtualization workhorses. I don't know what it is in oomd, but any time I make actual real use of it, things started getting killed.
Zero memory pressure. Talking sections of 60+GB free, so it's not fragmentation either.
The grouping of the entire session the link suggests seems likely
My desktops have 64GB+ of memory typically, they're virtualization workhorses. I don't know what it is in oomd, but any time I make actual real use of it, things started getting killed.
Zero memory pressure. Talking sections of 60+GB free, so it's not fragmentation either.
The grouping of the entire session the link suggests seems likely
Systemd-oomd, may be great for non-desktop server's needs for some, but definitely not great for single-user desktops, much less multi-user desktops.
THE Pain point remains in systemd-oomd's poor (and often, no) crash dump or traceback support.
Nevertheless, systemd-oomd remains an excellent OOM mass-killer (via cgroupv2), just not the discriminating assassin ... as (mis-)configured ... by Redhat/Fedora.
THE Pain point remains in systemd-oomd's poor (and often, no) crash dump or traceback support.
Nevertheless, systemd-oomd remains an excellent OOM mass-killer (via cgroupv2), just not the discriminating assassin ... as (mis-)configured ... by Redhat/Fedora.
This is non-init related systemd from IBM->Red Hat->Fedora in a nutshell. It's good for fleets of mega-corp servers. It's bad for human people running linux desktops. But since IBM is the primary entity paying people to write non-init systemd is everywhere and only niche distros avoid it.
Have you ever met someone running fleets of mega-corp servers who was a fan of systemd?
When I've seen them complain the systemd people tell them that they're just stuck in their ways and that systemd is a revelation on desktops where users don't have the historical unix experience holding them back...
When I've seen them complain the systemd people tell them that they're just stuck in their ways and that systemd is a revelation on desktops where users don't have the historical unix experience holding them back...
I and multiple of my coworkers love systemd. We work in a "hyperscale" environment and we moved from sysvinit through Upstart to systemd at scale, predominantly on RHEL and a little Debian. We were all skeptical at first. But sysvinit and Upstart both sucked for various reasons, and we had tried to work around the warts. systemd has mostly solved all of those problems for us. Some of the things that systemd improved:
Init scripts suck to test and debug. When manually-called, the caller's environment would get used, rather than a clean env, which would cause problems with the poorer-quality scripts. These showed up multiple times in production where an engineer would bounce a process manually and it got some configuration from their environment. We took some steps to alleviate this, like encouraging use of `service`, but using the scripts directly was reflexive for most because of years of indoctrination and it's portability to some other operating systems. systemd eliminated this entire class of issues, because unit files are just config files, not scripts.
Dependencies actually work. Usually, this came up when needing to wait on the network. With systemd, this is just a single dependency in the unit file. It also meant we stopped having to manage S## and K## which became something of a time-consuming art on machines with lots of services.
Not exactly init system, but systemd's triggered units absolutely destroy cron. Manually running a job (e.g. when testing or when a run failed) guarantees the same environment, whereas cron has no similar capabilities. It also actually keeping status means we can more easily see if the last run succeeded. This may ease our monitoring burden, too, but we haven't gotten to give it attention yet.
Init scripts suck to test and debug. When manually-called, the caller's environment would get used, rather than a clean env, which would cause problems with the poorer-quality scripts. These showed up multiple times in production where an engineer would bounce a process manually and it got some configuration from their environment. We took some steps to alleviate this, like encouraging use of `service`, but using the scripts directly was reflexive for most because of years of indoctrination and it's portability to some other operating systems. systemd eliminated this entire class of issues, because unit files are just config files, not scripts.
Dependencies actually work. Usually, this came up when needing to wait on the network. With systemd, this is just a single dependency in the unit file. It also meant we stopped having to manage S## and K## which became something of a time-consuming art on machines with lots of services.
Not exactly init system, but systemd's triggered units absolutely destroy cron. Manually running a job (e.g. when testing or when a run failed) guarantees the same environment, whereas cron has no similar capabilities. It also actually keeping status means we can more easily see if the last run succeeded. This may ease our monitoring burden, too, but we haven't gotten to give it attention yet.
Certainly no one loves whatever hellhole preceded systemd. It turns out that (some) complexity is non-reducible; if a declarative, properly solved service manager doesn’t cut it, then neither does that shitty ad-hoc bash script system abomination with race conditions.
If systemd were just a declarative sysvinit alternative I don't think there'd be near the pushback.
Manage a fleet of at least 15k systems here, I enjoy systemd at times
Other times, like oomd, I just want it to get out of the way.
Service dependencies are an area I love. With systemd making a service robust around dependencies like mounts and other services is a cake walk.
If you have manual hand-holding processes on services running under systemd, I suggest looking into supplying these kinds of directives.
However, forwarding is significantly different. I'd think twice about using it on a system doing NAT/masq
Other times, like oomd, I just want it to get out of the way.
Service dependencies are an area I love. With systemd making a service robust around dependencies like mounts and other services is a cake walk.
If you have manual hand-holding processes on services running under systemd, I suggest looking into supplying these kinds of directives.
- Requires / Wants
- Before / After
- PartOf
However, forwarding is significantly different. I'd think twice about using it on a system doing NAT/masq
I've just heard from people claiming the absurd way NIC are named in systemd is good for server fleet types and the like. Because it's obviously terrible for human people with desktop computers. I guess I just assumed the bad design decisions had some purpose.
I miss the old NIC names, but for systems with multiple NICs the old way would name them differently based on the winner of the init race. It often felt deterministic, but was not.
I've had critical systems fail on reboot because the NICs swapped names so WAN port got configured as LAN and vice versa. That can also create a terrifying security problem too
I've had critical systems fail on reboot because the NICs swapped names so WAN port got configured as LAN and vice versa. That can also create a terrifying security problem too
[deleted]
Some distros wrote config files so the assignment would remain stable.
Which also was wrong when switching hardware around, but that's a different issue.
Which also was wrong when switching hardware around, but that's a different issue.
It provides predictability in the device names. enp1s0f0 looks cryptic but it tells you the exact slot location of the card
eth0 tells you the first ethernet class interface that was activated that boot. Not particularly helpful.
I feel like if you need to care about these names as a user... the distribution has failed to provide effective wrappers. This is window dressing.
eth0 tells you the first ethernet class interface that was activated that boot. Not particularly helpful.
I feel like if you need to care about these names as a user... the distribution has failed to provide effective wrappers. This is window dressing.
Yes. When we got a shipment of 1000+ servers that are all supposed to be cabled the same way, both internally and externally, the device naming made it easy to see when something got messed up. This allowed us to hold the system builder accountable, simplify our configuration management, and reduce the amount of configuration discovery a engineer needed to do during an incident.
Very good examples of the practicality!
While sure, I would like shorter device names, the fact is... the only time I need to use them is when ambiguity is something I/we can't really afford.
So, I'm thankful for the cryptic-yet-specific names.
Everyone else (read: desktop people) can use whatever DHCP'd.
While sure, I would like shorter device names, the fact is... the only time I need to use them is when ambiguity is something I/we can't really afford.
So, I'm thankful for the cryptic-yet-specific names.
Everyone else (read: desktop people) can use whatever DHCP'd.
o/
I'm not a systemd fan but I'm not a hater either. I think it does its job and, although it was a bit immature when initially released (especially by Fedora) it has matured into a very good init system that does it job.
But yes, Fedora keeps pushing for the bleeding edge, but its users should know this by now.
Yeah. That's also mostly my opinion too. Systemd itself is okay. Just another init system, does some things better, a few things are kind of stirred around for no gain. What I've less been a fan of is the rider tech, networkd, their ntp support, oomd, all of which have more issues and don't seem to be converging. But if they do become stable I expect some new broken tentacle arm to form. :)
okayness meter decreased another notch today: system was ignoring fstab changes. turns out that systemd now handles fstab and you have to trigger the service to re-read the file if you edit it.
At least it doesn't require a reboot. Yet.
At least it doesn't require a reboot. Yet.
It’s particularly good on servers. All preceding systems were much worse.
It’s only useful on desktop because of its similarities with launchd, which is particularly good on desktop.
It’s only useful on desktop because of its similarities with launchd, which is particularly good on desktop.
They don't love systemd because they only interact with it when they're debugging Asok the intern's broken services.
Killing the user's graphical session is terrible.
On the other hand, not having a userspace OOM killer means (meant?) that if your system runs out of memory, it will become so slow as to be totally unresponsive and the only thing you can do is to reboot. Which is even worse. Is that still true today or did the kernel OOM killer improved?
How does earlyoom compare against systemd-oomd?
On the other hand, not having a userspace OOM killer means (meant?) that if your system runs out of memory, it will become so slow as to be totally unresponsive and the only thing you can do is to reboot. Which is even worse. Is that still true today or did the kernel OOM killer improved?
How does earlyoom compare against systemd-oomd?
I'll just say that earlyoom is much more bearable out-of-the-box than what this sounds like and much better than waiting for Linux to trigger the internal OOM killer (or trying to remember the SysRq key sequence to just trigger it. I always forget and don't have it close by when I need it :) ). I had to play around with the configuration a little to get it to match the conditions I could potentially get an OOM, but when it did reach it on my desktop, it would kill something like Slack, definitely not the whole X session. (FWIW, this happened when building a large Java project, that would burst with memory usage at certain stages of a build, on my computer that just has 16gb. Outside of managing OOM, it's something to play around with before giving up and just upping the RAM. :) )
Alt+SysRq+REISUB
Or, as a mnemonic: Reboot Even If System Utterly Borked*
*Being Irish, I prefer "Banjaxed" for the last word, personally.
Or, as a mnemonic: Reboot Even If System Utterly Borked*
*Being Irish, I prefer "Banjaxed" for the last word, personally.
Ah yeah, that :D But that's a full terminate process/filesystem sync/shutdown + restart ...
You got me to look it up: Alt-SysRq-F just for the kernel oom_killer ...
On ThinkPads, that's: Press Fn-Alt-S, then release Fn-S, then press the desired letter. Result from dmesg: "[154256.637037] sysrq: This sysrq operation is disabled." Bleh. Didn't work when you needed it :) Of course these can be enabled, but earlyoom is soo much easier than remembering the OOM dance or the REISUB dance (particularly if it is just an OOM :D)
You got me to look it up: Alt-SysRq-F just for the kernel oom_killer ...
On ThinkPads, that's: Press Fn-Alt-S, then release Fn-S, then press the desired letter. Result from dmesg: "[154256.637037] sysrq: This sysrq operation is disabled." Bleh. Didn't work when you needed it :) Of course these can be enabled, but earlyoom is soo much easier than remembering the OOM dance or the REISUB dance (particularly if it is just an OOM :D)
> it will become so slow at to be totally unresponsive and the only thing you can do is to reboot.
It's been well over a decade since I've seen that kind of behavior. Storage devices became much lower latency and the kernel swapper got much smarter.
It's been well over a decade since I've seen that kind of behavior. Storage devices became much lower latency and the kernel swapper got much smarter.
Can't confirm. I have more than 32GiB of memory and very fast storage and I see freezes more often than something being killed. The worst I saw was LUKS/dm-crypt being unable to allocate memory and slightly corrupting one of the mounted filesystems.
Linux Desktop in OOM situations is absolutely trash.
Linux Desktop in OOM situations is absolutely trash.
It still happens on an 8GiB laptop even with zram (I should probably buy a ram stick at this point).
Oh, definitely not my experience with 16 GB of RAM.
anecdotally, i've found that on a low-ram laptop having zram (i.e. compressed swap in ram) helps a lot, smoothing out the memory exhaustion. things start to get slower very gradually (as now swap churn means constant (de)compression rather than disk access), allowing you or the oom killer time to act.
I can also share this sentiment, though I have to add that unfortunately the kernel can still get into a state where it constantly churns pages to-from disk swap as well, freezing so hard that not even alt+printscr+f (oom killer) can react. Reisub works though. It seems to get fixed/reintroduced by version to version..
I found (no hard evidence though) that zswap behaves even more nicely, at least on a Raspberry Pi (and on top of that is easier to configure, it's just an on/off setting that can be done via procfs or boot argument, zswap.enabled=1).
Systemd is wonderful. But could they please stop trying to invent lightweight replacements?
You're not suckless. Nobody wants you to be suckless. I like you specifically because you aren't anything like suckless.
Don't replace anything unless you have close to full feature parity. Nobody wants your minimum viable product components. The people who do don't want systemd.
Init system? Absolutely. Timers? You've done a wonderful job. Time sync? Please no, crony does it better.
You're not suckless. Nobody wants you to be suckless. I like you specifically because you aren't anything like suckless.
Don't replace anything unless you have close to full feature parity. Nobody wants your minimum viable product components. The people who do don't want systemd.
Init system? Absolutely. Timers? You've done a wonderful job. Time sync? Please no, crony does it better.
> Achievement unlocked: I just had systemd-oomd kill my entire X session for unclear reasons.
> Does it log details about the session it killed so I could see, say, which program was using too much memory?
> Nope.
>
> Systemd-oomd has now been voted off my island.
https://github.com/systemd/systemd/blob/v252/src/oom/oomd-ma...
Chris's wiki has now been voted out of my sight.
https://github.com/systemd/systemd/blame/e8dc52766e1fdb4f8c0...
Yeah, looks like logging was added 1.5 years ago so hard to take his perspective seriously.
Yeah, looks like logging was added 1.5 years ago so hard to take his perspective seriously.
It's also problematic that he doesn't seem to have reported the issue to either his distributor or upstream systemd. If he did, it's not mentioned in his rant. There aren't any links to any bug reports where the developers demonstrably failed to care or anything rant-worthy.
Considering systemd-oomd is a relatively new addition to the project, contributed by FB staff, it wouldn't surprise me if there were desktop integration issues yet to be sorted out by the various distros.
The post doesn't strike me as made in good faith, seems written by someone with an axe to grind with systemd.
Considering systemd-oomd is a relatively new addition to the project, contributed by FB staff, it wouldn't surprise me if there were desktop integration issues yet to be sorted out by the various distros.
The post doesn't strike me as made in good faith, seems written by someone with an axe to grind with systemd.
To be totally fair, that also happens with the kernel's OOM killer.
Sometimes you wish X had been killed, instead of a total freeze or for example LUKS/dm-crypt being unable to allocate memory (and causing way way way worse issues than X being killed).
Sometimes you wish X had been killed, instead of a total freeze or for example LUKS/dm-crypt being unable to allocate memory (and causing way way way worse issues than X being killed).
Dumb question: why is this a thing at all if the kernel oom killer already exists? Doesn't this effectively do the same as the existing oom killer, but worse?
I believe the kernel OOM killer only fires when you're completely out of memory. But it's not unusual for swap thrashing to make the system unusable before that, forcing the user to either try to kill a likely culprit by hand, or hold down the power button.
systemd-oomd (and earlyoom, which Fedora used for a couple of releases before switching) try to detect that you're running out of RAM before the system becomes unusable, and kill a big memory user. It periodically killed Firefox before I bought some more RAM, which was a pain, but less of a pain than hard-rebooting the laptop.
The plans for Fedora to do this are public pages:
https://fedoraproject.org/wiki/Changes/EnableEarlyoom https://fedoraproject.org/wiki/Changes/EnableSystemdOomd
systemd-oomd (and earlyoom, which Fedora used for a couple of releases before switching) try to detect that you're running out of RAM before the system becomes unusable, and kill a big memory user. It periodically killed Firefox before I bought some more RAM, which was a pain, but less of a pain than hard-rebooting the laptop.
The plans for Fedora to do this are public pages:
https://fedoraproject.org/wiki/Changes/EnableEarlyoom https://fedoraproject.org/wiki/Changes/EnableSystemdOomd
Iirc, systemd-oomd comes from oomd which comes from Facebook who had issues with the kernel oom killer. (The kernel oom killer, especially with slow swap, can also take ages to activate, which means you're just down the entire time, and it's better to kill stuff more often than to sometimes freeze.)
I've had real-world downtime as a result of exactly this. The kernel OOM often takes forever to kick in. Like 10 to 15 minutes is not uncommon.
The most facepalm one was somebody uploaded their `.vimrc` which had a bug that infinitely recursed the parent directory loading ctags until it gobbled up all available memory and crashed. So every time he opened vim, it started about a 40 second timer of doom unless vim was exited before memory exhausted. Took forever to figure it out too, because he often only opened files for 10 seconds or less, short enough not to trigger the OOM.
The most facepalm one was somebody uploaded their `.vimrc` which had a bug that infinitely recursed the parent directory loading ctags until it gobbled up all available memory and crashed. So every time he opened vim, it started about a 40 second timer of doom unless vim was exited before memory exhausted. Took forever to figure it out too, because he often only opened files for 10 seconds or less, short enough not to trigger the OOM.
Yes it does. Basically everything under the systemd umbrella does the same thing as something that already existed (remember that we had perfectly good Linux boxes before its release in 2010), but worse than the old thing.
I have serious doubts about people's familiarity with the alternatives if they say something like this.
Is it really mentally healthy to hate to this degree a goddamn free, libre, open-source project?
Their post seemed pretty tame. No evocative emotions or deity-assisted emphasis. They just said that between the available free, libre, open-source projects they prefer what they had before -- akin to saying they like to install qutebrowser rather than firefox.
It's not like systemd gave us our freedom. The things it replaced were also FLOSS.
Mint opted out of systemd-oomd with v21: https://news.ycombinator.com/item?id=31960266
Ubuntu 22.04 was also affected: https://news.ycombinator.com/item?id=31530739
And flipped a config flag to make it less aggressive: https://www.phoronix.com/news/Ubuntu-Drops-Swap-Kill
Ubuntu 22.04 was also affected: https://news.ycombinator.com/item?id=31530739
And flipped a config flag to make it less aggressive: https://www.phoronix.com/news/Ubuntu-Drops-Swap-Kill
I confirm that the state of systemd-oomd is ridicolous right now and will easily kill important background processes that aren't even taking up that much RAM. It killed my Thunar file transfer, for example
I've been running nohang for quite a while now on all my desktop linux devices. In the rare cases where I mismanaged my memory or had a program hang it has always managed to kill the culprit and only the culprit.
It also gives me a nice desktop notification about what it killed, though I'm not sure where/if a more permanent log is written.
It also gives me a nice desktop notification about what it killed, though I'm not sure where/if a more permanent log is written.
OOMD seems to have a bug that if you have >64 GB of ram (or maybe >128GB?) it will randomly kill very large processes even though you're nowhere near running out of memory (much less swap).
Cost me god knows how much money or wasted power from lost computations until I figured that out and disabled it.
Cost me god knows how much money or wasted power from lost computations until I figured that out and disabled it.
Did you lodge a bug, i'd like to weigh in on that.
Cost me god knows how much money...
This might have made some IaaS firms happy?
This might have made some IaaS firms happy?
Maybe Off-topic, not sure if systemd is to blame or Apache or the package maintainers , but on a LAMP stack i was creating some images in the temp folder then do some work on them.
So for debugging I wanted to opent he temp folder and look at the images. The files were not there, the system is now full of magic, it pretends it gives you a temp file like /tmp/file1.png but the file is not there. I fcking Have o idea why they think is a good idea to lie to me what the file path is, I had luck that some other person hit this issue and somehow google managed to put it as a result to my searches.
It's because Apache httpd or other services are probably using their own "version"/"instance" of /tmp for better isolation from the rest of the system and extra security.
From the point of view of the program the files are written under /tmp, but in reality they're stored somewhere else. Try this to see the files:
[1]: https://www.freedesktop.org/software/systemd/man/systemd.exe...
From the point of view of the program the files are written under /tmp, but in reality they're stored somewhere else. Try this to see the files:
ls /tmp/systemd-private-$(systemd-id128 boot-id)-httpd.service-*/tmp
systemctl show --all httpd.service
systemctl edit httpd.service
[1]: https://www.freedesktop.org/software/systemd/man/systemd.exe...
Thanks. But this is such a antiuser feature, like the incident was like
1 I use std function to get a temp file path
2 I write content in that file
3 I do all the checks to ensure the write worked, I use std functions to check file exists
4 I use std function to print the file path
And the file path is a lie, It all would be fine if it would use the real path, and if a newb would hardocde "/tmp/file1" then I would be happy with a error message that I can't write there .
I assume they had some reasons for the lie, I am not convinced it was worth it, I do not like magic hidden stuff.
1 I use std function to get a temp file path
2 I write content in that file
3 I do all the checks to ensure the write worked, I use std functions to check file exists
4 I use std function to print the file path
And the file path is a lie, It all would be fine if it would use the real path, and if a newb would hardocde "/tmp/file1" then I would be happy with a error message that I can't write there .
I assume they had some reasons for the lie, I am not convinced it was worth it, I do not like magic hidden stuff.
I've been bitten by this, so I know the feeling. Once you know what's going on, it's not magic anymore :-)
The "antifeature" can be disabled if you don't like it, but keep in mind that it increases security. There have been vulnerabilities related to temporary files [1][2]. Usually there is no need for a web server running in production to make its temporary files accessible to the rest of the world.
The "lie" is similar to what happens when you're using virtual machines. The virtual machine thinks it's using some hardware, but in reality...
On a side note, Flatpak also does this by default sometimes. It can run applications in a sandbox in which even the home directory is partially inaccessible.
[1]: https://cwe.mitre.org/data/definitions/377.html
[2]: https://nvd.nist.gov/vuln/search/results?form_type=Advanced&...
The "antifeature" can be disabled if you don't like it, but keep in mind that it increases security. There have been vulnerabilities related to temporary files [1][2]. Usually there is no need for a web server running in production to make its temporary files accessible to the rest of the world.
The "lie" is similar to what happens when you're using virtual machines. The virtual machine thinks it's using some hardware, but in reality...
On a side note, Flatpak also does this by default sometimes. It can run applications in a sandbox in which even the home directory is partially inaccessible.
[1]: https://cwe.mitre.org/data/definitions/377.html
[2]: https://nvd.nist.gov/vuln/search/results?form_type=Advanced&...
P.S. When you're editing the service, write this in the drop-in:
[Service]
PrivateTmp=nothat sounds like my good old enemy https://www.freedesktop.org/software/systemd/man/tmpfiles.d....
Systemd seems great for servers and sysadmins. Does it's size, scope, and complexity actually do something for the grandmas of the world who use Linux? For that basic type of user, was the millions of lines of code and probably hundreds of thousands of hours of programmer's time worth it?
I'm a sysadmin for a medium-sized fleet of servers, and systemd definitely isn't great for me.
In what way?
It makes changes that break things (like suddenly kill -9'ing all of user's nohup and screen processes when they exited their SSH sessions, completely defeating the purpose of the tools) and replaces old simple ways of doing things with new complicated ways, without benefits of the new ways to justify the changes.
> like suddenly kill -9'ing all of user's nohup and screen processes when they exited their SSH sessions
This was surprising, I was unable to replicate it though. Did you set systemd to automatically clean up sessions?
From the man page of logind.conf,
This was surprising, I was unable to replicate it though. Did you set systemd to automatically clean up sessions?
From the man page of logind.conf,
KillUserProcesses= Takes a boolean argument. Configures whether the processes of a user should be killed when the user logs out. If true, the scope unit corresponding to the session and all processes inside that scope will be terminated.The default is 'yes'. For awhile Fedora overrode it no, but then abandoned that: https://fedoraproject.org/wiki/Changes/KillUserProcesses_by_...
That is what was doing it, but I didn't set it to do that. It was on by default when it was first released. Since then, the world realized how insane that was and major distros now turn it off for you.
They gave both a compile time and runtime configuration option to set the default behavior, package maintainers didn’t read the docs and shipped it incorrectly. I doubt systemd people are at fault here.
Plus, on a theoretical level that is the correct behavior, you don’t want to leave running processes gobbling up resources under a logged out user without explicitly allowing it.
Plus, on a theoretical level that is the correct behavior, you don’t want to leave running processes gobbling up resources under a logged out user without explicitly allowing it.
Speaking as one of many who has lost work to the systemd-kills-your-processes issue: No, it's 100% systemd's fault. They shipped a change that broke nohup, screen, tmux, and a dozen other things. All of the feedback they've gotten about the process-killing feature has been negative, but their response has been to shut their ears and blame the problem on distros for not overriding their insane design decision.
The point of a default is that it's what a setting should be if you don't go out of your way to change it. The default was on, and you're faulting the distro maintainers for not changing it to off, when it should have just defaulted to off all along. (Or are you saying that it did default to off, and that the distro maintainers originally went out of their way to turn it back on? If so, that would surprise me, but I'd concede the point.)
Correct behavior is that a demonized process doesn't ever get killed just because another process in its original family tree dies.
Correct behavior is that a demonized process doesn't ever get killed just because another process in its original family tree dies.
> Correct behavior is that a demonized process doesn't ever get killed just because another process in its original family tree dies
Who decides what is a demonized process, vs one that just froze and won’t respond to some random UNIX convention, that can’t even reliably signal its state (what is tmux when it is frozen?)
So the solution is to explicitly specify that a given process wishes to live outside of the seat’s lifetime (either by specifying it as a service, which your earlier example ssh is on basically any distro, or by executing it as with systemd-run)
Who decides what is a demonized process, vs one that just froze and won’t respond to some random UNIX convention, that can’t even reliably signal its state (what is tmux when it is frozen?)
So the solution is to explicitly specify that a given process wishes to live outside of the seat’s lifetime (either by specifying it as a service, which your earlier example ssh is on basically any distro, or by executing it as with systemd-run)
> explicitly specify that a given process wishes to live outside of the seat’s lifetime
Okay, so it does that now. But now how do you tell if it's frozen?
Okay, so it does that now. But now how do you tell if it's frozen?
That’s not really the point, but there are of course plenty of ways for that (output, watchdogs, etc).
The point is that the OS does actually know “statically” that a given process is a daemon or not.
The point is that the OS does actually know “statically” that a given process is a daemon or not.
How is this random systemd convention better than a random UNIX convention though?
One is decidable the other can have false positives.
Very little locality of behavior. A service can appear in something like 8 different folders, can depend on stuff that is who knows where. It is just a not very elegant system, it's baroque and confusing where it doesn't need to be.
The solution of course is more systemd. Instead of using generic Linux tools like grep, learn and remember a bunch of weird systemd-journald specific commands. Unfortunately I'm also a programmer, and only really have room in my head for one grep command, so it introduces frission and I have to look up the commands every time.
Also just a lot of random breakage over the years.
The solution of course is more systemd. Instead of using generic Linux tools like grep, learn and remember a bunch of weird systemd-journald specific commands. Unfortunately I'm also a programmer, and only really have room in my head for one grep command, so it introduces frission and I have to look up the commands every time.
Also just a lot of random breakage over the years.
It’s almost like service management is a hard problem, and that shitty bash one-liner won’t solve anything correctly. It is basically a dependency manager, yet you are not replacing apt/pacman whatever with unix tools piped together, do you?
It's a bit of a false choice to say the options are systemd or write everything in bash. Personally I do like OpenRC.
Service management is a hard problem when you start adding things like socket activation, home directory management, DNS resolving, container management, log management, and whatever else they want to throw in.
Now you might say "well DNS resolving isn't actually part of systemd, it's a seperate daemon", well they've been more and more tightly coupling all these different components together. Yes you can disable systemd's DNS resolver, but probably something else will break. They're all pretty tightly coupled together and they mostly seem to do that in a dumb way.
Service management is a hard problem when you start adding things like socket activation, home directory management, DNS resolving, container management, log management, and whatever else they want to throw in.
Now you might say "well DNS resolving isn't actually part of systemd, it's a seperate daemon", well they've been more and more tightly coupling all these different components together. Yes you can disable systemd's DNS resolver, but probably something else will break. They're all pretty tightly coupled together and they mostly seem to do that in a dumb way.
Probably, since all of those users rely on services provided by servers managed by sysadmins.
But it's such a misfit on desktop distros that the criticism there is valid and much louder. It never should've been a one-daemon-fits-all solution.
But it's such a misfit on desktop distros that the criticism there is valid and much louder. It never should've been a one-daemon-fits-all solution.
systemd has never been a one-daemon-fits-all solution... it's dozens of daemons.
Collectively known as legiond-systemd.
I'm not using systemd, but have been pretty happy with earlyoom. It tends to take out my web browser I think. Fairly inoffensive.
So many things wrong with this article. You can absolutely see which programs triggered systemd-oomd into action in journalctl logs. Even better, I now seem to get notifications with the culprit whenever it kicks in for my desktop setup.
I can also see the design decision made to kill the graphical session when the system becomes unresponsive. The alternative to that situation pre-systemd-oomd was a force reboot of the system.
I can also see the design decision made to kill the graphical session when the system becomes unresponsive. The alternative to that situation pre-systemd-oomd was a force reboot of the system.
> By having the [email protected] template be enrolled in systemd-oomd, Fedora made the cgroup that systemd-oomd would select to be killed be all of your processes
This is so obviously poorly thought-out as to border malicious negligence.
This is so obviously poorly thought-out as to border malicious negligence.
Great blog, I really enjoyed some blogposts about UNIX which may be found on this resource.