Ubuntu 22.04's new OOM kiler closes Firefox while in use(old.reddit.com)
old.reddit.com
Ubuntu 22.04's new OOM kiler closes Firefox while in use
https://old.reddit.com/r/Ubuntu/comments/uyl4i6/ubuntu_2204s_new_oom_killing_system_is_killing/
45 comments
To be clear, "Ubuntu 22.04's new OOM killer" is systemd. Why do we keep taking responsibilities away from the kernel and other programs that are good at handling them, only to give them to systemd for this kind of thing to happen?
The Linux OOMKiller is inflexible and as likely to kill the right thing as the wrong thing when there's memory pressure on the system. There's a reason why third-party userspace [0] ones are used in lieu of the builtin one. You can tweak the default implementation, but never fully rely on it behaving in a deterministic manner.
Whether this systemd implementation is any better is, of course, another question entirely.
[0] https://github.com/facebookincubator/oomd
Whether this systemd implementation is any better is, of course, another question entirely.
[0] https://github.com/facebookincubator/oomd
To be clear, Ubuntu 22.04's Firefox is snap Firefox, with horrible performance, so systemd decision to end its misery could be justified.
The kernel isn't good at handling OOM – to the nearest approximation it just kills whatever is using the most memory, which in my experience is almost always the thing that I least want it to kill. And it does it late enough that everything is broken anyway and the system sits thrashing forever.
All that seems to have happened here is that Ubuntu may have rolled out systemd-oomd with some overly-conservative settings, so if we could all avoid having a collective pointless shit-fit about systemd again that would be great.
All that seems to have happened here is that Ubuntu may have rolled out systemd-oomd with some overly-conservative settings, so if we could all avoid having a collective pointless shit-fit about systemd again that would be great.
> it just kills whatever is using the most memory, which in my experience is almost always the thing that I least want it to kill.
I've had the opposite experience. Usually when I'm out of memory, it's because some single process has gone crazy with memory allocation. Killing it brings the system back to "normal", but killing anything else would just lead to said crazy process gobbling up the newly-freed memory, and then being back to where I started.
> And it does it late enough that everything is broken anyway and the system sits thrashing forever.
This part I agree with.
I've had the opposite experience. Usually when I'm out of memory, it's because some single process has gone crazy with memory allocation. Killing it brings the system back to "normal", but killing anything else would just lead to said crazy process gobbling up the newly-freed memory, and then being back to where I started.
> And it does it late enough that everything is broken anyway and the system sits thrashing forever.
This part I agree with.
Are we REALLY still complaining about systemd? Systemd has been one of the best things to happen to Linux.
I am not someone who knows arcane system programming, but I use Linux since 20 years on commercial laptops, from Asus, DELL and Lenovo, etc. IMO the Linux of the 2000' where much better than today's Linux. The Linux I use are always Debian based, like Ubuntu.
For example Linux used to have a sleep mode which worked perfectly. Now it does not sleep reliably. For example if I move the mouse while it sleeps, it wakes up, even with the screen closed and uses the battery until there is no more energy in them. So the computer is unusable when I need it.
Another example is copy/paste. Today it works sometimes, some other times it paste another previous content. I use now Ubuntu 20.0.4 LTS. I just copied/pasted the version number from the system monitor panel to write this post, it pasted a previous content I copied in my browser!
A third example when I use a right click on Ubuntu's Chromium (a snap browser) then I have garbage on the screen. This browser can't display correctly non western characters.
So I do not know if its because systemd but a few years ago I used Trisquel Linux which did not used systemd at the time, and everything worked perfectly as I remember in the 2000'.
For example Linux used to have a sleep mode which worked perfectly. Now it does not sleep reliably. For example if I move the mouse while it sleeps, it wakes up, even with the screen closed and uses the battery until there is no more energy in them. So the computer is unusable when I need it.
Another example is copy/paste. Today it works sometimes, some other times it paste another previous content. I use now Ubuntu 20.0.4 LTS. I just copied/pasted the version number from the system monitor panel to write this post, it pasted a previous content I copied in my browser!
A third example when I use a right click on Ubuntu's Chromium (a snap browser) then I have garbage on the screen. This browser can't display correctly non western characters.
So I do not know if its because systemd but a few years ago I used Trisquel Linux which did not used systemd at the time, and everything worked perfectly as I remember in the 2000'.
This sounds more like issues with snap (and its isolation model) than systemd.
Power issues are always a pain. I built out the Linux OS for a fleet of mobile robots and found that the defaults for power management were very poor, but everything is there to get a bulletproof system. I could see how there would be issues with more unique laptop setups or external monitors/keyboards and whatnot.
Power issues are always a pain. I built out the Linux OS for a fleet of mobile robots and found that the defaults for power management were very poor, but everything is there to get a bulletproof system. I could see how there would be issues with more unique laptop setups or external monitors/keyboards and whatnot.
This could also just be Ubuntu. So many distros are based on Ubuntu and it’s pretty clear that Canonical doesn’t really care about Ubuntu on the desktop.
It’s little more than a sales funnel for them to make sure people use Ubuntu servers as well, at this point.
It’s little more than a sales funnel for them to make sure people use Ubuntu servers as well, at this point.
What is the best modern desktop distro? I am running Ubuntu on my newly-built "Gaming" linux box, and I am impressed with how far gaming on linux has come, but I am unimpressed with Ubuntu and Gnome Desktop.
I think Manjaro.
Arch is amazing; the newest software often works the best; the rolling model gives you new hardware support the fastest; the AUR has all the software ever (from Minecraft to the OldSchool RuneScape launcher to SVP for video interpolation); the Arch wiki is the most detailed resource ever.
But installing Arch is hard, and rarely there might be manual intervention. Enter Manjaro, which is easy to install, holds back packages for a very little while to preclude users dealing with any manual intervention breakage, has super-friendly forums (unlike Arch ;p), has an easy GUI for switching kernels and drivers, and yet has the AUR and can benefit from Arch wiki since it's Arch-based.
In my experience, both Arch and Manjaro have worked way better than Ubuntu (in terms of freezes and crashes), which seemed deeply paradoxical.
>I am unimpressed with Ubuntu and Gnome Desktop.
GNOME 3 is a laptop/tablet DE. It barely supports multiple monitors, unlike KDE, which is foremost a *desktop* DE. Amazing panel (aka taskbar), multimonitoring is a first-class concern (you can game on one monitor while you have a panel with tray that can have your CPU/GPU temps and the time on the other), deeply customizable, and my favorite bit is that you can disable desktop composition in its settings. Why? Because that makes it buttery smooth, which extends to games. Try dragging a window around with your mouse on a >=120 Hz monitor with composition on and then off to see what I'm talking about. (And all you lose by turning it off is terminal transparency.) In GNOME, of course, doing this is impossible (or at least extremely esoteric; it's not a setting).
GNOME and KDE are the two most polished DEs out there, but only one of them is a desktop DE. :p
Arch is amazing; the newest software often works the best; the rolling model gives you new hardware support the fastest; the AUR has all the software ever (from Minecraft to the OldSchool RuneScape launcher to SVP for video interpolation); the Arch wiki is the most detailed resource ever.
But installing Arch is hard, and rarely there might be manual intervention. Enter Manjaro, which is easy to install, holds back packages for a very little while to preclude users dealing with any manual intervention breakage, has super-friendly forums (unlike Arch ;p), has an easy GUI for switching kernels and drivers, and yet has the AUR and can benefit from Arch wiki since it's Arch-based.
In my experience, both Arch and Manjaro have worked way better than Ubuntu (in terms of freezes and crashes), which seemed deeply paradoxical.
>I am unimpressed with Ubuntu and Gnome Desktop.
GNOME 3 is a laptop/tablet DE. It barely supports multiple monitors, unlike KDE, which is foremost a *desktop* DE. Amazing panel (aka taskbar), multimonitoring is a first-class concern (you can game on one monitor while you have a panel with tray that can have your CPU/GPU temps and the time on the other), deeply customizable, and my favorite bit is that you can disable desktop composition in its settings. Why? Because that makes it buttery smooth, which extends to games. Try dragging a window around with your mouse on a >=120 Hz monitor with composition on and then off to see what I'm talking about. (And all you lose by turning it off is terminal transparency.) In GNOME, of course, doing this is impossible (or at least extremely esoteric; it's not a setting).
GNOME and KDE are the two most polished DEs out there, but only one of them is a desktop DE. :p
Considering that systemd-oomd as shipped by Ubuntu is happy to start gunning processes when there are upwards of 10 GB of free memory on the system, I think those complaints are entirely justified. This is hardly the only instance of some systemd appendage presumptuously doing something obnoxious or destructive.
I would be curious if Ubuntu is doing anything on top of it, though. systemd's oom-killer has been enabled in Fedora for like a year now, IIRC, and no one on Fedora has complained about it killing their Firefox browser or their entire DE session.
If you search there are many complaints of Fedora killing applications under similar circumstances. Ubuntu sets the memory pressure limit to 50% and Fedora does the same for user processes. The systemd-oomd default is a less aggressive 60%.
To be clear the kernel is horrible at handling OOM situations and Ubuntu's configuration of systemd-oomd is more aggressive at deciding when to start killing things than the default systemd-oomd settings.
Guys guys guys....
Stop using Ubuntu.
If you like Ubuntu but value reliability you should be using Debian.
Stop using Ubuntu.
If you like Ubuntu but value reliability you should be using Debian.
I made this switch over the last year and can second the recommendation. If you like Ubuntu, you actually like Debian and just put up with Ubuntu.
> If you like Ubuntu, you actually like Debian and just put up with Ubuntu.
I came to the same conclusion as well.
I've been using Debian for 1+ years. This is the best decision I've made regarding an OS ever since I started using Ubuntu many years ago.
I came to the same conclusion as well.
I've been using Debian for 1+ years. This is the best decision I've made regarding an OS ever since I started using Ubuntu many years ago.
This is because systemd-oomd developers failed the https://www.linuxatemyram.com/ test. If your cache/buffers fill up your RAM, the daemon will go to town on your applications. Because it was checking Free and not Available!
It isn't fixed until systemd v251, released 7 days ago.
https://github.com/systemd/systemd/commit/030bc91cb98385904b...
It isn't fixed until systemd v251, released 7 days ago.
https://github.com/systemd/systemd/commit/030bc91cb98385904b...
I'm curious what actually uses up all the memory that gets consumed in a web browser these days. It's very interesting to me.
I worked on UI compositors in my early twenties or so and they can consume large amounts of RAM generally because backing layers aren't typically compressed. Though I think most people associate web bloat with JavaScript, I would assume these two comprise a majority of consumption, maybe with JavaScript being a distant second place, though I'm not confident about that.
I worked on UI compositors in my early twenties or so and they can consume large amounts of RAM generally because backing layers aren't typically compressed. Though I think most people associate web bloat with JavaScript, I would assume these two comprise a majority of consumption, maybe with JavaScript being a distant second place, though I'm not confident about that.
Perhaps if web browsers in general returned to that crazy, old idea that ram is not an infinite resource and that I as a user should be able to set a reasonable limit on the maximum amount of ram a single web page or extension should be allowed to consume this wouldn't be a problem.
Instead, a few errant lines of javascript on a web page or in an extension is all that is needed to cause a web browser to consume every byte of available ram on a system and drive it into swap. :-/
I really miss the 90s web sometimes.
Instead, a few errant lines of javascript on a web page or in an extension is all that is needed to cause a web browser to consume every byte of available ram on a system and drive it into swap. :-/
I really miss the 90s web sometimes.
The person posting the bug says their 32GB machine has 10's of GB free. Later, someone points out the systemd-oom's oomctl is counting FS cache as "used". (Search the reddit thread for 'free -h'.)
It sounds like the issue can be easily avoided by ensuring that the size of your FS + the memory you're running applications are using never exceeds 60% of usable RAM.
I had to check to make sure this wasn't posted on April 1st.
systemd is glorious, but I'm happily using a 90's-era init, FWIW.
It sounds like the issue can be easily avoided by ensuring that the size of your FS + the memory you're running applications are using never exceeds 60% of usable RAM.
I had to check to make sure this wasn't posted on April 1st.
systemd is glorious, but I'm happily using a 90's-era init, FWIW.
Sounds like they are using cgroups to enforce memory limits then? I seem to recall that being a problem with containers getting killed as well because FS cache counts against their memory limit.
The 90s web that loaded an ActiveX which would automatically screw up your whole computer? Popups with ads that move around and cannot be closed? BSOD after entering the wrong website? Half of the browser window filled with toolbars? It seems that rose-colored glasses are very present here.
Not at all rose colored, I merely... avoided surfing sites that did that?
[deleted]
One line from the relevant reported bug [1] that caught my eye was:
"The current behavior gives the impression that Ubuntu 22.04 is unreliable and unsafe to use which is a problem for an LTS release that many people might want to use for critical production context."
I think a lot of people make the same, understandable, but incorrect, assumption that an LTS release is more stable than a normal release.
In my experience, Ubuntu LTS releases are equally stable/unstable as non-LTS release - the only distinction is that it is supported for a longer period of time, and hence tends to acquire more stability over time. Recently released LTS versions are quite frequently messy, and quite often new features make their debut in LTS releases - sometimes disruptively.
[Edit: Turns out this is one such case - systemd-oomd made its Ubuntu debut in 22.04]
[1] https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/19721...
I think a lot of people make the same, understandable, but incorrect, assumption that an LTS release is more stable than a normal release.
In my experience, Ubuntu LTS releases are equally stable/unstable as non-LTS release - the only distinction is that it is supported for a longer period of time, and hence tends to acquire more stability over time. Recently released LTS versions are quite frequently messy, and quite often new features make their debut in LTS releases - sometimes disruptively.
[Edit: Turns out this is one such case - systemd-oomd made its Ubuntu debut in 22.04]
[1] https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/19721...
I'd say it's not a terrible assumption given the fact you said yourself that the LTS release does become more stable over time. If picking between LTS and non-LTS, for stability, LTS is certainly the better choice.
That being said, the real solution here for "critical production" is to not run on a release that just came out, period. There's a reason Ubuntu LTS doesn't actually let you upgrade to the next LTS until the .1 release comes out. People should be deploying 20.04 if they want stability, and then upgrading to 22.04.1 when it comes out. Our infra team is still upgrading from 18.04->20.04, they won't be looking at 22.04 for a few years yet.
That being said, the real solution here for "critical production" is to not run on a release that just came out, period. There's a reason Ubuntu LTS doesn't actually let you upgrade to the next LTS until the .1 release comes out. People should be deploying 20.04 if they want stability, and then upgrading to 22.04.1 when it comes out. Our infra team is still upgrading from 18.04->20.04, they won't be looking at 22.04 for a few years yet.
Why is this so difficult on Linux? I've used Windows since 3.11, but I've never had anywhere close to the infuriating OOM issues I experience on Linux.
It feels like the whole system is designed so that it can only do the least sensible thing when it runs out of memory.
It feels like the whole system is designed so that it can only do the least sensible thing when it runs out of memory.
Linux allows memory overcommitment and Windows does not. On Windows, when the system is out of memory, attempts to allocate more will fail. Each running application has to decide from there on out exactly what to do. On Linux, running processes can allocate more memory than is available, and the OOM condition is triggered when too much of that memory is actually used. Once that happens, the OS has to decide which process to kill in order to reclaim some memory.
This has always seemed so strange to me, where are all of these applications that allocate a bunch of memory and then never use it? Isn't this just a chicken-and-egg thing where linux allowed this and then programs started depending on it?
This rather humorous take is still as relevant today as it was in 2004 - https://lwn.net/Articles/104185/
Windows has the best handling for when you run out of memory. It almost never kills stuff, and has a reasonable swap by default.
Macos will just kill your latest app and throw an idiotic dialog box that's of no use now that the biggest offender has already been killed.
It seems Linux has similar issues.
Macos will just kill your latest app and throw an idiotic dialog box that's of no use now that the biggest offender has already been killed.
It seems Linux has similar issues.
Fun times. Until Win XP SP1, i think, if the memory is full windows will just crash.
Thanks to being a n00b programmer at the time, I ran out of memory on NT4 plenty of times and it handled that more gracefully than my Linux desktop today. Whatever you're talking about must have been some specific issue.
Sounds more like a general memory allocation bug in some patch level of Windows XP SP1 than a change in OOM behavior.
Ah ah, same usual shit of systemd wanting to be this monolith that takes control of all of your linux system (login, home dir, oom, ...), with strong opinions and low quality code. And as usual, you introduce it and it starts to break for users.
If only it was the first time...
If only it was the first time...
Except systemd is almost the exact opposite of a monolith.
Systemd runs just fine even without systemd-oomd and can probably be disabled by a command as simple as:
> systemctl disable --now systemd-oomd
and doesn’t need to be installed in the first place.
> systemctl disable --now systemd-oomd
and doesn’t need to be installed in the first place.
It's not not-a-monolith just because you can disable or remove an ever increasing number of "tested"/supported functionalities and move further and further away from the supported configuration. This might be OK for personal or toy systems, but it's an immediate pass for anybody who cares about their time.
This comes up every time the systemd discussion happens and it's a little absurd that people need constant reminders that replacing a decent chunk of a base Linux install piecemeal is not viable.
This comes up every time the systemd discussion happens and it's a little absurd that people need constant reminders that replacing a decent chunk of a base Linux install piecemeal is not viable.
If this was chrome, wouldn't the OOM kill only kill the offending tab? Why can't firefox handle it this way too?
So this happened on a system that serves up user provided files; clam(d?) was doing some ram intensive things and oomd decided to kill it without much ceremony, this workload while ram tight worked without issue for 10+ years but this also could be a Ubuntu requiring more ram usage compared to previous versions. If I didn't notice this the end result could have been a bit rough.
This is not new for anyone who had used Ubuntu with GNOME on low RAM machines (<2GB). earlyoom is the answer.
I upgraded to 22.04 (with GNOME) in my 2GB laptop and still is very usable.
I upgraded to 22.04 (with GNOME) in my 2GB laptop and still is very usable.
Happened to me as well. I think it would be OK but only with a warning - since mostly it is obvious that an application start's to consume more and more ram.