Systemd 248 RC3: systemd-oomd is now considered fully supported(github.com)
github.com
Systemd 248 RC3: systemd-oomd is now considered fully supported
https://github.com/systemd/systemd/blob/main/NEWS
129 comments
That change log is HUGE and it’s the main reason we don’t use systemd. Creeping featurism isn’t good in your OS when you need things to be consistent and work the same every time. Systemd has a lot of state.
I'm very happily using systemd since a bit less than a decade now. It's fantastic to have a common standardized way to manage your services and jobs that doesn't rely on a fragile set of scripts, and unified logs that can be easily queried and exported to a machine readable format.
I've never had an issue with the shell scripts that run my systems, and I consider it a strength that they can be easily modified. Why do people always seem to make 'no more fragile scripts' their first point in favor of systemd?
Because sh and bash are horror shows. Maintaining and creating new service init scripts is a real pain and difficult to do with consistency. Bash scripts can do anything, meaning they have to be audited one by one, every single time they are updated. Every single distribution has its own way to run services via scripts. And that can change between upgrades.
In the systemd world, I can audit a unit file via a simple `systemctl cat <name>`, and edit it directly via `systemctl edit <name>`. I can easily see services dependencies, I can easily change the user running the service, the workdir, etc. There is a standard interface to start, stop, restart, enable at startup. All of this from one system that is consistent and the same between Linux systems.
But really the best features IMHO is journald.
In the systemd world, I can audit a unit file via a simple `systemctl cat <name>`, and edit it directly via `systemctl edit <name>`. I can easily see services dependencies, I can easily change the user running the service, the workdir, etc. There is a standard interface to start, stop, restart, enable at startup. All of this from one system that is consistent and the same between Linux systems.
But really the best features IMHO is journald.
sh isn't a horror show, it's a POSIX standard and it's pretty easy to learn. Does systemd conform to any known standard? Of course not, they're making it up as they go. When's it going to be finished? Nobody knows. You simply can't rely on it.
The core primitives of systemd units haven't changed in a long while. You can always rely on it being able to start your service and stop it. What might not work depending on what version of systemd you ship is additional parameters (such as requiring a path to exist outside mountpoints or setting resource limits).
Well, thanks for telling me how you see it. I can't say I agree with almost anything you've said, but it's definitely informative.
My perspective is that on the one hand, systemd is very 'cerimonial' in how it goes about things, while shell-based systems have the potential to be extremely well done. My go-to example is the OpenBSD init script - I once tried to boot OBSD over the network, and mount it on a RAM filesytem, and was able to simply open that boot script and change the mount commands. Ever since, I've been impressed by the clarity and simplicity of that shell script (in its original form, not after I was done with it (: ).
> Because sh and bash are horror shows.
I disagree. The syntax is obtuse, but it is at this point very well understood, and the rest of the system runs on it. Better to use fewer languages throughout, than more. Are you referring to something other than the syntax?
> Maintaining and creating new service init scripts is a real pain and difficult to do with consistency.
I'm not sure, but it sounds like you're talking about systemd services. If you are, I agree; multiple files with layers of cerimony are hard to compare to an extra line of shell core in an rc.local. If you're talking about non-systemd, I've never encountered this, and certainly found the process far less painful than the systemd way.
> Bash scripts can do anything, meaning they have to be audited one by one, every single time they are updated.
Just diff the scripts...not that hard. Just have to check the changes. That's no harder than any other program you care about.
> Every single distribution has its own way to run services via scripts.
vive la différence! No point to the stagnation caused by homogeniety; I don't see why I should expect debian services to run the same as FreeBSD.
> And that can change between upgrades.
Systemd is notorious for changing things. One great example is when they tried to change how processes can persist after the user logs out. I don't think you have a leg to stand on here.
In the systemd world, I can audit a unit file via a simple `systemctl cat <name>`, and edit it directly via `systemctl edit <name>`.
I can too: `cat /etc/rc.local` and `vim /etc/rc.local`. Added benefit, I'm using the same tools and interfaces that I do for the rest of my system administration.
> I can easily see services dependencies, I can easily change the user running the service, the workdir, etc.
Service dependencies are the strong point of systemd, I will give you that. However, `su` can change the user running the command, and `cd` is this awesome little tool I found for changing the directory I'm in. No extra benefit there, just new commands for doing the same old things.
> There is a standard interface to start, stop, restart, enable at startup.
That standard interface is only useful because the underlying complexity has increased so much that `systemctl start` has become a necessary abstraction to cover it up. Your standard interface is medicine for an ailment I do not have.
> All of this from one system that is consistent and the same between Linux systems.
Why is that beneficial? It just means that you've put all the innovation in the hands of the systemd devs. I'm over on systems like void, where my startup times are faster, my base system load is a third, and I can get my hands dirty in the system internals without googling for strange systemd incantations.
> But really the best features IMHO is journald.
You're going to have to spell that out, because I really like being able to use a proliferation of text-oriented tools to manage my logs. I'm not a fan of binary, no matter the supposed speed improvements.
My perspective is that on the one hand, systemd is very 'cerimonial' in how it goes about things, while shell-based systems have the potential to be extremely well done. My go-to example is the OpenBSD init script - I once tried to boot OBSD over the network, and mount it on a RAM filesytem, and was able to simply open that boot script and change the mount commands. Ever since, I've been impressed by the clarity and simplicity of that shell script (in its original form, not after I was done with it (: ).
> Because sh and bash are horror shows.
I disagree. The syntax is obtuse, but it is at this point very well understood, and the rest of the system runs on it. Better to use fewer languages throughout, than more. Are you referring to something other than the syntax?
> Maintaining and creating new service init scripts is a real pain and difficult to do with consistency.
I'm not sure, but it sounds like you're talking about systemd services. If you are, I agree; multiple files with layers of cerimony are hard to compare to an extra line of shell core in an rc.local. If you're talking about non-systemd, I've never encountered this, and certainly found the process far less painful than the systemd way.
> Bash scripts can do anything, meaning they have to be audited one by one, every single time they are updated.
Just diff the scripts...not that hard. Just have to check the changes. That's no harder than any other program you care about.
> Every single distribution has its own way to run services via scripts.
vive la différence! No point to the stagnation caused by homogeniety; I don't see why I should expect debian services to run the same as FreeBSD.
> And that can change between upgrades.
Systemd is notorious for changing things. One great example is when they tried to change how processes can persist after the user logs out. I don't think you have a leg to stand on here.
In the systemd world, I can audit a unit file via a simple `systemctl cat <name>`, and edit it directly via `systemctl edit <name>`.
I can too: `cat /etc/rc.local` and `vim /etc/rc.local`. Added benefit, I'm using the same tools and interfaces that I do for the rest of my system administration.
> I can easily see services dependencies, I can easily change the user running the service, the workdir, etc.
Service dependencies are the strong point of systemd, I will give you that. However, `su` can change the user running the command, and `cd` is this awesome little tool I found for changing the directory I'm in. No extra benefit there, just new commands for doing the same old things.
> There is a standard interface to start, stop, restart, enable at startup.
That standard interface is only useful because the underlying complexity has increased so much that `systemctl start` has become a necessary abstraction to cover it up. Your standard interface is medicine for an ailment I do not have.
> All of this from one system that is consistent and the same between Linux systems.
Why is that beneficial? It just means that you've put all the innovation in the hands of the systemd devs. I'm over on systems like void, where my startup times are faster, my base system load is a third, and I can get my hands dirty in the system internals without googling for strange systemd incantations.
> But really the best features IMHO is journald.
You're going to have to spell that out, because I really like being able to use a proliferation of text-oriented tools to manage my logs. I'm not a fan of binary, no matter the supposed speed improvements.
Wait until you see the Linux kernel changelogs...
Who is we? I use systemd on the operating systems I use (debian, centos and opensuse)
Our organization. It doesn’t make sense to rely on software which grows to consume all available niches. Systemd is a perfect example of this kind of software. I don’t wish to have to ascertain what new features it’s reimplemented, sometimes in ways which are not POSIX compliant, every time there’s an update.
Do I guess correctly that your organization doesn't value clean services initialization, solid services dependencies management and reliable log handling?
Well, there must be room for such organizations too. It's a colorful world after all.
Well, there must be room for such organizations too. It's a colorful world after all.
No, we get all of that with time-worn traditional methods which work well, are documented properly, and conform to standards. I’m not interested in beta testing Red Hat’s replacement for Linux.
Am I the only one excited about support for luks via TPM?
Systemd is such a wonderful example of "if I didn't develop it, it isn't good enough."
> A concept of system extension images is introduced. Such images may be used to extend the /usr/ and /opt/ directory hierarchies at runtime with additional files (even if the file system is read-only). When a system extension image is activated, its /usr/ and /opt/ hierarchies and os-release information are combined via overlayfs with the file system hierarchy of the host OS.
Oh, what fresh hell is this?
Oh, what fresh hell is this?
It's basically the ostree idea on a smaller scale. Did you ever want to install some proprietary software with access to system libraries/config, but without actually polluting the global namespace? It's this. Like running a docker service with host as your base layer. (Minus other namespaces)
Honestly, it seems great and not very tricky for a system which already manages namespaces with private mounts.
Honestly, it seems great and not very tricky for a system which already manages namespaces with private mounts.
There's a pretty big difference between "ghosting" /opt, /srv, /usr/local/ - and munging all/any of /usr?
Not sure what you mean either by ghosting or by munging to be honest...
Munging - messing with. Per the (somewhat abandoned)standard for filesystem layout /usr (except /usr/local) is the domainof the system/distribution. It's not very nice for a package to stomp on that. Although systemd already claim some authority over logs via journalctl of course.
Ghosting: by bind-mounting over the system fs -you might introduce a bug/condition that doesn't show up in a recovery boot scenario, or forensic analyzis of the filesystem/disk - you could introduce "ghost" files/artefacts.
Ghosting: by bind-mounting over the system fs -you might introduce a bug/condition that doesn't show up in a recovery boot scenario, or forensic analyzis of the filesystem/disk - you could introduce "ghost" files/artefacts.
> It's not very nice for a package to stomp on that.
Yes, sure. That doesn't stop hundreds of vendors with custom install scripts who will use /usr and /opt which is the current state of things. Systemd actually provides a workaround here which stops that package from changing those hierarchies globally, which... seems like something you'd want?
> you might introduce a bug/condition that doesn't show up in a recovery boot scenario
Why would you try to run anything but recovery tools in a recovery boot scenario?
> or forensic analyzis of the filesystem/disk
You can reproduce the same environment using the same overlay configuration - that setup is not hidden from you. It's not any more confusing in forensic than a docker service and we deal with them (including bind-mounts) just fine.
Yes, sure. That doesn't stop hundreds of vendors with custom install scripts who will use /usr and /opt which is the current state of things. Systemd actually provides a workaround here which stops that package from changing those hierarchies globally, which... seems like something you'd want?
> you might introduce a bug/condition that doesn't show up in a recovery boot scenario
Why would you try to run anything but recovery tools in a recovery boot scenario?
> or forensic analyzis of the filesystem/disk
You can reproduce the same environment using the same overlay configuration - that setup is not hidden from you. It's not any more confusing in forensic than a docker service and we deal with them (including bind-mounts) just fine.
Just so we’re clear, unless I’m mistaken this overlay only affects the target software. It’s not allowing said extensions to trample the “root” file system. The altered fs only applies to a limited namespace.
Just because it comes from systemd doesn’t mean it’s a bad idea. OpenWRT and other embedded Linux systems use overlay file systems very effectively.
It's a fine concept. Good for replacing stuff like appimage besides the packaging part?
Mixing this with OS package management will require some consideration.
So they are using overlayfs, which means that "upper" directories have visibility precedence on file name conflict. Which means that some auditing facility should be required that alerts if an upper sys image masks a lower one, or if an OS package some time later installs a file to the "lower" directory, perhaps the original rootfs, but a sysfs package still takes precedence…
Mixing this with OS package management will require some consideration.
So they are using overlayfs, which means that "upper" directories have visibility precedence on file name conflict. Which means that some auditing facility should be required that alerts if an upper sys image masks a lower one, or if an OS package some time later installs a file to the "lower" directory, perhaps the original rootfs, but a sysfs package still takes precedence…
To be fair overlay fs is a good way to manage network boot / nfs root systems, live cds... All sorts of things where I want an underlying fs that clients can't mess with but they still need to think they can write to it
Overlays are fine. I’ve used them for all sorts of stuff.
Tying them to pid 1, and forcing them to be used all the time for some hacked up thing where /usr is different depending on which process is looking at it? Not so much.
Tying them to pid 1, and forcing them to be used all the time for some hacked up thing where /usr is different depending on which process is looking at it? Not so much.
It’s not a hacked thing, it’s filesystem namespaces exactly as is done in Docker so that a container process doesn’t mess up its image.
It’s no more privilege than pid 1 had before because of .mount units.
It’s no more privilege than pid 1 had before because of .mount units.
> forcing them to be used all the time
You know this doesn't do anything unless explicitly configured in the system and/or unit configuration?
You know this doesn't do anything unless explicitly configured in the system and/or unit configuration?
There's actually a directly analogous case in the gaming world, of all things: the app Mod Organizer 2 uses a runtime virtual file system There's actually a directly analogous case in the gaming world, of all things: the app Mod Organizer 2 uses a runtime-only virtual file system to layer together the files from different mod packages, making it easy to adjust for and dynamically disable/enable overlapping mod resource files (of which there can be a huge number in games like Skyrim), as well as redirecting output files from mods to a sane holding area instead of letting them pollute your game installs.
Soon it'll include a browser and email client. Can't wait!
The issue here is branding, not engineering or design.
Systemd is an umbrella under which many key userspace tools are being developed. The actual init system itself is a lot bigger than the old SysV Init system was, but it is separate than all these other things that also say Systemd on them.
The best analogy is GNU. You don't complain that GNU contains a compiler, and also a text editor, and also C System Library do you? GNU is a project that develops a load of components that play well with each other. Systemd is the same.
Systemd is an umbrella under which many key userspace tools are being developed. The actual init system itself is a lot bigger than the old SysV Init system was, but it is separate than all these other things that also say Systemd on them.
The best analogy is GNU. You don't complain that GNU contains a compiler, and also a text editor, and also C System Library do you? GNU is a project that develops a load of components that play well with each other. Systemd is the same.
GNU/Linux will soon be replaced by Systemd/Linux
I actually seriously foresee a split occurring in the Linux community that will also affect users of other Unix-like operating systems. On one side you have modern Linux software projects like systemd and Wayland. These projects address deficiencies in traditional solutions, but certain design choices in these projects have not been well-received by more traditional users of Linux and the BSDs.
The increase in the amount of user-level software that depends on systemd and Wayland is a forcing factor. For example, my understanding is that official distributions of GNOME depend on systemd, and many Linux desktops are being actively developed for Wayland (though I don't think any major desktop environment or toolkit has abandoned X11). I expect this trend to continue, given that systemd is part of the most popular Linux distributions, Wayland will be default in Ubuntu (https://www.phoronix.com/scan.php?page=news_item&px=Ubuntu-2...), and GNOME runs on Wayland (with KDE's port to Wayland being actively developed: see https://www.phoronix.com/scan.php?page=news_item&px=KDE-2021...).
So, yes, I do see a split occurring between "modern Linux" (with systemd and Wayland) and "traditional Linux," with the BSDs fitting in closer to the "traditional Linux" camp.
The increase in the amount of user-level software that depends on systemd and Wayland is a forcing factor. For example, my understanding is that official distributions of GNOME depend on systemd, and many Linux desktops are being actively developed for Wayland (though I don't think any major desktop environment or toolkit has abandoned X11). I expect this trend to continue, given that systemd is part of the most popular Linux distributions, Wayland will be default in Ubuntu (https://www.phoronix.com/scan.php?page=news_item&px=Ubuntu-2...), and GNOME runs on Wayland (with KDE's port to Wayland being actively developed: see https://www.phoronix.com/scan.php?page=news_item&px=KDE-2021...).
So, yes, I do see a split occurring between "modern Linux" (with systemd and Wayland) and "traditional Linux," with the BSDs fitting in closer to the "traditional Linux" camp.
The split has already happened - Void and Alpine Linux seem to be on the traditional Linux side, everything with the musl libc in general.
Its the death of the meme of the "universal OS".
There will be Linux for desktops to run a web browser (aka poorly emulate a chromebook) and for docker and docker only.
If you want a non-containerized server or some kind of non-desktop embedded thingy, or anything that can be computed other than emulating a chromebook and doing containers, you'll have to run FreeBSD or some other OS.
The days of "just install Debian to do anything anywhere" are gone, which is sad, because there's really no technical reason or need for the destruction.
There will be Linux for desktops to run a web browser (aka poorly emulate a chromebook) and for docker and docker only.
If you want a non-containerized server or some kind of non-desktop embedded thingy, or anything that can be computed other than emulating a chromebook and doing containers, you'll have to run FreeBSD or some other OS.
The days of "just install Debian to do anything anywhere" are gone, which is sad, because there's really no technical reason or need for the destruction.
Gnome works without systemd under OpenBSD.
Actually, I don't think something like busybox included in systemd is necessarily bad. You could have a system recoverable even if everything else breaks. Of course, it would really enable systemd/linux.
It doesnt need to be included in systemd for that.
Most of what you would need to cobble together a busybox like binary is there in the systemd/src/basic directory. stat-util, procfs-util, sort-util, path-util, and so on.
OpenBSD has a ramdisk for that.
Shhhh. Don't give them ideas.
Including busybox in the initrd or whatever it's called now is probably a good idea. Having systemd developers write a new interpretation of that, rehashing old bugs is not great.
Including busybox in the initrd or whatever it's called now is probably a good idea. Having systemd developers write a new interpretation of that, rehashing old bugs is not great.
You meme, but they're eating so many userspace components that this is actually where I see them heading.
Just wait until ls has to have hooks into dbus to do its job.
Just wait until ls has to have hooks into dbus to do its job.
You mean GNU/Systemd.
No a Linux/Systemd,
which uses LLVM for building Linux and Systemd, replaces all core utils with non GNU versions. It use a non GNU libc and so on...
But that aside people often confuse/mixup the systemd service manager and the systemd project which bundles a wide variety of programs which can be used with systemd.
The reasons for this is because systemd itself confuses this. The change log is not split by sub-projects and the sub-projects have sometimes no clean boundary between them and the systemd service manager.
I think it would be nice if systemd changes in three ways:
1. Have more clean boundaries between the core service manager and it's sub-project.
2. Have a recurring periods where new features are blocked from being merged and development focused on cleaning up (e.g. fixing bugs). (I can recommend that to any projects, very few do so).
3. Get "outside" feedback for certain mainly UX aspects, some parts of systemd are out right counter intuitive unless you are not very familiar with it. A thinks which is very easy to happen to any project, but still could be improved on (and maybe already is improved on).
Oh and there where a very few security WTF are you doing moments, I hope there will be no more such moments in the future as systemd on itself it quite useful for Linux, especially desktop Linux.
which uses LLVM for building Linux and Systemd, replaces all core utils with non GNU versions. It use a non GNU libc and so on...
But that aside people often confuse/mixup the systemd service manager and the systemd project which bundles a wide variety of programs which can be used with systemd.
The reasons for this is because systemd itself confuses this. The change log is not split by sub-projects and the sub-projects have sometimes no clean boundary between them and the systemd service manager.
I think it would be nice if systemd changes in three ways:
1. Have more clean boundaries between the core service manager and it's sub-project.
2. Have a recurring periods where new features are blocked from being merged and development focused on cleaning up (e.g. fixing bugs). (I can recommend that to any projects, very few do so).
3. Get "outside" feedback for certain mainly UX aspects, some parts of systemd are out right counter intuitive unless you are not very familiar with it. A thinks which is very easy to happen to any project, but still could be improved on (and maybe already is improved on).
Oh and there where a very few security WTF are you doing moments, I hope there will be no more such moments in the future as systemd on itself it quite useful for Linux, especially desktop Linux.
Every time I see all the stuff packed into it I feel extreme urge to switch to freebsd again.
Oomd, oh yea, good idea, but WHY it has to be in systemd? The same applies to about 3/4 of things in there. Because of this i've started to call it LennartOS. And also I don't feel like linux is about freedom anymore but more like some sort of weird power display games.
Oomd, oh yea, good idea, but WHY it has to be in systemd? The same applies to about 3/4 of things in there. Because of this i've started to call it LennartOS. And also I don't feel like linux is about freedom anymore but more like some sort of weird power display games.
You should switch to FreeBSD, because imho it’s great.
However, you should probably switch to FreeBSD because you think it’s great too, not because you hate systemd.
However, you should probably switch to FreeBSD because you think it’s great too, not because you hate systemd.
Yes, please. Switch to FreeBSD. Every FreeBSD developer knows what a terrible idea SystemD is.
https://papers.freebsd.org/2018/bsdcan/rice-the_tragedy_of_s...
Joke's on you.
https://papers.freebsd.org/2018/bsdcan/rice-the_tragedy_of_s...
Joke's on you.
You're concerned about a project gaining too many features, so you… move to FreeBSD, an OS where everything is developed as one project?
Come to Gentoo! I've been booting with OpenRC since forever have have completely ignored the systemd drama. The only thing eating my userspace is me.
I can second this, recent Gentoo has been a pleasure to work with.
You can use Gnome 3 without systemd. Biggest plus point about Gentoo. Love it to bits even if it gives me headaches sometimes !
This is a common mistake. systemd-oomd is not "in" systemd the init system. Similar to systemd-resolved, systemd-networkd, systemd-boot, and others it just has the systemd name because it is developed by the systemd team. A minimal system using sytemd for the init system and other software for everything else only needs sytemd, dbus, and journald.
I use several of these projects because I like the systemd design philosophy (ex the way the CLI interface and config files are) and trust the developers of their software. However, there are popular altenatives available for all of these components, should you choose to use them:
systemd-resolved: bind9, dnsmasq, powerdns
systemd-networkd: NetworkManager, ConnMan
systemd-boot: GRUB2, Refind, EFISTUB
systemd-oomd: earlyoom, nohang, oomd (the facebook one, not systemd)
I use several of these projects because I like the systemd design philosophy (ex the way the CLI interface and config files are) and trust the developers of their software. However, there are popular altenatives available for all of these components, should you choose to use them:
systemd-resolved: bind9, dnsmasq, powerdns
systemd-networkd: NetworkManager, ConnMan
systemd-boot: GRUB2, Refind, EFISTUB
systemd-oomd: earlyoom, nohang, oomd (the facebook one, not systemd)
hexo(4)
I agree but I wouldn't call systemd a init system.
It's a service manager for your Linux OS which manage all services down to initial startup related services.
Just viewing it as a init system is misleading.
As a side note I think a service manager directly above the kernel is a must have for any modern server or desktop operating system ;=)
It's a service manager for your Linux OS which manage all services down to initial startup related services.
Just viewing it as a init system is misleading.
As a side note I think a service manager directly above the kernel is a must have for any modern server or desktop operating system ;=)
> I agree but I wouldn't call systemd a init system.
Systemd is the name of the init system. Systemd is also the name of the project developing multiple systemd-* services. Maybe they could help the situation a bit by renaming the service itself systemd-init, but we've had enough time for people to learn to deal with a tiny bit of ambiguity...
Systemd is the name of the init system. Systemd is also the name of the project developing multiple systemd-* services. Maybe they could help the situation a bit by renaming the service itself systemd-init, but we've had enough time for people to learn to deal with a tiny bit of ambiguity...
> but WHY it has to be in systemd?
«It (systemd) provides a system and service manager» (from https://www.freedesktop.org/wiki/Software/systemd/).
So it manages services and manages the system, as a whole.
> Every time I see all the stuff packed into it I feel extreme urge to switch to freebsd again.
Meh, a lot of people bring this up, but they never switch.
«It (systemd) provides a system and service manager» (from https://www.freedesktop.org/wiki/Software/systemd/).
So it manages services and manages the system, as a whole.
> Every time I see all the stuff packed into it I feel extreme urge to switch to freebsd again.
Meh, a lot of people bring this up, but they never switch.
> LennartOS
Lennax
Lennax
FreeBSD includes a kernel, drivers, and userland utilities under one project. If your complaint is that systemd is "bloated", it doesn't make any sense why you would want to switch to a system that's even more "bloated".
systemd-oomd wasn't even developed by Lennart, Anita Zhang of Facebook contributed it.
Try understanding a bit more than just the name's prefix before getting so bent out of shape.
Try understanding a bit more than just the name's prefix before getting so bent out of shape.
Linux oom behavior always has had complaints[0]. Hope it now finally fixes the desktop case[1].
[0] https://lwn.net/Articles/104185/
[1] https://lkml.org/lkml/2019/8/4/15
[0] https://lwn.net/Articles/104185/
[1] https://lkml.org/lkml/2019/8/4/15
For reference, [1] describes the SSD near-OOM thrash problem, where the speed of refaulting file pages is so high that the kernel detects enough activity and doesn't trigger OOM.
It's a heck of a problem to deal with, and a OOM killer will go part of the way to fixing it, but I still have two complaints about the thrash problem.
1. In similar situations on other operating systems, I don't usually see the GUI freeze to an unusable state. In Linux hitting a thrash situation will freeze the computer to the point where Ctrl-Alt-F2 will not switch the vterm even when waiting several minutes, and it has to be rebooted. I suppose it's possible that I just haven't used other OSes enough in a long time and they have this too, but this seems like a solvable problem. I assume it's not a CPU issue, since the scheduler should handle that just fine. So why not have a mechanism by which the user's desktop environment and init / login system can reserve itself enough memory to always be responsive under any memory condition?
2. Browsers. The state of browser memory management is atrocious. Just about every time I've run out of memory (other than compiling some complex software with -O3) it's been because my browser is hogging a huge chunk of it (even if the OOM killer blames something else). Now, I understand that in theory unused memory is wasted memory. So a browser using a ton of memory can be a good thing. But this is only true when memory the browser is holding but not using can be reclaimed to be used by another program, and this seems to basically never happen. If I close and reopen the browser (with the tabs automatically restored) the memory usage drops by 80% or more; the issue is that the browser simply never suspends tabs I haven't used in days or weeks. Modern browsers are not good "desktop citizens", if you will. They hog memory to the point where opening anything new will create a pressure stall.
I suspect solving these two issues would largely fix thrashing for most desktop users, without the need to ever kill anything, which is obviously undesirable.
Feel free to tell me if one or both of these is incorrect or impossible. These are just my observations, I'm not a kernel dev or anything close to one.
It's a heck of a problem to deal with, and a OOM killer will go part of the way to fixing it, but I still have two complaints about the thrash problem.
1. In similar situations on other operating systems, I don't usually see the GUI freeze to an unusable state. In Linux hitting a thrash situation will freeze the computer to the point where Ctrl-Alt-F2 will not switch the vterm even when waiting several minutes, and it has to be rebooted. I suppose it's possible that I just haven't used other OSes enough in a long time and they have this too, but this seems like a solvable problem. I assume it's not a CPU issue, since the scheduler should handle that just fine. So why not have a mechanism by which the user's desktop environment and init / login system can reserve itself enough memory to always be responsive under any memory condition?
2. Browsers. The state of browser memory management is atrocious. Just about every time I've run out of memory (other than compiling some complex software with -O3) it's been because my browser is hogging a huge chunk of it (even if the OOM killer blames something else). Now, I understand that in theory unused memory is wasted memory. So a browser using a ton of memory can be a good thing. But this is only true when memory the browser is holding but not using can be reclaimed to be used by another program, and this seems to basically never happen. If I close and reopen the browser (with the tabs automatically restored) the memory usage drops by 80% or more; the issue is that the browser simply never suspends tabs I haven't used in days or weeks. Modern browsers are not good "desktop citizens", if you will. They hog memory to the point where opening anything new will create a pressure stall.
I suspect solving these two issues would largely fix thrashing for most desktop users, without the need to ever kill anything, which is obviously undesirable.
Feel free to tell me if one or both of these is incorrect or impossible. These are just my observations, I'm not a kernel dev or anything close to one.
>In Linux hitting a thrash situation will freeze the computer to the point where Ctrl-Alt-F2 will not switch the vterm even when waiting several minutes, and it has to be rebooted.
You can recover system from this state by manually triggering OOM killer using Magic SysRq keys. (Alt+SysRq+F, provided you have SysRq enabled in /proc/sys/kernel/sysrq)
You can recover system from this state by manually triggering OOM killer using Magic SysRq keys. (Alt+SysRq+F, provided you have SysRq enabled in /proc/sys/kernel/sysrq)
Is there a version of this that works on typical laptop keyboards that don't have SysRq? I could map an unused key to it with xmodmap of course, but if my X server isn't responding I don't think that would help very much!
SysRq is PrintScreen, I think it should be present even on laptops?
Oh! News to me, I definitely have it. Thanks!
This is very useful! I have to look up this combination every other month when one of my systems starts thrashing.
How much memory is enough for 1? Back on Fedora 32, on a system with close to a year of uptime (it was attached to my TV), gnome-shell consumed nearly 20GB of RAM on a 32GB system! And I rarely interacted with it (outside of watching the occasional concert on it). Most of the time was spent on the lockscreen, with the TV on a different input.
Perhaps this falls outside the category of "any memory condition"... :)
Though, it was responsive. :D
Needless to say, I think 1 is harder than expected. Fedora has made some recent changes on the OOM killer front though, perhaps those are closer to what you're interested in.
Perhaps this falls outside the category of "any memory condition"... :)
Though, it was responsive. :D
Needless to say, I think 1 is harder than expected. Fedora has made some recent changes on the OOM killer front though, perhaps those are closer to what you're interested in.
Control groups are your friends. Run your browser in a separate control group with a constrained memory. It took me discovering memory leaks from qemu running X over virgl machines to actually spend a weekend figuring it out, but it is doable.
* I run separate cgroups for admin SSH with (1G min, 4G max ) - worst case scenario I can ssh into the desktop and kill offending processes.
* Separate cgroup for two browsers ( 8G each max ).
* Separate cgroup per qemu VM ( VM max ram + 512M each max ).
After implementing that I went from periodic lock ups to a quick browser crash and a restart maybe once a week.
* I run separate cgroups for admin SSH with (1G min, 4G max ) - worst case scenario I can ssh into the desktop and kill offending processes.
* Separate cgroup for two browsers ( 8G each max ).
* Separate cgroup per qemu VM ( VM max ram + 512M each max ).
After implementing that I went from periodic lock ups to a quick browser crash and a restart maybe once a week.
Yesterday I ran into this nice article on using cgroups to contain firefox memory usage, https://dev.to/msugakov/taking-firefox-memory-usage-under-co...
> the issue is that the browser simply never suspends tabs I haven't used in days or weeks. Modern browsers are not good "desktop citizens", if you will. They hog memory to the point where opening anything new will create a pressure stall.
The "Auto Tab Discard" extension[1] for Firefox unloads tab contents, keeping only the title and favicon. Tabs can be put in an allowlist to prevent discarding. I don't know if Chrome or Safari have something similar.
[1] https://addons.mozilla.org/en-US/firefox/addon/auto-tab-disc...
The "Auto Tab Discard" extension[1] for Firefox unloads tab contents, keeping only the title and favicon. Tabs can be put in an allowlist to prevent discarding. I don't know if Chrome or Safari have something similar.
[1] https://addons.mozilla.org/en-US/firefox/addon/auto-tab-disc...
For browser, reduce the number of parallel processes in preferences and try to use Auto Tab Discard plugin: https://www.youtube.com/watch?v=S0rHU38OnTE
> 2. Browsers. The state of browser memory management is atrocious. Just about every time I've run out of memory (other than compiling some complex software with -O3) it's been because my browser is hogging a huge chunk of it
> Modern browsers are not good "desktop citizens", if you will. They hog memory to the point where opening anything new will create a pressure stall.
I went the ballistic route: most of my browsers run inside local Docker containers, on which I put CPU and RAM quota.
It's my machine: I decide how "citizens" behave : )
> Modern browsers are not good "desktop citizens", if you will. They hog memory to the point where opening anything new will create a pressure stall.
I went the ballistic route: most of my browsers run inside local Docker containers, on which I put CPU and RAM quota.
It's my machine: I decide how "citizens" behave : )
If you're using systemd, drop docker and start your browser through something like this:
You can put resource limits on basically anything (see systemd.resource-control(5)).
#!/bin/sh
systemd-run \
--user \
--scope \
--property=MemoryHigh=4G \
--slice=background \
/usr/bin/firefox "$@"
MemoryHigh=bytes
Specify the throttling limit on memory usage of the executed processes in this unit. Memory usage may go above the limit if unavoidable, but the processes are heavily slowed down and memory is taken away aggressively in such cases. This is the main mechanism to control memory usage of a unit.
MemoryMax=bytes
Specify the absolute limit on memory usage of the executed processes in this unit. If memory usage cannot be contained under the limit, out-of-memory killer is invoked inside the unit. It is recommended to use MemoryHigh= as the main control mechanism and use MemoryMax= as the last line of defense.
You can put resource limits on basically anything (see systemd.resource-control(5)).
Docker is pretty overkill for this; you can just create a new cgroup with a maximum memory, CPU, etc.
OpenBSD has login.conf
https://man.openbsd.org/login.conf.5
https://man.openbsd.org/login.conf.5
Speaking of which, Firefox behaviour has improved on OpenBSD significantly recently. It used to either crash or act weird when it hit the memory limit in login.conf (which it would eventually do). Now it just works. No idea what changed.
This is what ulimit is for.
Regarding the file pages this is the solution https://github.com/hakavlad/le9-patch
> So why not have a mechanism by which the user's desktop environment and init / login system can reserve itself enough memory to always be responsive under any memory condition?
This! Always wanted a /proc/$PID flag to flag process' pages never to go to swap and reserve a number of cache pages specifically for the process. Something similar to mlock.
This! Always wanted a /proc/$PID flag to flag process' pages never to go to swap and reserve a number of cache pages specifically for the process. Something similar to mlock.
> So why not have a mechanism by which the user's desktop environment and init / login system can reserve itself enough memory to always be responsive under any memory condition?
Because it's not the memory shortage that's nailing the UI. It's disk contention. When pathological swap activity causes disk thrashing, every process that needs to use the disk is now waiting in a very long queue with all the swap page in/out requests and every other process that is now backed up.
If you have a metrics agent that emits a handful of metrics on a per second granularity, you can see a bunch of things happen:
* Memory rises
* Some pages get swapped out
* Even if the system has no other processes writing to disk, you'll see the swap-out correlating with disk writes
* Memory rises some more
* You see more pages get swapped out
* (...and see more disk writes)
* Memory rises some more
* You see pages get swapped out and in, because now we're shuffling out pages that are imminently required by another running process.
* Disk queue lengths rise
* One by one, all processes are waiting (driving up the CPU "wa" number, and causing load averages to shoot up)
This is why, if you're oncall and you're trying to remotely troubleshoot "high load", you likely can't log in because the system needs to access /etc/passwd, /etc/nsswitch.conf, /etc/ldap/ldap.conf, /bin/bash... Your login gets handed a ticket with "∞" written on it, and is told to stand in the line for the disk.
But aren't these files already in the disk cache? Possibly. But another thing that happens when the system runs out of memory is that disk caches get purged. So you have to go back to the disk again. Also, when disk caches get dropped, your disk read performance is dogshit.
So what can you do? Try zswap? Use a different disk devices for swap? Try to pin important processes so the can't be swapped out?
NOTE: I might not be right about this stuff - I've never worked on the Linux kernel, and I rarely even compile it these days. But I've worked as SRE / *Ops for years and saw this problem a lot in badly configured systems. Same patterns all the time.
Because it's not the memory shortage that's nailing the UI. It's disk contention. When pathological swap activity causes disk thrashing, every process that needs to use the disk is now waiting in a very long queue with all the swap page in/out requests and every other process that is now backed up.
If you have a metrics agent that emits a handful of metrics on a per second granularity, you can see a bunch of things happen:
* Memory rises
* Some pages get swapped out
* Even if the system has no other processes writing to disk, you'll see the swap-out correlating with disk writes
* Memory rises some more
* You see more pages get swapped out
* (...and see more disk writes)
* Memory rises some more
* You see pages get swapped out and in, because now we're shuffling out pages that are imminently required by another running process.
* Disk queue lengths rise
* One by one, all processes are waiting (driving up the CPU "wa" number, and causing load averages to shoot up)
This is why, if you're oncall and you're trying to remotely troubleshoot "high load", you likely can't log in because the system needs to access /etc/passwd, /etc/nsswitch.conf, /etc/ldap/ldap.conf, /bin/bash... Your login gets handed a ticket with "∞" written on it, and is told to stand in the line for the disk.
But aren't these files already in the disk cache? Possibly. But another thing that happens when the system runs out of memory is that disk caches get purged. So you have to go back to the disk again. Also, when disk caches get dropped, your disk read performance is dogshit.
So what can you do? Try zswap? Use a different disk devices for swap? Try to pin important processes so the can't be swapped out?
NOTE: I might not be right about this stuff - I've never worked on the Linux kernel, and I rarely even compile it these days. But I've worked as SRE / *Ops for years and saw this problem a lot in badly configured systems. Same patterns all the time.
Thanks for the comment, that's very interesting. I hadn't considered the possibility of the desktop environment needing to do disk reads, I was thinking everything was in memory already.
I suppose the solution is to figure out how to reduce or eliminate the number of disk reads after the DE initially loads, or cache everything needed in memory with a "do not evict" flag on it. Especially for the TTY, that's basically a crucial system function and keeping everything needed for it always available in memory wouldn't take much space relative to a typical desktop system.
I think at least some of the problem is the program code pages getting evicted to disk, so they have to be read back to memory too, which goes to the back of the disk queue. When I'm experiencing thrashing, even basic stuff like alt-tab to switch windows doesn't work. And I assume the window manager doesn't need to do disk reads to switch windows. Even the damn X cursor freezes!
I suppose the solution is to figure out how to reduce or eliminate the number of disk reads after the DE initially loads, or cache everything needed in memory with a "do not evict" flag on it. Especially for the TTY, that's basically a crucial system function and keeping everything needed for it always available in memory wouldn't take much space relative to a typical desktop system.
I think at least some of the problem is the program code pages getting evicted to disk, so they have to be read back to memory too, which goes to the back of the disk queue. When I'm experiencing thrashing, even basic stuff like alt-tab to switch windows doesn't work. And I assume the window manager doesn't need to do disk reads to switch windows. Even the damn X cursor freezes!
Desktops could be improved quite a bit by tweaking default nice and ionice priorities for apps, instead of relying on one scheduling algorithm to be able to just "figure it out" on desktops, servers, phones, etc. A memory eviction priority, if niceness isn't treated as memory priority, could also help, but the existing tools can already go a long way and they just aren't being used.
I was able to get glitch-free audio by tweaking software and soft-irq realtime and non-realtime priorities, when the default Ububtu (with low-latency kernel) was nearly unusable for audio.
I was able to get glitch-free audio by tweaking software and soft-irq realtime and non-realtime priorities, when the default Ububtu (with low-latency kernel) was nearly unusable for audio.
This problem is there even if you have no swap at all, so I don't see different devices or zswap helping.
> This problem
Which problem? Yes, there's always disk contention. But if you don't have swap enabled then you won't have pathological swap activity dominating the disk.
There are several bad things that happen when you run out of memory. One of them is pathological swapping. Another is loss of disk caches. Another is (eventual) oom. The one that has the most detrimental effect, and typically results in the system having to be reset, is swap activity saturating disk io. Even when you've got fast SSDs, because they're still orders of magnitude slower than memory.
Which problem? Yes, there's always disk contention. But if you don't have swap enabled then you won't have pathological swap activity dominating the disk.
There are several bad things that happen when you run out of memory. One of them is pathological swapping. Another is loss of disk caches. Another is (eventual) oom. The one that has the most detrimental effect, and typically results in the system having to be reset, is swap activity saturating disk io. Even when you've got fast SSDs, because they're still orders of magnitude slower than memory.
You still have pathological disk io even with swap disabled because Linux will evict file-backed pages. More aggressive OOM killing is currently the only solution, which is not ideal, but losing one process beats losing all of them. I use earlyoom, and I assume systemd-oomd is something similar.
> Which problem?
Thrashing and becoming unusable for a long time.
Thrashing and becoming unusable for a long time.
For 2) last time I looked I couldn't find a way to have a memory contract with the kernel that allowed for a given allocation to be evicted at will by the kernel while also telling the application that has happened. This would be ideal for apps that can grow their cache unbounded (e.g., photo processing apps with each photo, browsers with each tab) but can also regenerate that cache at will. You end up having tunables for how much cache to hold in RAM which is never correct as depending on what else the user is doing there may be a lot or very little free RAM.
You can use PSI for that. And it actually makes more sense when all applications listen on the global PSI indicators so that if one application would start thrashing another can free up memory cooperatively. But if needed it can also be done per cgroup.
Something like madvise(MADV_FREE) sounds closer than MADV_DONTNEED -- the app can check for zero where it wrote nonzero -- but doesn't guarantee the freeing will be delayed until there is memory pressure.
Would it be possible to run the cache in a separate process/thread and then somehow mark that process/thread as sacrificial?
In that regard Linux is much worse than Windows. In Windows you basically never go out of memory in a way to lock the system, sice the OS reserves some memory for its functions. It will block applications, but the operating system remain responsive.
In Linux I was never been able to generate a similar behaviour. No matter how aggressive I set the swappiness, the system didn't began to swap until it was practically locked up with the mouse pointer lagging.
Windows reserves some memory for its functions, for example the UI (maybe the fact that the GUI is in the kernel helps), and thus doesn't have these kind of problem, especially with an SSD (and I know that page file on SSDs is not ideal... but had a disk for 5 years swapping on it and never had a problem).
In Linux I was never been able to generate a similar behaviour. No matter how aggressive I set the swappiness, the system didn't began to swap until it was practically locked up with the mouse pointer lagging.
Windows reserves some memory for its functions, for example the UI (maybe the fact that the GUI is in the kernel helps), and thus doesn't have these kind of problem, especially with an SSD (and I know that page file on SSDs is not ideal... but had a disk for 5 years swapping on it and never had a problem).
I have some strong memories of paging death on Windows XP, on spinning rust. I recall taking some minutes to recover: Ctrl-Alt-Del and wait for secure authentication screen; hit shortcut for Task Manager and wait for it to start and come to the foreground; navigate to offending process and kill it, with blind keystrokes.
The mouse cursor did move smoothly throughout, though, as I recall.
The mouse cursor did move smoothly throughout, though, as I recall.
What do you mean “didn’t begin to swap”? It should always be using swap. It starts locking up when you run out of both swap and RAM (because it’s run out of things to swap out that aren’t critical to what’s currently running). What it fails to do in time is outright kill processes.
[0] was a humorous read. I love these little analogies.
It's kind of a funny analogy in the context of systemd-oomd, because the latter is sort of like saying "hmm, when the plane is out of fuel, sometimes there's not enough power to actually eject anyone from the plane, so we're going to start ejecting people while there's still a little bit of fuel left!"
I actually don't understand why systemd-oomd is needed given MemoryMax= ( https://man7.org/linux/man-pages/man5/systemd.resource-contr... ). What are they doing differently?
I think the distinction is that MemoryMax= is just an interface to the cgroupv2 setting, i.e., that rule is implemented inside the kernel and invokes the kernel's OOM killer within a cgroup. The manpage for systemd-oomd says, "systemd-oomd is a system service that uses cgroups-v2 and pressure stall information (PSI) to monitor and take action on processes before an OOM occurs in kernel space."
It looks like systemd-oomd is related to (based on? from the same people as?) Facebook's oomd https://github.com/facebookincubator/oomd , whose documentation gives a bunch of reasons as to why you would prefer a userspace oomd that takes in PSI data and can be configured to proactively kill misbehaving processes instead of just letting the kernel OOM killer handle it. The major reason is time to recovery: a misbehaving process can cause a system to be so far under pressure that the kernel OOM killer will take a long time to flush things out, but a userspace component can respond in advance with more configurable rules (and more flexibility, since the kernel doesn't believe you're at capacity yet).
It looks like systemd-oomd is related to (based on? from the same people as?) Facebook's oomd https://github.com/facebookincubator/oomd , whose documentation gives a bunch of reasons as to why you would prefer a userspace oomd that takes in PSI data and can be configured to proactively kill misbehaving processes instead of just letting the kernel OOM killer handle it. The major reason is time to recovery: a misbehaving process can cause a system to be so far under pressure that the kernel OOM killer will take a long time to flush things out, but a userspace component can respond in advance with more configurable rules (and more flexibility, since the kernel doesn't believe you're at capacity yet).
Yes, exactly.
> It looks like systemd-oomd is related to (based on? from the same people as?) Facebook's oomd
Yep, I authored the original standalone oomd (along with many others' help) and Anita has been working on systemd-oomd. There's plans to converge and simplify everything that's been learned from oomd into systemd-oomd so the community can benefit from all our experiments.
> It looks like systemd-oomd is related to (based on? from the same people as?) Facebook's oomd
Yep, I authored the original standalone oomd (along with many others' help) and Anita has been working on systemd-oomd. There's plans to converge and simplify everything that's been learned from oomd into systemd-oomd so the community can benefit from all our experiments.
When a cgroup hits its memory.max the kernel OOM killer is invoked. systemd-oomd enables finer grained policy than the kernel OOM killer. systemd-oomd can also prevent livelocks (which memory.max does not).
Could they maybe start handling the 1400+ bugs that has been a part of systemd for such a long time (some unresolved since 2015) rather than keep adding and adding and adding and adding code!
There are a lot of open issues but you are a bit unfair in calling them bugs.
If you filter out the RFEs (Requests For Enhancement) you find 617 issues. If you filter for only issues labeled as a bug you get 97 results.
If you filter out the RFEs (Requests For Enhancement) you find 617 issues. If you filter for only issues labeled as a bug you get 97 results.
OK, but they don't always label bugs that way (example: systemd/#5616) and the bugs that are labeled can get pretty old (systemd/#3700). Furthermore a lot of RFEs are really just disguised bug reports (systemd/#5917 is just an RFE rephrasing of system/#3700).
In my experience, unless you can get Red Hat to care about patching a specific bug, you can rely on opened bugs to stay open.
In my experience, unless you can get Red Hat to care about patching a specific bug, you can rely on opened bugs to stay open.
I didn't dig into others, but:
> don't always label bugs that way (example: systemd/#5616)
Nobody figured out if that issue is in logind or X yet. They did not close it so it's not like they say it can't possibly be on systemd side. (typical Lennart comments aside)
> don't always label bugs that way (example: systemd/#5616)
Nobody figured out if that issue is in logind or X yet. They did not close it so it's not like they say it can't possibly be on systemd side. (typical Lennart comments aside)
People add features they need, and fix bugs that impact them. Scratching your own itch is the fundamental principle in open source. I recommend you to do the same.
Feel free to contribute to open source projects!
I have some questions.
I started reading about systemd-oomd, then followed the link to its manpage (at https://www.man7.org/linux/man-pages/man8/systemd-oomd.8.htm...) then followed a link from there to this page: "In defence of swap: common misconceptions" at https://chrisdown.name/2018/01/02/in-defence-of-swap.html).
The first line of the tl;dr is: "Having swap is a reasonably important part of a well functioning system. Without it, sane memory management becomes harder to achieve."
Now, afaik linux as a kernel is pretty much designed to have swap memory and really not meant to be ran without it. Except disabling swap is required to run kubernetes, so much so that the kubelet will plain refuse to start if it detects swap memory enabled.
Why?
I started reading about systemd-oomd, then followed the link to its manpage (at https://www.man7.org/linux/man-pages/man8/systemd-oomd.8.htm...) then followed a link from there to this page: "In defence of swap: common misconceptions" at https://chrisdown.name/2018/01/02/in-defence-of-swap.html).
The first line of the tl;dr is: "Having swap is a reasonably important part of a well functioning system. Without it, sane memory management becomes harder to achieve."
Now, afaik linux as a kernel is pretty much designed to have swap memory and really not meant to be ran without it. Except disabling swap is required to run kubernetes, so much so that the kubelet will plain refuse to start if it detects swap memory enabled.
Why?
Linux works fine without swap. It also works fine with swap. It does not work fine under memory contention regardless of swap, but it is much, much worse if you have many GB of swap on a spinning disk.
Note that managing resources is much harder when swap is in the picture, because you can get the resident memory of a process, but not the swap usage of a process.
It used to be that the recommendation for swap was 2x ram. This recommendation was from the days were single-digit mega bytes of ram was common, and also from when linux core-dumps were both expected, and stored to swap.
Fast forward to 2010ish. You might have a 64GB or 128GB SSD, but you probably also have a spinning drive. you have 8 or 16GB of ram, so your 16-32GB swap would be 1/4 the SSD, plus you heard that putting swap on an SSD wears it out faster, so you put a 16-32GB swap file on the spinning drive.
Lets say you are developing software and you accidentally allocate in a tight loop. At some point the majority of the pages in ram not from the malloc loop end up in swap, so you try to do something to kill it, but Xorg and xterm both have their pages scattered across spinning drives, and your program is allocating memory faster than your machine saps in. If you are patient enough to not hard reset, an hour later, swap fills up and the OOM killer starts killing programs, and probably (but not definitely) kills the buggy malloc loop before it kills something more vital.
You are annoyed, so disable swap and do the same thing: the OOM killer kills probably just your web browser (which is virtually indistinguishable to a malloc loop with Web 2.0), and the malloc loop. The system recovers in minutes. You now swear off ever running with swap again.
Today: If you have a spinning disk 1GB or swap on it is fine, but zram is another good alternative. If you have slow and/or fragile SSD storage (e.g. sdcard or eMMC), zram is a good option. If you have fast nvme, I hear that you can go fairly crazy with swap sizes and have a system that works well under memory pressure, but I have not tried it myself.
Note that managing resources is much harder when swap is in the picture, because you can get the resident memory of a process, but not the swap usage of a process.
It used to be that the recommendation for swap was 2x ram. This recommendation was from the days were single-digit mega bytes of ram was common, and also from when linux core-dumps were both expected, and stored to swap.
Fast forward to 2010ish. You might have a 64GB or 128GB SSD, but you probably also have a spinning drive. you have 8 or 16GB of ram, so your 16-32GB swap would be 1/4 the SSD, plus you heard that putting swap on an SSD wears it out faster, so you put a 16-32GB swap file on the spinning drive.
Lets say you are developing software and you accidentally allocate in a tight loop. At some point the majority of the pages in ram not from the malloc loop end up in swap, so you try to do something to kill it, but Xorg and xterm both have their pages scattered across spinning drives, and your program is allocating memory faster than your machine saps in. If you are patient enough to not hard reset, an hour later, swap fills up and the OOM killer starts killing programs, and probably (but not definitely) kills the buggy malloc loop before it kills something more vital.
You are annoyed, so disable swap and do the same thing: the OOM killer kills probably just your web browser (which is virtually indistinguishable to a malloc loop with Web 2.0), and the malloc loop. The system recovers in minutes. You now swear off ever running with swap again.
Today: If you have a spinning disk 1GB or swap on it is fine, but zram is another good alternative. If you have slow and/or fragile SSD storage (e.g. sdcard or eMMC), zram is a good option. If you have fast nvme, I hear that you can go fairly crazy with swap sizes and have a system that works well under memory pressure, but I have not tried it myself.
> If you have fast nvme, I hear that you can go fairly crazy with swap sizes and have a system that works well under memory pressure, but I have not tried it myself
Not sure about Linux, but under macOS I had a bug that was leaking uncompressed 12 MP photos in a loop. I was browsing the web while waiting for my process to run and I got up to something like 150 GB of swap used (macOS uses dynamically growing swap files) before I even noticed that there was something going on.
Not sure about Linux, but under macOS I had a bug that was leaking uncompressed 12 MP photos in a loop. I was browsing the web while waiting for my process to run and I got up to something like 150 GB of swap used (macOS uses dynamically growing swap files) before I even noticed that there was something going on.
> It used to be that the recommendation for swap was 2x ram.
It always boiled down to never allocate more swap than you're willing to wait, and memory capacity multiplied over the years far faster than real world drive thruput.
So I was willing to slow everything down to use 8 megs of swap in 1994 in order to have enough virtual memory to compile a kernel, and in 2021 I'm still willing to slow everything down to use 8 megs of swap, in fact drive thruput has maybe quintupled since then, maybe, so I'd be willing to slow my system down into using 32 megs of swap in 2021.
The problem is 8 megs of swap in 1994 dramatically increased my horizons and I could use it to compile the kernel and stuff like that. In 2021, 32 megs of swap just makes my web browser OOM kill about 100 milliseconds later so why bother?
It always boiled down to never allocate more swap than you're willing to wait, and memory capacity multiplied over the years far faster than real world drive thruput.
So I was willing to slow everything down to use 8 megs of swap in 1994 in order to have enough virtual memory to compile a kernel, and in 2021 I'm still willing to slow everything down to use 8 megs of swap, in fact drive thruput has maybe quintupled since then, maybe, so I'd be willing to slow my system down into using 32 megs of swap in 2021.
The problem is 8 megs of swap in 1994 dramatically increased my horizons and I could use it to compile the kernel and stuff like that. In 2021, 32 megs of swap just makes my web browser OOM kill about 100 milliseconds later so why bother?
As I understand it, swap is used even when the system isn’t under direct memory pressure.
If the kernel sees that some memory is barely, if ever used, it will swap it out preemptively. This gives you more memory available for both programs and temporary caches.
As such it’s still advantageous to add much more swap than this, since it effectively gives you extra free space in RAM at essentially no cost.
If the kernel sees that some memory is barely, if ever used, it will swap it out preemptively. This gives you more memory available for both programs and temporary caches.
As such it’s still advantageous to add much more swap than this, since it effectively gives you extra free space in RAM at essentially no cost.
> I hear that you can go fairly crazy with swap sizes and have a system that works well under memory pressure, but I have not tried it myself.
We are running a distributed in-memory olap database. All nodes have 500gb ram used basically at capacity. One node has 1.5T of nvme swap (initially just for testing but it works well) so it hosts a shard and the manager/reduce node. We've tested forcing the nodes onto swap and saw a x10 performance decrease, but our implementation is pretty shit when it comes to memory access patterns so it's not too bad. The collocated node is slightly slower in terms of performance but doesn't impact total response by a meaningful margin.
We are running a distributed in-memory olap database. All nodes have 500gb ram used basically at capacity. One node has 1.5T of nvme swap (initially just for testing but it works well) so it hosts a shard and the manager/reduce node. We've tested forcing the nodes onto swap and saw a x10 performance decrease, but our implementation is pretty shit when it comes to memory access patterns so it's not too bad. The collocated node is slightly slower in terms of performance but doesn't impact total response by a meaningful margin.
> Note that managing resources is much harder when swap is in the picture, because you can get the resident memory of a process, but not the swap usage of a process.
Why can you not get swap usage? It’s in smaps, right?
Why can you not get swap usage? It’s in smaps, right?
Oh smaps is newer than my knowledge. I did since more research and your still can't limit over RSS+swap with ulimit though. There's a chance you could with cgroups though as there's lots of features there.
This is one of the primary reasons the Kubernetes folks are working on letting you run with swap! See recent discussion in https://github.com/kubernetes/kubernetes/issues/53533 .
The historic reason Kubernetes didn't let you run with swap is that it's complicated from a resource management perspective - it's not clear whether each pod should be requesting some amount of swap the same way it requests some amount of RAM, and also the container runtimes didn't really have support for that (largely because you need cgroupv2 to have a hope of that sort of per-cgroup swap resource control working at all). But it's not an insurmountable problem, and it sounds from that ticket like they recognize that letting systems have swap enabled for the sake of a userspace oomd is an important use case.
The historic reason Kubernetes didn't let you run with swap is that it's complicated from a resource management perspective - it's not clear whether each pod should be requesting some amount of swap the same way it requests some amount of RAM, and also the container runtimes didn't really have support for that (largely because you need cgroupv2 to have a hope of that sort of per-cgroup swap resource control working at all). But it's not an insurmountable problem, and it sounds from that ticket like they recognize that letting systems have swap enabled for the sake of a userspace oomd is an important use case.
Why is Linux not meant to be ran without swap? In 2021? I've never heard of this. I've operated massive clusters of thousands of hosts with no swap enabled with zero hiccups.
Swap comes with a lot of caveats -- when a performance-critical process begins swapping it can on occasion be more useful to just let it OOM instead.
My personal desktop has 64GB of memory, with no swap enabled. Why would I want swap it I don't need more than that?
Swap comes with a lot of caveats -- when a performance-critical process begins swapping it can on occasion be more useful to just let it OOM instead.
My personal desktop has 64GB of memory, with no swap enabled. Why would I want swap it I don't need more than that?
You don't need it. Linux works fine without swap, given ample memory.
In my experience, even with ample swap, when Linux gets memory starved it goes haywire anyways.
I'm an ex linux sysadmin, and haven't run swap on my personal systems in years.
In my experience, even with ample swap, when Linux gets memory starved it goes haywire anyways.
I'm an ex linux sysadmin, and haven't run swap on my personal systems in years.
If you want to understand why a oomd daemon is necessary and oom alone is not enough, you might be interested in this talk from FB "Linux memory management at scale" https://youtube.com/watch?v=cSJFLBJusVY
The finished talk with a (short) Q&A can be also be found.
https://media.ccc.de/v/arch-conf-online-2020-6390-linux-memo...
Q&A Questions: https://gitlab.archlinux.org/archlinux/conf-files/-/blob/mas...
https://media.ccc.de/v/arch-conf-online-2020-6390-linux-memo...
Q&A Questions: https://gitlab.archlinux.org/archlinux/conf-files/-/blob/mas...
Finally the author of nohang is working on arguably superior solutions: https://github.com/hakavlad/prelockd https://github.com/hakavlad/memavaild https://github.com/hakavlad/le9-patch It's unfortunate that systemd has chosen to integrate an inferior solution