A word about systemd (2016)(skarnet.org)
skarnet.org
A word about systemd (2016)
https://skarnet.org/software/systemd.html
187 comments
I was a systemd hater. I agree with a lot of the post but in practice I need a whole lot less custom scripts, forking code, etc to maintain a simple service whether it’s a custom c binary, go, rust, nodejs , ruby , python… I don’t need to mess with monit(for most things) and yeah in less words then my post here I could have a working service that would both restart on crash and on reboot…. Do I miss the init.d scripts and writing pid files forking twice to change user … no
> writing pid files forking twice to change user
No idea why software tries to "daemonize" anyway. Seems to be some kind of weird tradition. It's always been the wrong thing to do. They should just run normally and output to standard streams even if it's connected to a terminal. The reasons are outlined here:
https://mywiki.wooledge.org/ProcessManagement
Ironically, systemd is one of few service managers which can properly track those weird "daemonizing" processes. It uses Linux's cgroups. It doesn't matter how many times they fork, they can't hide from systemd.
No idea why software tries to "daemonize" anyway. Seems to be some kind of weird tradition. It's always been the wrong thing to do. They should just run normally and output to standard streams even if it's connected to a terminal. The reasons are outlined here:
https://mywiki.wooledge.org/ProcessManagement
Ironically, systemd is one of few service managers which can properly track those weird "daemonizing" processes. It uses Linux's cgroups. It doesn't matter how many times they fork, they can't hide from systemd.
Not sure what exactly you were pointing to, but section "Doing it right" (4.3.1) recommends:
- you ssh'd into server, restarted server, checked it worked. Then you logged out and server got killed.
- your server runs as root or power user
- your server output/error goes somewhere, not clear where
- your server's process got paused by SIGTTOU, because it was trying to write to console
etc...
The complex daemonization dance, with double-forks, closing random FDs, etc.. was required to avoid all this bad stuff before systemd days. Sysvinit/openrc folks came up with "start-stop-daemon" helper, but it was pretty terrible in general, especially in logging department (it had none).
mydaemon & daemonpid=$!
- you ssh'd into server, restarted server, checked it worked. Then you logged out and server got killed.
- your server runs as root or power user
- your server output/error goes somewhere, not clear where
- your server's process got paused by SIGTTOU, because it was trying to write to console
etc...
The complex daemonization dance, with double-forks, closing random FDs, etc.. was required to avoid all this bad stuff before systemd days. Sysvinit/openrc folks came up with "start-stop-daemon" helper, but it was pretty terrible in general, especially in logging department (it had none).
Handling all of those conditions is the job of a proper service manager like systemd. It does things like restart processes that died, redirect output to logs, set process user and group, etc.
The point is software should be written to make the service manager's job as easy as possible by keeping it simple.
The point is software should be written to make the service manager's job as easy as possible by keeping it simple.
I agree that handling all those conditions is the job of a proper service manager.
I disagree that systemd is a proper service manager. Systemd is horribly complicated in comparison with what is actually needed to implement a proper service manager, as demonstrated e.g. by D. J. Bernstein's daemontools and their many derivatives.
Perhaps systemd is better these days, but a few years ago when I have tried systemd on one of my servers I have stumbled upon a bug that could be explained not by some random programmer mistake but only by a bad global architecture of the program. That bug made me mistrust the competence of the systemd creators, so I have never tried to use it again. Moreover, systemd does not provide any feature that I could consider useful and that is not available in alternative service managers, so there were no reasons for further experiments with it.
(That systemd bug was a bug that I have never seen in a computer without systemd, from the mainframes of 50 years ago to the latest computers, regardless of the operating system. Due to a race condition in systemd, sometimes the shutdown or reboot of a computer could fail, with the computer remaining stuck in a deadlock, i.e. in an infinite loop, without properly saving all state and closing all resources.)
I disagree that systemd is a proper service manager. Systemd is horribly complicated in comparison with what is actually needed to implement a proper service manager, as demonstrated e.g. by D. J. Bernstein's daemontools and their many derivatives.
Perhaps systemd is better these days, but a few years ago when I have tried systemd on one of my servers I have stumbled upon a bug that could be explained not by some random programmer mistake but only by a bad global architecture of the program. That bug made me mistrust the competence of the systemd creators, so I have never tried to use it again. Moreover, systemd does not provide any feature that I could consider useful and that is not available in alternative service managers, so there were no reasons for further experiments with it.
(That systemd bug was a bug that I have never seen in a computer without systemd, from the mainframes of 50 years ago to the latest computers, regardless of the operating system. Due to a race condition in systemd, sometimes the shutdown or reboot of a computer could fail, with the computer remaining stuck in a deadlock, i.e. in an infinite loop, without properly saving all state and closing all resources.)
Systemd is complicated because software that does it all always is.
You might not need the features, but someone does, and putting it all in one places means everyone can use the same tool, and nobody has to learn 10 different systems for slightly different tasks.
You might not need the features, but someone does, and putting it all in one places means everyone can use the same tool, and nobody has to learn 10 different systems for slightly different tasks.
Software doing it all is antithetical to the Unix philosophy of "Do one thing and do it well."
Hence the hate that systemd has, and always will have, as long as there exists a community of people who use Linux because it is a Unix-like operating system.
But, Linux is versatile enough that everyone can have their own little fiefdom. The real rage came in years ago when tools that were widely used by all suddenly became reliant upon systemd to function at all. Thankfully, in the time since, enough development has happened that mostly people are free to choose what they do and don't want to run again. The manner in which it was foisted upon the community, however, was one that hopefully will be remembered as being far more aggressive than should be repeated in the future.
But, I imagine that as long as Lennart Poettering walks the Earth, we likely will have yet another similarly awful episode.
Hence the hate that systemd has, and always will have, as long as there exists a community of people who use Linux because it is a Unix-like operating system.
But, Linux is versatile enough that everyone can have their own little fiefdom. The real rage came in years ago when tools that were widely used by all suddenly became reliant upon systemd to function at all. Thankfully, in the time since, enough development has happened that mostly people are free to choose what they do and don't want to run again. The manner in which it was foisted upon the community, however, was one that hopefully will be remembered as being far more aggressive than should be repeated in the future.
But, I imagine that as long as Lennart Poettering walks the Earth, we likely will have yet another similarly awful episode.
I think a big issue is just that it takes more effort to support multiple init systems. It's hard to be particularly excited about maintaining support for something you don't have a use for yourself, and nobody is paying for(Or at least not as many people in your core audience).
Seems like we're in a pretty good place now, with separate development efforts by and for the people who are actually interested in anything but systemd.
Seems like we're in a pretty good place now, with separate development efforts by and for the people who are actually interested in anything but systemd.
Do you have the link to the bug you raised?
The bug you mentioned is very common, I've experienced it myself and others I know have too
[deleted]
As a previous user/exploiter of SysV init, I really like systemd.
-I can spawn/respawn daemons as someone other than root. SysV inittab runlevels are root-only.
-I used runlevel 4 for all my magic. Now I can have so much more magic.
-nspawn has let me craft containers from day 1.
-Path units are so much better than inotifywait (and somewhat better than incron).
-Do I prefer socket units to inetd? Mostly.
-Free automounter! Yes, crazy syntax, but gratis!
Do I wish it worked on BSD? Yes.
-I can spawn/respawn daemons as someone other than root. SysV inittab runlevels are root-only.
-I used runlevel 4 for all my magic. Now I can have so much more magic.
-nspawn has let me craft containers from day 1.
-Path units are so much better than inotifywait (and somewhat better than incron).
-Do I prefer socket units to inetd? Mostly.
-Free automounter! Yes, crazy syntax, but gratis!
Do I wish it worked on BSD? Yes.
I think there are two parts of systemd - the functionality and the implementation.
It IS functional - it does things that are helpful and important.
But the implementation is hard to swallow. It is just so far away from elegant, understandable and usable.
It IS functional - it does things that are helpful and important.
But the implementation is hard to swallow. It is just so far away from elegant, understandable and usable.
Can you give some examples of implementation that are hard to understand and use?
- symlinks - systemd took the "accepted" idea of symlinks from /etc/init.d -> /etc/rc*.d and put symlinks all over the filesystem for all kinds of units. It is less understandable and manageable.
- mediocre config files - they decided that config files are declarative, which could be good, but it pushed simple (and visible) logic in /etc/init.d/* files into more complicated logic encoded in binaries. The simplistic config files seem to be more simple for systemd to implement, but harder to understand, maintain and use.
- binary log files. everything has to go through systemd to access them.
- responsibility creep - systemd started taking over more functions, and it doesn't seem to stop. Will systemd turn into busybox, with simpler poorer implementations of every single thing in the system? dnsmasq is similar. The thing is, busybox and dnsmasq are cut down implementations for smaller system that are alternatives to their full-featured brethren. systemd takes over the core of a system.
- mediocre config files - they decided that config files are declarative, which could be good, but it pushed simple (and visible) logic in /etc/init.d/* files into more complicated logic encoded in binaries. The simplistic config files seem to be more simple for systemd to implement, but harder to understand, maintain and use.
- binary log files. everything has to go through systemd to access them.
- responsibility creep - systemd started taking over more functions, and it doesn't seem to stop. Will systemd turn into busybox, with simpler poorer implementations of every single thing in the system? dnsmasq is similar. The thing is, busybox and dnsmasq are cut down implementations for smaller system that are alternatives to their full-featured brethren. systemd takes over the core of a system.
> put symlinks all over the filesystem for all kinds of units. It is less understandable and manageable.
It is actually way better understandable and manageable: /usr/lib/systemd is for distro-supplied and packaged units. /etc/systemd is for your local overrides, /run/systemd is for units created at runtime by various generators. (Then there are $XDG_(RUNTIME|CONFIG)_DIR/systemd for user units, for consistency).
So no more diffing config files at update time: the distro provided ones are always updated, your overrides are always preserved, cleanly.
> - binary log files. everything has to go through systemd to access them.
So send them to external syslog, that saves them as you wish. There is a point in indexing them (and being able to have a newline in the log message).
- responsibility creep
This is a matter of perspective. If you take it as "init system at boot", then yes, it might looks so. If you take as an service manager that runs/stops/reloads/monitors units, based on different events (startup, dbus activation, socket activation, timer, etc): what's the difference in the bulk of action, except for the event that caused that action? The service handling, unit definition, etc. are all the same, just the initiation event is different. In this light, it doesn't take over functions; it consolidates and unifies them.
Unlike dnsmasq or busybox, it is not in cut-down fashion, but exactly opposite. It replaces rag-tag of tools with one with more complete and tested implementation.
It is actually way better understandable and manageable: /usr/lib/systemd is for distro-supplied and packaged units. /etc/systemd is for your local overrides, /run/systemd is for units created at runtime by various generators. (Then there are $XDG_(RUNTIME|CONFIG)_DIR/systemd for user units, for consistency).
So no more diffing config files at update time: the distro provided ones are always updated, your overrides are always preserved, cleanly.
> - binary log files. everything has to go through systemd to access them.
So send them to external syslog, that saves them as you wish. There is a point in indexing them (and being able to have a newline in the log message).
- responsibility creep
This is a matter of perspective. If you take it as "init system at boot", then yes, it might looks so. If you take as an service manager that runs/stops/reloads/monitors units, based on different events (startup, dbus activation, socket activation, timer, etc): what's the difference in the bulk of action, except for the event that caused that action? The service handling, unit definition, etc. are all the same, just the initiation event is different. In this light, it doesn't take over functions; it consolidates and unifies them.
Unlike dnsmasq or busybox, it is not in cut-down fashion, but exactly opposite. It replaces rag-tag of tools with one with more complete and tested implementation.
> Unlike dnsmasq or busybox, it is not in cut-down fashion
check out for systemd networking, the resolver, timesyncd, etc...
for example timesyncd IS cut-down and brute-force:
"does not bother with the full NTP complexity, focusing only on querying time from one remote server and synchronizing the local clock to it"
ntp is complex, but it does many things in safe and effective ways.
etc.
check out for systemd networking, the resolver, timesyncd, etc...
for example timesyncd IS cut-down and brute-force:
"does not bother with the full NTP complexity, focusing only on querying time from one remote server and synchronizing the local clock to it"
ntp is complex, but it does many things in safe and effective ways.
etc.
systemd-network is cut-down I agree. But it does not have an ambition to replace NetworkManager; it is usually installed only in cut-down environments (containers; ubuntu installs it for servers, but not for desktops). Redhat doesn't even have it in repo.
systemd-resolved, on other hand, is good. There's no other resolver stub, that has the functionality that resolved has. It is the only one for Linux, that can do per-link, split DNS properly.
systemd-resolved, on other hand, is good. There's no other resolver stub, that has the functionality that resolved has. It is the only one for Linux, that can do per-link, split DNS properly.
I recall systemd-resolved ignoring the resolv.conf order of preference of name servers. The native and thus relatively simple per-link DNS configuration is great though.
Addendum: https://news.ycombinator.com/item?id=15228940 This approach clearly deviates from widely expected behavior, whether it is in conformance with applicable RFCs or not.
Addendum: https://news.ycombinator.com/item?id=15228940 This approach clearly deviates from widely expected behavior, whether it is in conformance with applicable RFCs or not.
Resolv.conf does not specify order of preference of name servers! All the servers configured here are supposed to return the same answers!
See, many people have misconceptions how it works; see the comments in thread you linked to. Add to it the fact that resolv.conf syntax cannot express what is expected of modern resolvers (not even with apple-like extension - `/etc/resolver/*`; it cannot react to links going up and down), it is basically just compatibility shim for broken applications (they are not supposed to use it directly); that's why systemd-resolved configures it to point to itself and handles the specifics for them.
See, many people have misconceptions how it works; see the comments in thread you linked to. Add to it the fact that resolv.conf syntax cannot express what is expected of modern resolvers (not even with apple-like extension - `/etc/resolver/*`; it cannot react to links going up and down), it is basically just compatibility shim for broken applications (they are not supposed to use it directly); that's why systemd-resolved configures it to point to itself and handles the specifics for them.
Providing the same answers does not preclude an order (of preference). The man page clearly shows that such an order exists, though it does not explicitly label it as a preference. Whether you think it's right or wrong is irrelevant—the fact remains that the expectation of querying DNS servers in a specific order has been a common understanding for many. You're free to think systemd-resolved is the right way forward, but that's not an argument against the existence of the ordering in resolv.conf.
> the fact remains that the expectation of querying DNS servers in a specific order has been a common understanding for many.
That understanding has been wrong; it is an implementation detail of glibc and that's it. It could query them in random order, or round-robin them for each request; or time them how fast they respond and prefer the fastest, up until it starts timing out and then shake the order again. Some resolver stubs ask all of them and use the answer that arrives first. That's also valid. Since they all are returning the same answer, it doesn't matter. It also makes any argument for ordering superfluous.
So that "common understanding for many" is erroneous and those many should update it. The thread you linked to has exactly a discussion on this point, so in the years that passed since then, those affected many had ample time to fix up their assumptions.
That understanding has been wrong; it is an implementation detail of glibc and that's it. It could query them in random order, or round-robin them for each request; or time them how fast they respond and prefer the fastest, up until it starts timing out and then shake the order again. Some resolver stubs ask all of them and use the answer that arrives first. That's also valid. Since they all are returning the same answer, it doesn't matter. It also makes any argument for ordering superfluous.
So that "common understanding for many" is erroneous and those many should update it. The thread you linked to has exactly a discussion on this point, so in the years that passed since then, those affected many had ample time to fix up their assumptions.
That understanding is in line with using libresolv and the established behavior of the resolv.conf file, which accompanies the libresolv library. While it’s true that different implementations (e.g., some resolver stubs) might use various methods like round-robin, timing, or querying all servers simultaneously, this doesn’t invalidate the behavior outlined in resolv.conf.
Again, there can be other valid reasons for maintaining an order. Could be about latency, privacy etc.
I admit to not having read the full thread, and I apologize for any needless rehashing of arguments. It was just the first result confirming systemd-resolved breaking setups by its handling of what many (in that thread) consider expected behavior.
What I think you're primarily suggesting is that there are no formal standards prescribing glibc's behavior in terms of DNS resolution order. That is certainly true, and I don't disagree calling it an implementation detail.
Again, there can be other valid reasons for maintaining an order. Could be about latency, privacy etc.
I admit to not having read the full thread, and I apologize for any needless rehashing of arguments. It was just the first result confirming systemd-resolved breaking setups by its handling of what many (in that thread) consider expected behavior.
What I think you're primarily suggesting is that there are no formal standards prescribing glibc's behavior in terms of DNS resolution order. That is certainly true, and I don't disagree calling it an implementation detail.
I think it's possible that you just hold a non-majority opinion about what good usability looks like.
That's fine, and I'm OK with a world where we disagree about how usable systemd is, but I interact with the systemd config files and symlinks and logs pretty frequently, and have found them to be orders of magnitude more usable and understandable than /etc/rc#.d and /etc/init.d and "I wonder if this system puts sshd logs in /var/log/auth.log or /var/log/secure or somewhere else". Notably, I interact with a mix of Arch and Ubuntu and RHEL systems over the course of a week, and systemd means I can engage with all of the above using the same methodology.
In terms of creep... busybox is noteworthy for packing all of a system into a single binary. systemd doesn't do this. Their resolver and time daemon and networking and all the other bits and bobs are their own separate binary, and you aren't required to use them.
That's fine, and I'm OK with a world where we disagree about how usable systemd is, but I interact with the systemd config files and symlinks and logs pretty frequently, and have found them to be orders of magnitude more usable and understandable than /etc/rc#.d and /etc/init.d and "I wonder if this system puts sshd logs in /var/log/auth.log or /var/log/secure or somewhere else". Notably, I interact with a mix of Arch and Ubuntu and RHEL systems over the course of a week, and systemd means I can engage with all of the above using the same methodology.
In terms of creep... busybox is noteworthy for packing all of a system into a single binary. systemd doesn't do this. Their resolver and time daemon and networking and all the other bits and bobs are their own separate binary, and you aren't required to use them.
It is possible I haven't communicated the gist of the idea.
systemd is functional. It does the things you say. It unifies things, including config files and log files. It works for you.
I just don't think the way it did this is that elegant.
My busybox analogy should be more viewed as: what if arch/ubuntu/rh decided that busybox would replace all coreutils, etc on their systems today? All the same commands. Still usable. But something is missing...
systemd is functional. It does the things you say. It unifies things, including config files and log files. It works for you.
I just don't think the way it did this is that elegant.
My busybox analogy should be more viewed as: what if arch/ubuntu/rh decided that busybox would replace all coreutils, etc on their systems today? All the same commands. Still usable. But something is missing...
What is missing?
I have the feeling that you are mourning the old, bad days?
I have the feeling that you are mourning the old, bad days?
I disagree with most of this.
1. symlinks: I think the systemd structure is very intuitive and the order of overwriting makes sense. It's new, but it makes sense, and we only have 3 different locations, each with a specific order and precedence.
2. Config files: I think keeping the declarative syntax is the secret sauce of systemd. The whole point is you can orchestrate processes without needing to write code. They're much easier to understand if you assume a knowledge point of 0, and they're much more maintainable (no logic), and they're even more portable. Bonus is they're much faster to execute by the init system. Yes, most of the logic is in systemd now, but you tell it what to do. Reading the documentation goes a long way to making unit files less cryptic.
3. Binary logs: yeah I agree just not a great idea. Always sounds nice on paper because you can do neat things with permissions and can have faster performance, but in reality just not necessary. They got too fancy.
4. Responsibility creep: from where I'm standing, not a problem. systemd isn't a piece of software, it's just a name given to a collection of software, like GNU. systemd-init doesn't have feature creep. Sure, now we have systemd-file-my-taxes, but that's optional and it's a whole different program that just uses the systemd name.
1. symlinks: I think the systemd structure is very intuitive and the order of overwriting makes sense. It's new, but it makes sense, and we only have 3 different locations, each with a specific order and precedence.
2. Config files: I think keeping the declarative syntax is the secret sauce of systemd. The whole point is you can orchestrate processes without needing to write code. They're much easier to understand if you assume a knowledge point of 0, and they're much more maintainable (no logic), and they're even more portable. Bonus is they're much faster to execute by the init system. Yes, most of the logic is in systemd now, but you tell it what to do. Reading the documentation goes a long way to making unit files less cryptic.
3. Binary logs: yeah I agree just not a great idea. Always sounds nice on paper because you can do neat things with permissions and can have faster performance, but in reality just not necessary. They got too fancy.
4. Responsibility creep: from where I'm standing, not a problem. systemd isn't a piece of software, it's just a name given to a collection of software, like GNU. systemd-init doesn't have feature creep. Sure, now we have systemd-file-my-taxes, but that's optional and it's a whole different program that just uses the systemd name.
Unfortunately, there aren't any concrete examples (unless I skipped something accidentally), so it's hard to tell what exactly is meant as "does more" or that it doesn't "stand out of the way" etc.
Yes, systemd services can do a lot of things. Except that before rc scripts did all those things, but without any unified and standardized primitives. systemd provides those for a lot of common use cases - and that's how it ended up doing so much - because our rc scripts used to do that. systemd actually does less because unit files are not scripts (but one can hook any external program in a lot of places, if they need to).
Also, as I understand it, systemd isn't a single piece of software either, it's a whole project with a bunch of semi-autonomous pieces, tied with common conventions and assumptions. It's difficult to define the scope for an umbrella project, and I believe individual pieces all have their scopes relatively well-defined.
Yes, systemd services can do a lot of things. Except that before rc scripts did all those things, but without any unified and standardized primitives. systemd provides those for a lot of common use cases - and that's how it ended up doing so much - because our rc scripts used to do that. systemd actually does less because unit files are not scripts (but one can hook any external program in a lot of places, if they need to).
Also, as I understand it, systemd isn't a single piece of software either, it's a whole project with a bunch of semi-autonomous pieces, tied with common conventions and assumptions. It's difficult to define the scope for an umbrella project, and I believe individual pieces all have their scopes relatively well-defined.
>> The single, overarching problem with systemd is that it attempts, in every possible way, to do more instead of less.
The more systemd does, the better I like the power, features, consistency and functionality my linux systems have.
systemd is a beautiful work of art.
Software developers owe it to themselves to full understand what systemd has to offer, because if you really understand it then you'll find you can architect your systems around systemd and often let it to the hard work for you.
If anything, the old gnu/linux name should be dropped and replaced with linux/systemd to illustrate that in fact systemd is really a major part of what Linux is.
The more systemd does, the better I like the power, features, consistency and functionality my linux systems have.
systemd is a beautiful work of art.
Software developers owe it to themselves to full understand what systemd has to offer, because if you really understand it then you'll find you can architect your systems around systemd and often let it to the hard work for you.
If anything, the old gnu/linux name should be dropped and replaced with linux/systemd to illustrate that in fact systemd is really a major part of what Linux is.
> systemd is a beautiful work of art.
This reminds me of teenagers of the last generations who preferred distorted mp3 sound to original because that's the sound they have grew up with. Teenagers of today prefer distorted wide-angle selfies instead of properly shot faces because again that's what they have grew up with.
Calling systemd a work of art gives me the same vibe.
This reminds me of teenagers of the last generations who preferred distorted mp3 sound to original because that's the sound they have grew up with. Teenagers of today prefer distorted wide-angle selfies instead of properly shot faces because again that's what they have grew up with.
Calling systemd a work of art gives me the same vibe.
I can't tell if it's sarcasm or genuine, but I 90% agree. There's a lot of things that it makes easier. But there are areas which are just weird and overengineered and overcomplicated - for example resolved interacting with old files and creating lots of possible ways it can run. If you look at forums / SO it's obvious people are confused about what it tries to achieve. Instead of creating a whole new system, they could've started by implementing and standardizing the multi-interface behaviour in established projects like dnsmasq. Instead of that they went with NIH where it really wasn't necessary.
For resolve.conf, I think it is fundamentally bad design that is hard to get right. Before systemd, I've spent plenty of time trying to get networkmanager and resolvconf to play together. And then there was those dhcp scripts which would edit the files directly.
Also, what do you mean my "standardizing dnsmasq behaviour?" From my recollection of using it (and from manpage[0]), it's pretty basic. You (1) manually edit /etc/resolve.conf to point to localhost dnsmasq instance (2) use app-specific ways to configure nameservers, in dnsmasq's case it's an ad-hoc list of secondary resolv.conf files specified on command line, with no defaults. I don't think this is worth standardizing, as this scheme cannot even represent network interface -> dns entry associations.
https://thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html
Also, what do you mean my "standardizing dnsmasq behaviour?" From my recollection of using it (and from manpage[0]), it's pretty basic. You (1) manually edit /etc/resolve.conf to point to localhost dnsmasq instance (2) use app-specific ways to configure nameservers, in dnsmasq's case it's an ad-hoc list of secondary resolv.conf files specified on command line, with no defaults. I don't think this is worth standardizing, as this scheme cannot even represent network interface -> dns entry associations.
https://thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html
I didn't say standardizing dnsmasq behaviour. I said multi home/interface which was one of the big reasons to write resolved. They could've added the missing feature to the 2 most popular daemons and use one of them as a default, but instead went with a whole new system written from scratch.
> If you look at forums / SO it's obvious people are confused about what it tries to achieve.
Well, many people are confused about how DNS resolving works. Many of do not consider a problem, when they configure their resolv.conf to two different upstreams, each returning different answers (i.e. one responding for domains that the other doesn't) and then wondering, why stuff is broken.
Well, many people are confused about how DNS resolving works. Many of do not consider a problem, when they configure their resolv.conf to two different upstreams, each returning different answers (i.e. one responding for domains that the other doesn't) and then wondering, why stuff is broken.
Of course that's a bit of a troll comment at the end there. There are plenty of Linux distros still not using systemd. https://www.devuan.org/ as one example.
Is this a bit?
Trying to compare Windows updates favorably to updates on systemd seems a bit bizarre. I can't ever remember seeing an upgrade ask for a reboot on Raspberry PI OS (debian 12). And I can't ever remember seeing a Windows update that didn't reboot the system.
As a long-time Windows developer (both apps and device drivers) substantially coming to Linux long after the decision to use systemd was made, I am impressed as heck by Systemd.
And was completely horrified by my very limited experience, as a user, with pre-systemd Linuxes. That was insane. Manually edit what in to which system file? And my system invariably didn't even HAVE that file.
Even writing systemd units is virtually effortless.
As a long-time Windows developer (both apps and device drivers) substantially coming to Linux long after the decision to use systemd was made, I am impressed as heck by Systemd.
And was completely horrified by my very limited experience, as a user, with pre-systemd Linuxes. That was insane. Manually edit what in to which system file? And my system invariably didn't even HAVE that file.
Even writing systemd units is virtually effortless.
I get frustrated when people misstate the Unix philosophy.
The Unix philosophy is about function, not form. Say you have a tool to send something over a network. In 1993, that tool can get by being quite simple. In 2024, that same tool now needs to be more complex, just to do the same 'one job' and to 'do it well'.
It's a consistent insistence that the 'single job' be defined today just as it was in 1980 that results in Linux requiring hundreds of packages to be even remotely usable. Oh, you want Wi-Fi? Well, that's not really the job of the networking stack, the job of the networking stack is to connect PDP-11s together...
The Unix philosophy is about function, not form. Say you have a tool to send something over a network. In 1993, that tool can get by being quite simple. In 2024, that same tool now needs to be more complex, just to do the same 'one job' and to 'do it well'.
It's a consistent insistence that the 'single job' be defined today just as it was in 1980 that results in Linux requiring hundreds of packages to be even remotely usable. Oh, you want Wi-Fi? Well, that's not really the job of the networking stack, the job of the networking stack is to connect PDP-11s together...
There is systemd the init and there is systemd the project.
I counted no less then 64 related executables related to systemd on my system.
Accusing systemd of "not doing one thing and doing it well" is like accusing the GNU project of not following the Unix philosophy.
I counted no less then 64 related executables related to systemd on my system.
Accusing systemd of "not doing one thing and doing it well" is like accusing the GNU project of not following the Unix philosophy.
You think systemd-resolvd does its job well? :p
Actually, yes.
It is currently the only linux resolver stub, that supports link specific upstream resolvers. In the terms that user would understand, your split-horizon DNS works correctly over your VPN.
It is currently the only linux resolver stub, that supports link specific upstream resolvers. In the terms that user would understand, your split-horizon DNS works correctly over your VPN.
> Oh, you want Wi-Fi? Well, that's not really the job of the networking stack, the job of the networking stack is to connect PDP-11s together...
WiFi absolutely should be a separate component; the unix philosophy is an excellent defense against servers in a datacenter trying to install a WiFi stack.
WiFi absolutely should be a separate component; the unix philosophy is an excellent defense against servers in a datacenter trying to install a WiFi stack.
[deleted]
wifi happens to be a separate component; wpa_supplicant or iwl are separate user space tools that network management tools call into; on the kernel side, the drivers and their stack are separate modules.
Yes, as it should be; while we might quibble about the exact details, I think the Linux-world network stack(s) do a good job of separating components cleanly while still giving users a good experience (if you just install Debian on your laptop, you don't need don't need to know or care how many packages it takes to make networking work, it just does).
The Unix philosophy is about function, not form. Say you have a tool to send
something over a network. In 1993, that tool can get by being quite simple.
In 2024, that same tool now needs to be more complex, just to do the same
'one job' and to 'do it well'.
A core part of the Unix philosophy is being able to pipe standard streams between processes, files, sockets, etc, as the user sees fit.
Systemd, particularly journald, violates this severely by snarfing any stream you want to send to /var/log or wherever and locking it in a binary database somewhere, that you have to use journalctl to access.
I have no problem with journald as an idea, but the audacity that it knows better than the user what to do with stdout/stderr is insane, and 100% anti-Unix.
Systemd, particularly journald, violates this severely by snarfing any stream you want to send to /var/log or wherever and locking it in a binary database somewhere, that you have to use journalctl to access.
I have no problem with journald as an idea, but the audacity that it knows better than the user what to do with stdout/stderr is insane, and 100% anti-Unix.
> In 1993, that tool can get by being quite simple. In 2024, that same tool now needs to be more complex, just to do the same 'one job' and to 'do it well'.
1993's sendmail is more complex than qmail. The latter uses a modular system, where each subtool does its job well, rather than sendmail's monolith design.
1993's sendmail is more complex than qmail. The latter uses a modular system, where each subtool does its job well, rather than sendmail's monolith design.
The single major flaw with systemd...
Linux was(is?) the system. Shell scripts could configure the system, reasonably well. Systemd meant it was no longer possible to simply fix a linux distro that had it...a massive recompile and intricate understanding of many low level systems was needed.
No, you shouldn't need to know about PDP-11s to fix your buggy wifi, neither should you need to know a massive binary subsystem to configure your wifi either...
So as far as doing its "one job", it did/does it fairly poorly, even today. It is a lot more standard and it does get points from me in that respect, but it fails at being a configurable system completely.
Linux was(is?) the system. Shell scripts could configure the system, reasonably well. Systemd meant it was no longer possible to simply fix a linux distro that had it...a massive recompile and intricate understanding of many low level systems was needed.
No, you shouldn't need to know about PDP-11s to fix your buggy wifi, neither should you need to know a massive binary subsystem to configure your wifi either...
So as far as doing its "one job", it did/does it fairly poorly, even today. It is a lot more standard and it does get points from me in that respect, but it fails at being a configurable system completely.
I've fixed a lot of problems on a lot of systems and never needed to recompile systemd. The majority of them I've fixed by editing text files.
What kinds of problems are you hitting where you need to recompile systemd?
What kinds of problems are you hitting where you need to recompile systemd?
I could simply trace shell scripts with some file io, or echo statements. Now there are systems using python, even better I can set breakpoints and modify, no need to recompile.
Configuring the system with textfiles only works if the system anticipated all your needs, and to anticipate all needs is not only complex but error prone.
Configuring the system with textfiles only works if the system anticipated all your needs, and to anticipate all needs is not only complex but error prone.
Systemd unit files can exec shell scripts…
There have been times where I needed to do something that wasn’t available in the unit file syntax, so I just do it in a shell script and ExecStart that.
There have been times where I needed to do something that wasn’t available in the unit file syntax, so I just do it in a shell script and ExecStart that.
I suppose, can you arbitrarily replace portions of systemd with scripts, though?
If so, perhaps it is less of a problem than it used to be.
It still doesn't fit every need but it's better if it can get out of the way when told to.
If so, perhaps it is less of a problem than it used to be.
It still doesn't fit every need but it's better if it can get out of the way when told to.
I don't think systemd is meant to be fixed or modified, unless you are actually contributing to the upstream systemd project and have accepted that it will be a time consuming project.
It seems to be made for, and works well for, a modern ephemeral infrastructure workflow.
Build stuff to work with an unmodified distro, wipe it and reinstall if it goes wrong.
It seems to be made for, and works well for, a modern ephemeral infrastructure workflow.
Build stuff to work with an unmodified distro, wipe it and reinstall if it goes wrong.
I was indifferent to systemd until podman shipped quadlet. Now being able to bring up containers, tied to the system, in a reliable and efficient manner, without docker compose or big bespoke commands, has been incredible
Every time systemd comes up everyone says it's great and how much better it is than SysVinit. No one EVER talks about the actual alternatives that people use like OpenRC, runit, s6.
By all means, be the change you want to see. All I know about OpenRC is that it doesn't support user services the same way systemd does. I know even less about runit or s6, so I'd love to hear more from people who use them in their daily computing. Because unfortunately the impression I get from never seeing anyone talk about OpenRC, Shepherd, Upstart, et cetera; is that either nobody uses them, or they don't add anything worth noting over SysVinit.
Though I'm surprised that you say:
> Every time systemd comes up everyone says it's great and how much better it is
When my experience is usually that people complain about systemd and it being over-engineered and doing too much and having too many components (yes, I know that those projects are related to systemd, not part of it, I'm just saying what I see people mention as downsides).
Though I'm surprised that you say:
> Every time systemd comes up everyone says it's great and how much better it is
When my experience is usually that people complain about systemd and it being over-engineered and doing too much and having too many components (yes, I know that those projects are related to systemd, not part of it, I'm just saying what I see people mention as downsides).
OpenRC just works for me (Alpine) there’s a declarative side that’s more than enough for simple services, but you can override logic for starting, stopping, reloading,… as it’s just a shell script.
> unfortunately the impression I get from never seeing anyone talk about OpenRC, Shepherd, Upstart, et cetera; is that either nobody uses them
People do use them. Multiple distros either default to one of those, or support it: Alpine, Devuan, MX Linux, Slackware, Void, Guix, Gentoo and others.
> or they don't add anything worth noting over SysVinit.
Not looked at this myself, but do they mostly not add something - dependencies, easier config or something?
People do use them. Multiple distros either default to one of those, or support it: Alpine, Devuan, MX Linux, Slackware, Void, Guix, Gentoo and others.
> or they don't add anything worth noting over SysVinit.
Not looked at this myself, but do they mostly not add something - dependencies, easier config or something?
systemd is bringing Linux feature parity to Windows NT in the mid-90s. That's 30 years ago for those of you keeping score at home.
Funny the author mentions Solaris because Solaris has SMF and other systemd-similar tools.
Linux pre-systemd looked like a kernel duct taped to a bunch of random thrown-together shell scripts... because it was. A sorry state of affairs that a bunch of very stubborn people were afraid to let go.
The simple fact that configuring networking out of the box is completely different on every distro and sometimes within distros depending on which tools were loaded, in 2024, is simply insane.
Funny the author mentions Solaris because Solaris has SMF and other systemd-similar tools.
Linux pre-systemd looked like a kernel duct taped to a bunch of random thrown-together shell scripts... because it was. A sorry state of affairs that a bunch of very stubborn people were afraid to let go.
The simple fact that configuring networking out of the box is completely different on every distro and sometimes within distros depending on which tools were loaded, in 2024, is simply insane.
That's what makes Linux great. The entire Linux user space is replaceable. You can trash it all and make your own if you want to. You can trash systemd, write your own program and boot Linux directly into it if that's what you want. And people do, myself included. Not a single other operating system lets you do this. There's always some component that you cannot touch.
I'd rather have the chaos of Linux distributions than the tyranny of "standard" ways of doing things.
I'd rather have the chaos of Linux distributions than the tyranny of "standard" ways of doing things.
> You can trash it all and make your own if you want to.
It is important to note, that if you want to do it, you have to roll up your sleeves and do it. Not to expect free manpower of others to provide it to you.
Or in another words: http://islinuxaboutchoice.com/
It is important to note, that if you want to do it, you have to roll up your sleeves and do it. Not to expect free manpower of others to provide it to you.
Or in another words: http://islinuxaboutchoice.com/
Absolutely.
The biggest reason systemd is taking over is companies like Red Hat keep paying people to work on it. As time passes, systemd gets better, gets more features, makes the life of distribution maintainers easier and therefore gets adopted, and makes the lives of users easier too, mine included.
I'm making my own freestanding Linux thingy but I'm not under any illusions that it's going to replace systemd or any other software any time soon. I'm just one guy doing it in his free time because he enjoys it.
I really hate it when I see people complaining about systemd. The people behind systemd had a vision, they reimagined Linux user space and put the necessary work into making their vision a reality. They're an inspiration for me. Not only that, they took their work and they released it as GPL licensed free software. One would think the users would be more thankful. One would think people would not have moral objections to free software getting released to the world. Some technical criticism is certainly warranted but the sheer hatred that people display towards it and its developers certainly is not.
I agree that "Linux is about choice" is often misinterpreted. It's about my choice as a programmer and software developer to redefine the entire system downstream of the kernel however I want. Users are encouraged to themselves become technical and educated so that they too can enjoy this power. However, we're not about to drop everything and start maintaining some legacy system because people who aren't paying us keep insisting that it's how things gotta be done. We're programmers, we like it when things are made in our image.
The biggest reason systemd is taking over is companies like Red Hat keep paying people to work on it. As time passes, systemd gets better, gets more features, makes the life of distribution maintainers easier and therefore gets adopted, and makes the lives of users easier too, mine included.
I'm making my own freestanding Linux thingy but I'm not under any illusions that it's going to replace systemd or any other software any time soon. I'm just one guy doing it in his free time because he enjoys it.
I really hate it when I see people complaining about systemd. The people behind systemd had a vision, they reimagined Linux user space and put the necessary work into making their vision a reality. They're an inspiration for me. Not only that, they took their work and they released it as GPL licensed free software. One would think the users would be more thankful. One would think people would not have moral objections to free software getting released to the world. Some technical criticism is certainly warranted but the sheer hatred that people display towards it and its developers certainly is not.
I agree that "Linux is about choice" is often misinterpreted. It's about my choice as a programmer and software developer to redefine the entire system downstream of the kernel however I want. Users are encouraged to themselves become technical and educated so that they too can enjoy this power. However, we're not about to drop everything and start maintaining some legacy system because people who aren't paying us keep insisting that it's how things gotta be done. We're programmers, we like it when things are made in our image.
>the sheer hatred that people display towards it and its developers certainly is not [warranted].
Especially when many of the complaints can be summarized as, "I can no longer use my years of experience and knowledge about the old init system."
(The people who complain that systemd unnecessarily enlarges an install's attack surface I have more sympathy for.)
Especially when many of the complaints can be summarized as, "I can no longer use my years of experience and knowledge about the old init system."
(The people who complain that systemd unnecessarily enlarges an install's attack surface I have more sympathy for.)
Configuring networking is different not only between distros but also between different versions of a single distro, say, Ubuntu. Every major release lately brings new networking configuration method while abandoning the previous. Is it frustrating? You bet. Thankfully I can still `apt install ifupdown` and it works, but who knows for how long? systemd was optional for a couple of releases as well (`apt install upstart`) but of course not anymore.
Funnily enough, the network configuration problem is not due to systemd -- which has nothing to do with it -- but with Canonical being Canonical. Someone here has a new toy (netplan), so why not force it on users, with some excuse.
For the same reason, why Canonical forces netplan (consistency between desktop, using NetworkManager and server, using systemd-network), Redhat standardized on NetworkManager instead.
For the same reason, why Canonical forces netplan (consistency between desktop, using NetworkManager and server, using systemd-network), Redhat standardized on NetworkManager instead.
Sure, it has nothing to do with systemd. The vibes are extremely similar though. "We don't care what you think we are kids with a new toy and we will shove it down your throat no matter what" kind of thing.
(I hate NetworkManager just like I hate netplan though. Both can get off my lawn.)
(I hate NetworkManager just like I hate netplan though. Both can get off my lawn.)
If you allow distros it's going to lead to diversity. Even Solaris has multiple incompatible distos since they hired Ian Murdock.
A central point of their thesis is that Systemd folks bullied others into accepting it.
That's not really the case, though. The Debian folks had a very transparent discussion and vote about using it:
https://www.debian.org/vote/2019/vote_002
That's not really the case, though. The Debian folks had a very transparent discussion and vote about using it:
https://www.debian.org/vote/2019/vote_002
I have to wonder if this is some kind of statistical bias based on the type of engagement.
There are some people who are vehemently opposed to systemd, for a variety of reasons with varying degrees of merit. I think it's fair to say that systemd's developers broadly ignored those people, figuring out (correctly) that if they convinced a majority of distribution maintainers, and made the user experience for the average user smooth enough, that was actually going to be successful. As it should be: distro maintainers get to pick how their distros work, and users determine which distros they use.
And systemd has a clear sell for those groups: distro maintainers can standardize their service scripts, often just using what an upstream package author can write agnostically to distro, and users can handle the majority of base system functionality without having to pick things like "which of these cron daemons should I install".
There are some people who are vehemently opposed to systemd, for a variety of reasons with varying degrees of merit. I think it's fair to say that systemd's developers broadly ignored those people, figuring out (correctly) that if they convinced a majority of distribution maintainers, and made the user experience for the average user smooth enough, that was actually going to be successful. As it should be: distro maintainers get to pick how their distros work, and users determine which distros they use.
And systemd has a clear sell for those groups: distro maintainers can standardize their service scripts, often just using what an upstream package author can write agnostically to distro, and users can handle the majority of base system functionality without having to pick things like "which of these cron daemons should I install".
I'd also add that systemd appeals to application developers (no daemon code, one file for all distros). And it appeals to sysadmins too (no more /etc conflicts on any package update! Modify any system service! etc..)
Which brings me to the question of which category systemd does not appeal to? I am guessing "minimalists" and "people who like things done the old way" - and those are simply not that numerous compared to others.
Which brings me to the question of which category systemd does not appeal to? I am guessing "minimalists" and "people who like things done the old way" - and those are simply not that numerous compared to others.
> pick things like "which of these cron daemons should I install".
And we're back at it: systemd seemingly has plans to eliminate cron. I was not impressed when I have been pulling my hair out on why certain jobs in Ubuntu still run despite being removed from cron files. Turns out, systemd now has 'timers'.
So, mark my words: a few Ubuntu releases down the line, cron will become an optional package and after a couple more it won't even be available.
Up next: sudo.
And we're back at it: systemd seemingly has plans to eliminate cron. I was not impressed when I have been pulling my hair out on why certain jobs in Ubuntu still run despite being removed from cron files. Turns out, systemd now has 'timers'.
So, mark my words: a few Ubuntu releases down the line, cron will become an optional package and after a couple more it won't even be available.
Up next: sudo.
Systemd has had timers since 2012: https://github.com/systemd/systemd/commit/36697dc0199e25f09b...
So if timers were gonna replace cron I'd imagine it'd have happened already. That said, those Canonical folks are pretty crazy, so who knows.
Sudo getting replaced by doas may not be the worst thing.
So if timers were gonna replace cron I'd imagine it'd have happened already. That said, those Canonical folks are pretty crazy, so who knows.
Sudo getting replaced by doas may not be the worst thing.
Many parts of systemd were available for long time but fortunately were not used in distros. And then out of the blue Canonical decided that socket activation for sshd is the way.
Yes. Systemd timers are an order of magnitude better for the average distro maintainer and the average user than the various ways of invoking and running cron.
cron worked just fine and seemingly was perfect for the job for decades and dozens of operating systems.
What happened?
What happened?
https://news.ycombinator.com/item?id=41578676
See the last point.
See the last point.
It didn’t and wasn’t.
> broadly ignored those people, figuring out (correctly) that if they convinced a majority
So, politics at its worst. Being ignored on a matter, especially when there is merit involved, can make someone a highly motivated hater.
So, politics at its worst. Being ignored on a matter, especially when there is merit involved, can make someone a highly motivated hater.
Surely for each individual distro, they get to make their own decision about how to build their system.
I can’t stop you from hating, but if your pitch is that distros shouldn’t be allowed to make their own choices about what to include if you disagree… I guess your option is to start your own distro.
No linux distro was forced to adopt systemd. They did it on the merits, from their determination.
I can’t stop you from hating, but if your pitch is that distros shouldn’t be allowed to make their own choices about what to include if you disagree… I guess your option is to start your own distro.
No linux distro was forced to adopt systemd. They did it on the merits, from their determination.
I have to strongly disagree on the last one. Systemd is pretty much a canonical (pun intended) example of a solution in search of a problem.
Systemd as PID1 is nice on merits. I wouldn't want to get back to sysv initscripts.
Everything else is incredibly hostile to practical unix philosophy and solely exists out of spite and politics. This is my opinion and I realize I'm in the minority.
Systemd as PID1 is nice on merits. I wouldn't want to get back to sysv initscripts.
Everything else is incredibly hostile to practical unix philosophy and solely exists out of spite and politics. This is my opinion and I realize I'm in the minority.
Why do you think Linux distros adopted systemd, if not because they thought it was a good choice for themselves and their users?
That's a good question tbf.
Systemd as PID1 got its merits, hands down. Even coming from a person so destructive as LP, systemd as PID1 is still a good product.
Perhaps everything else is feature creep and architecture astronauts[0] type of things.
0 https://www.joelonsoftware.com/2001/04/21/dont-let-architect...
Systemd as PID1 got its merits, hands down. Even coming from a person so destructive as LP, systemd as PID1 is still a good product.
Perhaps everything else is feature creep and architecture astronauts[0] type of things.
0 https://www.joelonsoftware.com/2001/04/21/dont-let-architect...
How does that account for the distros using systemd-networkd, timesyncd, resolved, etc, where they’re separate binaries and services, instead of their predecessors?
If the other binaries aren’t better for maintainers and their users, why are they replacing other tools with them?
If the other binaries aren’t better for maintainers and their users, why are they replacing other tools with them?
I think it's change for the sake of change. I think it's mostly politics coming down from Red Hat. How exactly I'm not sure, but I fail to see any sane reason behind replacing things that worked well for decades with new things that are oftentimes inferior on merits.
Anyway. systemd vs unix philosophy is an insurmountable ravine and I chose to stay in what I think is sane camp.
Anyway. systemd vs unix philosophy is an insurmountable ravine and I chose to stay in what I think is sane camp.
I think maybe it's worth considering the possibility that the things being replaced did not work well, and were replaced because the new options solved problems and user pain.
To pick on cron, just from my own experience:
"I wonder where this scheduled task is configured" -> "Lemme check /etc/crontab" -> "nope, I guess maybe /etc/cron.d?" -> "Nope, lemme check /etc/cron.hourly/daily/weekly" -> "hmm, not there... maybe I want `crontab -l`... or it's running as a user, lemme check that list and run through `crontab -l -u $USER"
Finding out that I didn't notice a cronjob was failing because the stock methodology for notifications is email, and relies on the system having SMTP configured and having MAILTO set in cron.
Realizing a cronjob wasn't running because some, but not all, of the cron options have character limitations like "the file name can't contain any dots".
The total lack of dependency management, so anything I wanted to run @reboot needed to handle its own dependency checks and sleeps to make sure I didn't say, try to run puppet before the network was up.
On the networking side, I started using Arch when the config flow was "edit bash arrays in /etc/rc.local". I've used netcfg, netctl, and now systemd-networkd. systemd-networkd is easier to reason about, easier to debug, more powerful, and more portable between systems.
To pick on cron, just from my own experience:
"I wonder where this scheduled task is configured" -> "Lemme check /etc/crontab" -> "nope, I guess maybe /etc/cron.d?" -> "Nope, lemme check /etc/cron.hourly/daily/weekly" -> "hmm, not there... maybe I want `crontab -l`... or it's running as a user, lemme check that list and run through `crontab -l -u $USER"
Finding out that I didn't notice a cronjob was failing because the stock methodology for notifications is email, and relies on the system having SMTP configured and having MAILTO set in cron.
Realizing a cronjob wasn't running because some, but not all, of the cron options have character limitations like "the file name can't contain any dots".
The total lack of dependency management, so anything I wanted to run @reboot needed to handle its own dependency checks and sleeps to make sure I didn't say, try to run puppet before the network was up.
On the networking side, I started using Arch when the config flow was "edit bash arrays in /etc/rc.local". I've used netcfg, netctl, and now systemd-networkd. systemd-networkd is easier to reason about, easier to debug, more powerful, and more portable between systems.
As much as I don't want to, I have to admit you have merit here on all points.
My question though: how did we manage to run infrastructure using cron for decades and did not feel any of that pain?
My take on the answer is this. Failures: we have /var/log/something for this and typically it's obvious something is not being executed. Many files: yes, there are different configs but don't you remember where you typically put your cronjobs? For example I always use user's crontab, so this is where I would look for the job spec.
Dependency check is one of the main reasons for systemd as PID1 if not the main, so hands down, that's valid and true.
My question though: how did we manage to run infrastructure using cron for decades and did not feel any of that pain?
My take on the answer is this. Failures: we have /var/log/something for this and typically it's obvious something is not being executed. Many files: yes, there are different configs but don't you remember where you typically put your cronjobs? For example I always use user's crontab, so this is where I would look for the job spec.
Dependency check is one of the main reasons for systemd as PID1 if not the main, so hands down, that's valid and true.
We did feel pain. You could mitigate the pain by never making mistakes (like by ensuring all your cron scripts log to somewhere in /var/log, or that you have alerting, or that you always put them in crontab).
But also what we consider to be painful has evolved. When all systems were to some extent a hand-gardened entity, maintained by some specific system administrator, it was easier for them to have a personal standard. It didn't matter if I used cron.d and you used crontab, as long as on our own systems we kept one method. Increasingly, servers are more ephemeral, more horizontally scaled, and more automated, and systemd timers are just easier to work with in that model.
In a decade or so, I'm sure we'll have further modified how we handle systems, and something else will come along that solves painful things about systemd.
But also what we consider to be painful has evolved. When all systems were to some extent a hand-gardened entity, maintained by some specific system administrator, it was easier for them to have a personal standard. It didn't matter if I used cron.d and you used crontab, as long as on our own systems we kept one method. Increasingly, servers are more ephemeral, more horizontally scaled, and more automated, and systemd timers are just easier to work with in that model.
In a decade or so, I'm sure we'll have further modified how we handle systems, and something else will come along that solves painful things about systemd.
Didn't Arch also adopt it as the defaults before anyone else.
I can't see that being a project that would be bullied easily. Especially not "bullied so that it was adopted before almost everyone else".
I can't see that being a project that would be bullied easily. Especially not "bullied so that it was adopted before almost everyone else".
Related, it's also blatantly ignoring that it's the first software that even allows interoperability in its space, and was thus in huge demand by both developers and administrators.
(Edit: Other anti-systemd rant sites sometimes mention these obliquely as "upstream dependencies" and "workplace-mandated", while ignoring the fact that those people have reasons to demand those things)
But HN really doesn't need to litigate this all again.
(Edit: Other anti-systemd rant sites sometimes mention these obliquely as "upstream dependencies" and "workplace-mandated", while ignoring the fact that those people have reasons to demand those things)
But HN really doesn't need to litigate this all again.
Conformity is not interoperability. If anything, systemd is like Chrome, so popular that it persuades people to drop interoperability outright and just run with its nonstandard APIs.
what standard is systemd replacing? One of the big reasons it's popular is that there was no standard way to do most of the things it does.
Upstart and OpenRC both came years prior. As a matter of course anything which was self contained and standardized could have allowed interoperability. Systemd just actually convinced everyone to standardize not least because its integration with other parts of systemd provided both a carrot: standardization and simplicity for distro maintainers and a stick: the extra work required to use only part of it or especially if one used none of it and was expected to both write ones one unit files AND integration with stuff like gnome which was and remains the most popular singular GUI.
what do you mean by interoperability?
I (and this applies both with my dev hat and with my admin hat) can now write a unit file, depending on a different unit file, that works the same across all distros (note that this includes random end users whose systems I don't directly have access to). This even works for services that start per-user; no more write-only modifications to user startup files or DE-specific autostart dirs.
Pre-systemd this was unthinkable.
(similar stories apply to non-systemd-proper parts of the systemd project, but I personally haven't used them as much)
Pre-systemd this was unthinkable.
(similar stories apply to non-systemd-proper parts of the systemd project, but I personally haven't used them as much)
in the bad old times, every linux distro implemented its init scripts in a slightly different way, often with distro specific tooling and commands for accomplishing most management tasks
systemd unit files require minimal modification by distro repackagers.
With SysV-init, it was/is necessary to modify /etc/init.d/servicenamed shell scripts to get consistent logging and respawning behavior. With SysV init it's not possible to add edges between things to say that networkd must be started before apached; there are 6 numeric runlevels and filenames like S86apache, which you then can't diff or debsums against because the package default is S80apache and Apache starts with A which alphabetically sorts before the N in networkd.
And then add cgroups and namespaces to everything in /etc/init.d please
With SysV-init, it was/is necessary to modify /etc/init.d/servicenamed shell scripts to get consistent logging and respawning behavior. With SysV init it's not possible to add edges between things to say that networkd must be started before apached; there are 6 numeric runlevels and filenames like S86apache, which you then can't diff or debsums against because the package default is S80apache and Apache starts with A which alphabetically sorts before the N in networkd.
And then add cgroups and namespaces to everything in /etc/init.d please
There were and are other options other than systemd and SysV years prior in fact. OpenRC and upstart predate it and s6 was not much later.
It's possible to pipe stdout to syslog through logger by modifying the exec args. But I'd rather take journald's word for it.
Supervisord (and supervisorctl) are like s6. I don't think any of these support a config file syntax for namespaces, cgroups, seccomp, or SyscallFilter. Though it's possible to use process isolation with any init system by modifying the init script or config file to call the process with systemd-nspawn (which is like chroot, which most SysV-init scripts also fail to do at all, though a process can drop privs and chroot at init regardless of init system).
systemd works in containers now without weird setup. Though is it better to run a 'sidecar' container (for e.g. certbot) than to run multiple processes in a container?
Supervisord (and supervisorctl) are like s6. I don't think any of these support a config file syntax for namespaces, cgroups, seccomp, or SyscallFilter. Though it's possible to use process isolation with any init system by modifying the init script or config file to call the process with systemd-nspawn (which is like chroot, which most SysV-init scripts also fail to do at all, though a process can drop privs and chroot at init regardless of init system).
systemd works in containers now without weird setup. Though is it better to run a 'sidecar' container (for e.g. certbot) than to run multiple processes in a container?
> It's basically an integrated redesign of all the low-level userspace of a Linux system
Yes, and that's okay. Whatever it is that came before it sucked so much I don't even remember what life was like before systemd.
Also, it's amazing that we even get to redesign Linux user space like that. That's something special. Not a single other operating system allows you to do this. You could boot Linux straight into your program if you want, without any user space at all. Linux is such a solid foundation it lets you swap out the entire user space, you can trash systemd and even the entire GNU stuff and just do your own thing. It's inspired me to start working on a redesign of my own.
Yes, and that's okay. Whatever it is that came before it sucked so much I don't even remember what life was like before systemd.
Also, it's amazing that we even get to redesign Linux user space like that. That's something special. Not a single other operating system allows you to do this. You could boot Linux straight into your program if you want, without any user space at all. Linux is such a solid foundation it lets you swap out the entire user space, you can trash systemd and even the entire GNU stuff and just do your own thing. It's inspired me to start working on a redesign of my own.
> You could boot Linux straight into your program if you want, without any user space at all.
That was possible from around day one of linux kernel existence.
That was possible from around day one of linux kernel existence.
Yes. However it's not at all obvious that there is a clean separation between Linux and user space.
You look up the manual entry on some Linux system call and it gives you the glibc documentation. You look up the init system in the manual, expecting details on what the kernel requires of PID 1, and it just gives you systemd documentation. This stuff is completely mixed up.
You look Linux up on Wikipedia and the article eventually starts telling you about standard C libraries as if they were part of Linux. In some articles there's even a diagram showing glibc wrapping the kernel, as if it was Linux's equivalent to the Win32 API.
https://upload.wikimedia.org/wikipedia/commons/4/45/Linux_ke...
In reality, you can trash that entire thing and remake the entire user space in Rust or Zig or Lisp if you want. No libraries are necessary.
You look up the manual entry on some Linux system call and it gives you the glibc documentation. You look up the init system in the manual, expecting details on what the kernel requires of PID 1, and it just gives you systemd documentation. This stuff is completely mixed up.
You look Linux up on Wikipedia and the article eventually starts telling you about standard C libraries as if they were part of Linux. In some articles there's even a diagram showing glibc wrapping the kernel, as if it was Linux's equivalent to the Win32 API.
https://upload.wikimedia.org/wikipedia/commons/4/45/Linux_ke...
In reality, you can trash that entire thing and remake the entire user space in Rust or Zig or Lisp if you want. No libraries are necessary.
If you haven’t seen this talk[0], I think it covers in a fair shake the positives, negatives, and what even a post-systemd world would have to deal with.
[0]: https://youtu.be/o_AIw9bGogo
[0]: https://youtu.be/o_AIw9bGogo
It wouldn't surprise me if Benno Rice wrote that talk in response to this exact blog post – though there were enough posts in this same genre at the time, and he could have been responding to any/all of them.
a fair shake
I would think a BSD developer has a good idea of the challenges facing OS development and the implementation of an init system.
I do not think it is charitable to them to say they think only Linux matters. If that’s the case why are they a FreeBSD developer?
I do not think it is charitable to them to say they think only Linux matters. If that’s the case why are they a FreeBSD developer?
Have you not watched the video then? There are a couple sections on why portability is not important because only Linux matters. Benno's words:
The reality is UNIX is dead… these days I think you've basically got Linux and some rounding errors.
Look, I use linux, OpenBSD, Apple and even Microsoft operating systems. I thought the video provided a unique perspective to the systemd discourse that I thought people would be interested in seeing if they hadn’t already.
I couldn’t care less if Debian announced that Debian 13 is going to default to s6 or runit as the init system. I have no loyalty to systemd other than the Linux distros I use have integrated with it.
The linked article introduced me to s6 which is interesting. I do think there is value in simpler codebases a la wireguard, toybox, and doas. None of the hostilities towards one init project over the other have ever made sense to me but “the battle” was mostly fought before I got into Linux.
I couldn’t care less if Debian announced that Debian 13 is going to default to s6 or runit as the init system. I have no loyalty to systemd other than the Linux distros I use have integrated with it.
The linked article introduced me to s6 which is interesting. I do think there is value in simpler codebases a la wireguard, toybox, and doas. None of the hostilities towards one init project over the other have ever made sense to me but “the battle” was mostly fought before I got into Linux.
What's unique about it? That Benno tosses out a reference to BSD here and there? Otherwise it's just a video of someone dismissing criticism out of hand. Could've gotten that from Lennart.
The author seemed to be arguing at length about some sort of technical shortcomings, as though an argument had been made.
The author also makes it sound like something has been taken away from them - one can rip it out if they really want to. It's just that most of us have our focuses elsewhere and have no need to tinker with systemd.
My issue with the "systemd violates the unix philosophy" argument is that it... doesn't. I see systemd-resolved touted as an example of this. systemd-resolved is a different tool to systemd; it does one thing and does it well.
Might as well say GNU violates unix philosophy because it has text processors AND a shell.
Might as well say GNU violates unix philosophy because it has text processors AND a shell.
Related:
A word about systemd - https://news.ycombinator.com/item?id=19077561 - Feb 2019 (17 comments)
A word about systemd - https://news.ycombinator.com/item?id=19077561 - Feb 2019 (17 comments)
As a declarative process supervisor i really like systemd. I don't like all of the other things it wants to do though.
Server seems to have received the ol' HN hug. Archive:
https://web.archive.org/web/20240917143933/https://skarnet.o...
https://web.archive.org/web/20240917143933/https://skarnet.o...
Kind of a non-sequitur, but what is actually the point of systemd (and other similar tools)? Like compared to just spawning a bash processes that run a critical commands in a loop.
I can come up with a couple functions, but none of them seem like core things, more like tangential benefits:
- Startup ordering means you don't get log spam with "can't connect to X" until X starts
- Certain dependencies like mounts look the same whether they're "running" or not, so not starting before the mount is important. Or like, if a program enumerates interfaces or something, making sure all the interfaces exist first (via a dependency). Of course, a program could do its own checks and possibly be more stateless about it.
- As a standard, for packaging - you can distribute a service/socket file with your program and most distros can just plug it in and expect it to work (also thanks to having standard targets and directories).
Otherwise it seems like an ugly declarative DSL for gluing programs together, which you could do in bash or python or whatever other language too. Is reducing log spam really the point? FWIW I do think systemd is better than the alternatives ATM...
I can come up with a couple functions, but none of them seem like core things, more like tangential benefits:
- Startup ordering means you don't get log spam with "can't connect to X" until X starts
- Certain dependencies like mounts look the same whether they're "running" or not, so not starting before the mount is important. Or like, if a program enumerates interfaces or something, making sure all the interfaces exist first (via a dependency). Of course, a program could do its own checks and possibly be more stateless about it.
- As a standard, for packaging - you can distribute a service/socket file with your program and most distros can just plug it in and expect it to work (also thanks to having standard targets and directories).
Otherwise it seems like an ugly declarative DSL for gluing programs together, which you could do in bash or python or whatever other language too. Is reducing log spam really the point? FWIW I do think systemd is better than the alternatives ATM...
I'm not sure what you're asking.
On any system, something has to start "first", and be in charge of figuring out what the heck happens next. On modern Linux systems (and I'm oversimplifying massively), the hardware hands off to the kernel which launches some kind of init system the first userspace thing. That could be sysvinit or upstart or s6 or systemd or any number of other things. You could write it in any language, though the nature of when it has to launch strongly incentivizes writing it in some kind of compiled language so you aren't having to make sure the right interpreter bits are in place first.
That thing then, depending on its level of complexity, gets to handle things like "what else do I launch" and "what order to I launch it in" and "what do I do if a service I started stops on its own".
Beyond that, your life is way easier if more than one distro uses the same thing to do this, so that an upstream package author can write "this is how an init system launches my service" and distro maintainers can use that, vs having to create and maintain their own scripts for how their init system launches a service.
So, the "point" of init systems is "something has to be in charge of userspace".
On any system, something has to start "first", and be in charge of figuring out what the heck happens next. On modern Linux systems (and I'm oversimplifying massively), the hardware hands off to the kernel which launches some kind of init system the first userspace thing. That could be sysvinit or upstart or s6 or systemd or any number of other things. You could write it in any language, though the nature of when it has to launch strongly incentivizes writing it in some kind of compiled language so you aren't having to make sure the right interpreter bits are in place first.
That thing then, depending on its level of complexity, gets to handle things like "what else do I launch" and "what order to I launch it in" and "what do I do if a service I started stops on its own".
Beyond that, your life is way easier if more than one distro uses the same thing to do this, so that an upstream package author can write "this is how an init system launches my service" and distro maintainers can use that, vs having to create and maintain their own scripts for how their init system launches a service.
So, the "point" of init systems is "something has to be in charge of userspace".
Sorry, I guess my question wasn't "why does the tree need a root process" but "what is the critical feature that makes it worth using these tools instead of the dumbest possible alternative?"
So the dumb response to your questions is if I have a list of services: "what else do I launch?" everything, "what order do I launch it in?" launch it all at the same time, "what do I do if a service I started stops on its own?" restart it. I typoed, but I really meant my first sentence verbatim: suppose init is a bash script that just does:
It's largely touted that systemd's killer feature is that it represents services as a graph. But why is it important that it operates as a graph? Why is ordering startup important? Or does that not matter, and the real reason systemd matters is that it makes packaging easier (as you said, and as I said) and it has lots of built in conveniences? I have my guesses, but I'd like outside opinions.
So the dumb response to your questions is if I have a list of services: "what else do I launch?" everything, "what order do I launch it in?" launch it all at the same time, "what do I do if a service I started stops on its own?" restart it. I typoed, but I really meant my first sentence verbatim: suppose init is a bash script that just does:
(while true; do service1 &> log1.txt ; sleep 5; done) &
(while true; do service2 &> log1.txt ; sleep 5; done) &
(while true; do service3 &> log1.txt ; sleep 5; done) &
It's largely touted that systemd's killer feature is that it represents services as a graph. But why is it important that it operates as a graph? Why is ordering startup important? Or does that not matter, and the real reason systemd matters is that it makes packaging easier (as you said, and as I said) and it has lots of built in conveniences? I have my guesses, but I'd like outside opinions.
This very much reminds me of how old Slackware systems used to operate. It was workable, but had a number of problems:
(1) I want to stop "service1" for a few minutes, without affecting other services, how do I do this?
(2) I want to disable "service1" completely, or pass it an extra parameter, without affecting other services. How do I do this? Note you cannot edit the init file and restart, this'll restart all the other services as well.
(3) I just typed "my-package-manager install service4", and I want service4 to start right away (and naturally have full functionality - keep restarting on failure, save logs, etc..). How do I do this?
(4) service2 had a emitted too many log lines, and my log disk is full. How do I clean existing logs without restarting the service? (Note even if you remove the file, it'll still take up the disk space)
(5) You were debugging service3 performance, and noticed it has printed "ERROR out of foobars" to its log but didn't print the timestamp.. When did this happen?
(6) service2 needs to be started after NFS volume is mounted. service3 is responsible for cpu fans and should be started as early as possible. service1 will re-generate the userdb if the data directory is missing, so it should not be started automatically if that's the case.
(7) new version of service1 changes command-line and default restart interval. How do I apply this change to my heavily customized startup file?
I could go on and on. All of those are real problems, and before systemd, they were solved with tons of bespoke code. Either C/C++ code (making a trivial daemons complex beasts), or shell code of init system, with was typically highly distribution dependent. And of course it was fragile and full of bugs.
(1) I want to stop "service1" for a few minutes, without affecting other services, how do I do this?
(2) I want to disable "service1" completely, or pass it an extra parameter, without affecting other services. How do I do this? Note you cannot edit the init file and restart, this'll restart all the other services as well.
(3) I just typed "my-package-manager install service4", and I want service4 to start right away (and naturally have full functionality - keep restarting on failure, save logs, etc..). How do I do this?
(4) service2 had a emitted too many log lines, and my log disk is full. How do I clean existing logs without restarting the service? (Note even if you remove the file, it'll still take up the disk space)
(5) You were debugging service3 performance, and noticed it has printed "ERROR out of foobars" to its log but didn't print the timestamp.. When did this happen?
(6) service2 needs to be started after NFS volume is mounted. service3 is responsible for cpu fans and should be started as early as possible. service1 will re-generate the userdb if the data directory is missing, so it should not be started automatically if that's the case.
(7) new version of service1 changes command-line and default restart interval. How do I apply this change to my heavily customized startup file?
I could go on and on. All of those are real problems, and before systemd, they were solved with tons of bespoke code. Either C/C++ code (making a trivial daemons complex beasts), or shell code of init system, with was typically highly distribution dependent. And of course it was fragile and full of bugs.
Thanks! This is exactly the sort of stuff I was looking for.
So maybe this is too reductive, but basically:
- Tools for mutation of a running system
- Dealing with hard dependencies that don't cause errors when unavailable (like nfs in your example, or fan control, but also enumerating devices as I mentioned earlier, or for security, firewall rules since firewall starts entirely permissive)
- Standard packaging (edit: er, composability via standard packaging, or modularity with standard packaging for modules)
I think the logging stuff (datetimes, clearing, etc) is probably more the domain of a logger like journald, whose purpose seems is more concrete to me.
So maybe this is too reductive, but basically:
- Tools for mutation of a running system
- Dealing with hard dependencies that don't cause errors when unavailable (like nfs in your example, or fan control, but also enumerating devices as I mentioned earlier, or for security, firewall rules since firewall starts entirely permissive)
- Standard packaging (edit: er, composability via standard packaging, or modularity with standard packaging for modules)
I think the logging stuff (datetimes, clearing, etc) is probably more the domain of a logger like journald, whose purpose seems is more concrete to me.
> Why is ordering startup important?
This is pretty fundamental stuff. You can't run a program if the disk that it's sitting on hasn't been made accessible yet. You can't start a network service that listens on a certain IP address if that IP address hasn't been configured yet, or if the destination for service logs isn't ready, or if our service needs to talk to some other service like a database which isn't running yet. Etc etc. Booting up a modern system to a graphical environment requires hundreds of things to happen in the right order.
As for everything else, the project announcement blog post from 2010 does a pretty great job summing up the shortcomings of doing "the simplest possible thing": https://0pointer.de/blog/projects/systemd.html . You can certainly still do things that way if you want, but most people value the benefits gained from a more intricate approach.
This is pretty fundamental stuff. You can't run a program if the disk that it's sitting on hasn't been made accessible yet. You can't start a network service that listens on a certain IP address if that IP address hasn't been configured yet, or if the destination for service logs isn't ready, or if our service needs to talk to some other service like a database which isn't running yet. Etc etc. Booting up a modern system to a graphical environment requires hundreds of things to happen in the right order.
As for everything else, the project announcement blog post from 2010 does a pretty great job summing up the shortcomings of doing "the simplest possible thing": https://0pointer.de/blog/projects/systemd.html . You can certainly still do things that way if you want, but most people value the benefits gained from a more intricate approach.
Ah, thanks for the link! So it has an interesting point, which is socket/device activation. And services that aren't started immediately could also have dependencies you don't want to start immediately.
I can see that being important for a desktop, where you're plugging/unplugging stuff and want startup to be fast.
> You can certainly still do things that way if you want
I think you're putting words in my mouth.
I can see that being important for a desktop, where you're plugging/unplugging stuff and want startup to be fast.
> You can certainly still do things that way if you want
I think you're putting words in my mouth.
Sorry, I didn't mean you specifically. I meant, if someone values simplicity above all else, they can choose an init that matches those values (like the one you described). Your approach is perfectly valid given that set of values.
Debian and other distros asked their userbase which values to prioritize, and the users chose what systemd provides. Hope that helps explain "why does systemd exist / why have most distros chosen it?"
Debian and other distros asked their userbase which values to prioritize, and the users chose what systemd provides. Hope that helps explain "why does systemd exist / why have most distros chosen it?"
Oh, that answer is bigger but pretty boring: the dumbest possible alternative doesn't account for the complexity of real life systems.
Some examples:
On your desktop, while you launch programs through the window manager, things like the window manager itself, and also the network daemon, and time-keeping daemon, and the daemon that logs go to, etc etc, all need to be spun up by something.
Some of those daemons should restart themselves if they crash, and some of them shouldn't.
Some of those daemons misbehave, so it's nice to be able to tell them "don't eat all the RAM" that rendaw is trying to use to do whatever you're using the desktop for.
Some of those services shouldn't start until it's the right time in the sequence, but also you want booting to be fast, so where we can be parallel, it's nice to be parallel (this is where the graph comes in). And on the flip side, some of those services need to restart if a service they depend on restarts.
I've run s6 (the project built by skarnet, the author of the original post here) on a variety of systems. I would highly recommend it as a very high quality example of the kind of minimalist init system you're envisioning. And likewise, I'd encourage you to try it out! It's possible you'd find that for some/all of your use cases, the value of simplicity outweighs the value of the conveniences. But the reason that systemd has seen so much success on modern Linux distros is that for the average distro and average user, the conveniences in terms of functionality and standardization are worth it.
Some examples:
On your desktop, while you launch programs through the window manager, things like the window manager itself, and also the network daemon, and time-keeping daemon, and the daemon that logs go to, etc etc, all need to be spun up by something.
Some of those daemons should restart themselves if they crash, and some of them shouldn't.
Some of those daemons misbehave, so it's nice to be able to tell them "don't eat all the RAM" that rendaw is trying to use to do whatever you're using the desktop for.
Some of those services shouldn't start until it's the right time in the sequence, but also you want booting to be fast, so where we can be parallel, it's nice to be parallel (this is where the graph comes in). And on the flip side, some of those services need to restart if a service they depend on restarts.
I've run s6 (the project built by skarnet, the author of the original post here) on a variety of systems. I would highly recommend it as a very high quality example of the kind of minimalist init system you're envisioning. And likewise, I'd encourage you to try it out! It's possible you'd find that for some/all of your use cases, the value of simplicity outweighs the value of the conveniences. But the reason that systemd has seen so much success on modern Linux distros is that for the average distro and average user, the conveniences in terms of functionality and standardization are worth it.
Okay, I assumed there are services that shouldn't restart if they crash given that that's the systemd default, but do you have an example? I don't think I've ever encountered any, and I can't think of a situation where you'd want that.
If you have access to a systemd-based system, you can try:
grep -R Restart /etc/systemd/ /usr/lib/systemd/
One of the examples from my world is generally databases. If I’m running a large or clustered DB, I probably don’t want it to just blindly restart on a crash.
Oh interesting, here we have mysql on auto restart and IME it's critical to reliability since sometimes stuff happens and mysql goes down on a node.
I guess different databases might require more manual intervention to recover?
I guess different databases might require more manual intervention to recover?
I don't like systemd, but I don't like s6 either.
What we need is something simple that uses something easy like unit files. (Though I wouldn't choose the systemd unit file syntax.)
What we need is something simple that uses something easy like unit files. (Though I wouldn't choose the systemd unit file syntax.)
Here we go again.
I hate systemd. I sincerely and strongly believe it to be the worst thing that happened to the world of Linux. Note: this is just my opinion, of course I'm obviously wrong, can I have it please?
Having said that I have to admit that the battle has been lost and there are no practically suitable server distributions clear from it, let alone desktop. So the only fight worth fighting is to delay destruction of other parts of the OS. In Ubuntu you can easily remove systemd-resolved, get rid of ssh socket activation, bring back cronjobs to crond and install a sane NTP client. Hopefully, we will still have sudo available for many more years at least as an optional package and then as a third-party deb. And thankfully we still have logs in /var/log and journald is not a hard requirement for anything. Yet.
Otherwise resistance is futile as systemd people excel at politics and seemingly no opinion or input has any influence their decision making process.
Again: I'm totally wrong here and systemd is good.
I hate systemd. I sincerely and strongly believe it to be the worst thing that happened to the world of Linux. Note: this is just my opinion, of course I'm obviously wrong, can I have it please?
Having said that I have to admit that the battle has been lost and there are no practically suitable server distributions clear from it, let alone desktop. So the only fight worth fighting is to delay destruction of other parts of the OS. In Ubuntu you can easily remove systemd-resolved, get rid of ssh socket activation, bring back cronjobs to crond and install a sane NTP client. Hopefully, we will still have sudo available for many more years at least as an optional package and then as a third-party deb. And thankfully we still have logs in /var/log and journald is not a hard requirement for anything. Yet.
Otherwise resistance is futile as systemd people excel at politics and seemingly no opinion or input has any influence their decision making process.
Again: I'm totally wrong here and systemd is good.
>when said software is process 1 and basically the whole low-level userland layer, it is frightening.
So this is yet another person who apparently thinks all the different daemons made by the systemd project runs in the same process.
>Remember sendmail, BIND, INN, and, definitely a better analogy, the early days of Microsoft Windows ?
Honestly, no, I know next to nothing about any of these. Someone wanna fill me in?
So this is yet another person who apparently thinks all the different daemons made by the systemd project runs in the same process.
>Remember sendmail, BIND, INN, and, definitely a better analogy, the early days of Microsoft Windows ?
Honestly, no, I know next to nothing about any of these. Someone wanna fill me in?
[deleted]
This old chestnet eh?
Systemd being one big monolithic system locks you in to, frankly, just a lot of shitty decisions. Take for example the fact that it decides to bind to port 53 on localhost, for absolutely no good reason. Now if your response is that is not a big deal, or you push your glasses up on your nose and want to explain why its important, imagine for a second if it was port 80 or 443.
I have no real complaints about systemd as an init system. I have plenty to say about systemd not staying in its lane.
systemd's developers have formed a habit of deciding to implement something like an NTP client or DNS resolver, bringing a healthy dose of ignorance of how those services actually work, and then doubling down on their poor design decisions when problems are brought to their attention.
systemd's developers have formed a habit of deciding to implement something like an NTP client or DNS resolver, bringing a healthy dose of ignorance of how those services actually work, and then doubling down on their poor design decisions when problems are brought to their attention.
But that's systemd the project, not systemd the init system. You don't need to use any of those components to use systemd, which has no dependencies on them. Those services can live and die on their own merits.
It becomes default on all the main stream distros though.
Which should make you consider its validity. If all these distros chose it, independent of one another, there's gotta be a reason. The reason is that it's much, much simpler and more maintainable from a distro perspective and a sysadmin perspective.
Ultimately distros decide what software is on their distro and users decide what distros to use. So the solution here is to find a distro that meets your needs.
Ultimately distros decide what software is on their distro and users decide what distros to use. So the solution here is to find a distro that meets your needs.
They can ship as much trash as they like, so long as it doesn't become my problem.
Those components become my problem when the distributions I choose to use choose to adopt them, and then I have to work around them.
Those components become my problem when the distributions I choose to use choose to adopt them, and then I have to work around them.
Choose something else then. If there is one thing the world is not lacking it is Linux distros.
As a hobbyist, sure. It's not quite that simple when I have commercial apps I need to run in a regulated environment.
There's nothing about regulated environments or commercial apps that limit your choice to "systemd on Linux".
Everything is a tradeoff.
Everything is a tradeoff.
Those apps and the system's accreditation requirements limit my choice to a particular vendor's ecosystem so yes, they do.
Otherwise I could avoid Linux entirely if I so chose, and make the systemd consideration irrelevant.
Otherwise I could avoid Linux entirely if I so chose, and make the systemd consideration irrelevant.
Systemd doesn't.
The systemd monorepo also provides systemd-resolvd which does, but this is optional and last I checked not included or at least started by default on Debian, for example.
The systemd monorepo also provides systemd-resolvd which does, but this is optional and last I checked not included or at least started by default on Debian, for example.
Systemd-resolvd is started by default on my Ubuntu distro v24.
Good choice! It's great!
If you really don't want it, it's super easy to disable:
If you really don't want it, it's super easy to disable:
systemctl disable systemd-resolvedIf only it was that easy. Unfortunately, systemd-resolved is a sticky piece. You have to `apt-get uninstall` it, then `rm /etc/resolv.conf` and replace it with a sane file (`nameserver 1.1.1.1` or whatever).
For previous versions I had to `chattr +x` as well because systemd stuff of course will not let me have my resolver.
For previous versions I had to `chattr +x` as well because systemd stuff of course will not let me have my resolver.
No, disabling, or at works masking is enough.
`nameserver 1.1.1.1` is insane. Hope you don't ever have to use link-specific resolver.
`nameserver 1.1.1.1` is insane. Hope you don't ever have to use link-specific resolver.
Indeed. It has to be completely removed from the system and even then there is no guarantee it won't be brought back by some minor update in a couple of days. This is why I had to chattr +x in Ubuntu 22 and older. Not anymore in 24, but we'll see.
resolv.conf is an internal implementation detail of glibc. You are not supposed to touch it.
Congratulation to breaking your system.
Congratulation to breaking your system.
Wrong and wrong. It just needs to be stopped. And you can mask it to make sure it never ever ever is started by anyone again. C'mon man, 0 for 0 so far on your grumbles.
It absolutely wasn't enough to stop and mask it under ubuntu 22.04.
Do you know how to use netstat to see who is holding the port open? This seems like a major Problem Exists Between Keyboard And Chair issue. But also Ubuntu can be a special place. But if you stop the service & the port is still open... It's probably not that service, it's something else, and so throwing shade at systemd is again not accurate.
Ah. It might be a systemd-resolved.socket (or similarly named), allow socket activation. Systemd would warn you when you disable the service that it wouldn't be giving up the socket, that it could still be restarted. Because it's a great monorepo that's smart & courteous.
Ah. It might be a systemd-resolved.socket (or similarly named), allow socket activation. Systemd would warn you when you disable the service that it wouldn't be giving up the socket, that it could still be restarted. Because it's a great monorepo that's smart & courteous.
I think we're confusing two things here. ssh via socket activation is of course a brain damage done solely by Ubuntu. But systemd goes a long way to make sure it hurts, because `ps ax | grep sshd` will definitely show you that /usr/bin/sshd is running. Except it's not it. But maybe that's Ubuntu thing as well, not sure here.
`sudo netstat -nlp` or `sudo ss -nlp` (without root you won't know pid).
`systemctl status ssh` shows, that it is socket-activated:
`systemctl status ssh` shows, that it is socket-activated:
TriggeredBy: ● ssh.socket
Hindsight.
You would know to run netstat or lsof if you have expected something like that to happen. It certainly wouldn't ever cross my mind to have sshd socket-activated and systemd actively trying to hide the fact. This change by Canonical was out of the blue, totally unexpected.
You would know to run netstat or lsof if you have expected something like that to happen. It certainly wouldn't ever cross my mind to have sshd socket-activated and systemd actively trying to hide the fact. This change by Canonical was out of the blue, totally unexpected.
Link specific resolvers ONLY are insane.
Why am I forced into nothing but link specific resolvers?
Why cant I have a global default resolver anymore, with link specific as needed?
Why am I forced into nothing but link specific resolvers?
Why cant I have a global default resolver anymore, with link specific as needed?
Who said link specific resolvers ONLY?
You can have default one. Chances are, if you have only one configured, it will be default. How are you forced? Who did that? How?
Oth, with resolv.conf syntax, link-specific resolvers are impossible. Even the apple-like extension in /etc/resolver, they cannot respond to link doing up and down.
You can have default one. Chances are, if you have only one configured, it will be default. How are you forced? Who did that? How?
Oth, with resolv.conf syntax, link-specific resolvers are impossible. Even the apple-like extension in /etc/resolver, they cannot respond to link doing up and down.
maybe it's a skill issue on my part.
Can you tell me how to have a default global resolver with resolvd and network manager?
Should be pretty straightforward right?
It's a pretty basic task.
I looked all through the Man pages and couldn't find a way to do it with network manager.
Can you tell me how to have a default global resolver with resolvd and network manager?
Should be pretty straightforward right?
It's a pretty basic task.
I looked all through the Man pages and couldn't find a way to do it with network manager.
In NetworkManager, you set your DNS servers using `ipv{4,6}.dns`. Similarly, you set `ipv{4,6}.dns-search` with zones that you want to be looked up via DNS over this link. By default, the default DNS is the link with with default route. If you want to designate the default DNS via different link, set up the search for `~.`
Then `resolvectl status` will show you each link, what DNS is configured for it, what domains it will resolve and which one is the default.
Then `resolvectl status` will show you each link, what DNS is configured for it, what domains it will resolve and which one is the default.
that is a choice by ubuntu. it is not mandated by systemd itself.
You're right – if I imagine systemd doing something else than what it does, I don't like it!
resolvd was running by default on Ubuntu 24.
I was struggling with it while trying to troubleshoot a VPN connection because resolvd automates resolve.conf into a generated file from several other files and will overwrite any other functionality trying to add resolvers to resolve.conf, like an internal VPN company resolver for example.
Why is systemd even touching DNS?
Seems a little outside of it's scope.
This is kinda crazy to me personally when I found out resolvd was doing this.
https://www.man7.org/linux/man-pages/man8/systemd-resolved.s...
I was struggling with it while trying to troubleshoot a VPN connection because resolvd automates resolve.conf into a generated file from several other files and will overwrite any other functionality trying to add resolvers to resolve.conf, like an internal VPN company resolver for example.
Why is systemd even touching DNS?
Seems a little outside of it's scope.
This is kinda crazy to me personally when I found out resolvd was doing this.
https://www.man7.org/linux/man-pages/man8/systemd-resolved.s...
Ideally, with systemd-resolved, you would have only 127.0.0.53 in your resolv.conf. (Anything that reads it directly is broken anyway).
Then you configure your DNS using NetworkManager, on respective connections. I.e. on your VPN, you configure the DNS on the other side of the connection, and (optionally) which zones it is supposed to resolve. (NetworkManager then configures systemd-resolved accordingly). The usage of this DNS then goes up and down together with your VPN connection. That's it.
Then you configure your DNS using NetworkManager, on respective connections. I.e. on your VPN, you configure the DNS on the other side of the connection, and (optionally) which zones it is supposed to resolve. (NetworkManager then configures systemd-resolved accordingly). The usage of this DNS then goes up and down together with your VPN connection. That's it.
"Why is systemd even touching DNS?"
resolvd != systemd. It seems like you are mad that Ubuntu changed DNS resolver, but isn't that the actual job of a Linux distro?
resolvd != systemd. It seems like you are mad that Ubuntu changed DNS resolver, but isn't that the actual job of a Linux distro?
the full name of the service is systemd-resolvd...
yes, and full name of coreutils is GNU coreutils.
systemd-resolvd is in the literal systemd code base.
https://github.com/systemd/systemd/tree/main/src%2Fresolve
https://github.com/systemd/systemd/tree/main/src%2Fresolve
So are udev and nss modules. It's monorepo.
so if it's
1> in the systemd repo
2> if it's written and maintained by the same people who write systemd
3> if systemd is in the NAME
I don't really know what more is needed to say it a is part of systemd
1> in the systemd repo
2> if it's written and maintained by the same people who write systemd
3> if systemd is in the NAME
I don't really know what more is needed to say it a is part of systemd
After a quick check, in 100% of distributions checked, it is a separate package. In many of them, not even installed by default (which is too bad).
okayyy
I believe systemd-resolved is easily the absolute worst piece of systemd infrastructure. And especially so in Ubuntu! In 24.04 you can get away by uninstalling it then creating a sane /etc/resolv.conf; previous Ubuntu distros required a bit more than that.
Exactly - this is a perfect example of what has always bothered me about systemd.
Binding a DNS server to 127.0.01:53 is a perfect example of what has always bothered you about systemd? Seriously not getting it.
Providing DNS services on 127.0.0.1:53 has nothing to do with systemd. It's a convention used by NetworkManager, and associated network services. The very good reason being that network interfaces that are being brought up and down can choose to forward their DNS requests to 127.0.0.1, and allow a completely separate unit (resolved?) to deal with the details of where DNS requests get forwarded from there. (Or not, if they so choose).
Was there some OTHER burning compelling use-case that makes you want to use 127.0.0.1:53 for some other purpose?
Providing DNS services on 127.0.0.1:53 has nothing to do with systemd. It's a convention used by NetworkManager, and associated network services. The very good reason being that network interfaces that are being brought up and down can choose to forward their DNS requests to 127.0.0.1, and allow a completely separate unit (resolved?) to deal with the details of where DNS requests get forwarded from there. (Or not, if they so choose).
Was there some OTHER burning compelling use-case that makes you want to use 127.0.0.1:53 for some other purpose?
From the man pages:
"Additionally, systemd-resolved provides a local DNS stub listener on the IP addresses 127.0.0.53 and 127.0.0.54 on the local loopback interface"
https://www.man7.org/linux/man-pages/man8/systemd-resolved.s...
> Was there some OTHER burning compelling use-case that makes you want to use 127.0.0.1:53 for some other purpose?
The problem is... by default... resolvd routes ALL DNS REQUESTS to this stub by over writing resolv.conf with ONLY that socket and forcing you to configure resolvers elsewhere. And you have to manually reconfigure this if you want to change it and it forces you to get in the weeds way to much when all you want to do is add a DNS resolver.
https://www.man7.org/linux/man-pages/man8/systemd-resolved.s...
The confusing thing is why the heck is SYSTEMD an init system writing DNS resolvers? Seems out of it's scope in my opinion.
I was surprised resolvd existed when I was trying to troubleshoot a DNS error caused by it.
"Additionally, systemd-resolved provides a local DNS stub listener on the IP addresses 127.0.0.53 and 127.0.0.54 on the local loopback interface"
https://www.man7.org/linux/man-pages/man8/systemd-resolved.s...
> Was there some OTHER burning compelling use-case that makes you want to use 127.0.0.1:53 for some other purpose?
The problem is... by default... resolvd routes ALL DNS REQUESTS to this stub by over writing resolv.conf with ONLY that socket and forcing you to configure resolvers elsewhere. And you have to manually reconfigure this if you want to change it and it forces you to get in the weeds way to much when all you want to do is add a DNS resolver.
https://www.man7.org/linux/man-pages/man8/systemd-resolved.s...
The confusing thing is why the heck is SYSTEMD an init system writing DNS resolvers? Seems out of it's scope in my opinion.
I was surprised resolvd existed when I was trying to troubleshoot a DNS error caused by it.
resolved is neither started by default on systemd nor required in any way. Your distro chose that.
As most mainstream distros will just to fall in line with the standard.
unless you want to use Arch at some point it will be the default for most Linux.
unless you want to use Arch at some point it will be the default for most Linux.
Isn't mainstream distros all doing the same thing a Very Good Thing Indeed?
there is no standard. it's just a tool that ships with systemd that some folks use and some don't.
yeah just like systemd right?