Systemd 248(raw.githubusercontent.com)
raw.githubusercontent.com
Systemd 248
https://raw.githubusercontent.com/systemd/systemd/main/NEWS
236 comments
I'm a sysadmin myself and ever since I first worked with systemd when it was introduced in Arch, I fell in love with it and I've never understood why others of my profession hated it for replacing a cumbersome collection of shell scripts that previously were used for init/service management.
> replacing a cumbersome collection of shell scripts that previously were used for init/service management.
systemd-as-init was fine. systemd replacing udevd, syslog†, mount, ntpd, DNS, dhcpd, and probably a bunch more stuff I've forgotten or don't know about, is something else. If I wanted a tightly-coupled opaque binary blob of a system I'd run Windows.
† Of course its log file is not ACID, so when it poops its pants and corrupts the file there's no way to recover it besides moving it out of the way. Though the log entries on why this happened are also corrupted so there's no root cause analysis.‡ They could have just used SQLite as the logging format and it would have probably been awesome with regards to tooling and doing queries (instead of grep-foo). But no.
‡ And journald also cannot send logs off-host in any useful format, so I end up having to run rsyslogd anyway to get things into my SIEM through something useful like RFC 3164 or 5424.
But yeah: unit files are (non-sarcastically) pretty good.
systemd-as-init was fine. systemd replacing udevd, syslog†, mount, ntpd, DNS, dhcpd, and probably a bunch more stuff I've forgotten or don't know about, is something else. If I wanted a tightly-coupled opaque binary blob of a system I'd run Windows.
† Of course its log file is not ACID, so when it poops its pants and corrupts the file there's no way to recover it besides moving it out of the way. Though the log entries on why this happened are also corrupted so there's no root cause analysis.‡ They could have just used SQLite as the logging format and it would have probably been awesome with regards to tooling and doing queries (instead of grep-foo). But no.
‡ And journald also cannot send logs off-host in any useful format, so I end up having to run rsyslogd anyway to get things into my SIEM through something useful like RFC 3164 or 5424.
But yeah: unit files are (non-sarcastically) pretty good.
I'd be interested in how your logs actually got corrupted (was journald to blame or was there power/filesystems issues that were just hard to recover from)
I actually like journald quite a bit. Before, you'd have log files scattered all over the place with various cronjobs to try to keep them from filling up the disk (usually with the help of logrotate). Each application was free to put its logs wherever it decided.
Journald keeps them structured with first-class support for metadata like timestamp and unit so it's easy to get a service's logs in a 2 hour window on a given day and equally as easy to look at them in a stream with other service's logs (it aggregates them for you). From an application perspective, you just dump to stdout/err.
>And journald also cannot send logs off-host in any useful format
This seems directly at odds with your "tightly-coupled" beginning statement (you want it to do more or you think it does too much?). It's trivial to do this with fluentd or other log aggregation software
I actually like journald quite a bit. Before, you'd have log files scattered all over the place with various cronjobs to try to keep them from filling up the disk (usually with the help of logrotate). Each application was free to put its logs wherever it decided.
Journald keeps them structured with first-class support for metadata like timestamp and unit so it's easy to get a service's logs in a 2 hour window on a given day and equally as easy to look at them in a stream with other service's logs (it aggregates them for you). From an application perspective, you just dump to stdout/err.
>And journald also cannot send logs off-host in any useful format
This seems directly at odds with your "tightly-coupled" beginning statement (you want it to do more or you think it does too much?). It's trivial to do this with fluentd or other log aggregation software
In addition to the gratuitous tight coupling, the quality of implementation of most of these system service replacements is pretty bad! NTP and DNS clients are clearly worse (more limited and more error-prone and buggy than the separate packages they replace). The syslog replacement is only not a major issue for me because my distribution (K)Ubuntu has it configured to produce plaintext log files in addition to the binary files.
I also have a problem with socket activation. There is a reason why (x)inetd fell out of favor - its whole purpose was to provide socket activation. It's a hack that can produce new error states and it can actually slow down startup compared to starting services proactively.
That said, the OOM killer daemon seems to be a good thing.
I also have a problem with socket activation. There is a reason why (x)inetd fell out of favor - its whole purpose was to provide socket activation. It's a hack that can produce new error states and it can actually slow down startup compared to starting services proactively.
That said, the OOM killer daemon seems to be a good thing.
systemd-resolved and dnsmasq are the only two clients that can properly route DNS queries via the proper interface to their respective authoritative server based on the resolved hostname itself.
It would be hard to argue, that systemd-resolved is worse/more-limited/more error prone/buggy than dnsmasq.
The socket activation is intended for services, that are not normally needed, but occasionally they might be. It doesn't apply much to servers serving well-defined services, but for desktop, it is a godsend (together with dbus-activation). Just because you don't have use for it for your use cases doesn't mean it is not useful for other people.
It would be hard to argue, that systemd-resolved is worse/more-limited/more error prone/buggy than dnsmasq.
The socket activation is intended for services, that are not normally needed, but occasionally they might be. It doesn't apply much to servers serving well-defined services, but for desktop, it is a godsend (together with dbus-activation). Just because you don't have use for it for your use cases doesn't mean it is not useful for other people.
> It would be hard to argue, that systemd-resolved is worse/more-limited/more error prone/buggy than dnsmasq.
I'll take up that argument. On Ubuntu 14.04 (and I think also 16.04), systemd-resolved used to crash on my desktop machine after about a week of uptime. I never figured out the reason; I got in the habit of manually pointing `/etc/resolv.conf` at a real server whenever it happened.
I don't remember ever having that problem with dnsmasq, but I've seen so many DNS proxy things over the years that I can't keep them all straight. I think it's probably fair to compare systemd-resolved and nscd; both were there by default doing something with DNS, and both crashed often enough to annoy me.
It's not a new problem, either. I remember whatever came with RedHat or Mandrake or whatever I was running circa 2001 also seemed to stop working after a while. I ended up writing my own DNS proxy in C++ and ended up learning a lot. The protocol is wildly different enough from almost any other common protocol that I'm not surprised when software gets it wrong (I sure did!). The takeaway for me is that designing a protocol is as much a human problem as it is a technical one.
I'll take up that argument. On Ubuntu 14.04 (and I think also 16.04), systemd-resolved used to crash on my desktop machine after about a week of uptime. I never figured out the reason; I got in the habit of manually pointing `/etc/resolv.conf` at a real server whenever it happened.
I don't remember ever having that problem with dnsmasq, but I've seen so many DNS proxy things over the years that I can't keep them all straight. I think it's probably fair to compare systemd-resolved and nscd; both were there by default doing something with DNS, and both crashed often enough to annoy me.
It's not a new problem, either. I remember whatever came with RedHat or Mandrake or whatever I was running circa 2001 also seemed to stop working after a while. I ended up writing my own DNS proxy in C++ and ended up learning a lot. The protocol is wildly different enough from almost any other common protocol that I'm not surprised when software gets it wrong (I sure did!). The takeaway for me is that designing a protocol is as much a human problem as it is a technical one.
Yup, not-too-long-ago we setup OpenVPN at work with split tunneling which required DNS to work correctly. Windows worked out of the box (except for a small pool of users who had their DNS manually set which was a checkbox fix). macOS was more irritating because OpenVPN didn't native support the DNS API and it turned out the defacto way was Tunnelblick third party software.
Linux... Absolute nightmare. Every system had a slightly different DNS stack and trying to get queries to route over the VPN interface was an acrobatic exercise in docs navigation and trial and error. Ultimately, newer desktops with systemd-resolved were the easiest
systemd-resolved supporting DNSSEC is also a big win!
Linux... Absolute nightmare. Every system had a slightly different DNS stack and trying to get queries to route over the VPN interface was an acrobatic exercise in docs navigation and trial and error. Ultimately, newer desktops with systemd-resolved were the easiest
systemd-resolved supporting DNSSEC is also a big win!
Well. This http://0pointer.de/blog/projects/socket-activation.html sells socket activation as the best thing since sliced bread, including for boot time. It has its uses for some rarely used services and for some startup order problems, yes.
About the DNS service, I remember a bug where the fallback behavior was not what people wanted (it was something that seemed to be a good idea to systemd developers but didn't work well in completely reasonable real-world configurations), and the developers rejected the idea that systemd's behavior was wrong. Can't find it right now. I have disabled it on my system as well due to a problem I don't remember. Maybe it didn't use my router's DNS server and went straight to the internet. Maybe I just wanted it to leave my resolv.conf alone.
About the DNS service, I remember a bug where the fallback behavior was not what people wanted (it was something that seemed to be a good idea to systemd developers but didn't work well in completely reasonable real-world configurations), and the developers rejected the idea that systemd's behavior was wrong. Can't find it right now. I have disabled it on my system as well due to a problem I don't remember. Maybe it didn't use my router's DNS server and went straight to the internet. Maybe I just wanted it to leave my resolv.conf alone.
> About the DNS service, I remember a bug where the fallback behavior was not what people wanted (it was something that seemed to be a good idea to systemd developers but didn't work well in completely reasonable real-world configurations), and the developers rejected the idea that systemd's behavior was wrong.
It was a fallback DNS used when the user has configured nothing.
Not only that, it was a default fallback that can be changed by both the distribution package and the user (including disabling completely).
It was a tempest in the teapot, by people who have to find some drama.
It was a fallback DNS used when the user has configured nothing.
Not only that, it was a default fallback that can be changed by both the distribution package and the user (including disabling completely).
It was a tempest in the teapot, by people who have to find some drama.
I think I remember what it was now. Local configuration (DHCP?) contained two DNS servers. systemd-resolved switched to the global fallback when it tried one of the local servers and it failed, instead of falling back to the other local server first. It's a completely reasonable configuration to have two local servers for redundancy as well as load balancing, and systemd-resolved broke the redundancy aspect. It was not about a "wrong" global fallback, but about using it when a local alternative still works. It breaks local network name resolution.
If it behaved like that, yes, it is a bug that needs to be fixed.
I don't remember that (not saying that it didn't happen, after a quick search I didn't find any filed issue, #18769 could in theory have similar symptoms), but I remember several threads, including LWN article about the existence of the global fallback.
I don't remember that (not saying that it didn't happen, after a quick search I didn't find any filed issue, #18769 could in theory have similar symptoms), but I remember several threads, including LWN article about the existence of the global fallback.
It can both be true that there are a bunch of annoying things about systemd and their response to genuine bugs sometimes leave something to be desired, and that there's a lot of hate for the sake of hate directed at it.
This, naturally, just makes everything harder to have a sensible conversation about.
This, naturally, just makes everything harder to have a sensible conversation about.
What you may be referring to is systemd treating resolv.conf entries as identical DNS servers, where they used to be processed sequentially. Previous discussion: https://news.ycombinator.com/item?id=15228940
It's a breaking change for some users. I remember experiencing that bug.
It's a breaking change for some users. I remember experiencing that bug.
To be fair, there isn't one Uber-bin, but that doesn't change the fact that all these systems are now in lock step and under the control of one group.
Tell me, a naive one, why a throwaway for this?
Tell me, a naive one, why a throwaway for this?
> To be fair, there isn't one Uber-bin
Damning with faint praise? :) While an uber-bin would be a bigger problem, there is a reason why I italicized "tightly-coupled". Often many people simply respond with "but there are multiple binaries",† which kind of misses the point.
> Tell me, a naive one, why a throwaway for this?
It was created 2017-01-01.
† Edit: Case in point, see sibling comment: "These are separate services, with separate binaries, with separate packages […]"
* https://news.ycombinator.com/item?id=26646250
Damning with faint praise? :) While an uber-bin would be a bigger problem, there is a reason why I italicized "tightly-coupled". Often many people simply respond with "but there are multiple binaries",† which kind of misses the point.
> Tell me, a naive one, why a throwaway for this?
It was created 2017-01-01.
† Edit: Case in point, see sibling comment: "These are separate services, with separate binaries, with separate packages […]"
* https://news.ycombinator.com/item?id=26646250
> under the control of one group
Exactly.
Exactly.
Why do you assume a throwaway?
>... ntpd, DNS, dhcpd
These are separate services, with separate binaries, with separate packages for most distributions and their use is optional; they have their uses. For example, most distributions won't use systemd-networkd, because it is (intentionally) quite limited and they use NetworkManager anyway. RHEL8 for example doesn't even ship a package systemd-networkd.
These are separate services, with separate binaries, with separate packages for most distributions and their use is optional; they have their uses. For example, most distributions won't use systemd-networkd, because it is (intentionally) quite limited and they use NetworkManager anyway. RHEL8 for example doesn't even ship a package systemd-networkd.
> These are separate services, with separate binaries, with separate packages
Then why are they in the same repo as the udevd code? And why did udevd have to be pulled into the same repo as the init code when it was doing just fine outside of it?
Then why are they in the same repo as the udevd code? And why did udevd have to be pulled into the same repo as the init code when it was doing just fine outside of it?
Probably because it was not doing just fine outside of it.
If you think they took the wrong approach and yours is the right one, show them by doing.
If you think they took the wrong approach and yours is the right one, show them by doing.
> If you think they took the wrong approach and yours is the right one, show them by doing.
If Roger Ebert said a movie was awful, did people expect him to write, produce, direct, and/or act in one?
If Doug DeMuro said a car handles badly, are people expecting him to get a mechanical engineering degree and build a better one?
If Roger Ebert said a movie was awful, did people expect him to write, produce, direct, and/or act in one?
If Doug DeMuro said a car handles badly, are people expecting him to get a mechanical engineering degree and build a better one?
Ebert did write a movie: https://en.m.wikipedia.org/wiki/Beyond_the_Valley_of_the_Dol...
Yes, I know. :) And that would probably be the level of quality of any code I tried to write. :)
So what weight should throw0101a's word have? Why?
Since throw0101a has exactly zero public record of understanding the topic at hand, and everything he provided is an opinion, why should be that opinion taken into account? What exactly is his point, except for demonstrating that throw0101a doesn't like the existing approach?
The show them better is one way to establish your credentials that you know what you are talking about. There are other ways to achieve similar effect, but peanut gallery isn't it.
Since throw0101a has exactly zero public record of understanding the topic at hand, and everything he provided is an opinion, why should be that opinion taken into account? What exactly is his point, except for demonstrating that throw0101a doesn't like the existing approach?
The show them better is one way to establish your credentials that you know what you are talking about. There are other ways to achieve similar effect, but peanut gallery isn't it.
> So what weight should throw0101a's word have? Why?
About the same as any other rando's on the Internet. Take it or leave it. Up or down vote me.
As I type this: I have 23 imaginary Internet points on the post that kicked-off this sub-thread, and 0 points on a post bringing up Ebert and DeMuro.
Whatever.
About the same as any other rando's on the Internet. Take it or leave it. Up or down vote me.
As I type this: I have 23 imaginary Internet points on the post that kicked-off this sub-thread, and 0 points on a post bringing up Ebert and DeMuro.
Whatever.
Do you do the discussion for imaginary internet points?
I'm neither down nor up voting you; I consider doing that to people I discuss with a bad form (yes, that's opinion too).
However, it helps the discussion if there are arguments for the positions of those discussing; the point is not for either side to "win" (whatever that means and however you measure that "win"), but to find the best outcome after considering all valid arguments.
However, that does not work when there aren't any arguments. Vaguely liking or not liking something, and having no idea about the
I'm neither down nor up voting you; I consider doing that to people I discuss with a bad form (yes, that's opinion too).
However, it helps the discussion if there are arguments for the positions of those discussing; the point is not for either side to "win" (whatever that means and however you measure that "win"), but to find the best outcome after considering all valid arguments.
However, that does not work when there aren't any arguments. Vaguely liking or not liking something, and having no idea about the
> Do you do the discussion for imaginary internet points?
That's what Bitcoin is and people assign value to those useless bits don't they?
What's the point of life if you can't fret over magnetic ones and zeros?
That's what Bitcoin is and people assign value to those useless bits don't they?
What's the point of life if you can't fret over magnetic ones and zeros?
The value of any currency is in that, that you can get material stuff in exchange. The biggie historically used to be to be able pay the taxes, today getting extra heaps of atoms delivered to your doorstep does the job too.
Quite difficult to do with imaginary internet points tho ;)
Quite difficult to do with imaginary internet points tho ;)
What part of networkd do you find limited compared to other tools?
NM can manage VPNs and modems (LTE, etc) for example.
Don't get me wrong, systemd-networkd is fine for server or static usage; NM is better for desktop/laptop use.
Don't get me wrong, systemd-networkd is fine for server or static usage; NM is better for desktop/laptop use.
I found wireguard tunnels easier to set up in systemd-networkd then NM. systemd-networkd can handle VPNs for at least some use cases.
Most of these services have a dependency on the service manager and can't run without it.
For other services, such as journald and udevd, the dependency also works in reverse and the service manager can't run without the services.
In many cases, it's not entirely clear why they depend on the service manager itself. logind in particular was moved into systemd “in anticipation of the single writer cgroup architecture", an architecture that never came to be because it was quite clearly a very bad idea, and then elogind was forked to separate it again with no loss of functionality.
There is no reason for logind to exist, elogind should be the only thing that exists as it does the same thing without depending on systemd; elogind can also be used in conjunction with sytemd with no loss of functionality.
Even stranger things happen, such as DBus performing activation viā systemd by use of a private, nonsstandard a.p.i. for which there are standardized protocols specified by Linux-base that systemd also supports. That they chose to use a specific unstable, undocumented a.p.i. rather than a standardized one to do this is certainly a political rather than technical decision to create a dependency for it's own sake.
And that is indeed what many RedHat projects have done over the years; they have created dependencies on each other of little to no technical merit as form of product tying to encourage adoption of more RedHat software. — this is certainly not limited to sytemd.
For other services, such as journald and udevd, the dependency also works in reverse and the service manager can't run without the services.
In many cases, it's not entirely clear why they depend on the service manager itself. logind in particular was moved into systemd “in anticipation of the single writer cgroup architecture", an architecture that never came to be because it was quite clearly a very bad idea, and then elogind was forked to separate it again with no loss of functionality.
There is no reason for logind to exist, elogind should be the only thing that exists as it does the same thing without depending on systemd; elogind can also be used in conjunction with sytemd with no loss of functionality.
Even stranger things happen, such as DBus performing activation viā systemd by use of a private, nonsstandard a.p.i. for which there are standardized protocols specified by Linux-base that systemd also supports. That they chose to use a specific unstable, undocumented a.p.i. rather than a standardized one to do this is certainly a political rather than technical decision to create a dependency for it's own sake.
And that is indeed what many RedHat projects have done over the years; they have created dependencies on each other of little to no technical merit as form of product tying to encourage adoption of more RedHat software. — this is certainly not limited to sytemd.
I definitely understand that criticism w.r.t. all of the systemd-verse replacement tools.
I'm not an expert. I'm just a software dev and free software nut who wants a free software OS.
But (serious question), why does it matter that systemd has an ntp daemon or that it subsumed udev?
You say, disparagingly, that it is tightly-coupled and opaque (binary).
Why is loosely coupled better in these cases specifically?
Weren't `mount`, `ntpd`, `udev`, etc all compiled C code already? How does systmed make those service more opaque?
Everyone seems to hate journald. I'll go ahead and join in the chorus and say that it sounds really bad. But do we have to throw the baby out with the bath water? What else about systemd is actually worse than what we had before?
I'm not an expert. I'm just a software dev and free software nut who wants a free software OS.
But (serious question), why does it matter that systemd has an ntp daemon or that it subsumed udev?
You say, disparagingly, that it is tightly-coupled and opaque (binary).
Why is loosely coupled better in these cases specifically?
Weren't `mount`, `ntpd`, `udev`, etc all compiled C code already? How does systmed make those service more opaque?
Everyone seems to hate journald. I'll go ahead and join in the chorus and say that it sounds really bad. But do we have to throw the baby out with the bath water? What else about systemd is actually worse than what we had before?
> But (serious question), why does it matter that systemd has an ntp daemon or that it subsumed udev?
What does a time system have to with system start infrastructure? If you want to write a time daemon, write a time daemon: the chrony folks did, and it's pretty good. Lots of folks have switch away from the old school NTPd to it.
But it stood on its own merits, was able to mature over time, and when people thought it good, they could willing start using it (or not).
> Everyone seems to hate journald. I'll go ahead and join in the chorus and say that it sounds really bad. But do we have to throw the baby out with the bath water?
And that is exactly my point about things being tightly-coupled. If I (rightly or wrongly) perceive journald to be garbage, how do I not-use it?
What does a time system have to with system start infrastructure? If you want to write a time daemon, write a time daemon: the chrony folks did, and it's pretty good. Lots of folks have switch away from the old school NTPd to it.
But it stood on its own merits, was able to mature over time, and when people thought it good, they could willing start using it (or not).
> Everyone seems to hate journald. I'll go ahead and join in the chorus and say that it sounds really bad. But do we have to throw the baby out with the bath water?
And that is exactly my point about things being tightly-coupled. If I (rightly or wrongly) perceive journald to be garbage, how do I not-use it?
> What does a time system have to with system start infrastructure?
Accurate time is vital for many cryptographic operations. For example, in the embedded space you may want to have a check that your current disk image (ideally checked by dm-verity) is actually up to date. And to verify the metadata signing that disk image using Uptane/TUF/SUIT or similar protocols you need verifiable time. Or you might need to contact a provisioning server at boot time, and need to be able to verify that server's TLS cert.
(Now, systemd used to have a bug[1] where time-sync.target would be reached before time was actually synced, and Poettering's response to the bug was the all-too-typical typical "yeah, don't worry about it, it's intended behaviour that the time-sync target doesn't mean that time is synced"[2]. But that did get fixed a couple years ago [3].)
[1] https://github.com/systemd/systemd/issues/5097 [2] https://github.com/systemd/systemd/issues/5097#issuecomment-... [3] https://github.com/systemd/systemd/pull/8494
Accurate time is vital for many cryptographic operations. For example, in the embedded space you may want to have a check that your current disk image (ideally checked by dm-verity) is actually up to date. And to verify the metadata signing that disk image using Uptane/TUF/SUIT or similar protocols you need verifiable time. Or you might need to contact a provisioning server at boot time, and need to be able to verify that server's TLS cert.
(Now, systemd used to have a bug[1] where time-sync.target would be reached before time was actually synced, and Poettering's response to the bug was the all-too-typical typical "yeah, don't worry about it, it's intended behaviour that the time-sync target doesn't mean that time is synced"[2]. But that did get fixed a couple years ago [3].)
[1] https://github.com/systemd/systemd/issues/5097 [2] https://github.com/systemd/systemd/issues/5097#issuecomment-... [3] https://github.com/systemd/systemd/pull/8494
Thats a pretty uncharitable take on what Poettering said. IMO hes right that in many cases you dont want to hold up booting on whether or not time has been synced. I dont want my desktop, or most servers to not boot if the network or NTP server is down.
He also suggests what they can add for people who want this behaviour.
Oh woe is me that systemd does not support my use case of an embedded system needing to verify disk image with accurate time before booting further, as the _default_.
He also suggests what they can add for people who want this behaviour.
Oh woe is me that systemd does not support my use case of an embedded system needing to verify disk image with accurate time before booting further, as the _default_.
> > But (serious question), why does it matter that systemd has an ntp daemon or that it subsumed udev?
> What does a time system have to with system start infrastructure? If you want to write a time daemon, write a time daemon: the chrony folks did, and it's pretty good. Lots of folks have switch away from the old school NTPd to it.
Nothing, but you're making the common mistake of thinking that systemd is an init system. It isn't. It's a suite of components that more or less can be enabled independently, that no-one's forcing anyone to use. One of these components is an init. Another one in a time sync daemon, etc. etc.
Chrony's great, but if you don't need most of its pretty advanced features, why bother? A simpler, smaller, 'good enough' client might be more suitable.
Isn't software choice a good thing?
> What does a time system have to with system start infrastructure? If you want to write a time daemon, write a time daemon: the chrony folks did, and it's pretty good. Lots of folks have switch away from the old school NTPd to it.
Nothing, but you're making the common mistake of thinking that systemd is an init system. It isn't. It's a suite of components that more or less can be enabled independently, that no-one's forcing anyone to use. One of these components is an init. Another one in a time sync daemon, etc. etc.
Chrony's great, but if you don't need most of its pretty advanced features, why bother? A simpler, smaller, 'good enough' client might be more suitable.
Isn't software choice a good thing?
> Isn't software choice a good thing?
Yes, but tightly coupling can reduce the ability to make a choice.
Yes, but tightly coupling can reduce the ability to make a choice.
I'm not sure how that's true in this case.
I personally need the features of chrony, so despite running debian stable / systemd everywhere, I just turn systemd-timesyncd off.
Tight coupling would be "you can't boot your system unless you use this time sync daemon implementation".
I personally need the features of chrony, so despite running debian stable / systemd everywhere, I just turn systemd-timesyncd off.
Tight coupling would be "you can't boot your system unless you use this time sync daemon implementation".
There are definitely degrees of freedom, and I wonder if it would make sense to extend the Free Software Definition[0] backwards further by adding:
* The freedom to not run any program you do not wish to run, for any reason (freedom -1).
Presumably Stallman would agree that government-mandated spyware which happened to be GPL licensed would not really be granting users much freedom. The tight coupling of "you can't boot your system unless you use this time sync daemon implementation" might also count as a violation of this hypothetical freedom.
Applying this to systemd is more complicated though. I'm sure some people would say "Well it's released under a Free Software licence so you can remove or replace any part of the code you don't want.", but that seems almost as unhelpful as saying "Microsoft Windows is Free Software because you can always create a clean-room reimplementation of it".
[0] https://en.wikipedia.org/wiki/The_Free_Software_Definition
* The freedom to not run any program you do not wish to run, for any reason (freedom -1).
Presumably Stallman would agree that government-mandated spyware which happened to be GPL licensed would not really be granting users much freedom. The tight coupling of "you can't boot your system unless you use this time sync daemon implementation" might also count as a violation of this hypothetical freedom.
Applying this to systemd is more complicated though. I'm sure some people would say "Well it's released under a Free Software licence so you can remove or replace any part of the code you don't want.", but that seems almost as unhelpful as saying "Microsoft Windows is Free Software because you can always create a clean-room reimplementation of it".
[0] https://en.wikipedia.org/wiki/The_Free_Software_Definition
>> But (serious question), why does it matter that systemd has an ntp daemon or that it subsumed udev?
>
>What does a time system have to with system start infrastructure? If you want to write a time daemon, write a time daemon: the chrony folks did, and it's pretty good. Lots of folks have switch away from the old school NTPd to it.
>
>But it stood on its own merits, was able to mature over time, and when people thought it good, they could willing start using it (or not).
But my question was why does this matter? As far as I know, you don't have to use systemd's ntp thing. You might disagree with the project vision of systemd wanting to include an ntp daemon, but it sounds like it hasn't cause you or the chrony people any real harm. There was a good chance that, if systemd didn't have an ntp daemon, you'd have to replace your distro's default with chrony anyway. So just do that here, too, no?
> And that is exactly my point about things being tightly-coupled. If I (rightly or wrongly) perceive journald to be garbage, how do I not-use it?
You can't, AFAIK. And that sucks. I know you can disable its on-disk logs and enabled some other syslog service, though. In what way(s) is that insufficient?
Are there any other example of tight coupling that are bothersome in systemd?
But my question was why does this matter? As far as I know, you don't have to use systemd's ntp thing. You might disagree with the project vision of systemd wanting to include an ntp daemon, but it sounds like it hasn't cause you or the chrony people any real harm. There was a good chance that, if systemd didn't have an ntp daemon, you'd have to replace your distro's default with chrony anyway. So just do that here, too, no?
> And that is exactly my point about things being tightly-coupled. If I (rightly or wrongly) perceive journald to be garbage, how do I not-use it?
You can't, AFAIK. And that sucks. I know you can disable its on-disk logs and enabled some other syslog service, though. In what way(s) is that insufficient?
Are there any other example of tight coupling that are bothersome in systemd?
> But (serious question), why does it matter that systemd has an ntp daemon or that it subsumed udev?
> You say, disparagingly, that it is tightly-coupled and opaque (binary).
> Why is loosely coupled better in these cases specifically?
It creates extra work for others.
udev is far older than systemd; maintainership of the former was eventually inherited by one the latter's lead maintainers, who decided to move the latter into the former, at first promising that the former, which has applications reaching far outside systemd, could still be built and used independently.
That promise weakened after about two years and it became increasingly difficult to build and use udev without systemd; — some embedded systems that use udev can't even fit systemd into memory and thus using it is not an option.
So, udev in response was forked to eudev by other developers who now have to spend time and money on maintaining this fork and porting udev changes to it.
Loosely coupled is better because it allows one to be use without the other if there be such a need; tight coupling is essentially a form of product tying.
Why is it better that one can use an iPhone on any computer? rather than only computers designed by Apple?
> Everyone seems to hate journald. I'll go ahead and join in the chorus and say that it sounds really bad. But do we have to throw the baby out with the bath water? What else about systemd is actually worse than what we had before?
Many call for logind to replace acpid; — I strongly object to this and the difference in functionality between both highlights well the difference in philosophy between traditional Unix design and Lennart's brand.
acpid responds to acpi events by calling an executable file with a specific path; for instance `/etc/acpi/actions/powerbtn.sh` is called with arguments that describe the nature of the event when the power button is pressed. logind on the other hand simply allows for a limited list of I believe seven options in a configuration file of what to do when the power button is pressed, not an arbitrary file to be executed.
The possibilities of acpid are obviously limitless, and I use them as such. My machine is configured such that when I close my notebook's lid, the machine does not suspend, but rather disables the screen and goes into power-safe mode, but otherwise continues to operate, I can override this behavior by simply creating the file `/run/lid.disable`.
I cannot configure my machine to be so flexible with logind which highlights the difference in culture: Traditional Unix design was always about removing restrictions, and this is about providing features, but when the feature one seeks not be provided, then one is out of luck.
> You say, disparagingly, that it is tightly-coupled and opaque (binary).
> Why is loosely coupled better in these cases specifically?
It creates extra work for others.
udev is far older than systemd; maintainership of the former was eventually inherited by one the latter's lead maintainers, who decided to move the latter into the former, at first promising that the former, which has applications reaching far outside systemd, could still be built and used independently.
That promise weakened after about two years and it became increasingly difficult to build and use udev without systemd; — some embedded systems that use udev can't even fit systemd into memory and thus using it is not an option.
So, udev in response was forked to eudev by other developers who now have to spend time and money on maintaining this fork and porting udev changes to it.
Loosely coupled is better because it allows one to be use without the other if there be such a need; tight coupling is essentially a form of product tying.
Why is it better that one can use an iPhone on any computer? rather than only computers designed by Apple?
> Everyone seems to hate journald. I'll go ahead and join in the chorus and say that it sounds really bad. But do we have to throw the baby out with the bath water? What else about systemd is actually worse than what we had before?
Many call for logind to replace acpid; — I strongly object to this and the difference in functionality between both highlights well the difference in philosophy between traditional Unix design and Lennart's brand.
acpid responds to acpi events by calling an executable file with a specific path; for instance `/etc/acpi/actions/powerbtn.sh` is called with arguments that describe the nature of the event when the power button is pressed. logind on the other hand simply allows for a limited list of I believe seven options in a configuration file of what to do when the power button is pressed, not an arbitrary file to be executed.
The possibilities of acpid are obviously limitless, and I use them as such. My machine is configured such that when I close my notebook's lid, the machine does not suspend, but rather disables the screen and goes into power-safe mode, but otherwise continues to operate, I can override this behavior by simply creating the file `/run/lid.disable`.
I cannot configure my machine to be so flexible with logind which highlights the difference in culture: Traditional Unix design was always about removing restrictions, and this is about providing features, but when the feature one seeks not be provided, then one is out of luck.
> † Of course its log file is not ACID, so when it poops its pants and corrupts the file there's no way to recover it besides moving it out of the way. Though the log entries on why this happened are also corrupted so there's no root cause analysis.‡ They could have just used SQLite as the logging format and it would have probably been awesome with regards to tooling and doing queries (instead of grep-foo). But no.
When current journals are detected as dirty by journald @ startup, they are renamed and no longer written to. This is a very robust and conservative approach. It burns some space in the interest of simplicity and preserving the log data without risk of messing things up.
These renamed journals are still accessed when reading, so it's not like those dirty files no longer participate in journalctl operations; their contents are not lost.
Any logs lost in a crash are just what was in-flight IPC, or sitting dirty in the kernel's buffer cache and hadn't been sync'd to backing store yet. Journald also performs an explicit sync whenever an urgent message arrives, to try ensure it's made durable ASAP. Of course there's still a filesystem and potentially myriad layers below that which could lose data in a crash.
ACID would be overkill for this fairly simple single-writer multiple-reader mostly-appended situation.
Edit:
Your statement "when it poops its pants and corrupts the file" implies journald has a tendency to actively corrupt its own files.
The principal mechanism journald uses to detect potentially corrupt files is simply identifying a file as being ONLINE when opening for writing, typically at startup. This is almost always caused by an unclean shutdown of the host, though it could be due to a crash of journald itself.
The file isn't even necessarily corrupt. It's simply assumed inconsistent and treated as such WRT writability. I'm not aware of any journald bugs in recent history where journald actively corrupted the contents of journal files. Even crashy bugs where journald was littering dirty journals to be treated as corrupt I'm not aware of being actually corrupted internally by journald bugs.
Source: I've worked quite a bit on journald over the years
When current journals are detected as dirty by journald @ startup, they are renamed and no longer written to. This is a very robust and conservative approach. It burns some space in the interest of simplicity and preserving the log data without risk of messing things up.
These renamed journals are still accessed when reading, so it's not like those dirty files no longer participate in journalctl operations; their contents are not lost.
Any logs lost in a crash are just what was in-flight IPC, or sitting dirty in the kernel's buffer cache and hadn't been sync'd to backing store yet. Journald also performs an explicit sync whenever an urgent message arrives, to try ensure it's made durable ASAP. Of course there's still a filesystem and potentially myriad layers below that which could lose data in a crash.
ACID would be overkill for this fairly simple single-writer multiple-reader mostly-appended situation.
Edit:
Your statement "when it poops its pants and corrupts the file" implies journald has a tendency to actively corrupt its own files.
The principal mechanism journald uses to detect potentially corrupt files is simply identifying a file as being ONLINE when opening for writing, typically at startup. This is almost always caused by an unclean shutdown of the host, though it could be due to a crash of journald itself.
The file isn't even necessarily corrupt. It's simply assumed inconsistent and treated as such WRT writability. I'm not aware of any journald bugs in recent history where journald actively corrupted the contents of journal files. Even crashy bugs where journald was littering dirty journals to be treated as corrupt I'm not aware of being actually corrupted internally by journald bugs.
Source: I've worked quite a bit on journald over the years
> But yeah: unit files are (non-sarcastically) pretty good.
I personally like service managers more that have clean shell scripts such as runit or OpenRC where shell scripts more so look like this:
I personally like service managers more that have clean shell scripts such as runit or OpenRC where shell scripts more so look like this:
#!/usr/bin/openrc-run
depends="some list of services to depend on"
after="some list of services which need to start earlier"
cmd="/path/to/executable"
args="command line arguments"
See also the BSDs, which have similarly simple scripts
FWIW, I always found the "cumbersome collection of shell scripts" extremely transparent and easily hackable... I find systems that involve lots of native code that isn't like, clearly right there for me to read as I set things up, extremely opaque and difficult to debug.
The point is, did you ever have to debug systemd because it was systemd that was broken and not other parts?
Well, I would argue "the point is" the ability to easily understand when and how something works is useful; and, to that end, having the behaviors of something directly in front of you is thereby valuable in a way that shouldn't be dismissed, particularly when the language it is all written in is the "lineup franca" of system administration: shell. I have often wondered when something happens and why it happens at some point and simply grepped my way through /etc seeing exactly how the extremely simple startup behavior of those "cumbersome shell scripts" worked, and taking that away and hiding those behaviors into some complex C program to save a tiny bit of startup performance seems like a sad tradeoff, particularly for people just beginning their journey of discovery (like, I ostensibly don't need it anymore, as I can now read complex C programs and have in fact carefully scrutinized the code for the various eras of Apple's launchd, and so have a pretty solid understanding of exactly how everything works... but 25 years ago, being able to do that on Slackware because it effectively just had this beautifully simple shell script that just ran stuff as if it wasn't hard... that showed me the beauty of this system and how easy it is to bootstrap these things: the world didn't actually get more complex since then in a way where any of this cruft is actually necessary).
lineup -> lingua :(
> to save a tiny bit of startup performance
That is absolutely not the reason for systemd to exist, it is an additional benefit. How was early-boot logging, error handling, dependency handling between services implemented before? At best only in some ad-hoc way, different on each distro. The domain has an inherent complexity to it, init systems are not easy, there is no saving that. But I think systemd exposing the configurable parts in a declarative way is a net win over the previous solutions.
That is absolutely not the reason for systemd to exist, it is an additional benefit. How was early-boot logging, error handling, dependency handling between services implemented before? At best only in some ad-hoc way, different on each distro. The domain has an inherent complexity to it, init systems are not easy, there is no saving that. But I think systemd exposing the configurable parts in a declarative way is a net win over the previous solutions.
>> to save a tiny bit of startup performance
> That is absolutely not the reason for systemd to exist
If you read the original web document announcing systemd, one could come away with the impression that it was though (even if substantiated with quite questionable performance tests using virtual machines).
> That is absolutely not the reason for systemd to exist
If you read the original web document announcing systemd, one could come away with the impression that it was though (even if substantiated with quite questionable performance tests using virtual machines).
According to a core developer, it was a side effect of doing things the right way:
http://0pointer.de/blog/projects/the-biggest-myths.html
http://0pointer.de/blog/projects/the-biggest-myths.html
Not the application itself, no, but its configuration. Once I managed to screw up things by modifying the fstab: I forgot the (never used) fs_freq and fs_passno fields in a new entry for /tmp. mountall et. al. are fault tolerant and mount the fs regardless, but not so systemd -- it decided best course of action would be to not mount any local fs. You can imagine debugging that was a wee bit hairy, particularly since it was a head-less system.
When systemd came out I started experiencing "I upgraded and now my system won't boot". When I got fed up enough and removed systemd (and eventually had to switch distributions as non-systemd became unsupported) that stopped happening. I'm sure there will be some sophisticated argument about how this wasn't really systemd's fault and was someone else's fault, but the correlation is pretty damn strong.
That's a fair point, but where is the line? Do sysadmins worry about debugging Linux? Should it be written in Bash so that they can (obviously the kernel can't be written in Bash, but for the sake of argument)?
As a non-sysadmin, I just feel like it's totally fair to have PID 1 be about as trusted and uniform as the kernel. If there's a bug in systemd, that's bad. If there's a bug in Linux, that's bad. But can't we at least aim for an init that is not just a bunch of Bash scripts running as root?
As a non-sysadmin, I just feel like it's totally fair to have PID 1 be about as trusted and uniform as the kernel. If there's a bug in systemd, that's bad. If there's a bug in Linux, that's bad. But can't we at least aim for an init that is not just a bunch of Bash scripts running as root?
I have more trust in bash than in C. I can enable tracing in bash script and then see how it executed exactly.
IMHO, we should aim for an open standard for the init subsystem instead. For example, I want to implement my init system in Rust, but systemd is blob, which is hard to replace.
IMHO, we should aim for an open standard for the init subsystem instead. For example, I want to implement my init system in Rust, but systemd is blob, which is hard to replace.
> I have more trust in bash than in C. I can enable tracing in bash script and then see how it executed exactly.
If I had to work in either day-to-day, I'd chew my hands off. :p
> IMHO, we should aim for an open standard for the init subsystem instead. For example, I want to implement my init system in Rust, but systemd is blob, which is hard to replace.
Well, hold on. The person I replied to was talking about being able to debug systemd. So this is very tangential to that. If we had an standard for inits, with different implementations, it would be hard to debug all of the ones that are compiled binaries. My point is still that a sysadmin should not be expected to debug PID 1 any more than they should be expected to debug a driver.
Also, you can just treat systemd's APIs as the standard. GNOME, for example, depends on launchd. And, IIRC, there was some shim created so that GNOME could run on a BSD, which obviously doesn't have systemd. So, go ahead and write your init in Rust- Rust has great C API/ABI compatibility!
If I had to work in either day-to-day, I'd chew my hands off. :p
> IMHO, we should aim for an open standard for the init subsystem instead. For example, I want to implement my init system in Rust, but systemd is blob, which is hard to replace.
Well, hold on. The person I replied to was talking about being able to debug systemd. So this is very tangential to that. If we had an standard for inits, with different implementations, it would be hard to debug all of the ones that are compiled binaries. My point is still that a sysadmin should not be expected to debug PID 1 any more than they should be expected to debug a driver.
Also, you can just treat systemd's APIs as the standard. GNOME, for example, depends on launchd. And, IIRC, there was some shim created so that GNOME could run on a BSD, which obviously doesn't have systemd. So, go ahead and write your init in Rust- Rust has great C API/ABI compatibility!
Minimal initd is very short:
https://github.com/mesalock-linux/minit/blob/master/src/main...
http://git.suckless.org/sinit/file/sinit.c.html
However, systemd is huge: 1.3 million lines of code.
https://www.theregister.com/2020/01/06/linux_2020_kernel_sys...
https://github.com/mesalock-linux/minit/blob/master/src/main...
http://git.suckless.org/sinit/file/sinit.c.html
However, systemd is huge: 1.3 million lines of code.
https://www.theregister.com/2020/01/06/linux_2020_kernel_sys...
But those inits don't actually DO hardly anything. The Rust one hardcodes a bunch of stuff and has "TODO - setup signal handler" in the code...
You're not even comparing them to just the init part of systemd, but the entire repository.
What's the point here?
You're not even comparing them to just the init part of systemd, but the entire repository.
What's the point here?
The initd job IS to launch other processes and reap zombies. (To non-technical people: zombies are childs which died when their parent is already dead, so a daemon must collect their souls).
It's not a job of init daemon to manage containers or resolve DNS names.
IMHO, systemd must be PID 2+, not PID 1.
It's not a job of init daemon to manage containers or resolve DNS names.
IMHO, systemd must be PID 2+, not PID 1.
> The initd job IS to launch other processes and reap zombies. (To non-technical people: zombies are childs which died when their parent is already dead, so a daemon must collect their souls).
Well, those initds don't actually start processes, they start one other process.
> It's not a job of init daemon to manage containers or resolve DNS names.
systemd's init doesn't do these things, other parts of systemd do that.
> IMHO, systemd must be PID 2+, not PID 1.
It is, for the most part (certainly most of those '1 million lines of code' are not in PID 1). Certainly you could try to keep more of the service management code out of PID 1, but it's honestly not something with huge practical benefits (your service management is going to be priviledged anyway, in the event of a crash all a non-PID 1 solution allows is maybe a little bit of an easier time giving a chance to inspect the wreckage before you need to restart everything anyway), and you create more interfaces in your system to manage.
Well, those initds don't actually start processes, they start one other process.
> It's not a job of init daemon to manage containers or resolve DNS names.
systemd's init doesn't do these things, other parts of systemd do that.
> IMHO, systemd must be PID 2+, not PID 1.
It is, for the most part (certainly most of those '1 million lines of code' are not in PID 1). Certainly you could try to keep more of the service management code out of PID 1, but it's honestly not something with huge practical benefits (your service management is going to be priviledged anyway, in the event of a crash all a non-PID 1 solution allows is maybe a little bit of an easier time giving a chance to inspect the wreckage before you need to restart everything anyway), and you create more interfaces in your system to manage.
> cumbersome collection of shell scripts
Those shell scripts never had to be cumbersome. The BSDs, for example, do away with a lot of that complexity and boilerplate quite nicely via e.g. rc.subr and rc.conf.
Those shell scripts never had to be cumbersome. The BSDs, for example, do away with a lot of that complexity and boilerplate quite nicely via e.g. rc.subr and rc.conf.
>I've never understood why others of my profession hated it
Basically change aversion, and a misguided idea about the "unix way".
Basically change aversion, and a misguided idea about the "unix way".
That may be true for the init system replacement part. But I think asking (for example) "why is systemd killing things I started with nohup and also my tmux sessions?" or "why does systemd need to replace ntpd?" are valid questions.
"UNIX way" is not an idea or a religion. It's a natural selection of small, understandable tools, which can be supported by any professional programmer.
The UNIX way is a cargo cult.
I have done most commercial UNIXes starting with Xenix in 1993, none of them was (or are, regarding the surviving ones) following it.
In fact some of them have introduced systemd like ideas way longer.
I have done most commercial UNIXes starting with Xenix in 1993, none of them was (or are, regarding the surviving ones) following it.
In fact some of them have introduced systemd like ideas way longer.
"In fact some of them have introduced systemd like ideas way longer."
True enough. I spent a fair amount of time trying to wrangle with smit on AIX. That's not an endorsement :)
True enough. I spent a fair amount of time trying to wrangle with smit on AIX. That's not an endorsement :)
What you think about the idea of "function", a small piece of code which does one job, but does it well? Do you think we need one big function, which will unifies interfaces of multiple different functions, lets call it "system-function"?
Depends, functions that take booleans and enumeration flags to change their behaviour come to mind.
Also modules, objects and components packing functions together as means to make sense of disjointed functions.
Also modules, objects and components packing functions together as means to make sense of disjointed functions.
"Natural selection" in that it's a hodgepodge of varying quality tools, with no specific input and output structure (aside from stream of text/bytes), idiosyncratic flags (different between them for the same features), no coordination on various features/decisions, that let professional programmers hapilly work like it's 1979 and we've reached some global maxima.
I never liked the cumbersome mess of shell scripts it replaced. Here's what I really don't like about systemd:
(1) It's a "god object" at the system level, rolling massive amounts of functionality into a single monolithic "system manager." There are advantages to that, but it also makes it impossible to strip away complexity and leads to the core system becoming really bloated with functionality. That's the road Windows took that made Windows such a mess of cruft.
(2) Poor UX... it's obtuse and counter-intuitive. This is my biggest gripe. It replaced a hard to use ball of shell scripts with... something even more confusing! Every command seems designed to be hard to remember and clunky to type. Common tasks are "undiscoverable" in that you have to search docs to figure out how to do the simplest things. Git has this same problem by the way, but to a somewhat lesser degree.
(1) It's a "god object" at the system level, rolling massive amounts of functionality into a single monolithic "system manager." There are advantages to that, but it also makes it impossible to strip away complexity and leads to the core system becoming really bloated with functionality. That's the road Windows took that made Windows such a mess of cruft.
(2) Poor UX... it's obtuse and counter-intuitive. This is my biggest gripe. It replaced a hard to use ball of shell scripts with... something even more confusing! Every command seems designed to be hard to remember and clunky to type. Common tasks are "undiscoverable" in that you have to search docs to figure out how to do the simplest things. Git has this same problem by the way, but to a somewhat lesser degree.
I was ready to hate systemd after reading all the negative reactions over the years. Then it came to upgrade servers. Now, I appreciate that I never have to write an init script or cron job again.
> Since most major distros use systemd by default, it is important that also most of them fix the same fundamental problems the same way; and I really hope this new oomd finally fixes thrashing behavior.
Solving problems in multiple different ways is the only reason we have multiple distros serving different needs. Unification is not always a positive thing.
Solving problems in multiple different ways is the only reason we have multiple distros serving different needs. Unification is not always a positive thing.
I second that. I feel like I have less and less choice. I can either accept all systemd provides or switch to some fringe distro.
I switched from Windows to Linux 20 years ago to have more choice.
I switched from Windows to Linux 20 years ago to have more choice.
I really don’t get this mentality. Where are you limited exactly? You can trivially view and edit every single service file to achieve basically anything. But a modern system does require some building blocks. I mean, the linux kernel is similarly limiting you to eg. have a file-oriented abstraction, and you can’t just swap it out to a different kernel. Systemd is much more modular than that for example.
Pre-systemd you actually could switch out the kernel on Debian - you could be running Hurd or the FreeBSD kernel, but the rest of the OS would still be Debian.
I believe it is only correlation, not causation.
I think it has more to do with the linux kernel being much more advanced than the others, and some of the userspace depending more and more on these advanced features. But I’m not sure about this. Aren’t you referring instead to debian itself providing packages for all of them and the package manager running on each distro? Because I can’t really believe you could just swap out the kernel and be on your way in most cases. If it is the latter, than it was more due to the fact that debian maintainers voted against supporting other systems, because it is a great amount of additional work.
I think it has more to do with the linux kernel being much more advanced than the others, and some of the userspace depending more and more on these advanced features. But I’m not sure about this. Aren’t you referring instead to debian itself providing packages for all of them and the package manager running on each distro? Because I can’t really believe you could just swap out the kernel and be on your way in most cases. If it is the latter, than it was more due to the fact that debian maintainers voted against supporting other systems, because it is a great amount of additional work.
> I believe it is only correlation, not causation.
It's a bit more direct than that - previous init systems were kernel-agnostic whereas systemd is explicitly tightly coupled to Linux and udev, and encourages the rest of userland to be tightly coupled to it.
> I think it has more to do with the linux kernel being much more advanced than the others, and some of the userspace depending more and more on these advanced features.
I wouldn't say "more advanced" - in many ways FreeBSD jails are a better technical solution than Linux's various container systems.
It's a bit more direct than that - previous init systems were kernel-agnostic whereas systemd is explicitly tightly coupled to Linux and udev, and encourages the rest of userland to be tightly coupled to it.
> I think it has more to do with the linux kernel being much more advanced than the others, and some of the userspace depending more and more on these advanced features.
I wouldn't say "more advanced" - in many ways FreeBSD jails are a better technical solution than Linux's various container systems.
I think that lmilcin was referring to "either use systemd on distribution X or switch to another distribution".
E.g. in Gentoo you can choose to use SystemD or OpenRC, but I think that most other distros don't provide such a choice.
E.g. in Gentoo you can choose to use SystemD or OpenRC, but I think that most other distros don't provide such a choice.
Most don’t provide such a choice, because it’s a huge undertaking to basically package and maintain 2+ versions of everything, especially when one is unified among distros and often created upstream.
And of course after a certain size it has to do with network effects, but it was chosen based on merits — for example debian voted multiple times on the issue and I would say their voting system puts most countries’ to shame.
I believe up until recently the vote on debian was basically use systemd but if the maintainer wants to, they can also support alternative init systems, but there was not much interest. But of course anyone can help the fringe distros on another init systems to make them less fringe.
And of course after a certain size it has to do with network effects, but it was chosen based on merits — for example debian voted multiple times on the issue and I would say their voting system puts most countries’ to shame.
I believe up until recently the vote on debian was basically use systemd but if the maintainer wants to, they can also support alternative init systems, but there was not much interest. But of course anyone can help the fringe distros on another init systems to make them less fringe.
> Most don’t provide such a choice, because it’s a huge undertaking to basically package and maintain 2+ versions of everything
Yes, I absolutely agree regarding the effort. (not sure about that vote - I honestly didn't follow that discussion)
In parallel, some software (cannot recall - maybe the Gnome window mgr) does rely (in its original form) on SystemD being there => I don't think that's fair, kind of forcing the distros that want to offer it to adopt it => I don't like it - I do see the advantage of having that kind of integration, but I still don't like it :P
Yes, I absolutely agree regarding the effort. (not sure about that vote - I honestly didn't follow that discussion)
In parallel, some software (cannot recall - maybe the Gnome window mgr) does rely (in its original form) on SystemD being there => I don't think that's fair, kind of forcing the distros that want to offer it to adopt it => I don't like it - I do see the advantage of having that kind of integration, but I still don't like it :P
Gnome's dependency on logind came about because there originally was a dependency on a different library that one person had written, had pretty much stopped working on, and nobody wanted to take on maintaining it or adding it to Gnome itself.
The dependency no longer exists because some people took logind from systemd, made it a standalone package and called it elogind lol, and now you can use that instead of logind.
The dependency no longer exists because some people took logind from systemd, made it a standalone package and called it elogind lol, and now you can use that instead of logind.
Aha! Now I understand why some time ago when I updated my Gentoo distro that elogind-package came up! I had some package conflicts because of that... (but I'm not using Gnome, I have just XFCE4 ). Thanks!
Or disable the elements you don't want and switch to a different implementations. What exactly are you missing right now?
Switching to different implementations is getting harder with every systemd release.
I literally work on a device that is still uses an inittab (hopefully changing that in 2021). the only limiting factor is yourself.
1) eudev works
2) elogind works
3) you can still use sysvinit or literally the multitude of others
1) eudev works
2) elogind works
3) you can still use sysvinit or literally the multitude of others
> I literally work on a device that is still uses an inittab.
The fact that this is so rare it has to be pointed out like this kind of speaks for itself.
> 1) eudev works 2) elogind works 3) you can still use sysvinit or literally the multitude of others.
I am aware of these and use them on nearly all my machines.
Note that both elogind and eudev required a significant amount of work by the Gentoo community when they were pulled out of the systemd codebase.
The fact that this is so rare it has to be pointed out like this kind of speaks for itself.
> 1) eudev works 2) elogind works 3) you can still use sysvinit or literally the multitude of others.
I am aware of these and use them on nearly all my machines.
Note that both elogind and eudev required a significant amount of work by the Gentoo community when they were pulled out of the systemd codebase.
Linux is, or at least was, about Freedom; and Freedom is about choice.
You may not mind the decisions others make on your behalf, or care about the options that are no longer possible, but that doesn't mean that Linux hasn't lost some of its identity.
You may not mind the decisions others make on your behalf, or care about the options that are no longer possible, but that doesn't mean that Linux hasn't lost some of its identity.
Yes and no.
Yes, in the sense, that if you want, you can eventually make it the way you want.
No in the sense, that others won't do it for you; they will make linux distribution the way they consider it best, not how you consider it the best.
So if you want to pick and choose the components the way you want and not the way the distributions consider the best, be ready to put in the manhours.
Yes, in the sense, that if you want, you can eventually make it the way you want.
No in the sense, that others won't do it for you; they will make linux distribution the way they consider it best, not how you consider it the best.
So if you want to pick and choose the components the way you want and not the way the distributions consider the best, be ready to put in the manhours.
I mean there's always a way to configure/disable it, but a counter-argument would be that if there's an optimal solution to the OOM problem, I would prefer that all distros use it.
Just like you can mostly expect filesystem behavior to be consistent no matter which distro, so should be the OOM behavior if a "good" solution is found.
Just like you can mostly expect filesystem behavior to be consistent no matter which distro, so should be the OOM behavior if a "good" solution is found.
You can't expect filesystem to behave consistently across filesystem and media types because these exist to solve different issues and involve different design decisions.
The same thing goes for out of memory handling: no size fits all. The kind of handling strategy you need depends on the kind of system you build.
Exactly this kind of flexibility and lack of unification is what made Linux so ubiquitous.
The same thing goes for out of memory handling: no size fits all. The kind of handling strategy you need depends on the kind of system you build.
Exactly this kind of flexibility and lack of unification is what made Linux so ubiquitous.
I think what the OP means is that you can be in any filesystem without needing to put much thought into it (regardless of the very real differences in implementation). Advanced users who want to interface with the details can do that, but for the rest of us consistency can be good.
Isn't the same level of detail needed for init as for different filesystems?
Except that file systems aren't consistent on Linux. FAT and NTFS can't represent Unix permission bits. File systems reorder operations internally in some ways user space programs don't expect. NFS mounts may not be able to guarantee exclusive file locks...
Absolutely. What I'm saying is that I personally don't need to know about this. I suspect that OOM's behavior is of similar effect to most developers.
It was more like free beer source code did it.
> […] if there's an optimal solution to the OOM problem, I would prefer that all distros use it.
Except with the monoculture of systemd how can experiments be done with other approaches? Nowadays it seems like "use systemd or GTFO".
> Just like you can mostly expect filesystem behavior to be consistent […]
And how many file systems does the Linux support? One can try an approach and see how it works and others can perhaps copy it.
How does another approach towards OOM (or anything else in the kitchen sink that systemd handles) get traction when the 800lb/400kg of RH/IBM is bankrolling The One True Way™.
Except with the monoculture of systemd how can experiments be done with other approaches? Nowadays it seems like "use systemd or GTFO".
> Just like you can mostly expect filesystem behavior to be consistent […]
And how many file systems does the Linux support? One can try an approach and see how it works and others can perhaps copy it.
How does another approach towards OOM (or anything else in the kitchen sink that systemd handles) get traction when the 800lb/400kg of RH/IBM is bankrolling The One True Way™.
> Except with the monoculture of systemd how can experiments be done with other approaches? Nowadays it seems like "use systemd or GTFO".
Is that much different from let's say databases, where "One True Way" was SQL and despite that NoSQL became a thing and is appropriate for some use-cases?
> And how many file systems does the Linux support?
And yet ext4 appears to be the most widespread though. It's true that there are other options such as XFS, ZFS and BTRFS but in general, ext4 is a safe default that works well enough for most use cases unless you have specific reasons to choose otherwise.
Is it that far-fetched to believe that systemd might be the same, and it works "well enough" to be the default and even be bankrolled by some big names?
> How does another approach towards OOM [...] get traction
Presumably the systemd OOM killer can be disabled and something else put in its place if you so desire, just like you can still build or use a distro without systemd or selectively disabling its "bad" parts such as the journal, etc?
Is that much different from let's say databases, where "One True Way" was SQL and despite that NoSQL became a thing and is appropriate for some use-cases?
> And how many file systems does the Linux support?
And yet ext4 appears to be the most widespread though. It's true that there are other options such as XFS, ZFS and BTRFS but in general, ext4 is a safe default that works well enough for most use cases unless you have specific reasons to choose otherwise.
Is it that far-fetched to believe that systemd might be the same, and it works "well enough" to be the default and even be bankrolled by some big names?
> How does another approach towards OOM [...] get traction
Presumably the systemd OOM killer can be disabled and something else put in its place if you so desire, just like you can still build or use a distro without systemd or selectively disabling its "bad" parts such as the journal, etc?
When we talk about an OS that on desktop has minuscule user share, it is not too clever to further fragment that. I believe there are only a handful of people who knows enough to create a proper init system (similarly, sound and graphics) and do it for the community. The alternative is that more complicated features will simply not be implemented on linux.
Also, just look at the before-systemd landscape, there was much more difference in the repositories, and nowadays even more niche distros can trivially package almost everything since .service files are unified. Also, they are quite flexible, so I don’t feel that anything would be lost.
Also, just look at the before-systemd landscape, there was much more difference in the repositories, and nowadays even more niche distros can trivially package almost everything since .service files are unified. Also, they are quite flexible, so I don’t feel that anything would be lost.
> When we talk about an OS that on desktop has minuscule user share, it is not too clever to further fragment that.
True, but desktop usage is only one of many things linux is used for. All the "unification" efforts ala systemd try to handle all of those use cases. I don't see why the init system on my server has to be the same as the one on my ultrabook, because only a fraction of what they should do is shared. Having one tool that covers all the use cases just sounds like a very good way to increase the attack surface.
> I believe there are only a handful of people who knows enough to create a proper init system (similarly, sound and graphics) and do it for the community.
I don't think Linux graphics or sound situations have much in common with the init system situation, except perhaps the fact that Poettering tried to fix sound as well.
True, but desktop usage is only one of many things linux is used for. All the "unification" efforts ala systemd try to handle all of those use cases. I don't see why the init system on my server has to be the same as the one on my ultrabook, because only a fraction of what they should do is shared. Having one tool that covers all the use cases just sounds like a very good way to increase the attack surface.
> I believe there are only a handful of people who knows enough to create a proper init system (similarly, sound and graphics) and do it for the community.
I don't think Linux graphics or sound situations have much in common with the init system situation, except perhaps the fact that Poettering tried to fix sound as well.
> I don't see why the init system on my server has to be the same as the one on my ultrabook, because only a fraction of what they should do is shared
Well, not even the kernel should be shared —- but most of us is okay with having the same kernel, but at most configured differently. I see it the same way in the case of systemd, of course your ultra book doesn’t need I don’t know how many services, but the basics are very much the same. And to be honest, I’m happy to use the same knowledge/tools on a server machine and a laptop.
> Having one tool that covers all the use cases just sounds like a very good way to increase the attack surface.
That’s just security by obscurity...
Well, not even the kernel should be shared —- but most of us is okay with having the same kernel, but at most configured differently. I see it the same way in the case of systemd, of course your ultra book doesn’t need I don’t know how many services, but the basics are very much the same. And to be honest, I’m happy to use the same knowledge/tools on a server machine and a laptop.
> Having one tool that covers all the use cases just sounds like a very good way to increase the attack surface.
That’s just security by obscurity...
I really think fundamental problems should be solved the same way on major distros. Leave the exotic and creative ways for problems that are not fundamental or for minor distros. The feeling that you can solve fundamental problems the same way on any major distro is valuable.
Agreed, but systemd is leaning very much towards "every problem should be solved the systemd way". I'm totally fine with the idea of all distros having better oom behavior (which I hope is what systemd-oomd does), but I don't like how that implies having to use systemd for a lot of other problems as well.
It’s fairly clear to me that the problem isn’t that Systemd is leaning towards everything should be solved the SystemdD way.
The real problem is that almost no one has been able to create alternatives to SystemD’s solutions to the vast majority of problems it chooses to tackle that are as well developed or effective as the SystemD solution.
The real problem is that almost no one has been able to create alternatives to SystemD’s solutions to the vast majority of problems it chooses to tackle that are as well developed or effective as the SystemD solution.
Of course, if someone did create a better way to solve these problems, but it wasn't compatible with systemD, then no one would use it, because they are locked into systemD.
You might then ask why no one created these better solutions 6 years ago before systemD gained its stranglehold on distributions, but that's like asking why systemD itself didn't add all these features 6 years ago.
You might then ask why no one created these better solutions 6 years ago before systemD gained its stranglehold on distributions, but that's like asking why systemD itself didn't add all these features 6 years ago.
We were all "locked into" shell scripts for service management not so long ago.
Either open source and software freedom affords us endless amounts of choice and capability to create, inspect, modify and use whatever software we like, or we're all forced to use systemd against our will. Which is it?
Systemd doesn't have a stranglehold over anything - that misinterprets the power dynamic. It's got large adoption, because it's actually useful.
Either open source and software freedom affords us endless amounts of choice and capability to create, inspect, modify and use whatever software we like, or we're all forced to use systemd against our will. Which is it?
Systemd doesn't have a stranglehold over anything - that misinterprets the power dynamic. It's got large adoption, because it's actually useful.
> We were all "locked into" shell scripts for service management not so long ago.
I don't think that's the case. Systemd was introduced with a SysV init backwards compatibility mode, so it was trivial to port services over to using it. Porting a systemd-only service over to SysV init is much harder, though. I'll be charitable and say that this one-way "interoperability" wasn't deliberate, but it does seem reminiscent of the way some proprietary software supports open standards for importing files but not exporting them.
> Either open source and software freedom affords us endless amounts of choice and capability
Software freedom is not a binary matter (excuse the pun), and different licences have given people different amounts of freedom in the past. For example, the FSF considered the pre-1999 BSD licence to be non-free simply because it required a sentence to be included in any advertisement of the product[0]. Another example would be "the Dissident test"[1] that Debian applies to licenses, considering them non-free if they require that modifications to the source code be reported back to the original author.
I think that when trying to determine the freeness of a piece of software, we have to consider the second-order effects it has on the free software ecosystem generally. If the adoption of a (hypothetical) piece of nominally free software caused other unrelated pieces of software to run artificially slower, or become insecure, or to no longer be developed, then that could be a net loss to the free software community, even if the new software has some shiny new features.
> It's got large adoption, because it's actually useful.
That's a rather self-serving interpretation of events. As another comment in this thread[2] explains, systemd at every stage of its adoption was made hard to avoid, and adopting it was made a one-way process. I also gave a more thorough analysis in a comment on another thread a couple of years ago[3], but it seems that the myths about systemd's path to world domination are becoming more and more accepted as people forget the actual history.
[0] https://www.gnu.org/licenses/bsd.html
[1] https://en.wikipedia.org/wiki/Debian_Free_Software_Guideline...
[2] https://news.ycombinator.com/item?id=26647130
[3] https://news.ycombinator.com/item?id=18879657
I don't think that's the case. Systemd was introduced with a SysV init backwards compatibility mode, so it was trivial to port services over to using it. Porting a systemd-only service over to SysV init is much harder, though. I'll be charitable and say that this one-way "interoperability" wasn't deliberate, but it does seem reminiscent of the way some proprietary software supports open standards for importing files but not exporting them.
> Either open source and software freedom affords us endless amounts of choice and capability
Software freedom is not a binary matter (excuse the pun), and different licences have given people different amounts of freedom in the past. For example, the FSF considered the pre-1999 BSD licence to be non-free simply because it required a sentence to be included in any advertisement of the product[0]. Another example would be "the Dissident test"[1] that Debian applies to licenses, considering them non-free if they require that modifications to the source code be reported back to the original author.
I think that when trying to determine the freeness of a piece of software, we have to consider the second-order effects it has on the free software ecosystem generally. If the adoption of a (hypothetical) piece of nominally free software caused other unrelated pieces of software to run artificially slower, or become insecure, or to no longer be developed, then that could be a net loss to the free software community, even if the new software has some shiny new features.
> It's got large adoption, because it's actually useful.
That's a rather self-serving interpretation of events. As another comment in this thread[2] explains, systemd at every stage of its adoption was made hard to avoid, and adopting it was made a one-way process. I also gave a more thorough analysis in a comment on another thread a couple of years ago[3], but it seems that the myths about systemd's path to world domination are becoming more and more accepted as people forget the actual history.
[0] https://www.gnu.org/licenses/bsd.html
[1] https://en.wikipedia.org/wiki/Debian_Free_Software_Guideline...
[2] https://news.ycombinator.com/item?id=26647130
[3] https://news.ycombinator.com/item?id=18879657
> I don't think that's the case. Systemd was introduced with a SysV init backwards compatibility mode, so it was trivial to port services over to using it. Porting a systemd-only service over to SysV init is much harder, though.
(I meant before systemd/upstart existed).
Is it harder because systemd supports more features than sysv? What's stopping someone writing a new init / service manager that's systemd-compatible? Or improving sysv so that it supports those features?
> Software freedom is not a binary matter (excuse the pun)...
Of course there's nuance, and you're right to point out that different corners of the open source world have different values and disagree on things (BSD/GNU etc). But the point is that the users are in control and that the power lies with the users. If systemd was bad, or actively useless, people would stop using redhat, Debian, etc. As proof that the overall model works, there are distributions out there that shun or provide non-systemd options. People can use and contribute to those!
I also don't agree that anything is a "one way process". Everything is temporary. We'll use systemd until something better comes along, just like we did with perl, apache, Hadoop, or any technology that was the de facto canonical way of doing things right up until the point that there were better options.
I remember the history too - systemd would have gotten zero traction anywhere if it wasn't actually solving real problems. Sure, there were some pretty major bugs, and I wish Debian had done it right, but that doesn't take away from the recognition that there was a problem to solve and this was our best stab at solving it.
I guess I just don't understand the energy poured into hyperbole like "world domination" and "stranglehold". If you want to convince me that something I find useful every single day is bad, then build something better.
(I meant before systemd/upstart existed).
Is it harder because systemd supports more features than sysv? What's stopping someone writing a new init / service manager that's systemd-compatible? Or improving sysv so that it supports those features?
> Software freedom is not a binary matter (excuse the pun)...
Of course there's nuance, and you're right to point out that different corners of the open source world have different values and disagree on things (BSD/GNU etc). But the point is that the users are in control and that the power lies with the users. If systemd was bad, or actively useless, people would stop using redhat, Debian, etc. As proof that the overall model works, there are distributions out there that shun or provide non-systemd options. People can use and contribute to those!
I also don't agree that anything is a "one way process". Everything is temporary. We'll use systemd until something better comes along, just like we did with perl, apache, Hadoop, or any technology that was the de facto canonical way of doing things right up until the point that there were better options.
I remember the history too - systemd would have gotten zero traction anywhere if it wasn't actually solving real problems. Sure, there were some pretty major bugs, and I wish Debian had done it right, but that doesn't take away from the recognition that there was a problem to solve and this was our best stab at solving it.
I guess I just don't understand the energy poured into hyperbole like "world domination" and "stranglehold". If you want to convince me that something I find useful every single day is bad, then build something better.
> (I meant before systemd/upstart existed).
I still don't think we were "locked into shell scripts" though. Even if shell scripts were the only thing that existed, it was still possible for someone to make a service manager which supported something else, with a backwards compatibility mode for shell scripts.
> What's stopping someone writing a new init / service manager that's systemd-compatible?
My understanding is that systemd deliberately makes it hard to independently reimplement certain critical components like logind and journald. In practice, no distro is going to go to the trouble of packaging and supporting a mixture of components from the "official" systemd set and other competing implementations, even if those competing implementations were easier to use, or faster, or more secure, or more portable.
Compare this to the situation where system services were not so tightly coupled and it was possible for distros to change their default ntpd, for example. Just as monopolies in industries prevent new innovative start-ups from emerging, a monoculture in Linux system services prevents people from even trying new approaches.
> As proof that the overall model works, there are distributions out there that shun or provide non-systemd options.
I can't help being reminded of the (in)famous scene of a Microsoft lawyer holding up a boxed copy of Red Hat as a defence against the claim that they were abusing their monopoly[0]. Alternatively, it's like an authoritarian dictatorship pointing to the existence of nominal opposition parties to show that its citizens are free.
My point is, I think you're missing some of the power dynamics here, in particular the fact that people don't want to completely reinstall their operating system just to avoid the problems of systemd. Indeed, most users wouldn't even know that systemd is the cause of their problems, especially when the problem is something nebulous like a lack of innovation and competition in the system services space.
To put it more simply: Just because lots of people don't switch away from Windows (or Facebook for example) doesn't mean that they are happy with it.
> systemd would have gotten zero traction anywhere if it wasn't actually solving real problems.
It might have solved a few people's real problems, but I think that GNOME suddenly making it a hard requirement probably mattered to more people.
> If you want to convince me that something I find useful every single day is bad, then build something better.
If you find it useful every day then I'm happy for you. However, I don't think I have to convince you of anything. I'm not trying to make it harder for you to use systemd, whereas systemd is making it harder for people to use other inits, including SysV init which has been working fine for them for decades. The onus should be on the ones doing the coercion, to convince the people whose choices they are reducing, not the other way around.
[0] https://archive.is/eMmMB / https://www.washingtonpost.com/wp-srv/business/longterm/micr...
I still don't think we were "locked into shell scripts" though. Even if shell scripts were the only thing that existed, it was still possible for someone to make a service manager which supported something else, with a backwards compatibility mode for shell scripts.
> What's stopping someone writing a new init / service manager that's systemd-compatible?
My understanding is that systemd deliberately makes it hard to independently reimplement certain critical components like logind and journald. In practice, no distro is going to go to the trouble of packaging and supporting a mixture of components from the "official" systemd set and other competing implementations, even if those competing implementations were easier to use, or faster, or more secure, or more portable.
Compare this to the situation where system services were not so tightly coupled and it was possible for distros to change their default ntpd, for example. Just as monopolies in industries prevent new innovative start-ups from emerging, a monoculture in Linux system services prevents people from even trying new approaches.
> As proof that the overall model works, there are distributions out there that shun or provide non-systemd options.
I can't help being reminded of the (in)famous scene of a Microsoft lawyer holding up a boxed copy of Red Hat as a defence against the claim that they were abusing their monopoly[0]. Alternatively, it's like an authoritarian dictatorship pointing to the existence of nominal opposition parties to show that its citizens are free.
My point is, I think you're missing some of the power dynamics here, in particular the fact that people don't want to completely reinstall their operating system just to avoid the problems of systemd. Indeed, most users wouldn't even know that systemd is the cause of their problems, especially when the problem is something nebulous like a lack of innovation and competition in the system services space.
To put it more simply: Just because lots of people don't switch away from Windows (or Facebook for example) doesn't mean that they are happy with it.
> systemd would have gotten zero traction anywhere if it wasn't actually solving real problems.
It might have solved a few people's real problems, but I think that GNOME suddenly making it a hard requirement probably mattered to more people.
> If you want to convince me that something I find useful every single day is bad, then build something better.
If you find it useful every day then I'm happy for you. However, I don't think I have to convince you of anything. I'm not trying to make it harder for you to use systemd, whereas systemd is making it harder for people to use other inits, including SysV init which has been working fine for them for decades. The onus should be on the ones doing the coercion, to convince the people whose choices they are reducing, not the other way around.
[0] https://archive.is/eMmMB / https://www.washingtonpost.com/wp-srv/business/longterm/micr...
There’s nothing particularly cumbersome about init scripts.
On a system where the primary API is the shell, initialising everything with a sequence of imperative shell commands makes a lot of sense.
On a system where the primary API is the shell, initialising everything with a sequence of imperative shell commands makes a lot of sense.
Casual admins made race conditions and other security bugs in init scripts.
For me systemd created more race-conditions than it fixed (especially if the old init system started things serially).
Getting it right in serial fashion is fairly easy, but since systemd insists on doing as much as possible in parallel you really need to have every dependency correctly configured (which is not always that easy to get right if things take a couple of seconds to get going..).
Getting it right in serial fashion is fairly easy, but since systemd insists on doing as much as possible in parallel you really need to have every dependency correctly configured (which is not always that easy to get right if things take a couple of seconds to get going..).
What? You have to specify a dependency in the .service file, what does it matter how much time does any of them take? That’s the whole purpose of dependency management.
Either that is missing or you have some other config error, like the service believing it is ready, because it is not told about its state correctly?
But there is a setting to fall back to serial mode I believe, but it will also not work correctly if your service files are configured incorrectly.
Either that is missing or you have some other config error, like the service believing it is ready, because it is not told about its state correctly?
But there is a setting to fall back to serial mode I believe, but it will also not work correctly if your service files are configured incorrectly.
Exactly my point. You have to configure all dependencies correctly as well as have all the dependencies themselves configured correctly to make systemd aware at what point they are ready (which again can be non-trivial to do).
It is much less forgiving to mistakes (and yes.. the little annoyance I am having right now has the dependencies set correctly, correct service type, systemd-analyze chain looks good.. so mistake is somewhat well hidden).
It is much less forgiving to mistakes (and yes.. the little annoyance I am having right now has the dependencies set correctly, correct service type, systemd-analyze chain looks good.. so mistake is somewhat well hidden).
What exactly is your point? That systemd "created more race-conditions than it fixed", as you stated, or that systemd actually does solve race conditions, only that it is cumbersome to configure it?
Not OP. I suppose though, that the point is that a system in which things happen serially you don't really have race conditions. Things either work, or they don't. If they don't, it's because they happen in the wrong order.
One of the (very many) things that gave me and my colleagues a headache when switching to Systemd was two init scripts that could run one after the other (and the order didn't matter) but they could not run simultaneously. This had worked fine with System V style init scripts that ran serially, but started breaking in Systemd.
Of course you can now argue that this is a configuration error - however, it does not change the fact there there didn't use to be a race condition. Now there was one. One is a larger number than zero.
One of the (very many) things that gave me and my colleagues a headache when switching to Systemd was two init scripts that could run one after the other (and the order didn't matter) but they could not run simultaneously. This had worked fine with System V style init scripts that ran serially, but started breaking in Systemd.
Of course you can now argue that this is a configuration error - however, it does not change the fact there there didn't use to be a race condition. Now there was one. One is a larger number than zero.
Just for the sake of discussion, if we assume any general, generic computer process that works fine serially breaks in a parallel environment, is that really a valid argument for or against parallelism?
In the short term, things breaking like this are of course frustrating, but if it was my environment I’d prefer that this dependency was an explicit part of the configuration rather than something that works “by accident”, unless one prefers that initialization should never be anything but serial, but I personally at least don’t see the benefits of that outweighing what systemd offers.
In the short term, things breaking like this are of course frustrating, but if it was my environment I’d prefer that this dependency was an explicit part of the configuration rather than something that works “by accident”, unless one prefers that initialization should never be anything but serial, but I personally at least don’t see the benefits of that outweighing what systemd offers.
> is that really a valid argument for or against parallelism?
Not really. That wasn't the question though. The question was "were there more race condition with the parallel system than with the serial system?". The answer was undeniably, objectively yes.
Not really. That wasn't the question though. The question was "were there more race condition with the parallel system than with the serial system?". The answer was undeniably, objectively yes.
I think the question posed is really odd, because race conditions cannot really exist as long as you only assume serial processing. I mean, the definition of a race condition means it's dependent on timing, which is an excluded risk if you're not attempting to start your services in parallel. So a better question in my opinion is: does the tradeoffs mean we should go back to serial execution? My feeling is no.
I mean, if they really only work when not running at the same time, they do something really really shady, so I would not really use them on production systems even in serial unless I figure out what exactly is wrong with them. It may very well be the case that they are buggy even in serial, it just not as obvious/doesn’t happen that often that way.
Nonetheless, parallelism is an additional feature (in my opinion, greatly appreciated), but it can be turned off. Additional functionality is rarely bad over not having it.
Nonetheless, parallelism is an additional feature (in my opinion, greatly appreciated), but it can be turned off. Additional functionality is rarely bad over not having it.
> if they really only work when not running at the same time, they do something really really shady
They configured firewall rules using iptables. I'm not sure I consider configuring firewall rules at startup using an init script "really really shady". Turns out (quite unsurprisingly) that you can't run multiple instances of the iptables command simultaneously.
I don't really want to get into the whole debate about whether parallelizing startup scripts is better or not. I'm not saying it's not, but that's beside the point. The point was that parallelizing startup scripts leads to a set of problems that didn't exist before. It's annoying to see people argue that those problems don't exist. It's like arguing that the problem of flat tyres aren't more common for people driving cars than they are for people riding horses. It's utter nonsense, no matter how many other advantages cars may have.
They configured firewall rules using iptables. I'm not sure I consider configuring firewall rules at startup using an init script "really really shady". Turns out (quite unsurprisingly) that you can't run multiple instances of the iptables command simultaneously.
I don't really want to get into the whole debate about whether parallelizing startup scripts is better or not. I'm not saying it's not, but that's beside the point. The point was that parallelizing startup scripts leads to a set of problems that didn't exist before. It's annoying to see people argue that those problems don't exist. It's like arguing that the problem of flat tyres aren't more common for people driving cars than they are for people riding horses. It's utter nonsense, no matter how many other advantages cars may have.
Who is arguing that these problems don't exist?
Yeah I believe Rust is also much less forgiving to mistakes than C is, which makes it a bit less beginner friendly, but do we really want to allow badly written things to run?
Also, are you sure it is a race condition? Some other errors can also appear as races. What service are you trying to create?
Also, are you sure it is a race condition? Some other errors can also appear as races. What service are you trying to create?
Not sure your analogy really works here, since languages like Haskell, Rust, etc. will tell you at compile-time that you have made some mistake.
That's not how systemd (or any init system) works though. Things break at runtime.
It sure would be nice to have a system that could tell me stuff won't work before running it, but that's just not possible.
And my problem is the very definition of a race-conditions. It doesn't always occur and if I play with the timing a bit I can make it go away. The dependency unit is a oneshot script that adds some ip addresses to interfaces. The dependent job tries to bind to one of those ip addresses. And yes the interface is up before this all happens.
And my problem is the very definition of a race-conditions. It doesn't always occur and if I play with the timing a bit I can make it go away. The dependency unit is a oneshot script that adds some ip addresses to interfaces. The dependent job tries to bind to one of those ip addresses. And yes the interface is up before this all happens.
A lot of init script writing was made easy for years with tools like the lsb* functions and start-stop-daemon.
It would’ve been cool to have functions that supported dependency in the scripts themselves. A web server would have a call to...
Doing all this with INI files is very clever but it’s always felt very NIH. Good solutions solve existing problems without rewriting and embracing everything from the ground up.
It would’ve been cool to have functions that supported dependency in the scripts themselves. A web server would have a call to...
init_wait_for networking
init_ready networking
Doing all this with INI files is very clever but it’s always felt very NIH. Good solutions solve existing problems without rewriting and embracing everything from the ground up.
it is live in Fedora 34. You should try it - its really pretty good. between Gnome 40, Pipewire audio and oomd...works pretty good
I mostly like systemd. I definitely like writing unit files a heck of a lot more than writing shell scripts; it's really easy to write bad shell, and we didn't have shellcheck to help us out back in the old days. The parts that I don't like, I don't use - it's really more of a monorepo than a monolith.
That being said, I really wish there was more interest from the systemd folks in interoperating with containers and registries and all things OCI in general. Like the portable services they introduced in 239, the storage for the system extensions introduced in 248 is backed by either a directory or a disk image; meanwhile, there's an entire ecosystem of tools in the container world dedicated to the storage and distribution of root filesystems that they seem to be completely ignoring. I know there's nothing stopping me from using `umoci` or some such to pull and unpack a container and use the resulting directory as either a system extension or a source of portable services, but it'd be a lot cooler if that kind of functionality was baked in.
That being said, I really wish there was more interest from the systemd folks in interoperating with containers and registries and all things OCI in general. Like the portable services they introduced in 239, the storage for the system extensions introduced in 248 is backed by either a directory or a disk image; meanwhile, there's an entire ecosystem of tools in the container world dedicated to the storage and distribution of root filesystems that they seem to be completely ignoring. I know there's nothing stopping me from using `umoci` or some such to pull and unpack a container and use the resulting directory as either a system extension or a source of portable services, but it'd be a lot cooler if that kind of functionality was baked in.
> I definitely like writing unit files a heck of a lot more than writing shell scripts;
No argument there (even though declarative programming languages aren't universally favored over imperative ones), but why does this come up so often? On a typical Linux desktop or server there are hundreds, if not thousands of packages installed, yet only a few dozen init scripts. Almost no tool or application is typically run as service requiring such.
No argument there (even though declarative programming languages aren't universally favored over imperative ones), but why does this come up so often? On a typical Linux desktop or server there are hundreds, if not thousands of packages installed, yet only a few dozen init scripts. Almost no tool or application is typically run as service requiring such.
Possibly it's because initscripts were such a painful experience that they weren't used as extensively. I find myself making systemd units often for small things that I wouldn't have wanted to write a whole initscript for.
Just the dependency resolution features alone make it fantastic. It's only a few lines to specify a network dependency, a filesystem dependency, another service dependency, timeout and retry behaviors, monitoring and alerting on failures, etc.
Plus it goes beyond just init. The scheduling features mean I'm turning to it instead of something like cron for doing repeated tasks on a server. Again, there's just a bunch of quality-of-life improvements there, like dependencies, better and more granular logging, error reporting, retries with limits, etc.
Just the dependency resolution features alone make it fantastic. It's only a few lines to specify a network dependency, a filesystem dependency, another service dependency, timeout and retry behaviors, monitoring and alerting on failures, etc.
Plus it goes beyond just init. The scheduling features mean I'm turning to it instead of something like cron for doing repeated tasks on a server. Again, there's just a bunch of quality-of-life improvements there, like dependencies, better and more granular logging, error reporting, retries with limits, etc.
> Almost no tool or application is typically run as service requiring such.
I build embedded systems and appliances; it sounds like you might have to deal with this a lot less frequently than I do. I don't have to deal with it as frequently in the age of systemd though, because more projects are providing unit files now that it's everywhere, and it's a lot easier to glance at a systemd unit someone else wrote and decide if it's correct than it is to audit a random shell script and figure out what needs to be added or changed to get it to work correctly in your distro of choice.
I build embedded systems and appliances; it sounds like you might have to deal with this a lot less frequently than I do. I don't have to deal with it as frequently in the age of systemd though, because more projects are providing unit files now that it's everywhere, and it's a lot easier to glance at a systemd unit someone else wrote and decide if it's correct than it is to audit a random shell script and figure out what needs to be added or changed to get it to work correctly in your distro of choice.
> it sounds like you might have to deal with this a lot less frequently than I do.
That'll be the case. When I did work on an embedded system (enterprise storage system) different groups worked on different services (and some used hair-rising shell scripts to start their service), but our team's service was started via xinetd and we didn't spend all that much time on that aspect.
That'll be the case. When I did work on an embedded system (enterprise storage system) different groups worked on different services (and some used hair-rising shell scripts to start their service), but our team's service was started via xinetd and we didn't spend all that much time on that aspect.
Systemd continues to remind me of windows. It keeps fixing what wasn't broken, and you don't get to say 'no thank you' without a complete OS switch.
Yes you can argue it's the distros decision, but that ship has sailed and fairly soon there won't even be an effective choice.
Yes you can argue it's the distros decision, but that ship has sailed and fairly soon there won't even be an effective choice.
Seems perfectly useful to me. The stated intention is to use that to provide things like debugging tools in setups that are deliberately minimal by design and don't include any troubleshooting functionality, maybe not even a package manager to install any with.
A bit of a niche use perhaps, but seems to be very useful to me. And it's good that it's in systemd because otherwise you'd have to hack the boot process of every distro you might want this in.
And I wouldn't say it wasn't broken, it's just a need you personally didn't have. I do actually have this need in a virtualization context, and may well making use of it.
A bit of a niche use perhaps, but seems to be very useful to me. And it's good that it's in systemd because otherwise you'd have to hack the boot process of every distro you might want this in.
And I wouldn't say it wasn't broken, it's just a need you personally didn't have. I do actually have this need in a virtualization context, and may well making use of it.
Would an approach like this not also serve your needs?
https://www.unix.com/man-page/FreeBSD/8/rescue/
https://www.unix.com/man-page/FreeBSD/8/rescue/
There's other approaches would work for me, yes. I could also download a tar file with what's missing and unpack it over root, I suppose. But then root would need to be read-write, like your solution also requires.
Also, FreeBSD wouldn't work. We use many Linux specific features, and I have no interest whatsoever in porting all that stuff to BSD for no gain in functionality.
Also, I see no issue to somebody else solving my problems and not needing to write more stuff that's unrelated to the actual purpose of what I'm doing.
Also, FreeBSD wouldn't work. We use many Linux specific features, and I have no interest whatsoever in porting all that stuff to BSD for no gain in functionality.
Also, I see no issue to somebody else solving my problems and not needing to write more stuff that's unrelated to the actual purpose of what I'm doing.
> you can argue it's the distros decision
Is this not a fact, versus a mere argument?
Do you think the decision is not based on merit? b/c there's all the effective distros switching, versus you saying nothing is broken - one of you must be wrong?
Maybe the Linux ecosystem is broken, you just can't say it because you've already adjusted you expectations, or solidified your use case(s)?
Is this not a fact, versus a mere argument?
Do you think the decision is not based on merit? b/c there's all the effective distros switching, versus you saying nothing is broken - one of you must be wrong?
Maybe the Linux ecosystem is broken, you just can't say it because you've already adjusted you expectations, or solidified your use case(s)?
RedHat made the decision based on their commercial interests, not technical merit, and since RedHat de facto controls GNOME anyone who wanted to keep shipping GNOME had to fall into line. Debian were probably the only distro who had enough clout to push back against GNOME, and there was a bitter political fight there (the decision was certainly not based on technical merit, nor did the committee even follow their own rules). Slackware wisely chose to drop GNOME but most distros didn't dare; Ubuntu tried to go their own way for a while but they couldn't maintain everything on their own.
Honestly the whole thing made me more or less give up on open-source as a development model. Good programmers are expensive, especially if you want them to do the boring maintenance work, and that funding has to come from somewhere. The "give the program away for free and sell consulting" business model works up to a point, but it distorts maintainers' interests away from what would actually benefit the majority of users. While I do blame RedHat, the real problem is that no-one except RedHat is paying people to work full-time on GNOME or any of dozens of similarly important projects (KDE seems to be mostly funded by the EU, which is a much healthier model, but I don't know if you could make a whole distribution from projects like that).
Honestly the whole thing made me more or less give up on open-source as a development model. Good programmers are expensive, especially if you want them to do the boring maintenance work, and that funding has to come from somewhere. The "give the program away for free and sell consulting" business model works up to a point, but it distorts maintainers' interests away from what would actually benefit the majority of users. While I do blame RedHat, the real problem is that no-one except RedHat is paying people to work full-time on GNOME or any of dozens of similarly important projects (KDE seems to be mostly funded by the EU, which is a much healthier model, but I don't know if you could make a whole distribution from projects like that).
90% of the switch was because systemd made the distributions job much easier: it basically replaced a bunch of effort each distribution had to maintain themselves. There really was not a lot of political pressure as systemd opponents seem to imply.
Exactly, complain is easy, but only Red-Hat, Ubuntu and a couple of others are willing to put in the money.
Having been born during 70's I also saw all those ideals turn into business people the following decade.
It is easy to be idealistic when others put in the money.
Having been born during 70's I also saw all those ideals turn into business people the following decade.
It is easy to be idealistic when others put in the money.
No other package upgrade causes me to wonder "what will this subsume today?"
Few other packages are as fundamental
> Systemd continues to remind me of windows.
How is it anything like Windows? You're honestly, without bad faith, comparing it to a closed source operating systems that has telemetry enabled by default and forces you to run system updates?
I have no idea what systemd-oomd is, but from a cursory reading of the article I can tell that it's a .service, and from the description that it's a "userspace out-of-memory (OOM) killer". So either this service will (worst-case) be enabled by default, in which case I presume that you can disable it or mask it completely like any service, or (best-case) it will be disabled by default, in which case I really don't understand why it matters.
In addition to systemd fully allowing you to disable and re-configure much of its functionality, it's also fully open source. You can re-build it however you want, if you want to use it in another manner. This is what passes for being reminiscent of Windows?
How is it anything like Windows? You're honestly, without bad faith, comparing it to a closed source operating systems that has telemetry enabled by default and forces you to run system updates?
I have no idea what systemd-oomd is, but from a cursory reading of the article I can tell that it's a .service, and from the description that it's a "userspace out-of-memory (OOM) killer". So either this service will (worst-case) be enabled by default, in which case I presume that you can disable it or mask it completely like any service, or (best-case) it will be disabled by default, in which case I really don't understand why it matters.
In addition to systemd fully allowing you to disable and re-configure much of its functionality, it's also fully open source. You can re-build it however you want, if you want to use it in another manner. This is what passes for being reminiscent of Windows?
With windows, (and I suggest systemd) you're either all in or all out.
And as the scope continues to increase, you're still either all in or all out.
As the distros fall into line, you're either all in or all out.
Try and not use systemd, and then tell me it's not a problem. I think it's honestly easier to not use windows at this stage. The BSDs (love them as I do) just don't have the clout or the support for many day to day developer needs.
And as the scope continues to increase, you're still either all in or all out.
As the distros fall into line, you're either all in or all out.
Try and not use systemd, and then tell me it's not a problem. I think it's honestly easier to not use windows at this stage. The BSDs (love them as I do) just don't have the clout or the support for many day to day developer needs.
> With windows, (and I suggest systemd) you're either all in or all out.
This is not the case with systemd. Although these projects live in the same repository, distributions can pick and choose them depending on your priorities.
Granted, this is obviously not something most end users come in contact with and even know since very few people actually build distributions. As an end user, ripping out a core part of a distro to replace it is always a painful experience. Regardless of whether that component lives in the systemd repo or somewhere else. Systemd actually makes ripping out core parts easier, which is why arch adopted it so quickly, but that's another story.
You're just throwing vague and incorrect statements out there, sprinkle "Windows" on top in the hopes of eliciting an emotional response.
This is not the case with systemd. Although these projects live in the same repository, distributions can pick and choose them depending on your priorities.
Granted, this is obviously not something most end users come in contact with and even know since very few people actually build distributions. As an end user, ripping out a core part of a distro to replace it is always a painful experience. Regardless of whether that component lives in the systemd repo or somewhere else. Systemd actually makes ripping out core parts easier, which is why arch adopted it so quickly, but that's another story.
You're just throwing vague and incorrect statements out there, sprinkle "Windows" on top in the hopes of eliciting an emotional response.
* systemd-cryptsetup gained support for unlocking LUKS2 volumes using
TPM2 hardware, as well as FIDO2 security tokens (in addition to the
pre-existing support for PKCS#11 security tokens).
You've been able to do LUKS filesystems auto-unlocked by TPMs before; the new thing here is just that systemd-cryptsetup can do it.
I don't get why this is a feature that people want. In case of theft / loss it doesn't offer any security benefits over no encryption.
The main thing this is used for in the Linux world is as part of a trusted boot pathway. It allows you to use the TPM to perform measurements on the boot chain and refuse to decrypt the data on the encrypted filesystem if it fails those checks.
As the sibling has said, this is usually used with SecureBoot. The system won't unlock the drives if, for example, you attempt booting an unsigned kernel image, change the boot order or otherwise tamper with the system setup.
Granted, in case of theft, the computer will boot up if nothing is done to it. But in the case of a "user" computer where no services are listening, the risk is relatively low. In order to get access to the decrypted disk contents you'd have to exploit some vulnerability in Linux and / or the login manager.
Granted, in case of theft, the computer will boot up if nothing is done to it. But in the case of a "user" computer where no services are listening, the risk is relatively low. In order to get access to the decrypted disk contents you'd have to exploit some vulnerability in Linux and / or the login manager.
If you can boot it up there are ways to hack it. Almost every machine has some services exposed and there are also hardware-based attacks possible.
Your points are true. But I don't think we're talking about absolutes here.
This is a tradeoff. A "random user"[0] might be put off by having to type in a password every time they boot up their PC and forego disk encryption altogether. This approach is a net improvement over an unencrypted drive, even if it's not perfect.
As usual in matters of security, threat modelling is important. Who are you and who are your defending againts?
Are you some high(-enough)-profile target with potential access to interesting infrastructure or data? You may want to not rely on only this approach. As siblings have said, some kind of supplemental factor could be better. Say a challenge-response factor based on a Yubikey. Also, if the thief is determined, they may attempt to steal the laptop while it's running. You may want to look into desktops if that's the case. And not plug them into a UPS, of course.
But are you just a random Joe whose threat model is that you have some photos you wouldn't like seen, maybe some personal documents, but not much more? Then you need "just enough protection". If someone steals your PC, it's likely for the PC itself and not for the data it carries. They may want to check what's on it, see if there's anything juicy. Might even try "password" or something to unlock it. But they will probably not spend too much time trying to brute force Linux (or Windows for that matter).
> Almost every machine has some services exposed
How often is this the case on a random user's machine?
Even on my Windows machine I've only got SMB and RDP listening. Which I've manually enabled.
On my Arch Linux machine, which is my actual work machine, only SSH listens. Which I've manually installed and enabled. On an ubuntu desktop 20.10 install, there's no SSH server installed. There are no daemons listening on the external interface.
I guess you may attempt some attack against the DHCP client if you really want to get into that machine. I also suppose the pipe wrench exploit can still be tried.
> and there are also hardware-based attacks possible
Don't you think that if someone's going to invest the time and ressources to conduct such an attack against your PC you might be the kind of target who should be aware that's a possibility? Which brings us back to the point above about threat-modelling.
---
[0] I realize we're talking about LUKS and Linux, so maybe the "random user" isn't all that random. But maybe it's a "mom's computer" on which someone installed Ubuntu or something.
This is a tradeoff. A "random user"[0] might be put off by having to type in a password every time they boot up their PC and forego disk encryption altogether. This approach is a net improvement over an unencrypted drive, even if it's not perfect.
As usual in matters of security, threat modelling is important. Who are you and who are your defending againts?
Are you some high(-enough)-profile target with potential access to interesting infrastructure or data? You may want to not rely on only this approach. As siblings have said, some kind of supplemental factor could be better. Say a challenge-response factor based on a Yubikey. Also, if the thief is determined, they may attempt to steal the laptop while it's running. You may want to look into desktops if that's the case. And not plug them into a UPS, of course.
But are you just a random Joe whose threat model is that you have some photos you wouldn't like seen, maybe some personal documents, but not much more? Then you need "just enough protection". If someone steals your PC, it's likely for the PC itself and not for the data it carries. They may want to check what's on it, see if there's anything juicy. Might even try "password" or something to unlock it. But they will probably not spend too much time trying to brute force Linux (or Windows for that matter).
> Almost every machine has some services exposed
How often is this the case on a random user's machine?
Even on my Windows machine I've only got SMB and RDP listening. Which I've manually enabled.
On my Arch Linux machine, which is my actual work machine, only SSH listens. Which I've manually installed and enabled. On an ubuntu desktop 20.10 install, there's no SSH server installed. There are no daemons listening on the external interface.
I guess you may attempt some attack against the DHCP client if you really want to get into that machine. I also suppose the pipe wrench exploit can still be tried.
> and there are also hardware-based attacks possible
Don't you think that if someone's going to invest the time and ressources to conduct such an attack against your PC you might be the kind of target who should be aware that's a possibility? Which brings us back to the point above about threat-modelling.
---
[0] I realize we're talking about LUKS and Linux, so maybe the "random user" isn't all that random. But maybe it's a "mom's computer" on which someone installed Ubuntu or something.
It does raise the bar though compared to a completely unencrypted drive, and given it's seamless if done properly, it's a nice way to increase security without compromising user experience at all.
The alternative is to set a PIN on the TPM. It will not unlock without it and go into lockout after a few attempts.
Some people consider it more convenient to just boot the thing, take the protection against casual loss of confidentiality and accept the risk they might face from determined attackers.
Some people consider it more convenient to just boot the thing, take the protection against casual loss of confidentiality and accept the risk they might face from determined attackers.
I know at least one person using bitlocker (windows) with this TPM unlock feature and lost access to his data when he had to get his laptop repaired.
People need to be aware this can happen when using it and should remember to add a password key to their luks container.
People need to be aware this can happen when using it and should remember to add a password key to their luks container.
The whole song and dance where Windows forces you to save the recovery key on an external medium and tells you what it's for must have completely slipped his mind.
Though I'm fairly certain that Linux will forego this and just assumes you known what you're doing.
Though I'm fairly certain that Linux will forego this and just assumes you known what you're doing.
>Though I'm fairly certain that Linux will forego this and just assumes you known what you're doing.
Yes, even the documentation for the systemd feature implies generating the recovery key is a separate optional step.
Yes, even the documentation for the systemd feature implies generating the recovery key is a separate optional step.
The auto-unlock thing is more for Datacenters etc. If your SSD/HD explode you can safely throw them into the recycle-bin, without the fear that someone can access the data stored on.
> have encrypted disks and have a machine boot unassisted.
That has been possible (without TPM) since about 2009: Shameless plug: https://www.recompile.se/mandos
That has been possible (without TPM) since about 2009: Shameless plug: https://www.recompile.se/mandos
A cautionary tale on latest Linux, LUKS, and LVM’d swap partition: it still does desktop jankiness (intermittent screen freezes during swap).
Not for me.
Intel 960 graphic card here. XWayland. KDE. BTW, I meant encrypted swap->LUKS->LVM->md3->hds1
Ah, I'm missing the MD step. that might be the cause.
Does this mean I can boot and unlock a volume (/home for example) with just a Yubikey placed into the USB, and without it inserted it won't unlock it? (maybe falling back to a passphrase)
I think so. This was posted a few weeks ago ahead of the 248 release. http://0pointer.net/blog/unlocking-luks2-volumes-with-tpm2-f...
I'm gonna try it out once my Solo V2 arrives.
I'm gonna try it out once my Solo V2 arrives.
Very nice, will have to try it out! Thanks for the link.
https://www.freedesktop.org/software/systemd/man/systemd-sys...
Somebody has got to stop systemd from unilaterally fucking up every Linux distribution's design with their wacky features. Only /opt and /usr? Because no other directories might need/want an overlay? And why the fuck can't we just use a regular overlay fs? To say nothing of it actually breaking the expectations of users and operators ("In addition to your existing huge troubleshooting checklist, see if systemd is silently injecting fake files into your process's path")
This whole thing is designed like Kubernetes. Come up with a bad design, then hack in random functionality to make it as complicated as possible to use.
Somebody has got to stop systemd from unilaterally fucking up every Linux distribution's design with their wacky features. Only /opt and /usr? Because no other directories might need/want an overlay? And why the fuck can't we just use a regular overlay fs? To say nothing of it actually breaking the expectations of users and operators ("In addition to your existing huge troubleshooting checklist, see if systemd is silently injecting fake files into your process's path")
When the hostname is set explicitly to "localhost", systemd-hostnamed
will respect this. Previously such a setting would be mostly silently
ignored.
This whole thing is designed like Kubernetes. Come up with a bad design, then hack in random functionality to make it as complicated as possible to use.
They want to sell you hosted systemd ;-)
More likely they want to create an "app store" where they control which versions of each package you install, and when, and how much you pay for them.
Actually read the link. It reads like an OS change log.
e.g. https://www.freebsd.org/releases/11.0R/relnotes/
e.g. https://www.freebsd.org/releases/11.0R/relnotes/
Seems like it would, given all the spaces it's in:
systemd-networkd, systemd-resolved, systemd-timesyncd, systemd-udevd, systemd-journald, systemd-logind, systemd-nspawn, systemd-cron, systemd-notify, systemd-container, systemd-boot, systemd-oomd, systemd-mount, dbus, ...
systemd-networkd, systemd-resolved, systemd-timesyncd, systemd-udevd, systemd-journald, systemd-logind, systemd-nspawn, systemd-cron, systemd-notify, systemd-container, systemd-boot, systemd-oomd, systemd-mount, dbus, ...
and there are good & healthy improvements to each of these subsystems.
and oh good --json logging support is available everywhere. nice SYSTEMD_COLORS supports new 16 and 256 options for all these tools. my entire system will work much better. with high consistency.
this whole monorepo (not monolith) thing is working out.
and oh good --json logging support is available everywhere. nice SYSTEMD_COLORS supports new 16 and 256 options for all these tools. my entire system will work much better. with high consistency.
this whole monorepo (not monolith) thing is working out.
It varies. Their sntp client steps time instead of skewing, for example.
systemd-timesyncd is an SNTP client; that's what an SNTP client does. For a lot of use cases, desktops and even most servers, it's perfectly fine and a whole lot less complex.
I understand it's not a good fit in some cases; our standard server config disables all other NTP clients, systemd-timesyncd included, in favor of Chrony because we are concerned about highly accurate system clocks and slewing. Interestingly, it appears that systemd has some sort of provisions for this. The `timedatectl` command still works and reports that NTP is enabled and synchronized.
I understand it's not a good fit in some cases; our standard server config disables all other NTP clients, systemd-timesyncd included, in favor of Chrony because we are concerned about highly accurate system clocks and slewing. Interestingly, it appears that systemd has some sort of provisions for this. The `timedatectl` command still works and reports that NTP is enabled and synchronized.
"that's what an SNTP client does"
How so? Compare what msntp does to adjust the time with what systemd-timesync does. Though msntp does have an "immediate" option if you want to be abrupt.
How so? Compare what msntp does to adjust the time with what systemd-timesync does. Though msntp does have an "immediate" option if you want to be abrupt.
Previous discussion, for the v248 release candidate: https://news.ycombinator.com/item?id=26450960
There's a bunch of discussion of systemd-oomd and a bit of discussion of systemd extension images. Much of it is even calm and technical.
There's a bunch of discussion of systemd-oomd and a bit of discussion of systemd extension images. Much of it is even calm and technical.
Come on, what was wrong with the previous title? It was more informative, and "Systemd 248" is technically not the original title either.
`ExtensionImages` (second bullet point) seems it would be essentially the same as having a regular Portable Service [1]. I get they needed the field for the system-level sysext service, so they just made it available to every other unit too. But is there any reason to use `ExtensionImages` instead of making a Portable Service?
Edit: I guess if you have multiple services from multiple vendors that nevertheless agree to some common base image, it makes sense for those services to use `ExtensionImages`. Eg multiple services that all require a libglib base image. A Portable Service can expose multiple services from a single disk image, but that's the point - the vendors have to build a single disk image with their services.
[1]: https://github.com/systemd/systemd/blob/main/docs/PORTABLE_S...
Edit: I guess if you have multiple services from multiple vendors that nevertheless agree to some common base image, it makes sense for those services to use `ExtensionImages`. Eg multiple services that all require a libglib base image. A Portable Service can expose multiple services from a single disk image, but that's the point - the vendors have to build a single disk image with their services.
[1]: https://github.com/systemd/systemd/blob/main/docs/PORTABLE_S...
How efficient is overlayfs? Could you use this to build a regular packaging system where instead of installing .debs/.rpms directly, they come as extension images and you just mount them in?
I sometimes wonder why we even install stuff - mount it it in from remote with some kind of caching system, and you'd always have the latest, like on the web. Slap a delta algorithm on top, and it could even be more efficient.
This is not to say that there's no value in freezing stuff and staying in control - there is - but ideally we would be able to choose.
I sometimes wonder why we even install stuff - mount it it in from remote with some kind of caching system, and you'd always have the latest, like on the web. Slap a delta algorithm on top, and it could even be more efficient.
This is not to say that there's no value in freezing stuff and staying in control - there is - but ideally we would be able to choose.
isn't this what things like flatpack/snap try to offer?
Flatpak/snap/AppImage have been a godsend for me. I can now finally have a satisfactory easy balance between security/maturity of my system and run the latest software without having to compile it, and sometimes some of its dependencies, myself.
Flatpak/snap are other part of the same puzzle.
Think of them as the "app" part similar to mobile OSes; fedora silverblue/rpm-ostree/systemd-extensions as the "system" part.
Think of them as the "app" part similar to mobile OSes; fedora silverblue/rpm-ostree/systemd-extensions as the "system" part.
Yes, that's been one of Poettering's long-term goals [1] [2], and what Fedora SilverBlue and OSTree are trying to achieve. systemd features like the ability to create users via installable files [3] are geared towards this goal.
[1]: http://0pointer.net/blog/projects/stateless.html
[2]: http://0pointer.net/blog/revisiting-how-we-put-together-linu...
[3]: https://www.freedesktop.org/software/systemd/man/sysusers.d....
[1]: http://0pointer.net/blog/projects/stateless.html
[2]: http://0pointer.net/blog/revisiting-how-we-put-together-linu...
[3]: https://www.freedesktop.org/software/systemd/man/sysusers.d....
In my experience the performance is nearly the same as without. The code is fairly optimised.
And I agree: one could use systemd-sysext with something like rpm-ostree to create very robust systems with selective immutability.
And I agree: one could use systemd-sysext with something like rpm-ostree to create very robust systems with selective immutability.
The CERN have something exactly like that: https://news.ycombinator.com/item?id=25652503
Apparently not (yet), see Uses in https://www.freedesktop.org/software/systemd/man/systemd-sys...
A friend of mine built something like this 10y ago or so - a fuse filesystem backend for a NixOS/Nix-store with built-in caching.
So whenever you started an application/version you never ran before, it was just downloaded on demand and the only thing you might have noticed is a slight delay in startup duration.
By just regularly updating the configured Nix-channel it was always the latest software of this channel running in the local environment.
slax did this a long time ago.
Is the use case desktops / servers, or embedded systems? There's a lot of crypto / secure boot stuff in there that looks complicated, but could be pretty neat if it works...
"The primary use case for system images are immutable environments where debugging and development tools shall optionally be made available, but not included in the immutable base OS image itself (e.g. strace and gdb shall be an optionally installable addition in order to make debugging/development easier). System extension images should not be misunderstood as a generic software packaging framework, as no dependency scheme is available: system extensions should carry all files they need themselves, except for those already shipped in the underlying host system image. Typically, system extension images are built at the same time as the base OS image — within the same build system."
https://www.freedesktop.org/software/systemd/man/systemd-sys...
https://www.freedesktop.org/software/systemd/man/systemd-sys...
What's the difference between this and just adding an overlayfs mount to /etc/fstab?
[deleted]
[deleted]
I can’t wait to have to debug that in production.
And by "that" you mean which, exactly, of the 77 bullet points in the release notes?
That's hard to say upfront. Last time it was systemd-homed randomly blocking ssh access to remote systems where it was not even configured to be used, other than distro sticking its pam module to default pam config. What will be the killer future this time is yet to be seen.
My last systemd mystery was nohup, tmux, etc, not being able to leave a process running. "Working as designed", sure, but it took a while for me to suspect systemd was involved and figure it out.
Is there a document somewhere that shows what the systemd team thinks is a functional boundary? What things they don't plan on replacing?
Is there a document somewhere that shows what the systemd team thinks is a functional boundary? What things they don't plan on replacing?
systemd is god now and he’s not benevolent. More lord cthulhu.
It sounds like either your distro f*cked up here, or you didn’t read the update notes, not sure what does it have to do with systemd..
Why a disabled feature that should just do nothing even if its pam module is loaded, infinitely loops on D-BUS access in its pam module?
Anyway, I reported it and it presumably got fixed. I wouldn't know, because systemd dev handling this didn't say a thing to me at all. Just closed the report after some seemingly not very related commits to what I saw in gdb.
I just dropped the pam module from config anyway, and I'm not willing to lose access to my cluster again just to try.
Not sure how distro fucked up in this case, when it's homed pam module going into infinite loop once in a while unpredictably blocking all processes that use pam naively, unless you mean that they fucked up by using systemd or pam at all.
Edit: lol, I read through systemd release notes and there's no mention of new huge systemd-homed pam module at all. But good I looked, they're moving pam config for systemd to /usr/lib/pam.d and systemd-home pam module is again enabled there. So I'll have to check again all my systems if it's really disabled after upgrade.
Anyway, I reported it and it presumably got fixed. I wouldn't know, because systemd dev handling this didn't say a thing to me at all. Just closed the report after some seemingly not very related commits to what I saw in gdb.
I just dropped the pam module from config anyway, and I'm not willing to lose access to my cluster again just to try.
Not sure how distro fucked up in this case, when it's homed pam module going into infinite loop once in a while unpredictably blocking all processes that use pam naively, unless you mean that they fucked up by using systemd or pam at all.
Edit: lol, I read through systemd release notes and there's no mention of new huge systemd-homed pam module at all. But good I looked, they're moving pam config for systemd to /usr/lib/pam.d and systemd-home pam module is again enabled there. So I'll have to check again all my systems if it's really disabled after upgrade.
The main problem with systemd IMO is that it’s written in a critically insecure language. If they could rewrite it in something safe but keep the same design, that would be ideal.
I’m not sure why are you downvoted — yeah the “rewrite everything in rust” hype is bad/annoying (and no, I don’t have any problem with rust), but systemd simply should not have been started in C. It replaced goddamn shell scripts so performance can’t really be a good argument pro, but a not too scripty managed language would have more than enough performance.
If it had been written in rust or go, we already would have hundreds of MB worth of systemd binaries on our disks, like some other popular packages.
People all the time complain about systemd bloat, but it's very likely that in those languages it would be far worse.
People all the time complain about systemd bloat, but it's very likely that in those languages it would be far worse.
I don’t believe people complain about systemd binaries’ sizes at all. They complain about bloat (in my opinion wrongly) in terms of systemd incorporating more and more features that often seems orthogonal to an init system. But often these are not part of systemd itself, but created only under the same project.
What language would have been appropriate in 2013?
Well, C++ had been around, but there were plenty of managed languages also. I will be hated for this, but perhaps even Java would have been appropriate if one sets the heap size appropriately, I believe the GCs were already quite good then; and it wouldn’t have to create too much garbage so I think it could get away with only a bit of memory. The only problem is the not too comfortable way to do C calls.
Unix sockets aren't in the standard java libraries either, which probably meant it was a non-starter.
Golang comes to mind. Gobs of C boilerplate could have been avoided. Docker is written in Go after all and was first released in 2013
Go was very immature at the time. Literally a brand new language.
Wait until you hear what the Darwin, Linux and Windows kernels are written in...
Ooh ooh I know I know! They were all written in..
...the 90s. So it's too late to change them. We should definitely be considering safer languages in 2021.
...the 90s. So it's too late to change them. We should definitely be considering safer languages in 2021.
None of the safe languages I know of that are capable of writing an OS in are "safe" in the verification sense. They require extra, external processes for doing such. See the SEL4 kernel.
I would rather say, the main problem with systemd is, that it tries to solve to many things in a way that:
1. makes very common usecases needlessly hard for beginners (e.g. run something on boot)
2. is sometimes the opposite of self explainatory
I use systemd and am OK with it and I aggree with the problems it is trying to solve. It is just a bit convoluted and intransparent even after years of using it.
1. makes very common usecases needlessly hard for beginners (e.g. run something on boot)
2. is sometimes the opposite of self explainatory
I use systemd and am OK with it and I aggree with the problems it is trying to solve. It is just a bit convoluted and intransparent even after years of using it.
I'd say running something on boot tends to be similar for beginners with or without systemd. The support for drop-in configs, however, makes it easy to revert screw-ups without much effort. In addition, the fact that one does not have to write any bash to change the way a service is started is a godsend. Ini-files aren't perfect, I get that, but from the point of view of a beginner they're both easier to grok and modify as compared to bash scripts.
I had to find a way to start something on boot recently and I found it trivial. I just did a google search for starting thing on boot systemd, copied a 5 line service file, enabled it and it worked.
And if you just want to start a gui, Gnome Tweaks has a page which is super simple.
And if you just want to start a gui, Gnome Tweaks has a page which is super simple.
> makes very common usecases needlessly hard for beginners (e.g. run something on boot)
Ah yes, systemd unit files are well regarded as far more complex than shell scripts that are 99% boilerplate. /s
Ah yes, systemd unit files are well regarded as far more complex than shell scripts that are 99% boilerplate. /s
systemd unit files are a massive step up from the old way - they properly handle dependant services, running as a user, can handle restarts for you etc. And as you say, no more annoying boilerplate!
Actually, unit files are possibly the only part of SystemD that I've knowingly interacted with - I know it does about a thousand things, but whatever those other things are, they are transparent to me as a user.
Actually, unit files are possibly the only part of SystemD that I've knowingly interacted with - I know it does about a thousand things, but whatever those other things are, they are transparent to me as a user.
I have had to help a few Unix newbies, and systemd is way easier to explain than RedHat init scripts with comment blocks for chkconfig and no useful errors if things are wrong.
The biggest problem I remember now is that when testing your init script it picks up environment variables from your root shell. So, because of that it works when it shouldn't, and realizing what's missing is a big debugging nightmare. The systemd init starts every unit in the same environment.
Another one that resulted in hours of fun fixing it is init scripts happily starting services even when various filesystems failed to mount. Like /var. Getting those particular database files back in sync after the / filesystem filled up was a nightmare.
Systemd has pretty good documentation and I think it is a feature that its users are pushed to read it, rather than hacking out a shell script that doesn't use half the already existing functions because the author never bothered to read RedHat or Debian docs.
The biggest problem I remember now is that when testing your init script it picks up environment variables from your root shell. So, because of that it works when it shouldn't, and realizing what's missing is a big debugging nightmare. The systemd init starts every unit in the same environment.
Another one that resulted in hours of fun fixing it is init scripts happily starting services even when various filesystems failed to mount. Like /var. Getting those particular database files back in sync after the / filesystem filled up was a nightmare.
Systemd has pretty good documentation and I think it is a feature that its users are pushed to read it, rather than hacking out a shell script that doesn't use half the already existing functions because the author never bothered to read RedHat or Debian docs.
I'm not a sysadmin, so I never really felt the hate against systemd. For my use cases it was always not exactly transparent; it was essentially invisible.