A Caddy Cert Expired Because Systemd-Resolved Was Selectively Broken(rant.mvh.dev)
rant.mvh.dev
A Caddy Cert Expired Because Systemd-Resolved Was Selectively Broken
https://rant.mvh.dev/a-caddy-cert-expired-because-systemd-resolved-was-selectively-broken/
35 コメント
There's this adage about consuming whatever newspaper reporting, everything seems fine and dandy, until one day they report on something in your field. Say, chemistry, and you're a chemist, and... it's all wrong. Even the basics.
My field is networking, roughly from Ethernet physicals to TCP/UDP.
systemd has no f*cking clue what they're doing on networking. You need to not use systemd-resolved, and not use systemd-networkd or systemd-timesyncd either.
I really hope they are service manager people and know how to write a service manager, and I have no counterindications on that. But don't let them touch your networking, aside from service-managing on that.
(My recommendation for DNS - on servers - would be to install unbound locally and use that. It's not great for clients since it doesn't deal well with frequently changing network connectivity, or rather, it does, but only if you restart it on network attachment change, which implies flushing all cached data.)
My field is networking, roughly from Ethernet physicals to TCP/UDP.
systemd has no f*cking clue what they're doing on networking. You need to not use systemd-resolved, and not use systemd-networkd or systemd-timesyncd either.
I really hope they are service manager people and know how to write a service manager, and I have no counterindications on that. But don't let them touch your networking, aside from service-managing on that.
(My recommendation for DNS - on servers - would be to install unbound locally and use that. It's not great for clients since it doesn't deal well with frequently changing network connectivity, or rather, it does, but only if you restart it on network attachment change, which implies flushing all cached data.)
> systemd has no f*cking clue what they're doing on networking. You need to not use systemd-resolved, and not use systemd-networkd or systemd-timesyncd either.
Can you give some more details, or do you have some examples of where systemd is bad here? Because I'm quite happy with systemd-resolved and systemd-networkd, but I admittedly know hardly anything about networking.
> [...] But don't let them touch your networking, aside from service-managing on that.
What do you recommend as an alternative then? systemd-resolved and systemd-timesyncd are easy to replace, since there is lots of good DNS/NTP software available, but systemd-networkd seems trickier: I don't like NetworkManager, I'd rather not write a bunch of ad-hoc shell scripts, and I'm not aware of any other alternatives.
Can you give some more details, or do you have some examples of where systemd is bad here? Because I'm quite happy with systemd-resolved and systemd-networkd, but I admittedly know hardly anything about networking.
> [...] But don't let them touch your networking, aside from service-managing on that.
What do you recommend as an alternative then? systemd-resolved and systemd-timesyncd are easy to replace, since there is lots of good DNS/NTP software available, but systemd-networkd seems trickier: I don't like NetworkManager, I'd rather not write a bunch of ad-hoc shell scripts, and I'm not aware of any other alternatives.
> Can you give some more details, or do you have some examples of where systemd is bad here?
I can't, because it's death by a thousand cuts. I remember systemd-timesyncd having security issues, but not what exactly. systemd-resolved was initially missing basic security best practices (source port randomization, if I remember correctly), despite their being well established and well known in the DNS community - AFAIK the systemd people just didn't know of them.
> systemd-networkd seems trickier
Indeed; I was primarily speaking about servers, and there my answer is: either ifupdown-ng or ifupdown2. (The latter if your networking needs are more complex.)
On client devices the choice is between bad and worse. Personally speaking, I consider NetworkManager bad and systemd-networkd worse. So I'm using NetworkManager on my laptop... it still occasionally breaks IPv6 for me.
I will admit: this is burned-in experience for me, learned over several years, and the original reasons have expired from my brain's cache. And I probably didn't give systemd-networkd much of a chance (I don't remember anything on that, for resolved and timesyncd I have at least vague bits), with that being more recent (at least for me) than systemd-resolved and systemd-timesyncd.
Is it possible it has gotten better? Definitely. But TFA indicates at least systemd-resolved hasn't.
I can't, because it's death by a thousand cuts. I remember systemd-timesyncd having security issues, but not what exactly. systemd-resolved was initially missing basic security best practices (source port randomization, if I remember correctly), despite their being well established and well known in the DNS community - AFAIK the systemd people just didn't know of them.
> systemd-networkd seems trickier
Indeed; I was primarily speaking about servers, and there my answer is: either ifupdown-ng or ifupdown2. (The latter if your networking needs are more complex.)
On client devices the choice is between bad and worse. Personally speaking, I consider NetworkManager bad and systemd-networkd worse. So I'm using NetworkManager on my laptop... it still occasionally breaks IPv6 for me.
I will admit: this is burned-in experience for me, learned over several years, and the original reasons have expired from my brain's cache. And I probably didn't give systemd-networkd much of a chance (I don't remember anything on that, for resolved and timesyncd I have at least vague bits), with that being more recent (at least for me) than systemd-resolved and systemd-timesyncd.
Is it possible it has gotten better? Definitely. But TFA indicates at least systemd-resolved hasn't.
> systemd-resolved was initially missing basic security best practices (source port randomization, if I remember correctly), despite their being well established and well known in the DNS community
https://lists.dns-oarc.net/pipermail/dns-operations/2016-Jun...
It was fixed in 2016. RFC5452 is 2009.
As the first paragraph states it's not a big problem for a local forwarder but all other bullet points are on the case.
https://lists.dns-oarc.net/pipermail/dns-operations/2016-Jun...
It was fixed in 2016. RFC5452 is 2009.
As the first paragraph states it's not a big problem for a local forwarder but all other bullet points are on the case.
A lot of knowledgeable people hate systemd, and I start suspecting that they have good reasons for that. However, it's extremely hard to get a good list of related substantiated facts about systemd in order to convince a larger community. If you encounter something specific, please write some kind of article about it.
So, it's 12 days later and we upgraded Debian stable on our production DFZ BGP routers.
Result: systemd-networkd in a restart loop continuously consuming 100% CPU. None of the systemd restart-limiting options are being applied. The only message in the log is "Could not enumerate links: Connection timed out".
It's not a strong data point, but it's a data point. I did say "death by a thousand cuts". There's enough of these to sum up.
To add insult to injury, I'm relatively sure systemd-networkd was disabled before the update, but I can't easily confirm that.
Result: systemd-networkd in a restart loop continuously consuming 100% CPU. None of the systemd restart-limiting options are being applied. The only message in the log is "Could not enumerate links: Connection timed out".
It's not a strong data point, but it's a data point. I did say "death by a thousand cuts". There's enough of these to sum up.
To add insult to injury, I'm relatively sure systemd-networkd was disabled before the update, but I can't easily confirm that.
You're right in that I should really be able to back my position up better. I don't know if/when a situation where I have to deal with the systemd network components will come up though :/
> in order to convince a larger community
Even if you would get such a list it would be dismissed anyway, because 'works on my machine'. Eg: see a sibling response to the OP.
Even if you would get such a list it would be dismissed anyway, because 'works on my machine'. Eg: see a sibling response to the OP.
If I ever have a networking problem in a personal Linux system, it gets solved by removing the weird networking thing that the distro decided to use and move to full systemd-resolved / networkd. In my experience, that works every time.
I'm not doubting commenter's expertise. It's entirely possible that resolved and networkd are utter horror inside but still work from at least this user's POV.
I'm not doubting commenter's expertise. It's entirely possible that resolved and networkd are utter horror inside but still work from at least this user's POV.
hot take "predictable network names" should have been a kernel flag - give us all our eth0 in peace. i shouldn't need to set a flag to get back a default feature.
just what i needed to do: configure ens328452356aflhjdslhfda to get my network going.
just what i needed to do: configure ens328452356aflhjdslhfda to get my network going.
ln -s /dev/null /etc/systemd/network/99-default.link
Or
net.ifnames=0
Take your pick...
net.ifnames=0 should be the default: if someone needs it to be the stupid ENS nonsense, they should be the ones forced to set it to 1 at the command line, not me.
> hot take "predictable network names" should have been a kernel flag - give us all our eth0 in peace. i shouldn't need to set a flag to get back a default feature.
No, because the feature being on by default does nothing but make your interface names longer, but being off can be the difference between a restarted server having or not having network access because of undocumented behaviour with it's firmware causing the network interfaces to come up in different orders.
No, because the feature being on by default does nothing but make your interface names longer, but being off can be the difference between a restarted server having or not having network access because of undocumented behaviour with it's firmware causing the network interfaces to come up in different orders.
Personally, I find the predictable network names feature infuriating 100% of the time. Changing the name of network interfaces made nothing better, and actively made things worse for systems with a single NIC (virtually none of the systems I use with a single nic use the same name). Network interface configurations could already be bound to MAC addresses rather than names, and that had been implemented even before the enpXXX style names came into being.
Persistent ethX names were far better to me, and we had those before enpXXX via udev. At least then when I logged into a random system with a single NIC it was called eth0. With 2 NICs they were eth0 and eth1. Simple to predict, no thinking required (unless the system already had bound those names to other MAC addresses).
Usability dies by a thousand cuts. Forcing needlessly complex behaviour onto people with simple use-cases is not an improvement.
Persistent ethX names were far better to me, and we had those before enpXXX via udev. At least then when I logged into a random system with a single NIC it was called eth0. With 2 NICs they were eth0 and eth1. Simple to predict, no thinking required (unless the system already had bound those names to other MAC addresses).
Usability dies by a thousand cuts. Forcing needlessly complex behaviour onto people with simple use-cases is not an improvement.
< Persistent ethX names were far better to me
They weren't persistent, that was the problem.
They weren't persistent, that was the problem.
eth0 is predictable: a given system will always have the same interface for the same physical hardware
enp0s31f6 is unpredictable
The whole operation is a lie!
enp0s31f6 is unpredictable
The whole operation is a lie!
[deleted]
> My recommendation for DNS - on servers - would be to install unbound locally and use that.
At least on Ubuntu 26.04, you can't easily bind Unbound on a WireGuard interface because of services dependency order, so the Unbound service errors during system boot because the WireGuard interface doesn't exist yet. And IIRC neither `ip-transparent` nor `interface-automatic` fixed it.
On Alpine Linux all this just works.
At least on Ubuntu 26.04, you can't easily bind Unbound on a WireGuard interface because of services dependency order, so the Unbound service errors during system boot because the WireGuard interface doesn't exist yet. And IIRC neither `ip-transparent` nor `interface-automatic` fixed it.
On Alpine Linux all this just works.
> My recommendation for DNS - on servers - would be to install unbound locally and use that
And now your developers are running around and cursing you because nothing works anymore.
Because Docker silently retargets the interna Docker resolver to 8.8.8.8 if it sees 127.0.0.1 as a resolver address on the host.
Because people who wrote Docker have no fucking clue how the system works.
NB: see https://news.ycombinator.com/item?id=47441785 to solve the Docker issue with a local resolver.
> It's not great for clients since it doesn't deal well with frequently changing network connectivity
This is is something Linux specific I guess, I run Unbound locally on my Windows laptops for years and never had a problem which would require the Unbound restart.
> which implies flushing all cached data
It doesn't really matters in 2026. Just look in your cache and note the default TTLs for like 90% of records.
And now your developers are running around and cursing you because nothing works anymore.
Because Docker silently retargets the interna Docker resolver to 8.8.8.8 if it sees 127.0.0.1 as a resolver address on the host.
Because people who wrote Docker have no fucking clue how the system works.
NB: see https://news.ycombinator.com/item?id=47441785 to solve the Docker issue with a local resolver.
> It's not great for clients since it doesn't deal well with frequently changing network connectivity
This is is something Linux specific I guess, I run Unbound locally on my Windows laptops for years and never had a problem which would require the Unbound restart.
> which implies flushing all cached data
It doesn't really matters in 2026. Just look in your cache and note the default TTLs for like 90% of records.
Running on 127.0.0.53 isn't a systemd-resolved exclusive magic trick, you can just configure unbound to do the same thing.
Not 127/8
?
You need to configure your resolver to anything but 127.0.0.0/8 to not to trigger Docker automatic "help".
Most of the time local resolver is bound to 127.0.0.1 just because it shouldn't be available from the network and because this is how it's always done.
The problem is what Docker actually silently break things because it's devs are too good to bother, the same thing you note about systemd devs.
Hell, it was easier to just provide internal Docker resolver on a some network Docker uses internally by default for the NAT shenanigans - but Docker devs 'knew better' and instead made it to point to a 3rd-party absolutely not related service which brokes local resolution on the containers.
Most of the time local resolver is bound to 127.0.0.1 just because it shouldn't be available from the network and because this is how it's always done.
The problem is what Docker actually silently break things because it's devs are too good to bother, the same thing you note about systemd devs.
Hell, it was easier to just provide internal Docker resolver on a some network Docker uses internally by default for the NAT shenanigans - but Docker devs 'knew better' and instead made it to point to a 3rd-party absolutely not related service which brokes local resolution on the containers.
Want to know a really cursed dirty little magic trick?
(as root:)
And to be clear: I said "cursed dirty" and I mean "cursed dirty".
(as root:)
ip route add table local local 8.8.8.8/32 dev lo
echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind
cat <<EOF >/etc/unbound/unbound.conf.d/steal-8.8.8.8.conf
server:
interface: 8.8.8.8
EOF
service unbound restart
And to be clear: I said "cursed dirty" and I mean "cursed dirty".
Yikes. Container resolving is still broken though.
Not if your containers are L3 routed through the host ;)
(Or did you mean resolving the names of the containers? That'd be handled through unbound-control local_data)
(Or did you mean resolving the names of the containers? That'd be handled through unbound-control local_data)
Some docker issues archaeology says that there was tension between docker team and redhat / systemd
Yes?
RH wanted to control the stack and Docker was the thing what sidelined RH to a mere distro provider. And don't forget about RH dips into OpenStack at that time.
RH wanted to control the stack and Docker was the thing what sidelined RH to a mere distro provider. And don't forget about RH dips into OpenStack at that time.
> There's this adage about consuming whatever newspaper reporting, everything seems fine and dandy, until one day they report on something in your field. Say, chemistry, and you're a chemist, and... it's all wrong. Even the basics.
Gell-Mann amnesia effect: https://en.wiktionary.org/wiki/Gell-Mann_Amnesia_effect
> systemd has no fcking clue what they're doing on networking. You need to not use systemd-resolved, and not use systemd-networkd or systemd-timesyncd either.
I'm not sure how to reconcile that with enjoying each of those:
systemd-resolved is the default on many distributions, and does DNS-over-TLS stub resolution for me. What do you suggest as an alternative that handles changing network connectivity, and DoT?
* systemd-networkd is very well documented/organised. I find it pretty useful on servers.
* systemd-timesyncd doesn't seem to cause issues for me. It being an SNTP client (rather than NTP) seems appropriate for distributions. I would like to move to NTS or Roughtime at some point.
Gell-Mann amnesia effect: https://en.wiktionary.org/wiki/Gell-Mann_Amnesia_effect
> systemd has no fcking clue what they're doing on networking. You need to not use systemd-resolved, and not use systemd-networkd or systemd-timesyncd either.
I'm not sure how to reconcile that with enjoying each of those:
systemd-resolved is the default on many distributions, and does DNS-over-TLS stub resolution for me. What do you suggest as an alternative that handles changing network connectivity, and DoT?
* systemd-networkd is very well documented/organised. I find it pretty useful on servers.
* systemd-timesyncd doesn't seem to cause issues for me. It being an SNTP client (rather than NTP) seems appropriate for distributions. I would like to move to NTS or Roughtime at some point.
There was interesting commentary on https://lobste.rs/s/z0ozbb/caddy_cert_expired_because_system...
Was there? I see 14 comments, of which only 3 have any value at all which are: fanf finding an irrelevant bug, oliverpool explaining the use of LetsEncrypt staging, and hexadecimal suggesting setting up expiration alerts.
The rest is:
The rest is:
- People bitching about systemd and replies to their bitching totaling 7 posts (literally half the comments)
- conversational detritus like "well written blog post" and the whiteboard comment
might fix it.
[0]: https://github.com/systemd/systemd/issues/34897#issuecomment...