Letsencrypt, the Good, the Bad and the Ugly(jacquesmattheij.com)
jacquesmattheij.com
Letsencrypt, the Good, the Bad and the Ugly
https://jacquesmattheij.com/letsencrypt-good-bad-ugly/
227 comments
Historically the Certbot software was named "letsencrypt" which certainly didn't make this easier to understand.
The not-for-profit is named ISRG (Internet Security Research Group) and so that's the entity trusted to actually run the Certificate Authority. Let's Encrypt is in some sense branding for this their main (only?) activity the same way you can still buy certificates with Thawte branding even though Thawte hasn't really existed for many years.
ACME is an IETF standardized protocol and so one of the things which has changed at Let's Encrypt is they gradually migrated from the ACME prototype they'd built and shipped to something that's (more or less) compliant to the IETF standard.
This is akin to how today Google's own web sites can talk Google's QUIC protocol (sometimes referred to as "gQUIC") but Google intends to rip that out once the IETF QUIC standard is published and have their sites just speak the standard QUIC instead (there may be a brief overlap where they speak both but it's likely to be very short because maintaining two protocols is far from free)
The not-for-profit is named ISRG (Internet Security Research Group) and so that's the entity trusted to actually run the Certificate Authority. Let's Encrypt is in some sense branding for this their main (only?) activity the same way you can still buy certificates with Thawte branding even though Thawte hasn't really existed for many years.
ACME is an IETF standardized protocol and so one of the things which has changed at Let's Encrypt is they gradually migrated from the ACME prototype they'd built and shipped to something that's (more or less) compliant to the IETF standard.
This is akin to how today Google's own web sites can talk Google's QUIC protocol (sometimes referred to as "gQUIC") but Google intends to rip that out once the IETF QUIC standard is published and have their sites just speak the standard QUIC instead (there may be a brief overlap where they speak both but it's likely to be very short because maintaining two protocols is far from free)
> Historically the Certbot software was named "letsencrypt" which certainly didn't make this easier to understand.
And to this day strongly promoted (recommended first option) right from the letsencrypt.org starter page:
https://letsencrypt.org/getting-started/
And to this day strongly promoted (recommended first option) right from the letsencrypt.org starter page:
https://letsencrypt.org/getting-started/
That doesn't seem to be the case:
> Let’s Encrypt is a CA.
> We recommend that most people with shell access use the Certbot ACME client.
> Let’s Encrypt is a CA.
> We recommend that most people with shell access use the Certbot ACME client.
Huge stumbling block and source of confusion for me. During the steep learning curve setting it up, I'd often search online for help/tutorials and they'd all reference "certbot" which I couldn't find anywhere on my system. As a newbie, I frequently said "WTF is this certbot thing, I'm using letsencrypt!" Wasn't clear at all. At some point during my system updates, /usr/bin/letsencrypt became a symbolic link to /usr/bin/certbot and it became obvious.
Sounds like you were running a distro that didn't keep its packages updated. By the time everyone was calling it 'certbot', well, it was called certbot.
If you run something that issues 100 million or more certificates per year then backwards compatibility is not something that you toss out just because you can. Forcing that many web properties to upgrade their software (regardless of which party produced what) is discounting the combined effort that will take on the part of the users/sysadmins of those systems for something that could have just as easily been avoided.
You deprecate interfaces like these but you don't just shut them down, especially not when they are still seeing major use.
Just imagine that tomorrow IPV4 would be shut down because we've all had enough time to switch by now.
You deprecate interfaces like these but you don't just shut them down, especially not when they are still seeing major use.
Just imagine that tomorrow IPV4 would be shut down because we've all had enough time to switch by now.
The reason the old interface was deprecated was that a security hole was found in the protocol.
That is one of the few cases where it is reasonable to break backward compatability in this manner.
Especially when dealing with certificates, where the security is one of the top reasons to want to go there.
Especially when dealing with certificates, where the security is one of the top reasons to want to go there.
> that a security hole was found in the protocol
Is there any supporting evidence for that because the only thing I have been able to find so far is that it was simply superseded by a newer version, mostly to support wildcard certs. What holes there were in V1 were closed within a day or two at most.
Is there any supporting evidence for that because the only thing I have been able to find so far is that it was simply superseded by a newer version, mostly to support wildcard certs. What holes there were in V1 were closed within a day or two at most.
The article says "then the challenge protocol was changed" so that's why people are talking about the protocol.
The only challenge which changed was tls-sni-01 which was removed and eventually replaced with tls-alpn-01
The tls-sni-01 challenge is safe unless there are bulk hosting sites whose web server for some crazy reason accepts SNI for names that are nonsensical, and then serves up answers chosen by an attacker who is also one of the customers on that server instead of from a victim on the same server.
Unfortunately somebody actually did ship software which is crazy in that specific way, and it's named Apache HTTPD server. You might have heard of it. So that's a problem.
So, Let's Encrypt deprecated this challenge and you can no longer use it. They did tell everybody affected, by email to the address they provided for contact. Since they are not psychic they don't have a way to reach out to people who felt they didn't need to be contacted.
I suspect given you mention wildcards you're thinking of ACMEv2 which isn't a challenge protocol. But again there were plenty of email notifications about the ACMEv2 upgrade, and you've in fact encountered exactly the anticipated scenario, you decided to build out a new thing using the old service and it told you not to do that. Your old things are still working, for almost another year, after already two years notice that this was going away, it's just that new things can't be launched against this already deprecated service.
The only challenge which changed was tls-sni-01 which was removed and eventually replaced with tls-alpn-01
The tls-sni-01 challenge is safe unless there are bulk hosting sites whose web server for some crazy reason accepts SNI for names that are nonsensical, and then serves up answers chosen by an attacker who is also one of the customers on that server instead of from a victim on the same server.
Unfortunately somebody actually did ship software which is crazy in that specific way, and it's named Apache HTTPD server. You might have heard of it. So that's a problem.
So, Let's Encrypt deprecated this challenge and you can no longer use it. They did tell everybody affected, by email to the address they provided for contact. Since they are not psychic they don't have a way to reach out to people who felt they didn't need to be contacted.
I suspect given you mention wildcards you're thinking of ACMEv2 which isn't a challenge protocol. But again there were plenty of email notifications about the ACMEv2 upgrade, and you've in fact encountered exactly the anticipated scenario, you decided to build out a new thing using the old service and it told you not to do that. Your old things are still working, for almost another year, after already two years notice that this was going away, it's just that new things can't be launched against this already deprecated service.
You know this, but for the benefit of the thread: to say "tls-sni-01 is safe unless there are bulk hosting sites that break it" is to say that tls-sni-01 is unsafe. The "crazy" sites you're referring to included AWS and Heroku.
This all happened 2 years ago, so it's a bit odd to see it litigated today.
This all happened 2 years ago, so it's a bit odd to see it litigated today.
We briefly describe this history on page 6 of
https://jhalderm.com/pub/papers/letsencrypt-ccs19.pdf
in case anyone is more interested (there are also references there for further details). Twice, methods that seemed plausible for proving control over domain names turned out to make assumptions that were potentially violated by shared hosting environments.
Jacques, I'm really sorry for the hassle that these changes caused you.
https://jhalderm.com/pub/papers/letsencrypt-ccs19.pdf
in case anyone is more interested (there are also references there for further details). Twice, methods that seemed plausible for proving control over domain names turned out to make assumptions that were potentially violated by shared hosting environments.
Jacques, I'm really sorry for the hassle that these changes caused you.
Thanks for the link Seth. I wasn't aware this existed and it's sometimes nice to have something specific to cite as well as convenient that it's all in one place like this.
Edited to add: Wow the Sankey diagram (showing changes in which CA if any is used by a site) is something I hadn't seen anywhere else and is especially useful. Thanks again.
Edited to add: Wow the Sankey diagram (showing changes in which CA if any is used by a site) is something I hadn't seen anywhere else and is especially useful. Thanks again.
Heroku and (so far as I can tell) Cloudfront independently re-invented this stupidity. But if it was "just" say Heroku and Cloudfront you can imagine plausibly notifying those two providers to fix their broken infrastructure and then you're good.
Apache makes it unsalvageable by sheer numbers the same way it had already for HTTPS in http-01, so that's why I focused on Apache.
It's entirely possible for some fool to ship an exciting new cloud service that lets people bind to arbitrary ALPN values on a shared service and thereby re-introduce this problem for tls-alpn-01 - but unlike with tls-sni-01 that's not a bug common to hundreds of small bulk hosts using out of box Apache so I assume we'd tell the exciting upstart to knock it off and warn their customers what they're doing is inherently unsafe, rather than requiring Let's Encrypt to stop offering tls-alpn-01.
In fact we're already on the other side of this for the ordinary version of http-01 for a different reason. Apache really does potentially let an attacker who controls aaa-aardvark.example at some bulk host perform http-01 challenges for www.some-custom-site.example that has created A records pointing to the bulk host but hasn't currently actually got them serving www.some-custom-site.example maybe due to a typo or unpaid bill.
But most bulk hosts have specifically configured Apache to show a default "Did you pay? / Have you configured your hosting properly?" type site which is harmless in this case, and for the few that haven't users can understand that um, if they visit www.some-custom-site.example in their browser they get to the attacker's site, so like yeah, that's where the problem is, nothing new with http-01
Apache makes it unsalvageable by sheer numbers the same way it had already for HTTPS in http-01, so that's why I focused on Apache.
It's entirely possible for some fool to ship an exciting new cloud service that lets people bind to arbitrary ALPN values on a shared service and thereby re-introduce this problem for tls-alpn-01 - but unlike with tls-sni-01 that's not a bug common to hundreds of small bulk hosts using out of box Apache so I assume we'd tell the exciting upstart to knock it off and warn their customers what they're doing is inherently unsafe, rather than requiring Let's Encrypt to stop offering tls-alpn-01.
In fact we're already on the other side of this for the ordinary version of http-01 for a different reason. Apache really does potentially let an attacker who controls aaa-aardvark.example at some bulk host perform http-01 challenges for www.some-custom-site.example that has created A records pointing to the bulk host but hasn't currently actually got them serving www.some-custom-site.example maybe due to a typo or unpaid bill.
But most bulk hosts have specifically configured Apache to show a default "Did you pay? / Have you configured your hosting properly?" type site which is harmless in this case, and for the few that haven't users can understand that um, if they visit www.some-custom-site.example in their browser they get to the attacker's site, so like yeah, that's where the problem is, nothing new with http-01
I did provide an email address, never got any mail (I did actually check that).
> it's just that new things can't be launched against this already deprecated service.
Yes, I noticed. So, I now have the entirely unforced option of re-imaging a machine that is working just fine besides this little detail, which is in fact just one very small thing of a whole pile of much bigger things that run on that particular box. Not to mention migrating twentynine years of email to a new mail server.
I'm sure there is a lesson in there somewhere, but I'm not sure I'm overly receptive today, I had a lot of other stuff on my agenda.
> it's just that new things can't be launched against this already deprecated service.
Yes, I noticed. So, I now have the entirely unforced option of re-imaging a machine that is working just fine besides this little detail, which is in fact just one very small thing of a whole pile of much bigger things that run on that particular box. Not to mention migrating twentynine years of email to a new mail server.
I'm sure there is a lesson in there somewhere, but I'm not sure I'm overly receptive today, I had a lot of other stuff on my agenda.
If you let a server lag in OS version, at some point in time you're going to hit this kind of problem. If not with Let's Encrypt, then with some other dependency. I know, I've been in the exact same spot. I just don't blame the dependency, and included server OS updates as part of a yearly maintenance cycle.
I find that really ridiculous. Not you, but the fact that an OS needs to be upgraded because of some application level stuff that has to do with a protocol that is being run on some other server.
That's the kind of dependency snowball that we should work hard to avoid, not accept as some kind of new normality.
Servers should be able to live for years without re-imaging.
That's the kind of dependency snowball that we should work hard to avoid, not accept as some kind of new normality.
Servers should be able to live for years without re-imaging.
Is there a reason you can't just upgrade that one component on the server, why do you have to re-image it from scratch?
If you have external dependencies they are going to move around from time to time throughout their lifetimes, especially if they are beta. LetsEncrypt may not have signaled beta with v1, but I've been a cert-manager user for years in pre-1.0 and I've known that meant I might need to come up for air and read the docs for a specific upgrade instruction from one pre-1.0 minor version to another at any time.
Now cert-manager is 1.0+ and my expectations can change. It should remain backwards compatible until the next major version (hopefully for a while! And they will provide a migration path when that comes, with clear instructions and a fairly long sunset, godwilling)
But cert-manager depends on letsencrypt, and I depend on cert-manager, all of which depends on a protocol called acme, and this is the arrangement. We made this deal because it was going to turn out less complicated than managing the certificates by hand, and they made that deal because it was going to turn out better than rolling their own protocol from absolutely scratch, similarly. Eyes on the prize.
If you didn't want LetsEncrypt as a dependency there are other ways to connect cert-manager or another tool like it, including other acme providers... they all depend on the acme protocol, (or there might be some other protocol that you can use, with its own characteristics of change or stability, or roll your own) at some point you have to roll the dice and bet on something.
Occasionally these things happen. You suggest that servers should be able to go for years, (but they have allowed years for this transition! What more can be expected, realistically?)
If you have external dependencies they are going to move around from time to time throughout their lifetimes, especially if they are beta. LetsEncrypt may not have signaled beta with v1, but I've been a cert-manager user for years in pre-1.0 and I've known that meant I might need to come up for air and read the docs for a specific upgrade instruction from one pre-1.0 minor version to another at any time.
Now cert-manager is 1.0+ and my expectations can change. It should remain backwards compatible until the next major version (hopefully for a while! And they will provide a migration path when that comes, with clear instructions and a fairly long sunset, godwilling)
But cert-manager depends on letsencrypt, and I depend on cert-manager, all of which depends on a protocol called acme, and this is the arrangement. We made this deal because it was going to turn out less complicated than managing the certificates by hand, and they made that deal because it was going to turn out better than rolling their own protocol from absolutely scratch, similarly. Eyes on the prize.
If you didn't want LetsEncrypt as a dependency there are other ways to connect cert-manager or another tool like it, including other acme providers... they all depend on the acme protocol, (or there might be some other protocol that you can use, with its own characteristics of change or stability, or roll your own) at some point you have to roll the dice and bet on something.
Occasionally these things happen. You suggest that servers should be able to go for years, (but they have allowed years for this transition! What more can be expected, realistically?)
> Is there a reason you can't just upgrade that one component on the server, why do you have to re-image it from scratch?
Yes, I did this now and I have it working. But it leaves things in a messed up state and I don't like that so I will go back to this in a short while and fix it properly.
What I still wonder about is why their warning email never reached me, that I really need to figure out because then at least I would have dealt with this under a lot less time pressure.
> If you didn't want LetsEncrypt as a dependency there are other ways to connect cert-manager or another tool like it, including other acme providers...
There are some very good suggestions in this thread, I will probably adopt one of them.
> You suggest that servers should be able to go for years, (but they have allowed years for this!)
And somehow I missed that memo. Even so, I am still not convinced of the necessity, it is possible that it exists but I have yet to see a valid reason for shutting down the old protocol for new registrations like this. There also seems to be some confusion with people saying it should have worked for the same account, which I can prove did not work.
Yes, I did this now and I have it working. But it leaves things in a messed up state and I don't like that so I will go back to this in a short while and fix it properly.
What I still wonder about is why their warning email never reached me, that I really need to figure out because then at least I would have dealt with this under a lot less time pressure.
> If you didn't want LetsEncrypt as a dependency there are other ways to connect cert-manager or another tool like it, including other acme providers...
There are some very good suggestions in this thread, I will probably adopt one of them.
> You suggest that servers should be able to go for years, (but they have allowed years for this!)
And somehow I missed that memo. Even so, I am still not convinced of the necessity, it is possible that it exists but I have yet to see a valid reason for shutting down the old protocol for new registrations like this. There also seems to be some confusion with people saying it should have worked for the same account, which I can prove did not work.
> But it leaves things in a messed up state and I don't like that so I will go back to this in a short while and fix it properly.
You say this with confidence, I wish my own situation provided me with the confidence to say this and mean it. We do not have reproducible systems and depend in many ways wholly on backup images of live production systems. Someone is going to say this makes my life simpler than yours by some twisted math, but I have a doubt about that myself.
We are still talking about migrating from Amazon Linux v1 to Amazon Linux v2, and with a recent announcement from AWS, the pressure is off! We'll be able to continue talking about this transition for a good long time to come. Again, mixed blessing, is it better to have an operating system that can crawl along on life support? For those that can't upgrade, sure, it is better to get security maintenance than to have zombie servers which are not upgradeable, but who is to say what opportunity costs will arise because we are not on a formally supported leading-edge version of the platform.
You say this with confidence, I wish my own situation provided me with the confidence to say this and mean it. We do not have reproducible systems and depend in many ways wholly on backup images of live production systems. Someone is going to say this makes my life simpler than yours by some twisted math, but I have a doubt about that myself.
We are still talking about migrating from Amazon Linux v1 to Amazon Linux v2, and with a recent announcement from AWS, the pressure is off! We'll be able to continue talking about this transition for a good long time to come. Again, mixed blessing, is it better to have an operating system that can crawl along on life support? For those that can't upgrade, sure, it is better to get security maintenance than to have zombie servers which are not upgradeable, but who is to say what opportunity costs will arise because we are not on a formally supported leading-edge version of the platform.
Agreed, reproducible systems are an absolute must and it is a shame that we are still not even close to having a solid foundation under all this mission critical stuff we build.
It feels like we are building these huge castles on quicksand.
At the same time I think the whole 'treat your servers like cattle, not like pets' is exactly because we don't know how to do this properly. It is the cloud equivalent to hitting ctrl-alt-delete to solve issues.
It feels like we are building these huge castles on quicksand.
At the same time I think the whole 'treat your servers like cattle, not like pets' is exactly because we don't know how to do this properly. It is the cloud equivalent to hitting ctrl-alt-delete to solve issues.
I know you solved your issue; for others in the same boat, look into acme.sh. it's a shell only implementation, no python, no loads of dependencies. I used that to keep let's encrypt running on an ancient server (firewalled) that I cannot upgrade for reasons.
I decided to go with acme.sh instead of certbot on some servers because I am hoping that upgrading acme.sh will cause fewer headaches. But who knows...
Why TLS-SNI-01 was disabled:
https://community.letsencrypt.org/t/2018-01-09-issue-with-tl...
Explanation that renewals will be disallowed after 1 year deprecation period: https://community.letsencrypt.org/t/march-13-2019-end-of-lif...
And as you seem to be talking about ACMEv1/v2 instead of TLS-SNI-01 (which I originally thought); it will be supported as long as June 2021 in some cases: https://community.letsencrypt.org/t/end-of-life-plan-for-acm...
ACMEv2 was introduced, because it is much closer to the actual spec. Enforcing this ensures that there are actually ACME implementations out there, instead of proprietary "Let's Encypt ACME" implementations. https://tools.ietf.org/html/rfc8555 https://github.com/letsencrypt/boulder/blob/master/docs/acme...
To me this seems like a sensible compromise between backwards compatibility and their mission for standardized automated renewals.
Explanation that renewals will be disallowed after 1 year deprecation period: https://community.letsencrypt.org/t/march-13-2019-end-of-lif...
And as you seem to be talking about ACMEv1/v2 instead of TLS-SNI-01 (which I originally thought); it will be supported as long as June 2021 in some cases: https://community.letsencrypt.org/t/end-of-life-plan-for-acm...
ACMEv2 was introduced, because it is much closer to the actual spec. Enforcing this ensures that there are actually ACME implementations out there, instead of proprietary "Let's Encypt ACME" implementations. https://tools.ietf.org/html/rfc8555 https://github.com/letsencrypt/boulder/blob/master/docs/acme...
To me this seems like a sensible compromise between backwards compatibility and their mission for standardized automated renewals.
Yes, but that particular hole was fixed, wasn't it?
You can't "fix" the tls-sni-01 hole except by going back in a time machine to when Apache implements SNI and spraying all the involved developers with water. "No, bad developer, no biscuit. Do what the protocol specification actually says not whatever half-arsed nonsense you thought would work".
If there were like six web servers in the whole world that got this wrong, we could say "Fix those servers, fools" and sleep soundly knowing that those six servers are all that's affected. But Apache makes the scope too big to do that reasonably. It's a judgement call, but in this case the call was very easy.
If there were like six web servers in the whole world that got this wrong, we could say "Fix those servers, fools" and sleep soundly knowing that those six servers are all that's affected. But Apache makes the scope too big to do that reasonably. It's a judgement call, but in this case the call was very easy.
I don't see what that has to do with me because there is no Apache on that server (just Nginx).
But Let's Encrypt is part of the Web PKI, and the Web PKI is for all names on the public Internet, not just any operated by Jacques Mattheij. You sought certificates from the Web PKI, probably because you wanted somebody else other than Jacques Mattheij to trust them.
A large fraction of public Internet HTTPS servers run Apache, which means tls-sni-01 is unsafe for a non-trivial fraction of names, which means we need to tell Certificate Authorities not to use this method or those like it. Specifically 3.2.2.4.10. TLS Using a Random Number has to be approached differently if it's to be attempted. The tls-alpn-01 challenge implements 3.2.2.4.10 using ALPN instead of SNI and appears to be safe in practice.
A large fraction of public Internet HTTPS servers run Apache, which means tls-sni-01 is unsafe for a non-trivial fraction of names, which means we need to tell Certificate Authorities not to use this method or those like it. Specifically 3.2.2.4.10. TLS Using a Random Number has to be approached differently if it's to be attempted. The tls-alpn-01 challenge implements 3.2.2.4.10 using ALPN instead of SNI and appears to be safe in practice.
There was this joke when I was a fledgling programmer 35 years ago: If engineers would build bridges the way programmers build software the first woodpecker to come along would destroy civilization as we know it.
I think your comment is a nice illustration of that.
To me if a piece of software has a problem then it is that piece of software that should be fixed, not to push the burden onto everybody else as well. That's just so wrong.
But that does not mean I don't follow your reasoning and understand why this decision was made, still, the amount of waste here is incredible.
I think your comment is a nice illustration of that.
To me if a piece of software has a problem then it is that piece of software that should be fixed, not to push the burden onto everybody else as well. That's just so wrong.
But that does not mean I don't follow your reasoning and understand why this decision was made, still, the amount of waste here is incredible.
My interest in cryptography and the Web PKI is part of my interest in how to improve things (and the intersection with my interest in the Network), so that means as well as reading about stuff like Bleichenbacher I am interested in bridges.
Contrary to what your joke suggests, bridges do fall down because engineers don't always do a great job or learn from previous mistakes, they are after all only human.
England (where I live) has many railway bridges constructed by Victorian engineers using cast iron. The Victorians were not necessarily careful to properly document everything about these structures. And so a modern engineering team, responsible for safety of hundreds of these structures has to make some assumptions or else it would need a tremendous amount of investment to either replace every bridge or take it to pieces to figure out how it was built.
One obvious assumption is, if you can get at the two cast iron beams on the outer edges of a bridge that's clearly supported on four beams, the two inner beams you can't reach without demolishing the bridge are presumably identical.
So on particular bridge, over a road, the routine inspection team climbed into the guts of the bridge, they measured the accessible outer beams and concluded these were of a sufficient thickness that, allowing for the inevitable corrosion by the elements and the permitted loading of trains, the bridge should be "good" for another 20-25 years before needing replacement.
The bridge fell down before the next routine inspection. Fortunately nobody was killed, but the collapse exposed that Victorian engineers as well as not documenting their design work saw an opportunity from the fact that the inner beams aren't accessible. They'd used cheaper, thinner metal for those beams and really the only problem with doing that is that bridge will fall down sooner than otherwise expected...
Contrary to what your joke suggests, bridges do fall down because engineers don't always do a great job or learn from previous mistakes, they are after all only human.
England (where I live) has many railway bridges constructed by Victorian engineers using cast iron. The Victorians were not necessarily careful to properly document everything about these structures. And so a modern engineering team, responsible for safety of hundreds of these structures has to make some assumptions or else it would need a tremendous amount of investment to either replace every bridge or take it to pieces to figure out how it was built.
One obvious assumption is, if you can get at the two cast iron beams on the outer edges of a bridge that's clearly supported on four beams, the two inner beams you can't reach without demolishing the bridge are presumably identical.
So on particular bridge, over a road, the routine inspection team climbed into the guts of the bridge, they measured the accessible outer beams and concluded these were of a sufficient thickness that, allowing for the inevitable corrosion by the elements and the permitted loading of trains, the bridge should be "good" for another 20-25 years before needing replacement.
The bridge fell down before the next routine inspection. Fortunately nobody was killed, but the collapse exposed that Victorian engineers as well as not documenting their design work saw an opportunity from the fact that the inner beams aren't accessible. They'd used cheaper, thinner metal for those beams and really the only problem with doing that is that bridge will fall down sooner than otherwise expected...
This wasn't Let's Encrypt's choice. The Web is secure due to a series of rules agreed upon between the browsers and the CAs. If vulnerabilities are found, there are deadlines for fixing them; in fact, the deadline for revoking misissued certificates is 24 hours, and Let's Encrypt couldn't prove existing certificates weren't misissued, but they were able to get away without revocations, which is a huge benefit for their subscribers.
The point of these rules are to keep the web safe. The choice here is between inconveniencing Let's Encrypt users (forcing some of them to upgrade or switch validation methods), but keeping the web safe, or making the web unsafe, period, forever (because there is no way to force users of broken web servers and web hosts to upgrade to fix the approaches to certificate management that caused this problem). The only reasonable choice was the first.
I had to change all my servers from TLS-SNI-01 to another mechanism, and I absolutely do not blame Let's Encrypt for this. They did the right thing.
The point of these rules are to keep the web safe. The choice here is between inconveniencing Let's Encrypt users (forcing some of them to upgrade or switch validation methods), but keeping the web safe, or making the web unsafe, period, forever (because there is no way to force users of broken web servers and web hosts to upgrade to fix the approaches to certificate management that caused this problem). The only reasonable choice was the first.
I had to change all my servers from TLS-SNI-01 to another mechanism, and I absolutely do not blame Let's Encrypt for this. They did the right thing.
[deleted]
>Just imagine that tomorrow IPV4 would be shut down because we've all had enough time to switch by now.
Honestly? I would absolutely love to watch that shitshow.
Honestly? I would absolutely love to watch that shitshow.
Can you please wait until I'm past my 'best before' date when you pull that particular plug?
The anticipated order of events goes something like this:
Firstly the islands of IPv6 grow until they begin to dwarf the supposed generally interoperable ocean of IPv4. Big home ISPs, major CDNs, bulk hosts, AWS, and so on.
Somewhere around this time you'd start to see events reported where "the Internet" was down for lots of people but it was the IPv4 Internet, which they are increasingly not using so they didn't actually notice. "Your Internet was down" "No it wasn't, I was on Facebook all afternoon" "Right yeah, but other than Facebook" "I watched a movie on Netflix" "OK, other than Facebook and Netflix" "I got a mail from Jeremy on GMail" "OK, other than Facebook and Netflix and GMail" "Not much of an Internet". Happy Eyeballs, the algorithm that allowed IPv6 to be deployed in dual stack environment successfully, now allows IPv4 to ramp down imperceptibly.
Now, with the "ocean" so small, increasingly medium sized operators ignore it entirely, opting just to maintain translators at the edge of the IPv4 Internet, maybe your ISP does this, and you can't get "real" IPv4 addresses, although many of you already don't so this wouldn't be a change.
The last major steps taken by "the Internet" look like this:
The tier one providers who by that point are also more or less the global telecommunications companies, begin to deprecate IPv4 service, seeing it as a niche product that can better be serviced by specialists in your locale. Increasingly the only practical route from one IPv4 address to another IPv4 address is via two translators and IPv6.
The RIRs discontinue management of the namespace/ numberspace for IPv4 and so the allocation of IPv4 addresses ceases to be globally co-ordinated. The IPv4 Internet no longer formally exists, just many islands of legacy IPv4 in an IPv6 ocean which happen to have mostly discontiguous addressing.
Firstly the islands of IPv6 grow until they begin to dwarf the supposed generally interoperable ocean of IPv4. Big home ISPs, major CDNs, bulk hosts, AWS, and so on.
Somewhere around this time you'd start to see events reported where "the Internet" was down for lots of people but it was the IPv4 Internet, which they are increasingly not using so they didn't actually notice. "Your Internet was down" "No it wasn't, I was on Facebook all afternoon" "Right yeah, but other than Facebook" "I watched a movie on Netflix" "OK, other than Facebook and Netflix" "I got a mail from Jeremy on GMail" "OK, other than Facebook and Netflix and GMail" "Not much of an Internet". Happy Eyeballs, the algorithm that allowed IPv6 to be deployed in dual stack environment successfully, now allows IPv4 to ramp down imperceptibly.
Now, with the "ocean" so small, increasingly medium sized operators ignore it entirely, opting just to maintain translators at the edge of the IPv4 Internet, maybe your ISP does this, and you can't get "real" IPv4 addresses, although many of you already don't so this wouldn't be a change.
The last major steps taken by "the Internet" look like this:
The tier one providers who by that point are also more or less the global telecommunications companies, begin to deprecate IPv4 service, seeing it as a niche product that can better be serviced by specialists in your locale. Increasingly the only practical route from one IPv4 address to another IPv4 address is via two translators and IPv6.
The RIRs discontinue management of the namespace/ numberspace for IPv4 and so the allocation of IPv4 addresses ceases to be globally co-ordinated. The IPv4 Internet no longer formally exists, just many islands of legacy IPv4 in an IPv6 ocean which happen to have mostly discontiguous addressing.
Why should the author care? The point is the automated process isnt so automated.
Incidentally, my imaps certificate isnt renewing automatically any more. Cant bring myself to debug it. Which just proves the TFAs point.
Incidentally, my imaps certificate isnt renewing automatically any more. Cant bring myself to debug it. Which just proves the TFAs point.
the author should care because he's writing an article about it, and taking the time to make sure you have the terminology correct is a good thing to do.
but your main point is good: why should the average end user care. i've had the same frustrations with certbot - it tries to be too smart and too magic, and i want to better understand what it's doing. thankfully, the ansible letsencrypt module exists, and behaves in a much more understandable way to me, and that's why it's important to point out the distinction between letsencrypt and certbot - you don't have to use certbot, and everybody who uses letsencrypt should be aware of that. because the author is correct, certbot is kind of a turd, but letsencrypt is awesome and it would suck for people to stop using letsencrypt just because they don't like certbot.
but your main point is good: why should the average end user care. i've had the same frustrations with certbot - it tries to be too smart and too magic, and i want to better understand what it's doing. thankfully, the ansible letsencrypt module exists, and behaves in a much more understandable way to me, and that's why it's important to point out the distinction between letsencrypt and certbot - you don't have to use certbot, and everybody who uses letsencrypt should be aware of that. because the author is correct, certbot is kind of a turd, but letsencrypt is awesome and it would suck for people to stop using letsencrypt just because they don't like certbot.
Ok what's ansible? :)
Some script bunch that adds layers upon layers of VMs?
My website is all static (and private use) so I don't really have a reason to run SSL on it. Make it hard, and I'll give up on the security theater.
Some script bunch that adds layers upon layers of VMs?
My website is all static (and private use) so I don't really have a reason to run SSL on it. Make it hard, and I'll give up on the security theater.
Can we stop with this nonsense about static sites not needing https? It's not just there to protect secrecy. Integrity is vital with how many parties are happy to inject content into any unencrypted https connection these days. Browsers should be able to know that they're receiving the same bytes the server is sending.
I don't know, maybe the solution is to get a pro consumer FCC in the US :)
That would be great for a lot of reasons, but it's not a satisfying answer here.
For one, it'd only help web traffic that was entirely within the US, and only traversed networks within the scope of the FCC's enforcement operations. Do you really trust that coffee shop wifi? I don't, and I don't think it'd ever be big enough for the FCC to care.
Second, such an FCC could always revert back to current behavior. It's not any more reliable to depend on politics to fix a technical problem than it is to depend on technology to fix a political problem.
For one, it'd only help web traffic that was entirely within the US, and only traversed networks within the scope of the FCC's enforcement operations. Do you really trust that coffee shop wifi? I don't, and I don't think it'd ever be big enough for the FCC to care.
Second, such an FCC could always revert back to current behavior. It's not any more reliable to depend on politics to fix a technical problem than it is to depend on technology to fix a political problem.
I can sympathize with the frustration, but I disagree with the criticism. Certbot is capable of doing dry-runs, and is configured by default to try to renew with a month to spare. It logs failures. You should read your logs.
If you've built a system where silently failing is a problem, and you haven't configured any sort of "email me when anything isn't nominal" then you haven't finished configuring your server.
Software updates happen. Unattended updates are a thing. It's not the ninties anymore... You can't hook something up to the internet, chuck it in your closet, forget about it for five years, and expect zero issues. You need to pay attention to it.
If you've built a system where silently failing is a problem, and you haven't configured any sort of "email me when anything isn't nominal" then you haven't finished configuring your server.
Software updates happen. Unattended updates are a thing. It's not the ninties anymore... You can't hook something up to the internet, chuck it in your closet, forget about it for five years, and expect zero issues. You need to pay attention to it.
>You can't hook something up to the internet, chuck it in your closet, forget about it for five years, and expect zero issues. You need to pay attention to it.
This kind of contempt for users is why I try to avoid using any libraries/software written in Python. Somebody always thinks their minor backwards incompatible change is such a precious snowflake that it justifies wasting collectively thousands of hours of users' time debugging and updating their stuff to work around it.
This kind of contempt for users is why I try to avoid using any libraries/software written in Python. Somebody always thinks their minor backwards incompatible change is such a precious snowflake that it justifies wasting collectively thousands of hours of users' time debugging and updating their stuff to work around it.
I don't think this is "contempt for users," it's a position that configurations that appear to setup a web server to run without monitoring for years are deceptive and harmful to the admin taken in by them. It's an ethical (and technical) position that tools should push their users to adopt maintenance practices that are seen as responsible by the wider community. It's not contempt to inform people about the level of work you expect is required to do what they are planning. It's also, if letsencrypt happens to be wrong, not contemptuous to be wrong!
The one with contempt for users is reality itself. Computer security is hard, and whatever you have installed today will force you to update it later, directly or indirectly, due to security.
Don't blame the messenger.
Don't blame the messenger.
I don't think that's what the author says.
There's been some changes in the past to the acme protocol requiring to update whatever client you use.
It isn't contempt to realize that security landscapes change, and what was best practice in the past is insecure now.
That's not a Python specific problem, but otherwise I agree with your point.
No, but when it occurs, it's usually Python
> You need to pay attention to it.
My whole reason for switching to LE was that it was announced as an automatic way to upgrade certificates. So it required less of my attention than before. Or at least, that was the spiel.
My whole reason for switching to LE was that it was announced as an automatic way to upgrade certificates. So it required less of my attention than before. Or at least, that was the spiel.
You should demand a refund. Vote with your wallet.
Of course that was a bit trenchant, however you're complaining about a promise that was never made. No one ever promised that you could setup a old certbot instance and it was out of sight and mind for perpetuity. There are any number of issues that can occur, and honestly if one expected certbot to run without issue, having it automatically updating as well seems to be a base minimum.
Also worth noting that LE was early with ACMEv1, but a lot of alternatives started with ACMEv2. ACMEv2 became the common standard.
Of course that was a bit trenchant, however you're complaining about a promise that was never made. No one ever promised that you could setup a old certbot instance and it was out of sight and mind for perpetuity. There are any number of issues that can occur, and honestly if one expected certbot to run without issue, having it automatically updating as well seems to be a base minimum.
Also worth noting that LE was early with ACMEv1, but a lot of alternatives started with ACMEv2. ACMEv2 became the common standard.
Interesting perspective. For me it was clear from the beginning that I'm trading convenience for free certs.
Criticism of the "official" client was also very loud from the start.
I do agree that LetsEncrypt has been a constant source of maintenance problems over the years. For example now, I have to switch to another alternative acme client, since the old one doesn't seem to be in active development anymore...
However, certificates used to be expensive, even unaffordable for some projects.
Criticism of the "official" client was also very loud from the start.
I do agree that LetsEncrypt has been a constant source of maintenance problems over the years. For example now, I have to switch to another alternative acme client, since the old one doesn't seem to be in active development anymore...
However, certificates used to be expensive, even unaffordable for some projects.
My time isn't free.
> So it required less of my attention than before.
And that is why we run it. But we still monitor.
First we pipe stdout things to syslog via logger(1) to record each run in cron; however stderr is not redirected so we get that sent to us. Further, in our hook scripts, we have an "apache2ctl configtest" when a cert is renewed to verify that Apache (or equivalent) can read the files correctly, then we do the restart/reload. The configtest is sent to stderr as well so we can track when renewals happen and when there's an issue.
On top of all that, we don't actually trust our ACME client(s) to work properly all the time, so we check the expiration time on our certificates. We're monitoring the HTTP(S) service anyway, so adding another check doesn't add much cost.
And all this runs in the background silently and it only becomes top-of-mind really when it shows up in our service alert dashboard.
And that is why we run it. But we still monitor.
First we pipe stdout things to syslog via logger(1) to record each run in cron; however stderr is not redirected so we get that sent to us. Further, in our hook scripts, we have an "apache2ctl configtest" when a cert is renewed to verify that Apache (or equivalent) can read the files correctly, then we do the restart/reload. The configtest is sent to stderr as well so we can track when renewals happen and when there's an issue.
On top of all that, we don't actually trust our ACME client(s) to work properly all the time, so we check the expiration time on our certificates. We're monitoring the HTTP(S) service anyway, so adding another check doesn't add much cost.
And all this runs in the background silently and it only becomes top-of-mind really when it shows up in our service alert dashboard.
> And that is why we run it. But we still monitor.
The difference for me is that Let's Encrypt has required this kind of manual intervention more often than the previous status quo. Previously, I would put an item on my calendar to renew the free StartSSL cert, would do it on schedule, and things would generally go off without a hitch.
For me, the automatic renewer has fallen apart multiple times (for several reasons, including failing to reload nginx after renewing the cert), more than once a year on average, and worse still, they don't fail on any kind of regular schedule. I'm not a pro, just someone who wants a cert for my personal site, so I don't have any complicated monitoring system set up. This means I have to regularly remember to check up on certbot and see if it renewed the cert this time.
The difference for me is that Let's Encrypt has required this kind of manual intervention more often than the previous status quo. Previously, I would put an item on my calendar to renew the free StartSSL cert, would do it on schedule, and things would generally go off without a hitch.
For me, the automatic renewer has fallen apart multiple times (for several reasons, including failing to reload nginx after renewing the cert), more than once a year on average, and worse still, they don't fail on any kind of regular schedule. I'm not a pro, just someone who wants a cert for my personal site, so I don't have any complicated monitoring system set up. This means I have to regularly remember to check up on certbot and see if it renewed the cert this time.
We've had one minor hiccup when LE changed the company that they used for front-end load balancing, but other than that it's set and forget. Generally we don't think about LE/ACME at all at $WORK. We still have some annually-renewed certs for 'appliances', including a wildcard, but practically we don't think about either 'system' more than the other.
If spending $x per year solves the problem for you, then go for it, but it does seem odd to me that you're having so many problems.
As for monitoring, a simple thing to do is to install some well-tested, pre-canned check scripts (e.g., from Nagios):
* https://packages.debian.org/search?keywords=monitoring-plugi...
and have them kick off via cron on a regular basis. A non-zero error code will cause an e-mail to be sent to you.
While we use Nagios (for now) to monitor from a central host, we also leverage them in things like keepalived for running HA clusters to trigger fail-overs. No sense re-inventing the wheel.
If spending $x per year solves the problem for you, then go for it, but it does seem odd to me that you're having so many problems.
As for monitoring, a simple thing to do is to install some well-tested, pre-canned check scripts (e.g., from Nagios):
* https://packages.debian.org/search?keywords=monitoring-plugi...
and have them kick off via cron on a regular basis. A non-zero error code will cause an e-mail to be sent to you.
While we use Nagios (for now) to monitor from a central host, we also leverage them in things like keepalived for running HA clusters to trigger fail-overs. No sense re-inventing the wheel.
Since my college days, I've been running a Linux server, of some sorts, somewhere. Usually running Debian. To be honest, I also haven't configured any sort of notification when anything isn't nominal. Once I did set up Zabbix. But it turned out that Debian/Linux is incredibly reliable. When I migrated to a new server (or rather VPS), I simply haven't bothered installing any kind of notification. It just runs and runs, it's like an appliance.
I can see people falling into the trap. They run Linux their whole lives, and it never failed them. Then they introduce something new, like Certbot, and for the first time, there's a hiccup and they get frustrated :D
I can see people falling into the trap. They run Linux their whole lives, and it never failed them. Then they introduce something new, like Certbot, and for the first time, there's a hiccup and they get frustrated :D
> You need to pay attention to it
If I have to pay attention to it four times a year and regularly spend a day updating software I didn't want to have let alone update, I should have just paid GoDaddy or whoever their $100 grift and only had to update the cert once a year and not hunt down stupid error messages and look for release notes on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying 'Beware of the Leopard.'
Amazon haven't done something this stupid with their 'free" ssl certs for load balancers, and you know why? Because the provide them to "customers", not to "early adopting cheapskate beta testing schmucks"...
If I have to pay attention to it four times a year and regularly spend a day updating software I didn't want to have let alone update, I should have just paid GoDaddy or whoever their $100 grift and only had to update the cert once a year and not hunt down stupid error messages and look for release notes on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying 'Beware of the Leopard.'
Amazon haven't done something this stupid with their 'free" ssl certs for load balancers, and you know why? Because the provide them to "customers", not to "early adopting cheapskate beta testing schmucks"...
>and you know why?
because they control the software on both ends and can update certificate clients on the load balancers anytime they want?
because they control the software on both ends and can update certificate clients on the load balancers anytime they want?
It is actually easier to deploy a repeat certificate you bought as well. Once you have it all setup it is usually just a drop in replacement and a config reload of the web server. Multiple times now I have spent hours fiddling with certbot trying to get it to do its job properly and stop invalidating my certificates. SSL has never been as much hassle as it has been with letsencrypt. Its free and I feel like I am getting what I paid for!
I've been using let's encrypt nearly since it started issuing certificates, and I had to intervene only once so far.
I think one thing that's missing from the discussion is scalability.
Sure you can pay a CA for a cert that is valid for a year, but you still have to do your update dance once per year per site.
With let's encrypt, you write an Ansible playbook (or whatever it is you use for automation) once, run it everywhere. You select a TLS monitoring service (or run your own) once, enter your URLs, and you're done.
You only have to intervene when problems arise (and that can happen with CAs that you have paid as well).
I think one thing that's missing from the discussion is scalability.
Sure you can pay a CA for a cert that is valid for a year, but you still have to do your update dance once per year per site.
With let's encrypt, you write an Ansible playbook (or whatever it is you use for automation) once, run it everywhere. You select a TLS monitoring service (or run your own) once, enter your URLs, and you're done.
You only have to intervene when problems arise (and that can happen with CAs that you have paid as well).
The root CA certificate used to sign the AWS RDS certificates has recently expired and everyone needed to update the root store on their servers. Those who didn't pay attention lost connection to their database:
https://aws.amazon.com/blogs/database/amazon-rds-customers-u...
https://aws.amazon.com/blogs/database/amazon-rds-customers-u...
> It logs failures. You should read your logs.
> If you've built a system where silently failing is a problem, and you haven't configured any sort of "email me when anything isn't nominal" then you haven't finished configuring your server.
I mean, I built a system years ago then got a full time job. Shit breaking every 3 months is kinda bad for a hobbyist website / blog. Even when the monitoring is in place, there's ample room to decide 'you know, family emergency is more important than rearchitecting my TLS certs again.'
I mean, I built a system years ago then got a full time job. Shit breaking every 3 months is kinda bad for a hobbyist website / blog. Even when the monitoring is in place, there's ample room to decide 'you know, family emergency is more important than rearchitecting my TLS certs again.'
Running your own server connected to the Internet is not a hobby and nobody should get the impression that it will be. Otherwise you're just making a donation to a DDOS swarm.
Do we have any real evidence that hobbyist servers are significant hitters when it comes to DDOSes? The idea that hobbyists shouldn't run a server is, IMO, misguided, and dangerous to the health of the internet.
A simple debian VM + nginx + $app_server with automatic updates turned on is not rocket science, and I struggle to see why one shouldn't run that as a hobbyist.
A simple debian VM + nginx + $app_server with automatic updates turned on is not rocket science, and I struggle to see why one shouldn't run that as a hobbyist.
It hosts a static website with no meaningful network dependencies other than the control plane and letsencrypt. I'm relatively comfortable with the situation.
This is a great comment.
There are a lot of people who are going to get tripped up because they aren't really system admins. Or it might not be that important if it fails. If you only have to mess with it rarely, then it's probably not critical.
But for people who spend a substantial amount of time with their hands in servers, this comment is spot on.
There are a lot of people who are going to get tripped up because they aren't really system admins. Or it might not be that important if it fails. If you only have to mess with it rarely, then it's probably not critical.
But for people who spend a substantial amount of time with their hands in servers, this comment is spot on.
>" It's not the ninties anymore... You can't hook something up to the internet, chuck it in your closet, forget about it for five years, and expect zero issues"
Ouch! This is just too bad. I suggest we go back.
Ouch! This is just too bad. I suggest we go back.
You also get an email if your cert is about to expire.
> You can't hook something up to the internet, chuck it in your closet, forget about it for five years, and expect zero issues. You need to pay attention to it.
You should. 2 examples: - the average age of the planes for major airliners is 20 years. Maintenance - yes, retrofitting the cockpit every year just because a vendor decided to change the protocol for the sensor or control, no - about 18 years ago an intern in my company built (with external developers) an application for internal use; today the intern's career reached senior director level, left the company, the application is still used on a regular basis by every single employee; the only change was to move it from a physical server (there was no virtualization in 2002) to a VM. 5 years is not an unreasonable expectation from a damn thing to work without pampering
You should. 2 examples: - the average age of the planes for major airliners is 20 years. Maintenance - yes, retrofitting the cockpit every year just because a vendor decided to change the protocol for the sensor or control, no - about 18 years ago an intern in my company built (with external developers) an application for internal use; today the intern's career reached senior director level, left the company, the application is still used on a regular basis by every single employee; the only change was to move it from a physical server (there was no virtualization in 2002) to a VM. 5 years is not an unreasonable expectation from a damn thing to work without pampering
The first plane started in 1903, they have a bit more experience. Also they are crazy expensive and dangerous. Your mail server is not.
Mail and web servers should be a lot simpler to set up and maintain than aircraft too. But Letsencrypt, instead of making things simpler actually became a source of extra work, sometimes a lot of work.
>the average age of the planes for major airliners is 20 years. Maintenance - yes, retrofitting the cockpit every year just because a vendor decided to change the protocol for the sensor or control, no
Civilian airliners don't face a HUMAN DRIVEN ADVERSARIAL ENVIRONMENT for general airspace. In the rare occasions they do, everyone dies. Internet connected devices all do all the time, and indeed that is the entire point of all these security measures and upgrade requirements. I don't know why this is such a frequent screw up in comparisons by people complaining about security. Why bother with TLS at all? Plain old http is a ton simpler and easier. Why not just run everything as root? Etc etc. Systems were much simpler and more hackable back in the day, but running that way these days is simply not doable.
If you wanted to compare to planes, the proper comparison would be military aircraft. And those do indeed have constant block updates over their lifetimes to ECM, comms, weapons, etc. Dealing with an adversarial environment with intelligent adapter attackers requires adaption in turn.
Civilian airliners don't face a HUMAN DRIVEN ADVERSARIAL ENVIRONMENT for general airspace. In the rare occasions they do, everyone dies. Internet connected devices all do all the time, and indeed that is the entire point of all these security measures and upgrade requirements. I don't know why this is such a frequent screw up in comparisons by people complaining about security. Why bother with TLS at all? Plain old http is a ton simpler and easier. Why not just run everything as root? Etc etc. Systems were much simpler and more hackable back in the day, but running that way these days is simply not doable.
If you wanted to compare to planes, the proper comparison would be military aircraft. And those do indeed have constant block updates over their lifetimes to ECM, comms, weapons, etc. Dealing with an adversarial environment with intelligent adapter attackers requires adaption in turn.
Yes, but you are taking the analogy a bit too far. This is more on the level of vehicles requiring constant updates from the factory in order to be able to still use the roads.
>This is more on the level of vehicles requiring constant updates from the factory in order to be able to still use the roads.
No, it really isn't, unless you drive roads where people are constantly shooting at you and trying to hijack your vehicle. And even then, nothing stops you from just driving your regular vehicle, but insurance won't cover it and manufacturers won't support you and so on. If you want their armor/security systems and indemnities, you must agree to constant updates to counter constant new threats. Don't like that? You've got other options at your own risk/expense.
Adversarial situations are simply more dynamic no matter what they're applied to. What we gain is worth the cost, though it definitely can be a real pain. And to the extent the pain points can be reduced without diminishing security it's always well worth pursuing (and indeed easiness is of course a big part of good security). But that's going to be in the form of "trying to make regular updates easier/more reliable" not "throw it in a closet for 5 years" short of some sort of far future formalized bug free system that doesn't exist.
No, it really isn't, unless you drive roads where people are constantly shooting at you and trying to hijack your vehicle. And even then, nothing stops you from just driving your regular vehicle, but insurance won't cover it and manufacturers won't support you and so on. If you want their armor/security systems and indemnities, you must agree to constant updates to counter constant new threats. Don't like that? You've got other options at your own risk/expense.
Adversarial situations are simply more dynamic no matter what they're applied to. What we gain is worth the cost, though it definitely can be a real pain. And to the extent the pain points can be reduced without diminishing security it's always well worth pursuing (and indeed easiness is of course a big part of good security). But that's going to be in the form of "trying to make regular updates easier/more reliable" not "throw it in a closet for 5 years" short of some sort of far future formalized bug free system that doesn't exist.
Fair enough. Thank you for the insight.
I'm extremely sympathetic to the frustration that led to your post but, yeah, I think that was probably the best phrasing of the argument that you and I can stay grumpy if we want to but probably shouldn't get our way that I've encountered for a while.
[deleted]
> I don't know why this is such a frequent screw up in comparisons by people complaining about security.
I have a suspicion that many, many engineers don't conceive of networked systems as participants in a human-driven adversarial environment. Many conceive of their software systems as little artifacts that function mostly independent of the outside world. When complete and functional, they should require no touching at all to continue doing so.
In other words, many people regard their computer systems as mechanistic and discrete things that can be declared "done" and treated as such. Many engineers work in shops where this is indeed how they work - ship a package and someone else does all the touching to keep it going.
I'm sympathetic. It took me a long time to make the jump to understanding software as operating services that always require maintenance, monitoring, and other forms of operation. Modern computers on modern networks simply cannot be treated as toasters if we want that level of reliability.
The paradigm shift isn't always easy or fun.
I have a suspicion that many, many engineers don't conceive of networked systems as participants in a human-driven adversarial environment. Many conceive of their software systems as little artifacts that function mostly independent of the outside world. When complete and functional, they should require no touching at all to continue doing so.
In other words, many people regard their computer systems as mechanistic and discrete things that can be declared "done" and treated as such. Many engineers work in shops where this is indeed how they work - ship a package and someone else does all the touching to keep it going.
I'm sympathetic. It took me a long time to make the jump to understanding software as operating services that always require maintenance, monitoring, and other forms of operation. Modern computers on modern networks simply cannot be treated as toasters if we want that level of reliability.
The paradigm shift isn't always easy or fun.
If that system was mission critical, would you just leave it running there and not pay attention to it? No monitoring whether the VM is running, the disks are full, the processes are down?
Paying attention doesn't mean that you have to personally look at it every day. Set something up to let you know when something isn't right.
And as you mention: planes get maintained. Nobody lets a plane fly for 20 years without paying attention.
Paying attention doesn't mean that you have to personally look at it every day. Set something up to let you know when something isn't right.
And as you mention: planes get maintained. Nobody lets a plane fly for 20 years without paying attention.
Planes are checked between each flight for basic issues, have a lengthy checklist to go through before using and are often under support contract from the manufacturer.
If you want that level of support you need to pay for it, and you can by using other CA:s. The rest of us we can use letsencrypt.
If you want that level of support you need to pay for it, and you can by using other CA:s. The rest of us we can use letsencrypt.
I'd start by ignoring Letsencrypt's advice to start with Certbot (which I think was used here given the description). Instead look for something like lego[1] or even simpler clients, something like Acme.sh[2] or acme-tiny[3]. At least for me certbot has way to many moving parts and makes the acme protocol feel more complicated than it is.
[1]: https://github.com/go-acme/lego [2]: https://github.com/acmesh-official/acme.sh [3]: https://github.com/diafygi/acme-tiny
[1]: https://github.com/go-acme/lego [2]: https://github.com/acmesh-official/acme.sh [3]: https://github.com/diafygi/acme-tiny
Couldn't agree more. For a person that runs/admins their own DNS and mailserver infra, I'm surprised they didn't immediately gravitate towards one of the lightweight clients.
Been a while since I last needed to get involved with letsencrypt stuff but I've always used and liked: https://github.com/acmesh-official/acme.sh
Ansible's letsencrypt module was also not bad last time I used it: https://docs.ansible.com/ansible/latest/modules/acme_certifi...
Been a while since I last needed to get involved with letsencrypt stuff but I've always used and liked: https://github.com/acmesh-official/acme.sh
Ansible's letsencrypt module was also not bad last time I used it: https://docs.ansible.com/ansible/latest/modules/acme_certifi...
Mailinabox comes with its own pre-installed client that apparently does not upgrade unless you upgrade the whole machine. The upgrade instructions for that particular operation start with: take a newly installed Unbuntu machine. Which isn't always an option.
In the mailinabox case - I suspect this refers to the jump from v0.3x to v0.4x? Which was the point that they moved to Ubuntu 18.04, I think somewhere around the beginning of 2019?
Admittedly I'm somewhat slack about maintenance on my own mailinabox system, so it took me ~12 months to get around to going through the upgrade process, which was pretty painless.
It was just a case of backup/re-image/setup/restore on the same machine, took maybe an hour?
(But yes, it would have been nicer to have a better rollback point than that).
Admittedly I'm somewhat slack about maintenance on my own mailinabox system, so it took me ~12 months to get around to going through the upgrade process, which was pretty painless.
It was just a case of backup/re-image/setup/restore on the same machine, took maybe an hour?
(But yes, it would have been nicer to have a better rollback point than that).
That change happened about a week after I installed it the first time, just after migrating many (way too many) imap stores to it. I figured it should be good for a while at least. My mistake I guess. Or maybe two years counts as enough? Wonder what the lower border is for an operation like that.
Right, that makes things less fun indeed :(
Having issued the certificate manually I now have 90 days to fix this. I'm sure I'll get around to it before the end of September, but even so I am very hard at work to get a project ready for release and this ate up time that I didn't really have (and then of course I did find time to blog about it ;) ).
Once I'm told I need to start evaluating ad selecting alternative automation software to the one provided by the certificate provider to keep my 90 day expiry "free" certs alive - I'm just gonna tell the boss "Fuck that, lets just buy a 12 month cert again and add this onto the calendar like all the other annual updating crap we've always dealt with."
I'm with the OP here. They claimed they were going to simplify and solve a problem I had "OMG, remembering to renew certs every year???". They've replaced it with a three tier set of new unknown problems that fingerpoint at each other claiming not our fault!" and potentialy explode in my face at least four times as often. Sorry, but "free" is too expensive for that.
I'm with the OP here. They claimed they were going to simplify and solve a problem I had "OMG, remembering to renew certs every year???". They've replaced it with a three tier set of new unknown problems that fingerpoint at each other claiming not our fault!" and potentialy explode in my face at least four times as often. Sorry, but "free" is too expensive for that.
Depending on what you need, yet another option is to switch to a web server handles all of this for you. Caddy comes to mind.
I initially used Certbot on my server, then tried to switch to acme.sh, but eventually switch back to Certbot because it is the most painless way for my setup(Debian+Nginx). The problems I have with acme.sh are mostly permission related. Also, Certbot can update nginx configs for me automatically, so I only have to copy and paste commands from official website, then everything are done.
Check out dehydrated:
* https://github.com/dehydrated-io/dehydrated
The hook.sh allows for a lot of flexibility.
* https://github.com/dehydrated-io/dehydrated
The hook.sh allows for a lot of flexibility.
It looks similar to acme.sh too, providing lightweight and flexibility. In contrast, Certbot can modify Nginx config automatically to pass verification and install certificate for me.
I personally prefer tools to handle there things for me. Don't like writing script manually to achieve them.
I personally prefer tools to handle there things for me. Don't like writing script manually to achieve them.
> I personally prefer tools to handle there things for me. Don't like writing script manually to achieve them.
That's what configuration management (Ansible, Chef, etc) is for.
That's what configuration management (Ansible, Chef, etc) is for.
See also:
* https://github.com/dehydrated-io/dehydrated
All you need is Bash/Zsh, OpenSSL, and cURL.
* https://github.com/dehydrated-io/dehydrated
All you need is Bash/Zsh, OpenSSL, and cURL.
I've used acme-tiny for years to manage about 10 certificates and it's great for what it is.
But it has 1 problem in that you can't do DNS based validation with it, which means no wildcard certs. Generally speaking I prefer DNS validation even without wildcard certs too because it means your web server only ever has to think about serving the certificates. Your web server doesn't even need to be up to do a challenge request when you use DNS challenges.
I've moved onto using acme.sh and it's been very good. It supports a lot of DNS host APIs too (DigitalOcean, Cloudflare and like 50 others).
But it has 1 problem in that you can't do DNS based validation with it, which means no wildcard certs. Generally speaking I prefer DNS validation even without wildcard certs too because it means your web server only ever has to think about serving the certificates. Your web server doesn't even need to be up to do a challenge request when you use DNS challenges.
I've moved onto using acme.sh and it's been very good. It supports a lot of DNS host APIs too (DigitalOcean, Cloudflare and like 50 others).
I've been using acme.sh for years, and in that time, I've had to update it once (maybe last year) due to the move to ACMEv2. The interface stayed the same, at least as far as my use cases, but I spent an hour in the docs reading about new features. I've got a wildcard cert now!
Reading Jacques' post made me wonder WTF was going on, especially "The machine that this is all running on will need a much more recent version of its OS." Oh, right, the reference ACME client written in Python.
Reading Jacques' post made me wonder WTF was going on, especially "The machine that this is all running on will need a much more recent version of its OS." Oh, right, the reference ACME client written in Python.
I'm a huge fan of acme.sh. You can easily automate its use in an infrastructure pipeline, set up some simple alerts for failure cases, and then more or less forget about it.
I sympathise with the sentiment of this post and admit things can and should be better. But there are a few points to be made:
> the situation hasn’t changed and if it worked last time it should still work this time
The situation has changed though - who knows what vulnerabilities had to be addressed in any of the dependencies or the protocol. Just because you forgot about it doesn’t mean others did.
I’m not a fan of automated software updates and believe they should be restricted to just essential security patches (where the risk of a vulnerability is greater than the potential to break a system)
> You would not have known this unless you interacted with Letsencrypt on a daily basis, but like most of these things their whole value lies in NOT having to interact with them all the time.
There’s plenty of stuff that we interact with on an infrequent enough basis that we can get caught out when something changes and we realise we didn’t know and maybe had little way to know.
This is just impossible to avoid in the real world. The only way to address it is better communications.
I generate my LetsEncrypt certs in docker containers mainly because of all the dependencies and that it’s clear it’s still evolving. Maybe in 10 years time it’ll be as mature as a nix shell command like ‘ls’.
> the situation hasn’t changed and if it worked last time it should still work this time
The situation has changed though - who knows what vulnerabilities had to be addressed in any of the dependencies or the protocol. Just because you forgot about it doesn’t mean others did.
I’m not a fan of automated software updates and believe they should be restricted to just essential security patches (where the risk of a vulnerability is greater than the potential to break a system)
> You would not have known this unless you interacted with Letsencrypt on a daily basis, but like most of these things their whole value lies in NOT having to interact with them all the time.
There’s plenty of stuff that we interact with on an infrequent enough basis that we can get caught out when something changes and we realise we didn’t know and maybe had little way to know.
This is just impossible to avoid in the real world. The only way to address it is better communications.
I generate my LetsEncrypt certs in docker containers mainly because of all the dependencies and that it’s clear it’s still evolving. Maybe in 10 years time it’ll be as mature as a nix shell command like ‘ls’.
> who knows what vulnerabilities had to be addressed in any of the dependencies or the protocol.
Well, if there were unsolved vulnerabilities in the V1 of that protocol I'd like to know about them, but so far all I have seen are vulnerabilities that were typically solved within days at worst.
Well, if there were unsolved vulnerabilities in the V1 of that protocol I'd like to know about them, but so far all I have seen are vulnerabilities that were typically solved within days at worst.
> Somewhere someone has decided that Letsencrypt should auto-update even if absolutely nothing had been changed on the system that it was running on
Letsencrypt is not a piece of software you install on your systems. You're probably thinking of certbot, which is just one of many available ACME (Letsencrypt-compatible) clients that can manage your TLS certificates. I've used dehydrated and acme-tiny in the past, both work well.
The article is also devoid of any details on what the weird technical issues you ran into that make you think you need a completely new machine. For all I know the machine was running a 10 year old version of OpenSSL.
I assume the change in the challenge protocol being referred to is ACME v1 to ACME v2? Why not just say this outright?
Letsencrypt is not a piece of software you install on your systems. You're probably thinking of certbot, which is just one of many available ACME (Letsencrypt-compatible) clients that can manage your TLS certificates. I've used dehydrated and acme-tiny in the past, both work well.
The article is also devoid of any details on what the weird technical issues you ran into that make you think you need a completely new machine. For all I know the machine was running a 10 year old version of OpenSSL.
I assume the change in the challenge protocol being referred to is ACME v1 to ACME v2? Why not just say this outright?
A two year old install of mail-in-a-box. I am not in the habit of reinstalling my main mail and DNS server just for kicks, it is a lot of work to put it mildly, and very easy to mess up.
> I assume the change in the challenge protocol being referred to is ACME v1 to ACME v2? Why not just say this outright?
Because this has now happened twice, for entirely different reasons.
The first time letsencrypt-auto totally hosed a machine, the second time mail-in-a-box (which was otherwise humming along just fine) spat out an incomprehensible error about it not having enough authority to request a new certificate even though nothing whatsoever had changed on that machine.
> I assume the change in the challenge protocol being referred to is ACME v1 to ACME v2? Why not just say this outright?
Because this has now happened twice, for entirely different reasons.
The first time letsencrypt-auto totally hosed a machine, the second time mail-in-a-box (which was otherwise humming along just fine) spat out an incomprehensible error about it not having enough authority to request a new certificate even though nothing whatsoever had changed on that machine.
Without any details, it's uncertain what the real issue was. It could very well have been attributed to a misconfiguration on your machine.
I manage over 20 client domains over a dozen servers. Have letsencrypt protecting them all. The only time I needed to manually intervene after the initial domain + SSL setup was to upgrade to ACME v2. Everything generally works out of the box with Letsencrypt + Certbot.
I manage over 20 client domains over a dozen servers. Have letsencrypt protecting them all. The only time I needed to manually intervene after the initial domain + SSL setup was to upgrade to ACME v2. Everything generally works out of the box with Letsencrypt + Certbot.
Good, so this hasn't bitten you. I'm sincerely happy for you. But my experience so far has been pretty rotten.
The real issue was a deprecated protocol, one that - as far as I know - could have been kept running at zero cost to LE because it already worked just fine.
The real issue was a deprecated protocol, one that - as far as I know - could have been kept running at zero cost to LE because it already worked just fine.
Likewise I have had issues 3 times. In none of those cases did I really understand what went wrong, the fix was blow it away and redo it. Something just gets messed up the answer the community comes to is just start again. Its hard to answer the question of what went wrong when the error message is often garbage and gives no clue and just says the certificate can't be updated and there is no information on what just got broken.
The answer for me each time has been the age old turn it off and on again with redoing the install (or changing the way I utilised it in docker completely) and I have just found the whole thing more hassle than SSL certificates ever were.
The answer for me each time has been the age old turn it off and on again with redoing the install (or changing the way I utilised it in docker completely) and I have just found the whole thing more hassle than SSL certificates ever were.
The responses here are an interesting mix between those who have not had it happen to them (yet) and those who have (or have had it multiple times).
The former can't imagine the latter are anywhere near competent, it could obviously never happen to them so the person experiencing the problem must somehow be at fault.
It's the message board equivalent of 'can't reproduce', closed.
The former can't imagine the latter are anywhere near competent, it could obviously never happen to them so the person experiencing the problem must somehow be at fault.
It's the message board equivalent of 'can't reproduce', closed.
My comment was not accusation of incompetence, rather the lack of details provided. It's like receiving a ticket saying "This doesn't work. Fix it now"
> The first time letsencrypt-auto totally hosed a machine
Fair, I also don't like certbot as a software, hence naming alternative ACME clients. Though to get to a point where it completely hosed a machine sounds impressive.
> the second time mail-in-a-box (which was otherwise humming along just fine) spat out an incomprehensible error about it not having enough authority to request a new certificate
As far as I can tell, Mail-in-a-Box doesn't do any Letsencrypt certificate management so this confuses me.
I find it hard to support your conclusion that this is Letsencrypt's fault and not, for example, Mail-in-a-Box.
Fair, I also don't like certbot as a software, hence naming alternative ACME clients. Though to get to a point where it completely hosed a machine sounds impressive.
> the second time mail-in-a-box (which was otherwise humming along just fine) spat out an incomprehensible error about it not having enough authority to request a new certificate
As far as I can tell, Mail-in-a-Box doesn't do any Letsencrypt certificate management so this confuses me.
I find it hard to support your conclusion that this is Letsencrypt's fault and not, for example, Mail-in-a-Box.
Mailinabox uses letsencrypt to issue certificates for locally hosted web properties (mail in a box is a bit of a misnomer, it does much more than just mail).
Letsencrypt shut down a protocol version that Mailinabox critically relied upon and which worked flawlessly as little as a few months ago. There never was - as far as I could see - a user visible warning or a warning to the email address supplied with my Letsencrypt certificates that something was about to change and I'm not in the habit of monitoring each and every such component. Especially not because the whole idea of Letsencrypt the way it was originally promoted was that a free CA would allow for automatic updates of certificates taking away a pain point: to remember to upgrade your certs once per year. Even the biggest orgs got bitten by that one. So we now have 90 day valid certs that get replaced automatically because 90 days is much too short to do it manually.
But then renewal and issuing of new certificates should be as stable as they could possibly be and changing protocols without backwards compatibility does not fit in that particular playbook (at least, not according to me).
Letsencrypt shut down a protocol version that Mailinabox critically relied upon and which worked flawlessly as little as a few months ago. There never was - as far as I could see - a user visible warning or a warning to the email address supplied with my Letsencrypt certificates that something was about to change and I'm not in the habit of monitoring each and every such component. Especially not because the whole idea of Letsencrypt the way it was originally promoted was that a free CA would allow for automatic updates of certificates taking away a pain point: to remember to upgrade your certs once per year. Even the biggest orgs got bitten by that one. So we now have 90 day valid certs that get replaced automatically because 90 days is much too short to do it manually.
But then renewal and issuing of new certificates should be as stable as they could possibly be and changing protocols without backwards compatibility does not fit in that particular playbook (at least, not according to me).
To be clear on the facts, Let's Encrypt hasn't shut down ACMEv1 (yet).
Accounts with domains which were already renewing via ACMEv1 can continue to do so, until June 2021. Only new domains are shut out.
To me, seems fair enough. If you're booting up a new server, you may as well use a supported ACME client version.
I do think that Let's Encrypt/EFF did make some big mistakes all those years ago on the ACME client side, though, which has directly translated to ongoing user suffering.
Python was a not a good choice. Innumerable efforts have gone towards figuring out packaging issues purely caused by the choice of language. The CI alone is ridiculous. I don't understand why they didn't use something easier to package, like Go, which was sufficiently mature in 2014-2015 (well, duh, Boulder is written in Go). Certbot had to be available universally and needed to have a simple dependency and packaging story.
Relying on external, volunteer distro package maintainers to figure out the rat's nest of Python and native dependencies required by Certbot without blowing up other packages is a problem too. God bless those volunteers, but those roles needed to be funded and given priority, because being unable to get a remotely recent (i.e. not years out of date) version or any of the DNS plugins is honestly one of the worst parts of using Certbot. Today's solution (Certbot snaps) isn't a panacea either.
I somewhat suspect that Jaques' issues may have been avoided entirely if packaging had gone down a different route.
I've contributed a bunch of patches to both Boulder and Certbot (and will continue to) but it's honestly a bit heartbreaking to see that the CA side do such a good job and the client is stuck in a revolving door of issues relating to their choice of programming language.
That acme.sh is a fuckin' gigantic hacky shell script (this is a compliment Neil, I promise) and is, in many ways, superior to Certbot, is quite disappointing.
This post isn't meant to dump on the past and current maintainers either. I have an abundance of empathy for them. I can see how many of the decisions would have been reasonable at the time, and how the evolution of the protocol seriously complicated things. I see only happier days for Certbot ahead and hope to be a part of that.
Accounts with domains which were already renewing via ACMEv1 can continue to do so, until June 2021. Only new domains are shut out.
To me, seems fair enough. If you're booting up a new server, you may as well use a supported ACME client version.
I do think that Let's Encrypt/EFF did make some big mistakes all those years ago on the ACME client side, though, which has directly translated to ongoing user suffering.
Python was a not a good choice. Innumerable efforts have gone towards figuring out packaging issues purely caused by the choice of language. The CI alone is ridiculous. I don't understand why they didn't use something easier to package, like Go, which was sufficiently mature in 2014-2015 (well, duh, Boulder is written in Go). Certbot had to be available universally and needed to have a simple dependency and packaging story.
Relying on external, volunteer distro package maintainers to figure out the rat's nest of Python and native dependencies required by Certbot without blowing up other packages is a problem too. God bless those volunteers, but those roles needed to be funded and given priority, because being unable to get a remotely recent (i.e. not years out of date) version or any of the DNS plugins is honestly one of the worst parts of using Certbot. Today's solution (Certbot snaps) isn't a panacea either.
I somewhat suspect that Jaques' issues may have been avoided entirely if packaging had gone down a different route.
I've contributed a bunch of patches to both Boulder and Certbot (and will continue to) but it's honestly a bit heartbreaking to see that the CA side do such a good job and the client is stuck in a revolving door of issues relating to their choice of programming language.
That acme.sh is a fuckin' gigantic hacky shell script (this is a compliment Neil, I promise) and is, in many ways, superior to Certbot, is quite disappointing.
This post isn't meant to dump on the past and current maintainers either. I have an abundance of empathy for them. I can see how many of the decisions would have been reasonable at the time, and how the evolution of the protocol seriously complicated things. I see only happier days for Certbot ahead and hope to be a part of that.
> If you're booting up a new server, you may as well use a supported ACME client version.
This assumes that the new server is what will be running the new ACME client but in this case it was just an alias added to an older installation in order to redirect the naked domain to the actual web server, which indeed was installed from scratch. I'm sure that's an edge case that nobody at LE thought of or cared about but adding a domain does not necessarily mean you are going to be doing a lot of work on that particular machine.
I sympathize with your python pain. The fact that I can't run software I wrote three years ago in the current environment is what turned me off of python for the rest of my career. I consider it to be good enough for toy projects but not solid enough for any long term project. The investment in time and effort can all be negated by breaking changes and the fact that just about every application seems to want to ship a complete python runtime along, which will in some magical way interact with the system(s) already installed.
One machine I was working on had 3 different python installations on it and that caused no end of trouble. The only python distro that I think is good enough is Anaconda but it is super bloated.
This assumes that the new server is what will be running the new ACME client but in this case it was just an alias added to an older installation in order to redirect the naked domain to the actual web server, which indeed was installed from scratch. I'm sure that's an edge case that nobody at LE thought of or cared about but adding a domain does not necessarily mean you are going to be doing a lot of work on that particular machine.
I sympathize with your python pain. The fact that I can't run software I wrote three years ago in the current environment is what turned me off of python for the rest of my career. I consider it to be good enough for toy projects but not solid enough for any long term project. The investment in time and effort can all be negated by breaking changes and the fact that just about every application seems to want to ship a complete python runtime along, which will in some magical way interact with the system(s) already installed.
One machine I was working on had 3 different python installations on it and that caused no end of trouble. The only python distro that I think is good enough is Anaconda but it is super bloated.
Agree. There are a number of Go-based clients, e.g. acmetool and lego.
As Andrew "bunnie" Huang said recently: "Python is a great glue, but a terrible structural material. Do not build skyscrapers out of Python" (https://bunniestudios.com/blog/?p=5863).
As Andrew "bunnie" Huang said recently: "Python is a great glue, but a terrible structural material. Do not build skyscrapers out of Python" (https://bunniestudios.com/blog/?p=5863).
> That acme.sh is a fuckin' gigantic hacky shell script (this is a compliment Neil, I promise) and is, in many ways, superior to Certbot, is quite disappointing.
See also dehydrated. It's what we use in many places, and I find it much easier to understand what's going on that certbot.
There's quite a list of dependencies that get pulled in when one does an apt-get install certbot, and I'm not quite comfortable with that. I can generally follow a shell script though.
See also dehydrated. It's what we use in many places, and I find it much easier to understand what's going on that certbot.
There's quite a list of dependencies that get pulled in when one does an apt-get install certbot, and I'm not quite comfortable with that. I can generally follow a shell script though.
> Letsencrypt is not a piece of software you install on your systems. You're probably thinking of certbot
This is such a shitty mischaracterisation of how LetEncrypt markets themselves. Typical "move fast and break things" kind of victim blaming excuses for a total lack of professionalism on their part. Want to be a part of a significant portion of the web's security stance? Take some fucking responsibilty for making sure you don't break crap and then claim "It's not LetEncrypt, it's ACME!!!" This is why so many managers keep signing off on huge Microsoft bills. They might ship crap, but they rarely break your productions systems on you like this...
This is such a shitty mischaracterisation of how LetEncrypt markets themselves. Typical "move fast and break things" kind of victim blaming excuses for a total lack of professionalism on their part. Want to be a part of a significant portion of the web's security stance? Take some fucking responsibilty for making sure you don't break crap and then claim "It's not LetEncrypt, it's ACME!!!" This is why so many managers keep signing off on huge Microsoft bills. They might ship crap, but they rarely break your productions systems on you like this...
I can vouch for dehydrated. It only needed updating once (probably due to the mentioned protocol change), and it's running ever since on all my servers without any issues, or updates, or bloat. It's just a shellscript. (several years already, around 4-5)
> The basic idea is that certificates should be free, that the barrier to install them should be as low as possible and that updating certificates should be automated.
This is the wrong order to think about it, and that's a little bit important or I wouldn't mention it.
They want to drive automation and robots don't have wallets. The Let's Encrypt service has to be free at point of use because if it costs say $1 per certificate then automating it becomes much trickier as you need to give the automation a supply of money to buy certificates and now you're solving a hard problem you didn't need to have.
If the first goal was to make it very cheap, you can see how to get to say $1 certificates pretty easily from where we were already, and since name owners are likely spending a few bucks per year on the domain name that probably counts as success. But once you set the goal as automation a $1 certificate is only slightly better than a $100 certificate, because adding some sort of payment mechanism for an automated service is a big problem.
Now, being free at the point of use is also good for other reasons, but I think it'd be a harder sell without the technical argument.
This is the wrong order to think about it, and that's a little bit important or I wouldn't mention it.
They want to drive automation and robots don't have wallets. The Let's Encrypt service has to be free at point of use because if it costs say $1 per certificate then automating it becomes much trickier as you need to give the automation a supply of money to buy certificates and now you're solving a hard problem you didn't need to have.
If the first goal was to make it very cheap, you can see how to get to say $1 certificates pretty easily from where we were already, and since name owners are likely spending a few bucks per year on the domain name that probably counts as success. But once you set the goal as automation a $1 certificate is only slightly better than a $100 certificate, because adding some sort of payment mechanism for an automated service is a big problem.
Now, being free at the point of use is also good for other reasons, but I think it'd be a harder sell without the technical argument.
The technical argument is what sold me, not the price point.
If Jacques has been running the same system for five years, he probably missed five Nginx CVEs and 200+ kernel CVEs. Some of those required changes that ended up affecting the behavior of the system.
Certbot is not a replacement for a sysadmin.
DevOps is not a replacement for a sysadmin.
DevSecOps is not a replacement for a security engineer.
DevSecNetOps is not a replacement for a network engineer.
You need to have someone with the skills who applies those skills to your systems, or else you are hosed. This has always been true and is likely to always be true.
The good news is that cooperative projects make it easier. Let's Encrypt, the Debian project, the Google security hole finders, the whole CVE infrastructure -- the people working on these add an awful lot of value to the world.
Certbot is not a replacement for a sysadmin.
DevOps is not a replacement for a sysadmin.
DevSecOps is not a replacement for a security engineer.
DevSecNetOps is not a replacement for a network engineer.
You need to have someone with the skills who applies those skills to your systems, or else you are hosed. This has always been true and is likely to always be true.
The good news is that cooperative projects make it easier. Let's Encrypt, the Debian project, the Google security hole finders, the whole CVE infrastructure -- the people working on these add an awful lot of value to the world.
That system is about two years old.
As for the rest of your comment, you are somewhat underestimating what I can do but I like to automate away that sort of work because there are more important things in life.
And I expect things, once properly automated, to continue to function unless I mess with them. These cascading failures caused by some policy change elsewhere culminating in millions of installations having to go through sometimes costly upgrades are not the future I had in mind when I started my career in IT.
As for the rest of your comment, you are somewhat underestimating what I can do but I like to automate away that sort of work because there are more important things in life.
And I expect things, once properly automated, to continue to function unless I mess with them. These cascading failures caused by some policy change elsewhere culminating in millions of installations having to go through sometimes costly upgrades are not the future I had in mind when I started my career in IT.
I don't underestimate what you can do -- I saw the LEGO sorter. That's nothing short of amazing.
But I am surprised at the attitude of "this automated system will continue to function". It's never been true in connected systems that care about security, and it's only true for limited timelines for independent systems. In the long run, everything decays.
But I am surprised at the attitude of "this automated system will continue to function". It's never been true in connected systems that care about security, and it's only true for limited timelines for independent systems. In the long run, everything decays.
But in this particular case there was no good reason why it came out the way it did. And that's what bothers me more than anything about it, that such a popular package would force users to expend a lot of effort for zero gain. It's just make-work.
Unattended-upgrades plus checkrestart plus periodic reboots for kernel upgrades can get you a surprisingly long way.
I think one problem was that LetsEncrypt's marketing failed to make it clear that it's a tradeoff: You get automation and free, but at the cost of extra sysadminning headaches. I was initially sold on both the cost and the turnkey nature of the system, but realized only after 3 months that it needs to be constantly babysat.
Are you suggesting the three people should be required to use a mailserver?
I can't speak for Jacques, but when I deliver software, I try to avoid any breaking changes for as long as possible. In the event that breaking changes are absolutely unavoidable--and there seems to be a question about whether that applies in this case--then I communicate clearly beforehand in simple terms about WHY the change is necessary, and WHAT the steps are to implement it, and I ensure buy-in on the part of those using my code. I try to make such changes as simple as possible, preferably "run this script and check the simple expected output."
That does not seem to be the focus here. The update might not have been strictly necessary in the first place, and if it was, it wasn't communicated consistently well to customers, and the communication doesn't seem to have been so simple and straightforward, either. Barring the communication, humane error messages would be nice, but those are also lacking.
I guess I would add to your list: when a company hires sysadmins, security engineers, and network engineers, they should also hire some UI people and good product managers to communicate with the outside world.
I can't speak for Jacques, but when I deliver software, I try to avoid any breaking changes for as long as possible. In the event that breaking changes are absolutely unavoidable--and there seems to be a question about whether that applies in this case--then I communicate clearly beforehand in simple terms about WHY the change is necessary, and WHAT the steps are to implement it, and I ensure buy-in on the part of those using my code. I try to make such changes as simple as possible, preferably "run this script and check the simple expected output."
That does not seem to be the focus here. The update might not have been strictly necessary in the first place, and if it was, it wasn't communicated consistently well to customers, and the communication doesn't seem to have been so simple and straightforward, either. Barring the communication, humane error messages would be nice, but those are also lacking.
I guess I would add to your list: when a company hires sysadmins, security engineers, and network engineers, they should also hire some UI people and good product managers to communicate with the outside world.
I run several static websites. Thanks to S3, they do not require a sysadmin. Thanks to Google, they do require regularly updated certs. We run certbot from an AWS instance that exists for about 10 minutes/day. Are you telling me that my little projects, which I have purposefully designed to require minimal maintenance and which cost me ~$10/year in static hosting, require an actual sysadmin?
I'm saying that someone needs to know what they are doing, and check up on them when something goes wrong. That person is the sysadmin.
If you don't know who the sysadmin is, it's you.
If you don't know who the sysadmin is, it's you.
> If you don't know who the sysadmin is, it's you.
Brilliant. I'm printing this out and taping it to my wall.
Brilliant. I'm printing this out and taping it to my wall.
Put Cloudfront in front of your S3 bucket and they Amazon will handle your certificates for free.
Egress traffic is cheaper too I believe.
Egress traffic is cheaper too I believe.
From the very beginning I was sold by Let's Encrypt's concept but I loathed to install their bloated client. Instead I have been using acme-tiny [0] from the very beginning, and it saved me from all the frustration.
[0] https://github.com/diafygi/acme-tiny
[0] https://github.com/diafygi/acme-tiny
Same here. Note that acme-tiny requires to write scripts to manage the certificate update and reload the services to take in account the new certificates.
I've been using autocert (golang.org/x/crypto/acme/autocert) to fetch and renew certificates. A few weeks ago, I added new subdomains to my site, and noticed that it wasn't being issued new certificates.
After searching for the error in the logs, I landed here: https://community.letsencrypt.org/t/end-of-life-plan-for-acm.... I missed the whole ACMEv1 deprecation notice. I thought autocert would take care of this (and it does), except I had not updated to the latest version of the library. Everything was resolved after updating the library, but I still had to clear my local HSTS to verify. This would not have been acceptable if there were actual traffic going to that subdomain.
I guess the lesson to learn is that even though many of the barriers to automatic certificate renewal have been removed, I still have to keep up to date on breaking changes and deprecation notices, especially if updates only happen every couple of years.
After searching for the error in the logs, I landed here: https://community.letsencrypt.org/t/end-of-life-plan-for-acm.... I missed the whole ACMEv1 deprecation notice. I thought autocert would take care of this (and it does), except I had not updated to the latest version of the library. Everything was resolved after updating the library, but I still had to clear my local HSTS to verify. This would not have been acceptable if there were actual traffic going to that subdomain.
I guess the lesson to learn is that even though many of the barriers to automatic certificate renewal have been removed, I still have to keep up to date on breaking changes and deprecation notices, especially if updates only happen every couple of years.
The real lesson is to always have automated monitoring and alerting in place for everything. Certificate renewal, API schemas, service dependencies, core libraries and protocols - everything can and does change without notice. You could have a single dependency on www.google.com and it would still eventually break.
In case anyone is left wondering amout what specifically has changed and whether they might also have problems manifest suddenly:
https://letsencrypt.org/docs/acme-protocol-updates/
https://community.letsencrypt.org/t/end-of-life-plan-for-acm...
So this has been planned for quite some time, and if you have a current Letsencrypt client, it should keep working seamlessly.
But yeah, if you have servers running an old OS release that won't get a new certbot version, that's gonna suck.
https://letsencrypt.org/docs/acme-protocol-updates/
https://community.letsencrypt.org/t/end-of-life-plan-for-acm...
So this has been planned for quite some time, and if you have a current Letsencrypt client, it should keep working seamlessly.
But yeah, if you have servers running an old OS release that won't get a new certbot version, that's gonna suck.
Why force people to upgrade if it isn't necessary? Was there a huge price on the side of LE to keep the old protocol up and running until it was no longer in use? Is that price balanced out by the combined cost of the users having to upgrade their systems, something that typically costs a day or more per installation?
Reading between the lines, it seems the v1 protocol was basically a proof of concept implementation and differs in some points from the final RFC specification.
With Letsencrypt being by far the most popular ACME provider, if they kept running the old version even optionally, there would be very little incentive for clients to start supporting the standard instead of the POC.
I can understand how they don't want to be the IE6 of CAs.
With Letsencrypt being by far the most popular ACME provider, if they kept running the old version even optionally, there would be very little incentive for clients to start supporting the standard instead of the POC.
I can understand how they don't want to be the IE6 of CAs.
Unless there was an incurable security issue in the V1 protocol I see no reason to force an upgrade on the part of untold millions of installations.
From the linked article above
> In November of 2019 we will stop allowing new account registrations through our ACMEv1 API endpoint. _Existing accounts will continue to function normally._
> In November of 2019 we will stop allowing new account registrations through our ACMEv1 API endpoint. _Existing accounts will continue to function normally._
Except that a new domain counts as a new account.
That is not true. In fact you have an account key that identifies you and the account. With that account you can create and update the certificates of many domains.
Well, that's an implementation detail for Mailinabox, which I do not have sufficient insight in but as far as I know all the domains on that machine share the same account key, and an attempt to manually update one of these gave the exact same error as the mailinabox UI. So either miab uses one account per domain (which is very well possible, but that detail is not exposed) or something else is wrong.
I can't disagree more with the author of the post for the bad part. I manage quite a few things with letsencrypt, some having been set years ago. Each time I needed to take action I received an email from letsencrypt. I think they actually do this really well.
If you setup things in ways that you get no feedback, you'll have bad surprises but that's the same with absolutely everything in life.
The only difference with what was before letsencrypt is that certificates have a relatively short life, so a problem can appear sooner than it would before and it's automated. Like everything automated, it involve tools, infra and maintenance.
The only difference with what was before letsencrypt is that certificates have a relatively short life, so a problem can appear sooner than it would before and it's automated. Like everything automated, it involve tools, infra and maintenance.
Very true.
Generalizing: many projects would benefit from taking a more user-centric view of the world. Your project is not the center of the universe. It is not the main life focus for your users. Your users have things to do and places to be, and your project is just a tiny portion of their entire life. Take that into account when making decisions.
Generalizing: many projects would benefit from taking a more user-centric view of the world. Your project is not the center of the universe. It is not the main life focus for your users. Your users have things to do and places to be, and your project is just a tiny portion of their entire life. Take that into account when making decisions.
The problem is that Letsencrypt is, while useful and important (and free, let's not forget that!), one of those "if it works perfectly, you won't notice anything at all" things. Users don't need to interact actively or regularly with it at all, that's one of its main value proposition.
So how exactly are they supposed to handle a breaking change? Just never have any breaking changes, ever? What if a breaking change is necessary for security reasons, with security being, after all, the primary reason for the project to exist at all?
So how exactly are they supposed to handle a breaking change? Just never have any breaking changes, ever? What if a breaking change is necessary for security reasons, with security being, after all, the primary reason for the project to exist at all?
> Just never have any breaking changes, ever?
Absolutely, that is the correct answer.
If a breaking change is absolutely needed that implies a new major release number, but never, ever, break compatibility in the current major release family.
Absolutely, that is the correct answer.
If a breaking change is absolutely needed that implies a new major release number, but never, ever, break compatibility in the current major release family.
Absolutely not, that is the wrong answer.
If a security-critical issue cannot be fixed without a breaking change, you make that breaking change. Anything else is irresponsible, as is the desire of users to keep running insecure systems to avoid the effort of updating them.
If a security-critical issue cannot be fixed without a breaking change, you make that breaking change. Anything else is irresponsible, as is the desire of users to keep running insecure systems to avoid the effort of updating them.
> and free, let's not forget that!
If they can't run a reliable professional service doing it for free - stop lying about that and claiming you're a valid alternative to buying certs once a year from incumbent vendors.
If they can't run a reliable professional service doing it for free - stop lying about that and claiming you're a valid alternative to buying certs once a year from incumbent vendors.
This has nothing whatsoever to do with realiability or professionalism, much less is it related to funding, and it very much is a valid alternative.
A very expensive paid service might be able to offer direct support that walks you through the upgrade procedure in common setups, that's about it.
A very expensive paid service might be able to offer direct support that walks you through the upgrade procedure in common setups, that's about it.
The _worst_ of the shitty alternatives, let’s say GoDaddy, have this approach where the mechanism by which I update the ssl certs they provide has remained the same since, honestly? Probably the Mid/late 90’s which was the first time I added an ssl cert to an Apache config. GoDaddy have made exactly zero breaking changes in those intervening 20-25 year.
If this had been due to an actual critical security breach in ACMEv1, then maybe doing it would have been excusable - but even then, doing it without proactively contacting every user of their certs to let then know would have been inexcusable.
So far as I can tell though, the ACME 1->2 “upgrade” was closer to”Oh, we didn’t like the old guy’s code, so we rewrote it. It’s almost the same, but better! It passes almost 70% of the tests, so we decided versioning things so we could maintain backwards compatibility was too hard, so it sucks to be running a stone ages website that’s over 180 days old - who even are those losers? YOLO!”
If this had been due to an actual critical security breach in ACMEv1, then maybe doing it would have been excusable - but even then, doing it without proactively contacting every user of their certs to let then know would have been inexcusable.
So far as I can tell though, the ACME 1->2 “upgrade” was closer to”Oh, we didn’t like the old guy’s code, so we rewrote it. It’s almost the same, but better! It passes almost 70% of the tests, so we decided versioning things so we could maintain backwards compatibility was too hard, so it sucks to be running a stone ages website that’s over 180 days old - who even are those losers? YOLO!”
I've been using Letsencrypt since the beta and I enjoy it for the simple reason of not having to get private keys via email (remember the times before LE?).
So far, I had exactly one frustrating moment with LE (or was it certbot?). I've created AAAA records for a domain, but didn't set the nginx up to accept IPv6 connections. Certbot failed to fetch the certificate with some strange message. I figured it out. It was not that bad.
Letsencrypt is a blessing.
So far, I had exactly one frustrating moment with LE (or was it certbot?). I've created AAAA records for a domain, but didn't set the nginx up to accept IPv6 connections. Certbot failed to fetch the certificate with some strange message. I figured it out. It was not that bad.
Letsencrypt is a blessing.
If you were sending or receiving private keys via email then you were doing it wrong. CSRs and certificates do not contain private keys.
I was not sending them. I was receiving them as several SSL providers send them zippped via email.
I struggle with the same thing. LetsEncrypt/Certbot was the solution to maintenance free SSL certificates. Well it's been a bumpy ride so far. Both my FreeBSD and Linux servers has had issues with CertBot and some of the third party softwares, like acme.sh. Like 3-4 times a year I need to spend time to debug what has gone wrong. A $10 yearly certificate quickly becomes a no brainer.
I agree with some of the comments, title should be not letsencrypt but certbot, the provided implementation of certifying your server.
At least for Ubuntu 20.04, Letsencrypt does not require you to install the Python stack for certbot anymore and switched to a Snapd package. Although at least I had to install Snapd for that matter.
At least for Ubuntu 20.04, Letsencrypt does not require you to install the Python stack for certbot anymore and switched to a Snapd package. Although at least I had to install Snapd for that matter.
But that would require upgrading to Ubuntu 20.04, which is a major operation in its own right. And it's not as if snap is without problems of its own.
For the record, snapd is available all the way back to 14.04 [1]
I use acme.sh with dns, it just plonks a certificate in a directory of your choice and runs a reload command of your choice. I actually upload the certificate into a kubernetes cluster. All repeated by cron
[1] https://snapcraft.io/docs/installing-snap-on-ubuntu
I use acme.sh with dns, it just plonks a certificate in a directory of your choice and runs a reload command of your choice. I actually upload the certificate into a kubernetes cluster. All repeated by cron
[1] https://snapcraft.io/docs/installing-snap-on-ubuntu
I recently successfully switched to the latest version of Certbot using snap on a Ubuntu 16.04 server.
Tried doing the same with a Ubuntu 14.04 server. That failed totally while installing snapd on it, which replaced existing upstart init system with systemd. This caused no services to start on next reboot.
Tried doing the same with a Ubuntu 14.04 server. That failed totally while installing snapd on it, which replaced existing upstart init system with systemd. This caused no services to start on next reboot.
Note that the availability of snaps is both distro and release -independent.
> At least for Ubuntu 20.04, Letsencrypt does not require you to install the Python stack for certbot anymore and switched to a Snapd package.
To be clear: the snap is available and maintained by upstream. If you want the latest version of Certbot directly from upstream then the snap is the easiest way to get it. However Ubuntu 20.04 itself also ships the deb as packaged so you can use that if you want, and the "pip install" method in a virtualenv also works (but IMHO is the least maintainable method; it's more useful as a building block for other mechanisms).
> At least for Ubuntu 20.04, Letsencrypt does not require you to install the Python stack for certbot anymore and switched to a Snapd package.
To be clear: the snap is available and maintained by upstream. If you want the latest version of Certbot directly from upstream then the snap is the easiest way to get it. However Ubuntu 20.04 itself also ships the deb as packaged so you can use that if you want, and the "pip install" method in a virtualenv also works (but IMHO is the least maintainable method; it's more useful as a building block for other mechanisms).
A few people have already suggested alternatives to certbot, so I thought I'd throw another suggestion into the mix: dehydrated (formerly known as letsencrypt.sh)
https://github.com/dehydrated-io/dehydrated
I've got only good things to say about it. It's a single shell script, making it super easy to install and start using. It's quite configurable, but has sensible defaults and just works without demanding much operator attention.
https://github.com/dehydrated-io/dehydrated
I've got only good things to say about it. It's a single shell script, making it super easy to install and start using. It's quite configurable, but has sensible defaults and just works without demanding much operator attention.
I second your recommendation. After installing it on some of my now long-running servers back in 2016, I've only touched it a couple of times since. It's been working seamlessly since then.
Configuration was easy, I just ended up writing a simple shell script for the deploy challenge hook, to copy the certificates and reload configurations on nginx etc.
The two times I chose to intervene were straightforward too. The first was because of the ACME protocol upgrade, for which I just needed to drop in a newer copy of dehydrated, and rename references to its old name. The second was because I needed to configure the ACME DNS challenge to get a wildcard domain, and that just involved modifying the deploy challenge hook to temporarily run a custom DNS server that responded to the challenge.
I did take a look at certbot, or whatever it was called back then, and decided against it quite quickly due to the perceived complexity. Also I seem to recall it didn't support nginx properly.
After reading this user review, I'm quite glad I did that. Though I do wonder why the author persisted in using that client despite all the problems they were having with it, given the choices available in the wider Let's Encrypt ecosystem.
Configuration was easy, I just ended up writing a simple shell script for the deploy challenge hook, to copy the certificates and reload configurations on nginx etc.
The two times I chose to intervene were straightforward too. The first was because of the ACME protocol upgrade, for which I just needed to drop in a newer copy of dehydrated, and rename references to its old name. The second was because I needed to configure the ACME DNS challenge to get a wildcard domain, and that just involved modifying the deploy challenge hook to temporarily run a custom DNS server that responded to the challenge.
I did take a look at certbot, or whatever it was called back then, and decided against it quite quickly due to the perceived complexity. Also I seem to recall it didn't support nginx properly.
After reading this user review, I'm quite glad I did that. Though I do wonder why the author persisted in using that client despite all the problems they were having with it, given the choices available in the wider Let's Encrypt ecosystem.
I had the same experience with LE autoupdate. It's been working on a few years on my dedicated server and one day I woke up and all my certs were expired. LE failed on autoupdate and was failing to renew certs. It took me approximately 2hours to get it working. Had to change DNS TXT entries, get new ACME accounts as old accounts was creating new directories mydomain.com-0001 and mydomain.com-0002, which would mean I would have to change config in postfix, dovecot, nginx and more...
Given that OP is a self-proclaimed "update hater", I guess the reason they don't get a 5 year commercial cert is cost? In that case whining online about it seems uncalled for.
Also from an architecture standpoint, the post seems to suggest that OP has certbot running on a variety of machines throughout his infrastructure. If you have many servers relying on letsencrypt, you probably want to centralize generating certificates somewhere, and have other machines pull from there.
Also from an architecture standpoint, the post seems to suggest that OP has certbot running on a variety of machines throughout his infrastructure. If you have many servers relying on letsencrypt, you probably want to centralize generating certificates somewhere, and have other machines pull from there.
> I guess the reason they don't get a 5 year commercial cert is cost?
Web browsers no longer support 5 year certificates. The new maximum is 1 year as of last week, with a view to make that duration even shorter over time as certificate renewal & automated updates become more widespread.
https://nakedsecurity.sophos.com/2020/02/24/apple-chops-safa...
Web browsers no longer support 5 year certificates. The new maximum is 1 year as of last week, with a view to make that duration even shorter over time as certificate renewal & automated updates become more widespread.
https://nakedsecurity.sophos.com/2020/02/24/apple-chops-safa...
Thanks, wasn't aware of that, very interesting!
You apparently did not understand that article very well. I'm all for updates. I'm 100% on board with security related updates and bug fixes.
I'm very much against updates that break things, move things, surprise the user, include bloat and a hundred other violations of trust that get pushed upon users. And this is actually a nice case of such a thing: a protocol on a server that I have no control over got changed. And that now requires me to do a bunch of work to no gain for myself and at the expense of a bunch of time when when I had other plans.
I'm very much against updates that break things, move things, surprise the user, include bloat and a hundred other violations of trust that get pushed upon users. And this is actually a nice case of such a thing: a protocol on a server that I have no control over got changed. And that now requires me to do a bunch of work to no gain for myself and at the expense of a bunch of time when when I had other plans.
Personally I’ve been using acme.sh over certbot for a while now. Saves headaches about python dependence.
https://github.com/acmesh-official/acme.sh
https://github.com/acmesh-official/acme.sh
All my personal servers are either directly provided by or proxied through an instance of Caddy (https://caddyserver.com/), which has an Letsencrypt client built in.
I’m using Cloudfront, nginx and Apache atm between all the diff servers (should really clean up my stuff) as well as postfix for my mail. So while it won’t fix my needs I will keep it in mind for when I need to throw together a server for others.
If you are looking for ease of use and stability, there is an alternative solution. Issue self-signed certificates on your server, and then use a service like CloudFlare that handles the front-user certificates. The whole traffic is encrypted now, but you don't need a CA authority to deal with.
The people who are benefiting from Let's Encrypt are the least technical who can't even SSH into a server. They get the certificate from their hosting provider automagically. It's reasonable that LE and the people writing software for it will focus on them rather than another developer doing a manual integration.
The people who are benefiting from Let's Encrypt are the least technical who can't even SSH into a server. They get the certificate from their hosting provider automagically. It's reasonable that LE and the people writing software for it will focus on them rather than another developer doing a manual integration.
That's a valid point.
Completely relevant.
There's a variant where you don't need CF: compile/install Caddyserver on your box, let it handle the LE certificates by itself, and use it as a reverse proxy to deliver the request to your current webserver.
Caddy being a stand alone binary (it's even possible statically link GLIBC), you can even work yourself around an old distro and dependencies nightmares.
Caddy's config file is on par with Nginx in terms of simplicity, the learning phase is quiet small, I would even say nonexistent for someone with a technical background as OP.
There's a variant where you don't need CF: compile/install Caddyserver on your box, let it handle the LE certificates by itself, and use it as a reverse proxy to deliver the request to your current webserver.
Caddy being a stand alone binary (it's even possible statically link GLIBC), you can even work yourself around an old distro and dependencies nightmares.
Caddy's config file is on par with Nginx in terms of simplicity, the learning phase is quiet small, I would even say nonexistent for someone with a technical background as OP.
Just let it get the cert (DNS or HTTP challenge) and then restart/reload the web server. Do not let LE rewrite your server configuration. This was very error-prone before and likely still is. People structure their Apache, Nginx, HAProxy, etc. configuration files in very different ways and it's nearly impossible for the certbot to work correctly in all cases. I'm not sure if this was also the case for the article author, the article contains too little technical information to tell it.
That's the one thing that actually worked once I managed to get a chrooted copy of the latest certbot running on a copy of the config. But that took quite a bit of fiddling to set up and it should never have come to that in the first place.
One more than "good", actually phenomenal thing about LetsEncrypt from my experience is their community. It is pure gold. People are super helpful in the https://community.letsencrypt.org/ forum. I had issues on setting it up and lots of people got involved and helped me out, the responsiveness, the cordiality, that is high quality "support" that no money can pay.
I usually enjoy articles by Jacques, but this one has a different tone and I don't think I understand its point.
Saying things like
> hey should just shut up, sit in a corner and do their bloody job. seems unnecessarily blunt.
And if you don't like Let's Encrypt, don't use it. It's being compared to convenient paid services, so why not use those instead if they end up being more convenient for your usecase?
Saying things like
> hey should just shut up, sit in a corner and do their bloody job. seems unnecessarily blunt.
And if you don't like Let's Encrypt, don't use it. It's being compared to convenient paid services, so why not use those instead if they end up being more convenient for your usecase?
I don't like the 'if you don't like free service X that you've been using for Y years then don't use it'. There is a cost to switch, and that cost to switch alone outweighs all the benefit of using LE in the first place. That's not what I was sold when I went into it, and even 'free' users are entitled to being treated with some care.
On the assumption that you are using a free gmail account: would you agree that Google could shut down the free tier of gmail tomorrow unless we all move to Chrome and it would be our problem to deal with the consequences?
On the assumption that you are using a free gmail account: would you agree that Google could shut down the free tier of gmail tomorrow unless we all move to Chrome and it would be our problem to deal with the consequences?
> On the assumption that you are using a free gmail account: would you agree that Google could shut down the free tier of gmail tomorrow unless we all move to Chrome and it would be our problem to deal with the consequences?
Yes.
Yes.
The latter is a clear cut violation of antitrust law. How is this similar?
[deleted]
> This is unacceptable for a project that tries to make something easier and where convenience, not financial considerations are a big driver in adoption.
Financial considerations are the big driver for letsencrypt. Not sure why OP think otherwise ? Easier and convenience is secondary and served by certbot/acme.
Financial considerations are the big driver for letsencrypt. Not sure why OP think otherwise ? Easier and convenience is secondary and served by certbot/acme.
> Somewhere someone has decided that Letsencrypt should auto-update even if absolutely nothing had been changed on the system that it was running on.
This is why I always run certbot inside of a docker container.
> The machine that this is all running on will need a much more recent version of its OS.
Again, same thing: use Certbot inside Docker.
This is why I always run certbot inside of a docker container.
> The machine that this is all running on will need a much more recent version of its OS.
Again, same thing: use Certbot inside Docker.
Exactly.
https://hub.docker.com/r/certbot/certbot
certbot has an official docker image. By using the official docker image you don't manage the updates beyond having to call `docker pull certbot/certbot`. I just integrate that into the same cron task that triggers the renewals/checks. Could not be easier and more fault tolerant.
https://hub.docker.com/r/certbot/certbot
certbot has an official docker image. By using the official docker image you don't manage the updates beyond having to call `docker pull certbot/certbot`. I just integrate that into the same cron task that triggers the renewals/checks. Could not be easier and more fault tolerant.
Oh. I thought this was a Python library issues and had nothing to do with LE. I too have had two occasions where I couldn't get certs with MIAB after upgrading. The first was some weird library that was missing after upgrading from like 10 versions behind on MIAB. IIRC it was a core Python library that had changed. The other was something that had to do with there being multiple "instances" of certificate chains.
Basically all it made me do is make sure I was monitoring the administrator@ account for the primary MIAB domain to make sure I'm updating it regularly.
Also I think MIAB is one of the few pieces of software that can safely be upgraded regularly without pain. I really trust Dr. Tauberer's project.
Basically all it made me do is make sure I was monitoring the administrator@ account for the primary MIAB domain to make sure I'm updating it regularly.
Also I think MIAB is one of the few pieces of software that can safely be upgraded regularly without pain. I really trust Dr. Tauberer's project.
To add to the various anecdotes already shared: I've been running this for several years now with zero attention:
root@eighteendotfour:~# crontab -l|grep getssl
23 5 * * * /root/bin/getssl -u -a >> /root/bin/getssl.log 2>&1
root@eighteendotfour:~#
root@eighteendotfour:~# head -3 /root/bin/getssl
#!/usr/bin/env bash
# ---------------------------------------------------------------------------
# getssl - Obtain SSL certificates from the letsencrypt.org ACME serverTwo days ago i had a big problem with haproxy..well letsencrypt to be precise, getting a new cert was not possible but renew one was. It was the additional Cipher-suite setting in Haproxy ;(
It's a false promise for sure. Overly fragile and can bring down a system with little warning. Have 20 instances and if it fails 9 out of 10 times it's from LE.
[deleted]
> The machine that this is all running on will need a much more recent version of its OS.
Does anyone have any info on this problem? I have letsencrypt running just about everywhere (including my ubiquity edge router). It seems to "just work". However, I use the letsencrypt-nginx plugin to handle the certificates for me.
Does anyone have any info on this problem? I have letsencrypt running just about everywhere (including my ubiquity edge router). It seems to "just work". However, I use the letsencrypt-nginx plugin to handle the certificates for me.
I have run into many of these same issues. Certbot auto upgrade issues, v1 protocol deprecation, and time wasted fixing it.
Still, Letsencrypt is an amazing project. I would never want to go back the old way of buying / renewing / installing certificates. That was a true pain!
Still, Letsencrypt is an amazing project. I would never want to go back the old way of buying / renewing / installing certificates. That was a true pain!
I highly recommend https://github.com/acmesh-official/acme.sh to solve the annoyance of the terribly bloated and poorly built Certbot application.
Am not a sysadmin. However after struggling with LE for a while (esp. failed renewals) I just got an origin CA certificate from Cloudflare and stuck it in Nginx.
Haven't had to think about this again for years now.
Haven't had to think about this again for years now.
I've never had any of these issues, so I'm surprised by how spectacularly the author's setup broke. I can understand some Python library failing during the auto update process needing a rollback, but needing a complete reimage to fix the situation? How does that even happen?
> new protocol for the letsencrypt challenges was released, announced in some obscure corner of the web and bit by bit the old method of issuing new certificates was deprecated. You would not have known this unless you interacted with Letsencrypt on a daily basis, but like most of these things their whole value lies in NOT having to interact with them all the time.
Let's Encrypt also sent out emails to people still using the old method at least two months before the old method was phased out. You have to provide an email address when setting up LE exactly for this reason. The only explanation I can think of for why the author wouldn't have received an email is that they did not execute a renew check for their certificate in the three months leading up to the change of service, which is going straight against LE's guidelines (you should check more often, exactly for this reason). This is the
> which is odd because after all the situation hasn’t changed and if it worked last time it should still work this time
The situation obviously _has changed_ otherwise the venv wouldn't have been updated. Libraries get bugfixes even if those don't apply to you specifically; deal with it. You're not the only customer here.
> I understand that Letsencrypt needs to stay with the times. But the Letsencrypt folks should also understand that in the greater scheme of things certificates are not the most important thing in life and that causing your end users this level of grief is in the longer term not in the projects’ best interest.
Web security is not a "set and forget" thing. It's not a toggle button from "insecure" to "secure". If you want a zero-maintenance certificate solution, get a managed server and get someone else to handle the annoying stuff for you.
> The machine that this is all running on will need a much more recent version of its OS. That means I’m going to have to migrate a whole bunch of data to another machine in order to be able to do this without downtime.
This is unsurprising coming from someone who wrote a blog about why they don't like to update. However, the assumption that the entire host OS needs to update just to use a new piece of software is simply wrong. Things like Docker and LXC make it trivial to run newer software without upgrading the entire system. It also forces the software into a contained space to prevent the catastrophic failure the author had to deal with before. From what I can tell from a Shodan check for this domain, the author still seems to be running Ubuntu 14 (which went EOL last year). That probably explains why the author is afraid that they will need to upgrade their server. However, if you're running on a software stack whose LTS support has already been dropped, you're just postponing the inevitable by not planning an upgrade plan already. Blaming Let's Encrypt for potentially not supporting a platform nobody else supports is a silly excuse.
I agree that Let's Encrypt has its issues. Sure, their client can use some thinning, their decisions are sometimes confusing, but the normal situation is not nearly as bad as the author describes here. If you're bothered by the official client updating things, use a different client; there's plenty of them.
> new protocol for the letsencrypt challenges was released, announced in some obscure corner of the web and bit by bit the old method of issuing new certificates was deprecated. You would not have known this unless you interacted with Letsencrypt on a daily basis, but like most of these things their whole value lies in NOT having to interact with them all the time.
Let's Encrypt also sent out emails to people still using the old method at least two months before the old method was phased out. You have to provide an email address when setting up LE exactly for this reason. The only explanation I can think of for why the author wouldn't have received an email is that they did not execute a renew check for their certificate in the three months leading up to the change of service, which is going straight against LE's guidelines (you should check more often, exactly for this reason). This is the
> which is odd because after all the situation hasn’t changed and if it worked last time it should still work this time
The situation obviously _has changed_ otherwise the venv wouldn't have been updated. Libraries get bugfixes even if those don't apply to you specifically; deal with it. You're not the only customer here.
> I understand that Letsencrypt needs to stay with the times. But the Letsencrypt folks should also understand that in the greater scheme of things certificates are not the most important thing in life and that causing your end users this level of grief is in the longer term not in the projects’ best interest.
Web security is not a "set and forget" thing. It's not a toggle button from "insecure" to "secure". If you want a zero-maintenance certificate solution, get a managed server and get someone else to handle the annoying stuff for you.
> The machine that this is all running on will need a much more recent version of its OS. That means I’m going to have to migrate a whole bunch of data to another machine in order to be able to do this without downtime.
This is unsurprising coming from someone who wrote a blog about why they don't like to update. However, the assumption that the entire host OS needs to update just to use a new piece of software is simply wrong. Things like Docker and LXC make it trivial to run newer software without upgrading the entire system. It also forces the software into a contained space to prevent the catastrophic failure the author had to deal with before. From what I can tell from a Shodan check for this domain, the author still seems to be running Ubuntu 14 (which went EOL last year). That probably explains why the author is afraid that they will need to upgrade their server. However, if you're running on a software stack whose LTS support has already been dropped, you're just postponing the inevitable by not planning an upgrade plan already. Blaming Let's Encrypt for potentially not supporting a platform nobody else supports is a silly excuse.
I agree that Let's Encrypt has its issues. Sure, their client can use some thinning, their decisions are sometimes confusing, but the normal situation is not nearly as bad as the author describes here. If you're bothered by the official client updating things, use a different client; there's plenty of them.
The machine was installed two years ago with what was then the current version of Mailinabox, which in turn apparently relied on Ubuntu 14.04.
So, yes, that is outdated now but it wasn't as if there was much choice at the time and two years seems a bit short - maybe I'm wrong about that - on the expected lifespan of a server. But setting up a mailserver, migrating all of the data back on and then testing it is a boatload of work and I have plenty to do as it is.
And yes, I did eventually manage to get it to work but in a pretty roundabout way and that should just simply not have been necessary unless there was a huge security issue with the way LE was working before. I'm not aware of any such issues.
Essentially you are arguing that something like Mailinabox can not or should not exist because it falls somewhere between 'managed hosting' and something that can be set up for a few years in order to serve a purpose.
That's not a very encouraging future, maintaining basic internet infra such as mail, web and DNS servers should not be so hard that you can only do it if you want to pay third parties big bucks to do so.
So, yes, that is outdated now but it wasn't as if there was much choice at the time and two years seems a bit short - maybe I'm wrong about that - on the expected lifespan of a server. But setting up a mailserver, migrating all of the data back on and then testing it is a boatload of work and I have plenty to do as it is.
And yes, I did eventually manage to get it to work but in a pretty roundabout way and that should just simply not have been necessary unless there was a huge security issue with the way LE was working before. I'm not aware of any such issues.
Essentially you are arguing that something like Mailinabox can not or should not exist because it falls somewhere between 'managed hosting' and something that can be set up for a few years in order to serve a purpose.
That's not a very encouraging future, maintaining basic internet infra such as mail, web and DNS servers should not be so hard that you can only do it if you want to pay third parties big bucks to do so.
Mailinabox's use of such an old OS is the exact reason I skipped it and went for Mailcow instead. It also placed very strange limitations on the stuff you could do with the system once it was set up and running according to the documentation, presumably to make sure the project maintainers had only a few types of configurations to support.
I have nothing against the concept of Mailinabox but when you create a "hands-off" project like that, your project should also include stable major version upgrades to LTS versions. From what I can tell, MIAB switched from 14.04 directly to 18.04 (skipping 16.04). Their goals and ideas are noble, but I don't think the project is very reliable if they skip major OS releases just like that (and are very late to the party when it comes to support for newer versions, releasing a version compatible with 18.04 only months before 14.04 support was dropped entirely).
I appreciate the enormous amount of work it takes to achieve something like this, but marketing your product as something "you don’t need to be a technology expert to set up" brings with it certain expectations (or even, responsibilities). Your issue with an outdated LE client should have never happened if the project had achieved its goals, for example.
A modern setup with MIAB based on Ubuntu 18.04 will have OS support until 2028, so the situation has improved; Ubuntu has switched to 10 years of support instead of 5.
> Essentially you are arguing that something like Mailinabox can not or should not exist because it falls somewhere between 'managed hosting' and something that can be set up for a few years in order to serve a purpose.
The problem is that software doesn't work like that, unfortunately. Bugs and security issues are found and need to be patched. Volunteers don't want to fix five years of releases of their software and for companies it doesn't make financial sense to do such work for free. A Windows server box with regular reboots for security updates will work fine for a bunch of years, but that guarantee is hard to make when your software runs on volunteers and occasional corporate sponsors.
> That's not a very encouraging future, maintaining basic internet infra such as mail, web and DNS servers should not be so hard that you can only do it if you want to pay third parties big bucks to do so.
There's a reason companies have been providing paid services for this since the dawn of the web. Hosting software on the internet isn't maintenance free and it never was. If you desire a setup that keeps working with minimum maintenance without the risks that come with volunteer work, you'll have to pick a stable OS (like Ubuntu or its competitors) and software stack from the LTS repositories and configure it. You can apt-upgrade anything from the official Debian repositories without reconfiguring anything. You can upgrade and It Just Works as long as you stick to the packages provided by the people maintaining your OS. This works until your OS reaches end-of-life, which is the next time you should do intensive system admin effort. Luckily, this is announced long beforehand, so you can plan accordingly.
This still means you'll run into trouble when (free) external parties decide to drop your OS because of the growing amount of software stacks and configurations they have to support over the years. However, if you practice running external stuff in a contained environment (with tools like Docker) you'll get to keep the best of both worlds for as long as possible. Alternatively, you can use setups that don't hide behind magical one-command-fixes-all setups, like manually entering certificates instead of relying on Some Cloud System to do it for you.
If all that is too much for you, which I can completely understand, then maybe running your own servers might not be for you. The premise of bringing self-hosted, federated services to the every-day consumer is nice, but I'm afraid it'll always be little more than a dream. The modern web is either part of the Cloud (TM) or a bunch of projects glued together by random chance and best effort; running your own stuff unfortunately requires either money to be paid or effort to be made.
I have nothing against the concept of Mailinabox but when you create a "hands-off" project like that, your project should also include stable major version upgrades to LTS versions. From what I can tell, MIAB switched from 14.04 directly to 18.04 (skipping 16.04). Their goals and ideas are noble, but I don't think the project is very reliable if they skip major OS releases just like that (and are very late to the party when it comes to support for newer versions, releasing a version compatible with 18.04 only months before 14.04 support was dropped entirely).
I appreciate the enormous amount of work it takes to achieve something like this, but marketing your product as something "you don’t need to be a technology expert to set up" brings with it certain expectations (or even, responsibilities). Your issue with an outdated LE client should have never happened if the project had achieved its goals, for example.
A modern setup with MIAB based on Ubuntu 18.04 will have OS support until 2028, so the situation has improved; Ubuntu has switched to 10 years of support instead of 5.
> Essentially you are arguing that something like Mailinabox can not or should not exist because it falls somewhere between 'managed hosting' and something that can be set up for a few years in order to serve a purpose.
The problem is that software doesn't work like that, unfortunately. Bugs and security issues are found and need to be patched. Volunteers don't want to fix five years of releases of their software and for companies it doesn't make financial sense to do such work for free. A Windows server box with regular reboots for security updates will work fine for a bunch of years, but that guarantee is hard to make when your software runs on volunteers and occasional corporate sponsors.
> That's not a very encouraging future, maintaining basic internet infra such as mail, web and DNS servers should not be so hard that you can only do it if you want to pay third parties big bucks to do so.
There's a reason companies have been providing paid services for this since the dawn of the web. Hosting software on the internet isn't maintenance free and it never was. If you desire a setup that keeps working with minimum maintenance without the risks that come with volunteer work, you'll have to pick a stable OS (like Ubuntu or its competitors) and software stack from the LTS repositories and configure it. You can apt-upgrade anything from the official Debian repositories without reconfiguring anything. You can upgrade and It Just Works as long as you stick to the packages provided by the people maintaining your OS. This works until your OS reaches end-of-life, which is the next time you should do intensive system admin effort. Luckily, this is announced long beforehand, so you can plan accordingly.
This still means you'll run into trouble when (free) external parties decide to drop your OS because of the growing amount of software stacks and configurations they have to support over the years. However, if you practice running external stuff in a contained environment (with tools like Docker) you'll get to keep the best of both worlds for as long as possible. Alternatively, you can use setups that don't hide behind magical one-command-fixes-all setups, like manually entering certificates instead of relying on Some Cloud System to do it for you.
If all that is too much for you, which I can completely understand, then maybe running your own servers might not be for you. The premise of bringing self-hosted, federated services to the every-day consumer is nice, but I'm afraid it'll always be little more than a dream. The modern web is either part of the Cloud (TM) or a bunch of projects glued together by random chance and best effort; running your own stuff unfortunately requires either money to be paid or effort to be made.
I've self hosted web/mail/dns since roughly the beginning of the www and I find it quite frustrating that all these so called improvements actually make it only harder and harder to keep this up. Besides Google putting ever more pressure on independently operated mailservers to join the borg by randomly rejecting perfectly valid email.
I will also freely admit to having found that there are more important things in life than endless fiddling with system configurations and low level stuff. It used to fascinate me but the days when installing an operating system on some fresh piece of hardware (or a virtual) excited me are long gone. When you've done it 100's of times it is no longer as nice as when you did it the first few times.
I will also freely admit to having found that there are more important things in life than endless fiddling with system configurations and low level stuff. It used to fascinate me but the days when installing an operating system on some fresh piece of hardware (or a virtual) excited me are long gone. When you've done it 100's of times it is no longer as nice as when you did it the first few times.
The big fact of the matter is that https is usually unnecessary for most websites. So this entire effort is a massive waste. Why do I need https on my blog? What is the great risk I am exposing myself to by hosting a plain html page? Nothing.
This is basically Google bullying the internet for no reason. So of course lets encrypt will demand attention. It is the central artifact of your website now, since you took the time to set it up without any use or benefit. So deal with it.
This is basically Google bullying the internet for no reason. So of course lets encrypt will demand attention. It is the central artifact of your website now, since you took the time to set it up without any use or benefit. So deal with it.
Considering the entire post is whining about a free service, I don't get it.
Who is forcing you to use letsencrypt? If you don't like it, buy a cert and install it.
If the certbot is too complicated or "pulls in too much cruft" then use a different client. https://github.com/acmesh-official/acme.sh for example.
Who is forcing you to use letsencrypt? If you don't like it, buy a cert and install it.
If the certbot is too complicated or "pulls in too much cruft" then use a different client. https://github.com/acmesh-official/acme.sh for example.
Ahh i see...the old black and white worldview.
It's 2020 and it's time for you to learn about containers and docker.
Can people please stop to use "The Good, the Bad and the Ugly" every time they write some list of thoughts about some topic? It's maybe a little less creative and unique than you think.
This is what Nix/NixOS would solve..
m00dy(2)
You should have used certbot shipped by your Linux distribution.
I can confirm this experience. Back in the day, when startssl or wosign worked, I spent few minutes per year to issue a certificate and called it a day. I think that wosign even issued 3-year certificate, but I might be wrong about it. With letsencrypt I spent hours after hours trying to understand how does it work and set it up properly.
But moaning does not do any good, we're not getting back in those good old days, so it's better to adapt, I guess.
One thing that helps with broken letsencrypt setup is to monitor e-mail that was supplied. Letsencrypt will send angry mails when your certificate is not updated.
But moaning does not do any good, we're not getting back in those good old days, so it's better to adapt, I guess.
One thing that helps with broken letsencrypt setup is to monitor e-mail that was supplied. Letsencrypt will send angry mails when your certificate is not updated.
I think something like Certera, but designed more like a pull through proxy is the solution. So instead of making queries directly to LE you use the local proxy instead and it tracks issuances, expiry dates, CAA expectations, etc. in one centralized dashboard.
* Let's Encrypt - the actual CA
* certbot - probably the mentioned python software
* ACME - the protocol, which had a change in the supported challenges
Also there are alternatives for all three points. There are other CAs implementing ACME, there are other ACME clients and there are still CAs supporting the old 'send an email and click a link' domain validation.