Why Certificate Lifecycle Automation Matters(hezmatt.org)
hezmatt.org
Why Certificate Lifecycle Automation Matters
https://www.hezmatt.org/~mpalmer/blog/2024/01/30/why-certificate-automation-matters.html
30 comments
I do wish the ACME ecosystem was as fire and forget as it's always aimed to be. It feels like an annual occurrence where I need to fix... Something misfiring with acme cert issuance and I've tried certbot, acme.sh, Lego and they've all had issues. Like maybe when first issued the tool decided to use ZeroSSL but on reissue decides to use Let's Encrypt and fails because one requires an email and the other doesn't. Or the verification method gets deprecated and requires manual software updates. Or the tool decided to rename it's environment variables and auto updated so now it can't find it's DNS auth keys.
I'm using certbot and acme/autocert, and it has been a few years since I last fiddled with this (even though I have CD happening !).
I was a bit mad at certbot for having chosen snap as its recommended installation platform, but my solution to use pypi + CD seems to be quite stable (famous last words).
The go services I run with autocert have had no manual intervention ever since the ACMEv2 move.
I was a bit mad at certbot for having chosen snap as its recommended installation platform, but my solution to use pypi + CD seems to be quite stable (famous last words).
The go services I run with autocert have had no manual intervention ever since the ACMEv2 move.
EFF offers certbot in a Docker image, which is what I've been using for years:
https://eff-certbot.readthedocs.io/en/latest/install.html#al...
https://eff-certbot.readthedocs.io/en/latest/install.html#al...
This is why I use .onion urls instead of WebPKI.
Interesting. I was using acme.sh and switched to lego, largely because I manage my servers with Puppet and acme.sh’s config format didn’t lend itself well to config management.
I renew certs for several boring things like nginx instances, but also things like my printer’s web portal, a Synology NAS, and a Unifi console that require bespoke commands to reload the cert.
I’ve got monitors that fire if any of my certs drop below 27 days before expiry, and over the past ~2 years, 100% of the problems have been with the custom hooks. I’ll log into the server and find that it did reissue the cert, but the command to make the Synology ingest the new cert failed for some arcane reason. The servers where it’s just running systemctl restart nginx or similar don’t fail.
I renew certs for several boring things like nginx instances, but also things like my printer’s web portal, a Synology NAS, and a Unifi console that require bespoke commands to reload the cert.
I’ve got monitors that fire if any of my certs drop below 27 days before expiry, and over the past ~2 years, 100% of the problems have been with the custom hooks. I’ll log into the server and find that it did reissue the cert, but the command to make the Synology ingest the new cert failed for some arcane reason. The servers where it’s just running systemctl restart nginx or similar don’t fail.
Huh, the environment variable thing was specifically aimed at acme.sh which rather arbitrarily changed the config value from ACMEDNS_UPDATE_URL to ACMEDNS_BASE_URL, never acknowledged this in a changelog and then silently failed after an automatic upgrade as recommended by the default install:
https://github.com/acmesh-official/acme.sh/commit/2ce145f359...
It's also cleared out my .account.conf files when run on the suggested cron.
I've started using updown which also monitors my TLS certs simply because I no longer trust the process to work as documented.
https://github.com/acmesh-official/acme.sh/commit/2ce145f359...
It's also cleared out my .account.conf files when run on the suggested cron.
I've started using updown which also monitors my TLS certs simply because I no longer trust the process to work as documented.
Ah, I plausibly didn’t hit this because I was permanently on a fork of the acme.sh codebase, after I opened a PR to support Route53 subzones and it was ignored.
A reminder that if you an internal-only server where the typical http-01 verification connection method will not work, one can use dns-01 by using DNS aliasing/CNAME, especially if you cannot easily/dynamically update DNS records:
* https://dan.langille.org/2019/02/01/acme-domain-alias-mode/
* https://www.eff.org/deeplinks/2018/02/technical-deep-dive-se...
So if you want a cert for www.internal.example.com, you will first have do a one-time change to have a _acme-challenge.www.internal… CNAME created to point to any other (sub-)domain where you can easily update things dynamically, e.g., www-internal.example-dnsapi.com.
When request the cert for "www.internal…", LE/ACME will look up the corresponding _acme-challenge record, and go to "_acme-challenge.www-internal.example-dnsapi.com. The nonce token will be there in the 'final' destination following the CNAME in a TXT record, which shows LE/ACME that you control the DNS chain.
To do the DNS updating, you can use a CLI/Python library like Lexicon, which supports dozens of APIs:
* https://github.com/AnalogJ/lexicon
* https://dan.langille.org/2019/02/01/acme-domain-alias-mode/
* https://www.eff.org/deeplinks/2018/02/technical-deep-dive-se...
So if you want a cert for www.internal.example.com, you will first have do a one-time change to have a _acme-challenge.www.internal… CNAME created to point to any other (sub-)domain where you can easily update things dynamically, e.g., www-internal.example-dnsapi.com.
When request the cert for "www.internal…", LE/ACME will look up the corresponding _acme-challenge record, and go to "_acme-challenge.www-internal.example-dnsapi.com. The nonce token will be there in the 'final' destination following the CNAME in a TXT record, which shows LE/ACME that you control the DNS chain.
To do the DNS updating, you can use a CLI/Python library like Lexicon, which supports dozens of APIs:
* https://github.com/AnalogJ/lexicon
I own example.com and actually created NS records for internal.example.com, to manage that entire zone at an entirely different provider. I did that because the original provider had terrible API access for Caddy to use. The new one has better support, and a breach there does not pwn the root domain.
I was happy with that setup, but wasn’t aware one could delegate to an entirely different domain using ACME primitives. Makes sense though, it’s like the NS approach but simpler, and allows using some throwaway domain.
Thanks!
tls-alpn-01 woks really well as well. You don't have to mess with DNS at all, or have an HTTP port open; you just listen on HTTPS.
Yeah, except for the the internal-only server part where the hostname may not even be accessible to the public Internet (certainly not as an A(AAA) record).
And given the number of internal applications many large/r organizations have, giving all those internal dev teams update access to the main corporate domain is not something I'd really feel comfortable doing.
But being able to give API access to (e.g.) foo.dnsauth.example.com is something I have done: main domain needs to be touched once for the CNAME, and the sub-domain isn't critical for day-to-day operations of a company.
And given the number of internal applications many large/r organizations have, giving all those internal dev teams update access to the main corporate domain is not something I'd really feel comfortable doing.
But being able to give API access to (e.g.) foo.dnsauth.example.com is something I have done: main domain needs to be touched once for the CNAME, and the sub-domain isn't critical for day-to-day operations of a company.
The post I was replying to used www.internal.example.com , which looked like a public name. Internal-only names should not look like subdomains of public DNS names.
But yes, if you want a certificate for a name that isn't accessible externally and can't be made accessible externally, dns-01 works.
But yes, if you want a certificate for a name that isn't accessible externally and can't be made accessible externally, dns-01 works.
> Internal-only names should not look like subdomains of public DNS names.
Since when? I, and many other people I know, have been doing that for decades.
If I (the company) own example.com, I can choose to have some sub-domains internal-only, some external-only, some DNS records resolve to the same IP value regardless of in/outside, and other DNS records resolve to different IP values depending on in/outside.
* https://en.wikipedia.org/wiki/Split-horizon_DNS
Since when? I, and many other people I know, have been doing that for decades.
If I (the company) own example.com, I can choose to have some sub-domains internal-only, some external-only, some DNS records resolve to the same IP value regardless of in/outside, and other DNS records resolve to different IP values depending on in/outside.
* https://en.wikipedia.org/wiki/Split-horizon_DNS
However, internal domains should have their top domain externally registered and controlled... so ultimately there's no difference because you're going to need at least a dummy zone file to satisfy registrar.
> However, internal domains should have their top domain externally registered and controlled
Not necessarily. `.internal` exists for that purpose, and people have historically used `.lan` for similar purposes. There's no particular reason internal names need to have resolvable DNS names of any kind, if they're not going to be externally accessible, unless you're trying to do something like dns-01.
Not necessarily. `.internal` exists for that purpose, and people have historically used `.lan` for similar purposes. There's no particular reason internal names need to have resolvable DNS names of any kind, if they're not going to be externally accessible, unless you're trying to do something like dns-01.
> Not necessarily. `.internal` exists for that purpose …
.internal is being considered for that purpose:
> IANA has performed an evaluation to determine a suitable string to be reserved for the purpose of a top-level domain that may be used for internal or private use applications. The string “INTERNAL” has been identified as appropriate for this purpose.
> In accordance with the process described by ICANN, this assessment will be put for a public comment proceeding, followed by review and potential adoption by ICANN.
* https://itp.cdn.icann.org/en/files/root-system/identificatio...
* https://www.theregister.com/2024/01/29/icann_internal_tld/
There is now not (yet?), nor has there ever been, any label officially set aside for that purpose.
> … and people have historically used `.lan` for similar purposes.
People have also used .dev for their development environment, and we all saw how well that worked out:
* https://en.wikipedia.org/wiki/.dev
> There's no particular reason internal names need to have resolvable DNS names of any kind […]
Sure there are: for putting into a certificate which can be linked to a public certificate authority (CA) so that users don't have to install a custom/private CA just to use internal services. I'm at a place where the dumbass team before me used an internal-only TLD (lan/local/internal), and now we (the new team) can't issue certs for things internally.
.internal is being considered for that purpose:
> IANA has performed an evaluation to determine a suitable string to be reserved for the purpose of a top-level domain that may be used for internal or private use applications. The string “INTERNAL” has been identified as appropriate for this purpose.
> In accordance with the process described by ICANN, this assessment will be put for a public comment proceeding, followed by review and potential adoption by ICANN.
* https://itp.cdn.icann.org/en/files/root-system/identificatio...
* https://www.theregister.com/2024/01/29/icann_internal_tld/
There is now not (yet?), nor has there ever been, any label officially set aside for that purpose.
> … and people have historically used `.lan` for similar purposes.
People have also used .dev for their development environment, and we all saw how well that worked out:
* https://en.wikipedia.org/wiki/.dev
> There's no particular reason internal names need to have resolvable DNS names of any kind […]
Sure there are: for putting into a certificate which can be linked to a public certificate authority (CA) so that users don't have to install a custom/private CA just to use internal services. I'm at a place where the dumbass team before me used an internal-only TLD (lan/local/internal), and now we (the new team) can't issue certs for things internally.
I did specifically say "unless you're trying to do something like dns-01". If you want to issue public certificates for your internal domains, by all means use a public name for them.
In my experience, you can and will encounter a situation with conflicting resolvings of .internal unless the use is all properly scoped to broadcast and administrative zones (ha ha ha fat chance, though respect to anyone who really tries).
.lan, .home, and similar have the issue that they aren't actually reserved so they can conflict (see the land-grab of .dev that Google did after ICANN opened up TLDs).
The one way that has most stability (and is best recommendation I have seen) is to have a publicly reserved zone, which doesn't need to have anything other than NS records pointing to hostnames visible from your internal networks (I am always tempted to have essentially NS -> AAAA + PTR pointing at anycast internal v6 address).
Turns out it does not need to be separate domain from some other domain of yours, it can be let's say i.your-company-name.com. - I am partial to your-company-name.net. but that can be confusing for less computer crazy people.
.lan, .home, and similar have the issue that they aren't actually reserved so they can conflict (see the land-grab of .dev that Google did after ICANN opened up TLDs).
The one way that has most stability (and is best recommendation I have seen) is to have a publicly reserved zone, which doesn't need to have anything other than NS records pointing to hostnames visible from your internal networks (I am always tempted to have essentially NS -> AAAA + PTR pointing at anycast internal v6 address).
Turns out it does not need to be separate domain from some other domain of yours, it can be let's say i.your-company-name.com. - I am partial to your-company-name.net. but that can be confusing for less computer crazy people.
One advantage of the DNS01 approach is that you can also order certificates for standby instances behind a load balancer, all fronted by the same domain name. Since the standbys are not serving traffic, you can't get an HTTP01 challenge/response through, but you can still do so with DNS01 and get a certificate delivered to that machine, so that it's always ready to serve TLS in the event of a failover.
Can anyone with good experiences with a commercially-supported ACME stack recommend a vendor they’re happy with, where the stack was integrated into their own in-house certificate authority? I’m especially looking for vendors that have very good documentation and tracing introspection to let our own engineers who read the documentation rapidly troubleshoot the stack, and our own developers build robust integrations to the client to rotate the certificates.
I haven't used it (so I might be excluded from answering your question), but I think SmallStep may be relevant to you.
https://smallstep.com/docs/certificate-manager/acme/
At least they are doing a lot of "your own organizational PKI" tools and they care about ACME integration.
https://smallstep.com/docs/certificate-manager/acme/
At least they are doing a lot of "your own organizational PKI" tools and they care about ACME integration.
You should look at Venafi and their offerings. They have integrations that allow you to connect to inhouse CA for certificate issuing. You can also add your own integrations when provisioning certificates to machines or choose from some of the many existing integrations.
Disclosure: I work for Venafi
Disclosure: I work for Venafi
I get why it is important, but certs can be a tremendous pain to deal with when you're in a niche industry and reliant on the only vendor in town, and the consequences of a broken cert were minimal.
The whole cert regime is wrongheaded. Issuing certs is vulnerable to exploits, the methods are clunky and problematic, certs expire so quickly that stuff is breaking more often than security issues could ever come up, there's no consideration to private networks, we still don't have content integrity without encryption, content proxy caching is dead, CAs can do what they want (there's too many CAs, cert logs aren't required to be validated in real time, you can't force a CA not to do something shady under a government gag order, etc), and there's a fundamental conflict between the owner of a domain and the issuer of a cert and the owner of a cert and the owner of some IP space or DNS server records.
But because it's this giant behemoth involving multiple industries, technological components, and vested interests, it will never change, save to implement bandaid after bandaid after hack after kludge. Literally the only thing that changes it is when someone with enough power (a market-dominant browser) forces it and everyone has to comply, or face creating a total shit-show due to lack of conformance. And we're all hopelessly dependent on it so it's not like we can do anything about it. The security of the web is a Kafkaesque nightmare.
But because it's this giant behemoth involving multiple industries, technological components, and vested interests, it will never change, save to implement bandaid after bandaid after hack after kludge. Literally the only thing that changes it is when someone with enough power (a market-dominant browser) forces it and everyone has to comply, or face creating a total shit-show due to lack of conformance. And we're all hopelessly dependent on it so it's not like we can do anything about it. The security of the web is a Kafkaesque nightmare.
Here's the thing though: the alternatives are also god-awful, and in part it is because the people who make them wind up being very stupid about their choices - i.e. if you want a distributed trust environment fine, but your first, primary and most important use-cases are still going to be government, and corporate, and specifically financial.
If your "libre" plan does not involve a way to bring those entities onboard and direct priority to servicing their needs, then no one is going to use it because we all actually live in the real world where those are the things we need security in communicating with.
If your "libre" plan does not involve a way to bring those entities onboard and direct priority to servicing their needs, then no one is going to use it because we all actually live in the real world where those are the things we need security in communicating with.
Agreed 100%.
I would prefer a "kinda federated" model where the owner of a domain registers a private key with a Domain Registrar, and CAs send CSRs to Domain Registrars to validate, and when the CA issues a cert, the browser can validate the registrar signed it (and who the domain's registrar is). That avoids all 9 unique attacks on CA validation, and more importantly, actually cryptographically verifies if the CA should issue a cert, which does not happen at all today.
I would prefer a "kinda federated" model where the owner of a domain registers a private key with a Domain Registrar, and CAs send CSRs to Domain Registrars to validate, and when the CA issues a cert, the browser can validate the registrar signed it (and who the domain's registrar is). That avoids all 9 unique attacks on CA validation, and more importantly, actually cryptographically verifies if the CA should issue a cert, which does not happen at all today.
[deleted]
More revocations != more customers losing keys.
It may just as well mean more capable audits to detect lost/compromised keys.
The article also says nothing of certificate validity periods, which are critical to their lifecycle and analysis.
The conclusion that humans degrade information security is true but ultimately removed from the reality of operating a business.
It may just as well mean more capable audits to detect lost/compromised keys.
The article also says nothing of certificate validity periods, which are critical to their lifecycle and analysis.
The conclusion that humans degrade information security is true but ultimately removed from the reality of operating a business.
Shameless plug: I've built a tool that automatically generates certs and uploads to destinations. https://github.com/poundifdef/certmaster
It uses Lego under the hood to issue certs, and then has custom connectors to upload to destinations. Right now those are email (indeed, more humans handling certificates), sftp, and hetzner load balancers. I'm working on adding the ability for it to automatically renew and re-upload when certs are 30 days from expiration.
It uses Lego under the hood to issue certs, and then has custom connectors to upload to destinations. Right now those are email (indeed, more humans handling certificates), sftp, and hetzner load balancers. I'm working on adding the ability for it to automatically renew and re-upload when certs are 30 days from expiration.
A 'competitor' to this would be GetSSL which is a pure-shell ACME client (plus OpenSSL and cURL) and can be executed on one host, but send verification tokens to remote systems (where you may not have cron access):
> Get certificates for remote servers - The tokens used to provide validation of domain ownership, and the certificates themselves can be automatically copied to remote servers (via ssh, sftp or ftp for tokens). The script doesn't need to run on the server itself. This can be useful if you don't have access to run such scripts on the server itself, as it's a shared server for example.
* https://github.com/srvrco/getssl
> Get certificates for remote servers - The tokens used to provide validation of domain ownership, and the certificates themselves can be automatically copied to remote servers (via ssh, sftp or ftp for tokens). The script doesn't need to run on the server itself. This can be useful if you don't have access to run such scripts on the server itself, as it's a shared server for example.
* https://github.com/srvrco/getssl