Banish OEM self-signed certs forever and roll your own private LetsEncrypt(arstechnica.com)
arstechnica.com
Banish OEM self-signed certs forever and roll your own private LetsEncrypt
https://arstechnica.com/information-technology/2024/03/banish-oem-self-signed-certs-forever-and-roll-your-own-private-letsencrypt/
34 comments
People don't want their internal infrastructure to end up in CT logs.
Can confirm. Stood up a test site with SSL on a subdomain recently. Within 2hrs of let's encrypt registration, I was getting hits on file from around the globe with no index defined yet.
DNS-01 + a wildcard SSL cert takes care of that unless you need third level domain names.
I think the last time I issued non-wildcard SSL certs for my domains was when I was still making the mistake of self-hosting email and the entire setup was a rickety mess.
I think the last time I issued non-wildcard SSL certs for my domains was when I was still making the mistake of self-hosting email and the entire setup was a rickety mess.
This is a frequently-touted solution, but now somebody compromising printer.lan.example.com can impersonate auth.lan.example.com, which isn't great either.
One approach is to put the wildcard certificate on a proxy. Clients connect to the proxy, and see a valid TLS certificate. The connection from the proxy can use whatever you'd like, including a self-signed certificate that you pin. You have the impersonation problem if the proxy is compromised, but that's a smaller attack surface than "printer", etc.
To be frank that's one of those things where it depends on how much you care about what gets compromised. For a purely internal homelab, the main use is to get software working; the HTTPS is only needed because modern browsers get upset about a compromised secure context if you don't, which leads to them disabling all sorts of random APIs. The actual security component is pretty much secondary for the usecase where you're dealing with devices that exist solely on an intranet - you're already gating off access by different means.
Personally, the way I'd solve the problem in the situation where it'd be a problem is to not put auth on the same domain; auth would go to auth.secondexample.com. That's generally speaking the better choice to begin with due to browser weirdness regarding domain isolation (which is also why you often see software with CDNs use completely different domain names; reddit images being hosted on redd.it for example but also YouTube thumbnails being on ytimg. If they didn't, a maliciously formed file could read out browser storage for different subdomains).
Personally, the way I'd solve the problem in the situation where it'd be a problem is to not put auth on the same domain; auth would go to auth.secondexample.com. That's generally speaking the better choice to begin with due to browser weirdness regarding domain isolation (which is also why you often see software with CDNs use completely different domain names; reddit images being hosted on redd.it for example but also YouTube thumbnails being on ytimg. If they didn't, a maliciously formed file could read out browser storage for different subdomains).
> somebody compromising printer.lan.example.com can impersonate auth.lan.example.com
Nope: one can setup ACLs so that the script that can update the records for printer.lan… cannot also do foo.lan… or any *.lan….
Nope: one can setup ACLs so that the script that can update the records for printer.lan… cannot also do foo.lan… or any *.lan….
If someone is on your network, you have to start worrying about traffic spoofing, unless you've got a seriously locked down switch (I do). But even so.
Poor security practice to use a wildcard. I completely agree there. But I can understand how wildcard certs work well for the average Joe who just wants internal stuff to be served over TLS without having to think about it too hard.
I work in security, so in my view, having the most secure internal network possible (think ZTA) is certainly up there.
Although I have come up with an alternative approach to achieve this.
Poor security practice to use a wildcard. I completely agree there. But I can understand how wildcard certs work well for the average Joe who just wants internal stuff to be served over TLS without having to think about it too hard.
I work in security, so in my view, having the most secure internal network possible (think ZTA) is certainly up there.
Although I have come up with an alternative approach to achieve this.
> DNS-01 + a wildcard SSL cert takes care of that unless you need third level domain names.
This will allow internal-only services to get LE/ACME certs, but those cert will still show up in the CT logs and be visible to the entire world.
Some folks do not want that visibility, and that's what an internal-only CA gets you.
This will allow internal-only services to get LE/ACME certs, but those cert will still show up in the CT logs and be visible to the entire world.
Some folks do not want that visibility, and that's what an internal-only CA gets you.
A wildcard cert is a single cert that is valid for any subdomain. The specific ones you use are not exposed to the CT, just the fact that there is a *.example.com was minted
Let's Encrypt supports wildcard certificates now.
Wildcard certificates are sub-optimal. I have a personal policy of not using them. They become very much like a "shared secret" which you have to properly manage. I know LANs don't need world class security, but as someone who works in security I like to play around with doing things right.
Why CT logs include plaintext domain? Isn't hashing it would be enough?
Make a LE wildcard cert
>Replace self-signed certs with a self-signed CA? [...], why not just use LE and get a real cert?
I went back and forth between self-signed CA vs LetsEncrypt.
I liked the self-signed CA root because you can create end-entity certificates for internal ip addresses such as 192.168.1.13 instead of domain names. (No domain name purchase and no public DNS records pointing to 192.168.1.13 required.) The public SSL services like Let'sEncrypt and ZeroSSL can't issue certs for RFC1918 ip addresses -- for obvious reasons.
On the other hand, I didn't want to import my custom CA root certificate into all 8+ devices around the house. (desktops+laptops+phones+tablets+etc.) So, Let'sEncrypt won in my case on that basis.
I went back and forth between self-signed CA vs LetsEncrypt.
I liked the self-signed CA root because you can create end-entity certificates for internal ip addresses such as 192.168.1.13 instead of domain names. (No domain name purchase and no public DNS records pointing to 192.168.1.13 required.) The public SSL services like Let'sEncrypt and ZeroSSL can't issue certs for RFC1918 ip addresses -- for obvious reasons.
On the other hand, I didn't want to import my custom CA root certificate into all 8+ devices around the house. (desktops+laptops+phones+tablets+etc.) So, Let'sEncrypt won in my case on that basis.
If you're going to the effort to run a CA, why not also run DNS? I say this as a guy who runs an internal CA and eventually just implemented LE due to all the problems with trusting internal CAs and getting that to work across all devices.
I am running my own private CA as well, powered by Hashicorp Vault, Ansible and Jenkins.
The Vault initialization and configuration is more or less manual (just a bunch of commands, I have them in my notes). From there I am using an ansible role based on the hasi_vault module [1] which is run by a Jenkins job every night, logging into each target system, renewing certs if needed and reloading services.
Has been working very well for about a year now. Of course, there's a little more technical context needed - my CA needs to be present on all systems interacting with it, and my CI needs to be able to log into each target system (SSH keypair + sudo user). This ties into the rest of my infrastructure, which is managed by Terraform and Ansible.
I might write up a small blog post about this if I find the time.
[1] https://docs.ansible.com/ansible/latest/collections/communit...
The Vault initialization and configuration is more or less manual (just a bunch of commands, I have them in my notes). From there I am using an ansible role based on the hasi_vault module [1] which is run by a Jenkins job every night, logging into each target system, renewing certs if needed and reloading services.
Has been working very well for about a year now. Of course, there's a little more technical context needed - my CA needs to be present on all systems interacting with it, and my CI needs to be able to log into each target system (SSH keypair + sudo user). This ties into the rest of my infrastructure, which is managed by Terraform and Ansible.
I might write up a small blog post about this if I find the time.
[1] https://docs.ansible.com/ansible/latest/collections/communit...
I believe these instructions give you a CA root that can sign certificates for the entire web (a man-in-the-middle CA). I'd be hard pressed to trust such a CA if it was created by anyone other than myself (or my employer, for work owned devices).
Best practice is to use name constraints to prevent the trust root from impersonating, for example, google.com if your LAN only uses .corp or similar.
Best practice is to use name constraints to prevent the trust root from impersonating, for example, google.com if your LAN only uses .corp or similar.
Orrr just buy a domain and get a wildcard LE cert? All of the benefits, none of the hassle, and no subdomain leaking.
If the buying a domain part is outside your budget, consider gelocalcert (I work on this) or one of the other free subdomain providers [2].
[1] https://www.getlocalcert.net/
[2] https://free.wdh.gg/#/?id=domains
[1] https://www.getlocalcert.net/
[2] https://free.wdh.gg/#/?id=domains
> Orrr just buy a domain and get a wildcard LE cert?
If you have dozens (or more) of locations that need a cert, then on a yearly basis you have to run around updating things.
If you have dozens (or more) of locations that need a cert, then on a yearly basis you have to run around updating things.
The work to maintain this eventually became too much for me. Everything I did had to have the custom certificates installed, otherwise it would yell at me.
Then there are systems on which installing certificates is hard, like your niece's iPhone. Oh, just use Overseerr to request that movie. What you you mean? Certificate error? Oh, let me fix it.
I get a wildcard from ZeroSSL (after they bought acme.sh) and a couple of non-wildcard ones.
One thing to remember that a wildcard for .domain.com doesn't work for .foo.domain.com. You have to get a second-level *.foo.domain.com wildcard.
Then there are systems on which installing certificates is hard, like your niece's iPhone. Oh, just use Overseerr to request that movie. What you you mean? Certificate error? Oh, let me fix it.
I get a wildcard from ZeroSSL (after they bought acme.sh) and a couple of non-wildcard ones.
One thing to remember that a wildcard for .domain.com doesn't work for .foo.domain.com. You have to get a second-level *.foo.domain.com wildcard.
This could be an interesting learning opportunity. But for the love of god, I hope nobody copies this parrot-fashion just to get rid of the browser warnings. There is no reason to use HTTPS on your LAN. Save yourself the effort and just don't do it.
>There is no reason to use HTTPS on your LAN. Save yourself the effort and just don't do it.
SSL https is definitely a hassle which seems like overkill with private home networks but there can be reasons for it.
For example, self-hosted Bitwarden/Vaultwarden password manager doesn't work without https:// SSL.
The client UI for managing passwords running in web browsers depend on the Chrome/Firefox cryptography APIs and those don't work for non-encrypted http:// :
>Access to the WebCrypto API is restricted to secure origins (which is to say https:// pages). -- from https://www.chromium.org/blink/webcrypto/#:~:text=Access%20t....
SSL https is definitely a hassle which seems like overkill with private home networks but there can be reasons for it.
For example, self-hosted Bitwarden/Vaultwarden password manager doesn't work without https:// SSL.
The client UI for managing passwords running in web browsers depend on the Chrome/Firefox cryptography APIs and those don't work for non-encrypted http:// :
>Access to the WebCrypto API is restricted to secure origins (which is to say https:// pages). -- from https://www.chromium.org/blink/webcrypto/#:~:text=Access%20t....
Unless you’re building a web app that requires a secure context.
https://developer.mozilla.org/en-US/docs/Web/Security/Secure...
https://developer.mozilla.org/en-US/docs/Web/Security/Secure...
Browsers support *.localhost as a secure context.
yeh secure by default isn't harder than dealing with every edge case and quirk.
To be more exact, it's far easier to just shell out the cash for a domain name (or get a free subdomain from somewhere) and then just map the DNS on that domain to your internal infrastructure. That way you can just use letsencrypt certs without any hassle.
If you don't want to expose network details, use something like Tailscale to get unique internal IPs and map those. (Don't use Tailscale's own MagicDNS, that one logs individual certificates for machines into CT rather than giving you a wildcard cert for your tailnet).
If you don't want to expose network details, use something like Tailscale to get unique internal IPs and map those. (Don't use Tailscale's own MagicDNS, that one logs individual certificates for machines into CT rather than giving you a wildcard cert for your tailnet).
Then you'll need a server running locally and the firewall rules to expose that to the internet and either a static IP address or some dynamic DNS client running.
No, I think it's far easier to just not do it.
No, I think it's far easier to just not do it.
You don't; DNS-01 doesn't require contacting any servers besides the DNS registrar, which is what makes it so nice.
In practice, most domain registrars have certbot plugins - you just grab your API key from their customer portal and put it in a file, then run certbot in DNS-01 mode and instruct them to use that. Certbot places a temporary TXT record, which gets verified and the certificate is issued. Then certbot removes the TXT record and that's that. If there's no plugin, you do that by hand, which works but is a bit more involved.
You don't need a firewall for that or a DDNS client. Also if you have a weird registrar that only offers you to set NS records, most VPS providers have free DNS services.
In practice, most domain registrars have certbot plugins - you just grab your API key from their customer portal and put it in a file, then run certbot in DNS-01 mode and instruct them to use that. Certbot places a temporary TXT record, which gets verified and the certificate is issued. Then certbot removes the TXT record and that's that. If there's no plugin, you do that by hand, which works but is a bit more involved.
You don't need a firewall for that or a DDNS client. Also if you have a weird registrar that only offers you to set NS records, most VPS providers have free DNS services.
No, not really. You can obtain the certificate externally and use it internally. The DNS challenge type does not require a web server exposed externally either.
Sure, it's far easier not to do it. It's also far easier to have no root password.
Sure, it's far easier not to do it. It's also far easier to have no root password.
ZTA is fun to experiment with in your home LAN. And to be fair, most people's networks could benefit from it. That being said, the recommendation in this post is not for proper ZTA.
[deleted]
If you're going through all the effort to instrument everything with certbot, why not just use LE and get a real cert?