Automatic SSL Certificates for internal IP's for home k8 setup using LetsEncrypt(techprowd.com)
techprowd.com
Automatic SSL Certificates for internal IP's for home k8 setup using LetsEncrypt
https://www.techprowd.com/automatic-ssl-certificates-for-home-microk8s-setup-using-letsencrypt/
93 comments
I like DNS challenges, but I don't see how it matters for deprecation of an ACME challenge type. The dns-01 challenge could just as easily for some reason need to be deprecated.
The two likely reasons for such deprecation would apply just as well:
1. Updated Baseline Requirements or a programme policy requirement at any of the major root trust stores could forbid this challenge or require it to be substantially modified, obsoleting it in its current form.
2. The BRs don't change but Let's Encrypt finds they need to adjust this particular implementation in a non-compatible way so they deprecate the current challenge.
It can be easier to do DNS challenges, but it can also be very rough, depending on all the moving parts in your system.
The two likely reasons for such deprecation would apply just as well:
1. Updated Baseline Requirements or a programme policy requirement at any of the major root trust stores could forbid this challenge or require it to be substantially modified, obsoleting it in its current form.
2. The BRs don't change but Let's Encrypt finds they need to adjust this particular implementation in a non-compatible way so they deprecate the current challenge.
It can be easier to do DNS challenges, but it can also be very rough, depending on all the moving parts in your system.
Both of those are reasonable concerns, if all other factors were ignored.
However, in practice, the DNS challenge (which demonstrates control over DNS) is greatly preferred over HTTP/TLS challenges (which demonstrate control over a single port).
DNS is likely to be the only way to get a wildcard certificate, and HTTP/TLS will likely end up further restricted once SRVNames in certificates can be gracefully rolled out.
As such, deploying the DNS based control is absolutely the best thing to do, and HTTP/TLS should be seen as legacy-compat fallbacks that may become more difficult in time. Either certificates become less scoped than “entire domain” (as they are today with dNSName SANs) or it becomes more difficult to use a single port to prove authorization for an entire domain.
However, in practice, the DNS challenge (which demonstrates control over DNS) is greatly preferred over HTTP/TLS challenges (which demonstrate control over a single port).
DNS is likely to be the only way to get a wildcard certificate, and HTTP/TLS will likely end up further restricted once SRVNames in certificates can be gracefully rolled out.
As such, deploying the DNS based control is absolutely the best thing to do, and HTTP/TLS should be seen as legacy-compat fallbacks that may become more difficult in time. Either certificates become less scoped than “entire domain” (as they are today with dNSName SANs) or it becomes more difficult to use a single port to prove authorization for an entire domain.
But can’t DNS queries be altered man in the middle style?
I’m not sure your point?
Any HTTP/ALPN request first begins with DNS, so if you’re trying to compare those, they all share the same base issue. In theory, this can be mitigated by DNSSEC, but that’s not relevant when comparing these validation methods.
However, both the HTTP and ALPN methods only demonstrate control over a single port (or .well-known resource), while the DNS method demonstrates the full ability to alter any/all names.
Any HTTP/ALPN request first begins with DNS, so if you’re trying to compare those, they all share the same base issue. In theory, this can be mitigated by DNSSEC, but that’s not relevant when comparing these validation methods.
However, both the HTTP and ALPN methods only demonstrate control over a single port (or .well-known resource), while the DNS method demonstrates the full ability to alter any/all names.
Actually, I suppose DNS with DNSSEC or DNS over HTTPS would be better than any HTTP method.
Verification via DNS is not without issues. If you have more then one DNS server the verification record need to propagate to all servers. If you for example use anycast DNS you will run into issues. Letsencrypt uses Google name servers for lookup which is problematic because they do not behave, they will for example not try secondary dns servers if the first try fail, making the Letsencrypt verification also fail. And because of these issues and if you have many domains you will quickly reach Letsencrypt quota.
Where have you seen Let’s Encrypt using Google’s servers?
CAs are required to run full recursive resolvers, up to the root, and can’t just point at someone else’s DNS infrastructure. Which, if you think about it, is what you want: you don’t want the CA just trusting someone else is being honest, you want to go to the authoritative source.
CAs are required to run full recursive resolvers, up to the root, and can’t just point at someone else’s DNS infrastructure. Which, if you think about it, is what you want: you don’t want the CA just trusting someone else is being honest, you want to go to the authoritative source.
In which case it may be advisable to delegate the acme-challenge record to a different DNS provider, if doing so allows you to sidestep the anycast issues.
i had this issue but i just set the time to wait for propagation high enough to be somewhat certain and had no further issues since (its 10m i think). it does not really matter to me how long it takes as its an automated process... should be finished before expiration though ^^
having multiple nameservers is pretty standard and often mandatory requirement set by the NICs. Its just that my secondary is sometimes not fast enough to transfer the zone after a change notification which triggers this issue.
Also, retrying after an authoritative nameserver said there is defacto no record seems pretty wrong to me... i doubt they use google DNS or anything really. in order to avoid caching issues they very likely resolve names recursively without (the usual) caching
having multiple nameservers is pretty standard and often mandatory requirement set by the NICs. Its just that my secondary is sometimes not fast enough to transfer the zone after a change notification which triggers this issue.
Also, retrying after an authoritative nameserver said there is defacto no record seems pretty wrong to me... i doubt they use google DNS or anything really. in order to avoid caching issues they very likely resolve names recursively without (the usual) caching
> If you for example use anycast DNS you will run into issues.
You're not wrong, but this assumes that you use the your 'service hostname' for verification as well, rather than using CNAMEs.
So let us say you want to have "svc1.example.com" in your cert: you could put the ACME challenge under there, but if you have anycast delays that's a problem (as you mention). (A kludge is putting a 'sleep' somewhere to allow for propagation.)
So instead what you can do is have "_acme-challenge.svc1.example.com" be a CNAME that points to (say) "_acme-challenge.svc1.dnsauth.example.com". This sub-domain is not anycast, and may actually be a single machine that is used solely for this purpose.
The LE/ACME server goes to your main domain, finds a CNAME, and follows that to the real record and verification is achieved:
* https://www.eff.org/deeplinks/2018/02/technical-deep-dive-se...
* https://dan.langille.org/2019/02/01/acme-domain-alias-mode/
* https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mo...
The CNAME has to be set up initially, but can be left lying around otherwise.
This is how $WORK deals with getting LE certs for internal domains: we create a CNAME record (but no A records) for the internal hostname in our external DNS that point to our "dnsauth" domain which gets updating by internal clients via an API.
You're not wrong, but this assumes that you use the your 'service hostname' for verification as well, rather than using CNAMEs.
So let us say you want to have "svc1.example.com" in your cert: you could put the ACME challenge under there, but if you have anycast delays that's a problem (as you mention). (A kludge is putting a 'sleep' somewhere to allow for propagation.)
So instead what you can do is have "_acme-challenge.svc1.example.com" be a CNAME that points to (say) "_acme-challenge.svc1.dnsauth.example.com". This sub-domain is not anycast, and may actually be a single machine that is used solely for this purpose.
The LE/ACME server goes to your main domain, finds a CNAME, and follows that to the real record and verification is achieved:
* https://www.eff.org/deeplinks/2018/02/technical-deep-dive-se...
* https://dan.langille.org/2019/02/01/acme-domain-alias-mode/
* https://github.com/acmesh-official/acme.sh/wiki/DNS-alias-mo...
The CNAME has to be set up initially, but can be left lying around otherwise.
This is how $WORK deals with getting LE certs for internal domains: we create a CNAME record (but no A records) for the internal hostname in our external DNS that point to our "dnsauth" domain which gets updating by internal clients via an API.
Oh, this is an interesting trick... I think I'll need to investigate further.
Do you use a custom acme/dns updater for automatic renewals?
[ed: ie - if I understand correctly, I could point: _acme-challenge.example.com via CNAME to auth.other.example.net - but then I'd like a command to check renew my example.com certs - and it would ideally use an api/dns update to manipulate the auth.other.example.net TXT (or CNAME to something like a15ce5b2-f170-4c91-97bf-09a5764a88f6.auth.acme-dns.io) record when I ask for a check/update of '*.example.com' certificate.
As far as I'm aware, most tooling assumes that you can/will (programmatically or manually) update the _acme-challenge.example.com record directly when issuing/updating an example.com certificate?]
Do you use a custom acme/dns updater for automatic renewals?
[ed: ie - if I understand correctly, I could point: _acme-challenge.example.com via CNAME to auth.other.example.net - but then I'd like a command to check renew my example.com certs - and it would ideally use an api/dns update to manipulate the auth.other.example.net TXT (or CNAME to something like a15ce5b2-f170-4c91-97bf-09a5764a88f6.auth.acme-dns.io) record when I ask for a check/update of '*.example.com' certificate.
As far as I'm aware, most tooling assumes that you can/will (programmatically or manually) update the _acme-challenge.example.com record directly when issuing/updating an example.com certificate?]
The way it works for us:
First we use standard LE/ACME clients: either certbot or dehydrated. They ask for something like svc1.int.example.com ($DOMAIN).
In the hook script(s)† we manipulate the $DOMAIN string to put it into dnsauth.example.com ($AUTH_ZONE) sub-domain and send that new string to the DNS server that handles the dnsauth zone (and only that).
Before all of this we would have set up, in our public-external DNS, a CNAME record to point svc1.int to svc1.int.dnsauth.
The ACME client only thinks about $DOMAIN and the cert-issuing LE server only thinks about $DOMAIN. But the "in between" does not: by doing text manipulation (expr(1) is handy in shell scripts), and DNS redirects, the "in between" uses not-$DOMAIN for verification, but rather TXT records in $AUTH_ZONE.
All with standard ACME clients and some jiggery pokery.
We ended up creating some custom scripts called via SSH, but there are (now) DNS servers written specifically to handle REST API calls [0] and one can use lexicon [1] for just about any commercial DNS service.
[0] https://github.com/joohoi/acme-dns
[1] https://github.com/AnalogJ/lexicon
† dehydrated has deploy_challenge() and clean_challenge() functions in its example hook script. I'm sure most ACME clients have something similar.
First we use standard LE/ACME clients: either certbot or dehydrated. They ask for something like svc1.int.example.com ($DOMAIN).
In the hook script(s)† we manipulate the $DOMAIN string to put it into dnsauth.example.com ($AUTH_ZONE) sub-domain and send that new string to the DNS server that handles the dnsauth zone (and only that).
Before all of this we would have set up, in our public-external DNS, a CNAME record to point svc1.int to svc1.int.dnsauth.
The ACME client only thinks about $DOMAIN and the cert-issuing LE server only thinks about $DOMAIN. But the "in between" does not: by doing text manipulation (expr(1) is handy in shell scripts), and DNS redirects, the "in between" uses not-$DOMAIN for verification, but rather TXT records in $AUTH_ZONE.
All with standard ACME clients and some jiggery pokery.
We ended up creating some custom scripts called via SSH, but there are (now) DNS servers written specifically to handle REST API calls [0] and one can use lexicon [1] for just about any commercial DNS service.
[0] https://github.com/joohoi/acme-dns
[1] https://github.com/AnalogJ/lexicon
† dehydrated has deploy_challenge() and clean_challenge() functions in its example hook script. I'm sure most ACME clients have something similar.
Ok, thank you for the details and managing expectations. This still seems to warrant some experimentation.
In my case I'm mostly interested in delegating a domain/sub-domain somewhere I can easily update (be that run my own dns, host it somewhere with an api) - while having my main domains on a more boring/static dns infrastructure - yet still easily get certs for things like imap.example.com - which would not run a web server. And also split cert renewal to vps/container isolated from things like smtp/imap that need the certs.
In my case I'm mostly interested in delegating a domain/sub-domain somewhere I can easily update (be that run my own dns, host it somewhere with an api) - while having my main domains on a more boring/static dns infrastructure - yet still easily get certs for things like imap.example.com - which would not run a web server. And also split cert renewal to vps/container isolated from things like smtp/imap that need the certs.
[deleted]
> yet still easily get certs for things like imap.example.com - which would not run a web server.
Well, depending on the OS, you could start up a web server during the LE verification process and then bring it down once that's done. You'd only have to run in on port 80 for probably less than a minute.
But yes, you could this mechanism to have "_acme-challenge.imap.example.com" (which is what the ACME protocol uses) be a CNAME to point to something.auth.example.com that is more dynamic. Or even a completely different domain like foo.bar.example.ORG.
In your example.com zone file you'd put NS and A(AAA) records to point to the DNS server that handles the queries for the auth sub-domain.
> And also split cert renewal to vps/container isolated from things like smtp/imap that need the certs.
It's easier to run the ACME client on the host in question, and I'm not sure what it gains you to have it run somewhere else. That being said, there are ACME clients with a bit of a focus on being run 'remotely' from where the certs actually live:
* https://github.com/srvrco/getssl
This is probably for shared-hosting scenarios where cron is not accessible.
IMHO though, if you have access to the CLI on the host running the TLS service, it's best to run things there.
Well, depending on the OS, you could start up a web server during the LE verification process and then bring it down once that's done. You'd only have to run in on port 80 for probably less than a minute.
But yes, you could this mechanism to have "_acme-challenge.imap.example.com" (which is what the ACME protocol uses) be a CNAME to point to something.auth.example.com that is more dynamic. Or even a completely different domain like foo.bar.example.ORG.
In your example.com zone file you'd put NS and A(AAA) records to point to the DNS server that handles the queries for the auth sub-domain.
> And also split cert renewal to vps/container isolated from things like smtp/imap that need the certs.
It's easier to run the ACME client on the host in question, and I'm not sure what it gains you to have it run somewhere else. That being said, there are ACME clients with a bit of a focus on being run 'remotely' from where the certs actually live:
* https://github.com/srvrco/getssl
This is probably for shared-hosting scenarios where cron is not accessible.
IMHO though, if you have access to the CLI on the host running the TLS service, it's best to run things there.
Re: your latest point - typically I'd like imap/smptd to run in separate static containers/vms with read access to the cert, but not write (and a volume or db to write emails to etc).
In general I'd prefer the certs be something the services get via configuration mgmnt - while the cert service can run via cron and make sure certs are valid an present.
In particular, I don't want my smtpd server to have write access to my dns, if I can help it.
In general I'd prefer the certs be something the services get via configuration mgmnt - while the cert service can run via cron and make sure certs are valid an present.
In particular, I don't want my smtpd server to have write access to my dns, if I can help it.
See also lexicon, which supports over 50 APIs:
* https://github.com/AnalogJ/lexicon
* https://github.com/AnalogJ/lexicon
I do this with Traefik [0] internally in almost the same way. I use DNS-01 to get a Let's Encrypt wildcard cert and all my internal A records point to the ingress IP and Traefik happily proxies the communications to the appropriate service - container based and non-container based - which is the real win I was looking to solve for in my home environment. The thing I like about just using Traefik is it doesn't rely on a lot of extraneous tooling (can just use Docker without Swarm/K8s) and will automatically consume orchestration services if I'd like it to. But the reality is the majority of things I want valid certs for are static mappings. One config file update of a few new lines of boilerplate is all it takes to get a valid cert fronting any service. And then to get a dashboard of all my internal services I use Heimdall [1].
[0] https://docs.traefik.io/ [1] https://heimdall.site/
[0] https://docs.traefik.io/ [1] https://heimdall.site/
Appreciate this breakdown, I've been using an nginx proxy to hold a few things over and wanted to move in the direction of a managed service, which Traefik looks perfect for.
As time goes on the value of having services running as appliances is becoming more and more valuable.
As time goes on the value of having services running as appliances is becoming more and more valuable.
Keep in mind, adding local entries to your external DNS will expose internal details of your network, such as hostnames and IPs. Same goes for Let's Encrypt, due to Certificate Transparency logging.
While you'll get the hostnames leaked - you could register them as fake addresses (say an A record for 192.168.0.1 for every address), and have a local DNS server overriding with the real addresses.
Whether this is worthwhile or not is debatable. Is the fact your internal server 'gubbins.mydomain.com' exists, or even that it exists on 10.0.41.43 really much use?
The other option for internal certificates is to get a wildcard of *.internal.mydomain.com, and spread that wildcard certificate around your network.
The final solution is run your own certificate authority and trust it on every browser. For some reason when you import a root certificate you can't typically allow that CA to only be used to authenticate a given subdomain. There are x509 constraints you can use in setting up the CA, but that's rare too, and I'm not sure every tool uses it.
In any case, if you go for an internal DNS provision, make sure you set use-application-dns.net to NXDOMAIN on your internal dns server to override DoH too
Whether this is worthwhile or not is debatable. Is the fact your internal server 'gubbins.mydomain.com' exists, or even that it exists on 10.0.41.43 really much use?
The other option for internal certificates is to get a wildcard of *.internal.mydomain.com, and spread that wildcard certificate around your network.
The final solution is run your own certificate authority and trust it on every browser. For some reason when you import a root certificate you can't typically allow that CA to only be used to authenticate a given subdomain. There are x509 constraints you can use in setting up the CA, but that's rare too, and I'm not sure every tool uses it.
In any case, if you go for an internal DNS provision, make sure you set use-application-dns.net to NXDOMAIN on your internal dns server to override DoH too
> Whether this is worthwhile or not is debatable. Is the fact your internal server 'gubbins.mydomain.com' exists, or even that it exists on 10.0.41.43 really much use?
You're probably aware of tools such as dnsdumpster (dnsdumpster.com) that attempt to map internal networks using dns. It is common to use descriptive prefixes (eg. intranet, gw, fw, dns, jira, mysqldb, etc) that also give some insight on the internal tools and topology. Using the CT logs you can also collect over time changes in network, for servers/endpoints with non-wildcard certificates.
Its not a huge risk. But why give a map of the place to a potential attacker, when - as you quite well proposed - you can use simple ways of avoiding it?
You're probably aware of tools such as dnsdumpster (dnsdumpster.com) that attempt to map internal networks using dns. It is common to use descriptive prefixes (eg. intranet, gw, fw, dns, jira, mysqldb, etc) that also give some insight on the internal tools and topology. Using the CT logs you can also collect over time changes in network, for servers/endpoints with non-wildcard certificates.
Its not a huge risk. But why give a map of the place to a potential attacker, when - as you quite well proposed - you can use simple ways of avoiding it?
Downsides of a single wildcard certificate has its own issues, the more you pass a single certificate round the more opportunity for it to leak and the wider the risk
If you have a per-host DNS based with no valid A records, DOH could well mean worse performance in future as devices don't listen to canary domain and only query locally once getting an NXDOMAIN (and even then might not). You're still leaking the CN and SAN entries via CT.
If you run your own CA, every device has to have it installed, and it's a major security risk as losing control of the CA means all your machines can be compromised.
Between certificate transparency and DNS over HTTPS, there's no perfect answer.
If you have a per-host DNS based with no valid A records, DOH could well mean worse performance in future as devices don't listen to canary domain and only query locally once getting an NXDOMAIN (and even then might not). You're still leaking the CN and SAN entries via CT.
If you run your own CA, every device has to have it installed, and it's a major security risk as losing control of the CA means all your machines can be compromised.
Between certificate transparency and DNS over HTTPS, there's no perfect answer.
> when you import a root certificate you can't typically allow that CA to only be used to authenticate a given subdomain.
While, true, there are some workarounds, that are, ehm, workaroundy.
1. For _your_ domain, you can have CAA records in your DNS. (That solves it for your domains, not for others, if they do not use CAA)
2. Some CA implementations, like FreeIPA one, allow you to issue certificates only for your own domain. They will refuse to sign a certificate for any other domain.
So while it is not 100% solution, it is 80% one.
> In any case, if you go for an internal DNS provision, make sure you set use-application-dns.net to NXDOMAIN on your internal dns server to override DoH too
Also make sure that the browser honors this. Firefox honors canary domain only when in 'auto DoH' mode, but not, when the user has explicitly configured DoH. From https://support.mozilla.org/en-US/kb/canary-domain-use-appli...:
> The canary domain only applies to users who have DoH enabled as the default option. It does not apply for users who have made the choice to turn on DoH by themselves.
While, true, there are some workarounds, that are, ehm, workaroundy.
1. For _your_ domain, you can have CAA records in your DNS. (That solves it for your domains, not for others, if they do not use CAA)
2. Some CA implementations, like FreeIPA one, allow you to issue certificates only for your own domain. They will refuse to sign a certificate for any other domain.
So while it is not 100% solution, it is 80% one.
> In any case, if you go for an internal DNS provision, make sure you set use-application-dns.net to NXDOMAIN on your internal dns server to override DoH too
Also make sure that the browser honors this. Firefox honors canary domain only when in 'auto DoH' mode, but not, when the user has explicitly configured DoH. From https://support.mozilla.org/en-US/kb/canary-domain-use-appli...:
> The canary domain only applies to users who have DoH enabled as the default option. It does not apply for users who have made the choice to turn on DoH by themselves.
I'm thinking more that if you create your own CA and trust it as a root CA, you are responsible for the security of that CA - it can be used against you.
This is especially problematic if I want someone else to trust my certs in their browser, I'm asking them to trust any certificate I generate, including google.com, mybank.com, etc.
This is especially problematic if I want someone else to trust my certs in their browser, I'm asking them to trust any certificate I generate, including google.com, mybank.com, etc.
I don't want to trust your certificates in my browser, even if you invent all sorts of sophisticated technical hoops to jump through I have no reason to believe you actually did all that hoop jumping. Get certificates for the names you want from a PKI we both trust. The only one that's likely to be applicable is the Web PKI, which is what was done in this article.
For some other purposes, where the Web PKI isn't applicable, you might have a solid argument for why people should trust your PKI. But then they aren't trusting certificates in their browser but in some other likely very constrained system that's easier to reason about.
For example I've operated (for past employers) a service that was available over HTTPS to a handful of insurance and financial outfits using mutual TLS. I'd issue client certificates for them, and then they'd use their private key and their certificate to access the service over HTTPS. In that case I ran the CA, and so they're trusting me. But only narrowly to identify customers of the service I run, so it's not tricky to reason about.
Now, technically you could use certificates from Let's Encrypt as client certificates, they have the right EKU for that purpose but then your identity has to be an Internet FQDN and you need new certificates every ninety days, which sucks. So although I'd have accommodated a paying customer who firmly insisted on doing that, none of them did.
For some other purposes, where the Web PKI isn't applicable, you might have a solid argument for why people should trust your PKI. But then they aren't trusting certificates in their browser but in some other likely very constrained system that's easier to reason about.
For example I've operated (for past employers) a service that was available over HTTPS to a handful of insurance and financial outfits using mutual TLS. I'd issue client certificates for them, and then they'd use their private key and their certificate to access the service over HTTPS. In that case I ran the CA, and so they're trusting me. But only narrowly to identify customers of the service I run, so it's not tricky to reason about.
Now, technically you could use certificates from Let's Encrypt as client certificates, they have the right EKU for that purpose but then your identity has to be an Internet FQDN and you need new certificates every ninety days, which sucks. So although I'd have accommodated a paying customer who firmly insisted on doing that, none of them did.
Yes, I generate a client cert and give it to you, that's not a problem - you aren't trusting the cert at all.
I can secure my servers with my own CA, again that's fine - I trust myself.
If you want to verify my servers signed from my CA though you would need to trust my CA.
Currently that means you would need to import my CA's root certificate. That's bad, it's also the way many corporations work, I've had third party suppliers try to get me to trust their own internally generated root certificates too.
A better solution would be for the user to be able to import a root certificate for use for a set purpose. I'm happy to take foobar inc's certificate to authenticate *.foobar.inc, but not to trust anything else.
For some reason browsers don't allow that - it's all or nothing.
I can secure my servers with my own CA, again that's fine - I trust myself.
If you want to verify my servers signed from my CA though you would need to trust my CA.
Currently that means you would need to import my CA's root certificate. That's bad, it's also the way many corporations work, I've had third party suppliers try to get me to trust their own internally generated root certificates too.
A better solution would be for the user to be able to import a root certificate for use for a set purpose. I'm happy to take foobar inc's certificate to authenticate *.foobar.inc, but not to trust anything else.
For some reason browsers don't allow that - it's all or nothing.
> For some reason browsers don't allow that - it's all or nothing.
Absolutely. Name constraints. It's in the spec, but no one seems to implement it. This would fully make PKI useful. Typing x509 certificate (validation) to a delegated domain would allow people to import Root CAs from a variety of places while compartmentalizing security.
Right now, if you trust a Root CA cert, you trust any domain it signs. This has been a glaring problem for years. I never thought we'd have cheap certs for everything and LE came along and blew my mind.
Absolutely. Name constraints. It's in the spec, but no one seems to implement it. This would fully make PKI useful. Typing x509 certificate (validation) to a delegated domain would allow people to import Root CAs from a variety of places while compartmentalizing security.
Right now, if you trust a Root CA cert, you trust any domain it signs. This has been a glaring problem for years. I never thought we'd have cheap certs for everything and LE came along and blew my mind.
> Is the fact your internal server 'gubbins.mydomain.com' exists, or even that it exists on 10.0.41.43 really much use?
Pretty much this. What does it matter if you know certain hostnames or internal IPs on my network? It's all firewalled anyway, and if it wasn't it would be trivial to find them out on your own...
Pretty much this. What does it matter if you know certain hostnames or internal IPs on my network? It's all firewalled anyway, and if it wasn't it would be trivial to find them out on your own...
It may be of interest for attackers that have no visibility into your network and make cross site attacks against you easier. For example, if I know that your router is available at router.network.internal, I might just try and see if your browser is logged in and send you a link to a page that starts making requests against that interface. Enumerating network resources can certainly be done via other ways, but DNS is a particularly easy one.
Interesting but 192.168.0.1 or 192.168.1.1 would probably cover 80% of that use case.
This is just security by obscurity. It doesn't add much if you already implement actual security in the form of firewalls, network segregation, etc.
This is just security by obscurity. It doesn't add much if you already implement actual security in the form of firewalls, network segregation, etc.
I agree that it's certainly not a substitute for a proper security setup, but it definitely makes an attackers life easier - up to the point where it may get interesting to automate this. So as always, it's a tradeoff. Both options may be acceptable (and I've run a setup where I'd have internal names exposed on public DNS for years), but you need to be aware of the tradeoff to make that judgement.
I guess it could be useful for human level attacks, something like
"Hi it's Bob down in IT, any chance you could reboot 'funkyserver123', it's on 172.18.12.5, for some reason I can't get in"
"Hi it's Bob down in IT, any chance you could reboot 'funkyserver123', it's on 172.18.12.5, for some reason I can't get in"
Why would I answer a request from Bob down in IT about my home network?
Edit: I don't know why this is being downvoted. The OP is about a home k8s setup, so unless you live in some castle or palace I very much doubt there will be a Bob down in IT.
Edit: I don't know why this is being downvoted. The OP is about a home k8s setup, so unless you live in some castle or palace I very much doubt there will be a Bob down in IT.
Possibly useful for SSRF attack but the matter is SSRF.
There is no need to have an A record in the external DNS. The DNS challenge uses the contents of a TXT record, _acme-challenge.<YOUR_DOMAIN>, for verification. The certbot DNS plugins will automatically create the TXT record and delete it following verification.
[deleted]
You'd actually end up with a 'split horizon' setup wherein LetsEncrypt (and the public) see those DNS names with the external gateway's IP and ping that for the validation files.
Your options at that point are a central 'well known' directory that different hosts can write to (I recommend sshfs), different directories on one host that are checked for any valid file in any of them (by default) or by hostname match in specific, etc. The details depend on your security model.
Your options at that point are a central 'well known' directory that different hosts can write to (I recommend sshfs), different directories on one host that are checked for any valid file in any of them (by default) or by hostname match in specific, etc. The details depend on your security model.
In this case, I don’t think there is a gateway involved at all. I believe Let’s encrypt is verifying the authors domains by (essentially) querying their selected DNS provider to prove ownership over a domain, which resolved to a private IP.
So iiuc there is no split horizon, it’s just that the sites would only work for the author.
So iiuc there is no split horizon, it’s just that the sites would only work for the author.
I think you are half correct. The gateway has nothing to do with verifying the file during a DNS challenge. However, the IP of the machine requesting the cert IS saved with that cert information and made public. Let's encrypt will even warn you during the verification process.
The IP of the machine requesting the certificate is recorded by Let's Encrypt, but it is not (ordinarily) made public and certainly isn't (as you can see by inspecting it for yourself) saved with the certificate information.
ISRG is required to keep enough information about the issuances they make to allow them to usefully diagnose problems after the fact. Ideally when we discover a problem it will be possible for the issuer to go back and figure out which (if any) previously issued certificates were affected so that these certificates can be revoked if appropriate.
But although they had at one point planned to publish more of this information, they do not in fact do this routinely.
ISRG is required to keep enough information about the issuances they make to allow them to usefully diagnose problems after the fact. Ideally when we discover a problem it will be possible for the issuer to go back and figure out which (if any) previously issued certificates were affected so that these certificates can be revoked if appropriate.
But although they had at one point planned to publish more of this information, they do not in fact do this routinely.
Yeah, I was referring to the certbot warning of "logging" the IP publicly. But I guess that policy never actually came to fruition. Thanks for the clarification!
The host names wouldn't even need to resolve at all outside of the local network. Only the challenge subdomain _acme-challenge.foo.bar.org so certificate transparency will of course expose the existence of foo.bar.org but it wouldn't need to expose anything at all about those DNS records if you wanted to restrict them to only the local network.
[deleted]
Is that actually valuable information though? I mean internal IP addresses are fairly generic (like the 192.168.x range mentioned in the article).
Not always, but sometimes maybe. Check eg. dnsdumpster.com
I've solved the IP exposure issue with CNAME records and a local DNS server. So if I want to run a Plex server at example.com, I'll set plex.example.com to be a CNAME for plex.internal and just have the local DNS server resolve what IP address plex.internal is at. Easy way to get HTTPS addresses for internal IP addresses without having the IP address listed publicly.
You don't need to add those entries to your external DNS. It's possible to create and validate Let's Encrypt certificates for names which don't exist in public DNS, and use split-horizon DNS to define them on your local network.
With regards to DNS-01, only the temporary TXT record needs to be public. If you want everything to be internal you can run your own DNS server on your network and have all the actual A/AAAA/etc records there.
Anyway cert issued domain name is published for Certificate Transparency. Check crt.sh
for the domain problem I use wildcard dns.
for IP I dont think lets encrypt logging the ip address publicly (let me know if I wrong about it), since I use dns-01 I can generate SSL from anywhere.
for IP I dont think lets encrypt logging the ip address publicly (let me know if I wrong about it), since I use dns-01 I can generate SSL from anywhere.
> for IP I dont think lets encrypt logging the ip address publicly (let me know if I wrong about it), since I use dns-01 I can generate SSL from anywhere.
Yes, if you override DNS (or just provide it) internally then the IP is hidden. The hostname is still leaked though - for example Geodynamics limited have server names of the format fvdcsev01 and fvdccit03 -- https://crt.sh/?id=2381703940
Yes, if you override DNS (or just provide it) internally then the IP is hidden. The hostname is still leaked though - for example Geodynamics limited have server names of the format fvdcsev01 and fvdccit03 -- https://crt.sh/?id=2381703940
Its TLS and not SSL. Its TLS for a long time now...
And yes be aware that through this, it works fine but you are also exposing your internal infrastructure details through dns.
I'm not seeing a big issue, just be aware of it.
And yes be aware that through this, it works fine but you are also exposing your internal infrastructure details through dns.
I'm not seeing a big issue, just be aware of it.
> Its TLS and not SSL. Its TLS for a long time now...
Sure, that's technically correct but a wee bit overly pedantic. When technical people speak about SSL/TLS certificates it's common parlance to say "SSL" and everyone usually knows what you're talking about, which includes TLS, and whatever other new acronym might come down the pipe in the future.
Sure, that's technically correct but a wee bit overly pedantic. When technical people speak about SSL/TLS certificates it's common parlance to say "SSL" and everyone usually knows what you're talking about, which includes TLS, and whatever other new acronym might come down the pipe in the future.
You know, i do get this but we are in IT not in Marketing.
My most used skill is to make sure i'm pedantic aka 'so we need to calculate this from that after this? and we need accuracy of 0.32? And what should that button do exactly?'
It is not SSL its TLS and i don't expect everyone to get it but its still wrong.
The weirdes thing in IT is, that i don't know any other word which is so missused then SSL.
My most used skill is to make sure i'm pedantic aka 'so we need to calculate this from that after this? and we need accuracy of 0.32? And what should that button do exactly?'
It is not SSL its TLS and i don't expect everyone to get it but its still wrong.
The weirdes thing in IT is, that i don't know any other word which is so missused then SSL.
Good point even if it is pedantic. I shift between terms depending on my audience, but try to use TLS in technical circles as an example.
I worry that I'll forever have to use both terms, because while SSL is the term that communicates better to semi-literate audiences, I worry that some security expert will assume I don't know anything because I'm using the "wrong" term.
I worry that I'll forever have to use both terms, because while SSL is the term that communicates better to semi-literate audiences, I worry that some security expert will assume I don't know anything because I'm using the "wrong" term.
Jepp i actually assumed that at one point and i was right :D
Just setup a wildcard cert, apply that to wherever, and you limit how much you expose your internal infra.
I know the difference, but common usage is to refer to the whole group of protocols as SSL.
I have yet to apt-get install opentls on any webserver.
I have yet to apt-get install opentls on any webserver.
I wrote a similar post about a year ago[0], but even at the time I wasn't the first to come up with this idea. As someone who doesn't have a lot of experiences with DNS security, seeing other people floating similar setups without significant pushback gives me more confidence that the core idea isn't horribly unsafe. I'm pretty happy/relieved to see other people playing around in the same space.
My perspective was (and is) that for portable devices (phones/laptops) that are interfacing with locally hosted services, having SSL for those services is really important because your device probably isn't configured to check what network it's on before automatically pinging 192.168.1.x. This is doubly important if you have other people occasionally hopping onto your network and connecting to those same services. It's imo bad practice to ask everyone connecting to your network to install certificates or set up a certificate manager. I wouldn't do that for any of my personal devices if someone asked me to.
To push this a step farther, I imagined a world where my services could handle not just renewing their own certificates, but also updating their addresses if they were moved to a different network/address. If I build a physical device to give to someone, I'd like them to be able to plug it into their network, go to a web URL, and have everything just work -- no messing around with their internal DNS settings or worrying about whether they're using DNS over HTTPS in Firefox.
[0]: https://danshumway.com/blog/encrypting-internal-networks/
My perspective was (and is) that for portable devices (phones/laptops) that are interfacing with locally hosted services, having SSL for those services is really important because your device probably isn't configured to check what network it's on before automatically pinging 192.168.1.x. This is doubly important if you have other people occasionally hopping onto your network and connecting to those same services. It's imo bad practice to ask everyone connecting to your network to install certificates or set up a certificate manager. I wouldn't do that for any of my personal devices if someone asked me to.
To push this a step farther, I imagined a world where my services could handle not just renewing their own certificates, but also updating their addresses if they were moved to a different network/address. If I build a physical device to give to someone, I'd like them to be able to plug it into their network, go to a web URL, and have everything just work -- no messing around with their internal DNS settings or worrying about whether they're using DNS over HTTPS in Firefox.
[0]: https://danshumway.com/blog/encrypting-internal-networks/
I’ve tried to set up kubernetes at home a couple of times and I always freak out at the amount of layers and “just run this” style of tutorials. Am I crazy?
I’ve heard guix has some kind of container management thing. I’ve been thinking about trying it anyway.
I’ve heard guix has some kind of container management thing. I’ve been thinking about trying it anyway.
Ubuntu with Microk8s is very simple. Basically a few lines of commands, well documented and addons exist as well for most use cases.
MetalLB for your loadbalancing needs, prometheus/grafana etc.
https://microk8s.io/docs
And microk8s supports multiply nodes now as well.
MetalLB for your loadbalancing needs, prometheus/grafana etc.
https://microk8s.io/docs
And microk8s supports multiply nodes now as well.
"Kubernetes at home" called "Docker Swarm".
I've been using it couple of years for many types of workloads and it's been a pleasant experience.
Ansible + docker-compose.yml to manage apps on 1-3 servers. You just install docker and that's it. Swarm configuration is 95% similar to compose file you use locally if you're into developing with Docker.
It has couple of edges if you try to use it for complex setups but for apps with < 5 devs and < 10 services it's really simple. You can still migrate to K8s when you hire dedicated DevOps team because you're running containers and all your setup documented in docker-compose file.
I've been using it couple of years for many types of workloads and it's been a pleasant experience.
Ansible + docker-compose.yml to manage apps on 1-3 servers. You just install docker and that's it. Swarm configuration is 95% similar to compose file you use locally if you're into developing with Docker.
It has couple of edges if you try to use it for complex setups but for apps with < 5 devs and < 10 services it's really simple. You can still migrate to K8s when you hire dedicated DevOps team because you're running containers and all your setup documented in docker-compose file.
k8s needs a control plane, that needs security, hence all the tokens, certs (which need internal and external IPs and FQDNs), also it needs to set up an overlay network (so you need to configure the CNI provider, sysctl stuff for ebtables and iptables/nftables to work correctly), and DNS, and a dashboard would be nice too. oh, and unless you use k3s or something that budles a container runtime (CRI provider) you need to setup one (eg docker).
it's understandably complex, even if many parts are pretty standard (eg. the sysctl stuff, and installing dependencies is basically dnf/yum/apt/apk or exit and let the user do it).
since the most error prone parts were/are setting up the TLS stuff that got automated first (in the form of kubeadm install), and the rest just remains in "run this" form.
but the k3s installer is just a one liner call to a bash script. though then you have to make sure to include the magic env vars to get what you want.
it's understandably complex, even if many parts are pretty standard (eg. the sysctl stuff, and installing dependencies is basically dnf/yum/apt/apk or exit and let the user do it).
since the most error prone parts were/are setting up the TLS stuff that got automated first (in the form of kubeadm install), and the rest just remains in "run this" form.
but the k3s installer is just a one liner call to a bash script. though then you have to make sure to include the magic env vars to get what you want.
The k3sup project [1] takes this a step further and makes installing k3s even easier. k3s has been the most useful piece of infra I run at home. It gives me all the benefits of k8s with none of the complexity.
[1] https://github.com/alexellis/k3sup
[1] https://github.com/alexellis/k3sup
Honestly this feels overly complex when you can just create a CA and add the CA to ur devices. Still cool tho.
At home, with just a few devices, it's doable, but adding a ca on all the devices of a corporate network is a huge pain. On one hand, you have varying levels of control (from none to total) on the devices. On the other hand adding a ca is a bit of a pain, with the ca needed to be added in various ca collection s for different softwares (ex in linux, ca-certificates, java certificates, and to add them system wide for browsers, you need to recompile libnss).
We looked into this at $WORK, but it can be slightly annoying as you have to create a workflow for each operating system's trust store, but you also have to deal with many browsers independently as well, since many of them don't use the OS' trust store.
You don't want to be running a CA. Because devices don't implement "name constraints", once you import your CA Root cert into a device, all HTTPS communications on that device can be subverted if someone gets ahold of your Root CA's cert/private key.
I find it incredibly annoying that i can't tell chrome to use THIS CA Root Cert only for *.mydomain.com and not my banking domains, email, etc.
I find it incredibly annoying that i can't tell chrome to use THIS CA Root Cert only for *.mydomain.com and not my banking domains, email, etc.
I just created a wildcard with letsencrypt in the format of .internal.mydomain.com
My public services all run out of .mydomain.com and all my internal services run out of .internal.mydomain.com
I have my internal dns set to resolve any .internal calls to an internal load balancer which hosts the ca certs.
The downside is that all internal services are ssl terminated at the load balancer, but this makes handling internal certs easy as they're rotated in a single location. This is Good Enough for my homelab.
My public services all run out of .mydomain.com and all my internal services run out of .internal.mydomain.com
I have my internal dns set to resolve any .internal calls to an internal load balancer which hosts the ca certs.
The downside is that all internal services are ssl terminated at the load balancer, but this makes handling internal certs easy as they're rotated in a single location. This is Good Enough for my homelab.
inlets with the inlets-operator [0] does this by using the HTTP01 challenge, and gives you a LoadBalancer just like you'd have on AWS. The benefit is that you get a real IP and routable traffic, there's no tricks required. It would also work with DNS01 if that's of interest.
[0] https://github.com/inlets/inlets-operator
Feel free to check it out in this tutorial: https://docs.inlets.dev/#/get-started/quickstart-ingresscont...
[0] https://github.com/inlets/inlets-operator
Feel free to check it out in this tutorial: https://docs.inlets.dev/#/get-started/quickstart-ingresscont...
I haven't tried it yet but if you have control of your DNS and want automation:
https://github.com/joohoi/acme-dns/
https://github.com/joohoi/acme-dns-certbot
A simplified DNS server with a RESTful HTTP API to provide a simple way to automate ACME DNS challenges.
https://github.com/joohoi/acme-dns/
https://github.com/joohoi/acme-dns-certbot
A simplified DNS server with a RESTful HTTP API to provide a simple way to automate ACME DNS challenges.
This page is raising a ton of security alerts:
"NoScript detected a potential Cross-Site Scripting attack from https://www.techprowd.com to https://carbon.now.sh"
Images are failing to load too. Not sure what's going on.
edit: Probably some misusing of DNS rather than actual attack but who knows. Author should fix the site.
"NoScript detected a potential Cross-Site Scripting attack from https://www.techprowd.com to https://carbon.now.sh"
Images are failing to load too. Not sure what's going on.
edit: Probably some misusing of DNS rather than actual attack but who knows. Author should fix the site.
I think that's NoScript being overprotective. carbon.now.sh is a site that renders nice terminal sessions as html that you can include as iframes. - better than screenshots because you can actually copy the code. And as part of that request, the shell code is passed in the query string. I haven't investigated, but NoScript may be triggering on that.
It's hosted on Ghost Pro serviced blog platform. https://carbon.now.sh is those code blocks u see in the article it's code highlighter
It's not a certificate for internal ip address, it's a certificate for host name. Ip address is irrelevant here.
I use step-ca [0] for these sort of things and it works brilliantly. I barely see the point of having external DNS servers resolving your internal infrastructure.
[0] https://smallstep.com/certificates/
[0] https://smallstep.com/certificates/
I thought about that but passed because I didn't feel like telling all my browsers to trust that new CA. Yes, that's incredibly lazy.
I bought a real domain name, told my UBNT USG that was the domain for my network, set up the dns servers to use digital ocean, used jetstack's cert-manager [0] to acquire the a wildcart cert using DNS01 instead of HTTP01, and use kubed [1] to synchronize the TLS cert across namespaces. One key thing to consider is that you really should ensure that you use the staging let's encrypt server to test out issuance and see your browser complain about warnings before you switch to production let's encrypt.
Honestly, I don't mind that the cert requests for my domain show up in a CT log.
[0] https://cert-manager.io/
[1] https://cert-manager.io/docs/faq/kubed/
I bought a real domain name, told my UBNT USG that was the domain for my network, set up the dns servers to use digital ocean, used jetstack's cert-manager [0] to acquire the a wildcart cert using DNS01 instead of HTTP01, and use kubed [1] to synchronize the TLS cert across namespaces. One key thing to consider is that you really should ensure that you use the staging let's encrypt server to test out issuance and see your browser complain about warnings before you switch to production let's encrypt.
Honestly, I don't mind that the cert requests for my domain show up in a CT log.
[0] https://cert-manager.io/
[1] https://cert-manager.io/docs/faq/kubed/
No association what so ever to 'em but I so dearly love what they do, I'd encourage users to donate to keep them going healthy and strong!
https://letsencrypt.org/donate/
https://letsencrypt.org/donate/
I did that, the very hard way (I didn’t know better at the time): https://whynot.fail/homelab/lets-encrypt-the-house/
Another alternative is inlets which automates all of the steps necessary and offers Layer 4 as well as Layer 7: https://docs.inlets.dev/#/
Instead of using hacky fragile methods , use
https://github.com/FiloSottile/mkcert
to automate setting up local CA and making it trusted.
https://github.com/FiloSottile/mkcert
to automate setting up local CA and making it trusted.
mkcert may work fine if you're the only person using those network resources on a single machine, ever. Otherwise you've just traded yourself a trust management problem: Now you need to secure the key of that CA and distribute the CA certificate to all devices that should be trusting it. This may or may not be trivial.
The fundamental problem is that this CA that you generated gets basically the same trust level as a public CA, but it's just sitting there on your machine. An attacker could use it to generate certificates for almost every site and your devices would trust them. That's probably ok if only your machine trusts that CA since if the attacker rooted your box to the point that they gained access to that CA key, all is lost anyways. In a network with other devices - maybe even not under your direct control - that tradeoff looks substantially different.
The fundamental problem is that this CA that you generated gets basically the same trust level as a public CA, but it's just sitting there on your machine. An attacker could use it to generate certificates for almost every site and your devices would trust them. That's probably ok if only your machine trusts that CA since if the attacker rooted your box to the point that they gained access to that CA key, all is lost anyways. In a network with other devices - maybe even not under your direct control - that tradeoff looks substantially different.
I love that program and use that only on my laptop when I'm doing web development paired with nginx and dnsmasq.
I do have to set an environment variable so nodejs trusts it.
I do have to set an environment variable so nodejs trusts it.
[deleted]
[deleted]
[deleted]
Why not just get a wildcard LE cert and not worry about it?
TLDR:
- have a proper worldwide domain
- obtain a certificate for that domain
- point the domain to local IPs in your network and use the certificate on the local server.
Doesn't change that you'd need to self-sign certs for .local or other funky domains.
- have a proper worldwide domain
- obtain a certificate for that domain
- point the domain to local IPs in your network and use the certificate on the local server.
Doesn't change that you'd need to self-sign certs for .local or other funky domains.
Probably just give things which need names actual globally unique names from the Internet's DNS hierarchy.
More controversially I think you should give things globally unique addresses from the Internet's global address system but it's more important to at least give them names from the globally unique system even if you insist on using RFC1918 numbers.
More controversially I think you should give things globally unique addresses from the Internet's global address system but it's more important to at least give them names from the globally unique system even if you insist on using RFC1918 numbers.
split dns is typically the solution for this, is it not?
I recommend this not just for internal IP setups, for actually for all setups, since DNS verification is more robust than HTTP verification, particularly if you have issues with load balancers, or if Let's Encrypt decides to deprecate a protocol again [1].
[0] https://cert-manager.io/docs/configuration/acme/dns01/#suppo... [1] https://community.letsencrypt.org/t/upcoming-tls-sni-depreca...