Ask HN: Cloudflare broke my domain's DNSSEC making it unreachable since 4 days
122 comments
(It sucks that I had to see this on HN)
Can you email me - silverlock at cloudflare - with your ticket ID and domain name so I can understand what broke?
Can you email me - silverlock at cloudflare - with your ticket ID and domain name so I can understand what broke?
Thank you for the attention, e-mail on its way.
Just replied.
Ultimately it looks like the existing DS records for your domain weren't removed (and you can see that in your DNSViz output). Still have some questions for "how" it was working beforehand (see the email for those).
For others: I'll let the OP share what details they would like to, as this is their domain.
Ultimately it looks like the existing DS records for your domain weren't removed (and you can see that in your DNSViz output). Still have some questions for "how" it was working beforehand (see the email for those).
For others: I'll let the OP share what details they would like to, as this is their domain.
Come back and tell us what happened and the resolution.
Update for those who were curious:
Roughly one hour after I e-mailed @elithrar who kindly reached out and offered to expediate the issue, the broken DNSSEC records were partly fixed. The domain once again resolved through all major DNSes, and public access was restored. At that point dnsviz.net told me that A, MX, etc. records were "insecure", though name resolution worked fine. A few minutes ago I took another look with dnsviz and it's now telling me that all records are secure. Everything looks normal again.
Thanks a bunch for helping out, @elithrar. I really appreciate that you were proactive.
If the problem had somehow fixed itself or if the support ticket had gotten any attention or feedback at all within a day or two instead of just being "snoozed" by support staff, I wouldn't have made any noise about it. After four days of complete silence a bit of "cry-baby consumer activism" seemed like the only resort.
If CF reconnects to me with an update on why the domain dead-locked and why it took 4 days to untilt everything I'll add that info as well.
I've been OP and this has been an update about my domain woes.
Roughly one hour after I e-mailed @elithrar who kindly reached out and offered to expediate the issue, the broken DNSSEC records were partly fixed. The domain once again resolved through all major DNSes, and public access was restored. At that point dnsviz.net told me that A, MX, etc. records were "insecure", though name resolution worked fine. A few minutes ago I took another look with dnsviz and it's now telling me that all records are secure. Everything looks normal again.
Thanks a bunch for helping out, @elithrar. I really appreciate that you were proactive.
If the problem had somehow fixed itself or if the support ticket had gotten any attention or feedback at all within a day or two instead of just being "snoozed" by support staff, I wouldn't have made any noise about it. After four days of complete silence a bit of "cry-baby consumer activism" seemed like the only resort.
If CF reconnects to me with an update on why the domain dead-locked and why it took 4 days to untilt everything I'll add that info as well.
I've been OP and this has been an update about my domain woes.
(CF TAM here)
All plans come with support. Even the free plans (community, or email, the bot will deflect the request but if you email you're still stuck, you will get a reply _eventually_ (due to heavy support load, it can take a while though).
The correct procedure would be:
* turn off DNSsec on old registrar (and wait a day or two)
* update NS and/or migrate domain
* wait a while and make sure it works
* turn on DNSsec in CF dash and update DNSsec settings in the domain
It's not that DNSsec doesn't work -- it's doing exactly what it's supposed to be doing.
All plans come with support. Even the free plans (community, or email, the bot will deflect the request but if you email you're still stuck, you will get a reply _eventually_ (due to heavy support load, it can take a while though).
The correct procedure would be:
* turn off DNSsec on old registrar (and wait a day or two)
* update NS and/or migrate domain
* wait a while and make sure it works
* turn on DNSsec in CF dash and update DNSsec settings in the domain
It's not that DNSsec doesn't work -- it's doing exactly what it's supposed to be doing.
The correct steps are to completely disable DNSSEC and remove the relevant records. It is far too fragile for the majority of use cases. Effort should instead be put into deploying things like DNSCrypt that implement transport security and confidentiality.
Transport security is like HTTPS. DNSSEC was the equivalent of PGP signing every webpage. The former brings value to the end user, the latter not so much.
Even the government has issued memo M-18-23 ("Shifting From Low-Value to High-Value Work") that rescinds the requirements for the government to implement DNSSEC.
Transport security is like HTTPS. DNSSEC was the equivalent of PGP signing every webpage. The former brings value to the end user, the latter not so much.
Even the government has issued memo M-18-23 ("Shifting From Low-Value to High-Value Work") that rescinds the requirements for the government to implement DNSSEC.
I want both, DNSSEC and DNSCrypt.
WebPKI is a joke because it lacks name constraints and so isn't and can't be hierarchical. DNSSEC is a true PKI -- you can still have multiple roots if you like and don't trust ., but it's got name constraints, so whatever domain you graft an alternate PKI at, from there on down you get bound to that PKI. This is really, truly fantastic.
Add DANE and you have a complete replacement for WebPKI.
DNSCrypt is needed to increase confidentiality, it's cheaper than DNSQuic and such things. Unfortunately .'s and com's and major TLDs' NSes are unlikely to want to waste CPU cycles on any DNS confidentiality solution, and even if some TLDs did, unless clients use QName minimization, users gain no confidentiality -- . and the TLDs all have to adopt it.
The two, together, would be truly fantastic.
WebPKI is a joke because it lacks name constraints and so isn't and can't be hierarchical. DNSSEC is a true PKI -- you can still have multiple roots if you like and don't trust ., but it's got name constraints, so whatever domain you graft an alternate PKI at, from there on down you get bound to that PKI. This is really, truly fantastic.
Add DANE and you have a complete replacement for WebPKI.
DNSCrypt is needed to increase confidentiality, it's cheaper than DNSQuic and such things. Unfortunately .'s and com's and major TLDs' NSes are unlikely to want to waste CPU cycles on any DNS confidentiality solution, and even if some TLDs did, unless clients use QName minimization, users gain no confidentiality -- . and the TLDs all have to adopt it.
The two, together, would be truly fantastic.
DNSSEC is however the only way you can make TLS really work. The whole TLS ecosystem is dependent on CAs that are basically just a giant hack. They're signing a statement that they did a bunch of DNS resolutions at a point in time from different network vantage points (maybe, hopefully), and got consistent answers. DNSSEC+DANE lets you get the actual data you want (domain name->public key binding) from the root source, without needing the complicated middlemen.
DNSSEC+DANE is an affirmation by the United States government that you have followed a chain to an answer about a certificate pinning. Handshake (www.handshake.org, Trigger warning: crypto) will give you what you are talking about (CA non-reliance) anchored in the owner of the website itself.
If you aren't a fan of DV certificates (as you point out verified by resolution), you can always restrict your trust store to only CA certificates that sign EV certificates (verified by business records).
If you aren't a fan of DV certificates (as you point out verified by resolution), you can always restrict your trust store to only CA certificates that sign EV certificates (verified by business records).
The root zone doesn't change often. You can pin . or even run your own private . with pinned . content if you don't trust the root.
If you do, then you get MITM protection.
If you don't, but choose to use QName minimization, you still get a modicum of MITM protection: because the attacker would have to choose to get in the middle without having enough knowledge of whether a particular upcoming TLS connection (or whatever) will be of particular interest.
Really, DNSSEC is infinitely better than the WebPKI, even WebPKI+CT, especially when DNSSEC clients use QName minimization, and even more so when clients pin copies of . from time to time.
If you do, then you get MITM protection.
If you don't, but choose to use QName minimization, you still get a modicum of MITM protection: because the attacker would have to choose to get in the middle without having enough knowledge of whether a particular upcoming TLS connection (or whatever) will be of particular interest.
Really, DNSSEC is infinitely better than the WebPKI, even WebPKI+CT, especially when DNSSEC clients use QName minimization, and even more so when clients pin copies of . from time to time.
I mean when you are adding a new domain, CF can clearly tell if a domain already has dnssec on or not. Seems like something that should raise a warning to the user.
That would have been very helpful. The [Enable DNSSEC] button in the control panel is very assertive and confident through its casual and innocuous appearance.
Source: me, having lost access to my domain for 4 days for reasons that are not yet fully clear to me.
Source: me, having lost access to my domain for 4 days for reasons that are not yet fully clear to me.
DNSSEC is notorious for breaking things [1]. I use it on most of my domains, but I would not just 'enable' it on a domain that I cared about and that had real users without a lot of thought and planning. Nor should you.
[1] - https://ianix.com/pub/dnssec-outages.html
[1] - https://ianix.com/pub/dnssec-outages.html
I figured it would work since there were no problems for the handful of my domains using DNSSEC with the previous registrar. Maybe the button should come with a warning label. I'll certainly be a bit cautious from now on.
> DNSSEC is notorious for breaking things
My understanding is that people break things.
My understanding is that people break things.
https://en.wikipedia.org/wiki/Just_culture
> Just culture is a concept related to systems thinking which emphasizes that mistakes are generally a product of faulty organizational cultures, rather than solely brought about by the person or persons directly involved. In a just culture, after an incident, the question asked is, "What went wrong?" rather than "Who caused the problem?".
Prominent (and very effective) example: Aviation safety.
DNSSEC is both easy to break and hard to fix. https://sockpuppet.org/blog/2015/01/15/against-dnssec/
> Just culture is a concept related to systems thinking which emphasizes that mistakes are generally a product of faulty organizational cultures, rather than solely brought about by the person or persons directly involved. In a just culture, after an incident, the question asked is, "What went wrong?" rather than "Who caused the problem?".
Prominent (and very effective) example: Aviation safety.
DNSSEC is both easy to break and hard to fix. https://sockpuppet.org/blog/2015/01/15/against-dnssec/
So when organizations hire people that do not understand the DNS or PKC to maintain their DNS then it is the organization's fault (rather than the person who made the change). I accept that and agree.
But if a bunch of large, well-staffed, engineering-focused, otherwise competent organizations manage to fuck it up regularly, the problem's probably above the individual organizations. Potentially with the spec itself.
I've seen far more failed certificate renewal failures than DNSSEC failures from the same teams you appear to be suggesting are perfect and the standard is flawed.
There's no property of DNSSEC that makes it prone to breaking (and really any real problem on your link applies just as well to HTTPS). It just breaks because those large entities don't care about fixing it or care a big deal about breaking it on purpose.
Sure, everyone makes mistakes, but there’s still a difference between juggling rubber balls and juggling knives.
But I have seen professional knife jugglers, and none made a mistake.
> what's the fastest way to get technical assistance when on a free plan?
Upgrading to a non-free plan?
You don't have to upgrade to enterprise, but even their $20/mo plan comes with support.
(Also, I hate to victim-blame here but using DNSSEC was a bad idea in the first place)
Upgrading to a non-free plan?
You don't have to upgrade to enterprise, but even their $20/mo plan comes with support.
(Also, I hate to victim-blame here but using DNSSEC was a bad idea in the first place)
Why was enabling DNSSEC a bad idea? Clearly the origin registrar isn't handling DNSSEC requests properly, but the OP should still be able to revert to non-DNSSEC without issues.
This post makes a better argument than I could write. And nothing much has changed in the seven years since. https://sockpuppet.org/blog/2015/01/15/against-dnssec/
That article is a bad example of outdated arguments and whataboutisms. Sure, DNSSEC has some issues, but none are so bad that it means you shouldn't use it: It does help resolvers detect various attacks (e.g. cache poisoning, BGP hijacks, MitMs), which is fairly critical for security-minded organizations.
(from that article, that is from 2015 and woefully outdated)
> With TLS properly configured, DNSSEC adds nothing.
This is false. DNSSEC adds address lookup security through response integrity, whereas TLS (only!) adds transport layer security to the endpoint you're connected to (hence the name). If you find a record in DNS with DNSSEC enabled, you know that the response is exactly as the sender intended it to be (and when connecting to the address returned for A- or AAAA-records, you'll be connecting to the intended IP address). Without DNSSEC, this is impossible to guarantee and record interception / MitM would be an attack vector.
Additionally, "With TLS correctly configured" also implies CAA being set up, which only can be done securely using DNSSEC. As to why CAA must be configured: Not all CAs are made the same; and re-routing network traffic is fairly doable if you only need to target one of the many public CAs. Targeting only that CA allowed in the CAA record is presumably much harder.
> Securing DNS lookups isn’t a high-priority task
> DNSSEC’s real job is thus to replace the TLS CA system. This plan is called DANE.
No, DNSSEC's job _is_ to secure DNS lookups. DANE is only one scheme that is made possible by DNSSEC; Secure CAA checking being another.
> Real-world DNSSEC therefore relies on RSA with PKCS1v15 padding.
Correct, but also relies on Ed25519 and P-256. A lot of authorative servers are still using the legacy RSA keys, but another lot is using P-256 and Ed25519 too.
> [sections] DNSSEC is Expensive To Adopt / Deploy
This is partly true, but any security is expensive to adopt/deploy. DNSSEC is fairly easy nowadays, though, with many hosted DNS services providing some form of DNSSEC.
> DNSSEC doesn’t secure browser DNS lookups.
It would, if you allowed your browser to recurse.
> DNSSEC is Unsafe
> Authenticated denial. Offline signers. Secret hostnames. Pick two.
That's fine. Secrecy doesn't add security; Authenticated denial and Offline signers do.
> DNSSEC is Architecturally Unsound
I disagree with the conclusion here. Sure, it might be useful for US gov to be the writer of the spec, but what public scrutiny DNSSEC has had implies that the security part is sound.
(from that article, that is from 2015 and woefully outdated)
> With TLS properly configured, DNSSEC adds nothing.
This is false. DNSSEC adds address lookup security through response integrity, whereas TLS (only!) adds transport layer security to the endpoint you're connected to (hence the name). If you find a record in DNS with DNSSEC enabled, you know that the response is exactly as the sender intended it to be (and when connecting to the address returned for A- or AAAA-records, you'll be connecting to the intended IP address). Without DNSSEC, this is impossible to guarantee and record interception / MitM would be an attack vector.
Additionally, "With TLS correctly configured" also implies CAA being set up, which only can be done securely using DNSSEC. As to why CAA must be configured: Not all CAs are made the same; and re-routing network traffic is fairly doable if you only need to target one of the many public CAs. Targeting only that CA allowed in the CAA record is presumably much harder.
> Securing DNS lookups isn’t a high-priority task
> DNSSEC’s real job is thus to replace the TLS CA system. This plan is called DANE.
No, DNSSEC's job _is_ to secure DNS lookups. DANE is only one scheme that is made possible by DNSSEC; Secure CAA checking being another.
> Real-world DNSSEC therefore relies on RSA with PKCS1v15 padding.
Correct, but also relies on Ed25519 and P-256. A lot of authorative servers are still using the legacy RSA keys, but another lot is using P-256 and Ed25519 too.
> [sections] DNSSEC is Expensive To Adopt / Deploy
This is partly true, but any security is expensive to adopt/deploy. DNSSEC is fairly easy nowadays, though, with many hosted DNS services providing some form of DNSSEC.
> DNSSEC doesn’t secure browser DNS lookups.
It would, if you allowed your browser to recurse.
> DNSSEC is Unsafe
> Authenticated denial. Offline signers. Secret hostnames. Pick two.
That's fine. Secrecy doesn't add security; Authenticated denial and Offline signers do.
> DNSSEC is Architecturally Unsound
I disagree with the conclusion here. Sure, it might be useful for US gov to be the writer of the spec, but what public scrutiny DNSSEC has had implies that the security part is sound.
I'm with you here. I automated my dnssec setup, and I barely think about it.
I really wish my bank would use it and stop calling javascript from domains unknown to me :)
I really wish my bank would use it and stop calling javascript from domains unknown to me :)
>but the OP should still be able to revert to non-DNSSEC without issues.
Slack had a 24 hour outage doing exactly that; so I don't think "revert to non-DNSSEC without issues" is trivial at all.
Slack had a 24 hour outage doing exactly that; so I don't think "revert to non-DNSSEC without issues" is trivial at all.
Wasn't there a thread on HN within the last couple of months that says switching back when things go wrong is actually very difficult? Intermediate servers or something cache the DNSSEC issues and things tend to break for 24 hours at a time. Unfortunately I can't remember the details.
Can you please explain why DNSSEC was a bad idea in the first place? It worked perfectly fine with the old registrar.
It like https. A lot of people in the past viewed HTTPS as a terrible idea that just broke things, and every example where someone had their website go down because of broken certificates or mixed content was proof that https as a concept was broken. Usually people brought up x.509 or revocation lists as the definitive proof that https would never be common.
From a site reliability perspective HTTPS is still broken. Some 15yo OS can't access any site because it doesn't have the certificates or cipher suites. And as you mentioned we need to update certs, webservers and DNS all the time to keep up to date. We only put up with it because it protects users from from snoopers. But that means we live in an inadequate equilibrium. If we abolished mass surveillance rather than impeding it with technical measures then we wouldn't need encryption for read-only sites, we could have our cake and eat it too.
HTTPS also guarantees integrity of the content; even for read-only sites it's important to ensure content isn't modified, code isn't injected, etc. There's an argument that a protocol which does integrity-checks only (without data encryption) might be good, that's the NONE cipher in TLS but it was removed in TLS 1.3 I believe; but it wouldn't fundamentally change the problem with 15yo operating systems lacking the software support or whatever.
(More broadly though the operational overhead of software is really high these days in a lot of ways. I think that's true of anything, not just HTTPS, but there are a lot of other historical factors leading to that.)
I think it's a bit of a leap to suggest that just doing things like banning mass surveillance would magically make systems more stable or make 15yo operating systems suddenly relevant on the net again. We'd probably still need a lot of the stuff we have in place already. However, I suggest we try it anyway because there's only one way to find out and oh well we won't lose anything valuable anyway.
(More broadly though the operational overhead of software is really high these days in a lot of ways. I think that's true of anything, not just HTTPS, but there are a lot of other historical factors leading to that.)
I think it's a bit of a leap to suggest that just doing things like banning mass surveillance would magically make systems more stable or make 15yo operating systems suddenly relevant on the net again. We'd probably still need a lot of the stuff we have in place already. However, I suggest we try it anyway because there's only one way to find out and oh well we won't lose anything valuable anyway.
> If we abolished mass surveillance then we wouldn't need encryption for read-only sites, we could have our cake and eat it too.
Mass surveillance is not the only reason to have HTTPS everywhere. It protects not just from snoopers, but from MITM attacks.
Mass surveillance is not the only reason to have HTTPS everywhere. It protects not just from snoopers, but from MITM attacks.
Did you notice the "read only sites" part? MITM is hardly relevant for those.
You should still protect against MITM attacks even with read-only websites - not all attacks are based on stealing user input.
What's the threat model here?
Random examples of MITM attacks I could do on a read-only website:
* Inserting malicious JavaScript
* Changing content on trusted websites in order to mislead people
* Replacing downloadable application binaries with versions that contain malicious code
* Inserting malicious JavaScript
* Changing content on trusted websites in order to mislead people
* Replacing downloadable application binaries with versions that contain malicious code
Malicious JS can be served directly, e.g. via ad iframes. Injecting it into a low-stakes (read-only) site doesn't gain much, does it?
Points 2 and 3 are the same, they're about integrity which could be had cheaper with content-addressing (hashes uniquely identifying the content) rather than pulling in the full TLS+CA machinery.
Points 2 and 3 are the same, they're about integrity which could be had cheaper with content-addressing (hashes uniquely identifying the content) rather than pulling in the full TLS+CA machinery.
Your ISP inserts random javascript and pop-ups into HTTP sites to tell you that you're nearing your data cap and that you should go buy an additional-data-pack.
Like Airtel used to (still does?) in India.
Like Airtel used to (still does?) in India.
Ok, you have a site with signed firmware downloads. I mean, they are signed securely right? A user messing with the stream can only send you another signed firmware the device takes, and not anything they attempt to create (unless they guess your signing key somehow).
But, you make a mistake in firmware version XYZ and there is an RCE in it. So you pull it off your site and now XZZ is the latest version.
Only problem is, anyone that can MITM you can serve version XYZ that the client will accept and make the machine exploitable by an RCE.
But, you make a mistake in firmware version XYZ and there is an RCE in it. So you pull it off your site and now XZZ is the latest version.
Only problem is, anyone that can MITM you can serve version XYZ that the client will accept and make the machine exploitable by an RCE.
>Did you notice the "read only sites" part? MITM is hardly relevant for those.
I'm sorry but you're not thinking this through very carefully. People still care about authentication of read-only stuff, in the same way much (and it should be all) open source software, particularly from repositories, is signed these days. A great deal of mischief can be done by modifying info in flight, even ignoring privacy concerns entirely. Plenty of not just governmental but corporate bodies could benefit by being able to trivially rewrite whatever populations (or targeted segments of populations) read. Even ignoring what a vector it could be for other attacks. In principle, we could have some universal standard for signing and authenticating as unaltered websites without bothering to encrypt them. But frankly that seems pointless vs just having encryption as well.
Further, like all practical public crypto use in the face of adversaries, there is a lot of benefit from using it universally and thus "hiding in the herd". Otherwise, the mere usage of crypto itself is a signal, and also easier to target and block (not just technically but via laws). Whereas when it's just baked into literally everything that's much harder to outright infeasible, and also destroys that extra bit of signal.
This was all debated and considered extensively while the moves to universal HTTPS were happening. People moved read-only sites as well for a reason.
I'm sorry but you're not thinking this through very carefully. People still care about authentication of read-only stuff, in the same way much (and it should be all) open source software, particularly from repositories, is signed these days. A great deal of mischief can be done by modifying info in flight, even ignoring privacy concerns entirely. Plenty of not just governmental but corporate bodies could benefit by being able to trivially rewrite whatever populations (or targeted segments of populations) read. Even ignoring what a vector it could be for other attacks. In principle, we could have some universal standard for signing and authenticating as unaltered websites without bothering to encrypt them. But frankly that seems pointless vs just having encryption as well.
Further, like all practical public crypto use in the face of adversaries, there is a lot of benefit from using it universally and thus "hiding in the herd". Otherwise, the mere usage of crypto itself is a signal, and also easier to target and block (not just technically but via laws). Whereas when it's just baked into literally everything that's much harder to outright infeasible, and also destroys that extra bit of signal.
This was all debated and considered extensively while the moves to universal HTTPS were happening. People moved read-only sites as well for a reason.
> This was all debated and considered extensively while the moves to universal HTTPS were happening. People moved read-only sites as well for a reason.
And the entire discussion was primarily motivated by pervasive surveillance. Which is the point I was making that we're living in a bad equilibrium (low trust society) where there is an attacker and there is a costly defense against the attacker that demonstrates that this particular kind of attack can be rendered useless but we cannot stop paying for the defense because as soon as we do the attacks would resume. If we could instead solve the regulatory problem and forbid surveillance then we would not have to pay that price.
> Further, like all practical public crypto use in the face of adversaries, there is a lot of benefit from using it universally and thus "hiding in the herd". Otherwise, the mere usage of crypto itself is a signal, and also easier to target and block (not just technically but via laws). Whereas when it's just baked into literally everything that's much harder to outright infeasible, and also destroys that extra bit of signal.
That's just a different angle on the surveillance, no? If we had no surveillance then nobody would be there to observe those bits of information you would leak by using or not using encryption.
> In principle, we could have some universal standard for signing and authenticating as unaltered websites without bothering to encrypt them. But frankly that seems pointless vs just having encryption as well.
There are plenty of benefits such as lower latency, much simpler zero-copy IO on the server side (sendfile), improved caching, less energy and silicon area wasted on encryption, less technological obsolescence, a smaller your-ciphersuite/OpenSSL-has-flaws maintenance treadmill. To some extent we could even do without CAs (via content-addressable data).
These may all be papercuts, but we're still getting cut because we can't collectively just tell the NSA to get off our lawn even though we have the means to keep them out anyway.
And the entire discussion was primarily motivated by pervasive surveillance. Which is the point I was making that we're living in a bad equilibrium (low trust society) where there is an attacker and there is a costly defense against the attacker that demonstrates that this particular kind of attack can be rendered useless but we cannot stop paying for the defense because as soon as we do the attacks would resume. If we could instead solve the regulatory problem and forbid surveillance then we would not have to pay that price.
> Further, like all practical public crypto use in the face of adversaries, there is a lot of benefit from using it universally and thus "hiding in the herd". Otherwise, the mere usage of crypto itself is a signal, and also easier to target and block (not just technically but via laws). Whereas when it's just baked into literally everything that's much harder to outright infeasible, and also destroys that extra bit of signal.
That's just a different angle on the surveillance, no? If we had no surveillance then nobody would be there to observe those bits of information you would leak by using or not using encryption.
> In principle, we could have some universal standard for signing and authenticating as unaltered websites without bothering to encrypt them. But frankly that seems pointless vs just having encryption as well.
There are plenty of benefits such as lower latency, much simpler zero-copy IO on the server side (sendfile), improved caching, less energy and silicon area wasted on encryption, less technological obsolescence, a smaller your-ciphersuite/OpenSSL-has-flaws maintenance treadmill. To some extent we could even do without CAs (via content-addressable data).
These may all be papercuts, but we're still getting cut because we can't collectively just tell the NSA to get off our lawn even though we have the means to keep them out anyway.
Examples of read only sites that adversaries might want to alter the content you see:
Your banks support numbers, election poll information, binary downloads/checksums are some really critical ones but the list is really endless given the wide range of possible adversaries and their motives.
One big advantage to pushing HTTPS everywhere is that we don't have to trust people to be able to correctly predict which read only content is sensitive.
I do wish browsers handled expired certs for longstanding sites in a way that was clearer to nontechnical people. We should have the ability to look at a project like the Internet Archive to know the history of a site in terms of both content and the certs it was served under.
Your banks support numbers, election poll information, binary downloads/checksums are some really critical ones but the list is really endless given the wide range of possible adversaries and their motives.
One big advantage to pushing HTTPS everywhere is that we don't have to trust people to be able to correctly predict which read only content is sensitive.
I do wish browsers handled expired certs for longstanding sites in a way that was clearer to nontechnical people. We should have the ability to look at a project like the Internet Archive to know the history of a site in terms of both content and the certs it was served under.
It really isn’t hard now there’s letsencrypt. We’ll never live in a world where a connection between the client and server can be completely trusted.
HTTPS is wonderful because it offers a guarantee that the data isn’t tampered with (except with corporate root CAs, but that is fuckery).
HTTPS is wonderful because it offers a guarantee that the data isn’t tampered with (except with corporate root CAs, but that is fuckery).
LetsEncrypt removes the cost of certificates, and ACME removes the work of getting certificate issues, and decent integrations remove the work of loading new certificates and that's all great.
But LE doesn't remove the compatability challenges. If you needed to ship a device today that would sit in a box for 10 years and then get online and get an update via https, that's really hard to do. TLS protocols sometimes get discouraged, and CA changes happen, etc.
But LE doesn't remove the compatability challenges. If you needed to ship a device today that would sit in a box for 10 years and then get online and get an update via https, that's really hard to do. TLS protocols sometimes get discouraged, and CA changes happen, etc.
Do you think criminals care about the law?
Criminals are much less likely to engage in MITM attacks, besides TLAs it's usually shady ISPs who want to inject some content (similar to surveillance that could be made illegal too, ISPs would in fact care). And criminals also have little incentive to attack read-only sites. Even if they did it might be more efficient to allocate resources to law enforcement rather than securing everything that could theoretically be attacked by criminals.
They're not attacking the sites, they're attacking the users. Incentives are exactly the same with readonly sites.
I'm a web programmer and I have no idea how the law enforcement could in any way help. Nor do I want them to. The idea that I have to cooperate with law enforcement to put a site online is absurd.
See "Tech support scams" on YouTube to see what's being done today. We're talking about billion-dollar crime organizations.
I'm a web programmer and I have no idea how the law enforcement could in any way help. Nor do I want them to. The idea that I have to cooperate with law enforcement to put a site online is absurd.
See "Tech support scams" on YouTube to see what's being done today. We're talking about billion-dollar crime organizations.
Another one for the list of attacking users....
You're updating the firmware on a server. The firmware is signed, so the attacker cannot outright put their own firmware on your system. The version you're using currently is secure, and the version you want to go to is secure, but there are versions in between that are insecure. All an attacker needs to do is modify the DNS and http stream to feed the firmware with an RCE to you, and then they can directly take over your server.
You're updating the firmware on a server. The firmware is signed, so the attacker cannot outright put their own firmware on your system. The version you're using currently is secure, and the version you want to go to is secure, but there are versions in between that are insecure. All an attacker needs to do is modify the DNS and http stream to feed the firmware with an RCE to you, and then they can directly take over your server.
Wordpress begs to differ. There are tons of examples of malicious JS on read only sites. Doesn’t have to be MitM. Usually it’s to generate ad views on another site, but can be more nefarious.
>Criminals are much less likely to engage in MITM attacks
Can you provide a source, or even just reasoning, to why this would be true? In my experience, MiTM is a common enough attack vector used by criminals.
Can you provide a source, or even just reasoning, to why this would be true? In my experience, MiTM is a common enough attack vector used by criminals.
> From a site reliability perspective HTTPS is still broken. Some 15yo OS can't access any site because it doesn't have the certificates or cipher suites
Sorry, but this argument doesn't hold water
And this coming from someone who supports systems still running NT 4
I have fallback rules enabled on all of my domains - TLS 1.3 is preferred, but older editions will be supported if the need arises (1.2, 1.1, and 1.0 (on a single domain))
Sorry, but this argument doesn't hold water
And this coming from someone who supports systems still running NT 4
I have fallback rules enabled on all of my domains - TLS 1.3 is preferred, but older editions will be supported if the need arises (1.2, 1.1, and 1.0 (on a single domain))
If your service must meet some strict certifications (e.g. PCI DSS), simply enabling old tls/ssl protocols for backward compatibility is not an option.
HTTPS is indeed broken when viewing it from a site reliability perspective. Anyone who has maintained more than a handful of domains simultaneously will agree (personally I’ve managed hundreds, each with their own certificate … it’s an awful experience).
Awful in what sense though? I also maintain many domains and have not touched them in years since their initial setup, with LetsEncrypt (and the Certbot renewal timer).
I've had several sites with HTTPS work for many years now with zero effort or SRE time. Let's Encrypt via certbot handles it all for me
It’s very sad that new SSL sites just don’t work on older computers that work just fine still. We’re not talking about complicated sites that wouldn’t work anyway without new browsers. Just basic HTML.
Eh, except that HTTPS actually has tangible benefits, unlike DNSSEC.
Much of the Internet right now uses DNS as proof of authentication. Having authentication system be a plain text protocol without any integrity or validation is a recipe for abuse. Right now the work-around is to have multiple resolver spread out all over the world and query the name servers multiple times to detect malicious actors, which is a much worse solution that dnssec if you ask me. It doesn't scale well and is a hack on top of an insecure protocol in order to create a sense of security.
We could return back to IPsec, or tunnel everything under https as a more modern version of IPsec, but those solutions are all disliked depending on who you ask.
We could return back to IPsec, or tunnel everything under https as a more modern version of IPsec, but those solutions are all disliked depending on who you ask.
Without DNSSEC anyone can intercept your email. The TLS cert verified by mail is the domain pointed to by the MX record. Plus with DKIM keys store in DNS people can spoof email (if they can fool the receiver to trust their records). If you can fool DNS resolution for LetsEncrypt (pretty hard since IIRC they fetch DNS from multiple perspectives on the internet to mitigate this) you can get certificates for any hostname.
There are other solutions such as MTA-STS and DNS-over-HTTP but the end-to-end validation of DNSSEC is pretty powerful.
There are other solutions such as MTA-STS and DNS-over-HTTP but the end-to-end validation of DNSSEC is pretty powerful.
DNSSEC doesn’t really solve any problems that you have nor does it meaningfully prevent any security risks.
It does create a lot of operational risk, as you’ve discovered. It also checks a box if you’re building a system for the US Federal .gov.
tptacek has written about this at length on this site and other places.
It does create a lot of operational risk, as you’ve discovered. It also checks a box if you’re building a system for the US Federal .gov.
tptacek has written about this at length on this site and other places.
Basically, it is lots of extra work effort for no real security advantages. Other people wrote a lot about it, here is an example:
https://sockpuppet.org/blog/2015/01/15/against-dnssec/
https://sockpuppet.org/blog/2015/01/15/against-dnssec/
[deleted]
Reading this hurts. I see that @elithrar has given out his email address and is following up but I will also be following this internally to understand what happened.
Since the issue was made public, we'd love to see some details in case OP (throwaway) doesn't come back.
> How can I get their attention without paying for an Enterprise plan?
Just comment on HN and they'll crawl out of the woodwork.
Just comment on HN and they'll crawl out of the woodwork.
Also, the community forums are generally quite useful when something isn't working. I've posted there with an actual problem maybe twice, but the problems were resolved within 30 minutes.
Which is a bit strange. I would have guessed that the support ticket system would be prioritized by staff.
English language usage note: "since" takes a past point in time, not a duration. You want either "unreachable for 4 days", or "unreachable since 4 days ago".
I had the same problem.
I registered a domain at Google Domains.
Then I configured the domain at CloudFlare.
At first it worked OK then I started getting SERVFAIL.
I found the problem was there was still DNSSEC configuration set up at Google Domains. I deleted that and everything worked OK.
Cloudflare was not at fault in my case.
I registered a domain at Google Domains.
Then I configured the domain at CloudFlare.
At first it worked OK then I started getting SERVFAIL.
I found the problem was there was still DNSSEC configuration set up at Google Domains. I deleted that and everything worked OK.
Cloudflare was not at fault in my case.
> How can I get their attention without paying for an Enterprise plan?
By paying for the cheapest plan, or any plan at all for that matter.
By paying for the cheapest plan, or any plan at all for that matter.
Enterprise plans no longer come with "premium" support either, you are looking at 20% over contract value to get a similar level of previously included support and an SLA. To be fair, CloudFlare provides a lot of services for free and $20 premium plan with upgraded support seems like a pretty good deal!
For any form of business endeavour that would be the obvious lowest starting point. For private users a $20/mo. price tag for registering one or a few domains would turn effective TLD pricing on its head. Suddenly owning a single .net domain is no longer $20 per year like it is with all the other thousand registrars, but instead it would be $260 per year.
Cloudflare's kind policy of zero markup on domain registrations on a free plan is remarkably generous. OK, sure, the traffic data has an obvious value to them, but maybe the support environment could improve with, I dunno, a tiny 5% markup.
Cloudflare's kind policy of zero markup on domain registrations on a free plan is remarkably generous. OK, sure, the traffic data has an obvious value to them, but maybe the support environment could improve with, I dunno, a tiny 5% markup.
I had a problem with a similar effect some time ago but I run my own DNS (no Cloudflare). I accidentally clicked a button in my control panel to regenerate the zone keys, which means the published keys mismatched the new zone signature for a couple of days until I was able to get the registrar to update them and everything propagated (even when a registrar supports the .eu TLD they are usually severely lacking in automation). The control panel devs have since added a confirmation dialog!
Sorry to hear about your problem, a quick recommendation would be to keep the temporary DNS name you bought and simply redirect to your previous name once you have the issue resolved (or vice versa if the branding is less important to you.) This way your users won't need to know or care about the change anymore aside from this temporary setback.
>I've been forced to migrate the project and its (few) users to a completely different domain. I cannot inconvenience users by bouncing them back and forth, so the domain Cloudflare ruined for me is now effectively lost, as is the "branding" of the project which was reflected in the domain's name.
If this was that important then you should not have used the free plan.
If this was that important then you should not have used the free plan.
Why do you presume the issue would have gotten immediate attention for the sum of $20? Customers don't make Cloudflare's terms, and customers didn't decide for Cloudflare to offer a free plan with zero markup for their registrar operations.
There is by users' own hands no way out of domain registration issues like these, sooner than 30-45 days when the domain can be transferred once again. Those who decide to offer registrar services, even for free, must hold some liability towards the users and the ecosystem and offer some support to make sure their product actually works.
There is by users' own hands no way out of domain registration issues like these, sooner than 30-45 days when the domain can be transferred once again. Those who decide to offer registrar services, even for free, must hold some liability towards the users and the ecosystem and offer some support to make sure their product actually works.
They are also the only domain registrar that I've interacted with this decade that does not allow you to set your own nameservers. Effectively locking you into using cloudflare DNS and related services until you can transfer again.
Its frankly, disgusting.
Its frankly, disgusting.
I'm also noticing that Cloudflare support is going terribly downhill.
I have an issue with the Cloudflare infrastructure on my domain since WEEKS, giving me thousands of 503 Service Temporarily Unavailable errors per day (cloudflare side, not the origin server) and nobody seems to care or able to resolve.
Removing the ability to create support tickets on free plan doesn't help at all, I mean, I get it why they're doing it, but asking on their community forum as an alternative it's not an acceptable solution. Neither going after Cloudflare employees on social media platforms hoping for a reply.
If I'm also going to pay for their services such as Zero Trust, domains registrar and R2, why do I have to switch to a Pro plan just to open a support ticket? Perhaps a middle-ground solution like 1 free support ticket per month on a free plan would be a good compromise?
I still think they're giving an incredible service and value for free, but this sucks.
I have an issue with the Cloudflare infrastructure on my domain since WEEKS, giving me thousands of 503 Service Temporarily Unavailable errors per day (cloudflare side, not the origin server) and nobody seems to care or able to resolve.
Removing the ability to create support tickets on free plan doesn't help at all, I mean, I get it why they're doing it, but asking on their community forum as an alternative it's not an acceptable solution. Neither going after Cloudflare employees on social media platforms hoping for a reply.
If I'm also going to pay for their services such as Zero Trust, domains registrar and R2, why do I have to switch to a Pro plan just to open a support ticket? Perhaps a middle-ground solution like 1 free support ticket per month on a free plan would be a good compromise?
I still think they're giving an incredible service and value for free, but this sucks.
There's also no real support for paid Warp+ access. The app's bug reporter seems to go to the app development team, who don't seem to be triaging reports much lately. That team has never responded well to service quality reports, seeing them outside their responsibility. Which is largely true but being able to forward our reports somewhere would be nice.
Recently I had to reach out to @CloudflareSupport on Twitter to get my several day old report of bad Warp+ routing on the forums looked at. It was eventually fixed but it was done in silence and really wasn't something that should've happened in the first place. Nor has there been a followup report on what went wrong.
Recently I had to reach out to @CloudflareSupport on Twitter to get my several day old report of bad Warp+ routing on the forums looked at. It was eventually fixed but it was done in silence and really wasn't something that should've happened in the first place. Nor has there been a followup report on what went wrong.
> Removing the ability to create support tickets on free plan doesn't help at all
It’s weird; If I was affected by a bug on Cloudflare, my first instinct would not be to start giving them money in order to be allowed to inform them about it.
It’s weird; If I was affected by a bug on Cloudflare, my first instinct would not be to start giving them money in order to be allowed to inform them about it.
[deleted]
Can you email me ([email protected]) with details?
I can also confirm the support seems to be terrible at the moment, which is disappointing as in general I think cloudflare is a great service and I've had good experiences with most of the features (though the docs often leave a bit to be desired) and have been a user since cloudflare's early beta days.
We're on a pro plan and have had an outstanding support ticket since March 22nd. With the last cloudflare response being 19 days ago.
I can't seem to get cloudflare to talk directly to backblaze (it's a domain mapping issue) and playing the middle-man in a back and forth between cloudflare and backblaze support seems to be recipe for not getting things resolved promptly.
I know it's covid times and organizations may be short staffed but compare this to cloud66 support who implemented a whole code update to support a special edge case for a non-paying customer within 48 hours. That makes an almost 2 month old unresolved ticket seem a bit tired.
@jgrahamc I'll email you ticket details in case you'd like to take a look.
We're on a pro plan and have had an outstanding support ticket since March 22nd. With the last cloudflare response being 19 days ago.
I can't seem to get cloudflare to talk directly to backblaze (it's a domain mapping issue) and playing the middle-man in a back and forth between cloudflare and backblaze support seems to be recipe for not getting things resolved promptly.
I know it's covid times and organizations may be short staffed but compare this to cloud66 support who implemented a whole code update to support a special edge case for a non-paying customer within 48 hours. That makes an almost 2 month old unresolved ticket seem a bit tired.
@jgrahamc I'll email you ticket details in case you'd like to take a look.
I'm pleased to say that my problem has been promptly resolved now as well.
First problem - trusting Cloudflare :|
I've had nothing but problems with them personally
I know some people swear by them ... I'm in the "swear at them" camp
I've had nothing but problems with them personally
I know some people swear by them ... I'm in the "swear at them" camp
dnsviz.net is awesome.
Does your TLD definitely support DNSSEC?
Yes. It had DNSSEC enabled for over a year when it was with the old registrar.
You would usually need to disable DNSSEC, wait 24h, transfer, and then wait for at least 24h before enabling DNSSEC again.
Thanks for the info, I'll keep it in mind for eventual future transfers. But shouldn't I be able to disable DNSSEC regardless, instead of the domain being stuck in limbo and hijacked by what appears to be a deadlock type of bug?
Talking as a admin at a registrar, Yes, Indeed Yes, you should be able to disable DNSSEC regardless.
DNSSEC signed is basically just that the TLD servers has a DS record listed for the domain. In order to remove dnssec you remove the DS record. This can be easy or hard depending on the interface that the TLD, but in theory very simple.
The reason why its recommended to remove dnssec before transfer is to allow caches to timeout with the old DS record to expire. Some TLD also automatically remove DS when you do a transfer and a name server change, as it is a rather clear signal that the old key won't be useful. There is however some exciting new technology called multi-signer which is intended to resolve this problem in the future.
DNSSEC signed is basically just that the TLD servers has a DS record listed for the domain. In order to remove dnssec you remove the DS record. This can be easy or hard depending on the interface that the TLD, but in theory very simple.
The reason why its recommended to remove dnssec before transfer is to allow caches to timeout with the old DS record to expire. Some TLD also automatically remove DS when you do a transfer and a name server change, as it is a rather clear signal that the old key won't be useful. There is however some exciting new technology called multi-signer which is intended to resolve this problem in the future.
Disabling DNSSEC doesn't propagate instantly. Have you queried the CF nameservers for the domain directly? In my experience everything involving DNSSEC requires a 24h wait (unless the domain hasn't been queried from anywhere - but that's usually not the case, something might have triggered distributed DNS lookups e.g. LE doing DNS validation for cert issuance etc).
CF's authorative servers ("hasslo" and "crystal") respond correctly when queried directly, but that doesn't really help the situation.
Then it sounds like you are caught in cache limbo. It might be prudent for CF to have their DNSSEC setup so that users can't disable it instantly (or enable again instantly) and have a minimum of 12/24/48h between changing DNSSEC state. I'd guess that by now most caching DNS resolvers might have different signatures (old registrar, first CF DNSSEC and second CF DNSSC).
Are there any case where DNSSEC can be kept enabled? I though it need to be disabled for transferring.
There is, but it requires cooperation between everyone and double signing between the old and new hosting service (for a short period of time). That rarely works out in the real world.
[deleted]
There are some tools being worked on: https://github.com/DNSSEC-Provisioning/Multi-signer
It shouldn't be a problem, at least i never had one, if you don't change the authoritative DNS servers.
-- removed. My apologies. --
My domain and users have ended up in limbo beyond anyone's but Cloudflare's control. I cannot transfer it back to the working registrar, or I would without being "angry at some free service". Why do you think berating me with snide remarks is helpful?
My apologies if you read it as a snide remark. I'm usually baffled when support is demanded on free stuff, but i see your point in this particular scenario.
When it comes to dnssec, the keys are handed down through the same layer of delegation that gives you your nameservers. This is out of Cloudflare's hands as well, you have bad data up through the registry and out to (your tld's) root nameservers. Just like changes to your NS record delegation don't take immediate effect, these keys don't take immediate effect either.
It's still pretty early days for DNSSEC, if you're going to use it it's worthwhile to know a lot about it. Just look at the several Slack outages caused by their attempts to implement it. Eventually the tooling will catch up, and registrars will all give you warnings about moving DNS and registration and the importance of syncing up your keys but we just aren't there yet.
It's still pretty early days for DNSSEC, if you're going to use it it's worthwhile to know a lot about it. Just look at the several Slack outages caused by their attempts to implement it. Eventually the tooling will catch up, and registrars will all give you warnings about moving DNS and registration and the importance of syncing up your keys but we just aren't there yet.
If free implies "if it breaks don't ask us anything, hope it works" they should state so when you signup. Pretty sure it's not the case.
Not really a service if it doesn't work. :)
Last week I transferred a domain used for a personal project from my old registrar to Cloudflare. After the transfer was finalized and new NS records had propagated, everything resolved normally and everything was working fine. I then enabled DNSSEC, and after a while the domain would no longer resolve. Every DNS server I try - Google, Quad9, OpenDNS, even Cloudflare's own DNS on 1.1.1.1 - returns SERVFAIL. The excellent diagnostic tool on dnsviz.net tells me that the domain is returning bogus DNSKEY/DS/NSEC responses and bogus delegation status. "no SEP matching the DS found".
I tried canceling the DNSSEC setup and waiting for over a day, with no effect. I re-enabled DNSSEC setup and waited for 3 days, with no effect. Cloudflare's control panel has since several days now been saying that DNSSEC will be enabled "in the next 24 hours". My site cannot be reached, and Cloudflare's support cannot be reached.
I've been forced to migrate the project and its (few) users to a completely different domain. I cannot inconvenience users by bouncing them back and forth, so the domain Cloudflare ruined for me is now effectively lost, as is the "branding" of the project which was reflected in the domain's name.
How can I get their attention without paying for an Enterprise plan? I would like to think that basic functional service should be accessible even when using Cloudflare only as a registrar with fundamental DNS on a free plan.