Comcast Shot Themselves in the Foot with MTA-STS(agwa.name)
agwa.name
Comcast Shot Themselves in the Foot with MTA-STS
https://www.agwa.name/blog/post/comcast_shot_themselves_in_the_foot_with_mta-sts
63 comments
It's not that people don't "like" DNSSEC.
The author themselves states in a different post on their blog that DNSSEC is avoided like the plague because it is a royal pain in the ass to set up and outages at major corporations (coughSlackcough) that clearly have talented, knowledgeable, experienced ops people show that the standard was not designed with practical implementation in mind.
"Security that is difficult to implement and can cause catastrophic-level outages not used by majority of internet; film at 11."
Instead of lecturing everyone on how they should be using DNSSEC, what we should be doing is working on "DNSSEC 2.0", from lessons learned in the 2-3 decades people have been trying to use DNSSEC.
The author themselves states in a different post on their blog that DNSSEC is avoided like the plague because it is a royal pain in the ass to set up and outages at major corporations (coughSlackcough) that clearly have talented, knowledgeable, experienced ops people show that the standard was not designed with practical implementation in mind.
"Security that is difficult to implement and can cause catastrophic-level outages not used by majority of internet; film at 11."
Instead of lecturing everyone on how they should be using DNSSEC, what we should be doing is working on "DNSSEC 2.0", from lessons learned in the 2-3 decades people have been trying to use DNSSEC.
Exactly this. DNSSEC is needlessly complicated in large part because its service model was fixed in the mid-late 1990s, when the conventional wisdom was that DNS servers would never be powerful enough to actually perform cryptographic operations. A true DNSSEC 2.0 could simply forego the concept of offline signers (which has largely been foregone already, because online signing is necessary to serve secure NXDOMAINs without revealing all your hostnames) and sign/encrypt on demand.
But DNSSEC 2.0 is unlikely to happen, because DoH solves most of the real-world on-the-wire problems with DNS security. There have always been two visions for how to secure the DNS: the top-down way, with signatures rolling down from the roots, a la DNSSEC, and the bottom-up way, where leaf servers allow individual clients to make secure queries, and we gradually converge on a secure DNS as more and more leafs make secure queries available. Top-down won at the IETF, but bottom-up is going to win the market.
But DNSSEC 2.0 is unlikely to happen, because DoH solves most of the real-world on-the-wire problems with DNS security. There have always been two visions for how to secure the DNS: the top-down way, with signatures rolling down from the roots, a la DNSSEC, and the bottom-up way, where leaf servers allow individual clients to make secure queries, and we gradually converge on a secure DNS as more and more leafs make secure queries available. Top-down won at the IETF, but bottom-up is going to win the market.
aka DNSCurve.
It may turn out that DNSSEC is useful, for exactly one thing: downloading the root zone securely. This circumvents any refusal to do signing-on-the-server by the root zone server operators.
Here are the instructions: https://localroot.isi.edu/
also: https://datatracker.ietf.org/doc/html/rfc8806
It may turn out that DNSSEC is useful, for exactly one thing: downloading the root zone securely. This circumvents any refusal to do signing-on-the-server by the root zone server operators.
Here are the instructions: https://localroot.isi.edu/
also: https://datatracker.ietf.org/doc/html/rfc8806
See also: IPv6 instead of RFC 1710.
https://datatracker.ietf.org/doc/html/draft-ietf-sipp-ipae-t...
Twenty-six years later, we're still waiting for IPv6 to be considered "the Internet".
https://datatracker.ietf.org/doc/html/draft-ietf-sipp-ipae-t...
Twenty-six years later, we're still waiting for IPv6 to be considered "the Internet".
comcast.net is DNSSEC signed and Comcast was one of the first major ISPs to implement DNSSEC validation. Comcast's MX records are DNSSEC signed.
> dig +dnssec comcast.net MX
This has nothing to do with DNSSEC.
> dig +dnssec comcast.net MX
This has nothing to do with DNSSEC.
It has nothing to do with Comcast not liking DNSSEC.
But if the ecosystem as a whole was happy about DNSSEC and DANE, it's somewhat doubtful we'd have both specs and the involvement of https.
But if the ecosystem as a whole was happy about DNSSEC and DANE, it's somewhat doubtful we'd have both specs and the involvement of https.
We would not. There'd be no reason to have MTA-STS if there was universal DNSSEC deployment. Of course, outages like this would be vastly more common, because DNSSEC includes a large superset of the failure modes of MTA-STS.
Does anyone know why SNI style approach is not used for email? The client could indicate in the TLS handshake (or in STARTTLS command) that it wants to send email to comcast.com and the server would use comcast.com certificate to prove that it is authentic destination for those emails. That way you'd avoid this whole mess of needing to fetch another file using another protocol based on another DNS record etc. SNI seems to work well for web, so why not for email too?
This is a good question. It's a bit more complicated than just using SNI.
First is certificate management. There is currently no automated way for mail providers to obtain TLS certificates for domains whose MX records point to them - i.e. there is no SMTP/MX equivalent to the Baseline Requirement's "Agreed-Upon Change to Website" domain validation method, or ACME's HTTP-01 challenge type. And in general domain operators don't want their mail providers to have certificates which could be used to impersonate their websites.
The certificate problem could be solved with SRVName certificates, which permit certificates to be scoped to a particular service on a host. Combined with an automated SMTP/MX-based validation method, this would permit mail providers to easily and securely get certificates for the domains whose mail they host. But there are thorny security-critical compatibility problems which are blocking the deployment of SRVName certificates. I wrote about the problem near the end of this blog post: https://www.agwa.name/blog/post/avoiding_dnssec
Second is that you need a mechanism for servers to remember that they should use TLS with a particular recipient domain, similar to HSTS. This could probably be accomplished with a simple SMTP extension that's included in the EHLO response.
First is certificate management. There is currently no automated way for mail providers to obtain TLS certificates for domains whose MX records point to them - i.e. there is no SMTP/MX equivalent to the Baseline Requirement's "Agreed-Upon Change to Website" domain validation method, or ACME's HTTP-01 challenge type. And in general domain operators don't want their mail providers to have certificates which could be used to impersonate their websites.
The certificate problem could be solved with SRVName certificates, which permit certificates to be scoped to a particular service on a host. Combined with an automated SMTP/MX-based validation method, this would permit mail providers to easily and securely get certificates for the domains whose mail they host. But there are thorny security-critical compatibility problems which are blocking the deployment of SRVName certificates. I wrote about the problem near the end of this blog post: https://www.agwa.name/blog/post/avoiding_dnssec
Second is that you need a mechanism for servers to remember that they should use TLS with a particular recipient domain, similar to HSTS. This could probably be accomplished with a simple SMTP extension that's included in the EHLO response.
> And in general domain operators don't want their mail providers to have certificates which could be used to impersonate their websites.
> The certificate problem could be solved with SRVName certificates, which permit certificates to be scoped to a particular service on a host.
What about extended key usage extension, doesn't it function similarly without needing to fiddle with SANs? https://datatracker.ietf.org/doc/html/rfc5280#section-4.2.1....
> The certificate problem could be solved with SRVName certificates, which permit certificates to be scoped to a particular service on a host.
What about extended key usage extension, doesn't it function similarly without needing to fiddle with SANs? https://datatracker.ietf.org/doc/html/rfc5280#section-4.2.1....
Yeah it seems like EKU could be used for this. But I might be overlooking something.
You're not always talking directly to the destination mailserver when sending a message via SMTP.
What do you do when the mail server you're talking to (or its impersonator) doesn't offer startssl?
if the server does not have tls then mta-sts does not help much either?
edit: I actually read the spec, and apparently the listing of allowed servers is a side aspect of the spec, the main meat is opting in to tls overall; that was not immediately obvious from the article. In that context your question makes more sense. To answer more comprehensively, I don't see why similar mechanism that functions well with HTTP would not work here; that the server would send STS policy to the client after connection which the client then caches and consults on subsequent connections. This is how HSTS works.
edit: I actually read the spec, and apparently the listing of allowed servers is a side aspect of the spec, the main meat is opting in to tls overall; that was not immediately obvious from the article. In that context your question makes more sense. To answer more comprehensively, I don't see why similar mechanism that functions well with HTTP would not work here; that the server would send STS policy to the client after connection which the client then caches and consults on subsequent connections. This is how HSTS works.
I agree they could have added HAVESTS to the EHLO reply.
[deleted]
> MTA-STS is a relatively new standard that allows domain owners such as Comcast to opt in to authenticated encryption for their mail servers. (By default, SMTP traffic between mail servers uses opportunistic encryption, which can be defeated by active attackers to intercept email.) MTA-STS requires the domain owner to duplicate their MX record (the DNS record that lists a domain's mail servers) in a text file served over HTTPS. Sending mail servers, like Alex's, refuse to contact mail servers that aren't listed in the MTA-STS text file. Since HTTPS uses authenticated encryption, the text file can't be altered by active attackers. (In contrast, the MX record is vulnerable to manipulation unless DNSSEC is used, but people don't like DNSSEC which is why MTA-STS was invented.)
Is this just a way of validating the MX record? That is, the public MX record in DNS can't be trusted, but because a cert verifies the identity of the sending mail server, the one they send can be trusted?
If someone is successfully impersonating the sender, what stops them from sending a false MX record. If they aren't impersonators, why not just trust them regardless - why do they need the MX record?
Obviously, I'm missing something.
Is this just a way of validating the MX record? That is, the public MX record in DNS can't be trusted, but because a cert verifies the identity of the sending mail server, the one they send can be trusted?
If someone is successfully impersonating the sender, what stops them from sending a false MX record. If they aren't impersonators, why not just trust them regardless - why do they need the MX record?
Obviously, I'm missing something.
This allows you to verify the mx record of the domain you're sending to, not the sender.
Thanks. Without trusting the MX record, how do you know from what address ask for the MX over HTTPS?
Maybe I should just look it up myself!
Maybe I should just look it up myself!
In theory, you cache a good copy from sometime before the attack.
This is for the receiver, not the sender.
Wouldn't the simplest work around be to just remove mx2h1.comcast.net and mx1a1.comcast.net from the MX records for at least 2592000 seconds and instead add the IP addresses of those mail servers vips to other existing DNS records that were properly in the STS file?
e.g.
e.g.
dig +short mx2h1.comcast.net
96.102.157.179
dig +short mx1a1.comcast.net
96.103.145.165
Comcast just needs to rotate the ID in the _mta-sts.comcast.net TXT record. That will cause sending mail servers to download the new policy.
Well that's way too easy. I supposed I improperly assumed that the results would be cached longer than the TTL of the TXT record based on the expiration in the well-known text file. In that case they should have only seen a tiny window of time that emails remained in queues. I will have to read up on the RFC for MTA-STS a bit more.
In that case I would suggest they did not really shoot themselves in the feet. Rather they may have stubbed their toe.
In that case I would suggest they did not really shoot themselves in the feet. Rather they may have stubbed their toe.
Oh boy. I sure hope you guys aren’t using your ISP’s email address.
I'm not, but far too many folks do as a matter of course.
That comcast.net email address is only good for storing all the DMCA complaints that pile up and get automatically deleted.
Good Lord, someone took the dumpster fire that was HSTS and made it for mail servers? Heaven help us all and have mercy on our souls.
I wanted to then follow up with an insightful or interesting comment but I am at a loss.
I wanted to then follow up with an insightful or interesting comment but I am at a loss.
HSTS itself is fine? It just means "please only contact me over TLS for X amount of time". The only complaint I heard about was adtech companies using it for fingerprinting.
Are you sure you're not thinking of HPKP?
Are you sure you're not thinking of HPKP?
Until an application chooses to enable HSTS, and their only documented way to update the certificate is the web interface... and there's HSTS. I lost a weekend to HSTS not long ago.
Essentially the problem with HSTS is it's a malicious choice to remove user agency because the spec intentionally prohibits web browsers from allowing a bypass option, despite the browser being an agent of the user, not the server. It also tells you to talk to your administrator, which is unhelpful when you are the administrator but the browser refuses to allow the administrator to continue.
Every time I've encountered HSTS in practice, it's been because of a perfectly normal certificate error I'm aware of the reason for (usually expiration) and have no real concerns about bypassing, except my own web browser won't let me.
The problem with HSTS (and MTA-STS) is that these are specs designed for internet centralization, where Google or Microsoft controls everything and manages everything[1], and a certificate never lapses. It's not designed for the real world, and neither of these specs would've made it except for that big tech has complete capture of web standards.
[1] Didn't know who the authors were when writing this comment, but the authors of the MTA-STS RFC are in fact, Google, Google, Microsoft, Comcast, and Yahoo.
Essentially the problem with HSTS is it's a malicious choice to remove user agency because the spec intentionally prohibits web browsers from allowing a bypass option, despite the browser being an agent of the user, not the server. It also tells you to talk to your administrator, which is unhelpful when you are the administrator but the browser refuses to allow the administrator to continue.
Every time I've encountered HSTS in practice, it's been because of a perfectly normal certificate error I'm aware of the reason for (usually expiration) and have no real concerns about bypassing, except my own web browser won't let me.
The problem with HSTS (and MTA-STS) is that these are specs designed for internet centralization, where Google or Microsoft controls everything and manages everything[1], and a certificate never lapses. It's not designed for the real world, and neither of these specs would've made it except for that big tech has complete capture of web standards.
[1] Didn't know who the authors were when writing this comment, but the authors of the MTA-STS RFC are in fact, Google, Google, Microsoft, Comcast, and Yahoo.
This is a weird analysis. HSTS allows site operators to stand up sites that are HTTPS-only. They ultimately choose the services they want to offer, not you (or else every webserver would just be a remote shell).
Standing up sites that are HTTPS only are a matter of not opening port 80 on your server. HSTS is an action to force access to fail even when you know the content being served, know the server is correct, and understand the issues with certificates.
HSTS would be an annoyance, but still manageable if not for the complete removal of our user agency. I think that's really where HSTS moves from a security specification to an abhorrent abuse of monopoly control of web standards. It's less about protecting the user and more about giving Google and such power over what you can and can't view with your browser.
HSTS would be an annoyance, but still manageable if not for the complete removal of our user agency. I think that's really where HSTS moves from a security specification to an abhorrent abuse of monopoly control of web standards. It's less about protecting the user and more about giving Google and such power over what you can and can't view with your browser.
No, it's plainly not as simple as "not opening port 80", because an attacker can spoof the server side of port 80. This attack, "SSL stripping", is literally the reason we have HSTS.
I think you are missing my core point: That the web browser, as my agent, should do what I tell it. Which means any browser that refuses to load a site because of a cert issue because of HSTS, that browser has been hijacked by the server operator.
For any other certificate issue, my browser will allow me to bypass it if I know what I am doing. But because HSTS has hijacked the browser, it will no longer act in my stead unless I fork the browser and recompile it. I consider HSTS to be a malicious specification in this respect.
For any other certificate issue, my browser will allow me to bypass it if I know what I am doing. But because HSTS has hijacked the browser, it will no longer act in my stead unless I fork the browser and recompile it. I consider HSTS to be a malicious specification in this respect.
Your web browser should do what you tell it to, but it doesn't follow that people who provide apps are obligated to allow you to speak HTTP to them. I don't know what a "malicious specification" is, but your logic about how to lock apps into HTTPS (by simply not listening on 80/tcp) doesn't hold, at all.
That's not the request being made. What people want is for the button that says "this expired cert is fine, whatever" to still be available for hsts https sites.
I guess I don't understand what that has to do with HSTS, which was not designed for that button.
Basically, on any ordinary HTTPS site, every browser can allow you to proceed despite a certificate error. Such as knowing the certificate is expired or revoked or not from a known CA or something.
HSTS by specification prohibits browsers from offering that option. So I can know a server is serving valid, authentic traffic via HTTPS, be aware of why the certificate has an issue, and HSTS requires that my browser prohibit me from connecting to the server.
HSTS by specification prohibits browsers from offering that option. So I can know a server is serving valid, authentic traffic via HTTPS, be aware of why the certificate has an issue, and HSTS requires that my browser prohibit me from connecting to the server.
When an HSTS locked site has an expired cert, the browser takes away the "it's okay" button.
No, I get that. I get how you can reasonably argue about that feature. I don't get how you can argue that feature suggests we shouldn't stop SSL stripping at all.
HTTPS-only is fine, and I like it overall. But HSTS also cranks up certificate enforcement in a way that is sometimes excessive.
You could start by explaining why you think HSTS is a dumpster fire. The idea behind HSTS is to ensure that sites intended only to be used over HTTPS are used only over HTTPS; it's not the kind of thing you're supposed to toggle or fiddle with.
I have written a neighbor comment. But in short: It removes user agency in the browser, and more than often solely exists to interfere with troubleshooting for the administrator trying to fix the issue.
It exists because without HSTS, attackers can defeat HTTPS trivially by spoofing HTTP responses to attempts to visit the (unintended) "http" site.
Everyone can easily move to email at their own domain. A com domain is about $14/year and you can get free email hosting from your domain on zoho.
Google can cut off your account on a whim with no recourse and God forbid you want to escape your isp but keep your email address with them!
Google can cut off your account on a whim with no recourse and God forbid you want to escape your isp but keep your email address with them!
Right, but now you're on the hook for setting up MTA-STS yourself, which as this post shows is no easy task.
For sure. It's not trivial but I also wouldn't say it's difficult either, maybe more of a moderate difficulty? The policy file format is straight forward, as are the necessary TXT and CNAME records.
In my case, I use Cloudflare for my domain so I set up a Cloudflare Pages project to generate and host the policy file at the mta-sts subdomain which is nice because I don't have to worry about hosting or SSL/TLS.
As long as your provider supports MTA-STS, you're only on the hook for setting up the records and hosting the policy file.
In my case, I use Cloudflare for my domain so I set up a Cloudflare Pages project to generate and host the policy file at the mta-sts subdomain which is nice because I don't have to worry about hosting or SSL/TLS.
As long as your provider supports MTA-STS, you're only on the hook for setting up the records and hosting the policy file.
If you never make changes, that will work once it's set up.
But if you ever decide to change your mail provider, you'll have to remember to correctly update your MTA-STS policy. If your change to Cloudflare Pages doesn't propagate through the Cloudflare CDN right away, you could end up in the situation described in the blog post, with the TXT record updated before the text file, and other mail servers unable to send you email.
But if you ever decide to change your mail provider, you'll have to remember to correctly update your MTA-STS policy. If your change to Cloudflare Pages doesn't propagate through the Cloudflare CDN right away, you could end up in the situation described in the blog post, with the TXT record updated before the text file, and other mail servers unable to send you email.
If you're using a provider, you're not on the hook for it.
Unless I'm mistaken in how this is implemented (not just server side)
Unless I'm mistaken in how this is implemented (not just server side)
Your provider is on the hook for properly configuring TLS on their mail server, but you still need to publish an MTA-STS policy saying who your provider is.
Unless your provider provides MTA-STS endpoints that you can CNAME your _mta-sts and mta-sts subdomains to. I haven't seen anyone do this yet.
Unless your provider provides MTA-STS endpoints that you can CNAME your _mta-sts and mta-sts subdomains to. I haven't seen anyone do this yet.
Does using your own domain stop Google from cutting off your account? My understanding was that the biggest risk is ending up on a spam filter, and Google could easily do that to you with a small or less heard of domain.
Interesting. I'm curious if popular MTA software supports some kind of configuration to reload caches after some amount of MTA-STS failures.
Isn't by design it must cache for the amount of time specified in the TXT file? So it has to be durable.
Of course the MTA probably should provide ways for admins to override this. Maybe as simple as deleting a file, modifying a sql(ite) DB.
Of course the MTA probably should provide ways for admins to override this. Maybe as simple as deleting a file, modifying a sql(ite) DB.
> Isn't by design it must cache for the amount of time specified in the TXT file? So it has to be durable.
Kind of. The max_age in the text file is the max time a cached policy should be used if a new policy can't be retrieved (e.g. it's being blocked by an attacker). If a new policy can be retrieved over HTTPS, it's totally OK to use it even if the max_age of the cached policy isn't up yet.
Thus, tyingq's suggestion is reasonable. In fact, the RFC already tells mail servers to look for a new policy if just one MTA-STS failure occurs (https://datatracker.ietf.org/doc/html/rfc8461#section-5.1 step 3). The problem is that it checks for a new policy using the ID in the TXT record, which as explained in the blog post is unreliable.
Kind of. The max_age in the text file is the max time a cached policy should be used if a new policy can't be retrieved (e.g. it's being blocked by an attacker). If a new policy can be retrieved over HTTPS, it's totally OK to use it even if the max_age of the cached policy isn't up yet.
Thus, tyingq's suggestion is reasonable. In fact, the RFC already tells mail servers to look for a new policy if just one MTA-STS failure occurs (https://datatracker.ietf.org/doc/html/rfc8461#section-5.1 step 3). The problem is that it checks for a new policy using the ID in the TXT record, which as explained in the blog post is unreliable.
Thanks for the detailed explanation.
It seems silly to introduce that extra level of indirection when there's already a well-known URL to try to fetch a new version.
Is there any rationale for this? Also if DNS is explicitly a not trusted component this seems to strictly lead to a less secure system :o
(But I guess the designers did not want to "completely" override MX records just with yet another out of band magic.)
It seems silly to introduce that extra level of indirection when there's already a well-known URL to try to fetch a new version.
Is there any rationale for this? Also if DNS is explicitly a not trusted component this seems to strictly lead to a less secure system :o
(But I guess the designers did not want to "completely" override MX records just with yet another out of band magic.)
> Is there any rationale for this?
Do you mean for the TXT record? The rationale for that was performance. I don't know if this rationale was backed up by hard data or just a theoretical concern that HTTPS requests are more expensive than DNS requests.
> Also if DNS is explicitly a not trusted component this seems to strictly lead to a less secure system
The TXT record doesn't contain any security-sensitive information so it doesn't actually reduce security. The worst an attacker can do is block the retrieval of a new policy, but they could also accomplish this by removing the mta-sts A/AAAA record. Either way, the cached policy will be used until its max_age expires.
Do you mean for the TXT record? The rationale for that was performance. I don't know if this rationale was backed up by hard data or just a theoretical concern that HTTPS requests are more expensive than DNS requests.
> Also if DNS is explicitly a not trusted component this seems to strictly lead to a less secure system
The TXT record doesn't contain any security-sensitive information so it doesn't actually reduce security. The worst an attacker can do is block the retrieval of a new policy, but they could also accomplish this by removing the mta-sts A/AAAA record. Either way, the cached policy will be used until its max_age expires.
> Do you mean for the TXT record?
I mean that during the max_age time the MTA knows that the target MTA has/had a policy, so it could directly query that again instead of doing the DNS lookup. But probably the spec writers did not want to risk MTAs getting hammered with HTTPS requests on every failure.
> Either way, the cached policy will be used until its max_age expires.
Ah, thanks. I got confused by the DNS spoofing, but of course since the policy is protected by PKI TLS even if the attacker can redirect the MTA to evil.com/policy.txt the cert won't match, and the email recipient domain is already in the email to be delivered. (And if that can be spoofed, then the mail contents are also very likely accessible.)
I mean that during the max_age time the MTA knows that the target MTA has/had a policy, so it could directly query that again instead of doing the DNS lookup. But probably the spec writers did not want to risk MTAs getting hammered with HTTPS requests on every failure.
> Either way, the cached policy will be used until its max_age expires.
Ah, thanks. I got confused by the DNS spoofing, but of course since the policy is protected by PKI TLS even if the attacker can redirect the MTA to evil.com/policy.txt the cert won't match, and the email recipient domain is already in the email to be delivered. (And if that can be spoofed, then the mail contents are also very likely accessible.)
> the MX record is vulnerable to manipulation unless DNSSEC is used, but people don't like DNSSEC which is why MTA-STS was invented
This speaks volumes about the way these protocols are developed and subsequently implemented in software. People are not going to like complex solutions and that should be kept in mind when those protocols are being developed and implemented, otherwise its deployment is hampered just like we're seeing now with DNSSEC. In fact, now we've got MTA-STS because "people don't like DNSSEC", involving a whole HTTPS stack on both sides with so much more failure points just to send an email.