SpamChannel: Spoofing emails from 2M domains and virtually becoming Satan [pdf](media.defcon.org)
media.defcon.org
SpamChannel: Spoofing emails from 2M domains and virtually becoming Satan [pdf]
https://media.defcon.org/DEF%20CON%2031/DEF%20CON%2031%20presentations/byt3bl33d3r%20-%20SpamChannel%20Spoofing%20Emails%20From%202%20Million%20Domains%20and%20Virtually%20Becoming%20Satan.pdf
121 comments
I think these might be the demos that were unable to play during the talk:
https://www.youtube.com/watch?v=61PIOBp30vA
https://www.youtube.com/watch?v=eODw4t4WaCw
https://www.youtube.com/watch?v=61PIOBp30vA
https://www.youtube.com/watch?v=eODw4t4WaCw
SPF is broken in many more ways than described in this presentation. I work as an email hardening / deliverability support engineer and our advice is always to focus on DKIM + DMARC, rather than SPF. You still need SPF for legacy reasons, but it should never be relied on for deliverability or anti-impersonation.
Slide 54 says that DKIM + DMARC does not help against this attack, but that is not completely true.
If (and only if) you have set up DKIM for all your delegated senders, then (and only then) can you safely enable a DMARC p=reject policy. Once you have reached that level, you can start opting out of SPF for third party senders, by using the '?' (neutral) modifier in SPF.
So this:
Granted, it is not a perfect solution, but email will never be 100% reliable, or secure anyway.
Slide 54 says that DKIM + DMARC does not help against this attack, but that is not completely true.
If (and only if) you have set up DKIM for all your delegated senders, then (and only then) can you safely enable a DMARC p=reject policy. Once you have reached that level, you can start opting out of SPF for third party senders, by using the '?' (neutral) modifier in SPF.
So this:
v=spf1 include:relay.mailchannels.net ~all
v=spf1 ?include:relay.mailchannels.net ~all
Granted, it is not a perfect solution, but email will never be 100% reliable, or secure anyway.
What's wrong with SPF then?
Since source IP spoofing is rather tricky¹ with TCP, I've always wondered what advantage DKIM has over SPF. I've always just configured SPF to allow $myIP only: good luck sending spam in the name of my domain, you'd have to compromise my ISP or registrar first, at which point you can also get TLS certs for my domain for example.
Even for larger organisations which need to allowlist multiple sending systems, how would one spoof being one of the legitimate SPF sender in a situation where it is not also possible to spoof the DKIM records?
That is, besides allowlisting a range of IPs that anyone can publicly use (as in the submitted talk), which is just dumb.
¹ although not impossible, spoofing an entire mail exchange requires terabytes of traffic for one email of a few bytes, iff starttls is not enforced in which case it becomes impossible
Since source IP spoofing is rather tricky¹ with TCP, I've always wondered what advantage DKIM has over SPF. I've always just configured SPF to allow $myIP only: good luck sending spam in the name of my domain, you'd have to compromise my ISP or registrar first, at which point you can also get TLS certs for my domain for example.
Even for larger organisations which need to allowlist multiple sending systems, how would one spoof being one of the legitimate SPF sender in a situation where it is not also possible to spoof the DKIM records?
That is, besides allowlisting a range of IPs that anyone can publicly use (as in the submitted talk), which is just dumb.
¹ although not impossible, spoofing an entire mail exchange requires terabytes of traffic for one email of a few bytes, iff starttls is not enforced in which case it becomes impossible
A lot of normal email setups, e.g. forwarding, break SPF. As log as the message isn't modified, DKIM will work in these setups.
It's easy to have too wide of a range specified for SPF. You can't properly DKIM sign a mail unless you have a valid key (or you control DNS). As a medium/large corp, you might blanket allow mail from your whole IP space, but you're probably not pushing DKIM keys to all your hosts.
> What's wrong with SPF then?
- SPF breaks with forwarding. This is by far the biggest problem.
You typically end up with very large IP blocks being included in your policy (for example: MS365 includes a total of half a million IPv4 addresses, and billions of IPv6 addresses). All of which are now allowed to send on behalf of your domain. You'll have to trust (or hope) that the third party properly prevents other tenants from sending email on behalf of your domain. This is the problem with MailChannels as explained in the PDF.
- SPF can be very tricky to get right, one simple mistake and you either allow way to many IPs, or break your entire email domain. Even most seasoned IT administrators don't fully understand the entire RFC with all its errata and amendments. Now imagine some less technical user trying to set up SPF.
- You can only have one SPF record. If you want to add some external service, you'll need to merge your existing SPF record with the example given by your vendor. This can be highly confusing to many.
- The SPF DNS record is placed at the domain root (apex), which as shown in other comment here can be problematic for large orgs. This also prevents using a CNAME for SPF.
- SPF limits to 10 DNS lookups, which has proven far to low for modern use. The rules on how to calculate the number of required lookups is also ambiguous, so some receivers may accept your email, some may reject based on lookup limits.
- SPF contains many rarely used features (such as macros) that may or may not be implemented by the receiver, so you can't really rely on it.
- IP spoofing is an issue, but as you noted, this is not always a practical attack.
DKIM is in any regard more robust, more secure, and more important: easier to set up for the domain administrator. Just copy/paste the DKIM record provided by the third party email service. The third party vendor can (and most do) easily verify that the DKIM record is set up correctly. This is about as easy as it gets for DNS.
> Even for larger organisations which need to allowlist multiple sending systems, how would one spoof being one of the legitimate SPF sender in a situation where it is not also possible to spoof the DKIM records?
That is the problem with the (current) DMARC specification, you need SPF or DKIM to pass DMARC. Emphasis on or. So no matter how well you manage your DKIM keys, if someone can spoof (or has access to) one IP in your SPF policy, then the email is delivered. This is also explained in the talk.
Also, with DKIM you don't 'spoof' the record, you'd have to MITM the receiver when they fetch the DKIM DNS record, this is very hard to pull of in practice.
- SPF breaks with forwarding. This is by far the biggest problem.
You typically end up with very large IP blocks being included in your policy (for example: MS365 includes a total of half a million IPv4 addresses, and billions of IPv6 addresses). All of which are now allowed to send on behalf of your domain. You'll have to trust (or hope) that the third party properly prevents other tenants from sending email on behalf of your domain. This is the problem with MailChannels as explained in the PDF.
- SPF can be very tricky to get right, one simple mistake and you either allow way to many IPs, or break your entire email domain. Even most seasoned IT administrators don't fully understand the entire RFC with all its errata and amendments. Now imagine some less technical user trying to set up SPF.
- You can only have one SPF record. If you want to add some external service, you'll need to merge your existing SPF record with the example given by your vendor. This can be highly confusing to many.
- The SPF DNS record is placed at the domain root (apex), which as shown in other comment here can be problematic for large orgs. This also prevents using a CNAME for SPF.
- SPF limits to 10 DNS lookups, which has proven far to low for modern use. The rules on how to calculate the number of required lookups is also ambiguous, so some receivers may accept your email, some may reject based on lookup limits.
- SPF contains many rarely used features (such as macros) that may or may not be implemented by the receiver, so you can't really rely on it.
- IP spoofing is an issue, but as you noted, this is not always a practical attack.
DKIM is in any regard more robust, more secure, and more important: easier to set up for the domain administrator. Just copy/paste the DKIM record provided by the third party email service. The third party vendor can (and most do) easily verify that the DKIM record is set up correctly. This is about as easy as it gets for DNS.
> Even for larger organisations which need to allowlist multiple sending systems, how would one spoof being one of the legitimate SPF sender in a situation where it is not also possible to spoof the DKIM records?
That is the problem with the (current) DMARC specification, you need SPF or DKIM to pass DMARC. Emphasis on or. So no matter how well you manage your DKIM keys, if someone can spoof (or has access to) one IP in your SPF policy, then the email is delivered. This is also explained in the talk.
Also, with DKIM you don't 'spoof' the record, you'd have to MITM the receiver when they fetch the DKIM DNS record, this is very hard to pull of in practice.
Thanks! Your reply and its siblings have convinced me why DKIM is better overall; my opinion was changed, but there's one part that doesn't make much sense to me:
> and more important: easier to set up for the domain administrator. Just copy/paste the DKIM record provided by the third party email service
Pasting the same known value (my IP address) into a TXT record of the domains I want to send email from is an order of magnitude easier than
1. configuring the mail delivery software to ask opendkim for a signature, which in turn needs multiple config files and lookup tables for every domain
2. deploying a cryptographic key that doesn't even fit into one record, with various guides telling you different things on how you should be inputting the data
3. still failing the DKIM check because it turns out that 4k or even 3k rsa keys (let alone ecc! But I wasn't trying to be adventurous...) are not commonly enough supported and so you get to start over, wait another 24 hours for caches to flush, and ask another favor from that friend using that failing provider if they can send me the headers of a new test email to see if it passes now.
Even in your scenario where you hand your emails off to a third party, they can give you the SPF record to plug in literally, same as with DKIM, and then SPF is still easier because the input format doesn't depend on the registrar.
> and more important: easier to set up for the domain administrator. Just copy/paste the DKIM record provided by the third party email service
Pasting the same known value (my IP address) into a TXT record of the domains I want to send email from is an order of magnitude easier than
1. configuring the mail delivery software to ask opendkim for a signature, which in turn needs multiple config files and lookup tables for every domain
2. deploying a cryptographic key that doesn't even fit into one record, with various guides telling you different things on how you should be inputting the data
3. still failing the DKIM check because it turns out that 4k or even 3k rsa keys (let alone ecc! But I wasn't trying to be adventurous...) are not commonly enough supported and so you get to start over, wait another 24 hours for caches to flush, and ask another favor from that friend using that failing provider if they can send me the headers of a new test email to see if it passes now.
Even in your scenario where you hand your emails off to a third party, they can give you the SPF record to plug in literally, same as with DKIM, and then SPF is still easier because the input format doesn't depend on the registrar.
> Pasting the same known value (my IP address) into a TXT record of the domains I want to send email from is an order of magnitude easier than
For your own domain this may be true, but or the majority of domain the SPF record will be far more complex. Adding a new third party sender to an existing SPF record is error prone. Unless you work with these sorts of things on a regular basis, it can be very confusing.
> 1. configuring the mail delivery software to ask opendkim for a signature, which in turn needs multiple config files and lookup tables for every domain
99.9% of domains do not self-host their email (and rightfully so!). Generally you don't set up DKIM on the sending side yourself. All you typically need to do is copy/paste the provided DKIM DNS record (the public key) into your DNS zone. Most major email services will also verify for you of you did that correctly.
> 3. still failing the DKIM check because it turns out that 4k or even 3k rsa keys (let alone ecc! But I wasn't trying to be adventurous...)
The recommended key length for DKIM is 2048 bit for this very reason. Don't try to use >2048 bit keys since there are plenty of legacy email systems that can't handle keys larger than that. ECC keys are much shorter, but can't be relied on for the coming decades (or ever) since you'll have to deal with legacy systems.
> Even in your scenario where you hand your emails off to a third party, they can give you the SPF record to plug in literally, same as with DKIM, and then SPF is still easier because the input format doesn't depend on the registrar.
It's true (and very unfortunate) that DNS is treated as a second class citizen by most registrars. The UI is often horrible. However, I do want to point out that SPF records can also easily exceed the maximum UDP length, so you'll need to start splitting your SPF with includes, which can be confusing to many
Again, my job is to assist domain administrators with email infrastructure related stuff. So I'm telling from experience that most of the configuration errors come from SPF, not DKIM.
For your own domain this may be true, but or the majority of domain the SPF record will be far more complex. Adding a new third party sender to an existing SPF record is error prone. Unless you work with these sorts of things on a regular basis, it can be very confusing.
> 1. configuring the mail delivery software to ask opendkim for a signature, which in turn needs multiple config files and lookup tables for every domain
99.9% of domains do not self-host their email (and rightfully so!). Generally you don't set up DKIM on the sending side yourself. All you typically need to do is copy/paste the provided DKIM DNS record (the public key) into your DNS zone. Most major email services will also verify for you of you did that correctly.
> 3. still failing the DKIM check because it turns out that 4k or even 3k rsa keys (let alone ecc! But I wasn't trying to be adventurous...)
The recommended key length for DKIM is 2048 bit for this very reason. Don't try to use >2048 bit keys since there are plenty of legacy email systems that can't handle keys larger than that. ECC keys are much shorter, but can't be relied on for the coming decades (or ever) since you'll have to deal with legacy systems.
> Even in your scenario where you hand your emails off to a third party, they can give you the SPF record to plug in literally, same as with DKIM, and then SPF is still easier because the input format doesn't depend on the registrar.
It's true (and very unfortunate) that DNS is treated as a second class citizen by most registrars. The UI is often horrible. However, I do want to point out that SPF records can also easily exceed the maximum UDP length, so you'll need to start splitting your SPF with includes, which can be confusing to many
Again, my job is to assist domain administrators with email infrastructure related stuff. So I'm telling from experience that most of the configuration errors come from SPF, not DKIM.
We use cloudflare workers + mailchannels in production. Yikes.
We were already sorking on migrating away from cf workers to real servers, and now also we'll also have to move away from mailchannels I guess.
Security risks aren't worth the convenience.
We were already sorking on migrating away from cf workers to real servers, and now also we'll also have to move away from mailchannels I guess.
Security risks aren't worth the convenience.
Note that since June 2023, it is impossible to send email from Workers through MailChannels unless you publish a _mailchannels record in your DNS.
This doesn’t seem like a real fix for the issue. This brings the number of spoofable sites way down from all MC users to just those who send mail through CF. But it is still a big vulnerability.
For any domain that is using mailchannels through cloudflare, you can see what region they are using. And you can continue to spoof them. You just have to do it from the same region.
And this is terrible positioning from Cloudflare’s POV. Why would anyone send email through a CF worker since it requires advertising through a public record (DNS) that is by design accessible/scrapable by bots that you are using an insecure service. It’s like asking people to spoof you.
I don’t get why CF doesn’t do something more sensible, like limit sender addresses to domains that are already set up in the cloudflare account where the worker was created. Basically every other provider does this.
For any domain that is using mailchannels through cloudflare, you can see what region they are using. And you can continue to spoof them. You just have to do it from the same region.
And this is terrible positioning from Cloudflare’s POV. Why would anyone send email through a CF worker since it requires advertising through a public record (DNS) that is by design accessible/scrapable by bots that you are using an insecure service. It’s like asking people to spoof you.
I don’t get why CF doesn’t do something more sensible, like limit sender addresses to domains that are already set up in the cloudflare account where the worker was created. Basically every other provider does this.
The _mailchannels record includes the Cloudflare worker id, other workers can not send from that domain.
[deleted]
Source? The talk slides seem to contradict that:
>● CF workers will be required to set their “Domain Lockdown” record in order to send emails after a specific cutoff date (TBA). You’re still able to spoof all the things until then!
>● CF workers will be required to set their “Domain Lockdown” record in order to send emails after a specific cutoff date (TBA). You’re still able to spoof all the things until then!
> Source?
You're replying to the CEO of MailChannels
You're replying to the CEO of MailChannels
This seems like a hack. why doesn't mail channels do a proper authentication of the sender?
Ahh yes, another "standard". https://xkcd.com/927/
Absolutely. For reference, here is a TXT lookup for a random large corporation:
% host -t txt bosch.com bosch.com descriptive text "b5r8gr465ydgqlvwlwy6x7dshccwg37c" bosch.com descriptive text "mindmanager-verification=5eacd59de90dd7d9933da220294f2906a881fa6701d64a1a4667c05abac12546" bosch.com descriptive text "cisco-ci-domain-verification=20e8d94f041a742a6de560a038d031baf659330970e6f05bb03fb2468bba379a" bosch.com descriptive text "v=spf1 mx redirect=_spf.bosch.de" bosch.com descriptive text "r7bNBcUhS/tsO45+nP1dycu5UDrZ7TniKe858vhLNJwAcCNDQlpdaks8iBm3TxV9r3fCYRvup/QpaC1rdp4Dpg==" bosch.com descriptive text "google-site-verification=avGZ684w6lq0UwXmOHxz9l6u9GL8r-sXoV7io9KzZOc" bosch.com descriptive text "facebook-domain-verification=20380cm6fdq6h7tdcqzmhysa70ctdo" bosch.com descriptive text "google-site-verification=jWBLaKM6WZQoAV6crbGGsAre3rSvqaDtwnCKuwvDPCg" bosch.com descriptive text "adobe-idp-site-verification=e13fd7b27a30ca44261bc7ad3f0b2e0994b724830fe8afc38a6565a40d81bde4" bosch.com descriptive text "google-site-verification=yeXu97OTorw4QioF3yNiDvTn-0GL8__7-iTNLEp6Vbk" bosch.com descriptive text "docusign=4f8c3289-c1e1-42a5-a541-f3459d2da55e" bosch.com descriptive text "docusign=95ecd2d2-2737-4b07-9d40-95d065bfbc49" bosch.com descriptive text "atlassian-domain-verification=1XOeeaiW02aX/CXX/725tFzKFh4PA18JpZoyq9qBDqDxR2PP/9LDxCqlmYYgyb4D" bosch.com descriptive text "77D8-D4C6-1CCC-8D85-3127-5140-11D2-956B" bosch.com descriptive text "axway-amplify=af03bdb0-d4a8-429f-bf96-b7ba41051d49" bosch.com descriptive text "apple-domain-verification=VYw9jmavDBjP0C9S" bosch.com descriptive text "mongodb-site-verification=gf5347Wa7YvVkq4C51l3srxrco2GLGPl" bosch.com descriptive text "docusign=ebea8618-97e6-4a64-82a9-2e3fa036d5bb" bosch.com descriptive text "docusign=df164c89-5239-42d4-97f5-8f5710b1b929" bosch.com descriptive text "miro-verification=95bb46ca27717c388e091411d7fe643e3a3d2b3d" bosch.com descriptive text "docker-verification=224577a6-972f-4e1d-a8b9-c5fc2e8da018" bosch.com descriptive text "klaviyo-site-verification=THKvhQ" bosch.com descriptive text "atlassian-sending-domain-verification=e4597f31-7a29-47fd-b150-5e1eaeac437b" bosch.com descriptive text "sFHU8FVI0Jt2PIXJAn2DWGmT7UJmZJzyq2THJmabTvEcw0IGtPu2UHU9Wf/zdvZoGAvtmO5tD3rOSvXjQWZ57Q=="
% host -t txt bosch.com bosch.com descriptive text "b5r8gr465ydgqlvwlwy6x7dshccwg37c" bosch.com descriptive text "mindmanager-verification=5eacd59de90dd7d9933da220294f2906a881fa6701d64a1a4667c05abac12546" bosch.com descriptive text "cisco-ci-domain-verification=20e8d94f041a742a6de560a038d031baf659330970e6f05bb03fb2468bba379a" bosch.com descriptive text "v=spf1 mx redirect=_spf.bosch.de" bosch.com descriptive text "r7bNBcUhS/tsO45+nP1dycu5UDrZ7TniKe858vhLNJwAcCNDQlpdaks8iBm3TxV9r3fCYRvup/QpaC1rdp4Dpg==" bosch.com descriptive text "google-site-verification=avGZ684w6lq0UwXmOHxz9l6u9GL8r-sXoV7io9KzZOc" bosch.com descriptive text "facebook-domain-verification=20380cm6fdq6h7tdcqzmhysa70ctdo" bosch.com descriptive text "google-site-verification=jWBLaKM6WZQoAV6crbGGsAre3rSvqaDtwnCKuwvDPCg" bosch.com descriptive text "adobe-idp-site-verification=e13fd7b27a30ca44261bc7ad3f0b2e0994b724830fe8afc38a6565a40d81bde4" bosch.com descriptive text "google-site-verification=yeXu97OTorw4QioF3yNiDvTn-0GL8__7-iTNLEp6Vbk" bosch.com descriptive text "docusign=4f8c3289-c1e1-42a5-a541-f3459d2da55e" bosch.com descriptive text "docusign=95ecd2d2-2737-4b07-9d40-95d065bfbc49" bosch.com descriptive text "atlassian-domain-verification=1XOeeaiW02aX/CXX/725tFzKFh4PA18JpZoyq9qBDqDxR2PP/9LDxCqlmYYgyb4D" bosch.com descriptive text "77D8-D4C6-1CCC-8D85-3127-5140-11D2-956B" bosch.com descriptive text "axway-amplify=af03bdb0-d4a8-429f-bf96-b7ba41051d49" bosch.com descriptive text "apple-domain-verification=VYw9jmavDBjP0C9S" bosch.com descriptive text "mongodb-site-verification=gf5347Wa7YvVkq4C51l3srxrco2GLGPl" bosch.com descriptive text "docusign=ebea8618-97e6-4a64-82a9-2e3fa036d5bb" bosch.com descriptive text "docusign=df164c89-5239-42d4-97f5-8f5710b1b929" bosch.com descriptive text "miro-verification=95bb46ca27717c388e091411d7fe643e3a3d2b3d" bosch.com descriptive text "docker-verification=224577a6-972f-4e1d-a8b9-c5fc2e8da018" bosch.com descriptive text "klaviyo-site-verification=THKvhQ" bosch.com descriptive text "atlassian-sending-domain-verification=e4597f31-7a29-47fd-b150-5e1eaeac437b" bosch.com descriptive text "sFHU8FVI0Jt2PIXJAn2DWGmT7UJmZJzyq2THJmabTvEcw0IGtPu2UHU9Wf/zdvZoGAvtmO5tD3rOSvXjQWZ57Q=="
Indent with four spaces to render as preformatted text which preserves newlines... I also removed the redundant info from each line and sorted alphabetically while I was at it
TXT records for bosch.com:
TXT records for bosch.com:
77D8-D4C6-1CCC-8D85-3127-5140-11D2-956B
adobe-idp-site-verification=e13fd7b27a30ca44261bc7ad3f0b2e0994b724830fe8afc38a6565a40d81bde4
apple-domain-verification=VYw9jmavDBjP0C9S
atlassian-domain-verification=1XOeeaiW02aX/CXX/725tFzKFh4PA18JpZoyq9qBDqDxR2PP/9LDxCqlmYYgyb4D
atlassian-sending-domain-verification=e4597f31-7a29-47fd-b150-5e1eaeac437b
axway-amplify=af03bdb0-d4a8-429f-bf96-b7ba41051d49
b5r8gr465ydgqlvwlwy6x7dshccwg37c
cisco-ci-domain-verification=20e8d94f041a742a6de560a038d031baf659330970e6f05bb03fb2468bba379a
docker-verification=224577a6-972f-4e1d-a8b9-c5fc2e8da018
docusign=4f8c3289-c1e1-42a5-a541-f3459d2da55e
docusign=95ecd2d2-2737-4b07-9d40-95d065bfbc49
docusign=df164c89-5239-42d4-97f5-8f5710b1b929
docusign=ebea8618-97e6-4a64-82a9-2e3fa036d5bb
facebook-domain-verification=20380cm6fdq6h7tdcqzmhysa70ctdo
google-site-verification=avGZ684w6lq0UwXmOHxz9l6u9GL8r-sXoV7io9KzZOc
google-site-verification=jWBLaKM6WZQoAV6crbGGsAre3rSvqaDtwnCKuwvDPCg
google-site-verification=yeXu97OTorw4QioF3yNiDvTn-0GL8__7-iTNLEp6Vbk
klaviyo-site-verification=THKvhQ
mindmanager-verification=5eacd59de90dd7d9933da220294f2906a881fa6701d64a1a4667c05abac12546
miro-verification=95bb46ca27717c388e091411d7fe643e3a3d2b3d
mongodb-site-verification=gf5347Wa7YvVkq4C51l3srxrco2GLGPl
r7bNBcUhS/tsO45+nP1dycu5UDrZ7TniKe858vhLNJwAcCNDQlpdaks8iBm3TxV9r3fCYRvup/QpaC1rdp4Dpg==
sFHU8FVI0Jt2PIXJAn2DWGmT7UJmZJzyq2THJmabTvEcw0IGtPu2UHU9Wf/zdvZoGAvtmO5tD3rOSvXjQWZ57Q==
v=spf1 mx redirect=_spf.bosch.deI cannot believe how absurd and out of hand this has gotten.
Why did we not all standardise on some _well_known. subdomain or similar for these dozens of records?
The part that gives me an eye twitch is so many domains have multiple verifications for the same group, your own example has Google three times for example. The reason being one marketing company sets up Analytics. A few months later, a different company expects to be authorised to run another set, but all the admins get told is "you have to add this record".
Why did we not all standardise on some _well_known. subdomain or similar for these dozens of records?
The part that gives me an eye twitch is so many domains have multiple verifications for the same group, your own example has Google three times for example. The reason being one marketing company sets up Analytics. A few months later, a different company expects to be authorised to run another set, but all the admins get told is "you have to add this record".
Or just allocate proper DNS records as the standard intended.
There was a SPF record type in DNS, TXT was suggested as a transition, but we all know how that went.
Incidentally, SPF is a good example how a large company hijacked the process and tried to stuff the standard with things they had taken out patents on. That made the end standard less useful than what it could have been.
There was a SPF record type in DNS, TXT was suggested as a transition, but we all know how that went.
Incidentally, SPF is a good example how a large company hijacked the process and tried to stuff the standard with things they had taken out patents on. That made the end standard less useful than what it could have been.
They also generally don't need to stay there it's usually one-shot (at least the ones I've seen) but no one bothers to delete them.
I give talks to IT teams on the importance of cleaning up transient auth records in DNS and, as the night follows the day, I don't do it on my own personal ones.
What would you have used?
I would have implemented DKIM and SPF.
Great. But that doesn't solve this particular problem. Why do you think every legit ESP makes you do domain level validation using a custom TXT record?
The only reason most do that is to enable branded tracking links.
You are thinking of CNAMEs, and no, it isn't related.
To be clear, this issue is really with mail channels not authenticating the sender as the verified owner of the sending domain. CF workers are aren't the issue here.
> Display a banner on any email that doesn’t have a DKIM
signature but comes from a domain that’s implemented DKIM
(I’ve never seen this, I’m assuming possible?)
IIUC, it’s mostly not possible, because there is no sure way to know if a domain has “implemented DKIM”.
You can, in theory, do a DNS query for “_domainkey.example.com” and see if you get NXDOMAIN or NOERROR; the latter usually¹ means that there are subdomains present, which in turn would imply that there are some DKIM keys present in DNS (although you can’t know what the selector names are). But you don’t know if these keys are active, or if they are meant to be activated later. Or there might be multiple authenticated senders for a domain name, and some might use DKIM signing, but others might not.
1. Not all DNS servers obey the standards correctly, so the NXDOMAIN/NOERROR distinction only mostly works.
IIUC, it’s mostly not possible, because there is no sure way to know if a domain has “implemented DKIM”.
You can, in theory, do a DNS query for “_domainkey.example.com” and see if you get NXDOMAIN or NOERROR; the latter usually¹ means that there are subdomains present, which in turn would imply that there are some DKIM keys present in DNS (although you can’t know what the selector names are). But you don’t know if these keys are active, or if they are meant to be activated later. Or there might be multiple authenticated senders for a domain name, and some might use DKIM signing, but others might not.
1. Not all DNS servers obey the standards correctly, so the NXDOMAIN/NOERROR distinction only mostly works.
It is possible for providers who see a statically significant potion of mail traffic to see all or almost all DKIM selectors.
I think what the line you quote is trying to say is reject mail from MailChannels that isn't DKIM signed.
I think what the line you quote is trying to say is reject mail from MailChannels that isn't DKIM signed.
> It is possible for providers who see a statically significant potion of mail traffic to see all or almost all DKIM selectors.
Yes, but you can’t know that every authenticated sender is using DKIM at all. If the mail you just recieved from a new sender (even if it is from a domain which you have recieved DKIM signed mail from before) passes SPF but is not signed using DKIM, they are quite probably from a valid, albeit rarely used, mail sender who simply does not use DKIM.
> I think what the line you quote is trying to say is reject mail from MailChannels that isn't DKIM signed.
No, they can’t be saying that, since elsewhere they say that only 105 out of 2 million MailChannel domains are using DKIM. Therefore, they can’t be reasonably suggesting to block all mail from all but 105 of these domains.
Yes, but you can’t know that every authenticated sender is using DKIM at all. If the mail you just recieved from a new sender (even if it is from a domain which you have recieved DKIM signed mail from before) passes SPF but is not signed using DKIM, they are quite probably from a valid, albeit rarely used, mail sender who simply does not use DKIM.
> I think what the line you quote is trying to say is reject mail from MailChannels that isn't DKIM signed.
No, they can’t be saying that, since elsewhere they say that only 105 out of 2 million MailChannel domains are using DKIM. Therefore, they can’t be reasonably suggesting to block all mail from all but 105 of these domains.
You're pointing out the inherent challenge of domain authentication. Large domain owners like Google lock their domains down by publishing restrictive SPF records that do not authorize anyone other than their own IP space. For everyone else, SPF is a gigantic hole that you can drive a truck through.
For this and other reasons, people who actually work in the email industry do not trust SPF when authenticating domains. An SPF pass is necessary but not sufficient to know that someone is responsible for the email you just received. A far more trustworthy element is a valid DKIM signature; this certifies that the domain owner signed the message contents with a key they presumably control themselves.
For this and other reasons, people who actually work in the email industry do not trust SPF when authenticating domains. An SPF pass is necessary but not sufficient to know that someone is responsible for the email you just received. A far more trustworthy element is a valid DKIM signature; this certifies that the domain owner signed the message contents with a key they presumably control themselves.
The logical way to fix this seems to me to be for DMARC to implement some way for the domain to not just require SPF or DKIM, but explicitly require DKIM specifically.
This is how other similar things have solved similar problems, like DANE, CAA DNS records, and HSTS headers in HTTP. CAA records, in particular, long had a similar problem which was only solved with RFC 8657; discussion here: <https://news.ycombinator.com/item?id=34035148>
This is how other similar things have solved similar problems, like DANE, CAA DNS records, and HSTS headers in HTTP. CAA records, in particular, long had a similar problem which was only solved with RFC 8657; discussion here: <https://news.ycombinator.com/item?id=34035148>
[deleted]
I thought it already was required and if you specify adkim in your record.
No, my understanding is that the “adkim” tag in DMARC only affects how strictly the DKIM check is done, but not whether a DKIM signature is actually required.
> MailChannels main customers are web hosting providers who don’t
own the domains they send emails from!
That's the lamest excuse I've heard in a while.
Web hosting providers generally don't "own" the domains they host, but they absolutely know which domains they host. Routing domains to customer accounts/directories is the whole point of web hosting!
All they need is some sort of cPanel integration to report the list of domains and tie each domain to a randomly generated key. All of this can (and should) be automated without bothering the end users in any way.
That's the lamest excuse I've heard in a while.
Web hosting providers generally don't "own" the domains they host, but they absolutely know which domains they host. Routing domains to customer accounts/directories is the whole point of web hosting!
All they need is some sort of cPanel integration to report the list of domains and tie each domain to a randomly generated key. All of this can (and should) be automated without bothering the end users in any way.
I wish there was some way to evolve the DMARC spec such that only DKIM and not SPF is used for validation. Sadly things like Google Calendar invites still fail DKIM…
The next release of DMARC will likely include an option to exclude SPF from DMARC verification. Google's team is pushing for this on the IETF DMARC mailing list:
https://mailarchive.ietf.org/arch/msg/dmarc/PDktxOYkB28k6ukL...
We think this is a very good idea as it will allow domain owners to use DMARC but specify that they prefer DKIM to be the only mechanism that truly authenticates traffic as their own.
The industry has many workarounds for the weaknesses of SPF and DMARC, such as using SPF macros to dynamically shape authentication based on criteria one only knows at the time of SPF resolution, but none of these workarounds is better than just saying, "DKIM only for my domain, please."
https://mailarchive.ietf.org/arch/msg/dmarc/PDktxOYkB28k6ukL...
We think this is a very good idea as it will allow domain owners to use DMARC but specify that they prefer DKIM to be the only mechanism that truly authenticates traffic as their own.
The industry has many workarounds for the weaknesses of SPF and DMARC, such as using SPF macros to dynamically shape authentication based on criteria one only knows at the time of SPF resolution, but none of these workarounds is better than just saying, "DKIM only for my domain, please."
I would love this. Right now I am running a service that sends email. For cost reasons I try to do this directly. It works surprisingly well even from cloud-provider IP addresses. Most providers quickly learn to trust my domain. But there are some big players that have an outright block on public cloud ranges (notably Microsoft and Apple). I end up needing to use a relay for these, but I would prefer not to give the relay a DKIM key or allow it to munge my messages.
This mostly works today, for example SES allows the origin to sign messages (although as of a few months ago I started hitting an SES bug where they modify a header field that they said they wouldn't, breaking the signature) and there are a few other providers that allow this capability as well.
However I basically still need to mark them as trusted in the SPF record otherwise the spam score goes up. This effectively allows them to spoof messages from my domain. I would love to close off this loophole.
It also isn't great with sending from public cloud VM instances either since I have to update the SPF record with the changing IPs and DNS caching can cause false fails (if a new instance sends mail before the cache is refreshed) and false passes (if the old IP gets reused for a different customer before the cache expires).
Yeah, I know. I should just rent a public IP. But that raises costs and would require me to raise my prices. Especially at the current small scale.
This mostly works today, for example SES allows the origin to sign messages (although as of a few months ago I started hitting an SES bug where they modify a header field that they said they wouldn't, breaking the signature) and there are a few other providers that allow this capability as well.
However I basically still need to mark them as trusted in the SPF record otherwise the spam score goes up. This effectively allows them to spoof messages from my domain. I would love to close off this loophole.
It also isn't great with sending from public cloud VM instances either since I have to update the SPF record with the changing IPs and DNS caching can cause false fails (if a new instance sends mail before the cache is refreshed) and false passes (if the old IP gets reused for a different customer before the cache expires).
Yeah, I know. I should just rent a public IP. But that raises costs and would require me to raise my prices. Especially at the current small scale.
As someone whose recently facechecked getting my domains email ready, including the absolute frustration of dealing with an ISP who has decreed that "those who want to send email from their own metal shalt have a business account", shit like this makes my blood boil.
I bust my ass to be a responsible member of the Net and do my damnedest to ensure that my systems are up to snuff, up to and including ripping through/tearing apart the current SotA... and yet assholes like that not only exist, but flippantly run as an open relay. With almost half of the Net paying them to do so. Are you frigging kidding me?!
I bust my ass to be a responsible member of the Net and do my damnedest to ensure that my systems are up to snuff, up to and including ripping through/tearing apart the current SotA... and yet assholes like that not only exist, but flippantly run as an open relay. With almost half of the Net paying them to do so. Are you frigging kidding me?!
TL;DR They found an open relay included in many SPF records.
Where out of those 2M domains with SPF records opening themselves up, <1k had set up DKIM/DMARC records and somehow even those with DKIM configured and neglected still pass as authentic in Gmail.
That part did not make any sense to me. I do not see how this person determined that these domains are not using DKIM. There is no standard DKIM domain name, so what did they check?
[deleted]
Worse than that. The platform itself didn't even attempt to verify domain ownership so you could send as anyone.
Except it's not an open relay. MailChannels would not exist on the internet if it didn't aggressively control spam and phishing. The DEFCON talk did not prove the existence of a gaping hole; it showed something that has existed since the dawn of internet email: the sender domain cannot be adequately authenticated unless you use message signatures such as S/MIME and DKIM. And even with DKIM, DKIM replay attacks allow widespread abuse.
That's a terribly incomplete understanding of what's going on here.
It is incumbent upon them to validate that their API users own the domain they intend to send from. All the reputable senders do this. Allowing unchecked user-defined from field is the definition of an open relay.
It is incumbent upon them to validate that their API users own the domain they intend to send from. All the reputable senders do this. Allowing unchecked user-defined from field is the definition of an open relay.
Why is it not valid for someone to send email from another domain? I’m asking seriously. That’s not an invalid thing to do. The RFC explicitly advises that flexibility in the envelope address is a necessary feature of internet email.
It may be a convention that transactional email services require proof of domain ownership, but that is a convention, not a standard.
It may be a convention that transactional email services require proof of domain ownership, but that is a convention, not a standard.
I think the fact this was possible via the use of CF workers invalidates your point. Also, you choose to add the Domain Lockdown feature as a direct results of these findings.
I'd feel pretty silly too if my "email filters" kept me from acting sooner on this problem.
I'd feel pretty silly too if my "email filters" kept me from acting sooner on this problem.
It's valid, but it needs to be authorized. Back when initial specs of SMTP were it didn't matter that much as users were mainly academics and it was primarily used for discussion/passing information.
These days you don't really want your finance team getting email which say it's from CEO and is asking them to make wire transfer, email from IT asking to password reset and so on. While it probably shouldn't be, the sender address is treated as very strong signal that email is legitimate so you should be taking steps to make it so that if someone tries to spoof it without authorization, it should be treated as suspicious.
These days you don't really want your finance team getting email which say it's from CEO and is asking them to make wire transfer, email from IT asking to password reset and so on. While it probably shouldn't be, the sender address is treated as very strong signal that email is legitimate so you should be taking steps to make it so that if someone tries to spoof it without authorization, it should be treated as suspicious.
IDK. First thing I say in every single security talk I give is "The Sender Address displayed in an email is just what the sender put there, nothing more". If the email needs some kind of validation it needs to carry that token itself (and here begins my long graybeard rant on why we should all be digitally signing emails as a matter of course).
Sorry, no. Any service that will issue emails masquerading as 3rd-party domains, which does not use an effective authentication mechanism, is an open relay by definition.
I disagree. The “open” part implies there is no control at all on what email can be relayed. That is not at all the case here.
Spammers try their best to send spam through MailChannels all day long, through compromised WordPress sites, hacked user accounts, etc. We block 28% of all the messages that are submitted, and of what remains, only 2% is rejected by receivers.
That’s not at all what an open relay looks like. An open relay would accept 100% and likely be blocked within an hour across the internet.
Spammers try their best to send spam through MailChannels all day long, through compromised WordPress sites, hacked user accounts, etc. We block 28% of all the messages that are submitted, and of what remains, only 2% is rejected by receivers.
That’s not at all what an open relay looks like. An open relay would accept 100% and likely be blocked within an hour across the internet.
Ironically it's not possible to use S/MIME with MailChannels here because you don't allow sending raw MIME content.
Yeah that restriction isn't immediately apparent when you look at the documentation that mailchannels link to on their site.[1]
However, it is mentioned in the openapi schema.[2]
I suppose they intend that people send up to 20M with just text/plain and text/html. (Send all the base64 encoded images I guess)
[1]https://api.mailchannels.net/tx/v1/documentation [2]https://api.mailchannels.net/tx/v1/openapi.yaml
However, it is mentioned in the openapi schema.[2]
I suppose they intend that people send up to 20M with just text/plain and text/html. (Send all the base64 encoded images I guess)
[1]https://api.mailchannels.net/tx/v1/documentation [2]https://api.mailchannels.net/tx/v1/openapi.yaml
You don't understand the situation at all
If you check their profile they’re the founder of MC. That would explain the comments more than understanding of the topic.
Oh yikes. Yeah, I've seen them arguing on the Cloudflare Discord channel and it is the definition of sticking your head in the sand. Their response to this serious problem has been absolutely awful
One of those "It is difficult to get a man to understand something, when his salary depends on his not understanding it." situations I guess.
The effect that ARC headers have on spam scores is interesting. As someone who runs his own private mail server, I can seemingly improve the deliverability of my emails by just adding a set of useless ARC headers to my own emails?
I want to say no, purely based on the idea that if that were the case all my inboxes - spreads across several providers - would be overwhelmed with spam? Surely the (organised, educated) spammers are all over this sort of thing and would be using that to break through?
That's what I would assume too, but one of the slides very specifically claims:
> The presence of an arc=pass generally guarantees a better spam score. Confirmed with ProtonMail. Seems to be the case with Gmail and Outlook as well.
And it doesn't seem like there's anything that would prevent me from adding this to my own emails. I don't think it would turn blatant spam into non-spam, but it seems like it could help something that's already teetering on the line.
> The presence of an arc=pass generally guarantees a better spam score. Confirmed with ProtonMail. Seems to be the case with Gmail and Outlook as well.
And it doesn't seem like there's anything that would prevent me from adding this to my own emails. I don't think it would turn blatant spam into non-spam, but it seems like it could help something that's already teetering on the line.
I want to test this now heh!
Nobody in the email world considers ARC to be a bulletproof way to bypass spam filtering at a major receiver. The DEFCON presenter was ill-equipped to make that determination.
> bulletproof way to bypass spam filtering
Yeah that's definitely not what I meant or what the presenter seems to be implying. If it helps spam scores even a little bit, that's very interesting and potentially worth implementing in private hosts that often get dinged by larger hosts just for not being well-known. It doesn't need to get anywhere near turning actual spam into non-spam.
Yeah that's definitely not what I meant or what the presenter seems to be implying. If it helps spam scores even a little bit, that's very interesting and potentially worth implementing in private hosts that often get dinged by larger hosts just for not being well-known. It doesn't need to get anywhere near turning actual spam into non-spam.
I've worked in anti-spam since 2003. I am skeptical that his claims about ARC work at scale. Google does not have a rules-based spam filtering system. It is entirely driven by a massive machine learning model that adjusts its weights in response to user interaction. You can't beat that model.
> It is entirely driven by a massive machine learning model that adjusts its weights in response to user interaction. You can't beat that model.
Why not? Spammers do. I get spam to my gmail all the time.
Why not? Spammers do. I get spam to my gmail all the time.
Yep, I see phishing style spam (email from random domain that claims to be my bank) in my gmail in the past few weeks.
> You can't beat that model.
Sign up for a google apps account and send outbound cold sales spam via gmail. Seems to work exceedingly well judging by my inbox.
Sign up for a google apps account and send outbound cold sales spam via gmail. Seems to work exceedingly well judging by my inbox.
… I should have carved out a space for cold prospecting. I’m using GPT-3.5 to detect it in my own inbox because bulk spam filters don’t seem to have the requisite power to deal with such highly customized messages at scale.
They’ll be detecting based on spam received from the sending domain soon enough.
> I've worked in anti-spam since 2003.
By that you mean 2003 was the first year this vulnerability was reported to you, and every year since you've been auto-replying "nothing to see here" when people kept reporting this vulnerability to you?
By that you mean 2003 was the first year this vulnerability was reported to you, and every year since you've been auto-replying "nothing to see here" when people kept reporting this vulnerability to you?
looks like this was even ack'd in May 2022: https://news.ycombinator.com/item?id=30533032
From CEO, apparently:
> We have extensive spam and phishing detection capabilities and can handle the abuse.
Ah yes :D
> We have extensive spam and phishing detection capabilities and can handle the abuse.
Ah yes :D
Since about September 1st, email forwarding from my Gmail account to another Gmail account fails with a DMARC-related bounce depending on the original sender's configuration. Email security has become too hard for email providers to get right.
Does the issue still exist if you register an $80 SMTP outbox and spoof the address?
The slides say yes:
>● Anyone is still able to sign up on MCs website and for 80$ spoof all their customers via their SMTP relay. You’re fully trusting in their mitigating controls.
>● Anyone is still able to sign up on MCs website and for 80$ spoof all their customers via their SMTP relay. You’re fully trusting in their mitigating controls.
Thought experiment: Would an email sending service that sends email for 42% of the domains globally that send through a transactional service [1] possibly exist on today's internet if it _wasn't_ incredibly good at blocking spam?
The DEFCON presenter makes the assertion that you can set up a spamming operation for $80/mo, but has he tried?
[1] https://trends.builtwith.com/mx/transactional-email/traffic/...
The DEFCON presenter makes the assertion that you can set up a spamming operation for $80/mo, but has he tried?
[1] https://trends.builtwith.com/mx/transactional-email/traffic/...
It's incredibly infuriating to read your replies here. You clearly don't care at all about email security, you don't take responsibility for your mistakes, and you have the GALL to comment like this. Awful.
Most drugs are not legal, still they exist.
The examples in the slides show that your company allowed spam. Why is it so tough to admit a mistake? We all do sometimes misconceptions, some even at scale.
SendGrid isn't great at blocking spam, and they're a billion dollar company.
So yes.
So yes.
SendGrid send roughly 8% of all internet email. At their scale, controlling abuse is extremely difficult. Gmail originates a huge amount of spam and phishing. The bigger you are, the more spammers will work to target you and the harder you have to work to fight back against them.
I can't sign up for Sendgrid and then send signed email on behalf of another Sendgrid customer.
Wait.. isn’t this now the argument against your original thought experiment?
Why is spam the only consideration? What about ad hoc fraud?
Yes, 100% yes. As can be seen with your company.
I think this is primarily a problem of the domain owners/sysadmins, all my domains have SPF, DKIM, and DMARC and I also keep them monitored for any blacklist/alerts, if you are not aware of these, you should hire someone to audit it, email protocol is broken by design and all of these are duct tape solutions to prevent OP attempts. This whole document gives me nostalgia back in the late 90s early 2000s how there were tools you install to spoof emails!
Is this why I'm getting bounces from emails I didn't send with broken DKIM signatures for the past couple of weeks?
(Edit: I'm not using MailChannels, though.)
(Edit: I'm not using MailChannels, though.)
Can anyone confirm whether that this short article I wrote a while ago is "good enough" for avoiding spoofing?
https://www.uxwizz.com/blog/stop-others-use-your-domain-emai...
https://www.uxwizz.com/blog/stop-others-use-your-domain-emai...
[deleted]
But... What about BIMI
Well this explains all the email2text spam with spoofed emails lately.
The level of complacency of mailchannels when they were contacted about the issue is just unbelievable. Those guys are responsible for a sizeable portion of the corporate email traffic, and don't seem to give a shit about security or spam (the cancer of the very thing they sell) until publicly shamed about it.
Entertaining presentation
Entertaining presentation
Eh, it's not just their security that has complacency. I contacted their support and explained how to fix their incorrect handling of text/x-amp-html content (they insist on putting text/html first, which breaks Apple Mail when sending AMP emails), and this is the eventual response I got after wasting time trying to help:
```
I do understand what you want, but we're not going to make that change at this time.
This probably isn't the answer you wanted, but I'm afraid we can't be any more help with this issue. I'm marking this ticket solved. Please open a new ticket if you need anything else.
```
```
I do understand what you want, but we're not going to make that change at this time.
This probably isn't the answer you wanted, but I'm afraid we can't be any more help with this issue. I'm marking this ticket solved. Please open a new ticket if you need anything else.
```
sure, otoh they build big money faucet upon "don't worry, we'll take care of it" which you've gotta respect
[deleted]
Due to laziness / ignorance / some combination of both, many domains that use Amazon can be trivially spoofed by anyone else who can send via Amazon. Last I checked, that was true of outlook.com, too. I think there was a recent article about it...
These companies are so hungry for money, they forget that their products need anything else at all after they get to the "charge customer money for service" part. Companies have been told about issues like these for ages now, yet they all act so surprised when the same ancient problem pops up, time and time again.
These companies are so hungry for money, they forget that their products need anything else at all after they get to the "charge customer money for service" part. Companies have been told about issues like these for ages now, yet they all act so surprised when the same ancient problem pops up, time and time again.
And if you want to setup an honest mail server as Random Joe, you will struggle to not get blocked by spam filters while these too-big-to-fail mail senders can get away with being on open relay.
[deleted]
cyclotron3k(3)
Ken is a good friend in the industry and always has the best interest of email security at heart. This may have been an architectural oversight but they are not wrong that SPF is surely a cause for concern, as is misconfigured DNS based trust and recertifications via arc (which was supposed to solve a problem for forwarding scenarios).
The centralization of email services to a handful of providers basically has led to multihoming of millions of domains that open SPF auth to the same handful. Any integrations by them or changes to existing stack can cause issues to pop up, because delegation of sending rights isn't strictly auth controlled. The same also happens with dkim delegation to saas providers who share backend keys across other customers of theirs and if their API is open to experiment (or an account gets popped) then the customer domains are possibly at risk.
Email is hard to do right. No auth no entry should be the default. But majority of domain owners aren't very good at figuring out how to secure things, or have business/product interests that are a priority, specially when delegated and authorized to third party senders on their behalf.
The centralization of email services to a handful of providers basically has led to multihoming of millions of domains that open SPF auth to the same handful. Any integrations by them or changes to existing stack can cause issues to pop up, because delegation of sending rights isn't strictly auth controlled. The same also happens with dkim delegation to saas providers who share backend keys across other customers of theirs and if their API is open to experiment (or an account gets popped) then the customer domains are possibly at risk.
Email is hard to do right. No auth no entry should be the default. But majority of domain owners aren't very good at figuring out how to secure things, or have business/product interests that are a priority, specially when delegated and authorized to third party senders on their behalf.
> Ken is a good friend in the industry and always has the best interest of email security at heart.
Based on the information in OP we know as a fact this is false. Please stop spreading misinformation.
Based on the information in OP we know as a fact this is false. Please stop spreading misinformation.
or here (video format didn't work in firefox for me but VLC plays it): https://media.defcon.org/DEF%20CON%2031/DEF%20CON%2031%20vid...