After self-hosting my email for twenty-three years I have thrown in the towel(cfenollosa.com)
cfenollosa.com
After self-hosting my email for twenty-three years I have thrown in the towel
https://cfenollosa.com/blog/after-self-hosting-my-email-for-twenty-three-years-i-have-thrown-in-the-towel-the-oligopoly-has-won.html
745 comments
I completly agree with you. While Hetzner is my actual neighbor as their headquarters is right in the neighboring town, I still use a server at very small scale provider. I have no problem with my email server. I receive some spam here and there, mostly from Russia. But I immediatly block the according IP addresses for some time.
For years I avoided to use any external service to decide whether its Spam or not. But about 2 years ago I started to rely on some of the external Blocklists.
Till today I have no problem sending Email. Even as I don't use DKIM or DMARC.
For years I avoided to use any external service to decide whether its Spam or not. But about 2 years ago I started to rely on some of the external Blocklists.
Till today I have no problem sending Email. Even as I don't use DKIM or DMARC.
If you are using external blocklists, you are perpetuating the problem here. Small mail operators end up on blocklists through no fault of their own, and it sounds like your server would reject them just like the big servers do.
You don’t have to use blocklists to block messages.
I use blocklists in my self-hosted setup but only for the purpose of adding header fields that my Bayesian anti-spam filter can use to classify messages. I don’t reject anything out-right, aside from attempted spoofs of the domains my server is authoritative for. Everything is received— it just may end up in an “Unsure” folder if it seems too shady for the filter to put in my inbox.
I use blocklists in my self-hosted setup but only for the purpose of adding header fields that my Bayesian anti-spam filter can use to classify messages. I don’t reject anything out-right, aside from attempted spoofs of the domains my server is authoritative for. Everything is received— it just may end up in an “Unsure” folder if it seems too shady for the filter to put in my inbox.
That sounds like a better solution and not at all what I would assume someone means when they say that they use blocklists to filter out spam.
Not using block lists will inundate mail servers with spam. It's a necessary evil.
I don't use any kind of block list on my mail server, as I find the concept to be fundamentally flawed. I accept all incoming email as long as basic sanity on the connection is met and then apply bayesian filtering after the email has been accepted.
I get just about no spam at all (<10 per month, maybe).
I get just about no spam at all (<10 per month, maybe).
I use block lists, fail2ban, gray listing, most of rspamd's widgets, and Bayesian filtering. Still getting 10s of spams a day just to my accounts.
Any guess on why people's experiences with spam are so diverse? I self-host and get fewer than 10 true spam emails per year (not counting marketing newsletters).
> Any guess on why people's experiences with spam are so diverse?
That would be an interesting research project. The domains getting most spams on my servers are the ones that are old (20+ years) which I guess makes sense.
Checking the rspamd logs for the last month gives just shy of 8500 emails with what I'd consider "definitely a spam" score. There's probably another 1000-1500 sneaking under that.
That would be an interesting research project. The domains getting most spams on my servers are the ones that are old (20+ years) which I guess makes sense.
Checking the rspamd logs for the last month gives just shy of 8500 emails with what I'd consider "definitely a spam" score. There's probably another 1000-1500 sneaking under that.
> > Any guess on why people's experiences with spam are so diverse?
> That would be an interesting research project.
Indeed. When you say get spam, do you mean pre or post-filtering?
Over the last 30 days (I don't keep them longer than that) I've received 43 spam emails which were sent to the spam folder (so I wouldn't ever see these, other than because I went and looked now for the sake of this discussion). In the same time period there was only 1 spam email which was missed by spamprobe and made it into my inbox spool.
There's a fair amount of would-be spam that gets blocked during the SMTP transaction due to things like bogus host in HELO, etc. I don't keep any stats on those, I did at one time but it was too much noise.
My email isn't secret, it should be on every spammer list I'd guess. It has been the same since the mid 90s and is all over usenet, email list archives, websites, etc and I've never made any effort to mask it.
> That would be an interesting research project.
Indeed. When you say get spam, do you mean pre or post-filtering?
Over the last 30 days (I don't keep them longer than that) I've received 43 spam emails which were sent to the spam folder (so I wouldn't ever see these, other than because I went and looked now for the sake of this discussion). In the same time period there was only 1 spam email which was missed by spamprobe and made it into my inbox spool.
There's a fair amount of would-be spam that gets blocked during the SMTP transaction due to things like bogus host in HELO, etc. I don't keep any stats on those, I did at one time but it was too much noise.
My email isn't secret, it should be on every spammer list I'd guess. It has been the same since the mid 90s and is all over usenet, email list archives, websites, etc and I've never made any effort to mask it.
> Indeed. When you say get spam, do you mean pre or post-filtering?
All my numbers are post-rspamd. There's still sieve after that which does the "is it scored more than X?" to redirect into spam/not-spam.
> There's a fair amount of would-be spam that gets blocked during the SMTP transaction
Yeah, the blocklists and protocol strictness rules definitely cut down a lot before it even gets to rspamd.
All my numbers are post-rspamd. There's still sieve after that which does the "is it scored more than X?" to redirect into spam/not-spam.
> There's a fair amount of would-be spam that gets blocked during the SMTP transaction
Yeah, the blocklists and protocol strictness rules definitely cut down a lot before it even gets to rspamd.
You mentioned receiving 10s of spam per day - are they being classified correctly and put in your spam folder or are you seeing them in your inboxes?
My mx hosted with hetzner also runs rspamd. Of the 32k mails received in the last month, 40% were rejected (postfix DISCARD, so the sender sees the mail as accepted but its sent to /dev/null - this only happens to mails scored very highly as spam, or sent to a spam trap address), 10% were greylisted and 1% were delivered to the spam folder.
So I'm also receiving 10s of spam per day, but they're all delivered to my spam folder with rare errors.
With the exception of an issue delivering to AT&T recently, I haven't had any outbound deliverability problems in a few years, but then again I don't send very much mail at all - perhaps I'd have more trouble if I did but most of my mail is incoming.
My mx hosted with hetzner also runs rspamd. Of the 32k mails received in the last month, 40% were rejected (postfix DISCARD, so the sender sees the mail as accepted but its sent to /dev/null - this only happens to mails scored very highly as spam, or sent to a spam trap address), 10% were greylisted and 1% were delivered to the spam folder.
So I'm also receiving 10s of spam per day, but they're all delivered to my spam folder with rare errors.
With the exception of an issue delivering to AT&T recently, I haven't had any outbound deliverability problems in a few years, but then again I don't send very much mail at all - perhaps I'd have more trouble if I did but most of my mail is incoming.
> are they being classified correctly and put in your spam folder
Mostly. There's probably 3-5 spams a day which get into my inbox; also 10-15 a day that wrongly end up in the "maybespam" folder (generally bulk that I'm not worried about seeing 100%.)
Mostly. There's probably 3-5 spams a day which get into my inbox; also 10-15 a day that wrongly end up in the "maybespam" folder (generally bulk that I'm not worried about seeing 100%.)
> That would be an interesting research project. The domains getting most spams on my servers are the ones that are old (20+ years) which I guess makes sense.
The old lists may be more plausible.
Some spammer once associated 'Robin Bennett' with my email addresses at some point 20 years ago and kept reselling it. Never used that name, didn't even know Bennett was a surname. I assume they linked me up with that name to fill up empty cells in their list and make it look more plausible.
It's a good way to filter spam. The last decade almost all spam calling me 'Bennett' is from some US political group, which mostly reminds me that it will be a long time before anything GDPR-like will pass in the US..
But almost all spam goes to that old 20 year old email address I don't use. My current email addresses are much cleaner even after 10+ year of use
The old lists may be more plausible.
Some spammer once associated 'Robin Bennett' with my email addresses at some point 20 years ago and kept reselling it. Never used that name, didn't even know Bennett was a surname. I assume they linked me up with that name to fill up empty cells in their list and make it look more plausible.
It's a good way to filter spam. The last decade almost all spam calling me 'Bennett' is from some US political group, which mostly reminds me that it will be a long time before anything GDPR-like will pass in the US..
But almost all spam goes to that old 20 year old email address I don't use. My current email addresses are much cleaner even after 10+ year of use
[deleted]
I thought self hosting is for aliases? Shouldn't those fix spam for good unless you publish your address globally?
> I thought self hosting is for aliases? Shouldn't those fix spam for good
Not quite sure I follow? Unless you mean "only allow emails to specific email addresses you've noted down", in which case, yeah, that works but also means a lot of admin when you want to use a new one (plus there's 10+ other people who use my servers for email, not just me.)
> unless you publish your address globally?
At least one of my email addresses has been published globally since ~1995. Others since ~2000.
Not quite sure I follow? Unless you mean "only allow emails to specific email addresses you've noted down", in which case, yeah, that works but also means a lot of admin when you want to use a new one (plus there's 10+ other people who use my servers for email, not just me.)
> unless you publish your address globally?
At least one of my email addresses has been published globally since ~1995. Others since ~2000.
By lot of admin you mean that it's not automated? That's sad indeed.
> By lot of admin you mean that it's not automated? That's sad indeed.
How would I automate "[someone with an account] just entered abc-xyz@domain into a web form to subscribe to something, add that to the valid alias file"?
How would I automate "[someone with an account] just entered abc-xyz@domain into a web form to subscribe to something, add that to the valid alias file"?
As I understand, spam filters like postscreen are designed as reverse proxies or relay servers. IIRC spamgourmet supports your scenario.
> IIRC spamgourmet supports your scenario.
Not really - that's a limited use alias. Doesn't really work for when you want to keep getting email from places. Also if I'm reading that correctly, spammers can just send mail to <randomword>.20.<knownaccount>@spamgourmet.com and you can't protect against those 20 spams.
Not really - that's a limited use alias. Doesn't really work for when you want to keep getting email from places. Also if I'm reading that correctly, spammers can just send mail to <randomword>.20.<knownaccount>@spamgourmet.com and you can't protect against those 20 spams.
You are not a valuable target to spammers. At some point even storing the spam letters makes no financial sense.
One spam every three days is insanely high IMO (compared to what you will get on gmail)
I get > 1 spam per day on gmail.com.
But I don't find that burdensome at all, in fact I'd prefer that gmail let more stuff through to my spam folder instead of swallowing things that it misclassifies, because I have seen several instances of lost legitimate email, something I still don't understand how is deemed acceptable to the people who wrote it.
But I need to vote with my feet instead of complaining.
But I don't find that burdensome at all, in fact I'd prefer that gmail let more stuff through to my spam folder instead of swallowing things that it misclassifies, because I have seen several instances of lost legitimate email, something I still don't understand how is deemed acceptable to the people who wrote it.
But I need to vote with my feet instead of complaining.
> One spam every three days is insanely high IMO (compared to what you will get on gmail)
That's funny since on gmail I get tons more spam than on any of my self-hosted domain addresses. What's worse, I see emails in gmail misfiled as spam when they are not, which is far worse. I don't ever see that happen in my self-hosted system.
That's funny since on gmail I get tons more spam than on any of my self-hosted domain addresses. What's worse, I see emails in gmail misfiled as spam when they are not, which is far worse. I don't ever see that happen in my self-hosted system.
And for most people, using a large mail provider is the necessary evil. I don't see much difference between a small mail provider using block lists and a large mail provider using the same block lists + their own proprietary lists and metrics.
Both result in other small mail providers getting blocked for arbitrary reasons with limited recourse.
Both result in other small mail providers getting blocked for arbitrary reasons with limited recourse.
Using large mail provider is not necessary evil in my book. It is just so much more convenient than self hosting.
Today, with ready-made containers and stuff we start having a fighting change, but the moat is possibly too big to fight with email falling out of favor
I think that gray listing is a great idea to filter out spam. Here's how it works: the very first time an email server get an email from a new IP it answers with a "temporary unavaiable" error. According to specs, a server should retry to deliver the email after a while, so a legitimate server will retry and it's IP will be put in a "withelist" (here in quotes because you can still do further processing of the emails to determine if it's good or bad). But a spammer will most likely not retry to send it, as their goal is to quickly send a big amount of mails.
I didn't try this personally as I'm not hosting my own emails (I tried but gave up soon), but I heard it works very well, no third party blocklist needed.
The trade off is that even legitimate mail that you are actively expecting (like an account confirmation or password reset mail) will be delayed by however long the sending server's retry interval is.
You also either need to apply the greylisting to some larger IP range (rspamd e.g. apparently uses /19 by default for IPv4) or otherwise specially handle some of the bigger mail providers, because some of them rotate through their servers between retries, so you could be in for a quite a long wait if you do per-individual-IP greylisting.
The biggest culprit I noticed this with was Amazon SES – a former mail provider of mine used per-individual-IP, non-configurable greylisting, and any mail sent through Amazon (which isn't just Amazon itself – quite a few companies are using Amazon SES for transactional mail and suchlike) would consequently almost always arrive several hours late (however randomly long it would take Amazon to finally re-use an IP during a subsequent retry attempt).
Even more infuriating, my mail provider's support would then claim that it wasn't their fault and they didn't know anything about any supposed greylisting.
You also either need to apply the greylisting to some larger IP range (rspamd e.g. apparently uses /19 by default for IPv4) or otherwise specially handle some of the bigger mail providers, because some of them rotate through their servers between retries, so you could be in for a quite a long wait if you do per-individual-IP greylisting.
The biggest culprit I noticed this with was Amazon SES – a former mail provider of mine used per-individual-IP, non-configurable greylisting, and any mail sent through Amazon (which isn't just Amazon itself – quite a few companies are using Amazon SES for transactional mail and suchlike) would consequently almost always arrive several hours late (however randomly long it would take Amazon to finally re-use an IP during a subsequent retry attempt).
Even more infuriating, my mail provider's support would then claim that it wasn't their fault and they didn't know anything about any supposed greylisting.
You just solved a random mystery I was having with mail forwarding from a single particular provider.
> I heard [gray listing] works very well
Mostly. But then you get the "click the link in the email within 10 minutes" problem. There's also a non-zero number of "our mail didn't get through first attempt, oh well, give up" people. From running GL on my servers over a couple of years, it mildly cut down spam (on top of blocklists and fail2ban) but I'm now wavering over whether it's worth the hassle.
Mostly. But then you get the "click the link in the email within 10 minutes" problem. There's also a non-zero number of "our mail didn't get through first attempt, oh well, give up" people. From running GL on my servers over a couple of years, it mildly cut down spam (on top of blocklists and fail2ban) but I'm now wavering over whether it's worth the hassle.
It cut spam tremendeously for me, but that's on an email address I've used and published openly since early 2000s.
Still, I've given up on it since plenty of email senders are not standard-abiding (they fail to retry), and I've kept losing email. I only caved in in the last 12 months after 15+ years of doing graylisting.
Still, I've given up on it since plenty of email senders are not standard-abiding (they fail to retry), and I've kept losing email. I only caved in in the last 12 months after 15+ years of doing graylisting.
I have greylisting enabled for years. But recently some Russian spamers were able to circumvent those by using compliant SMTP servers. They even support proper SPF, DKIM, DMARC and all that stuff what you can think of.
That is the reason why I switched on some external block lists into the mix.
That is the reason why I switched on some external block lists into the mix.
I wonder if anyone compared the effectiveness of postscreen vs greylisting. Can't find any
Both rely on filtering out non compliant senders, but postscreen's filtering might be less disruptive. Are there spammers out there who cannot pass a graylist but can pass postscreen ?
Both rely on filtering out non compliant senders, but postscreen's filtering might be less disruptive. Are there spammers out there who cannot pass a graylist but can pass postscreen ?
What we found 20+ years ago when we started to provide a very poor service to attempted spam delivery was that the spammers realised our servers were very 'slow' and stopped the attempts. We used to get 1000s of new IPV4 tuples / day, now we get < 10. If you put the inbound SMTP sessions to sleep this is wrecking the spammers bus. model, they need to deliver 1,000,000,000+ / day. Don't u fell sorry 4 them. The only time we don't tuck them in 4 a snooze is if they are in a 3rd party BL !!!
This is pure victim-blaming. The problem is that bulletproof hosting exists, not that people are sick of spam.
I don't follow. I am pointing out that using the same kinds of lists that the big providers use will result in smaller providers being arbitrarily blocked with little recourse. Do you disagree with that assessment?
First, yes. They (the small providers) have recourse, it’s just annoying. They voluntarily signed up for that annoyance because of their ideology.
But much more importantly, that question is orthogonal to my argument. Using blocklists is a good way to cut down on spam, the fact that it might block some trivial percentage of people who for ideological reasons might or might not be on those lists isn’t the receivers problem.
You want this to not be the case? Contribute meaningfully to solving the problem in a more effective way. Don’t blame the people who just don’t want the spam.
But much more importantly, that question is orthogonal to my argument. Using blocklists is a good way to cut down on spam, the fact that it might block some trivial percentage of people who for ideological reasons might or might not be on those lists isn’t the receivers problem.
You want this to not be the case? Contribute meaningfully to solving the problem in a more effective way. Don’t blame the people who just don’t want the spam.
The small providers typically don't have recourse if they approach a blocklist maintainer and ask to be taken off a list. That "trivial percentage of people" make up a substantial portion of service providers (for every Google there are many <100 user mail servers), which is what this thread was about.
I pointed out that adopting blocklists just makes it harder to operate a mail server as a small provider, and nobody in this thread appears to disagree with that assessment. They instead seem to take issue with the tone of the message.
I pointed out that adopting blocklists just makes it harder to operate a mail server as a small provider, and nobody in this thread appears to disagree with that assessment. They instead seem to take issue with the tone of the message.
>The small providers typically don't have recourse if they approach a blocklist maintainer and ask to be taken off a list. [...] I pointed out that adopting blocklists just makes it harder to operate a mail server as a small provider, and nobody in this thread appears to disagree with that assessment. They instead seem to take issue with the tone of the message.
I don't think it's the tone. While the inability for senders to get off blocklists can be true, you're still not addressing why the cost to the sender to not be blocked should have higher priority than the cost to the receiver to avoid blocklists to make email admin more of a burden.
The gp you first replied to (PinguTS) is running a personal mailserver and resorted to using blocklists because reducing spam -- at the cost of some legit people not being able to send email to him -- is a tradeoff he's willing to make. You haven't convinced every user running mailservers that they should increase their spam burden because some small providers can't get off blocklists.
As another example in another communication channel... Here's a similar "blocklist" for cell phones that some users take advantage of: https://about.att.com/pages/cyberaware/ae/cp
Are "legitimate" phone numbers getting caught up in that block filter?!? Of course. But phone owners are trying to stop spam telemarketing calls. Telling them that some phone users are incorrectly on AT&T's list isn't going to convince cell users to quit using the block filter. They're willing to live with a few legit callers getting blocked.
I don't think it's the tone. While the inability for senders to get off blocklists can be true, you're still not addressing why the cost to the sender to not be blocked should have higher priority than the cost to the receiver to avoid blocklists to make email admin more of a burden.
The gp you first replied to (PinguTS) is running a personal mailserver and resorted to using blocklists because reducing spam -- at the cost of some legit people not being able to send email to him -- is a tradeoff he's willing to make. You haven't convinced every user running mailservers that they should increase their spam burden because some small providers can't get off blocklists.
As another example in another communication channel... Here's a similar "blocklist" for cell phones that some users take advantage of: https://about.att.com/pages/cyberaware/ae/cp
Are "legitimate" phone numbers getting caught up in that block filter?!? Of course. But phone owners are trying to stop spam telemarketing calls. Telling them that some phone users are incorrectly on AT&T's list isn't going to convince cell users to quit using the block filter. They're willing to live with a few legit callers getting blocked.
[deleted]
> ideological reasons might or might not be on those lists
What's the ideological reasons here?
What's the ideological reasons here?
Apparently people who want to self host email instead of paying some big tech
[deleted]
Agree completely.
To those that still persist, there is a page I found recently that helps you make sure that your outgoing mail is configured correctly: https://appmaildev.com/en/dkim. They generate an e-mail address, you send a mail to them, they do the check and display results (not affiliated, just a happy user).
To those that still persist, there is a page I found recently that helps you make sure that your outgoing mail is configured correctly: https://appmaildev.com/en/dkim. They generate an e-mail address, you send a mail to them, they do the check and display results (not affiliated, just a happy user).
EU's MECSA is nice as well https://mecsa.jrc.ec.europa.eu/en/
I used this one: https://www.mail-tester.com/
I will check out your recommendations. Thanks
I will check out your recommendations. Thanks
I've also been self-hosting email for years, and the only deliverability problem I've ever had has been with AT&T. If I try to send something to an AT&T customer, I get an automated "your message has been eaten" notice, and following its directions accomplishes precisely nothing. At this point, I can only guess they're hellbanning the IP block in which my VPS resides, because it does not show up on any public DNSBLs.
Google? No problem. Comcast? No problem. Charter? No problem. AT&T? Problem.
Google? No problem. Comcast? No problem. Charter? No problem. AT&T? Problem.
At least you get notified. Microsoft/Outlook on the other hand silently drops emails leaving you and the recipient in the dark.
The correct response would be to set them on the blacklist for a time. People would complain and they would quickly change their behavior. Microsoft currently really wants everyone to have a Microsoft account.
that seems new, or maybe a different beast from the MS zoo of madness.
gmail on the other hand does what others said, report smtp 250 and silently discard some emails. (mostly those that lack DKIM)
gmail on the other hand does what others said, report smtp 250 and silently discard some emails. (mostly those that lack DKIM)
I think we are currently in a phase because of phishing problems. Corporate filters are currently turned up to eleven, but hosting your own server is still very well possible. SPF/DKIM/DMARC helps, but isn't required. You really think the corps that do host their own servers have set that up? It is far more widely spread by enthusiasts that host their own servers.
We also should keep mail for pseudonymous and anonymous user authentication. There are a lot of threats to the free internet right now and user account consolidation is one of it. I agree that people should keep hosting themselves. Someone blocked you? Their loss, let them rant to their internal IT about it. Corporations or institutions that use Google as a provider should be seen with scepticism.
We also should keep mail for pseudonymous and anonymous user authentication. There are a lot of threats to the free internet right now and user account consolidation is one of it. I agree that people should keep hosting themselves. Someone blocked you? Their loss, let them rant to their internal IT about it. Corporations or institutions that use Google as a provider should be seen with scepticism.
Everyone that can should host their own HTTP/1.1, DNS and SMTP at home on a battery backed Raspberry 1, 2 or Zero if they have an external IP.
To host anything beyond those protocols and/or on more powerful hardware is often counter productive.
The problem is getting the ports opened, you need to fight for that right even if it makes spam worse in the short term.
Fight for external IP, ports and static IP in that order.
To host anything beyond those protocols and/or on more powerful hardware is often counter productive.
The problem is getting the ports opened, you need to fight for that right even if it makes spam worse in the short term.
Fight for external IP, ports and static IP in that order.
Would using a VPS with a static IP not work?
Also does it help to use Google/Azure to host with regard to IP reputation with Gmail or Outlook?
Also does it help to use Google/Azure to host with regard to IP reputation with Gmail or Outlook?
Yes, as long as the IP doesn't already have some reputation attached to it. Here's a list of resources: https://sendgrid.com/blog/5-ways-check-sending-reputation/
was just gonna ask how you would handle DKIM and SPF stuff. Hetzner? Digitalocean?
I use a small datacentre in my country, actually not far from where I live. DKIM/SPF are independent of the provider. The easiest way to understand is to consider how receiving works. If I'm getting an email from hnemail.example, the first thing I do is consider the IP address. Oh, 257.257.257.1? Ok. So I then ask DNS "what is the SPF record for hnemail.example?" and it returns
For DKIM, when the email was sent it was signed with a key by the sending server. It is identified by a UUID in the incoming email. So the receiving server again queries DNS for TXT <UUID>._domainkey.hnemail.example and receives the public key as a response. Signature verification passes? Accept email. It fails? Mark as spam.
This doesn't have a lot to do with IP reputation. This is different. If you are a very large email provider, you might develop custom spam filters. IPs are allocated to 'autonomous systems' i.e. who actually uses them and hands them out to users, and depending on the business you might make some decisions about reputation. For example, if the IP address is part of a consumer ISP block that is handed out to users of broadband, chances are high that if they're sending email, it is probably a Windows PC compromised by malware.
Similarly, you might decide some ASNs are better than others. Some hosters are more liberal in what they will accept, such as VPN endpoints, tor nodes and such and as a consequence of this more spam comes from these ranges.
Rightly or wrongly, larger email providers try to add these extra filters to the process to protect their users from spam. This obviously sucks if you are genuinely trying to run an email server on your symmetric home fibre connection with a dedicated IP, but that's the world we live in.
I can't make any general statement on which providers might be best, and some people will have no issue whereas others will find themselves unable to send anything. I don't work for Outlook/Microsoft or Google and never have, so I don't know exactly what rules they use, and in all likeliness they shift constantly depending on spammer patterns. I can only say I've found running from a small DC to work pretty well.
v=spf1 mx -all
For DKIM, when the email was sent it was signed with a key by the sending server. It is identified by a UUID in the incoming email. So the receiving server again queries DNS for TXT <UUID>._domainkey.hnemail.example and receives the public key as a response. Signature verification passes? Accept email. It fails? Mark as spam.
This doesn't have a lot to do with IP reputation. This is different. If you are a very large email provider, you might develop custom spam filters. IPs are allocated to 'autonomous systems' i.e. who actually uses them and hands them out to users, and depending on the business you might make some decisions about reputation. For example, if the IP address is part of a consumer ISP block that is handed out to users of broadband, chances are high that if they're sending email, it is probably a Windows PC compromised by malware.
Similarly, you might decide some ASNs are better than others. Some hosters are more liberal in what they will accept, such as VPN endpoints, tor nodes and such and as a consequence of this more spam comes from these ranges.
Rightly or wrongly, larger email providers try to add these extra filters to the process to protect their users from spam. This obviously sucks if you are genuinely trying to run an email server on your symmetric home fibre connection with a dedicated IP, but that's the world we live in.
I can't make any general statement on which providers might be best, and some people will have no issue whereas others will find themselves unable to send anything. I don't work for Outlook/Microsoft or Google and never have, so I don't know exactly what rules they use, and in all likeliness they shift constantly depending on spammer patterns. I can only say I've found running from a small DC to work pretty well.
DKIM selectors aren't UUIDs. You can of course use a UUID as a selector, but you don't have to. My selectors are named S-YYYYMM (when I rotate the keys), so my current public key is at S-202001._domainkey.example.com.
A lot of tools generate UUIDs for the selector, just to get something unique without having to ask the user for something relevant or defining some other heuristic. For instance: the built-in helper tool for Zimbra generates a UUID by default, unless you provide something specific. I think a lot of people assume it should be a UUID just because they see UUIDs used in common examples.
Few people think about key cycling for DKIM as it isn't a built-in requirement at all, so once a UUID is set they just keep it until some point in the future that may never happen when they need to revoke the key because the private half is compromised.
Few people think about key cycling for DKIM as it isn't a built-in requirement at all, so once a UUID is set they just keep it until some point in the future that may never happen when they need to revoke the key because the private half is compromised.
Find a clueful small provider, local to you if possible. On huge providers like Hetzner and DO, you are guaranteed to have spammers as neighbours some of the time, even if the provider rapidly shuts them down. On the other hand, a good-quality small provider may rarely if ever host spammers.
Counterpoint, our mail admins spend a lot of time trying to convince small-scale providers to shutdown the spam email coming from them. Lots of people who host at small scale providers don’t care about patches, so they send tons of spam.
Doesn't seem like a counterpoint to me. The provider you're describing isn't clueful. A clueful provider pays attention to what's happening on their network and knows how to make themselves an unattractive host for spammers.
I suppose that's why the person you're replying to specified clueful!
I've been hosting my mail for 20+ years now, with minor issues. I guess I've been lucky.
Reading the comments here makes me incredibly sad. Every answer that tells me to use a provider misses the point. The Internet was created so that there could be many independent nodes, not so that everybody has to rely on one of several blessed providers. I should be able to run my own E-mail.
The real problem is lack of incentives. The big corps do not care about e-mail. It doesn't make money and isn't easily controllable. You can't turn it into a walled garden and lock users in. So, it gets minimal attention, and only defensive measures are developed.
Either we solve the spam problem, or things will get worse. The big tech corps won't solve it for us.
Reading the comments here makes me incredibly sad. Every answer that tells me to use a provider misses the point. The Internet was created so that there could be many independent nodes, not so that everybody has to rely on one of several blessed providers. I should be able to run my own E-mail.
The real problem is lack of incentives. The big corps do not care about e-mail. It doesn't make money and isn't easily controllable. You can't turn it into a walled garden and lock users in. So, it gets minimal attention, and only defensive measures are developed.
Either we solve the spam problem, or things will get worse. The big tech corps won't solve it for us.
I have also been self-hosting email for 15 years and only had couple of problems at the beginning, mainly until my IP got enough reputation. I have been hosting it on a bare metal Supermicro server in a proper datacenter, though. It has reverse-DNS, SPF, DKIM, TLS, MTA-STS and even DANE with DNSSEC (on a self-hosted BIND but that's another story). It is implemented using Exim, Dovecot, SpamAssasin, DNSBL and Roundcube with OpenLDAP auth. I can recommend this awesome hand-on guide provided by Netherlands Domain Registration Foundation as a basis of a nice configuration https://www.sidn.nl/en/news-and-blogs/hands-on-implementing-...
I had some troubles with IMAP search. I set up CLucene, it was easy and enough for me (no need for Java Lucene). It just took me a long time to figure out why it wouldn't search a domain part of email addresses. It just required to set up the tokenizations in such a way to split words also on @ character, i.e. don't consider a full email address as a word. :P I also had some troubles with OpenLDAP until I finally decided to read the docs and examples there properly. Since then I have been using this setup happily and it appears I will continue to do so! I also share the LDAP with NextCloud btw.
I had some troubles with IMAP search. I set up CLucene, it was easy and enough for me (no need for Java Lucene). It just took me a long time to figure out why it wouldn't search a domain part of email addresses. It just required to set up the tokenizations in such a way to split words also on @ character, i.e. don't consider a full email address as a word. :P I also had some troubles with OpenLDAP until I finally decided to read the docs and examples there properly. Since then I have been using this setup happily and it appears I will continue to do so! I also share the LDAP with NextCloud btw.
> I have been hosting it on a bare metal Supermicro server in a proper datacenter, though.
This is a key difference. Many people who have a bad time self-hosting mail set up with a minimum-effort major provider like Hetzner or DO — you're guaranteed to have spammers as neighbours some of the time, and other networks will behave accordingly when deciding whether to accept your mail.
A real server in a proper datacentre is great — I do the same — but as a lower-cost option a virtual server with a small, clueful provider who knows their customers also works just as well.
This is a key difference. Many people who have a bad time self-hosting mail set up with a minimum-effort major provider like Hetzner or DO — you're guaranteed to have spammers as neighbours some of the time, and other networks will behave accordingly when deciding whether to accept your mail.
A real server in a proper datacentre is great — I do the same — but as a lower-cost option a virtual server with a small, clueful provider who knows their customers also works just as well.
I've run my email server in VM's at a variety of ISP's like GoDaddy, Linode and AWS Lightsail and never had an issue, well once when Microsoft banned a group of IP's in a drag net. I filed a request with them and they fixed it in a week or so.
Real hardware servers are not required to host email.
Real hardware servers are not required to host email.
> Real hardware servers are not required to host email.
I agree, and said that in my parent comment.
I agree, and said that in my parent comment.
I had tried it on a very inexpensive VPS before having my own physical server. The IP was blocked almost everywhere. Even though I filled out a lot of forms in Outlook, Yahoo, etc, it appeared that those major email providers had blacklisted the entire subnet of the VPS provider, and IP reputation seemed to have made little difference. Choosing the right provider is crucial!
A new VPS provider may work well and make you happy, but later on, many spammers may start abusing it to send spam. Then, major email providers begin to block the entire IP subnets that belong to your VPS provider and you are screwed.
Since spammers typically can't afford to buy and house servers, having your own physical or dedicated server will likely increase the likelihood of everything being fine for a long time or forever.
A new VPS provider may work well and make you happy, but later on, many spammers may start abusing it to send spam. Then, major email providers begin to block the entire IP subnets that belong to your VPS provider and you are screwed.
Since spammers typically can't afford to buy and house servers, having your own physical or dedicated server will likely increase the likelihood of everything being fine for a long time or forever.
Selecting the right host is very important. They should be unattractive as a host for spammers. Get this part right and you don't have to worry that tomorrow your neighbour is a spammer. The virtual/dedicated thing is not as important — many providers comingle virtual and dedicated servers in the same address space anyway.
One example of a good signal: if you cannot sign up online, but must make personal contact with the provider (via email or other means) to set up a server. It's a minor inconvenience for you — you're only going to do it once — but spammers expect to have to do it many times and won't bother.
One example of a good signal: if you cannot sign up online, but must make personal contact with the provider (via email or other means) to set up a server. It's a minor inconvenience for you — you're only going to do it once — but spammers expect to have to do it many times and won't bother.
I don't agree, but it is true that some aggressive DNS RBL may include Digital Ocean, for example, just because [1].
The point is not that much that there are such blocking lists, the point is why admins use them. Surely that's blocking a good chunk of spam, but also legit sites.
1: "As you should know now: It is not you, it is your complete provider which got [redacted] listed."
The point is not that much that there are such blocking lists, the point is why admins use them. Surely that's blocking a good chunk of spam, but also legit sites.
1: "As you should know now: It is not you, it is your complete provider which got [redacted] listed."
DO regularly shows up on RBLs, even "less aggressive" ones, because it's regularly a source of spam. Same with Hetzner.
I don't think it matters which provider you choose. Big corps ban entire IP blocks, and you will get spammers as neighbors both at a low-cost provider and at a "proper datacenter".
FWIW, I've been using Hetzner for the last 10 years or so (but bare metal server, not a VPS) and haven't had issues.
FWIW, I've been using Hetzner for the last 10 years or so (but bare metal server, not a VPS) and haven't had issues.
It absolutely matters which provider you choose, assuming the provider is big enough to have their own address allocations (which is not that big, really — plenty of relatively small hosting providers all over the world have their own ASN and address allocations).
My recommendation is to use a provider that doesn't host spammers. Hetzner is not great at that in my experience, but sounds like they've been fine for you. Smaller providers who know their customers and are able to pay attention to how their network is being used are generally quite good at making things difficult enough for spammers that they go elsewhere.
My recommendation is to use a provider that doesn't host spammers. Hetzner is not great at that in my experience, but sounds like they've been fine for you. Smaller providers who know their customers and are able to pay attention to how their network is being used are generally quite good at making things difficult enough for spammers that they go elsewhere.
The problem is that collectively we love 'free' (at the point of sale) so much that we'll gladly allow gmail to just walk in and own almost the entirety of the email infrastructure. Then later we realize this gives them the ability to unilaterally make the rules, and we complain. But it's too late.
Also 20+ years, only very minor issues. The spam situation has gotten much better over that time, too. This topic comes up every so often on HN and I feel like the "never self-host, it's guaranteed to be a disaster, you have to use a centralised provider" crowd are just louder. Self-hosting my mail isn't something I think about much or talk about much. It's so obvious to me that it's worth doing, and it's extremely low effort.
> It's so obvious to me that it's worth doing, and it's extremely low effort.
DMARC, IP reputation and spams are not obvious.
DMARC, IP reputation and spams are not obvious.
Amazing — even though you quoted me, you still managed to reply to something I didn't say!
You say that self-hosting a mail server is obvious and extremely low effort, which simply is not true. But ok, let's say that you happen to have a clean IP and no spam or dmarc issues. You still had to choose what to deploy, which is not obvious and actually setup/maintain your server which is not extremely low effort.
Please read the comment you are replying to again. I said that it's obvious that it's worth doing, not that the work to set it up is "obvious".
Of course you need to learn some things, but it's not that much work. If you're the type that enjoys learning this sort of thing, you'll have a good time. If you're not, don't do it!
Of course you need to learn some things, but it's not that much work. If you're the type that enjoys learning this sort of thing, you'll have a good time. If you're not, don't do it!
I think the problem was this guy was providing email accounts for other people. Other people who probably reused their passwords and had their email in the same DB in the apps they used. That DB was compromised. Hackers got an account to send spam. Then the domain was blacklisted.
He alluded to this in
> At some point your IP range is bound to be banned, either by one asshole IP neighbor sending spam, one of your users being pwned
He alluded to this in
> At some point your IP range is bound to be banned, either by one asshole IP neighbor sending spam, one of your users being pwned
I understood that sentence as something that can happen to you even if you believe it won't.
> Please believe me. My current email server IP has been managed by me and used exclusively for personal email with zero spam, zero, for the last ten years.
> Please believe me. My current email server IP has been managed by me and used exclusively for personal email with zero spam, zero, for the last ten years.
> one of your users being pwned
Wow, good catch! He really buried the lede there -- "I kept getting blacklisted because my servers kept sending out spam." I don't know what to tell you pal -- spam is a serious problem, and if your users are sending it out, you're contributing to that. If you can't keep your users in line, other providers don't have any choice but to blacklist you.
Wow, good catch! He really buried the lede there -- "I kept getting blacklisted because my servers kept sending out spam." I don't know what to tell you pal -- spam is a serious problem, and if your users are sending it out, you're contributing to that. If you can't keep your users in line, other providers don't have any choice but to blacklist you.
Except that's not what he said happened to him.
He has that there as an example of things outside your direct control that can lead to blacklisting.
He has that there as an example of things outside your direct control that can lead to blacklisting.
> The real problem is lack of incentives. The big corps do not care about e-mail. It doesn't make money and isn't easily controllable. You can't turn it into a walled garden and lock users in. So, it gets minimal attention, and only defensive measures are developed.
Is this true? Even for consumer stuff, the only Google product besides search that seems widely popular is Gmail.
For enterprise, Microsoft is pulling in billions from office products. Without digging into their statements I know from my own experience that enterprise email services constitute a core piece of this.
For consumers, switching costs are low (mostly just inconvenient to go through all of your accounts) but not relative to the drawbacks of using corporate email providers which is zero unless you value your privacy.
Is this true? Even for consumer stuff, the only Google product besides search that seems widely popular is Gmail.
For enterprise, Microsoft is pulling in billions from office products. Without digging into their statements I know from my own experience that enterprise email services constitute a core piece of this.
For consumers, switching costs are low (mostly just inconvenient to go through all of your accounts) but not relative to the drawbacks of using corporate email providers which is zero unless you value your privacy.
I agree, the core of the enterprise suite of Microsoft is still very much based on domain controllers and Exchange. This enables the needs for user management and access policy configuration. Where I live self-hosting with Exchange is still the usual way too. No serious company would ever use Gmail.
I wouldn't use my corporate account for private mails though. It is used to register at services and interdict the spam.
I wouldn't use my corporate account for private mails though. It is used to register at services and interdict the spam.
>the Internet was created so that there could be many independent nodes, not so that everybody has to rely on one of several blessed providers.
any community that grows large enough needs some mechanism to manage trust, this is a universal issue. The early internet was more permissive and less differentiated simply because it was smaller.
The big corps do an alright job at managing spam given the sheer size of the problem, and more importantly you don't just need to solve spam, you need to do so economically, because for your system to stay distributed the nodes need to do the job competitively.
Given that there's intrinsic benefits to managing these things at scale that's not really realistic, in large systems you're always going to have division of labor and stratification for that reason.
any community that grows large enough needs some mechanism to manage trust, this is a universal issue. The early internet was more permissive and less differentiated simply because it was smaller.
The big corps do an alright job at managing spam given the sheer size of the problem, and more importantly you don't just need to solve spam, you need to do so economically, because for your system to stay distributed the nodes need to do the job competitively.
Given that there's intrinsic benefits to managing these things at scale that's not really realistic, in large systems you're always going to have division of labor and stratification for that reason.
spam is solved between the big players, they already use various feedback mechanisms ... it's just not enabled for small fish.
https://en.wikipedia.org/wiki/Feedback_loop_(email)
gmail silently drops emails (while reports smtp 250 accepted) - they could just as easily report that it's blackholed. spammers already do get through their fancy AI filters.
microsoft proactively blocks half of the world, rejects the incoming mail, and sends the sysadmin on a wild goose chase to get the IP/domain/whatever allowed. then you diligently register, and wait. and then no signal from them, and the problem still persists.
so big corps, small shops, everyone and their dog flocks to the good old microsoft/google duopoly.
and at this point if someone asks what to use for email knowing ... well, it's hard to not recommend folks to just use google workspace and have some kind of backup ready for when G bans their whole account just because.
https://en.wikipedia.org/wiki/Feedback_loop_(email)
gmail silently drops emails (while reports smtp 250 accepted) - they could just as easily report that it's blackholed. spammers already do get through their fancy AI filters.
microsoft proactively blocks half of the world, rejects the incoming mail, and sends the sysadmin on a wild goose chase to get the IP/domain/whatever allowed. then you diligently register, and wait. and then no signal from them, and the problem still persists.
so big corps, small shops, everyone and their dog flocks to the good old microsoft/google duopoly.
and at this point if someone asks what to use for email knowing ... well, it's hard to not recommend folks to just use google workspace and have some kind of backup ready for when G bans their whole account just because.
Use Microsoft or Google. Use your own domain. Know how to spin up the other quickly. Spend your time on things that matter.
I hope at some point the routine crushing of smaller providers gains the attention of the competition and markets authority, or its equivalents. It is crazy how Google and MS use their market power in this way and get away with it.
Not only they don't care about other e-mail providers, they often don't care about supporting widely used e-mail clients.
Massive corps that have a clue about how the internet works today badly care about email.
Email is your identity in many cases (for lack of a better solution).
Email -> identity -> tracking -> advertising revenue
Email is your identity in many cases (for lack of a better solution).
Email -> identity -> tracking -> advertising revenue
After 30 years, I don't think spam is going to get solved with email.
Here are some ideas that could solve spam:
1. Require the email receiver to add the sender to their contact list, then put all other emails in "Junk". The UX of this could be quite good nowadays with smartphones and QR codes. The changes to email apps are minimal:
Here are some ideas for improving the current system, but not actually solving spam:
3. Create a protocol for automatically reporting emails marked as spam back to their admins. Servers will only accept signed emails. Then servers can do lots of things like:
5. Build a shared spam reporting system that accepts only signed emails and supports searching by email address. Receivers can use it to identify compromised user accounts and reject their email. Senders use it to identify receivers acting in bad faith (reporting ham as spam). A centralized version would be straightforward. A decentralized version would be a challenging project.
6. Add support to the SMTP protocol for reporting rate limits and cooling-off times to email senders. Admins of large shared email systems can feed these metrics into a monitoring system and receive alerts of problems early.
[0] https://github.com/mleonhard/hipp
EDIT: tone
1. Require the email receiver to add the sender to their contact list, then put all other emails in "Junk". The UX of this could be quite good nowadays with smartphones and QR codes. The changes to email apps are minimal:
- 1) When a user opens a "mailto:" URL, the email program shows the normal "send email" screen with a "Just add to Contacts" button at the top.
- 2) Email program has a function to show a QR code with the user's "mailto:" URL.
Here are some ideas for improving the current system, but not actually solving spam:
3. Create a protocol for automatically reporting emails marked as spam back to their admins. Servers will only accept signed emails. Then servers can do lots of things like:
- Mark sent emails as "Receiver flagged this as spam" and inform the user.
- When an account begins sending spam, rate limit it, disable it pending password reset, or alert the admin.
5. Build a shared spam reporting system that accepts only signed emails and supports searching by email address. Receivers can use it to identify compromised user accounts and reject their email. Senders use it to identify receivers acting in bad faith (reporting ham as spam). A centralized version would be straightforward. A decentralized version would be a challenging project.
6. Add support to the SMTP protocol for reporting rate limits and cooling-off times to email senders. Admins of large shared email systems can feed these metrics into a monitoring system and receive alerts of problems early.
[0] https://github.com/mleonhard/hipp
EDIT: tone
I think the real problem is how bad most e-mail interfaces are (makes you want to use it less) or that there is not a dead simple way to do it other than a provider. It should be easy enough that I can download an app on my phone and be sending messages.
I think matrix might get rid of a lot of the infrastructure barriers, but I don't have much experience with it.
I think matrix might get rid of a lot of the infrastructure barriers, but I don't have much experience with it.
Maybe there needs to be a self-hosting association/union that self-hosters can join? It could advocate for adherence to open standards, and an equal standing for small servers. It could also be a repository of advice for current best practise in small server administration and configuration. Should it be under the auspices of an existing group such as FreedomBox?
Great idea.
As you say though, there are existing groups. EFF[0] comes to mind. They advocated for net neutrality back in the day.
[0] https://www.eff.org/
As you say though, there are existing groups. EFF[0] comes to mind. They advocated for net neutrality back in the day.
[0] https://www.eff.org/
G'Day Femto
Over the past 15+ years I have joined a number of groups like a web site 'web hosting talk' and all of them jumped all over my privacy and passed on my details to spammers / hackers.
I know what you are thinking -> how would he know that ?
Well its really simple I use DedicatedEmailAddressing ( DEA ) and our system tells me when a 3rd party tried to deliver a spam message using one of these DEAs.
Some of our customers charge suppliers for a new replacement DEA when they find the supplier has 'leaked' :)
Yeah, the host I use sells on webhostingtalk and I'm 99% sure they're selling my info on. My cPanel username there is the only place that I have that username, and that's what a massive chunk of my spam is directed to.
I'm a newbie to YC, can't c 'Edit' button, so I'm assuming it is like Twit. I'm not saying I wouldn't be interested, I think it is a really good idea, but really concerned that I don't find it is a waste of time after my privacy was again splattered
+1. Maybe it should be like ham radio, where the association tries to police its own members.
This sounds like a great idea. Email is not something you can drop from one day to another, there's no real alternative as most services require it.
This would require certain rules to avoid SPAM, but the outcome is totally worth for me.
This would require certain rules to avoid SPAM, but the outcome is totally worth for me.
I really like this Idea and would love to join such an organization!
The sweet spot for having control over your email while simultaneously minimizing unforseen headaches is to simply own your domain name and point the MX record to whatever hosting provider you want instead of self-hosting a server at home.
Same philosophy for exposing a your personal blog of html files or content like mp4 videos. The sweet spot is to focus on buying a domain name you control. Then let Amazon S3, or Cloudflare, Hezner etc, host your html or mp4 files.
I quit self-hosting email at home over 15 years ago. It's just not something I want to babysit anymore because I have other things to focus on. As long as I control the MX record on my own domain, that's really all that's necessary.
Same philosophy for exposing a your personal blog of html files or content like mp4 videos. The sweet spot is to focus on buying a domain name you control. Then let Amazon S3, or Cloudflare, Hezner etc, host your html or mp4 files.
I quit self-hosting email at home over 15 years ago. It's just not something I want to babysit anymore because I have other things to focus on. As long as I control the MX record on my own domain, that's really all that's necessary.
There is also a happy medium. Host your own MX servers but use someone else's SMTP servers. You have complete control over the incoming mail but dodge the filters by using the established business for sending mail.
> Host your own MX servers but use someone else's SMTP servers.
Yes. My outgoing email goes out via Sonic's SMTP server, with the SPF records to allow it to have a source address of my own domain. Incoming email goes to my own domains and gets forwarded.
This seems to be trouble-free. The domain is on a cheap shared hosting account. I'm not running a server. I own the domain, and not though the hosting company, so I can switch to another provider if necessary. In 27 years, I've had to do that twice, because the hosting provider went out of business.
This is easy to do, and I don't have to deal with Google. I don't even get much spam. All the spammers seem to be targeting the big services now.
I just looked at my spam folder. I'm regularly being offered dental supplies, large hydraulic sheet metal bending presses, and ammonium sulfate fertilizer. It seems that having your own domain now means you get mostly business-to-business spam. I subscribe to Machine Design, which gets me some heavy industrial marketing, but I have no idea why I get dental supply ads. The fertilizer spams, from China, look like a scam - there's a fertilizer shortage, so that's a spam which might get replies.
Yes. My outgoing email goes out via Sonic's SMTP server, with the SPF records to allow it to have a source address of my own domain. Incoming email goes to my own domains and gets forwarded.
This seems to be trouble-free. The domain is on a cheap shared hosting account. I'm not running a server. I own the domain, and not though the hosting company, so I can switch to another provider if necessary. In 27 years, I've had to do that twice, because the hosting provider went out of business.
This is easy to do, and I don't have to deal with Google. I don't even get much spam. All the spammers seem to be targeting the big services now.
I just looked at my spam folder. I'm regularly being offered dental supplies, large hydraulic sheet metal bending presses, and ammonium sulfate fertilizer. It seems that having your own domain now means you get mostly business-to-business spam. I subscribe to Machine Design, which gets me some heavy industrial marketing, but I have no idea why I get dental supply ads. The fertilizer spams, from China, look like a scam - there's a fertilizer shortage, so that's a spam which might get replies.
>I just looked at my spam folder. I'm regularly being offered dental supplies, large hydraulic sheet metal bending presses, and ammonium sulfate fertilizer. It seems that having your own domain now means you get mostly business-to-business spam. I subscribe to Machine Design, which gets me some heavy industrial marketing, but I have no idea why I get dental supply ads. The fertilizer spams, from China, look like a scam - there's a fertilizer shortage, so that's a spam which might get replies.
I used to have a 'Shirley' email all the time from China about elevator parts. It was such a niche kind of spam that I eventually replied and requested pictures of a bunch of parts. They sent them!
I used to have a 'Shirley' email all the time from China about elevator parts. It was such a niche kind of spam that I eventually replied and requested pictures of a bunch of parts. They sent them!
> I used to have a 'Shirley' email all the time from China about elevator parts. It was such a niche kind of spam that I eventually replied and requested pictures of a bunch of parts. They sent them!
https://web.archive.org/web/20030412191437/http://www.penny-...
https://web.archive.org/web/20030412191437/http://www.penny-...
I remember in grad school, one of my classmates being upset about getting a porn spam email in Chinese and saying, “but don’t they know that I’m a girl?” I pointed out to her that I got the same emails (but had no idea what they were saying) and explaining to her that they send them to everybody.
It can very much depend on your domain. In the early 2000s I ran a domain for a firm and the domain had 'UT' in it. Well spammers thought it was related to university of Texas at some point and we went from around 10 messages a day to over quarter of a million. The immediate issue I has was poor back scatter protection so I had to configure that in a few days. That stopped 99.99% of the spam, but still caused massive issues with the mailboxes that did exist getting hundreds of messages per day even though we were blocking tens of thousands of mails per valid email addresses.
Spammers are miserable, I don't fault anyone for giving up.
Spammers are miserable, I don't fault anyone for giving up.
SPF and DKIM no longer have any real value. Just look at SA and the values it assigns by default to mail with those headers. Couple that will the fact that spam now has valid SPF and DKIM - just makes them pointless.
DKIM and SPF, along with DMARC are more about authentication - they let you know that the message has come from where it says it does. This makes spoofing harder, not necessarily spam. Greylisting deals with most of the "spray and prey" spammers, though it does have it's own issues.
Agreed.
I do this too with a different provider whose whole business is relaying for self hosters. I have had exactly 0 problems since I started a couple of years ago. Its kind of perfect.
Which provider is this? I'm thinking of switching to this middle ground option too.
This should be made a standard practice.
Third party email providers, the so-called "established businesses", get a free pass as sending SMTP servers that are accepted by almost all receiving STMP servers. Everyone just assumes everything coming from those SMTPs is legit. Establishing this "legitimacy" and getting the "free pass" is difficult and some have suggested, the third parties may employ anticompetive tactics.
However, IMHO the receiving SMTPs is a different issue. Why do we let these third parties receive and store our email. (Why do our homes have their own mailboxes. Why not use a "P.O. Boxes" instead.) Eventually we could move away from letting third parties control the receipt of our mail. Neither "POP3" nor "webmail" was part of the original concept of email.
Today, it is easier than ever to set up overlay networks where we can assign our own IP addresses and run our own SMTP servers that can communicate directly with other SMTP servers on the overlay network. These networks are not open to the world, they may only be open to people we know. Much of our mail is between people who know each other, e.g., friends, family, colleagues. Or businesses that we contact first. We can separate different social and business networks on different overlay networks.
Anyway, the sending and receiving of mail can be separated. We do not need to let a third party control both.
Third party email providers, the so-called "established businesses", get a free pass as sending SMTP servers that are accepted by almost all receiving STMP servers. Everyone just assumes everything coming from those SMTPs is legit. Establishing this "legitimacy" and getting the "free pass" is difficult and some have suggested, the third parties may employ anticompetive tactics.
However, IMHO the receiving SMTPs is a different issue. Why do we let these third parties receive and store our email. (Why do our homes have their own mailboxes. Why not use a "P.O. Boxes" instead.) Eventually we could move away from letting third parties control the receipt of our mail. Neither "POP3" nor "webmail" was part of the original concept of email.
Today, it is easier than ever to set up overlay networks where we can assign our own IP addresses and run our own SMTP servers that can communicate directly with other SMTP servers on the overlay network. These networks are not open to the world, they may only be open to people we know. Much of our mail is between people who know each other, e.g., friends, family, colleagues. Or businesses that we contact first. We can separate different social and business networks on different overlay networks.
Anyway, the sending and receiving of mail can be separated. We do not need to let a third party control both.
Feel free to do this, nobody will join you so the spam will be virtually nonexistent!
Adding to the happy medium is to teach your friends and family to use Thunderbird so they can easily GPG encrypt [1] their emails keeping the nosey email providers off the email body. Also teach them to use the IMAPS (TLS) endpoint for their mail provider, usually port 993. There are probably simpler how-to's with pictures, I just do not have any of them handy.
[1] - https://support.mozilla.org/en-US/kb/openpgp-thunderbird-how...
[1] - https://support.mozilla.org/en-US/kb/openpgp-thunderbird-how...
I'm sorry to say, that's nowhere near a medium stance or ask.
It could be we have different circles of acquaintances. I have managed to get non technical friends to GPG encrypt their emails. I also talked 2 lawyers into using this and the two lawyers are not only non technical but have nearly zero patience.
I guess a lawyer might know a thing or two about confidential messages.
I think you’d be very surprised at how much lawyers don’t know or care about any of that. They store stuff in the cloud without ever having heard of the Third Party Doctrine (which allows the US government warrantless access to those documents).
> store stuff in the cloud without ever having heard of the Third Party Doctrine (which allows the US government warrantless access to those documents)
Or because they know the third-party doctrine doesn’t apply to attorney work product and privileged materials.
Or because they know the third-party doctrine doesn’t apply to attorney work product and privileged materials.
After doing work for lawyers for years, they really are the worst when it comes to understanding computers.
It does no such thing, that’s absurd. See carpenter v. United States (among many, many others).
[deleted]
They do. Most of them use proprietary https web interfaces that usually have "secure email" or "secure messaging" in the description but they are just fancy web portals. I despise those systems. The content is not encrypted at rest and can be leaked. With OpenGPG the emails are only decrypted on the recipients end points and can be deleted by request.
Not all of them. I keep seeing lawyers sending photos of sensitive documents using their cellphone and Whatsapp (before encryption), and most of them don't even care about all those documents still residing on their cellphone, at the mercy of whoever steals it since all the security they have in place is that swipe thing that anyone with good eyes spots in 2 seconds straight.
ok, this is the text book definition of "out of touch"
Yep. On any other site I’d assume that comment was satire. Nothing’s changed in the last decade:
https://moxie.org/2015/02/24/gpg-and-me.html
https://blog.cryptographyengineering.com/2014/08/13/whats-ma...
https://moxie.org/2015/02/24/gpg-and-me.html
https://blog.cryptographyengineering.com/2014/08/13/whats-ma...
even zimmeman says it's too complicated for the unwashed masses and was meant for the techy crowd of the time he came up with it.
My canned response to the second article/rant:
* https://articles.59.ca/doku.php?id=pgpfan:wtmwp
* https://articles.59.ca/doku.php?id=pgpfan:wtmwp
Most of my friends and family don’t own computers, only phones and sometimes tablets.
K-9 Mail is an app available for Android which supports PGP (if clunkily).
<https://k9mail.app/>
I'm not going to remotely pretend that this would make PGP attractive or viable for the greater public. I do want to address your valid point that the general-purpose computer seems to be a dying breed, but that there are still options for mobile devices.
I've used K-9 Mail, though rarely do these days (it went poorly-maintained for a while).
There seem to be some options for iOS as well, such as iPG Mail, though I've no experience at all with it.
<https://www.openpgp.org/software/ipgmail/>
<https://k9mail.app/>
I'm not going to remotely pretend that this would make PGP attractive or viable for the greater public. I do want to address your valid point that the general-purpose computer seems to be a dying breed, but that there are still options for mobile devices.
I've used K-9 Mail, though rarely do these days (it went poorly-maintained for a while).
There seem to be some options for iOS as well, such as iPG Mail, though I've no experience at all with it.
<https://www.openpgp.org/software/ipgmail/>
Most of my friends and family don't have Android devices.
Perhaps you might care to share with us what types of devices they do have, seeing as my mental telepathy transgizmatron happens to be in the shop presently.
Note that I also listed an iOS option, and mentioned that there are others.
Again: PGP-enabled email apps are available on many platforms, though, again, this addresses only a small part of the larger challenge at encouraging adoption.
Note that I also listed an iOS option, and mentioned that there are others.
Again: PGP-enabled email apps are available on many platforms, though, again, this addresses only a small part of the larger challenge at encouraging adoption.
there is definitely software to help with pgp on ANY tablet, phone, or PC your friends and family have access to. The universal problem is no one cares to encrypt except a few computer nerds, spooks, and journalists.
It means you now have to worry about key management or else lose access to all your old mail, it possibly complicates logging in on new devices or via webmail, partially breaks server-side (spam) filtering, breaks server-side full-text search… so there are some definitive trade-offs there to be made.
Not a single member of my friends or family is going to use GPG encryption, not even my mother who loves me dearly. I had a computer geek "penpal" for a while and we did monthly emails to each once a month to catch up via gpg, and even that died off lol
[deleted]
Apple supports encryption on Mail:
https://support.apple.com/en-gb/guide/mail/mlhlp1180/mac
https://support.apple.com/en-gb/guide/mail/mlhlp1180/mac
protonmail also has nice tools for it if you use them. I suspect tutanota probably does as well.
Apple mail support GPG easily. No need to use Thunderbird.
Can you please provide a few references to that claim?
They might be talking about S/MIME.
I use https://github.com/Free-GPGMail/Free-GPGMail which is a plugin for GNUPG, without the "support" plan.
I use https://github.com/Free-GPGMail/Free-GPGMail which is a plugin for GNUPG, without the "support" plan.
https://gpgtools.org
GPG Mail is a paid for add-on, but it works and it works well.
GPG Mail is a paid for add-on, but it works and it works well.
It's not a paid add-on... GnuPG is open source. They are selling a support plan in case you need help with open source.
Here is the same thing, without the paid support. https://github.com/Free-GPGMail/Free-GPGMail
- GPG Suite was released under an Open Source license in the past. Is that stil the case?
You GPG Suite including GPG Mail is still going to be released under an Open Source license.
- Am I still allowed to compile my own version of GPG Mail / GPG Suite removing any code regarding the trial or activation?
You absolutely are, the GPL makes sure of that. You will find the source code on this website next to the download link for GPG Suite.
We would kindly like to ask you not to use our names or icons if you plan to publish a binary for others to use. Also please do not release any complete installers as that may suggest that they are official releases.
https://gpgtools.org/faq
Here is the same thing, without the paid support. https://github.com/Free-GPGMail/Free-GPGMail
- GPG Suite was released under an Open Source license in the past. Is that stil the case?
You GPG Suite including GPG Mail is still going to be released under an Open Source license.
- Am I still allowed to compile my own version of GPG Mail / GPG Suite removing any code regarding the trial or activation?
You absolutely are, the GPL makes sure of that. You will find the source code on this website next to the download link for GPG Suite.
We would kindly like to ask you not to use our names or icons if you plan to publish a binary for others to use. Also please do not release any complete installers as that may suggest that they are official releases.
https://gpgtools.org/faq
To be pedantic, the version I linked to is a paid addin. However, you are correct - there is/are unofficial unpaid and unsupported versions available that I was unaware of, per the FAQ and the repository that you linked, which has nothing to do with GPGTools GmbH; from the repo, “This does not constitute any endorsement or promotion of our releases by the respective copyright or trademark holders.” Thanks for sharing this.
You can also use an MX backup service which will accept mail when your server is offline (and it will resend it when the server comes online)
You can also even keep Gmail as your MX server! Just move messages off of it as soon as they arrive. It's just a mailbox, after all
You can also even keep Gmail as your MX server! Just move messages off of it as soon as they arrive. It's just a mailbox, after all
> You can also even keep Gmail as your MX server! Just move messages off of it as soon as they arrive. It's just a mailbox, after all
Do you have more information about this?
I've been looking to do something similar so that I can have my mail sorted into folders without setting up the same rules on multiple clients. (Gmail's sorting doesn't seem to support some of the sorting I'm currently doing in Thunderbird)
Do you have more information about this?
I've been looking to do something similar so that I can have my mail sorted into folders without setting up the same rules on multiple clients. (Gmail's sorting doesn't seem to support some of the sorting I'm currently doing in Thunderbird)
I've been hosting and operating my own MTA MX with Postfix since 2005. I have Postfix set to use "MailDir" format storage (one file per email) and then use Procmail filters to direct emails into per-sender or per-topic specific folders when they arrive on the server - nothing needed to be done in the email client. Dovecot provides the IMAP4 client interface. Thunderbird connects via IMAP4.
Each domain has its own user home directory so for each there is a /home/${domain_name}/Maildir/ directory as the base for storing emails, and each IMAP4 folder has an associated directory. Snippet:
Each domain has its own user home directory so for each there is a /home/${domain_name}/Maildir/ directory as the base for storing emails, and each IMAP4 folder has an associated directory. Snippet:
$ ls -1da Maildir/.Technology.FOSS.Projects.Linux*
Maildir/.Technology.FOSS.Projects.Linux
Maildir/.Technology.FOSS.Projects.LinuxContainers
Maildir/.Technology.FOSS.Projects.Linux.drbd
Maildir/.Technology.FOSS.Projects.Linux.kernel
Maildir/.Technology.FOSS.Projects.Linux.linaro.dev
Maildir/.Technology.FOSS.Projects.Linux.linux-i2c
Maildir/.Technology.FOSS.Projects.Linux.linux-input
Maildir/.Technology.FOSS.Projects.Linux.linux-pci
Maildir/.Technology.FOSS.Projects.Linux.linux-usb
:0H
* ^List-id: .*linux-usb\.vger\.kernel\.org
$HOME/Maildir/.Technology.FOSS.Projects.Linux.linux-usb/
:0H
* ^List-id: .*linux-wireless\.vger\.kernel\.org
$HOME/Maildir/.Technology.FOSS.Projects.Linux.linux-wireless/
:0H
* ^List-id: .*yaffs\.lists\.aleph1\.co\.uk
$HOME/Maildir/.Technology.FOSS.Projects.Linux.yaffs/
:0H
* ^List-id: .*util-linux\.vger\.kernel\.org
$HOME/Maildir/.Technology.FOSS.Projects.Linux.util-linux/
### LinuxContainers
:0H
* ^List-id: .*lxc-devel\.lists\.linuxcontainers\.org
$HOME/Maildir/.Technology.FOSS.Projects.LinuxContainers/
### Linaro
:0H
* ^List-id:.*linaro-dev\.lists\.linaro\.org
$HOME/Maildir/.Technology.FOSS.Projects.Linux.linaro.dev/
:0H
* ^List-id:.*linaro-kernel\.lists\.linaro\.org
$HOME/Maildir/.Technology.FOSS.Projects.Linux.linaro.kernel/[deleted]
Whose SMTP service you'd recommend?
I use zoho.com, $12 a year and I haven't had any issues with blacklists.
Sendgrid works well for small volumes
Yep SMTP relay is the only sane choice nowadays sadly. Sending your own mail is a losing battle against the email blacklist cartel.
No...it isn't the only sane choice. If it works for you, great. Forunately for Internet freedom, there many of us small timers left who have no issues hosting our own email.
I have no issue hosting, it's just tiresome keeping clear of Spamhaus etc.
Using an SMTP relay was a suitable trade-off, for me, but I respect those who are able to endure the pains of sending non-flagged emails without it.
Using an SMTP relay was a suitable trade-off, for me, but I respect those who are able to endure the pains of sending non-flagged emails without it.
[deleted]
You are totally missing the simple fact that the number of blessed email providers to choose from is slowly going down. I've seen ISPs with thousands of clients to give up and move the mailboxes to large players simply because their clients' email was ending up in the spam so often that running the support has gotten too expensive.
It's definitely an anticompetitive practice.
It's definitely an anticompetitive practice.
> You are totally missing the simple fact that
Please make your substantive points without swipes. Your comment would be fine without that bit.
https://news.ycombinator.com/newsguidelines.html
Please make your substantive points without swipes. Your comment would be fine without that bit.
https://news.ycombinator.com/newsguidelines.html
Indeed. I host my domains with dreamhost and it turns out that outgoing mail from their servers will get marked as spam by Google. The exact same mail sent through a gmail address (whether under gmail.com or a custom domain) will be delivered no problem (although after the sudden closing of legacy free email, I found that emails that I had been sending via gmail with my own domain that had been getting blocked by spam filters were being delivered when they came from a gmail.com address).
The problem is your outbound mail server does not have a good reputation. You also probably do not have a large enough swath of IP space to prevent bad neighbors from becoming your problem. Lots of tricks to getting your email sent reliably to most mail servers. What got me out of hosting mail was trying to send an email to a potential lead I met at a local meetup. His email was hosted at some very small and relatively unknown university, but my email was flat out rejected. Something about that moment just clicked that it's not worth my time to chase down the admins and resolve the problem. I'll just start shunting my email through O365. I still selfhost everything else I can, but I offload mail management to a provider
It's more than that. I'm on a mailing list for a social organization that I've belonged to for 20 years, and has been hosted at the same place for all of that time. I have marked hundreds of the messages not spam, and still Google sends about 80% of them to spam.
Google does not care even if you have regular correspondence with an address -- if the server isn't big enough, it's going to spam. The user's wishes or ideas about what is or isn't spam are irrelevant.
Google does not care even if you have regular correspondence with an address -- if the server isn't big enough, it's going to spam. The user's wishes or ideas about what is or isn't spam are irrelevant.
I'm surprised that you didn't just whitelist the email after the repeated ham events...!
As a counterpoint, I host a couple of domains with Dreamhost and have had no such issues. They've been serving the email for my main domain for 15 years and AFAIK I've never had any of those emails go directly to spam within Gmail or elsewhere. I do periodic smoke testing by sending test emails to different place, plus I have a reasonable number of external users making use of the domains for email as well. FWIW I moved all my DNS records to Cloudflare, but that was only in last couple of years.
I can pick any ISP today, including Dreamhost, and have clean email reputation and sending ability with FAANG and Microsoft in less than 30 days.
>It's definitely an anticompetitive practice.
yeah, but in this climate what are you gonna do? It's not like there's any kinda recourse for monopolistic behavior that has any teeth to it.
yeah, but in this climate what are you gonna do? It's not like there's any kinda recourse for monopolistic behavior that has any teeth to it.
Sure there is.. unless you mean no recourse in this environment. The natural recourse would be to force Google to divest GMail from it's core business.
No, the recourse is to force net neutrality. You must process all emails the same way and they must go through the same filters. In fact if spam processing is part of modern email it should be part of the standard and then all mail servers must be forced to conform to the standard.
GDPR worked.
I just randomly looked at 8 different emails in my inbox. All of them were from different email providers (except google which was there twice). There's hundreds or thousands of email providers you can chose from.
iphmx 1 google 2 kornet 1 linkedin 1 secureserver 1 amazonses 1 self hosted university email 1
iphmx 1 google 2 kornet 1 linkedin 1 secureserver 1 amazonses 1 self hosted university email 1
Agreed.
It’s just because of spam. All open systems that do not impose a cost to participate are destroyed by spam.
True. But they are not calling for a completely open system. I like this proposal from the author.
> Change blacklisting protocols so they are not permanent and use an exponential cooldown penalty. After spam is detected from an IP, it should be banned for, say, ten minutes. Then, a day. A week. A month, and so on. This discourages spammers from reusing IPs after the ban is lifted and will allow the IP pool to be cleaned over time by legitimate owners.
> There should be a recourse for legitimate servers. I'm not asking for a blank check. I don't mind doing some paperwork or paying a fee to prove I'm legit. Spammers will not do that, and if they do, they will get blacklisted anyways after sending more spam.
But Big Tech will not do that because they will gain more from eliminating the competition.
> Change blacklisting protocols so they are not permanent and use an exponential cooldown penalty. After spam is detected from an IP, it should be banned for, say, ten minutes. Then, a day. A week. A month, and so on. This discourages spammers from reusing IPs after the ban is lifted and will allow the IP pool to be cleaned over time by legitimate owners.
> There should be a recourse for legitimate servers. I'm not asking for a blank check. I don't mind doing some paperwork or paying a fee to prove I'm legit. Spammers will not do that, and if they do, they will get blacklisted anyways after sending more spam.
But Big Tech will not do that because they will gain more from eliminating the competition.
> I don't mind doing some paperwork or paying a fee to prove I'm legit.
Then how about this: The big email companies all declare one day that any newly registered domain (with an MX record) needs to post a bond for good behaviour in escrow somewhere. If any of them find the domain being used to send spam, they can slash the bond (sending it to some charity or something).
This has the advantage that it doesn't affect any existing senders (so there's no one to complain about it), and it makes transparent the cartel-like power that these companies have over email. Perhaps, to democratise the process a bit, the ITU could organise a ballot (one vote per country) to elect 5 companies/non-profits who would have this bond-slashing power.
Unfortunately to implement something like this, they'd also probably have to demand that DKIM signing become mandatory (so there are cryptographic proofs of any evidence of spamming), and this sort of global consensus / money processing scheme would probably end up being built using a blockchain, whether that was a good idea or not.
Then how about this: The big email companies all declare one day that any newly registered domain (with an MX record) needs to post a bond for good behaviour in escrow somewhere. If any of them find the domain being used to send spam, they can slash the bond (sending it to some charity or something).
This has the advantage that it doesn't affect any existing senders (so there's no one to complain about it), and it makes transparent the cartel-like power that these companies have over email. Perhaps, to democratise the process a bit, the ITU could organise a ballot (one vote per country) to elect 5 companies/non-profits who would have this bond-slashing power.
Unfortunately to implement something like this, they'd also probably have to demand that DKIM signing become mandatory (so there are cryptographic proofs of any evidence of spamming), and this sort of global consensus / money processing scheme would probably end up being built using a blockchain, whether that was a good idea or not.
I can just imagine the headline. “Ask HN: Google sent my mail bond to charity for no reason and has torpedoed my small business, and I can’t get in touch with anyone to make it right”
I can imagine headlines like that too, but the idea of electing 5 (or some other odd number of) entities is that they would be able to share among themselves the cryptographically signed evidence of the spam they detected, and then the bond slashing would require a majority vote.
So instead, the headline should be something like "Ask HN: Google, Amazon, and the Shanghai Cooperation Organisation forced me to send $100 of Ether to UNICEF and I couldn't send any new emails until I sent another $100 payment to my domain registrar. How do I take them to the World Court to force them to reimburse me?". That's not a great situation, but it's slightly better than the status quo.
So instead, the headline should be something like "Ask HN: Google, Amazon, and the Shanghai Cooperation Organisation forced me to send $100 of Ether to UNICEF and I couldn't send any new emails until I sent another $100 payment to my domain registrar. How do I take them to the World Court to force them to reimburse me?". That's not a great situation, but it's slightly better than the status quo.
You're describing one of the solutions made by Ironport, "Bonded Sender". Their solutions were sold to Return Path and Cisco later bought them out, presumably with the bonded sended solution still belonging to Return Path? [1,2]
I've never seen discussion of this in the mainstream though... so I'm not sure if it's actually being used or just shelved.
At this point, I think any proprietary they've created is game for usage. But it's very hard to get multiple large organizations to adopt this.
I definitely think it's a solution.
[1] https://archive.ph/CH98s [2] https://www.computerworld.com/article/2548788/cisco-to-acqui...
I've never seen discussion of this in the mainstream though... so I'm not sure if it's actually being used or just shelved.
At this point, I think any proprietary they've created is game for usage. But it's very hard to get multiple large organizations to adopt this.
I definitely think it's a solution.
[1] https://archive.ph/CH98s [2] https://www.computerworld.com/article/2548788/cisco-to-acqui...
Is there any actually money to be made in hosting email for people? I genuinely don’t know but my suspicion is that GMail, Yahoo, Outlook, et al are loss leaders for their owner companies. I suspect people at those companies would be quite happy if the protocol got unfucked enough that it small players could participate without negatively impacting the network.
O365 web costs me 5 bucks per month and I only use it for a few emails a week so I doubt it's a loss leader.
If I'd actually use all of it a lot, sure but I don't.
If I'd actually use all of it a lot, sure but I don't.
Do you just get email with that? Or maybe is there an office suite also?
There's an office suite included yes but only web based which really sucks. Half the office features are not supported.
It also doesn't work properly on Firefox on FreeBSD.
Even though on other OSes it works ok, it's so limited I can't imagine anyone using the web version of office 365 for any serious activity.
It also doesn't work properly on Firefox on FreeBSD.
Even though on other OSes it works ok, it's so limited I can't imagine anyone using the web version of office 365 for any serious activity.
Far be it for me to tell people what setup they should use but it sure sounds like you are unhappy with the value you are getting for your money.
I am. But services providing just email usually are about the same price.
I gladly pay for Fastmail and I assume they’re not running a charity. Also, I think hey.com is charging $100 per year.
I am not saying there aren’t paid options. But Fastmail isn’t who we are talking about being a bad actor here and is much smaller than GMail.
You asked it there is any money to be made in email, I provided you with anecdotal evidence that there was.
Spam has been been fought for decades, you can rest assured any obvious solution has been tried and either doesn’t have the desired effect or is impossible to implement.
You ignore the fact that there are perverse incentives among the participants. It's possible to implement, and I'm doing it myself. If I had more time to spend on it, we could end spam. Instead I am fine as is: most of the spammers have given up.
I think we ought to move email (or some future incarnation of email, like matrix) to a completely whitelist (opt-in to receive messages) basis.
signup confirm emails are like that.
similarly, any "hello pls add me to your allow-list" emails could be made auto-disappear to the "will be deleted in 30 days" folder in ~10-15 minutes, so even if you get a 100 spam messages per day you only see the last of those, you can easily pick what you are looking for, and don't worry about the rest, they'll just disappear.
(and you still have 30 days to look for messages that might be interesting/important/etc.)
...
the real missing piece is the feedback mechanism. DMARC is meh. of course large senders have implemented FBL, but they are not available for mere mortals.
https://en.wikipedia.org/wiki/Feedback_loop_(email)
similarly, any "hello pls add me to your allow-list" emails could be made auto-disappear to the "will be deleted in 30 days" folder in ~10-15 minutes, so even if you get a 100 spam messages per day you only see the last of those, you can easily pick what you are looking for, and don't worry about the rest, they'll just disappear.
(and you still have 30 days to look for messages that might be interesting/important/etc.)
...
the real missing piece is the feedback mechanism. DMARC is meh. of course large senders have implemented FBL, but they are not available for mere mortals.
https://en.wikipedia.org/wiki/Feedback_loop_(email)
signup confirm emails are not what i'm describing, because you need to establish and filter the initial offer that they send you via email itself, which is still prone to phishing.
What I'm describing is a situation where users themselves have to proactively subscribe to a connection using some sort of out-of-band mechanism. For example, if a website wanted to send you emails, they could produce some sort of "connection ticket" that you can give to your email client in order to subscribe to them.
What I'm describing is a situation where users themselves have to proactively subscribe to a connection using some sort of out-of-band mechanism. For example, if a website wanted to send you emails, they could produce some sort of "connection ticket" that you can give to your email client in order to subscribe to them.
This is useless because users are stupid, people sending the mail are stupid, people getting the mail are stupid, UI people creating interfaces are so stupid society could be improved by putting them in a box and mailing them all to some wasteland and hoping they form their own society there or starve.
This would result in half the planet being frustrated all the time and the other half never getting their mail.
If your goal is to secretly destroy email this is the way.
This would result in half the planet being frustrated all the time and the other half never getting their mail.
If your goal is to secretly destroy email this is the way.
This makes sense for service emails and is similar to how push notification services like Pushbullet work, but can't work for humans. You need to be able to give your email to someone IRL so they can send you a message. Mutual approval would be possible in the "we just met and want to exchange emails" situation, but that too breaks when you legitimately want to give anyone the chance to message you.
> but that too breaks when you legitimately want to give anyone the chance to message you.
Yup. I do want people to be able to contact me regarding my homepage. It's only a niche page on a niche subject, so only a handful of people has written in, but it was nice hearing from them and some of them did make quite a few valuable contributions.
Some minimal obfuscation seems to be enough to keep mail harvesters away, and beyond that those mails go through the same spam filter as all my other mail traffic. Putting up a contact form would definitively be more of a hassle than just a simple mailto:-link, and then I would additionally have to start worrying about how to keep the bots away from that contact form.
Yup. I do want people to be able to contact me regarding my homepage. It's only a niche page on a niche subject, so only a handful of people has written in, but it was nice hearing from them and some of them did make quite a few valuable contributions.
Some minimal obfuscation seems to be enough to keep mail harvesters away, and beyond that those mails go through the same spam filter as all my other mail traffic. Putting up a contact form would definitively be more of a hassle than just a simple mailto:-link, and then I would additionally have to start worrying about how to keep the bots away from that contact form.
I know, but anything out of band won't really work, because that can be phished even more, plus as described above, there's no real need for it either (IMHO).
That's essentially how www.hey.com works! I thought it would be tedious at first, but I don't mind it, and it's done a great job of making it so I only see what I want to see in my inbox.
From a quick read, hey.com still allows arbitrary people to message you and just initially puts them in "The Screener", which doesn't seem like quite the same thing as an absolutely "completely whitelist (opt-in to receive messages) basis".
That's fine by me because
a) I do want to give out some sort of contact info on my homepage and people being able to message me in relation to that (and putting up a contact form leads to its own spam problems), and
b) if you happen to swap contact details offline, you then have to remember that you still need to additionally whitelist that person inside of that message service, which also seems somewhat of a hassle.
Somebody who gets inundated in unwanted messages might have a different opinion on that subject, though, and might indeed prefer a strict opt-in mode, with no exceptions…
That's fine by me because
a) I do want to give out some sort of contact info on my homepage and people being able to message me in relation to that (and putting up a contact form leads to its own spam problems), and
b) if you happen to swap contact details offline, you then have to remember that you still need to additionally whitelist that person inside of that message service, which also seems somewhat of a hassle.
Somebody who gets inundated in unwanted messages might have a different opinion on that subject, though, and might indeed prefer a strict opt-in mode, with no exceptions…
Spam has been a thing forever. Literally forever. Yet people have been dealing with spam far better than the big boys, and doing so for decades.
Want to talk about anti-competitive? Gmail will accept mails, provide a 250 SMTP response, then drop the email internally.
That's not right. At all. You can reject the email easily during SMTP exchange, and people have been doing that literally for 20+ years.
No valid excuse here. None. Zero.
And if your 250 OK accept then drop the message, and provide no way to notify, or discuss, or find out why, you make it impossible for a remote admin to fix the problem.
This is 100% on purpose. Yahoo, outlook/hotmail, gmail, collude to resolve issues like this, while blocking all others to resolve issues their own purposefully broken policies cause.
If you see Alphabet with a policy, or action, you can be 100% sure it is aligned to increase market dominance.
Want to talk about anti-competitive? Gmail will accept mails, provide a 250 SMTP response, then drop the email internally.
That's not right. At all. You can reject the email easily during SMTP exchange, and people have been doing that literally for 20+ years.
No valid excuse here. None. Zero.
And if your 250 OK accept then drop the message, and provide no way to notify, or discuss, or find out why, you make it impossible for a remote admin to fix the problem.
This is 100% on purpose. Yahoo, outlook/hotmail, gmail, collude to resolve issues like this, while blocking all others to resolve issues their own purposefully broken policies cause.
If you see Alphabet with a policy, or action, you can be 100% sure it is aligned to increase market dominance.
Agreed. It's the increasing difficulty in getting things delivered to the big providers making it harder rather than spam.
At this rate, eventually we'll have a handful of mail senders (Mailchimp, Sendgrid), and a handful of mail receivers (Google, MS).
At this rate, eventually we'll have a handful of mail senders (Mailchimp, Sendgrid), and a handful of mail receivers (Google, MS).
I'd imagine they do this to not tip off spammers when a message goes through. It's the same idea behind returning a 404 when trying to access a resource you don't have permission to, or not telling users if an account actually exists under an email when doing a password reset.
It's simplistic for a spammer to tell if mail is getting delivered to end point. EG, just have a gmail account, and spam that account. Simple. Done.
Breaking reliable mail delivery for everyone, is inline with "there's no excuse, ever". It's inline with "making it worse", not better.
If you 250 accept, you deliver the email. Worst case, it ends up in a spam folder. You do not drop it on the floor. Ever. No excuse, no reason is valid here.
And I certainly won't accept "But it's so hard!", considering how easy it is to handle email, including SPAM, for everyone... until Google purposefully breaks it.
Breaking reliable mail delivery for everyone, is inline with "there's no excuse, ever". It's inline with "making it worse", not better.
If you 250 accept, you deliver the email. Worst case, it ends up in a spam folder. You do not drop it on the floor. Ever. No excuse, no reason is valid here.
And I certainly won't accept "But it's so hard!", considering how easy it is to handle email, including SPAM, for everyone... until Google purposefully breaks it.
[deleted]
Incorrect.
> It's definitely an anticompetitive practice.
Or maybe it's an overly competitive protocol? Like playing Monopoly. Even a neophyte can end sweeping the game despite not understanding any of the underlying mechanics that drive the game's outcome. Those remaining mail providers are also fighting back the insanity. They 'just' won.
Don't hate the player, hate the game.
Or maybe it's an overly competitive protocol? Like playing Monopoly. Even a neophyte can end sweeping the game despite not understanding any of the underlying mechanics that drive the game's outcome. Those remaining mail providers are also fighting back the insanity. They 'just' won.
Don't hate the player, hate the game.
A normal company could engage in these practices, but it is settled law that a monopolist may not. A monopolist may not use their monopoly in one are to supress activities in another area.
You can't put these big tech monopolists into the same bucket as normal companies. They have way too much power, and even when they do not intend to smash things, they end up smashing things.
You can't put these big tech monopolists into the same bucket as normal companies. They have way too much power, and even when they do not intend to smash things, they end up smashing things.
I wonder if we can’t build a far simpler postoffice app on http (self hosted domain space) and have a whitelisted encrypted exchange.
Substitute “HTTP” with “SMTP” and you already have that today… SMTP in plain text isn’t generally used anymore, it’s always transported over TLS. Of course you can run whitelist-only, but how is that ever going to work?
SMTP is text based and we’ll defined, so I would also argue that the transport being “simpler” is nonsense.
SMTP is text based and we’ll defined, so I would also argue that the transport being “simpler” is nonsense.
Whitelist is easier. If I added an mail of a friend in this system, we can communicate. Everything else goes blackhole.
Nothing is easy about email! Having worked for years just to get reliable in and outboxes is definitely not trivial. Also SMTP is a system out of your control if you want anything verified and actually delivered.
Nothing is easy about email! Having worked for years just to get reliable in and outboxes is definitely not trivial. Also SMTP is a system out of your control if you want anything verified and actually delivered.
Of course whitelist is easier, but how valuable would your email be if you only allowed your friends to email you? Goodbye order confirmations, subscription reminders, recruiter emails, notifications…
Sounds like a dream, really. And you could always have it be user-driven action, like "Copy this line and paste it into your email provider's whitelist", framing it as a positive win for the user since they have total control over who is allowed to communicate with them.
Seems like you could easily do this yourself with a couple of rules in your email client.
I do want arbitrary people to reach out to me regarding my home page, so any just-allow-whitelisted-messages proposals are useless in that regard.
And in practice some minimal address obfuscation has been enough to block any address harvesters, and setting up a contact form or something comparable (and I guess especially also subsequently keeping it spam-free!) would have been much more of a hassle.
And in practice some minimal address obfuscation has been enough to block any address harvesters, and setting up a contact form or something comparable (and I guess especially also subsequently keeping it spam-free!) would have been much more of a hassle.
Yeah, I feel to that is a much better system. You could have a “grey”list postbox that uses a one time code for that company.
Say you enter a form at a company,and in the message box you just leave that code so they can mail you.
Say you enter a form at a company,and in the message box you just leave that code so they can mail you.
[deleted]
Of course we can, but good luck getting people to use it. They'll keep using what everyone else uses.
Yeah, I've been doing exactly this for over 20 years. The only problem I can recall is related to the fact that the hosting provider uses a single SSL cert for the machine that hosts my domain (and many others, presumably), so of course the cert doesn't match my domain name. It's pretty easy to work around, and I only have to deal with it every few years when they do a hardware upgrade, which sometimes means moving my domain to a different machine.
Certs for MX servers are supposed to have the MX as subject or SAN, not your email domain. It's important when the sender enforces encryption with a valid cert (e.g. MTA-STS, or config in the mail server, or many hosted solutions like Google Workspace also support enforcing this for selected or all domains).
Example:
example.com. MX aspmx.l.google.com.
Cert should have aspmx.l.google.com as subject or SAN.
Example:
example.com. MX aspmx.l.google.com.
Cert should have aspmx.l.google.com as subject or SAN.
I agree with this assessment, and it's what I do. There is still the single point of failure of losing your domain due to a hostile registrar or mismanagement e.g. allowing registration to lapse.
Ideally there would be some a decentralized permanent domain registry keyed on certs (I know these exist but have not been adopted), or at least a fallback domain you could configure somehow in case you lost control of your main domain.
Ideally there would be some a decentralized permanent domain registry keyed on certs (I know these exist but have not been adopted), or at least a fallback domain you could configure somehow in case you lost control of your main domain.
> It's just not something I want to babysit anymore because I have other things to focus on
Dont know about you, but I have setup my mailserver years ago, and outside of regular OS updates, havent had to touch it.
Dont know about you, but I have setup my mailserver years ago, and outside of regular OS updates, havent had to touch it.
Where is it hosted? Isn’t the primary issue being blocked by the major providers due to spam filters?
I have used Mailinabox on a Hetzner server for about an year. My email delivers to all the major providers. However, small providers will occasionally block my email. So I continue to use my Gmail address for now.
With small amounts of evidence, I think if my contacts on those providers email me first, and I reply to those emails, then my domain is not blocked.
With small amounts of evidence, I think if my contacts on those providers email me first, and I reply to those emails, then my domain is not blocked.
server4you.
there is one blocklist im aware of that simply categorically blocks all their IPs, but thankfully none of the "big tech" players use it, so its really of no consequence to me
there is one blocklist im aware of that simply categorically blocks all their IPs, but thankfully none of the "big tech" players use it, so its really of no consequence to me
How do you make sure your emails don't end up in spam?
For me, the issue wasn't that I had to fix it frequently, but that when I did it was urgent, stressful and disruptive. Eventually the VDS I was renting had a fatal HD crash and I gave up.
this would apply to any self hosted stuff you rely on, but yeah, self hosting is not for everyone
Agreed. Salient.
> simply own your domain name and point the MX record to whatever hosting provider you want
That's not necessarily a sure cure, depending on the hosting provider. RoadRunner (Spectrum / Charter) in the US and Shaw in Canada won't deliver emails from my domain hosted at Runbox.com (or sent directly from the runbox.com domain.) Spectrum's bounce message references an error code that translates to "Spectrum limits the number of concurrent connections from a sender, as well as the total number of connections allowed. Limits vary based on the reputation of the IP address. Reduce your number of connections and try again later."
That's not necessarily a sure cure, depending on the hosting provider. RoadRunner (Spectrum / Charter) in the US and Shaw in Canada won't deliver emails from my domain hosted at Runbox.com (or sent directly from the runbox.com domain.) Spectrum's bounce message references an error code that translates to "Spectrum limits the number of concurrent connections from a sender, as well as the total number of connections allowed. Limits vary based on the reputation of the IP address. Reduce your number of connections and try again later."
Some of us use Wireguard or OpenVPN to host where we want and use a static cheap or free (like Cloudflare) public point of presence IP.
> point the MX record to whatever hosting provider you want
You can even have a hybrid solution where incoming mail goes directly to your self-hosted server and (some) outgoing mail is relayed through a third party.
You can even have a hybrid solution where incoming mail goes directly to your self-hosted server and (some) outgoing mail is relayed through a third party.
What would be the advantage?
There are many benefits to running your own server. The three biggies for me are:
1. Control. A third party can change anything about the service any time they want, and if you don't like the change they made you're screwed.
2. Expectation of privacy. Because I am not contracting with a third party, the government cannot argue that I have waived my right to privacy. (As a practical matter of course this matters not at all. If the government -- or anyone with the right technical skill and access -- wants to read your email they will. But if push ever comes to shove in a court of law it could matter.)
3. Spam filtering. I think the whole industry is doing it wrong. The Right Way to filter spam is to use your outgoing mail as ground truth for what is not spam. I have a custom spam filter that I wrote based on this idea and it works like a charm. No Bayesian analysis needed. I don't even look at content at all. Just the headers are enough to achieve >99% accuracy.
1. Control. A third party can change anything about the service any time they want, and if you don't like the change they made you're screwed.
2. Expectation of privacy. Because I am not contracting with a third party, the government cannot argue that I have waived my right to privacy. (As a practical matter of course this matters not at all. If the government -- or anyone with the right technical skill and access -- wants to read your email they will. But if push ever comes to shove in a court of law it could matter.)
3. Spam filtering. I think the whole industry is doing it wrong. The Right Way to filter spam is to use your outgoing mail as ground truth for what is not spam. I have a custom spam filter that I wrote based on this idea and it works like a charm. No Bayesian analysis needed. I don't even look at content at all. Just the headers are enough to achieve >99% accuracy.
> The Right Way to filter spam is to use your outgoing mail as ground truth for what is not spam
Could you elaborate? Does this mean that email from people/domains you haven't corresponded with before is spam?
Could you elaborate? Does this mean that email from people/domains you haven't corresponded with before is spam?
Anything that comes in from an address I have never seen before is handled specially. But it's not hard to filter out the obvious spam. Just a handful of heuristics on the from and subject lines (e.g. if the sender's name contains common English words it's probably spam) takes care of >90% of the cold calls. The rest I just look through manually once a day or so.
I was planning to institute a system where my contact page included a special keyword to include in the subject line to get past the spam filter, but that has turned out not to be necessary so I haven't implemented that yet.
The only remaining case is things like confirmation emails for new accounts, but those just get lumped in with the other cold calls. They are super-easy to spot because I'm almost always expecting them, so they are always at the top of the list.
I was planning to institute a system where my contact page included a special keyword to include in the subject line to get past the spam filter, but that has turned out not to be necessary so I haven't implemented that yet.
The only remaining case is things like confirmation emails for new accounts, but those just get lumped in with the other cold calls. They are super-easy to spot because I'm almost always expecting them, so they are always at the top of the list.
Benefiting from the large provider's reputation in regards to spam blocklists etc.
"The sweet spot for having control over your email while simultaneously minimizing unforseen headaches is to simply own your domain name"
You would think. The issue no one seems to think about is that you need to make sure to pay for the domain for the duration of your life(at least). Otherwise, as soon as you lose your domain you lose ownership of your email. Any one that has control of the domain has control of your e-mail.
This dawned on me after a place I worked at reactivated the email I used for work when I worked there. It has my name but I have 0 control over it. Lucky for me I never used the email for things other than work so it's not a big deal. It is still bothersome that they can do that and I have no say so on its use.
You would think. The issue no one seems to think about is that you need to make sure to pay for the domain for the duration of your life(at least). Otherwise, as soon as you lose your domain you lose ownership of your email. Any one that has control of the domain has control of your e-mail.
This dawned on me after a place I worked at reactivated the email I used for work when I worked there. It has my name but I have 0 control over it. Lucky for me I never used the email for things other than work so it's not a big deal. It is still bothersome that they can do that and I have no say so on its use.
> The issue no one seems to think about is that you need to make sure to pay for the domain for the duration of your life
But that's true for any delivery endpoint in any medium, including physical mail and phone numbers.
It's pretty easy to set up auto-pay for domain names so that all you really need to do is keep the billing info up to date. After that it all runs on autopilot.
But that's true for any delivery endpoint in any medium, including physical mail and phone numbers.
It's pretty easy to set up auto-pay for domain names so that all you really need to do is keep the billing info up to date. After that it all runs on autopilot.
Unless your registry messes up and forgets to renew your domain. It’s happened to me before (don’t use them anymore though).
name and shame? that's a pretty drastic mistake in terms of the consequences it must have had for you.
> The issue no one seems to think about is that you need to make sure to pay for the domain for the duration of your life
Is there a registrar that will allow you to do this? Most of the ones I’ve looked at have an upper limit of around 15 years or so
Is there a registrar that will allow you to do this? Most of the ones I’ve looked at have an upper limit of around 15 years or so
The maximum bound for most gTLD registries is ten years (have a dig around the ICANN website if you'd like to check), not fifteen. Sure, there might be a ccTLD that allows you to register a domain for longer than that, but I'm not aware of any, and ccTLDs usually have lower maximum renewal/registration bounds than the gTLDs.
If a registrar is letting you renew a gTLD for 15 years, they're basically sitting on a third of your money until the domain's expiration date comes up.
If a registrar is letting you renew a gTLD for 15 years, they're basically sitting on a third of your money until the domain's expiration date comes up.
You don't have to do it all in one shot but you should make sure it gets paid so you don't lose access.
[deleted]
While this is true I don't see it as a huge deal. Treat it like any other annual bill (e.g Insurance, property taxes) and you will be fine.
You can always use a different email address to manage the domain if that's the issue?
I would imagine the issue is someone else buying the domain, and subsequently receiving any email meant for you (such as a password reset request).
This is what I do, though I do think helm is also pretty cool: https://thehelm.com/products/helm-personal-server-v2?variant...
I like the idea of having a node I can just plug into my network. I run my Urbit on a Mac mini with tailscale (which works great).
The core of what he writes about is correct though, email failed to be truly peer to peer (as imo all non-urbit-like federated systems will) because of the incentives that lead to centralization (spam, difficulty of running nodes, etc.)
We’re suffering the consequences of the local max we’re trapped in currently because of this. The promise of the 90s internet was a bunch of people using decentralized services they controlled - instead we're primarily thin clients connecting to a small handful of powerful ad companies. We're mostly serfs [0] allowed access if we give up our data for ad targeting, follow EULAs nobody reads, and don't say anything the company earls disagrees with.
[0]: https://zalberico.com/essay/2020/07/14/the-serfs-of-facebook...
I like the idea of having a node I can just plug into my network. I run my Urbit on a Mac mini with tailscale (which works great).
The core of what he writes about is correct though, email failed to be truly peer to peer (as imo all non-urbit-like federated systems will) because of the incentives that lead to centralization (spam, difficulty of running nodes, etc.)
We’re suffering the consequences of the local max we’re trapped in currently because of this. The promise of the 90s internet was a bunch of people using decentralized services they controlled - instead we're primarily thin clients connecting to a small handful of powerful ad companies. We're mostly serfs [0] allowed access if we give up our data for ad targeting, follow EULAs nobody reads, and don't say anything the company earls disagrees with.
[0]: https://zalberico.com/essay/2020/07/14/the-serfs-of-facebook...
Controlling your domain/mx is the most valuable thing.
Email reception has not been a problem for me so I enjoy having my mx pointing to my own mail server. It gives me more control and it requires very little maintenance.
If I was going to outsource anything the first choice would be outbound. My email system does everything right for reputation protection. All senders are authenticated on secure connections and the senders are people I trust. Nothing bad gets sent and my static IP is on a reputable server host and is not on any public black lists. I maintain SPF, DKIM etc. If someone decides to block my IP for no reason by accident or on purpose there is very little I can do or care to do anymore.
I have an alternate path for emails setup via a server I host elsewhere ready to go. If I run into a widespread delivery issue due to massive indiscriminate ip blacklisting of my provider I can enable it. That has happened once in 20 years. If delivery gets too hard I will change that policy to send all outgoing emails through a commercial smtp delivery service and let them deal with the problems.
Email reception has not been a problem for me so I enjoy having my mx pointing to my own mail server. It gives me more control and it requires very little maintenance.
If I was going to outsource anything the first choice would be outbound. My email system does everything right for reputation protection. All senders are authenticated on secure connections and the senders are people I trust. Nothing bad gets sent and my static IP is on a reputable server host and is not on any public black lists. I maintain SPF, DKIM etc. If someone decides to block my IP for no reason by accident or on purpose there is very little I can do or care to do anymore.
I have an alternate path for emails setup via a server I host elsewhere ready to go. If I run into a widespread delivery issue due to massive indiscriminate ip blacklisting of my provider I can enable it. That has happened once in 20 years. If delivery gets too hard I will change that policy to send all outgoing emails through a commercial smtp delivery service and let them deal with the problems.
It isn’t about what’s convenient for you. It’s an individual cost benefit analysis. Your needs are different than mine.
This is what I do. The downside is that a lot of email providers make it really difficult to set up 3rd party clients outside of their own clients. I've struggled to get mutt to work with a lot of email providers because they use their own auth mechanisms.
I'm pretty happy using protonmail bridge with local clients. It does take some effort to setup though.
I am ashamed to admit that I have no idea how email works. Is there a dumb down explanation of what are the moving parts and how you can achieve that sweet spot?
You send your email through a client. That client then sends (transfers / SMTPs) that email through an MTA either bundled with it or provided by your mail server.
The MTA parses the message, figures out who it needs to go to (To, CC, BCC headers), figures out what servers receives mail for those recipients, and then transfers (SMTPs) it to the server.
What OP is referring to is that the MTA essentially does a DNS lookup for the recipients domain for a record of type MX (Mail eXchanger).
If you own the domain you have complete control over where that mail goes: you own the MX record.
The MTA parses the message, figures out who it needs to go to (To, CC, BCC headers), figures out what servers receives mail for those recipients, and then transfers (SMTPs) it to the server.
What OP is referring to is that the MTA essentially does a DNS lookup for the recipients domain for a record of type MX (Mail eXchanger).
If you own the domain you have complete control over where that mail goes: you own the MX record.
There is no such thing as a BCC header. That's the entire point of the BCC.
What you've described might be correct in some cases but is not universal. The mainstream way that messages are sent is your "email client" or MUA speaks the ESTMP protocol to a mail transfer or submission agent (MTA or MSA). The client directly specifies the envelope recipients, which are not, generally speaking, parsed out of the formatted message. That is why it is possible for me to send a message to myself but the message is subsequently delivered to hundreds of unnamed recipients.
What you've described might be correct in some cases but is not universal. The mainstream way that messages are sent is your "email client" or MUA speaks the ESTMP protocol to a mail transfer or submission agent (MTA or MSA). The client directly specifies the envelope recipients, which are not, generally speaking, parsed out of the formatted message. That is why it is possible for me to send a message to myself but the message is subsequently delivered to hundreds of unnamed recipients.
> The MTA parses the message...
That may happen, but not for the purpose you stated. MTA just uses addresses you told it to use in the SMTP dialog. It doesn't use addresses from the message. In fact the addresses in the message may be totally different.
That may happen, but not for the purpose you stated. MTA just uses addresses you told it to use in the SMTP dialog. It doesn't use addresses from the message. In fact the addresses in the message may be totally different.
Email is complicated but the jist is there are relays and mailbox/mail-exchanger servers (and email clients). And a million different anti-spam measures that make making sure people actually get you email difficult.
When you send an email you mail client contacts you mail-exchanger (MX) and drops the mail in your outbox. Then the MX will look up the MX record for the domains in the TO field and attempt to send the email to it using SMTP.
The first thing receiving server ussally does and look up the IP of the sending server and see if it's on a spam black list. If it is it will probably just drop the connection. Then it will look up the SPF record for the domain in the FROM address and see if the sending server is allowed to send that mail.
Larger email services will have an internal 'reputation' scoring system that will use data from reported spam to figure out what IP's and domains are sending spam emails and filter them out. They'll also look at if it's a residential IP, in an IP pool for a major cloud provider etc. Each provider has a different system and it can be really difficult to get a provider to trust your IP or whitelist your IP so you can make sure that your mail actually gets to who you're trying to send it to.
Relays are pretty simple they'll take email from one place and send it to the recipient. The mail exchanger usually has a built-in relay. A lot of people will use a third party relay service so they don't have to worry about managing the reputation of the IP of their sending mail server. They'll just add an SPF record for the relays service to their domain. And then configure their mail exchanger to send all outbound mail to the relay and then the relay will do the MX lookup.
A lot of mail services will also have their MX records pointed at relays. These inbound relays will often have a lot of those anti-spam services bolted onto them and they can also be used for load balancing to make sure that the service is always able to accept mail even if it doesn't make it in the mailbox immediately.
When you send an email you mail client contacts you mail-exchanger (MX) and drops the mail in your outbox. Then the MX will look up the MX record for the domains in the TO field and attempt to send the email to it using SMTP.
The first thing receiving server ussally does and look up the IP of the sending server and see if it's on a spam black list. If it is it will probably just drop the connection. Then it will look up the SPF record for the domain in the FROM address and see if the sending server is allowed to send that mail.
Larger email services will have an internal 'reputation' scoring system that will use data from reported spam to figure out what IP's and domains are sending spam emails and filter them out. They'll also look at if it's a residential IP, in an IP pool for a major cloud provider etc. Each provider has a different system and it can be really difficult to get a provider to trust your IP or whitelist your IP so you can make sure that your mail actually gets to who you're trying to send it to.
Relays are pretty simple they'll take email from one place and send it to the recipient. The mail exchanger usually has a built-in relay. A lot of people will use a third party relay service so they don't have to worry about managing the reputation of the IP of their sending mail server. They'll just add an SPF record for the relays service to their domain. And then configure their mail exchanger to send all outbound mail to the relay and then the relay will do the MX lookup.
A lot of mail services will also have their MX records pointed at relays. These inbound relays will often have a lot of those anti-spam services bolted onto them and they can also be used for load balancing to make sure that the service is always able to accept mail even if it doesn't make it in the mailbox immediately.
Typically, if you sign up for an email account, you get an email address like [email protected] or [email protected]. Alternatively, if you own/host skywal.com, you can have an email address like [email protected] served from a computer in your home.
The "sweet spot" is combining the two, where you own skywal.com, and have your email send/receive through Google or whoever. Then, if Google decides to ban you, you just register skywal.com with another company who provides that same service, and you keep your same email address.
That's the broad strokes anyway.
The "sweet spot" is combining the two, where you own skywal.com, and have your email send/receive through Google or whoever. Then, if Google decides to ban you, you just register skywal.com with another company who provides that same service, and you keep your same email address.
That's the broad strokes anyway.
I can type out an explanation relatively easily.
Let's imagine I'm sending from [email protected] to [email protected]. I'm doing it from say Thunderbird or Outlook, and you're using the same.
I need to send, and to do this I typically use an SMTP server. This is something configured, probably on zahllos.example, maybe with the domain smtp.zahllos.example. My mail client contacts this and 'logs in' with my details, then transmits the email message I want to send to this server. The server says 'right fine' and closes the connection.
At this point, the message is in a mail queue ready to go. The SMTP server then does a DNS request for the MX record of skywall.example. Let's say this is 'mail.skywall.example'. My server, smtp.zahllos.example, then connects to `mail.skywall.example', also speaking SMTP, and says "hey, I have this message to deliver".
It is at this point that mail.skywall.example can decide to do some things. It might check SPF, so it will query the SPF record of zahllos.example to find the list of servers that may send email for that domain. In this example, let's assume smtp.zahllos.example is in the list. Great.
It may then also check the mail headers for a signature, called a dkim signature. My server signed the message before sending; mail.skywall.example can query <uid>._domainkey.zahllos.example and find a public key (or not) and check that this signature matches (or not). Again, let's assume it matches.
It might also check something like TXT _dmarc.zahllos.example to see what my DMARC policies are. If I have something like v=DMARC1;p=reject;sp=reject;pct=100;ri=86400;fo=1;aspf=s;adkim=s;rua=mailto:[email protected];ruf=mailto:[email protected];" this tells you I'd like you to outright reject anything not matching policy, that I expect everything to match SPF and have DNS signatures, and you can send reports if you support that to [email protected]. Your server can then enforce these checks as it likes.
One of the first things that will happen is that my server will announce itself via an EHLO statement. An obvious check to do is to check that the sending IP actually matches smtp.zahllos.example. by querying 'reverse ptr' records.
Your receiving server will also likely hand the message over to various spam-checking tools for analysis, such as against DNS blocklists and so on. Larger providers likely have much more sophisticated infrastructure here. Ultimately, you're going to do one of a few things: 1) deliver to inbox or apply user-specified rules and deliver to a folder; 2) deliver to junk (which is typically just another folder, but treated specially by clients), 3) reject, and tell smtp.zahllos.example you don't want the email.
Once the email passes through the smtp dameon, assuming either 1 or 2, it then gets stored somehow and in some way. I'm being a little bit vague here, because 'it depends', but in the simplest scenario, the smtp daemon will write the message to an mbox or maildir-style format. More complex setups definitely exist, indeed, there can be multiple layers of servers doing analysis on separate machines, but for simplicity, mail.skywall.example is one VM that makes its decisions and the result ends up in /var/mail/user@domain/ or some such.
A key aspect of this step that makes email very nice is that if smtp.zahllos.example cannot, for some reason, reach your server now it will queue the message and try again at set intervals. You can reasonably safely turn off mail.skywall.example for a couple of hours.
Another aspect is that you can have multiple MX records where you are prepared to accept email, with priorities. So if you can't accept at one address because the server is down for maintenance, another will accept.
So, now you've technically got an email, but you don't know it. So you open thunderoutlook, and you connect to an IMAP server imap.skywall.example - in our example let us assume this is really the same thing as mail.skywall.example. The server checks you are really you with your credentials. At the backend, this is just another daemon that knows to read /var/mail/... and find new messages; it finds one, downloads its headers and displays it on your screen.
Since we're in a slightly more modern world now, in the case your client was already open you might have an "idle" connection with the server at all times, in which case it can push the message down to you.
In the case of webmail, it is really all the same thing, except you point your browser at a webpage, and that webpage communicates with the servers instead of your client communicating directly. Open source webmail might even use IMAP underneath; things like Zimbra use their own java mail agent, while Google is entirely custom.
That might seem complicated, but in the end it isn't: between two domains at the 'edge', in the end, there's an SMTP conversation. One sending server tells a receiving server it has mail for delivery, and it finds that server by asking DNS where it is. The receiving server may do a bunch of checks against DNS also before making a decision on what to do with the email.
Let's imagine I'm sending from [email protected] to [email protected]. I'm doing it from say Thunderbird or Outlook, and you're using the same.
I need to send, and to do this I typically use an SMTP server. This is something configured, probably on zahllos.example, maybe with the domain smtp.zahllos.example. My mail client contacts this and 'logs in' with my details, then transmits the email message I want to send to this server. The server says 'right fine' and closes the connection.
At this point, the message is in a mail queue ready to go. The SMTP server then does a DNS request for the MX record of skywall.example. Let's say this is 'mail.skywall.example'. My server, smtp.zahllos.example, then connects to `mail.skywall.example', also speaking SMTP, and says "hey, I have this message to deliver".
It is at this point that mail.skywall.example can decide to do some things. It might check SPF, so it will query the SPF record of zahllos.example to find the list of servers that may send email for that domain. In this example, let's assume smtp.zahllos.example is in the list. Great.
It may then also check the mail headers for a signature, called a dkim signature. My server signed the message before sending; mail.skywall.example can query <uid>._domainkey.zahllos.example and find a public key (or not) and check that this signature matches (or not). Again, let's assume it matches.
It might also check something like TXT _dmarc.zahllos.example to see what my DMARC policies are. If I have something like v=DMARC1;p=reject;sp=reject;pct=100;ri=86400;fo=1;aspf=s;adkim=s;rua=mailto:[email protected];ruf=mailto:[email protected];" this tells you I'd like you to outright reject anything not matching policy, that I expect everything to match SPF and have DNS signatures, and you can send reports if you support that to [email protected]. Your server can then enforce these checks as it likes.
One of the first things that will happen is that my server will announce itself via an EHLO statement. An obvious check to do is to check that the sending IP actually matches smtp.zahllos.example. by querying 'reverse ptr' records.
Your receiving server will also likely hand the message over to various spam-checking tools for analysis, such as against DNS blocklists and so on. Larger providers likely have much more sophisticated infrastructure here. Ultimately, you're going to do one of a few things: 1) deliver to inbox or apply user-specified rules and deliver to a folder; 2) deliver to junk (which is typically just another folder, but treated specially by clients), 3) reject, and tell smtp.zahllos.example you don't want the email.
Once the email passes through the smtp dameon, assuming either 1 or 2, it then gets stored somehow and in some way. I'm being a little bit vague here, because 'it depends', but in the simplest scenario, the smtp daemon will write the message to an mbox or maildir-style format. More complex setups definitely exist, indeed, there can be multiple layers of servers doing analysis on separate machines, but for simplicity, mail.skywall.example is one VM that makes its decisions and the result ends up in /var/mail/user@domain/ or some such.
A key aspect of this step that makes email very nice is that if smtp.zahllos.example cannot, for some reason, reach your server now it will queue the message and try again at set intervals. You can reasonably safely turn off mail.skywall.example for a couple of hours.
Another aspect is that you can have multiple MX records where you are prepared to accept email, with priorities. So if you can't accept at one address because the server is down for maintenance, another will accept.
So, now you've technically got an email, but you don't know it. So you open thunderoutlook, and you connect to an IMAP server imap.skywall.example - in our example let us assume this is really the same thing as mail.skywall.example. The server checks you are really you with your credentials. At the backend, this is just another daemon that knows to read /var/mail/... and find new messages; it finds one, downloads its headers and displays it on your screen.
Since we're in a slightly more modern world now, in the case your client was already open you might have an "idle" connection with the server at all times, in which case it can push the message down to you.
In the case of webmail, it is really all the same thing, except you point your browser at a webpage, and that webpage communicates with the servers instead of your client communicating directly. Open source webmail might even use IMAP underneath; things like Zimbra use their own java mail agent, while Google is entirely custom.
That might seem complicated, but in the end it isn't: between two domains at the 'edge', in the end, there's an SMTP conversation. One sending server tells a receiving server it has mail for delivery, and it finds that server by asking DNS where it is. The receiving server may do a bunch of checks against DNS also before making a decision on what to do with the email.
So in the example of the parent, he's got a domain name registered somewhere (mydomain.com) and he set, in its MX records, the gmail server. But how does gmail make the connection between that address and your gmail address then?
So normally in the cases like this, you also have to tell Google about this, and you typically do this by using one of their paid-for products like workspaces or apps for business or whatever they call it. So let's say that you decide to host [email protected] with Google. You pay them for workspaces and you likely tell them "I would like to use this domain I already have, with email". They then tell you "OK, add our servers as your MX record in your DNS, or transfer the whole domain to us and we'll do it for you". In this case we're doing the 'changing the MX records' part.
Now when a sending server asks "where should I deliver skywall.example email" by querying MX skywall.example it gets google's servers and starts an smtp conversation with them, saying "I'd like to deliver a message for [email protected]". At this point, Google knows it can accept that, so they say yes, and then continue doing whatever they do to check for spam beyond that, including queries for spf and friends.
The reason the parent suggests this is that if at any point you decide to move off Google, you can pay someone else, e.g. fastmail, for their services, and modify your MX record. 24-48 hours later, DNS around the world catches up and everyone will get fastmail as a response when they ask for your MX record. Any new email goes there instead of Google, and thus you aren't 'tied' to the provider: you just have to move all your old email over. Whereas Google cannot let you move a [email protected] address, because they can only change the MX records for the whole of gmail.
DNS is the source of truth here. Whatever your MX records are is where other servers will try to contact to send email. The MX record is typically just another DNS address that will be queried for AAAA/A (i.e. what is the IP), and that doesn't need to be on the same domain at all.
Here's an example of what it looks like:
I'll cover off the SMTP outgoing as well while I'm at it. You _can_ also not run your own outgoing smtp server but use someone else's. The key here is that if you use SPF and DKIM, you should put their IPs into SPF and their keys into DKIM, as that is what the receiving server will use. So smtp.zahllos.example could be replaced by sendgrid, provided in my DNS I say so. This may work better, as sendgrid may have a better reputation than the server I chose.
Now when a sending server asks "where should I deliver skywall.example email" by querying MX skywall.example it gets google's servers and starts an smtp conversation with them, saying "I'd like to deliver a message for [email protected]". At this point, Google knows it can accept that, so they say yes, and then continue doing whatever they do to check for spam beyond that, including queries for spf and friends.
The reason the parent suggests this is that if at any point you decide to move off Google, you can pay someone else, e.g. fastmail, for their services, and modify your MX record. 24-48 hours later, DNS around the world catches up and everyone will get fastmail as a response when they ask for your MX record. Any new email goes there instead of Google, and thus you aren't 'tied' to the provider: you just have to move all your old email over. Whereas Google cannot let you move a [email protected] address, because they can only change the MX records for the whole of gmail.
DNS is the source of truth here. Whatever your MX records are is where other servers will try to contact to send email. The MX record is typically just another DNS address that will be queried for AAAA/A (i.e. what is the IP), and that doesn't need to be on the same domain at all.
Here's an example of what it looks like:
delv MX ycombinator.com @9.9.9.9
; unsigned answer
ycombinator.com. 295 IN MX 20 alt2.aspmx.l.google.com.
ycombinator.com. 295 IN MX 10 aspmx.l.google.com.
ycombinator.com. 295 IN MX 20 alt1.aspmx.l.google.com.
ycombinator.com. 295 IN MX 30 aspmx4.googlemail.com.
delv NS ycombinator.com @9.9.9.9
ycombinator.com. 159148 IN NS ns-225.awsdns-28.com.
ycombinator.com. 159148 IN NS ns-1914.awsdns-47.co.uk.
ycombinator.com. 159148 IN NS ns-1411.awsdns-48.org.
ycombinator.com. 159148 IN NS ns-556.awsdns-05.net.
I'll cover off the SMTP outgoing as well while I'm at it. You _can_ also not run your own outgoing smtp server but use someone else's. The key here is that if you use SPF and DKIM, you should put their IPs into SPF and their keys into DKIM, as that is what the receiving server will use. So smtp.zahllos.example could be replaced by sendgrid, provided in my DNS I say so. This may work better, as sendgrid may have a better reputation than the server I chose.
I learnt about delv for the first time reading your comment and struggled [1] to get the commands you used here to work on my Mac.
Turns out that I have to treat Apple-included binaries of OSS utilities with suspicion and instead use the version maintained by the Homebrew guys.
1: If anyone else is interested, I wrote a short article about it https://ayewo.com/how-to-get-delv-working-on-macos/
Turns out that I have to treat Apple-included binaries of OSS utilities with suspicion and instead use the version maintained by the Homebrew guys.
1: If anyone else is interested, I wrote a short article about it https://ayewo.com/how-to-get-delv-working-on-macos/
delv is the successor to dig, and both are tools that come as part of BIND the DNS server for making queries and debugging DNS. Delv happens to understand DNSSEC a bit better, so if you ever need to debug that, it is handy to have.
In this case dig or delv would be fine. You can also use https://mxtoolbox.com/SuperTool.aspx and pick "MX lookup" to find the MX records. It will also resolve the IPs, both IPv4 and IPv6 (A and AAAA) for you for the returned records.
Their supertool also breaks down SPF records, for example, into their meaning and tells you if they're valid or not. I think other links were posted that do similar things but I haven't had a chance to try them yet.
I've mentioned reverse pointers in various cases, delv can do those lookups too. Here is how you do it: first you find the IP of say mail-ed1-x529.google.com, which happens to be a mailserver of Google that was the last step before my mail server when sending myself a message from gmail. For a bit of a change, this is an IPv6 only server, and it has IP 2a00:1450:4864:20::529. So let's look it up with -x:
One of, but my no means the only, reasons you can't send email from a home connection is because you announce yourself as smtp.zahllos.example but your reverse dns resolves to something like <somegeneratedhostname>.dynamic.residential.isp.com. You can find out yours: type "what is my ip" into google, copy-paste that and do a delv -x on it and see what your "hostname" is according to the internet.
In this case dig or delv would be fine. You can also use https://mxtoolbox.com/SuperTool.aspx and pick "MX lookup" to find the MX records. It will also resolve the IPs, both IPv4 and IPv6 (A and AAAA) for you for the returned records.
Their supertool also breaks down SPF records, for example, into their meaning and tells you if they're valid or not. I think other links were posted that do similar things but I haven't had a chance to try them yet.
I've mentioned reverse pointers in various cases, delv can do those lookups too. Here is how you do it: first you find the IP of say mail-ed1-x529.google.com, which happens to be a mailserver of Google that was the last step before my mail server when sending myself a message from gmail. For a bit of a change, this is an IPv6 only server, and it has IP 2a00:1450:4864:20::529. So let's look it up with -x:
delv -x 2a00:1450:4864:20::529
; unsigned answer
9.2.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.ip6.arpa. 43200 IN PTR mail-ed1-x529.google.com.
One of, but my no means the only, reasons you can't send email from a home connection is because you announce yourself as smtp.zahllos.example but your reverse dns resolves to something like <somegeneratedhostname>.dynamic.residential.isp.com. You can find out yours: type "what is my ip" into google, copy-paste that and do a delv -x on it and see what your "hostname" is according to the internet.
> The sweet spot for having control over your email while simultaneously minimizing unforseen headaches is to simply own your domain name and point the MX record to whatever hosting provider you want instead of self-hosting a server at home.
So, mandatory third-party doctrine/warrantless surveillance.
So, mandatory third-party doctrine/warrantless surveillance.
It’s a misnomer to say you can “own” a domain name. You cannot. The better word would be “rent”. Because of that you should think about what could happen if your domain name registration lapses and someone else registers it.
I think everyone knows this who owns a domain. Probably people say this because (like I do too) they have all the control over the name. If I rent an apartment, I do have to consult the owner to make major changes.
I think people who get their email on their own domain generally consider it as their permanent home and don’t think about what will happen when the domain no longer is registered to them. Using the apartment rental analogy works up to a point. With an apartment you can get the postal service to forward your mail to a new address. With a domain name there is no way to get this to happen. Whoever holds the registration controls the MX record and can do whatever they want.
In case of death it's indeed an issue if someone isn't planning ahead.
With grace periods of the domain expiry process it's quite safe. People have plenty of time to recover it. It's also hard not to notice if your mail stops arriving.
With grace periods of the domain expiry process it's quite safe. People have plenty of time to recover it. It's also hard not to notice if your mail stops arriving.
How does this get me unlimited accounts for my shady biz? Every provider I know of wants money to make new ones, those bastards.
> The sweet spot for having
no forth amendment protection for your email because its stored by a third party.
no forth amendment protection for your email because its stored by a third party.
Most emails are still going through servers owned by Microsoft or Google, so what does self-hosting email accomplish in reality? Some government entity likely has warrant-less access to most of your emails regardless. I like email as a way to communicate, but speaking pragmatically, it’s just not a secure means of communication in 2022.
My server uses encrypted connections and my own certificate. If I send an email to a family member on my server, no one on the Internet can read its contents. If I add a recipient and FAANG or Microsoft then yes it wouldn't matter. If that is truly important, we can learn to encrypt emails locally first.
What does self-hosting accomplish? I'd say it's important in the way that having HAM radio operators is important.
What does self-hosting accomplish? I'd say it's important in the way that having HAM radio operators is important.
what does self-hosting email accomplish in reality
Well, for one thing it prevents Google from building a profile of my online shopping behavior based on my email receipts. Also avoids them knowing every time I go on a trip based on booking confirmation emails.
Like most people, I'm not trying to hide from the government.
Well, for one thing it prevents Google from building a profile of my online shopping behavior based on my email receipts. Also avoids them knowing every time I go on a trip based on booking confirmation emails.
Like most people, I'm not trying to hide from the government.
Using a 3rd party email host that isn't Google has the same effect with a lot less effort. I switched to FastMail for this reason, even if Google claims not to scan paid Gmail accounts for marketing purposes.
I support the author but let me tell you a counterargument I don’t think he devotes enough to:
Spam is a real issue.
The amount of spam emails which get sent are absurd and likely orders of magnitude more than non-spam. And spammers do a lot to mimic real emails, including just hacking legitimate addresses and adding them to botnets.
Even on gmail, I still get spam sent to my inbox. Fortunately very rarely, but it still happens.
And even if it isn’t bad today, spam has the potential to be much worse in the future with transformer networks and hostile state actors.
And even if it really isn’t that bad and never will be, the big companies and those arguing against self-hosting will claim it is. They don’t want to allow a relative few self-hosted email servers in exchange for much more difficult and less effective spam detection. Forget Gmail and Outlook, why not just use Fastmail or Protonmail?
If you want a legitimate argument for self-hosted emails you need to address the spam. It may be as simple as registering your official email with some organization sponsored by open-source, and all the big companies can trust that one organization. Then the org has to deal with spam registrations but maybe there won’t be much and it will work out. idk much about self-hosting so this org might already exist.
But this article doesn’t mention that org, in fact doesn’t say much at all about spam besides “keep existing spam-prevention because it already works”. But you should at least explain why. Because spam is a legitimate argument for big-co forming an oligarchy that’s not just “so they can make more money”, and it’s the main argument that big-co uses.
Spam is a real issue.
The amount of spam emails which get sent are absurd and likely orders of magnitude more than non-spam. And spammers do a lot to mimic real emails, including just hacking legitimate addresses and adding them to botnets.
Even on gmail, I still get spam sent to my inbox. Fortunately very rarely, but it still happens.
And even if it isn’t bad today, spam has the potential to be much worse in the future with transformer networks and hostile state actors.
And even if it really isn’t that bad and never will be, the big companies and those arguing against self-hosting will claim it is. They don’t want to allow a relative few self-hosted email servers in exchange for much more difficult and less effective spam detection. Forget Gmail and Outlook, why not just use Fastmail or Protonmail?
If you want a legitimate argument for self-hosted emails you need to address the spam. It may be as simple as registering your official email with some organization sponsored by open-source, and all the big companies can trust that one organization. Then the org has to deal with spam registrations but maybe there won’t be much and it will work out. idk much about self-hosting so this org might already exist.
But this article doesn’t mention that org, in fact doesn’t say much at all about spam besides “keep existing spam-prevention because it already works”. But you should at least explain why. Because spam is a legitimate argument for big-co forming an oligarchy that’s not just “so they can make more money”, and it’s the main argument that big-co uses.
email spam is not a real issue. big uncooperative gigacorps fucking it up for everyone is.
responsibility for spam (and other kinds of abuse) can be delegated via simple reputation scoring for netblocks, sender authentication and a proper feedback mechanism along the chain.
when the user clicks "spam" gmail already uses that to train their fancy AI and if you are not a small nobody, then they already alert you that ooops you sent something spammy via a feedback loop [1]. (see also how Mailchimp proudly claims they work with big integration partners like gmail ... https://mailchimp.com/help/how-mailchimp-prevents-and-handle... )
whitelist clearinghouses exist [2] but they are not terribly useful, because there's most of the signal to use for reputation is hidden :/
[1] https://en.wikipedia.org/wiki/Feedback_loop_(email) [2] https://www.dnswl.org/
responsibility for spam (and other kinds of abuse) can be delegated via simple reputation scoring for netblocks, sender authentication and a proper feedback mechanism along the chain.
when the user clicks "spam" gmail already uses that to train their fancy AI and if you are not a small nobody, then they already alert you that ooops you sent something spammy via a feedback loop [1]. (see also how Mailchimp proudly claims they work with big integration partners like gmail ... https://mailchimp.com/help/how-mailchimp-prevents-and-handle... )
whitelist clearinghouses exist [2] but they are not terribly useful, because there's most of the signal to use for reputation is hidden :/
[1] https://en.wikipedia.org/wiki/Feedback_loop_(email) [2] https://www.dnswl.org/
> responsibility for spam (and other kinds of abuse) can be delegated via simple reputation scoring for netblocks, sender authentication and a proper feedback mechanism along the chain.
Are there any real-world examples of this? I know other decentralized networks like Tor and BitTorrent have some sort of reputation and feedback system. What type of "spam" do they deal with and how well do they deal with it? Are there any systems more similar to mail with mechanisms to prevent spam?
Spam is a serious issue, not just in email, not just in decentralized systems - it's one of the main issues in technology today. If there's a solution, even a proof-of-concept in a smaller system would be great
Are there any real-world examples of this? I know other decentralized networks like Tor and BitTorrent have some sort of reputation and feedback system. What type of "spam" do they deal with and how well do they deal with it? Are there any systems more similar to mail with mechanisms to prevent spam?
Spam is a serious issue, not just in email, not just in decentralized systems - it's one of the main issues in technology today. If there's a solution, even a proof-of-concept in a smaller system would be great
As a practitioner, I can tell you that this proposal would be quite a bit behind state of the art for email spam filter accuracy. For example, there's a surprising amount of legitimate mail that gets sent without authentication, and which you don't want to block because your end users will get mad if it goes missing.
I may also point out that "reputation scoring for netblocks" is the exact problem the original blog post was complaining about. He was trying to send from residential ISP and VPS netblocks that had poor reputation, and saw delivery problems as a result.
I may also point out that "reputation scoring for netblocks" is the exact problem the original blog post was complaining about. He was trying to send from residential ISP and VPS netblocks that had poor reputation, and saw delivery problems as a result.
if netblock reputation were known small non-spam senders would congregate, this would help weed out the spammers.
but google et al. don't provide any feedback. nor any way to build reputation for your IP. (where are all the staking blockfoo chainbar solutions? or at least let people pay a one time fee for some exception)
anyway, since we already have certificate transparency, we could have a similar one to look up MTA/domain repu provider, then sending DKIM signed HTTP calls to the reputation report address, and let people aggregate it.
but google et al. don't provide any feedback. nor any way to build reputation for your IP. (where are all the staking blockfoo chainbar solutions? or at least let people pay a one time fee for some exception)
anyway, since we already have certificate transparency, we could have a similar one to look up MTA/domain repu provider, then sending DKIM signed HTTP calls to the reputation report address, and let people aggregate it.
> "... small non-spam senders would congregate"
Congratulations, you've invented the email service provider. Most small non-spam senders don't want to spend the time figuring out IP reputation, DKIM, or SPF already, so they pay someone to do it for them. What makes netblock reputation different?
Reputation isn't public because then spammers game it and you get worse filtering outcomes. ISPs learned this the hard way.
Reputation isn't public because then spammers game it and you get worse filtering outcomes. ISPs learned this the hard way.
sure, there is no problem with having 95+ % or more handled by professional industrial companies
the problem is that they literally don't even allow others at the table, only way is to force yourself a seat by having enough traffic
the problem is that they literally don't even allow others at the table, only way is to force yourself a seat by having enough traffic
Is it getting harder to self-host? Yes. Impossible? No. Most senders just find it easier to pay for it.
Everything is getting harder as tech gets more and more complex. (As IT matures.) That's okay. The problem is that the incentives suck and there's too much business in email. The big ones have built their moats, and there's not much to do.
"SMTP" was simply never made to be resistant to this. (Nor the web in general.)
"SMTP" was simply never made to be resistant to this. (Nor the web in general.)
Very well-put.
> And even if it isn’t bad today, spam has the potential to be much worse in the future with transformer networks and hostile state actors.
> Even on gmail, I still get spam sent to my inbox. Fortunately very rarely, but it still happens.
It is as good as it is exactly because of the requirements author finds tedious.
> It may be as simple as registering your official email with some organization sponsored by open-source, and all the big companies can trust that one organization. Then the org has to deal with spam registrations but maybe there won’t be much and it will work out. idk much about self-hosting so this org might already exist.
They do exist but there are a bunch of problems with that. People have varying definitions of spam, getting paid to whitelist someone creates a perverse incentive or not getting paid will quickly overwhelm the organisation.
Things could be better if we could enforce sender authentication (SPF/DKIM). It would assign a direct cost to getting your domain blacklisted. But if nobody is taking away rest of the spammer's domains (or keeps selling them new ones), they'll continue.
> And even if it isn’t bad today, spam has the potential to be much worse in the future with transformer networks and hostile state actors.
> Even on gmail, I still get spam sent to my inbox. Fortunately very rarely, but it still happens.
It is as good as it is exactly because of the requirements author finds tedious.
> It may be as simple as registering your official email with some organization sponsored by open-source, and all the big companies can trust that one organization. Then the org has to deal with spam registrations but maybe there won’t be much and it will work out. idk much about self-hosting so this org might already exist.
They do exist but there are a bunch of problems with that. People have varying definitions of spam, getting paid to whitelist someone creates a perverse incentive or not getting paid will quickly overwhelm the organisation.
Things could be better if we could enforce sender authentication (SPF/DKIM). It would assign a direct cost to getting your domain blacklisted. But if nobody is taking away rest of the spammer's domains (or keeps selling them new ones), they'll continue.
Spam is an issue, but it's not the one that impacts me the most. I rarely get true 'spam' in my inbox.
What impacts me the most is a never-ending problem with Gmail classifying messages as spam that were actually important to me. Time sensitive announcements of meetings or events, for example. Many are coming from senders I've been receiving email from for years.
Gmail is convinced that almost every technical-related email mailing list I'm on is a spam source, despite my constantly going into my spam folder and telling it dozens of messages from those mailing lists are not actually spam.
Meanwhile, what I do get is a barrage of promotional emails - what I consider very much to be spam - from corporations that at some point have had my email address and now email me multiple times a week. Those sail right past the spam filter into my "promotions" folder and accumulate...
What impacts me the most is a never-ending problem with Gmail classifying messages as spam that were actually important to me. Time sensitive announcements of meetings or events, for example. Many are coming from senders I've been receiving email from for years.
Gmail is convinced that almost every technical-related email mailing list I'm on is a spam source, despite my constantly going into my spam folder and telling it dozens of messages from those mailing lists are not actually spam.
Meanwhile, what I do get is a barrage of promotional emails - what I consider very much to be spam - from corporations that at some point have had my email address and now email me multiple times a week. Those sail right past the spam filter into my "promotions" folder and accumulate...
For those who might choose to run their own email these days, you don't have to postfix + dovecot:
https://github.com/foxcpp/maddy
https://blitiri.com.ar/p/chasquid/
These options are much easier to set up, will do things like generate DKIM for you, etc.
I talk about this a lot[0]. There are positively awesome tools for email out there.
[EDIT] - Since I'm repeating myself I've collected all the options into a post[1] I can just link to.
[0]: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
[1]: https://vadosware.io/post/its-never-been-easier-or-harder-to...
https://github.com/foxcpp/maddy
https://blitiri.com.ar/p/chasquid/
These options are much easier to set up, will do things like generate DKIM for you, etc.
I talk about this a lot[0]. There are positively awesome tools for email out there.
[EDIT] - Since I'm repeating myself I've collected all the options into a post[1] I can just link to.
[0]: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
[1]: https://vadosware.io/post/its-never-been-easier-or-harder-to...
I recommend people do it themselves or not at all, these are my tools: postfix/dovecot/mysql/opendkim/opendmarc.
I agree with you, but I think that stack you're using is actively harmful to newcomers at this point.
Something like Maddy is much easier to set up, comes with workable SMTP + IMAP, saves to object storage.
If you pair maddy with Thunderbird/Outlook/Mail/whatever else, I think we can actually get new people self-hosting without getting discouraged.
[EDIT] I misunderstood the earlier comment as a recommendation -- it's just a statement of fact.
Something like Maddy is much easier to set up, comes with workable SMTP + IMAP, saves to object storage.
If you pair maddy with Thunderbird/Outlook/Mail/whatever else, I think we can actually get new people self-hosting without getting discouraged.
[EDIT] I misunderstood the earlier comment as a recommendation -- it's just a statement of fact.
Is Maddy's IMAP+object storage setup mature now? Last time I tried it I was testing against minio and trying to get my 25gb inbox synced over. The transfer was in b/s, would've taken forever. I love the project but haven't checked in on it for over a year now.
I use IMAP+on-disk SQLite, and make sure to take backups, I haven't run it purely off of object storage yet so can't comment on whether it's better with such a large inbox!
I use maddy + dovecot. maddy's IMAP implementation is good for small stuff (and trivial to set up!), but still performs very slow for few-GB inboxes or larger.
Add to this list spamassassin with amavis and clamav.
Thanks for the post!
o/t: Could you add a close button to that ethical ads element? It's obscuring a decent chunk of the screen on mobile and I can't figure out how to dismiss it
o/t: Could you add a close button to that ethical ads element? It's obscuring a decent chunk of the screen on mobile and I can't figure out how to dismiss it
Hey apologies, I just enabled EthicalAds this week, I'll look into this and add something... I'll also forward your concern to them.
Thank you very much!
just for visibility -- it should be fixed, added a media query just now.
Am I correct that neither of those support SIEVE filters and ManageSieve protocol?
IE something like: https://blog.tinned-software.net/setup-sieve-mail-filter-wit...
Ed: Does anyone have experience with https://modoboa.org/ ?
IE something like: https://blog.tinned-software.net/setup-sieve-mail-filter-wit...
Ed: Does anyone have experience with https://modoboa.org/ ?
Good point but maddy's README says "IMAP storage is beta" and AFAICT some of the other suggestions don't support SPF+DKIM out of the box.
The ecosystem can definitely use more work, but I've found maddy to be really good, because it's all in one, makes very reasonable choices, and is easy to configure.
I put maddy first because it's what I personally use (without dovecot) -- I make sure to back it up as well, and that's enough for me/most people, I think. It's the closest to the whole package which doesn't rely on.
Oh you know what, I completely forgot about iRedMail:
https://www.iredmail.org/
I haven't run this, but it's another great option when combined with SoGo.
I put maddy first because it's what I personally use (without dovecot) -- I make sure to back it up as well, and that's enough for me/most people, I think. It's the closest to the whole package which doesn't rely on.
Oh you know what, I completely forgot about iRedMail:
https://www.iredmail.org/
I haven't run this, but it's another great option when combined with SoGo.
Mailinabox is much simpler https://mailinabox.email/
I'm a happy user of mailcow:
https://mailcow.email/
https://mailcow.email/
It was a huge mistake for email receivers to take on the cost of filtering spam. Of course given the evolution of the internet and email it is easy to see how that mistake happened. Nobody had a crystal ball. But the only solution here is to raise the cost of sending email to the point where spam is no longer profitable.
It seems like one solution is to bcrypt hash (or some similarly expensive algorithm) the email and include the hash in a header. Of course you need to hash per receiver or a spammer can just hash it once and spam away.
The receiving client hashes the email and compares the result with the value in the header and discards emails that don't match.
You'll never get industry buy in though - the FAANG companies don't want to pay that cost for their semi-legitimate email. They prefer to keep that cost externalized.
I believe there have been attempts at something like this, but it clearly never went anywhere.
It seems like one solution is to bcrypt hash (or some similarly expensive algorithm) the email and include the hash in a header. Of course you need to hash per receiver or a spammer can just hash it once and spam away.
The receiving client hashes the email and compares the result with the value in the header and discards emails that don't match.
You'll never get industry buy in though - the FAANG companies don't want to pay that cost for their semi-legitimate email. They prefer to keep that cost externalized.
I believe there have been attempts at something like this, but it clearly never went anywhere.
This indeed looks like a good direction. A decade ago Freenet (not sure if it still exists?) had a problem with spam on its equivalent of USENET. It was pretty bad, until they changed the protocol so that it's the sending node that keeps the message, which is then pulled (or not) by the recipients. It made a lot of sense to me: I'm the one sending you the message, I want you to see it, while you don't even know whether you want to get that message. So it's me that should pay for storage and propagation of the message in terms of bandwidth and disk space. Not sure how it turned out in the end, but the approach seems right to me. It's like phone calls: it's the caller that pays the cost, not the one receiving the call.
Wow that's ingenious - and obvious when you think about it. I guess evolving the email standard is a much bigger obstacle though, no matter how clever the proposal.
An attempt was made.
https://en.wikipedia.org/wiki/Internet_Mail_2000
https://en.wikipedia.org/wiki/Internet_Mail_2000
it's ridiculously easy to create a template that is stored and when the remote client pulls the message only a few variables has to be substituted. it's how spam is generated after all :)
Yes, but in the context of Freenet specifically, there's a lot of encryption and hashing involved in every transaction, which are computationally expensive, and there's no CGI equivalent at all (the whole network is basically static, append-only content addressed with the content's hash). So to create many messages you'd need to first generate the content, then encrypt and hash it, then insert it into the network starting at your node, then provide signature and checksum verification on every request. After some time, if the content is widely requested, some other nodes will cache it, but you can't really expect spam to propagate via web of trust; it's more probable that it will be added to spam filters which, themselves, will propagate. As everything is signed, you'd need to create a fresh node for each spam, but that means your new identity is not trusted by anyone - you'd need to convince at least some users that you're a user that won't abuse the system. Only to get your whole identity blacklisted once you do abuse it.
It wasn't perfect, and Freenet's usage was a huge PITA from the usability/ux perspective, but in a constrained environment like that this change in the protocol had much bigger impact than it would on the open web. Not sure how it turned out, but the trend until I left the community was very positive with regards to spam.
It wasn't perfect, and Freenet's usage was a huge PITA from the usability/ux perspective, but in a constrained environment like that this change in the protocol had much bigger impact than it would on the open web. Not sure how it turned out, but the trend until I left the community was very positive with regards to spam.
Something different: a hash which is expensive to calculate but cheap to verify. E.g. calculating a string of bytes to append to a hash stream on order to produce a hash with a certain number of leading zeros; you provide the hash and the bytes, and it's trivial to verify.
There were proposals for this sort of thing a while back, but they never caught on:
https://en.wikipedia.org/wiki/Hashcash
https://en.wikipedia.org/wiki/Hashcash
That's almost exactly my off the cuff suggestion. Of course my brain is seeded by both bitcoin (finding a hash as proof of work) and HLL (which counts leading hash digits to estimate set cardinality).
Like cryptocurrency this will be a moving target. You would need a difficulty setting - but where as cryptocurrency only has to track one thing "total hash rate" this would need to track two things "minimum supported hardware for legit sender" and "hardware threshold for attacker to be successful with a mining rig they can afford, based on the income from the spam".
Each email provider might come up with different difficulty levels based on what they thing this is. So some handshaking might be required. And less computer literate people would be stressed why their email is taking 6 minutes to send. I think it would be hard to implement.
Each email provider might come up with different difficulty levels based on what they thing this is. So some handshaking might be required. And less computer literate people would be stressed why their email is taking 6 minutes to send. I think it would be hard to implement.
Not good from an energy-wasting perspective.
Oh I know. The point is something hard that the sender needs to do. Ideally it wouldn't waste lots of power, but it must consume some scarce resource, or else it wouldn't cost anything, and the cost should be borne by the sender, so e.g. a RAM-intensive hash function wouldn't be right.
That is a clever idea but I think it'll still fail so long as email (SMTP) is a fire-and-forget architecture. As long as you have that asymmetry, your SNR is going to suck.
If it were a back-and-forth protocol, more like TCP, then you have way more options for congestion control, error reporting, load balancing, and the like. The server can choose to accept the incoming request, ask for more verification, or interrogate the client in various ways. This could be something just like DKIM / DMARC / SPF, or even something more exotic, like making the client do proof-of-work with difficulty tied to how suspicious that client is to the server, and also the delivery scope/scale. Or forcing the client to wait for ACK for valid delivery while slow-walking it.
This gets around some of the issues in cousin comments, with respect to punishing botnets and rewarding lawful players. Established, high-trust players pay no cost. Suspicious players can still get through, albeit with a tax (that should be trivial for low-volume personal MX, but expensive for high-volume spam). Furthermore, it's adaptable.
If it were a back-and-forth protocol, more like TCP, then you have way more options for congestion control, error reporting, load balancing, and the like. The server can choose to accept the incoming request, ask for more verification, or interrogate the client in various ways. This could be something just like DKIM / DMARC / SPF, or even something more exotic, like making the client do proof-of-work with difficulty tied to how suspicious that client is to the server, and also the delivery scope/scale. Or forcing the client to wait for ACK for valid delivery while slow-walking it.
This gets around some of the issues in cousin comments, with respect to punishing botnets and rewarding lawful players. Established, high-trust players pay no cost. Suspicious players can still get through, albeit with a tax (that should be trivial for low-volume personal MX, but expensive for high-volume spam). Furthermore, it's adaptable.
> If it were a back-and-forth protocol, more like TCP, then you have way more options for congestion control, error reporting, load balancing, and the like. The server can choose to accept the incoming request, ask for more verification, or interrogate the client in various ways.
That's basically what graylisting aims to achieve.
That's basically what graylisting aims to achieve.
Yeah, this is essentially a form of greylisting. The difference is (as I understand it, this is fairly outside my domain), with the current setup, MTAs can accept an email, and it ends up getting blackhole'd or spam-folder'd anyways. My hypothetical scheme would put more onus on the first "boundary node" to report on errors/compliance. Basically the MTA tells the client what hoops to jump through, and the client gets some indication what will happen once those conditions are met.
That could be an exchange like: "Sign this nonce, and your message will be vetted", or "this is very suss, you have to do X difficulty hashes to have any chance of delivery, and regardless it'll be flagged as potential spam". Or perhaps just a guarantee on how an action would affects the message's "spam score".
This could be used alongside nested packets/envelopes and various headers/trust levels in a network of trust to give a message some overall trust level.
That could be an exchange like: "Sign this nonce, and your message will be vetted", or "this is very suss, you have to do X difficulty hashes to have any chance of delivery, and regardless it'll be flagged as potential spam". Or perhaps just a guarantee on how an action would affects the message's "spam score".
This could be used alongside nested packets/envelopes and various headers/trust levels in a network of trust to give a message some overall trust level.
The problem to adding a cost to email is that it affects everyone. The amount of CPU power you need to waste to make most spam not viable is so much that it isn't worth it.
Most spam is sent by hijacked machines and botnets is it not? They don't care about wasting CPU power; they aren't paying for it.
It definitely does not - you can allow different work loads for different senders. Mailing lists you actually want can be dropped to zero for instance.
Most spam comes from new address pairs, not existing ones. Requiring high cost to get past a first-contact filter, then near zero forever after, is completely reasonable and would practically eliminate unsolicited spam.
Most spam comes from new address pairs, not existing ones. Requiring high cost to get past a first-contact filter, then near zero forever after, is completely reasonable and would practically eliminate unsolicited spam.
But now the sender needs to know the receivers policy and if they remember that there has been contact before. Or I guess you change SMTP but we still allow unencrypted connections so good luck with that.
For newsletter style stuff, nah. The "confirm your registration" email can "pay" to get past the wall, and then you're done - approved pair established, future letters can probably be zero cost and everyone receives the same one.
I wholly admit that this is arguing theoretical setups and that's always problematic, but of course patterns would be established pretty quickly. There are loads of simple tactics that would still make spam dramatically harder, and legitimate use nearly unaffected. The current reputation system has clear, massive gaps that really don't need to exist.
I wholly admit that this is arguing theoretical setups and that's always problematic, but of course patterns would be established pretty quickly. There are loads of simple tactics that would still make spam dramatically harder, and legitimate use nearly unaffected. The current reputation system has clear, massive gaps that really don't need to exist.
yeah, I believe it is called "HashCash" and works similarly to "proof of work" in cryptocurrencies
HashCash is actually referenced as part of the inspiration for Bitcoin by Satoshi themself. It's the birthplace of proof of work.
Right, I can´t believe people are re-discovering HashCash. It was a brilliant idea way ahead of its time. Sadly it was not adopted for email.
Isn't this DKIM bh= ?
The address [email protected], which must be used by Australian permanent residents to update their personal details, refuses to accept email messages unless they are from big tech.
Shame on you, Australian Department of Home Affairs.
And shame on Telstra, which provides the service.
---
Remote-MTA: dns; dibp-ibmail2.msng.telstra.com.au
Diagnostic-Code: smtp; 554-mx.msng.telstra.com.au 554 Your access to this mail system has been rejected due to the sending MTA's poor reputation. If you believe that this failure is in error, please contact the intended recipient via alternate means.
Shame on you, Australian Department of Home Affairs.
And shame on Telstra, which provides the service.
---
Remote-MTA: dns; dibp-ibmail2.msng.telstra.com.au
Diagnostic-Code: smtp; 554-mx.msng.telstra.com.au 554 Your access to this mail system has been rejected due to the sending MTA's poor reputation. If you believe that this failure is in error, please contact the intended recipient via alternate means.
> The address [email protected], which must be used by Australian permanent residents to update their personal details
You update personal details with the government via email? That's interesting. I would have assumed they would use at least a secure web form.
You update personal details with the government via email? That's interesting. I would have assumed they would use at least a secure web form.
929 is the form. The email address is for completed copies of the form.
Permanent residents are a class of non citizens who have the right to permanently reside in Australia. Most cannot vote.
All of it can be done online through their portal. The Australian government spies on everyone anyway so using the portal presents no increased attack surface for anyone who would care.
Really it's there as a legacy technology. Most people under 35 would certainly use a portal. Probably most of them over 35 as well at this point.
Permanent residents are a class of non citizens who have the right to permanently reside in Australia. Most cannot vote.
All of it can be done online through their portal. The Australian government spies on everyone anyway so using the portal presents no increased attack surface for anyone who would care.
Really it's there as a legacy technology. Most people under 35 would certainly use a portal. Probably most of them over 35 as well at this point.
If you used an immigration lawyer to apply for permanent residency, you can't use the portal to make those updates, because the original visa is not associated to your portal account.
So your only other option besides emailing the form, is to print it and bring it in person to an office.
I know this, because this was my situation until recently.
A few months ago I changed my PR subclass and the new application was done using the portal. Which means that I can finally use the portal to update my details.
So your only other option besides emailing the form, is to print it and bring it in person to an office.
I know this, because this was my situation until recently.
A few months ago I changed my PR subclass and the new application was done using the portal. Which means that I can finally use the portal to update my details.
[deleted]
[deleted]
That might be something worth talking to a journalist about.
Government department accepts mail from the providers that basically everyone uses, details at 0300.
E-mail is complicated, sure. But I’ve had it up to here with people who give up running their own server and then go on to vastly exaggerate how infeasible it is, in order to placate their own conscience. It’s not that they’ve gotten tired of doing it, oh no; (they say,) it’s entirely the fault of Google, Microsoft, etc. who’ve made it literally impossible to run your own e-mail server. Except it’s not impossible – lots of us do it, still. And now there’s one fewer of us, so the rest of us have to work that much harder when the next monopolizing standard comes along (BIMI, anyone?). Sure, you don’t owe us anything, but thanks for nothing when making these public rants; you are scaring away people who might still be inclined to help!
I’m genuinely interested. Where would you recommend a newcomer starts?
There's alot that goes into a mail server stack, but it's no more complicated than k8s or other stacks these days. My preferred setup is rspamd/postfix/dovecot/roundcube. The docs are good and the mailing lists are active & archived for easy searching
For a pre-packaged mailserver environment, take a look at mailcow or mailinabox
https://mailcow.email
https://mailinabox.email/
There's a variety of ansible/chef/puppet up on github that can also be used to setup the invididual component
For a pre-packaged mailserver environment, take a look at mailcow or mailinabox
https://mailcow.email
https://mailinabox.email/
There's a variety of ansible/chef/puppet up on github that can also be used to setup the invididual component
> it's no more complicated than k8s
That's not really saying much.
That's not really saying much.
I've found that Mailinabox provides a pretty much turn-key setup. (https://www.mailinabox.email)
I've been self hosting my own email on a Dell Poweredge for just over a year now, if you want my smtpd, dovecot, and rspamd confs I'll share them :x
I haven’t followed any such guide, so I wouldn’t know, but there are usually lots of links in all these threads to such things.
How is BIMI "monopolizing"? It's a trivial DNS record.
There's the catch - VMCs are not a mandatory part of BIMI. Though if you want to establish trust, someone has to be willing to put their name on the line and verify everything required. If you have a better approach in mind, I'm sure a lot of people would love to know.
Trust (by way of VMC) is the whole point of BIMI; it’s what’s in all the marketing copy: The fact that nobody else can send mail with your logo. If you merely wanted to send e-mail with your icon on it, that already existed: the X-Face header has been around for decades.
Absolutely not, BIMI is a way to provide brand identity for recognition/marketing purposes. Trust or security is not the standard's goal. Being able to trust the logo is an optional extra.
Also nobody really uses X-Face, it's irrelevant.
Also nobody really uses X-Face, it's irrelevant.
You’re talking nonsense. What is the improvement of BIMI (without VMC) over X-Face? The only things I can think of are:
1. BIMI logos are in color.
oh yeah, and:
2. BIMI logos are inherently a tracking pixel so the sender can see when readers read the mail.
1. BIMI logos are in color.
oh yeah, and:
2. BIMI logos are inherently a tracking pixel so the sender can see when readers read the mail.
You are free to read the BIMI standard's section 2.1 containing its high-level goals.
The next section also literally says "This document does not cover the different verification and reputation mechanisms available, but BIMI relies upon them to be in deployed in order to control abuse." It's not a standard meant for establishing trust, it does not mandate requiring a VMC.
The next section also literally says "This document does not cover the different verification and reputation mechanisms available, but BIMI relies upon them to be in deployed in order to control abuse." It's not a standard meant for establishing trust, it does not mandate requiring a VMC.
Just like with e-mail, it doesn’t matter what’s in the standard, what matters is what the big providers actually do. If Google (Gmail), Microsoft, etc. will simply show any BIMI logo without VMC verification (which will never happen), then I will concede that BIMI is not a monopolizing standard. It’ll just be a tracking pixel.
Showing any BIMI logo is an absolutely unreasonable thing to demand from a large-scale BIMI implementation. It does not make it "monopolizing", I don't think you even know what the word means.
So VMC isn’t optional?
Is it that difficult to grasp that there's a "depends" option between "optional" and "not optional"?
Nobody really forces you to use HTTPS either, it's not a "monopolizing" standard if someone doesn't trust you without.
And again, if you have a way of establishing just as much trust without such a labour-intensive/expensive verification process, please do share.
Nobody really forces you to use HTTPS either, it's not a "monopolizing" standard if someone doesn't trust you without.
And again, if you have a way of establishing just as much trust without such a labour-intensive/expensive verification process, please do share.
We still are fighting the oligopoly with our vaulted c1.fi email service. Please feel free to check us at https://c1.fi/v/hn20220905/?lang=en.
Here's EU's JRC-MECSA report on our service: https://mecsa.jrc.ec.europa.eu/en/finderRequest/b5daceffc76e....
Support for client's own domain is currently under works. Our webmail supports PGP and one can use IMAPS/SMTPS or ActiveSync based native email clients too.
All servers self hosted (we run C1/Gentoo!) in our own computing facility in Finland. =)
Here's EU's JRC-MECSA report on our service: https://mecsa.jrc.ec.europa.eu/en/finderRequest/b5daceffc76e....
Support for client's own domain is currently under works. Our webmail supports PGP and one can use IMAPS/SMTPS or ActiveSync based native email clients too.
All servers self hosted (we run C1/Gentoo!) in our own computing facility in Finland. =)
Love this site design. Looks so good on mobile.
My fear is that something similar will slowly happen to everything "compute". How long before my bank's website won't let me login if I don't use a computer with secure boot and a browser installed from an app store? McDonalds app on Android won't run on a de-googled or rooted device... At this point one may argue that there will always be computers that you can compile and install your own Linux. Yes, that is true. But just like I am not likely to have un-googled andorid for some apps and googled one for others, the same way it won't be practical to have one computer for some apps and the other one for others. And the one that will win will be the one that lets you login into your bank account, for simple and practical reasons.
I agree with the pains, but the options are not juts Big Tech or self-hosting. There's a myriad of not-big-tech email providers out there, for example there's Posteo, who use open source software and green energy. They are going strong for 13 years now with 400+k accounts.
https://en.wikipedia.org/wiki/Posteo
https://en.wikipedia.org/wiki/Posteo
Sending email out is a royal pain. Trying to deal with a Microsoft ban on my IP even though it’s sparkling clean for several years. DKIM, DMARC, SPF etc all ser up, reverse dns, you name it. Looks like Linode is being blocked as a whole pretty much?
Hate the level of centralization, particularly since there’s still a shit ton of spam still around. Sorry for the rant.
https://docs.microsoft.com/en-us/answers/questions/674558/55...
Hate the level of centralization, particularly since there’s still a shit ton of spam still around. Sorry for the rant.
https://docs.microsoft.com/en-us/answers/questions/674558/55...
I've been running a private mailserver for myself and a couple of friends since 2002. I hit a similar issue with Microsoft last year.
I route outgoing mail to hotmail.com, live.com, outlook.com and msn.com domains via email-smtp.us-east-2.amazonaws.com. I started doing this when I noticed a large portion of e-commerce email I was receiving originated from amazon vms and they clearly had no deliverability issues.
The ability to do this is a paid-for service, and needs to be configured appropriately. I was expecting to have to pay for a VM when I set out to do this. However, it turns out you do not need that - you can just configure the SMTP forwarding without needing to purchase any other products, and the cost for this is a few cents per 10k emails, which I can see would be significant for commercial services and/or spammers, but for a private mail server essentially means I will never have to pay. Amazon go to great pains to maintain deliverability for those servers, so you don't have to.
I don't really see a problem with adding one extra SMTP hop to outgoing emails tbh, but as no-one other than Microsoft has (to date) a permanent IP block ban with no recourse whatsoever policy, while I've had transient issues with most other big providers at one time or another, those four Microsoft domains remain the only destination I have to route this way.
I route outgoing mail to hotmail.com, live.com, outlook.com and msn.com domains via email-smtp.us-east-2.amazonaws.com. I started doing this when I noticed a large portion of e-commerce email I was receiving originated from amazon vms and they clearly had no deliverability issues.
The ability to do this is a paid-for service, and needs to be configured appropriately. I was expecting to have to pay for a VM when I set out to do this. However, it turns out you do not need that - you can just configure the SMTP forwarding without needing to purchase any other products, and the cost for this is a few cents per 10k emails, which I can see would be significant for commercial services and/or spammers, but for a private mail server essentially means I will never have to pay. Amazon go to great pains to maintain deliverability for those servers, so you don't have to.
I don't really see a problem with adding one extra SMTP hop to outgoing emails tbh, but as no-one other than Microsoft has (to date) a permanent IP block ban with no recourse whatsoever policy, while I've had transient issues with most other big providers at one time or another, those four Microsoft domains remain the only destination I have to route this way.
office 365 hosts a bunch of domains though. My bounce was to a .berlin domain hosted there. Yes, I could route all smtp via a ESP and pay for it, but that’s centralization.
Interesting - whatever permaban it is that my IP block got didn't affect office 365 hosted services (verified with several different domains at the time) - just literally the domains I mention. Guess I've lucked out so far.
Wouldn't surprise me if Linode were entirely blocked. Also anything in M247 Ltd's ASN. They host a lot of VPN endpoints, including Mullvad's.
Step 1, I'd move off Linode. Find a local DC or business you can support by hosting a VPS or dedi box with them. LowEndBox might be an interesting place to search but avoid anything too famous.
Step 2, join this https://sendersupport.olc.protection.outlook.com/pm/services.... They don't actually email me when they block my IP, but at least when I contact them I can argue I'm already in their program and they didn't actually notify me of any sending issues. I've got myself unblocked relatively quickly this way.
Step 1, I'd move off Linode. Find a local DC or business you can support by hosting a VPS or dedi box with them. LowEndBox might be an interesting place to search but avoid anything too famous.
Step 2, join this https://sendersupport.olc.protection.outlook.com/pm/services.... They don't actually email me when they block my IP, but at least when I contact them I can argue I'm already in their program and they didn't actually notify me of any sending issues. I've got myself unblocked relatively quickly this way.
IMO, any low-cost VPS provider is going to have a poor reputation. I've even had trouble running websites on some of them, as some "endpoint security" products have their IP addresses blacklisted, causing users to get alarming warnings if they try to visit sites hosted there.
You'll run into the same issues with the "free/trial" tiers of bulk email services like Mailgun.
I agree with OP, you pretty much have to at least send your outgoing email via an established, widely-accepted SMTP service provider, or in some other way pay a lot for a "clean" reputation.
You'll run into the same issues with the "free/trial" tiers of bulk email services like Mailgun.
I agree with OP, you pretty much have to at least send your outgoing email via an established, widely-accepted SMTP service provider, or in some other way pay a lot for a "clean" reputation.
I'd still say it depends. I've used memset (www.memset.com) in the UK in the past without issue. They have low end VPS offerings, but they're not as well known as say Linode.
It isn't so easy to categorise such a service by pure ASN, as they also offer dedicated and colocation services, so you might be blocking a licensed 'on prem' exchange box for a local UK council.
It isn't so easy to categorise such a service by pure ASN, as they also offer dedicated and colocation services, so you might be blocking a licensed 'on prem' exchange box for a local UK council.
"Wouldn't surprise me if Linode were entirely blocked."
I'd be very surprised. I hosted email on Linode for years without issue.
I'd be very surprised. I hosted email on Linode for years without issue.
I have my personal e-mail on linode for probably 15 years or so and haven't had any issues with e-mail delivery (knock on wood). Had the same two static IPs there for years, and have always had things locked down so I couldn't be used as a relay for spammers.
mine is maybe 10 years? I don’t recall. No issues so far, but started seeing those Microsoft blocks when I moved my brother’s domain over to my server. I hardly send email out, so it could be unrelated to his domain, just that he sends much more email out.
> Trying to deal with a Microsoft ban on my IP
https://sendersupport.olc.protection.outlook.com/snds/index....
https://sendersupport.olc.protection.outlook.com/snds/index....
Thanks I’ll check it out. The bounce link was to an office 365 spam delist portal. My request was auromatically rejected and my only option was to escalate it to support (one click luckily). I’m yet to hear back, but from the stories on the link I shared, I doubt it will do anything.
Wow that is a wonderful link, they really seem to be improving on interacting with the public in these matters.
The link used to be <https://postmaster.live.com/snds/index.aspx>, and has existed since at least 2018.
I vaguely recall seeing in Linode docs that Linode itself blocks outgoing mail, unless you contact them to arrange for them not to.
I had that issue with them once in the past, I found a form on their site, filled it out and they had it fixed in a week or so. No issues since then.
This isn't actually that hard to fix, it's just that for whatever reason, we seem to frequently have this blindspot that we don't seem to have in other industries.
Namely that "do it yourself at home" and "massive oligopolist" aren't the only two options. It's like saying "You can only have hamburgers two ways, cook them yourself or McDonalds."
I do the third and it's been great. I let my paid webhost handle it. (hostdime if you're interested, but I'm sure others do it well also)
Namely that "do it yourself at home" and "massive oligopolist" aren't the only two options. It's like saying "You can only have hamburgers two ways, cook them yourself or McDonalds."
I do the third and it's been great. I let my paid webhost handle it. (hostdime if you're interested, but I'm sure others do it well also)
> like saying "You can only have hamburgers two ways, cook them yourself or McDonalds."
and your comment seems to be saying it’s OK if we lose the ability to cook hamburgers at home, because there are other (more ethical) restaurants that aren’t McDonalds. am i misunderstanding?
and your comment seems to be saying it’s OK if we lose the ability to cook hamburgers at home, because there are other (more ethical) restaurants that aren’t McDonalds. am i misunderstanding?
The metaphor is kinda good. It's okay for people to refuse your home-cooked burgers because they don't trust them to be safe. There are actual small food vendors that comply with sanitation rules and you can have them provide catering services, professionally. And that is okay.
People aren't choosing. Businesses are choosing to reject them before the human ever knows it was sent.
we mandate public health measures because it's kind of easy to check for "safe to eat" quality.
also arguably it's too ineffective considering how hard it is to open a restaurant. (because instead of harassing random taco trucks we should focus on other preventative measures, better visibility of food safety, better reporting and tracking)
sure maybe it's time for a "Let's Email" (after letsencrypt) service that handles reputation for email senders.
we already have certificate transparency logs, OCSP, CAA records, and all the fancy stuff, what's needed is something similar.
also arguably it's too ineffective considering how hard it is to open a restaurant. (because instead of harassing random taco trucks we should focus on other preventative measures, better visibility of food safety, better reporting and tracking)
sure maybe it's time for a "Let's Email" (after letsencrypt) service that handles reputation for email senders.
we already have certificate transparency logs, OCSP, CAA records, and all the fancy stuff, what's needed is something similar.
> sure maybe it's time for a "Let's Email" (after letsencrypt) service that handles reputation for email senders.
> what's needed is something similar.
There are more than a few out there. No offense but if you would have had dealt with spam, you would have known that already.
There are more than a few out there. No offense but if you would have had dealt with spam, you would have known that already.
I know, and they are quite useless since the big mailbox hosts don't interact with them.
the wonder of letsencrypt is that it does have the backing of Google et al.
the wonder of letsencrypt is that it does have the backing of Google et al.
Public health is not at stake. I want to cook my own damn email burger thank you.
Absolutely!
Another way to look at it: In what world are you more likely to be able to cook your own burger: One where it's literally just McDonalds who can now control how people talk about burgers, or one where there's also other restaurants as well.
Another way to look at it: In what world are you more likely to be able to cook your own burger: One where it's literally just McDonalds who can now control how people talk about burgers, or one where there's also other restaurants as well.
If you're trying to deliver to others, it is.
Google "is email delivery guaranteed", it isn't.
What's your point?
I'll take a shot here. I'm not sure why you think Gmail or the other biggies can actually guarantee delivery any better or worse than anyone else. In a technical sense, they definitely can't.
In an anecdotal sense, I mean, I've dealt with my hostdime stuff, gmail and outlook. My stuff and gmail deliver just fine. Outlook does fail.
In a practical sense, see issue above. I think "non-delivery" via the big-boys getting things wrong on spam (and worse future outcomes, pretty easy to imagine gmail trying to "amp" or something similar email such that it cuts out little guys) is a far far greater danger or problem than an individual or smaller outfit tinkering with their own servers and occasionally getting things wrong.
In an anecdotal sense, I mean, I've dealt with my hostdime stuff, gmail and outlook. My stuff and gmail deliver just fine. Outlook does fail.
In a practical sense, see issue above. I think "non-delivery" via the big-boys getting things wrong on spam (and worse future outcomes, pretty easy to imagine gmail trying to "amp" or something similar email such that it cuts out little guys) is a far far greater danger or problem than an individual or smaller outfit tinkering with their own servers and occasionally getting things wrong.
No. To beat up the metaphor, I'm saying if we support the restaurants that aren't McDonalds and that are more "mom and pop" (and get others on board) then we can beat McDonalds or at least live in harmony with them.
To colinsane's point, I don't want to be a restaurant, I want to cook my burger at home. A lot of us do. You're OK with the Hostdime restaurant. I'm OK cooking my own burger. A lot of us are and we don't want to lose that right.
Said it above; but I 100% agree with you. My point has "second-order" effects.
I'm not necessarily saying your email has to go away and you HAVE to use hostdime. I'm saying the goal of McDonalds is to dominate and obsucre everything and anything that cuts into that is useful, especially third-parties like Hostdime. For someone like me that finds it too hard, but also finds gmail unacceptable this is a good (and should be obvious) solution.
How I explain this in real life:
"You'd be stupid to run a business off of gmail. When your phone or internet or electricity goes down, you can call a human and yell at them because you pay them. What happens for google? This is why I pay someone."
I'm not necessarily saying your email has to go away and you HAVE to use hostdime. I'm saying the goal of McDonalds is to dominate and obsucre everything and anything that cuts into that is useful, especially third-parties like Hostdime. For someone like me that finds it too hard, but also finds gmail unacceptable this is a good (and should be obvious) solution.
How I explain this in real life:
"You'd be stupid to run a business off of gmail. When your phone or internet or electricity goes down, you can call a human and yell at them because you pay them. What happens for google? This is why I pay someone."
I am not a Google employee, but I do work with email anti-spam at scale. There's a lot to critique here but it boils down to three points:
1. Spam filter behavior has changed because spam has increased in volume and sophistication, not because ISPs want to save money, or to eliminate competition. Some techniques that worked well 5 years ago aren't as effective anymore. One of the consequences of this has been a reduction in the value of IP reputation, from a spam signal perspective, particularly for low-volume IPs.
2. IP range reputation does matter. The increase in the value of IP range reputation, as a spam signal, has paralleled the decline in value of low-volume IP reputation. In practice, this means you need to either send enough volume to outweigh the reputation of your IP range (exact quantity varies based on a lot of variables, but as a very rough approximation, 1000 messages a day), or find an IP range with good reputation.
IP range reputation is not easy to assess, sometimes even for email professionals. So you can either gamble with a residential ISP IP, or a VPS IP, or you can find a provider that spends time, effort, and expertise on managing IP range reputation. The practical solution for most senders is the latter. Many of these offer a free tier, and many options are available among providers of all sizes.
3. The filtering behavior reported here is either misunderstood or misrepresented. First, no, no major ISP (Gmail, Yahoo, Microsoft/Outlook, icloud) is going to permanently block an IP range; filters are designed to be dynamic. In severe, ongoing, high-volume spam scenarios, you could see a 2-week block, maybe occasionally 30 days. But never "one strike".
Mail deletion without a bounce also cam happen, particularly at Microsoft, but again it's almost never seen for legitimate mail - that response is reserved for long-term, severe spam scenarios, where anyone reasonable would agree that a block is warranted. And, again, this is dynamic.
So it looks like OP is either exaggerating, or has been trying to send from IP ranges with unusually bad spam problems.
1. Spam filter behavior has changed because spam has increased in volume and sophistication, not because ISPs want to save money, or to eliminate competition. Some techniques that worked well 5 years ago aren't as effective anymore. One of the consequences of this has been a reduction in the value of IP reputation, from a spam signal perspective, particularly for low-volume IPs.
2. IP range reputation does matter. The increase in the value of IP range reputation, as a spam signal, has paralleled the decline in value of low-volume IP reputation. In practice, this means you need to either send enough volume to outweigh the reputation of your IP range (exact quantity varies based on a lot of variables, but as a very rough approximation, 1000 messages a day), or find an IP range with good reputation.
IP range reputation is not easy to assess, sometimes even for email professionals. So you can either gamble with a residential ISP IP, or a VPS IP, or you can find a provider that spends time, effort, and expertise on managing IP range reputation. The practical solution for most senders is the latter. Many of these offer a free tier, and many options are available among providers of all sizes.
3. The filtering behavior reported here is either misunderstood or misrepresented. First, no, no major ISP (Gmail, Yahoo, Microsoft/Outlook, icloud) is going to permanently block an IP range; filters are designed to be dynamic. In severe, ongoing, high-volume spam scenarios, you could see a 2-week block, maybe occasionally 30 days. But never "one strike".
Mail deletion without a bounce also cam happen, particularly at Microsoft, but again it's almost never seen for legitimate mail - that response is reserved for long-term, severe spam scenarios, where anyone reasonable would agree that a block is warranted. And, again, this is dynamic.
So it looks like OP is either exaggerating, or has been trying to send from IP ranges with unusually bad spam problems.
> The filtering behavior reported here is either misunderstood or misrepresented. First, no, no major ISP (Gmail, Yahoo, Microsoft/Outlook, icloud) is going to permanently block an IP range; filters are designed to be dynamic. In severe, ongoing, high-volume spam scenarios, you could see a 2-week block, maybe occasionally 30 days. But never "one strike".
You say you're knowledgable, but I also ran my own email server for a couple of decades, and gave up for precisely this reason. Maybe this is even true, but when you're trying to get mail sent, YOU NEED THE MAIL SENT, and you can hardly wait 2 weeks or a month. Besides, your email server is going to give up and report it as undeliverable by that time. So you have to do the legwork to get off the block regardless of how long the "bigs" have the timeout set to. And that's happens so often, and is so onerous, it's not worth the hassle any more. Like many others here, I could have written this article.
You say you're knowledgable, but I also ran my own email server for a couple of decades, and gave up for precisely this reason. Maybe this is even true, but when you're trying to get mail sent, YOU NEED THE MAIL SENT, and you can hardly wait 2 weeks or a month. Besides, your email server is going to give up and report it as undeliverable by that time. So you have to do the legwork to get off the block regardless of how long the "bigs" have the timeout set to. And that's happens so often, and is so onerous, it's not worth the hassle any more. Like many others here, I could have written this article.
The overwhelming majority of legitimate mail will never see anything remotely near a 14-day block. Again, it's reserved for severe, ongoing, high-volume spam scenarios; ISPs use other methods like short-term deferrals for more common/less severe problems.
If you find yourself in an IP range involved in a severe, ongoing, high-volume spam scenario that's affecting your delivery, then it means your provider is not managing IP range reputation, or not doing it very well, and you should vote with your dollars and move somewhere else.
As a rule of thumb, email-specific service providers tend to do a better job of managing IP range reputation than more general purpose providers like VPSes.
If you find yourself in an IP range involved in a severe, ongoing, high-volume spam scenario that's affecting your delivery, then it means your provider is not managing IP range reputation, or not doing it very well, and you should vote with your dollars and move somewhere else.
As a rule of thumb, email-specific service providers tend to do a better job of managing IP range reputation than more general purpose providers like VPSes.
The overwhelming majority of my legitimate mail has seen consistent and irrecoverable 14 day blocks for the last decade+. One on an IP that only did my email for over 6 years. Yet the oligopoly started to block blocks of IPs, including my IP, without recourse.
If you say this problem doesn't exist, you are either not familiar with the problem, misunderstanding it, or simply lying. I hope it is just miscommunication.
Like many here, I gave up self hosting email. To the point that I buy an SMTP service from one of the oligopoly. The incoming mail is handled fine by my server still.
If you say this problem doesn't exist, you are either not familiar with the problem, misunderstanding it, or simply lying. I hope it is just miscommunication.
Like many here, I gave up self hosting email. To the point that I buy an SMTP service from one of the oligopoly. The incoming mail is handled fine by my server still.
Exceptions to the rule do happen; I'm sorry your experience was the exception. Dealing with that kind of thing can be quite a pain if you're not experienced with it.
But again: "If you find yourself in an IP range involved in a severe, ongoing, high-volume spam scenario that's affecting your delivery, then it means your provider is not managing IP range reputation, or not doing it very well, and you should vote with your dollars and move somewhere else."
But again: "If you find yourself in an IP range involved in a severe, ongoing, high-volume spam scenario that's affecting your delivery, then it means your provider is not managing IP range reputation, or not doing it very well, and you should vote with your dollars and move somewhere else."
Did that. Three times. It's whackamole.
You know why? Because any IP-adress that I can afford, is going to be blackholed.
You make it sound like the blame lies with the Linodes, Digital oceans or even the Hezners or ISPs. This is not their fault. The blame lies, entirely, with Microsoft and Google (And to lesser extend Yahoo) using a cannon to shoot a mosquito.
Again: My IP (the address, not the range) was fine. It had been fine for many years. Why then, must Google and/or Microsoft, randomly, block this address? Why can't they make exceptions for reputable addresses within a range of bad ones? (I know why: they are lazy and use the easy path: just block everything and accept some "collateral damage", especially when that "collateral damage" cements their oligopoli a bit more, and when avoiding that collateral damage not only costs more work, but enables competition to exist (and grow))
You know why? Because any IP-adress that I can afford, is going to be blackholed.
You make it sound like the blame lies with the Linodes, Digital oceans or even the Hezners or ISPs. This is not their fault. The blame lies, entirely, with Microsoft and Google (And to lesser extend Yahoo) using a cannon to shoot a mosquito.
Again: My IP (the address, not the range) was fine. It had been fine for many years. Why then, must Google and/or Microsoft, randomly, block this address? Why can't they make exceptions for reputable addresses within a range of bad ones? (I know why: they are lazy and use the easy path: just block everything and accept some "collateral damage", especially when that "collateral damage" cements their oligopoli a bit more, and when avoiding that collateral damage not only costs more work, but enables competition to exist (and grow))
> IP range reputation is not easy to assess, sometimes even for email professionals.
The article is about a man who has been sending emails from the same address for the last couple of decades, and it has always been in his control. Is it difficult to assess the reputation of his address? You seem to be deflecting some blame onto the victim of the shortcomings of your system.
The article is about a man who has been sending emails from the same address for the last couple of decades, and it has always been in his control. Is it difficult to assess the reputation of his address? You seem to be deflecting some blame onto the victim of the shortcomings of your system.
This is not my system, to be clear, but you might be surprised how little 20 years of data matters to filter accuracy, empirically speaking; recent sending behavior carries orders of magnitude more weight.
Agreed, well written counterpoints.
Have around 100 users on my self-hosted mailserver. Works alright for the most part. Once or twice a year, there are connection issues to small companies with weird settings. I just route those over an external ESP.
Then there is also mxroute.com, which is an indie email provider. He seems to do fine too. Didn't use them yet.
So I think having at least some sending volume is key to running an indie server. You can't do it just for a few mailboxes/users.
I still wouldn't recommend to learn or start with email in 2022. There are better uses of your time.
Then there is also mxroute.com, which is an indie email provider. He seems to do fine too. Didn't use them yet.
So I think having at least some sending volume is key to running an indie server. You can't do it just for a few mailboxes/users.
I still wouldn't recommend to learn or start with email in 2022. There are better uses of your time.
He? Is it a one-man-band? If so, that's a scary single point of failure for something as critical as your email.
The owner is from the lowendtalk.com hosting forums.
It was a side project for a few years and has been full time for last couple of years now.
He's very hands on. Has very strict rules about spam - both from sending and receiving perspective.
When he brings in new ip ranges, he babysits them by slowly warming them up until they have a good enough reputation to be accepted by major providers.
It was a side project for a few years and has been full time for last couple of years now.
He's very hands on. Has very strict rules about spam - both from sending and receiving perspective.
When he brings in new ip ranges, he babysits them by slowly warming them up until they have a good enough reputation to be accepted by major providers.
Nah, he should have some staff. I just know the founder ("Jar") from forum posts.
Reminds me of this fun story:
A person at a company mistakenly created an email list segment (or lack thereof) resulting in an email to the entire email list of hundred of thousands of emails. This combined with inexistent (we were a naive startup without an email specialist role) list hygiene practices meant we were blacklisted by Gmail after some time.
Took a year to get a hold of someone on Gmail's spam team. We found out were on 4+ Gmail blocklists, some of which were ML-based. We couldn't do anything to remove ourselves after we fixed the issues. A $1-2 million revenue channel dried up because we couldn't get out of the Gmail blackhole (short of rebranding completely, rewriting content, and using a different ESP). Fun times.
A person at a company mistakenly created an email list segment (or lack thereof) resulting in an email to the entire email list of hundred of thousands of emails. This combined with inexistent (we were a naive startup without an email specialist role) list hygiene practices meant we were blacklisted by Gmail after some time.
Took a year to get a hold of someone on Gmail's spam team. We found out were on 4+ Gmail blocklists, some of which were ML-based. We couldn't do anything to remove ourselves after we fixed the issues. A $1-2 million revenue channel dried up because we couldn't get out of the Gmail blackhole (short of rebranding completely, rewriting content, and using a different ESP). Fun times.
This thread is quite enlightening. It seems the engineering community predominantly would like to be able to self host email (and presumably other services). By proxy of that I guess that email hosting is not just for one user but maybe also friends and family or some community that you are part of. It's quite clear that managed centralised services accelerated the adoption of the internet, email, etc and the majority of consumers don't really care about self hosting but it also speaks to the fact that we've given up ownership of this aspect of our lives without clear understanding of how it will affect us in the long term. If no one but large corporations own all the services we use then it puts us in a pretty precarious position.
> ... [They use] spam as a scapegoat to nerf deliverability and stifle competition.
Disagree. It was a way too open protocol to begin with. From a time of innocence best suited for places with inherit trust like inside a business. And it's not just spam. Phishing is also a huge issue.
As much as I want to sympathise, Email for the big WWW is unsalvageable IMO. Too many bad actors are out there.
> [Solution:...] * There should be a recourse for legitimate servers
This is the same Big tech story. They want to cut cost, you want a human touch. You can see similar stories here in HN every week. Which is why I think it will never happen.
Disagree. It was a way too open protocol to begin with. From a time of innocence best suited for places with inherit trust like inside a business. And it's not just spam. Phishing is also a huge issue.
As much as I want to sympathise, Email for the big WWW is unsalvageable IMO. Too many bad actors are out there.
> [Solution:...] * There should be a recourse for legitimate servers
This is the same Big tech story. They want to cut cost, you want a human touch. You can see similar stories here in HN every week. Which is why I think it will never happen.
> This is the same Big tech story. They want to cut cost, you want a human touch. You can see a similar story here in HN every week.
Dang, sounds like our monopolized tech dystopia is probably inevitable. If only there was some state level mechanism to help balance the interests of industries maximizing profits and those of users who need a robust system which doesn't harm them.
Dang, sounds like our monopolized tech dystopia is probably inevitable. If only there was some state level mechanism to help balance the interests of industries maximizing profits and those of users who need a robust system which doesn't harm them.
A previous thread (5m old) here showed it is not that cut and dry, a good read: https://news.ycombinator.com/item?id=30788681
There is a really great bit from Robert Reich (labor admin from the Clinton years) about how if you're business would collapse if you need to pay people more than minimum wage, you shouldn't be in business.
If your business suddenly fails as soon as you need to... support your customers, then you probably shouldn't be in business and only exist as a fluke in the current economic model.
If your business suddenly fails as soon as you need to... support your customers, then you probably shouldn't be in business and only exist as a fluke in the current economic model.
The topic often comes up. Can't say I share the experience. My servers have never been put on a blacklist in the 7 years they've been running, and one of them operates from my residential DSL connection. Standard postfix+dovecot stack on an Archlinux VPS, I log in once a year to update the packages and make sure there is enough disk space left.
Essentially all residential IPs are blacklisted for email servers, at least on port 25. Many ISPs blackhole any traffic on port 25 to residential IPs. Do you have port 25 working on your residential connection?
Know what will kill self-hosted email? Giving up and hosting your email with the big guys, the less self hosted email servers there are the less the big guys feel any requirement to support it
All that "security" just to fight spam. IIRC it was estimated that globally spammers make $300M per year from their spam. It doesn't seem like much. Somebody joked that it would be better if we just paid them that much to do nothing.
I wonder what the underlying problem is.
Gabe Newell once said that "Piracy is not a pricing issue. It's a service issue". I believe this has been proven by netflix/spotify as well.
Is spam just a symptom of a much deeper problem? If so what is it? Or is it naive to think of spam this way?
Gabe Newell once said that "Piracy is not a pricing issue. It's a service issue". I believe this has been proven by netflix/spotify as well.
Is spam just a symptom of a much deeper problem? If so what is it? Or is it naive to think of spam this way?
I think it is different. With Piracy people want something (product/service/media). The quote from Gabe is about the fact that they are willing to pay for it, if it is convenient and affordable.
Nobody wants spam or the things promoted in the spam. The companies want attention from the users. I think the better version is targeted ads on Facebook, etc. Maybe that’s the closest analogy to Gabe’s quote.
Nobody wants spam or the things promoted in the spam. The companies want attention from the users. I think the better version is targeted ads on Facebook, etc. Maybe that’s the closest analogy to Gabe’s quote.
> Nobody wants spam or the things promoted in the spam.
Lots of people want Viagra without having to see a doctor. There definitely is a market for it, and now there are companies that do provide it, I wonder if this will cause spam to shift to other products.
Lots of people want Viagra without having to see a doctor. There definitely is a market for it, and now there are companies that do provide it, I wonder if this will cause spam to shift to other products.
Many also wants to be heir to a Nigerian prince worth a few million ...
That's true! I had considered those to be scams rather than "normal" spam. When I look at what my filter catches, it broadly falls into three categories: scams, viagra/dating/sex-related products, and every day products ("buy this great ladder for your garage"). Too bad you usually can't analyze how many clicks and sales they generate.
> I wonder what the underlying problem is.
The underlying problem is that a sufficiently motivated spammer can target tens or hundreds of millions of people with their spam without too much effort.
As a result, every possible scam and spam with even the slightest possibility of converting 0.00001% of recipients can now be a viable spam campaign.
The underlying problem is that it’s so easy to scale spam to a lot of targets.
The underlying problem is that a sufficiently motivated spammer can target tens or hundreds of millions of people with their spam without too much effort.
As a result, every possible scam and spam with even the slightest possibility of converting 0.00001% of recipients can now be a viable spam campaign.
The underlying problem is that it’s so easy to scale spam to a lot of targets.
Yup. Just requesting the sender to solve some riddle (and waste their energy in process) would turn the tables completely because the cost of sending would be non-zero. Unfortunately it would also mean that we would be sacrificing our planet again. But maybe the difficulty of the challenge could adapt according to some trust score?
I think the deeper problem is sending an email (or a billion emails) triggers someone else's servers to do most of the work. Email is beautifully cooperative computing, but that means it can easily be taken advantage of.
I think you can exclude Netflix from that list. Streaming services have fractured into a dozen different ones so piracy has come back.
Still applies to games/music and probably audio books.
Still applies to games/music and probably audio books.
Spam is just the opposite, it is actually a pricing issue and not a service issue.
As long as the expected value per message of spam sent is positive, someone in the world will send as many messages as they are able to. You either fix this by raising their costs so that spam no longer has a positive expected value, remove their ability to send an unlimited number of messages, or both.
This is not exclusive to the e-mail system, robocalls are still a major annoyance, and the global telephone system is much more regulated. Mobile phones now automatically filter calls, even! It's wild.
As long as the expected value per message of spam sent is positive, someone in the world will send as many messages as they are able to. You either fix this by raising their costs so that spam no longer has a positive expected value, remove their ability to send an unlimited number of messages, or both.
This is not exclusive to the e-mail system, robocalls are still a major annoyance, and the global telephone system is much more regulated. Mobile phones now automatically filter calls, even! It's wild.
Steam and Netflix made it possible for people to play/watch media more easily than going to the pirate bay.
What are spammers trying to accomplish, that a better product would prevent them from using email (that isn't just them shitting up the other service the way they shit up email)?
What are spammers trying to accomplish, that a better product would prevent them from using email (that isn't just them shitting up the other service the way they shit up email)?
> Somebody joked that it would be better if we just paid them that much to do nothing.
I know this is in jest, but in economics there's this concept called "induced demand" that comes to mind.
"Public extortion" would be an interesting challenge, as it would be difficult to solve the problem of "Hey why don't you also pay ME to do nothing too?"
I know this is in jest, but in economics there's this concept called "induced demand" that comes to mind.
"Public extortion" would be an interesting challenge, as it would be difficult to solve the problem of "Hey why don't you also pay ME to do nothing too?"
Transaction costs! https://en.wikipedia.org/wiki/Coase_theorem
I am still doing it, on a cheap VPS no less. Yes, it is hard, and yes, some large email vendors drop my emails for no reason. However, if everyone throws in the towel, they (the monopolies) won.
I relate to this. I also stopped hosting my own mail server for this exact reason.
However I do think it’s a case of damned if you do damned if you don’t. As a consumer of big tech email I become equally frustrated when spam makes it past the filter and I expect them to do more.
If it’s easy for the average person to setup a mail sever with high reputation then it’s easy for spammers to do the same. I can’t think of a great way to manage this at scale for the average person using a $5 a month Digital Ocean VPS sending < 10 emails a month.
One thing I have noticed is that there’s still a load of large organisations failing to implement basic deliverability best practices like SPF records. These organisations have themselves to blame.
However I do think it’s a case of damned if you do damned if you don’t. As a consumer of big tech email I become equally frustrated when spam makes it past the filter and I expect them to do more.
If it’s easy for the average person to setup a mail sever with high reputation then it’s easy for spammers to do the same. I can’t think of a great way to manage this at scale for the average person using a $5 a month Digital Ocean VPS sending < 10 emails a month.
One thing I have noticed is that there’s still a load of large organisations failing to implement basic deliverability best practices like SPF records. These organisations have themselves to blame.
The paradox here is the same one patio11 discussed in "The optimal amount of fraud is non-zero", on the front page yesterday [0]. The more non-tech people have an email address, the more we have to prevent fraudulent email, and the harder it becomes to run your own email address.
The original email users were much more savvy and needed less protecting against fraud. Now my grandma has an email, and if we're not careful she ends up on the phone with "Microsoft customer support" giving them full access to her computer. Spam filters aren't just a question of irritation anymore, people's life savings are at risk.
[0] https://news.ycombinator.com/item?id=32701913
The original email users were much more savvy and needed less protecting against fraud. Now my grandma has an email, and if we're not careful she ends up on the phone with "Microsoft customer support" giving them full access to her computer. Spam filters aren't just a question of irritation anymore, people's life savings are at risk.
[0] https://news.ycombinator.com/item?id=32701913
I actually would rephrase it as there weren’t enough users with money to make using email a worthwhile scam medium.
Email is low trust and almost any user is susceptible to being scammed, because there aren’t good trust markers in emails, and companies use it in ways that make it indistinguishable from spam.
Email is low trust and almost any user is susceptible to being scammed, because there aren’t good trust markers in emails, and companies use it in ways that make it indistinguishable from spam.
I think both are true. The kinds of scams my grandparents (and even parents) fall for are trivially recognizable as scams to me. It takes more work to defraud someone who knows more about the way thing are supposed to work.
But yes, it's definitely still possible for anyone to fall for more sophisticated scams, and there being more money to be had is a huge part of it. Either way the effect is the same: more protection is necessary than was before.
But yes, it's definitely still possible for anyone to fall for more sophisticated scams, and there being more money to be had is a huge part of it. Either way the effect is the same: more protection is necessary than was before.
Oddly enough I do not have these problems after self-hosting for close to 27 years now (i.e. from before spam became a problem). I hardly ever get any spam and my messages seem to arrive where they need to be even when that destination lies in Microsoft- or Google-land. I have the usual assortment of DKIM/SPF configured for my domain, I send mail through a smart host operated by my IAP (at no extra charge) but for the rest I do not do anything special. Am I the exception to the rule, am I just lucky that my IAP's smart host has not been blacklisted by the likes of Microsoft and Google or is the perception of self-hosting mail to be fraught with problems erroneous? I suspect the latter to be true, self-hosting is neither difficult nor bound to fail just as long as a) you have some good spam filters (easy), b) your MTA is set up with the correct SPF and DKIM records (also easy) and c) you send outgoing mail through a smart host (easy to configure once you've found one).
I tried to operate my own email server as well on a VPS, and I have been thinking that the way to solve these problems is to solve the problem of spam itself. Detecting spam puts the costs on the email providers... when the costs should be born by spammers.
Perhaps some sort of digital stamp (digital signatures similar to stamps on physical envelopes) for each email sent paid for with micropayments in a cryptocurrency like nano (note: I don't own any crypto). Small cost per email like 0.01 cents that is trivial for legitimate senders but not for bulk-sending spammers. SMTP servers should put all incoming unsigned emails into spam folders. This will disincentivize spamming (probably not eliminate it) enough that self-hosting emails might be possible again without having to swim against the tide.
Perhaps some sort of digital stamp (digital signatures similar to stamps on physical envelopes) for each email sent paid for with micropayments in a cryptocurrency like nano (note: I don't own any crypto). Small cost per email like 0.01 cents that is trivial for legitimate senders but not for bulk-sending spammers. SMTP servers should put all incoming unsigned emails into spam folders. This will disincentivize spamming (probably not eliminate it) enough that self-hosting emails might be possible again without having to swim against the tide.
Paper mail, phone calls, and text messages all cost money, and yet you get massive amounts of spam from each of those. No, putting a price on those does NOT help. What will happen is that one source of low-quality spam that will be priced out of existence will subside, and now companies with more money will be priced into the game, because they will decide that sending spam is competitive again, as there's so much less of it nowadays. So then those companies are sending you spam and there's no escape from that. What's the next step? How do you fix that? The "digital stamp" thing hasn't been thought out too far into the future.
Those are the equivalent of your banner ads and pre-roll ads. The next step is ad blocking. Since there's not a lot of companies that actually want to pay for this stuff now, there's actually not so many of them, and you can enumerate them. They'll try to randomize the text using auto-gen and eventually things like GPT-3. This might be successful. How do you protect against that?
Let's say you protected against that. The next step is ad integration. The kind of stuff SponsorBlock removes from YouTube videos. Small mentions of ads and sponsors, integrated into the content. Interaction reminders. Donation begging. SponsorBlock works well because all that stuff is public, but that might not work well for email. Are you willing to let an equivalent of SponsorBlock read your email? Would you trust it? It would require a completely new paradigm for such blocking addons, where we're sure - by means of technological assurances - that such a blocking program cannot spy on us by leaking emails back to the mothership. That's a tough one, and I have a feeling in the current browser-centric environment the effort is just so large, and the required approaches are so far from what's being done nowadays, and the payoff of satisfying a fringe mail-midroll-ad-blocking need is so small, that it simply won't be done.
What's next after that? SponsorBlock hasn't caught up widely enough to cause people to find workarounds. I don't know, but I'm sure it'll come.
Those are the equivalent of your banner ads and pre-roll ads. The next step is ad blocking. Since there's not a lot of companies that actually want to pay for this stuff now, there's actually not so many of them, and you can enumerate them. They'll try to randomize the text using auto-gen and eventually things like GPT-3. This might be successful. How do you protect against that?
Let's say you protected against that. The next step is ad integration. The kind of stuff SponsorBlock removes from YouTube videos. Small mentions of ads and sponsors, integrated into the content. Interaction reminders. Donation begging. SponsorBlock works well because all that stuff is public, but that might not work well for email. Are you willing to let an equivalent of SponsorBlock read your email? Would you trust it? It would require a completely new paradigm for such blocking addons, where we're sure - by means of technological assurances - that such a blocking program cannot spy on us by leaking emails back to the mothership. That's a tough one, and I have a feeling in the current browser-centric environment the effort is just so large, and the required approaches are so far from what's being done nowadays, and the payoff of satisfying a fringe mail-midroll-ad-blocking need is so small, that it simply won't be done.
What's next after that? SponsorBlock hasn't caught up widely enough to cause people to find workarounds. I don't know, but I'm sure it'll come.
> Small cost per email like 0.01 cents that is trivial for legitimate senders but not for bulk-sending spammers
Given how much is spent on ads, I’d say that an email is still worth more than that. So, what you get is spammers paying minuscole amount of money to skip the spam filter and killing the idea at the root.
Or they can simply steal these credentials from one of the millions of hacked sites and cause additional trouble to them.
Given how much is spent on ads, I’d say that an email is still worth more than that. So, what you get is spammers paying minuscole amount of money to skip the spam filter and killing the idea at the root.
Or they can simply steal these credentials from one of the millions of hacked sites and cause additional trouble to them.
Within the confines of "a digital stamp is a good idea" [1], what you're saying isn't quite true: instead of being treated as a whitelist, a digital stamp can just be seen as an finite increase in credibility of some quantum that's decided over time by adaptive filters.
[1] It's not quite. See sibling post to yours.
[1] It's not quite. See sibling post to yours.
Cryptocurrency is a pyramid scheme and even if it weren't, the technology crushes itself under its own weight.
After 19+ years of hosting my own email - It's worth it!
Imagine someone revokes your access or deletes all your emails because of an error, at the scale of gmail or outlook.com it just happens.
For spam there is one solution:
- implement greylisting. It just solves the problem.
Imagine someone revokes your access or deletes all your emails because of an error, at the scale of gmail or outlook.com it just happens.
For spam there is one solution:
- implement greylisting. It just solves the problem.
..or imagine if google just shuts down your account. Gives you no reason and suddenly stuff on your phone, your email and all those related services stop.
The issue with greylisting happens when you receive a lot of email from Google or Outlook servers. Different one each time, unless you're whitelisting them all - which defeats the purpose.
The issue with greylisting happens when you receive a lot of email from Google or Outlook servers. Different one each time, unless you're whitelisting them all - which defeats the purpose.
Hard to untangle the incentive for collusion + need to police legit bad behavior.
This could be a few large providers saying 'we control most email traffic, let's control all email traffic'. Or it could be serious players saying 'spam hurts our users, let's stop criminals using a blanket rule'.
More likely it's a schelling point where large players are rent-seeking (crowding out some competition), but only to the extent they can preserve the illusion this is about policing spam.
Suspect we'll start asking platforms to offer something like due process in the law -- administrative checks that increase the cost to administer a system, and reduce the quality to end users, but increase transparency and make it harder for the platform to engage in corruption.
This could be a few large providers saying 'we control most email traffic, let's control all email traffic'. Or it could be serious players saying 'spam hurts our users, let's stop criminals using a blanket rule'.
More likely it's a schelling point where large players are rent-seeking (crowding out some competition), but only to the extent they can preserve the illusion this is about policing spam.
Suspect we'll start asking platforms to offer something like due process in the law -- administrative checks that increase the cost to administer a system, and reduce the quality to end users, but increase transparency and make it harder for the platform to engage in corruption.
"Newsletters from my alumni organization go to spam. Medical appointments from my doctor who has a self-hosted server with a patient intranet go to spam. Important withdrawal alerts from my bank go to spam. Purchase receipts from e-commerces go to spam. Email notifications to users of my company's SaaS go to spam."
Are you talking here about incoming emails? I expected that these would be reliably delivered to you and that the problem is only with emails you send to the large providers?
Are you talking here about incoming emails? I expected that these would be reliably delivered to you and that the problem is only with emails you send to the large providers?
Part of the problem is that the wrong people are complaining.
Using Google as an example, the author has no right to push anything to a gmail inbox. Google has no contract with the author to accept mail from him.
What Google is doing, it's failing its customers, the people who signed on gmail to have an address where other people could send data to.
And now those people are not receiving everything they could, but it's only up to them to decide whether this is actually a problem and whether it's serious enough to contact gmail support.
I do understand the point and the spirit of the author, but he is actually conflating the freedom of speech with the right to be listened to.
Using Google as an example, the author has no right to push anything to a gmail inbox. Google has no contract with the author to accept mail from him.
What Google is doing, it's failing its customers, the people who signed on gmail to have an address where other people could send data to.
And now those people are not receiving everything they could, but it's only up to them to decide whether this is actually a problem and whether it's serious enough to contact gmail support.
I do understand the point and the spirit of the author, but he is actually conflating the freedom of speech with the right to be listened to.
I didn't see any reference to freedom of speech in the article.
OP Carlos Fenollosa mentioned this:
> Over time I realized that residential IP blocks were banned on most servers. > You just cannot create another first-class node of this network. > Email is now an oligopoly, a service gatekept by a few big companies which does not follow the principles of net neutrality.
It's unfortunately true. However, the reason that how we end up like this is more nuanced than just the big players trying to power grab (perhaps) but rather because of the rise of spam/scams/phishing/malware. All big players like Google (Gmail), Microsoft(Outlook/Live.com/Hotmail), Yahoo!, Apple (iCloud) are suffering from those threats, wasted bandwidth and compute on spam detection heuristic AI.
There are industry consortiums like Spamhause and commercial entities like Barracuda to maintain blacklist/whitelist to restrict access of major MTA network interconnect to fight off spams/malwares/phishing/malware delivery from botnets and individuals. And it helps, at the mean time, it consolidated the control of who can send outbound emails.
We are seeing this trend repeatedly in other communication channels like phone calls (due to robocalls, VoIP numbers are being blacklisted by all major players' services) or Text messaging (due to spam texts, major U.S. wireless carriers band together established Campaign Registry to control who can mass send outbound text messages. This is also known as 10DLC registration).
I think the vulnerabilities of previous communication protocols (email, VoIP, SMS/MMS) lie in the fact those protocols are designed with security in mind. Modern community protocols like Push Notification has been designed with security in mind, which make it less susceptible to abuse and spamming. That's probably the way go forward.
> Over time I realized that residential IP blocks were banned on most servers. > You just cannot create another first-class node of this network. > Email is now an oligopoly, a service gatekept by a few big companies which does not follow the principles of net neutrality.
It's unfortunately true. However, the reason that how we end up like this is more nuanced than just the big players trying to power grab (perhaps) but rather because of the rise of spam/scams/phishing/malware. All big players like Google (Gmail), Microsoft(Outlook/Live.com/Hotmail), Yahoo!, Apple (iCloud) are suffering from those threats, wasted bandwidth and compute on spam detection heuristic AI.
There are industry consortiums like Spamhause and commercial entities like Barracuda to maintain blacklist/whitelist to restrict access of major MTA network interconnect to fight off spams/malwares/phishing/malware delivery from botnets and individuals. And it helps, at the mean time, it consolidated the control of who can send outbound emails.
We are seeing this trend repeatedly in other communication channels like phone calls (due to robocalls, VoIP numbers are being blacklisted by all major players' services) or Text messaging (due to spam texts, major U.S. wireless carriers band together established Campaign Registry to control who can mass send outbound text messages. This is also known as 10DLC registration).
I think the vulnerabilities of previous communication protocols (email, VoIP, SMS/MMS) lie in the fact those protocols are designed with security in mind. Modern community protocols like Push Notification has been designed with security in mind, which make it less susceptible to abuse and spamming. That's probably the way go forward.
> So, starting today, the MX records of my personal domain no longer point to the IP of my personal server. They now point to one of the Big Email Providers.
A MX records don't have to point to an IP; it can point to a host name.
My MX record is a dynamic DNS host name.
> Big email servers permanently blacklist whole IP blocks and delete their emails without processing or without notice. Some of those blacklists are public, some are none.
OK, but if you're having trouble sending, that's no reason to do anything with your MX record, which is for receive only. Just route outbound SMTP through someone forwarding service.
I've run my mail domain for twelve years. In that time, I've not sent SMTP directly to anyone; always through the SMTP forwarding host run by my ISP.
Well, you know, the mail is going through that ISP anyway! If I could directly connect to port 25 of various hosts around the net, I would still be routing through that ISP's hardware. So the fact that mail is routed at a higher semantic level through their SMTP server, rather than just at the IP level, just almost just a footnote.
A MX records don't have to point to an IP; it can point to a host name.
My MX record is a dynamic DNS host name.
> Big email servers permanently blacklist whole IP blocks and delete their emails without processing or without notice. Some of those blacklists are public, some are none.
OK, but if you're having trouble sending, that's no reason to do anything with your MX record, which is for receive only. Just route outbound SMTP through someone forwarding service.
I've run my mail domain for twelve years. In that time, I've not sent SMTP directly to anyone; always through the SMTP forwarding host run by my ISP.
Well, you know, the mail is going through that ISP anyway! If I could directly connect to port 25 of various hosts around the net, I would still be routing through that ISP's hardware. So the fact that mail is routed at a higher semantic level through their SMTP server, rather than just at the IP level, just almost just a footnote.
According to Hoyle, MX records must point to a domain name. The cannot contain addresses or point to aliases. Some mailers can tolerate IP address literals in MX records, but that is not universal and will cause problems.
Unless you're doing something special, there is a big difference between sending your mail to the recipient's smtp server and relaying it through you ISP's smtp server. The difference is that if you send it direct the ISP can't read it, because it's encrypted. If you relay it, the ISP can read your mail, and even tamper with it, unless the message itself has been encrypted with something like pgp.
Your ISP can log and analyze all SMTP traffic if they want to, whether it's being processed by their mail relays or not.
I believe most SMTP servers talk SSL/TLS, so, yes, sending mail directly to an SMTP server that doesn't belong to the ISP prevent the ISP from reading the mail.
Well sure, but if you're worried about traffic analysis you've got bigger problems. The difference here is that if you relay through your ISP they can read the contents of your messages. If you don't, they can't.
But this is the fault of the party you'd like to connect to; they are the ones refusing your direct SMTP connection because your IP address smells funny so that you have to resort to forwarding.
Maybe it's possible to work things out with particular such destinations. If there are certain parties to whom you'd like to send mail more securely, maybe they can give you an account on their SMTP server.
To everywhere else you go through your SMTP provider.
Maybe it's possible to work things out with particular such destinations. If there are certain parties to whom you'd like to send mail more securely, maybe they can give you an account on their SMTP server.
To everywhere else you go through your SMTP provider.
I think the author didn't want to say something technical about this and just wanted to say, that with changing his MX record, that his email server has changed.
My little e-mail Server on an OVH VPS is happily sending and receiving e-Mails to/from the big ones without problems for my 20+ domains. Just a basic postfix/dovecot setup with letsencrypt certificates and SPF/DKIM/DMARC working the way it should. I described everything in a short blog series at https://jan.wildeboer.net/2022/08/Email-0-The-Journey-2022/ in case you are interested.
I started hosting my own email in 2004 before finally giving up and migrating my email to Fastmail last year.
Besides the problems mentioned in this post the real problem I had was dealing with spam. The open source community around spam has really degraded over time, to the point where most solutions are extremely high maintenance and require regular tweaking. Methods that used to work, like greylisting, cause problems when dealing with GMail because google doesn't play nicely with it. The big spam blacklists have also gotten a bit less trustworthy over the years.
Besides the problems mentioned in this post the real problem I had was dealing with spam. The open source community around spam has really degraded over time, to the point where most solutions are extremely high maintenance and require regular tweaking. Methods that used to work, like greylisting, cause problems when dealing with GMail because google doesn't play nicely with it. The big spam blacklists have also gotten a bit less trustworthy over the years.
I started self hosting in 2017. (With a fresh domain. I'm the only user.) My sever closes the incoming connection if it doesn't has a rDNS or the HELO doesn't match the rDNS. Apart from that I have no anti spam measures! Friends and family have my firstname.lastname@domain address. Websites and companies get a <random>@domain address. So far I have only reserved 3 spam mails. Two times they got the address from a public mailing list I participated. One time it was from a webshop that sold or leaked my address.
I’ve noticed since switching to Fastmail from gmail my spam filtering is very problematic. I always end up with legit emails in spam, and I just can change if I want even more legit in spam or more spam in inbox, which still gets through.
I think it’s both a tough problem, as well as something that only becomes easier when you have both talent and scale to have a wide perspective.
I think it’s both a tough problem, as well as something that only becomes easier when you have both talent and scale to have a wide perspective.
After 30 years running my own MTA I also gave up and moved to fastmail.
Yeah, I ran my own e-mail server from 1995 until about 10 years ago. It used to be fairly simple, but between incoming spam and attacks, the various standards that were developed, and my e-mail not getting through... this became a problem that was worth paying someone a few dollars a month to solve for me.
I still use my own domain, but I'll let actual delivery and security experts deal with the day-to-day running of things, while I run my business (which definitely isn't that).
It was a bit sad to give up, but the time and frustration it saved have been more than worthwhile.
I still use my own domain, but I'll let actual delivery and security experts deal with the day-to-day running of things, while I run my business (which definitely isn't that).
It was a bit sad to give up, but the time and frustration it saved have been more than worthwhile.
The fix here is use a commercial provider for outbound smtp but continue to self-host inbound.
Not ideal, but it works.
Not ideal, but it works.
what do you use for outbound SMTP? do you need to add/authenticate each domain or from address with it? (I host several domains and mailboxes)
For my personal mail server I ended up using the free tier of SMTP2GO as outbound smtp relay. You can just register your domain as a whole, and the setup was pretty painless. They let you send up to 1000 mails/month for free, which is plenty for my own personal use, but if you need any more than that, these relay services can get quite expensive. You usually end up paying the same price as the fully hosted solution that most of those providers offer alongside the relay service.
Actually, smarthosting is precisely ideal for deliverability problems, particularly with IP reputation that can't be fixed.
Proper, artisanal self-hosting of email can still be viable depending on your expectations and tolerance levels for random issues.
These days, I operate with the medium-temperature bowl of porridge: AWS WorkMail with custom domains & users. My use case is basically "Replace gmail for personal email". I don't have a lot of patience for running an actual email server, so this is about as custom as I can get.
Running a custom email domain can have other practical implications, such as having to carefully re-iterate spelling when mentioning your email address over the phone to a customer support agent. With a gmail or hotmail account, virtually everyone can type that hostname in without thinking about it. This concern is moderated by being able to select a username with fewer than 5 characters, rather than your full legal name appended with your date of birth.
These days, I operate with the medium-temperature bowl of porridge: AWS WorkMail with custom domains & users. My use case is basically "Replace gmail for personal email". I don't have a lot of patience for running an actual email server, so this is about as custom as I can get.
Running a custom email domain can have other practical implications, such as having to carefully re-iterate spelling when mentioning your email address over the phone to a customer support agent. With a gmail or hotmail account, virtually everyone can type that hostname in without thinking about it. This concern is moderated by being able to select a username with fewer than 5 characters, rather than your full legal name appended with your date of birth.
The cloud services providers have some of the worst IP blocks for spam. Cheap hosting blocks are a distant second place.
Google for example, doesn't even hide the fact you have to request whitelisting using their online business services portal. They don't give a toss what kind of sender authentication/signing hoops you have jumped through already, and user letters may still end up in the spam bin.
Many users have indeed migrated to the web platforms, and don't care about people data-mining their business communications. The real issue is so have many spammers/scams, a side-effect keeping their trade on life support by removing technological administrative barriers for the desperate... and you can't block Google/Outlook/Redmond.
Google for example, doesn't even hide the fact you have to request whitelisting using their online business services portal. They don't give a toss what kind of sender authentication/signing hoops you have jumped through already, and user letters may still end up in the spam bin.
Many users have indeed migrated to the web platforms, and don't care about people data-mining their business communications. The real issue is so have many spammers/scams, a side-effect keeping their trade on life support by removing technological administrative barriers for the desperate... and you can't block Google/Outlook/Redmond.
Hosting an email server on a consumer IP does seem to a losing proposition,
Hosting an email server on a cheap (reputable) cloud server and doing the basics (PTR records, SPF etc) still works well.
Hosting an email server on a cheap (reputable) cloud server and doing the basics (PTR records, SPF etc) still works well.
How do you know? SMTP wasn't designed for reliable delivery. You don't know if your emails are being received, and even i they are, you don't know if they'll be received tomorrow.
You can know from getting responses and sending test mails to different mailboxes.
Doesn't give full certainty, but can be well enough. (If all my recipients respond I don't care about the ones I don't send to anyways)
Doesn't give full certainty, but can be well enough. (If all my recipients respond I don't care about the ones I don't send to anyways)
If you're not getting bounces, then the receiving email servers are not RFC-compliant.
If you're really that worried about it, smarthost.
If you're really that worried about it, smarthost.
Statistically. I communicate with a lot of people via email, and if I don't get an expected response to something of consequence I follow up - it's far more common the recipient simply missed it/didn't get around to it than it is to get "I never got it/I found it in my spam folder".
Completely agree this is anticompetitive and worthly of examination under the light of antitrust laws. Of course the defense that will be offered and one which is partially true will be one of providing safety from spam, phishing, and other evils prevalent in the email system today. Maybe texting is the new email after all...
Very interesting, I've self hosted my e-mail since 1999, which incidentally is 23 years ago. My current server is at Hetzner in Germany.
do you have your own asn? The entire Hetzner network has been banned by Microsoft, so you can't send to Hotmail addresses
Do you have a source for this? Because I don't think it's true.
Not at all, just a /29
Is that really a problem?
The real problem is and always has been "reputation". Now the big boys fuck that up by definition from the start and require you to do the heavy lifting: DNS, SPF, DMARC, DKIM and so on.
I run several email domains quite happily in the UK. I know why it works and I don't resort to magical thinking. My ISP is considered a business one and my IPs are static. I've owned both my work and personal ranges for a while.
Feel free to contact: [email protected] - I'd love to hear your ideas.
Cheers Jon
I run several email domains quite happily in the UK. I know why it works and I don't resort to magical thinking. My ISP is considered a business one and my IPs are static. I've owned both my work and personal ranges for a while.
Feel free to contact: [email protected] - I'd love to hear your ideas.
Cheers Jon
I've been self-hosting since 2004. I currently route via a VPS. The only issues I have seem to be with outlook.com/hotmail.com/etc - free Microsoft accounts. That goes to junk, though replies seem to work fine. Paid Outlook365 seems fine.
Even after speaking with Microsoft's email admin team on the phone a couple times, I still have issues. It's kind of infuriating.
I have properly configured SPF+DKIM (selector rotated daily)+DMARC, and I've gotten set up with dnswl.org.
Even after speaking with Microsoft's email admin team on the phone a couple times, I still have issues. It's kind of infuriating.
I have properly configured SPF+DKIM (selector rotated daily)+DMARC, and I've gotten set up with dnswl.org.
I hosted my mailman stack on VPS for some time, it worked well
I stopped self-hosting because it's too much hassle, but it was any difficult to maintain, (by difficult I mean complex)
It didn't worth the time I spent though, so I quit, but I would do it again if I need to
If I had to maintain a server at home and my ISP blocks it, I would get a VPS and host proxies on the VPS and use VPN tunnel to keep the mails stored locally
But I don't have any reason to do that currently, as well as most people
I stopped self-hosting because it's too much hassle, but it was any difficult to maintain, (by difficult I mean complex)
It didn't worth the time I spent though, so I quit, but I would do it again if I need to
If I had to maintain a server at home and my ISP blocks it, I would get a VPS and host proxies on the VPS and use VPN tunnel to keep the mails stored locally
But I don't have any reason to do that currently, as well as most people
I already wrote this also in another thread. Which was not also only was suggested by me:
* implement and publish policies which emails you accept (regex/strings on domains, emails, headers, signatures and so on)
* found a association where all use this strict (which shall potentially be stricter than gmail and so on) settings and if gmail does not accept these emails, sue them for discrimination.
* implement and publish policies which emails you accept (regex/strings on domains, emails, headers, signatures and so on)
* found a association where all use this strict (which shall potentially be stricter than gmail and so on) settings and if gmail does not accept these emails, sue them for discrimination.
Totally naive questions:
Could we come up with a new protocol (possibly based on SMTP/IMAP/whatever), that would only guarantee to get your email to its recipient if you included some sort of token generated by the recipient and given to you? Something where you could text/message/whatever a unique token to a friend/business/etc. and then they can send you email? And if you email someone, your outgoing email includes the token necessary for them to reply? The contents (including who it’s being sent to) would be encrypted by default rather than being plain text that anyone in between sender and recipient (or at least sender and recipient servers) could read. Is something like that possible?
Obviously at first nobody would have it implemented, so you’d have to get developers interested in writing server and client software, and convince people and companies to use it instead of or in addition to regular email. But I wonder how many people would be interested in such a system and whether it would be workable?
Could we come up with a new protocol (possibly based on SMTP/IMAP/whatever), that would only guarantee to get your email to its recipient if you included some sort of token generated by the recipient and given to you? Something where you could text/message/whatever a unique token to a friend/business/etc. and then they can send you email? And if you email someone, your outgoing email includes the token necessary for them to reply? The contents (including who it’s being sent to) would be encrypted by default rather than being plain text that anyone in between sender and recipient (or at least sender and recipient servers) could read. Is something like that possible?
Obviously at first nobody would have it implemented, so you’d have to get developers interested in writing server and client software, and convince people and companies to use it instead of or in addition to regular email. But I wonder how many people would be interested in such a system and whether it would be workable?
There is no commercial interest in setting up a new decentralized system, because there is no money in it. And you’d have to get buy-in from all major tech companies for something like that to even remotely have a chance to replace email. Everything else already exists in the vein of Slack, Matrix, iMessage, Line, WeChat, and the like.
I gave up self-hosting ages ago. For a while I even used the old Google for Families grandfathred in setup. That also went the way of the dodo a few years ago for me.
I signed up for ImprovMX, setup my domains there, and just route my emails to whatever service I want. I use a random gmail account that I use for my login and Google services, but the email itself is never exposed anywhere, I only give out the custom domain one.
ImproveMX handles routing for my whole family. My mom uses Outlook, and it routes there for her. If google, microsoft, or whatever give me trouble or ban me, I just quickly switch the email and nothing lost.
If you pay for their service, since they have a super generous free tier, you also get SMTP servers to use as outbound, which lets me send emails through them and not have the `on behalf of` email thing. Also they do all the work to make sure their IPs aren't blocked and in good standing with MS, Google, etc.
I signed up for ImprovMX, setup my domains there, and just route my emails to whatever service I want. I use a random gmail account that I use for my login and Google services, but the email itself is never exposed anywhere, I only give out the custom domain one.
ImproveMX handles routing for my whole family. My mom uses Outlook, and it routes there for her. If google, microsoft, or whatever give me trouble or ban me, I just quickly switch the email and nothing lost.
If you pay for their service, since they have a super generous free tier, you also get SMTP servers to use as outbound, which lets me send emails through them and not have the `on behalf of` email thing. Also they do all the work to make sure their IPs aren't blocked and in good standing with MS, Google, etc.
Another very happy (paying) customer of ImprovMX here. I have a dozen or so domains for family and friends configured this way, using ImprovMX for both inbound MX and outbound SMTP (using Gmail as the 'front end' mailbox, but using the custom domain/addresses everywhere).
I too gave up self hosting this all a few years ago - especially when hosting domains for others (family) who aren't tech savvy, I got sick of having to troubleshoot why their emails weren't being delivered. Outsourcing the delivery component to ImprovMX and the mail storage and even inbound spam filtering to Gmail, made things so much easier, even if it does mean relying on a centralised party like ImprovMX.
I too gave up self hosting this all a few years ago - especially when hosting domains for others (family) who aren't tech savvy, I got sick of having to troubleshoot why their emails weren't being delivered. Outsourcing the delivery component to ImprovMX and the mail storage and even inbound spam filtering to Gmail, made things so much easier, even if it does mean relying on a centralised party like ImprovMX.
Well I'm slowly seeing competitors a alternatives pop up, so if it falls apart, we might have places to go. Even Cloudflare now offers a similar service. Minus the SMTP part.
I know a lot of older guys will disagree with me here, the guys that use mailing lists to work on FOSS projects, many of whom I respect very much, but I think email sucks. I only use it for the same reason I have a phone number: because some people need me to have one in order to contact me.
I don't like email. I think the problems with it are shortcomings of the protocol. I'd rather not use it. But I do, as a last resort contact method.
For me, people that use email are like people that primarily communicate over SMS. If I need to talk to you and that's all you'll use, I can. But if there's another way to talk to you I'd rather use that. Xmpp, matrix, signal, shit even telegram and discord if I have to, are preferable to SMS or email. But otherwise, yeah I have some email addresses and a phone number if you insist on doing things that way.
I don't like email. I think the problems with it are shortcomings of the protocol. I'd rather not use it. But I do, as a last resort contact method.
For me, people that use email are like people that primarily communicate over SMS. If I need to talk to you and that's all you'll use, I can. But if there's another way to talk to you I'd rather use that. Xmpp, matrix, signal, shit even telegram and discord if I have to, are preferable to SMS or email. But otherwise, yeah I have some email addresses and a phone number if you insist on doing things that way.
Author do not mention the frictions of going the other side (gmail/hotmail/live):
- you might not receive all the emails people sent to you, for the reasons mentionned in this very article
- you will receive much more spam/unwanted email. I quit using a gmail account because I kept reveiving newsletter and notifications from other people who kept mistaking his account with mine. There was probably a one letter difference in their real email address and mine. I surrendered and gave up trying to tell his relative they weren't reaching the right person, only had fun once by powning his NAS with a "cloud function", and gave up after receiving tons of newsletters and other shit.
- you can lose your account any day, for any reason without any possibility to get it back. I've seen it happen to 2 people with hotmail.
- you might not receive all the emails people sent to you, for the reasons mentionned in this very article
- you will receive much more spam/unwanted email. I quit using a gmail account because I kept reveiving newsletter and notifications from other people who kept mistaking his account with mine. There was probably a one letter difference in their real email address and mine. I surrendered and gave up trying to tell his relative they weren't reaching the right person, only had fun once by powning his NAS with a "cloud function", and gave up after receiving tons of newsletters and other shit.
- you can lose your account any day, for any reason without any possibility to get it back. I've seen it happen to 2 people with hotmail.
Of course it all started with spam. But the response was just as destructive. I worked for a small ISP (25K email customers) back in the mid-2000s, and it was so hard to keep email flowing. All it took was one random customer somewhere on our DSL who had a misconfigured open relay, and spamhaus would blacklist us. They wouldn't respond when we'd try to get the block lifted. We tried and tried to work with them to streamline the process, nobody wanted spam to go through our servers, but they weren't really interested in any cooperation. They were perfectly happy to stop all legitimate email if it stopped a single spam.
I don't work there any more, but I'd be surprised if that little ISP hosts their own email servers nowadays. It's so expensive to deal with such issues, it's just not worth it.
I don't work there any more, but I'd be surprised if that little ISP hosts their own email servers nowadays. It's so expensive to deal with such issues, it's just not worth it.
It’s time for the US mail service to digitise and fulfill its constitutional obligation to provide a private, secure mail service for all American citizens.
How I wish spam was handled:
1. Any email that isn't signed with DKIM is blocked (having a signature specific to the source email address would be better, but that is probably too much to ask for)
2. If the sender isn't in your contact list, block it, or at least mark it as spam.
3. Have an easy way to add new entries to you contact list, maybe a new url scheme similar to mailto:? So that it reduces the friction of say getting a confirmation email when signing up for something, or making sure you get emails from a new acquaintance. It would probably be good to have a way to add a full domain to the allowlist as well.
But that would make selling/sharing email addresses a lot less valuable, so there might be some resistance to that from marketing and adtech.
1. Any email that isn't signed with DKIM is blocked (having a signature specific to the source email address would be better, but that is probably too much to ask for)
2. If the sender isn't in your contact list, block it, or at least mark it as spam.
3. Have an easy way to add new entries to you contact list, maybe a new url scheme similar to mailto:? So that it reduces the friction of say getting a confirmation email when signing up for something, or making sure you get emails from a new acquaintance. It would probably be good to have a way to add a full domain to the allowlist as well.
But that would make selling/sharing email addresses a lot less valuable, so there might be some resistance to that from marketing and adtech.
> Blacklists should not include whole IP blocks. I am not responsible for what my IP neighbor is doing with their server.
This is obviously laughably naive and creates infinite sources of spam.
Before doing a proposal on a core Internet technology you should be required to be on the other side for a while. Do anti-spam at a large retail e-mail service provider for a year and then you can understand the problem space.
You might not be responsible for what your neighbor is doing with their server, but the ESP is responsible for filtering it. The idea that they need to treat each and every comcast IP with equal weight is nuts. IP reputation is the single most valuable tool in the industry; the largest statistical predictor of whether or not an email is abusive.
This is obviously laughably naive and creates infinite sources of spam.
Before doing a proposal on a core Internet technology you should be required to be on the other side for a while. Do anti-spam at a large retail e-mail service provider for a year and then you can understand the problem space.
You might not be responsible for what your neighbor is doing with their server, but the ESP is responsible for filtering it. The idea that they need to treat each and every comcast IP with equal weight is nuts. IP reputation is the single most valuable tool in the industry; the largest statistical predictor of whether or not an email is abusive.
I have self hosted for 20 years. If you want perfect delivery then completely self-hosting isn't for you. It is a good learning experience and once setup it does not require much attention.
A compromise solution is to outsource delivery which is by far the shittiest part of self hosting because of bad business practices and lack of regulation. It is the least interesting bit anyway.
My email server is setup with policies commented out to send outbound emails through another host I maintain if required. When a very large company hosting lots of email for many domains mass ip blacklisted my hosting provider late last year I used this to maintain connectivity while the companies sorted out their dispute.
A compromise solution is to outsource delivery which is by far the shittiest part of self hosting because of bad business practices and lack of regulation. It is the least interesting bit anyway.
My email server is setup with policies commented out to send outbound emails through another host I maintain if required. When a very large company hosting lots of email for many domains mass ip blacklisted my hosting provider late last year I used this to maintain connectivity while the companies sorted out their dispute.
It's strange that in his list of possible solutions to keep e-mail spam free, there is zero mentioning of hashcash [1].
No, that is not a shitty crypto coin. It's just a computational proof that your computer spend some seconds on hashing, which is fine if you send 1-to-1 emails like real people do, but not if you are a spammer who bought a file with 6 million leaked e-mail addresses.
See Back's 2002 paper "Hashcash - A Denial of Service Counter-Measure".[2]
No, that is not a shitty crypto coin. It's just a computational proof that your computer spend some seconds on hashing, which is fine if you send 1-to-1 emails like real people do, but not if you are a spammer who bought a file with 6 million leaked e-mail addresses.
See Back's 2002 paper "Hashcash - A Denial of Service Counter-Measure".[2]
[1] https://wikiless.org/wiki/Hashcash
[2] http://www.hashcash.org/papers/hashcash.pdfIt should be possible for everyone to participate in internet systems equally as peers.
Somehow it seems like the Overton window has shifted such that people find it acceptable that ordinary individuals can no longer take part in the email infrastructure as equal peers.
Somehow it seems like the Overton window has shifted such that people find it acceptable that ordinary individuals can no longer take part in the email infrastructure as equal peers.
Spammers are also equal peers, and are a huge problem. The concept of every computer being an equal peer is in fact part of the problem with email.
This is pretty much exactly the reason I stopped running my own mail. I had the mail host on a static IP, part of a block of "pristine" IPs from a local colocation operation (iNOC in Albany, NY). Ran fine for years, but I started having to call more and more customers about "why haven't you answered my email from two weeks ago?" only to find out I ended up in the spam bin. Wasn't worth it when the risk was losing business.
I moved to Proton Mail as I like their simple interface and support their goals. Pretty good service so far, worth paying for, but I do sort of miss running my own services.
If you run your own mail server in 2022, you are the resistance.
I moved to Proton Mail as I like their simple interface and support their goals. Pretty good service so far, worth paying for, but I do sort of miss running my own services.
If you run your own mail server in 2022, you are the resistance.
I host my mail for around 12 years so only half as long as the author of the article, I faced the same problem- that my emails land in spam for some people. In most cases it's those people who care to receive my email and I always tell them it's their provider who is at wrong and that they should switch. We have some laugh together, exchange some jokes and continue with our lives. I'm one of not many who love postfix and dovecot enough to use it to self host, but I'm fine with that, I won't throw towel and will continue to run my email, hopefully I'll be lucky enough to run it for next 12 years :) Peace everyone :)
>I implemented all the acronyms, secured antispam measures, verified my domain, made sure my server is neither breached nor used to relay actual spam, added new servers with supposedly clean IPs from reputable providers, tried all the silver bullets recommended by Hacker News, used kafkaesque request forms to prove legitimity, contacted the admins of some blacklists.
I cloned a repo, edited two lines in a yaml file, ran docker-compose, logged into a web ui, added my domain, added a couple of dns records (MX, spf, dkim, dmarc) and everything worked (yes, I can deliver emails to gmail and outlook).
I honestly have no idea why so many people say that self hosting emails is hard.
I cloned a repo, edited two lines in a yaml file, ran docker-compose, logged into a web ui, added my domain, added a couple of dns records (MX, spf, dkim, dmarc) and everything worked (yes, I can deliver emails to gmail and outlook).
I honestly have no idea why so many people say that self hosting emails is hard.
You can deliver email to gmail and outlook until you ... can't. Whether the IP block your mail server is on gets blacklisted, some heuristic shifts against you (domain name becomes "bad" and shifts a point score over a threshold for being spam), or some other external factor happens, your perfectly configured mail server will suddenly and possibly with no warning or sign that it's failed, fail.
For even personal mail this is pretty annoying, but if you're relying on mail for business reasons, this is completely unacceptable. You need to be able to assume that mail you send reaches your clients/customers. The chance of your private mail server getting banned might be low, but it's not low enough, and over time that chance only increases (especially if you're hosting from a server on big shared IP blocks with naughty tenets like on most major cloud providers).
For even personal mail this is pretty annoying, but if you're relying on mail for business reasons, this is completely unacceptable. You need to be able to assume that mail you send reaches your clients/customers. The chance of your private mail server getting banned might be low, but it's not low enough, and over time that chance only increases (especially if you're hosting from a server on big shared IP blocks with naughty tenets like on most major cloud providers).
IP addresses and whole address blocks may have a bad karma: if somebody ever sent spam from them, they may be marked as untrusted in various databases, and unblocking them is pretty hard. The filters prefer to err on the side of mis-marking a few legitimate self-hosted emails, instead of passing a spam salvo.
Decentralized trust is still hard.
Decentralized trust is still hard.
Maybe it's a joke, but a problem I see in every "open source self hosted alternative" is that people tend to underestimate how much work is to self-host everything.
It's either paid hosting like AWS, some intermediate docker-compose solution or your own personal server machine. In every case someone has to do the gritty work. It's either a paid service, a volunteering open-source contributor, or you.
It's either paid hosting like AWS, some intermediate docker-compose solution or your own personal server machine. In every case someone has to do the gritty work. It's either a paid service, a volunteering open-source contributor, or you.
I guess try to use it as your main e-mail for a few years and you may see...
[deleted]
I'll keep going with my private server (on a vps). I just moved it to another hoster, meaning new IP block. This caused blocking issues:
Some but not all gmail address delivered to the spam folder. Another big mail provider asked me to set up a web page with contact info on the same domain I use for mail. And another self hosted mail server from a public service agency had me on block list.
This caused me some headaches and I was thinking this could be the end and I have to use one of the big players. But I did not give up, invested time and it works now again!
This caused me some headaches and I was thinking this could be the end and I have to use one of the big players. But I did not give up, invested time and it works now again!
I've been debating whether to do this myself (throw in the towel on mail). I am getting tired of fighting all the assorted battles.
This said, my concern is that the big players seem like they could, at a whim, drop you as a customer, with no recourse. This is what is giving me pause going to the big providers.
I've been looking at mail distributions like Mail-in-a-box, and modoba as an intermediate, though none of them seem to be great. Basically I don't want to stitch together several different opinionated tools into a working mail system anymore.
This said, my concern is that the big players seem like they could, at a whim, drop you as a customer, with no recourse. This is what is giving me pause going to the big providers.
I've been looking at mail distributions like Mail-in-a-box, and modoba as an intermediate, though none of them seem to be great. Basically I don't want to stitch together several different opinionated tools into a working mail system anymore.
Modoboa? https://modoboa.org/en/
I was skeptical about running a mailserver but a friend set up a few mailboxes with Modoboa and so far it's going better than expected. (Mostly we just needed a mail relay.)
I was skeptical about running a mailserver but a friend set up a few mailboxes with Modoboa and so far it's going better than expected. (Mostly we just needed a mail relay.)
Yeah ... Modoboa ... my misspelling ...
I could never get it to work. Installation went great, but then I could not pull up admin panel. Not sure why.
I could never get it to work. Installation went great, but then I could not pull up admin panel. Not sure why.
I am self hosting. It works and I have no problems.
I use https://cloudron.io for orchestration, security - to run it on a VPS. Everything just works.
I use https://cloudron.io for orchestration, security - to run it on a VPS. Everything just works.
I did training provided by a large email security firm, and one thing the presenter said was along the lines of "this spam filter defaults to block the senders domain & IP, you can set an expiration on that block butI don't see a reason why you would". One misconfigured server sending out a single email and I assume by extension someone impersonating your domain could get you perma-blocked from sending emails to that company, and I assume it'd reduce your trust rating for other orgs using that provider.
to me, the root cause of this is just money. a LOT of people have zero scruples when it comes to money, so they consider sending email to every email address they can find, in order to hopefully make a sale, to be a perfectly valid tradeoff.
these people deny that they are causing a problem, or that they ever caused a problem, because admitting that would mean they are a bad person, and they're not a bad person! they're "just trying to feed [their] kids, man."
making decisions based on money alone is always a bad idea. ALWAYS. I do not care if it is one person and one decision, or if it is a business making a decision on behalf of their stock holders, or anything else.
if money alone is your decision-making criteria, you are making a bad decision, or you are making a decision on bad criteria.
someone always pays for everyone's scramble for money. someone always pays, and it is always an unjust payment.
in this case, spammers have cost us our ability to self-host email, which is a very significant problem, as described by the author of the linked article, with rather severe consequences if you hope to have any freedom on the internet.
so, if you work for a company that will, over time, do just about anything to get people to click on ads, you are slowly destroying the internet as it was intended to be, and was, for 2-4 decades, depending on how you define that.
these people deny that they are causing a problem, or that they ever caused a problem, because admitting that would mean they are a bad person, and they're not a bad person! they're "just trying to feed [their] kids, man."
making decisions based on money alone is always a bad idea. ALWAYS. I do not care if it is one person and one decision, or if it is a business making a decision on behalf of their stock holders, or anything else.
if money alone is your decision-making criteria, you are making a bad decision, or you are making a decision on bad criteria.
someone always pays for everyone's scramble for money. someone always pays, and it is always an unjust payment.
in this case, spammers have cost us our ability to self-host email, which is a very significant problem, as described by the author of the linked article, with rather severe consequences if you hope to have any freedom on the internet.
so, if you work for a company that will, over time, do just about anything to get people to click on ads, you are slowly destroying the internet as it was intended to be, and was, for 2-4 decades, depending on how you define that.
Funny thing happened to me today: Gmail sent its own Google Fi customer support email to spam. Haha, wish I noticed that before spending my morning going in circles with chat support.
gmail also puts mails from mailing lists into spam, despite repeated "not spam" tagging. Wondering whether mail getting flagged as spam is even a problem anymore as people get used to erratic results, at which point we can get rid of spam filtering, or what's left of it, altogether when on balance it does more harm than good, such as preventing SMTP self-hosting.
And yet I continue to get "Walmart Confirmation Receipt" or "Verizon Confirmation Receipt" all coming from addresses like "[email protected]" which means google are't even trying for some.
If you continually mark those emails as read without opening them, then gmail will learn that they are junk for your account.
I see this same behaviour. It's infuriating. At this point, I'd almost rather be able to disable all their spam filtering and simply run an App Script, or email client with its own rules that then syncs changes back to the server.
I self host my private email server since 2005. Never had big problems. One time my server was wrong configured and then marked on spamhaus. Just fill in a form and all works fine again. Maybee gmail will blacklist me in some weeks, but fuck you gmail, I have nearby no one that uses a gmail account.
I had in all this year no problem with any other peer. So, maybee the writer has just a malconfigured mailserver or don't know that he is marked on spamhaus because any reason.
The range IP bans hit home. Been hit by this more than a few times. You don’t have to be sending spam, but if someone in the range did, ever, you will be blacklisted all the same, and treated like a spammer.
The people who run these blacklists are unreasonable. I can understand why, they tend to interact with the bowels of the internet and the heuristic is effective. Usually people who need to talk to them are doing something naughty, so why bother taking a chance?
Guilty until proven innocent would be an improvement.
The people who run these blacklists are unreasonable. I can understand why, they tend to interact with the bowels of the internet and the heuristic is effective. Usually people who need to talk to them are doing something naughty, so why bother taking a chance?
Guilty until proven innocent would be an improvement.
I know of some small servers that get a lot of spam and hacking attempts, and their most effective tool against abuse is an IPv4 block ban. Increasingly this became more and more difficult, and I assume email servers are at the same point. Thanks to VPNs, people appear to be able to spawn insane numbers of random IPs.
One solution this decentralized server system came up with is the concept of accounts that have some barrier to entry to create (which involves a delay and proving identity). This account has a private key and it uses this to access the servers through any IP. Abuse on this account and any connected accounts of course leads to the key being temporarily revoked. Lots of positive interactions with well established accounts increases your credibility. Lots of reports decreases you credibility.
If you have been sending credible emails with multiple hosts for 10 years, even if you did get flagged, you would be given the benefit of the doubt. Hell, it should be easy to email the host and give them the headers and the reason why the email was flagged.
About the email space now being owned by big tech, it could simply be time for a boycott until they improve their practices. There is far too much centralization on the web now, and we all contribute to it every time we use an external service rather than host our own.
One solution this decentralized server system came up with is the concept of accounts that have some barrier to entry to create (which involves a delay and proving identity). This account has a private key and it uses this to access the servers through any IP. Abuse on this account and any connected accounts of course leads to the key being temporarily revoked. Lots of positive interactions with well established accounts increases your credibility. Lots of reports decreases you credibility.
If you have been sending credible emails with multiple hosts for 10 years, even if you did get flagged, you would be given the benefit of the doubt. Hell, it should be easy to email the host and give them the headers and the reason why the email was flagged.
About the email space now being owned by big tech, it could simply be time for a boycott until they improve their practices. There is far too much centralization on the web now, and we all contribute to it every time we use an external service rather than host our own.
> One solution this decentralized server system came up with is the concept of accounts that have some barrier to entry to create (which involves a delay and proving identity). This account has a private key and it uses this to access the servers through any IP. Abuse on this account and any connected accounts of course leads to the key being temporarily revoked. Lots of positive interactions with well established accounts increases your credibility. Lots of reports decreases you credibility.
That's essentially DKIM being fed into your average domain reputation system.
That's essentially DKIM being fed into your average domain reputation system.
> That's essentially DKIM being fed into your average domain reputation system.
Yeah, and they are not particularly happy with this solution either. A lot of people complaining, etc, as we see here. I'm still not sure of a better system.
Yeah, and they are not particularly happy with this solution either. A lot of people complaining, etc, as we see here. I'm still not sure of a better system.
I am still hosting my emails via docker mailserver[1]
I got some trouble with outlook.com bans, but Linode helped me to switch to a “good” ip address and it is working fine for now.
I will try ti resist as much as possible, because email is your primary identity “link” on the Internet, and you deserves to own it if you want.
[1]: https://gioorgi.com/2020/mail-server-on-docker/
I will try ti resist as much as possible, because email is your primary identity “link” on the Internet, and you deserves to own it if you want.
[1]: https://gioorgi.com/2020/mail-server-on-docker/
Here's the GitHub link, not just the blog post (although it mentions it): https://github.com/docker-mailserver/docker-mailserver
And the container image: https://hub.docker.com/r/mailserver/docker-mailserver
I'm also using this solution very efficiently myself! So far I haven't had many issues with receiving or sending e-mails and what's especially nice is that I can easily set up as many accounts as I need, for example, one for Zabbix, one for Drone CI, one for OpenProject etc.
Even a cheap VPS is enough to run this setup (though you might need to selectively disable certain components) and the administration could be web based, but frankly the scripts that are present are simple enough and there's nothing unusual about, say, connecting Thunderbird to it.
And the container image: https://hub.docker.com/r/mailserver/docker-mailserver
I'm also using this solution very efficiently myself! So far I haven't had many issues with receiving or sending e-mails and what's especially nice is that I can easily set up as many accounts as I need, for example, one for Zabbix, one for Drone CI, one for OpenProject etc.
Even a cheap VPS is enough to run this setup (though you might need to selectively disable certain components) and the administration could be web based, but frankly the scripts that are present are simple enough and there's nothing unusual about, say, connecting Thunderbird to it.
1999 was a pretty late adopter to the self-hosting email cause since it was pretty apparent since 1997 that free email was going to win out. As Gmail filters get smarter then it becomes an altogether more fruitless cause. Email hosting at the individual level died out for most due to this and the pointless rivers of spam you had to deal with. If you've only now come to the realization that you can't make it then you've been wasting a lot of your life.
I've been hosting my email on my own server since the 90s, but got tired of dealing with keeping up with spam filters. Updated the MX to deliver inbound to mailroute.net and have them do the filtering before forwarding to my server and that's been working great for years. Not free, but not expensive and still gives me 99% of the control i want with very good spam filtering too.
Outbound mail is relayed via mailroute too, which solves the tainted IP delivery problem.
Outbound mail is relayed via mailroute too, which solves the tainted IP delivery problem.
In fairness, things like postfix usually ship with very poor (not to say “moronic”) defaults.
Like, postfix won’t even try to connect to tls-enabled smtp for outgoing email by default, and you have to explicitly point it at the certificate bundle it’s supposed to consider valid.
And you have to tell explicitly to reject incoming plaintext connections from the public internet.
And quite a bit more… Like, why doesn’t postfix have its own freaking spf/dkim implementation BUILT IN?
Like, postfix won’t even try to connect to tls-enabled smtp for outgoing email by default, and you have to explicitly point it at the certificate bundle it’s supposed to consider valid.
And you have to tell explicitly to reject incoming plaintext connections from the public internet.
And quite a bit more… Like, why doesn’t postfix have its own freaking spf/dkim implementation BUILT IN?
Are you referring to this? http://www.postfix.org/postconf.5.html#smtpd_tls_security_le...
" encrypt
Mandatory TLS encryption: announce STARTTLS support to remote SMTP clients, and require that clients use TLS encryption. According to RFC 2487 this MUST NOT be applied in case of a publicly-referenced SMTP server. Instead, this option should be used only on dedicated servers. "
" encrypt
Mandatory TLS encryption: announce STARTTLS support to remote SMTP clients, and require that clients use TLS encryption. According to RFC 2487 this MUST NOT be applied in case of a publicly-referenced SMTP server. Instead, this option should be used only on dedicated servers. "
There are so many mesolithic defaults in email software. So many things have to be constantly reinvented. I really wish it weren't like that.
Things like Maddy (https://maddy.email/) aim to simplify all this. Really great potential, but they're still work in progress.
Things like Maddy (https://maddy.email/) aim to simplify all this. Really great potential, but they're still work in progress.
Is there actually a "big tech" email provider that accepts a message with a 2xx SMTP code and then deletes it? The only one I personally know of never does that. That one also does not use anything like an IP address blacklist. This article doesn't name names, it just waves its hands and throws around some innuendo. But as far as my own personal experience goes, this author has no idea what they are talking about.
Yes, it's Microsoft (the author confirmed in a thread on Twitter).
> Is there actually a "big tech" email provider that accepts a message with a 2xx SMTP code and then deletes it?
Agreed, that is really broken behavior. Once you accept mail for delivery, it should be treated as a contract to deliver that mail. Rejects during SMTP conversation are fine as they notify the sender and do not generate backscatter.
I have heard complaints that Microsoft's hosted mail offerings accept then silently delete mail. It is somewhat believable, as MS Exchange server would respond 2xx for any message to any (including non-existent) destination address and later spam the possibly spoofed sender with bounce messages. Maybe MS has broken behavior like this in their hosted offerings too? And, rather than fix their software, they silently delete mail since they have finally learned that backscatter is a bad thing?
And, MS o365 allows 'delete' as an option for the centralized spam rules maintained by the admin. These mails are accepted 2xx then silently deleted.
MS does other questionable things on their 'free' hosted offerings to mitigate their abysmal spam filtering. I have a couple burner @outlook.com addresses, and they no longer receive any mail reliably from any sender. MS provides the user a place to whitelist senders and domains, but after wasting a bunch of time whitelisting domains, mail still is marked spam. "Junk" is effectively the inbox on those accounts.
Disclaimer: experience is dated, from a past job, running Postfix MTAs for a large organization and dealing with / mitigating MS issues, but never directly involved with any MS stuff.
Agreed, that is really broken behavior. Once you accept mail for delivery, it should be treated as a contract to deliver that mail. Rejects during SMTP conversation are fine as they notify the sender and do not generate backscatter.
I have heard complaints that Microsoft's hosted mail offerings accept then silently delete mail. It is somewhat believable, as MS Exchange server would respond 2xx for any message to any (including non-existent) destination address and later spam the possibly spoofed sender with bounce messages. Maybe MS has broken behavior like this in their hosted offerings too? And, rather than fix their software, they silently delete mail since they have finally learned that backscatter is a bad thing?
And, MS o365 allows 'delete' as an option for the centralized spam rules maintained by the admin. These mails are accepted 2xx then silently deleted.
MS does other questionable things on their 'free' hosted offerings to mitigate their abysmal spam filtering. I have a couple burner @outlook.com addresses, and they no longer receive any mail reliably from any sender. MS provides the user a place to whitelist senders and domains, but after wasting a bunch of time whitelisting domains, mail still is marked spam. "Junk" is effectively the inbox on those accounts.
Disclaimer: experience is dated, from a past job, running Postfix MTAs for a large organization and dealing with / mitigating MS issues, but never directly involved with any MS stuff.
Microsoft and maybe sometimes very rarely Google does it.
> But as far as my own personal experience goes, this author has no idea what they are talking about.
He's not very informed about being on the receiving end.
> But as far as my own personal experience goes, this author has no idea what they are talking about.
He's not very informed about being on the receiving end.
Sending emails with dkim + sfp and I never had big problems with reachability and a postfix server on Digital Ocean.
I haven't done it since last year though. Has something gone terribly wrong?
I remember debugging issues with email sent via aws ses to Hotmail addresses at $dailyJob but I can't think of a single Microsoft product that works well (windows, teams, azure, now even GitHub is starting to work every other day) so it doesn't surprise me.
I haven't done it since last year though. Has something gone terribly wrong?
I remember debugging issues with email sent via aws ses to Hotmail addresses at $dailyJob but I can't think of a single Microsoft product that works well (windows, teams, azure, now even GitHub is starting to work every other day) so it doesn't surprise me.
Big email companies were never threatened by self-host. Spammers ruined it, not big email companies. Spam is an incredibly difficult problem to solve.
I managed 3+Mail at 3Com 35 years ago, and in fact, it gets its own subplot in The Big Bucks (https://www.albertcory.io/the-big-bucks), back when email was brand-new (well, for most people).
However, nowadays I'm bored with stuff like this. PITA. So I totally sympathize with the author.
However, nowadays I'm bored with stuff like this. PITA. So I totally sympathize with the author.
My company exists to solve this problem at scale for the web hosting industry. It’s too bad that self-hosting isn’t viable because of IP reputation problems, but it’s a reality that is unlikely to change any time soon.
I’d say if you want to continue self-hosting, just let go of the delivery part. Use a service like SendGrid; it probably won’t cost you anything and it’s easy to set up.
I’d say if you want to continue self-hosting, just let go of the delivery part. Use a service like SendGrid; it probably won’t cost you anything and it’s easy to set up.
I've been trying for the past week or so to break into the self-hosted email game since I had a VPS and found some tutorials and documentation for postfix + dovecot. I did eventually get inbound and outbound email working although I was using an alias when I think I needed to use a virtual mailbox (to both receive and send email from a user with a different name from the unix user behind it, as right now outbound still leaks my unix username).
But then I looked at Protonmail's cost and it's less than I'm paying for my VPS (which is already cheap for the use I get out of it) so I'm on the fence whether I keep hammering away at that (and then have to wrestle with the big players treating me like a spammer and do my own spam filtering) vs just pay for that convenience. The VPS is staying in any case so it's just a question of whether I pay a little more every month for convenient secure mail.
But then I looked at Protonmail's cost and it's less than I'm paying for my VPS (which is already cheap for the use I get out of it) so I'm on the fence whether I keep hammering away at that (and then have to wrestle with the big players treating me like a spammer and do my own spam filtering) vs just pay for that convenience. The VPS is staying in any case so it's just a question of whether I pay a little more every month for convenient secure mail.
G'Day Carlos
We started hosting own SMTP servers on IBM OS2 using a BBS. Long before retail NET was in Oz.
Fast forward to today and we have a DB with millions of IPV4 address and have never used IPV6, cloudflare or a 3rd party to 100% stop spam like we do day in day out.
You are absolutely right, there are a lot of criminals you are wearing suits who are selling snake oil.
We have X simple rules, one is that inbound port 25 sessions from spammers are put to sleep and we give them a very poor service. Our servers seem overworked and they often go away never to return.
I read your blog post and from approx. 30 years experience including SMTP at gov.au and gov.nz content filtering does not work.
Our NoSpamAccepted ( NSA ) AI powered SMTP servers use the network to stop spam.
How many domains & sub-domains are you self hosting ?
TIA
KG
BTW - We have never been in the BLs as we only relay our outbound
I have been self hosting email for just over 23 years and I am more emboldened than ever to keep doing so. Even with SPF/DKIM/DMARC setup, I am constantly asking people to check their spam folder, add me to their contacts, etc. I refuse to pay an email tax to one of the larger players to solidify their hold over the protocols.
Are the big tech email servers truly not used for spam at all? Because if they are, shouldn't they be permabanning each other as hard as they ban the little guy?
Clearly they're using different rules against each other than against small email servers, and I think that's all the evidence you need to get the EU to take action here.
Clearly they're using different rules against each other than against small email servers, and I think that's all the evidence you need to get the EU to take action here.
They actually do end up banning each other's endpoints (specific IP addresses, not entire ranges) from time to time. --Even the big companies aren't able to prevent their services from being used for spam even though they shut it down as quickly as they can. And usually the blocks are due to the blacklists like SpamHaus, rather than the email providers explicitly blocking competitors' endpoints.
Did you properly set up DKIm, Dmarc and SPF? I have been hosting my own email through a VM in Texas for about two decades. Whenever I had problems with rejected deliveries, they were solved by moving to the new standards the internet invented to fight spam. Did you use a dmarc analyzer to verify you were set up correctly?
I self-host email for a bit less than 20 years and still not given up. Being blackholed by Microsoft or Google is a real risk (though I was lucky so far), but at least I fully control receiving side - if I expect a message I can check my logs to see if there was a delivery attempt if it was not successful why.
I tried to host a home email server a few times and it was a pain to say the least.
Finally I created a droplet on Digital Ocean and used Mail in a Box https://mailinabox.email/ with a glue record to act as a name server.
So far so good
I like a lot of what’s being discussed here. One thing to consider is a similar problem I am facing is people trying to hack into a login page. We see thousands of requests per minute from different IPs… many from VPS, some from obviously hacked TVs/devices. We could implement an exponential back off on abusive IPs but detection requires observation of action that action could result in a compromised account… so another idea is we simply block large ranges of know bad IPs from blacklists… I think this is similar to the email sending issue… it’s not fair and I think a solution could be some kind of “block chain” - make it expensive to login… make it expensive to send an email… but I’m not sure and for email it’s way harder because you need agreement from the oligopoly of email providers… not sure what the solution is
There is even a much larger problem here other than not being able to self-host your own email. The issue is that large providers and large companies have no accountability. No way to ever have a conversation with an actual person (email or by phone) to resolve a simple problem or mistake in an algorithm or automated process (or even a manual process if that's what it is). Canned replies not reply to follow up question not a care in the world or a conscience. And the fact that the service is free (to someone) does not mean a company should be able to so easily do what they want and cause aggravation to others.
And it happens even with paid services at large companies.
That is not 'just business' and has never been the way business operated pre-internet except in a few super rare (and perhaps rare monopoly) situations.
And it happens even with paid services at large companies.
That is not 'just business' and has never been the way business operated pre-internet except in a few super rare (and perhaps rare monopoly) situations.
The Helm email server — funded by future YC CEO Garry Tan’s VC firm, but I bought one before he returned – is a really great compromise between privacy and convenience.
The IP block is managed by the Helm co., they tunnel connections and sell you the (tiny, silent) server and software. Each Helm server generates its own TLS cert, so the tunneling does not violate your privacy (unless it was delivered without TLS, in which case your privacy already vanished upstream).
The only delivery issues I hit are sometimes with Outlook/Microsoft managed domains. It’s been at least a year since I had that issue. When I first bought one someone on gmail had to move a message of mine out of spam, but it’s been fine since. Last I checked their infra is hosted on AWS but apparently they have some screening technique for getting clean IPs.
The IP block is managed by the Helm co., they tunnel connections and sell you the (tiny, silent) server and software. Each Helm server generates its own TLS cert, so the tunneling does not violate your privacy (unless it was delivered without TLS, in which case your privacy already vanished upstream).
The only delivery issues I hit are sometimes with Outlook/Microsoft managed domains. It’s been at least a year since I had that issue. When I first bought one someone on gmail had to move a message of mine out of spam, but it’s been fine since. Last I checked their infra is hosted on AWS but apparently they have some screening technique for getting clean IPs.
The general rule of thumb for my home server is that messages to people I've contacted before, or to addressees that were in a thread that I'm also part of get delivered reliably. Messages to new people I've never emailed before often go to spam. I've learned to accept this.
The biggest challenge is in the sending.
If, for whatever reason, Gmail doesn't like your setup, even if it's 100% according to specs, it's effectively broken. And there's no one to resort to, often no previous relevant search result, because the errors are vague when not silent (you don't get to know the email was hard rejected, i.e. has not even reached the recipient's Spam box).
Your only hope is for your complain to go viral on HN or Twitter and some Googler takes pitty on you.
If, for whatever reason, Gmail doesn't like your setup, even if it's 100% according to specs, it's effectively broken. And there's no one to resort to, often no previous relevant search result, because the errors are vague when not silent (you don't get to know the email was hard rejected, i.e. has not even reached the recipient's Spam box).
Your only hope is for your complain to go viral on HN or Twitter and some Googler takes pitty on you.
Not to mention that your email is tied to your online identity in many ways when mostly every account you make asks for an email to use as your username, verify your account, or recover access to your lost account.
Handing over control of your online identity like that to a centralized third party when you could be cut off at any point [insert here any recent news about people wrongfully losing access to their Gmail account and not getting it reinstated] seems like the wrong solution to be allowed the privilege of sending email.
I guess the very least we could do to keep some control is to own (read: rent) your email domain you could move elsewhere in case you lose access, but then you gotta make sure you don't also lose access to your email domain.
Is there an actual solution to all of this?
Handing over control of your online identity like that to a centralized third party when you could be cut off at any point [insert here any recent news about people wrongfully losing access to their Gmail account and not getting it reinstated] seems like the wrong solution to be allowed the privilege of sending email.
I guess the very least we could do to keep some control is to own (read: rent) your email domain you could move elsewhere in case you lose access, but then you gotta make sure you don't also lose access to your email domain.
Is there an actual solution to all of this?
Email issued by your government, in addition to authentication services needed for online banking etc.
Giving large providers a taste of their own medicine might be interesting, let gmail spend 24 hours on a black hole list, make sure the removal forms have a captcha so they can’t be filled out by automation, when people complain on social media then let people know what it’s about. Social media really gets Google’s attention, it draws them out from behind their algorithms and silence when nothing else will.
Utilizing IPv6 more might help also since at some point it becomes absurd to have quintillions of addresses blocked for all eternity.
There are monopoly and racketeering angles also, since Google and Outlook are suppressing independent mailers in favor of their own paid products. Nobody will make any money except the lawyers but that’s ok as long as the situation improves.
Utilizing IPv6 more might help also since at some point it becomes absurd to have quintillions of addresses blocked for all eternity.
There are monopoly and racketeering angles also, since Google and Outlook are suppressing independent mailers in favor of their own paid products. Nobody will make any money except the lawyers but that’s ok as long as the situation improves.
Gmail wouldn’t care at all, they know the pressure would go the other way. Racketeering doesn’t mean what you think it means.
Racketeering is when the only solution offered to you out of a problem someone created for you is to pay the person who created the problem. In the context of email: "you have to pay us to send your email because we have created a system in which only we can send email." Seems pretty close to racketeering.
Anyway, telling someone they don't understand what something means is almost never a good way to start a conversation.
Anyway, telling someone they don't understand what something means is almost never a good way to start a conversation.
That’s not what racketeering is. In order to be racketeering it must be extortative, fraudulent, or otherwise illegal. Your definition would make PG&E guilty of racketeering, which they are not. Selling fire insurance is also not racketeering.
When someone is using a word incorrectly making them aware of that fact is important. They’re welcome to google it.
When someone is using a word incorrectly making them aware of that fact is important. They’re welcome to google it.
How would selling fire insurance or an electric hookup match my definition? For fire insurance to be racketeering the ones selling the insurance have to be the ones starting the fires. For PG&E... IDK they'd have to have invented work (J/s)? From the Wikipedia:
> Originally and often still specifically, racketeering may refer to an organized criminal act in which the perpetrators offer a service that will not be put into effect, offer a service to solve a nonexistent problem, or offer a service that solves a problem that would not exist without the racket.
It's unlikely (IMO) that GGP meant that they thought it would literally meet the legal definition of racketeering in a court room, more that the situation with email can be described in terms of the textbook definition.
1. https://en.m.wikipedia.org/wiki/Racketeering
> Originally and often still specifically, racketeering may refer to an organized criminal act in which the perpetrators offer a service that will not be put into effect, offer a service to solve a nonexistent problem, or offer a service that solves a problem that would not exist without the racket.
It's unlikely (IMO) that GGP meant that they thought it would literally meet the legal definition of racketeering in a court room, more that the situation with email can be described in terms of the textbook definition.
1. https://en.m.wikipedia.org/wiki/Racketeering
A big problem with the way the big companies fight spam is that sometimes it is a bit too aggressive. Happened a couple of times where legitimate government-agency emails had trouble getting through.
Personally, I prefer defining my own spam list rather having an algo decide what pops into my inbox.
Personally, I prefer defining my own spam list rather having an algo decide what pops into my inbox.
What do I do? I got rid of Spamassassin after being challenged by someone on HN or Reddit, can't recall which. Spamassassin is pretty effective but was consuming too much of my time.
So I stopped. I analyzed mail for a few weeks to look for patterns in the wild with my server. I came to the conclusion to block all but the top level TLD's. That decision yielded very positive results.
I then wrote a simple SPAM blocking server to allow me to block habitual or suspicious TLD SPAM sending domains, as well as a few custom checks for common sense things.
As a result of those two decisions, I am now at or better at blocking SPAM than I was with Spamassassin and 2 other related tools I just remembered I also stopped using, spamass-milter and postgrey.
So I stopped. I analyzed mail for a few weeks to look for patterns in the wild with my server. I came to the conclusion to block all but the top level TLD's. That decision yielded very positive results.
I then wrote a simple SPAM blocking server to allow me to block habitual or suspicious TLD SPAM sending domains, as well as a few custom checks for common sense things.
As a result of those two decisions, I am now at or better at blocking SPAM than I was with Spamassassin and 2 other related tools I just remembered I also stopped using, spamass-milter and postgrey.
RBLs and other blacklists have been annoying for decades, even more so when they were maintained by annoyed individuals rather than corporations like today. You had to beg a single person to remove you off a list that an ISP somewhere used without thinking twice about it.
Honestly wish there were a mode where you pay me $x to send me email and I can sort by spend.
The name evades me, but this was a thing a few years back. IIRC, it was the equivalent of what a stamp for an envelope would cost, and the recipient got a chunk, as did the service provider. Will try and find the name for you, but AFAIK, it's no longer doing business.
Would that be Hashcash you're thinking of?
http://www.hashcash.org/
http://www.hashcash.org/
Alas not! Much more recent, had a very polished of-its-era SaaS style website. IIRC, you might have even been given a specific email address for it, which then forwarded to your chosen personal address.
>In many countries politicians are forced to deploy their own email servers for security and confidentiality reasons. We only need one politician's emails not delivered due to poorly implemented or arbitrary hellbans and this will be a hot button issue.
"I just the other day got... an Internet was sent by my staff at 10 o'clock in the morning on Friday. I got it yesterday!", as Ted Stevens would say.
Unfortunately the man said this as part of a massive, uninformed speech[0] about why big tech[1] needs less regulation.
[0] https://en.wikipedia.org/wiki/Series_of_tubes
[1] Comcast inclusive
"I just the other day got... an Internet was sent by my staff at 10 o'clock in the morning on Friday. I got it yesterday!", as Ted Stevens would say.
Unfortunately the man said this as part of a massive, uninformed speech[0] about why big tech[1] needs less regulation.
[0] https://en.wikipedia.org/wiki/Series_of_tubes
[1] Comcast inclusive
> Unfortunately the man said this as part of a massive, uninformed speech about why big tech needs less regulation.
Sometimes they'll regulate themselves to avoid the exact problem you've cited.
https://www.fec.gov/files/legal/aos/2022-14/202214R_1.pdf
Sometimes they'll regulate themselves to avoid the exact problem you've cited.
https://www.fec.gov/files/legal/aos/2022-14/202214R_1.pdf
[deleted]
[deleted]
I'd like to consider migrating from my self-hosted solution but it seems like all major email providers want $ per month per user|mailbox. Are there any providers out there that charge a reasonable fixed amount for say 50 mailboxes?
Same here, I’ve been hosting my email since 2007 and gave up in the beginning of this year, mostly due to constant delivery issues with microsoft infrastructure, which can be mitigated for a couple of months before coming back again
> The industry should fix email interoperability before politicians do. We will all win.
Not sure if by "politicians" he means legislators, but given very few players that control today's email deliverability, while doing very little to provide observability (=feeback loop) to the users who needs it most (that is users who cannot afford to build an expensive pipeline that optimize deliverability), given all that, I think regulation around distributed protocols observability/fairness is not unlike AI explainability regulation, only I expect that with mail it shouldn't be as hard to implement.
Not sure if by "politicians" he means legislators, but given very few players that control today's email deliverability, while doing very little to provide observability (=feeback loop) to the users who needs it most (that is users who cannot afford to build an expensive pipeline that optimize deliverability), given all that, I think regulation around distributed protocols observability/fairness is not unlike AI explainability regulation, only I expect that with mail it shouldn't be as hard to implement.
For those interested, I made a post related to this topic a few months ago: https://news.ycombinator.com/item?id=31180379
You start with the premise that you can't host your own email, which is problematic. I don't know why the people who fail at self-hosting email are so adamant about telling others that nobody should self-host, but it seems more like a squeaky wheel problem than a real one.
In every scenario, deliverability problems can be solved by smarthosting through a reputable email provider. Period.
You get all the benefits of your own filtering, your own logs showing every delivery attempt, you get to store your data on your own systems, access it however you prefer, et cetera - all the reasons to self-host are there except for delivery logs, and those can be arranged through your smarthost provider.
Simple, huh? So why are so many people emphatically telling us to NOT self-host email?
In every scenario, deliverability problems can be solved by smarthosting through a reputable email provider. Period.
You get all the benefits of your own filtering, your own logs showing every delivery attempt, you get to store your data on your own systems, access it however you prefer, et cetera - all the reasons to self-host are there except for delivery logs, and those can be arranged through your smarthost provider.
Simple, huh? So why are so many people emphatically telling us to NOT self-host email?
> In every scenario, deliverability problems can be solved by smarthosting through a reputable email provider. Period.
Could you provide an example of such a smarthost email provider that has the benefits you mentioned?
Not challenging you, just genuinely curious.
Could you provide an example of such a smarthost email provider that has the benefits you mentioned?
Not challenging you, just genuinely curious.
Sure! I would never recommend, for instance, Dreamhost, but I set up a company's smarthosting with Dreamhost because the company was already using them. Dreamhost in general, though, is quite spammy.
Linode and Panix come to mind.
Really, though, many hosting companies won't necessarily list "smart hosting" / "smarthosting" as a service. If they offer a cheap VPS and offer outgoing SMTP, you can set up smarthosting through a VPS.
One can even smarthost through Outlook / Gmail, if you really want.
Linode and Panix come to mind.
Really, though, many hosting companies won't necessarily list "smart hosting" / "smarthosting" as a service. If they offer a cheap VPS and offer outgoing SMTP, you can set up smarthosting through a VPS.
One can even smarthost through Outlook / Gmail, if you really want.
Deliverability problems sending from VPS providers like Linode are common. The article mentioned using a VPS.
Sending via a VPS is not the same as sending through the VPS provider's email servers. Providers usually have their own outgoing SMTP servers for their clients with people who make sure that deliverability is always working, so sending through those is much more likely to work than sending directly from a VPS.
You said through a VPS. Not the VPS provider's email servers. And you suggested Linode. It looks like you can't send through their servers.
If you're sending through someone else then you're not self hosting!
Actually, yes, you are still self-hosting:
You're in control of all deliveries to you, so you can grep through your logs and see any and every delivery attempt, which you can't do with, say, Gmail.
You can store your data however you like, with as many (or as few) security considerations as you please. Want full disk and OS encryption with a Yubikey that has to be physically present when you boot the system? Sure! Want to encase it in a cube of concrete? Install your server at the top of a tree? Why not? Want to store all of your email encrypted in memory? Go for it!
You can back it up as you like, you can make sure nobody else sees or indexes it, you can access it via command line, webmail, IMAP, POP, whatever. You can more (less) your spool file directly. You can overwrite the disk where the mail was stored when you delete it. It's up to you.
Smarthosting does mean you have less control at delivery time, though. For instance, I can choose to refuse to deliver email to domains that don't negotiate TLS. Smarthosting doesn't easily allow this (or at least I don't know of any way to do this).
But every other part will still be 100% in your control.
You're in control of all deliveries to you, so you can grep through your logs and see any and every delivery attempt, which you can't do with, say, Gmail.
You can store your data however you like, with as many (or as few) security considerations as you please. Want full disk and OS encryption with a Yubikey that has to be physically present when you boot the system? Sure! Want to encase it in a cube of concrete? Install your server at the top of a tree? Why not? Want to store all of your email encrypted in memory? Go for it!
You can back it up as you like, you can make sure nobody else sees or indexes it, you can access it via command line, webmail, IMAP, POP, whatever. You can more (less) your spool file directly. You can overwrite the disk where the mail was stored when you delete it. It's up to you.
Smarthosting does mean you have less control at delivery time, though. For instance, I can choose to refuse to deliver email to domains that don't negotiate TLS. Smarthosting doesn't easily allow this (or at least I don't know of any way to do this).
But every other part will still be 100% in your control.
Yup. I have to pay Google a chunk of change every month to do something I used to be able to do for myself for free. Because suddenly they flagged my domain and no one would get any of my e-mails.
And there ain’t nothing I can do but pay.
And there ain’t nothing I can do but pay.
I've been self-hosting my mail for 17 or 18 years now, by purchasing just some managed webhosting package from somebody who cares about their services not being used for shady stuff (any reputable managed hosting provider) and I think I've never "lost" an outgoing mail for personal use.
I don't understand what the author thinks it's so hard here and why he's painting it so black and white. There's lots of more to "my own e-mail" than choosing between some old notebook running and collecting dust in your garage and using GMail.
Some people just want to find a hair in the soup.
I don't understand what the author thinks it's so hard here and why he's painting it so black and white. There's lots of more to "my own e-mail" than choosing between some old notebook running and collecting dust in your garage and using GMail.
Some people just want to find a hair in the soup.
I've been using exchange online for my personal email. I had to do a bit of fenagling with my dns records, which is old hat at this point, to get it to work but it basically costs $4 per month per user and since I'm the only user it gives me the freedom to use my own domain name and not have to worry about selfhosting. Prior to that I was using exim as my SMTP server and dovecot as my imap/pop3. But managing it was a huge hassle. $4 a month is worth the effort of not maintaining it myself.
[deleted]
Totally unrelated. But this guy has an amazing OS-from-scratch tutorial [0].
[0] https://github.com/cfenollosa/os-tutorial
[0] https://github.com/cfenollosa/os-tutorial
> Email is now an oligopoly, a service gate kept by a few big companies which does not follow the principles of net neutrality.
While I understand what they're getting at here, I disagree. There are certainly other providers you can go with - You just need to pay up.
Happy customer of FastMail here. All of my personal domains, and multiples businesses are tied to it. Wonderful service, no deliverability issues, great features, etc.
So - clearly other companies are able to get it right. Self-hosting is probably difficult, but it's not like you're forced to go to Big Tech.
While I understand what they're getting at here, I disagree. There are certainly other providers you can go with - You just need to pay up.
Happy customer of FastMail here. All of my personal domains, and multiples businesses are tied to it. Wonderful service, no deliverability issues, great features, etc.
So - clearly other companies are able to get it right. Self-hosting is probably difficult, but it's not like you're forced to go to Big Tech.
Yeah, this is my compromise too. I'm not able to spend the effort setting up my own mail server right now, but don't want to just use one of the really big players, so I use my own domain and FastMail. Feels good to support a smaller player.
> So - clearly other companies are able to get it right.
These companies are part of the oligopoly.
One can become part of said oligopoly, but it requires an outsized effort for an individual to keep the service running just for himself.
These companies are part of the oligopoly.
One can become part of said oligopoly, but it requires an outsized effort for an individual to keep the service running just for himself.
I would just like to point out, you _don't_ actually own your domain...
It's ironic, 23 years for me too and my towel is not thrown in at all.
Just did a test with my own mailserver hosted on a small-ish local vps.
Outlook: OK (had to bother with this one when I started out) Google: OK iCloud: OK
I have a pristine track record and not a single byte of outgoing spam so I cannot attest how easy is it to get back into the game after an incident. I do agree with the larger point being made here. It is clear some kind of anti racketeering legislation would be the only fix. Sadly, currently there is zero will from both the EU and the US to fix any of these blatant anti competitive issue on the internet.
Outlook: OK (had to bother with this one when I started out) Google: OK iCloud: OK
I have a pristine track record and not a single byte of outgoing spam so I cannot attest how easy is it to get back into the game after an incident. I do agree with the larger point being made here. It is clear some kind of anti racketeering legislation would be the only fix. Sadly, currently there is zero will from both the EU and the US to fix any of these blatant anti competitive issue on the internet.
My experience is that Outlook will accept email from a new IP ... for awhile. But rather quickly they block it. I think they watch for a commercial amount of traffic, enough to get a reputation from various services. Not enough traffic for a reputation? Very suspicious, adios... How else would legit commercial email senders bring up a new server?
From the MailOp commercial mailers mailing list, it seems Outlook will eventually unblock you, but you have to keep appealing, etc., for multiple rounds.
From the MailOp commercial mailers mailing list, it seems Outlook will eventually unblock you, but you have to keep appealing, etc., for multiple rounds.
The dream is a self-hosting email, which is also super easy to setup, so much that any person in the world could do so without specific technical knowledge.
How? I don't know. But it would be great.
How? I don't know. But it would be great.
I just moved from fastmail to self hosted and it works with no email blocked so far. Had that setup for 6 months. The most important point being sure to have a static IP with reverse DNS.
I am running my own mail server for 3 years now. There were some issues (outlook pushing emails to spam folder) and an IP change (not sure how that affected things but I resubmitted the IP to the Microsoft email tools, can't remember the name of the service).
I would like to join some movement or a class-action lawsuit against big tech for anti-competitive practices regarding private mail servers if such thing exist, and if not i would suggest we make one.
I love the idea of self-hosting my email, but there's no way I'm going through all that work if getting my mail delivered will be a toss-up. Complete motivation-killer.
> We are all experiencing what happened when politicians regulated the web. I hope you are enjoying your cookie modals; browsing the web in 2022 is an absolute hell.
What would they do with email?
To be honest, I kinda like seeing a lot of cookie modals out there. Yeah, the experience can be hellish, but it highlights how many sites are actually collecting data from their users.
With that said, I wonder what alternative regulations are feasible if we don’t rely on politician-mandated regulations.
What would they do with email?
To be honest, I kinda like seeing a lot of cookie modals out there. Yeah, the experience can be hellish, but it highlights how many sites are actually collecting data from their users.
With that said, I wonder what alternative regulations are feasible if we don’t rely on politician-mandated regulations.
I also came to the same conclusions 8 years ago after hosting my own email for more than 2 decades. You just can't do it on your own anymore. There are some proposals how to fix this, but I think a more sane approach is whole email system overhaul/replacement with something better. Many people are thinking about this and come up with good ideas. I hope eventually one of them will get enough traction to replace our frankly outdated email technology.
I have had the exact opposite experience, I used mail in a box and set it up on digital ocean on a 5$ droplet.
Have not had any spam or blacklisting issues and it was super easy to setup.
Have not had any spam or blacklisting issues and it was super easy to setup.
Part of the problem is silent deliverability issues. You can send to someone @gmail.com that you've never messaged before and it won't get rejected. It will probably end up in their Junk folder. You don't know it, they don't know it, and unless they check, which many people don't, it will be as if you never sent it.
>You can send to someone @gmail.com that you've never messaged before and it won't get rejected. It will probably end up in their Junk folder.
That happens even if you send from @gmail to @gmail. Well, it happened to me not long ago.
But I've mostly been using lesser known email providers and I haven't noticed any problems.
That happens even if you send from @gmail to @gmail. Well, it happened to me not long ago.
But I've mostly been using lesser known email providers and I haven't noticed any problems.
I am somewhere in the middle. Have been running my personal mail server for 15y, it mostly works but my emails do get often flagged as spam by the major providers (but not deleted). Though those very same providers are themselves a major source of the spam I receive. Do as I say, not as I do.
The nice thing if you control your domain is to be able to create unique email aliases, which is a way to cut spam to zero. A company starts spamming or leaks your email address, just delete the alias.
The nice thing if you control your domain is to be able to create unique email aliases, which is a way to cut spam to zero. A company starts spamming or leaks your email address, just delete the alias.
Wanted to get my own email specifically for this. Is it possible on one of the serviced emails?
[deleted]
At what level of the net does IP address reputation operate? Do the blocklists ban whole ASNs based on some measured volume of spam?
More transparency (or more likely, less ignorance on my part) here would be helpful.
Can anyone recommend a one stop tool / script for looking oneself up in the reputation services? (And on that note, it is abhorrent that Big Email providers don’t have open reputation databases, or at least ones where I can look myself up.)
More transparency (or more likely, less ignorance on my part) here would be helpful.
Can anyone recommend a one stop tool / script for looking oneself up in the reputation services? (And on that note, it is abhorrent that Big Email providers don’t have open reputation databases, or at least ones where I can look myself up.)
«So, starting today, the MX records of my personal domain no longer point to the IP of my personal server. They now point to one of the Big Email Providers.»
I would have kept the MX records pointing to my personal server, and I would have changed only my configuration to send outgoing email through a third-party relay (eg. Gandi). This would have solved all the author's problems (deliverability issues) while staying 99% self-hosted.
I would have kept the MX records pointing to my personal server, and I would have changed only my configuration to send outgoing email through a third-party relay (eg. Gandi). This would have solved all the author's problems (deliverability issues) while staying 99% self-hosted.
Wrote my smtp server 5 years ago. Still running.
BTW, did you know the smtp protocol works without DNS?
You just need to puth the ipv4 between brackets @[xxx.xxx.xxx.xxx] and for ipv6 @[ipv6:...].
spam? simplicity and freedom has a price (personnaly, I have have very, very little spam since I am self-hosted), and don't think corpos won't try to force you to use their servers one way or another... Whose coding the virus? It is sane to presume it is the seller of anti-virus software...
BTW, did you know the smtp protocol works without DNS?
You just need to puth the ipv4 between brackets @[xxx.xxx.xxx.xxx] and for ipv6 @[ipv6:...].
spam? simplicity and freedom has a price (personnaly, I have have very, very little spam since I am self-hosted), and don't think corpos won't try to force you to use their servers one way or another... Whose coding the virus? It is sane to presume it is the seller of anti-virus software...
>BTW, did you know the smtp protocol works without DNS? //
I did but assume no-one uses that as it's not practical because most IP addresses serve more than one domain? Or does SMTP handle that, like some extra "really-to:" header to encode the user@domain if you use IPvX for delivery?
I did but assume no-one uses that as it's not practical because most IP addresses serve more than one domain? Or does SMTP handle that, like some extra "really-to:" header to encode the user@domain if you use IPvX for delivery?
RFCs.
I self-hosted my email during some years. My IP was banned in certain providers a few times, I guess due to neighbors. I also found quite complex to manage the service using dovecot at that time, but I guess today we have simpler servers.
However, I ended up moving away. Sadly, dealing with deny-lists and the management overhead was not worth for me. Fortunately, there are alternatives that doesn't require you to go to big providers :)
However, I ended up moving away. Sadly, dealing with deny-lists and the management overhead was not worth for me. Fortunately, there are alternatives that doesn't require you to go to big providers :)
> We are all experiencing what happened when politicians regulated the web. I hope you are enjoying your cookie modals; browsing the web in 2022 is an absolute hell.
This. Just This.
Seriously though, as someone fairly uneducated in the space of how standards, protocols, and regulations get set in place, how can the ship be moved on this issue? Tech companies will likely only move / allocate resources if there is financial incentive. So what do we do?
This. Just This.
Seriously though, as someone fairly uneducated in the space of how standards, protocols, and regulations get set in place, how can the ship be moved on this issue? Tech companies will likely only move / allocate resources if there is financial incentive. So what do we do?
Regulate them.
[deleted]
A similar problem does exist when hosting your own crawler. No matter how polite your crawler is, after a few days your IP address gets blocked.
The suggestions are good. Do Postfix and other FOSS email servers implement them?
> - Let's keep antispam measures. ...
> - Change blacklisting protocols so they are not permanent and use an exponential cooldown penalty. ...
> - Blacklists should not include whole IP blocks. ...
> - Stop blackholing. ...
> - There should be a recourse for legitimate servers. ... doing some paperwork or paying a fee to prove I'm legit ...
> - Let's keep antispam measures. ...
> - Change blacklisting protocols so they are not permanent and use an exponential cooldown penalty. ...
> - Blacklists should not include whole IP blocks. ...
> - Stop blackholing. ...
> - There should be a recourse for legitimate servers. ... doing some paperwork or paying a fee to prove I'm legit ...
[deleted]
Nah. I'm glad that people like the OP are squeezed out of sending their own email. I have no problem with an email oligopoly at this point.
For every "good" email server owner, there's probably a million bad ones. And the problem of spam is a big one. If you want to send your own email, get used to telling people to check their spam lists and/or add your email account.
For every "good" email server owner, there's probably a million bad ones. And the problem of spam is a big one. If you want to send your own email, get used to telling people to check their spam lists and/or add your email account.
The same philosophy could be applied to retail and other areas, forget small time hospitals, let the U. S. Government take care of your health.
I'm very much a proponent of Universal Healthcare. I think whatever we currently have for health care is a mix of the worst of both worlds.
Not seeing anyone talking about email lists, now you need to buy a mailchimp type service, but many mailchimp type companies ban political customers if your not on their team.
Setup a private listserv or mailman use to be easy, but now you need to have a smtp provider in front, or you will quickly get blacklisted. Even then, get too big, and you will trigger some email email providers.
Setup a private listserv or mailman use to be easy, but now you need to have a smtp provider in front, or you will quickly get blacklisted. Even then, get too big, and you will trigger some email email providers.
The competition argument is getting weaker and weaker, IMO. The point of competition, in theory, is to facilitate a kind of darwinian process that iterates toward an optimal solution. We have basically found that: Gmail pretty much just works for 99% of people. I sympathize with the 1% it doesn’t work for, but appeals to competition are not the way forward.
The domestic ISP I use has all its IP ranges assigned to spamhaus's Policy Block List. https://www.spamhaus.org/pbl/
However, the ISP does operate its own SMTP proxy that's free to use. So I can send and receive emails using home IP address.
This is sad, but it's a gross and inaccurate oversimplification. Let's look at the summary of "What are we left with?":
> You cannot set up a home email server.
This is true enough to not care about edge cases.
> You cannot set it up on a VPS.
This is definitely not true.
> You cannot set it up on your own datacenter.
This is absolutely, unambiguously untrue.
I get that there are many people out there who don't want to administer an email server, or who administer one (or more) and are tired of trying to train users to DTRT and care about security. The truth is that if you have lots of users, it's likely that one will get compromised, and their account will be used to send spam.
Is it the end of the world? Heck, no, unless you let it go on for days. "It's not if, it's when. Say goodbye to your email. Game over. No recourse." That's just plain not the case at all, unless, again, you don't have monitors in place.
A super simple example: a script which counts the number of email sent by any given user in a certain timeframe is really not complicated. I've used something like this and it has caught a mail loop which wouldn't end because the entity causing the looping was rewriting so much that typical anti-loop checks failed.
So a user gets compromised. If this is a real concern (say, for instance, you have a lot of Windows users), your script should send an alert to you when this user's account has sent several hundred messages over the past hour. You disable the user's account, you clean the mail queue, and you deal with the fallout. Sure, that may mean watching your logs for a few days for rejections and visiting other networks' delisting pages, but it happens.
So there's the largest problem with running your own email server handled. Boom. Done. If you've hosted email for years yet can't / won't do this little bit of work, then that's you. The rest of us understand this.
What about deliverability in general? Isn't that the largest problem, you ask? No. No, it isn't at all. You can even run an email server on your home Internet connection, if your ISP allows incoming connections, the same way you can handle any other general deliverability issue: smarthosting.
If you want to claim that there are NO ISPs out there that can reliably send email outside of Yahoo / Outlook / Google / Amazon, then you might say smarthosting isn't a solution. However, you'd be flatly wrong, so wrong you shouldn't be hosting email.
If your home network can't send email (it almost certainly can't), and your VPS can't send email (it'd probably have issues), and your datacenter can't send email (you're clearly doing something wrong, but let's pretend), then you can smarthost through an email provider that has a good reputation. Period.
Anyone who wants to argue that hosting your own server can't be done today because of deliverability ignores this super obvious solution, which negates this entire article.
Let's move past that and look at the suggestions this article makes:
Should we throw in the towel, proverbially speaking? Certainly not. I disagree with this emphatically.
"This doesn't only affect contrarian nerds." No, it doesn't, but discouraging others isn't the solution. Your lack of solutions isn't a good reason for others to throw in the towel. But why are so many "contrarian nerds" so quick to tell others to NOT do something? Do you tell people to not paint or draw, because it's too hard for you? Or to carve, or write fiction, because you're not good at those things?
"You can no longer set up postfix to manage transactional emails for your business. The emails just go to spam or disappear." Nope. You're accepting that as normal and equal. It isn't. This is the same basic idea as "I can't afford to not run Windows, because everyone else runs Windows" - it's a fundamental misunderstanding on your part that leads you to assume you're the victim, and you're powerless. If your email is being silently dropped, then you need to tell the recipients that they need to 1) complain to their provider, and / or 2) find real, deterministic email services. I've told many people that I'm not responsible for overzealous spam filtering, and I provide proof that the email was delivered. It's on them after that. "But I can't afford to do that!" Then smarthost. This isn't difficult.
"One strike and you're out. For the rest of your life." Nope. Demonstrably, nope, unless you're letting spam flow from your servers for days at a time.
Your recommendations:
"Let's keep antispam measures." Sure, but consider the fact that they're part of the problem. Spam filtering shouldn't be arbitrary - for instance, I do ZERO content filtering, unless or until I can prove to myself that there are no false positives. Email with "storage.googleapis.com" URLs? 100% spam. Email from random addresses / networks with Gmail Reply-To? Absolutely 100% spam. Email from servers with a HELO / EHLO name that doesn't exist? Rejected. But keywords? No. That's stupid. I've seen, for instance, too many abuse email addresses that don't accept spam complaints because of content-based, rather than behavior-based, spam filtering. The problem with Gmail is that they do too much content based filtering, with no rules and no logs that we can see.
"Change blacklisting protocols so they are not permanent and use an exponential cooldown penalty." Fair.
"Blacklists should not include whole IP blocks." I disagree. If your network neighbors are shitty, then you should 1) ask for your IPs to be SWIP'd to you, 2) find a better company that punishes spammers / scammers, and/or 3) smarthost.
"Stop blackholing." Yep. But, "No need to bounce every email" - 100% disagree. If you're sending so many messages that you're overwhelmed by returns, then you're doing something horribly wrong. Every email needs a bounce. This is how email works.
"There should be a recourse for legitimate servers." 100% agree. I think someone who has the time and resources should take all the large providers to court to compel them to have methods for correcting interoperability. If Google, for instance, wants to be like a utility, then they should be forced to act like one and they should have real ways to interoperate. As it is right now, it it not possible to reach an actual human at Google about anything via email. Every single message goes nowhere. They shouldn't be allowed to operate like that, or if they want to be arbitrary, they should lose the right to be called RFC compliant email and the use of Gmail accounts shouldn't be usable for anything public. That's another whole battle, though - why should a company get to call themselves an email provider when they don't provide reliable, repeatable service? Sigh.
"Email discrimination is not only unethical; it's a risk for the industry." Agreed. I think there's already legislation proposed, if not already passed, making certain types of communication unblockable. It's shitty legislation, but it's a first step at a precedent we all need - we need to be able to dictate to large corporations the parameters of what they can do and can't do if they want to be considered email.
> You cannot set up a home email server.
This is true enough to not care about edge cases.
> You cannot set it up on a VPS.
This is definitely not true.
> You cannot set it up on your own datacenter.
This is absolutely, unambiguously untrue.
I get that there are many people out there who don't want to administer an email server, or who administer one (or more) and are tired of trying to train users to DTRT and care about security. The truth is that if you have lots of users, it's likely that one will get compromised, and their account will be used to send spam.
Is it the end of the world? Heck, no, unless you let it go on for days. "It's not if, it's when. Say goodbye to your email. Game over. No recourse." That's just plain not the case at all, unless, again, you don't have monitors in place.
A super simple example: a script which counts the number of email sent by any given user in a certain timeframe is really not complicated. I've used something like this and it has caught a mail loop which wouldn't end because the entity causing the looping was rewriting so much that typical anti-loop checks failed.
So a user gets compromised. If this is a real concern (say, for instance, you have a lot of Windows users), your script should send an alert to you when this user's account has sent several hundred messages over the past hour. You disable the user's account, you clean the mail queue, and you deal with the fallout. Sure, that may mean watching your logs for a few days for rejections and visiting other networks' delisting pages, but it happens.
So there's the largest problem with running your own email server handled. Boom. Done. If you've hosted email for years yet can't / won't do this little bit of work, then that's you. The rest of us understand this.
What about deliverability in general? Isn't that the largest problem, you ask? No. No, it isn't at all. You can even run an email server on your home Internet connection, if your ISP allows incoming connections, the same way you can handle any other general deliverability issue: smarthosting.
If you want to claim that there are NO ISPs out there that can reliably send email outside of Yahoo / Outlook / Google / Amazon, then you might say smarthosting isn't a solution. However, you'd be flatly wrong, so wrong you shouldn't be hosting email.
If your home network can't send email (it almost certainly can't), and your VPS can't send email (it'd probably have issues), and your datacenter can't send email (you're clearly doing something wrong, but let's pretend), then you can smarthost through an email provider that has a good reputation. Period.
Anyone who wants to argue that hosting your own server can't be done today because of deliverability ignores this super obvious solution, which negates this entire article.
Let's move past that and look at the suggestions this article makes:
Should we throw in the towel, proverbially speaking? Certainly not. I disagree with this emphatically.
"This doesn't only affect contrarian nerds." No, it doesn't, but discouraging others isn't the solution. Your lack of solutions isn't a good reason for others to throw in the towel. But why are so many "contrarian nerds" so quick to tell others to NOT do something? Do you tell people to not paint or draw, because it's too hard for you? Or to carve, or write fiction, because you're not good at those things?
"You can no longer set up postfix to manage transactional emails for your business. The emails just go to spam or disappear." Nope. You're accepting that as normal and equal. It isn't. This is the same basic idea as "I can't afford to not run Windows, because everyone else runs Windows" - it's a fundamental misunderstanding on your part that leads you to assume you're the victim, and you're powerless. If your email is being silently dropped, then you need to tell the recipients that they need to 1) complain to their provider, and / or 2) find real, deterministic email services. I've told many people that I'm not responsible for overzealous spam filtering, and I provide proof that the email was delivered. It's on them after that. "But I can't afford to do that!" Then smarthost. This isn't difficult.
"One strike and you're out. For the rest of your life." Nope. Demonstrably, nope, unless you're letting spam flow from your servers for days at a time.
Your recommendations:
"Let's keep antispam measures." Sure, but consider the fact that they're part of the problem. Spam filtering shouldn't be arbitrary - for instance, I do ZERO content filtering, unless or until I can prove to myself that there are no false positives. Email with "storage.googleapis.com" URLs? 100% spam. Email from random addresses / networks with Gmail Reply-To? Absolutely 100% spam. Email from servers with a HELO / EHLO name that doesn't exist? Rejected. But keywords? No. That's stupid. I've seen, for instance, too many abuse email addresses that don't accept spam complaints because of content-based, rather than behavior-based, spam filtering. The problem with Gmail is that they do too much content based filtering, with no rules and no logs that we can see.
"Change blacklisting protocols so they are not permanent and use an exponential cooldown penalty." Fair.
"Blacklists should not include whole IP blocks." I disagree. If your network neighbors are shitty, then you should 1) ask for your IPs to be SWIP'd to you, 2) find a better company that punishes spammers / scammers, and/or 3) smarthost.
"Stop blackholing." Yep. But, "No need to bounce every email" - 100% disagree. If you're sending so many messages that you're overwhelmed by returns, then you're doing something horribly wrong. Every email needs a bounce. This is how email works.
"There should be a recourse for legitimate servers." 100% agree. I think someone who has the time and resources should take all the large providers to court to compel them to have methods for correcting interoperability. If Google, for instance, wants to be like a utility, then they should be forced to act like one and they should have real ways to interoperate. As it is right now, it it not possible to reach an actual human at Google about anything via email. Every single message goes nowhere. They shouldn't be allowed to operate like that, or if they want to be arbitrary, they should lose the right to be called RFC compliant email and the use of Gmail accounts shouldn't be usable for anything public. That's another whole battle, though - why should a company get to call themselves an email provider when they don't provide reliable, repeatable service? Sigh.
"Email discrimination is not only unethical; it's a risk for the industry." Agreed. I think there's already legislation proposed, if not already passed, making certain types of communication unblockable. It's shitty legislation, but it's a first step at a precedent we all need - we need to be able to dictate to large corporations the parameters of what they can do and can't do if they want to be considered email.
What the author doesn’t mention is he uses his email to discuss his penis enlargement company with his friend, a Nigerian prince.
OP mentioned on Twitter that he chose iCloud Mail as the provider. How do You handle issues with losing incoming mails? Apple very often rejects mails from financial companies or ones that contain particular words and they won't report that to You. That's the only issue that keeps me from moving from own server.
So what do the companies with Exchange and Lotus do today? Do they still host their email-servers themselves, or already gave up and moved to big-providers? I can hardly imagine companies will leave their emails on Google or Microsoft for them to scan and abuse. But I'm so many years out of corporate environment.
Majority outsource to the big providers. Way better total cost of ownership, and corp accounts are safe from all the scanning bullshit because they’re paid for accounts. They’re actually the customer not the commodity like on the free versions.
A bit crazy to call this unethical. By that logic McDonalds is unethical for not building a restaurant in the middle of the amazon rainforest. Hey, I want my chicken tenders! Well tough cookies, friend, if there's no economical way to support a customer, said customer shall not be supported. Nothing personal. Nothing unethical.
Couldn’t agree more. I had to leave Zoho even though I didn’t want to. Google just blocked emails to my Gmail customers and left me with no choice.
Write about it here https://bitbytebit.substack.com/p/customer-hacquisition
Write about it here https://bitbytebit.substack.com/p/customer-hacquisition
Email is a protocol from the seventies (unmixed, unencrypted by default), not surprised if it leads to centralisation.
Maybe the approach is all wrong? Maybe we should be educating users to scan their spam folder regularly because spam filters will never be perfect. Then it will not be as big an issue if self-hosted email gets marked as spam, and the users of providers like Gmail can complain and push for filter improvements.
We have lost the old internet where you could have a first class host and interoperate with others. I just shared photos with my family and it was super easy… except my sister who uses Android. Everything is like this now. Big Tech is playing gate keeper and the days of rapid innovation are gone.
It would be possible to solve the spam problem once and for all with a crypto currency:
if (Sender is whitelisted by receiver):
All emails arrive in the inbox
else:
Sender has to send $1 for their email to arrive in the inbox
The $1 will be returned if the receiver repliese-postage, knows-SMTP-4, knows-SMTP-5 -> https://www.rhyolite.com/anti-spam/you-might-be.html
We have cryptographic identity in SMTP already. In fact, almost all our protocols and formats for signing and identity were first adapted from email standards. This has already failed.
"You might be an anti-spam kook" does need to be updated to also include cryptocurrency solutions. You're neglecting the transaction costs associated with cryptocurrency, which might be punitive both for senders and for those attempting to collect the spam bounty. Of course the big email providers could aggregate settlements... but now we're back to having big email providers.
We have cryptographic identity in SMTP already. In fact, almost all our protocols and formats for signing and identity were first adapted from email standards. This has already failed.
"You might be an anti-spam kook" does need to be updated to also include cryptocurrency solutions. You're neglecting the transaction costs associated with cryptocurrency, which might be punitive both for senders and for those attempting to collect the spam bounty. Of course the big email providers could aggregate settlements... but now we're back to having big email providers.
> "You might be an anti-spam kook" does need to be updated to also include cryptocurrency solutions
I was scanning this thread to see if someone would post that old copypasta email solution checklist form again.
Here's one version: https://craphound.com/spamsolutions.txt
I was scanning this thread to see if someone would post that old copypasta email solution checklist form again.
Here's one version: https://craphound.com/spamsolutions.txt
I can’t believe it, there is already a checkbox for “weird new form of money”.
I feel a little bit bad for dismissing the solution… but not very bad.
If low-cost microtransactions were viable, we could do many interesting things. For some reason cryptocurrency is often conflated with microtransactions. Yeah yeah Lightning Network blah blah, but I haven’t heard of anything that could scale to 300 billion emails per day.
I feel a little bit bad for dismissing the solution… but not very bad.
If low-cost microtransactions were viable, we could do many interesting things. For some reason cryptocurrency is often conflated with microtransactions. Yeah yeah Lightning Network blah blah, but I haven’t heard of anything that could scale to 300 billion emails per day.
This was a bad idea when it was first proposed 30 years ago
I will stop replying to my mail for sure
I think ideally replies would have a token to bypass it (since replies are expected)
Been self-hosting my email since 2018 on my homeserver. Luckily my ISP's static IP block is not blacklisted by any public list. Only had one instance so far where I couldn't register to a website because they were only allowing 10 or so domains from big providers in your address.
> The industry must self-establish clear rules which are harsh on spammers but give everybody a fair chance.
The author is dreaming, sadly. Why would the big email providers (corporations) change rules or do anything, when the failing system drives more people (such as the author) into their arms?
The author is dreaming, sadly. Why would the big email providers (corporations) change rules or do anything, when the failing system drives more people (such as the author) into their arms?
Funny seeing this on the top of HN when this is also on the top of HN: https://jan.wildeboer.net/2022/08/Email-0-The-Journey-2022/
Couple of suggestions: Oracle's OCI has a free offering. One could use that as a mail relay.
Also SpamHero has been a pleasure to work with with very reasonable pricing to filter most of the junk that comes in, re-delivery service and outbound relay
How does one verify that their server has never ever sent spam, for instance through a security breach?
> You can no longer set up postfix to manage transactional emails for your business
Well, actually you can. But it's tough : https://news.ycombinator.com/item?id=20553028
Well, actually you can. But it's tough : https://news.ycombinator.com/item?id=20553028
Looks like the op decided to go w/ iCloud for the cfenollosa.com domain[1].
[1] https://files.littlebird.com.au/Shared-Image-2022-09-05-07-2...
[1] https://files.littlebird.com.au/Shared-Image-2022-09-05-07-2...
Do not throw your towel.
I also manage since over 20 years my own mail server. Had a few problem but less than expected.
I love the flexibility and low costs besides the time which is needed to understand what is going. But this was a good invest if I look back.
Keep your mailservers running
I also manage since over 20 years my own mail server. Had a few problem but less than expected.
I love the flexibility and low costs besides the time which is needed to understand what is going. But this was a good invest if I look back.
Keep your mailservers running
I've hosted for several decades as well and intend to continue to exercise my right to do so. Whenever one of these monthly or bi-monthly doom and gloom woe is me posts shows up on HN; I wonder if there are any other motivations behind them.
I have self-hosted my personal email account for years without issues. At some point early on I started relying on AWS for sending emails, but that can be easily switched out or removed at any time (and its free for my small volumes of sent emails).
>You cannot set it up on your own datacenter.
There are still new email providers appearing every few years right? What special incantations are they performing to be allowed in the club? Do they buy large IP ranges? Do they pay protection money to Google et al?
There are still new email providers appearing every few years right? What special incantations are they performing to be allowed in the club? Do they buy large IP ranges? Do they pay protection money to Google et al?
They buy their own IP range, establish their own AS, and pay for peering/bandwidth as an independent network and not as a single user/customer on someone else's network.
This is fairly straightforward to do, but requires more commitment than renting some CPU time and bandwidth on someone else's machine.
This is fairly straightforward to do, but requires more commitment than renting some CPU time and bandwidth on someone else's machine.
As someone who was actually looking into utilising my own mail server for building my technical knowledge and understanding, I am dissuaded a bit that my efforts would end in a host of hinderances caused by some auto moderators and filters.
For those who are still self-hosting, what tools & services do you use and find useful?
I know the pain and the annoyances of hosting Email.
But still, I won't switch to big providers, I use Proton Mail personally, and Postale.io for many projects.
There is also mailbox.org and many others, you have a choice to not use the big providers, it totally possible.
But still, I won't switch to big providers, I use Proton Mail personally, and Postale.io for many projects.
There is also mailbox.org and many others, you have a choice to not use the big providers, it totally possible.
“ This concept may sound familiar to you. It's called a racket.”
Wouldn’t RICO statutes apply then?
Wouldn’t RICO statutes apply then?
The author is confusing not being independent one person setup friendly to an oligopoly.
There are plenty of ISPs that offer email services , plenty of small businesses, government orgs, universities host on their own, most have no problems .
Email is still very much small business friendly, it is no longer easy for indie one person setups anymore.
Many industries are not one person shop friendly , that is just nature of the industry, doesn’t make it a racket
There are plenty of ISPs that offer email services , plenty of small businesses, government orgs, universities host on their own, most have no problems .
Email is still very much small business friendly, it is no longer easy for indie one person setups anymore.
Many industries are not one person shop friendly , that is just nature of the industry, doesn’t make it a racket
Partially. Mail reputation is managed by companies like this:
https://senderscore.org/get-your-score/
This website should get a very, very low trust score, just as the company itself.
https://senderscore.org/get-your-score/
This website should get a very, very low trust score, just as the company itself.
I use a third-party spam filter service, for both inbound and outbound. My MX records point to the service provider. I appreciate hiding behind them so to speak and I only open the firewall to the service provider.
Forget about self hosting. Even a custom domain is a pain sometimes.
According to a lot of web apps, my email is not valid and can’t use it.
Also I have been told by a customer support person that my email is not right as it has to end with gmail.com
According to a lot of web apps, my email is not valid and can’t use it.
Also I have been told by a customer support person that my email is not right as it has to end with gmail.com
My dad used to self-host email for his company. It worked fine for a long time -- until MS started silently discarding his emails.
A lot of his clients (very large companies) use MS for email, and his emails to them got silently discarded (not rejected as spam, not showing up in spam folder, simply discarded while responding that the email had been accepted). Notably, the invoices he was sending didn't reach the intended inbox, so he eventually had to move on to use another provider for email.
I've heard this story dozens (if not hundreds) of times over the last couple of decades. It gets worse and worse, since huge corporations only whitelist other huge corporations, and can choose to blacklist the rest. The oligopoly has won.
A lot of his clients (very large companies) use MS for email, and his emails to them got silently discarded (not rejected as spam, not showing up in spam folder, simply discarded while responding that the email had been accepted). Notably, the invoices he was sending didn't reach the intended inbox, so he eventually had to move on to use another provider for email.
I've heard this story dozens (if not hundreds) of times over the last couple of decades. It gets worse and worse, since huge corporations only whitelist other huge corporations, and can choose to blacklist the rest. The oligopoly has won.
I can't send mail to mit.edu addresses. It gets rejected with a message that says I should talk to Microsoft.
I frequently end up in GMail's spam folders. No idea why.
I see no more than one piece of 'spam' a week; everything else is caught by a combination of a 15 minute greylist, the zen spamhaus BL, and SpamAssassin evaluating things. There are a bunch of spammers who send from accounts with valid SPF and DKIM, by the way.
Spam is a reasonably solved problem, but SPF and DKIM aren't much help.
(I get lots and lots of business spam, where the sender clearly believes that they have a right to try to sell me crap. The difference is that if I respond to business spam, someone will answer, trying to sell me crap.)
I frequently end up in GMail's spam folders. No idea why.
I see no more than one piece of 'spam' a week; everything else is caught by a combination of a 15 minute greylist, the zen spamhaus BL, and SpamAssassin evaluating things. There are a bunch of spammers who send from accounts with valid SPF and DKIM, by the way.
Spam is a reasonably solved problem, but SPF and DKIM aren't much help.
(I get lots and lots of business spam, where the sender clearly believes that they have a right to try to sell me crap. The difference is that if I respond to business spam, someone will answer, trying to sell me crap.)
SPF/DKIM are not intended to prevent people from sending spam.
They are intended to let a domain owner tell other email providers what servers have the right to send email from that domain (and sign email).
I can send email all day from [email protected]. But most email providers will check SPF and see that my VPS IP is not authorized by google.com to send email, and it will go to spam.
Those standards wouldn't keep me from registering example.com, setting up SPF correctly, and then spamming. There are other techniques to filter out that kind of behavior.
They are intended to let a domain owner tell other email providers what servers have the right to send email from that domain (and sign email).
I can send email all day from [email protected]. But most email providers will check SPF and see that my VPS IP is not authorized by google.com to send email, and it will go to spam.
Those standards wouldn't keep me from registering example.com, setting up SPF correctly, and then spamming. There are other techniques to filter out that kind of behavior.
> I can send email all day from [email protected]. But most email providers will check SPF and see that my VPS IP is not authorized by google.com to send email, and it will go to spam.
Actually, it will be discarded completely.
Google's DMARC policy is for mail from unrecognized origins to be rejected:
Actually, it will be discarded completely.
Google's DMARC policy is for mail from unrecognized origins to be rejected:
host -t txt _dmarc.google.com
_dmarc.google.com descriptive text "v=DMARC1; p=reject; rua=mailto:[email protected]"
I might be reading into this a little, but you seem to be arguing that "Spam is a reasonably solved problem at my scale, therefore spam is a reasonably solved problem at global scale", and those are very different things. One of the key differences is: a small domain like yours is multiple orders of magnitude less attractive of a target, vs. Gmail, and as a result Gmail's filters are subject to constant, high-volume adversarial attacks, and you are not.
SPF and DKIM are pretty explicit in their RFCs that passing authentication isn't a sign the mail is legitimate. The presence of passing auth in a message does change how filters should handle it, but for most larger-scale production filtering systems (not spamassassin) that mostly ends up as "change the weight of certain reputation identifiers in the spam filtering inputs", more or less.
SPF and DKIM are pretty explicit in their RFCs that passing authentication isn't a sign the mail is legitimate. The presence of passing auth in a message does change how filters should handle it, but for most larger-scale production filtering systems (not spamassassin) that mostly ends up as "change the weight of certain reputation identifiers in the spam filtering inputs", more or less.
Sorry to hear you go!
I still self host my email. I ended up on a blacklist last year probably because some IP in my VPS block got marked. It sucked for about a week. But it came back eventually.
I still self host my email. I ended up on a blacklist last year probably because some IP in my VPS block got marked. It sucked for about a week. But it came back eventually.
I run my own email server for receiving, but use the ISP's server for sending (and setting up Exim on Ubuntu provides such an option at installation time). I have no problem.
Despite my friends repeatedly marking my email as "not spam" google keeps classifying them as spam. Still hoping that one day European anti trust laws will fix that.
> There should be a recourse for legitimate servers
... one of the "big three" being google, this will never happen, there is no recourse in anything, even when you pay them for it.
... one of the "big three" being google, this will never happen, there is no recourse in anything, even when you pay them for it.
The worst is Telekom in DE. They required you to host a website with your address, contact e-mail and phone number. If you don't, your e-mail is just blocked.
Everyone that can should host their own HTTP/1.1, DNS and SMTP at home on a battery backed Raspberry 1, 2 or Zero if they have an external IP.
Preferably with their own implementations, I have implemented all 3.
To host anything beyond those protocols and/or on more powerful hardware is often counter productive.
The problem is getting the ports opened, you need to fight for that right even if it makes spam worse in the short term.
Fight for external IP, ports and static IP in that order.
Edit: Reposting as separate comment because very important!
Edit2: How I wish downvoting required an argument. 500 karma is too low a barrier.
Preferably with their own implementations, I have implemented all 3.
To host anything beyond those protocols and/or on more powerful hardware is often counter productive.
The problem is getting the ports opened, you need to fight for that right even if it makes spam worse in the short term.
Fight for external IP, ports and static IP in that order.
Edit: Reposting as separate comment because very important!
Edit2: How I wish downvoting required an argument. 500 karma is too low a barrier.
If some group wants to get serious about self-hosting email, first they need to define a new subset of the email standard for them all to conform to. Unencrypted C2S, unauthenticated senders, and outdated protocols like POP shouldn't be included. Make it simple to self-host.
XMPP has similar flaws except even worse. Too many different feature sets floating around, too many weird/insecure defaults. It also lost the federation game.
Think of HTTPS, a success story of self-hosting. The weird cases are just out of the question. You basically can't use encryption without signing, or old versions of TLS.
XMPP has similar flaws except even worse. Too many different feature sets floating around, too many weird/insecure defaults. It also lost the federation game.
Think of HTTPS, a success story of self-hosting. The weird cases are just out of the question. You basically can't use encryption without signing, or old versions of TLS.
how many of posters here have used mailinabox ? or something similar? how was your experience?
i am almost 2 years into it now and beyond the first months hiccups, it just works
i am almost 2 years into it now and beyond the first months hiccups, it just works
MIAB user here. I pay ~$4 for a small VPS hosted here in Hungary. The company did a pretty good job of keeping it's IP range clean apparently. I only had to request whitelisting from 2 spamlists. One being Outlook's. I filled out their form and my mail was delivering a few days later.
I wonder how the ubuntu 22.04 upgrade is progressing though. The EOL of 18.04 is getting pretty close.
I wonder how the ubuntu 22.04 upgrade is progressing though. The EOL of 18.04 is getting pretty close.
my point exactly. was your experience an exercise in futility as the OP wants us to believe?
Not at all. My only problem is that I can never be 100% sure my emails are going through. Shame that some large servers don't send bounce notifications (khm, outlook)
I was with the author up until the end. Why not have have regulation that makes the big company's spend money on processing? The can surely afford it.
If you believe the hype, the Digital Markets Act is that regulation. https://en.wikipedia.org/wiki/Digital_Markets_Act
How about each email recipient has an Absolute Whitelist and messages from addresses or domains on this list are always delivered come hell or high water?
I think the author of the article makes a few mistakes in his
diagnostic.
You should never use a service where you cannot report a problem or where the administrators are out of reach. Of course it includes the "big techs" but they're not alone.
The underlying issue is that most of them delegate their filtering decisions to third parties. And many of them rely on the same centralized IP and domain blacklists.
Because of how the filtering is done, the End User generally never discovers that any filtering is happening. Only the sender may be notified by his mail relay of the delivery failure.
Of course, the blacklisters are not going to put the IP addresses of the big Email Vendors into their lists, if they did, millions of people would be notified of delivery failures with the risk of them discovering who is responsible.
""" Unfortunately, the computing power required to filter millions of emails per minute is huge. That's why the email industry has chosen a shortcut to reduce that cost. """
Even 20 years ago the computing power wasn't a problem. He probably has this impression because he's been using SpamAssassin. The real reason for why they are taking the "shortcut" is carelessness toward their service and users.
The excuse of saying that you should block messages before delivering them because it takes disk space is also heavily promoted by the blacklisters. Indeed, if the message was simply delivered to a Spam folder with the actual reason for which it has been classified as such, users could discover who is responsible for the filtering.
The good news is that there is some success in getting the big email services to remove centralized blacklists.
Another problem is that too many administrators of smaller services are not even aware of their reliance on blacklists. Sometimes this is because they have used an easy installation script for convenience, or because they've copy/pasted a configuration. And of course, there are those who do not understand the ethical implications of doing such a thing or are just foolish.
""" So, starting today, the MX records of my personal domain no longer point to the IP of my personal server. They now point to one of the Big Email Providers. """
This doesn't make any sense. The MX records are for inbound, not outbound, he could have used a different relay for sending mails yet still use his own relay for receiving (perhaps he doesn't know that?). Instead, he switches to a provider that is known for contributing to the problem. This is... disturbing.
You should never use a service where you cannot report a problem or where the administrators are out of reach. Of course it includes the "big techs" but they're not alone.
The underlying issue is that most of them delegate their filtering decisions to third parties. And many of them rely on the same centralized IP and domain blacklists.
Because of how the filtering is done, the End User generally never discovers that any filtering is happening. Only the sender may be notified by his mail relay of the delivery failure.
Of course, the blacklisters are not going to put the IP addresses of the big Email Vendors into their lists, if they did, millions of people would be notified of delivery failures with the risk of them discovering who is responsible.
""" Unfortunately, the computing power required to filter millions of emails per minute is huge. That's why the email industry has chosen a shortcut to reduce that cost. """
Even 20 years ago the computing power wasn't a problem. He probably has this impression because he's been using SpamAssassin. The real reason for why they are taking the "shortcut" is carelessness toward their service and users.
The excuse of saying that you should block messages before delivering them because it takes disk space is also heavily promoted by the blacklisters. Indeed, if the message was simply delivered to a Spam folder with the actual reason for which it has been classified as such, users could discover who is responsible for the filtering.
The good news is that there is some success in getting the big email services to remove centralized blacklists.
Another problem is that too many administrators of smaller services are not even aware of their reliance on blacklists. Sometimes this is because they have used an easy installation script for convenience, or because they've copy/pasted a configuration. And of course, there are those who do not understand the ethical implications of doing such a thing or are just foolish.
""" So, starting today, the MX records of my personal domain no longer point to the IP of my personal server. They now point to one of the Big Email Providers. """
This doesn't make any sense. The MX records are for inbound, not outbound, he could have used a different relay for sending mails yet still use his own relay for receiving (perhaps he doesn't know that?). Instead, he switches to a provider that is known for contributing to the problem. This is... disturbing.
G'Day Guys
I have gone and had a another look at EFF and the issue for me is they have a 'Donate' in the menu. I'm a OG coder and have been giving away my efforts for decades, I believe that $$$ is the root of all evil, so I don't have any spare cash. What about a BC list of IPV4 that is a referral WL, I'm not gr8 at branding and don't believe in marketing, so the best I can come up with is 'BEN - Blockchain Email Net' ???
I pretty much stopped using email altogether once I realised this is what was happening. Maybe 10 years ago now.
Email was really useful if you wanted to send a message or notification to multiple people at the same time, but that got abused so much literally everyone disabled it, at which point email was no longer useful.
People can still send me emails (e.g. for plane tickets), but the chances of me replying to one are nearly zero. Now I send maybe 4 or 5 emails a year and only to people who wont use literally anything else.
Email was really useful if you wanted to send a message or notification to multiple people at the same time, but that got abused so much literally everyone disabled it, at which point email was no longer useful.
People can still send me emails (e.g. for plane tickets), but the chances of me replying to one are nearly zero. Now I send maybe 4 or 5 emails a year and only to people who wont use literally anything else.
Here’s an idea
Make a new protocol for this decade, that isn’t email.
HTTP is supported nearly everywhere SMTP is. Just build something over that, and this time around make sure to avoid SPAM bullshit.
People shouldn’t be able to just message you based on your address. They receive a capability to email you. People can be empowered to give out your capabilities. If a particular such branch leads to spam, you simply cut off that branch and boom, no new user can reach you with that capability anymore. Hashes of Public keys can identify users.
Make a new protocol for this decade, that isn’t email.
HTTP is supported nearly everywhere SMTP is. Just build something over that, and this time around make sure to avoid SPAM bullshit.
People shouldn’t be able to just message you based on your address. They receive a capability to email you. People can be empowered to give out your capabilities. If a particular such branch leads to spam, you simply cut off that branch and boom, no new user can reach you with that capability anymore. Hashes of Public keys can identify users.
https://xkcd.com/927/
The problem isn't the port. The problem is the behaviour.
HTTP itself is on its way out, you should be recommending HTTPS if you're going to go this route.
Increasingly, I find HTTP(S) requests are blocked by various mechanisms, typically CAPTCHAs or similar.
And the problem ultimately with email isn't the network requests, by and large, it's the spam. Reimplement some universally-accessible messaging protocol over another port or service and you'll simply find you've kicked the can and inherited the old problems.
I'm not saying SMTP is faultless (for a rather tongue-in-cheek take, see <https://news.ycombinator.com/item?id=32587346> and responses within that subthread and elsewhere on the post, particularly <https://news.ycombinator.com/item?id=32591880>). But you'll have to do more than just swap port numbers to fix it.
The problem isn't the port. The problem is the behaviour.
HTTP itself is on its way out, you should be recommending HTTPS if you're going to go this route.
Increasingly, I find HTTP(S) requests are blocked by various mechanisms, typically CAPTCHAs or similar.
And the problem ultimately with email isn't the network requests, by and large, it's the spam. Reimplement some universally-accessible messaging protocol over another port or service and you'll simply find you've kicked the can and inherited the old problems.
I'm not saying SMTP is faultless (for a rather tongue-in-cheek take, see <https://news.ycombinator.com/item?id=32587346> and responses within that subthread and elsewhere on the post, particularly <https://news.ycombinator.com/item?id=32591880>). But you'll have to do more than just swap port numbers to fix it.
actually something similar was proposed by djb with email2000.
sadly it got never picked up.
sadly it got never picked up.
There are companies that claim to offer decentralization by hosting them on their own servers. I think we've already lost.
Alright blockchain nerds, now is your time to shine. An actual problem that can be attacked through distributed consensus.
If you insist. I put my suggestion as a reply to another comment in this discussion, specifically here:
https://news.ycombinator.com/item?id=32717921
https://news.ycombinator.com/item?id=32717921
I threw in the towel a few years ago, after 15 years running my own server on top of dan bernstein's qmail.
We need kubernetes level of distributed messaging system to beat email. What’s most popular next option?
Matrix protocol. Can be bridged with existing platforms and protocols, including Facebook Messenger, WhatsApp, and email. 60m+ publicly addressable users, not including EU military, gov, healthcare, emergency services et al.
Is email itself broken? What could replace it, that isn't a "platform" like WhatsApp?
Email is broken. Fix that first so that hosting it isn’t such a terrible chore.
This was the first thing in Google so caveat emptor: https://www.templafy.com/blog/how-many-emails-are-sent-every....
Now for the quote:
"Globally, as of 2019, a staggering 293. 6 billion emails were sent each day and there are currently now over 4 billion email users worldwide."
If we suppose that 95% of that traffic is SPAM, 14,650,000,000 or 14 Billion legitimate emails are sent around the world every day on a planet with 8 billion people and not sure how many of those are connected to email.
Even at 14 billion, email is staggeringly successful, this is why FAANG wants us small time operators who do this as a hobby, for free for ourselves and family ... out of their pool.
Now for the quote:
"Globally, as of 2019, a staggering 293. 6 billion emails were sent each day and there are currently now over 4 billion email users worldwide."
If we suppose that 95% of that traffic is SPAM, 14,650,000,000 or 14 Billion legitimate emails are sent around the world every day on a planet with 8 billion people and not sure how many of those are connected to email.
Even at 14 billion, email is staggeringly successful, this is why FAANG wants us small time operators who do this as a hobby, for free for ourselves and family ... out of their pool.
Why can't we have an open source mail hosting solution that self-updates?
You've actually asked that question before and the answer is still the same: you can't embed "trusted sender status" into source code for a mailserver or a software package. The invisible rules for rejecting email is an emergent external property that exists outside of the software package: https://news.ycombinator.com/item?id=20853157
E.g. Software that "auto-updates" cannot solve the problem of how different participants change their criteria on which ip blocks are "bad".
E.g. Software that "auto-updates" cannot solve the problem of how different participants change their criteria on which ip blocks are "bad".
Good catch :)
But I'm not completely convinced. Sender reputation is made up of at least two parts: the software (rules) used to send the emails, AND the actual emails sent, frequency of emails and number of unique recipients.
If you're a spammer you can still use the same software as everyone else, but your reputation will be bad because of the number of bad emails you sent.
In other words, if everyone used the same software to send emails, then anti-spam systems will have to use other metrics to blacklist people.
But I'm not completely convinced. Sender reputation is made up of at least two parts: the software (rules) used to send the emails, AND the actual emails sent, frequency of emails and number of unique recipients.
If you're a spammer you can still use the same software as everyone else, but your reputation will be bad because of the number of bad emails you sent.
In other words, if everyone used the same software to send emails, then anti-spam systems will have to use other metrics to blacklist people.
>In other words, if everyone used the same software to send emails, then anti-spam systems will have to use other metrics to blacklist people.
Right, and the "other metrics" is what spam heuristics already use now.
In other words, you replied to this author's problem with "self-updating software" but software cannot solve his outgoing email getting rejected/spamholed. It's those "other metrics" that made him give up running his mailserver from home.
Right, and the "other metrics" is what spam heuristics already use now.
In other words, you replied to this author's problem with "self-updating software" but software cannot solve his outgoing email getting rejected/spamholed. It's those "other metrics" that made him give up running his mailserver from home.
Who is any middle man to decide for me what is spam? Send in the regulators!
How to charge people for sending me an email? With hashcash or something?
Hey [0] built a very successful email business in only a couple of years with no deliverability issues. There are hundreds of thousands of happy Hey users who continue to pay for email. So much for anticompetitiveness. There may be a difference between a guy with a server and a real business serving thousands of customers.
[0] hey.com
[0] hey.com
we shold come up with a more universal approach to tackle spam
Nowadays, if you want to build services on top of email, you have to pay an email sending API which has been blessed by others in the industry ...
This concept may sound familiar to you. It's called a racket.
This concept may sound familiar to you. It's called a racket.
seems to me that email transport is by now a fairly robust and universal delivery mechanism at least just above the IP layer, and it's utility decline creates the perfect opportunity to piggyback a new system on top of the existing infrastructure:
high concept: encrypted email with cryptographic postage stamps paid directly to the recipient.
Want me to read your email, pay the postage, and the same vice versa. Who cares what gmail is doing, I don't want them collecting my postage anyway.
this would simultaneously bootstrap a universal micropayment currency
high concept: encrypted email with cryptographic postage stamps paid directly to the recipient.
Want me to read your email, pay the postage, and the same vice versa. Who cares what gmail is doing, I don't want them collecting my postage anyway.
this would simultaneously bootstrap a universal micropayment currency
Seems like we need to write a law (draft a bill).
No, we don't need to do this because more laws will be piled on top and only FAANG will have the lawyers and people required to comply with those laws, forcing more control of the Internet out of our hands, the people, and resting control with the last people on Earth who should have any power whatsoever...politicians.
We need new protocols. Blockchain protocols can somewhat solve the spam problem by charging to send. Receiving addresses can also whitelist or charge to receive.
Why are we still using email?
No, I'm serious, why?
No, I'm serious, why?
First, yes, it sucks. See <https://news.ycombinator.com/item?id=32587346>
But:
- It exists.
- It's ... kind of ... not owned. (Though as TFA notes, this is only vanishingly true any more.) But compared to most alternatives, it is less owned by any one entity.
- It's supported over a wide range of devices, from smartphones to mainframes.
- It has extensive mindshare, use-culture, associated processes, etc.
- It is largely agnostic as to content. Yes, plain text is most native, but through the majick of MIME, et al, it'll manage just about anything.
If you're going to call for email's end, you'd best have a highly viable alternative or replacement in mind.
But:
- It exists.
- It's ... kind of ... not owned. (Though as TFA notes, this is only vanishingly true any more.) But compared to most alternatives, it is less owned by any one entity.
- It's supported over a wide range of devices, from smartphones to mainframes.
- It has extensive mindshare, use-culture, associated processes, etc.
- It is largely agnostic as to content. Yes, plain text is most native, but through the majick of MIME, et al, it'll manage just about anything.
If you're going to call for email's end, you'd best have a highly viable alternative or replacement in mind.
One of the few means of online communication that isn't a proprietary walled garden?
So in the few other means of online communication that aren't proprietary walled gardens, what else we got?
Another compromise between self hosting and using "big tech" email providers is to use smaller email providers. I have been using https://kolabnow.com/ for a five years or so and am a happy customer. Of course it's more expensive, has less bells and whistles, etc, but I'm glad I don't have to participate in the surveillance capitalism machine of big email providers.
how about trying a distributed censorship resistant version like skiff.com?
Sad, but true...
This was a great read. Thanks for posting it here!
I found this line to be especially intriguing:
> Hellbanning everybody except for other big email providers is lazy and conveniently dishonest. It uses spam as a scapegoat to nerf deliverability and stifle competition.
The big tech firms criticized in this article are guilty of these sorts of transgressions in other arenas, as well. It's always been my contention that the "hellbanning" of user-generated content by big media and big tech alike comes from the same motivations. YouTube and CBS alike want to make niche content difficult to consume in order to stifle any competition that might get vaulted up as a result of that niche audience finding the new distribution endpoints. This comes with the added bonus of reducing cost of goods sold, by reducing the firehose of new content to process. Or, as the article puts it:
> Unfortunately, the computing power required to filter millions of emails per minute is huge. That's why the email industry has chosen a shortcut to reduce that cost. The shortcut is to avoid processing some email altogether. Selected email does not either get bounced nor go to spam. That would need processing, which costs money.
I would be very curious to learn if there are any proposed explanation as to why this phenomenon is so commonly spread throughout the big tech space. Do we get the same kind of behavior out of other enormous multi-national firms like oil producers, ocean freight companies, defense contractors, and chemical suppliers?
I found this line to be especially intriguing:
> Hellbanning everybody except for other big email providers is lazy and conveniently dishonest. It uses spam as a scapegoat to nerf deliverability and stifle competition.
The big tech firms criticized in this article are guilty of these sorts of transgressions in other arenas, as well. It's always been my contention that the "hellbanning" of user-generated content by big media and big tech alike comes from the same motivations. YouTube and CBS alike want to make niche content difficult to consume in order to stifle any competition that might get vaulted up as a result of that niche audience finding the new distribution endpoints. This comes with the added bonus of reducing cost of goods sold, by reducing the firehose of new content to process. Or, as the article puts it:
> Unfortunately, the computing power required to filter millions of emails per minute is huge. That's why the email industry has chosen a shortcut to reduce that cost. The shortcut is to avoid processing some email altogether. Selected email does not either get bounced nor go to spam. That would need processing, which costs money.
I would be very curious to learn if there are any proposed explanation as to why this phenomenon is so commonly spread throughout the big tech space. Do we get the same kind of behavior out of other enormous multi-national firms like oil producers, ocean freight companies, defense contractors, and chemical suppliers?
bashblog!
Totally agree with the tone & premise, but
> At some point your IP range is bound to be banned, either by one asshole IP neighbor sending spam, *one of your users being pwned*,
feels like a hint that
> My current email server IP has been managed by me and used exclusively for personal email with zero spam, zero, for the last ten years.
Might not be entirely accurate.
> At some point your IP range is bound to be banned, either by one asshole IP neighbor sending spam, *one of your users being pwned*,
feels like a hint that
> My current email server IP has been managed by me and used exclusively for personal email with zero spam, zero, for the last ten years.
Might not be entirely accurate.
Agree with a sibling comment that many major providers fail to operate the SPF/DKIM/DMARC tools they insist you do.
Each to their own, but ultimately if we don't hold on to the freedom to operate our own mailservers, it will be taken away through inaction. This means doing some things right: DMARC, DKIM, SPF of course, server maintenance, good password policies and of course IP reputation. The best way I can recommend for IP reputation is to use a dedicated provider or VPS provider that disallows things like VPN endpoints, where it is less likely they'll assign an address with a poor reputation. A good provider might also ask you what you intend to host, and you might be able to discuss IP addresses with them.