The Hostile Email Landscape(liminality.xyz)
liminality.xyz
The Hostile Email Landscape
http://liminality.xyz/the-hostile-email-landscape/
250 comments
> suddenly Gmail was connecting to my mail server over an IPv6 interface
Nitpick: when sending email, the sending mail server establishes a connection to the receiving server, not vice versa. The sender is responsible for looking up the domain name's MX record, choosing a hostname from those listed, initiating a connection, establishing an SMTP conversation, and negotiating protocol enhancements like SSL.
That said, I could see a change in behavior like this resulting from action on the receiver's side. Though the sending server makes the choice of how to connect, that choice can be influenced by the presentation of the MX record. I haven't specifically run into this before, but I could see this happening if, for example, the MX record advertised a host with an A record first (IPv4), and then was changed to list a host with only an AAAA record first (IPv6).
There's quite a lot of arcana in the email space that's difficult to manage without careful, deliberate effort. Sending email is harder than it should be, and a lot of that difficulty results from anti-spam or general anti-abuse efforts. If it was easy to bring a server up and begin sending with a new domain name and IP address while successfully reaching the inbox, then spammers would do exactly that. So new identities are put through the wringer to prove they're legitimate. Well-known platforms have an easier time sending email because they can relay traffic from infrastructure that the ISPs already trust; and in turn they maintain that trust by keeping abuse off their platform.
Nitpick: when sending email, the sending mail server establishes a connection to the receiving server, not vice versa. The sender is responsible for looking up the domain name's MX record, choosing a hostname from those listed, initiating a connection, establishing an SMTP conversation, and negotiating protocol enhancements like SSL.
That said, I could see a change in behavior like this resulting from action on the receiver's side. Though the sending server makes the choice of how to connect, that choice can be influenced by the presentation of the MX record. I haven't specifically run into this before, but I could see this happening if, for example, the MX record advertised a host with an A record first (IPv4), and then was changed to list a host with only an AAAA record first (IPv6).
There's quite a lot of arcana in the email space that's difficult to manage without careful, deliberate effort. Sending email is harder than it should be, and a lot of that difficulty results from anti-spam or general anti-abuse efforts. If it was easy to bring a server up and begin sending with a new domain name and IP address while successfully reaching the inbox, then spammers would do exactly that. So new identities are put through the wringer to prove they're legitimate. Well-known platforms have an easier time sending email because they can relay traffic from infrastructure that the ISPs already trust; and in turn they maintain that trust by keeping abuse off their platform.
[deleted]
> suddenly Gmail was connecting to my mail server over an IPv6 interface, and I had never bothered to put the IPv6 block into the SPF record
Whoa, I think you just solved the problem I've been having with my outbound e-mail for the last two months! Thanks so much!
Whoa, I think you just solved the problem I've been having with my outbound e-mail for the last two months! Thanks so much!
To confirm: If my server is ipv4 only, there is nothing special I need to add to my SPF record regarding ipv6 addresses, correct?
You probably should add the v6 representation of your V4 address, in case your ISP adds a 4to6 gateway in front of you at some point down the line.
This one caught me out too, Comcast Business decided to give me an IP6 block, didn't notify me, and my mail server ended up with an IPV6 address. Everything still worked (Yay IP V6 I guess) except that now Google was throwing everything away. The "quick" fix was to tell my mail server not to use IPV6 and the "longer" fix was to update the DNS records according.
> Greylisting still works amazingly well. With a long, long whitelist and greylisting plus DNSBL, I don't even bother running a spam filter, since the little bit of spam and emails from new senders ends up in its own directory as it came from a non-whitelisted sender.
Any good tips on this section in particular? If I'm running my own mail server, how would I get started making sure this is in order?
Any good tips on this section in particular? If I'm running my own mail server, how would I get started making sure this is in order?
I have been running exim4 for years, but I'm in the process of moving to postfix, as postfix is considerably easier to set up all the DKIM, etc., machinery that is now required. Inbound email comes through procmail and is mainly read in emacs (mh-e), which is kind of old fashioned. I have a small script that makes a new email address within my domain for each new use. I sign up for a lot of mailing lists and groups, and my /etc/aliases is more than 5300 lines. I can track if domainA's address starts coming from domainB and disable that address, but that doesn't happen very often, which is a pleasant surprise.
I also have a small script that puts a new sender on my whitelist of sender email addresses. My whitelist is 12000+ lines right now, collected over many years. Procmail sorts to mailing lists and vendor folders, and finally puts things that are not on the whitelist into a "possible spam" folder. From the five or ten items a day, it is easy to spot legitimate emails and I add those to the whitelist. The majority of spam is blocked by the combination of greylisting and DNSBL lists, as the delay of greylisting (ten minutes for me) is enough for them to make the blackhole list, if they happen to ever attempt delivery again.
I was thinking recently that I should be collecting statistics on the use of a lot of those aliases and whitelisted emails, and maybe start garbage collecting my lists.
There are various reputation reports and services that can tell you how your mail is doing in the major ISPs, but a lot of those require higher traffic than a personal or small business generates. There is one service, DMARC[1], that is free and can give you some visibility into how email from your domain is being processed. I put the txt record in my DNS, and Google, Facebook, Comcast, Yahoo, Fastmail, and a few others send me reports about email they have processed from my domain. It's not that interesting at the moment because things are working, but it might help to debug issues if your email was being rejected. At least I see a few spammers are trying to use my domain from their servers.
[1] https://dmarc.org/
I also have a small script that puts a new sender on my whitelist of sender email addresses. My whitelist is 12000+ lines right now, collected over many years. Procmail sorts to mailing lists and vendor folders, and finally puts things that are not on the whitelist into a "possible spam" folder. From the five or ten items a day, it is easy to spot legitimate emails and I add those to the whitelist. The majority of spam is blocked by the combination of greylisting and DNSBL lists, as the delay of greylisting (ten minutes for me) is enough for them to make the blackhole list, if they happen to ever attempt delivery again.
I was thinking recently that I should be collecting statistics on the use of a lot of those aliases and whitelisted emails, and maybe start garbage collecting my lists.
There are various reputation reports and services that can tell you how your mail is doing in the major ISPs, but a lot of those require higher traffic than a personal or small business generates. There is one service, DMARC[1], that is free and can give you some visibility into how email from your domain is being processed. I put the txt record in my DNS, and Google, Facebook, Comcast, Yahoo, Fastmail, and a few others send me reports about email they have processed from my domain. It's not that interesting at the moment because things are working, but it might help to debug issues if your email was being rejected. At least I see a few spammers are trying to use my domain from their servers.
[1] https://dmarc.org/
If you are using emacs, you might want to try the emacs UI for notmuch:
http://notmuchmail.org/
http://notmuchmail.org/
old but valid HOWTO blog post. http://blog.philippheckel.com/2010/01/28/how-to-postfix-as-m...
One "could" have a VPS SMTP server transporting inbound mail to a mail server at home and relay outbound mail through e.g. Amazon SES via the same vps-smtp host. But still.. this is bullshit.
I have a VPS for sending, it is set up as a smarthost to relay from my internal mail server. It is included the SPF record for my domains along with MX records. I receive on my firewall at home, that host is what the MX record points at.
It is very easy to set up.
The full setup for inbound mail is - OpenBSD spamd (greylisting) -> internal postfix
Outbound is - internal postfix -> VPS postfix
It is very easy to set up.
The full setup for inbound mail is - OpenBSD spamd (greylisting) -> internal postfix
Outbound is - internal postfix -> VPS postfix
What kind of bandwith usage do you see? I'm not sure I'd be brave enough to try this anyway, but one thing I worry about is exceeding bandwith quotas and ending up with a large bill.
Also, do you just have one server or a backup as well?
Also, do you just have one server or a backup as well?
I setup my own server recently and could share my experience.
Using postfix + dovecot(imap) + thunderbird. 1 user, set to check mail every 9 minutes. Not even sure if I have IDLE on my server (unless it's on by default).
The SolusVM control panel shows 15-30 bytes/second (in+out) when it's idling. Now add how much your email weights and you got your bandwidth.
Using postfix + dovecot(imap) + thunderbird. 1 user, set to check mail every 9 minutes. Not even sure if I have IDLE on my server (unless it's on by default).
The SolusVM control panel shows 15-30 bytes/second (in+out) when it's idling. Now add how much your email weights and you got your bandwidth.
I've run into similar issues with a similar setup. It's frustrating. You can convince gmail user A to whitelist your messages, and so they'll get through to user A, but gmail user B probably still won't see messages from you unless you tell him to dig them out of the spam trap. And your messages to A might still be classified as spam if they have attachments or hyperlinks in them. (Even if you've been corresponding with A for several years!)
Once upon a time, to send email, you needed to use SMTP. Now, you must use SMTP, from a IP block that isn't categorized as residential, and which has never before had any association with outgoing spam, and you also must implement several ad-hoc identification protocols like SPF and reverse DNS. You should also use a domain name that you've owned for some time and which is not expiring soon. Every system to which you want to send mail will give different weights to all these signals. If they don't like you, their behavior is to report successful delivery and then silently hide messages from their intended recipients.
Spam is no fun, but the present situation is pretty weak too.
Once upon a time, to send email, you needed to use SMTP. Now, you must use SMTP, from a IP block that isn't categorized as residential, and which has never before had any association with outgoing spam, and you also must implement several ad-hoc identification protocols like SPF and reverse DNS. You should also use a domain name that you've owned for some time and which is not expiring soon. Every system to which you want to send mail will give different weights to all these signals. If they don't like you, their behavior is to report successful delivery and then silently hide messages from their intended recipients.
Spam is no fun, but the present situation is pretty weak too.
The email deliverability issues we have had as a legitimate business are insane. Moving to an ESP years ago has helped, but it's far from perfect. I've wondered how a small business without the tech resources could manage this.
For instance, several months back some of our account holders suddenly stopped receiving important account info as well as newsletters from us. Tracked it down to a third party filtering service used by various email providers. Contacted them and they indicated that we were sending to honeypot addresses and/or doing other things that landed us on their list. We refuted all of their assertions (all of our recipients are double out-in, etc.) But , they insisted, as if their processes were infallible and refused to remove us.
Here they were, a third party with whom we had no agreement, yet they were interfering with our ability to do business (and profiting from it). We also had paid advertising in the newsletters per agreements with advertisers. Consulted counsel and the next step was to send a cease and desist, followed by an injunction on the grounds of tortious interference. Then, just as suddenly, our emails started going through again.
So, all of this, just to send email.
Yet, I still have to deal with a ton of spam personally when wading through my quarantine folder for routine false positives. Current processes are harder on the good guys than the spammers.
For instance, several months back some of our account holders suddenly stopped receiving important account info as well as newsletters from us. Tracked it down to a third party filtering service used by various email providers. Contacted them and they indicated that we were sending to honeypot addresses and/or doing other things that landed us on their list. We refuted all of their assertions (all of our recipients are double out-in, etc.) But , they insisted, as if their processes were infallible and refused to remove us.
Here they were, a third party with whom we had no agreement, yet they were interfering with our ability to do business (and profiting from it). We also had paid advertising in the newsletters per agreements with advertisers. Consulted counsel and the next step was to send a cease and desist, followed by an injunction on the grounds of tortious interference. Then, just as suddenly, our emails started going through again.
So, all of this, just to send email.
Yet, I still have to deal with a ton of spam personally when wading through my quarantine folder for routine false positives. Current processes are harder on the good guys than the spammers.
> Here they were, a third party with whom we had no agreement, yet they were interfering with our ability to do business (and profiting from it).
Well... that's an oversimplification. You don't have an agreement, sure, but your recipient has an agreement with them --- your recipient has decided that it's with paying the spam filtering supplier to filter their mail for them.
I realise that this doesn't help you, and frankly they don't sound like they're doing their job very well, but it's important to remember that the recipient chose to use their service.
Well... that's an oversimplification. You don't have an agreement, sure, but your recipient has an agreement with them --- your recipient has decided that it's with paying the spam filtering supplier to filter their mail for them.
I realise that this doesn't help you, and frankly they don't sound like they're doing their job very well, but it's important to remember that the recipient chose to use their service.
No. Our recipients had no knowledge of the third party. Instead, their email providers contracted with the third party for filtering services.
EDIT: Not sure why it would preclude their liability in any case though. Their customers would be paying to have spam blocked, not emails from legitimate companies with whom they've agreed to do business. If they were blocking these demonstrably legit businesses in error and refused to do anything about it, then their "service" is helping no one, but is damaging those businesses.
It is simply a risk inherent to their business model and it is absolutely their responsibility to address it.
EDIT: Not sure why it would preclude their liability in any case though. Their customers would be paying to have spam blocked, not emails from legitimate companies with whom they've agreed to do business. If they were blocking these demonstrably legit businesses in error and refused to do anything about it, then their "service" is helping no one, but is damaging those businesses.
It is simply a risk inherent to their business model and it is absolutely their responsibility to address it.
Right... their email providers, which they are paying to provide email from them. The money chain is still at your recipient's end.
Note that I'm not suggesting you did the wrong thing here --- I don't see anything else you could have done. I'm just saying that pinning the entirety of the blame on the third party provider is wrong.
...
Having read the rest of the thread: I'm sorry to say but the reason why you're having so much difficulty, and the reason why you're getting hostile replies here, is that as an email advertiser people are going to automatically assume you're in the wrong. You are on the edge of an astonishingly dirty industry, and anyone who works in email has been so burnt by the Neptune-sized tidal wave of diarrhoea that is spam is likely to spit in your face before you have the chance to explain that you are not, in fact, a spammer. And chances are that they won't actually care. To a most people, email + advertising = spam.
I really don't have anything to suggest. I have written antispam software, and I've read some of the forums of both sides, and frankly I don't know who's scarier.
Note that I'm not suggesting you did the wrong thing here --- I don't see anything else you could have done. I'm just saying that pinning the entirety of the blame on the third party provider is wrong.
...
Having read the rest of the thread: I'm sorry to say but the reason why you're having so much difficulty, and the reason why you're getting hostile replies here, is that as an email advertiser people are going to automatically assume you're in the wrong. You are on the edge of an astonishingly dirty industry, and anyone who works in email has been so burnt by the Neptune-sized tidal wave of diarrhoea that is spam is likely to spit in your face before you have the chance to explain that you are not, in fact, a spammer. And chances are that they won't actually care. To a most people, email + advertising = spam.
I really don't have anything to suggest. I have written antispam software, and I've read some of the forums of both sides, and frankly I don't know who's scarier.
>* pinning the entirety of the blame on the third party provider is wrong.*
We disagree. It's certainly understandable that the third-party might raise false-positives. However, what is wrong is when a business can demonstrate that it is not engaged in such practices, yet the third-party provider still refuses to cease penalizing them.
So, who else's fault would it be? Mine? The customer? The customer's e-mail provider who contracts with the third-party? Some unrelated spammer who makes the third-party's business really difficult to execute well? I don't think so. The third-party is in the business of profiting from the proper classification of senders. It's their risk and responsibility to fix these problems.
>the reason why you're getting hostile replies here.
Well, I didn't think the replies were particularly hostile. A little misguided maybe, but not hostile.
Anyway, I think you're also jumping to a lot of conclusions and, frankly, not reading very carefully.
Firstly, we're not an "email advertiser" in the sense you're implying. We're a member-based site with a newsletter, into which only members double opt-in out of their interest in our service. The ads I referenced are occasional sponsorships by businesses that are also highly-relevant to our service and members' interests. But, we don't just push ads and we follow all rules regarding proper email management.
Secondly, our content was never the issue anyway. As I mentioned, the issue was that we were falsely claimed to have been sending to honeypot addresses and/or defunct addresses. I enumerated the reasons why we knew this to be in error.
I hate spam as much as anyone. In fact, my reaction to it is probably borderline irrational. But, there are legitimate businesses with legitimate customers between whom e-mail is a mutually beneficial channel. There's nothing "scary" about that. And, my initial point was in response to an ancestor comment, wherein I agreed that certain "anti-spam" practices make it far too difficult for those who use the channel properly and respectfully.
We disagree. It's certainly understandable that the third-party might raise false-positives. However, what is wrong is when a business can demonstrate that it is not engaged in such practices, yet the third-party provider still refuses to cease penalizing them.
So, who else's fault would it be? Mine? The customer? The customer's e-mail provider who contracts with the third-party? Some unrelated spammer who makes the third-party's business really difficult to execute well? I don't think so. The third-party is in the business of profiting from the proper classification of senders. It's their risk and responsibility to fix these problems.
>the reason why you're getting hostile replies here.
Well, I didn't think the replies were particularly hostile. A little misguided maybe, but not hostile.
Anyway, I think you're also jumping to a lot of conclusions and, frankly, not reading very carefully.
Firstly, we're not an "email advertiser" in the sense you're implying. We're a member-based site with a newsletter, into which only members double opt-in out of their interest in our service. The ads I referenced are occasional sponsorships by businesses that are also highly-relevant to our service and members' interests. But, we don't just push ads and we follow all rules regarding proper email management.
Secondly, our content was never the issue anyway. As I mentioned, the issue was that we were falsely claimed to have been sending to honeypot addresses and/or defunct addresses. I enumerated the reasons why we knew this to be in error.
I hate spam as much as anyone. In fact, my reaction to it is probably borderline irrational. But, there are legitimate businesses with legitimate customers between whom e-mail is a mutually beneficial channel. There's nothing "scary" about that. And, my initial point was in response to an ancestor comment, wherein I agreed that certain "anti-spam" practices make it far too difficult for those who use the channel properly and respectfully.
> However, what is wrong is when a business can demonstrate that it is not engaged in such practices, yet the third-party provider still refuses to cease penalizing them.
They don't have any responsibility to help you do business. Comcast and your recipients are not required to accept your email. It's in their interest to do so, of course, because that's what their customers are paying them to do so... but their customers are also paying them not to accept email, because most email is spam, and they don't want to receive that.
Email is not a common carrier medium. You can't demand people accept your traffic. It just doesn't work like that.
> The ads I referenced are occasional sponsorships by businesses[...]
...which means, unfortunately, that people are paying you to send advertisements by email, which means that you're in the email advertising business. Which means that people will automatically assume you're malicious. Which sucks, but...
They don't have any responsibility to help you do business. Comcast and your recipients are not required to accept your email. It's in their interest to do so, of course, because that's what their customers are paying them to do so... but their customers are also paying them not to accept email, because most email is spam, and they don't want to receive that.
Email is not a common carrier medium. You can't demand people accept your traffic. It just doesn't work like that.
> The ads I referenced are occasional sponsorships by businesses[...]
...which means, unfortunately, that people are paying you to send advertisements by email, which means that you're in the email advertising business. Which means that people will automatically assume you're malicious. Which sucks, but...
>They don't have any responsibility to help you do business
No, but in the course of doing their business, this third party cannot wrongfully interfere with our business. It's a key distinction that you are missing.
As to the rest of your comment, you are assuming that the problem was the content (advertising). However, I have stated at least three times on this thread the actual issue; we were erroneously classified as sending to honeypots and/or defunct addresses.
No, but in the course of doing their business, this third party cannot wrongfully interfere with our business. It's a key distinction that you are missing.
As to the rest of your comment, you are assuming that the problem was the content (advertising). However, I have stated at least three times on this thread the actual issue; we were erroneously classified as sending to honeypots and/or defunct addresses.
> No, but in the course of doing their business, this third party cannot wrongfully interfere with our business.
Um... they're an email filtering company, and you're an email sending company. Interfering with your business is their business! That's what Comcast is paying them to do --- Comcast want your business interfered with!
> As to the rest of your comment, you are assuming that the problem was the content (advertising).
I'm afraid you've missed my point here. The content is important, because having advertising in the content means that they will have already written you off as a reputable company. I'm pretty sure the conversation at their end went something like:
A. Hey, Bob, I've just got an email from someone claiming we shouldn't be blocking their email, but they're on the honeypot list.
B. Sigh. Again? Do we have one of their messages?
A. Yup. Here.
B. <spends two seconds skimming message> It's got advertising in it. They're a spammer. Screw 'em.
(It may not even have gone that far. I have no idea what format your newsletters are in, but if they score badly they may not even have bothered to look at them.)
My point is: you may not be a bad guy, but to an awful lot of people in the SMTP industry, you're going to look just like one.
Um... they're an email filtering company, and you're an email sending company. Interfering with your business is their business! That's what Comcast is paying them to do --- Comcast want your business interfered with!
> As to the rest of your comment, you are assuming that the problem was the content (advertising).
I'm afraid you've missed my point here. The content is important, because having advertising in the content means that they will have already written you off as a reputable company. I'm pretty sure the conversation at their end went something like:
A. Hey, Bob, I've just got an email from someone claiming we shouldn't be blocking their email, but they're on the honeypot list.
B. Sigh. Again? Do we have one of their messages?
A. Yup. Here.
B. <spends two seconds skimming message> It's got advertising in it. They're a spammer. Screw 'em.
(It may not even have gone that far. I have no idea what format your newsletters are in, but if they score badly they may not even have bothered to look at them.)
My point is: you may not be a bad guy, but to an awful lot of people in the SMTP industry, you're going to look just like one.
>Um... they're an email filtering company, and...
It's funny when people start their sentences with "Um...", as if they are about to unleash some potent wisdom of which their target is unaware, then instead immediately go on to prove that they have absolutely no idea what they're talking about.
And, now you're literally inventing imaginary conversations.
It's gotten really silly by now and I've repeated nearly half a dozen times on this thread why "points" such as yours are not applicable here. So, hope you don't mind that I'm going to get off this ride now.
Take care.
It's funny when people start their sentences with "Um...", as if they are about to unleash some potent wisdom of which their target is unaware, then instead immediately go on to prove that they have absolutely no idea what they're talking about.
And, now you're literally inventing imaginary conversations.
It's gotten really silly by now and I've repeated nearly half a dozen times on this thread why "points" such as yours are not applicable here. So, hope you don't mind that I'm going to get off this ride now.
Take care.
How about let your members keep missing their emails. If they really want them, encourage them to sort out the problem with their email provider which is apparently failing to do its job? Since the recipients are the ones who choose to receive it, then they're also the ones to be upset when something goes wrong - especially since it's caused by their choice of an unreliable email provider.
I use email for a slightly more spammy purpose (double opt in but mainly to promote sales to existing users) so I wouldn't be able to make this claim if it happened to me - my recipients won't notice if they don't get my email and many won't care. I have to deal with these problems too, usually by manually asking the blacklister to unblacklist me, which they seem to do.
I use email for a slightly more spammy purpose (double opt in but mainly to promote sales to existing users) so I wouldn't be able to make this claim if it happened to me - my recipients won't notice if they don't get my email and many won't care. I have to deal with these problems too, usually by manually asking the blacklister to unblacklist me, which they seem to do.
Unfortunately engaging users in a campaign to encourage their providers to sort it out is a bit of a quagmire. Meantime, our business suffers.
It's a good thought. Just wish it were simpler to pull off.
It's a good thought. Just wish it were simpler to pull off.
What I mean is passively let them take up the problem themselves if they want your newsletters. Since it's something they expect and want, they might choose to pressure their email provider into solving it.
Of course if they don't really want it or just half-heartedly signed up because it seemed like it might be useful at the time then they're not going to make the effort. And I guess that's the situation almost everybody is in when they find they're not receiving emails due to false positive spam filtering. So it's basically a terrible situation for everyone :(
At least we have these magic solutions like MailChip or MailJet (what I use because the free plan fits my usage pattern) so you could turn to something like that if you really had no luck chasing the spam list company.
Of course if they don't really want it or just half-heartedly signed up because it seemed like it might be useful at the time then they're not going to make the effort. And I guess that's the situation almost everybody is in when they find they're not receiving emails due to false positive spam filtering. So it's basically a terrible situation for everyone :(
At least we have these magic solutions like MailChip or MailJet (what I use because the free plan fits my usage pattern) so you could turn to something like that if you really had no luck chasing the spam list company.
In fairness, I've worked with a number of generally legitimate shops who've had exceptionally poor email hygiene.
Sending messages in the millions to domains which have been inactive and unregistered for over a decade, for example, suggests a certain lack of diligence in bounce-tracking or list-cleaning.
The emails had been initially legitimately added, but clients and/or email providers had long since gone out of business. Re-validating email every so often isn't a bad idea. Especially if you're relying on it for other purposes (e.g., account recovery).
Sending messages in the millions to domains which have been inactive and unregistered for over a decade, for example, suggests a certain lack of diligence in bounce-tracking or list-cleaning.
The emails had been initially legitimately added, but clients and/or email providers had long since gone out of business. Re-validating email every so often isn't a bad idea. Especially if you're relying on it for other purposes (e.g., account recovery).
We're clean.
For instance, our ESP detects hard bounces and auto removes them from our lists. We also use Webhooks to update internally on unsubscribes, etc.
For transactional emails, our ESP auto-adds bounces to an outbound block list, so any future attempts to send fail. We only send those to accounts with recent activity anyway, but either way, one bounce stops future sends.
For instance, our ESP detects hard bounces and auto removes them from our lists. We also use Webhooks to update internally on unsubscribes, etc.
For transactional emails, our ESP auto-adds bounces to an outbound block list, so any future attempts to send fail. We only send those to accounts with recent activity anyway, but either way, one bounce stops future sends.
> Consulted counsel and the next step was to send a cease and desist, followed by an injunction on the grounds of tortious interference. Then, just as suddenly, our emails started going through again.
Dangerous path to go down; because so many spammers throw around legal threats too, doing so can get you on a different set of blacklists.
Dangerous path to go down; because so many spammers throw around legal threats too, doing so can get you on a different set of blacklists.
At that point, you're boxed in. You're already on a list. Begging and hoping when your revenue and relationships are under attack is not an option.
Filing for injunctive relief isn't a threat. It's legal action. Unlike spammers, we could easily demonstrate our legitimacy and best practices, etc. They would be compelled to remove us and not retaliate if they are a legit business and care about the integrity of their service for their own reputation and customers.
If, however, that does not describe them, then the outcome would seem to be bleak in any case.
Filing for injunctive relief isn't a threat. It's legal action. Unlike spammers, we could easily demonstrate our legitimacy and best practices, etc. They would be compelled to remove us and not retaliate if they are a legit business and care about the integrity of their service for their own reputation and customers.
If, however, that does not describe them, then the outcome would seem to be bleak in any case.
so... your customers were paying a third party to block you, so rather than working with your customers to figure out what the problem was, you threaten to sue the third party.
That's a rather glib and obtuse way to put it.
I'm an Amazon customer. If Amazon can't reach my gmail address because Google started using some third party filtering service, what exactly would you expect me to do? Even if I had the ability, I certainly don't have the inclination to spend time resolving this issue. I'll go so far as to mark something "not spam", but that's about it. The cost to me of not hearing from Amazon is virtually zero. The cost to Amazon of not being able to communicate with all its customers that use gmail is huge. It's an Amazon problem, not a customer problem.
But GP did figure out what the problem was and took steps to resolve it without involving lawyers before finally issuing the C&D and getting an injunction.
I'm an Amazon customer. If Amazon can't reach my gmail address because Google started using some third party filtering service, what exactly would you expect me to do? Even if I had the ability, I certainly don't have the inclination to spend time resolving this issue. I'll go so far as to mark something "not spam", but that's about it. The cost to me of not hearing from Amazon is virtually zero. The cost to Amazon of not being able to communicate with all its customers that use gmail is huge. It's an Amazon problem, not a customer problem.
But GP did figure out what the problem was and took steps to resolve it without involving lawyers before finally issuing the C&D and getting an injunction.
>The cost to me of not hearing from Amazon is virtually zero. The cost to Amazon of not being able to communicate with all its customers that use gmail is huge. It's an Amazon problem, not a customer problem.
And... this is exactly the point. If the filtering company works for the end user, and the end-user doesn't want the mail (or at least doesn't care if it doesn't get delivered) - then the filtering company should block it.
My point here is that you don't have a fundamental right to my attention. If I'm paying someone to sort my messages for me, and they throw out a message that is important to you, but that I don't really want to see... as far as I am concerned, the filtering company has done it's job.
(obviously, if I want that email, it's another matter entirely - see my other response.)
And... this is exactly the point. If the filtering company works for the end user, and the end-user doesn't want the mail (or at least doesn't care if it doesn't get delivered) - then the filtering company should block it.
My point here is that you don't have a fundamental right to my attention. If I'm paying someone to sort my messages for me, and they throw out a message that is important to you, but that I don't really want to see... as far as I am concerned, the filtering company has done it's job.
(obviously, if I want that email, it's another matter entirely - see my other response.)
> That's a rather glib and obtuse way to put it.
Glib, yes. Obtuse? less so.
>I'm an Amazon customer. If Amazon can't reach my gmail address because Google started using some third party filtering service, what exactly would you expect me to do?
If your free email address doesn't receive mail you want, I expect you might move to a different mail service. I think that if you want anything but automated support out of a free service, you are expecting far too much. Because you still seem to think sticking with your free email provider is the only reasonable course of action, I can only assume that said free provider is meeting your needs; that the automated support is good enough and that you don't mind the odd false positive.
Sure, if you want service, you are going to have to pay a few dollars, but there are plenty of spamfiltering services that are fairly cheap that do have a real person who will fix it if you tell them you aren't getting legitimate mail.
In reality, users consider "legitimate" newsletters to be spam... spending a bunch of time digging through the "legitimate sender" settings to get the mail you want and not the "legitimate newsletters" is often so much work that users just mark the newsletters as spam, and hope their provider will figure it out. Of course, this works really badly when it comes to reputation systems, as the billing stuff and the "newsletters" often come from the same server.
(really, if you must send "newsletters" in addition to your billing, you should do them from a different mailserver and different email address.)
Glib, yes. Obtuse? less so.
>I'm an Amazon customer. If Amazon can't reach my gmail address because Google started using some third party filtering service, what exactly would you expect me to do?
If your free email address doesn't receive mail you want, I expect you might move to a different mail service. I think that if you want anything but automated support out of a free service, you are expecting far too much. Because you still seem to think sticking with your free email provider is the only reasonable course of action, I can only assume that said free provider is meeting your needs; that the automated support is good enough and that you don't mind the odd false positive.
Sure, if you want service, you are going to have to pay a few dollars, but there are plenty of spamfiltering services that are fairly cheap that do have a real person who will fix it if you tell them you aren't getting legitimate mail.
In reality, users consider "legitimate" newsletters to be spam... spending a bunch of time digging through the "legitimate sender" settings to get the mail you want and not the "legitimate newsletters" is often so much work that users just mark the newsletters as spam, and hope their provider will figure it out. Of course, this works really badly when it comes to reputation systems, as the billing stuff and the "newsletters" often come from the same server.
(really, if you must send "newsletters" in addition to your billing, you should do them from a different mailserver and different email address.)
You may be right in theory but way off in practice. MS for example accepts the emails and dumps them. It does not even make it to users' spam folders. So they don't know and we don't know. We only send emails that customers pay us to receive yet email no longer works. This is not how internet supposed to work.
Spam is a major problem and as it is often the case the defensive measures cause just as much, if not more, problems.
>Spam is a major problem and as it is often the case the defensive measures cause just as much, if not more, problems.
more problems for people who send bulk email, sure...
the senders of bulk email vastly overestimate how much of a problem that is for end users.
more problems for people who send bulk email, sure...
the senders of bulk email vastly overestimate how much of a problem that is for end users.
Exactly. I would add though that it can be a customer problem too. For instance, when customers order from Amazon, they expect an order confirmation, and generally find it helpful. Likewise, if customers need to reset passwords, etc.
In fact, we learned of the problem when customers suddenly began to complain that they weren't receiving their password reset emails from us.
In fact, we learned of the problem when customers suddenly began to complain that they weren't receiving their password reset emails from us.
No.
Our customers have email accounts with providers like Comcast. Comcast uses a third-party filtering service to handle spam.
So, our customers had no knowledge of the third-party, and some contacted our support staff to question why they weren't receiving our emails.
And, of course we attempted to identify the problem with the third party. If you read more carefully, you'll find that in my comment. Only after they insisted their classification was correct and refused to remove us from the list did we explore our legal options.
Our customers have email accounts with providers like Comcast. Comcast uses a third-party filtering service to handle spam.
So, our customers had no knowledge of the third-party, and some contacted our support staff to question why they weren't receiving our emails.
And, of course we attempted to identify the problem with the third party. If you read more carefully, you'll find that in my comment. Only after they insisted their classification was correct and refused to remove us from the list did we explore our legal options.
>And, of course we attempted to identify the problem with the third party. If you read more carefully, you'll find that in my comment. Only after they insisted their classification was correct and refused to remove us from the list did we explore our legal options.
My point here is that the recipients of the email and the senders of the email have very different ideas as to what constitutes a "correct evaluation." - and in this case, it's the recipient who matters, so complaints from senders of email should be simply ignored by spam-filtering companies. Now, if the recipients can't complain, that's a problem, but it's not like people don't have a choice of email providers.
My point here is that the recipients of the email and the senders of the email have very different ideas as to what constitutes a "correct evaluation." - and in this case, it's the recipient who matters, so complaints from senders of email should be simply ignored by spam-filtering companies. Now, if the recipients can't complain, that's a problem, but it's not like people don't have a choice of email providers.
You are really missing the point here and you have some key assumptions wrong.
>recipients of the email and the senders of the email have very different ideas
What does that mean? They sign up for our service and expect to get e-mails from us, which they don't receive. We have the same "idea", but the third-party is interfering with that "idea".
>if the recipients can't complain, that's a problem
I will repeat that by definition of the business model, the recipients don't complain to the third-party because they have no knowledge that the third-party even exists.
>but it's not like people don't have a choice of email providers
Even if recipients did know about the third party, do you know the switching costs of e-mail? And, you're suggesting that we should wait until all of our affected members get fed up and change providers (hoping that provider doesn't use the same service, etc.)? All of this, when the problem is obviously being caused by the third-party's error, yet they have no responsibility?
Sorry, but of course the third-party is absolutely compelled to provide some recourse for legitimate senders that are mis-classified. Too hard with all the spammers out there, you say? Sorry. Pick another business.
>recipients of the email and the senders of the email have very different ideas
What does that mean? They sign up for our service and expect to get e-mails from us, which they don't receive. We have the same "idea", but the third-party is interfering with that "idea".
>if the recipients can't complain, that's a problem
I will repeat that by definition of the business model, the recipients don't complain to the third-party because they have no knowledge that the third-party even exists.
>but it's not like people don't have a choice of email providers
Even if recipients did know about the third party, do you know the switching costs of e-mail? And, you're suggesting that we should wait until all of our affected members get fed up and change providers (hoping that provider doesn't use the same service, etc.)? All of this, when the problem is obviously being caused by the third-party's error, yet they have no responsibility?
Sorry, but of course the third-party is absolutely compelled to provide some recourse for legitimate senders that are mis-classified. Too hard with all the spammers out there, you say? Sorry. Pick another business.
>What does that mean? They sign up for our service and expect to get e-mails from us, which they don't receive. We have the same "idea", but the third-party is interfering with that "idea".
That's not quite how it works from the end-user's perspective.
Say I buy a product or service from you. Of course, you send me billing emails, etc. and that's fine.
The problem is that companies take this further and start thinking they have the "right" to send all sorts of garbage to that customer.
Now, they of course let the customer opt out... if the customer is willing to spend ten minutes digging through their interface. But most don't.
I know you love those newsletters; they make you money. But don't kid yourself that your customers do. Many are going to click 'report spam' if you send too many. Even if it's not spam by the definition that marketers use, it's still very clearly unwanted email, and marking it as spam is so much easier than going through the unique-to-you opt-out procedure. And if your list is like most, some of the members signed up years ago (or bought something from you years ago) and long ago forgot doing so, so of course they are going to mark it as spam.
If you want to send email to your customers, you need to understand this.
Note, I'm not saying that you can't send that email; you can. I send email to my customers, and some of it borders on being "newsletters" - I'm just saying that you need to be aware of the cost- that cost is that it will annoy some of your users. Getting your email marked as spam is only one part of that cost.
>Even if recipients did know about the third party, do you know the switching costs of e-mail? And, you're suggesting that we should wait until all of our affected members get fed up and change providers (hoping that provider doesn't use the same service, etc.)? All of this, when the problem is obviously being caused by the third-party's error, yet they have no responsibility?
My point is that third party has no responsibility to you. They have a responsibility to their customers, but none to you.
That's not quite how it works from the end-user's perspective.
Say I buy a product or service from you. Of course, you send me billing emails, etc. and that's fine.
The problem is that companies take this further and start thinking they have the "right" to send all sorts of garbage to that customer.
Now, they of course let the customer opt out... if the customer is willing to spend ten minutes digging through their interface. But most don't.
I know you love those newsletters; they make you money. But don't kid yourself that your customers do. Many are going to click 'report spam' if you send too many. Even if it's not spam by the definition that marketers use, it's still very clearly unwanted email, and marking it as spam is so much easier than going through the unique-to-you opt-out procedure. And if your list is like most, some of the members signed up years ago (or bought something from you years ago) and long ago forgot doing so, so of course they are going to mark it as spam.
If you want to send email to your customers, you need to understand this.
Note, I'm not saying that you can't send that email; you can. I send email to my customers, and some of it borders on being "newsletters" - I'm just saying that you need to be aware of the cost- that cost is that it will annoy some of your users. Getting your email marked as spam is only one part of that cost.
>Even if recipients did know about the third party, do you know the switching costs of e-mail? And, you're suggesting that we should wait until all of our affected members get fed up and change providers (hoping that provider doesn't use the same service, etc.)? All of this, when the problem is obviously being caused by the third-party's error, yet they have no responsibility?
My point is that third party has no responsibility to you. They have a responsibility to their customers, but none to you.
Thanks for the advice. I know you mean well, but none of it applies.
I do agree with you on principle. We've been doing this for over a decade and have learned best practices. We also know what our customers appreciate and are very mindful of content, frequency, etc.
But, you seem to be overlooking a key point, so I will state it for at least the third time: the problem was NOT the content, and it was NOT being marked as spam by recipients. Instead, we were erroneously classified by the service as sending to honeypot addresses and/or defunct addresses. So, your assumptions about our customers, newsletters, etc. are just not relevant here.
>the third party has no responsibility to you. They have a responsibility to their customers...
Their customers complained to us that they weren't receiving our emails, so they weren't serving their customers very well in this instance either.
But, what you really don't seem to be getting is that they do have a responsibility not to harm us. That's why there is a legal cause of action known as "tortious interference". I sincerely don't know how to make that point any clearer. Perhaps researching that term a bit might help.
I do agree with you on principle. We've been doing this for over a decade and have learned best practices. We also know what our customers appreciate and are very mindful of content, frequency, etc.
But, you seem to be overlooking a key point, so I will state it for at least the third time: the problem was NOT the content, and it was NOT being marked as spam by recipients. Instead, we were erroneously classified by the service as sending to honeypot addresses and/or defunct addresses. So, your assumptions about our customers, newsletters, etc. are just not relevant here.
>the third party has no responsibility to you. They have a responsibility to their customers...
Their customers complained to us that they weren't receiving our emails, so they weren't serving their customers very well in this instance either.
But, what you really don't seem to be getting is that they do have a responsibility not to harm us. That's why there is a legal cause of action known as "tortious interference". I sincerely don't know how to make that point any clearer. Perhaps researching that term a bit might help.
>But, what you really don't seem to be getting is that they do have a responsibility not to harm us. That's why there is a legal cause of action known as "tortious interference". I sincerely don't know how to make that point any clearer. Perhaps researching that term a bit might help.
Yes, yes, reading up on that is probably a good idea. I am not a lawyer... but in the case of a mail-filtering company blocking emails that their customers have asked them to block, they are harming the senders of those mails. That is a direct result of what their customers are paying them to do.
Or ad-block programs- really, it's exactly the same thing; what the ad-blocker does for the user does harm the advertisers. There's many, many places in the free market where helping one side of a transaction hurts the other.
That probably means that "tortious interference" is more complex than you are making it out to be. There are a lot of situations where helping one party necessarily hurts another, and helping one party over the other is generally not illegal.
Yes, yes, reading up on that is probably a good idea. I am not a lawyer... but in the case of a mail-filtering company blocking emails that their customers have asked them to block, they are harming the senders of those mails. That is a direct result of what their customers are paying them to do.
Or ad-block programs- really, it's exactly the same thing; what the ad-blocker does for the user does harm the advertisers. There's many, many places in the free market where helping one side of a transaction hurts the other.
That probably means that "tortious interference" is more complex than you are making it out to be. There are a lot of situations where helping one party necessarily hurts another, and helping one party over the other is generally not illegal.
Their customers are not paying them to harm legitimate senders from whom they wish to receive emails. It's unfathomable that you appear to see no dinstinction there.
Neither is it "exactly" nor even remotely the same as ad blockers.
>That probably means that "tortious interference" is more complex than you are making it out to be.
No, it means that your analogies are invalid. Further, perhaps you should look up the term before musing about what it "probably means".
Neither is it "exactly" nor even remotely the same as ad blockers.
>That probably means that "tortious interference" is more complex than you are making it out to be.
No, it means that your analogies are invalid. Further, perhaps you should look up the term before musing about what it "probably means".
[deleted]
http://cr.yp.to/im2000.html
Internet Mail 2000 IM2000 is a project to design a new Internet mail infrastructure around the following concept: Mail storage is the sender's responsibility.
Internet Mail 2000 IM2000 is a project to design a new Internet mail infrastructure around the following concept: Mail storage is the sender's responsibility.
Bernstein is an incredibly smart guy. But he's also an example of how the social stuff matters. His software would run the internet if he put a little more effort into the social/political side of things.
The big problem with IM2000 is that it doesn't solve the real problems. It's focused on the economics of storage of mail, which, yeah, are a thing for a few mail administrators, but generally is considered less important than the spam-filtering problem in general.
I mean, I'm not willing to pay for infinite storage, but I certainly would be willing to pay the cost of receiving 100x my current monthly spam load if I didn't have to read any of it. Maybe even 1000x.
The major benefit of the IM2000 standard is that if you can squash the spammer's mail servers, you squash the mail in question, which actually could change things, because it would mean that spammers would need to control the servers they use to send for longer than they do now.
The big problem with IM2000 is that it doesn't solve the real problems. It's focused on the economics of storage of mail, which, yeah, are a thing for a few mail administrators, but generally is considered less important than the spam-filtering problem in general.
I mean, I'm not willing to pay for infinite storage, but I certainly would be willing to pay the cost of receiving 100x my current monthly spam load if I didn't have to read any of it. Maybe even 1000x.
The major benefit of the IM2000 standard is that if you can squash the spammer's mail servers, you squash the mail in question, which actually could change things, because it would mean that spammers would need to control the servers they use to send for longer than they do now.
That his software does not "run the internet" does not make it any less good. It's there for whoever chooses to use it.
That he focuses on the software instead of pandering probably makes his software better than the alternatives that aim to please even the most foolish of users. At least I think so.
Maybe I interpreted the proposal incorrectly, but I always saw IM2000 (minus the "notifications") as a "pull" solution.
By contrast, conventional email relies on "pushing" spam to the recipient (in practice, a middleman called an "email provider").
A smart IM2000 recipient perhaps would not pull spam from the sender's server.
As such, the spam would never enter the network. It would just sit on the sender's server.
Therefore, IM2000 not only conserves storage but also conserves bandwidth.
That he focuses on the software instead of pandering probably makes his software better than the alternatives that aim to please even the most foolish of users. At least I think so.
Maybe I interpreted the proposal incorrectly, but I always saw IM2000 (minus the "notifications") as a "pull" solution.
By contrast, conventional email relies on "pushing" spam to the recipient (in practice, a middleman called an "email provider").
A smart IM2000 recipient perhaps would not pull spam from the sender's server.
As such, the spam would never enter the network. It would just sit on the sender's server.
Therefore, IM2000 not only conserves storage but also conserves bandwidth.
"A smart IM2000 recipient perhaps would not pull spam from the sender's server."
Based on what criteria? You wouldn't have the message headers/content, so you couldn't filter based on that. All you can filter on is the IP address. But you can already do that with SMTP by just responding with a 5xx code as soon as the sending server connects anyway... Or you could actually fetch the message and then spam filter it. But then, you may as well use SMTP...
What difference does it make which side initiates the TCP connection? If I was a spammer, IM2000 wouldn't concern me one little bit.
Based on what criteria? You wouldn't have the message headers/content, so you couldn't filter based on that. All you can filter on is the IP address. But you can already do that with SMTP by just responding with a 5xx code as soon as the sending server connects anyway... Or you could actually fetch the message and then spam filter it. But then, you may as well use SMTP...
What difference does it make which side initiates the TCP connection? If I was a spammer, IM2000 wouldn't concern me one little bit.
> ... because it would mean that spammers would need to control the servers they use to send for longer than they do now.
This is the reason IM2000 is exciting. Spammers only survive using hit and run tactics. We might see a 99% reduction in total generated spam.
This is the reason IM2000 is exciting. Spammers only survive using hit and run tactics. We might see a 99% reduction in total generated spam.
Really? More than 99% of the junk email in my box is from "legitimate" senders who use their own mailservers. The truth of the matter is that it's already pretty easy to block the hit and run spammers. Most of the junk that gets through is sent by people with the resources to bully the spamfilters into accepting it.
Spam is viagra ads and the like (unsolicited commercial email). Promotional emails from companies you have a relationship with are a different thing. Users should be able to easily filter and ignore promotional email, but email hosting companies should never block it as spam.
SMTP is not where email organization should happen. That's what email clients are for. Blocking DoS attacks and spam makes sense. Blocking Amazon Kindle promotions does not.
SMTP is not where email organization should happen. That's what email clients are for. Blocking DoS attacks and spam makes sense. Blocking Amazon Kindle promotions does not.
>Users should be able to easily filter and ignore promotional email, but email hosting companies should never block it as spam.
See, you still seem to think that it's a negotiation between the senders and the receivers.
My point is that you don't have the right to the end-user's attention. If the end-user wants to hire someone to filter out messages they don't want to read, using whatever criteria for "don't want to read" that they like, this is the user's right.
(more relevant to the discussion, the email that is technically spam is pretty easy to block already, so a solution that requires a lot of social/political change is inferior to the technical measures we have in place.)
See, you still seem to think that it's a negotiation between the senders and the receivers.
My point is that you don't have the right to the end-user's attention. If the end-user wants to hire someone to filter out messages they don't want to read, using whatever criteria for "don't want to read" that they like, this is the user's right.
(more relevant to the discussion, the email that is technically spam is pretty easy to block already, so a solution that requires a lot of social/political change is inferior to the technical measures we have in place.)
Re-read the bit you quoted.
> Users should be able to easily filter and ignore promotional email, but email hosting companies should never block it as spam.
i.e. Yes, it should be the user's control, not the hosting company's, not the sending server's.
> Users should be able to easily filter and ignore promotional email, but email hosting companies should never block it as spam.
i.e. Yes, it should be the user's control, not the hosting company's, not the sending server's.
If it's all the same to you, it seems to me that everybody and their mother has a daily or bi-daily newsletter advertising things I don't care about. I think maybe 15 companies are responsible for the vast majority of mail arriving in my inbox, and I care for none of it. Marketing routinely mistakes quantity for quality, as if yelling your message louder is sufficient for mindshare.
I'm really glad that gmail sorts these emails into a place where I don't have to ignore them, because I used to flag them as spam whether or not I might have accidentally given consent 10 years ago by buying something once. It was making email as useful as physical mail(read: not at all).
Just because I bought something from you once does not mean we have a relationship, just as a one-night stand does not entitle you to ask me to help you move.
I'm really glad that gmail sorts these emails into a place where I don't have to ignore them, because I used to flag them as spam whether or not I might have accidentally given consent 10 years ago by buying something once. It was making email as useful as physical mail(read: not at all).
Just because I bought something from you once does not mean we have a relationship, just as a one-night stand does not entitle you to ask me to help you move.
I used to have your problem, then I clicked on the "unsubscribe" link in those 15 company's emails and now I mostly just get email from real people and robots I need information from.
Spam is anything that clogs my inbox and makes the emails I want to receive harder to find.
But I agree, SMTP is not where email organization should happen. Blocking illegitimate senders at the protocol level is the most that should be done, and the blocking of unwanted content should be pushed into a more user oriented layer. Current structure isn't optimal.
But I agree, SMTP is not where email organization should happen. Blocking illegitimate senders at the protocol level is the most that should be done, and the blocking of unwanted content should be pushed into a more user oriented layer. Current structure isn't optimal.
> Spam is viagra ads and the like (unsolicited commercial email). Promotional emails from companies you have a relationship with are a different thing. Users should be able to easily filter and ignore promotional email, but email hosting companies should never block it as spam.
If you didn't ask my permission to send me those promotional emails, they absolutely are spam and anyone who sends them should have their emails blocked. This rationalization that spamming is ok because the recipient is a customer and there is an unsubscribe mechanism is delusional self-interest.
If you didn't ask my permission to send me those promotional emails, they absolutely are spam and anyone who sends them should have their emails blocked. This rationalization that spamming is ok because the recipient is a customer and there is an unsubscribe mechanism is delusional self-interest.
Its also law. Do Not Call for instance doesn't apply to companies you have a relationship with. That's why some major banks 'affiliate' with 100's of companies, essentially selling the right to spam their customers.
Its screwed up, sure. But its the state we're in right now.
Its screwed up, sure. But its the state we're in right now.
I made no claims about legality. Yes, CAN SPAM means you can spam legally. It doesn't make unsolicited commercial email anything other than spam.
Your inbox != the internet. The majority of email traffic has been spam for a decade now.
99% of them have unsubscribe buttons, too.
You might like http://the.mailman.ninja/
If mail storage is the sender's responsibility, then perhaps senders should be allowed to seed their outgoing messages directly from their personal and home devices, and rely on others who have included their identity in their web-of-trust\address-book to propagate notifications of changes to their outbox, without relying on their ISP to do so.
If the sender did not have at least one personal device connected to the internet 24/7, then they can perhaps purchase storage and bandwidth from a competitive online market, similar to the seedbox market for torrents, without surrendering their online identity or address to a commercial entity.
If the sender did not have at least one personal device connected to the internet 24/7, then they can perhaps purchase storage and bandwidth from a competitive online market, similar to the seedbox market for torrents, without surrendering their online identity or address to a commercial entity.
It would probably be possible to build a system like this on top of https://ipfs.io/, see https://github.com/ipfs/apps/issues/10 for example
Anything which loses immutability loses in my book. Once I get an email it's my own copy, and it never goes away. How I'm reading im2000, the sender can retract or change the message, because they have the copy. I guess if I marked every message as "keep a copy for me" as soon as I read it, it would be OK - hard to do that while not having to read spam just in case I want to keep a copy though.
Detecting moifications can be solved with a hash or signature of the message that is sent out and always kept by the recipient. Doesn't solve the retracting though, or the social aspects of what to do if you actually detect a hash mismatch.
IM2000 is a good idea. However, I would propose at least one additional idea [0]: Linked attachments. That should drastically reduce storage space in a corporate environment.
[0] http://beza1e1.tuxen.de/articles/better_email.html
[0] http://beza1e1.tuxen.de/articles/better_email.html
This is also the crux of the streams concept [1][2] posted here a few months back. I have even started an implementation as a side project.
1 - http://tonsky.me/blog/streams/ 2 - https://news.ycombinator.com/item?id=9829614
1 - http://tonsky.me/blog/streams/ 2 - https://news.ycombinator.com/item?id=9829614
In wich ways is this different and superior to a fee?
People have tried and failed with fee-based email before (e.g. HashCash). I do wonder if the postal service could do something here: every citizen has an address, people pay some pittance to deliver mail to that address, and it's a federal crime to tamper with that email. Physical junk mail is annoying, but nowhere near as bad as spam.
Because spammers can't afford a couple of TB of disk space?
Spammers using botnets could not use hit and run tactics. Spammer would need reliable server that stays up long time and receive requests. One way to fight spam would be to actually retrieve it and throw it into trash. It would be equal to DOS-attack.
There would be more time to determine if the message is spam. If enough people mark it as spam and site is blacklisted or taken down, spam would be removed from everyone.
There would be more time to determine if the message is spam. If enough people mark it as spam and site is blacklisted or taken down, spam would be removed from everyone.
If storage is the sender's responsibility, then there would most likely be no reason for recipients to ever bother downloading unsolicited messages from addresses originating outside of their web of trust in the first place.
You could push spam blocking to the end user and client, and simply let them not download messages originating from unknown origins (similar to phone + Caller ID), without having to worry about dealing with spam in a centralized manner.
You could push spam blocking to the end user and client, and simply let them not download messages originating from unknown origins (similar to phone + Caller ID), without having to worry about dealing with spam in a centralized manner.
The problem is not the few kilobytes of body. The problem is the slot that the notification takes up in my inbox and attention, which IM2000 does nothing to solve.
Well none of this is happening unless we are using a new mail protocol, and if we are using a new protocol we might as well throw in the kitchen sink and give you a new mail transfer agent and mail client application as well.
And in the new application: Address == Public Key, Address book == web of trust.
Your mail agent probably doesn't even bother to check or propagate incoming message notifications from addresses you haven't manually added to your address book (100% trust), or addresses within a certain number of hops from someone you manually added to your address book (friends of friend, boss's employees, etc.). And the mail client could further hide and ignore all of the notifications from below a certain user customizable trust score as well.
And in the new application: Address == Public Key, Address book == web of trust.
Your mail agent probably doesn't even bother to check or propagate incoming message notifications from addresses you haven't manually added to your address book (100% trust), or addresses within a certain number of hops from someone you manually added to your address book (friends of friend, boss's employees, etc.). And the mail client could further hide and ignore all of the notifications from below a certain user customizable trust score as well.
If storage is the sender's responsibility, then not only can they potentially get some of your attention, but also your location, what kind of device(s) you use to get email, when you're active, and precisely what kind of content is enticing you to click and what isn't.
A very tight feedback loop for generating content that will trick you, and hosting it for just long enough to see if it works.
A very tight feedback loop for generating content that will trick you, and hosting it for just long enough to see if it works.
What do you mean? A single message/template sent to a million people would be a couple KB... only the ongoing operational costs of hosting the server... Beyond that, blacklisting would be more affective if there was a pull based email model.
I setup my own email server about 2 years ago and gmail accepted our emails without issues. But outlook/hotmail blocked us for a long time until the second time I submitted an unblock application.
Maybe Google is stricter now. I often get captchas when using google.com now.
Maybe Google is stricter now. I often get captchas when using google.com now.
The problem is not so much the attitude of the big guys. It is that smtp is fundamentally broken. we need a better mail protocol that ensures:
1. Traffic always encrypted and content always signed
2. Guarantee that the sender is who it claims he is
3. Decorrelating the email from the domain, a lot of users are prisoners of their current provider just because the address they gave everyone ends with the provider's domain name, very much like it is very hard to switch bank accounts
4. Ability to provide disposable adresses which can be deleted when spammed
3 and 4 would require a sort of token system
3 and 4 would require a sort of token system
> It is that smtp is fundamentally broken.
Because the SMTP cabal is fanatically opposed to changing it.
The response I had 15 years ago was "It takes 10 years to deploy any serious changes to a protocol".
Uh... so? Does that mean we shouldn't do it?
Apparently, yes. 15 years later, there have been no substantive changes to SMTP.
Repeat after me, SMTP is perfect. It's perfect! Well, there are a few known issues. But it's deployed! It's too hard to change. You're trying to change it? You're a terrible person who doesn't understand SMTP.
I got that for 5 years before giving up and going to more productive lines of work.
Because the SMTP cabal is fanatically opposed to changing it.
The response I had 15 years ago was "It takes 10 years to deploy any serious changes to a protocol".
Uh... so? Does that mean we shouldn't do it?
Apparently, yes. 15 years later, there have been no substantive changes to SMTP.
Repeat after me, SMTP is perfect. It's perfect! Well, there are a few known issues. But it's deployed! It's too hard to change. You're trying to change it? You're a terrible person who doesn't understand SMTP.
I got that for 5 years before giving up and going to more productive lines of work.
Changing SMTP would create a lot of interoperability problems. I agree with this. This is why I concluded that we need a new protocol that solves the problem of SMTP. If it's good it will take over SMTP's market share. That's the way to go. It's like IMAP taking over market share of POP, ...
It's been some years now that I'm thinking on this messaging and spamming problem. It is not a protocol or technical issue. The problem is the abuse of the communication system by spammer.
The worst is that people have to hide their contact address or use semi private messaging systems. This is totally contradictory to the evolution of the communication systems. We have smartphones and can't reliably send a message to any body in the world ? Internet is lacking a solution for that.
It's been some years now that I'm thinking on this messaging and spamming problem. It is not a protocol or technical issue. The problem is the abuse of the communication system by spammer.
The worst is that people have to hide their contact address or use semi private messaging systems. This is totally contradictory to the evolution of the communication systems. We have smartphones and can't reliably send a message to any body in the world ? Internet is lacking a solution for that.
> Decorrelating the email from the domain, a lot of users are prisoners of their current provider just because the address they gave everyone ends with the provider's domain name, very much like it is very hard to switch bank accounts
This one seems completely uninteresting in a world in which $15/year can get you your own domain name, complete with reliable email servers, IMAP, and as many email aliases as you like at that domain. Given that, why create an entire infrastructure to "decouple" email addresses from domains?
This one seems completely uninteresting in a world in which $15/year can get you your own domain name, complete with reliable email servers, IMAP, and as many email aliases as you like at that domain. Given that, why create an entire infrastructure to "decouple" email addresses from domains?
That's $15 a year that some people can't, won't or haven't paid before. If someone decides later on that they wish to do so, they have to set up forwarding and try and convince contacts to use the new e-mail address instead.
Anyone can sign up with a provider and become "prisoner" but not everyone knows how to get out of it. How can you even compare the complexity of someone signing up for Internet service and getting [email protected] for free, and then later having to somehow figure out to pay $15 to set up another domain with E-mail to replace it?
Each individual using a GUID as domain name and another GUID as username for each correspondant is probably the solution. I am not suggesting it will require some fundamental changes. But the one thing that would never work is having to ask users to mess with MX entries themselves. So it take some infrastructure change somehow.
That being said my suggestion doesn't work from a privacy point of view as the domain GUID would become a tatoo and it wouldn't take long for directories mapping GUID to real identities to appear.
That being said my suggestion doesn't work from a privacy point of view as the domain GUID would become a tatoo and it wouldn't take long for directories mapping GUID to real identities to appear.
Maybe something like this?
https://github.com/ssbc https://github.com/ssbc/secure-scuttlebutt https://github.com/ssbc/scuttlebot
https://github.com/ssbc https://github.com/ssbc/secure-scuttlebutt https://github.com/ssbc/scuttlebot
Or djb's IM2000 (later fleshed out by JdeBP): http://homepage.ntlworld.com/jonathan.deboynepollard/Proposa...
That might have since fallen out of favor, I'm unsure.
That might have since fallen out of favor, I'm unsure.
I wrote about the further balkanization of SMTP-based Internet electronic mail, lamented by the headline article here, some ten years ago in http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/sm... .
I am surprised and encouraged to see IM2000 mentioned in this discussion.
As to whether such ideas have fallen out of favour, I offer this observation:
You are using a pull-style electronic communications system right now. Your WWW browser is the originator and receipient UA, and Hacker News is the message store. A copy of a message is only made and sent across the network when you request it with your User Agent. A lot of Internet-mediated communications are not SMTP-based any more. The shift to pull-style systems has, in some ways, actually already happened.
As such, many of the things that people ask about IM2000 are questions that they can answer from their own direct experiences, with a little reflection. How do you deal with senders who alter their messages? Well: how do you the receipient deal with the fact that right here and now I can hit an edit button and modify this comment after people have pulled it for reading it, with no edit history and nothing but the honour system for dealing with letting people know? The fleshing out of IM2000 set out the idea of downloading to folders in the recipient MUA the messages that one wanted to take off the senders' mail stores, by the way.
I am surprised and encouraged to see IM2000 mentioned in this discussion.
As to whether such ideas have fallen out of favour, I offer this observation:
You are using a pull-style electronic communications system right now. Your WWW browser is the originator and receipient UA, and Hacker News is the message store. A copy of a message is only made and sent across the network when you request it with your User Agent. A lot of Internet-mediated communications are not SMTP-based any more. The shift to pull-style systems has, in some ways, actually already happened.
As such, many of the things that people ask about IM2000 are questions that they can answer from their own direct experiences, with a little reflection. How do you deal with senders who alter their messages? Well: how do you the receipient deal with the fact that right here and now I can hit an edit button and modify this comment after people have pulled it for reading it, with no edit history and nothing but the honour system for dealing with letting people know? The fleshing out of IM2000 set out the idea of downloading to folders in the recipient MUA the messages that one wanted to take off the senders' mail stores, by the way.
I agree and disagree. SMTP is bad, but it isn't the primary problem. The problem is social conditioning.
Since the 19th century, postal services have been monopolized by government institutions with a fee levied for storage, processing and transportation costs.
Physical mail delivery in bulk is costly, which makes it more profitable to send directly rather than en-masse.
But SMTP requires no fee. Originally, your ISP would provide you with e-mail in exchange for the fee you paid for your connection. Since the advent of "free" email services, people got used to not paying for mail anymore.
With electronic mail, it's cheap and easy to send in large quantities. Before we never needed to verify who was sending us mail, but now it's the only way we can filter out the noise.
To make electronic mail better we have to make it more expensive. Requiring a payment to send mail would reduce the cost-effectiveness of spam, and I would argue it would make the systems more verifiable. You could even preserve anonymity by having anonymous mail services, as long as you paid to send those mails, say with bitcoin.
Want to run your own mail server? Fine, but relaying will cost you. FedEx doesn't deliver mail from the USPS for free, and so relays shouldn't be expected to pass on mail without transportation costs.
Since the 19th century, postal services have been monopolized by government institutions with a fee levied for storage, processing and transportation costs.
Physical mail delivery in bulk is costly, which makes it more profitable to send directly rather than en-masse.
But SMTP requires no fee. Originally, your ISP would provide you with e-mail in exchange for the fee you paid for your connection. Since the advent of "free" email services, people got used to not paying for mail anymore.
With electronic mail, it's cheap and easy to send in large quantities. Before we never needed to verify who was sending us mail, but now it's the only way we can filter out the noise.
To make electronic mail better we have to make it more expensive. Requiring a payment to send mail would reduce the cost-effectiveness of spam, and I would argue it would make the systems more verifiable. You could even preserve anonymity by having anonymous mail services, as long as you paid to send those mails, say with bitcoin.
Want to run your own mail server? Fine, but relaying will cost you. FedEx doesn't deliver mail from the USPS for free, and so relays shouldn't be expected to pass on mail without transportation costs.
"3 and 4 would require a sort of token system"
Not to get all handwavey, but I think this is why some people are super excited about bitcoin becoming 'part of the internet'. There are definitely some areas where we need a concept of identity & trust, and bitcoin seems like one of the first truly distributed ways of doing it.
Not to get all handwavey, but I think this is why some people are super excited about bitcoin becoming 'part of the internet'. There are definitely some areas where we need a concept of identity & trust, and bitcoin seems like one of the first truly distributed ways of doing it.
The thing is while the Internet is all excited about bitcoin, everyone else has (kind of) stopped using e-mail for communication. It's all phone based now. E-mail is just for the things that doesn't have an app. And maybe resetting your password. Unfortunately, the "geek" age is out in favor of the startup age.
That may be the case for personal communications (well, not me, I am old school) but I don't think it the case at all for professional communications.
True, although now shared with cloud tools, at least intra-company. In some markets, like China, they use messaging for business. It's better in some ways e.g. you get a response faster.
I doubt it. No-one wants to pay to send an email. Even if they did, it'd involve waiting 10+ minutes for a payment to go through and limit the global email rate to a maximum of 5 messages per second. Bitcoin is not a good fit.
Not to pay per email, but to pay per email address. If you aren't a spammer, how many email addresses do you need? I don't think asking for a registration fee akin to a domain name is unreasonable.
You know, I've been wondering about this. I wrote about an email token system here [0], let me know what you think.
[0] https://roj.is/blog/a-different-approach-to-online-account-m...
[0] https://roj.is/blog/a-different-approach-to-online-account-m...
We had a better system back in the day: X.400.
Howso?
The X protocols have typically struck me as grossly overdesigned and complicated.
The X protocols have typically struck me as grossly overdesigned and complicated.
More overcomplicated than the complex system of headers and whitelists trying to bolt some basic security and authenticity features onto SMTP?
The latter didn't win because it was "better" just because it was easier to implement. Like a car with no brakes or seatbelts.
The latter didn't win because it was "better" just because it was easier to implement. Like a car with no brakes or seatbelts.
Why the X protocols didn’t win is up for debate, but a good reason why they shouldn’t win, ever, is that, aside from being complicated and over-engineered, they are incredibly centralized in their design.
NB: that doesn't answer my question at all. I didn't ask what the deficiencies of X protocol alternatives were. But for the specific advantages of X.400 and related standards.
Many of the issues we're running into with online systems, particularly those relating to quality and reputation (spam, collaborative filtering or content rating as on HN or Reddit, etc.) have strong analogs in real-world social spaces. And there were real-world mechanisms for dealing with these.
For a new businessman or professional setting out in the world, pre-Internet, "establishing your name" was a requirement. These days the concept's often referred to as "creating a personal brand", but the reality was pretty straightforward: how does an unknown quantity become a known quantity?
A common method was the professional or social introduction. This is still practiced, where a third-party _matchmaker_ will introduce two parties. The matchmaker usually knows both, and can vouch for the newcomer and smooth the path for introductions with the established party. Essentially, the matchmaker stakes _their_ reputation by speaking for another.
Lawrence Lessig describes a similar concept in his book Code and Other Laws of Cypberspace, in a passage describing a physical messaging system, the Yale Wall. This was a board onto which messages could be posted, with the proviso that that they be signed. Unsigned messages would be posted from time to time, effectively presenting an anonymous viewpoint. Removal wasn't instantaneous or automatic, but rather, at some point prior to garbage collection, another individual could review the piece, and, if they felt it warranted posting, sign for it. They weren't registering themselves as author, but as vouching for the merits of the viewpoint -- not necessarily agreement.
A messaging system in which a new peer might be able to indicate that, hey, peers X, Y, and Z, with established reputations can vouch for me, and for which those peers could confirm their endorsement, might address the "how to build reputation" issues of new mailservers.
I've also found that even large mail systems will frequently have some procedure for getting at least provisionally vetted, effectively a workfactor cost to coming online. Though the process absolutely could be improved.
For a new businessman or professional setting out in the world, pre-Internet, "establishing your name" was a requirement. These days the concept's often referred to as "creating a personal brand", but the reality was pretty straightforward: how does an unknown quantity become a known quantity?
A common method was the professional or social introduction. This is still practiced, where a third-party _matchmaker_ will introduce two parties. The matchmaker usually knows both, and can vouch for the newcomer and smooth the path for introductions with the established party. Essentially, the matchmaker stakes _their_ reputation by speaking for another.
Lawrence Lessig describes a similar concept in his book Code and Other Laws of Cypberspace, in a passage describing a physical messaging system, the Yale Wall. This was a board onto which messages could be posted, with the proviso that that they be signed. Unsigned messages would be posted from time to time, effectively presenting an anonymous viewpoint. Removal wasn't instantaneous or automatic, but rather, at some point prior to garbage collection, another individual could review the piece, and, if they felt it warranted posting, sign for it. They weren't registering themselves as author, but as vouching for the merits of the viewpoint -- not necessarily agreement.
A messaging system in which a new peer might be able to indicate that, hey, peers X, Y, and Z, with established reputations can vouch for me, and for which those peers could confirm their endorsement, might address the "how to build reputation" issues of new mailservers.
I've also found that even large mail systems will frequently have some procedure for getting at least provisionally vetted, effectively a workfactor cost to coming online. Though the process absolutely could be improved.
It bothers me that this entire thread is about technology. This is anticompetitive behavior by a small group of oligopolists, plain and simple.
The big providers need to build an accessible system for entering the email market or face tremendous fines from worldwide governments. There are plenty ways imaginable to do this, especially if you allow the process to involve real people and paperwork.
It shouldn't matter whether this is simple negligence or conscious work to keep out competition. The end result is that it's made nigh on impossible by a few market leaders to enter said market.
The big providers need to build an accessible system for entering the email market or face tremendous fines from worldwide governments. There are plenty ways imaginable to do this, especially if you allow the process to involve real people and paperwork.
It shouldn't matter whether this is simple negligence or conscious work to keep out competition. The end result is that it's made nigh on impossible by a few market leaders to enter said market.
I feel like this is a solvable problem without making any changes to email whatsoever. The problem is the email recipient hosts are suspicious of the sender (as opposed to the message itself being suspicious). So the solution is to have a standardized way for senders to acquire an instantaneous reputation by tying their real-world identity to it (which lets them be held accountable if they do spam), and perhaps by throwing some money at it too. If there was some company that did identity checks, similar to how EV certificates are given out, then that ties your real-world identity to it (this could in fact be done by literally requiring an EV certificate for the hostname of the sender). This company could also take a decent-sized deposit (so you're staking money on not being a spammer) and hold it in trust for a set amount of time. Once the time has passed, and you've sent enough emails for recipients to draw meaningful conclusions, if you have in fact not spammed, then you get your deposit back (minus a service fee). Then all the big email hosts would pay this company to query it about senders the host doesn't already trust, and similarly they'd report any spam from these hosts back to the service.
Heck, this doesn't even have to be a new company. A big host like Google could just start offering this service anyway, as a way to simplify their own handling of unknown senders, although I'd feel more comfortable if this was done by someone else.
Heck, this doesn't even have to be a new company. A big host like Google could just start offering this service anyway, as a way to simplify their own handling of unknown senders, although I'd feel more comfortable if this was done by someone else.
The core problem to this is that reputation is valuable. Google or Twitter or Facebook or any of the other big web presences could offer a method to determine "reputation" of any of their users, and a way to canonically tie your identity to that reputation, but they'd be undermining their own business model. They want you to stay in their network, because the safety of interacting with only reputable users is the big selling point of their walled-garden social networks over open networks like email
What do you mean by "stay in their network", when it comes to sending emails? Google is not in the business of being a paid mass email sender for businesses. What business email they handle is specifically for being the email host for a business's internal email, not sending their newsletter / marketing / account notifications / whatever else.
You mean extortion. Give us money or we will drop your mail.
Here is an almost related story: I am in italy and was waiting for an email from the local Apple store to tell me my macbook's repair was complete. After 6 days waiting, I called the store. They said the computer had been ready for a few days.
I checked the spam folder. Gmail had plopped the apple email into the spam folder. Gmail's reason was that the email was in a foreign language, Italian. Didn't matter that the previous 2 weeks all my google.com searches were done from Italy.
I checked the spam folder. Gmail had plopped the apple email into the spam folder. Gmail's reason was that the email was in a foreign language, Italian. Didn't matter that the previous 2 weeks all my google.com searches were done from Italy.
Google's killed access to various accounts of mine at different times for:
1. Accessing an account one time over Tor. Subsequent reconnect attempts from the same device I'd previously used, same IP, etc., failed. Took a couple of weeks to get back.
2. Accessing accounts from different IPs while travelling. Ultimately ending up at the same IP as one of the email accounts I'd previously corresponded heavily with.
3. Accessing accounts from new device(s). Old (Android) devices could access email but not Web-based account recovery tools.
"Who are you" is the most expensive question in technology. No matter how you get it wrong, you're fucked.
https://www.reddit.com/r/dredmorbius/comments/3mo7l6/that_go...
1. Accessing an account one time over Tor. Subsequent reconnect attempts from the same device I'd previously used, same IP, etc., failed. Took a couple of weeks to get back.
2. Accessing accounts from different IPs while travelling. Ultimately ending up at the same IP as one of the email accounts I'd previously corresponded heavily with.
3. Accessing accounts from new device(s). Old (Android) devices could access email but not Web-based account recovery tools.
"Who are you" is the most expensive question in technology. No matter how you get it wrong, you're fucked.
https://www.reddit.com/r/dredmorbius/comments/3mo7l6/that_go...
I was once disappointed not to hear back from a Google recruiter after an interview. Turns out it was in my Gmail spam folder. Along with any other @google.com address.
>This isn't how the internet is supposed to work.
The email architecture was started back when it was a smaller network of researchers at universities, governments, etc. Everybody basically trusted each other.
Once the "internet" is available to the general public and commercial interests, it becomes vulnerable to the "bad actors" problem (e.g. spam abuse). That's why we have the inevitable situation today of a few entities (e.g. gmail, hotmail) being "trusted", and random residential SMTP servers run by homeowners being "untrusted".
I haven't seen a realistic de-centralized trust proposal for email. Even if a proposal is theoretically sound, what incentive is there for other big players to adopt it?
The email architecture was started back when it was a smaller network of researchers at universities, governments, etc. Everybody basically trusted each other.
Once the "internet" is available to the general public and commercial interests, it becomes vulnerable to the "bad actors" problem (e.g. spam abuse). That's why we have the inevitable situation today of a few entities (e.g. gmail, hotmail) being "trusted", and random residential SMTP servers run by homeowners being "untrusted".
I haven't seen a realistic de-centralized trust proposal for email. Even if a proposal is theoretically sound, what incentive is there for other big players to adopt it?
Combining a notify/pull system with a requirement for a valid domain certificate from a pre-approved list of CA's, similar to those already in the browsers, would go a step farther... increasing the costs for operating a a badly acting domain only to be blacklisted relatively quickly.
Unfortunately, that would be less than decentralized, but it may turn out to be the best option in combating spam.
I've just opted to pay for sendgrid for my small hobby BBS server's outbound email, because it's easier than setting up an appropriate outbound system myself, and as TFA points out, even then odds are you'll be bitbucketted before you even start.
Unfortunately, that would be less than decentralized, but it may turn out to be the best option in combating spam.
I've just opted to pay for sendgrid for my small hobby BBS server's outbound email, because it's easier than setting up an appropriate outbound system myself, and as TFA points out, even then odds are you'll be bitbucketted before you even start.
How do you solve the problem of "cold call" with a trusting system ? How does one build up trust with a new account ? Spammers could create million accounts at once and fake trust build up between them. That's why I'm not convinced that trust will solve the abuse of messaging. A messaging application must allow "cold calls". Relying on trust built up by others is not a reliable system because it can be abused.
>How do you solve the problem of "cold call" with a trusting system ? [...] That's why I'm not convinced that trust will solve the abuse of messaging.
You can't remove "trust" or "reputation" from a viable messaging system. The concept of trust always exists whether informally or formally.
When the internet was a small private network between universities and government, how did a new unknown scientist "cold-call" an email? It was not an issue. The scientist just sent the cold email. There was already implicit trust within the system to allow that scenario. Everybody trusted that [email protected] didn't send a million emails about Nigerian money transfers to [email protected].
Back then, email users didn't need "spam filters". The trust was informally created because the institutional "system" outside of the email system vetted email users. (The "system" being the hiring procedures of SomeUniversity, DepartmentOfDefense, etc.)
All those gates for reputation are gone when we expose that simplistic email architecture to the general public. Trust doesn't go away. You now must recreate trust at a massive scale.
>Spammers could create million accounts at once and fake trust build up between them.
That's not the type of "trust" people are talking about in this thread. That type of incestuous "trust" between bad actors won't work because it doesn't have an initial authority (good actor) to "bless" them which starts an acceptable "chain of trust". The concept is similar to Certificate Authorities. We trust Chrome because we trust Google Inc; and in turn, Google Inc trusts Verisign; and in turn, Verisign EV-certificate processing trusts government offices that register businesses.
You can't remove "trust" or "reputation" from a viable messaging system. The concept of trust always exists whether informally or formally.
When the internet was a small private network between universities and government, how did a new unknown scientist "cold-call" an email? It was not an issue. The scientist just sent the cold email. There was already implicit trust within the system to allow that scenario. Everybody trusted that [email protected] didn't send a million emails about Nigerian money transfers to [email protected].
Back then, email users didn't need "spam filters". The trust was informally created because the institutional "system" outside of the email system vetted email users. (The "system" being the hiring procedures of SomeUniversity, DepartmentOfDefense, etc.)
All those gates for reputation are gone when we expose that simplistic email architecture to the general public. Trust doesn't go away. You now must recreate trust at a massive scale.
>Spammers could create million accounts at once and fake trust build up between them.
That's not the type of "trust" people are talking about in this thread. That type of incestuous "trust" between bad actors won't work because it doesn't have an initial authority (good actor) to "bless" them which starts an acceptable "chain of trust". The concept is similar to Certificate Authorities. We trust Chrome because we trust Google Inc; and in turn, Google Inc trusts Verisign; and in turn, Verisign EV-certificate processing trusts government offices that register businesses.
Ok my assumption of trust credit working model was inaccurate. Could you give an example how trust could be used to control abuse ? If a user is given initial trust credit, how can one prevent spammers to create million of such "trusted" accounts and let them be banned after the hit and run spamming ? That model us used in messaging systems like Twitter but has its limitation. But is known to have its limit. Or did I still got it wrong ?
>, how can one prevent spammers to create million of such "trusted" accounts
Notice what you did there? You characterized spam accounts as "trusted" even though you didn't specify why or how they'd get trusted in the first place. There's still a gap in your thinking of how the origination of real (not fake) trust happens.
It's like a 1970s email user asking, "how does the system prevent a 'trusted' Nigerian money scammer from spamming everybody?" Well, why would Department of Defense hire a Nigerian scammer? Would a UniversityOfWhatever hire a Nigerian scammer? No. In that scenario, the trust begins with the institution. For the Nigerian scammer to be trusted, somebody authoritative has to "bless" him.
>Could you give an example how trust could be used to control abuse ?
Well, we're still using trust right now to help control abuse. Today, reputable corporations and also millions of honest users (that's us) "trust" Google Inc, Microsoft, Yahoo, FastMail, etc. Spam on a massive scale does not originate from gmail/hotmail/etc because those corporations shut them down.
The issue is that it's centralized trust concentrated in a few big players instead of de-centralized trust. That's the thorny problem that's very hard to solve. All the clever decentralized proposals (sender pays for storage, outbound payment bonds, cpu-proof-of-work, etc) are basically roundabout proxies for recreating "trust" on a bigger scale. I don't know what the final answer will be. I just know that trust/reputation is an unavoidable part of a new solution.
Notice what you did there? You characterized spam accounts as "trusted" even though you didn't specify why or how they'd get trusted in the first place. There's still a gap in your thinking of how the origination of real (not fake) trust happens.
It's like a 1970s email user asking, "how does the system prevent a 'trusted' Nigerian money scammer from spamming everybody?" Well, why would Department of Defense hire a Nigerian scammer? Would a UniversityOfWhatever hire a Nigerian scammer? No. In that scenario, the trust begins with the institution. For the Nigerian scammer to be trusted, somebody authoritative has to "bless" him.
>Could you give an example how trust could be used to control abuse ?
Well, we're still using trust right now to help control abuse. Today, reputable corporations and also millions of honest users (that's us) "trust" Google Inc, Microsoft, Yahoo, FastMail, etc. Spam on a massive scale does not originate from gmail/hotmail/etc because those corporations shut them down.
The issue is that it's centralized trust concentrated in a few big players instead of de-centralized trust. That's the thorny problem that's very hard to solve. All the clever decentralized proposals (sender pays for storage, outbound payment bonds, cpu-proof-of-work, etc) are basically roundabout proxies for recreating "trust" on a bigger scale. I don't know what the final answer will be. I just know that trust/reputation is an unavoidable part of a new solution.
This is a true story:
I rent some servers from the Rackspace cloud for personal use. I have my own sites on these machines, and my own email servers.
Meanwhile, I have a day job, and lately it has been consuming 12 hours a day. We missed a deadline and we have all been working like crazy to catch up. I have fallen behind reading my personal email.
Roughly a month ago, my friends who use Gmail stopped getting my email. Or rather, they did not know I was sending them email, because all of my email to them was going to spam.
After a few weeks, I finally had a free weekend to catch up on my personal life, so I did some investigations. Turns Rackspace had switched over to IP6 in a way that impacted my email. I did not have a Sender Policy Framework for IP6, only IP4.
It's likely that Rackspace sent me an email about this, though I never read it because I was busy.
This was easy to fix: I added a SPF for IP6.
However, these kinds of issues do make it harder to maintain a personal email server. Its tough for us to keep up with the changes.
I rent some servers from the Rackspace cloud for personal use. I have my own sites on these machines, and my own email servers.
Meanwhile, I have a day job, and lately it has been consuming 12 hours a day. We missed a deadline and we have all been working like crazy to catch up. I have fallen behind reading my personal email.
Roughly a month ago, my friends who use Gmail stopped getting my email. Or rather, they did not know I was sending them email, because all of my email to them was going to spam.
After a few weeks, I finally had a free weekend to catch up on my personal life, so I did some investigations. Turns Rackspace had switched over to IP6 in a way that impacted my email. I did not have a Sender Policy Framework for IP6, only IP4.
It's likely that Rackspace sent me an email about this, though I never read it because I was busy.
This was easy to fix: I added a SPF for IP6.
However, these kinds of issues do make it harder to maintain a personal email server. Its tough for us to keep up with the changes.
Drinking game for this thread. Start at the top of the comments:
* every time someone implies spam filtering is a massive conspiracy against "the little guy", drink
* every time someone suggests Bitcoin be used for something, drink
* every time a solution utterly fails to account for compromised end-user machines sending spam, drink
* every time a comment can be summed up as "assuming a PKI exists, this problem is trivially solved", CHUG THE REST OF THE BOTTLE
* every time someone implies spam filtering is a massive conspiracy against "the little guy", drink
* every time someone suggests Bitcoin be used for something, drink
* every time a solution utterly fails to account for compromised end-user machines sending spam, drink
* every time a comment can be summed up as "assuming a PKI exists, this problem is trivially solved", CHUG THE REST OF THE BOTTLE
LOL
That reminds me of an old classic: http://craphound.com/spamsolutions.txt
There are a lot of obvious problems with email.
* It's hard to run your own mail server, as described in the original post
* Your address is tied to your provider
* The standards involved in delivering mail (SMTP, POP, IMAP, etc) are complex, layered with hacks (SPF, STARTTLS, etc), hard to implement correctly, hard to secure.
* The standards that govern the actual body of email (multipart MIME, a 90s-vintage subset of "HTML" and "CSS") are even worse
But despite all those flaws, email is built on ubiquitous open standards running on a federated infrastructure.
It's the best thing we have.
Proprietary standards and centralized designs like AIM, ICQ, MSN, FB Messenger, Slack, and Twitter DMs come and go.
Email is forever, and is more useful than all of those combined. The flipside is that it's very hard to "fix".
That reminds me of an old classic: http://craphound.com/spamsolutions.txt
There are a lot of obvious problems with email.
* It's hard to run your own mail server, as described in the original post
* Your address is tied to your provider
* The standards involved in delivering mail (SMTP, POP, IMAP, etc) are complex, layered with hacks (SPF, STARTTLS, etc), hard to implement correctly, hard to secure.
* The standards that govern the actual body of email (multipart MIME, a 90s-vintage subset of "HTML" and "CSS") are even worse
But despite all those flaws, email is built on ubiquitous open standards running on a federated infrastructure.
It's the best thing we have.
Proprietary standards and centralized designs like AIM, ICQ, MSN, FB Messenger, Slack, and Twitter DMs come and go.
Email is forever, and is more useful than all of those combined. The flipside is that it's very hard to "fix".
> This isn't how the internet is supposed to work. As we continue to consolidate on a few big mail services, it's only going to become more difficult to start new servers.
And this is exactly the reason I setup my own mail server. I'm only 1 man, but I hope more people will do so with time, thus requiring the "big ones" to work on better algorithms for filtering and not base it on reputation.
And this is exactly the reason I setup my own mail server. I'm only 1 man, but I hope more people will do so with time, thus requiring the "big ones" to work on better algorithms for filtering and not base it on reputation.
Create an email network where is would cost a penny to send email. It would be payed into bitcoin wallet of folks maintaining infrastructure.
Every email would be digitally signed and encrypted. Certificate with keys would connected to email address (and bitcoin wallet).
Spam would die.
Go build it please.
Every email would be digitally signed and encrypted. Certificate with keys would connected to email address (and bitcoin wallet).
Spam would die.
Go build it please.
As adamrt said, this won't kill off spam. It might reduce it a bit, though.
I keep thinking instead about something that works more like a refundable bond:
To send a message to you, I have to include some amount. (I don't know how much. Not a penny, though. Maybe $10 or $100.) If you want to keep receiving messages from me afterward, you return the bond. If not, you keep the money.
And after a couple of minutes of Googling, I just found a name for schemes like this: Attention Bonds.
I keep thinking instead about something that works more like a refundable bond:
To send a message to you, I have to include some amount. (I don't know how much. Not a penny, though. Maybe $10 or $100.) If you want to keep receiving messages from me afterward, you return the bond. If not, you keep the money.
And after a couple of minutes of Googling, I just found a name for schemes like this: Attention Bonds.
I think that is an interesting idea, but my first reaction is that it wouldn't end spam. In the US there is a much larger cost associated with sending physical spam in the US (stamps). But there seems to be unlimited physical spam still.
Maybe it would just become more targeted?
Maybe it would just become more targeted?
Probably not end spam completely, but reduce it to a point it wouldn't be noticable.
In any case, I would love to only send/receive encrypted/signed email. I don't think the problems are technical.
In any case, I would love to only send/receive encrypted/signed email. I don't think the problems are technical.
Please read
http://craphound.com/spamsolutions.txt
and check all boxes that apply.
http://craphound.com/spamsolutions.txt
and check all boxes that apply.
Those reasons are pretty stale. It may not work, but not for any of those reasons. I'd still be in.
Thats actually a good idea. You don't even have to make it a fixed value. The receiver uses how much money was paid as a (spam) signal.
The bitcoin transaction can just go to /dev/null with a hash of the message as data, so don't have to pay somebody ( ;) ) and you cannot track who sent data to whom via tracking the bitcoin transactions.
The bitcoin transaction can just go to /dev/null with a hash of the message as data, so don't have to pay somebody ( ;) ) and you cannot track who sent data to whom via tracking the bitcoin transactions.
The minute you say bitcoin is the moment you cut out 99% of the population; unless it is in the background and one does not have to interact with it directly.
Other than that, I agree paying to send will reduce spam.
But also look at your postal mail box. There is arguably more spam there than in your digital inbox, and that one costs (stamps).
Other than that, I agree paying to send will reduce spam.
But also look at your postal mail box. There is arguably more spam there than in your digital inbox, and that one costs (stamps).
Yes, but if the email is digitally signed I can be certain who sent it (at least the address) and, unlike postal mail, I can electronically sort it. Email from folks I know would receive the highest priority.
Are folks afraid of bitcoin? What can be worse than just having a 16 digit number printed on a card -- that is a 1000x more vulnerable.
Are folks afraid of bitcoin? What can be worse than just having a 16 digit number printed on a card -- that is a 1000x more vulnerable.
Are folks afraid of bitcoin?
You are part of the tiny percentage of the tiny percentage of the population that understands bitcoin.
You are part of the tiny percentage of the tiny percentage of the population that understands bitcoin.
I get far more spam than legitimate mail in the US Postal mail.
Me too. Take the contents from one spam envelope and place it in the return envelope of another spam. Keep 'em churning.
Ask Bill Gates : http://www.cnet.com/forums/discussions/bill-gates-proposes-c...
Is the c|net link broken? I remember Bill Gates bringing this up -- I would to like read what the issues would be with technology today.
[deleted]
I have a bit of experience with running email servers. I can't really say that I had similar encounters.
In my experience if you get blocked by big mail providers it's almost always due to some reason. What's tricky is that it may be hard to tell what exactly is wrong, because they won't necessarily tell you (or not in an easy way).
Some advice what I'd do to try to find out what's going on:
1. Take a sent example mail that is like the blocked one (but obviously one that reached its target destination) with all headers and run it through spamassassin. Don't just look if it hit the spam score (then you did something terribly wrong), look at each individual rule that spamassassin hit. They might give you a clue. A proper mail usually shouldn't hit any or very few positive spamassassin rules.
2. Check your IP at a service like valli where you can query multiple DNS black lists. If it is on any blacklist try to find out how you can be delisted. There are some rogue blacklists that make it impossible to be delisted at all, you may ignore them (google for them, their behavior is well documented), but these shouldn't be more than 1 or 2. As already said by other commenters, don't forget IPv6.
3. Read whatever error message you can get your hands on. If you're blocked on the SMTP level read the error message. If your message got sorted into a spam folder look at all the headers. If the provider blocking you has some online docs about their spam filtering read that. If they have some sort of service for mail ISPs where you can sign up to get warnings sign up there.
Of course also the obvious stuff. If you do anything that is mass mailing you are in extra danger. Make sure that you allow people to unsubscribe easily, don't ignore manual attempts by them to unsubscribe ("I want to get off this mailing list") and delete invalid mail addresses.
In my experience if you get blocked by big mail providers it's almost always due to some reason. What's tricky is that it may be hard to tell what exactly is wrong, because they won't necessarily tell you (or not in an easy way).
Some advice what I'd do to try to find out what's going on:
1. Take a sent example mail that is like the blocked one (but obviously one that reached its target destination) with all headers and run it through spamassassin. Don't just look if it hit the spam score (then you did something terribly wrong), look at each individual rule that spamassassin hit. They might give you a clue. A proper mail usually shouldn't hit any or very few positive spamassassin rules.
2. Check your IP at a service like valli where you can query multiple DNS black lists. If it is on any blacklist try to find out how you can be delisted. There are some rogue blacklists that make it impossible to be delisted at all, you may ignore them (google for them, their behavior is well documented), but these shouldn't be more than 1 or 2. As already said by other commenters, don't forget IPv6.
3. Read whatever error message you can get your hands on. If you're blocked on the SMTP level read the error message. If your message got sorted into a spam folder look at all the headers. If the provider blocking you has some online docs about their spam filtering read that. If they have some sort of service for mail ISPs where you can sign up to get warnings sign up there.
Of course also the obvious stuff. If you do anything that is mass mailing you are in extra danger. Make sure that you allow people to unsubscribe easily, don't ignore manual attempts by them to unsubscribe ("I want to get off this mailing list") and delete invalid mail addresses.
I was thinking about this a while ago and have been meaning to write it up and post it somewhere, so I guess this is as good a time as any.
Hashcash (also known as the precursor to Bitcoin) was proposed to solve this problem in 1997:
https://en.wikipedia.org/wiki/Hashcash
The trouble with it is that it requires computation for each sent message, which is bad for senders with low resource devices or legitimate mailing lists. I want to propose a variant.
Instead of creating an expensive hash against the message, create an even more expensive hash against the (sender TLS certificate, receiver domain name) pair. This implies using TLS but it works just as well even if the certificate is self-signed. Then each mail server only has to generate a hash once per recipient domain, ever. Every message that mail server sends to that domain is tagged with that hash. A legitimate mail server will have already computed hashes for all the domains its users regularly correspond with and rarely if ever need to do any more expensive computations.
If spammers do the same thing then the receiving server can mark all messages sent with that hash as spam. So there is a highly disproportionate cost to spammers (even if they have more computing power) because to avoid that they have to continuously generate expensive new hashes. Which can be made arbitrarily expensive because legitimate servers only need to do it once. And a new hash is much less valuable to a spammer than a domain name or IP address is today because each hash can only be used against one recipient domain. The required amount of computation can be set by the receiving server so domains with more users can require more computation.
If a legitimate mail server is compromised by a spammer then it will have to generate all new hashes (because the spammer will presumably immediately ruin the reputation of the compromised ones), but the reputation of the legitimate sender's email domains is unharmed because the reputation is tied to the hash computation, not the sender's domain name(s).
And adding support to mail servers would require no configuration whatsoever. You install server version N+1 and it starts tagging outgoing messages with hashes that receiving servers can verify.
So here's the traditional form: http://craphound.com/spamsolutions.txt
How'd I do?
Hashcash (also known as the precursor to Bitcoin) was proposed to solve this problem in 1997:
https://en.wikipedia.org/wiki/Hashcash
The trouble with it is that it requires computation for each sent message, which is bad for senders with low resource devices or legitimate mailing lists. I want to propose a variant.
Instead of creating an expensive hash against the message, create an even more expensive hash against the (sender TLS certificate, receiver domain name) pair. This implies using TLS but it works just as well even if the certificate is self-signed. Then each mail server only has to generate a hash once per recipient domain, ever. Every message that mail server sends to that domain is tagged with that hash. A legitimate mail server will have already computed hashes for all the domains its users regularly correspond with and rarely if ever need to do any more expensive computations.
If spammers do the same thing then the receiving server can mark all messages sent with that hash as spam. So there is a highly disproportionate cost to spammers (even if they have more computing power) because to avoid that they have to continuously generate expensive new hashes. Which can be made arbitrarily expensive because legitimate servers only need to do it once. And a new hash is much less valuable to a spammer than a domain name or IP address is today because each hash can only be used against one recipient domain. The required amount of computation can be set by the receiving server so domains with more users can require more computation.
If a legitimate mail server is compromised by a spammer then it will have to generate all new hashes (because the spammer will presumably immediately ruin the reputation of the compromised ones), but the reputation of the legitimate sender's email domains is unharmed because the reputation is tied to the hash computation, not the sender's domain name(s).
And adding support to mail servers would require no configuration whatsoever. You install server version N+1 and it starts tagging outgoing messages with hashes that receiving servers can verify.
So here's the traditional form: http://craphound.com/spamsolutions.txt
How'd I do?
Sounds like a plan!
With a memory-bound proof-of-work system like my Cuckoo Cycle, computing the hash could require the use of more than 4GB of memory for over 5 minutes on a 20-thread server, thus preventing the use of botnets for avoiding the expense.
With a memory-bound proof-of-work system like my Cuckoo Cycle, computing the hash could require the use of more than 4GB of memory for over 5 minutes on a 20-thread server, thus preventing the use of botnets for avoiding the expense.
Not sure Internet providers or google will be enthusiast about this proposal. This is equivalent as asking to pay to register a new destination address. In this case the mail relay owner has to pay. It would be more fair to ask users who send mail to pay to register a new destination address. Google and mail relay provider would be more happy with this variant.
It's very friendly to large providers. The amount of computation needed to send to a domain is the same regardless of how many emails you send so large providers can amortize it over more emails. And large providers generally operate their own infrastructure, which implies they already have a lot of idle computing capacity during off peak hours, and "off peak" is staggered across regions.
But even if some large providers refuse to hash outgoing mail, they can still verify incoming mail. The interesting benefit is that a small email provider can burn some dollars in computation once to avoid having their mail discarded by a large provider's spam filter indefinitely.
But even if some large providers refuse to hash outgoing mail, they can still verify incoming mail. The interesting benefit is that a small email provider can burn some dollars in computation once to avoid having their mail discarded by a large provider's spam filter indefinitely.
I sometimes see similar tales of woe, and I can only say that this does not match my experience. I’ve done this many times, you set up the mail server, configure DNS correctly (including reverse lookup), and that’s it. Never had problems being blacklisted or mail getting classified as spam.
I suspect that people having trouble are sending a lot of mail, like “newletters”, etc. But I can’t prove this hypothesis.
I suspect that people having trouble are sending a lot of mail, like “newletters”, etc. But I can’t prove this hypothesis.
Mail reputations are very real and pose an issue with mail servers. I had a gaming server for years and had many issues with Gmail, Microsoft and Yahoo, to name a few, filtering or blocking emails. Just last week, I setup a mail server using mail in a box on a new server I spun up on Digital Ocean using an IP that was on no blacklists and still had issues with sending emails to various Gmail subscribers.
Even when I used to work at HostGator and handled many of their abuse issues, they had many issues with being blacklisted just because RBLs didn't recognize new HostGator IPs or the rate of email being sent from their new gateways.
So, at least with my personal and professional experience, I can attest to the issues with using self-hosted mail servers.
Even when I used to work at HostGator and handled many of their abuse issues, they had many issues with being blacklisted just because RBLs didn't recognize new HostGator IPs or the rate of email being sent from their new gateways.
So, at least with my personal and professional experience, I can attest to the issues with using self-hosted mail servers.
New IPs are often old IPs that have been used for other purposes before, and often blacklisted. Especially IPs on easily spun-up and spun-down hosts like DO. On DO, after every VPS I've removed, and then started up another a week later, it has a different IP. It would be easy for a spammer/phisher/whoever to abuse this quick address re-allocation, but hurts regular users.
One more point of data from my experience: I have never set up mail servers at hosting providers in remote datacenters, only local servers in my own server rooms which I could physically touch. I guess it’s quite possible that Gmail, etc. have figured out by now where all the IP blocks of hosting providers are, and are very suspicious of them.
Ack. Same here. I had a mail server colo'd with Peer1 in Canada since mid 2000's, which I moved to a European colo few months ago. I use it to periodically send newsletters to few thousand people, so it's not exactly low-volume.
The one and only issue was with some AT&T block list - and judging by their supporting forums they end up with a boatload of false positives - I filed a complain with them and they removed the block few days after.
I suspect that OP ended up getting an IP from a netblock that has lots of transient customers, which included some spammers. It's not his server's reputation that was weak, it's the netblock's rep that was bad.
The one and only issue was with some AT&T block list - and judging by their supporting forums they end up with a boatload of false positives - I filed a complain with them and they removed the block few days after.
I suspect that OP ended up getting an IP from a netblock that has lots of transient customers, which included some spammers. It's not his server's reputation that was weak, it's the netblock's rep that was bad.
I've had similar issues to those described in the article. Especially "outlook.com accepted my email, but discarded it."
When people buy my book, I my server sends them literally one single email containing a link to the PDF they have purchased. And that's it.
Google delivers my email fine, in most cases. But hotmail/outlook was just deleting them most of the time for over a year.
I know I'm just a single data point, but I swear I'm not a newsletter!
When people buy my book, I my server sends them literally one single email containing a link to the PDF they have purchased. And that's it.
Google delivers my email fine, in most cases. But hotmail/outlook was just deleting them most of the time for over a year.
I know I'm just a single data point, but I swear I'm not a newsletter!
Exchange is much more discerning of email structure than Google servers.
That means that, if you have any configuration mistake, an Exchange server will reject your email. But if you have a good reputation, Gmail will deliver it anyway.
That means that, if you have any configuration mistake, an Exchange server will reject your email. But if you have a good reputation, Gmail will deliver it anyway.
Exchange servers weren't "rejecting" my email. They were giving me SMTP 250 (Queued mail for delivery) then silently failing to deliver it.
Hotmail sender support kept saying "I do not see anything offhand that would be preventing your mail from reaching our customers from this IP"
Also, my emails were and are text/plain.
I'm sure Exchange does reject malformed emails, but that wasn't my problem.
Hotmail sender support kept saying "I do not see anything offhand that would be preventing your mail from reaching our customers from this IP"
Also, my emails were and are text/plain.
I'm sure Exchange does reject malformed emails, but that wasn't my problem.
My experience with a malconfigured server interfacing Exchange is that Exchanges pretends everything is ok while receiving (with a 250 response at the end), and may sometimes notify its user (with a bounce message) while sending, but may also not.
But, of course, YMMV. And I have no experience at all sending email to Hotmail. I just thought it was worth checking your server settings.
But, of course, YMMV. And I have no experience at all sending email to Hotmail. I just thought it was worth checking your server settings.
I think I might be able to offer another explanation.
People talk about the reputation of their individual IPs, but networks matter a lot too. DigitalOcean for instance is an occasional source of spam; Linode is very rarely; 1and1 is a HUGE headache; and so on.
I have some custom software in place that temporarily drops entire netblocks for abuse, and I did that to keep up with the spam filtering offered by larger services. With so many spammers now switching between cheap VPS hosts, it was getting impossible to do single-IP-only bans.
Ever since the software was put into place the amount of spam actually making it into users mailboxes has fallen off a cliff.
You have to be very very careful about which network you choose to host your mail server from.
People talk about the reputation of their individual IPs, but networks matter a lot too. DigitalOcean for instance is an occasional source of spam; Linode is very rarely; 1and1 is a HUGE headache; and so on.
I have some custom software in place that temporarily drops entire netblocks for abuse, and I did that to keep up with the spam filtering offered by larger services. With so many spammers now switching between cheap VPS hosts, it was getting impossible to do single-IP-only bans.
Ever since the software was put into place the amount of spam actually making it into users mailboxes has fallen off a cliff.
You have to be very very careful about which network you choose to host your mail server from.
I was actually hosting on Linode (UK). My hunch was that they would be one of the more reputable networks; I've run small mail servers on them before (2+ years ago) without any problems.
It's a pity there's no easy way (that I know of) to look up the reputation for an entire network block. I'm glad to hear your experience with Linode matches mine.
It's a pity there's no easy way (that I know of) to look up the reputation for an entire network block. I'm glad to hear your experience with Linode matches mine.
It's different from one AS to another. I've spent a lot of time maintaining reputation for my IP at a certain major ISP (no bulk email) and I'm still getting blocked from Google and others.
Then I got a VPS at Linode (for some other purpose), installed Postfix and emails were just working - no SPF or even reverse DNS set up.
IP reputation IS a thing.
Then I got a VPS at Linode (for some other purpose), installed Postfix and emails were just working - no SPF or even reverse DNS set up.
IP reputation IS a thing.
OP, I don't know if you are reading the comments here but in case you do: Don't get discouraged so quickly.
The reason this is happening is as the blurb from the MS postmaster help page: Your IP doesn't have a reputation yet.
The reason these rules are in place aren't about email monopoly, it's about spam. If anybody could setup a SMTP server and start firing off large amounts of mail, spam would be even more endemic than today.
You can configure your server perfectly, but that doesn't mean much, since it's your IP that's the problem.
If you have legit objectives, it's a pain in the ass for sure. But you are not the only one having this problem, and there's a solution for it.
All the big email service providers (ESP's) like Neolane, Exact Target, Mailchimp, Campaign monitor etc share this problem when they onboard a new client, who requires their own IP.
Deliverability is a surprisingly deep, technical topic, and all major ESP's have entire teams of specialists working on this.
If you want to make such a service as Fastmail, you need to get really into deliverability. It's not a walk in the park, but it's not impossible either.
I'm not a specialist in this particular area myself, so I can't give you that much specific advice. I've just worked elbow to elbow with a lot of these guys, so I know what kind of challenges they work with.
One thing I know for sure is really important, is the "warming up" of IP's. Basically the IP you are sending from needs to accumulate some reputation over a period of time, typically a month or two.
If you send out reasonably small amounts of mail to email addresses that exists and the recipients does not explicitly report you for junk mail, your IP get whitelisted and you will get a much higher delivery rate.
There's no quick fix unfortunately, and email reputation is hard to gain and fast to lose.
But it certainly can be done. You sound very competent on the server side of things, so to get your fastmail-like service up, I think it's just a matter of a bit more persistence and studying deliverability as a technical subject.
Hope this helps.
The reason this is happening is as the blurb from the MS postmaster help page: Your IP doesn't have a reputation yet.
The reason these rules are in place aren't about email monopoly, it's about spam. If anybody could setup a SMTP server and start firing off large amounts of mail, spam would be even more endemic than today.
You can configure your server perfectly, but that doesn't mean much, since it's your IP that's the problem.
If you have legit objectives, it's a pain in the ass for sure. But you are not the only one having this problem, and there's a solution for it.
All the big email service providers (ESP's) like Neolane, Exact Target, Mailchimp, Campaign monitor etc share this problem when they onboard a new client, who requires their own IP.
Deliverability is a surprisingly deep, technical topic, and all major ESP's have entire teams of specialists working on this.
If you want to make such a service as Fastmail, you need to get really into deliverability. It's not a walk in the park, but it's not impossible either.
I'm not a specialist in this particular area myself, so I can't give you that much specific advice. I've just worked elbow to elbow with a lot of these guys, so I know what kind of challenges they work with.
One thing I know for sure is really important, is the "warming up" of IP's. Basically the IP you are sending from needs to accumulate some reputation over a period of time, typically a month or two.
If you send out reasonably small amounts of mail to email addresses that exists and the recipients does not explicitly report you for junk mail, your IP get whitelisted and you will get a much higher delivery rate.
There's no quick fix unfortunately, and email reputation is hard to gain and fast to lose.
But it certainly can be done. You sound very competent on the server side of things, so to get your fastmail-like service up, I think it's just a matter of a bit more persistence and studying deliverability as a technical subject.
Hope this helps.
Thanks for the encouragement! I might take another shot at it sometime. I had this server running for about 3 months with just my personal mail, sending probably 1 email per day on average. It's possible slightly higher volume would do a better job of "warming up" the IP.
I self-hosted from about 2007-2008 and 2011-2013, and had nowhere near as much trouble with deliverability. It came as a bit of a surprise how much more difficult it is these days.
I self-hosted from about 2007-2008 and 2011-2013, and had nowhere near as much trouble with deliverability. It came as a bit of a surprise how much more difficult it is these days.
If you try again, try to see if you can get a client with a decent sized mailing list. Warming up is usually done with progressively larger sendouts. So you'd start with maybe 500 sendouts, then a thousand, then 3000 etc.
But anyway yeah the barrier to entry has risen really much. It's much harder these days than just a couple of years ago, and it probably won't get easier either!
But anyway yeah the barrier to entry has risen really much. It's much harder these days than just a couple of years ago, and it probably won't get easier either!
Email has been the my last hold out from switch away from gapps completely. I don't want to have to deal with any of this, especially as I do business communication with clients via it. Email wasn't suppose to be like this, and there has to be a better way to enable non-giants to successfully deliver email.
Joke answer: the blockchain!
Real answer: It comes down to trust (duh). But, how do we manage trust online? How do we manage trust in real life? Real life trust is through association of groups. But, groups online are meaningless. A "gmail user" doesn't belong to a community, they belong to The Nation of Google.
How do we break down online identities into manageable, trustable communities? How do we bootstrap new communities into the system so we don't end up with a pre-selected list of blessed communities while others languish in obscurity?
Open questions, no answers, and if you say Facebook you have to run 20 laps and give me 1000 pushups.
Real answer: It comes down to trust (duh). But, how do we manage trust online? How do we manage trust in real life? Real life trust is through association of groups. But, groups online are meaningless. A "gmail user" doesn't belong to a community, they belong to The Nation of Google.
How do we break down online identities into manageable, trustable communities? How do we bootstrap new communities into the system so we don't end up with a pre-selected list of blessed communities while others languish in obscurity?
Open questions, no answers, and if you say Facebook you have to run 20 laps and give me 1000 pushups.
You joke about the blockchain, but HashCash (a similar technique to Bitcoin's Blockchain-difficulty (not the metric itself, but how it's used)) was originally conceived as a means of making email computationally costly to send. The main issue was that spammers would use botnets while normal people would be stuck taking a while sending emails (iirc).
Personally, I would like everyone to have their own rsa or ecc key. While not as expensive as HashCash, it does require more computational effort to send an encrypted email. (This precludes web portals from sending mail, but user convince seems to always trump security! :( ). Still not a great solution.
Web of Trusts are an interesting idea, but fails for the same reason not everyone has a public encryption key: bad ui, complex idea, and no one cares. I honestly think this is the way forward. If someone isn't a few hops from me, they probably don't have any business contacting me anyway -- if they really want to, a.k.a. read my blog and want to ask me a question, they can use HashCash or something else expensive to make the initial contact and slowly build up trust/reputation as we exchange emails.
In the end, everyone is always told they need to check their spam folder for false positives anyway. Maybe we should all just say "screw it" and read all our email. If some combination of the sender and subject doesn't make us think it's legit, then we can delete it.
Maybe we keep spam filters, but stop calling it a "Spam Box" and treat it like trash. Maybe we start calling it "Unknown Box" and treat it with a little skepticism instead of a dumpster.
Personally, I would like everyone to have their own rsa or ecc key. While not as expensive as HashCash, it does require more computational effort to send an encrypted email. (This precludes web portals from sending mail, but user convince seems to always trump security! :( ). Still not a great solution.
Web of Trusts are an interesting idea, but fails for the same reason not everyone has a public encryption key: bad ui, complex idea, and no one cares. I honestly think this is the way forward. If someone isn't a few hops from me, they probably don't have any business contacting me anyway -- if they really want to, a.k.a. read my blog and want to ask me a question, they can use HashCash or something else expensive to make the initial contact and slowly build up trust/reputation as we exchange emails.
In the end, everyone is always told they need to check their spam folder for false positives anyway. Maybe we should all just say "screw it" and read all our email. If some combination of the sender and subject doesn't make us think it's legit, then we can delete it.
Maybe we keep spam filters, but stop calling it a "Spam Box" and treat it like trash. Maybe we start calling it "Unknown Box" and treat it with a little skepticism instead of a dumpster.
With regards to HashCash, the explanations I've seen didn't persuade me (the counter arguments most often cited have been made by some researchers in some IMHO very weak papers). To me it seems that the real reason why it's not being used is that ISPs would have to allocate resources, which would make them less competitive (who wants to offer free email then pay for a lot of infrastructure, or deal with the mess explaining that users have to pay for better delivery). The CPU cost could be moved to the clients, but somehow no IMAP based mail clients seem to ever have implemented it (chicken and egg problem?), and JavaScript has not been an efficient way to do it.
Also, I guess the big ISPs have an easy enough time (in their view?) analyzing trust, which is probably a better solution in general if applicable. The problem is that it's easier to judge (newly created) internal users (accounts) than external ones (random IPs).
I would think improving the mail system would be a research topic, but perhaps not interesting enough from the point of view of researchers (would it be best done in social sciences or computer science?, and there are many real-world variables, taking time to collect and verify, and working out simulations to predict possible remedies).
Also, I guess the big ISPs have an easy enough time (in their view?) analyzing trust, which is probably a better solution in general if applicable. The problem is that it's easier to judge (newly created) internal users (accounts) than external ones (random IPs).
I would think improving the mail system would be a research topic, but perhaps not interesting enough from the point of view of researchers (would it be best done in social sciences or computer science?, and there are many real-world variables, taking time to collect and verify, and working out simulations to predict possible remedies).
This largely true and hugely disappointing. Our startup (https://portal.cloud) is making it possible for non-hackers to self-host their own email servers. It works very well for the most part, but we have had to explain to a number of them why their email sometimes gets bounced by the big proprietary cloud services.
All of our users have their own domain names, IP addresses, SPF records, and correctly configured (and up to date) Postfix SMTP servers. There is absolutely no excuse for not always delivering their email, and yet this is what the big companies do.
All of our users have their own domain names, IP addresses, SPF records, and correctly configured (and up to date) Postfix SMTP servers. There is absolutely no excuse for not always delivering their email, and yet this is what the big companies do.
Edit: They said they checked the spam lists but the ones that I've listed below may be helpful to others.
OP might have had an IP previously used by spammers or in a spam block. (See edit).
Important to check the IP before using it if it's been given to you by, say, the VPS Provider or the upstream. To see what, if any, reputation it has.
Here are two tools to use:
http://multirbl.valli.org/
http://mxtoolbox.com/SuperTool.aspx
And there are a bunch of similar tools.
OP might have had an IP previously used by spammers or in a spam block. (See edit).
Important to check the IP before using it if it's been given to you by, say, the VPS Provider or the upstream. To see what, if any, reputation it has.
Here are two tools to use:
http://multirbl.valli.org/
http://mxtoolbox.com/SuperTool.aspx
And there are a bunch of similar tools.
Outlook.com is possibly the worst offender at present:
1. They set SPF policies of "-all" on customer domains. These customers have no idea about SPF and often use 3rd party software to send account statements that never get delivered. This breaks email for their customers and for everyone else. They should set "~all" by default as Google Apps does.
2. They claim not to, but they definitely block entire IP ranges rather than a single IP address. A noisy neighbour in the same datacenter as you can get you blacklisted. Use http://www.senderbase.org/lookup/ip to monitor your neighbours and report them to your hosting company. If you spot a bad neighbour, you will no doubt see your own IP blacklisted at Outlook.com's https://postmaster.live.com/snds/ipStatus.aspx within a few hours.
3. They are well known for accepting mail and then silently dropping it, without bouncing sender or receiver and without putting it into a spam folder, even when the sender is a longtime sender to that receiver and when the sender is sending personal anticipated email. They should either reject at the SMTP transaction (sending a bounce later is bad practice and leads to backscatter) or else deliver to a spam folder. Silently dropping email breaks email for everyone.
The sad thing is that many businesses are switching to Office 365 and Outlook.com.
1. They set SPF policies of "-all" on customer domains. These customers have no idea about SPF and often use 3rd party software to send account statements that never get delivered. This breaks email for their customers and for everyone else. They should set "~all" by default as Google Apps does.
2. They claim not to, but they definitely block entire IP ranges rather than a single IP address. A noisy neighbour in the same datacenter as you can get you blacklisted. Use http://www.senderbase.org/lookup/ip to monitor your neighbours and report them to your hosting company. If you spot a bad neighbour, you will no doubt see your own IP blacklisted at Outlook.com's https://postmaster.live.com/snds/ipStatus.aspx within a few hours.
3. They are well known for accepting mail and then silently dropping it, without bouncing sender or receiver and without putting it into a spam folder, even when the sender is a longtime sender to that receiver and when the sender is sending personal anticipated email. They should either reject at the SMTP transaction (sending a bounce later is bad practice and leads to backscatter) or else deliver to a spam folder. Silently dropping email breaks email for everyone.
The sad thing is that many businesses are switching to Office 365 and Outlook.com.
This is indeed worrying. I've been running my own E-mail servers for the last 20 years or so and even though my problems weren't as severe, I did run into a few cases where the "big ones" were at least delaying my E-mail.
But this kind of problem invariably arises when we go from a fragmented Internet with lots of small hosts/providers to an Internet of several walled gardens, run by the big guys.
The real answer is to offer a reasonable self-hosting competitor to GMail.
But this kind of problem invariably arises when we go from a fragmented Internet with lots of small hosts/providers to an Internet of several walled gardens, run by the big guys.
The real answer is to offer a reasonable self-hosting competitor to GMail.
I have self-hosted my mailserver for a long time and started to get problems a couple of years ago. The main issue is corporate networks running McAfee's "MxLogic" product that claim to bounce my mail and tell me so, but then go on to deliver it almost all the time.
The difficulty in getting feedback is extremely frustrating, particularly compared with getting feedback from, eg, google's webcrawlers.
The difficulty in getting feedback is extremely frustrating, particularly compared with getting feedback from, eg, google's webcrawlers.
What has completely decimated mail management in my experience had been the rise of cryptolocker related emails.
You could always deal with traditional viruses by blocking certain types of attachments. A false negative on spam just annoyed someone with some Viagra sale they probably didn't want.
Nowadays, you fail to block an email and suddenly a user loses his entire department's file share for the day.
You could always deal with traditional viruses by blocking certain types of attachments. A false negative on spam just annoyed someone with some Viagra sale they probably didn't want.
Nowadays, you fail to block an email and suddenly a user loses his entire department's file share for the day.
It is simple: the more fear uncertainty and doubt the "big email providers" can cast on no using one of the big email providers the more they chase everyone into their business (when they can read it). You can go on and on how it is "technically hard to fix email" but that is a second order effect to not even trying.
"no using" -> "not using". sorry.
Having run my own email server about a decade ago I can kind of understand why they are just rejecting things. Dealing with spam for even just a handful of domains was a nightmare.
Gmail has Postmaster tools that let domains with moderately large mail flow be able to monitor their spam reputation scores, email authentication e.g DKIM / SPF, DMARC rejection rates, etc. This tool can help address a number issues folks here have mentioned wrt Gmail. In particular please look at the "Delivery Errors Dashboard" description in the help center article.
help center post: https://goo.gl/7QHoqc Postmaster tool: https://gmail.com/postmaster/
(disclosure I work in Gmail)
help center post: https://goo.gl/7QHoqc Postmaster tool: https://gmail.com/postmaster/
(disclosure I work in Gmail)
We are using Gmail Postmaster tools (I work at a small ESP), and while this tool is certainly useful for diagnosing rejections, delivery to Gmail inbox / promotions is still a black box.
In our case, a number of IP addresses we are using started showing up as "bad" in Postmaster tools, despite having no increase in complaint rates (actually, in Spam rate dashboard it shows 0.0% for the past 90 days). We have implemented Gmail feedback loop headers into outgoing emails to diagnose the issue, but the Feedback Loop dashboard also does not show any data.
All of these IPs have a Senderscore reputation in the high 90s and deliver perfectly fine to all ISPs, except Gmail. Contacting Gmail team through their form also yields no response.
My takeaway from this situation is that once Gmail starts not liking your emails for some obscure reason, you have practically no way to fix that, apart from getting a new IP / sending domain and starting to build sending reputation from scratch.
In our case, a number of IP addresses we are using started showing up as "bad" in Postmaster tools, despite having no increase in complaint rates (actually, in Spam rate dashboard it shows 0.0% for the past 90 days). We have implemented Gmail feedback loop headers into outgoing emails to diagnose the issue, but the Feedback Loop dashboard also does not show any data.
All of these IPs have a Senderscore reputation in the high 90s and deliver perfectly fine to all ISPs, except Gmail. Contacting Gmail team through their form also yields no response.
My takeaway from this situation is that once Gmail starts not liking your emails for some obscure reason, you have practically no way to fix that, apart from getting a new IP / sending domain and starting to build sending reputation from scratch.
It's not a solution for low-traffic personal email servers. I've tested 3 domains in Postmaster Tools and for all 3 there's no data (I've added them a month ago or so - there should be data already)
No data to display at this time. Please come back later. Postmaster Tools requires that your domain satisfies certain conditions before data is visible for this chart. Refer to the help page for more details.
No data to display at this time. Please come back later. Postmaster Tools requires that your domain satisfies certain conditions before data is visible for this chart. Refer to the help page for more details.
What about for the small time, personal servers? How do we get our mail delivered without having to use a third party service like Mandrill?
I'm not blaming you personally, but the way Gmail handles email originating from personal servers is reprehensible.
I'm not blaming you personally, but the way Gmail handles email originating from personal servers is reprehensible.
Did you try to acquire reputation by having real people with accounts in "known" services send emails to your server?
If this sort of shit was required to get a basic www-server up and running, do you think anyone would have cared about the internet at all?
The internet was supposed to be for everyone and anyone. Current internet email is terribly flawed. There's just no way around it and the current status cannot be defended.
The internet was supposed to be for everyone and anyone. Current internet email is terribly flawed. There's just no way around it and the current status cannot be defended.
It may be infeasible to run a new SMTP-based mail service from "residential IP's" that can interact with the existing email empire, dominated by store and forward middlemen who expect to make money from the "free" email service they provide.
That empire amounts to a junk email delivery service and later a way to gather information about email users. The later purpose is probably why you want to run a new email service?
However it is certainly feasible to run a new SMTP-based email service from residential IP's that does NOT interact with the existing email empire. One with no middlemen. The sender's SMTP server talks directly to the recipient's SMTP server. You decide what port you want to use. There are thousands to choose from.
There are multiple ways to do this, but I rarely if ever see this option discussed. I suspect it's because like DNS most users are not comfortable configuring mail servers nor with NAT traversal.
If indeed the motivation for running your own mail service is because you do not want your mail stored on third part servers (whether in the sender's mail folders or the recipient's), then the ability to interact with the existing store and forward email providers seems a counterproductive requirement.
That empire amounts to a junk email delivery service and later a way to gather information about email users. The later purpose is probably why you want to run a new email service?
However it is certainly feasible to run a new SMTP-based email service from residential IP's that does NOT interact with the existing email empire. One with no middlemen. The sender's SMTP server talks directly to the recipient's SMTP server. You decide what port you want to use. There are thousands to choose from.
There are multiple ways to do this, but I rarely if ever see this option discussed. I suspect it's because like DNS most users are not comfortable configuring mail servers nor with NAT traversal.
If indeed the motivation for running your own mail service is because you do not want your mail stored on third part servers (whether in the sender's mail folders or the recipient's), then the ability to interact with the existing store and forward email providers seems a counterproductive requirement.
OP might still have some configuration issue (check the headers of the mail that has been placed in the spam folder. Gmail usually tells you what the problem is).
Last August, I have added IPv6 to our mail server, so its address for sure didn't have any prior reputation. Once the PTR and SPF records were correct, sending mail over v6 to gmail was no problem at all.
I have yet to see any delivery issue over v6 that's caused by a lack of reputation
Last August, I have added IPv6 to our mail server, so its address for sure didn't have any prior reputation. Once the PTR and SPF records were correct, sending mail over v6 to gmail was no problem at all.
I have yet to see any delivery issue over v6 that's caused by a lack of reputation
Surprised he didn't mention third party reputation providers such at Return Path.
I'm surprised as well, since these services reinforce his point even more. In other words, good reputation = $$$.
Also, I currently use mailgun (and dabbling with mandrill on some new projects) but I'm intrigued by postmarkapp.com take on dedicated IP addresses. Apparently they won't sell dedicated IP addresses because of the time needed to warm them up, so you get a shared IP and their TOS is stricter anti-spam than other ESP's: http://blog.postmarkapp.com/post/14127210172/the-false-promi...
Also, I currently use mailgun (and dabbling with mandrill on some new projects) but I'm intrigued by postmarkapp.com take on dedicated IP addresses. Apparently they won't sell dedicated IP addresses because of the time needed to warm them up, so you get a shared IP and their TOS is stricter anti-spam than other ESP's: http://blog.postmarkapp.com/post/14127210172/the-false-promi...
Mmm... not really. Dedicated IPs are hard to beat if you have a steady traffic of high quality and low-latency delivery is a priority (think of PagerDuty). Most legit businesses/customers have that. Most likely they're not offering dedicated IPs because they're tough to get, especially if you are a small company in email business.
The reason Mailgun and Mandrill can do this is because they belong to much larger companies with better access to IPv4 stockpiles: Rackspace and Mailchimp.
Source: I worked at Mailgun.
The reason Mailgun and Mandrill can do this is because they belong to much larger companies with better access to IPv4 stockpiles: Rackspace and Mailchimp.
Source: I worked at Mailgun.
It's only a matter of time before we start seeing peering issues between major email providers. This is not a system for sharing, it is competition.
Email is such a broken system. It's unacceptably hard to get a message from one machine to another and have it not marked as spam.
Furthermore I can't stand when I have to setup all of my accounts across the multitude of devices that I own. Way too many settings to configure regarding ports, domain names, encryption techniques, folder mappings, etc.
Don't get me wrong, I'm super impressed that email has managed to continue to be an essential tool in many people's day to day workflow despite being old enough to be found in a museum. But it's time for a change, and for us to put all we've learned about encryption, data serialization, and user workflow out on the table to design a better future for the universal method of digital asynchronous threaded communication.
And if we've learned anything from Google Wave and friends, these changes have to make sense, and not stray too far from what we're already familiar with and know that works.
Anyone want to weigh in?
Furthermore I can't stand when I have to setup all of my accounts across the multitude of devices that I own. Way too many settings to configure regarding ports, domain names, encryption techniques, folder mappings, etc.
Don't get me wrong, I'm super impressed that email has managed to continue to be an essential tool in many people's day to day workflow despite being old enough to be found in a museum. But it's time for a change, and for us to put all we've learned about encryption, data serialization, and user workflow out on the table to design a better future for the universal method of digital asynchronous threaded communication.
And if we've learned anything from Google Wave and friends, these changes have to make sense, and not stray too far from what we're already familiar with and know that works.
Anyone want to weigh in?
My experience has been that sending from my server works fine if it's in DNS and RDNS. Sending in that server's name works fine if the SPF records are present. But I don't bulk send; everything I send is something I typed, other than some messages the server sends to me periodically.
You can talk about HTTPS and forcing encryption all over the web however much you want but as long as you're stuck with SMTP and the few large coorps as the only viable mail solutions you can forget individual citizen data privacy... at least thats my 2cents.
Perhaps the problem here is that there is no verified identity for email servers?
There is - this is effectively what DKIM does. Reputation systems calculate reputations over email server identities, not email sender identities (subtle distinction).
Yes true, there is SPF and DKIM which are used by the big providers, though I'm not sure how strongly they trust those, but they shouldn't be using IP as a signal to determine whether the server is legitimate, or it leads to the catch 22 this article talks about - unknown IP = SPAM.
What's that even mean?
Certs or some other method to identify servers, so that gmail etc don't attempt to use IP for server identity, which is obviously flawed. The SMTP protocol is pretty flawed itself though, so probably that needs to change to see any real progress here.
something like EV certificates for smtp servers?
Certificates for SMTP servers are meaningless without DNSSEC because crossdomain email servers are a thing.
Crossdomain web hosts are also a thing, and HTTPS works fine. (Sometimes with particularly hilarious definitions of "fine", like CloudFlare's former practice of putting dozens of customers' websites in the same certificate, via subject alternative names.)
If you're worried about the fact that your mail host and web host can now impersonate each other, we can just define a new X.509 extension for "I can only be used for email". You might even be able to get away adding a new option to the existing Extended Key Usage field, but it's possible that enough clients don't enforce it rigorously enough.
Ideally, you'd also want an SMTP equivalent of strict transport security.
If you're worried about the fact that your mail host and web host can now impersonate each other, we can just define a new X.509 extension for "I can only be used for email". You might even be able to get away adding a new option to the existing Extended Key Usage field, but it's possible that enough clients don't enforce it rigorously enough.
Ideally, you'd also want an SMTP equivalent of strict transport security.
SMTP can use TLS, though, right? It doesn't _have_ to use STARTTLS? You _could_ use SNI.
My concern is that it doesn't get you anywhere. phishing sites can and do get TLS/SSL certificates. The process isn't particularly difficult or labour intensive if you own the domain. As far as spam goes, so what? This only proves I'm talking to the server I intended to, not that it's a reputable and upstanding member of the server society.
My concern is that it doesn't get you anywhere. phishing sites can and do get TLS/SSL certificates. The process isn't particularly difficult or labour intensive if you own the domain. As far as spam goes, so what? This only proves I'm talking to the server I intended to, not that it's a reputable and upstanding member of the server society.
What about EV itself? Currently there's no EV equivalent for individuals, but it would be a step forward.
If I can't reasonably get it, then how is it a step forward?
It would at least help companies/freelancers to host their own mailserver. The same infrastructure may lead to EV for individuals (or equivalent) once we can figure that out. Waiting for everything to be equally ready leads to inaction.
If someone spoofs a CNAME to evil.com in response to DNS query for google.com, your browser will not accept a certificate from evil.com as valid for google.com, whereas if you send mail to gmail.com and someone spoofs an MX record for mx.evil.com, a vaild mx.evil.com certificate will be accepted.
Right, so "don't do that." There isn't a clear standard for what subject to accept, out of the two options. The obvious correct one is to check for google.com, even if you're talking to a server named mx.google.com -- just as you don't trust CNAMEs and you check the original name, don't trust MXes either. It's just that most people don't have those certs configured in their mail servers, so doing that check today would lead to widespread failures and it's a bad default.
Postfix, for instance, lets you configure what it checks via the "smtp_tls_verify_cert_match" option. If you set it to "nexthop", it'll require google.com. The default, "hostname", will check against mx.evil.com.
http://www.postfix.org/postconf.5.html#smtp_tls_verify_cert_...
Postfix, for instance, lets you configure what it checks via the "smtp_tls_verify_cert_match" option. If you set it to "nexthop", it'll require google.com. The default, "hostname", will check against mx.evil.com.
http://www.postfix.org/postconf.5.html#smtp_tls_verify_cert_...
That proves I own the domain to some extent. Don't DKIM records do the same? For the purposes of anti-spam, a cert doesn't do anything except prove that you showed some CA that you own the domain (which can be done and is done in other ways currently).
At present SMTP is so broken that arbitrary servers are allowed to send mail for any domain, with fake headers, there is no proper verification of sending server or sending identity, the mail is accepted (along with legitimate mail), and put in spam folders (sometimes, as is real mail sometimes). SPF/DKIM are an attempt to solve that of course, but they are not enforced rigorously or universally, and tactics like IP reputation and email content sniffing still seem to be widely used too (the problem reported in the article). Compared to the web, email seems pretty behind on verifying server identity.
Yes, that's my point. It doesn't really solve any issues that SPF/DKIM don't already solve -- pointing to certs isn't a new solution, it's a rehash of the same solution.
And yes, the concept of relays should disappear along with much of the other cruft SMTP brings. (And while we're at it, can we fix/replace IMAP, or at least make the spec say that message ids are eternal and can't be invalidated?)
And yes, the concept of relays should disappear along with much of the other cruft SMTP brings. (And while we're at it, can we fix/replace IMAP, or at least make the spec say that message ids are eternal and can't be invalidated?)
IMO Message ids should be defined as a sha512 of headers and content to ensure they are unique. A new protocol is required.
Currently headers get modified in-flight, so a content-based ID wouldn't work.
As for a new protocol, there are some out there, but all fail in some way or another. I figured I'd throw my ideas out there and see how much they fail https://github.com/jimktrains/email_ng
As for a new protocol, there are some out there, but all fail in some way or another. I figured I'd throw my ideas out there and see how much they fail https://github.com/jimktrains/email_ng
EV certificates are to be bit more extensive than just proving ownership of domain. Most importantly they should associate the owner with some legal entity. So using EV certs would force spammers to establish shell companies, which typically is bit more expensive than just buying a domain and leaves stronger paper trail.
The problem goes both ways.
Spam needs to stop and so too does the convergence of Internet services in general (not just email).
Frankly, the solution already exists, and just like how billion-dollar companies start with the tech community, techies need to embrace the idea of decentralization by adopting some trust-based model.
Frankly, I do not see why encrypted emails cannot be accepted by users through some peer-sharing agreement where the public-key is online.
There's tons of solutions that are easy for tech people and if this community embraces it, it trickles down, like how a lot of other things do in tech.
Spam needs to stop and so too does the convergence of Internet services in general (not just email).
Frankly, the solution already exists, and just like how billion-dollar companies start with the tech community, techies need to embrace the idea of decentralization by adopting some trust-based model.
Frankly, I do not see why encrypted emails cannot be accepted by users through some peer-sharing agreement where the public-key is online.
There's tons of solutions that are easy for tech people and if this community embraces it, it trickles down, like how a lot of other things do in tech.
I don't think we will see a trickle down effect unless there are substantial benefits for the average user to outweigh the cost of switching and learning something new.
If there is a single abstract advantages, like 'more security', that might not be good enough.
We might as well throw in the kitchen sink and come up with a detailed list of every possible advantage an ideal replacement protocol should offer:
- no domain names required to create addresses
- 1 terabyte+ attachment file sizes
- ability to update and delete messages which recipients have not yet downloaded
- mail transfer and storage agent which don't have access to unencrypted message contents
- mail agents which can be distributed across multiple personal and home devices
- client apps which always and non-optionally use local encryption\decryption
- ability to purchase mail serving and storage resources from a competitive commercial market directly from client application.
- ability to switch to a different commercial serving\storage resource provider without loosing your previous identity\address.
If there is a single abstract advantages, like 'more security', that might not be good enough.
We might as well throw in the kitchen sink and come up with a detailed list of every possible advantage an ideal replacement protocol should offer:
- no domain names required to create addresses
- 1 terabyte+ attachment file sizes
- ability to update and delete messages which recipients have not yet downloaded
- mail transfer and storage agent which don't have access to unencrypted message contents
- mail agents which can be distributed across multiple personal and home devices
- client apps which always and non-optionally use local encryption\decryption
- ability to purchase mail serving and storage resources from a competitive commercial market directly from client application.
- ability to switch to a different commercial serving\storage resource provider without loosing your previous identity\address.
Everyone seems to agree that email is broken (and yet incredibly useful and almost universal in reach).
So moving on from there, how do we fix it? Who is currently working on fixing it? What would a new protocol look like?
So moving on from there, how do we fix it? Who is currently working on fixing it? What would a new protocol look like?
Chat services such as Slack, and social media like Twitter, are excellent ways to communicate. It's pretty simple. If you implement a request system so that both parties have to agree to let the other send messages, all of the problems are solved. There's no good reason that we need a system that allows anyone to send an arbitrary number of messages to anyone else.
[deleted](1)
This is the big question I'd like to hear an answer for as well.
You only need read the discussion that was already here prior to your comments. (-:
* https://news.ycombinator.com/item?id=10406040
* https://news.ycombinator.com/item?id=10405864
* https://news.ycombinator.com/item?id=10406040
* https://news.ycombinator.com/item?id=10405864
[deleted]
I suspect many of my messages are getting flagged as spam as well. Is there an easy way of checking whether emails are going through?
I've looked into this a bunch as we've been building Nylas. Our backend is essentially a cloud mail user agent (MUA), but can have similar issues to a MTA or mailbox provider.
It turns out that creating a successful product in the email space requires you to build relationships and partnerships with the existing vendors/providers. It takes a lot of work, and is part of what you pay for when using Mailgun/Mandrill/etc.
These "greylists" and systems that drop mail from unknown IPs have been pretty carefully designed and tuned to combat the insane amount of spam out there. They work remarkably well, and for most of today's email users, spam is no longer an issue. (The current state of email abuse innovation is promotions/marketing/etc. which is a more subtle challenge.)
At the end of this article, the author writes, "This isn't how the internet is supposed to work." Ironically this system of obscure reputation-based email is a direct result of how the email system was actually designed to work, with a total lack of permissions or feedback loop. Many SMTP servers used to not even require a password.
Stuff like DKIM/SPF and DMARC is a step in the right direction. But the RFCs upon which our email system is based were written decades ago, and in many cases have fundamental flaws, like SMTP leaking metadata no matter what. I could go on and on about issues with email, and why I care about getting them fixed, but let's just say it was designed in a different era of Internet with different constraints and opportunities.
So how do you build a new email service and not get blocked? Well, you spend a few weeks or months emailing, calling, Skyping, and meeting with folks in the current space. You work your way up through marketing support and random protocol discussion lists until you are talking with the folks who can influence which IPs are blocked/unblocked. Then you convince them you are (1) building a legit venture, (2) are worthy of their trust, and (3) don't directly compete with them. Then you'll get a small number of clean IPs and you must not screw up!
There are a few hacks, like sometimes a single partner/vendor will sell you a block of clean IPs and help manage the spam reputation. But usually it's just a lot of sweat and annoying phone calls. It takes way, way longer than setting up SPF/DKIM. The challenge is more relationships than technical.
Once you have a new email sending provider, the burden shifts to you for doing abuse/spam prevention. And you find yourself implementing many of the strategies and systems you cursed when getting started. But that's the circle of rfc2822 life I guess.
Oh, and never use EC2 IPs for sending mail. Most of them have been burned by spammers.
It turns out that creating a successful product in the email space requires you to build relationships and partnerships with the existing vendors/providers. It takes a lot of work, and is part of what you pay for when using Mailgun/Mandrill/etc.
These "greylists" and systems that drop mail from unknown IPs have been pretty carefully designed and tuned to combat the insane amount of spam out there. They work remarkably well, and for most of today's email users, spam is no longer an issue. (The current state of email abuse innovation is promotions/marketing/etc. which is a more subtle challenge.)
At the end of this article, the author writes, "This isn't how the internet is supposed to work." Ironically this system of obscure reputation-based email is a direct result of how the email system was actually designed to work, with a total lack of permissions or feedback loop. Many SMTP servers used to not even require a password.
Stuff like DKIM/SPF and DMARC is a step in the right direction. But the RFCs upon which our email system is based were written decades ago, and in many cases have fundamental flaws, like SMTP leaking metadata no matter what. I could go on and on about issues with email, and why I care about getting them fixed, but let's just say it was designed in a different era of Internet with different constraints and opportunities.
So how do you build a new email service and not get blocked? Well, you spend a few weeks or months emailing, calling, Skyping, and meeting with folks in the current space. You work your way up through marketing support and random protocol discussion lists until you are talking with the folks who can influence which IPs are blocked/unblocked. Then you convince them you are (1) building a legit venture, (2) are worthy of their trust, and (3) don't directly compete with them. Then you'll get a small number of clean IPs and you must not screw up!
There are a few hacks, like sometimes a single partner/vendor will sell you a block of clean IPs and help manage the spam reputation. But usually it's just a lot of sweat and annoying phone calls. It takes way, way longer than setting up SPF/DKIM. The challenge is more relationships than technical.
Once you have a new email sending provider, the burden shifts to you for doing abuse/spam prevention. And you find yourself implementing many of the strategies and systems you cursed when getting started. But that's the circle of rfc2822 life I guess.
Oh, and never use EC2 IPs for sending mail. Most of them have been burned by spammers.
I guess big providers has to deal with "newbies" all the time, that don't know how to configure their server and run open relays. And don't know how to add extra headers etc.
That said, silently dropping messages without a notification is probably illegal! And pretty serious! So if you know what you are doing (and you are not a spammer) you should send a cease and desist!
Just make sure you use double opt-in and that providing an e-mail address in sign-ups/etc is optional!
Some will however put your mail in a spam-folder and it's not much you can do about it, just hope your readers complain to their provider.
So basically: setup your smtp relay correctly! Make sure you are not on any black-lists. Add extra headers like dkmi / precedence, add spf (don't froget ipv6) and ptr. Add your relays to white-lists. Publicly publish a privacy and e-mail policy (important! with opt in and optional clause); Link to them and fill out some "email provider" forms at gmail/microsoft. Send out a bunch of test mails. This will take a whole day, but if you do this, you will have no problems unless your IP or domain is perma-banned.
That said, silently dropping messages without a notification is probably illegal! And pretty serious! So if you know what you are doing (and you are not a spammer) you should send a cease and desist!
Just make sure you use double opt-in and that providing an e-mail address in sign-ups/etc is optional!
Some will however put your mail in a spam-folder and it's not much you can do about it, just hope your readers complain to their provider.
So basically: setup your smtp relay correctly! Make sure you are not on any black-lists. Add extra headers like dkmi / precedence, add spf (don't froget ipv6) and ptr. Add your relays to white-lists. Publicly publish a privacy and e-mail policy (important! with opt in and optional clause); Link to them and fill out some "email provider" forms at gmail/microsoft. Send out a bunch of test mails. This will take a whole day, but if you do this, you will have no problems unless your IP or domain is perma-banned.
There is absolutely no law that says you have to accept an email, nor that you have to send a notification if you don't. A cease and desist?! Are you kidding?
Nobody else has mentioned it, so i will; This is a great situation for the NSA. Get all 400M accounts in one fell swoop by using a gag ordered warrant on Google (or microsoft) Ez peasy. Much harder than to contact a sysadmin about their 1 account that isn't being fed into the behemoth..
> Nobody else has mentioned it, so i will
Do we really need a comment against the NSA in every single thread mentioning emails or hosting in general? Not that I'm a NSA supporter or anything.
Do we really need a comment against the NSA in every single thread mentioning emails or hosting in general? Not that I'm a NSA supporter or anything.
I use a private email provider (privateemail.com, via Namecheap), which doesn't seem to stop others receiving my emails, but sites sometimes have trouble emailing me, oddly enough.
Though that might just be because I have a 6-character domain (ajf.me).
Though that might just be because I have a 6-character domain (ajf.me).
SMTP Mail is broken. The best justification is that error messages can't even be sent back to avoid abuse by spammers.
People are constrained to use services like gmail to avoid troubles. This is a serious privacy threat.
The anti-competitive consequences of this are really interesting. Does anyone know if there are historical statistics on email provider market shares? It would be interesting to see how things have changed over time.
I did not realize that I am running my private SMTPd (+ imaps) for nearly 8 years. I never had issues. My score on test-mail is 9/10 because of lack of DKIM. I will implement DKIM, see if I can get 10/10.
The irony is that at least half the spam I get comes from @gmail.com addresses.
Charge the sender one cent per email through a combination of legislation and technology. The problem will instantly go away.
The transfer of mailing lists to online forums will be a very small price to pay.
McAfee Mail Gateway also uses a reputation, called GTI. Appliance hashes the message, looks up it on this online service and get back a score.
There is a KB explaining that new senders get a high score.
There is a KB explaining that new senders get a high score.
This guys story is a sad one but I'm sure we're missing some important information here. New mail servers get set up all the time, it's not impossible for such servers to be accepted by the big players.
If none of these major services ever learned that his server was OK, it's likely that users weren't unmarking the mail he sent as spam. And that leads to the question of why not.
If none of these major services ever learned that his server was OK, it's likely that users weren't unmarking the mail he sent as spam. And that leads to the question of why not.
In general, everybody uses a service like MailChimp or SendGrid to do this. That allows them to send email, but keeps them from building up a good reputation for no spam on their IP addresses -- that reputation is built by the email provider, not by you as the customer.
For awhile I wondered why so many SaaS apps bothered to pay external (not cheap!) mail services when it's so easy to set up an email server.
This is why. My "easy to set up an email server" info was years out of date.
For awhile I wondered why so many SaaS apps bothered to pay external (not cheap!) mail services when it's so easy to set up an email server.
This is why. My "easy to set up an email server" info was years out of date.
sadly, it's even hard to get some email across when using those ESPs (Email Service Providers).
Our domain has all the correct SPF records, DKIM etc and we're using a reputable email provider (Mandrill). All is sparking clean. Validated etc. Still we see some activation emails to google get delivered, but severely delayed on the Google side (after we see an OK from google for the email being delivered). Our users can't reset their password or activate their account until a certain time passes...
Emails from my mum (who I've emailed on the same account for nearly a decade) suddenly get flagged as spam by gmail. Go figure.
Our domain has all the correct SPF records, DKIM etc and we're using a reputable email provider (Mandrill). All is sparking clean. Validated etc. Still we see some activation emails to google get delivered, but severely delayed on the Google side (after we see an OK from google for the email being delivered). Our users can't reset their password or activate their account until a certain time passes...
Emails from my mum (who I've emailed on the same account for nearly a decade) suddenly get flagged as spam by gmail. Go figure.
It's a lost cause.
The blacklisting ugliness is one more sign email was badly designed. One more nail in the coffin.
The blacklisting ugliness is one more sign email was badly designed. One more nail in the coffin.
[deleted]
So, why they look at the IP address at all when there is DKIM in place?
I see lots of threads shitting on the guy for doing it wrong vis-a-vis his configuration whilst ignoring his actual problem: An IP address without a reputation score. I've had the same problem and reached the same conclusion. The address can't just be clean, as in not on a blacklist, but has to essentially already be whitelisted via a "known good" reputation score or mail automatically gets blackholed. How do I get my VPS provider of choice to give me an IP address with a good reputation score?
Another post lamenting this situation:
https://sfconservancy.org/blog/2015/sep/15/email/
https://sfconservancy.org/blog/2015/sep/15/email/
Just goes to show reputation isn't the end all, be all, solution to everything. It's so often championed by the Linux kernel team as a reason why the distributed model works, and it's evident that in that case it certainly does. Here though, much outcry about how you can't setup a box and instantly be as respected as established boxes that have earned rep over time. This post just comes off way too butt hurt for my taste.
[deleted]
Mail spammers are truly vile people. They have ruined it for everyone.
I think it's exaggerated. I am happily running a private mail server on a tiny vserver. My emails get through to whomever I might mail and I do so a lot.
I once heard that my email got flagged as spam.
Of course, if you don't set up your mail server correctly it might be that it's flagged as spam, but it's not really harder than setting up various other things correctly.
It sounds like it was set up correctly by the author. Most people use SpamAssassin, also big companies do. So it should be good.
Maybe the network itself wasn't considered to be good by the mentioned big companies.
I once heard that my email got flagged as spam.
Of course, if you don't set up your mail server correctly it might be that it's flagged as spam, but it's not really harder than setting up various other things correctly.
It sounds like it was set up correctly by the author. Most people use SpamAssassin, also big companies do. So it should be good.
Maybe the network itself wasn't considered to be good by the mentioned big companies.
Greylisting still works amazingly well. With a long, long whitelist and greylisting plus DNSBL, I don't even bother running a spam filter, since the little bit of spam and emails from new senders ends up in its own directory as it came from a non-whitelisted sender.
Comcast finally started blocking residential mail server ports inbound a few years ago, so I had to migrate to a smarthost environment using a VPS as email server for $15/yr.[1]
Last year for a few months, Gmail was dropping everything I sent into the spam folder, even after recipients were marking it not spam. I eventually discovered the "Authentication-Results:" header that Gmail adds to every inbound message. It is under the "Show Original" dropdown menu. That showed that I "hadn't changed anything"(!) on my mail server, but suddenly Gmail was connecting to my mail server over an IPv6 interface, and I had never bothered to put the IPv6 block into the SPF record. Gmail was nice enough to explain exactly what it didn't like about those emails.
[1] http://lowendbox.com/blog/top-provider-poll-2014-q3-the-resu...