Ask HN: How do I build email server like AWS Ses
25 comments
I'm hosting my mail on my own server with postfix and dovecot for 6 years now. It's running smoothly, but I invested alot of work into it. it was fun to figure out how everything is working, anything I needed to know I had to google and stitch the pieces together for my needs, but it was totally worth it because I learned so much when doing it.
Since I know how easy the whole system could fail, I would never run a professional/commercial mail server by myself, not even for a small company. Office365, always!
So google for postfix, dovecot, mysql, dkim, roundcube and figure out how to glue it all together. Advise: start with an unused domain, not with your everyday mail address, migration can be done when everything is tested and running.
So google for postfix, dovecot, mysql, dkim, roundcube and figure out how to glue it all together. Advise: start with an unused domain, not with your everyday mail address, migration can be done when everything is tested and running.
Thanks , what failure points should I be carful ?
Don't get yourself onto spam block lists, which can relatively easy happen if your mail server is misconfigured. You can test the spam-level of your outgoing mail on this site: https://www.mail-tester.com
I saw this site to day ...
How does it give you level of spam ?
How can it give score based on what ?
you send a mail to the displayed address and it will parse the incoming mail and analyzes the message headers. the headers contain valuable information about the authenticity of the mail. If a mail looks authentic to the spam filter, it will be marked with a lower spam score.
As others pointed out, there is software for it. Many people use Postfix as main software and then combine it with Dovecot for IMAP.
But of course there are other softwares and the choice pretty much depends on your requirements regarding scale, security and ease of setup or maintenance. The reason why things are so scattered is probably because even in basic setups, there are a number of technologies involved, like for Spam filtering, SPF and DKIM. And not many people do this kind of thing because there's Gmail. ;-)
But if you are just looking for anything to get started, there are some Docker setups which do exactly what you ask for: https://github.com/tomav/docker-mailserver
But of course there are other softwares and the choice pretty much depends on your requirements regarding scale, security and ease of setup or maintenance. The reason why things are so scattered is probably because even in basic setups, there are a number of technologies involved, like for Spam filtering, SPF and DKIM. And not many people do this kind of thing because there's Gmail. ;-)
But if you are just looking for anything to get started, there are some Docker setups which do exactly what you ask for: https://github.com/tomav/docker-mailserver
Thanks I do want to learn about using spf and dkim
Don't. Even most of us that know what we're doing choose not to handle this drudgery ourselves.
I want to know how to do it
Have a look at Mail in a Box: https://mailinabox.email/
I'm actually trying to implement this right now. A serverless only email solution with SES to front the email sending and receiving.
First thing important to note, for email clients to be compatible you need to have your system support IMAP/POP3 which you cannot do in a serverless way because it requires an open Socket connection, so you need at least one cloud machine. So either 1) you make your own email client for your system (speaks to a custom backend which interacts with SES), or 2) you create a custom implementation of IMAP/POP3 which makes calls to your backend and bridges SES with the IMAP/POP3 Right now my solution is number 1, but 2 makes more sense in the long term.
I'll have something up and running to show in a couple of weeks hopefully; and I'll post it here when I get time to write it all up. If you want more info now, I can give it but it's a bit of a long writeup in itself.
First thing important to note, for email clients to be compatible you need to have your system support IMAP/POP3 which you cannot do in a serverless way because it requires an open Socket connection, so you need at least one cloud machine. So either 1) you make your own email client for your system (speaks to a custom backend which interacts with SES), or 2) you create a custom implementation of IMAP/POP3 which makes calls to your backend and bridges SES with the IMAP/POP3 Right now my solution is number 1, but 2 makes more sense in the long term.
I'll have something up and running to show in a couple of weeks hopefully; and I'll post it here when I get time to write it all up. If you want more info now, I can give it but it's a bit of a long writeup in itself.
Well if I understand you right you are using Aws smtp service
I want to learn how does it work
Ok I'll bite and give a bit more info.
SES sets up the ability to send or receive email. The receiving is handled by SES by either creating an SNS notification and/or storing the email (in MIME format) into an S3 bucket. Now let's say you've configured SES to store the email to S3 and emit an SNS notification, if you want to create a backend that will allow you to read the emails, you'll want to handle the SNS notification (via a subscribed Lambda, or a service in EC2 polling a SQS which is subscribed to the SNS) and store some of the email metadata in a database (like DynamoDB or Aurora) so that you can search the emails easier. The SNS notification tells you the email information as well as telling you the name of the file that it will create in S3, so if you store all this in the database you can load the raw email later (e.g. to get the attachments) Now let's say you have a searchable database, you'll want to make a service that allows a front-end (website or app) to read this information. So you'll want to make a backend REST API which allows users to get emails for a particular user by searching the database for emails to that user. If you're making this for users other than yourself, you'll also probably want some more user specific information in another, user-orientated, database like which stores which users have read which emails. And you'll also want to setup some user authentication/authorisation so only the right users can read the right emails.
Anyways that's how you setup recieving. To setup sending is much simpler, and I'd just follow the documentation.
SES sets up the ability to send or receive email. The receiving is handled by SES by either creating an SNS notification and/or storing the email (in MIME format) into an S3 bucket. Now let's say you've configured SES to store the email to S3 and emit an SNS notification, if you want to create a backend that will allow you to read the emails, you'll want to handle the SNS notification (via a subscribed Lambda, or a service in EC2 polling a SQS which is subscribed to the SNS) and store some of the email metadata in a database (like DynamoDB or Aurora) so that you can search the emails easier. The SNS notification tells you the email information as well as telling you the name of the file that it will create in S3, so if you store all this in the database you can load the raw email later (e.g. to get the attachments) Now let's say you have a searchable database, you'll want to make a service that allows a front-end (website or app) to read this information. So you'll want to make a backend REST API which allows users to get emails for a particular user by searching the database for emails to that user. If you're making this for users other than yourself, you'll also probably want some more user specific information in another, user-orientated, database like which stores which users have read which emails. And you'll also want to setup some user authentication/authorisation so only the right users can read the right emails.
Anyways that's how you setup recieving. To setup sending is much simpler, and I'd just follow the documentation.
So ignore this ^ comment unless you're interested in setting up for own email system wrapping around SES
Ok I just reread your comment. Yeah I'm using SES to setup an email system. If you want to do the SMTP by yourself then that's a shit tonne more effort than what I've just described. Other people have linked some useful stuff so I won't comment any further.
I've had good experiences with https://www.iredmail.org/
> With iRedMail, you can deploy an OPEN SOURCE, FULLY FLEDGED, FULL-FEATURED mail server in several minutes, for free.
> We did the heavy lifting of putting all the open source components together and applying best practices. Our product does all the major tasks for you. Furthermore we offer professional support to back you up in case you have some problems.
> With iRedMail, you can deploy an OPEN SOURCE, FULLY FLEDGED, FULL-FEATURED mail server in several minutes, for free.
> We did the heavy lifting of putting all the open source components together and applying best practices. Our product does all the major tasks for you. Furthermore we offer professional support to back you up in case you have some problems.
It is a great experience to have and know that you will not want to do it again :)
I think setting it up is not as difficult as to maintain and make sure emails are not bounced back, etc.
I think setting it up is not as difficult as to maintain and make sure emails are not bounced back, etc.
Thanks , well this is part of what I want to learn
How mailchip started to build there mail infra?
Learn SMTP. Use telnet to send an email. That is the basics. It gets more and more nuanced after that. Auth. Deliverability (avoiding looking like spam and doing dkim and spf). Dealing with retries and bounces. Writing valid mimes. If you want to be like an email service provider, you will have to deal with multi tenant issues too. These are the basics for sending mail. For receiving mail, you have spam report handling, delayed bounces, inbox validation, mime parsing, etc. Because of all this, most people just use something like postfix.
The best full guide is the ISPmail tutorials: https://workaround.org/ispmail
Thanks!!
i think it a bingo , reading this now looks like good starting point
Most recently, I've done this with postfix, dovecot, and rspamd. In previous lives, I've used sendmail and qmail...
Listen on a port. Parse bytes. Encode bytes. Parse text.
Implement SMTP.
Implement SMTP.
Be sure to check out open source repos.
https://github.com/haraka/Haraka
https://github.com/haraka/Haraka
Dovecot email server may be a good starting point?
how can I build real private email server with incoming spam filters for domain There are so many pieces scattered around the web but no complite picture.
Thanks