Help Needed: Programmatic Email Creation and Management with Custom SMTP Server
11 comments
You don't need your own SMTP server for this. I'm working on a project with a similar need at the moment and Mailgun's Email Routing feature is exactly what you need.
https://www.mailgun.com/products/send/inbound-routing/
https://www.mailgun.com/products/send/inbound-routing/
thank you, this is useful
If you want to implement this yourself just realize that email is a text format. HTTP (2616) is actually based on RFC822 for format. That is important because receiving email requires a full understanding of the routing mechanisms defined in 821 and successive documents. Sending, though, just requires that you get 822 right because sending an email message only requires writing the contents correctly over a socket and DNS. Knowing that is how spammers can blast out millions of emails at almost no cost, because you don’t have to be an email server or client to send emails.
Lots of managed providers allow you to set up catch all inboxes that you can then get webhooks for. SES and Mailgun are the two I've used successfully.
I've implemented an email relay before. We had an email address like relay+<code>@site.com that was delivered to a webhook via Mailgun. The code after the plus gave us all the unique information we needed to handle the email.
You can use something like Mail in a box [0] to setup your own SMTP server.
[0] https://github.com/mail-in-a-box/mailinabox
[0] https://github.com/mail-in-a-box/mailinabox
Do you know about managed SMTP services like AWS SES? Running your own SMTP is possible but would take a lot more work to setup and maintain.
SES sounds like a good option, will look into that. when you say alot more work, could you elaborate on that longer term vs getting the same alternative on a paid provider? Also are there any providers that offer this level of control?
I am running my own SMTP for my SaaS. I used managed SMTP services, including SES, before. It took be roughly 2 months to get it all set up: research the SMTP server options, trial a few of them, pick one, learn it thoroughly, write deployment scripts, deploy it, switch email traffic over to it gradually to warm up the sending IP, and monitor it. Blogged about it here: https://blog.healthchecks.io/2023/08/notes-on-self-hosted-tr...
If you are starting from zero, and are not running in production yet, I suppose it's easier as you can afford to make mistakes and lose some email. I had to measure everything 7 times, and still had one major oh-no event :-)
If you are starting from zero, and are not running in production yet, I suppose it's easier as you can afford to make mistakes and lose some email. I had to measure everything 7 times, and still had one major oh-no event :-)
I am running Dovecot and Postfix in production for decades now. Both are stable and reliable.
First, let me start with saying what you are wanting is a bad idea. Parts can be done by using dovecot as the MDA with a backed mysql/mariadb database (where you host your own mailboxes).
You will have a number of problems that may or may not be recoverable. This is the front lines.
I've worked with email (and related components) for years, and it is easily one of the most automated and targeted set of protocols for bad actors.
Any server that contains an MX record is going to be flooded with bots testing your defenses, and they will often be from large botnets; we easily reached 1 million unique IPv4 addresses within a month with fail2ban and ipset doing quite a lot of the work for our public facing server (these were just the repeat offenders that failed graduated response). Fail2ban has problems as well (where it may not work properly if you didn't verify).
Parsing to LLM is one of the worst ideas you mention because you are effectively deserializing user input into a trusted zone. Please review OWASP. Any scriptkiddie is going to have a field day with you.
https://cheatsheetseries.owasp.org/cheatsheets/Deserializati...
Additionally, as a result of spam, mail delivery is almost more important than setting up a barebones server. You can set up a server but not have the mail received by your intended recipients.
This is because mail delivery includes both the sending, and the receiving portions. Email service providers (ESP) will reject mail by default in many cases depending on various reputational factors they do not publicly disclose.
There are processes you must follow to build up a reputation which is tied to both your domain records, and your IP history. These requirements change arbitrarily depending on the ESP recipient. For example, google at one point required that any email sent include duplicates of the message as both a plain, and a html mimetype. This may no longer be the case, but its an example of arbitrary requirements.
The general business processes required to get a good reputation score involves parsing the many whitepapers located at M3AAWG.org. There are a lot of them, but these are where the general consensus requirements for email to be delivered are being posted.
If you get on a block list because you didn't properly secure your system, or failed to follow the posted whitepaper guidelines, you may not be removed from those private lists for up to a year or more, and there is no technical contact to reach out for re-evaluation even then even when it is received, it may go straight to the SPAM folder (each one lowering your reputation score).
Most ESPs have postmaster services available (if you sign up) which are crucial in discovering and heading these potential problems off.
There are a number of metrics beyond the sending process that also go into reputation. For example, Google lowers reputation if the emails being received in a mailbox are never opened, read by the recipient, or interacted with through various widgets in their web GUI. Microsoft and other companies do the same.
Needless to say, this is a very nuanced subject area, with the downsides of doing something wrong, often requiring a start from scratch/back to formula approach.
You will have a number of problems that may or may not be recoverable. This is the front lines.
I've worked with email (and related components) for years, and it is easily one of the most automated and targeted set of protocols for bad actors.
Any server that contains an MX record is going to be flooded with bots testing your defenses, and they will often be from large botnets; we easily reached 1 million unique IPv4 addresses within a month with fail2ban and ipset doing quite a lot of the work for our public facing server (these were just the repeat offenders that failed graduated response). Fail2ban has problems as well (where it may not work properly if you didn't verify).
Parsing to LLM is one of the worst ideas you mention because you are effectively deserializing user input into a trusted zone. Please review OWASP. Any scriptkiddie is going to have a field day with you.
https://cheatsheetseries.owasp.org/cheatsheets/Deserializati...
Additionally, as a result of spam, mail delivery is almost more important than setting up a barebones server. You can set up a server but not have the mail received by your intended recipients.
This is because mail delivery includes both the sending, and the receiving portions. Email service providers (ESP) will reject mail by default in many cases depending on various reputational factors they do not publicly disclose.
There are processes you must follow to build up a reputation which is tied to both your domain records, and your IP history. These requirements change arbitrarily depending on the ESP recipient. For example, google at one point required that any email sent include duplicates of the message as both a plain, and a html mimetype. This may no longer be the case, but its an example of arbitrary requirements.
The general business processes required to get a good reputation score involves parsing the many whitepapers located at M3AAWG.org. There are a lot of them, but these are where the general consensus requirements for email to be delivered are being posted.
If you get on a block list because you didn't properly secure your system, or failed to follow the posted whitepaper guidelines, you may not be removed from those private lists for up to a year or more, and there is no technical contact to reach out for re-evaluation even then even when it is received, it may go straight to the SPAM folder (each one lowering your reputation score).
Most ESPs have postmaster services available (if you sign up) which are crucial in discovering and heading these potential problems off.
There are a number of metrics beyond the sending process that also go into reputation. For example, Google lowers reputation if the emails being received in a mailbox are never opened, read by the recipient, or interacted with through various widgets in their web GUI. Microsoft and other companies do the same.
Needless to say, this is a very nuanced subject area, with the downsides of doing something wrong, often requiring a start from scratch/back to formula approach.
I've been working on a project where I need to programmatically create and manage email addresses under my own domain for each user onboarded. I currently use Resend for sending emails (e.g., welcome emails), and my domain's DNS is managed on Cloudflare. However, Resend doesn't seem to support programmatically creating email addresses.
My goal is to programatically create emails, monitor email and parse them to LLM etc
I've done some research and considered setting up a custom SMTP server to handle this. Has anyone here implemented a similar system? What are the best practices for setting up a custom SMTP server for this purpose? Any recommendations on tools or libraries to make this easier?
I'm looking for advice on the best way forward. Appreciate any insights!