Mailit: A Tiny Drop-In REST API to Send Emails(github.com)
github.com
Mailit: A Tiny Drop-In REST API to Send Emails
https://github.com/dthree/mailit
74 comments
I don't really understand what you gain from this. SMTP is much more widespread than this API. If you want to migrate to a service later on, all you need to do is change the hostname to that service, how much easier is it to have to babysit a service in a container?
For a first beta (MVP?) I would strongly advice against using your own mail server. It takes a while for a server to build up a reputation, hence mails from an "unknown" server are quicker marked as spam (especially by common services like gmail and outlook).
Instead configure a local sendmail (so you have all the benefits of local mail delivery, like queuing and availability so on) to relay through your provider's mail servers.
Instead configure a local sendmail (so you have all the benefits of local mail delivery, like queuing and availability so on) to relay through your provider's mail servers.
MailIt doesn't provide a SMTP server.
Instead it takes your existing SMTP settings, so you can take the advantage of your existing reputation, or your provider's one.
(assuming you already have an email account somewhere, and it can be gmail, outlook, [popularprovider]...)
Instead it takes your existing SMTP settings, so you can take the advantage of your existing reputation, or your provider's one.
(assuming you already have an email account somewhere, and it can be gmail, outlook, [popularprovider]...)
Author here. There seems to be a lot of discussion over use cases. Here's the use case I built it for:
All traffic for my (large) application hits a load balanced API gateway. Its role is to authenticate and forward requests to one of many services.
The gateway is the only point of exposure to internal services. After something has cleared the gateway on a route with roles approved for that user, there is little worry about security. Certificates between servers and containerized deployments such as Kubernetes help on this as well.
I'm not going to write email-sending logic in my gateway. It just handles AAA and then proxies the request.
By forwarding to microservices instead of a monolith, I can scale workload better and have less risk of a mail bug taking down other services.
All traffic for my (large) application hits a load balanced API gateway. Its role is to authenticate and forward requests to one of many services.
The gateway is the only point of exposure to internal services. After something has cleared the gateway on a route with roles approved for that user, there is little worry about security. Certificates between servers and containerized deployments such as Kubernetes help on this as well.
I'm not going to write email-sending logic in my gateway. It just handles AAA and then proxies the request.
By forwarding to microservices instead of a monolith, I can scale workload better and have less risk of a mail bug taking down other services.
It migt be just me, but I honestly don't understand why people would use an HTTP API to send mail.
SMTP really is a very simple protocol, that's matured for decades now. Also it is fully specified by RFCs.
I can see why mail services that provide templating (eg Mandrill) have special APIs, they offer something different from sending regular emails.
This specific REST API seems more like a standin for sendmail, which IMO makes things only more complicated.
SMTP really is a very simple protocol, that's matured for decades now. Also it is fully specified by RFCs.
I can see why mail services that provide templating (eg Mandrill) have special APIs, they offer something different from sending regular emails.
This specific REST API seems more like a standin for sendmail, which IMO makes things only more complicated.
Playing Devil's advocate: SMTP has some back-and-forth stateful chatter. With a custom HTTP API, you can authenticate and send an email in a single request. SMTP Pipelining can reduce this problem, but not all libraries support it (e.g. Python's smtplib).
>SMTP has some back-and-forth stateful chatter.
So what ? Is not like you implement the SMTP protocol into your application to send emails. Any language under the sun has a SMTP implementation exposed as a library.
So what ? Is not like you implement the SMTP protocol into your application to send emails. Any language under the sun has a SMTP implementation exposed as a library.
It's slower, which is important when you're trying to push a lot of emails. If you keep the TLS connection open, sending an HTTP request is very fast. Waiting for the roundtrips of that chatter reduces the throughput, especially on higher latency links.
> If you keep the TLS connection open, sending an HTTP request is very fast. Waiting for the roundtrips of that chatter reduces the throughput, especially on higher latency links.
I agree that for a high load of sending email it can be beneficial to use an intermediate service. IMO a local sendmail is still preferred, but I see your point.
However, the Mailit service makes the client wait on the SMTP traffic too [1]. So with this particular service you now have to wait for both SMTP /and/ HTTP communication. Doesn't really speed thinks up :)
[1] https://github.com/dthree/mailit/blob/master/src/routes.js#L...
I agree that for a high load of sending email it can be beneficial to use an intermediate service. IMO a local sendmail is still preferred, but I see your point.
However, the Mailit service makes the client wait on the SMTP traffic too [1]. So with this particular service you now have to wait for both SMTP /and/ HTTP communication. Doesn't really speed thinks up :)
[1] https://github.com/dthree/mailit/blob/master/src/routes.js#L...
However, the Mailit service makes the client wait on the SMTP traffic too [1]. So with this particular service you now have to wait for both SMTP /and/ HTTP communication. Doesn't really speed thinks up :)
That depends, you could host the Mailit service in the same machine as the email server, which your application instances on other machines would call. This could be useful if you have multiple machines but only want to run a single email server.
That depends, you could host the Mailit service in the same machine as the email server, which your application instances on other machines would call. This could be useful if you have multiple machines but only want to run a single email server.
[deleted]
We have configured at work an intermediate postfix for this effect, but it takes a bit to learn how to configure it
So I can see the benefits of this approach
So I can see the benefits of this approach
Another reason i would do is , i have only port 80/443 allowed outside my network. so using https api helps with that
Why is that? What security does it provide? This very project just demonstrated that you can send pretty much anything in HTTP(S) so any malware (I presume) you're trying to defeat can do the same.
Spam malware can scan IPs to find open SMTP ports to send spam so some ISPs and hosting providers lock down these ports to avoid getting their IPs blacklisted when a compromised host starts sending a bunch of spam through an open relay. I can see this being used to send email out from some restricted hosting providers.
I want to send mails from embedded devices, similar to e.g. an ESP8266, with intermittent connectivity and a semi-stable power source. An HTTP API seems simpler for me to interface with than SMTP.
I can't comment on what seems simpler to you, but ... what does intermittent connectivity have to do with whether you transmit the email via SMTP or via HTTP?
> what does intermittent connectivity have to do with whether you transmit the email via SMTP or via HTTP
SMTP uses a stateful connection where HTTP is basically a single request/response. I.e. the device can close the HTTP connection after a single roundtrip where SMTP requires waiting for server responses after commands and then continuing the request.
Not sure if it's simpler on those devices to use an HTTP interface (because HTTP requires a bit more overhead data) but it might be simpler, sure.
SMTP uses a stateful connection where HTTP is basically a single request/response. I.e. the device can close the HTTP connection after a single roundtrip where SMTP requires waiting for server responses after commands and then continuing the request.
Not sure if it's simpler on those devices to use an HTTP interface (because HTTP requires a bit more overhead data) but it might be simpler, sure.
HTTP runs on top of TCP, which already gives you at least two round trips. Also, if it's a device that's connected via the public internet, you probably should be using TLS, which gives you a couple more round trips.
Really, realistically, if your internet connectivity is too bad to support SMTP, it most likely won't work for HTTP either, and you probably should be using some custom UDP thingy with an appropriate retry strategy.
Really, realistically, if your internet connectivity is too bad to support SMTP, it most likely won't work for HTTP either, and you probably should be using some custom UDP thingy with an appropriate retry strategy.
I've looked at setting up sendmail as a "LAMP guy" and I said nope nope nope, just use mailgun.
You still need to set up the SMTP server with this solution.
Creating the relay is the hard part of any mail service because even SMTP aside, you need to set DNS entries on your FROM domain to authorise your SMTP relay's IP (otherwise spam filters will just block your email). And if it's regular commercial mailshots then you'd need to register with one of the trusted services (which isn't free) and have all sorts of controls like rate limiting in place - again so you don't just get rejected as spam.
Honestly, I would welcome a complete ground up re-implementation of email - SMTP, POP/IMAP and all. The "very simple protocol, that's matured for decades now" is really just a mess of different protocols - different authentication models, different encryption methods, etc and all operating without a standard format for error handling. Plus all of the additional bloat bolted on to handle the swell of abuse that the email "network" has had to endure. There's nothing simple about email any more.
However going back to the topic: the unfortunate thing about this REST API is it seeks to solve the least complicated part of the whole SMTP stack.
Creating the relay is the hard part of any mail service because even SMTP aside, you need to set DNS entries on your FROM domain to authorise your SMTP relay's IP (otherwise spam filters will just block your email). And if it's regular commercial mailshots then you'd need to register with one of the trusted services (which isn't free) and have all sorts of controls like rate limiting in place - again so you don't just get rejected as spam.
Honestly, I would welcome a complete ground up re-implementation of email - SMTP, POP/IMAP and all. The "very simple protocol, that's matured for decades now" is really just a mess of different protocols - different authentication models, different encryption methods, etc and all operating without a standard format for error handling. Plus all of the additional bloat bolted on to handle the swell of abuse that the email "network" has had to endure. There's nothing simple about email any more.
However going back to the topic: the unfortunate thing about this REST API is it seeks to solve the least complicated part of the whole SMTP stack.
Thanks, I missed that part, I thought drop in means it handles most of the things :)
I'd welcome that too, it's kinda sad nobody succeeded with "email 2.0", sure we have lots of attempts but I can't recall any major players. The best solutions to "fix email" were companies that advocated to not use email and use their closed platform.
What I can think of would be cool is something XMPP based with email fallback.
I'd welcome that too, it's kinda sad nobody succeeded with "email 2.0", sure we have lots of attempts but I can't recall any major players. The best solutions to "fix email" were companies that advocated to not use email and use their closed platform.
What I can think of would be cool is something XMPP based with email fallback.
Yes, good point about email being complex to implement - both the server and the clients that use it. Example: Some years back, I remember reading a book (I think from O'Reilly) about using programming email, by an author called David Wood [1].
Update: I just did this google search:
java and email book david wood
and looked at the results.
The first hit is that book - it's name was actually Programming Internet Email - and it was by O'Reilly Media:
http://shop.oreilly.com/product/9781565924796.do
It has info on both the fundamental concepts and protocols of email, as well as examples in both Perl and Java.
Other results of that search show that there are multiple books about programming email (as can be expected), including a list of books about Internet email on an Oracle site:
Books on the JavaMail API and Internet Mail:
http://www.oracle.com/technetwork/java/javamail/index-139773...
Not all of those are exclusively about email, and some probably have overlapping content, but still, it shows (as Wood's book did too) that it is a big topic:
JavaMail API, by Elliotte Rusty Harold.
Internet Email, by David Wood.
Programmer's Guide to Internet Mail, by John Rhoton.
Internet Email Protocols: A Developer's Guide, by Kevin Johnson.
Java Network Programming, 2nd Edition, by Elliotte Rusty Harold.
Essential Email Standards: RFCs and Protocols Made Practical, by Pete Loshin.
Internet Messaging: From the Desktop to the Enterprise, by Marshall T. Rose and David Strom.
Internet E-Mail: Protocols, Standards, & Implementation, by Lawrence Hughes.
Managing IMAP, by Dianna Mullet and Kevin Mullet.
Professional Java Server Programming J2EE Edition.
Java Cookbook: Solutions and Examples for Java Developers, by Ian F. Darwin.
JavaServer Pages (JSP) Fast & Easy Web Development, by Aneesha Bakharia.
[1] From what I read about David Wood, he seemed to have a lot of experience with email (and other stuff) and had for some time, a company providing products and/or services in that area.
http://www.oreilly.com/pub/au/565
And that Java Cookbook by Ian Darwin is pretty good too - I have it.
Update: I just did this google search:
java and email book david wood
and looked at the results.
The first hit is that book - it's name was actually Programming Internet Email - and it was by O'Reilly Media:
http://shop.oreilly.com/product/9781565924796.do
It has info on both the fundamental concepts and protocols of email, as well as examples in both Perl and Java.
Other results of that search show that there are multiple books about programming email (as can be expected), including a list of books about Internet email on an Oracle site:
Books on the JavaMail API and Internet Mail:
http://www.oracle.com/technetwork/java/javamail/index-139773...
Not all of those are exclusively about email, and some probably have overlapping content, but still, it shows (as Wood's book did too) that it is a big topic:
JavaMail API, by Elliotte Rusty Harold.
Internet Email, by David Wood.
Programmer's Guide to Internet Mail, by John Rhoton.
Internet Email Protocols: A Developer's Guide, by Kevin Johnson.
Java Network Programming, 2nd Edition, by Elliotte Rusty Harold.
Essential Email Standards: RFCs and Protocols Made Practical, by Pete Loshin.
Internet Messaging: From the Desktop to the Enterprise, by Marshall T. Rose and David Strom.
Internet E-Mail: Protocols, Standards, & Implementation, by Lawrence Hughes.
Managing IMAP, by Dianna Mullet and Kevin Mullet.
Professional Java Server Programming J2EE Edition.
Java Cookbook: Solutions and Examples for Java Developers, by Ian F. Darwin.
JavaServer Pages (JSP) Fast & Easy Web Development, by Aneesha Bakharia.
[1] From what I read about David Wood, he seemed to have a lot of experience with email (and other stuff) and had for some time, a company providing products and/or services in that area.
http://www.oreilly.com/pub/au/565
And that Java Cookbook by Ian Darwin is pretty good too - I have it.
>it's name was
Sorry, "its name was" :)
Still forget that rule sometimes.
http://data.grammarbook.com/blog/pronouns/1-grammar-error/
Sorry, "its name was" :)
Still forget that rule sometimes.
http://data.grammarbook.com/blog/pronouns/1-grammar-error/
Mailgun works with SMTP [0]. Using SMTP means you're not locking yourself into one vendor, in case Mailgun does something like Mandrill where they dramatically increase the price
[0] https://documentation.mailgun.com/en/latest/quickstart-sendi...
[0] https://documentation.mailgun.com/en/latest/quickstart-sendi...
Sendmail is everything but simple. The question was SMTP in general, which indeed is an extremely simple protocol. Here, this is how you send a plain text* mail in python: https://petermolnar.net/not-mime-email-python-3/
Using an HTTP API for this is a massive overkill.
* I'm aware using utf-8 like this may confuse older clients but I didn't have problem with anything relatively modern I tested the receiving on.
Using an HTTP API for this is a massive overkill.
* I'm aware using utf-8 like this may confuse older clients but I didn't have problem with anything relatively modern I tested the receiving on.
nowadays you have the option of using postfix+dovecot, which for me at least seems to be a lot easier.
Sending is not the tricky part, implementing it right is probably another story.
I wouldn't call it massive overkill. REST is extremely simple as well. You have to make an request anyways so it's just a question of preference how you connect to that the email sending service.
I wouldn't call it massive overkill. REST is extremely simple as well. You have to make an request anyways so it's just a question of preference how you connect to that the email sending service.
> implementing it right is probably another story
Do you implement your own HTTP communication every time you need it or do you use a library?
Just as for HTTP there are many good libraries for SMTP communication that you can easily use. No need to roll your own.
Do you implement your own HTTP communication every time you need it or do you use a library?
Just as for HTTP there are many good libraries for SMTP communication that you can easily use. No need to roll your own.
> I wouldn't call it massive overkill.
You're running a web server to emulate telnet. It's a massive overkill.
You're running a web server to emulate telnet. It's a massive overkill.
Nobody I know in the SA community uses sendmail much anymore. Maybe there are a few holdouts but if someone gave me the task of standing up an MTA it would be postfix.
We were sending email via SMTP from our Android apps for a while until we ran into quite a few cases where SMTP was blocked at the router level. We had to move to HTTP mailing to let the mails go through.
In our case the reason we switched from SMTP to an HTTP API was mostly speed. At least for Sendgrid, their HTTP API is much faster than SMTP.
In general I agree that I'm not seeing a big advantage in having another service to keep up and maintain, just to wrap SMTP.
In general I agree that I'm not seeing a big advantage in having another service to keep up and maintain, just to wrap SMTP.
For instance to send messages from client only web applications where the only thing you have is an AJAX call.
For example to make it possible for a static (client only) web app to send mail. Watch out for spammers though! It should be secure by default, because most people either don't care or don't know how to change the defaults!
Gmail is IMAP and SMTP, and Google Talk is (or was) XMPP, all on port 80 in the same browser window. Others are more metaphorical. Twitter is IRC on port 80, although one primarily listens to users instead of channels — but listening to channels is still possible, if one uses a client that can follow hashtags, and the syntax is even the same. Dropbox is FTP on port 80. Reddit is Usenet on port 80.
https://medium.com/@maradydd/on-port-80-d8d6d3443d9a
https://medium.com/@maradydd/on-port-80-d8d6d3443d9a
> SMTP really is a very simple protocol, that's matured for decades now. Also it is fully specified by RFCs.
So simple and so mature it needs an RFC.
I'll take the http API I can figure out in 10 seconds because I make API requests all day over some ancient tech I don't even want to begin to understand.
Does that make me lazy and a shit developer? Probably. Do I care. Definitely not.
So simple and so mature it needs an RFC.
I'll take the http API I can figure out in 10 seconds because I make API requests all day over some ancient tech I don't even want to begin to understand.
Does that make me lazy and a shit developer? Probably. Do I care. Definitely not.
Having an RFC hardly means it's not simple; it means it has a well-defined specification.
Do you actually think HTTP isn't specified in an RFC?
Do you actually think HTTP isn't specified in an RFC?
The total opposite of this project would be way more useful :
A sendmail replacement that can call a REST / Webhook endpoint. You'll get free alerting directly from crontab/smartd/zfs to your monitoring / logging system
A sendmail replacement that can call a REST / Webhook endpoint. You'll get free alerting directly from crontab/smartd/zfs to your monitoring / logging system
Kenneth Reitz's inbox.py does 99% of that for you. Just add a call to requests (also by reitz) and you are good.
I've worked with APIs like this. The email gets more complicated, and inevitably, the API starts breaking down. E.g., with this API, how do I send attachments?
To essentially every email endpoint in the world, I always end up wondering: why is there not just an endpoint:
(Of course, this starts to very much resembled SMTP. I'm frankly okay with the above: I have tons of readily available tooling for HTTP, and I understand how TLS works and doesn't with it. But I do also understand why some might balk at this with a "that's just SMTP!")
(And just to note: I don't want to trivialize building an email. Email's format is super complex. But I have in my standard library a module that handles that for me…)
To essentially every email endpoint in the world, I always end up wondering: why is there not just an endpoint:
POST /email?from=<sender>&to=<recipient1>&to=<recipient2>
Content-Type: message/rfc822
… the actual email to send, encoded according to RFC 822. I.e., an email.
(Of course, this starts to very much resembled SMTP. I'm frankly okay with the above: I have tons of readily available tooling for HTTP, and I understand how TLS works and doesn't with it. But I do also understand why some might balk at this with a "that's just SMTP!")
(And just to note: I don't want to trivialize building an email. Email's format is super complex. But I have in my standard library a module that handles that for me…)
The example in the readme uses curl to send an email.
But curl can already send email through SMTP directly.
I'm sure the author has a good reason for creating this but it needs a better example.
But curl can already send email through SMTP directly.
I'm sure the author has a good reason for creating this but it needs a better example.
Good catch. I suppose it's because curl seems to be the standard tool when showing an example for a REST API.
Is this authless? You just send mail as the account configured and anyone can send mail?
A service like this might only be exposed internally to trusted servers. Depends on the setup.
> This service defaults to no authentication.
That default really did not work out well for MongoDB[1]. Why are we perpetuating this mistake?
[1] https://www.bleepingcomputer.com/news/security/mongodb-apoca...
[1] https://www.bleepingcomputer.com/news/security/mongodb-apoca...
Perhaps, more importantly, that didn't work out very well for smtp either ; aka "open relay".
I haven't tried it myself (and won't) but given this section "Passing Secrets" in the README [1] I dont think its authless.
[1] https://github.com/dthree/mailit/blob/master/readme.md#passi...
[1] https://github.com/dthree/mailit/blob/master/readme.md#passi...
Those secrets are for Mailit itself to connect and authenticate against the email server.
[deleted]
I wouldn't add another HTTP layer for a functionality that is within my own app / network, unless I were to expose this as a service for outside the network as a service.
I would think that this is bad design - call a http service that wraps smtp, when pretty much every language has an SMTP client.
I would think that this is bad design - call a http service that wraps smtp, when pretty much every language has an SMTP client.
I really like this library (from a conceptual level) id love to see more "drop in" API's, rather than proxying to a bunch of other services.
Could do with an end-point to do some basic validation of email addresses. I.e a syntax check, and also a DNS check to make sure that the domain has MX or A/AAAA records, and potentially a check to make sure that there is an SMTP server listening at the destination on port 25. Perhaps all 3 of those things configurable at request time.
The latter is a terrible idea: Mail servers as well as networks can and do fail temporarily, that should not prevent you from sending emails to addresses hosted on those servers, that is exactly what MTA queuing is there for.
I'm not going to make assumptions about other peoples use cases. If your use case demands that an email be immediately accepted, then the idea is not necessarily terrible.
Regardless, that's why I said it should be configurable.
Regardless, that's why I said it should be configurable.
But it doesn't give you that guarantee. Just because you can connect now, does not mean you can actually send an email later, let alone that particular email. If you want that, then you should provide feedback whether the email that you want to send actually was accepted by the destination server. But even that is probably not a sensible idea: Just because the MX accepted the messages does not in any way guarantee that it will actually be delivered immediately.
Nothing you've said is non-obvious. What if the thing you want to do is not "send an email", but is: "test if an address is likely to be able to receive an email". Could you conceive of a scenario where a message pops up to a user stating:
"It looks like you may have entered your email address incorrectly. Are you sure it is ok?"
I'm sure there are lots of use cases. Just because you can't think of one does not mean that none exist.
"It looks like you may have entered your email address incorrectly. Are you sure it is ok?"
I'm sure there are lots of use cases. Just because you can't think of one does not mean that none exist.
Except that I have seen way too many incompetent uses of APIs that support bad ideas, and so, in reality, everyone would use "the best setting!", aka, all validations switched on, and thus probably lead to all kinds of false positives with hard rejects. See also the common rejection of +-suffix addresses, for example.
For address validation, connection checks are a bad idea. For address validation, turning any temporary failure into something that could end up as a hard reject is a bad idea.
For address validation, connection checks are a bad idea. For address validation, turning any temporary failure into something that could end up as a hard reject is a bad idea.
I'm not a fan of leaving out useful features because bad developers don't RTFM. This is all hypothetical though. I don't personally need the feature, it was just an idea.
We wrote a similar service in Erlang [1] which is used for bug reporting of client side software (since most corporate networks block outgoing SMTP).
[1] https://gitHub.com/lindenbaum/http2smtp
[1] https://gitHub.com/lindenbaum/http2smtp
There are no hypermedia controls. This isn't REST, merely HTTP (with tight and brittle coupling).
Love the idea.. although we can call SMTP directly, having the SMTP APIs encapsulated with a HTTP interface is pretty neat. Plus its a readymade microservice that you just spin up and use.
I worked for 2 years on a monolith that used to call SMTP directly and it was a pain to debug.
I worked for 2 years on a monolith that used to call SMTP directly and it was a pain to debug.
Built this for a hackathon the other day. The ability to send email attachments is a must.
Is there some kind of a template out there to create the API documentation used by this project?
Looks like it is using https://www.npmjs.com/package/swagpi
Like a reverse ssmtp?
Let's say you want to build a first beta. You are going to use your own server for sending mail to reduce costs and also not introduce dependencies early on.
There are two ways to do it:
1. Call the SMTP library like some here suggested.
2. Create your own encapsulation of the SMTP functionality, and call these functions to send email.
It is obvious why you want to use 2. But, from experience, it makes your code base bigger and debugging more problematic.
By using a Docker container, you are separating matters: Now sending emails is a small service that can scale in the future, but doesn't consume lots of resources today.
With a docker/rest api, it also means that have you decided to switch to Mailchimp in the future; it'll be an easy task. You don't have to go through all your code and look which parts send emails and update them. You don't have to update your library encapsulation. You don't have to redeploy your whole instance to update the code.
All you do is create a new docker instance and then shutdown the old one. Tada!