Dovecot audit complete(wiki.mozilla.org)
wiki.mozilla.org
Dovecot audit complete
https://wiki.mozilla.org/MOSS/Secure_Open_Source/Completed#dovecot
80 comments
Those are probably fairly common in image compression standards - designers of image compression are not, in general, going to make their compression less efficient than it has to be. JPEG2000 in particular is likely to be far worse.
Not at all. Decode performance is generally a very important consideration, as it has to go in all kinds of devices. Unbounded decoding time would be terrible (it would "lock up" embedded devices), and it's almost certainly an oversight.
You probably have encoding performance in mind, where there's much more liberty to make things very heavy.
You probably have encoding performance in mind, where there's much more liberty to make things very heavy.
That's a fantastic endorsement for Dovecot. It's interesting that security is not highlighted on the homepage[0], just mentioned in passing. Also interesting is the "special C variant that makes it much more difficult to write security holes"[1]
[0]: https://www.dovecot.org [1]: https://www.dovecot.org/security.html
[0]: https://www.dovecot.org [1]: https://www.dovecot.org/security.html
I was curious about what this special variant entailed and it looks to be some sensible guidelines/practices:
https://github.com/dovecot/core/blob/master/doc/securecoding...
https://github.com/dovecot/core/blob/master/doc/securecoding...
> it looks to be some sensible guidelines/practices
But more notably I think, a whole bunch of custom functions and data types to be used in place of more typical C coding practices. The document references a number of (presumably custom) header files (lib/buffer.h, lib/str.h, lib/strfuncs.h, lib/array.h, lib/data-stack.h, lib/mempool.h), the Boehm GC, and use of "GCC Extensions".
Also, there seems to be one person contributing an overwhelming proportion of commits (just going by the "contributors" page). Hmm, this seems to be an example of largely a single person implementing a program in C with a priority on code safety. Impressively successfully. But with substantial effort and individual discipline employed to avoid the pitfalls of C.
I'm not sure how scalable/transferable this kind of approach is. Perhaps the takeaway here is
i) This is an impressive (mostly) individual achievement.
ii) Dovecot may have some useful libraries for improving code safety if you're forced to or are committed to using C.
iii) The effort and discipline required to achieve code safety in C in this case, might also be interpreted as a validation of the effort and time required to achieve code safety by adopting newer alternatives like Rust and SaferCPlusPlus[1] when those choices are available.
[1] shameless plug: https://github.com/duneroadrunner/SaferCPlusPlus
But more notably I think, a whole bunch of custom functions and data types to be used in place of more typical C coding practices. The document references a number of (presumably custom) header files (lib/buffer.h, lib/str.h, lib/strfuncs.h, lib/array.h, lib/data-stack.h, lib/mempool.h), the Boehm GC, and use of "GCC Extensions".
Also, there seems to be one person contributing an overwhelming proportion of commits (just going by the "contributors" page). Hmm, this seems to be an example of largely a single person implementing a program in C with a priority on code safety. Impressively successfully. But with substantial effort and individual discipline employed to avoid the pitfalls of C.
I'm not sure how scalable/transferable this kind of approach is. Perhaps the takeaway here is
i) This is an impressive (mostly) individual achievement.
ii) Dovecot may have some useful libraries for improving code safety if you're forced to or are committed to using C.
iii) The effort and discipline required to achieve code safety in C in this case, might also be interpreted as a validation of the effort and time required to achieve code safety by adopting newer alternatives like Rust and SaferCPlusPlus[1] when those choices are available.
[1] shameless plug: https://github.com/duneroadrunner/SaferCPlusPlus
>Don't rely on input validation. Maybe you missed something. Maybe someone
calls your function somewhere else where you didn't originally intend it.
Maybe someone makes the input validation less restrictive for some reason.
Point is, it's not an excuse to cause a security hole just because input
wasn't what you expected it to be.
This point makes me mad because we have both ways to statically verify that calling conditions are "correct" through the right kind of design, and it should be a zero-cost abstraction but you can't do it in C.
Plus this advice is messy when it comes to things like string escaping. This could easily lead to issues like double encoding. Issues that don't exist if you could set up "proper" type checking in C without having to use things like wrapper structs that incur performance penalties.
Defensive programming is always a bit weird because it always feels un-necessary, but in C it's really necessary because you have so few assurances. Pretty scary
I wonder if there's a tool out there that lets you use Rust-style linear types over C code, and other things that would make some of this advice redundant (because the CI server would check the code). You could still write C, but wrong C would error in the checking step.
This point makes me mad because we have both ways to statically verify that calling conditions are "correct" through the right kind of design, and it should be a zero-cost abstraction but you can't do it in C.
Plus this advice is messy when it comes to things like string escaping. This could easily lead to issues like double encoding. Issues that don't exist if you could set up "proper" type checking in C without having to use things like wrapper structs that incur performance penalties.
Defensive programming is always a bit weird because it always feels un-necessary, but in C it's really necessary because you have so few assurances. Pretty scary
I wonder if there's a tool out there that lets you use Rust-style linear types over C code, and other things that would make some of this advice redundant (because the CI server would check the code). You could still write C, but wrong C would error in the checking step.
I don't think they meant what you described here.
> Plus this advice is messy when it comes to things like string escaping. This could easily lead to issues like double encoding.
That's escaping - it's not the same as validation. What they mean is that you shouldn't expect that the invariants you see right now to hold in the future. It still means that you should only escape right before the data sink that requires it. But on the other hand it means: escape it right before the data sink, even if you see that the caller only passes in 2 different static strings - it may change. Or escape even if the current filter before rejects anything apart from digits.
> Plus this advice is messy when it comes to things like string escaping. This could easily lead to issues like double encoding.
That's escaping - it's not the same as validation. What they mean is that you shouldn't expect that the invariants you see right now to hold in the future. It still means that you should only escape right before the data sink that requires it. But on the other hand it means: escape it right before the data sink, even if you see that the caller only passes in 2 different static strings - it may change. Or escape even if the current filter before rejects anything apart from digits.
Their advice seems to be "do not assume invariants on inputs". In which case you can only assume that your input is unescaped, right?
If you could assume escaping, that means you're limiting your input, which seems to go against the advice from my reading.
I gave escaping as an example, but one of the primary ways to reduce bugs is to reduce the scope of possible inputs as much as possible (thus reducing the scope of output to manage).
The requirements are different depending on the use case, but for application-internal code, not relying on that means that so much more global knowledge about the project is necessary to work on it. And, of course, a greater possibility of mistakes (because your requirements are more vast)
If you could assume escaping, that means you're limiting your input, which seems to go against the advice from my reading.
I gave escaping as an example, but one of the primary ways to reduce bugs is to reduce the scope of possible inputs as much as possible (thus reducing the scope of output to manage).
The requirements are different depending on the use case, but for application-internal code, not relying on that means that so much more global knowledge about the project is necessary to work on it. And, of course, a greater possibility of mistakes (because your requirements are more vast)
If you hack on the source (as I have for both fun and profit) then the secure coding style and defensive internal infrastructure quickly become obvious and natural to work with.
It's a great product, inside & out.
It's a great product, inside & out.
I think including security in the first paragraph you read can be called a highlight.
First sentence: "Dovecot is an open source IMAP and POP3 email server for Linux/UNIX-like systems, written with security primarily in mind"
First sentence: "Dovecot is an open source IMAP and POP3 email server for Linux/UNIX-like systems, written with security primarily in mind"
It actually says "secure imap server" besides the logo in the page header too.
is this is a good usecase for building this in Rust instead ?
One could argue that in 2017 software that is safety-critical and/or handles potentially untrusted input should be written in memory-safe languages. Rust is one such choice, but there are certainly other valid choices too.
But, given that dovecot just got a glowing security audit, perhaps there are other software projects that should be prioritized higher in the "rewrite everything in rust" list..
But, given that dovecot just got a glowing security audit, perhaps there are other software projects that should be prioritized higher in the "rewrite everything in rust" list..
There's currently going a discussion, whether Rust is ready yet for such a project. See http://esr.ibiblio.org/?p=7303
ESR claims: But you guys have competition out there: Go, Nim, Swift, D, and maybe other languages that aren’t on anyone’s radar yet.
All those languages are garbage collected or use automatic reference counting. Rust is unique in using its type system to automatically allocate and free memory. For the use cases where the overhead of GC is too high - Wich are the same use cases that generally call for C or C++ these days - Rust has no competition. I think this explains why Rust is still in the explore phase.
All those languages are garbage collected or use automatic reference counting. Rust is unique in using its type system to automatically allocate and free memory. For the use cases where the overhead of GC is too high - Wich are the same use cases that generally call for C or C++ these days - Rust has no competition. I think this explains why Rust is still in the explore phase.
Having a garbage collector is no showstopper for writing an email server though.
That claim gets even stronger if one considers real-time GC's or ref counting. Like how they do with Java for embedded systems. The places where Rust is uniquely advantageous gets smaller.
HN thread: https://news.ycombinator.com/item?id=13398068
It's hard to take Eric S. Raymond seriously. He starts by claiming to be responsible for the open source development model, continues to make some generic remarks about library quality that applies equally to many other established languages (including C), then has the gall to offer advice to the designers of a language that he is openly struggling with [1].
His arguments aren't all wrong, but it's weirdly presumptious to level them so unequivocally at a new language/community that is still finding its feet and whose adoption rate hasn't even reached the point where you can talk about library curation/fragmentation being a huge problem. One has to give Rust some slack just for being young and immature.
You also have to give Rust some slack for just being more different and complicated than the languages you are used to. If you read his previous post about Rust, it's clear that he's blaming Rust for his inability to grapple with its fairly advanced concepts, and his complaints mostly amount to knee-jerk reactions, not carefully reasoned criticisms.
While it's true that Rust doesn't offer a learning experience that's as friendly as Go, by far, Go has the benefit of having a head start of several years, a larger organization funding its development, and of being a much, much simpler language. Based on his skill level, it sounds like Go would be a better fit for him. (I use Go myself and see a huge benefit being that it's easy to learn for any developer, not just highly experienced ones.)
[1] http://esr.ibiblio.org/?p=7294
It's hard to take Eric S. Raymond seriously. He starts by claiming to be responsible for the open source development model, continues to make some generic remarks about library quality that applies equally to many other established languages (including C), then has the gall to offer advice to the designers of a language that he is openly struggling with [1].
His arguments aren't all wrong, but it's weirdly presumptious to level them so unequivocally at a new language/community that is still finding its feet and whose adoption rate hasn't even reached the point where you can talk about library curation/fragmentation being a huge problem. One has to give Rust some slack just for being young and immature.
You also have to give Rust some slack for just being more different and complicated than the languages you are used to. If you read his previous post about Rust, it's clear that he's blaming Rust for his inability to grapple with its fairly advanced concepts, and his complaints mostly amount to knee-jerk reactions, not carefully reasoned criticisms.
While it's true that Rust doesn't offer a learning experience that's as friendly as Go, by far, Go has the benefit of having a head start of several years, a larger organization funding its development, and of being a much, much simpler language. Based on his skill level, it sounds like Go would be a better fit for him. (I use Go myself and see a huge benefit being that it's easy to learn for any developer, not just highly experienced ones.)
[1] http://esr.ibiblio.org/?p=7294
"Discussion" isn't the word I'd use to describe that mess.
Criticizing a low-level language that makes it cheap and easy to directly call C APIs directly for not providing special syntax for select/kqueue/epoll when he could just call epoll if that's what he wants makes it clear that he wants to be writing some other, higher level language entirely.
It tells us nothing interesting about whether or not Dovecot, a project that wants low-level direct binding to POSIX APIs, is not suited for Rust.
Criticizing a low-level language that makes it cheap and easy to directly call C APIs directly for not providing special syntax for select/kqueue/epoll when he could just call epoll if that's what he wants makes it clear that he wants to be writing some other, higher level language entirely.
It tells us nothing interesting about whether or not Dovecot, a project that wants low-level direct binding to POSIX APIs, is not suited for Rust.
No.
On the one hand, the audit is a bit of a counter to "writing C is the same as writing unsafe code". On the other hand, there are a number of potential efficiency (as in, time-to-develop, maintenance effort required) gains for using a language, such as Rust, which provides the abstractions used in the C code out of the box and "for free" (in terms of implementation time--not necessarily execution performance). For a mature and stable project, especially one that has already built-in the effort to set up and maintain the appropriate abstractions, the effort to re-write is probably not worth it.
"Rewrite all the things in Rust" is asinine in general, but in this particular case it's especially bad. It would be much better to focus on C and C++ projects with actual problems.
On the one hand, the audit is a bit of a counter to "writing C is the same as writing unsafe code". On the other hand, there are a number of potential efficiency (as in, time-to-develop, maintenance effort required) gains for using a language, such as Rust, which provides the abstractions used in the C code out of the box and "for free" (in terms of implementation time--not necessarily execution performance). For a mature and stable project, especially one that has already built-in the effort to set up and maintain the appropriate abstractions, the effort to re-write is probably not worth it.
"Rewrite all the things in Rust" is asinine in general, but in this particular case it's especially bad. It would be much better to focus on C and C++ projects with actual problems.
Anyone know how much MOSS Secure Open Source (SOS) budgets a year for audits and how they select a vendor to do an audit?
EDIT - Answer to the budget appears to be $500k to start: https://blog.mozilla.org/blog/2016/06/09/help-make-open-sour...
EDIT - Answer to the budget appears to be $500k to start: https://blog.mozilla.org/blog/2016/06/09/help-make-open-sour...
https://www.mozilla.org/en-US/moss/ says the yearly budget is about $3 million.
> Mozilla Open Source Support (MOSS) is Mozilla’s program for supporting the Open Source and Free Software movement, with a yearly budget of around $3 million.
That's the yearly budget for the whole Open Source Support program.
The Secure Open Source "pillar" which saycheese was asking about is just one of three (the other two being Foundational Technology and Missing Partners) which presumably split that $3 MM.
That's the yearly budget for the whole Open Source Support program.
The Secure Open Source "pillar" which saycheese was asking about is just one of three (the other two being Foundational Technology and Missing Partners) which presumably split that $3 MM.
Agree, MOSS is the "boss" of the SOS project; as stated in theSOS's budget appears to be $500k to start per:
https://blog.mozilla.org/blog/2016/06/09/help-make-open-sour...
That's really interesting. I've been considering moving back to my own mail server from gmail for a long time but have been worried about the security issues.
This is one more thing pushing me in that direction.
This is one more thing pushing me in that direction.
I've been hosting my personal email on a $5/month DigitalOcean server, running Postfix + Dovecot, for almost two years. I think it's reasonably secure. I run updates regularly and trust the distro maintainers to release timely fixes for new vulns. Aside from that, I mostly ignore it because it works. I should probably look at the logs, but eh.
After the typical host hardening stuff (which isn't much work with modern OS defaults), I have configured SPF, DKIM, DMARC policy, opportunistic TLS for server-to-server, and mandatory TLS for IMAP client connections.
From a data privacy perspective, I know that nobody is mining the contents of my mailbox (except the messages I send to Gmail, etc. users!), and my server is not a high-value target for compromise. (Yes, DigitalOcean could snoop on my non-GPG-encrypted messages if they wanted. I guess I could migrate the server back to my own hardware.) I also encourage friends to use GPG, though this is orthogonal to one's choice of email host.
For clients I use Thunderbird on desktop and K-9 Mail on Android. Mobile push via IMAP IDLE works out of the box. (I also run a CalDAV/CardDAV server to sync contacts/calendar/todo across devices, but that is technically separate.)
Overall I'm really happy with the arrangement. The only annoyance was having my messages to Gmail users consistently marked as spam, but after doing everything suggested by mail-tester.com, I think I'm making it through most of the time.
After the typical host hardening stuff (which isn't much work with modern OS defaults), I have configured SPF, DKIM, DMARC policy, opportunistic TLS for server-to-server, and mandatory TLS for IMAP client connections.
From a data privacy perspective, I know that nobody is mining the contents of my mailbox (except the messages I send to Gmail, etc. users!), and my server is not a high-value target for compromise. (Yes, DigitalOcean could snoop on my non-GPG-encrypted messages if they wanted. I guess I could migrate the server back to my own hardware.) I also encourage friends to use GPG, though this is orthogonal to one's choice of email host.
For clients I use Thunderbird on desktop and K-9 Mail on Android. Mobile push via IMAP IDLE works out of the box. (I also run a CalDAV/CardDAV server to sync contacts/calendar/todo across devices, but that is technically separate.)
Overall I'm really happy with the arrangement. The only annoyance was having my messages to Gmail users consistently marked as spam, but after doing everything suggested by mail-tester.com, I think I'm making it through most of the time.
Inability to reliably send messages to gmail is huge. I'm using my own server, I own my domain for years, I asked a lot of people to remove "Spam" label from my messages, I did everything (reverse DNS, DKIM, SPF, checked that Gmail likes it all) I could, yet for new recipients gmail might mark my e-mail as spam. Gmail is worse e-mail provider, I liked it before, but I really don't like it, because it seems to punish people for decentralization. And I don't have this problem with other e-mail providers that I checked, they are happy to receive my e-mails.
Something is definitely missing in your setup, either entirely or just incorrectly setup.
I've been hosting my own email for about 3 years now, in an even worse-for-my-domain-reputation way: by using SRS.
Basically any email that arrives at my domain gets forwarded to my Gmail account.
And because I use SRS, it will re-write the envelope so it's something like hash-original_domain=user@mydomain.
Yet I've never had a problem with my emails getting marked as spam.
Actionable advice:
- setup DMARC and use something like dmarcian[1] so you have a pretty dashboard of your DMARC reports (otherwise you have to find some tool that will read the XML and aggregate the reports).
- use mxtoolbox to verify your IPs aren't blacklisted, you aren't an open relay, etc.
- if you have enough email traffic (not likely if you're using it for personal email only), signup for Google's Postmaster[3]
Ask if you need any help.
[1] https://dmarcian.com/
[2] http://mxtoolbox.com/
[3] https://postmaster.google.com
I've been hosting my own email for about 3 years now, in an even worse-for-my-domain-reputation way: by using SRS.
Basically any email that arrives at my domain gets forwarded to my Gmail account.
And because I use SRS, it will re-write the envelope so it's something like hash-original_domain=user@mydomain.
Yet I've never had a problem with my emails getting marked as spam.
Actionable advice:
- setup DMARC and use something like dmarcian[1] so you have a pretty dashboard of your DMARC reports (otherwise you have to find some tool that will read the XML and aggregate the reports).
- use mxtoolbox to verify your IPs aren't blacklisted, you aren't an open relay, etc.
- if you have enough email traffic (not likely if you're using it for personal email only), signup for Google's Postmaster[3]
Ask if you need any help.
[1] https://dmarcian.com/
[2] http://mxtoolbox.com/
[3] https://postmaster.google.com
I don't think vbezhenar necessarily does anything wrong. I've been (and still am?) running into the same issues with Gmail, as well as with Yahoo (but not any other providers). I'm owning domain and IPs for about a decade, using reverse DNS, DKIM, SPF, on no blacklists, also using SRS for forwarding, registered the domain with Google, not bouncing possible spam, older recipients who moved mail out of spam folder: no problem, new recipients: tagged as spam. I'm using my mail server for a few small-volume mailing lists now, with an added comment on the subscription instructions that subscribers should watch out for mails moved to the spam folder; that has been working well enough for over a year now, but I'm not sure whether that has helped alleviate the problem (I should do a test run again, but it's time consuming and worrisome to create new Gmail user accounts for testing as I fear Google linking those accounts to myself might actually contribute to the bad scoring).
I know someone who does his own mail and does less than me (e.g. no SRS), and yet he doesn't have problems and thinks I'm doing something wrong...
My best hypothesis is that Google doesn't like the network that my server is in (Hetzner, various neighbours in the network are listed on a couple blacklists). When I find the time I'll set up a server with a different provider and see if it improves things. Other hypotheses are the software used (buddy uses OpenBSD and its mail server instead of Linux and QPSMTPD/Qmail+patches, perhaps they care about software headers or are doing OS fingerprinting), the fact that I'm using a fallback server on another continent (US), network timings, and plain noise (perhaps they use some kinds of machine learning that have persistent low-level irregularities).
PS. yes I'm not doing DMARC yet, but neither does my buddy.
I know someone who does his own mail and does less than me (e.g. no SRS), and yet he doesn't have problems and thinks I'm doing something wrong...
My best hypothesis is that Google doesn't like the network that my server is in (Hetzner, various neighbours in the network are listed on a couple blacklists). When I find the time I'll set up a server with a different provider and see if it improves things. Other hypotheses are the software used (buddy uses OpenBSD and its mail server instead of Linux and QPSMTPD/Qmail+patches, perhaps they care about software headers or are doing OS fingerprinting), the fact that I'm using a fallback server on another continent (US), network timings, and plain noise (perhaps they use some kinds of machine learning that have persistent low-level irregularities).
PS. yes I'm not doing DMARC yet, but neither does my buddy.
> it (GMail) seems to punish people for decentralization.
I wonder why.
I wonder why.
My guess is that the ratio of false positives is extremely small and hence this may be a very low hanging fruit when it comes to spam filtering.
Having an honest reason to belong to a minority that is on average very dishonest is one of the worst things that can happen to you.
This is a real dilemma, not just with spam filtering.
Having an honest reason to belong to a minority that is on average very dishonest is one of the worst things that can happen to you.
This is a real dilemma, not just with spam filtering.
Shame on the big providers to force this dilemma on us.
I bet that's not the whole story. There was a reason they were lazy in the first place, but I bet there's another reason why they stay lazy. Punishing decentralization is a great way to get more users, after all: sent from gmail? works. Sent from little provider? Doesn't work.
The only way out of this I see right now is the generalization of the Freedom Box. Though even then, one would need to run protests to be able to send email from home. Between ISP wide firewalls, interdictions on home servers (by contract with some ISP), or the blacklisting of all residential IPs (Hotmail), it will take a lot of collective action before we can send mail from home again.
I bet that's not the whole story. There was a reason they were lazy in the first place, but I bet there's another reason why they stay lazy. Punishing decentralization is a great way to get more users, after all: sent from gmail? works. Sent from little provider? Doesn't work.
The only way out of this I see right now is the generalization of the Freedom Box. Though even then, one would need to run protests to be able to send email from home. Between ISP wide firewalls, interdictions on home servers (by contract with some ISP), or the blacklisting of all residential IPs (Hotmail), it will take a lot of collective action before we can send mail from home again.
>Punishing decentralization is a great way to get more users
I very much doubt that Google sees decentralization as a major threat to Gmail at this point.
I very much doubt that Google sees decentralization as a major threat to Gmail at this point.
It wouldn't take much for it to be. We basically need 3 things: a usable Freedom Box (some commercial implementation of this idea are starting to pop up), the authorization to send email, and then end of "little provider" blacklisting. The last one is basically under Google's and Microsoft's control. If they put an end to it, we now have only 2 hurdles to overcome, and both are already partly solved.
Since Snowden, people know they are being spied on. The only reason so many of them still use Gmail is because they don't know how not to. Give them a little box that's as usable as Gmail, and they will use it.
While I agree decentralization is hardly a threat to Gmail right now, it could be, and I don't see them taking any step to make it even more threatening.
Since Snowden, people know they are being spied on. The only reason so many of them still use Gmail is because they don't know how not to. Give them a little box that's as usable as Gmail, and they will use it.
While I agree decentralization is hardly a threat to Gmail right now, it could be, and I don't see them taking any step to make it even more threatening.
The pattern -- and volume -- of messages that Gmail would receive from an independent server with legitimate users would look very different than someone running a similar server to send spam.
Perhaps a spammer with a new server could gain Gmail's trust by passing genuine-looking correspondence, but this would stop working as soon as they start to send bulk messages.
Seems to me that tuning Gmail's filters to recognize and trust small-volume email servers would not be difficult or time-consuming.
Perhaps a spammer with a new server could gain Gmail's trust by passing genuine-looking correspondence, but this would stop working as soon as they start to send bulk messages.
Seems to me that tuning Gmail's filters to recognize and trust small-volume email servers would not be difficult or time-consuming.
That could well be. It's hard to know without access to the data, but they could probably do better if they tried.
However, I fear if Google were to try harder, spammers would also jump on that opportunity. It's easier to mimic the patterns of a small legit SMTP server than to pose as one of a few big well established email providers with known IP ranges. They could potentially use millions of compromised PCs, each sending spam in low volumes mixed with non-spam traffic.
However, I fear if Google were to try harder, spammers would also jump on that opportunity. It's easier to mimic the patterns of a small legit SMTP server than to pose as one of a few big well established email providers with known IP ranges. They could potentially use millions of compromised PCs, each sending spam in low volumes mixed with non-spam traffic.
Did you register with https://www.dnswl.org/? That should help with deliverability (in general, don't know about Gmail).
What CalDAV/CardDAV server do you run?
I can recommend radicale [1]. It's very simple and doesn't need a lot of resources. It also supports authentication via an IMAP server so you don't have to keep a separate user database.
1: http://radicale.org/
1: http://radicale.org/
I also use Radicale. My only annoyance is that when you use its built-in HTTP server, the process eventually stops working if one of your clients has an intermittent connection (e.g. phone with a poor signal).
https://github.com/Kozea/Radicale/issues/266 https://github.com/Kozea/Radicale/issues/388
I'm told that using a dedicated HTTP + WSGI server solves this, just haven't done the work yet.
https://github.com/Kozea/Radicale/issues/266 https://github.com/Kozea/Radicale/issues/388
I'm told that using a dedicated HTTP + WSGI server solves this, just haven't done the work yet.
Would it be easier to just have a cron job that checks every five minutes if Radicale is running and if it has fallen over restart it?
Can we stop hacking ill-conceived one-offs like this and just start to assume that a monitoring utility (Monit, Supervisor, Munin, Nagios, ServerDensity, Solarwinds to name a few) in some form is the standard for notifying of anomalies and automation of process recovery?
Moreover, systemd has a OnFailure directive that activates other units when the process exits uncleanly. Upstart has a similar directive. With a more classical approach, `respawn' is a utility that can invoke and maintain the running status of a process launched from System V Init, or you could use the regarded inittab solution.
I've seen answers to questions similar to this on StackOverflow respond with scripts that are nothing more than a while(true) loop that checks to see if a PID exists in languages ranging from Bourne shell to Python and NodeJS. This is the wrong way to do it, especially the Node one. Lets take a language intended to be used inside a web browser client and slap it onto a server to be used for process monitoring. I'm sure it does a fine job with it's while loop, but we're hitting copper pipe with a tire iron wondering why our lightbulb isn't turning on here.
Moreover, systemd has a OnFailure directive that activates other units when the process exits uncleanly. Upstart has a similar directive. With a more classical approach, `respawn' is a utility that can invoke and maintain the running status of a process launched from System V Init, or you could use the regarded inittab solution.
I've seen answers to questions similar to this on StackOverflow respond with scripts that are nothing more than a while(true) loop that checks to see if a PID exists in languages ranging from Bourne shell to Python and NodeJS. This is the wrong way to do it, especially the Node one. Lets take a language intended to be used inside a web browser client and slap it onto a server to be used for process monitoring. I'm sure it does a fine job with it's while loop, but we're hitting copper pipe with a tire iron wondering why our lightbulb isn't turning on here.
The GP's using it for personal use (I think). OP merely suggested a hack to let GP get on with their life as quickly as possible in 2 lines of bash and one crontab entry - and without learning a new process monitoring utility's configuration. Obviously this wouldn't be suitable for a production site for many users.
I have the same experience. It works perfectly for me with ldap authentication (which I also use for my mailserver) over multiple devices (macOS, Windows, Android, Linux).
Started using it after Owncloud/Nextcloud broke their ldap auth.
Started using it after Owncloud/Nextcloud broke their ldap auth.
Using it as well, pretty simple to setup & dockerization-friendly.
Not the parent, but I use Nextcloud for this (previously ownCloud). There are carddav and caldav sync applications for Android. I assume other operating systems have them too.
https://nextcloud.com
https://nextcloud.com
I've found Baïkal quite reliable over what must be a half decade of use. Integrates well with Apple devices (iOS/OS X 10.9), with the one Android I ever used, and via Lightning and Sogo Connector, also reasonably well with Thunderbird.
Thank you for that post. I've been trying some time ago, mostly for fun, to set up an email server. I did get one up and running with a setup similar to what you describe but was spam-filtered by Gmail as you describe. And because of this I kind of gave up.
But now I might try it again some time.
But now I might try it again some time.
You can expect a fresh IP from one of the cloud hosting services to be spam filtered out of the box by Google et al these days.
If you’re lucky, the IP in question hasn’t previously been used to spam & by very slowly ramping up your email rate Google will eventually decide that you’re probably OK. Getting this right appears to be something of a black art however :(
If you’re lucky, the IP in question hasn’t previously been used to spam & by very slowly ramping up your email rate Google will eventually decide that you’re probably OK. Getting this right appears to be something of a black art however :(
A discussion from long ago about Gmail deliverability:
https://news.ycombinator.com/item?id=9855030
https://news.ycombinator.com/item?id=9855030
I run my own mail server, I can tell you this: Buffer overflows and format string vulns in your imap implementation will be the least of your worries.
It is certainly still valuable work to audit the software used for the mailserver. But the challenges lie elsewhere. The biggest chunks are configuration issues and - if you have users that aren't very security savvy - stolen passwords and subsequent abuse of mail accounts for spam.
It is certainly still valuable work to audit the software used for the mailserver. But the challenges lie elsewhere. The biggest chunks are configuration issues and - if you have users that aren't very security savvy - stolen passwords and subsequent abuse of mail accounts for spam.
> configuration issues
This is where having a mail server with a sane configuration file comes handy. OpenSMTPd is simply awesome in that regard. Plus, it too was recently audited and vulnerabilities fixed so it should be secure.
OpenSMTPd + Dovecot makes for an awesome combination for a mail server.
This is where having a mail server with a sane configuration file comes handy. OpenSMTPd is simply awesome in that regard. Plus, it too was recently audited and vulnerabilities fixed so it should be secure.
OpenSMTPd + Dovecot makes for an awesome combination for a mail server.
you might be interested in https://github.com/sovereign/sovereign
Installing 10 other services doesn't seem like a good way to run a secure mail server.
Well, that setup offers a personal "cloud", not just mail. And the focus is more on "independent" than "secure".
I wouldn't worry too much about running many services. They all run as their own unprivileged users. (Of course chroots or even jails/containers would be better.)
I wouldn't worry too much about running many services. They all run as their own unprivileged users. (Of course chroots or even jails/containers would be better.)
That seems to be a great repo, both as a starting point for self hosting and as a starting point for learning ansible!
I run dovecot on my laptop, for a few years now, for local email storage. I usually move my messages down from fastmail once or twice a year.
I just recently set it up to use TLS on port 993. It was easy to do.
I just recently set it up to use TLS on port 993. It was easy to do.
Makes one wonder how they achieved this. I can find a passing mention of a "special C variant that makes it much more difficult to write security holes"[0], but no more details on whether they enforce this (and how), nor on other techniques to improve security.
Do they have a rules checker? Are they running static analysis? Which security-related compiler flags are they enabling? Other tools they use to achieve this? Or is it just a lead developer who really groks C and security, and who reviews all PRs?
It'd be interesting to see what that other projects can learn here to improve security...
[0] https://www.dovecot.org/security.html
Do they have a rules checker? Are they running static analysis? Which security-related compiler flags are they enabling? Other tools they use to achieve this? Or is it just a lead developer who really groks C and security, and who reviews all PRs?
It'd be interesting to see what that other projects can learn here to improve security...
[0] https://www.dovecot.org/security.html
Much like djb's code it's forcing the most error prone parts (string handling, memory) to go through a library, or simply forbidding the error prone constructs entirely (static buffers).
Could anyone here point me in a good direction for hosting my own mail server? Is Dovecot a good solution? I can figure out a hosting solution but I would like to know about best practices for implementation and security. Thanks in advance ~
Dovecot + postfix are a good combination. Getting whitelisted by major mail providers could be more of an issue.
If you're on DO or Linode or similar, you start with a new VPS and check the IP it's assigned in all the RBLs and spam blacklists. (Google "multi rbl check", there's a bunch.) If it's on any of them, tear down the VPS and stand up a fresh one, and repeat until you get a clean IP. Then keep it that way by not running an open relay.
Beyond that, the main deliverability improvements, in approximate order of value, are: PTR record with the hostname you send mail from; DKIM; SPF; DMARC.
My mail server runs all of these, and I've yet to encounter trouble sending to anyone, including Gmail, which is often mentioned in threads like these as being a special issue. Admittedly I've been running mail from that IP since 2004, so have a long history of not being a spammer, but as long as you start out with a clean IP and keep it so, and make sure all your sender legitimacy boxes are properly checked, I don't see any reason why you should have undue difficulty.
Beyond that, the main deliverability improvements, in approximate order of value, are: PTR record with the hostname you send mail from; DKIM; SPF; DMARC.
My mail server runs all of these, and I've yet to encounter trouble sending to anyone, including Gmail, which is often mentioned in threads like these as being a special issue. Admittedly I've been running mail from that IP since 2004, so have a long history of not being a spammer, but as long as you start out with a clean IP and keep it so, and make sure all your sender legitimacy boxes are properly checked, I don't see any reason why you should have undue difficulty.
(That's approximate descending order of value, to be clear. In particular, if you don't have a matching PTR record, you're hosed whatever else you try to do.)
Looking at the incoming mail in my server, I discovered that there are big legit mail senders who have a non-matching PTR. I suppose that at some point gmail et al stopped using that rule, or they just look for certain patterns in the PTR name, or something.
If you follow the basic steps of checking RBLs, using SPF and DKIM, you usually can do decent. I still have the occasional issue with hotmail/outlook/live mail though.
Another option if you just don't want to deal with it, but do want to run your own mail server is to use something like MailGun which can act as an SMTP relay and will be approved by those webmail providers out of the box.
Another option if you just don't want to deal with it, but do want to run your own mail server is to use something like MailGun which can act as an SMTP relay and will be approved by those webmail providers out of the box.
We're building a set of Ansible roles for self-hosting services with a complete mailserver based on dovecot and postfix. See https://wiki.base48.cz/Nestihacky and https://github.com/nestihacky/ansible-role-mailserver for more details. Originally inspired by https://github.com/sovereign/sovereign/ but with focus on Centos/Fedora (due to SELinux) and re-usability (separate Ansible roles).
Check out mailinabox – been using that for years now and it's fabulously hassle-free.
Wow, congrats. Good engineering. I remember Dovecot for being the easy option back when I ran my own mailserver.
What is a good smtp server to go with it in terms of security and reliability?
The most secure in absolute terms is qmail[1] but I found OpenSMTPd[2] to be a joy to work with, something that can not be said for any other SMTPd system I worked over the years (postfix, sendmail horror, exim, qmail and OpenSMTPd). OpenSMTPd seems to be fairly secure[3].
[1]: http://www.qmail.org/top.html
[2]: https://www.opensmtpd.org/
[3]: https://www.opensmtpd.org/security.html
[1]: http://www.qmail.org/top.html
[2]: https://www.opensmtpd.org/
[3]: https://www.opensmtpd.org/security.html
Thanks
Use postfix. It's (one of the, it not the) most common, and it's got a security secord to rival dovecot (the author is competent and cares). It integrates with everything under the sun for filtering. There's good documentation in the form of bundled text files.
Don't use qmail unless you really want to, as you need to patch it up a lot in any modern setting. And don't use sendmail if postfix will suffice. Exif will also do the job but there's really nothing it does better than the others.
Don't use qmail unless you really want to, as you need to patch it up a lot in any modern setting. And don't use sendmail if postfix will suffice. Exif will also do the job but there's really nothing it does better than the others.
This title badly needs a (semi)colon before "auditor".
I hit the 80 char limit :(
It was a struggle to get everything in there.
Was going to use DONE instead of complete, but that didn't read right.
You can drop the "by" and have
> Mozilla sponsored Dovecot audit complete; auditor Cure53 “extremely impressed”
> Mozilla sponsored Dovecot audit complete; auditor Cure53 “extremely impressed”
I had that as one the options, but wanted to have Dovecot first, not Mozilla as it is their product.
But you have given me an idea :)
EDIT: Doh, cannot edit. Email sent.
Dovecot audit: Mozilla sponsored complete. Auditor Cure53 “extremely impressed”
But you have given me an idea :)
EDIT: Doh, cannot edit. Email sent.
Dovecot audit: Mozilla sponsored complete. Auditor Cure53 “extremely impressed”
That is even more confusing. How about:
Dovecot audit complete: Mozilla sponsored auditor, Cure53 "extremely impressed"
Dovecot audit complete: Mozilla sponsored auditor, Cure53 "extremely impressed"
However, more interesting were the two medium vulnerabilities, which were initially reported as DoS bugs in the libjpeg-turbo library but on further investigation were found to be issues with the JPEG standard itself. These issues were reproduced across multiple JPEG implementations, can be triggered by entirely legal JPEGs, and so are not easy to mitigate in any JPEG library itself.