FastMail gains push notifications in iOS Mail.app, joining only Yahoo and iCloud(blog.fastmail.com)
blog.fastmail.com
FastMail gains push notifications in iOS Mail.app, joining only Yahoo and iCloud
http://blog.fastmail.com/2015/07/17/push-email-now-available-in-ios-mail/
67 comments
Really a shame that the most important access method that gives full access can't have 2FA.
Fastmail are also greedy: they monetize 2fa via sms to the tune of $0.12 (!!!) per text, and for the cherry on top, they expire your credits after a year. It's just skeevy cheap.
I'm a disappointed subscriber.
I'm a disappointed subscriber.
This doesn't really bother me because you can use Google Authenticator-style TOTP codes instead, which are free, and I don't expect Fastmail to run their own SMS infrastructure, so they presumably have to pay someone. It would be nice not to have to manually append the code to your password though, as the set-up page suggests, because this would mess up password managers. Why not detect when someone's logging in with 2-factor authentication and present an additional panel for the verification code, like most websites do?
Also agreed that it's not 'true' 2-factor authentication if you can get product support or log in with only a master password. That opens up a lot of social engineering attacks. But then, true 2-factor authentication is also a support nightmare when people lose their passwords, so I'm sympathetic, but it doesn't change the fact it's not ideal.
Also agreed that it's not 'true' 2-factor authentication if you can get product support or log in with only a master password. That opens up a lot of social engineering attacks. But then, true 2-factor authentication is also a support nightmare when people lose their passwords, so I'm sympathetic, but it doesn't change the fact it's not ideal.
I can promise you that we make no money on those SMSes, we just haven't updated the pricing since it really cost that much - and we're actually paying a minimum per-month rate that means it's costing us more than that for how rarely it's used.
Since you work there.. why did you guys choose that approach for 2fa? Reading what's written on the help page about the master password, it sounds like you want the master password to serve as a recovery code. So why not just have recovery codes instead?
Also, since you would generate the recovery codes, there wouldn't be a need to explain why the master password should be long & random; and it would reduce the risk of customers using an easy master password (either from failing to change it, or failing to use sufficient length/randomness).
[Many services w/ 2fa support recovery codes.. here's github for example: https://help.github.com/articles/downloading-your-two-factor...]
Also, since you would generate the recovery codes, there wouldn't be a need to explain why the master password should be long & random; and it would reduce the risk of customers using an easy master password (either from failing to change it, or failing to use sufficient length/randomness).
[Many services w/ 2fa support recovery codes.. here's github for example: https://help.github.com/articles/downloading-your-two-factor...]
> Since you work there.. why did you guys choose that approach for 2fa? Reading what's written on the help page about the master password, it sounds like you want the master password to serve as a recovery code. So why not just have recovery codes instead?
History, mostly. Ten years ago 2FA wasn't really a thing for most sites. We built an "alternative login" system, where you could create secondary passwords for your account and optionally set that password to be a "restricted" login. This was when using FastMail on untrusted machines (eg public libraries) was not at all uncommon, and HTTPS and even cookie-based login wasn't widespread and being able to create a throwaway password was very useful!
Over time we added paper-based OTP, Yubikey, SMS and so on but at its core its still just for secondary passwords.
We have started work on a rewriting the whole way this system works, which will work like more "conventional" 2FA systems, and will allow a second-factor on the master password. We're expecting it to be available later this year.
History, mostly. Ten years ago 2FA wasn't really a thing for most sites. We built an "alternative login" system, where you could create secondary passwords for your account and optionally set that password to be a "restricted" login. This was when using FastMail on untrusted machines (eg public libraries) was not at all uncommon, and HTTPS and even cookie-based login wasn't widespread and being able to create a throwaway password was very useful!
Over time we added paper-based OTP, Yubikey, SMS and so on but at its core its still just for secondary passwords.
We have started work on a rewriting the whole way this system works, which will work like more "conventional" 2FA systems, and will allow a second-factor on the master password. We're expecting it to be available later this year.
twilio simply doesn't charge that much
plus expiring the credits after a year (or I'd just buy $50 worth so they'd never run out at an inopportune time) is, again, just chintzy
plus expiring the credits after a year (or I'd just buy $50 worth so they'd never run out at an inopportune time) is, again, just chintzy
I haven't been happier paying for an internet service in a long time. FastMail keeps releasing more and more useful features. I can't wait for CardDAV syncing.
Even after so many years, Gmail still doesn't support push notifications in the Mail app, but FastMail threaded the needle to do it. Such a wonderful change; works great in my short testing. (I had to toggle the account on/off a few times to force it, but their blog post promises it propagates out automatically, too.)
Even after so many years, Gmail still doesn't support push notifications in the Mail app, but FastMail threaded the needle to do it. Such a wonderful change; works great in my short testing. (I had to toggle the account on/off a few times to force it, but their blog post promises it propagates out automatically, too.)
> I can't wait for CardDAV syncing.
Its actually enabled for all accounts and fully supported. We just haven't advertised it widely yet because its a big, complex feature and we've got enough people travelling at the moment that we didn't want everyone to start using it all at the same time!
Documentation is at: https://www.fastmail.com/help/clients/applist.html
You can see that it is leasted on our pricing pages: https://www.fastmail.com/signup/
We'll announce it officially in the next 2-3 weeks.
Its actually enabled for all accounts and fully supported. We just haven't advertised it widely yet because its a big, complex feature and we've got enough people travelling at the moment that we didn't want everyone to start using it all at the same time!
Documentation is at: https://www.fastmail.com/help/clients/applist.html
You can see that it is leasted on our pricing pages: https://www.fastmail.com/signup/
We'll announce it officially in the next 2-3 weeks.
I fully agree with your sentiment. One correction though, if you have a Google apps account, you can also use push mail, since it supports ActiveSync:
https://support.google.com/a/answer/135937?hl=en
So, when comparing paid and paid accounts, there is not a difference here.
(Of course, the GMail app gives push for all GMail accounts.)
https://support.google.com/a/answer/135937?hl=en
So, when comparing paid and paid accounts, there is not a difference here.
(Of course, the GMail app gives push for all GMail accounts.)
> I can't wait for CardDAV syncing.
It has been available in beta for qiuite some time really.[1]
You can start using it now if you like. No need to wait.
I'm a fastmail subscriber and this is my only way of syncing contacts on my Android-devices right now.
[1] http://blog.fastmail.com/2014/12/22/carddav-beta-release/
It has been available in beta for qiuite some time really.[1]
You can start using it now if you like. No need to wait.
I'm a fastmail subscriber and this is my only way of syncing contacts on my Android-devices right now.
[1] http://blog.fastmail.com/2014/12/22/carddav-beta-release/
We're hoping it automatically starts working for everyone - it did with our couple of test accounts and test devices, but we don't have everything from the test lab with us - we did this all very hush-hush while we're in the Bay Area. We didn't even tell the gang back in Melbourne that we were working on it until about 15 minutes before rollout.
How standards compliant is Fastmail? Gmail is sadly a trainwreck. Their IMAP implementation has tons of quirks.
Furthermore, GTalk is getting phased out in favour of proprietary Google Hangouts. GTalk was always quirky as well, and fails to support many XEPs (XMPP extensions).
Furthermore, GTalk is getting phased out in favour of proprietary Google Hangouts. GTalk was always quirky as well, and fails to support many XEPs (XMPP extensions).
FastMail's mail infrastructure is largely built on top of Cyrus IMAP (which tracks reasonably closely to RFCs). FastMail and Kolab (along with many others) also have numerous developers who contribute to Cyrus IMAP.
Sounds great, thanks.
So far the majority of devices haven't had a problem. We've got one confirmed case of it not working automatically on iOS 9 beta, and a couple of other reports that we haven't been able to identify yet. We'll keep testing.
Yay! I am so happy to hear this. Congratulations and thank you to the Fastmail team and people at Apple who made this happen. This has been my only gripe as a Fastmail customer and the only reason I haven't been using Fastmail as my main email account yet. Very cool :)
fyi: if you're a gmail user, you're likely to be disappointed with fastmail unless you are -- as I am -- using it specifically to avoid using a google product. I wish their web UI were as good as gmail's but it really isn't.
I still agree with these notes: https://news.ycombinator.com/item?id=9664594
I also think -- and I've tried almost all of them except outlook -- that it's the best non-gmail webmail available today. But that's a big caveat since gmail is free except for your privacy.
I still agree with these notes: https://news.ycombinator.com/item?id=9664594
I also think -- and I've tried almost all of them except outlook -- that it's the best non-gmail webmail available today. But that's a big caveat since gmail is free except for your privacy.
I am a Gmail user. I personally find the Fastmail interface a lot more performant; it's very responsive for me in a way that Gmail just isn't, though I'm not a Gmail 'power' user, so my experiences might be different (and I don't have 10,000s of emails in Fastmail yet). However, I can say I've found Google's Inbox product really useful a number of times recently (automatically classifying emails as travel, purchases, etc.) and I'll definitely miss that.
> Congratulations and thank you to the Fastmail team and people at Apple who made this happen.
No, sincerely no, no congratulations.
My congratulations will go to those who fight to have that new kind of notifications standardised in a RFC or to those that will reverse engineer the protocol. I am sorry, but my congratulations cannot go to those who are willing to accept _and are proud_ of being part of a closed network. This is another step in screwing even more small business and private people that want to run their own internet services. Do you like these new notifications? Abandon your independence and join our infrastructure, what do you have to lose?
No, sincerely no, no congratulations.
My congratulations will go to those who fight to have that new kind of notifications standardised in a RFC or to those that will reverse engineer the protocol. I am sorry, but my congratulations cannot go to those who are willing to accept _and are proud_ of being part of a closed network. This is another step in screwing even more small business and private people that want to run their own internet services. Do you like these new notifications? Abandon your independence and join our infrastructure, what do you have to lose?
Well, you're right, but let's put this into context: the only viable alternative at the moment is Microsoft's ActiveSync, another proprietary protocol[1]. IMAP IDLE apparently isn't sufficient for power conscious mobile devices. So what alternative do they have? FastMail's already pushing their own JMAP[2] protocol which I imagine they'd be very happy to become a standard. Perhaps everyone should wait for a standards body to come up with a solution, but personally I'd rather they be pragmatic and actually help end-users until that happens, even though I agree with you on principle.
[1] ActiveSync does have open implementations, so it's not like you're locked out of push notifications to iOS Mail if you are a private individual or small business who wants to use this functionality, it's just that for whatever reason Fastmail doesn't want to use ActiveSync, hence this alternative. But as an iOS end user, both solutions offer the same experience.
[2] http://jmap.io
[1] ActiveSync does have open implementations, so it's not like you're locked out of push notifications to iOS Mail if you are a private individual or small business who wants to use this functionality, it's just that for whatever reason Fastmail doesn't want to use ActiveSync, hence this alternative. But as an iOS end user, both solutions offer the same experience.
[2] http://jmap.io
An HTTP-based IMAP replacement makes me want to put a bullet in my brain.
IMAP is actually really good for the most part. It just has a few legacy warts (like EXPUNGE notifications being index based instead of UID-based), but a lot of the newer extensions fix these problems.
The CONDSTORE and QRESYNC extensions drastically simplify re-synchronizing the client and server and have been around for quite a few years at this point.
Honestly, if I was to design a brand new protocol for mail, it would be very close to what IMAP is today and just make a number of the newer extensions mandatory (UIDPLUS, LITERAL+ with Google's caveat for APPEND, NAMESPACE, SPECIAL-USE, SASL-IR, CONDSTORE, QRESYNC, perhaps a few others). I'd also include IDLE, but not as a distinct command - rather just have it always-on (at least as long as a folder is selected).
The #1 issue most developers have with IMAP is that every server implements a different subset of extensions. This isn't an IMAP-specific problem, however, it's just what happens to any protocol that gets as much adoption as IMAP and lives as long as IMAP has lived. There will always be new extensions for any protocol to improve upon it in various ways just like no framework API is ever complete.
Forcing all mail clients to go through HTTP for mail access is just retarded. It's depressing that so many people love the idea of using HTTP for everything. Honestly, wtf?
IMAP is actually really good for the most part. It just has a few legacy warts (like EXPUNGE notifications being index based instead of UID-based), but a lot of the newer extensions fix these problems.
The CONDSTORE and QRESYNC extensions drastically simplify re-synchronizing the client and server and have been around for quite a few years at this point.
Honestly, if I was to design a brand new protocol for mail, it would be very close to what IMAP is today and just make a number of the newer extensions mandatory (UIDPLUS, LITERAL+ with Google's caveat for APPEND, NAMESPACE, SPECIAL-USE, SASL-IR, CONDSTORE, QRESYNC, perhaps a few others). I'd also include IDLE, but not as a distinct command - rather just have it always-on (at least as long as a folder is selected).
The #1 issue most developers have with IMAP is that every server implements a different subset of extensions. This isn't an IMAP-specific problem, however, it's just what happens to any protocol that gets as much adoption as IMAP and lives as long as IMAP has lived. There will always be new extensions for any protocol to improve upon it in various ways just like no framework API is ever complete.
Forcing all mail clients to go through HTTP for mail access is just retarded. It's depressing that so many people love the idea of using HTTP for everything. Honestly, wtf?
I've worked on an IMAP server for a long time.
HTTP is just one potential transport for JMAP, the spec itself is:
* heavily based on IMAP concepts, borrowing particularly from CONDSTORE and QRESYNC but also taking the better bits from Card/CalDAV around opaque syncToken rather than forcing a sequential modseq, because that's harder to cope with distributed systems on the server with exact ordering * designed a lot more around server side sort/search, because UID ordering sucks in various ways - try moving some messages into your INBOX and noticing that iOS mail displays them in APPEND order rather than date order. * connectionless / stateless as far as possible, because connections drop in the IMAP world, particularly in mobile, and the cost of a new SELECT is high, even with QRESYNC. It still has to create a COW view of the message space because it has to keep message numbers that don't track EXPUNGEs just in case the client wants to run a pipelined command. * it's a regular datastructure rather than the abomination which is IMAP. Things like system flags starting with a \ which makes them not an atom, yet sent like an atom without quoting.
In general, IMAP has many things which are done implicitly rather than explicitly, which add cost that you can't avoid paying at both ends, even if you don't want it. Oh yeah, and it's really quite chatty. We're 280ms away from our server, so we really notice the chattyness. Go watch the video at http://jmap.io (recorded in Australia against a server in New York) and tell me you could do that via IMAP.
JMAP is a batch protocol that happens to be easy to implement over HTTP.
HTTP is just one potential transport for JMAP, the spec itself is:
* heavily based on IMAP concepts, borrowing particularly from CONDSTORE and QRESYNC but also taking the better bits from Card/CalDAV around opaque syncToken rather than forcing a sequential modseq, because that's harder to cope with distributed systems on the server with exact ordering * designed a lot more around server side sort/search, because UID ordering sucks in various ways - try moving some messages into your INBOX and noticing that iOS mail displays them in APPEND order rather than date order. * connectionless / stateless as far as possible, because connections drop in the IMAP world, particularly in mobile, and the cost of a new SELECT is high, even with QRESYNC. It still has to create a COW view of the message space because it has to keep message numbers that don't track EXPUNGEs just in case the client wants to run a pipelined command. * it's a regular datastructure rather than the abomination which is IMAP. Things like system flags starting with a \ which makes them not an atom, yet sent like an atom without quoting.
In general, IMAP has many things which are done implicitly rather than explicitly, which add cost that you can't avoid paying at both ends, even if you don't want it. Oh yeah, and it's really quite chatty. We're 280ms away from our server, so we really notice the chattyness. Go watch the video at http://jmap.io (recorded in Australia against a server in New York) and tell me you could do that via IMAP.
JMAP is a batch protocol that happens to be easy to implement over HTTP.
> * designed a lot more around server side sort/search, because UID ordering sucks in various ways - try moving some messages into your INBOX and noticing that iOS mail displays them in APPEND order rather than date order.
Why not just sort the messages? Just because iOS Mail shows the message in APPEND order doesn't you have to show them in that order. You would expect to see them in APPEND order if you are listing them in APPEND order.
As I'm sure you know, IMAP also has server-side SORT extensions. I will agree that they could be expanded a bit, but you can do client-side sorting just fine (and in fact need to if you want offline support). JMAP doesn't fix that.
SORT is really only needed on the server side for calculating which subset of messages to FETCH summary information for (to display in your message-list) if you are incrementally displaying messages as the user scrolls and you have a sort-order applied that isn't "APPEND order".
> * connectionless / stateless as far as possible, because connections drop in the IMAP world, particularly in mobile, and the cost of a new SELECT is high, even with QRESYNC. It still has to create a COW view of the message space because it has to keep message numbers that don't track EXPUNGEs just in case the client wants to run a pipelined command.
It's really not that bad.
I already mentioned the EXPUNGES being tied to sequence id's (aka indexes) as being a wart, but IIRC a newer extension replaces EXPUNGE events with VANISHED (I think it was CONDSTORE or QRESYNC) which uses UIDs and vastly reduces the chattyness of a large number of messages being expunged because it gives has a uid-set argument rather than getting 1 EXPUNGE event per message.
You argue that a stateful protocol is bad because it forces the client to re-sync with the server by re-SELECTing the folder and that it is slow even with QRESYNC. Okay, so.... what? A stateless protocol means that new messages can't arrive if I get momentarily disconnected? And no other client can change flags on messages while I was momentarily disconnected?
That's ridiculous. You need to re-sync even with a stateless client to make sure your local mirror is identical.
Sure, with a stateless protocol you don't need to do that, your client could blissfully ignore the fact that another client could have changed something, but then again, you could do the same thing with IMAP - nothing forces you to have to do anything more than re-SELECT a folder and just blindly assume everything is identical to where you left it when you got disconnected. It's foolish to do so, but it's foolish to do so even with a stateless protocol...
> Go watch the video at http://jmap.io (recorded in Australia against a server in New York) and tell me you could do that via IMAP.
Okay, I watched the video. The IMAP client on the right was fetching far more information than it needed to which is why it was so much slower. You are comparing apples to oranges.
What do I mean by that? Well, it was pretty clear that the IMAP client was fetching information for far more messages than just the 5 or 6 that it was showing on the screen which is completely unnecessary. The "snippet" portion is slow, yes, because unfortunately right now, the client must wait for the initial FETCH request to return the BODYSTRUCTURE of all of the messages so that it can figure out which MIME part contains the message text and then issue FETCH requests for each of the messages one at a time (which is why you see each snippet load individually).
There is a current discussion on solving this problem with a SNIPPET extension for IMAP so that the client could request it in its initial FETCH request to get the summary info for the messages it will be displaying on the screen.
A lot of IMAP clients are extremely inefficient in the requests that they make and the fact that very few make use of PIPELINE'd commands.
> JMAP is a batch protocol
IMAP is also a batch protocol.
Let me ask you this: you say that IMAP is really chatty, and I will agree that some parts of it are, but fetching summary information needed to populate a message list is really not that bad.
Let's take a look at a FETCH response for a single message:
* 1 FETCH (UID 1 ENVELOPE ("Fri, 17 Jul 2015 10:49:03 -0400" "This is the subject" (("Example From" NIL "from" "example.com")) (("Example Sender" NIL "sender" "example.com")) (("Example Reply-To" NIL "reply-to" "example.com")) (("Example To" NIL "to" "example.com")) (("Example Cc" NIL "cc" "example.com")) (("Example Bcc" NIL "bcc" "example.com")) "<[email protected]>" "<[email protected]>") FLAGS (\Seen \Draft \Answered \Deleted))
Now let's take a look at what this would look like in JMAP:
{ messageId: "f123u457", date: "2015-07-17T10:49:03Z", subject: "This is the subject", from: [{name: "Example From", email: "[email protected]"}], sender: [{name: "Example Sender", email: "[email protected]"}], replyTo: [{name: "Example Reply-To", email: "[email protected]"}], to: [{name: "Example To", email: "[email protected]"}], cc: [{name: "Example Cc", email: "[email protected]"}], bcc: [{name: "Example Bcc", email: "[email protected]"}], inReplyTo: "<[email protected]">, isUnseen: false, isDraft: true, isAnswered: true, isDeleted: true }
This is longer than the IMAP response!
Do you see why I called bullshit on your video comparison? It was obvious that your example IMAP client is purposely inefficient to make JMAP look good.
But let's ignore that for a moment... let's pretend JMAP is all that and a bag of chips.
What now? Since IMAP with all of the latest extensions to optimize/fix all of the problems with the original protocol have been around for years and yet many servers still do not implement them (if they did, there wouldn't really be a need for JMAP), what makes you think mail hosts will suddenly jump at the chance to implement a whole new protocol when they couldn't be bothered to implement existing extensions that would have drastically improved the user experience for their customers?
Why not just sort the messages? Just because iOS Mail shows the message in APPEND order doesn't you have to show them in that order. You would expect to see them in APPEND order if you are listing them in APPEND order.
As I'm sure you know, IMAP also has server-side SORT extensions. I will agree that they could be expanded a bit, but you can do client-side sorting just fine (and in fact need to if you want offline support). JMAP doesn't fix that.
SORT is really only needed on the server side for calculating which subset of messages to FETCH summary information for (to display in your message-list) if you are incrementally displaying messages as the user scrolls and you have a sort-order applied that isn't "APPEND order".
> * connectionless / stateless as far as possible, because connections drop in the IMAP world, particularly in mobile, and the cost of a new SELECT is high, even with QRESYNC. It still has to create a COW view of the message space because it has to keep message numbers that don't track EXPUNGEs just in case the client wants to run a pipelined command.
It's really not that bad.
I already mentioned the EXPUNGES being tied to sequence id's (aka indexes) as being a wart, but IIRC a newer extension replaces EXPUNGE events with VANISHED (I think it was CONDSTORE or QRESYNC) which uses UIDs and vastly reduces the chattyness of a large number of messages being expunged because it gives has a uid-set argument rather than getting 1 EXPUNGE event per message.
You argue that a stateful protocol is bad because it forces the client to re-sync with the server by re-SELECTing the folder and that it is slow even with QRESYNC. Okay, so.... what? A stateless protocol means that new messages can't arrive if I get momentarily disconnected? And no other client can change flags on messages while I was momentarily disconnected?
That's ridiculous. You need to re-sync even with a stateless client to make sure your local mirror is identical.
Sure, with a stateless protocol you don't need to do that, your client could blissfully ignore the fact that another client could have changed something, but then again, you could do the same thing with IMAP - nothing forces you to have to do anything more than re-SELECT a folder and just blindly assume everything is identical to where you left it when you got disconnected. It's foolish to do so, but it's foolish to do so even with a stateless protocol...
> Go watch the video at http://jmap.io (recorded in Australia against a server in New York) and tell me you could do that via IMAP.
Okay, I watched the video. The IMAP client on the right was fetching far more information than it needed to which is why it was so much slower. You are comparing apples to oranges.
What do I mean by that? Well, it was pretty clear that the IMAP client was fetching information for far more messages than just the 5 or 6 that it was showing on the screen which is completely unnecessary. The "snippet" portion is slow, yes, because unfortunately right now, the client must wait for the initial FETCH request to return the BODYSTRUCTURE of all of the messages so that it can figure out which MIME part contains the message text and then issue FETCH requests for each of the messages one at a time (which is why you see each snippet load individually).
There is a current discussion on solving this problem with a SNIPPET extension for IMAP so that the client could request it in its initial FETCH request to get the summary info for the messages it will be displaying on the screen.
A lot of IMAP clients are extremely inefficient in the requests that they make and the fact that very few make use of PIPELINE'd commands.
> JMAP is a batch protocol
IMAP is also a batch protocol.
Let me ask you this: you say that IMAP is really chatty, and I will agree that some parts of it are, but fetching summary information needed to populate a message list is really not that bad.
Let's take a look at a FETCH response for a single message:
* 1 FETCH (UID 1 ENVELOPE ("Fri, 17 Jul 2015 10:49:03 -0400" "This is the subject" (("Example From" NIL "from" "example.com")) (("Example Sender" NIL "sender" "example.com")) (("Example Reply-To" NIL "reply-to" "example.com")) (("Example To" NIL "to" "example.com")) (("Example Cc" NIL "cc" "example.com")) (("Example Bcc" NIL "bcc" "example.com")) "<[email protected]>" "<[email protected]>") FLAGS (\Seen \Draft \Answered \Deleted))
Now let's take a look at what this would look like in JMAP:
{ messageId: "f123u457", date: "2015-07-17T10:49:03Z", subject: "This is the subject", from: [{name: "Example From", email: "[email protected]"}], sender: [{name: "Example Sender", email: "[email protected]"}], replyTo: [{name: "Example Reply-To", email: "[email protected]"}], to: [{name: "Example To", email: "[email protected]"}], cc: [{name: "Example Cc", email: "[email protected]"}], bcc: [{name: "Example Bcc", email: "[email protected]"}], inReplyTo: "<[email protected]">, isUnseen: false, isDraft: true, isAnswered: true, isDeleted: true }
This is longer than the IMAP response!
Do you see why I called bullshit on your video comparison? It was obvious that your example IMAP client is purposely inefficient to make JMAP look good.
But let's ignore that for a moment... let's pretend JMAP is all that and a bag of chips.
What now? Since IMAP with all of the latest extensions to optimize/fix all of the problems with the original protocol have been around for years and yet many servers still do not implement them (if they did, there wouldn't really be a need for JMAP), what makes you think mail hosts will suddenly jump at the chance to implement a whole new protocol when they couldn't be bothered to implement existing extensions that would have drastically improved the user experience for their customers?
You put a lot of effort into your response, and I appreciate that.
But hey - that was a 36 message folder in that video. Show me an IMAP client that would do better and I'll be sufficiently impressed. They don't exist, because you can't do that.
I can show you, right now, a 500,000 message view across multiple folders infinite scrolling with JMAP and I can jump to the middle of that view and load a screenful of messages.
You CAN NOT do an IMAP sort without fetching one record for every message in the folder. If you have a million messages in a folder (yes, we have customers who do that) then you need to fetch a million records.
QRESYNC does improve matters a lot - except almost nobody implements it (we do: I fixed the initial implementation to be exactly correct in all the edge cases) - and JMAP's synchronisation is based on QRESYNC with extra protections against a flood of traffic if the server has many more changes than you expected.
I suspect your allergy to all things JSON and HTTP has coloured your attitude to the protocol, which is a pity.
(IMAP isn't a batch protocol - it supports pipelining, and that's good - you'll notice we also tag commands in JMAP)
Anyway, I look forward to your demonstration of an IMAP client fetching and preparing a view of a 36 message folder alongside our web interface loading the same folder from scratch, no local data present.
Until then, screw your calling bullshit. Go get yourself an iPhone (or ANYTHING with an IMAP client on it) and create yourself a FastMail free trial and reproduce the scenario in the video yourself. You don't need to take my word for it, it's trivial to recreate.
Basically, put up or shut up if you think IMAP is so good. If you think we're missing an IMAP extension that would make your demo work better, let me know and I'll implement it. If you need more than a month, let me know and I'll extend your trial as long as you like.
But hey - that was a 36 message folder in that video. Show me an IMAP client that would do better and I'll be sufficiently impressed. They don't exist, because you can't do that.
I can show you, right now, a 500,000 message view across multiple folders infinite scrolling with JMAP and I can jump to the middle of that view and load a screenful of messages.
You CAN NOT do an IMAP sort without fetching one record for every message in the folder. If you have a million messages in a folder (yes, we have customers who do that) then you need to fetch a million records.
QRESYNC does improve matters a lot - except almost nobody implements it (we do: I fixed the initial implementation to be exactly correct in all the edge cases) - and JMAP's synchronisation is based on QRESYNC with extra protections against a flood of traffic if the server has many more changes than you expected.
I suspect your allergy to all things JSON and HTTP has coloured your attitude to the protocol, which is a pity.
(IMAP isn't a batch protocol - it supports pipelining, and that's good - you'll notice we also tag commands in JMAP)
Anyway, I look forward to your demonstration of an IMAP client fetching and preparing a view of a 36 message folder alongside our web interface loading the same folder from scratch, no local data present.
Until then, screw your calling bullshit. Go get yourself an iPhone (or ANYTHING with an IMAP client on it) and create yourself a FastMail free trial and reproduce the scenario in the video yourself. You don't need to take my word for it, it's trivial to recreate.
Basically, put up or shut up if you think IMAP is so good. If you think we're missing an IMAP extension that would make your demo work better, let me know and I'll implement it. If you need more than a month, let me know and I'll extend your trial as long as you like.
It'll take me longer than a month to implement my own IMAP client to do things efficiently.
As it just so happens, the company I work for is having a "work on anything you want as long as it dog-foods our products" week next week and I was already planning on writing an IMAP mail client for iOS that didn't suck. Unfortunately, due to the complexity of writing a mail client, I doubt I'll finish in a week.
That said, thinking about the problem a bit more since last night, there's 1 major performance bottleneck in the IMAP protocol for this particular use-case but it could be solved with an extension to SORT.
So, as I'm sure you know, the SORT command (even with ESORT) only allows returning MIN, MAX, MODSEQ, and COUNT (from memory there might be 1 more, but I forget what it is) in addition to the matching UIDs/indexes. It's not that these aren't useful, but to optimize the "open a mailbox for the first time and display a message list to the user in the fastest way possible", they aren't all that helpful. Instead, what is needed is something more akin to the following (made up) command:
TAG001 UID SORT RETURN (OFFSET 0 LIMIT 50 FETCH (ENVELOPE FLAGS ...)) UTF-8 REVERSE DATE ALL
And this command could return untagged FETCH responses with the requested info - since we'd need a way to determine order, we could say that the FETCH results should be in the order requested or we could also have it return an untagged ESEARCH response like extended SORT commands do now to define the UID ordering.
In other words, make it so that SORT (and SEARCH, for that matter) return the information that we ultimately care about rather than the list of UIDs which we then would have to use to fetch the information we want.
(in hindsight, this is probably what you meant by "chatty" whereas I interpreted it to mean "verbosity" last night)
Basically, make it a bit more SQL-like.
Problem solved.
As it just so happens, the company I work for is having a "work on anything you want as long as it dog-foods our products" week next week and I was already planning on writing an IMAP mail client for iOS that didn't suck. Unfortunately, due to the complexity of writing a mail client, I doubt I'll finish in a week.
That said, thinking about the problem a bit more since last night, there's 1 major performance bottleneck in the IMAP protocol for this particular use-case but it could be solved with an extension to SORT.
So, as I'm sure you know, the SORT command (even with ESORT) only allows returning MIN, MAX, MODSEQ, and COUNT (from memory there might be 1 more, but I forget what it is) in addition to the matching UIDs/indexes. It's not that these aren't useful, but to optimize the "open a mailbox for the first time and display a message list to the user in the fastest way possible", they aren't all that helpful. Instead, what is needed is something more akin to the following (made up) command:
TAG001 UID SORT RETURN (OFFSET 0 LIMIT 50 FETCH (ENVELOPE FLAGS ...)) UTF-8 REVERSE DATE ALL
And this command could return untagged FETCH responses with the requested info - since we'd need a way to determine order, we could say that the FETCH results should be in the order requested or we could also have it return an untagged ESEARCH response like extended SORT commands do now to define the UID ordering.
In other words, make it so that SORT (and SEARCH, for that matter) return the information that we ultimately care about rather than the list of UIDs which we then would have to use to fetch the information we want.
(in hindsight, this is probably what you meant by "chatty" whereas I interpreted it to mean "verbosity" last night)
Basically, make it a bit more SQL-like.
Problem solved.
Yeah, OK - so you're doing what JMAP does but with a ton of search extension which STILL don't get you cross-folder support. Nice, problem solved - except you don't have a month to implement a good IMAP client, and neither does anyone else in the world.
Has ANYONE ever implemented a good IMAP client, in your opinion? If not, why not?
(JMAP is basically IMAP but more SQL-like, and without a bunch of custom syntax, yay)
Has ANYONE ever implemented a good IMAP client, in your opinion? If not, why not?
(JMAP is basically IMAP but more SQL-like, and without a bunch of custom syntax, yay)
I think plenty of people have time to implement a good IMAP client, just not me because I have a job that is completely unrelated to writing mail clients.
http://trojita.flaska.net/
two developers, 5 years, not yet feature complete - but it's the closest thing to someone attempting to write a really good IMAP client based on the RFCs, and they've been active in mailing lists and IRC too.
Funny how nobody has actually built this mythical IMAP client you say is so easy.
two developers, 5 years, not yet feature complete - but it's the closest thing to someone attempting to write a really good IMAP client based on the RFCs, and they've been active in mailing lists and IRC too.
Funny how nobody has actually built this mythical IMAP client you say is so easy.
Wow, way to attack a straw man there.
> I can show you, right now, a 500,000 message view across multiple folders infinite scrolling with JMAP and I can jump to the middle of that view and load a screenful of messages. You CAN NOT do an IMAP sort without fetching one record for every message in the folder. If you have a million messages in a folder (yes, we have customers who do that) then you need to fetch a million records.
If I've got the sort order (from a UID SORT REVERSE DATE command), and a user jumps to the middle of the list, I can FETCH just the visible subset of messages by calculating what the UIDs would be and then doing:
TAG UID FETCH <visible-uid-set> (ENVELOPE FLAGS ...)
I'm not sure why you think you'd have to issue a FETCH command for each message individually.
If I've got the sort order (from a UID SORT REVERSE DATE command), and a user jumps to the middle of the list, I can FETCH just the visible subset of messages by calculating what the UIDs would be and then doing:
TAG UID FETCH <visible-uid-set> (ENVELOPE FLAGS ...)
I'm not sure why you think you'd have to issue a FETCH command for each message individually.
Yeh, no shit - but you need to either have downloaded EVERY UID in that sort order (no updates either - if ANYTHING changes in the folder while you were disconnected you need to re-fetch the entire sort list) or you need sort extensions which don't even exist yet.
Then why did you say you couldn't if "yea, no shit" you knew you could?
If you've already grabbed ENVELOPE info for all of the older messages (i.e. all of the messages from a previous session), you can just download the ENVELOPE info of new messages and then sort locally.
I mean, if only a few messages have arrived since the previous session, it's probably faster to grab the ENVELOPEs and then sort locally.
If there's a lot of new mail, it might be more efficient to sort server side and just fetch the ENVELOPEs for the messages in view and lazy-load the rest when the user is idle (which is most of the time).
A dead-simple heuristic would be if the number of new messages is fewer than the number of messages you can show in a list, then it's probably not worth doing server-side sorting, right? Does that make sense? Especially as mobile devices get more and more powerful. You just need to design your sqlite database such that it's setup to do optimal sorting by whatever sorting method you are using (by date should be easy, right?).
I've actually done this sort of thing before (although not with IMAP), so I know it's doable. It just takes thought to design it and time to implement it.
I've studied the IMAP specs in my spare time over the past year or so and know them pretty well and I'm confident I could get good results if I had the time to implement it, but it's not like anyone in the world could implement a solid IMAP client in a week or even a month working on it full time and seeing as how I couldn't spend full-time working on writing a mail client due to the fact that I have a job that doesn't involve me writing said mail client, I don't see how you could possibly expect me to implement one in a month.
If you've already grabbed ENVELOPE info for all of the older messages (i.e. all of the messages from a previous session), you can just download the ENVELOPE info of new messages and then sort locally.
I mean, if only a few messages have arrived since the previous session, it's probably faster to grab the ENVELOPEs and then sort locally.
If there's a lot of new mail, it might be more efficient to sort server side and just fetch the ENVELOPEs for the messages in view and lazy-load the rest when the user is idle (which is most of the time).
A dead-simple heuristic would be if the number of new messages is fewer than the number of messages you can show in a list, then it's probably not worth doing server-side sorting, right? Does that make sense? Especially as mobile devices get more and more powerful. You just need to design your sqlite database such that it's setup to do optimal sorting by whatever sorting method you are using (by date should be easy, right?).
I've actually done this sort of thing before (although not with IMAP), so I know it's doable. It just takes thought to design it and time to implement it.
I've studied the IMAP specs in my spare time over the past year or so and know them pretty well and I'm confident I could get good results if I had the time to implement it, but it's not like anyone in the world could implement a solid IMAP client in a week or even a month working on it full time and seeing as how I couldn't spend full-time working on writing a mail client due to the fact that I have a job that doesn't involve me writing said mail client, I don't see how you could possibly expect me to implement one in a month.
> FastMail's already pushing their own JMAP[2] protocol which I imagine they'd be very happy to become a standard. Perhaps everyone should wait for a standards body to come up with a solution, but personally I'd rather they be pragmatic and actually help end-users until that happens, even though I agree with you on principle.
That's pretty much it. We want open standards as much as anyone, and our efforts with JMAP and building solid IMAP support into Cyrus reflect that. That said, the iOS Mail app is our #1 IMAP client by a long away, and a very common complaint is that we haven't been able to do push with it - its even been enough to stop people moving from iCloud. As a business decision its a no-brainer.
That's pretty much it. We want open standards as much as anyone, and our efforts with JMAP and building solid IMAP support into Cyrus reflect that. That said, the iOS Mail app is our #1 IMAP client by a long away, and a very common complaint is that we haven't been able to do push with it - its even been enough to stop people moving from iCloud. As a business decision its a no-brainer.
Why would you buy an iphone if you are against closed implementations and protocols?
I thought push is a standardized imap extension and any capable client can use it, if the server is configured properly. Does push on iOS require apple to somehow allow you to use it with your provider, or why is it such a big deal?
IMAP "push" usually means the IDLE extension, where a client holds a connection open and waits for the server to report that something has happened. This works ok, but isn't great on mobile because holding a TCP connection open is usually difficult on flaky networks and consumes battery.
iOS Mail doesn't implement it, instead doing a poll every 15 minutes. However it also implements a separate push system which allows true push if the server supports it. The details on how to support this aren't public information.
We talked to Apple, and they were kind enough to give us access to this system. We implemented it on our side, and now when an email arrives at FastMail we can immediately signal to Mail.app that there's new mail available.
iOS Mail doesn't implement it, instead doing a poll every 15 minutes. However it also implements a separate push system which allows true push if the server supports it. The details on how to support this aren't public information.
We talked to Apple, and they were kind enough to give us access to this system. We implemented it on our side, and now when an email arrives at FastMail we can immediately signal to Mail.app that there's new mail available.
> IMAP "push" usually means the IDLE extension [...]
> iOS Mail doesn't implement it, instead doing a poll every 15 minutes. However it also implements a separate push system which allows true push if the server supports it. The details on how to support this aren't public information.
"The details on how to support this aren't public information." My biggest computer-related fears summarized in a single sentence.
> iOS Mail doesn't implement it, instead doing a poll every 15 minutes. However it also implements a separate push system which allows true push if the server supports it. The details on how to support this aren't public information.
"The details on how to support this aren't public information." My biggest computer-related fears summarized in a single sentence.
So there's absolutely no way to enable that on custom little VPS? You have to make an agreement with Apple for that?
It looks like the interface is called XAPS and someone was able to build a working implementation last year: http://www.dovecot.org/list/dovecot/2014-September/097721.ht...
From skimming the notes it looks like it's essentially a way to send APNS notifications to Mail.app and requires a cert from Apple. You could run it from your VPS once you get the cert.
From skimming the notes it looks like it's essentially a way to send APNS notifications to Mail.app and requires a cert from Apple. You could run it from your VPS once you get the cert.
NuevaSync has supported this for years, via ActiveSync.
https://www.nuevasync.com/nuevasync-pemail.html
Disclosure: have family connection to owners.
Disclosure: have family connection to owners.
Well, wireshark + decompilation of existing software.
What I imagine, though, is a system similar to GCM, which would be not very helpful.
What I imagine, though, is a system similar to GCM, which would be not very helpful.
I guess that this is achieved using a server-side communication: the Fastmail server send the notification to the Apple servers and the Apple server sends the notification to the user's device. So wireshark probably won't help.
yeah, as I speculated, similar to GCM. Sadly, there isn’t an open standard for this.
Sounds like it uses APNS, rather than IMAP push. For the latter, you need a daemon listening, which you can't do on iOS. Plenty of Android apps have IMAP push, for example.
The article says "We’ve enabled Push IMAP" and their instructions tell you how to setup imap email account.
But yes, using APNS for notifications probably makes sense (bandwidth, less connections, battery, ...).
This is fantastic news! I can finally turn off push notifications for the FastMail iOS app and just use Mail.app (I had the iOS app installed primarily just for push notifications, although I'll keep it around for the rare case when I want to use its email management features). And FWIW, I just tested (on my iOS 9 device) and it worked great.
I'm an Android not an iOS user, so maybe I don't get the significance of push notifications.
Isn't this simply the kind of annoying notification that productivity experts tell us to turn off?
I certainly have configured all devices (phone, tablets, pc) to not ding or give notification when new mail arrives.
Isn't this simply the kind of annoying notification that productivity experts tell us to turn off?
I certainly have configured all devices (phone, tablets, pc) to not ding or give notification when new mail arrives.
In theory this should improve battery life. Instead of waking up the CPU/radios to ask whether there's new mail every few minutes, the phone lets the server push the mail to it.
Headline is faulty. Lots of third party mail providers support push in mail.app, they just need to use active sync.
I have been using mailbox.org Hvis supports active sync.
I have been using mailbox.org Hvis supports active sync.
I noticed that it doesn’t issue push notifications for emails that enter the inbox as read. I do that for a bunch of notifications and mailing list traffic, and with the FastMail iOS app, it was annoying that those would generate a push notification too.
This was the only thing stopping me moving over to FastMail, so delighted to hear this!
Welcome to 2015, where Push Notifications are a feature and not default anymore.
IMAP doesn't have "push". This is something Apple built to stop wasting your battery by trying to use the terrible IMAP IDLE
TCP Connections can be upheld in IDLE-Mode for 25 Minutes without causing any traffic.
Congratulations, but I still won't be putting my data in the US.
FastMail is an Australian company; whether that changes your feeling is another matter.
But their servers are in the U.S.
We've written about this before:
http://blog.fastmail.com/2013/10/07/fastmails-servers-are-in...
http://blog.fastmail.com/2013/10/07/fastmails-servers-are-in...
Thank you, I wouldn't want to be subsidizing tech in Sweden.
Hasn't started working for me yet.
Might try removing the fastmail account and adding it again so I can get the push option.
Really happy this has been implemented.
Really happy this has been implemented.
You can create an alternative login and put 2 factor on that, but not the master login. In theory you should never have to use the master login, and they have written pages of text justifying not having two factor on the master login, but thats not how it works in real life [0]
In real life, they require you to submit your master password to get any support. [1] .... so there I am overseas needing support and having to submit my master password over a dubious hotel connection.
They have promised real 2FA for years and years. One of these days I will ditch them for a provider who can bother supporting real 2FA.
[0] https://www.fastmail.com/help/account/2fa.html
[1] https://www.fastmail.com/html/?MSignal=Support-**