JMAP: It’s Like IMAP but Not Really (2019)(unencumberedbyfacts.com)
unencumberedbyfacts.com
JMAP: It’s Like IMAP but Not Really (2019)
https://unencumberedbyfacts.com/2019/01/24/jmap-its-like-imap-but-not-really/
136 comments
I have more faith in JMAP's eventual success than you do, but I do agree that it's frustratingly slow. We knew that adoption would be frustrating slow of course, but I do hope that in 3 years' time we'll have a much better story than we have had in the three years since initial standardisation of the core and mail parts.
Certainly once calendars and contacts are done, and hopefully tasks as well, that will make a more complete ecosystem that is even more attractive than just the email component!
Certainly once calendars and contacts are done, and hopefully tasks as well, that will make a more complete ecosystem that is even more attractive than just the email component!
My personal take is that JMAP would start being widely used if we had some simple Python/Go bindings to easily manipulate the emails.
My first reaction as I have seen this standard popping up was not "my email will be faster/better", but "I will be able to setup a nice [email protected]" email address and have both email/web integration with it.
If developers start to embrace JMAP for a derivative usage, I think it could drive the adoption.
My first reaction as I have seen this standard popping up was not "my email will be faster/better", but "I will be able to setup a nice [email protected]" email address and have both email/web integration with it.
If developers start to embrace JMAP for a derivative usage, I think it could drive the adoption.
I think one of the cooler 'derivative' uses is the integration between fastmail and 1password via "masked email".
https://blog.1password.com/making-masked-email-with-jmap/
https://blog.1password.com/making-masked-email-with-jmap/
Are you all thinking about, or planning to allow plugins?
eg say I wanted to use jmap to implement some of hey's functionality -- for instance the screener that isolates and allows ban of new emails on first contact. Rather than a client, can I have a server that listens to my fastmail inbox and (potentially) moves emails as they're delivered?
Any chance you're thinking about allowing UI space for things like that?
eg say I wanted to use jmap to implement some of hey's functionality -- for instance the screener that isolates and allows ban of new emails on first contact. Rather than a client, can I have a server that listens to my fastmail inbox and (potentially) moves emails as they're delivered?
Any chance you're thinking about allowing UI space for things like that?
Hi Bron - love Fastmail and the innovation you leading.
Question: what do you think is the rough timing on Calendar/Sharing/Contacts/Tasks to be ratified? Within the next year, 2-years, etc?
Question: what do you think is the rough timing on Calendar/Sharing/Contacts/Tasks to be ratified? Within the next year, 2-years, etc?
Hi, I am one of the people writing and coding these standards at Fastmail. There are IETF standard drafts for JMAP calendars, contacts and tasks in the works (not just by Fastmail). We aim at publishing them in 2023. You can check the progress in the IETF calext and jmap working groups and we appreciate any input on the mailing lists!
Awesome. Thanks for much for all you and your colleagues do.
I've had it confirmed by on of the top two mobile OS providers, that their email developers have JMAP on the roadmap.
> Maybe Fastmail doesn’t have enough money and/or enough interest to make it easier to use and to evangelize it with major email providers and email client developers.
All the other major email providers are selling their own proprietary services or only offering webmail clients.
They have no interest in implementing modern open specifications. They only support things like SMTP and IMAP for legacy reasons.
All the other major email providers are selling their own proprietary services or only offering webmail clients.
They have no interest in implementing modern open specifications. They only support things like SMTP and IMAP for legacy reasons.
You are likely right about all of these things. I agree commercial E-mail providers don't currently have a reason to offer JMAP support. Thunderbird already has robust IMAP support so why change it. But JMAP provides a new avenue to E-mail client development that is both easier for existing clients to deal with and encouraging for new development.
> Outlook365 or Gmail
Why would either of these build a bridge over their lock-in moat?
Why would either of these build a bridge over their lock-in moat?
Weird comment considering how both of them already support POP3 and IMAP.
I think the cynical response would be that these a) may have added these protocols before their market dominance when it was of clearer benefit to them, and b) are widely used in business where businesses have certain demands like interoperability.
I think Microsoft is deprecating the use of anything but Outlook.
[deleted]
"to evangelize"
It's sad that superior standards need to be evangelized to even hope to gain traction.
It's sad that superior standards need to be evangelized to even hope to gain traction.
Is it too late for any significant improvements to email? It seems like, to our lament, we're headed in the direction of proprietary instant chat services and protocols.
Also, you can't overlook that a significant number of people use Gmail/Google Apps and Outlook.com/Office 365 which each use a proprietary communication protocol that is unlikely to change in the future.
So, it does make me wonder, who does JMAP benefit?
Also, you can't overlook that a significant number of people use Gmail/Google Apps and Outlook.com/Office 365 which each use a proprietary communication protocol that is unlikely to change in the future.
So, it does make me wonder, who does JMAP benefit?
JMAP benefits people who want to use broadly accepted, open standards. Especially standards better adapted to present and future needs: mobile use - lower latency & fewer round trips, lower end-user device power usage.
JMAP benefits FastMail (and others). The CEO of FastMail is also a heavy-lifter in standardizing JMAP, Bron Gondwana. It's probably appropriate to mention he's also a lead developer of the Cyrus IMAP free-software project, so it's hard to accuse him of making a private moat for his company.
Self-hosters, and small businesses, who can run software, and possibly contribute, but not write their own email software from scratch—-benefit.
JMAP is targeting email, and soon calendar, contacts, and maybe more general messaging, to run better suited to current network and user conditions.
https://jmap.io/
https://www.ietf.org/blog/jmap/
Who does TCP benefit, JSON, .well-known?
who does vcf/vCard benefit, or .ics (even if it started as a proprietary format)?
JMAP benefits FastMail (and others). The CEO of FastMail is also a heavy-lifter in standardizing JMAP, Bron Gondwana. It's probably appropriate to mention he's also a lead developer of the Cyrus IMAP free-software project, so it's hard to accuse him of making a private moat for his company.
Self-hosters, and small businesses, who can run software, and possibly contribute, but not write their own email software from scratch—-benefit.
JMAP is targeting email, and soon calendar, contacts, and maybe more general messaging, to run better suited to current network and user conditions.
https://jmap.io/
https://www.ietf.org/blog/jmap/
Who does TCP benefit, JSON, .well-known?
who does vcf/vCard benefit, or .ics (even if it started as a proprietary format)?
Thanks for the callout :) I'm not doing so much development on Cyrus any more - I still attend the weekly team meetings and do code and design review, but trying to keep my focus more on company operations and standardisation work.
JMAP core and email standards haven't changed in the past few years because they're done! In the JMAP working group at the IETF we're currently heavily focused on getting the calendaring and contacts bits finished. That unlocks significantly more benefit for a JMAP client because you don't need multiple different authentications.
As for email clients - yeah, it's been slow going, partly because when COVID hit we all shrank back into our shells a bit, and certainly my travel and networking was cut back! It's good to see how it's perceived from outside our little bubble where JMAP is doing great and we're moving more and more of Fastmail's internal APIs to be JMAP-style because it's so easy to multiplex requests together and route them separately through our middleware into different systems, then combine the results back to together in a single response to the calling system or browser.
JMAP core and email standards haven't changed in the past few years because they're done! In the JMAP working group at the IETF we're currently heavily focused on getting the calendaring and contacts bits finished. That unlocks significantly more benefit for a JMAP client because you don't need multiple different authentications.
As for email clients - yeah, it's been slow going, partly because when COVID hit we all shrank back into our shells a bit, and certainly my travel and networking was cut back! It's good to see how it's perceived from outside our little bubble where JMAP is doing great and we're moving more and more of Fastmail's internal APIs to be JMAP-style because it's so easy to multiplex requests together and route them separately through our middleware into different systems, then combine the results back to together in a single response to the calling system or browser.
The multiplexing of requests has me pretty interested.
If I made myself the spare time, or had a business model, I would love to implement chat bridges (for personal, self-hosted) small-scale use, using JMAP.
I like, I'm hopeful for the potential of, the promise of, REST-ful state transfer but with multiplexing of requests internal to the protocol.
Think of the protocol as sending a git-patch to modify state, instead of sending a diff per file, "resource".
Analogous to how QUIC (which is still adding standardization features) works around the problem of head of line blocking on TCP connections (and on http2's HTTP sequentially multiplex-able response streams), a core theme supporting in existing JMAP for email & attachments is multiple application requests per web-request, on-the-wire payload.
Said another way, I appreciate the multiplexing in: - per web request, - per application request (JMAP adds multiplexing here) - per resource, accessible to the user, on the server, - you can create, read, update, delete, move, etc.
If I made myself the spare time, or had a business model, I would love to implement chat bridges (for personal, self-hosted) small-scale use, using JMAP.
I like, I'm hopeful for the potential of, the promise of, REST-ful state transfer but with multiplexing of requests internal to the protocol.
Think of the protocol as sending a git-patch to modify state, instead of sending a diff per file, "resource".
Analogous to how QUIC (which is still adding standardization features) works around the problem of head of line blocking on TCP connections (and on http2's HTTP sequentially multiplex-able response streams), a core theme supporting in existing JMAP for email & attachments is multiple application requests per web-request, on-the-wire payload.
Said another way, I appreciate the multiplexing in: - per web request, - per application request (JMAP adds multiplexing here) - per resource, accessible to the user, on the server, - you can create, read, update, delete, move, etc.
> Especially standards better adapted to present and future needs: mobile use - lower latency & fewer round trips, lower end-user device power usage.
IMAP development started in the 1980s (RFC 1064), and was run over POTS modems for many decades: I find it hard to believe that it can't handle the 'low-end' of things.
The fact that many clients assume high-end / high-speed links is not a fault of the protocol.
IMAP development started in the 1980s (RFC 1064), and was run over POTS modems for many decades: I find it hard to believe that it can't handle the 'low-end' of things.
The fact that many clients assume high-end / high-speed links is not a fault of the protocol.
> Also, you can't overlook that a significant number of people use Gmail/Google Apps and Outlook.com/Office 365 which each use a proprietary communication protocol that is unlikely to change in the future.
All of those support IMAP currently, an open standard. Exchange has MAPI as well.
Email is the best federated system we might ever have, and it has thousands of clients that can work with nearly any provider.
JMAP has not taken off, but there is a chicken and egg problem. Most clients don't support it so there is no pressure on services to support it. Though I imagine if Gmail supported it then suddenly all the clients would support it
All of those support IMAP currently, an open standard. Exchange has MAPI as well.
Email is the best federated system we might ever have, and it has thousands of clients that can work with nearly any provider.
JMAP has not taken off, but there is a chicken and egg problem. Most clients don't support it so there is no pressure on services to support it. Though I imagine if Gmail supported it then suddenly all the clients would support it
> All of those support IMAP currently, an open standard. Exchange has MAPI as well.
Gmail supports IMAP, with custom extensions to support threading. As far as I understand it they use a custom protocol for their own gmail apps because of IMAP's shortcomings.
Its basically impossible to build a modern email app directly on top of gmail's public APIs. You can do it - but only by mirroring and re-indexing all the email.
Gmail supports IMAP, with custom extensions to support threading. As far as I understand it they use a custom protocol for their own gmail apps because of IMAP's shortcomings.
Its basically impossible to build a modern email app directly on top of gmail's public APIs. You can do it - but only by mirroring and re-indexing all the email.
Email has already largely become a newsletter and notification distribution system. Average people don't write or read emails anymore outside of some work settings but even then its usually Slack or MS Teams.
Technology and people move quickly so fossilized systems such as email get left behind. There is too much friction to improve a system which has a large number of different implementations. Where any change is hotly debated and people still resist features added to email over a decade ago. Naturally the proprietary systems win when they can roll out a new feature and have it universally work on day one.
Technology and people move quickly so fossilized systems such as email get left behind. There is too much friction to improve a system which has a large number of different implementations. Where any change is hotly debated and people still resist features added to email over a decade ago. Naturally the proprietary systems win when they can roll out a new feature and have it universally work on day one.
> Average people don't write or read emails anymore outside of some work settings but even then its usually Slack or MS Teams.
Do others believe/experience this?
I ask because this sounds like a very Bay Area comment.
Do others believe/experience this?
I ask because this sounds like a very Bay Area comment.
I write emails, usually long form (several paragraphs). And of course, I read emails too. I don't use any chat apps, nor Slack nor Teams. I regret the tendency of chat apps to integrate email; the result is that my correspondents tend to reply to my considered emails with "LOL <smiley>" or something like that.
The IMAP spec is poor; it's ambiguous and unclear. The only way to build a reliable client is to test interworking against as many IMAP servers as you can find. A better protocol would be a good thing.
The SMTP spec is NOT poor. There's no need to replace SMTP. It works fine.
It's some time since I read an explanation of JMAP (with the article doesn't provide). As far as I'm aware, it provides roughly the same service, but using HTTP and JSON. I'm not a supporter of the "replace everything with JSON" movement; I don't see the point.
I'm not that keen on the "replace everything with HTTP" movement either; if you piggy-back your new protocol on an existing protocol, there's a risk that your new application will start dragging the underlying standard along with it, to the detriment of the original application the underlying protocol was designed for. I.e., it would be regrettable if JMAP resulted in pressure on the HTTP standard to start including features motivated by the requirements of mail retrieval.
The IMAP spec is poor; it's ambiguous and unclear. The only way to build a reliable client is to test interworking against as many IMAP servers as you can find. A better protocol would be a good thing.
The SMTP spec is NOT poor. There's no need to replace SMTP. It works fine.
It's some time since I read an explanation of JMAP (with the article doesn't provide). As far as I'm aware, it provides roughly the same service, but using HTTP and JSON. I'm not a supporter of the "replace everything with JSON" movement; I don't see the point.
I'm not that keen on the "replace everything with HTTP" movement either; if you piggy-back your new protocol on an existing protocol, there's a risk that your new application will start dragging the underlying standard along with it, to the detriment of the original application the underlying protocol was designed for. I.e., it would be regrettable if JMAP resulted in pressure on the HTTP standard to start including features motivated by the requirements of mail retrieval.
I'm Australian and have worked for a variety of companies and its been over a year since someone internal at the company sent me a non broadcast newsletter email. Email is much like SMS these days, you use it to arrange job interviews or talk occasionally between businesses but you would never email someone you know directly.
I'm Australian and have the direct counter experience. Most of the business I've worked for consider Slack as ephermal.
Email is for significant communications.
Outside of work, I'm using daily for things like booking bands, talking to strata, etc.
While email in a lot of large companies might tend to be becoming a one-way distribution system; it is still vibrant and heavily used in smaller enterprises.
Email is for significant communications.
Outside of work, I'm using daily for things like booking bands, talking to strata, etc.
While email in a lot of large companies might tend to be becoming a one-way distribution system; it is still vibrant and heavily used in smaller enterprises.
How do companies that don’t use email keep track of long standing communications? Stuff that isn’t resolved within days but may take weeks, months, or even years to be resolved?
And that’s just internally. How do you handle stuff that also requires external contributors?
And that’s just internally. How do you handle stuff that also requires external contributors?
Jira and Wiki pages. A problem spanning months to years does not belong in email but somewhere public where people can join and leave the company without breaking the knowledge.
> you would never email someone you know directly.
Across countries everyone uses a different messenger app. For some whatsapp, other LINE, etc. Email is faster to make sure you reach them all.
Across countries everyone uses a different messenger app. For some whatsapp, other LINE, etc. Email is faster to make sure you reach them all.
I work in the r&d lab of a major testing/instruments manufacturer (I'm in AI but I work with engineers with expertise in radar, phased arrays, wave physics, FPGAs... so not really "SF tech"). Yet even here it seems like everyone is moving heavily towards just using teams. Now that I think about it, we only really use email when we deal with HR or to accept invitations to meetings (on teams).
In my experience this is ridiculously far off. I know that pretty much all universities still communicate through email, organisations like the IEEE as well, but more importantly how does the OP think companies communicate with external suppliers, candidates etc?
That is really a scary comment I agree. Apparently according to the user using paid private services have replaced email functionality.
Side Note: My wife's work went all in on MS Teams. They held multiple trainings and kept showing how it could handle all the functionality of email. You could schedule appointments and even share file..... Six months later everyone still uses email and MS Teams is barely used at all.... email just works.
From my end it looks like the "no one uses email anymore" is just marketing hype.
Side Note: My wife's work went all in on MS Teams. They held multiple trainings and kept showing how it could handle all the functionality of email. You could schedule appointments and even share file..... Six months later everyone still uses email and MS Teams is barely used at all.... email just works.
From my end it looks like the "no one uses email anymore" is just marketing hype.
> Do others believe/experience this?
I don't think many companies operate this way. Any formal things, e.g. regulation, finance, HR, will likely use email much more. And talking to a third party company, which some people will talk to 200 of in a year, will all be email, at least to start with.
I don't think many companies operate this way. Any formal things, e.g. regulation, finance, HR, will likely use email much more. And talking to a third party company, which some people will talk to 200 of in a year, will all be email, at least to start with.
That sounds right to me. Academia is run on email.
> Do others believe/experience this?
No. This doesn't even come close to representing my real-world experience.
No. This doesn't even come close to representing my real-world experience.
I mostly use email for sending invoices.
This is not the case. One of my jobs is taking funerals, which means that I meet a random selection of the population, of all social classes. This includes people down to about 20 years old, not just the older relatives you might expect. The only people I meet who do not use email are a minority of people over 80 years old (perhaps 3% of my total contacts), and pretty obviously they are not using the more recent proprietary platforms either. Sometimes clients will use messaging for something short like an update on the time of a meeting, and occasionally they might use Teams or Skype for meetings rather than face to face, but the two great constants are voice telephone calls and email.
I think you may have become bubbled.
I think you may have become bubbled.
>fossilized systems
What a weird way to call a technologically superior solution.
What a weird way to call a technologically superior solution.
See here for a list of software supporting JMAP: https://jmap.io/software.html
Whoa I love SMTP and email, as a distributed systems nerd I'm a huge fan of the federated protocol and it's simplicity but JMAP I'm sorry is garbage. The http/json part is great, the potential to simplify email access and use is great, even using DNS SRV records is a potential good use of DNS and a compliment to MX records. But the actual JMAP protocol is horrible. This is something built incrementally for fastmail, it's not how you design a protocol. We need to go back to square one and redesign and access and communication protocol as one thing that's ultra simplistic as a building block and starts with very basic primitives. JMAP combines too many things while being very ad-hoc and custom not taking into account a lot of progress made in things like gRPC protos, etc. Tempted to build it from scratch just to prove a point.
On a related note: the author links to his open source web mail platform Cypht. It looks cool and sounds promising. Anyone tried it? I can't remember if it has been on HN. The git history of the project is very active.
I have the Helm email appliance, and what it does _not_ come with is a webmail interface. (It does have Nextcloud, though). I put Cypht on a different box to access my Helm mail, and it was very useful.
At the moment it is down because I did a migration from one host to another, there is some sort of login problem on the new box with the old database. RSN I'll have time to diagnose and fix.
At the moment it is down because I did a migration from one host to another, there is some sort of login problem on the new box with the old database. RSN I'll have time to diagnose and fix.
Hello, author here! Not sure who posted this old blog article but great to see an interesting discussion and new interest in the project. Development is active and we always appreciate new feedback!
JMAP sounds like a crutch for mobile devices that can't keep a TCP connection open. The internet is slowly being gimped to support mobile devices.
There is a lot of writings explaining that the reason open email clients largely suck / become unmaintained is that IMAP is extremely difficult to work with. It's not just the connection, you also have to parse and work with this custom protocol as well as deal with massive inefficiencies. Doing bulk actions like moving emails between folders requires massive amounts of requests rather than a single one describing the change.
JSON protocols have completely taken over the market because they are massively easier to deal with and trivially parse in to structs which don't require a 200 page document to detail the protocol.
JSON protocols have completely taken over the market because they are massively easier to deal with and trivially parse in to structs which don't require a 200 page document to detail the protocol.
I've written multiple IMAP clients, and I agree it's a huge pain to work with. But the real issue is the API design and the fact that more advanced searching, synchronization, etc, is only available via extensions that are not guaranteed to exist on a given server.
You can write a JSON API that's equally bad, doesn't support paging, doesn't let you select just the fields you need, is missing key information in the extracted fields forcing you to parse the entire RFC822 text as a fallback, etc. Dealing with the parser is just one part of the battle.
You can write a JSON API that's equally bad, doesn't support paging, doesn't let you select just the fields you need, is missing key information in the extracted fields forcing you to parse the entire RFC822 text as a fallback, etc. Dealing with the parser is just one part of the battle.
Optional extensions pretty much signal the death of any protocol. It shows there is no longer any leadership or direction. Same thing happened to XMPP. A functioning and healthy protocol would just start with proposals and once they are well tested, bump the version number and list everything as mandatory. Then implementations can just say they will be dropping legacy_version in x months, update or become incompatible.
I think that would depend a lot on what the extensions contain. I think extensions are a decent idea when they're obscure functionality that would take a lot of work to implement.
Complexity has exploded as of late. It used to be a joke that programs grew until they became able to read mail. These days programs grow until they contain a web engine. It's not enough anymore to just have a chat window, the chat window has to generate thumbnails of webpages for you.
Complexity has exploded as of late. It used to be a joke that programs grew until they became able to read mail. These days programs grow until they contain a web engine. It's not enough anymore to just have a chat window, the chat window has to generate thumbnails of webpages for you.
Something should either be in the protocol or not. No one likes a situation where you see a feature on your client and it sometimes works or sometimes doesn't depending on the situation.
We would be better off having a smaller selection of better software over 100 different email server implementations which can't agree on a basic set of features gmail had a decade ago.
We would be better off having a smaller selection of better software over 100 different email server implementations which can't agree on a basic set of features gmail had a decade ago.
That means a protocol can not evolve in time to integrate new needs of clients and / or servers.
It can, you bump the protocol version number and move on. Give some grace period of compatibility and then shut off the old version.
That's an optional extension by any other name.
You can't "give some grace period of compatibility" in an open / distributed ecosystem, every version of the protocol lives forever.
YAML's dumb fuzzy booleans still cause issues all the time even though Yaml 1.2 was released 13 years ago.
You can't "give some grace period of compatibility" in an open / distributed ecosystem, every version of the protocol lives forever.
YAML's dumb fuzzy booleans still cause issues all the time even though Yaml 1.2 was released 13 years ago.
HTTP manages this pretty well. It's pretty common practice to turn off old http and TLS versions and things move along fine because the vast majority of users are on the latest version.
Most modern languages have a JSON encoder/decoder in their standard library too, making for one fewer dependency to worry about and that's never a bad thing.
Pretty much every device is mobile now. For example, people will probably start reading emails on the center console of their autonomous cars on the way to work over 5G.
Nevertheless, a valid current-day example for non-mobile devices requiring a tolerance to dropped connections: I regularly dock and undock my laptop when heading to meeting rooms, and simply switching from docked Ethernet to WiFi causes a network switch that breaks a lot of apps. Engineering tolerance for this kind of thing is only good.
Nevertheless, a valid current-day example for non-mobile devices requiring a tolerance to dropped connections: I regularly dock and undock my laptop when heading to meeting rooms, and simply switching from docked Ethernet to WiFi causes a network switch that breaks a lot of apps. Engineering tolerance for this kind of thing is only good.
Yes. That's reality. Most devices people use are terrible, terrible computers and they can't do what computers used to be able to do. This is a bad thing.
Honestly this sounds like a "if your computer weighs less than 5 tons and can fit on a desk, it's not a real computer" argument someone would have made 30 years ago.
Reality is people use multiple different and mobile devices, roam across connections, and intermittently go offline. Building software and protocols that are designed around this is beneficial to every use case - even those using whatever you consider a non-"terrible computer" to be.
Reality is people use multiple different and mobile devices, roam across connections, and intermittently go offline. Building software and protocols that are designed around this is beneficial to every use case - even those using whatever you consider a non-"terrible computer" to be.
I don't believe you really think that. Not being able to keep a TCP connection open is not some trivial aesthetic or nostalgic issue. Pretty much all information transfer on the internet is over TCP.
What the OP was talking about, which I totally agree with, is protocols that can't tolerate interruptions to their connection. This can be done with TCP.
When I worked in an office, I had the same issue as the OP - docking/undocking my laptop resulted in switching from wired to wireless, and connections (eg: SSH, RDP) would often drop. I ended up unplugging the ethernet at my desk and using wireless most of the time because it was just easier.
What would help is protocols actually support the concept of a "reconnection" -- ie: a TCP session that has the same identifier or token as a previous one that lets it it silently resume instead of starting fresh with new authentication. This isn't even that hard to implement if designed in from the start.
When I worked in an office, I had the same issue as the OP - docking/undocking my laptop resulted in switching from wired to wireless, and connections (eg: SSH, RDP) would often drop. I ended up unplugging the ethernet at my desk and using wireless most of the time because it was just easier.
What would help is protocols actually support the concept of a "reconnection" -- ie: a TCP session that has the same identifier or token as a previous one that lets it it silently resume instead of starting fresh with new authentication. This isn't even that hard to implement if designed in from the start.
Users don't care what you consider to be a good computer. They use the proprietary email provider and it just works and shows emails on their car. While FOSS fanboys are sitting around complaining about the decay of tech because of JSON protocols.
Less-stateful and less-connection oriented are good for things like this. Lessons from NFS and HTTP.
I would go further and also replace SMTP with the same for clients. As I understand we still need SMTP to pass mail between servers but a client already having access to IMAP/JMAP/whatever could just put the mail into the outbox folder for the server to deliver it.
Agreed, I think the separation between SMTP (as used for submission) and IMAP makes sending emails unnecessarily complicated.
Edit: From a quick glance it seems like JMAP actually supports submission.
Edit: From a quick glance it seems like JMAP actually supports submission.
There are 3 different kinds of email services that only have limited interoperability:
* Corporate email services, aka Outlook/Exchange
* Surveillance email services, aka Gmail, etc
* Real email services that expose the raw email database, such as Maildir, to the users.
For the first 2 kinds, I'd rather POP and keep my interaction with them to the minimum. For the last kind, IMAP/JMAP is too limiting in capability.
* Corporate email services, aka Outlook/Exchange
* Surveillance email services, aka Gmail, etc
* Real email services that expose the raw email database, such as Maildir, to the users.
For the first 2 kinds, I'd rather POP and keep my interaction with them to the minimum. For the last kind, IMAP/JMAP is too limiting in capability.
hrm, handful of new github stars in one day?
That's odd. Must have gotten picked up by something.
Oh, HN repost of a blog post I wrote 3 years ago, cool!
To sum up my thoughts about this thread:
Appreciate the great discussion! Wherever you land on the validity of JMAP, as a long (long) time IMAP client developer I stand by my position that JMAP is a huge improvement over legacy IMAP. Not perfect, but walks a reasonable line between what IMAP client devs understand and the modern world of APIs 30 years later. Is it REST compliant or a SOAP derivative? Most likely no and yes. Does it only benefit client developers? Probably, but that is a big benefit to the E-mail ecosystem. IMO the biggest barrier to new and innovative E-mail clients is the obtuseness of the protocol itself, and JMAP goes a long way to bridge that gap.
To sum up my thoughts about this thread:
Appreciate the great discussion! Wherever you land on the validity of JMAP, as a long (long) time IMAP client developer I stand by my position that JMAP is a huge improvement over legacy IMAP. Not perfect, but walks a reasonable line between what IMAP client devs understand and the modern world of APIs 30 years later. Is it REST compliant or a SOAP derivative? Most likely no and yes. Does it only benefit client developers? Probably, but that is a big benefit to the E-mail ecosystem. IMO the biggest barrier to new and innovative E-mail clients is the obtuseness of the protocol itself, and JMAP goes a long way to bridge that gap.
The article explains the technical advantages of JMAP, but I'm not clear how (if at all) it benefits email users. Without clear end-user advantages its hard to see why email providers would want to implement it when they already support just-good-enough IMAP.
Key things that are hard in IMAP and easy in JMAP are well summarized in this comment:
https://lobste.rs/s/er97xr/so_when_did_pop_imap_become_legac...
Global search is clearly important. The other "hard thing" are a little... niche?
AFAIK search in IMAP is folder-based, so you have to iterate over the folders, do multiple searches, and merge the results. Not elegant, and probably not fast, but it doesn't seem like a protocol killer to me.
AFAIK search in IMAP is folder-based, so you have to iterate over the folders, do multiple searches, and merge the results. Not elegant, and probably not fast, but it doesn't seem like a protocol killer to me.
One persons 'niche' is anothers 'essential feature'.
Fastmail are surely in a better position than you or I to figure out which kinds of searches are commonly performed!
Fastmail are surely in a better position than you or I to figure out which kinds of searches are commonly performed!
> Almost all requests in JMAP are to the same URL using an HTTP POST to submit a JSON body of “methods”. A method is an action or query you want to perform like “give me the 20 newest messages in this mailbox”
Can the HTTP spec just add a CALL verb already?
Can the HTTP spec just add a CALL verb already?
[deleted]
JMAP is nice if you sell self-hosting data mining software. One less connector to write.
Does JMAP do anything about the fact that it takes on average multiple whole number seconds, with frequent spikes into 10s of seconds, for email delivered to show up in people's inboxes? Source: https://status.postmarkapp.com/
If not, is anyone doing anything to improve this situation?
Sending packets over the internet across the farthest ends of the earth takes about 300ms roundtrip. What's preventing email from approaching this?
Latency is the biggest UX issue facing email IMHO, and a huge part of why it is continuing to lose ground vs alternative messaging systems, most of which are proprietary, but offer sub-second latency delivery. The world will become a worse place if the trend continues to its logical conclusion and proprietary silos eventually fully supplants email's remaining use cases.
The high average latency and even higher variance leads to people second guessing if an email they sent will ever arrive, and if an email they're expecting has ever been sent, resulting in crude workarounds like resend buttons becoming commonplace and seen as necessary, perpetuating the uncertainty around deliverability, and even further eroding trust in the system.
But most importantly, the latency problem makes email unsuitable for the most compelling use case for messaging platforms: real-time conversation.
Holding a spontaneous real-time conversation over email is painful, as anyone who has tried can probably attest. Both sides are constantly left wondering if the other side is simply taking their sweet time to reply, or if the email is just taking even longer than email usually takes to get delivered. Conversations that could have taken mere seconds end up taking minutes as a result.
This is why email has been consistently losing to messaging apps for fun conversations between friends where email's latency would suck much of the fun out of them, to Slack for mission critical work conversations where wasting time means literally wasting money, to live chat widgets for businesses looking to provide a customer support experience that delights, whereas conversations over email frequently tends to frustrate, the list goes on...
Solving the latency problem for email I believe is the key to slowing and possibly reversing the trend of email's slow but steady decline, and society will benefit massively from the continued prosperity of this rare miracle of an open system that managed to survive for so long despite all its shortcomings.
If not, is anyone doing anything to improve this situation?
Sending packets over the internet across the farthest ends of the earth takes about 300ms roundtrip. What's preventing email from approaching this?
Latency is the biggest UX issue facing email IMHO, and a huge part of why it is continuing to lose ground vs alternative messaging systems, most of which are proprietary, but offer sub-second latency delivery. The world will become a worse place if the trend continues to its logical conclusion and proprietary silos eventually fully supplants email's remaining use cases.
The high average latency and even higher variance leads to people second guessing if an email they sent will ever arrive, and if an email they're expecting has ever been sent, resulting in crude workarounds like resend buttons becoming commonplace and seen as necessary, perpetuating the uncertainty around deliverability, and even further eroding trust in the system.
But most importantly, the latency problem makes email unsuitable for the most compelling use case for messaging platforms: real-time conversation.
Holding a spontaneous real-time conversation over email is painful, as anyone who has tried can probably attest. Both sides are constantly left wondering if the other side is simply taking their sweet time to reply, or if the email is just taking even longer than email usually takes to get delivered. Conversations that could have taken mere seconds end up taking minutes as a result.
This is why email has been consistently losing to messaging apps for fun conversations between friends where email's latency would suck much of the fun out of them, to Slack for mission critical work conversations where wasting time means literally wasting money, to live chat widgets for businesses looking to provide a customer support experience that delights, whereas conversations over email frequently tends to frustrate, the list goes on...
Solving the latency problem for email I believe is the key to slowing and possibly reversing the trend of email's slow but steady decline, and society will benefit massively from the continued prosperity of this rare miracle of an open system that managed to survive for so long despite all its shortcomings.
> unsuitable for the most compelling use case for messaging platforms: real-time conversation.
If that's what you need, you use a real-time messaging application. That's not what email is for. I disapprove of the incorporation of email functionality behind a messaging UI; I get fed up of sending a properly-composed email, and getting a one-liner reply, because it's hard to compose a thought-out, formatted email on a phone.
If that's what you need, you use a real-time messaging application. That's not what email is for. I disapprove of the incorporation of email functionality behind a messaging UI; I get fed up of sending a properly-composed email, and getting a one-liner reply, because it's hard to compose a thought-out, formatted email on a phone.
Now, to build a better IMAP on top of UMAP.
I want hashes for email and not just UIDs. Does any protocol do this currently where it hashes the entire message?
Email hashes are not very useful as additional headers are added to the email every step on the way. Even if you send an email to yourself, the copy in the Sent Mail folder has a different hash than the one in Inbox.
They are useful for verification though that an email is the original, and for detecting changed messages when syncing.
At least for the verifying that an email is original I agree with the other poster, DKIM exists for that purpose, to sign emails while in transit.
As for changed messages while synching, do your messages change after delivery? If not, a simple delivered after X would be sufficient.
As for changed messages while synching, do your messages change after delivery? If not, a simple delivered after X would be sufficient.
Yes messages can change after syncing. Using mbsync for example, if you change the contents of an email locally, there are no ways to detect the change. I guess the preferred solution may be to do a more generic sync directly from the mail server and bypass IMAP entirely.. not possible with most email providers like Gmail, Microsoft, etc. Hopeful to setup my own email server someday.
DKIM offers something similar.
2019.
[deleted]
Almost all requests in JMAP are to the same URL using an HTTP POST to submit a JSON body of “methods”.
Tunnelling everything through post is the normal way to do e.g. rpc over http. It’s not a soap thing, it’s also how xml-rpc, json-rpc, and graphql work.
More generally it’s how you layer over http when you can’t necessarily guarantee respecting its requirements like purity, idempotence, or cacheability (or don’t want to bother), or when you fear hitting it’s practical implementation limits.
More generally it’s how you layer over http when you can’t necessarily guarantee respecting its requirements like purity, idempotence, or cacheability (or don’t want to bother), or when you fear hitting it’s practical implementation limits.
And SOAP inherited it from xml-rpc.
And more importantly, that's _extremely_ un-REST-like, at least in the modern use of the term.
Yeah, what can't be fixed by shoveling JSON shit into HTTP instead of a dedicated protocol?
JMAP was stupid when the Fastmail CEO or whoever was here pushing it, and it's still stupid.
JMAP was stupid when the Fastmail CEO or whoever was here pushing it, and it's still stupid.
IMAP is a protocol that you can only love if you haven’t had to try to implement it. I say “try to” because there are so many incompatible almost-implementations that you will never actually implement IMAP.
Replacing it with something more sane seems like a good use of time.
Replacing it with something more sane seems like a good use of time.
If you want to use JSON and paging but are stuck with regular IMAP servers, you can use https://EmailEngine.app between your app and the mail server.
I'd rather implement IMAP than HTTP, TLS, JSON, and whatever other shit is involved here. All of those points apply to the WWW more than anything else.
No one implements HTTP, JSON, and TLS. There are libraries for every single language imaginable. The more shared parts of the stack, the less wasted effort so you only have to implement the parts specific to sending and receiving email.
> No one implements HTTP, JSON, and TLS.
> There are libraries for every single language imaginable.
These statements are contradictory.
> There are libraries for every single language imaginable.
These statements are contradictory.
Only if you're insistent on willfully misreading them.
[no application protocol implementer] implements HTTP, JSON, and TLS [because there are well-worn implementations that can just be used for the purpose].
[no application protocol implementer] implements HTTP, JSON, and TLS [because there are well-worn implementations that can just be used for the purpose].
HTTP is a massive standard that takes thousands of man hours to implement. It's not something that "just exists" out there. I guess maybe the intent is that if you don't have Google writing your standard library you shouldn't even be able to check your mail.
I'm not a huge fan of IMAP (I wrote a client implementation for IMAPv4 a long time ago) but it's ridiculous that using what amounts to "SMTP for hypertext" as a transport layer is now common practice.
HTTP libraries do "just exist" for every usable language because almost everything is based on HTTP. Its a piece of common platform that someone puts the effort once and then you have opened up almost everything relating to network protocols. Basing your protocol on HTTP now means you have an absolutely huge amount of stuff for free. Error handling, encryption, verification, streaming, the whole set of tooling around hosting HTTP protocols like nginx, load balancers, etc.
I'd challenge you to find a single production suitable language which _doesn't_ have a http library.
I'd challenge you to find a single production suitable language which _doesn't_ have a http library.
TCP/IP is a standard that takes a long time to implement well. I've had to implement TCP/IP exactly once in my career (maybe a couple times more if you count just "lobbing" crafted datagrams), but I've just "used" existing implementations hundreds of times.
Similarly, HTTP is a big standard that takes a long time to implement well. But very few people will have to implement it: they will use existing implementations.
Similarly, HTTP is a big standard that takes a long time to implement well. But very few people will have to implement it: they will use existing implementations.
Why do you call JSON shit? It's incredibly useful and I use it all the time, am I missing something?
JSON has a few rough points that make it frustrating to parse. Like for example, most values requiring open and ending tags, no trailing commas allowed (so you must either be lax and non-compliant or backtrack the parser on last element), and other similar edge cases.
This results in making it extraordinarily easy to break most parsers in use, today. For example, to kill Python:
So when you're handling something like email, on a mobile device which may drop a partial connection, you need a hell of a lot of extra work to show the partial download, which might even be everything except the closing braces, or you have to discard the whole thing - which isn't kind to your user.
This results in making it extraordinarily easy to break most parsers in use, today. For example, to kill Python:
n="$(python3 -c 'import math; import sys; sys.stdout.write(str(math.floor(sys.getrecursionlimit() - 4)))')"
left="$(yes [ | head -n "$n" | tr -d '\n')"
echo "$left" | python3 -c 'import json; print(json.loads(input()))'
So when you're handling something like email, on a mobile device which may drop a partial connection, you need a hell of a lot of extra work to show the partial download, which might even be everything except the closing braces, or you have to discard the whole thing - which isn't kind to your user.
> JSON has a few rough points that make it frustrating to parse.
Even so, parsing JSON is a well-understood problem, and there are tons of mature libraries to parse and generate it. I'd much rather work with a JSON-based protocol than one which made up its own wacky data format.
The same goes for the use of HTTP. It may not be the ideal protocol for accessing mail, but everyone knows how to work with HTTP, and tools to interact with it are plentiful.
Even so, parsing JSON is a well-understood problem, and there are tons of mature libraries to parse and generate it. I'd much rather work with a JSON-based protocol than one which made up its own wacky data format.
The same goes for the use of HTTP. It may not be the ideal protocol for accessing mail, but everyone knows how to work with HTTP, and tools to interact with it are plentiful.
> Its JSON-centred roots and HTTP are conventional,
> so special parsers are not needed: this is quite intentional.
Two lines from a song rjbs and I made (but never published) about JMAP a few years ago, back when I was a Fastmail employee, set to the tune of the Major-General’s Song.
> so special parsers are not needed: this is quite intentional.
Two lines from a song rjbs and I made (but never published) about JMAP a few years ago, back when I was a Fastmail employee, set to the tune of the Major-General’s Song.
I almost always enjoy the faffing around "well I don't like JSON personally therefore it's a bad fit for email" in every one of these threads.
> Even so, parsing JSON is a well-understood problem, and there are tons of mature libraries to parse and generate it. I'd much rather work with a JSON-based protocol than one which made up its own wacky data format.
The same could easily be said for mbox and maildir. These have been around forever, and if you have a library for JSON, you've probably got a library for those two, as well. But parsing them tends to be less error-intensive if you hit some kind of connection interruption.
The complaint for JSON in JMAP tends to come down to - does it improve the current status quo?
The same could easily be said for mbox and maildir. These have been around forever, and if you have a library for JSON, you've probably got a library for those two, as well. But parsing them tends to be less error-intensive if you hit some kind of connection interruption.
The complaint for JSON in JMAP tends to come down to - does it improve the current status quo?
> But parsing them tends to be less error-intensive if you hit some kind of connection interruption.
Never heard of the ">From" problem? :)
mbox has a well-documented escaping problem -- many tools which parse mbox files will mistake a line which begins with the word "from" for the start of a new message. As a result, many tools which generate mbox data will insert a ">" before the start of those lines, mangling the data slightly to prevent corruption.
But that's besides the point. mbox/maildir are data storage formats, not network protocols. The corresponding network protocols (POP and IMAP) have their respective problems as well, though.
Never heard of the ">From" problem? :)
mbox has a well-documented escaping problem -- many tools which parse mbox files will mistake a line which begins with the word "from" for the start of a new message. As a result, many tools which generate mbox data will insert a ">" before the start of those lines, mangling the data slightly to prevent corruption.
But that's besides the point. mbox/maildir are data storage formats, not network protocols. The corresponding network protocols (POP and IMAP) have their respective problems as well, though.
JSON's not a network protocol, either. It's a data storage format, too. :p
Yes, they all have problems. That's... Our industry. You pick and choose your problems. Hence why it's easy to point out rough edges in JSON, and it's easy with mbox or maildir, too.
Yes, they all have problems. That's... Our industry. You pick and choose your problems. Hence why it's easy to point out rough edges in JSON, and it's easy with mbox or maildir, too.
> JSON's not a network protocol, either. It's a data storage format, too. :p
JSON is a data exchange format, it’s always been intended as a way to communicate between systems. You can use it for data storage just fine, but it’s not the primary role of the format.
Mbox was not designed as an exchange / interchange format, but as a storage format for MUA, with some possibility of interoperability.
JSON is a data exchange format, it’s always been intended as a way to communicate between systems. You can use it for data storage just fine, but it’s not the primary role of the format.
Mbox was not designed as an exchange / interchange format, but as a storage format for MUA, with some possibility of interoperability.
How well supported are mbox and maildir relative to JSON?
Do you have multiple high quality implementations for supporting mbox or maildir in nearly every language and framework in active use today?
If not, then they’re nothing like JSON.
Do you have multiple high quality implementations for supporting mbox or maildir in nearly every language and framework in active use today?
If not, then they’re nothing like JSON.
> Do you have multiple high quality implementations for supporting mbox or maildir in nearly every language and framework in active use today?
Yes.
They're builtin to Python's stdlib [0]. There's dozens for JS. Part of the PHP stdlib. [1] A number for Rust. Several for Go. And so on, and so forth.
[0] https://docs.python.org/3/library/mailbox.html
[1] https://www.php.net/manual/en/function.imap-createmailbox.ph...
Yes.
They're builtin to Python's stdlib [0]. There's dozens for JS. Part of the PHP stdlib. [1] A number for Rust. Several for Go. And so on, and so forth.
[0] https://docs.python.org/3/library/mailbox.html
[1] https://www.php.net/manual/en/function.imap-createmailbox.ph...
> Part of the PHP stdlib.
That's an IMAP library, not mbox/maildir, and it's part of an extension, not the standard library.
That's an IMAP library, not mbox/maildir, and it's part of an extension, not the standard library.
The parsing of mbox and maildir storage formats is orthogonal to the mailstore access protocol, isn't it? Does JMAP propose a new mailstore format?
No you aren't missing anything. Most HN users have some weird nostalgia about obsolete technology. Custom protocols and XML are good, JSON is bad because Facebook and Google use it to deliver adverts and it reminds them of Single Page Apps.
S-expressions are better, and something that sees so much usage should have a dedicated format. In a saner world, this work would already be done so that each protocol needn't define the octets on the line. Regardless, IMAP already exists.
It's a diseased mind that takes a shitty programming language, JavaScript, and turns it into a shitty structured data format.
It's a diseased mind that takes a shitty programming language, JavaScript, and turns it into a shitty structured data format.
IMAP is an ancient behemoth in strong need of modernization. JMAP is the replacement we have. Could modernization have been done with a dedicated protocol? I guess so. But no one (including you) did the work to get it done, so JMAP it is.
> Could modernization have been done with a dedicated protocol? I guess so.
But probably not because it adds an other layer of friction and complexity to deployment, which doesn’t do much good to your odds.
And it requires an entire additional ecosystem of inspection and protection capabilities to appear.
Not to mention the parsing and serialisation of a bespoke protocol, and the risk of all the usual framing and escaping errors in the protocol.
It also requires going to Iana for a port.
But probably not because it adds an other layer of friction and complexity to deployment, which doesn’t do much good to your odds.
And it requires an entire additional ecosystem of inspection and protection capabilities to appear.
Not to mention the parsing and serialisation of a bespoke protocol, and the risk of all the usual framing and escaping errors in the protocol.
It also requires going to Iana for a port.
> so JMAP it is
Except it isn't. As you say, IMAP could do with modernisation; JMAP isn't that, it's a replacement.
If you want to replace a protocol that's in very widespread use, then your replacement needs to offer decisive benefits to operators and users, otherwise they won't switch. JMAP seems to offer benefits only to developers.
Except it isn't. As you say, IMAP could do with modernisation; JMAP isn't that, it's a replacement.
If you want to replace a protocol that's in very widespread use, then your replacement needs to offer decisive benefits to operators and users, otherwise they won't switch. JMAP seems to offer benefits only to developers.
>As you say, IMAP could do with modernisation
In theory that's doable. In practice modernizing a protocol with extensions is though, since support ends up patchy. Especially when the big email providers have few reasons to make a documented, non-propriety extension. Why would they when they can instead push people to their propriety apps?
It's definitely possible that JMAP doesn't catch on. In that case, we'll probably just keep languishing with non-modernised IMAP.
The key is getting client support, especially in mobile (where there is a noticeable user benefit) and in open source software.
In theory that's doable. In practice modernizing a protocol with extensions is though, since support ends up patchy. Especially when the big email providers have few reasons to make a documented, non-propriety extension. Why would they when they can instead push people to their propriety apps?
It's definitely possible that JMAP doesn't catch on. In that case, we'll probably just keep languishing with non-modernised IMAP.
The key is getting client support, especially in mobile (where there is a noticeable user benefit) and in open source software.
> JMAP seems to offer benefits only to developers.
I find this argument shortsighted. If it benefits developers then it does benefit users. Making developers' lives easier means they produce code faster (users like things done sooner), have fewer bugs in their code (users don't like buggy code), and have more time to implement other features (users like featureful software). I don't see how that doesn't benefit users.
I find this argument shortsighted. If it benefits developers then it does benefit users. Making developers' lives easier means they produce code faster (users like things done sooner), have fewer bugs in their code (users don't like buggy code), and have more time to implement other features (users like featureful software). I don't see how that doesn't benefit users.
That's not the way this works. I needn't accept the WWW garbage graphomaniacs attempt to force upon others.
(shrug) I could probably say about IMAP that it looks like it has been developed by hardcore Lisp fanboys. Boils down to someone’s yum being someone else’s yuck.
Hey, I love IMAP. For all its faults it is pretty powerful and good.
I've spent a lot of time on it, but the reasons that JMAP was developed still exist and are still valid :)
I've spent a lot of time on it, but the reasons that JMAP was developed still exist and are still valid :)
I don’t see Outlook365 or Gmail (or even Apple for its iCloud email that has a relatively smaller user base) announcing any implementations of JMAP on their services. I don’t see smaller paid email service providers announce support for it. As it stands now, I don’t see a future for JMAP outside of Fastmail.
A popular client like Mozilla Thunderbird still hasn’t started implementing JMAP [1] and one of the reasons is because…the client and server documentation for JMAP haven’t been updated for a few years. [2]
So yeah, JMAP is like IMAP, but not really. We probably need a successor to JMAP that is liked enough to be implemented by email service providers and by email clients.
[1]: https://bugzilla.mozilla.org/show_bug.cgi?id=1322991
[2]: https://github.com/jmapio/jmap/issues/323