Google's CardDAV server isn't standards compliant (2014)(evertpot.com)
evertpot.com
Google's CardDAV server isn't standards compliant (2014)
https://evertpot.com/google-carddav-issues/
64 comments
Embrace, Extend, Extinguish.
I can confirm just about all of this because of a recent job in this space. Whatever you do, do not do two way sync with Google's CardDAV servers or you will end up with a lot of lost data.
I wonder if they should even be allowed to call it CardDAV since you'd expect it to actually adhere to the standard.
Google is large enough not to care about this though, as long as it works with their address books on Android phones it is probably good enough for them.
Note that this post seems to be from 2014 and the situation isn't much better today!
I wonder if they should even be allowed to call it CardDAV since you'd expect it to actually adhere to the standard.
Google is large enough not to care about this though, as long as it works with their address books on Android phones it is probably good enough for them.
Note that this post seems to be from 2014 and the situation isn't much better today!
"Google is large enough not to care about this"
They also operate "SMTP servers" that violate the applicable RFPs, have done so for many years, and are not concerned about the potential damage that can result: https://lee-phillips.org/gmailRewriting/
They also operate "SMTP servers" that violate the applicable RFPs, have done so for many years, and are not concerned about the potential damage that can result: https://lee-phillips.org/gmailRewriting/
You don’t seem to understand the standards you are complaining about. You are using mail submission, not relaying. RFC5321 specifically says that it does not cover submission, and defers to RFC4409 (3.6.1) which specifically allows From header rewriting (8.8).
If you’re going to complain about the standards-compliance of other people's services it’s important to understand the standard first.
If you’re going to complain about the standards-compliance of other people's services it’s important to understand the standard first.
I discuss exactly this in the article. I even referred to sec. 8.8 (of the current 6409; 4409 is superseded (but 8.8 is the same)).
Yeah but you don’t seem to really grok it.
“ The MSA MAY rewrite local parts and/or domains in the SMTP envelope and, optionally, in address fields of the header, according to local policy.“
Generally MSAs are permitted to do almost anything. Your reading of the RFC which requires MSAs to merely reject instead of modifying messages is not backed by any of the language in the RFC. Essentially, you seem to misunderstand “may” “must” and “should” as used in these documents.
“ The MSA MAY rewrite local parts and/or domains in the SMTP envelope and, optionally, in address fields of the header, according to local policy.“
Generally MSAs are permitted to do almost anything. Your reading of the RFC which requires MSAs to merely reject instead of modifying messages is not backed by any of the language in the RFC. Essentially, you seem to misunderstand “may” “must” and “should” as used in these documents.
You're extracting one sentence out of several paragraphs.
The following paragraph restricts the permissible rewriting, and Google is going beyond what is permitted. If your interpretation differs from mine, that's fine. I don't think you need to support your view by selective quotations and accusing me of not being aware of what I explicitly dealt with.
The meaning of the whole section is clear, and I stand behind what I wrote.
The meaning of the whole section is clear, and I stand behind what I wrote.
Again, the only thing that seems clear here is that you don't know how to read an RFC. The only "must not" in 8.8 is the prohibition of changing the destination local parts. "must not" appears nowhere else in section 8, and 8 is not strongly prescriptive as it "describes a number of such modifications that are often considered useful". The described modifications are not thought to be exhaustive. Moreover, 5321 describes MSA as "arrangements are private and fall outside the scope of this specification, they are not described here." 5321 does not control MSA and 4409/6409 do not defer to 5321 on any point. 4409/6490 do _refer_ to 5321, of course.
There are only 5 MUST NOTs in 6409 and there are zero SHOULD NOTs. The standard permits the MSA to do pretty much anything it wants to do. I think you should begin by reading 2119 "Key words for use in RFCs to Indicate Requirement Levels".
There are only 5 MUST NOTs in 6409 and there are zero SHOULD NOTs. The standard permits the MSA to do pretty much anything it wants to do. I think you should begin by reading 2119 "Key words for use in RFCs to Indicate Requirement Levels".
'The only "must not" in 8.8 is the prohibition of changing the destination local parts.'
That's not even true. If anyone cares about this they should just read the texts. I'm certainly not going to argue this any further. We're looking at the same page and seeing different words.
It actually says this:
"However, only addresses, local-parts, or domains that match specific local MSA configuration settings should be altered"
In other words, the opposite of what you claim. You can keep banging on this if you find it useful, but I'm not going to read any more of your comments.
That's not even true. If anyone cares about this they should just read the texts. I'm certainly not going to argue this any further. We're looking at the same page and seeing different words.
It actually says this:
"However, only addresses, local-parts, or domains that match specific local MSA configuration settings should be altered"
In other words, the opposite of what you claim. You can keep banging on this if you find it useful, but I'm not going to read any more of your comments.
According to RFC2119, which defines these terms, "should" and "should not" are recommendations with the caveat that individuals not following said recommendations need to understand the full implications what they are doing.
On the other hand, the words "must" and "must not" specify absolute requirements and prohibitions of the standard, respectively.
On the other hand, the words "must" and "must not" specify absolute requirements and prohibitions of the standard, respectively.
I think the confusion comes from parsing the poorly worded "only X should be altered" which doesn't really make it clear how the "should" applies.
The clear wording would be "... should only alter X" which makes it clear that altering things other than X needs be be done with full understanding of the implications.
The clear wording would be "... should only alter X" which makes it clear that altering things other than X needs be be done with full understanding of the implications.
I know nothing about these RFCs, but this seems trivial to clear up with a reference to the text. Here's RFC6409 section 8.8: https://tools.ietf.org/html/rfc6409#section-8.8. There is exactly one "MUST NOT" and it looks like it only prohibits what your parent said it does.
There seems to be an idea floating around that anything at all is permitted under the RFC for MSAs unless it is specifically prohibited using the phrase "must not", regardless of context and other guidance in this RFP and the other RFPs that refer to it. Perhaps this is true. I have a sincere question for anyone holding this opinion: would Google be in compliance if their software altered the message body by adding the sentence "Google is the best search engine ever." here and there, at random? If not, why not, exactly?
yes they would be compliant.
They wouldn't be a good server to use, but they'd be one compliant to the specification.
--> What this tells you is that abiding by the spec is not everything, it just sets minimum interoperability guarantees.
They wouldn't be a good server to use, but they'd be one compliant to the specification.
--> What this tells you is that abiding by the spec is not everything, it just sets minimum interoperability guarantees.
Thanks for this reply. It's food for thought.
"should be" is not the same as "must be"
[deleted]
[deleted]
After considering some of the comments below, I retract my statement that Google is violating the relevant RFPs. They are operating their servers in violation of reasonable user expectations, and potentially causing harm in doing so, but they are still, strictly speaking, in compliance. I plan to update my article to reflect this.
This does suck, but you can actually use From: aliases if you set them up first in the Gmail interface. You also have to enable two factor auth. Hard to be too mad about spam fighting measures.
Sounds like I need to update my article. Thanks for this information.
But note that they are still violating the RFPs, if they rewrite headers for people who fail to jump through these hoops. They can implement their spam protections (if that's the purpose) while remaining compliant by rejecting the emails in question.
But note that they are still violating the RFPs, if they rewrite headers for people who fail to jump through these hoops. They can implement their spam protections (if that's the purpose) while remaining compliant by rejecting the emails in question.
Maybe I've misunderstood something, but if gmail didn't do this, what's to stop me from sending emails that look like they come from your address?
Nothing. The standard allows you to put anything you want in the From: header. Sometimes I send emails from Santa Claus or God. The envelope information is still there.
Actually, not "nothing." If the MSA doesn't like this, it is free to reject the email. But not to change its content.
Yes, I’ve actually lost data because of this... :/
Google, Microsoft, both wants to put us in a cage and milk us for every cent. Perhaps even Apple, but their cage seems to have a bit more gilded...
I doubt their poor compliance to standards is done maliciously. I'm guessing it's more about needing some service, finding an open standard or code that does what they need, making changes that are convenient for them, then moving on to the next project.
If not malice then willful harm through inaction. The service should be shut down or fixed.
From the Article:
> "Google’s CardDAV server silently wipes out all kinds of data it doesn’t care about."
How is this "bad" functionality treated differently than issues found and publicized after a 90-day-count-down in the security disclosure universe?
They likely have known about their noncompliance, and they must know the impact; they are a technically savvy group. Plus I presume they have the financial resources to fix this with a year-to-year rising stock price currently at 1,095.50 USD.
"... moving on to the next project" is irresponsible after unleashing a service which when used correctly WILL corrupt users' data.
From the Article:
> "Google’s CardDAV server silently wipes out all kinds of data it doesn’t care about."
How is this "bad" functionality treated differently than issues found and publicized after a 90-day-count-down in the security disclosure universe?
They likely have known about their noncompliance, and they must know the impact; they are a technically savvy group. Plus I presume they have the financial resources to fix this with a year-to-year rising stock price currently at 1,095.50 USD.
"... moving on to the next project" is irresponsible after unleashing a service which when used correctly WILL corrupt users' data.
> How is this "bad" functionality treated differently than issues found and publicized after a 90-day-count-down in the security disclosure universe?
For one, these aren't security issues.
Also, they do document the parts of the standards that they don't support or adhere to:
https://developers.google.com/google-apps/carddav/
Their documentation could probably be more complete, but it does say, for example, that they don't support user defined fields on WebDAV requests.
I suppose they could have opted to not mention CardDAV or the other standards and just published an API called GCard or something like that.
They do mention that interoperating with Apple's operating systems is working, so that may be what determined the subset they care about.
> service which when used correctly WILL corrupt users' data
So that comes back to who gets to define what correct use is? If you follow the Google documentation and are losing data, then I agree with you. They should fix the problem or update the documentation.
For one, these aren't security issues.
Also, they do document the parts of the standards that they don't support or adhere to:
https://developers.google.com/google-apps/carddav/
Their documentation could probably be more complete, but it does say, for example, that they don't support user defined fields on WebDAV requests.
I suppose they could have opted to not mention CardDAV or the other standards and just published an API called GCard or something like that.
They do mention that interoperating with Apple's operating systems is working, so that may be what determined the subset they care about.
> service which when used correctly WILL corrupt users' data
So that comes back to who gets to define what correct use is? If you follow the Google documentation and are losing data, then I agree with you. They should fix the problem or update the documentation.
Aaaaaah they are in the clear then - I am alarmist and retract my assertion:
> "We have not implemented the full specification, however many clients such as Apple iOS™ Contacts and Apple Mac™ OS should interoperate correctly."
They've explained they're non-conformant so all is well IMO.
Thanks for the corrections and critique criddell.
> "We have not implemented the full specification, however many clients such as Apple iOS™ Contacts and Apple Mac™ OS should interoperate correctly."
They've explained they're non-conformant so all is well IMO.
Thanks for the corrections and critique criddell.
Ah just like when I worked in x.400 Sprint was notorious for just ignoring stuff in the standard.
Also Goole doesn't even parse the date time in xml sitemaps 100% correctly and that's only a 4 page RFC!
Also Goole doesn't even parse the date time in xml sitemaps 100% correctly and that's only a 4 page RFC!
We need to change the culture of software development such that this behaviour is considered unprofessional. Perhaps a professional software engineering association might help?
How is that kind of carelessness not a form of malice?
You think on some PowerPoint a manager had the bullet:
* break compatibility with Outlook
* break compatibility with Outlook
No, but I think a huge company saying, “Oh who gives a tupenny fuck about compatibility” is sort of malicious.
I don't think they have a "compatibility" slide anywhere at all at Google.
Quote from the DAVdroid project:
Address books, calendars and task list contain private and sensitive data by their nature. We believe that individuals and companies should be able to be in full control of their own data. This means the freedom to choose where and how the data are stored, and freedom in choice of software. CalDAV/CardDAV are open protocols and thus can be used with various server and client software.
Setting up your own CalDAV + CardDAV server such as Radicale is easy. Chances are if you are reading HN, you can do it yourself. Do it.
Address books, calendars and task list contain private and sensitive data by their nature. We believe that individuals and companies should be able to be in full control of their own data. This means the freedom to choose where and how the data are stored, and freedom in choice of software. CalDAV/CardDAV are open protocols and thus can be used with various server and client software.
Setting up your own CalDAV + CardDAV server such as Radicale is easy. Chances are if you are reading HN, you can do it yourself. Do it.
I recently upgraded from Radicale 0.10 to 2.x. One of their migration scripts didn't work correctly for the calendar, but since I was in control of the data, I could just look at the ics file, see what they were doing, and write a small python script to do it myself.
If anyone wants to do it in Docker, I have a really simple Dockerfile here:
https://github.com/sumdog/bee2/tree/master/dockerfiles/Radic...
If anyone wants to do it in Docker, I have a really simple Dockerfile here:
https://github.com/sumdog/bee2/tree/master/dockerfiles/Radic...
Recently switched to Radicale, and it is pretty much setup-once-and-forget. Highly recommended. I'm using the tomsquest/docker-radicale image, if anyone else wants to set it up.
Like many google skeptics I moved to Fastmail. Fastmail have accounts and calendar that you can access over CalDav and CardDav with DAVdroid, which is exactly what I do.
Can you recommend any good webGUIs when running my own CalDAV server? I'm struggling to find one and I really need one.
Nextcloud works for me
I've found infcloud [0] to be decent.
It's a bit messy to set up but the config file is extensively documented and you generally only need to touch it in the initial setup.
[0]: https://www.inf-it.com/open-source/clients/infcloud/
It's a bit messy to set up but the config file is extensively documented and you generally only need to touch it in the initial setup.
[0]: https://www.inf-it.com/open-source/clients/infcloud/
For CalDAV: http://agendav.org/ is much prettier than infCloud, but actually has it's own php-backend and mysql database. A bit heavier and is not strictly only-CalDAV. Could be a viable solution though.
For CardDAV: https://github.com/owncloud/contacts could be used as a frontend, since it advertises it's talking to CardDAV-backend, but it's not actually deployable as a standalone interface. It's better than infCloud both architecture-wise and UI-wise, and could be the best opensource solution available, if you're willing to adapt this frontend to stand-alone usage (or integrate into whatever you need it for)
For CardDAV: https://github.com/owncloud/contacts could be used as a frontend, since it advertises it's talking to CardDAV-backend, but it's not actually deployable as a standalone interface. It's better than infCloud both architecture-wise and UI-wise, and could be the best opensource solution available, if you're willing to adapt this frontend to stand-alone usage (or integrate into whatever you need it for)
Seconded. I did this myself a few months ago and it is not hard at all.
I used to feel sufficiently uncomfortable about handing reams of personal data to Google that it stopped me using calendar services on my phone - so it's actually made an appreciable difference to my life.
I used to feel sufficiently uncomfortable about handing reams of personal data to Google that it stopped me using calendar services on my phone - so it's actually made an appreciable difference to my life.
In the CalDAV space a lot of people do whatever they want, so there is a standard but most people are not compliant with it.
I would imagine the same is true for CardDAV, not only at Google.
I would imagine the same is true for CardDAV, not only at Google.
Tangent: What are people's experience with Google's CalDAV support? Is it just as bad?
It isn't. CalDAV is a separate team. The support is quite good. The deviations from the standard are documented.
> this isn’t just a 400 Bad Request [...] Any (valid) vCard that the CardDAV server does not understand, will result in our HTTP requests timing out
That's just terrible. Why would such a thing be implemented ?
The only valid explaination I can think of, is to deter abuse. But even for that there are better solutions than this.
That's just terrible. Why would such a thing be implemented ?
The only valid explaination I can think of, is to deter abuse. But even for that there are better solutions than this.
I get the sense that Google cared about open protocols, and then Android changed that, to simply being another mechanism to lock users into their walled garden.
I don't think you can pin the change on Android. It's a slow progression at Google from the days of the unofficial "don't be evil." Even their official standards have been awful. IMAP was, and still is, hopelessly broken on gmail. Where GTalk once had XMPP with federation, they silently removed federation and XMPP entirely. I didn't realize how bad their Card/Caldav implementations where because I switched to Radicalie back in 2013.
There just isn't any money is letting users access and control their own data in a standard way.
There just isn't any money is letting users access and control their own data in a standard way.
Google doesn't like existing standards, and have generally been bad stewards of their power in that place, they like to make new ones though that they expect everyone to jump on. I am wondering if the engineers and product developers are still in charge at Google?
This (good, but sad article) is actually about "How" it's different, not "Why".
Why don't they care at all about standards? I don't know; they talk a good game. At least some companies who violate standards are open about it -- Google trumpets it but often simply pays lip service.
Why don't they care at all about standards? I don't know; they talk a good game. At least some companies who violate standards are open about it -- Google trumpets it but often simply pays lip service.
I think to get past such issues with providers like this, you first need to understand what it is and what it is not.
Forget the brochure, the ads and the nice tutorial, those are NOT CardDAV/SMTP/IMAP/... servers. What they are instead are ways to access their own proprietary special thing using standard protocols that your software and tools already supports, if you can't use the dedicated api.
It applies to pretty much every system for which Google has made their own thing, with their own api (contact api, mail api, ...), and then offer an alternate way to access their system by standard protocols.
And what this means is that the ruling system is the underlying one, the one which can be accessed in several ways, including that standard protocol. If their contact system discards the field "foobar" because it doesn't want it, then it doesn't matter if CardDAV says it should be kept.
Sure Google should totally make this clearer, and sure as an user I'm not a big fan, and yes I totally understand and agree with the "but as a result they break the standard, lose data and cause unexpected behavior and that's terrible", but from a purely engineering perspective it also makes total sense as to why those things happen this way, and is actually rather predictable.
You're feeding data to a special system that just so happens to have a CardDAV/SMTP/IMAP/... endpoint plugged on top of it. Of course that endpoint doesn't rule how the special system will behave.
And frankly any one who has had to build/support that kind of "api on top of an api" for some time has faced that sort of situation before.
In fact you don't need to go that far; I mean how many of your web applications take PUT or DELETE requests ? The HTTP protocol certainly says it should work and the http server you're using supports it. And let's not talk about all those sites that give a full page answer to a HEAD request.
PS: again, I am most definitely NOT saying I like that it works that way, I'm just saying ... "well no big surprise there".
Forget the brochure, the ads and the nice tutorial, those are NOT CardDAV/SMTP/IMAP/... servers. What they are instead are ways to access their own proprietary special thing using standard protocols that your software and tools already supports, if you can't use the dedicated api.
It applies to pretty much every system for which Google has made their own thing, with their own api (contact api, mail api, ...), and then offer an alternate way to access their system by standard protocols.
And what this means is that the ruling system is the underlying one, the one which can be accessed in several ways, including that standard protocol. If their contact system discards the field "foobar" because it doesn't want it, then it doesn't matter if CardDAV says it should be kept.
Sure Google should totally make this clearer, and sure as an user I'm not a big fan, and yes I totally understand and agree with the "but as a result they break the standard, lose data and cause unexpected behavior and that's terrible", but from a purely engineering perspective it also makes total sense as to why those things happen this way, and is actually rather predictable.
You're feeding data to a special system that just so happens to have a CardDAV/SMTP/IMAP/... endpoint plugged on top of it. Of course that endpoint doesn't rule how the special system will behave.
And frankly any one who has had to build/support that kind of "api on top of an api" for some time has faced that sort of situation before.
In fact you don't need to go that far; I mean how many of your web applications take PUT or DELETE requests ? The HTTP protocol certainly says it should work and the http server you're using supports it. And let's not talk about all those sites that give a full page answer to a HEAD request.
PS: again, I am most definitely NOT saying I like that it works that way, I'm just saying ... "well no big surprise there".
The distinction between being a "real CardDAV server" and being a server that exposes CardDAV is undefined and irrelevant, possibly non-existent. There is a spec, and if you implement it, you have a server that implements that spec. That's it.
You can of course deviate from certain parts of the spec... dropping unknown fields as you mentioned might be a reasonable tradeoff. But then that deviation has to be documented. It isn't.
You're completely leaving out the other issues that the author describes: Timing out TCP sessions, discarding UIDs. Both things are not reasonable design choices in any world. And then the response to bug reports.
You can of course deviate from certain parts of the spec... dropping unknown fields as you mentioned might be a reasonable tradeoff. But then that deviation has to be documented. It isn't.
You're completely leaving out the other issues that the author describes: Timing out TCP sessions, discarding UIDs. Both things are not reasonable design choices in any world. And then the response to bug reports.
I am not sure what you mean by 'the deviation has to be documented.'
Has to be documented in order to do what? To claim to be standards compliant? Do they ever make that claim?
Has to be documented in order to do what? To claim to be standards compliant? Do they ever make that claim?
Ahh, the good old days of “embrace and extend” are back.
Delightful.
Delightful.
Relevant: The author of the post worked on Sabre/dav, an open source CardDAV/CalDAV server (linked at bottom of the post). Unfortunately, it is no longer under development: http://sabre.io/blog/2017/development-on-hold/
A bunch of people have since picked up development and are working on new releases.
Slightly related: try to sync an external calendar feed in iCalendar format (also used in CalDAV) with Google Calendar.
Google Calendar will use random intervals to synchronize. Often taking more than 24 hours... which is, in most use cases, quite useless.
Sync in the same way with Facebook's event feed (same format) and changes appear within minutes.
This has been the case for years. Google Groups are full of questions about this.
I don't want to demand anything from a free product (Google Calendar is very useful in many ways). But somehow some feedback about 'shortcomings' like these would be hugely appreciated.
Google Calendar will use random intervals to synchronize. Often taking more than 24 hours... which is, in most use cases, quite useless.
Sync in the same way with Facebook's event feed (same format) and changes appear within minutes.
This has been the case for years. Google Groups are full of questions about this.
I don't want to demand anything from a free product (Google Calendar is very useful in many ways). But somehow some feedback about 'shortcomings' like these would be hugely appreciated.
https://developers.google.com/people/v1/libraries
Is the current recommended solution for this sort of thing.
Libraries look to be available for
GO JAVA JAVASCRIPT .NET NODE.JS OBJ-C PHP PYTHON RUBY
and you can roll your own using HTTP.
CardDAV is not well maintained, and even the contacts API using GData is superseded with the above.
With over 1 billion active users - my guess is engineering efforts are focused on serving that customer base, especially the growth in paying customers through google apps. Not sure where strict compliance with CardDAV would fall in the analysis.
A note that actually using some of these API's through their client libraries isn't totally the nightmare described - though my experience is very limited.