XMPP Myths(wiki.xmpp.org)
wiki.xmpp.org
XMPP Myths
http://wiki.xmpp.org/web/index.php?title=Myths
139 comments
While I agree that XMPP is not the future for app-to-app communication, I do so for a slightly different reasons.
> XML is unnecessarily complex for this kind of usecase
XMPP is too complex, but the XML stuff is really the smallest issue, compared with other complexities in the protocol. For example, every client that successfully connected to an XMPP server must first send a "<presence>" message. Otherwise, it won't receive any messages despite their successful connection. And there are more such surprises in XMPP. Good luck in understanding which "devices" (#xxxx) of the same Jabber ID will receive a message targeted at that ID.
It's really annoying and discouraging to debug that kind of issues - especially since these aren't bugs. These work as specified, but the specifications were obviously designed with direct human chatting or direct human collaboration in mind, programs being an afterthought.
> The correct technology is JSON-over-WebSockets
Not really. Whenever I saw someone trying this "in the wild", I see them quickly switching to HTTP long-polling. (Not to be confused with polling! We are still talking about instant push notifications with very low network overhead.)
For example, there are some libraries that used to provide wrappers around WebSockets, which switched to HTTP long-polling as fallback for old browsers. Nowadays, it seems they use long-polling by default, even though browser support becomes better and better.
Don't underestimate the role of firewalls and HTTP proxies! Many firewalls are aggressive configured, and many companies have forced HTTP proxies in place.
> XML is unnecessarily complex for this kind of usecase
XMPP is too complex, but the XML stuff is really the smallest issue, compared with other complexities in the protocol. For example, every client that successfully connected to an XMPP server must first send a "<presence>" message. Otherwise, it won't receive any messages despite their successful connection. And there are more such surprises in XMPP. Good luck in understanding which "devices" (#xxxx) of the same Jabber ID will receive a message targeted at that ID.
It's really annoying and discouraging to debug that kind of issues - especially since these aren't bugs. These work as specified, but the specifications were obviously designed with direct human chatting or direct human collaboration in mind, programs being an afterthought.
> The correct technology is JSON-over-WebSockets
Not really. Whenever I saw someone trying this "in the wild", I see them quickly switching to HTTP long-polling. (Not to be confused with polling! We are still talking about instant push notifications with very low network overhead.)
For example, there are some libraries that used to provide wrappers around WebSockets, which switched to HTTP long-polling as fallback for old browsers. Nowadays, it seems they use long-polling by default, even though browser support becomes better and better.
Don't underestimate the role of firewalls and HTTP proxies! Many firewalls are aggressive configured, and many companies have forced HTTP proxies in place.
That is a browser issue, and does not exist in native clients. If we're talking about XML-stream-over-sockets, then comparing it to browsers makes no sense (because they can't do that!), and thus long-polling and JSON-over-HTTP become irrelevant.
EDIT: Hmm, did you edit your post? I don't recall the first part being there... anyhow:
> XMPP is too complex, but the XML stuff is really the smallest issue, compared with other complexities in the protocol.
That is one factor, but the protocol complexities often seem to derive from the question 'how do we represent this in XML?', and that's an issue.
Another problem is the jargon - the MUC specification is utterly incomprehensible, because the terms it uses for things are so wildly different from how they are used everywhere else, that building a mental model is really hard. I've seen this issue in other XEPs as well.
EDIT: Hmm, did you edit your post? I don't recall the first part being there... anyhow:
> XMPP is too complex, but the XML stuff is really the smallest issue, compared with other complexities in the protocol.
That is one factor, but the protocol complexities often seem to derive from the question 'how do we represent this in XML?', and that's an issue.
Another problem is the jargon - the MUC specification is utterly incomprehensible, because the terms it uses for things are so wildly different from how they are used everywhere else, that building a mental model is really hard. I've seen this issue in other XEPs as well.
> For example, every client that successfully connected to an XMPP server must first send a "<presence>" message. Otherwise, it won't receive any messages despite their successful connection.
That statement is wrong. Sending an initial presence merely indicates your availability to entities subscribed to your presence. Not sending the initial presence does not prevent you from receiving message stanzas.
That statement is wrong. Sending an initial presence merely indicates your availability to entities subscribed to your presence. Not sending the initial presence does not prevent you from receiving message stanzas.
Addendum: As Zash pointed out[1], this is documented behavior for offline messages, i.e. messages send while the user was offline. Those are only send if the user announced availability via a presence or if XEP-13: Flexible Offline Message Retrieval is used. This is to prevent storms of offline messages when the client (re-)connects.
But it's certainly not true that you "won't receive any message despite ... successful connection".
1: https://news.ycombinator.com/item?id=10043962
But it's certainly not true that you "won't receive any message despite ... successful connection".
1: https://news.ycombinator.com/item?id=10043962
Indeed, my description was rough and from memories a few year ago. But I believe it still demonstrates the point I was trying to illustrate:
There is unneccessary complex behaviour in XMPP that you must write extra code in your client for. Otherwise it will work mostly, but not always and not reliably.
There is unneccessary complex behaviour in XMPP that you must write extra code in your client for. Otherwise it will work mostly, but not always and not reliably.
Pretty sure you want your XMPP client to announce that it is online, so you'd have to implement that anyways.
This is all well described in the RFCs and http://www.xmpp.org/extensions/xep-0160.html
IIRC, the ejabberd server will wait until you are "present". Only then it will send you all messages so far received for you.
Really? The old Google Talk service used to do this, but I didn't think ejabberd did. There's a bunch of non-IM use-cases which need presence-less messaging.
That ejabberd does something doesn't mean it's the proper behaviour. Maybe it's a bug?
Seems like Prosody has the same behavior. Ran into just this morning.
+1 that JSON-over-WS is not a panacea. Long-polling is inelegant but is pretty efficient with HTTP/2.0, and trivially degrades on older HTTP stacks (e.g. in embedded devices) which don't know anything about WS.
Dave Cridland's wiki page makes some good points, but predictably it paints a very black and white view of the world. Some contentious points are:
1. Myth One: XMPP is XML, so it's too slow.
...there are obviously faster encodings than XML out there (e.g. CBOR), so this feels like a slightly specious point to argue. Sure the difference isn't that important on the clientside (perhaps more of an issue for squeezing every last drop out of a server), but why not acknowledge it?
2. Myth Two: The baseline is minimal, therefore XMPP is useless.
I'm not sure anyone claims XMPP is useless just because its baseline is minimal. The issue is instead the fragmentation caused by lack of interoperability profiles for all the different extensions. Statistically, if I pick an arbitrary XMPP client and an arbitrary XMPP server, the overlap of supported XEPs is unlikely to provide a good modern comms experience. If it was a "bug" for your client or server not to implement a given critical XEP, then fragmentation would be less of an issue.
3. Myth Three: It's too bandwidth-inefficient for mobile.
Just because you can successfully talk XMPP over a 9600bps link with 30s of latency doesn't mean that the default transport is bandwidth efficient(!) That said, the XEPs for constrained networks are a better justification.
4. Myth Four: XMPP is unreliable without a bunch of extensions.
As Dave repeatedly says, XMPP is a NAK protocol - by default messages are not acknowledged, so whenever a TCP stream drops you have no way of knowing whether the last messages in the stream got through or not. This is to me the definition of an unreliable delivery mechanism, and you have to mitigate it with extensions (XEP-198). I'm still not sure how this one's a myth...
5. Myth Five : XMPP is unsuited to the web.
The hypothesis of "XMPP is unsuited to the web because angle brackets and stuff" isn't exactly well articulated here. I think the actual complaint here is "why bother putting an XMPP stack in your browser when you already have an HTTP stack there"? What value do the remnants of XMPP's transport semantics have when you already have HTTP as a transport? I agree that Stanza.io & XMPP-FTW make this less clear cut, however.
For what it's worth, the FAQ entry that triggered this wiki page can be found at http://matrix.org/blog/faq/ (What’s the difference between Matrix and XMPP?).
Dave Cridland's wiki page makes some good points, but predictably it paints a very black and white view of the world. Some contentious points are:
1. Myth One: XMPP is XML, so it's too slow.
...there are obviously faster encodings than XML out there (e.g. CBOR), so this feels like a slightly specious point to argue. Sure the difference isn't that important on the clientside (perhaps more of an issue for squeezing every last drop out of a server), but why not acknowledge it?
2. Myth Two: The baseline is minimal, therefore XMPP is useless.
I'm not sure anyone claims XMPP is useless just because its baseline is minimal. The issue is instead the fragmentation caused by lack of interoperability profiles for all the different extensions. Statistically, if I pick an arbitrary XMPP client and an arbitrary XMPP server, the overlap of supported XEPs is unlikely to provide a good modern comms experience. If it was a "bug" for your client or server not to implement a given critical XEP, then fragmentation would be less of an issue.
3. Myth Three: It's too bandwidth-inefficient for mobile.
Just because you can successfully talk XMPP over a 9600bps link with 30s of latency doesn't mean that the default transport is bandwidth efficient(!) That said, the XEPs for constrained networks are a better justification.
4. Myth Four: XMPP is unreliable without a bunch of extensions.
As Dave repeatedly says, XMPP is a NAK protocol - by default messages are not acknowledged, so whenever a TCP stream drops you have no way of knowing whether the last messages in the stream got through or not. This is to me the definition of an unreliable delivery mechanism, and you have to mitigate it with extensions (XEP-198). I'm still not sure how this one's a myth...
5. Myth Five : XMPP is unsuited to the web.
The hypothesis of "XMPP is unsuited to the web because angle brackets and stuff" isn't exactly well articulated here. I think the actual complaint here is "why bother putting an XMPP stack in your browser when you already have an HTTP stack there"? What value do the remnants of XMPP's transport semantics have when you already have HTTP as a transport? I agree that Stanza.io & XMPP-FTW make this less clear cut, however.
For what it's worth, the FAQ entry that triggered this wiki page can be found at http://matrix.org/blog/faq/ (What’s the difference between Matrix and XMPP?).
The specific claims I was responding to when writing this (and it's still a draft) were an HTTP-based JSON protocol whose FAQ makes all these claims and more. And yeah, I think it's dumb to do things that way (both JSON/HTTP and making ill-informed and misleading comments in your FAQ).
As to your comments:
(1) I'm just not seeing this. Even without mobile-style push, apps like Conversations appear way down the list in Android. Note that because XMPP maintains a continuous TCP session (or WebSocket, or whatever), then you get "push" in a general sense; that is, there is no polling.
Using an XMPP client on your mobile will, of course, have an impact on battery life, but I don't think a good client will drain your battery. For me, Conversations uses less power than Solitaire according to Android.
(2) I'll agree that XML does make the development of XMPP libraries more complex, but we gain because it grants very simple permissionless extension capability, and that's hugely valuable. If developers working with (rather than on) XMPP are having to hit XML directly, that is absolutely a problem.
Finally, resources are the way they are for all sorts of reasons, but Message Carbons really does work for having conversation sync across multiple devices. I often walk away from my desktop and continue conversations on my mobile and/or tablet, and wander back to the desktop when I feel the need to continue with a keyboard.
Hope that helps.
As to your comments:
(1) I'm just not seeing this. Even without mobile-style push, apps like Conversations appear way down the list in Android. Note that because XMPP maintains a continuous TCP session (or WebSocket, or whatever), then you get "push" in a general sense; that is, there is no polling.
Using an XMPP client on your mobile will, of course, have an impact on battery life, but I don't think a good client will drain your battery. For me, Conversations uses less power than Solitaire according to Android.
(2) I'll agree that XML does make the development of XMPP libraries more complex, but we gain because it grants very simple permissionless extension capability, and that's hugely valuable. If developers working with (rather than on) XMPP are having to hit XML directly, that is absolutely a problem.
Finally, resources are the way they are for all sorts of reasons, but Message Carbons really does work for having conversation sync across multiple devices. I often walk away from my desktop and continue conversations on my mobile and/or tablet, and wander back to the desktop when I feel the need to continue with a keyboard.
Hope that helps.
> I'm just not seeing this. Even without mobile-style push, apps like Conversations appear way down the list in Android. Note that because XMPP maintains a continuous TCP session (or WebSocket, or whatever), then you get "push" in a general sense; that is, there is no polling.
> Using an XMPP client on your mobile will, of course, have an impact on battery life, but I don't think a good client will drain your battery. For me, Conversations uses less power than Solitaire according to Android.
I don't know how accurate Android's power estimates are, but I consistently have people telling me that they are switching back to proprietary messengers because every XMPP client they've tried drained their battery. Whatever causes it, there's a real problem here.
> (2) I'll agree that XML does make the development of XMPP libraries more complex, but we gain because it grants very simple permissionless extension capability, and that's hugely valuable. If developers working with (rather than on) XMPP are having to hit XML directly, that is absolutely a problem.
The problem is that theoretical future benefits are worthless if you never get a base implementation going. XMPP is just too hard for many developers - it's not a thing that you're going to understand well unless you work on it full-time, and that's not acceptable for a protocol.
Hitting XML directly is almost always a requirement. Libraries are woefully incomplete due to the large number of XEPs, and to implement anything yourself, you must understand the XML structure.
If the intention of XMPP or XMPP frameworks was to abstract away the XML, then they have failed.
> Finally, resources are the way they are for all sorts of reasons, but Message Carbons really does work for having conversation sync across multiple devices. I often walk away from my desktop and continue conversations on my mobile and/or tablet, and wander back to the desktop when I feel the need to continue with a keyboard.
I experience constant issues with OTR, most servers don't seem to even support message carbons (nor do major clients like Pidgin have any useful indications of it), and frankly, I have never seen a single 'regular' user get synchronization set up correctly.
This is really a pervasive problem in the XMPP ecosystem: everything is too hard. Little to no thought seems to have gone into the UX, and unless you are an 'experienced XMPP user', you really have no clue what is going on. The experienced users say "meh, works for me" and don't set out to fix it.
This is true for the frameworks, the clients, and most of the servers. It's just all-around user-hostile, and that needs to change.
> Using an XMPP client on your mobile will, of course, have an impact on battery life, but I don't think a good client will drain your battery. For me, Conversations uses less power than Solitaire according to Android.
I don't know how accurate Android's power estimates are, but I consistently have people telling me that they are switching back to proprietary messengers because every XMPP client they've tried drained their battery. Whatever causes it, there's a real problem here.
> (2) I'll agree that XML does make the development of XMPP libraries more complex, but we gain because it grants very simple permissionless extension capability, and that's hugely valuable. If developers working with (rather than on) XMPP are having to hit XML directly, that is absolutely a problem.
The problem is that theoretical future benefits are worthless if you never get a base implementation going. XMPP is just too hard for many developers - it's not a thing that you're going to understand well unless you work on it full-time, and that's not acceptable for a protocol.
Hitting XML directly is almost always a requirement. Libraries are woefully incomplete due to the large number of XEPs, and to implement anything yourself, you must understand the XML structure.
If the intention of XMPP or XMPP frameworks was to abstract away the XML, then they have failed.
> Finally, resources are the way they are for all sorts of reasons, but Message Carbons really does work for having conversation sync across multiple devices. I often walk away from my desktop and continue conversations on my mobile and/or tablet, and wander back to the desktop when I feel the need to continue with a keyboard.
I experience constant issues with OTR, most servers don't seem to even support message carbons (nor do major clients like Pidgin have any useful indications of it), and frankly, I have never seen a single 'regular' user get synchronization set up correctly.
This is really a pervasive problem in the XMPP ecosystem: everything is too hard. Little to no thought seems to have gone into the UX, and unless you are an 'experienced XMPP user', you really have no clue what is going on. The experienced users say "meh, works for me" and don't set out to fix it.
This is true for the frameworks, the clients, and most of the servers. It's just all-around user-hostile, and that needs to change.
> I don't know how accurate Android's power estimates are, but I consistently have people telling me that they are switching back to proprietary messengers because every XMPP client they've tried drained their battery. Whatever causes it, there's a real problem here.
You have to differentiate between the specification and the implemetation. Nothing in the XMPP specification prevents you from implementing a battery friendly XMPP client. But some implementations suck/have room for improvement.
You have to differentiate between the specification and the implemetation. Nothing in the XMPP specification prevents you from implementing a battery friendly XMPP client. But some implementations suck/have room for improvement.
Right. But when I have suggested a number of different XMPP clients to people, and every single one of them has caused this issue, then there's a problem.
Where that problem is, I don't know. I'm just going by what I hear from people more experienced with the effect of XMPP on battery life.
Where that problem is, I don't know. I'm just going by what I hear from people more experienced with the effect of XMPP on battery life.
> I don't know how accurate Android's power estimates are, but I consistently have people telling me that they are switching back to proprietary messengers because every XMPP client they've tried drained their battery. Whatever causes it, there's a real problem here.
I honestly cannot help you. Could be that my usage patterns just happen to make the battery usage better (though I'm a heavy user of multiple accounts), or could be I'm a really serious Solitaire player.
I do know that over the past year or so the situation has really improved, though (mostly with the merging of aSmack and Smack, but Conversations too).
> The problem is that theoretical future benefits are worthless if you never get a base implementation going. XMPP is just too hard for many developers - it's not a thing that you're going to understand well unless you work on it full-time, and that's not acceptable for a protocol.
I mostly agree; I'd hope that we could get to a place where the requirement to "know" XMPP was roughly that of needing to "know" HTTP. So simple use cases could be done without truly understanding the guts.
> Hitting XML directly is almost always a requirement. Libraries are woefully incomplete due to the large number of XEPs, and to implement anything yourself, you must understand the XML structure.
> If the intention of XMPP or XMPP frameworks was to abstract away the XML, then they have failed.
I'll confess to not being your typical XMPP developer, but I was under the impression that successfully hitting the XML with libraries like Smack was actually quite hard.
> I experience constant issues with OTR, most servers don't seem to even support message carbons (nor do major clients like Pidgin have any useful indications of it), and frankly, I have never seen a single 'regular' user get synchronization set up correctly.
OK, so Pidgin is a multi-protocol client which means the UI is abstracted out... But... https://developer.pidgin.im/ticket/15508 I don't understand their logic there, I have to say.
For OTR, by the way, that just breaks with multi-device anyway. There's some experimental work with Axolotl underway which looks very promising.
> This is really a pervasive problem in the XMPP ecosystem: everything is too hard. Little to no thought seems to have gone into the UX, and unless you are an 'experienced XMPP user', you really have no clue what is going on. The experienced users say "meh, works for me" and don't set out to fix it.
I won't disagree there. The lack of a consensus on UX considerations (in fact, the lack of any attempt to get such a consensus) means that different clients present even simple things like jids differently. That's something the community (or the XSF) could seek to address.
> This is true for the frameworks, the clients, and most of the servers. It's just all-around user-hostile, and that needs to change.
And again, I agree. It's sad, especially, that you qualify servers yet not the clients - I'd rather see clients as really easy. I mostly think Conversations is there, mind, but I'm hardly the target audience here.
That said, I'm aware that one key problem faced by a federated network is that client on-boarding can't, so easily, create your account for you, and early attempts to streamline that process have left serious abuse problems in their wake.
I honestly cannot help you. Could be that my usage patterns just happen to make the battery usage better (though I'm a heavy user of multiple accounts), or could be I'm a really serious Solitaire player.
I do know that over the past year or so the situation has really improved, though (mostly with the merging of aSmack and Smack, but Conversations too).
> The problem is that theoretical future benefits are worthless if you never get a base implementation going. XMPP is just too hard for many developers - it's not a thing that you're going to understand well unless you work on it full-time, and that's not acceptable for a protocol.
I mostly agree; I'd hope that we could get to a place where the requirement to "know" XMPP was roughly that of needing to "know" HTTP. So simple use cases could be done without truly understanding the guts.
> Hitting XML directly is almost always a requirement. Libraries are woefully incomplete due to the large number of XEPs, and to implement anything yourself, you must understand the XML structure.
> If the intention of XMPP or XMPP frameworks was to abstract away the XML, then they have failed.
I'll confess to not being your typical XMPP developer, but I was under the impression that successfully hitting the XML with libraries like Smack was actually quite hard.
> I experience constant issues with OTR, most servers don't seem to even support message carbons (nor do major clients like Pidgin have any useful indications of it), and frankly, I have never seen a single 'regular' user get synchronization set up correctly.
OK, so Pidgin is a multi-protocol client which means the UI is abstracted out... But... https://developer.pidgin.im/ticket/15508 I don't understand their logic there, I have to say.
For OTR, by the way, that just breaks with multi-device anyway. There's some experimental work with Axolotl underway which looks very promising.
> This is really a pervasive problem in the XMPP ecosystem: everything is too hard. Little to no thought seems to have gone into the UX, and unless you are an 'experienced XMPP user', you really have no clue what is going on. The experienced users say "meh, works for me" and don't set out to fix it.
I won't disagree there. The lack of a consensus on UX considerations (in fact, the lack of any attempt to get such a consensus) means that different clients present even simple things like jids differently. That's something the community (or the XSF) could seek to address.
> This is true for the frameworks, the clients, and most of the servers. It's just all-around user-hostile, and that needs to change.
And again, I agree. It's sad, especially, that you qualify servers yet not the clients - I'd rather see clients as really easy. I mostly think Conversations is there, mind, but I'm hardly the target audience here.
That said, I'm aware that one key problem faced by a federated network is that client on-boarding can't, so easily, create your account for you, and early attempts to streamline that process have left serious abuse problems in their wake.
IMO, a lot of F/OSS seems to really ignore the UX/UI part. It doesn't seem to be just XMPP-community.
For another example, Mumble (VoIP) is effectively leader in bandwidth, quality, encryption, and extensibility - but have you ever tried to teach an end user gamer or otherwise about certificates, checking cert fingerprints, backing up their client certs so they don't lose admin privileges on servers, it's more or less a UX/onboarding clusterfuck.
For another example, Mumble (VoIP) is effectively leader in bandwidth, quality, encryption, and extensibility - but have you ever tried to teach an end user gamer or otherwise about certificates, checking cert fingerprints, backing up their client certs so they don't lose admin privileges on servers, it's more or less a UX/onboarding clusterfuck.
I don't think that aspect is intentionally ignored, it's just that most FOSS projects are seriously understaffed, and the UX/UI part is not a high priority item compared to getting it to work.
I think that's a mistake, though. Users often become contributors, and by offering a substandard UX, you remain forever understaffed - you don't gain the momentum, because you're not attracting users.
UX should be a high-priority item.
UX should be a high-priority item.
I guess thats when you build the client with good UX and charge for it. But it would have to be code under a license that is friendly to that kind of extension.
Good idea on charging for better interface. License might not be an issue, though. A cheat that many use is to run the GPL code in its own process, the proprietary one in its, and use a high-speed IPC method to marshal data between them. You can even do this as a pipeline with a proprietary UI on front, complex GPL in middle, and proprietary middleware as last step. Many models.
> I mostly agree; I'd hope that we could get to a place where the requirement to "know" XMPP was roughly that of needing to "know" HTTP. So simple use cases could be done without truly understanding the guts.
I'm afraid that will never happen, though, due to the way extensibility is built into XMPP. HTTP is only very superficially extensible - you can provide custom headers (which are really just formatted strings) and custom-encoded body data. Both can be rather challenging to understand, in some cases, but the core protocol is easy to abstract, and usually is all you need.
For XML, that is much harder - without the right set of XEPs, XMPP isn't very useful for a real-world application. If only one single XEP is not implemented on your platform, you have to write your own, and immediately have to deal with the entirety of the XMPP/XML (protocol) design. You can't superficially string something together, like with a HTTP header.
> I'll confess to not being your typical XMPP developer, but I was under the impression that successfully hitting the XML with libraries like Smack was actually quite hard.
Things seem slightly more well-supported in Java-land, but in most other ecosystems, there are no implementations of XMPP that are both feature-complete and easy to use. A complex protocol invites developers who build complex abstractions, not easy ones.
> OK, so Pidgin is a multi-protocol client which means the UI is abstracted out... But... https://developer.pidgin.im/ticket/15508 I don't understand their logic there, I have to say.
I have no idea, but some of my contacts use other clients, and they haven't gotten it to work either.
> For OTR, by the way, that just breaks with multi-device anyway. There's some experimental work with Axolotl underway which looks very promising.
Depends. XMPP seems somewhat lacking in its notification of which resources are 'active' (in common implementations), which is something that would help to at least synchronize online devices (by sending differently encrypted messages to different resources). Log synchronization is trickier, but really should be possible as well.
> I won't disagree there. The lack of a consensus on UX considerations (in fact, the lack of any attempt to get such a consensus) means that different clients present even simple things like jids differently. That's something the community (or the XSF) could seek to address.
I am not just refering to end-user UX, though - also to developer UX. Complex protocols with complex libraries aren't developer-friendly, and the same issue occurs as above: complex abstractions invite developers who write complex end user products.
It's very hard to get simple end user products built for complex protocols, because those require two fundamentally different types of developers.
> That said, I'm aware that one key problem faced by a federated network is that client on-boarding can't, so easily, create your account for you, and early attempts to streamline that process have left serious abuse problems in their wake.
I think the big mistake there was to leave the exact registration data undefined. Rather, there should have been a standardization of 'common registration fields' (including eg. CAPTCHA data), and later amendment XEPs if things like CAPTCHAs were to have to change.
As it is, in-band registration is disabled almost everywhere, because no client can deal well with bot prevention measures.
I'm afraid that will never happen, though, due to the way extensibility is built into XMPP. HTTP is only very superficially extensible - you can provide custom headers (which are really just formatted strings) and custom-encoded body data. Both can be rather challenging to understand, in some cases, but the core protocol is easy to abstract, and usually is all you need.
For XML, that is much harder - without the right set of XEPs, XMPP isn't very useful for a real-world application. If only one single XEP is not implemented on your platform, you have to write your own, and immediately have to deal with the entirety of the XMPP/XML (protocol) design. You can't superficially string something together, like with a HTTP header.
> I'll confess to not being your typical XMPP developer, but I was under the impression that successfully hitting the XML with libraries like Smack was actually quite hard.
Things seem slightly more well-supported in Java-land, but in most other ecosystems, there are no implementations of XMPP that are both feature-complete and easy to use. A complex protocol invites developers who build complex abstractions, not easy ones.
> OK, so Pidgin is a multi-protocol client which means the UI is abstracted out... But... https://developer.pidgin.im/ticket/15508 I don't understand their logic there, I have to say.
I have no idea, but some of my contacts use other clients, and they haven't gotten it to work either.
> For OTR, by the way, that just breaks with multi-device anyway. There's some experimental work with Axolotl underway which looks very promising.
Depends. XMPP seems somewhat lacking in its notification of which resources are 'active' (in common implementations), which is something that would help to at least synchronize online devices (by sending differently encrypted messages to different resources). Log synchronization is trickier, but really should be possible as well.
> I won't disagree there. The lack of a consensus on UX considerations (in fact, the lack of any attempt to get such a consensus) means that different clients present even simple things like jids differently. That's something the community (or the XSF) could seek to address.
I am not just refering to end-user UX, though - also to developer UX. Complex protocols with complex libraries aren't developer-friendly, and the same issue occurs as above: complex abstractions invite developers who write complex end user products.
It's very hard to get simple end user products built for complex protocols, because those require two fundamentally different types of developers.
> That said, I'm aware that one key problem faced by a federated network is that client on-boarding can't, so easily, create your account for you, and early attempts to streamline that process have left serious abuse problems in their wake.
I think the big mistake there was to leave the exact registration data undefined. Rather, there should have been a standardization of 'common registration fields' (including eg. CAPTCHA data), and later amendment XEPs if things like CAPTCHAs were to have to change.
As it is, in-band registration is disabled almost everywhere, because no client can deal well with bot prevention measures.
Mind sharing which desktop client you use (and on which OS)?
I use Gajim on Linux and Conversations on Android, but the technology you're looking for is "XEP-0280: Message Carbons".
XML is unnecessarily complex but I think JSON also is.
IRC and MSNP (the early versions, before they started XML-ifying everything) are what I'd consider a good format protocol for the application - a mostly line-based, textual one you can use pretty easily without any client more complex than netcat. They are also extensible enough even without XML.
(Personally, I think it is absurd that a protocol needs to wrap a single text message, the most common form of message that would be encountered, in several times more bytes than the message itself.)
IRC and MSNP (the early versions, before they started XML-ifying everything) are what I'd consider a good format protocol for the application - a mostly line-based, textual one you can use pretty easily without any client more complex than netcat. They are also extensible enough even without XML.
(Personally, I think it is absurd that a protocol needs to wrap a single text message, the most common form of message that would be encountered, in several times more bytes than the message itself.)
IRC has major issues, actually - notably, it's very tricky to parse correctly (and very easy to think you're parsing it correctly when you aren't), and doesn't support more than one space-containing value per 'line'. It's also implemented differently by every single server.
You will need some kind of serialization format for a reliable message protocol. JSON is just the simplest possible format that can reasonably represent common data, and works in 95% of the usecases.
You will need some kind of serialization format for a reliable message protocol. JSON is just the simplest possible format that can reasonably represent common data, and works in 95% of the usecases.
I'm fond of Avro. It has both a compact binary format and a human-readable plaintext format that's nearly identical to JSON (the only difference being that order of fields is significant in Avro but insignificant in JSON), with tools to convert between them.
I personally found BSON* (or maybe protobuf) a better format for a wire protocol. I really miss support for a date type and binary data in JSON (for my own applications).
* http://bsonspec.org/
* http://bsonspec.org/
msgpack[1] or LEON[2] might be more useful alternatives, in that case, depending on your interoperability requirements.
[1] http://msgpack.org/ [2] https://github.com/raypulver/LEON
[1] http://msgpack.org/ [2] https://github.com/raypulver/LEON
> XMPP is not battery-friendly on mobile devices (this primarily has/had to do with the absence of push notifications, it seems).
There is actually an unbelievably good XEP (XMPP extension protocol) on mobile battery life.[1] I learned things from it: radio power states and so forth. It's incredibly detailed and educational and well worth the read even if you don't use XMPP. That being said, so far as actual implementations go, I have to agree with you.
> XML is unnecessarily complex for this kind of usecase, and too hard and time-consuming to work with (and I agree). That is most likely what 'slow' refers to - not slow in parsing it, but in developing for it.
I have an ever-evolving conformant XMPP server pet project. What I found is that the framework takes bloody ages and I have to agree with you there. However, XMPP is a protocol that is ripe for elegant architecture and you can make a great framework. If you spend the time making a good framework the protocol is an absolute pleasure to develop against: but you have to make that initial time expenditure.
> The correct technology is JSON-over-WebSockets, and it doesn't really make sense to compare XMPP to a known-bad implementation and say "see? XMPP isn't that bad!"
Sort of. XMPP is one of the few applications that actually really makes use of the "eXtensible" bit in XML (where it's often misused as a dumb object graph). This means that you can do something as follows:
Either way, XMPP inherits a lot of traits from XML. You either need it or you don't and if you are using it there is a good chance that you are misusing it.
[1]: http://xmpp.org/extensions/xep-0286.html
There is actually an unbelievably good XEP (XMPP extension protocol) on mobile battery life.[1] I learned things from it: radio power states and so forth. It's incredibly detailed and educational and well worth the read even if you don't use XMPP. That being said, so far as actual implementations go, I have to agree with you.
> XML is unnecessarily complex for this kind of usecase, and too hard and time-consuming to work with (and I agree). That is most likely what 'slow' refers to - not slow in parsing it, but in developing for it.
I have an ever-evolving conformant XMPP server pet project. What I found is that the framework takes bloody ages and I have to agree with you there. However, XMPP is a protocol that is ripe for elegant architecture and you can make a great framework. If you spend the time making a good framework the protocol is an absolute pleasure to develop against: but you have to make that initial time expenditure.
> The correct technology is JSON-over-WebSockets, and it doesn't really make sense to compare XMPP to a known-bad implementation and say "see? XMPP isn't that bad!"
Sort of. XMPP is one of the few applications that actually really makes use of the "eXtensible" bit in XML (where it's often misused as a dumb object graph). This means that you can do something as follows:
<message from='...' to='...'>
<body>You have work, go to http://... to action it.</body>
<form xmlns='urn:my-proprietary-stuff'>AF3BD2</form>
</message>
Either way, XMPP inherits a lot of traits from XML. You either need it or you don't and if you are using it there is a good chance that you are misusing it.
[1]: http://xmpp.org/extensions/xep-0286.html
JSON name conflicts are easily resolved by giving every extension its own key in the object. That's the equivalent of namespaces.
I disagree. XML namespaces are URIs and give XMPP distributed extensibility. If you do this in JSON you are just replacing pointy brackets with curly ones.
I do think this is a clever solution (so have an upvote), but it's not exactly the same thing.
A lot of things get a bad name because they are used incorrectly. Look at SQL: it's actually great if you use it for the correct problems. The same thing applies to XML: if you're actually using it for the correct problems it's somewhat irreplaceable (XMPP is a beautiful example of how XML should be used).
Putting namespaces into JSON seems to fall into that dangerous territory - rather use the tool that's designed for that job from the ground up. JSON is great at the 99% of things that XML gets abused for, but it simply can't replace the 1% of correct use-cases for XML.
A lot of things get a bad name because they are used incorrectly. Look at SQL: it's actually great if you use it for the correct problems. The same thing applies to XML: if you're actually using it for the correct problems it's somewhat irreplaceable (XMPP is a beautiful example of how XML should be used).
Putting namespaces into JSON seems to fall into that dangerous territory - rather use the tool that's designed for that job from the ground up. JSON is great at the 99% of things that XML gets abused for, but it simply can't replace the 1% of correct use-cases for XML.
XMPP isn't a correct usecase for XML, though. It could easily be accomplished with a format like JSON without notable architectural defects. In a way, namespaces are a part of the JSON design - it supports deeply nested structures.
For a correct use of XML, I'd more suggest looking at static datasets with lots of complex inter-references and data types - for example, public transport data.
For a correct use of XML, I'd more suggest looking at static datasets with lots of complex inter-references and data types - for example, public transport data.
XML namespaces aren't just keys in a JSON hash, unless you're planning on naming all your keys like so:
Further, you have to understand that the key thing about the XML namespaces is primarily social, not technical: The use of the URI in the namespace (which is otherwise just an advisory string with no meaning) is also a licence to stick things in the value that you fully expect no-one else to understand, in a way that you know is guaranteed not to cause a conflict (in the absense of hostile intent). The guaranteed lack of conflict gives you that license. If you intend to match that with JSON keys, you must either copy the XML namespace's approach or something very similar, or run the risk of stomping on each other's keys, which socially creates pressure back towards some sort of licensing mechanism/standards body where people must agree on things before they can be used. (XML namespaces are also important because URIs of domains you own are also one of the few relatively unique keys that you can get without permission from a central (XMPP) authority. It's not the only option, but it's worth observing that "just pick something!" still has problems.)
On the one hand, people have been complaining about XMPP for years and years... on the other hand, I'd point out that given that it is also surviving, one must ask the question "why?" And I'd submit that this may be a significant part of the reason why. If you want sane and correct extension on an IM protocol, XMPP is in fact the best choice.
{"http://joepie91.com/XMPP/rtf_content content": "..."}
Further, you have to understand that the key thing about the XML namespaces is primarily social, not technical: The use of the URI in the namespace (which is otherwise just an advisory string with no meaning) is also a licence to stick things in the value that you fully expect no-one else to understand, in a way that you know is guaranteed not to cause a conflict (in the absense of hostile intent). The guaranteed lack of conflict gives you that license. If you intend to match that with JSON keys, you must either copy the XML namespace's approach or something very similar, or run the risk of stomping on each other's keys, which socially creates pressure back towards some sort of licensing mechanism/standards body where people must agree on things before they can be used. (XML namespaces are also important because URIs of domains you own are also one of the few relatively unique keys that you can get without permission from a central (XMPP) authority. It's not the only option, but it's worth observing that "just pick something!" still has problems.)
On the one hand, people have been complaining about XMPP for years and years... on the other hand, I'd point out that given that it is also surviving, one must ask the question "why?" And I'd submit that this may be a significant part of the reason why. If you want sane and correct extension on an IM protocol, XMPP is in fact the best choice.
> If you intend to match that with JSON keys, you must either copy the XML namespace's approach or something very similar, or run the risk of stomping on each other's keys, which socially creates pressure back towards some sort of licensing mechanism/standards body where people must agree on things before they can be used.
I understand the problem. I just don't feel that the way XML solves it is the correct one.
> (XML namespaces are also important because URIs of domains you own are also one of the few relatively unique keys that you can get without permission from a central (XMPP) authority. It's not the only option, but it's worth observing that "just pick something!" still has problems.)
Right, the point is that it's over-engineered. For usecases like this, you don't need guaranteed-unique namespaces - they just need to unique enough within your application that collisions are unlikely to ever happen.
For all practical purposes and intents, a key along the lines of `joepie91:carbons` (ie. a URN-like) would be sufficient - nickname and company name reuse is already non-zero but limited, and the cumulative chance of both having a name reused and working with the same kind of data (ie. conflict-sensitive) is negligible.
Domain URIs aren't quite decentralized (socially) either, by the way - in practice, it relies very heavily on the DNS system, which is very centralized.
> On the one hand, people have been complaining about XMPP for years and years... on the other hand, I'd point out that given that it is also surviving, one must ask the question "why?" And I'd submit that this may be a significant part of the reason why. If you want sane and correct extension on an IM protocol, XMPP is in fact the best choice.
"It's still around" in no way leads to "it's the best choice", logically. There are many, many reasons why an inferior, poorly-designed or overly complex protocol might still be in active use.
Conversely, it could be asked: why is XMPP not universally used, like HTTP is, and why is the tooling so substandard?
I understand the problem. I just don't feel that the way XML solves it is the correct one.
> (XML namespaces are also important because URIs of domains you own are also one of the few relatively unique keys that you can get without permission from a central (XMPP) authority. It's not the only option, but it's worth observing that "just pick something!" still has problems.)
Right, the point is that it's over-engineered. For usecases like this, you don't need guaranteed-unique namespaces - they just need to unique enough within your application that collisions are unlikely to ever happen.
For all practical purposes and intents, a key along the lines of `joepie91:carbons` (ie. a URN-like) would be sufficient - nickname and company name reuse is already non-zero but limited, and the cumulative chance of both having a name reused and working with the same kind of data (ie. conflict-sensitive) is negligible.
Domain URIs aren't quite decentralized (socially) either, by the way - in practice, it relies very heavily on the DNS system, which is very centralized.
> On the one hand, people have been complaining about XMPP for years and years... on the other hand, I'd point out that given that it is also surviving, one must ask the question "why?" And I'd submit that this may be a significant part of the reason why. If you want sane and correct extension on an IM protocol, XMPP is in fact the best choice.
"It's still around" in no way leads to "it's the best choice", logically. There are many, many reasons why an inferior, poorly-designed or overly complex protocol might still be in active use.
Conversely, it could be asked: why is XMPP not universally used, like HTTP is, and why is the tooling so substandard?
You've added in a lot of things to my post that I didn't say. Just to take one for instance:
""It's still around" in no way leads to "it's the best choice", logically."
Yes, I know. It's why I never made the claim that it's the best. I think XMPP has some serious flaws that kept it from being much more successful.
I also think XML isn't any of them. Convert XMPP naively to JSON, and it's still too difficult for people to bring up a stream and send an IM. One change I'd make, for instance, is for initiating a stream to default in one tag to a sensible initialized state, the state people expect a new IM stream to be in, and allow the user to override the defaults to be more manual, rather than what it is now. The spec is full of things like that, though, to some extent IM is a harder problem than people think.
"XML" is just a cognitively-available excuse. It's not the core problem. Personally, if I had to write a new protocol, JSON wouldn't actually be in the running; I'd either stick to XML or go to something like Protocol Buffers. I'd say JSON is actually in a really bad place for a big IM protocol, where it's still very, very verbose and has bad parsing characteristics (text-based, must parse whole message before you can do anything with it) but also fails at things XML is good at, like rich text and extensibility (JSON is bad at mixed text). And both are pretty bad at being streamed, for what it's worth.
"For all practical purposes and intents, a key along the lines of `joepie91:carbons` (ie. a URN-like) would be sufficient - nickname and company name reuse is already non-zero but limited, and the cumulative chance of both having a name reused and working with the same kind of data (ie. conflict-sensitive) is negligible."
That's not really disagreement. You just want to run the risk of using a shorter URN than everybody else, for the convenience of... uh... not sure actually, since this is all auto-generated JSON anyhow. It's not like you're saving the user typing, and choosing JSON means you've already thrown away wire-space efficiency. (Or are using gzip or something, and gzip'ed XML-XMPP and gzip'ed JSON-XMPP will be virtually identical in size on a stable stream.) You can already use that exact URN as an XML namespace today, if you so choose.
""It's still around" in no way leads to "it's the best choice", logically."
Yes, I know. It's why I never made the claim that it's the best. I think XMPP has some serious flaws that kept it from being much more successful.
I also think XML isn't any of them. Convert XMPP naively to JSON, and it's still too difficult for people to bring up a stream and send an IM. One change I'd make, for instance, is for initiating a stream to default in one tag to a sensible initialized state, the state people expect a new IM stream to be in, and allow the user to override the defaults to be more manual, rather than what it is now. The spec is full of things like that, though, to some extent IM is a harder problem than people think.
"XML" is just a cognitively-available excuse. It's not the core problem. Personally, if I had to write a new protocol, JSON wouldn't actually be in the running; I'd either stick to XML or go to something like Protocol Buffers. I'd say JSON is actually in a really bad place for a big IM protocol, where it's still very, very verbose and has bad parsing characteristics (text-based, must parse whole message before you can do anything with it) but also fails at things XML is good at, like rich text and extensibility (JSON is bad at mixed text). And both are pretty bad at being streamed, for what it's worth.
"For all practical purposes and intents, a key along the lines of `joepie91:carbons` (ie. a URN-like) would be sufficient - nickname and company name reuse is already non-zero but limited, and the cumulative chance of both having a name reused and working with the same kind of data (ie. conflict-sensitive) is negligible."
That's not really disagreement. You just want to run the risk of using a shorter URN than everybody else, for the convenience of... uh... not sure actually, since this is all auto-generated JSON anyhow. It's not like you're saving the user typing, and choosing JSON means you've already thrown away wire-space efficiency. (Or are using gzip or something, and gzip'ed XML-XMPP and gzip'ed JSON-XMPP will be virtually identical in size on a stable stream.) You can already use that exact URN as an XML namespace today, if you so choose.
The correct technology is JSON-over-WebSockets,
No, it's ASN.1 DER over TCP. :)
No, it's ASN.1 DER over TCP. :)
Or canonical S-expressions over TCP…
You'll be glad that someone got the IMAP reference... (Unless you're as niche as I, and were referring to ACAP).
Actually, I was thinking of SPKI (my current nominee for 'best idea which failed').
I really do wish that canonical S-expressions had won over XML for data transfer (as opposed to publishing markup): much more readable, incredibly easy to parse, fairly lightweight. Really, nothing wrong with them except for some reason people seem to be allergic to parentheses.
Why is that? Is it that one has to hit shift to type them on a standard keyboard?
I really do wish that canonical S-expressions had won over XML for data transfer (as opposed to publishing markup): much more readable, incredibly easy to parse, fairly lightweight. Really, nothing wrong with them except for some reason people seem to be allergic to parentheses.
Why is that? Is it that one has to hit shift to type them on a standard keyboard?
It's a platform thing. XML was pushed by SOAP and Java frameworks, while S-expressions were only used by simple software that solved small problems.
True but earlier: it was the HTML replacement that was also a data, description language. It's how I was introduced to it. I remember the tools were ridiculously overpriced: $100-200 for XML editor. Sold as better than SGML, which nobody wanted to even look at. Yet, any use of it for years required expensive 3rd party libs or my own custom PERL scripts. I said "I'll pass..." ;)
> XML was pushed by SOAP and Java frameworks, while S-expressions were only used by simple software that solved small problems.
I'd argue that S-expressions are used by simple software that solves complex problems, but I'm a bit biased
I'd argue that S-expressions are used by simple software that solves complex problems, but I'm a bit biased
I used S-expressions and/or Sun's XDR over a UDP/UDT variant. Could you believe there were efficiency and predictability gains over the prior HTTP method? It's like the mainstream approaches were hacked together rather than engineered for best fit. ;)
Or Bencode. Does anything use Bencode besides torrent files?
I used Bencode to encode messages for a presence and messaging system. Easy to write parsers for, binary safe, worked well overall. Probably the only real issue was that the sizes of lists and dictionaries aren't known in advance. This actually is a benefit in some streaming contexts, but calls for caution when dealing with user supplied data. JSON has the same problem, for what it's worth.
Or protobuf over TCP.
Bingo. If you don't need HTTP as the transport layer, protobuf is the way to go.
> XML is unnecessarily complex for this kind of usecase, and too hard and time-consuming to work with (and I agree)
What is 'difficult' in XML?
What is 'difficult' in XML?
Dealing with it. Namespaces, attribute access, type-casting, the entire shebang.
JSON can be very easily natively represented in just about any language: you have objects/hashmaps/dicts, arrays, booleans, strings, numbers, and null. This means that accessing something in a JSON object is simple - you just use standard member access in your language of choice.
XML is considerably harder to use - you have children, and attributes. You have namespaces, and so on. These make it impossible to use a basic 'data model' like for JSON - instead you need interactive objects, that let you traverse the tree.
You can no longer simply map/filter/reduce over data. You can't easily access deeply nested structures without lots of syntax clutter. You can't easily serialize back and forth. Instead of using what your language already provides for data structures, you now need to learn an entirely new one, specifically for XML.
That is what is 'difficult' in XML - it considerably increases the cognitive load for working with it. You can't keep the structure in your head easily, but need to look at documentation instead, and need to learn a specific (often bulky!) API just to interact with the data.
Realistically, XML is useful for storage of complex data, but you really shouldn't be using it for wire serialization like XMPP does. XML is useful in a small subset of usecases, and trying to apply it to everything else is a mistake.
JSON can be very easily natively represented in just about any language: you have objects/hashmaps/dicts, arrays, booleans, strings, numbers, and null. This means that accessing something in a JSON object is simple - you just use standard member access in your language of choice.
XML is considerably harder to use - you have children, and attributes. You have namespaces, and so on. These make it impossible to use a basic 'data model' like for JSON - instead you need interactive objects, that let you traverse the tree.
You can no longer simply map/filter/reduce over data. You can't easily access deeply nested structures without lots of syntax clutter. You can't easily serialize back and forth. Instead of using what your language already provides for data structures, you now need to learn an entirely new one, specifically for XML.
That is what is 'difficult' in XML - it considerably increases the cognitive load for working with it. You can't keep the structure in your head easily, but need to look at documentation instead, and need to learn a specific (often bulky!) API just to interact with the data.
Realistically, XML is useful for storage of complex data, but you really shouldn't be using it for wire serialization like XMPP does. XML is useful in a small subset of usecases, and trying to apply it to everything else is a mistake.
Namespaces is something I consider a strong plus for XMPP.
You may want to extend the data model at some point. You need a way to say "this is my extension, please ignore it". Or at least you need it if you want multiple implementations to work together.
This makes it possible to experiment over the live xmpp network.
map/filter/reduce etc is possible with XML. Scala does it. Other languages can do it to. It's purely depending on your language. Even JavaScript has better XML support than the language you're portraying here :-)
This makes it possible to experiment over the live xmpp network.
map/filter/reduce etc is possible with XML. Scala does it. Other languages can do it to. It's purely depending on your language. Even JavaScript has better XML support than the language you're portraying here :-)
> Namespaces is something I consider a strong plus for XMPP. You may want to extend the data model at some point. You need a way to say "this is my extension, please ignore it". Or at least you need it if you want multiple implementations to work together.
Sure. The problem is that in every implementation I've seen, they're unreasonably hard to work with. Giving each extension its own key in a JSON object accomplishes the same, without breaking the native data model.
> map/filter/reduce etc is possible with XML. Scala does it. Other languages can do it to. It's purely depending on your language. Even JavaScript has better XML support than the language you're portraying here :-)
Yes, it's possible, but it's too hard. It always introduces more complexity when compared to standard deserialized JSON (in any language that can natively represent the JSON data types, which seems to be most).
Sure. The problem is that in every implementation I've seen, they're unreasonably hard to work with. Giving each extension its own key in a JSON object accomplishes the same, without breaking the native data model.
> map/filter/reduce etc is possible with XML. Scala does it. Other languages can do it to. It's purely depending on your language. Even JavaScript has better XML support than the language you're portraying here :-)
Yes, it's possible, but it's too hard. It always introduces more complexity when compared to standard deserialized JSON (in any language that can natively represent the JSON data types, which seems to be most).
In fairness....in some statically typed languages like Go, working with JSON isn't a ton of fun either, if you need things like optional fields.
Well, that’s not a limitation of statically typed languages, but a limitation of go.
Just like Generics.
Java, for example, does support Optional types similar to Haskell’s Maybe.
Haskell in supports them, too
Just like Generics.
Java, for example, does support Optional types similar to Haskell’s Maybe.
Haskell in supports them, too
Namespaces are a nice idea but almost all XML libraries make them unnecessarily cumbersome by requiring full specification everywhere. I sometimes wonder whether just fixing that would have made XML more popular after having seen so many people try to write something like a selector using the format in the document (ns:element), burning time until they learned that you have to use FULL_URL:element and in some cases even declare the namespace names you can't use.
No, it's not the biggest challenge but it's so transparently a case of conplete disregard for developer productivity that I've heard it mentioned by a lot of people as why they avoid XML.
No, it's not the biggest challenge but it's so transparently a case of conplete disregard for developer productivity that I've heard it mentioned by a lot of people as why they avoid XML.
> No, it's not the biggest challenge but it's so transparently a case of conplete disregard for developer productivity that I've heard it mentioned by a lot of people as why they avoid XML.
This is the core issue with XML/XMPP. UX simply doesn't seem to have ever been a factor when designing either of them - whether towards the developers or the end users.
This is the core issue with XML/XMPP. UX simply doesn't seem to have ever been a factor when designing either of them - whether towards the developers or the end users.
Agreed: the XML and Java communities from the late 90s were plagued by the assumption that complexity was intrinsically good and that everyone wants to spend the time required to completely internalize a massive specification and all of its dependencies.
That is not at all a good characterization of what was going on. Do you remember what it was like before XML and Java? Could we have built anything on the scale of the web with CORBA?
(XMPP would've used JSON had it existed in 1998, fwiw.)
(XMPP would've used JSON had it existed in 1998, fwiw.)
In what sense was the web built on "XML and Java"?
The long arc from "what came before" has periodically involved cleaning up old complexity. There have always been arguments about how simple is simple enough, but these systems weren't designed with complexity as a goal.
The web is a lot of things, but broad consumer adoption of web technology also involves boring things like banks. SOAP and WSDL weren't created in a vacuum. They exist because Java / XML made it even possible to bridge the gap from the old world.
The web is a lot of things, but broad consumer adoption of web technology also involves boring things like banks. SOAP and WSDL weren't created in a vacuum. They exist because Java / XML made it even possible to bridge the gap from the old world.
I don't know what "old world" you're talking about. When the web was first a thing, it was written in C and PHP and Perl. All the Java crap was layered on top later, temporarily, and unsatisfactorily. The enterprise repeatedly misunderstands what the internet is and how to use it, but that is the enterprise's problem, no one else's.
Banks.
As much as we like all sorts of features of the web, integrating commerce was what brought the enterprise in, drove down costs, etc etc etc.
As much as we like all sorts of features of the web, integrating commerce was what brought the enterprise in, drove down costs, etc etc etc.
> Do you remember what it was like before XML and Java?
Yes – in particular, I remember liking the initial promise of XML 1.0 and then being disappointed as the initial simplicity was buried under layers of gratuitous complexity as the enterprise computing crowd jumped on board. Some neat ideas (e.g. XPath)
Similarly, my first database-backed web app was a Java servlet in 1996. I assumed it would become richer in the future but it was still pretty easy write and deployment was pretty simple at that point. Instead, the community chased off after more complexity (JavaBeans, the near-pervasive obsession with runtime configuration and the ensuing massive config files, nested abstraction layers, etc.) rather than focusing on basic developer productivity. We got massive piles of code like RMI (i.e. CORBA) before basic features like regular expressions, logging, etc.
> Could we have built anything on the scale of the web with CORBA?
No. We also failed, repeatedly, to do it with XML for reasons which are obvious to anyone who's ever used SOAP/WSDL/etc.
The common theme was seeing complexity and, especially, abstraction as a universal good rather than something with real costs and forgetting how important high-quality implementations are, particularly when you're relying on them to help with all of that complexity.
Yes – in particular, I remember liking the initial promise of XML 1.0 and then being disappointed as the initial simplicity was buried under layers of gratuitous complexity as the enterprise computing crowd jumped on board. Some neat ideas (e.g. XPath)
Similarly, my first database-backed web app was a Java servlet in 1996. I assumed it would become richer in the future but it was still pretty easy write and deployment was pretty simple at that point. Instead, the community chased off after more complexity (JavaBeans, the near-pervasive obsession with runtime configuration and the ensuing massive config files, nested abstraction layers, etc.) rather than focusing on basic developer productivity. We got massive piles of code like RMI (i.e. CORBA) before basic features like regular expressions, logging, etc.
> Could we have built anything on the scale of the web with CORBA?
No. We also failed, repeatedly, to do it with XML for reasons which are obvious to anyone who's ever used SOAP/WSDL/etc.
The common theme was seeing complexity and, especially, abstraction as a universal good rather than something with real costs and forgetting how important high-quality implementations are, particularly when you're relying on them to help with all of that complexity.
> The common theme was seeing complexity and, especially, abstraction as a universal good rather than something with real costs
Rich Hickey did a great service outlining some common problems when thinking about the word complexity itself: http://www.infoq.com/presentations/Simple-Made-Easy
I don't think it's about the engineers wanting to see complexity, so much as the problems you mention stemming from design-by-committee.
Rich Hickey did a great service outlining some common problems when thinking about the word complexity itself: http://www.infoq.com/presentations/Simple-Made-Easy
I don't think it's about the engineers wanting to see complexity, so much as the problems you mention stemming from design-by-committee.
I don't know. Namespaces can be complex, yes -- if you use them. You are not obliged to. Also, XML schemas and XSLT are immensely useful (the lack of schemas is one of the most annoying issues with JSON).
JSON doesn't lack schemas: http://json-schema.org/
And yes, you absolutely do have to use namespaces when working with a third-party protocol that chooses to use them. You can't opt out.
And yes, you absolutely do have to use namespaces when working with a third-party protocol that chooses to use them. You can't opt out.
You can't quote a word and ask them to explain it if they didn't even use that word.
I don't know if he has edited the original, if not, then sorry - I should have quoted 'hard'. (Not that it's unclear for anyone what I was referring to.)
The notion that XML is difficult comes from the most profoundly lazy evaluations possible, where the most minimal of up front understanding are effort are eschewed if you can just launch mongodb and insert some giant mystery meat bags of json. We've seen this same mistake play out over, and over, and over again, and it is incredible that in a field where projects often end up taking thousands or more of man hours, people sweat and complain about 15 minutes up front gaining a basic understanding of a structure or API, and instead pursue solutions that, for instance, doesn't even have a date type or even consistent hack standard of how to handle it.
This is how we have the rapid trends in this field, where the new panacea is tomorrow's nightmare.
This is how we have the rapid trends in this field, where the new panacea is tomorrow's nightmare.
To add to this, it is hardly justified to compare XML and JSON - XML and the associated standards are a lot more powerful and feature-rich than JSON and the same holds for the available tools. JSON works well for simple things, XML proofs its usefulness when things become more complex.
By the way, XMPP predates JSON or at least stems from a time when JSON was not jet standardized or widely accepted, so at least in the context of XMPP the XML vs JSON debate is pretty futile to begin with.
By the way, XMPP predates JSON or at least stems from a time when JSON was not jet standardized or widely accepted, so at least in the context of XMPP the XML vs JSON debate is pretty futile to begin with.
That's an important thing to note. The problem, however, is that a lot of people seem to be defending XML retroactively, because that's the choice that was made. An appeal to authority, if you will.
Realistically, it's completely understandable that XMPP was built on XML, and it made perfect sense at the time - but that should not preclude us from acknowledging that there are serious usability issues with it.
Realistically, it's completely understandable that XMPP was built on XML, and it made perfect sense at the time - but that should not preclude us from acknowledging that there are serious usability issues with it.
Was a reductio ad absurdum really necessary?
The significance of cognitive load in software development is well-understood, so it seems you're barking up to the wrong tree.
Yes, making things 'simpler' at the cost of architectural issues is bad - but that does not give you a wildcard to claim that complexity doesn't matter, as you seem to be doing right now.
A protocol like XMPP could easily use JSON without any significant architectural issues resulting from it.
Screaming "get off my lawn" to anybody who cares about developer efficiency, without any rational arguments underlying it, is not going to encourage them to use XMPP. It will just make them see you as a hostile dick that they don't want anything to do with. And that is not an insult - that is the reality of how people interpret these kind of posts.
The significance of cognitive load in software development is well-understood, so it seems you're barking up to the wrong tree.
Yes, making things 'simpler' at the cost of architectural issues is bad - but that does not give you a wildcard to claim that complexity doesn't matter, as you seem to be doing right now.
A protocol like XMPP could easily use JSON without any significant architectural issues resulting from it.
Screaming "get off my lawn" to anybody who cares about developer efficiency, without any rational arguments underlying it, is not going to encourage them to use XMPP. It will just make them see you as a hostile dick that they don't want anything to do with. And that is not an insult - that is the reality of how people interpret these kind of posts.
The most valuable thing XMPP gains from using XML is namespaces. The way namespaces have been defined to work on top of XML, is what allows the protocol to be distributely extensible. I.e. anyone can define new protocol without having to clear this with a centralized body, not even the XSF.
If you'd want this property in a JSON-based wire protocol, you'd invariably going to end up with XML-in-curly-brackets. This is why developing with/for XMPP is slightly harder than a protocol that has a limited feature scope.
However, as Dave mentions elsewhere, one shouldn't have to deal with the wire protocol directly to use XMPP in your application. A proper library should abstract from that. Arguably in some platforms, that's still a weak spot.
If you'd want this property in a JSON-based wire protocol, you'd invariably going to end up with XML-in-curly-brackets. This is why developing with/for XMPP is slightly harder than a protocol that has a limited feature scope.
However, as Dave mentions elsewhere, one shouldn't have to deal with the wire protocol directly to use XMPP in your application. A proper library should abstract from that. Arguably in some platforms, that's still a weak spot.
Was a reductio ad absurdum really necessary?
Yet there was none. Your very post, though, not only attempts to paint my comment as an "old person thing" (hilariously), it then attacks the speaker ("hostile dick"). Before you (incorrectly) start claiming fallacies, maybe commit to witholding them yourself.
My comment absolutely rises defensiveness among some people. But this is an endemic issue in this industry, and your personal feeling of diminished professionalism in the face of it needn't merit attack responses -- countless projects chose technologies not because they were efficient in any real world sense of that word (saving 15 minutes on the first day to pay thousands of hours later is hardly anyone's measure of "efficiency", and that you attempt to attack the very clear statement I made boggles the mind), but because they were the laziest, most accessible option available, saving a tiny amount of effort at the forefront.
Generating code is easy. Generating tonnes of code is easy. Generating good, well considered solutions, learning and adopting appropriate technologies and choices...well that's hard.
Yet there was none. Your very post, though, not only attempts to paint my comment as an "old person thing" (hilariously), it then attacks the speaker ("hostile dick"). Before you (incorrectly) start claiming fallacies, maybe commit to witholding them yourself.
My comment absolutely rises defensiveness among some people. But this is an endemic issue in this industry, and your personal feeling of diminished professionalism in the face of it needn't merit attack responses -- countless projects chose technologies not because they were efficient in any real world sense of that word (saving 15 minutes on the first day to pay thousands of hours later is hardly anyone's measure of "efficiency", and that you attempt to attack the very clear statement I made boggles the mind), but because they were the laziest, most accessible option available, saving a tiny amount of effort at the forefront.
Generating code is easy. Generating tonnes of code is easy. Generating good, well considered solutions, learning and adopting appropriate technologies and choices...well that's hard.
> Yet there was none.
There was. You were painting an overly ridiculous consequence of the argument, and then arguing against that overly-ridiculed consequence.
> Your very post, though, not only attempts to paint my comment as an "old person thing" (hilariously), it then attacks the speaker ("hostile dick").
As I explicitly pointed out, that wasn't an insult. It was a pointer as to how other people will likely interpret your comment. Which was more or less confirmed by the downvotes on it.
I wasn't attacking you. I was trying to make you understand how other people would read your comment.
The rest of your post again uses an overly ridiculous version of the argument to argue against (that really has nothing to do with my original statement), so there's really not much point in me responding to that.
There was. You were painting an overly ridiculous consequence of the argument, and then arguing against that overly-ridiculed consequence.
> Your very post, though, not only attempts to paint my comment as an "old person thing" (hilariously), it then attacks the speaker ("hostile dick").
As I explicitly pointed out, that wasn't an insult. It was a pointer as to how other people will likely interpret your comment. Which was more or less confirmed by the downvotes on it.
I wasn't attacking you. I was trying to make you understand how other people would read your comment.
The rest of your post again uses an overly ridiculous version of the argument to argue against (that really has nothing to do with my original statement), so there's really not much point in me responding to that.
A Scheme Machine [1] with plenty code and hardware I just found: 60 pages
The R5RS Scheme specification w/ example code: 50 pages
XML 1.0 specification for structure only: 49 pages
Sun's efficient, binary XDR: 29 pages
Wirth's Oberon Programming Language spec: 21 pages
ECMA JSON specification: 14 pages
Jsmn - portable, JSON parser [2] in C: 7 pages
I'm sure the above illustrates quite well how difficult and complex XML is. Additionally, while students everywhere implement LISP, there's a whole chapter in Beautiful Code where writing a good XML parser is an accomplishment people pay to see. Far from minimal understanding, reading XML's specification is a HOW-TO guide for how non-programmers that haven't seen superior works (eg Scheme, Oberon) try to handle data management. It was horrible, I used s-expressions/XDR instead, and implemented whole thing in a few hundred lines of straight-forward code. Best of all, unlike XML, my code could've run through a tool to prove it free of the types of bugs hackers love to exploit.
So, there's my proof that XML is garbage. Essentially everything it does can be done better by using one or more tools with less complexity, more cost-effectiveness, and more straight-forward implementation. Nobody should waste time learning it. If work demands, they should just learn a library that handles the awful details for them and internally use something else. Also, they should avoid any protocol that depends on it: that's just bad design.
[1] https://www.cs.indiana.edu/ftp/techreports/TR413.pdf
[2] http://zserge.com/jsmn.html
The R5RS Scheme specification w/ example code: 50 pages
XML 1.0 specification for structure only: 49 pages
Sun's efficient, binary XDR: 29 pages
Wirth's Oberon Programming Language spec: 21 pages
ECMA JSON specification: 14 pages
Jsmn - portable, JSON parser [2] in C: 7 pages
I'm sure the above illustrates quite well how difficult and complex XML is. Additionally, while students everywhere implement LISP, there's a whole chapter in Beautiful Code where writing a good XML parser is an accomplishment people pay to see. Far from minimal understanding, reading XML's specification is a HOW-TO guide for how non-programmers that haven't seen superior works (eg Scheme, Oberon) try to handle data management. It was horrible, I used s-expressions/XDR instead, and implemented whole thing in a few hundred lines of straight-forward code. Best of all, unlike XML, my code could've run through a tool to prove it free of the types of bugs hackers love to exploit.
So, there's my proof that XML is garbage. Essentially everything it does can be done better by using one or more tools with less complexity, more cost-effectiveness, and more straight-forward implementation. Nobody should waste time learning it. If work demands, they should just learn a library that handles the awful details for them and internally use something else. Also, they should avoid any protocol that depends on it: that's just bad design.
[1] https://www.cs.indiana.edu/ftp/techreports/TR413.pdf
[2] http://zserge.com/jsmn.html
This is the most breathtakingly ironic defense of XML I have ever seen.
In what way is my post "ironic"? I would dearly love to know. Do you think my defense of XML is somehow that I am too lazy to know alternatives. You would be painfully wrong, dear friend, though such is a go to among the intellectually lazy (how many failed NoSQL projects were justified on notions of "people who criticize elements of it are just old and fear change". It's the laziest, most unprofessional retort possible, and fueled about a year of content on Hacker News, and was the meat of a million failed adventures)
My post will be pummeled by the defensive, and that's okay.
My post will be pummeled by the defensive, and that's okay.
All of your invective was exactly what programmers were saying to XML proponents in the early days of XML. I still can't entirely tell if you're deliberately doing this or if you really are just wholly unaware of yourself.
> The notion that binary serialization is difficult comes from the most profoundly lazy evaluations possible, where the most minimal of up front understanding are effort are eschewed if you can just open a text file and insert some giant mystery meat bags of XML.
etc etc
> The notion that binary serialization is difficult comes from the most profoundly lazy evaluations possible, where the most minimal of up front understanding are effort are eschewed if you can just open a text file and insert some giant mystery meat bags of XML.
etc etc
I recall approximately no one making that argument. That isn't to say there aren't terrible misuses of XML (the database within a database notion that, while sometimes necessary, is every beginner's mistake), but you're juxtaposing an absurd invention that isn't actually real at all. XML filled a role.
> 1. XMPP is not battery-friendly on mobile devices (this primarily has/had to do with the absence of push notifications, it seems).
I never really understood this complaint. Why are the notifications sent via XMPP's TCP channel not considered "push"? How do you expect to get push notifications without an open TCP connection anyway?
I never really understood this complaint. Why are the notifications sent via XMPP's TCP channel not considered "push"? How do you expect to get push notifications without an open TCP connection anyway?
The idea is that push notifications are handled by a single platform vendor, and you thus only have one TCP connection to maintain for all applications together, instead of a separate connection for each application.
It's a form of batching.
It's a form of batching.
Read: Route everything trough one vendors servers.
Instead of something like a network traffic scheduler or a signal that the radio is on and now is a good time to send stuff.
Instead of something like a network traffic scheduler or a signal that the radio is on and now is a good time to send stuff.
> 1. XMPP is not battery-friendly on mobile devices (this primarily has/had to do with the absence of push notifications, it seems).
I do use XMPP in a mobile environment without any noticeable impact on battery runtime. It's not the protocol which drains your battery, but how one uses it: If there is frequent activity in terms of XMPP stanzas being exchanged between your device and the device's XMPP server, then the device will be unable to e.g. put the radio to sleep. Which, in turn, is of course causing battery drain.
The golden rules to prevent that:
1. Only send XMPP stanzas when necessary
2. Bundle and defer outgoing stanzas if possible
3. Tell your XMPP service when the device is inactive, so that the service is able to optimize the traffic (XEP-352: Client State Indiation [1])
4. Consider service-side stanza blocking (e.g. XEP-16: Privacy Lists) for stanzas of "unknown" origin to prevent malicious users from draining your battery.
1: https://xmpp.org/extensions/xep-0352.html
I do use XMPP in a mobile environment without any noticeable impact on battery runtime. It's not the protocol which drains your battery, but how one uses it: If there is frequent activity in terms of XMPP stanzas being exchanged between your device and the device's XMPP server, then the device will be unable to e.g. put the radio to sleep. Which, in turn, is of course causing battery drain.
The golden rules to prevent that:
1. Only send XMPP stanzas when necessary
2. Bundle and defer outgoing stanzas if possible
3. Tell your XMPP service when the device is inactive, so that the service is able to optimize the traffic (XEP-352: Client State Indiation [1])
4. Consider service-side stanza blocking (e.g. XEP-16: Privacy Lists) for stanzas of "unknown" origin to prevent malicious users from draining your battery.
1: https://xmpp.org/extensions/xep-0352.html
What are your thoughts on http://op-co.de/blog/posts/mobile_xmpp_in_2014/ ?
tl;dr It's being worked on (or already solved), but I'm sure the XMPP community welcomes additional contributors.
1-3. Could be solved with upcoming versions of XEP-313 MAM.
4. e2e There is currently a XSF GSOC project running, bringing axolotl to Conversations (and eventually describing the protocol as an XEP).
5. There are a few standards on how to perform encrypted file transfers, but implementations are lacking. You are invited to code one.
6. XEP-357: https://xmpp.org/extensions/xep-0357.html
1-3. Could be solved with upcoming versions of XEP-313 MAM.
4. e2e There is currently a XSF GSOC project running, bringing axolotl to Conversations (and eventually describing the protocol as an XEP).
5. There are a few standards on how to perform encrypted file transfers, but implementations are lacking. You are invited to code one.
6. XEP-357: https://xmpp.org/extensions/xep-0357.html
> I also don't really understand the comparison with "JSON over HTTP"
One of the complainers is matrix (https://matrix.org/), which does JSON over HTTP.
One of the complainers is matrix (https://matrix.org/), which does JSON over HTTP.
Which has it's own set of problems, btw.
Try to find an API for "delete".
Oh, and I'm eager to see how their API survives mutual parties extending it. (There is a reason why namespaces exists)
Oh, and I'm eager to see how their API survives mutual parties extending it. (There is a reason why namespaces exists)
http://matrix.org/docs/spec/#m-room-redaction is the current API for removing messages.
Matrix uses namespaces for extension to survive mutual parties extending it (http://matrix.org/docs/spec/#events), and we haven't seen any problems there yet.
That said, Matrix is still beta and we certainly do have a whole bunch of other problems :)
Matrix uses namespaces for extension to survive mutual parties extending it (http://matrix.org/docs/spec/#events), and we haven't seen any problems there yet.
That said, Matrix is still beta and we certainly do have a whole bunch of other problems :)
JSON over HTTP is not that bad concept. With the HTML5 server-sent events, this becomes a really cool stuff. And surprisingly it is working very well. For developers this is an order of magnitude easier and faster to develop rather than using low-level websockets which is reinventing the wheel.
> absence of push
If you are connected and your buddy sends a message you get it right away.
If you are connected and your buddy sends a message you get it right away.
On mobile you want messages to come through without each app maintaining a connection.
Because?
Because iOS doesn't let you do that (see http://op-co.de/blog/posts/mobile_xmpp_in_2014/#index6h2)
This is the real answer - long-standing connections are banned by the platform monopolist - and should be much closer to the top.
Yes, but I think the claim was that a persistent TCP connection has a negative effect on battery life, which is not obvious to me. An idle connection should have no impact whatsoever.
Because TCP connections are expensive on your battery life. Especially if your network is flaky, and all those connections need to constantly repeat their handshakes.
Not really, TCP connections are how GCM does things even. Unless you're using keep alives or otherwise preventing sleep, I believe these devices are capable of sending an interrupt when there's activity on a network connection though I could be mistaken.
Idle TCP connections do not consume any battery.
And on a data connectivity change, e.g. GSM <-> WiFi switch, there is a good chance that a dozen other components start trying to re-establish their connections, which means the radio is awake anyway (for, usually, at most a few seconds).
And on a data connectivity change, e.g. GSM <-> WiFi switch, there is a good chance that a dozen other components start trying to re-establish their connections, which means the radio is awake anyway (for, usually, at most a few seconds).
> Idle TCP connections do not consume any battery.
An idle connection doesn't consume anything, but a useful idle connection will have some keepalive every X minutes. Multiply that by the number of connections your application will have and by the number of background application you run on your phone, and the radio will never truly sleep or enter the "low" mode. The solution wanted by Google and Apple is to maintain a single connection to their server on which every push is aggregated and sent, and that the OS itself manages.
An idle connection doesn't consume anything, but a useful idle connection will have some keepalive every X minutes. Multiply that by the number of connections your application will have and by the number of background application you run on your phone, and the radio will never truly sleep or enter the "low" mode. The solution wanted by Google and Apple is to maintain a single connection to their server on which every push is aggregated and sent, and that the OS itself manages.
I use my mobile XMPP connections without a TCP keepalive but send a server ping if there has been stanzas received in the last 30 minutes and get a useful and reliable XMPP connection without a noticeable impact on battery. If you use Android's AlarmManager to trigger the check, then Android will even take care of scheduling the "alarm" with other alarms for efficiency.
> Multiply that by the number of connections your application will have and by the number of background application you run on your phone, and the radio will never truly sleep or enter the "low" mode.
Why would X x Y idle connections (X: TCP connections per app, Y: applictions) prevent your radio from sleeping?
> Multiply that by the number of connections your application will have and by the number of background application you run on your phone, and the radio will never truly sleep or enter the "low" mode.
Why would X x Y idle connections (X: TCP connections per app, Y: applictions) prevent your radio from sleeping?
> Why would X x Y idle connections (X: TCP connections per app, Y: applictions) prevent your radio from sleeping?
In the worst case, if the pings aren't sent at the same moment and all applications have a timeout set to, let's say, 30 minutes, your phone will send a ping every 30/(X*Y) minutes. I don't know what the usual X and Y would be, but if they're high enough your radio will always be up (check out these slides, 23 in particular: https://www.igvita.com/slides/2013/io-radio-performance.pdf)
And that's only counting idle applications. If you have one that periodically polls a server, it gets even worse.
In the worst case, if the pings aren't sent at the same moment and all applications have a timeout set to, let's say, 30 minutes, your phone will send a ping every 30/(X*Y) minutes. I don't know what the usual X and Y would be, but if they're high enough your radio will always be up (check out these slides, 23 in particular: https://www.igvita.com/slides/2013/io-radio-performance.pdf)
And that's only counting idle applications. If you have one that periodically polls a server, it gets even worse.
Course, you shouldn't poll and if those mechanisms aren't coordinated, then you you will suffer. But that's why Android provides the AlarmManager API and I would expect other mobile platforms to provide something similar.
FWIW, I added the following to the Wiki page when I saw it was on HN, which might help give some context to what you're reading:
Hi. I'm Dave Cridland. There's a note just up there which says "(Draft blog post)", and that's been there from the start - but when I asked for some feedback from the public [email protected] mailing list, people posted it on Twitter, and then it got onto Hacker News. Which is all cool, it's not secret, but...
This has not been properly reviewed yet (though the community is starting to), and so may contain inaccuracies, errors, or simply bad wording that's inadvertently as misleading as some of the comments I'm trying to rebut, and so feedback is genuinely welcomed.
Also equally welcome is feedback of the kind that says "Yeah, but I think that XMPP is entirely made by KILLING KITTENS", or whatever, so I can either write a rebuttal, or agree with you and try to get the community to fix it.
This is, currently, my personal opinion, and may not represent that of the XMPP community or the XMPP Standards Foundation.
Just for the record, I don't think we kill any kittens.
Hi. I'm Dave Cridland. There's a note just up there which says "(Draft blog post)", and that's been there from the start - but when I asked for some feedback from the public [email protected] mailing list, people posted it on Twitter, and then it got onto Hacker News. Which is all cool, it's not secret, but...
This has not been properly reviewed yet (though the community is starting to), and so may contain inaccuracies, errors, or simply bad wording that's inadvertently as misleading as some of the comments I'm trying to rebut, and so feedback is genuinely welcomed.
Also equally welcome is feedback of the kind that says "Yeah, but I think that XMPP is entirely made by KILLING KITTENS", or whatever, so I can either write a rebuttal, or agree with you and try to get the community to fix it.
This is, currently, my personal opinion, and may not represent that of the XMPP community or the XMPP Standards Foundation.
Just for the record, I don't think we kill any kittens.
As always, when making these kind of statements, the question is: relative to what, and for what purpose? Without knowing that, it’s quite hard to know what the author of this page is getting at. Who is he arguing against, and what’s the context for these “myths”?
XMPP is sort of similar to all the browser technologies (HTML/DOM/CSS/etc.): it’s a large and complex spec which takes a large amount of work to implement even a basic version of, with lots of possible things to screw up.
It mixes various concerns together into a big jumble without the cleanest layering, which makes certain types of systems a pain to build on top of it. It’s based on a technology (XML) which is inherently heavyweight and not all that well suited to streaming applications. Tooling which exists around XML is mostly fairly irrelevant to an XMPP-like use case. If someone wanted to make an open IM protocol from scratch, they could make something much simpler, cleaner, and better organized.
On the other hand, all the other popular person-to-person IM protocols are proprietary, and XMPP is at least open, reasonably well documented, and with a relatively large community built up around it. Which counts for a lot.
If someone wants to build a new protocol, for example Google Wave, I would very strongly recommend against trying to shoehorn it into XMPP. On the other hand, if someone wants to make their chat client or server interoperate with existing chat networks, then XMPP is great.
XMPP is sort of similar to all the browser technologies (HTML/DOM/CSS/etc.): it’s a large and complex spec which takes a large amount of work to implement even a basic version of, with lots of possible things to screw up.
It mixes various concerns together into a big jumble without the cleanest layering, which makes certain types of systems a pain to build on top of it. It’s based on a technology (XML) which is inherently heavyweight and not all that well suited to streaming applications. Tooling which exists around XML is mostly fairly irrelevant to an XMPP-like use case. If someone wanted to make an open IM protocol from scratch, they could make something much simpler, cleaner, and better organized.
On the other hand, all the other popular person-to-person IM protocols are proprietary, and XMPP is at least open, reasonably well documented, and with a relatively large community built up around it. Which counts for a lot.
If someone wants to build a new protocol, for example Google Wave, I would very strongly recommend against trying to shoehorn it into XMPP. On the other hand, if someone wants to make their chat client or server interoperate with existing chat networks, then XMPP is great.
Don't forget Whisper Systems protocol used by TextSecure[1] which is based on Google's protobuf[2] and JSON in a REST api.
[1] https://github.com/WhisperSystems/TextSecure-Server/wiki/API...
[2] https://developers.google.com/protocol-buffers/
[1] https://github.com/WhisperSystems/TextSecure-Server/wiki/API...
[2] https://developers.google.com/protocol-buffers/
The wiki page was in response to the XMPP bit of http://matrix.org/blog/faq. Coincidentally Matrix is a Wave-like protocol that hasn't been shoehorned into XMPP.
One could get started with PSYC. It's open.
http://about.psyc.eu/Introduction
http://about.psyc.eu/Introduction
Apparently for me it is a myth that XMPP teams can make anything that works. And their site is logically down.
I am gonna make my grumpy old man. It still takes 1 day for an apprentice coder to :
- make an irc bot; - program a rs232 port; - to draw a pitcure accessing a framebuffer; - to make hello world in Tcl/Tk that is centered horizontally and vertically in a bigger canvas.
But seasoned devs cannot easily do the same with:
- xmpp; - usb; - css/js...
Either the quality of the devs is going down, or some progresses are not.
ADDENDUM technical issues in jabber: http://webcache.googleusercontent.com/search?q=cache:yGdLTF4...
I am gonna make my grumpy old man. It still takes 1 day for an apprentice coder to :
- make an irc bot; - program a rs232 port; - to draw a pitcure accessing a framebuffer; - to make hello world in Tcl/Tk that is centered horizontally and vertically in a bigger canvas.
But seasoned devs cannot easily do the same with:
- xmpp; - usb; - css/js...
Either the quality of the devs is going down, or some progresses are not.
ADDENDUM technical issues in jabber: http://webcache.googleusercontent.com/search?q=cache:yGdLTF4...
While I'll join you in demanding the kids off our lawns, I'm looking forward to seeing 120V AC run through the very last serial port. There's nothing like sending a message down the serial power and seeing nothing come back. So many wonderful ways to fail.
Does this device require a null modem cable?
When the manual said that all commands needed to be terminated with a '\n', did they actually mean "\r\n" and not mention it?
Am I sending ASCII bytes to a piece of hardware that expects strings of UTF-16, without documenting it?
Did one of the students swap the cables and I'm just talking to a deactivated motor controller?
Has this particular serial cable finally bit the dust?
Is the hardware expecting me to transmit data on pin seven for some ungodly reason?
Did one of the students swap the the detector cable and the pre-amp high voltage line and fry the board I'm trying to talk to?
These are all issues that I've had to contend with on serial communications. The big catch is that it's usually not just one of these problems, so I wind up needing to check every possible combination. Having a device on USB saves me a boatload of trouble. The devices hooks up to the computer through a single, standard type of cable. If I can see the device after I plug it in, it's a software problem. If I don't see it, there's a hardware problem. Just from the above scenarios, seeing the equipment NOT connect eliminates 75% of the phase space for the problem. If I DO see the hardware, I've cut out 96%. So much easier to debug with USB.
Does this device require a null modem cable?
When the manual said that all commands needed to be terminated with a '\n', did they actually mean "\r\n" and not mention it?
Am I sending ASCII bytes to a piece of hardware that expects strings of UTF-16, without documenting it?
Did one of the students swap the cables and I'm just talking to a deactivated motor controller?
Has this particular serial cable finally bit the dust?
Is the hardware expecting me to transmit data on pin seven for some ungodly reason?
Did one of the students swap the the detector cable and the pre-amp high voltage line and fry the board I'm trying to talk to?
These are all issues that I've had to contend with on serial communications. The big catch is that it's usually not just one of these problems, so I wind up needing to check every possible combination. Having a device on USB saves me a boatload of trouble. The devices hooks up to the computer through a single, standard type of cable. If I can see the device after I plug it in, it's a software problem. If I don't see it, there's a hardware problem. Just from the above scenarios, seeing the equipment NOT connect eliminates 75% of the phase space for the problem. If I DO see the hardware, I've cut out 96%. So much easier to debug with USB.
This is so true. XMPP is quite complex. Complex to the point that bringing up a possible bug on either GTalk generating XML or Mcabber parsing it will leave some seasoned XMPP developers scratching their heads for a while, till they find the source of the problem.
It's sad, because we don't have that many decentralized multiplatform protocols. I had high hopes for Signal, but so far there is just one closed server.
It's sad, because we don't have that many decentralized multiplatform protocols. I had high hopes for Signal, but so far there is just one closed server.
Protocols are getting more complicated. Sometimes needlessly - XMPP; sometimes out of necessity - we can't let you poke bits in the framebuffer directly, because the graphics card is now this almost-blackbox superscalar coprocessor that only incidentally maybe has a display hooked to a tiny little region of memory that you could perhaps think of as a "framebuffer", but the framebuffer is actually not linear, but tiled in some weird way for performance.
Actually making the complex and simple stuff coexist on the same system is a very... complicated challenge.
Actually making the complex and simple stuff coexist on the same system is a very... complicated challenge.
So here's a question (and a personal manifesto)
Would things be better if they were worse?
That is, what if we gave a little bit less of a shit about optimizations and preferred simple over fast?
We live in a world of magical black boxes that even experts in the field have trouble manipulating. We need to protect and promote the Freedom to Tinker and that means carefully considering when optimizations are really critical hindrances.
Would things be better if they were worse?
That is, what if we gave a little bit less of a shit about optimizations and preferred simple over fast?
We live in a world of magical black boxes that even experts in the field have trouble manipulating. We need to protect and promote the Freedom to Tinker and that means carefully considering when optimizations are really critical hindrances.
I have a feeling that USB's complexity is there for good reasons. As far as xmpp and css/js go, there definitely is a lot of complexity that could have been avoided with a different initial design, although hindsight is 20/20.
[deleted]
The advantage of JSON is that it's way easier to handle because it maps directly to structures that are available in every common programming language.
I agree that XML is only slightly more verbose than JSON but they should still make a JSON version of the spec or it will die. It should be pretty straight forward to mechanically change the XML into JSON.
They did: "XEP-0295: JSON Encodings for XMPP" https://xmpp.org/extensions/xep-0295.html
:-)
:-)
How about that! Then they need to publicize it. Make a website, say, jmpp.org (pronounced "jump") and a reference implementation.
You might want to read it. :-) The problem has mostly been that nobody's come up with a better JSON representation than one in a April 1st XEP (and they have been proposed - as ralphm says, they're mostly XML-with-curly-brackets, which makes them have the disadvantages of both).
On the other hand, both stanza.io and XMPP-FTW both use the same fixed-format (ie, inextensible) JSON format for XMPP just fine. They can do this because they drop unknown namespaces and don't intend to provide round-trip conversions, etc. So if you're after a JSON "interface" to XMPP, there is one, and it's pretty good.
Worth a look.
On the other hand, both stanza.io and XMPP-FTW both use the same fixed-format (ie, inextensible) JSON format for XMPP just fine. They can do this because they drop unknown namespaces and don't intend to provide round-trip conversions, etc. So if you're after a JSON "interface" to XMPP, there is one, and it's pretty good.
Worth a look.
Oh yeah (about it being a joke). Thanks for the other pointers.
> NOTICE: This document is Humorous. It MAY provide amusement but SHOULD NOT be taken seriously.
EDIT: Also the X in XMPP stands for eXtensible not XML
EDIT: Also the X in XMPP stands for eXtensible not XML
Serialization and de-serialisation should be abstracted. I should be able to use JSON, XML, TOML, YAML or MessagePack and just write a plugin.
Here's an old critique of XMPP by author of PSYC protocol:
http://about.psyc.eu/XMPP
I think it showed how its complexity vs value delivered tradeoff isnt so good. Decided against trying to implement the protocol.
I think it showed how its complexity vs value delivered tradeoff isnt so good. Decided against trying to implement the protocol.
Worth noting that two of the people fairly heavily quoted on the shortcomings of XMPP both serve on the XMPP technical Council at the moment. (One of them is me, the other is Philipp Hancke).
That is interesting. My initial guess would be one of the two:
(a) it's improved remarkably and such short comings don't exist
(b) it won in adoption, it's getting all the labor, and you two are simply trying to improve it on the technical council.
I'm going with (b) for now. Was I right? Or is XMPP now efficient, easy to understand, and easy to implement?
(a) it's improved remarkably and such short comings don't exist
(b) it won in adoption, it's getting all the labor, and you two are simply trying to improve it on the technical council.
I'm going with (b) for now. Was I right? Or is XMPP now efficient, easy to understand, and easy to implement?
Actually a mixture. Some shortcomings have vanished because while they're problems, they're solved problems. And adoption etc, too.
I'd note that I was never against XMPP, though; I just don't see the need to pretend it's perfect. I do, however, object when problems are claimed for it that it doesn't have.
I'd note that I was never against XMPP, though; I just don't see the need to pretend it's perfect. I do, however, object when problems are claimed for it that it doesn't have.
Makes sense. Is there an up-to-date reference on the problems it currently has? I'm not talking whatever technical papers or submissions to committees. More an article like the one I cited.
So did you implement PSYC instead?
No. I already had a reliable communication system over UDP. I tweaked it with some stuff from PSYC and elsewhere. Let me achieve needed goals without doing a whole PSYC system. Had I needed full XMPP alternative I might have built PSYC just to see what would happen.
Ofcourse if i (or anybody else) would write a chat protocol today i would make
changes. But i'm not even sure if i would abandon xml. I'm in the same boat as
you and think xml is too verbose and all that other stuff but i've recently
programmed some xmpp projects and i came to the conclusion that xml actually
realy does work well. And i didn't even use a library like sleekxmpp because
i wanted to know everything.
I'll just say: good luck designing something that'll be able to grow for 20 years
and still work today. You can realy feel where the X in XMPP comes from.
I have to mention the Conversations people. For me they are a big reason why i use xmpp. It's not only the client which is very good, but the stuff like http-uploads fix some very real problems in very little amount of code.
Also once axolotl-xmpp works and prosody 0.10 is released i see no reason not to use xmpp honestly, which i will regardless, even if everybody uses something else (like WhatsApp) i will simply write myself a transport server and continue to use Conversations.
While i don't use it, projects like biboumi are realy cool. With a simple combination of biboumi[1] and httpuploadcomponent[2] you now can share files in irc-rooms.
I still have more to say. I hope that once i have my perfect chat-setup i will write about it.
[1]: https://github.com/louiz/biboumi [2]: https://github.com/siacs/HttpUploadComponent
I have to mention the Conversations people. For me they are a big reason why i use xmpp. It's not only the client which is very good, but the stuff like http-uploads fix some very real problems in very little amount of code.
Also once axolotl-xmpp works and prosody 0.10 is released i see no reason not to use xmpp honestly, which i will regardless, even if everybody uses something else (like WhatsApp) i will simply write myself a transport server and continue to use Conversations.
While i don't use it, projects like biboumi are realy cool. With a simple combination of biboumi[1] and httpuploadcomponent[2] you now can share files in irc-rooms.
I still have more to say. I hope that once i have my perfect chat-setup i will write about it.
[1]: https://github.com/louiz/biboumi [2]: https://github.com/siacs/HttpUploadComponent
From where I sit typing this, I can look out of my window and see Bryn Myrddin - Merlin's Hill - underneath which, legend says, Merlin lies asleep, awaiting a time when England in is peril once more. This is a problematic myth, since Bryn Myrddin isn't in England...
"Merlin" might or might not be based on any historical figure, and such figure might or might not lie under that hill, and if so he probably knew a few Angles, but he never heard of "England", because it didn't exist for centuries after he might or might not have been active.
"Merlin" might or might not be based on any historical figure, and such figure might or might not lie under that hill, and if so he probably knew a few Angles, but he never heard of "England", because it didn't exist for centuries after he might or might not have been active.
Thanks for picking up on the important content there. ;-)
If you want to be pedantic, Merlin - assuming he existed in the most likely timeframe - probably was aware of Angleland to his east, and would have been defending against its encroachment. (This based on the rise of Arthur as a popular name around 550AD as I recall, during the "Saxon" invasion largely carried out by Angles). The country he lived in was probably called something like Britain, though may have simply been closer to Cymru.
But I was largely paraphrasing T.H. White's Once And Future King, which in turn cribs the La Morte d'Arthur, which forms the basis for the "modern" Arthurian legend, and recasts it in an idealized England.
If you want to be pedantic, Merlin - assuming he existed in the most likely timeframe - probably was aware of Angleland to his east, and would have been defending against its encroachment. (This based on the rise of Arthur as a popular name around 550AD as I recall, during the "Saxon" invasion largely carried out by Angles). The country he lived in was probably called something like Britain, though may have simply been closer to Cymru.
But I was largely paraphrasing T.H. White's Once And Future King, which in turn cribs the La Morte d'Arthur, which forms the basis for the "modern" Arthurian legend, and recasts it in an idealized England.
I'm always playing around with messaging for pet projects, and tried xmpp a couple of times.
Honest question for edification and future play around projects: what alternatives do exist, fixing the issues presented here?
I'd love a lean, multiplatform api, with presence, multi-user chat, extensible for my use case, etc.
Recommendations?
Honest question for edification and future play around projects: what alternatives do exist, fixing the issues presented here?
I'd love a lean, multiplatform api, with presence, multi-user chat, extensible for my use case, etc.
Recommendations?
I find it strange that, nobody has mentioned MQTT so far in this discussion. Isn't MQTT also a 'competitor' for XMPP, besides JSON-over-Websockets? especially on mobile devices ?
It is not really a competitor for XMPP. It transports messages and you could build a chat service on top of it, so it is in a somewhat related space, but XMPP and MQTT aim at very different things.
1. XMPP is not battery-friendly on mobile devices (this primarily has/had to do with the absence of push notifications, it seems).
2. XML is unnecessarily complex for this kind of usecase, and too hard and time-consuming to work with (and I agree). That is most likely what 'slow' refers to - not slow in parsing it, but in developing for it.
Neither of these are addressed.
I also don't really understand the comparison with "JSON over HTTP" - why would you ever use that for a realtime application?
The correct technology is JSON-over-WebSockets, and it doesn't really make sense to compare XMPP to a known-bad implementation and say "see? XMPP isn't that bad!"
Another issue with XMPP is the strange implementation of both resources and the existing options for sending to all resources (especially when taking into account tools like OTR). People want device synchronization, and it just isn't viable with XMPP right now.