JSON5 Data Interchange Format(json5.org)
json5.org
JSON5 Data Interchange Format
https://json5.org/
156 comments
It allows you to put double quotes in the string without escaping them, which is pretty nice.
But then you can't include single quotes into the string any more. Same problem again.
The most practical solution for this problem I saw was in F#. You can use tripple-quoted strings.
The most practical solution for this problem I saw was in F#. You can use tripple-quoted strings.
let a = """The diner is called "John's", but you can't let the string end with a double quote"""
Rust has similar but it scales infinitely. They're called raw strings,
"normal string can contain ' "
r#" raw string can contain " ' "#
r###" if for some reason you need to write "# "###
Most (newer) programming languages have a concept of multi-line string literals.
Most newer languages only have 1 variation, not infinite. Go only has backtick, python only has triple quotes. Rust has #", ##", ###", ##...#"
My favourite variation is from Zig:
Pair this with the fact that it only has the C++ style // comments, it means that every line can be tokenized independently. Not that that particularly matters for programmers, but I know Vim sometimes gets confused about where Python's strings end and it turns the syntax highlighting to garbage.
const hello_world_in_c =
\\#include <stdio.h>
\\
\\int main(int argc, char **argv) {
\\ printf("hello world\n");
\\ return 0;
\\}
;
Pair this with the fact that it only has the C++ style // comments, it means that every line can be tokenized independently. Not that that particularly matters for programmers, but I know Vim sometimes gets confused about where Python's strings end and it turns the syntax highlighting to garbage.
it also fixes the indentation problem
I would prefer PostgreSQL-style arbitrary quotes, but with nesting. Something like:
// the object key here is "weird object key '\""
{%{weird object key '":}%:"see how it works?"}
// ditto
{%""{weird object key '":}""%:"see how it works?"}This is why I like backticks. ` should almost never show up in normal English text.
And, luckily, all transmitted text is normal English.
I'd be interested to see a counter example where a standalone backtick is part of normal non-computing related text. I really don't find your comment helpful. The strings where I see single and double quotes being used most frequently is with normal, yes, English text. The backtick is useful there, and that's all I wanted to say. French doesn't really have the same issue, since guillemets (« and ») are often used in place of double quotes, for example.
According to this Stack Exchange answer[0], the backtick is strictly for use in computing. I understand that it represents the grave accent, but that isn't of concern as a string delimiter, because once it is applied to a letter, for example è, it has its own unique representation.
0: https://english.stackexchange.com/questions/249406/are-backt...
According to this Stack Exchange answer[0], the backtick is strictly for use in computing. I understand that it represents the grave accent, but that isn't of concern as a string delimiter, because once it is applied to a letter, for example è, it has its own unique representation.
0: https://english.stackexchange.com/questions/249406/are-backt...
I think the point was that it doesn't make sense to expect that strings in a format like JSON or a programming language will only be used for non-computing related plain text.
If English was good enough for Jesus Christ it's good enough for me
[deleted]
An annoying half-solution that breaks down as soon as you encounter content with both. Better imho to have one way and become good at it. I think I might even prefer trailing commas to be mandatory.
It solves the problem like 99% of the time in my experience. It's very rare that I have content with both single and double quotes.
That's exactly my point: 99% solutions are the worst. Every 99% solution will eventually find someone relying on it, willingly or not. And it's always someone else cleaning up the mess (or failing to find the cause).
There is no mess.
The 99% case is a pretty string. The 1% case is a string with escaped delimiters.
The 99% case is a pretty string. The 1% case is a string with escaped delimiters.
Why does it need commas at all? What purpose do they serve?
They don’t serve one. Clojure uses only white space for distinguishing items in lists and hashmaps and after a short period of “this is weird” you embrace it and wonder why you ever needed commas at all
Checksums, basically, drawing a line between random bytes and structured JSON in a way that comes at zero cost for a human mind (arguably less than zero). We might as well leave out all closing tokens at the end of the stream, in a way those are completely redundant but still worth it for the warm, fuzzy feeling of knowing that you haven't just read a truncated stream.
It's either commas or significant whitespace. No matter what you'll find someone who hates one or the other.
But everyone puts the whitespace in anyway so why not just make it significant?
What if you want both?
E.g., how would you put the following sentence in a string:
You can use both " and ' to delimit a string.
E.g., how would you put the following sentence in a string:
You can use both " and ' to delimit a string.
You'd use escapes:
"You can use both \" and ' to delimit a string"
"You can use both \" and ' to delimit a string"
That was their point, you inevitably have to escape one or the other, so just pick one
My only disagreement is that you imply that this is no better than plain old JSON – I think it's positively _worse_. It makes it difficult to remember which of the two quote marks to escape, without checking the beginning of the string to see what quote marks were there (and this difficulty exists not only for humans but for machines, which introduces more complexity in parsers).
The benefit of JSON is that it's simple, opinionated, and pretty. Putting bells and whistles on it ruins the very ethos of JSON. If you want JSON++, use XML or Jsonnet or whatever.
The benefit of JSON is that it's simple, opinionated, and pretty. Putting bells and whistles on it ruins the very ethos of JSON. If you want JSON++, use XML or Jsonnet or whatever.
I didn't imply that, in fact my comment agrees with what you are saying. Did you reply to the wrong person?
Haha, I only meant that as a "I agree with you, but I'd go even further" preface. In hindsight, this being HN, I should have been more precise!
Getting over your pet peeves is a superpower.
Yeah exactly. That's why they should have stuck with just double quotes. Double quotes are far more popular than single quotes anyway.
How about sticking with JavaScript, as long as that's what the J and the S in JSON stand for. And as we all know, JavaScript supports both single and double quotes.
Or they could pull a YAML, and retroactively redefine what JSON stands for, the way YAML originally stood for "Yet Another Markup Language" until some party pooper pointed out to their great surprise that YAML was not actually a document markup language, but a data serialization language, so they redefined it to mean "YAML Ain't Markup Language" -- the opposite of what it originally meant.
https://en.wikipedia.org/wiki/YAML#History_and_name
https://en.wikipedia.org/wiki/Markup_language
Pro Tip: When naming you file format, look up what the words in your acronym mean, first.
Or they could pull a YAML, and retroactively redefine what JSON stands for, the way YAML originally stood for "Yet Another Markup Language" until some party pooper pointed out to their great surprise that YAML was not actually a document markup language, but a data serialization language, so they redefined it to mean "YAML Ain't Markup Language" -- the opposite of what it originally meant.
https://en.wikipedia.org/wiki/YAML#History_and_name
https://en.wikipedia.org/wiki/Markup_language
Pro Tip: When naming you file format, look up what the words in your acronym mean, first.
[deleted]
Maybe they don't think it should be up to them to decide which one you should go with.
What's the point of standards then?
The standard says you can use either. Why do you disagree with the standard?
I disagree with the OP, not the standard. The standard says you can't use backticks for quoting strings, so it's still "up to them to decide which one you should go with".
[deleted]
To support people's use cases?
… could’ve at least disallowed for one of the most batshit crazy aspects of JSON, the legal but undefined behaviour of duplicate keys in objects.
> An object whose names are all unique is interoperable in the sense that all software implementations receiving that object will agree on the name-value mappings. When the names within an object are not unique, the behavior of software that receives such an object is unpredictable.
> An object whose names are all unique is interoperable in the sense that all software implementations receiving that object will agree on the name-value mappings. When the names within an object are not unique, the behavior of software that receives such an object is unpredictable.
It would lose its "backwards compatible" property though.
Having a defined, predictable behaviour in JSON5 doesn't seem like it would break backwards compatibility with JSON's undefined, unpredictable behaviour. I don't think someone with an existing JSON document would complain that a JSON5 parser behaved "too predictably" when interpreting their document.
It could change, and therefore break, the real-world behavior of a given parser that some code is knowingly or unknowingly relying on.
So could switching to another JSON parser. The response of the parser writer should be "I detected a bug in your code for you. You're welcome."
This attitude willfully ignores real-world constraints. There are plenty of "shoulds", but at the end of the day, changing behavior is changing behavior. Maybe it's worth it, maybe it isn't. But it's never free.
This can be solved if you base JSON5 on I-JSON (RFC 7493) instead of JSON (RFC 8259). Many (but not all) JSON implementations actually expect I-JSON, at least for object keys.
Those are fightin' words you know...
I've seen some of the endless flamewars about removing that.
The problem is ultimately that you kinda can't remove duplicate keys. You can only explain the ways in which handling them varies. I'm glossing over a lot of stuff though and I'll understand your skepticism.
I've seen some of the endless flamewars about removing that.
The problem is ultimately that you kinda can't remove duplicate keys. You can only explain the ways in which handling them varies. I'm glossing over a lot of stuff though and I'll understand your skepticism.
Regardless of how good this is or not, every time it pops up I just wonder if anybody actually gives a crap or not.
JSON is one of those formats that's "good enough" even if it's stinky and it's also universally accepted that a small iteration on it with small conveniences doesn't make that much sense imho.
JSON is one of those formats that's "good enough" even if it's stinky and it's also universally accepted that a small iteration on it with small conveniences doesn't make that much sense imho.
And XML is a very battle-tested alternative that supports every conceivable edge case and arbitrary depths of strictness and validation. There was soooo much energy poured in to making XML the One Format To Rule Them All. It has so many advanced features like strict definitions, includes syntax, support for mixed content, entities and references, comment syntax. But it turns out that 99% of users never needed any of that and just wanted nested key-value pairs.
That and any serialization format that has security implications is doomed to fail (or at least be derided).
What?!? BREH, read Effective Java on Serialization.
Every serialization format has security implications.
Some try to make this explicit and probably fail in capturing all conceivable holes. Some leave this implicit to give you "fun surprises."
Every serialization format has security implications.
Some try to make this explicit and probably fail in capturing all conceivable holes. Some leave this implicit to give you "fun surprises."
What "fun surprises" does JSON have, aside from "the object could be arbitrarily big" ?
A serialization format is more than just how things are described on the wire. It's also about how both sides are expected to receive and transmit it. Naive processing of JSON in certain situations can run into sharp edges like this:
https://rules.sonarsource.com/java/RSPEC-4544
Serialization formats are a part of Protocols, which essentially form the software equivalent of a spoken language between two people. Like words, the bytes can have a general syntax and expected semantic, but still be open to problematic interpretation. Like the conceptual difference between a language like English and Ithkuil. (http://www.newyorker.com/magazine/2012/12/24/utopian-for-beg...)
Like Ithkuil, some formats can specify "a little more" in a way that tries to prescribe semantic meaning and syntax such that implementing according to spec limits the opportunities for issues like this to happen.
https://rules.sonarsource.com/java/RSPEC-4544
Serialization formats are a part of Protocols, which essentially form the software equivalent of a spoken language between two people. Like words, the bytes can have a general syntax and expected semantic, but still be open to problematic interpretation. Like the conceptual difference between a language like English and Ithkuil. (http://www.newyorker.com/magazine/2012/12/24/utopian-for-beg...)
Like Ithkuil, some formats can specify "a little more" in a way that tries to prescribe semantic meaning and syntax such that implementing according to spec limits the opportunities for issues like this to happen.
It's true. I doubt this spec will pick up even a little traction either since it's not backwards compatible.
You trade slightly more concrete data structures for hard incompatibility.
Maybe an internal api could leverage this in some way?
But it really isn't that hard to wrap hex, floating points, etc. in quotes and parse that server side.
Any application that's not a pet project should be using json schemas anyways, which is the layer you could put your custom datatypes into.
You trade slightly more concrete data structures for hard incompatibility.
Maybe an internal api could leverage this in some way?
But it really isn't that hard to wrap hex, floating points, etc. in quotes and parse that server side.
Any application that's not a pet project should be using json schemas anyways, which is the layer you could put your custom datatypes into.
If I had to pick one feature add, it’d be allow trailing commas. That would make serializing a bit easier and less error prone.
It would also make slightly nicer diffs
JSON is "good enough" for a textual, mostly human readable data interchange format. But as a configuration format, for example the package.json file for node/npm/yarn, it is most definitely not "good enough". For two reasons: no comments and no multiline strings. The latter might not be important for some types of configuration, but it is for things that need something like a description that is more than a single sentence.
> But as a configuration format, for example the package.json file for node/npm/yarn, it is most definitely not "good enough".
Then perhaps don't use it for a use case it was not designed to address? I understand that JSON is a very handy hammer to have, but not all things are nails.
Then perhaps don't use it for a use case it was not designed to address? I understand that JSON is a very handy hammer to have, but not all things are nails.
It is just that people keep insisting on using JSON for human-maintained configuration files, and JSON sucks for that: no comments, trailing commas, or multi-line strings.
Since JSON5 is trivially pre-processed into JSON, it tends to be useful as a tack-on to such projects to let your startup scripts generate the real file given the useful on without having unrelated syntax.
Since JSON5 is trivially pre-processed into JSON, it tends to be useful as a tack-on to such projects to let your startup scripts generate the real file given the useful on without having unrelated syntax.
In practice, I've found most program configs allow comments and non quoted keys, for what it's worth.
Couple of gripes:
- Still no date type
- Why make it more loose? Like "Strings may be single quoted." doesn't bring any value.What would a date type look like? An ISO-8601 string is probably as good as it'll get, else you end up with an object containing loads of additional information like timezones, offsets, etc. There's also https://json-schema.org/understanding-json-schema/reference/... for a formalized date format in JSON.
The looseness seems to be aimed at human authors who want to write JSON as if it were JS, so mainly package.json. But, json is not ideal for configuration for multiple reasons.
The looseness seems to be aimed at human authors who want to write JSON as if it were JS, so mainly package.json. But, json is not ideal for configuration for multiple reasons.
If you wanted JSON to parse into actual date objects, you could always do:
{
thisIsADate: new Date('1995-12-17T03:24:00Z')
}
I think allowing a whitelisted set of constructors (so it's extensible, but with some standard ones) as types in a future JSON-like could be good. e.g.
Of course, my ideal NuJSON would also include bigints, multiline strings, comments, bare keys (i.e. without having to wrap them in quotes).
Stretch nuJSON would also include the ability to serialise circular graph structures, perhaps through the use of a Reference([up, up, "aSymbol"]) data type.
{
birthdayParty: Date(1632817436514),
sounds: Map([["cow", "moo"], ["fox", "?"]]),
possibilities: Set(["she loves me", "she loves me not"]),
aSymbol: Symbol("tag"),
aBuffer: UInt8Array(ArrayBuffer([104, 101, 108, 108, 111])),
}
const serialised = NuJSON.stringify(data)
const result = NuJSON.parse(serialised, allowedConstructors, fallbackConstructor)
Of course, my ideal NuJSON would also include bigints, multiline strings, comments, bare keys (i.e. without having to wrap them in quotes).
Stretch nuJSON would also include the ability to serialise circular graph structures, perhaps through the use of a Reference([up, up, "aSymbol"]) data type.
I know this is how Javascript is done nowadays but using constructors for maps and arrays just seems like unnecessary syntax to me. And if you're going to initialize an array buffer too, you might as well just not use JSON at all and send plain Javascript code over the wire.
I think all JSON needs in this regard is type hinting. Let everything else be handled by the application:
I think all JSON needs in this regard is type hinting. Let everything else be handled by the application:
{
birthdayParty: Date 1632817436514,
sounds: String? { // zomgwtfbbq comments too
cow:"moo",
fox: null
},
possibilities: ["she loves me", "she loves me not", 42],
aSymbol: Symbol "tag",
aBuffer: Uint8 [104, 101, 108, 108, 111]
}I think the main difference between my suggestion and yours is that mine has more brackets. I think type hinting and 'constructors' are basically the same thing, but I like the types to be extensible, so even in your version, I'd make the type hints things that could be provided into the NuJSON.parse function.
Obviously, in your version my approach would be
Obviously, in your version my approach would be
sounds: Map {
cow: "moo",
fox: null
},
possibilities: Set ["she loves me", "she loves me not", 42]
I suppose my point was that you don't have to specify types for maps and sets at all (which I didn't do in my example,) the existing syntax already does that. [] is already a set, {} is already a map. Type hints would just be used for the values.
Sending type definitions seems like a good idea. Although at that point it might as well not even be called any kind of JSON.
Sending type definitions seems like a good idea. Although at that point it might as well not even be called any kind of JSON.
I don't agree that you don't have to specify types for maps and sets. For example
I want to be able to stringify records with Sets, Maps, Arrays and Objects in, and have that work, even if they have behavior not supported by objects and arrays, and I don't think the user should have to fix up the output after parsing of JSON containing standard javascript objects.
{true: "yes", false: "no"}
Map [[true, "yes"], [false, "no"]]
I want to be able to stringify records with Sets, Maps, Arrays and Objects in, and have that work, even if they have behavior not supported by objects and arrays, and I don't think the user should have to fix up the output after parsing of JSON containing standard javascript objects.
Useful when you need to input already double-quoted value.
But it's a matter of taste still and nothing that we couldn't do before. Such changes will lead to meetings where programmers discuss JSON5 quoting choices (yes, it will be a thing).
But it's a matter of taste still and nothing that we couldn't do before. Such changes will lead to meetings where programmers discuss JSON5 quoting choices (yes, it will be a thing).
It brings it closer to JavaScript behavior I guess.
I guess it's nice to have ability to copy random js object declaration to actual json without editing
I agree about the single quotes, but many use single quotes for strings in JS and so it's understandable to have this consistent.
Because:
‘I am “really” angry’
“I am \”really\” angry”and now you have two problems
is this any better?
is this any better?
'I\'m "really" angry, don\'t do it again!'
Nobody's forcing you to use single quotes as delimiters.
"I'm \"really\" angry, don't do it again!"
"I'm \"really\" angry, don't do it again!"
Json is not meant to be human readable or writeable.
All text based data formats are meant to be human readable and human writable, and humans read and write JSON all the time.
You wouldn't say that mammals write C all the time, would you? Front end development would be so much better if you could just spew json at people in a table.
>You wouldn't say that mammals write C all the time, would you?
Yes. Mammals do write C all the time, because humans are mammals. And C is meant to be human readable and writable.
> Front end development would be so much better if you could just spew json at people in a table.
That's literally what a JSON API is, and there are tons of them. And for each of them, a human had to be able to read and comprehend JSON responses and write JSON requests in code. There's a reason JSON is pretty-printed in the debug consoles of browsers.
To say nothing of all of the package definitions or framework configs that are written (by hand) in JSON.
The fact that most JSON is machine generated at present has nothing to do with its original design intent. Most Javascript, CSS and HTML are machine generated nowadays as well, but they were meant to be written by a person using a text editor.
Yes. Mammals do write C all the time, because humans are mammals. And C is meant to be human readable and writable.
> Front end development would be so much better if you could just spew json at people in a table.
That's literally what a JSON API is, and there are tons of them. And for each of them, a human had to be able to read and comprehend JSON responses and write JSON requests in code. There's a reason JSON is pretty-printed in the debug consoles of browsers.
To say nothing of all of the package definitions or framework configs that are written (by hand) in JSON.
The fact that most JSON is machine generated at present has nothing to do with its original design intent. Most Javascript, CSS and HTML are machine generated nowadays as well, but they were meant to be written by a person using a text editor.
I've worked on 4+ years of technology that was entirely based on humans hand-tweaking, editing, and updating JSON configs. My text editors validate it quickly, and know how to fold, sort, prettify. Based on my experience (and editing it all day long) It's a very human readable and editable configuration.
Second sentence from json.org: "It is easy for humans to read and write."
But JSON5 is.
Single quote strings are useful when the string contains many double quotes (e.g. XML with attributes).
I know it’s not trendy but consider XML and schema at this point. XML allows all this to be expressed and more and schema allows the creation of integration contracts that can be validated at both ends. Tooling and libraries are mature.
The problem is not that XML isn't "trendy". It's annoying to use, especially in an environment where JSON is a first-class citizen.
JSON-schema is a thing as well, maybe not quite as mature (or "aged"), but the tooling is fine.
JSON-schema is a thing as well, maybe not quite as mature (or "aged"), but the tooling is fine.
Also massively overengineered.
XML has two big problems:
1. It's super verbose. Everybody hates writing it. And yes, people do write JSON by hand.
2. It has a weird and confusing data model with both attributes and child elements. Do you do <foo a="1"/> or <foo><a>1</a></foo>? In many many cases it's ambiguous and you end up with a weird mix, whereas in JSON it's obvious and easy.
1. It's super verbose. Everybody hates writing it. And yes, people do write JSON by hand.
2. It has a weird and confusing data model with both attributes and child elements. Do you do <foo a="1"/> or <foo><a>1</a></foo>? In many many cases it's ambiguous and you end up with a weird mix, whereas in JSON it's obvious and easy.
I want to love XML schemas, but to call them mature is a stretch. There's a bunch of important features hidden behind XSD 1.1, which is almost 10 years old by now, yet most software that validates XML on schemas does not support that yet.
The XML Schema standard is terrible, and its mistakes and weaknesses were what drove James Clark and Murata Makoto to develop RELAX NG (REgular LAnguage for XML Next Generation).
https://en.wikipedia.org/wiki/RELAX_NG
RELAX NG has a clearly focused sound mathematical underpinning: regular expressions applied to trees, while XML Schema is an ad-hoc hot mess designed by committee.
https://en.wikipedia.org/wiki/XML_schema#RELAX_NG
Any decent JSON schema language should take its cues and learn from the design of RELAX NG, and not repeat the mistakes of XML Schema.
Makoto Murata raised some critical points about XML Schema, which was beyond repair, so they both invented new regexp-based schema languages to address those problems, and combined their respective work into Relax/NG:
https://en.wikipedia.org/wiki/Makoto_Murata#RELAX_and_RELAX_...
>Some people, including Murata and James Clark, had critical attitudes toward XML Schema. XML Schema is a modern XML schema language designed by W3C XML Schema Working Group. W3C intended XML Schema to supersede traditional DTD (Document Type Definition). XML Schema supports so many features that its specification is large and complex. Murata, James Clark and those who criticised XML Schema, pointed out the following:
>It is difficult to implement all features of XML Schema.
>It is difficult for engineers to read and write XML Schema definitions.
>It does not permit nondeterministic content models.
James Clark compared Relax NG to XML Schema and its predecessor, SGML Document Type Definitions, in his paper, "The Design of RELAX NG":
https://relaxng.org/jclark/design.html
James Clark has a huge amount of experience designing and implementing SGML and XML standards:
https://en.wikipedia.org/wiki/James_Clark_(programmer)
I've written about XML Schema, Relax NG, and James Clark eariler:
https://news.ycombinator.com/item?id=26122033
>Some of the most incredibly awfully bad XML DSLs are official standards, themselves. COUGH XMLSchema COUGH: [...]
https://news.ycombinator.com/item?id=22756875
>James Clark used Haskell to design and implement an algorithm for validating Relax NG XML schemas (he co-designed Relax NG, and designed its predecessor TREX), to work the ideas out before re-implementing it in (many many more lines of tedious brittle) Java (JING). Haskel works wonderfully as a design and standard definition language, that way. [...]
A Triumph of Simplicity: James Clark on Markup Languages and XML
https://www.drdobbs.com/a-triumph-of-simplicity-james-clark-...
>If you peek under the hood of high-profile open-source projects such as Mozilla, Apache, Perl, and Python, you'll find a little program called "expat" handling the XML parsing. If you've ever used the man command on your GNU/Linux distribution, then you've also used groff, the GNU version of the UNIX text formatting application, troff. If you've ever done any work with SGML, from generating documentation from DocBook to building your own SGML applications, you've undoubtedly come across sgmls, SP, and Jade.
This article "Schema Wars: XML Schema vs. Relax NG (1/2) - exploring XML" enumerates some of James Clark's objections to XML Schema:
https://web.archive.org/web/20190403171512/http://webreferen...
>James Clark, leader of the technical committee at OASIS for RELAX NG, and author of one of the first XML parsers, recently described the problems of the XML Schema language in a newsgroup posting:
>1) XML Schema definitions require considerable expertise to understand and can contain quite a few surprises.
>As an example, if you derive a complex type by restriction you have to specify the new restricted content model explicitly. However, attributes are treated in the opposite way: by default you get all the attributes and you have to explicitly rule out the ones you don't want. A similar inconsistency exists in that if you merge two attribute definitions you get the union of the concrete attribute definitions, but the intersection of attribute wildcards (specified by the anyAttribute element). While these might be convenient choices for the specification, it easily creates confusion for the human reader of XML schemas.
>2) The XML Schema Recommendation is hard to read and understand.
>To avoid the possible misinterpretations mentioned above you might have to reference the specification in order to fully understand a specific schema definition. I have to agree with Clark that the W3C's XML Schema Recommendation is by far the hardest to read and understand, making it even more difficult to make sense of a particular schema presented to you.
>3) W3C XML Schema's support for attributes provides no advance over DTDs.
>As with DTDs, W3C XML Schema only allows the specification of whether attributes are required or optional. There is no way to specify more complex constraints between attributes or between attributes or elements, for instance that either attribute X or attribute Y is allowed or that either attribute X or element Y is allowed. The mechanism that is used to constrain the co-occurrence of child elements should be extended to attributes and the combinations of attributes and child elements.
>4) W3C XML Schema provides very weak support for unordered content.
>When the designer of an XML vocabulary does not wish to force child elements to occur in a particular order, it can be impractical to describe the XML vocabulary using XML Schema, because XML Schema imposes such limitations.
>5) Datatype handling in W3C XML Schema lacks modularity.
>W3C XML Schema is tied to the single collection of datatypes defined in Part 2 of W3C XML Schema. Yet this collection of datatypes is a very ad-hoc collection. It includes datatypes of highly debatable relevance (gYearMonth, gDay etc). Yet it lacks many datatypes that are important for many applications. A modular approach where a schema language can be combined with one or more standard collections of datatypes, some general-purpose and some domain-specific, is called for here.
>6) W3C XML Schema does not define a single notion of validity of a document with respect to a schema.
>There are different varieties of validation (lax and strict) and many different ways to validate a document against a schema. From a W3C XML Schema alone, it is not possible to know what is a valid document. For instance there is no way to specify what is allowed as the root element.
>7) Magic schema attributes in documents
>W3C XML Schema provides the xsi:schemaLocation attribute, which allows an XML document instance to indicate the schema that should be used to validate the document. This creates problems with security (the destination might have changed or tampered with), interoperability (use of schemaLocation is optional) and "purity" of schema definition: There is no way to prevent the document containing magic xsi:* attributes, so the use of W3C XML Schema "infects" the grammar you are defining.
>8) Another problematic area in W3C XML Schema is the support for infoset augmentation, such as default attributes.
>Apart from being a violation of modularity, this tends to cause interoperability problems, because it leads to the possibility of the application getting different information depending on whether or not validation has been performed.
https://en.wikipedia.org/wiki/RELAX_NG
RELAX NG has a clearly focused sound mathematical underpinning: regular expressions applied to trees, while XML Schema is an ad-hoc hot mess designed by committee.
https://en.wikipedia.org/wiki/XML_schema#RELAX_NG
Any decent JSON schema language should take its cues and learn from the design of RELAX NG, and not repeat the mistakes of XML Schema.
Makoto Murata raised some critical points about XML Schema, which was beyond repair, so they both invented new regexp-based schema languages to address those problems, and combined their respective work into Relax/NG:
https://en.wikipedia.org/wiki/Makoto_Murata#RELAX_and_RELAX_...
>Some people, including Murata and James Clark, had critical attitudes toward XML Schema. XML Schema is a modern XML schema language designed by W3C XML Schema Working Group. W3C intended XML Schema to supersede traditional DTD (Document Type Definition). XML Schema supports so many features that its specification is large and complex. Murata, James Clark and those who criticised XML Schema, pointed out the following:
>It is difficult to implement all features of XML Schema.
>It is difficult for engineers to read and write XML Schema definitions.
>It does not permit nondeterministic content models.
James Clark compared Relax NG to XML Schema and its predecessor, SGML Document Type Definitions, in his paper, "The Design of RELAX NG":
https://relaxng.org/jclark/design.html
James Clark has a huge amount of experience designing and implementing SGML and XML standards:
https://en.wikipedia.org/wiki/James_Clark_(programmer)
I've written about XML Schema, Relax NG, and James Clark eariler:
https://news.ycombinator.com/item?id=26122033
>Some of the most incredibly awfully bad XML DSLs are official standards, themselves. COUGH XMLSchema COUGH: [...]
https://news.ycombinator.com/item?id=22756875
>James Clark used Haskell to design and implement an algorithm for validating Relax NG XML schemas (he co-designed Relax NG, and designed its predecessor TREX), to work the ideas out before re-implementing it in (many many more lines of tedious brittle) Java (JING). Haskel works wonderfully as a design and standard definition language, that way. [...]
A Triumph of Simplicity: James Clark on Markup Languages and XML
https://www.drdobbs.com/a-triumph-of-simplicity-james-clark-...
>If you peek under the hood of high-profile open-source projects such as Mozilla, Apache, Perl, and Python, you'll find a little program called "expat" handling the XML parsing. If you've ever used the man command on your GNU/Linux distribution, then you've also used groff, the GNU version of the UNIX text formatting application, troff. If you've ever done any work with SGML, from generating documentation from DocBook to building your own SGML applications, you've undoubtedly come across sgmls, SP, and Jade.
This article "Schema Wars: XML Schema vs. Relax NG (1/2) - exploring XML" enumerates some of James Clark's objections to XML Schema:
https://web.archive.org/web/20190403171512/http://webreferen...
>James Clark, leader of the technical committee at OASIS for RELAX NG, and author of one of the first XML parsers, recently described the problems of the XML Schema language in a newsgroup posting:
>1) XML Schema definitions require considerable expertise to understand and can contain quite a few surprises.
>As an example, if you derive a complex type by restriction you have to specify the new restricted content model explicitly. However, attributes are treated in the opposite way: by default you get all the attributes and you have to explicitly rule out the ones you don't want. A similar inconsistency exists in that if you merge two attribute definitions you get the union of the concrete attribute definitions, but the intersection of attribute wildcards (specified by the anyAttribute element). While these might be convenient choices for the specification, it easily creates confusion for the human reader of XML schemas.
>2) The XML Schema Recommendation is hard to read and understand.
>To avoid the possible misinterpretations mentioned above you might have to reference the specification in order to fully understand a specific schema definition. I have to agree with Clark that the W3C's XML Schema Recommendation is by far the hardest to read and understand, making it even more difficult to make sense of a particular schema presented to you.
>3) W3C XML Schema's support for attributes provides no advance over DTDs.
>As with DTDs, W3C XML Schema only allows the specification of whether attributes are required or optional. There is no way to specify more complex constraints between attributes or between attributes or elements, for instance that either attribute X or attribute Y is allowed or that either attribute X or element Y is allowed. The mechanism that is used to constrain the co-occurrence of child elements should be extended to attributes and the combinations of attributes and child elements.
>4) W3C XML Schema provides very weak support for unordered content.
>When the designer of an XML vocabulary does not wish to force child elements to occur in a particular order, it can be impractical to describe the XML vocabulary using XML Schema, because XML Schema imposes such limitations.
>5) Datatype handling in W3C XML Schema lacks modularity.
>W3C XML Schema is tied to the single collection of datatypes defined in Part 2 of W3C XML Schema. Yet this collection of datatypes is a very ad-hoc collection. It includes datatypes of highly debatable relevance (gYearMonth, gDay etc). Yet it lacks many datatypes that are important for many applications. A modular approach where a schema language can be combined with one or more standard collections of datatypes, some general-purpose and some domain-specific, is called for here.
>6) W3C XML Schema does not define a single notion of validity of a document with respect to a schema.
>There are different varieties of validation (lax and strict) and many different ways to validate a document against a schema. From a W3C XML Schema alone, it is not possible to know what is a valid document. For instance there is no way to specify what is allowed as the root element.
>7) Magic schema attributes in documents
>W3C XML Schema provides the xsi:schemaLocation attribute, which allows an XML document instance to indicate the schema that should be used to validate the document. This creates problems with security (the destination might have changed or tampered with), interoperability (use of schemaLocation is optional) and "purity" of schema definition: There is no way to prevent the document containing magic xsi:* attributes, so the use of W3C XML Schema "infects" the grammar you are defining.
>8) Another problematic area in W3C XML Schema is the support for infoset augmentation, such as default attributes.
>Apart from being a violation of modularity, this tends to cause interoperability problems, because it leads to the possibility of the application getting different information depending on whether or not validation has been performed.
I know this is a stupid objection, but I suspect one of the main impedances to takeup is going to be its stupid name. It's trivial I know, bikeshedding I know, but come on, "RELAX NG"?
Like, how do you pronounce that? I want to say 'Relax-ung', but it sounds odd, like you're trying to say "relaxing" but failing; or maybe 'Relax Nig', but that sounds like something you'd see in an HN thread on Timnit Gebru's firing. It doesn't feel like the naming was considered from a normal human point of view, as opposed to from the golden mountain of abstraction.
Like, how do you pronounce that? I want to say 'Relax-ung', but it sounds odd, like you're trying to say "relaxing" but failing; or maybe 'Relax Nig', but that sounds like something you'd see in an HN thread on Timnit Gebru's firing. It doesn't feel like the naming was considered from a normal human point of view, as opposed to from the golden mountain of abstraction.
RELAX NG was a combination of Makoto Murata's work on RELAX, an acronym for "REgular LAnguage description for Xml", and James Clark's work on "TREX", an acronym for "Tree Regular EXpressions".
They could have pulled a slick smooth move and called it "TREX-LAX", for "Tree Regular EXpressions for LAnguages of Xml", which rolls off the tongue more mellifluously!
TREX-LAX: It makes your XML nice and regular, and keeps you shit moving smoothly.
https://www.browncafe.com/community/threads/fedex-and-ex-lax...
>FedEx and Ex-Lax battle for rights to promotional slogan
They could have pulled a slick smooth move and called it "TREX-LAX", for "Tree Regular EXpressions for LAnguages of Xml", which rolls off the tongue more mellifluously!
TREX-LAX: It makes your XML nice and regular, and keeps you shit moving smoothly.
https://www.browncafe.com/community/threads/fedex-and-ex-lax...
>FedEx and Ex-Lax battle for rights to promotional slogan
Interesting. I have validation of our internal (overly) complex configuration files on my todo list, and I'll have a look at using that, thanks!
Another nice thing about RELAX NG is that not only does it have an XML format, but it also has a nice clean, much simpler to read and write, "compact syntax", which is equivalent to the XML format, and can be translated back and forth without losing any information.
It's not all goofy and arbitrary and complex line noise like XML DTDs, and it's not all verbose and clumsy namespaced XML tags and attributes like XML Scheme documents.
It even supports inheritance, and grammar as well as element level annotations (useful for schema and meta processing tools and editors).
RELAX NG's Compact Syntax:
https://www.xml.com/pub/a/2002/06/19/rng-compact.html
RELAX NG Compact Syntax: Committee Specification 21 November 2002:
https://www.oasis-open.org/committees/relax-ng/compact-20021...
RELAX NG Compact Syntax Tutorial:
https://relaxng.org/compact-tutorial-20030326.html
It's not all goofy and arbitrary and complex line noise like XML DTDs, and it's not all verbose and clumsy namespaced XML tags and attributes like XML Scheme documents.
It even supports inheritance, and grammar as well as element level annotations (useful for schema and meta processing tools and editors).
RELAX NG's Compact Syntax:
https://www.xml.com/pub/a/2002/06/19/rng-compact.html
RELAX NG Compact Syntax: Committee Specification 21 November 2002:
https://www.oasis-open.org/committees/relax-ng/compact-20021...
RELAX NG Compact Syntax Tutorial:
https://relaxng.org/compact-tutorial-20030326.html
James Clark argues that it's useful to have multiple specialized schema definition languages, each for their own purpose, so putting one reference to one "official" schema in one "official" language inside the document doesn't make sense.
For example, you might want to perform one kind of quick simple input sanity checking validation that ensures the input won't crash your processor, to perform in real time on millions of XML documents you receive every minute.
Then you might also want another more elaborate schema with an anal retentive formal specification, including text syntax checking, data type validation, cross-reference checking, constraints between values, and documentation, for creating and editing the same document in a schema-driven editor.
https://github.com/michmech/xonomy
It's also a bad idea to interpret the URI of a schema as the URL from which you can download the schema definition. That introduce security and privacy and interoperability problems.
Internet Explorer used to try to download the schema definition every time you opened an XML document, which lets whoever controls that web site know you're looking at one of their documents, and what its url is in the referer field. It's like XML's answer to the 1x1 transparent GIF tracking cookie!
Unlike DTDs and XML Schema, RELAX NG doesn't presume that it's the only schema language in the world that you will ever need to use: there are other useful languages like Schematron, which do things that RELEX NG was designed not to do on purpose.
https://www.schematron.com/
(There's a great Ven diagram comparing XSD, DTD, RELAX NG and Schematron's scope on that page.)
https://www.schematron.com/document/2755.html
>Schematron can be useful in conjunction with many grammar-based structure-validation languages: DTDs, XML Schemas, RELAX, TREX, etc. Indeed, Schematron is part of an ISO standard (DSDL: Document Schema Description Languages) designed to allow multiple, well-focussed XML validation languages to work together. You can even embed a Schematron schema inside an XML Schema <appinfo> element or inside a RELAX NG schema!
James Clark on RELAX NG and xml:id:
https://blog.jclark.com/2009/01/relax-ng-and-xmlid.html
>RELAX NG and xml:id
>One part of the vision underlying RELAX NG is that validation should not be monolithic: it is not necessary or desirable to have one schema language that can handle every possible kind of validation you might want to do; it is better instead to have multiple specialized languages, each of which does one kind validation, really well. Consistent with this vision, RELAX NG provides only grammar-based validation. There's no implicit claim that other kinds of validation aren't useful and important.
>One kind of validation that is clearly useful and important and that can't be done by grammars is checking of cross-references. One possibility is to use Schematron for this. The designers of RELAX NG anticipated that there would be a little schema language specialized to this, which would be created as part of the ISO DSDL effort (as part 6); this wouldn't be a million miles from the kind of thing that XSD provides with xs:key/xs:unique/xs:keyref. Unfortunately this hasn't happened yet.
[...]
For example, you might want to perform one kind of quick simple input sanity checking validation that ensures the input won't crash your processor, to perform in real time on millions of XML documents you receive every minute.
Then you might also want another more elaborate schema with an anal retentive formal specification, including text syntax checking, data type validation, cross-reference checking, constraints between values, and documentation, for creating and editing the same document in a schema-driven editor.
https://github.com/michmech/xonomy
It's also a bad idea to interpret the URI of a schema as the URL from which you can download the schema definition. That introduce security and privacy and interoperability problems.
Internet Explorer used to try to download the schema definition every time you opened an XML document, which lets whoever controls that web site know you're looking at one of their documents, and what its url is in the referer field. It's like XML's answer to the 1x1 transparent GIF tracking cookie!
Unlike DTDs and XML Schema, RELAX NG doesn't presume that it's the only schema language in the world that you will ever need to use: there are other useful languages like Schematron, which do things that RELEX NG was designed not to do on purpose.
https://www.schematron.com/
(There's a great Ven diagram comparing XSD, DTD, RELAX NG and Schematron's scope on that page.)
https://www.schematron.com/document/2755.html
>Schematron can be useful in conjunction with many grammar-based structure-validation languages: DTDs, XML Schemas, RELAX, TREX, etc. Indeed, Schematron is part of an ISO standard (DSDL: Document Schema Description Languages) designed to allow multiple, well-focussed XML validation languages to work together. You can even embed a Schematron schema inside an XML Schema <appinfo> element or inside a RELAX NG schema!
James Clark on RELAX NG and xml:id:
https://blog.jclark.com/2009/01/relax-ng-and-xmlid.html
>RELAX NG and xml:id
>One part of the vision underlying RELAX NG is that validation should not be monolithic: it is not necessary or desirable to have one schema language that can handle every possible kind of validation you might want to do; it is better instead to have multiple specialized languages, each of which does one kind validation, really well. Consistent with this vision, RELAX NG provides only grammar-based validation. There's no implicit claim that other kinds of validation aren't useful and important.
>One kind of validation that is clearly useful and important and that can't be done by grammars is checking of cross-references. One possibility is to use Schematron for this. The designers of RELAX NG anticipated that there would be a little schema language specialized to this, which would be created as part of the ISO DSDL effort (as part 6); this wouldn't be a million miles from the kind of thing that XSD provides with xs:key/xs:unique/xs:keyref. Unfortunately this hasn't happened yet.
[...]
For many use cases, the way I'd do this these days is just to adopt Avro or Flatbuffers, or some other such format, and use it in JSON mode.
One shouldn't have to think about which features of a deserializer to enable or disable in order to avoid security issues.
XML is a terrible data interchange format because it embeds its schema in its data. That makes it sometimes an order of magnitude larger, and it takes longer to parse.
If you want to drop JSON altogether, something like Protobufs would be better. But JSON + JSONSchema has all the benefits of XML and none of the drawbacks.
If you want to drop JSON altogether, something like Protobufs would be better. But JSON + JSONSchema has all the benefits of XML and none of the drawbacks.
XML DTDs were inherited from SGML. They can also be defined externally, but with internal links to them embedded in the document.
XML Schema were supposed to fix all the problems of DTDs, but they don't support unordered content, and have a host of their own problems. XML Schema are typically external XML documents referenced by an xsi:schemaLocation attribute, but apparently you can embed them too. But I don't know if that's a common practice, since it's a pretty bad idea for a lot of reasons:
https://www.herongyang.com/XML/XSD-Statements-Embedded-in-XM...
You raise a good point that it's a bad idea to embed schemas in documents, or even references to schemas in documents, because, as James Clark points out, magic schema attributes in documents (like xsi:schemaLocation) pre-suppose that there can be only one way of validating a document, they create security and interoperability problems, and it infects documents with the namespace of the grammar that you're using to validate them, when documents shouldn't depended on such knowledge, and be forced to include hot messes of namespace attributes like:
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:aop="http://www.springframework.org/schema/aop" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0... http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.0.xsd">
Schema Wars: XML Schema vs. RELAX NG (1/2) - exploring XML:
https://web.archive.org/web/20190403171512/http://webreferen...
>James Clark, leader of the technical committee at OASIS for RELAX NG, and author of one of the first XML parsers, recently described the problems of the XML Schema language in a newsgroup posting: [...]
>7) Magic schema attributes in documents
>W3C XML Schema provides the xsi:schemaLocation attribute, which allows an XML document instance to indicate the schema that should be used to validate the document. This creates problems with security (the destination might have changed or tampered with), interoperability (use of schemaLocation is optional) and "purity" of schema definition: There is no way to prevent the document containing magic xsi:* attributes, so the use of W3C XML Schema "infects" the grammar you are defining.
Here's more of what he has to say about JSON in his blog post "XML vs the Web":
https://blog.jclark.com/2010/11/xml-vs-web_24.html
>[...] From this perspective, my reaction to JSON is a combination of "Yay" and "Sigh".
>It's "Yay", because for important use cases JSON is dramatically better than XML. In particular, JSON shines as a programming language-independent representation of typical programming language data structures. This is an incredibly important use case and it would be hard to overstate how appallingly bad XML is for this. The fundamental problem is the mismatch between programming language data structures and the XML element/attribute data model of elements. This leaves the developer with three choices, all unappetising:
>- live with an inconvenient element/attribute representation of the data;
>- descend into XML Schema hell in the company of your favourite data binding tool;
>- write reams of code to convert the XML into a convenient data structure.
>By contrast with JSON, especially with a dynamic programming language, you can get a reasonable in-memory representation just by calling a library function. [...]
>There's a bigger point that I want to make here, and it's about the relationship between XML and the Web. When we started out doing XML, a big part of the vision was about bridging the gap from the SGML world (complex, sophisticated, partly academic, partly big enterprise) to the Web, about making the value that we saw in SGML accessible to a broader audience by cutting out all the cruft. In the beginning XML did succeed in this respect. But this vision seems to have been lost sight of over time to the point where there's a gulf between the XML community and the broader Web developer community; all the stuff that's been piled on top of XML, together with the huge advances in the Web world in HTML5, JSON and JavaScript, have combined to make XML be perceived as an overly complex, enterprisey technology, which doesn't bring any value to the average Web developer.
>This is not a good thing for either community (and it's why part of my reaction to JSON is "Sigh"). XML misses out by not having the innovation, enthusiasm and traction that the Web developer community brings with it, and the Web developer community misses out by not being able to take advantage of the powerful and convenient technologies that have been built on top of XML over the last decade.
>So what's the way forward? I think the Web community has spoken, and it's clear that what it wants is HTML5, JavaScript and JSON. XML isn't going away but I see it being less and less a Web technology; it won't be something that you send over the wire on the public Web, but just one of many technologies that are used on the server to manage and generate what you do send over the wire.
>In the short-term, I think the challenge is how to make HTML5 play more nicely with XML. In the longer term, I think the challenge is how to use our collective experience from building the XML stack to create technologies that work natively with HTML, JSON and JavaScript, and that bring to the broader Web developer community some of the good aspects of the modern XML development experience.
XML Schema were supposed to fix all the problems of DTDs, but they don't support unordered content, and have a host of their own problems. XML Schema are typically external XML documents referenced by an xsi:schemaLocation attribute, but apparently you can embed them too. But I don't know if that's a common practice, since it's a pretty bad idea for a lot of reasons:
https://www.herongyang.com/XML/XSD-Statements-Embedded-in-XM...
You raise a good point that it's a bad idea to embed schemas in documents, or even references to schemas in documents, because, as James Clark points out, magic schema attributes in documents (like xsi:schemaLocation) pre-suppose that there can be only one way of validating a document, they create security and interoperability problems, and it infects documents with the namespace of the grammar that you're using to validate them, when documents shouldn't depended on such knowledge, and be forced to include hot messes of namespace attributes like:
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:aop="http://www.springframework.org/schema/aop" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0... http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop-2.0.xsd">
Schema Wars: XML Schema vs. RELAX NG (1/2) - exploring XML:
https://web.archive.org/web/20190403171512/http://webreferen...
>James Clark, leader of the technical committee at OASIS for RELAX NG, and author of one of the first XML parsers, recently described the problems of the XML Schema language in a newsgroup posting: [...]
>7) Magic schema attributes in documents
>W3C XML Schema provides the xsi:schemaLocation attribute, which allows an XML document instance to indicate the schema that should be used to validate the document. This creates problems with security (the destination might have changed or tampered with), interoperability (use of schemaLocation is optional) and "purity" of schema definition: There is no way to prevent the document containing magic xsi:* attributes, so the use of W3C XML Schema "infects" the grammar you are defining.
Here's more of what he has to say about JSON in his blog post "XML vs the Web":
https://blog.jclark.com/2010/11/xml-vs-web_24.html
>[...] From this perspective, my reaction to JSON is a combination of "Yay" and "Sigh".
>It's "Yay", because for important use cases JSON is dramatically better than XML. In particular, JSON shines as a programming language-independent representation of typical programming language data structures. This is an incredibly important use case and it would be hard to overstate how appallingly bad XML is for this. The fundamental problem is the mismatch between programming language data structures and the XML element/attribute data model of elements. This leaves the developer with three choices, all unappetising:
>- live with an inconvenient element/attribute representation of the data;
>- descend into XML Schema hell in the company of your favourite data binding tool;
>- write reams of code to convert the XML into a convenient data structure.
>By contrast with JSON, especially with a dynamic programming language, you can get a reasonable in-memory representation just by calling a library function. [...]
>There's a bigger point that I want to make here, and it's about the relationship between XML and the Web. When we started out doing XML, a big part of the vision was about bridging the gap from the SGML world (complex, sophisticated, partly academic, partly big enterprise) to the Web, about making the value that we saw in SGML accessible to a broader audience by cutting out all the cruft. In the beginning XML did succeed in this respect. But this vision seems to have been lost sight of over time to the point where there's a gulf between the XML community and the broader Web developer community; all the stuff that's been piled on top of XML, together with the huge advances in the Web world in HTML5, JSON and JavaScript, have combined to make XML be perceived as an overly complex, enterprisey technology, which doesn't bring any value to the average Web developer.
>This is not a good thing for either community (and it's why part of my reaction to JSON is "Sigh"). XML misses out by not having the innovation, enthusiasm and traction that the Web developer community brings with it, and the Web developer community misses out by not being able to take advantage of the powerful and convenient technologies that have been built on top of XML over the last decade.
>So what's the way forward? I think the Web community has spoken, and it's clear that what it wants is HTML5, JavaScript and JSON. XML isn't going away but I see it being less and less a Web technology; it won't be something that you send over the wire on the public Web, but just one of many technologies that are used on the server to manage and generate what you do send over the wire.
>In the short-term, I think the challenge is how to make HTML5 play more nicely with XML. In the longer term, I think the challenge is how to use our collective experience from building the XML stack to create technologies that work natively with HTML, JSON and JavaScript, and that bring to the broader Web developer community some of the good aspects of the modern XML development experience.
Is there an official standard body behind this (and json 6 mentioned in another comment), or is it a kind of loose / informal standard left up to libraries to implement?
I mean comments and trailing commas will make JSON a bit more human friendly when used in e.g. configuration (package.json), but as a pure data exchange format it doesn't add much. Mind you, there's better data exchange formats than json, like XML (yes I said it) which can be transferred via a binary protocol (https://www.w3.org/TR/exi/) instead of the, when you think about it, awkward and inefficient data -> text format -> compression -> transfer -> decompress -> text format -> parse cycle of both 'traditional' XML and JSON both.
I wouldn't be averse to the idea of browser manufacturers or standards bodies to come up with a more efficient data transfer method that can be human-read with the right tooling like browser development tools, but which is fully compressed binary and efficient on the line. I feel like tools like GRPC are a bit too far out there.
I mean comments and trailing commas will make JSON a bit more human friendly when used in e.g. configuration (package.json), but as a pure data exchange format it doesn't add much. Mind you, there's better data exchange formats than json, like XML (yes I said it) which can be transferred via a binary protocol (https://www.w3.org/TR/exi/) instead of the, when you think about it, awkward and inefficient data -> text format -> compression -> transfer -> decompress -> text format -> parse cycle of both 'traditional' XML and JSON both.
I wouldn't be averse to the idea of browser manufacturers or standards bodies to come up with a more efficient data transfer method that can be human-read with the right tooling like browser development tools, but which is fully compressed binary and efficient on the line. I feel like tools like GRPC are a bit too far out there.
Msgpack and protobuf do that too.
I believe the rise of JSON is attributed to that it's easily human-readable, and because Javascript.
Not sure human-readability was a major consideration of JSON really, but maybe.
One of the major considerations that many seem to miss today though was simplicity of parsing: it's a much more strict syntax than actual JS so there's a lot less ambiguity in parsing it. E.g. there's no need to have layered logic for recognizing unquoted identifiers, no need to check for multiple quote types on IDs & strings, consistent comma-ing etc. In general giving the author less options will make your parser simpler.
This is notably the exact opposite of YAML's approach.
JSON5 is nowhere near as bad as YAML, but it does seem to leaving behind some of the raison d'être of JSON. Like, I hate double-quoting in JS/JSON (personal bike-shed colour preference), but surely variable quoting characters is the worst of both worlds.
One of the major considerations that many seem to miss today though was simplicity of parsing: it's a much more strict syntax than actual JS so there's a lot less ambiguity in parsing it. E.g. there's no need to have layered logic for recognizing unquoted identifiers, no need to check for multiple quote types on IDs & strings, consistent comma-ing etc. In general giving the author less options will make your parser simpler.
This is notably the exact opposite of YAML's approach.
JSON5 is nowhere near as bad as YAML, but it does seem to leaving behind some of the raison d'être of JSON. Like, I hate double-quoting in JS/JSON (personal bike-shed colour preference), but surely variable quoting characters is the worst of both worlds.
Agreed. Every single feature here complicates parsing, and the only two that are worth it IMO are comments and multi-line strings (though I'm sure others disagree on which ones are most important, which is how we end up with something like this)
Comments maybe. Multi-line strings though I don't really see as very valuable.
Lack of multi-line strings in JS was a very large, but infrequent pain for years before it was added. That infrequency tends toward zero for JSON: most of it is generated, and even when it is hand-authored, long strings are very much an exception in those cases (in my experience).
Lack of multi-line strings in JS was a very large, but infrequent pain for years before it was added. That infrequency tends toward zero for JSON: most of it is generated, and even when it is hand-authored, long strings are very much an exception in those cases (in my experience).
Two use-cases come to mind:
a) Short series' of shell commands in package.json
b) Longform content
I disagree that JSON is rarely hand-authored. You could argue it isn't the best medium for these usecases, but it definitely gets used for them either way
a) Short series' of shell commands in package.json
b) Longform content
I disagree that JSON is rarely hand-authored. You could argue it isn't the best medium for these usecases, but it definitely gets used for them either way
JSON is often hand-authored as small config formats (in which your use-case (a) is relevant*), but I have never seen it hand-authored for readable "content".
Firstly longform content is often rich-text; if that's hand-authored in something that's not a wysiwyg UI, it's typically a dedicated separate file (.html or .md) which may-or-may not be later programmatically compiled to JSON.
For longform content that's not rich-text, and doesn't warrant a separate dedicated .txt file: how long is it? Do you really need multiline strings?
The only real use-case I've seen is i18n string map files, where some translations can contain a newline or two, but these are an exception; the majority of strings in those files are short.
----
* On use-case (a), this is a personal preference but I tend to prefer referencing single independent executables in these use-cases, to the modern trend of having long unreadable bash one-liners in config files.
Firstly longform content is often rich-text; if that's hand-authored in something that's not a wysiwyg UI, it's typically a dedicated separate file (.html or .md) which may-or-may not be later programmatically compiled to JSON.
For longform content that's not rich-text, and doesn't warrant a separate dedicated .txt file: how long is it? Do you really need multiline strings?
The only real use-case I've seen is i18n string map files, where some translations can contain a newline or two, but these are an exception; the majority of strings in those files are short.
----
* On use-case (a), this is a personal preference but I tend to prefer referencing single independent executables in these use-cases, to the modern trend of having long unreadable bash one-liners in config files.
> Is there an official standard body behind this (and json 6 mentioned in another comment), or is it a kind of loose / informal standard left up to libraries to implement?
There's a spec: https://spec.json5.org/
There's a spec: https://spec.json5.org/
Previously on HN:
The JSON5 Data Interchange Format (https://spec.json5.org/) - 2 Years ago - 0 Comments - https://news.ycombinator.com/item?id=21113613
The JSON5 Data Interchange Format (https://spec.json5.org/) - 2 Years ago - 0 Comments - https://news.ycombinator.com/item?id=21113613
Killer feature: comments.
Sure, for APIs comments are bad. However, for configuration things are so much better when you can document or temporarily remove things, without having hacks which pollute the data.
Trailing commas are also nice to have - although we have linters for that, unfortunately some people think they're too good to need a linter.
Sure, for APIs comments are bad. However, for configuration things are so much better when you can document or temporarily remove things, without having hacks which pollute the data.
Trailing commas are also nice to have - although we have linters for that, unfortunately some people think they're too good to need a linter.
> Trailing commas are also nice to have - although we have linters for that, unfortunately some people think they're too good to need a linter.
IMO the real value in trailing commas is that it's easier to produce simple JSON programmatically.
IMO the real value in trailing commas is that it's easier to produce simple JSON programmatically.
Past related threads:
JSON5 – a superset of JSON with comments, multiline string, trailing comma, etc. - https://news.ycombinator.com/item?id=22598401 - March 2020 (1 comment)
JSON5: JSON for humans - https://news.ycombinator.com/item?id=17375850 - June 2018 (3 comments)
JSON5 – JSON for the ES5 Era - https://news.ycombinator.com/item?id=10956943 - Jan 2016 (8 comments)
JSON5 is a proposed extension to JSON - https://news.ycombinator.com/item?id=7325735 - March 2014 (246 comments)
Show HN: JSON5 — modern JSON - https://news.ycombinator.com/item?id=4031699 - May 2012 (84 comments)
JSON5 – a superset of JSON with comments, multiline string, trailing comma, etc. - https://news.ycombinator.com/item?id=22598401 - March 2020 (1 comment)
JSON5: JSON for humans - https://news.ycombinator.com/item?id=17375850 - June 2018 (3 comments)
JSON5 – JSON for the ES5 Era - https://news.ycombinator.com/item?id=10956943 - Jan 2016 (8 comments)
JSON5 is a proposed extension to JSON - https://news.ycombinator.com/item?id=7325735 - March 2014 (246 comments)
Show HN: JSON5 — modern JSON - https://news.ycombinator.com/item?id=4031699 - May 2012 (84 comments)
json-ld files (json line delimited, means one json object per line) are used extensively in data processing.
A lot of processing of these json-ld files uses normal builtins instead of special libraries eg in python doing a for line in or a stream map and the standard json lib etc.
From a glance at this spec, it seems that where an object ends is no longer something that the normal file processing most languages possess work. Which is a problem.
I’d be surprised if a extended-json ever got traction because everyone already has code that uses the lowest common denominator json parser and line splitting built into whatever language and libraries they are using and those they interchange with already use.
Having said that, the pain points I wrestle with are lack of date types and how to safely pass large integers and currencies etc. Things not addressed in the json5 spec?
A lot of processing of these json-ld files uses normal builtins instead of special libraries eg in python doing a for line in or a stream map and the standard json lib etc.
From a glance at this spec, it seems that where an object ends is no longer something that the normal file processing most languages possess work. Which is a problem.
I’d be surprised if a extended-json ever got traction because everyone already has code that uses the lowest common denominator json parser and line splitting built into whatever language and libraries they are using and those they interchange with already use.
Having said that, the pain points I wrestle with are lack of date types and how to safely pass large integers and currencies etc. Things not addressed in the json5 spec?
JSON-LD is JSON Linked Data.
What you describe seems to be called line-delimited JSON, LDJSON, NDJSON or JSONL.
https://en.wikipedia.org/wiki/JSON_streaming
What you describe seems to be called line-delimited JSON, LDJSON, NDJSON or JSONL.
https://en.wikipedia.org/wiki/JSON_streaming
I call it JSONL but yeah it's super useful. The ability to split and join using standard Unix tools, or just pipe through GNU parallel for automatic speedup without any boilerplate code
A JSON5 parser comes in handy for scraping, when you want to extract data that is embedded directly inside the javascript source of a website.
But I wouldn't ever use JSON5 over JSON for serialization, or over YAML for parsing a configuration file.
But I wouldn't ever use JSON5 over JSON for serialization, or over YAML for parsing a configuration file.
These days, JS has moved on while JSON has remained stuck in the past. There are some fairly important ones it'd be great to get a standard format for in JSON. e.g.
I'd quite like a next gen JSON not to get upset on circular references too, although that would most likely require some sort of link or reference data type.
JSON.stringify({z: 1n})
I'd quite like a next gen JSON not to get upset on circular references too, although that would most likely require some sort of link or reference data type.
Bigints, dates, and buffers I can understand to a degree.
Sets can easily be serialized as a json array. Maps can be serialized as a json array of arrays or as a single json dictionary if the key types are compatible.
Sets can easily be serialized as a json array. Maps can be serialized as a json array of arrays or as a single json dictionary if the key types are compatible.
Everything can be serialised as something else with various degress of difficulty, but one of the beauties of JSON was that it had support for most of the native types of javascript. It's nice not to have to write special logic to serialise / deserialise the way you want. In particular, the ability to round trip all the standard types would be really useful.
While I think this would make JSON more fun to edit by-hand, I don’t feel the breaking changes across the stack, and mixed adoption, are worth the benefits.
I'd love if they added support for dates, e.g.
{ justDate: @2020-02-08 }
{ dateAndTime: @2007-03-01T13:00:00Z }
{ justTime: @@13:00:00 }
{ justDate: @2020-02-08 }
{ dateAndTime: @2007-03-01T13:00:00Z }
{ justTime: @@13:00:00 }
Well, that has nothing to do with "Javascript" notation
So first get that kind of notation into javascript, then wait 15 years
[deleted]
I still use the original Java JSON that I compiled from Douglas source back in ~2008 maybe.
It hasen't changed since then and it will never need to change ever in the history of mankind.
Problem solved!
Edit: Maybe this solves another problem though, but I doubt it!
It hasen't changed since then and it will never need to change ever in the history of mankind.
Problem solved!
Edit: Maybe this solves another problem though, but I doubt it!
The syntax for line breaks make it incompatible with JavaScript.
Probably not terrible because doing an eval of untrusted json would be a bad idea but they could have used back ticks which would have these useful properties:
- still JavaScript
- fewer characters
Probably not terrible because doing an eval of untrusted json would be a bad idea but they could have used back ticks which would have these useful properties:
- still JavaScript
- fewer characters
Don't we already have Yet Another Markup Language (YAML)? :D
"JSON (JavaScript Object Notation) is a lightweight data-interchange format."
So says JSON.org for as long as anyone can remember.
Why is anyone caring about using single or double quotes, or wanting trailing commas? It solves no problems. It seems to me the only answer is it's for manually created JSON. That is, all of this is for people whose only exposure to JSON is doing some config in vscode. It adds nothing else.
Call it JSON-CONFIG and leave well enough alone.
So says JSON.org for as long as anyone can remember.
Why is anyone caring about using single or double quotes, or wanting trailing commas? It solves no problems. It seems to me the only answer is it's for manually created JSON. That is, all of this is for people whose only exposure to JSON is doing some config in vscode. It adds nothing else.
Call it JSON-CONFIG and leave well enough alone.
Keep going, next sentence from JSON.org: "It is easy for humans to read and write. It is easy for machines to parse and generate. "
Non-quoted keys is easier to read and write by humans, trailing commas are easier for a machine to write, single quotes can make it easier for a human to read or write. Comments help humans read and write.
I remember when it was javascript object notation and that stuff was allowed because thats what javascript allows and felt simple and effective for both human and machine, then it became strict json, what a step backwards.
Non-quoted keys is easier to read and write by humans, trailing commas are easier for a machine to write, single quotes can make it easier for a human to read or write. Comments help humans read and write.
I remember when it was javascript object notation and that stuff was allowed because thats what javascript allows and felt simple and effective for both human and machine, then it became strict json, what a step backwards.
I like that it has a new name. I've seen too many stupid libraries support their own superset while still calling it JSON (looking at you, Python and .NET ecosystem!)
It's a standard. If we change it, it needs a new name..
However, I am worried that this new standard is harder for languages and ecosystems to support consistently, because it requires a much more involved parser.
Then again, if YAML can do it.. (and that must be a parser created in the depths of hell).
It's a standard. If we change it, it needs a new name..
However, I am worried that this new standard is harder for languages and ecosystems to support consistently, because it requires a much more involved parser.
Then again, if YAML can do it.. (and that must be a parser created in the depths of hell).
Why wouldn't they follow ECMA and use backticks for multiline strings?
Ah, it's based on ECMAScript 5.1. I guess we can wait for a JSON6, then.
Ah, it's based on ECMAScript 5.1. I guess we can wait for a JSON6, then.
If you're going to the trouble of using something not backwards compatible with json, use something that isn't crap.
[deleted]
> Strings may include character escapes
Wait, that wasn't possible before?
I like it, most of the additions seem more like filling gaps in the original version. And the lack of trailing commas & comments have always annoyed me a bit.
But as this is a library I'm guessing NPM doesn't support it for the package.json?
Wait, that wasn't possible before?
I like it, most of the additions seem more like filling gaps in the original version. And the lack of trailing commas & comments have always annoyed me a bit.
But as this is a library I'm guessing NPM doesn't support it for the package.json?
Somewhat related, I was toying around with what an html5 based csv alternative could look like and called it hsv5 partly as a play on json5 and hdf5.
https://github.com/elcritch/hsv5
https://github.com/elcritch/hsv5
IIRC there is a JSON6 format which adds features and improvements on top of JSON5.
HJSON [0] is also another format that tries to make it easier for humans to read / write json.
[0] https://hjson.github.io/
[0] https://hjson.github.io/
> Strings may span multiple lines by escaping new line characters.
What if the escape character comes before a trailing whitespace? Like this (`_` is whitespace):
What if the escape character comes before a trailing whitespace? Like this (`_` is whitespace):
"hello,\_\n
Some luckier alternate universes have standardized on EDN instead.
https://github.com/edn-format/edn
https://github.com/edn-format/edn
>> by expanding its syntax to include some productions from ECMAScript 5.1.
I think at this point a new json spec should have little/nothing to do with javascript.
I think at this point a new json spec should have little/nothing to do with javascript.
"The standard has to be sufficiently simple that it makes sense to have multiple implementations." -James Clark
I have to admit that I once used python repr() as a way to pass json data. It worked. I think.
I've heard horror stories from a friend who had to fix such code introduced by a contractor. Management didn't want to throw any of it awayz because they paid a lot of money to the contractor. Of course broke in a hard to debug fashion.
It will work only as long as the object(s) interpreted by repr are serialzable by JSON, which is not a sound approach, unless you write a any-type serializer.
These are the monsters who told us "XML is too bloated" and "who'd ever want a self-describing, self-transformable interchange format anyway?"
The more I see things like JSONSchema and JSON5 get away from Crockford's original overly simplistic syntax, the more I think we can all agree with Richard Gabriel. Worse Is Better!
The more I see things like JSONSchema and JSON5 get away from Crockford's original overly simplistic syntax, the more I think we can all agree with Richard Gabriel. Worse Is Better!
No booleans?
It's a superset of JSON, so it should natively support true and false identifiers.
what a piece of shit...
Should've named it JackSON5.
What's the point? I cringe every time a project uses both double quote and single quote, a mix of semicolon and no semicolon and a mix of tabs and space, .... Which one to pick isn't relevant as soon as the code base is consistent