JCOF: JSON-like Compact Object Format(github.com)
github.com
JCOF: JSON-like Compact Object Format
https://github.com/mortie/jcof
68 comments
Hey, JCOF author here. I didn't notice this had made it to HN before now.
* JCOF allows for streaming serialization if you want, you can just use the key-value object syntax and use inline strings everywhere. You won't get much size improvement over JSON when using a streaming approach, but you can even do a half-and-half approach; maybe you have a set of strings which you know you're using a lot, so you put those strings in the string table, then you do a streaming encode of the rest of your data, with string references to the string table. You could even do the same with object shapes, where you hard-code a couple of object shapes you know you use a lot, and use the JSON-style key-value syntax for other kinds of objects. Basically, you can choose where you want to be on the efficiency vs streaming curve.
* Nested objects aren't difficult; an object can go in any place where a value can go, just like in JSON. This object can be either a "shaped" object where the keys are listed in the object shape table, or it can be a "keyed" object where you write out the keys inline (as either literal strings or references to the string table).
* JCOF is human-writable; you can write a document which looks a whole lot like JSON. This is a valid document: `;;{"people":[{"name":"Bob",Aage":36},{"name":"Alice","age":40}]}`. But optimal JCOF is mostly unreadable. So it's less human friendly than JSON, more human friendly than binary formats; it's a trade-off.
I think JCOF hits enough interesting, unexplored points on enough relevant curves that it has some value. It also ends up being smaller than any of the binary formats I've tried like MessagePack and CBOR (even with the string references extension), while being text-based. It's obviously not going to replace JSON, but it's useful in some cases.
* JCOF allows for streaming serialization if you want, you can just use the key-value object syntax and use inline strings everywhere. You won't get much size improvement over JSON when using a streaming approach, but you can even do a half-and-half approach; maybe you have a set of strings which you know you're using a lot, so you put those strings in the string table, then you do a streaming encode of the rest of your data, with string references to the string table. You could even do the same with object shapes, where you hard-code a couple of object shapes you know you use a lot, and use the JSON-style key-value syntax for other kinds of objects. Basically, you can choose where you want to be on the efficiency vs streaming curve.
* Nested objects aren't difficult; an object can go in any place where a value can go, just like in JSON. This object can be either a "shaped" object where the keys are listed in the object shape table, or it can be a "keyed" object where you write out the keys inline (as either literal strings or references to the string table).
* JCOF is human-writable; you can write a document which looks a whole lot like JSON. This is a valid document: `;;{"people":[{"name":"Bob",Aage":36},{"name":"Alice","age":40}]}`. But optimal JCOF is mostly unreadable. So it's less human friendly than JSON, more human friendly than binary formats; it's a trade-off.
I think JCOF hits enough interesting, unexplored points on enough relevant curves that it has some value. It also ends up being smaller than any of the binary formats I've tried like MessagePack and CBOR (even with the string references extension), while being text-based. It's obviously not going to replace JSON, but it's useful in some cases.
Oh, I think you have something, but I don't think JSON and JCOF are comparable. I think it makes a very interesting replacement for formats where there is a header line like CSV, TSV. JCOF allows for some level of consistent nested structures (which would be nice for header/detail records which are really common in the real world) and would be so much nicer than many of the bespoke formats you see out there... and also, sooo much more efficient than XML, and without the issues that come with XML. Incidentally, these kinds of uses are where JSON really doesn't work well because you always have the potential of a mis-shaped object.
In short: nice work. Might be fun to roll an implementation or two :-)
In short: nice work. Might be fun to roll an implementation or two :-)
JCOF is just not radical enough.
It should stop pretending to remain human-readable, go binary, and become a variant of protocol buffers, thrift, etc.
BTW being schemaless is a boon during initial hacking things together, and an impediment when operating and developing the system further down the line. It's the data equivalent of dynamic typing vs static typing, interpreted vs compiled in code.
It should stop pretending to remain human-readable, go binary, and become a variant of protocol buffers, thrift, etc.
BTW being schemaless is a boon during initial hacking things together, and an impediment when operating and developing the system further down the line. It's the data equivalent of dynamic typing vs static typing, interpreted vs compiled in code.
> It should stop pretending to remain human-readable, go binary, and become a variant of protocol buffers, thrift, etc.
There are lots of ways standard ways to do serialization to binary formats. https://en.wikipedia.org/wiki/Comparison_of_data-serializati...
> BTW being schemaless is a boon during initial hacking things together, and an impediment when operating and developing the system further down the line. It's the data equivalent of dynamic typing vs static typing, interpreted vs compiled in code.
Sounds like XML, which is a whole other discussion :-)
There are lots of ways standard ways to do serialization to binary formats. https://en.wikipedia.org/wiki/Comparison_of_data-serializati...
> BTW being schemaless is a boon during initial hacking things together, and an impediment when operating and developing the system further down the line. It's the data equivalent of dynamic typing vs static typing, interpreted vs compiled in code.
Sounds like XML, which is a whole other discussion :-)
People keep reinventing JSON trying to beat gzip, and it never beats the simplicity of gzipping JSON...
comets.json:
JSON: 51949 bytes
JCOF: 37480 bytes (0.721x)
JSON ZIPED: 15178 bytes
Now I am curious – how does zipped JCOF turn out? I know that’s not the author’s intention, but I’m curious how it compresses
It's nearly irrelevant, given that I can gzip JSON and send it to literally anything built in the last 10+ years. It's so ubiquitous now that many web servers will just accept gzipped content without even exposing that fact to the back end servers. You can get all the benefits of highly compressed JSON and never have interacted with gzip.
If it could somehow produce a significant reduction with fewer CPU cycles, there may be a really niche use case, but I don't see those often.
If it could somehow produce a significant reduction with fewer CPU cycles, there may be a really niche use case, but I don't see those often.
Oh, it is completely irrelevant! I just want a data point for my mental model of gzip performance. Specifically, I am curious if the optimizations (object tables and separator elimination) are made irrelevant by gzip's dictionary, or if they result in better compression
In my experience it should not make much or any difference.
I had devs try to "optimise" the JSON our services were returning by doing things like abbreviating field names that were repeated very often or replacing long string constants with short integers or removing formatting, and so on. When I asked them to do actual test when the service returns gzipped output and they found out that even reducing the JSON size this way by half does not make any measurable difference on the compressed stream in most cases.
I had devs try to "optimise" the JSON our services were returning by doing things like abbreviating field names that were repeated very often or replacing long string constants with short integers or removing formatting, and so on. When I asked them to do actual test when the service returns gzipped output and they found out that even reducing the JSON size this way by half does not make any measurable difference on the compressed stream in most cases.
I found the same re: compressed stream, however I also found that hand optimizing the decompressed JSON made for significant performance improvements on underpowered devices (e.g. some old Android phone).
Well, that's true. But if performance is more important to you, just skip it and go for binary formats that can be orders of magnitude more efficient.
In general text formats are a compromise. They are bulky and and inefficient to parse. What you get in exchange is ease of development.
If you are willing to complicate the format just to improve performance, at some point you no longer get ease of development and you should just switch to binary altogether.
In general text formats are a compromise. They are bulky and and inefficient to parse. What you get in exchange is ease of development.
If you are willing to complicate the format just to improve performance, at some point you no longer get ease of development and you should just switch to binary altogether.
This actually does. I used Python gzip to see how the minified would compress and came up with 157 bytes to JCOF's 134. Using the same mechanism to gzip the JCOF saves only 4 bytes. I'm actually rather impressed.
I think people would reject your samples because the data size is too small. Production JSON comes in all shapes and sizes and situations where you care about performance usually indicates much larger payloads.
If your egress is paramount, you should of course benchmark everything to shreds. Then pick what fits.
It might on small files but it doesn't on (some) bigger ones. e.g. I dumped a Minecraft region file into JSON giving 85M. `jcof` turns this into 66M. `gzip` on the original json gives 14M for `-1` and 11M for `-9`. `zstd` also gives 11M.
But because the Minecraft JSON is highly redundant, I also tried it on 84M of fake social graph data. `jcof` gave 44M, `gzip -9` gave 19M and `zstd` gave 18M.
In summary, neat idea, but you're maybe better off with `gzip` or `zstd` in the real world.
But because the Minecraft JSON is highly redundant, I also tried it on 84M of fake social graph data. `jcof` gave 44M, `gzip -9` gave 19M and `zstd` gave 18M.
In summary, neat idea, but you're maybe better off with `gzip` or `zstd` in the real world.
JCOF and gzip are not mutex.
No, but when you get ~95% of the benefit (per other people's data) while keeping all the flexibility, compatibility with everything under the sun, and native-performance parsing everywhere, the supposed improvement looks a lot less like one.
A sibling has the data. For that case it's 10% of egress which can be a healthy chunk of change on your bill. For all such use cases you should of course benchmark a lot and then pick what fits. I think this tool is a nice one to have in the toolbox. The low-rate files seem to be close to optimal.
> For that case it's 10% of egress which can be a healthy chunk of change on your bill. For all such use cases you should of course benchmark a lot and then pick what fits.
Provided that you don't run into the issue of one end only talking in language X, which does not have JCOF support due to there not being any libraries for it yet. In the GitHub repo, it seems like there is only the JavaScript reference implementation for now.
Provided that you don't run into the issue of one end only talking in language X, which does not have JCOF support due to there not being any libraries for it yet. In the GitHub repo, it seems like there is only the JavaScript reference implementation for now.
The design documents are really thorough. I don't think I would be more than a day making a serializer in Python for example. Lack of libraries just means an interested developer needs to tend to the matter and solve the issue.
> A sibling has the data.
And now has updated data showing that sorting your keys lets `lzma` beat `jcof|lzma` at every level. The egress bill has been reduced! All rejoice!
And now has updated data showing that sorting your keys lets `lzma` beat `jcof|lzma` at every level. The egress bill has been reduced! All rejoice!
EDIT: Nope, I was wrong, I was doing `lzma` against `jcof|zstd`. `jcof|lzma` is still sneaking ~1M below `lzma` at all levels.
See https://news.ycombinator.com/item?id=32111239 where I tested gzip vs jcof vs jcof+gzip (tl;dr: you save 2M at the cost of an extra CPU intensive process)
People keep trying to re-invent a human-readable, parsable, efficient, robust format that can be validated.
But "easy" wins so we keep returning to CSV and JSON. ^_^
But "easy" wins so we keep returning to CSV and JSON. ^_^
Gzip may be standard and supported transparently by your http library.
OTOH zstd [1] should be significantly more efficient, while being as fast or faster.
[1]: https://en.wikipedia.org/wiki/Zstd
OTOH zstd [1] should be significantly more efficient, while being as fast or faster.
[1]: https://en.wikipedia.org/wiki/Zstd
no offense, but this seems pointless tbh.
If I need a more compact no longer at all human readable format I can use a compact binary format. If not I can use JSON which has wide spread support.
or just use a compression (after which the difference between JSON and JCOF should be negligible)
If I need a more compact no longer at all human readable format I can use a compact binary format. If not I can use JSON which has wide spread support.
or just use a compression (after which the difference between JSON and JCOF should be negligible)
I'm curious, which more compact format would you use? Because I have looked a bit and been unable to find one, both MessgePack and CBOR produce pretty large objects. CBOR with the string reference extension comes closer, but it's still bigger, and string references don't seem very well supported in the CBOR ecosystem.
Do you mean you'd use a format with a schema?
Do you mean you'd use a format with a schema?
The comments here are really negative.
I think this is a great idea. JSON is incredibly wasteful and any improvement is welcome.
And yes, you can gzip JSON. still wastes a lot of space when you actually need it in a format where you might want to read individual fields.
I think this is a great idea. JSON is incredibly wasteful and any improvement is welcome.
And yes, you can gzip JSON. still wastes a lot of space when you actually need it in a format where you might want to read individual fields.
If a specific use-case can benefit from it, fine.
I love me some jq, not moving much data as JSON, and so this amounts to an interesting exercise.
I love me some jq, not moving much data as JSON, and so this amounts to an interesting exercise.
I would like to see gzip compression added to the benchmark
curl https://raw.githubusercontent.com/mortie/jcof/main/tests/corpus/meteorites.json | gzip -9 | wc
Gives me 34569
JSON: 244920 bytes
JCOF: 87028 bytes
GZIP: 34569 bytesTo be fair you would gzip the JCOF encoding in this example too.
Author mentions gzip doesn’t work for some use cases. For use case mentioned I’d expect sqlite to be similar, at least that is the default thing I’d reach for.
If for some reason sqlite wasn’t sufficient probably a custom binary encoding controlled and updated via code instead of config would be next.
Author mentions gzip doesn’t work for some use cases. For use case mentioned I’d expect sqlite to be similar, at least that is the default thing I’d reach for.
If for some reason sqlite wasn’t sufficient probably a custom binary encoding controlled and updated via code instead of config would be next.
> To be fair you would gzip the JCOF encoding in this example too.
Just tested my 84M fake social media file - `jcof` gives 44M, `gzip` gives 19M, `jcof+gzip` gives 17M. In essence, you've gained 2M for two CPU intensive procedures instead of one. Doesn't seem all that worth it?
Just tested my 84M fake social media file - `jcof` gives 44M, `gzip` gives 19M, `jcof+gzip` gives 17M. In essence, you've gained 2M for two CPU intensive procedures instead of one. Doesn't seem all that worth it?
10% is a lot of your egress.
A fair point.
Prompted me to check if the higher zstd levels worked any better on my 84MB fake social graph - nope - and then if LZMA was any good - yes, `lzma` at 5 or higher on the raw JSON beats `jcof | lzma` by ~2M every time. `lzma -4` beats it by ~400k.
If I sort my object keys (a la `jq -S`), `lzma` beats `jcof|lzma` at every level (`gzip` never gets close, `zstd` gets closer.)
Prompted me to check if the higher zstd levels worked any better on my 84MB fake social graph - nope - and then if LZMA was any good - yes, `lzma` at 5 or higher on the raw JSON beats `jcof | lzma` by ~2M every time. `lzma -4` beats it by ~400k.
If I sort my object keys (a la `jq -S`), `lzma` beats `jcof|lzma` at every level (`gzip` never gets close, `zstd` gets closer.)
EDIT: Nope, I was wrong, I was doing `lzma` against `jcof|zstd`. `jcof|lzma` is still sneaking ~1M below `lzma` at all levels.
meteorites.json, re-encoded via both JSON.stringify() and jcof.stringify():
json jcof jcof/json
plain 244975 87083 0.355
gzip -6 35829 33152 0.925
gzip -9 34384 32875 0.956
xz -9 27864 28696 1.030
Serialization formats that dedupe and perform other fancy processing trade increased CPU time for decreased memory usage. People who care about CPU time will want to measure the CPU time hit as well as the memory savings.
This format is also not interoperable without a decoding library, which kind of invalidates the comparison to JSON. If you're going to do this, why not just go full binary and save even more memory?
This format is also not interoperable without a decoding library, which kind of invalidates the comparison to JSON. If you're going to do this, why not just go full binary and save even more memory?
You're decoding JSON in all cases. Usually network dominates over cpu. Sometimes it's useful to have human readable serializations, especially if you expect to operate in anger.
Depends on your needs. When i'm really concerned about deserialization time, i try to skip it entirely. Partial|Total zero copy deserialization is the thing to reach for. Various libs exist, https://rkyv.org/ is what i've been toying with recently.
For those unfamiliar, here is the interesting bit:
> rkyv implements total zero-copy deserialization, which guarantees that no data is copied during deserialization and no work is done to deserialize data. It achieves this by structuring its encoded representation so that it is the same as the in-memory representation of the source type.
For those unfamiliar, here is the interesting bit:
> rkyv implements total zero-copy deserialization, which guarantees that no data is copied during deserialization and no work is done to deserialize data. It achieves this by structuring its encoded representation so that it is the same as the in-memory representation of the source type.
Thank you for linking to this!
It is super cool that it serializes hash tables and b-trees because that is where serde’s zero copy parsing ends
It seems perfect for my use case, which is putting stuff in a chunk of memory shared between processes
It is super cool that it serializes hash tables and b-trees because that is where serde’s zero copy parsing ends
It seems perfect for my use case, which is putting stuff in a chunk of memory shared between processes
Slight awkwardness with the spoken pronunciation.
Glad I'm not the only one who reads it that way.
J cough?
Jack off, I'm afraid.
Interesting, I wonder which type of English speakers would see that as their initial hunch for pronunciation.
While Oscar Wilde would undoubtedly have something witty to say about that, I guess it's a matter of stress: if you read it as a word and assume stress on the (invisible) first vowel, it's almost inevitable: juh-cof.
But we dont say juh sahn for JSON? Hence why Jcough seemed intuitive.
Language is not very deterministic. An unknown word can remind you of more than one other word, which can influence the way you pronounce it, certainly in English where pronunciation is so irregular. That’s not a fact of course, just a possible factor.
Guess you could say if its a 'jack off' here, its 'jiss on' there.
There is a very, very small difference between placing a slight break in the middle of the acronym as J-COF or as JC-OF, because there's a major unwritten vowel sound after J in any case.
Thing about JSON is the ease of reading and writing it, regardless the size of the document. Compression deals with the rest.
It may be easy to read and write, but few libraries implement it to the letter because the format doesn’t support extremely common use cases.
For those who disagree, try finding a json library that can reliably round-trip the data in a json file (i.e. ignoring spacing, tabs, and extra line breaks)
Common breaking points are discrimination between integers and doubles (e.g read 0.0 and write 0) and limits to the number of digits in a number.
It’s futile to try and change it now, but I think json would have been better if it had disjunct sets of integers and floats, did 64-bit ints (1) and IEEE doubles (1) as opposed to arbitrary length integers and floats, standardized serialization of time stamps and durations.
Requiring parsers to keep keys in file order also might have been a good thing.
(1) I know that opens a rat’s nest of users wanting unsigned 64-bit integers, signed 16-bit, 128-bit quad floats, etc, but I don’t see how not addressing that problem at all is better than picking reasonable values, event though that gives up on supporting those alternatives. Now, we have libraries making choices there, hopefullly in compatible ways.
For those who disagree, try finding a json library that can reliably round-trip the data in a json file (i.e. ignoring spacing, tabs, and extra line breaks)
Common breaking points are discrimination between integers and doubles (e.g read 0.0 and write 0) and limits to the number of digits in a number.
It’s futile to try and change it now, but I think json would have been better if it had disjunct sets of integers and floats, did 64-bit ints (1) and IEEE doubles (1) as opposed to arbitrary length integers and floats, standardized serialization of time stamps and durations.
Requiring parsers to keep keys in file order also might have been a good thing.
(1) I know that opens a rat’s nest of users wanting unsigned 64-bit integers, signed 16-bit, 128-bit quad floats, etc, but I don’t see how not addressing that problem at all is better than picking reasonable values, event though that gives up on supporting those alternatives. Now, we have libraries making choices there, hopefullly in compatible ways.
Neither fully human-editable nor really compact, but an interesting experiment nonetheless.
Nevermind readable
I'm still not sure wtf the leading 0, on every object is supposed to represent.
I think that's representing the keys, so if you'd have multiple types of objects in an array the key table would have multiple rows and the leading number index for each object would define what the keys for that object would be.
> gzipping stuff from JavaScript in the browser is inconvenient until TextEncoderStream is supported in Firefox
There are a pure-JS DEFLATE implementations with reasonable performance-- I recommend fflate.
There are a pure-JS DEFLATE implementations with reasonable performance-- I recommend fflate.
JSON is supposed to be human readable this doesn’t even seem like it would address the same issues.
I'm always conflicted.
They're improvements, they're experiments, they're an attempt to move forward. Somebody cares!
but Randall Munroe said it pretty well: https://xkcd.com/927/
They're improvements, they're experiments, they're an attempt to move forward. Somebody cares!
but Randall Munroe said it pretty well: https://xkcd.com/927/
JSON isn't mean to be efficient, but to be a good compromise between being human readable/writable, easily generated/parsed by a machine, and familiar. CBOR and BSON make more sense, to be frank, for any use case I could think of using this in.
I don't think BSON de-dupes the keys in the manner that JCOF does? But it has other nice properties. In a game I worked on we used a BSON library that didn't really "parse" the data when you loaded a file; it just kept a pointer to the original binary data and let you perform queries against it.
We used this to keep our memory footprint low; instead of loading BSON files we would memmap them (getting a pointer that can be used to read the data right off the disk) and passed that to the BSON library. Bang, we could map a 10MB file and it wouldn't use 10MB of system memory; it would just swap portions in and out as they were used.
This has performance tradeoffs but for us (on mobile, where there is no swap space) the memory savings was worth it.
We used this to keep our memory footprint low; instead of loading BSON files we would memmap them (getting a pointer that can be used to read the data right off the disk) and passed that to the BSON library. Bang, we could map a 10MB file and it wouldn't use 10MB of system memory; it would just swap portions in and out as they were used.
This has performance tradeoffs but for us (on mobile, where there is no swap space) the memory savings was worth it.
* JSON is really schemaless so I don't have to assume all objects are shaped the same or that they are even the same kind of object. This allows for streaming serialization, and does not require the data structure to be known or to introspect data to create the heading lines.
* Nested objects look to be difficult, especially if the schema is not known.
* JSON is very human friendly, JCOF is not.
I do think the format is better than CSV, TSV and other delimited flat record formats. JCOF appears to be able to handle objects that are not tabular, which CSV just can't really do. A replacement for JSON this is not, and storage efficiency isn't really relevant because of the difference in purpose.