flatbuffers and capnproto are in the game of trying to make serialization to binary format as efficient as possible. Their goal is trying to beat benchmarks: how long it takes to convert an object to bytes and vice-versa. It's cool, but I personally think that for most use cases (not all), serialization efficiency shouldn't be the primary goal: serialization time is often negligible compared to time it takes to send data over the wire, and it's less important than other features (e.g. quality of the generated API) that some of these techs might neglect.
I have an example to illustrate this. With Proto3, Google decided that when encoding a `string` field in C++, it would not perform UTF-8 validation. This leads to better benchmark metrics. This has also been a horrible mistake that led to many bugs which have costed so much in eng hours, since for example the same protobuf C++ API fails at deserialization when it encounters an invalid UTF-8 string.
As per messagepack, jsonbinpack, these seem to be layers on top of JSON to make JSON more compact. They still use field names for field identity, which I think can be problematic for long-term data persistence since it prevents renaming fields. I think the Protobuf/Thrift approach of using meaningless field numbers in serialization forms is better.
Thanks for the comment! Agree with you about the horrible Protobuf-to-Python bidding, it was a big frustration and definitely contributed to me wanting to build Skir.
1. You can create an enum with just "wrapper" fields, that's exactly like a oneof
2. Totally fair, I'm planning to work on this later this year, probably Q3 (priority is adding support to 4 more languages, and then I'll get to it)
Hey, thanks a lot for the comment! I share your frustration with protobuf: although I think it's great, it carries a few design flaws which are hard to fix at this point and they create pain points which are not going away.
I completely agree with you about timing, wish I had done this 10 years ago :)
"Here is an idea to contemplate as a side gig with your favorite Ai assistant: A tool to convert proto to Skir. Or at least as much as possible. As someone who had to maintain larger and complex proto files, a lot of proto specific pain points are addressed." < I tried asking Claude: "Migrate this project from protobuf to Skir, see https://skir.build/" and it works pretty well. I created http://skir.build/llms.txt which helps with this.
The pain point is data migration though, and as much as I want Skir to succeed: I cannot recommend people migrating from protobuf to Skir if they have some persisted data to migrate, the effort is probably not worth it.
Supposedly you remove a field after you have migrated all code that uses the field and you have deployed all binaries.
In your DB, you still have [10,[email protected]","john"], [11,null,"jane"], which you are able to deserialize fine (the email field is ignored).
New values that you serialize are stored as [12,0,"jack"].
If you happen to have old binaries which still use the old email field and which are still running (which you shouldn't, but let's imagine you accidentally didn't deploy all your binaries before you removed the field), these new binaries will indeed decode the email field for new values (Jack) as an empty string instead of null.
I looked at Prisma, I very much prefer the Protobuf/Thrift model of using numbers to identify fields, which allows 2 important things: fields to be renamed without breaking backward compatibility, and a compact wire format.
I think the Protobuf language (which Skir is heavily influenced by) has some flaws in its core design, e.g. the enum/oneof mess, the fact that it allows spare field numbers which makes the "dense JSON" format (core feature of Skir) harder to get, the fact that it does not allow users to optionally specify a stable identifier to a message to get compatibility checks to work.
I get your point about "why building another language", but also that point taken too far means that we would all be programming in Haskell.
Thanks for the comment. I am very familiar with Buf+Protobuf, I think it's a great system overall but has many limitations which I think can be overcome by redesigning the language from scratch instead of building on top of the .proto syntax.
In the Skir vs Protobuf part of the blog post [https://medium.com/@gepheum/i-spent-15-years-with-protobuf-t...], only 2 out of 10 pertain to "syntax" (and they're a bit more than syntax). Since you mention compatibility check, Buf's compatibility check prevents message renaming, which is a huge limitation. With Skir, that's not the case. You also get the compatibility checks verified in the IDE.
That is correct and that is a good catch, the idea though is that when you remove a field you typically do that after having made sure that all code no longer read from the removed field and that all binaries have been deployed.
Protobuf is battle-tested and excellent. If your team already runs on Protobuf and has large amounts of persisted protobuf data in databases or on disk, a full migration is often a major effort: you have to migrate both application code and stored data safely. In many cases, that cost is not worth it.
For new projects, though, the choice is open. That is where Skir can offer a meaningful long-term advantage on developer experience, schema evolution guardrails, and day-to-day ergonomics.
"""
0. Yes, I looked at Avro, Ion. I like Protobuf much better because I think using field numbers for field identity, meaning being able to rename fields freely, is a must.
1. Yes. Skir also supports that with binary format (you can serialize and deserialize a Skir schema to JSON, which then allows you to convert from binary format to readable JSON). It just requires to build many layers of extra tooling which can be painful. For example, if you store your data in some SQL engine X, you won't be able to quickly visualize your data with a simple SELECT statement, you need to build the tooling which will allow you to visualize the data.
Now dense JSON is obviously not idea for this use case, because you don't see the field names, but for quick debugging I find it's "good enough".
3. I agree there are definitely cases where it can be painful, but I think the cases where it actually is helpful are more numerous.
One thing worth noting is that you can "opt-out" of this feature by using `ClassName.partial(...)` instead of `ClassName()` at construction time. See for example `User.partial(...)` here: https://skir.build/docs/python#frozen-structs
I mostly added this feature for unit tests, where you want to easily create some objects with only some fields set and not be bothered if new fields are added to the schema.
Thanks!
Main use case (similarly to Protobuf) is when you need to exchange data types between systems written in different languages. Like Protobuf, it can also be used in a mono-linguistic system, when you want to serialize systems and have strong guarantees that you will be able to deserialize your data in the future (when you use classic serialization libraries like Pydantic, Java Serialization etc., it's easy to accidentally modify a schema and break the ability to deserialize old data.)
> Also, I disagree on the constructor front. Proto forces you to grapple with the reality that a field may be missing. In a production system, when adding a new field, there will be a point where that field isn’t present on only one side of the network call. The compiler isn’t saving you.
I agree it's important for users to understand that newer fields won't be set when they deserialize old data -- whether that's with Protobuf or Skir. I disagree with the idea that not forcing you to update all constructor call sites when you add a field will help (significantly) with that. Are you saying that because Protobuf forces you to manually search for all call sites when you add a field, it forces you to think about what happens if the field is not set at deserialization, hence, it's a good thing? I'm not sure that outweighs the cost of bugs introduced by cases where you forget to update a constructor call site when you add a field to your schema.
I think you may be referring to JSPB. It's used internally at Google but has little support in the open-source. I know about it, but I wouldn't say I was inspired by it. It's particularly unreadable, because it needs to account for field numbers being possible sparse.
Google built it for frontend-backend communication, when both the frontend and the backend use Protobuf dataclasses, as it's more efficient than sending a large JSON object and also it's faster to deserialize than deserializing a binary string on the browser side. I think it's mostly deprecated nowadays.
Skir has exactly the same goals as Protobuf, so yes, that sentence can apply to Protobuf as well (and buf.build).
I listed some of the reasons to prefer Skir over Protobuf in my humble opinion here: https://medium.com/@gepheum/i-spent-15-years-with-protobuf-t...
Built-in compatibility checks, the fact that you can import dependencies from other projects (buf.build offers this, but makes you pay for it), some language designs (around enum/oneof, the fact that adding fields forces you to update all constructor code sites), the dense JSON format, are examples.
Links: - Website: https://skir.build/ -- Skir Go doc: https://skir.build/docs/go -- Skir Rust doc: https://skir.build/docs/rust -- Skir Swift doc: https://skir.build/docs/swift - Github: https://github.com/gepheum/skir - Blog post: https://medium.com/@gepheum/i-spent-15-years-with-protobuf-t... - Discord: https://discord.gg/mruvDuybJ