The Protobuf Language Specification(buf.build)
buf.build
The Protobuf Language Specification
https://buf.build/blog/protobuf-language-specification
95 comments
Actually, I thought about it twice and I retract what I said about this specification being pointless for building tools compatible with protobuf. Reasons:
* The language itself is unlikely to change much given it's been public for so long. A non-official spec that captures the current implementation is probably going to survive for some time.
* There's no official spec (which I would prefer) for me to base my tool on. This spec is about my only choice. The more tools targeting this spec, the hardest would be for Google to break compatibility with it, reinforcing my previous point.
I will keep the parent comment for context, and I don't retract the fact that I think the title is misleading. Otherwise, great work!!
* The language itself is unlikely to change much given it's been public for so long. A non-official spec that captures the current implementation is probably going to survive for some time.
* There's no official spec (which I would prefer) for me to base my tool on. This spec is about my only choice. The more tools targeting this spec, the hardest would be for Google to break compatibility with it, reinforcing my previous point.
I will keep the parent comment for context, and I don't retract the fact that I think the title is misleading. Otherwise, great work!!
Agreed. If IDEs and alternative compilers are all building off of the spec because it's the path of least resistance, and there are no bugs for a while, the defacto standard impl is going to face serious scrutiny for parting from it.
And, as you said, proto isn't in a great position to be making crazy changes anyway.
And, as you said, proto isn't in a great position to be making crazy changes anyway.
Our aim is to make the spec accurately match Google's reference compiler -- for as long as that is the source of truth, which is hopefully not forever :)
Even for those not using Buf, we expect this documentation to be of interest to the community as it describes a large number of facets of protoc that were previously undocumented (and required examining the source for protoc or playing around with test source code to see what it expects and what descriptors it generates).
If issues are found with this spec, it is true that we'll most likely have to revise the spec to match the compiler, not the other way around. But no software is perfect: some variations will be due to bugs in protoc, which can be fixed in the compiler to properly match the spec. Over time, we'd love to see an outcome where a formal specification is the source of truth.
For now, our commitment is to make (and keep) this spec as accurate as possible to describe the Protobuf language, not some Buf dialect.
Even for those not using Buf, we expect this documentation to be of interest to the community as it describes a large number of facets of protoc that were previously undocumented (and required examining the source for protoc or playing around with test source code to see what it expects and what descriptors it generates).
If issues are found with this spec, it is true that we'll most likely have to revise the spec to match the compiler, not the other way around. But no software is perfect: some variations will be due to bugs in protoc, which can be fixed in the compiler to properly match the spec. Over time, we'd love to see an outcome where a formal specification is the source of truth.
For now, our commitment is to make (and keep) this spec as accurate as possible to describe the Protobuf language, not some Buf dialect.
That's basically the equivalent of RubySpec — reverse engineered from MRI (the original Ruby implementation) for use by Rubinius. It was adopted by the other alternative Ruby implementations too.
It looks like the original is now gone, but a fork has taken over. Looking at some comments, fighting to get MRI to adopt it may have burnt out the people behind it.
It looks like the original is now gone, but a fork has taken over. Looking at some comments, fighting to get MRI to adopt it may have burnt out the people behind it.
Googler, opinions are my own. I don't work on protobuf at all, just use it all the time (like most Googlers)
I haven't dug into this in great detail yet, but the hard thing about the proto "spec" is that there isn't one, and protoc lets you do all kinds of crazy things that are really hard to model in languages like Antlr. There were some poor choices in protoc dating back to when proto1 was first created that have been carried forward. Having 20 years of proto definitions lets people come up with some crazy use cases.
Definitely interesting for this company to create an EBNF definition for protobuf.
I haven't dug into this in great detail yet, but the hard thing about the proto "spec" is that there isn't one, and protoc lets you do all kinds of crazy things that are really hard to model in languages like Antlr. There were some poor choices in protoc dating back to when proto1 was first created that have been carried forward. Having 20 years of proto definitions lets people come up with some crazy use cases.
Definitely interesting for this company to create an EBNF definition for protobuf.
It's possible that the internal version of protoc is very different from the open-source version. (I know there are numerous differences, but not sure how pervasive they are in the parser.)
The open-source version has a hand-written tokenizer and recursive descent parser that is not too difficult to translate to EBNF. You'll notice that the section on numeric literals is a little wonky, because the tokenizer does a check that is hard to describe in EBNF. But it isn't too bad.
Also, some of the constraints of the language are in prose in this spec because they are easier to enforce using a semantic validation pass, instead of trying to model purely with a CFG. (Optionality of the colon in the text format, used in message literals, comes to mind.)
There are some things that technically _could_ be handled in the grammar, but they would make the grammar much more cumbersome to read and understand. So those things are also extracted into prose.
> Definitely interesting for this company to create an EBNF definition for protobuf.
For what it's worth, Google has also published an EBNF definition (the subject blog post contains links to those specs). But they are incomplete and not entirely accurate, which is a non-trivial part of what led us to writing and publishing this spec.
The open-source version has a hand-written tokenizer and recursive descent parser that is not too difficult to translate to EBNF. You'll notice that the section on numeric literals is a little wonky, because the tokenizer does a check that is hard to describe in EBNF. But it isn't too bad.
Also, some of the constraints of the language are in prose in this spec because they are easier to enforce using a semantic validation pass, instead of trying to model purely with a CFG. (Optionality of the colon in the text format, used in message literals, comes to mind.)
There are some things that technically _could_ be handled in the grammar, but they would make the grammar much more cumbersome to read and understand. So those things are also extracted into prose.
> Definitely interesting for this company to create an EBNF definition for protobuf.
For what it's worth, Google has also published an EBNF definition (the subject blog post contains links to those specs). But they are incomplete and not entirely accurate, which is a non-trivial part of what led us to writing and publishing this spec.
One place protoc doesn't align well is the descriptor object. https://developers.google.com/protocol-buffers/docs/referenc...
Comment placement is basically allowed anywhere by protoc, but how to get those comments within a Descriptor object for a proto is not well defined (there are places where you can put comments that are not available within Descriptor). It provides leading/trailing comments, but there are many other cases that are missed today (like comments embedded within a list of items in an array). Maybe this is a mismatch between what protoc allows and what Descriptor presents, but it's definitely annoying.
Comment placement is basically allowed anywhere by protoc, but how to get those comments within a Descriptor object for a proto is not well defined (there are places where you can put comments that are not available within Descriptor). It provides leading/trailing comments, but there are many other cases that are missed today (like comments embedded within a list of items in an array). Maybe this is a mismatch between what protoc allows and what Descriptor presents, but it's definitely annoying.
I agree! Protoc allows comments anywhere, but it doesn't bother preserving them all in the descriptor. At one point I thought this was a bug, since comments _could_ be preserved in far greater contexts (though definitely not all). But then I realized that the descriptor.proto comments do actually state the places where comments are retained:
> If this SourceCodeInfo represents a complete declaration, these are any > comments appearing before and after the declaration which appear to be > attached to the declaration.
https://github.com/protocolbuffers/protobuf/blob/v21.5/src/g...
After re-reading with this caveat in mind -- that comments are only preserved before and after complete declarations -- I realized that it does retain the comments that are expected.
Long story short: the descriptor is bad as an AST if you care about recovering the original source. The descriptor is lossy. I'd recommend using a real AST for use cases that want something non-lossy: https://pkg.go.dev/github.com/jhump/protoreflect/desc/protop...
> If this SourceCodeInfo represents a complete declaration, these are any > comments appearing before and after the declaration which appear to be > attached to the declaration.
https://github.com/protocolbuffers/protobuf/blob/v21.5/src/g...
After re-reading with this caveat in mind -- that comments are only preserved before and after complete declarations -- I realized that it does retain the comments that are expected.
Long story short: the descriptor is bad as an AST if you care about recovering the original source. The descriptor is lossy. I'd recommend using a real AST for use cases that want something non-lossy: https://pkg.go.dev/github.com/jhump/protoreflect/desc/protop...
I totally forgot about .proto comments. I spent a while re-implementing that [1] for the IntelliJ editor and even longer testing it [2]. Apparently I opened a bug while doing so; is b/33539835 still open?
1: https://github.com/jvolkman/intellij-protobuf-editor/blob/ma... 2: https://github.com/jvolkman/intellij-protobuf-editor/blob/ma...
1: https://github.com/jvolkman/intellij-protobuf-editor/blob/ma... 2: https://github.com/jvolkman/intellij-protobuf-editor/blob/ma...
if comfortable answering, why protobuf instead of gob?
Gob is Go-specific. Our mission is to make schema-driven APIs easy, regardless of what language you use. Protobuf already has official support for nearly a dozen languages, and unofficial support for many more. Protobuf also has a compiler with a plugin model, which facilitates supporting even more in the future.
Furthermore, Protobuf is an IDL, not a full-blown programming language. This makes it ideal for this use case, for describing APIs and data structures.
Gob-encoded data structures are described with Go. While Go is great for writing server-side business logic, it is not as well-suited as a description language for data that you need to share with non-Go systems.
Furthermore, Protobuf is an IDL, not a full-blown programming language. This makes it ideal for this use case, for describing APIs and data structures.
Gob-encoded data structures are described with Go. While Go is great for writing server-side business logic, it is not as well-suited as a description language for data that you need to share with non-Go systems.
thanks for the super clear explanation, which i definitely thought was the case (thinking back to NeXTstep era IDL and all thereafter)
proto predates gob by quite a bit. gob was introduced with golang, and it's not really used anywhere else.
My takeaway from Java serialization (which became passé) was that a schema-driven encoding that’s supported in many languages is a lot more useful.
Java serialization is passe largely because it has huge gaping holes in how one can abuse the serialization loaders to perform very unsafe security holes. And it's incredibly slow, and tied to JVMs, but sadly only really tied to JVM that have the exact same version and serialized objects, etc.. did I mention java serialization sucks? (Spoken by a true java lover that that subjected himself to serialization way more than he wanted to)
This is interesting. So one company invents and maintains a compiler, then a different company found that documentation to be insufficient (big surprise), so wrote out a lengthy standard that conformed to what the first company's compiler happened to do? Seems very useful, but also seems risky and hard to maintain. What happens when Google tightens a constraint or adds a new feature next week?
I think the hope is that this could be a situation like CommonMark and Markdown, where google's implementation will continue to exist, but that the community just totally moves over to this new specification / tooling, and that anytime someone says "protobuf", they don't even necessarily realize that Google has a thing, they just know this specification.
Trouble is that I don’t think everyone’s moved on with Markdown, I still encounter different dialects that are subtly different. Even within one suite of tools from a single vendor … [angry stares in the direction of Atlassian]
... except for whenever you've got to communicate or exist inside the Google ecosystem. Now you've got "protobuf as per Google" and "protobuf as per internet randos" with twice the dependency graph.
It's unfortunate, and not surprising, that Google hasn't made a protobuf spec. I realize there are a ton of protobuf fans out there but, personally, it just feels like a massive, awkwardly maintained mess that I'm forced to live with for (mostly) their benefit.
It's unfortunate, and not surprising, that Google hasn't made a protobuf spec. I realize there are a ton of protobuf fans out there but, personally, it just feels like a massive, awkwardly maintained mess that I'm forced to live with for (mostly) their benefit.
We keep an eye on the protobuf repo, so if a change is made we can both incorporate it into our products (https://buf.build) and into this spec.
A great outcome for the ecosystem would be that Google chooses to engage with the community before making language changes. And the best possible outcome would be that the authority is eventually inverted: a specification doc becomes the definitive source of truth on the language and `protoc` is updated to conform to it, instead of vice versa.
A great outcome for the ecosystem would be that Google chooses to engage with the community before making language changes. And the best possible outcome would be that the authority is eventually inverted: a specification doc becomes the definitive source of truth on the language and `protoc` is updated to conform to it, instead of vice versa.
Generally new features in protobuf are just new features, so if a new feature is added then the worst case is the documentation will be out of date/incomplete for a short period of time.
For the most part the "constraint tightening" thing doesn't affect the language specification, at least in my experience. For example, there have been some changes in protobuf that affect things like serialization order. A change like that can break brittle tests that do things like checking that a function produces some exact serialized string/byte sequence, but they don't affect the semantics of the language.
For the most part the "constraint tightening" thing doesn't affect the language specification, at least in my experience. For example, there have been some changes in protobuf that affect things like serialization order. A change like that can break brittle tests that do things like checking that a function produces some exact serialized string/byte sequence, but they don't affect the semantics of the language.
This is a similar situation to Ruby and Rubinius (an alternative implementation of Ruby). Because there was no Ruby specification other than the original MRI implementation, the Rubinius project (a new alternative implementation) created their own test suite to codify expected Ruby behavior. However, the MRI developers didn't use it, and the behavior diverged.
The original creator gave up on the idea [0] but it was immediately taken over by others and is still maintained [1]. In case of conflict, though, Matz (lead developer on MRI), not the specification, is the source of truth [2].
[0] https://github.com/rubinius/rubinius-website-archive/blob/87...
[1] https://github.com/ruby/spec
[2] http://ruby.github.io/rubyspec.github.io/bugs_found/
The original creator gave up on the idea [0] but it was immediately taken over by others and is still maintained [1]. In case of conflict, though, Matz (lead developer on MRI), not the specification, is the source of truth [2].
[0] https://github.com/rubinius/rubinius-website-archive/blob/87...
[1] https://github.com/ruby/spec
[2] http://ruby.github.io/rubyspec.github.io/bugs_found/
(deleted)
Matt, that is not our intention. But we are trying to build a business around making schema-driven APIs easy, and protobuf is at the core of our current products. So we are trying to improve the ecosystem around protobuf, and a critical aspect of that in our esteem is having a spec.
While `protoc` remains the source of truth, this spec captures the syntax and rules accepted and enforced by `protoc` in a far more detailed way than the official developer site.
While `protoc` remains the source of truth, this spec captures the syntax and rules accepted and enforced by `protoc` in a far more detailed way than the official developer site.
Ah, hey Josh. I didn't realize you were involved in this effort. FWIW, seeing your name associated with this definitely gives it a bit more authority than I had originally assumed.
If it helps, I (original author of proto2) have been advising Buf and like what they're doing. (Disclosure: I also invested a small amount.)
Buf is founded by engineers who spent a LOT of time working with Protobufs outside of Google. I was always the one saying "please don't write your own .proto parser" but I am convinced Buf actually knows what they are doing here and probably have all the details right.
Our industry has a whole lot of tooling and infrastructure built around JSON, and almost every piece of it could be way better if it were operating with well-defined types instead, in the same way that TypeScript tooling benefits vs. JavaScript. Google has had an all-protobuf ecosystem internally for a long time, but much of it will never be released publicly. So that leaves someone like Buf to really build it out. I'm pretty interested to see where they're able to take it.
Buf is founded by engineers who spent a LOT of time working with Protobufs outside of Google. I was always the one saying "please don't write your own .proto parser" but I am convinced Buf actually knows what they are doing here and probably have all the details right.
Our industry has a whole lot of tooling and infrastructure built around JSON, and almost every piece of it could be way better if it were operating with well-defined types instead, in the same way that TypeScript tooling benefits vs. JavaScript. Google has had an all-protobuf ecosystem internally for a long time, but much of it will never be released publicly. So that leaves someone like Buf to really build it out. I'm pretty interested to see where they're able to take it.
Ahhh, is that the play? That makes a lot of sense.
> As of today, Protobuf is now a fully-defined language:
(Etc.)
I'm not sure what this is meant to achieve in reality. There is still only one implementation that defines what the language actually is, and that is Google's protoc.
I'm my experience working with and writing alternative parsers for the '.proto' language, I've found that time and again Google's documentation for the format is either woefully vague, or directly contradicts the actual implementation. I don't see the value in a third party "spec" if what I have to do in practice will always be "whatever Google did in protoc".
(Etc.)
I'm not sure what this is meant to achieve in reality. There is still only one implementation that defines what the language actually is, and that is Google's protoc.
I'm my experience working with and writing alternative parsers for the '.proto' language, I've found that time and again Google's documentation for the format is either woefully vague, or directly contradicts the actual implementation. I don't see the value in a third party "spec" if what I have to do in practice will always be "whatever Google did in protoc".
The intent of this spec is to actually put "whatever Google did in protoc" into a readable format, so you don't have to read the C++ code. The official docs fall short on providing much of the details that are included.
That sounds great, but what's the governance story? Are the authors of the spec document committing to keeping the document up to date here henceforth? Are the protoc maintainers committing to have these folks involved in project direction decisions?
As of right now, the former. That is currently required for our tools to remain functioning (https://buf.build). If/when changes are made to the language, we update our tools (and this spec) to continue to be accurate.
Protobuf isn't too complicated, I've found the wire format docs to be some of the best among the avro/msgpack/thrift/etc competitors.
Maybe you mean something besides the wire format. In that case, good luck, because that shit ain't protobuf.
Maybe you mean something besides the wire format. In that case, good luck, because that shit ain't protobuf.
The wire format is fairly straightforward if you've seen a few binary encodings. The language used to write the schemas isn't quite as simple and regular as you might hope, though.
> Maybe you mean something besides the wire format. In that case, good luck, because that shit ain't protobuf.
Naming's hard :) Being really pedantic, I think even Google calls the schema description language "Protocol Buffers" and uses phrases like "the Protobuf binary format" or "the Protocol Buffer wire format" to refer to the wire format. Colloquially, it's never confused me to just use "Protobuf" for both.
> Maybe you mean something besides the wire format. In that case, good luck, because that shit ain't protobuf.
Naming's hard :) Being really pedantic, I think even Google calls the schema description language "Protocol Buffers" and uses phrases like "the Protobuf binary format" or "the Protocol Buffer wire format" to refer to the wire format. Colloquially, it's never confused me to just use "Protobuf" for both.
Except I've written thousands of lines of protobuf format handling that never, or only extremely distantly, touch a schema file. But there's no reason you'd ever do the inverse, pushing protobuf schemas around with no intent to handle the wire format. As an abstract data definition format it's exceptionally poor, it only makes sense if you also want to use the wire format (which is... better than poor, especially as the commodity ones go).
> But there's no reason you'd ever do the inverse, pushing protobuf schemas around with no intent to handle the wire format.
You could be writing a linter, a formatter, an implementation of the Language Server Protocol, a compiler that's not protoc, a way to apply semantic patches to large numbers of Protobuf schemas, or any number of other useful tools. There's clearly at least some demand for tools like this - partial implementations of most of these exist, often with some corporate backing.
Unless you're implementing a Protobuf runtime (google.golang.org/protobuf in Go, upb for Python, etc.), your experience seems unusual to me - most developers I've encountered read and write the wire format using one of the existing runtimes.
That said, it does sound like a lot of fun - especially if it's in lisp!
You could be writing a linter, a formatter, an implementation of the Language Server Protocol, a compiler that's not protoc, a way to apply semantic patches to large numbers of Protobuf schemas, or any number of other useful tools. There's clearly at least some demand for tools like this - partial implementations of most of these exist, often with some corporate backing.
Unless you're implementing a Protobuf runtime (google.golang.org/protobuf in Go, upb for Python, etc.), your experience seems unusual to me - most developers I've encountered read and write the wire format using one of the existing runtimes.
That said, it does sound like a lot of fun - especially if it's in lisp!
The wire format isn't great compared to these, particularly in its handling of nested messages. Or rather, non-handling, since you have to encode the sub message to a bytestring. That makes encoders a bit slower (or complex), and you can't look at a message and see the nesting without a schema. Not great.
Or super-great, because decoders can't fail on nested messages they don't parse. This is a perennial problem in the large corpus of JSON we parse at my day job, and it would be fairly awesome to treat all nested objects as thunks to be decoded on access instead of crashing when synchronizing a crapton of JSON-encoded application state.
Funny, I have the opposite view. If you have invalid data, you should fail as early and loudly as possible, not wait until it's decoded later and you have no idea where the faulty data comes from.
I have both views - if you're going to do something with the data yes please reject as soon as possible - but also it sure is nice to be able to make a dumb pipe for either efficiency or architecture reasons sometimes.
> There is still only one implementation that defines what the language actually is, and that is Google's protoc.
From the article, it seems this is not true anymore?
> We've built the [new Buf proto] compiler within the buf CLI to accurately match protoc.
From the article, it seems this is not true anymore?
> We've built the [new Buf proto] compiler within the buf CLI to accurately match protoc.
Our intent with the compiler in Buf is to match protoc as perfectly as we can. We want to instill maximum confidence in our users that Buf is a trustworthy tool and a suitable replacement for protoc. And to do that, we need the behavior to match.
But we do hope that eventually the official definition of the language will be a proper specification, not a particular implementation. (And maybe this document could be the start of that shift.)
So while there are multiple implementations (Buf, Square/Wire, probably others), the protoc implementation is canon.
But we do hope that eventually the official definition of the language will be a proper specification, not a particular implementation. (And maybe this document could be the start of that shift.)
So while there are multiple implementations (Buf, Square/Wire, probably others), the protoc implementation is canon.
Have you thought of creating your own version of the protobuf language, sort of like GNU C? You could have an optional flag to enable it, which would allow you to create your own official specification.
There is a specification: https://developers.google.com/protocol-buffers/docs/referenc....
It took two minutes to find but now I wonder, is there something wrong with it…?
It took two minutes to find but now I wonder, is there something wrong with it…?
I've used the Google docs as a reference to build software that parses '.proto' files and then generate code based on what I found. The docs are... incomplete, ambiguous, and in some places directly contradict the behaviour of protoc. So they're okay as a guide for learning how to write '.proto' files by hand, but a specification they are not.
It's been several years, so I can't point you to specifics. But if you look at open source projects that parse '.proto' files and do anything interesting with the result, you'll probably find lots of comments to the effect of "docs say X, but we are doing Y because that's what protoc does".
It's been several years, so I can't point you to specifics. But if you look at open source projects that parse '.proto' files and do anything interesting with the result, you'll probably find lots of comments to the effect of "docs say X, but we are doing Y because that's what protoc does".
Out of curiosity: why write proto language implementations rather than protoc plugins?
(I don't work at Buf, but happen to be able to answer this) - the post at [1] describes the rationale for wanting something different than the Google compiler.
To my mind I'd rather have something written in Go that I can pull in and version using `go.mod` instead of having to special case a single tool, as well.
[1]: https://docs.buf.build/reference/internal-compiler
To my mind I'd rather have something written in Go that I can pull in and version using `go.mod` instead of having to special case a single tool, as well.
[1]: https://docs.buf.build/reference/internal-compiler
A great question: We do plan to add content about the plugin protocol to this site. While documentation for plugins is light, it is easier to find and to get a working plugin than it is to find the information needed, for example, to write a tool that performs static analysis on a protobuf source files.
The biggest omission in the existing docs was the specification of the language.
Plugins generally require a library for a particular implementation language, so content we write would likely have to focus on a single language and library (at least to start). Whereas a spec is more broadly useful, regardless of what implementation language one is using with Protobuf.
The biggest omission in the existing docs was the specification of the language.
Plugins generally require a library for a particular implementation language, so content we write would likely have to focus on a single language and library (at least to start). Whereas a spec is more broadly useful, regardless of what implementation language one is using with Protobuf.
So that you can analyze and troubleshoot protobuf network traffic? Wireshark has a protobuf parser that integrates with our dissection API:
https://gitlab.com/wireshark/wireshark/-/blob/master/epan/pr...
https://gitlab.com/wireshark/wireshark/-/blob/master/epan/pr...
Hrmm, I don't see why you would need to think about proto files to do this. You can dissect protocol messages on the wire using the descriptor. In fact, I would say that would be a good improvement to the code you just showed me.
> we are standing on the shoulders of giants, those who have built and battle-tested it, and brought it to its current mature state
I would rewrite this maybe to:
> we are making Google's internal problems into everyone's problems
There are benefits to an IDL in the abstract, but an IDL for everyone should be built with the benefit of hindsight looking at the lessons of protobuf, ion, thrift, etc. Not just baking Google's internal backwards compatibility obligations into a formal spec everyone should follow.
I think any time google takes an internal tool and flips the "open source" bit on it, it turns out to be a bad match for the rest of the world. When they instead take the time to build a new system that learns from the internal tool, like Kubernetes learned from Borg, I think the end result is significantly more valuable.
I would rewrite this maybe to:
> we are making Google's internal problems into everyone's problems
There are benefits to an IDL in the abstract, but an IDL for everyone should be built with the benefit of hindsight looking at the lessons of protobuf, ion, thrift, etc. Not just baking Google's internal backwards compatibility obligations into a formal spec everyone should follow.
I think any time google takes an internal tool and flips the "open source" bit on it, it turns out to be a bad match for the rest of the world. When they instead take the time to build a new system that learns from the internal tool, like Kubernetes learned from Borg, I think the end result is significantly more valuable.
I quite like Protobuf definitions. I find them very easy to read and I love the fact I can distribute them to a bunch of different languages via a library.
Are these Google’s internal problems? Or, what google-internal problems do protobufs solve that nobody else needs to care about?
Edit: to your edit, I find it hard to see a different way to do things.
Are these Google’s internal problems? Or, what google-internal problems do protobufs solve that nobody else needs to care about?
Edit: to your edit, I find it hard to see a different way to do things.
A different way: learn from protobufs, then design a new language from the ground up and define a spec very early.
Eh, I've been corrupted by a decade of working with protobufs, but I've ultimately settled on "protos are good enough".
Maybe not perfect, maybe there are use cases and patterns where they totally suck, but they exist and are already being used and they generally get the job done. While a part of me will always want to spend time optimizing and carefully crafting the perfect system from the bits up, there are more interesting things to build on top of your serialization and RPC frameworks so I don't see any real value in spending more time creating a 15th standard.
Maybe not perfect, maybe there are use cases and patterns where they totally suck, but they exist and are already being used and they generally get the job done. While a part of me will always want to spend time optimizing and carefully crafting the perfect system from the bits up, there are more interesting things to build on top of your serialization and RPC frameworks so I don't see any real value in spending more time creating a 15th standard.
>> we are making Google's internal problems into everyone's problems
I think you're right. One example that I thought about the last time Buf was on the front page, but didn't post at the time, was the lesson that Google (and @kentonv with Cap'n Proto) took from Google's specific challenges with required fields in proto2. The Cap'n Proto FAQ [1] includes this:
> Imagine a production environment in which two servers, Alice and Bob, exchange messages through a message bus infrastructure running on a big corporate network. The message bus parses each message just to examine the envelope and decide how to route it, without paying attention to any other content. Often, messages from various applications are batched together and then split up again downstream.
> Now, at some point, Alice’s developers decide that one of the fields in a deeply-nested message commonly sent to Bob has become obsolete. To clean things up, they decide to remove it, so they change the field from “required” to “optional”. The developers aren’t idiots, so they realize that Bob needs to be updated as well. They make the changes to Bob, and just to be thorough they run an integration test with Alice and Bob running in a test environment. The test environment is always running the latest build of the message bus, but that’s irrelevant anyway because the message bus doesn’t actually care about message contents; it only does routing. Protocols are modified all the time without updating the message bus.
> Satisfied with their testing, the devs push a new version of Alice to prod. Immediately, everything breaks. And by “everything” I don’t just mean Alice and Bob. Completely unrelated servers are getting strange errors or failing to receive messages. The whole data center has ground to a halt and the sysadmins are running around with their hair on fire.
> What happened? Well, the message bus running in prod was still an older build from before the protocol change. And even though the message bus doesn’t care about message content, it does need to parse every message just to read the envelope. And the protobuf parser checks the entire message for missing required fields. So when Alice stopped sending that newly-optional field, the whole message failed to parse, envelope and all. And to make matters worse, any other messages that happened to be in the same batch also failed to parse, causing errors in seemingly-unrelated systems that share the bus.
How common is this type of message bus outside of a very big, very custom infrastructure like Google's? Also, I think the right lesson to take from this would be that the message bodies passing through the bus should be opaque; they should probably be defined as byte arrays in the schema for the message envelope. The message bus shouldn't parse or validate those bodies any more than an IP router is supposed to parse or validate the bodies of IP packets (leaving aside, for the moment, stateful packet inspection).
And so the power of Protobuf and Cap'n Proto schemas, particularly the power to validate messages which is supposed to be a primary advantage of strong static typing, is limited because of a specific issue with Google's infrastructure, from which I think they took the wrong lesson.
[1]: https://capnproto.org/faq.html
I think you're right. One example that I thought about the last time Buf was on the front page, but didn't post at the time, was the lesson that Google (and @kentonv with Cap'n Proto) took from Google's specific challenges with required fields in proto2. The Cap'n Proto FAQ [1] includes this:
> Imagine a production environment in which two servers, Alice and Bob, exchange messages through a message bus infrastructure running on a big corporate network. The message bus parses each message just to examine the envelope and decide how to route it, without paying attention to any other content. Often, messages from various applications are batched together and then split up again downstream.
> Now, at some point, Alice’s developers decide that one of the fields in a deeply-nested message commonly sent to Bob has become obsolete. To clean things up, they decide to remove it, so they change the field from “required” to “optional”. The developers aren’t idiots, so they realize that Bob needs to be updated as well. They make the changes to Bob, and just to be thorough they run an integration test with Alice and Bob running in a test environment. The test environment is always running the latest build of the message bus, but that’s irrelevant anyway because the message bus doesn’t actually care about message contents; it only does routing. Protocols are modified all the time without updating the message bus.
> Satisfied with their testing, the devs push a new version of Alice to prod. Immediately, everything breaks. And by “everything” I don’t just mean Alice and Bob. Completely unrelated servers are getting strange errors or failing to receive messages. The whole data center has ground to a halt and the sysadmins are running around with their hair on fire.
> What happened? Well, the message bus running in prod was still an older build from before the protocol change. And even though the message bus doesn’t care about message content, it does need to parse every message just to read the envelope. And the protobuf parser checks the entire message for missing required fields. So when Alice stopped sending that newly-optional field, the whole message failed to parse, envelope and all. And to make matters worse, any other messages that happened to be in the same batch also failed to parse, causing errors in seemingly-unrelated systems that share the bus.
How common is this type of message bus outside of a very big, very custom infrastructure like Google's? Also, I think the right lesson to take from this would be that the message bodies passing through the bus should be opaque; they should probably be defined as byte arrays in the schema for the message envelope. The message bus shouldn't parse or validate those bodies any more than an IP router is supposed to parse or validate the bodies of IP packets (leaving aside, for the moment, stateful packet inspection).
And so the power of Protobuf and Cap'n Proto schemas, particularly the power to validate messages which is supposed to be a primary advantage of strong static typing, is limited because of a specific issue with Google's infrastructure, from which I think they took the wrong lesson.
[1]: https://capnproto.org/faq.html
The latest version of protobuf does support passing opaque messages [1] and disallows required fields [2]. So I guess lessons were learned.
1: https://developers.google.com/protocol-buffers/docs/referenc...
2: https://stackoverflow.com/questions/31801257/why-required-an...
1: https://developers.google.com/protocol-buffers/docs/referenc...
2: https://stackoverflow.com/questions/31801257/why-required-an...
Yes, and I'm disagreeing with the second one.
Required fields really are a terrible idea. Having the validation performed in code you don't control (generated) and that is pervasive (middle services that you don't control) is a nightmare for API evolution.
If something is truly required, you're better off writing the validation logic yourself, so that you can change it when your requirements change.
If something is truly required, you're better off writing the validation logic yourself, so that you can change it when your requirements change.
> If something is truly required, you're better off writing the validation logic yourself
Validation logic is often tricky, or at least tedious and error-prone (in part due to its tediousness). I would much rather have a battle-tested piece of software generate validation code for me.
I don't really get this aversion to required fields. Some data is actually required, and this happens fairly often in APIs in my experience. And if that changes, then often you are fundamentally changing what the API does, so it should be a new API.
Validation logic is often tricky, or at least tedious and error-prone (in part due to its tediousness). I would much rather have a battle-tested piece of software generate validation code for me.
I don't really get this aversion to required fields. Some data is actually required, and this happens fairly often in APIs in my experience. And if that changes, then often you are fundamentally changing what the API does, so it should be a new API.
What you are now arguing for is that every protobuf field addition or removal needs to be accompanied by a version bump of the underlying message, or in other words proto schema definitions are write-once, because if you add or remove a field, you're fundamentally changing what the api does too. This is, I guess, a technically valid approach to the problem, but there's a reason no one does it.
> Some data is actually required, and this happens fairly often in APIs in my experience. And if that changes, then often you are fundamentally changing what the API does, so it should be a new API.
I can think of some fabulous counterexamples to this that aren't really public, but suffice to say that there are lots of places in code with
> Some data is actually required, and this happens fairly often in APIs in my experience. And if that changes, then often you are fundamentally changing what the API does, so it should be a new API.
I can think of some fabulous counterexamples to this that aren't really public, but suffice to say that there are lots of places in code with
msg.foo = "" // This field is required, but has been deprecated in
// favor of msg.bar for <efficiency/external change/time>
// reasons.
Ah, got it. Well as far as I know, proto2 (with required) isn't going anywhere and Any can still be used with it. One nice thing about protobuf is its extensibility, so you can still get required fields with something like protoc-gen-validate, but interpret them closer to the app's logic.
https://github.com/envoyproxy/protoc-gen-validate
https://github.com/envoyproxy/protoc-gen-validate
> How common is this type of message bus outside of a very big
Cloudflare is a pretty big one. There are others, but you're forgetting the much more common and pernicious problem: a proto serialized in a database. The reader, writer, and database itself need to be synced across time as the database takes the shape of a message bus that stores the proto, potentially forever, so now to make something un-required, you have to first do an in place column type change on a database, as opposed to a 3-phase column migration, which is a safe thing everyone adores undertaking.
Just don't use required fields, they're bad. Learn from Google's mistakes!
> Also, I think the right lesson to take from this would be that the message bodies passing through the bus should be opaque
Proto has support for this (in varying forms, there's the `bytes` type and the `Any` type), but these are distinct from required, so if you want to keep required, you need to make sure that the [every] message bus wraps all clients in any, which has, if I recall, a nontrivial performance penalty.
Cloudflare is a pretty big one. There are others, but you're forgetting the much more common and pernicious problem: a proto serialized in a database. The reader, writer, and database itself need to be synced across time as the database takes the shape of a message bus that stores the proto, potentially forever, so now to make something un-required, you have to first do an in place column type change on a database, as opposed to a 3-phase column migration, which is a safe thing everyone adores undertaking.
Just don't use required fields, they're bad. Learn from Google's mistakes!
> Also, I think the right lesson to take from this would be that the message bodies passing through the bus should be opaque
Proto has support for this (in varying forms, there's the `bytes` type and the `Any` type), but these are distinct from required, so if you want to keep required, you need to make sure that the [every] message bus wraps all clients in any, which has, if I recall, a nontrivial performance penalty.
I kind of wish I didn't say "message bus" in that example, that sounds too enterprise-y.
In practice the use case I had in mind was the Google Search pipeline. Simplifying quite a bit, there is (or was, 15 years ago) a front-end server (mostly renders HTML), several back-end indexes (serving different corpora, like web, images, maps, etc.), and a middle part that fetches data from all the back-ends and serves them up to the front-end.
The middle part is where the problems would sometimes appear. A back-end index would be updated to stop filling in some field of a search result which had previously been required. They would dutifully update the front-end code that depended on the field before changing the back-end to stop sending it. But if they knew the middle part didn't use the field, then they often believed they didn't need to wait for the middle part to do a release -- forgetting that the `required` part is enforced everywhere.
The middle part wasn't a dumb message broker, it would actually inspect the data too, for a variety of reasons. Many ranking algorithms were implemented in the middle part and they might want to inspect pretty much any part of the information attached to search results. So your idea that the middle part should just treat all the "payload" as opaque wouldn't have worked -- it would have been hard to draw the line between what should be opaque and what shouldn't be, and impossible to keep it updated as requirements changed. The whole exercise would have just been a huge burden.
No, this situation is not at all unique to Google. It is, in fact, quite common in any arrangement that involves three or more independently-managed components.
> And so the power of Protobuf and Cap'n Proto schemas, particularly the power to validate messages which is supposed to be a primary advantage of strong static typing, is limited
I would argue that a much more important power of these systems is their ability to manage change. It's extremely hard to fully test what happens when two different versions of your code interact. Almost all automated test setups only test the current version of the code at HEAD; they rarely attempt to set up old versions of some services. Even if they do, how many combinations do you test?
The genius of Protobuf (which Cap'n Proto mostly copied) is that it's pretty easy to reason logically about how a change will interact with older versions of the code. You add a new field, you know old code will just ignore it, and new code receiving data from old code will fill in the default value. It's rare -- surprisingly rare, to be perfectly honest! -- that people can't reason through this in their head.
The problem with `required` was that the implications of changing it were not intuitive, because it's not intuitive that it had implications for servers that don't even inspect the field in question.
The lesson for me is that validation has to happen at the point of consumption. You should not try to validate your data early in the pipeline because your validation rules will likely become out-of-sync with what the consumer actually wants over time. The only way to maintain sync is for the consumer itself to decide whether the data meets its requirements. This makes a lot of theorists uncomfortable, but I've seen in play out in practice so many times...
In practice the use case I had in mind was the Google Search pipeline. Simplifying quite a bit, there is (or was, 15 years ago) a front-end server (mostly renders HTML), several back-end indexes (serving different corpora, like web, images, maps, etc.), and a middle part that fetches data from all the back-ends and serves them up to the front-end.
The middle part is where the problems would sometimes appear. A back-end index would be updated to stop filling in some field of a search result which had previously been required. They would dutifully update the front-end code that depended on the field before changing the back-end to stop sending it. But if they knew the middle part didn't use the field, then they often believed they didn't need to wait for the middle part to do a release -- forgetting that the `required` part is enforced everywhere.
The middle part wasn't a dumb message broker, it would actually inspect the data too, for a variety of reasons. Many ranking algorithms were implemented in the middle part and they might want to inspect pretty much any part of the information attached to search results. So your idea that the middle part should just treat all the "payload" as opaque wouldn't have worked -- it would have been hard to draw the line between what should be opaque and what shouldn't be, and impossible to keep it updated as requirements changed. The whole exercise would have just been a huge burden.
No, this situation is not at all unique to Google. It is, in fact, quite common in any arrangement that involves three or more independently-managed components.
> And so the power of Protobuf and Cap'n Proto schemas, particularly the power to validate messages which is supposed to be a primary advantage of strong static typing, is limited
I would argue that a much more important power of these systems is their ability to manage change. It's extremely hard to fully test what happens when two different versions of your code interact. Almost all automated test setups only test the current version of the code at HEAD; they rarely attempt to set up old versions of some services. Even if they do, how many combinations do you test?
The genius of Protobuf (which Cap'n Proto mostly copied) is that it's pretty easy to reason logically about how a change will interact with older versions of the code. You add a new field, you know old code will just ignore it, and new code receiving data from old code will fill in the default value. It's rare -- surprisingly rare, to be perfectly honest! -- that people can't reason through this in their head.
The problem with `required` was that the implications of changing it were not intuitive, because it's not intuitive that it had implications for servers that don't even inspect the field in question.
The lesson for me is that validation has to happen at the point of consumption. You should not try to validate your data early in the pipeline because your validation rules will likely become out-of-sync with what the consumer actually wants over time. The only way to maintain sync is for the consumer itself to decide whether the data meets its requirements. This makes a lot of theorists uncomfortable, but I've seen in play out in practice so many times...
> This makes a lot of theorists uncomfortable, but I've seen in play out in practice so many times...
I believe you, and I'm sorry I was quick to criticize. Thanks for taking time to clarify.
I believe you, and I'm sorry I was quick to criticize. Thanks for taking time to clarify.
This seems like a great resource. Kudos.
> But most of them are based on the incomplete specs from Google's developer site. None of them can correctly predict what source files protoc will actually accept or reject 100% of the time.
I'd like to think I got pretty close with the plugin that now ships with IntelliJ. It even supports the 65-bit integer literal [1] that protoc happens to accept for proto2-style float and double default values.
With this as a starting point, it'd be nice to fix some of the pecularities that arise from "implementation as spec", such as that literal value, and the fact that colon optionality in text format is based on value type, not syntax.
1: https://github.com/jvolkman/intellij-protobuf-editor/blob/6e...
> But most of them are based on the incomplete specs from Google's developer site. None of them can correctly predict what source files protoc will actually accept or reject 100% of the time.
I'd like to think I got pretty close with the plugin that now ships with IntelliJ. It even supports the 65-bit integer literal [1] that protoc happens to accept for proto2-style float and double default values.
With this as a starting point, it'd be nice to fix some of the pecularities that arise from "implementation as spec", such as that literal value, and the fact that colon optionality in text format is based on value type, not syntax.
1: https://github.com/jvolkman/intellij-protobuf-editor/blob/6e...
Out of curiosity, why does Protobuf allow a negative 64-bit value in the first place? No CPU architecture supports such as far as I know.
It's been 15 years since I wrote this so I don't remember exactly, but I think it's just an implementation quirk. See:
https://github.com/protocolbuffers/protobuf/blob/main/src/go...
The parser consumes a "-", then consumes a number. The number can be any 64-bit unsigned integer. It is then converted to a double. Finally, if a "-" was seen earlier, it is negated. So by accident, it ends up allowing the range of a 65-bit signed integer.
https://github.com/protocolbuffers/protobuf/blob/main/src/go...
The parser consumes a "-", then consumes a number. The number can be any 64-bit unsigned integer. It is then converted to a double. Finally, if a "-" was seen earlier, it is negated. So by accident, it ends up allowing the range of a 65-bit signed integer.
It's just a long-standing quirk in the parser. It parses the numeric part as an unsigned 64-bit number, then applies the sign afterwards. And the result can be approximately stuffed into a floating point value.
The behavior for integer fields is different; compilation will fail with an out of range error.
The behavior for integer fields is different; compilation will fail with an out of range error.
> Protobuf is the most stable and widely adopted IDL today
I've run into this misconception so many times over the last decade. Protobuf is much less than an IDL (intentionally so). It's used to describe the data of an interface but is completely unopinionated about all other aspects of an IDL. GRPC is a great example of how to use ProtoBuf in an IDL, but it could be used for other categories of interfaces (object oriented, etc). People treating ProtoBuf as an IDL make the mistake of concentrating too much on the data format and not about (imho) more important aspects like pre and postconditions etc, that make an interface an interface.
I've run into this misconception so many times over the last decade. Protobuf is much less than an IDL (intentionally so). It's used to describe the data of an interface but is completely unopinionated about all other aspects of an IDL. GRPC is a great example of how to use ProtoBuf in an IDL, but it could be used for other categories of interfaces (object oriented, etc). People treating ProtoBuf as an IDL make the mistake of concentrating too much on the data format and not about (imho) more important aspects like pre and postconditions etc, that make an interface an interface.
Lately I've been reading up on Amazon's Ion format and think it's underappreciated. It's got a decent spec for both binary and text serialization formats, under an Apache-2.0 license even. Maybe I'm weird but I just don't like using serialization formats that don't include a specification for the binary format.
I also like where Ion goes with it's symbol tables. They provide much of the space efficiency that Protobufs provide with schemas, but gives you much more freedom and flexibility. I'm a longtime fan of msgpack but it becomes inefficient if you transport field names with it. The Ion schema also seems nice as an alternative to Protobuf schemas, but able to describe arbitrary data constraints too.
I also like where Ion goes with it's symbol tables. They provide much of the space efficiency that Protobufs provide with schemas, but gives you much more freedom and flexibility. I'm a longtime fan of msgpack but it becomes inefficient if you transport field names with it. The Ion schema also seems nice as an alternative to Protobuf schemas, but able to describe arbitrary data constraints too.
(removing my unfair characterization)
A large corporate sponsor that has done a terrible job of shepherding the protocol, maintaining docs, etc. I'm all for a community push to divorce the protobuf specification/implementation from Google and to have it be much more community maintained, as it's clear that Google doesn't seem to care to.
Google actually requested that the community contribute to a real specification.
https://github.com/protocolbuffers/protobuf/issues/6188#issu...
So we've taken the initiative. No rent-seeking.
So we've taken the initiative. No rent-seeking.
bufdev brought up some good concerns on that issue:
> This as a community project defeats the purpose of creating a definitive grammar/spec that protocolbuffers/protobuf holds itself to - without that component, having a community-written spec is no improvement over the current situation from what I can see, and in fact could create even more fragmentation and confusion, just my two cents.
It looks like the protobuf team created a repo intended for this (https://github.com/protocolbuffers/protobuf-grammar), perhaps the spec could be contributed there, including a review/blessing from the official team.
> This as a community project defeats the purpose of creating a definitive grammar/spec that protocolbuffers/protobuf holds itself to - without that component, having a community-written spec is no improvement over the current situation from what I can see, and in fact could create even more fragmentation and confusion, just my two cents.
It looks like the protobuf team created a repo intended for this (https://github.com/protocolbuffers/protobuf-grammar), perhaps the spec could be contributed there, including a review/blessing from the official team.
Here is protobuf parser I wrote for a client w/ ragel/c++. We needed way to parse the proto specification to generate our own code, since the google's own C++ runtimes were way too huge for our use case especially with reflection. The nanopb in addition to the code generation w/ this parser was used in the end to keep the binary bloat in check.
https://gist.github.com/Cloudef/7a69eba054f61ee63d6ee59262e1...
I also did this benchmark for cap'n'proto, flatbuffers, protobuf and raw structs. This is already years old so it might not be accurate anymore, but what surprised me most was how different flatbuffers was compared to raw struct, as the design sounds like it should be merely extension to what is essentially raw struct + making sure the data can be portably encoded and decoded.
https://cloudef.pw/protobug.png
https://gist.github.com/Cloudef/7a69eba054f61ee63d6ee59262e1...
I also did this benchmark for cap'n'proto, flatbuffers, protobuf and raw structs. This is already years old so it might not be accurate anymore, but what surprised me most was how different flatbuffers was compared to raw struct, as the design sounds like it should be merely extension to what is essentially raw struct + making sure the data can be portably encoded and decoded.
https://cloudef.pw/protobug.png
Question: if there’s no ebnf protobuf spec, what’s this then? https://developers.google.com/protocol-buffers/docs/referenc...
The text at the top of the page says:
> This is a language specification reference for version 3 of the Protocol Buffers language (proto3). The syntax is specified using Extended Backus-Naur Form (EBNF) …
The text at the top of the page says:
> This is a language specification reference for version 3 of the Protocol Buffers language (proto3). The syntax is specified using Extended Backus-Naur Form (EBNF) …
It appears Google has released a spec as of 11 days ago: https://github.com/protocolbuffers/protobuf/issues/6188#issu...
That is for the text format, which is a serialized representation of Protobuf data. As they specify in the linked doc, it is not the format for the actual language:
> This format is distinct from the format of text within a .proto schema.
> This format is distinct from the format of text within a .proto schema.
Is there a license on this spec?
Never having had the opportunity to put protobuf into action but having some interest I've had these questions:
1) What would you say is the best use case
2) and what unfortunate misusecases have you come across
1) What would you say is the best use case
2) and what unfortunate misusecases have you come across
I do most of my work with protos in Java, and it's nice to have a schema like this that will build a bunch of immutable POJOs with builder classes and enough infrastructure to be able to do some interesting reflection-style stuff on top of the serialization / deserialization.
The wide variety of client languages is really nice. I'm fairly certain that I can parse a proto in any language I'm ever going to use.
The binary wire format is fairly straightforward, and is pretty tight without using compression. Fields are byte-aligned, and if you wanted to generate a binary proto message by concatenating a few things together it isn't very hard. And then you can use your proto definitions in whatever language you want to parse it. You can even parse your proto definition into a proto (Google provides the proto proto definitions, I think I got that grammar right) and write tools that generate whatever code you want easily.
I think the text format is under-utilized. It's my go-to for configuration files. You create the proto definitions with whatever structures you want to structure your config settings, and then use the official parser to parse a text file. It supports comments (I'm throwing shade at you, JSON), and is simpler than YAML, while adding that structure. You can also use command line tools to validate the file as a pre-commit, or translate it into a binary format if you don't want to rely on the text format.
The wide variety of client languages is really nice. I'm fairly certain that I can parse a proto in any language I'm ever going to use.
The binary wire format is fairly straightforward, and is pretty tight without using compression. Fields are byte-aligned, and if you wanted to generate a binary proto message by concatenating a few things together it isn't very hard. And then you can use your proto definitions in whatever language you want to parse it. You can even parse your proto definition into a proto (Google provides the proto proto definitions, I think I got that grammar right) and write tools that generate whatever code you want easily.
I think the text format is under-utilized. It's my go-to for configuration files. You create the proto definitions with whatever structures you want to structure your config settings, and then use the official parser to parse a text file. It supports comments (I'm throwing shade at you, JSON), and is simpler than YAML, while adding that structure. You can also use command line tools to validate the file as a pre-commit, or translate it into a binary format if you don't want to rely on the text format.
The key "killer" use case is for describing RPC schemas. By describing the schema in an IDL, you can then generate client stubs and server interfaces in a variety of implementation languages, allowing interop between heterogenous clients and servers.
They are also useful for describing domain models. This isn't surprising since domain models usually find their way into RPC schemas (since RPCs will often query or define model data). But they can also be used in other cases, such as for persistence and structured logging.
Some misuse I have seen involves trying to make a protobuf model the _only_ representation of a domain model: it is almost inevitable that a physical model (a representation of your data in a SQL database, for example) will need to vary from a logical model, and trying to make a single protobuf representation serve double duty can be a source of problems. Making protobuf schemas conform to physical storage constraints often makes for worse abstraction. The model becomes de-normalized (which can make constraints and relationships harder to model/enforce), and can even leak details that consumers shouldn't know about or care about.
Another misuse is using it for data that never leaves a process -- a program's private, internal state. If a data structure never needs to be serialized (to persist or send to another process), then you're better off using native data structures in the implementation language (which generally have far greater flexibility/expressibility as well as better performance).
They are also useful for describing domain models. This isn't surprising since domain models usually find their way into RPC schemas (since RPCs will often query or define model data). But they can also be used in other cases, such as for persistence and structured logging.
Some misuse I have seen involves trying to make a protobuf model the _only_ representation of a domain model: it is almost inevitable that a physical model (a representation of your data in a SQL database, for example) will need to vary from a logical model, and trying to make a single protobuf representation serve double duty can be a source of problems. Making protobuf schemas conform to physical storage constraints often makes for worse abstraction. The model becomes de-normalized (which can make constraints and relationships harder to model/enforce), and can even leak details that consumers shouldn't know about or care about.
Another misuse is using it for data that never leaves a process -- a program's private, internal state. If a data structure never needs to be serialized (to persist or send to another process), then you're better off using native data structures in the implementation language (which generally have far greater flexibility/expressibility as well as better performance).
1) Serialization for storage or transit
2) Serialization for storage or transit
Unless the spec has changed, there's a fun issue. In protobuf there's no notion of null. So for instance a null list and an empty list are serialized identically. This leads to a nightmare in any system where null can have meaning by itself. I was always confused by this decision since there would be an endless number of ways to represent null that would have near zero impact on size or performance. The only cost I can see is the abstract one of making the representation less "elegant."
There are a of course endless ways to overcome this issue, but I don't think serialization implementation details should ever be allowed to leak into the model of what's being serialized. Just begging for issues down the road.
2) Serialization for storage or transit
Unless the spec has changed, there's a fun issue. In protobuf there's no notion of null. So for instance a null list and an empty list are serialized identically. This leads to a nightmare in any system where null can have meaning by itself. I was always confused by this decision since there would be an endless number of ways to represent null that would have near zero impact on size or performance. The only cost I can see is the abstract one of making the representation less "elegant."
There are a of course endless ways to overcome this issue, but I don't think serialization implementation details should ever be allowed to leak into the model of what's being serialized. Just begging for issues down the road.
This is true for proto3, but not for proto2 (except for repeated fields, which are always non-null but can contain 0 items).
Proto2 has `HasFoo` methods, which distinguish between empty (unset/null) and set explicitly to zero fields.
I don't like that proto3 made this choice, I think it was wrong, but there was some rationale behind it beyond just "elegance".
Proto2 has `HasFoo` methods, which distinguish between empty (unset/null) and set explicitly to zero fields.
I don't like that proto3 made this choice, I think it was wrong, but there was some rationale behind it beyond just "elegance".
Can you elaborate on the reasons for the decision? I've always been quite curious on the reasoning there.
A good reference. I don't think it was really needed in the same way that e.g. a JSON or C++ spec was, since the language is so simple there's not much room for ambiguity. Definitely nice to have anyway.
It is a simple language, being just an IDL (no expressions, no logic, no state, no memory model, etc).
However, you might be surprised about the room for ambiguity. For example, there are several mistakes in the grammars on the official developer site. And there is no place on the developer site that, for example, clearly explains how option names are formulated and interpreted. Even the way that relative references are resolved is not thoroughly described; it is probably the most complicated part of the spec because coherence/consistency wasn't keenly considered when the reference implementation in protoc was devised (https://www.protobuf.com/docs/language-spec#relative-referen...).
So if you wanted to write a tool for the language, one that could correctly parse and understand a source file, without the details in this spec that tool will almost certainly be incorrect and disagree with how protoc parses and understands the same source.
However, you might be surprised about the room for ambiguity. For example, there are several mistakes in the grammars on the official developer site. And there is no place on the developer site that, for example, clearly explains how option names are formulated and interpreted. Even the way that relative references are resolved is not thoroughly described; it is probably the most complicated part of the spec because coherence/consistency wasn't keenly considered when the reference implementation in protoc was devised (https://www.protobuf.com/docs/language-spec#relative-referen...).
So if you wanted to write a tool for the language, one that could correctly parse and understand a source file, without the details in this spec that tool will almost certainly be incorrect and disagree with how protoc parses and understands the same source.
It does make me hope that some protobuf libraries will integrate their own compilers. The official one can be a bit of a pain to install. This will definitely make that easier!
Paging kentonv for his thoughts on this
oh .. cool .. pricing for protobuf
Golang is an example of a language defined by a spec not an implementation? Discuss.
I see your point, but the implementers of Go do pay a lot more attention than most "defined by an implementation" languages to specifying what they're doing before they do it. And if you find a difference between the specification and the implementation, generally the specification will prevail.
This is certainly The Buf Language Specification, which is useful in itself. Specs are like contracts. If I were to build a tool to be compatible with Buf, I would definitely aim it to work with this spec.
The problem is that the Protobuf project simply didn't sign this contract. Whatever it says is, sorry for the choice of word, a bit pointless if I'm trying to build a tool compatible with Protobuf, specially around forward compatibility.
The industry does need better tooling around protobuf/efficient RPC, and being dependent on a single company (i.e. Google) is definitely not healthy. I hope you guys succeed in what your are trying to doing.