Unexpected security footguns in Go's parsers(blog.trailofbits.com)
blog.trailofbits.com
Unexpected security footguns in Go's parsers
https://blog.trailofbits.com/2025/06/17/unexpected-security-footguns-in-gos-parsers/
138 comments
This is not unavoidable in typescript at all. It really depends a lot on how you have structured your application, but it's basically standard practice at this point to use e.g. zod or similar to parse data at the boundaries. You may have to be careful here (remember to use zod's .strict, for example), but it's absolute not unavoidable.
I should have been clear. Yes it is avoidable but only by adding more checking machinery. The bare language doesnt help.
In Go etc. If a struct doesn't a field foo then there will not be a field foo at runtime. In JS there might be. Unless you bring in libraries to help prevent it.
You are relying on someone remembering to use zod on every fetch
In Go etc. If a struct doesn't a field foo then there will not be a field foo at runtime. In JS there might be. Unless you bring in libraries to help prevent it.
You are relying on someone remembering to use zod on every fetch
OpenAPI can do this too
huh? does Java even support marshall/unmarshall json in the 'bare' language?
Does the origin of "DTO" come from Java? I've never seen it used anywhere else.
DTO comes from OOP for which Java is the poster child.
But you can find it many places.
https://codewithstyle.info/typescript-dto/
But you can find it many places.
https://codewithstyle.info/typescript-dto/
Why is Java the poster child of OO when c++ was literally c with classes
The poster child for statically typed OO is actually Simula 67. And Java is basically Simula with C-style syntax and packages.
1. Java was made OO from ground up. No pointer arithmetic, all code is encapsulated in classes, etc.
2. C++ has some unusual OO features -- friends, multiple-inheritance.
3. Most importantly, Java is significantly more approachable than C++ due to automatic memory management.
2. C++ has some unusual OO features -- friends, multiple-inheritance.
3. Most importantly, Java is significantly more approachable than C++ due to automatic memory management.
It’s worth noting that if you DisallowUnknownFields it makes it much harder to handle forward/backward compatible API changes - which is a very common and usually desirable pattern
While this is a common view, recently I’ve begun to wonder if it may be secretly an antipattern. I’ve run into a number of cases over the years where additional fields don’t break parsing, or even necessarily the main functionality of a program, but result in subtle incorrect behavior in edge cases. Things like values that are actually distinct being treated as equal because the fields that differ are ignored. More recently, I’ve seen LLMs get confused because they hallucinated tool input fields that were ignored during the invocation of a tool.
I’m a little curious to try and build an API where parsing must be exact, and changes always result in a new version of the API. I don’t actually think it would be too difficult, but perhaps some extra tooling around downgrading responses and deprecating old versions may need to be built.
I’m a little curious to try and build an API where parsing must be exact, and changes always result in a new version of the API. I don’t actually think it would be too difficult, but perhaps some extra tooling around downgrading responses and deprecating old versions may need to be built.
It's a convenience and a labor saver, so of course it's fundamentally at odds with security. It's all trade-offs.
If you’re writing a client, I could see this being a problem.
If you’re writing a server, I believe the rule is that any once valid input must stay valid forever, so you just never delete fields. The main benefit of DisallowUnknownFields is that it makes it easier for clients to know when they’ve sent something wrong or useless.
If you’re writing a server, I believe the rule is that any once valid input must stay valid forever, so you just never delete fields. The main benefit of DisallowUnknownFields is that it makes it easier for clients to know when they’ve sent something wrong or useless.
No, once-valid input can be rejected after a period of depreciation.
What actually makes sense is versioning your interfaces (and actually anything you serialize at all), with the version designator being easily accessible without parsing the entire message. (An easy way to have that is to version the endpoint URLs: /api/v1, /api/v2, etc).
For some time, you support two (or more) versions. Eventually you drop the old version if it's problematic. You never have to guess, and can always reject unknown fields.
What actually makes sense is versioning your interfaces (and actually anything you serialize at all), with the version designator being easily accessible without parsing the entire message. (An easy way to have that is to version the endpoint URLs: /api/v1, /api/v2, etc).
For some time, you support two (or more) versions. Eventually you drop the old version if it's problematic. You never have to guess, and can always reject unknown fields.
This would also be easy to do if an API is designed around versioning from the beginning, because often it isn’t and it requires a lot of boilerplate and duplication, and it just results in everything being slapped into v1.
Especially the case in frameworks that prescribe a format for routing.
Especially the case in frameworks that prescribe a format for routing.
This was all very interesting, but that polyglot json/yaml/xml payload was a big surprise to me! I had no idea that go's default xml parser would accept proceeding and trailing garbage. I'd always thought of json as one of the simpler formats to parse, but I suppose the real world would beg to differ.
It's interesting that decisions made about seemingly-innocuous conditions like 'what if there are duplicate keys' have a long tail of consequences
It's interesting that decisions made about seemingly-innocuous conditions like 'what if there are duplicate keys' have a long tail of consequences
Tangent for breaking Python's JSON parser: This has worked for five years. The docs do not say that parsing an invalid piece will result in a RecursionError. They specify JSONDecodeError and UnicodeDecodeError. (There is a RecursionError reference to a key that is off by default - but if its off, we can still raise this...)
#!/bin/sh
# Python will hit it's recursion limit
# If you supply just 4 less than the recursion limit
# I assume this means there's a few objects on the call stack first
# Probably: __main__, print, json.loads, and input.
n="$(python3 -c 'import math; import sys; sys.stdout.write(str(math.floor(sys.getrecursionlimit() - 4)))')"
echo "N: $n"
# Obviously invalid, but unparseable without matching pair
# JSON's grammar is... Not good at being partially parsed.
left="$(yes [ | head -n "$n" | tr -d '\n')"
# Rather than exploding with the expected decodeError
# This will explode with a RecursionError
# Which naturally thrashes the memory cache.
echo "$left" | python3 -c 'import json; print(json.loads(input()))'Reported as <https://github.com/python/cpython/issues/135835>.
I did report this five years ago. And got told that CPython don't consider it a problem. Anything is allowed to raise RecursionError without documentation, in the same way anything is allowed to raise MemoryError.
I don't really agree, as "surprising" is a stench in any API. But it's their project.
I don't really agree, as "surprising" is a stench in any API. But it's their project.
This is why we can't have nice things
> I'd always thought of json as one of the simpler formats to parse, but I suppose the real world would beg to differ
Parsing JSON Is a Minefield (2018) - https://news.ycombinator.com/item?id=40555431 - June, 2024 (56 comments)
et al https://hn.algolia.com/?query=parsing%20json%20is%20a%20mine...
Parsing JSON Is a Minefield (2018) - https://news.ycombinator.com/item?id=40555431 - June, 2024 (56 comments)
et al https://hn.algolia.com/?query=parsing%20json%20is%20a%20mine...
In the case of Attack scenario 2, I do not get why in a secure design you would ever forward the client originating data to the auth service. This is more of a broken best practise then a footgun to me.
The logic should be "Parse, don't validate"[0] and after that you work on those parsed data.
[0]: https://hn.algolia.com/?q=https%3A%2F%2Flexi-lambda.github.i...
The logic should be "Parse, don't validate"[0] and after that you work on those parsed data.
[0]: https://hn.algolia.com/?q=https%3A%2F%2Flexi-lambda.github.i...
See: https://bsky.app/profile/filippo.abyssdomain.expert/post/3le...
that was about a signature wrapping attack in crypto, but it also applies here.
What is "IsAdmin" doing in the "create user" request DTO in the first place? The examples seem to indicate inappropriate re-use of data models.
Would it not be better to:
No need to just have just 1 model that maps 1:1 to your DB row. This applies to all languages
Would it not be better to:
type CreateUserRequest struct {
Username string
Password string
}
type UserView struct {
Username string
IsAdmin boolean
}
No need to just have just 1 model that maps 1:1 to your DB row. This applies to all languages
Yeah its a bad pattern. And we can expect to see it in a sizable number of production codebases.
ORMs love to put people and projects in the situation where it's much easier for data types in your program to match exactly with the tables in your database, but reality often requires them to be decoupled instead. That "create user" request contains "isAdmin" smells like someone mapped that exact request to a table, and just because the table has "isAdmin", the data type now also has it.
Aside from security issues it presents, it’s just a really bad idea to tightly couple your data model to your API like that.
Any change to your DB schema is liable to become a breaking change on your API. If you need separate types for your requests and responses so be it.
Any change to your DB schema is liable to become a breaking change on your API. If you need separate types for your requests and responses so be it.
Exactly. This isn't a flaw in the runtime or the metaphor, the bug here is that the app exposes private data in a public API. That's a mistake that doesn't care about implementation choice. You can make the same blunder with a RESTful interface or a CGI script.
At most, you can argue that simple serialization libraries (Go's is indeed one of the best) make it more tempting to "just send the data" in such a design, so if you squint really (really) hard, you can call this a "footgun" I guess.
But the rest of the headline is 100% nonsense. This is not about "Go" or "parsers". At all.
At most, you can argue that simple serialization libraries (Go's is indeed one of the best) make it more tempting to "just send the data" in such a design, so if you squint really (really) hard, you can call this a "footgun" I guess.
But the rest of the headline is 100% nonsense. This is not about "Go" or "parsers". At all.
Except in the case of encoding/xml garbage, that's really a limitation of the Go parser. It's a very non-strict parser and doesn't actually validate that the input conforms to XML. Think of it more as a query language for a stream of XML fragments.
What? It's about parsers in Go's standard library.
A "parser" is the piece of software that translates unstructured input (usually text) into structured output that better reflects the runtime of a programming language. A "security bug" in a parser is normally construed to be the existence of an input that causes the software to do something incorrect/undocumented/unexpected.
Nothing in the article discusses a parser or anything like a parser bug.
The article doesn't like that the semantics of the user-facing API wrapped around the parser is, I guess, "easy to make mistakes with". That's an article about API design, at most. But that's boring and specious and doesn't grab clicks, so they want you to think that Go's parsers are insecure instead.
Nothing in the article discusses a parser or anything like a parser bug.
The article doesn't like that the semantics of the user-facing API wrapped around the parser is, I guess, "easy to make mistakes with". That's an article about API design, at most. But that's boring and specious and doesn't grab clicks, so they want you to think that Go's parsers are insecure instead.
I would consider the case-insensitive key handling in Go’s JSON parser and the malleability in the XML parser to both be “classic” examples of differential-prone behavior, which is the ultimate point made by the article.
(This is why the more formal definition of a parser is useful to ground on: a parser is a type of recognizer, and disagreements between recognizers that claim to recognize the same thing can be exploited. This doesn’t require a bug per se, only difference, which is why it’s a footgun.)
(This is why the more formal definition of a parser is useful to ground on: a parser is a type of recognizer, and disagreements between recognizers that claim to recognize the same thing can be exploited. This doesn’t require a bug per se, only difference, which is why it’s a footgun.)
> case-insensitive key handling in Go’s JSON parser
This was an explicit decision for convenience, because the Go struct fields will be Capitalized to export them but JSON tends to follow a lower case convention.
This was an explicit decision for convenience, because the Go struct fields will be Capitalized to export them but JSON tends to follow a lower case convention.
Just because it’s convenient doesn’t make it a good idea! It’s also not what Go’s other parsers do per the rest of the post.
The "other parsers" in stdlib are gob and encoding/xml.
Gob is a format made for Go, so it does not have a field naming convention mismatch; what's on wire is the Go convention.
encoding/xml isn't really structured as a "convert between equivalent-shaped structs and a message" utility, its struct tags are more like a query language, like a simpler cousin of XPath. Most real-world code using it does things like
Gob is a format made for Go, so it does not have a field naming convention mismatch; what's on wire is the Go convention.
encoding/xml isn't really structured as a "convert between equivalent-shaped structs and a message" utility, its struct tags are more like a query language, like a simpler cousin of XPath. Most real-world code using it does things like
type Person struct {
FirstName string `xml:"name>first"`
LastName string `xml:"name>last"`
}
{"email": "[email protected]", "name": "John Doe"}
type Person struct {
Email string
Name string
}Again, I understand the rationale. I just don’t think it was a good idea.
> Nothing in the article discusses a parser or anything like a parser bug.
And it doesn't claim to. The article is titled "footguns" not "bugs". A footgun is just something that is easy to misuse due to unintuitive or unexpected behavior.
And it doesn't claim to. The article is titled "footguns" not "bugs". A footgun is just something that is easy to misuse due to unintuitive or unexpected behavior.
> And it doesn't claim to.
Yes it does. The title is literally "Unexpected security footguns in Go's parsers". The article didn't identify a single footgun. This is just bad design.
Yes it does. The title is literally "Unexpected security footguns in Go's parsers". The article didn't identify a single footgun. This is just bad design.
The article links to CVEs directly caused by some of the less intuitive behavior here. I feel like that’s sufficient to qualify as a footgun
That title is clearly constructed to imply that Go's parsers are insecure. The text of the article isn't about parsers at all, and only tangentially about Go[1].
It's deliberately misleading clickbait. You know it. I know it. We all know it.
If you want to have a considered discussion about pitfalls with the use of automatic serialization paradigms across trust boundaries, I'm here for it. If you just want to flame about Go, get better source material. This one isn't the hill to stand on.
[1] Which, again, has a really first rate serialization story; but not one fundamentally different from any of a zillion others. Cooking data from untrusted sources is just plain hard, and not something that anyone (much less the author of this awful blog post) is going to solve with a serialization API.
It's deliberately misleading clickbait. You know it. I know it. We all know it.
If you want to have a considered discussion about pitfalls with the use of automatic serialization paradigms across trust boundaries, I'm here for it. If you just want to flame about Go, get better source material. This one isn't the hill to stand on.
[1] Which, again, has a really first rate serialization story; but not one fundamentally different from any of a zillion others. Cooking data from untrusted sources is just plain hard, and not something that anyone (much less the author of this awful blog post) is going to solve with a serialization API.
I think you and I read a different article. For example, about treating JSON keys without case sensitivity:
> In our opinion, this is the most critical pitfall of Go’s JSON parser because it differs from the default parsers for JavaScript, Python, Rust, Ruby, Java, and all other parsers we tested.
It would be kind of difficult to argue that this is not about Go.
Don't get me wrong, I love Go just as much as the next person, but this article definitely lays bare some issues with serialization specific to Go.
> In our opinion, this is the most critical pitfall of Go’s JSON parser because it differs from the default parsers for JavaScript, Python, Rust, Ruby, Java, and all other parsers we tested.
It would be kind of difficult to argue that this is not about Go.
Don't get me wrong, I love Go just as much as the next person, but this article definitely lays bare some issues with serialization specific to Go.
I disagree.
It was a bad design choice to allow JSON keys with different capitalisation and to serialise all public struct members by default.
These decisions can easily result in the creation of an insecure system.
It was a bad design choice to allow JSON keys with different capitalisation and to serialise all public struct members by default.
These decisions can easily result in the creation of an insecure system.
I don’t think the title is clickbait. You’re fixating on just one example in the post, but the others demonstrate a clear pattern: there are footguns in Go’s various parsers, including forms of malleability or flexibility that enable parser differentials.
This isn’t intended to be a ding against Go; it’s a universal problem across programming languages (otherwise there’d be no differentials at all). But they’re worth noting, and I think the post amply demonstrates their security relevance.
This isn’t intended to be a ding against Go; it’s a universal problem across programming languages (otherwise there’d be no differentials at all). But they’re worth noting, and I think the post amply demonstrates their security relevance.
Of course it's about Go. They decided to build not just opt-out unmarshaling of objects, but fragile opt-out.
See eg rails' strong params for the opposite approach: opt-in.
See eg rails' strong params for the opposite approach: opt-in.
Exactly right. Better have a domain layer with data types representing the domain object 1:1 and add one or more API layers on top for interacting with those for some modality. Creation, deletion, verification, auth etc.
The security failure is not the parsing library, but failing to model your application architecture properly.
The security failure is not the parsing library, but failing to model your application architecture properly.
It's a concept from Fieldings REST thesis that is important to the current meaning of REST:
Transfer Objects are not the Storage Objects.
Transfer Objects are not the Storage Objects.
> No need to just have just 1 model that maps 1:1 to your DB row. This applies to all languages
I agree with your stance, but creating lots of DTOs is more work and we developers are lazy. It's incredibly common to see a single "User" type used everywhere. This also makes all User-related functions re-usable in all contexts (again: it's the fast and lazy approach).
I agree with your stance, but creating lots of DTOs is more work and we developers are lazy. It's incredibly common to see a single "User" type used everywhere. This also makes all User-related functions re-usable in all contexts (again: it's the fast and lazy approach).
> No need to just have just 1 model that maps 1:1 to your DB row. This applies to all languages
One reason is to avoid copying data constantly. I don't just mean this from an efficiency perspective, but also (and maybe more so) from a simplicity one. If you have a library for shoving data into a struct mechanistically, but you then take the data from that struct and shove it into an additional struct, what's the point of the library? You're writing the code move the data anyway.
In my dayjob I see this tendency constantly to have a lot of different very narrow structs that somehow integrate into some library, and then a TON of supporting code to copy between those structs. Only to then do very little actually useful work with any of the data at the end. I generally think you'd be happier with fatter structs that integrated less with weird "struct-filling" libraries.
One reason is to avoid copying data constantly. I don't just mean this from an efficiency perspective, but also (and maybe more so) from a simplicity one. If you have a library for shoving data into a struct mechanistically, but you then take the data from that struct and shove it into an additional struct, what's the point of the library? You're writing the code move the data anyway.
In my dayjob I see this tendency constantly to have a lot of different very narrow structs that somehow integrate into some library, and then a TON of supporting code to copy between those structs. Only to then do very little actually useful work with any of the data at the end. I generally think you'd be happier with fatter structs that integrated less with weird "struct-filling" libraries.
1. As shown in the article, exposing the internal model to APIs directly is a security footgun. I've seen sites get hacked because the developer serialized internal objects without any oversight just like in the article and accidentally exposed secrets.
2. Exposing internal models to APIs directly also makes it hard to refactor code because refactoring would change APIs, which would require updating the clients (especially problematic when the clients are owned by other teams). I've seen this firsthand too in a large legacy project, people were afraid to refactor the code because whenever they tried, it broke the clients downstream. So instead of refactoring, they just added various complex hacks to avoid touching the old core code (and of course, their models also mapped directly to the UI).
In the end, codebases like that, with no layer separation, become really hard to maintain and full of security problems.
All because they thought it was "simpler" to skip writing ~10 lines of extra boilerplate per model to map models to DTOs.
Lack of layer separation becomes a problem in the long term. When you're just starting out, it may seem like overengineering, but it isn't
2. Exposing internal models to APIs directly also makes it hard to refactor code because refactoring would change APIs, which would require updating the clients (especially problematic when the clients are owned by other teams). I've seen this firsthand too in a large legacy project, people were afraid to refactor the code because whenever they tried, it broke the clients downstream. So instead of refactoring, they just added various complex hacks to avoid touching the old core code (and of course, their models also mapped directly to the UI).
In the end, codebases like that, with no layer separation, become really hard to maintain and full of security problems.
All because they thought it was "simpler" to skip writing ~10 lines of extra boilerplate per model to map models to DTOs.
Lack of layer separation becomes a problem in the long term. When you're just starting out, it may seem like overengineering, but it isn't
> Lack of layer separation becomes a problem in the long term. When you're just starting out, it may seem like overengineering, but it isn't
I actually agree, but you're setting of a false dichotomy. I do believe in strong layering at the interfaces, for exactly the reasons you line up. What I don't believe in is what I might call "struct annotation based parsing" at those interfaces.
Typically, you don't want to pass DTO's around your code. Usually, you take in that struct, and then immediatly have some code to poke it into the appropriate places in your actual data structures. It's very often much easier to simply take a well structured but more direct and generic interpretation of the input data, and write the code to poke it into the correct places directly.
It is not that you should define your inputs separately from your internal data storage. It's that the specification of your input structure shouldn't exist as a struct, it should exist as the consequence of your parsing code.
> When you're just starting out, it may seem like overengineering, but it isn't
It's a real shame that the internet has driven us to assume everybody is a novice.
I actually agree, but you're setting of a false dichotomy. I do believe in strong layering at the interfaces, for exactly the reasons you line up. What I don't believe in is what I might call "struct annotation based parsing" at those interfaces.
Typically, you don't want to pass DTO's around your code. Usually, you take in that struct, and then immediatly have some code to poke it into the appropriate places in your actual data structures. It's very often much easier to simply take a well structured but more direct and generic interpretation of the input data, and write the code to poke it into the correct places directly.
It is not that you should define your inputs separately from your internal data storage. It's that the specification of your input structure shouldn't exist as a struct, it should exist as the consequence of your parsing code.
> When you're just starting out, it may seem like overengineering, but it isn't
It's a real shame that the internet has driven us to assume everybody is a novice.
>It's a real shame that the internet has driven us to assume everybody is a novice.
Sorry, English is not my native language. I didn't mean to say you're a novice.
>Usually, you take in that struct, and then immediatly have some code to poke it into the appropriate places in your actual data structures
>It's that the specification of your input structure shouldn't exist as a struct, it should exist as the consequence of your parsing code.
>It is not that you should define your inputs separately from your internal data storage. It's that the specification of your input structure shouldn't exist as a struct, it should exist as the consequence of your parsing code.
Can you give an example?
Sorry, English is not my native language. I didn't mean to say you're a novice.
>Usually, you take in that struct, and then immediatly have some code to poke it into the appropriate places in your actual data structures
>It's that the specification of your input structure shouldn't exist as a struct, it should exist as the consequence of your parsing code.
>It is not that you should define your inputs separately from your internal data storage. It's that the specification of your input structure shouldn't exist as a struct, it should exist as the consequence of your parsing code.
Can you give an example?
> Can you give an example?
Sure. I like taking Jackson (the Java library) as an example, since it actually supports both models. The way I've seen it used mostly is with jackson-databind. Here you define classes and annotate the fields with data that tells the library how to marshall them to/from json. Superficially, I find that similar to how Go or SerDe (from rust) suggests you handle that. In that programming model, I agree it makes total sense to declare some classes separately from your core structures, for all the reasons we've talked about.
The other model Jackson has is what they call the Jackson Tree Model. In this model you get back a representation of a Json Object. From that object you can get fields, those fields can themselves be objects, arrays, or immediate. It's an AST if you're a compiler person.
The first model might lead to code like this:
Sure. I like taking Jackson (the Java library) as an example, since it actually supports both models. The way I've seen it used mostly is with jackson-databind. Here you define classes and annotate the fields with data that tells the library how to marshall them to/from json. Superficially, I find that similar to how Go or SerDe (from rust) suggests you handle that. In that programming model, I agree it makes total sense to declare some classes separately from your core structures, for all the reasons we've talked about.
The other model Jackson has is what they call the Jackson Tree Model. In this model you get back a representation of a Json Object. From that object you can get fields, those fields can themselves be objects, arrays, or immediate. It's an AST if you're a compiler person.
The first model might lead to code like this:
public class Person {
@JsonProperty
String name;
@JsonProperty
String phoneNo;
}
if(personDto.phoneNo.length() != 10) return Http(400, "Phone number must be 10 chars");
person.phone = personDto.phoneNo
var phoneNoObj = jsonBody.get("phoneNo");
if(phoneNoObj == null) return Http(400, "Phone number is required");
var phoneNoStr = phoneNoObj.asString()
if(phoneNoStr == null) return Http(400, "Phone number must be a string");
if(phoneNoStr.length() != 10) return Http(400, "Phone number must be 10 chars");
person.phone = phoneNoStr;
In Go, you can marshal JSON to a map, so you can extract fields manually, too.
> In my dayjob I see this tendency constantly to have a lot of different very narrow structs that somehow integrate into some library, and then a TON of supporting code to copy between those structs.
Maybe that's the problem to solve, rather than exposing the entire internal world to the outside? Because different views of the same entities is pretty critical otherwise it's way too easy to start e.g. returning PII to public endpoints because some internal process needed it.
Maybe that's the problem to solve, rather than exposing the entire internal world to the outside? Because different views of the same entities is pretty critical otherwise it's way too easy to start e.g. returning PII to public endpoints because some internal process needed it.
> exposing the entire internal world to the outside
That's not at all what I said.
You don't need a struct to avoid exposing internal data. If you're building a JSON object, you can just not write the code to format some fields out. You don't need a new data layout for that.
That's not at all what I said.
You don't need a struct to avoid exposing internal data. If you're building a JSON object, you can just not write the code to format some fields out. You don't need a new data layout for that.
That's the whole point of the first item in the article, and the original comment you were replying to. In Go (and some other languages) that formatting is implicit and automatic. So you need to write to code to NOT format the fields out. Which leads to the potential security issues where data is leaked, or you can inject data into "hidden" fields. So since it's implicit and automatic, it's safer to, as a rule, define separate structs for the data input and map them, so that there is absolutely no chance (implicit or explicit) to leak or inject data.
> If you're building a JSON object, you can just not write the code to format some fields out.
Did you fail to read the article somehow?
Few if any modern language requires you to write the code to format individual fields out. Even Go does not, though when it comes to json most fields need to be annotated because of the choices it made to tie casing and visibility, but then it doesn't even require you to opt the type itself into being serializable, every struct is serializable by default.
Skipping fields on a serializable structure is what requires extra work.
Did you fail to read the article somehow?
Few if any modern language requires you to write the code to format individual fields out. Even Go does not, though when it comes to json most fields need to be annotated because of the choices it made to tie casing and visibility, but then it doesn't even require you to opt the type itself into being serializable, every struct is serializable by default.
Skipping fields on a serializable structure is what requires extra work.
> In my dayjob I see this tendency constantly to have a lot of different very narrow structs that somehow integrate into some library, and then a TON of supporting code to copy between those structs. Only to then do very little actually useful work with any of the data at the end.
Don’t think of it as doing a little useful work at the end; think of it as doing all the useful work in the centre. Your core logic should be as close to a pure implementation without external libraries as possible (ideally zero, but that is often not easily achievable), but call out to external libraries and services to get its work done as appropriate. That does mean a fair amount of data copying, but IMHO it’s worth it. Testing copies is easy and localised, whereas understand the implications of a JSON (or Protobuf, or Django, or whatever) object carried deep into one’s core logic and passed into other services and libraries is very very difficult.
There’s a common theme with the ORM trap here. The cost of a little bit of magic is often higher than the cost of a little bit of copying.
Don’t think of it as doing a little useful work at the end; think of it as doing all the useful work in the centre. Your core logic should be as close to a pure implementation without external libraries as possible (ideally zero, but that is often not easily achievable), but call out to external libraries and services to get its work done as appropriate. That does mean a fair amount of data copying, but IMHO it’s worth it. Testing copies is easy and localised, whereas understand the implications of a JSON (or Protobuf, or Django, or whatever) object carried deep into one’s core logic and passed into other services and libraries is very very difficult.
There’s a common theme with the ORM trap here. The cost of a little bit of magic is often higher than the cost of a little bit of copying.
> also (and maybe more so) from a simplicity one. If you have a library for shoving data into a struct mechanistically, but you then take the data from that struct and shove it into an additional struct, what's the point of the library? You're writing the code move the data anyway.
Super annoying if you need to do it by hand, and wastes compute and memory if you actually need to do copies of copies, but this is the mapping part of "object relational mapping", the M in ORM. Skipping it is a bad idea.
Your business/domain model should not be tied directly to your persistence model. It's a common mistake that's responsible for like half of the bad rep ORMs get. Data structures may look superficially similar, but they represent different concepts with different semantics and expectations. If you skip on that, you'll end up with tons of stupid mistakes like 'masklinn mentions, and more subtle bugs when the concepts being squashed together start pulling in opposite directions over time.
Super annoying if you need to do it by hand, and wastes compute and memory if you actually need to do copies of copies, but this is the mapping part of "object relational mapping", the M in ORM. Skipping it is a bad idea.
Your business/domain model should not be tied directly to your persistence model. It's a common mistake that's responsible for like half of the bad rep ORMs get. Data structures may look superficially similar, but they represent different concepts with different semantics and expectations. If you skip on that, you'll end up with tons of stupid mistakes like 'masklinn mentions, and more subtle bugs when the concepts being squashed together start pulling in opposite directions over time.
I don't know about Go, but in Java and .NET world this is trivially solvable with libraries like MapStruct. If you have a model with 20 fields and need to create a tiny slice of it (with let's say three fields), you need a few lines of boilerplate: create a record with those fields (1-3 LOC):
It also handles nested/recursive entities, nulls, etc. It's also using codegen, not reflection, so performance is exactly the same as if you had written it by hand, and the code is easy to read.
https://mapstruct.org
Go developers usually "don't need these complications", so this is just another self-inflicted problem. Or maybe it's solved, look around.
record Profile(int id, String login, boolean isAdmin) {}
interface UserMapper {
// arrays are just one example, plain models and plenty
// of other data structures are supported
Profile[] usersToProfiles(User[] user);
// other mappers...
}
class UserController {
//
@GET("/profiles")
Profile[] getUserProfiles() {
var users = userRepo.getUsers();
return userMapper.usersToProfiles(users);
}
}
It also handles nested/recursive entities, nulls, etc. It's also using codegen, not reflection, so performance is exactly the same as if you had written it by hand, and the code is easy to read.
https://mapstruct.org
Go developers usually "don't need these complications", so this is just another self-inflicted problem. Or maybe it's solved, look around.
Actually it's even easier in Go with struct embedding:
type SmallerThing struct {
Id int
Login string
IsAdmin bool
}
type UserController struct {
SmallerThing
OtherField Whatever
OtherField2 SomethingElse
}
>you have a model with 20 fields and need to create a tiny slice of it (with let's say three fields), you need a few lines of boilerplate: create a record with those fields (1-3 LOC):
>create a mapper for it:
> ...
>Go developers usually "don't need these complications", so this is just another self-inflicted problem.
In Go:
>create a mapper for it:
> ...
>Go developers usually "don't need these complications", so this is just another self-inflicted problem.
In Go:
type DTO struct {
A, B, C string
}
// copy the fields to the DTO
return DTO{A: o.A, B: o.B, C: o.C}
Yes!
[deleted]
Kudos to the author for making this very clear and easy to understand. More technical writing should be like this.
On another note, it's mind-blowing that a single string can parse as XML, JSON, and YAML.
On another note, it's mind-blowing that a single string can parse as XML, JSON, and YAML.
> Maintain consistency across boundaries. When input in processed in multiple services, ensure consistent parsing behavior by always using the same parser or implement additional validation layers, such as the strictJSONParse function shown above.
Or make some "entry gate"-service not only validate/authorize requestes but also re-encode them into certainly valid shape. In the example with AuthService/ProxyService from "Attack scenario 2", make the Auth Service return not a simple "yep/nope" in response, but properly re-serialized request instead (if it's allowed in). So if e.g. AuthService takes a request with two fields "UserAction" and "AdminAction" and allows the "UserAction", it would response with a request object that has "UserAction" field in it but not "AdminAction" because the service ignored that field and so did not copy it into the response.
Or make some "entry gate"-service not only validate/authorize requestes but also re-encode them into certainly valid shape. In the example with AuthService/ProxyService from "Attack scenario 2", make the Auth Service return not a simple "yep/nope" in response, but properly re-serialized request instead (if it's allowed in). So if e.g. AuthService takes a request with two fields "UserAction" and "AdminAction" and allows the "UserAction", it would response with a request object that has "UserAction" field in it but not "AdminAction" because the service ignored that field and so did not copy it into the response.
These kinds of issues (parser differentials in particular) are why you shouldn't trust Go SAML implementations that use `encoding/xml`, which was never designed for that application to begin with; I just wrote my own for my SAML.
(I mean, don't use SAML to begin with, but.)
(I mean, don't use SAML to begin with, but.)
Issue is not with go's parser, but instead about processing layer using different input than verifying layer [1]
We patched the gosaml2 (and other go saml libraries), by ensuring only the authenticated bytes are processed (not the original XML document). You can see the patches here: https://github.com/russellhaering/goxmldsig/commit/e1c8a5b89... https://github.com/russellhaering/gosaml2/commit/99574489327...
> I just wrote my own for my SAML.
Curious to see your implementation for SAML and XML Signatures.
[1]: https://bsky.app/profile/filippo.abyssdomain.expert/post/3le...
We patched the gosaml2 (and other go saml libraries), by ensuring only the authenticated bytes are processed (not the original XML document). You can see the patches here: https://github.com/russellhaering/goxmldsig/commit/e1c8a5b89... https://github.com/russellhaering/gosaml2/commit/99574489327...
> I just wrote my own for my SAML.
Curious to see your implementation for SAML and XML Signatures.
[1]: https://bsky.app/profile/filippo.abyssdomain.expert/post/3le...
I know these problems are easily avoidable... but I'm finally starting to see the appeal of protocol buffers.
Just to have the assurance that, regardless of programming language, you're guaranteed a consistent ser/de experience.
Just to have the assurance that, regardless of programming language, you're guaranteed a consistent ser/de experience.
It would nice if it were true :-)
But it’s not, for reasons that have more to do with the languages themselves, than parsing
e.g. C++ numbers are different than Java numbers are different than Python numbers are different than JavaScript numbers
ditto for strings
But it’s not, for reasons that have more to do with the languages themselves, than parsing
e.g. C++ numbers are different than Java numbers are different than Python numbers are different than JavaScript numbers
ditto for strings
I imagine that one of the points of a solid protocol buffers library would be to align the types even across programming languages. E.g. explicitly force a 64-bit integer rather than "int" relying on the platform. And to have some custom "string" type which is always UTF-8 encoded in memory rather than depending on the platform-specific encoding.
(I have no idea if that is the case with protobuf, I don't have enough experience with it.)
(I have no idea if that is the case with protobuf, I don't have enough experience with it.)
Why would you "imagine" that?
Again, the problem has more to do with the programming languages themselves, rather than with protobufs or parsing.
Protobuf has both signed and unsigned integers - the initial use case was C++ <-> C++ communication
Java doesn't have unsigned integers
Python has arbitrary precision integers
JavaScript traditionally only had doubles, which means it can represent integers up to 53 bit exactly. It has since added arbitrary size integers -- but that doesn't mean that the protobuf libraries actually use them
---
These aren't the only possibilities -- every language is fundamentally different
OCaml has 31- or 63-bit integers IIRC
https://protobuf.dev/programming-guides/encoding/#int-types
And again, strings also differ between all these languages -- there are three main choices, which are basically 8-bit, 16-bit, or 32-bit code units
Go and Rust favor 8-bit units; Java and JavaScript favor 16-bit units; and Python/C/C++ favors 32-bit units (which are code points)
Again, the problem has more to do with the programming languages themselves, rather than with protobufs or parsing.
Protobuf has both signed and unsigned integers - the initial use case was C++ <-> C++ communication
Java doesn't have unsigned integers
Python has arbitrary precision integers
JavaScript traditionally only had doubles, which means it can represent integers up to 53 bit exactly. It has since added arbitrary size integers -- but that doesn't mean that the protobuf libraries actually use them
---
These aren't the only possibilities -- every language is fundamentally different
OCaml has 31- or 63-bit integers IIRC
https://protobuf.dev/programming-guides/encoding/#int-types
And again, strings also differ between all these languages -- there are three main choices, which are basically 8-bit, 16-bit, or 32-bit code units
Go and Rust favor 8-bit units; Java and JavaScript favor 16-bit units; and Python/C/C++ favors 32-bit units (which are code points)
I think I didn't explain myself well.
As long as a language has bytes and arrays, you can implement anything on top of them, like unsigned integers, 8-bit strings, UTF-8 strings, UCS-2, whatever you want. Sure it won't be native types, so it will probably be slower and could have an awkward memory layout, but it's possible
Granted, if a language is so gimped that it doesn't even have integers (as you mentioned JavaScript), then that language will not be able to fully support it indeed.
As long as a language has bytes and arrays, you can implement anything on top of them, like unsigned integers, 8-bit strings, UTF-8 strings, UCS-2, whatever you want. Sure it won't be native types, so it will probably be slower and could have an awkward memory layout, but it's possible
Granted, if a language is so gimped that it doesn't even have integers (as you mentioned JavaScript), then that language will not be able to fully support it indeed.
Unfortunately that doesn't solve the problem -- it only pushes it around
I recommend writing a protobuf generator for your favorite language. The less it looks like C++, the more hard decisions you'll have to make
If you try your approach, you'll feel the "tax" when interacting with idiomatic code, and then likely make the opposite decision
---
Re: "so gimped" --> this tends to be what protobuf API design discussion are like. Users of certain languages can't imagine the viewpoints of users of other languages
e.g. is unsigned vs. signed the way the world is? Or an implementation detail.
And it's a problem to be MORE expressive than C/C++ -- i.e. from idiomatic Python code, the protobuf data model also causes a problem
Even within C/C++, there is more than one dialect -- C++ 03 versus C++ 11 with smart pointers (and probably more in the future). These styles correspond to the protobuf v1 and protobuf v2 APIs
(I used both protobuf v1 and protobuf v2 for many years, and did a design review for the protobuf v3 Python API)
In other words, protobufs aren't magic; they're another form of parsing, combined with code generation, which solve some technical problems, and not others. They also don't resolve arguments about parsing and serialization!
I recommend writing a protobuf generator for your favorite language. The less it looks like C++, the more hard decisions you'll have to make
If you try your approach, you'll feel the "tax" when interacting with idiomatic code, and then likely make the opposite decision
---
Re: "so gimped" --> this tends to be what protobuf API design discussion are like. Users of certain languages can't imagine the viewpoints of users of other languages
e.g. is unsigned vs. signed the way the world is? Or an implementation detail.
And it's a problem to be MORE expressive than C/C++ -- i.e. from idiomatic Python code, the protobuf data model also causes a problem
Even within C/C++, there is more than one dialect -- C++ 03 versus C++ 11 with smart pointers (and probably more in the future). These styles correspond to the protobuf v1 and protobuf v2 APIs
(I used both protobuf v1 and protobuf v2 for many years, and did a design review for the protobuf v3 Python API)
In other words, protobufs aren't magic; they're another form of parsing, combined with code generation, which solve some technical problems, and not others. They also don't resolve arguments about parsing and serialization!
> you're guaranteed a consistent ser/de experience
Are there that many implementations of protobuf? How many just wrap the C lib and proto compiler? Consistency can be caused by an underlying monoculture, although that's turtles all the way down because protobuf is not YAML is not JSON, etc.
Off in the weeds already, and all because I implemented a pure Python deserializer / dissector simply because there wasn't one.
Are there that many implementations of protobuf? How many just wrap the C lib and proto compiler? Consistency can be caused by an underlying monoculture, although that's turtles all the way down because protobuf is not YAML is not JSON, etc.
Off in the weeds already, and all because I implemented a pure Python deserializer / dissector simply because there wasn't one.
I think you can get similar benefits here from writing an RPC style JSON API into an OpenAPI spec and generating structs and route handlers from that. That's what I do for most of my Go projects anyway.
But since the article isn't really about parser bugs, I don't think using a different data format will save you from most of the problems described there.
As someone who isn't a Go programmer, on the face of it using strings (struct tags) for field metadata seems pretty backwards compared to Rust macros (which parses the metadata at compile time) or Java annotations (which are processed at runtime but at least don't require parsing a string to break apart options).
The accidental omitempty and - are a good example of the weirdness even if they might not cause problems in practice.
The accidental omitempty and - are a good example of the weirdness even if they might not cause problems in practice.
As a .net programmer, the "stringly typed" nature of the metadata horrifies me, but the choices of Go have long confused me.
So in .NET, like Java as you mention, we have attributes, .
e.g.
This is simple, and obvious. The JsonPropertyName attribute is an override, you can set naming policies for the whole class. camelCase by default, with kebab-case, snake_case etc as alternative defaults.
C#/.NET of course has the benefit of having public properties, which are serialised by default, and private properties, which aren't, so you're unlikely to be exposing things you don't want to expose.
This contrasts to Go's approach, much like python, of using casing convention to determine private vs public fields. ( Please correct me if I'm wrong on this? )
The first example still confuses me though, because either you want IsAdmin to come from the user, in which case you still want to deserialise it, or you don't, in which case it shouldn't even be in your DTO at all.
Deserialisation there is a bit of a red-herring, as there should be a validation step which includes, "Does this user have the rights to create an admin?".
The idea of having a user class, which gets directly updated using properties straight from deserialized user input, feels weird to me, but I'd probably be dismissed as an "enterprise programmer" who wants to put layers between everything.
So in .NET, like Java as you mention, we have attributes, .
e.g.
[JsonPropertyName("username")]
[JsonIgnore]
This is simple, and obvious. The JsonPropertyName attribute is an override, you can set naming policies for the whole class. camelCase by default, with kebab-case, snake_case etc as alternative defaults.
C#/.NET of course has the benefit of having public properties, which are serialised by default, and private properties, which aren't, so you're unlikely to be exposing things you don't want to expose.
This contrasts to Go's approach, much like python, of using casing convention to determine private vs public fields. ( Please correct me if I'm wrong on this? )
The first example still confuses me though, because either you want IsAdmin to come from the user, in which case you still want to deserialise it, or you don't, in which case it shouldn't even be in your DTO at all.
Deserialisation there is a bit of a red-herring, as there should be a validation step which includes, "Does this user have the rights to create an admin?".
The idea of having a user class, which gets directly updated using properties straight from deserialized user input, feels weird to me, but I'd probably be dismissed as an "enterprise programmer" who wants to put layers between everything.
> This contrasts to Go's approach, much like python, of using casing convention to determine private vs public fields. ( Please correct me if I'm wrong on this? )
I think calling it a convention is misleading.
In Python, you can access an _field just by writing obj._field. It's not enforced, only a note to the user that they shouldn't do that.
But in Go, obj.field is a compiler error. Fields that start with a lowercase letter really are private, and this is enforced.
So I think it's better to think of it as true private fields, just with a... unique syntax.
I think calling it a convention is misleading.
In Python, you can access an _field just by writing obj._field. It's not enforced, only a note to the user that they shouldn't do that.
But in Go, obj.field is a compiler error. Fields that start with a lowercase letter really are private, and this is enforced.
So I think it's better to think of it as true private fields, just with a... unique syntax.
> This contrasts to Go's approach, much like python, of using casing convention to determine private vs public fields. ( Please correct me if I'm wrong on this? )
Go actually ties visibility to casing, instead of using separate annotations. And it will not serialise private fields, only public.
Python has no concept of visibility at all, conventionally you should not access attributes prefixed with `_` but it won't stop you.
Go actually ties visibility to casing, instead of using separate annotations. And it will not serialise private fields, only public.
Python has no concept of visibility at all, conventionally you should not access attributes prefixed with `_` but it won't stop you.
> conventionally you should not access attributes prefixed with `_` but it won't stop you.
Any serious Python project will use at least one linter or typechecker, which can easily enforce this.
Any serious Python project will use at least one linter or typechecker, which can easily enforce this.
Not just weird, it’s dangerous to do this - to easy to miss validation as fields are added over time. Better to explicitly validate all fields IMO.
As someone who is regular Go programmer, yes struct tags do suck. It gets even worse if you want to try and combine multiple "annotations" into one struct tag string.
The reason its like that is that Go philosophically is very much against the idea of annotations and macros, and very strongly about the idea of a clear upfront control flow, and this is one of the reasons I love the language. But it does come at the cost of a few highly useful usecases for annotations (like mapping JSON and XML, etc.) becoming obtuse to use.
The idea of more compile-time macros in Go is interesting to me, but at the same time the ease of debugging and understanding the Go control flow in my programs is one of the reasons I love it so much, and I would not want to invite the possibility of "magic" web frameworks that would inevitably result from more metaprogramming ability in Go. So I guess I'm prepared to live with this consequence. :/
The reason its like that is that Go philosophically is very much against the idea of annotations and macros, and very strongly about the idea of a clear upfront control flow, and this is one of the reasons I love the language. But it does come at the cost of a few highly useful usecases for annotations (like mapping JSON and XML, etc.) becoming obtuse to use.
The idea of more compile-time macros in Go is interesting to me, but at the same time the ease of debugging and understanding the Go control flow in my programs is one of the reasons I love it so much, and I would not want to invite the possibility of "magic" web frameworks that would inevitably result from more metaprogramming ability in Go. So I guess I'm prepared to live with this consequence. :/
> The reason its like that is that Go philosophically is very much against the idea of annotations and macros, and very strongly about the idea of a clear upfront control flow
Annotations have no control flow, they just attach metadata to items. The difference with struct tags being that that metadata is structured.
Annotations have no control flow, they just attach metadata to items. The difference with struct tags being that that metadata is structured.
Yes, apologies I was not very clear here: What I meant to say is that Go avoids annotations because it invites magic frameworks and other types of metaprogramming that do the work for you behind the scenes.
I understand your feeling. There are many magical frameworks like e.g. Spring that do these things and it's super hard to figure out what's going on.
The solution is usually to have an even better language. One, where the typesystem is so powerful, that such hacks are not necessary. Unfortunately, that also means you have to learn that typesystem to be productive in language, and you have to learn it more or less upfront - which is not something that Google wanted for golang due to the turnover.
The solution is usually to have an even better language. One, where the typesystem is so powerful, that such hacks are not necessary. Unfortunately, that also means you have to learn that typesystem to be productive in language, and you have to learn it more or less upfront - which is not something that Google wanted for golang due to the turnover.
I don't think it's so much a question of a better language so much as a different language. There are obviously tradeoffs one makes with all language designs.
What might be interesting is a language ecosystem, where one can write parts of a system in one language and other parts in another. The BEAM and JVM runtimes allow for this but I don't think I've seen any good examples of different languages commingling and playing to their strengths.
What might be interesting is a language ecosystem, where one can write parts of a system in one language and other parts in another. The BEAM and JVM runtimes allow for this but I don't think I've seen any good examples of different languages commingling and playing to their strengths.
Well, what I meant is that there is some language feature missing. So either the language gets "better" in the sense that this feature is being added, or a different language is chosen, that is already "better" in that the feature already exists. So yeah.
> The BEAM and JVM runtimes allow for this but I don't think I've seen any good examples of different languages commingling and playing to their strengths.
Probably because the runtime is always the lowest common denominator. That being said, there are lots of tools e.g. written in Scala but then being used by Java, such as Akka or Spark. And the other way around of course.
> The BEAM and JVM runtimes allow for this but I don't think I've seen any good examples of different languages commingling and playing to their strengths.
Probably because the runtime is always the lowest common denominator. That being said, there are lots of tools e.g. written in Scala but then being used by Java, such as Akka or Spark. And the other way around of course.
I find the most annoying thing to be that there is no standard for separate values or field in the tag string. Sometimes you use a comma (json) sometimes a semicolon (Gorm IIRC) etc. This has caused me several headaches.
For some it's stupidity. For others it's brilliance.
It's one of many examples of 80/20 design in Go: 80% of functionality with 20% of complexity and cost.
Struct tags address an important scenario in an easy to use way.
But they don't try to address other scenarios, like annotations do. They are not function tags. They're not variable tags. They are not general purpose annotations. They are annotations for struct fields and struct fields only.
Are they are as powerful as annotations or macros? Of course not, not even close.
Are they as complex to implement, understand, use? Also not.
80/20 design. 80% of functionality at 20% of cost.
It's one of many examples of 80/20 design in Go: 80% of functionality with 20% of complexity and cost.
Struct tags address an important scenario in an easy to use way.
But they don't try to address other scenarios, like annotations do. They are not function tags. They're not variable tags. They are not general purpose annotations. They are annotations for struct fields and struct fields only.
Are they are as powerful as annotations or macros? Of course not, not even close.
Are they as complex to implement, understand, use? Also not.
80/20 design. 80% of functionality at 20% of cost.
Go's simplifications often introduce complexities elsewhere, however, as this article demonstrates with the complexities of correctness of a stringly-typed DSL.
There's no free lunch here, and the compromises Go makes to achieve its outcomes have shown themselves to be error-prone in ways that were entirely predictable at design time.
There's no free lunch here, and the compromises Go makes to achieve its outcomes have shown themselves to be error-prone in ways that were entirely predictable at design time.
> Go's simplifications often introduce complexities elsewhere
It does occasionally, although I'll push back on the "often". Go's simplifications allow most of the codebase to be... well... simple.
This does come at the cost of some complexity on the edge cases. That's a trade off I'm perfectly willing to make. The weird parts being complex is something I'm willing to accept in exchange for the normal parts being simple, as opposed to constantly dealing with a higher amount of complexity to make the edge cases easier.
> There's no free lunch here
This I'll agree with as well. The lunch is not free, but it's very reasonably priced (like one of those hole in the wall restaurants that serves food way too good for what you pay for it).
> the compromises Go makes to achieve its outcomes have shown themselves to be error-prone in ways that were entirely predictable at design time.
I also agree here, although I see this as a benefit. The things that are error prone are clear enough that they can be seen at design time. There's no free lunch here either, something has to be error prone, and I like the trade offs that go has made on which parts are error prone. Adding significant complexity to reduce those error prone places has, in my experience, just increased the surface area of the error prone sections of other languages.
Could you make the case that some other spot in design space is a better trade-off? Absolutely, especially for a particular problem. But this spot seems to work really well for ~95% of things.
It does occasionally, although I'll push back on the "often". Go's simplifications allow most of the codebase to be... well... simple.
This does come at the cost of some complexity on the edge cases. That's a trade off I'm perfectly willing to make. The weird parts being complex is something I'm willing to accept in exchange for the normal parts being simple, as opposed to constantly dealing with a higher amount of complexity to make the edge cases easier.
> There's no free lunch here
This I'll agree with as well. The lunch is not free, but it's very reasonably priced (like one of those hole in the wall restaurants that serves food way too good for what you pay for it).
> the compromises Go makes to achieve its outcomes have shown themselves to be error-prone in ways that were entirely predictable at design time.
I also agree here, although I see this as a benefit. The things that are error prone are clear enough that they can be seen at design time. There's no free lunch here either, something has to be error prone, and I like the trade offs that go has made on which parts are error prone. Adding significant complexity to reduce those error prone places has, in my experience, just increased the surface area of the error prone sections of other languages.
Could you make the case that some other spot in design space is a better trade-off? Absolutely, especially for a particular problem. But this spot seems to work really well for ~95% of things.
I dunno, I'm not willing to overlook null values or default zero-values that easily. Those cause me problems all the time, and they are not meaningfully simpler than the alternative (explicit assignment).
Or mutability modifiers. Yes, that's an extra feature, and there's an undeniable appeal to having fewer features. But being able to flag things as immutable will make all the code you deal with easier in future.
Or consider how they left out generics for 15 years. It simplifies the language in some ways, sure, but when you needed generics, you had to use reflection, which is way more complicated than generics. Even macros, unpopular as they are, are better than codegen.
Again, I understand the appeal of minimalism, but a cost/benefit analysis of any of these features shows them to be a massive net gain. A blanket policy of "no, we can't have nice things" is needlessly austere, and it leaves everyone worse off imo.
Or mutability modifiers. Yes, that's an extra feature, and there's an undeniable appeal to having fewer features. But being able to flag things as immutable will make all the code you deal with easier in future.
Or consider how they left out generics for 15 years. It simplifies the language in some ways, sure, but when you needed generics, you had to use reflection, which is way more complicated than generics. Even macros, unpopular as they are, are better than codegen.
Again, I understand the appeal of minimalism, but a cost/benefit analysis of any of these features shows them to be a massive net gain. A blanket policy of "no, we can't have nice things" is needlessly austere, and it leaves everyone worse off imo.
> Go's simplifications often introduce complexities elsewhere
Exactly this.
Basically: have a complex compression algorithm? Yes, it's complex, but the resulting filesize (= program complexity) will be low.
If you use a very basic compression algorithm, it's easier the understand the algorithm, but the filesize will be much bigger.
It's a trade-off. However, as professionals, I think we should really strive to put time to properly learn the good complex compression algorithm once and then benefit for all the programs we write.
Exactly this.
Basically: have a complex compression algorithm? Yes, it's complex, but the resulting filesize (= program complexity) will be low.
If you use a very basic compression algorithm, it's easier the understand the algorithm, but the filesize will be much bigger.
It's a trade-off. However, as professionals, I think we should really strive to put time to properly learn the good complex compression algorithm once and then benefit for all the programs we write.
> I think we should really strive to put time to properly learn
[insert Pike's Google young programmers quote here]
That's just not the philosophy of the language. The convention in Go is to be as obvious as possible, at the cost of more efficient designs. Some people like it, others don't. It bothers me, so I stopped using Go.
[insert Pike's Google young programmers quote here]
That's just not the philosophy of the language. The convention in Go is to be as obvious as possible, at the cost of more efficient designs. Some people like it, others don't. It bothers me, so I stopped using Go.
Exactly. Look how we are getting downvoted for the truth.
Starting to get feeling that 80/20 design is not good thing. Many things seems to be driven by worse is better but, looking at things like Climate Change... was it worth it?
Yes they are a horrible idea for many reasons, not just security. It’s like a hidden ill-defined poorly understood dsl in strings.
You can just not use them though - you can unmarshal to a map instead and select the keys you want, perform validation etc and then set the values.
Same when publishing - I prefer to have an explicit view which defines the keys exposed rather than than publishing all by default based on these poorly understood string keys attached to types.
You can just not use them though - you can unmarshal to a map instead and select the keys you want, perform validation etc and then set the values.
Same when publishing - I prefer to have an explicit view which defines the keys exposed rather than than publishing all by default based on these poorly understood string keys attached to types.
[deleted]
struct tags greatly reduce the boilerplate code required to map fields to fields. It’s really quite novel once you understand it.
> struct tags greatly reduce the boilerplate code required to map fields to fields.
Are you somehow under the impression that Go is unique in having a terse way to map fields to fields?
> It’s really quite novel once you understand it.
It's the opposite of novel, putting ad-hoc annotations in unstructured contexts is what people used to do before java 5.
Are you somehow under the impression that Go is unique in having a terse way to map fields to fields?
> It’s really quite novel once you understand it.
It's the opposite of novel, putting ad-hoc annotations in unstructured contexts is what people used to do before java 5.
No, not at all, but considering C has no such thing, I’ll take it.
It's not very novel. There's far better ways of solving this than allowing a random string to be embedded as aux information to a struct field. Examples: F# type providers, or OCamls PPX system for extending the language in a well defined way. Macro rewriting systems also allow for better safety in this area.
This allows you to derive a safe parser from the structural data, and you can make said parser be really strict. See e.g., Wuffs or Langsec for examples of approaches here.
This allows you to derive a safe parser from the structural data, and you can make said parser be really strict. See e.g., Wuffs or Langsec for examples of approaches here.
I’m not disagreeing that there are better ways to solve this given how other languages have implemented theirs but considering the constraints they had at the time the Go team designed this, it allowed them to implement marshaling fairly easily and leaves it open for extensions by the community.
> considering the constraints they had at the time the Go team designed this
What constraints? Ignoring decades of programming language developments since C89?
What constraints? Ignoring decades of programming language developments since C89?
I didn't see this mentioned in the article: wouldn't the obvious way be to make the private field private (lowercase)?
(I'm not a Go developer, just tried the language casually).
(I'm not a Go developer, just tried the language casually).
type User struct {
Username string `json:"username_json_key,omitempty"`
Password string `json:"password"`
isAdmin bool
}
> wouldn't the obvious way be to make the private field private (lowercase)?
That may break other things - `gorm`, for example, will ignore private fields - inconvenient if you want to serialise `User` to your DB.
That may break other things - `gorm`, for example, will ignore private fields - inconvenient if you want to serialise `User` to your DB.
If you use the same struct in both an HTTP API and an ORM, you're Doing It Wrong in my opinion. These should be completely separated. Exactly to prevent accidental leaking or injection of data.
> If you use the same struct in both an HTTP API and an ORM, you're Doing It Wrong in my opinion.
If you mean "public API", yep, 100% agree. Internal API between microservices though? Perfectly safe and cromulent, I'd say.
If you mean "public API", yep, 100% agree. Internal API between microservices though? Perfectly safe and cromulent, I'd say.
I tend to disagree with that, also. :) Even within one codebase there's immense value in having separate structs/classes per "layer" or domain. E.g. a different set of structs for the database layer than for the "business layer" (or whatever your application's internal setup is).
When that boundary is moved to outside the application, so an HTTP API between microservices, I feel even more strongly (though indeed still not as strongly as in what you call a "public API").
E.g. I have seen plenty of times a situation where a bunch of applications were managed within one team, the team split up and now this "internal API" has become an API between teams, suddenly making it "public" (when viewed from the teams perspective).
When that boundary is moved to outside the application, so an HTTP API between microservices, I feel even more strongly (though indeed still not as strongly as in what you call a "public API").
E.g. I have seen plenty of times a situation where a bunch of applications were managed within one team, the team split up and now this "internal API" has become an API between teams, suddenly making it "public" (when viewed from the teams perspective).
Humanity should ignore gorm in all honesty
You can annotate `json:"-"` which is equivalent of @JsonIgnore
Covered in the article along with a potential snafu - adding anything else (`-,omitempty`) turns it into a key "-", not the "ignore this field" indicator.
[deleted]
i am not aware of any parser that does that differently, but i would also argue that this is not the job of parsers. after parsing (or before exporting) there should be a data validation step based on whitelists.
so the user can send in unknown fields all they want, the code will only accept the username and firstname strings, and ignore the other ones.
same with fetching data and sending it to the user. i fetch only the fields i want and create the correct datastructures before invoking the marshaling step.
there are no footguns. if you expect your parser to protect you you are using it wrong. they were not designed for that.
input -> parse -> extract the fields we want, which are valid -> create a data-structure with those fields.
data -> get fields i want -> create datastructures with only wanted fields -> write to output format
so the user can send in unknown fields all they want, the code will only accept the username and firstname strings, and ignore the other ones.
same with fetching data and sending it to the user. i fetch only the fields i want and create the correct datastructures before invoking the marshaling step.
there are no footguns. if you expect your parser to protect you you are using it wrong. they were not designed for that.
input -> parse -> extract the fields we want, which are valid -> create a data-structure with those fields.
data -> get fields i want -> create datastructures with only wanted fields -> write to output format
This is correct.
In blog post they say:
> Other examples exist, but most follow the same pattern: the component that does security checks and the component that performs the actions differ in their view of the input data.
This would be solved (as you described), by ensuring that the downstream layer uses only contents that are verified in the security check layer.
If they are using a microservice then: Security check API -> return verified data (i.e. re-serialize the verified JSON or XML into byte form, NOT the original input) -> Processing layer i.e. userCreate API uses verified data.
This is the method we used in fixing the ruby-saml example.
See: https://bsky.app/profile/filippo.abyssdomain.expert/post/3le...
This would be solved (as you described), by ensuring that the downstream layer uses only contents that are verified in the security check layer.
If they are using a microservice then: Security check API -> return verified data (i.e. re-serialize the verified JSON or XML into byte form, NOT the original input) -> Processing layer i.e. userCreate API uses verified data.
This is the method we used in fixing the ruby-saml example.
See: https://bsky.app/profile/filippo.abyssdomain.expert/post/3le...
> this is not the job of parsers
The part of the article that I read before getting annoyed at the clickbaity title is basically "if you trust external data here's how you can blame that design decision on the parser".
The part of the article that I read before getting annoyed at the clickbaity title is basically "if you trust external data here's how you can blame that design decision on the parser".
[deleted]
There's a lot of good information in here, but if you think this is a time to go all language supremicist about how much better your language is and how this proves Go sucks, you might want to double-check the CVE database real quick first. A lot of these issues are more attributable to plain ol' user error and the fundamental messiness of JSON and XML than Go qua Go and I've seen many similar issues everywhere.
For instance, there simply isn't a "correct" way for a parser to handle duplicate keys. Because the problem they have is different layers seeing them differently, you can have the problem anywhere duplicate keys are treated differently, and it's not like Go is the only thing to implement "last wins". It doesn't matter what you do. Last wins? Varies from the many "first wins" implementations. First wins? Varies from the many "last wins" implementations. Nondeterministically choose? Now you conflict with everyone, even yourself, sometimes. Crash or throw an exception or otherwise fail? Now you've got a potential DOS. There's no way for a parser to win here, in any langauge. The code using the parser has some decisions to make.
Another example, the streaming JSON decoder "accepts" trailing garbage data because by the act of using the streaming JSON decoder you have indicated a willingness to potentially decode more JSON data. You can use this to handle newline-separated JSON, or other interesting JSON protocols where you're not committing to the string being just one JSON value and absolutely nothing else. It's not "an issue they're not planning on fixing", it's a feature with an absolutely unavoidable side effect in the context of streams. The JSON parser stops reading the stream at the end of the complete JSON object, by design, and anything else would be wrong because it would be consuming a value to "check" whether the next thing is JSON or not, when you may not even be "claiming" that the "next thing" is JSON, and whatever input it consumed to verify a claim that nobody is even making would itself be a bug.
Accepting user input into sensitive variables is a common mistake I've seen multiple times in a number of langauges. The root problem there is more the tension between convenience and security than languages themselves; any language can make it so convenient to load data that developers accidentally load more data than they realize.
Etc. The best lesson to take away from this is that there is more than meets the eye with JSON and XML and they're harder to use safely than its ease-of-use suggests.
Although in the interests of fairness I also consider case insensitivity in the JSON field names to be a mistake; maybe it should be an option, JSON can get messy in the real world, but it's a bad default. I have other quibbles but most of them are things where there isn't a correct answer where you can unambiguously say that some choice is wrong. JSON is really quite fundamentally messier than people realize, and XML, while generally more tightly specified at the grammer level than JSON is, is generally quite messy in the protocols people build on top of it.
For instance, there simply isn't a "correct" way for a parser to handle duplicate keys. Because the problem they have is different layers seeing them differently, you can have the problem anywhere duplicate keys are treated differently, and it's not like Go is the only thing to implement "last wins". It doesn't matter what you do. Last wins? Varies from the many "first wins" implementations. First wins? Varies from the many "last wins" implementations. Nondeterministically choose? Now you conflict with everyone, even yourself, sometimes. Crash or throw an exception or otherwise fail? Now you've got a potential DOS. There's no way for a parser to win here, in any langauge. The code using the parser has some decisions to make.
Another example, the streaming JSON decoder "accepts" trailing garbage data because by the act of using the streaming JSON decoder you have indicated a willingness to potentially decode more JSON data. You can use this to handle newline-separated JSON, or other interesting JSON protocols where you're not committing to the string being just one JSON value and absolutely nothing else. It's not "an issue they're not planning on fixing", it's a feature with an absolutely unavoidable side effect in the context of streams. The JSON parser stops reading the stream at the end of the complete JSON object, by design, and anything else would be wrong because it would be consuming a value to "check" whether the next thing is JSON or not, when you may not even be "claiming" that the "next thing" is JSON, and whatever input it consumed to verify a claim that nobody is even making would itself be a bug.
Accepting user input into sensitive variables is a common mistake I've seen multiple times in a number of langauges. The root problem there is more the tension between convenience and security than languages themselves; any language can make it so convenient to load data that developers accidentally load more data than they realize.
Etc. The best lesson to take away from this is that there is more than meets the eye with JSON and XML and they're harder to use safely than its ease-of-use suggests.
Although in the interests of fairness I also consider case insensitivity in the JSON field names to be a mistake; maybe it should be an option, JSON can get messy in the real world, but it's a bad default. I have other quibbles but most of them are things where there isn't a correct answer where you can unambiguously say that some choice is wrong. JSON is really quite fundamentally messier than people realize, and XML, while generally more tightly specified at the grammer level than JSON is, is generally quite messy in the protocols people build on top of it.
If a parser throwing an error due to duplicate fields causes a DoS, then the same is going to happen when it throws an error due to invalid encoding...
So, I don't think that's a relevant critique. I think any ambiguous case in parsing untrusted user input should raise an error, and anyone working on code with untrusted data should be ready to handle errors
So, I don't think that's a relevant critique. I think any ambiguous case in parsing untrusted user input should raise an error, and anyone working on code with untrusted data should be ready to handle errors
The article didn’t claim that “last wins” is in and of itself an issue, but that the differences between who wins between parsers across services/languages can cause issues. Their position was that everyone should standardize on “last wins,” since that is the most common.
The correct conclusion is: https://news.ycombinator.com/item?id=44337330
The problem of trying to ensure that each parser behaves the same for all input is twofold: - JSON and XML specifications are complex, lots of quirks. So not feasible. - Does not solve the fundamental issue of the processing layer not using the same data that is verified in the verification layer.
Note: the processing layer parses the original input bytes, while the verification layer verifies a struct that is parsed using another parser.
Processed: Proc(input) Verified: VerifyingParser(input)
The problem of trying to ensure that each parser behaves the same for all input is twofold: - JSON and XML specifications are complex, lots of quirks. So not feasible. - Does not solve the fundamental issue of the processing layer not using the same data that is verified in the verification layer.
Note: the processing layer parses the original input bytes, while the verification layer verifies a struct that is parsed using another parser.
Processed: Proc(input) Verified: VerifyingParser(input)
Been seeing these same problems in services for decades now. It's almost like they made these protocol languages exploitable on purpose.
I think it's just kinda dumb parsing. E.g. JSON is an extremely simple spec. Most of those issues that the Go JSON parser has, are because of specific choices of the Go implementation, not about JSON. The fact that it allows case-insensitive key matching is just insane. Also that it parses invalid XML documents (with garbage) into valid structs without returning an error is very much a problem with the parser and not with XML.
> The fact that it allows case-insensitive key matching is just insane.
It's probably a side effect of what is IMO another bad design of that language: letter casing determining field visibility, instead of using a keyword or a sigil. If your field has to be named "User" to be public, and the corresponding entry in the JSON has all-lowercase "user" as the key (probably because the JSON was defined first, and most languages have "field names start with lowercase" as part of their naming conventions), you have to either ignore case when matching, or manually map every field. They probably wanted to be "intuitive" and not require manual mapping.
It's probably a side effect of what is IMO another bad design of that language: letter casing determining field visibility, instead of using a keyword or a sigil. If your field has to be named "User" to be public, and the corresponding entry in the JSON has all-lowercase "user" as the key (probably because the JSON was defined first, and most languages have "field names start with lowercase" as part of their naming conventions), you have to either ignore case when matching, or manually map every field. They probably wanted to be "intuitive" and not require manual mapping.
> If your field has to be named "User" to be public, and the corresponding entry in the JSON has all-lowercase "user" as the key
then you specify the key to be "user"? Isn't that the point of the ability to remap names? Except you can't, because you don't have a choice whether or not your data is deserialised with case sensitivity enabled or not.
I've written plenty of Rust code to turn camelCase into snake_case and "it's too much effort" has never been a problem. It's a minor bother that helps prevents real security issues like the ones listed in this article.
Even if you want to help lazy programmers, I don't think there's a good reason to confuse "User" and "uſER" by default.
then you specify the key to be "user"? Isn't that the point of the ability to remap names? Except you can't, because you don't have a choice whether or not your data is deserialised with case sensitivity enabled or not.
I've written plenty of Rust code to turn camelCase into snake_case and "it's too much effort" has never been a problem. It's a minor bother that helps prevents real security issues like the ones listed in this article.
Even if you want to help lazy programmers, I don't think there's a good reason to confuse "User" and "uſER" by default.
I mean, one of the more silly things here is that even when you manually map, with the tag `json:"user"`, it still ignores the case of that while deserializing. It respects it while serializing though
Indeed...
Great informational post. I've been guilty of forgetting that pesky leading comma and changing the field name.
pitfalls
And people keep on hating on PHP... Case-insensitive JSON parsing is another level, one even PHP wouldn't cross.
vlowther(2)
It never occurred to me to ever (in any language) have a DTO with fields I wish (let alone require for security) not to unmarshall.
This seems doubly strange in Go the language of "Yes absolutely DO repeat yourself!"
Side rant:
JS (even with Typescript) pisses me off as this is unavoidable. Splats make it worse. But still a distinct DTO and business object and don't use splats would help. (Maybe make ... a lint error)