Map[string]interface{} in Go: a user's manual(bitfieldconsulting.com)
bitfieldconsulting.com
Map[string]interface{} in Go: a user's manual
https://bitfieldconsulting.com/golang/map-string-interface
72 comments
This article would be great to be revised once parametric types (AKA "generics") become an official part of the language.
https://go.googlesource.com/proposal/+/master/design/15292/2...
https://go2goplay.golang.org/
https://go.googlesource.com/proposal/+/master/design/15292/2...
https://go2goplay.golang.org/
map in Go is already generic. If you need map[string]int or map[int]float, you can already do it.
Generics do not obviate the need for interface{} or the described use case of map[string]interface{}
interface{} is like Java or C# object. It packages a type and a value in a single value that can be queried at runtime to extract the underlying value in a type safe matter.
When people use map[string]interface{] it's because they don't know the type of value at compile time.
Generics do not obviate the need for interface{} or the described use case of map[string]interface{}
interface{} is like Java or C# object. It packages a type and a value in a single value that can be queried at runtime to extract the underlying value in a type safe matter.
When people use map[string]interface{] it's because they don't know the type of value at compile time.
I'm not sure why you're being downvoted, the only alternatives I've seen are to use methods or enums, which constrain the types to those supported by JSON, but ultimately you still have to switch on the type at runtime, the tradeoff being between closed enums vs. open interfaces, or the runtime tradeoffs of methods. Rust's serde_json uses enums for this, but the maps are generic in the same way that they already are in Go.
> the maps are generic in the same way that they already are in Go.
Go maps are actually not generic in the same way they are in other languages, there is just a single map struct with type descriptors used to do some compiler magic: https://dave.cheney.net/2018/05/29/how-the-go-runtime-implem...
Go maps are actually not generic in the same way they are in other languages, there is just a single map struct with type descriptors used to do some compiler magic: https://dave.cheney.net/2018/05/29/how-the-go-runtime-implem...
[deleted]
Yes I'm aware that there are differences, but it's not clear to me how that's relevant.
Well you said "maps are generic in the same way that they already are in Go", which I took to mean that Go already has monomorphized generics under the hood, they just aren't exposed publicly.
I'm specifically talking about the subset of ways in which they are similar and also relevant to this scenario of handling arbitrary structured data like JSON. There are of course ways in which they are not similar, but I'm not sure how those are relevant to this scenario being discussed....could you elaborate on that?
Yeah, no. That's not what generics mean. A map of any single type with any type for the key is different from a map using generic types.
For generics what you want is not interface{}, which says nothing, but a type T that has meaningful constraints.
For generics what you want is not interface{}, which says nothing, but a type T that has meaningful constraints.
Doing nothing is the point; it serves the same purpose as a `void*` in C, except that there are (runtime-) safe ways to take a value out of `interface{}` and back to its concrete type in Go. The most obvious use for it is heterogenous containers.
> The most obvious use for it is heterogenous containers.
You mean runtime generic container without the benefits of compile-time type checking, opening the developer to runtime type errors purely for ideological reasons...
You mean runtime generic container without the benefits of compile-time type checking, opening the developer to runtime type errors purely for ideological reasons...
That Go would benefit from generics doesn't change the fact that there are also lots of use for heterogenous containers. As it is, Go already has "slices" for type safe containers. The fair complaint about Go is it's hard to do other type safe container than the built in "slice" and "map" containers. But the problem is on track to be fixed next year, so it's silly to complain about it now.
It's really hard to take people who complain about Go seriously when they're pretty obviously just zealots who don't know much about the language besides "types good" and "Go bad". My stance: types good in moderation; Go a useful tradeoff of engineering attributes.
It's really hard to take people who complain about Go seriously when they're pretty obviously just zealots who don't know much about the language besides "types good" and "Go bad". My stance: types good in moderation; Go a useful tradeoff of engineering attributes.
> so it's silly to complain about it now.
No it isn't, Go has other flaws stemming from the exact same root cause and it has nothing to do with programming...
No it isn't, Go has other flaws stemming from the exact same root cause and it has nothing to do with programming...
> purely for ideological reasons
There are no ideological reasons.
https://github.com/golang/go/commit/dd64f86e0874804d0ec5b713... "Generics may well come at some point."
https://golang.org/doc/faq#generics "A language proposal implementing a form of generic types has been accepted for inclusion in the language. If all goes well it will be available in the Go 1.18 release."
One of the language designers explaining the motivation for adding parametric polymorphism to Go: https://www.youtube.com/watch?v=RIvL2ONhFBI&t=1064s
Explaining the benefits of generics: https://www.reddit.com/r/golang/comments/jditu9/what_do_gene...
From the Featherweight Go paper: https://arxiv.org/pdf/2005.11710.pdf "Recently, the Go team mooted a design to extend Go with generics, and Rob Pike wrote Wadler to ask: Would you be interested in helping us get polymorphism right (and/or figuring out what “right” means) for some future version of Go?"
There are no ideological reasons.
https://github.com/golang/go/commit/dd64f86e0874804d0ec5b713... "Generics may well come at some point."
https://golang.org/doc/faq#generics "A language proposal implementing a form of generic types has been accepted for inclusion in the language. If all goes well it will be available in the Go 1.18 release."
One of the language designers explaining the motivation for adding parametric polymorphism to Go: https://www.youtube.com/watch?v=RIvL2ONhFBI&t=1064s
Explaining the benefits of generics: https://www.reddit.com/r/golang/comments/jditu9/what_do_gene...
From the Featherweight Go paper: https://arxiv.org/pdf/2005.11710.pdf "Recently, the Go team mooted a design to extend Go with generics, and Rob Pike wrote Wadler to ask: Would you be interested in helping us get polymorphism right (and/or figuring out what “right” means) for some future version of Go?"
Yup, it's a common misconception that Go language designers are anti-generics. They wanted to include generics, the reason they were left out is that there was no clear way to include them without significant downsides:
> The generic dilemma is this: do you want slow programmers, slow compilers and bloated binaries, or slow execution times?
> I would be happy to learn about implementations that somehow manage to avoid all three of these bad outcomes, especially if there were good written descriptions (papers, blog posts, etc.) about what was hard and why it's a good approach. I'd also be interested to see good written descriptions of trying one of those approaches and what went wrong (or right).
- Russ Cox (2009): https://research.swtch.com/generic
> The generic dilemma is this: do you want slow programmers, slow compilers and bloated binaries, or slow execution times?
> I would be happy to learn about implementations that somehow manage to avoid all three of these bad outcomes, especially if there were good written descriptions (papers, blog posts, etc.) about what was hard and why it's a good approach. I'd also be interested to see good written descriptions of trying one of those approaches and what went wrong (or right).
- Russ Cox (2009): https://research.swtch.com/generic
> do you want slow programmers, slow compilers and bloated binaries, or slow execution times?
Which one did Rust choose?
Which one did Rust choose?
slow compiler
I think you read my comment as an invitation to debate dynamic vs. static safety checking. That seems like a deeply uninteresting debate. I'm just saying, there's a clear purpose to `interface{}`, and the same pattern shows up (though obviously much less frequently) in Rust.
In the interest of supplying information to the curious, the Rust equivalent is to use an Any trait object with down casting.
https://doc.rust-lang.org/std/any/index.html
https://doc.rust-lang.org/std/any/index.html
The two are orthogonal. Even with generics, languages can have an "escape" type like interface{} which is used/necessary for when you do want (or require) run-time type checking.
That Go would benefit from generics doesn't change the fact that there are also lots of use for heterogenous containers. As it is, Go already has "slices" for type safe containers. The fair complaint about Go is it's hard to do other type safe container than the built in "slice" and "map" containers. But the problem is on track to be fixed next year, so it's silly to complain about it now.
> opening the developer to runtime type errors purely for ideological reasons...
To be fair, there are also language-implementor-laziness reasons.
To be fair, there are also language-implementor-laziness reasons.
Runtime safe and statically type checked are two fundamentally different things is my point. Interfaces in most typed languages are static concepts, though there are always exceptions.
You can think of interface{} like Object in Java.
Interfaces in go are simply saying "this thing satifies these properties". interface{} is an interface that anything can satisfy.
Interfaces in go are simply saying "this thing satifies these properties". interface{} is an interface that anything can satisfy.
I will continue to think of interface{} as Any in Rust.
`interface{}` used as a type is almost exactly `Box<dyn Any>`.
I don't think "empty" is quite the right mental model. It's an object that has at least 0 methods, which is a characteristic of every data type. An actually empty type in Go is struct{} (good for map values when the map represents a set).
To be clear, "empty interface" is the official term: ctrl+f "empty interface" on https://golang.org/ref/spec
Well the interface's method set is empty, not the underlying type.
There's a nice trick you can use to make your map[string]interface{} code a bit easier to write and arguably also to read (less {}s), useful especially when you need e.g. to serialize some deeply nested JSON, for one-off request:
func doSomeRequest(...) {
type loose map[string]interface{}
payload, err := json.Marshal(loose{
"fooBar": true,
"nested": loose{
"subfield": bazArg,
},
})
// ...
}
That is a good trick and I have a place where I can use this idea to make something more clear. You should write that blog post!
`map[string]interface{}` is just a map with string keys and `any` as a value, and is commonly used as the type for a JSON object. Unless you have a completely specified type for your JSON object (at that point might as well use a `struct` with all of the fields defined), the best you _could_ do would be something like a recursive type `type JSON = bool | string | []JSON | map[string]JSON` in some hypothetical version of Golang with recursive types and unions.
Also, if you're making a set, use `map[string]struct{}` instead of `map[string]interface{}` as `struct{}` takes 0 bytes whereas `interface{}` takes 8.
Also, if you're making a set, use `map[string]struct{}` instead of `map[string]interface{}` as `struct{}` takes 0 bytes whereas `interface{}` takes 8.
You could have a JSON value interface with methods like getType and getAsString, which would remove the overhead and type safety issues of reflection.
depending on what you're trying to do a map[string]interface{} is not really want you want. Most likely you want map[string]json.RawMessage if the key can indicate what the next step is in umarshalling.
if the serialized structure doesn't indicate its "Type" somewhere/somehow then you would have to resort to map[string]interface{}
if the serialized structure doesn't indicate its "Type" somewhere/somehow then you would have to resort to map[string]interface{}
Why does the syntax specifically use `interface{}` instead of just `interface`? `{}` isn't use for any other types
`interface` isn't a type, it's the keyword used to declare an interface:
type Foo interface {
Foo() int
}
I'm excited for generics to land in Golang so we can have efficient set implementations instead of abusing maps.
In many languages (including Python & Scala), sets are not much more than syntactic sugar on a hashmap-like datatype.
This is true for Java as well, a `HashSet` is just a `HashMap<T, Object>` where the value is:
A thread-safe implementation is available with `ConcurrentHashMap.newKeySet()`, which uses a similar approach:
// Dummy value to associate with an Object in the backing Map
private static final Object PRESENT = new Object();
A thread-safe implementation is available with `ConcurrentHashMap.newKeySet()`, which uses a similar approach:
public static <K> KeySetView<K,Boolean> newKeySet() {
return new KeySetView<K,Boolean>
(new ConcurrentHashMap<K,Boolean>(), Boolean.TRUE);
}
In rust, hashsets are just hashmaps with a zero-sized type as the value, which results in all the extra getting optimized away.
A zero-sized type + hashmap for sets is also done in Go with a `struct{}`. [1][2]
But without generics it must be implemented again for each entry type ...
Kubernetes uses a code generator to automate this, but it's a bit kludge. [3]
[1]: https://pkg.go.dev/k8s.io/apimachinery/pkg/util/sets#Empty
[2]: https://dave.cheney.net/2014/03/25/the-empty-struct
[3]: https://github.com/kubernetes/code-generator/blob/master/cmd...
But without generics it must be implemented again for each entry type ...
Kubernetes uses a code generator to automate this, but it's a bit kludge. [3]
[1]: https://pkg.go.dev/k8s.io/apimachinery/pkg/util/sets#Empty
[2]: https://dave.cheney.net/2014/03/25/the-empty-struct
[3]: https://github.com/kubernetes/code-generator/blob/master/cmd...
There probably aren't many languages where that's different, whether a hashset on a hashmap or a treeset on a treemap.
However sets tend to provide support for… set operations. Which currently can't be done generically in go except through interface{} or specialised, neither of which is a great option.
However sets tend to provide support for… set operations. Which currently can't be done generically in go except through interface{} or specialised, neither of which is a great option.
Even if that's the case, the thing we want is syntactic sugar. It'd be nice to just do `sets.New[int]()` rather than manipulate maps manually or create bespoke types.
`map[string]interface{}` screams disconnected from reality when building web software. What an awful experience.
I only rarely ever use `interface{}` anywhere in Go code. The idiomatic thing to do with JSON messages you're going to hold onto in Go web code is to type them, just as you would in Rust with serde.
Yeah, nah. I’m not typing 5 levels of “structs” for one or two properties I want.
I'm not sure what you're after, then. You have basically the same safety problems in Ruby and Python dealing with nested JSON; the thing you seem to be bothered by is that Go makes it possible to get extra safety here.
I’m not after anything. Why do I need “safety” from a web response? You should be liberal with what you’re accepting from a service.
What “type safety” could you possibly hope for from a system you don’t control?
What “type safety” could you possibly hope for from a system you don’t control?
> Why do I need “safety” from a web response?
I’m not even sure where to start from that. Responses provided to an HTTP request are one of the most obvious places where enforcing structure and type safety is clearly a win.
I’m not even sure where to start from that. Responses provided to an HTTP request are one of the most obvious places where enforcing structure and type safety is clearly a win.
Go's nested struct syntax actually makes doing so quite nicer than Rust :)
Yeah, the formality of embedded structs can lead to some very verbose bloat. And it obviously only works on very reliable payloads.
What's the solution for an unreliable payload? When things aren't where you expect them in a JSON blob with a Python Flask app, your app also just breaks on that input.
What solution? There is no solution. The web just works that way.
You receive some bytes and you do stuff with them. Any other expectation is a farce.
Is a property missing after serialization? Can you even serialize it? Maybe a different status code returns a different mime type! OK, whatever. That’s not always an error, nor is it remotely “exceptional.”
You receive some bytes and you do stuff with them. Any other expectation is a farce.
Is a property missing after serialization? Can you even serialize it? Maybe a different status code returns a different mime type! OK, whatever. That’s not always an error, nor is it remotely “exceptional.”
I mean, I agree. That's why I'm sort of confused about what the problem is here. Yes, you can get unreliable inputs. Most apps barf on them, even if they think they don't. The barfing mechanism is pretty much the same everywhere.
try:
val = input['foo']['bar']
except KeyError, TypeError:
raise HttpErrors.BadRequestCan someone explain to me why python uses try/except so often? I see that promoted as a solution instead of;
if 'bar' in input['foo'].keys(): val = input['foo']['bar']
Generally, exceptions cause code to run slower and I feel like in any other language is considered bad practices.
I never understood why Python promotes EAFP vs LBYL. Both performance and accuracy wise, checking before doing is safer and faster performance.
if 'bar' in input['foo'].keys(): val = input['foo']['bar']
Generally, exceptions cause code to run slower and I feel like in any other language is considered bad practices.
I never understood why Python promotes EAFP vs LBYL. Both performance and accuracy wise, checking before doing is safer and faster performance.
Well, firstly input.get("foo", {}).get("bar", default) is a bit nicer, imho.
Python exceptions are only slower on the exception. They are almost exactly the same speed on success.
LBYL forces you to look for all the ways in which you might leap. EAFP lets you handle even unexpected conditions. The classic use case is checking if a condition exists, then doing the thing, except the condition changed between check and run. So you gotta catch anyways.
I've been moving away from this actually and towards a rust-like Result[T, Exception]. I'm using the package called "result". For fastapi api_request -> handler -> some_func_chain -> return response_to_client patterns, if something barfs, you normally get a 422 and useless "Internal server error". It's far simpler imho to map over some Results and then pack the error(s) into a more useful response.
Also testing is just a lot simpler.
https://pypi.org/project/result/
Python exceptions are only slower on the exception. They are almost exactly the same speed on success.
LBYL forces you to look for all the ways in which you might leap. EAFP lets you handle even unexpected conditions. The classic use case is checking if a condition exists, then doing the thing, except the condition changed between check and run. So you gotta catch anyways.
I've been moving away from this actually and towards a rust-like Result[T, Exception]. I'm using the package called "result". For fastapi api_request -> handler -> some_func_chain -> return response_to_client patterns, if something barfs, you normally get a 422 and useless "Internal server error". It's far simpler imho to map over some Results and then pack the error(s) into a more useful response.
Also testing is just a lot simpler.
https://pypi.org/project/result/
Your example will throw a KeyError if "input" doesn't contain "foo", and an AttributeError if "foo" is some unexpected data type that doesn't have keys().
You just generated a 502, and spent a bunch of extra lines doing it. The 502 is bad, and if that's all I need to do, I get that almost for free from net/http's panic handler. It sounds like you don't actually have a problem here.
I kept the example simple, of course I could now customize the error message further (or take some other sensible action). Note that I returned a 400, not a generic 502 here. If I didn't care at all, I wouldn't need any extra lines. Does the panic handler send a response at all, or does it just drop the connection?
Also, in Go, using panic/recover is considered unidiomatic. You can use it like an exception system, but it's not what you're supposed to do.
Also, in Go, using panic/recover is considered unidiomatic. You can use it like an exception system, but it's not what you're supposed to do.
I'm not suggesting that you would build a custom exception recovery system; I'm pointing out that net/http, which is definitionally idiomatic, already has that recovery mechanism.
I believe that that recovery mechanism isn't supposed to be routinely triggered just due to bad user input, and that code that does so would be considered defective.
It's doing almost literally the same thing as the Python code upthread. Just, with less code.
I think that was more of an example. There are lots of ways to fail gracefully when you don't have the data you want in the dict.
Right, which is why json.Unmarshal returns an error, and why map lookups return an extra `ok` value in Go.
It would be a lot more than 4 lines in Go though. Either the struct definition, or a lot of `if !ok return error`. Especially once the nesting gets deeper.
I'm pretty sure it would just be an `if err != nil {http.Error(w, err.Error(), 501); return }`.
one of these for each nesting level.
[deleted]
This article almost seems like a falseflag, in how it cheerfully provides a benediction for an ugly hack to deal with controversial inadequacies of the Go type system.
I don't see interface{} as a deficiency in the type system. Just about every type system has a method for storing "any" value such that it can then be upcast at runtime.
The problem in Go isn't with interface{} but that the simple type system can't handle common use cases so you end up needing it use interface{} much more often that you would need to in other languages.
Basically interface{} is useful when you truly need it. But it is often used to workaround deficiencies in the type system.
The problem in Go isn't with interface{} but that the simple type system can't handle common use cases so you end up needing it use interface{} much more often that you would need to in other languages.
Basically interface{} is useful when you truly need it. But it is often used to workaround deficiencies in the type system.
An empty interface should be effectively useless. What could it possibly do!? This pattern is much more like a dynamic container with a type tag.