netaddr.IP: a new IP address type for Go(tailscale.com)
tailscale.com
netaddr.IP: a new IP address type for Go
https://tailscale.com/blog/netaddr-new-ip-type-for-go/
101 comments
I just want to say I love how the information is provided and broken down in this blog post. Masterclass.
Reading the description of the stdlib type, I can't help but be reminded of the statement that the go way of solving a problem is to ignore real-world problems and half-ass things for the sake of a simplicity that merely pushes the complexity elsewhere. It certainly does seem to apply here too. Glad this library exists now though.
The standard library IP type has worked for numerous programs for a literal decade. Just because a specialized application can do a better job by leveraging, again, decades of real world experience with the language doesn't invalidate that or mean it's half-assed.
I think your statement reflects more on your own biases than reality.
I think your statement reflects more on your own biases than reality.
I am not saying that the type is unusable. Plenty of high profile software has been written with C strings too yet we almost universally recognize them today as a mistake.
What I'm saying is that the design flaws of net.IP, the absence of which would have benefited the whole ecosystem, fall into a familiar pattern that can be learned from. It is good to see that addressed, even if by third parties.
What I'm saying is that the design flaws of net.IP, the absence of which would have benefited the whole ecosystem, fall into a familiar pattern that can be learned from. It is good to see that addressed, even if by third parties.
Brad Fitzpatrick isn't exactly a third party. He was on the golang team until 2020 and you'll find his contributions all over the language.
It wouldn't surprise me one bit if this ended up in the stdlib for Go 2.0.
It wouldn't surprise me one bit if this ended up in the stdlib for Go 2.0.
That's great to hear! Hope you're right.
If you can look at any widely used API that has existed for an extended period of time and not find a single tradeoff or hindsight flaw, you aren't looking hard enough.
I think being able to compare or hash IP addresses is a pretty baseline feature for an IP address type. I don't think that's some kind of unpredictable hindsight or tradeoff.
Obviously, people routinely compare and hash IP addresses in Go programs.
Out of interest, how do you compare/store in a map? Do you convert to a string, or a [4]byte array (for IPv4), or some other comparable/hashable type?
Either strings or (in the common case where I know I'm dealing with v4) uint32's. In fact, I probably do v6 stuff with uint32s too (I'm never working with an arbitrarily large v6 prefix).
I think this post is right about net.IP, though I also think the solution they've reached is Lovecraftian. But I write Rust, too, and I'd take net.IP over Sockaddr any day of the week.
I think this post is right about net.IP, though I also think the solution they've reached is Lovecraftian. But I write Rust, too, and I'd take net.IP over Sockaddr any day of the week.
I assume you mean std::net::SocketAddr? Out of curiosity, what's wrong with it?
I'd assume the lackluster selection of operations and how deeply nested some of the information is within the type. Definitely not the most ergonomic thing.
My personal favorite is probably the python ones. Only annoyance is difficulty of creating modified objects from existing ones.
My personal favorite is probably the python ones. Only annoyance is difficulty of creating modified objects from existing ones.
I can't tell if your're being sarcastic but it's hard to find a program that uses UDP and doesn't at some point store IP addresses in a hashmap. Many more will do the same for per-IP rate limiting. I think those are reasonable use cases for go.
Yes, I ship several of those commercially. Like I said: obviously, people hash and compare IP addresses routinely in Go.
Just not using the built in IP type, which is stupid.
I think it's more likely that numerous programs have worked around the issues of the built in type for a decade and/or changed scope/modified functionality to avoid it.
I had to work around one of the issues here in my first use of the built in ip type - I needed to parse ip addresses and pass them to a command that needed a tcp4 or tcp6 flag depending on the address type.
I had to work around one of the issues here in my first use of the built in ip type - I needed to parse ip addresses and pass them to a command that needed a tcp4 or tcp6 flag depending on the address type.
Thompson and Pike are both Bell Labs, the origin of https://en.wikipedia.org/wiki/Worse_is_better#New_Jersey_sty... - so that's by design.
That's a pretty lazy criticism given how much of the most modern and popular networking and system software is written in Go (Kubernetes, Docker, all the related infrastructure, etc).
The "net" standard library is still very nice and works extremely well.
The proof of the pudding is in the eating, and the Go pudding has been eaten successfully many times.
The "net" standard library is still very nice and works extremely well.
The proof of the pudding is in the eating, and the Go pudding has been eaten successfully many times.
> networking and system software is written in Go (Kubernetes, Docker, all the related infrastructure, etc).
Kubernetes and Docker are not networking. They orchestrate some networking components that do the actual network processing in the kernel (iptables, ebpf) or other high performance user space c/c++ (dpdk, etc).
Kubernetes and Docker are not networking. They orchestrate some networking components that do the actual network processing in the kernel (iptables, ebpf) or other high performance user space c/c++ (dpdk, etc).
[deleted]
And Facebook was written in PHP. That hardly means that PHP doesn't have flaws or quirks or drawbacks.
And to your other point, I've never used docker or kubernetes and I am alive and fully functional without them.
As far as I'm concerned containers are a gimmick just like VPNs and just like go. It's a very hands off and ineffective approach to security that just makes you more comfortable being more lazy.
And to your other point, I've never used docker or kubernetes and I am alive and fully functional without them.
As far as I'm concerned containers are a gimmick just like VPNs and just like go. It's a very hands off and ineffective approach to security that just makes you more comfortable being more lazy.
I remember a time when Facebook would make a quite buggy appearance for a year or two. Must have been around 2015.
Docker and k8s are tools to make Linux cgroups more accessible for the user. In total they add more structure to files and processes. I cannot really say that about VPNs since OpenVPN for instance just overwrites all my network settings. But I guess it's possible to write bad code in every language.
Docker and k8s are tools to make Linux cgroups more accessible for the user. In total they add more structure to files and processes. I cannot really say that about VPNs since OpenVPN for instance just overwrites all my network settings. But I guess it's possible to write bad code in every language.
The uintptr / SetFinalize trick to implement "weak" references is pretty cool, but I'd be curious to hear why
type IP struct {
hi, lo uint64
z string
}
Yeah, that would be the obvious representation if we were willing to go to 32 bytes. (on 64-bit)
But paying 8x memory cost for IPv4 and 2x for IPv6 was a bit hard to swallow. Especially considering how rare IPv6+zones are.
And we didn't want to have be worse in any regard compared to std's net.IP. If we went to 32 bytes then Go's net.IP would win if you parsed an IP and stored it multiple times. Because we'd be a 32 byte value type and Go would be a 24 byte value type containing a pointer to the shared-for-all-copies address bits. Sure, they'd all be aliasing the same memory (danger zone!) but in Go the culture is to behave and risk that.
But staying at 24 bytes, we're never worse than net.IP and strictly better in all regards. (at least by the metrics we highlighted in the blog post... perhaps not by "don't play unsafe shenanigans with IPv6 zone interning")
But paying 8x memory cost for IPv4 and 2x for IPv6 was a bit hard to swallow. Especially considering how rare IPv6+zones are.
And we didn't want to have be worse in any regard compared to std's net.IP. If we went to 32 bytes then Go's net.IP would win if you parsed an IP and stored it multiple times. Because we'd be a 32 byte value type and Go would be a 24 byte value type containing a pointer to the shared-for-all-copies address bits. Sure, they'd all be aliasing the same memory (danger zone!) but in Go the culture is to behave and risk that.
But staying at 24 bytes, we're never worse than net.IP and strictly better in all regards. (at least by the metrics we highlighted in the blog post... perhaps not by "don't play unsafe shenanigans with IPv6 zone interning")
Fair enough. I agree 8x for IPv4 would be hard to swallow.
In either case, having the interning wrapped up in a reusable library is great.
In either case, having the interning wrapped up in a reusable library is great.
> Especially considering how rare IPv6+zones are.
Who uses IPv6+zones and why?
I'm confused what zones are about. They don't seem to be part of IPv6 packets. Where do they come from? Why does IP type have to deal with them?
Who uses IPv6+zones and why?
I'm confused what zones are about. They don't seem to be part of IPv6 packets. Where do they come from? Why does IP type have to deal with them?
A scoped IPv6 address contains a network interface name, which is used to disambiguate which network a link-local address belongs to.
It’s relatively common to encounter them in some network configurations where the routers use link-local addresses rather than global addresses in their RA packets. They are also used in SLAAC, because a device can configure a link-local address without knowing anything about its environment.
[ I think in the past there might have been an idea to use them to disambiguate site-local IPv6 addresses, but network interfaces are not sufficient in that case, and site-local addresses were problematic in lots of other ways, so they have been dropped from the specifications. ]
It’s relatively common to encounter them in some network configurations where the routers use link-local addresses rather than global addresses in their RA packets. They are also used in SLAAC, because a device can configure a link-local address without knowing anything about its environment.
[ I think in the past there might have been an idea to use them to disambiguate site-local IPv6 addresses, but network interfaces are not sufficient in that case, and site-local addresses were problematic in lots of other ways, so they have been dropped from the specifications. ]
One answer to "who?" question: https://news.ycombinator.com/item?id=26419035
Still unclear why and where do zones come from.
Still unclear why and where do zones come from.
This was helpful:
https://en.wikipedia.org/wiki/IPv6_address#Scoped_literal_IP...
Apparently it's possible to embed interface name into an address:
fe80::1ff:fe23:4567:890a%eth2
The article also points to an alternative solution - embed interface index into an address:
fe80:3::1ff:fe23:4567:890a
https://en.wikipedia.org/wiki/IPv6_address#Scoped_literal_IP...
Apparently it's possible to embed interface name into an address:
fe80::1ff:fe23:4567:890a%eth2
The article also points to an alternative solution - embed interface index into an address:
fe80:3::1ff:fe23:4567:890a
> The story begins in April 2019 with 89476f8cb5 in which David Crawshaw, aware of all these problems, created an IP type like:
It's in a PR here: https://github.com/WireGuard/wireguard-go/pull/11/files
Which mentions:
> Typically throughout WireGuard, we've used [4]byte for v4 and [16]byte for v6, considering the Go standard library's choice of v6-mapped-v4 to be a mistake.
It actually looks useful, there's something similar in wgtypes, and I wanted to use it recently [0] but that package unfortunately doesn't expose a way to write it out to a string/buffer, so I ended up almost replicating it.
[0] https://github.com/OJFord/terraform-provider-wireguard/blob/...
(There's a lot of ways of presenting/parsing/constructing the same config options there! I'm sure if I were more fluent in Go there's some more succinct way.)
It's in a PR here: https://github.com/WireGuard/wireguard-go/pull/11/files
Which mentions:
> Typically throughout WireGuard, we've used [4]byte for v4 and [16]byte for v6, considering the Go standard library's choice of v6-mapped-v4 to be a mistake.
It actually looks useful, there's something similar in wgtypes, and I wanted to use it recently [0] but that package unfortunately doesn't expose a way to write it out to a string/buffer, so I ended up almost replicating it.
[0] https://github.com/OJFord/terraform-provider-wireguard/blob/...
(There's a lot of ways of presenting/parsing/constructing the same config options there! I'm sure if I were more fluent in Go there's some more succinct way.)
"But then I thought of something gross. Or awesome."
The programmer's mantra.
The programmer's mantra.
> Basically, it’s playing unsafe games behind the Go garbage collector’s back, hiding pointers in untyped uintptr integers so Go will be forced to eventually garbage collect things which then causes the finalizer to be invoked to step in and either clean up its lies or clean up the map.
I love this. I hate to love this. It has that fast inverse sqrt flavor to it. Does some crazy stuff, but cordons off the well-documented crazy behind a clean API. Bravo! Golang "worse is better" at it's finest.
Gonna send this to my FP-obsessed friend who thinks humans can never be trusted to write code like this and all future languages should forbid any kind of manual memory management. I told him "have fun writing a kernel or driver in that kind of language", he wasn't having it. I'm sure this will irk him to no end.
I love this. I hate to love this. It has that fast inverse sqrt flavor to it. Does some crazy stuff, but cordons off the well-documented crazy behind a clean API. Bravo! Golang "worse is better" at it's finest.
Gonna send this to my FP-obsessed friend who thinks humans can never be trusted to write code like this and all future languages should forbid any kind of manual memory management. I told him "have fun writing a kernel or driver in that kind of language", he wasn't having it. I'm sure this will irk him to no end.
I don't understand why SetFinalizer+resurrected could prevent a memory block which will be collected from being collected.
Update: never mind, I re-read the docs of SetFinalizer to get the explanation:
Update: never mind, I re-read the docs of SetFinalizer to get the explanation:
SetFinalizer sets the finalizer associated with obj to the provided
finalizer function. When the garbage collector finds an unreachable block
with an associated finalizer, it clears the association and runs
finalizer(obj) in a separate goroutine. This makes obj reachable again,
but now without an associated finalizer. Assuming that SetFinalizer
is not called again, the next time the garbage collector sees
that obj is unreachable, it will free obj.Then we will refer you to Midori powering the Asian nodes for Bing, until Microsoft management decided to pull the plug.
Or how F-Secure is shipping secure USB keys with bare metal Go firmware.
Or how F-Secure is shipping secure USB keys with bare metal Go firmware.
The brittle string interning implementation could be replaced with a proposed Golang weak reference map [0].
[0] https://github.com/golang/go/issues/43615
[0] https://github.com/golang/go/issues/43615
Year and a half of work. Impressive. I wonder if it’s actually worth it for a company to spend such amount of time on stuff like this. Nevertheless, I really appreciate it’s open source.
Perhaps I missed something (and I don’t know Go at all), but if you’re keeping a background map in-sync for the pointer equality stuff to work out, doesn’t that imply it’s not actually 0-allocation? Or is this a go’ism where allocations refer to some subset of things?
It's a mild tradeoff the post glossed over: if you're handling IPv6 addresses with a zone specifier, the first use of a particular zone specifier will cause an allocation to intern that string.
Thankfully, the vast majority of IP-wielding programs never touch zones, and IPs sans zone specifiers are always allocation-free. And even for the long tail of zone-aware programs (e.g. corerad, a router advertisement daemon), you end up with one allocation per unique zone string, not one per IP value, which generally amortizes to alloc-free in steady state.
Thankfully, the vast majority of IP-wielding programs never touch zones, and IPs sans zone specifiers are always allocation-free. And even for the long tail of zone-aware programs (e.g. corerad, a router advertisement daemon), you end up with one allocation per unique zone string, not one per IP value, which generally amortizes to alloc-free in steady state.
That's only used for the IPv6 scopes and it's amortized over all parses to effectively zero. Only allocates once per unique new scope name seen.
I could clarify in the post.
I could clarify in the post.
I don't quite understand why you need the messy/unsafe string interning, instead of a simple (mutex-protected) map with the m[string(b)] optimization that avoids allocations, or josharian's sync.Pool approach? https://commaok.xyz/post/intern-strings/
A simple map was discussed in the post. The problem it has is that it has unbounded growth. The sync.Pool approach is a best effort to avoid allocations only. If its API was changed to return sentinel pointers rather than the string, it wouldn't have the guarantee that Intern(x) == Intern(x) when x == x.
Thanks. I re-read this paragraph from the original article and it makes more sense now:
> use a zone mapping table, mapping between zone index integers and zone name strings. That’s what the Go standard library does internally. But then we’re left susceptible to an attack where an adversary forces us to parse a bunch of IP addresses with scopes and we forever a bloat a mapping table that we don’t have a good opportunity to ever shrink. The Go standard library doesn’t need to deal with this, as it only ever maps interfaces that exist on the machine and doesn’t expose the integers to users in representations; its net.IPAddr.Zone field is a string.
> use a zone mapping table, mapping between zone index integers and zone name strings. That’s what the Go standard library does internally. But then we’re left susceptible to an attack where an adversary forces us to parse a bunch of IP addresses with scopes and we forever a bloat a mapping table that we don’t have a good opportunity to ever shrink. The Go standard library doesn’t need to deal with this, as it only ever maps interfaces that exist on the machine and doesn’t expose the integers to users in representations; its net.IPAddr.Zone field is a string.
Whoops, I just fixed that typo too. (not yet deployed)
I know nothing about Go or IP, but i do know Erlang where atoms (= interned strings) are a little bit of a security risk because it makes it very easy for an attacker to cause out-of-memory by flooding an endpoint that converts input strings to atoms with random data.
Is that a risk here? Could i connect to a Go service that uses this library with a patched IP client and somehow push random data into that "zone" string and crash the server?
Is that a risk here? Could i connect to a Go service that uses this library with a patched IP client and somehow push random data into that "zone" string and crash the server?
Cool! One thing that confused me, though, is why they used the inet.af import path. I get that it looks cool, but it'd be better if they just used tailscale.com for everything.
On a related note, I'm not sure which problems Tailscale aims to solve (although I suspect it's one of those "you'll know when you need it" tools).
On a related note, I'm not sure which problems Tailscale aims to solve (although I suspect it's one of those "you'll know when you need it" tools).
> Cool! One thing that confused me, though, is why they used the inet.af import path. I get that it looks cool, but it'd be better if they just used tailscale.com for everything.
Why would it be better to use tailscale.com?
It has nothing to do with Tailscale. It's just an IP address.
A neutral package name seemed best.
Why would it be better to use tailscale.com?
It has nothing to do with Tailscale. It's just an IP address.
A neutral package name seemed best.
Love the redirect from inet.af to the POSIX socket spec, nice easter egg :)
This should point to AF_INET flag which you pass when creating a socket, and means that the address family is the internet (IP) address, as opposed to e.g. a domain socket, or some other transport.
Isn't intern essentially the behavior of a Ruby symbol? If that's the case it's a very powerful package, no doubt
Nitpick: the correct title of the blog post starts with "netaddr.IP", but the one currently on HN starts with "netaddr.ip".
The former looks like the name of a Go type (which is what it is) and the latter looks like a domain name or something.
The former looks like the name of a Go type (which is what it is) and the latter looks like a domain name or something.
What an insane amount of work for something that would be fixed, in a better language, by better value types and algebraic enums.
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."
https://news.ycombinator.com/newsguidelines.html
https://news.ycombinator.com/newsguidelines.html
5 days of work is insane for you?
No, the time isn't really what I'm referring to—ip addressing is complicated. But it shouldn't require having to work around the language to do this kind of thing.
I'm not sure how this is a "work around the language" rather then the standard library in Golang is allocation happy for IP types. I imagine this would look very similar in almost any language.
For example, look at rust's [0]. It claims to vary in size depending on os, but on Linux, at least, it's 20 bytes.
[0]: https://doc.rust-lang.org/std/net/enum.IpAddr.html
[0]: https://doc.rust-lang.org/std/net/enum.IpAddr.html
This type doesn't appear to support IPv6 Address Zones which was the most complicated part of the post. Actually from my limited googling Rust doesn't seem to support Address Zones at all in it's standard library.
That's a good point.
It also doesn't include the ipv6 zone scope. Adding that would take it to at least 28 bytes, and still have to handle all the interning problems for cheap equality.
Appears to be 16 bytes on Windows (larger of the two variants): https://github.com/rust-lang/rust/blob/master/library/std/sr...
The entire section about comparisons and string interning is extremely "work around the language".
Interning is to reduce memory usage. I will agree the uintptr trick is a work around for the lack of weak refs in golang but the code isn't that different then what you would write if you have a referenced counted cache.
I trust Brad and the team’s take on things. If they decide they needed a new library for this then so be it.
> [it improves upon] 28-56 bytes [of IPv4 record size]
that's still 28-56 MiB for 1024000 (~1M) IP records. Who needs to manipulate more than that?
that's still 28-56 MiB for 1024000 (~1M) IP records. Who needs to manipulate more than that?
Google, Cisco, Microsoft, AWS, Linux, OpenBSD, ...
... Tailscale...
You may not be in the class. I only live next to people in that class, I'm not in it myself. But it exists.
... Tailscale...
You may not be in the class. I only live next to people in that class, I'm not in it myself. But it exists.
[deleted]
I would think that once you start working with even half that many ips it starts to make sense to use a trie-like data structure that shares common pieces of the ips.
That doesn't help much if IP addresses are randomly distributed, as they are in a lot of our use cases. You use tries generally to do range and prefix lookups, not so much as a storage trick.
Network code tends to be performance critical, so the more you can fit into a cache line the better. Exact numbers depend on the application, of course, but those memory overheads can give you a very serious performance hit in cache-heavy applications.
Size is a bonus though, the primary push seems to be all the other issues. Either way imho, needless bloat should be shorn regardless of the ability to accommodate it.
You should also think in terms of CPU caches.
Well they manage around 500 addresses for me, and I assume they'd like more than 2k users like me some day.
Also, we run on platforms like iOS, where we have very severe memory constraints (the NetworkExtension is allowed 15MiB of total RAM). Given that we juggle a lot of IP-related data types, even modest optimizations help.
That said, I'd think of the size thing as more of a constraint on the other desired features. We wanted the other nice things (less GC pressure, no allocations, comparable type...) while also being _lean enough_ that there's no reason not to use the netaddr types.
That said, I'd think of the size thing as more of a constraint on the other desired features. We wanted the other nice things (less GC pressure, no allocations, comparable type...) while also being _lean enough_ that there's no reason not to use the netaddr types.
Do you have to do anything special from a runtime instance perspective to keep it within 15MB? That seems really small for a GCed language.
Mostly we optimize at the application layer (e.g. we do lazy on-demand WireGuard tunnel configuration, to minimize how much state sits around idle). We also have a small fork of Go with some space-optimizing changes: https://github.com/golang/go/compare/master...tailscale:tail... (most of these changes get upstreamed, albeit delayed by one release cycle).
Aside from that, Go's also just not a big garbage generator in general, compared to older GC'd languages like Java, so with a little care you can keep the pressure off.
That said, if anyone at Apple's listening, it would sure be nice to have, say, 30MiB ;). 15MiB makes it possible to cram a whole mesh networking stack in, but doesn't make it fun.
Aside from that, Go's also just not a big garbage generator in general, compared to older GC'd languages like Java, so with a little care you can keep the pressure off.
That said, if anyone at Apple's listening, it would sure be nice to have, say, 30MiB ;). 15MiB makes it possible to cram a whole mesh networking stack in, but doesn't make it fun.
It's less that you need to store many of them at the same time, but that it increases the amount of garbage collections. If you can create half as much garbage per packet or request, that's double the time until you hit the threshold for gc invocation.
How the heck does one even turn an IPv4 address into 28-56 bytes?
Obviously it "should" be exactly 4 bytes, but even if you store the whole thing as an ASCII string it's only up to 16 bytes, including a null terminator.
UTF-16, perhaps? LOL
Obviously it "should" be exactly 4 bytes, but even if you store the whole thing as an ASCII string it's only up to 16 bytes, including a null terminator.
UTF-16, perhaps? LOL
> How the heck does one even turn an IPv4 address into 28-56 bytes?
It's covered in the blog post.
It's covered in the blog post.
"It’s large. A Go slice is 3 words (so 24 bytes total on 64-bit machines) just for the slice header, without counting the underlying array that the slice points to"
from the article:
> The underlying type of a net.IP is just a []byte
> ...
> It’s large. A Go slice is 3 words (so 24 bytes total on 64-bit machines) just for the slice header, without counting the underlying array that the slice points to (background). So an IP address with Go’s net.IP is two parts: the 24 byte slice header, and then also the 4 or 16 bytes of IP address. If you want an IPv6 zone, you have to use net.IPAddr with a 16 byte string header also.
> The underlying type of a net.IP is just a []byte
> ...
> It’s large. A Go slice is 3 words (so 24 bytes total on 64-bit machines) just for the slice header, without counting the underlying array that the slice points to (background). So an IP address with Go’s net.IP is two parts: the 24 byte slice header, and then also the 4 or 16 bytes of IP address. If you want an IPv6 zone, you have to use net.IPAddr with a 16 byte string header also.
Thanks, that's a bit interesting. I only know a little bit of Go, but I would have assumed you could just use an array ([4]byte) directly without a slice ([]byte) on top of it. Guess not.
[deleted]
Web infrastructure companies keeping track of hundreds of millions of users.
I really wonder whether there is an executive at Google who has some connection to revenue for Google and how control of Golang ties to that. And, whether Brad leaving makes them really nervous. Google has never, AFAIK, invested in Jetbrains, but they care a lot about Android Studio and Kotlin. It's interesting to see this kind of post and wonder if this kind of work makes Google nervous at all. That would be a good book I'd like to read in 10 or 20 years.