In Go, pointers (mostly) don't go with slices in practice(utcc.utoronto.ca)
utcc.utoronto.ca
In Go, pointers (mostly) don't go with slices in practice
https://utcc.utoronto.ca/~cks/space/blog/programming/GoSlicesVsPointers
97 comments
> This is safe, but confuses some people.
Rust has created a weird perception that memory safety equals safety. Language is a tool and it should work with me: it is extremely important that my understanding of what the program should do aligns with what it actually does.
The way you describe go's behavior is "takes snapshot of the underlying data", which usually means "deep copy container". Taking a pointer/reference usually means quite an opposite. So it is "safe" in a sense that the pointer points to valid data, but is "incorrect" in a sense that it does wrong thing without warning.
Sure, one could argue that value-returning modification functions are a giveaway of invalidated data. But this is not C, go has reference counting and instead of "forcing" underlying array to maintain the same address it just keeps original pointer pointing to dereferenceable, but wrong data.
Rust has created a weird perception that memory safety equals safety. Language is a tool and it should work with me: it is extremely important that my understanding of what the program should do aligns with what it actually does.
The way you describe go's behavior is "takes snapshot of the underlying data", which usually means "deep copy container". Taking a pointer/reference usually means quite an opposite. So it is "safe" in a sense that the pointer points to valid data, but is "incorrect" in a sense that it does wrong thing without warning.
Sure, one could argue that value-returning modification functions are a giveaway of invalidated data. But this is not C, go has reference counting and instead of "forcing" underlying array to maintain the same address it just keeps original pointer pointing to dereferenceable, but wrong data.
This is how I think about Go slices (may help other understand them).
A slice itself is just a window into a backing array of fixed size. The slice carries three data members. The pointer to the backing array and its remaining capacity and the length of the slice data.
Typically slices are passed around by value but you can take their address and modify a "shared" slice.
The built-in append() returns a new slice by value.
What happens is simply that when appending data to a slice and there is no room in the backing array, a new backing array is allocated that the returned slice points into. The old "input" slice to append is still intact and if some code has access to it, it will look at data stored in the old backing array.
I've constructed similar utility types in C and find them quite convenient. It's very convenient to have the distinction between the backing memory (array) and a slice viewing a portion of it instead of just a dynamic array.
A slice itself is just a window into a backing array of fixed size. The slice carries three data members. The pointer to the backing array and its remaining capacity and the length of the slice data.
Typically slices are passed around by value but you can take their address and modify a "shared" slice.
The built-in append() returns a new slice by value.
What happens is simply that when appending data to a slice and there is no room in the backing array, a new backing array is allocated that the returned slice points into. The old "input" slice to append is still intact and if some code has access to it, it will look at data stored in the old backing array.
I've constructed similar utility types in C and find them quite convenient. It's very convenient to have the distinction between the backing memory (array) and a slice viewing a portion of it instead of just a dynamic array.
> I've constructed similar utility types in C and find them quite convenient. It's very convenient to have the distinction between the backing memory (array) and a slice viewing a portion of it instead of just a dynamic array.
Of course that's convenient, the issue of Go's slices is that they act as both a dynamic array and a slice viewing a portion of one. The two uses conflict with one another, and the interactions are full of traps.
Of course that's convenient, the issue of Go's slices is that they act as both a dynamic array and a slice viewing a portion of one. The two uses conflict with one another, and the interactions are full of traps.
If it were true in the typical sense that “append() returns a new slice by value”, then you would expect to be able to mutate the old slice and the new slice independently from each other. But in reality, you can only do this if append() decided to reallocate, which only happens at some implementation-defined exponential pattern of sizes.
package main
import "fmt"
func main() {
a := []int{0}
for i := 0; i < 40; i++ {
b := append(a, 0)
a[i] = 1
fmt.Printf("%d ", b[i])
a = b
}
}
Does that mean that a) the pointer to the single element is only invalidated on “append” if the slice has no more capacity (this is how C++ vectors work), or b) does the fact that there’s an active reference into the slice always cause a reallocation (copy-on-write style)? If it’s the former, we’re literally in the C++ iterator invalidation nightmare, only without the debugging tools.
The array is only reallocated when it runs out of space. There is no magic about the "old" array. The GC just won't collect it, as long as a pointer to it exists.
I don't think there are many reasons, if any at all, to keep a pointer to an array element of a slice around in Go. Usually I only get the address of an array element only when passing it to some C code or doing some low level manipulation, but then I don't keep the pointer around. In Go code you usually just keep the slice object - which contains the necessary pointer anyway.
I don't think there are many reasons, if any at all, to keep a pointer to an array element of a slice around in Go. Usually I only get the address of an array element only when passing it to some C code or doing some low level manipulation, but then I don't keep the pointer around. In Go code you usually just keep the slice object - which contains the necessary pointer anyway.
Thank you for clarifying this. I was scratching my head reading this discussion, wondering why one would do this in the first place.
Without knowing much Go I believe it's neither a nor b. The pointer to the single element will always stay valid, no matter whether reallocation happens or not (and having a pointer doesn't influence whether reallocation happens or not).
Re-allocation might be a confusing word here because afaik it's actually always a new allocation (the old one is not touched) and only if there are no more pointers to the old allocation will the next GC cycle deallocate it.
So there is never iterator invalidation like in C++ but of course you still need to be careful because you might accidentally share or not share the same underlying data.
This is correct. A pointer like `p := &x[0]` will always point at the original backing array even if an append on the slice causes the slice to allocate a new backing array. This means that you can update `x[0]` on the new slice without changing `*p`.
https://play.golang.org/p/Hl58VW-Yvhn
https://play.golang.org/p/Hl58VW-Yvhn
AFAIK, neither.
Since slice API is pass-by-value, in theory ANY method will invalidate the pointer. In practice only resizing methods actually NEED to reallocate the underlying array, but magic can happen. However, refcounting will make sure that a previously underlying array having pointers to it will remain allocated. This means that 1. pointers to single elements will always dereference 2. slice structure modification can leave pointers pointing to stale data
Since slice API is pass-by-value, in theory ANY method will invalidate the pointer. In practice only resizing methods actually NEED to reallocate the underlying array, but magic can happen. However, refcounting will make sure that a previously underlying array having pointers to it will remain allocated. This means that 1. pointers to single elements will always dereference 2. slice structure modification can leave pointers pointing to stale data
Yes, this is entirely correct.
> Rust has created a weird perception that memory safety equals safety.
I think it's a bit more than that. They're also riding on the static typing trend that's happening right now, so type-safety is also part of the equation. From the website:
> A language empowering everyone to build reliable and efficient software.
> Reliability: Rust’s rich type system and ownership model guarantee memory-safety and thread-safety — enabling you to eliminate many classes of bugs at compile-time.
That means that people like me that still have a hard time with C and C++ can build efficient software using the same workflow as I'm used to in my usual "web languages" (Python and JS mostly).
I think it's a bit more than that. They're also riding on the static typing trend that's happening right now, so type-safety is also part of the equation. From the website:
> A language empowering everyone to build reliable and efficient software.
> Reliability: Rust’s rich type system and ownership model guarantee memory-safety and thread-safety — enabling you to eliminate many classes of bugs at compile-time.
That means that people like me that still have a hard time with C and C++ can build efficient software using the same workflow as I'm used to in my usual "web languages" (Python and JS mostly).
> They're also riding on the static typing trend that's happening right now, so type-safety is also part of the equation
I think the "static typing trend" is a product of Rust and Go showing people that static typing doesn't have to be cumbersome like it was in 90s-00s Java, C++, and C#. Indeed, I suspect that the quality of life improvements that Java, C#, and C++ made also improved the stock price of static typing (and building on that foundation, things like TypeScript are exposing JavaScript developers to the utility of types). Which is to say, static typing isn't an empty trend or fad (no idea if that's your intended meaning) but rather people were previously averse to static typing because the mainstream statically typed languages weren't ergonomic and people assumed that the bad ergonomics was caused by static typing--now we have many mainstream languages that show that this isn't the case.
I think the "static typing trend" is a product of Rust and Go showing people that static typing doesn't have to be cumbersome like it was in 90s-00s Java, C++, and C#. Indeed, I suspect that the quality of life improvements that Java, C#, and C++ made also improved the stock price of static typing (and building on that foundation, things like TypeScript are exposing JavaScript developers to the utility of types). Which is to say, static typing isn't an empty trend or fad (no idea if that's your intended meaning) but rather people were previously averse to static typing because the mainstream statically typed languages weren't ergonomic and people assumed that the bad ergonomics was caused by static typing--now we have many mainstream languages that show that this isn't the case.
That wasn't my intended meaning, I don't think static typing is a "fad". I think the "new" typed languages (Go, Rust, Typescript) are more ergonomic than 90s-00s Java, C++, C#, as you said. This is also forcing them to improve, with features like type inference, sealed classes, records. I also think that the combination of gradual typing and type inference is playing a big role in the adoption.
However, I called static typing a "trend", and I'll try to explain why. I think attempts to type Python, JS, Ruby and the popularity of Go and Rust are the natural consequences of people departing from the Java/C++ ecosystem 10-20 years earlier (for good reasons). Now they are rediscovering the good parts of this ecosystem (ease of deployment with binaries/fat jars, static typing, performance). Since Twitter, Github, Youtube, Shopify, Instagram, etc have all that code around, they are going to either improve it, or try to migrate from it. For example, Shopify is working on a compiler to native for Ruby based on LLVM https://sorbet.org/blog/2021/07/30/open-sourcing-sorbet-comp.... Instagram is working on a performance-oriented CPython fork https://github.com/facebookincubator/cinder. Twitter, from what I understand, went back to Java, going through Scala first (which is another example of "better type system"). KhanAcademy is migrating services from a Django monolith to Go services https://blog.khanacademy.org/half-a-million-lines-of-go/. Whatsapp even had a project to do a statically typed "Erlang 2".
The "trend" here is that some companies that use "new" dynamic languages in the 00s are now very large companies that have enough money to invest in language, tooling and things like that.
However, I called static typing a "trend", and I'll try to explain why. I think attempts to type Python, JS, Ruby and the popularity of Go and Rust are the natural consequences of people departing from the Java/C++ ecosystem 10-20 years earlier (for good reasons). Now they are rediscovering the good parts of this ecosystem (ease of deployment with binaries/fat jars, static typing, performance). Since Twitter, Github, Youtube, Shopify, Instagram, etc have all that code around, they are going to either improve it, or try to migrate from it. For example, Shopify is working on a compiler to native for Ruby based on LLVM https://sorbet.org/blog/2021/07/30/open-sourcing-sorbet-comp.... Instagram is working on a performance-oriented CPython fork https://github.com/facebookincubator/cinder. Twitter, from what I understand, went back to Java, going through Scala first (which is another example of "better type system"). KhanAcademy is migrating services from a Django monolith to Go services https://blog.khanacademy.org/half-a-million-lines-of-go/. Whatsapp even had a project to do a statically typed "Erlang 2".
The "trend" here is that some companies that use "new" dynamic languages in the 00s are now very large companies that have enough money to invest in language, tooling and things like that.
Makes sense. Thanks for clarifying!
The problem here is identical to the problem of pointers to array elements in C after a 'realloc', except that Go at least guarantees that you're not going to modify some other object's memory.
Of course, since append neither guarantees nor prevents a copy, the semantics of modifying a value through a pointer to a slice element after an append are unspecified, so it is not a useful construct.
Of course, since append neither guarantees nor prevents a copy, the semantics of modifying a value through a pointer to a slice element after an append are unspecified, so it is not a useful construct.
> except that Go at least guarantees that you're not going to modify some other object's memory.
Or that you’re way off in UB (UAF) land.
Or that you’re way off in UB (UAF) land.
> The way you describe go's behavior is "takes snapshot of the underlying data", which usually means "deep copy container"
No, there is no mention of a "snapshot". You get a reference to the current backing array, which may or may not continue being used by the slice (depending on reallocations). You're pointing to the live slice backing array, and the values in it may change if someone else is manipulating the slice, up to the point where the slice backing array must be reallocated, at which point you'll continue pointing to the old backing array and be keeping it from getting GC'd.
No, there is no mention of a "snapshot". You get a reference to the current backing array, which may or may not continue being used by the slice (depending on reallocations). You're pointing to the live slice backing array, and the values in it may change if someone else is manipulating the slice, up to the point where the slice backing array must be reallocated, at which point you'll continue pointing to the old backing array and be keeping it from getting GC'd.
And that’s really the problem with it. If you want to ensure that you have exclusive access to the element(s), then you have to explicitly copy them first or you get silent data corruption.
And if you want to ensure that multiple things have access to the elements, then you have to avoid reallocations or you get silent data loss.
No matter what you’re doing, a pointer to an element of an array or slice is usually the wrong thing in Go. The language would be better off without them.
And if you want to ensure that multiple things have access to the elements, then you have to avoid reallocations or you get silent data loss.
No matter what you’re doing, a pointer to an element of an array or slice is usually the wrong thing in Go. The language would be better off without them.
> it does wrong thing without warning
It's not without warning, it's a well documented behavior.
Just think of slices as "immutable", "pass-by-value" data structures (with a relatively efficient implementation) and everything falls into place.
Mutating them in any way is actually a special case that you do only for performance reason (i.e. you can pre-allocate and fill if you know the size ahead of time) but - as always - you try to keep those abstracted away and to the minimum.
It's not without warning, it's a well documented behavior.
Just think of slices as "immutable", "pass-by-value" data structures (with a relatively efficient implementation) and everything falls into place.
Mutating them in any way is actually a special case that you do only for performance reason (i.e. you can pre-allocate and fill if you know the size ahead of time) but - as always - you try to keep those abstracted away and to the minimum.
> It's not without warning, it's a well documented behavior.
Ah yes, the usual excuse for it being fine that C APIs are completely broken and half of them can not be used correctly.
> Just think of slices as "immutable", "pass-by-value" data structures (with a relatively efficient implementation) and everything falls into place.
Except when they don't because `append` itself amortises allocations, which means if you treat slices as immutable and pass by value you will end up with slices sharing a backing array with leftover capacity and stomping on one another's data.
> Mutating them in any way is actually a special case that you do only for performance reason
Mutating them is literally what the normal Go API usage has you do. If you want to avoid mutating slices you need to write this abortion:
Ah yes, the usual excuse for it being fine that C APIs are completely broken and half of them can not be used correctly.
> Just think of slices as "immutable", "pass-by-value" data structures (with a relatively efficient implementation) and everything falls into place.
Except when they don't because `append` itself amortises allocations, which means if you treat slices as immutable and pass by value you will end up with slices sharing a backing array with leftover capacity and stomping on one another's data.
> Mutating them in any way is actually a special case that you do only for performance reason
Mutating them is literally what the normal Go API usage has you do. If you want to avoid mutating slices you need to write this abortion:
s1 := append(append([]int(nil), s0...), item)
Ok, I think I just understood the problem.
The following would cause a problem:
s0 := append([]string{}, "zoo")
sa := append(s0, "foo")
sb := append(s0, "bar") // overwrites sa[1]
> The way you describe go's behavior is "takes snapshot of the underlying data", which usually means "deep copy container".
There’s no need for Go to copy anything in the circumstance the OP described. It just doesn’t shrink the underlying array.
There’s no need for Go to copy anything in the circumstance the OP described. It just doesn’t shrink the underlying array.
Go tells you very explicitly how it resizes slices by forcing you to write this:
slice = append(slice, item)
> Go tells you very explicitly how it resizes slices by forcing you to write this:
No, what most readers intuit from that is that `append` performs no mutation and that this is fine:
Except it's absolutely not fine.
That's why other language separate slices and vectors and avoid confusing two objects which have different behaviours and uses even if their representation is very similar.
No, what most readers intuit from that is that `append` performs no mutation and that this is fine:
s2 := append(s1, item)
(def s2 (conj s1 item))
Except it's absolutely not fine.
That's why other language separate slices and vectors and avoid confusing two objects which have different behaviours and uses even if their representation is very similar.
This is fine:
> other language separate slices and vectors
Go has a separate type for arrays, which is a value type
s2 := append(s1, item)
> other language separate slices and vectors
Go has a separate type for arrays, which is a value type
var arr [10]int> This is fine:
No, it is not.
> It's only not fine if you some how assume that s2 and s1 will always refer to the same data.
It's not fine if you assume anything about the interaction of s1 and s2. It's not fine if you assume they do alias, it's not fine if you assume they don't.
There is, fundamentally, no situation in which that construct is anything other than a footgun. `append` should only ever be used with the same slice on the LHS and RHS[0], or a brand new slice object constructed for the occasion on the RHS.
> Go has a separate type for arrays, which is a value type
An array is neither a vector nor a slice.
The issue is that Go's slices serve as both a vector and an actual slice, and the union of these interfaces creates footguns which don't exist in either.
[0] IFF that RHS is a either a non-parameter local, or a pointer to a slice
No, it is not.
> It's only not fine if you some how assume that s2 and s1 will always refer to the same data.
It's not fine if you assume anything about the interaction of s1 and s2. It's not fine if you assume they do alias, it's not fine if you assume they don't.
There is, fundamentally, no situation in which that construct is anything other than a footgun. `append` should only ever be used with the same slice on the LHS and RHS[0], or a brand new slice object constructed for the occasion on the RHS.
> Go has a separate type for arrays, which is a value type
An array is neither a vector nor a slice.
The issue is that Go's slices serve as both a vector and an actual slice, and the union of these interfaces creates footguns which don't exist in either.
[0] IFF that RHS is a either a non-parameter local, or a pointer to a slice
the idea that append performs no mutation is fairly insane. its name implies a mutation.
No, it does not. Appending is the addition of a suffix, it does not say anything about how it works.
Example: https://hackage.haskell.org/package/bytestring-0.11.1.0/docs...
Example: https://hackage.haskell.org/package/bytestring-0.11.1.0/docs...
if you add a suffix, you've changed the value. that is a mutation.
In Go, a []int ("slice of int") is just a C struct like this, passed by value:
This can be confusing if you're used to C++'s std::vector (which owns the memory) or Python's slices. Go's slices are a shallow pointer/length system exactly like is used in C all the time. For example:
struct intSlice {
int* addr;
int len;
int cap;
};
This can be confusing if you're used to C++'s std::vector (which owns the memory) or Python's slices. Go's slices are a shallow pointer/length system exactly like is used in C all the time. For example:
void sort(int* addr, int len);
func sort(a []int)
In Pascal, there are no slices
No slices, no problems
If you need to work with a part of a string, you can make two ordinary integer variables for offset and length
No slices, no problems
If you need to work with a part of a string, you can make two ordinary integer variables for offset and length
Cool
Actually, Pascal has slices
They are just so obscure, I forgot about them and no one uses them. No users, no problems
They are not part of the normal type system. You cannot declare a variable of a type slice. Nor a field. But when a parameter of a function is an (open) array, you can call the function with a slice of an existing array
That avoids most problems
The backing array exists when the function is called, and the function cannot store the slice, so the slice cannot outlive the array. It is like the function borrows the array. Only problem is if the function gets another reference to the array, through a global variable or something, and resizes it
They are just so obscure, I forgot about them and no one uses them. No users, no problems
They are not part of the normal type system. You cannot declare a variable of a type slice. Nor a field. But when a parameter of a function is an (open) array, you can call the function with a slice of an existing array
That avoids most problems
The backing array exists when the function is called, and the function cannot store the slice, so the slice cannot outlive the array. It is like the function borrows the array. Only problem is if the function gets another reference to the array, through a global variable or something, and resizes it
The author seems confused. The following is simply not true:
The author seems to have failed to distinguish the operation of copying a fat pointer (which opens the possibility of divergence) and the operation of the taking the address of a fat pointer (which involves no copying, so divergence is not possible - where would the divergent version be stored?).
See this code snippet: https://play.golang.org/p/tdb-O8a6hDN
When you take a pointer to a slice, you get a pointer to the current version of this tuple of information for the slice. This pointer may or may not refer to a slice that anyone else is using; for instance:
ps := &s
s = append(s, 50)
At this point, '*ps' may or may not be the same thing as 's', and so it might or might not have the new '50' element at the end.
The author seems to have failed to distinguish the operation of copying a fat pointer (which opens the possibility of divergence) and the operation of the taking the address of a fat pointer (which involves no copying, so divergence is not possible - where would the divergent version be stored?).
See this code snippet: https://play.golang.org/p/tdb-O8a6hDN
Yep, that immediately stood out to me too.
s is a local variable (or global, doesn't matter). ps simply points to that local variable. You can modify the local variable all day long and ps will still point to it, not some old version of it.
s is a local variable (or global, doesn't matter). ps simply points to that local variable. You can modify the local variable all day long and ps will still point to it, not some old version of it.
the line `s = append(s, 50)` redefines what "s" actually is. And after this line `ps` points to some previous version of what "s" used to be.
No, that line modifies the value of the variable s to represent the value of a new slice returned by append (assuming append did need to reallocate). Any pointer to s will point to this new value.
A variable in Go always maintains its address after it is allocated. Assignments to that variable copy the assigned value to the original address.
Somewhat unhelpfully, this rule is even true for iteration variable - when you write 'for i,v := range arr {...}', i and v get allocated a memory address, and they get successively assigned the indices and values in arr. This implies that each element in arr is copied into the value of v, and that doing &v inside the loop gives you a completely different pointer than &arr[i]. In fact, &v will always point to the last element of arr after the loop is over.
A variable in Go always maintains its address after it is allocated. Assignments to that variable copy the assigned value to the original address.
Somewhat unhelpfully, this rule is even true for iteration variable - when you write 'for i,v := range arr {...}', i and v get allocated a memory address, and they get successively assigned the indices and values in arr. This implies that each element in arr is copied into the value of v, and that doing &v inside the loop gives you a completely different pointer than &arr[i]. In fact, &v will always point to the last element of arr after the loop is over.
No, ps still points to s:
https://play.golang.org/p/iSjoqGTg20_O
https://play.golang.org/p/iSjoqGTg20_O
var s []int
s = append(s, 10, 20, 30)
pe := &s[0]
ps := &s
s = append(s, 50)
s[0] = 100
pe2 := &s[0]
fmt.Println("s: ", s, ", ps: ", ps, ", pe: ", pe, ", pe2: ", pe2)
// s: [100 20 30 50] , ps: &[100 20 30 50] , pe: 0xc0000be000 , pe2: 0xc0000b8030I don't get why address of slice returned from "append" does not change. Maybe in a trivial program like this the backing array can always be extended in-place, because there in memory fragmentation.
Is that still true in an app that has considerable memory pressure and has GC running now and then?
Is that still true in an app that has considerable memory pressure and has GC running now and then?
Even if slice is reallocated, the information about new reallocated slice is still stored in variable s. ps is merely pointing to that variable. The fact that the contents of the variable changed does not mean that its location has to change.
In other words, ps is a pointer to a pointer to array data. Append may change the inner pointer's value but that's about it.
In other words, ps is a pointer to a pointer to array data. Append may change the inner pointer's value but that's about it.
Append returns a value, the new slice struct generated by append. Append always generates a new struct and returns it by value, because even if the array pointed to doesn't change, the length property of the slice changes. This value is then assigned to the local variable s, which didn't change its memory location.
>Maybe in a trivial program like this the backing array can always be extended in-place,
In this example the backing array didn't get extended in-place. The first backing array starts at 0xc0000be000. After the append() call, there's a new backing array, which starts at 0xc0000b8030.
In this example the backing array didn't get extended in-place. The first backing array starts at 0xc0000be000. After the append() call, there's a new backing array, which starts at 0xc0000b8030.
Despite some syntax glosses in Go making it superficially sometimes look like a dynamic language, it is actually in the C heritage of what a variable is. A variable is not just a label pointing at an arbitrary value that can be moved to other arbitrary values like it is in most dynamic languages (like Python), it is a specific chunk of memory. (Or, more precisely, it is something that will be a specific chunk of memory if anything ever needs it to be, or the compiler decides it needs to be; recently things can in theory be register-only, but this is a detail the Go programmer operating at the Go level need not worry about.)
Specifically, what happens in
In Go, allocation is very important and it is always explicit, except this explicitness is sometimes masked by the fact that the := operator is not a simple "allocate everything on the left using the types of the thing on the right" operator, but "allocate at least one thing on the left using the types on the right (it is a compile-time error if there isn't at least one new value) but use normal equality for everything already allocated", which is very convenient to use, but can make developing the proper mental model for how Go works trickier. Arguably, there's some sort of design mistake in Go here, though the correct solution doesn't immediately leap to mind. (It's easy to come up with more abstractly correct options but they have poor usability compared to the current operator.)
Similarly, it can be easy to miss that Go, like C, cares deeply about the size of structures, and every = statement has, at compile time, full awareness by the compiler of exactly how much memory is going to be involved in that equality statement. Interfaces may make it seem like maybe I can have an "io.Reader" value, and first I set it to some struct that implements it with a small amount of RAM, then maybe later I can set it to a struct that uses a large amount of RAM, but the interface value itself is actually a two-word structure with two pointers in it that is all you are ever changing, and, again, those two words are given a specific location in RAM (possibly virtually, if you never use it they could conceivably never been out of a register, but the Go compiler and runtime will transparently make it live in RAM if you ever need the address for any reason) and any setting of the value of the variable that has an interface value will set only those two values, with no other RAM changing as a result. You can use the same io.Reader variable through its interface implementation as a "handle" on a wide variety of differently-sized values under the hood, even in the same function (I do this all the time when progressively "decorating" an interface value within a function), but the in-memory size of the handle itself never changes no matter what value you ask it to handle.
This is not intended as criticism, praise, defense, attack, or anything else on Go itself; it is descriptive of what it is.
Specifically, what happens in
s = append(s, "something")
In Go, allocation is very important and it is always explicit, except this explicitness is sometimes masked by the fact that the := operator is not a simple "allocate everything on the left using the types of the thing on the right" operator, but "allocate at least one thing on the left using the types on the right (it is a compile-time error if there isn't at least one new value) but use normal equality for everything already allocated", which is very convenient to use, but can make developing the proper mental model for how Go works trickier. Arguably, there's some sort of design mistake in Go here, though the correct solution doesn't immediately leap to mind. (It's easy to come up with more abstractly correct options but they have poor usability compared to the current operator.)
Similarly, it can be easy to miss that Go, like C, cares deeply about the size of structures, and every = statement has, at compile time, full awareness by the compiler of exactly how much memory is going to be involved in that equality statement. Interfaces may make it seem like maybe I can have an "io.Reader" value, and first I set it to some struct that implements it with a small amount of RAM, then maybe later I can set it to a struct that uses a large amount of RAM, but the interface value itself is actually a two-word structure with two pointers in it that is all you are ever changing, and, again, those two words are given a specific location in RAM (possibly virtually, if you never use it they could conceivably never been out of a register, but the Go compiler and runtime will transparently make it live in RAM if you ever need the address for any reason) and any setting of the value of the variable that has an interface value will set only those two values, with no other RAM changing as a result. You can use the same io.Reader variable through its interface implementation as a "handle" on a wide variety of differently-sized values under the hood, even in the same function (I do this all the time when progressively "decorating" an interface value within a function), but the in-memory size of the handle itself never changes no matter what value you ask it to handle.
This is not intended as criticism, praise, defense, attack, or anything else on Go itself; it is descriptive of what it is.
s changes but &s doesn't: https://play.golang.org/p/Xs1SYXqEl9i
Because there’s still space left in that slice (capacity > len), and the strategy of pre allocate capacity is to double the current (1,2,4,8,…), in this case 3 elements added => that slice was having a capacity of 4.
No, you can see that it was reallocated. At first the backing array started at 0xc0000be000. The append needed to do a reallocation and created a new backing array that starts at 0xc0000b8030.
`s` is some value that occupies some memory, starting at say address 1000. `ps` contains the address of `s`, 1000. It will do this no matter what you put in memory at address 1000. You can assign new values to `s`, e.g. via append, but ps will still point to its address, 1000.
The confusion stems from the fact that a slice object contains a pointer in itself, pointing to a backing array, thus adding an additional layer of indirection. Append will return a new slice that may point to a different backing array. This doesn't matter, because you put the new slice in the memory location pointed at by ps, 1000.
This is not so different from pointer pointers. If you have `int i = 0; int pi = &i; int *ppi = π` you can change `pi` (analogous to the slice) to your heart's content and `ppi` will reflect the changes.
The confusion stems from the fact that a slice object contains a pointer in itself, pointing to a backing array, thus adding an additional layer of indirection. Append will return a new slice that may point to a different backing array. This doesn't matter, because you put the new slice in the memory location pointed at by ps, 1000.
This is not so different from pointer pointers. If you have `int i = 0; int pi = &i; int *ppi = π` you can change `pi` (analogous to the slice) to your heart's content and `ppi` will reflect the changes.
Did you check GP's link? It demonstrates that ps points to s, not any particular version of it.
This is why passing a pointer to a slice as a function parameter is required if you want to avoid subtle bugs in functions that alter the length of the contents of slices.
To be clear the author is incorrect.
To be clear the author is incorrect.
> Honestly, this is a strange and peculiar situation, although Go programmers have acclimatized to it. To programmers from other languages, such as C or C++, the concept of pointers to dynamically extensible arrays seems like a perfectly decent idea that surely should exist and work in Go. Well, it exists, and it "works" in the sense that it yields results and doesn't crash your program, but it doesn't "work" in the sense of doing what you'd actually want.
In C++, what happens is that the "iterators are invalidated" when you add something to an array. This is CONSTANTLY a source of bugs and frustration for new programmers. In C++, it may yield results or crash your program, and you are never sure quite which will happen. The best you can do (as a senior engineer) is design your software to avoid ever creating this situation in the first place and throw address sanitizer at things to try and catch them when they arise. The difference with Go is that in Go this will never result in a memory error.
Strictly speaking, the situation in Go is way better. I will take "incorrect behavior, but not a memory error" over "memory error" any day of the week.
We may forget what it's like for new programmers, but for those of us who hang out on Discord channels, Stack Overflow, and Reddit giving people help with programming, simple things like iterator invalidation are a major pain point.
"You have a memory error in your program", I say to someone. "Now that you know that you have a memory error, it is probably your highest priority to find and fix this error." And now you start walking someone through the steps of finding and fixing a memory error, which is nontrivial. You'll tell them about Address Sanitizer, GDB, and Valgrind, and you'll wish them luck.
In C++, what happens is that the "iterators are invalidated" when you add something to an array. This is CONSTANTLY a source of bugs and frustration for new programmers. In C++, it may yield results or crash your program, and you are never sure quite which will happen. The best you can do (as a senior engineer) is design your software to avoid ever creating this situation in the first place and throw address sanitizer at things to try and catch them when they arise. The difference with Go is that in Go this will never result in a memory error.
Strictly speaking, the situation in Go is way better. I will take "incorrect behavior, but not a memory error" over "memory error" any day of the week.
We may forget what it's like for new programmers, but for those of us who hang out on Discord channels, Stack Overflow, and Reddit giving people help with programming, simple things like iterator invalidation are a major pain point.
"You have a memory error in your program", I say to someone. "Now that you know that you have a memory error, it is probably your highest priority to find and fix this error." And now you start walking someone through the steps of finding and fixing a memory error, which is nontrivial. You'll tell them about Address Sanitizer, GDB, and Valgrind, and you'll wish them luck.
I agree that it is better than C and C++ but not much better.
C and C++ are "It is very easy to make this mistake which leads to undefined behaviour and a maybe incorrect program."
Go is "It is very easy to make this mistake which leads to a maybe incorrect program."
Yes, better! But the problem is still there. I much prefer the Rust solution where there is no common mistake.
C and C++ are "It is very easy to make this mistake which leads to undefined behaviour and a maybe incorrect program."
Go is "It is very easy to make this mistake which leads to a maybe incorrect program."
Yes, better! But the problem is still there. I much prefer the Rust solution where there is no common mistake.
The catch is that code is hard to translate into Rust.
"I have this code, you see... and it takes a couple mutable references into an array... how do I translate this into Rust?"
There is no one-size-fits-all answer to that question. The code may be correct in C or C++, but the Rust type system may give you one hell of a hard time proving that it is correct to the Rust type system's satisfaction... so you refactor your code completely, or you use integer indexes into arrays rather than references, or you use unsafe code...
I've written some amount of Rust code at this point. About half of the time, when I write a project in Rust, there comes a point at which I'm fighting with the type system. I feel like this should stop happening, at some point.
"I have this code, you see... and it takes a couple mutable references into an array... how do I translate this into Rust?"
There is no one-size-fits-all answer to that question. The code may be correct in C or C++, but the Rust type system may give you one hell of a hard time proving that it is correct to the Rust type system's satisfaction... so you refactor your code completely, or you use integer indexes into arrays rather than references, or you use unsafe code...
I've written some amount of Rust code at this point. About half of the time, when I write a project in Rust, there comes a point at which I'm fighting with the type system. I feel like this should stop happening, at some point.
For sure. The solution has downsides but it does largely solve the problem. It isn't strictly better than go, but it is an important difference.
Mostly off-topic but FWIW "mutable reference into an array" is typically very easy in Rust you just accept a &mut [T]. This also nicely enforces that you don't try to append because that would be wrong 99% of the time.
Mostly off-topic but FWIW "mutable reference into an array" is typically very easy in Rust you just accept a &mut [T]. This also nicely enforces that you don't try to append because that would be wrong 99% of the time.
> Mostly off-topic but FWIW "mutable reference into an array" is typically very easy in Rust you just accept a &mut [T].
I had specifically worded it as "a couple mutable references into an array". How do I take a small number of references into an array, say two or three?
I had specifically worded it as "a couple mutable references into an array". How do I take a small number of references into an array, say two or three?
Off the top of my head you do something like https://play.rust-lang.org/?version=stable&mode=debug&editio...
Yeah you gotta write it out per number of times. Luckily this rarely comes up in this specific form.
Yeah you gotta write it out per number of times. Luckily this rarely comes up in this specific form.
Wow, that code is as bad as I thought it would be. Yeesh.
> Luckily this rarely comes up in this specific form.
Well, yeah—it would come up rarely because it only solves the problem under very specific circumstances!
> Luckily this rarely comes up in this specific form.
Well, yeah—it would come up rarely because it only solves the problem under very specific circumstances!
Yeah I mean you might be able to make it better, this is just what I would reach for first.
It is so rare that I don’t think I’ve ever seen it in any rust codebase I’ve ever looked at.
It is so rare that I don’t think I’ve ever seen it in any rust codebase I’ve ever looked at.
Ah, missed that. You are right, in that case you need to start making some decisions depending on the situation.
It would be more accurate to say that pointers don’t work with append() or any other way of growing an array, since they all depend on reallocating it sometimes.
Incidentally, this is equally true of creating additional pointers or slices pointing into an growable array. They aren’t safe after the next append(). If you grow an array then you need to refer to its elements using array indices.
But if you have a fixed-length array, or between appends, you can use both pointers and slices to point to parts of it, and it will work fine,
This all works the same as C if you think of a slice as a glorified pointer. If you’re thinking of a slice as a JavaScript array then you’ll have trouble.
Incidentally, this is equally true of creating additional pointers or slices pointing into an growable array. They aren’t safe after the next append(). If you grow an array then you need to refer to its elements using array indices.
But if you have a fixed-length array, or between appends, you can use both pointers and slices to point to parts of it, and it will work fine,
This all works the same as C if you think of a slice as a glorified pointer. If you’re thinking of a slice as a JavaScript array then you’ll have trouble.
> If you’re thinking of a slice as a JavaScript array then you’ll have trouble.
The problem of Go is that it has you uses slices as that as well as actual slices, there is no vector type. So the confusion is very much understandable and to be expected.
The problem of Go is that it has you uses slices as that as well as actual slices, there is no vector type. So the confusion is very much understandable and to be expected.
>Honestly, this is a strange and peculiar situation, although Go programmers have acclimatized to it. To programmers from other languages, such as C or C++, the concept of pointers to dynamically extensible arrays seems like a perfectly decent idea that surely should exist and work in Go. Well, it exists, and it "works" in the sense that it yields results and doesn't crash your program, but it doesn't "work" in the sense of doing what you'd actually want.
I think it works in the same way as a pointer to std::span in C++. (Or pointer to std::string_view with the exception that std::string_view doesn't allow modification of the elements.)
I guess the difference is that std::span doesn't let you append to the backing array through the std::span directly. So with C++ you have to write more code which makes it clearer what's happening.
I think it works in the same way as a pointer to std::span in C++. (Or pointer to std::string_view with the exception that std::string_view doesn't allow modification of the elements.)
I guess the difference is that std::span doesn't let you append to the backing array through the std::span directly. So with C++ you have to write more code which makes it clearer what's happening.
C++ has std::vector, which is one level of abstraction above a slice; you push to a vector, and maybe the backing slice changes, but it's still the same vector.
Go is unusual in not having a equivalent of vector.
Go is unusual in not having a equivalent of vector.
Yes, std::vector and slice are different. My point is that a slice is similar to std::span.
With both slice and std::span, = does a shallow copy of just 2 or 3 pointers.
With std::vector = does a deep copy of every element, you have 2 distinct backing arrays.
Go forces you to use copy() and append() and rely on the garbage collector to make a slice fill the roll of std::vector. IMO it leads to some confusing code.
With both slice and std::span, = does a shallow copy of just 2 or 3 pointers.
With std::vector = does a deep copy of every element, you have 2 distinct backing arrays.
Go forces you to use copy() and append() and rely on the garbage collector to make a slice fill the roll of std::vector. IMO it leads to some confusing code.
Not really, if reallocation takes place.
So if you got a pointer to a vector element, it now points to garbage.
So if you got a pointer to a vector element, it now points to garbage.
I guess the difference is this:
C++:
C++:
std::vector<int> v {1, 2, 3};
void foo(std::vector<int> *ref) {
ref.push_back(4);
}
foo(&v);
//v[3] == 4 is true here
v := []int{1, 2, 3}
func foo(ref *[]int) {
append(ref, 4)
}
foo(&v)
//v[3] == 4 may or may not be true here.
append(ref, 4) // error: first argument to append must be slice
append(*ref, 4) // error: append(*ref, 4) evaluated but not used
this is the correct version and it also removes the incertitude:
*ref = append(*ref, 4)
https://play.golang.org/p/-xDqaxvqWhm
append(*ref, 4) // error: append(*ref, 4) evaluated but not used
this is the correct version and it also removes the incertitude:
*ref = append(*ref, 4)
https://play.golang.org/p/-xDqaxvqWhm
Yes, I made a significant error in forgetting what slice variables actually represent. In my example, even fixing the compilation errors (with `_ = append(*ref, 4)` ), v[3] would always be an array index out of bounds error, since v itself always points to just the first 3 elements of the array, regardless of resizing. This is another significant difference between slices and C++ vectors.
A more interesting example showing that resizing can be observed (getting rid of the pointer to the slice, since it's not useful anyway):
https://play.golang.org/p/UJ-t63bKyJJ
A more interesting example showing that resizing can be observed (getting rid of the pointer to the slice, since it's not useful anyway):
https://play.golang.org/p/UJ-t63bKyJJ
Too late to edit, but I made a significant mistake in the Go code: v[3] would always be a runtime error unless we explicitly modify *ref inside the function (ref = append(ref, 4), in which case v[3] == 4 would always be true).
This happens regardless of resizing, since append() at best modifies the array that *ref/v points to, but it does not modify *ref/v itself; and slices in Go have a pointer to an underlying storage AND a start and end index into that storage (multiple slices can point to different parts of the same storage).
Created here an example that shows how this interacts with resizing:
https://play.golang.org/p/UJ-t63bKyJJ
This happens regardless of resizing, since append() at best modifies the array that *ref/v points to, but it does not modify *ref/v itself; and slices in Go have a pointer to an underlying storage AND a start and end index into that storage (multiple slices can point to different parts of the same storage).
Created here an example that shows how this interacts with resizing:
https://play.golang.org/p/UJ-t63bKyJJ
> //v[3] == 4 is true here
Note that C++ std::vector has an operator overload so this is actually just calling the method named operator[] on the object v and that method returns you a reference to the object in the backing array if in fact it is a suitable size (no checks are made). In particular if foo doesn't for any reason extend the vector then our program now has Undefined Behaviour.
This is not merely the C syntactic sugar array subscript operation v[3] == *(v+3) as the vector is not necessarily just a backing array pointer and some magic.
In contrast Go is really offering array syntax for this slice, that's the built-in array subscript operation and it cares whether v[3] exists when you try to compare it to 4.
Note that C++ std::vector has an operator overload so this is actually just calling the method named operator[] on the object v and that method returns you a reference to the object in the backing array if in fact it is a suitable size (no checks are made). In particular if foo doesn't for any reason extend the vector then our program now has Undefined Behaviour.
This is not merely the C syntactic sugar array subscript operation v[3] == *(v+3) as the vector is not necessarily just a backing array pointer and some magic.
In contrast Go is really offering array syntax for this slice, that's the built-in array subscript operation and it cares whether v[3] exists when you try to compare it to 4.
I'm not sure I understand how this is relevant. In C++, arr[non_existent_index] is UB both if arr is an std::vector and if it is a C-style array. In Go it's a runtime error instead.
Sure, std::vector::operator[]() is not just syntax sugar for *(v+i), but I don't think any of this is relevant for the discussion at hand, unless I'm missing something.
Sure, std::vector::operator[]() is not just syntax sugar for *(v+i), but I don't think any of this is relevant for the discussion at hand, unless I'm missing something.
> To programmers from other languages, such as C or C++, the concept of pointers to dynamically extensible arrays seems like a perfectly decent idea that surely should exist and work in Go.
Ah, I would beg to differ!
You should never be taking pointers to a dynamically resizable array, in any language. (Well, caveat, its fine if you do it only for a time period where you know the array won't be growing.) The whole point of a dynamically resizable array is that its addresses can change!
If you did this in C++, you'd get undefined behavior. In Go you get "safe" but probably-not-what-you-wanted behavior. In Rust it simply wouldn't be possible (w/o unsafe), and you'd have to use indices (which is the correct thing to do, in any language).
Ah, I would beg to differ!
You should never be taking pointers to a dynamically resizable array, in any language. (Well, caveat, its fine if you do it only for a time period where you know the array won't be growing.) The whole point of a dynamically resizable array is that its addresses can change!
If you did this in C++, you'd get undefined behavior. In Go you get "safe" but probably-not-what-you-wanted behavior. In Rust it simply wouldn't be possible (w/o unsafe), and you'd have to use indices (which is the correct thing to do, in any language).
I love Go, my favourite language. But I've been bitten before by passing slices around and then finding out that they got disassociated and are now pointing at two different backing arrays without telling me.
I kinda know enough now to avoid this, but I have to be careful and remind myself it's a possibility.
I'd love some built-in method to be able to tell whether altering a value in slice A will also alter the value in slice B (i.e. whether A and B are referring to the same backing array). As far as I'm aware there's no easy way of doing this in Go.
I kinda know enough now to avoid this, but I have to be careful and remind myself it's a possibility.
I'd love some built-in method to be able to tell whether altering a value in slice A will also alter the value in slice B (i.e. whether A and B are referring to the same backing array). As far as I'm aware there's no easy way of doing this in Go.
[deleted]
Outside of a few very specific situations, if you’re working with a pointer to a slice or string you’re doing something very wrong. Slices are “fat” pointers.
> To programmers from other languages, such as C or C++, the concept of pointers to dynamically extensible arrays seems like a perfectly decent idea
Write Go in Go, don’t write C in Go. (Which applies to every language, tbh.)
> To programmers from other languages, such as C or C++, the concept of pointers to dynamically extensible arrays seems like a perfectly decent idea
Write Go in Go, don’t write C in Go. (Which applies to every language, tbh.)
[deleted]
Problems with memory addressing are bound to happen with asynchronous memory manipulation: threads in C/C++ or concurrent GC in go.
The biggest issue here is that memory manipulation happens behind the scenes and the runtime does not offer effective tools to synchronise memory state manipulations.
That’s not really the issue here. The issue is that slices are mutation proxies to the backing array, so in a concurrent context you’re sharing mutable state with all that implies. And worse because of the behaviour of append that sharing is not systematic, so it can look like there’s no mutable state sharing… until you append on a slice with leftover capacity and now there is.
The GC only operates on “dead” allocations (afaik it remains non-moving) so it’s not a concern for now.
The GC only operates on “dead” allocations (afaik it remains non-moving) so it’s not a concern for now.
I get the confusion (to people unfamiliar with pointers) about pointers to elements in reference types, but why would anyone want pointers to reference types? They're basically pointers with extra features
Go doesn’t have reference types, and slices certainly do not behave like them: because the slice is composed of the pointer, length, and capacity (on the stack) if you want to modify a slice on behalf of a caller you absolutely must get a pointer to a slice, unless your mutations consist solely of setting existing indices.
Go’s map is (AFAIK) a pointer to the hmap stucture where everything happens, so it does behave like a reference type (all updates in the callee will be visible to the caller) but even then it can be useful to have a pointer to one so you can reset it without mutating it in-place. While that can be a bit weirder, it is also much safer. Especially given Go’s maps are not thread-safe (and in fact not memory-safe under concurrent updates).
Go’s map is (AFAIK) a pointer to the hmap stucture where everything happens, so it does behave like a reference type (all updates in the callee will be visible to the caller) but even then it can be useful to have a pointer to one so you can reset it without mutating it in-place. While that can be a bit weirder, it is also much safer. Especially given Go’s maps are not thread-safe (and in fact not memory-safe under concurrent updates).
Isn’t passing in pointers to slices considered an anti-pattern? I thought the go way would be to take the slice as argument by value and return possibly a different slice, like append does. I don’t think I’ve seen any (idiomatic) code that took slices by pointer.
There are no such things as "reference types" in Go, though slices do have the extremely odd behavior that they can take the value `nil`, similarly to interfaces and unlike any other non-pointer type in Go.
Pointers to "fat pointer" types are sometimes needed, just like you sometimes need pointers-to-pointers.
Pointers to "fat pointer" types are sometimes needed, just like you sometimes need pointers-to-pointers.
What about maps? Non-nil maps sure seem like they're "reference types", look at [0].
[0] https://play.golang.org/p/xtY_ASExQzR
[0] https://play.golang.org/p/xtY_ASExQzR
"reference types" is a very specific concept from a specific category of languages: types which are always heap-allocated and sitting behind an invisible (and un-interactible) pointer.
But Go doesn't have that distinction, and has actual pointers you can use directly. A map is just a heap-allocated structure sitting behind a pointer.
If you create a type which is a pointer to a struct, sure you can say you've built a reference type if you want, but that doesn't actually say much to anyone, because that's not a distinction the language makes, unlike Java or C#.
But Go doesn't have that distinction, and has actual pointers you can use directly. A map is just a heap-allocated structure sitting behind a pointer.
If you create a type which is a pointer to a struct, sure you can say you've built a reference type if you want, but that doesn't actually say much to anyone, because that's not a distinction the language makes, unlike Java or C#.
> types which are always heap-allocated and sitting behind an invisible (and un-interactible) pointer
> A map is just a heap-allocated structure sitting behind a pointer.
And the difference is?.. Because maps in Go behave exactly as if they were un-referenceable pointers to the hidden, heap-allocated hashtables.
> A map is just a heap-allocated structure sitting behind a pointer.
And the difference is?.. Because maps in Go behave exactly as if they were un-referenceable pointers to the hidden, heap-allocated hashtables.
Considering the confusion of the author, it seems like not all junior programmers can understand Go, which makes me wonder: is it simple enough?
One pitfall is when getting a slice by value in a function. You cannot be sure that someone is not going to pass you a slice into a buffer that they themselves use, so you have to be careful when appending - someone might be using that buffer and you’ll be writing over it.
One pitfall is when getting a slice by value in a function. You cannot be sure that someone is not going to pass you a slice into a buffer that they themselves use, so you have to be careful when appending - someone might be using that buffer and you’ll be writing over it.
I don't feel its a matter of complexity per se, more like bad UX. We could imagine exposing why this doesn't work to end users instead oh hiding behind a leaky abstraction.
> which makes me wonder: is it simple enough?
Obviously not given Go was never simple in the first place. Go was built to be easy — for a certain value of easy.
Simple tools are often not easy, and simple programming languages are definitely not easy: they tend to be built out of a small set of very powerful concepts which are directly exposed to the language user, said language user has close to the power of the language designer in building abstractions. Lisps, and Smalltalks and Forths are simple, which means they are mind-bending and not only can you build what you want out of them (hello turing equivalence) you can build how you want.
And of course the simplest of languages (the turing tarpits) are barely usable at all.
Obviously not given Go was never simple in the first place. Go was built to be easy — for a certain value of easy.
Simple tools are often not easy, and simple programming languages are definitely not easy: they tend to be built out of a small set of very powerful concepts which are directly exposed to the language user, said language user has close to the power of the language designer in building abstractions. Lisps, and Smalltalks and Forths are simple, which means they are mind-bending and not only can you build what you want out of them (hello turing equivalence) you can build how you want.
And of course the simplest of languages (the turing tarpits) are barely usable at all.
Continuing that thought along the line of "Simple made easy," I think ease has a simplicity all its own.
A simple language, e.g. C or lisp, simple in that their grammar is simple, are definitely less easy for the programmer, than say Go. But C is not simple as an experience, since it forces the dev to mentally complect so many concepts in order to get things done: macros, memory management, etc. Lisp is complex in a different way: metaprogamming, DSLs, and deep abstractions are the norm. So simplicity/complexity tends to be something of a whack-a-mole. It's a lower bound, much like the uncertainty principle; you can always add complexity.
Go makes a lot of choices that try to really optimize the user_complexity * language_complexity product.
A simple language, e.g. C or lisp, simple in that their grammar is simple, are definitely less easy for the programmer, than say Go. But C is not simple as an experience, since it forces the dev to mentally complect so many concepts in order to get things done: macros, memory management, etc. Lisp is complex in a different way: metaprogamming, DSLs, and deep abstractions are the norm. So simplicity/complexity tends to be something of a whack-a-mole. It's a lower bound, much like the uncertainty principle; you can always add complexity.
Go makes a lot of choices that try to really optimize the user_complexity * language_complexity product.
There are conventions that help, but they could be made explicit:
- If you don't know where a slice came from (such as getting it passed as an argument), treat it as read-only. Any exceptions should be documented.
- If you're going to mutate a slice, make it a private member of a struct and don't give out direct access to it.
- If you don't know where a slice came from (such as getting it passed as an argument), treat it as read-only. Any exceptions should be documented.
- If you're going to mutate a slice, make it a private member of a struct and don't give out direct access to it.
No it's not, it can be even simpler, but it's a great start and evolution
s = append(s, elem)
I read this immediately as "create a new copy of the original slice with one additional element", so I presumed that was the case. It would actually be shocking the opposite, if I could end up modifying the original one (before the append) with a pointer to the new s, which seems to be the case!
Big gotcha there: treat slices as stateful at all time.
Since it has an assignment operation, it must be creating something new, otherwise it would have been a method of the slice itself
EDIT: I just realized the gotcha is not there at all, Go would consider the first slice to be of N length and the second slice of length N+1. Comparing the two slices would give an error at some point because one is shorter than the other, so the fact that the address changes or not is irrelevant. However I can see this becoming problematic with pointers, which proves the point of the article.
I read this immediately as "create a new copy of the original slice with one additional element", so I presumed that was the case. It would actually be shocking the opposite, if I could end up modifying the original one (before the append) with a pointer to the new s, which seems to be the case!
Big gotcha there: treat slices as stateful at all time.
Since it has an assignment operation, it must be creating something new, otherwise it would have been a method of the slice itself
EDIT: I just realized the gotcha is not there at all, Go would consider the first slice to be of N length and the second slice of length N+1. Comparing the two slices would give an error at some point because one is shorter than the other, so the fact that the address changes or not is irrelevant. However I can see this becoming problematic with pointers, which proves the point of the article.
Go-s behavior is the ONLY sensible one in _any_ language that supports pointers. This is a faster and safe(er) way. You simply cannot modify (move or reallocate) a data-structure that has pointers pointing to it without invalidating all pointers. Not in C++, not in any language with pointers (that I know if). This is not "strange and peculiar". What's "strange and peculiar" is that the author thinks that doing this in C++ is a "perfectly decent idea". In fact it's a huge no no and more often than not will crash.
Edit: we would both learn something if you offered a counter example instead of downvoted.
Edit: we would both learn something if you offered a counter example instead of downvoted.
It's clearly not the "only sensible one" given that no other language works the way Go does here.
In C++ this is UB, which is bad, but in keeping with the rest of the language.
In Rust, the compiler will not allow you to do any operation that would re-allocate the backing store whilst there are outstanding references into it.
In most other languages (eg. Java, C#, python, etc.), you can't get a pointer/reference to an array index, only a pointer/reference to the item at that index at the time you looked.
Go's decision here is especially weird given that this same thing is seemingly prevented for maps (why the inconsistency?).
Given the three goals of memory-safety, "simplicity" and performance, it's true there are not many other options Go could have chosen, but personally I think Go's interpretation of "simplicity" is incredibly warped: it's a kind of superficial simplicity that leads to programs that are much more complicated to reason about.
In C++ this is UB, which is bad, but in keeping with the rest of the language.
In Rust, the compiler will not allow you to do any operation that would re-allocate the backing store whilst there are outstanding references into it.
In most other languages (eg. Java, C#, python, etc.), you can't get a pointer/reference to an array index, only a pointer/reference to the item at that index at the time you looked.
Go's decision here is especially weird given that this same thing is seemingly prevented for maps (why the inconsistency?).
Given the three goals of memory-safety, "simplicity" and performance, it's true there are not many other options Go could have chosen, but personally I think Go's interpretation of "simplicity" is incredibly warped: it's a kind of superficial simplicity that leads to programs that are much more complicated to reason about.
In Go, you get a stable version of the old data, and the garbage collector tracks that you still have a reference to it. This is safe, but confuses some people.
In Rust, the borrow checker won't let you modify the array while you have a reference to a slice of it. So you can't do this at all.
In C++, you get a mention on the Department of Homeland Security's US-CERT site.