Go automatically downloads a newer toolchain if needed(kokada.capivaras.dev)
kokada.capivaras.dev
Go automatically downloads a newer toolchain if needed
https://kokada.capivaras.dev/blog/quick-bits-go-automatically-downloads-a-newer-toolchain-if-needed/
76 comments
Link to the proposal behind this change:
https://go.googlesource.com/proposal/+/master/design/57001-g...
> Many people believe the go line in the go.mod file specifies which Go toolchain to use. This proposal would correct this widely held misunderstanding by making it reality.
That doesn't sound like a good reason to automatically download binaries and run them.
Is it difficult to update or install a new version of Go and are there frequent updates in Go spec introducing new features that it is necessary to auto install the compiler itself ?
Supply chain attacks are on rise and not a new concept and yet we see these changes.
This is not the first time Go lang has introduced a questionable opt-out feature [1]. They backed out but looks like there were no takeaways from that episode.
1. https://news.ycombinator.com/item?id=34771472
https://go.googlesource.com/proposal/+/master/design/57001-g...
> Many people believe the go line in the go.mod file specifies which Go toolchain to use. This proposal would correct this widely held misunderstanding by making it reality.
That doesn't sound like a good reason to automatically download binaries and run them.
Is it difficult to update or install a new version of Go and are there frequent updates in Go spec introducing new features that it is necessary to auto install the compiler itself ?
Supply chain attacks are on rise and not a new concept and yet we see these changes.
This is not the first time Go lang has introduced a questionable opt-out feature [1]. They backed out but looks like there were no takeaways from that episode.
1. https://news.ycombinator.com/item?id=34771472
The binaries are fully reproducible from source[1] and are published in a transparency log to provide assurance that your go command is downloading the same binary as every other go command. This is state-of-the-art in preventing supply chain attacks and better than every other language.
[1] https://go.dev/rebuild
[1] https://go.dev/rebuild
> Is it difficult to update or install a new version of Go
Go is the easiest language I know to build from source, let alone run a binary distribution.
Go is the easiest language I know to build from source, let alone run a binary distribution.
As usual, Go team shows how much they have learned from the history of computing.
> are there frequent updates in Go spec introducing new features that it is necessary to auto install the compiler itself ?
From what I've seen, the need for frequent updates is more about keeping security scans happy.
From what I've seen, the need for frequent updates is more about keeping security scans happy.
Are you really surprised that people who explicitly refused to learn from 50 years of computer science and software engineering history are refusing to learn from their own mistakes, or even acknowledge them as such?
I'll be raising an issue with my distribution of choice to disable this behavior by default. They're the last line of defense for users' security and privacy, and we as an industry have been trying hard to circumvent them. Maybe precisely because of that reason, who knows.
I'll be raising an issue with my distribution of choice to disable this behavior by default. They're the last line of defense for users' security and privacy, and we as an industry have been trying hard to circumvent them. Maybe precisely because of that reason, who knows.
I'm not sure I like this feature. I prefer to be in control of the software that runs on my system, without any automated downloads. But I'm a nixos user so...
Author here, also a big user of NixOS.
But strangely I liked this feature, mostly because it allows me to use newer features without needing to wait nixpkgs to update the Go compiler to the version 1.23.
It is easy to disable if you don't like though, something like `environment.sessionVariables.GOTOOLCHAIN = "local"` or `home.sessionVariables.GOTOOLCHAIN = "local"` should do the trick.
But strangely I liked this feature, mostly because it allows me to use newer features without needing to wait nixpkgs to update the Go compiler to the version 1.23.
It is easy to disable if you don't like though, something like `environment.sessionVariables.GOTOOLCHAIN = "local"` or `home.sessionVariables.GOTOOLCHAIN = "local"` should do the trick.
Setting the GOTOOLCHAIN environment variable to local disables this. I don't use nix so I'm not sure if they set it by default but you can with set it with `go env`
> Setting the GOTOOLCHAIN environment variable to local disables this
I should be setting the GOTOCHAIN environment variable to enable this, opt-in not opt-out.
I should be setting the GOTOCHAIN environment variable to enable this, opt-in not opt-out.
Why? The majority of people are happy with this change.
Why not opt-in?
It forces what the foundation wants you to do. It takes away control from you, it's unethical.
Not only that, it opens a can of worms. A compiler should be a compiler.
Not an self updating application, nor an dependency modulator from GitHub. How can I trust it when it does all these things?
Call me old fashioned at 35.
If you want the latest then go download the latest. Is that now to hard for the user?
Just because the latest is out doesn't mean it's any better than the previous version. What happens in a CrowdStrike scenario? What happens when Go gets retired in 50 years?
I don't want to work with the latest. Should I? TCL 9 is getting there but TCL 8.7 is still perfectly operatable. Should I be using 9 because it exists? My work only has 8.6 on production.
So your toolchain updates and they've removed a thing. You've got to hunt down the previous version, let alone needing to discover why it was working yesterday and not today. Unnecessary overhead.
You use a dependency that's not updated for the future version?
What stops someone from crafting a malicious binary? Malware hijacking the download path?
Auto-updating takes away your integrity. Your making blind trust that everything is what it is.
How can I be sure that the updated compiler is the compiler and not a malicious crafted version? If you can't trust the compiler how can you trust your code?
Yes, I could turn it off, but could I turn it on instead.
I shouldn't need to turn it off, I'll update when I want to update tyvm.
It forces what the foundation wants you to do. It takes away control from you, it's unethical.
Not only that, it opens a can of worms. A compiler should be a compiler.
Not an self updating application, nor an dependency modulator from GitHub. How can I trust it when it does all these things?
Call me old fashioned at 35.
If you want the latest then go download the latest. Is that now to hard for the user?
Just because the latest is out doesn't mean it's any better than the previous version. What happens in a CrowdStrike scenario? What happens when Go gets retired in 50 years?
I don't want to work with the latest. Should I? TCL 9 is getting there but TCL 8.7 is still perfectly operatable. Should I be using 9 because it exists? My work only has 8.6 on production.
So your toolchain updates and they've removed a thing. You've got to hunt down the previous version, let alone needing to discover why it was working yesterday and not today. Unnecessary overhead.
You use a dependency that's not updated for the future version?
What stops someone from crafting a malicious binary? Malware hijacking the download path?
Auto-updating takes away your integrity. Your making blind trust that everything is what it is.
How can I be sure that the updated compiler is the compiler and not a malicious crafted version? If you can't trust the compiler how can you trust your code?
Yes, I could turn it off, but could I turn it on instead.
I shouldn't need to turn it off, I'll update when I want to update tyvm.
> it's unethical.
lmao, I stopped reading after this.
lmao, I stopped reading after this.
Shame, shows the type of person you are. Keep lapping that kool-aid.
lol keep editing your message clown.
I would've preferred that it be opt out or at least for it to be mentioned prominently enough at some point that it isn't a surprise.
https://tip.golang.org/doc/toolchain
But I guess I was hyper-sensible, because I've missed 'rustup'
But I guess I was hyper-sensible, because I've missed 'rustup'
[deleted]
The way to look at it: The Go toolchain is just another dependency to your project, declared in go.mod, just like the other modules, and handled accordingly (download, verification etc.)
Best worst thing about nixos is that none of these terrifying tricks work because the binaries can never find libc.
Author here and NixOS user.
This works even in NixOS, because `go` toolchain is statically compiled.
This works even in NixOS, because `go` toolchain is statically compiled.
I thought go binaries were purely static, aren't they ?
If you import net or os then you'll automatically link to libc unless you disable cgo or set the "netgo" and "osusergo" build tag respectively.
Author here.
You are confusing the `go` toolchain and the binaries generated with Go toolchain.
The `go` toolchain itself is compiled statically, this is why this works without issues even in NixOS. Binaries built with the toolchain may or may not be linked against libc depending if you enable or disable CGO. But this is not related to the toolchain, that will work regardless since the toolchain itself is statically compiled.
Edit: or in other words, the `go` toolchain is built with CGO disabled.
You are confusing the `go` toolchain and the binaries generated with Go toolchain.
The `go` toolchain itself is compiled statically, this is why this works without issues even in NixOS. Binaries built with the toolchain may or may not be linked against libc depending if you enable or disable CGO. But this is not related to the toolchain, that will work regardless since the toolchain itself is statically compiled.
Edit: or in other words, the `go` toolchain is built with CGO disabled.
... Or do some other flags to statically link to cgo.
If I understand correctly, Rust acts similarly: if you have a `rust-toolchain.toml` file in your repo, any usages of `cargo` will automatically install and use that version of Rust. https://rust-lang.github.io/rustup/overrides.html
It seems like that will change in the (near) future according to the following github issue[0]. A quote from one of the developers, rami3l, in that thread[1]:
> My current plan is indeed to remove implicit installations entirely.
[0]: https://github.com/rust-lang/rustup/issues/3635
[1]: https://github.com/rust-lang/rustup/issues/3635#issuecomment...
> My current plan is indeed to remove implicit installations entirely.
[0]: https://github.com/rust-lang/rustup/issues/3635
[1]: https://github.com/rust-lang/rustup/issues/3635#issuecomment...
Oh sad! Dang I actually really liked the feature, it's super convenient for keeping developer environments in sync. I left a comment in that thread asking for clarification.
>> Oh sad! Dang I actually really liked the feature, it's super convenient for keeping developer environments in sync. I left a comment in that thread asking for clarification.
Here is a slightly contrived, but realistic example of why it is a bad idea:
Perhaps they could make it a configuration setting that developers could opt-in? That would let developers who want automatic toolchain installs to have it and others who do not want it (or whose employers will not allow it) to not have it.
Here is a slightly contrived, but realistic example of why it is a bad idea:
1) Attacker discovers vulnerability in an older version of the Rust toolchain
2) Attacker creates useful crate and helps it to get widely adopted or becomes trusted contributor to a crate that is already popular
3) Attacker creates and publishes crate changes with exploit code and rust-toolchain.toml to trigger use of older, vulnerable Rust toolchain
4) Unsuspecting developers build the trapped crate or something that depends on it and get owned
Perhaps they could make it a configuration setting that developers could opt-in? That would let developers who want automatic toolchain installs to have it and others who do not want it (or whose employers will not allow it) to not have it.
Author here.
In Go case though the version can only go higher, not lower (e.g.: it will not download a toolchain if the Go version is set to lower than your current one, only higher). So I can't see the same attack being executed here.
In Go case though the version can only go higher, not lower (e.g.: it will not download a toolchain if the Go version is set to lower than your current one, only higher). So I can't see the same attack being executed here.
>> In Go case though the version can only go higher, not lower
That is good to know, assuming that the newer hypothetical toolchain is not vulnerable (e.g. a zero-day in the newer toolchain).
My opinion is still that toolchains (newer or older) should not be implicitly installed without the developer's explicit permission. This could be a configuration setting that the developer has opted-in or a "This package requires toolchain version X. Install it? (y/n)" prompt.
That is good to know, assuming that the newer hypothetical toolchain is not vulnerable (e.g. a zero-day in the newer toolchain).
My opinion is still that toolchains (newer or older) should not be implicitly installed without the developer's explicit permission. This could be a configuration setting that the developer has opted-in or a "This package requires toolchain version X. Install it? (y/n)" prompt.
> That is good to know, assuming that the newer hypothetical toolchain is not vulnerable (e.g. a zero-day in the newer toolchain).
This would still be really difficult to explore:
Again, not saying that this is an impossible scenario, just find it highly unlikely.
> This could be a configuration setting that the developer has opted-in or a "This package requires toolchain version X. Install it? (y/n)" prompt.
I concur at this part though, however keep in mind that this `toolchain` feature is more akin to adding a module dependency, since it reuses the whole module system.
I think there is already lots of trust that is implicit when you are managing modules, and this new feature doesn't make it worse. If anything, considering that the binaries can only come from golang.org/toolchain and I assume Go already check the checksum of every module that it downloads (and also that Go is perfectly reproducible: https://go.dev/blog/rebuild), if anything this is probably more trusted than a random module that you add to your project.
This would still be really difficult to explore:
0. An undiscovered zero-day in the newest version of Go
1. The only way to force users to use the newest toolchain is if your project has dependencies where the attacker has control, so e.g.: they can change their go.mod and set a `go` directive that upgrades to the later version
2. You need to update to this new version, because this will make Go complain that your `go` directive inside your `go.mod` is out-of-date
3. After all this, yes, Go will download the newest version of Go (that is vulnerable)
Again, not saying that this is an impossible scenario, just find it highly unlikely.
> This could be a configuration setting that the developer has opted-in or a "This package requires toolchain version X. Install it? (y/n)" prompt.
I concur at this part though, however keep in mind that this `toolchain` feature is more akin to adding a module dependency, since it reuses the whole module system.
I think there is already lots of trust that is implicit when you are managing modules, and this new feature doesn't make it worse. If anything, considering that the binaries can only come from golang.org/toolchain and I assume Go already check the checksum of every module that it downloads (and also that Go is perfectly reproducible: https://go.dev/blog/rebuild), if anything this is probably more trusted than a random module that you add to your project.
Thanks, I understand this; but what I don't understand is, wouldn't it be easier for the same attacker to do the same thing by exploiting a vulnerability in a different crate, and include that other crate as a dependency?
As for configuration: to me, having it be opt-in negates the entire benefit. My point is that automatically installing the correct toolchain makes it far easier to collaborate with others who aren't nearly as obsessive about Rust as I am.
As for configuration: to me, having it be opt-in negates the entire benefit. My point is that automatically installing the correct toolchain makes it far easier to collaborate with others who aren't nearly as obsessive about Rust as I am.
>> wouldn't it be easier for the same attacker to do the same thing by exploiting a vulnerability in a different crate, and include that other crate as a dependency?
Possibly, which is why the example is a bit contrived. In most cases, the toolchains will likely be more trusted and be on approved lists whereas binaries created by third-party crates are not.
For more secure environments, explicitness is valued and automatic installation of anything is frowned upon because it can introduce unvetted changes which could include vulnerabilities.
It depends on what work is being done and how much toolchains and ecosystem can be trusted.
Possibly, which is why the example is a bit contrived. In most cases, the toolchains will likely be more trusted and be on approved lists whereas binaries created by third-party crates are not.
For more secure environments, explicitness is valued and automatic installation of anything is frowned upon because it can introduce unvetted changes which could include vulnerabilities.
It depends on what work is being done and how much toolchains and ecosystem can be trusted.
That seems like a lot of hoops to jump through considering that rust allows arbitrary code execution during compile time anyway.
>> That seems like a lot of hoops to jump through considering that rust allows arbitrary code execution during compile time anyway.
If you mean build.rs build scripts, yes, those do run, but it is not arbitrary code. You can view and inspect them before building. If you need more security, you can download all the dependencies and build inside an isolated container.
If you mean build.rs build scripts, yes, those do run, but it is not arbitrary code. You can view and inspect them before building. If you need more security, you can download all the dependencies and build inside an isolated container.
> but it is not arbitrary code
uhh ya it is. There's also https://github.com/eleijonmarck/do-not-compile-this-code
uhh ya it is. There's also https://github.com/eleijonmarck/do-not-compile-this-code
No. The code in question is plainly visible in the crate:
https://github.com/eleijonmarck/do-not-compile-this-code/blo...
This is true for all third-party libraries. If you blindly download and execute code from the Internet, this is a risk you are assuming.
As I stated above, if you need more security, you can download all the dependencies and build inside an isolated container.
https://github.com/eleijonmarck/do-not-compile-this-code/blo...
This is true for all third-party libraries. If you blindly download and execute code from the Internet, this is a risk you are assuming.
As I stated above, if you need more security, you can download all the dependencies and build inside an isolated container.
icholy(1)
Yeah, I've run into that and don't love it. Build.rs files are probably a bigger concern from a supply chain attack standpoint.
I can see the appeal, but hope they keep it opt-in so that people can choose their preferred workflow without surprises.
Automatic updates are always a good idea... until they aren't ! It's an open door to problems ranging from subtle version semantic difference (try to find why what was working yesterday is not working today without any change) to hacking.
In corporate environment, it might trigger some unexpected alarms (or be blocked)
Gentoo got this right: by default, the "old" behaviour should be kept (no automatic update) and it should only be opt-in
In corporate environment, it might trigger some unexpected alarms (or be blocked)
Gentoo got this right: by default, the "old" behaviour should be kept (no automatic update) and it should only be opt-in
It's not automatic though. You explicitly change your go.mod file and then it downloads the toolchains.
What i'd want is a way to revert back, if the user sees a problem.
The automatic updating toolchain is not a bad idea, and in most cases it works great. Once in a while though, there's a fuck up and breaks your build or something changes that you didnt expect.
The best way forward is to keep the old tool in a backup location, and have the user be able to revert the update with a single command. Then the user is able to safely return to the old one when the inevitable breakage happens. You could even opt-in to analytics for this action, and the owner of the tool would know if a particular update is a screwup (aka, many people reverted it).
The automatic updating toolchain is not a bad idea, and in most cases it works great. Once in a while though, there's a fuck up and breaks your build or something changes that you didnt expect.
The best way forward is to keep the old tool in a backup location, and have the user be able to revert the update with a single command. Then the user is able to safely return to the old one when the inevitable breakage happens. You could even opt-in to analytics for this action, and the owner of the tool would know if a particular update is a screwup (aka, many people reverted it).
To be clear, the new toolchain doesn't replace the old toolchain, but is installed alongside it, and is only used for modules that explicitly request the newer toolchain in their go.mod file.
Author here.
I think you misundertood how this feature works.
If you have a `go.mod` that sets a `go` directive with a version bigger than the version your Go toolchain is currently running, it will download the binary automatically and pass the commands to the new binary. However this doesn't replace your `go` binary in say, PATH, the new toolchain will be downloaded at `GOPATH`, the same as it does with your modules. This is why there is nothing to backup here, nothing is deleted.
Also, keep in mind that if Go is downloading a new toolchain, it means your project will not work with the current toolchain: because if you set `go 1.23` in your `go.mod`, and your current `go` is version 1.22, it would fail regardless until you either fix your `go.mod` or upgrade your `go` toolchain. What Go is doing here is the later, automatically for you.
I think you misundertood how this feature works.
If you have a `go.mod` that sets a `go` directive with a version bigger than the version your Go toolchain is currently running, it will download the binary automatically and pass the commands to the new binary. However this doesn't replace your `go` binary in say, PATH, the new toolchain will be downloaded at `GOPATH`, the same as it does with your modules. This is why there is nothing to backup here, nothing is deleted.
Also, keep in mind that if Go is downloading a new toolchain, it means your project will not work with the current toolchain: because if you set `go 1.23` in your `go.mod`, and your current `go` is version 1.22, it would fail regardless until you either fix your `go.mod` or upgrade your `go` toolchain. What Go is doing here is the later, automatically for you.
[deleted]
If you didn't commit the breaking change to the repo, couldn't you just revert your local changes? Trashing the mod and sum file and running "tidy" should take care of the rest, I would think. Maybe I'm missing something?
> By the way, this only works well because Go binaries are static, one of the things that make the language reasonable good.
Only by default, as the Go toolchain supports dynamic linking for ages, and without third party dependencies called via cgo, or stuff like DNS resolution on Linux.
Only by default, as the Go toolchain supports dynamic linking for ages, and without third party dependencies called via cgo, or stuff like DNS resolution on Linux.
Author here.
What I meant is that the toolchain itself is linked statically, this is why it can be just downloaded from the internet and it will work without extra setup from the user.
Yes, Go will build the libraries once you call `go build`, and yes, they may link to the system C libraries for things like you said. But the `go` binary itself is static.
What I meant is that the toolchain itself is linked statically, this is why it can be just downloaded from the internet and it will work without extra setup from the user.
Yes, Go will build the libraries once you call `go build`, and yes, they may link to the system C libraries for things like you said. But the `go` binary itself is static.
I'm broadly a Go fan, but this strikes me as a terrible decision. Critical tools should not be working through magic like this. Just fail with a clear error and let the user fix it as they see fit.
If they want magic like this, it should be opt-in, not opt-out.
If they want magic like this, it should be opt-in, not opt-out.
I like this feature, I used this to be able to use iterators before Go was updated in Homebrew.
And yes you need to make sure GOTOOLCHAIN=auto, my colleague had to explicitly set it because it defaults to local on Gentoo.
And yes you need to make sure GOTOOLCHAIN=auto, my colleague had to explicitly set it because it defaults to local on Gentoo.
GoTV does similar things but plays it explicitly: https://go101.org/apps-and-libs/gotv.html
GoTV builds needed toolchain versions with the Go source code instead of downloading the pre-built toolchain packages.
GoTV builds needed toolchain versions with the Go source code instead of downloading the pre-built toolchain packages.
It should be optional to activate this feature.
This sounds amazingly convenient! Language creators, please take note.
Imagine if Python 2 automatically upgraded itself to Python 3.
Author here.
This doesn't make sense. The toolchain will be updated only IF you say your project only runs in Go >= XX. This is what `go` directive in `go.mod` says: my projects doesn't work in versions of Go older than this. This is why it happened to me, I declared I need `go 1.23` because of range-over-func, and the toolchain respected my wishes.
This is not an "enforcing you to use the latest Go version, even if you don't wish for it".
This doesn't make sense. The toolchain will be updated only IF you say your project only runs in Go >= XX. This is what `go` directive in `go.mod` says: my projects doesn't work in versions of Go older than this. This is why it happened to me, I declared I need `go 1.23` because of range-over-func, and the toolchain respected my wishes.
This is not an "enforcing you to use the latest Go version, even if you don't wish for it".
Python 3 is not backwards compatible with Python 2 so why would it do that?
Maybe you meant "imagine if Python 3.6 automatically updated itself to Python 3.12"... Which would be amazing! Except that Python 3.12 is also not backwards compatible with Python 3.6. There's no saving Python.
Maybe you meant "imagine if Python 3.6 automatically updated itself to Python 3.12"... Which would be amazing! Except that Python 3.12 is also not backwards compatible with Python 3.6. There's no saving Python.
Same is also starting to apply to Go runtime, standard library and even some minor fixes like the for loop variables lifetime.
All possible places for existing code to break, either via compiler errors, or changes in the expected runtime semantics.
All possible places for existing code to break, either via compiler errors, or changes in the expected runtime semantics.
Even in that case they did the behaviour change properly:
> To ensure backwards compatibility with existing code, the new semantics will only apply in packages contained in modules that declare go 1.22 or later in their go.mod files.
CMake and Rust and Android and lots of other systems have mechanisms like this that allow you to introduce opt-in breaking changes, but unsurprisingly Python doesn't. They just decided it was ok to start breaking backwards compatibility.
> To ensure backwards compatibility with existing code, the new semantics will only apply in packages contained in modules that declare go 1.22 or later in their go.mod files.
CMake and Rust and Android and lots of other systems have mechanisms like this that allow you to introduce opt-in breaking changes, but unsurprisingly Python doesn't. They just decided it was ok to start breaking backwards compatibility.
> Even in that case they did the behaviour change properly:
Not in fact. Go core team has admitted that Go toolchain 1.22.n versions break backward compatibility: https://github.com/golang/go/issues/66092
There is another breakage case: https://github.com/golang/go/issues/66070 and the team leader doesn't plan to fix it.
Not in fact. Go core team has admitted that Go toolchain 1.22.n versions break backward compatibility: https://github.com/golang/go/issues/66092
There is another breakage case: https://github.com/golang/go/issues/66070 and the team leader doesn't plan to fix it.
That doesn't apply to runtime and standard library changes.
Enough of them happened already, that a clarification post was needed,
https://go.dev/blog/compat
Enough of them happened already, that a clarification post was needed,
https://go.dev/blog/compat
That's still among the strongest backwards compatibility guarantees in any major language. Maybe Java would be close? It's certainly better than C++ which is famed for backwards compatibility, and light years ahead of Python.
> That's still among the strongest backwards compatibility guarantees in any major language.
Not at all. See https://go101.org/blog/2024-03-01-for-loop-semantic-changes-... for reasons.
Not at all. See https://go101.org/blog/2024-03-01-for-loop-semantic-changes-... for reasons.
What reasons? You need to pass one flag to the compiler to specify the language version? That's still very very good.
Please tell me what flag I can pass to Python 3.12 so `distutils` still exists.
Please tell me what flag I can pass to Python 3.12 so `distutils` still exists.
Reasons are in the article.
I'm not familiar with Python, so I don't understand what is the problem with Python. But I think the problem of Go is unrelated to Python.
I'm not familiar with Python, so I don't understand what is the problem with Python. But I think the problem of Go is unrelated to Python.
Yet it isn't without zero issues, as claimed.
I don't like Python much, but they will definitely not pull off something like this (again), because they've actually learned from that mistake and broadly agree that it was one.
It’s more like if virtualenv looked in requirements.txt for the Python version and used that version locally. Other languages like Haskell and Scala have been doing that sort of thing for ages.
After explicitly setting the python version to 3 in package.json? Yes please!
Imagine if Python had as strong a compatibility promise as Go
Made a little harder to track down initially as we have a replace directive pointing to a local proto module in each service. It was here that toolchain was set which forced it to keep appearing in the top level module.
Should’ve been opt in behaviour by default not enforced.
We’ve decided to sit on n-1 of Go and I’ve already had an example last month of a well used OSS library setting toolchain to 1.22 as a patch release (which could be considered a breaking change) which fortunately was quickly reverted.