How Go mitigates supply chain attacks(go.dev)
go.dev
How Go mitigates supply chain attacks
https://go.dev/blog/supply-chain
249 comments
There are some subtleties here, but go.mod files are not lockfiles, including go.mod files don't follow a traditional constraint/lockfile split, which I think is part of the point in that snippet you quoted.
Another way they differ is when installing a top-level tool by default npm does not use the exact version from a library's lockfile to pick the selected version of another library as far I as understand, whereas Go does by default use the exact version required by a library's go.mod file in that scenario.
In other words, go.mod files play a bigger role for libraries than a traditional lockfile does by default for libraries in most other ecosystems.
Here's a good analysis on the contrast between go.mod and the default behavior of more traditional lockfiles (using the npm 'colors' incident as a motivating example):
https://research.swtch.com/npm-colors
That link also includes some comments on 'npm ci' and 'shrinkwrap' that I won't repeat here.
All that said, go.mod files do record precise dependency requirements and provide reproducible builds, so it's possible to draw some analogies between go.mod & lockfiles if you want. I just wouldn't say "go.mod files are lockfiles". ;-)
Another way they differ is when installing a top-level tool by default npm does not use the exact version from a library's lockfile to pick the selected version of another library as far I as understand, whereas Go does by default use the exact version required by a library's go.mod file in that scenario.
In other words, go.mod files play a bigger role for libraries than a traditional lockfile does by default for libraries in most other ecosystems.
Here's a good analysis on the contrast between go.mod and the default behavior of more traditional lockfiles (using the npm 'colors' incident as a motivating example):
https://research.swtch.com/npm-colors
That link also includes some comments on 'npm ci' and 'shrinkwrap' that I won't repeat here.
All that said, go.mod files do record precise dependency requirements and provide reproducible builds, so it's possible to draw some analogies between go.mod & lockfiles if you want. I just wouldn't say "go.mod files are lockfiles". ;-)
Wouldn't build size increase a lot if transitive dependencies were pinned to direct dependency lockfiles? Like if library A says "use version 1.0.0 of library X" and library B says "use version 1.0.1 of library X", then you'd likely end up bundling duplicate code in your build.
Not saying the tradeoff isn't worth it, but pinning to dependency lockfiles isn't without downsides.
Not saying the tradeoff isn't worth it, but pinning to dependency lockfiles isn't without downsides.
FWIW, that's not what Go does. In your scenario, a Go binary ends up with a single copy of library X -- the 1.0.1 version. That's because library A is stating "I require at least v1.0.0 of X", and library B is stating "I require at least v1.0.1 of X". The minimal version that satisfies both of those requirements is v1.0.1, and that's what ends up in the binary.
That behavior is Go's "Minimal Version Selection" or "MVS". There are many longer descriptions out there, but a concise graphical description I saw recently and like is:
https://encore.dev/guide/go.mod
That's the default behavior, but a human can ask for other versions. For example, a consumer of A and B could do 'go get X@latest', or edit their own go.mod file to require X v1.2.3, or do 'go get -u ./...' to update all their direct and indirect dependencies, which would include X in this case, etc.
That behavior is Go's "Minimal Version Selection" or "MVS". There are many longer descriptions out there, but a concise graphical description I saw recently and like is:
https://encore.dev/guide/go.mod
That's the default behavior, but a human can ask for other versions. For example, a consumer of A and B could do 'go get X@latest', or edit their own go.mod file to require X v1.2.3, or do 'go get -u ./...' to update all their direct and indirect dependencies, which would include X in this case, etc.
Continuing that example -- in Go you end up with v1.0.1 of X by default even if v1.0.2 is the latest version of X.
That is a difference with many other package managers that can default to using the latest v1.0.2 of X (even if v1.0.2 was just published) when doing something like installing a command line tool. That default behavior is part of how people installing the 'aws-sdk' tool on a Saturday started immediately experiencing bad behavior due to the deliberate 'colors' npm package sabotage that happened that same Saturday.
In any event, it's certainly reasonable to debate pros and cons of different approaches. I'm mainly trying to clarify the actual behavior & differences.
That is a difference with many other package managers that can default to using the latest v1.0.2 of X (even if v1.0.2 was just published) when doing something like installing a command line tool. That default behavior is part of how people installing the 'aws-sdk' tool on a Saturday started immediately experiencing bad behavior due to the deliberate 'colors' npm package sabotage that happened that same Saturday.
In any event, it's certainly reasonable to debate pros and cons of different approaches. I'm mainly trying to clarify the actual behavior & differences.
What if the requirement was pinned specifically to 1.0.0 in order to avoid a bug introduced in 1.0.1. With a package that also requires a minimum 1.0.1, that should be unresolvable set of requirements and your package manager should fail to make a lockfile out of it.
how are you not describing package.json right now what is the difference
The difference is the npm ecosystem actively encourages automatically following SemVer because by default it uses a ^ to prefix the version number. https://heynode.com/tutorial/how-use-semantic-versioning-npm...
The practice of dependencies’ dependencies being specified using SemVer version constraints to auto-accept minor or patch changes is the difference compared to Go, and why lockfiles will not always save you in the npm ecosystem. That said, approaches like Yarn zero-install can make very explicit the versions installed because they are distributed with the source. Similarly, the default of using npm install is bad because it will update lockfiles, you have to use npm ci or npm install —ci both of which are less well-known.
So it’s not impossible to fix, just a bad choice of defaults for an ecosystem of packages that has security implications about the same as updating your Go (or JS) dependencies automatically and not checking the changes first as part of code review. Blindly following SemVer to update dependencies is bad, from a security perspective, regardless of why or how you’re doing it.
The practice of dependencies’ dependencies being specified using SemVer version constraints to auto-accept minor or patch changes is the difference compared to Go, and why lockfiles will not always save you in the npm ecosystem. That said, approaches like Yarn zero-install can make very explicit the versions installed because they are distributed with the source. Similarly, the default of using npm install is bad because it will update lockfiles, you have to use npm ci or npm install —ci both of which are less well-known.
So it’s not impossible to fix, just a bad choice of defaults for an ecosystem of packages that has security implications about the same as updating your Go (or JS) dependencies automatically and not checking the changes first as part of code review. Blindly following SemVer to update dependencies is bad, from a security perspective, regardless of why or how you’re doing it.
> The difference is the npm ecosystem actively encourages automatically following SemVer because by default it uses a ^ to prefix the version number.
So does Go. In fact, Go only supports the equivalent of ^, there is no way to specify a dependency as '=1.2.3'. That is, whenever you have two different dependencies which use the same dependency at different (semver compatible) versions, go mod will always download the newer of the two, effectively assuming that the one depending on an older version will also work with the newer.
The only difference in this respect compared to NPM (and perhaps also Cargo or NuGet? I don't know) is that Go will never download a version that is not explicitly specified in some go.mod file - which is indeed a much better policy.
So does Go. In fact, Go only supports the equivalent of ^, there is no way to specify a dependency as '=1.2.3'. That is, whenever you have two different dependencies which use the same dependency at different (semver compatible) versions, go mod will always download the newer of the two, effectively assuming that the one depending on an older version will also work with the newer.
The only difference in this respect compared to NPM (and perhaps also Cargo or NuGet? I don't know) is that Go will never download a version that is not explicitly specified in some go.mod file - which is indeed a much better policy.
It’s very subtle, but there are some important differences. For example, lockfiles are not recursive in NPM: the NPM package (usually?) does not contain the lockfile and does not adhere to it when installed as a dependency. It will pick the newest version of dependencies that matches the spec in package.json.
Go mod files are used recursively, and rather than try to pick the newest possible version, it will go with the oldest version.
This avoids the node-ipc issue entirely, at least until you update the go.mod.
Go mod files are used recursively, and rather than try to pick the newest possible version, it will go with the oldest version.
This avoids the node-ipc issue entirely, at least until you update the go.mod.
This really depends on the specific package manager: if you're building an application in Rust, its lockfile will contain the full tree of dependencies, locked to a specific version.
I might be misunderstanding GP, but I think what they're saying is that when package A depends on package B, building package A will use B's lockfile. Assuming that's the case, I think this is generally not how Rust does things, as by default Cargo.lock is explicitly listed in the .gitignore when making a library crate, although there's nothing stopping anyone from just removing that line. I think I remember reading in documentation somewhere that checking in Cargo.lock for libraries is discouraged (hence the policy), but I don't recall exactly where since it's been so long. (That being said, there's a pretty decent chance you were the one who wrote that documentation, so maybe you might remember!)
> I think this is generally not how Rust does things
That is correct.
> as by default Cargo.lock is explicitly listed in the .gitignore when making a library crate
Even if it is included in the contents of the package, Cargo will not use it for the purpose of resolution.
The "don't check it in" thing is related, but not because it will be used if it's included. It's because of the opposite; that way new people who download your package to hack on it will get their own, possibly different Cargo.lock, so you end up testing more versions naturally. Some people dislike this recommendation and include theirs in the package, but that never affects resolution behavior.
That is correct.
> as by default Cargo.lock is explicitly listed in the .gitignore when making a library crate
Even if it is included in the contents of the package, Cargo will not use it for the purpose of resolution.
The "don't check it in" thing is related, but not because it will be used if it's included. It's because of the opposite; that way new people who download your package to hack on it will get their own, possibly different Cargo.lock, so you end up testing more versions naturally. Some people dislike this recommendation and include theirs in the package, but that never affects resolution behavior.
Ah, good to know! I'm glad I asked
Yes, this is true; I think the article is definitely making the distinction vs NPM and not Rust.
The article seemed to go out of its way not to mention any specific package manager or ecosystem. So I think comparing to Rust is completely reasonable.
It didn’t mention one by name, but Rust hasn’t been subject to any widely publicized supply chain attacks. They do, however, mention left-pad by name. I think it can be implied that they really did just mean npm.
They definitely should have said that then. Simply referring to "lockfiles" paints with a very broad brush that includes a number of package managers that don't have the problems that NPM does.
The default way Go handles go.mod is fairly different than the default way Cargo handles Cargo.lock files, including for example with libraries.
Also, when the blog says:
> Moreover, when a dependency is added with go get, its transitive dependencies are added at the version specified in the dependency’s go.mod file, not at their latest versions, thanks to Minimal version selection.
I believe that is significantly different than default Cargo behavior and for example default 'pub' behavior for Flutter (though I know approximately nothing about Flutter package management beyond a cursory search just now ;-)
To my knowledge, both Cargo and Flutter 'pub' prefer the most recent / highest allowed version by default when asked to solve constraints, whereas Go does not.
Cargo: [1]
> When multiple packages specify a dependency for a common package, the resolver attempts to ensure that they use the same version of that common package, as long as they are within a SemVer compatibility range. It also attempts to use the greatest version currently available within that compatibility range.
Flutter 'pub': [2]
> For each package in the graph, pub looks at everything that depends on it. It gathers together all of their version constraints and tries to simultaneously solve them. (Basically, it intersects their ranges.) Then it looks at the actual versions that have been released for that package and selects the best (most recent) one that meets all of those constraints.
[1]: https://doc.rust-lang.org/cargo/reference/resolver.html
[2]: https://dart.dev/tools/pub/versioning#constraint-solving
Also, when the blog says:
> Moreover, when a dependency is added with go get, its transitive dependencies are added at the version specified in the dependency’s go.mod file, not at their latest versions, thanks to Minimal version selection.
I believe that is significantly different than default Cargo behavior and for example default 'pub' behavior for Flutter (though I know approximately nothing about Flutter package management beyond a cursory search just now ;-)
To my knowledge, both Cargo and Flutter 'pub' prefer the most recent / highest allowed version by default when asked to solve constraints, whereas Go does not.
Cargo: [1]
> When multiple packages specify a dependency for a common package, the resolver attempts to ensure that they use the same version of that common package, as long as they are within a SemVer compatibility range. It also attempts to use the greatest version currently available within that compatibility range.
Flutter 'pub': [2]
> For each package in the graph, pub looks at everything that depends on it. It gathers together all of their version constraints and tries to simultaneously solve them. (Basically, it intersects their ranges.) Then it looks at the actual versions that have been released for that package and selects the best (most recent) one that meets all of those constraints.
[1]: https://doc.rust-lang.org/cargo/reference/resolver.html
[2]: https://dart.dev/tools/pub/versioning#constraint-solving
In that same section, the blog describes the behavior of 'go install foo@latest' and contrasts it to how the default install "in some ecosystems bypass pinning."
That is also a difference in default behavior between Go and Cargo.
To install a 'foo' binary, 'go install foo@latest' gives you the latest version of foo, but the direct and indirect dependencies used are the versions listed in foo's go.mod or a dependency’s go.mod file (and not whatever the latest versions of those direct and indirect dependencies might be at the moment the install is invoked).
'cargo install foo' supports the optional --locked flag, but its not the default behavior: [1]
> By default, the Cargo.lock file that is included with the package will be ignored. This means that Cargo will recompute which versions of dependencies to use, possibly using newer versions that have been released since the package was published. The --locked flag can be used to force Cargo to use the packaged Cargo.lock file if it is available.
There are definitely pros and cons here, but to my knowledge it is not "just NPM" that is being contrasted in the blog.
Finally, I'm no world-class Rust expert, but I like using Cargo. I think Cargo is a fantastic tool that set the bar for package mangers, and it has done great things for the Rust community. But it is easier for communities to learn from each other with a base understanding of where & why different choices have been made, which is part of what is behind some of my comments around Go's behavior. ;-)
[1]: https://doc.rust-lang.org/cargo/commands/cargo-install.html
That is also a difference in default behavior between Go and Cargo.
To install a 'foo' binary, 'go install foo@latest' gives you the latest version of foo, but the direct and indirect dependencies used are the versions listed in foo's go.mod or a dependency’s go.mod file (and not whatever the latest versions of those direct and indirect dependencies might be at the moment the install is invoked).
'cargo install foo' supports the optional --locked flag, but its not the default behavior: [1]
> By default, the Cargo.lock file that is included with the package will be ignored. This means that Cargo will recompute which versions of dependencies to use, possibly using newer versions that have been released since the package was published. The --locked flag can be used to force Cargo to use the packaged Cargo.lock file if it is available.
There are definitely pros and cons here, but to my knowledge it is not "just NPM" that is being contrasted in the blog.
Finally, I'm no world-class Rust expert, but I like using Cargo. I think Cargo is a fantastic tool that set the bar for package mangers, and it has done great things for the Rust community. But it is easier for communities to learn from each other with a base understanding of where & why different choices have been made, which is part of what is behind some of my comments around Go's behavior. ;-)
[1]: https://doc.rust-lang.org/cargo/commands/cargo-install.html
So, I believe that you're confusing two different cases here.
`cargo install` is not how you add a dependency, it's not like `npm install`. `cargo install` downloads the source for and installs a runnable binary program, like `npm install -g`. Cargo does not currently have an "add this dependency to my Cargo.toml" command built-in, but `cargo add` is coming from this purpose.
With `cargo install`, the default behavior is to completely recompute the dependency tree before doing the build. The `--locked` flag modifies that to use the lockfile included in the package to do that build instead (and in fact fail compilation if it does not exist). That lockfile will still be a full graph of all dependencies and transitive dependencies to build the binary, it doesn't like, recurse or use any lockfiles that exist in any of the dependencies' packages.
`cargo install` is not how you add a dependency, it's not like `npm install`. `cargo install` downloads the source for and installs a runnable binary program, like `npm install -g`. Cargo does not currently have an "add this dependency to my Cargo.toml" command built-in, but `cargo add` is coming from this purpose.
With `cargo install`, the default behavior is to completely recompute the dependency tree before doing the build. The `--locked` flag modifies that to use the lockfile included in the package to do that build instead (and in fact fail compilation if it does not exist). That lockfile will still be a full graph of all dependencies and transitive dependencies to build the binary, it doesn't like, recurse or use any lockfiles that exist in any of the dependencies' packages.
Hi Steve, first, thanks for weighing in here!
I might have misunderstood your comment, but in my GP comment I was indeed attempting to contrast 'go install foo@latest' with 'cargo install foo', which both install binaries. (I wasn't talking about 'go get bar@latest', which now is just for updating or adding dependencies to a project).
Also, I'm contrasting what happens by default at the moment either binary install command is run. My understanding is Cargo's (non-default) 'cargo install --locked foo' behavior is similar to the default behavior of 'go install foo@latest'. In other words, the default behavior is fairly different between 'cargo install foo' (without --locked) vs. 'go install foo@latest'.
I edited my GP comment to simplify the example to use 'foo' in both cases. Maybe that helps?
I might have misunderstood your comment, but in my GP comment I was indeed attempting to contrast 'go install foo@latest' with 'cargo install foo', which both install binaries. (I wasn't talking about 'go get bar@latest', which now is just for updating or adding dependencies to a project).
Also, I'm contrasting what happens by default at the moment either binary install command is run. My understanding is Cargo's (non-default) 'cargo install --locked foo' behavior is similar to the default behavior of 'go install foo@latest'. In other words, the default behavior is fairly different between 'cargo install foo' (without --locked) vs. 'go install foo@latest'.
I edited my GP comment to simplify the example to use 'foo' in both cases. Maybe that helps?
Ah yes, I did miss that, thank you / sorry :) Too many sub-threads around here!
I don't know go install's semantics well enough to know if that comparison is true or not, I'm just trying to make sure that Cargo's semantics are clear :)
I don't know go install's semantics well enough to know if that comparison is true or not, I'm just trying to make sure that Cargo's semantics are clear :)
[deleted]
I believe Cargo does still have less strictness over dep versions than Go modules, since it will never use a module newer than the one specified in any go.mod file. Lockfiles are generally not honored recursively, and I don’t think Cargo is different here? Hope I’m not spreading misinformation, though I couldn’t find any docs with a cursory glance.
I don’t want to make assertions that I’m less sure of, but I think NPM and Cargo are actually more similar than different here. They both specify exact versions in lock files, for all nested dependencies, but don’t honor the lock files present inside dependencies, instead calculating the nested deps from the constraints.
I don’t want to make assertions that I’m less sure of, but I think NPM and Cargo are actually more similar than different here. They both specify exact versions in lock files, for all nested dependencies, but don’t honor the lock files present inside dependencies, instead calculating the nested deps from the constraints.
Cargo does not do "recursive lockfiles", that's correct.
This is also true of npm and yarn, as far as I can tell: package-lock.json and yarn.lock contain the exact version of every transitive dependency.
I always forget the exact semantics, but the parent's description of them as "recursive" is not the same as Cargo; Cargo determines the full tree and writes out its own lockfile, if dependencies happen to have a Cargo.lock inside the package, it's ignored, not used.
How does go deal with the diamond dependency issue when transitively you've a dep on the same package via two different paths?
As far as I understand it... the import path that you use to import a package acts as its identity, and only one version of any given package will be installed. The way that it will determine this is by choosing the lowest version specified in any package that depends on a given package. Major versions of packages are required to have different import paths with Go modules, so when depending on two different major versions of the same package, they are treated effectively as their own package.
I think you might have a small typo here:
> by choosing the lowest version specified in any package that depends on a given package
It picks the highest version specified in any of the requirements. (That's the minimal version that simultaneously satisfies each individual requirement, where each individual requirement is saying "I require vX.Y.Z or higher". So if A requires Foo v1.2.3 and B requires Foo v1.2.4, v1.2.4 is selected as the minimal version that satisfies both A and B, and that's true even if v1.2.5 exists).
> by choosing the lowest version specified in any package that depends on a given package
It picks the highest version specified in any of the requirements. (That's the minimal version that simultaneously satisfies each individual requirement, where each individual requirement is saying "I require vX.Y.Z or higher". So if A requires Foo v1.2.3 and B requires Foo v1.2.4, v1.2.4 is selected as the minimal version that satisfies both A and B, and that's true even if v1.2.5 exists).
Yep, I regrettably described the system wrong. It’s the highest version specified, and thus minimum version possible.
That's a nice & concise way to describe it.
Minimalistic Version Selection might have been a better term for it.
That did confuse me a lot. Selecting the maximum makes a lot more sense, since that gets SemVer assumptions correct.
That did confuse me a lot. Selecting the maximum makes a lot more sense, since that gets SemVer assumptions correct.
Okay but if I depend on A and A depends on C and has it pinned at 1.5, but I also depend on B and B has C pinned on 1.8, then I get 1.5? But what happens if C is on 1.8 because it doesn't work with 1.5 because an API it needs doesn't exist in 1.5?
Are we not talking about the transitively pinned dependencies in the "lock" section, or are we talking about logical constraints?
Logical constraints would make more sense, but if the constraints on C across various transitive deps are > 1.5 and > 1.8, and 1.9 or 1.10 exists, I probably want the last version of those.
Are we not talking about the transitively pinned dependencies in the "lock" section, or are we talking about logical constraints?
Logical constraints would make more sense, but if the constraints on C across various transitive deps are > 1.5 and > 1.8, and 1.9 or 1.10 exists, I probably want the last version of those.
I'm fairly sure Go Modules does not support what you’re describing. It specifically avoided having a SAT solver (or something similar), unlike most package managers. You specify a minimum version, and that’s it. 1.8 would be selected because it is the highest minimum version out of the options 1.5 and 1.8 that the dependencies require. Unless you edit your go.mod file to require an even higher version, which is an option. Alternatively, you can always replace that transitive dependency with your own fork that fixes the problems, using a “replace” directive in your go.mod file.
If your dependencies are as broken as you’re describing, you’re in for a world of hurt no matter the solution. I also can't remember ever encountering that situation.
If your dependencies are as broken as you’re describing, you’re in for a world of hurt no matter the solution. I also can't remember ever encountering that situation.
Well after 12 years of using ruby and bundler and maintaining a bundler-ish depsolver at work, and playing around a bit with cargo, I can say that it is becoming clearer as to why I don't grok go modules at all.
The lack of a depsolver is a curious choice...
I don't think my example was remotely "broken" at all, that's just another day doing software development.
The lack of a depsolver is a curious choice...
I don't think my example was remotely "broken" at all, that's just another day doing software development.
[deleted]
> I don't think my example was remotely "broken" at all, that's just another day doing software development.
It's not the norm for me. If this is what you consider to be the norm, then this kind of statement doesn't make me feel any better about Ruby.
I will say that Bundler is one of the better package managers, but the existence of the constraint solver doesn't fix this problem -- Bundler doesn't allow you to have multiple versions of a single dependency. The problem is fundamentally the dependency not maintaining its compatibility guarantees, which I would definitely call "broken". Sometimes breakage is unavoidable, like with security fixes that you want to be available even to users of existing SemVer versions, but it should not be a common situation.
It's not the norm for me. If this is what you consider to be the norm, then this kind of statement doesn't make me feel any better about Ruby.
I will say that Bundler is one of the better package managers, but the existence of the constraint solver doesn't fix this problem -- Bundler doesn't allow you to have multiple versions of a single dependency. The problem is fundamentally the dependency not maintaining its compatibility guarantees, which I would definitely call "broken". Sometimes breakage is unavoidable, like with security fixes that you want to be available even to users of existing SemVer versions, but it should not be a common situation.
> Okay but if I depend on A and A depends on C and has it pinned at 1.5, but I also depend on B and B has C pinned on 1.8, then I get 1.5?
No, you end up with the highest explicitly required version. So 1.8 in that scenario, if I followed. (Requiring 1.5 is declaring support for "1.5 or higher". Requiring 1.8 is declaring support for "1.8 or higher". 1.8 satisfies both of those requirements).
> if the constraints on C across various transitive deps are > 1.5 and > 1.8, and 1.9 or 1.10 exists, I probably want the last version of those.
By default, you get 1.8 (for reasons outlined upthread and in the blog post & related links), but you have the option of getting the latest version of C at any time of your choosing (e.g., 'go get C@latest', or 'go get -u ./...' to get latest versions of all dependencies, and so on).
Also, you are using the word "pin". The way it works is that the top-level module in a build has the option to force a particular version of any direct or indirect dependency, but intermediate modules in a build cannot. So as the author of the top-level module, you could force a version of C if you needed to, but for example your dependency B cannot "pin" C in your build.
No, you end up with the highest explicitly required version. So 1.8 in that scenario, if I followed. (Requiring 1.5 is declaring support for "1.5 or higher". Requiring 1.8 is declaring support for "1.8 or higher". 1.8 satisfies both of those requirements).
> if the constraints on C across various transitive deps are > 1.5 and > 1.8, and 1.9 or 1.10 exists, I probably want the last version of those.
By default, you get 1.8 (for reasons outlined upthread and in the blog post & related links), but you have the option of getting the latest version of C at any time of your choosing (e.g., 'go get C@latest', or 'go get -u ./...' to get latest versions of all dependencies, and so on).
Also, you are using the word "pin". The way it works is that the top-level module in a build has the option to force a particular version of any direct or indirect dependency, but intermediate modules in a build cannot. So as the author of the top-level module, you could force a version of C if you needed to, but for example your dependency B cannot "pin" C in your build.
If I'm understanding you properly, recursive lockfiles means that if I depend on some chain of dependencies A->B->C->D->E, and E has a security vulnerability that they patch in a new version, I have to wait for A B C D and E to all update their lockfiles before the security vulnerability will be patched on my system?
That's not correct. You can unilaterally decide to update the version of E without waiting for anyone. Alternatively, if only C for example decides to update their required version of E, you would get that version of E if you updated your version of C (directly or indirectly), without needing to directly do anything with E yourself.
Slightly longer explanation of the mechanics here: https://news.ycombinator.com/item?id=30871730
The best complete explanation is probably here: https://research.swtch.com/vgo-principles
Slightly longer explanation of the mechanics here: https://news.ycombinator.com/item?id=30871730
The best complete explanation is probably here: https://research.swtch.com/vgo-principles
But wouldn't that have the same issue then? Developers decide to update their dependencies to patch any security vulnerabilities, and wind up adding installing node-ipc's malicious update
The difference is that it's an explicit choice instead of other package managers who'd happily install latest compromised versions of packages by default.
But if its a manual explicit choice, that adds more friction to these patches, and many developers may not update them at all. It's a trade off
A manual choice can be easily automated; an automatic choice can be difficult-to-impossible to de-automate.
It is trivial to manually upgrade dependencies in NPM. You just use `npm update <package>` with an optional version number if you want. And upgrading all dependencies of a Go package is also a single command. So honestly it seems like there is very little difference. My main point here is the trade-off. Either you reduce the friction for upgrades, and run the risk of malicious upgrades like node-ipc. Or you increase the friction, and run the risk of security vulnerabilities being unpatched in many projects.
I personally prefer the former. Encourage upgrades, but then NPM should also have a separate repository for community verified / trusted packages to reduce the chance of a random single developer damaging the entire ecosystem (left-pad, node-ipc, etc)
I personally prefer the former. Encourage upgrades, but then NPM should also have a separate repository for community verified / trusted packages to reduce the chance of a random single developer damaging the entire ecosystem (left-pad, node-ipc, etc)
If I set up a new Node project I get the highest 'supported' version of whatever. If I add a new dependency I get the latest version of any transitive dependency I didn't already have. As far as I know that's impossible to disable. That's the automated upgrade I mean.
I see, in that case yes Go does have more tooling for being able to install the minimum vs the latest of all packages, using their `update` command if you want the latest. But it would also be trivial for Node to add a command to grab the minimum of all dependencies when installing new packages. They just haven't felt the need to add such a feature. Because again, it comes down to which side you want to encourage: installing minimum versions to prevent malicious updates, or installing latest to patch security vulnerabilities.
Perhaps
"A module may have a text file named go.sum in its root directory, alongside its go.mod file. The go.sum file contains cryptographic hashes of the module’s direct and indirect dependencies."
And
"If the go.sum file is not present, or if it doesn’t contain a hash for the downloaded file, the go command may verify the hash using the checksum database, a global source of hashes for publicly available modules."
Should be stressed on. If I committed a dependency version (go.mod) and checksum (go.sum) along with the code, either I get a repeatable build everywhere, or build fails if dependency not found or found to be modified.
I am not sure if all other package managers include checksum with dependency version.
"A module may have a text file named go.sum in its root directory, alongside its go.mod file. The go.sum file contains cryptographic hashes of the module’s direct and indirect dependencies."
And
"If the go.sum file is not present, or if it doesn’t contain a hash for the downloaded file, the go command may verify the hash using the checksum database, a global source of hashes for publicly available modules."
Should be stressed on. If I committed a dependency version (go.mod) and checksum (go.sum) along with the code, either I get a repeatable build everywhere, or build fails if dependency not found or found to be modified.
I am not sure if all other package managers include checksum with dependency version.
> the go command may verify the hash
If we're talking about reproducible builds, the word "may" seems concerning here?
If we're talking about reproducible builds, the word "may" seems concerning here?
I suspect the primary purpose of the word "may" in that sentence is that you can choose to disable checking the hash against the Certificate Transparency style https://sum.golang.org. In other words, you can opt out. If you do, you fall back to your local go.sum file, which is more-or-less a "TOFU" security model:
https://en.wikipedia.org/wiki/Trust_on_first_use
More on sum.golang.org: https://go.googlesource.com/proposal/+/master/design/25530-s...
More on sum.golang.org: https://go.googlesource.com/proposal/+/master/design/25530-s...
Thank you for the clarification!
Discussing "what is a lockfile" is a bit of a headache because different languages have different files which do different things. Generally speaking, there's some file which specifies the dependency versions and some file with cryptographic checksums of the all transitive dependencies.
In Go it's go.mod / go.sum. In NPM, it's package.json / package-lock.json. In Rust it's Cargo.toml / Cargo.lock.
Diving into the exact details of what the author is saying is a bit outside my headspace at the moment. I think the author of the article may not actually understand the scenario where Go's package system differs. (I'm not sure I do, either.)
Suppose you have your project, projectA, and its direct dependency, libB. Then libB has a dependency on libC.
If projectA has a lockfile, you get exactly the same versions of libA and libB. This is true for Go, NPM, and Cargo. However, suppose projectA is a new project. You just created it. In Go, the version of libB that makes it into the lockfile will be the minimum version that libA requires, which means that any new, poisoned version of libB will not transitively affect anything that depends on libA, such as projectA. With NPM, you get the latest version of libB which is compatible with libA--this version may be poisoned.
In Go it's go.mod / go.sum. In NPM, it's package.json / package-lock.json. In Rust it's Cargo.toml / Cargo.lock.
Diving into the exact details of what the author is saying is a bit outside my headspace at the moment. I think the author of the article may not actually understand the scenario where Go's package system differs. (I'm not sure I do, either.)
Suppose you have your project, projectA, and its direct dependency, libB. Then libB has a dependency on libC.
If projectA has a lockfile, you get exactly the same versions of libA and libB. This is true for Go, NPM, and Cargo. However, suppose projectA is a new project. You just created it. In Go, the version of libB that makes it into the lockfile will be the minimum version that libA requires, which means that any new, poisoned version of libB will not transitively affect anything that depends on libA, such as projectA. With NPM, you get the latest version of libB which is compatible with libA--this version may be poisoned.
> any new, poisoned version of libB
Conversely, you will get any old security-buggy version of libB instead.
Most package managers when adding a new dependency assume newer versions are "better" than older versions. Go's minimum version system assumes older is better than newer.
I don't think there's any clear argument you can make on first principles for which of those is actually the case. You'd probably have to do an empirical analysis of how often mailicious packages get published versus how often security bug fix versions get published. If the former is more common than the latter, then min version is likely a net positive for security. If the latter is more common than the former, then max version is probably better. You'd probably also have to evaluate the relative harm of malicious versions versus unintended security bugs.
Conversely, you will get any old security-buggy version of libB instead.
Most package managers when adding a new dependency assume newer versions are "better" than older versions. Go's minimum version system assumes older is better than newer.
I don't think there's any clear argument you can make on first principles for which of those is actually the case. You'd probably have to do an empirical analysis of how often mailicious packages get published versus how often security bug fix versions get published. If the former is more common than the latter, then min version is likely a net positive for security. If the latter is more common than the former, then max version is probably better. You'd probably also have to evaluate the relative harm of malicious versions versus unintended security bugs.
> I don't think there's any clear argument you can make on first principles for which of those is actually the case.
I don't understand why someone would try to argue from first principles here, it just seems like such a bizarre approach.
Anyway, it's not just a security issue. Malicious packages and security fixes are only part of the picture. Other issues:
- Despite a team's promise to use semantic versioning, point releases & "bugfix" releases will break downstream users
- Other systems for determining the versions to use are much more unpredictable and hard to understand than estimated (look at Dart and Cargo)
https://github.com/dart-lang/pub/blob/master/doc/solver.md
https://github.com/rust-lang/cargo/blob/1ef1e0a12723ce9548d7...
I don't understand why someone would try to argue from first principles here, it just seems like such a bizarre approach.
Anyway, it's not just a security issue. Malicious packages and security fixes are only part of the picture. Other issues:
- Despite a team's promise to use semantic versioning, point releases & "bugfix" releases will break downstream users
- Other systems for determining the versions to use are much more unpredictable and hard to understand than estimated (look at Dart and Cargo)
https://github.com/dart-lang/pub/blob/master/doc/solver.md
https://github.com/rust-lang/cargo/blob/1ef1e0a12723ce9548d7...
> Other systems for determining the versions to use are much more unpredictable and hard to understand than estimated (look at Dart and Cargo)
I'm one of the co-authors of Dart's package manager. :)
Yes, it is complex. Code reuse is hard and there's no silver bullet.
I'm one of the co-authors of Dart's package manager. :)
Yes, it is complex. Code reuse is hard and there's no silver bullet.
Nice! I hope I wasn't coming across as critical of Dart's package manager, or Cargo for that matter.
It's OK. There are always valid criticisms of all possible package managers. It's just a hard area with gnarly trade-offs.
Every change that fixes a security issue implies the existence of a change that introduced the security issue in the first place. Why is bumping a version more likely to remove security issues instead of introduce them?
The reason why older is better than newer has more to do with the fact that the author has actually tested their software with that specific version, and so there's more of a chance that it actually works as they intended.
The reason why older is better than newer has more to do with the fact that the author has actually tested their software with that specific version, and so there's more of a chance that it actually works as they intended.
Security issues aren't introduced intentionally, oftentimes they are found much later on in code that was assumed to be secure. Like the SSL heartbleed vulnerability. Once a vulnerability like that is discovered, you _want_ every developer to update their deps to the most secure version
My statement had nothing to do with intent. Conversely, once a vulnerability is introduced (intentionally or not), you don't want every developer to update their deps to the newly insecure version.
Exactly, so it's a trade-off, do you want to encourage updates at the risk of malicious updates (like with node-ipc). Or do you want to add friction to updates and thus risk security vulnerabilities persisting for longer. Node chooses one approach, Go chooses the other.
Again, it's not just malicious updates. Normal updates can also introduce security vulnerabilities. For example, I have a dependency at v1.0 and v1.0.1 introduces a security bug unintentionally. It is eventually fixed in v1.1. If I wait to update until v1.1, then I am not vulnerable to that bug whereas an automatic update to v1.0.1 would be vulnerable. My point is that in expectation, updating your dependency could be just as likely to remove a security vulnerability as it is to add one.
I'd back that down to "most security issues aren't introduced intentionally".
Go just expects you to manually trigger the updates. Thats all. It still is in favor of updating to take security fixes, so i think your argument is wrong.
Let's say my_app uses package foo which uses package bar.
It turns out there is a security bug in bar. The bar maintainers release a patch version that fixes it.
In most package managers, users of my_app can and will get that fix with no work on the part of the author of foo. I'm not very familiar with Go's approach but I thought that unless foo's author puts out a version of foo that bumps its minimum version dependency on bar, my_app won't get the fix. Version solving will continue to select the old buggy version of bar because that's the minimum version the current version of foo permits.
It turns out there is a security bug in bar. The bar maintainers release a patch version that fixes it.
In most package managers, users of my_app can and will get that fix with no work on the part of the author of foo. I'm not very familiar with Go's approach but I thought that unless foo's author puts out a version of foo that bumps its minimum version dependency on bar, my_app won't get the fix. Version solving will continue to select the old buggy version of bar because that's the minimum version the current version of foo permits.
> I thought that unless foo's author puts out a version of foo that bumps its minimum version dependency on bar, my_app won't get the fix. Version solving will continue to select the old buggy version of bar because that's the minimum version the current version of foo permits.
That is incorrect. The application's go.mod defines all dependencies, even indirect ones. Raise the version there, and you raise it for all dependencies. You cannot have one more than one minor version of a dependency in the dependency graph.
That is incorrect. The application's go.mod defines all dependencies, even indirect ones. Raise the version there, and you raise it for all dependencies. You cannot have one more than one minor version of a dependency in the dependency graph.
That still implies that I need to know to update the version constraint of what may be a very deep transitive dependency, doesn't it?
The version constraint is always listed in your top level go.mod file, so you know the dependency exists, no digging into the dependency tree required at all, and it’s not hidden in some lock file no one ever looks at. Plus, there are plenty of tools that help you with this problem, including the language server helping you directly in your editor and Dependabot on GitHub.
I’m not aware of any languages that send you an email when your dependencies are out of date, so yes, you need to check them. Dependabot can do this for you and open a PR automatically, which will result in an email, so this is one way for people to stay on top of this stuff even for projects they deploy but don’t work on every single week.
If you’re suggesting that indirect dependencies should automatically update themselves, then you are quite literally saying those code authors should have a shell into your production environments that you have no control over, compromising all your systems with a single package update that no one but the malicious author got to review. It is possible with tools like Dependabot to be notified proactively when updates are required so you can review and apply those, but it is not possible to go back in time and un-apply a malicious update that went straight to prod.
Repeatedly assuming that the Go core team never thought through the design of Go Modules and how it relates to security updates is such a strange choice. Go is a very widely used language with tons of great tooling.
I’m not aware of any languages that send you an email when your dependencies are out of date, so yes, you need to check them. Dependabot can do this for you and open a PR automatically, which will result in an email, so this is one way for people to stay on top of this stuff even for projects they deploy but don’t work on every single week.
If you’re suggesting that indirect dependencies should automatically update themselves, then you are quite literally saying those code authors should have a shell into your production environments that you have no control over, compromising all your systems with a single package update that no one but the malicious author got to review. It is possible with tools like Dependabot to be notified proactively when updates are required so you can review and apply those, but it is not possible to go back in time and un-apply a malicious update that went straight to prod.
Repeatedly assuming that the Go core team never thought through the design of Go Modules and how it relates to security updates is such a strange choice. Go is a very widely used language with tons of great tooling.
I'm sorry but I'm not super familiar with the workflow for working with dependencies in Go, I've only read about it. You say:
> Raise the version there.
Am I to understand that it's common to hand-edit the version constraint on a transitive dependency in your go.mod file?
But that transitive dependency was first added there by the Go tool itself, right?
How does a user easily keep track of what bits of data in the go.mod file are hand-maintained and need to be preserved and which things were filled in implicitly by the tool traversing dependency graphs?
> Repeatedly assuming that the Go core team never thought through the design of Go Modules
I'm certainly not assuming that. But I'm also not assuming they found a perfect solution to package management that all other package management tools failed to find. What's more likely is that they chose a different set of trade-offs, which is what this thread is exploring.
> Raise the version there.
Am I to understand that it's common to hand-edit the version constraint on a transitive dependency in your go.mod file?
But that transitive dependency was first added there by the Go tool itself, right?
How does a user easily keep track of what bits of data in the go.mod file are hand-maintained and need to be preserved and which things were filled in implicitly by the tool traversing dependency graphs?
> Repeatedly assuming that the Go core team never thought through the design of Go Modules
I'm certainly not assuming that. But I'm also not assuming they found a perfect solution to package management that all other package management tools failed to find. What's more likely is that they chose a different set of trade-offs, which is what this thread is exploring.
> Am I to understand that it's common to hand-edit the version constraint on a transitive dependency in your go.mod file?
No, run `go get <package with vuln>@<version that fixes vuln>` and Go will do it for you.
No, run `go get <package with vuln>@<version that fixes vuln>` and Go will do it for you.
>No, run `go get <package with vuln>@<version that fixes vuln>` and Go will do it for you.
The point GP was making is that it's not a given you'll know that there's a security vuln in a sub-sub-sub-dependency of your app. Is it reasonable to expect developers to manually keep tabs on what could be dozens of libraries that may or may not intersect with the dependencies of any other apps you have on the go?
Maybe for Google scale where you can "just throw more engineers at the problem".
The point GP was making is that it's not a given you'll know that there's a security vuln in a sub-sub-sub-dependency of your app. Is it reasonable to expect developers to manually keep tabs on what could be dozens of libraries that may or may not intersect with the dependencies of any other apps you have on the go?
Maybe for Google scale where you can "just throw more engineers at the problem".
> Is it reasonable to expect developers to manually keep tabs on what could be dozens of libraries that may or may not intersect with the dependencies of any other apps you have on the go?
Well, in the NPM model you need at least one transitive dependency to notice it and upgrade, and you need to notice your transitive dependency upgraded. But also, it might upgrade despite nobody asking for it just because you set up a new dependency.
In the Go model... you need at least one transitive dependency to notice it and upgrade, and you need to notice your transitive dependency upgraded. But at least it won't ever upgrade unless someone asked for it.
Well, in the NPM model you need at least one transitive dependency to notice it and upgrade, and you need to notice your transitive dependency upgraded. But also, it might upgrade despite nobody asking for it just because you set up a new dependency.
In the Go model... you need at least one transitive dependency to notice it and upgrade, and you need to notice your transitive dependency upgraded. But at least it won't ever upgrade unless someone asked for it.
Well, the easy thing is just let something like Dependabot update your stuff.
If you are just wanting "update all my stuff to the latest version", just run `go get -u ./...`?
> Am I to understand that it's common to hand-edit the version constraint on a transitive dependency in your go.mod file?
"Common" is probably not accurate, but what's wrong with hand editing it? If you mess up, your project won't compile until you fix it. You can update the versions a dozen different ways: editing by hand, `go get -u <dependency>` to update a specific dependency, `go get -u ./...` to update all dependencies, using your editor to view the go.mod file and select dependencies to update with the language server integration or by telling the language server to update all dependencies, by using Dependabot, or however else you like to do it. The options are all there for whatever way you're most comfortable with.
> But that transitive dependency was first added there by the Go tool itself, right?
So what? It's still your dependency now, and you are equally as responsible for watching after it as you are for any direct dependency. Any package management system that hides transitive dependencies is encouraging the user to ignore a significant fraction of the code that makes up their application. Every dependency is important.
> How does a user easily keep track of what bits of data in the go.mod file are hand-maintained and need to be preserved and which things were filled in implicitly by the tool traversing dependency graphs?
You already know[0] the answer to the most of that question, so I don't know why you're asking that part again. As a practical example, here is Caddy's go.mod file.[1] You can see the two distinct "require" blocks and the machine-generated comments that inform the reader that the second block is full of indirect dependencies. No one looking at this should be confused about which is which.
But the other part of that question doesn't really make sense. If you don't "preserve" the dependency, your code won't compile because there will be an unsatisfied dependency, regardless of whether you wrote the dependency there by hand or not, and regardless of whether it is a direct or indirect dependency. If you try to compile a program that depends on something not listed in the `go.mod` file, it won't compile. You can issue `go mod tidy` at any time to have Go edit your go.mod file for you to satisfy all constraints, so if you delete a line for whatever reason, `go mod tidy` can add it back, and Go doesn't update dependencies unless you ask it to, so `go mod tidy` won't muck around with other stuff in the process. There are very few ways the user can shoot themselves in the foot here.
Regardless, it is probably uncommon for people to manually edit anything in the go.mod file when there are so many tools that will handle it for you. The typical pattern for adding a dependency that I've observed is to add the import for the dependency from where you're trying to use it, and then tell your editor to update the go.mod file to include that dependency for you. The only time someone is likely to add a dependency to the go.mod file by hand editing it is when they need to do a "replace" operation to substitute one dependency in place of another (which applies throughout the dependency tree), usually only done when you need to patch a bug in third party code before the upstream is ready to merge that fix.
In my experience, most people either use Dependabot to keep up to date with their dependencies, or they update the dependencies using VS Code to view the go.mod file and click the "buttons"(/links/whatever) that the language server visually adds to the file to let you do the common tasks with a single click. They're both extremely simple to use and help you to update your direct and indirect dependencies.
> But I'm also not assuming they found a perfect solution to package management that all other package management tools failed to find. What's more likely is that they chose a different set of trade-offs, which is what this thread is exploring.
They were extremely late to the party, so they were able to learn from everyone else's mistakes. I really don't think it should be surprising that a latecomer is able to find solutions that other package management tools didn't, because the latecomer has the benefit of hindsight. They went from having some of the worst package management in the industry (effectively none; basically only beating out C and C++... and maybe Python, package management for which has been a nightmare for forever) to having arguably the best. Rust's Cargo comes extremely close, and I've used them both (as well as others) for the past 5+ years in several professional contexts. (Yes, that includes time before Go Modules existed, in the days when there was no real package management built in.) It seems like humans often want to assume that "things probably suck just as much everywhere else, just in different ways, and those people must simply be hiding it", but that's not always the case.
Some "trade-offs":
- For awhile, the `go` command would constantly be modifying your `go.mod` file for you whenever it didn't like what was in there, and that was definitely something I would have chalked up as a "trade-off", but they fixed it. Go will not touch your `go.mod` file unless you explicitly tell it to, which is a huge improvement in the consistency of the user experience.
- Go Modules requires version numbers to start with a "v", which annoys some people, because those people had been using git tags for years to track versions without a "v", so you could argue that's a trade-off too.
- There has been some debate about the way that major versions are implemented, since it requires you to change the name of the package for each major version increment. (By appending “/v2”, “/v3”, etc to the package name. The justification for this was to allow you to import multiple major versions of the same package into your dependency graph — and even the same file — without conflict.)
- The fact that the packages are still named after URLs is a source of consternation for some people, but it's only annoying in practice when you need to move the package from one organization to another or from one domain name to another. It's simply not an issue the rest of the time. Some people are also understandably confused into thinking that third party Go modules can vanish at any time because they're coming from URLs, but there is a transparent, immutable package proxy enabled by default that keeps copies of all[#] versions of all public dependencies that are fetched, so even if the original repo is deleted, the dependency will generally still continue to work indefinitely, and the lock sums will be retained indefinitely to prevent any malicious updates to existing dependency versions, which means that tampering is prevented both locally by your go.sum file and remotely by the proxy as an extra layer of protection for new projects that don't have a go.sum file yet. It is possible to disable this proxy (in whole or in part) or self-host your own if desired, but... I haven't encountered any use case that would dictate either. ([#]: There are a handful of rare exceptions involving packages without proper licenses which will only be cached for short periods of time plus the usual DMCA takedown notices that affect all "immutable" package registries, from what I understand.)
Beyond that... I don't know of any trade-offs. Seriously. I have taken the time to think through this and list what I could come up with above. A "trade-off" implies that some decision they made has known pros and cons. What are the cons? Maybe they could provide some "nicer" commands like `go mod update` to update all your dependencies instead of the somewhat obtuse `go get -u ./...` command? You have complained in several places about how Go combines indirect dependencies into the `go.mod` file, but... how is that not objectively better? Dependencies are dependencies. They all matter, and hiding some of them doesn't actually help anything except maybe aesthetics? I would love to know how making some of them exist exclusively in the lock file helps at all. Before Go Modules, I was always fine with that because I had never experienced anything better, but now I have.
There are plenty of things I would happily criticize about Go these days, but package management isn't one of them... it is simply stellar these days. It definitely did go through some growing pains, as any existing language adopting a new package manager would, and the drama that resulted from such a decision left a bitter taste in the mouth of some people, but I don't believe that bitterness was technical as much as it was a result of poor transparency from the core team with the community.
[0]: https://news.ycombinator.com/item?id=30870862
[1]: https://github.com/caddyserver/caddy/blob/master/go.mod
"Common" is probably not accurate, but what's wrong with hand editing it? If you mess up, your project won't compile until you fix it. You can update the versions a dozen different ways: editing by hand, `go get -u <dependency>` to update a specific dependency, `go get -u ./...` to update all dependencies, using your editor to view the go.mod file and select dependencies to update with the language server integration or by telling the language server to update all dependencies, by using Dependabot, or however else you like to do it. The options are all there for whatever way you're most comfortable with.
> But that transitive dependency was first added there by the Go tool itself, right?
So what? It's still your dependency now, and you are equally as responsible for watching after it as you are for any direct dependency. Any package management system that hides transitive dependencies is encouraging the user to ignore a significant fraction of the code that makes up their application. Every dependency is important.
> How does a user easily keep track of what bits of data in the go.mod file are hand-maintained and need to be preserved and which things were filled in implicitly by the tool traversing dependency graphs?
You already know[0] the answer to the most of that question, so I don't know why you're asking that part again. As a practical example, here is Caddy's go.mod file.[1] You can see the two distinct "require" blocks and the machine-generated comments that inform the reader that the second block is full of indirect dependencies. No one looking at this should be confused about which is which.
But the other part of that question doesn't really make sense. If you don't "preserve" the dependency, your code won't compile because there will be an unsatisfied dependency, regardless of whether you wrote the dependency there by hand or not, and regardless of whether it is a direct or indirect dependency. If you try to compile a program that depends on something not listed in the `go.mod` file, it won't compile. You can issue `go mod tidy` at any time to have Go edit your go.mod file for you to satisfy all constraints, so if you delete a line for whatever reason, `go mod tidy` can add it back, and Go doesn't update dependencies unless you ask it to, so `go mod tidy` won't muck around with other stuff in the process. There are very few ways the user can shoot themselves in the foot here.
Regardless, it is probably uncommon for people to manually edit anything in the go.mod file when there are so many tools that will handle it for you. The typical pattern for adding a dependency that I've observed is to add the import for the dependency from where you're trying to use it, and then tell your editor to update the go.mod file to include that dependency for you. The only time someone is likely to add a dependency to the go.mod file by hand editing it is when they need to do a "replace" operation to substitute one dependency in place of another (which applies throughout the dependency tree), usually only done when you need to patch a bug in third party code before the upstream is ready to merge that fix.
In my experience, most people either use Dependabot to keep up to date with their dependencies, or they update the dependencies using VS Code to view the go.mod file and click the "buttons"(/links/whatever) that the language server visually adds to the file to let you do the common tasks with a single click. They're both extremely simple to use and help you to update your direct and indirect dependencies.
> But I'm also not assuming they found a perfect solution to package management that all other package management tools failed to find. What's more likely is that they chose a different set of trade-offs, which is what this thread is exploring.
They were extremely late to the party, so they were able to learn from everyone else's mistakes. I really don't think it should be surprising that a latecomer is able to find solutions that other package management tools didn't, because the latecomer has the benefit of hindsight. They went from having some of the worst package management in the industry (effectively none; basically only beating out C and C++... and maybe Python, package management for which has been a nightmare for forever) to having arguably the best. Rust's Cargo comes extremely close, and I've used them both (as well as others) for the past 5+ years in several professional contexts. (Yes, that includes time before Go Modules existed, in the days when there was no real package management built in.) It seems like humans often want to assume that "things probably suck just as much everywhere else, just in different ways, and those people must simply be hiding it", but that's not always the case.
Some "trade-offs":
- For awhile, the `go` command would constantly be modifying your `go.mod` file for you whenever it didn't like what was in there, and that was definitely something I would have chalked up as a "trade-off", but they fixed it. Go will not touch your `go.mod` file unless you explicitly tell it to, which is a huge improvement in the consistency of the user experience.
- Go Modules requires version numbers to start with a "v", which annoys some people, because those people had been using git tags for years to track versions without a "v", so you could argue that's a trade-off too.
- There has been some debate about the way that major versions are implemented, since it requires you to change the name of the package for each major version increment. (By appending “/v2”, “/v3”, etc to the package name. The justification for this was to allow you to import multiple major versions of the same package into your dependency graph — and even the same file — without conflict.)
- The fact that the packages are still named after URLs is a source of consternation for some people, but it's only annoying in practice when you need to move the package from one organization to another or from one domain name to another. It's simply not an issue the rest of the time. Some people are also understandably confused into thinking that third party Go modules can vanish at any time because they're coming from URLs, but there is a transparent, immutable package proxy enabled by default that keeps copies of all[#] versions of all public dependencies that are fetched, so even if the original repo is deleted, the dependency will generally still continue to work indefinitely, and the lock sums will be retained indefinitely to prevent any malicious updates to existing dependency versions, which means that tampering is prevented both locally by your go.sum file and remotely by the proxy as an extra layer of protection for new projects that don't have a go.sum file yet. It is possible to disable this proxy (in whole or in part) or self-host your own if desired, but... I haven't encountered any use case that would dictate either. ([#]: There are a handful of rare exceptions involving packages without proper licenses which will only be cached for short periods of time plus the usual DMCA takedown notices that affect all "immutable" package registries, from what I understand.)
Beyond that... I don't know of any trade-offs. Seriously. I have taken the time to think through this and list what I could come up with above. A "trade-off" implies that some decision they made has known pros and cons. What are the cons? Maybe they could provide some "nicer" commands like `go mod update` to update all your dependencies instead of the somewhat obtuse `go get -u ./...` command? You have complained in several places about how Go combines indirect dependencies into the `go.mod` file, but... how is that not objectively better? Dependencies are dependencies. They all matter, and hiding some of them doesn't actually help anything except maybe aesthetics? I would love to know how making some of them exist exclusively in the lock file helps at all. Before Go Modules, I was always fine with that because I had never experienced anything better, but now I have.
There are plenty of things I would happily criticize about Go these days, but package management isn't one of them... it is simply stellar these days. It definitely did go through some growing pains, as any existing language adopting a new package manager would, and the drama that resulted from such a decision left a bitter taste in the mouth of some people, but I don't believe that bitterness was technical as much as it was a result of poor transparency from the core team with the community.
[0]: https://news.ycombinator.com/item?id=30870862
[1]: https://github.com/caddyserver/caddy/blob/master/go.mod
> You already know[0] the answer to the most of that question, so I don't know why you're asking that part again.
My point is that once you start editing (either by hand or through a tool) the version numbers in the second section, then that's no longer cached state derived entirely from the go.mod files of your dependencies which can be regenerated from scratch. It contains some human-authored decisions and regenerating that section without care will drop information on the floor.
Imagine you:
1. Add a dependency on foo, which depends on bar 1.1.
2. Decide to update the version of bar to 1.2.
3. Commit.
4. Weeks later, remove the dependency on foo and tweak some other dependencies. Tidy your go.mod file.
5. Change your mind and re-add the dependency on foo.
At this point, if you look at the diff of your go.mod file, you see that the indirect dependency on bar has changed from 1.2 to 1.1. Is that because:
A. You made a mistake and accidentally lost a deliberate upgrade to that version and you should revert that line.
B. It's a correct downgrade because your other dependency changes which have a shared dependency on bar no longer need 1.2 and it is correctly giving you the minimum version 1.1.
Maybe the answer here is that even when you ask the tool to remove unneeded transitive dependencies, it won't roll back to an earlier version of bar? So it will keep it at 1.2?
With other package management tools, this is obvious: Any hand-authored intent to pin versions—including for transitive dependencies—lives in one file, and all the state derived from that is in another.
> In my experience, most people either use Dependabot to keep up to date with their dependencies, or they update the dependencies using VS Code to view the go.mod file and click the "buttons"(/links/whatever) that the language server visually adds to the file to let you do the common tasks with a single click. They're both extremely simple to use and help you to update your direct and indirect dependencies.
This sounds like you more or less get the same results as you would in other package managers, but with extra steps.
I don't know. I guess I just don't understand the system well enough.
My point is that once you start editing (either by hand or through a tool) the version numbers in the second section, then that's no longer cached state derived entirely from the go.mod files of your dependencies which can be regenerated from scratch. It contains some human-authored decisions and regenerating that section without care will drop information on the floor.
Imagine you:
1. Add a dependency on foo, which depends on bar 1.1.
2. Decide to update the version of bar to 1.2.
3. Commit.
4. Weeks later, remove the dependency on foo and tweak some other dependencies. Tidy your go.mod file.
5. Change your mind and re-add the dependency on foo.
At this point, if you look at the diff of your go.mod file, you see that the indirect dependency on bar has changed from 1.2 to 1.1. Is that because:
A. You made a mistake and accidentally lost a deliberate upgrade to that version and you should revert that line.
B. It's a correct downgrade because your other dependency changes which have a shared dependency on bar no longer need 1.2 and it is correctly giving you the minimum version 1.1.
Maybe the answer here is that even when you ask the tool to remove unneeded transitive dependencies, it won't roll back to an earlier version of bar? So it will keep it at 1.2?
With other package management tools, this is obvious: Any hand-authored intent to pin versions—including for transitive dependencies—lives in one file, and all the state derived from that is in another.
> In my experience, most people either use Dependabot to keep up to date with their dependencies, or they update the dependencies using VS Code to view the go.mod file and click the "buttons"(/links/whatever) that the language server visually adds to the file to let you do the common tasks with a single click. They're both extremely simple to use and help you to update your direct and indirect dependencies.
This sounds like you more or less get the same results as you would in other package managers, but with extra steps.
I don't know. I guess I just don't understand the system well enough.
> It contains some human-authored decisions and regenerating that section without care will drop information on the floor.
> Maybe the answer here is that even when you ask the tool to remove unneeded transitive dependencies, it won't roll back to an earlier version of bar? So it will keep it at 1.2?
I'm not aware of any package management system that will remember dependencies you no longer depend on, except by human error when you forget to remove that dependency but keep punishing yourself and others by making them build and install that unused dependency. No matter where the information lived before it was deleted, it's still up to the human to do the right thing in a convoluted scenario like you're describing.
> With other package management tools, this is obvious: Any hand-authored intent to pin versions—including for transitive dependencies—lives in one file, and all the state derived from that is in another.
That doesn't solve anything if you don't look go back to the commit that had that information. If you do go back to that commit, you have all the information you need right there anyways. You can add your own comments to the `go.mod` file, so if something changed for an important reason, you can keep track of that (and the why) just as easily as you can in any other format. Actually, easier than some... does package.json even support comments yet? But it only matters if you go back and look at the previously-deleted dependency information instead of just blindly re-adding it as I imagine most people would do, which is a huge assumption.
>> It seems like humans often want to assume that "things probably suck just as much everywhere else, just in different ways, and those people must simply be hiding it", but that's not always the case.
> This sounds like you more or less get the same results as you would in other package managers, but with extra steps.
> I don't know. I guess I just don't understand the system well enough.
I've done my best to explain it to you, literally multiple hours of my day writing these comments. Maybe I suck at explaining, or maybe you're not interested in what I have to say, or maybe I'm somehow completely wrong on everything (but no one has bothered to explain how). "More or less the same" is not the same. The nuances make a huge difference, and Go Modules has done things incredibly well on the whole. Package managers aren't a zero sum game where you shuffle a deck of pros and cons, and you end up with the same number of pros and cons at the end.
> Maybe the answer here is that even when you ask the tool to remove unneeded transitive dependencies, it won't roll back to an earlier version of bar? So it will keep it at 1.2?
I'm not aware of any package management system that will remember dependencies you no longer depend on, except by human error when you forget to remove that dependency but keep punishing yourself and others by making them build and install that unused dependency. No matter where the information lived before it was deleted, it's still up to the human to do the right thing in a convoluted scenario like you're describing.
> With other package management tools, this is obvious: Any hand-authored intent to pin versions—including for transitive dependencies—lives in one file, and all the state derived from that is in another.
That doesn't solve anything if you don't look go back to the commit that had that information. If you do go back to that commit, you have all the information you need right there anyways. You can add your own comments to the `go.mod` file, so if something changed for an important reason, you can keep track of that (and the why) just as easily as you can in any other format. Actually, easier than some... does package.json even support comments yet? But it only matters if you go back and look at the previously-deleted dependency information instead of just blindly re-adding it as I imagine most people would do, which is a huge assumption.
>> It seems like humans often want to assume that "things probably suck just as much everywhere else, just in different ways, and those people must simply be hiding it", but that's not always the case.
> This sounds like you more or less get the same results as you would in other package managers, but with extra steps.
> I don't know. I guess I just don't understand the system well enough.
I've done my best to explain it to you, literally multiple hours of my day writing these comments. Maybe I suck at explaining, or maybe you're not interested in what I have to say, or maybe I'm somehow completely wrong on everything (but no one has bothered to explain how). "More or less the same" is not the same. The nuances make a huge difference, and Go Modules has done things incredibly well on the whole. Package managers aren't a zero sum game where you shuffle a deck of pros and cons, and you end up with the same number of pros and cons at the end.
As I understand it, that's true in the simple case. If you have `my_app -> foo -> bar` then there's only one path to bar, and you only get a new bar when you upgrade foo.
It's more complicated in general, with diamond dependencies. There needs to be a chain of module updates between you and foo, with the minimum case being a chain of length one where you specify the version of foo directly.
So, people do need to pay attention to security patch announcements. But popular modules, at least, are likely to be get updated relatively quickly, because only one side of a diamond dependency needs to notice and do a release.
It's more complicated in general, with diamond dependencies. There needs to be a chain of module updates between you and foo, with the minimum case being a chain of length one where you specify the version of foo directly.
So, people do need to pay attention to security patch announcements. But popular modules, at least, are likely to be get updated relatively quickly, because only one side of a diamond dependency needs to notice and do a release.
> As I understand it, that's true in the simple case. If you have `my_app -> foo -> bar` then there's only one path to bar, and you only get a new bar when you upgrade foo.
This is not correct. You can update bar independent of foo directly from the top-level go.mod file in your project.
This is not correct. You can update bar independent of foo directly from the top-level go.mod file in your project.
Yes, you can do that by adding a direct dependency on foo. I started by talking about when there isn't a direct dependency on foo.
I explicitly talked about having a direct dependency at the end of the second paragraph.
I explicitly talked about having a direct dependency at the end of the second paragraph.
I'm not talking about direct dependencies. Direct and indirect are all listed in the go.mod file. If they aren't listed there, then they aren't in your final binary. If you delete indirect dependencies from the top-level go.mod, your project will fail to compile.
You keep harping on old buggy version where Go has been very clear that it is operator's explicit responsibility to have correct/updated/fixed versions of dependency running.
It specially does not look good in your case considering you work for Google on a different programing language. If you have a clear point to make then compare it with your approach. Instead of making neutral sounding arguments when they are not.
It specially does not look good in your case considering you work for Google on a different programing language. If you have a clear point to make then compare it with your approach. Instead of making neutral sounding arguments when they are not.
Don't drag in ad-hominem attacks. If you want to defend Go's approach, explain why having it be the "operator's explicit responsibility" is a good policy, likely to make apps (in general) more secure. The obvious implication of the example given is that, on average, it will be a mess.
The code itself isn’t the only risk factor; it’s weighted by others, like discoverability, which are asymmetric in time. If the white hats fix an issue in version n+1, they’re going to make sure people know. If black hats (or normal devs making a mistake) introduce an issue, no one will tell you about it.
I.e. even if both strategies win just as often, min-version pulls ahead by taking less of a hit from losses.
I.e. even if both strategies win just as often, min-version pulls ahead by taking less of a hit from losses.
The author is saying that Go provides the same guarantees with just a package list in the go.mod file that other package managers need both a package list and lock file to solve.
go.sum is essentially a distributed / community maintained transparency log of published versions of packages.
go.sum is essentially a distributed / community maintained transparency log of published versions of packages.
Maybe I'm just not familiar with it enough but I don't see how merging a package manifest and lockfile into a single file is a net win.
This means it's no longer clear which dependencies are immediate and which are transitive. It's not clear which versions are user-authored constraints versus system-authored version selections. For dependencies that are transitive, it's not clear why the dependency is in there and which versions of which other dependencies require it.
Other packages separate these into two files because they are very different sets of information. Maybe Go's minimum version selection makes that not the case, but it still seems user-unfriendly to me to lump immediate and transitive dependencies together.
This means it's no longer clear which dependencies are immediate and which are transitive. It's not clear which versions are user-authored constraints versus system-authored version selections. For dependencies that are transitive, it's not clear why the dependency is in there and which versions of which other dependencies require it.
Other packages separate these into two files because they are very different sets of information. Maybe Go's minimum version selection makes that not the case, but it still seems user-unfriendly to me to lump immediate and transitive dependencies together.
FWIW, there are two machine-formatted sections of go.mod -- the first for direct dependencies, the second section for indirect dependencies.
(That's as of Go 1.17. Previously, that information was communicated via machine-generated comments in a single section).
(That's as of Go 1.17. Previously, that information was communicated via machine-generated comments in a single section).
That seems reasonable.
I think I personally lean towards keeping them in separate files entirely because I like a clearer separation between human-authored content and machine-derived state.
I think I personally lean towards keeping them in separate files entirely because I like a clearer separation between human-authored content and machine-derived state.
but if you ever bump up the version of an indirect dependency (maybe to pick up a bugfix earlier), is this now a direct or indirect dependency?
In other systems, you do that by creating a direct dependency. Even though your code doesn't directly import the module, you author an explicit dependency because you care about the version of the module that your app ends up using.
This way, there's a clear separation between human-authored intentional dependencies and the automatically-derived solved versions and transitive dependencies based on that.
If you see a diff in your package manifest, you know a human did that. If you see a diff in the lockfile, you know that was derived from your manifest and the manifests you depend on. The only human input is when they choose to regenerate the lockfile and which packages they request to be unlocked and upgraded. That's still important data because regenerating the lockfile at different points in time will produce different results, but it's a different kind of data and I think it's helpful to keep it separated.
This way, there's a clear separation between human-authored intentional dependencies and the automatically-derived solved versions and transitive dependencies based on that.
If you see a diff in your package manifest, you know a human did that. If you see a diff in the lockfile, you know that was derived from your manifest and the manifests you depend on. The only human input is when they choose to regenerate the lockfile and which packages they request to be unlocked and upgraded. That's still important data because regenerating the lockfile at different points in time will produce different results, but it's a different kind of data and I think it's helpful to keep it separated.
Direct means that your code actually imports it directly instead of transitively. It has nothing to do with the version of that dependency being selected in the go.mod file.
[deleted]
Npm lockfile will refer to same package because you can't republish npm with the same tag (there is also content hash in lockfile).
Referring to version tag in git from go's mod can't guarantee that because you can overwrite tag in git.
Am I wrong?
Referring to version tag in git from go's mod can't guarantee that because you can overwrite tag in git.
Am I wrong?
Yes. The go.sum file that sits alongside go.mod keeps track of the hashes so that no modification like that can be made, and dependency fetches actually transparently go through a module proxy/mirror that keeps those same hashes as well, and it will prevent you from getting an altered version of a known module even if you’re starting a new project and don’t have a sum file yet. Versions can’t be republished.
Thanks for clarification, indeed I can see go.sum being checked in on few go package repos I've checked, nice.
Most package managers have lockfiles. Yes, npm's decision to have both "npm install" and "npm ci", just so you can confuse and mislead developers, is a bit silly.
But Ruby's Bundler, for example, has been refusing to run your code if your lockfile is inconsistent for as long as I remember.
Locking dependencies is, generally, a solved problem across most ecosystems (despite node botching its UX). Go doesn't get to claim that it's superior here.
But of course, supply chain attacks are still possible with lock files. Because somebody is going to update your dependencies at some point (often for security reasons). And at that point you might be pulling in a malicious dependency which you haven't carefully vetted (because nobody has time to vet all their dependencies thoroughly nowadays).
That's still an unsolved problem, as far as I know. I don't think that Go has solved it.
But Ruby's Bundler, for example, has been refusing to run your code if your lockfile is inconsistent for as long as I remember.
Locking dependencies is, generally, a solved problem across most ecosystems (despite node botching its UX). Go doesn't get to claim that it's superior here.
But of course, supply chain attacks are still possible with lock files. Because somebody is going to update your dependencies at some point (often for security reasons). And at that point you might be pulling in a malicious dependency which you haven't carefully vetted (because nobody has time to vet all their dependencies thoroughly nowadays).
That's still an unsolved problem, as far as I know. I don't think that Go has solved it.
> Most package managers have lockfiles
> Locking dependencies is, generally, a solved problem across most ecosystems
As someone who is building a package manager for work, and has looked at pretty much every package manager out there (and their ecosystem adoption), I can only say that those don't reflect the current reality of package management (no matter how much I wish it were true).
Bundler was the first mainstream package manager to adopt a lockfile (AFAIK) a mere 12 years ago. Many many language ecosystems predate that and are still lacking lockfiles (or even widespread adoption of a single compatible package manager).
NPM only got lockfiles 5 years ago (after being pressured by yarn). Gradle got them less than 3.5 years ago, and Maven still doesn't have them (though a niche plugin for it exists). The Python ecosystem is still a hot mess, with ~3 competing solutions (Poetry, Pipenv, Conda), of which Conda just got a 1.0 of their decent conda-lock incubation project a month ago, but due to how setuputils works, the cross-platform package management story is broken almost beyond recovery. In Conan lockfiles are still an experimental feature today.
I could go on and on, but I hope that I could paint a picture that while one could argue that with the advent of lockfiles, locking dependencies has become a solved problem _conceptually_, the current status of implementation across ecosystems is still horrible. I'm also constantly amazed about how little love is put into package managers in most language communities, even though they are so crucial for their respective ecosystems.
As far as I can tell nowadays Go does have one of the better package managers, which given their horrible starting point is quite the feat. As a nice side-effect of experiments in the Go package ecosystem, one of the people working on go dep also created one of the best resources around package managers: https://medium.com/@sdboyer/so-you-want-to-write-a-package-m...
> Locking dependencies is, generally, a solved problem across most ecosystems
As someone who is building a package manager for work, and has looked at pretty much every package manager out there (and their ecosystem adoption), I can only say that those don't reflect the current reality of package management (no matter how much I wish it were true).
Bundler was the first mainstream package manager to adopt a lockfile (AFAIK) a mere 12 years ago. Many many language ecosystems predate that and are still lacking lockfiles (or even widespread adoption of a single compatible package manager).
NPM only got lockfiles 5 years ago (after being pressured by yarn). Gradle got them less than 3.5 years ago, and Maven still doesn't have them (though a niche plugin for it exists). The Python ecosystem is still a hot mess, with ~3 competing solutions (Poetry, Pipenv, Conda), of which Conda just got a 1.0 of their decent conda-lock incubation project a month ago, but due to how setuputils works, the cross-platform package management story is broken almost beyond recovery. In Conan lockfiles are still an experimental feature today.
I could go on and on, but I hope that I could paint a picture that while one could argue that with the advent of lockfiles, locking dependencies has become a solved problem _conceptually_, the current status of implementation across ecosystems is still horrible. I'm also constantly amazed about how little love is put into package managers in most language communities, even though they are so crucial for their respective ecosystems.
As far as I can tell nowadays Go does have one of the better package managers, which given their horrible starting point is quite the feat. As a nice side-effect of experiments in the Go package ecosystem, one of the people working on go dep also created one of the best resources around package managers: https://medium.com/@sdboyer/so-you-want-to-write-a-package-m...
You are probably right that, practically, not every language has "solved" dependency locking and that I was probably unduly extrapolating from my experiences with ecosystems where this has been solved, but there are enough package managers (Bundler, Yarn, Poetry, that I know of, and from what people claim, also Composer and Cargo) that have solved it so that go claiming credit for it seems unwarranted. If anything, this should be credited to Bundler (though it's possible that it wasn't the first package manager to do so).
I can give no credit to npm, it would never have had lockfiles if not for yarn, and even its current attempt seems half-assed.
Python has the problem of too many package managers, some which are bad, unfortunately (the list of open bugs for Pipenv is especially alarming; I once had to rip it out of a project because the dependency resolution failed after half an hour with a stack overflow). That said, poetry solves the problem well and correctly, IMHO.
Over in the Java world, you're right that dependency locking is a bit rarer. But I also think the situation is not nearly as dire there. BOMs, where lists of compatible dependencies are curated, are relatively common, so that alleviates some of the pain. Plus, there seems to be less churn than in some other ecosystems. Still, it would probably be technically better to use dependency locking.
I can give no credit to npm, it would never have had lockfiles if not for yarn, and even its current attempt seems half-assed.
Python has the problem of too many package managers, some which are bad, unfortunately (the list of open bugs for Pipenv is especially alarming; I once had to rip it out of a project because the dependency resolution failed after half an hour with a stack overflow). That said, poetry solves the problem well and correctly, IMHO.
Over in the Java world, you're right that dependency locking is a bit rarer. But I also think the situation is not nearly as dire there. BOMs, where lists of compatible dependencies are curated, are relatively common, so that alleviates some of the pain. Plus, there seems to be less churn than in some other ecosystems. Still, it would probably be technically better to use dependency locking.
To be fair the title of the article is "How Go Mitigates Supply Chain Attacks" not how it solves.
And I think it does a good job at that.
For example if you had any JavaScript package that depended on node-ipc in your project, a simple npm install after cloning the project would download code that tries to corrupt files in your disk if the malicious code determined that your IP was from Russia. (before the malicious package was taken down/fixed)
With Go you would have to explicitly bump dependency versions. Simply cloning the project and installing dependencies would not have downloaded the malicious version. And bumping would at the very least appear as a diff in a Pull Request.
And I think it does a good job at that.
For example if you had any JavaScript package that depended on node-ipc in your project, a simple npm install after cloning the project would download code that tries to corrupt files in your disk if the malicious code determined that your IP was from Russia. (before the malicious package was taken down/fixed)
With Go you would have to explicitly bump dependency versions. Simply cloning the project and installing dependencies would not have downloaded the malicious version. And bumping would at the very least appear as a diff in a Pull Request.
Yes, node's attempt to include lockfiles is botched, unfortunately. Not only are there UX issues (it's completely unintuitive that you should be using "npm ci", for example), but the lock file can also get corrupted e.g. during a merge conflict and npm performs no sanity checking on it.
I once had a case where a build was suddenly failing. The reason turned out to be that (for whatever reason) a dev had managed to corrupt the lock file, probably during a merge conflict, in such a way that the entry for package A actually contained the URL for package B. It turns out that npm didn't realise that this was inconsistent (with the package.json, and with the npm registry) and downloaded package B but exported it as package A, making the error incredibly hard to pin down.
I once had a case where a build was suddenly failing. The reason turned out to be that (for whatever reason) a dev had managed to corrupt the lock file, probably during a merge conflict, in such a way that the entry for package A actually contained the URL for package B. It turns out that npm didn't realise that this was inconsistent (with the package.json, and with the npm registry) and downloaded package B but exported it as package A, making the error incredibly hard to pin down.
Doesn't node have lockfiles? Cloning a project and running npm install would install the exact dependencies declared in the lockfile right? To quote the docs[1]:
> The goal of package-lock.json file is to keep track of the exact version of every package that is installed so that a product is 100% reproducible in the same way even if packages are updated by their maintainers.
[1]: https://nodejs.dev/learn/the-package-lock-json-file
> The goal of package-lock.json file is to keep track of the exact version of every package that is installed so that a product is 100% reproducible in the same way even if packages are updated by their maintainers.
[1]: https://nodejs.dev/learn/the-package-lock-json-file
Maven and gradle don't have lockfiles(by default), and have never really had a serious need for them, because dependency declarations generally don't use ranges.
The central repositories don't allow versions to be replaced, and artifacts are all signed with PGP keys of the developers(although most people don't verify these).
I've never really seen the value in dependency ranges, they make builds more complicated, and bring minimal value.
I've never really seen the value in dependency ranges, they make builds more complicated, and bring minimal value.
As far as I know that doesn't solve the problem for transitive dependencies, which can still be resolved to different versions without a lock file.
No it’s the same behaviour for transitive dependencies, if two libraries require different versions of the same transitive dependency, the newer one is chosen.
Deterministic no lock file required.
> if two libraries require different versions of the same transitive dependency, the newer one is chosen.
Unfortunately, this is not how maven works. It picks the version required by the dependency nearest in depth to the project root, breaking ties by first listed in the file. It is deterministic, but it's not what anyone ever wants by default.
(This is called "dependency mediation" if you want to Google it.)
Unfortunately, this is not how maven works. It picks the version required by the dependency nearest in depth to the project root, breaking ties by first listed in the file. It is deterministic, but it's not what anyone ever wants by default.
(This is called "dependency mediation" if you want to Google it.)
OK, but that has a whole host of other issues such as the dependency that relies on the older version potentially breaking with the newer version. I think BOMs get around this issue, hence why they're so common, but then it just means that you trade one issue for the other and that's why you have different solutions. A Ruby project, for example, doesn't need BOMs.
The elephant in the room here is NPM, and I think the obvious problem there is the culture. I have a tiny app I've been playing with using create-react-app. There are over 800 directories in node_modules. That absolutely dwarfs the number of any other language I've used. Even in a medium sized rails app, you likely have some awareness of what every dependency is. It's just impossible with npm.
One thought I've had to "reboot" the npm culture is to somehow curate packages that are proven to have minimal and safe dependencies, and projects can shift to using those. I imagine there has to be some sort of manual review to make that happen.
One thought I've had to "reboot" the npm culture is to somehow curate packages that are proven to have minimal and safe dependencies, and projects can shift to using those. I imagine there has to be some sort of manual review to make that happen.
It's part of the culture of frontend programming.
In the past I've needed to display a timestamp as something like "n weeks ago" (in a mac app). My first instinct was to write a quick function to do the transform. Then I can tweak it and extend it later to fit my app's needs.
However when I asked the web app team at my company to see their code so I could use the same initial set of intervals, it turns out they use a library to do it. The first instinct of a frontend dev seems to be - even for very tiny, single function solutions - download a library.
In the past I've needed to display a timestamp as something like "n weeks ago" (in a mac app). My first instinct was to write a quick function to do the transform. Then I can tweak it and extend it later to fit my app's needs.
However when I asked the web app team at my company to see their code so I could use the same initial set of intervals, it turns out they use a library to do it. The first instinct of a frontend dev seems to be - even for very tiny, single function solutions - download a library.
To be fair, there are problems that look easy at the first sight, but turn out surprisingly difficult -- and working with time is almost always one of those problems.
npm may be an elephant, but why is it in a room talking about go.mod?
Why not drag APT and the Debian unstable repository in as well? i think install-debian-os also has around 800 dependencies. Debian unstable is vulnerable to supply chain attacks by package maintainers, and Debian stable suffers from lack of volunteers backporting and verifying security patches. Yet, if their numbers increase, the small chance increases a cabal of them attacks the supply chain by pushing and vouching for a fake patch that introduces malicious code ...
The elephant in the room is not some package manager unrelated to the topic. The elephant is trust.
GO argues to mitigate attacks by verifying and locking all dependencies yourself. Which, does not scale.
Debian argues with a proof of work, carefully curating what they publish, and who can publish to fast tracks like the security repo. In theory these people could go insane or be hacked in the same way node module devs go insane or get hacked. But curated publishing mitigates some of the risk.
As far as i know there is only one "third party" GO module repository (actually its generic, supporting both go.mod, npm, and many more) that has a multi-stage system of curation. It promises to integrate lots of tools, global cooperation, configurable policies, an AI and a team of specialists to help with curation. It is also proprietary, stupidly expensive and may not fully deliver on those promises.
Why not drag APT and the Debian unstable repository in as well? i think install-debian-os also has around 800 dependencies. Debian unstable is vulnerable to supply chain attacks by package maintainers, and Debian stable suffers from lack of volunteers backporting and verifying security patches. Yet, if their numbers increase, the small chance increases a cabal of them attacks the supply chain by pushing and vouching for a fake patch that introduces malicious code ...
The elephant in the room is not some package manager unrelated to the topic. The elephant is trust.
GO argues to mitigate attacks by verifying and locking all dependencies yourself. Which, does not scale.
Debian argues with a proof of work, carefully curating what they publish, and who can publish to fast tracks like the security repo. In theory these people could go insane or be hacked in the same way node module devs go insane or get hacked. But curated publishing mitigates some of the risk.
As far as i know there is only one "third party" GO module repository (actually its generic, supporting both go.mod, npm, and many more) that has a multi-stage system of curation. It promises to integrate lots of tools, global cooperation, configurable policies, an AI and a team of specialists to help with curation. It is also proprietary, stupidly expensive and may not fully deliver on those promises.
The creator of NodeJS talks about how one of the things he regrets is hard-coupling Node to the NPM registry[1]. I imagine this makes it hard to have curated or trusted third-party registries (although note that it is possible to configure private or third-party registries in Node). This is also one of the problems the creator tries to solve in his new runtime, Deno.
[1]: https://www.youtube.com/watch?v=M3BM9TB-8yA
[1]: https://www.youtube.com/watch?v=M3BM9TB-8yA
I think the NPM organization is completely aware just how dangerous this all is, and is eager to hide it. For example, if you look up an NPM package, it'll list its direct dependencies. But, there's no acknowledgement whatsoever of all the stuff that comes along for the ride.
I'd love to have a well-supported ranking of NPM packages in terms of their dependencies (and their dependencies' dependencies, etc). Knowing the breadth of immediate dependencies, PLUS the depth of the total dependency tree, would give you some inkling of just how much you're taking for granted when you start using a package.
I'd love to have a well-supported ranking of NPM packages in terms of their dependencies (and their dependencies' dependencies, etc). Knowing the breadth of immediate dependencies, PLUS the depth of the total dependency tree, would give you some inkling of just how much you're taking for granted when you start using a package.
I agree that it would be nice for NPM to show the total footprint of a module, especially if that provides some social incentive to reduce the dependency count.
Something this articles glosses over is that some of these approaches, especially the way 'All builds are “locked”' is achieved with minimum version selection, and “A little copying is better than a little dependency” are tradeoffs against an alternative security model, where transitive dependencies are automatically updated to pick up security fixes.
Part of the churn and noise in the Node.js dependency ecosystem actually stems from security-related issues being noted in a low-level module, and the ripple effects caused by that when a bunch of maintainers have to go around bumping lockfiles and versions.
Part of the churn and noise in the Node.js dependency ecosystem actually stems from security-related issues being noted in a low-level module, and the ripple effects caused by that when a bunch of maintainers have to go around bumping lockfiles and versions.
There is a deeper strategy here with go vs. node; having a standard library maintained by professionals.
I would rather build on a common set of libraries secured by people who are paid full-time to maintain them, and maybe have slightly worse ergonomics, than have a community of libraries that come and go and have inconsistent quality.
This standard library approach yields fewer dependencies, fewer changes over time, and better consistency between projects.
I would rather build on a common set of libraries secured by people who are paid full-time to maintain them, and maybe have slightly worse ergonomics, than have a community of libraries that come and go and have inconsistent quality.
This standard library approach yields fewer dependencies, fewer changes over time, and better consistency between projects.
The downside of the standard library approach is that things tend to ossify. While I agree that slower change can be a good thing sometimes, putting things like a HTTP server in the standard library means less experimentation around different ways of doing things, and more difficulty getting performance and other improvements into the hands of language users.
Sure, people can make a third-party module that implements a HTTP server, but the incumbent default that's shipped with the language has an inherent (and often unfair) advantage and a lot of inertia behind it.
I don't really care about the whole "professionals" bit. Sure, I don't want to be relying on something mission-critical to me that's maintained by one person doing it in their spare time. But there is a world of possibilities between that and having a dedicated paid team. Consider, also, that the Go team is only funded so long as Go is important to Google's corporate strategy. Once it isn't, funding will start to dry up, and Go will have to look for a new funding and governance model. That's not necessarily a bad thing, and I'm sure Go would still succeed despite that. But that's kinda my point: this whole "maintained by funded professionals" thing doesn't really matter all that much.
Sure, people can make a third-party module that implements a HTTP server, but the incumbent default that's shipped with the language has an inherent (and often unfair) advantage and a lot of inertia behind it.
I don't really care about the whole "professionals" bit. Sure, I don't want to be relying on something mission-critical to me that's maintained by one person doing it in their spare time. But there is a world of possibilities between that and having a dedicated paid team. Consider, also, that the Go team is only funded so long as Go is important to Google's corporate strategy. Once it isn't, funding will start to dry up, and Go will have to look for a new funding and governance model. That's not necessarily a bad thing, and I'm sure Go would still succeed despite that. But that's kinda my point: this whole "maintained by funded professionals" thing doesn't really matter all that much.
Isn't Node API an equivalent of go standard library?
Yes but it's super barebones. Its successor, Ryan Dahl's second attempt at JS runtime, Deno, has a much fuller standard library (inspired by Go).
I wish we'd stop trying to make broken languages work. This feels like hill-climbing into the strangest local optimum possible. JS is not the best example of an interpreted language. Wouldn't it be better to put Python in the browser than to put JS on the server? Can't wait for WASM to be a first-rate citizen on the web so we don't have to deal with this anymore.
I think you would be surprised to learn that more developers love TypeScript than Python these days, according to one popular survey.[0]
All of this is subjective, of course. WASM isn't going to make Python the language of choice for browsers any time soon.
[0]: https://insights.stackoverflow.com/survey/2021#section-most-...
All of this is subjective, of course. WASM isn't going to make Python the language of choice for browsers any time soon.
[0]: https://insights.stackoverflow.com/survey/2021#section-most-...
I don't think the comparison is entirely fair since one of the main attractions of TS is that it runs in the browser. Python can unfortunately not fill the same role right now. So I'd keep that in mind while looking at that ranking. But yes, I see many people like it. Maybe I'm missing something, but it's still too JavaScript-y for me.
> Wouldn't it be better to put Python in the browser than to put JS on the server?
I think that's a categorical "no", because Python isn't an objectively better language than JavaScript. I'm saying this as a Python developer since v1.5 (>20 years).
Subjective opinions are a different matter.
I think that's a categorical "no", because Python isn't an objectively better language than JavaScript. I'm saying this as a Python developer since v1.5 (>20 years).
Subjective opinions are a different matter.
Yes Node.js ships with what is effectively a very thin standard library for some low level things like interacting with the file system, the process model, some security features like TLS.
The Fetch API, supported by browsers for over 5 years now, is only now making it into the official Node API https://blog.logrocket.com/fetch-api-node-js/
Node.js had `http` in its standard library for a long time though.
so, cathedral vs bazaar
> tradeoffs against an alternative security model, where transitive dependencies are automatically updated to pick up security fixes.
One thing to keep in mind is that Go doesn't stop you from updating.
For example, its common to do 'go get -u ./...' or 'go get -u=patch ./...' from your project root to update all of your direct and indirect dependencies.
The built-in tooling & language server give you nudges, and if desired it can be automated via things like dependabot or otherwise.
In practice, it means it is often a slightly slower cadence for typical projects in the Go ecosystem compared to say the Node.js ecosystem, but the upgrades still happen. That slightly slower pace I think has worked out so far, and was a conscious choice[1]:
> Many developers recoil at the idea that adding the latest B would not automatically also add the latest C, but if C was just released, there's no guarantee it works in this build. The more conservative position is to avoid using it until the user asks. For comparison, the Go 1.9 go command does not automatically start using Go 1.10 the day Go 1.10 is released. Instead, users are expected to update on their own schedule, so that they can control when they take on the risk of things breaking.
[1] https://go.googlesource.com/proposal/+/master/design/24301-v...
One thing to keep in mind is that Go doesn't stop you from updating.
For example, its common to do 'go get -u ./...' or 'go get -u=patch ./...' from your project root to update all of your direct and indirect dependencies.
The built-in tooling & language server give you nudges, and if desired it can be automated via things like dependabot or otherwise.
In practice, it means it is often a slightly slower cadence for typical projects in the Go ecosystem compared to say the Node.js ecosystem, but the upgrades still happen. That slightly slower pace I think has worked out so far, and was a conscious choice[1]:
> Many developers recoil at the idea that adding the latest B would not automatically also add the latest C, but if C was just released, there's no guarantee it works in this build. The more conservative position is to avoid using it until the user asks. For comparison, the Go 1.9 go command does not automatically start using Go 1.10 the day Go 1.10 is released. Instead, users are expected to update on their own schedule, so that they can control when they take on the risk of things breaking.
[1] https://go.googlesource.com/proposal/+/master/design/24301-v...
> transitive dependencies are automatically updated to pick up security fixes
Does Node do this? That seems like an awful idea. People should be manually updating dependencies, never automatically. Stuff like dependabot need to die.
Does Node do this? That seems like an awful idea. People should be manually updating dependencies, never automatically. Stuff like dependabot need to die.
Dependabot gives you an easy way to review every single commit that went into a dependency update before you merge it.
Dependabot is by far the most convenient way that I’ve seen to actually check that your dependency updates are not overtly malicious.
It's not some tool that just removes your lockfiles behind your back, as you seem to be implying.
Dependabot is by far the most convenient way that I’ve seen to actually check that your dependency updates are not overtly malicious.
It's not some tool that just removes your lockfiles behind your back, as you seem to be implying.
I'm annoyed by the false dichotomy that colors most discussions around package management that there are only two solutions to publishing software packages: 1. a carefully curated professionally maintained standard library, 2. the complete wild west where anything goes. It's not really "false" because this is the reality of how package managers are designed today, but it's false in the sense that it doesn't have to be this way.
You can see this tension in virtually every discussion, users resisting using packages that aren't published in the standard library for fear of attacks and poor quality, and maintainers that resist publishing in the standard library for fear of changing requirements and the appearance of better designs. Sure there are admissible entitlement / responsibility arguments against these respective positions, but that's mostly a distraction because both have a valid point.
The problem is that there's no space for intermediate solutions. We need packaging tools to aggregate and publish groups of packages that relate to a particular domain, and organizational tools to ensure quality and continuity of these package groups over time. This mitigates users' fears and reduces their cognitive load by curating the solution space, and it mitigates maintainers fears of ossification and backcompat hell by enabling them to create new package groups.
I'm saying there's an entire dimension of valid tradeoffs in this space, but the current design trend of package managers force us into one extreme or the other.
You can see this tension in virtually every discussion, users resisting using packages that aren't published in the standard library for fear of attacks and poor quality, and maintainers that resist publishing in the standard library for fear of changing requirements and the appearance of better designs. Sure there are admissible entitlement / responsibility arguments against these respective positions, but that's mostly a distraction because both have a valid point.
The problem is that there's no space for intermediate solutions. We need packaging tools to aggregate and publish groups of packages that relate to a particular domain, and organizational tools to ensure quality and continuity of these package groups over time. This mitigates users' fears and reduces their cognitive load by curating the solution space, and it mitigates maintainers fears of ossification and backcompat hell by enabling them to create new package groups.
I'm saying there's an entire dimension of valid tradeoffs in this space, but the current design trend of package managers force us into one extreme or the other.
I'm unclear from reading your comment if you know this or not... but, what the Go team is describing is an intermediate solution. Not the exact one you describe, but it is intermediate. There is no particular requirement to get into the Go package ecosystem, no gatekeepers, it's all namespaced by the URLs you store your source code at, but between what the proxies do and the way the version requirements were specified, you are also not simply naked to every update someone somewhere pushes.
Please note I'm not claiming it's perfect or that you'll like every aspect of it, I'm just saying it is an intermediate solution between the two extremes.
Please note I'm not claiming it's perfect or that you'll like every aspect of it, I'm just saying it is an intermediate solution between the two extremes.
The original solution is the intermediate solution. You can build an entire useful userspace around nothing but libc and an ssl lib (take your pick between OpenSSL or GnuTLS usually). That is effectively what busybox is. Need to do some heavyweight math in Fortran? BLAS and LAPACK probably have everything you need. You can get really far with a C++ application using nothing but Boost.
For whatever reason, newer language ecosystems migrated away from that in the direction of increasingly smaller libraries until npm practically became a parody of it.
There is no reason you can't have lots and lots of useful functionality packed into a few large, well-maintained, well-packaged, well-vetted and trusted libraries, but you need trustworthy organizations willing to that maintaining and vetting. Historically, that seemed to largely be universities and research labs, where the funding and incentives are a lot different from the weekend warriors and solo devs that dominate open source landscapes today. Interestingly, I think library projects that still have large organizations behind them keep with the larger old-school ethos. Look at the world of ML and scientific computing. NumPy and SciPy are still huge libraries. Same with PyTorch and Tensorflow. QuantLib is an interesting example because it actually doesn't have a single large organization behind it. A bunch of Quants just got tired of doing the same things from scratch over and over and decided to aggregate their work for their common good. But it was 22 years ago, so maybe it was still just different back then and the trend toward small libraries hadn't kicked in yet.
For whatever reason, newer language ecosystems migrated away from that in the direction of increasingly smaller libraries until npm practically became a parody of it.
There is no reason you can't have lots and lots of useful functionality packed into a few large, well-maintained, well-packaged, well-vetted and trusted libraries, but you need trustworthy organizations willing to that maintaining and vetting. Historically, that seemed to largely be universities and research labs, where the funding and incentives are a lot different from the weekend warriors and solo devs that dominate open source landscapes today. Interestingly, I think library projects that still have large organizations behind them keep with the larger old-school ethos. Look at the world of ML and scientific computing. NumPy and SciPy are still huge libraries. Same with PyTorch and Tensorflow. QuantLib is an interesting example because it actually doesn't have a single large organization behind it. A bunch of Quants just got tired of doing the same things from scratch over and over and decided to aggregate their work for their common good. But it was 22 years ago, so maybe it was still just different back then and the trend toward small libraries hadn't kicked in yet.
I don't think your suggestion works for Web Development.
Imagine an application that talks to a Postgres DB, a Redis cache, several AWS services (e.g. S3, Lex) and also has to be able to parse excel documents. I've worked on applications like that. That's a whole lot of libraries you'll need to include. But I don't think you should include all of them in some "standard library", most people are not gonna need most of these dependencies.
I agree that left-pad is ridiculous (over in the JVM space, there is apache-commons, which also provides e.g. left-pad but of course it's not the only functionality that it provides), but using only 2-3 libraries isn't realistic either.
Imagine an application that talks to a Postgres DB, a Redis cache, several AWS services (e.g. S3, Lex) and also has to be able to parse excel documents. I've worked on applications like that. That's a whole lot of libraries you'll need to include. But I don't think you should include all of them in some "standard library", most people are not gonna need most of these dependencies.
I agree that left-pad is ridiculous (over in the JVM space, there is apache-commons, which also provides e.g. left-pad but of course it's not the only functionality that it provides), but using only 2-3 libraries isn't realistic either.
This is kind of interesting. In the Linux world, you have package maintainers who (among other things) vet and vouch for the quality of the packages they maintain. I think there are similar things in the Docker ecosystem these days (since Docker really did/does seem to be the wild west).
It could be interesting if there was a similar concept for Go (and/or other ecosystems), except that instead of actually packaging the packages into artifacts (especially with the licensing headache that entails), it could be essentially a registry of verified package versions. So the "maintainers" in this sense are just validating the dependencies and maintaining a list of the approved dependencies (including their versions/checksums) and then automated tooling could be used by consumers ("consumer" here may or may not imply payment depending on whether this hypothetical venture is open or closed) to identify unverified dependencies in the consumer's project.
I'm sure someone has thought of this already--link me to relevant projects if you know about any.
It could be interesting if there was a similar concept for Go (and/or other ecosystems), except that instead of actually packaging the packages into artifacts (especially with the licensing headache that entails), it could be essentially a registry of verified package versions. So the "maintainers" in this sense are just validating the dependencies and maintaining a list of the approved dependencies (including their versions/checksums) and then automated tooling could be used by consumers ("consumer" here may or may not imply payment depending on whether this hypothetical venture is open or closed) to identify unverified dependencies in the consumer's project.
I'm sure someone has thought of this already--link me to relevant projects if you know about any.
Seems like this would be equivalent to running your own module proxy / sumdb with a whitelist, which e.g. Athens can do.
> We need packaging tools to aggregate and publish groups of packages that relate to a particular domain, and organizational tools to ensure quality and continuity of these package groups over time
Whats stopping you? golang.org/x is kind of like that. Theres nobody stopping you from aggregating packages under foo.bar domain and build a reputation for high quality.
Whats stopping you? golang.org/x is kind of like that. Theres nobody stopping you from aggregating packages under foo.bar domain and build a reputation for high quality.
Good question! The thing that the standard library has that I don't is version aggregation. The problem is not publishing under a single domain, the problem is publishing under a single version. Publishing a bunch of packages that I claim are high quality doesn't help users decide which versions to use, they still have to make this decision on a package-by-package basis. Note that I may be in the middle of a big redesign and some individually-complete packages use the new design but others don't, so you can't just say use the latest of each because they don't all work together yet. In this case I still want to publish each package individually for people who do want fine-grained control, but I wouldn't publish a new version of my "package group" with these versions because they're not integrated yet. My point is that aggregations of verified-interoperable packages is a separate problem domain and existing tools don't suffice to solve it.
You could sort of do that using a Go module that points to all your other modules. Then anyone who depends on that will get the versions you specify (at a minimum).
But a problem is that they would also download all the modules you point at, whether they use them or not.
To fix that, the package system would need a "soft dependency" where, if a module exists, it must be at least the version indicated.
But a problem is that they would also download all the modules you point at, whether they use them or not.
To fix that, the package system would need a "soft dependency" where, if a module exists, it must be at least the version indicated.
Good idea with "soft dependencies". You'd also need to make the go.mod RequireSpec version argument [1] optional so it can be overridden by the module group.
I also noticed Workspaces [2] which I hadn't seen before. They look interesting, but appear to exist for a different purpose. Maybe workspaces with a bunch of replace directives & some cli tooling could emulate a system like what is described here.
[2]: https://go.dev/ref/mod#workspaces
require example.com/my-module-group 1.2.3
require example.com/some/module // selects minimum version specified by my-module-group
I also noticed Workspaces [2] which I hadn't seen before. They look interesting, but appear to exist for a different purpose. Maybe workspaces with a bunch of replace directives & some cli tooling could emulate a system like what is described here.
[2]: https://go.dev/ref/mod#workspaces
[deleted]
I suspect licensing may stop you from doing this, but I'm not an expert. See my sibling comment for an alternative: https://news.ycombinator.com/item?id=30871181
The article, and the comments praising this approach, don’t do a great job of explaining how any of this is substantively different from running the likes of yarn install --frozen-lockfile, or cargo build --frozen.
Here’s the thing: You can argue about being secure by default and encouraging better CI practices. I’d fully agree it isn’t great that one has to know a somewhat obscure flag to get a secure CI build in those environments.
But claiming in what I perceive to be in parts a somewhat grandiose tone to have reinvented the wheel, when you’re just describing a standard approach, can make you sound uninformed.
Here’s the thing: You can argue about being secure by default and encouraging better CI practices. I’d fully agree it isn’t great that one has to know a somewhat obscure flag to get a secure CI build in those environments.
But claiming in what I perceive to be in parts a somewhat grandiose tone to have reinvented the wheel, when you’re just describing a standard approach, can make you sound uninformed.
the difference is in the part where you don't have a lock file yet (new dependency, upgrades), and need to choose versions for those.
I think at most there's pride in their own solution, which is not something anyone should object to - it's pretty good. It's better than some other systems, but no point in being specific.
Not doing specific comparisons is likely a deliberate strategy, since it means the blog post is less likely to go out of date, and it avoids controversy if they get something wrong.
Comparisons will need to be written by people familiar with both systems, and they're likely to go out of date quickly.
Not doing specific comparisons is likely a deliberate strategy, since it means the blog post is less likely to go out of date, and it avoids controversy if they get something wrong.
Comparisons will need to be written by people familiar with both systems, and they're likely to go out of date quickly.
> Unlike most other package managers files, Go modules don’t have a separate list of constraints and a lock file pinning specific versions.
The weird thing about the Go devs is there is always that little bit of elitism under the surface that I detect in their writing (whether it be colors in the playground, the GC, etc). I spent years writing Go and have now moved to Rust. What I find odd is the Rust team has done (IMO) one of the greater achievements in PL history and yet they seem to not have this elitism thing going on (or maybe I just haven't noticed). Go on the other hand, IMO, made some "interesting" language choices (like keeping null) and they seem to want to be celebrated for it and claim their achievements as new and novel.
EDIT: To clarify, I'm talking about the core Go devs - those that work on stdlib and the compiler
The weird thing about the Go devs is there is always that little bit of elitism under the surface that I detect in their writing (whether it be colors in the playground, the GC, etc). I spent years writing Go and have now moved to Rust. What I find odd is the Rust team has done (IMO) one of the greater achievements in PL history and yet they seem to not have this elitism thing going on (or maybe I just haven't noticed). Go on the other hand, IMO, made some "interesting" language choices (like keeping null) and they seem to want to be celebrated for it and claim their achievements as new and novel.
EDIT: To clarify, I'm talking about the core Go devs - those that work on stdlib and the compiler
I don't see that elitism in this article. Supply chain attacks are a hot topic right now, so it makes sense for them to make a statement about where the language stands with them. They make compelling points, and they're not calling out specific language or package manager as a comparison.
This is a pretty genuinely confounding response, and I mean that with absolutely no offense intended. There is a tremendous amount of fighting between devs who prefer Go and Rust, and a tremendous amount of elitism as well, truly from both perspectives. Rust gained a reputation for elitism long before Go did; “Rust Evangelism Strike Force” was never meant to be pejorative, and “Rewrite it in Rust” was never meant to be a joke, but it became one anyways. It’s not hard to see why; Rust is genuinely novel in a way that few other programming languages are. It feels the most like the “future.”
But I still like Go a lot. I like Go because of how easy and simple it feels. There is definitely elitism over simplicity, but the elitism I’ve seen and even received from Rust and C++ programmers (…despite that I have been coding C++ forever and do have a few Rust projects as well…) has been pretty much the opposite: Go is too stupid and simple; real programmers need absurdly complex metaprogramming to make basic CLI tools or what have you. Now for what it’s worth, that has cooled down in many regards, and also, Rust is amazing and there’s nothing wrong with advanced metaprogramming. (It’s just another set of tradeoffs, after all. Unquestionably has its benefits.)
However, whereas people who have hated on Rust have often come to see it for what it is (an immensely cool, novel programming language,) Go has received the opposite treatment. People soured on it. Now everyone seems sure the GC latency (which of course is just about state of the art) is simply too much for most use cases. It’s seen adoption in all sorts of places and even been competitive with Rust software in performance, but it is commonly discussed as if Go is inherently obsolete because Rust is a better option in every way that matters. Bringing up Go in certain places often subjects you to ridicule, and I’m not joking. The memory ballast is a favorite among detractors to prove that the language is stupid and bad for production environments.
So when people do try to tout the benefits of Go, it’s routinely discredited and downplayed for some reason. It’s a nice language to use with a stellar standard library, nice tooling, and pretty good runtime performance.
This article doesn’t mention Rust (that I noticed) and Go is still being measured up to Rust in the comments. They both trade blows in different categories, but I truly believe that the fact that Go lacks the novelty of Rust with its borrow checker and language design has caused a lot of people to view it very negatively, and I think that is sad. People loved C for a lot of what it didn’t have. Go is a lot different than C, but for me, the sentiment is very much the same.
I think people see what they want to see. I like Go and Rust, but I find myself going back to Go for various reasons and it feels like every year it leads more and more people to ask for justification that they wouldn’t for other languages. It’s a little tiring.
But I still like Go a lot. I like Go because of how easy and simple it feels. There is definitely elitism over simplicity, but the elitism I’ve seen and even received from Rust and C++ programmers (…despite that I have been coding C++ forever and do have a few Rust projects as well…) has been pretty much the opposite: Go is too stupid and simple; real programmers need absurdly complex metaprogramming to make basic CLI tools or what have you. Now for what it’s worth, that has cooled down in many regards, and also, Rust is amazing and there’s nothing wrong with advanced metaprogramming. (It’s just another set of tradeoffs, after all. Unquestionably has its benefits.)
However, whereas people who have hated on Rust have often come to see it for what it is (an immensely cool, novel programming language,) Go has received the opposite treatment. People soured on it. Now everyone seems sure the GC latency (which of course is just about state of the art) is simply too much for most use cases. It’s seen adoption in all sorts of places and even been competitive with Rust software in performance, but it is commonly discussed as if Go is inherently obsolete because Rust is a better option in every way that matters. Bringing up Go in certain places often subjects you to ridicule, and I’m not joking. The memory ballast is a favorite among detractors to prove that the language is stupid and bad for production environments.
So when people do try to tout the benefits of Go, it’s routinely discredited and downplayed for some reason. It’s a nice language to use with a stellar standard library, nice tooling, and pretty good runtime performance.
This article doesn’t mention Rust (that I noticed) and Go is still being measured up to Rust in the comments. They both trade blows in different categories, but I truly believe that the fact that Go lacks the novelty of Rust with its borrow checker and language design has caused a lot of people to view it very negatively, and I think that is sad. People loved C for a lot of what it didn’t have. Go is a lot different than C, but for me, the sentiment is very much the same.
I think people see what they want to see. I like Go and Rust, but I find myself going back to Go for various reasons and it feels like every year it leads more and more people to ask for justification that they wouldn’t for other languages. It’s a little tiring.
Without commenting on anything else here:
> “Rust Evangelism Strike Force” was never meant to be pejorative, and “Rewrite it in Rust” was never meant to be a joke
Maybe it's just been such a long time, but my recollection was exactly that: both of these terms were invented by outsiders intending to denigrate the Rust community, and became jokes inside the community as a means of recuperating them.
Okay, I will mention one other thing here:
> This article doesn’t mention Rust (that I noticed) and Go is still being measured up to Rust in the comments.
I agree with this in 99.999% of threads, this happens all the time, and probably shouldn't. However, in this thread in particular with the way that the Go package management story developed, including all of the drama there, I don't think it's surprising that Rust/Cargo get mentioned in comparison.
> “Rust Evangelism Strike Force” was never meant to be pejorative, and “Rewrite it in Rust” was never meant to be a joke
Maybe it's just been such a long time, but my recollection was exactly that: both of these terms were invented by outsiders intending to denigrate the Rust community, and became jokes inside the community as a means of recuperating them.
Okay, I will mention one other thing here:
> This article doesn’t mention Rust (that I noticed) and Go is still being measured up to Rust in the comments.
I agree with this in 99.999% of threads, this happens all the time, and probably shouldn't. However, in this thread in particular with the way that the Go package management story developed, including all of the drama there, I don't think it's surprising that Rust/Cargo get mentioned in comparison.
I won’t comment on Rust Evangelism Strike Force too much; it seemed to be unironically used as a term of endearment at some point, but that could’ve been after its use as a pejorative. At this point, I can’t remember, and frankly, the world is probably better off forgetting.
I understand. In the earlier days of Go package management, it was pretty common for folks to compare it to Cargo. In retrospect, this was probably bad, but it did serve to highlight some pretty damning issues with Go at the time. But I feel they addressed the shortcomings significantly with Go modules, and now it has become much more a matter of taste.
I enjoy Go’s idea of trying to make source control the only source of truth, but I don’t think it’s as well-received as the more tried-and-true approach of Cargo and other centralized package repositories. I suppose time will tell.
I understand. In the earlier days of Go package management, it was pretty common for folks to compare it to Cargo. In retrospect, this was probably bad, but it did serve to highlight some pretty damning issues with Go at the time. But I feel they addressed the shortcomings significantly with Go modules, and now it has become much more a matter of taste.
I enjoy Go’s idea of trying to make source control the only source of truth, but I don’t think it’s as well-received as the more tried-and-true approach of Cargo and other centralized package repositories. I suppose time will tell.
To be clear, I have no issue with anyone who prefers Go. I was speaking about the core devs in particular. I would agree that the users of the language definitely go back and forth "trading blows".
Interesting and apologies for misunderstanding. I didn’t read the article as being elitist, though I can see how it reads as self-congratulatory to a degree. Maybe the matter-of-fact way that Go’s developers state its advantages comes off poorly compared to, for example, coming from the standpoint of trying to explain how they got to their current design based on the challenges. Personally, I find articles like this easier to read because they tend to be more terse when written this way versus some other approaches that are perhaps more humble.
Lovely comment. I share your sentiment. Go is truly hated on hacker news these days (and, if you're brave enough to venture there, reviled on r/programming).
The elitism stems not from solving cutting-edge PL problems, but from making technical decisions informed by engineering experience.
How much of that is justified is up to you, of course.
How much of that is justified is up to you, of course.
Go is a distillation of many decades of software engineering experience. The people behind Go (e.g., Russ Cox) have learned from history.
The peanut gallery loves to complain about superficial aspects of Go. Typically these are people with little or no actual experience using the language and tools. They fixate on imagined problems that don't matter in practice.
But anyone who has used Go full-time for a few years is likely to deeply respect and appreciate it.
The peanut gallery loves to complain about superficial aspects of Go. Typically these are people with little or no actual experience using the language and tools. They fixate on imagined problems that don't matter in practice.
But anyone who has used Go full-time for a few years is likely to deeply respect and appreciate it.
I've been using Go for a couple of years now and it's funny: I do not love Go, I just work effectively in it. I am not passionate about it, but I recommend it for absolutely all appropriate use-cases.
It doesnt go for intellectual satisfaction, it goes for getting shit done. You have to respect it for being so radically bland.
It doesnt go for intellectual satisfaction, it goes for getting shit done. You have to respect it for being so radically bland.
I haven't used Go full-time, but I have used it on and off for more than a few years. There are certainly things I respect and appreciate about it, but there are also a lot of things that annoy me about it. Some that you might consider "superficial", I consider important. If I'm going to be spending all day in a language, I want the ergonomics of the language itself to work with me, not against me, and Go often does not fit the bill there (for me -- others' opinions are free to differ).
I don't find most of this article to be all that persuasive. Rust, for example, has a separate lock file, which the article derides. That doesn't really make sense, as lock files are also checked into source control, so you get the same benefit that Go touts of go.mod. My threat model doesn't consider having a separate module/package repository to be much of a risk, so I don't care about that point all that much. Admittedly, having source control be the source of truth is just simpler, which is good, but it also means that module publishers can pull versions (or the entire module) for arbitrary, selfish reasons, and then the community is left with a lot of difficulty (there's also a big problem if someone wants to move their code from GitHub to GitLab or something like that). Centralized module repositories can remove this problem if they choose to. The Go Module Mirror appears to be a hack that tacitly admits this problem.
I did find the "a little copying..." bit to be interesting, and I agree with it. With Rust, pulling in a single dependency tends to pull in many tens of transitive dependencies, which I don't like.
I don't find most of this article to be all that persuasive. Rust, for example, has a separate lock file, which the article derides. That doesn't really make sense, as lock files are also checked into source control, so you get the same benefit that Go touts of go.mod. My threat model doesn't consider having a separate module/package repository to be much of a risk, so I don't care about that point all that much. Admittedly, having source control be the source of truth is just simpler, which is good, but it also means that module publishers can pull versions (or the entire module) for arbitrary, selfish reasons, and then the community is left with a lot of difficulty (there's also a big problem if someone wants to move their code from GitHub to GitLab or something like that). Centralized module repositories can remove this problem if they choose to. The Go Module Mirror appears to be a hack that tacitly admits this problem.
I did find the "a little copying..." bit to be interesting, and I agree with it. With Rust, pulling in a single dependency tends to pull in many tens of transitive dependencies, which I don't like.
"but it also means that module publishers can pull versions (or the entire module) for arbitrary, selfish reasons, and then the community is left with a lot of difficulty"
By default go get will download the source code into the pgk/mod folder. So if a module is pulled by the author, you can just use your copy of the source to fork it.
By default go get will download the source code into the pgk/mod folder. So if a module is pulled by the author, you can just use your copy of the source to fork it.
That's fine if you've already downloaded it, but doesn't help for people trying to pull it for the first time, either as a direct or transitive dependency.
that's what the proxy infrastructure is for: retracted versions won't be selected by default if you're adding it as a new direct dependency, but if you request a version directly (because you need it as an indirect dependency, you have a fresh machine, etc), it will always return the cached code.
For a decade+, people complained that go lacks generics. Would you say those were all people with no experience fixated on an imaginary problem that doesn't matter, or were the complaints valid?
I would say that it is legitimately annoying to have to copy and paste data structures, but mostly doesn't matter, and the intensity of the complaining does not match the intensity of the problem.
People mostly just dont like the expertise of the go devs.
People mostly just dont like the expertise of the go devs.
Some of those "superficial aspects" are really annoying.
For example, Go forced version tags to have a `v` prefix in git repos for their dependency system, which broke a whole host of CI tools that expected plain numeric values for release versions. There's a outsized amount of Go-specific special casing for this one seemingly arbitrary decision in multi-language CI systems.
For example, Go forced version tags to have a `v` prefix in git repos for their dependency system, which broke a whole host of CI tools that expected plain numeric values for release versions. There's a outsized amount of Go-specific special casing for this one seemingly arbitrary decision in multi-language CI systems.
> Go forced version tags to have a `v` prefix in git repos for their dependency system
"Forced" is a bit strong - you can pin any ref, the vX tags just also have some default semver-ish treatment.
> a whole host of CI tools that expected plain numeric values for release versions
Like what? Pure numeric tags are also ambiguous with git refs; this can be worked around by careful arguments, but it means most tooling was already broken when dealing with such things.
If you were one of the people using "release-X.Y.Z" or "rXYZ" I feel for you though.
> Go-specific special casing
'v' tags have been idiomatic for ages. Semver 1.0 went so far as to mandate it in 2010, though that was taken out in 2.0. https://semver.org/spec/v1.0.0.html#tagging-specification-se...
"Forced" is a bit strong - you can pin any ref, the vX tags just also have some default semver-ish treatment.
> a whole host of CI tools that expected plain numeric values for release versions
Like what? Pure numeric tags are also ambiguous with git refs; this can be worked around by careful arguments, but it means most tooling was already broken when dealing with such things.
If you were one of the people using "release-X.Y.Z" or "rXYZ" I feel for you though.
> Go-specific special casing
'v' tags have been idiomatic for ages. Semver 1.0 went so far as to mandate it in 2010, though that was taken out in 2.0. https://semver.org/spec/v1.0.0.html#tagging-specification-se...
I love Go as well, but I don't think being the first to kick up a flame war will do anything for us.
Couldn't disagree more (I spent years writing Go, and now Rust - will NEVER go back)
I couldn't have said it better.
Other ecosystems decided to use lots of little dependencies and I can't imagine a worse engineering decision. This way lies madness.
Other ecosystems decided to use lots of little dependencies and I can't imagine a worse engineering decision. This way lies madness.
A few years of Golang under my belt and I still hate it. Russ Cox and co. seem incredibly arrogant to me in that they can ignore decades of PL research only to reinvent a bizarre way of achieving what other languages do in a more standard way (package management, error handling), or just adopt that standard super late (generics).
Go has some great qualities and you can make great software with it no doubt. But I find the development of the language frustrating to witness, not inspiring.
Go has some great qualities and you can make great software with it no doubt. But I find the development of the language frustrating to witness, not inspiring.
Who is doing "PL research" in package management?
What is your favorite PL?
C# probably. Though I enjoy Typescript and Python as well.
C# has generics, exceptions, is the birthplace of async/await, has LINQ, an unmatched standard library, a great build system and package management system, is open source, cross platform, fast..
It doesn't compile to a single native binary unlike golang which is a bummer. But 95% of my software ships as a container so this isn't too big of deal for me. The MS-provided base images are really good too.
C# has generics, exceptions, is the birthplace of async/await, has LINQ, an unmatched standard library, a great build system and package management system, is open source, cross platform, fast..
It doesn't compile to a single native binary unlike golang which is a bummer. But 95% of my software ships as a container so this isn't too big of deal for me. The MS-provided base images are really good too.
laerus(1)
Dependencies being immutable and identified by hash was such an obvious thing 20 years ago. The problem was fitting that into the flow of these crappy UN*X based build systems where to do any mundane task you need to fiddle with files and encodings and semi documented folder heirarchies and use a CLI tool to change other stuff that is too cumbersome to encode directly into files / text. The most obvious concrete instance of this being that it's cumbersome to import a dependency by hash (in Java for example, you really want hashes everywhere instead of reverse TLDs) but obviously better tools (such as a structural code editor) solves this perfectly. It's also annoying every time a company takes a nano step asymptotically towards these proper solutions and spams their crap and everyone buys it, but that's to be expected when your field is broken industrially and academically and even in hobbiest communities.
I feel like much of the current software complexity is mostly caused by the pain that is compiling C dependencies and the outdated Unix build and configuration tools.
Ah, the old “we’ve already solved this 40 years ago”. Lambdas, immutable-by-default, async/await, etc.
Having dependencies not change is literally trivial. Just link to them by hash. Async/await is a pragmatist garbage hack, and does not fit in the archetype you are trying to name.
There are many gems in its design and implementation in addition to those mentioned in OP.
https://verdverm.com/go-mods/
https://verdverm.com/go-mods/
There is a spelling error:
but also for security resaons. << reasons not resaons
but also for security resaons. << reasons not resaons
Off topic but: Since I assumed this was about physical supply chain attacks (where someone nefarious will either intercept your package to install custom firmware etc. or even change the physical device in some way) - does anyone know where I could find a good guide on mitigating such attacks?
Worth noting is that, while go build doesn't run arbitrary code, go generate does:
package evil
import "fmt"
// the echo is intentional, in case someone actually tries this for some reason
//go:generate echo rm -rf /
func PretendGood() {
fmt.Println("I am good")
}
One downside of such pinning of versions is that if one of your transitive dependencies has a security vulnerability affecting your package, your package will remain affected until you update.
As I understand it dependencies-of-dependencies are fetched from the config provided by the library that includes the dependency, in the case of a transitive dependency, everyone between you and the vulnerable package needs to update, and they need to do so in order (i.e. if you depend on A, A depends on B, B depends on C, and C fixes a vulnerability, then first B, then A have to update in order for you to pull in the change).
As I understand it dependencies-of-dependencies are fetched from the config provided by the library that includes the dependency, in the case of a transitive dependency, everyone between you and the vulnerable package needs to update, and they need to do so in order (i.e. if you depend on A, A depends on B, B depends on C, and C fixes a vulnerability, then first B, then A have to update in order for you to pull in the change).
You can force newer transient dependencies: https://stackoverflow.com/questions/70100325/force-a-transit...
I feel like everyone is beating around the bush here: NPM is a garbage fire, from the interface to the tooling implementation to the theory to the governance. We can talk at each other back and forth about theoretical benefits, and "friction" vs. "usability" or whatever, but NPM has been and continues to be an unmitigated security disaster. The module proxy could have a package takeover a month for the next three years and still not even come close to the ridiculous shit that has happened on NPM.
Keep it simple, just don't use dependencies.
One problem: unless you have an endless amount of time, money, patience, and...did I mention patience? that just isn't a viable approach for anything but the smallest of hobby projects
Build things by KISS principle.
Dependencies will complicate system you are building. ( with things like supply chain attacks, bloat, liscensing etc...)
Dependencies will complicate system you are building. ( with things like supply chain attacks, bloat, liscensing etc...)
So we just don't build GUI applications unless we understand all of the nuances of layout, text rendering, graphics programming, etc sufficient to implement it ourselves (and without the bugs that even domain experts have introduced but which have been found and fixed over time in libraries)? Or maybe we just say "fuck users who speak languages that aren't easily expressed in ASCII"?
There's a reason libraries exist. It's not like they were the default state of computing and no one has tried to write applications without them. On the contrary, we tried to build applications by writing everything ourselves, but that doesn't survive encounters with the real world. And "well if you can't do it yourself, you don't need it" (which may or may not be your argument, I genuinely can't tell) is just technologically regressive ideology.
There's a reason libraries exist. It's not like they were the default state of computing and no one has tried to write applications without them. On the contrary, we tried to build applications by writing everything ourselves, but that doesn't survive encounters with the real world. And "well if you can't do it yourself, you don't need it" (which may or may not be your argument, I genuinely can't tell) is just technologically regressive ideology.
Software programmer usually need to solve specific task, not worlds problems. If you go too generalized you will need 10x, 100x or 1000x more amount of time and code.
Look at Big tech, this is what they are doing, they employ thousands workers that write millions of lines of code every day, only to make it work for every case in a world. And still can't compete with specialized solution.
Look at Big tech, this is what they are doing, they employ thousands workers that write millions of lines of code every day, only to make it work for every case in a world. And still can't compete with specialized solution.
> Software programmer usually need to solve specific task, not worlds problems. If you go too generalized you will need 10x, 100x or 1000x more amount of time and code.
Yes, I accept this is true, but your earlier claim was much more specific: that using dependencies at all makes things worse. I gave you a specific example (GUI libraries) but you completely ignored it. How does your 0-dependencies theory survive an encounter with that basic example?
Yes, I accept this is true, but your earlier claim was much more specific: that using dependencies at all makes things worse. I gave you a specific example (GUI libraries) but you completely ignored it. How does your 0-dependencies theory survive an encounter with that basic example?
It's usually needed only minor part of it. You dont need to code whole GUI library. Only what you need to solve specific task.
No, but if your GUI involves displaying text in multiple languages (i.e., virtually everything), you now need to become an expert in text rendering (which implies a significant breadth of knowledge in linguistics, graphics programming, constraint solving systems [for things like word wrapping], etc).
Except KISS is subjective, as are most things. Please don't take this the wrong way, but anyone who touts any one, true "religion" in sw development I have learned to take with 6 grains of salt. There is no one magic way or solution. Everything is trade offs and I've learned this over the last 30 years through trying every new magic elixir that was going to "save us all".
That said, most sw projects are determined by business need. I can't think of a single one I've ever written that hasn't needed dependencies or would have been viable had I not used some. Going to a theory level, there is no reason to think the stdlib is magically immune to the issues of a very popular dependency either. Shun absolutes and make the correct trade offs based on your business goals.
That said, most sw projects are determined by business need. I can't think of a single one I've ever written that hasn't needed dependencies or would have been viable had I not used some. Going to a theory level, there is no reason to think the stdlib is magically immune to the issues of a very popular dependency either. Shun absolutes and make the correct trade offs based on your business goals.
Kind of ironic for a language that was the first that I know about where you can straight up import libraries with a URL directly in your source.
It's not a URL, it's just an identifier that sort-of resembles a URL similar to Kubernetes annotation conventions.
It may not be quite a URL, but it contains the URL. It definitely more than just "sort-of resembles a URL". See how the module path to URL lookup is done here: https://go.dev/ref/mod#vcs-find
I understand your meaning, but that link supports my claims. The package path helps the Go tool infer the actual URL, but it doesn't contain the URL itself (e.g., the actual URL has a scheme/protocol component and potentially a `go-get=1` query string argument which don't exist as part of the path). This is what I meant when I said it "sort-of resembles a URL", but I understand from this conversation how that wording wasn't clear.
Fair. Strictly speaking it doesn't even "contain" a URL. But I think in the context of this conversation it acts like a URL -- it allows the Go tooling to fetch code from an arbitrary domain and path on the internet.
[deleted]
I guess that's fine so long as you're covered by the standard library and/or are willing to reimplement a lot of stuff yourself, but that's a significant trade-off you're asking.
It sounds defensive but Go stdlib is all you need. I believe I'm qualified to say that as last year I challenged myself to only use stdlib, out of several languages I used over the course of the year on big projects, Go was painless and that was a unique experience. So far this year I haven't seen much need to add libraries to my work because everything is already within grasp.
sure, just don't use Postgres, OAuth, payments, RPCs, metrics, non-standard crypto, or anything important
This is the exact point of the final section of the article, yes.
>> The only commands that will change the go.mod (and therefore the build) are go get and go mod tidy. These commands are not expected to be run automatically or in CI, so changes to dependency trees must be made deliberately and have the opportunity to go through code review.
GO doesn't do jack shit to mitigate supply chain attacks. Version pinning with checksum and that is it. But what could go do? Solve supply chain attacks as a language feature? That doesn't even make sense.
Application developers using Go must prevent supply chain attacks against their applications. So go get some SAST for your pipeline.
Sure there is truth in saying: always verify your dependencies (and their dependencies) yourself with a code review on every update. But you should not do that alone, so let's talk about collaborative vulnerability management. (there is more to sast than vulnerability assessment, but we have to start somewhere)
Let's say repositories that publishe go modules should also publish a curated list of known vulnerabilities (including known supply chain attacks) for the modules they publish. This curation is work: reports must be verified before being included in the list and they must be verified quickly. This work scales with the number of packages published. And worse, modules could be published in more than one repository, module publishing repositories can be different from the modules source code repository, and vulnerability lists can exist independent from these repositories - so reports should be synced between different list providers. Different implementations and lack of common standards make this a hard problem. And implicit trust for bulk imports could open the door for takedown attacks.
There is an argument that vulnerability listing should be split from source and module publishing: each focusing on their core responsibility. For supply chain attacks this split in responsibilities also makes it harder for an attacker to both attack suppliers and suppress reports. But for all other issues it increase distance as reports must travel upstream. And it creates perverse incentives, like trying to keep reports exclusive to paying customers.
To pile on the insanity: reports can be wrong. And there are unfixed CVEs that are many years old (well ok maybe not for go... yet). Downstream there are "mitigated" and "wont-fix" classifications for reports about dependencies and many SAST tooling can't parse that for transitive dependencies.
Really, supply chain attacks are the easy case in vulnerability management, because they are so obviously a "must-fix" when detected. (and to please the never update crowd: for a downstream project "fix" can mean not updating a dependency into an attacked version. you are welcome)
Long story short: go get some SAST in your pipelines to defend against supply chain attacks. Don't pretend pinning the version and half-assing a code review when you update them is actually solving supply chain attacks. Don't tell me everyone who uses go can find a sophisticated data bomb or intentional rce in some transitive dependency of some lib they update to a new feature release. And don't give me some "well if its transitive then the lib dev should have." Should have doesn't solve shit.
Examples for Vulnerability Assessment in GO dependency graphs are GitLabs Gemnasium ( https://gitlab.com/gitlab-org/security-products/gemnasium-db... ) or GitHubs Dependabot ( https://github.com/advisories?query=type%3Areviewed+ecosyste... ) among many, many others. Not recommendations, just examples!
Tools like these help you sort out supply chain attacks that other people have already found, before you update into them and push them downstream. Collaboration is useful. Sure you are still left with reading the source changes of all dependency update, because who knows, you may be the first one to spot one, but hey, good for you.
GO doesn't do jack shit to mitigate supply chain attacks. Version pinning with checksum and that is it. But what could go do? Solve supply chain attacks as a language feature? That doesn't even make sense.
Application developers using Go must prevent supply chain attacks against their applications. So go get some SAST for your pipeline.
Sure there is truth in saying: always verify your dependencies (and their dependencies) yourself with a code review on every update. But you should not do that alone, so let's talk about collaborative vulnerability management. (there is more to sast than vulnerability assessment, but we have to start somewhere)
Let's say repositories that publishe go modules should also publish a curated list of known vulnerabilities (including known supply chain attacks) for the modules they publish. This curation is work: reports must be verified before being included in the list and they must be verified quickly. This work scales with the number of packages published. And worse, modules could be published in more than one repository, module publishing repositories can be different from the modules source code repository, and vulnerability lists can exist independent from these repositories - so reports should be synced between different list providers. Different implementations and lack of common standards make this a hard problem. And implicit trust for bulk imports could open the door for takedown attacks.
There is an argument that vulnerability listing should be split from source and module publishing: each focusing on their core responsibility. For supply chain attacks this split in responsibilities also makes it harder for an attacker to both attack suppliers and suppress reports. But for all other issues it increase distance as reports must travel upstream. And it creates perverse incentives, like trying to keep reports exclusive to paying customers.
To pile on the insanity: reports can be wrong. And there are unfixed CVEs that are many years old (well ok maybe not for go... yet). Downstream there are "mitigated" and "wont-fix" classifications for reports about dependencies and many SAST tooling can't parse that for transitive dependencies.
Really, supply chain attacks are the easy case in vulnerability management, because they are so obviously a "must-fix" when detected. (and to please the never update crowd: for a downstream project "fix" can mean not updating a dependency into an attacked version. you are welcome)
Long story short: go get some SAST in your pipelines to defend against supply chain attacks. Don't pretend pinning the version and half-assing a code review when you update them is actually solving supply chain attacks. Don't tell me everyone who uses go can find a sophisticated data bomb or intentional rce in some transitive dependency of some lib they update to a new feature release. And don't give me some "well if its transitive then the lib dev should have." Should have doesn't solve shit.
Examples for Vulnerability Assessment in GO dependency graphs are GitLabs Gemnasium ( https://gitlab.com/gitlab-org/security-products/gemnasium-db... ) or GitHubs Dependabot ( https://github.com/advisories?query=type%3Areviewed+ecosyste... ) among many, many others. Not recommendations, just examples!
Tools like these help you sort out supply chain attacks that other people have already found, before you update into them and push them downstream. Collaboration is useful. Sure you are still left with reading the source changes of all dependency update, because who knows, you may be the first one to spot one, but hey, good for you.
node, python, ruby...etc including go modules can lock their dependencies, Go really wins when it can be wrapped in a reasonably-sized one binary, nothing beats that in deployment, for all the others you have to pull in lots of packages into the target system
How about getting safe compiler?
It doesn't. Some other tools are worse. Great, saved you a read.
I don't like the authoritative tone of the article. Especially
given the fact that author "conveniently forgets" about
go mod edit
go work
> Unlike most other package managers files, Go modules don’t have a separate list of constraints and a lock file pinning specific versions. The version of every dependency contributing to any Go build is fully determined by the go.mod file of the main module.
I don't know if this was intentional on the author's part, but this reads to me like it's implying that with package managers that do use lockfiles a new version of a dependency can automatically affect a build.
The purpose of a lockfile is to make that false. If you have a valid lockfile, then fetching dependencies is 100% deterministic across machines and the existence of new versions of a package will not affect the build.
It is true that most package managers will automatically update a lockfile if it's incomplete instead of failing with an error. That's a different behavior from Go where it fails if the go.mod is incomplete. I suspect in practice this UX choice doesn't make much of a difference. If you're running CI with an incomplete lockfile, you've already gotten yourself into a weird state. It implies you have committed a dependency change without actually testing it, or that you tested it locally and then went out of your way to discard the lockfile changes.
Either way, I don't see what this has to do with lockfiles as a concept. Unless I'm missing something, go.mod files are lockfiles.