Building Go programs with Nix flakes(xeiaso.net)
xeiaso.net
Building Go programs with Nix flakes
https://xeiaso.net/blog/nix-flakes-go-programs
25 comments
Nice to have some development in this space. However I would be very interested in some discussion why this approach should be superior to goBuildModule, which is also mentioned in the article. goBuildModule works well for us and it is even arguably simpler to use, you just have to update the vendor hash whenever you updated the dependencies.
The only real downside for us is that it doesn't handle private dependencies well.
The announcement post for gomod2nix has some additional info on the problems with buildGoModule's approach to vendorization:
> The buildGoModule package is designed around fixed-output derivations, which means that a single derivation is created where all the dependencies of the package you want to build are wrapped, and only a single hash of the derivation is specified. It fetches all dependencies in the fixed output, creating a vendor directory which is used for the build.
> This has several issues, most notably there is no sharing of dependencies between packages that depend on the same Go module.
> The other notable issue is that it forces developers to remember editing the vendorSha256 attribute separately from the already existing hash/sha256 attribute on the derivation. Forgetting to do so can not only lead to incorrect builds but also be frustrating when working with larger packages that takes a long time to build, and only very late in the build notice that something was broken so you have to start over from scratch.
> Because of the lack of hash granularity the build needs to clone every dependency every time the vendorSha256 is invalidated, and cannot use the cache from previous builds.
https://www.tweag.io/blog/2021-03-04-gomod2nix/
So you get greater granularity here, and thus also greater cache reuse between builds.
> The buildGoModule package is designed around fixed-output derivations, which means that a single derivation is created where all the dependencies of the package you want to build are wrapped, and only a single hash of the derivation is specified. It fetches all dependencies in the fixed output, creating a vendor directory which is used for the build.
> This has several issues, most notably there is no sharing of dependencies between packages that depend on the same Go module.
> The other notable issue is that it forces developers to remember editing the vendorSha256 attribute separately from the already existing hash/sha256 attribute on the derivation. Forgetting to do so can not only lead to incorrect builds but also be frustrating when working with larger packages that takes a long time to build, and only very late in the build notice that something was broken so you have to start over from scratch.
> Because of the lack of hash granularity the build needs to clone every dependency every time the vendorSha256 is invalidated, and cannot use the cache from previous builds.
https://www.tweag.io/blog/2021-03-04-gomod2nix/
So you get greater granularity here, and thus also greater cache reuse between builds.
Another advantage of the gomod2nix approach concerns long term build reproducibility [1]. Having each source dependency in its own derivation would make easier source code mirroring. Since Nix knows the hash of each sources, we could imagine to teach Nix how to get the source code from some other places than GitHub. For instance, Nix could use the hash of the dependency to query Software Heritage [2].
Note there is also an opened issue to restrict Nix fixed output derivation which would make impossible the buildGoModule "hack".
[1] https://github.com/NixOS/nixpkgs/issues/84826
[2] https://www.tweag.io/blog/2020-06-18-software-heritage/
Note there is also an opened issue to restrict Nix fixed output derivation which would make impossible the buildGoModule "hack".
[1] https://github.com/NixOS/nixpkgs/issues/84826
[2] https://www.tweag.io/blog/2020-06-18-software-heritage/
That's actually really important. Sometimes upstream sources do disappear, and the caches we have don't help very much, or only help for the exact collection used in a specific package, when they're one big chunk like with buildGoModule.
I hope that FODs in Nix itself stay flexible, though, even if a more limited form becomes preferable in Nixpkgs. New fetchers being possible to implement in Nixpkgs rather than in Nix itself seems really valuable, and clunky uses of FODs to wrap upstream vendorization are still an important fallback option for wrapping new or ill-behaved upstream package managers, until better implementations are worked out or become possible.
I hope that FODs in Nix itself stay flexible, though, even if a more limited form becomes preferable in Nixpkgs. New fetchers being possible to implement in Nixpkgs rather than in Nix itself seems really valuable, and clunky uses of FODs to wrap upstream vendorization are still an important fallback option for wrapping new or ill-behaved upstream package managers, until better implementations are worked out or become possible.
Is there any reason this isn't just merged into the buildGoModule package? It seems like a net positive?
See https://code.tvl.fyi/tree/nix/buildGo as an alternative.
"Most language-specific Nix tooling outsources the build to existing language-specific build tooling, which essentially means that Nix ends up being a wrapper around all sorts of external build systems.
However, systems like Bazel take an alternative approach in which the compiler is invoked directly and the composition of programs and libraries stays within a single homogeneous build system.
Users don't need to learn per-language build systems and especially for companies with large monorepo-setups (like Google) this has huge productivity impact.
This project is an attempt to prove that Nix can be used in a similar style to build software directly, rather than shelling out to other build systems. "
"Most language-specific Nix tooling outsources the build to existing language-specific build tooling, which essentially means that Nix ends up being a wrapper around all sorts of external build systems.
However, systems like Bazel take an alternative approach in which the compiler is invoked directly and the composition of programs and libraries stays within a single homogeneous build system.
Users don't need to learn per-language build systems and especially for companies with large monorepo-setups (like Google) this has huge productivity impact.
This project is an attempt to prove that Nix can be used in a similar style to build software directly, rather than shelling out to other build systems. "
The main difference is that it's easier to track what changed in the vendor closure rather than just waiting for the build to fail and updating the hash with the result that Nix gives you.
but how does it react when go.mod and the toml get out of sync? For us this will just break on CI and requires then a commit with the updated vendorSha to be fixed. I would expect the same workflow from this approach?
The only reason they would get out of sync is if you updated dependency versions without also running gomod2nix. The benefit here is that you can do that without the intermediate step of first building the project and updating the hash after the build fails.
Now if only there were a reliable way to do this with Python programs with a requirements.txt.
Last time I tried (maybe 2 months ago on NixOS 22.05), I found several different tools that claimed to be able to do this and they all failed in various ways; some failed quickly, others took a long time to fail. I ended up using my brain as an SAT solver and manually copying in nix expressions from older versions of nixpkgs that matched versions of the python libs that were needed.
Last time I tried (maybe 2 months ago on NixOS 22.05), I found several different tools that claimed to be able to do this and they all failed in various ways; some failed quickly, others took a long time to fail. I ended up using my brain as an SAT solver and manually copying in nix expressions from older versions of nixpkgs that matched versions of the python libs that were needed.
Yeah the situation with Python is kind of crap to begin with. The main problem with requirements.txt is that it doesn't lock the SHAsums of the packages being depended on.
The Python ecosystem also doesn't seem to have a sensible solution for dependency declaration in general. It's kind of remarkable that anything works at all in Python land, it's a lot of "let's hope it works often enough". To be fair it does work out a lot of the time, but when that fails you end up coming to people like me.
I should do some research and see if I can figure out the best way to do this in Python, but I don't use Python at all personally or professionally.
The Python ecosystem also doesn't seem to have a sensible solution for dependency declaration in general. It's kind of remarkable that anything works at all in Python land, it's a lot of "let's hope it works often enough". To be fair it does work out a lot of the time, but when that fails you end up coming to people like me.
I should do some research and see if I can figure out the best way to do this in Python, but I don't use Python at all personally or professionally.
What would you consider a sensible solution?
What about
* Pip-tools(requirements.in + requirements.txt)
* Poetry (pyproject.toml+poetry.lock)
* Pipenv(Pipfile+Pipfile.lock)
Those and others enable easy declaration of direct dependencies and compile (resolve) a lock file of the actual dependency graph with hashes. Those tools have their issues and idiosyncrasies, but the dependency declaration aspect seems sensible and functional.
* Pip-tools(requirements.in + requirements.txt)
* Poetry (pyproject.toml+poetry.lock)
* Pipenv(Pipfile+Pipfile.lock)
Those and others enable easy declaration of direct dependencies and compile (resolve) a lock file of the actual dependency graph with hashes. Those tools have their issues and idiosyncrasies, but the dependency declaration aspect seems sensible and functional.
Does https://pypi.org/project/pypi2nix/ work?
Now that pip supports pinning with a sha (I say “now that”, this was added years ago but it wasn’t always thus), it seems it should be trivial to map onto the Nix model.
I have been working with poetry and playing with poetry2nix so haven’t tried raw pip requirements files with Nix.
Now that pip supports pinning with a sha (I say “now that”, this was added years ago but it wasn’t always thus), it seems it should be trivial to map onto the Nix model.
I have been working with poetry and playing with poetry2nix so haven’t tried raw pip requirements files with Nix.
If I were a better engineer, I'd have recorded a list of all the things I tried, but I didn't. Most of them ended in the substring "2nix" but the names kind of blur together so I can't definitively say I tried it. I have some free time on the 23rd and may see if pypi2nix works with the project or not, and file a bug if it doesn't.
PEP 518 has been around for a long time. All of the projects must be running pyproject.toml + poetry.lock. Like really, it's the new modern standard, which solves a lot of headaches. The configs, the Python version, the dependencies, are all nicely crammed together. There's even a thing poetry2nix, which seems pretty well supported.
"go" the übertool is so well thought out and handy, I'm missing how this doesn't essentially add complexity.
Dependency management in go is…underwhelming to say the least. Go get? Go mod download? Go install? Go tool error messages related to the same? All a mess. This issue, implicit interfaces, and a CS-3/400 class project quality implementation of generics (harsh, I know) are three reasons that the language frustrates me to no end.
Agreed on most points, but why the distaste for implicit interfaces? I find myself really, really liking that feature.
Accidental implementation of an interface is a footgun at best. These are contracts that should be explicit and deliberate. “Write” doesn’t always mean “I/O,” for example, but if you have data for which “Write” is a perfectly reasonable method name and isn’t I/O you’re inviting eventual disaster by implementing a method with the same signature as oi.Write.
I think there's two angles to come at gomodule2nix from, basically. You might either be
Here's an answer more tailored towards that second group, Nix-curious maintainers of Go packages. You might want to add a Nix flake to your existing Go project if:
Case (b) for Go-first users is kind of similar to the general appeal for Nix-first users, but with less commitment. Nix has uniform ways of managing ephemeral applications that don't permanently live on your usual PATH. They're awesome for trying out new software in a non-committal way. Adding a Nix flake to your project is thus morally a bit like providing a Homebrew recipe: it makes it easy to run or install your software via a general tool that prospective new users may already have and be willing to use. Could prospective new users 'just' use `go install` to install your little CLI app instead of using a Nix package? Sure! Are they likely to prefer to do that if they're not themselves to developers? Maybe not.
In the Go-first use case, I think the sensible complexity considerations to make are the cost (to you) of maintaining the flake. The problems to consider would be whether or not it will divide your users/contributors into groups using different toolchains (it won't, since it reuses your existing Go übertool's resolved dependencies, and whether users/contributors who don't plan on using that integration would nonetheless be confronted with its complexity (they won't; not using the flake is free and this tooling doesn't modify any non-Nix files). So that really just leaves the question of whether you think any potential users/contributors will enjoy being able to leverage your package via Nix, and how much effort it will take from you to offer them that.
If you think none of your current or potential users/contributors use Nix, then this probably is an increase in complexity for you with no payoff for you. But otherwise, it might be worth doing or involve less complexity than alternative solutions.
1. someone who is already a Nix user, figuring out if/how you want to build a Go package you have for use with other Nix tooling; or
2. someone who is already a maintainer of a Go package, trying to decide if/how you want to offer Nix integration to your users/contributors
- are NixOS users
- use Nix in CI/CD
- use Nix to create/deploy server images
- use Nix to create/deploy container images
- use Nix to provide reproducible development environments across different projects written in different programming languages
Here's an answer more tailored towards that second group, Nix-curious maintainers of Go packages. You might want to add a Nix flake to your existing Go project if:
(a) your project uses CGo or any foreign libs via FFI
(b) you have or would like to have users/contributors who are new to Go but not to Nix
(c) your application needs to communicate with external programs at runtime to do anything useful (e.g., it's a microservice)
(d) you would like to provide users/contributors ready-to-use debugging, monitoring, or editing tools
Case (b) for Go-first users is kind of similar to the general appeal for Nix-first users, but with less commitment. Nix has uniform ways of managing ephemeral applications that don't permanently live on your usual PATH. They're awesome for trying out new software in a non-committal way. Adding a Nix flake to your project is thus morally a bit like providing a Homebrew recipe: it makes it easy to run or install your software via a general tool that prospective new users may already have and be willing to use. Could prospective new users 'just' use `go install` to install your little CLI app instead of using a Nix package? Sure! Are they likely to prefer to do that if they're not themselves to developers? Maybe not.
In the Go-first use case, I think the sensible complexity considerations to make are the cost (to you) of maintaining the flake. The problems to consider would be whether or not it will divide your users/contributors into groups using different toolchains (it won't, since it reuses your existing Go übertool's resolved dependencies, and whether users/contributors who don't plan on using that integration would nonetheless be confronted with its complexity (they won't; not using the flake is free and this tooling doesn't modify any non-Nix files). So that really just leaves the question of whether you think any potential users/contributors will enjoy being able to leverage your package via Nix, and how much effort it will take from you to offer them that.
If you think none of your current or potential users/contributors use Nix, then this probably is an increase in complexity for you with no payoff for you. But otherwise, it might be worth doing or involve less complexity than alternative solutions.
excellent clarity; thanks for this thoughtful response
Has anyone followed this guide and gotten it to work? I've followed the instructions but get a command not found error when I run 'gomod2nix.'
I think a step that isn't directly clear is that you need to be in the nix env for the flake where it should then be available