Nix at work: FlakeHub Cache and private flakes(determinate.systems)
determinate.systems
Nix at work: FlakeHub Cache and private flakes
https://determinate.systems/posts/flakehub-cache-and-private-flakes/
Hey folks, Determinate Systems CEO Graham here. Happy to answer questions about our launch today!
31 comments
I appreciate someone from the project commenting here. I left Nix because it just has too many warts, and unfortunately, they've only increased over time, which is the opposite of what I expected.
- I'd update Nixpkgs, and my build would stop working. The logging was awful, so fixing it took hours.
- The mono repo sounds nice, but selective updates are almost impossible.
- Building our Go project in a Nix package was nearly impossible with private GitHub repos, so we moved on.
- Bash scripts never work because binaries are never where they should be in Linux.
I could go on, but my experience with Nix and NixOS left me with significant scars. I’m hopeful that someone will eventually create a more user-friendly deterministic OS.
- I'd update Nixpkgs, and my build would stop working. The logging was awful, so fixing it took hours.
- The mono repo sounds nice, but selective updates are almost impossible.
- Building our Go project in a Nix package was nearly impossible with private GitHub repos, so we moved on.
- Bash scripts never work because binaries are never where they should be in Linux.
I could go on, but my experience with Nix and NixOS left me with significant scars. I’m hopeful that someone will eventually create a more user-friendly deterministic OS.
> - Building our Go project in a Nix package was nearly impossible with private GitHub repos, so we moved on.
Yes. This is a nightmare, and is the literal step-one problem FlakeHub is designed to solve.
Regarding the monorepo causing big-bang updates that broke all your stuff: continued, strong, agreement. I think as nixpkgs continues to grow, and usage grows, we'll see a wider adoption of flakes from a pure "how do we all work together?" perspective. That's the flip side of FlakeHub's initial concept.
Yes. This is a nightmare, and is the literal step-one problem FlakeHub is designed to solve.
Regarding the monorepo causing big-bang updates that broke all your stuff: continued, strong, agreement. I think as nixpkgs continues to grow, and usage grows, we'll see a wider adoption of flakes from a pure "how do we all work together?" perspective. That's the flip side of FlakeHub's initial concept.
Please keep at it! Doing what GitHub did for Git is a real opportunity I think for Nix. We'll be keeping an eye on Determinate's work...
Thank you, we will! =). I think there is a lot of opportunity here around a cohesive workflow around Nix. That is a big part about what I was consulting on for so many years.
Maybe check out our demo on our home page: https://determinate.systems/ -- you might like it.
Maybe check out our demo on our home page: https://determinate.systems/ -- you might like it.
Not sure if you know this, but this is the way to add dependencies to bash scripts:
https://nixos.wiki/wiki/Nix-shell_shebang
It should add everything to $PATH, so unless you've got hardcoded bin paths, there shouldn't be any problems.
https://nixos.wiki/wiki/Nix-shell_shebang
It should add everything to $PATH, so unless you've got hardcoded bin paths, there shouldn't be any problems.
Given the extensive infrastructure you and others are building around flakes in the Nix ecosystem, despite flakes still being experimental, what does the roadmap look like for your efforts, and more broadly for officially establishing flakes??
Sure. To start with, we see flakes as stable as they are today: https://determinate.systems/posts/experimental-does-not-mean.... The ecosystem has broadly adopted flakes as their preferred way to use Nix, and we're standing by them and are committing to not breaking their flakes in the future.
On Monday, we launched Determinate Nix to help make this promise more real: https://determinate.systems/posts/announcing-determinate-nix.... Note: Determinate Nix is not a fork, it is a downstream. Our plan, and intent, is to keep all our patches sent to the upstream project first.
On Monday, we launched Determinate Nix to help make this promise more real: https://determinate.systems/posts/announcing-determinate-nix.... Note: Determinate Nix is not a fork, it is a downstream. Our plan, and intent, is to keep all our patches sent to the upstream project first.
> Note: Determinate Nix is not a fork, it is a downstream. Our plan, and intent, is to keep all our patches sent to the upstream project first.
And what happens if the Nix community doesn't pull those patches, and instead goes with a different solution? Will your downstream adapt to the upstream project, possibly breaking things for your customers?
And what happens if the Nix community doesn't pull those patches, and instead goes with a different solution? Will your downstream adapt to the upstream project, possibly breaking things for your customers?
We won't break our customers.
Indeed, part of the motivation for our downstream distribution is to be able to ship some of our patches faster than upstream wants to. However, these patches are generally about usability improvements that are not incompatible.
If the upstream project evolves in a different direction, it will be on us to move with them too.
Indeed, part of the motivation for our downstream distribution is to be able to ship some of our patches faster than upstream wants to. However, these patches are generally about usability improvements that are not incompatible.
If the upstream project evolves in a different direction, it will be on us to move with them too.
The roadmap is closed-source solutions as described here: https://discourse.nixos.org/t/introducing-flakehub/32044/3
lol. We have two closed source projects that are meaningful in any way: FlakeHub, and determinate-nixd.
Everything else we have is, and all of our improvements to Nix are, open source and permissively licensed. That includes Determinate Nix Installer and zero-to-nix, both of which are permissively licensed with the hope and intention of the upstream project adopting or integrating the material as they saw fit.
Everything else we have is, and all of our improvements to Nix are, open source and permissively licensed. That includes Determinate Nix Installer and zero-to-nix, both of which are permissively licensed with the hope and intention of the upstream project adopting or integrating the material as they saw fit.
Has the situation between you guys and the Nix community cooled down yet?
I have honestly avoided NixOS for quite a while because the situation seemed "unstable". I don't particularly care for the complaints brought about by the open letter, I do however worry if a distribution/community is imploding.
I have honestly avoided NixOS for quite a while because the situation seemed "unstable". I don't particularly care for the complaints brought about by the open letter, I do however worry if a distribution/community is imploding.
There is a lot of turbulance in the project, but I think a lot of that is a natural result of twenty years of history meeting standout commercial success. Our approach here is to continue our pattern of improving Nix and sending patches upstream. We'll also continue improving the ecosystem around Nix, most of that being open source, and only a couple specific projects being proprietary.
Regarding the stability of the project: The project and distribution is incredibly vibrant, and not going anywhere. There are tens of thousands of organizations that depend on it every day, and projects that are funded on the time scale of decades. It's here to stay.
Regarding the stability of the project: The project and distribution is incredibly vibrant, and not going anywhere. There are tens of thousands of organizations that depend on it every day, and projects that are funded on the time scale of decades. It's here to stay.
As someone who just went back to Arch after using NixOS as my daily driver for more than six months, what's in dire need of improvement is the documentation, the rest of the ecosystem was all right.
Agreed. Thank you for saying so. We're working on this in two ways:
1. Reducing the number of moving parts, so there are fewer interfaces that need to be documented and understood ... because they don't exist.
2. Resources like https://zero-to-nix.com/, and https://docs.determinate.systems/.
This is a big lift, though, so anything we do here in the near future will definitely not be enough.
1. Reducing the number of moving parts, so there are fewer interfaces that need to be documented and understood ... because they don't exist.
2. Resources like https://zero-to-nix.com/, and https://docs.determinate.systems/.
This is a big lift, though, so anything we do here in the near future will definitely not be enough.
Those seem like good steps in the right direction. Other things I'd like to see when I try Nix again would be more consensus of what are good conventions to follow for packages and configurations.
Zero-to-Nix looks better than the documentation I used to get started, seems a great resource. But even with the worse documentation I found, I was able to overcome the pretty steep initial learning curve. What did it for me was the lack of documentation for many packages and having to dive into the code every time I needed to configure anything remotely advanced. Whereas in Arch there's always a couple examples that get me 95% of where I want to go, making it very easy to figure out the rest.
But thanks for the effort, I'll be looking forward to giving Nix another go in one or two years to see the progress and maybe switch again, as I really liked the whole concept.
Zero-to-Nix looks better than the documentation I used to get started, seems a great resource. But even with the worse documentation I found, I was able to overcome the pretty steep initial learning curve. What did it for me was the lack of documentation for many packages and having to dive into the code every time I needed to configure anything remotely advanced. Whereas in Arch there's always a couple examples that get me 95% of where I want to go, making it very easy to figure out the rest.
But thanks for the effort, I'll be looking forward to giving Nix another go in one or two years to see the progress and maybe switch again, as I really liked the whole concept.
I'm still driving nixos, but for me, the stacktraces are the hard part when discerning where and why things have gone wrong
Ugh. I feel you. I think one thing that makes this hard is how big and intertwined Nixpkgs and NixOS is. All the modules are loaded all the time, and it can make the displayed errors propagate out to somewhere intractably far away from where the error actually is. There's a ton of work still being done in Nix to make that better, and I'm hopeful for an easier future :').
> There are tens of thousands of organizations that depend on it every day, and projects that are funded on the time scale of decades. It's here to stay.
I like Nix, but what turns me off most is this kind of hyperbole from its evangelists. "tens of thousands of organizations" depend on Nix every day? How do you even come up with that kind of number? And as someone who has actually worked on projects that are funded for decades: the first rule is that things are changing, so stay flexible, chose boring technology, and always have a plan for migration. Binding yourself to a niche framework that is also strongly opinionated with an arcane syntax is pretty much the worst you could do.
As you say, Nix is already over 20 years old, so yes, it's probably not going anywhere, but that goes both ways: it will probably not vanish in the near future, but it'll probably also not become really popular. And that's fine.
I like Nix, but what turns me off most is this kind of hyperbole from its evangelists. "tens of thousands of organizations" depend on Nix every day? How do you even come up with that kind of number? And as someone who has actually worked on projects that are funded for decades: the first rule is that things are changing, so stay flexible, chose boring technology, and always have a plan for migration. Binding yourself to a niche framework that is also strongly opinionated with an arcane syntax is pretty much the worst you could do.
As you say, Nix is already over 20 years old, so yes, it's probably not going anywhere, but that goes both ways: it will probably not vanish in the near future, but it'll probably also not become really popular. And that's fine.
I think it's worth jumping in. Fifteen years ago, it wasn't clear whether Git, Mercurial, or Bazaar would win, but it was evident that something fundamental had forever changed in revision control systems.
Nix feels similar, but for build systems, package management, and configuration management all at once.
Nix feels similar, but for build systems, package management, and configuration management all at once.
Right, I don't know if nix will manage to evolve, or a new implementation will come up, but the box with the wonders of nix has been already open and there's no way people will want to go back to mutable and non-deterministic systems.
I love the idea of Nix, but the current implementation seems too impossibly complex to ever go mainstream. Dynamic hard-to-debug DSL, loads of novel concepts, difficult to do a piecemeal transition, etc. One poster in this thread reported how their supposedly immutable system configuration broke with an update.
Is there anything on the horizon which could embrace the ideas in a better package? Does Guix smooth over enough rough edges?
Is there anything on the horizon which could embrace the ideas in a better package? Does Guix smooth over enough rough edges?
I believe a lot of this comes from Nix being largely "low policy", meaning you can do anything, any way you want, you just have to figure out how. Our goal is to make a more high-policy version that has more workflow and "rails" out of the box to make a good experience.
Can someone do a ELI5 of Nix and flakes for me? I've looked at the docs numerous times and bounced back each time after I failed to understand the USP of Nix...I just don't know what's the main use case(s).
Ah! You asked about flakes as well. Just saw that. Basically flakes are a construct in Nix that enable you to (a) ensure that your Nix code is always "pinned" to specific revisions of your inputs (for example Nixpkgs but other inputs as well) and (b) to provide a simple and well understood system for sharing Nix code. So this command, for example, enables me to build Tailscale:
nix build github:tailscale/tailscale
I point Nix at an "address" and it knows which set of Nix build instructions to use. Basically that Tailscale repo has a flake.nix in the root that Nix looks at, specifically at the flake outputs. From there, it sees that there is an attribute called "packages" and then it sees that there's a package for the current system called "default," and then it builds that. So Nix expressions have universal "addresses" (they also did before flakes but flakes provide a standard structure for outputs and built-in pinning via flake.lock files).
nix build github:tailscale/tailscale
I point Nix at an "address" and it knows which set of Nix build instructions to use. Basically that Tailscale repo has a flake.nix in the root that Nix looks at, specifically at the flake outputs. From there, it sees that there is an attribute called "packages" and then it sees that there's a package for the current system called "default," and then it builds that. So Nix expressions have universal "addresses" (they also did before flakes but flakes provide a standard structure for outputs and built-in pinning via flake.lock files).
you lost me at the first sentence. all i understood is that it is a thingamajig that I can manipulate to package the stuff into window Puerto Rico the fox flies over area 54. ok.
I've been meaning to write something pithy on this and this is a great spur to action. Basically Nix is something you use to build software reproducibly, that is, where you can take the same inputs and get the exact same bytes on disk every time you build it. The "software" here could be something simple-ish like a CLI tool or a UNIX utility, and it could be as complex as an entire functioning Linux system, and it could be something that you usually build using other tools, like Docker or Podman containers. Nix can do all of this. I'd say that the things people most often use Nix for are:
1. Project-specific development environments. You want everyone on your team to have the exact same version of Python, jq, Terraform, Cargo, and Bun in a specific project? Nix can do that. 2. Home environments. You want to replace your dotfiles and Homebrew and apt-get with something declarative that you have an actual language for? Nix can do that.
So yeah, development environments, home environments, whole Linux systems, OCI containers, standard packages (like CLI tools), things like VS Code extension, Neovim plugins... all of this stuff can be built using Nix. Pretty powerful stuff.
1. Project-specific development environments. You want everyone on your team to have the exact same version of Python, jq, Terraform, Cargo, and Bun in a specific project? Nix can do that. 2. Home environments. You want to replace your dotfiles and Homebrew and apt-get with something declarative that you have an actual language for? Nix can do that.
So yeah, development environments, home environments, whole Linux systems, OCI containers, standard packages (like CLI tools), things like VS Code extension, Neovim plugins... all of this stuff can be built using Nix. Pretty powerful stuff.
now that makes a lot of sense. in fact id like to have both! does this work in wsl?
Yes!
1. Enable systemd: https://devblogs.microsoft.com/commandline/systemd-support-i...
2. Install as normal:
curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install --determinate
3. https://zero-to-nix.com/ :)
1. Enable systemd: https://devblogs.microsoft.com/commandline/systemd-support-i...
2. Install as normal:
curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install --determinate
3. https://zero-to-nix.com/ :)
[deleted]
[deleted]