Nuenv: An experimental Nushell builder for Nix packages(determinate.systems)
determinate.systems
Nuenv: An experimental Nushell builder for Nix packages
https://determinate.systems/posts/nuenv
29 comments
I use NixOS as my daily driver on my two workstations, and package manager on my Mabook Pro now.
It's super helpful if you have an extra computer you can reformat and spend a couple months experimenting with NixOS on, learning it, getting it set up just right, and verifying it can do everything you need your main OS to. Nix is great for this kind of rapid experimental iteration with system config, since if something doesn't work you either quickly fix it and make a new build, or just roll back to the prior working build.
The beauty of Nix is, if that succeeds, the declarative config can instantly apply it to your other computer/s too, and fully transition with minimal risk. Then you can use multiple Nix computers in a sort of A-B testing setup, where you test new config options on one till they work, and copy them over to the others only after you're certain they're ready. Repeat.
And if the initial experiment with it doesn't work out for any reason, you still have your non-Nix computer chugging along doing all the things you need it to do.
It's super helpful if you have an extra computer you can reformat and spend a couple months experimenting with NixOS on, learning it, getting it set up just right, and verifying it can do everything you need your main OS to. Nix is great for this kind of rapid experimental iteration with system config, since if something doesn't work you either quickly fix it and make a new build, or just roll back to the prior working build.
The beauty of Nix is, if that succeeds, the declarative config can instantly apply it to your other computer/s too, and fully transition with minimal risk. Then you can use multiple Nix computers in a sort of A-B testing setup, where you test new config options on one till they work, and copy them over to the others only after you're certain they're ready. Repeat.
And if the initial experiment with it doesn't work out for any reason, you still have your non-Nix computer chugging along doing all the things you need it to do.
I’m curious about nix in macOS. I’ll Kagi / chatgpt it in a bit, I’m sick of brew. But was hoping to hear your experiences as well (assuming you’re not a generative AI). I run a nixos home server so am familiar, just not with using it for macOS.
Not the person you're asking, but I use nix on my MacBook and ditched brew after a few weeks. The main draw for me was that I was already using home-manager on NixOS, and being able to share the same config on macOS was pretty compelling. I had to sprinkle a couple conditionals in the home-manager config to account for things that are linux-only, but it was pretty painless.
Here's an example of using `pkgs.stdenv.isLinux` to conditionally install stuff (there's also an `isDarwin`, if you prefer): https://github.com/yusefnapora/nixos-system-flake/blob/660dd...
For my own use case, I install a few things "globally" in my home-manager config (neovim, vscode, fish shell, etc.). Then each of my projects gets its own flake with a devShell, and I use nix-direnv to bring things into the PATH when I cd into the project directory. I also use `nix shell` a fair bit if I just want to run something in a one-off shell session.
You can also use nix profiles if you're looking for a more brew-like experience, but I never really mess with those. I figure if I want something to be persistently installed, it's better to just stuff it into my home-manager config so it will be there on all my machines.
On the whole, I much prefer nix to brew. I'd still recommend brew to people that don't want to bother with learning nix, but if you're already running NixOS I'd say go for it.
Here's an example of using `pkgs.stdenv.isLinux` to conditionally install stuff (there's also an `isDarwin`, if you prefer): https://github.com/yusefnapora/nixos-system-flake/blob/660dd...
For my own use case, I install a few things "globally" in my home-manager config (neovim, vscode, fish shell, etc.). Then each of my projects gets its own flake with a devShell, and I use nix-direnv to bring things into the PATH when I cd into the project directory. I also use `nix shell` a fair bit if I just want to run something in a one-off shell session.
You can also use nix profiles if you're looking for a more brew-like experience, but I never really mess with those. I figure if I want something to be persistently installed, it's better to just stuff it into my home-manager config so it will be there on all my machines.
On the whole, I much prefer nix to brew. I'd still recommend brew to people that don't want to bother with learning nix, but if you're already running NixOS I'd say go for it.
Thanks! Super helpful.
Nix support on macOS is pretty good, but it's not the main OS of most Nixpkgs contributors and that does show in some ways, and there are some differences in using it as a 'guest' package manager rather than having it run the whole show.
First, the bad:
Apple does minorly break things on every OS upgrade by nuking all of your systemwide shell RC files, besides the more substantial. (There's been some investigation into more robust ways to get that integration working, but so far none of the leads are very promising.)
Nix on macOS has to do some weird stuff to achieve integration due to macOS' security features like SIP and non-writable /. Nowadays the installer just takes care of this.
macOS GUI apps don't get built by Hydra, so they're not in the binary cache. This is because they require proprietary software (Xcode) to build, and Hydra doesn't build or cache non-free derivations as a legal precaution. For grabbing things like iTerm or whatever, you might prefer using Homebrew in a limited way ('casks' only) with a non-standard prefix. This is what I have done and recommend doing. (That said, macOS GUI apps do exist in Nixpkgs and they build just fine, so if you want you can still use Nix for your iTerm and your custom Emacs build and whatever.)
Now the good parts:
Package availability on macOS is still great.
Nix feels really fast compared to Homebrew!
There's a new, alternative Nix installer available from a credible company in the Nix community that seems to cope a bit better with macOS' integration quirks, making uninstalling and reinstalling a bit easier: https://github.com/DeterminateSystems/nix-installer
If you like NixOS' declarative style and service/configuration management capabilities, there's an awesome module system that lets you get a very similar experience on macOS: https://daiderd.com/nix-darwin/
The ugly(?):
The default build environment for packages on macOS uses a completely different toolchain than Nix uses on Linux. LLVM instead of GCC, libsystem instead of glibc. This is appropriate and generally invisible, but you may notice it and think about it when writing your own packages or something like that.
Nix builds and the environment in general are a bit less 'pure' on macOS. Some of that is part of what I just mentioned above. Another aspect is that Nix builds are not sandboxed on macOS.
I hope that helps! Overall, if you like NixOS, I think you will find lots to like about Nix on Mac. :)
First, the bad:
Apple does minorly break things on every OS upgrade by nuking all of your systemwide shell RC files, besides the more substantial. (There's been some investigation into more robust ways to get that integration working, but so far none of the leads are very promising.)
Nix on macOS has to do some weird stuff to achieve integration due to macOS' security features like SIP and non-writable /. Nowadays the installer just takes care of this.
macOS GUI apps don't get built by Hydra, so they're not in the binary cache. This is because they require proprietary software (Xcode) to build, and Hydra doesn't build or cache non-free derivations as a legal precaution. For grabbing things like iTerm or whatever, you might prefer using Homebrew in a limited way ('casks' only) with a non-standard prefix. This is what I have done and recommend doing. (That said, macOS GUI apps do exist in Nixpkgs and they build just fine, so if you want you can still use Nix for your iTerm and your custom Emacs build and whatever.)
Now the good parts:
Package availability on macOS is still great.
Nix feels really fast compared to Homebrew!
There's a new, alternative Nix installer available from a credible company in the Nix community that seems to cope a bit better with macOS' integration quirks, making uninstalling and reinstalling a bit easier: https://github.com/DeterminateSystems/nix-installer
If you like NixOS' declarative style and service/configuration management capabilities, there's an awesome module system that lets you get a very similar experience on macOS: https://daiderd.com/nix-darwin/
The ugly(?):
The default build environment for packages on macOS uses a completely different toolchain than Nix uses on Linux. LLVM instead of GCC, libsystem instead of glibc. This is appropriate and generally invisible, but you may notice it and think about it when writing your own packages or something like that.
Nix builds and the environment in general are a bit less 'pure' on macOS. Some of that is part of what I just mentioned above. Another aspect is that Nix builds are not sandboxed on macOS.
I hope that helps! Overall, if you like NixOS, I think you will find lots to like about Nix on Mac. :)
Amazing thank you super helpful!
Yes, but if it doesn't work there's an indirection between you and the actual problem. Which feels like having to use chopsticks to build a Lego project.
How do you mean? I ran into a number of problems back when learning NixOS, but the error messages were usually sufficient to track them down.
It's possible on NixOS to feel like you are struggling with a problem that you already know how to solve, dammit! This frustration hits harder earlier on in one's Nix journey, imo, when your first instinct may be to just edit that dang config file under /etc.
As you progress, though, the 'layer of indirection' phenomenon almost reverses, as the Nixpkgs repository becomes a quick and reliable resource you can use to learn about how to configure and manage software that you didn't already know all that well.
As you progress, though, the 'layer of indirection' phenomenon almost reverses, as the Nixpkgs repository becomes a quick and reliable resource you can use to learn about how to configure and manage software that you didn't already know all that well.
Agree, once you realize how to traverse search.nixos.org and module definitions in nixpkgs it gets a lot easier.
> just edit that dang config file under /etc.
Using syncthing as an example do search here instead first:
https://search.nixos.org/options?channel=22.11&from=0&size=5...
Or say you are using home-manager for something like neovim, use this options search:
https://mipmip.github.io/home-manager-option-search/
and search "programs.neovim".
Using syncthing as an example do search here instead first:
https://search.nixos.org/options?channel=22.11&from=0&size=5...
Or say you are using home-manager for something like neovim, use this options search:
https://mipmip.github.io/home-manager-option-search/
and search "programs.neovim".
Yep! The first habit one builds with NixOS is first searching the module systems to examine existing support.
And these days, the module systems are so extensive that the result is almost always a pleasant surprise. :)
And these days, the module systems are so extensive that the result is almost always a pleasant surprise. :)
I'm kind of surprised this didn't link to the Nix RFC that considers changing the default builder from Bash to Oil: https://github.com/NixOS/rfcs/pull/99
Of course it does have a different goal (of replacing Bash with OSH instead of only offering an alternative), but it has a lot of good discussion.
Personally I think having alternative builders available is a great idea to improve the developer experience here.
Of course it does have a different goal (of replacing Bash with OSH instead of only offering an alternative), but it has a lot of good discussion.
Personally I think having alternative builders available is a great idea to improve the developer experience here.
(Oil author here) Thanks for mentioning it -- Oil is intended for exactly that kind of thing, in theory. That is, a clean shell language for gluing together distros, which are inherently heterogeneous and messy.
In practice, I think there is limited time / resources / appetite for cleaning up a bunch of 15 year old bash, with the potential to break many things. Shell, and especially bash, can be very fragile.
(I'm not speaking for the Nix team, I'd be interested their viewpoint)
OSH runs many shell scripts unmodified, but not Nix's, even though we implemented many bash features that Nix uses. I mentioned that in the latest release: https://www.oilshell.org/blog/2023/03/release-0.14.2.html
I'd still be interested in bug reports -- non-interactive language features tend to be prioritized, and are realistic
---
So to "concede to reality", I think the approach of having new builders makes sense -- it's ADDING rather than REPLACING. It's sort of like what Rich Hickey says about functional programming -- you don't refactor your code, you just keep adding stuff.
Having only used Nix a tiny bit, I didn't even realize new builders are possible. Is this the only other one?
Is it possible to create some sort of "interface" for them? If there was a documented interface, maybe it will help people understand how to build packages, in addition to plugging in new builders.
---
BTW for people wondering about the project, I just wrote a FAQ:
Oils 2023 FAQ - https://www.oilshell.org/blog/2023/03/faq.html
tl;dr We're stabilizing OSH, the "good parts" that runs your existing scripts. In theory Nix could adopt this, with the harder "replacing" strategy.
And we're renaming Oil to YSH to delineate the stable and unstable parts. YSH could be used with the "adding" strategy.
Though the idea is that you can seamlessly upgrade from OSH and YSH, and not have two different things forever, but the costs/benefits are different for each project. And of course as I say, YSH isn't yet stable. (nushell is also pre - 1.0 from what I understand, but I guess version pinning helps with that.)
Much more detail in the post. We can still use help :)
In practice, I think there is limited time / resources / appetite for cleaning up a bunch of 15 year old bash, with the potential to break many things. Shell, and especially bash, can be very fragile.
(I'm not speaking for the Nix team, I'd be interested their viewpoint)
OSH runs many shell scripts unmodified, but not Nix's, even though we implemented many bash features that Nix uses. I mentioned that in the latest release: https://www.oilshell.org/blog/2023/03/release-0.14.2.html
I'd still be interested in bug reports -- non-interactive language features tend to be prioritized, and are realistic
---
So to "concede to reality", I think the approach of having new builders makes sense -- it's ADDING rather than REPLACING. It's sort of like what Rich Hickey says about functional programming -- you don't refactor your code, you just keep adding stuff.
Having only used Nix a tiny bit, I didn't even realize new builders are possible. Is this the only other one?
Is it possible to create some sort of "interface" for them? If there was a documented interface, maybe it will help people understand how to build packages, in addition to plugging in new builders.
---
BTW for people wondering about the project, I just wrote a FAQ:
Oils 2023 FAQ - https://www.oilshell.org/blog/2023/03/faq.html
tl;dr We're stabilizing OSH, the "good parts" that runs your existing scripts. In theory Nix could adopt this, with the harder "replacing" strategy.
And we're renaming Oil to YSH to delineate the stable and unstable parts. YSH could be used with the "adding" strategy.
Though the idea is that you can seamlessly upgrade from OSH and YSH, and not have two different things forever, but the costs/benefits are different for each project. And of course as I say, YSH isn't yet stable. (nushell is also pre - 1.0 from what I understand, but I guess version pinning helps with that.)
Much more detail in the post. We can still use help :)
> I didn't even realize new builders are possible. Is this the only other one?
The Nix Pills is the document I'd look to to understand what a builder is in Nix: https://nixos.org/guides/nix-pills/working-derivation.html
A builder is any program whose invocation is specified by a Nix derivation (think .drv file, specifically— not just any program that gets called in the course of a Nix build).
Most packages in Nixpkgs are defined using the stdenv.mkDerivation function, which uses a bash builder. This is why a lot of the arguments used in package definitions are bash snippets. They get interpolated and sourced and otherwise woven together to make the bash scripts that run the actual builds.
But for special purposes, it's not unusual to use builders in other languages, especially scripting languages, like Perl, Ruby, or Python. The first use case for special-purpose, non-bash builders that comes to mind for me is Nix code generation tools. When some enterprising Nixer writes a tool to partially or completely auto-generate Nix package definitions from the upstream packages native to their favorite language ecosystem, they often reach for that same language to do the job.
This experiment and the Oil/Nix RFC are about creating another reusable/extensible, general purpose builder that could be used to build many packages, so that snippets in the new shell language could play a similar role to the current role of bash snippets in Nixpkgs. For some, the hope is that eventually such builders could replace the prevalent bash builders in Nixpkgs!
The Nix Pills is the document I'd look to to understand what a builder is in Nix: https://nixos.org/guides/nix-pills/working-derivation.html
A builder is any program whose invocation is specified by a Nix derivation (think .drv file, specifically— not just any program that gets called in the course of a Nix build).
Most packages in Nixpkgs are defined using the stdenv.mkDerivation function, which uses a bash builder. This is why a lot of the arguments used in package definitions are bash snippets. They get interpolated and sourced and otherwise woven together to make the bash scripts that run the actual builds.
But for special purposes, it's not unusual to use builders in other languages, especially scripting languages, like Perl, Ruby, or Python. The first use case for special-purpose, non-bash builders that comes to mind for me is Nix code generation tools. When some enterprising Nixer writes a tool to partially or completely auto-generate Nix package definitions from the upstream packages native to their favorite language ecosystem, they often reach for that same language to do the job.
This experiment and the Oil/Nix RFC are about creating another reusable/extensible, general purpose builder that could be used to build many packages, so that snippets in the new shell language could play a similar role to the current role of bash snippets in Nixpkgs. For some, the hope is that eventually such builders could replace the prevalent bash builders in Nixpkgs!
I'm very happy alternative builders such as nushell and Nickel are being worked on.
Even happier that concurrently the other side of Nix DX that is documentation is very actively being worked on.
Even happier that concurrently the other side of Nix DX that is documentation is very actively being worked on.
Apologies for my "well, actually" pedantry but Nickel is a type-safe alternative to Nix for creating expressions and, as far as I know, not a builder. _But_, that being said, there's indeed a lot of fun areas to explore here that, with enough work and care, could be pretty transformative.
Maybe with the effects system Nickel itself could actually be used as a builder as well as the expression language, although I don't think that's really the intent atm.
Nice to see an alternative to dealing with bash, though was wondering why a shell based on a proper language like python wasn't considered
Because the main business of most Nix builders is shelling out to external programs, and for that there isn't anything that's a better fit than a shell language.
At the same time, the whole mission of these next-gen shell languages (Nushell, Elvish, Oil, Crush, Murex, NGS) is to provide a language that is at once a shell language suitable for daily interactive usage and a capable, general purpose programming language in its own right.
Truthfully, I don't think a better fit exists for this use case than one of these shell languages.
At the same time, the whole mission of these next-gen shell languages (Nushell, Elvish, Oil, Crush, Murex, NGS) is to provide a language that is at once a shell language suitable for daily interactive usage and a capable, general purpose programming language in its own right.
Truthfully, I don't think a better fit exists for this use case than one of these shell languages.
But Python is also a shell language suitable for daily interactive usage, e.g., within a python-based shell called xonsh, so the naming alone doesn't explain much re. which of the next-gen shell languages are better suitable
Imo if you spend a little while using a shell like Xonsh or Ammonite, it becomes clear why the internal DSL approach is not as suitable for this as a new language that is designed with interactive shell usage in mind from the start. (Lisp-y languages might be a bit of an exception since command invocations are very close to the same thing syntactically anyway. But idrk since I've never tried using an internal DSL in a Lisp as a Unix shell.)
But sure, some people do use and love Xonsh, and you could make the case for using a Xonsh builder in this way as well.
But sure, some people do use and love Xonsh, and you could make the case for using a Xonsh builder in this way as well.
For an better shell language that also allows extension in python see xonsh
I know that Python builders do exist out there. Personally, I've found Nushell much more concise than Python et al and quite adequate for the task at hand. But who knows, maybe I should do a whole series :)
i do not want my shell to depend on whatever idea of package management python has this week, nor do i necessarily want to warm up the jit of a ~proper~ general-purpose scripting language in the process of execing a one-line script
default links bash with well know linter and one may make bash or cp other bash into nix and tune. also with nix it is easy to add alternative cli tools like sad, rg, etc.
if i want to be heavy and code real stuff, in nix it is easy to build and run any language. many things people do in bash to glue, i glue using nix declaratively.
so imho until other alt shell wins without nix, will use ugly weird non scalable bash.
if i want to be heavy and code real stuff, in nix it is easy to build and run any language. many things people do in bash to glue, i glue using nix declaratively.
so imho until other alt shell wins without nix, will use ugly weird non scalable bash.
Thanks for mentionind sad, didn't know it yet. To save others a search: https://github.com/ms-jpq/sad
Space Age seD
Space Age seD
With so many fruitful experiments in the Nix ecosystem being carried out by people who can dedicate more time to Nix than ever, I increasingly feel like I'm wasting my life by not seriously working on or with the Nix ecosystem as my main daily occupation.
I know there are still unresolved difficulties and growing pains, but later seems like such a stupid time to really dive in, for people who already know Nix and are already excited about it.