NixOS is a good server OS, except when it isn't(sidhion.com)
sidhion.com
NixOS is a good server OS, except when it isn't
https://sidhion.com/blog/posts/nixos_server_issues/
172 comments
What's so bad about it?
Putting aside the poor typing (the lack of proper typing is a shame, so valid criticism), I actually really like the language - it's genuinely a great DSL for the particular problems it's supposed to handle.
It does take a bit of use for it to click, though. A lot of it has to do not with Nixlang itself but about learning nixpkgs' idioms.
Putting aside the poor typing (the lack of proper typing is a shame, so valid criticism), I actually really like the language - it's genuinely a great DSL for the particular problems it's supposed to handle.
It does take a bit of use for it to click, though. A lot of it has to do not with Nixlang itself but about learning nixpkgs' idioms.
It's not just poor typing. It's the lack of discoverability. "What are all my options here in this expression? What are the symbols available?"
I think a good IDE integration could solve this, but not sure how much is possible.
I think a good IDE integration could solve this, but not sure how much is possible.
Like (iirc) systemd-resolved has `enabled` which is false by default but then gets silently turned on if you use systemd-networkd. How are you supposed to figure that out without reading the source?
But I think this also stems from the fact that the default state of nixos is "a general purpose linux system" and so instead of just starting at 0 and adding the things you need, you have to mix adding and removing things which IMO makes things much more complicated (except maybe for newbies to linux who don't know what's necessary for a running system).
But I think this also stems from the fact that the default state of nixos is "a general purpose linux system" and so instead of just starting at 0 and adding the things you need, you have to mix adding and removing things which IMO makes things much more complicated (except maybe for newbies to linux who don't know what's necessary for a running system).
nixos-rebuild repl and then you can inspect things like config.services.resolved.enable or :p options.services.resolved.enable.definitionsWithLocations
With a default config you start with a console, systemd, dbus and some things to make it boot. There is barely anything.
With a default config you start with a console, systemd, dbus and some things to make it boot. There is barely anything.
You instantiate the nix config and then lookup the value
`nix-instantiate --eval -A config.services.resolved.enabled /etc/nixos/configuration.nix`
This is way better than a stateful package manager making a non-revertible change without even telling me.
`nix-instantiate --eval -A config.services.resolved.enabled /etc/nixos/configuration.nix`
This is way better than a stateful package manager making a non-revertible change without even telling me.
For other distros, that kind of behind the scenes changes are triggered by package installation and ad-hoc commands. It's not always easy to figure out exactly what has changed, especially after the fact. This, in turn, makes changes difficult to revert.
NixOS is much better because you can inspect the changes after the fact. You also know which code to look for, which is a luxury. If the code seems too much, there's the repl to help. Changes are also much easier to revert.
NixOS is much better because you can inspect the changes after the fact. You also know which code to look for, which is a luxury. If the code seems too much, there's the repl to help. Changes are also much easier to revert.
The nix repl can be a very valuable tool in answering these questions.
That said, I strive to structure my nix source so that portions of it can easily be pasted into a repl. ReadTree goes a long way in that regard: https://github.com/tvlfyi/kit/tree/canon/readTree
More to your point, though: I think a lot is possible. Although nix is very dynamic, it is also, for all intents and purposes, side effect free. I've had this idea that a sufficiently advanced IDE should be able to evaluate your nix code and tell you exactly what the possible values (not just types, but value!) are for any particular variable.
That said, I strive to structure my nix source so that portions of it can easily be pasted into a repl. ReadTree goes a long way in that regard: https://github.com/tvlfyi/kit/tree/canon/readTree
More to your point, though: I think a lot is possible. Although nix is very dynamic, it is also, for all intents and purposes, side effect free. I've had this idea that a sufficiently advanced IDE should be able to evaluate your nix code and tell you exactly what the possible values (not just types, but value!) are for any particular variable.
> I've had this idea that a sufficiently advanced IDE should be able to evaluate your nix code and tell you exactly what the possible values (not just types, but value!)
Similarly to the REPL, I'm often using `nix-instantiate --eval -E 'somethingsomething'` so it should definitely be possible.
Similarly to the REPL, I'm often using `nix-instantiate --eval -E 'somethingsomething'` so it should definitely be possible.
This is inherently a Hard Problem™, since completions may require evaluating arbitrary derivations (e.g. building a custom Linux kernel).
For "what symbols are available", the nil LSP implementation[1] works for anything in scope that doesn't require evaluation. It also includes completions for the stdlib and NixOS options (in certain contexts).
Another LSP implementation is nixd[2], which is trying to tackle the problem of evaluations for completion.
[1] https://github.com/oxalica/nil/
[2] https://github.com/nix-community/nixd
For "what symbols are available", the nil LSP implementation[1] works for anything in scope that doesn't require evaluation. It also includes completions for the stdlib and NixOS options (in certain contexts).
Another LSP implementation is nixd[2], which is trying to tackle the problem of evaluations for completion.
[1] https://github.com/oxalica/nil/
[2] https://github.com/nix-community/nixd
Maybe then IDE is not the best target, but rather a clear "REPL-friendliness at all times" is, e.g. everything is built around constant re-evaluation where you can discover things where interested, maybe put stop gaps and prints easily.
The community has built two LSPs, and https://search.nixos.org/options (or "man configuration.nix", if you prefer) shows what NixOS options are available
There is this: https://github.com/nix-community/nixd
It has jump to definition and autocomplete. Which is very nice.
It's not perfect. But it's pretty good
It has jump to definition and autocomplete. Which is very nice.
It's not perfect. But it's pretty good
> What are all my options here in this expression? What are the symbols available?"
Unlike most languages, the symbols available are completely determined by the scope. Just look at the let expressions in effect. There's no magic.
As for nested expressions, that's a typing problem, which was already mentioned above as a pain point (although there are several efforts to fix this).
Unlike most languages, the symbols available are completely determined by the scope. Just look at the let expressions in effect. There's no magic.
As for nested expressions, that's a typing problem, which was already mentioned above as a pain point (although there are several efforts to fix this).
There is https://search.nixos.org/ and on the command line you can play around with:
$ nix repl
nix-repl> :l <nixpkgs>
nix-repl> {press tab for auto-complete}
$ nix repl
nix-repl> :l <nixpkgs>
nix-repl> {press tab for auto-complete}
But that’s not the fault of the language. insert another DSL would have the same issues.
Way too much sugar/"idioms", which makes it hard for someone new to the language to figure out what a given piece of code is actually doing. Confusing use of semicolons for what almost every other language uses commas or newlines (or nothing) for. It's the same feeling as writing bash, and needing to always look up again exactly what the syntax is and where the semicolons go.
Here's all the cases for using a semicolon. (estimated 30 seconds read)
1. At the end of local variables
1. At the end of local variables
let
a = 1;
b = 2;
in
a + b
result: 3
{
a = 1;
b = 2;
}
result: { a = 1; b = 2; }
with pkgs;
coreutils
result: (the coreutils attribute in the pkgs attribute set)
assert a != null;
a
result: (the value of a)
This sounds like a skill problem if you don't even know bash or rust or PHP syntax.
The Nix language also doesn't need to exist. They want to write pure lazy declarative derivations - great, you can do that in any existing language. It's a matter of style and APIs. You don't need to spend years developing a brand new language from scratch. Not to mention that many derivations end up calling a Bash script underneath anyway because at some point you actually need to perform an action in the real world. How would a derivation looked like if written in <insert your favorite scripting language> with lazy APIs?
The Nix language is basically what you'd get if you were designing a DSL for declarative configuration of extremely deeply nested trees. Nearly every "feature" in the language is for making that easier. You could probably write a Nix -> JSON compiler and end up with something completely unreadable because the language hides so much of what's actually present in the fully resolved tree.
That's not true. The language needs to be declarative. if you use your favorite lang and develop a DSL around those APIS, that DSL would be declarative.. also the language itself, isn't that complicated.... it's really quite minimal. It has its roots in ML. I honestly think people expect everything to look like javascript or C++ and it's a shame honestly.
Pulumi is a great example of declarative APIs built with imperative programming languages. SwiftUI is another.
Personally I have nothing against the Nix language, and use it without issue, but it's untrue to suggest that the language itself requires uncommon support for this kind of thing.
Personally I have nothing against the Nix language, and use it without issue, but it's untrue to suggest that the language itself requires uncommon support for this kind of thing.
Ooof... Pulumi et al are terrible to write and read. Why should I care about writing 'new' in front of all my declarative configuration? What happens when an if statement depends on a concrete value? How would that even work? The leakiness of the abstraction is too terrible to even consider.
Terraform et al, despite not being my favorite, have much simpler semantics than Pulumi. It's not always a good idea to write DSLs into languages with huge paradigm mismatches.
Terraform et al, despite not being my favorite, have much simpler semantics than Pulumi. It's not always a good idea to write DSLs into languages with huge paradigm mismatches.
Terraform and Pulumi have basically the same semantics.
> Why should I care about writing 'new' in front of all my declarative configuration?
Because that’s how your choice of language instantiates an object. Try F# or Swift or Go if it’s that annoying to you.
> What happens when an if statement depends on a concrete value?
What do you think “count = var.concrete_value ? 1 : 0” is doing in Terraform, exactly?
> The leakiness of the abstraction is too terrible to even consider.
While you are are entitled to your opinion, I’d suggest you are very much mistaken, and would implore you to actually consider it for a minute.
> Why should I care about writing 'new' in front of all my declarative configuration?
Because that’s how your choice of language instantiates an object. Try F# or Swift or Go if it’s that annoying to you.
> What happens when an if statement depends on a concrete value?
What do you think “count = var.concrete_value ? 1 : 0” is doing in Terraform, exactly?
> The leakiness of the abstraction is too terrible to even consider.
While you are are entitled to your opinion, I’d suggest you are very much mistaken, and would implore you to actually consider it for a minute.
Just as a point of order. You offer no case for Pulumi and your one actual discussion of the semantics is misplaced as it deals with if expressions, not statements. Stratified ifs that occur at the non-recursive areas of the language are usually not a problem for these change management systems.
I'm not offering a case for or against either, to be clear - and neither are you - indeed you went so far as to say effectively you wouldn't even bother considering the semantics since they _must_ be awful.
The semantics of the Pulumi runtime are probably fine, but the semantics of the DSL layer that preceeds that when mixed with the sensible semantics of the Pulumi runtime are a recipe for disaster. This is based on my extensive experience with declarative DSLs in imperative languages. The impedance mismatch is high. Nothing to do with Pulumi's internal state management, but it's beholden to javascript.
Personally I see it as similar to typed vs untyped languages. You can add typing to untyped languages or you can just use a typed language. The language used shapes the structure and some are easier to reason about than others (to some people).
Some people don't want to hear this, but it is 100% true.
Meh... the nix language being as it is makes it a lot easier to write these things with less cruft. Every attempt I've seen at introducing laziness into a language like python, c++, rust, javascript, etc just seems to require a lot more unnecessary keywords and helper functions and cruft.
There is Guix, which replaces the Nix language with Scheme, but which has some limitations related to a smaller user base, e.g. a smaller package collection.
Replacing the language requires duplicating all the work that went into Nix, to reach parity, so it is not easy.
Replacing the language requires duplicating all the work that went into Nix, to reach parity, so it is not easy.
> Replacing the language requires duplicating all the work that went into Nix, to reach parity, so it is not easy.
That seems like a design flaw in Nix, there's no reason the data model should be so tightly coupled to the scripting implementation that you can't reuse packages written in a different language.
That seems like a design flaw in Nix, there's no reason the data model should be so tightly coupled to the scripting implementation that you can't reuse packages written in a different language.
There is no technical barrier against doing that. But much of the power and flexibility in nixpkgs arises from the nix language, not the data model (which is comparatively simple).
For example, see zb: https://www.zombiezen.com/blog/2024/09/zb-early-stage-build-...
Using a different language to depend on packages derived from .nix would be very much akin to depending on a docker image whose Dockerfile you can not inspect.
For example, see zb: https://www.zombiezen.com/blog/2024/09/zb-early-stage-build-...
Using a different language to depend on packages derived from .nix would be very much akin to depending on a docker image whose Dockerfile you can not inspect.
> Using a different language to depend on packages derived from .nix would be very much akin to depending on a docker image whose Dockerfile you can not inspect.
Speaking of Docker images and Dockerfiles, that's actually a real-world example of how you can achieve this kind of effect without relying on a specific language. Ironically, you can use Nix to build Docker images; there's a bunch of other alternative builders (e.g. Kaniko, Buildah); you can also just stitch together some files&metadata into tarballs, and then 'docker import' it.
Nix or Guix are of course much more powerful and expressive than Docker images, but there's always a cost to complexity.
Speaking of Docker images and Dockerfiles, that's actually a real-world example of how you can achieve this kind of effect without relying on a specific language. Ironically, you can use Nix to build Docker images; there's a bunch of other alternative builders (e.g. Kaniko, Buildah); you can also just stitch together some files&metadata into tarballs, and then 'docker import' it.
Nix or Guix are of course much more powerful and expressive than Docker images, but there's always a cost to complexity.
That sounds like a design flaw in the data model if the flexibility exists at a higher level.
What would a better design look like?
One where the "power and flexibility" of nixpkgs is encoded in the data model?
If there is something that can be done in the nix language that can't be expressed in the underlying model that needs to be used by another frontend then it should be represented in the underlying model so another frontend can use it.
To put it another way, if you're designing a client-server model where there may be multiple client implementations you don't bake big chunks of the implementation into the clients, you provide it in the server interfaces and data types.
If there is something that can be done in the nix language that can't be expressed in the underlying model that needs to be used by another frontend then it should be represented in the underlying model so another frontend can use it.
To put it another way, if you're designing a client-server model where there may be multiple client implementations you don't bake big chunks of the implementation into the clients, you provide it in the server interfaces and data types.
Okay, but... how? You can't serialize a function or a closure very easily. I'm unaware of any language which attempts to do so.
Not having functions as values (true of pretty much any serialization scheme I've ever seen) makes serialized data structures strictly less powerful than data structures in code.
> that needs to be used by another frontend
I don't think this was ever a goal of Nix. But if it was, well, you would end up with something considerably less powerful for the reasons I stated.
Not having functions as values (true of pretty much any serialization scheme I've ever seen) makes serialized data structures strictly less powerful than data structures in code.
> that needs to be used by another frontend
I don't think this was ever a goal of Nix. But if it was, well, you would end up with something considerably less powerful for the reasons I stated.
The biggest limitation IMO it's that they are HPC-centric, not caring the desktop, which is the was to allow more people discovering a distro. Also the lack of a proper zfs and lvm/mdraid/luks support it's a big showstopper.
IIRC guix started off by being compatible with nix package derivations, but they broken it (i also may remember incorrectly).
If nothing changed, they also have a strong ideological drive and funny support any non-free software.
If nothing changed, they also have a strong ideological drive and funny support any non-free software.
What is a "package derivation"? Packages and derivations are two different things in Guix.
Also, Guix supports proprietary software just fine. It's just not in the main official repo. But there are other repos that have it, e.g. nonguix.
Also, Guix supports proprietary software just fine. It's just not in the main official repo. But there are other repos that have it, e.g. nonguix.
Scheme is so much worse. It has so much repetition and is so much more verbose for no reason.
A macro-capable lisp means that this isn't permanently or definitionally true, even if it is today - which is pretty subjective.
https://www.gnu.org/software/guile/manual/html_node/Macros.h...
Racket is IMO a pretty compelling environment for prototyping DSLs because of how malleable it can be, so I think the ceiling for ergonomics can be pretty high.
https://www.gnu.org/software/guile/manual/html_node/Macros.h...
Racket is IMO a pretty compelling environment for prototyping DSLs because of how malleable it can be, so I think the ceiling for ergonomics can be pretty high.
I find the nix language to be quite pleasant. There are some syntax quirks and types would be nice, but in general the “json with functions” vibe is imo great and a very nice fit for the domain. Lots of other modern config languages (e.g. dhall, jsonnet) have ended up in this part of the design space too.
With that said tweag has been working on a kind of nix 2.0 / nix with types for a while with the aim (I think) of being able to use it in nixpkgs: https://github.com/tweag/nickel
With that said tweag has been working on a kind of nix 2.0 / nix with types for a while with the aim (I think) of being able to use it in nixpkgs: https://github.com/tweag/nickel
I also quite like nixlang for config tasks - in theory! In practice its really annoying. I think the main problem is the interpreter and the bad error messages / bad debuggability.
Part of that just comes from lazy evaluation, which makes debugging a lot harder in general (you feel this in Haskell...), but also just from nix not being a big popular language that gets lots of polish, and being completely dynamically typed.
Part of that just comes from lazy evaluation, which makes debugging a lot harder in general (you feel this in Haskell...), but also just from nix not being a big popular language that gets lots of polish, and being completely dynamically typed.
Nix is ok. I like jsonnet more, and once I've tried to write a converter from jsonnet to nix, but it turned out this is much harder than expected (some idioms don't transfer from nix to jsonnet well.
Me as well. As a haskell / ML programmer, it is extremely intuitive. It's non-innovative (In a good way). Literally it's just a functional scripting language.
I don't think the language is the issue here[1].
It seems inevitable to me that some of the design choices around immutability and isolation are going to result in a larger server image (both on disk and in memory) than if you are prepared to forgo those things. For most people that tradeoff is probably worth it but if you want something to run in an embedded server or with a very low disk footprint it's probably not right for you.
Around 20 years ago people who wanted to do this[2] used to make tiny immutable redhat servers by remounting /usr and a few other things read-only after boot so it's certainly doable but it's a lot more of a pain than what nix does and there is no process isolation and no rollback etc when things go wrong.
[1] ...or generally in fact but that's a matter of opinion and I know people feel differently about this.
[2] me for one, but others also.
It seems inevitable to me that some of the design choices around immutability and isolation are going to result in a larger server image (both on disk and in memory) than if you are prepared to forgo those things. For most people that tradeoff is probably worth it but if you want something to run in an embedded server or with a very low disk footprint it's probably not right for you.
Around 20 years ago people who wanted to do this[2] used to make tiny immutable redhat servers by remounting /usr and a few other things read-only after boot so it's certainly doable but it's a lot more of a pain than what nix does and there is no process isolation and no rollback etc when things go wrong.
[1] ...or generally in fact but that's a matter of opinion and I know people feel differently about this.
[2] me for one, but others also.
> Have there been any good efforts into getting rid of the language or providing an alternative?
Guix is conceptually similar to Nix but uses scheme.
Guix is conceptually similar to Nix but uses scheme.
It seems to be an issue with testing and debugging, rather than the language itself. The same issue would also be present if you could switch to any other language for configuration.
Nickel lang is such an effort. Id say the syntax is a mix of json and lua and aims for a non-touring complete program. It is still a bit early but it looks promising
No, Nickel is Turing-complete. That's been one of the characteristics intended to distinguish it from most other configuration languages from the start.
See the 'RATIONALE' document: https://github.com/tweag/nickel/blob/378ece30b3e3c0ab488f659...
See the 'RATIONALE' document: https://github.com/tweag/nickel/blob/378ece30b3e3c0ab488f659...
What do you want to replace it with? YAML?
The language won't go away and you should try to look at it for more than I don't like it.
The language won't go away and you should try to look at it for more than I don't like it.
No, writing this in any other language will definitely make for a much shittier Nix experience.
The problem isn't the language, the problem is that nixpkgs (and NixOS) are just huge.
The problem isn't the language, the problem is that nixpkgs (and NixOS) are just huge.
yes, https://github.com/garnix-io/garn
But weirdly there was little interest and they rebranded themselves around being a more general build system based on nix instead of what they originally said about being a nix reimplmentation in typescript.
I just finished moving all of my servers off NixOS and I'm finally breathing a sigh of relief.
Deterministic systems are a cool idea, but we're just not there yet. The headaches and pain involved in maintaining these systems and warping the software to obey are too great.
Everything in NixOS works, until it doesn't. And when it doesn't, woe be unto you.
Deterministic systems are a cool idea, but we're just not there yet. The headaches and pain involved in maintaining these systems and warping the software to obey are too great.
Everything in NixOS works, until it doesn't. And when it doesn't, woe be unto you.
As an opposing viewpoint, I migrated all my self hosted systems (physical machines in a data center) to use nixos and have truly reached self-hosting nirvana. Nothing breaks because I build everything before deployment, and when things do break during system build, it's fairly trivial to fix. I've successfully run all my own infrastructure on NixOS for about 5 years now. No downtime. Very easy to replace computers
What did you replace it with? I've been been using NixOS for a (non-critical) server which replaces another server that was "managed" by Ansible. Now that's it running nicely on NixOS, and while it's far from perfect, I'm really struggling to see how I would manage it in any other way. The experience is so much nicer now.
I use Proxmox on top of Debian. Surprisingly, I'm back to Bash scripts to set things up (because Ansible wouldn't allow me to orchestrate between a host and a guest - so if I have to change container config during setup, I'm back to scripts again, AND the complications of Ansible).
I basically build up Proxmox container templates, and then build upon those similar to how Docker does it (I don't use Docker because they don't allow you to specify your MAC address, so you can't control them from a separate LAN-based DHCP server - instead you have to map a bunch of ports on your host and then configure all external clients to match... so dumb).
I've basically gone full circle at this point:
- Docker
- LXD with Bash scripts
- LXD with a ton of Python
- NixOS
- Proxmox with Ansible
- Proxmox with Bash scripts (albeit much simpler and flatter than last time)
Everything is containerized and has its own IP address on the physical LAN, the templates can be regenerated with a simple script, important data is mapped to a host directory (/home/data/my-container, which gets backed up), and destroying and rebuilding an instance container is a cinch.
One really nice feature of this setup is that I can tear down and rebuild a template, launch a test container from that, copy the instance data in /home/data to the new container, make sure it works with the new stuff, and then launch it for real.
Now it doesn't matter what technology (container or VM) I use. Everything is a completely separate machine as far as the LAN is concerned, which greatly simplifies things.
Everything, from host to software to containers & VMs is built "deterministically" (i.e. deterministically enough) from the scripts. Rebuilding the whole thing (server and all) from scratch takes about an hour and a half. I just use the same set of scripts on all of my servers to make management easier. Hosts have minimal software and configuration, and guests do all the real work. Migrating is an rsync /home/data away.
I basically build up Proxmox container templates, and then build upon those similar to how Docker does it (I don't use Docker because they don't allow you to specify your MAC address, so you can't control them from a separate LAN-based DHCP server - instead you have to map a bunch of ports on your host and then configure all external clients to match... so dumb).
I've basically gone full circle at this point:
- Docker
- LXD with Bash scripts
- LXD with a ton of Python
- NixOS
- Proxmox with Ansible
- Proxmox with Bash scripts (albeit much simpler and flatter than last time)
Everything is containerized and has its own IP address on the physical LAN, the templates can be regenerated with a simple script, important data is mapped to a host directory (/home/data/my-container, which gets backed up), and destroying and rebuilding an instance container is a cinch.
One really nice feature of this setup is that I can tear down and rebuild a template, launch a test container from that, copy the instance data in /home/data to the new container, make sure it works with the new stuff, and then launch it for real.
Now it doesn't matter what technology (container or VM) I use. Everything is a completely separate machine as far as the LAN is concerned, which greatly simplifies things.
Everything, from host to software to containers & VMs is built "deterministically" (i.e. deterministically enough) from the scripts. Rebuilding the whole thing (server and all) from scratch takes about an hour and a half. I just use the same set of scripts on all of my servers to make management easier. Hosts have minimal software and configuration, and guests do all the real work. Migrating is an rsync /home/data away.
Funny, I can imagine bash is a real improvement on ansible. I once started with bash, switched to bash in ansible and then to "good style" ansible and then to a nixops cluster.
There wasn't really a good point between bash and nix.
There wasn't really a good point between bash and nix.
As someone who uses Proxmox without bash scripts and is scared of the day I have to re-work out all the config files I (often randomly through trial and error) changed:
Do you have any tips on how to get started? Do you simply make the change and then paste the commands needed into a script?
Also I assume you have a script to set up the (Proxmox) host machine?
I also have quite a few Proxmox specific things that I had to change (e.g. GPU pass-through) which seems to break your "Now it doesn't matter what technology (container or VM) I use" advantage.
Do you have any tips on how to get started? Do you simply make the change and then paste the commands needed into a script?
Also I assume you have a script to set up the (Proxmox) host machine?
I also have quite a few Proxmox specific things that I had to change (e.g. GPU pass-through) which seems to break your "Now it doesn't matter what technology (container or VM) I use" advantage.
Edit: I've made the repo public: https://github.com/kstenerud/proxmox-containers
Here's one of my host setup files (run this immediately after installing Proxmox from the iso):
Here's my basic Ubuntu container template:
Here's one of my host setup files (run this immediately after installing Proxmox from the iso):
DATA_SIZE=600GiB
# Switch to free mode
rm /etc/apt/sources.list.d/ceph.list
rm /etc/apt/sources.list.d/pve-enterprise.list
echo "deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription" >/etc/apt/sources.list.d/pve-no-subscription.list
# Setup locale
sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen
locale-gen
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US LC_ALL=en_US.UTF-8
apt update
apt dist-upgrade -y
apt install -y software-properties-common
add-apt-repository -y non-free
apt install -y vainfo intel-media-va-driver-non-free lm-sensors
apt install -y ansible pip python3-proxmoxer htop strace git
# Setup user data area
lvcreate -V ${DATA_SIZE} -T pve/data -n home_data
mke2fs -t ext4 -O ^has_journal /dev/pve/home_data
mkdir /home/data
echo "/dev/pve/home_data /home/data ext4 rw,discard,relatime 0 2" >>/etc/fstab
mkdir -p /mnt/containers/media
echo "//media/media /mnt/containers/media cifs uid=100000,gid=100000,ro,guest,x-systemd.automount 0 0" >>/etc/fstab
mkdir -p /mnt/containers/files
echo "//files/files /mnt/containers/files cifs uid=100000,gid=100000,ro,credentials=/etc/samba/credentials,x-systemd.automount 0 0" >>/etc/fstab
systemctl daemon-reload
mount -a
Here's my basic Ubuntu container template:
# ============
# Local config
# ============
TEMPLATE_IMAGE="ubuntu-24.04-standard_24.04-2_amd64.tar.zst"
INSTANCE_CT=200
INSTANCE_NAME=template-ubuntu
INSTANCE_MEMORY=2048
# ======
# Script
# ======
pveam download local ${TEMPLATE_IMAGE} || true
pct create $INSTANCE_CT local:vztmpl/${TEMPLATE_IMAGE} \
--hostname ${INSTANCE_NAME} \
--memory ${INSTANCE_MEMORY} \
--rootfs local-lvm:1 \
--net0 name=eth0,ip=dhcp,ip6=dhcp,bridge=vmbr0 \
--ostype ubuntu \
--start 1 \
--timezone host \
--features nesting=1 \
--unprivileged 1
# Use a fast mirror
pct exec $INSTANCE_CT -- sed -i 's/archive.ubuntu.com/ftp.uni-stuttgart.de/g' /etc/apt/sources.list
# Set up locale
pct exec $INSTANCE_CT -- sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen
pct exec $INSTANCE_CT -- locale-gen
pct exec $INSTANCE_CT -- update-locale LANG=en_US.UTF-8 LANGUAGE=en_US LC_ALL=en_US.UTF-8
# Bring everything up to date
pct exec $INSTANCE_CT -- apt clean
pct exec $INSTANCE_CT -- apt update
pct exec $INSTANCE_CT -- apt dist-upgrade -y
pct exec $INSTANCE_CT -- apt install -y software-properties-common curl
# Turn this into a template
pct stop $INSTANCE_CT
pct template $INSTANCE_CT
# ============
# Local config
# ============
TEMPLATE_CT=200
INSTANCE_CT=201
INSTANCE_NAME=template-plex
INSTANCE_MEMORY=2048
# ======
# Script
# ======
passthrough_gpu() {
local instance_id="$1"
echo 'lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.mount.entry: /dev/dri/renderD128 dev/dri/renderD128 none bind,optional,create=file
lxc.hook.pre-start: sh -c "chown 0:108 /dev/dri/renderD128"
' >> /etc/pve/lxc/${instance_id}.conf
}
pct clone $TEMPLATE_CT $INSTANCE_CT --full 1
pct resize $INSTANCE_CT rootfs 2G
pct set $INSTANCE_CT \
--hostname ${INSTANCE_NAME} \
--memory ${INSTANCE_MEMORY} \
--net0 name=eth0,ip=dhcp,ip6=dhcp,bridge=vmbr0
passthrough_gpu $INSTANCE_CT
pct start $INSTANCE_CT
pct exec $INSTANCE_CT -- apt update
pct exec $INSTANCE_CT -- apt dist-upgrade -y
# Install Plex
apt_add_key $INSTANCE_CT plexmediaserver https://downloads.plex.tv/plex-keys/PlexSign.key CD665CBA0E2F88B7373F7CB997203C7B3ADCA79D
apt_add_repo $INSTANCE_CT plexmediaserver "https://downloads.plex.tv/repo/deb public main"
pct exec $INSTANCE_CT -- apt update
pct exec $INSTANCE_CT -- apt install -y plexmediaserver
# Turn this into a template
pct stop $INSTANCE_CT
pct template $INSTANCE_CT
# ============
# Local config
# ============
TEMPLATE_CT=201
INSTANCE_CT=20001
INSTANCE_NAME=plex
INSTANCE_ADDRESS=99
INSTANCE_MEMORY=2048
# ===========
# Host config
# ===========
HOST_DATA=/home/data
HOST_BASE_UID=100000
HOST_BASE_GID=100000
# ======
# Script
# ======
pct clone $TEMPLATE_CT $INSTANCE_CT --full 1
pct set $INSTANCE_CT \
--onboot 1 \
--hostname ${INSTANCE_NAME} \
--memory ${INSTANCE_MEMORY} \
--net0 name=eth0,hwaddr=12:4B:53:00:00:${INSTANCE_ADDRESS},ip=dhcp,ip6=dhcp,bridge=vmbr0
# Mount the media dir
pct set $INSTANCE_CT -mp0 /mnt/containers/media,mp=/media
# Get the UID and GID of the plex user
pct start $INSTANCE_CT
PLEX_UID=$(pct exec $INSTANCE_CT -- id -u plex)
PLEX_GID=$(pct exec $INSTANCE_CT -- id -g plex)
PLEX_HOST_UID=$(($HOST_BASE_UID+$PLEX_UID))
PLEX_HOST_GID=$(($HOST_BASE_GID+$PLEX_GID))
pct stop $INSTANCE_CT
# Mount the Plex config dir
mkdir -p /home/data/${INSTANCE_NAME}/var-lib-plexmediaserver
chown -R ${PLEX_HOST_UID}:${PLEX_HOST_GID} ${HOST_DATA}/${INSTANCE_NAME}
chown -R ${PLEX_HOST_UID}:${PLEX_HOST_GID} ${HOST_DATA}/${INSTANCE_NAME}/var-lib-plexmediaserver
pct set $INSTANCE_CT -mp1 ${HOST_DATA}/${INSTANCE_NAME}/var-lib-plexmediaserver,mp=/var/lib/plexmediaserver
pct start $INSTANCE_CT
pct exec $INSTANCE_CT -- apt update
pct exec $INSTANCE_CT -- apt dist-upgrade -y
IP_ADDR=$(pct exec $INSTANCE_CT -- hostname -I | xargs)
set +x
echo "==========================================================="
echo "FOR FIRST TIME SETUP (no existing config):"
echo
echo "Go to https://www.plex.tv/claim/"
echo "pct exec $INSTANCE_CT -- curl -X POST 'http://127.0.0.1:32400/myplex/claim?token=PASTE_TOKEN_HERE'"
echo "Then, go to ${IP_ADDR}:32400"
echo "==========================================================="Maybe chuck that on GitHub/gist (or some other online repo) so people can potentially suggest stuff in an easy way? :)
Hmm OK. I already had it on github so I'll just make it public:
https://github.com/kstenerud/proxmox-containers
https://github.com/kstenerud/proxmox-containers
Thank you. I've been meaning to try out proxmox.
Proxmox really needs to get their cloud-init story together. If they supported cloud-init for LXC a lot of automation and setup issues would just go away…
I'll echo this. I can't imaging going back.
And yes, I have put a lot of blood sweat and tears into making things work in nix/NixOS. The thing that keeps me invested is once I get something working, it is far easier to keep it working. If nixpkgs updates break my things, I'm one git bisect away from figuring out what happened.
And yes, I have put a lot of blood sweat and tears into making things work in nix/NixOS. The thing that keeps me invested is once I get something working, it is far easier to keep it working. If nixpkgs updates break my things, I'm one git bisect away from figuring out what happened.
Same experience with Debian. Preaching religious, unquestionable stability but a soon as you install an out of tree software, everything falls apart in your hands.
Any Debian system I used fell apart on upgrades in some unrecoverable way and it was a giant pain in the ass.
The last NixOS stable release I deployed in ~3 hours onto ~30 VMs with only minor issues and everything continued to just work.
The last NixOS stable release I deployed in ~3 hours onto ~30 VMs with only minor issues and everything continued to just work.
I don't actually care about uncommon software stability on Debian. The hypervisor is Debian, so it runs very basic happy path software that always gets tested by the Debian team (and/or Proxmox team) for upgrade problems, and then all of my crazy-ass software runs in containers/VMs which are simple to destroy/rebuild if necessary.
The Proxmox team ensures that the hypervisor stuff works, and now I don't have to worry about a basic change to a nix file resulting in "Unknown entity flibblefrazzle" coming from some random place 18 levels deep in the bowels of the package system/os. It really got to the point where I was afraid to touch anything anymore.
Now I can run an up-to-date Plex! And Chrome Remote Desktop! Without spending 3 whole weekends knee deep in nix guts!
The Proxmox team ensures that the hypervisor stuff works, and now I don't have to worry about a basic change to a nix file resulting in "Unknown entity flibblefrazzle" coming from some random place 18 levels deep in the bowels of the package system/os. It really got to the point where I was afraid to touch anything anymore.
Now I can run an up-to-date Plex! And Chrome Remote Desktop! Without spending 3 whole weekends knee deep in nix guts!
The OP’s article mostly seemed to be about disk space size, specifically when creating a bunch of micro vms. As a NixOS personal dev user, I’m curious about your insights here. Do you have other issues besides the disk space one? I’ve always wanted to do a NixOS server but I only know enough nix to get by on a dev machine. I’m worried about the complexity in deploying a live server.
NixOS is EXTREMELY opaque and obtuse. The documentation also, since everything assumes that you already know everything.
I thought that I could just power through it, but there's just no end to the edge cases, bizarre magic, lack of useful error feedback (if any at all - the most common result is: nothing happens), and situations where you simply cannot do what you want to do.
So you either conform to their model and live with the limitations (and spend countless weekends debugging your builds), or you give up and move on.
I just got tired it all. I want to spend my time USING the computer, not setting it up. So now I use Debian, because everything includes a build for Debian and drivers for Debian. And the best part is: SOMEONE ELSE is maintaining it and keeping it current with the latest security fixes.
I thought that I could just power through it, but there's just no end to the edge cases, bizarre magic, lack of useful error feedback (if any at all - the most common result is: nothing happens), and situations where you simply cannot do what you want to do.
So you either conform to their model and live with the limitations (and spend countless weekends debugging your builds), or you give up and move on.
I just got tired it all. I want to spend my time USING the computer, not setting it up. So now I use Debian, because everything includes a build for Debian and drivers for Debian. And the best part is: SOMEONE ELSE is maintaining it and keeping it current with the latest security fixes.
There is a middle ground imo: run your application/services as containers on top of NixOS. Granted, this doesn’t work for all cases.
That's basically what I was doing. But it still became too stressful to manage them.
Proxmox has its own issues (they all do), but it's a much more inviting experience and nicer community.
Proxmox has its own issues (they all do), but it's a much more inviting experience and nicer community.
Fair enough. I actually run Proxmox too, but have most services running on NixOS as a VM. Best of both worlds haha :)
I do the same. Raw proxmox as the host, running mostly NixOS in the VMs
For me it came down to an epiphany: I'm doing all this careful setup on a tricky, esoteric platform to get reproducible builds, except actually, some of my containers have to run Debian because I can't build the software they run deterministically. So now I only sort-of have reproducible, immutable builds.
So I'm jumping through all these hoops, stepping so far out of the norm that I become the effective maintainer and debugger of all of this, and ... Why was I doing this again?
Oh yeah, deterministic builds and immutability. And I needed these ... why?
Turns out I can get a similar effect on a mainstream platform with some scripts and a little bit of discipline.
So I'm jumping through all these hoops, stepping so far out of the norm that I become the effective maintainer and debugger of all of this, and ... Why was I doing this again?
Oh yeah, deterministic builds and immutability. And I needed these ... why?
Turns out I can get a similar effect on a mainstream platform with some scripts and a little bit of discipline.
What didn't work in your NixOS servers?
> I was trying to bring NixOS to a bare minimum, which is an exercise similar to building containers with the bare minimum required for the software in the container to run. I think this is a worthy endeavour. I think we have all the tools in regular non-docker, non-kubernetes linux to get to a similar outcome, except we won’t need docker or kubernetes or whatever in this new land, thus removing quite a bunch of complexity from the systems we build.
> But doing it on top of NixOS currently feels like a bad path to take.
The author of this blog post might be interested in playing with not-os, another, much smaller OS built with Nix: https://github.com/cleverca22/not-os
Could be a decent source of inspiration!
> But doing it on top of NixOS currently feels like a bad path to take.
The author of this blog post might be interested in playing with not-os, another, much smaller OS built with Nix: https://github.com/cleverca22/not-os
Could be a decent source of inspiration!
(I'm the post author)
Thanks! I have to admit that I've had the itch to build my own NixOS-inspired system more than once, and I haven't done that because I just don't have time to dedicate to this among all the other projects I'm working on. I wasn't aware of not-os before, but I'll definitely dig into the code!
Thanks! I have to admit that I've had the itch to build my own NixOS-inspired system more than once, and I haven't done that because I just don't have time to dedicate to this among all the other projects I'm working on. I wasn't aware of not-os before, but I'll definitely dig into the code!
I don't know any other OS where you could even go on such a trip so easily! Figuring out why things are where and being able to disable them like this is pretty cool.
I was really surprised you were able to replace systemdMinimal with systemd in dbus though.
I thought it was there to break the cyclic dependency between systemd and dbus
I was really surprised you were able to replace systemdMinimal with systemd in dbus though.
I thought it was there to break the cyclic dependency between systemd and dbus
Depends on what you consider "easy"! Nix itself has a pretty high barrier to entry.
Personally I believe systems that start simple (e.g. Alpine) are easier to mess with. Plus you don't have to give up all benefits of declarative configuration; for example apk has a single file (/etc/apk/world) that defines the exact package set that needs to remain installed. You can edit it and run "apk fix", much like you can edit /etc/nixos/configuration.nix and rerun "nixos-rebuild switch". It's not as powerful as Nixos, but power (and complexity) always has a price.
Personally I believe systems that start simple (e.g. Alpine) are easier to mess with. Plus you don't have to give up all benefits of declarative configuration; for example apk has a single file (/etc/apk/world) that defines the exact package set that needs to remain installed. You can edit it and run "apk fix", much like you can edit /etc/nixos/configuration.nix and rerun "nixos-rebuild switch". It's not as powerful as Nixos, but power (and complexity) always has a price.
He didn't mention that nix uses a lot of ram. If your server is tiny and doesn't have swap enabled, running nix command s will make it unresponsive. Are there any nixos alternative that allow you to do system wide configuration from a single source similar to configuration.nix?
(I'm the post author)
As others said, I've moved away from doing nix builds on servers and into a less wasteful (if you're running multiple servers) approach of building once, deploying the bits into a cache, and making the servers fetch them from a cache. I've been slowly working on my own tool to make this workflow easier (nixless-agent, you can find an early version on GitHub).
As others said, I've moved away from doing nix builds on servers and into a less wasteful (if you're running multiple servers) approach of building once, deploying the bits into a cache, and making the servers fetch them from a cache. I've been slowly working on my own tool to make this workflow easier (nixless-agent, you can find an early version on GitHub).
Do you have a post on your setup? I'm relatively new to NixOS server management and I've been leaning on NixOps for the coordination bits. I'd love to see what you're doing as it sounds more elegant, efficient, and supported.
I don't have one yet, but I will definitely make one once I make a public release of nixless-agent. At the moment I use a combination of nixos-anywhere (for initial provisioning), colmena (mostly for quick testing/dev setups, because it lets me push things directly to a machine), and nixless-agent (still experimental and going through lots of testing). Ideally my final setup will be just a mix of nixos-anywhere and nixless-agent.
If you have a different machine with more ram and compute, you can use 'nixos-rebuild --target-host=<server> switch'. That does all the nix building on the local machine, and then just copies binaries and activates the built configuration on the remote machine.
I don't run nixos on my PC
you don't need to have nixos to have nixos-rebuild installed. Just nix
You need not perform your nix evaluation on the same device you are targeting. You can nix copy a system closure to your target and activate it, and there are a number of tools in the nix ecosystem to make this easier.
Granted, if you local machine is low on RAM, or isn't Linux, then you will be in trouble.
Granted, if you local machine is low on RAM, or isn't Linux, then you will be in trouble.
The best thing about NixOS is that it's extremely easy to replace systems. On my infrastructure, I have a centralized server where I store all my nix artifacts in /nix/store.
Then on boot for my other systems, they boot into a minimalist nixos image (via netboot) that (1) lookups up the hostname assigned to the system, (2) uses `nix copy` to move the closure of the current system from the main host where I store my builds to the local one, (3) switches into the system (explained below), and (4) then uses nixos's kexec support to switch into the proper kernel booting into the new system.
How to switch into a system: every nixos closure has a top level `bin/switch-to-configuration` script that nixos-rebuild suse. Just run `/nix/store/my-hash/bin/switch-to-configuration switch` and your system will silently be replaced by the new NixOS configuration. Very easy!
Then on boot for my other systems, they boot into a minimalist nixos image (via netboot) that (1) lookups up the hostname assigned to the system, (2) uses `nix copy` to move the closure of the current system from the main host where I store my builds to the local one, (3) switches into the system (explained below), and (4) then uses nixos's kexec support to switch into the proper kernel booting into the new system.
How to switch into a system: every nixos closure has a top level `bin/switch-to-configuration` script that nixos-rebuild suse. Just run `/nix/store/my-hash/bin/switch-to-configuration switch` and your system will silently be replaced by the new NixOS configuration. Very easy!
I haven't tried it personnally but Guix is similar with config in guile scheme. Have a look at the documentation [1][2].
Caveat: it is a gnu project so no proprietary stuff like firmwares and drivers included out of the box (but there is a community guix nonfree project available [3]). I believe that isn't a problem for virtual machine servers anyway.
[1] guix cookbook: https://guix.gnu.org/cookbook/en/html_node/index.html#Top
[3] guix manual: https://guix.gnu.org/manual/en/html_node/index.html#Top
[3] https://github.com/guix-users/guix-nonfree
Caveat: it is a gnu project so no proprietary stuff like firmwares and drivers included out of the box (but there is a community guix nonfree project available [3]). I believe that isn't a problem for virtual machine servers anyway.
[1] guix cookbook: https://guix.gnu.org/cookbook/en/html_node/index.html#Top
[3] guix manual: https://guix.gnu.org/manual/en/html_node/index.html#Top
[3] https://github.com/guix-users/guix-nonfree
The widely used nonfree guided channel is https://gitlab.com/nonguix/nonguix, not the one you linked.
Thanks for the clarification. Can't seem to be able to edit my comment.
Not similar to nix, but you can look into Yocto. You can use it to generate an OS. It is, much more involved than using nix, but suitable for low memory environments.
The problem with nix is that compiling a system uses lots of memory, but when deployed there's little overhead.
Like you would with Yocto, I just build my systems on a proper host then remotely deploy them.
Like you would with Yocto, I just build my systems on a proper host then remotely deploy them.
Yocto is also used by Lenovo to build their software appliances like XClarity Administrator.
Also it’s very famous and loved in embedded software circles.
Also it’s very famous and loved in embedded software circles.
Well, "loved" is not the word I would use. It's an improvement over the chaos before, where everyone used some other bespoke tool to build their BSPs. Yocto at least is some kind of standard now, but not a particularly good one (granted, it's a really difficult problem to solve). I don't know anyone actually enjoying working with Yocto.
Yocto is also hated for its unnessesary complexity in embedded software circles.
It won't necessarily do that if you deploy to your server from a remote machine over ssh
author does not use nix on his nixos and removed it, so no ram issue.
author does not activate new config on host machine, but deploys new host machines as needed.
he evals on build/dev/ci machine only.
author does not activate new config on host machine, but deploys new host machines as needed.
he evals on build/dev/ci machine only.
guix?
Doesn't it use nixos underneath?
guix does not use nixos underneath, it's a completely independent implementation (of the same underlying concepts)
guix is built on top of the nix daemon
It does have the nix daemon in-tree: https://git.savannah.gnu.org/cgit/guix.git/tree/nix
Not anymore, I think.
It’s a forked and tweaked version of the nix daemon. There have been some efforts to build a pure guile replacement, but afaik it’s not yet the default.
[deleted]
The post is much more interesting than the title suggests. Great deep dive into slimming a NixOS system.
I agree. The title made me expect a low-quality, clickbait article. The actual article provides a lot of educational value to users of Nix and those interested in trying to build the smallest Linux distribution that can run Nix. Despite using Nix (the package manager) for so long, I haven't considered what it would take to get this. The article answers the question perfectly.
Nix tries to be too many things with too little resources.
As a language it's plagued by very sparse documentation. As an package manager, it still has the same issues with documentation, (compounded by tools providing the same functionality superseding each other) and then it requires maintaining EVERY software package ever created.
I think the base idea is great, the language is a little weird (but manageable), the DX is unintuitive and badly documented.
The killer reason that made me abandon NixOs is packages not being up to date. I can't fix that and I don't want to spend my life writing my own derivations when the alternative is to use Arch repos + AUR for exotic stuff.
As a language it's plagued by very sparse documentation. As an package manager, it still has the same issues with documentation, (compounded by tools providing the same functionality superseding each other) and then it requires maintaining EVERY software package ever created.
I think the base idea is great, the language is a little weird (but manageable), the DX is unintuitive and badly documented.
The killer reason that made me abandon NixOs is packages not being up to date. I can't fix that and I don't want to spend my life writing my own derivations when the alternative is to use Arch repos + AUR for exotic stuff.
Couple of things spring to mind:
- didn't check if nixos uses it, but coreutil has a single binary mode (like busybox, a single binary is built and symlinks or hardlinks are used to provide all the commands); that might save some space
- instead of trying to strip down the system, maybe go the other way around: only include the command you need with its closure? closure computation is done in a few places (apparmor profile or systemd.confinement come to mind) and it should be possible to just copy whatever your server binary needs, your kernel (since microVM and not container), and run the binary directly as init (maybe with a simple wrapper that hardcodes network or whatsnot)
good luck!
- didn't check if nixos uses it, but coreutil has a single binary mode (like busybox, a single binary is built and symlinks or hardlinks are used to provide all the commands); that might save some space
- instead of trying to strip down the system, maybe go the other way around: only include the command you need with its closure? closure computation is done in a few places (apparmor profile or systemd.confinement come to mind) and it should be possible to just copy whatever your server binary needs, your kernel (since microVM and not container), and run the binary directly as init (maybe with a simple wrapper that hardcodes network or whatsnot)
good luck!
> - didn't check if nixos uses it, but coreutil has a single binary mode (like busybox, a single binary is built and symlinks or hardlinks are used to provide all the commands); that might save some space
It does. If you have coreutils from Nixpkgs installed you can check this with
It does. If you have coreutils from Nixpkgs installed you can check this with
basename "$(realpath "$(which ls)")"
> go the other way around
I'm not sure about just the one closure since the result needs to boot, but more generally that looks reasonable if your use case fits. There's a couple efforts, IIRC, mostly centered around router firmware where "real" nixos doesn't fit, ex. https://gti.telent.net/dan/liminix
I'm not sure about just the one closure since the result needs to boot, but more generally that looks reasonable if your use case fits. There's a couple efforts, IIRC, mostly centered around router firmware where "real" nixos doesn't fit, ex. https://gti.telent.net/dan/liminix
2 is very interesting to me. I'd love to see an example of a "scratch" vm build, in a similar spirit to scratch docker builds. Assume I have all my software and kernel built, what is the minimal filesystem that will boot? How much easier can it get when I am only targeting a VM?
This might be something you're interested in: https://appfs.rkeene.org/web/dir?ci=trunk&name=build/qemu
It's for running AppFS inside a VM on QEMU. It uses a statically linked Tcl (which AppFS is written in) to bring the system up.
It's for running AppFS inside a VM on QEMU. It uses a statically linked Tcl (which AppFS is written in) to bring the system up.
One technique employed by microvm.nix[0] is to mount the hosts /nix/store into the guest. This won't shrink the size of the system, but should allow it to be amortized across many different VMs.
I'm not sure how exploitable a read-only virtiofs share is, so this is perhaps not appropriate in some circumstances.
[0]: https://github.com/astro/microvm.nix
I'm not sure how exploitable a read-only virtiofs share is, so this is perhaps not appropriate in some circumstances.
[0]: https://github.com/astro/microvm.nix
I was thinking of a similar approach, but mounting /nix/store from the host into the guest will only work if you have a single guest.
For multiple guests, you should rely instead on: * A snapshot-able filesystem with the option to create clones (like ZFS). I think this is a great idea actually. * Exporting /nix/store via NFS, so you can have multiple writers (but this creates some tight coupling in that accidentally deleting stuff there may disrupt all guests).
For multiple guests, you should rely instead on: * A snapshot-able filesystem with the option to create clones (like ZFS). I think this is a great idea actually. * Exporting /nix/store via NFS, so you can have multiple writers (but this creates some tight coupling in that accidentally deleting stuff there may disrupt all guests).
mounting /nix/store from the host into the guest works with as many guests as you like - this is what the nixos tests framework does all the time, you just need a different "system" derivation for each (different) VM.
The problem with that is that the VM can see all the needless software, so if your goal is isolation, having a smaller closure is much better from a security point of view: if there's no coreutils bash etc then there's no risk of getting a shell spawned by an attack...
The problem with that is that the VM can see all the needless software, so if your goal is isolation, having a smaller closure is much better from a security point of view: if there's no coreutils bash etc then there's no risk of getting a shell spawned by an attack...
If you don't setup the mount, microvm.nix will create an erofs/squashfs image for /nix/store.
Good stuff! From the title I thought this would be about sysadmin woes, but this is interesting too :)
Another little addendum: you can trivially create bootable custom NixOS installer images with whatever configuration you want pre-applied[0].
[0] https://nix.dev/tutorials/nixos/building-bootable-iso-image
Another little addendum: you can trivially create bootable custom NixOS installer images with whatever configuration you want pre-applied[0].
[0] https://nix.dev/tutorials/nixos/building-bootable-iso-image
You can just use them as immutable live systems too, with the unfortunate requirement that it'll call your system an installer: https://github.com/NixOS/nixpkgs/issues/338738
Disk space is why one of my laptops doesn't run NixOS; 16GB soldered storage fit one generation passably, but it was clear that it wasn't gonna work (of course, that's with X and a browser; totally different use case). I vaguely recall that locales were shockingly large and I couldn't figure out how to trim them down. So that box runs Alpine now, which easily fits with no effort.
Have you tried putting /nix on an filesystem with compression? I've experienced >50% in storage savings with Bcachefs + zstd and dedupe
I too use Alpine. I like how you start small and add what you need. The option of having a read-only root FS is convenient on flaky SBCs.
The locale default seeing has been fixed by me over a year ago but 16GB is pretty small to fit two systems which you need to switch.
How "stable" are these kinds of slimming tweaks though? In a rolling release setup, aren't you going to have extraneous packages semi-regularly drag in all of Python or Perl accidentally? The setup might break and you'd not even notice?
I find the premise of a carefully re-compilable/re-creatable system very appealing, but not having a stable LTS style release rather incongruous. It takes a huge effort to get all the pieces working together - and if it's rolling and the sand is shifting/breaking underneath you it feels you never reach a meaningful stable system. Sure you can recreate your well tested working configuration, but the configurations is effectively immediately out of date and unmaintained once any packages are updated
I think this is why they effectively only target x64. I'm not a "distro guy" so maybe I'm missing something. It seems it'd be sensible to just 1-to-1 copy Ubuntu LTS package versions (+ patches) and build a NixOS "stable" version that can be patched and updated as people find issues
I find the premise of a carefully re-compilable/re-creatable system very appealing, but not having a stable LTS style release rather incongruous. It takes a huge effort to get all the pieces working together - and if it's rolling and the sand is shifting/breaking underneath you it feels you never reach a meaningful stable system. Sure you can recreate your well tested working configuration, but the configurations is effectively immediately out of date and unmaintained once any packages are updated
I think this is why they effectively only target x64. I'm not a "distro guy" so maybe I'm missing something. It seems it'd be sensible to just 1-to-1 copy Ubuntu LTS package versions (+ patches) and build a NixOS "stable" version that can be patched and updated as people find issues
> I think this is why they effectively only target x64
Nix and Nixpkgs is the best in class when it comes to cross platform & cross architecture support. It has good support for x86_64 / aarch64 /macOS / Linux. Getting Musl or static variants of existing packages just work for many packages. There's even some work on BSD / Windows support. Cross compiling is far easier to setup compared to other package managers. If anything, other projects should be copying what Nix is doing.
Nix and Nixpkgs is the best in class when it comes to cross platform & cross architecture support. It has good support for x86_64 / aarch64 /macOS / Linux. Getting Musl or static variants of existing packages just work for many packages. There's even some work on BSD / Windows support. Cross compiling is far easier to setup compared to other package managers. If anything, other projects should be copying what Nix is doing.
NetBSD's pkgsrc has always done extremely well for me for that.
I'm not sure how feasible it would be to compare nixpkgs and pkgsrc given how different they are, but I'd encourage people who need that to poke around at both and see which one feels like a better fit for their use case.
I'm not sure how feasible it would be to compare nixpkgs and pkgsrc given how different they are, but I'd encourage people who need that to poke around at both and see which one feels like a better fit for their use case.
Almost all of the changes flip an official setting. Those stay around for a long time and get a proper deprecation notice when they go away, so you won't be surprised. Replacing systemd-minimal with the full version may potentially cause some edge case issues, but it's the same package with more features enabled, so I wouldn't really expect any.
Nothing will break when the package gets updated as long as you keep to your specific release - backported changes are backwards compatible.
Nothing will break when the package gets updated as long as you keep to your specific release - backported changes are backwards compatible.
[deleted]
NixOS has stable releases built every 6 months.
What is the security update policy with NixOS? I've always had the impression that I would need to upgrade every 6 months to keep getting critical security fixes, which is unappealing compared to e.g. Debian + Nix
I know I can revert easily if there are problems when upgrading, but that doesn't really apply if security fixes only land in the new branch
I know I can revert easily if there are problems when upgrading, but that doesn't really apply if security fixes only land in the new branch
You get backported security fixes in release channels as well. Unless anything changed recently, there's no explicit guarantee around them, but the core packages typicaly land just as fast or faster then other distros. Keep in mind though that more esoteric software with a small number of users and auto-update disabled may lag a bit.
Maybe I'm missing something, but the only branch in github:nixos/nixpkgs I can see receiving fixes is the 24.05 branch getting fixes backported from unstable. The last commit I can see to the 23.11 branch is about 3 months ago
This would imply only 9 months of security patches before I would need to upgrade the server. That is of course a far less risky process with NixOS, so perhaps that is ok, but it is a lot more work than the 5 years you get (free) with Ubuntu/Debian
https://github.com/NixOS/nixpkgs/branches/active
This would imply only 9 months of security patches before I would need to upgrade the server. That is of course a far less risky process with NixOS, so perhaps that is ok, but it is a lot more work than the 5 years you get (free) with Ubuntu/Debian
https://github.com/NixOS/nixpkgs/branches/active
Updating on NixOS is so much less painful that doing an update every 6 months is actually viable. Also the update already contains large parts of your config and can be easily tested in a VM.
It's more like 7 months of patches. Release (n-1) gets EOL'd 1 month after release (n), and releases are 6 months apart (in May and November). So 23.11 would've been EOL in July 2024.
And since a release happens every 6 months, while you do have an extra month's window, you still have to upgrade... every 6 months.
And since a release happens every 6 months, while you do have an extra month's window, you still have to upgrade... every 6 months.
You're correct, it's 9 months max, there's no lts equivalent. The updates are much less risky, but in business context you'd really need to discuss the decision. (I'd still prefer that way)
Taking the time to delve in to things like this can teach you more about the OS you run than many other things can. It's not too unlike making a crunchgen OS, or even just a read-only OS with overlays. It's sometimes quite enlightening to see what will work with much less than we assumed it needs.
Having just installed the entirety of NetBSD on an i386 system (a 200 MHz Pentium), I see it weighs in at around 1 gig. But that's everything including X11 with WM and the toolchain (gcc 10). That's not bad, but it's really amazing how much of that isn't necessary for running the OS on a server. Particularly where you might want tiny VM images
Having just installed the entirety of NetBSD on an i386 system (a 200 MHz Pentium), I see it weighs in at around 1 gig. But that's everything including X11 with WM and the toolchain (gcc 10). That's not bad, but it's really amazing how much of that isn't necessary for running the OS on a server. Particularly where you might want tiny VM images
Have you taken a look at https://github.com/nix-community/srvos ? It’s a collection of profiles for servers by the Numtide folks.
I'm not sure if this approach is being used already or even viable, but there are good tools for the following:
- creating minimal OCI images from Nix packages
- creating microvms from OCI images
There are certainly some tradeoffs with this approach, but given that the author is trying to optimize for size, in addition to one of the primary benefits to this approach being a really clean, structured build/deploy loop, it seems like it could be worth exploring.
- creating minimal OCI images from Nix packages
- creating microvms from OCI images
There are certainly some tradeoffs with this approach, but given that the author is trying to optimize for size, in addition to one of the primary benefits to this approach being a really clean, structured build/deploy loop, it seems like it could be worth exploring.
Yes: https://nix.dev/tutorials/nixos/building-and-running-docker-...
They OCI images generated are super light weight.
They OCI images generated are super light weight.
Could you run ’nix store optimise’ before you remove ’nix’?
[deleted]
On borderline, thinking to learn nix. Few questions:
Can I define VM images with nix? Can those VM images be loaded into VirtualBox? Also, possible to do similar to build AMIs or other cloud VMs such as Hetzner where my "Dockerfile" is a nix file which defines the system to be built and then it has everything in place once built including tools, libraries, configuration and such?
Thoughts?
EDIT: Typos
Can I define VM images with nix? Can those VM images be loaded into VirtualBox? Also, possible to do similar to build AMIs or other cloud VMs such as Hetzner where my "Dockerfile" is a nix file which defines the system to be built and then it has everything in place once built including tools, libraries, configuration and such?
Thoughts?
EDIT: Typos
And for cloud/dedicated servers perhaps
https://github.com/nix-community/nixos-anywhere
(There's also a terraform module ) And for state changes https://github.com/serokell/deploy-rs Or Colmena /nixops/ and x other
For secret handling perhaps https://github.com/ryantm/agenix/ + https://github.com/oddlama/agenix-rekey
The ecosystem is in my experience very well fleshed out (7 yrs of use), as long as you don't require a knowledgebase/wiki/ up2date documentation, it's not been a issue for me since I could always fall back on Linux knowledge and just looking for how other distributions do x / how the thing itself is configured , and looking at how perhaps a existing nix module wraps that
(There's also a terraform module ) And for state changes https://github.com/serokell/deploy-rs Or Colmena /nixops/ and x other
For secret handling perhaps https://github.com/ryantm/agenix/ + https://github.com/oddlama/agenix-rekey
The ecosystem is in my experience very well fleshed out (7 yrs of use), as long as you don't require a knowledgebase/wiki/ up2date documentation, it's not been a issue for me since I could always fall back on Linux knowledge and just looking for how other distributions do x / how the thing itself is configured , and looking at how perhaps a existing nix module wraps that
Oh I am converted! Will try to setup system using nix instead of packer.
Yes but with an asterisk. Nixos generators linked elsewhere is a really great tool to take a given nix configuration and build it for consumption in many contexts (eg i use the same one to both build initial proxmox vm images and then also update those running images). The problem is that you will need a hardware-configuration.nix that caters to the end environment. Ive worked around this by pre-generating it and then copying it into the result but I wish there was a better, less manual, process.
Yep for sure! We even have a neat upload-ami cli tool:
https://github.com/NixOS/amis
https://github.com/NixOS/amis
That's awesome! I think all the missing pieces are in place. :)
But why would you run nixos as a micro vm os. It is not really designed for that purpose.
Nixos as a general purpose headless server OS with nix defined services, or as a desktop system with declarative and portable config is pretty neat.
Nixos as a general purpose headless server OS with nix defined services, or as a desktop system with declarative and portable config is pretty neat.
Any OS without stable releases and a security team is not a server OS.
https://status.nixos.org/
https://nixos.org/community/teams/security/
Welcome to NixOS, then! :)
Welcome to NixOS, then! :)
Then you only have Ubuntu, Fedora, Red Hat/CentOS/Alma/Rocky and Suse/OpenSuse.
and Debian.
You mean rot and forget LTS?
After the blog post the system was around 600mb, but what would be a reasonable image size for a server? Also is this bare metal or in the cloud (for the worker machines)?
This reminds me a little bit of trying to slim Docker images.
This is highly relevant for me, will probably end up using some of the same tricks they've done to trim the size of some of my micro VMs.
[deleted]
NixOS IME have few issues:
- a language friendly as Haskell, so while fit for purpose definitively it's not well digested by most, also by various longtime NixOS users;
- an unclear direction, there are countless of "side projects" and no clear path, most are not even indexed in a wiki page so you just discover by accident interacting with someone else or after a search;
- a terrible documentation probably due to the lack of a clear direction stated above.
The biggest "mean install" is true, but it's not that much impacting in the real world, NixOS real purpose is AVOIDING containers in designing an infra, not being wrapped by them or wrapping them and true x86 zero-overhead virtualization does not exists. So far only IBM Power Systems with AiX seems to have something nearly-zero-overhead built-in in the (big)iron.
IMVO that's the main point: most people, NixOS devs included, fails to see a world different than the current one. A possible answer could be keeping up the evolution of zfs and mirror some IllumOS features so we can have light paravirtualization thanks to zones on zfs clones. But as per NixOS most people fails to see a different storage than the most common today, a relict model from the '80s (does anyone remember the infamous "zfs is a rampant layer violation" phrase?). A damn real modern system should be: a SINGLE application, yes, the OS as a framework, development environment who produce a running system live out of itself. A coupled package-manager/installer/storage, because those are effectively a unique thing so we do not need a network of symlinks or containers, we have a storage behind that simply expose needed software pieces together, also a system who manage in-memory stuff the same way. Zfs was the first step in this direction, with boot environments, clones, zones glued by the Image Package System, lacking the language for a proper system integration, unfortunately almost nobody have taken care of that. NixOS and Guix System offer another piece, the language to integrate package management, installers but they lack the storage integration to generate a unique new system model.
Rediscover IllumOS (OpenSolaris) would bridge the gap providing all needed piece to start a new kind of distro and infra management for a FLOSS world where there is no need of monsters to deploy simple infra and those simple tools could scale at monster level, killing the commercial IT model of the giants and given the humanity the desktop model, the "pioneering internet" of interconnected personal system model a new start.
The lack of independent universities, big labs, is probably the root cause but as always good things tend to happen anyway sooner or later, it's the interim the bad part.
- a language friendly as Haskell, so while fit for purpose definitively it's not well digested by most, also by various longtime NixOS users;
- an unclear direction, there are countless of "side projects" and no clear path, most are not even indexed in a wiki page so you just discover by accident interacting with someone else or after a search;
- a terrible documentation probably due to the lack of a clear direction stated above.
The biggest "mean install" is true, but it's not that much impacting in the real world, NixOS real purpose is AVOIDING containers in designing an infra, not being wrapped by them or wrapping them and true x86 zero-overhead virtualization does not exists. So far only IBM Power Systems with AiX seems to have something nearly-zero-overhead built-in in the (big)iron.
IMVO that's the main point: most people, NixOS devs included, fails to see a world different than the current one. A possible answer could be keeping up the evolution of zfs and mirror some IllumOS features so we can have light paravirtualization thanks to zones on zfs clones. But as per NixOS most people fails to see a different storage than the most common today, a relict model from the '80s (does anyone remember the infamous "zfs is a rampant layer violation" phrase?). A damn real modern system should be: a SINGLE application, yes, the OS as a framework, development environment who produce a running system live out of itself. A coupled package-manager/installer/storage, because those are effectively a unique thing so we do not need a network of symlinks or containers, we have a storage behind that simply expose needed software pieces together, also a system who manage in-memory stuff the same way. Zfs was the first step in this direction, with boot environments, clones, zones glued by the Image Package System, lacking the language for a proper system integration, unfortunately almost nobody have taken care of that. NixOS and Guix System offer another piece, the language to integrate package management, installers but they lack the storage integration to generate a unique new system model.
Rediscover IllumOS (OpenSolaris) would bridge the gap providing all needed piece to start a new kind of distro and infra management for a FLOSS world where there is no need of monsters to deploy simple infra and those simple tools could scale at monster level, killing the commercial IT model of the giants and given the humanity the desktop model, the "pioneering internet" of interconnected personal system model a new start.
The lack of independent universities, big labs, is probably the root cause but as always good things tend to happen anyway sooner or later, it's the interim the bad part.
Re: unclear direction - I have never encountered another ecosystem more prone than Nix is to have people (and startups) trying to build their own little technical/commercial/social fiefdoms around some specific idea that X is the One True Way to do Y. Fragmentation is pervasive and detrimental and one of the major reasons I'm moving away from the tech, second only to governance.
Haskell is so much worse with its custom operators.
So a vibrant community is now bad? Also there where big improvements in the last years like freeform settings.
Try reading Debians Postgres Documentation and you get a sense what terrible doc is. Not only do they point to each withput instructions but they are also about 9.5 which is stone age old.
So a vibrant community is now bad? Also there where big improvements in the last years like freeform settings.
Try reading Debians Postgres Documentation and you get a sense what terrible doc is. Not only do they point to each withput instructions but they are also about 9.5 which is stone age old.
> So a vibrant community is now bad?
It's not bad experimenting, but a thing is experimenting and have a mainline with a clear path, another is having only experiments, mostly undocumented, hard to discover, and so on.
What most people want from a distro? Being rock solid and functional with the minimum effort for anything. NixOS offer that formally, well, stating it's legacy because Flakes are the future, but their are still not there, than NixOps/Disnix and so on, this does not play with this other etc etc etc. Essentially a generic user, not a developer, have hard time to craft a stable infra with stable tech and so most fear the change reducing the community to a nearly devs-only show and enterprise player show.
Debian docs in general aren't excellent but Debian is a well known dinosaurs so most of it's users already know it, there is no need to teach anything to most, those who do not know simply ask their side friend. NixOS while not new it's still unknown to many so it have to teach well from zero various people, who aren't interested in NixOS development itself but only in it's use, model, that will contribute with return of experience, translations, casual patches and no more. They might be seen as a burden to devs, but they are "the base" that grant any distro enough popularity to really thrive.
Ubuntu back than succeed over Debian because of that. They gives a sane, ready-to-work base. There is no need to a NixOS GUI installer or so, that's not a potential target, but there is a damn need to tell anyone "start with that, learn that, in 5 and 10 years it will be the same evolved properly" No one want to learn and relearn things.
It's not bad experimenting, but a thing is experimenting and have a mainline with a clear path, another is having only experiments, mostly undocumented, hard to discover, and so on.
What most people want from a distro? Being rock solid and functional with the minimum effort for anything. NixOS offer that formally, well, stating it's legacy because Flakes are the future, but their are still not there, than NixOps/Disnix and so on, this does not play with this other etc etc etc. Essentially a generic user, not a developer, have hard time to craft a stable infra with stable tech and so most fear the change reducing the community to a nearly devs-only show and enterprise player show.
Debian docs in general aren't excellent but Debian is a well known dinosaurs so most of it's users already know it, there is no need to teach anything to most, those who do not know simply ask their side friend. NixOS while not new it's still unknown to many so it have to teach well from zero various people, who aren't interested in NixOS development itself but only in it's use, model, that will contribute with return of experience, translations, casual patches and no more. They might be seen as a burden to devs, but they are "the base" that grant any distro enough popularity to really thrive.
Ubuntu back than succeed over Debian because of that. They gives a sane, ready-to-work base. There is no need to a NixOS GUI installer or so, that's not a potential target, but there is a damn need to tell anyone "start with that, learn that, in 5 and 10 years it will be the same evolved properly" No one want to learn and relearn things.
NixOS and flakes are orthogonal. Flakes are not a replacement for NixOS, they are simply a _lockfile specification_, akin to your requirements.txt or Cargo.toml or such. NixOS is not legacy.
I mean NixOS classic config instead of flakes.
I spend a weekend every so often defining the core of what I want next time I upgrade, but just find it so annoying I'm sure I won't use anything I've written until there's a major change in the ecosystem.