How I develop with Nix(ocharles.org.uk)
ocharles.org.uk
How I develop with Nix
https://ocharles.org.uk/blog/posts/2014-02-04-how-i-develop-with-nixos.html
57 comments
I was just trying to get Nix working after needing a version of base (GHC) newer than what I had installed.
The situation seems to have gotten a bit simpler with newer versions of the nix tools (this is of no fault of the author of this post, the post is more than a year old, their blog has great haskell stuff!).
Anyway, for haskell development you can just:
The situation seems to have gotten a bit simpler with newer versions of the nix tools (this is of no fault of the author of this post, the post is more than a year old, their blog has great haskell stuff!).
Anyway, for haskell development you can just:
1. Write cabal file (e.g., ./project.cabal)
2. cabal2nix --shell . >shell.nix
3. nix-shell
4. cabal sandbox init
5. cabal buildIs step 4 really required ? nix-shell already gives you a sandbox environment.
Also their user manual[0] is really good now.
[0]https://hydra.nixos.org/job/nixpkgs/trunk/manual/latest/down...
Also their user manual[0] is really good now.
[0]https://hydra.nixos.org/job/nixpkgs/trunk/manual/latest/down...
Recently I've been in various threads on HN complaining about how frustrating the Nix story is for development.
I do not read this and go "This is a great idea and process!" I read this and say, "I can see how this might be better than the existing haskell toolchain. Cabal-dev has always been a bit of a compromise." (Edit: showing how long it's been since I have used Haskell in anger, cabal-dev got deprecated in favor of native sandboxing about 2 years ago, my bad.) Haskell seems to be the common thread for nearly all users of Nix I've encountered in the wild.
Honestly, do other people read this and go, "This seems better than my current development toolchain?" ESPECIALLY if you're producing static binaries? As much as I criticize Maven, I'm not sure what this process buys you over maven except... well... what DOES it buy you over Maven dependency management? Unification with OS packages?
It's not immutable, reproducible deploys. The deploy side is easy to render immutable and reproducible with layerd, controlled, tagged containers. It seems to be a story that's more about the build and dev side of things. Heck, I showed this to a co-worker and he said, "Wow, cabal must be really bad if someone was so frustrated by it that their solution was to change their OS."
I don't mean to throw shade on the work that the Nix and NixOS people have been doing. I understand it has value to some people. It just seems very oversold as a solution to the problem of reproducible dev environments. And by adopting Nix, you MUST deploy your executable to something that can play nice with Nix's expectations, even if you are making a static executable. This seems like a heavy price to pay.
I do not read this and go "This is a great idea and process!" I read this and say, "I can see how this might be better than the existing haskell toolchain. Cabal-dev has always been a bit of a compromise." (Edit: showing how long it's been since I have used Haskell in anger, cabal-dev got deprecated in favor of native sandboxing about 2 years ago, my bad.) Haskell seems to be the common thread for nearly all users of Nix I've encountered in the wild.
Honestly, do other people read this and go, "This seems better than my current development toolchain?" ESPECIALLY if you're producing static binaries? As much as I criticize Maven, I'm not sure what this process buys you over maven except... well... what DOES it buy you over Maven dependency management? Unification with OS packages?
It's not immutable, reproducible deploys. The deploy side is easy to render immutable and reproducible with layerd, controlled, tagged containers. It seems to be a story that's more about the build and dev side of things. Heck, I showed this to a co-worker and he said, "Wow, cabal must be really bad if someone was so frustrated by it that their solution was to change their OS."
I don't mean to throw shade on the work that the Nix and NixOS people have been doing. I understand it has value to some people. It just seems very oversold as a solution to the problem of reproducible dev environments. And by adopting Nix, you MUST deploy your executable to something that can play nice with Nix's expectations, even if you are making a static executable. This seems like a heavy price to pay.
The thing about Nix is that it's the first project, as far as I know, that tries to solve the problem of package management with a fundamentally different approach. Most of the solutions to date were just okay: they worked with a tolerable degree of failure.
A solution that works good enough is often a serious obstacle for developing a solution that works really well. E.g. Unix works just good enough to preclude the adoption of a new paradigm in OS'es such as Plan 9.
Treating packages like immutable values sounds like a genuinely new idea. So what I think is that people don't go "this seems better than my current development toolchain", instead they think that the idea that underlies Nix might result in disproportionately more reliable tools in package management. That's worth fussing about.
Disclaimer: I am not associated with Nix in any way.
A solution that works good enough is often a serious obstacle for developing a solution that works really well. E.g. Unix works just good enough to preclude the adoption of a new paradigm in OS'es such as Plan 9.
Treating packages like immutable values sounds like a genuinely new idea. So what I think is that people don't go "this seems better than my current development toolchain", instead they think that the idea that underlies Nix might result in disproportionately more reliable tools in package management. That's worth fussing about.
Disclaimer: I am not associated with Nix in any way.
The first but not the last, there is also Guix the package manager and GuixSD the system distribution, which is written in Scheme, as well as all the package definitions. For lispers it is heaven.
I am sort of curious. What problems are you having with package managers that building decent packages wont solve ?
> As much as I criticize Maven, I'm not sure what this process buys you over maven except... well... what DOES it buy you over Maven dependency management?
I agree that using Nix doesn't make sense for application development where all the dependencies are resolvable within the language runtime. (Provided the runtime itself can be well-specified in the project configuration and the language's dependency management system defaults to exact immutable declarations.)
But throw in a dependency on Redis and Postgres, and you get into territory where traditional tools fall over pretty quickly, especially if different branches of your codebase depend upon different versions of Redis.
Personally Guix is more interesting to me as a way to keep my own stuff declarative rather than for sharing across a team of developers. But the cost doesn't seem too high if your application has external dependencies which traditional tools suck at; I've felt that pain and it isn't pretty.
I agree that using Nix doesn't make sense for application development where all the dependencies are resolvable within the language runtime. (Provided the runtime itself can be well-specified in the project configuration and the language's dependency management system defaults to exact immutable declarations.)
But throw in a dependency on Redis and Postgres, and you get into territory where traditional tools fall over pretty quickly, especially if different branches of your codebase depend upon different versions of Redis.
Personally Guix is more interesting to me as a way to keep my own stuff declarative rather than for sharing across a team of developers. But the cost doesn't seem too high if your application has external dependencies which traditional tools suck at; I've felt that pain and it isn't pretty.
> But throw in a dependency on Redis and Postgres, and you get into territory where traditional tools fall over pretty quickly, especially if different branches of your codebase depend upon different versions of Redis.
Surely you would just depend on the driver, which is most likely going to be written in your high-level language.
A better point may be languages which rely a lot on C libraries (think Ruby and Python), and where having something like pip won't magically install the ImageMagick dependency your software needs.
Surely you would just depend on the driver, which is most likely going to be written in your high-level language.
A better point may be languages which rely a lot on C libraries (think Ruby and Python), and where having something like pip won't magically install the ImageMagick dependency your software needs.
The driver's not much good on its own.
Traditional dependency systems force you to track the version of the server itself outside the project's configuration.
Traditional dependency systems force you to track the version of the server itself outside the project's configuration.
It's very strange to imagine that you'd host a DB in a conceptually local manner to application services. This couples things in a way that's very awkward for scaling. Even if you're not using containers, you almost never organize deploys this way outside of beta/dev environments.
And if you were to make such a decision, then you'd capture it in the closure required to run the code you built, so you'd sort of bake in the inclusion of those services into any deploy of your code, using Nix's philosophy.
Or am I misunderstanding you?
And if you were to make such a decision, then you'd capture it in the closure required to run the code you built, so you'd sort of bake in the inclusion of those services into any deploy of your code, using Nix's philosophy.
Or am I misunderstanding you?
What I'm saying is that you should have a nix package for your application and one for your database server. Whether these run on the same machine or not is a question for how they're deployed, but it should be the same package that's pulled in by the production recipe and the dev recipe in any case.
So? This kind of service often sits on a different box anyway in production.
I'm not sure what point you're making. Are you saying it's not important to run the same version of all your dependencies in development as in production?
I'm saying that in production, your application will likely connect to a database on a remote machine, which may or not have the same version as the one you packaged and you use for local development.
Nobody's used cabal-dev in Haskell-land for ages. Cabal has had native sandboxing since mid-2013 and Stack is a more recent alternative.
I'd argue Nix is nicer for everything but Haskell because the way Nix thinks about versions doesn't fit projects as neatly as one would think. As a Linux distro it's nifty (fully systemd-ified).
Cabal sandboxes have "just worked" since release in my experience (my own, plus hundreds of IRC users I've helped).
I'd argue Nix is nicer for everything but Haskell because the way Nix thinks about versions doesn't fit projects as neatly as one would think. As a Linux distro it's nifty (fully systemd-ified).
Cabal sandboxes have "just worked" since release in my experience (my own, plus hundreds of IRC users I've helped).
Sure, the last time I deployed Haskell for a paycheck it was still cabal-dev. I'm absolutely out of date there.
But I sort of struggle to imagine using NixOS to ship products, as it would mandate every dev maintain a comparable NixOS environment for development. In a world where even editor choice can be contentious and user-facing Linux only runs well on a small handful of machine types, this seems a little unfair.
But I sort of struggle to imagine using NixOS to ship products, as it would mandate every dev maintain a comparable NixOS environment for development. In a world where even editor choice can be contentious and user-facing Linux only runs well on a small handful of machine types, this seems a little unfair.
You don't have to switch to NixOS; you can run the nix package manager on any linux distribution or on OS X.
https://nixos.org/nix/
https://nixos.org/wiki/Nix_on_OS_X
A development team has to standardize on some things. Everyone might use their own text editor, but everyone can't use their own dependency management system.
https://nixos.org/nix/
https://nixos.org/wiki/Nix_on_OS_X
A development team has to standardize on some things. Everyone might use their own text editor, but everyone can't use their own dependency management system.
This does't actually answer any real criticism I leveled except...
> but everyone can't use their own dependency management system.
This is untrue. Different disciplines use different tools for dependency management and it's not clear to me that the world is crying out for a unification here. Golang developers use one, Nim developers use another. Java developers have different requirements still.
Let's look at a pair of closely related langauges, Clojure and ClojureScript. While we might want a uniform method for DECLARING dependencies, the specific mechanisms used to fetch those dependencies may (and possibly should?) be different.
> but everyone can't use their own dependency management system.
This is untrue. Different disciplines use different tools for dependency management and it's not clear to me that the world is crying out for a unification here. Golang developers use one, Nim developers use another. Java developers have different requirements still.
Let's look at a pair of closely related langauges, Clojure and ClojureScript. While we might want a uniform method for DECLARING dependencies, the specific mechanisms used to fetch those dependencies may (and possibly should?) be different.
I didn't say everyone globally, I said everyone on a development team, i.e. a group of people working together to build a product. Creating a reliable development environment for a team can be a PITA. That's why Vagrant, and more recently Docker, gained a lot of traction. People could use Nix to solve the same problem, like ris describes at https://news.ycombinator.com/item?id=10050681
I just have a hard time envisioning package management as the light and the hope for this problem, at least at the level of granularity offered.
As sciurus said, Nix can be used on other systems than NixOS. The package manager is the main idea, and the distribution is an extension of it to provide its benefits also for system configuration. The package manager doesn't touch your normal system file hierarchy at all; everything is isolated in Nix's own hierarchy; users can install packages without root; etc.
How does that actually answer the criticism though? You lose binary portability, you mandate Nix as a deploy AND dev target. You mandate that every development environment either is Nix-friendly OR that you're on the hook for writing said management.
And, as I've noted in the past, these packages people make as afterthoughts to product development are often broken. You end up welding your dev environment directly to your deployment environment and you complicate cross-platform builds quite a bit (closures cross-platform are, unless they have changed since early this year, not really a well-formed proposition).
And, as I've noted in the past, these packages people make as afterthoughts to product development are often broken. You end up welding your dev environment directly to your deployment environment and you complicate cross-platform builds quite a bit (closures cross-platform are, unless they have changed since early this year, not really a well-formed proposition).
Yup. This is pretty much what I do as well, but for Node and Python projects. The best part is that I develop in exactly the same environment that the app has in deployment.
I use nix with python at work, but I haven't yet used it (much) with our node libraries. `npm2nix` was a little clunky in my experience, generating an enormous expression for each package with no code reuse. Is it easier in practice?
Yeah, I use npm2nix a lot, and it does produce very large files (a few thousand lines of code). But in practice it's fine. I very rarely look at the generated code. If I need to update a dependency, I change my package.json file, run npm2nix and start a new nix-shell.
I also wrote a bit of nix code to read the package.json file and figure out what the direct dependencies are, which keeps derivations for my own packages simple.
I also wrote a bit of nix code to read the package.json file and figure out what the direct dependencies are, which keeps derivations for my own packages simple.
[deleted]
Seems like it would be perfect for declarative deployments!! So neat!
That's kind of the whole point of nix-os. Take the nix package manager and describe your entire OS decoratively. With atomic upgrades and rollbacks!
How does this compare to Docker?
So, nix actually tries to solve the problem of "what are your dependencies, how are they declared, how are interdependent parts related to each other?" where docker just goes "screw it. nobody's ever going to be able to tame this tangle. let's just throw it all in behind an os container and let it do what it wants"
I've recently nixified the (really very complicated) deployment of our main webapp at work and I now have the entire setup of the webapp defined in about 5 .nix files. That's a really relaxing place to be in (compared to how things were before).
I've recently nixified the (really very complicated) deployment of our main webapp at work and I now have the entire setup of the webapp defined in about 5 .nix files. That's a really relaxing place to be in (compared to how things were before).
That was a brilliant description of Docker that reflects precisely how I feel.
The appeal of Docker is that containers give you repeatable deployments. Once you have an image, you can deploy it in a well-defined way. But it punts on the question of how you build the image.
Nix provides repeatable deployments at the package level. The destination environment can be a user's environment, a container, a VM, a cloud instance, an NixOS machine, an HTTP server or whatever.
Once you're using nix, I don't see any point in using Docker. If you're running Mesos or something like that, it might be worthwhile to build containers for deployment, but if all you want is your packages on a server, Nix gives you that out of the box.
Nix provides repeatable deployments at the package level. The destination environment can be a user's environment, a container, a VM, a cloud instance, an NixOS machine, an HTTP server or whatever.
Once you're using nix, I don't see any point in using Docker. If you're running Mesos or something like that, it might be worthwhile to build containers for deployment, but if all you want is your packages on a server, Nix gives you that out of the box.
I wonder what Joyent and/or @bcantrill would say about that? It'd be super-cool to be able to run nix deployments in a Triton (instance|container|whateveritis)!
Yeah, Nix on SmartOS would be awesome. With the right filesystem jiggery, the nix store could be shared across zones. That would make the whole system faster and more efficient. Also, specifying a Manta job via next would be amazing.
I've brought it up a few times with Joyent folk, but they haven't looked at it seriously. I think they're so spooked by Docker that they can't pay attention to anything else.
I've brought it up a few times with Joyent folk, but they haven't looked at it seriously. I think they're so spooked by Docker that they can't pay attention to anything else.
Docker lets you run arbitrary N to M for Hardware to Software ala virtualization, with low enough overhead. Does Nix have a story for this?
I suppose so. Nix allows me to run N services on M machines arbitrarily with no virtualization and no overhead.
My team builds a distributed application. We have nix expressions that define packages for all the parts of the application, and systemd services to run them on NixOS. Using those packages:
My team builds a distributed application. We have nix expressions that define packages for all the parts of the application, and systemd services to run them on NixOS. Using those packages:
• Our production deployment is defined by some Nix expressions that assign the services to different machines in the cluster.
• Our staging deployment is a Nix expression that runs all the services on a single machine.
• Our development deployment is a Nix expression that creates an environment where all dependencies are available and the application can be executed by hand.
i think ris's description pretty much covers it.
What is cool though is that containers are supported natively in nixos (through systemd) https://nixos.org/releases/nixos/14.12/nixos-14.12.374.61adf...
Also, quite similar to 'cloud-config' I guess, there's a way to declaritively describe deployments to your favorite Cloud Provider TM https://nixos.org/nixops/
As the world is moving to declarative deployments, and once the docker hype settles, I think operating systems like NixOS will prevail as they're a great solution to the problem.
What is cool though is that containers are supported natively in nixos (through systemd) https://nixos.org/releases/nixos/14.12/nixos-14.12.374.61adf...
Also, quite similar to 'cloud-config' I guess, there's a way to declaritively describe deployments to your favorite Cloud Provider TM https://nixos.org/nixops/
As the world is moving to declarative deployments, and once the docker hype settles, I think operating systems like NixOS will prevail as they're a great solution to the problem.
NixOS is the whole OS. It doesn't use the kernel sandboxing at all, for itself at least, any more than any other distro does. Conceivably it could be used to replace CoreOS, though I'm sure that while at a high level that makes perfect sense there's an arbitrary number of incidental details that make that impractical, as is the way of these things.
It actually has kernel sandboxing built in. You can run nix expressions in linux containers. https://nixos.org/releases/nixos/14.12/nixos-14.12.374.61adf...
Much better and not restricted to only containers.
See also NixOps.
nix-shell is a killer tool. A generic virtualenv that doesn't duplicate dependencies if you use it for multiple projects.
For me it has a few too many annoying shell quirks for me to use daily (try a ctrl-c inside a psql prompt - always screws up my screen session for some reason...)
Are you using --pure? I'd imagine these problems are from lost environment variables, but only --pure should clobber those.
I see people mentioning it on other threads, but is anyone usigin Guix? Or the dmd daemon system? I am curious because the GNU package set is quite limtied for obvious reasons, but I still would love to work my way up to a homegrown OS or fork where I add the non-GNU-compliant apps and build my own master OS for the elitist Lispter (see what I did there) I aspire to be.
GUIX has non-gnu stuff available too.
Oh I am aware, I just did not want to get in to the OSI and GNU spectrum of licensing.
I was going to say Debianesque, but I kept deleting whatever I wrote in its place because I just thought it would lead to off-topic trolling.
In short: I wanna build my own Arch Linux, and Arch User Repository, but using Guix. We shall see if I ever do that as I promised myself in the next 5-10 years. Haha.
I was going to say Debianesque, but I kept deleting whatever I wrote in its place because I just thought it would lead to off-topic trolling.
In short: I wanna build my own Arch Linux, and Arch User Repository, but using Guix. We shall see if I ever do that as I promised myself in the next 5-10 years. Haha.
I love Nix, but I wish there were a way to easily install packages without having sudo access. Programs like pip only need a '--user' flag, while Nix requires you to use chroot magic[1] to be able to install packages in your home directory.
[1] https://nixos.org/wiki/How_to_install_nix_in_home_(on_anothe...
[1] https://nixos.org/wiki/How_to_install_nix_in_home_(on_anothe...
I sort of "version" or segregate my $HOME directory into a separate Workspaces directory in my real home for which I then have a script to activate a particular work space. For the most part, this script just sets $HOME and execs a new /bin/bash. This works pretty well. Most everything respects the new $HOME and works from there (so far only SSH and Java seem to ignore this to which I think I could fix Java by setting $JAVA_OPTS and explicitly setting the user.home property). On OS X I can install a separate copy of homebrew for each work space and have only the packages in each work space that I need or want. I was really looking forward to doing the same thing with Nix when I recently switched to Linux at my new job, but was very disappointed to find out that it really wanted to be installed somewhere in the root file system. Thanks for the link, I hadn't come across that one yet. I might give that a try.
http://wiki.ocharles.org.uk/Nix
The official documentation also describes how to work with Cabal/Haskell:
https://nixos.org/nixpkgs/manual/#users-guide-to-the-haskell...