Automate Project Environments with Devbox and Direnv (2023)(jetify.com)
jetify.com
Automate Project Environments with Devbox and Direnv (2023)
https://www.jetify.com/blog/automated-dev-envs-with-devbox-and-direnv/
54 comments
I'm in the same boat here. I'm very not over the Nix learning curve, so I'm left to conclude that it's actually not easy for some projects.
Last time I tried to use Nix to build a dev environment, it did not go well. I wound up in a Turing tarpit of dependencies, as my Python program needed natively compiled libraries and other things to even get off the ground. I kept having to pull in more of the Nix environment/OS to achieve functional isolation from the parent Ubuntu OS. I also ran into problems with online docs and advice skewing wildly from what the CLI options required of me (probably a skill issue).
And that's when it struck me. Docker has pretty much raised the bar for the developer experience. It can achieve the same amount of isolation for far fewer keystrokes, in less time, and (with docker-compose) has a sharable configuration for others. At the end of the day, a container is just a well isolated process whether it's a webserver or a user shell. And really, that's what you need if you're trying to solve repeatability and isolation. As a bonus, if your software is going to deploy as a container, you're that much closer to emulating the production environment (read: fewer bugs).
I like the Direnv concept and have nothing against Nix or folks trying to share the NixOS experience with others. But from a DevEx perspective, my expectations are set at "as easy or easier than Docker."
Last time I tried to use Nix to build a dev environment, it did not go well. I wound up in a Turing tarpit of dependencies, as my Python program needed natively compiled libraries and other things to even get off the ground. I kept having to pull in more of the Nix environment/OS to achieve functional isolation from the parent Ubuntu OS. I also ran into problems with online docs and advice skewing wildly from what the CLI options required of me (probably a skill issue).
And that's when it struck me. Docker has pretty much raised the bar for the developer experience. It can achieve the same amount of isolation for far fewer keystrokes, in less time, and (with docker-compose) has a sharable configuration for others. At the end of the day, a container is just a well isolated process whether it's a webserver or a user shell. And really, that's what you need if you're trying to solve repeatability and isolation. As a bonus, if your software is going to deploy as a container, you're that much closer to emulating the production environment (read: fewer bugs).
I like the Direnv concept and have nothing against Nix or folks trying to share the NixOS experience with others. But from a DevEx perspective, my expectations are set at "as easy or easier than Docker."
Maybe the author is confusing direnv and devenv, but I'm not sure.
https://devenv.sh
https://direnv.net/
https://devenv.sh
https://direnv.net/
I don't think so. The author accurately describes Direnv and also makes a big deal out of writing .envrc.
The author might have confused a few things, I don't think direnv and Nix are related. I've used direnv for years without touching Nix.
Direnv and Nix are related in the sense that direnv has a `use flake` command that you can put in your .envrc to load the devShell defined in the project's flake.nix file. It's essentially just an automatic entry to the project's Nix-native devShells.
Edit: this is the only thing I use direnv for - I haven't used direnv without touching Nix.
Edit: this is the only thing I use direnv for - I haven't used direnv without touching Nix.
The author is definitely confused. Direnv and Nix are not related.
Made me wonder if the article was GPT-generated and they didn't catch that particular hallucination
Hey I'm the author of this blog post. Definitely not GPT-generated, when I was exploring direnv, it was at the same time as I was exploring nix, so the two seemed adjacent in my mind. Admittedly, I should have phrased it better and not confused myself and anyone who read the blog post. What I tried to convey in the blog post was that, trying to setup isolated environments with Nix and automated env switching with direnv, requires investing time in learning about those tools. Devbox abstracts the Nix part and `devbox generate direnv` abstracts the direnv part.
Understood. Apologies if my comment made you feel bad, I hate when people do it to me and I should have the same respect for you.
Direnv is very useful for switching between projects, we have an article:
https://nixos.asia/en/direnv
But if you are comfortable with Nix you don't need Devbox per se, as you can get by with using Flakes. Here are some examples -- just `git clone` and run `direnv allow` (once), then you are put in the corresponding devShell
Haskell: https://github.com/srid/haskell-template
Rust: https://github.com/srid/rust-nix-template
home-manager: https://github.com/juspay/nix-dev-home
Dioxus: https://github.com/srid/dioxus-desktop-template
But if you are comfortable with Nix you don't need Devbox per se, as you can get by with using Flakes. Here are some examples -- just `git clone` and run `direnv allow` (once), then you are put in the corresponding devShell
Haskell: https://github.com/srid/haskell-template
Rust: https://github.com/srid/rust-nix-template
home-manager: https://github.com/juspay/nix-dev-home
Dioxus: https://github.com/srid/dioxus-desktop-template
Here's an open source "Uber" project, written largely in Haskell and heavily uses Nix/ direnv:
https://github.com/nammayatri/nammayatri
There's dedicated direnv configuration for backend and frontend development: checkout `.envrc.backend` and `.envrc.frontend` files. They both use `use flake`, specifying the appropriate devShell argument.
https://github.com/nammayatri/nammayatri
There's dedicated direnv configuration for backend and frontend development: checkout `.envrc.backend` and `.envrc.frontend` files. They both use `use flake`, specifying the appropriate devShell argument.
I started messing around with Devbox a few weeks ago and I like it as a faster alternative to toolbox/distrobox containers. I've been adding my environment variables directly into the devbox.json file as normal EXPORT statements. From the article tho, I couldn't grasp what Direnv adds to this. Same Devbox shell, different environment variables per directory?
With direnv you can activate your shell automatically by changing your directory. This is pretty useful if you have multiple projects using devbox, or a monorepo type setup with a few devbox.json files in your subfolders
I try to contribute a lot, which has resulting in me adding .direnv and .envrc to my global gitignores (for other Nix+hm folks[1]). .direnv for obvious reasons, it's a cache, and doesn't frequently show up in project gitignores.
I add .envrc to not only avoid committing it to projects that don't use it, but also because it turns out that it's still pretty environment dependent. There's 100 different ways to skin the dependency problem: Nix, pipenv, nvm, rustup, etc. There's no telling which a contributor prefers to use, so .envrc doesn't belong in git IMHO. What could be committed is something like .envrc-flake (which can then be sourced into .envrc).
As for needing a flake in the repo, I have found a workaround. `use flake` accepts parameters, including the path to the flake. I have a bunch of shells[2] that I can import with e.g. `use flake ~/Nix#rust`.
[1]: https://codeberg.org/jcdickinson/nix/src/branch/main/home/co... [2]: https://codeberg.org/jcdickinson/nix/src/branch/main/flake.n...
I add .envrc to not only avoid committing it to projects that don't use it, but also because it turns out that it's still pretty environment dependent. There's 100 different ways to skin the dependency problem: Nix, pipenv, nvm, rustup, etc. There's no telling which a contributor prefers to use, so .envrc doesn't belong in git IMHO. What could be committed is something like .envrc-flake (which can then be sourced into .envrc).
As for needing a flake in the repo, I have found a workaround. `use flake` accepts parameters, including the path to the flake. I have a bunch of shells[2] that I can import with e.g. `use flake ~/Nix#rust`.
[1]: https://codeberg.org/jcdickinson/nix/src/branch/main/home/co... [2]: https://codeberg.org/jcdickinson/nix/src/branch/main/flake.n...
This is the way. We should make it a standard practice for oss projects that one is able to git clone, direnv allow and build the project.
Regardless of the distro or even OS. An easy thing to say, but hard to execute on actual projects with patched versions of cPython and some obscure in-house build tool.
Regardless of the distro or even OS. An easy thing to say, but hard to execute on actual projects with patched versions of cPython and some obscure in-house build tool.
I would love for an ARCHITECTURE.md to become standard as well... I've definitely skipped contributing to a lot of projects because I'd have to basically have to reverse engineer the project, configuring a dev environment and add a lot of breakpoints to even begin to understand the structure
I wouldn't put an .envrc file in source control. This seems like an anti-pattern.
I think it should be standard practice that you can pull and run a project without configuring environment variables. I agree with that. The defaults should work.
This is not always possible (when you have something that absolutely relies on a third party thing, which you need an environment variable for) - in which case you should be greeted with a helpful error telling you what environment variables you need to set up and why.
I think it should be standard practice that you can pull and run a project without configuring environment variables. I agree with that. The defaults should work.
This is not always possible (when you have something that absolutely relies on a third party thing, which you need an environment variable for) - in which case you should be greeted with a helpful error telling you what environment variables you need to set up and why.
It is absolutely not an anti pattern to check Direnv's `.envrc` in to source control, and if you don't then you're passing up on much of Direnv's value proposition.
Obviously don't put secrets in there. For secrets and overrides you can use separate a `.env` file which is not checked in, and source it in `.envrc` with `dotenv_if_exists`
Obviously don't put secrets in there. For secrets and overrides you can use separate a `.env` file which is not checked in, and source it in `.envrc` with `dotenv_if_exists`
The moment you commit an envrc file, you disallow the developer from using direnv for that directory to configure stuff related to their machine. It is an anti pattern, it was never meant to be committed. Put an envrc.example in repo if you want, or an envrc.project in there and tell the dev to load it (load_if_exists in direnv)
You can use direnv in conjunction with sops or your favorite cloud-centric secrets management utility to automate secrets handling without writing any plaintext secrets (or any secrets at all, if you want) to version control.
.envrc is a whole bash script, so it can invoke command line tools that fetch secrets over the network or decrypt them from the disk.
.envrc is a whole bash script, so it can invoke command line tools that fetch secrets over the network or decrypt them from the disk.
Yep that too
This is an awful idea.
It's very hard to define in general what it means to build a project, what are the expected inputs and outputs. Not every build aims to produce the same artifacts, not using the same inputs etc.
What you want could be based on some typical practice in a particular field with a particular project size or structure, but this doesn't transfer well to other fields.
It's very hard to define in general what it means to build a project, what are the expected inputs and outputs. Not every build aims to produce the same artifacts, not using the same inputs etc.
What you want could be based on some typical practice in a particular field with a particular project size or structure, but this doesn't transfer well to other fields.
As long as it doesn't interfere with supporting normal, non container builds, sure.
What interferes with supporting “normal” builds is the effort required to actually support them. If that was a given, there’d be no justification for us having gotten ourselves in this mess in the first place. The realities of increasingly choose-your-own-adventure nature of Linux have come home to roost, and people are quite rightfully wanting to spend their effort elsewhere.
I say this as someone that was using Gentoo before lxc even existed. So this isn’t code for “I don’t get it”.
I say this as someone that was using Gentoo before lxc even existed. So this isn’t code for “I don’t get it”.
What about non Linux operating systems? Those exist and Open Source software written for them, developed on them exists.
Been using Devbox recently both at work and for personal projects and I can only say good things.
It abstracts away the complexity of nix, defining the dependencies in a simple json with a lock file, the shell starts quite fast and from then on you have everything under your PATH, and it works the same in macOS and Linux.
Also, as a benefit, by tracking the JSON and lockfile, the CI can work with the exact same environment. I highly recommend giving it a try.
Also, as a benefit, by tracking the JSON and lockfile, the CI can work with the exact same environment. I highly recommend giving it a try.
Also, as a benefit, by tracking the JSON and lockfile, the CI can work with the exact same environment.
This benefit is also possible under both Nix flakes (inherently) and vanilla Nix. The latter takes mild deliberate effort to not use channels-based import syntax but instead a fetcher with a pinned URL argument.
This benefit is also possible under both Nix flakes (inherently) and vanilla Nix. The latter takes mild deliberate effort to not use channels-based import syntax but instead a fetcher with a pinned URL argument.
How does it compare to docker? Does it run stuff on baremetal?
> despite it being a very useful tool, its setup requires investing time and getting over the Nix ecosystem's learning curve.
They mention Nix but I guess they are referring to the use_nix integration built-in to direnv? There is a fairly detailed (but old) comparison in https://github.com/direnv/direnv/wiki/Nix#some-factors-to-co... and in the meantime lorri and nix-direnv are the only ones still maintained. But you aren't limited to Nix when using direnv, for example there is nvm. https://github.com/direnv/direnv/wiki/Node#using-nvm
Actually, the post's solution, besides being built on top of direnv, uses Devbox, which itself is built on top of Nix... https://www.jetify.com/devbox/docs/faq/#how-does-devbox-work So they are saying they are outside the Nix ecosystem when in fact they are in it. It is yet another set of nice scripts on top of Nix.
They mention Nix but I guess they are referring to the use_nix integration built-in to direnv? There is a fairly detailed (but old) comparison in https://github.com/direnv/direnv/wiki/Nix#some-factors-to-co... and in the meantime lorri and nix-direnv are the only ones still maintained. But you aren't limited to Nix when using direnv, for example there is nvm. https://github.com/direnv/direnv/wiki/Node#using-nvm
Actually, the post's solution, besides being built on top of direnv, uses Devbox, which itself is built on top of Nix... https://www.jetify.com/devbox/docs/faq/#how-does-devbox-work So they are saying they are outside the Nix ecosystem when in fact they are in it. It is yet another set of nice scripts on top of Nix.
My experience of Devbox on Linux has been highly disappointing. I gave it a good go, had it running on my main project from February to May.
In case you hadn't realised, the very concept of having two sets of binary distributions on one machine, vying for superiority and the correct version of glibc... is fraught.
Most of my use was with rails projects, and I can't recommend it.
Coupled with an abstraction that tries to save you from Nix, but almost entirely fails, you end up with a bloated hellscape where every time you load your project it will unnecessarily reinstall your packages and several times an hour it will have forgotten curl exists and so you have to manually reinstall curl (not-so-slowly increasing your /nix folder's size), every week or two a new version of devbox completely changes the workarounds you need to do, and don't try to garbage collect nix or it will delete vital files, and you end up scrubbing it all and starting again.
In python, it overrode the path so I couldn't get it to reliably use the binaries in the venv. Pip and Python were using packages in different places and I couldn't get them to converge for love nor money.
The devbox team were great and really tried to get things working, but in the end I couldn't get it to work with enough stability to properly recommend it to my team, and if I wanted it to half-work for any substantial length of time I had to lock to a version of devbox.
Obviously, ymmv, please do give it a try, it's an impressive project. But my view is that it's trying to do something that is very very hard, and for that you need a very clever solution. And this is a very clever solution, with very clever bugs, and so it's not something I'd recommend jumping into with both feet.
In case you hadn't realised, the very concept of having two sets of binary distributions on one machine, vying for superiority and the correct version of glibc... is fraught.
Most of my use was with rails projects, and I can't recommend it.
Coupled with an abstraction that tries to save you from Nix, but almost entirely fails, you end up with a bloated hellscape where every time you load your project it will unnecessarily reinstall your packages and several times an hour it will have forgotten curl exists and so you have to manually reinstall curl (not-so-slowly increasing your /nix folder's size), every week or two a new version of devbox completely changes the workarounds you need to do, and don't try to garbage collect nix or it will delete vital files, and you end up scrubbing it all and starting again.
In python, it overrode the path so I couldn't get it to reliably use the binaries in the venv. Pip and Python were using packages in different places and I couldn't get them to converge for love nor money.
The devbox team were great and really tried to get things working, but in the end I couldn't get it to work with enough stability to properly recommend it to my team, and if I wanted it to half-work for any substantial length of time I had to lock to a version of devbox.
Obviously, ymmv, please do give it a try, it's an impressive project. But my view is that it's trying to do something that is very very hard, and for that you need a very clever solution. And this is a very clever solution, with very clever bugs, and so it's not something I'd recommend jumping into with both feet.
We had a similar experience with devenv. It didn't really prevent us from needing to learn nix, and then we had two things to deal with instead of one. We eventually switched back to just nix.
We're pretty happy devenv users at work, despite an annoying bug (with devenv in particular, although it's technically a Nix bug).
One difference that may have been decisive for our success is that when I selected devenv, my goal was not to avoid Nix at all, but rather to choose a nice convention for defining our project-specific Nix environments. I chose devenv because I trust the author's technical leadership (and it also has a decent community of contributors), it is very Nix-forward for this type of tool (most of defining your environment can be done in Nix, and you can even use it just as a Nix library rather than an executable tool (which we do, for one project)), it supports flakes in a first-class way, and it's built on my favorite library for defining flakes (flake-parts) which works via a NixOS-like module system. It also takes care of the direnv/caching optimizations we'd otherwise want to roll ourselves. I also love some of the small conveniences it offers, even when they're pretty easy to do with Nix alonw, like the `scripts` functionality. I also find it pleasantly easy to extend— my team has a couple partially completed upstream contributions on the backlog right now, and they were delightfully easy to get working.
Devenv does admittedly have a lot to it. Each of our projects generally only needs a subset of its functionality, but that breadth of functionality seems justified by the fact that it's not always the same subset that we need. Devenv does currently do some things I don't love, like use a custom Nix build for evaluation and recommend a custom Nixpkgs fork for best compatibility, but why it does those things is clear once you dive in and the future direction in those areas seems sound to me.
All of this is from my perspective as my small team's 'Nix guy'. My manager has some casual Nix experience outside of work, and has had success creating his own devenv environments for some projects I'm not involved in as well. Our other guy is totally Nix naive, but has a solid Unix background. He's never initiated any new work with devenv, but he's used our existing devenv integrations without issue.
I think for a team where everyone is still getting their feet wet with Nix, the approach you guys settled on is quite a sound option. But for teams with more mixed Nix experience levels like mine, maybe devenv can work better. For us, it does a nice job of providing a well-documented, idiomatic, highly compatible Nix library to experienced Nix users on the one hand, while presenting a nice porcelain for the other members of the team on the other.
One difference that may have been decisive for our success is that when I selected devenv, my goal was not to avoid Nix at all, but rather to choose a nice convention for defining our project-specific Nix environments. I chose devenv because I trust the author's technical leadership (and it also has a decent community of contributors), it is very Nix-forward for this type of tool (most of defining your environment can be done in Nix, and you can even use it just as a Nix library rather than an executable tool (which we do, for one project)), it supports flakes in a first-class way, and it's built on my favorite library for defining flakes (flake-parts) which works via a NixOS-like module system. It also takes care of the direnv/caching optimizations we'd otherwise want to roll ourselves. I also love some of the small conveniences it offers, even when they're pretty easy to do with Nix alonw, like the `scripts` functionality. I also find it pleasantly easy to extend— my team has a couple partially completed upstream contributions on the backlog right now, and they were delightfully easy to get working.
Devenv does admittedly have a lot to it. Each of our projects generally only needs a subset of its functionality, but that breadth of functionality seems justified by the fact that it's not always the same subset that we need. Devenv does currently do some things I don't love, like use a custom Nix build for evaluation and recommend a custom Nixpkgs fork for best compatibility, but why it does those things is clear once you dive in and the future direction in those areas seems sound to me.
All of this is from my perspective as my small team's 'Nix guy'. My manager has some casual Nix experience outside of work, and has had success creating his own devenv environments for some projects I'm not involved in as well. Our other guy is totally Nix naive, but has a solid Unix background. He's never initiated any new work with devenv, but he's used our existing devenv integrations without issue.
I think for a team where everyone is still getting their feet wet with Nix, the approach you guys settled on is quite a sound option. But for teams with more mixed Nix experience levels like mine, maybe devenv can work better. For us, it does a nice job of providing a well-documented, idiomatic, highly compatible Nix library to experienced Nix users on the one hand, while presenting a nice porcelain for the other members of the team on the other.
I do use Nix and direnv for my work Ubuntu machine. The glibc issue rings true, it was crashing my file manager for quite a long time (I suspect for thumbnail generation). I learned to become better at cli file management ;).
Apart from glibc, I have never had issues with two sets of binaries vying for superiority. Nix binaries take preference with my "vanilla" setup. I think that might be Devbox doing something strange with $PATH?
Apart from glibc, I have never had issues with two sets of binaries vying for superiority. Nix binaries take preference with my "vanilla" setup. I think that might be Devbox doing something strange with $PATH?
Yeah, it was paper cuts - for example, if you don't have git installed inside your devbox, it wouldn't work because of different glibc versions. Which would be fine, but my shell prompt uses git. So there has to be a nix version of git installed for every project for my machine, despite almost no projects technically needing it.
There were a couple of other libraries, can't remember which ones. I remember once having a fun chain of a library that depended on a library that depended on two libraries that in turn depended on glibc, and for some reason the last link of the chain, only one of the libraries was hitting the system libc incorrectly - that was a fun one to debug. I think I ditched that dependency in the end, it was the only solution (and was clearly badly written).
One of my projects used an older version of ruby. In that case, there was a gem to connect to the database, and that gem links to the db client library, but the db is new and the ruby is old and guess what? Two different versions of glibc, both being used within the nix ecosystem.
I worked around a lot of it with LD_LIBRARY_PATH (I think? from memory) which I had to unset for everything in devbox, and used aliases to set it to a backup of that env whenever I found a binary that needed it - and then they tried to fix that, but it just seemed to stop my workarounds from working, so I had to come up with new ones.
But yeah, it was a wild ride. Most of it came back to glibc or environment variables or both, and probably me doing something I really ought not to do (like support old projects). Alas, for me, it wasn't worth the effort - but I sure learned a lot.
There were a couple of other libraries, can't remember which ones. I remember once having a fun chain of a library that depended on a library that depended on two libraries that in turn depended on glibc, and for some reason the last link of the chain, only one of the libraries was hitting the system libc incorrectly - that was a fun one to debug. I think I ditched that dependency in the end, it was the only solution (and was clearly badly written).
One of my projects used an older version of ruby. In that case, there was a gem to connect to the database, and that gem links to the db client library, but the db is new and the ruby is old and guess what? Two different versions of glibc, both being used within the nix ecosystem.
I worked around a lot of it with LD_LIBRARY_PATH (I think? from memory) which I had to unset for everything in devbox, and used aliases to set it to a backup of that env whenever I found a binary that needed it - and then they tried to fix that, but it just seemed to stop my workarounds from working, so I had to come up with new ones.
But yeah, it was a wild ride. Most of it came back to glibc or environment variables or both, and probably me doing something I really ought not to do (like support old projects). Alas, for me, it wasn't worth the effort - but I sure learned a lot.
Great project which you can also use as your package manager on your system.
This works nice if your developers have a MacBook with an Arm processor and want to have the build process close to what you do in your ci server which might be a Linux amd64
I wrote myself a small tool for dev workspaces in containers: https://github.com/queil/rooz - still WIP but I've been using it for quite a while
I created container-shell with a similar goal in mind. It’s super light weight, but does not yet support a package manager. It simply gives you a shell using a specified docker image inside the current directory, isolating both software installs as well as the rest of your filesystem
https://github.com/jrz/container-shell
https://github.com/jrz/container-shell
I think this is my biggest pet peeves when working on different projects. Plus port allocation conflict; if you have multiple docker databases or multiple project using the same default port it's not smooth and obviously changing port is not always an option on legacy projects :/
[deleted]
Why do not use asdf?
Asdf slows down the shell
Use mise instead; it's fast, it gets out of your way, it's easy to use, and it's compatible with asdf plugins.
http://mise.jdx.dev/
Can't manage lower-level deps like dynamic libraries and C compilers, though. I just containerize those.
http://mise.jdx.dev/
Can't manage lower-level deps like dynamic libraries and C compilers, though. I just containerize those.
I really gotta give mise a shot one of these days...
It looks quite smaller in scope than devenv (https://devenv.sh/)
I think the scope is the same. Devenv uses nix expressions. Devbox uses json and has an easy syntax for version pinning.
This looks like a huge overkill to solve the problem that's while real, isn't really so difficult to solve with existing tools... Kind of reminds me of Jucero.
Also, any solution of this kind will inevitably introduce a number of new problems proportionate to the internal complexity of such solution. I.e. this is just asking for a lot of problems, while offering very little in return.
Also, any solution of this kind will inevitably introduce a number of new problems proportionate to the internal complexity of such solution. I.e. this is just asking for a lot of problems, while offering very little in return.
Devbox is really interesting stuff, gonna try it out soon. Thank you!
This is so cool. Thanks!
[deleted]
I'm a bit confused. Direnv is rather simple, is it not.. ?
1. put env variables into .envrc 2. direnv allow
I don't use nix and I don't think I'm over its learning curve. I don't remember investing any time into it..