Python Project-Local Virtualenv Management Redux(hynek.me)
hynek.me
Python Project-Local Virtualenv Management Redux
https://hynek.me/articles/python-virtualenv-redux/
35 comments
If you're targeting just one version of python, on just one version of one OS, only running on a computer you control, then there are very few problems. The problems only really come when you remove one or more of the above constraints.
True, though I'm not sure .venv is a solution for any of those problems.
Multiple python versions still isn't exactly hard with pip+venv. If you want a different python version for your venv, just use that python version to install the venv. Package repositories not providing the python version you need? Python is surprisingly easy to compile from source.
Different OSes is a bit more annoying. I tend to just get VMs involved and do the build on the target OS.
Different OSes is a bit more annoying. I tend to just get VMs involved and do the build on the target OS.
Being conservative with library and language feature choices helps a lot too
The problem I had was that the plethora of current and former alternatives out there obscured that fact that this was a straightforward and adequate option.
pip + venv with requirements.txt is 90% of the way there. the bare minimum most people need on top of this is some form of lockfile. without this, builds are not reproducible. it's frustrating to ask users to install an extra tool (like poetry or whatever) when python comes with almost everything. at least `pyproject.toml` is standardized in a PEP, so you don't have to force users to choose a specific tool to resolve it.
I've been doing largely the same as you up until just about a month ago, and maybe once a year I'd have to track down and deal with breakage of my environment because of it. Things periodically get into a weird state for reasons I don't understand.
About a month ago I set up a "homedir python" using pyenv, to hopefully solve this problem once and for all. So far it seems to be working great.
Though for projects I've been playing with poetry and rye the last year, basically just as syntactic sugar over venv, mostly just to try them out. They work fine, though I don't understand why "rye shell" is being deprecated.
About a month ago I set up a "homedir python" using pyenv, to hopefully solve this problem once and for all. So far it seems to be working great.
Though for projects I've been playing with poetry and rye the last year, basically just as syntactic sugar over venv, mostly just to try them out. They work fine, though I don't understand why "rye shell" is being deprecated.
Rye is replaced by UV.
See https://lucumr.pocoo.org/2024/2/15/rye-grows-with-uv/
See https://lucumr.pocoo.org/2024/2/15/rye-grows-with-uv/
Thumbs up, plus a well-maintained Makefile that I share among my projects. Never had any issues. Github matricx test runners ensure that everything works across platforms, Python and dep versions, ...
what do you put in your Makefile?
make env = create environment and pip install editable mode (I typically list dev and test dependencies as optional-dependencies in pyproject.toml)
make test = test, coverage, ...
make docs = sphinx-build ...
make release = build, twine, ...
I try to keep things simple.
I try to keep things simple.
It gets really hard in accelerated computing, where there are a number of stories that are kinda ok (vanilla virtualenv with some hand-rolled special casing can get you like, off the ground, conda/miniconda are workable in some scenarios, Poetry can kinda do it), but it just gets really, really hard if performance matters.
pip’s depset builder is also non-deterministic up to its stated capabilities: at the stated capabilities dependency resolution is IIRC exactly NP-hard, it’s in that “you’re using a SAT solver but it will usually be fine” neck of the woods which is why more modern stuff (including I think apt and other Debian-adjacent stuff) use solvers now. You can limit the use of version ranges, you can pin like Bazel does with depsets (usually). This is why running pip repeatedly often solves the problem: you rolled a better character and/or it was hinted by a previous run.
Docker and friends can help but you get the usual Docker headaches, especially with qemu in the mix. I find OrbStack is just a strictly better option everywhere I’ve tried it, but docker over emulation/virtualization is rarely any kind of fun in the presence of accelerators.
So, I want to run an LLM or SD or some other now common thing, but I’ve got an NVIDIA box on x86_64-linux and a Mac on aarch64-darwin. What do I do? That sounds reasonable right?
Good luck. I went the whole way with Nix last time, and it solved lots of other problems (and in that world, problems stay solved until you are compelled to do a non-trivial update), but Python was still a nightmare.
This time I’m trying flox: and so far it’s been friggin awesome. It’s a little rough here and there for advanced users, but flox install pip, pip install torch/etc, and go? That works. On like, 7 os/cpu/accelerator combinations I’ve tried.
It probably sounds like I’m on their payroll but I’m legitimately just a very, very happy user who posts on HN in between excellent interactions with my bedrock env tool now. I hang out in their Slack but I don’t work there.
pip’s depset builder is also non-deterministic up to its stated capabilities: at the stated capabilities dependency resolution is IIRC exactly NP-hard, it’s in that “you’re using a SAT solver but it will usually be fine” neck of the woods which is why more modern stuff (including I think apt and other Debian-adjacent stuff) use solvers now. You can limit the use of version ranges, you can pin like Bazel does with depsets (usually). This is why running pip repeatedly often solves the problem: you rolled a better character and/or it was hinted by a previous run.
Docker and friends can help but you get the usual Docker headaches, especially with qemu in the mix. I find OrbStack is just a strictly better option everywhere I’ve tried it, but docker over emulation/virtualization is rarely any kind of fun in the presence of accelerators.
So, I want to run an LLM or SD or some other now common thing, but I’ve got an NVIDIA box on x86_64-linux and a Mac on aarch64-darwin. What do I do? That sounds reasonable right?
Good luck. I went the whole way with Nix last time, and it solved lots of other problems (and in that world, problems stay solved until you are compelled to do a non-trivial update), but Python was still a nightmare.
This time I’m trying flox: and so far it’s been friggin awesome. It’s a little rough here and there for advanced users, but flox install pip, pip install torch/etc, and go? That works. On like, 7 os/cpu/accelerator combinations I’ve tried.
It probably sounds like I’m on their payroll but I’m legitimately just a very, very happy user who posts on HN in between excellent interactions with my bedrock env tool now. I hang out in their Slack but I don’t work there.
Works until you need a reproducible setup that will alert you, if anything in your dependencies chances.
That's because you don't need different versions of python. The author needs to test his stuff against different versions, including ARM and intel versions.
I'd say it is a good idea to pin python version in your project just like you'd pin any library - after all, there are changes to the standard library all the time. But how to do that in practice. Fuck me if I know a good way.
I'd say it is a good idea to pin python version in your project just like you'd pin any library - after all, there are changes to the standard library all the time. But how to do that in practice. Fuck me if I know a good way.
Can try with a pinned container image.
My Python virtual environment and dependency management journey has been the following:
1. Just `pip` and `venv`; specifying dependencies via "requirements.txt" and "dev_requirements.txt", and tool configuration options in bespoke files. Too involved and with too many parts, yet it made do for my use cases.
2. `poetry` together with "pyproject.toml". Big quality of life improvement: commands like `poetry add`, `poetry build` make the process much less involved so one can just focus on actual development. Having all of the tooling configuration in a single place (that being "pyproject.toml") also helped with context switching.
3. `uv` for quick and dirty Python projects, `rye` (with the `uv` backend) for anything above that. In short, I think of `uv` and `rye` as faster siblings of `pip` and `poetry`, respectively. Once you get used to creating virtual and environments and installing dependencies in milliseconds or a few seconds it's difficult to go back to it always taking many seconds.
In hindsight, I would have been content just using `poetry`, but `uv` and `rye` do make for a faster and hence better developer experience.
P.S.: how messed up is it that only now we are agreeing on a convention for putting a project’s virtual environment into an in-project directory called `.venv`?
1. Just `pip` and `venv`; specifying dependencies via "requirements.txt" and "dev_requirements.txt", and tool configuration options in bespoke files. Too involved and with too many parts, yet it made do for my use cases.
2. `poetry` together with "pyproject.toml". Big quality of life improvement: commands like `poetry add`, `poetry build` make the process much less involved so one can just focus on actual development. Having all of the tooling configuration in a single place (that being "pyproject.toml") also helped with context switching.
3. `uv` for quick and dirty Python projects, `rye` (with the `uv` backend) for anything above that. In short, I think of `uv` and `rye` as faster siblings of `pip` and `poetry`, respectively. Once you get used to creating virtual and environments and installing dependencies in milliseconds or a few seconds it's difficult to go back to it always taking many seconds.
In hindsight, I would have been content just using `poetry`, but `uv` and `rye` do make for a faster and hence better developer experience.
P.S.: how messed up is it that only now we are agreeing on a convention for putting a project’s virtual environment into an in-project directory called `.venv`?
Are we agreeing on that name for the venv? I usually just go "python3 -m venv venv". And usually I don't need to activate that manually anyway, but have it handled in a Makefile, whenever activation is necessary for a task.
It's funny how people seem to love this node style of a virtualenv in the project.
I much, much prefer virtualenvwrapper with it's style of keeping the environment outside the project.
This way, I don't end up with lots of huge directories hidden in the many many projects I check out.
The other thing that can't happen is accidentally adding the folder to version control.
I can have various versions of my project checkout out, and use the same venv between them.
I can quickly fire up a venv for the project with a different version of python.
I can quickly list all my venvs, and prune them as needed.
While people may say "we have infinite storage" my laptop only has 2tb and I fill it up.
But each to their own, a lot of people seem to like it.
I also use direnv, and the the virtualenvwrapper plugin for zsh, and have defaults - in this case it automatically defaults to the virtualenv with the same name as the directory I enter, but I didn't have to do any config for that to happen.
I much, much prefer virtualenvwrapper with it's style of keeping the environment outside the project.
This way, I don't end up with lots of huge directories hidden in the many many projects I check out.
The other thing that can't happen is accidentally adding the folder to version control.
I can have various versions of my project checkout out, and use the same venv between them.
I can quickly fire up a venv for the project with a different version of python.
I can quickly list all my venvs, and prune them as needed.
While people may say "we have infinite storage" my laptop only has 2tb and I fill it up.
But each to their own, a lot of people seem to like it.
I also use direnv, and the the virtualenvwrapper plugin for zsh, and have defaults - in this case it automatically defaults to the virtualenv with the same name as the directory I enter, but I didn't have to do any config for that to happen.
> The other thing that can't happen is accidentally adding the folder to version control.
A few other ways I've seen embedded venvs interfering, off the top of my head:
* Polluting output from tools like `find`, `grep` and `tree` (yes, some have .gitignore options, but you have to remember to use them and some don't have the option at all, and often that's not appropriate anyway),
* Increasing docker build times due to not adding it to .dockerignore,
* Ending up in source packages somehow (e.g. serverless framework).
A few other ways I've seen embedded venvs interfering, off the top of my head:
* Polluting output from tools like `find`, `grep` and `tree` (yes, some have .gitignore options, but you have to remember to use them and some don't have the option at all, and often that's not appropriate anyway),
* Increasing docker build times due to not adding it to .dockerignore,
* Ending up in source packages somehow (e.g. serverless framework).
There is a certain quality in simplicity.
With inside project venvs what you see is what you get. VS Code can find them, no magic involved.
> my laptop only has 2tb and I fill it up
With venvs? Even assuming 4 GB pytorch/cuda/whatnot venvs that's a lot of venvs.
With inside project venvs what you see is what you get. VS Code can find them, no magic involved.
> my laptop only has 2tb and I fill it up
With venvs? Even assuming 4 GB pytorch/cuda/whatnot venvs that's a lot of venvs.
Nice! I've recently discovered direnv too and can't believe I've coped this long without it. I currently use virtualenvwrapper and direnv to activate those (I much prefer having the venv root outside of the project dirs; not everything respects gitignore files by default).
I didn't know about uv and PDM. I use pip-tools and there has been some work to adapt it to support multiple Python interpreters. My main gripe is it can't be installed once with pipx, for example, and just work. But it seems difficult. Perhaps uv suffers from the same problem. Interested to see how PDM deals with it.
> While direnv has built-in support for the standard library’s venv module, it’s glacial. virtualenv is much faster, but uv is faster, still.
I've never been bothered by the speed of venv. It does what it needs to do in milliseconds and it's not like I'm calling it that often. Why is the speed of it important?
I didn't know about uv and PDM. I use pip-tools and there has been some work to adapt it to support multiple Python interpreters. My main gripe is it can't be installed once with pipx, for example, and just work. But it seems difficult. Perhaps uv suffers from the same problem. Interested to see how PDM deals with it.
> While direnv has built-in support for the standard library’s venv module, it’s glacial. virtualenv is much faster, but uv is faster, still.
I've never been bothered by the speed of venv. It does what it needs to do in milliseconds and it's not like I'm calling it that often. Why is the speed of it important?
I can't believe people aren't using devcontainers
This is especially significant for all the stuff outside the virtual environment.
Is there a standard method of making a "deploycontainer" from a devcontainer? Something that strips out everything not necessary to execute the application (however that is defined)?
Is there a standard method of making a "deploycontainer" from a devcontainer? Something that strips out everything not necessary to execute the application (however that is defined)?
I haven't embraced devcontainers specifically, but I've standardized on putting all apps in Docker Compose for a few years now.
Tell me more...
> I use different mechanisms to manage the virtual environments depending on whether I’m developing a *package* without strict pinning or an *application* with strict pinning.
I assume that by “package,” the author means library?
I assume that by “package,” the author means library?
The docs define only module[0] and package[1] but the term library is often used to refer to a collection of modules such as the "standard library."
0. https://docs.python.org/3/glossary.html#term-module
1. https://docs.python.org/3/glossary.html#term-package
A non-canonical source claims broader meaning for "Library:"
A library is an umbrella term referring to a reusable chunk of code. Usually, a Python library contains a collection of related modules and packages. Actually, this term is often used interchangeably with “Python package” because packages can also contain modules and other packages (subpackages). However, it is often assumed that while a package is a collection of modules, a library is a collection of packages.
https://learnpython.com/blog/python-modules-packages-librari...
0. https://docs.python.org/3/glossary.html#term-module
1. https://docs.python.org/3/glossary.html#term-package
A non-canonical source claims broader meaning for "Library:"
A library is an umbrella term referring to a reusable chunk of code. Usually, a Python library contains a collection of related modules and packages. Actually, this term is often used interchangeably with “Python package” because packages can also contain modules and other packages (subpackages). However, it is often assumed that while a package is a collection of modules, a library is a collection of packages.
https://learnpython.com/blog/python-modules-packages-librari...
In the Python world "package" implies something that can be published to PyPI - the Python Package Index - and then installed using pip, a recursive acronym for "pip installs packages".
Package is a better term than library, because some of the things you can install with pip, such as command-line tools, don't really act like libraries.
Package is a better term than library, because some of the things you can install with pip, such as command-line tools, don't really act like libraries.
I tried poetry for the first time last night and it was honestly a breeze to get set up and felt much more full featured than pipping about with a requirements.txt
[deleted]
I bring this up because this is such a common topic to complain about, and there is this sentiment that Python packaging is completely broken and unusable. But for me, and I suspect a large quiet userbase, stuff just works fine, and I don't think this side gets brought up nearly enough. And for people not actively using Python, it's important to note that you don't need a rube-goldberg setup like in TFA.