Classifying Python virtual environment workflows(snarky.ca)
snarky.ca
Classifying Python virtual environment workflows
https://snarky.ca/classifying-python-virtual-environment-workflows/
90 comments
> # when you cd into a folder that doesn't
Is this the behaviour you want, for subdirectories of your project root? Other tools (e.g. git, but also version managers like asdf) tend to look in the current directory, and then recursively try the parent directory to find what the current configuration should be.
Is this the behaviour you want, for subdirectories of your project root? Other tools (e.g. git, but also version managers like asdf) tend to look in the current directory, and then recursively try the parent directory to find what the current configuration should be.
I don't really know the last time I cd'd to a subdirectory in a project, I typically just use fzf or nnn for any of those interactions. Not a bad idea though, thanks!
Do you have any issues with many virtualenvs/packages taking up a lot of disk space?
For instance, numerical packages (scipy, torch, etc) can add up if they’re duplicated across many projects, and, as I understand it, virtualenv doesn’t do any caching/symlinking.
For instance, numerical packages (scipy, torch, etc) can add up if they’re duplicated across many projects, and, as I understand it, virtualenv doesn’t do any caching/symlinking.
I don't work on too many projects at a time and tend to archive and remove the virtualenv when I finish working on something for the foreseeable future. I can see why that would be a problem for some though.
>...virtualenv doesn’t do any caching/symlinking.
Well, one of us is incorrect then. I had been thinking for years that one of the key benefits was that virtualenv would not do a blind copy of files (except on Windows...sigh). The `venv` command even has a `symlinks` vs `copies` option.
`python3 -m venv --help`
Well, one of us is incorrect then. I had been thinking for years that one of the key benefits was that virtualenv would not do a blind copy of files (except on Windows...sigh). The `venv` command even has a `symlinks` vs `copies` option.
`python3 -m venv --help`
Sorry, this was my mistake - I ought to have said that pip doesn't cache/symlink installed packages (unless I am wrong about this also!). You are, of course, correct that the python binaries are symlinked.
I really just run the same python module as you directly, its the most predictable across platforms.
Why don't you run
`python3.11 -m venv .venv`
One option I never see mentioned in any of these discussions about virtualenvs/stuff is my approach to python environments: Each project gets it's own VM/Container.
You then have one python version in the container, and no virtualenvs at all. All libraries get installed right into the main python libraries folder.
I've been using this approach for 5+ years now, and I've yet to hit issues. I read people lamenting issues with python's packaging infrastructure, and with the approach I've never run into any of the issues they talk about.
Pip complains a lot about being told to install stuff globally, but from what I can tell as long as you are careful enough to never use distro-sourced python packages (they're almost always out of date anyways, at least on ubuntu), it's never caused problems for me.
For web-service type things, this has a lot of knock on effects that are beneficial. Each project winds up with it's own IP, if you need to restart things you can often just reboot the VM/container (assuming you have init scripts), process/library isolation issues are effectively impossible, etc...
You then have one python version in the container, and no virtualenvs at all. All libraries get installed right into the main python libraries folder.
I've been using this approach for 5+ years now, and I've yet to hit issues. I read people lamenting issues with python's packaging infrastructure, and with the approach I've never run into any of the issues they talk about.
Pip complains a lot about being told to install stuff globally, but from what I can tell as long as you are careful enough to never use distro-sourced python packages (they're almost always out of date anyways, at least on ubuntu), it's never caused problems for me.
For web-service type things, this has a lot of knock on effects that are beneficial. Each project winds up with it's own IP, if you need to restart things you can often just reboot the VM/container (assuming you have init scripts), process/library isolation issues are effectively impossible, etc...
same.
but I'm still wishing for a native container support from pipenv/pyenv/poetry etc.
but I'm still wishing for a native container support from pipenv/pyenv/poetry etc.
Separation of concerns...
Can other data scientists comment?
I'm 15 years in with python and scientific work. For a lot of years I liked conda but then it got crazy slow. Next I started making conda environments and installing packages with pip. Now I'm experimenting with mamba ("fast conda") and that's pretty good.
Conda envs mean I can experiment with different versions of Python (I'm a co author for O'Reilly's High Performance Python so eg 3.11 and 3.12 are pretty interesting right now). Conda "should" also make identical teaching environments (I teach my own courses). Pip was a pragmatic choice to get installations in minutes not hours in the years when conda was silly-slow.
The above is also all for short -lived research work (my typical client mode for scientific work), so it is probably different to anyone doing long-run dev work, production deploys, or for those not needing non-Python binary support (eg GPU/C/Fortan lib support).
I'm 15 years in with python and scientific work. For a lot of years I liked conda but then it got crazy slow. Next I started making conda environments and installing packages with pip. Now I'm experimenting with mamba ("fast conda") and that's pretty good.
Conda envs mean I can experiment with different versions of Python (I'm a co author for O'Reilly's High Performance Python so eg 3.11 and 3.12 are pretty interesting right now). Conda "should" also make identical teaching environments (I teach my own courses). Pip was a pragmatic choice to get installations in minutes not hours in the years when conda was silly-slow.
The above is also all for short -lived research work (my typical client mode for scientific work), so it is probably different to anyone doing long-run dev work, production deploys, or for those not needing non-Python binary support (eg GPU/C/Fortan lib support).
> Now I'm experimenting with mamba ("fast conda") and that's pretty good.
+1 for this. I really like Conda because I use a lot of packages that cannot be installed with only pip (E.g. GDAL). Conda would sometimes take 10 minutes to make an environment.
Mamba has been an absolute game changer. It has rewritten parts of Conda in C++ and added multithreading, so every part of the process - the solving, download, and the extraction - is all Lightning quick in comparison. A 10 minute process with Conda can now take 2.5 mins with Mamba. It also helps that it is also prettier to look at!
+1 for this. I really like Conda because I use a lot of packages that cannot be installed with only pip (E.g. GDAL). Conda would sometimes take 10 minutes to make an environment.
Mamba has been an absolute game changer. It has rewritten parts of Conda in C++ and added multithreading, so every part of the process - the solving, download, and the extraction - is all Lightning quick in comparison. A 10 minute process with Conda can now take 2.5 mins with Mamba. It also helps that it is also prettier to look at!
I found that the best way to install GDAL was:
- on Windows, using Christoph Gohlke’s wheels,
- on Linux, installing GDAL with the OS’ package manager, then the Python bindings by fixing the version to $(gdal-config —-version) and setting C_INCLUDE_DIR and CPP_INCLUDE_DIR.
It’s a bit more involved, but the speed of pip and conda’s multiple channels and dependency resolution issues make this worth it.
- on Windows, using Christoph Gohlke’s wheels,
- on Linux, installing GDAL with the OS’ package manager, then the Python bindings by fixing the version to $(gdal-config —-version) and setting C_INCLUDE_DIR and CPP_INCLUDE_DIR.
It’s a bit more involved, but the speed of pip and conda’s multiple channels and dependency resolution issues make this worth it.
Agreed. I recently saw on a Mamba team's blog post saying that conda is also starting to incorporating libmamba, so probably the issue could be resolved someday.
I am a Lisp programmer who has mostly used Python for deep learning. I found the long runtimes for some conda operations worth while in order to have several very different environments setup. I now try to use Google Colab for as much of my deep learning work as possible, so I have thought of dropping conda and try an alternative setup because I am writing a Python book that covers deep learning but also covers a lot of use cases that I usually use Common Lisp for. I will probably take the time to try most of the options in this blog article.
Mamba is great! You won't find many self-promotional blog posts about it because it just works.
Mamba doesn't work with optional dependencies (those in square braces e.g. pytorch[gpu])
> Conda "should" also make identical teaching environments (I teach my own courses).
It's probably easier for your students if you provide containers for them, then everyone has an identical environment.
It's probably easier for your students if you provide containers for them, then everyone has an identical environment.
Thinking about how python is the only ecosystem where you need to deal with any of this, and how many thousands of developer-hours are spent on it
It's extremely annoying. The reality is, python can deploy or isolate any way you'd like, but because there isn't one canonical way to do it, everyone comes up with their own system. My best guess about why this is the case is because python is over 30 years old, so they've reinvented the packaging wheel (ha, .whl) four of five times, leading to massive confusion.
Good point. I am wondering what are the specific reasons say Java or JS don’t suffer from this, and hoping for rant-free responses that don’t turn into language wars.
In part because Python is often used as an OS language (Debian has it in the base system and a bunch of OS subsystems use it) so if you upgrade some library at the OS level it affects more than just your own code.
In part it is because Python has modules, not libraries. Modules can have executables bundled with them.
In part, it’s because Python modules packages don’t exactly care about backwards compatibility. By and large, people who put out packages will keep a changelog but not keep the same API going for years. So slightly different version of the same package can give you different results.
Lastly it is because Python does not have a clear definition of a project. If at the language level I could say “this is the root of my project. Do not use any OS-level libraries. Here are my library dependencies and they are to be stores in $PROJECT_ROOT/libs/“ then none of this would be required. I have rarely found that the version of Python itself was a big problem for me, but having the ability for multiple versions to be installed at the OS level combined with the above mechanism would entirely eliminate the need for virtual envs.
In part it is because Python has modules, not libraries. Modules can have executables bundled with them.
In part, it’s because Python modules packages don’t exactly care about backwards compatibility. By and large, people who put out packages will keep a changelog but not keep the same API going for years. So slightly different version of the same package can give you different results.
Lastly it is because Python does not have a clear definition of a project. If at the language level I could say “this is the root of my project. Do not use any OS-level libraries. Here are my library dependencies and they are to be stores in $PROJECT_ROOT/libs/“ then none of this would be required. I have rarely found that the version of Python itself was a big problem for me, but having the ability for multiple versions to be installed at the OS level combined with the above mechanism would entirely eliminate the need for virtual envs.
A key thing for Node is that NPM (in addition to being first-party and having a standard manifest format) installs all dependencies in the project directory, under node_modules
People complain about node_modules, but the benefit is that every Node project on a system is isolated automatically and can re-download all of its dependencies trivially (after cleaning them, or after being checked out onto a new machine)
A Node project's system-wide dependencies are:
1. A new-enough Node installation, and
...that's it
The same is true for Go and Rust and other modern languages. Python is the odd one out.
People complain about node_modules, but the benefit is that every Node project on a system is isolated automatically and can re-download all of its dependencies trivially (after cleaning them, or after being checked out onto a new machine)
A Node project's system-wide dependencies are:
1. A new-enough Node installation, and
...that's it
The same is true for Go and Rust and other modern languages. Python is the odd one out.
Go, Rust, and JS are unique in having package management solutions that prevents the diamond dependency problem. Most other languages suffer from the same problems as python in that they cannot have more than one version of the same package globally. The dependency resolution algorithms for Ruby/Dart/Julia are all NP class and require constraint solving which often fails to find a solution if your dependencies are complex.
For what it’s worth, the Poetry version solver works in the same way as Dart’s: https://github.com/python-poetry/poetry/blob/286f4ddb70394dd...
You can face the diamond problem in NPM, it just deals with it internally (usually by installing both versions under node_modules). But I don't see that as related to whether dependencies are project-specific or system-wide
To be fair, trying to figure out Maven and proper Java IDE integration (which is essential) can also be pretty miserable. Even though there aren't as many tool choices to deal with, the ones that exist can get really complicated and have poor documentation.
I would say that one important factor is the tooling. Instead of having one or two standard tools, as seen in Java (maven/gradle), Node (npm/yarn), .NET (dotnet CLI and MSBuild/maybe FAKE), Python has a plethora of opinionated tools. venv+pip is built-in, but doesn’t handle any of the “where to put the venv” parts, and doesn’t handle making packages installable (and the classic solution is setuptools). There are many competing projects, like pipenv (which does not work at all for libraries), poetry (which uses a non-standard way of specifying metadata and dependencies, and isn’t too friendly with the rest of the packaging world), flit, hatch, etc.
Why are there so many competing tools? Why can’t the community just pick one and make it a good tool for all use-cases? There’s an organization called “Python Packaging Authority”, and it maintains almost all tools mentioned in the previous paragraph (except the stdlib venv, and poetry). If they’re an authority, they should just say “___ is the way to go, we’ll add the missing use-cases to it, here’s a tool to migrate everything else to ___”. Instead, they have done things that support the proliferation of tools, like the PEP 517 (which is a standard API for package managers to talk to build tools).
Another deficiency in the Python package management system is the fact that you need virtual environments to get things done, and that they are often finicky. Some of the modern tools hide the venv from you, but this gets less practical if you’re trying to use things from within scripts, or trying to point your WSGI server at the venv, or in multi-user scenarios. Some of the modern tools put it in .venv and manage it for you, but venvs can randomly break in cases the tool might not be aware of (eg. in some system package managers, upgrading Python to a new minor version would cause venvs to break, because symlink targets moved.)
Node solves this by having a `node_modules` directory. .NET does stuff at build-time that most people don’t need to care about, and then just puts the .dll files for your dependencies next to the .dll with your thing, and the loader looks in the directory your code came from. Python has a proposal for `__pypackages__` [0], but it’s been there since May 2018 with not much progress.
[0] https://peps.python.org/pep-0582/
Why are there so many competing tools? Why can’t the community just pick one and make it a good tool for all use-cases? There’s an organization called “Python Packaging Authority”, and it maintains almost all tools mentioned in the previous paragraph (except the stdlib venv, and poetry). If they’re an authority, they should just say “___ is the way to go, we’ll add the missing use-cases to it, here’s a tool to migrate everything else to ___”. Instead, they have done things that support the proliferation of tools, like the PEP 517 (which is a standard API for package managers to talk to build tools).
Another deficiency in the Python package management system is the fact that you need virtual environments to get things done, and that they are often finicky. Some of the modern tools hide the venv from you, but this gets less practical if you’re trying to use things from within scripts, or trying to point your WSGI server at the venv, or in multi-user scenarios. Some of the modern tools put it in .venv and manage it for you, but venvs can randomly break in cases the tool might not be aware of (eg. in some system package managers, upgrading Python to a new minor version would cause venvs to break, because symlink targets moved.)
Node solves this by having a `node_modules` directory. .NET does stuff at build-time that most people don’t need to care about, and then just puts the .dll files for your dependencies next to the .dll with your thing, and the loader looks in the directory your code came from. Python has a proposal for `__pypackages__` [0], but it’s been there since May 2018 with not much progress.
[0] https://peps.python.org/pep-0582/
There's a couple of reasons for the current situation in terms of the plethora of tools.
One, Python and its packaging story is old (remember, Python predates Linux). That has given folks plenty of time to either come up with their own solutions since Python predates widespread internet usage (PyPI has not always been around).
Two, a lack of standards. Tying back into the "old" point, not everything was initially designed. There has been work to chip away at this and get more standards behind things, but getting folks to update their packaging code is *hard*; most people copy their packaging code from their last project and don't really try to update it. Getting changes to propagate through an ecosystem as large as Python's takes years; a decade is the typical time frame considered for complete uptake.
Three, the lack of standards means tools come up with their own solution which then isn't compatible with anyone else. That means when someone innovates, it can very quickly get locked up behind a single tool. That means folks end up reinventing the wheel for various reasons (e.g. lock files). Different approaches leads to different opinions, which leads to folks choosing different tools for different reasons. And when that happens, there ends up being a lack of consensus.
And four, this is almost entirely driven by volunteers with very few people paid in any way to work on this stuff (I think there might be like 2 people who contribute to pip, 2.5 folks for PyPI).
And good luck telling someone that their preferred workflow isn't the "chosen" workflow that the whole community is going to switch to. People don't like being told they are going to have to change, especially if they believe their approach is superior for whatever reason. Multiply that by the size of the Python community and you can see the pitchforks quite clearly on the horizon.
Now that isn't to say work isn't being done to improve the situation. The PyPA side of packaging has been working on standards for quite some time and is getting some traction with them (e.g. pyproject.toml is a great example of that). But we do have some more standards to work out. We are also regularly discussing how to come up with some singular tool/UX that people can get behind for most use cases, but see the above comments about the size of a challenge that it is and thus why it hasn't happened yet. But people are aware and trying to figure all of this stuff out.
One, Python and its packaging story is old (remember, Python predates Linux). That has given folks plenty of time to either come up with their own solutions since Python predates widespread internet usage (PyPI has not always been around).
Two, a lack of standards. Tying back into the "old" point, not everything was initially designed. There has been work to chip away at this and get more standards behind things, but getting folks to update their packaging code is *hard*; most people copy their packaging code from their last project and don't really try to update it. Getting changes to propagate through an ecosystem as large as Python's takes years; a decade is the typical time frame considered for complete uptake.
Three, the lack of standards means tools come up with their own solution which then isn't compatible with anyone else. That means when someone innovates, it can very quickly get locked up behind a single tool. That means folks end up reinventing the wheel for various reasons (e.g. lock files). Different approaches leads to different opinions, which leads to folks choosing different tools for different reasons. And when that happens, there ends up being a lack of consensus.
And four, this is almost entirely driven by volunteers with very few people paid in any way to work on this stuff (I think there might be like 2 people who contribute to pip, 2.5 folks for PyPI).
And good luck telling someone that their preferred workflow isn't the "chosen" workflow that the whole community is going to switch to. People don't like being told they are going to have to change, especially if they believe their approach is superior for whatever reason. Multiply that by the size of the Python community and you can see the pitchforks quite clearly on the horizon.
Now that isn't to say work isn't being done to improve the situation. The PyPA side of packaging has been working on standards for quite some time and is getting some traction with them (e.g. pyproject.toml is a great example of that). But we do have some more standards to work out. We are also regularly discussing how to come up with some singular tool/UX that people can get behind for most use cases, but see the above comments about the size of a challenge that it is and thus why it hasn't happened yet. But people are aware and trying to figure all of this stuff out.
I’m not exactly some hip and happening ‘on the inside’ developer, but I’m aware enough of the various Big Personalities in the Python community to be quite confident in my belief that politics played a role in PyPA not taking a more consistent stance.
Which is…not at all justification, certainly.
Which is…not at all justification, certainly.
For JS, which I'm intepreti as node because I think that's the real parallel, the answer is NVN and NPM. Or alternatives such as yarn or pnpm. But even the default package manager is great compared to python.
I havent tried it but my understanding is poetry is a bit more like npm.
I havent tried it but my understanding is poetry is a bit more like npm.
> python is the only ecosystem where you need to deal with any of this
Coming from decades spent in C/C++ ecosystem, this is definitely not true. I would say Python users have it rather easy. A lot of Python packaging problems stems from dependencies on messy C/C++ ecosystem.
Coming from decades spent in C/C++ ecosystem, this is definitely not true. I would say Python users have it rather easy. A lot of Python packaging problems stems from dependencies on messy C/C++ ecosystem.
Sure, fair. I guess in my head I meant "only modern ecosystem" and/or "only major ecosystem that gets used in web development"
Well python is also the only language have a full set of numerical / scientific / data-science software that is well-maintained. Seeing the Gigs of storage used by Rust / node without building any of more complex libraries, I cannot imagine how much more they will use when doing scientific / data stuff...
Looking for a suggestion from the audience here.
I lean on Makefile for automation wherever possible, even if it is just a centralized trigger command to run a packaged script (eg scripts/dothething.py). One of my sticking points is how to ensure the Python virtual environment is installed and up to date before triggering any scripts. That is, I clone a new project, cd into it, and want to run some package command inside the expected environment (eg `poetry run scripts/dothething.py`). This will fail if the environment has not previously installed all of the defined dependencies. My annoyance is that requiring `poetry install` to run prior to every command is quite slow, and rarely has anything to do.
Does anyone have a suggestion as to how to ensure the right thing is always done without the slow Poetry resolution? My dream is that in all of my projects you could, say `make test` and Python would optionally create the environment, install the dependencies, and run pytest in the fresh case. The typical development case would confirm that the environment exists and just run pytest.
I lean on Makefile for automation wherever possible, even if it is just a centralized trigger command to run a packaged script (eg scripts/dothething.py). One of my sticking points is how to ensure the Python virtual environment is installed and up to date before triggering any scripts. That is, I clone a new project, cd into it, and want to run some package command inside the expected environment (eg `poetry run scripts/dothething.py`). This will fail if the environment has not previously installed all of the defined dependencies. My annoyance is that requiring `poetry install` to run prior to every command is quite slow, and rarely has anything to do.
Does anyone have a suggestion as to how to ensure the right thing is always done without the slow Poetry resolution? My dream is that in all of my projects you could, say `make test` and Python would optionally create the environment, install the dependencies, and run pytest in the fresh case. The typical development case would confirm that the environment exists and just run pytest.
You can check that a venv exists by looking for .venv/pyvenv.cfg. Is that not enough? Unless you also need to check that all packages are installed and up to date.
1) Confirming that the environment is in sync with my definition file (pyproject.toml or lockfile) is my true objective. If I suddenly add libraryFoo to pyproject.toml, but I forget to run `poetry install`, I want my `make test` command to note the discrepancy and correct the situation.
2) I think that has some brittleness of detecting where the environment lives (as per the author's post). For my use case, I would be fine assuming the defaults, but it is something else to go wrong
Edit: Thinking out loud, I guess the quickest solution is to checksum the pyproject.toml file and store that somewhere. If current hash matches previous, do not run `poetry install`. While that would get me there, I am hoping for a first-party, standard solution.
2) I think that has some brittleness of detecting where the environment lives (as per the author's post). For my use case, I would be fine assuming the defaults, but it is something else to go wrong
Edit: Thinking out loud, I guess the quickest solution is to checksum the pyproject.toml file and store that somewhere. If current hash matches previous, do not run `poetry install`. While that would get me there, I am hoping for a first-party, standard solution.
> I guess the quickest solution is to checksum the pyproject.toml
This is the kind of stuff bazel/ninja do.
https://bazel.build
https://ninja-build.org
This is the kind of stuff bazel/ninja do.
https://bazel.build
https://ninja-build.org
Thanks for the heads-up about that file; I use a similar development setup and have been slacking on wiring Make rules together because I thought I would have to touch my own files. It would be nice if the files Python installed had better timestamps, though. I'm the only one using Make on the team and install some tooling of my own, so have to chain upgrade Pip -> install deps -> install tools in the correct order. I'm not sure how to automate that order without running all three steps every time.
As long as a venv exists and is activated, your Makefile can simply “pip install -r ./requirements.txt” and pip should make sure that all constraints in the requirements are met.
Make sure that the package management tools are up-to-date first with “python -m pip install—upgrade pip setuptools wheel”.
Pip is very fast now and shouldn’t take more than a second if all dependencies are already installed. The only part that’s too slow is installing a package from source.
Make sure that the package management tools are up-to-date first with “python -m pip install—upgrade pip setuptools wheel”.
Pip is very fast now and shouldn’t take more than a second if all dependencies are already installed. The only part that’s too slow is installing a package from source.
I've settled with: Pyenv > python venv > Poetry
CFLAGS="-march=native -mtune=native" CONFIGURE_OPTS="--enable-optimizations --with-lto=full" pyenv install 3.11.1 --verboseSame. Though I’m debating swapping out poetry with pip tools.
I started using poetry for my OSS projects and I'm happy with it, but I'd never use it for closed source projects. Poetry simplifies deps, config, pkg build etc as it's stored in a single place, but it's way to opinionated. For managing dependencies only, pip tools is easier to use and customize. There's no breaking changes, it's a finished product.
Also, I don't want my CI to fail randomly. https://github.com/python-poetry/poetry/pull/6297
Look at the number of issues in poetry. Some things are not backwards compat, some are broken. If you want reliable, KISS with pip tools.
That's my opinion.
Also, I don't want my CI to fail randomly. https://github.com/python-poetry/poetry/pull/6297
Look at the number of issues in poetry. Some things are not backwards compat, some are broken. If you want reliable, KISS with pip tools.
That's my opinion.
Agreed. We tried using poetry in a monorepo that contains dozens of projects and it seemed annoying to get right. Poetry seems to want only one main project folder?
Have you used pip-tools before? I have used PDM instead of Poetry and was very impressed
https://github.com/pdm-project/pdm
https://github.com/pdm-project/pdm
For what its worth, I evaluated pdm, poetry, and pyenv before settling on pip-tools for my workflow. pdm is the only one I would consider trying again (once it is more mature).
I have. At my last gig at Shopify we used it to manage our requirements since poetry didn’t play nice with some internal dev tools.
I’ve tried PDM as well. It’s definitely nice. Part of me just wants to avoid “all encompassing” tools and use a specific tool for a specific job.
I’ve tried PDM as well. It’s definitely nice. Part of me just wants to avoid “all encompassing” tools and use a specific tool for a specific job.
For scientific calculation I just use mamba (sometimes micromamba) + managed environment in home folder. It is impossible to keep an environment for each project due to the library size and storage limitation. These I keep one copy each machine (including on HPCs and local machines). For bootstraping, I keep a copy of the environment.yaml file.
I used to have a dedicated folder to manage these environments myself, but seeing conda/mamba does a good job managing these; and they are also easier to load with command, so I switched to the managed approach.
Conda does not only work for Python, but many software necessary for our workflow that are written in C/C++/Fortran has been precompiled and uploaded to the conda-forge index as well.
I used to have a dedicated folder to manage these environments myself, but seeing conda/mamba does a good job managing these; and they are also easier to load with command, so I switched to the managed approach.
Conda does not only work for Python, but many software necessary for our workflow that are written in C/C++/Fortran has been precompiled and uploaded to the conda-forge index as well.
My flow/boilerplate for apps:
./scripts/create-venv.sh # create venv using specific python version, update pip, wheel, setuptools, install requirements.txt
./scripts/freeze-requirements.sh # create tmp venv with requirements-base.txt and pip freeze > requirements.txt
./requirements-base.txt # dependencies with loose versions
./requirements.txt # frozen/locked dependenciesOnly reason I use virtual environments instead of just building everything inside Docker containers is the debug experience, which is not good enough inside a Docker container.
How is debugging inside a container different than on the host os?
Well to use something like "debugpy" It's the port mapping that just dosn't work straight out of the box, but needs custom settings inside my editor (and often times additional code inside the startup script) to get it working between the container and my local machine.
It sounds like you want to do the debugging outside of the container?
I just do it inside the container with pudb.
I just do it inside the container with pudb.
How does your LSP pick up everything like imports if they only live in a docker container?
LSP?
https://code.visualstudio.com/api/language-extensions/langua... Basically just your auto-completion, linting, quick-actions, etc...
Why does it care if the application runs in a container?
When you run a project in a container, you mount the projects directory into the container.
That is nothing my editor cares about. It does not even know it. It works on the files locally on the host os and that's it. The the files are executed somewhere inside a container is none of the editors business.
When you run a project in a container, you mount the projects directory into the container.
That is nothing my editor cares about. It does not even know it. It works on the files locally on the host os and that's it. The the files are executed somewhere inside a container is none of the editors business.
It doesn't technically care, but if your container is reporting that it has hit a breakpoint in file /mnt/some_module/script.py, then your LSP / debugger / editor is not going to know what it is talking about without some more info.
I do debugging inside the container with pudb.
You run the LSP also inside the container and connect to it remotely.
https://code.visualstudio.com/docs/devcontainers/containers
https://code.visualstudio.com/docs/devcontainers/containers
Okay that makes sense, I don't really see how it's simpler but I completely see the use case. I was unclear on what he/she meant by the editor "just working" without having some config or plugin to point to the installed packages
In my view, you'd normally still want to use a virtual environment or ignore the system site packages.
The system will have a bunch a bunch of (site) packages already installed, and hence you lose control.
The system will have a bunch a bunch of (site) packages already installed, and hence you lose control.
I hope the convention becomes to also consider ./venv as another default location.
Are virtual environments still needed now that we have containers?
I just run everything in a container and never need a virtual environment. Am I missing out on anything?
I just run everything in a container and never need a virtual environment. Am I missing out on anything?
For me it's the debug experience that is lacking when only using containers, I have some work arounds but they are still not good enough.
Can you give an example?
I don't experience any difference, whether I am inside a container or on the host os.
I don't experience any difference, whether I am inside a container or on the host os.
Can you debug in a container, eg VS code debugger with breakpoints and steps?
I have no experience with that as I use pudb for debugging, which I simply run inside the container.
Yea it is a supported use case. VS code remote extension can work with every container just like remote ssh development.
yes, devcontainers work very well. Full debugger. It's also possible to run the container in the cloud and still get a full local debugger.
I did have some issues with some projects and want to revisit this soon.
I did have some issues with some projects and want to revisit this soon.
Not all Python users run containers, especially in the sciences. When you are teaching basics of data manipulation to people that have barely touched any programming language before, getting people to grok even virtual environments is not trivial.
To me, containers are simpler than virtual environments.
Just do "docker run --rm -it debian:11" and you are ready to go.
Just do "docker run --rm -it debian:11" and you are ready to go.
This is the version of containers where they are most useful - a clean, throwaway environment with exactly the one application or userspace you want to test inside it.
Unfortunately as a developer it's more fiddly than that.
You need to mount your code into it. Then if you try commit changes to your code, you're a different user, so the commit is now messed up. Also you lost your dotfiles, so none of your aliases or other shortcuts work when you're in the container. You might need to connect to other services which now need to run in their own containers, and then they all need to talk to each other. Before you know it there's a whole litany of commands you need to run and configuration files you need to put together just to get started in the morning. Imo using containers on development machines is a big step backwards for developer productivity.
Virtual environment is just... cd in the directory. Run magic incantation to set that directory as the canonical one for your current shell. The end. IDEs will pick it up automatically.
The Python virtual environment and package management experience is still worse than pretty much any other programming language, but for me it's far and away better than using containers on a development machine.
Unfortunately as a developer it's more fiddly than that.
You need to mount your code into it. Then if you try commit changes to your code, you're a different user, so the commit is now messed up. Also you lost your dotfiles, so none of your aliases or other shortcuts work when you're in the container. You might need to connect to other services which now need to run in their own containers, and then they all need to talk to each other. Before you know it there's a whole litany of commands you need to run and configuration files you need to put together just to get started in the morning. Imo using containers on development machines is a big step backwards for developer productivity.
Virtual environment is just... cd in the directory. Run magic incantation to set that directory as the canonical one for your current shell. The end. IDEs will pick it up automatically.
The Python virtual environment and package management experience is still worse than pretty much any other programming language, but for me it's far and away better than using containers on a development machine.
You make it sound like you have to edit your code inside the container.
I don't do that. I edit it on the host OS.
I just run the code inside the container.
I don't do that. I edit it on the host OS.
I just run the code inside the container.
Do you have to mount the location of the code as a volume in the container also?
I use containers regularly but adding them to a development workflow for a standalone python application I could just as easily run directly without having the additional complexity of navigating container boundaries always mystifies me.
I use containers regularly but adding them to a development workflow for a standalone python application I could just as easily run directly without having the additional complexity of navigating container boundaries always mystifies me.
Yes, I have one terminal window in which I run / use the code. I start it like this:
docker run --rm -it -v $(pwd):/some/where debian:11
Most of the time I don't use plain Debian, but an image based on Debian which I have peppered with some convenience stuff.
docker run --rm -it -v $(pwd):/some/where debian:11
Most of the time I don't use plain Debian, but an image based on Debian which I have peppered with some convenience stuff.
> I just run everything in a container and never need a virtual environment. Am I missing out on anything?
I don't think it is practical for compute heavy scientific visualization. Setting up GUI with docker was a nightmare for me. There is also some overhead for CPU/GPU heavy code.
I don't think it is practical for compute heavy scientific visualization. Setting up GUI with docker was a nightmare for me. There is also some overhead for CPU/GPU heavy code.
At $WORK, I more or less can only develop on a Windows machine. Getting Docker approval is needlessly painful. Until Docker and WSL are Microsoft standard tools, then yes, these solutions are required.
How is WSL not a Microsoft standard tool? What makes it different from Visual Studio or VSCode or whatever, which is also never going to be built into Windows? Why can you get Python installed but not WSL?
WSL is a full Linux/Ubuntu?
So you can install whatever you can apt-get/wget into it?
I can see that being a problem in restricted environments.
So you can install whatever you can apt-get/wget into it?
I can see that being a problem in restricted environments.
For a software developer, who can run any software they want (because they need to be able to run the software they’re paid to work on), including software that could subvert all the restrictions anyway?
If that's really an issue, corporate IT can have an air gapped repo used for apt-get. I use purely air gapped systems at $WORK and it's not too bad.
I use both. Containers for server code, environments for scripting. Containerizing scripts is a bit excessive, especially if it's only going to run on your machine.
And I have this in my `.zshrc` so whenever I `cd` to that directory, it auto-activates the virtual environment. Having the local directory in the project also makes it really easy for my editor (neovim or VS Code) to pick up