Python environment setup seems complicated and unsolvable(stackoverflow.com)
stackoverflow.com
Python environment setup seems complicated and unsolvable
https://stackoverflow.com/questions/75267582/python-environment-setup-seems-complicated-and-unsolvable
33 comments
Poetry is an all-in-one tool that does dependency management using `pyproject.toml` files and its own `poetry.lock` files, as well as package building, virtualenv management, and also has hooks for entrypoints and scripts.
It's rather a useful tool and I'm personally using it for dependency management and packing for all my projects moving forwards, though for venvs I'm using `pyenv` and the `pyenv-virtualenv` plugin.
Once you figure out your workflow it can be quite nice, but it's figuring it out that's a huge mess in Python at the minute. Hopefully PEP 582 (Python local packages directory) solves it a bit...
It's rather a useful tool and I'm personally using it for dependency management and packing for all my projects moving forwards, though for venvs I'm using `pyenv` and the `pyenv-virtualenv` plugin.
Once you figure out your workflow it can be quite nice, but it's figuring it out that's a huge mess in Python at the minute. Hopefully PEP 582 (Python local packages directory) solves it a bit...
PEP 582 was rejected, FYI.
https://peps.python.org/pep-0582/
https://peps.python.org/pep-0582/
Bugger.
Thanks for letting me know (embarrassingly I did load up the PEP page to make sure I remembered the right number but I didn't check its status).
Was hoping that it would make things simpler for smaller projects and newbie developers but the rejection reason is solid.
Thanks for letting me know (embarrassingly I did load up the PEP page to make sure I remembered the right number but I didn't check its status).
Was hoping that it would make things simpler for smaller projects and newbie developers but the rejection reason is solid.
Funnily enough, Poetry only just very recently started to properly support PyTorch (you know, which seemingly 90% of Python usecases nowadays require). Like, it would not work across different setups with Arm, Cpu and Gpu machines, which is the point of poetry.
For years, youd need to use one of a few hacks that had serious downsides. Our deployments still install pytorch stuff separately in every container.
I think this is still one of the top (open) issues on the repo.
Also, for sufficiently sophisticated projects, poetry was known to lead to hour long dependency resolving sessions. For example, if you feel like using different pypy sources (eg internal registry, plus some wheels etc), poetry will try to install every package from every of these source registry meaning creating a lock file takes now hours. Not sure if this was ever resolved, but at my company, poetry is know as the thing that never works.
For years, youd need to use one of a few hacks that had serious downsides. Our deployments still install pytorch stuff separately in every container.
I think this is still one of the top (open) issues on the repo.
Also, for sufficiently sophisticated projects, poetry was known to lead to hour long dependency resolving sessions. For example, if you feel like using different pypy sources (eg internal registry, plus some wheels etc), poetry will try to install every package from every of these source registry meaning creating a lock file takes now hours. Not sure if this was ever resolved, but at my company, poetry is know as the thing that never works.
I’ve created about 50 Python projects in the last 3 years that are in production, not a single one uses PyTorch.
Actually PDM is the best python package manager tool!
I was going to say that too, but then I was like "yep, that's exactly the problem!"
Yes, but unironically
I’ve worked with Python for about a decade professionally and have settled on just always using containers to manage project environments, which is much less messy than many of the other tools in my experience. I was resistant to learning Docker in 2016, but I couldn’t be happier to have done so now. It has also led me away from the Mac as a work device, as Docker for Mac is workable but not as good as being on Linux.
I’m also not surprised that there are still some growing pains with the M1 architecture changeover–when Apple switched from PowerPC to Intel there were software stragglers for a good while.
I’m also not surprised that there are still some growing pains with the M1 architecture changeover–when Apple switched from PowerPC to Intel there were software stragglers for a good while.
Python packages are mostly fine if they're pure Python, the issue comes when you're installing native C/C++/Fortran dependencies, which most ML and scientific codes have. People who complain about Python vs Go or Rust or Node in this area I think are not comparing like for like at all.
In academia, everyone pretty much uses Conda to get around these issues, because packages there if you stick to the main channel 'just work' but the licensing is prohibitive elsewhere. Many years ago, you couldn't generally expect to install a Python package on your machine if it had compiled dependencies without installing those libraries yourself with your system package manager and having a local compiler toolchain. That's changed because PyPi introduced wheels which allow you to bundled shared objects, and is why Python is now usable on Windows, but it's not perfect as the this post shows. Wheels generally are available for the most popular OS/architecture combinations - x86 processors, Windows, Mac, and 'manylinux' (an old Linux distribution with low version of glibc to maintain compatibility with newer distros. If you fall outside of that (ARM Mac, want to use a distro with MUSL rather than glibc, etc. etc.) then you basically are on your own and need to compile everything from scratch, and you're going to have a lot of... fun... doing so. In my previous job I worked for a University with a POWER9 GPU cluster, and so we used tools like EasyBuild and Spack to try and manage this, but many packages required manual patches in order to get them working.
In academia, everyone pretty much uses Conda to get around these issues, because packages there if you stick to the main channel 'just work' but the licensing is prohibitive elsewhere. Many years ago, you couldn't generally expect to install a Python package on your machine if it had compiled dependencies without installing those libraries yourself with your system package manager and having a local compiler toolchain. That's changed because PyPi introduced wheels which allow you to bundled shared objects, and is why Python is now usable on Windows, but it's not perfect as the this post shows. Wheels generally are available for the most popular OS/architecture combinations - x86 processors, Windows, Mac, and 'manylinux' (an old Linux distribution with low version of glibc to maintain compatibility with newer distros. If you fall outside of that (ARM Mac, want to use a distro with MUSL rather than glibc, etc. etc.) then you basically are on your own and need to compile everything from scratch, and you're going to have a lot of... fun... doing so. In my previous job I worked for a University with a POWER9 GPU cluster, and so we used tools like EasyBuild and Spack to try and manage this, but many packages required manual patches in order to get them working.
I use Nix to manage python environments, and it’s quite revealing.
For instance, a few months ago I was working on an environment for playing around with some libraries that you’d think would commonly go together. But within the community there are very popular libraries that cannot be used together (at recent enough commits to contain features I want) due to version conflicts.
But as I’m a nix person I obviously try to solve that by going rogue. This was my process before I decided to pack it in and wait for the community to sort their shit out:
1. Pull projects at their latest commit instead. This is the “hope the communities already have their shit sorted out” method. nope, no luck.
2. Write a patch that changes the older requirement. Maybe it is API compatible? LOL no such luck.
3. Write a patch that changes the outdated dependency name to some random name, then add an expression to map the outdated dependency to that name instead. Ugh no, there are version incompatibility with its own dependencies too.
4. Write a patch to update the lib with the outdated dependency to use the new API instead - success! On its own. Now use it together with the other libraries… uh oh.
It has a C++ backend that requires a GLIBC version that is incompatible with that used by the other libraries. It needs this because it needs to be built with a specific CUDA-enabled compiler that is quite outdated now, and if you compile it on a newer version it fails to build because the author hardcoded the compatibility matrix into the codebase with c macros.
So yeah there comes a point where you think “hey I want to try out using some fun tools I’ve heard about” and, before you know it, you’re writing a patch for a dependency that allows you to use a patched version of another dependency that allows you to use a patched version of a dependency that you want to add to your project alongside something else, and at some point you’ve just got to quit and hope that in a few weeks there will be version parity.
For instance, a few months ago I was working on an environment for playing around with some libraries that you’d think would commonly go together. But within the community there are very popular libraries that cannot be used together (at recent enough commits to contain features I want) due to version conflicts.
But as I’m a nix person I obviously try to solve that by going rogue. This was my process before I decided to pack it in and wait for the community to sort their shit out:
1. Pull projects at their latest commit instead. This is the “hope the communities already have their shit sorted out” method. nope, no luck.
2. Write a patch that changes the older requirement. Maybe it is API compatible? LOL no such luck.
3. Write a patch that changes the outdated dependency name to some random name, then add an expression to map the outdated dependency to that name instead. Ugh no, there are version incompatibility with its own dependencies too.
4. Write a patch to update the lib with the outdated dependency to use the new API instead - success! On its own. Now use it together with the other libraries… uh oh.
It has a C++ backend that requires a GLIBC version that is incompatible with that used by the other libraries. It needs this because it needs to be built with a specific CUDA-enabled compiler that is quite outdated now, and if you compile it on a newer version it fails to build because the author hardcoded the compatibility matrix into the codebase with c macros.
So yeah there comes a point where you think “hey I want to try out using some fun tools I’ve heard about” and, before you know it, you’re writing a patch for a dependency that allows you to use a patched version of another dependency that allows you to use a patched version of a dependency that you want to add to your project alongside something else, and at some point you’ve just got to quit and hope that in a few weeks there will be version parity.
Part of the problem is that nix takes an opposite stance from where python seems to be heading…
Poetry, pipenv, etc are all about building dependency tree solvers into python package management.. so if you have package a that depends on package c >= 2.0 and package b that depends on package c < 2.5, you will get package c installed in version 2.5…
Nix basically says “Python packages are lying when they say they need this specific package version. Ignore that and install whatever version is in nixpkgs”.
Which sucks! There are some badly maintained workarounds (poetry2nix and pip2nix come to mind… neither of which works for m1 macs), but the whole stance is just wrong.
Poetry, pipenv, etc are all about building dependency tree solvers into python package management.. so if you have package a that depends on package c >= 2.0 and package b that depends on package c < 2.5, you will get package c installed in version 2.5…
Nix basically says “Python packages are lying when they say they need this specific package version. Ignore that and install whatever version is in nixpkgs”.
Which sucks! There are some badly maintained workarounds (poetry2nix and pip2nix come to mind… neither of which works for m1 macs), but the whole stance is just wrong.
I believe, having recently installed Python libraries on it, that Debian stable has the best handle on what to do to produce defaults that a mere mortal can debug: Either use what's vendored in apt, or use venv and pip for your project's additional dependencies. Pip has been configured to be venv-only, which is good for the needs of Debian's maintainers, and clarifies what your project has to be responsible for. So, while I haven't needed it yet, I have some confidence that I can get to a reproducible dev environment with the appropriate version of whatever's needed, even if the resolution process is imperfectly automated.
In terms of tooling Python is a mess. Poetry is indeed quite good, but it still has issues.
I think what python misses is things like cargo, go and npm (dare I say it..), tools that handle packages and run your app in context, making that things generally just work.
Might be anecdotal, but I have had way less cross platform/binary issues in other languages. For node things are usually handled very well these days, rust just compiles stuff (or great docs with clear instructions).
It’s not just environments, it’s formatting, linting, types.
I think python used to be ahead in usability, in the last decade it has fallen behind a bit.
It’s still a good language to use because it can do so much, but I can’t say the experience is enjoyable.
I think what python misses is things like cargo, go and npm (dare I say it..), tools that handle packages and run your app in context, making that things generally just work.
Might be anecdotal, but I have had way less cross platform/binary issues in other languages. For node things are usually handled very well these days, rust just compiles stuff (or great docs with clear instructions).
It’s not just environments, it’s formatting, linting, types.
I think python used to be ahead in usability, in the last decade it has fallen behind a bit.
It’s still a good language to use because it can do so much, but I can’t say the experience is enjoyable.
IMHO, Poetry is the best we have in the Python dep mgmt space, and it's still endlessly frustrating. It's especially hard to recommend it for newbies looking to get up and running with even a simple ML stack. Check out this thread[1] on the Kafkaesque nightmare that is trying to install PyTorch with Poetry.
[1]: https://github.com/python-poetry/poetry/issues/6409
[1]: https://github.com/python-poetry/poetry/issues/6409
Poetry won’t ever be the answer because the Poetry maintainers haven’t shown the maturity to be real leaders in the space. It’s good, as many narrowly opinionated projects are, but ultimately the core maintainers are not interested in use cases they see as outside their vision for the tool. Which means it will never be the one tool to rule them all.
For ML projects, conda is still better since it usually manages to resolve a working environment including pytorch and cuda.
Sure, it doesn’t lead to the same exact environment on every machine, but that stuff never ever works anyway at least with portry.
Sure, it doesn’t lead to the same exact environment on every machine, but that stuff never ever works anyway at least with portry.
The whole "Python is how science gets done" meme is one of the dumbest things we've allowed to be foisted on otherwise unaware/unsuspecting users (such as the kinds of academics who end up being the victims of the Python ecosystem shitshow). Who knows how many setbacks in science we've suffered, not to mention billions of dollars of productivity lost, sticking to such an unworthy programming system/language/environment. All because, like, colons and significant whitespace make programming so much easier to pick up when you compare it to making someone look at curly braces—which, as we all know is the hardest part of programming.
Once you start to look at scientific packages in other languages, they have the same issues as Python does, because they start to use scientific libraries written in C and Fortran, as rewriting 50+ years of code is actually really hard.
Working with python environments made me sorry for all the times I may have complained about the Node.js way of doing things.
I just gave up on learning Python because of this mess. It's really too bad, because I'd like to write utilities with Python that would run on almost any machine.
Oh well.
When I was still trying to wrangle this, I had followed this guide on setting up Python "the right way." I wonder what y'all think: https://opensource.com/article/19/5/python-3-default-mac
Oh well.
When I was still trying to wrangle this, I had followed this guide on setting up Python "the right way." I wonder what y'all think: https://opensource.com/article/19/5/python-3-default-mac
Looks mostly good, but also has extra elements you don’t really need.
Pyenv - managing / installing multiple python versions.
Then all you need to start is regular virtual environments ‘python -m venv .venv’ with ‘requirements.txt’, but I also recommend poetry. This is for dependency isolation between projects.
Packaging up projects for distribution is a whole other venture though!
Pyenv - managing / installing multiple python versions.
Then all you need to start is regular virtual environments ‘python -m venv .venv’ with ‘requirements.txt’, but I also recommend poetry. This is for dependency isolation between projects.
Packaging up projects for distribution is a whole other venture though!
Thanks!
My 2 cents:
Since I use a Arch based Linux distribution, I like to update my (main) Python environment just as frequently: at least once a week. My findings so far:
1. Do not interfere with the system managed packages. I.e. use --user when pip install or make the system python lib dir(s) non-writable for the user
2. If you don't like to switch frequently into another venv, just put the packages that interfere with your main (user) env into separate venvs. These are normally young and intensely developed modules, like e.g. textual, or old non-updated modules, like e.g. ecs, or complicated, cross-language stuff, like e.g. numba, scipy. They will separate themselves out after a while.
3. If nonetheless some local packages interfere with system packages (btw, I hate the --break-system-packages crap), make separate users each with the appropriate environment.
Rinse and repeat.
Since I use a Arch based Linux distribution, I like to update my (main) Python environment just as frequently: at least once a week. My findings so far:
1. Do not interfere with the system managed packages. I.e. use --user when pip install or make the system python lib dir(s) non-writable for the user
2. If you don't like to switch frequently into another venv, just put the packages that interfere with your main (user) env into separate venvs. These are normally young and intensely developed modules, like e.g. textual, or old non-updated modules, like e.g. ecs, or complicated, cross-language stuff, like e.g. numba, scipy. They will separate themselves out after a while.
3. If nonetheless some local packages interfere with system packages (btw, I hate the --break-system-packages crap), make separate users each with the appropriate environment.
Rinse and repeat.
Arch has set python to externally managed since June, so you can't mix system python packages with pip installed ones.
https://gitlab.archlinux.org/archlinux/packaging/packages/py...
https://peps.python.org/pep-0668/
https://gitlab.archlinux.org/archlinux/packaging/packages/py...
https://peps.python.org/pep-0668/
With sudo pip you can.
But I was thinking of other Linux distributions. The ones that caused the introduction of --break-system-packages.
And it prevents less a mixing than a shadowing of system packages with local packages.
But I was thinking of other Linux distributions. The ones that caused the introduction of --break-system-packages.
And it prevents less a mixing than a shadowing of system packages with local packages.
Yes on arch you can only install packages with `--break-system-packages`. But you can't accidentally mix up system packages with pip.
Not to mention most of the pain people experience with activating venvs can be mitigated by running the Python binary directly
./venv/bin/python3
./venv/bin/python3
I've settled on miniconda for virtual environment management and poetry for dependency management. I'm happier with this setup than anything I've tried before.
I use python pretty much all day long, and would never use a python environment besides the basics. A Docker is more general, serves the same purpose, less mental overhead etc.
Really, could anyone explain why I should care about python environment tools at all? Because we do fine without them.
Really, could anyone explain why I should care about python environment tools at all? Because we do fine without them.
I agree with everyone here that Python is a wonderful tool and it is frustrating when everything does not work ootb.
Doctor, it hurts when I do that!
It seems pretty emblematic of the shitshow that is Python environment management that the top-rated comment is recommending something I've never heard of. I know pyenv and pyvenv (not the same thing!) and virtualenv and venv (not the same thing!) and pipenv, but apparently those are all now out of date and I should be moving to this sixth new tool. I'm sure this time it will be fixed for good.