Pyenv – lets you easily switch between multiple versions of Python(github.com)
github.com
Pyenv – lets you easily switch between multiple versions of Python
https://github.com/pyenv/pyenv
339 comments
Actually, it only builds it locally if it can't find a pre-packaged version for your system/arch. Admittedly that's most of the recent ones on a Mac, but there is a difference (I've been using pyenv for nearly ten years[1] now).
The big advantage for me is that I can match whatever runtime and standard library a target has (and yes, that's needed more times than not, even in this new age of Docker).
Additionally, you can build an _optimized_ Python. I have this set for my builds:
The big advantage for me is that I can match whatever runtime and standard library a target has (and yes, that's needed more times than not, even in this new age of Docker).
Additionally, you can build an _optimized_ Python. I have this set for my builds:
env PYTHON_CONFIGURE_OPTS='--enable-optimizations --with-lto' PYTHON_CFLAGS='-march=native -mtune=native' pyenv install 3.12.2
That happens way more often than people want to admit, and I think it's not genuine to present the tool that way.
E.G, I'm on Ubuntu 20.04 on an Dell XPS, a fairly standard machine, I'll get:
But by default it downloads the source, and compiles it.
On top of that it will use a shim which comes with its own world of possible pain.
Again, I don't want to bash on pyenv.
But I do want to lower people's expectations.
It's a tool for experts, not beginners.
E.G, I'm on Ubuntu 20.04 on an Dell XPS, a fairly standard machine, I'll get:
pyenv install 3.9 -v
/tmp/python-build.20240325124651.73089 ~
Downloading Python-3.9.19.tar.xz...
-> https://www.python.org/ftp/python/3.9.19/Python-3.9.19.tar.xz
...
LD_LIBRARY_PATH=/tmp/python-build.20240325124651.73089/Python-3.9.19 CC='gcc -pthread' LDSHARED='gcc -pthread -shared -L/home/user/.pyenv/versions/3.9.19/lib -Wl,-rpath,/home/user/.pyenv/versions/3.9.19/lib -L/home/user/.pyenv/versions/3.9.19/lib -Wl,-rpath,/home/user/.pyenv/versions/3.9.19/lib ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall' _TCLTK_INCLUDES='' _TCLTK_LIBS='' ./python -E ./setup.py build
running build
running build_ext
... etc
But by default it downloads the source, and compiles it.
On top of that it will use a shim which comes with its own world of possible pain.
Again, I don't want to bash on pyenv.
But I do want to lower people's expectations.
It's a tool for experts, not beginners.
I’ve quite literally never seen it download a pre-compiled Python - I didn’t actually know it had that functionality, and have been using it for probably a decade.
Ditto. I had no idea that it ever looked for a pre-compiled version.
Actually, I'm pretty sure that binaries for Python 3.9 aren't available on Ubuntu 20.04 without adding an apt repository to get them from, so your statement that you "can apt install it" isn't entirely correct in this case - by default, binaries aren't available, and therefore pyenv has to compile Python 3.9.
Just tried, I get the same result with 3.7, 3.8, 3.10 and 3.11.
Currently going through this myself on a new mac I bought lol. Still haven't gotten anywhere. Oh well
$ PYTHON_CONFIGURE_OPTS='--disable-ipv6 --enable-optimizations --with-lto' PYTHON_CFLAGS='-march=native -mtune=native' pyenv install 3.12.2 python-build: use openssl@3 from homebrew python-build: use readline from homebrew Downloading Python-3.12.2.tar.xz... -> https://www.python.org/ftp/python/3.12.2/Python-3.12.2.tar.x... Installing Python-3.12.2... python-build: use readline from homebrew python-build: use ncurses from homebrew python-build: use zlib from xcode sdk
BUILD FAILED (OS X 14.4 using python-build 20180424)
Inspect or clean up the working tree at /var/folders/w9/xvxzj68j6kx7m480rnwq6hvh0000gn/T/python-build.20240325114837.43577 Results logged to /var/folders/w9/xvxzj68j6kx7m480rnwq6hvh0000gn/T/python-build.20240325114837.43577.log
Last 10 log lines: ./Include/internal/pycore_interp.h:193:24: error: field has incomplete type 'struct _dtoa_state' struct _dtoa_state dtoa; ^ ./Include/internal/pycore_interp.h:193:12: note: forward declaration of 'struct _dtoa_state' struct _dtoa_state dtoa; ^ 1 error generated. make[2]: ** [Objects/boolobject.o] Error 1 make[1]: ** [profile-gen-stamp] Error 2 make: ** [profile-run-stamp] Error 2
$ PYTHON_CONFIGURE_OPTS='--disable-ipv6 --enable-optimizations --with-lto' PYTHON_CFLAGS='-march=native -mtune=native' pyenv install 3.12.2 python-build: use openssl@3 from homebrew python-build: use readline from homebrew Downloading Python-3.12.2.tar.xz... -> https://www.python.org/ftp/python/3.12.2/Python-3.12.2.tar.x... Installing Python-3.12.2... python-build: use readline from homebrew python-build: use ncurses from homebrew python-build: use zlib from xcode sdk
BUILD FAILED (OS X 14.4 using python-build 20180424)
Inspect or clean up the working tree at /var/folders/w9/xvxzj68j6kx7m480rnwq6hvh0000gn/T/python-build.20240325114837.43577 Results logged to /var/folders/w9/xvxzj68j6kx7m480rnwq6hvh0000gn/T/python-build.20240325114837.43577.log
Last 10 log lines: ./Include/internal/pycore_interp.h:193:24: error: field has incomplete type 'struct _dtoa_state' struct _dtoa_state dtoa; ^ ./Include/internal/pycore_interp.h:193:12: note: forward declaration of 'struct _dtoa_state' struct _dtoa_state dtoa; ^ 1 error generated. make[2]: ** [Objects/boolobject.o] Error 1 make[1]: ** [profile-gen-stamp] Error 2 make: ** [profile-run-stamp] Error 2
Actually, I installed rye and got everything working. There goes any usage of pyenv going forward lol.
PYTHON_CONFIGURE_OPTS='--disable-ipv6
> Actually, it only builds it locally if it can't find a pre-packaged version for your system/arch. Admittedly that's most of the recent ones on a Mac, but there is a difference (I've been using pyenv for nearly ten years[1] now).
Really? I regularly use this on Linux exactly because it compiles from source rather than using the system provided Python which often have patches in them that breaks pip's test suite.
I've never encountered it trying to use the system provided Python, but maybe there's a weird quirk in the way I am using it.
Really? I regularly use this on Linux exactly because it compiles from source rather than using the system provided Python which often have patches in them that breaks pip's test suite.
I've never encountered it trying to use the system provided Python, but maybe there's a weird quirk in the way I am using it.
If you don’t care which specific version of python you are using, do not use pyenv.
You will know when you care. These days, doing common tasks, the constraint solver in poetry will often tell you:
“Hey! I can’t find a version of sentencepiece with metadata that lets me combine this with python 3.12.2. Sorry! I give up!”
Now, if you aren’t concerned with using your bare metal for CUDA or fancy new MPS or AMD stuff. Just ignore this and use containers. I’d use podman compose.
However, I use pyenv on every machine. Because it compiles specific versions I actually need, even to create various virtual environments. If compiling python automatically sounds tough, you probably don’t need to anyway.
To describe the problem you’d see. I try to use poetry by default, though I think it became popular before it was PEP-compliant or useful in devops. It is impossible to control the behavior of other package managers, and poetry is/was strict about that. Which means you can’t force deploy in many cases. (Better lately.)
For the problem pyenv helps to solve, I back-up my pyproject.toml setuptools backend with pip and requirements.txt. These days, requirements-cuda.txt and requirements-mps.txt.
The landscape is still a disaster for binary compatibility, but it can be done lol. (I’ve been doing python packaging professionally since prom, which was python 2.6 give or take.)
You will know when you care. These days, doing common tasks, the constraint solver in poetry will often tell you:
“Hey! I can’t find a version of sentencepiece with metadata that lets me combine this with python 3.12.2. Sorry! I give up!”
Now, if you aren’t concerned with using your bare metal for CUDA or fancy new MPS or AMD stuff. Just ignore this and use containers. I’d use podman compose.
However, I use pyenv on every machine. Because it compiles specific versions I actually need, even to create various virtual environments. If compiling python automatically sounds tough, you probably don’t need to anyway.
To describe the problem you’d see. I try to use poetry by default, though I think it became popular before it was PEP-compliant or useful in devops. It is impossible to control the behavior of other package managers, and poetry is/was strict about that. Which means you can’t force deploy in many cases. (Better lately.)
For the problem pyenv helps to solve, I back-up my pyproject.toml setuptools backend with pip and requirements.txt. These days, requirements-cuda.txt and requirements-mps.txt.
The landscape is still a disaster for binary compatibility, but it can be done lol. (I’ve been doing python packaging professionally since prom, which was python 2.6 give or take.)
This kind of error surprises me:
> I can’t find a version of sentencepiece with metadata that lets me combine this with python 3.12.2
I'm a reasonably advanced python user. I've shipped web apps, Desktop GUIs, cli-tools, and even written cpython extension modules for custom hardware control. I typically target the system `python3` of whatever linux distribution I'm shipping to and I use the system 'python3-virtualenv' for venvs.
But I have never encountered a dependency resolution issue or been forced to use poetry. What am I doing wrong?
> I can’t find a version of sentencepiece with metadata that lets me combine this with python 3.12.2
I'm a reasonably advanced python user. I've shipped web apps, Desktop GUIs, cli-tools, and even written cpython extension modules for custom hardware control. I typically target the system `python3` of whatever linux distribution I'm shipping to and I use the system 'python3-virtualenv' for venvs.
But I have never encountered a dependency resolution issue or been forced to use poetry. What am I doing wrong?
Sometimes, an app or library developer wants to target multiple Linux distributions.
For example, I want my own Python apps to work equally well on Debian oldstable (which currently provides Python 3.9) and Arch Linux (currently on 3.11). That means that I’m going to choose the lowest common version (3.9) as the language level for my app or library.
And if I program against a 3.9 language level, I absolutely *refuse* to use any other interpreter than a 3.9 one at development time. (If I used a newer one, then my linter and type checker would helpfully give me false alarms all the time, unaware of the actual language level.)
Hence, I use pyenv and poetry to get exactly the interpreter for the language level I want, and to allow pylint and mypy to be perfectly aware of the language level I’m using at development time, allowing them to give me findings that actually make sense.
For example, I want my own Python apps to work equally well on Debian oldstable (which currently provides Python 3.9) and Arch Linux (currently on 3.11). That means that I’m going to choose the lowest common version (3.9) as the language level for my app or library.
And if I program against a 3.9 language level, I absolutely *refuse* to use any other interpreter than a 3.9 one at development time. (If I used a newer one, then my linter and type checker would helpfully give me false alarms all the time, unaware of the actual language level.)
Hence, I use pyenv and poetry to get exactly the interpreter for the language level I want, and to allow pylint and mypy to be perfectly aware of the language level I’m using at development time, allowing them to give me findings that actually make sense.
> But I have never encountered a dependency resolution issue or been forced to use poetry. What am I doing wrong?
Not using python on a system without the necessary build tools while trying to use dependencies with native code that don't have binaries built for your combination of python version and platform?
This used to be a huge problem with Python on Windows, as things that weren't pure Python would very often not have binary packages or have them for a narrow set of python versions, often not the same set as other dependencies. (Not just a windows problem, but it was definitely big on windows.)
Tooling and practices have advanced so that more packages are automatically built against a wider set of targets, so the problem is a lot smaller than it used to be.
Not using python on a system without the necessary build tools while trying to use dependencies with native code that don't have binaries built for your combination of python version and platform?
This used to be a huge problem with Python on Windows, as things that weren't pure Python would very often not have binary packages or have them for a narrow set of python versions, often not the same set as other dependencies. (Not just a windows problem, but it was definitely big on windows.)
Tooling and practices have advanced so that more packages are automatically built against a wider set of targets, so the problem is a lot smaller than it used to be.
(Edit2 tl:dr: it’s that other package maintainers don’t always keep up with new semver constants in their metadata, particularly on anything cutting-edge)
It’s always the inclusion of a specific dependency we added for a feature, and based on that dev’s knowledge and experience. It’s often me, but not always.
This doesn’t happen in ecosystems with a base package versioning. This is arguably why anaconda became popular, and why we target base docker images of ubuntu.
Doesn’t work in complex deployments based on money rather than ideals, every time. At least in my career.
Edit: first time I dealt with this, we ended up forking a dependency chain instead of using pip. I lost that war and they ended up reviving a legacy PHP app instead of funding python dev.
It’s always the inclusion of a specific dependency we added for a feature, and based on that dev’s knowledge and experience. It’s often me, but not always.
This doesn’t happen in ecosystems with a base package versioning. This is arguably why anaconda became popular, and why we target base docker images of ubuntu.
Doesn’t work in complex deployments based on money rather than ideals, every time. At least in my career.
Edit: first time I dealt with this, we ended up forking a dependency chain instead of using pip. I lost that war and they ended up reviving a legacy PHP app instead of funding python dev.
So, what is "the tool for beginners fighting with python packaging problems"?
That is, the pattern seems to be that someone mentions a solution, then a zillion responses as to why it sucks.
Is there any tool or pair that sucks least for most cases and beginners? I get that every case is different, but perhaps there are some useful starting points?
That is, the pattern seems to be that someone mentions a solution, then a zillion responses as to why it sucks.
Is there any tool or pair that sucks least for most cases and beginners? I get that every case is different, but perhaps there are some useful starting points?
While there are no silver bullet, for beginners, I found that "Relieving your Python packaging pain" (https://www.bitecode.dev/p/relieving-your-python-packaging-p...) is the Pareto solution. That is, the solution that has the best ratio of effort, reward, but also the lower risk of failure. It's not no risk, but I've been helping beginners for 15 years with Python, and have tried everything you can think of.
It does imply, on linux, to limit yourself to the choices of Python you can install. This constraint is, for most people, preferable than the alternative, even if it gets frustrating to our geeky soul.
It does imply, on linux, to limit yourself to the choices of Python you can install. This constraint is, for most people, preferable than the alternative, even if it gets frustrating to our geeky soul.
There isn't one. Python distribution and packaging is just fundamentally horribly broken.
I think his point was that you shouldn't pretend to users that just switching to pyenv is the solution.
I think his point was that you shouldn't pretend to users that just switching to pyenv is the solution.
> Python distribution and packaging is just fundamentally horribly broken
It's clearly not because most people successfully use it fine.
The problem of distribution and packaging is often a matter of user expectations vs. the actual problem.
The user expectations is that Python is a high level language and will run the same across different machines regardless of OS and Hardware.
The actual problem is Python is a glue language often depending on lots of libraries that are sensitive to how they were compiled and what hardware they are targeting. So people can end up moving the the glue of the project first and then expect everything else to just work.
Things are getting better (e.g. https://lukeplant.me.uk/blog/posts/python-packaging-must-be-...), a lot of people have put a lot of work into standards and improving the ecosystem, but it is a hard problem that most other popular languages don't interface with nearly as much.
It's clearly not because most people successfully use it fine.
The problem of distribution and packaging is often a matter of user expectations vs. the actual problem.
The user expectations is that Python is a high level language and will run the same across different machines regardless of OS and Hardware.
The actual problem is Python is a glue language often depending on lots of libraries that are sensitive to how they were compiled and what hardware they are targeting. So people can end up moving the the glue of the project first and then expect everything else to just work.
Things are getting better (e.g. https://lukeplant.me.uk/blog/posts/python-packaging-must-be-...), a lot of people have put a lot of work into standards and improving the ecosystem, but it is a hard problem that most other popular languages don't interface with nearly as much.
> It's clearly not because most people successfully use it fine.
Well, no because it's perfectly possible to successfully use a horribly broken system. I use Python "successfully", it just meant I have spend probably literal weeks of my life fighting pip and virtualenv and relative imports and finding I need flags like `--config-settings editable_mode=compat`.
> The problem of distribution and packaging is often a matter of user expectations vs. the actual problem.
Ha yes, I expect it to work reliably and simply and it doesn't!
> The actual problem is Python is a glue language often depending on lots of libraries that are sensitive to how they were compiled and what hardware they are targeting.
That's completely irrelevant to the kind of problems I was talking about. I run into issues with Python failing to compile C libraries relatively rarely! Even compiling Python itself seems to work quite well (maybe not surprising since that's one of the only ways to get a new version on Linux).
It's all the packaging infrastructure that's a mess. Pip, virtualenv, setuptools, and also the module import system is a total disaster. You don't see questions like this for Go:
https://stackoverflow.com/questions/14132789/relative-import...
Well, no because it's perfectly possible to successfully use a horribly broken system. I use Python "successfully", it just meant I have spend probably literal weeks of my life fighting pip and virtualenv and relative imports and finding I need flags like `--config-settings editable_mode=compat`.
> The problem of distribution and packaging is often a matter of user expectations vs. the actual problem.
Ha yes, I expect it to work reliably and simply and it doesn't!
> The actual problem is Python is a glue language often depending on lots of libraries that are sensitive to how they were compiled and what hardware they are targeting.
That's completely irrelevant to the kind of problems I was talking about. I run into issues with Python failing to compile C libraries relatively rarely! Even compiling Python itself seems to work quite well (maybe not surprising since that's one of the only ways to get a new version on Linux).
It's all the packaging infrastructure that's a mess. Pip, virtualenv, setuptools, and also the module import system is a total disaster. You don't see questions like this for Go:
https://stackoverflow.com/questions/14132789/relative-import...
> It's all the packaging infrastructure that's a mess. Pip, virtualenv, setuptools, and also the module import system is a total disaster. You don't see questions like this for Go:
> https://stackoverflow.com/questions/14132789/relative-import...
I think you'll find the further you delve into it the less the problems are distinct, a lot of the issues with the module import system, pip, virtualenv, setuptools, etc. is because they are designed with having to support a vast range of things, from being depended on to interact with system libraries to downloading sdists and compiling arbitrary languages, etc.
Though the specific example you linked was largely solved with Python 3, there was a lot of confusion during the 2 to 3 transition because people had to support both behaviors, but most people don't have to think about Python 2 any more.
> https://stackoverflow.com/questions/14132789/relative-import...
I think you'll find the further you delve into it the less the problems are distinct, a lot of the issues with the module import system, pip, virtualenv, setuptools, etc. is because they are designed with having to support a vast range of things, from being depended on to interact with system libraries to downloading sdists and compiling arbitrary languages, etc.
Though the specific example you linked was largely solved with Python 3, there was a lot of confusion during the 2 to 3 transition because people had to support both behaviors, but most people don't have to think about Python 2 any more.
> Though the specific example you linked was largely solved with Python 3
I can assure you it absolutely was not.
> I think you'll find the further you delve into it the less the problems are distinct, a lot of the issues with the module import system, pip, virtualenv, setuptools, etc. is because they are designed with having to support a vast range of things, from being depended on to interact with system libraries to downloading sdists and compiling arbitrary languages, etc.
Not really. There are plenty of systems that have to "support a vast range of things" that aren't this bad.
In my opinion it's because the core Python devs didn't particularly care about the issue, never really tried to solve it, and as a result we have 10 incompatible half-baked third party solutions.
It's similar to the situation with C/C++ - worse in some ways, better in others (at least there is a de facto package registry in Python). In some ways it's because both languages are very old and predate the idea that packaging should be easy and reliable. That's fine, but please don't pretend that it is easy and reliable now.
I can assure you it absolutely was not.
> I think you'll find the further you delve into it the less the problems are distinct, a lot of the issues with the module import system, pip, virtualenv, setuptools, etc. is because they are designed with having to support a vast range of things, from being depended on to interact with system libraries to downloading sdists and compiling arbitrary languages, etc.
Not really. There are plenty of systems that have to "support a vast range of things" that aren't this bad.
In my opinion it's because the core Python devs didn't particularly care about the issue, never really tried to solve it, and as a result we have 10 incompatible half-baked third party solutions.
It's similar to the situation with C/C++ - worse in some ways, better in others (at least there is a de facto package registry in Python). In some ways it's because both languages are very old and predate the idea that packaging should be easy and reliable. That's fine, but please don't pretend that it is easy and reliable now.
> I can assure you it absolutely was not.
The question, as posted, was a confusion about how Python 2 relative importing worked, which was indeed bad. I don't know what you think you are pointing out, you haven't said, and the question *is* about Python 2.
> Not really. There are plenty of systems that have to "support a vast range of things" that aren't this bad. > > In my opinion it's because the core Python devs didn't particularly care about the issue, never really tried to solve it, and as a result we have 10 incompatible half-baked third party solutions.
I agree that a lot of the solutions were created when there was not an understanding, or thought out design, on what would be a good packaging solution.
But these ill thought out solutions were exactly because of trying to support this wide range of situations, from working on weird OSes, to integrating with strange build systems.
However, what you seemed to have missed is there is now a first party standard on:
* How package installers (pip, poetry, PDM, etc.) should interact with package builders (setuptools, hatchling, etc.)
* How and where build configuration and project metadata should be stored (pyproject.toml)
Almost every popular Python package tool now supports these standards, meaning they all interact with each other pretty well.
Dropping all legacy configuration, and updating the standards to support edge cases is still a long road, but it is a road being travelled and things are getting better.
The question, as posted, was a confusion about how Python 2 relative importing worked, which was indeed bad. I don't know what you think you are pointing out, you haven't said, and the question *is* about Python 2.
> Not really. There are plenty of systems that have to "support a vast range of things" that aren't this bad. > > In my opinion it's because the core Python devs didn't particularly care about the issue, never really tried to solve it, and as a result we have 10 incompatible half-baked third party solutions.
I agree that a lot of the solutions were created when there was not an understanding, or thought out design, on what would be a good packaging solution.
But these ill thought out solutions were exactly because of trying to support this wide range of situations, from working on weird OSes, to integrating with strange build systems.
However, what you seemed to have missed is there is now a first party standard on:
* How package installers (pip, poetry, PDM, etc.) should interact with package builders (setuptools, hatchling, etc.)
* How and where build configuration and project metadata should be stored (pyproject.toml)
Almost every popular Python package tool now supports these standards, meaning they all interact with each other pretty well.
Dropping all legacy configuration, and updating the standards to support edge cases is still a long road, but it is a road being travelled and things are getting better.
I don't disagree with what you're saying but that article seems odd. It's just a story of someone installing something once that didn't break immediately. Not only is it anecdotal, it doesn't even seem to confirm whether the pip story is getting better or if they just got lucky.
I agree, but there is no large scale study on this.
As someone who has managed Python distributions in a large company and who triages issues on the pip github issue page that my anecdotal experience is things are getting better.
The only hard statistic I can point to is the number of top packages on PyPI that offer wheels has substantially gone up, and is close to 100% in the top 500.
As someone who has managed Python distributions in a large company and who triages issues on the pip github issue page that my anecdotal experience is things are getting better.
The only hard statistic I can point to is the number of top packages on PyPI that offer wheels has substantially gone up, and is close to 100% in the top 500.
They use it fine by using Docker. So many major Python repos on GitHub come with a Dockerfile compared to, say, NodeJS. It's unfortunate, but having dealt with Python packages before, I don't blame them.
Lots of analysts, data scientists, traders, engineers, etc., use Python third party packages successfully, and have never touched, or maybe even head of, Docker.
And yeah, in general there are significantly less NodeJS third party packages interfacing with packages that directly depend on OSes and hardware. Python has many third party packages that are older than NodeJS that depend on foreign function interfaces, win32 com apis, directly talking to graphics shaders, etc.
And yeah, in general there are significantly less NodeJS third party packages interfacing with packages that directly depend on OSes and hardware. Python has many third party packages that are older than NodeJS that depend on foreign function interfaces, win32 com apis, directly talking to graphics shaders, etc.
Python packages are painful even if nothing native is involved. There are NodeJS packages that rely on native code too, difference is the packaging system is a lot simpler to use, though it's starting to get worse with the `import` vs `require` nonsense and Typescript.
The silver bullet is Docker. It's not the best solution in every situation, but at least it'll work.
For your own projects built from scratch (rather than big multi-dep projects off GitHub), there's a smaller learning curve if you go to the vanilla https://www.python.org/downloads/ , install the latest, and use the included pip to install packages. That'll probably get you very far. It's not like the older days when you needed both Py3 and Py2.
For experts working frequently in Python, tools like Pyenv can make more sense.
For your own projects built from scratch (rather than big multi-dep projects off GitHub), there's a smaller learning curve if you go to the vanilla https://www.python.org/downloads/ , install the latest, and use the included pip to install packages. That'll probably get you very far. It's not like the older days when you needed both Py3 and Py2.
For experts working frequently in Python, tools like Pyenv can make more sense.
You could just give up and resort to using Docker.
Using docker is not free, you are effectively trading one type of complexity for another.
And of course you'll have to make sure people equally have good practices, since incorrectly using sudo pip install in docker and not a venv are very common.
So again, one possible solution for a certain context, but I wouldn't sell that to most people. Certainly not to beginners.
People writing their pythonanywhere website won't pop up a container, won't they ?
And of course you'll have to make sure people equally have good practices, since incorrectly using sudo pip install in docker and not a venv are very common.
So again, one possible solution for a certain context, but I wouldn't sell that to most people. Certainly not to beginners.
People writing their pythonanywhere website won't pop up a container, won't they ?
For a beginner trying to run someone else's Python project, it is free. You run one command and it works. There's tons more complexity under the hood, but you don't have to deal with it. This is why so many Python projects do resort to Docker.
And btw, JS doesn't have this problem, they use NPM.
And btw, JS doesn't have this problem, they use NPM.
> For a beginner trying to run someone else's Python project, it is free.
Its absolutely, by far, the highest friction alternative on Windows for anyone that doesn't use Docker for other purposes.
Its absolutely, by far, the highest friction alternative on Windows for anyone that doesn't use Docker for other purposes.
- Not if you have to edit the code. Try to explain your beginner with notepad++ how to do that.
- Not if you have to deploy the code.
- Not if you have to import the code in your own project.
Basically, not if you want to do anything by running the code. For which I would suggest an installer instead.
- Not if you have to deploy the code.
- Not if you have to import the code in your own project.
Basically, not if you want to do anything by running the code. For which I would suggest an installer instead.
I don't see what editing has to do with this. Whether you're running the main.py or the Dockerfile, if you want to edit the code then you gotta use Notepad.
Is it not just install and run? The alternative is installing Python, installing pip, etc.
Installing python with the python.org installer installs the entirety of Python and its standard library and tools, including pip, venv, etc., and the py launcher which supports selecting from multiple python versions. Linux distros may split this up into different packages, but on Windows (or Mac, afaik, though the py launcher is a Windows-only feature) using the official installers is one-stop shopping.
And, no, docker isn't just install and run on Windows. Before Docker made a heavy push for paid Docker Desktop for even personal use, it was close to that.
But now the way to get a free usable docker command line is to install WSL and a Linux environment, install docker there, and then invoke docker via wsl. (Which, of course, you will not find via Docker’s own information, which will try to sell you a paid subscription.)
And, no, docker isn't just install and run on Windows. Before Docker made a heavy push for paid Docker Desktop for even personal use, it was close to that.
But now the way to get a free usable docker command line is to install WSL and a Linux environment, install docker there, and then invoke docker via wsl. (Which, of course, you will not find via Docker’s own information, which will try to sell you a paid subscription.)
I tried it just now on my spare Win10 desktop, and it went like this:
1. Download and install Docker desktop for Windows
2. Restart Windows (guess you don't have to do this with just Python)
3. Run Docker Desktop
4. Say "no" to signing into a Docker account
5. Wait for engine to start, which took a few minutes the first time, a bit annoying
6. Pull and run an image (I tried nginx)
It was weird being asked to log in, but the "no" button was pretty clear. I didn't feel like I was forced to use WSL to avoid paying; maybe they've backtracked from something. Unity was far more convincing that I had to pay.
1. Download and install Docker desktop for Windows
2. Restart Windows (guess you don't have to do this with just Python)
3. Run Docker Desktop
4. Say "no" to signing into a Docker account
5. Wait for engine to start, which took a few minutes the first time, a bit annoying
6. Pull and run an image (I tried nginx)
It was weird being asked to log in, but the "no" button was pretty clear. I didn't feel like I was forced to use WSL to avoid paying; maybe they've backtracked from something. Unity was far more convincing that I had to pay.
Honestly this. I've moved three companies now to docker based dev for teams on python. It's better for teams who want to customize how they in particular work without inadvertently also blocking themselves from quickly contributing to another team because that team uses a totally different python management scheme. So a team can be conda-in-docker and another team can be virtualenv-in-docker and another team can be raw-python-in-docker and that one group of weirdos can be jython or whatever other sad shit they cook up.
The common interface is they all use `docker-compose up` and have their editors hooked into the containers.
The common interface is they all use `docker-compose up` and have their editors hooked into the containers.
this is the way.
If you use Homebrew: you can use ‘brew pyenv-sync’ to use Homebrew’s pythons with pyenv. Similar commands are available for rbenv/nodenv (which always feel like it is missing an ‘e’ to me)
> it compiles python on your machine
What is the alternative? Every solution that I know of on Linux requires you to build Python on the machine: asdf, official Python downloads, etc.
What is the alternative? Every solution that I know of on Linux requires you to build Python on the machine: asdf, official Python downloads, etc.
For now most people would do well to stick to python.org installers for mac and windows.
For linux, official repos are ideal. If you really, really can't (which is different than wanting to), ubuntu deadsnake and red hat epl are the best second plan, while already more finicky.
If you use something more exotic, you chose hardship, and you will have to be up to the task.
Anything else will come with bigger caveats than the people promoting them will really admit to.
The story is of course a bit richer because linux packaging is fun, so I'll complete with:
https://www.bitecode.dev/p/installing-python-the-bare-minimu...
Thete are currently no silver bullet to bootstrap python on linux. I tried all of them with hundred of colleagues and trainees.
I do have hope for astral to come up with one but don't run on rye thinking your life is saved.
For linux, official repos are ideal. If you really, really can't (which is different than wanting to), ubuntu deadsnake and red hat epl are the best second plan, while already more finicky.
If you use something more exotic, you chose hardship, and you will have to be up to the task.
Anything else will come with bigger caveats than the people promoting them will really admit to.
The story is of course a bit richer because linux packaging is fun, so I'll complete with:
https://www.bitecode.dev/p/installing-python-the-bare-minimu...
Thete are currently no silver bullet to bootstrap python on linux. I tried all of them with hundred of colleagues and trainees.
I do have hope for astral to come up with one but don't run on rye thinking your life is saved.
Even on Macs, you can use MacPorts. They provide a lot of versions pre-compiled (and everything is BSD-solid).
I know you mean well, but such advice is why so many beginners have painful Python experience.
Macports and homebrew Pythons are dependencies of other packages. They can be used by you, but they are not meant for you.
This means at some point, they will contain a surprise, and not a good one.
Macports and homebrew Pythons are dependencies of other packages. They can be used by you, but they are not meant for you.
This means at some point, they will contain a surprise, and not a good one.
> but such advice is why so many beginners have painful Python experience
Sample of one, but I never encountered anything even broken in MacPorts. They seem to embrace the BSD ethos of doing everything right (even if at a slower pace, as some packages lag a few releases behind Homebrew).
Sample of one, but I never encountered anything even broken in MacPorts. They seem to embrace the BSD ethos of doing everything right (even if at a slower pace, as some packages lag a few releases behind Homebrew).
Personally I only use MacPorts or Homebrew to install random smaller tools. It's easier to install Python, NodeJS, Postgres, etc binaries from the main website.
These builds are an alternative: https://github.com/indygreg/python-build-standalone
Those are what Rye and hatch use.
Drawbacks: late availability of patch versions, various quirks from how they are built (missing readline, missing some build info that self-compiled C python modules might need.)
Those are what Rye and hatch use.
Drawbacks: late availability of patch versions, various quirks from how they are built (missing readline, missing some build info that self-compiled C python modules might need.)
I think PEP711 (https://peps.python.org/pep-0711/) is (eventually) a better alternative, because it builds on top of the proven manylinux approach to binary compatibility.
Those are proof of concept builds now and will eventually hopefully replace indygreg's builds.
Not only does the format need to exist but the service of building and publishing them is needed too.
Not only does the format need to exist but the service of building and publishing them is needed too.
Very cool, this is the first time I'm hearing of PEP 711! I hope this (or some other PEP like it) will get accepted eventually!
I hope this will be merged with indygreg builds at some point, but the community move slowly on those issues so it may take years.
> Every solution that I know of on Linux requires you to build Python on the machine
Unless you need a Python that's not supported by your Linux distribution, you can just use what's available.
On macOS, MacPorts provides compiled versions for 3.2 all the way to 3.13, as well as 2.6 and 2.7. Right now, I have 3.8, 3.9, 3.10, 3.11, 3.12, and a 3.13 development build. The fact it's not Linux (or x86) might cause some frustration.
Unless you need a Python that's not supported by your Linux distribution, you can just use what's available.
On macOS, MacPorts provides compiled versions for 3.2 all the way to 3.13, as well as 2.6 and 2.7. Right now, I have 3.8, 3.9, 3.10, 3.11, 3.12, and a 3.13 development build. The fact it's not Linux (or x86) might cause some frustration.
If you’re lucky enough to be on a Linux system that uses apt some thankless soul maintains a repo called deadsnakes with all these binaries. Fabulous if you’re using any somewhat old version of Python in CI for instance. Yum based systems are SOL as far as I can tell. Build and host your own binary for that. Apk doesn’t have this either IIRC
If you are deploying to such an ancient OS, it's perhaps easier to have the whole OS packaged as a container or a VM and use that. It's not only different Python versions that might bite you.
What makes you think that GP’s comment is talking about older OSes?
My understanding of their comment is that they’re talking about getting older Python interpreters to run on more modern OSes, modern enough that they don’t carry the older Python as a system package anymore. Hence, deadsnakes.
My understanding of their comment is that they’re talking about getting older Python interpreters to run on more modern OSes, modern enough that they don’t carry the older Python as a system package anymore. Hence, deadsnakes.
This is not about system packages or OSes, this is about your application needing Python 3.9.7 specifically, and locking to that, and 3.9.7 not being available in repositories anymore (3.9.7 is just an example). So normally you would either need to self host 3.9.7 somewhere or compile it from source on every new machine (which is terrible for CI but fine for local dev, a one off in local dev to build a Python version is nothing but paying 4 minutes every time on ci to compile your version of Python from source is a very angry amount of time for a lot of people).
> needing Python 3.9.7 specifically
That's indeed extremely specific. I'd imagine this pain could be self-inflicted with C-based extensions that were compiled (and can't be recompiled) with structures that don't exist in other versions.
I don't want to imagine what other eldritch horrors await developers working on this application.
That's indeed extremely specific. I'd imagine this pain could be self-inflicted with C-based extensions that were compiled (and can't be recompiled) with structures that don't exist in other versions.
I don't want to imagine what other eldritch horrors await developers working on this application.
Usually it is not the patch version being important eg being specifically 3.9.7, but rather picking one supported version and pinning it. Yes we should all be on the latest Python version that is available in the mainline repos and distros as a binary but a lot of orgs I’ve seen don’t invest the time and effort to properly be upgrading to be on the latest version of deps. So you are usually having one specific supported Python version that everyone builds against which may or may not be new enough to be in the official repo.
I also imagine the scenario you mentioned causing a lock to a patch version. But I see this as just a normal steady state a lot of orgs drift to if not staying on top of version updates
I also imagine the scenario you mentioned causing a lock to a patch version. But I see this as just a normal steady state a lot of orgs drift to if not staying on top of version updates
The thankless soul is the awesome Anthony Sottile, who is also the author of pre-commit.ci: https://twitter.com/codewithanthony
Drop by and send thanks cause he really needs them.
Drop by and send thanks cause he really needs them.
deadsnakes is useful, but also a bit of a misnomer. One can use it to get pretty much anything python version, even ones that are too new to be available in your default apt repos. That is, it's not just for 'dead'/EOL versions.
That's exactly why I don't really on that for Ubuntu though. It's a single point of failure, when I can just use `asdf` to install Python and easily switch between it, automatically if setup correctly on projects.
I use asdf on my local machine, stuff like deadsnakes is for when you need your CI to use VaguelyOldButStillSupported Python version and don’t want to either compile Python from source or host your own binary
My use case is that I'm almost always using the latest Python version, which many Linux distros don't have in their packaging yet, and I'm upgrading code using old versions to the latest. asdf is the best tool for that.
Rye downloads pre-built binaries.
I'll leave further discussion of the reliability and provenance of said binaries to someone else.
I'll leave further discussion of the reliability and provenance of said binaries to someone else.
Binaries are quite reliable and from a serious project, but rye is young and uses shim.
Still, it's part of those new generation of tooling that bringing hope for the next few years.
Still, it's part of those new generation of tooling that bringing hope for the next few years.
> What is the alternative?
Using per-distro official installation channels.
E.g. using deadsnakes PPA on Ubuntu, AUR on Arch Linux, etc.
Using per-distro official installation channels.
E.g. using deadsnakes PPA on Ubuntu, AUR on Arch Linux, etc.
try pyoxidizer
I think that you are painting a bit of an unreasonably bleak view of pyenv. I think that one can easily get value out of it without being a Python expert. I’m not sure how I can refute your “yes, but you’ll eventually run into trouble. You just haven’t clocked enough hours yet”. I can’t prove a negative. But I’ll say that I’ve been writing Python in my day job for a decade.
But, to add to the list of problems, these Python versions IIRC do not compile with optimisation turned on, so they’re by default quite a bit slower than they need to be.
But, to add to the list of problems, these Python versions IIRC do not compile with optimisation turned on, so they’re by default quite a bit slower than they need to be.
I have not suffered any issues due to pyenv compiling. In our prod boxes we compile Python ourselves anyway. It's a really trivial build process tbh.
I generally like bite codes newsletter but this one was a miss for me.
I’m sure he is trying to promote some sort of work flow in that article but I don’t understand which.
I’m sure he is trying to promote some sort of work flow in that article but I don’t understand which.
I call it the curse of Python.
When the God Of Programming made Python, all other languages were jealous of its elegance, simplicity and intuitive beauty. When the other programming languages went to complain he said..."Wait until you see what package systems I will give them...They will never get an environment properly setup..." :-)
When the God Of Programming made Python, all other languages were jealous of its elegance, simplicity and intuitive beauty. When the other programming languages went to complain he said..."Wait until you see what package systems I will give them...They will never get an environment properly setup..." :-)
And then the Devil made CUDA…
”The CUDA version on your system does not match the CUDA version the realm of mortal men was compiled with”
”The CUDA version on your system does not match the CUDA version the realm of mortal men was compiled with”
Oh man, you just confirm that I’m not crazy. For every single time I use pyenv it just downloads source and build. I got tangled into so many issues such as TLS headers in my AL2 boxes. I thought the whole thing is about getting the official binary and install and not compile from source even for a 2 year old release.
Data point: pyenv works just fine for me. It helps me with installing/managing many python versions (multiple years).
I guess, I'm in the "expert" category. I'm saying it so that people won't be afraid to try it.
I guess, I'm in the "expert" category. I'm saying it so that people won't be afraid to try it.
Python is quite self contained, it's not a chore to build it.
Can I ask - why the dislike for virtualenvwrapper? If you are saying that it can have occasional problems, I agree with you. But it makes the process so much simpler. The occasional problems I’ve had (had an issue once or twice deleting a venv) pales in comparison to the advantage I get in remembering a couple fast commands. Is there something better or am I relegated to “source what/directory/was/it/again/bin/activate?
> See also: "Why not tell people to "simply" use pyenv, poetry or anaconda"
> https://www.bitecode.dev/p/why-not-tell-people-to-simply-use
Just curious: what are the downsides of poetry installed with pipx? The article mentions having to install poetry in another venv, but that's hardly an issue with pipx (you just add an 'x' after 'pip'), and installing pipx is as straight-forward as it can be.
> https://www.bitecode.dev/p/why-not-tell-people-to-simply-use
Just curious: what are the downsides of poetry installed with pipx? The article mentions having to install poetry in another venv, but that's hardly an issue with pipx (you just add an 'x' after 'pip'), and installing pipx is as straight-forward as it can be.
I eventually landed on pipx as fighting with Pyenv and Anaconda - via Miniconda - was an exercise in frustration. There's some mucking about in `$HOME/.local`, but this is mostly self-contained and not a huge chore to keep running.
Coming from the Homebrew/Ruby ecosystem - Hey @mikemcquaid - installing a entirely separate package manager just to deal with a few projects felt like the wrong thing to do.
Occasionally, I have still needed to compile Python myself in order to get things to work, which isn't guaranteed not to blow up w/ `brew`, but this has become far less common of late.
Coming from the Homebrew/Ruby ecosystem - Hey @mikemcquaid - installing a entirely separate package manager just to deal with a few projects felt like the wrong thing to do.
Occasionally, I have still needed to compile Python myself in order to get things to work, which isn't guaranteed not to blow up w/ `brew`, but this has become far less common of late.
Agreed pipx solves a lot of packaging issues with no downside to speak of. Not just with poetry but also for tools like virtualenv, ruff and black and non-dev command line tools.
> People that love pyenv are so adamant at telling everybody they should use it. How it solved all their problems, and got rich and lost 5 pounds.
> It doesn’t support Windows. That’s game over right there, for half of the community.
What do you mean by this? I use pyenv on windows all the time.
> It doesn’t support Windows. That’s game over right there, for half of the community.
What do you mean by this? I use pyenv on windows all the time.
Why does it compile Python on your machine?
What's the problem? The repo clearly states that you need to install such and such build prerequisites.
Also, the issues the article you link are things that damn near every programmer will eventually need to know. Things like PATH are IMO the basics -- if you don't understand this or how to `$packageManager install` something, you're gonna have a rough time programming in general, regardless of language.
Also, the issues the article you link are things that damn near every programmer will eventually need to know. Things like PATH are IMO the basics -- if you don't understand this or how to `$packageManager install` something, you're gonna have a rough time programming in general, regardless of language.
Tools you can use to make sure the Python program you wrote keeps working: requirements.txt, pip, pipenv, pyenv, virtualenv, pyenv-virtualenv, virtualenvwrapper, pyenv-virtualenvwrapper, venv, pyvenv, conda, miniconda, poetry, docker, nix.
Which ones did I miss? Which of them actually ensure your program always works the same as when you first wrote it, without asterisks?
Which ones did I miss? Which of them actually ensure your program always works the same as when you first wrote it, without asterisks?
You've missed: pdm, uv, pip-tools, pipx, rye, and probably some others.
Only pdm and poetry generate cross-platform lock files by default as far as I know, but there are a lot of people trying to solve this problem right now.
It's not an easy problem to solve. Python's package management predates package managers from most other programming languages and Python itself predates Linux. There is a lot of baggage so change is very slow.
Only pdm and poetry generate cross-platform lock files by default as far as I know, but there are a lot of people trying to solve this problem right now.
It's not an easy problem to solve. Python's package management predates package managers from most other programming languages and Python itself predates Linux. There is a lot of baggage so change is very slow.
Shameless plug: don't forget the wonderful pip-chill for simplifying gigantic requirements files (and for stripping out version numbers to make canaries easier to do).
python -m venv myenv
. myenv/bin/activate
<install reqs>
<your program>
> Ensures you have the same python environment.
But not the same Python runtime, which does not fulfil the request of guaranteeing it works the same later without asterisks.
But not the same Python runtime, which does not fulfil the request of guaranteeing it works the same later without asterisks.
I'm not sure what you might mean by runtime, but a python venv does make sure you're using the same runtime.
There are symlinks in the environments bin directory to the specific runtime. If you're plonking a python3.12 binary over the top of the python3.11 one, then yeah, you'll be using a different one, but that'd be an issue nomatter what you're using.
Just don't uninstall your python binaries and you're fine.
If you want to package everything up into am image or a zip, you can do that too if you want, but in my 10+ year career I've not had much of an issue just using boring venvs.
Just don't uninstall your python binaries and you're fine.
If you want to package everything up into am image or a zip, you can do that too if you want, but in my 10+ year career I've not had much of an issue just using boring venvs.
> but a python venv does make sure you're using the same runtime.
It does not.
> There are symlinks in the environments bin directory to the specific runtime.
Precisely. They symlink to a path. Which means that if you have a certain version of Python at one location (let’s say /usr/bin/python3) and later update that (let’s say by upgrading macOS and the Xcode developer tools), the same virtual environment will point to a different version of Python.
> Just don't uninstall your python binaries and you're fine.
That’s not a practical solution. Sometimes you don’t have a choice, as demonstrated above. By that logic one could say “don’t change anything about your system and you don’t even need virtual environments”. Which is somewhat true, but also profoundly unhelpful.
The point of the question was to reproduce the same thing without asterisks and you’ve introduced a major one and called it a day.
It does not.
> There are symlinks in the environments bin directory to the specific runtime.
Precisely. They symlink to a path. Which means that if you have a certain version of Python at one location (let’s say /usr/bin/python3) and later update that (let’s say by upgrading macOS and the Xcode developer tools), the same virtual environment will point to a different version of Python.
> Just don't uninstall your python binaries and you're fine.
That’s not a practical solution. Sometimes you don’t have a choice, as demonstrated above. By that logic one could say “don’t change anything about your system and you don’t even need virtual environments”. Which is somewhat true, but also profoundly unhelpful.
The point of the question was to reproduce the same thing without asterisks and you’ve introduced a major one and called it a day.
The binary installs aren't that generic in the machines I've used (mac and debian based, so covers a fair bit).
It's python3.11, not python3, and the python3 "executable" is, itself a symlink to the particular binary (in this example python3.11). Upgrading just changes the symlink, which wouldn't affect the venv, which isn't using python3, it's using python3.11.
I didn't introduce anything, I explained how the links are to the versioned binaries, which isn't what you're stating is happening.
Edit to add: as someone else also points out, you don't even have to use the symlinked versions, you can use --copies
It's python3.11, not python3, and the python3 "executable" is, itself a symlink to the particular binary (in this example python3.11). Upgrading just changes the symlink, which wouldn't affect the venv, which isn't using python3, it's using python3.11.
I didn't introduce anything, I explained how the links are to the versioned binaries, which isn't what you're stating is happening.
Edit to add: as someone else also points out, you don't even have to use the symlinked versions, you can use --copies
> It's python3.11, not python3, and the python3 "executable" is, itself a symlink to the particular binary (in this example python3.11).
Not for the example I gave. If you’re seeing Python 3.11, you’re definitely not using the /usr/bin/python3 on macOS with is made available by the Xcode CLI tools. That’s at 3.9.6 even on Sonoma.
> as someone else also points out, you don't even have to use the symlinked versions, you can use --copies
Which, again, doesn’t work for the stated case.
https://news.ycombinator.com/item?id=39818732
Not for the example I gave. If you’re seeing Python 3.11, you’re definitely not using the /usr/bin/python3 on macOS with is made available by the Xcode CLI tools. That’s at 3.9.6 even on Sonoma.
> as someone else also points out, you don't even have to use the symlinked versions, you can use --copies
Which, again, doesn’t work for the stated case.
https://news.ycombinator.com/item?id=39818732
Sorry, but yes I am using macOS for work. I use homebrew as xcode is the issue here, not python or macOS. There are solutions to your problem, you just seem resistant to using them.
> There are solutions to your problem, you just seem resistant to using them.
No, it is you who are failing to understand. I’m describing to you a real scenario, but it’s not one that bothers me. I don’t need you to come up with a solution and didn’t ask you for it. You need to understand not everyone has the same requirements and tradeoffs you do.
No, it is you who are failing to understand. I’m describing to you a real scenario, but it’s not one that bothers me. I don’t need you to come up with a solution and didn’t ask you for it. You need to understand not everyone has the same requirements and tradeoffs you do.
You can use --copies if you don't want symlinks [0].
[0]: https://docs.python.org/3/library/venv.html
[0]: https://docs.python.org/3/library/venv.html
Not with the macOS Python from the Xcode developer tools:
> Error: This build of python cannot create venvs without using symlinks
> Error: This build of python cannot create venvs without using symlinks
Don't use that one
— Doctor, it hurts when I raise may arm.
— Then don’t raise your arm.
Like I said in the original comment you replied to and are now ignoring:
> That’s not a practical solution. Sometimes you don’t have a choice, as demonstrated above. By that logic one could say “don’t change anything about your system and you don’t even need virtual environments”. Which is somewhat true, but also profoundly unhelpful.
— Then don’t raise your arm.
Like I said in the original comment you replied to and are now ignoring:
> That’s not a practical solution. Sometimes you don’t have a choice, as demonstrated above. By that logic one could say “don’t change anything about your system and you don’t even need virtual environments”. Which is somewhat true, but also profoundly unhelpful.
No what you're saying is "Doc I want to only live on bread", to which a reasonable response is "it sounds like you're in jail and have bigger problems."
I'm not ignoring you; I just don't think your use case of "I want to capture the Python from Xcode in my dev env so it's resilient to changes and upgrades" is something anyone wants (or should want) to do. Do you really want to ship on that version? Are you asking your users to install Xcode?
> That’s not a practical solution. Sometimes you don’t have a choice, as demonstrated above. By that logic one could say “don’t change anything about your system and you don’t even need virtual environments”. Which is somewhat true, but also profoundly unhelpful.
In other words: when are you forced to use Xcode's Python for development, and why would that be a good idea? I'm earnestly asking; there may be reasons; I'm just not aware of any.
I'm not ignoring you; I just don't think your use case of "I want to capture the Python from Xcode in my dev env so it's resilient to changes and upgrades" is something anyone wants (or should want) to do. Do you really want to ship on that version? Are you asking your users to install Xcode?
> That’s not a practical solution. Sometimes you don’t have a choice, as demonstrated above. By that logic one could say “don’t change anything about your system and you don’t even need virtual environments”. Which is somewhat true, but also profoundly unhelpful.
In other words: when are you forced to use Xcode's Python for development, and why would that be a good idea? I'm earnestly asking; there may be reasons; I'm just not aware of any.
> I just don't think your use case of "I want to capture the Python from Xcode in my dev env so it's resilient to changes and upgrades" is something anyone wants (or should want) to do.
Then you are wrong. Simple as that. I’m describing a very real scenario.
> Do you really want to ship on that version?
Holy moly, is it really that hard to understand the difference between wanting and having to? For the use case, and older version in a consistent place which is easy to install is the best solution.
> Are you asking your users to install Xcode?
Triggering an Xcode CLI tools installation is simple and done graphically. And it’s one step removed from installing Homebrew or pyenv, which both need them (even for the scripted installation, pyenv requires git).
> In other words: when are you forced to use Xcode's Python for development, and why would that be a good idea?
See, for a moment there you understood it’s about users, not just your own dev environment, but then went back. Unfortunately, after this conversation with you two I no longer have the energy to go through it in detail in an uphill explanation. Another time, maybe.
Then you are wrong. Simple as that. I’m describing a very real scenario.
> Do you really want to ship on that version?
Holy moly, is it really that hard to understand the difference between wanting and having to? For the use case, and older version in a consistent place which is easy to install is the best solution.
> Are you asking your users to install Xcode?
Triggering an Xcode CLI tools installation is simple and done graphically. And it’s one step removed from installing Homebrew or pyenv, which both need them (even for the scripted installation, pyenv requires git).
> In other words: when are you forced to use Xcode's Python for development, and why would that be a good idea?
See, for a moment there you understood it’s about users, not just your own dev environment, but then went back. Unfortunately, after this conversation with you two I no longer have the energy to go through it in detail in an uphill explanation. Another time, maybe.
>> I just don't think your use case of "I want to capture the Python from Xcode in my dev env so it's resilient to changes and upgrades" is something anyone wants (or should want) to do.
> Then you are wrong. Simple as that. I’m describing a very real scenario.
>> Do you really want to ship on that version?
> Holy moly, is it really that hard to understand the difference between wanting and having to? For the use case, and older version in a consistent place which is easy to install is the best solution.
Tone is hard on the internet, but I'm honestly trying to understand your use case. The way I understand it now is "I want to be able to develop Python programs locally using only Xcode's Python." I'm still not sure why you want to (generally when people ship Python programs they bundle a runtime) but let's set that to the side. For this use case, I don't understand why symlinks don't work for you. Xcode installs to versioned folders so you can have multiple Xcode installs side by side, that way new versions won't overwrite things. I'm obviously not an expert here though; am I missing something?
> Then you are wrong. Simple as that. I’m describing a very real scenario.
>> Do you really want to ship on that version?
> Holy moly, is it really that hard to understand the difference between wanting and having to? For the use case, and older version in a consistent place which is easy to install is the best solution.
Tone is hard on the internet, but I'm honestly trying to understand your use case. The way I understand it now is "I want to be able to develop Python programs locally using only Xcode's Python." I'm still not sure why you want to (generally when people ship Python programs they bundle a runtime) but let's set that to the side. For this use case, I don't understand why symlinks don't work for you. Xcode installs to versioned folders so you can have multiple Xcode installs side by side, that way new versions won't overwrite things. I'm obviously not an expert here though; am I missing something?
This is more like:
- Doctor, it hurts when I hit my head on the wall
- Then don't hit your head on the wall
You have solutions available, use them
- Doctor, it hurts when I hit my head on the wall
- Then don't hit your head on the wall
You have solutions available, use them
Apparently, “sometimes you don’t have a choice” is a foreign language.
— Hey, so you know how we have this requirement, which came about after years of dealing with and understanding a problem and the available solutions?
— Yes, what about it?
— Well, a random commenter on Hacker News who has zero context of the problem has suggested we use a method we already found inadequate for our specific use case.
— Oh wow, in that case let’s replace the whole system right now.
— Hey, so you know how we have this requirement, which came about after years of dealing with and understanding a problem and the available solutions?
— Yes, what about it?
— Well, a random commenter on Hacker News who has zero context of the problem has suggested we use a method we already found inadequate for our specific use case.
— Oh wow, in that case let’s replace the whole system right now.
I think people are really looking for some examples of scenarios where this might be a requirement and your time might be better spent providing one vs. fighting with them. Maybe something like an airgapped gov network with very specific approved and audited software?
This is not the way you're supposed to be using pyenv
You're supposed to install a specific version of python in a specific place, with a specific name. Say, /usr/local/python-3.10.6
Use pyenv to use that python. Control that by creating a `.python-version` file that says 3.10.6
You now have a project that uses 3.10.6. Unless, of course, somebody installs a different version in that path - at which point you've got bigger issues
Using pyenv to use `/usr/bin/python3` and hoping for the best misses the point
You're supposed to install a specific version of python in a specific place, with a specific name. Say, /usr/local/python-3.10.6
Use pyenv to use that python. Control that by creating a `.python-version` file that says 3.10.6
You now have a project that uses 3.10.6. Unless, of course, somebody installs a different version in that path - at which point you've got bigger issues
Using pyenv to use `/usr/bin/python3` and hoping for the best misses the point
> This is not the way you're supposed to be using pyenv
Because that’s not what the conversation is about. It’s about virtual environments.
Because that’s not what the conversation is about. It’s about virtual environments.
If you attempt to reproduce a project in a repo that only includes source and requirements.txt you will not have metadata about the version of Python used.
It seems that there is a way to do this in pip but I don’t think it is widely used: https://stackoverflow.com/questions/19559247/requirements-tx...
It seems that there is a way to do this in pip but I don’t think it is widely used: https://stackoverflow.com/questions/19559247/requirements-tx...
>The other part is OS state
And that's the whole problem. "The same Python environment" doesn't mean much because it underspecifies the full runtime dependency chain.
And that's the whole problem. "The same Python environment" doesn't mean much because it underspecifies the full runtime dependency chain.
It's inherit problem for all languages based on some runtime. Java, PHP, Ruby, whatever have the same issue. And TBH it's not very major issue to start with.
No other solution besides fixed OS (containers, nix) could solve this problem.
No other solution besides fixed OS (containers, nix) could solve this problem.
>No other solution besides fixed OS (containers, nix) could solve this problem.
You don't have to use NixOS to use Nix. You can install it on any Linux distro, or MacOS, and use it as your build system.
You don't have to use NixOS to use Nix. You can install it on any Linux distro, or MacOS, and use it as your build system.
This approach is full of footguns.
Forget to install virtualenv? Have fun getting rid of packages in your global system.
Sub-dependencies? You'll have to think of a strategy for it. Most people don't and the reproducibility of their project suffers.
Developer vs production dependencies ? You'll have to think of a strategy for it.
And I think I'm forgetting one, but that's already enough.
Forget to install virtualenv? Have fun getting rid of packages in your global system.
Sub-dependencies? You'll have to think of a strategy for it. Most people don't and the reproducibility of their project suffers.
Developer vs production dependencies ? You'll have to think of a strategy for it.
And I think I'm forgetting one, but that's already enough.
In this case you don’t count the standard library as part of your Python environment. Since Python version is not specified neither is the standard library.
I am always curious how many people, outside of those building code for third party clients, actually hit this problem? In the 10+ years of using Python I have never had a problem using the core tools. The ecosystem is far from perfect but it has never cause me a problem.
Edit: Wow y'all are some sour people for voting down this question. I truly wonder how often people run into this problem compared to just complaining about it.
Edit: Wow y'all are some sour people for voting down this question. I truly wonder how often people run into this problem compared to just complaining about it.
Just a requirements.txt that you can install with pip in a venv works 95% of the time. Unless you need a specific python version, which you only figure out halfway through by reading the documentation. Or you are using pytorch, because on windows and linux the version on pypi lacks GPU support, but there's no good way to encode the download url in your dependencies. Or when people don't properly maintain the requirements.txt, because you are supposed to somehow manually keep it in sync.
I run into some kind of packaging or dependency problem for basically every non-trivial project. Hence the many "solutions", but somehow most end up worse than the problems they attempt to solve
I run into some kind of packaging or dependency problem for basically every non-trivial project. Hence the many "solutions", but somehow most end up worse than the problems they attempt to solve
Working in CI in a company that has couple dozens of Python packages: I'd say about once a week. There are some weeks with no incidents, and there are weeks when everything is broken for many days straight.
NB. The latest incident was Friday when I've discovered that some CI pipeline ran `setup.py install` that down the lane invoked easy_install, which doesn't have a policy of ignoring bizarre versions s.a. X.Y.Zrc1 or X.Y.Zb2 etc. It ran aground when it was trying to install scikit-learn which wanted NumPy >=X.Y.Z, but it already installed X.Y.Zb1, and it didn't realize that this version should be OK (also, it shouldn't have installed non-release versions anyways).
NB. The latest incident was Friday when I've discovered that some CI pipeline ran `setup.py install` that down the lane invoked easy_install, which doesn't have a policy of ignoring bizarre versions s.a. X.Y.Zrc1 or X.Y.Zb2 etc. It ran aground when it was trying to install scikit-learn which wanted NumPy >=X.Y.Z, but it already installed X.Y.Zb1, and it didn't realize that this version should be OK (also, it shouldn't have installed non-release versions anyways).
I just hit it a few days ago. 3rd party script that I did not write. The dev has lost interest in it 3 years ago. It is written for 2.7.x (and it is a nontrivial amount of code). So my choices are some sort of pyenv thing (or one of them), or fix up the script myself so it runs on 3.12. I lucked out and someone else had already done the second thing. One project I worked on was 2.4.x. If they ever have to update that (which I hope they have) to something more recent I could see them doing it until they port it over. That is not even a lib or anything that is just the main prog.
download python 2.7 and run it? no pyenv needed
2.7 has been EOL for five years. It doesn’t get security updates nor bug fixes.
Python Foundation support ended that long ago, but vendors maintained it for longer. Red Hat support for it through RHEL 7 ends 30 June 2024, however, and my guess is they’re the last ones.
For this use case it is probably fine to have 2.7.x installed to use for this. Just annoyed. The downside to having it that way is this weird stray python installed. Would rather use something to switch it around to make it easier to keep track. But in the end it didnt mater as someone else had already updated it and I did not have to bother. Maybe next time :)
If I were writing applications only for myself it'd be fine, but I've definitely run into issues when writing code with others - teammates, open source repos, etc.
Just getting me and one coworker in sync on our dev machines often causes problems. Even more when we try to deploy to a cloud service which relies on pip.
I am not sure about what 'core' means in this context.
In my edperience I had many many problems with OS packages vs pip installed ones. There were really strange dependency issues.
In some cases I encountered dependency hell.
Even if somebody said to me that the core means also using virtualized en I would disagree as places over Internet not always explicitly guide you in that route.
In my edperience I had many many problems with OS packages vs pip installed ones. There were really strange dependency issues.
In some cases I encountered dependency hell.
Even if somebody said to me that the core means also using virtualized en I would disagree as places over Internet not always explicitly guide you in that route.
> Which of them actually ensure your program always works the same as when you first wrote it, without asterisks?
Nix does, if you want to actually invest the effort into the "without asterisks" part.
Nix does, if you want to actually invest the effort into the "without asterisks" part.
You didn’t miss Poetry, but I have to say up until I started using Poetry python in large projects was a pain in the tooling department to setup and maintain over longer periods of time.
It’s no panacea, but feels more stable and usable (especially from onboarding new team members PoV) than other tooling I’ve tried
It’s no panacea, but feels more stable and usable (especially from onboarding new team members PoV) than other tooling I’ve tried
What does poetry deliver beyond what pyenv gets you? I haven't used either tool super extensively.
pyenv only manages python versions, while poetry manages dependencies and virtual environments. They are complimentary, but do not overlap.
To add to this, with poetry, you basically do `poetry env use python3.12` to create a python virtualenv on python3.12 (you can use whatever python version pyenv supports, doesn't even have to be CPython).
The generated virtual env name is a little wonky. I'm not sure exactly what the scheme is, but it's basically `$(package)-$(some sort of hash?)-$(pyversion)`. I can't speak for all tools, but at least VS Code detects the poetry env and suggests it (and indicates it as a Poetry env) when you go to configure the project interpreter.
The generated virtual env name is a little wonky. I'm not sure exactly what the scheme is, but it's basically `$(package)-$(some sort of hash?)-$(pyversion)`. I can't speak for all tools, but at least VS Code detects the poetry env and suggests it (and indicates it as a Poetry env) when you go to configure the project interpreter.
You can use the poetry config to enable poetry putting the virtualenv into the project folder itself, which allows most IDEs to discover it.
Only Nix does. The builds are deterministic, transitively all the way back to gcc and libc.
What package managers are people using in other languages to make sure that software "always works the same as when you first wrote it, without asterisks"? I'd like to understand how they solve the "package no longer exists in a central registry" problem.
This is not as much about a package manager as it is about conventions and necessary dependencies.
Standards ensure that going forward language semantics and syntax don't change. Having minimal dependencies ensures program longevity. Package manager cannot solve these problems, no matter how good it is at its job.
Python doesn't have a standard, it's heavily reliant on dependencies which are very plentiful and similarly unregulated. A program written in C that uses only functionality described in some POSIX standard will endure decades unmodified. Even Python helloworld program went stale sometime ago, even though it's just one line.
Standards ensure that going forward language semantics and syntax don't change. Having minimal dependencies ensures program longevity. Package manager cannot solve these problems, no matter how good it is at its job.
Python doesn't have a standard, it's heavily reliant on dependencies which are very plentiful and similarly unregulated. A program written in C that uses only functionality described in some POSIX standard will endure decades unmodified. Even Python helloworld program went stale sometime ago, even though it's just one line.
> I'd like to understand how they solve the "package no longer exists in a central registry" problem.
This is, of course, an infrastructure/maintenance issue as much as a package manager design issue. But in Nix's case, the public 'binary cache' (for Nixpkgs/NixOS) of build outputs includes not only final build outputs but also the source tarballs that go into them. As Nix disallows network access at build time, all dependencies are represented this way, including jar files or source tarballs, or whatever— Nix itself must be the one to fetch your dependencies. Consequently, everything you fetch from the Internet for your build is a kind of intermediary Nix build that can be cached using the usual Nix tools. The Nix community's public cache has a policy of retaining copies of upstream sources forever (there is recently talk of limiting storage of the final built packages to a retention period of only 2 years, but sources will continue to be retained indefinitely. So far the cache reaches back to its inception around a decade ago.)
Taken together, these things mean that when a project disappears entirely from GitHub or Maven Central or whatever, people building against old versions of it with Nix/Nixpkgs don't even notice. Nix just fetches those upstream sources from the public cache without even reaching out to that central repository from which those sources have been removed.
For private use cases where your project and its dependencies won't be mirrored to the public cache of Nixpkgs builds, you can achieve the same effect by running your own cache or paying a hosted service to do that.
For builds outside the Nix universe, you can make special arrangements for each type of package your various builds fetch, and mirroring those repos. Then configure your builds to pull from your mirrors instead of the main/public ones.
This is, of course, an infrastructure/maintenance issue as much as a package manager design issue. But in Nix's case, the public 'binary cache' (for Nixpkgs/NixOS) of build outputs includes not only final build outputs but also the source tarballs that go into them. As Nix disallows network access at build time, all dependencies are represented this way, including jar files or source tarballs, or whatever— Nix itself must be the one to fetch your dependencies. Consequently, everything you fetch from the Internet for your build is a kind of intermediary Nix build that can be cached using the usual Nix tools. The Nix community's public cache has a policy of retaining copies of upstream sources forever (there is recently talk of limiting storage of the final built packages to a retention period of only 2 years, but sources will continue to be retained indefinitely. So far the cache reaches back to its inception around a decade ago.)
Taken together, these things mean that when a project disappears entirely from GitHub or Maven Central or whatever, people building against old versions of it with Nix/Nixpkgs don't even notice. Nix just fetches those upstream sources from the public cache without even reaching out to that central repository from which those sources have been removed.
For private use cases where your project and its dependencies won't be mirrored to the public cache of Nixpkgs builds, you can achieve the same effect by running your own cache or paying a hosted service to do that.
For builds outside the Nix universe, you can make special arrangements for each type of package your various builds fetch, and mirroring those repos. Then configure your builds to pull from your mirrors instead of the main/public ones.
Dotnet uses Nuget[1]. Packages in the system are immutable, & never changing. They can be unlisted, but never deleted (except in limited & extreme cases, like malware), which means even if a package maintainer stops publishing new versions to the repository, existing packages will continue to be publicly available for as long as Microsoft continues to exist.
[1] https://www.nuget.org/
[1] https://www.nuget.org/
Often a package says it works with OS version X, and not X+1. It may be true or false. But what you described does not solve either version of that problem.
Or the "says it will work with > X," but doesn't.
Or the "says it will work with > X," but doesn't.
I think you're referring to vendoring dependencies? In python/pip for example, you can download the source for a package and point to the folder directly as a dependency instead of the version or a git URL. Most package managers/languages support some version of that. I suppose if you wanted to vendor all dependencies by default, keep them updated etc it would take a little more scripting or extra tools.
> Which of them actually ensure your program always works the same as when you first wrote it, without asterisks?
There aren't such tools. Python not being a standard and heavily reliant on the OS that runs it and on third-party components that are also not standard leaves you with no choice by to "be at the wheel" all the time. Virtually anything written in Python will go stale in a mater of few years. In other words, you need to constantly update and test as the environment changes just to stand still.
There aren't such tools. Python not being a standard and heavily reliant on the OS that runs it and on third-party components that are also not standard leaves you with no choice by to "be at the wheel" all the time. Virtually anything written in Python will go stale in a mater of few years. In other words, you need to constantly update and test as the environment changes just to stand still.
> Which ones did I miss?
Using python and python libraries only from your package manager (like APT) for a specific OS version.
Micromamba as well, lightweight version of conda/miniconda.
Using python and python libraries only from your package manager (like APT) for a specific OS version.
Micromamba as well, lightweight version of conda/miniconda.
I have been looking at Rye, I like it so far.
The other one is Docker, which some people hate. I don't mind it, in some cases I prefer it.
The other one is Docker, which some people hate. I don't mind it, in some cases I prefer it.
None. The software container image is the best bet but you need to keep the image and not only the building scripts.
This is not true. Nix actually solves the problem. If you package with Nix, you'll get the exact same version of Python with the exact same version of dependencies, including the exact same version of system libraries. Your build will work in a year just as it does today. You don't even need to keep any binary artifacts.
Of course this doesn't come free: packaging with Nix may involve nontrivial effort for some projects.
Of course this doesn't come free: packaging with Nix may involve nontrivial effort for some projects.
How does nix solves the problem of Python dependencies not specifying their dependencies well or exhaustively? Do you have to find them out and fix them manually or can you generate a lock file and hope for the best?
Ultimately the true bellwether of compatibility is the package's own test suite, which we try to get working (and integrated into the build process) for as many packages as possible. For packages which have poor/non-existent test suites, often a downstream package's test suite will expose compatibility problems (we've found bugs in zlib security patches using curl's test suite for instance).
nixpkgs maintainers are frequently the first to notify a project author of incompatibility with new versions of another package.
nixpkgs maintainers are frequently the first to notify a project author of incompatibility with new versions of another package.
It depends. If the dependency is in nixpkgs (Nix' own package repository), someone will have done the work of figuring out the dependencies properly and you can just use that. If it's not, you can either do that yourself and pin particular versions, or you can integrate with a tool like Poetry: Poetry could generate a lockfile for you that you then reference from Nix to get the versions of Python packages. You'd still need to specify any native dependencies manually, though.
> How does nix solves the problem of Python dependencies not specifying their dependencies well or exhaustively?
In Nix lingo, a package successfully building or running against an implicit or unmanaged dependency is called 'impurity'. To help keep builds 'pure', Nix builds everything in a sandbox and does various other tricks to ensure that a package being built can't/doesn't find any dependencies at build time that you don't explicitly tell Nix to include in the build environment.
(This is also increasingly how Linux distros build their packages, to solve the same problem.)
For the most part, if building (and running the test suite during the build) a Python package with Nix succeeds, you can be confident that you've got all the dependencies sorted out— even the ones upstream forgot to tell you about.
> Do you have to find them out and fix them manually or can you generate a lock file and hope for the best?
At install time, you can be confident that Nix will bring along all of the system-level dependencies your Python package needs. You don't have to find and fill any gaps at install time 10 years from now or whatever.
When you're writing your Nix package for the first time, you'll be doing a mix of generating the Nix code that defines your package from some upstream, Python-specific lockfile and making manual corrections when the build fails. Nix doesn't have any magic for figuring out dependencies that are left out of poetry.lock or whatever, or for disambiguating guaranteed-compatible exact versions from requirements.txt.
In Nix lingo, a package successfully building or running against an implicit or unmanaged dependency is called 'impurity'. To help keep builds 'pure', Nix builds everything in a sandbox and does various other tricks to ensure that a package being built can't/doesn't find any dependencies at build time that you don't explicitly tell Nix to include in the build environment.
(This is also increasingly how Linux distros build their packages, to solve the same problem.)
For the most part, if building (and running the test suite during the build) a Python package with Nix succeeds, you can be confident that you've got all the dependencies sorted out— even the ones upstream forgot to tell you about.
> Do you have to find them out and fix them manually or can you generate a lock file and hope for the best?
At install time, you can be confident that Nix will bring along all of the system-level dependencies your Python package needs. You don't have to find and fill any gaps at install time 10 years from now or whatever.
When you're writing your Nix package for the first time, you'll be doing a mix of generating the Nix code that defines your package from some upstream, Python-specific lockfile and making manual corrections when the build fails. Nix doesn't have any magic for figuring out dependencies that are left out of poetry.lock or whatever, or for disambiguating guaranteed-compatible exact versions from requirements.txt.
Yeah, containers don't fully solve this problem.
We still need a generated lock file with every top level dependency and sub-dependencies locked down to their most precise version commit to version control so that when you build your image today or in 6 months you end up with the same result.
Using pip to freeze your dependencies and writing a tiny shell script to generate a lock file at build time is better than nothing to solve this problem with nothing more than pip. It's what I do in https://github.com/nickjj/docker-flask-example and https://github.com/nickjj/docker-django-example. It's not perfect but it solves 80% with minimal complexity.
We still need a generated lock file with every top level dependency and sub-dependencies locked down to their most precise version commit to version control so that when you build your image today or in 6 months you end up with the same result.
Using pip to freeze your dependencies and writing a tiny shell script to generate a lock file at build time is better than nothing to solve this problem with nothing more than pip. It's what I do in https://github.com/nickjj/docker-flask-example and https://github.com/nickjj/docker-django-example. It's not perfect but it solves 80% with minimal complexity.
I remember when I was a childish Perl fan trading insults with the childish Python fans of the time, one of the things "they" always chided "us" about was that we had too many different ways to accomplish the same thing...
Pyinstaller, provided you build it in a reasonably old glibc to avoid glibc incompatibilities. I know that is a caveat but it is a one time build time operation.
Doesn't Pyinstaller also require CPython to have been built with '--enable-shared'? I _think_ this is the default for most system-installed Pythons that I've seen, but it's not the default if you build Python from source or install via pyenv.
asdf can be used as an alternative to pyenv. (In fact, it is not only meant for the Python ecosystem, so it can also replace nvm and others.)
For me, the combination of asdf and Poetry has worked quite well recently: I use asdf to pin the Python & Poetry version and then use Poetry to pin everything else.
For me, the combination of asdf and Poetry has worked quite well recently: I use asdf to pin the Python & Poetry version and then use Poetry to pin everything else.
Point of clarification: asdf uses python-build which is from pyenv.
https://github.com/asdf-community/asdf-python?tab=readme-ov-...
https://github.com/asdf-community/asdf-python?tab=readme-ov-...
True. I mentioned asdf mainly because it can be used with many languages & tools and is not restricted to the Python ecosystem.
[deleted]
You only need pip and virtualenvwrapper
requirements.txt is a text file, not a separate tool...
if you want your program to work exactly as intended in the future, there are tools like py2exe and py2app for that.
requirements.txt is a text file, not a separate tool...
if you want your program to work exactly as intended in the future, there are tools like py2exe and py2app for that.
Don't need any wrappers either. venv has been built into python for ages.
yes but `workon` is too good not to have
Looks like what could be a shell alias to cd and activate? I always keep a few tabs open to my repos folder, so not sure it would help me much. Doesn't seem to support fish, though activate does.
I modified my bash prompt to detect and auto-activate Python environments, and show if one is currently active. Haven’t thought about activate in years. It’s great.
Yes, I did that on my work laptop with fish, since I only ever used it with the work project.
On my personal machine I don't bother with venvs, so also haven't thought about it in years. But was trying to figure out what the "workon" command did for GP.
On my personal machine I don't bother with venvs, so also haven't thought about it in years. But was trying to figure out what the "workon" command did for GP.
`workon` lists all virtualenvs you have in your virtualenvs folder and `workon foo` activates environment foo
this way I have all my envs in ~/virtual and all my projects in ~/projects and just `workon xyz` when I want to be in a certain venv for a given project (which doesn't always map one-to-one)
this way I have all my envs in ~/virtual and all my projects in ~/projects and just `workon xyz` when I want to be in a certain venv for a given project (which doesn't always map one-to-one)
While gross ugly Maven lets you specify the bytecode version and every JDK can produce backwards-compatible jars and has done for many years
You missed hatch, uv, rye, pdm.
I think rye hits most of the points pretty well, it ensures both python and package versions.
I think rye hits most of the points pretty well, it ensures both python and package versions.
Dependencies are for the weak. You only need Python. Just support a range of versions if you intend to distribute.
[deleted]
pyenv is all you need. pip is part of python. requirements.txt is just a convention used to store the list of stuff to hand to pip.
You missed Pipenv :-)
Edit: Oh I see you didn't. I can't read.
Edit: Oh I see you didn't. I can't read.
Go
You're missing Tox[0] & Nox[1]. Tox is actually quite nice when dealing with testing code across different versions of Python.
[0] https://tox.wiki [1] https://nox.thea.codes/en/stable/
[0] https://tox.wiki [1] https://nox.thea.codes/en/stable/
I really find it quite astounding just how much absolute nonsense python programmers are willing to put up with.
Needing more than one, MAYBE two versions of your language installed is insanity. If I build a Python widget and send it to my non-programmer coworker, there is exactly zero chance it will work until I walk over and manually set up the correct language. Instead I use a language that natively compiles to an exe.
Python is pretty neat, but the concept of a program that just works anywhere is so utterly alien that I genuinely cannot find any practical use for it.
I can't just install a pip package to use globally, I have to set up some goddamn virtual environment because everything is so brittle that nothing works without the exact correct language version.
It's like NPM all over again. Dependency hell is so bad that your package manager is now a dependency so we need a package manager manager to manage installing the correct package manager version to then install the correct packages.
Every single time I've come up to some Python thing or other, I spend about fifteen minutes fucking with it before giving up and using a tool built in a sane language.
The fact that I can't just send any random person a program and expect it to run at all is just lunacy. And no, compiled Python does not count because that's even more brittle tooling on top of all the other bullshit I have to deal with. If it doesn't work out of the box, it's a bad tool and I have far better things to do with my very valuable time.
Needing more than one, MAYBE two versions of your language installed is insanity. If I build a Python widget and send it to my non-programmer coworker, there is exactly zero chance it will work until I walk over and manually set up the correct language. Instead I use a language that natively compiles to an exe.
Python is pretty neat, but the concept of a program that just works anywhere is so utterly alien that I genuinely cannot find any practical use for it.
I can't just install a pip package to use globally, I have to set up some goddamn virtual environment because everything is so brittle that nothing works without the exact correct language version.
It's like NPM all over again. Dependency hell is so bad that your package manager is now a dependency so we need a package manager manager to manage installing the correct package manager version to then install the correct packages.
Every single time I've come up to some Python thing or other, I spend about fifteen minutes fucking with it before giving up and using a tool built in a sane language.
The fact that I can't just send any random person a program and expect it to run at all is just lunacy. And no, compiled Python does not count because that's even more brittle tooling on top of all the other bullshit I have to deal with. If it doesn't work out of the box, it's a bad tool and I have far better things to do with my very valuable time.
I've been programming in Python for years and I agree: Python isn't really for consumers. But hey it is used for LLMs that can generate React apps, so there's that.
I am only half-kidding. But it appears the industry as a whole is concentrating effort on fewer languages and Python is one of them. If I want to distribute my app - I write it in React, but I still often use python in the process: prototyping, asking GPT to rewrite X from python to typescript, making mock servers for tests / preparing data for unit tests.
I am only half-kidding. But it appears the industry as a whole is concentrating effort on fewer languages and Python is one of them. If I want to distribute my app - I write it in React, but I still often use python in the process: prototyping, asking GPT to rewrite X from python to typescript, making mock servers for tests / preparing data for unit tests.
It’s totally up to your knowledge and experience to produce code that supports multiple versions. I had no issue with doing it. That’s what you pay for dynamic language that evolves rapidly.
You're just using the wrong tool for the job.
Plug for Mise (https://mise.jdx.dev/) which is like asdf (which is like pyenv, but for any language you might want all in one tool) but written in Rust and without shims (by default) such that it’s much faster at running the activated runtime. It installs runtimes in parallel and will even download the correct plugin if you don’t already have it installed. It can also replace direnv, which I haven’t done because I’m using it as a drop-in replacement for asdf with the .tool-versions files my team already has in every repo (it supports .python-version as well if you’ve been using pyenv). It works great as a “better asdf”/“better pyenv” for my purposes.
https://asdf-vm.com/ ASDF is better because it works with many more languages, other than only Python, like Rust, Go, Node, etc, and other tools, such as AWS/Google/Firebase/Azure CLIs.
You should check out mise (https://github.com/jdx/mise), it is very similar to ASDF (multi langs) but written in Rust instead of Bash, and doesnt use the shim technique that ASDF does. My shell startup went from a couple hundred ms to a couple dozen ms.
Yeah, ASDF is fantastic, I'm glad stuff like pyenv and rvm existed and paved the way for it but ASDF is the way to go nowadays.
asdf for python is a wrapper around pyenv, it didn't "pave the way", it's an important part of asdf.
(Also I like mise better currently: https://github.com/jdx/mise)
(Also I like mise better currently: https://github.com/jdx/mise)
Curious what you like about mise! I've been a strong advocate for asdf for a few years now. Aside from smooshing together tools that are often used together, but not always (direnv, make), I can't see what mise adds to the picture. And I like still having direnv for what direnv does and make for make does, rather than an all-in-one.
the asdf CLI has really sharp edges and feels kind of aggressively unhelpful.
Even if you just use mise as an asdf alternative, it has a nicer CLI and interacts with the same plugin ecosystem more smoothly.
(I've found it to be an improvement on direnv, but I still use make)
the comparison to asdf page has more: https://mise.jdx.dev/dev-tools/comparison-to-asdf.html
Even if you just use mise as an asdf alternative, it has a nicer CLI and interacts with the same plugin ecosystem more smoothly.
(I've found it to be an improvement on direnv, but I still use make)
the comparison to asdf page has more: https://mise.jdx.dev/dev-tools/comparison-to-asdf.html
Damn. I'm convinced. The asdf CLI was always a bit of a pain ... mostly I don't interact with it that often. But mise seems light years better designed. Thanks!
Well that's excellent!
I used to be all-in for pyenv and rbenv both but have loved using asdf so much more.
[deleted]
After some trial and error I have now settled on the following "stack": 1)I manage python versions with pyenv 2)For each new project I create a new virtualenv with venv "PYENV_VERSION=3.10 python -m venv .venv" 3)Then I start jump into the venv and initiate the project with Poetry ("poetry init -n") and manage dependencies with Poetry.
If I'm keeping the projects under Dropbox, then I'll just add the .venv folder to the ignore list ("attr -s com.dropbox.ignored -V 1 .venv") to reduce the amount of files that need to be synced. If I need to get back to old project, I simply recreate the venv and install dependecies using Poetry.
A good habit would be to add a .pythonversion file on the project folder. Pyenv can pick that up and it is then obvious which version was used for the project.
I do development on Windows, but run all the Python stuff on WSL2 and use VS code for notebooks and everything. One thing I haven't looked at is using pipx to manage Poetry. Now I simply have one Poetry version installed.
Pyenv has worked fine once I managed to collect all the necessary dependencies to Ubuntu. For me it's not just to switch between Python 3.x. I like the fact that you can easily have different patch versions and match exactly the version I would use in production (via containers).
If I'm keeping the projects under Dropbox, then I'll just add the .venv folder to the ignore list ("attr -s com.dropbox.ignored -V 1 .venv") to reduce the amount of files that need to be synced. If I need to get back to old project, I simply recreate the venv and install dependecies using Poetry.
A good habit would be to add a .pythonversion file on the project folder. Pyenv can pick that up and it is then obvious which version was used for the project.
I do development on Windows, but run all the Python stuff on WSL2 and use VS code for notebooks and everything. One thing I haven't looked at is using pipx to manage Poetry. Now I simply have one Poetry version installed.
Pyenv has worked fine once I managed to collect all the necessary dependencies to Ubuntu. For me it's not just to switch between Python 3.x. I like the fact that you can easily have different patch versions and match exactly the version I would use in production (via containers).
Odd to me that you don’t let poetry create the venv? Why do this separately?
Our flow is similar: pyenv (windows and linux), pipx, poetry.
We’ve also defaulted poetry to utilize the current global version of Python and build the venv within the project folder.
Our flow is similar: pyenv (windows and linux), pipx, poetry.
We’ve also defaulted poetry to utilize the current global version of Python and build the venv within the project folder.
Personally I've had problems with poetry managing virtualenvs in the past so I just don't let it touch them any more. Maybe it's better now, but I don't see any reason to risk it, given how often Python seems to like to ruin my day.
I also don't like that by default it wants you to use `poetry run` to run things. Sure you can configure it not to, but it still annoys me
One issue I’ve been encountering is that Poetry isn’t aware of pyenv or `.python-version`. So what I do to initially build my venv is:
And every time that `.python-version` changes (which I at most do a few times per year per project,) I throw away the `.venv`, do `pyenv install -s` and start over.
Your `poetry run` point still stands, though.
pyenv exec pip install poetry && pyenv exec poetry install
And every time that `.python-version` changes (which I at most do a few times per year per project,) I throw away the `.venv`, do `pyenv install -s` and start over.
Your `poetry run` point still stands, though.
[deleted]
Now you have poetry installed for every version of python and need to keep track of updating all of those poetry installs.
Python really makes things difficult :D
Python really makes things difficult :D
> Now you […] need to keep track of updating all of those poetry installs.
You’re absolutely right.
On second thought, I guess I only need these Poetry installs just once anyway: for creating the venv and then never again. That’s not worth it.
So here’s a new method that does away with these extra Poetry installs:
Going to adopt this variant from now on, I guess. Thank you for your thoughts!
You’re absolutely right.
On second thought, I guess I only need these Poetry installs just once anyway: for creating the venv and then never again. That’s not worth it.
So here’s a new method that does away with these extra Poetry installs:
pyenv exec python -m venv .venv && poetry install
Going to adopt this variant from now on, I guess. Thank you for your thoughts!
OP - I'm curious why you've submitted this today, you seem experienced enough to guess this is not the first time you've used it. Has there been a major change or something happened in the community I need to know about? Or are you just spreading the love for a great piece of open source software?
I used Python a lot, but missed pyenv. Used it for some weeks now and was thinking this is something I would like to know about earlier (thats normally when I post it to HN).
There are always new developers coming onboard that might have missed this handy tool. It's worth firing up the conversation on occasion.
Also a bit confused as to why this is currently the #1 post.
Because it's not a solved problem, pyenv is niche and people discover it once again and think that they finally solved their pain.
They will try it, and most of them will pay the price for it.
It's a cycle.
They will try it, and most of them will pay the price for it.
It's a cycle.
I've used python lightly for a few decades now. I've never got far enough in to figure out how packages work. Once, at least 10 years ago, I briefly tried to use some published packages. I never figure out how to get it to work at all. Other than that, I've limited myself to what's available in the standard library. I've seen a lot of conflicting information about how python packages work, or should work. If there was a "one true way", I'd be interested. I don't want to have to compare a list of pros and cons. I want one officially blessed tool. I'll live with the constraints.
Packages is another story though. Pyenv is for Python versions. For me it’s either that or using Docker images. The later performed poorly with PyCharm when I tested a few years ago. I have seen an issue with Pyenv only once and not even on my machine.
I used pyenv in my previous company. At first it was fine, but after a while with multiple versions of Python installed, things stopped working. The right virtual env was not activated, etc. (It could have also been because our software updates like OS and security upgrades were pushed by desktop support). I removed all of it, and just resorted to installing multiple versions of Python the old way (downloading from python.org) and then use `python<version> -m venv /my/virtual/env` to manage my virtual environments. Things are more stable, and I don't feel like its magic. I am not going back to pyenv.
Works fine for me on macOS. I am using venv the same way you do except some global ones for certain tools.
I thought one of the advantages of pyenv was that you switched to also creating virtual envs using pyenv and managed virtual environments with it too. Maybe not.
Pyenv is for Python versions. There is a plugin for virtualenvs that can be useful for some specific cases but I rarely use it nowadays.
Also, when working on packages supporting multiple versions it was very helpful to use tox with pyenv.
I've given up on trying to manage runtime dependencies with language specific tooling. Instead, I've moved to mise which handles the majority of them (Python, Node, Ruby, Terraform, etc.) the same way. It will also activate Python virtual envs:
https://mise.jdx.dev/lang/python.html
It has other nice helpers for development environments (tasks, env variables, etc.).
https://mise.jdx.dev/lang/python.html
It has other nice helpers for development environments (tasks, env variables, etc.).
And for those of you that are already on asdf-vm, it is compatible with most of those plugins. I have not yet made the jump but I'm tempted.
I'll toss in my (macOS) workflow too:
- Download whatever Python binaries I need from python.org
- virtualenv --python=$PYTHON_VERSION ~/.virtualenvs/$PROJECT_NAME
- pyproject.toml
- pip-compile --generate-hashes
There has been tons of churn in the Python project management space, and I feel like I've been blissfully unaware of all of it with this workflow. Can't recommend enough.
Why regular Python binaries and not pyenv? I've had enough pyenv snafus and meltdowns that I started looking for alternatives, and it turned out I could just install whatever Python binary I wanted and specify it wherever I wanted. What could be easier?
python -m venv is fine, I've just gotten used to virtualenv.
pyproject.toml is the future and almost everything supports it pretty well; it also has fewer implicit weirdnesses than setup.py or setup.cfg.
pip-tools are really simple, and they let you have hashed dependencies which are super important IMO.
- Download whatever Python binaries I need from python.org
- virtualenv --python=$PYTHON_VERSION ~/.virtualenvs/$PROJECT_NAME
- pyproject.toml
- pip-compile --generate-hashes
There has been tons of churn in the Python project management space, and I feel like I've been blissfully unaware of all of it with this workflow. Can't recommend enough.
Why regular Python binaries and not pyenv? I've had enough pyenv snafus and meltdowns that I started looking for alternatives, and it turned out I could just install whatever Python binary I wanted and specify it wherever I wanted. What could be easier?
python -m venv is fine, I've just gotten used to virtualenv.
pyproject.toml is the future and almost everything supports it pretty well; it also has fewer implicit weirdnesses than setup.py or setup.cfg.
pip-tools are really simple, and they let you have hashed dependencies which are super important IMO.
Does it handle OS differences?
Yep
How exactly? Isn't a requirements.txt file, generated from pip-compile, incapable of specifying OS stuff?
It's also my understanding that pip-tools does not let you make layered requirements files with pyproject.toml files, only with .in files
It's also my understanding that pip-tools does not let you make layered requirements files with pyproject.toml files, only with .in files
You can specify different dependencies in pyproject.toml and then use --extra with pip-compile to pull them in. I do this with dev and test sections for example. Constraints seem like they work like you'd expect here. pip-compile also has the -c flag if you want to use that.
I guess I don't know what you mean by OS stuff, but maybe this works for that too?
I guess I don't know what you mean by OS stuff, but maybe this works for that too?
Hmm. So you generate a requirements.txt and a dev-requirements.txt. It sounds like they have overlapping dependencies? I can't get them to be "layered" where the dev-requirements.txt only has the dev dependencies.
Ah, yeah that's what I do--I don't think there's any way to do that no.
I have been using pyenv for years and it's the best tool for managing Python versions. You can install any version, switch between them, use different versions for different projects, have multiple versions installed at the same type without problems. It really solved all my Python version managament problems, I never looked for any other tool like this since I started using it.
I've been using python on and off since Django 0.96, which was released in 2007.
I don't recall ever needing anything other than virtualenvwrapper and pip—and even some of the annoyances these tools had early on have been solved by now...
https://virtualenvwrapper.readthedocs.io/en/latest/
If you really need different versions of python, you can just `mkvirtualenv -p python3 venvname`
I feel like every other tool out there has to explain what problem they solve that virtualenvwrapper doesn't
Don't install anything globally, creates lots of envs, and feel free to have different versions of python installed side-by-side with some "main" version preferably symlinked as `python` and `python3`
The end
I don't recall ever needing anything other than virtualenvwrapper and pip—and even some of the annoyances these tools had early on have been solved by now...
https://virtualenvwrapper.readthedocs.io/en/latest/
If you really need different versions of python, you can just `mkvirtualenv -p python3 venvname`
I feel like every other tool out there has to explain what problem they solve that virtualenvwrapper doesn't
Don't install anything globally, creates lots of envs, and feel free to have different versions of python installed side-by-side with some "main" version preferably symlinked as `python` and `python3`
The end
For managing versions of Python packages, I take a similar route and keep it simple:
For example, if you need the latest version of Python and it is not available through your OS:
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
For example, if you need the latest version of Python and it is not available through your OS:
pyenv install 3.12
pyenv global 3.12on Mac OS I just do
on Linux it should be just as easy to have multiple versions side-by-side
it's been a while since I've done something similar on Windows but it's also possible (although I think I would prefer to use WSL if I were on Windows these days)
brew install [email protected]
brew install [email protected]
on Linux it should be just as easy to have multiple versions side-by-side
it's been a while since I've done something similar on Windows but it's also possible (although I think I would prefer to use WSL if I were on Windows these days)
Some Linux distros don’t have the latest Python versions in their package repos, but there are definitely other ways to install versions side-by-side.
You don't even need virtualenv much less virtualenv wrapper. On Debian/Ubuntu distros (and most others, with small modifications) you just need python3-venv. Then create your venv with `python3 -m venv myenv` or `python3.11 -m venv my311venv` or whatever version of Python you need. Less is more!
You should look at the command `python -m venv`. Its built in, and is a breeze to create virtual environments. I guess it does not provide shortcuts for activation etc, but I am ok with that.
Glad I’m not the only one. As soon as I started using Pyenv for local development I bloated my machine with more versions of python than I could manage. Obviously user error, but it never solved a problem I couldn’t get around with virtualenv.
> feel free to have different versions of python installed side-by-side with some "main" version preferably symlinked as `python` and `python3`
This is what pyenv does.
This is what pyenv does.
by compiling from source? god no.
also it's terribly named because it conflates the idea of python environments with python versions
just install pre-compiled python from official sources side-by-side. no pyenv needed
also it's terribly named because it conflates the idea of python environments with python versions
just install pre-compiled python from official sources side-by-side. no pyenv needed
But that’s the point of the tool. Let it manage that for you.
right, but now you know even less about your setup when you some roadblock. and compiling python from source is very slow and error prone because it requires a bunch of flags and other libraries to be available. I have no problem compiling stuff from source generally, but python specifically is a major PITA
and I already have homebrew to install python (and anything else), which I trust to have more users and better formulas. other OSes have their own tools
and I already have homebrew to install python (and anything else), which I trust to have more users and better formulas. other OSes have their own tools
> right, but now you know even less about your setup when you some roadblock
This is the same with a binary though. And with homebrew, you can't follow patches or flags used or if they change.
- https://github.com/Homebrew/homebrew-core/blob/c964ad7fa53ad... - https://github.com/Homebrew/homebrew-core/blob/c964ad7fa53ad... - https://github.com/Homebrew/homebrew-core/blob/c964ad7fa53ad...
Not arguing for good or bad, just saying that the problem isn't limited to this.
This is the same with a binary though. And with homebrew, you can't follow patches or flags used or if they change.
- https://github.com/Homebrew/homebrew-core/blob/c964ad7fa53ad... - https://github.com/Homebrew/homebrew-core/blob/c964ad7fa53ad... - https://github.com/Homebrew/homebrew-core/blob/c964ad7fa53ad...
Not arguing for good or bad, just saying that the problem isn't limited to this.
More than Python versions, I use pyenv to keep from polluting the system’s site-packages.
It also ensures that my requirements.txt is sufficient for the code to run.
It also ensures that my requirements.txt is sufficient for the code to run.
I used to use this a lot when working on a Windows machine. Worked pretty well. But nowadays Nix solves the same problem in a fully general way, for all software rather than a single language. You can have whatever versions of whatever you want, side by side, without interfering with each other.
What is your solution here? Are you using a project-specific flake? Do you still install packages with pip?
I was more talking about version management of the Python interpreter itself. So far all the packages I need are already in nixpkgs so I haven't needed to stray off that path yet.
This is the way. I moved everyone I work with at over to the pyenv, virtualenv, poetry stack. Once you get set up it’s pretty smooth sailing. It is a shame that so many tools are needed to work with python, but I was happy to finally find something that works.
Curiously, post-checkout, how do you go about creating your venv and installing your project dependencies?
The shortest ritual I could figure out so far is:
The shortest ritual I could figure out so far is:
pyenv install -s && pyenv exec pip install poetry && pyenv exec poetry install
Python (and Ruby) handling of versions and dependencies is truly awful... :-(
I've only used Python occasionally, and more than half of that time I ran into issues with the installed Python version or dependencies. Not only version 2 vs 3, but for example a certain project required exactly 3.11, not 3.10 or 3.12. Last time, I tried to install Caddy and broke all previously installed packages/commands - and after searching for a solution and trying different things, the best I could come up with was to switch versions (manually) before and after running a command to restore the environment.
Now I understand I should have been using Pyenv the whole time. But it sure felt unfriendly for a newcomer to the language/ecosystem. Granted my learning approach is to jump in without much preparation, I wish the language designers or the ecosystem provided a better experience when things don't work right.
Now I understand I should have been using Pyenv the whole time. But it sure felt unfriendly for a newcomer to the language/ecosystem. Granted my learning approach is to jump in without much preparation, I wish the language designers or the ecosystem provided a better experience when things don't work right.
How is it awful on the Ruby side? In my experience it is a solved problem since bundler was invented 15 years ago. It's been very free from pain during the last 10 years or so.
AFAIR it was mostly because Ruby still wanted to use not-100%-fixed versions so there was still an issue with something updating slightly breaking the build. Just a cursory experience when trying to deal with Ruby. Though it _felt_ somewhat more stable than Python.
In my experience, python has been far, far worse than ruby. Now, to be fair, I’m much more comfortable in the ruby ecosystem, but still, everything feels much more brittle in python package management.
It's inevitable for non-dead interpreted languages.
What's wrong with rvm?
can I interest you in R?
I thought the recommended approach these days was to use the venv package built in to python3 with `python3 -mvenv ...`
It is, but the problem with `venv` is that it can only use the same version of Python that you're using. E.g. if your Python installation is 3.8, your `venv` will also use 3.8.
Pyenv allows you to use whatever version of Python you want.
Pyenv allows you to use whatever version of Python you want.
Most systems allow you to keep multiple python installations, no? I think the best way is to specify the python when you do this: python3.11 -m venv env
On most Linux distros you probably don't want to change your system python to whatever, because there are a lot of system programs that may not be compatible with a version other than what is on the system by default.
Then once you activate it you can make sure your python
On most Linux distros you probably don't want to change your system python to whatever, because there are a lot of system programs that may not be compatible with a version other than what is on the system by default.
Then once you activate it you can make sure your python
it's not a bug, it's a feature.
just install the different python versions by downloading the binaries or through your package manager and run python3.<whatever_version_you_want> -m venv and the virtual environment will automatically, forever, use that python version when you activate it.
just install the different python versions by downloading the binaries or through your package manager and run python3.<whatever_version_you_want> -m venv and the virtual environment will automatically, forever, use that python version when you activate it.
You're right, but venv requires an existing Python install to create a venv. Pyenv is for installing multiple different versions of Python to create those virtual environments.
This is a truly, truly terrible idea. It adds several failure modes, some subtle so you can go a long way in a state of error, just so beginners can type `python` instead of e.g. `python3.10`.
Many developers, not just me, have a similar setup: we use virtual environments everywhere, and if you aren't in one, `python` doesn't even resolve to a symbol.
If I want to write a quick script with no dependencies, I directly call `python3.xx` on it. Otherwise, I create a virtualenv.
Yes, it's a bit harder for beginners, but from a huge amount of experience helping people who are starting up in programming, people have little issue in following a few more instructions. What demolishes beginners is getting into a bad state where nothing works and you don't know why.
Many developers, not just me, have a similar setup: we use virtual environments everywhere, and if you aren't in one, `python` doesn't even resolve to a symbol.
If I want to write a quick script with no dependencies, I directly call `python3.xx` on it. Otherwise, I create a virtualenv.
Yes, it's a bit harder for beginners, but from a huge amount of experience helping people who are starting up in programming, people have little issue in following a few more instructions. What demolishes beginners is getting into a bad state where nothing works and you don't know why.
This is not a terrible idea. It is a tool that makes it possible for people to either retrofit a newer version of Python onto an ageing OS or to develop for a specific older version of Python on a newer OS. If you don't need the features the tool uses, you shouldn't use it--I don't see why you're comparing it to virtualenvs, it provides completely different runtimes with a different engine and a standard library, and the ability to switch between those by just cd'ing to a folder with a .python-version file in them.
Isn't this a completely different concept?
Virtual environments are for your installed dependencies, whilst pyenv is for installing python.
I have a client that uses Python X and another that strictly uses Python X+1.
The virtual environments are so that I can have the project dependencies installed and the pyenv lets different companies have different cadence for their Python upgrades.
I could be completely mistaken and mixing up my Python support utils as I've not had a client request Python for a couple of years.
Virtual environments are for your installed dependencies, whilst pyenv is for installing python.
I have a client that uses Python X and another that strictly uses Python X+1.
The virtual environments are so that I can have the project dependencies installed and the pyenv lets different companies have different cadence for their Python upgrades.
I could be completely mistaken and mixing up my Python support utils as I've not had a client request Python for a couple of years.
[deleted]
But how do I get a .venv with a new/different version of Python in your setup? And how do I add some kind of Python version requirement in my project?
though, with openinterpreter, you can just ask it to fix your python and it'll help you out of a weird situation that you as an newbie don't understand. Still, node's node_modules implementation isn't a bad one, at the expense of disk space, you don't run into the same problems as you do python.
Pls dont.
Just install several pythons with different binary names.
ex. python3.8, python3.9, python3.11
No need to complicate things.
If using Poetry, you can just do `poetry env use python3.8`.
Just install several pythons with different binary names.
ex. python3.8, python3.9, python3.11
No need to complicate things.
If using Poetry, you can just do `poetry env use python3.8`.
What do you do when (not if) your system package manager eventually drops that Python version from their core repositories?
The good thing about pyenv is that it makes you rely less on package maintainers. You can come back to your Python project 10 years later and have a real chance that it still works, even if you have changed workstations in between.
The good thing about pyenv is that it makes you rely less on package maintainers. You can come back to your Python project 10 years later and have a real chance that it still works, even if you have changed workstations in between.
Here's all the project boilerplate you need with pyenv when working on a new project.
> $ pyenv install 3.8
> $ pyenv local 3.8
> $ python3 -m venv .venv
> $ source .venv/bin/activate
and you're done. Not really sure what you might consider difficult about pyenv, but it's just a tool to instigate your python venvs which is a built tool for most modern versions of python.
> $ pyenv install 3.8
> $ pyenv local 3.8
> $ python3 -m venv .venv
> $ source .venv/bin/activate
and you're done. Not really sure what you might consider difficult about pyenv, but it's just a tool to instigate your python venvs which is a built tool for most modern versions of python.
> Not really sure what you might consider difficult about pyenv
- compiling from source
- need to install build-essentials on docker images which takes up lot of space and takes a long time
- compiling from source
- need to install build-essentials on docker images which takes up lot of space and takes a long time
I use this every day everywhere. Together with pyenv-virtualenv and poetry it dramatically improved my python experience…
We use Pyenv successfully for developing the Flower open-source project. We use a few simple Bash scripts to manage virtual environments with different Python versions via pyenv and the pyenv-virtualenv plugin.
The main scripts are `venv-create.sh`, `venv-delete.sh` and `bootstrap.sh`. `venv-reset.sh` pulls these three scripts together to make reinstalling your venv a single command.
Here's the link if anyone is interested: https://github.com/adap/flower/tree/main/dev
The main scripts are `venv-create.sh`, `venv-delete.sh` and `bootstrap.sh`. `venv-reset.sh` pulls these three scripts together to make reinstalling your venv a single command.
Here's the link if anyone is interested: https://github.com/adap/flower/tree/main/dev
I'm using pyenv to manage different projects on my machine. I use --system-site-packages to share libraries like torch, and some others -- especially to ensure those libraries are built with the same cuda version, avoid some issues when upgrading drivres, and to avoid having to keep multiple copies of several gigabyte libraries around.
I'm not using things like anaconda for a similar reason I'm not using docker. -- I don't want a separate OS install for each service I'm running on my server. It works perfectly fine.
I'm not using things like anaconda for a similar reason I'm not using docker. -- I don't want a separate OS install for each service I'm running on my server. It works perfectly fine.
Is anyone is the AI/ML area finding success with anything other than conda, where installation of CUDA/CUDnn is required? Although I often have to pip install a lot of packages, I find conda's nvidia/pytorch/conda-forge channels are still by far the easiest way to get a deep learning stack up and running, and so I just stick with conda environments. I've tried poetry in the past but getting the NVidia deep learning stack up and running was really tough.
For anything related to CUDA/CuDNN, use one of NVIDIA base Docker images. Then whether you use Conda / Pip / Poetry / Pipenv does not matter much. Not at all a Conda fan myself and avoid it like the plague
What's surprising to me is that this isn't better known. The only reliable solution I've found is to go with the pytorch or deepstream images from NGC. Conda is probably a good idea for noobs who need Cuda installed for them on windows, but otherwise I find it an endless source of finicky issues, especially for unsavvy ML scientists who are looking for a silver bullet for package management.
This link shows which package versions come in which Docker tag and is invaluable: https://docs.nvidia.com/deeplearning/frameworks/support-matr...
This link shows which package versions come in which Docker tag and is invaluable: https://docs.nvidia.com/deeplearning/frameworks/support-matr...
10 years ago, « Data Science » work past the experimental stage was performed by SWE with a knack for applied maths. So investing in tooling to do things properly was a given.
Nowadays, most DS people only want to do ML at the experimental stage only and get lost when things get on the engineering side of things. But for their defense, nowadays the bare minimum skills require to do programming, containerization, CI/CD, etc. More experienced and swiss army knife SWE/MLE have to educate the willing.
It was already the same 10 years ago with MATLAB dudes not wanting to get dirty with C/C++/ASM SIMD. The history repeats itself, only at a faster pace
Nowadays, most DS people only want to do ML at the experimental stage only and get lost when things get on the engineering side of things. But for their defense, nowadays the bare minimum skills require to do programming, containerization, CI/CD, etc. More experienced and swiss army knife SWE/MLE have to educate the willing.
It was already the same 10 years ago with MATLAB dudes not wanting to get dirty with C/C++/ASM SIMD. The history repeats itself, only at a faster pace
Yes. I simply do
There's a very good chance I've installed cuda on my system before this though. And usually cudnn and some other packages because this is part of my standard install. And then I also never run into the issue where a package is looking for nvcc.
python -m pip install torch torchvision
There's a very good chance I've installed cuda on my system before this though. And usually cudnn and some other packages because this is part of my standard install. And then I also never run into the issue where a package is looking for nvcc.
I love poetry but have found it pretty hard once you move off of anything that doesn’t manage to get wheels on pypi.
We make extensive use of conda/mamba to solve this, and are pretty happy with it, especially with conda-forge.
We make extensive use of conda/mamba to solve this, and are pretty happy with it, especially with conda-forge.
I have successfully transitioned an ML/AI team of seasoned researchers away from conda and to poetry. Some also use pyenv, I suspect a lot don't bother but may get bitten eventually.
It's definitely a learning curve, but it turns out every conda user has been bit by the irreproducible tendencies of conda quite often. Nobody uses the conda env file, they just start an env and pip install things into it. They don't realize the base env has stuff, too, and conda envs are hierarchical rather than isolated. I know it's possible to use conda in an isolated and reproducible way, but have yet to meet someone that does so.
So it hasn't been hard to pitch poetry to these folks, and while many complain about the learning curve they appreciate the outcomes.
We're a pytorch shop, and torch mostly just works with pip or poetry these days, as long as you skip the versions the torch maintainers mispackaged. We rarely need anything higher-level that only conda could install.
We really like having more than two dependency groups as this allows us to keep research and production in the same repository. main, dev, research. Then researchers contribute to the core library of a project and keep research and production using the same code for running and evaluating models.
It's definitely a learning curve, but it turns out every conda user has been bit by the irreproducible tendencies of conda quite often. Nobody uses the conda env file, they just start an env and pip install things into it. They don't realize the base env has stuff, too, and conda envs are hierarchical rather than isolated. I know it's possible to use conda in an isolated and reproducible way, but have yet to meet someone that does so.
So it hasn't been hard to pitch poetry to these folks, and while many complain about the learning curve they appreciate the outcomes.
We're a pytorch shop, and torch mostly just works with pip or poetry these days, as long as you skip the versions the torch maintainers mispackaged. We rarely need anything higher-level that only conda could install.
We really like having more than two dependency groups as this allows us to keep research and production in the same repository. main, dev, research. Then researchers contribute to the core library of a project and keep research and production using the same code for running and evaluating models.
I use pipenv and I've found it to be much more usable than conda. For my use cases, it's generally faster and I've run into fewer dependency issues.
uv has been really awesome as a replacement for pip:
https://github.com/astral-sh/uv
So fast it finally made virtual environments usable for me. But it's not (yet) a full replacement for conda, e.g. it won't install things outside of Python packages
So fast it finally made virtual environments usable for me. But it's not (yet) a full replacement for conda, e.g. it won't install things outside of Python packages
How about prefix then?
https://prefix.dev/blog/uv_in_pixi
Pyenv just worked for me. I am actually using Fedora Silverblue and have GCC and the CUDA SDK available only inside a toolbox container. Therefore, I have to enter that toolbox to install things like FlashAttention.
Have you tried https://pixi.sh/ ? It brings Cargo/NPM/Poetry like commands and lock files to the Conda ecosystem, and now can manage and lock PyPI dependencies alongside by using uv under the hood.
I haven't been using anything CUDA, but the scientific geospatial stack is often a similar mess to install, and it's been handling it really well.
I haven't been using anything CUDA, but the scientific geospatial stack is often a similar mess to install, and it's been handling it really well.
I use poetry and direnv. Coming from node/npm, it feels natural for me to just do this. I have really no troubles of installing Pytorch with poetry
How are you installing Pytorch with CUDA with Poetry? I stopped using Poetry because it wouldn't automatically get the CUDA version; instead, it would install the CPU version. I migrated to PDM, which does the right thing.
Before CUDA 12.0 you have to specify a field in pyproject.toml like this
[tool.poetry.dependencies]
python = ">=3.10,<3.12.0"
torch = {version = "^2.0.1+cu118", source = "torch118"}
torchvision = {version = "^0.15.2+cu118", source = "torch118"}
[[tool.poetry.source]]
name = "torch118"
url = "https://download.pytorch.org/whl/cu118"
priority = "explicit"
poetry add torch torchvisionI stand corrected. I was familiar with the first option, which coupled the dependencies with the platform, whereas I wanted a CUDA version on Linux and a Metal version on macOS.
However, this works perfectly with Poetry 1.8 and Pytorch 2.2. I suppose the only problem is what PDM also does, where the lock file is platform-dependent. I'm not sure whether Poetry allows you to select a specific lock file, however.
However, this works perfectly with Poetry 1.8 and Pytorch 2.2. I suppose the only problem is what PDM also does, where the lock file is platform-dependent. I'm not sure whether Poetry allows you to select a specific lock file, however.
was this before torch 2.0? With the very notable exceptions of a few mispackaged versions, torch now includes all the relevant Nvidia libs, and I haven't seen it grab the CPU version on a GPU box yet, though I'm not sure what it looks for.
A notable open issue in poetry is we can't currently specify one dependency on torch, and have it grab CPU version on some systems and GPU on others. Does PDM solve that?
A notable open issue in poetry is we can't currently specify one dependency on torch, and have it grab CPU version on some systems and GPU on others. Does PDM solve that?
I don't think PDM solves that directly. What I do is have different lock files for different platforms (e.g. Linux/CUDA and macOS/Metal), but pyproject.toml lists only "torch".
Why not just use a tool like asdf (https://asdf-vm.com/) or mise (https://mise.jdx.dev/)?
These tools have the advantage of not being multi-taskers and can manage version for all your tools. You wouldn’t need pyenv and npm and rvm and…
We’ve even started committing the .mise.toml files for projects to our repos. That way, since we work on multiple projects that may need multiple versions of the same tool, it’s handled and documented.
These tools have the advantage of not being multi-taskers and can manage version for all your tools. You wouldn’t need pyenv and npm and rvm and…
We’ve even started committing the .mise.toml files for projects to our repos. That way, since we work on multiple projects that may need multiple versions of the same tool, it’s handled and documented.
I really love miniconda for managing python versions and deps myself.
Conda/Miniconda randomly breaks every few months or so when updating and I can't fix it. Idk why I stick with it, its been like this for years.
Odd. I always do `conda activate base && conda update conda` on Mac and never have any issues.
I recently used Pyenv to install an older version of Python on my system. The reason for this, is I wanted to create a Docker image using 3.8 slim-buster and I wanted to be explicit about the Python libraries I needed. By using Pyenv to install the same version of python as is used by slim-buster I was able to pip freeze my virtual env, and use the same version of libraries for the Docker image no problem. Oh the things I have to do in Python to avoid dependency hell
Was a pyenv user for the longest time, but I switched to Docker containers instead. I put together bash functions that take in a positional argument for the image and port (or randomizes it if not specified), and it automatically mounts the current working directory to the volume. Now I can just start a container without having to think about any of this stuff. And a config file per project if I don't want to pass in any variables at all.
I've been using Rye[0] lately, which has been pretty good. It's really just a wrapper around a bunch of underlying tools - it's nice to not have to worry about those and let Rye do it's thing.
All that being said, the creator of Rye is 100% cognizant of that XKCD comic, this [1] is a nice read.
I'm not super well versed in Python tooling at all. I've had to work a lot in Python in the past 6+ months, and I become super confused when I tried making a Python project in my spare time.
I settled on Rye because it just seemed to be the easiest to use.
[0]: https://rye-up.com/ [1]: https://github.com/astral-sh/rye/discussions/6
All that being said, the creator of Rye is 100% cognizant of that XKCD comic, this [1] is a nice read.
I'm not super well versed in Python tooling at all. I've had to work a lot in Python in the past 6+ months, and I become super confused when I tried making a Python project in my spare time.
I settled on Rye because it just seemed to be the easiest to use.
[0]: https://rye-up.com/ [1]: https://github.com/astral-sh/rye/discussions/6
In windows I just use subst to switch python versions. Python always sits on drive P:. Never understood what all the fuzz about this was about. Sure, I have the same package on my disk multiple times, but if I cared about that there'd be easy ways to get around that problem just using OS functionality.
From my perspective, it doesn't come easier than running a batch file to switch between python versions.
From my perspective, it doesn't come easier than running a batch file to switch between python versions.
pip+venv mostly work fine for me, but there are some libraries like say numpy/scipy that are heavy so I want to share them across projects instead of installing multiple copies. I use pyenv to create a python version separate from the system version and install global dependencies into that. It's been useful for that and is just a nice way to manage multiple versions of python. So pyenv is a great addition to the traditional pip+venv setup.
unrelated rant -
Why is the "requirements.txt" file a stupid flat listing of all transitive dependencies with pinned versions? It makes it harder to change library versions even if there are no true conflicts. I've resorted to making an "actual_requirements.txt" file manually listing only direct dependencies and whatever version constraints make sense. I wish pip would fix this.
unrelated rant -
Why is the "requirements.txt" file a stupid flat listing of all transitive dependencies with pinned versions? It makes it harder to change library versions even if there are no true conflicts. I've resorted to making an "actual_requirements.txt" file manually listing only direct dependencies and whatever version constraints make sense. I wish pip would fix this.
> Why is the "requirements.txt" file a stupid flat listing of all transitive dependencies with pinned versions? It makes it harder to change library versions even if there are no true conflicts.
My friend, here is what you seek: https://github.com/jazzband/pip-tools
requirements.txt is flat because it's really the output of `pip freeze`. It's supposed to completely and exactly rebuild the environment. Unfortunately it's far too flexible and people abuse it by putting in only direct dependencies etc.
If you're writing packages, you don't need a requirements.txt at all, by the way. Package dependencies (only direct dependencies) live in pyproject.toml with the rest of the package config. requirements.txt (and pip tools) are only for when you want to freeze the whole environment, like for a server deployment.
My friend, here is what you seek: https://github.com/jazzband/pip-tools
requirements.txt is flat because it's really the output of `pip freeze`. It's supposed to completely and exactly rebuild the environment. Unfortunately it's far too flexible and people abuse it by putting in only direct dependencies etc.
If you're writing packages, you don't need a requirements.txt at all, by the way. Package dependencies (only direct dependencies) live in pyproject.toml with the rest of the package config. requirements.txt (and pip tools) are only for when you want to freeze the whole environment, like for a server deployment.
> I've resorted to making an "actual_requirements.txt" file manually listing only direct dependencies and whatever version constraints make sense.
Pip-tools has an elegant solution for this. You write an requirements.in where you list packages and optionally versions, then pip-compile turns that into a requirements.txt with every package including dependencies specified by versions. You can use it to update specific packages as well.
Pip-tools has an elegant solution for this. You write an requirements.in where you list packages and optionally versions, then pip-compile turns that into a requirements.txt with every package including dependencies specified by versions. You can use it to update specific packages as well.
I wonder if it ever occurs to people the whole concept of versioned environments for compatibility is fundamentally a terrible design kludge.
Someone needs to standardize a single Python environment at least annually. As these tools are just turning Python into a perpetual Alpha build.
Best of luck, =)
Someone needs to standardize a single Python environment at least annually. As these tools are just turning Python into a perpetual Alpha build.
Best of luck, =)
I've been using pyenv for several years now, but for some reason, the basic commands and overall integration don't feel as smooth as Node's nvm package. I wonder if that's because Python setup is technically harder than Node.
Several years in ops in a large company where I had to hand-hold couple hundreds of programmers a lot of whom needed to install Python for development on their computers...
One of the troubleshooting steps when "something mysterious" happens on developer's computer (eg. package installation fails for inexplicable reasons, Python "standard" library components missing or present when shouldn't be, incorrect component version etc.) was to remove pyenv.
This step was often met with resentment and arguments... but, in most cases I was able to win :)
----
Now, here's a larger point: some programs solve the problem by actually solving the problem, while other programs solve the problem by adding more code around the problem, which, usually, creates new problems while only partially solving the original problem.
An example of the former: fsck -- you run it, it looks at your filesystem, tries to fix it, if it's broken and then gets out of the way entirely. An example of the later: Kubernetes -- you start by having a problem of resource allocation / management and you end up with a problem of resource allocation / management compounded by problems with component version management, configuration management etc.
pyenv falls into the second category of programs. The problem it's trying to solve is: install and use multiple versions of Python. There's really no need for an extra helper program to solve this problem. Multiple versions of CPython can be installed and used together without the use of any extra tools. I.e. the solution to this "problem" is simply to learn how to do that.
Those who advocate for the use of pyenv and the likes usually make an argument for "simplicity". I.e. in their mind, not needing to know how to install multiple versions of Python is a bonus. Something that, potentially, saves them several hours of reading the documentation and perhaps saving them a tiny bit of typing when setting up Python initially.
I contend that this calculation is off because it doesn't account for the problems down the lane. In other words: pyenv helps until it doesn't, and then it becomes a liability. Debugging is always more difficult if you have more wrappers between you and your problem. So, while individual users will not face a lot of problems with "wrapper solutions", those who service such users will face such problems a lot more frequently. That's why, as an ops person, I really dislike "wrapper solutions" -- for me, they complicate the task, never help.
One of the troubleshooting steps when "something mysterious" happens on developer's computer (eg. package installation fails for inexplicable reasons, Python "standard" library components missing or present when shouldn't be, incorrect component version etc.) was to remove pyenv.
This step was often met with resentment and arguments... but, in most cases I was able to win :)
----
Now, here's a larger point: some programs solve the problem by actually solving the problem, while other programs solve the problem by adding more code around the problem, which, usually, creates new problems while only partially solving the original problem.
An example of the former: fsck -- you run it, it looks at your filesystem, tries to fix it, if it's broken and then gets out of the way entirely. An example of the later: Kubernetes -- you start by having a problem of resource allocation / management and you end up with a problem of resource allocation / management compounded by problems with component version management, configuration management etc.
pyenv falls into the second category of programs. The problem it's trying to solve is: install and use multiple versions of Python. There's really no need for an extra helper program to solve this problem. Multiple versions of CPython can be installed and used together without the use of any extra tools. I.e. the solution to this "problem" is simply to learn how to do that.
Those who advocate for the use of pyenv and the likes usually make an argument for "simplicity". I.e. in their mind, not needing to know how to install multiple versions of Python is a bonus. Something that, potentially, saves them several hours of reading the documentation and perhaps saving them a tiny bit of typing when setting up Python initially.
I contend that this calculation is off because it doesn't account for the problems down the lane. In other words: pyenv helps until it doesn't, and then it becomes a liability. Debugging is always more difficult if you have more wrappers between you and your problem. So, while individual users will not face a lot of problems with "wrapper solutions", those who service such users will face such problems a lot more frequently. That's why, as an ops person, I really dislike "wrapper solutions" -- for me, they complicate the task, never help.
> Multiple versions of CPython can be installed and used together without the use of any extra tools. I.e. the solution to this "problem" is simply to learn how to do that.
How do you do that? For example, if you're on Ubuntu LTS but want to use the latest Python version, how do you do that? The system package manager won't have it. Do you rely on a third-party PPA? What about other distros, where you don't have that?
How do you do that? For example, if you're on Ubuntu LTS but want to use the latest Python version, how do you do that? The system package manager won't have it. Do you rely on a third-party PPA? What about other distros, where you don't have that?
I personally? -- Check out CPython Github repository and build w/e version I need. CPython's Makefile has altinstall command that installs as pythonX.Y rather than just python. And if I need to have multiple Pythons of the same version, but, say, different in whether they have debug info or something like that, I modify the installation prefix.
But, you can also download CPython source tarballs and skip the Git part. I haven't used CPython installers in a very long time... but, I'd imagine that the MSI would have a way to specify at least the location where Python is to be installed...
But, you can also download CPython source tarballs and skip the Git part. I haven't used CPython installers in a very long time... but, I'd imagine that the MSI would have a way to specify at least the location where Python is to be installed...
A similar tool for Go: https://go101.org/apps-and-libs/gotv.html
This is my favorite toolset -- pyenv and pyenv-virtualenv! So good. Such a great system imo. This is my preferred path through the forest of options here. :)
Not really a complaint - but I am surprised to see this on the front page of HN. This is an enormously popular project that's been around for over a decade that basically anyone who writes Python regularly is aware of.
Only marginally less surprising than seeing a link directly to python.org.
Only marginally less surprising than seeing a link directly to python.org.
Casual Python user. I found all these things very confusing.
Then I found Pipenv. Sticking with it. (Emacs has support for this).
Then I found Pipenv. Sticking with it. (Emacs has support for this).
I think the only problem of pyenv is that you can't upgrade an old virtualenv to a newer Python version.
Isn't that more of a limitation with virtualenv? I've been using virtualfish as my venv manager in the fish shell and it has a `vf upgrade` command for upgrading virtualenvs to newer versions, either by updating in-place or rebuilding the entire virtualenv. So far has worked nicely with my pyenv Python installs.
Yeah. I had a shell snippet someplace that was essentially pip freeze | remove virtualenv | make virtualenv | pip install. It's not that hard.
That has nothing to do with pyenv.
Also, this really isn't a problem -- just blast out of the old venv, make a new one with a new version, and you're good.
Also, this really isn't a problem -- just blast out of the old venv, make a new one with a new version, and you're good.
Migrated some small scripts from Python to Go because of the virtual env madness.
Seeing pyenv inside of docker containers is always a treat
Just use the builtin "python -m venv" and life is good, why the others?
I tried almost all of them, with various issues, now staying with the default venv, it's solid and get the job done.
The premise of controlling python version is that - you're sending the project to your friend / colleague and you want both of you to use the same python version so that you get the same behaviour.
And then the same for your friend the buildserver.
It's all about reproducibility.
And then the same for your friend the buildserver.
It's all about reproducibility.
which venv does really well, what am I missing.
in the "worst" case you can always do a docker
in the "worst" case you can always do a docker
venv only works if you already have the required version of python installed and have it active as the python for your shell (or use a launcher to which you specify the python to use in the command line), since venv just creates a virtual environment based on the version of python it is run with.
So you actually need something that can read a specification of the required python version and use the correct one from the available options (maybe even reaching out and getting it if it isn't already locally available, though I don5 remeber if any of the existing python build tools will do this; docker obviously will, but its not really a python build tool), across different OS flavors (maybe not that last bit, depending on the use case, especially for internal development), and with minimal overhead for the build tool itself.
So you actually need something that can read a specification of the required python version and use the correct one from the available options (maybe even reaching out and getting it if it isn't already locally available, though I don5 remeber if any of the existing python build tools will do this; docker obviously will, but its not really a python build tool), across different OS flavors (maybe not that last bit, depending on the use case, especially for internal development), and with minimal overhead for the build tool itself.
Yeah, my thoughts exactly. I’ve never found myself in a situation where venv did not do exactly what I needed it to do.
The problem there is that by default a virtual environment version of Python is just a symbolic link to the actual binaries installed elsewhere on the machine. If you do an OS upgrade or do anything that increments the system Python version, it could cause problems in the environment.
pyenv manages python versions, not virtual environments. venv won't help you grab python versions you don't have, and will happily install your requirements into a venv with the wrong python version.
I long for the day when python is mature enough for an implementation to arise where one binary can run as all previous versions of the language and stdlib, so you can do
I mean, imagine if C compilers required that you have a separate compiler binary and copy of the standard headers (or even worse, all of stdlib!) for each version of the C standard?!? Instead of just putting new language features on version flags, and make stdlib functions available/marked as deprecated based on version checks?
allpy --std=py3.4
gcc -std=c90
I mean, imagine if C compilers required that you have a separate compiler binary and copy of the standard headers (or even worse, all of stdlib!) for each version of the C standard?!? Instead of just putting new language features on version flags, and make stdlib functions available/marked as deprecated based on version checks?
I wonder if "maturity" is the problem, or just different design goals. You speak like this is an inherent feature that comes with some magical level of "maturity", but if the feature isn't a legitimate design goal of the product, it isn't "immature" not to have it.
to switch, I use on the command like "python", or "python3"... seems a lot easier to me and you dont even need an extra package.
Obligatory relevant xkcd: https://xkcd.com/1987/
Need a new one with pyenv in it? (or to replace it)
This is the sort of tool that experienced devs all use, but nobody teaches or writes it into a tutorial on the larger language as a whole, because it feels like a side tool.
For now I'm not seeing a lot of reasons to use PyEnv over the `venv` module that ships with Python 3.3+ [1]. I'm sure there are some thing PyEnv does that `venv` doesn't, but the fact that `venv` ships with Python greatly simplifies things.
Basically my workflow is:
1. I'll create a Python project with, `mkdir dirname`, `cd dirname`, `git init`, `python3 -m venv .env`. This creates a hidden folder named `.env/` which contains the virtual environment. I'll then usually add `.env/` to my `.gitignore`, and also run `pip install --upgrade pip` and `pip install wheel`.
2. If I'm using an existing project, I'll do `git clone `projname`, `cd dirname`, `python3 -m venv .env`, `pip install -m requirements.txt` (requirements.txt is the idiomatic name for the dependencies list in Python projects).
3. I have the following lines in my `~/.bashrc` (hidden file that contains Bash settings):
4. When I install a dependency, I use `pip freeze` to get the dependency string, and I append that line to the file `requirements.txt`. For example, if I do `pip install django`, I get `Django==5.0.1` in my output from `pip freeze`, so I'll append that line to my `requirements.txt`.
5. To upgrade dependencies, I edit the version number in the `requirements.txt` file, and then run `pip install -m requirements.txt`. This makes sure that the requirements file stays up-to-date with my locally installed dependencies.
6. To get valid version numbers for a file, you can do `pip install <package>==`, which is basically asking pip to install an invalid version number. This causes it to list valid version numbers. For example, you can do `pip install django==` to view all available Django versions.
[1] https://docs.python.org/3/library/venv.html
For now I'm not seeing a lot of reasons to use PyEnv over the `venv` module that ships with Python 3.3+ [1]. I'm sure there are some thing PyEnv does that `venv` doesn't, but the fact that `venv` ships with Python greatly simplifies things.
Basically my workflow is:
1. I'll create a Python project with, `mkdir dirname`, `cd dirname`, `git init`, `python3 -m venv .env`. This creates a hidden folder named `.env/` which contains the virtual environment. I'll then usually add `.env/` to my `.gitignore`, and also run `pip install --upgrade pip` and `pip install wheel`.
2. If I'm using an existing project, I'll do `git clone `projname`, `cd dirname`, `python3 -m venv .env`, `pip install -m requirements.txt` (requirements.txt is the idiomatic name for the dependencies list in Python projects).
3. I have the following lines in my `~/.bashrc` (hidden file that contains Bash settings):
# Gets a directory named .env or .venv if it exists in the currend directory or any of its parents
get_env() {
if [ -d "$1/.env" ] ; then
echo "$1/.env"
else
if [ -d "$1/.venv" ] ; then
echo "$1/.venv"
else
if [ -d "$1/.." ] ; then
get_env "$1/.."
fi
fi
fi
}
get_absolute_path() {
python3 -c "import os; print(os.path.realpath('$1'))"
}
on_prompt() {
# Load a virtualenv environment if it exists in a file named .env
env_folder=$(get_env $(pwd))
if [ -d "$env_folder" ] ; then
if [[ $VIRTUAL_ENV != $(get_absolute_path $env_folder) ]] ; then
echo "Activating env '$env_folder'"
source "$env_folder/bin/activate"
fi
else
if [ -d "$VIRTUAL_ENV" ] ; then
deactivate
fi
fi
}
# Call on_prompt() every time the command prompt executes
PROMPT_COMMAND=on_prompt
4. When I install a dependency, I use `pip freeze` to get the dependency string, and I append that line to the file `requirements.txt`. For example, if I do `pip install django`, I get `Django==5.0.1` in my output from `pip freeze`, so I'll append that line to my `requirements.txt`.
5. To upgrade dependencies, I edit the version number in the `requirements.txt` file, and then run `pip install -m requirements.txt`. This makes sure that the requirements file stays up-to-date with my locally installed dependencies.
6. To get valid version numbers for a file, you can do `pip install <package>==`, which is basically asking pip to install an invalid version number. This causes it to list valid version numbers. For example, you can do `pip install django==` to view all available Django versions.
[1] https://docs.python.org/3/library/venv.html
[deleted]
Python package management is so broken. Think of the gymnastics that every python package has to go through in order for users to install it properly. Absolutely insane that this is still happening in such a widely used programming language.
Homebrew really bugs me. The devs want you to use the most current version of Python. Except the requirement is set at the package level and there's no enforcement. So you just keep getting python versions installed.
I don't get why you just don't set a minimum version like every other manager. Or if you want to do this crazy thing, don't rely on package maintainers to enforce it. I just ends up in major bloat
I don't get why you just don't set a minimum version like every other manager. Or if you want to do this crazy thing, don't rely on package maintainers to enforce it. I just ends up in major bloat
This is misleading. The older versions get cleaned out as brew does the upgrades (if not, then check your brew settings). My take on brew's Python is that it is there to make sure there is a uniform runtime for everything with a Python dependency, but I am not required to use it.
As. Homebrew maintainer: confirmed. You can also adjust your settings to pin versions or avoid cleanup, too.
‘brew pyenv-sync’ may be useful around this area, too.
‘brew pyenv-sync’ may be useful around this area, too.
If I pin a version of Python, isn't that going to wreck any tooling that depends on it? Unless you're saying have multiple versions of Python installed.
This is practically the only remaining annoyance I have with the Python ecosystem (relative imports aside). I use some tools, like Glances [0] whose formula relies on a much newer version (3.12) than the actual package requires (3.8) [1].
So when there's a Python update, all of those update as well. I thought I'd fixed this with pipx, but in a way that's worse, because the venvs it builds depend on a specific version of Python existing, which doesn't work well with brew always wanting to upgrade it.
I want a stable, system-level Python that I don't touch, don't add packages to, and which only exists as a dependency for anything that needs it. If an update would break a package I have installed (due to Python library deprecation, etc.), it should warn me before updating. Otherwise, I don't care, as long as any symlinks are taken care of.
Separately, I want a stable, user-level Python that I can do whatever I want to. Nothing updates it automatically. I can accomplish this by compiling Python and using `make altinstall`, but if there's a better way, I'd love to hear about it.
[0]: https://github.com/Homebrew/homebrew-core/blob/20e744191e74d...
[1]: https://github.com/nicolargo/glances
This is practically the only remaining annoyance I have with the Python ecosystem (relative imports aside). I use some tools, like Glances [0] whose formula relies on a much newer version (3.12) than the actual package requires (3.8) [1].
So when there's a Python update, all of those update as well. I thought I'd fixed this with pipx, but in a way that's worse, because the venvs it builds depend on a specific version of Python existing, which doesn't work well with brew always wanting to upgrade it.
I want a stable, system-level Python that I don't touch, don't add packages to, and which only exists as a dependency for anything that needs it. If an update would break a package I have installed (due to Python library deprecation, etc.), it should warn me before updating. Otherwise, I don't care, as long as any symlinks are taken care of.
Separately, I want a stable, user-level Python that I can do whatever I want to. Nothing updates it automatically. I can accomplish this by compiling Python and using `make altinstall`, but if there's a better way, I'd love to hear about it.
[0]: https://github.com/Homebrew/homebrew-core/blob/20e744191e74d...
[1]: https://github.com/nicolargo/glances
Pinning won’t break things but it may block upgrades (where they would break something or unpin).
Isn't it using the Python version in the formula?
Yes but there are multiple versioned formulae.
Thanks for taking the time. Can you help me understand? Because I didn't get this from the devs.
Just looking at a random formula, am I correct to understand that this will use python 3.10 and NOT 3.12?[0] I understand ones like this[1] where there's a note about the issue with newer versions.
What I'm trying to understand is if the python version is specified by the formula or it will default to the newest version. If it requires it to be specified in the formula then doesn't this make it contingent on the maintainer upgrading it every python version? I didn't verify [0], but it looks like it should work with later versions of python, and it it is still using 3.10 then isn't that essentially the maintainers "fault?" Because that's my concern. I can't see how something like this stays updated when it requires a maintainer to update. Seems better to have a >=3.10 and then do ==3.10 if only 3.10 works (odd) or >=3.10 <3.12 if it works for 10 and 11 but not 12. Especially since formulas are often made by people that are not the package developers themselves and we're just reliant upon someone keeping up. I'd rather break upon a new version than have many old pythons installed. I know there's no perfect solution, but we're talking about failure modes. And fwiw, I'd rather it try to use the system or environment python rather than installing a unique version. It just gets confusing when you need to add dependencies for optional stuff that wasn't included in the formula and is extremely non-obvious to a new user.
[0] https://github.com/Homebrew/homebrew-core/blob/12a0f6bbbeda8...
[1] https://github.com/Homebrew/homebrew-core/blob/12a0f6bbbeda8...
Just looking at a random formula, am I correct to understand that this will use python 3.10 and NOT 3.12?[0] I understand ones like this[1] where there's a note about the issue with newer versions.
What I'm trying to understand is if the python version is specified by the formula or it will default to the newest version. If it requires it to be specified in the formula then doesn't this make it contingent on the maintainer upgrading it every python version? I didn't verify [0], but it looks like it should work with later versions of python, and it it is still using 3.10 then isn't that essentially the maintainers "fault?" Because that's my concern. I can't see how something like this stays updated when it requires a maintainer to update. Seems better to have a >=3.10 and then do ==3.10 if only 3.10 works (odd) or >=3.10 <3.12 if it works for 10 and 11 but not 12. Especially since formulas are often made by people that are not the package developers themselves and we're just reliant upon someone keeping up. I'd rather break upon a new version than have many old pythons installed. I know there's no perfect solution, but we're talking about failure modes. And fwiw, I'd rather it try to use the system or environment python rather than installing a unique version. It just gets confusing when you need to add dependencies for optional stuff that wasn't included in the formula and is extremely non-obvious to a new user.
[0] https://github.com/Homebrew/homebrew-core/blob/12a0f6bbbeda8...
[1] https://github.com/Homebrew/homebrew-core/blob/12a0f6bbbeda8...
Yes you have to use the package version in the formula. Yes all python formulae will be moved to a newer version eventually. The other versions are for human consumption and use with tools like ‘brew pyenv-sync’.
(I am a Homebrew maintainer).
(I am a Homebrew maintainer).
The number of possible modes of failure in this situation is huge.
See also: "Why not tell people to "simply" use pyenv, poetry or anaconda"
https://www.bitecode.dev/p/why-not-tell-people-to-simply-use
I'm not saying pyenv is not a useful tool, but it is not a tool for beginners fighting with python packaging problems. It's a specialist tool to normalize your setup.
Very often, I see people tell me they don't have problems with pyenv, but later on have other unrelated problems with their dependencies. Analysis then prove it was because of pyenv, they just didn't know it. The cost is not obvious.