PEP 704 – Require virtual environments by default for package installers(peps.python.org)
peps.python.org
PEP 704 – Require virtual environments by default for package installers
https://peps.python.org/pep-0704/
61 comments
Written more Python than I'd like, and I still hate + avoid venvs.
They're purely incidental complexity: it should be able to set up a project-local venv by themselves without me having to micromanage anything. I don't care about activating/deactivating venvs. It should just work when I navigate to the project folder. It should let me tweak where packages are stored for the 1% of the time that I care.
They're purely incidental complexity: it should be able to set up a project-local venv by themselves without me having to micromanage anything. I don't care about activating/deactivating venvs. It should just work when I navigate to the project folder. It should let me tweak where packages are stored for the 1% of the time that I care.
>It should just work when I navigate to the project folder.
You still need to do the setup when you create the project, but once you do you can use direnv[0] to activate/deactivate the venv[1]. I target GCP often, so I have direnv switch over to the correct project [3] also.
[0] https://direnv.net/
[1] https://stackabuse.com/managing-python-environments-with-dir...
[2] https://stackoverflow.com/questions/56781493/gcloud-is-it-po...
You still need to do the setup when you create the project, but once you do you can use direnv[0] to activate/deactivate the venv[1]. I target GCP often, so I have direnv switch over to the correct project [3] also.
[0] https://direnv.net/
[1] https://stackabuse.com/managing-python-environments-with-dir...
[2] https://stackoverflow.com/questions/56781493/gcloud-is-it-po...
It always seems so backwards to me when I use Python... and kinda crazy to see in a "beginner" language.
I've always felt that it's weird that Python is considered a "beginner" language. Yeah, it uses "and" instead of "&&", but it took me a loooonnng time to figure out how to manage dependencies when I was starting out (and then figuring out how to deploy to targets, how to do reproducible builds, etc). Compared with Rust or Go which just do the right thing by default (personally I've always felt that the tooling is the way more important problem than language concepts which often feel like bike shedding to me).
I also despair that it's used as a beginner language. There are so many better options. I even include Java in that list.
So why is it used? As it has a fantastic eco-system for applied statistics / machine learning / various math things. If you're teaching an introduction to programming then it's a very effective language.
That and once people progress it's decent - much prefer the code written in Python by competent developers to that written in Java or even Go. It also sets people up to use more robust languages such as C# or (for those with a Math background) Haskell.
So why is it used? As it has a fantastic eco-system for applied statistics / machine learning / various math things. If you're teaching an introduction to programming then it's a very effective language.
That and once people progress it's decent - much prefer the code written in Python by competent developers to that written in Java or even Go. It also sets people up to use more robust languages such as C# or (for those with a Math background) Haskell.
There is not an actual need to activate for most use cases. I almost always just do
python3 -m venv .venv
.venv/bin/pip install whatever
.venv/bin/python foo.pyJust to give an example where activate is needed, if you have the tool programs `a` and `b` installed in the venv, and `a` invokes `b` (invokes as opposed to importing python modules), then:
.venv/bin/a
This is what I do in 99% of my projects. Or I use a tool like Hatch that creates envs for me, and I can use `hatch run ...` instead of manually invoking the path.
People like to complain about Python, but `./venv/bin/foo` is basically identical to `./node_modules/.bin/foo` and I don't see why the former deserves so much hate.
People like to complain about Python, but `./venv/bin/foo` is basically identical to `./node_modules/.bin/foo` and I don't see why the former deserves so much hate.
I think it's because you "have to" run a command to activate the virtual environment AKA use files that are already in the directory. With NodeJS (or any other platform/language), it's implied that the files in your current directory are the files you want to use for building/execution
Explicit is better than implicit, after all.
This defeats one of the coolest parts of virtualenvs which is that they can be placed anywhere. It’s just a directory.
Nothing about that proposal changes how venvs work. It just changes what the global python command does. Anything you do explicitly with a venv today, you could still do under that proposal.
This pep already partially does that.
> The error message SHOULD inform the user that a virtual environment is required, SHOULD provide shell-specific instructions on how to create and activate a virtual environment named .venv, and SHOULD provide a link to a documentation page that explains how to create and activate a virtual environment.
> The error message SHOULD inform the user that a virtual environment is required, SHOULD provide shell-specific instructions on how to create and activate a virtual environment named .venv, and SHOULD provide a link to a documentation page that explains how to create and activate a virtual environment.
I'd call this a minor convenience at best. Symlinks would still make it possible, of course.
I know it's not the end of the world, but I prefer to keep my dependencies outside of the project root in deployed scenarios. Ex, /srv/venv and /srv/app
Why do you say this is one of the coolest part?
Because the best part about unix/linux is that everything is a file (or more broadly, everything are files and folders) and this is congruent with that.
The more I program, the more I appreciate:
- 1. tools that are extremely small and functional and can be composed together (re: unix pipes, cat foo | grep bar)
- 2. things that are just files and folders on disk, so that they can get introspected, moved to different disks, symlinked, put on FUSE filesystems, etc. The common interface of file or folder is very flexible.
The more I program, the more I appreciate:
- 1. tools that are extremely small and functional and can be composed together (re: unix pipes, cat foo | grep bar)
- 2. things that are just files and folders on disk, so that they can get introspected, moved to different disks, symlinked, put on FUSE filesystems, etc. The common interface of file or folder is very flexible.
At some point i did this with gradle (i'm a Java guy):
https://hg.sr.ht/~twic/devtools/browse/bin/gradle?rev=tip
If i'm in a directory with a gradle wrapper, i essentially never want to run any globally installed gradle, i want to run the one in the wrapper.
It's more of a mixed bag with python. If i do:
If i do:
On the other hand, if i do:
So maybe resolve the file that will get run, then look for a venv in the vicinity of that?
What happens when you do:
Another option would be to introduce a new set of commands - vpython runs python using a nearby venv, vpip runs pip using it, etc. Or call the ppython, ppip, etc - less mnemonic, but faster to type.
https://hg.sr.ht/~twic/devtools/browse/bin/gradle?rev=tip
If i'm in a directory with a gradle wrapper, i essentially never want to run any globally installed gradle, i want to run the one in the wrapper.
It's more of a mixed bag with python. If i do:
cd ~/myproject
python main.py
If i do:
cd ~/myproject
salt '*' status.all_status
On the other hand, if i do:
python ~/myproject/main.py
So maybe resolve the file that will get run, then look for a venv in the vicinity of that?
What happens when you do:
cd ~/myproject
python -m json.tool data.json
Another option would be to introduce a new set of commands - vpython runs python using a nearby venv, vpip runs pip using it, etc. Or call the ppython, ppip, etc - less mnemonic, but faster to type.
I like activate setting the path. “python3”, “pip3” is the best idea anyone ever got since Solomon suggested cutting a baby in half.
I mean, soon it becomes “python3.5” (don’t really try that one!), “python3.6”, “python3.7”, “python3.8”, and maybe “python3.9-pypi” and who knows what else.
If the noobs you are pandering are too lazy to create a venv, they will never be able to change your script that says
I mean, soon it becomes “python3.5” (don’t really try that one!), “python3.6”, “python3.7”, “python3.8”, and maybe “python3.9-pypi” and who knows what else.
If the noobs you are pandering are too lazy to create a venv, they will never be able to change your script that says
pip3.12.8a-lyrical-magical install xyzyzy
This is in discussion, but is not straighforward. We need a story for system python which does not want this behavior, a security story so that an attacker cannot just pop a dir and make you execute it, a story for entry points, a story for parent env handling and so on.
pypackages is a proposal for this, but since it's harder to do, and this one easier, one step after the other...
pypackages is a proposal for this, but since it's harder to do, and this one easier, one step after the other...
That's more or less what PEP 582 plans to do, but it's been stalled and mired in discussions for years. The PDM tool went ahead and implemented it though if you want to use it: https://github.com/pdm-project/pdm
Loading code from the current working directory is not a good idea. It would make it unsafe to run a Python program from any directory that's not completely trusted, e.g. if we imagine "ls" having this feature:
Windows used to do this with DLL files and has since had to add various modes where it's either disabled or (in the default "safe DLL search mode") deprioritized in the search order: https://learn.microsoft.com/en-us/windows/win32/dlls/dynamic...
[root@localhost]# cd /home/someguy
[root@localhost]# ls
Windows used to do this with DLL files and has since had to add various modes where it's either disabled or (in the default "safe DLL search mode") deprioritized in the search order: https://learn.microsoft.com/en-us/windows/win32/dlls/dynamic...
That's not the proposal, the idea with PEP 582 is python's executable will check a local directory for a venv and activate it if necessary, but only when you actually run python (not automatically in your shell).
I was responding to the parent comment which suggested that Python "should look for a .venv directory in the current working directory".
Doing this "only when you actually run python" doesn't help. It would make it unsafe to run for example (looking at my /usr/bin) "ufw", "hg", "smbinfo", "samba-tool" or "routel" without first making sure that the current working directory (i.e. the one that you have currently set with "cd" in your shell) is not writable by any other user and doesn't contain anything unexpected.
Doing this "only when you actually run python" doesn't help. It would make it unsafe to run for example (looking at my /usr/bin) "ufw", "hg", "smbinfo", "samba-tool" or "routel" without first making sure that the current working directory (i.e. the one that you have currently set with "cd" in your shell) is not writable by any other user and doesn't contain anything unexpected.
>Doing this "only when you actually run python" doesn't help
Node.js does it this way, and it works just fine. Not sure what you think the problem is.
>Doing this "only when you actually run python" doesn't help. It would make it unsafe to run for example (looking at my /usr/bin) "ufw", "hg", "smbinfo", "samba-tool" or "routel" without first making sure that the current working directory (i.e. the one that you have currently set with "cd" in your shell) is not writable by any other user and doesn't contain anything unexpected
Why would the current working directory matter as to that?
First, if somebody has write access to your local directory you have worse issues.
They could for example overwrite the py/pyc dependencies or replace the python inside the virtual env, and their code will be executed if you run any of the items in the bin or even import those python deps.
Whether Python "looks for a .venv directory in the current working directory" or not doesn't mattter. It will just mean that you need to activate the .venv first to get those in your PATH. But that's why you had the .venv there in the first place, to activate and use it.
Whether you activate it manually or not doesn't matter...
Node.js does it this way, and it works just fine. Not sure what you think the problem is.
>Doing this "only when you actually run python" doesn't help. It would make it unsafe to run for example (looking at my /usr/bin) "ufw", "hg", "smbinfo", "samba-tool" or "routel" without first making sure that the current working directory (i.e. the one that you have currently set with "cd" in your shell) is not writable by any other user and doesn't contain anything unexpected
Why would the current working directory matter as to that?
First, if somebody has write access to your local directory you have worse issues.
They could for example overwrite the py/pyc dependencies or replace the python inside the virtual env, and their code will be executed if you run any of the items in the bin or even import those python deps.
Whether Python "looks for a .venv directory in the current working directory" or not doesn't mattter. It will just mean that you need to activate the .venv first to get those in your PATH. But that's why you had the .venv there in the first place, to activate and use it.
Whether you activate it manually or not doesn't matter...
To be clear, by "current working directory", I mean the directory returned by os.getcwd(), not the directory where the script is located. This is what I think the problem is:
1. One might want to run utility programs from arbitrary working directories, for example running "ls" in another user's directory (the example I gave in my first post), with "ls" being the utility program in this example.
2. One might want to implement such utility programs in Python, which is the case with e.g. "ufw", "hg" and "smbinfo" (the examples I gave in my second post).
3. If the suggestion to load Python modules from the current working directory is implemented, and a utility program is implemented in Python as in (2) and one runs it with a current working directory controlled by another party as in (1), then that party is able to execute arbitrary code as the user running the utility program, resulting in privilege escalation.
To make this more concrete, let's say we reimplement "ls" in Python as "ls.py" and install it in /usr/bin. If we then (quite reasonably, I think?) use our "ls.py" as root to list the files in /tmp, which is writable by other users...
1. One might want to run utility programs from arbitrary working directories, for example running "ls" in another user's directory (the example I gave in my first post), with "ls" being the utility program in this example.
2. One might want to implement such utility programs in Python, which is the case with e.g. "ufw", "hg" and "smbinfo" (the examples I gave in my second post).
3. If the suggestion to load Python modules from the current working directory is implemented, and a utility program is implemented in Python as in (2) and one runs it with a current working directory controlled by another party as in (1), then that party is able to execute arbitrary code as the user running the utility program, resulting in privilege escalation.
To make this more concrete, let's say we reimplement "ls" in Python as "ls.py" and install it in /usr/bin. If we then (quite reasonably, I think?) use our "ls.py" as root to list the files in /tmp, which is writable by other users...
# cd /tmp/
# /usr/bin/ls.py
It already is though. Python loads modules from the current directory. As soon as you import a non default module, its already over.
Does it? As far as I know, it loads modules from the script's directory (which is usually fine), not from the current working directory.
$ mkdir bin_dir
$ echo 'print("mymod loaded as expected.")' >bin_dir/mymod.py
$ echo 'import mymod' >bin_dir/script.py
$ mkdir working_dir
$ echo 'print("Oops, game over!")' >working_dir/mymod.py
$ cd working_dir/
$ python3 ../bin_dir/script.py
mymod loaded as expected.But you can have multiple virtual environments in the same directory, which one would it activate? You could also be using an environment that's in a different directory, or have one already loaded.
I like the concept potentially meaning less friction, but the implementation may make things more confusing for beginners when you're not explicitly activating an environment in your shell. I think maybe a flag to enable the behavior at runtime would be better than it'd be opt out.
I like the concept potentially meaning less friction, but the implementation may make things more confusing for beginners when you're not explicitly activating an environment in your shell. I think maybe a flag to enable the behavior at runtime would be better than it'd be opt out.
>But you can have multiple virtual environments in the same directory, which one would it activate?
So don't have many in the same directory. Since the name shall be fixed, you wouldn't be able to anyway (e.g. .venv, similar to .node_modules).
Simple as that.
So don't have many in the same directory. Since the name shall be fixed, you wouldn't be able to anyway (e.g. .venv, similar to .node_modules).
Simple as that.
.venv, obviously. Just like node and node_modules. No need to make this configurable beyond VIRTUAL_ENV
[deleted]
Different tool but:
https://asdf-vm.com/
Works like this
This would be a mess with python updates fwiw. venvs in my experience break when the system version of python is updated. By solving it this way, you open a whole other can of issues.
Virtualenvwrapper used to do this. It worked pretty flawlessly if I recall.
Not doing much Python now so I don't know what the current situation is.
Not doing much Python now so I don't know what the current situation is.
I have my Fish shell do this automatically when I cd into a directory with a .venv
Which plugin do you use?
I don't use a plugin, just a custom fish function that responds to directory changes. Mine is quite simple but also very specific to our systems, however this one looks very solid: https://gist.github.com/tommyip/cf9099fa6053e30247e5d0318de2...
Correct. This PEP makes sense if venvs are easier to work with
One of the biggest problem in python packaging is... not packaging.
Bootstrapping a workable dev env is hard: you can fail at installing python, setting it up, creating a venv, activating it...
This PEP is a step in the right direction: we don't want people that don't know what they are doing (which is a lot of python users, since it's a popular language among non developers) to install something outside of a venv.
First, it can break things. But also, it will probably not work, and if it does, it will bite them later on.
So the PEP is willing to break backward compatibility for pip to say "nope, you are not in a venv, you can't install a package" by default, and give instructions on how to do so to the user.
Ideally, we should have something like cargo or npm, which solve the entire problem altogether, but believe it or not, it's not easy to do for a project as old and successful as python.
This is a healthy compromise while something better is decided.
Bootstrapping a workable dev env is hard: you can fail at installing python, setting it up, creating a venv, activating it...
This PEP is a step in the right direction: we don't want people that don't know what they are doing (which is a lot of python users, since it's a popular language among non developers) to install something outside of a venv.
First, it can break things. But also, it will probably not work, and if it does, it will bite them later on.
So the PEP is willing to break backward compatibility for pip to say "nope, you are not in a venv, you can't install a package" by default, and give instructions on how to do so to the user.
Ideally, we should have something like cargo or npm, which solve the entire problem altogether, but believe it or not, it's not easy to do for a project as old and successful as python.
This is a healthy compromise while something better is decided.
> This PEP is a step in the right direction: we don't want people that don't know what they are doing (which is a lot of python users, since it's a popular language among non developers) to install something outside of a venv.
And even more "just someone that wants to install python app"
> So the PEP is willing to break backward compatibility for pip to say "nope, you are not in a venv, you can't install a package" by default, and give instructions on how to do so to the user.
I'd like if the default was just "create venv for user with default name and put it there".
So the above case of "user that just wants to install app" just works, and the crap pip spews is nicely contained.
And even more "just someone that wants to install python app"
> So the PEP is willing to break backward compatibility for pip to say "nope, you are not in a venv, you can't install a package" by default, and give instructions on how to do so to the user.
I'd like if the default was just "create venv for user with default name and put it there".
So the above case of "user that just wants to install app" just works, and the crap pip spews is nicely contained.
We cannot do that yet because not all systems come with venv install, thanks to linux.
I'm sure there's a reason that I'm just not aware of, but why is the PEP for __pypackages__ [1] in draft and not the obvious solution? It just seems that having those dependencies in the directory would resolve a good few issues without the need for the virtual environment. The big downside would be versioning of the Python interpreter, but at that point, we could still keep virtual environments for that case, right?
I assume I'm missing something big and someone who is more in-sync with the Python ecosystem can correct me.
[1] https://peps.python.org/pep-0582/
I assume I'm missing something big and someone who is more in-sync with the Python ecosystem can correct me.
[1] https://peps.python.org/pep-0582/
Too many cooks in the kitchen. There's a ton of competing ideas to 'fix' packaging in python, but no will or desire from the core to step up and own/shepherd a solution.
There's a recent podcast with most of the folks important to the ecosystem and they pretty quickly get to the realization that yes it sucks for users but it's not their domain and no one really owns or can fix python packaging: https://talkpython.fm/episodes/show/406/reimagining-pythons-...
There's a recent podcast with most of the folks important to the ecosystem and they pretty quickly get to the realization that yes it sucks for users but it's not their domain and no one really owns or can fix python packaging: https://talkpython.fm/episodes/show/406/reimagining-pythons-...
Can't say I'm a fan of this one. I get the idea, but I don't like the execution of this.
It's a very myopic view of python package management; python packages are also often just used to distribute complete applications (ie. youtube-dl), not just libraries. This is one of the reasons why pip nowadays warns you if you do a sudo installation to site-packages and otherwise defaults to user; binaries need to be installed to the right place and all this will likely end up doing is further mangling the amount of PATH fiddling necessary to get those packages working.
I also don't like that we're getting yet another expected reserve name for virtual environments. Originally it was virtualenv, then it became venv and now it could end up being .venv.
Python's package management could use quite a bit of work, especially with versioned dependencies co-existing, and venvs are a useful solution to that but this isn't the answer.
It's a very myopic view of python package management; python packages are also often just used to distribute complete applications (ie. youtube-dl), not just libraries. This is one of the reasons why pip nowadays warns you if you do a sudo installation to site-packages and otherwise defaults to user; binaries need to be installed to the right place and all this will likely end up doing is further mangling the amount of PATH fiddling necessary to get those packages working.
I also don't like that we're getting yet another expected reserve name for virtual environments. Originally it was virtualenv, then it became venv and now it could end up being .venv.
Python's package management could use quite a bit of work, especially with versioned dependencies co-existing, and venvs are a useful solution to that but this isn't the answer.
I run Python applications in containers. No need for a virtual environment.
Don't you need a container anyhow? Who want's any Python program and its 1000 dependencies to have access to the guest OS?
Don't you need a container anyhow? Who want's any Python program and its 1000 dependencies to have access to the guest OS?
Containers can be broken out of. They are not a security tool.
However, from a configuration perspective I totally agree. Python ecosystem is broken beyond repair. Just chuck it all in Docker and get back to programming!
However, from a configuration perspective I totally agree. Python ecosystem is broken beyond repair. Just chuck it all in Docker and get back to programming!
Containers can be broken out of.
Did you ask a skilled pen-tester?
I would really like this for the single reason that while there are many virtual environment or similar systems I see this as still a win for the beginner but, still having the flexibility of using other systems.
Honestly, having multiple virtual environment like systems is very confusing. I generally use condo and the venv module but I couldn't understand how to get poetry to work.
Honestly, having multiple virtual environment like systems is very confusing. I generally use condo and the venv module but I couldn't understand how to get poetry to work.
> I couldn't understand how to get poetry to work.
I recommend looking into it again. Conda is great for managing software environments, but when developing pure python packages, poetry is brilliant.
I recommend looking into it again. Conda is great for managing software environments, but when developing pure python packages, poetry is brilliant.
I always hated updates between python3 versions, with or without venvs, but I still installed many packages with pip from time to time because it was a reflex.
But Debian is currently pushing for this new requirement and it made me discover PIPX, which is a PIP including venvs and python version upgrades without any hassle.
Now I use APT or PIPX all the time.
But Debian is currently pushing for this new requirement and it made me discover PIPX, which is a PIP including venvs and python version upgrades without any hassle.
Now I use APT or PIPX all the time.
It’s about time. Other than “pip’s solving strategy is unsound” there are a handful of problems that make Python environments unreliable and this is one of the worst of them.
If eggs go away the “pip is unsound” problem will be easier to solve.
If eggs go away the “pip is unsound” problem will be easier to solve.
This is great, and I would make sure to move to 3.13+ because of this.
Why is Python the only language for which this is such a headache?
Now give me relocatable virtual environments and I'm game
If someone runs python it should look for a .venv directory in the current working directory. If it finds one, it should use the python interpreter located in that virtual environment instead of the system one. Obviously a flag can be used to disable this behavior and run the system interpreter no matter what the current directory is.
There’s an argument to be made that it should search upwards in the directory tree as well. In case the current directory is /home/user/myproject/otherdir/deeperdir/ and the .venv is in /home/user/myproject/.venv/ then the venv should still be used.
This not only saves a lot of trouble for users in the terminal, but also for various shell scripts, systemd service definitions, etc. Anyone who wants to run a Python program correctly just has to make sure they cd to the correct place first. Then they can just run python without having to know anything at all about virtualenvs.