Python Environment(xkcd.com)
xkcd.com
Python Environment
https://xkcd.com/1987/
185 comments
For a language that has "There should be one-- and preferably only one --obvious way to do it." as part of its core philosophy, this is very ironic.
It's aspirational, not prescriptive.
Just consider urllib, urllib2, urllib3, and requests. Or getopt, optparse, argparse, and docopt. Or countless others.
I appreciate that the Python community continually strives for ever-more obvious and ergonomic ways of accomplishing tasks. While there is an ecosystem complexity cost to the solutions-that-weren't, it's no greater than the churn than I've seen in other popular languages.
The Zen of Python even pokes fun at the "one obvious way" by writing its three em-dashes three different ways in the same document.
Just consider urllib, urllib2, urllib3, and requests. Or getopt, optparse, argparse, and docopt. Or countless others.
I appreciate that the Python community continually strives for ever-more obvious and ergonomic ways of accomplishing tasks. While there is an ecosystem complexity cost to the solutions-that-weren't, it's no greater than the churn than I've seen in other popular languages.
The Zen of Python even pokes fun at the "one obvious way" by writing its three em-dashes three different ways in the same document.
>> The Zen of Python even pokes fun at the "one obvious way" by writing its three em-dashes three different ways in the same document.
I did not know this! Thanks :)
I did not know this! Thanks :)
Was thinking the same thing. When python was getting started it was up against Perl, which toon the opposite approach and there was a myriad of ways to do something in the language.
Package managers for languages were just starting to become a thing back then. (At least I think that is true...). Now days you wouldn’t dream of releasing a new language without some kind of standard package manager...
Dunno where I’m going with this comment but it is definitely interesting to see how much has changed since then. Hell even Microsoft has a package manager for it’s languages...
Package managers for languages were just starting to become a thing back then. (At least I think that is true...). Now days you wouldn’t dream of releasing a new language without some kind of standard package manager...
Dunno where I’m going with this comment but it is definitely interesting to see how much has changed since then. Hell even Microsoft has a package manager for it’s languages...
> Now days you wouldn’t dream of releasing a new language without some kind of standard package manager...
Isn't this one of the major criticisms of Go?
Isn't this one of the major criticisms of Go?
My feeling is that a good package manager is the exception to the rule. Particularly around managing versions of the language runtime and the versions of packages available within projects using that runtime.
Rust is the exception here in that Cargo is a very nice piece of kit.
In terms of generic runtime version managers, ASDF is very nice cross language version manager for scoping versions to projects and switching between them easily.
Rust is the exception here in that Cargo is a very nice piece of kit.
In terms of generic runtime version managers, ASDF is very nice cross language version manager for scoping versions to projects and switching between them easily.
> Rust is the exception here in that Cargo is a very nice piece of kit
Also Rustup, which is pretty key to Cargo remaining flexible going forward. There's a huge advantage to having the build/dependency management piece being highly coupled with the language runtime/compiler. Making cargo work with all versions of Rust would mean having to build in backwards compability and make it much more difficult to do projects like the new modules changes.
It's a key insight that there needs to be two tools surrounding a language, not just one. You need good dependency management and good version management. Node also has this right with nvm and npm/yarn.
Also Rustup, which is pretty key to Cargo remaining flexible going forward. There's a huge advantage to having the build/dependency management piece being highly coupled with the language runtime/compiler. Making cargo work with all versions of Rust would mean having to build in backwards compability and make it much more difficult to do projects like the new modules changes.
It's a key insight that there needs to be two tools surrounding a language, not just one. You need good dependency management and good version management. Node also has this right with nvm and npm/yarn.
Well, it's not like Go if feature packed in other areas people ask it to either...
Yeah, and the exact same thing is happening.
That philosophy is pure and well intentioned. It will always fight with reality.
Pipenv solves a very real problem, but it has a lot of problems of its own:
- It's very slow: re-locking after updating 1 dependency often takes me ~1 minute.
- It has lots of bugs. To name a few in 11.6.9: clobbers comments in the Pipfile, inconsistently includes dependencies for other build environments in Pipfile.lock, stores the wrong index in Pipfile.lock for packages not on PyPI.
- They release multiple times per day, often breaking things in patch releases.
- Kenneth Reitz is quite unpleasant to deal with in GitHub issues, which I often have to because of the previous 2 issues.
From what I've heard, Pipenv "has been officially blessed" only insofar as its maintainer got commit access to PyPA's documentation and inserted a recommendation.
- It's very slow: re-locking after updating 1 dependency often takes me ~1 minute.
- It has lots of bugs. To name a few in 11.6.9: clobbers comments in the Pipfile, inconsistently includes dependencies for other build environments in Pipfile.lock, stores the wrong index in Pipfile.lock for packages not on PyPI.
- They release multiple times per day, often breaking things in patch releases.
- Kenneth Reitz is quite unpleasant to deal with in GitHub issues, which I often have to because of the previous 2 issues.
From what I've heard, Pipenv "has been officially blessed" only insofar as its maintainer got commit access to PyPA's documentation and inserted a recommendation.
I apologize for not meeting perfect standards of mood at all times. I have bipolar disorder, for what it's worth, and working on a packaging tool is very stress-inducing.
It’s not easy to develop for large communities, and even tougher to speak frankly about oneself.
Keep up the good work.
Keep up the good work.
Also:
> From what I've heard, Pipenv "has been officially blessed" only insofar as its maintainer got commit access to PyPA's documentation and inserted a recommendation.
This is blatantly false.
> From what I've heard, Pipenv "has been officially blessed" only insofar as its maintainer got commit access to PyPA's documentation and inserted a recommendation.
This is blatantly false.
The recommendation was added by Jon Wayne Parrott:
https://github.com/pypa/python-packaging-user-guide/pull/443
He doesn't seem to be a pipenv maintainer:
https://github.com/pypa/pipenv/commits?author=jonparrott
https://github.com/pypa/python-packaging-user-guide/pull/443
He doesn't seem to be a pipenv maintainer:
https://github.com/pypa/pipenv/commits?author=jonparrott
If what you've heard is true, it could be pretty toxic. Do you have any evidences to back up?
Sharing your awesome Python program is its Achilles heel. I have 5 computers I work on a weekly basis. I stopped using Python and Haskell due to all of this.
Racket's exe are so simple and quick to share.
(Edited tendon to heel)
Racket's exe are so simple and quick to share.
raco exe foo.rkt
(Edited tendon to heel)
I've thought about starting a "build an ecosystem for Racket" project where I document gaps in the extended ecosystem and review best-of-breed solutions from other languages as a kind of literature review / review of prior art.
Racket has such a great story for so many things that are painful in other environments. It's an under-appreciated language.
Racket has such a great story for so many things that are painful in other environments. It's an under-appreciated language.
Would you be willing to collaborate on this? Racket was my first programming language and I miss it dearly. An excuse to write code in it would be great!
Oh my yes. Of course. PM me, and let’s figure out the right vehicle to get started.
The only issues that this really could address. Would be nice to have a "Wall of Shame" for these gaps.
A) Racket is so enjoyable everyone builds their own solutions
B) "The view that full-fledged problem solving almost always calls for language design" Which sadly falls onto deaf ears especially people from Python.
A) Racket is so enjoyable everyone builds their own solutions
B) "The view that full-fledged problem solving almost always calls for language design" Which sadly falls onto deaf ears especially people from Python.
Yes, packaging Python applications for distribution is yet another nightmare situation.
Back in the day, we used one of the various "freeze" applications (cxfreeze, bbfreeze, probably others) and some tools developed in-house to package and distribute them for various platforms.
It has been a while since I actually needed to do this, so I don't know what the current state of the art is.
Back in the day, we used one of the various "freeze" applications (cxfreeze, bbfreeze, probably others) and some tools developed in-house to package and distribute them for various platforms.
It has been a while since I actually needed to do this, so I don't know what the current state of the art is.
> Python, in its noble quest for backwards compatibility, has accumulated so many different ways of packaging, distributing and installing libraries and apps that it's rivaling Google's chat apps.
I'm not sure it's because of backwards compatibility. Perl has amazing backwards compatibility, but doesn't seem to suffer from this.
That's not said to pump up Perl or degrade Python, but because if you misidentify the problem, your proposes solution has a much lower likelihood of fixing it.
(Then again, maybe you mean something different by "backwards compatibility" than what I thought you meant)
I'm not sure it's because of backwards compatibility. Perl has amazing backwards compatibility, but doesn't seem to suffer from this.
That's not said to pump up Perl or degrade Python, but because if you misidentify the problem, your proposes solution has a much lower likelihood of fixing it.
(Then again, maybe you mean something different by "backwards compatibility" than what I thought you meant)
As a Python dev it looks like I need to do some reading as I am still using virtualenv (it usually works fine but I have do encounter the occasional issue).
Can anyone give me a rundown of the benefits of other systems over virtualenv?
On a side note I hope Python isn't going to turn into a shambles like JavaScript - its certainly starting to look like it as far as package managers go.
Can anyone give me a rundown of the benefits of other systems over virtualenv?
On a side note I hope Python isn't going to turn into a shambles like JavaScript - its certainly starting to look like it as far as package managers go.
JS isn't in shambles; in fact, their package manager situation is excellent: Use yarn, period. It could be better (yarn isn't default yet), but I would love for the Python ecosystem to get to: "Use pipenv and setup.cfg, period".
But as I said, it's too young. If you use pipenv, you still have to interact with pip, so that's two binaries whose commands you have to remember. And urgh, it's so slow. And it's kinda useless for libraries... pipenv doesn't publish stuff, pipfiles aren't appropriate for packages.
But we'll get there. One day. Hopefully it won't be too late.
But as I said, it's too young. If you use pipenv, you still have to interact with pip, so that's two binaries whose commands you have to remember. And urgh, it's so slow. And it's kinda useless for libraries... pipenv doesn't publish stuff, pipfiles aren't appropriate for packages.
But we'll get there. One day. Hopefully it won't be too late.
So what are the benefits over virtualenv? You have given me the downside of another binary to manage. What do I get in return?
I've not given you anything other than a warning that it's too young.
Pipenv is promising. That means at some point I'm fairly certain it'll be and deserve to be the standard in Python packaging. In the mean time, it doesn't.
Pipenv is promising. That means at some point I'm fairly certain it'll be and deserve to be the standard in Python packaging. In the mean time, it doesn't.
That still doesn't answer the question of what it will (eventually) give me over virtualenv.
> Pipenv [...] has been officially blessed as "The One True Way" of installing stuff by the Python documentation.
Oh? I don't see any mention of pipenv in the cpython git repo.
Oh? I don't see any mention of pipenv in the cpython git repo.
Nor in the Python Packaging Authority documentation!
Pipenv is mentioned in the PYPA Python Packaging User Guide tool recommendations:
> Application dependency management
> Use Pipenv to manage library dependencies when developing Python applications. See Managing Application Dependencies for more details on using pipenv.
https://packaging.python.org/guides/tool-recommendations/
> Application dependency management
> Use Pipenv to manage library dependencies when developing Python applications. See Managing Application Dependencies for more details on using pipenv.
https://packaging.python.org/guides/tool-recommendations/
You're quite right. I skimmed Installing Packages, which mentions it in passing is a tool for managing multiple venvs, and Packaging & Distributing Projects, which doesn't mention it at all.
> To those suggesting it, Docker is great but you're still dealing with a package manager inside Docker, so that's a moot point.
Package manager/dependency manager is must for any sort of mature mass production coding environment (e.g. cargo for Rust, Maven for Java etc.) Docker doesn't solve all that but at least you are doing it only once and replicate easily and predictably. And a lot of times you can reuse Docker images built by someone else.
Package manager/dependency manager is must for any sort of mature mass production coding environment (e.g. cargo for Rust, Maven for Java etc.) Docker doesn't solve all that but at least you are doing it only once and replicate easily and predictably. And a lot of times you can reuse Docker images built by someone else.
If the "built by someone else" docker container contains a bunch of incomprehensible hacks or black magic, then you're really just handing yourself a nightmare down the road when it comes time to update that thing or apply security fixes.
Please forgive me for venting my frustration here, but my reaction to this is basically: oh great, yet another blinking tool to add to the list of misery that is the Python packaging experience. May as well check it out now before I have to do it in a hurry...
First of all it's not in Debian unstable. Ok, maybe it's just super new... so I'll install it with pip. A python3 -m pip install pipenv later and let's try it out!
$ cd ~/src/nexsan-exporter/nexsan-exporter $ pipenv install Creating a virtualenv for this project… Using /usr/bin/python3.6m (3.6.5) to create virtualenv… ⠋Running virtualenv with interpreter /usr/bin/python3.6m Using base prefix '/usr' New python executable in /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3.6m Also creating executable in /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python Installing setuptools, pip, wheel...done.
Virtualenv location: /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG Installing dependencies from Pipfile.lock (ca72e7)… 0/0 — 00:00:00 To activate this project's virtualenv, run the following: $ pipenv shell
Cute emoji and nice colours but... ~/.local/share? For a directory that will end up containing arch-specific libraries? Ok, I guess no one gives a shit about this in the modern world, oh well. Wait... "python3.6m"? That isn't the Python interpreter I asked for... but it seems to be a hardlink to the same file as /usr/bin/python3.6 so I guess maybe this is intentional? Anyway let's check out the venv...
$ ls -l ~/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3* lrwxrwxrwx 1 yrro yrro 10 Apr 30 17:06 /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3 -> python3.6m lrwxrwxrwx 1 yrro yrro 10 Apr 30 17:06 /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3.6 -> python3.6m -rwxr-xr-x 1 yrro yrro 4576440 Apr 30 17:06 /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3.6m
So this is the Python folk's what, fourteenth attempt to get this right, and they are still copying the python executable into the virtual environment instead of symlinking it in? This seems to be a regression from venv, which seemed to get this right!
Right, time to install my dependencies... according to 'pipenv' the right command for this is 'pipenv install -e .' Hm, I wonder exactly what the -e option does?
$ pipenv install --help
* no menition of -e in the output *
:sadface:
Oh well, let's just run it blind!
$ pipenv install -e . Installing -e .… ⠏ Error: An error occurred while installing -e .! Directory '.' is not installable. File 'setup.py' not found.
Maybe I screwed up and ran this from the wrong directory?
$ ls setup.py setup.py
Weird, what's going on here?
$ strace -f pipenv install -e . 2>&1 | grep setup\\.py stat("/home/yrro/src/nexsan-exporter/nexsan-exporter/setup.py", {st_mode=S_IFREG|0644, st_size=1769, ...}) = 0 stat("/home/yrro/src/nexsan-exporter/setup.py", 0x7ffc013a0200) = -1 ENOENT (No such file or directory) [pid 7365] stat("./setup.py", 0x7fff84176e30) = -1 ENOENT (No such file or directory) stat("/home/yrro/src/nexsan-exporter/setup.py", 0x7ffc013a0300) = -1 ENOENT (No such file or directory) write(2, "Directory '.' is not installable"..., 62Directory '.' is not installable. File 'setup.py' not found.
Oh FFS, I give up. I think I'll let pipenv pass me by for now.
First of all it's not in Debian unstable. Ok, maybe it's just super new... so I'll install it with pip. A python3 -m pip install pipenv later and let's try it out!
$ cd ~/src/nexsan-exporter/nexsan-exporter $ pipenv install Creating a virtualenv for this project… Using /usr/bin/python3.6m (3.6.5) to create virtualenv… ⠋Running virtualenv with interpreter /usr/bin/python3.6m Using base prefix '/usr' New python executable in /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3.6m Also creating executable in /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python Installing setuptools, pip, wheel...done.
Virtualenv location: /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG Installing dependencies from Pipfile.lock (ca72e7)… 0/0 — 00:00:00 To activate this project's virtualenv, run the following: $ pipenv shell
Cute emoji and nice colours but... ~/.local/share? For a directory that will end up containing arch-specific libraries? Ok, I guess no one gives a shit about this in the modern world, oh well. Wait... "python3.6m"? That isn't the Python interpreter I asked for... but it seems to be a hardlink to the same file as /usr/bin/python3.6 so I guess maybe this is intentional? Anyway let's check out the venv...
$ ls -l ~/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3* lrwxrwxrwx 1 yrro yrro 10 Apr 30 17:06 /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3 -> python3.6m lrwxrwxrwx 1 yrro yrro 10 Apr 30 17:06 /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3.6 -> python3.6m -rwxr-xr-x 1 yrro yrro 4576440 Apr 30 17:06 /home/yrro/.local/share/virtualenvs/nexsan-exporter-Eq5p1XVG/bin/python3.6m
So this is the Python folk's what, fourteenth attempt to get this right, and they are still copying the python executable into the virtual environment instead of symlinking it in? This seems to be a regression from venv, which seemed to get this right!
Right, time to install my dependencies... according to 'pipenv' the right command for this is 'pipenv install -e .' Hm, I wonder exactly what the -e option does?
$ pipenv install --help
* no menition of -e in the output *
:sadface:
Oh well, let's just run it blind!
$ pipenv install -e . Installing -e .… ⠏ Error: An error occurred while installing -e .! Directory '.' is not installable. File 'setup.py' not found.
Maybe I screwed up and ran this from the wrong directory?
$ ls setup.py setup.py
Weird, what's going on here?
$ strace -f pipenv install -e . 2>&1 | grep setup\\.py stat("/home/yrro/src/nexsan-exporter/nexsan-exporter/setup.py", {st_mode=S_IFREG|0644, st_size=1769, ...}) = 0 stat("/home/yrro/src/nexsan-exporter/setup.py", 0x7ffc013a0200) = -1 ENOENT (No such file or directory) [pid 7365] stat("./setup.py", 0x7fff84176e30) = -1 ENOENT (No such file or directory) stat("/home/yrro/src/nexsan-exporter/setup.py", 0x7ffc013a0300) = -1 ENOENT (No such file or directory) write(2, "Directory '.' is not installable"..., 62Directory '.' is not installable. File 'setup.py' not found.
Oh FFS, I give up. I think I'll let pipenv pass me by for now.
You're doing God's work, man. Thank you for helping me avoid that headache. I'm sticking with pip
I wouldn't go that far. I would absolutely love for some tool, and pipenv as as fine as any other, to replace everything else. But right now it doesn't seem to be ready.
It's absolutely not ready, as I made clear in several comments :) I just said pipenv is the most promising one.
I agree it's frustrating. I'm planning to donate some of my time to the project to help the situation. I hope others can too.
I agree it's frustrating. I'm planning to donate some of my time to the project to help the situation. I hope others can too.
Well I filed these as bugs this morning and got mixed feedback; the 'install -e .' but was caused by a stale Pipefile in the parent directory that I did not create, but at least removing it makes things work. Unfortunately upstream does not consider a stale copy of the python interpreter copied to dozens of directories on the filesystem that will never be patched to be an issue, so from my PoV pipenv gets a big raspberry and I'll stick with manually running 'python -m venv' which has gotten this right since Python 3.3!
What best practices do I teach to complete beginners to Python?
I am teaching adult beginners and it would be best to avoid the messy parts in the beginning.
I thought I could get by with just pip and briefly touching on virtualenv.
I sin myself by avoiding all the mess by cloning a full bare bones Lubuntu virtualization whenever I need a clean project.
Theoretically Docker should suffice but it is yet another layer of complexity.
I am teaching adult beginners and it would be best to avoid the messy parts in the beginning.
I thought I could get by with just pip and briefly touching on virtualenv.
I sin myself by avoiding all the mess by cloning a full bare bones Lubuntu virtualization whenever I need a clean project.
Theoretically Docker should suffice but it is yet another layer of complexity.
There was Zope Buildout for years, that had declarative packaging format, used the cheese shop (biggest issue with conda) and isn't slow as molasses but it was and still is largely ignored by the larger community.
I just started using pipenv for a small personal project, and I like it. The usage is similar to git which I prefer over the weird bash activate mangling that a normal virtualenv does.
Python environment headaches have been largely solved for me thanks to Miniconda[1]. Not only do the environments isolate dependencies, they can easily use different versions of Python, and can include arbitrary binary packages too. It helps that for my work in biology an extensive number of packages are available from the bioconda[2] channel (with many non-bio packages from conda-forge[3]). Environments can be described via environment files, allowing them to be transferred to collaborators, managed with source control, or included with publications to support reproducible science. If you like virtualenvs, you may want to give Miniconda a try. One current limitation is that environment files cannot specify a source channel for specific packages--they're installed from channels based on the global channel precedence.
1. https://conda.io/miniconda.html
2. https://bioconda.github.io
3. https://conda-forge.org
1. https://conda.io/miniconda.html
2. https://bioconda.github.io
3. https://conda-forge.org
I used Conda recently and it was OK, but it's annoying that it can't install stuff directly from PyPI. Unless you're lucky, you still have to use Pip inside the Conda environment to install some packages.
A sparkle of hope is that the conda people recently announced in a github ticket that this would be supported as a first class citizen soon.
But it's annoying how many packages like qiime make it basically a requirement to use Anaconda/miniconda. I really don't want to pollute my computer with multiple installations of even things like R just to run a package.
What if the right answer to the packaging nightmare that plagues languages with native extensions, is to actually have multiple isolated copies of the interpreter, each with well-known build chain metadata and sane, isolated locations for shared libraries?
With 8 comments so far, I count seven solutions presented.
I thought I was the only one that noticed. It's certainly makes it clear why this type of situation persists.
The alt-text is especially funny, now that we're in the world of USB-C and Thunderbolt 3, etc...
Dongles for days.
Dongles for days.
Honestly, Python on OSX is the most confusing of all.
The OS comes with a python runtime, which is different from the runtime you can download from python.org.
You can then also install it from Homebrew or from MacPorts.
I'm probably even missing something.
The OS comes with a python runtime, which is different from the runtime you can download from python.org.
You can then also install it from Homebrew or from MacPorts.
I'm probably even missing something.
Should I set up this project with conda, miniconda, virtualenv, docker, some combination of the above...? Fuck it, I'll just use R.
You can install R and some commonly used R packages with:
conda install r-essentials
conda install r-essentials
Homebrew installing Python and installing virtualenv and virtualenvwrapper makes life easy.
All of these methods people are listing that requiring me to do more than manage one file (requirements.txt) is sort of missing the point.
I get that a lot of Node developers are coming to Python to do things, and that's great. But the environments don't need the exact same tools. pipenv seems to fix a problem I never encounter with pip/virtualenv during daily development work.
All of these methods people are listing that requiring me to do more than manage one file (requirements.txt) is sort of missing the point.
I get that a lot of Node developers are coming to Python to do things, and that's great. But the environments don't need the exact same tools. pipenv seems to fix a problem I never encounter with pip/virtualenv during daily development work.
When I first got mac, I've spent the whole evening trying to install python3+pipenv and get it working properly. Ended up nuking system's python 2.7 and installing it through homebrew.
This was also frustrating me til recently. My imperfect solution is to use pyenv, which you can get through homebrew. Try it out.
Still missing from the picture: pipenv, virtualenv, virtualenvwrapper, pyenv
I think the comic should be more aptly be titled "Life before pipenv".
Pipenv is the first (popular) sane tool to make Python accessible to programmers coming from other languages. Sure you can also just use bare virtualenv, but then you have to figure out how it works together with everything displayed in the picture.
Pipenv is the first (popular) sane tool to make Python accessible to programmers coming from other languages. Sure you can also just use bare virtualenv, but then you have to figure out how it works together with everything displayed in the picture.
Pipenv has been my solution as well. It handles venvs for all my projects, dev/prod configuration files, and auto-generation of requirements files.
> Sure you can also just use bare virtualenv, but then you have to figure out how it works together with everything displayed in the picture.
How so? I use virtualenv and don't have any issues. Seems pretty straight forward to me. I haven't moved to pipenv because it forces a folder structure on you, which I hate. The great thing about virtualenv is that you can be anywhere on your system, it doesn't matter, as long as you see the project name in paren's next to your prompt you know you are good.
How so? I use virtualenv and don't have any issues. Seems pretty straight forward to me. I haven't moved to pipenv because it forces a folder structure on you, which I hate. The great thing about virtualenv is that you can be anywhere on your system, it doesn't matter, as long as you see the project name in paren's next to your prompt you know you are good.
I don't know the specific problems anymore, but I've set up virtualenv on ~5 separate occasions for my machines. Every time it was a long painful session of trial end error, that ended with a setup that was mostly stable (but also sometimes fell apart when I updated the system/homebrew ?) and also seemed to never be completely shielded off from othe dependencies I had installed globally. The workflow with Pipenv was the first stable, no hassle solution I experienced with Python.
> The great thing about virtualenv is that you can be anywhere on your system
That sounds like an anti-pattern to me. Each project (= directory) should bring its own environment with it and should be completely decoupled from the environment of other projects.
> The great thing about virtualenv is that you can be anywhere on your system
That sounds like an anti-pattern to me. Each project (= directory) should bring its own environment with it and should be completely decoupled from the environment of other projects.
Well, my experience has been drastically different than yours. Virutalenv has been flawless for me.
>That sounds like an anti-pattern to me. Each project (= directory) should bring its own environment with it and should be completely decoupled from the environment of other projects.
Yes they should be decoupled, but why would I want in the same folder? If am building a Django app I don't want the entire instance of python inside my project folder for a few reasons. 1) It clutters it up. 2) The web service (user/group) needs to access the Django folder, why would I put python in a folder that the web service has control over? I much rather have my web application in /svr/www/webapp and my python virtualenv in /opt/python-virtual/webappname
>That sounds like an anti-pattern to me. Each project (= directory) should bring its own environment with it and should be completely decoupled from the environment of other projects.
Yes they should be decoupled, but why would I want in the same folder? If am building a Django app I don't want the entire instance of python inside my project folder for a few reasons. 1) It clutters it up. 2) The web service (user/group) needs to access the Django folder, why would I put python in a folder that the web service has control over? I much rather have my web application in /svr/www/webapp and my python virtualenv in /opt/python-virtual/webappname
Regarding '1)': It doesn't have to bin _in_ the directory, and AFAIK it isn't with pipenv.
Regarding '2)': That sounds like a deployment issue? Certainly nothing I've experienced on my development machine. With pipenv your still free to generate a requirements.txt and deploy your app with whatever webserver setup you want.
>With pipenv your still free to generate a requirements.txt and deploy your app with whatever webserver setup you want
yeah but then I can't be in the Django web application folder and just type "python manage.py makemigrations", for example. I would have to be the pipenv folder then "python /srv/www/webapp/manage.py makemigrations".
Unless I am missing something, which could be the case.
yeah but then I can't be in the Django web application folder and just type "python manage.py makemigrations", for example. I would have to be the pipenv folder then "python /srv/www/webapp/manage.py makemigrations".
Unless I am missing something, which could be the case.
You would be in the Django web application folder (which has a Pipfile) and run "pipenv run python manage.py makemigrations" or run "pipenv shell" and then "python manage.py makemigrations". There is no "pipenv folder" you need to think about. There is a virtualenv folder somewhere on the system, but activation of that etc. is all managed by the pipenv command and tied to a project.
Let’s say I want Anaconda. Is pipenv the right way to go? Do they play nicely?
Not to be glib, but your comment ironically reminds me of this: https://xkcd.com/927/
No amount of adding things to the Python comic is going to simplify it.
No amount of adding things to the Python comic is going to simplify it.
I think the intent here is to convince adherents of other tools to switch to pipenv. Having used most of the solutions in this thread, pipenv has been the simplest and the one that "just works" most often.
Yeah, that one also popped into my mind when I wrote the comment.
> No amount of adding things to the Python comic is going to simplify it.
It doesn't simplify it, but it puts you in a whole new canvas where you don't have to worry about whats in the messed up one.
> No amount of adding things to the Python comic is going to simplify it.
It doesn't simplify it, but it puts you in a whole new canvas where you don't have to worry about whats in the messed up one.
they also forgot the part, where every package manager has his own way of installing python packages
All of which work fine, the real problem is installing Python with homebrew. If you just install Python from the binary on the website then any of those solutions will work with minimal drama.
Why is this? What's the difference between Homebrew Python and website Python?
https://github.com/Homebrew/homebrew-core/blob/master/Formul...
Homebrew pulls it directly from the Python.org, but it's also slightly opinionated in what else it installs. I would suggest just reading through the formula to see what it does (it's straighforward).
Likewise, you can quickly see what any Homebrew package is doing with `brew info <package>` and looking at the "From:" link.
Homebrew pulls it directly from the Python.org, but it's also slightly opinionated in what else it installs. I would suggest just reading through the formula to see what it does (it's straighforward).
Likewise, you can quickly see what any Homebrew package is doing with `brew info <package>` and looking at the "From:" link.
Don’t forget the shell specific stuff that wraps virtualenv. Thinking of the OhMyZSh plugins that do cool things on top of virtualenvwrapper...
And `pyenv-virtualenv`!
It's funny because within the comment section there are already 2 different opinions on why this is not funny because $xyz is what every sane python dev should use.
There was a time where I would have strongly related to this. Thankfully, now I've started using pipenv whenever possible and it basically just works(TM).
I hope it becomes the standard going forward.
I hope it becomes the standard going forward.
pipenv doesn't solve any problems well, especially if you require multiple versions of Python, which is _always_ a problem if you're developing libraries. You still need something like pyenv to manage multiple versions of Python.
If you're using pyenv, it's easy to do virtualenv with the pyenv-virtualenv plugin, at which point you don't need Pipenv because it doesn't solve anything (except locking, see blow) that you can't do with requirements files. Furthermore, if you're testing under multiple versions of Python, you'd use tox to manage your test environments anyway.
_Further_ furthermore, they authors recommend _against_ use pipenv to manage dependencies for anything but development, so it's on you to have a copy of runtime dependencies isolated from Pipfile. And since you have to put your runtime requirements in setup.py anyway, there was never any way around the issue of duplicating requirements to begin with.
The one thing Pipenv does that pip and requirements files can't is lock dependencies down the entire tree... except it's abominably slow at that because it has to download dependencies and walk the entire tree. There's a debate ongoing about using pip's cache to solve the problem but there's no end in sight.
If you're really sensitive to having the entire dependency tree frozen to exact versions of you requirements, just `pip freeze` them occasionally. It's way less hassle.
Packaging in Python is far better than it was a few years ago. There's still a lot to improve but pipenv shouldn't be part of the solution.
If you're using pyenv, it's easy to do virtualenv with the pyenv-virtualenv plugin, at which point you don't need Pipenv because it doesn't solve anything (except locking, see blow) that you can't do with requirements files. Furthermore, if you're testing under multiple versions of Python, you'd use tox to manage your test environments anyway.
_Further_ furthermore, they authors recommend _against_ use pipenv to manage dependencies for anything but development, so it's on you to have a copy of runtime dependencies isolated from Pipfile. And since you have to put your runtime requirements in setup.py anyway, there was never any way around the issue of duplicating requirements to begin with.
The one thing Pipenv does that pip and requirements files can't is lock dependencies down the entire tree... except it's abominably slow at that because it has to download dependencies and walk the entire tree. There's a debate ongoing about using pip's cache to solve the problem but there's no end in sight.
If you're really sensitive to having the entire dependency tree frozen to exact versions of you requirements, just `pip freeze` them occasionally. It's way less hassle.
Packaging in Python is far better than it was a few years ago. There's still a lot to improve but pipenv shouldn't be part of the solution.
> _Further_ furthermore, they authors recommend _against_ use pipenv to manage dependencies for anything but development, so it's on you to have a copy of runtime dependencies isolated from Pipfile.
I think the use-case is slightly different than that.
Pipenv is a tool that enables reproducible builds for python app development and deployment. For example, many web apps, aren't going to be packaged up using setuptools, they are just going to be deployed somewhere (possibly containerized first). And with pipenv you can avoid the need to vendor all your dependancies to ensure reproducible builds.
But its not a tool that solves problems for library packaging and distribution (on say pypi). You still have to use setup.py for that, and shouldn't be pinning your dependancies on exact package versions. But you can still use pipenv to manage your dev environment in those cases, and for reproducible dev environments.
I think the use-case is slightly different than that.
Pipenv is a tool that enables reproducible builds for python app development and deployment. For example, many web apps, aren't going to be packaged up using setuptools, they are just going to be deployed somewhere (possibly containerized first). And with pipenv you can avoid the need to vendor all your dependancies to ensure reproducible builds.
But its not a tool that solves problems for library packaging and distribution (on say pypi). You still have to use setup.py for that, and shouldn't be pinning your dependancies on exact package versions. But you can still use pipenv to manage your dev environment in those cases, and for reproducible dev environments.
Seconding this recommendation of pipenv.
https://github.com/pypa/pipenv
It combines many features that people manage separately with virtualenv, pip, and any custom scripts. I think it reduces boilerplate and simplifies env management.
It is still under active development, but I think it is stable enough to use for production. I think it's probably better for people learning Python to use too.
https://github.com/pypa/pipenv
It combines many features that people manage separately with virtualenv, pip, and any custom scripts. I think it reduces boilerplate and simplifies env management.
It is still under active development, but I think it is stable enough to use for production. I think it's probably better for people learning Python to use too.
I believe pipenv still uses virtualenv instead of python's venv. Isn't that a drawback to use it?
Who would know?
Well, if more and more packages in future move to python's offering venv. It would certainly be a problem that right now we are already facing
Yes, pipenv is great and I am hopeful it gets adopted into the standard.
Agreed. pipenv is wonderful.
Highly recommend pipenv.
"This is not a place of honor"
https://news.ycombinator.com/item?id=11851871 https://archive.is/MVVU1
Explanation edit:
Image alt text is "The Python environmental protection agency wants to seal it in a cement chamber, with pictoral messages to future civilizations warning them about the danger of using sudo to install random Python packages."
https://news.ycombinator.com/item?id=11851871 https://archive.is/MVVU1
Explanation edit:
Image alt text is "The Python environmental protection agency wants to seal it in a cement chamber, with pictoral messages to future civilizations warning them about the danger of using sudo to install random Python packages."
Care to explain a little? The link in the discussion is a broken error page.
"This place is not a place of honor" is the first line of the most basic signage created by a Sandia National Lab project aiming to keep future explorers/archaeologists/looters/settlers unharmed by a nuclear waste dump.
The problem space is very difficult - it'll be something like AD 12000 before the waste is effectively harmless. That's further in the future than all of civilization[1] is behind us. The earliest surviving language specimens are proto-heiroglyphics that are less than 5000 years old.
It's difficult to craft a message that won't be taken like all the myriad curse-promising looter-deterrents guarding e.g. plundered tombs throughout the world. It's especially unconvincing when the hazards we're trying to protect future humans from include things like a hundredfold increased risk of cancer.....after inhabiting the site for twenty years.
It's further difficult to attempt to ensure that the message survives that massive amount of time. The designers came up with a linked system where elements are supposed to reinforce and index each other, with redundancy of information and presentation at multiple levels of complexity, with written and pictographic forms, attempting to avoid overstatement, attempting to ensure that protective structures cannot be usefully scrapped and re-used for new construction, ensure that the communications resist deliberate vandalization, etc.
1: https://en.wikipedia.org/wiki/History_of_the_world#Rise_of_c... claims "Though early 'cities' appeared at Jericho and Catal Huyuk around 6000 BCE,[32] the first civilizations did not emerge until around 3000 BCE in Egypt[33] and Mesopotamia.[34]"
The problem space is very difficult - it'll be something like AD 12000 before the waste is effectively harmless. That's further in the future than all of civilization[1] is behind us. The earliest surviving language specimens are proto-heiroglyphics that are less than 5000 years old.
It's difficult to craft a message that won't be taken like all the myriad curse-promising looter-deterrents guarding e.g. plundered tombs throughout the world. It's especially unconvincing when the hazards we're trying to protect future humans from include things like a hundredfold increased risk of cancer.....after inhabiting the site for twenty years.
It's further difficult to attempt to ensure that the message survives that massive amount of time. The designers came up with a linked system where elements are supposed to reinforce and index each other, with redundancy of information and presentation at multiple levels of complexity, with written and pictographic forms, attempting to avoid overstatement, attempting to ensure that protective structures cannot be usefully scrapped and re-used for new construction, ensure that the communications resist deliberate vandalization, etc.
1: https://en.wikipedia.org/wiki/History_of_the_world#Rise_of_c... claims "Though early 'cities' appeared at Jericho and Catal Huyuk around 6000 BCE,[32] the first civilizations did not emerge until around 3000 BCE in Egypt[33] and Mesopotamia.[34]"
[deleted]
To add on to the mess, Poetry seems like a strong contestant for a well-designed Python dependency management solution: https://github.com/sdispater/poetry
Did they forget site-packages vs dist-packages?
It’s so tempting to just apt install everything until it’s too late.
It’s so tempting to just apt install everything until it’s too late.
I forgot about that. That annoys me to no end, and that's purely due to the Debian/Ubuntu package maintainers right?
What's wrong with dist-packages?
Sure, it caused a lot of grief for Debian packagers (because the name "site-packages" was hardcoded everywhere) when it was introduced back in the Python 2.6 era; but I don't think I've ever had to worry about it as a user.
Sure, it caused a lot of grief for Debian packagers (because the name "site-packages" was hardcoded everywhere) when it was introduced back in the Python 2.6 era; but I don't think I've ever had to worry about it as a user.
Users that are programmers also have to package Python packages. Sometimes in .deb form for Ubuntu too...
It's not just official Debian packages that someone needs to maintain...
It's not just official Debian packages that someone needs to maintain...
Sorry, idk. I just went that route and then was like well, maybe this time I should simply return and rethink the choice of a language. Didn’t do much credit to an engineer, but honestly, I didn’t feel myself a mess cleaner that day.
One thing that amazes me about the modern PHP ecosystem is that when it comes to distributable PHP packages, there is only one package manager (composer) and one central package repo (packagist).
When it comes to binaries it's still a bit of a mess (homebrew? pecl? yum?) but the native PHP code story is clearer than any other language ecosystem I've seen.
When it comes to binaries it's still a bit of a mess (homebrew? pecl? yum?) but the native PHP code story is clearer than any other language ecosystem I've seen.
Am I the only one using Nix (the package manager) for this? It's lovely and works for other languages and binary package management as well.
I encourage people to try it out, interactive usage looks something like `nix-shell -p python36Packages.pyaml -p python36Packages.aiohttp --run "python some-script-with-dependencies.py" `.
I encourage people to try it out, interactive usage looks something like `nix-shell -p python36Packages.pyaml -p python36Packages.aiohttp --run "python some-script-with-dependencies.py" `.
But why.. every sane programmer uses Python 3 and pip in a virtualenvwrapper environment, cleanly separated for non-pip packages!
I've never understood the purpose of virtualenvwrapper. I only use three commands for administering virtual environments: "virtualenv -p python3 ~/envs/foo" to make one, ". ~/envs/foo/activate" to activate it, and "rm -r ~/envs/foo" to delete it. That is really not something that needs further simplification. What am I missing?
Virtualenvwrapper (I hate the name, and as a fish user I actually use virtualfish instead) has as the main advantage in keeping all virtualenvs in one place (by default ~/.virtualenvs), so you don't have to worry about the actual location of your virtualenv. This allows for a number of useful features like activating, listing and deleting available environments with autocomplete, and creating a virtualenv is also quite simplified.
But yes, if you're making sure that you're always careful about passing the location of your environments, you don't really need it.
But yes, if you're making sure that you're always careful about passing the location of your environments, you don't really need it.
It always felt like virtualenvwrapper was moving things in the wrong direction. Why should the virtualenv be located in the home directory, instead of in the project directory? The whole point of virtualenv is so that you don't have environment dependencies, and each project can have its own environment.
Treating the environment files as a build artifact does make more sense than having a directory of environments somewhere else.
I'm not sure there are any practical advantages to doing it that way, though. I name my environments after the project, so it's more like the environment's folder is a secondary project folder than part of the environment.
Edit: It looks like pipenv, which hopefully will replace virtualenvwrapper and the like, supports storing the environment in the project folder: https://docs.pipenv.org/advanced/#changing-where-pipenv-stor...
Edit: It looks like pipenv, which hopefully will replace virtualenvwrapper and the like, supports storing the environment in the project folder: https://docs.pipenv.org/advanced/#changing-where-pipenv-stor...
I'm not sure how the location of the environment affects dependencies; the environment is isolated regardless of where it is located. Also, the virtualenv location doesn't have to be in the home directory; it can be anywhere your user has access to, the home is simply most convenient.
Using a common location means that unique names must be maintained across multiple projects, which is a recipe for disaster.
Most definitely in automated setups, i.e. in production. (I personally never depend on virtualenvs in production, preferring to have a fully isolated environment either via Docker or even better a dedicated VM.)
But in development, where you manually switch between environments, a centralised setup is great. You don't have to worry about gitignoring the virtualenv directory, or maintaining paths in general -- a common problem with virtualenv in your code directory is that IDEs and linters and similar tools tend to just cut through and parse everything, unless explicitly prevented. With virtualfish/virtualenvwrapper, the process is simply `workon {envname}` and you have everything in place.
But in development, where you manually switch between environments, a centralised setup is great. You don't have to worry about gitignoring the virtualenv directory, or maintaining paths in general -- a common problem with virtualenv in your code directory is that IDEs and linters and similar tools tend to just cut through and parse everything, unless explicitly prevented. With virtualfish/virtualenvwrapper, the process is simply `workon {envname}` and you have everything in place.
> But why
Legacy setups from previous decades, laziness, automated installers written by others (so they play by different rules), semi-broken packages with messed-up dependency graphs that require manual treatment. The comic is, of course, exaggerating - or at least I hope no one have things gone that bad.
> every sane programmer uses ... pip in a virtualenvwrapper environment
Check out Pipenv <https://docs.pipenv.org/> - you may like it. It aims to make things saner that bare pip+virtualenv{,wrapper}, and IMHO it really does.
Legacy setups from previous decades, laziness, automated installers written by others (so they play by different rules), semi-broken packages with messed-up dependency graphs that require manual treatment. The comic is, of course, exaggerating - or at least I hope no one have things gone that bad.
> every sane programmer uses ... pip in a virtualenvwrapper environment
Check out Pipenv <https://docs.pipenv.org/> - you may like it. It aims to make things saner that bare pip+virtualenv{,wrapper}, and IMHO it really does.
I guess what Python needs to do is to crown one solution as the official, and then start imposing on PyPA packages to convert before a deadline or exclude. Then they can move the non-compliant packages to a legacy archive, and build on the one solution that the community chose, say in a poll. Granted I've only ever interacted pip and virtualenv and though it was simple; but looking at this, things don't have to be this complex.
The heart of the problem is that different people are doing different things with Python, and nobody wants to pay cognitive load for what they don't want to use.
The packagers and package tool makers of the world inherit ALL of the technical debt from ALL of the upstream software devs. They're either the liver or the colon. An upstream dev decided to not document which compiler they use during dev time? That's now your problem. A compiler maker (GNU, MSFT, etc.) decides to change how they distribute the C runtime? Congrats, now it's your packaging system's job to know how to differentiate between Windows 7 and Windows 10 running particular versions of Visual Studio.
Nobody wants a "single simple unified solution" for packaging more than the packaging tool makers and distro vendors. Believe me. But it's not going to happen until software devs become take more responsibility for what they build upstream, and how they build it.
The packagers and package tool makers of the world inherit ALL of the technical debt from ALL of the upstream software devs. They're either the liver or the colon. An upstream dev decided to not document which compiler they use during dev time? That's now your problem. A compiler maker (GNU, MSFT, etc.) decides to change how they distribute the C runtime? Congrats, now it's your packaging system's job to know how to differentiate between Windows 7 and Windows 10 running particular versions of Visual Studio.
Nobody wants a "single simple unified solution" for packaging more than the packaging tool makers and distro vendors. Believe me. But it's not going to happen until software devs become take more responsibility for what they build upstream, and how they build it.
Pypi and Python devs themselves could start doing the curation. The package just isn't accepted into the repo if it doesn't follow X, Y, Z guidelines.
Python is big and important enough that it needs these kinds of quality assurances. Punting on them is as bad as compromising on language design decisions.
Python is big and important enough that it needs these kinds of quality assurances. Punting on them is as bad as compromising on language design decisions.
This doesn't reflect the reality of what devs should be using: virtualenv or docker.
I use Docker and like it over virutalenvs, but even without it, if you're just playing around/sandboxing, pip3 --user is your friend. Don't use pip for system packages ever! Let your distros package manager handle that.
Docker doesn't solve the problem, and virtualenv is irrelevant to the discussion. It's just another option -- all other options already exist, will continue to exist, and can interact with your Python setup in subtle ways (not to mention one might need to distribute their python code).
Why does the official python documentation recommend something else then?
https://packaging.python.org/guides/tool-recommendations/#ap...
https://packaging.python.org/guides/tool-recommendations/#ap...
This is what I learned. This is what I use. It's nice to have more options though. Unfortunately that makes it all the more confusing.
I guess if you view it from a newcomers perspective it can be quite daunting, especially on operating systems like OSX.
Yea macos is kinda a clusterfuck when it comes to dealing with Linux packages. Now you're fighting between system, brew and pip and none of it is fun. At least in Linux you're pretty much dealing with system python packages and user python packages.
Maybe this is my experience biasing things, but it doesn't seem that complicated to me even on Mac.
- Install brew w/ the one-liner on https://brew.sh
- `brew install python3` (which includes pip) and any system dependencies you need: C libraries, etc. For example, `brew install openssl libpq libffi libsass libsodium ossp-uuid` and so on
- Create a virtualenv and install all python stuff in there. Doing `virtualenv -p python3 venv` in the root of your project works. But, if you want to get fancy, there's pyenv, pyvenv, and the like. Use one that works for you.
This seems to be the same as the experience with Linux, except you don't need to install the package manager. In Docker, you also don't need a virtualenv.
Or maybe people are using more difficult to install dependencies than I am? Even if you have to install from source though, it's not a huge deal.
Another possibility is people are using IDE's that want you to do things "their way". I don't know anything about this b/c I use a terminal-based text editor (not trying to start a flamewar).
And the final thought is just that people are not experienced with package managers or general terminal use. You obviously have to be fluent in terminal to use the workflow I outlined above.
- Install brew w/ the one-liner on https://brew.sh
- `brew install python3` (which includes pip) and any system dependencies you need: C libraries, etc. For example, `brew install openssl libpq libffi libsass libsodium ossp-uuid` and so on
- Create a virtualenv and install all python stuff in there. Doing `virtualenv -p python3 venv` in the root of your project works. But, if you want to get fancy, there's pyenv, pyvenv, and the like. Use one that works for you.
This seems to be the same as the experience with Linux, except you don't need to install the package manager. In Docker, you also don't need a virtualenv.
Or maybe people are using more difficult to install dependencies than I am? Even if you have to install from source though, it's not a huge deal.
Another possibility is people are using IDE's that want you to do things "their way". I don't know anything about this b/c I use a terminal-based text editor (not trying to start a flamewar).
And the final thought is just that people are not experienced with package managers or general terminal use. You obviously have to be fluent in terminal to use the workflow I outlined above.
Surely you mean venv or docker! ;)
Maybe I just never do anything complicated with Python, but I never have any the problems captured in this comic.
I install all my Python modules into the system's Python using my OS's package manager. I have both Python 2 and Python 3 installed but beyond having the same module installed in each of those locations, I have never found the need for multiple versions of the same module installed at the same time.
On the rare occasion where my OS doesn't provide a package I need, I use pip install --user. If it turns out to be something I'll want for the long term, I just knock together a quick package that I can install/uninstall using standard OS packaging tools.
I install all my Python modules into the system's Python using my OS's package manager. I have both Python 2 and Python 3 installed but beyond having the same module installed in each of those locations, I have never found the need for multiple versions of the same module installed at the same time.
On the rare occasion where my OS doesn't provide a package I need, I use pip install --user. If it turns out to be something I'll want for the long term, I just knock together a quick package that I can install/uninstall using standard OS packaging tools.
>I install all my Python modules into the system's Python using my OS's package manager.
If that's the case, then it's as you said: you "just never do anything complicated with Python".
If that's the case, then it's as you said: you "just never do anything complicated with Python".
Most of the time the language package manager for a very popular language is an overkill. You can get all the packages you need for web development, AI or statistical/mathematical stuff in python from say debian packages and build arbitrarily complex applications. And you have the additional benefit of being able to easily package your application for many distributions that package stable versions of popular libraries for languages like python, perl, ruby, etc. Another additional benefit is that because most of the time you fetch packages precompiled, you don't need to run probably insecure compilations on your system (many popular packages for Python for example require you to compile C extensions). And the trade-off is quite palatable: you use the APIs hygenically, don't do things like leftpad, and sometimes forego some cutting edge features (and many dynamic languages enable you to easily backport them without modifying the package itself most of the time).
> You can get all the packages you need for web development, AI or statistical/mathematical stuff in python from say debian packages
Um... not quite that simple. If you are just developing for yourself, and you have full control over the deployment environment, then of course things are quite simple. (Assuming you don't need to track a fast-moving ecosystem, like machine learning, where packages update constantly and there is a huge rat's nest of interdependencies between OS, kernel, hardware drivers, and algorithmic libraries.)
Um... not quite that simple. If you are just developing for yourself, and you have full control over the deployment environment, then of course things are quite simple. (Assuming you don't need to track a fast-moving ecosystem, like machine learning, where packages update constantly and there is a huge rat's nest of interdependencies between OS, kernel, hardware drivers, and algorithmic libraries.)
Again, those do not concern complicated things with Python.
You might program very complicated algorithms in Python, but you don't anything complicated with them (shipping them to customers, supporting 100 servers, different distributions, and so on).
You might program very complicated algorithms in Python, but you don't anything complicated with them (shipping them to customers, supporting 100 servers, different distributions, and so on).
not really the case if you work in a team or deploy your application.
Unless you streamline the development and deployment environments, that is. You can just work in a VM, a chroot/jail, or probably use Docker (I haven't ever used it tho, thus not sure).
soner or later you will need to solve the same set of issues that virtualenv and pipenv try to solve.
Then I can reach for virtualenv and pip. I did not mean that package distribution and management in itself is an overkill, but that many types of projects just don't need these tools. And if one has to use these tools for all projects and from the get-go, then that means the library ecsosystem is unhygienic and/or the devs are not really responsible on how they use the APIs.
How would you define "complicated"? Does it go beyond having different versions of the same Python module installed?
It can go way beyond that. Different Python versions for legacy programs, Python servers and mass deployment via dev-ops, creating Python RPMs or Debs or other packages to ship to clients, and many other cases besides...
One of the reasons I try to use Docker for everything is because it's a cross-language solution. It does add some complexity and has its drawbacks, but for me, it's worth it. I'd rather have one global dependency (Docker) than several (virtualenv for Python, nvm for Node, rbenv for Ruby, etc.), which all have their own idiosyncrasies.
I also like using Docker because it handles non-language resources like databases. Even installing one version of Postgres locally was painful. It's hard to imagine trying to deal with multiple instances and versions on the same machine (without something like Docker or Vagrant) if two projects use 9.x, and another uses 10.x.
I also like using Docker because it handles non-language resources like databases. Even installing one version of Postgres locally was painful. It's hard to imagine trying to deal with multiple instances and versions on the same machine (without something like Docker or Vagrant) if two projects use 9.x, and another uses 10.x.
I use the stock Dockerhub python container with a requirements.txt. Similar to my Ruby code. It's a lot nicer than virtual envs for deployment.
It's still difficult to debug stuff inside your container (unless you do a docker exec and then you're limited to the tools you installed) so it's useful to have the same environment outside of it. `pip3 --user install` has been my friend; and you can create new users instead of full virtulenvs if you really need to.
Still, my python environment isn't as screwed up as this comic. System python is always only controlled by the OS package manager. My own stuff is installed with --user and things I need to deploy somewhere have a Dockerfile or I have a Jenkins workflow that create rpms/debs with fpm and push them to my repo and sign them.
It's still difficult to debug stuff inside your container (unless you do a docker exec and then you're limited to the tools you installed) so it's useful to have the same environment outside of it. `pip3 --user install` has been my friend; and you can create new users instead of full virtulenvs if you really need to.
Still, my python environment isn't as screwed up as this comic. System python is always only controlled by the OS package manager. My own stuff is installed with --user and things I need to deploy somewhere have a Dockerfile or I have a Jenkins workflow that create rpms/debs with fpm and push them to my repo and sign them.
Maybe this is obvious, but to informally install executables supplied by python packages, use a separate virtualenv for each one and place a symlink to the venv/bin/theexecutable in a location like ~/bin that you have on your $PATH.
pipsi is great for that, it takes care of creating the virtualenv, etc.
It's almost - but not quite - this bad when using a "simple" local development environment. As others have said it's not nearly this bad when you're deploying to sane infrastructure, be it containerized or not.
One more thing to add to the mess: PyBOMBS [1] A while back GNU Radio went and reinvented the whole package manager thing on their own. Since then I constantly have problems integrating anything GNU Radio-related with the rest of the Python ecosystem. Random Python 2 modules being imported by Python 3? You bet it's some random gr module in the PYTHONPATH somewhere.
[1] https://www.gnuradio.org/blog/pybombs-the-what-the-how-and-t...
[1] https://www.gnuradio.org/blog/pybombs-the-what-the-how-and-t...
It's funny because it's true
What's wrong with virtualenv/venv and pip?
* The package formats (source or wheel) do not sufficiently capture assumptions that devs make at release time, in order to produce a usable installable artifact at install time.
* Worse, when different package builders release binaries, there is insufficient information in the binaries metadata in order to know which sets of binaries will actually install AND run properly on any given system.
* Finally, the Python packaging ecosystem suffers from a "Tesla Autopilot" problem: some kinds of things are actually WORSE if they only work 90% of the time. So, many people get by just fine with pip and virtualenv... until they don't. Since package building and dependency-solving are not exactly the sexiest or most fun part of software development, most devs in the modern era don't necessarily take the time to understand the actual roots of the problem, but instead bat around lore and cut-and-paste stackoverflow until things seem to kind of work.
A major part of Python's current success is due to its numerical and data analysis libraries. These, in turn, are successful because they take advantage of deep capabilities in C, C++, and other "native" code. This means that Python packaging inherits the original sins of C (the dynamic linker) and of C++ (no standard binary ABI). Most other languages do not have to solve such a hard problem: Perl, Ruby, Node, etc. all don't go nearly as deep as Python does in terms of leveraging a rich ecosystem of native code extension modules.
Even Java avoids this and lives almost entirely within the JVM runtime - but even then, classpath conflicts show that dependency management in ANY language is a hard problem unless treated holistically and intentionally. Python got its package system bolted-on after the fact. Then several tools came and went as maintainers entered and exited the ecosystem. With Anaconda and conda, we are just now finally at a point where people can reliably install the basic scientific and numerical libraries, across hardware and OSes..... 20 years after I started using the language. :-)
* Worse, when different package builders release binaries, there is insufficient information in the binaries metadata in order to know which sets of binaries will actually install AND run properly on any given system.
* Finally, the Python packaging ecosystem suffers from a "Tesla Autopilot" problem: some kinds of things are actually WORSE if they only work 90% of the time. So, many people get by just fine with pip and virtualenv... until they don't. Since package building and dependency-solving are not exactly the sexiest or most fun part of software development, most devs in the modern era don't necessarily take the time to understand the actual roots of the problem, but instead bat around lore and cut-and-paste stackoverflow until things seem to kind of work.
A major part of Python's current success is due to its numerical and data analysis libraries. These, in turn, are successful because they take advantage of deep capabilities in C, C++, and other "native" code. This means that Python packaging inherits the original sins of C (the dynamic linker) and of C++ (no standard binary ABI). Most other languages do not have to solve such a hard problem: Perl, Ruby, Node, etc. all don't go nearly as deep as Python does in terms of leveraging a rich ecosystem of native code extension modules.
Even Java avoids this and lives almost entirely within the JVM runtime - but even then, classpath conflicts show that dependency management in ANY language is a hard problem unless treated holistically and intentionally. Python got its package system bolted-on after the fact. Then several tools came and went as maintainers entered and exited the ecosystem. With Anaconda and conda, we are just now finally at a point where people can reliably install the basic scientific and numerical libraries, across hardware and OSes..... 20 years after I started using the language. :-)
This is so true and so sad that it’s actually funny!
This issue is one of the things that drove me to utilize a dedicated VPS for each project. This keeps environments clean, ensures project/client isolation and allows me to utilize the exact resources needed, as opposed to worrying about the local machine.
Once done or if the prod environment changes, I can blow away the VPS and rebuild as needed.
This issue is one of the things that drove me to utilize a dedicated VPS for each project. This keeps environments clean, ensures project/client isolation and allows me to utilize the exact resources needed, as opposed to worrying about the local machine.
Once done or if the prod environment changes, I can blow away the VPS and rebuild as needed.
I'm using pyvenv even though it has started warning me I should move. I won't change from it for exactly the reasons in this comic.
When I do a new OS install I will try pipenv again...but it dind't work well for me the first time due to conflicting with something I had installed outside of a virtualenv!
When I do a new OS install I will try pipenv again...but it dind't work well for me the first time due to conflicting with something I had installed outside of a virtualenv!
No docker containers?
That would just make the chart recursive.
That's how I've solved this problem. Containers (i.e., clean-slate, defined media) give me something I can reason about and control.
The 'framework build and venv on osx' thing ruined my night yesterday so this post had great timing.
Seriously, starting from a clean osx, how do you install python? I always liked using vex for venv management. But I would also like to have anaconda for some tasks..
Seriously, starting from a clean osx, how do you install python? I always liked using vex for venv management. But I would also like to have anaconda for some tasks..
[deleted]
I love this comic.
I was just going through an old server of mine. Python 2 and Python 3 dependencies, installed over years, in a bit of a mess.
We may have virtualenv and the like today, but a lot of old stuff was never setup to use it. :p
I was just going through an old server of mine. Python 2 and Python 3 dependencies, installed over years, in a bit of a mess.
We may have virtualenv and the like today, but a lot of old stuff was never setup to use it. :p
Try installing pipsi such that it uses python3 to install simple scripts like httpie or youtube-dl and you'll see how big of a mess it really is.
I use pipenv and couldn't be happier. It's dead simple and it works. It is a bit slow. The problem presented in this xkcd has been fixed.
I think people are missing that he seems to be aware this is a poor way of doing things...
sudo pip3 install everything
Need add pypy, cython, etc.
In an earlier xkcd blag post, Randall talked about messing up his Linux box so badly that someone trying to help him fix it said "your computer looks like a redshirt after a transporter accident". That analogy has stuck with me ever since and I use it most often to describe Python installations.
LOL, haven't check if this XKCD was posted today, but it was particularly timely as I've spend the day debugging a Python + Jenkins + autotools + setuptools + RPM spec (mock etc) setup...
Anaconda. That's all you need to know.
This thread is an instance of https://xkcd.com/927/
[deleted]
[deleted]
All jokes are only funny when they are resembling to the truth but they are not when it's far from it. So this maybe funny five years ago. But it doesn't any more since Docker[1].
docker pull; docker run, you are golden. Even if you are going to customize the official image, it's much less cumbersome than that xkcd cartoon.
[1]: https://hub.docker.com/_/python/
docker pull; docker run, you are golden. Even if you are going to customize the official image, it's much less cumbersome than that xkcd cartoon.
[1]: https://hub.docker.com/_/python/
Sorry, can't run Docker on the Windows Server 2012 machines provided by the client. Or on the CentOS 6 machines provided by the other.
Not sure what your specific situation is, afaik it's possible. So your problem might not be pertaining to Docker Python.
https://www.liquidweb.com/kb/how-to-install-docker-on-centos...
https://www.liquidweb.com/kb/how-to-install-docker-on-centos...
Yes, but you'd have to run a 2.5+ years old - and unsupported - version of Docker (1.7.1).
What version of Python you'd be running? And Windows Server 2012 is 5 years old, no? If you are running a five years old OS, perhaps unsuported Docker version 1.7 (released 2015) might not be your only and most import concern right?
We are running the latest Python 2 release, compiled from source. Windows Server 2012 will be supported until 2023, and CentOS 6 until 2020.
Oh look, we found one more arrow to add to the chart!
You mean one more solution to add to the chart to solve that mess?
Every solution has drawbacks. What are some for the solution you propose?
I can assure you that this state of things is still pretty relevant to some of us.
Is this your comic, perhaps?
Nah, but I was helping the author clean up his Python environment on Friday, which I think prompted this comic. His environment isn't even the messiest I've seen. I helped an astronomer friend with theirs at one point, and it had something like six different "manage your whole Python workflow easily with just this tool!" things installed.
I think what a lot of comments here miss is that while new tools come out that can help make things better, they don't suddenly fix people's existing installs. It seems weird that there isn't more documentation out there to help people get out of these kinds of messes.
I think what a lot of comments here miss is that while new tools come out that can help make things better, they don't suddenly fix people's existing installs. It seems weird that there isn't more documentation out there to help people get out of these kinds of messes.
Exactly right. There is an irreversible entropy to this: once a certain number of implementations exist, it becomes almost impossible to create a "new standard" that fixes them all, because it has to be backwards compatible with too many of them.
As someone responsible for probably one of your astronomer friend's tools (Anaconda), I have to say that we can only do so much: the nature of Python itself is that we cannot ignore what the user puts into PYTHONPATH, PATH, etc. On Windows, at least, we do create shortcuts in the Start Menu to ensure that we get a well-defined command shell. But if the user forces things like PYTHONPATH, there's not much we can do to un-fsck the setup.
Maybe it would be useful to have a python-safemode (or python-portable) binary that explicitly ignores all environment variables, configuration files, registry settings, dotfiles, etc. etc. and only looks at files that are next to it in the path....
As someone responsible for probably one of your astronomer friend's tools (Anaconda), I have to say that we can only do so much: the nature of Python itself is that we cannot ignore what the user puts into PYTHONPATH, PATH, etc. On Windows, at least, we do create shortcuts in the Start Menu to ensure that we get a well-defined command shell. But if the user forces things like PYTHONPATH, there's not much we can do to un-fsck the setup.
Maybe it would be useful to have a python-safemode (or python-portable) binary that explicitly ignores all environment variables, configuration files, registry settings, dotfiles, etc. etc. and only looks at files that are next to it in the path....
We've been discussing this, R users have asked for the same. The argument is that Anaconda should give them isolation. I think it should be opt-in though, probably. Tricky thing is that to do it you need to muck about in the internals of CPython and R and that'll upset some other people.
No disagreement from me on the sorry state of things.
Pipenv is the most promising solution today but is still very, very new. It's modeled after yarn and has been officially blessed as "The One True Way" of installing stuff by the Python documentation. It has a way to go still to be as good as yarn (especially in terms of speed). The Python ecosystem has never had proper declarative packages like package.json (setup.cfg can be used to have fully declarative package metadata, but I seem to be the only one using it that way), which is a problem for package managers.
To those suggesting it, Docker is great but you're still dealing with a package manager inside Docker, so that's a moot point. It avoids the need for virtualenv, kind of, but so does pipenv and it does so more reliably and reproducibly (pipenv implements lockfiles).