A not so unfortunate sharp edge in Pipenv(0xda.de)
0xda.de
A not so unfortunate sharp edge in Pipenv
https://0xda.de/blog/2022/12/a-not-so-unfortunate-sharp-edge-in-pipenv/
72 comments
> I am sick and tired of the python packaging space pushing this godawful tool for Python. Please just use pip-tools or poetry instead.
As an outsider, Python packaging seems like a shit show, sorry. This bug in pipenv is bad, yes. But poetry was deliberately non-deterministically failing CI builds that used a deprecated feature.
Maybe pip-tools is reliable and has a reasonable maintainer, but having to manually create venvs is mediocre UX.
PDM sounds interesting because it doesn't need venvs under the hood, thanks to a recent PEP.
As an outsider, Python packaging seems like a shit show, sorry. This bug in pipenv is bad, yes. But poetry was deliberately non-deterministically failing CI builds that used a deprecated feature.
Maybe pip-tools is reliable and has a reasonable maintainer, but having to manually create venvs is mediocre UX.
PDM sounds interesting because it doesn't need venvs under the hood, thanks to a recent PEP.
> Python packaging seems like a shit show
It does, but primarily because most of the tools - and that includes pipenv, Poetry, pdm and hatch - have crammed up to three separate functionalities into one (probably following the example of npm): packaging on one hand, dependency management on the other, and virtual environments on the third.
The packaging question has mostly been resolved: distribute your packages as wheels and install them using pip; that's all. Dependency managament inherently has more complexity, and adding virtual environments in the mix, especially with the introduction of PEP 582 (as you mention), doesn't help things.
It does, but primarily because most of the tools - and that includes pipenv, Poetry, pdm and hatch - have crammed up to three separate functionalities into one (probably following the example of npm): packaging on one hand, dependency management on the other, and virtual environments on the third.
The packaging question has mostly been resolved: distribute your packages as wheels and install them using pip; that's all. Dependency managament inherently has more complexity, and adding virtual environments in the mix, especially with the introduction of PEP 582 (as you mention), doesn't help things.
> It does, but primarily because most of the tools - and that includes pipenv, Poetry, pdm and hatch - have crammed up to three separate functionalities into one (probably following the example of npm): packaging on one hand, dependency management on the other, and virtual environments on the third.
Many build tools in other languages also do all of these three things - for example Ruby's Bundler or maven/gradle for JVM languages. And they're pretty reliable at all of these tasks.
(I guess, "virtual environments" is maybe not the right description. What these tools all do is isolating dependencies of different projects from each other, although they might do it in slightly different ways.)
Many build tools in other languages also do all of these three things - for example Ruby's Bundler or maven/gradle for JVM languages. And they're pretty reliable at all of these tasks.
(I guess, "virtual environments" is maybe not the right description. What these tools all do is isolating dependencies of different projects from each other, although they might do it in slightly different ways.)
>primarily because most of the tools - and that includes pipenv, Poetry, pdm and hatch - have crammed up to three separate functionalities into one (probably following the example of npm): packaging on one hand, dependency management on the other, and virtual environments on the third.
That never stopped npm to be way better than all of them...
>The packaging question has mostly been resolved: distribute your packages as wheels and install them using pip; that's all. Dependency managament inherently has more complexity, and adding virtual environments in the mix, especially with the introduction of PEP 582 (as you mention), doesn't help things.
So, the "solution" is "just give up"...
That never stopped npm to be way better than all of them...
>The packaging question has mostly been resolved: distribute your packages as wheels and install them using pip; that's all. Dependency managament inherently has more complexity, and adding virtual environments in the mix, especially with the introduction of PEP 582 (as you mention), doesn't help things.
So, the "solution" is "just give up"...
Npm doesn't do environments. You have the choice of "in this directory" or "globally" and pray there are no symlinks involved or things fail mysteriously. It doesn't quite do packaging either.
So it lands in a similar space to what the parent describes - mostly a good dependency manager.
So it lands in a similar space to what the parent describes - mostly a good dependency manager.
>Npm doesn't do environments.
"This directory" (which can be overriden with env setting or similar advise for where to lookup) works fine for me, and never have been a problem in quite complex setups. Not sure why "symlinks" would be involved.
In any case, what NPM does is leaps and bounds ahead of any shit Python offering we have to deal with, which is a constant source of trouble.
IIRC, it's also the model ("this directory") that there's a PEP to imitatate, not sure what happened to that...
"This directory" (which can be overriden with env setting or similar advise for where to lookup) works fine for me, and never have been a problem in quite complex setups. Not sure why "symlinks" would be involved.
In any case, what NPM does is leaps and bounds ahead of any shit Python offering we have to deal with, which is a constant source of trouble.
IIRC, it's also the model ("this directory") that there's a PEP to imitatate, not sure what happened to that...
> which can be overriden with env setting
Support for this is not great. Lots of packages still don't support this properly. My experience matches the 2015 comment https://github.com/npm/npm/issues/775#issuecomment-71294085
> Not sure why "symlinks" would be involved.
If you make your node_modules a symlink, multiple packages will fail. Even if you're not interested in doing that, others are. I tried the be symlinks because the env var option is not reliable.
> What NPM does is leaps and bounds ahead
Unless you change your node / gyp version. It doesn't really have a concept of runtime version. You can restrict it, but not have two concurrent versions if they conflict.
Support for this is not great. Lots of packages still don't support this properly. My experience matches the 2015 comment https://github.com/npm/npm/issues/775#issuecomment-71294085
> Not sure why "symlinks" would be involved.
If you make your node_modules a symlink, multiple packages will fail. Even if you're not interested in doing that, others are. I tried the be symlinks because the env var option is not reliable.
> What NPM does is leaps and bounds ahead
Unless you change your node / gyp version. It doesn't really have a concept of runtime version. You can restrict it, but not have two concurrent versions if they conflict.
Back in the days we had the .eggs directory in python it worked fairly well except for binary packages so we got virtual environments.
What was the problem with binary packages for .eggs?
Because, iirc, npm does handle e.g. .so files in packages that are based on C/C++ code, and puts them and everything else in the same, relocatable even (another pain point with virtualenvs), .node_modules directory.
Because, iirc, npm does handle e.g. .so files in packages that are based on C/C++ code, and puts them and everything else in the same, relocatable even (another pain point with virtualenvs), .node_modules directory.
You meant to write „I am more used to npm and intuitively work around the quieks while in python i run into these“
No, I meant to write "I've worked with Python for 20+ years, and Node for 8, and Python has always been a shit show on that front, whereas npm haven't given me any trouble (1)".
But thanks for the uncharitable strawman. Couldn't be that I know both and don't like it, must be that I'm not familiar enough with the wonderland that is the Python package management...
(1) At least not any trouble of that kind. I've had some issues regarding codependencies for add-ons, but that's a problem far beyond the kind of problems the Python package management has. If only Python's packaging stuff only had the same kind of issues...
But thanks for the uncharitable strawman. Couldn't be that I know both and don't like it, must be that I'm not familiar enough with the wonderland that is the Python package management...
(1) At least not any trouble of that kind. I've had some issues regarding codependencies for add-ons, but that's a problem far beyond the kind of problems the Python package management has. If only Python's packaging stuff only had the same kind of issues...
What makes NPM a cut above Python package managers is that I can clone a JavaScript project, run npm install, and it will just work in such a way that it will not conflict with other projects.
With python I have to work out which package manager it's using (the JavaScript ecosystem also has multiple package managers - yarn and pnpm - but they're compatible with each other!), and there's a significant chance it's using pip which installs packages globally unless I manually create and manage a virtualenv.
The gold standard is probably Rust's "cargo" (and "rustup" for toolchain version management), which broadly follows the NPM model but with a better implementation with fewer quirks. But NPM has gotten much better (it uses lock files by default now for example), and is mostly painless at this point.
With python I have to work out which package manager it's using (the JavaScript ecosystem also has multiple package managers - yarn and pnpm - but they're compatible with each other!), and there's a significant chance it's using pip which installs packages globally unless I manually create and manage a virtualenv.
The gold standard is probably Rust's "cargo" (and "rustup" for toolchain version management), which broadly follows the NPM model but with a better implementation with fewer quirks. But NPM has gotten much better (it uses lock files by default now for example), and is mostly painless at this point.
> But NPM has gotten much better (it uses lock files by default now for example)
Last time I checked, "npm install" will still overwrite your lock file, you have to explicitly use "npm ci" for that not to happen.
Or maybe that changed recently?
Last time I checked, "npm install" will still overwrite your lock file, you have to explicitly use "npm ci" for that not to happen.
Or maybe that changed recently?
It depends on the context.
* If the package.json file can be satisfied by the lock file (i.e. if the two files match) then the lock file will be used unchanged, even if there are newer versions available. This is (mostly) identical to the `npm ci` command.
* If the package.json requires more recent versions of a particular package than are available in the lock file then this package will be updated (but other packages will remain the same) and a new lock file will be written. This happens if, for example, the package.json file has been changed by hand, and so the lock file is no longer in date. In contrast, the `npm ci` command exits and fails to install the dependencies.
* Similarly, if the package.json file has new dependencies that aren't in the lock file, it if dependencies exist in the lock file that are no longer present in package.json, then these packages will be installed or deleted and the lock file updated. Theoretically, only the relevant dependency and its dependencies will be affected, but because if the deduping algorithm, NPM may make other changes if it thinks it can simplify the dependency tree. Again, in this situation, `npm ci` will just fail.
There is one other minor exception. The `npm ci` command will never make any changes to the lock file, but `npm install` may still choose to update the lock file with additional metadata (for example if the lock file was created with an older version of npm). This will never change the versions, but it may look a bit messy in the commit history, particularly if different people on a team are using different versions of npm. As a result, I tend to use npm ci as my default anyway.
However, as a role of thumb, if `npm ci` is able to successfully install dependencies, then running `npm install` will always install the same dependencies. The only difference is when the package.json file disagrees with the lock file, in which case `npm install` will probably do what you want, unless you want some very specific results in which case you still have the npm ci command available.
(This has been true since at least npm v5, although I think there were some bugs in the initial implementation.)
* If the package.json file can be satisfied by the lock file (i.e. if the two files match) then the lock file will be used unchanged, even if there are newer versions available. This is (mostly) identical to the `npm ci` command.
* If the package.json requires more recent versions of a particular package than are available in the lock file then this package will be updated (but other packages will remain the same) and a new lock file will be written. This happens if, for example, the package.json file has been changed by hand, and so the lock file is no longer in date. In contrast, the `npm ci` command exits and fails to install the dependencies.
* Similarly, if the package.json file has new dependencies that aren't in the lock file, it if dependencies exist in the lock file that are no longer present in package.json, then these packages will be installed or deleted and the lock file updated. Theoretically, only the relevant dependency and its dependencies will be affected, but because if the deduping algorithm, NPM may make other changes if it thinks it can simplify the dependency tree. Again, in this situation, `npm ci` will just fail.
There is one other minor exception. The `npm ci` command will never make any changes to the lock file, but `npm install` may still choose to update the lock file with additional metadata (for example if the lock file was created with an older version of npm). This will never change the versions, but it may look a bit messy in the commit history, particularly if different people on a team are using different versions of npm. As a result, I tend to use npm ci as my default anyway.
However, as a role of thumb, if `npm ci` is able to successfully install dependencies, then running `npm install` will always install the same dependencies. The only difference is when the package.json file disagrees with the lock file, in which case `npm install` will probably do what you want, unless you want some very specific results in which case you still have the npm ci command available.
(This has been true since at least npm v5, although I think there were some bugs in the initial implementation.)
Looks like that's still the case. It does still use lockfiles, it just updates them by default. There is also as you mention a built-in command that does the right thing.
Right, it's just one of these things that inexperienced people (or people coming from other languages) would get wrong, so no idea why they have this behaviour.
> That never stopped npm to be way better than all of them...
npm isn't my favourite, but it's definitely improved with the competition from yarn and pnpm.
I think that Stack (Haskell) and Cargo (Rust) are state of the art, for ease of use and reproducibility. The other user would describe these as cramming three functionalities into one, but they do it so well.
npm isn't my favourite, but it's definitely improved with the competition from yarn and pnpm.
I think that Stack (Haskell) and Cargo (Rust) are state of the art, for ease of use and reproducibility. The other user would describe these as cramming three functionalities into one, but they do it so well.
Stack is good, but the Haskell community seems split between Stack and pure Cabal users... plus there's also ghcup and it's not easy to understand whether one should have stack manage GHC versions or no... every source tells you something different.
I also hate that there's no good way to install a depedency globally so you can play around with it (you kinda can, but it's very hard to remove it afterwards).
But the real shitshow is the HLS that just crashes on me all the time and doesn't always work correctly with Stack.
I also hate that there's no good way to install a depedency globally so you can play around with it (you kinda can, but it's very hard to remove it afterwards).
But the real shitshow is the HLS that just crashes on me all the time and doesn't always work correctly with Stack.
what? npm is, for a lack of better word, shit. not worse than Python situation (coming from somebody who was excited when new-style classes were introduced), but that is a very low bar. ($DAYJOB makes me use npm _and_ pnpm, nx, nvm, corepack and I guess some other tools I don't know about just to make the goddamn thing build.)
Any argument why is it shit?
it's so slow there's a whole ecosystem to avoid using it, edited my comment above.
I think it's not npm/yarn that is slow, it's that Node project have tons of dependencies and transitive dependencies and such (leftpad, anyone?)
But that's more a fault of the library ecosystem and culture, than the design of npm/yarn
But that's more a fault of the library ecosystem and culture, than the design of npm/yarn
Alas, wheels are terrible for providing anything but a Python API. If you have headers, compiled tools, or some other complicated SDK, good luck (NumPy is excepted here because it only ships headers this way and chooses to live with wheel limitations). Wheels depending on you also ship copies of your library (if you use PyPI), so the lackluster dependency management is just waiting to mix things up and give your users "fun" linker/runtime issues.
Package management in Python is mostly solved to me with conda+conda devenv+conda lock.
Sometimes I still need to get packages from pip in that scenario, but conda makes package management a lot better than pip+alternatives for me.
Sometimes I still need to get packages from pip in that scenario, but conda makes package management a lot better than pip+alternatives for me.
> But poetry was deliberately non-deterministically failing CI builds that used a deprecated feature.
I’m having difficulty finding a reference on this, could you elaborate?
I’m having difficulty finding a reference on this, could you elaborate?
Interesting that you call out venv creation as a UX problem. I never considered it any worse than having to `cargo new` a project. I find having to activate and deactivate vs trust a 3rd party project that watches my every `cd` a much bigger issue.
PDM in my experience has been the best, plus it's rapidly growing, and is more PEP compliant than poetry. My go-to choice, been advocating it since I found out about it!
I have used python for 15 years but less frequently nowadays because engineering manager. What irks me out about poetry is that it tries to be too magic and in the end just is oddly surprising. Will pdm feel „more solid“ to me?
There is no good alternative. Maybe poetry is the least worst, but it is a fractal trash fire like the rest of the python ecosystem.
To name some issues i've had with poetry(and keep having):
- Updating packages does not properly take effect sometimes. It will sometimes keep using old code, and only deleting the entire underlying venv fixed it. In this instance, I was using a wheel package locally. - It has incredibly poor UX and will just vomit raw python errors when you use it slightly incorrectly. - There's so many bugs in its handling of CLI args and getting a typical python None error is very common - It is ridiculously slow and doesn't cache things properly
I could probably go on but I'd have to sit and think a bit about it, and that's pretty much the opposite of what I want to do: forget all about python and poetry.
To name some issues i've had with poetry(and keep having):
- Updating packages does not properly take effect sometimes. It will sometimes keep using old code, and only deleting the entire underlying venv fixed it. In this instance, I was using a wheel package locally. - It has incredibly poor UX and will just vomit raw python errors when you use it slightly incorrectly. - There's so many bugs in its handling of CLI args and getting a typical python None error is very common - It is ridiculously slow and doesn't cache things properly
I could probably go on but I'd have to sit and think a bit about it, and that's pretty much the opposite of what I want to do: forget all about python and poetry.
> pipenv's dependency resolution is the slowest among all the tools in this space including poetry, pip-tools, hatch and so on
"Slow" doesn't just mean "slightly annoying", mind you.
I was once tasked to properly package up an Airflow installation. Not being a Python guy, I tried doing it with pipenv because we were using it for other projects.
It was impossible to resolve the dependencies because the resolver would routinely throw a timeout error. In the end I gave up and switched to Poetry. That was still a bit slow (I think Airflow just has a messed up dependency graph...), but it wrote the lockfile in a reasonable amount of time and just worked thereafter...
"Slow" doesn't just mean "slightly annoying", mind you.
I was once tasked to properly package up an Airflow installation. Not being a Python guy, I tried doing it with pipenv because we were using it for other projects.
It was impossible to resolve the dependencies because the resolver would routinely throw a timeout error. In the end I gave up and switched to Poetry. That was still a bit slow (I think Airflow just has a messed up dependency graph...), but it wrote the lockfile in a reasonable amount of time and just worked thereafter...
Personally i've never really seen the advantages any of these tools over having a requirements.txt, or slightly fancier and more modern, using a pyproject.toml file.
e.g. in a recent project the entire package configuration is this pyproject.toml file:
e.g. in a recent project the entire package configuration is this pyproject.toml file:
[project]
name = "vera"
version = "0.0.0"
requires-python = ">=3.10"
authors = [
{ name = "Doxin" }
]
dependencies = ["moderngl==5.7.3", "pysdl2==0.9.14", etc etc]
[project.optional-dependencies]
dev = ["pre-commit==2.20.0"]
[tool.setuptools]
packages = ["vera"]
python3.10 -m venv venv
source venv/bin/activate
pip install -e vera[dev]
Not to mention the horrible configuration anti-pattern where everything depends on environment variables.
Environment variable tweaks are for local-only specific fixes, not for reproducible/distributable configuration.
Environment variable tweaks are for local-only specific fixes, not for reproducible/distributable configuration.
This is partially addressed by .env files (many projects support them) and an accompanying file with empty values or defaults, that lists and documents the variables (the .env file itself is then in .gitignore)
> not for reproducible/distributable configuration
At my work, we use environment variables all the time for configurations, e.g. for telling a kubernetes pod where to find the database. In your opinion, what is the better option(s)?
At my work, we use environment variables all the time for configurations, e.g. for telling a kubernetes pod where to find the database. In your opinion, what is the better option(s)?
The problem with environment variables as configuration is that it's unstructured, hardly documented, and overall hard to reproduce and inspect.
Nothing is worse than trying to understand an issue with a program that heavily relies on environment variables for configuration, as environment variables are designed to be short-lived, memory only.
A good old configuration file is the best. You can version it, distribute it, it's explicit, possibly structured, easily documented.
That doesn't mean that there is no room for environment variables, but these should be for local-only hacks and tweaks.
Nothing is worse than trying to understand an issue with a program that heavily relies on environment variables for configuration, as environment variables are designed to be short-lived, memory only.
A good old configuration file is the best. You can version it, distribute it, it's explicit, possibly structured, easily documented.
That doesn't mean that there is no room for environment variables, but these should be for local-only hacks and tweaks.
> Nothing is worse than trying to understand an issue with a program that heavily relies on environment variables for configuration
I totally agree with this.
> A good old configuration file is the best.
But the issue with this is that you often don't know what your configuration is ahead of time. Sometimes it is only generated just before execution of your code.
I totally agree with this.
> A good old configuration file is the best.
But the issue with this is that you often don't know what your configuration is ahead of time. Sometimes it is only generated just before execution of your code.
Secret managers. All cloud providers have them built in, and on other platforms you can easily whip up a self-hosted one. See https://blog.diogomonica.com//2017/03/27/why-you-shouldnt-us...
I recommend against with using secrets managers as opposed to config files for storing configuration. Ideally, your configuration is versioned and code-reviewed. Config files are optimal. Even secrets used by your deployment should be stored in configuration - encrypted, of course. The benefit of versioned configuration files, secrets included, cannot be overstated.
You can encrypt sensitive configuration fields with tools like `mozilla/sops` [1], which will reach out to your KMS or secrets manager of choice to encrypt/decrypt sensitive fields on the fly.
This way, you are minimizing the splitting of state across secrets managers and your code. Your configuration is stored at code, code reviewed, and versioned, which has devops and security benefits of its own.
[1]: https://github.com/mozilla/sops
You can encrypt sensitive configuration fields with tools like `mozilla/sops` [1], which will reach out to your KMS or secrets manager of choice to encrypt/decrypt sensitive fields on the fly.
This way, you are minimizing the splitting of state across secrets managers and your code. Your configuration is stored at code, code reviewed, and versioned, which has devops and security benefits of its own.
[1]: https://github.com/mozilla/sops
Config files, and command line arguments as a very distant second
Every solution in the space is bad and it's a wonder that any python software is maintainable and I think the only real answer is everyone gets fucked all the time with these bugs and rough edges
I say this as someone who has done an unfortunately large amount of work in the python packaging and python package distribution space and still get bitten every few months
I say this as someone who has done an unfortunately large amount of work in the python packaging and python package distribution space and still get bitten every few months
Plus pipenv falls down the stairs whenever I have tried to use it with packages that rely on native extensions like NumPy and GDAL. Not sure about pip-tools or poetry. My path of least resistance has been to use a local package manager (MacPorts, Homebrew before that), and then use Conda in CICD to test compatibility with different versions of Python.
> exactly like how `npm install` updates the package-lock.json and requires additional flags to use the lockfile instead
`npm install` doesn't do that anymore since at least a year
`npm install` doesn't do that anymore since at least a year
Pipenv was artificially elevated to its position of “popularity” by its maintainer who was able to use various connections to make it seem more official than it was. I think this has done a lot of harm to Python packaging over the years. Packaging wasn’t in a great place, but the confusion, and Pipenv’s promises for newcomers really threw everything off course for a generation of new Python engineers.
I think this is part of it, it did not help that during the process it got abandoned and that poetry filled the void. I liked pipenv from its user interface it just never got from the PoC stage to „polished“. The way forward would actually be to unify the dependency resolution mechanisms into a PIP implementation (iirc pip has drastically improved its dependency resolution in recent years) and converge the various projects.
I use a small shell script I wrote in jest a year ago (https://github.com/senko/pvp), it boils down to virtualenv and pip, and it basically solves all Python package management issues for me.
I was burned by pipenv before (naively trying to use it because a certain prominent member of Python community hyped it up, and I hadn't known he'd gone off the rails). I can find no redeeming qualities to it whatsoever, and it scarred me enough that I don't want to touch other newer tools (poetry, pdm, whatever) with a 10ft pole.
I find solace in pip and virtualenv just working and not trying to be too clever.
I pin all my immediate reqs, and if there's a conflict with indirect deps (which happened maybe once or twice), I figure out the version I need and pin that manually.
For packaging I rely on specialized tools that do just that (build, twine), and don't need one-with-everything spaceship of a tool.
I also develop in Node and have suffered much more grief by npm. I lost count of the number of times where I had to rm-rf node modules and npm cache, rerun and hope for the best.
I was burned by pipenv before (naively trying to use it because a certain prominent member of Python community hyped it up, and I hadn't known he'd gone off the rails). I can find no redeeming qualities to it whatsoever, and it scarred me enough that I don't want to touch other newer tools (poetry, pdm, whatever) with a 10ft pole.
I find solace in pip and virtualenv just working and not trying to be too clever.
I pin all my immediate reqs, and if there's a conflict with indirect deps (which happened maybe once or twice), I figure out the version I need and pin that manually.
For packaging I rely on specialized tools that do just that (build, twine), and don't need one-with-everything spaceship of a tool.
I also develop in Node and have suffered much more grief by npm. I lost count of the number of times where I had to rm-rf node modules and npm cache, rerun and hope for the best.
I wholeheartedly recommend PDM as a modern python package management tool: https://pdm.fming.dev/latest/
* greater PEP compliance with respect to poetry (and no upper boundary constraints on packages)
* support for scripting
* parallel installation
Here is the opinion of somebody way more qualified than me:
- https://iscinumpy.dev/post/poetry-versions/
- https://iscinumpy.dev/post/bound-version-constraints/
* greater PEP compliance with respect to poetry (and no upper boundary constraints on packages)
* support for scripting
* parallel installation
Here is the opinion of somebody way more qualified than me:
- https://iscinumpy.dev/post/poetry-versions/
- https://iscinumpy.dev/post/bound-version-constraints/
I've been experimenting with a new requirements.txt and requirements.txt.lock route.
Pip allows your requirements file to omit versions. That means: you can manually add your root requirements into your requirements.txt. Then you can
pip install -U -r requirements.txt
and it will update your dependencies. If anything breaks, you can uninstall it. If it all works out, you can lock again.
This way I can stay within pip tools and speed up package installs. One negative is having to manually fix dependency conflicts. But that is a price I'd be willing to pay for ease if development and deployment.
Edit: Also note that the article makes an excellent point concerning supply chain attacks which pipenv would have successfully mitigated in this particular case.
Pip allows your requirements file to omit versions. That means: you can manually add your root requirements into your requirements.txt. Then you can
pip install -r requirements.txt
pip freeze > rewuirements.txt.lock
pip install -U -r requirements.txt
and it will update your dependencies. If anything breaks, you can uninstall it. If it all works out, you can lock again.
This way I can stay within pip tools and speed up package installs. One negative is having to manually fix dependency conflicts. But that is a price I'd be willing to pay for ease if development and deployment.
Edit: Also note that the article makes an excellent point concerning supply chain attacks which pipenv would have successfully mitigated in this particular case.
Pretty much all stuff that tools like pipenv do, is something that anyone proficient with pip can do on their own. Their "advantage" lies in standardizing, so people don't have to find out if you called the lockfile "requirements.txt.lock", "requirement.lock", or something else.
Unfortunately they create another learning curve, another leaky abstraction layer, another point of failure, etc etc...
Unfortunately they create another learning curve, another leaky abstraction layer, another point of failure, etc etc...
To add to that, the suspicious steps are:
manually adding root dependencies. What if you forget?
Not locking. What if you forget?
Locking invalid versions. What if you don't check all is compatible first?
How do you handle dev requirements? requirements_dev? but then how do you lock?
The good thing is: you got rid of dependency walking and slow locking. As with everything, there is a tradeoff that I was willing to take.
manually adding root dependencies. What if you forget?
Not locking. What if you forget?
Locking invalid versions. What if you don't check all is compatible first?
How do you handle dev requirements? requirements_dev? but then how do you lock?
The good thing is: you got rid of dependency walking and slow locking. As with everything, there is a tradeoff that I was willing to take.
Check out pip-tools [1] which does exactly that, albeit in a slightly more polished way.
[1]: https://github.com/jazzband/pip-tools
[1]: https://github.com/jazzband/pip-tools
Just FYI this doesn't actually pin hashes, unlike pipenv and pip-tools. If this functionality is important to you I suggest you use the latter.
[deleted]
> and it will update your dependencies.
I'm pretty sure you need an extra step, since pip freeze will put "==" constraints in the lock file. I do the same thing, but I have this extra step to replace "==" with ">="
I'm pretty sure you need an extra step, since pip freeze will put "==" constraints in the lock file. I do the same thing, but I have this extra step to replace "==" with ">="
sed -i "s/==/>=/g" requirements.txt.lockThere is a misunderstanding.
This is why I don't use the lock file to update dependencies in the first place. You have to use the requirements.txt for updates. The lock is for deploying, or initializing to last stable revision.
requirements.txt also ensures that you only install what you need.
This is why I don't use the lock file to update dependencies in the first place. You have to use the requirements.txt for updates. The lock is for deploying, or initializing to last stable revision.
requirements.txt also ensures that you only install what you need.
But pip -U will only update first level dependencies (the ones directly listed in requirements.txt). It will not update secondary dependencies.
This is why every 3 months or so I completely delete the venv and recreate it.
This is why every 3 months or so I completely delete the venv and recreate it.
I see, yes unfortunately even if we lock the requirements, people just yank versions or update them in place. It all feels a little bit finicky pardon the fun.
Isn't the purpose of the lock file to lock precise versions for max reproducibility?
There is an issue from 2019 on the pipenv tracker for this issue: https://github.com/pypa/pipenv/issues/3893
According to the last comment there it should be fixed:
> I don't think this should be an issue in the current 2022.4.8 release because of the requirement to use index restricted packages for locking. Please advise if you think this is still a concern and we can revisit it.
But apparently it is still an issue (assuming the author did use a recent enough pipenv version)
From the article:
> Overall this is pipenv working exactly as it should.
I don't agree, IMO pipenv should just install from the source package and ignore the newer wheel, maybe showing a warning about the new wheel.
Edit: Just noticed that the author agrees in the end:
> I suppose the behavior I’d like to see here is that if my Pipfile.lock has hashes in it for a distribution, then even if additional distributions are available, pipenv sync should be allowed to continue to install from the distribution corresponding to the “trusted” hash.
I prefer to use https://python-poetry.org/ instead of pipenv, since it has less issues in my experience. But I wonder how it would behave in this situation.
According to the last comment there it should be fixed:
> I don't think this should be an issue in the current 2022.4.8 release because of the requirement to use index restricted packages for locking. Please advise if you think this is still a concern and we can revisit it.
But apparently it is still an issue (assuming the author did use a recent enough pipenv version)
From the article:
> Overall this is pipenv working exactly as it should.
I don't agree, IMO pipenv should just install from the source package and ignore the newer wheel, maybe showing a warning about the new wheel.
Edit: Just noticed that the author agrees in the end:
> I suppose the behavior I’d like to see here is that if my Pipfile.lock has hashes in it for a distribution, then even if additional distributions are available, pipenv sync should be allowed to continue to install from the distribution corresponding to the “trusted” hash.
I prefer to use https://python-poetry.org/ instead of pipenv, since it has less issues in my experience. But I wonder how it would behave in this situation.
Mandatory read whenever the topic of pipenv comes back: https://chriswarrick.com/blog/2018/07/17/pipenv-promises-a-l...
Orthogonal to the Python-specific issues, the problem of "All of my past commits in version control are now broken" is an interesting one.
My first instinct would be to rebase the entire project tree onto a new commit that follows the original addition of the dependency. This isn't a great solution, mostly because git doesn't make such an operation easy, but also because there's something a little spooky about a mass history rewrite like that.
My first instinct would be to rebase the entire project tree onto a new commit that follows the original addition of the dependency. This isn't a great solution, mostly because git doesn't make such an operation easy, but also because there's something a little spooky about a mass history rewrite like that.
What do I use?
- Pipenv?
- Hatch?
- Bawl in a corner?
Thanks in advance.
- Pipenv?
- Nice deterministic builds
- Dogslow to lock
- Has some quirky defaults (pipenv install also updates other packages)
- Pretty
- 10x locking speed
- Heard (in several places) problems about it failing to deterministically resolve a dep tree in some edge cases, which makes me a little scared (am I going to hit the dep jackpot at some point?)
- Hatch?
- Bawl in a corner?
Thanks in advance.
as other commenters have stated, pip-tools is a decent alternative. It does have some issues around platform dependent packages (like libuv and windows), but that's solved by managing a different compiled requirements per environment.
If you or anyone else enjoying pip-tools is a Zsh user and interested in trying out my higher level functions to ease interactive use of pip-tools, venvs, and also isolated app installs (like pipx), I would love some feedback on zpy: https://github.com/AndydeCleyre/zpy
I'm very happy to answer any questions about it right here or as GitHub issues.
I'm very happy to answer any questions about it right here or as GitHub issues.
well, another example that pythons dependency management is seriously broken. Why does the lock file not contain information about whether it was wheel or not?
It's not broken, it's incomplete. For example, there is no universally accepted lock file standard, so every tool whips up its own.
Sidenote: this website performs poorly on Firefox for Android; every time the background updates, the page freezes for a second, so scrolling is jumpy
- pipenv install updates your lockfile on install exactly like how `npm install` updates the package-lock.json and requires additional flags to use the lockfile instead
- pipenv's dependency resolution is the slowest among all the tools in this space including poetry, pip-tools, hatch and so on
- pipenv has consistently failed to do its job to simply install dependencies because of issues like the one described in the article where it fails to resolve changes in hash simply because a new wheel was added for the platform
- Tons of longstanding bugs that fail to be resolved in any sort of timely manner like this one: https://github.com/pypa/pipenv/issues/2413
I am sick and tired of the python packaging space pushing this godawful tool for Python. Please just use pip-tools or poetry instead.