Changes to Homebrew adversely affect Python development(justinmayer.com)
justinmayer.com
Changes to Homebrew adversely affect Python development
https://justinmayer.com/posts/homebrew-python-is-not-for-you/
108 comments
Use Homebrew to install pyenv and pyenv-virtualenv. Then use those to install pythons and collections of packages. You'll have stable development environments.
Feels strange doing this, but on M1 I am using the system Python 3, and installing packages as —user. This will cause me trouble sometime.
Not on M1 Apple Silicon. So many issues with pyenv on the M1. Brew python works fine natively.
Many of the problems people appear to be having with pyenv on M1 are related to it pulling include paths for homebrew that include homebrew libffi, but not passing the library path. For cases like cffi this results in header/library mismatch and you get runtime dynamic linker errors.
Other replies here appear to assume basic compilation issues, but the issues I've seen on various GitHub projects are mostly related to these more insidious C toolchain challenges. They're not M1-specific, but they are manifesting on M1 machines.
Other replies here appear to assume basic compilation issues, but the issues I've seen on various GitHub projects are mostly related to these more insidious C toolchain challenges. They're not M1-specific, but they are manifesting on M1 machines.
Make sure that Xcode is installed, and available, then these instructions work a treat:
https://github.com/pyenv/pyenv/issues/1643#issuecomment-7293...
For installing Python on the M1 Apple Silicon. This will install x86_64 versions, but those will work just fine.
https://github.com/pyenv/pyenv/issues/1643#issuecomment-7293...
For installing Python on the M1 Apple Silicon. This will install x86_64 versions, but those will work just fine.
I have this setup running on M1 and Intel. I'm not saying you are having no issues, but I don't think that's the reason.
If you're having trouble installing something with brew on M1, run arch x86_64 brew instead of brew to do a Rosetta installation.
If you're having trouble installing something with brew on M1, run arch x86_64 brew instead of brew to do a Rosetta installation.
What? I did "pyenv install 3.9.1" on M1 and it worked absolutely fine, it built a arm64 build and everything. Maybe it's because I had Xcode installed?
Ah, I was wondering why none of these things have happened to me. That said, the cascading upgrades does feel like a foot gun in search of feet.
`brew install openssl` or `brew upgrade openssl` will still break pyenv virtualenvs, though they can then be fixed with `pyenv rehash`. Maybe this makes sense? I didn't sort through why it was happening.
I prefer pyenv-virtualenvwrapper but yes
Still, Python 3.6.x installation is broken on Big Sur (non M1)
My current Mac is going to be my last. Linux on Thinkpad is next
My current Mac is going to be my last. Linux on Thinkpad is next
I am far from an expert but this is what fixed Vscode's Python extension and enabled a "stable development environment"
Ran into this myself. Starting using `pyenv` instead. Never looked back. Allows you to maintain multiple versions of python outside brew's control and the system's.
https://github.com/pyenv/pyenv
What's nice is you can find analog programs for a lot of different languages and platforms including
https://github.com/pyenv/pyenv
What's nice is you can find analog programs for a lot of different languages and platforms including
Node: https://github.com/ekalinin/nodeenv
Ruby: https://github.com/rbenv/rbenv
Terraform: https://github.com/tfutils/tfenvThe asdf Python plugin uses python-build, the same back-end that pyenv uses. So the tools are very similar. I prefer asdf to the collection of programs you mentioned because with asdf you get a unified CLI syntax for adding/updating plugins and installing/removing versions. I like being able to use the same toolchain and UI to manage different versions of Python, Node.js, Ruby, etc. Check out the list of asdf plugins [1] — it's quite impressive.
[1] https://asdf-vm.com/#/plugins-all?id=plugin-list
[1] https://asdf-vm.com/#/plugins-all?id=plugin-list
Not trying to spam, but have you looked at `anyenv`? It installs and updates all of these installers (and more), and keeps plugins up to date. I've been very happy with it.
Oh noes, I will soon need to use tfenv too!?!
Article author here. I think my biggest complaint is that there’s no disclaimer on the box [1]. Homebrew folks know about these problems, but they don’t warn users that their virtual environments will be obliterated. If Homebrew explained that these problems can happen, and that developers might want to use a different source for installing Python interpreters, my ire would be lessened considerably.
[1] https://docs.brew.sh/Homebrew-and-Python.html
[1] https://docs.brew.sh/Homebrew-and-Python.html
I don't think the Homebrew team cares much about UX. I still don't appreciate how pretty much all build options were deprecated.
We actually care a great deal about UX!
The decision to remove build options was informed (in part) by user experience: for every one user who was using them correctly to configure a package slightly differently for their uses, there were fifteen users following outdated SO answers and reporting their own breakages to us. Like every other package manager that distributes binary builds, we made a decision to improve the 99% use case.
The decision to remove build options was informed (in part) by user experience: for every one user who was using them correctly to configure a package slightly differently for their uses, there were fifteen users following outdated SO answers and reporting their own breakages to us. Like every other package manager that distributes binary builds, we made a decision to improve the 99% use case.
Hi there! I'm one of the Homebrew maintainers, albeit not a very active one for the past year or so.
I agree that we could definitely improve our documentation on what people can and should expect from a Homebrewed version of Python; I'll look into opening a PR that improves these docs tonight.
I agree that we could definitely improve our documentation on what people can and should expect from a Homebrewed version of Python; I'll look into opening a PR that improves these docs tonight.
That would be really helpful. In another thread you mentioned that "individual developers will always be best served to isolate their development environments from the overarching runtime environment that's been distributed to them". I agree, but it took me way longer to figure that out than I think was necessary, precisely because the "Homebrew and Python" docs don't mention that. I imagine a lot of other developers have been similarly affected, so many thanks in advance for looking into opening a pull request to enhance the documentation.
MacPorts is installing Python just fine here, fwiw. There's a package for each Python 3.x version so you can have whatever random versions you need installed. They can be used in virtualenvs easily and they don't break when you have to install an update for a 3.x.y release.
Not only that, MacPorts also kept all versions installed even after upgrade unless it's explicitly uninstalled (port echo inactive / port uninstall inactive); suffix every binaries with versions by default (e.g. pip-3.8, ansible-3.8) and only linked them as unversioned when explicitly set (port select --set python3 python38) which should be more familiar to people coming from Linux/BSD world.
I wish more people try out MacPorts.
I wish more people try out MacPorts.
I was using MacPorts when Homebrew came out. I was amazed how quickly Homebrew spread, given that it did everything worse.
I honestly put its popularity down to it being written in Ruby, which made it the default choice for the large population of Ruby programmers on the Mac.
I honestly put its popularity down to it being written in Ruby, which made it the default choice for the large population of Ruby programmers on the Mac.
Some years ago I tried submitting some fixes for MacPorts. It usually took a long time before they were picked up.
When I did the same for some homebrew formula I got a response a few hours later.
For me homebrew has a much more responsive community and due to their use of Github also a lower bar for contributions. Both might add to homebrew popularity.
When I did the same for some homebrew formula I got a response a few hours later.
For me homebrew has a much more responsive community and due to their use of Github also a lower bar for contributions. Both might add to homebrew popularity.
Disclaimer: I'm a MacPorts committer.
MacPorts uses GitHub too now. We maintain much more ports than Homebrew, and few of our committers have time to regularly go through the PR queue, so our community seems inactive. But actually, we just need more hands on PR management.
One important distinction is that most active ports in MacPorts have specific maintainers, and we usually put off merging PRs until the port maintainer approves. With more than 20,000 ports, it hard for the committers to test every change thoroughly, so we rely on our port maintainers to resolve issues and review complex PRs whenever possible.
In Homebrew, any maintainer could review and merge a PR instantly, combined with fewer ports to maintain, average time to merge would be much shorter.
MacPorts uses GitHub too now. We maintain much more ports than Homebrew, and few of our committers have time to regularly go through the PR queue, so our community seems inactive. But actually, we just need more hands on PR management.
One important distinction is that most active ports in MacPorts have specific maintainers, and we usually put off merging PRs until the port maintainer approves. With more than 20,000 ports, it hard for the committers to test every change thoroughly, so we rely on our port maintainers to resolve issues and review complex PRs whenever possible.
In Homebrew, any maintainer could review and merge a PR instantly, combined with fewer ports to maintain, average time to merge would be much shorter.
That's fair. I did submit a new package to MacPorts once, and it was a rather arduous process.
maybe I'm misremembering this era but macports required building everything from scratch, whereas homebrew offered pre-built binaries
As mentioned by the other poster they were both source distributions at first, followed by binaries later.
The pain is a bit lost to memory, but even as a long-time MacPorts user I understand why people wanted to switch. If everything has to be built from source you also have to build all your build-time dependencies from source, even weird ones for building docs or whatever. The behavior of MacPorts then (and still unchanged today) is to upgrade to the latest versions of all dependencies when installing or upgrading a requested package. So even if you had zlib sitting around if an update was pushed to zlib you're going to be building a new one. A lot of packages were forced to autoreconf because the Mac was still gaining popularity and a lot of source distributions didn't have ./configure scripts that supported this or that Mac-specific or MacPorts-specific functionality. So now you have to download and install whatever autotools you need to do autoreconf. And of course MacPorts ships a separate package for autoconf 1.3, 1.4, 1.5, and individual packages can depend on specific major versions of autoconf, so even if you had an up-to-date autoconf 1.4 you're still getting an upgrade to autoconf 1.3 if your package happened to ask for it.
It was ugly. Homebrew's initial insistence on requiring the re-use of system packages cut down on flexibility (you can't get a new zlib, you have to wait for Apple to ship one) but greased a lot of wheels people did care about. But as time has gone by both systems have improved on their faults and are in a better place than before. This particular bug sucks for the people affected but I think that the Homebrew people will eventually figure this one out, even if it means making their system more MacPorts-like. Like they have been for the past ten years. ;)
The pain is a bit lost to memory, but even as a long-time MacPorts user I understand why people wanted to switch. If everything has to be built from source you also have to build all your build-time dependencies from source, even weird ones for building docs or whatever. The behavior of MacPorts then (and still unchanged today) is to upgrade to the latest versions of all dependencies when installing or upgrading a requested package. So even if you had zlib sitting around if an update was pushed to zlib you're going to be building a new one. A lot of packages were forced to autoreconf because the Mac was still gaining popularity and a lot of source distributions didn't have ./configure scripts that supported this or that Mac-specific or MacPorts-specific functionality. So now you have to download and install whatever autotools you need to do autoreconf. And of course MacPorts ships a separate package for autoconf 1.3, 1.4, 1.5, and individual packages can depend on specific major versions of autoconf, so even if you had an up-to-date autoconf 1.4 you're still getting an upgrade to autoconf 1.3 if your package happened to ask for it.
It was ugly. Homebrew's initial insistence on requiring the re-use of system packages cut down on flexibility (you can't get a new zlib, you have to wait for Apple to ship one) but greased a lot of wheels people did care about. But as time has gone by both systems have improved on their faults and are in a better place than before. This particular bug sucks for the people affected but I think that the Homebrew people will eventually figure this one out, even if it means making their system more MacPorts-like. Like they have been for the past ten years. ;)
How well does macports and home brew play together on the same system these days?
I used to use ports back in the days and appreciated it coming from FreeBSD previously but at some point most things I wanted were simply only available in brew, at that time brew and ports didn’t play well together on the same system so sometime around SL/Lion I switched to brew completely.
I used to use ports back in the days and appreciated it coming from FreeBSD previously but at some point most things I wanted were simply only available in brew, at that time brew and ports didn’t play well together on the same system so sometime around SL/Lion I switched to brew completely.
Default prefix of Homebrew is now /opt/homebrew on macOS on ARM, which should not conflict with MacPorts anymore.
For macOS on Intel, MacPorts does not play well with Homebrew because it took /usr/local, so ports built from source might use Homebrew files and break.
For macOS on Intel, MacPorts does not play well with Homebrew because it took /usr/local, so ports built from source might use Homebrew files and break.
I see so as an Intel user I'd better stay away from mac ports if I want to use software only published on brew. I probably won't buy a new computer in the coming 5 years, so arm isn't relevant for me at this moment. But good to know for the future.
MacPorts has definitely always been the more correct and better way to do the job Brew claims to do.
But "brew" & "homebrew" is just a more charming and hipster name for a newbie to write in his blog for other newbies to follow, repeat eleventy-seven-hundred times.
But "brew" & "homebrew" is just a more charming and hipster name for a newbie to write in his blog for other newbies to follow, repeat eleventy-seven-hundred times.
Long-term homebrew user here, I have no blog and find the homebrew naming annoying. But being developed in Ruby and on GitHub made it a lot more welcoming than MacPorts ever felt when I used it.
I also never understood the concerns about correctness. Very few people want homebrew to be an apt/rpm replacement for servers that will provide stability for decades. Most Macs are effectively single-user systems, and if something goes wrong then nuking the whole homebrew directory and starting over is not a big deal.
I also never understood the concerns about correctness. Very few people want homebrew to be an apt/rpm replacement for servers that will provide stability for decades. Most Macs are effectively single-user systems, and if something goes wrong then nuking the whole homebrew directory and starting over is not a big deal.
Nuking the whole directory and starting over is a big deal for many people, myself included. I don't have time to waste, I want my software to just work, and that is why I use MacPorts.
I know this is frustrating, and I expect it affects more than just Python developers, but I think the author sums it up pretty well here:
> Homebrew’s Python is not for you. It exists to serve Homebrew, or more accurately, Homebrew’s other formulae. The primary purpose of Homebrew’s Python formula is to enable other Python-dependent Homebrew packages to work.
If you want to develop Python (likely Ruby/ PHP/ whatever), you are best keeping your dev environment separate from the interpreters installed for the purpose of running apps managed by Home-brew (or apt/ rpm/ etc).
> Homebrew’s Python is not for you. It exists to serve Homebrew, or more accurately, Homebrew’s other formulae. The primary purpose of Homebrew’s Python formula is to enable other Python-dependent Homebrew packages to work.
If you want to develop Python (likely Ruby/ PHP/ whatever), you are best keeping your dev environment separate from the interpreters installed for the purpose of running apps managed by Home-brew (or apt/ rpm/ etc).
Speaking as a Homebrew maintainer (and separately as someone who does both Ruby and Python development locally): this is the correct answer.
Homebrew's guarantees are no different from those of apt or any other system package manager; the only difference is that we don't have a stable distribution version to lock our Python or Ruby versions against. Individual developers will always be best served to isolate their development environments from the overarching runtime environment that's been distributed to them.
Homebrew's guarantees are no different from those of apt or any other system package manager; the only difference is that we don't have a stable distribution version to lock our Python or Ruby versions against. Individual developers will always be best served to isolate their development environments from the overarching runtime environment that's been distributed to them.
I think it would be wonderful if this last sentence could be expanded into its own section of the "Homebrew and Python" docs. [1]
[1] https://docs.brew.sh/Homebrew-and-Python.html
[1] https://docs.brew.sh/Homebrew-and-Python.html
Thanks, I'll base the language in the PR off of it. If you give me your GitHub handle, I'll tag you when I create the PR.
Certainly — that would be great. You can find me at: http://github.com/justinmayer
It was already merged, so I'll just go ahead and link it here: https://github.com/Homebrew/brew/pull/10513
Thanks again for bringing it to our attention!
Thanks again for bringing it to our attention!
Would it make sense to enforce this by making the Homebrew python keg-only? Would other Homebrew packages that need Python still work? This way I can keep my own separate installation of Python that does not interfere with Homebrew.
> Would it make sense to enforce this by making the Homebrew python keg-only?
Maybe. I wasn't privy to any discussions around making Python keg-only, but my experience with other formulae that are keg-only is that users will force-link keg-only packages into `/usr/local` anyways. What's more, there are probably enough people out there who do just want a barebones Python 3 installation and don't really care if it gets upgraded underneath them, and we'd like to provide that by default.
The Python ecosystem already contains mature tooling on both Linux and macOS for managing Python installations that are isolated from Homebrew and other system-level package managers, like pyenv. In general, I'd recommend using pyenv in parallel with Homebrew; that will get you the best of both worlds.
Maybe. I wasn't privy to any discussions around making Python keg-only, but my experience with other formulae that are keg-only is that users will force-link keg-only packages into `/usr/local` anyways. What's more, there are probably enough people out there who do just want a barebones Python 3 installation and don't really care if it gets upgraded underneath them, and we'd like to provide that by default.
The Python ecosystem already contains mature tooling on both Linux and macOS for managing Python installations that are isolated from Homebrew and other system-level package managers, like pyenv. In general, I'd recommend using pyenv in parallel with Homebrew; that will get you the best of both worlds.
A postgres update seriously fucked up my work Ruby install as well. We were using an old version and homebrew updated openssl on us (from 1.0.1 to 1.1 I think?) with no way of going back. (The Ruby install I was running was linked to 1.0.1, so it was not happy with having its library deleted.) I spent a month trying to figure out how to get that shit working again.
I really like Homebrew but its dependency management is horrible. The moment I saw that dyld "image not found" error message I didn't even have to finish reading the article to know what happened.
I really like Homebrew but its dependency management is horrible. The moment I saw that dyld "image not found" error message I didn't even have to finish reading the article to know what happened.
Homebrew Python recently had a problem with pasting in the REPL, because it uses GNU Readline and Homebrew recently updated that from 8.0 to 8.1. GNU Readline changed the default for bracketed paste mode from off in 8.0 to on in 8.1.
Python does not explicitly set this to what it wants (off), so if you didn't happen to have "set enable-bracketed-paste off" in your ~/.inputrc, pasting got messed up.
This isn't just a Homebrew thing. It also hit Arch Linux, and probably others.
As part of trying to figure out who to file a bug report with, I installed Apple's Python3 and I installed the Mac version of Python3 from python.org. It worked fine in the later two, so I filed the bug with Homebrew [1] (and they figured out that it is really a Python bug and they submitted a pull request upstream to have Python explicitly turn off bracketed paste).
The python.org Python3 was easy to install, and was the same version the Homebrew installed.
This raises a question. The article talks about alternate systems to manage Python installations instead of using Homebrew or using Apple's Python. But why do I want any of them? The python.org install seemed fine. Why not just use that and not have to deal with any third party Python installation managers?
[1] https://github.com/Homebrew/homebrew-core/issues/68193
Python does not explicitly set this to what it wants (off), so if you didn't happen to have "set enable-bracketed-paste off" in your ~/.inputrc, pasting got messed up.
This isn't just a Homebrew thing. It also hit Arch Linux, and probably others.
As part of trying to figure out who to file a bug report with, I installed Apple's Python3 and I installed the Mac version of Python3 from python.org. It worked fine in the later two, so I filed the bug with Homebrew [1] (and they figured out that it is really a Python bug and they submitted a pull request upstream to have Python explicitly turn off bracketed paste).
The python.org Python3 was easy to install, and was the same version the Homebrew installed.
This raises a question. The article talks about alternate systems to manage Python installations instead of using Homebrew or using Apple's Python. But why do I want any of them? The python.org install seemed fine. Why not just use that and not have to deal with any third party Python installation managers?
[1] https://github.com/Homebrew/homebrew-core/issues/68193
Because the Python.org installer does not have an uninstaller or an upgrader, so you end up with files dumped in various locations on your system.
You should give MacPorts a try...
MacPorts allows you to install multiple Python versions at the same time and I personally have several venvs with Python 2.7, 3.6, 3.7, 3.8, 3.9... all running without a glitch.
MacPorts allows you to install multiple Python versions at the same time and I personally have several venvs with Python 2.7, 3.6, 3.7, 3.8, 3.9... all running without a glitch.
This was one of the pain points that made me move back to Linux on my devices, and I'm glad I did it. Homebrew is infuriating when you're used to real package managers.
When you switch from Linux to Mac/Windows, you think "wow, this OS was actually made for GUIs. Everything just works. I don't have to restart ALSA every 3 days."
When you switch from Mac/Windows to Linux, you think "wow, this OS was actually made for software development. Everything just works. I don't have to resolve a library version conflict every 3 days."
When you switch from Mac/Windows to Linux, you think "wow, this OS was actually made for software development. Everything just works. I don't have to resolve a library version conflict every 3 days."
I have never needed to restart ALSA, but I agree that it can be a bit rougher around the edges. Bluetooth is flakey everywhere, but I do seem to encounter more "edge cases" in Linux than my Mac.
If you can live with the rough edges, I find the general experience with applications or my printer suddenly being incompatible under MacOS for "reasons" is much less of an issue under Linux.
If you can live with the rough edges, I find the general experience with applications or my printer suddenly being incompatible under MacOS for "reasons" is much less of an issue under Linux.
Agree that Mac OS backwards compatibility is atrocious!
I think another big challenge for Linux is battery life and sleep/wake on laptops. Last time I considered replacing Mac OS with Ubuntu on my laptop, I got scared away by reports that battery life would decrease by half.
I think another big challenge for Linux is battery life and sleep/wake on laptops. Last time I considered replacing Mac OS with Ubuntu on my laptop, I got scared away by reports that battery life would decrease by half.
My Lenovo laptop gets better battery life under Linux than it did under Windows, and I've generally found the battery life estimations (for my use cases anyway) to be complete bullshit even under MacOS.
I'm excited to be able to upgrade to the M1 series Mac's as it sounds like I might actually get battery life, though I have the 2019 16" Pro and can't justify the upgrade just yet, even if that thing overheats with just a Slack video.
I'm excited to be able to upgrade to the M1 series Mac's as it sounds like I might actually get battery life, though I have the 2019 16" Pro and can't justify the upgrade just yet, even if that thing overheats with just a Slack video.
I'd be perfectly happy with homebrew if it didn't auto-update all packages when installing something new. E.g. I do `brew install ffmpeg`, and as part of that it upgrades python from 3.8 to 3.9 (???), breaking imports in all my scripts. I know how to work around this, but still, surprising choice for the default behavior.
> I know how to work around this
For others similarly curious, it appears setting [`$HOMEBREW_NO_AUTO_UPDATE=1`](https://github.com/Homebrew/brew/blob/2.7.7/docs/Manpage.md#...) is the magic sauce (or the perhaps more accurate link: https://github.com/Homebrew/brew/blob/2.7.7/Library/Homebrew... )
For others similarly curious, it appears setting [`$HOMEBREW_NO_AUTO_UPDATE=1`](https://github.com/Homebrew/brew/blob/2.7.7/docs/Manpage.md#...) is the magic sauce (or the perhaps more accurate link: https://github.com/Homebrew/brew/blob/2.7.7/Library/Homebrew... )
That's what the author is talking about. The Python that comes with HomeBrew is not for you, it's only for the dependencies in the HomeBrew world.
Everything in homebrew auto-updates on new installs, not just python. Databases, CLI tools, etc.
I'm a little surprised more people aren't using `anyenv`. That lets you install & update pyenv, nodenv, rbenv, etc. (though some support for certain languages is spotty) and thus get everything built/installed the way you want it.
Also, when using pyenv, I almost always set `env PYTHON_CONFIGURE_OPTS="--enable-framework" pyenv install -v ${pylatest}` (macOS) or `env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install -v ${pylatest}` (Linux). (I also set some CFLAGS, but these are personal preference.) I've had plenty of situations where not having the shared/framework build has screwed something up (and in fact that was my original reason for switching from Homebrew or package manager-installed Python years ago), and I've never had issues where the shared build screwed something up.
Also, when using pyenv, I almost always set `env PYTHON_CONFIGURE_OPTS="--enable-framework" pyenv install -v ${pylatest}` (macOS) or `env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install -v ${pylatest}` (Linux). (I also set some CFLAGS, but these are personal preference.) I've had plenty of situations where not having the shared/framework build has screwed something up (and in fact that was my original reason for switching from Homebrew or package manager-installed Python years ago), and I've never had issues where the shared build screwed something up.
asdf[1] mentioned in the article is similar to anyenv but instead of being a wrapper around *env, it implements its own simple framework for creating a version manager (e.g. [2]). Usually asdf plugin uses the same underlying installer as its *env equivalent (e.g. asdf-python uses python-build, asdf-ruby uses ruby-build, asdf-erlang uses kerl).
[1]: https://asdf-vm.com/
[1]: https://asdf-vm.com/#/plugins-all
[1]: https://asdf-vm.com/
[1]: https://asdf-vm.com/#/plugins-all
Can someone explain why nobody seems to use the official Python installers from python.org?
In my experience they work great, you can install multiple versions, and even pip3 seems to just work with them.
I'm not a heavy Python user, but ever since I started using the official installers it just works.
In my experience they work great, you can install multiple versions, and even pip3 seems to just work with them.
I'm not a heavy Python user, but ever since I started using the official installers it just works.
Article author here. I think folks like asdf and pyenv because of the ease with which new versions can be installed and removed. For example, I can run asdf install python 3.9.1 and have a new version installed in minutes, without leaving the terminal console. I don't think that's possible via the official python.org installers.
My experience on MacOS was that it doesn't update and cant be uninstalled without using 3rd party scripts.
I could be mistaken, but I didn't find any documentation on how to do it and the fact that other people had projects to try and do it mean that I wasnt alone on finding this a bit of a mess.
I could be mistaken, but I didn't find any documentation on how to do it and the fact that other people had projects to try and do it mean that I wasnt alone on finding this a bit of a mess.
Where my conda crew at?
I was worried that even miniconda seemed kinda bloaty, but it was the only way I could consistently use numpy on apple silicon. Really good option for local dev environments
If you haven't yet, try out mamba. It's more or less a drop-in conda replacement but written in C++ using a super fast SAT solver. The speed difference is incredible--I rarely even have time to finish reading the output before it's done installing a conda environment.
Thank you very much for this suggestion. My team's workflow is conda heavy and this will speed up my development by orders of magnitude.
I believe I have noticed this as well with Ruby as well. All Gems work great, then, after a brew operation, I need to reinstall/rebuild against an updated Ruby. It's certainly frustrating and can bite you, especially since it is unexpected.
I have had good experiences with Conda [1], and generally use Miniconda.
[1] https://docs.conda.io/en/latest/
[1] https://docs.conda.io/en/latest/
I totally get that package management is Hard. However, I feel like Homebrew should be able to track what's there because it's a dependency vs what's there because the user wants it to be and act accordingly.
Granted, I've only lived in these tools just long enough to install what I need, so I'm sure I'm naively overlooking a lot of nuance.
Granted, I've only lived in these tools just long enough to install what I need, so I'm sure I'm naively overlooking a lot of nuance.
Ah interesting. It appears that the reason for this difference is that Python on Homebrew is delivered as a single package for development and execution. The true intention should have been to have had Python-Homebrew to create the environment for Homebrew and Python-Dev which you can use for your development.
Sort of analogous to the way the JVM is distributed usually as separate packages with the JRE and the JDK.
I don't write very much prod-grade Python, so I just recreate the venv with my requirements.txt instead.
Sort of analogous to the way the JVM is distributed usually as separate packages with the JRE and the JDK.
I don't write very much prod-grade Python, so I just recreate the venv with my requirements.txt instead.
As someone who doesn't often use Python what is the current "best" solution for installing and managing your environment?
Last time I used Python on macOS and Windows I manually installed the official version from python.org and used pip and virtualenv. It wasn't too bad but it seems things have changed in the last 6 years or so? Or is this still the most flexible way to go?
I would be grateful for any guidance.
Last time I used Python on macOS and Windows I manually installed the official version from python.org and used pip and virtualenv. It wasn't too bad but it seems things have changed in the last 6 years or so? Or is this still the most flexible way to go?
I would be grateful for any guidance.
For totally greenfield python projects check out poetry, it is an opinionated python project workflow that's very close to the nodejs/npm ecosystem's package.json. It does a virtualenv behind the scenes, pins your python dependencies with a lock file, and even has niceties like dev scripts as first class citizens.
Otherwise if you're trying to use some python project with particular dependencies pyenv is a good way to get a specific python version. But if the project has a lot of native dependencies (think scientific computing with lots of fortran, C++, etc. code to compile) then you're almost certainly better off with conda as a higher-level entire binary system manager.
Otherwise if you're trying to use some python project with particular dependencies pyenv is a good way to get a specific python version. But if the project has a lot of native dependencies (think scientific computing with lots of fortran, C++, etc. code to compile) then you're almost certainly better off with conda as a higher-level entire binary system manager.
I use asdf and its Python plugin [1] to install Python interpreter versions.
For managing virtual environments on my preferred shell (Fish), I use (and maintain) VirtualFish [2].
For managing project dependencies, I activate environments via VirtualFish and then use Poetry [3] to update the dependencies within the environments.
[1]: https://github.com/danhper/asdf-python
[2]: http://github.com/justinmayer/virtualfish
[3]: http://python-poetry.org
For managing virtual environments on my preferred shell (Fish), I use (and maintain) VirtualFish [2].
For managing project dependencies, I activate environments via VirtualFish and then use Poetry [3] to update the dependencies within the environments.
[1]: https://github.com/danhper/asdf-python
[2]: http://github.com/justinmayer/virtualfish
[3]: http://python-poetry.org
Homebrew and its excellent community was once indispensible on the Mac, but various pain points including this flakiness made me switch to (mini)conda for nearly everything, despite initial apprehension. There are some wrinkles and the resolver is slow, but in my hands conda is just so much more reliable than Homebrew ever was.
I loved Homebrew, but it really seems to have gone to shit the last couple years. Are they spreading themselves too thin?
The general trend I've noticed over the years is that they've been reducing edge-cases at the cost of making normal behavior worse. The change noted in TFA is one of these: always upgrading a package's dependencies when upgrading a package greatly reduces the numbers of combinations of versions that users can possibly have and so probably reduces the number of bug reports that come down to local environment things, but it comes at the cost of effectively removing the ability to upgrade a specific tool without fucking with the rest of your system.
Similarly I'm sure that making `brew install X` implicitly perform `brew upgrade` simplifies things a lot for them, but it's been a major source of hassle for me and it's making me consider switching away from homebrew.
Similarly I'm sure that making `brew install X` implicitly perform `brew upgrade` simplifies things a lot for them, but it's been a major source of hassle for me and it's making me consider switching away from homebrew.
Have you explored the `$HOMEBREW_NO_AUTO_UPDATE=1` env var? https://github.com/Homebrew/brew/blob/2.7.7/docs/Manpage.md#...
I can't tell if that would influence the behavior you're experiencing, but it sounds like it makes the brew command more deterministic
I can't tell if that would influence the behavior you're experiencing, but it sounds like it makes the brew command more deterministic
That sounds unrelated. It disables automatically calling `brew update` (which I'm fine with), but not `brew upgrade`.
Relevant xkcd:
https://xkcd.com/1987/
https://xkcd.com/1987/
I gave up and just started using the OSX system copy (along w/ Google Colab), the main reason being that nearly every time there was a system update something would break and I would have to pin/unpin [email protected]
So many different ways to install Python - doesn’t anybody just use the official installer from python.org/downloads?
People doing first-timer Python classes in school.
Python's insistence that it be a system-level utility in one global location, with one global set of libraries, etc. is kind of why we're in this mess of many different ways to install and use python (and the myriad of workarounds like virtual envs, pyenv, pipx, etc.). The official CPython installer is fine for some simple scripting or learning the language, but as soon as you need to tightly control and define multiple different python installs, different versions of libraries, etc. to satisfy different constraints it becomes a huge headache.
Meanwhile, they could have fixed this shit with 3... had time until 3.4 or so (when few were using it anyway).
>https://asdf-vm.com/
I have successfully confused people who sought meaning in 'asdf' when the string is absolutely meaningless, actually being the left-hand home keys.
'asdf' could be interpreted as "totally blew off that 'naming' thing".
I have successfully confused people who sought meaning in 'asdf' when the string is absolutely meaningless, actually being the left-hand home keys.
'asdf' could be interpreted as "totally blew off that 'naming' thing".
I will almost resist the urge to truncate the title to "Python is Not for You" -- but I'd like to second the benefits of using a python version manager such as pyenv. Yes, all the versions waste an ungodly amount of space, but you probably can't/won't use Clojure instead I guess.
I gave up on Homebrew for Python development and now use Conda. Never looked back.
Why does everything related to versioning and packaging have to be so difficult with Python? I never understood that.
I had a whole comment written up saying basically this, but in less-kind terms. Python just seems so incredibly brittle.
I'd hazard a guess that it is because Python is so old. It was literally created in the 80s. Package management, explosion of dependencies, multiple implementations of the interpreter, being added to the base operating system, and all that fancy stuff came after. I remember doing versioning and packaging in C++ in early 2000s, and it being just as annoying but for different reasons.
People are trying to fix it, and their are some great packages out there that make it easier. Pyenv, Poetry, this asdf tool the fine article mentions, all seem like good solutions.
People are trying to fix it, and their are some great packages out there that make it easier. Pyenv, Poetry, this asdf tool the fine article mentions, all seem like good solutions.
>I'd hazard a guess that it is because Python is so old. It was literally created in the 80s. Package management, explosion of dependencies, multiple implementations of the interpreter, being added to the base operating system, and all that fancy stuff came after.
So? You can still create nice tools for it, even decades after a language has been created...
So? You can still create nice tools for it, even decades after a language has been created...
It's not. People arbitrarily complicate their personal workflows, especially having years of habit predating the existence of modern tools, but this is a voluntary decision to make local development complicated and you can choose not to work that way.
As an example, any time Homebrew ships a new update I run “pipx reinstall-all” and, for local projects, I let pipenv / poetry rebuild the virtualenv the next time I use it. This is automated and takes far less of my time than, say, managing dependency updates.
As an example, any time Homebrew ships a new update I run “pipx reinstall-all” and, for local projects, I let pipenv / poetry rebuild the virtualenv the next time I use it. This is automated and takes far less of my time than, say, managing dependency updates.
I think part of the challenge (from my perspective, as someone who only rarely needs to mess with python) is not that the workflow you describe is particularly complicated in isolation, it's the shear number of different tools that ostensibly have the same goal: to manage python virtual environments and packages. Because there are so many different tools, there is a near-limitless supply of different articles and SO posts in which people describe the "best" way to accomplish the most basic setup and maintenance of a virtual environment. It's quite difficult for new and/or occasional users to choose a good way to get started. Even if there is an "official" way documented on python.org for each Python version, when you run into a problem with it, inevitably you're stuck filtering through lots of advice to "just use <third-party tool xyz> instead".
>As an example, any time Homebrew ships a new update I run “pipx reinstall-all” and, for local projects, I let pipenv / poetry rebuild the virtualenv the next time I use it. This is automated and takes far less of my time than, say, managing dependency updates.
So, giving up (and re-installing everything from scratch or rebuilding virtualenvs, using several third party tools (pipx on top of pip, pipenv and/or poetry in lieu/in place of virtualenv) is your argument to how "it's not complicated" and people merely "arbitrarily complicate their personal workflows"?
Oh, the irony!
So, giving up (and re-installing everything from scratch or rebuilding virtualenvs, using several third party tools (pipx on top of pip, pipenv and/or poetry in lieu/in place of virtualenv) is your argument to how "it's not complicated" and people merely "arbitrarily complicate their personal workflows"?
Oh, the irony!
The key part which you missed is that everything is automated, always installed in an isolated virtual environment, the package are automatically tracked and updated, and the tools which make this happen are installed from Homebrew and update at the same time.
As to why there are two tools, it's because they have separate goals: pipx provides a robust way to manage global installs of Python utilities in per-tool virtualenvs while tools like poetry or pipenv are aimed at projects you're developing and handle things like automating the process of installing updates & updating the corresponding metadata files so you can ship clean updates. All of them use virtualenvs and pip underneath — it's just a modern workflow.
The key part is recognizing that the problem comes from trying to fight updates: software changes and accepting that means moving to a workflow which automates that rather than spending time trying to freeze everything in amber. Learning to use a tool every decade or so is a LOT less time consuming than doing this by hand. The average poster in this thread has already spent more time on this topic than my projects have needed for the mechanics of installing Python packages in any given year.
As to why there are two tools, it's because they have separate goals: pipx provides a robust way to manage global installs of Python utilities in per-tool virtualenvs while tools like poetry or pipenv are aimed at projects you're developing and handle things like automating the process of installing updates & updating the corresponding metadata files so you can ship clean updates. All of them use virtualenvs and pip underneath — it's just a modern workflow.
The key part is recognizing that the problem comes from trying to fight updates: software changes and accepting that means moving to a workflow which automates that rather than spending time trying to freeze everything in amber. Learning to use a tool every decade or so is a LOT less time consuming than doing this by hand. The average poster in this thread has already spent more time on this topic than my projects have needed for the mechanics of installing Python packages in any given year.
>The key part which you missed is that everything is automated, always installed in an isolated virtual environment, the package are automatically tracked and updated, and the tools which make this happen are installed from Homebrew and update at the same time.
That we can automate our way out of a mess doesn't mean there is a mess.
Even more so if the automation involves extra inconvenience (like, from writing the automation to waiting to rebuild), which with a different design in the Python packaging wouldn't be needed.
>The key part is recognizing that the problem comes from trying to fight updates: software changes and accepting that means moving to a workflow which automates that rather than spending time trying to freeze everything in amber.
I can find and install updates just fine in other package managers for other languages, without messing my setup.
I can also move my whole environment to any path I like (with just copying).
That we can automate our way out of a mess doesn't mean there is a mess.
Even more so if the automation involves extra inconvenience (like, from writing the automation to waiting to rebuild), which with a different design in the Python packaging wouldn't be needed.
>The key part is recognizing that the problem comes from trying to fight updates: software changes and accepting that means moving to a workflow which automates that rather than spending time trying to freeze everything in amber.
I can find and install updates just fine in other package managers for other languages, without messing my setup.
I can also move my whole environment to any path I like (with just copying).
Your position seems to be that Python packaging is a mess for non-specific reasons but also that nobody is allowed to write tools to improve usability. Think about pip – that wasn't the first tool to do that job but it (gasp) automated several things which didn't have great solutions. Virtualenv has the same story. This will be no different.
> I can find and install updates just fine in other package managers for other languages, without messing my setup.
Just like Python, in other words? (remember, this thread is not about installing package updates)
> I can also move my whole environment to any path I like (with just copying).
The problem we're talking about here is what happens when you get a new build of the language itself. This is also true of most other dynamic languages unless you manage to avoid using any sort of native code linked against the language runtime. A compiled language which produces a static binary won't need that, of course, but that doesn't avoid the need to periodically recompile to stay current with dependencies and language changes. In all of these cases, automation makes the process fast and easy.
> I can find and install updates just fine in other package managers for other languages, without messing my setup.
Just like Python, in other words? (remember, this thread is not about installing package updates)
> I can also move my whole environment to any path I like (with just copying).
The problem we're talking about here is what happens when you get a new build of the language itself. This is also true of most other dynamic languages unless you manage to avoid using any sort of native code linked against the language runtime. A compiled language which produces a static binary won't need that, of course, but that doesn't avoid the need to periodically recompile to stay current with dependencies and language changes. In all of these cases, automation makes the process fast and easy.
>Your position seems to be that Python packaging is a mess for non-specific reasons
No, for very specific reasons. Historical cruft, too many competing tools (from pip to conda, and from venv to poetry, not to mention setuptools), conceptual garbage like wheels, pths, etc., non-movable virtual environments, and so on.
>The problem we're talking about here is what happens when you get a new build of the language itself. This is also true of most other dynamic languages unless you manage to avoid using any sort of native code linked against the language runtime.
And yet node handles this much more elegantly (with C++/gyp modules). And minor releases should absolutely no need this.
No, for very specific reasons. Historical cruft, too many competing tools (from pip to conda, and from venv to poetry, not to mention setuptools), conceptual garbage like wheels, pths, etc., non-movable virtual environments, and so on.
>The problem we're talking about here is what happens when you get a new build of the language itself. This is also true of most other dynamic languages unless you manage to avoid using any sort of native code linked against the language runtime.
And yet node handles this much more elegantly (with C++/gyp modules). And minor releases should absolutely no need this.
Python was created in the 90s when most unix users were on huge minicomputer-like shared systems (mostly at universities). In those days you had entire IT staff dedicated to curating and maintaining all of the software on the system, making sure it ran well, keeping it updated, etc. Guido explicitly did not want to bog the language development down with time spent on building packaging, deployment, etc.--those were matters for the system admins. We've all kind of been dealing with the consequences of that decision for many years.
This all stems from the official policy of only caring about themselves and putting themselves above the users. https://docs.brew.sh/Maintainers-Avoiding-Burnout#3-prioriti...
Is this somehow a bad thing? Your comment certainly seems to imply that it is.
At the end of the day, they're people volunteering their time for little to no benefit. Of course they should put themselves above their users; their users aren't even paying them for said time.
At the end of the day, they're people volunteering their time for little to no benefit. Of course they should put themselves above their users; their users aren't even paying them for said time.
It certainly is a bad thing, it has led to them making a lot of user hostile changes that have made Homebrew a much worse project than it used to be. This article describes just one example of the kind of things that are considered ok by the Homebrew maintainers.
Once again, they're volunteers doing unpaid work. They're not beholden to anyone's interests except their own.
If you don't like their decisions, you're free to criticize them, but to imply that (rightly) putting themselves above their users is a bad thing is an entirely different beast when they don't owe anything to their users.
At the end of the day, if you don't like their decisions, you're free to work towards becoming a maintainer and make your voice heard, or fork it and do whatever you want and think is right. But nobody owes you or any other user anything unless you explicitly are paying them for the time and effort.
If you don't like their decisions, you're free to criticize them, but to imply that (rightly) putting themselves above their users is a bad thing is an entirely different beast when they don't owe anything to their users.
At the end of the day, if you don't like their decisions, you're free to work towards becoming a maintainer and make your voice heard, or fork it and do whatever you want and think is right. But nobody owes you or any other user anything unless you explicitly are paying them for the time and effort.
You can even fork an individual formula by creating a tap, adding the one formula you disagree with, and customize it to your heart’s content.
The only drawback is that the Homebrew organization won’t support, update, or build bottles for, custom taps.
The only drawback is that the Homebrew organization won’t support, update, or build bottles for, custom taps.
To be absolutely clear: what Homebrew is doing here is consistent with what every other system package manager does. If you've ever done `do-release-upgrade` to bump your Ubuntu LTS, you'll have run into the exact same problem.
The only difference is that Homebrew doesn't have a stable system distribution to pin to (we're not Apple, and we have no say over macOS's development), so we make upgrade decisions for packages like Python based on their stability with our packaged ecosystem. It's a poor practice to use your system Python distribution for any sort of virtualized development environment, and Homebrew behaves identically to every other system package manager in that regard.
Please keep in mind that we're a community of volunteers answering to millions of users. Your idea of "user hostile" is our idea of "simple for the sake of preserving our sanity."
The only difference is that Homebrew doesn't have a stable system distribution to pin to (we're not Apple, and we have no say over macOS's development), so we make upgrade decisions for packages like Python based on their stability with our packaged ecosystem. It's a poor practice to use your system Python distribution for any sort of virtualized development environment, and Homebrew behaves identically to every other system package manager in that regard.
Please keep in mind that we're a community of volunteers answering to millions of users. Your idea of "user hostile" is our idea of "simple for the sake of preserving our sanity."
No other package manager I've used automatically performs destructive actions on a package installation without asking me, homebrew does. But aside from that, homebrew used to be much nicer, it had options on formulae, it worked with the system ruby instead of installing its own periodically, it allowed patches that improved system integration even if they made the resulting binary less command line focused (for example emacs has macOS specific patches locked behind building the gui app).
And yeah, it's the homebrew maintainer's project, but when you're in a position to help people or make their life worse, it's immoral to do the latter.
And yeah, it's the homebrew maintainer's project, but when you're in a position to help people or make their life worse, it's immoral to do the latter.
As someone who has suffered burnout at various points I think that policy is a great idea.