Homebrew 1.9.0(brew.sh)
brew.sh
Homebrew 1.9.0
https://brew.sh/2019/01/09/homebrew-1.9.0/
115 comments
I wanted to comment about this as a LPT because of past experiences, then went to run `brew update` and `brew upgrade` which took more than an hour because it had to build a new version of LLVM from source... Then I ran `brew cleanup` and boom, 4.4GB of free space on a MacBook Air with a small disk.
Overall I hate how slow brew is, especially `update` should be 10x faster, it should maybe warn about source builds that could take forever, and it should maybe give a summary like `apt` whether there are packages to be `autoremove`d.
Overall I hate how slow brew is, especially `update` should be 10x faster, it should maybe warn about source builds that could take forever, and it should maybe give a summary like `apt` whether there are packages to be `autoremove`d.
You may know it already but as a PSA: you can usually avoid source builds on `brew install` and `brew upgrade` if you do it without options (such as `--with-lldb`). Unless you customize your installation using options, Homebrew will install your package from bottle, which is infinity times faster than building from source.
Upgrading Postgresql between major version requires previous version to be installed[1]. With auto-cleanup, when new version is installed, old will be uninstalled automatically, unless they added some exclusion mechanism for it, but I don't see anything about it in "postgres" formula[2].
[1] https://www.postgresql.org/docs/11/pgupgrade.html
[2] https://github.com/Homebrew/homebrew-core/blob/master/Formul...
[1] https://www.postgresql.org/docs/11/pgupgrade.html
[2] https://github.com/Homebrew/homebrew-core/blob/master/Formul...
Wow. That's so true. I just got back 5GB. I didn't know about this. Thanks :-)
I'm excited about it. `brew upgrade` now runs `brew update` and if you set the environment variable `brew upgrade` will run `brew cleanup` too.
> `brew upgrade` now runs `brew update`
Is that documented anywhere? I guess it makes sense for most people, but I often do `brew update && brew fetch $(brew outdated)`, then later rely on `brew upgrade` working without hitting the network. I'd hate for that to change.
Is that documented anywhere? I guess it makes sense for most people, but I often do `brew update && brew fetch $(brew outdated)`, then later rely on `brew upgrade` working without hitting the network. I'd hate for that to change.
It's not explicitly documented but it's the same behaviour as has been on `brew tap` and `brew install` since we introduced auto-update. `man brew` details how to customise this behaviour.
I didn't know cleanup existed, so never ran it until now.
==> This operation has freed approximately 8.9GB of disk space. ==> This operation has freed approximately 343.7MB of disk space.
Possibly helpful to some macOS folks: I converted all of my cronjobs to launchd agents/daemons and the increased dependability has been helpful, since I use a laptop which sleeps a lot.
What I mean: "Unlike cron which skips job invocations when the computer is asleep, launchd will start the job the next time the computer wakes up. If multiple intervals transpire before the computer is woken, those events will be coalesced into one event upon wake from sleep."[1]
[1]: http://www.launchd.info
What I mean: "Unlike cron which skips job invocations when the computer is asleep, launchd will start the job the next time the computer wakes up. If multiple intervals transpire before the computer is woken, those events will be coalesced into one event upon wake from sleep."[1]
[1]: http://www.launchd.info
Ha, I wish I knew about that command earlier. I was wondering where all my disk space was going.
==> This operation has freed approximately 41.3GB of disk space.
Automatic cleanup would have saved me a lot of confusion.
==> This operation has freed approximately 41.3GB of disk space.
Automatic cleanup would have saved me a lot of confusion.
==> This operation has freed approximately 3.3GB of disk space.
Wow, thanks
Wow, thanks
I really wish these release notes would explain what is going on for people who don't know the internals of homebrew. Especially since homebrew force-updates itself and there's no way for users to avoid these changes.[1]
There are several updates in here that make me worry that this is somehow going to break my environment or otherwise bite me in the ass when I do something innocent like install Ghostscript and find that Emacs no longer works and to figure out how to fix it's going to be hours of reading github issues. Which is an experience I've had before, repeatedly.
"brew update no longer migrates legacy keg symlinks, tap names, repository locations, cache locations or cache entries."
Is this going to break something I installed years ago? Maybe the homebrew maintainers know, but average tool users don't.
"Homebrew/homebrew-core requires all formulae to be open source by the OSI definition."
Does this mean that something I used homebrew to install in a previous version will no longer be getting updates? Will things that depend on it still be getting updates? Does whether or not updates happen depend on some random committer's legalistic interpretation of an open source definition? Why exactly is this kind of purist decision being imposed on users?
[in 2.0] "brew cleanup will be run periodically and for trigger individual formula cleanup on reinstall, install or upgrade. You can enable this behaviour now on 1.9.0 by setting the HOMEBREW_INSTALL_CLEANUP variable."
So you're going to periodically make maintenance changes to my system without my permission. Goodie.
Someone really needs to write a guide to migrating away from homebrew for people in the unfortunate situation of having used this tool for a long time, and find it taking more and more liberties with one's computer, but also having numerous essential packages now managed by it and not sure how to move to alternatives without weeks of breakage.
[1] https://paultopia.github.io/posts-output/anti-homebrew/
There are several updates in here that make me worry that this is somehow going to break my environment or otherwise bite me in the ass when I do something innocent like install Ghostscript and find that Emacs no longer works and to figure out how to fix it's going to be hours of reading github issues. Which is an experience I've had before, repeatedly.
"brew update no longer migrates legacy keg symlinks, tap names, repository locations, cache locations or cache entries."
Is this going to break something I installed years ago? Maybe the homebrew maintainers know, but average tool users don't.
"Homebrew/homebrew-core requires all formulae to be open source by the OSI definition."
Does this mean that something I used homebrew to install in a previous version will no longer be getting updates? Will things that depend on it still be getting updates? Does whether or not updates happen depend on some random committer's legalistic interpretation of an open source definition? Why exactly is this kind of purist decision being imposed on users?
[in 2.0] "brew cleanup will be run periodically and for trigger individual formula cleanup on reinstall, install or upgrade. You can enable this behaviour now on 1.9.0 by setting the HOMEBREW_INSTALL_CLEANUP variable."
So you're going to periodically make maintenance changes to my system without my permission. Goodie.
Someone really needs to write a guide to migrating away from homebrew for people in the unfortunate situation of having used this tool for a long time, and find it taking more and more liberties with one's computer, but also having numerous essential packages now managed by it and not sure how to move to alternatives without weeks of breakage.
[1] https://paultopia.github.io/posts-output/anti-homebrew/
> "brew update no longer migrates legacy keg symlinks, tap names, repository locations, cache locations or cache entries."
The biggest thing that has kept me on MacPorts and away from Homebrew is that I don't understand half of the terms that they use.
What is a keg? What is a tap? From the article you linked, what is a cask? What is the Caskroom? Elsewhere I've heard talk of "bottles" - what is a bottle, in this context? What are formulae?
The biggest thing that has kept me on MacPorts and away from Homebrew is that I don't understand half of the terms that they use.
What is a keg? What is a tap? From the article you linked, what is a cask? What is the Caskroom? Elsewhere I've heard talk of "bottles" - what is a bottle, in this context? What are formulae?
When I started using Homebrew, I found those terms super intimidating and overly quirky but that went away quickly!
Keg: a dedicated subdirectory for each installed package.
Cellar: this is where the kegs are. Usually `/usr/local/Cellar`.
Formula: a file that tells Homebrew how to build and install a specific CLI package. (Similar to a portfile, I guess?)
Tap: repository of formulae, e. g. `homebrew-core` (the repository where Homebrew keeps all the core formulae)
Bottle: pre-built binary package. Homebrew usually builds and hosts them for you. At install time, Homebrew downloads and uses them whenever it can so you won’t need to wait through unnecessary source builds.
Cask: Represents a package that can’t be built from source. In practice, a Cask usually means something that ends with `.app`. Technically, Homebrew Cask is very much a separate package manager in itself. (You can tell by looking at the individual `brew cask` commands.) That’s because it caters to somewhat different use cases, and because it used to be a separate project before it went on to merge with Homebrew.
Caskroom: directory hierarchy for Cask-related bookkeeping. Caskroom is also the name of the former organization in charge of Casks; but that’s history now given all things Cask have officially become part of Homebrew.
It’s cool that MacPorts works well for you but hey, maybe try Homebrew some time to find out how you like it? (Disclosure: am one of the core maintainers.)
Keg: a dedicated subdirectory for each installed package.
Cellar: this is where the kegs are. Usually `/usr/local/Cellar`.
Formula: a file that tells Homebrew how to build and install a specific CLI package. (Similar to a portfile, I guess?)
Tap: repository of formulae, e. g. `homebrew-core` (the repository where Homebrew keeps all the core formulae)
Bottle: pre-built binary package. Homebrew usually builds and hosts them for you. At install time, Homebrew downloads and uses them whenever it can so you won’t need to wait through unnecessary source builds.
Cask: Represents a package that can’t be built from source. In practice, a Cask usually means something that ends with `.app`. Technically, Homebrew Cask is very much a separate package manager in itself. (You can tell by looking at the individual `brew cask` commands.) That’s because it caters to somewhat different use cases, and because it used to be a separate project before it went on to merge with Homebrew.
Caskroom: directory hierarchy for Cask-related bookkeeping. Caskroom is also the name of the former organization in charge of Casks; but that’s history now given all things Cask have officially become part of Homebrew.
It’s cool that MacPorts works well for you but hey, maybe try Homebrew some time to find out how you like it? (Disclosure: am one of the core maintainers.)
>So you're going to periodically make maintenance changes to my system without my permission. Goodie.
I suspect this is partly because the cache can consume a huge amount of disk space over time and many users might never run `brew cleanup` or doesn't even know about it. And then wonder where all their disk space has gone.
I suspect this is partly because the cache can consume a huge amount of disk space over time and many users might never run `brew cleanup` or doesn't even know about it. And then wonder where all their disk space has gone.
Yup, I just deleted 13GB of cache a few days ago.
That would be a great reason to communicate with users in a prominent place in the docs. Sadly, the maintainers seem to always choose to force the user behavior that they see as desirable rather than communicate with users about it. For what is, at bottom, a developer tool this is madness.
Git also does internal housekeeping in the background when you pull. Actually most developer tools perform operations on their caches and other internal stores over time without asking or telling you about it, so this shouldn't really be a concern.
(a secret: we copied the time period that Git uses for its periodic housekeeping)
Isn't it that what mac is all about ? "We know better, let us do the thinking for you" ?
Not sure if this is tongue-in-cheek or not but we do try to follow the/our perceived Apple approach of "sensible defaults which take the thinking out of using the tool". This means doing things automatically for users and letting them opt-out rather than requiring them to read all our documentation to know how to avoid e.g. a bloated cache directory.
I appreciate it. I can RTFM and figure out how to do a million ad-hoc maintenance chores for the thousand tools I use daily, but I'd rather not have to if those tools care more or less take care of themselves.
And let's be honest, most often Apple does know what's better for the average user.
> Especially since homebrew force-updates itself and there's no way for users to avoid these changes.[1]
This is not true. `man brew` documents how to disable auto-updates.
> Why exactly is this kind of purist decision being imposed on users?
Because there's a lot of current debate about whether things being relicensed under the Commons Clause is "open source" or not and we'd rather lean on an established, well-documented authority on what that definition is. Additionally, part of our agreed membership of the Software Freedom Conservancy (who help us with a lot) involves ensuring our core formulae are open source software.
> So you're going to periodically make maintenance changes to my system without my permission. Goodie.
Again, `brew man` documents how to disable this.
---
The complaint behind the individual complaints is "why doesn't Homebrew work the way I want it to". That's a reasonable complaint and there may be better tools for you to use rather than Homebrew but we're going to continue to build Homebrew to work in a way that works best for the majority of our users and avoids our (entirely voluntary) maintainers having to deal with the same issues again and again.
This is not true. `man brew` documents how to disable auto-updates.
> Why exactly is this kind of purist decision being imposed on users?
Because there's a lot of current debate about whether things being relicensed under the Commons Clause is "open source" or not and we'd rather lean on an established, well-documented authority on what that definition is. Additionally, part of our agreed membership of the Software Freedom Conservancy (who help us with a lot) involves ensuring our core formulae are open source software.
> So you're going to periodically make maintenance changes to my system without my permission. Goodie.
Again, `brew man` documents how to disable this.
---
The complaint behind the individual complaints is "why doesn't Homebrew work the way I want it to". That's a reasonable complaint and there may be better tools for you to use rather than Homebrew but we're going to continue to build Homebrew to work in a way that works best for the majority of our users and avoids our (entirely voluntary) maintainers having to deal with the same issues again and again.
As someone who uses homebrew religiously, but isn't familiar with the inner-workings, I'll second the sentiment that some whys on the changes would be much appreciated.
Otherwise it just looks like a bunch of scary behavior changes w/o a clear upside.
Otherwise it just looks like a bunch of scary behavior changes w/o a clear upside.
that's a really good suggestion. As someone that writes a lot of changelogs, they would really benefit from including _why_ and not just _what_.
I do think, however, that users greatly underestimate how much time it takes to write up a good changelog. Even one that is as simple as this.
I do think, however, that users greatly underestimate how much time it takes to write up a good changelog. Even one that is as simple as this.
Yeh, that's it in a nutshell, unfortunately. I wish I had more time to write great changelogs for Homebrew but I don't :(
Not for me to tell you how to prioritize your time, of course, but I'll just say in my experience programmers tend to vastly over-index on the importance of stuff like crushing code and under-index on the importance of stuff like communicating w/ the community. Non-tech tasks are still hugely important.
Cheers, and thanks for your work on homebrew. It really is a great tool.
Cheers, and thanks for your work on homebrew. It really is a great tool.
Thanks for the kind words.
For what it's worth, I really do value this stuff which is why we have release notes at all, a Discourse forum, etc. most of which I've setup and run :)
For what it's worth, I really do value this stuff which is why we have release notes at all, a Discourse forum, etc. most of which I've setup and run :)
I'd rather you write the code. Keep up the good work.
Hopefully these changes will allow you more time to focus on adding value to the vast majority of users. If a small subset of folks need the marginal functionality, it should be up to them to step up to the plate and make it happen.
Hopefully these changes will allow you more time to focus on adding value to the vast majority of users. If a small subset of folks need the marginal functionality, it should be up to them to step up to the plate and make it happen.
Thanks for the kind words.
> Hopefully these changes will allow you more time to focus on adding value to the vast majority of users. If a small subset of folks need the marginal functionality, it should be up to them to step up to the plate and make it happen.
100% agree.
> Hopefully these changes will allow you more time to focus on adding value to the vast majority of users. If a small subset of folks need the marginal functionality, it should be up to them to step up to the plate and make it happen.
100% agree.
Re auto-updating: thanks for that, which is something.
For those who haven't had a chance to dig through, prompted by this, I searched through the man page [1] on GitHub and apparently the environment variable to set is:
HOMEBREW_NO_AUTO_UPDATE
https://github.com/Homebrew/brew/blob/master/docs/Manpage.md
For those who haven't had a chance to dig through, prompted by this, I searched through the man page [1] on GitHub and apparently the environment variable to set is:
HOMEBREW_NO_AUTO_UPDATE
https://github.com/Homebrew/brew/blob/master/docs/Manpage.md
There's also `HOMEBREW_AUTO_UPDATE_SECS` (which I personally set) so you can still auto-update but do so a little bit less often.
Save your time and soul. Just use MacPorts.
Good job! Homebrew is, at least for me, essential for using macOS as a true *nix, and most of the changes (implemented or proposed) seem good (e.g., the ability to automatically run cleanup)
I am a little bit worried for the removal of all options that is planned for the 2.0.0 version: I have some formulae installed from source (with additional options) and I like the flexibility that they give. Hopefully the migration to an "options-free" Homebrew will be smooth and some alternative when options are needed (not requiring a lot of third-party taps) will emerge.
I am a little bit worried for the removal of all options that is planned for the 2.0.0 version: I have some formulae installed from source (with additional options) and I like the flexibility that they give. Hopefully the migration to an "options-free" Homebrew will be smooth and some alternative when options are needed (not requiring a lot of third-party taps) will emerge.
Have you tried Macports?
I'm seriously considering it. I have several formula installed with options and I really don't want to have to go through every formula I currently have installed and see if they decided to "bless" the options I need/want.
You can defer that until the option is actually removed. You’ll get an error on the next upgrade. You can then reinstall without the option and decide if that’s still what you want.
If that’s not what you want, you’re free to dig up the formula with the option, and continue to maintain it on your own tap. There’s good documentation for that [1], and there are always people in our forums willing to help if you get stuck [2].
[1]: https://docs.brew.sh/How-to-Create-and-Maintain-a-Tap [2]: https://discourse.brew.sh
If that’s not what you want, you’re free to dig up the formula with the option, and continue to maintain it on your own tap. There’s good documentation for that [1], and there are always people in our forums willing to help if you get stuck [2].
[1]: https://docs.brew.sh/How-to-Create-and-Maintain-a-Tap [2]: https://discourse.brew.sh
Editing ports to add options is pretty easy and they are willing to help.
We're using analytics data to ensure that the most commonly used options (or those that don't add extra dependencies) are enabled by default. We're doing our best to make it a smooth migration (and given we're 95% of the way there already and you hopefully haven't noticed problems: I'm confident).
Is there any easy way to list all installed packages for which I've customised options, and what the options I've used are?
Could be better, but this is what I did:
for i in `brew ls`; do brew info $i | grep -e 'Built.*with:' >/dev/null && brew info $i; done
No need to feel dependent on Homebrew maintainers’ whims. You’re free to copy the formula, customize it the way you want, push it to your repo on GitHub, and `brew tap` it.
No need to use the Cask version if you don’t want to. If you feel stuck with the tap thing, you’re welcome to get in touch on discourse.brew.sh and we’re glad to help!
No need to use the Cask version if you don’t want to. If you feel stuck with the tap thing, you’re welcome to get in touch on discourse.brew.sh and we’re glad to help!
> I have some formulae installed from source
Homebrew has gone out of its way to make this more difficult lately, to the point where 'brew doctor' complains when HOMEBREW_BUILD_FROM_SOURCE is set, and now it's deprecated.
Homebrew's bottle infrastructure is a tempting target if you want to get malware deployed to Macs, and as best I can tell, there is scant documentation out there about what they do to secure it. For all I know, the bottles are built on some random bozo's Mac.
The snarky side of me thinks that if build from source is discouraged then the program should rename itself to Macrobrew (as I'm sure Budwiser would sue their pants off if they used that name).
Homebrew has gone out of its way to make this more difficult lately, to the point where 'brew doctor' complains when HOMEBREW_BUILD_FROM_SOURCE is set, and now it's deprecated.
Homebrew's bottle infrastructure is a tempting target if you want to get malware deployed to Macs, and as best I can tell, there is scant documentation out there about what they do to secure it. For all I know, the bottles are built on some random bozo's Mac.
The snarky side of me thinks that if build from source is discouraged then the program should rename itself to Macrobrew (as I'm sure Budwiser would sue their pants off if they used that name).
> there is scant documentation out there about what they do to secure it
We have private operations documentation and have had private security reviews of our infrastructure. We are actively moving away from self-hosted infrastructure this year.
We have private operations documentation and have had private security reviews of our infrastructure. We are actively moving away from self-hosted infrastructure this year.
Do you have published security reviews of your infrastructure? Otherwise you're asking us to just trust you.
Note: I'm not denying that you have a stellar track record, but it would be nice to have more evidence that this is not down to luck or obscurity but is instead due to process, best practice, and understanding of risk.
Note: I'm not denying that you have a stellar track record, but it would be nice to have more evidence that this is not down to luck or obscurity but is instead due to process, best practice, and understanding of risk.
Almost everything we have is open for you to examine (including the code, obviously). If you're not willing or able to do that: yes, you have to trust us (like you do with pretty much all software and infrastructure you don't personally control).
Yes, I know that I have to make a judgement call about whether or not to trust infrastructure I don't own, and I have made that judgement, but I'm asking for myself and others if the Homebrew project haa any additional information I and others can use to make that evaluation.
It is frustrating that you are being so opaque and dismissive. You still haven't answered my earlier question if the Homebrew project has published a summary of the results of these security reviews (I understand not posting the entire review publicly). A quick Google search did not turn up anything, which is why I am asking.
For contrast: I can find out the build process for a Debian package from their website[1]. While they do have some private operation documentation, they also publish the process by which packages get pulled into their system, built, and pushed to the mirrors[2][3]. They have documentation for how to replicate their build environment and build packages on my own[4]. This documentation is open, and I can verify packages with it as they move toward reproducible builds[5]. I understand that Debian is a much larger operation with a much longer history. I understand that it takes time to develop these things. This is not an attack on the Homebrew project or the work they do.
[1] https://www.debian.org/devel/buildd/
[2] https://www.debian.org/devel/buildd/operation
[3] https://wiki.debian.org/Teams/FTPMaster
[4] https://wiki.debian.org/BuilddSetup
[5] https://wiki.debian.org/ReproducibleBuilds
It is frustrating that you are being so opaque and dismissive. You still haven't answered my earlier question if the Homebrew project has published a summary of the results of these security reviews (I understand not posting the entire review publicly). A quick Google search did not turn up anything, which is why I am asking.
For contrast: I can find out the build process for a Debian package from their website[1]. While they do have some private operation documentation, they also publish the process by which packages get pulled into their system, built, and pushed to the mirrors[2][3]. They have documentation for how to replicate their build environment and build packages on my own[4]. This documentation is open, and I can verify packages with it as they move toward reproducible builds[5]. I understand that Debian is a much larger operation with a much longer history. I understand that it takes time to develop these things. This is not an attack on the Homebrew project or the work they do.
[1] https://www.debian.org/devel/buildd/
[2] https://www.debian.org/devel/buildd/operation
[3] https://wiki.debian.org/Teams/FTPMaster
[4] https://wiki.debian.org/BuilddSetup
[5] https://wiki.debian.org/ReproducibleBuilds
It is not my intention to be dismissive. I admire the work Debian has done on the above but they are a project with many more maintainers and a lot more resources. We have some documentation for our process already under https://docs.brew.sh but ultimately if it's not good enough for you we need people who are willing to step up and do the work to do so.
I understand you do not intend to be dismissive, however, not answering a simple yes or no question in your last two responses appears to be dismissive. I could not find any information about the security audits performed and a summary of their results, so I ask again; is there a summary of the security audits published publicly?
I have looked at the Homebrew docs page. There is a document linked that describes how bottles are built[1], but it's not clear who has access to it and what safeguards are in place to prevent a malicious maintainer from spreading malware through it (such as who reviews the commits) and it doesn't list the precautions taken by bintray to prevent and detect tampering with packages (and a user has to trust that they as in place, sufficient, and trust bintray to not tamper with them). Another page says that most formula pull requests need to be reviewer but does not go over what this entails[2].
This alarming text, however, does appear in your maintainer guidelines [3]:
>Verify the formula works if possible. If you can’t tell (e.g. if it’s a library) trust the original contributor, it worked for them, so chances are it is fine. If you aren’t an expert in the tool in question, you can’t really gauge if the formula installed the program correctly. _At some point an expert will come along, cry blue murder that it doesn’t work, and fix it. This is how open source works._ Ideally, request a test do block to test that functionality is consistently available.
Is there a set of maintainers who handle security sensitive formula, like openssl, gnupg, and tor?
[1] https://docs.brew.sh/Brew-Test-Bot
[2] https://docs.brew.sh/How-To-Open-a-Homebrew-Pull-Request
[3] https://docs.brew.sh/Maintainer-Guidelines
I have looked at the Homebrew docs page. There is a document linked that describes how bottles are built[1], but it's not clear who has access to it and what safeguards are in place to prevent a malicious maintainer from spreading malware through it (such as who reviews the commits) and it doesn't list the precautions taken by bintray to prevent and detect tampering with packages (and a user has to trust that they as in place, sufficient, and trust bintray to not tamper with them). Another page says that most formula pull requests need to be reviewer but does not go over what this entails[2].
This alarming text, however, does appear in your maintainer guidelines [3]:
>Verify the formula works if possible. If you can’t tell (e.g. if it’s a library) trust the original contributor, it worked for them, so chances are it is fine. If you aren’t an expert in the tool in question, you can’t really gauge if the formula installed the program correctly. _At some point an expert will come along, cry blue murder that it doesn’t work, and fix it. This is how open source works._ Ideally, request a test do block to test that functionality is consistently available.
Is there a set of maintainers who handle security sensitive formula, like openssl, gnupg, and tor?
[1] https://docs.brew.sh/Brew-Test-Bot
[2] https://docs.brew.sh/How-To-Open-a-Homebrew-Pull-Request
[3] https://docs.brew.sh/Maintainer-Guidelines
> not answering a simple yes or no question
Here you go:
> Do you have published security reviews of your infrastructure?
Beyond https://hackerone.com/homebrew/hacktivity, https://brew.sh/2018/08/05/security-incident-disclosure/: no.
> Otherwise you're asking us to just trust you.
Yes.
> it's not clear who has access to it
A subset of the documented Homebrew maintainers who manage our ops/infrastructure.
> such as who reviews the commits
The maintainers and additional trusted users.
> the precautions taken by bintray to prevent and detect tampering with packages
packages require a manual publishing step, are checksummed and frozen on their CDN a while after publishing.
> most formula pull requests need to be reviewer but does not go over what this entails
This is part of the on boarding and training of new maintainers.
> Is there a set of maintainers who handle security sensitive formula, like openssl, gnupg, and tor?
Not a specific set but they receive extra time, review and attention to their source URLs.
---
I expect some of that will not be good enough for you in which case: please either get involved to help us make it better (we are unpaid volunteers) or use another tool. This is how open source works.
Here you go:
> Do you have published security reviews of your infrastructure?
Beyond https://hackerone.com/homebrew/hacktivity, https://brew.sh/2018/08/05/security-incident-disclosure/: no.
> Otherwise you're asking us to just trust you.
Yes.
> it's not clear who has access to it
A subset of the documented Homebrew maintainers who manage our ops/infrastructure.
> such as who reviews the commits
The maintainers and additional trusted users.
> the precautions taken by bintray to prevent and detect tampering with packages
packages require a manual publishing step, are checksummed and frozen on their CDN a while after publishing.
> most formula pull requests need to be reviewer but does not go over what this entails
This is part of the on boarding and training of new maintainers.
> Is there a set of maintainers who handle security sensitive formula, like openssl, gnupg, and tor?
Not a specific set but they receive extra time, review and attention to their source URLs.
---
I expect some of that will not be good enough for you in which case: please either get involved to help us make it better (we are unpaid volunteers) or use another tool. This is how open source works.
I had exactly the same security concern. It seems it would be quite easy to slip in an attack via brew.
I had an experience years ago where I discovered the lack of quality control: a typo (presumably) caused a package to regress back a couple of years. My issue used wording like "Are you serious?" which was taken as offensive and I was banned, unable to even apologize or defend myself from resulting attacks.
brew is convenient and I still use it, but I have a much higher degree of trust in packages shipping with OS distributions.
I had an experience years ago where I discovered the lack of quality control: a typo (presumably) caused a package to regress back a couple of years. My issue used wording like "Are you serious?" which was taken as offensive and I was banned, unable to even apologize or defend myself from resulting attacks.
brew is convenient and I still use it, but I have a much higher degree of trust in packages shipping with OS distributions.
> It seems it would be quite easy to slip in an attack via brew.
You can put your money where your mouth is and consider submitting one to our HackerOne. Otherwise: perhaps it's not "quite easy" after all.
You can put your money where your mouth is and consider submitting one to our HackerOne. Otherwise: perhaps it's not "quite easy" after all.
I wonder why Linux users would want to use Homebrew? Are the packages not available via the normal Linux package managers?
Consider someone writing a guide on how to install packages on multiple platforms. You only need one guide and not three based on brew. Especially interesting for Windows 10 IMO.
Generally speaking all that documentation should consist of is "click this link to download <thing> for <os>" and maybe "extract its contents somewhere". Anything else is over-complicating the situation for no good reason. Unfortunately that has become such an ingrained part of the open source culture that we keep imagining things like this are a good idea.
Homebrew on Linux (known as Linuxbrew) does not require root access.
Of course, Nix also works without root and has many other benefits ;).
Of course, Nix also works without root and has many other benefits ;).
However, Nix requires you to create a /nix directory, which is not possible in some cases.
It doesn't require it:
- You can create the nix store in some directory you own. But this entails recompiling every package, because the precompiled binaries have dependencies on the store being in /nix.
- If user namespaces are enabled, you can bind mount /nix to some directory that you own:
https://nixos.wiki/wiki/Nix_Installation_Guide
Of course, it is more convenient if you can convince a sysadmin to install multi-user Nix. From there on, every user can install their own packages (in a non-conflicting manner).
- You can create the nix store in some directory you own. But this entails recompiling every package, because the precompiled binaries have dependencies on the store being in /nix.
- If user namespaces are enabled, you can bind mount /nix to some directory that you own:
https://nixos.wiki/wiki/Nix_Installation_Guide
Of course, it is more convenient if you can convince a sysadmin to install multi-user Nix. From there on, every user can install their own packages (in a non-conflicting manner).
That’s good to know, thanks for sharing!
I don't think recompilation is required. You can use proot (like chroot) to make a local dir look like "/nix" to subprocesses. I've seen it here (but not tested it): https://gist.github.com/mbbx6spp/4f467adb4e0133063fd87e264c6...
Yes, that's a similar approach to the use of user namespaces that I mentioned, which also doesn't require recompilation. I guess proot also works on systems with user namespaces disabled (which is sometimes done for security reasons).
Package managers can only offer what they are feeded with. And that does not include all software in all incarnations. A usual problem is that certain distributions are lacking newer versions. Or the ability to install multiple versions parallel. Something like homebrew is a solution for this.
For many packages it is difficult to get packaged for all Linux distributions. Hence if your package does not get packaged for a particular Linux distribution users can still install your package using homebrew.
For many distributions (redhat, ubuntu lts, debian) packages can grow rather old. If you want something more up-to-date, linuxbrew can help. I suppose that's less useful on rolling distros like gentoo or arch.
I ran into this the other day with ubuntu (on a windows laptop with the linux subsystem) opting for node version 8 instead of 10.
One of those annoying things that is valid from a "lets keep things backwards compatible" point of view that ends you up with the wrong versions of everything you care about as a developer. I fixed it by uninstalling node and then installing nvm and then doing a nvm install 10. Of course having per tool custom hacks around "your distribution comes with the wrong version of everything you care about, 100% guaranteed" gets old pretty quickly. Brew fixes this on Mac.
On most servers on the other hand, you should not be installing developer tools or runtimes at all and use something like docker instead. Installing stuff like node.js, python, ruby, etc. via the default package manager is almost always the wrong thing to do on both development machines and production machines.
Speaking of docker, having something like brew on a lightweight image could be a pretty awesome way of installing what you need without relying on downloading tar balls and other madness that you find in a typical Dockerfile. Why reinvent that wheel?
I haven't come across linux brew. Sounds like a good idea. Merging the two ecosystems sounds like a better one. Having that on windows as well; awesome. Simplifies a lot of stuff developers have to do on different platforms to end up with the same tools.
One of those annoying things that is valid from a "lets keep things backwards compatible" point of view that ends you up with the wrong versions of everything you care about as a developer. I fixed it by uninstalling node and then installing nvm and then doing a nvm install 10. Of course having per tool custom hacks around "your distribution comes with the wrong version of everything you care about, 100% guaranteed" gets old pretty quickly. Brew fixes this on Mac.
On most servers on the other hand, you should not be installing developer tools or runtimes at all and use something like docker instead. Installing stuff like node.js, python, ruby, etc. via the default package manager is almost always the wrong thing to do on both development machines and production machines.
Speaking of docker, having something like brew on a lightweight image could be a pretty awesome way of installing what you need without relying on downloading tar balls and other madness that you find in a typical Dockerfile. Why reinvent that wheel?
I haven't come across linux brew. Sounds like a good idea. Merging the two ecosystems sounds like a better one. Having that on windows as well; awesome. Simplifies a lot of stuff developers have to do on different platforms to end up with the same tools.
I think that's a good example of how the whole paradigm of package management as an application distribution mechanism (at least as implemented) is really terrible. What good is all this management if I have to bypass it and everything it supposedly provides just to get up to date software, or two versions of the same software?
I totally agree with you. I think it's pretty pointless to have terabytes of repos for software that then grows stale.
But distributions often say that it's the only way to keep the whole "ecosystem" of a distribution stable.
So? What I think is that Linux "distributions" are growing less and less useful nowadays. On the desktop, no Linux was able to gain significant traction, and many developers are just using Macs and Windows with WSL.
On the server side, ways to "build your own distro" for creating a tuned stack for your purposes ( read: docker images ) exist and seem more useful than standard distro components. Nixos is another take.
Yes, "just apt install foo" is nice, until it isn't. I think we're at a tipping point between the old and new approach.
But distributions often say that it's the only way to keep the whole "ecosystem" of a distribution stable.
So? What I think is that Linux "distributions" are growing less and less useful nowadays. On the desktop, no Linux was able to gain significant traction, and many developers are just using Macs and Windows with WSL.
On the server side, ways to "build your own distro" for creating a tuned stack for your purposes ( read: docker images ) exist and seem more useful than standard distro components. Nixos is another take.
Yes, "just apt install foo" is nice, until it isn't. I think we're at a tipping point between the old and new approach.
> But distributions often say that it's the only way to keep the whole "ecosystem" of a distribution stable.
In practice, it doesn't work. I did a Debian Stretch `apt-get upgrade` and it pulled in a new version of firefox-esr that did not run on my Rockchip Chromebook. Debian was forced to do it because it was end-of-life for the older firefox-esr, and Debian's security backports policy doesn't scale for projects like Chromium and Firefox.
In practice, it doesn't work. I did a Debian Stretch `apt-get upgrade` and it pulled in a new version of firefox-esr that did not run on my Rockchip Chromebook. Debian was forced to do it because it was end-of-life for the older firefox-esr, and Debian's security backports policy doesn't scale for projects like Chromium and Firefox.
I think this is a testament of Homebrew its success. Build a package manager for a OS that doesn't natively has one (macOS) and eventually expand to OS's that do have one (linux), because users request it / like Homebrew.
Chrome OS. Gentoo-based, but no package management available.
But not Linuxbrew, Chromebrew[1]!
1: https://skycocker.github.io/chromebrew/
1: https://skycocker.github.io/chromebrew/
Some people don't have root access.
Homebrew can be used as a package manager for C/C++ projects development and meanwhile doesn't mess up the system.
There are many small utilities that's not available especially if you stick to LTS. (Even ripgrep isn't available on 18.04. I feel this more when these 'remake better alternatives with rust/go' movement started to occur.)
And all you can do is hopefully find a trustworthy PPA that gets timely updates or you end up downloading it yourself and get them updated manaually.
I wanted to use Linuxbrew but this seems still not too mature.
And all you can do is hopefully find a trustworthy PPA that gets timely updates or you end up downloading it yourself and get them updated manaually.
I wanted to use Linuxbrew but this seems still not too mature.
Really not a fan of the removal of options, imo it is one of the greatest perks of homebrew. Can I manually build my own software? Yes. Do I want to have to maintain a tap for every single thing I want to use options on? No, I do not.
This move along with the python2 vs python3 change that was made previously has left a seriously sour taste in my mouth about Homebrew. I'm actively looking at other options.
This move along with the python2 vs python3 change that was made previously has left a seriously sour taste in my mouth about Homebrew. I'm actively looking at other options.
It’s less than 12 months until Python 2 will no longer be maintained. https://pythonclock.org/
I really encourage you to create a tap for the formulas you like to customize. Taps are a powerful, much overlooked feature baked into Homebrew.
I really encourage you to create a tap for the formulas you like to customize. Taps are a powerful, much overlooked feature baked into Homebrew.
My objection is more about how it was handled on a communication and technical level than any of the specifics of switching to Python 3 over 2.
> Homebrew 1.9.0 no longer runs on 32-bit Intel CPUs.
Where does this constraint come from? Homebrew is written in Ruby.
To cut down on compilation from source issues? Or Homebrew uses language features from Ruby versions greater than what Apple shipped in their latest 32-bit supported OSX versions (as homebrew insists on using the system default binaries)?
Where does this constraint come from? Homebrew is written in Ruby.
To cut down on compilation from source issues? Or Homebrew uses language features from Ruby versions greater than what Apple shipped in their latest 32-bit supported OSX versions (as homebrew insists on using the system default binaries)?
From: https://docs.brew.sh/Linuxbrew
> Linuxbrew does not currently support 32-bit x86 platforms. It would be possible for Linuxbrew to work on 32-bit x86 platforms with some effort. An interested and dedicated person could maintain a fork of Homebrew to develop support for 32-bit x86.
> Linuxbrew does not currently support 32-bit x86 platforms. It would be possible for Linuxbrew to work on 32-bit x86 platforms with some effort. An interested and dedicated person could maintain a fork of Homebrew to develop support for 32-bit x86.
Homebrew builds individual binary bottles per package and per macOS version, but not per target architecture.
In other words: at bottle creation time (server-side), Homebrew tells the compiler to output code that runs on older CPUs. The downside of this is that end users who install bottles never get to enjoy the benefits of latest CPU instruction sets, such as SSE 4.1/4.2.
From time to time, Homebrew raises the bar to keep the balance between performance and compatibility. From Homebrew 1.9.0, bottles are built to use the features of the Penryn/Core 2 microarchitecture or higher, or in other words: Homebrew no longer supports CPU microarchitectures older than Penryn/Core 2.
In other words: at bottle creation time (server-side), Homebrew tells the compiler to output code that runs on older CPUs. The downside of this is that end users who install bottles never get to enjoy the benefits of latest CPU instruction sets, such as SSE 4.1/4.2.
From time to time, Homebrew raises the bar to keep the balance between performance and compatibility. From Homebrew 1.9.0, bottles are built to use the features of the Penryn/Core 2 microarchitecture or higher, or in other words: Homebrew no longer supports CPU microarchitectures older than Penryn/Core 2.
Your explanation makes sense for bottles, not for Homebrew itself. Homebrew is still at its core "build from source".
This is exactly what the bottle bot does, building the same formulas you have on your machine. Why should this no longer work?
This is exactly what the bottle bot does, building the same formulas you have on your machine. Why should this no longer work?
Unless the user sets specific build options, or runs `brew install --build-from-source`, Homebrew installs core packages from a bottle hosted on Bintray by default.
While I agree in that building from source is somewhat at Homebrew’s core (as that’s what happens at bottle creation time), the intention is to deliver core packages to end users via bottle as often as possible.
While I agree in that building from source is somewhat at Homebrew’s core (as that’s what happens at bottle creation time), the intention is to deliver core packages to end users via bottle as often as possible.
It's not so much a constraint as something that has enabled us to cleanup code used only on extremely old Macs.
Windows support! That's a surprise.
Chocolatey is... ok. It works, mostly, but it's a common occurrence for upgrades to fail because the package maintainer forgot to update the hash, or the program's built-in autoupdater already ran and moved things around, or a phantom .install dependency is missing, or some other inscrutable problem. Competition will be good for it (and the developer sells a premium version, so I do expect proper competition). All the other package management tools on windows have fairly tiny repositories and are, as far as I can tell, optimized for the workflow of sysadmins building images.
I wonder if Brew has any plans to support that package management framework Microsoft is apparently still working on? (https://github.com/OneGet/oneget)
Chocolatey is... ok. It works, mostly, but it's a common occurrence for upgrades to fail because the package maintainer forgot to update the hash, or the program's built-in autoupdater already ran and moved things around, or a phantom .install dependency is missing, or some other inscrutable problem. Competition will be good for it (and the developer sells a premium version, so I do expect proper competition). All the other package management tools on windows have fairly tiny repositories and are, as far as I can tell, optimized for the workflow of sysadmins building images.
I wonder if Brew has any plans to support that package management framework Microsoft is apparently still working on? (https://github.com/OneGet/oneget)
Did you take a look at scoop.sh? I find it very useful commpared to chocolatey. It only installs portable versions, requires no admin and has everything my dev heart needs. Even installing stuff like miniconda and fluttr is a blaze.
Seconded. Scoop.sh is spectacularly simple and to the point for all the areas where chocolatey feels needlessly complex.
It's not windows support, it's WSL support.
[deleted]
> brew link --force will not link software already provided by macOS.
An explanation for this change would be useful.
An explanation for this change would be useful.
If I had to guess. Probably to prevent tainting the base system.
It depends on the defintion of taint. Homebrew cannot touch the base system, since it lives in /usr/local in macOS. Also macOS SIP prevents modification of most system files.
One major problem is that system and Homebrew versions of library may have incompatible ABIs.
One major problem is that system and Homebrew versions of library may have incompatible ABIs.
That's the exact reason why I consider MacPorts much more elegant.
Most likely. I still remember the incident where I upgraded my system wide python installation manually and had to reinstall mac os
From https://github.com/Homebrew/brew/pull/5383:
> This would have avoided Homebrew/homebrew-core#34754
which links to: https://github.com/Homebrew/homebrew-core/issues/34754
tl;dr One user had binutils force-linked, causing cmake to pick up the user-installed flavor of binutils, altering cmake’s behavior and causing issues.
> This would have avoided Homebrew/homebrew-core#34754
which links to: https://github.com/Homebrew/homebrew-core/issues/34754
tl;dr One user had binutils force-linked, causing cmake to pick up the user-installed flavor of binutils, altering cmake’s behavior and causing issues.
Yup, this exactly.
It is not a good idea to change the provided software versions that other parts of the base system can rely on. I would go further than that, modern OS design should make every part of the operating system immutable by default. We could get rid off entire class of security problems if it was like that.
How php is handled by Homebrew has changed last year. Now you do
brew install [email protected]
It was my understanding that if you wanted in your PATH you just did `brew link --force [email protected]`. That's not possible anymore? (Since MacOS provides a php binary) What's the recommended way for easily getting a specific version of php in your PATH?
It was my understanding that if you wanted in your PATH you just did `brew link --force [email protected]`. That's not possible anymore? (Since MacOS provides a php binary) What's the recommended way for easily getting a specific version of php in your PATH?
I've never had to use --force to change my php version. You probably just need to change your PATH so it finds the homebrew version before the system version.
Remember that Homebrew by default records some of your usage data via Google Analytics. Disable that if you are not comfortable, or switch to MacPorts.
Technically speaking, why would anyone wants to use brew on linux over snap or nix?
My saviour when forced to use a Mac recently has been Nix. Homebrew has only ever caused me pain.
It's a bit weird how they're announcing a new mission statement about how this is a small focused project for macOS at the same time they're announcing Windows and Linux compatibility.
Hi. I am one of the Linuxbrew maintainers. I understand that this may seem contradictory.
Linuxbrew has existed since a few years now (https://github.com/Linuxbrew). We just merged the linux part of the code into the main brew repo.
https://github.com/Linuxbrew/homebrew-core, where the formula reside, is still a separate project. But we all work closely together. The 4-5 linuxbrew maintainers joined the homebrew team and we are now a single team.
This does not add too much work for the initial homebrew maintainers.
The scope has grown a little bit, but nothing to be worried about. And the windows part is "just" Linux running on Windows, so actually there is not much difference there. That part is not so well tested though, because we do not have so many users on Windows, but we hope to be able to improve this part if needed.
On the other side some decisions have been made lately: less from source builds, less options ... this will reduce our workload too.
Linuxbrew has existed since a few years now (https://github.com/Linuxbrew). We just merged the linux part of the code into the main brew repo.
https://github.com/Linuxbrew/homebrew-core, where the formula reside, is still a separate project. But we all work closely together. The 4-5 linuxbrew maintainers joined the homebrew team and we are now a single team.
This does not add too much work for the initial homebrew maintainers.
The scope has grown a little bit, but nothing to be worried about. And the windows part is "just" Linux running on Windows, so actually there is not much difference there. That part is not so well tested though, because we do not have so many users on Windows, but we hope to be able to improve this part if needed.
On the other side some decisions have been made lately: less from source builds, less options ... this will reduce our workload too.
If the scope has grown and you're all a single team, shouldn't that be reflected in the mission statement?
While I’m as excited about the Linux part as my fellow maintainers are, I feel it’s important to set reasonable expectations.
Currently, about 25 % of Homebrew’s active maintainers work on the Linux/WSL part. I don’t believe that ratio is going to change much during 2019.
If Linux and Windows were to become part of Homebrew’s mission statement, it’d be all too easy for Linux/WSL users to interpret that as Homebrew committing to the same level of support to which macOS users are accustomed at this time.
I feel there’s no way we’d be able to deliver on that commitment and not disappoint users. We want happy users.
Currently, about 25 % of Homebrew’s active maintainers work on the Linux/WSL part. I don’t believe that ratio is going to change much during 2019.
If Linux and Windows were to become part of Homebrew’s mission statement, it’d be all too easy for Linux/WSL users to interpret that as Homebrew committing to the same level of support to which macOS users are accustomed at this time.
I feel there’s no way we’d be able to deliver on that commitment and not disappoint users. We want happy users.
Is it me or homebrew became slower recently?
I recall that some commands where much faster. It takes on average 2.5 seconds to do brew search.
$ time brew search mpv ==> Formulae mpv
==> Casks mpv brew search mpv 2.34s user 0.59s system 67% cpu 4.348 total
This is not autoupdate — this is with set HOMEBREW_AUTO_UPDATE_SECS to 14400.
I recall that some commands where much faster. It takes on average 2.5 seconds to do brew search.
$ time brew search mpv ==> Formulae mpv
==> Casks mpv brew search mpv 2.34s user 0.59s system 67% cpu 4.348 total
This is not autoupdate — this is with set HOMEBREW_AUTO_UPDATE_SECS to 14400.
It loads all "casks" when doing search. I don't know if it did it in earlier version, but now `brew search` takes more than minute on my system (mostly because of apfs on hdd)
https://discourse.brew.sh/t/solved-brew-search-is-too-slow-s...
https://discourse.brew.sh/t/solved-brew-search-is-too-slow-s...
That explains everything... Thanks.
Since I don't install 'mission critical' application through brew, I've always had the following command run daily though a cronjob: