Orphaning bcachefs-tools in Debian(jonathancarter.org)
jonathancarter.org
Orphaning bcachefs-tools in Debian
https://jonathancarter.org/2024/08/29/orphaning-bcachefs-tools-in-debian/
223 comments
Hey at least it's not the worst behavior we've seen from a Linux file system creator...
I thought Carl Thompson's response was very good and constructive: https://lore.kernel.org/lkml/1816164937.417.1724473375169@ma...
What I don't understand is that IIUC Kent has his development git history well broken up into small tight commits. But he seems to be sending the Linux maintainers patches that are much larger than they want. I don't get why he doesn't take the feedback and work with them to send smaller patches.
EDIT: The culture at Google (where Kent used to work) was small patches, although that did vary by team. At Google you have fleet-wide control and can roll back changes that looked good in testing but worked out poorly in production. You can't do that across all organizations or people who have installed bcachefs. Carl pointed out that Kent seemed to be missing some social aspects, but I feel like he's also not fully appreciating the technical aspects behind why the process is the way it is.
I thought Carl Thompson's response was very good and constructive: https://lore.kernel.org/lkml/1816164937.417.1724473375169@ma...
What I don't understand is that IIUC Kent has his development git history well broken up into small tight commits. But he seems to be sending the Linux maintainers patches that are much larger than they want. I don't get why he doesn't take the feedback and work with them to send smaller patches.
EDIT: The culture at Google (where Kent used to work) was small patches, although that did vary by team. At Google you have fleet-wide control and can roll back changes that looked good in testing but worked out poorly in production. You can't do that across all organizations or people who have installed bcachefs. Carl pointed out that Kent seemed to be missing some social aspects, but I feel like he's also not fully appreciating the technical aspects behind why the process is the way it is.
Honesty, I think I just presented that pull request badly.
I included the rcu_pending and vfs inode rhashtable conversion because I was getting user reports that it fixed issues that were seriously affecting system usability, and because they were algorithmically simple and well tested.
Back in the day, on multiple occasions Linus and others were rewriting core mm code in RC kernels; bcachefs is still experimental, so stuff like this should still be somewhat expected.
I included the rcu_pending and vfs inode rhashtable conversion because I was getting user reports that it fixed issues that were seriously affecting system usability, and because they were algorithmically simple and well tested.
Back in the day, on multiple occasions Linus and others were rewriting core mm code in RC kernels; bcachefs is still experimental, so stuff like this should still be somewhat expected.
> bcachefs is still experimental, so stuff like this should still be somewhat expected.
I really think you need to realign your expectations here. The Linux kernel is in a different place now than "back in the day" and you are not Linus Torvalds.
That PR would have been better off had it been split into multiple ones and timed differently.
I really think you need to realign your expectations here. The Linux kernel is in a different place now than "back in the day" and you are not Linus Torvalds.
That PR would have been better off had it been split into multiple ones and timed differently.
They Kent. I love your work and I have succesfully used bcachefs in my main workstation since 6.7. I also happily donate monthly in patreon, which I do rarely if ever.
Hope you don't get too much into trouble with Linus. I do not want to see you or the project get into the wrong side of the old guard...
Hope you don't get too much into trouble with Linus. I do not want to see you or the project get into the wrong side of the old guard...
I second this. Please keep pushing through and don't let the peanut gallery get to you. bcachefs is our only realistic chance to bring our filesystem game into the 21st century, given that the little hope we might have had in Oracle has not been realized.
Who's the "old guard" - Linus Torvalds?
Yeah I see where you're coming from. By the way, I only heard of bcachefs yesterday and I watched a great video where you were presenting about it. I'm excited about the file system and it's super cool to hear from you!
Likewise!
> Hey at least it's not the worst behavior we've seen from a Linux file system creator...
I think that dubious distinction would go to Hans Reiser.
I think that dubious distinction would go to Hans Reiser.
[deleted]
It's not about the size of each individual patches but about the large amount of changes in total *during the freeze•.
It's very clear from that thread that he doesn't understand the purpose of the stable branch. It doesn't mean "stable" as in "the best possible experience", it means it as in "this code has been tested for a long period of time with no serious defects found" so that when the stable branch is promoted to release, everything has undergone a long testing period by a broad user base.
If there is a defect found, the change to a stable branch should literally be the minimal code change to fix the reported issue. Ideally, if it's a newly introduced issue (i.e. since being on the stable branch), the problematic code reverted and a different fix to the original defect applied instead (or left if it's deemed less of an issue than taking another speculative fix). Anything that requires a re-organisation of code, by definition, isn't a minimal fix. Maybe it's the correct long-term solution, but that can be done on the unstable branch, but for the stable branch, the best fix is the simplest work around. If there isn't a simple work around, the best fix is to revert everything back to the previous stable version and keep iterating on the unstable branch.
The guy even admits it as well with his repeated "please don't actually use this in production" style messages - it's hard to give a greater indication than this that the code isn't yet ready for stable.
I can understand why from his perspective he wants his changes in the hands of users as soon as possible - it's something he's poured his heart and soul and he strongly believes it will improve his users' experience. It's also the case that he is happy running the very latest and probably has more confidence in it that an older version. The rational choice from his perspective is to always use the latest code. But, discounting the extremely unlikely situation that his code is entirely bug free, that just means he hasn't yet found the next serious bug. If a big code change is rushed out into the stable branch, it just increases the likelihood that any serious bug won't have the time it needs in testing to have the confidence that's the branch is suitable for promotion to release.
If there is a defect found, the change to a stable branch should literally be the minimal code change to fix the reported issue. Ideally, if it's a newly introduced issue (i.e. since being on the stable branch), the problematic code reverted and a different fix to the original defect applied instead (or left if it's deemed less of an issue than taking another speculative fix). Anything that requires a re-organisation of code, by definition, isn't a minimal fix. Maybe it's the correct long-term solution, but that can be done on the unstable branch, but for the stable branch, the best fix is the simplest work around. If there isn't a simple work around, the best fix is to revert everything back to the previous stable version and keep iterating on the unstable branch.
The guy even admits it as well with his repeated "please don't actually use this in production" style messages - it's hard to give a greater indication than this that the code isn't yet ready for stable.
I can understand why from his perspective he wants his changes in the hands of users as soon as possible - it's something he's poured his heart and soul and he strongly believes it will improve his users' experience. It's also the case that he is happy running the very latest and probably has more confidence in it that an older version. The rational choice from his perspective is to always use the latest code. But, discounting the extremely unlikely situation that his code is entirely bug free, that just means he hasn't yet found the next serious bug. If a big code change is rushed out into the stable branch, it just increases the likelihood that any serious bug won't have the time it needs in testing to have the confidence that's the branch is suitable for promotion to release.
> The guy even admits it as well with his repeated "please don't actually use this in production" style messages - it's hard to give a greater indication than this that the code isn't yet ready for stable.
True that, and yet the kernel has zero issues keeping Btrfs around even though it's been eating people data since 2010. Kent Overstreet sure is naive at times, but I just can't not sneer at the irony that an experimental filesystem is arguably better than a 15-years old one that's been in the Linux kernel for more than a decade.
True that, and yet the kernel has zero issues keeping Btrfs around even though it's been eating people data since 2010. Kent Overstreet sure is naive at times, but I just can't not sneer at the irony that an experimental filesystem is arguably better than a 15-years old one that's been in the Linux kernel for more than a decade.
> True that, and yet the kernel has zero issues keeping Btrfs around even though it's been eating people data since
I can imagine scenarios where known failure modes on an "inferior" tool are better than unknown failure modes on a "superior" one.
I can imagine scenarios where known failure modes on an "inferior" tool are better than unknown failure modes on a "superior" one.
honestly, it's mostly just a matter of trying to give myself the time to work with bugs as people hit them, and stage the rollout. I don't want clueless newbies running it until it's completely bulletproof.
It seems to be a difficult situation: he has bug fixes against the version in the stable kernel for bugs which haven't been reported.
I can see both perspectives: on stable you don't want to do development, but also you want all bugfixes you can get. I can also see the point of Linus, who wants just to add bug fixes and to minimize the risk of introducing new bugs.
Considering that Kent himself warns against general use right now, I don't quite see the urgency to get the bug fixes out - in my understanding Linus would happily merge them in the next development kernel. And whoever is set to to run bcachefs right now, might also be happy to run a dev kernel.
Considering that Kent himself warns against general use right now, I don't quite see the urgency to get the bug fixes out - in my understanding Linus would happily merge them in the next development kernel. And whoever is set to to run bcachefs right now, might also be happy to run a dev kernel.
I agree. If the author himself is telling people it's not ready for production, it doesn't really matter what bugs this code has, unless it affects other subsystems or is a dangerous regression from the previous stable release.
If the bug it's fixing was already in the current release branch, wasn't noticed before and has only shown up now late in the stable branch lifetime, then it definitely doesn't seem like something that needs an urgent fix.
If the bug it's fixing was already in the current release branch, wasn't noticed before and has only shown up now late in the stable branch lifetime, then it definitely doesn't seem like something that needs an urgent fix.
He is not submitting changes for stable. He is submitting non-regression fixes after the merge window. It's clear he understands the rules and the reasons for them but feels like his own internal development process is equivalent at reducing the chance of major regressions introduced in such a PR such that he can simply persuade Linus to let things go through anyway.
Whether this internal process gives him a pass for getting his non-regression fixes in after the merge window is at the end of the day for Linus to decide. And Linus is finally erring on the side of "Please just do what everyone else is doing" rather than "Okay, fine Kent, just this once".
I would say it's ironic to start a comment saying: "It's very clear from that thread that he doesn't understand the purpose of the stable branch" when it's "very clear" from your opening paragraph that you don't understand the thread.
Whether this internal process gives him a pass for getting his non-regression fixes in after the merge window is at the end of the day for Linus to decide. And Linus is finally erring on the side of "Please just do what everyone else is doing" rather than "Okay, fine Kent, just this once".
I would say it's ironic to start a comment saying: "It's very clear from that thread that he doesn't understand the purpose of the stable branch" when it's "very clear" from your opening paragraph that you don't understand the thread.
Perhaps you might enlighten me how it's "very clear" from my opening paragraph that I don't understand the thread. Granted, the initial post could be interpreted a number of different says, but having read the whole thread, I think I have a pretty good understanding of the intent. But clearly, you have a different interpretation, so please - enlighten me to your way of thinking.
Taken at it's most charitable, the opening of the first message "no more bug reports related to the disk accounting rewrite, things are looking good over here as far as regressions go" would suggest a meaning of "there are no significant bugs, the changes below are optional".
The next section in the change description then says that this fixes a number of very serious bugs. Straight away, I can see the potential for an interpretation difference. Is it "heads up, no changes required" or "these fixes are critical"?
He's told "no" by Linus, for reasons that seem to correlate with what I said (unless you'd like to point out in what way I don't understand the thread), and then rather than saying "yeah, then can wait until the next stable branch", he doubled down on the importance of getting these changes in and basically saying that the rules should only apply to everyone else and not him because he knows that there won't be any new bugs because of $REASONS. $REASONS that didn't apply when the bugs were introduced. $REASONS that include automated testing, but that didn't find these bugs originally.
The thread (which apparently I don't understand) contains a perfect summary from Linus himself: "But it doesn't even change the issue: you aren't fixing a regression, you are doing new development to fix some old probl;em, and now you are literally editing non-bcachefs files too."
All this for some changes to a system that he's actively discouraging people from using because it's not production ready anyway, and so none of these bug fixes are actually critical for right now.
It's good he ultimately backs down, but he should never have been pushing for these changes this late in the stable branch timeline anyway.
So, that's my understanding of the thread. I'd be interested to hear how your understanding of the thread is so radically different from that.
Taken at it's most charitable, the opening of the first message "no more bug reports related to the disk accounting rewrite, things are looking good over here as far as regressions go" would suggest a meaning of "there are no significant bugs, the changes below are optional".
The next section in the change description then says that this fixes a number of very serious bugs. Straight away, I can see the potential for an interpretation difference. Is it "heads up, no changes required" or "these fixes are critical"?
He's told "no" by Linus, for reasons that seem to correlate with what I said (unless you'd like to point out in what way I don't understand the thread), and then rather than saying "yeah, then can wait until the next stable branch", he doubled down on the importance of getting these changes in and basically saying that the rules should only apply to everyone else and not him because he knows that there won't be any new bugs because of $REASONS. $REASONS that didn't apply when the bugs were introduced. $REASONS that include automated testing, but that didn't find these bugs originally.
The thread (which apparently I don't understand) contains a perfect summary from Linus himself: "But it doesn't even change the issue: you aren't fixing a regression, you are doing new development to fix some old probl;em, and now you are literally editing non-bcachefs files too."
All this for some changes to a system that he's actively discouraging people from using because it's not production ready anyway, and so none of these bug fixes are actually critical for right now.
It's good he ultimately backs down, but he should never have been pushing for these changes this late in the stable branch timeline anyway.
So, that's my understanding of the thread. I'd be interested to hear how your understanding of the thread is so radically different from that.
> enlighten me
Fundamentally on the whole I don't think most of your interpretation is comment worthy. (To clarify, I don't think its particularly objectionable following from the premise in your opening paragraph.) But...
> in the stable branch timeline
Again. Like I outlined in my initial reply. This has nothing to do with stable. I don't know why you keep talking about stable.
The discussion is about bleeding edge mainline Linux. It's clear to me because:
* It is a PR for Linus. I don't know enough about stable to know if they use PRs but most stable stuff I do know about involves marking patches for stable on the specific mailing lists oriented around stable.
* Linus doesn't handle stable.
* Linus and Kent are taking about the merge window and Kent submitting non-regression fixing patches after it. This doesn't make any sense if it was in a stable context. The process is different.
* If this was stable the discussion would be with GKH.
So, your comment is based on the premise of this being a discussion surrounding stable. It's not, so I don't know what to make of the rest of your comment on the basis of this incorrect premise.
Fundamentally on the whole I don't think most of your interpretation is comment worthy. (To clarify, I don't think its particularly objectionable following from the premise in your opening paragraph.) But...
> in the stable branch timeline
Again. Like I outlined in my initial reply. This has nothing to do with stable. I don't know why you keep talking about stable.
The discussion is about bleeding edge mainline Linux. It's clear to me because:
* It is a PR for Linus. I don't know enough about stable to know if they use PRs but most stable stuff I do know about involves marking patches for stable on the specific mailing lists oriented around stable.
* Linus doesn't handle stable.
* Linus and Kent are taking about the merge window and Kent submitting non-regression fixing patches after it. This doesn't make any sense if it was in a stable context. The process is different.
* If this was stable the discussion would be with GKH.
So, your comment is based on the premise of this being a discussion surrounding stable. It's not, so I don't know what to make of the rest of your comment on the basis of this incorrect premise.
> So, your comment is based on the premise of this being a discussion surrounding stable. It's not, so I don't know what to make of the rest of your comment on the basis of this incorrect premise.
The repo is literally called "linux-stable-rc"
It wasn't an incorrect premise, just incorrect terminology. Sorry, my bad, I shouldn't have referred to it prematurely as "stable" when it is just undergoing the process of stabilization.
> > in the stable branch timeline
> This has nothing to do with stable. I don't know why you keep talking about stable.
Yes, technically, this isn't officially called "stable" until after the last release candidate, however every release candidate should be considered as an attempt to create the stable release (although pragmatically, nobody expects the first few to have had enough testing to surface all the bugs that are likely to show up) and I don't think it's particularly egregious to talk about this change in the context of the stable branch timeline as -rc releases are just as much part of the timeline as the initial stable release and the later point releases.
For context, this change was being requested for inclusion in -rc6, which was over 4 weeks after the merge window ended. This very well could end up being promoted to the stable release if no more significant bugs are found. There is no way a change of this complexity should have been accepted, and when Linus pointed that out, Kent shouldn't have been arguing about it at all, instead he should have just waited to get it merged into 6.12 as he originally intended.
> The discussion is about bleeding edge mainline Linux.
Yes, it's mainline, but also "bleeding edge" is kind of a misnomer, as it hadn't been accepting feature changes and was in stabilization for producing stable release candidates for a month already, and by that point would have had significant testing.
Sorry for causing confusion my referring to it prematurely as "stable". I don't look at the kernel all that often, and we use a different process with different terminology in our environment. We keep mainline open all the time for ongoing feature work, fork that to "stable" which only accepts bug-fixes and from that we periodically create release candidates which get released for testing and possibly get relabeled as the actual release. Sorry I was still thinking in that mindset when I replied and didn't properly map the concepts back to those used in the kernel.
The repo is literally called "linux-stable-rc"
It wasn't an incorrect premise, just incorrect terminology. Sorry, my bad, I shouldn't have referred to it prematurely as "stable" when it is just undergoing the process of stabilization.
> > in the stable branch timeline
> This has nothing to do with stable. I don't know why you keep talking about stable.
Yes, technically, this isn't officially called "stable" until after the last release candidate, however every release candidate should be considered as an attempt to create the stable release (although pragmatically, nobody expects the first few to have had enough testing to surface all the bugs that are likely to show up) and I don't think it's particularly egregious to talk about this change in the context of the stable branch timeline as -rc releases are just as much part of the timeline as the initial stable release and the later point releases.
For context, this change was being requested for inclusion in -rc6, which was over 4 weeks after the merge window ended. This very well could end up being promoted to the stable release if no more significant bugs are found. There is no way a change of this complexity should have been accepted, and when Linus pointed that out, Kent shouldn't have been arguing about it at all, instead he should have just waited to get it merged into 6.12 as he originally intended.
> The discussion is about bleeding edge mainline Linux.
Yes, it's mainline, but also "bleeding edge" is kind of a misnomer, as it hadn't been accepting feature changes and was in stabilization for producing stable release candidates for a month already, and by that point would have had significant testing.
Sorry for causing confusion my referring to it prematurely as "stable". I don't look at the kernel all that often, and we use a different process with different terminology in our environment. We keep mainline open all the time for ongoing feature work, fork that to "stable" which only accepts bug-fixes and from that we periodically create release candidates which get released for testing and possibly get relabeled as the actual release. Sorry I was still thinking in that mindset when I replied and didn't properly map the concepts back to those used in the kernel.
> The repo is literally called "linux-stable-rc"
It's not? What repo? The only two repos which are involved are https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... which is mainline (implicitly) which is where these patches would land and git://evilpiepirate.org/bcachefs.git which is the source repo for the PR. The only branches being referenced are "master" (implicitly) for mainline and the tag "bcachefs-2024-08-23".
Regardless, to respond to the rest of your comment:
The reasons for why Linus is rejecting the change have nothing to do with the stable process and everything to do with the set release process. The mainline merge window opens, you (not you specifically unless you are a subsystem maintainer, if you want to contribute a patch as a non-maintainer, the process is completely separate and goes via the subsystem maintainers) submit features and bug fixes, the merge window closes, somewhere in the ballpark of 7 release candidates happen, and it's released as mainline. The goal of the RCs is to incorporate subsequent waves of fixes for any regressions introduced specifically by the bug fixes and new features.
Kent is claiming that, because he himself implements effectively an equivalently rigorous (according to him) feature testing and stabilisation process that his patches which do not fix regressions introduced by previous patches submitted during the merge window, but which do fix some real bugs, should be accepted outside the merge window.
In the past, Linus has let it slide, and he has also let it slide this time too: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... . Linus is just asking Kent to stop doing this as he doesn't want to keep giving him special treatment.
It's not? What repo? The only two repos which are involved are https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... which is mainline (implicitly) which is where these patches would land and git://evilpiepirate.org/bcachefs.git which is the source repo for the PR. The only branches being referenced are "master" (implicitly) for mainline and the tag "bcachefs-2024-08-23".
Regardless, to respond to the rest of your comment:
The reasons for why Linus is rejecting the change have nothing to do with the stable process and everything to do with the set release process. The mainline merge window opens, you (not you specifically unless you are a subsystem maintainer, if you want to contribute a patch as a non-maintainer, the process is completely separate and goes via the subsystem maintainers) submit features and bug fixes, the merge window closes, somewhere in the ballpark of 7 release candidates happen, and it's released as mainline. The goal of the RCs is to incorporate subsequent waves of fixes for any regressions introduced specifically by the bug fixes and new features.
Kent is claiming that, because he himself implements effectively an equivalently rigorous (according to him) feature testing and stabilisation process that his patches which do not fix regressions introduced by previous patches submitted during the merge window, but which do fix some real bugs, should be accepted outside the merge window.
In the past, Linus has let it slide, and he has also let it slide this time too: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... . Linus is just asking Kent to stop doing this as he doesn't want to keep giving him special treatment.
> It's not? What repo?
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux...
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux...
Yes I can imagine there are a bunch of repos with "stable" in their name in git.kernel.org but where did you find a reference to this repo in that thread?
There are repos on git.kernel.org for microemacs, that doesn't mean this thread relates to microemacs.
There are repos on git.kernel.org for microemacs, that doesn't mean this thread relates to microemacs.
I thought stable means "doesn't change"?
Not in the kernel land. Stable branches feature tens of thousands of patches.
Patches are expected but the kernel interfaces shouldn't change right? Like if I write a kernel module no patch should break my compatibility and make my module not build anymore (I think)? I don't care if it changes underneath as long as it doesn't change where I interface.
Userspace doesn't break, but if you don't want your module to break, upstream it (which is an important lesson about hardware selection: if it's not upstream and not being upstreamed, then you're going to get stuck on an old kernel at some point).
ZFS has broken on new releases (I don't recall if they were stable, I think they were), and that is one reason I won't use as the main filesystem on linux.
ZFS has broken on new releases (I don't recall if they were stable, I think they were), and that is one reason I won't use as the main filesystem on linux.
Usually. There are no hard rules though.
Upstream stable kernel certainly does not care about compatibility with your particular thirdparty module. You'll just have to add another KERNEL_VERSION #if. Maybe if you're nvidia, or something, things are different.
Upstream stable kernel certainly does not care about compatibility with your particular thirdparty module. You'll just have to add another KERNEL_VERSION #if. Maybe if you're nvidia, or something, things are different.
Let’s not misrepresent Kent over a single incident of sending too much after a merge window. He’s extremely helpful and nice in every interaction I’ve ever read.
My 2 cents: these are the types of people that actually get the job done. All good software in my experience starts thanks to overachieving human representations of Pareto's law - people that can do alone in months what a team of average-skilled developers do in years.
In this industry it's very, very easy to run in circle, keeping doing stuff over stuff without realising you are in a bikeshedding loop, you're overengineering or simply wasting time. We need people that want to just push forward and assume the responsibility of anything that breaks, otherwise I'm sure that in 30 years we'd all still be using the same stuff we've always used because it's just human nature to stick with what we know works, quirks and all.
In this industry it's very, very easy to run in circle, keeping doing stuff over stuff without realising you are in a bikeshedding loop, you're overengineering or simply wasting time. We need people that want to just push forward and assume the responsibility of anything that breaks, otherwise I'm sure that in 30 years we'd all still be using the same stuff we've always used because it's just human nature to stick with what we know works, quirks and all.
> he wants to "move fast and break things"
That's not how I read that thread. This is just about where the diligence happens, not that it can be avoided; and exactly how small a fix is mergable in the fixes phase of kernel development.
I don't see that thread as being particularly angry either. There have been ones where both of them have definitely lost their cool; here they are having a (for them) calm disagreement. Linus is just not going to merge this one until the next development phase, which is fine.
There have been arguments involving this developer that do raise questions; I just don't see this as one of them.
That's not how I read that thread. This is just about where the diligence happens, not that it can be avoided; and exactly how small a fix is mergable in the fixes phase of kernel development.
I don't see that thread as being particularly angry either. There have been ones where both of them have definitely lost their cool; here they are having a (for them) calm disagreement. Linus is just not going to merge this one until the next development phase, which is fine.
There have been arguments involving this developer that do raise questions; I just don't see this as one of them.
That's a normal LKML conversation. Nowhere do I see actual bugs pointed out, in fact the only testimony by Carl said that bcachefs has been quite stable so far.
This is just about following procedure. There are people who follow procedure and introduce many bugs, and people who don't but write perfect software.
The bcachefs author cautiously marks bcachefs as unstable, which is normal for file systems. The only issue here is that the patch touched other areas, but in the kernel development model Linus is free not to pull it.
This is just about following procedure. There are people who follow procedure and introduce many bugs, and people who don't but write perfect software.
The bcachefs author cautiously marks bcachefs as unstable, which is normal for file systems. The only issue here is that the patch touched other areas, but in the kernel development model Linus is free not to pull it.
Is bcachefs-tools going into the mainline distros or into something that’s meant to be less stable and experimental? Linus makes it sound like there’s a more appropriate place for this work.
Edit: Reading through the thread, it seems like there is a claim of rigorous but effectively private testing. Without the ability to audit those results easily it’s causing a lot of worry.
Edit: Reading through the thread, it seems like there is a claim of rigorous but effectively private testing. Without the ability to audit those results easily it’s causing a lot of worry.
I don't think Linus is particularly concerned about bcachefs-tools, and whether a particular distributions ships with it or not isn't a concern for the kernel. Presumably though, distributions that don't ship the tools may also want to disable it in the kernel, although I'd imagine they'd leave it alone if it was previously supported.
Linus' complaint was about adding feature work into a supposed minor bug fix, especially because (going from Linus' comments) they were essentially refactors of major systems that impacted other areas of the kernel.
Linus' complaint was about adding feature work into a supposed minor bug fix, especially because (going from Linus' comments) they were essentially refactors of major systems that impacted other areas of the kernel.
Great and now the top thread on the HN discussion is about that drama only tangentially referenced in the article.
This would be a good one for Rust enthusiasts to weigh in on.
The issue raised is that some program written in rust insists on a very specific version of various dependencies. If other people change the metadata, it builds and seems to run ok with different versions. (Developer on reddit clarifies that it builds and does the wrong thing, and recommends dropping Debian as a solution).
Linux likes to pack a finite set of versions of libraries (ideally just one version at a time) and use that dependency for multiple programs, totally at odds with the vendoring specific versions strategy.
I'm not clear what a solution is to this one. In the specific case, sure, drop it from Debian. But this isn't the only program choosing to vendor dependencies instead of use them from the environment.
The issue raised is that some program written in rust insists on a very specific version of various dependencies. If other people change the metadata, it builds and seems to run ok with different versions. (Developer on reddit clarifies that it builds and does the wrong thing, and recommends dropping Debian as a solution).
Linux likes to pack a finite set of versions of libraries (ideally just one version at a time) and use that dependency for multiple programs, totally at odds with the vendoring specific versions strategy.
I'm not clear what a solution is to this one. In the specific case, sure, drop it from Debian. But this isn't the only program choosing to vendor dependencies instead of use them from the environment.
Maintenance is much more practical if everything in your dependency tree uses the same version of every dependency. Achieving this in practice is much of the work distribution maintainers have to do.
Doing this has a couple of key advantages:
1. when a security vulnerability needs patching in a stable distribution release, the security team only have to patch one version instead of a dozen or more different versions bundled all over the place
2. a library that depends on dependencies A and B, both of which depend on X, can actually work properly if it needs to pass API objects created by X across between A and B, since X is of the same version
In an ecosystem where it's considered acceptable to "pin" versions of dependencies and also call any system that doesn't use the pinned versions "unsupported", both of the above two cases become impractical.
Whether you use shared libraries or static libraries, the above matters still exist.
Doing this has a couple of key advantages:
1. when a security vulnerability needs patching in a stable distribution release, the security team only have to patch one version instead of a dozen or more different versions bundled all over the place
2. a library that depends on dependencies A and B, both of which depend on X, can actually work properly if it needs to pass API objects created by X across between A and B, since X is of the same version
In an ecosystem where it's considered acceptable to "pin" versions of dependencies and also call any system that doesn't use the pinned versions "unsupported", both of the above two cases become impractical.
Whether you use shared libraries or static libraries, the above matters still exist.
> 2. a library that depends on dependencies A and B, both of which depend on X, can actually work properly if it needs to pass API objects created by X across between A and B, since X is of the same version
That feels like an advantage for the developer, not the distro maintainer.
I find that amusing, because you're saying it's easy if they can use one version of the library. Well, that's also true if they're writing an application, hence pinning.
That feels like an advantage for the developer, not the distro maintainer.
I find that amusing, because you're saying it's easy if they can use one version of the library. Well, that's also true if they're writing an application, hence pinning.
> That feels like an advantage for the developer, not the distro maintainer.
It's an advantage for the ecosystem, yes. Not the developer of the library (X in my example), but a different developer. Someone who is trying to write an app to do something interesting with A and B.
> ...because you're saying it's easy if they can use one version of the library. Well, that's also true if they're writing an application
I don't think that's true.
Perhaps library A has pinned dependencies for X such that it is in conflict with unrelated library B that also uses pinned dependencies for X. Both libraries are now less useful to the ecosystem, because they are now inadvertently incompatible with each other (if their APIs expose X's objects).
My example demonstrates why this doesn't work in the general case, and therefore why it's wrong to expect that everyone does it this way.
It's an advantage for the ecosystem, yes. Not the developer of the library (X in my example), but a different developer. Someone who is trying to write an app to do something interesting with A and B.
> ...because you're saying it's easy if they can use one version of the library. Well, that's also true if they're writing an application
I don't think that's true.
Perhaps library A has pinned dependencies for X such that it is in conflict with unrelated library B that also uses pinned dependencies for X. Both libraries are now less useful to the ecosystem, because they are now inadvertently incompatible with each other (if their APIs expose X's objects).
My example demonstrates why this doesn't work in the general case, and therefore why it's wrong to expect that everyone does it this way.
Generally speaking libraries don't pin versions, applications do (in rust-land, a Cargo.lock is respected for the folder you're building, not your dependencies. A dependency can specify an exact version if it wants but an application can override that kind of thing, and it's generally not considered a good idea). This makes 2) a non-issue for the ecosystem (if an application needs to pass objects from X between A and B, then they'll need to pin to a single version of X). 1) is more of a disadvantage, but it's unclear to me that the effort distro maintainers put in to fix to a single version actually results in a reduction in effort overall.
One package's app is another package's dependency, so this distinction cannot work in the general case.
It can work perfectly well, at least as far as API/ABI compatibility. An app is an executable, a library is a static or dynamic library. Something isn't generally both at once (a project or package might have a library and some tool applications related to the library, but those tend to get split out by distribution package maintainers anyway, and with cargo there's always a top-level package which sets the pinning, the rest are libraries whether or not they're the top level in other contexts). If you're talking about IPC or RPC, then yes, this matters more in terms of compatibility, but that's an area where a lot more attention is paid to compatibility in the first place, by everyone.
Maintenance is much more practical when you use the versions upstream tests in their CI and not whatever mishmash of ancient/silently incompatible deps that each distro separately decides to combine together.
Upstream CI isn't magic that only they have. Distributions run the same tests upstreams do if they are made available.
If tests falsely pass, then that's a quality problem in the upstream tests.
If tests falsely pass, then that's a quality problem in the upstream tests.
No.
1. Rust is as hostile to dynamic libraries as glibc to static.
2. With everything static you have to rebuild every dependent on any security patch anyways. If you meant with multiple versions maintainers have to backport patches to multiple versions. Maybe don't backport at all? Some people appreciate having backports. Users of software written in Rust does NOT. So why bother doing backports for them?
3.
> a library that depends on dependencies A and B, both of which depend on X, can actually work properly
This will be detected during compile (since it's all static! sorry dynamic linking fans) and dealt with.
1. Rust is as hostile to dynamic libraries as glibc to static.
2. With everything static you have to rebuild every dependent on any security patch anyways. If you meant with multiple versions maintainers have to backport patches to multiple versions. Maybe don't backport at all? Some people appreciate having backports. Users of software written in Rust does NOT. So why bother doing backports for them?
3.
> a library that depends on dependencies A and B, both of which depend on X, can actually work properly
This will be detected during compile (since it's all static! sorry dynamic linking fans) and dealt with.
Dynamic libraries have had interface versions and linking strategies for decades.
For statically compiled apps, if the version is over specified recompiling will have no effect. The dependent will have to be updated with a new version string.
For statically compiled apps, if the version is over specified recompiling will have no effect. The dependent will have to be updated with a new version string.
> Some people appreciate having backports. Users of software written in Rust does NOT. So why bother doing backports for them?
This is completely false. First and foremost, people who use applications don't care at all what language those apps are written in, and there is no "community of people who use Rust tools" that could have a different world-view from everyone else.
And beyond that, there is no one in the world who doesn't want backports, because it's never safe to just take the latest version of some piece of software and expect that everything will still work as expected. Having to take even a month of new features just because you need a security fix.
And note that Rust even at the language level has strong support for backwards compatibility and support for older versions. It's definitely not true that Rust people, even developers, live at the bleeding edge and don't need backports.
This is completely false. First and foremost, people who use applications don't care at all what language those apps are written in, and there is no "community of people who use Rust tools" that could have a different world-view from everyone else.
And beyond that, there is no one in the world who doesn't want backports, because it's never safe to just take the latest version of some piece of software and expect that everything will still work as expected. Having to take even a month of new features just because you need a security fix.
And note that Rust even at the language level has strong support for backwards compatibility and support for older versions. It's definitely not true that Rust people, even developers, live at the bleeding edge and don't need backports.
> This will be detected during compile (since it's all static! sorry dynamic linking fans) and dealt with.
If it is correctly detected, then the only "dealing with" that can be done is to fail to build.
If it is correctly detected, then the only "dealing with" that can be done is to fail to build.
This isn't Rust specific. The same issue exists in all languages where the versions can be restricted at project level. It's an issue in Debian because they can't handle multiple concurrent versions (beyond renaming the package) and want every package built without internet access.
But that same issue affects Ruby, Python, etc. if the project specifies strict versions. And if you deal with filesystems, you really want strict versions - I understand why the author wants to keep it that way.
It's more of a self-inflicted Debian policy pain rather than anything to do with Rust. The author could be nice and vendor, but also he's completely right to refuse. The request is basically "we put handcuffs on ourselves, so please you do the work instead".
But that same issue affects Ruby, Python, etc. if the project specifies strict versions. And if you deal with filesystems, you really want strict versions - I understand why the author wants to keep it that way.
It's more of a self-inflicted Debian policy pain rather than anything to do with Rust. The author could be nice and vendor, but also he's completely right to refuse. The request is basically "we put handcuffs on ourselves, so please you do the work instead".
In fact, I am tired of the same issue e.g. when evaluating machine learning tools with Python. It's as if everyone just says "dependency = ${exact_version_I_have}" and never considers anything else. Not helped by the fact that most of these projects break ABI every other day which I still think it is just plain evil.
Please don't say "just use containers/virtualenvs/vendoring/whatever" because at some point you obviously want these "containers" to use your real hardware. And your GPU driver is only going to be at one version, no matter how many statically linked executables you have.
Please don't say "just use containers/virtualenvs/vendoring/whatever" because at some point you obviously want these "containers" to use your real hardware. And your GPU driver is only going to be at one version, no matter how many statically linked executables you have.
> It's as if everyone just says "dependency = ${exact_version_I_have}" and never considers anything else.
This is what I call "version soup".
The idea that every project can choose an near-arbitrary set of very exact version numbers of dependencies and expect it to work. And that every project on earth has the burden of continually stirring their soup, often through a tool like dependabot, hoping (cross fingers) no compatibility problems come about.
Of course, there's no guarantee that those versions will work together. The extremely limited abilities packaging tools have for expressing dependency restrictions (>= this or that version) generally lack the ability to even handle the concept of stable branches with backports. i.e. "No, it must be >= 1.2.3.. what do you mean they backported the relevant fix to 1.1.4? 1.1.4 < 1.2.3 so it's unacceptable". The way these specifications are then used by authors brings an extra layer of noise to the compatibility situation - very few (understandably) will actually go and check the limits of version compatibility.
In nixpkgs we attempt to address this situation for the python ecosystem by providing one version of each package (with few exceptions) per release. But in return, we put in work to make sure those versions actually work with each other - generally by getting the projects' test suites integrated into the build system. The idea is that an app built to depend on nixpkgs packages should be able to expect to do dependency upgrades as a "jump" every 6 months when there's a new nixpkgs release, but otherwise be able to depend on a stable suite of packages that still receives security updates & backports.
This is what I call "version soup".
The idea that every project can choose an near-arbitrary set of very exact version numbers of dependencies and expect it to work. And that every project on earth has the burden of continually stirring their soup, often through a tool like dependabot, hoping (cross fingers) no compatibility problems come about.
Of course, there's no guarantee that those versions will work together. The extremely limited abilities packaging tools have for expressing dependency restrictions (>= this or that version) generally lack the ability to even handle the concept of stable branches with backports. i.e. "No, it must be >= 1.2.3.. what do you mean they backported the relevant fix to 1.1.4? 1.1.4 < 1.2.3 so it's unacceptable". The way these specifications are then used by authors brings an extra layer of noise to the compatibility situation - very few (understandably) will actually go and check the limits of version compatibility.
In nixpkgs we attempt to address this situation for the python ecosystem by providing one version of each package (with few exceptions) per release. But in return, we put in work to make sure those versions actually work with each other - generally by getting the projects' test suites integrated into the build system. The idea is that an app built to depend on nixpkgs packages should be able to expect to do dependency upgrades as a "jump" every 6 months when there's a new nixpkgs release, but otherwise be able to depend on a stable suite of packages that still receives security updates & backports.
> In nixpkgs we attempt to address this situation for the python ecosystem by providing one version of each package (with few exceptions) per release. But in return, we put in work to make sure those versions actually work with each other - generally by getting the projects' test suites integrated into the build system. The idea is that an app built to depend on nixpkgs packages should be able to expect to do dependency upgrades as a "jump" every 6 months when there's a new nixpkgs release, but otherwise be able to depend on a stable suite of packages that still receives security updates & backports.
Is this different from any other Linux distribution?
Is this different from any other Linux distribution?
Essentially no, but
- most Linux distributions don't reach very far into e.g. the python ecosystem (python packages that exist are generally there to support packaged applications)
- nixpkgs isn't linux specific - most packages work on macos, and nixpkgs happily works on top of other distributions.
- most Linux distributions don't reach very far into e.g. the python ecosystem (python packages that exist are generally there to support packaged applications)
- nixpkgs isn't linux specific - most packages work on macos, and nixpkgs happily works on top of other distributions.
In Rust, it is frowned upon to pin to an exact version. We do encourage people to specify what semver version range will work. We don't have great support for depending on and verifying multiple-major version ranges when a library broke compatibility but not in a way that affects you.
Is this being tracked somewhere?
> And your GPU driver is only going to be at one version, no matter how many statically linked executables you have.
This is a non-issue.
libcuda.so (which is part of the NVIDIA driver, their userland part) can not be statically linked into anything, since it's closed-source and no .a shipped at all, so no possibility of misconfigure it to be statically linked. And NVIDIA drivers, as a whole (including a pair of libcuda.so and corresponding .ko), is infinitely backward-compatible, you can always use new driver with old libraries. So the problem you mentioned does not exist at all.
Statically linking to cuBLAS / cuDNN etc never add more pain and can only reduce pain. Unless you hold it wrong.
This is a non-issue.
libcuda.so (which is part of the NVIDIA driver, their userland part) can not be statically linked into anything, since it's closed-source and no .a shipped at all, so no possibility of misconfigure it to be statically linked. And NVIDIA drivers, as a whole (including a pair of libcuda.so and corresponding .ko), is infinitely backward-compatible, you can always use new driver with old libraries. So the problem you mentioned does not exist at all.
Statically linking to cuBLAS / cuDNN etc never add more pain and can only reduce pain. Unless you hold it wrong.
That is hardly true for NVIDIA over the years, as I have experienced personally, and it's definitely not true for about every other GPU driver in existence -- particularly AMD's RoCM, which even recommends specific Python versions to go with each ROCM version. In addition, replace "GPU" with any other piece of hardware, such as NPU. Or even software, such as your desktop environment. Do you think a statically linked binary will run with Wayland+1 ? It's already a chore to run games that link with older SDL versions, and the only reason you can at all is that SDL was dynamically linked...
Even the kernel itself does a pretty poor job at maintaining backwards compatibility (e.g. contents of /sys changing every other year). Statically linking just hardcodes one of these things, and then forgets about the rest, such as IPC APIs, desktop APIs, filesystem structure, etc. which due to the poor culture out there are moving as much as the libraries themselves.
Even the kernel itself does a pretty poor job at maintaining backwards compatibility (e.g. contents of /sys changing every other year). Statically linking just hardcodes one of these things, and then forgets about the rest, such as IPC APIs, desktop APIs, filesystem structure, etc. which due to the poor culture out there are moving as much as the libraries themselves.
> Do you think a statically linked binary will run with Wayland+1 ?
Yes, yes it will. The baseline stuff (the base protocol and xdg_shell) has been stable for years. Of course, one might argue that the caveman byte poking provided there turns a modern machine into a slightly faster 15-year-old one, and I agree it does. (It’s still fun to do as an experiment.)
But the moment you try to actually leverage the GPU the modern Linux desktop forces you to abandon static linking in order to load pieces of the graphics driver into your address space, and unfortunately those pieces will in turn force you to dynamically link the distro’s preferred libc (likely Glibc) and libwayland (as opposed to any other protocol implementation).
So either you poke pixels into a byte buffer by hand, and your question gets an answer in the affirmative simply because that foundational layer is fairly small, or you take advantage of the hardware, and your question becomes moot because you can no longer statically link.
Yes, yes it will. The baseline stuff (the base protocol and xdg_shell) has been stable for years. Of course, one might argue that the caveman byte poking provided there turns a modern machine into a slightly faster 15-year-old one, and I agree it does. (It’s still fun to do as an experiment.)
But the moment you try to actually leverage the GPU the modern Linux desktop forces you to abandon static linking in order to load pieces of the graphics driver into your address space, and unfortunately those pieces will in turn force you to dynamically link the distro’s preferred libc (likely Glibc) and libwayland (as opposed to any other protocol implementation).
So either you poke pixels into a byte buffer by hand, and your question gets an answer in the affirmative simply because that foundational layer is fairly small, or you take advantage of the hardware, and your question becomes moot because you can no longer statically link.
It's 100% culture: this happens across all languages.
It should be noted that, particularly for rust, it's a silly policy as rust does not support dynamic linking of rust libraries. All rust applications are statically linked.
So trying to force all projects using foo onto the same version of foo is just a huge headache with no real benefit to anything.
So trying to force all projects using foo onto the same version of foo is just a huge headache with no real benefit to anything.
> So trying to force all projects using foo onto the same version of foo is just a huge headache with no real benefit to anything.
It creates a security nightmare.
Distributions expect to be able to security patch without waiting for laggard upstreams to bump their pinned dependency versions. If the ecosystem insists that only upstream-sanctioned versions are acceptable, then that runs contrary to this expectation, because it blocks distributions from patching or bumping dependency versions. If this case should be an exception, then you end up with a security team having to patch myriad different versions. That's the security nightmare.
It creates a security nightmare.
Distributions expect to be able to security patch without waiting for laggard upstreams to bump their pinned dependency versions. If the ecosystem insists that only upstream-sanctioned versions are acceptable, then that runs contrary to this expectation, because it blocks distributions from patching or bumping dependency versions. If this case should be an exception, then you end up with a security team having to patch myriad different versions. That's the security nightmare.
Fair point.
I do think we need to reevaluate how security is managed in terms of software. It's often the case that we say "this library version has a CVE, we need to update it!" However, it's not often the case that the CVE in the library is even exploitable from the parent application.
I'm thinking of a recent 9 CVE against zlib... But actually it was a rarely used extension API for working with compressed files.
Something like path tracing to see if a vulnerable API is even accessible from the parent application/library before instructing the universe to adopt a new version.
I do think we need to reevaluate how security is managed in terms of software. It's often the case that we say "this library version has a CVE, we need to update it!" However, it's not often the case that the CVE in the library is even exploitable from the parent application.
I'm thinking of a recent 9 CVE against zlib... But actually it was a rarely used extension API for working with compressed files.
Something like path tracing to see if a vulnerable API is even accessible from the parent application/library before instructing the universe to adopt a new version.
Rust has the advantage of aggressively gating many library APIs behind feature flags as a convention. A lot of third-party libraries consist of a core of the most essential functionality that the entire library just can't go without, and everything else, particularly things that require yet more external dependencies to work, is gated behind a feature.
You could imagine a model where security vulnerability reports include a combination of features required to trigger that vulnerability. If the vulnerability was in a rarely-used extension, like it was for Zlib, many dependents would be automatically marked as non-vulnerable because they wouldn't have the necessary feature flag active.
You could imagine a model where security vulnerability reports include a combination of features required to trigger that vulnerability. If the vulnerability was in a rarely-used extension, like it was for Zlib, many dependents would be automatically marked as non-vulnerable because they wouldn't have the necessary feature flag active.
I think it's already complicated to find all the software that uses a library version that has a known vulnerability, because of the many slightly different versions floating around, static linking, software running in docker containers etc. Expecting to further identify which applications use a particular part of a library gets even more difficult, and lazy developers will use this as an excuse not to patch far more often than you'll save on unnecessary updates.
Not to mention, this already happens to some extent, especially for older versions of software: distributors and developers do sometimes say "we will not provide a patch because even if we use the affected library, we don't use it in a way that triggers the vulnerability".
Not to mention, this already happens to some extent, especially for older versions of software: distributors and developers do sometimes say "we will not provide a patch because even if we use the affected library, we don't use it in a way that triggers the vulnerability".
Not all distributions suffer from this, because they have the tooling to patch all crate versions given a name. Compile times are actually the worst part but can be mitigated. But these distros also fundamentally accept the language's compilation model in question, so they have to develop solutions if they want to supply software to their users rather than make up excuses. Most distros don't like vendoring, for reasons. But many code maintainers do like it, for reasons. So it is what it is.
There are also more tools than ever to do things like correlate versions of OSS packages with reported vulnerabilities e.g. Google's osv.dev. 95% or more of Rust vendoring isn't actually "vendoring" in the sense you have to figure anything out. The dependencies are literally written in the lock file with the version and that can be traced back to a Git hash automatically in almost all cases. Some crates that bind C libraries might require this work, where you have to look at the code to see "What version of the code did the `cp -R` into here?" but they are practically limited in number, and detectable. You aren't doing any detective work for 98% of this stuff. It can be scaled pretty well with a few (good) heuristics.
Rust even has the advantage that large, vast amounts of code is actually statically analyzable under known sets of configurations/features. In the future I suspect a tool like `govulncheck` for Rust will be developed, making accurate vuln tracing even easier.
While I'm admittedly biased because I work on a distro where this problem isn't as directly relevant, my answer is that in 2024 if you don't have the tooling to do this stuff, it's absolutely on you. Whether as a policy decision or a technical limitation, whatever.
There are also more tools than ever to do things like correlate versions of OSS packages with reported vulnerabilities e.g. Google's osv.dev. 95% or more of Rust vendoring isn't actually "vendoring" in the sense you have to figure anything out. The dependencies are literally written in the lock file with the version and that can be traced back to a Git hash automatically in almost all cases. Some crates that bind C libraries might require this work, where you have to look at the code to see "What version of the code did the `cp -R` into here?" but they are practically limited in number, and detectable. You aren't doing any detective work for 98% of this stuff. It can be scaled pretty well with a few (good) heuristics.
Rust even has the advantage that large, vast amounts of code is actually statically analyzable under known sets of configurations/features. In the future I suspect a tool like `govulncheck` for Rust will be developed, making accurate vuln tracing even easier.
While I'm admittedly biased because I work on a distro where this problem isn't as directly relevant, my answer is that in 2024 if you don't have the tooling to do this stuff, it's absolutely on you. Whether as a policy decision or a technical limitation, whatever.
What do you mean that they can patch all crate versions given a name? Sometimes the same patch might be trivially applied to many different versions of a library, but in the general case, this is not true: the amount of work is in direct proportion with the number of library versions you have to patch, even if you know exactly which versions those are. So the difference between maintaining, say, 3 versions of a library, and 30, is going to be ~10× the work, even with tools that instantly identify the exact versions that need to be patched.
Tooling to apply the patches is trivial. This isn't the problem I'm describing.
As the sibling comment says, the issue is when the patch doesn't apply cleanly to some subset of versions in use, or even worse if they appear to apply cleanly but leave behind logic bugs.
If distributions did approach this by "distro-patching" specific versions, then that would reintroduce the upstream complaint that distros are using versions of dependencies not sanctioned by them. So your approach, even if it did work in the general case, would take you back to square one.
As the sibling comment says, the issue is when the patch doesn't apply cleanly to some subset of versions in use, or even worse if they appear to apply cleanly but leave behind logic bugs.
If distributions did approach this by "distro-patching" specific versions, then that would reintroduce the upstream complaint that distros are using versions of dependencies not sanctioned by them. So your approach, even if it did work in the general case, would take you back to square one.
Yeah, I don't think either of these are really big problems. People identify the vuln, the fix, and they tend to share the patches for backports across versions + across distros, and/or immediately update downstream users to fixed versions because, again, the nifty lockfiles literally tell you if they are vulnerable as they contain the dep graph. You query every package's dep graph and look at the bounds. Maintainers with RUSTSEC advisories will yank the crate on crates.io, forcing upgrades across the stack. You can apply patches to most versions of a crate (again: you can find all of them!) and loosen or make more targeted fixes if needed. But even if 30 versions are vulnerable, in practice 30 patches don't need to be written in the p99 case. So I just disagree with the sibling on that, based on my experience. Most of these packages aren't the velocity of the Linux kernel. They aren't rewriting 10,000 lines of code a version so you have to re-interrogate the fix.
But most importantly, I don't see any argument this is all meaningfully more work than all the manual shit Debian maintainers do to get all this stuff working, on top of the fact they try to ship frankenstein versions of packages that silently break and piss off users (every distro ships some busted packages but it's a matter of give and take.) Other maintainers of other distros don't spend time on all the stuff the OP spent time on. They don't spend time loosening lockfile bounds on Rust crates for filesystem-sensitive tooling(!!!) in an effort to make a 35-year-old cultural rule about C/C++'s compilation model apply to them. Mostly, they run some script like 'import crates.io package XYZ version 1.2.3', commit the result, and get it code reviewed and merged. They can then spend time on other things. And often, these same processes are used for all updates, including many security updates. This stuff is insanely well oiled in other projects.
Sorry to be the hater, but this is squarely a Debian dysfunction as far as I can see? If they spent more time actually solving infrastructural problems they had, they would be able to spend time on actually delivering more working software to their users in the world they actually exist in.
But most importantly, I don't see any argument this is all meaningfully more work than all the manual shit Debian maintainers do to get all this stuff working, on top of the fact they try to ship frankenstein versions of packages that silently break and piss off users (every distro ships some busted packages but it's a matter of give and take.) Other maintainers of other distros don't spend time on all the stuff the OP spent time on. They don't spend time loosening lockfile bounds on Rust crates for filesystem-sensitive tooling(!!!) in an effort to make a 35-year-old cultural rule about C/C++'s compilation model apply to them. Mostly, they run some script like 'import crates.io package XYZ version 1.2.3', commit the result, and get it code reviewed and merged. They can then spend time on other things. And often, these same processes are used for all updates, including many security updates. This stuff is insanely well oiled in other projects.
Sorry to be the hater, but this is squarely a Debian dysfunction as far as I can see? If they spent more time actually solving infrastructural problems they had, they would be able to spend time on actually delivering more working software to their users in the world they actually exist in.
> in an effort to make a 35-year-old cultural rule about C/C++'s compilation model apply to them
> this is squarely a Debian dysfunction as far as I can see
Nothing I have said is specific to Debian or to C/C++. I've presented general arguments that hold regardless of distribution policies or the language toolchain used. Nothing you have said demonstrates why the general points I have made would apply to Debian or to C/C++ but not to Rust-based projects.
If you still believe that this is the case, then I think that demonstrates that you do not understand the matter deeply enough.
The only thing that Debian is insisting on is basic software supply chain hygiene. Every so often, there is a very public failure in ecosystems that don't bother with this hygiene, while Debian continues unaffected.
> You can apply patches to most versions of a crate
The point of the complaint that spawned this discussion is that there's an upstream that considers it unacceptable for downstreams to be doing this patching (or indeed running any dependency version that is not sanctioned by upstream). It is a contradictory position to justify downstream patches as a supposedly easy solution while at the same time complaining that downstreams patch at all.
The fact is that distributions users do not expect to be beholden to laggard upstreams while awaiting security fixes. This requires distributions to modify the dependency versions (or patch, which is the equivalent "off-piste" behaviour) used to build their packages.
> this is squarely a Debian dysfunction as far as I can see
Nothing I have said is specific to Debian or to C/C++. I've presented general arguments that hold regardless of distribution policies or the language toolchain used. Nothing you have said demonstrates why the general points I have made would apply to Debian or to C/C++ but not to Rust-based projects.
If you still believe that this is the case, then I think that demonstrates that you do not understand the matter deeply enough.
The only thing that Debian is insisting on is basic software supply chain hygiene. Every so often, there is a very public failure in ecosystems that don't bother with this hygiene, while Debian continues unaffected.
> You can apply patches to most versions of a crate
The point of the complaint that spawned this discussion is that there's an upstream that considers it unacceptable for downstreams to be doing this patching (or indeed running any dependency version that is not sanctioned by upstream). It is a contradictory position to justify downstream patches as a supposedly easy solution while at the same time complaining that downstreams patch at all.
The fact is that distributions users do not expect to be beholden to laggard upstreams while awaiting security fixes. This requires distributions to modify the dependency versions (or patch, which is the equivalent "off-piste" behaviour) used to build their packages.
If the only way a security team can cope is with a single version policy, that sounds to me like a tooling problem.
Take Nixpkgs, for example. They have this concept called "default crate overrides" which their rust builder uses, which offers the capability to patch all versions of a crate. That's not to say that Nixpkgs does rust perfectly, but just an example of the sort of tooling a security team should have at their disposal.
Take Nixpkgs, for example. They have this concept called "default crate overrides" which their rust builder uses, which offers the capability to patch all versions of a crate. That's not to say that Nixpkgs does rust perfectly, but just an example of the sort of tooling a security team should have at their disposal.
How does it patch all versions automatically? If the library has been refactored, you still need to modify the patch for the pre-refactor and the post-refractor versions, for example. The closest you might get is a ChatGPT-based tool, but that is nowhere near safe enough for security patches.
I didn't say it's automatic. You can have overrides that target a more narrow range of versions, if you need. If there has been a major refactor, then you need two patches. But it's not rocket science, and you definitely don't need chatgpt.
You can also blend both strategies - you might have a version minimization strategy, where you only use, for example, one major version of a library at a time.
You can also blend both strategies - you might have a version minimization strategy, where you only use, for example, one major version of a library at a time.
The security team got used to c/c++ being the only way to build native binaries, and would prefer to peg other square languages into this round hole.
FUD enuses.
FUD enuses.
[deleted]
> rust does not support dynamic linking of rust libraries. All rust applications are statically linked.
That's not true.
What Rust currently doesn't support (there's a desire to fix this someday), is dynamically linking with a Rust library compiled with a different version of the Rust compiler, unless the library has been restricted to a C-compatible ABI. That's because Rust currently does not have a stable ABI; things like structure layouts can change between versions of the compiler, and Rust makes heavy use of inlining.
That's not true.
$ ldd /usr/bin/rustc
linux-vdso.so.1 (0x00007fde30c44000)
librustc_driver-adb68da129cf8bba.so => /lib64/librustc_driver-adb68da129cf8bba.so (0x00007fde2cc00000)
libstd-aa46ce58aa106bb2.so => /lib64/libstd-aa46ce58aa106bb2.so (0x00007fde2cad3000)
libc.so.6 => /lib64/libc.so.6 (0x00007fde2c8e2000)
[...]
What Rust currently doesn't support (there's a desire to fix this someday), is dynamically linking with a Rust library compiled with a different version of the Rust compiler, unless the library has been restricted to a C-compatible ABI. That's because Rust currently does not have a stable ABI; things like structure layouts can change between versions of the compiler, and Rust makes heavy use of inlining.
> What Rust currently doesn't support (there's a desire to fix this someday), is dynamically linking with a Rust library compiled with a different version of the Rust compiler
Note that even with the same compiler there a lot of factors that must remain the same to make the dynamic linking work properly. For example changing any compiler flag or dependency feature flag will likely result in incompatibility. The best way to guarantee that the dynamic library and the executable that loads it are compatible is to build everything as part of a single workspace (and yes, this means that if anything changes in the dynamic library crate then the executable that depend on it must also be rebuilt).
Note that even with the same compiler there a lot of factors that must remain the same to make the dynamic linking work properly. For example changing any compiler flag or dependency feature flag will likely result in incompatibility. The best way to guarantee that the dynamic library and the executable that loads it are compatible is to build everything as part of a single workspace (and yes, this means that if anything changes in the dynamic library crate then the executable that depend on it must also be rebuilt).
Almost—even if you need to rebuild dependencies, you can still patch security issues just once instead of doing so in every vendored copy. That's the first point in the parent comment.
The same applies to Go.
The same applies to Go.
It remains a problem for source packages, which are an important part of Debian necessary for offline and reproducible builds.
> It should be noted that, particularly for rust, it's a silly policy as rust does not support dynamic linking of rust libraries.
Almost. You can write dynamic libraries in Rust (using the cdylib target type), but their external interface will be limited to the C ABI. For all intents and purposes, they'll be 'C shared libraries' that just happen to be written in Rust. (You can also create C headers for the exported C API using cbindgen.)
Almost. You can write dynamic libraries in Rust (using the cdylib target type), but their external interface will be limited to the C ABI. For all intents and purposes, they'll be 'C shared libraries' that just happen to be written in Rust. (You can also create C headers for the exported C API using cbindgen.)
> This isn't Rust specific. The same issue exists in all languages where the versions can be restricted at project level
Can confirm, I occasionally use an ai program that has to hard code all of their python dependencies, because it's the only way to get it to compile, let alone run properly... and then they go and change the underlying package manager, and figure out they have to hard code even more... it's a bloody mess, but thankfully you can just run it as a docker container once it's all working...
Can confirm, I occasionally use an ai program that has to hard code all of their python dependencies, because it's the only way to get it to compile, let alone run properly... and then they go and change the underlying package manager, and figure out they have to hard code even more... it's a bloody mess, but thankfully you can just run it as a docker container once it's all working...
I think that 2 different things are mixed up.
For Rust it is language specific, because the thing being statically built, doing differently is very hard.
For other languages like Python, there is not a need and you should not do that. The dynamic part of it make it so that developer should be resilient to different versions. And python also provide a lot of conveniences for that (just think about six).
But in your case, you faced the current situation of lot of "data scientist" jumping the band wagon of AI are creating a lot of things like apprentice sorcerers without a real experience of software engineering.
For Rust it is language specific, because the thing being statically built, doing differently is very hard.
For other languages like Python, there is not a need and you should not do that. The dynamic part of it make it so that developer should be resilient to different versions. And python also provide a lot of conveniences for that (just think about six).
But in your case, you faced the current situation of lot of "data scientist" jumping the band wagon of AI are creating a lot of things like apprentice sorcerers without a real experience of software engineering.
No, in both cases the issue is the same: whether the actual interface of the library is staying the same or not, and over what time period. Python packages have plenty of the same kind of breakages. usually it's unintentional, a bug or similar, sometimes it's a deliberate breaking change in a point release. Either way the effect is the same. Version X works in project A but newer version Y doesn't, and the reverse can be true for project B. It's just that in python you're most likely to get the problem when you run the code, wheras compiled languages can also fail at the build step as well as at runtime.
It can be pretty bad with the bigger ML/scientific computing packages, hence all the version pinning that happens, though I will concur that generally speaking AI researchers don't understand package management in the slightest and you get all kinds of horrors as a result (e.g. mixing anaconda and pip in the same project, bonus points if it's blasted over the distro's python packaging for even more chaos. works on one very broken machine if you are lucky and don't touch anything).
Debian's approach is similarly frustrating: they will try to make your code work with an older version of a library than the one you developed with, which doesn't even work within semver (semver only defines backwards compatibility, not forwards). That's what caused the problem in this case: rolling forward dependencies was not recommended but didn't break anything, rolling a dependency backwards meant it stopped working, which is something absolutely no-one should be surprised or upset by (at least it meant the build broke as opposed to debian shipping a broken package like would have likely happened with python. It just meant that debian was shipping a version with bugs fixed in upstream). If you're going to do that you've got to be prepared to take responsibility for fixing the problem that you've introduced.
It can be pretty bad with the bigger ML/scientific computing packages, hence all the version pinning that happens, though I will concur that generally speaking AI researchers don't understand package management in the slightest and you get all kinds of horrors as a result (e.g. mixing anaconda and pip in the same project, bonus points if it's blasted over the distro's python packaging for even more chaos. works on one very broken machine if you are lucky and don't touch anything).
Debian's approach is similarly frustrating: they will try to make your code work with an older version of a library than the one you developed with, which doesn't even work within semver (semver only defines backwards compatibility, not forwards). That's what caused the problem in this case: rolling forward dependencies was not recommended but didn't break anything, rolling a dependency backwards meant it stopped working, which is something absolutely no-one should be surprised or upset by (at least it meant the build broke as opposed to debian shipping a broken package like would have likely happened with python. It just meant that debian was shipping a version with bugs fixed in upstream). If you're going to do that you've got to be prepared to take responsibility for fixing the problem that you've introduced.
> For other languages like Python, there is not a need and you should not do that.
Normally I would agree, but when it comes to cuda and rocm, and the one that intel uses... you find that you have to pin torch/audio/video, and then you have to start pinning its dependencies (and their dependencies etc etc) or else you start getting random build or runtime failures...
Normally I would agree, but when it comes to cuda and rocm, and the one that intel uses... you find that you have to pin torch/audio/video, and then you have to start pinning its dependencies (and their dependencies etc etc) or else you start getting random build or runtime failures...
Not disagreeing, but IME much more often I have to run something that locks specific versions of their python dependencies because ... well ... why not -- kool kids run everything in a virtual environment anyway.
Version locking is a powerful tool that should be used very sparingly. My 2c.
Version locking is a powerful tool that should be used very sparingly. My 2c.
if you deal with filesystems, you really want strict versions
This is a bad dependency management, nothing more. Vendoring is a PITA.
Level of potentially messing up. If you have a chat app, relax the deps and it crashes - oh well. But if you have a tool which can cause data loss, you really want to be sure everyone's running what you tested.
It's similar to why restic (backup software) has official, static, reproducible binaries in their releases.
It's similar to why restic (backup software) has official, static, reproducible binaries in their releases.
Oh well, it was probably just a crash in my Internet connected software where users can just sent inputs to be processed on my machine.
RCE would have been worse than file system bugs, as it leads directly to a superset of problems including potentially losing all your files.
RCE would have been worse than file system bugs, as it leads directly to a superset of problems including potentially losing all your files.
And if my kernel crashes ?
And if whatever library bugged, oh well, random people can access my systems ?
And if my bgpd bugged, oh well, we just inject random routes et forward data randomly ?
Makes no sense.
And if whatever library bugged, oh well, random people can access my systems ?
And if my bgpd bugged, oh well, we just inject random routes et forward data randomly ?
Makes no sense.
Your kernel has no external library dependencies.
Relaxing library requirements can introduce security issues when the dependency behaviour changes, so I'm not sure what the argument is.
If your bgp is bugged, yeah it sucks, but it's recoverable. We're using encrypted traffic these days to ensure that even captured traffic is safe.
The difference here is that data corruption is not recoverable.
Relaxing library requirements can introduce security issues when the dependency behaviour changes, so I'm not sure what the argument is.
If your bgp is bugged, yeah it sucks, but it's recoverable. We're using encrypted traffic these days to ensure that even captured traffic is safe.
The difference here is that data corruption is not recoverable.
[deleted]
Linux uses at least libc
Linux kernel doesn't use libc. Libc provides the interface for userspace to call the kernel.
It does includes its headers (stdlib.h, at least), in various subsystems.
In user space tools and tests only. The kernel part does not include stdlib.h anywhere. Nowhere where it matters in the context of "And if my kernel crashes?"
You are right, thank you for the insight
Filesystems are complicated and have nasty failure modes. Strict versioning reduces the impact of software development QoI properties, i.e. that basically all code is substantially wrong.
[deleted]
I would expect it to be the opposite (especially for a linux file system): you want to test with as wide a variety of versions as possible, because that what you're going to encounter in the real world. If your code only works in a limited set of cases, what happens when you mount the file system on a different system, is the file system going to be trashed?
These are all compile time dependencies in this case. You're not going to encounter a variety of versions - that's exactly what the strict version bounds are doing. The app will be built with the tested version and everyone gets the same results.
There's at least 3 different versions that can happen (the filesystem version, the kernel version (including stable backports), and the tooling version). I'm presuming udev being a dependency means bcachefs links to that (which will be different on different OS versions).
Those are completely different things than the dependencies you can make a choice about. You have to handle different filesystem/kernel versions regardless - you don't get a choice here.
it is rust specific because rust is the first attemp to replace proper system engineering languages with one that while nicer on memory management and overall ergonomics, is worse in tooking. cargo brings many malpractices from java/JavaScript (maven, npm, etc) that were always shunned in systems engineering and, mark my words, will be a security nightmare for linux in the near future.
Go, D, Ocaml, Nim have the same issue, most are older than Rust and are arguably systems languages.
C/C++ built with Bazel or Conan will have the same issue too.
C/C++ built with Bazel or Conan will have the same issue too.
Go would only be comparable if you had hundreds of vendored dependencies but Go's community and culture have encouraged use of the standard library over external dependencies where possible. A core problem with Rust is the lack of an adequate standard library. The problem with Cargo is when you have an application with hundreds of dependencies. That does not exist in any mainstream Go application.
1. Why Rust Libraries May Never Exist - https://www.youtube.com/watch?v=769VqNup21Q
2. The Rust Cargo ecosystem is broken - https://www.youtube.com/watch?v=OdMQwDtumJ8
1. Why Rust Libraries May Never Exist - https://www.youtube.com/watch?v=769VqNup21Q
2. The Rust Cargo ecosystem is broken - https://www.youtube.com/watch?v=OdMQwDtumJ8
Having a big standard library doesn't help that much since it needs separate updates. Every other golang app I use requires updated crypto, net, or sync. In theory they come in the standard library, but the moment you need updates, they're exactly the same as other dependencies for packaging purposes. (it's very common, yes I do packaging)
Go is unlike other languages mentioned here (and this arguably applies to OCaml too).
As much as its standard library superficially looks close to C, it is as equally unlike C in its performance and abstraction level. It does not have suitable tools to do systems programming properly and its compiler and interop capabilities are too weak.
As much as its standard library superficially looks close to C, it is as equally unlike C in its performance and abstraction level. It does not have suitable tools to do systems programming properly and its compiler and interop capabilities are too weak.
those are userland system applications at most... which i grant is also what the original article talked about, but still.
What makes a system engineering language "proper"? And what languages besides C fulfill those criteria?
The grammar might be confusing, but the comment you're replying to clearly considers rust a (new) proper systems language, and nicer in many ways (memory management and overall ergonomics) than C, but with worse (from the perspective of systems programming) tooling.
thanks for making it more succinct. exactly.
The author would be fine with vendoring. The problem is that Debian doesn't allow it.
> The problem is that Debian doesn't allow it.
Vendoring used to be more common, even in Debian. But then there was an important zlib vulnerability (which IIRC could be exploited through manipulated compressed data), and they had to chase and fix all the many copies of zlib embedded all over the place. To make things worse, some of them were not only old versions of zlib, but also had modified that zlib code, so each had to be reviewed before applying the fix.
Debian (and other traditional distributions) learned from that incident, and started a strong push to not only remove vendored copies of libraries, but also when possible use a single version of these libraries. Which means that, whenever an important security issue is once again found in a widely used library, only a single copy has to be fixed and updated.
Vendoring used to be more common, even in Debian. But then there was an important zlib vulnerability (which IIRC could be exploited through manipulated compressed data), and they had to chase and fix all the many copies of zlib embedded all over the place. To make things worse, some of them were not only old versions of zlib, but also had modified that zlib code, so each had to be reviewed before applying the fix.
Debian (and other traditional distributions) learned from that incident, and started a strong push to not only remove vendored copies of libraries, but also when possible use a single version of these libraries. Which means that, whenever an important security issue is once again found in a widely used library, only a single copy has to be fixed and updated.
I'm interested in hearing more about the history behind Debian and zlib! I did some searching and the closest thing I could find was a nod to the same incident in the Upstream Guide[0]. Do you know of a place where I could read more about it?
[0]: https://wiki.debian.org/UpstreamGuide#No_inclusion_of_third_...
[0]: https://wiki.debian.org/UpstreamGuide#No_inclusion_of_third_...
It's been so long ago, that it's hard to find all the discussions I had seen back then. I recall that it was after a long time without any zlib release, so looking at the zlib history, I think it was this one fixed in zlib 1.1.4 from 11 March 2002: http://www.zlib.org/advisory-2002-03-11.txt
Looking at the debian-devel archives around that date, I found a Debian developer complaining about the vendored zlib copies (https://lists.debian.org/debian-devel/2002/03/msg00716.html), but not the full discussion about getting rid of vendored libraries, so it must have happened elsewhere.
Looking at the debian-devel archives around that date, I found a Debian developer complaining about the vendored zlib copies (https://lists.debian.org/debian-devel/2002/03/msg00716.html), but not the full discussion about getting rid of vendored libraries, so it must have happened elsewhere.
Thanks for taking the time these references up for me!
And that's because the users of Debian are not fine with vendoring.
I think this is the same shape of issue that I’ve experienced with Debian for as long as I’ve used it - close to 15 years now.
Debian is a great OS, but it targets stability and long term support for its releases. That just isn’t compatible with newer, faster moving software that’s still working towards stability. I remember it being an issue when I was playing around with Mono around 2010 ish, and it’s an issue now with bcachefs - a very new and fast moving technology.
For motivated users, the solution is to install bcachefs-tools direct from upstream, or from a third party packager (if one exists). When bcachefs stabilises, I’m sure it’ll find its way back into Debian official.
Debian is a great OS, but it targets stability and long term support for its releases. That just isn’t compatible with newer, faster moving software that’s still working towards stability. I remember it being an issue when I was playing around with Mono around 2010 ish, and it’s an issue now with bcachefs - a very new and fast moving technology.
For motivated users, the solution is to install bcachefs-tools direct from upstream, or from a third party packager (if one exists). When bcachefs stabilises, I’m sure it’ll find its way back into Debian official.
What I think is somewhat interesting in this case (if I've read the post correctly) half the changes were for newer versions of dependencies. If that's the case, then I'm inclined to start wondering what's going on with bcachefs upstream (and the whole dependency tree), if they're not keeping up with versions.
Imagine that an update of bcache-fs is released and 3 dependencies (could even be transitive) were updated and the newer versions were not packaged in Debian. The maintainer now has one package to update but another 3 to package for the first time. That's additional work. AFAIK Fedora has a tool to automatize this work, but maybe Debian has additional requirements that make this harder.
Debian (and derivs) have similar tooling (and the vast majority of rust packages are tracked in a monorepo https://salsa.debian.org/rust-team/debcargo-conf).
My point was "some of the upstream versions were older than Debian's" which means the "Debian is behind, Debian is slow" is not true in the case. That usually only happens where an upstream is either very conservative with bumping versions, or where upstream is under-maintained, neither which at first glance seem to be true of becache-fs.
My point was "some of the upstream versions were older than Debian's" which means the "Debian is behind, Debian is slow" is not true in the case. That usually only happens where an upstream is either very conservative with bumping versions, or where upstream is under-maintained, neither which at first glance seem to be true of becache-fs.
It looks like the two packages that were bumped forward are ones where the changes were very minimal (it's not obvious why errno even bumped their 'major' version, and udev seem to do it on every release even when backwards compatible). Also errno 0.4 doesn't seem to exist, so I think that's a typo.
(There is a somewhat frustrating trend in rust packages for them to sit on 0.x versions for a very long time, even if they're fairly heavily used in the ecosystem and pretty stable in practice)
From what the author of bcachefs is saying, the forward moves were not a problem, it was moving bindgen backwards which broke the build, meaning the packaged version of bcachefs-tools stayed very out of date, causing him to get bug reports for known and fixed fairly serious bugs (ones which rendered machines unbootable).
(There is a somewhat frustrating trend in rust packages for them to sit on 0.x versions for a very long time, even if they're fairly heavily used in the ecosystem and pretty stable in practice)
From what the author of bcachefs is saying, the forward moves were not a problem, it was moving bindgen backwards which broke the build, meaning the packaged version of bcachefs-tools stayed very out of date, causing him to get bug reports for known and fixed fairly serious bugs (ones which rendered machines unbootable).
My read is that whatever subset of the ecosystem bcachefs-tools just hasn’t stabilized. Presumably it will, but once everything is functionally at 1.x even if they don’t reach it officially this will be become a non-issue.
That said, having dabled in rust there’s a temptation to name breaking changes to improve ergonomics because the type system is so expressive. I almost wonder if this is an argument for simpler type systems to reduce churn.
That said, having dabled in rust there’s a temptation to name breaking changes to improve ergonomics because the type system is so expressive. I almost wonder if this is an argument for simpler type systems to reduce churn.
I love Debian for this stability. It is a great host OS for all my VMs.
I agree. Debian is a waterfall OS living in an agile world. There's a fundamental mismatch between Debian's philosophy and the reality of today's open source software ecosystem.
Gentoo has SLOTs and they work well. But for Go and Rust, I think they chose the vendoring path because it's just too damn Don Quixotesque to try to mirror their NPM tier package repositories.
As others have said this isn't just Rust, distro "release" packaging is untenable now for non-base-system packages: there are an ever increasing number of packages, they release with updates users need/want at a rate far faster than the distro release schedule and have too many conflicting dependencies.
To deal with this today you install a stable base system with an overlaid "user" package system so Debian/Fedora + nixpkgs/guix/brew/snap/flatpak/mpr/pip/cargo/etc. Unfortunately because because although you can get most packages on nixpkgs you can't get all and you'll need to install multiple so it becomes a nightmare to maintain (how do you remember all the package managers you have to update during the xz vulnerability?) and extremely bloated due to duplicated dependencies (especially for GUI packages that require gnome or kde).
You do get pretty far for CLI-only packages by just adding nixpkgs though. Too bad it's so terrible to use.
MPR is also pretty interesting: You basically leverage the AUR for Debian and with a few changes you could probably make the dependency names translation automatic and transparent. It solves the bloat problem (since you'll mostly use system packages) but doesn't help with library versions.
To deal with this today you install a stable base system with an overlaid "user" package system so Debian/Fedora + nixpkgs/guix/brew/snap/flatpak/mpr/pip/cargo/etc. Unfortunately because because although you can get most packages on nixpkgs you can't get all and you'll need to install multiple so it becomes a nightmare to maintain (how do you remember all the package managers you have to update during the xz vulnerability?) and extremely bloated due to duplicated dependencies (especially for GUI packages that require gnome or kde).
You do get pretty far for CLI-only packages by just adding nixpkgs though. Too bad it's so terrible to use.
MPR is also pretty interesting: You basically leverage the AUR for Debian and with a few changes you could probably make the dependency names translation automatic and transparent. It solves the bloat problem (since you'll mostly use system packages) but doesn't help with library versions.
Shouldn't the filesystem utilities be base-system packages, though? Like e2fsprogs, xfsprogs ... I guess rust just isn't appropriate for base-system packages, because the dependencies move too fast, and so rust components need to depend on different specific versions of all dependencies, and are "horrifically out-of-date" in just a few months ... and rust developers wouldn't stoop so low as to recognize that e.g. bindgen is a particularly hairy and touchy dependency so they should just vendor the bindings output by bindgen, rather than the entire bindgen tool plus all other dependencies in full ...
I'm not that familiar with bcachefs-tools and was speaking more generally but I thought bcachefs-tools wasn't critical for normal booting (On Ubuntu 24.04 nothing seems to depend on that package and it seems optional. The same for btrfs-tools). Some other comments suggested you need it to mount a degraded file system but it seems to me that is a case where either 1. your system can't boot and you need external recovery boot tools anyway or 2. you can boot and can run whatever tools you want (you can install the latest version of bcachefs-tools via nixpkgs for example). The real problem there seems to be that the program was silently broken due to the changes made to get it to build on Debian.
Though I agree that it is closer to a system-critical package than most packages.
Though I agree that it is closer to a system-critical package than most packages.
IIRC the brokenness was simply due to a bug in an upstream release of bindgen, which "was quickly fixed upstream, but as far as I know it's still broken in Debian" (which must have some reason of not updating bindgen again ...) so it wasn't some subtle incompatibility or modification to bcachefs-tools IMHO, it was just bindgen.
> and rust developers wouldn't stoop so low as to recognize that e.g. bindgen is a particularly hairy and touchy dependency so they should just vendor the bindings output by bindgen
Why can't Debian do that though?
Moreover vendoring is generally considered a bad practice, more so by Debian, so this just seems a step backwards.
Why can't Debian do that though?
Moreover vendoring is generally considered a bad practice, more so by Debian, so this just seems a step backwards.
> Developer on reddit clarifies that it builds and does the wrong thing, and recommends dropping Debian as a solutio
Can you link to the post on Reddit you are referring to?
Can you link to the post on Reddit you are referring to?
I think this is the comment:
https://www.reddit.com/r/bcachefs/comments/1f4erbg/comment/l...
https://www.reddit.com/r/bcachefs/comments/1f4erbg/comment/l...
> rust insists on a very specific version of various dependencies
It only insists on semver-compatible versions (if a Rust/Cargo package specifies libfoo=5.1, it will work with libfoo=5.9). It's one per major version, not that different from Debian packaging "libfoo5" and "libfoo7" when both are needed.
The difference is that Cargo unifies versions by updating them to the newest compatible, and Debian unifies them by downgrading them to old unsupported versions, ignores compatibility, and reintroduces bugs, and disables required functionality.
It only insists on semver-compatible versions (if a Rust/Cargo package specifies libfoo=5.1, it will work with libfoo=5.9). It's one per major version, not that different from Debian packaging "libfoo5" and "libfoo7" when both are needed.
The difference is that Cargo unifies versions by updating them to the newest compatible, and Debian unifies them by downgrading them to old unsupported versions, ignores compatibility, and reintroduces bugs, and disables required functionality.
Related discussion in the bcachefs subreddit: https://www.reddit.com/r/bcachefs/comments/1f4erbg/debiansta...
Kent's reply in that thread has more details.
Kent's reply in that thread has more details.
This is pretty dumb on Debian's part. First of all I don't understand why they insist crate dependencies must be pulled from their repository. They are just source code, not built binary. AFAIK there is no other distro that does this, what they do is that they would download crates from crates.io (`cargo vendor` is a command that does this automatically) and build against that. Arch does this, Gentoo does this, NixOS does this, why does Debian has to be different?
Secondly, even if they have to use crates from their repository, I don't understand what's so hard to just have multiple versions of the same crate? That will solve the problem too.
This is just all-around weird what Debian is doing.
(Full disclosure, I am the one who introduced the first piece of Rust code into bcachefs-tools)
Secondly, even if they have to use crates from their repository, I don't understand what's so hard to just have multiple versions of the same crate? That will solve the problem too.
This is just all-around weird what Debian is doing.
(Full disclosure, I am the one who introduced the first piece of Rust code into bcachefs-tools)
Debian has a Social Contract[1] as well as guidelines {the DFSG}[2] regarding the commitment to only distribute free and open source software that all package maintainers must adhere to. This means that package maintainers must check the licenses of source code and documentation files, clear up any ambiguities by talking to upstream, and (as a last resort) even excise code and/or documentation from Debian's copy of the codebase if it doesn't meet the requirements of the DFSG.
In practise, this means that Debian has to make its own copy of the source code available from a Debian-controlled repository, to ensure that no (accidental or otherwise) change to an upstream source archive can cause non-DFSG compliant Debian source or binary packages to be distributed.
[1] https://www.debian.org/social_contract
[2] https://wiki.debian.org/DebianFreeSoftwareGuidelines
In practise, this means that Debian has to make its own copy of the source code available from a Debian-controlled repository, to ensure that no (accidental or otherwise) change to an upstream source archive can cause non-DFSG compliant Debian source or binary packages to be distributed.
[1] https://www.debian.org/social_contract
[2] https://wiki.debian.org/DebianFreeSoftwareGuidelines
That justification doesn't work here - I provide vendored tarballs, so the source code availability argument is moot.
But it’s not guaranteed. The Debian way provides a method of allocating responsibility. So if anything does go wrong they can point to a responsible party, the package maintainer. By providing tarball source you’re trying to placate responsibility of some code. You could build those tarballs on a different machine/different OS and any issues wouldn’t technically be your problem because “it’s just deps”.
> Arch does this, Gentoo does this, NixOS does this, why does Debian has to be different?
I say this as someone who ran Gentoo for years and daily drives Arch today.
Because sometimes you don't want entire swaths of your server being rebuilt/tinkered with on a regular basis under the hood. "Move fast, break everything" is great in dev/test land, or a prod environment where the entire fleet is just containers treated like cattle, but contrary to what the SREs of the valley would believe, there's a whole ecosystem of 'stuff' out there that will never be containerized, where servers are still treated like pets, or rather, at least "cherished mules", that just do their job 24/7 and get the occasional required security updates/patches and then go right back to operating the same way they did last year.
I say this as someone who ran Gentoo for years and daily drives Arch today.
Because sometimes you don't want entire swaths of your server being rebuilt/tinkered with on a regular basis under the hood. "Move fast, break everything" is great in dev/test land, or a prod environment where the entire fleet is just containers treated like cattle, but contrary to what the SREs of the valley would believe, there's a whole ecosystem of 'stuff' out there that will never be containerized, where servers are still treated like pets, or rather, at least "cherished mules", that just do their job 24/7 and get the occasional required security updates/patches and then go right back to operating the same way they did last year.
> AFAIK there is no other distro that does this, what they do is that they would download crates from crates.io (`cargo vendor` is a command that does this automatically) and build against that.
AFAIK, most traditional distributions do that, not just Debian. They consider it important that software can be rebuilt, even in the far future, with nothing more than a copy of the distribution's binary and source packages. Doing anything which depends on network access during a build of the software is verboten (and AFAIK the automated build hosts block the network to enforce that requirement).
Keep also in mind that these distributions are from before our current hyper-connected time; it was common for a computer to be offline most of the time, and only dial up to the Internet when necessary. You can still download full CD or DVD sets containing all of the Debian binaries and source code, and these should be enough to rebuild anything from that distribution, even on an air-gaped computer.
> Secondly, even if they have to use crates from their repository, I don't understand what's so hard to just have multiple versions of the same crate? That will solve the problem too.
That is often done for C libraries; for instance, Debian stable has both libncurses5 and libncurses6 packages. But it's a lot of work, since for technical reasons, each version has to be an independent package with a separate name, and at least for Debian, each new package has to be reviewed by the small ftpmaster team before being added to the distribution. I don't know whether there's anything Rust-specific that makes this harder (for C libraries, the filenames within the library packages are different, and the -dev packages with the headers conflict with each other so only one can be installed at a time).
There's also the issue that having multiple versions means maintaining multiple versions (applying security fixes and so on).
AFAIK, most traditional distributions do that, not just Debian. They consider it important that software can be rebuilt, even in the far future, with nothing more than a copy of the distribution's binary and source packages. Doing anything which depends on network access during a build of the software is verboten (and AFAIK the automated build hosts block the network to enforce that requirement).
Keep also in mind that these distributions are from before our current hyper-connected time; it was common for a computer to be offline most of the time, and only dial up to the Internet when necessary. You can still download full CD or DVD sets containing all of the Debian binaries and source code, and these should be enough to rebuild anything from that distribution, even on an air-gaped computer.
> Secondly, even if they have to use crates from their repository, I don't understand what's so hard to just have multiple versions of the same crate? That will solve the problem too.
That is often done for C libraries; for instance, Debian stable has both libncurses5 and libncurses6 packages. But it's a lot of work, since for technical reasons, each version has to be an independent package with a separate name, and at least for Debian, each new package has to be reviewed by the small ftpmaster team before being added to the distribution. I don't know whether there's anything Rust-specific that makes this harder (for C libraries, the filenames within the library packages are different, and the -dev packages with the headers conflict with each other so only one can be installed at a time).
There's also the issue that having multiple versions means maintaining multiple versions (applying security fixes and so on).
> There's also the issue that having multiple versions means maintaining multiple versions (applying security fixes and so on).
This is the most important part. Debian LTS maintains packages for 5 years. Canonical takes Debian sources, and offers to maintain their LTS for 10 years. Red Hat also promises 10 years of support. They don't want anything in the core part of their stable branches that they can't promise to maintain for the next 5-10 years, when they have no assurance that upstream will even exist that long.
If you want to move fast and break things, that's also fine. Just build and distribute your own .deb or .rpm. No need to bother distro maintainers who are already doing so much thankless work.
This is the most important part. Debian LTS maintains packages for 5 years. Canonical takes Debian sources, and offers to maintain their LTS for 10 years. Red Hat also promises 10 years of support. They don't want anything in the core part of their stable branches that they can't promise to maintain for the next 5-10 years, when they have no assurance that upstream will even exist that long.
If you want to move fast and break things, that's also fine. Just build and distribute your own .deb or .rpm. No need to bother distro maintainers who are already doing so much thankless work.
I didn't get the impression anyone bothered the maintainers to do this, the article implies they chose to package it on their own accord.
Yes, and they chose to stop packaging it, again on their own accord.
Except this time, they seem to have made the decision because upstream was bothering them too much.
Except this time, they seem to have made the decision because upstream was bothering them too much.
No arguments there, I'm more talking what that means for the "build and distribute your own dev/rpm" part that follows. Why are the only options "do work for maintainers" or "provide a prebuilt package for the distro", what happened to "nobody said this needed to be done yet"?
No problem, if upstream doesn't want their software packaged for Distro X, nobody needs to do anything.
The thing about Linux filesystems, though, is that they consist of two parts: the kernel patch and the userspace tooling. Bcachefs is already in the kernel, so it's a bit awkward to leave out bcachefs-tools. Which is probably why it got packaged in the first place. Stable distros generally don't want loose ends flailing about in such a critical part of their system. If nobody wants to maintain bcachefs-tools for Debian, Debian will probably remove bcachefs entirely from their kernel as well.
The thing about Linux filesystems, though, is that they consist of two parts: the kernel patch and the userspace tooling. Bcachefs is already in the kernel, so it's a bit awkward to leave out bcachefs-tools. Which is probably why it got packaged in the first place. Stable distros generally don't want loose ends flailing about in such a critical part of their system. If nobody wants to maintain bcachefs-tools for Debian, Debian will probably remove bcachefs entirely from their kernel as well.
I’d consider the issue to be the opposite. Why does every programming language now have a package manager and all of the infrastructure around package management rather than rely on the OS package manager? As a user I have to deal with apt, ports, pkg, opkg, ipkg, yum, flatpak, snap, docker, cpan, ctan, gems, pip, go modules, cargo, npm, swift packages, etc., etc., which all have different opinions of how and where to package files.
On packaged operating systems (Debian, FreeBSD) - you have the system’s package manager to deal with (apt, pkg respectively). I can have an offline snapshot of _all_ packages that can be mirrors from one place.
IMHO, every programming language having its own package system is the weird thing.
On packaged operating systems (Debian, FreeBSD) - you have the system’s package manager to deal with (apt, pkg respectively). I can have an offline snapshot of _all_ packages that can be mirrors from one place.
IMHO, every programming language having its own package system is the weird thing.
If you are a developer you almost always eventually need some dependencies that don’t ship with the os package manager, and once some of your dependencies are upstream source, you very quickly find that some dependencies of the sources you download rely on features from newer versions of libraries. If you have multiple clients, you may also need to support both old and new versions of the same dependencies depending on who the work is for. Package managers for a Linux distribution have incompatible goals to these (except maybe nix)
We want to make our software available to any system without every library maintainer being a packaging expert in every system.
The user experience is much better when working within tkese packaging systems.
You can control versions of software independent of the machine (or what distros ship).
Or in other words, the needs of software development and software distribution are different. You can squint and see similarities but the fill different roles.
The user experience is much better when working within tkese packaging systems.
You can control versions of software independent of the machine (or what distros ship).
Or in other words, the needs of software development and software distribution are different. You can squint and see similarities but the fill different roles.
So every user has to be an expert in every package manager instead? Makes sense. Make life easy for the developer and pass the pain on to thousands of users. 20 years ago you may or may not support RPM and DEB and for everyone else a tarball with a make file that respected PREFIX was enough. (Obviously a tarball doesn’t support dependencies.)
Why would users need to be an expert in them?
[deleted]
Because OS packaging stuff sucks. It adds an enourmous barrier to sharing and publishing stuff.
Imagine that I make a simple OS-agnostic library in some programming language and want to publish it to allow others to use it. Do I need to package for every possible distro? That's a lot of work, and might still not cover everyone. And consider that I might not even use Linux!
A programming language will never get successful if that is what it takes to built up a community.
Moreover in the case of Rust distos are not even forced to build using crates.io. However the downside is that they have to package every single dependency version required, which due to the simplicity of publishing and updating them have become quite a lot and change much often than they would like.
The funny thing is that in the C/C++ world it's common to reimplement functionality due to the difficulty of using some dependencies for them. The result is not really different from vendoring dependencies, except for the reduced testing of those components, and this is completly acceptable to distros compared to vendoring. It makes no sense!
Imagine that I make a simple OS-agnostic library in some programming language and want to publish it to allow others to use it. Do I need to package for every possible distro? That's a lot of work, and might still not cover everyone. And consider that I might not even use Linux!
A programming language will never get successful if that is what it takes to built up a community.
Moreover in the case of Rust distos are not even forced to build using crates.io. However the downside is that they have to package every single dependency version required, which due to the simplicity of publishing and updating them have become quite a lot and change much often than they would like.
The funny thing is that in the C/C++ world it's common to reimplement functionality due to the difficulty of using some dependencies for them. The result is not really different from vendoring dependencies, except for the reduced testing of those components, and this is completly acceptable to distros compared to vendoring. It makes no sense!
1. Because Windows/macOS/iOS/Android don't have a built-in package manager at the same granularity of individual libraries, but modern programming languages still want to have first-class support for all these OSes, not just smugly tell users their OS is inferior.
2. Because most Linux distros can only handle very primitive updates based on simple file overwrites, and keep calling everything impossible to secure if it can't be split and patched within limitations of C-oriented dynamic linker.
3. Because Linux distros have a very wide spread of library versions they support, and they often make arbitrary choices on which versions and which features are allowed, which is a burden for programmers who can't simply pick a library and use it, and need to deal with extra compatibility matrix of outdated buggy versions and disabled features.
From developer perspective with lang-specific packages
• Use 1 or 2 languages in the project, and only need to deal a couple of package repositories, which give the exact deps they want, and it works the same on every OS, including cross-compilation to mobile.
From developer perspective of using OS package managers:
• Different names of packages on each Linux distro, installed differently with different commands. There's no way to specify deps in a universal way. Each distro has a range of LTS/stable/testing flavors, each with a different version of library. Debian has super old useless versions that are so old it's worse than not having them, plus bugs reintroduced by removal of vendored patches.
• macOS users may not have any package manager, may have an obscure one, and even if they have the popular Homebrew, there's no guarantee they have the libs you want installed and kept up-to-date. pkg-config will give you temporary paths to precise library version, and unless you work around that, your binary will break when the lib is updated.
• Windows users are screwed. There are several fragmented package managers, which almost nobody has installed. They have few packages, and there's a lot of fiddly work required to make anything build and install properly.
• Supporting mobile platforms means cross-compilation, and you can't use your OS's package manager.
OS-level packaging suuuuuuuucks. When people say that dependency management in C and C++ is a nightmare, they mean the OS-level package managers are a nightmare.
2. Because most Linux distros can only handle very primitive updates based on simple file overwrites, and keep calling everything impossible to secure if it can't be split and patched within limitations of C-oriented dynamic linker.
3. Because Linux distros have a very wide spread of library versions they support, and they often make arbitrary choices on which versions and which features are allowed, which is a burden for programmers who can't simply pick a library and use it, and need to deal with extra compatibility matrix of outdated buggy versions and disabled features.
From developer perspective with lang-specific packages
• Use 1 or 2 languages in the project, and only need to deal a couple of package repositories, which give the exact deps they want, and it works the same on every OS, including cross-compilation to mobile.
From developer perspective of using OS package managers:
• Different names of packages on each Linux distro, installed differently with different commands. There's no way to specify deps in a universal way. Each distro has a range of LTS/stable/testing flavors, each with a different version of library. Debian has super old useless versions that are so old it's worse than not having them, plus bugs reintroduced by removal of vendored patches.
• macOS users may not have any package manager, may have an obscure one, and even if they have the popular Homebrew, there's no guarantee they have the libs you want installed and kept up-to-date. pkg-config will give you temporary paths to precise library version, and unless you work around that, your binary will break when the lib is updated.
• Windows users are screwed. There are several fragmented package managers, which almost nobody has installed. They have few packages, and there's a lot of fiddly work required to make anything build and install properly.
• Supporting mobile platforms means cross-compilation, and you can't use your OS's package manager.
OS-level packaging suuuuuuuucks. When people say that dependency management in C and C++ is a nightmare, they mean the OS-level package managers are a nightmare.
Programming language package managers are more for development than deployment.
Yet as a user who programs only in a handful of those languages, I’ve used them all and continue to come across new ones.
>AFAIK there is no other distro that does this, what they do is that they would download crates from crates.io (`cargo vendor` is a command that does this automatically) and build against that.
Note your examples are all bleeding edge / rolling distributions. Debian and the non-bleeding edge distributions go a different route and focus on reproduce-ability and security, among other things.
With the "get from crates.io" route, if someone compromises/hijacks a crate upstream, you're in trouble, immediately. By requiring vendoring of sources, that requires at least some level of manual actions by maintainers to get that compromised source in to the debian repositories to then be distributed out to the users. As you get in towards distributions like RHEL, they get even more cautious on this front.
Note your examples are all bleeding edge / rolling distributions. Debian and the non-bleeding edge distributions go a different route and focus on reproduce-ability and security, among other things.
With the "get from crates.io" route, if someone compromises/hijacks a crate upstream, you're in trouble, immediately. By requiring vendoring of sources, that requires at least some level of manual actions by maintainers to get that compromised source in to the debian repositories to then be distributed out to the users. As you get in towards distributions like RHEL, they get even more cautious on this front.
NixOS is not a rolling distro.
debian is substantially older than all of those distros, and you named three that happen to, in my view, been designed specifically in reaction against the debian style of maintenance (creating an harmonious, stable set of packages that's less vulnerable to upstream changes), so it's strange to say that debian is the odd one out
keeping a debian-hosted copy of all source used to build packages seems like a very reasonable defensive move to minimize external infrastructure dependencies and is key in the reproducible builds process
there's definitely a conflict with the modern style of volatile language-specific package management, and I don't think debian's approach is ideal, but there's a reason so many people use debian as a base system
also it seems like the idea of maintaining stable branches of software has fallen out of vogue in general
keeping a debian-hosted copy of all source used to build packages seems like a very reasonable defensive move to minimize external infrastructure dependencies and is key in the reproducible builds process
there's definitely a conflict with the modern style of volatile language-specific package management, and I don't think debian's approach is ideal, but there's a reason so many people use debian as a base system
also it seems like the idea of maintaining stable branches of software has fallen out of vogue in general
Perhaps they want to have builds they can reproduce years later?
When libraries are removed you have to update your build. This will not be the case if you use your own infrastructure (crates.io mirror).
When libraries are removed you have to update your build. This will not be the case if you use your own infrastructure (crates.io mirror).
Part of a distros job is to audit all the packages (even if just to a minimal extent), and in many cases patch for various reasons. This is much harder if the source is external and there are N copies of everything.
Because Debian and similar distros have a goal of maintaining all of the software that users are expected to use. And this means they commit to fixing security issues in every single piece of software they distribute.
A consequence of this is that they need to fix security issues in every library version that any application they distribute uses, including any statically-linked library. So, if they allow 30 applications to each have their own version of the same library, even Rust crates, and those versions all have a security issue, then the Debian team needs to find or patch themselves 30 different pieces of code. If instead they make sure that all 30 of those applications use the same version of a Rust crate, then they only need to patch one version. Maybe it's not 30 times less work, but it's definitely 10 times work. At the size of the Debian repos, this is an extremely significant difference.
Now, it could be that this commitment from Debian is foolish and should be done away with. I certainly don't think my OS vendor should be the one that maintains all the apps I use - I don't even understand the attraction of that. I do want the OS maintainer to handle the packaging of the base system components, and patch those as needed for every supported version and so on - and so I understand this requirement for them. And I would view bcachefs-tools as a base system component, so this requirement seems sane for it.
A consequence of this is that they need to fix security issues in every library version that any application they distribute uses, including any statically-linked library. So, if they allow 30 applications to each have their own version of the same library, even Rust crates, and those versions all have a security issue, then the Debian team needs to find or patch themselves 30 different pieces of code. If instead they make sure that all 30 of those applications use the same version of a Rust crate, then they only need to patch one version. Maybe it's not 30 times less work, but it's definitely 10 times work. At the size of the Debian repos, this is an extremely significant difference.
Now, it could be that this commitment from Debian is foolish and should be done away with. I certainly don't think my OS vendor should be the one that maintains all the apps I use - I don't even understand the attraction of that. I do want the OS maintainer to handle the packaging of the base system components, and patch those as needed for every supported version and so on - and so I understand this requirement for them. And I would view bcachefs-tools as a base system component, so this requirement seems sane for it.
It's pretty dumb (your words if you don't like them) not to understand (your words if you don't like them) that, how, and why, different distributions "has to be different". Debian is not Arch or Nix. A tractor is not a race car is not a submarine, even though they are all vehicles.
This seems like a very basic concept for anyone purporting to do any sort of engineering or designing in any field.
If it were me, I would not be so eager to advertize my failure to understsnd such basics, let alone presume to call anyone else pretty dumb.
This seems like a very basic concept for anyone purporting to do any sort of engineering or designing in any field.
If it were me, I would not be so eager to advertize my failure to understsnd such basics, let alone presume to call anyone else pretty dumb.
The same bcachefs that Linus just ripped apart for inconsiderate and irresponsible PRs?
Sensing a pattern and it's not both Debian and the kernel. I mean can you even point at 2 more proven solid large and long term projects that have proven how to do it right?
Sensing a pattern and it's not both Debian and the kernel. I mean can you even point at 2 more proven solid large and long term projects that have proven how to do it right?
this is "static vs shared libraries" fight again. no one remembers the first iterations, and there's more layers of "stuff" in the way now, so now one sees the actual issue for what it is.
"shared libs" was the product of storage constraints; a bobble in the smooth pace of progress that shouldn't be needed now. Our faith in "the way things are done now is the right way" and our (justified) fear of messing with the foundations of running systems will make excising the notion take longer.
"shared libs" was the product of storage constraints; a bobble in the smooth pace of progress that shouldn't be needed now. Our faith in "the way things are done now is the right way" and our (justified) fear of messing with the foundations of running systems will make excising the notion take longer.
It’s rather the “vendoring is hostile to distros” fight. It’s adjacent to the shared vs static one but not the same: it’s absolutely possible to have statically linked binaries tracking distro-wide dependency versions, provided the distro’s build system is robust and automated enough. Not all are.
"Vendoring is hostile to distros" in turn directly leads to "loose dependencies are hostile to upstream developers".
Distros want to be able to mix & match upstream code with whatever version of the dependency they happen to have lying around in a drawer somewhere. Understandable, as the alternative is having to support dozens of versions of the same library.
Upstream developers want to use a single fixed version of their dependencies for each release. Understandable, as the alternative is having to support support a massive test matrix for every possible version of every dependency, just on the odd chance that some obscure distro wants to mix a brand-new libfoo with a 5-year-old libbar.
And all of this boils down to "versioning is actually really hard", because even with SemVer one's bug is another's feature[0] so there's still no unambiguous way to choose a version number, which in turn means upstream can't choose a version range for their dependencies and be reasonably sure that simply testing with the latest version won't result in breakage with other versions in that range.
If dependency management was easy, we would've solved it decades ago.
[0]: https://xkcd.com/1172/
Distros want to be able to mix & match upstream code with whatever version of the dependency they happen to have lying around in a drawer somewhere. Understandable, as the alternative is having to support dozens of versions of the same library.
Upstream developers want to use a single fixed version of their dependencies for each release. Understandable, as the alternative is having to support support a massive test matrix for every possible version of every dependency, just on the odd chance that some obscure distro wants to mix a brand-new libfoo with a 5-year-old libbar.
And all of this boils down to "versioning is actually really hard", because even with SemVer one's bug is another's feature[0] so there's still no unambiguous way to choose a version number, which in turn means upstream can't choose a version range for their dependencies and be reasonably sure that simply testing with the latest version won't result in breakage with other versions in that range.
If dependency management was easy, we would've solved it decades ago.
[0]: https://xkcd.com/1172/
Shared libraries also mean that you can patch a large number of programs, simply by updating the shared library.
If you have a large number of statically linked program, you need to recompile all of them, which means that you need to know exactly which applications uses the affected library, when a bug shows up, and recompile all those applications and redeploy them.
Without knowing, I imagine that's also partly why Debian wants to have all programs use dependencies installed via APT, it makes it easier to keep track of which other packages needs to be rebuilt.
Personally I'm a huge fan of having applications ship with all their dependencies, like statically compile Go, Rust or Java jar/war/ear files or even containers. It's super easy to deploy, but you do need to be constantly rebuilding and keeping track of dependencies, and I'm not seeing a ton of people doing that.
For Python programs we use apt as a package manager for libraries, because that removed the burden of keeping on top of security issues, to some extend. We do sometimes need to back-port or build our own packages, but try to push those upstream whenever possible.
If you have a large number of statically linked program, you need to recompile all of them, which means that you need to know exactly which applications uses the affected library, when a bug shows up, and recompile all those applications and redeploy them.
Without knowing, I imagine that's also partly why Debian wants to have all programs use dependencies installed via APT, it makes it easier to keep track of which other packages needs to be rebuilt.
Personally I'm a huge fan of having applications ship with all their dependencies, like statically compile Go, Rust or Java jar/war/ear files or even containers. It's super easy to deploy, but you do need to be constantly rebuilding and keeping track of dependencies, and I'm not seeing a ton of people doing that.
For Python programs we use apt as a package manager for libraries, because that removed the burden of keeping on top of security issues, to some extend. We do sometimes need to back-port or build our own packages, but try to push those upstream whenever possible.
This is unrelated to shared libraries. Those strict versions are for compile-time dependencies.
I would rather something not be packaged at all than packaged badly; this whole experience reads like a lesson in what not to do.
What the author left out in his blog post is that I specifically explained why this was going to be an issue and what was going to happen when he and I first talked about Debian packaging; and why for a package that needs to work correctly for the system to boot (fsck, mount helper, encryption unlock), playing these kinds of games with swapping dependencies out was not a good idea.
But with Debian and Fedora people alike it's been like talking to a brick wall. "No, your concerns don't matter, this is our policy and we're going to do what we want with your code". They've made it very clear that I have no voice or say in how they package it.
So then he breaks the build by switching to the packaged version of bindgen, an old unsupported version.
And he makes no attempt to debug, doesn't even send me the build error until months later.
As a result, Debian users were stuck on a broken version of -tools that wasn't passing mount options correctly, which meant that when a drive died and they needed to mount in degraded mode, they weren't able to. This happened to multiple people - that bothered to report it. This was quickly fixed upstream, but as far as I know it's still broken in Debian; and Jonathan Carter wasn't fielding any of those bug reports, I was.
I'm tired, frustrated and demoralized by this drama and tired of constantly having to respond to it. This has taken up an enormous amount of my time.
I really wish this guy hadn't packaged -tools for Debian in the first place. I never asked for it. I have enough on my plate to do without a bunch of distro drama to deal with as well.
What the author left out in his blog post is that I specifically explained why this was going to be an issue and what was going to happen when he and I first talked about Debian packaging; and why for a package that needs to work correctly for the system to boot (fsck, mount helper, encryption unlock), playing these kinds of games with swapping dependencies out was not a good idea.
But with Debian and Fedora people alike it's been like talking to a brick wall. "No, your concerns don't matter, this is our policy and we're going to do what we want with your code". They've made it very clear that I have no voice or say in how they package it.
So then he breaks the build by switching to the packaged version of bindgen, an old unsupported version.
And he makes no attempt to debug, doesn't even send me the build error until months later.
As a result, Debian users were stuck on a broken version of -tools that wasn't passing mount options correctly, which meant that when a drive died and they needed to mount in degraded mode, they weren't able to. This happened to multiple people - that bothered to report it. This was quickly fixed upstream, but as far as I know it's still broken in Debian; and Jonathan Carter wasn't fielding any of those bug reports, I was.
I'm tired, frustrated and demoralized by this drama and tired of constantly having to respond to it. This has taken up an enormous amount of my time.
I really wish this guy hadn't packaged -tools for Debian in the first place. I never asked for it. I have enough on my plate to do without a bunch of distro drama to deal with as well.
> I would rather something not be packaged at all than packaged badly; this whole experience reads like a lesson in what not to do.
You might consider adding a big warning on your official documentation about unsupported distribution packages.
Add links to relevant issue tracker/bug reports or mailing list discussions saying until So and so issue are resolved, the official statement is that distribution package is unsupported, not recommended and deemed dangerous to use.
This is as much leverage as you can have with the distribution community. Then wait for them to upstream patches and attempt to fix the issues. Accept fixes as you consider appropriate or not.
It's also important to at least respect the distribution's opinions in regards to having only one version of a library's major release . Just respect and understanding, you don't have to agree.
Also, all the problematic libs cited by the packager are 0.x.x numbered which sounds like a very young and immature ecosystem of dependencies. Of course, this is bound to cause pain to packagers. I think this speaks volumes about the high level of interest for bcachefs, that they actually tried to make it work instead of not bothering.
You might consider adding a big warning on your official documentation about unsupported distribution packages.
Add links to relevant issue tracker/bug reports or mailing list discussions saying until So and so issue are resolved, the official statement is that distribution package is unsupported, not recommended and deemed dangerous to use.
This is as much leverage as you can have with the distribution community. Then wait for them to upstream patches and attempt to fix the issues. Accept fixes as you consider appropriate or not.
It's also important to at least respect the distribution's opinions in regards to having only one version of a library's major release . Just respect and understanding, you don't have to agree.
Also, all the problematic libs cited by the packager are 0.x.x numbered which sounds like a very young and immature ecosystem of dependencies. Of course, this is bound to cause pain to packagers. I think this speaks volumes about the high level of interest for bcachefs, that they actually tried to make it work instead of not bothering.
0.2 and 0.4 are different "major releases" of rust crates as you say. The major release is determined by the first non-0 component in the version number. The issue is that debian appears to only allow one version even if there are multiple major versions.
If debian is fine with packaging versions 2.0 and 4.0 but not 0.2 and 0.4, then debian does not understand rust version numbers.
If debian is fine with packaging versions 2.0 and 4.0 but not 0.2 and 0.4, then debian does not understand rust version numbers.
Debian allows this. However, it's extra work for package maintainers and extra headache on the security front.
nah, I'm not taking the fight to Jörg Schilling levels :)
Thanks for taking the time to comment here!
> So then he breaks the build by switching to the packaged version of bindgen, an old unsupported version.
Could bcachefs-tools solve this problem upstream of distros by releasing a source tarball with the bindings already generated, so that you can use whatever version of bindgen you want, and bindgen is not needed at all when building from this release tarball?
This would be similar in nature to traditional autotools approach where some amount of pre-compilation is done to produce a release tarball from what actually lives in the source repository.
It's also what I do for some of my ffi crates. I have a script in the root of the repo which regenerates the bindings, but when published to crates.io, the bindings are already generated and bindgen is not part of the build downstream of my crate.
> So then he breaks the build by switching to the packaged version of bindgen, an old unsupported version.
Could bcachefs-tools solve this problem upstream of distros by releasing a source tarball with the bindings already generated, so that you can use whatever version of bindgen you want, and bindgen is not needed at all when building from this release tarball?
This would be similar in nature to traditional autotools approach where some amount of pre-compilation is done to produce a release tarball from what actually lives in the source repository.
It's also what I do for some of my ffi crates. I have a script in the root of the repo which regenerates the bindings, but when published to crates.io, the bindings are already generated and bindgen is not part of the build downstream of my crate.
I already provide tarballs with vendored dependencies - I did this for Fedora, so that package builds wouldn't have to download anything.
I'm not going to special case the bindgen bindings because this really isn't a bindgen specific issue - this could just as easily been any other dependency.
I'm not going to special case the bindgen bindings because this really isn't a bindgen specific issue - this could just as easily been any other dependency.
Speaking from practical experience, bindgen is a particularly heavyweight dependency. It requires libclang to be available and also adds a considerable amount of time to the compile. If it makes compiling the software easier - and end users compiling the software themselves would benefit here as well as distro maintainers - then it would seem like a pragmatic move to me.
This really isn't about bindgen.
Any time a dependency is changed, the package version should be bumped and retested. This is a massive improvement in reproducibility, bisectability and our ability to QA.
Before this model, bisecting breakage that was a result of a change in library 'a' breaking something in consumer 'b' was effectively impossible; now it is.
What Debian and Fedora have been doing is a big step backwards, this whole 'unbundling of dependencies' (that were going to be statically linked anyways) needs to die.
Any time a dependency is changed, the package version should be bumped and retested. This is a massive improvement in reproducibility, bisectability and our ability to QA.
Before this model, bisecting breakage that was a result of a change in library 'a' breaking something in consumer 'b' was effectively impossible; now it is.
What Debian and Fedora have been doing is a big step backwards, this whole 'unbundling of dependencies' (that were going to be statically linked anyways) needs to die.
> What Debian and Fedora have been doing is a big step backwards, this whole 'unbundling of dependencies' (that were going to be statically linked anyways) needs to die.
They, and most others, have been doing that since their respectively beginnings, and for good reasons: everybody expects distros to fix security issues, which is greatly aided by ensuring an-as-small-as-necesary dep tree and thus a single version of any particular dep.
Changing that isn't impossible, but I am not aware of any distro that's managing this and be widely considered 'stable'.
How can distros change while keeping up their end which is stability and robustness in the face of bugs and security issues?
They, and most others, have been doing that since their respectively beginnings, and for good reasons: everybody expects distros to fix security issues, which is greatly aided by ensuring an-as-small-as-necesary dep tree and thus a single version of any particular dep.
Changing that isn't impossible, but I am not aware of any distro that's managing this and be widely considered 'stable'.
How can distros change while keeping up their end which is stability and robustness in the face of bugs and security issues?
> They, and most others, have been doing that since their respectively beginnings
As I explained on another comment of mine (https://news.ycombinator.com/item?id=41409199), it's not since their beginning, but since a particular zlib security incident which showed them the risks of vendored libraries. Any discussion of vendoring policy which does not consider that incident is necessarily incomplete.
As I explained on another comment of mine (https://news.ycombinator.com/item?id=41409199), it's not since their beginning, but since a particular zlib security incident which showed them the risks of vendored libraries. Any discussion of vendoring policy which does not consider that incident is necessarily incomplete.
That's very nearly a solved problem at this point; I get notified by bots (github, mainly) if a dependency has as a security vulnerability, and then it's very nearly a one click action to do a cargo update and commit the new lockfile.
The distro people could've been working on the tooling for automating this at the distro level (and some people in debian are doing work that would enable this); we don't need to go this insane "unbundle everything" route.
The distro people could've been working on the tooling for automating this at the distro level (and some people in debian are doing work that would enable this); we don't need to go this insane "unbundle everything" route.
Before broadband, and before forges like github with free CI and bots, which is all fairly recent, distros packaged everything was a godsend, not insane.
How long ago was that? And did they quantify the problem?
> everybody expects distros to fix security issues
IMO this is a huge mistake in the Linux distro world. It doesn't scale. It's similar to the weird idea that Debian should contain every program in the world and if the authors want their programs to be installable on Debian they must make a Debian package for it. And Fedora and Arch and Gentoo and...
Clearly bonkers.
IMO this is a huge mistake in the Linux distro world. It doesn't scale. It's similar to the weird idea that Debian should contain every program in the world and if the authors want their programs to be installable on Debian they must make a Debian package for it. And Fedora and Arch and Gentoo and...
Clearly bonkers.
Debian does not prevent you from installing stuff from elsewhere, it's just that their packages come with certain availability and stability guarantees. You may not care for them, but clearly bonkers it is clearly not.
boris(1)
> this whole experience reads like a lesson in what not to do.
What lessons have you drawn from it?
What lessons have you drawn from it?
Splitting a piece of software into multiple pieces and shipping the pieces (dependencies) independently is sometimes a good idea, but it has its limits. Maybe the limit should be for dependencies which are very stable and used by many packages (libc, etc.). The hard line policy enforced by Debian here obviously is not working. Happy to see other distros solve this better. This might become really problematic for Debian in the future.
See also on the same general topic: “The modern packager’s security nightmare” (2021) [1] by Gentoo maintainer Michał Górny, and Drew DeVault’s posts on why FOSS developers should not distribute software directly or try to impose their will on distros[2,3].
[1] https://blogs.gentoo.org/mgorny/2021/02/19/the-modern-packag..., discussed at https://news.ycombinator.com/item?id=26203853 (320 points, 278 comments)
[2] https://drewdevault.com/2019/12/09/Developers-shouldnt-distr..., no notable HN discussions
[3] https://drewdevault.com/2021/09/27/Let-distros-do-their-job...., no notable HN discussions
[1] https://blogs.gentoo.org/mgorny/2021/02/19/the-modern-packag..., discussed at https://news.ycombinator.com/item?id=26203853 (320 points, 278 comments)
[2] https://drewdevault.com/2019/12/09/Developers-shouldnt-distr..., no notable HN discussions
[3] https://drewdevault.com/2021/09/27/Let-distros-do-their-job...., no notable HN discussions
exactly. the sad part is that this is being coopted into an aegument for flatpack! it's insane the discussions on some distros.
they cannot see that by giving in to static libs you make every little application maintainer into a full-time distro maintainer responsible for it forever. ... good luck with that.
they cannot see that by giving in to static libs you make every little application maintainer into a full-time distro maintainer responsible for it forever. ... good luck with that.
I actually don’t entirely agree with Górny’s argument that we should use the technical tool of shared libraries to solve the social problem of not organizing the package archive in a way that allows an automatic bump of every dependency on a static library change. But I do agree with a weakened version where “static libraries” is replaced with “vendored code”, and that’s the (part of) problem TFA describes.
(The rest of the problem is that the goals of Debian are incompatible with upstream refusing to support versions of their own code more than a couple of months old. Which they transparently are, so there isn’t much to argue about here.)
(The rest of the problem is that the goals of Debian are incompatible with upstream refusing to support versions of their own code more than a couple of months old. Which they transparently are, so there isn’t much to argue about here.)
I'm also on (several) the fence on the solution. but the description of the problem is spot on
The list of dependencies doesn't really seem crazy. Logging, parsing command line arguments, band-aiding error traits, working with uuids, reading binary data, "memset_s". All of those should be a part of the standard library as in any other programming language. The fact that you simply cannot build anything but trivial examples without taking a dependency on a third party library is just plain ridiculous and hinders Rust's adoption (as seen in this case). It's especially ridiculous considering Rust positions itself to be used in mission critical systems and yet essentially requires you to implicitly trust hundreds of maintainers for any non trivial project to not be malicious.
https://github.com/koverstreet/bcachefs-tools/blob/b422b19f6...
https://github.com/koverstreet/bcachefs-tools/blob/b422b19f6...
Still, if you include transitive dependencies you end up with a total of 90 dependencies[0] which is unheard of in systems programming. This for some reason includes stuff like "winapi-i686-pc-windows-gnu" which really has no place for a set of linux software tools.
[0]:https://github.com/gentoo/gentoo/blob/master/sys-fs/bcachefs...
[0]:https://github.com/gentoo/gentoo/blob/master/sys-fs/bcachefs...
> with a total of 90 dependencies[0] which is unheard of in systems programming
If the comparison is with C/C++ then the comparison is flawed:
- some you would never see in C/C++ because there's no point to them (e.g. safe bindings to system libraries)
- some are the same dependency but split in multiple crates to improve compile times (e.g. `anstyle`, `clap` and `regex`)
- some in C/C++ would just be vendored due to the difficulty of package management (see for example [0])
> This for some reason includes stuff like "winapi-i686-pc-windows-gnu" which really has no place for a set of linux software tools.
I don't know how Gentoo's build system works, but Cargo resolves the versions for *all dependencies*, including those that are conditionally compiled (e.g. based on the target OS). This is because the `Cargo.lock` file, which contains the actual dependencies versions used, is supposed to be versioned in e.g. git, and thus needs to be cross platform and be the same for everyone compiling the crate.
[0]:https://wiki.alopex.li/LetsBeRealAboutDependencies#gotta-go-...
If the comparison is with C/C++ then the comparison is flawed:
- some you would never see in C/C++ because there's no point to them (e.g. safe bindings to system libraries)
- some are the same dependency but split in multiple crates to improve compile times (e.g. `anstyle`, `clap` and `regex`)
- some in C/C++ would just be vendored due to the difficulty of package management (see for example [0])
> This for some reason includes stuff like "winapi-i686-pc-windows-gnu" which really has no place for a set of linux software tools.
I don't know how Gentoo's build system works, but Cargo resolves the versions for *all dependencies*, including those that are conditionally compiled (e.g. based on the target OS). This is because the `Cargo.lock` file, which contains the actual dependencies versions used, is supposed to be versioned in e.g. git, and thus needs to be cross platform and be the same for everyone compiling the crate.
[0]:https://wiki.alopex.li/LetsBeRealAboutDependencies#gotta-go-...
The winapi dependency comes from the rust tooling (cargo specifically) putting a great value on portability, and practical concerns like people collaborating from different platforms on the same project. Many Rust projects check their Cargo.lock into git (recommended for binaries, and also many libraries do it). Now imagine if the Cargo.lock changes depending on the platform, which one would you track in git?
Now, there is the separate concern of cargo-vendor downloading the winapi crate, which it does, also on Linux. It doesn't have to because winapi is never built, yet it still does it. That one I agree is a bit wasteful, but it's a known issue.
Now, there is the separate concern of cargo-vendor downloading the winapi crate, which it does, also on Linux. It doesn't have to because winapi is never built, yet it still does it. That one I agree is a bit wasteful, but it's a known issue.
Where are those things in the standard library of C or C++, which most software is written in?
Most C Debian packages have a few dependencies in my experience, they are just “standard”, because they are old and stable.
Most C Debian packages have a few dependencies in my experience, they are just “standard”, because they are old and stable.
Parsing command line arguments, reading/manipulating binary data, memset_s are of course all available in the C standard library. I’m not familiar with rust, but if it’s true that dependencies are required for those types of basic operations... yikes, to say the least. If you include boost (which, admittedly, is a heavy dependency, but still a single one), everything mentioned is possible.
Just to be clear about it, Rust has the same level of this stuff in the standard library as C does. Like, the equivalent of argc/argv is in the rust standard library. The library being included is more like getopt conceptually; a package built on top of that that makes things nicer.
> If you include boost (which, admittedly, is a heavy dependency, but still a single one)
Then in principle, we could curate a set of useful Rust crates, treat the set as a single dependency, and be on the same page as C++ with Boost.
Then in principle, we could curate a set of useful Rust crates, treat the set as a single dependency, and be on the same page as C++ with Boost.
> positions itself to be used in mission critical systems and yet essentially requires you to implicitly trust hundreds of maintainers
It doesn't require anything. You can write those things yourself if you want. And for some critical things you should.
But given the tiny number of languages with logging, uuids, and error reporting built in... what are you taking about? They're doing just fine.
It doesn't require anything. You can write those things yourself if you want. And for some critical things you should.
But given the tiny number of languages with logging, uuids, and error reporting built in... what are you taking about? They're doing just fine.
Nonsense. If these were written in C, and depended on exact versions of getopt, log4c, libuuid, etc and refused to vendor them do you think that would be any easier to package?
This is a result of every Linux distros' self-centred requirement for the entire world to be packaged by them, and a refusal to accept vendoring.
Python has exactly the same problem. You can't even `pip install` on Debian any more because they insist on copying a random selection of Python packages into Apt and they can conflict.
This is a result of every Linux distros' self-centred requirement for the entire world to be packaged by them, and a refusal to accept vendoring.
Python has exactly the same problem. You can't even `pip install` on Debian any more because they insist on copying a random selection of Python packages into Apt and they can conflict.
> You can't even `pip install` on Debian any more because they insist on copying a random selection of Python packages into Apt and they can conflict.
And, they insist on not including venv by default, which IMO seems like a tiny little "fuck you" to users.
And, they insist on not including venv by default, which IMO seems like a tiny little "fuck you" to users.
Sure buddy. Not only Debian but the kernel itself too.
Those immature unproven little projects and their silly "best practices". What do Debian or the kernel know about building and maintaining bedrock software?
https://www.phoronix.com/news/Linus-Torvalds-Bcachefs-Regret...
Those immature unproven little projects and their silly "best practices". What do Debian or the kernel know about building and maintaining bedrock software?
https://www.phoronix.com/news/Linus-Torvalds-Bcachefs-Regret...
These are some of the sentiments I'm seeing where people are tired of Rust and the related crowd. Very young and/or very inexperienced Rust evangelists are showing up to experienced, long-term, and stable projects and effectively dictating that they know better because "the future" is on their side. And then when they are rebuffed, they turn to the mentally unhealthy echo chamber that is social media for reinforcement. It is a cycle of unhealthy and even toxic behavior. I'm fortunate that I haven't seen this with my mentees but it does appear that there are non-mainstream social media platforms allowing these problems to fester rather than addressing them head-on.
Apologies I got this kinda backwards. The Debian article was written from Debians perspective not bcachefs. "sure buddy" and the rest of the sarcasm was directed at bcachefs.
So I guess the gauntlet is now down for any other maintainer to come and decide they do want to package it for debian/ubuntu...
Not really.
Debian has a Rust policy, which among others prohibits this kind of extreme vendoring. Upstream insists on exact versioning (thus requiring vendoring), including hacked-up versions of some upstream crates, and also insists that Debian maintainers go on a war to change Debian's Rust policy to his liking (this is why I left #bcache; I don't set these policies, I'm not interested in fighting against them, and I'm not interested in an infinite rant about how I should spend my energy on such a fight). A couple of other people and myself made some attempts to get the Rust parts of bcachefs-tools buildable and working (and yes, it was really working, and no, the version changes led to no real breakage that I know of), but you can't reasonably package software from an upstream who is so hostile to everything about your distribution. If it's “my way or the highway” about everything, I guess it's the highway, then?
Maybe someone who cares deeply about bcachefs and cares deeply about Debian will pick it up, but that person would better have a really thick skin.
Debian has a Rust policy, which among others prohibits this kind of extreme vendoring. Upstream insists on exact versioning (thus requiring vendoring), including hacked-up versions of some upstream crates, and also insists that Debian maintainers go on a war to change Debian's Rust policy to his liking (this is why I left #bcache; I don't set these policies, I'm not interested in fighting against them, and I'm not interested in an infinite rant about how I should spend my energy on such a fight). A couple of other people and myself made some attempts to get the Rust parts of bcachefs-tools buildable and working (and yes, it was really working, and no, the version changes led to no real breakage that I know of), but you can't reasonably package software from an upstream who is so hostile to everything about your distribution. If it's “my way or the highway” about everything, I guess it's the highway, then?
Maybe someone who cares deeply about bcachefs and cares deeply about Debian will pick it up, but that person would better have a really thick skin.
oguz-ismail(1)
Debian is going to look as ridiculous for doing this as Alma Linux is for insisting btrfs isn’t an “enterprise file system” due to it lacking RAID 5/6.
AlmaLinux has made no such statement, for what it's worth.
Red Hat Enterprise Linux excludes btrfs support in its distribution because Red Hat does not believe btrfs to be stable enough to be worth considering. That decision trickles down to recompilation projects that pretty much amount to "RHEL, but without having to pay for it".
And rightfully so, imo. btrfs is the only filesystem I've used where it's gotten so corrupted due to faulty memory on a computer that I had to recover the files and reinstall from scratch after replacing the memory (obviously not using btrfs the second time ..).
Thanks for that correction. I misunderstood a forum post that was probably someone being snarky/cynical.
btrfs lacking RAID 5/6 is exactly the sort of thing that makes it get written off as a toy. ZFS has had working raidz[12] since 2005, raidz3 since 2007, and draid since 2021. Completely stable and enterprise-ready.
It feels like whenever the author of bcachefs comes up, it's always because of some drama.
Just the other day he clashed with Linus Torvalds: https://lore.kernel.org/lkml/CAHk-=wj1Oo9-g-yuwWuHQZU8v=VAsB...
My reading is that he's very passionate, so he wants to "move fast and break things" and doesn't get why the others aren't necessarily very happy about it.