Traditional packaging is not suitable for modern applications?(theevilskeleton.gitlab.io)
theevilskeleton.gitlab.io
Traditional packaging is not suitable for modern applications?
https://theevilskeleton.gitlab.io/2022/08/29/traditional-packaging-is-not-suitable-for-modern-applications.html
133 comments
> If anything, this argument dismisses most work done by distros
> It seems that the author has a solution - containers - and tries to find a problem for said solution.
I don't really follow this. Distro maintainers are working in a problem and some people are happy with the results.
But the fact Nix, Guix flatpack and others exist shows other folks are not and are coming up with their solutions.
It seems you're painting the author as some container weirdo just for not liking your favorite distribution method.
> It seems that the author has a solution - containers - and tries to find a problem for said solution.
I don't really follow this. Distro maintainers are working in a problem and some people are happy with the results.
But the fact Nix, Guix flatpack and others exist shows other folks are not and are coming up with their solutions.
It seems you're painting the author as some container weirdo just for not liking your favorite distribution method.
Distro maintainers are working on a problem (conflict resolution in dependency trees) and solving it. Use of containers is sidestepping of the problem.
Containers solve entirely different problem - how do you make conflicting dependency trees coexist in isolation. Layered containers solve this particular problem quite well. But it's a different problem than distros try to solve.
Containers solve entirely different problem - how do you make conflicting dependency trees coexist in isolation. Layered containers solve this particular problem quite well. But it's a different problem than distros try to solve.
There is no "conflict" other than that which is created by most distros; there is no inherent problem to having different applications use different versions of libraries. Containers/nix/whatever do not really sidestep this, so much as not having this problem in the first place.
>Sure, containerizing an application is /easier/ than working out the dependency trees and solving dependency conflicts between them. It is by no means fundamentally better approach. [...] The [distro] packaging system is transparent to contents: nothing forbids you to package versions 1.0 and 2.0 and have them happily coexist as long as they do not share common paths.
I think your comment is analyzing the problem from a different abstraction level than the author's examples.
In other words, your alternative approach of "solving dependency conflicts" is abstract advice unless you try to make it concrete by focusing who solves those dependency conflicts.
Using the author's example of a VSCode conflict (collision of "/usr/bin/code") with Actor in italics:
- Microsoft owns commercial the build of /packages/visual-studio-code-bin
- Filipe Laíns & Massimiliano Torromeo owns the open-source-license build of /packages/community/x86_64/code/
- Levente Polyak, et al owns the Arch Linux distribution
- end user owns the computer that needs to run conflicting software
Generalize the above matrix to every software package and consider which of those 4 actors can practically solve the conflicts in a scalable way?
Filipe Laíns & Massimiliano Torromeo can't control what Microsoft does and vice-versa. (Ironically, the freedom to not be controlled by the other actor is one reason there's a package conflict!)
And Levente Polyak could "solve" it by blessing Filipe Laíns's package and rejecting Microsoft's but this creates a new "problem" for users who specifically want Microsoft's commercial build.
The fundamental issue is that you have independent actors that can't control what others do. Result is that sometimes software packages clash (namespace collisions, dependent libraries not in sync, incompatible licenses, etc).
The approach of using containers -- however flawed it is -- gets past the stalemate of independent actors in the software landscape being incompatible with each other.
I think your comment is analyzing the problem from a different abstraction level than the author's examples.
In other words, your alternative approach of "solving dependency conflicts" is abstract advice unless you try to make it concrete by focusing who solves those dependency conflicts.
Using the author's example of a VSCode conflict (collision of "/usr/bin/code") with Actor in italics:
- Microsoft owns commercial the build of /packages/visual-studio-code-bin
- Filipe Laíns & Massimiliano Torromeo owns the open-source-license build of /packages/community/x86_64/code/
- Levente Polyak, et al owns the Arch Linux distribution
- end user owns the computer that needs to run conflicting software
Generalize the above matrix to every software package and consider which of those 4 actors can practically solve the conflicts in a scalable way?
Filipe Laíns & Massimiliano Torromeo can't control what Microsoft does and vice-versa. (Ironically, the freedom to not be controlled by the other actor is one reason there's a package conflict!)
And Levente Polyak could "solve" it by blessing Filipe Laíns's package and rejecting Microsoft's but this creates a new "problem" for users who specifically want Microsoft's commercial build.
The fundamental issue is that you have independent actors that can't control what others do. Result is that sometimes software packages clash (namespace collisions, dependent libraries not in sync, incompatible licenses, etc).
The approach of using containers -- however flawed it is -- gets past the stalemate of independent actors in the software landscape being incompatible with each other.
This is why we need packaging standards, especially around metadata, instead of further decamping into language-specific ecosystems. Because even with containerized apps, you still want a few launch scripts and a compiled program (for instance) that might contain multiple dependencies from different languages (C, Go, and Fortran, maybe).
> The fundamental issue is that you have independent actors that can't control what others do. Result is that sometimes software packages clash (namespace collisions, dependent libraries not in sync, incompatible licenses, etc).
Yes, I agree. Where we are in disagreement is what the role of a distro is here. Distros are there to solve that stalemate in one way or another. In Code's case distro maintainers chose to mark these packages as conflicting. They could have changed namespace of one (or both) package[s].
As I have mentioned in reply to a sibling comment this is exactly the work distros are doing - solving the stalemate between upstreams.
> The approach of using containers -- however flawed it is -- gets past the stalemate of independent actors in the software landscape being incompatible with each other.
Sure, containers do try to get past the stalemate, but they try to solve a different problem - how to make conflicting dependency trees coexist. Containers can also fail to solve the stalemate. Sure, containers can isolate `/usr/bin/code` between containers, but contention on `/etc/code`, `$HOME/.code` or other static user data still has to be solved one way or another.
Yes, I agree. Where we are in disagreement is what the role of a distro is here. Distros are there to solve that stalemate in one way or another. In Code's case distro maintainers chose to mark these packages as conflicting. They could have changed namespace of one (or both) package[s].
As I have mentioned in reply to a sibling comment this is exactly the work distros are doing - solving the stalemate between upstreams.
> The approach of using containers -- however flawed it is -- gets past the stalemate of independent actors in the software landscape being incompatible with each other.
Sure, containers do try to get past the stalemate, but they try to solve a different problem - how to make conflicting dependency trees coexist. Containers can also fail to solve the stalemate. Sure, containers can isolate `/usr/bin/code` between containers, but contention on `/etc/code`, `$HOME/.code` or other static user data still has to be solved one way or another.
> The approach of using containers -- however flawed it is -- gets past the stalemate of independent actors in the software landscape being incompatible with each other
What does that is bundling your dependencies, not containers. You could always bundle everything an app needs into a single tarball or a package. Containers didn't make it possible, they made it easy and popular.
What does that is bundling your dependencies, not containers. You could always bundle everything an app needs into a single tarball or a package. Containers didn't make it possible, they made it easy and popular.
There are design choices that can be made to guarantee namespace isolation - having different software packages installed within the same /usr/bin/ directory is a guarantee for namespace collisions.
I generally think these arguments (containers, flatpak) boil down to: I don't want to figure out how a distro works. I'm unsympathetic to that; it's actually pretty easy to figure that out: they almost all use systemd and making a deb/rpm is also straightforward (there are great guides). Distros are also great at updates, to the degree devs can largely outsource a lot of it to them (compared with Windows where you're writing update functionality into your app, or whatever).
I'm not even sympathetic to weird cases like OBS needing a forked version of ffmpeg. 1 this sounds like a wacky thing to do, 2 it sounds like they created a new project that is a different dependency. Call it something else (I suggest obsmpeg), package it for distros, pull in updates from ffmpeg whenever you want, reap the benefits of FOSS. Inventing an entirely new software delivery system instead seems like an overreaction.
I am sympathetic to working this out across multiple distros though. Finding the intersection of packages across multiple versions of Ubuntu, Debian and Fedora isn't super easy, and it can really hamstring you when building an app (you want to use libcool-1.8.x, Ubuntu LTS has libcool-1.2.25, :sad:). The ecosystem isn't set up for this and it's a pain to deal with in your app (you more or less #ifdef functionality out, which product managers will freak out about).
I'm not even sympathetic to weird cases like OBS needing a forked version of ffmpeg. 1 this sounds like a wacky thing to do, 2 it sounds like they created a new project that is a different dependency. Call it something else (I suggest obsmpeg), package it for distros, pull in updates from ffmpeg whenever you want, reap the benefits of FOSS. Inventing an entirely new software delivery system instead seems like an overreaction.
I am sympathetic to working this out across multiple distros though. Finding the intersection of packages across multiple versions of Ubuntu, Debian and Fedora isn't super easy, and it can really hamstring you when building an app (you want to use libcool-1.8.x, Ubuntu LTS has libcool-1.2.25, :sad:). The ecosystem isn't set up for this and it's a pain to deal with in your app (you more or less #ifdef functionality out, which product managers will freak out about).
As a regular Linux user who doesn't publish apps but has been reading this argument for years I've never understood the value distros provide.
They're clearly doing a huge amount of work editing everything, but it also doesn't seem like they have the resources to properly test their edits. And while their edits are necessary to make everything work with their system, from a high level perspective they don't seem to be making the software better at what I want it to do. Given that, their model looks like dangerous busywork that at best delays updates and at worst introduces new bugs.
Distros use the word "maintain" to describe what they're doing, but they don't seem to be doing that. To maintain a piece of software I'd expect the maintainer to need a deep understanding of the source code and to continually monitor changes. The complaints maintainers post that software changed seem to prove they aren't doing that.
When I first saw a maintainer complain a library only posted about a breaking build change in their release notes and mailing list I was shocked. I had been assuming they were reading diffs, when they weren't even reading the changelog.
They're clearly doing a huge amount of work editing everything, but it also doesn't seem like they have the resources to properly test their edits. And while their edits are necessary to make everything work with their system, from a high level perspective they don't seem to be making the software better at what I want it to do. Given that, their model looks like dangerous busywork that at best delays updates and at worst introduces new bugs.
Distros use the word "maintain" to describe what they're doing, but they don't seem to be doing that. To maintain a piece of software I'd expect the maintainer to need a deep understanding of the source code and to continually monitor changes. The complaints maintainers post that software changed seem to prove they aren't doing that.
When I first saw a maintainer complain a library only posted about a breaking build change in their release notes and mailing list I was shocked. I had been assuming they were reading diffs, when they weren't even reading the changelog.
Generally, distros supply security updates. For example, Debian picks a version for a release and makes sure it has security updates for the entire release. Sometimes this requires backporting patches because the version is no longer supported upstream. This means an app can be shipped that will work for the life of the release without changes, while staying secure.
In theory, yes. In practice I'm very skeptical that maintainers can correctly backport patches without having a solid understanding of the source code. And I'm skeptical that maintainers can have a solid understanding of the source code without reading a substantial portion of it, and I know they haven't got the time to do that.
I’m sure what you’re saying happens. There’s 1000s of packages with maintainers of varying skill.
That said the track record speaks for itself. I can only remember one time a maintainer introduced a vuln in Debian. The system works even though you’ll find cracks if you look.
That said the track record speaks for itself. I can only remember one time a maintainer introduced a vuln in Debian. The system works even though you’ll find cracks if you look.
Yeah, quality varies. But the vast majority of this is volunteer work. If I were developer of some app and I wanted it to work on Fedora, I'd probably be the maintainer of its package. But if I didn't want to do that, I'd at least be in contact with the person who was. "Hey heads up, we broke compat. Let me know if we need to huddle or something." Also seems a lot easier than inventing Flatpak and getting distros and users to use/like it.
So much of this stuff is like a default unwillingness to work with humans over the internet, but that's fundamentally how FOSS works. A couple years ago there was a kerfluffle between the scientific community and SciPy--the community wanted SciPy to maintain compatibility literally forever and also continue adding new features. SciPy was like, "that's impossible, and even if we wanted to it would take the resources of Microsoft to do". But they never talked about it! Instead there were open letters and forum threads.
The quickest route to fixing social problems like this--and they are social--is to deal with people. I'm pretty misanthropic, but even I have to admit this is true.
> Distros use the word "maintain" to describe what they're doing, but they don't seem to be doing that. To maintain a piece of software I'd expect the maintainer to need a deep understanding of the source code and to continually monitor changes. The complaints maintainers post that software changed seem to prove they aren't doing that.
I just want to point out that this is asking a lot. You're essentially asking Debian to find volunteers to become as knowledgeable about, say, Libreoffice, as the developers themselves. That's probably not gonna happen. I think a more reasonable definition of "maintainer" is "ensures the package continues to work on ${DISTRO}, reaches out to the developers if a new update breaks that". I'm not defending the case you brought up, that sounds pretty lazy to me, but in the vast majority of cases it works great.
So much of this stuff is like a default unwillingness to work with humans over the internet, but that's fundamentally how FOSS works. A couple years ago there was a kerfluffle between the scientific community and SciPy--the community wanted SciPy to maintain compatibility literally forever and also continue adding new features. SciPy was like, "that's impossible, and even if we wanted to it would take the resources of Microsoft to do". But they never talked about it! Instead there were open letters and forum threads.
The quickest route to fixing social problems like this--and they are social--is to deal with people. I'm pretty misanthropic, but even I have to admit this is true.
> Distros use the word "maintain" to describe what they're doing, but they don't seem to be doing that. To maintain a piece of software I'd expect the maintainer to need a deep understanding of the source code and to continually monitor changes. The complaints maintainers post that software changed seem to prove they aren't doing that.
I just want to point out that this is asking a lot. You're essentially asking Debian to find volunteers to become as knowledgeable about, say, Libreoffice, as the developers themselves. That's probably not gonna happen. I think a more reasonable definition of "maintainer" is "ensures the package continues to work on ${DISTRO}, reaches out to the developers if a new update breaks that". I'm not defending the case you brought up, that sounds pretty lazy to me, but in the vast majority of cases it works great.
I feel the article is missing the point of distros. They are supposed to be fully self-contained integrated operating systems, not just container platforms.
Additionally, essentially vendoring full dependency trees for each application separately would massively increase the maintenance burden because instead of just maintaining one version of a library the distro will need to maintain n versions
> Modern applications are inherently fragile.
There is nothing inherent in the fragility, it is self-inflicted problem of some applications.
Additionally, essentially vendoring full dependency trees for each application separately would massively increase the maintenance burden because instead of just maintaining one version of a library the distro will need to maintain n versions
> Modern applications are inherently fragile.
There is nothing inherent in the fragility, it is self-inflicted problem of some applications.
This is written from the point of view that implicitly assumes only a single application installed (= a container), where distro is reduced to API provider and some common utils the app author doesn't care about. It would be correct only if I was building an appliance. But as the user I have a general purpose computer which gets the work done over multiple apps.
I think there lies deeper, cultural division between developers, who think the world (at least the computer) revolves around their app and their downstream, who desperately try to avoid more and more code shipped unto them for the upstream's convenience. It's not really a difference for upstream if they bundle a dependency or not (or 10k+, in the case of npm), but it quickly becomes a big support burden if you have multiple such "apps" installed.
I think there lies deeper, cultural division between developers, who think the world (at least the computer) revolves around their app and their downstream, who desperately try to avoid more and more code shipped unto them for the upstream's convenience. It's not really a difference for upstream if they bundle a dependency or not (or 10k+, in the case of npm), but it quickly becomes a big support burden if you have multiple such "apps" installed.
I think you are going to have 20 gcc available at same time if you do want this model. Windows literally showed you what would happen if everyone just require their own dependency version.
It is choice between a system with bleeding edge / ancient(or sometimes insecure) program mixed together or a system with well audited recent enough programs.
Windows/Snaps go for the first, and apt/yum..etc chose the later. It's just all about it.
It is choice between a system with bleeding edge / ancient(or sometimes insecure) program mixed together or a system with well audited recent enough programs.
Windows/Snaps go for the first, and apt/yum..etc chose the later. It's just all about it.
It's not really a difference for upstream if they bundle a dependency or not (or 10k+, in the case of npm), but it quickly becomes a big support burden if you have multiple such "apps" installed.
But only because distributions want to take control of everything. If a distribution just provided the necessary infrastructure to empower developers to maintain their own dependencies (like eg. Flatpak does), we wouldn’t have this issue.
But only because distributions want to take control of everything. If a distribution just provided the necessary infrastructure to empower developers to maintain their own dependencies (like eg. Flatpak does), we wouldn’t have this issue.
> distributions want to take control of everything.
I'm not sure where that's coming from. Distros/packagers (should) rightly feel a responsibility for a packaged application's quality, security, interoperability.
> If a distribution just provided the necessary infrastructure to empower developers to maintain their own dependencies
To stick with the Arch Linux example in the article, is the flatpak you get from `# pacman -S flatpak` not suitable for some reason?
I'm not sure where that's coming from. Distros/packagers (should) rightly feel a responsibility for a packaged application's quality, security, interoperability.
> If a distribution just provided the necessary infrastructure to empower developers to maintain their own dependencies
To stick with the Arch Linux example in the article, is the flatpak you get from `# pacman -S flatpak` not suitable for some reason?
> I'm not sure where that's coming from. Distros/packagers (should) rightly feel a responsibility for a packaged application's quality, security, interoperability.
This seems like false advertising. Ensuring an application is secure would require reading it's source. Based on maintainer complaints (like when that python package added a rust dependency) they don't seem to have the resources to even read all the changelogs.
This seems like false advertising. Ensuring an application is secure would require reading it's source. Based on maintainer complaints (like when that python package added a rust dependency) they don't seem to have the resources to even read all the changelogs.
It's not really fair to extrapolate from this one instance you keep bringing up to all package maintenance everywhere. I think a more balanced assessment of the current situation would be that by and large the system works, there are 1000s of packages that are up to date, well-maintained and secure to the best of our knowledge, and they function in concert with each other. That's a big job!
General purpose computing (and particularly Free Software) is about empowering users, not developers. This quickly becomes important when interests misalign, for example when developers want to push incompatible update to users and stop supporting previous version, or the upstream simply vanishes and stop providing security updates. Why is it OK for developers to "manage dependencies" (pin older versions of libraries) and somehow not for the end user agents (distros) to "manage" their dependencies (i.e. apps)?
Moreover, why shouldn't we empower libraries' developers to push their versions as they want? They also have limited manpower and support capabilities.
Moreover, why shouldn't we empower libraries' developers to push their versions as they want? They also have limited manpower and support capabilities.
General purpose computing (and particularly Free Software) is about empowering users, not developers.
Empowering users is also making it easy to install what they want or need to install, not just what a distribution decides to offer. To give an example, many people need Slack for work, but distributions don't offer it, because it is non-free.
This quickly becomes important when interests misalign, for example when developers want to push incompatible update to users and stop supporting previous version,
Why should a distributor decide what my interests are?
Also, this argument is kind of strange, if a developer is pushing an incompatible update or stops supporting a previous version, will the distribution continue to maintain the application? Not really. Usually, it just means that the application is frozen in time and you can only hope that security updates are applied. If the application happens to be (e.g.) in Ubuntu Universe, you are usually out of luck and you just get to keep a bunch of applications with known vulnerabilities.
Empowering users is also making it easy to install what they want or need to install, not just what a distribution decides to offer. To give an example, many people need Slack for work, but distributions don't offer it, because it is non-free.
This quickly becomes important when interests misalign, for example when developers want to push incompatible update to users and stop supporting previous version,
Why should a distributor decide what my interests are?
Also, this argument is kind of strange, if a developer is pushing an incompatible update or stops supporting a previous version, will the distribution continue to maintain the application? Not really. Usually, it just means that the application is frozen in time and you can only hope that security updates are applied. If the application happens to be (e.g.) in Ubuntu Universe, you are usually out of luck and you just get to keep a bunch of applications with known vulnerabilities.
Looking at the Slack .deb I see that there isn't even any info on what the license is for Slack, so distros couldn't distribute it even in their non-free archives.
https://downloads.slack-edge.com/releases/linux/4.28.171/pro...
Not only that, but it contains ffmpeg and other LGPL stuff without distributing source code, which seems like an LGPL license violation to me.
https://downloads.slack-edge.com/releases/linux/4.28.171/pro...
Not only that, but it contains ffmpeg and other LGPL stuff without distributing source code, which seems like an LGPL license violation to me.
> But only because distributions want to take control of everything.
This misses entirety of what distributions are. Distribution is a collection of software packages that in tandem create uniform coherent unit. They only want to control what they distribute.
As I have mentioned in another comment, one of the core problems is that we do not understand our dependency trees. If you use version "latest" (or even do not audit your dependency trees not to contain such meta-versions) you consciously relinquish control of dependencies. At this point there is no real difference in power and control between trusting apt and npm not to break anything. Trusting npm (or pip, or whatever) is actually even worse, because they explicitly are just registries without any supervision or soft guarantees.
This misses entirety of what distributions are. Distribution is a collection of software packages that in tandem create uniform coherent unit. They only want to control what they distribute.
As I have mentioned in another comment, one of the core problems is that we do not understand our dependency trees. If you use version "latest" (or even do not audit your dependency trees not to contain such meta-versions) you consciously relinquish control of dependencies. At this point there is no real difference in power and control between trusting apt and npm not to break anything. Trusting npm (or pip, or whatever) is actually even worse, because they explicitly are just registries without any supervision or soft guarantees.
> There is nothing inherent in the fragility, it is self-inflicted problem of some applications.
The architecture is fragile due to too many layers. Anything is "fragile" in the sense that you can't really design a really complex, generic system with all possible state pre-computed and handled up front, the point is how to handle the fragility.
Classic systems was "the OS is a single application where anything is ready available source code", this was utterly fragile in potential faults BUT far easier to correct any error so in the end far more robust. Think about some CL systems: you'll easily get far more issues in normal usage, but essentially any issue is easy to handle.
The real underneath issue is business: software can't be a commercial product, must be a common open knowledge, like a kitchen recipe, where any party involved can see and act and anything is designed ONLY for the human. We need desktops in a network not dummy terminals of a central service designed to serve their owner.
The architecture is fragile due to too many layers. Anything is "fragile" in the sense that you can't really design a really complex, generic system with all possible state pre-computed and handled up front, the point is how to handle the fragility.
Classic systems was "the OS is a single application where anything is ready available source code", this was utterly fragile in potential faults BUT far easier to correct any error so in the end far more robust. Think about some CL systems: you'll easily get far more issues in normal usage, but essentially any issue is easy to handle.
The real underneath issue is business: software can't be a commercial product, must be a common open knowledge, like a kitchen recipe, where any party involved can see and act and anything is designed ONLY for the human. We need desktops in a network not dummy terminals of a central service designed to serve their owner.
> software can't be a commercial product, must be a common open knowledge, like a kitchen recipe
Kitchen recipes rarely need features added or even bugs fixed. But when this happens, it's usually done by a cook on the spot.
This is only possible due to utter simplicity of a typical recipe, compared to any substantial piece of software. Also, small variations in cooking usually result in pretty edible dishes, while small variations in a program can have drastic results.
So no, sadly, software already cannot be like a cookbook.
Kitchen recipes rarely need features added or even bugs fixed. But when this happens, it's usually done by a cook on the spot.
This is only possible due to utter simplicity of a typical recipe, compared to any substantial piece of software. Also, small variations in cooking usually result in pretty edible dishes, while small variations in a program can have drastic results.
So no, sadly, software already cannot be like a cookbook.
Try to compare commercial support, 99% of the time crappy and utterly slow vs community support and you'll get the answer. Even hyper-expensive commercial software have terrible support. Most FLOSS devs acts far better. Surely without formal legal guarantees.
While I can agree in many cases (YMMV), the point is that there still need to be expert cooks, and pretty few of them cook a particular piece of software right. Very unlike NYT plum pie recipe which exists as common knowledge and which even I can cook adequately.
Sure, but this is another problem: we need a common culture in IT, with specialists of that field that are missed because most schools of any degree do not form them being tied to some specific corporate and political interests.
Let's say we have a widespread classic desktop system, we start learning it when we start using a desktop computer in the childhood, when we arrive at the high school most already know it a bit, not differently by those who start driving cars: they do not really start from sketch. After university some will know some aspects, some others some other aspects but in general the current common social brain is known enough by most to be mastered enough by most.
At that point a CAD user do know how to fix certain CAD issues and who to ask if he/she can't because at that point IT is a common knowledge enough.
Instead if we keep up the refrain "just learn to click around and there you go, no need to study more" we end up in present impasse where in most professions, not just IT, there is a so deep lack of competence that anything fall apart because even those who know a bit can't act lacking tools to act.
We can't design of course a really perfect system, there always be issues, but we can and should try to tend to. Aiming high to still collect something good.
Let's say we have a widespread classic desktop system, we start learning it when we start using a desktop computer in the childhood, when we arrive at the high school most already know it a bit, not differently by those who start driving cars: they do not really start from sketch. After university some will know some aspects, some others some other aspects but in general the current common social brain is known enough by most to be mastered enough by most.
At that point a CAD user do know how to fix certain CAD issues and who to ask if he/she can't because at that point IT is a common knowledge enough.
Instead if we keep up the refrain "just learn to click around and there you go, no need to study more" we end up in present impasse where in most professions, not just IT, there is a so deep lack of competence that anything fall apart because even those who know a bit can't act lacking tools to act.
We can't design of course a really perfect system, there always be issues, but we can and should try to tend to. Aiming high to still collect something good.
[deleted]
I'm not sure distro needs to "maintain" N versions. The distro maintain 1, but all N remain available and able to fulfill the dependencies of the underlying application
It's not the distro's job to ensure the application is secure and up to date, its up to the application packager (i.e. docker container maintainer) to do that. i.e. to rev new releases of the application image even if nothing has changed in the core app, just to take care of security updates in distro provided packages). In pr
It's not the distro's job to ensure the application is secure and up to date, its up to the application packager (i.e. docker container maintainer) to do that. i.e. to rev new releases of the application image even if nothing has changed in the core app, just to take care of security updates in distro provided packages). In pr
> It's not the distro's job to ensure the application is secure and up to date
I would prefer a distro that ensures applications in its repository are up to date.
Consider a log4j-type issue. In one case, the distro quickly fixes the vulnerability in a library package, the user installs the update and that is that.
In the other case, the user has to track down which installed applications use the affected library, update them if there is an update available, and figure out what to do if there is no update available.
I would prefer a distro that ensures applications in its repository are up to date.
Consider a log4j-type issue. In one case, the distro quickly fixes the vulnerability in a library package, the user installs the update and that is that.
In the other case, the user has to track down which installed applications use the affected library, update them if there is an update available, and figure out what to do if there is no update available.
> Consider a log4j-type issue. … the user installs the update and that is that.
I hear what you’re saying. But - I’ve been writing a lot of rust lately. For better and worse, most rust programs pull in dozens of crates as direct or transitive dependencies. Apt isn’t up to the task of replacing cargo:
- Apt’s dynamic library mechanism is designed around C’s dynamic library support. This doesn’t work with many other languages. It certainly doesn’t work with rust, which doesn’t have an ABI.
- There are too many rust packages, and they update too frequently for apt to keep up. Apt could be a bad, out of date, partial clone of cargo. But that sounds worse than cargo in every way.
- Shipping deps with cargo (or whatever) is preferred by app developers because cargo is better integrated with the language. And it works reliably on every platform. A dpkg wouldn’t even work on every version of Debian.
We have a similar problem with ruby gems, nodejs packages (in npm), etc etc. I could imagine apt wrapping cargo, npm, and friends and providing equivalent functionality. But the authors of apt seem to think it’s already good enough. It’s not. Cargo, npm, and friends run rings around apt from the point of view of developer mindshare, ease of use, platform independent packaging, and so on.
Apt and friends don’t have the features we want, and all the browbeating in the world won’t change that.
I hear what you’re saying. But - I’ve been writing a lot of rust lately. For better and worse, most rust programs pull in dozens of crates as direct or transitive dependencies. Apt isn’t up to the task of replacing cargo:
- Apt’s dynamic library mechanism is designed around C’s dynamic library support. This doesn’t work with many other languages. It certainly doesn’t work with rust, which doesn’t have an ABI.
- There are too many rust packages, and they update too frequently for apt to keep up. Apt could be a bad, out of date, partial clone of cargo. But that sounds worse than cargo in every way.
- Shipping deps with cargo (or whatever) is preferred by app developers because cargo is better integrated with the language. And it works reliably on every platform. A dpkg wouldn’t even work on every version of Debian.
We have a similar problem with ruby gems, nodejs packages (in npm), etc etc. I could imagine apt wrapping cargo, npm, and friends and providing equivalent functionality. But the authors of apt seem to think it’s already good enough. It’s not. Cargo, npm, and friends run rings around apt from the point of view of developer mindshare, ease of use, platform independent packaging, and so on.
Apt and friends don’t have the features we want, and all the browbeating in the world won’t change that.
Rust has dynamic linking and a symbol name mangling ABI now and it is potentially useful enough that distros could use it.
https://wiki.debian.org/StaticLinking#Rust https://github.com/rust-lang/rfcs/pull/2603 https://github.com/rust-lang/rust/issues/60705
https://wiki.debian.org/StaticLinking#Rust https://github.com/rust-lang/rfcs/pull/2603 https://github.com/rust-lang/rust/issues/60705
The symbol mangling ABI isn't considered stable yet. And even if it was stable, rust's struct packing is still compiler-defined.
Rust libraries can be dynamically linked if they look like a C library, or get wrapped by abi_stable[1] or something. But this isn't the standard. Probably less than 5% of the crate ecosystem is packaged like this.
And a 5% solution is nowhere near good enough. Right now, I can build any rust package on any system, no matter what dependencies it has by just running cargo build. For apt to be competitive, it'd need to pull the ~100k cargo packages into apt's repository, wrap all of them in abi_stable or some equivalent and parse Cargo.toml files into the equivalent dpkg config.
Ideally it should also work across every OS (though starting with ubuntu, debian and mint would be a start).
This is possible. But for all the hand wringing about how modern packages should use apt, nobody seems to actually care enough to make any of this happen.
[1] https://crates.io/crates/abi_stable
Rust libraries can be dynamically linked if they look like a C library, or get wrapped by abi_stable[1] or something. But this isn't the standard. Probably less than 5% of the crate ecosystem is packaged like this.
And a 5% solution is nowhere near good enough. Right now, I can build any rust package on any system, no matter what dependencies it has by just running cargo build. For apt to be competitive, it'd need to pull the ~100k cargo packages into apt's repository, wrap all of them in abi_stable or some equivalent and parse Cargo.toml files into the equivalent dpkg config.
Ideally it should also work across every OS (though starting with ubuntu, debian and mint would be a start).
This is possible. But for all the hand wringing about how modern packages should use apt, nobody seems to actually care enough to make any of this happen.
[1] https://crates.io/crates/abi_stable
Well, there are significant differences between apt and cargo though right? Apt by and large installs pre-built binaries and ancillary stuff for projects written in... every language. Cargo fetches and builds Rust projects. If you wanted Cargo to do what Apt does, what it currently does would be a small percentage of its new total functionality.
It might be cool if we "united" all the build apps (npm, pip, cargo, go, etc) under a similar interface, but that also sounds like a lot of work, which is why I think these package managers stick with binaries.
It might be cool if we "united" all the build apps (npm, pip, cargo, go, etc) under a similar interface, but that also sounds like a lot of work, which is why I think these package managers stick with binaries.
I got the impression from that issue that the v0 ABI is frozen and is only not the default yet because they are working on getting support for it in external tools and writing up the documentation.
https://github.com/rust-lang/rust/issues/60705 https://github.com/rust-lang/rfcs/pull/2603
https://github.com/rust-lang/rust/issues/60705 https://github.com/rust-lang/rfcs/pull/2603
> Apt’s dynamic library mechanism is designed around C’s dynamic library support. This doesn’t work with many other languages. It certainly doesn’t work with rust, which doesn’t have an ABI.
Nothing's stopping them from shipping them as static libraries - with some simple tooling it's easy enough to figure out what needs rebuilds and automate it.
> There are too many rust packages, and they update too frequently for apt to keep up.
And what do you think cargo does to keep up? It uses a metadata system and a common build system across packages to make automatic discovery of dependency trees seamless. There's nothing stopping a debian maintainer from writing automatic tooling around this, in fact, it already exists[1].
> A dpkg wouldn’t even work on every version of Debian.
And neither does cargo as soon as there's a crate that requires a too new version of openssl or whatever other C library. The reality of the matter is that cargo can't express the full dependency tree including any cross-language or system-level dependencies.
> We have a similar problem with ruby gems, nodejs packages (in npm), etc etc.
I can name a few problems that make nodejs slightly different, and it's the fact that npm doesn't deal with source packages the way cargo does, it deals with binary packages that can include whatever flavor of bundled shared libraries du jour, probably only built for x86_64 and if you're lucky arm64, and with unclear source and/or build instructions or OS compatibility. Its build system also has support for defining custom build procedures and calling random binaries during build like the npm package for 7zip, which is also prebuilt. This, and the tendency to make microdependencies just doesn't mesh well with distributions that need to make sure everything has corresponding source and can be built reproducibly.
Ruby gems have been solved since they've been around for longer, as have python packages, even though the latter keeps introducing wrenches to make packaging harder.
> Cargo, npm, and friends run rings around apt from the point of view of developer mindshare, ease of use, platform independent packaging, and so on.
Maybe, but they introduce their fair share of problems as well. I've already mentioned the piss poor support for cross-language dependencies, but let's mention that the "stores" the packages for these are published to are only moderated after the fact - it's really easy to introduce malware into these stores without anyone noticing, and it's happened a bunch of times. Distributions make sure the binaries correspond to the sources[2], and the maintainers are the first to spot any obvious malware or breakage in software as a result of a change. Sometimes it goes under their noses as well[3], but these cases are much more rare than npm breakage. In general, the distribution model allows for better trust and security, and I think this is something that should be valued more than it currently is.
[1]: https://salsa.debian.org/rust-team/debcargo/
[2]: https://reproducible-builds.org/
[3]: https://www.theregister.com/2015/06/17/debian_chromium_hubbu...
Nothing's stopping them from shipping them as static libraries - with some simple tooling it's easy enough to figure out what needs rebuilds and automate it.
> There are too many rust packages, and they update too frequently for apt to keep up.
And what do you think cargo does to keep up? It uses a metadata system and a common build system across packages to make automatic discovery of dependency trees seamless. There's nothing stopping a debian maintainer from writing automatic tooling around this, in fact, it already exists[1].
> A dpkg wouldn’t even work on every version of Debian.
And neither does cargo as soon as there's a crate that requires a too new version of openssl or whatever other C library. The reality of the matter is that cargo can't express the full dependency tree including any cross-language or system-level dependencies.
> We have a similar problem with ruby gems, nodejs packages (in npm), etc etc.
I can name a few problems that make nodejs slightly different, and it's the fact that npm doesn't deal with source packages the way cargo does, it deals with binary packages that can include whatever flavor of bundled shared libraries du jour, probably only built for x86_64 and if you're lucky arm64, and with unclear source and/or build instructions or OS compatibility. Its build system also has support for defining custom build procedures and calling random binaries during build like the npm package for 7zip, which is also prebuilt. This, and the tendency to make microdependencies just doesn't mesh well with distributions that need to make sure everything has corresponding source and can be built reproducibly.
Ruby gems have been solved since they've been around for longer, as have python packages, even though the latter keeps introducing wrenches to make packaging harder.
> Cargo, npm, and friends run rings around apt from the point of view of developer mindshare, ease of use, platform independent packaging, and so on.
Maybe, but they introduce their fair share of problems as well. I've already mentioned the piss poor support for cross-language dependencies, but let's mention that the "stores" the packages for these are published to are only moderated after the fact - it's really easy to introduce malware into these stores without anyone noticing, and it's happened a bunch of times. Distributions make sure the binaries correspond to the sources[2], and the maintainers are the first to spot any obvious malware or breakage in software as a result of a change. Sometimes it goes under their noses as well[3], but these cases are much more rare than npm breakage. In general, the distribution model allows for better trust and security, and I think this is something that should be valued more than it currently is.
[1]: https://salsa.debian.org/rust-team/debcargo/
[2]: https://reproducible-builds.org/
[3]: https://www.theregister.com/2015/06/17/debian_chromium_hubbu...
applications in the distro should be up to date by simply being in the distro in that case, i.e. we are more talking about applications that are outside the distro and installed on top of it. Because we have dependency issues, containerization is a good method. anything the distro ships itself should always be using the latest of everything the distro ships.
I don't think distros are going to be fixing vulnerabilities in all their packages. Is that what you mean?
There are two actions that need to be taken:
1) fix the bug in the vulnerable package (cut a new version)
2) make sure the new, bug-free version is actually in use
1 is the responsibility of upstream (though sometimes distros will do this too, and release it themselves while they try to upstream the patch).
2 has to be done by somebody for every application using the vulnerable package, either a) automatically (for example, by Debian releasing the new log4j version and all dependent applications picking it up at runtime) or b) individually (by each dependent project's upstream updating the dependency, cutting a new Flatpak or container image, and getting users to update it).
a is much more pleasant than b, at least in the universe as currently constituted. We can imagine a world where b is mostly automated so it is about the same or less work, but we don't live in that world yet.
1) fix the bug in the vulnerable package (cut a new version)
2) make sure the new, bug-free version is actually in use
1 is the responsibility of upstream (though sometimes distros will do this too, and release it themselves while they try to upstream the patch).
2 has to be done by somebody for every application using the vulnerable package, either a) automatically (for example, by Debian releasing the new log4j version and all dependent applications picking it up at runtime) or b) individually (by each dependent project's upstream updating the dependency, cutting a new Flatpak or container image, and getting users to update it).
a is much more pleasant than b, at least in the universe as currently constituted. We can imagine a world where b is mostly automated so it is about the same or less work, but we don't live in that world yet.
Yes, that is what I mean.
That's not really their job. For log4j they could update that package. Any app that breaks as a result needs to be fixed upstream anyway, and then the distro can rebuild that. Fixes like that do get through the process, but it's not specifically the distros job to fix it.
These problems are all about dependencies, which fall to upstream to make managable. If your favorite app breaks all the time it's probably because the code base is a giant mess. Oh, and some of the mess comes from too many different licenses in use, which prevents tighter integration of source code.
These problems are all about dependencies, which fall to upstream to make managable. If your favorite app breaks all the time it's probably because the code base is a giant mess. Oh, and some of the mess comes from too many different licenses in use, which prevents tighter integration of source code.
That is the entire value dynamic linking and a stable ABI and API provide me. The ability to seamlessly replace an updated dependency by restarting an application and loading the library.
If I'm just going to completely rebuild the app and all of it's dependencies I may as well statically link them and get the unique optimizations we can do after link time by analyzing the resulting executable.
If I'm just going to completely rebuild the app and all of it's dependencies I may as well statically link them and get the unique optimizations we can do after link time by analyzing the resulting executable.
It's up to the distro to define what their job is. As I said, I prefer distros that proactively patch security issues regardless of what upstream does :-)
Point being a log4j issue would be downstream of the distro, not upstream.
The fix was to remove a feature from the library. That's upstream. That would also break things relying on the feature, so those apps would need to be fixed. That's also upstream. In each case, upstream WILL have to fix it. It's up to the distro whether they want to do anything about it prior to upstream getting it done.
It's not upstream of the distro.
> It's not the distro's job to ensure the application is secure and up to date
On the contrary, it's the most important feature of distributions.
On the contrary, it's the most important feature of distributions.
so you install Oracle on a Debian installation, it's Debian's job to ensure Oracle is secure?
I'm not talking applications provided by the distribution, those aren't "applications" (i.e. in my usage, something installed onto an existent operating system), but part of the provided operating system itself, and yes, I'd agree it's their job to keep their provided operating system secure. It's not their job to keep external apps secure (but to provide the means for whoever is maintaining those apps to keep them secure).
I'm not talking applications provided by the distribution, those aren't "applications" (i.e. in my usage, something installed onto an existent operating system), but part of the provided operating system itself, and yes, I'd agree it's their job to keep their provided operating system secure. It's not their job to keep external apps secure (but to provide the means for whoever is maintaining those apps to keep them secure).
A distro often has to maintain more than one version of a,library: consider Debian stable, oldstable, and backports.
> self-inflicted problem of some applications
It sure seems distro inflicted to me. The architecture induces fragility. Apps needing to very carefully step to avoid breaking things is the definition of fragile.
It sure seems distro inflicted to me. The architecture induces fragility. Apps needing to very carefully step to avoid breaking things is the definition of fragile.
> massively increase the maintenance burden
The maintenance burden for who? Using isolated apps (Assuming that no-shared-libraries is what we want, which is probably true for desktop systems like and less true for security critical systems or "traditional" linux systems) just places the maintenance burden on the apps where it belongs. Yes, unmaintained apps should run out-of-date libraries.
The maintenance burden for who? Using isolated apps (Assuming that no-shared-libraries is what we want, which is probably true for desktop systems like and less true for security critical systems or "traditional" linux systems) just places the maintenance burden on the apps where it belongs. Yes, unmaintained apps should run out-of-date libraries.
> Assuming that no-shared-libraries is what we want
That's certainly not what I want on any system, desktop or otherwise.
That's certainly not what I want on any system, desktop or otherwise.
Nonsense. There are essentially two conflicting ideals here: what developers want and what distros want. The problem is that distros are generally right and what distros want is also what users want. So it's devs against the world.
Devs want to ship updates whenever they want and force them into their users' machines as soon as possible, and forget about updating dependencies and pin them to ten versions ago so they never have to eat the cost of upgrading them. Devs don't want to make sure their application plays well with others, they just want to package it up in its own universe, the user's hard drive space and bandwidth be damned. Devs never want to finish their software or declare it stable, they just want to ship their machine on Monday morning so users have it installed by Monday afternoon, just to be updated again on Wednesday. Devs want to ship spyware and adware and malware without any oversight. All of this is cheap and profitable for devs but highly antisocial.
Everything is better about the distro model: security, efficiency, and stability, and they represent the users interests even when they conflict with the dev's interests.
"Modern applications" are developed by children and distros are the teacher slapping them on the wrist for misbehaving. Don't bitch about it -- just grow up.
Devs want to ship updates whenever they want and force them into their users' machines as soon as possible, and forget about updating dependencies and pin them to ten versions ago so they never have to eat the cost of upgrading them. Devs don't want to make sure their application plays well with others, they just want to package it up in its own universe, the user's hard drive space and bandwidth be damned. Devs never want to finish their software or declare it stable, they just want to ship their machine on Monday morning so users have it installed by Monday afternoon, just to be updated again on Wednesday. Devs want to ship spyware and adware and malware without any oversight. All of this is cheap and profitable for devs but highly antisocial.
Everything is better about the distro model: security, efficiency, and stability, and they represent the users interests even when they conflict with the dev's interests.
"Modern applications" are developed by children and distros are the teacher slapping them on the wrist for misbehaving. Don't bitch about it -- just grow up.
As a user I just want the app to be exactly like you describe: a) isolated from all other apps b) self-updating and have a release cycle decoupled from all other apps on my system.
In this scenario (for my personal desktop OS) I don't care about bandwidth, disk space or whether an app uses an out of date version of a library. Having a release per week isn't uncommon and isn't somehow a sign that an app has poor quality.
I'm not going to say a generalization is dead because of one counterexample, but It's perhaps not as clear cut as you make it out to be.
In this scenario (for my personal desktop OS) I don't care about bandwidth, disk space or whether an app uses an out of date version of a library. Having a release per week isn't uncommon and isn't somehow a sign that an app has poor quality.
I'm not going to say a generalization is dead because of one counterexample, but It's perhaps not as clear cut as you make it out to be.
> As a user I just want the app to be exactly like you describe
...and with zero security. No 3rd party reviewing it for vulnerable dependencies, homecalling and telemetries or worse, or breach of licenses.
No guarantee of receiving security updates. Especially given that most upstream do not maintain stable release train and do not backport security fixes.
And keep in mind that flatpak sandboxing is only partially effective and it also does not protect the data you put in the sandboxed app in any way.
(Not to mention that more than 4 billion people on this planet have very slow, expensive and unreliable Internet access, and old devices with limited storage space.)
No thanks, I'll stick with distributions.
...and with zero security. No 3rd party reviewing it for vulnerable dependencies, homecalling and telemetries or worse, or breach of licenses.
No guarantee of receiving security updates. Especially given that most upstream do not maintain stable release train and do not backport security fixes.
And keep in mind that flatpak sandboxing is only partially effective and it also does not protect the data you put in the sandboxed app in any way.
(Not to mention that more than 4 billion people on this planet have very slow, expensive and unreliable Internet access, and old devices with limited storage space.)
No thanks, I'll stick with distributions.
For what it’s worth, I’m in the same boat as you.
I have different thoughts in different use cases (for example, I have servers farms and other specific systems where I more strictly batch updates and value a slower cadence), but for my desktop, my mobile devices, and essentially all of my personal systems, I want constant rolling forward progress.
I have different thoughts in different use cases (for example, I have servers farms and other specific systems where I more strictly batch updates and value a slower cadence), but for my desktop, my mobile devices, and essentially all of my personal systems, I want constant rolling forward progress.
As a user you are asking for a measurably worse experience. And given context, I suspect you are a developer, not just a user.
A release per week is a sign that an app has poor quality.
A release per week is a sign that an app has poor quality.
It seems wild for you to posit what users want as a sweeping generalization, have somebody say ~“that’s not what I want and here’s why”, and then just say full stop that their desires are wrong and don’t count anyway.
Maybe your sweeping generalization is just wrong. Users, developers, and distro maintainers are not uniform groups with rigid pre-determined opinions and desires.
Maybe your sweeping generalization is just wrong. Users, developers, and distro maintainers are not uniform groups with rigid pre-determined opinions and desires.
> As a user you are asking for a measurably worse experience.
Worse than what? A program with a million features is going to have a hundred known bugs at any point. Getting ten fixed every week is better than getting 100 fixed 3 months from now.
That's the thing though: as a user (not as a developer, I assure you) I want the million features, and accept the thousand bugs in order to get them. But that also means I want the weekly patches. A desktop app with 1000 features and 1 bug is good quality. As is a desktop app with a million features and the same number of bugs-to-features. You might argue that one should count bugs in the absolute, but that's just moving goalposts. With that bar there simply are no complex high-quality apps! Any useful definition of complex high quality app must be relative.
> I suspect you are a developer, not just a user
Indeed (this is HN after all).
Worse than what? A program with a million features is going to have a hundred known bugs at any point. Getting ten fixed every week is better than getting 100 fixed 3 months from now.
That's the thing though: as a user (not as a developer, I assure you) I want the million features, and accept the thousand bugs in order to get them. But that also means I want the weekly patches. A desktop app with 1000 features and 1 bug is good quality. As is a desktop app with a million features and the same number of bugs-to-features. You might argue that one should count bugs in the absolute, but that's just moving goalposts. With that bar there simply are no complex high-quality apps! Any useful definition of complex high quality app must be relative.
> I suspect you are a developer, not just a user
Indeed (this is HN after all).
> A program with a million features is going to have a hundred known bugs at any point. Getting ten fixed every week is better than getting 100 fixed 3 months from now
Ideally.
In real world though it's more like fix 10, introduce 5 more. So modern software is always in constant beta.
I'd rather have them release every 3 months with proper QA process in place so it'd be fix 100, introduce 20 new or something.
Ideally.
In real world though it's more like fix 10, introduce 5 more. So modern software is always in constant beta.
I'd rather have them release every 3 months with proper QA process in place so it'd be fix 100, introduce 20 new or something.
Thanks for the voice of sanity, Drew.
> However, Electron based applications vary, as they are built on top of different versions of Electron.
I thought one of the selling points of Electron et al. was building apps on web standards. What is changing so fundamentally between versions of Electron that prevents apps from running on the latest release? I wouldn't expect Element to not work on Firefox 432[0] after using it on Firefox 431...
[0] I lost track of Firefox version numbers somewhere between 3 and 50
I thought one of the selling points of Electron et al. was building apps on web standards. What is changing so fundamentally between versions of Electron that prevents apps from running on the latest release? I wouldn't expect Element to not work on Firefox 432[0] after using it on Firefox 431...
[0] I lost track of Firefox version numbers somewhere between 3 and 50
> Traditional packaging introduces several issues, such as dependency and package conflicts, which usually require workarounds that differ from one distribution to another.
I feel that there was no thought here at all regarding the reverse perspective:
Current application development and deployment practices introduce several issues, such as dependency and package conflicts, which usually require workarounds that differ from one application to another.
I feel that there was no thought here at all regarding the reverse perspective:
Current application development and deployment practices introduce several issues, such as dependency and package conflicts, which usually require workarounds that differ from one application to another.
Better title would be "Traditional Packaging Is Not Suitable For Low Skilled Developers"
People are generally opposed to forking, right? If you fork a project, you effectively take over all the responsibilities to fix bugs and security issues etc. in that project, which is, most of the time, more work that the fork is worth. Now, a popular recent trend is “vendoring” your dependencies; i.e. bundling the dependencies with your software. However, vendoring does not differ much from forking. The responsibility may be slightly less, since you only have to fix bugs and issues affecting your application’s usage of the software. But, just like a fork, the longer you go without merging from the upstream version, the more work it is to maintain your fork.
I suggest that vendoring, just like forking, should be avoided, for largely the same reasons.
I suggest that vendoring, just like forking, should be avoided, for largely the same reasons.
vscode example notwithstanding, I do not encounter the 'conflicting packages' problem very often on Arch. In fact, I don't remember the last time I encountered it, other than when it's an explicit choice - like, Python can only have one version of a package installed for the system Python interpreter, so if you try to install both python-numpy and python-numpy-mkl-bin, well, you have to choose. That's how Python works, though you can of course make additional Python environments (venv/conda/whatever) with different packages.
I notice the vscode example involves the AUR, which is not the official repositories. I consider that a broken package, and it should be fixed. One of the packages should rename its main executable or whatever.
Chrome, for example, has various versions, but they ship binaries with different names, e.g. /usr/bin/google-chrome-stable.
The great thing about Arch is that there is one canonical user repository, the AUR. So the official repos should not have conflicts unless absolutely necessary, and the AUR should endeavour not to have conflicts, though it's more of a wild-west so they're somewhat more likely.
However, you generally don't have to think about how to not conflict with whatever random packages may be out there - you only need to consider the repos and the AUR. Anyone making their own packages that are not in the official repos or the AUR, and yet conflict with them, well, that's their problem.
Arch being super up to date and the existence of the AUR really does obviate the need for containerising apps to a large degree, at least from the perspective of ensuring dependencies are satisfied (there are still security reasons to want containerisation). I don't believe I have any containerised apps, and yet I'm running loads of modern common programs like zoom, chrome, discord, spotify, the list goes on.
I notice the vscode example involves the AUR, which is not the official repositories. I consider that a broken package, and it should be fixed. One of the packages should rename its main executable or whatever.
Chrome, for example, has various versions, but they ship binaries with different names, e.g. /usr/bin/google-chrome-stable.
The great thing about Arch is that there is one canonical user repository, the AUR. So the official repos should not have conflicts unless absolutely necessary, and the AUR should endeavour not to have conflicts, though it's more of a wild-west so they're somewhat more likely.
However, you generally don't have to think about how to not conflict with whatever random packages may be out there - you only need to consider the repos and the AUR. Anyone making their own packages that are not in the official repos or the AUR, and yet conflict with them, well, that's their problem.
Arch being super up to date and the existence of the AUR really does obviate the need for containerising apps to a large degree, at least from the perspective of ensuring dependencies are satisfied (there are still security reasons to want containerisation). I don't believe I have any containerised apps, and yet I'm running loads of modern common programs like zoom, chrome, discord, spotify, the list goes on.
> if you try to install both python-numpy and python-numpy-mkl-bin, well, you have to choose. That's how Python works
It's possible to have as many Python versions as you like installed at the same time. All of the libraries and headers include the major/minor version, packages are installed in `lib/pythonX.Y` and the Python binary itself is available as `pythonX.Y`. Few distributions support this well for anything other than a single 2.x and 3.x but it's nothing to do with how Python itself works.
It's possible to have as many Python versions as you like installed at the same time. All of the libraries and headers include the major/minor version, packages are installed in `lib/pythonX.Y` and the Python binary itself is available as `pythonX.Y`. Few distributions support this well for anything other than a single 2.x and 3.x but it's nothing to do with how Python itself works.
That's for different Python interpreters altogether, you still can't have multiple numpy packages in any one of them.
Arch actually supports multiple Pythons just fine, I currently have a bunch of 3.x Python interpreters I installed from the AUR. But installing other Python packages with these interpreters indeed needs to be done outside the system's package manager, e.g. with pip -U or a venv (usually a venv for me).
Arch actually supports multiple Pythons just fine, I currently have a bunch of 3.x Python interpreters I installed from the AUR. But installing other Python packages with these interpreters indeed needs to be done outside the system's package manager, e.g. with pip -U or a venv (usually a venv for me).
> Modern applications are inherently fragile.
Well, there's your problem.
Well, there's your problem.
The problem is that containers are opaque and authors simply don't care about quality. If say, fedora would allow containers by having deviating Versions of rpms per app, it would be quite cool. The OS could inform me about apps using deprecated Versions, dependencies could be shared, etc. But people don't want to put up much effort when shipping their app. They want to build on some version of Ubuntu, add some manual hacks and expect everyone to use that container for the next decade or so. No one cares about dependencies, security or other nonfunctional aspects, sadly.
Yet another article arguing against distribution package managers without understanding them a bit.
Traditional packaging is perfectly suitable for modern applications.
What author is arguing against is the distributions' packaging policies. Distribution policies are what prevents shipping multiple versions of the same library or program, not package managers. These policies are in place because they're necessary for releasing and maintaining a 50000 package distribution.
But for shipping your own software you're free to ignore distribution policies. Package managers have always provided the necessary mechanism. Nothing is preventing you from shoving in all library, binary or data dependencies your program needs into a deb package. You're also free to embed version number in package name and installation paths so different versions could be installed in parallel. The package manager will obey dutifully. Build systems have always been capable of installing libraries into arbitrary paths, and deb/rpm packages are just dumb archive files with metadata slapped on top. If you shove everything needed into it and declare no dependencies, it'll be no different than a dumb tar file and it'll work just fine if you did it right.
You can even imitate flatpak runtimes if you want. Put same things a flatpak runtime has into /usr/lib/freedesktop-sdk-21.08 into a package like freedesktop-sdk-2108.deb, and make your application deb depend on freedesktop-sdk-2108 and you're good to go.
I'm really tired hearing of these baseless arguments over and over over the years. I just wish everyone could be honest and just say "We don't like deb/rpm, they're not cool, they're not hip, we just want to NIH them" so we could be spared of these made up justifications.
P.S. I have nothing against nix. Nix is legit innovation.
Traditional packaging is perfectly suitable for modern applications.
What author is arguing against is the distributions' packaging policies. Distribution policies are what prevents shipping multiple versions of the same library or program, not package managers. These policies are in place because they're necessary for releasing and maintaining a 50000 package distribution.
But for shipping your own software you're free to ignore distribution policies. Package managers have always provided the necessary mechanism. Nothing is preventing you from shoving in all library, binary or data dependencies your program needs into a deb package. You're also free to embed version number in package name and installation paths so different versions could be installed in parallel. The package manager will obey dutifully. Build systems have always been capable of installing libraries into arbitrary paths, and deb/rpm packages are just dumb archive files with metadata slapped on top. If you shove everything needed into it and declare no dependencies, it'll be no different than a dumb tar file and it'll work just fine if you did it right.
You can even imitate flatpak runtimes if you want. Put same things a flatpak runtime has into /usr/lib/freedesktop-sdk-21.08 into a package like freedesktop-sdk-2108.deb, and make your application deb depend on freedesktop-sdk-2108 and you're good to go.
I'm really tired hearing of these baseless arguments over and over over the years. I just wish everyone could be honest and just say "We don't like deb/rpm, they're not cool, they're not hip, we just want to NIH them" so we could be spared of these made up justifications.
P.S. I have nothing against nix. Nix is legit innovation.
I don't want to start a war about static/dynamic linking / shared libraries, so: some people want some shared libraries for some reasons.
One problem with having different applications use different versions of libraries is that you can have multiple versions of a library installed.
Having multiple versions of a library installed undermines some of the reasons for using shared libraries in the first place.
A lot of (most?) software was never meant to be used this way. It's uncommon to see an application that separates its data/config with unique filenames per version, for example. And there's not a global answer to whether or not that data should be shared between versions anyway. [0]
Container-style software distribution largely avoids these issues by not having shared libraries[1] for the most part. This has some benefits like the sandboxing and security features and making consistent behaviour across environments more achievable[2]. But you miss out on benefits from using shared libraries. Surprise -- a software development decision is a trade-off!
[0] Blender stands out for separating config files per version and also letting you easily import an existing config from an earlier version after an update.
[1] or not sharing shared libraries
[2] consistent behaviour across environments is more achievable as the dependencies are shipped with the application (you can ship dependencies without containers as well, e.g. static linking)
One problem with having different applications use different versions of libraries is that you can have multiple versions of a library installed.
Having multiple versions of a library installed undermines some of the reasons for using shared libraries in the first place.
A lot of (most?) software was never meant to be used this way. It's uncommon to see an application that separates its data/config with unique filenames per version, for example. And there's not a global answer to whether or not that data should be shared between versions anyway. [0]
Container-style software distribution largely avoids these issues by not having shared libraries[1] for the most part. This has some benefits like the sandboxing and security features and making consistent behaviour across environments more achievable[2]. But you miss out on benefits from using shared libraries. Surprise -- a software development decision is a trade-off!
[0] Blender stands out for separating config files per version and also letting you easily import an existing config from an earlier version after an update.
[1] or not sharing shared libraries
[2] consistent behaviour across environments is more achievable as the dependencies are shipped with the application (you can ship dependencies without containers as well, e.g. static linking)
I literally wrote part of my phd dissertation on this (can call it proto docker as it predated it and they seemed to use much of the same nomenclature I used, also was sort of a toss in an the very in, though arguably my most important contribution).
see - https://www.usenix.org/legacy/event/lisa11/tech/full_papers/...
see - https://www.usenix.org/legacy/event/lisa11/tech/full_papers/...
Nix solves the mentioned issues without using containers. Not sure why they keep referring to containers as only solution when Nix is included. Also not sure why they talk specifically about graphical applications. The issues mentioned apply to every program running on the system. That said, agree on those issues.
I bet it has something to do with Dockerfile having far simpler syntax than Nix config files.
How can one write an article like this and fail to mention guix? It solves _a_lot_ of the problems mentioned. Guix (and all its channels) is a great piece of software, but unfortunately its potential for populartity is severely diminished by the insistence of the core user/dev group that its goals are not purely technological, but ideological(perhaps even primarily ideological).
It seems that for many devs guix is less of a technical achievemt,and more a way of preventing people from being able to run what they call "non free software". BTW, "non free software" also includes open source software that was released under certain non-compliant licenses.
The biggest issue I have with guix is that any effort of packaging "non-free" software (such as cuda, intel MKL, nvidia drivers etc). Is heavily discouraged, and if it it does happen its discussion is banned from official channels and all authors which make such software available in their personal repos seem to meekly include various pledges in the READMEs "to not advertise this horrible evil thing, or otherwise let people know of its existence". To me this behaviour is very far away from encouraging "free software", but my and their definition of free differ a lot.
My second issue with guix is that the language used to write everything(guile) still lacks many crucial debugging and ease-of-use features such as: being able to display more meanigful error messages, being able to show on which line of compiled code the error/exception happens at, being able to easily interrogate variable value types (there seem to be lots of is_this_value_a_string?, or is_this_value_an_integer? functions, but no what_type_is_this?), no standard way to convert nested objects to string, or to pretty-print deeply nested objects. To this one has to add pretty substantial, but really poorely laid out documentation of the language (from the very first chapter the authors in their examples use all language features that may not be explained until 300 pages in - so the reader basically has to read the whole thing multiple times to "get it"). And so on... All those things matter a lot to beginners and they matter less to advanced users. If this language and guix in general is to catch on, it has to be more beginner friendly.
However as a package manager it really is pretty good. It delivers everything nix does, but uses a more standardised and powerfull general purpose language.
I can't wait until someone forks guix without the whole ideology thing.
It seems that for many devs guix is less of a technical achievemt,and more a way of preventing people from being able to run what they call "non free software". BTW, "non free software" also includes open source software that was released under certain non-compliant licenses.
The biggest issue I have with guix is that any effort of packaging "non-free" software (such as cuda, intel MKL, nvidia drivers etc). Is heavily discouraged, and if it it does happen its discussion is banned from official channels and all authors which make such software available in their personal repos seem to meekly include various pledges in the READMEs "to not advertise this horrible evil thing, or otherwise let people know of its existence". To me this behaviour is very far away from encouraging "free software", but my and their definition of free differ a lot.
My second issue with guix is that the language used to write everything(guile) still lacks many crucial debugging and ease-of-use features such as: being able to display more meanigful error messages, being able to show on which line of compiled code the error/exception happens at, being able to easily interrogate variable value types (there seem to be lots of is_this_value_a_string?, or is_this_value_an_integer? functions, but no what_type_is_this?), no standard way to convert nested objects to string, or to pretty-print deeply nested objects. To this one has to add pretty substantial, but really poorely laid out documentation of the language (from the very first chapter the authors in their examples use all language features that may not be explained until 300 pages in - so the reader basically has to read the whole thing multiple times to "get it"). And so on... All those things matter a lot to beginners and they matter less to advanced users. If this language and guix in general is to catch on, it has to be more beginner friendly.
However as a package manager it really is pretty good. It delivers everything nix does, but uses a more standardised and powerfull general purpose language.
I can't wait until someone forks guix without the whole ideology thing.
>I can't wait until someone forks guix without the whole ideology thing.
There's no reason to fork Guix since ideology doesn't affect the software itself at all. Rather what you want is make an alternative definitions repo. Pretty sure I've seen such one that includes software not accepted on official channels.
There's no reason to fork Guix since ideology doesn't affect the software itself at all. Rather what you want is make an alternative definitions repo. Pretty sure I've seen such one that includes software not accepted on official channels.
Indeed alternative "channels" exist. I mentioned them in my post. However, their authors don't question the ideological side of things. As I mentioned they even include statements how this software is "evil" etc.
What does it matter, you may say, the software is available! Yes, it is, but because of the whole "please don't advertise non-guix non-free(by their definition) software" it is hard to find to new users. Additionally pretty much all support there is for guix is provided by maybe 5 people on the irc channel. If you go to nonguix that number shrinks to perhaps 3 or 2. It is not easy to get support when the whole community is few people.
By "forking without the ideology" I would like to see guix under a new name, the name new users would search and find an installer that used normal Linux Kernel that supports a lot of software,after installing they would find a (hopefully) large community that is helpful in how to package any kind of software. One where a user doesn't have to hide the fact their trying top package software with proprietary components to get help etc. The "pure" guix has a different mission and it can continue with its libre-kernel . As of now, the software things I mention (guix with normal Kernel) exist, but they are not popular, as a result they are not as up to date as they could be. Also the whole project popularity suffers as a result.
What does it matter, you may say, the software is available! Yes, it is, but because of the whole "please don't advertise non-guix non-free(by their definition) software" it is hard to find to new users. Additionally pretty much all support there is for guix is provided by maybe 5 people on the irc channel. If you go to nonguix that number shrinks to perhaps 3 or 2. It is not easy to get support when the whole community is few people.
By "forking without the ideology" I would like to see guix under a new name, the name new users would search and find an installer that used normal Linux Kernel that supports a lot of software,after installing they would find a (hopefully) large community that is helpful in how to package any kind of software. One where a user doesn't have to hide the fact their trying top package software with proprietary components to get help etc. The "pure" guix has a different mission and it can continue with its libre-kernel . As of now, the software things I mention (guix with normal Kernel) exist, but they are not popular, as a result they are not as up to date as they could be. Also the whole project popularity suffers as a result.
> I can't wait until someone forks guix without the whole ideology thing.
What's wrong with the ideology behind it?
What's wrong with the ideology behind it?
Strict ideology itself is wrong if for no other reason than that reality is not black and white.
While I understand where are you coming from, GNU is a great counterbalance to the currents and trends what we have today.
It single-handedly made open source ubiquitous, normalized open source licensing and changed the how software is built.
I'm a strong proponent of the philosophy GNU is putting forward (the reality is GNU is just something which aligns my ideology, I'm not a convert), but I understand that both things are not that easy and non-GNU world will always exist in some form.
On the other hand, regardless of the software I use, all software I develop is strictly GNU/GPLv3. I also contribute to projects with more permissive licenses, but my software is always GPL.
GNU may not be able to convert the whole world to Free software, but it's a necessary force to protect our current environment and evolve it for the better.
Moreover, while not pointed at you directly, GNU and GPL is vastly misunderstood concepts. They are neither evil, nor unreasonable. It just aims to move the freedoms developers enjoy with MIT/Expat licenses to users by making them irrevocable.
It single-handedly made open source ubiquitous, normalized open source licensing and changed the how software is built.
I'm a strong proponent of the philosophy GNU is putting forward (the reality is GNU is just something which aligns my ideology, I'm not a convert), but I understand that both things are not that easy and non-GNU world will always exist in some form.
On the other hand, regardless of the software I use, all software I develop is strictly GNU/GPLv3. I also contribute to projects with more permissive licenses, but my software is always GPL.
GNU may not be able to convert the whole world to Free software, but it's a necessary force to protect our current environment and evolve it for the better.
Moreover, while not pointed at you directly, GNU and GPL is vastly misunderstood concepts. They are neither evil, nor unreasonable. It just aims to move the freedoms developers enjoy with MIT/Expat licenses to users by making them irrevocable.
Guix is basically an open source focused variant of Nix, with a few different design decisions.
It's also considerably less popular than Nix, which is already niche.
It's also considerably less popular than Nix, which is already niche.
The main difference between guix and nix(other than ideology) is the language used to define the packages. Nix uses its own language, guix uses guile(compatible with scheme). Many people will find the nix language a lot more readable and easy to learn, but one may argue scheme/guie is a more general purpose language. It has been in existence for a long time. As such it has a large number of libraries, editing tools and it is easier to do stuff without having to resort to third party tools and scripts. Therefore in the long run guile (for one that learns it well) will prove more useful.
I have to admit I'm not 100% convinced by the argument. I've been learning guile for a couple of weeks and I just start to "get it". Compared with other languages I learned the difficulty with guile was many many times more (for various reasons). I now see it as a pretty powerfull language, but not without many faults.
The above notwithstanding I think something like Guix/Nix will become really popular when a truly easy for beginners to learn and powerfull language is used (such as python for example). Perhaps it would be an interesting project to attempt to auto-translate guix from guile into python. I can definitely see it takeoff then.
I have to admit I'm not 100% convinced by the argument. I've been learning guile for a couple of weeks and I just start to "get it". Compared with other languages I learned the difficulty with guile was many many times more (for various reasons). I now see it as a pretty powerfull language, but not without many faults.
The above notwithstanding I think something like Guix/Nix will become really popular when a truly easy for beginners to learn and powerfull language is used (such as python for example). Perhaps it would be an interesting project to attempt to auto-translate guix from guile into python. I can definitely see it takeoff then.
The question is why now? Distros have for the longest time worked as curated sets of dependencies without "nuclear" options such as containers. At the same time, there have been exactly zero new end-user apps on Linux for over a decade. Incidentally just today I bought a MBP after 12 years on Linux. Won't miss the stagnation, bike shedding, nerding around with new unproven languages, projecting everything into the "next big thing" such as rust, systemd, containers while delivering very little, the DE regressions, comical lack of apps, qa, taste, professionalism, and common sense.
Just spit balling ideas here, how much of the problems containerizing is meant to solve would be solved by merely static linking? Because it feels like that's what containers are mostly used for. Static linking + a sandbox to get most of everything.
Using shared libraries but then shipping the .so's in a container defeats the point of a .so in the first place. If the reason containers are okay now since disk space is cheap, why not return to static linking of executables?
Using shared libraries but then shipping the .so's in a container defeats the point of a .so in the first place. If the reason containers are okay now since disk space is cheap, why not return to static linking of executables?
There is, I think, a bit of a paradigm shift in action.
In the old world, disk space was limited and download speeds were slow. So it made sense for every app to use the system-provided version of Library X.
Additionally, stability and backwards compatibility was assumed. So if the distro updated to Library X v1.2, and your app relied on v1.1, everything would still work.
Today's world is very different (for many users). Disk space is cheap and plentiful and downloading a couple of GB takes an instant. So why not bundle everything together?
Additionally, developers can't be sure of stability in their dependencies. And the app developer is the person who receives the complaints from angry users, not the library developer.
So it makes perfect sense to use a self-contained set of libraries and dependencies.
The pendulum will swing the other way in time. We'll all get annoyed at having a dozen different Electron libraries taking up space, and we'll go back to one-library-to-rule-them all.
But - as someone who used to hate battling dependency hell - I love Flatpack.
In the old world, disk space was limited and download speeds were slow. So it made sense for every app to use the system-provided version of Library X.
Additionally, stability and backwards compatibility was assumed. So if the distro updated to Library X v1.2, and your app relied on v1.1, everything would still work.
Today's world is very different (for many users). Disk space is cheap and plentiful and downloading a couple of GB takes an instant. So why not bundle everything together?
Additionally, developers can't be sure of stability in their dependencies. And the app developer is the person who receives the complaints from angry users, not the library developer.
So it makes perfect sense to use a self-contained set of libraries and dependencies.
The pendulum will swing the other way in time. We'll all get annoyed at having a dozen different Electron libraries taking up space, and we'll go back to one-library-to-rule-them all.
But - as someone who used to hate battling dependency hell - I love Flatpack.
> Disk space is cheap and plentiful and downloading a couple of GB takes an instant.
True in a data center. Not true for many people.
> So why not bundle everything together?
Security updates, RAM usage.
True in a data center. Not true for many people.
> So why not bundle everything together?
Security updates, RAM usage.
> True in a data center. Not true for many people.
Even the cheapest laptop you can buy today has literally 1,000x the disk space of a 1990's tower PC. The packaging format APT was developed in the late 1990s when the average user's modem speed was unlikely to hit 56kbps.
Times have changed. Time to update the way we package software.
> Security updates, RAM usage.
True in a data centre. Not true for many people.
Even the cheapest laptop you can buy today has literally 1,000x the disk space of a 1990's tower PC. The packaging format APT was developed in the late 1990s when the average user's modem speed was unlikely to hit 56kbps.
Times have changed. Time to update the way we package software.
> Security updates, RAM usage.
True in a data centre. Not true for many people.
Even the simplest of program today is way bigger than an alternative of yesterday. I don't see your point.
There's also the option that we could finally end the hardware upgrade cycle. After all, the laptop that I already have costs me less both in money and setup time than even the cheapest laptop I could buy if I was forced to. For desktop app developers it's other people's money so it's not as important.
> Additionally, stability and backwards compatibility was assumed. So if the distro updated to Library X v1.2, and your app relied on v1.1, everything would still work.
I thought all the cool kids had adopted `semver` these days, so this shouldn't be a problem? Except for crappy libraries/dependencies, in which case surely the answer is: Don't use crappy dependencies; find an alternative that does the same thing, without sucking.
> Additionally, developers can't be sure of stability in their dependencies.
Again, be more judicious in which dependencies you use. Stay away from the crap ones, find ones that do the right thing.
> Disk space is cheap and plentiful and downloading a couple of GB takes an instant.
Maybe for you. But not for everyone on the planet. Or, maybe people who live in places with intermittent internet connectivity don't deserve new software, or for their needs to be considered?
> So why not bundle everything together?
Security. If a serious vulnerability is found in Library X v1.1, I want to be able to upgrade it to Library X v1.1.1 and know that every single app that uses it is safe, without even needing to know which apps use it. I don't want to have to figure out which apps use it (which, OK, might not be too hard if the tools are written to find that out), and then have to download an update for every single one. Even if I do have unlimited gigabit broadband, that could still be awkward if the library is sufficiently popular and widely used.
I thought all the cool kids had adopted `semver` these days, so this shouldn't be a problem? Except for crappy libraries/dependencies, in which case surely the answer is: Don't use crappy dependencies; find an alternative that does the same thing, without sucking.
> Additionally, developers can't be sure of stability in their dependencies.
Again, be more judicious in which dependencies you use. Stay away from the crap ones, find ones that do the right thing.
> Disk space is cheap and plentiful and downloading a couple of GB takes an instant.
Maybe for you. But not for everyone on the planet. Or, maybe people who live in places with intermittent internet connectivity don't deserve new software, or for their needs to be considered?
> So why not bundle everything together?
Security. If a serious vulnerability is found in Library X v1.1, I want to be able to upgrade it to Library X v1.1.1 and know that every single app that uses it is safe, without even needing to know which apps use it. I don't want to have to figure out which apps use it (which, OK, might not be too hard if the tools are written to find that out), and then have to download an update for every single one. Even if I do have unlimited gigabit broadband, that could still be awkward if the library is sufficiently popular and widely used.
>Additionally, developers can't be sure of stability in their dependencies. And the app developer is the person who receives the complaints from angry users, not the library developer.
That seems like the main issue to me. Why has it become acceptable for libraries to break backwards compatibility left and right?
That seems like the main issue to me. Why has it become acceptable for libraries to break backwards compatibility left and right?
> In the old world, disk space was limited and download speeds were slow
No, the problem is that production systems need to receive fine-grained and timely security updates.
You can't rebuild, test and deploy a whole OS every time a vulnerability in OpenSSL is fixed. It would be extremely risky for any non-trivial production workload.
Disk space is not the issue. Distributions do the crucial and painstaking work of backporting security fixes.
This cannot be done when applications are shipped with huge blobs of dependencies, like with flatpack or containers, it's simply too much work for any organization to do that.
No, the problem is that production systems need to receive fine-grained and timely security updates.
You can't rebuild, test and deploy a whole OS every time a vulnerability in OpenSSL is fixed. It would be extremely risky for any non-trivial production workload.
Disk space is not the issue. Distributions do the crucial and painstaking work of backporting security fixes.
This cannot be done when applications are shipped with huge blobs of dependencies, like with flatpack or containers, it's simply too much work for any organization to do that.
The text suggests to just containerise any app, so I am wondering, what is the current status for containers (on Linux):
Can you access files outside of a container from inside a container?
Can you access devices like an eGPU from inside a container?
Can you access files outside of a container from inside a container?
Can you access devices like an eGPU from inside a container?
This is addressed in the article. Search for permissions and e.g. XDG portals.
I have seen the link to XDG portals, but it doesn't say what is possible with it.
Yes to both.
Is there a good book about all of that?
I dont know about books but I mostly learned from experimentation and https://docs.podman.io/en/latest/
I dunno about "modern" - I can recall distro packaging making a total disaster of Ruby gems over a decade ago.
The solution is simple: ship your fucking dependencies.
Every application should ship every dependency it requires. Attempting to use random shit that may or may not be installed on the system is a disaster.
Containers that include all dependencies achieves this. Infact I like to argue that the only reason containers exist is because reliably launching software is so outrageously complex the only way to do is create a full system imagine. It’s a modern day tragedy.
Disk space is cheap. If every application shipped duplicate dependencies it’d be… totally fine. The downside is inability to update security vulnerabilities. But if everyone is shipping containers that seems kinda moot.
iOS and Android don’t have this issue. When you install an app it just works. Windows has the issue somewhat, but it’s mostly a problem with dev tools that are trying to pretend their in Linux. If you ship your fucking dependencies it’s a non-issue.
Every application should ship every dependency it requires. Attempting to use random shit that may or may not be installed on the system is a disaster.
Containers that include all dependencies achieves this. Infact I like to argue that the only reason containers exist is because reliably launching software is so outrageously complex the only way to do is create a full system imagine. It’s a modern day tragedy.
Disk space is cheap. If every application shipped duplicate dependencies it’d be… totally fine. The downside is inability to update security vulnerabilities. But if everyone is shipping containers that seems kinda moot.
iOS and Android don’t have this issue. When you install an app it just works. Windows has the issue somewhat, but it’s mostly a problem with dev tools that are trying to pretend their in Linux. If you ship your fucking dependencies it’s a non-issue.
Every security update means updating every big bundle of packages when it's a small change in some commonly used core library.
Even IF my bandwidth and time were free (neither are) that is still a ton of duplicated work and wear on non-volatile memory life.
Even IF my bandwidth and time were free (neither are) that is still a ton of duplicated work and wear on non-volatile memory life.
> Every security update means updating every big bundle of packages when it's a small change in some commonly used core library.
Replacing all instances of libFFMoeg.so on a drive is not particularly harder than replacing one.
> that is still a ton of duplicated work and wear on non-volatile memory life.
Nonsense. We’re talking about replacing libraries on the order tens to hundreds of megabytes during a monthly update. It’s peanuts.
Time I’ll give you. But the current system of system-wide libraries is objectively broken. As evidenced by the increasing usage of containers to work around the brokenness. I’d wager far more time is wasted on docker orchestration.
It’s probably worth differentiating between servers and personal machines. Lord knows how many vulnerabilities are in your Steam library. It’s a lot.
Replacing all instances of libFFMoeg.so on a drive is not particularly harder than replacing one.
> that is still a ton of duplicated work and wear on non-volatile memory life.
Nonsense. We’re talking about replacing libraries on the order tens to hundreds of megabytes during a monthly update. It’s peanuts.
Time I’ll give you. But the current system of system-wide libraries is objectively broken. As evidenced by the increasing usage of containers to work around the brokenness. I’d wager far more time is wasted on docker orchestration.
It’s probably worth differentiating between servers and personal machines. Lord knows how many vulnerabilities are in your Steam library. It’s a lot.
I always feel that takes like "a few hundreds of MB is peanuts" are the marker of someone so privileged, that they have never thought about those not living in a developed country.
We really need to get this mentality sorted, as we are quite literally blocking everyone else from better technology.
We really need to get this mentality sorted, as we are quite literally blocking everyone else from better technology.
Even in developed countries there are people and areas that don't have fast internet connections, at least not without significant cost.
> Replacing all instances of libFFMoeg.so on a drive is not particularly harder than replacing one.
but how would you know the new version works with the app?
And how do you even know the app needed this library? Are you going to now investigate every app, and check that the library needed updating?
It's work, even if each individual app has only a small amount, it will add up. And not to mention _every_ user would need to do it for themselves.
but how would you know the new version works with the app?
And how do you even know the app needed this library? Are you going to now investigate every app, and check that the library needed updating?
It's work, even if each individual app has only a small amount, it will add up. And not to mention _every_ user would need to do it for themselves.
> but how would you know the new version works with the app?
How is this any different than replacing one instance of a library that is then used by all applications?
How is this any different than replacing one instance of a library that is then used by all applications?
It’s not harder for you, when you do it, for your package which you maintain. However, when your machine has N large bundles of packages installed, and an issue is found with a vendored dependency in all of them, you have to wait for every single one of them to be upgraded, which requires the work of N people, individually, none of whom work for you.
> Replacing all instances of libFFMoeg.so on a drive is not particularly harder than replacing one.
Instead, ensuring that your applications do not break is extremely hard.
Instead, ensuring that your applications do not break is extremely hard.
It's only due to the way docker containers and the like are constructed. Predating docker by a few years I built a docker like mechanism that viewed what docker calls layers (i also used the term) as packages in a traditional distribution.
Instead of unpackaging a package into a file systme, the package is just a layer that can be unioned in on demand (for live updates, though in the age of containizerization, this hasn't aged as well, so in practice it be running a new container).
link to paper is somewhere in the threads here
Instead of unpackaging a package into a file systme, the package is just a layer that can be unioned in on demand (for live updates, though in the age of containizerization, this hasn't aged as well, so in practice it be running a new container).
link to paper is somewhere in the threads here
> The solution is simple: ship your fucking dependencies.
And then you get Electron, and people start complaining about the fact that you ship all your dependencies instead of using the dependencies that are already there.
And then you get Electron, and people start complaining about the fact that you ship all your dependencies instead of using the dependencies that are already there.
Who is complaining about the install size of Electron? The problem with Electron is it’s laggy and slow. Using a different source for dependencies wouldn’t solve performance. It’d just slightly reduce storage footprint.
I mean there’s always someone who complains about something. But if Electron apps had a large folder of binaries but it was super fast and responsive that would eliminate the vast majority of complaints.
I mean there’s always someone who complains about something. But if Electron apps had a large folder of binaries but it was super fast and responsive that would eliminate the vast majority of complaints.
> The problem with Electron is it’s laggy and slow.
electron isn't laggy or slow, the app might be because it's poorly written. There are fast electron apps!
The complaint about electron i hear the most are 1) the memory footprint - each instance of electron takes up at least a gig of ram. 2) the browser is duplicated, and cannot be upgraded independently (but i argue that is the point of electron). The third, but irrelevant complaint is from people on high horses talking about wanting native apps, and look down on html based apps.
electron isn't laggy or slow, the app might be because it's poorly written. There are fast electron apps!
The complaint about electron i hear the most are 1) the memory footprint - each instance of electron takes up at least a gig of ram. 2) the browser is duplicated, and cannot be upgraded independently (but i argue that is the point of electron). The third, but irrelevant complaint is from people on high horses talking about wanting native apps, and look down on html based apps.
My complaint with electron is they don't keep up with security updates to the browser engine that electron is a fork of. Even electron-nightly is based on an outdated version of chromium.
Enough people where all of Electron's competitors (Tauri, et al) list their smaller install size as one of the main selling points.
Install size matters to me. Not as much as other things such as performance and memory footprint, but it still matters.
Flutter is as good if not better than electron and requires a minimal set of dependencies (gtk, libc). Everything else can be statically linked using plugins.
And this is what NeXTStep/macOS has done for 30 years. Well you could also ship common Frameworks but they all have a version number so should not cause a problem if different apps need different versions of a library.
The AppStore or using a library like Sparkle will update when the developer finds an issue or security issue with a dependency.
The advantage here is that the OS knows about bundles and so gives an organisation above the file. Anything in that bundle should not be used by anything else. I suspect this is what using containers gives Linux
Alternative do as work did 30 years ago - explicitly link to a versioned number of a library. OK then you need to identify which apps use a particular library that needs to be updated for security reasons.
The AppStore or using a library like Sparkle will update when the developer finds an issue or security issue with a dependency.
The advantage here is that the OS knows about bundles and so gives an organisation above the file. Anything in that bundle should not be used by anything else. I suspect this is what using containers gives Linux
Alternative do as work did 30 years ago - explicitly link to a versioned number of a library. OK then you need to identify which apps use a particular library that needs to be updated for security reasons.
What is the difference between shipping dependencies and shipping container?
Thin layer of metadata? Or am I missing something?
Yes. For a desktop system where you want to update complex multi-gig apps perhaps once per week this isn't just the better option, this is the ONLY option.
For a smartphone, the same.
For a server or security sensitive system? Perhaps a different scenario. You wouldn't update a dozen large complex apps every week on such a system.
I think the issue is that the distros with their application trees are trying to solve a handful of completely different problems. Me updating a 5GB IDE and a Game once a week is not the same problem to solve as running a web server.
For a smartphone, the same.
For a server or security sensitive system? Perhaps a different scenario. You wouldn't update a dozen large complex apps every week on such a system.
I think the issue is that the distros with their application trees are trying to solve a handful of completely different problems. Me updating a 5GB IDE and a Game once a week is not the same problem to solve as running a web server.
I think many, too many still fail to see two thing:
- packages/package management is intimately tied to filesystems and so installers. Actual package models are based on '80s Unix model that do not scale anymore since decades. Such model was designed decoupling "system code" to "user code" (script, mostly) while all along the IT history, both before and after unix we see a better model (see below);
- packages in FLOSS are NOT meant do be made by upstream developers, that's a need of commercial only software. In FLOSS systems packagers of any distro package upstream code.
Now a step at a time: classic systems where designed as a single application, indefinitively extensible by many, and users included. At a certain point in time AT&T/IBM and others say "ah, no that's bad" IMVHO just to deprive users of much power and gain the ability to sell limited, limiting and poorly designed crap. For more than a decade such decoupling take shape. After decades of big mess frameworks spring to life as a kind of middle-ground and they prove to be no better than the decoupled crap they try to solve without coming back to the classic OS-as-a-single-application model. Now essentially ALL non trivial applications being unable to effectively integrate each others try to be "the only one application" and finding no way to do so some pull out of the hat an even worse cage: hiding the OS on someone else computer, used as a bootloader of a modern more limited and limiting Java VM: the WebVM improperly named browser for legacy reasons. Pushing "the network is the computer" to "the cloud is the system".
To solve this mess what we need is the original solution: the OS as a single application where source code is in the runtime, no "packaging" just SCM. Oh, surely that does not solve version conflict between source, these actually are ALREADY solved in architectural terms by the FLOSS distribution model: packagers assemble code. But working directly on source makes that far easier with simple sharable patches who happen to be also a bit distro-neutral. Nix/Guix offer the actual middle ground.
To solve the storage mess it's far more complex. Today most solutions simply use a backend storage service, a kind of DB, no matter if SQL-alike or key-value or something else. SUN with zfs realize a very first limited evolution in that sense, but far from current needs. BeOS with if raw "query-able file system" have done another very small part before. https://twizzler.io seems to offer something more modern, but that's still an open issue.
- packages/package management is intimately tied to filesystems and so installers. Actual package models are based on '80s Unix model that do not scale anymore since decades. Such model was designed decoupling "system code" to "user code" (script, mostly) while all along the IT history, both before and after unix we see a better model (see below);
- packages in FLOSS are NOT meant do be made by upstream developers, that's a need of commercial only software. In FLOSS systems packagers of any distro package upstream code.
Now a step at a time: classic systems where designed as a single application, indefinitively extensible by many, and users included. At a certain point in time AT&T/IBM and others say "ah, no that's bad" IMVHO just to deprive users of much power and gain the ability to sell limited, limiting and poorly designed crap. For more than a decade such decoupling take shape. After decades of big mess frameworks spring to life as a kind of middle-ground and they prove to be no better than the decoupled crap they try to solve without coming back to the classic OS-as-a-single-application model. Now essentially ALL non trivial applications being unable to effectively integrate each others try to be "the only one application" and finding no way to do so some pull out of the hat an even worse cage: hiding the OS on someone else computer, used as a bootloader of a modern more limited and limiting Java VM: the WebVM improperly named browser for legacy reasons. Pushing "the network is the computer" to "the cloud is the system".
To solve this mess what we need is the original solution: the OS as a single application where source code is in the runtime, no "packaging" just SCM. Oh, surely that does not solve version conflict between source, these actually are ALREADY solved in architectural terms by the FLOSS distribution model: packagers assemble code. But working directly on source makes that far easier with simple sharable patches who happen to be also a bit distro-neutral. Nix/Guix offer the actual middle ground.
To solve the storage mess it's far more complex. Today most solutions simply use a backend storage service, a kind of DB, no matter if SQL-alike or key-value or something else. SUN with zfs realize a very first limited evolution in that sense, but far from current needs. BeOS with if raw "query-able file system" have done another very small part before. https://twizzler.io seems to offer something more modern, but that's still an open issue.
Containers are golden images. You find combination of stuff that works, carefully package them together and ship this whole package. Gold images run on bare metal, containers run on container runtime enabled kernel.
Sure, containerizing an application is /easier/ than working out the dependency trees and solving dependency conflicts between them. It is by no means fundamentally better approach.
If anything, this argument dismisses most work done by distros - building and solving those dependency trees. Maintainers of free OSes sometimes do not bother to solve all possible conflicts, however instead of breaking the system/packages, they do inform you of possible conflicts that are not solved.
Inability to cherry-pick dependencies is deceptively wrong. Traditional distributions resort to use of so called meta-packages and alternatives to keep the system up to date. Deb/rpm packages are simple archives with additional metadata slapped on. The packaging system is transparent to contents: nothing forbids you to package versions 1.0 and 2.0 and have them happily coexist as long as they do not share common paths. In fact, a lot of packages are distributed in exactly the same way.
It seems that the author has a solution - containers - and tries to find a problem for said solution.