Ask HN: Why does every package+module system become a Rube Goldberg machine?
221 comments
> In contrast, we apparently have no "package management theory". We have not figured out the primitives required to express dependencies
We have a good hunch. The basic theory behind Nix definitely goes in the right direction, and if we look away from all the surface-level nonsense going on in Nix, it's conceptually capable (e.g. [0]) of being a first-class language dependency manager.
For this to work at scale we'd need to overcome a couple of large problems though (in ascending order of complexity):
1. A better technical implementation of the model (working on it [1]).
2. A mindset shift to make people understand that "binary distribution" is not a goal, but a side-effect of a reasonable software addressing and caching model. Without this conceptual connection, everything is 10x harder (which is why e.g. Debian packaging is completely incomprehensible - their fundamental model is wrong).
3. A mindset shift to make people understand that their pet programming language is not actually a special snowflake. No matter what the size of your compilation units is, whether you call modules "modules", "classes" or "gorboodles", whether you allow odd features like mutually-recursive dependencies and build-time arbitrary code execution etc.: Your language fits into the same model as every other language. You don't have to NIH a package manager.
This last one is basically impossible at the current stage. Maybe somewhere down the line, if we manage to establish such a model successfully in a handful of languages and people see for themselves, but for now we have to just hold out.
[0]: https://code.tvl.fyi/about/nix/buildGo
[1]: https://cs.tvl.fyi/depot/-/tree/tvix/
We have a good hunch. The basic theory behind Nix definitely goes in the right direction, and if we look away from all the surface-level nonsense going on in Nix, it's conceptually capable (e.g. [0]) of being a first-class language dependency manager.
For this to work at scale we'd need to overcome a couple of large problems though (in ascending order of complexity):
1. A better technical implementation of the model (working on it [1]).
2. A mindset shift to make people understand that "binary distribution" is not a goal, but a side-effect of a reasonable software addressing and caching model. Without this conceptual connection, everything is 10x harder (which is why e.g. Debian packaging is completely incomprehensible - their fundamental model is wrong).
3. A mindset shift to make people understand that their pet programming language is not actually a special snowflake. No matter what the size of your compilation units is, whether you call modules "modules", "classes" or "gorboodles", whether you allow odd features like mutually-recursive dependencies and build-time arbitrary code execution etc.: Your language fits into the same model as every other language. You don't have to NIH a package manager.
This last one is basically impossible at the current stage. Maybe somewhere down the line, if we manage to establish such a model successfully in a handful of languages and people see for themselves, but for now we have to just hold out.
[0]: https://code.tvl.fyi/about/nix/buildGo
[1]: https://cs.tvl.fyi/depot/-/tree/tvix/
And Guix has put that in an beautiful form. There are two things which make Guix special:
1. The package definitions are just a normal, battle-proven, very well defined general-purpose, functional-style supporting programming language (Scheme).
2. There is no conceptual difference between a package definition in the public Guix system, and a self-written package definition which a developers makes to build and test his own package, or to build and run a specific piece of software. The difference is equally small as between using an Emacs package, and configuring that package in ones .emacs configuration file.
1. The package definitions are just a normal, battle-proven, very well defined general-purpose, functional-style supporting programming language (Scheme).
2. There is no conceptual difference between a package definition in the public Guix system, and a self-written package definition which a developers makes to build and test his own package, or to build and run a specific piece of software. The difference is equally small as between using an Emacs package, and configuring that package in ones .emacs configuration file.
I think 3 is probably true for language developers, but for users, language repos feel like they are needed because like none of the linux distros really ship everything you need, or even a reliable fraction without installing a bunch of dumb hacks to get it to work. It's much easier just to do "pip install" so that's where the demand is.
And sure, may be you're right that distro packaging is the "wrong model," again, that is the problem then for distros, users are stuck using ubuntu or whatever so they don't have the option to do the "right" thing, so they do use the mishmash of packaging/repo systems as just the cost of doing business.
And sure, may be you're right that distro packaging is the "wrong model," again, that is the problem then for distros, users are stuck using ubuntu or whatever so they don't have the option to do the "right" thing, so they do use the mishmash of packaging/repo systems as just the cost of doing business.
Apart from not shipping what you need, you often only get a fraction of the versions you need.
For development you may want to test your code against multiple versions of the system libraries. This is not easy using a distro package manager.
For development you may want to test your code against multiple versions of the system libraries. This is not easy using a distro package manager.
Maybe our definition of "easy" doesn't agree, but for Ksplice, we had to build kernels using the exact same, often old versions of GCC and everything else (binutils, etc). We made schroots (this predated docker) and then we used the system package manager to install the requisite old version of packages. Wasn't trivial to get the whole system setup, but getting old versions of packages installed wasn't impossibly hard either.
Nix is not capable of becoming a first class dependency manager because it does not manage dependencies. It does not attempt any version or feature resolution. Besides, saying Nix could become a package manager is like saying that C could become a package manager.
Nix is an umbrella for a stack of various solutions.
Nixpkgs doesn't do version or feature resolution, but other tools (that are not nixpkgs) can and do.
Nixpkgs doesn't do version or feature resolution, but other tools (that are not nixpkgs) can and do.
Under this abuse of terminology, a pocket calculator is capable of becoming a package/dependency manager. Or any other technology which conceivably could be put to work toward the package management problem, but has not yet been adapted to that need. I concede that these statements are "true", but they fail the relevancy test of communication.
Like I said in another comment, the problem is curating the giant database that maps version numbers to commit refs. Once you pass this roadblock, the tools Nix gives you make dependency resolution a quite a bit simpler problem to solve.
The next step is then realizing you actually don't need the useless legacy version numbering scheme at all, and that you wasted that effort for nothing.
The next step is then realizing you actually don't need the useless legacy version numbering scheme at all, and that you wasted that effort for nothing.
Something like this maybe? https://lazamar.co.uk/nix-versions/
I'm not sure what you mean, through, by not needing the actual program versions.
I'm not sure what you mean, through, by not needing the actual program versions.
What we actually want is a content-addressable global namespace of source code. (Maybe it could be marketed as "blockchain for software" or something, lol.)
Version numbers are an attempt at content-addressing from the time when software came on floppy disks and CD-ROMs. It sorta made sense given the constraints of that technology.
Nowadays trusting git tags over git revision hashes makes no real sense. (Except that git tags are kinda shorter to type, but who cares.)
Version numbers are an attempt at content-addressing from the time when software came on floppy disks and CD-ROMs. It sorta made sense given the constraints of that technology.
Nowadays trusting git tags over git revision hashes makes no real sense. (Except that git tags are kinda shorter to type, but who cares.)
tazjin(1)
2) In Debian a binary package has almost always just been a cache, are we thinking about Debian packaging in completely different ways? Are you perhaps talking about the aspect of doing an "build world"[0] that bsd, Gentoo and now Nix are better at?
[0] Edit: rebuilding your dependencies if you need it and handling that seemlesly CAN be hard on Debian. Something that even Nix struggles with even if they are best in class by far. It is also completely different from the notion of compiling a program. I wonder what you consider the goal of a package system.
[0] Edit: rebuilding your dependencies if you need it and handling that seemlesly CAN be hard on Debian. Something that even Nix struggles with even if they are best in class by far. It is also completely different from the notion of compiling a program. I wonder what you consider the goal of a package system.
> are we thinking about Debian packaging in completely different ways?
Quite likely! The whole concept of separately building a source/binary package, and then uploading/"deploying" that binary package, already violates the notion of being "just a cache" for me. There might be a tool in Debian where I can seamlessly say "do not download this package from the repository, but build it locally" - but even if so, it would surprise me if it can give me the same guarantees as Nix (i.e. guarantees about the artifact being repeatable, and being addressable by its inputs rather than a human-curated tag or version number).
> Something that even Nix struggles with
Nix only struggles with it to the degree that some of the higher-level abstractions in the existing package set (which are built on top of the fundamental model, not part of it) can be confusing/underdocumented, but conceptually this is a simple thing to do in Nix. Even practically at this point it is usually simple - unless you're dealing with Haskell or Javascript, of course.
> I wonder what you consider the goal of a package system
I want it to let me describe a desired state, and then make it so. That state is best represented as a graph, like a Merkle tree, of the instructions for the individual steps and a way to address them. An individual step is a transformation (i.e. some program) executed over some sources, yielding some result (likely in the filesystem). I want any distribution of binaries to be a result of using this addressing scheme, and looking up/copying an already built equivalent artifact for that thing. I want a text file containing the string "Hello HN" to be represented by the same abstraction that represents a specific .so file, the `emacs` package, the configuration of my entire system, the configuration of a cluster of systems and so on. I want this system to be programmable so I can work around the shortcomings that its original designers missed.
Nix (and Guix) do large parts of this already, and are conceptually (though not yet technically) capable of doing all of it. Technically, so is something like Bazel - but its complexity and maintenance requirements make that prohibitively expensive (basically only Google can use Bazel like that, and even they have delineated areas where they "give up" on wrapping things in Bazel).
Quite likely! The whole concept of separately building a source/binary package, and then uploading/"deploying" that binary package, already violates the notion of being "just a cache" for me. There might be a tool in Debian where I can seamlessly say "do not download this package from the repository, but build it locally" - but even if so, it would surprise me if it can give me the same guarantees as Nix (i.e. guarantees about the artifact being repeatable, and being addressable by its inputs rather than a human-curated tag or version number).
> Something that even Nix struggles with
Nix only struggles with it to the degree that some of the higher-level abstractions in the existing package set (which are built on top of the fundamental model, not part of it) can be confusing/underdocumented, but conceptually this is a simple thing to do in Nix. Even practically at this point it is usually simple - unless you're dealing with Haskell or Javascript, of course.
> I wonder what you consider the goal of a package system
I want it to let me describe a desired state, and then make it so. That state is best represented as a graph, like a Merkle tree, of the instructions for the individual steps and a way to address them. An individual step is a transformation (i.e. some program) executed over some sources, yielding some result (likely in the filesystem). I want any distribution of binaries to be a result of using this addressing scheme, and looking up/copying an already built equivalent artifact for that thing. I want a text file containing the string "Hello HN" to be represented by the same abstraction that represents a specific .so file, the `emacs` package, the configuration of my entire system, the configuration of a cluster of systems and so on. I want this system to be programmable so I can work around the shortcomings that its original designers missed.
Nix (and Guix) do large parts of this already, and are conceptually (though not yet technically) capable of doing all of it. Technically, so is something like Bazel - but its complexity and maintenance requirements make that prohibitively expensive (basically only Google can use Bazel like that, and even they have delineated areas where they "give up" on wrapping things in Bazel).
> There might be a tool in Debian where I can seamlessly say "do not download this package from the repository, but build it locally"
apt-get --build source package_name
It's unlikely to be fully repeatable to the point of producing an identical binary, but it's also not some esoteric set of procedures.
apt-get --build source package_name
It's unlikely to be fully repeatable to the point of producing an identical binary, but it's also not some esoteric set of procedures.
> There might be a tool in Debian where I can seamlessly say "do not download this package from the repository, but build it locally" - but even if so, it would surprise me if it can give me the same guarantees as Nix
It's been there for 15 or 20 years. Also Debian has been doing reproducible builds before Nix.
It's been there for 15 or 20 years. Also Debian has been doing reproducible builds before Nix.
Even in the best case where Debian satisfies all the reproducibility goals it sets for itself (i.e. the ~96% at https://tests.reproducible-builds.org/debian/reproducible.ht... becomes 100%), each Debian package is still only a reproducible function of the current versions of its build dependencies at the time it was built. Upgrading those dependencies can change the build output, and this is still normal and expected in Debian; it’s the entire point of a binNMU.
In order to reproduce a given Debian package from scratch, one would need to track down long chains of historical versions of packages from sid. Due to circular dependencies, these chains probably snake back to the very first versions of Debian before the availability of snapshot.debian.org. It’s possible that human judgment could be used to break some chains at an earlier point by substituting different versions, but this goes outside the guarantees provided by Debian’s reproducibility.
Nix guarantees that every package is built with the specified versions of its dependencies, and so on recursively. Some more rebuilding is necessary to maintain this guarantee, but in exchange, it enables stronger notions of reproducibility that Debian can’t.
In order to reproduce a given Debian package from scratch, one would need to track down long chains of historical versions of packages from sid. Due to circular dependencies, these chains probably snake back to the very first versions of Debian before the availability of snapshot.debian.org. It’s possible that human judgment could be used to break some chains at an earlier point by substituting different versions, but this goes outside the guarantees provided by Debian’s reproducibility.
Nix guarantees that every package is built with the specified versions of its dependencies, and so on recursively. Some more rebuilding is necessary to maintain this guarantee, but in exchange, it enables stronger notions of reproducibility that Debian can’t.
Well of course it snakes back, and likely beyond debian's inception, your initial c compiler has to come from somewhere.
Right. Nix bootstraps from a fixed collection of seed binaries, and everything in any given commit of nixpkgs is derived from the seed binaries and versions of software that exist in that commit of nixpkgs. It doesn’t solve the “trusting trust” problem with those seed binaries, but it does solve the “arbitrarily long chains” problem.
(The work at https://www.gnu.org/software/mes/ aims to minimize the needed binary seed to a human-auditable size, and should be applicable to Nix and Guix.)
(The work at https://www.gnu.org/software/mes/ aims to minimize the needed binary seed to a human-auditable size, and should be applicable to Nix and Guix.)
> It's been there for 15 or 20 years.
Has it? What I remember being the standard practice for customizing packages was
(Some of that may also apply to `apt-get source --build`, idk.)
With Nix or Guix, when a cached binary artifact for a given package is missing, you may not even notice, the fallback to building from source is so transparent.
I encourage you to actually try Nix or Guix, including writing packages for them, and including adding a patch or overriding a build flags in an existing package, to see how the model is really different.
> Also Debian has been doing reproducible builds before Nix.
What Debian does for reproducible builds and how Nix embodies reproducibility are qualitatively different. If you understood both, you'd understand why this is a silly point to raise.
(Debian's work on reproducible builds is great. Every distro benefits from most of it, and everyone in the Nix community is grateful for it.)
Has it? What I remember being the standard practice for customizing packages was
apt-get source && sudo apt-get build-dep && vim <something> && debuild -b
(Some of that may also apply to `apt-get source --build`, idk.)
With Nix or Guix, when a cached binary artifact for a given package is missing, you may not even notice, the fallback to building from source is so transparent.
I encourage you to actually try Nix or Guix, including writing packages for them, and including adding a patch or overriding a build flags in an existing package, to see how the model is really different.
> Also Debian has been doing reproducible builds before Nix.
What Debian does for reproducible builds and how Nix embodies reproducibility are qualitatively different. If you understood both, you'd understand why this is a silly point to raise.
(Debian's work on reproducible builds is great. Every distro benefits from most of it, and everyone in the Nix community is grateful for it.)
With Nix or Guix, when a cached binary artifact for a given package is missing, you may not even notice, the fallback to building from source is so transparent.
I consider this to be one of the main drawbacks of these systems -- it assumes that every machine is build-capable. I do not want compilers installed on my production servers (nor would I want my RPi to attempt to rebuild half the world because the package server was down). I'm not sure how Nix handles this, but I'm pretty sure that's not possible with Guix.
I consider this to be one of the main drawbacks of these systems -- it assumes that every machine is build-capable. I do not want compilers installed on my production servers (nor would I want my RPi to attempt to rebuild half the world because the package server was down). I'm not sure how Nix handles this, but I'm pretty sure that's not possible with Guix.
> it assumes that every machine is build-capable
It doesn't, fortunately. You can tell Nix to rely exclusively on the binary cache, or configure it to have 0 build threads locally and use remote builders.
For a server deployment, many popular choices don't have the server build anything, and in some cases it may not even have any users who are allowed to invoke `nix-*` or anything.
It doesn't, fortunately. You can tell Nix to rely exclusively on the binary cache, or configure it to have 0 build threads locally and use remote builders.
For a server deployment, many popular choices don't have the server build anything, and in some cases it may not even have any users who are allowed to invoke `nix-*` or anything.
To be fair, the whole concept of manually installing individual packages on production servers (especially when they run NixOS!) is a complete anti-pattern, so that situation wouldn't even occur during normal operation.
Please do not assume I don't know what Nix and Guix do.
> you'd understand why this is a silly point to raise
It isn't.
> you'd understand why this is a silly point to raise
It isn't.
Debian packages can have pre/postinstall scripts that mutate the overall state of the system in arbitrary ways. This is both (a) necessary to make some thing work and (b) ruins the theoretical model.
> This is ... necessary to make some thing work
Things work in NixOS without the need of any of that. So, no it is not necessary.
It's a need created purely from the other directives of the Debian organization.
Things work in NixOS without the need of any of that. So, no it is not necessary.
It's a need created purely from the other directives of the Debian organization.
Not necessary, but certainly it's easier to implement and get stuff done. It starts to break down and show its cracks the larger and larger it gets, though.
The difference between almost all Linux or BSD packaging systems and nixpkgs is like the difference between dynamic and compile-time-type-checked programming languages.
The difference between almost all Linux or BSD packaging systems and nixpkgs is like the difference between dynamic and compile-time-type-checked programming languages.
You can have a much more conventional package manager without pre- or post-install hooks. One ex-Debian developer created such a package manager as a research project, presumably to prove a point to other Debian people with the same head-in-the-sand attitude about the state of Debian's package tooling.
It doesn't require the same kind of patching as Nix does (although sometimes patching does help avoid requiring those scripts).
Check it out: https://distr1.org/
It doesn't require the same kind of patching as Nix does (although sometimes patching does help avoid requiring those scripts).
Check it out: https://distr1.org/
Yes!
I'm in a team that works on a pet prog lang for distributed systems, and we did some research of using an existing package managing systems. We've settled on NPM for now, but god I wish there would be a better generic package manager out there.
I'm in a team that works on a pet prog lang for distributed systems, and we did some research of using an existing package managing systems. We've settled on NPM for now, but god I wish there would be a better generic package manager out there.
Not a generic package manager, but it's probably worth calling out asdf as the generic version manager[0] (maybe you're already aware of it, but it's a generic replacement for nvm, rvm, virtualenv, *vm, which supports any language based on plugins.)
Again, maybe you're already aware of it, but I think it's a nice example of genericising a concern common to many languages which sounds similar to what you're asking for (albeit unfortunately in a slightly different space).
[0] https://github.com/asdf-vm/asdf
Again, maybe you're already aware of it, but I think it's a nice example of genericising a concern common to many languages which sounds similar to what you're asking for (albeit unfortunately in a slightly different space).
[0] https://github.com/asdf-vm/asdf
At a minimum it's useful having the `.tool-versions` dotfile in a project directory which lists the versions of each language, db, and package manager version (our Rails project lists versions for ruby, bundler, postgres, node, yarn, redis). Even if all devs don't use asdf, it's a useful reference point.
https://asdf-vm.com/manage/configuration.html#tool-versions
Similar to an `.editorconfig` to globally define stuff like tab widths for IDEs.
https://asdf-vm.com/manage/configuration.html#tool-versions
Similar to an `.editorconfig` to globally define stuff like tab widths for IDEs.
Check out Denxi. Might be the closest thing to what you're looking for today, and is informed by the state of the art in package management without totally imposing strict discipline on all packages.
https://docs.racket-lang.org/denxi-guide/index.html
https://docs.racket-lang.org/denxi-guide/index.html
Did you compare nix to npm? Is there a good comparison out there?
If the community just focused on creating a great package manager instead of an Everything Monster, and “meeting developers where they are”, then Nix might do something great.
What's a good source, for someone who has never used nix, to read about its conceptual model and how it is different from contemporary dependency managers?
We wrote a blog post on how Nix helps us solve some of the problems that we encountered with other package managers like e.g. apt and pip.
https://www.channable.com/tech/nix-is-the-ultimate-devops-to...
(It doesn't go very much in depth on the conceptual model, but touches on the the main ideas)
https://www.channable.com/tech/nix-is-the-ultimate-devops-to...
(It doesn't go very much in depth on the conceptual model, but touches on the the main ideas)
There really isn't one that I know of. There's the original thesis[0], but it's from a different time and it focuses only on the package management (~ for a distribution) aspect.
Everything else is mostly written to teach people how to use Nix, and the more recent the thing is the more it will focus on surface-level features of the C++ implementation of Nix.
[0]: https://edolstra.github.io/pubs/phd-thesis.pdf
Everything else is mostly written to teach people how to use Nix, and the more recent the thing is the more it will focus on surface-level features of the C++ implementation of Nix.
[0]: https://edolstra.github.io/pubs/phd-thesis.pdf
[deleted]
Have you used Go or Rust package ecosystems?
My experience is that the older gen languages you mention had to invent package management, made lots of understandable mistakes and now are in a backwards compat hellscape.
Rust and Go built their packaging story with the benefit of lessons learned from those other systems, and in my experience the difference is night and day.
My experience is that the older gen languages you mention had to invent package management, made lots of understandable mistakes and now are in a backwards compat hellscape.
Rust and Go built their packaging story with the benefit of lessons learned from those other systems, and in my experience the difference is night and day.
Go package management is a mess of hacks. It doesn't even have a repository, instead relying on source control systems to do the actual work, with special hacks for each source control system to define the artifacts that can be downloaded (e.g. using Git tags with some magic format, or Perforce commit Metadata). It requires you to physically move all code to a new folder in your version control if you want to increase the major version number. It requires users of your code to update all of their import statements throughout their code whenever you move your hosting. It relies on DNS for package identity. It takes arcane magic to support multiple Go modules in the same repo.
I can go on, but it's a terrible hodge-podge of systems. It works nicely for simple cases (consuming libraries off Github), but it's awful when you go into details. And it's not even used by its creators - since Google has a monorepo and they actually use their internal universal build tool to just compile everything from source.
I can go on, but it's a terrible hodge-podge of systems. It works nicely for simple cases (consuming libraries off Github), but it's awful when you go into details. And it's not even used by its creators - since Google has a monorepo and they actually use their internal universal build tool to just compile everything from source.
> It relies on DNS for package identity.
The flip side of this is that it never has to worry about naming collisions or namespacing: Your public package name must be a URL you control.
Additionally, there is no requirement for a centralized package facility to be run. The Golang project is currently running pkg.go.dev, but that's only been in the last few years; and if they decided to get rid of it, it wouldn't significantly impact the development environment.
Finally, the current system makes "typo-squatting attacks" harder to do. Consider the popular golang package github.com/mattn/go-sqlite3. The only way to "typosquat" the package is to typosquat somewhere up the dependency tree; e.g., by creating github.com/matn/go-sqlite3 or something. You can't typosquat github.com/mattn/sqlite3, or github.com/mattn/go-sqlite, because you don't own those namespaces; whereas with non-DNS-based package systems, the package would be called `go-sqlite3`, and `sqlite3` or `go-sqlite` would be much easier to typosquat.
All those things I find really valuable; and honestly it's something I wish the Rust ecosystem had picked up.
> It requires users of your code to update all of their import statements throughout their code whenever you move your hosting.
This is a necessary cost of the item above. It can be somewhat annoying, but I believe this can be done with a one-line change to the go.mod. I'd much rather occasionally deal with this.
> It requires you to physically move all code to a new folder in your version control if you want to increase the major version number.
And the benefit of this is that legacy code will continue to compile into the future. I do tend to find this annoying, but it was explicit trade-off that was decided back when they were developing their packaging system.
Packaging is a hard problem, with lots of trade-offs; I think Go has done a pretty good job.
One way in which Go and Rust have it easier than Python or Node is that the former only have to deal with developers; the latter have to deal with both developers and users, whose requirements are often at odds with one another.
The flip side of this is that it never has to worry about naming collisions or namespacing: Your public package name must be a URL you control.
Additionally, there is no requirement for a centralized package facility to be run. The Golang project is currently running pkg.go.dev, but that's only been in the last few years; and if they decided to get rid of it, it wouldn't significantly impact the development environment.
Finally, the current system makes "typo-squatting attacks" harder to do. Consider the popular golang package github.com/mattn/go-sqlite3. The only way to "typosquat" the package is to typosquat somewhere up the dependency tree; e.g., by creating github.com/matn/go-sqlite3 or something. You can't typosquat github.com/mattn/sqlite3, or github.com/mattn/go-sqlite, because you don't own those namespaces; whereas with non-DNS-based package systems, the package would be called `go-sqlite3`, and `sqlite3` or `go-sqlite` would be much easier to typosquat.
All those things I find really valuable; and honestly it's something I wish the Rust ecosystem had picked up.
> It requires users of your code to update all of their import statements throughout their code whenever you move your hosting.
This is a necessary cost of the item above. It can be somewhat annoying, but I believe this can be done with a one-line change to the go.mod. I'd much rather occasionally deal with this.
> It requires you to physically move all code to a new folder in your version control if you want to increase the major version number.
And the benefit of this is that legacy code will continue to compile into the future. I do tend to find this annoying, but it was explicit trade-off that was decided back when they were developing their packaging system.
Packaging is a hard problem, with lots of trade-offs; I think Go has done a pretty good job.
One way in which Go and Rust have it easier than Python or Node is that the former only have to deal with developers; the latter have to deal with both developers and users, whose requirements are often at odds with one another.
People lose DNS names by accident all the time. It's also easy to typosquat many DNS domains, and even Github projects occasionally.
Non-DNS-based packages don't have to be named "go-sqlite". You can easily require some namespacing, and even use DNS as a base for that, but having an abstraction over it that recognizes the specific needs of package management is better. For example, Maven packages are called things like org.apache.commons, and registering a new package requires control of an equivalent DNS domain. However, if you later lose control of that domain, the new owners don't simply get to replace the packet in Maven just because they snipes your domain.
Go's choice to require full paths for import in each file is also not a direct implication of the previous item - they could have allowed the go.mod file to specify the path and a name, and then allow source files to import based on that name. Instead, this comes from Go tooling that existed before modules support, when the tooling would scour all files in your project to find dependencies.
Moving code to a v2 dir does not specifically help (or hinder) with backwards compatibility. Old code can always simply keep using the old versions of the package anyway. It is also a very unpopular decision, with very few packages actually adopting v2 versions precisely because of this requirement, even when making major breaking changes. Even the team maintaing the Go protobuf bindings decided not to use v2 when they rehauled their code (opting instead to create a new v1 at a new location with minor versions starting at 1.20...).
Sure, packaging is hard, but the Go team has chosen to go against the flow of essentially all other package managers, and instead of learning from their mistakes, they seem to have decided to make original mistakes all their own.
Non-DNS-based packages don't have to be named "go-sqlite". You can easily require some namespacing, and even use DNS as a base for that, but having an abstraction over it that recognizes the specific needs of package management is better. For example, Maven packages are called things like org.apache.commons, and registering a new package requires control of an equivalent DNS domain. However, if you later lose control of that domain, the new owners don't simply get to replace the packet in Maven just because they snipes your domain.
Go's choice to require full paths for import in each file is also not a direct implication of the previous item - they could have allowed the go.mod file to specify the path and a name, and then allow source files to import based on that name. Instead, this comes from Go tooling that existed before modules support, when the tooling would scour all files in your project to find dependencies.
Moving code to a v2 dir does not specifically help (or hinder) with backwards compatibility. Old code can always simply keep using the old versions of the package anyway. It is also a very unpopular decision, with very few packages actually adopting v2 versions precisely because of this requirement, even when making major breaking changes. Even the team maintaing the Go protobuf bindings decided not to use v2 when they rehauled their code (opting instead to create a new v1 at a new location with minor versions starting at 1.20...).
Sure, packaging is hard, but the Go team has chosen to go against the flow of essentially all other package managers, and instead of learning from their mistakes, they seem to have decided to make original mistakes all their own.
Java enforces domain name ownership as well; I can’t publish to maven central under io.netty.
This removes a whole slew of attack surfaces that JS and Python have, and leverages an existing “prove you are who you say you are” infra
This removes a whole slew of attack surfaces that JS and Python have, and leverages an existing “prove you are who you say you are” infra
Note that Maven Central only imposes this once when creating a new artifact - which is, I believe, much better than Go, which imposes this for every go mod download.
The advantage being, if you later lose access to the DNS domain that you used to publish an artifact to Maven Central, the new owner doesn't automatically get to compromise your artifact for all (new) users.
The advantage being, if you later lose access to the DNS domain that you used to publish an artifact to Maven Central, the new owner doesn't automatically get to compromise your artifact for all (new) users.
Some of your criticism is reasonable, and I’m no fan of Go’s module system as a standalone artifact, but much of your criticism is unfounded.
> It requires you to physically move all code to a new folder in your version control if you want to increase the major version number.
This is untrue.
> It requires users of your code to update all of their import statements throughout their code whenever you move your hosting.
This is only true if not using a vanity URL, but is sadly often the case.
> It takes arcane magic to support multiple Go modules in the same repo.
I don’t know what you’re calling arcane magic here, but we maintain repos at work with 6-7 go modules in without it being an issue whatsoever, and no “arcane magic” required, so I’m going to go ahead and say this is untrue too.
> It requires you to physically move all code to a new folder in your version control if you want to increase the major version number.
This is untrue.
> It requires users of your code to update all of their import statements throughout their code whenever you move your hosting.
This is only true if not using a vanity URL, but is sadly often the case.
> It takes arcane magic to support multiple Go modules in the same repo.
I don’t know what you’re calling arcane magic here, but we maintain repos at work with 6-7 go modules in without it being an issue whatsoever, and no “arcane magic” required, so I’m going to go ahead and say this is untrue too.
> This is untrue. [needing to move code to a new folder to increase major version]
I was indeed wrong, this is not necessary, though it is the original strong recommendation of how to do it from the original modules proposal.
Still, slightly changing the critique to say that changing major version is a big hassle, and it requires touching all of your project files, and pointing out how bizarre the official recommendation is compared to other packaging systems keeps my point intact, I believe.
> This is only true if not using a vanity URL, but is sadly often the case.
Yes, I am aware that you can buy a custom DNS and point it to your repo to release under a better name, but it is almost never done (I think the only dependency the project I work on has that does this are Google's Go protobuf bindings and several k8s.io projects).
> I don’t know what you’re calling arcane magic here, but we maintain repos at work with 6-7 go modules in without it being an issue whatsoever, and no “arcane magic” required, so I’m going to go ahead and say this is untrue too.
If you want to maintain multiple Go modules in the same git repo, and you want others to be able to download them, you need to tag commits for each module you are releasing. These tags must be formatted to match the dir path to the specific module's go.mod file (which also "conveniently" becomes part of the module name), except for the afore-mentionet vX directories if you chose to follow the Go team's recommendations for major versions. Then, for local development, you also need each go.mod to contain a REPLACE directive for each other module inside the same repo (or maybe directives in a go.work file, or who knows what else). Overall, this ends up creating lots and lots of useless tags, and is the definition of what I'd call arcane.
You'll not find the tag format in any "getting started" doc, and you'll not get any help from go mod itself if you get any of this wrong - just some "not found" style errors.
Edit: oh, and I should note: I have no idea how this is supposed to be done if your repo is not in Git, even though go modules are supposed to support other VCSs too.
I was indeed wrong, this is not necessary, though it is the original strong recommendation of how to do it from the original modules proposal.
Still, slightly changing the critique to say that changing major version is a big hassle, and it requires touching all of your project files, and pointing out how bizarre the official recommendation is compared to other packaging systems keeps my point intact, I believe.
> This is only true if not using a vanity URL, but is sadly often the case.
Yes, I am aware that you can buy a custom DNS and point it to your repo to release under a better name, but it is almost never done (I think the only dependency the project I work on has that does this are Google's Go protobuf bindings and several k8s.io projects).
> I don’t know what you’re calling arcane magic here, but we maintain repos at work with 6-7 go modules in without it being an issue whatsoever, and no “arcane magic” required, so I’m going to go ahead and say this is untrue too.
If you want to maintain multiple Go modules in the same git repo, and you want others to be able to download them, you need to tag commits for each module you are releasing. These tags must be formatted to match the dir path to the specific module's go.mod file (which also "conveniently" becomes part of the module name), except for the afore-mentionet vX directories if you chose to follow the Go team's recommendations for major versions. Then, for local development, you also need each go.mod to contain a REPLACE directive for each other module inside the same repo (or maybe directives in a go.work file, or who knows what else). Overall, this ends up creating lots and lots of useless tags, and is the definition of what I'd call arcane.
You'll not find the tag format in any "getting started" doc, and you'll not get any help from go mod itself if you get any of this wrong - just some "not found" style errors.
Edit: oh, and I should note: I have no idea how this is supposed to be done if your repo is not in Git, even though go modules are supposed to support other VCSs too.
Not the one you we're replying to, but I have an idea of the arcane magic he might be referring to. Back when I started learning Go I wanted to make a few example projects to test how well the language works, but I didn't want to push anything anywhere. When I tried to break my project into modules it ended up being that I needed to give them fake urls or something and then tell it to redirect the fake url to a folder instead. This[1] and this[2] gives a good example of the stuff I was running into at the time. I also realize that I should have probably been using sub-packages and not sub-modules, but coming from other languages I didn't realize they were suppose to be different. The thing that screwed with me most though was that there were half solutions for sub-modules to work so I kept using that terminology when searching google.
[1] https://stackoverflow.com/questions/10687627/relative-import... [2] https://stackoverflow.com/questions/57806081/how-to-access-n...
[1] https://stackoverflow.com/questions/10687627/relative-import... [2] https://stackoverflow.com/questions/57806081/how-to-access-n...
Now a days you can use go workspaces to do this so you don't have to modify your go.mod files anymore.
Thanks for the info, I'll look into that the next time I pick up go again.
> It requires you to physically move all code to a new folder in your version control if you want to increase the major version number
That's simply not true. That's only one way you can do it. Another way is to create a branch.
That's simply not true. That's only one way you can do it. Another way is to create a branch.
It's the recommended way, but you're right it's not the only one. Still, updating your major version number is much harder than in any other version control system I've seen (since however you do it, it requires you to update every single file in your repo to point to the new module). It also complicates the relationship between git tags, go.mod file location, and location within the Git repo significantly.
Interesting read on the official golang.org proxy DoSing git.sr.ht
https://sourcehut.org/blog/2023-01-09-gomodulemirror/
https://sourcehut.org/blog/2023-01-09-gomodulemirror/
Idk about Go, but Rust’s cargo seems nice, clean yet powerful.
That was my impression some time ago.
But last week I attempted to compile a couple of (not very big) tools from cargo. And it ended up downloading hundreds of dependencies and gigabytes of packages.
Looks like node_modules.jpg all over again :(
That was my impression some time ago.
But last week I attempted to compile a couple of (not very big) tools from cargo. And it ended up downloading hundreds of dependencies and gigabytes of packages.
Looks like node_modules.jpg all over again :(
As someone who contributed somewhat extensively to the node_modules problem early on, Cargo is definitely better than the JS ecosystem in this regard.
Further, another major difference is that you don't need those dependencies after you've built. You can blow them away. Doing that with node is not as straightforward, and in many cases, not possible.
Further, another major difference is that you don't need those dependencies after you've built. You can blow them away. Doing that with node is not as straightforward, and in many cases, not possible.
> Idk about Go
I wrote a post highlighting Go's mod system: https://verdverm.com/go-mods/
imo, it is the best designed dependency system I know of. One of the nice things is that Go uses a shared module cache so there is only one copy on your computer when multiple projects use the same dependency@version
I wrote a post highlighting Go's mod system: https://verdverm.com/go-mods/
imo, it is the best designed dependency system I know of. One of the nice things is that Go uses a shared module cache so there is only one copy on your computer when multiple projects use the same dependency@version
I really like a lot of the distribution- and safety-related decisions Go modules made. Domain names and the proxy+sumdb are wonderfully clear, flexible, and scalable, and I think we'll see copies of it in many future languages.
The rest of the stuff around modules, like crippled constraints, zero control over contents (which they have changed!), and completely non-existent "x is available, upgrade" or release tooling: constant, unnecessary pain, and it'll be inflicting serious damage on the ecosystem for many years to come.
The rest of the stuff around modules, like crippled constraints, zero control over contents (which they have changed!), and completely non-existent "x is available, upgrade" or release tooling: constant, unnecessary pain, and it'll be inflicting serious damage on the ecosystem for many years to come.
> crippled constraints
Do you mean ranges on dep versions? The way it is currently, the version you set is the minimum, and the algo finds the highest minimum set across all deps and uses that.
If ranges were introduced, you'd end up with an NP hard problem and need a SAT solver for your deps again
> Release tooling
What are you looking for here? Libraries only need to push a git tag, binaries do require a bit of work, but Goreleaser fills that pretty nicely. It would seem hard to standard where binaries would be pushed
> completely non-existent "x is available, upgrade"
https://go.dev/doc/modules/managing-dependencies#discovering...
"go list -m -u all"
Do you mean ranges on dep versions? The way it is currently, the version you set is the minimum, and the algo finds the highest minimum set across all deps and uses that.
If ranges were introduced, you'd end up with an NP hard problem and need a SAT solver for your deps again
> Release tooling
What are you looking for here? Libraries only need to push a git tag, binaries do require a bit of work, but Goreleaser fills that pretty nicely. It would seem hard to standard where binaries would be pushed
> completely non-existent "x is available, upgrade"
https://go.dev/doc/modules/managing-dependencies#discovering...
"go list -m -u all"
I'm continually confused why SAT solving is seen as a bad thing. It automatically solves common real-world problems in less than a second even in absurdly extreme cases, and a couple milliseconds normally - why would you avoid it?
Did I say SAT was bad?
Go's algorithm is much simpler and does not need a lockfile while still giving deterministic results. Ranged deps without a lockfile cannot. There is benefit to two people running the same command and getting the same dependency versions. Most projects do not start with a lockfile, so it is quite easy to have different versions when running getting started commands.
Another example, if I install two ranged dependencies in both orders, will I get the same final deps@version list?
Why choose SAT over simpler deterministic algos?
Go's algorithm is much simpler and does not need a lockfile while still giving deterministic results. Ranged deps without a lockfile cannot. There is benefit to two people running the same command and getting the same dependency versions. Most projects do not start with a lockfile, so it is quite easy to have different versions when running getting started commands.
Another example, if I install two ranged dependencies in both orders, will I get the same final deps@version list?
Why choose SAT over simpler deterministic algos?
I think, this is not a problem with package manager per se. But with extremities of coding culture. On one side of the spectrum is “NIH” and reinvent everything yourself. On the other side is: let’s pull left-pad from package, because packages are good, we need MORE PACKAGES.
The best solution always lies somewhere in the middle. But finding this “middle” (and adhering to this approach) is the hard part.
The best solution always lies somewhere in the middle. But finding this “middle” (and adhering to this approach) is the hard part.
I think I'm on the NIH side of things. Not because I like inventing things (well, partially), but because of the security problem [0].
I don't see it being talked about here, either. Our current system of pulling in packages made by random people on the internet is going to burn us. We assume that everyone who creates a package is an honest, reliable, developer who will not inject malicious code into their package. This assumption is similar to the assumptions we made around SMTP, HTTP, DNS, and every other internet protocol. Turns out we were wrong and surprise! you can't trust people on the internet.
I'm not sure if we can solve this with package managers. But package managers are part of the culture that has created this problem, and are probably a reasonable starting point to try and address it.
[0]:https://david-gilbertson.medium.com/im-harvesting-credit-car...
I don't see it being talked about here, either. Our current system of pulling in packages made by random people on the internet is going to burn us. We assume that everyone who creates a package is an honest, reliable, developer who will not inject malicious code into their package. This assumption is similar to the assumptions we made around SMTP, HTTP, DNS, and every other internet protocol. Turns out we were wrong and surprise! you can't trust people on the internet.
I'm not sure if we can solve this with package managers. But package managers are part of the culture that has created this problem, and are probably a reasonable starting point to try and address it.
[0]:https://david-gilbertson.medium.com/im-harvesting-credit-car...
Being in the Rust full time for last 2 or 3 years: it is quite a pain to setup a release process for big Rust workspace.
Version incrementing, packaging wasms, dancing around code generation – all doable, but not standardized.
There's a release-please to automate all that, but it's not an easy task to set it up in all of your repos.
Besides, if in addition to Rust projects, you have projects in other languages like JavaScript, then you have to do it twice and struggle with understanding all of the package management systems provided by all languages you have.
A single swiss-army-knife package manager would be amazing.
Version incrementing, packaging wasms, dancing around code generation – all doable, but not standardized.
There's a release-please to automate all that, but it's not an easy task to set it up in all of your repos.
Besides, if in addition to Rust projects, you have projects in other languages like JavaScript, then you have to do it twice and struggle with understanding all of the package management systems provided by all languages you have.
A single swiss-army-knife package manager would be amazing.
Doesn't look like it, see module drama in Go, while Rust is having an npm like ecosystem of tiny crates.
Plus none of them handle binary library distribution as some of the packing models that came before them.
Plus none of them handle binary library distribution as some of the packing models that came before them.
Can you give me a one-liner about module drama in Go?
First they weren't supported, so the community created various ways of dealing it including a kind of Google's blessed implementation, then Google decided to create their own official way, then there was the transition time which I guess not everyone has done, having SCM urls as imports is just bad, and how GOPROXY DoS some SCM repos outside of Github.
[0] https://go.dev/blog/appengine-gopath gopath / workspaces original
[1] https://go.dev/blog/migrating-to-go-modules current system
[2] https://news.ycombinator.com/item?id=34310674 goproxy agressively polling sourcehut
It's worth recognizing that go isn't alone in dns-based namespacing : java's maven/gradle use the same strategy: https://repo1.maven.org/maven2/gov/nih/imagej/imagej/1.47/
[1] https://go.dev/blog/migrating-to-go-modules current system
[2] https://news.ycombinator.com/item?id=34310674 goproxy agressively polling sourcehut
It's worth recognizing that go isn't alone in dns-based namespacing : java's maven/gradle use the same strategy: https://repo1.maven.org/maven2/gov/nih/imagej/imagej/1.47/
Thank you. I didn’t quite see all that happening but heard echoes of it from time to time, then really jumped in after they adopted modules.
That sounds like a chatgpt prompt lol
Thanks pal. I’m not feeling self-conscious at all now
Npm has lots of not understandable mistakes: it should already be as good with ES6+TS as other systems. Import should just work everywhere.
We should have left Commonjs a long time ago, while keeping backwards compatibility.
At the same time what I see with the node+npm system is that everything is just ,,it just doesn't work by default''.
Having 10 other package managers doesn't work either, they are faster, but don't solve this problem.
We should have left Commonjs a long time ago, while keeping backwards compatibility.
At the same time what I see with the node+npm system is that everything is just ,,it just doesn't work by default''.
Having 10 other package managers doesn't work either, they are faster, but don't solve this problem.
Just use .mjs extension for your code, add extensions to your imports and everything does just work.
Except in typescript, where you have to import a file with the .js extension, despite the actual file being .ts: https://github.com/microsoft/TypeScript/issues/42151, https://github.com/microsoft/TypeScript/issues/49083, https://github.com/microsoft/TypeScript/issues/16577
This one issue (IMO) is preventing parts of the ecosystem from switching to esm.
This one issue (IMO) is preventing parts of the ecosystem from switching to esm.
How does the js extension stop a project from targeting ESM?
But it is a pain for a distro to upgrade a vulnerable crate dependency
Go and Rust don't have packages. They have tooling to pull library code from git and build it in-place to be linked into a local project. That's not (really) packaging.
Cargo does not “pull library code from git” unless you expressly ask for it to.
And given that packages have to depend on other packages, and cannot depend on a git repository, that feature is mostly useful for testing bug fixes, private repos for leaf packages, stuff like that.
And given that packages have to depend on other packages, and cannot depend on a git repository, that feature is mostly useful for testing bug fixes, private repos for leaf packages, stuff like that.
I thought Rust doesn't have binary libraries? If you declare a Rust dependency, cargo pulls its source and builds it together with your own code? I assume that's what the GP meant, anyway.
Source code is stored in an S3 bucket owned by crates.io, and not from any form of source control, including git.
That the entire code of all dependencies is hosted in one place is an important differentiator of crates.io vs what Go does, which is why this is relevant in this context. Crates.io is centralized, with all of the advantages and disadvantages that that brings, and Go is decentralized, with all of the advantages and disadvantages that brings.
That the entire code of all dependencies is hosted in one place is an important differentiator of crates.io vs what Go does, which is why this is relevant in this context. Crates.io is centralized, with all of the advantages and disadvantages that that brings, and Go is decentralized, with all of the advantages and disadvantages that brings.
Julia and Rust seem to have package systems that are fine and manageable. I think these are really just major problems in Javascript, Python, and C/C++ (exactly the languages you mention) because the kind of widespread OSS code sharing just didn't exist to the extent it does today back when those languages were designed. People shared code through email, but didn't expect one button to pull in 200 Github repositories, and thus weren't built with the expectations required to make that be stable.
Back when those languages were designed, you'd manually download the few modules you need, if you downloaded any packages at all. In C you'd normally build your own world, since it came before the www times, and C++ kind of inherited that. But languages which came out later decided that we now live in a world where most of the code that is executed is packages, most likely packages which live on Github. So Julia and Rust build this into the language. Julia in particular with the Project.toml and Manifest.jl for fully reproducing environments, its package manager simply uses git and lets you grab the full repository with `]dev packagename`, its package registry system lets you extend with private package worlds.
I think the issue is that dependencies are central, so you can never remove old package systems because if that's where the old (and rarely updated) dependencies live, then you need to keep it around. But for dependencies to work well, you need all dependencies to be resolved using the same package system. So package systems don't tend to move very fast in any language, whatever you had early has too much momentum.
Back when those languages were designed, you'd manually download the few modules you need, if you downloaded any packages at all. In C you'd normally build your own world, since it came before the www times, and C++ kind of inherited that. But languages which came out later decided that we now live in a world where most of the code that is executed is packages, most likely packages which live on Github. So Julia and Rust build this into the language. Julia in particular with the Project.toml and Manifest.jl for fully reproducing environments, its package manager simply uses git and lets you grab the full repository with `]dev packagename`, its package registry system lets you extend with private package worlds.
I think the issue is that dependencies are central, so you can never remove old package systems because if that's where the old (and rarely updated) dependencies live, then you need to keep it around. But for dependencies to work well, you need all dependencies to be resolved using the same package system. So package systems don't tend to move very fast in any language, whatever you had early has too much momentum.
Tying package management to GitHub seems convenient in the short term, but will be the baggage of the next generation.
I cringe hard when I see projects depending on git repos, without pinning a version or commit.
I cringe hard when I see projects depending on git repos, without pinning a version or commit.
It's not actually tied to GitHub. If GitHub died tomorrow, they would easily be able to move on and host the packages somewhere else. There's no way to do this without hosting. Also, there is a way to pin a version or commit. Julia for example always stores the exact commit information for all packages in the "Manifest" file. There are also straightforward ways to demand certain versions and package maintainers have the opportunity to specify compatibility and requirements precisely.
Strictly speaking, hosting like github isn't the only way, an alternative being some peer to peer network like torrents
There will be a mess of dependencies of old projects that depend of projects that have since moved off GitHub, there will be name clashes between GitHub projects and BitBucket projects etc. These ecosystems are very much dependent on GitHub, and if/when anything permanent changes, it will be a mess to move off of it.
> there will be name clashes between GitHub projects and BitBucket projects etc.
The package manager requires a canonical URL for the git repo. The location of dependencies doesn't matter, for example Github repos with Bitbucket dependencies work fine. The FastLapack packages lived on BitBucket at one point, and then migrated to Github, and that happened without a hitch to upstream packages like LinearSolve.jl or DifferentialEquations.jl.
Also, it takes a one line change to General change a package's from Github to BitBucket. It didn't take me more than 2 minutes when I tried. Did you not have a similar experience? Can you show the PR where you gave it a try? Or is your response just speculation?
The package server mirrors the repos, so even if these services went down it would not break the dependency chain (developers wouldn't be able to easily update the packages, but the installations and all would still work). So if tomorrow Github decided to make all of the code private for some odd reason, Julia's package server would continue to serve the current versions.
The package manager requires a canonical URL for the git repo. The location of dependencies doesn't matter, for example Github repos with Bitbucket dependencies work fine. The FastLapack packages lived on BitBucket at one point, and then migrated to Github, and that happened without a hitch to upstream packages like LinearSolve.jl or DifferentialEquations.jl.
Also, it takes a one line change to General change a package's from Github to BitBucket. It didn't take me more than 2 minutes when I tried. Did you not have a similar experience? Can you show the PR where you gave it a try? Or is your response just speculation?
The package server mirrors the repos, so even if these services went down it would not break the dependency chain (developers wouldn't be able to easily update the packages, but the installations and all would still work). So if tomorrow Github decided to make all of the code private for some odd reason, Julia's package server would continue to serve the current versions.
I was mostly speculating, and thinking more of how Go does Github dependencies. I am happy to hear that Cargo doesn't have the same sort of issues.
I really should have looked into it more before commenting, especially since I knew Cargo is actually a well-designed and considered package system.
I really should have looked into it more before commenting, especially since I knew Cargo is actually a well-designed and considered package system.
Just to make sure there are no misunderstandings. ChrisRackauckas was talking about Julia's package manager, (which is called Pkg, if you need to name it, although I would just call it "the package manager"). Cargo is the Rust package manager.
[deleted]
If the git repo url is content addressable, using p2p network like ipfs / zeronet / magnet link, then should be easier to move the storage provider to any host as long as it's connected to the p2p network.
Julia and Rust all work with using versions tied to git tags. Package management isn't tied to Github but git: packages can live on Gitlab or BitBucket, though they generally don't and that's the choice of package developers. Because of that there are tie-ins to make Github really nice to use, but for example with Julia the only piece that is truly Github based is the fact that the General registry lives in a Github repo (https://github.com/JuliaRegistries/General), but could easily migrate to another git platform if it needed to.
What I really want to know, is why package development is such a Rube Goldberg machine. Not for programming-language packages, per se, but rather for OS packages of simple programming-language packages.
Have you tried to package a random Python/Ruby/etc. CLI program, for Debian? Or how about for Homebrew? Each one involves a cacophony of PL scripts calling shell-scripts calling PL scripts, and internal/undocumented packager subcommands calling other internal/undocumented packager subcommands. It takes ~forever to do a checked build of a package in these systems, and 99% of it is just because of how spread out the implementation of such checks is over 100 different components implemented at different times by different people. It could all be vastly simplified by reimplementing all the checks and transforms in a single pass that gradually builds up an in-memory state, making assertions about it and transforming it as it goes, and then emitting it if everything works out. You know — like a compiler.
Have you tried to package a random Python/Ruby/etc. CLI program, for Debian? Or how about for Homebrew? Each one involves a cacophony of PL scripts calling shell-scripts calling PL scripts, and internal/undocumented packager subcommands calling other internal/undocumented packager subcommands. It takes ~forever to do a checked build of a package in these systems, and 99% of it is just because of how spread out the implementation of such checks is over 100 different components implemented at different times by different people. It could all be vastly simplified by reimplementing all the checks and transforms in a single pass that gradually builds up an in-memory state, making assertions about it and transforming it as it goes, and then emitting it if everything works out. You know — like a compiler.
That's a Debian specific quirk. There are simpler, more cohesive package management systems, like eg. pacman, where package maintainers's tools mastery is not an arcane art in itself.
It annoys me that every gee-whizz new language needs to have it's own package-management system.
There's no reason why a package-management system needs to be language-specific; dependencies are often cross-language. Hell, even some blocks of code contain more than one language.
The package-management system is responsible for deploying packages. The way a package is deployed should depend on the operating environment, not on the language. These language-specific packaging arrangements typically deploy into some private part of the file-system, organized in its own idiosyncratic way.
Using git as a repository is just nuts. Git is privately-owned, and stuffed with all kinds of unaudited junk. You can't audit everything you install by hand; so these systems force you to install unaudited code, or simply not install.
I've been using Debian derivatives for years. I appreciate having an audited repository, and an installation system that deploys code to more-or-less predictable filesystem locations.
There's no reason why a package-management system needs to be language-specific; dependencies are often cross-language. Hell, even some blocks of code contain more than one language.
The package-management system is responsible for deploying packages. The way a package is deployed should depend on the operating environment, not on the language. These language-specific packaging arrangements typically deploy into some private part of the file-system, organized in its own idiosyncratic way.
Using git as a repository is just nuts. Git is privately-owned, and stuffed with all kinds of unaudited junk. You can't audit everything you install by hand; so these systems force you to install unaudited code, or simply not install.
I've been using Debian derivatives for years. I appreciate having an audited repository, and an installation system that deploys code to more-or-less predictable filesystem locations.
> Using git as a repository is just nuts. Git is privately-owned, and stuffed with all kinds of unaudited junk.
Do you mean github? Git is open source and one of the few pieces of software that works in a truly distributed fashion.
Do you mean github? Git is open source and one of the few pieces of software that works in a truly distributed fashion.
You're right; I meant github. Sorry.
> Git is privately-owned, and stuffed with all kinds of unaudited junk.
Also: what do I care that people store unaudited and insecure stuff in there?
Also: what do I care that people store unaudited and insecure stuff in there?
Because cross platform packages isn't a thing, and not everyone wants to create a package for every platform out there.
The reasons language-specific package managers are popular:
1) "npm install" or "go get" or what have you, works on every platform (barring bugs), while "apt install" only works on some.
2) Most platform package managers aren't good at handling multiple versions of dependencies (which, neither are many language package managers, but they're easier to sandbox away with supplemental tools than system package managers are)
3) Most platform package managers lag way behind language-specific package managers, and may also lack tons and tons of packages that are available on those.
> Git is privately-owned
Git... hub, you mean?
1) "npm install" or "go get" or what have you, works on every platform (barring bugs), while "apt install" only works on some.
2) Most platform package managers aren't good at handling multiple versions of dependencies (which, neither are many language package managers, but they're easier to sandbox away with supplemental tools than system package managers are)
3) Most platform package managers lag way behind language-specific package managers, and may also lack tons and tons of packages that are available on those.
> Git is privately-owned
Git... hub, you mean?
Probably because your Debian doesn't work on other Linux distros like CentOS, Arch Linux ..., let alone MacOS and Windows. The holy grail would be a package manager that is both cross language and cross platform at the same time, but it seems too hard to be implemented in general.
Anaconda is probably the closest as it also package many non python packages. Nix is also similar but it doesn't support Windows at all (without WSL)
Anaconda is probably the closest as it also package many non python packages. Nix is also similar but it doesn't support Windows at all (without WSL)
> Git is privately-owned, and stuffed with all kinds of unaudited junk. [...] I've been using Debian derivatives for years
Git is owned by the same owner as Linux. If you've been using Debian derivatives for years it seems you must have some trust to give that private entity? Unless the derivative you speak of is Debian GNU/k*BSD?
Furthermore, if you give trust to Debian derivative projects, why not trust their Git builds? If you trust everything else in their distribution Git is a curious omission. Do you have a personal beef with Torvalds or something?
Git is owned by the same owner as Linux. If you've been using Debian derivatives for years it seems you must have some trust to give that private entity? Unless the derivative you speak of is Debian GNU/k*BSD?
Furthermore, if you give trust to Debian derivative projects, why not trust their Git builds? If you trust everything else in their distribution Git is a curious omission. Do you have a personal beef with Torvalds or something?
> There's no reason why a package-management system needs to be language-specific; dependencies are often cross-language. Hell, even some blocks of code contain more than one language.
This sounds a lot like a case of https://xkcd.com/927/ . Languages have different ways of importing and installing dependencies, trying to create a package manager over all of those is just going to end up making things even more complex, especially if you target all platforms at once.
> Using git as a repository is just nuts. Git is privately-owned, and stuffed with all kinds of unaudited junk.
Git is fully open source. Are you confusing Git and GitHub?
This sounds a lot like a case of https://xkcd.com/927/ . Languages have different ways of importing and installing dependencies, trying to create a package manager over all of those is just going to end up making things even more complex, especially if you target all platforms at once.
> Using git as a repository is just nuts. Git is privately-owned, and stuffed with all kinds of unaudited junk.
Git is fully open source. Are you confusing Git and GitHub?
> Languages have different ways of importing and installing dependencies
They're not actually different. They call things differently, and they have different methods of passing the required lookup paths/artifacts/sources to their compilers/interpreters/linkers, but in the end all of them are conceptually the same thing.
They're not actually different. They call things differently, and they have different methods of passing the required lookup paths/artifacts/sources to their compilers/interpreters/linkers, but in the end all of them are conceptually the same thing.
>but in the end all of them are conceptually the same thing
The concept of a programming language isn't what people use, but the concrete reality of specific implementations on specific hardware.
You could also say that a coffee cup and a donut aren't actually different because they're topologically the same... but the facts on the ground do actually matter.
The concept of a programming language isn't what people use, but the concrete reality of specific implementations on specific hardware.
You could also say that a coffee cup and a donut aren't actually different because they're topologically the same... but the facts on the ground do actually matter.
It absolutely does matter in this case. We have written implementations of this concept for very different languages (Go, Common Lisp, Erlang, C) and the conceptual equality means that nothing is different at the core, only at the edges (where you integrate with each snowflake's particular flavour of "what are the names of the command-line arguments to the compiler" and so on).
I'd say we have a fairly good existing theory that explains modern package management: Conway's Law.
We self-organize into communities of practice and select the package management strategy that works best. Reaching across communities to develop a grand centralized strategy that fits everyone's needs would be __possible__, but involves significant communication and coordination overhead. So instead we fracture, and the tooling ecosystem reflects ad-hoc organizational units within the community.
Ecosystems like Rust cargo that have batteries included from the start have an advantage, virtually all Rust developers have a single obvious path to package management because of this emergent social organization.
Ecosystems like Python's seem like the wild west, there is deep fracturing (particularly between data science and software engineering) and no consensus on the requirements or even the problems. So Python fractures further, ironically in a search for something that can eventually unify the community. And python users feel the strain of this additional overhead every day, needing to side with a team just to get work done.
I'd argue both of these cases are driven by consequences easily predictable from Conway's Law.
We self-organize into communities of practice and select the package management strategy that works best. Reaching across communities to develop a grand centralized strategy that fits everyone's needs would be __possible__, but involves significant communication and coordination overhead. So instead we fracture, and the tooling ecosystem reflects ad-hoc organizational units within the community.
Ecosystems like Rust cargo that have batteries included from the start have an advantage, virtually all Rust developers have a single obvious path to package management because of this emergent social organization.
Ecosystems like Python's seem like the wild west, there is deep fracturing (particularly between data science and software engineering) and no consensus on the requirements or even the problems. So Python fractures further, ironically in a search for something that can eventually unify the community. And python users feel the strain of this additional overhead every day, needing to side with a team just to get work done.
I'd argue both of these cases are driven by consequences easily predictable from Conway's Law.
Is there a name for a phenomena/theory that states "the more people you add to a problem while asking them to solve it in an opinionated way, the more likely you are to get conflict/fragmentation/less people agreeing overall as consensus gets diluted and number of possibilities/opinions increases?"
Maybe it should be called Lydgate’s Law: “You can please some of the people all of the time, you can please all of the people some of the time, but you can’t please all of the people all of the time.”
The Racket module system, designed by Matthew Flatt, is great.
But to fully appreciate it, it helps to understand syntax transformation in Racket. Once the rigorous phase system forces non-kludgy static rules about when things are evaluated, your syntax transformers and the code on which they depend could cause a mess of shuffling code among multiple files to solve dependency problems... until you use submodules with the small set of visibility rules, and then suddenly your whole tricky package once again fits in a single file cleanly.
I leveraged this for some of my embedded doc and test experiments, without modifying the Racket core. (I really, really like single-source-file modules that embed doc, test, and package metadata all in the same file, in logical places.)
But to fully appreciate it, it helps to understand syntax transformation in Racket. Once the rigorous phase system forces non-kludgy static rules about when things are evaluated, your syntax transformers and the code on which they depend could cause a mess of shuffling code among multiple files to solve dependency problems... until you use submodules with the small set of visibility rules, and then suddenly your whole tricky package once again fits in a single file cleanly.
I leveraged this for some of my embedded doc and test experiments, without modifying the Racket core. (I really, really like single-source-file modules that embed doc, test, and package metadata all in the same file, in logical places.)
You mention Python and Node, which are programming language that unusually require end users to have the text of your program and all its dependencies on their own machine, and the languages store some parts of your program in /usr/lib and other parts of your program in your source directory. (npm does a little better here and at least puts the dependencies in your project directory.) Those constraints make development and packaging hard no matter what.
Python and Node both need a way to compile the code down to a single statically-linked binary like more modern languages (Go, Rust), solving the distribution problem once and for all.
There are module systems that aren't insane, like Go's module system. It uses semantic versioning to mediate version conflicts. Programs can import multiple major versions of the same module. The module requirements files ensure that every checkout of the code gets the exact same bytes of the code's dependencies. The compiler is aware of modules and can fetch them, so on a fresh workstation, "go test ./..." or "go install ./cmd/cool-thing" in a module-aware project works without running any other command first. It is actually so pleasant to use that I always think twice "do I want to deal with modules" before using a language like Javascript or Python, and usually decide "no".
npm and pip are the DARK AGES. That's why you're struggling. The community has been unable to fix these fundamental flaws for decades, despite trying in 100 incompatible ways. I personally have given up on the languages as a result. The standard library is never enough. You HAVE to solve the module problem on day 1.
Python and Node both need a way to compile the code down to a single statically-linked binary like more modern languages (Go, Rust), solving the distribution problem once and for all.
There are module systems that aren't insane, like Go's module system. It uses semantic versioning to mediate version conflicts. Programs can import multiple major versions of the same module. The module requirements files ensure that every checkout of the code gets the exact same bytes of the code's dependencies. The compiler is aware of modules and can fetch them, so on a fresh workstation, "go test ./..." or "go install ./cmd/cool-thing" in a module-aware project works without running any other command first. It is actually so pleasant to use that I always think twice "do I want to deal with modules" before using a language like Javascript or Python, and usually decide "no".
npm and pip are the DARK AGES. That's why you're struggling. The community has been unable to fix these fundamental flaws for decades, despite trying in 100 incompatible ways. I personally have given up on the languages as a result. The standard library is never enough. You HAVE to solve the module problem on day 1.
if npm and pip are DARK AGES what does that mean perl's CPAN is?
I haven't used Perl for 15 years, but it was pretty miserable back then. Obviously I had my workflow and didn't run into too many problems, but I was certainly nervous about how I could share my work with other people. (Putting it into prod wasn't that bad. I don't know why, but it always went OK. CPANPLUS helped at the time.) I used to teach Perl trainings as a side job, and people would literally be in tears over @INC. It was bad.
I don't use Python much these days, but it's not as bad as Perl 15 years ago. I see blog posts like "how to set up the perfect Python dev environment with Docker" and it makes me very sad, but at least teams are getting their work done. The edge cases of Python packaging, though, are really really bad. For example, the C compiler that Python was compiled with (and C library; musl vs. glibc) affects installability of modules through pip. This really forces your Linux distribution to be the arbiter of what modules you can use, which is always too out of date for development. Also, the exact requirements (what are the sha256s and URLs of the code files that this code depends on) depends on architecture, compiler, etc. As far as I can tell, you have to run code to find the whole dependency tree. That is the fatal flaw I see with Python's packaging system.
I spent a lot of time trying to use Bazel to unify all the work my company does across languages. Python was the killer here. We do depend on a lot of C extensions, and I have a C toolchain built with Bazel (so that arm64 mac users can cross-compile to produce Linux releases; maybe a dumb requirement, but it would be too unpopular to discriminate against certain developer devices), but getting Python built with that toolchain, and using that Python to evaluate requirements.txt didn't work. (There are bugs open; they all end with "Python has to fix this".) As a result, you don't get compatible versions of, say, numpy with this setup. Dealbreaker. Partially Bazel's fault, partially Pyton's fault.
(It pained me to use Bazel for Go, but it did all work. While the workflow wasn't as nice as what Go has built in, easy things were hard and hard things were possible. I had working binaries within a few hours, buildable on any supported workstation type, with object and test caching in the cloud.)
I don't use Python much these days, but it's not as bad as Perl 15 years ago. I see blog posts like "how to set up the perfect Python dev environment with Docker" and it makes me very sad, but at least teams are getting their work done. The edge cases of Python packaging, though, are really really bad. For example, the C compiler that Python was compiled with (and C library; musl vs. glibc) affects installability of modules through pip. This really forces your Linux distribution to be the arbiter of what modules you can use, which is always too out of date for development. Also, the exact requirements (what are the sha256s and URLs of the code files that this code depends on) depends on architecture, compiler, etc. As far as I can tell, you have to run code to find the whole dependency tree. That is the fatal flaw I see with Python's packaging system.
I spent a lot of time trying to use Bazel to unify all the work my company does across languages. Python was the killer here. We do depend on a lot of C extensions, and I have a C toolchain built with Bazel (so that arm64 mac users can cross-compile to produce Linux releases; maybe a dumb requirement, but it would be too unpopular to discriminate against certain developer devices), but getting Python built with that toolchain, and using that Python to evaluate requirements.txt didn't work. (There are bugs open; they all end with "Python has to fix this".) As a result, you don't get compatible versions of, say, numpy with this setup. Dealbreaker. Partially Bazel's fault, partially Pyton's fault.
(It pained me to use Bazel for Go, but it did all work. While the workflow wasn't as nice as what Go has built in, easy things were hard and hard things were possible. I had working binaries within a few hours, buildable on any supported workstation type, with object and test caching in the cloud.)
I think package management is something so important you either have to get it right on the first released version of the language, or never release an official one at all.
So if you look at C, there was basically no internet or even HTTP, so if they wanted to fetch + store dependencies from centralized repositories, it would have been difficult and then need to change a lot over the years.
Python is the worst to me, I love the language, but I hate the tooling, managing installations, packages, modules, etc.
Sometimes languages are relased with just a spec and don't want to force any choice of tool or way of doing things on you, so you just manage that yourself or create third party tools, and that's where it gets wild in every direction, but it also creates room for new ideas, innovation, which later are used in "official" modern package managers built in the language tools.
And yeah, nowadays I use Rust even for scripting stuff that would be easier to do in Python, just because I don't want to create a thousandth virtualenv, find a lib that does what I want but it's for Python<3.6 ou Python2 etc., so in the end it's easier in Rust even though some new libs require nightly builds of the toolchain.
Sometimes languages are relased with just a spec and don't want to force any choice of tool or way of doing things on you, so you just manage that yourself or create third party tools, and that's where it gets wild in every direction, but it also creates room for new ideas, innovation, which later are used in "official" modern package managers built in the language tools.
And yeah, nowadays I use Rust even for scripting stuff that would be easier to do in Python, just because I don't want to create a thousandth virtualenv, find a lib that does what I want but it's for Python<3.6 ou Python2 etc., so in the end it's easier in Rust even though some new libs require nightly builds of the toolchain.
I’ve also found it not that much harder to do scripting kinds of things in Rust vs Ruby or Python. It’s a bit of extra effort of course but having a nice self contained binary at the end is pretty handy.
We've been using Go to build CLI tools rather than writing scripts for certain tasks recently, and I've found that Golang is very well suited for that.
we use the urfave/cli library, which works great, and gets you a ton of things for almost free (passing and documenting flags and arguments) that you just don't get easily in bash
we use the urfave/cli library, which works great, and gets you a ton of things for almost free (passing and documenting flags and arguments) that you just don't get easily in bash
I was going to disagree that Python's is the worst, but then I thought about it and.. yeah, it's up there. I think there are definitely some other ecosystems with worse package management systems but they're certainly not as mainstream as Python.
Relevant XKCD: https://xkcd.com/1987/
As I've said, I still love it and use it a lot, easy to write, easy to explain and share, easy to run without dependencies and if your code runs the same on different versions of Python, but hard to choose which type of virtualenv you want to use, then pick the right versions of dependencies that work together (if you have big dependencies like in data science for example, and some of them are less maintained than others, which happens because the Python ecosystem is rich but people start projects and move on, happens everywhere but because it's easy to start a project in Python, it's also easy to leave it unmaintained).
I love explaining Python code to non coders!
As I've said, I still love it and use it a lot, easy to write, easy to explain and share, easy to run without dependencies and if your code runs the same on different versions of Python, but hard to choose which type of virtualenv you want to use, then pick the right versions of dependencies that work together (if you have big dependencies like in data science for example, and some of them are less maintained than others, which happens because the Python ecosystem is rich but people start projects and move on, happens everywhere but because it's easy to start a project in Python, it's also easy to leave it unmaintained).
I love explaining Python code to non coders!
I find cargo (rust package manager) relatively pleasant to use. It's not perfect, but it also is not a constant source of confusion like sbt (scala package manger) was back when I did scala.
In the long term, I think the whole notion of a "package" is obsolete. In unison https://www.unison-lang.org/ , code is just a content-addressed merkle tree. So dependencies are both more precise and more fine grained than packages.
E.g. if there is a new version of a library, but the part of the library that you actually use is unchanged, unison will detect this and not trigger a full rebuild.
The whole notion of a package becomes less important.
In the long term, I think the whole notion of a "package" is obsolete. In unison https://www.unison-lang.org/ , code is just a content-addressed merkle tree. So dependencies are both more precise and more fine grained than packages.
E.g. if there is a new version of a library, but the part of the library that you actually use is unchanged, unison will detect this and not trigger a full rebuild.
The whole notion of a package becomes less important.
Cargo and Leiningen are the only two build/package tools I've used that are legitimately pleasant and unobtrusive. (And Cargo is more pleasant than Leiningen but I think that's the rest of the toolchain like rustc, clippy, etc)
Correlation is not causation, but some food for thought is that Rust and Clojure are also consistently the top two languages in Stack Overflow survey "most-loved languages" lists.
Correlation is not causation, but some food for thought is that Rust and Clojure are also consistently the top two languages in Stack Overflow survey "most-loved languages" lists.
Because anyone can write a mediocre "gets the immediate problem solved" type of package manager, while designing and implementing a programming language that someone else would want to use requires way more knowledge and skill.
Eventually the shortcomings of the package manager will become more evident, but at first it will seem to work and solve some pain points programmer have.
Something similar goes for build systems.
Eventually the shortcomings of the package manager will become more evident, but at first it will seem to work and solve some pain points programmer have.
Something similar goes for build systems.
Modern languages like Rust and Elixir are fine.
JavaScript, C/C++, Python are very old and the package systems were pretty much bolt-on. And of course a lot of historical factors were playing a bit role.
In the old days, people usually start minimal. When they get burned they also find they were locked-in - thus the bolt-ons.
> we apparently have no "package management theory"
People are definitely standing on top of each others' shoulders. Node.js npm actually felt like Ruby package tools but also being first-party (they also cut some corners, and made some design choices - some worked well and some didn't).
i.e. Hex for Elixir feels like nothing new, but avoided most common pitfalls others have experienced. I believe most language designers nowadays can do the same, given they want to play it safe and want nothing too novel.
JavaScript, C/C++, Python are very old and the package systems were pretty much bolt-on. And of course a lot of historical factors were playing a bit role.
In the old days, people usually start minimal. When they get burned they also find they were locked-in - thus the bolt-ons.
> we apparently have no "package management theory"
People are definitely standing on top of each others' shoulders. Node.js npm actually felt like Ruby package tools but also being first-party (they also cut some corners, and made some design choices - some worked well and some didn't).
i.e. Hex for Elixir feels like nothing new, but avoided most common pitfalls others have experienced. I believe most language designers nowadays can do the same, given they want to play it safe and want nothing too novel.
Avoiding past pitfalls is the best part of a new language ecosystem. I remember when Python was the new kid on the block and how joyous it was to work with in comparison to Perl. By the time Ruby came around, the same thing was being said about it, sometimes with aspersions cast at Python and how wonky it was.
Maybe after a few more generational cycles we'll develop the One True Language, bug-free, intuitive tooling, and sane to use.
Maybe after a few more generational cycles we'll develop the One True Language, bug-free, intuitive tooling, and sane to use.
We have such a theory, it's called "modules". But unfortunately, modules are even less popular than proper typesystems have been for decades.
It's no coincidence that Rust (which has basically copied Haskell's type system to a large degree) and Julia (which is essentially a Lisp in disguise) have somewhat sound packaging systems. C/C++ does not even have a tidy core language, let alone proper modules (yes, yes, C++ modules might change that, but they are still a complex mix of templates and normal code).
It's no coincidence that Rust (which has basically copied Haskell's type system to a large degree) and Julia (which is essentially a Lisp in disguise) have somewhat sound packaging systems. C/C++ does not even have a tidy core language, let alone proper modules (yes, yes, C++ modules might change that, but they are still a complex mix of templates and normal code).
I do not understand what you mean by modules, and how Rust and not Java or Python is helped by this.
Modules typically refers to a unit of code that packages together several types and functions (and/or other top-level concepts in your language, e.g. macros, templates, metaclasses, namespaces, packages) with an explicitly declared public interface, while still allowing these types and functions to have additional internal APIs that are not exposed. A module has to declare what other modules it depends on explicitly.
The public interface of a module has to be explicit enough that a compiler for that language only needs the public interface description of that module to compile code that depends on it.
A good example of the most widely used and successful module system is C compiled libraries. The header files represent the public description of a library (or, for dynamic linking, the .so/.dll itself embeds this info as well), and a C compiler only requires these header files to be able to compile dependent code and later link the two. It is also impossible to access non-public members of the library from standard C (without resorting to direct memory access, of course).
Java packages don't have these properties because of two problems: there is no way to indicate dependencies between packages, and there is no standard way to have a bundle that corresponds to a package (since different .jar files can all load classes into the same package). The Java module system was created specifically to address these problems (and is still not fully adopted throughout the ecosystem).
C++ breaks C's module system because of its heavy reliance on templates, which have to be entirely declared in the public API and can't have any internal API of their own; and because the compiler needs to know the total size of a class, including the size of its private members, to compile any code which references objects of that class.
For Python, I'm honestly not sure if Python packages fit this definition of a module or not.
The public interface of a module has to be explicit enough that a compiler for that language only needs the public interface description of that module to compile code that depends on it.
A good example of the most widely used and successful module system is C compiled libraries. The header files represent the public description of a library (or, for dynamic linking, the .so/.dll itself embeds this info as well), and a C compiler only requires these header files to be able to compile dependent code and later link the two. It is also impossible to access non-public members of the library from standard C (without resorting to direct memory access, of course).
Java packages don't have these properties because of two problems: there is no way to indicate dependencies between packages, and there is no standard way to have a bundle that corresponds to a package (since different .jar files can all load classes into the same package). The Java module system was created specifically to address these problems (and is still not fully adopted throughout the ecosystem).
C++ breaks C's module system because of its heavy reliance on templates, which have to be entirely declared in the public API and can't have any internal API of their own; and because the compiler needs to know the total size of a class, including the size of its private members, to compile any code which references objects of that class.
For Python, I'm honestly not sure if Python packages fit this definition of a module or not.
> C++ breaks C's module system because of its heavy reliance on templates, which have to be entirely declared in the public API and can't have any internal API of their own; and because the compiler needs to know the total size of a class, including the size of its private members, to compile any code which references objects of that class.
Other languages with generics and modules do just fine by having the required metadata on the module, e.g. Eiffel, Modula-3, Ada, OCaml,..., which is what C++20 modules do as well.
I have been using C++ modules on VC++ for quite some time, and while there are still some warts on the overall experience, they work quite alright.
Other languages with generics and modules do just fine by having the required metadata on the module, e.g. Eiffel, Modula-3, Ada, OCaml,..., which is what C++20 modules do as well.
I have been using C++ modules on VC++ for quite some time, and while there are still some warts on the overall experience, they work quite alright.
Sure, I should have said "C++ before the modules feature". I was thinking more to explain why the modules feature was required at all (when it's not really required too much for C).
Also, C# Assemblies solved this problem for generics too, for a more mainstream example.
Also, C# Assemblies solved this problem for generics too, for a more mainstream example.
You haven't explicitly enumerated the things about those packaging systems which seem "Rube Goldberg" to you; nor have you mentioned either Golang or Rust's package management. I'm pretty happy overall with Golang's package management, and from my little exposure to Rust's package manager, it seems fairly decent as well. Do either of them strike you as "Rube Goldberg"? What aspects, and why?
Package management in Go is weird but it works well. You just run go mod tidy from time to time and you're good. Add a module proxy if you want to prevent issues with repositories disappearing and you're gooder.
There's already a module proxy (pkg.go.dev) by default in new versions of Go, for just that purpose. In fact, one could argue that the automatic proxy is "Rube-Goldberg"-like: It works great normally, but if you need, for instance, to pull from a repo you just pushed to, you have to track down the magic rune to type to get `go get -u` to pull directly from your repo, rather than using the cached copy at pkg.go.dev.
In go, if you just pushed to a repo a tagged version, say "v1.4.2", then you just specify "go get -u "myrepo.com/[email protected]". Internally it will fail to fetch from the moduel proxy, but then (by default) it will then look directly for the git repo, and it will still work. But you can also use go workspaces if you are co-developing modules in different repositories.
> There's already a module proxy (pkg.go.dev) by default in new versions of Go, for just that purpose
Is it really a proxy in this sense? I was under the impression that it did not keep cached copies of the code, just some metadata, so that it didn't actually protect about someone removing their repo. Maybe I'm wrong about that though.
Is it really a proxy in this sense? I was under the impression that it did not keep cached copies of the code, just some metadata, so that it didn't actually protect about someone removing their repo. Maybe I'm wrong about that though.
> so that it didn't actually protect about someone removing their repo.
My impression, particularly based on the timing, was that the introduction of the proxy was in part directly in response to the "left-pad" fiasco. One thing it will also do is automatically publish your docs, if it can detect an open-source license. That would certainly require keeping a copy around.
My impression, particularly based on the timing, was that the introduction of the proxy was in part directly in response to the "left-pad" fiasco. One thing it will also do is automatically publish your docs, if it can detect an open-source license. That would certainly require keeping a copy around.
> Is it just a "law of software" that this must happen to any successful language? Or are there examples of where this it has not happened, and what can we learn from them?
Dotnet has nuget and it is quite pleasant to use in my experience. I think the big deal is having some standardized way of managing them as a layer on top. In teh dotnet case, it is having the solution pull in the nuget packages, rather than individual files. Along with having a standardized way to index available packages from each source (In this case the nuget json)
Dotnet has nuget and it is quite pleasant to use in my experience. I think the big deal is having some standardized way of managing them as a layer on top. In teh dotnet case, it is having the solution pull in the nuget packages, rather than individual files. Along with having a standardized way to index available packages from each source (In this case the nuget json)
Have you read https://medium.com/@sdboyer/so-you-want-to-write-a-package-m... ?
The author does a great job pointing out the problems, theory, and ecosystem change that makes it a Rube Goldberg machine.
By definition, a package manager is always "incomplete" because it cannot catch all the security vulnerabilities, binary compatibility, or knowing the dependencies ahead of time nor their interactions together. Thus it can be unsuccessful at managing dependencies - its primary job.
Source: Also work on a notable package manager.
The author does a great job pointing out the problems, theory, and ecosystem change that makes it a Rube Goldberg machine.
By definition, a package manager is always "incomplete" because it cannot catch all the security vulnerabilities, binary compatibility, or knowing the dependencies ahead of time nor their interactions together. Thus it can be unsuccessful at managing dependencies - its primary job.
Source: Also work on a notable package manager.
Part of it is deliberately ignoring the lessons of the past because they seem like overkill. Python has the excuse that it is old enough to predate the lessons of Java and Maven. But the node and js worlds deliberately ignored things like namespaces and mandatory version specification because they were uncool. And then they had to introduce package-lock and typo-squatting defences because the complexity of the world isn't always negotiable.
Why do we need different systems for all languages, all Linux/Windows/Mac/Android distributions. Why are no one reusing what is already there. It might seem like an easy problem to fix. Rust, Java and Go certainly has their problems too, not quite sure why people feel they are good examples.
I think the world is complicated and you will in the end need to support compilations/installations on AmigaOS 2.05 on m68k running in kubernetes native.
I think the world is complicated and you will in the end need to support compilations/installations on AmigaOS 2.05 on m68k running in kubernetes native.
Because I don't want my side project to share dependencies with system-wide python.
It's far simpler than you think
It all boils down to NIHS - "not invented here syndrome"
Between developers being suspicious of anyone else's work, and their innate love to build things they'll use (your wheel is cool, I guess...but MY wheel does THIS!) ...you get a gajillion libraries that overlaps 95% of the time (but never on the 5% you "care" about)
Eons ago David Pogue wrote in his review of Word (98, I think) for the Mac, "MacWrite fit on one floppy disk and had 95% of all the features I've ever needed in a word processor. Word take a CD, and has 96% of all the features I've ever needed in a word processor."
Why is Word so big? Is it because a 'word processor' needs to be that big?
Why are there so many libraries with incredibly weird interdependencies (version 1.9.2a of this lib needs 2.1.x2 of that one, not 2.1.x3 or 2.1.g7; but if you have version 1.9.3 of this lib, you can use 2.1.x2 up through 2.1.x9 ot he other one)?
Same thing - somebody somewhere sometime somewhy decided to use that library in their own stuff, and is now forever and always bound to it (...until they refactor or rewrite a perfectly good tool that was in Scala into Haskell)
It all boils down to NIHS - "not invented here syndrome"
Between developers being suspicious of anyone else's work, and their innate love to build things they'll use (your wheel is cool, I guess...but MY wheel does THIS!) ...you get a gajillion libraries that overlaps 95% of the time (but never on the 5% you "care" about)
Eons ago David Pogue wrote in his review of Word (98, I think) for the Mac, "MacWrite fit on one floppy disk and had 95% of all the features I've ever needed in a word processor. Word take a CD, and has 96% of all the features I've ever needed in a word processor."
Why is Word so big? Is it because a 'word processor' needs to be that big?
Why are there so many libraries with incredibly weird interdependencies (version 1.9.2a of this lib needs 2.1.x2 of that one, not 2.1.x3 or 2.1.g7; but if you have version 1.9.3 of this lib, you can use 2.1.x2 up through 2.1.x9 ot he other one)?
Same thing - somebody somewhere sometime somewhy decided to use that library in their own stuff, and is now forever and always bound to it (...until they refactor or rewrite a perfectly good tool that was in Scala into Haskell)
I think the issue is how hard it is to test with a range of package versions. Maven initially supported package ranges, but that got dropped in favour of repeatable builds.
They all followed the same process:
Step 1: look at all the existing package+module systems, and realise they are all horrible Rube Goldberg machines, way too bloated for the simple thing you want to do
Step 2: write your own package+module system! It's lean, mean and just does the simple obvious thing your simple one-man project needs.
Step 3: your system hits the real world! Your system is used with projects that involve more than one person and grow organically, and that spread to more and more esoteric environments. The Rube Goldberg projects have Rube Goldberg aims and their target ecosystems have Rube Goldberg requirements while goalposts shift continuously with tight deadlines. You add features and toggles to your system to support these things.
Step 4: congratulations! You are the proud inventor and owner of yet another Rube Goldberg machine. It is now time to move on to better things; the system, meanwhile, is handed over to some committee and will only grow in complexity from here.
https://www.kiplingsociety.co.uk/poem/poems_palace.htm
Step 1: look at all the existing package+module systems, and realise they are all horrible Rube Goldberg machines, way too bloated for the simple thing you want to do
Step 2: write your own package+module system! It's lean, mean and just does the simple obvious thing your simple one-man project needs.
Step 3: your system hits the real world! Your system is used with projects that involve more than one person and grow organically, and that spread to more and more esoteric environments. The Rube Goldberg projects have Rube Goldberg aims and their target ecosystems have Rube Goldberg requirements while goalposts shift continuously with tight deadlines. You add features and toggles to your system to support these things.
Step 4: congratulations! You are the proud inventor and owner of yet another Rube Goldberg machine. It is now time to move on to better things; the system, meanwhile, is handed over to some committee and will only grow in complexity from here.
https://www.kiplingsociety.co.uk/poem/poems_palace.htm
> See JavaScript/TypeScript, Python, or C/C++.
I feel like you've picked the 3 worst ecosystems as an example.
Java has a wonderful ecosystem. You can use Maven/Gradle/Bazel as a dependency manager/build tool. All support the Maven repository format. Packages are published as jars and easily consumed. Of course you can still end up in dependency hell and you should take steps to mitigate those issues, but I'm quite happy with it.
> As a result, we keep building new variants and features, until we end up with <script>, require(), import, npm, yarn, pnpm, (py)?(v|virtual|pip)?env, (ana)?conda, easy_install, eggs and wheels ...
IMO the problem here is that there isn't a monopoly and its easy to swap in a replacement. JavaScript/Python developers are happy (or willing) to try something new, so new alternatives are created. Convincing a Java developer to try a new build system is difficult, so changes are made to existing systems rather than creating new ones.
Go and Rust are two other systems that work well and are (almost?) universally adopted. They come with the language and there is little reason to look around.
I feel like you've picked the 3 worst ecosystems as an example.
Java has a wonderful ecosystem. You can use Maven/Gradle/Bazel as a dependency manager/build tool. All support the Maven repository format. Packages are published as jars and easily consumed. Of course you can still end up in dependency hell and you should take steps to mitigate those issues, but I'm quite happy with it.
> As a result, we keep building new variants and features, until we end up with <script>, require(), import, npm, yarn, pnpm, (py)?(v|virtual|pip)?env, (ana)?conda, easy_install, eggs and wheels ...
IMO the problem here is that there isn't a monopoly and its easy to swap in a replacement. JavaScript/Python developers are happy (or willing) to try something new, so new alternatives are created. Convincing a Java developer to try a new build system is difficult, so changes are made to existing systems rather than creating new ones.
Go and Rust are two other systems that work well and are (almost?) universally adopted. They come with the language and there is little reason to look around.
> JavaScript/Python developers are happy (or willing) to try something new, so new alternatives are created.
FWIW as far as JavaScript on NodeJS goes, npm did set the standard quite firmly. You can also use yarn or pnpm, which promise improvements over npm while using the same package metadata (`package.json`) and produce npm-compatible structures in the same subdirectory (`node_modules`), so overall the user experience is muche like going from `sudo apt-get install` to the newer, more polished and otherwise largely equivalent `sudo apt install`.
FWIW as far as JavaScript on NodeJS goes, npm did set the standard quite firmly. You can also use yarn or pnpm, which promise improvements over npm while using the same package metadata (`package.json`) and produce npm-compatible structures in the same subdirectory (`node_modules`), so overall the user experience is muche like going from `sudo apt-get install` to the newer, more polished and otherwise largely equivalent `sudo apt install`.
There is no real theory because it's a solved problem--packages and dependencies form a DAG or graph and it's just an ordering and traversal problem that's easily solved.
The pain and fragmentation you're mentioning is that everyone has different opinions about how they want to configure/bundle/organize code and package metadata. That's really the only core difference between deb/rpm, npm/yarn, pip/poetry/conda/<a million other python tools>, handmade makefile/cmake/autotools/etc. People just had different ideas about how they wanted to do things at the periphery. At their core all of those tools are just simple DAG walkers.
Basically, you're thinking it's a technical problem when in reality it's just a social issue. Some people prefer something one way, others prefer it the other way--there is no consensus (and likely never will be, people will always be building tools to do things their way).
The pain and fragmentation you're mentioning is that everyone has different opinions about how they want to configure/bundle/organize code and package metadata. That's really the only core difference between deb/rpm, npm/yarn, pip/poetry/conda/<a million other python tools>, handmade makefile/cmake/autotools/etc. People just had different ideas about how they wanted to do things at the periphery. At their core all of those tools are just simple DAG walkers.
Basically, you're thinking it's a technical problem when in reality it's just a social issue. Some people prefer something one way, others prefer it the other way--there is no consensus (and likely never will be, people will always be building tools to do things their way).
Well, not DAG, because the A is for acyclic, and dependency graphs can definitely contain cycles.
In the case of immutable builds (and hence immutable dependencies), dependency graphs cannot contain cycles. At most they can have the same package occur twice in the graph, but with different dependencies or different build options.
You might be referring to something specific thing I'm not familiar with... generally, an immutable build doesn't impose a structure on your dependency graph, just that it can't change (in any detail, hopefully) from build to build, without an explicit change in source.
They really shouldn't, and some dependency systems like rubygems/bundler after the molinillo switch ban them entirely.
Yeah, that's a misfeature. It should definitely be a DAG. Which ecosystems 'support' that besides Node?
> we apparently have no "package management theory"
Honestly, I'd like to have a lowest common denominator of "just dump the contents of the tar/zip archive" here.
Only check required would be to ensure you're not overwriting anything.
Dependencies and PATH management etc could be done out-of-band. That way you would not have to re-download stuff at least.
Honestly, I'd like to have a lowest common denominator of "just dump the contents of the tar/zip archive" here.
Only check required would be to ensure you're not overwriting anything.
Dependencies and PATH management etc could be done out-of-band. That way you would not have to re-download stuff at least.
I actually wrote something of that sorts for my personal projects:
https://masysma.net/11/maartifact.xhtml
Downloads tgz if not already downloaded, saves it to directory `../x-artifacts` and optionally extracts it.
https://masysma.net/11/maartifact.xhtml
Downloads tgz if not already downloaded, saves it to directory `../x-artifacts` and optionally extracts it.
I am starting to see some consistency these days. I mostly work in Go, so I'm not intimately familiar with the state of the art in other languages, but it looks like most ecosystems have a tool that support this flow.
A declarative (go.mod) file that always installs the same dependencies every build, a tool to update that lock file (go mod tidy), and an indication in the source that indicates how the dependency is being used (import).
A declarative (go.mod) file that always installs the same dependencies every build, a tool to update that lock file (go mod tidy), and an indication in the source that indicates how the dependency is being used (import).
And a way to locally override a dependency to conveniently work on multiple parts at the same time (go.work)
This is Go Workspaces for those unfamiliar
https://go.dev/doc/tutorial/workspaces
https://go.dev/doc/tutorial/workspaces
Have you used nuget for .NET? I have to say I've never once run into any of the problems I encounter sometimes weekly with NPM for Node.
Indeed, people act as if package managers were some Turing award level problem because they havent tried ecosystems that at least try to provide good ux like rust or .net
It's actually a nice change when I'm working on a .NET project knowing that some random dependency or NPM type problem is literally never going to be a problem with nuget.
I like golang package management in practice - sure, you have the "v2" thing and "every package can bump any package if you're not careful", but it works in practice because people are careful, and the minimal version selection is always deterministic.
The proxy thing and the drama with sr.ht is annoying, but maybe it was solved while I didn't look.
The proxy thing and the drama with sr.ht is annoying, but maybe it was solved while I didn't look.
It's because the software ITSELF is structured poorly, not just the package managers
Big pyramids of dependencies are inherently fragile (whether dynamically or statically typed)
Dependency inversion can make it so that your dependency tree is exactly 1 deep, and it's dynamically wired together in main()
But most people don't write code in that style. It takes extra effort
Big pyramids of dependencies are inherently fragile (whether dynamically or statically typed)
Dependency inversion can make it so that your dependency tree is exactly 1 deep, and it's dynamically wired together in main()
But most people don't write code in that style. It takes extra effort
In C#, I liked the pre-Nuget approach where you just made sure 3rd party dlls were in a location relative to source code control, (or in source code control if they were small.)
Not really "package management," but it was straightforward and worked very well... (Until you had to juggle 32/64 bit native dlls.)
Not really "package management," but it was straightforward and worked very well... (Until you had to juggle 32/64 bit native dlls.)
You can do that with nuget too.
If I wanted to point out what the greatest mistake in the conception of the NodeJS package manager—npm—is, its 'original sin', if you will, is the conceit that SemVer ('semantic versioning', a great idea in itself) is 100% reliable and that, therefore, it would be heretic or contradictory for a single module or sub-package in a project to ever want to `require()` two distinct versions of a module, say, [email protected] and [email protected]. This is now fixed[1] but the developers stubbornly refused to see the occasional necessity of doing just this for many years.
[1](https://medium.com/weekly-webtips/how-to-install-multiple-ve...)
[1](https://medium.com/weekly-webtips/how-to-install-multiple-ve...)
If you let everyone and their dog create and maintain packages and package dependencies and leave it to them to declare forward/backward compatibility between versions of packages, and your package manager ends up being popular, I think it’s unavoidable that you end up with a mess.
Foo may not be updated for Bar 2.0, while the Baz you use states it needs it, Bar 3.0 may incorrectly declare to be a 100% stand-in for anything between Bar 1.5 and Bar 3.0, Baz and Qux may contain similar functions that would better be part of their shared dependency Quux, etc.
Tooling can make navigating that mess more or less difficult, but the mess still is there.
Foo may not be updated for Bar 2.0, while the Baz you use states it needs it, Bar 3.0 may incorrectly declare to be a 100% stand-in for anything between Bar 1.5 and Bar 3.0, Baz and Qux may contain similar functions that would better be part of their shared dependency Quux, etc.
Tooling can make navigating that mess more or less difficult, but the mess still is there.
You might want to check PackagingCon https://www.youtube.com/watch?v=bdX1XXpLFYY&list=PLl386dCR5Q...
[deleted]
A few reasons:
1) Every developer no matter how new wants to contribute something. 2) There is no centralized authority in most languages/package system to stop them doing so 3) Imagine all the bad code you write in the first 2 years but now you can't delete it because 100 other developers rely on it and 100 other developers rely on them
The rube goldberg machine exists because of the matryoshka dolls that is the dependency tree. Sometimes rewriting things is actually easier than trying to untangle the wire of dependencies.
1) Every developer no matter how new wants to contribute something. 2) There is no centralized authority in most languages/package system to stop them doing so 3) Imagine all the bad code you write in the first 2 years but now you can't delete it because 100 other developers rely on it and 100 other developers rely on them
The rube goldberg machine exists because of the matryoshka dolls that is the dependency tree. Sometimes rewriting things is actually easier than trying to untangle the wire of dependencies.
[deleted]
I think the problem is the answer to this question:
What is harder regarding computation - arranging for a computation or doing the computation?
What I mean is that the computation of an addition or subtraction is easy but the arranging of instructions of code into an algorithm is hard. Everything has to be in the right place for the algorithm to work right.
This is why packaging is so difficult. Each algorithm and code expects things to be in a certain place. ABIS and packaging for them is tedious.
What is harder regarding computation - arranging for a computation or doing the computation?
What I mean is that the computation of an addition or subtraction is easy but the arranging of instructions of code into an algorithm is hard. Everything has to be in the right place for the algorithm to work right.
This is why packaging is so difficult. Each algorithm and code expects things to be in a certain place. ABIS and packaging for them is tedious.
Pythoners have themselves to blame.
The worst thing that happened to Python was that distributions like Red Hat not only shipped Python as an rpm but shipped system scripts in Python. This made it practically impossible to upgrade the system Python without breaking scripts and is fundamentally incompatible with how Python packaging works. (You are just one 'sudo pip install' from breaking your system)
Like people who are impressed with ChatGPT (wow that answer seemed so confident even if it is wrong), Pythoners systematically mistake footguns for solutions. For instance, use 'pip install --user' and now the package you've included is visible when you type 'python', 'python3', use a venv, conda, whatever. I've had people tell me that having a 'python3' is the best idea anyone's had since Solomon proposed cutting a baby in half, but if you stop and think you realize that pretty soon you have 'python3', 'python3.5' (broken), 'python3.6' (maybe broken), 'python3.7', 'python3.8', 'python3.9', ...
I would say venv's really solve the problem with the exception that you can currently trash all your venv's with 'pip install --user'.
The trouble is that people who are bothered by this stop using Python, the people who are still using Python are people who have difficulty perceiving the nature of the problem never mind that there is a problem.
The algorithm used by pip to resolve packages is not sound. It goes off trying to install things and can only find out late in the game that it installed a package which is not compatible with another package. This can be solidly blamed on the egg package format which can only determine the dependencies of a package when it tries to install it by running a setup.py. This is nice because it can adapt to the environment dynamically (say add a timezone db on windows) but it is not like Java's maven that knows it has a consistent solution before it downloads the JARs.
I'm pretty sure you could make a Python package manager that works like Maven if it only installed wheels because you can read the directory of a wheel (a ZIP file) with one or two http range requests and then read the dependency data with another http range request. Poetry doesn't quite do this. Poetry does a lot better than pip does, but whatever it does takes a really, really, really, long time.
Poetry might be the best thing going but it has the fundamental flaw of trying to be a 95% or 98% solution which, to the Pythoner, sounds like a good idea. The trouble is that writing programs that are 98% is to computer science what Bigfoot is to zoology or ESP is to psychology. The gap between that and 100% is not 2% but more like 200% because figuring out exactly what is wrong with the conceptual model and working around it in a hard case is at best like pushing a bubble under a rug. Imagine how much trouble you could get into with a 98% correct sort or binary search algorithm... It's a place where professional developers just don't want to go.
Java has the advantage of extreme xenophobia and bad feelings all around that mean that: (1) people don't expect to link very much C code into Java, and (2) people don't feel entitled to use the Java bundled with their Linux system and have it really work. Because of (1), Java code tends to be self-sufficient and avoids a whole sector of 'dependency hell' involving .so and .dll files. Because of (2) people feel both responsible and empowered to install a JDK that works unlike the typical who Pythoner doesn't.
Don't think that docker helps in any way, what I found what that Docker accelerates the ability of Pythoner to find pythons that are broken in odd ways, configured with strange default character sets and such.
Javascript has the nice feature that file A can import from package version B and file C can import from package version D so that the 'diamond dependency' problems that are so terrible in Java (Like that bug in Guava post-13 that broke HDFS) and still problematic in Python rarely produce problems. (You only have problems if an object from A migrates to C and it gets used in incompatible way, contrast that to Java where probably the classloader blows up)
I was wishing over the weekend that Python had something like that because once more I have been building and packaging machine learning models for inference in Python and it would be really sweet to pack up a model that uses scikit-learn or pytorch or whatever into a Python package and then load it into a web server or a batch job. It's totally practical to do that, using joblib to unpickle a few MB of code. Once you end up needing two different versions of pytorch though, you are really screwed.
The worst thing that happened to Python was that distributions like Red Hat not only shipped Python as an rpm but shipped system scripts in Python. This made it practically impossible to upgrade the system Python without breaking scripts and is fundamentally incompatible with how Python packaging works. (You are just one 'sudo pip install' from breaking your system)
Like people who are impressed with ChatGPT (wow that answer seemed so confident even if it is wrong), Pythoners systematically mistake footguns for solutions. For instance, use 'pip install --user' and now the package you've included is visible when you type 'python', 'python3', use a venv, conda, whatever. I've had people tell me that having a 'python3' is the best idea anyone's had since Solomon proposed cutting a baby in half, but if you stop and think you realize that pretty soon you have 'python3', 'python3.5' (broken), 'python3.6' (maybe broken), 'python3.7', 'python3.8', 'python3.9', ...
I would say venv's really solve the problem with the exception that you can currently trash all your venv's with 'pip install --user'.
The trouble is that people who are bothered by this stop using Python, the people who are still using Python are people who have difficulty perceiving the nature of the problem never mind that there is a problem.
The algorithm used by pip to resolve packages is not sound. It goes off trying to install things and can only find out late in the game that it installed a package which is not compatible with another package. This can be solidly blamed on the egg package format which can only determine the dependencies of a package when it tries to install it by running a setup.py. This is nice because it can adapt to the environment dynamically (say add a timezone db on windows) but it is not like Java's maven that knows it has a consistent solution before it downloads the JARs.
I'm pretty sure you could make a Python package manager that works like Maven if it only installed wheels because you can read the directory of a wheel (a ZIP file) with one or two http range requests and then read the dependency data with another http range request. Poetry doesn't quite do this. Poetry does a lot better than pip does, but whatever it does takes a really, really, really, long time.
Poetry might be the best thing going but it has the fundamental flaw of trying to be a 95% or 98% solution which, to the Pythoner, sounds like a good idea. The trouble is that writing programs that are 98% is to computer science what Bigfoot is to zoology or ESP is to psychology. The gap between that and 100% is not 2% but more like 200% because figuring out exactly what is wrong with the conceptual model and working around it in a hard case is at best like pushing a bubble under a rug. Imagine how much trouble you could get into with a 98% correct sort or binary search algorithm... It's a place where professional developers just don't want to go.
Java has the advantage of extreme xenophobia and bad feelings all around that mean that: (1) people don't expect to link very much C code into Java, and (2) people don't feel entitled to use the Java bundled with their Linux system and have it really work. Because of (1), Java code tends to be self-sufficient and avoids a whole sector of 'dependency hell' involving .so and .dll files. Because of (2) people feel both responsible and empowered to install a JDK that works unlike the typical who Pythoner doesn't.
Don't think that docker helps in any way, what I found what that Docker accelerates the ability of Pythoner to find pythons that are broken in odd ways, configured with strange default character sets and such.
Javascript has the nice feature that file A can import from package version B and file C can import from package version D so that the 'diamond dependency' problems that are so terrible in Java (Like that bug in Guava post-13 that broke HDFS) and still problematic in Python rarely produce problems. (You only have problems if an object from A migrates to C and it gets used in incompatible way, contrast that to Java where probably the classloader blows up)
I was wishing over the weekend that Python had something like that because once more I have been building and packaging machine learning models for inference in Python and it would be really sweet to pack up a model that uses scikit-learn or pytorch or whatever into a Python package and then load it into a web server or a batch job. It's totally practical to do that, using joblib to unpickle a few MB of code. Once you end up needing two different versions of pytorch though, you are really screwed.
This is a very good rant, worth bookmarking to refer others to who are curious about packaging problems
Because of a surplus of worn out boots, candles, chickens, bowling balls (all bygone common, everyday, understandable items), and human invention (meaning "copying" badly, wheel reinvention - seldomly punctuated with novel synthesis).
Or was that open source software? Or was that any private corporate software ecosystem? If I squint the problems all start to look the same. I must need glasses, considering my squint frequency.
Or was that open source software? Or was that any private corporate software ecosystem? If I squint the problems all start to look the same. I must need glasses, considering my squint frequency.
It sounds like you’re familiar with the status quo, so let’s look at this as a matter of requirements:
- Cross-platform - Wraps another packaging format - Lowest disk space - Dependency version solving - An OS can run make changes to its own low-level state using its own tools - User can configure when and what are installed
Even Nix, which has its own language and isolation model, can’t meet all these.
- Cross-platform - Wraps another packaging format - Lowest disk space - Dependency version solving - An OS can run make changes to its own low-level state using its own tools - User can configure when and what are installed
Even Nix, which has its own language and isolation model, can’t meet all these.
Nix almost does meet those though.
It's cross-platform to some degree. If Windows is a requirement for this, I see why it doesn't qualify.
Nix wraps other packaging formats all the time. There's endless x2nix converters.
In a way, Nix is "lowest disk space" - by nature of lazy evaluation, Nix closures are always exactly what they need. Now, they may "actually" need less and are holding onto more due to how packages are written by people. And you do need to GC.
Dependency version solving can actually be punted to other tools. In Haskell, Nix uses a combination of cabal and pkg-config to handle Haskell + C deps.
NixOS can swap out the kernel along with anything else with a line or two in your config.
It's cross-platform to some degree. If Windows is a requirement for this, I see why it doesn't qualify.
Nix wraps other packaging formats all the time. There's endless x2nix converters.
In a way, Nix is "lowest disk space" - by nature of lazy evaluation, Nix closures are always exactly what they need. Now, they may "actually" need less and are holding onto more due to how packages are written by people. And you do need to GC.
Dependency version solving can actually be punted to other tools. In Haskell, Nix uses a combination of cabal and pkg-config to handle Haskell + C deps.
NixOS can swap out the kernel along with anything else with a line or two in your config.
At least in Python's case, I think people are much too hard on the packaging system. I don't think anyone has significant issues with pip itself as opposed to the runtime it exists within.
Following from this, I think that most back-end applications should try solve their messy runtime environment issues with some containerization. Java/C(++) however...
Following from this, I think that most back-end applications should try solve their messy runtime environment issues with some containerization. Java/C(++) however...
Java's Maven has been the nicest package management tool I've ever worked with. Apart from the fact that definitions are a bit verbose, it supported everything I've ever wanted from such a tool:
1. Clear definitions of what is an artifact that it manages, easily described in a versioned file but still separate from your source code
2. Simple support to combine internal and external sources
3. Support for interacting with other package ecosystems (you can easily write a pom.xml for any other kind of package and reference it from your Java projects)
4. Not picky about version number formats
5. Not reliant on other tools to handle any step of package management (except for compilation, of course)
6. Works at the project level and has no problems about different versions being used in different projects on the same machine (with a shared cache for efficiency)
1. Clear definitions of what is an artifact that it manages, easily described in a versioned file but still separate from your source code
2. Simple support to combine internal and external sources
3. Support for interacting with other package ecosystems (you can easily write a pom.xml for any other kind of package and reference it from your Java projects)
4. Not picky about version number formats
5. Not reliant on other tools to handle any step of package management (except for compilation, of course)
6. Works at the project level and has no problems about different versions being used in different projects on the same machine (with a shared cache for efficiency)
And because of that, maven infrastructure is and has been the target of multiple build frameworks and languages.
I would like to second that Maven works for me as well.
A lot of posters have said that Rust and Go work well.
Honestly, the OP sounds like the stereotype of a JS developer: "This is a problem for Javascript, therefore it must be a problem for everyone else, too."
No, guys, it's just you. Every single time.
A lot of posters have said that Rust and Go work well.
Honestly, the OP sounds like the stereotype of a JS developer: "This is a problem for Javascript, therefore it must be a problem for everyone else, too."
No, guys, it's just you. Every single time.
> I don't think anyone has significant issues with pip itself as opposed to the runtime it exists within.
Pip has lots of issues (IMO it's one of the worst package managers):
- It installs dependencies globally by default! This makes it a massive pain to work with unless you get virtualenvs setup correctly. Which you have to remember to do every time you interact with a project.
- It doesn't use lock files by default. You have to remember to lock your dependencies.
- Packages that depend on C libraries have a tendency to fail with cryptic error messages if you don't have the library installed (which can be contrasted with node.js packages which tend to bundle the library, and will even compile it from source for you if there isn't a binary version available for your platform).
Pip has lots of issues (IMO it's one of the worst package managers):
- It installs dependencies globally by default! This makes it a massive pain to work with unless you get virtualenvs setup correctly. Which you have to remember to do every time you interact with a project.
- It doesn't use lock files by default. You have to remember to lock your dependencies.
- Packages that depend on C libraries have a tendency to fail with cryptic error messages if you don't have the library installed (which can be contrasted with node.js packages which tend to bundle the library, and will even compile it from source for you if there isn't a binary version available for your platform).
> I don't think anyone has significant issues with pip
and boy would you be wrong. I just tried to install `datasette-scraper` on my system. This didn't work because it would consistently pick up outdated versions of Django and `more-itertools` from the system's Python site-packages directory.
By the way I learned to (1) install an updated version of Python that has the advantage of not being the one that the system uses; (2) do not use `pip` the executable but `python3.9 -m pip`; (3) use `pip` with argument `--target vendor` to install everything in a dedicated project-owned directory; (4) prefix my invocation of the software with `PYTHONPATH=./vendor` to add that to the import path; (5) add a `vendor/sitecustomize.py` file with the lines `import sys; sys.path.pop()` to remove the system's `site-packages` from the path.
That's a local installation of a Python package in five easy steps, no 'virtual-env' or some such! Pip is great! /s
and boy would you be wrong. I just tried to install `datasette-scraper` on my system. This didn't work because it would consistently pick up outdated versions of Django and `more-itertools` from the system's Python site-packages directory.
By the way I learned to (1) install an updated version of Python that has the advantage of not being the one that the system uses; (2) do not use `pip` the executable but `python3.9 -m pip`; (3) use `pip` with argument `--target vendor` to install everything in a dedicated project-owned directory; (4) prefix my invocation of the software with `PYTHONPATH=./vendor` to add that to the import path; (5) add a `vendor/sitecustomize.py` file with the lines `import sys; sys.path.pop()` to remove the system's `site-packages` from the path.
That's a local installation of a Python package in five easy steps, no 'virtual-env' or some such! Pip is great! /s
> no 'virtual-env' or some such!
I'm pretty sure you're manually doing the same thing virtualenv (venv in python3) does for you...
I'm pretty sure you're manually doing the same thing virtualenv (venv in python3) does for you...
I have personally never had a problem with activating a virtualenv and `pip install`-ing inside it. Most libraries nowadays also specify minimum versions for packages (e.g., >=3.6, <4.0).
I'd guess that, yes, and going forward I may take a 2nd look at virtualenv. That said, I am a little proud and glad I managed to take it this far in this particular instance because I feel it has improved my ability to deal with Pythonic installation gotchas. Pip is a great teaching tool! /s
Everybody uses PIP, but the Python devs (well single BDFL) refuse to accept this and standardize the package manager. As a result, if you want to make a package, you must support everything.
Also, PIP can't evolve, because nobody will make their packages even more complex to support PIP only idiosyncrasies. It doesn't matter that everybody uses PIP, you have to spend 90% of your time dealing with other package managers, so they are the ones on your mind all the time.
Also, PIP can't evolve, because nobody will make their packages even more complex to support PIP only idiosyncrasies. It doesn't matter that everybody uses PIP, you have to spend 90% of your time dealing with other package managers, so they are the ones on your mind all the time.
I find myself in nightmarish scenarios with Python dependencies more often than I'd like to. This dependency is missing this wheel which is missing this bit of C tooling which needs system-level permissions to be installed through XCode on Mac (what?). I've somehow corrupted the package lock, now I'm looking at diffs and checking dependencies of dependencies, inspecting the virtual environment folder and questioning my life choices. PyCharm has somehow lost the sense of where the virtual environment is located, I now have a squiggly line and warning explosion in my IDE.
You could argue many of these are not Python's fault, some can probably be attributed to carelessness on third-party and tooling devs or on my part. But let's invent a metric here: "% of time spent dependency troubleshooting over total programming time". I've had better experiences with this in other languages, and intuition tells me that my carelessness or that of third-party devs being equal, there is something not quite right about the Python dependency management story. I put up with it because like many others I think that the language is otherwise dope.
You could argue many of these are not Python's fault, some can probably be attributed to carelessness on third-party and tooling devs or on my part. But let's invent a metric here: "% of time spent dependency troubleshooting over total programming time". I've had better experiences with this in other languages, and intuition tells me that my carelessness or that of third-party devs being equal, there is something not quite right about the Python dependency management story. I put up with it because like many others I think that the language is otherwise dope.
> I don't think anyone has significant issues with pip itself as opposed to the runtime it exists within.
I'm running macOS with Macports, and oh boy. For a long time, the aws and eb CLI tools had different package requirements or whatnot, and you couldn't install both of them simultaneously for whatever reason. Some stuff installs fine with the system Python, some stuff needs Macports for a specific Python version, some stuff needs root access to install itself... anything Python is a hot mess.
I'm running macOS with Macports, and oh boy. For a long time, the aws and eb CLI tools had different package requirements or whatnot, and you couldn't install both of them simultaneously for whatever reason. Some stuff installs fine with the system Python, some stuff needs Macports for a specific Python version, some stuff needs root access to install itself... anything Python is a hot mess.
System Python? Root access? Oh. That's a terrible, terrible practice.
First, one never touches the system Python. It's there for OS-managed stuff and to run OS components.
Then, every Python-based tool should be installed in a separate, unprivileged virtualenv, with executables symlinked where you find them appropriate (usually ~/bin or /usr/local/bin).
If you find yourself ever issuing a `sudo pip install`, chances are million to one that you are doing a disservice to to yourself.
First, one never touches the system Python. It's there for OS-managed stuff and to run OS components.
Then, every Python-based tool should be installed in a separate, unprivileged virtualenv, with executables symlinked where you find them appropriate (usually ~/bin or /usr/local/bin).
If you find yourself ever issuing a `sudo pip install`, chances are million to one that you are doing a disservice to to yourself.
> First, one never touches the system Python. It's there for OS-managed stuff and to run OS components.
Why would one want to not use what the system provides? Everything not provided by the OS is additional maintenance burden on myself. The 'nix world has managed just fine with OS-provided bash, perl and other runtime dependencies for decades.
> Then, every Python-based tool should be installed in a separate, unprivileged virtualenv, with executables symlinked where you find them appropriate (usually ~/bin or /usr/local/bin).
No one has time for all that. I'm not a Python developer - as a user I'm happy enough if it barely works.
When an ecosystem requires messing around with completely separate instances of the runtime for each program, that's not a good sign for the quality of the ecosystem as a whole.
Why would one want to not use what the system provides? Everything not provided by the OS is additional maintenance burden on myself. The 'nix world has managed just fine with OS-provided bash, perl and other runtime dependencies for decades.
> Then, every Python-based tool should be installed in a separate, unprivileged virtualenv, with executables symlinked where you find them appropriate (usually ~/bin or /usr/local/bin).
No one has time for all that. I'm not a Python developer - as a user I'm happy enough if it barely works.
When an ecosystem requires messing around with completely separate instances of the runtime for each program, that's not a good sign for the quality of the ecosystem as a whole.
While I'm trained to use venvs for everything, it would be nicer to have a flat global list of dependencies in every version required of every package, and specify allowed versions in each project, which will use what you have and/or download what's missing.
Almost any trivial application can be trivially implemented using these build systems. People encounter their own “unique” edge cases and build their own “unique” solutions to them. Over time, all these edge cases result in a proliferation of bubble gum, duct tape and baling wire that you’ve described.
Because most developers don't learn from other projects and redo the same mistakes.
It's a form of not-invented-here syndrome and you can see it in all the comments defending their favorite language..
> we apparently have no "package management theory".
Apparently. In reality there's been plenty of research - and for decades - that is not being used.
It's a form of not-invented-here syndrome and you can see it in all the comments defending their favorite language..
> we apparently have no "package management theory".
Apparently. In reality there's been plenty of research - and for decades - that is not being used.
> In reality there's been plenty of research - and for decades - that is not being used.
Can you link what you're referring to? Genuine request, I'm curious and keen to learn more.
Can you link what you're referring to? Genuine request, I'm curious and keen to learn more.
A solution can not be simpler than the problem.
The problem is a classic "looks simple until you really examine it".
The problem is a classic "looks simple until you really examine it".
This is the answer. If you really want to learn about the issue, try designing a package manager in detail, and be sure to handle the thousands of edge cases which arise in real world usage.
[deleted]
I'll say that on the whole it seems to be getting better with time. Most people have already mentioned Rust / Go as examples of this.
I do agree it is an issue. Doing away with node_modules completely on the last rails 7 app I worked on was the most cathartic experience of my life.
I do agree it is an issue. Doing away with node_modules completely on the last rails 7 app I worked on was the most cathartic experience of my life.
Have you experienced nuget? Seems pretty solved and solid and unchanged since... forever now.
I'd think that with all the CS PhDs, there'd be some academic work in this area. I haven't gone looking.
My experience as a .Net developer has not been that it's a Rube Goldberg machine. I isolate all my projects and minimize 3rd party dependencies.
My experience as a .Net developer has not been that it's a Rube Goldberg machine. I isolate all my projects and minimize 3rd party dependencies.
I don't see many complaints about Free Pascal/Delphi. You want a unit? Get it, add it, use it.
No 3rd party Rube Goldberg contraption needed. No package managers or dependency resolvers, no special build tooling, no lock files. No constantly having to run updates and figure out what they broke by way of transitive dependencies. Just get what you need and use it.
I haven't, however, developed enough professionally with it to know how that plays out long-term in practice. But I'd have to imagine it would compare favorably to just about everything else nowadays. The things I've been using for the last couple of decades are all pretty abysmal in comparison.
I'd say they were a solution in search of a problem, but people have managed to create a problem to justify the solution, which really still isn't very good. At least, not nearly as good as the simple get, add, use pattern.
No 3rd party Rube Goldberg contraption needed. No package managers or dependency resolvers, no special build tooling, no lock files. No constantly having to run updates and figure out what they broke by way of transitive dependencies. Just get what you need and use it.
I haven't, however, developed enough professionally with it to know how that plays out long-term in practice. But I'd have to imagine it would compare favorably to just about everything else nowadays. The things I've been using for the last couple of decades are all pretty abysmal in comparison.
I'd say they were a solution in search of a problem, but people have managed to create a problem to justify the solution, which really still isn't very good. At least, not nearly as good as the simple get, add, use pattern.
There are two problems that make package management hard, neither which are tackled by a "Get it, add it, use it" approach. Dealing with cross platform issues and package dependencies. If we ignore these two problems then package management is easy. We could make a 'rule' that packages cannot depend on other packages, and that packages only support the single OS/CPU combo they were originally developed for, but that will cause other problems.
Your mistake is in the use of the word "every" when in fact you've only considered a few. Rust, for instance, uses cargo for packaging and it's probably the most intuitive part of Rust (/s).
TBH I think you may just have a sampling problem - the JS and Python spaces have two of the messiest package-management ecosystems.
> neat-and-tidy (...) C++.
This is a huge oxymoron. Additionally, C++ does not even have a package manager.
This is a huge oxymoron. Additionally, C++ does not even have a package manager.
What about haskell cabal and stack
I appreciate Hackage and its lack of garbage, although I think the exclusivity can't scale. I also like Haskell's module system
All other aspects of Haskell package management feel unfinished, like "an exercise left to the reader." The lack of polish is astounding.
See also: "Why I Don't Code in Haskell Anymore" (https://www.youtube.com/watch?v=SPwnfSmyAGI) from TsodingDaily
All other aspects of Haskell package management feel unfinished, like "an exercise left to the reader." The lack of polish is astounding.
See also: "Why I Don't Code in Haskell Anymore" (https://www.youtube.com/watch?v=SPwnfSmyAGI) from TsodingDaily
You don't have to use Hackage.
What part is unfinished? Just docs and the verbose CLI? It solves for versions, you can declare native dependencies in various ways, it has multiple solid integrations with Nix (which in turn automatically resolves both Haskell and native deps for you).
Also that video is not a compelling source. That guy is just wrong lol. Or, rather, it's just his opinion that seems to be based on the 5-10 years ago state of Haskell.
I'm an engineer (professionally - in Haskell - for years), and I don't find anything about the packaging or "engineering" painful.
If we're talking engineering, the optimizing compiler + RTS are extremely impressive.
What part is unfinished? Just docs and the verbose CLI? It solves for versions, you can declare native dependencies in various ways, it has multiple solid integrations with Nix (which in turn automatically resolves both Haskell and native deps for you).
Also that video is not a compelling source. That guy is just wrong lol. Or, rather, it's just his opinion that seems to be based on the 5-10 years ago state of Haskell.
I'm an engineer (professionally - in Haskell - for years), and I don't find anything about the packaging or "engineering" painful.
If we're talking engineering, the optimizing compiler + RTS are extremely impressive.
Well sure, there's Stackage, and there's Nix, and there's some other options.
Haskell, for my money, is similar to Node.js in that I don't have confidence I could reliably expect the state of Haskell development _today_ to resemble Haskell development 5~10 years from now. There's a cost to falling behind (who wants to work on a Haskell project using 10-year-old conventions?) and there's also a cost to keeping up.
Don't get me wrong, I think it's getting better. Stack is easy enough to learn, Stackage has nice for guarantees on compatibility. Nix solves some issues for deployments, although "multiple solid integrations" is maybe a too-rosy description from my experience... but I don't think that Haskell's ecosystem is in its final form yet. Not in the way that Maven or Cargo feel like they're stupid simple and here to stay.
Haskell, for my money, is similar to Node.js in that I don't have confidence I could reliably expect the state of Haskell development _today_ to resemble Haskell development 5~10 years from now. There's a cost to falling behind (who wants to work on a Haskell project using 10-year-old conventions?) and there's also a cost to keeping up.
Don't get me wrong, I think it's getting better. Stack is easy enough to learn, Stackage has nice for guarantees on compatibility. Nix solves some issues for deployments, although "multiple solid integrations" is maybe a too-rosy description from my experience... but I don't think that Haskell's ecosystem is in its final form yet. Not in the way that Maven or Cargo feel like they're stupid simple and here to stay.
I just spent a few hours in npm/node/heroku hell, so... yeah. It's awful.
Rube Goldberg was a prophet.
"The entropy-pool, a reservoir of dumped complexity"
[deleted]
Scope creep.
Lots of brain cycles are spent on "programming language theory". We've roughly figured out the primitives required to express real-world computation.
In contrast, we apparently have no "package management theory". We have not figured out the primitives required to express dependencies. As a result, we keep building new variants and features, until we end up with <script>, require(), import, npm, yarn, pnpm, (py)?(v|virtual|pip)?env, (ana)?conda, easy_install, eggs and wheels ...
Is it just a "law of software" that this must happen to any successful language? Or are there examples of where this it has not happened, and what can we learn from them? Is there a "theory of package management", or a "lambda calculus of package management" out there?