“We do not plan to publish the AWS CLI project to PyPI at this time”(github.com)
github.com
“We do not plan to publish the AWS CLI project to PyPI at this time”
https://github.com/aws/aws-cli/issues/4947
43 comments
You can't, as they apparently depend on unreleased versions of botocore, or did depend on them in the past, according to a few people in the thread who tried that.
Should also be able to do
pipx install https://github.com/aws/aws-cli/archive/refs/tags/2.2.24.zip
Honest question. The package management ecosystem for Python is terrible, why would anyone want to use it for command line apps?
Every time I've done that, everything ends in disaster.
Every time I've done that, everything ends in disaster.
It works fine for pure Python applications. Especially if you use Python virtual environments to keep your application's dependencies isolated from the system-level installed libraries. An application is just a Python package or module, with a small stub stuck in $PATH that does the equivalent of 'python -m mystuf'. One of the nice benefits is users get to install things into $HOME if they don't want to pollute the 'system' install, even if they don't want to mess with the virtual environment features.
You know what, that's actually quite true. I realize that it's good to make sure everyone on a team has access to the same command line tools as well.
Just pip install the virtualenv and boom, cli tools in, for everyone.
Just pip install the virtualenv and boom, cli tools in, for everyone.
> we want the fact that the AWS CLI is written in Python to be more of an implementation detail than a feature.
Sounds about right. Nothing more to see here.
Sounds about right. Nothing more to see here.
The final response from the team at Amazon is incredibly gracious considering how condescending and entitled almost all the other comments are. Please don’t be like those commenters.
You can call it entitled, but:
- Lots of people don't have a choice if they want to use AWS, someone in the business decided it for them
- AWS is not some non-profit open source pet project but actively tries to pull as many devs and companies as possible into their ecosystem
- Breaking users' workflows is always going to cause issues. AWS CLI v1 was distributed on pypi, lots of people had already built workflows around that.
- It is my understanding that for some people / use cases (e.g. using pyenv), having aws cli being tied to the system python causes significant issues for some reason (I can't remember exactly why, but a coworker struggled with this for days). It might have something to do with the PYTHONPATH, not sure.
- The whole response from Amazon reads as "we know what's best for you" and frankly, I find that kind of attitude (which seems to be increasing among big corps) infuriating. Large tech companies already have way too much control over the ways we build software, it would be nice if they didn't also control our workflows.
Full agreement that many people in that GitHub issue were not being constructive and behaving childishly, but I still believe that Amazon deserves criticism here.
- Lots of people don't have a choice if they want to use AWS, someone in the business decided it for them
- AWS is not some non-profit open source pet project but actively tries to pull as many devs and companies as possible into their ecosystem
- Breaking users' workflows is always going to cause issues. AWS CLI v1 was distributed on pypi, lots of people had already built workflows around that.
- It is my understanding that for some people / use cases (e.g. using pyenv), having aws cli being tied to the system python causes significant issues for some reason (I can't remember exactly why, but a coworker struggled with this for days). It might have something to do with the PYTHONPATH, not sure.
- The whole response from Amazon reads as "we know what's best for you" and frankly, I find that kind of attitude (which seems to be increasing among big corps) infuriating. Large tech companies already have way too much control over the ways we build software, it would be nice if they didn't also control our workflows.
Full agreement that many people in that GitHub issue were not being constructive and behaving childishly, but I still believe that Amazon deserves criticism here.
Aws cli v2 installs it's own self contained python so it shouldn't be tied to any system install.
Frankly it's good to see devs not ceding to the internet rage mob. I remember before aws hired the main guy behind boto and I had to install 10 different tar.gz tools from 10 different aws teams and install all of them and setup 20 different env variables.
Running their installer isn't difficult and I've included it in our docker and ami builds easily. If you want a fixed version, download whatever from their repo and mirror it to your internal storage and you can always install from there and ensure you're running the exact version you like.
This response (from a year ago) seems perfectly reasonable and explains in detail a good number of reasons why they aren't using pip: https://github.com/aws/aws-cli/issues/4947#issuecomment-5860...
Frankly it's good to see devs not ceding to the internet rage mob. I remember before aws hired the main guy behind boto and I had to install 10 different tar.gz tools from 10 different aws teams and install all of them and setup 20 different env variables.
Running their installer isn't difficult and I've included it in our docker and ami builds easily. If you want a fixed version, download whatever from their repo and mirror it to your internal storage and you can always install from there and ensure you're running the exact version you like.
This response (from a year ago) seems perfectly reasonable and explains in detail a good number of reasons why they aren't using pip: https://github.com/aws/aws-cli/issues/4947#issuecomment-5860...
The first note in the issue was also straightforward and clearly reasoned. Aws-cli is a lot more than a Python script. It's used by hundreds of thousands if not millions of developers and admins. I completely understand their desire to avoid being caught up in the complexity of Python package management.
seems pretty reasonable, if you want an AWS python package that's distributed through pip, boto exists. but there's no reason for CLI tools that are not python libaries to be distributed through pip.
and as one of the commenters on the bug mentions, if you really want to install the aws cli through pip, that works because pip will happily install the git repo url.
and as one of the commenters on the bug mentions, if you really want to install the aws cli through pip, that works because pip will happily install the git repo url.
How about "that's what many users want"? Why is that not a reason?
Historically:
Microsoft: Prioritize what users demand. This leads to massive spaghetti systems which become unmaintainable in the long term.
Amazon: Talk to users, and do what users need. This seems to do okay.
Google: Do what developers think is right for users, having never interacted with a user, and convinced of the intrinsic superiority of Googlers.
The Google approach does okay for B2C, but doesn't work for B2B. The Microsoft and Amazon approaches do well in different niches.
Microsoft: Prioritize what users demand. This leads to massive spaghetti systems which become unmaintainable in the long term.
Amazon: Talk to users, and do what users need. This seems to do okay.
Google: Do what developers think is right for users, having never interacted with a user, and convinced of the intrinsic superiority of Googlers.
The Google approach does okay for B2C, but doesn't work for B2B. The Microsoft and Amazon approaches do well in different niches.
Users want lots of things. I want a free car with a Bluetooth stereo. That doesn’t mean it’s in anyone’s interest to provide it to me.
pip is not a general operating system package manager, it's a python package manager. despite many users wanting it, distributing random packages which may or may not be python-based is not what pip is for, and clearly not what the aws-cli team wants to use pip for.
just because a customer wants something, doesn't mean it's a reasonable request. i'm sure there are plenty of people who would also like to install the official aws-cli through cargo, gem, or npm.
just because a customer wants something, doesn't mean it's a reasonable request. i'm sure there are plenty of people who would also like to install the official aws-cli through cargo, gem, or npm.
I understand the argument that they might not want to stay on Python so they want to get people used to downloading from their zip instead of PyPi. But why not provide an rpm or deb? I don't understand why the only two official options are PyPi or downloading a zip and running the installer.
The last comment from Amazon before they locked the thread mentions they are putting together more packaging options.
After a year and a half, they claimed to be prioritizing this for Q2, which seemingly ended last month, and they still haven't offered anything.
Heck, there's people on the thread offering them build scripts that will create debs and rpms for them. It isn't that hard to do.
I can understand why they don't give a crap, though. The people responsible for setting up recipes to build and deploy a few thousand servers with AWS tooling on them that would sure like to be able to automate updates of the packages aren't the ones forcing their companies to use AWS, so Amazon prioritizes meeting the needs of the people making purchasing decisions rather than the people who actually have to use the tooling. With enough vendor lock-in and long term contracts already signed, they don't need to care about improving user experience to come up to par with system maintainer expectations from the early 90s.
Heck, there's people on the thread offering them build scripts that will create debs and rpms for them. It isn't that hard to do.
I can understand why they don't give a crap, though. The people responsible for setting up recipes to build and deploy a few thousand servers with AWS tooling on them that would sure like to be able to automate updates of the packages aren't the ones forcing their companies to use AWS, so Amazon prioritizes meeting the needs of the people making purchasing decisions rather than the people who actually have to use the tooling. With enough vendor lock-in and long term contracts already signed, they don't need to care about improving user experience to come up to par with system maintainer expectations from the early 90s.
@joguSD's second comment, preceding most of the discussion, also mentions that they're looking to get it into brew and linux package managers
You're right, I missed that, thanks.
[deleted]
Could unrelated third parties republish the code (unmodified or with injected malware) to PyPI without the permission of Amazon?
Not without forking it under a new name, assuming its license allows that.
It's comical to me how poor packaging solutions are across distributions and languages. It's all just namespace management at slightly different levels.
Can anyone give me one good reason a programming language should bother with dependency management?
Nix has the right idea, but the execution is spotty and poorly documented IMO. Not to mention they have a serious problem with package naming, which strikes me as a glaring issue.
Can anyone give me one good reason a programming language should bother with dependency management?
Nix has the right idea, but the execution is spotty and poorly documented IMO. Not to mention they have a serious problem with package naming, which strikes me as a glaring issue.
Can anyone give me one good reason a programming language should bother with dependency management?
When a programming language tries to ignore dependency management, like Go and Python did for too long, an ecosystem will soon spring up of a bunch of informal third-party tools for managing dependencies, all of which are only partially compatible with each other. It works better when the designers of a programming language include dependency management in the design and promote one official way to do it, like C# or Rust.
(I say this even though in general I prefer Go and Python to C# and Rust - you have to respect what the different tools do better.)
When a programming language tries to ignore dependency management, like Go and Python did for too long, an ecosystem will soon spring up of a bunch of informal third-party tools for managing dependencies, all of which are only partially compatible with each other. It works better when the designers of a programming language include dependency management in the design and promote one official way to do it, like C# or Rust.
(I say this even though in general I prefer Go and Python to C# and Rust - you have to respect what the different tools do better.)
> an ecosystem will soon spring up of a bunch of informal third-party tools for managing dependencies, all of which are only partially compatible with each other.
Start standardizing language agnostic interface, thus fixing the compatibility issue and I only see this as a good thing.
Start standardizing language agnostic interface, thus fixing the compatibility issue and I only see this as a good thing.
AppFS [0] solves this by getting rid of the install step, which prevents many problems.
[0] https://AppFS.net/
Disclaimers: I work for AWS, but not on the AWS CLI; AppFS is my own personal project
[0] https://AppFS.net/
Disclaimers: I work for AWS, but not on the AWS CLI; AppFS is my own personal project
I'll start.
A programming language with the ability to dispatch on version could be very interesting, and possibly allow for safer upgrades on a per function basis.
Still it feels like this should be part of an interface between a PL and some other set of packaging options. A convention for where to place libraries (with versions) like /lib is basically the solution. But I guess the various communities don't really want to get into a turf war, so maybe it's /lib/ruby and /lib/python in addition to the rest?
A programming language with the ability to dispatch on version could be very interesting, and possibly allow for safer upgrades on a per function basis.
Still it feels like this should be part of an interface between a PL and some other set of packaging options. A convention for where to place libraries (with versions) like /lib is basically the solution. But I guess the various communities don't really want to get into a turf war, so maybe it's /lib/ruby and /lib/python in addition to the rest?
Perl almost does this. Various CPAN packages have different "use" lines where they can declare themselves to be written in a particular version. These can all live together on an interpreter recent enough to know all the dialects. This is at the module level, rather than the function level though.
I'm imagining something like `[email protected](x)`. Of course, `foo(x)` would still work, but it would probably just choose the top version, like how things generally work today. Integrate with signed VCS and call it a day.
I very much empathize with the AWS team's decision here — pip's requirement for a single shared version for each library makes it a bad system-level package manager, since tools have limited control over the versions of their dependencies.
I'm curious why they even chose python to implement this — maybe it's sufficiently close to v1 of the tool?
I'm curious why they even chose python to implement this — maybe it's sufficiently close to v1 of the tool?
Python is an implementation detail.
Maybe they’ll rewrite to Rust one day, but maintain interface parity.
Maybe they’ll rewrite to Rust one day, but maintain interface parity.
It's unfortunate that it's not so uncommon for upstream software developers to express distaste for packages managers. While the reasons given aren't totally misguided in many cases, it quickly degrades user experience when each software has their own mechanism for installation.
I agree that this is a bothersome trend, but the use of pip to install system level packages is one of those user experience-degrading custom installation mechanisms. From that perspective, I don't have a problem with this refusal, I actually think it's a good thing, for the reason mentioned in the first comment:
> In the long term we want the fact that the AWS CLI is written in Python to be more of an implementation detail than a feature.
For Arch, there is a PKGBUILD for the AWS CLI v2 on AUR [1], which produces a package that can be installed, like any other, with the system package manager.
There seem to be some problems with packaging it for Fedora, but those issues look orthogonal to the decision not to publish on PyPI [2].
1: https://aur.archlinux.org/packages/aws-cli-v2-bin/
2: https://bugzilla.redhat.com/show_bug.cgi?id=1964128
> In the long term we want the fact that the AWS CLI is written in Python to be more of an implementation detail than a feature.
For Arch, there is a PKGBUILD for the AWS CLI v2 on AUR [1], which produces a package that can be installed, like any other, with the system package manager.
There seem to be some problems with packaging it for Fedora, but those issues look orthogonal to the decision not to publish on PyPI [2].
1: https://aur.archlinux.org/packages/aws-cli-v2-bin/
2: https://bugzilla.redhat.com/show_bug.cgi?id=1964128
[deleted]
Starting in April, we’re going to begin posting monthly roadmap updates as issues on our GitHub repository. These will lay out what we’re actively working on and what’s on the horizon. Our backlog includes a mix of priorities, some driven by the CLI team and the community, some which align with larger AWS goals. Our focus will be on sharing the priorities emerging from our team, which will include Github issues we are planning to work on. We will iterate on the exact format and approach for these updates as we determine what works best, but they will be at least monthly for the rest of 2021.
The writing style here just oozes AWS manager. This was probably ripped straight out of a business report. Its not surprising that this manager has posted only 1 update in May. https://github.com/aws/aws-cli/issues/6186 . The update ends with basically we fight fires first and dev second, there are always more fires, and we are hiring. Not so much of a status update as a plea for help.
The writing style here just oozes AWS manager. This was probably ripped straight out of a business report. Its not surprising that this manager has posted only 1 update in May. https://github.com/aws/aws-cli/issues/6186 . The update ends with basically we fight fires first and dev second, there are always more fires, and we are hiring. Not so much of a status update as a plea for help.
> Those sound as decent and understandable reasons — except perhaps the #1. I'd avoid going into language comparison contests here — but it's hard to refute that Python is among the few must know languages for DevOps folks and the like.
- [taken from the GitHub thread]
I understand where the poster is coming from but I think it’s hard to overstate the reach and compatibility needs of the AWS CLI. Also, of course, times change and Python man be supplanted by another scripting language.
Anyway, I think the AWS team is making the right call though I am a big package manager fan.
- [taken from the GitHub thread]
I understand where the poster is coming from but I think it’s hard to overstate the reach and compatibility needs of the AWS CLI. Also, of course, times change and Python man be supplanted by another scripting language.
Anyway, I think the AWS team is making the right call though I am a big package manager fan.
Anyway, i can just
and keep moving. Whatever.