Yes, compression being part of the OCI image's digest was (in hindsight) a poor decision. _Technically_ OCI images allow uncompressed layers, and the layers could be included without compression (and transport compression to be used); this would allow layers to be fully reproducible. We explored some options to do this (and made some preparations; https://github.com/containerd/containerd/pull/8166), but also discovered that various implementations of registry clients didn't handle transport-compression correctly (https://github.com/distribution/distribution/pull/3754), which could result in client either pulling the full, uncompressed, content, or image validation failing.
I'm a maintainer of the Moby project (which is used to build the Docker Engine), and saw this post got some attention, so let me try to outline some of the changes and motivation. Happy to answer questions if there's any.
First of all, some history; the Docker Engine was a monolith daemon that provided many services; this worked well when using Docker as a standalone solution, but when used as runtime for Kubernetes, this wasn't ideal; many components were not designed for this purpose, which meant they had to be replaced / overridden with hacks to make it work. The containerd project was created to provide a more modular runtime for the container ecosystem, providing separate subcomponents (a containerd runtime, image/content storage) for the container ecosystem to build on. It was created "from scratch" with lessons learned over the Years, providing a modern foundation.
While docker has used containerd as a runtime for many Years, it still used its own implementation for storing images ("graph-drivers"); this implementation started to show its age and had many limitations; graph-drivers have no native support for multi-platform ("multi-arch") images, no support for OCI Artifacts, and no reproducible images when pushing to different registries (among others).
Around 4 Years ago, we started to re-implement the image storage using containerd "snapshotters"; our initial goal was to provide a mostly seamless transition; add multi-arch support, but keep the UX as close as possible to the graph-drivers. Around 2 Years ago, Docker Desktop changed to using the containerd image storage (snapshotters) as a default for new installations, and Docker v29 made it the default for Linux installations.
While we kept most of the UX similar, there are some differences; when storing an image with graph-drivers, docker would pull the OCI image, extract the content (layers), and discard the (compressed) layers. While this reduced storage, it also made images non-reproducible as the image had to be re-constructed when pushing to a registry (which also resulted in slower pushes).
The containerd image storage uses a different design, where a copy of the compressed artifacts are preserved (by default); this requires more storage to keep these extra blobs, but reduces duplication and increases push performance. It was the decision containerd maintainers made early in their design process, and all containerd-based tools have used this model since the start of the containerd project.
We have a couple of roadmap items to improve this in future; some are outlined in this ticket; https://github.com/moby/moby/issues/51581, but there's other options that will become availeble through the containerd image store; support for erofs as an alternative to (tar) compressed image layers, as well as automatic garbage-collection (which would reduce the need for manually pruning content through `docker system prune` (and related commands).
I wrote that comment at the time; to add some context to that comment: removal of the "login to download" was shortly after the "Docker Enterprise" business went to Mirantis. While Docker started as a developer centric tool, focus shifted to Enterprise products (Docker Enterprise Engine, UCP, Docker Trusted Registry, Docker Desktop Enterprise). After the move of the enterprise products to Mirantis, Docker's focus went back to developer products.
> I'm aware of runC but don't know if Docker images are realistically portabl
(It wasn't clear from your comment if you were aware of this)
Since last year (docker 1.11) docker itself no longer is a runtime, and uses runC as the default runtime (https://blog.docker.com/2016/04/docker-engine-1-11-runc/)
The quarterly ("stable channel") CE releases (17.03, 17.06 and so on) are supported for 4 months, and will not get new features during that period. EE quarterly releases have a 1 year support period, and also won't get new features.
During the support period, bug fixes will get back ported to those versions and released as "patch" releases (e.g. 17.03.1).
When installing, you can choose to install either the "stable" (quarterly) channel, or the "edge" (monthly) channel.
Thanks so much for writing this up. Happy to help getting those shirts out. Let me know if there's any way I can help out, with the shirts, or any other way. This just has to stop.
Not disputing that the "verified" message is misleading, but the comment you referred to was about "not yet being able to create a signed image".
I don't think there's anything yet in the docs with regard to signed image, apart for the release notes[1] mentioning it being a "sneak peak" of a coming feature, that is under development
> because the command is "docker run", with additional settings
The `-m` flag was already present in docker run. Swarm makes use of that flag to decide on which host to put a container. The `constraint:` setting is indeed new, but (afaik) the values behind it are not pre-defined, so if someone develops an alternative clustering-backend they should be able to make use of that for their needs.
In any case, Swarm will be pluggable, so other back ends are possible and can be developed by other parties.
> confused over how many components and services you have to contend with now
You don't have to. It really depends on what you're trying to do. Using Docker alone, you'll be able to build and run containers, link them together to build a "stack" etc.
If you want to make building a stack (a group of containers that together form your application) easier, you can use an orchestration tool to automate this, for example, Fig, Crane, or now Compose. Or, create a bash script to do this; it's up to you.
If you want to build a cluster (run your containers distributed over several servers), you can do that with docker alone, but it will get hard to manage. You can build a tool for that (making use of the Docker API), or use an existing tool, like Flocker, Shipyard or now Swarm.
So if all you need is running a few containers on a single host, Docker alone may be enough for you, in that case you can safely ignore the other stuff for now.
+1 if debugging is present out-of-the-box, even first-time users will have no excuse not to use proper debugging.
While installing XDebug may be no rocket-science, I've encountered too many situations that it just didn't work reliably, causing developers to revert to nasty var_dumps() for debugging.
I agree completely, maybe PHP should use the same approach as the jQuery migration-plugin, which;
- Logs usage of old or (to-be) deprecated functionality
- Provides deprecated functionality so that existing code doesn't break
This way;
- The PHP API can finally be cleaned up (please, also move those old functions to a separate section in the documentation)
- Older code can still run, but enabling the 'migration/compatibility' plug-in (should be possible to enable/disable this per site, not for the whole server at once)
- Developers are informed that their code uses deprecated functionality, giving them time to migrate code.
Although this approach is comparable to the E_STRICT flag, it is going one step further, in that the deprecated functions are actually removed from the core, and must be manually enabled/added via an optional plugin.
Reorganising the documentation is an important part of this; cleaning up the old stuff will make PHP less confusing (should I use 'implode()' or 'join()'?), and will silently guide new developers away from the old stuff.
On a final note; if this approach allows hosting-providers to upgrade PHP without breaking their client's code, adoption rate of newer PHP versions will probably be faster as well!