Fun fact: some older articles were originally written using the term man-in-the-middle, but at some point were updated... except that the diagrams still use man-in-the-middle because search-and-replace doesn't work on images.
Controlling both the builder and the registry is super nice btw, because they can work together. If the builder knows some of the layers already exist in the registry, it merely has to create and push the remaining ones (without downloading any of the other layers, not even those from the base image). That gives you near-instant builds once the biggest layers are cached in the registry!
Since builds usually happen in CI, and pulls happen elsewhere (e.g., a kubernetes node), in the end layers are only downloaded when the resulting container image is actually used.
I can't comment on BuildKit, unfortunately, since I haven't used it. My experience comes from building bespoke systems (an image builder and a custom registry) fully from scratch, because we needed to have full control in order to achieve the performance we were aiming at.
I'd say the limitations are relative to your use case: `docker build` works well enough in most cases, but sometimes it's not the right tool for the job (e.g., you cannot use `docker build` inside an AWS Lambda, you can get way better performance from specialized tools in some cases [1], etc)
The idea is that, in many cases, you can create a layer "by hand" without running actual Linux programs. Layers don't need to be pre-existing, the only requirement is that they can be built programmatically (inside the browser, in this case). The demo actually does that: it "manually" creates a layer from the user-specified entrypoint script, then creates an image from the pre-existing base image's layers and the new entrypoint layer.
In a more real scenario, you can e.g., turn pip wheels into layers without actually using docker's RUN command. All it takes is to massage the data from one archive format into another, programmatically. This unlocks lots of potential (e.g., it becomes embarrassingly parallel to build a container image comprised of pip wheels). Combine that with a good layer caching strategy and a registry that takes advantage of it, and you can have near-instant container builds for arbitrary sets of pip dependencies.
I sometimes wonder why there aren't more bespoke container tools (like yours). Would people be willing to pay for stuff like what you have built, if someone took the time to "productionize" it? Or is there no market?
There are many ways to achieve this. If you are just copying artifacts, and not using RUN, then there is in principle no need to download the base image at all. Feel free to message me if you'd like to chat (email is in my profile).
Out of curiosity, is there a build setup you have seen in the past that you think could be a good replacement for this complex GitHub CI setup? Asking for a friend ;)
Update: now I've finished reading the article, my impression is that complexity is mostly inherent to this problem space. I'd be glad to be proven wrong, though!
Thanks! It's always scary when container technology fiddles with your network... I wonder how they can be at peace with such defaults. Another reason to stick with good old containerd units, I guess.
Could you share more details about this? Do you mean that e.g., if I run LXD/Incus on a machine with a public IP address, anyone on the internet could route traffic through it?
Many QUIC features get implemented while in draft stage as people iterate on the RFC's design (e.g., ACK Frequency[0] is currently on draft version 14 and I implemented support for it three years ago in quinn).
Thanks for posting this! As a long-time Rust user (and contributor, in the good old days), the thing that has always fascinated me about Rust is the healthy balance it strikes between academic brilliance and industry pragmatism. Radical changes like the ones suggested by the OP risk damaging that balance IMO. I'd rather put up with some language quirks and see Rust achieve "boring technology" status...
But who knows, maybe the "academic brilliance" from the article is more pragmatic than I give it credit for. I sure hope for it if these changes ever go through.