This false sense of reproducability is why I funded https://docs.stablebuild.com/ some years ago. It lets you pin stuff in dockerfiles that are normally unpinnable like OS package repos, docker hub tags and random files on the internet. So you can go back to a project a year from now and actually get the same container back again.
The removal (or moving) of the Bitnami images from Docker Hub is going to break a ton of systems that depend on them. I helped set up https://www.stablebuild.com/ some years ago to counter these types of issues, it provides (among other things) a transparent cache to Docker Hub which automatically caches image tags and makes them immutable - underlying tag might be deleted or modified, but you’ll get the exact same original image back.
FYI, I've helped set up StableBuild (https://www.stablebuild.com) to help pin stuff in Docker that's normally virtually impossible to pin (e.g. OS package repos, Docker base images, random files from the internet, etc.)
Not never. E.g. all the capital we as founders put in the business before we raised our seed round was converted into Series Seed Preferred shares at the same rights as angels / seed VC. Small portion of total equity but still.
Packages and versions can be deleted from PyPI, which can be a massive pain in the ass for anyone consuming these packages. Can have your whole Python dependency tree pinned => author pulls a package version => builds broken. As part of StableBuild (https://www.stablebuild.com) we create full daily snapshots of the PyPI registry - so figured it would be nice to make an overview of deleted packages/versions and make the wheels available for download.
Yes, fantastic work. Downside is that snapshot.debian.org is extremely slow, times out / errors out regularly - very annoying. See also e.g. https://github.com/spesmilo/electrum/issues/8496 for complaints (but it's pretty apparent once you integrate this in your builds).
Until someone overwrites or deletes the Docker base image (regularly happens), or when you depend on some packages installed through apt - as you'll get the latest version (impossible to pin those).
Yeah, but it's impossible to properly pin w/o running your own mirrors. Anything you install via apt is unpinnable, as old versions get removed when a new version is released; pinning multi-arch Docker base images is impossible because you can only pin on a tag which is not immutable (pinning on hashes is architecture dependent); Docker base images might get deleted (e.g. nvidia-cuda base images); pinning Python dependencies, even with a tool like Poetry is impossible, because people delete packages / versions from PyPI (e.g. jaxlib 0.4.1 this week); GitHub repos get deleted; the list goes on. So you need to mirror every dependency.
Reading through the comments in https://news.ycombinator.com/item?id=39720007 I saw a common misconception pop up again: Dockerfiles are not deterministic. But they _look_ like they are, and even are for a while: Build a Dockerfile on your local machine; then build it again => most likely exactly the same container. Stuff starts to break down quickly though; so I did this writeup some time ago that should be informative for the wider community.
That's the interesting bit about Dockerfiles. They look _looks_ deterministic, and they even are for a while while you're looking at it as a developer. I've done a detailed writeup of how it's not deterministic in https://docs.stablebuild.com/why-stablebuild
I've gone down the same path. I love deterministic builds, and I think Docker's biggest fault is that to the average developer a Dockerfile _looks_ deterministic - and it even is for a while (build a container twice in a row on the same machine => same output), but then packages get updated in the package manager, base images get updated w/ the same tag, and when you rebuild a month later you get something completely different. Do that times 40 (the number of containers my team manages) and now fixing containers is a significant part of your job.
So in theory Nix would be perfect. But it's not, because it's so different. Get a tool from a vendor => won't work on Nix. Get an error => impossible to quickly find a solution on the web.
Anyway, out of that frustration I've funded https://www.stablebuild.com. Deterministic builds w/ Docker, but with containers built on Ubuntu, Debian or Alpine. Currently consists of an immutable Docker Hub pull-through cache, full daily copies of the Ubuntu/Debian/Alpine package registries, full daily copies of most popular PPAs, daily copies of the PyPI index (we do a lot of ML), and arbitrary immutable file/URL cache.
So far it's been the best of both worlds in my day job: easy to write, easy to debug, wide software compatibility, and we have seen 0 issues due to non-determinism in containers that we moved over to StableBuild in my day job.
Not sure about Earthly as I’ve never used it, but for Nexus and other artifactory solutions around package repos:
1. Don’t need to set up anything. No need to set up an apt repo and push packages or configure a mirror, because StableBuild already caches the complete registry and thus has everything.
2. Don’t need to think about the complete package list when pushing files to artifactory. Have packages cached from 3 months ago, now want to add another one? Oops, not in Nexus, and the current versions in the Ubuntu package registry are not compatible with your cached versions -> need to update the full dependency tree.
3. Integration is trivial. Three lines to your Dockerfile and done.
4. Can retroactively fix things. Knew that this container built 4 weeks ago? Ok use that as a pin date -> fixed.
Then there’s some stuff, like the immutable Docker pullthrough cache and history pypi mirror that I haven’t seen before (but I’d like to learn if others are doing this :-)).
Things Edge Impulse customers have in production: Sleep stage prediction, fall detection for elderly, fire detection in power lines, voice command recognition on headsets, predicting heath exhaustion for first responders, pet feeders that recognize animals, activity trackers for pets, and many more.