Preventing Kubernetes from pulling the pause image from the internet(kyle.cascade.family)
kyle.cascade.family
Preventing Kubernetes from pulling the pause image from the internet
https://kyle.cascade.family/posts/preventing-kubernetes-from-pulling-the-pause-image-from-the-internet/
53 comments
Relying on an hosted image also caused some disruptions for Nomad (the scheduler from Hashicorp), because the default pause image was hosted at gcr.io which google killed, and it moved to registry.k8s.io.
The nomad team made this configurable afterwards.
The nomad team made this configurable afterwards.
That nomad was hit with this after years of notice and deprecation extension, seems a sign of serious maintenance issues
It's implementation of cri plugin.
> This should be part of the containerd distribution
containerd is not the only CRI runtime out there.
> This should be part of the containerd distribution
containerd is not the only CRI runtime out there.
> It's implementation of cri plugin.
Right, that’s the point. A user of the CRI should not have to care about this implementation detail.
> containerd is not the only CRI runtime out there.
Any CRI that needs a pause executable should come with one.
Right, that’s the point. A user of the CRI should not have to care about this implementation detail.
> containerd is not the only CRI runtime out there.
Any CRI that needs a pause executable should come with one.
> A user of the CRI should not have to care about this implementation detail.
This detail is implemented by K8s, not a container runtime. User has to care about the source of it due to supply chain attacks, though.
> Any CRI that needs a pause executable should come with one.
Which would introduce a simultaneous update across different projects, which is a problem harder than a line of config.
This detail is implemented by K8s, not a container runtime. User has to care about the source of it due to supply chain attacks, though.
> Any CRI that needs a pause executable should come with one.
Which would introduce a simultaneous update across different projects, which is a problem harder than a line of config.
More general one would wish that Kubernetes had a few extra ways to get images, so you could grow on a scale from "minimal infrastructure" to "fully CI/CD". Starting with just sending the image in the RPC itself or even just on local disk (you figure out how to get it there), all the way up to registries with tightly controlled versioning.
It's possible to do that, as kubernetes only passes the image information to CRI.
You can also setup a separate service to "push" images directly to your container runtime, someone even demoed one in Show HN post some time ago I think.
You can also setup a separate service to "push" images directly to your container runtime, someone even demoed one in Show HN post some time ago I think.
It's almost as if Docker (the company) wants it this way.
Just to save someone 5 minutes of research, if you are using the EKS AMIs based on AL2023 or Bottlerocket, this is already done for you by pointing to an image on ECR. At least on Bottlerocket, I haven't checked AL2023, the image is baked into the AMI so you don't even need to pull it from ECR.
We removed the image registry dependency on AL2023 as well. :)
https://github.com/awslabs/amazon-eks-ami/pull/2000
https://github.com/awslabs/amazon-eks-ami/pull/2000
Thank you, was just about to task my team with figuring out how affected we are by this.
I went down this rabbit hole not so long ago too.
There was a discussion open on containerd's GitHub on removing the dependency on the pause image but it has been closed as won't fix: https://github.com/containerd/containerd/issues/10505
Also, if you are using kubeadm to create your cluster, beware that kubeadm may be pre-pulling a different pause image if it does not match your containerd configuration: https://github.com/kubernetes/kubeadm/issues/2020
There was a discussion open on containerd's GitHub on removing the dependency on the pause image but it has been closed as won't fix: https://github.com/containerd/containerd/issues/10505
Also, if you are using kubeadm to create your cluster, beware that kubeadm may be pre-pulling a different pause image if it does not match your containerd configuration: https://github.com/kubernetes/kubeadm/issues/2020
O/T, but I'm getting a cert error on this page - wonder if it's just me or if this site is just serving a weird cert. Looks like it's signed by some Fortinet appliance - maybe I'm getting MITMed? Would be kind of exciting/frightening if so.
EDIT: I loaded the page from a cloud box, and wow, I'm getting MITMed! Seems to only be for this site, wonder if it's some kind of sensitivity to the .family TLD.
EDIT: I loaded the page from a cloud box, and wow, I'm getting MITMed! Seems to only be for this site, wonder if it's some kind of sensitivity to the .family TLD.
Ooft. If it helps, this is the PEM I'm getting. LetEncrypt signed.
-----BEGIN CERTIFICATE-----
MIIFAjCCA+qgAwIBAgISBZR6PR4jNhx4fBFvqKwzJWx4MA0GCSqGSIb3DQEBCwUA
MDMxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQwwCgYDVQQD
EwNSMTMwHhcNMjUwOTE4MTM1OTEwWhcNMjUxMjE3MTM1OTA5WjAeMRwwGgYDVQQD
ExNreWxlLmNhc2NhZGUuZmFtaWx5MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB
CgKCAQEA55JknkVzyq5QGaRXn2TAzaOGYTHUVxl89lGOFgEEaWEvH5pcZL7xkqfv
Edee7l5MeRKuK1zJ+ISPQQaEjGTk51y1aXXfOKs62NiNy6QQUbzQ+euecqrKsJVN
l3PC3EYlEGibKI1gZ2x/ht8WJU9o4KiswCLqHrY7nC7BeEByv/ehiYyRTTxAXJsr
2X4LgPX6MQ1Iu10S2Bp9jnOlEV7n4RCTPFeWtfQ0CdXH45ykuwL/zrTaD111oNQE
BQPNq7Ig7OihLZcJQo8TMJ3FUgzDI9z6kMy7QHNR1I8uODVUohQCO6E7A29x8nRJ
UBV5DN1as3aHYFJ4FbX9s2tuLwCTiwIDAQABo4ICIzCCAh8wDgYDVR0PAQH/BAQD
AgWgMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAA
MB0GA1UdDgQWBBTXwJ21Mudr9rplbA970jxJk44pEDAfBgNVHSMEGDAWgBTnq58P
LDOgU9NeT3jIsoQOO9aSMzAzBggrBgEFBQcBAQQnMCUwIwYIKwYBBQUHMAKGF2h0
dHA6Ly9yMTMuaS5sZW5jci5vcmcvMB4GA1UdEQQXMBWCE2t5bGUuY2FzY2FkZS5m
YW1pbHkwEwYDVR0gBAwwCjAIBgZngQwBAgEwLwYDVR0fBCgwJjAkoCKgIIYeaHR0
cDovL3IxMy5jLmxlbmNyLm9yZy8xMjEuY3JsMIIBAwYKKwYBBAHWeQIEAgSB9ASB
8QDvAHYApELFBklgYVSPD9TqnPt6LSZFTYepfy/fRVn2J086hFQAAAGZXVTEhwAA
BAMARzBFAiAVfJZ/XSqNq0sdf49o/8Mhs1uG9H/iPAHynYubtxfw4wIhAPiDa5S5
DoawcZlWePa+uKZRiIaZwlVVOigiZEfm+75VAHUAzPsPaoVxCWX+lZtTzumyfCLp
hVwNl422qX5UwP5MDbAAAAGZXVTEmAAABAMARjBEAiAJTtUg1SkZlRsuvXiWbeon
ehJiRiOvQBBjCrDhPk+EmAIgRy7+96Uq7sFF2iQqlDbBJTbfxqVxsLAKKsv/4mUQ
76gwDQYJKoZIhvcNAQELBQADggEBADwJpGkcEI9YQJBcBqJ1k4lkUBI4zdhjYYuv
Z2kbXFRkl041ulyel48qZZW20T9myTL4dI/2kqLP4VSrz+vk4xTzyXtTCJZHDeke
dzoQ7lplxTfZRwDVx19PkJIFPIShHyS/Ia0XTqHC8F81PmwXULRAgMYrBS3sKLXg
aIyf00xq7W6s0uPd0XDn5CsmJgHzEcBZ0F423V42iedwgGNv6GnlgzKP3Q8fkf21
4KdRYBgyYBfi33jQFf5fuMuSTtFak++BYe/ZWVAoehlw0gLh5BBmBXtCFrVFZc+q
uXXe4q5MVQmDRa0A+QtKbwkyZxIiwJ8Xi+eBTKQSscpdINy5bUs=
-----END CERTIFICATE-----I've used k8s before a lot and at several companies. I am convinced that 99.9% of the people who use it should not be. But it's more fun than deploying VM images at least.
I'm running k3s at home on single node with local storage. Few blogs, forum, minIO.
Very easy, reliable.
Without k3s I would have use Docker, but k3s really adds important features: easier to manage network, more declarative configuration, bundled Traefik...
So, I'm convinced that quite a few people can happily and efficiently use k8s.
In the past I used other k8s distro (Harvester) which was much more complicated to use and fragile to maintain.
Very easy, reliable.
Without k3s I would have use Docker, but k3s really adds important features: easier to manage network, more declarative configuration, bundled Traefik...
So, I'm convinced that quite a few people can happily and efficiently use k8s.
In the past I used other k8s distro (Harvester) which was much more complicated to use and fragile to maintain.
Check out Talos Linux if you haven't already, it's pretty cool (if you want k8s).
I tried Talos few month ago. Found it unstable and complicated; reported few bugs.
And because they are "immutable" - I found it's significantly more complicated to use with no tangible benefits. I do not want to learn and deal declarative machine configs, learn how to create custom images with GPU drivers...
Quite a few things which I get done on Ubuntu / Debian under 60 seconds - takes me half an hour to figure out with Talos.
And because they are "immutable" - I found it's significantly more complicated to use with no tangible benefits. I do not want to learn and deal declarative machine configs, learn how to create custom images with GPU drivers...
Quite a few things which I get done on Ubuntu / Debian under 60 seconds - takes me half an hour to figure out with Talos.
Learning new things takes time.
It sounds like an immutable kubernetes distro doesn't solve any problems for you.
It sounds like an immutable kubernetes distro doesn't solve any problems for you.
I use k3s for my home and for dev envs I think it's completely fine especially when it comes to deployment documentation.
I am way more comfortable managing a system that is k3s rather than something that is still using tmux that gets wiped every reboot.
Well... it's what I would have said until bitnami pulled the rug and pretty much ruined the entire ecosystem as now you don't have a way to pull something that you know is trusted with similar configuration and all from a single repository which makes deployments a pain in the ass.
However, on the plus side I've just been creating my own every time I need one with the help of claude using bitnami as reference and honestly it doesn't take that much more time and keeping them up to date is relatively easy as well with ci automations.
I am way more comfortable managing a system that is k3s rather than something that is still using tmux that gets wiped every reboot.
Well... it's what I would have said until bitnami pulled the rug and pretty much ruined the entire ecosystem as now you don't have a way to pull something that you know is trusted with similar configuration and all from a single repository which makes deployments a pain in the ass.
However, on the plus side I've just been creating my own every time I need one with the help of claude using bitnami as reference and honestly it doesn't take that much more time and keeping them up to date is relatively easy as well with ci automations.
The situation with bitnami is getting fixed, but it takes time for all the holes to be plugged.
I knew bitnami were trouble when I saw their paid tier prices. Relevant article: https://devoriales.com/post/402/from-free-to-fee-how-broadco...
Oh, and it's owned by Broadcom.
I knew bitnami were trouble when I saw their paid tier prices. Relevant article: https://devoriales.com/post/402/from-free-to-fee-how-broadco...
Oh, and it's owned by Broadcom.
> I am way more comfortable managing a system that is k3s rather than something that is still using tmux that gets wiped every reboot.
Thoughts on Tmux-resurrect[1] , it can even resurrect programs running inside of it as well. It feels like it can as such reduce complexity from something like k3s back to tmux. What are your thoughts on it?
[1]:https://github.com/tmux-plugins/tmux-resurrect?tab=readme-ov...
Thoughts on Tmux-resurrect[1] , it can even resurrect programs running inside of it as well. It feels like it can as such reduce complexity from something like k3s back to tmux. What are your thoughts on it?
[1]:https://github.com/tmux-plugins/tmux-resurrect?tab=readme-ov...
I had it break enough times to where I just don't bother.
Well firstly I would love to know more about your workflow where it actually broke etc. because I feel like tmux-ressurect team could help or something for sure.
I haven't used the tool itself so I am curious as I was thinking of a similar workflow as well sometime ago
Now please answer the above questions but also I am going to assume that you are right about tmux-ressurect, even then there are other ways of doing the same thing as well.
https://www.baeldung.com/linux/process-save-restore
This mentions either Criu if you want a process to persist after a shutdown, or the shutdown utility's flags if you want to temporarily do it.
I have played around with Criu and docker, docker can even use criu with things like docker checkpoint and I have played with that as well (I used it to shutdown mid compression of a large file and recontinue compression exactly from where I left)
What are your thoughts on using criu+docker/criu + termux, I think that it itself might be an easier thing than k3s for your workflow.
Plus, I have seen some people mention vps where they are running the processes for 300 days or even more without a single shutdown iirc and I feel like modern VPS providers are insanely good at uptime, even more so than sometimes cloud providers.
I haven't used the tool itself so I am curious as I was thinking of a similar workflow as well sometime ago
Now please answer the above questions but also I am going to assume that you are right about tmux-ressurect, even then there are other ways of doing the same thing as well.
https://www.baeldung.com/linux/process-save-restore
This mentions either Criu if you want a process to persist after a shutdown, or the shutdown utility's flags if you want to temporarily do it.
I have played around with Criu and docker, docker can even use criu with things like docker checkpoint and I have played with that as well (I used it to shutdown mid compression of a large file and recontinue compression exactly from where I left)
What are your thoughts on using criu+docker/criu + termux, I think that it itself might be an easier thing than k3s for your workflow.
Plus, I have seen some people mention vps where they are running the processes for 300 days or even more without a single shutdown iirc and I feel like modern VPS providers are insanely good at uptime, even more so than sometimes cloud providers.
failure scales exponentially with servers due to design limitations
even using tmux resurrect on my personal machine I've had it fail to resurrect anything
again - lack of documentation and loosy tmux resurrect state is not what I want to go thru when working in unfamilar environments
why are you getting downvoted
docker compose also has issues but at least it is defined, again if you are managing 10+ machines docker becomes a challenge to maintain especially when you have 4 to 5 clusters, when you are familiar with kubernetes there's virtually no difference between docker tmux or raw k8s, although I heavily recommend k3s due to its ability to maintain itself.
even using tmux resurrect on my personal machine I've had it fail to resurrect anything
again - lack of documentation and loosy tmux resurrect state is not what I want to go thru when working in unfamilar environments
why are you getting downvoted
docker compose also has issues but at least it is defined, again if you are managing 10+ machines docker becomes a challenge to maintain especially when you have 4 to 5 clusters, when you are familiar with kubernetes there's virtually no difference between docker tmux or raw k8s, although I heavily recommend k3s due to its ability to maintain itself.
Skill issue. It works just fine
Same here, I went through a few projects since 2021 where doing Kubernetes setups were part of my role on the consulting project, and I would say prefer managed containers solutions, e.g. Azure Web Apps, or when running locally plain systemd or Docker Compose.
Anything else, most companies aren't Web scale enough to set their full Kubernetes clusters with failover regions from scratch.
Anything else, most companies aren't Web scale enough to set their full Kubernetes clusters with failover regions from scratch.
I like Docker(compose) + Portainer for small deployments
I believe this has been patched time and time again in on-premises variants like OpenShift. Curious to check if it’s there in small variants like microk8s, k3s, etc., as I’m considering moving a few offline services to Talos.
Talos' KubeSpan is backed by Sidero-hosted disovery service that cannot be self-hosted without a commercial license
Hmmm. So I need to find something equivalent as a base OS. Maybe it's time to do Fedora Core.
Flatcar Linux has looked decent
Nice to know, though I wonder how many companies are really using all private images? I've certainly had a client running their own Harbor instance, but almost all others pulled from Docker Hub or Github (ghcr.io).
Lots of medical and governmental organisations are not allowed to run in public cloud environments. It's part of my job to help them get set up. However, in reality that often boils down to devs wining about adding a registry to Harbor to cache; nobody is going to recompile base images and read through millions of lines of third party code.
A lot of security is posturing and posing to legally cover your ass by following an almost arbitrary set of regulations. In practice, most end up running the same code as the rest of us anyway. People need to get stuff done.
A lot of security is posturing and posing to legally cover your ass by following an almost arbitrary set of regulations. In practice, most end up running the same code as the rest of us anyway. People need to get stuff done.
Pretty much all enterprises are using their own ECR/GCR/ACR.
I work on the container registry team at my current company running a custom container registry service!
How does this require a whole team? Unless you're working at a hyperscaler
Not a hyperscaler, but we’re multi-cloud and probably one to two steps down.
My team’s service implements a number of performance and functionality improvements on top of your typical registry to support the company’s needs.
I can’t say much more than that sadly.
My team’s service implements a number of performance and functionality improvements on top of your typical registry to support the company’s needs.
I can’t say much more than that sadly.
Please describe their system for us, including system throughput, the hardware they're on, networking constraints, and how many people are allowed to be needed to operate it.
Maybe they work for docker
[deleted]
The Public Sector and anyone concerned with compliance under the Cyber Resilience Act should really use their own private image store. Some do, some don't.
I think once your eng org > 300 people and you have a dedicated infra and security team, it's going to be on their radar to do at some point.
Some people run Artifactory as a cache in front of Docker Hub etc, which allows some governance
Yeah that pause image was really annoying when I was hosting a k8s cluster on Hetzner, since the `registry.k8s.io` -registry was blocking some Hetzner IPs, since its hosted on Google.
This is insane, am I the only one being shocked to learn about this? This reeks of bad engineering, is there at least a plan to make this go away by embedding "pause" with the distribution?
Easy. Dont use kubernetes. You'll thank me later.
Instead of just swapping out the registry, try baking it into your machine image.