The point that I wanted to make about systemd is that in practice I'm again and again running into systemd vs. containers problems. Systemd isn't really intended to run in containers. There are workarounds (such as sharing special files from the host with the container or using Podman that has implemented similar workarounds for it). The reason is that on modern "fat" Linux distributions (like RHEL/CentOS or Ubuntu) more and more things depend on systemd. Postfix is just a simple example. Because of this clash the original promise of containers (package any user mode stuff into a container and run in anywhere) often feels broken. This isn't Podmans fault (Podman is doing what it can to ease the pain, but it only works when you use Podman). A container image that contains systemd needs special treating (not possible on any host or platform that can run containers) and isn't really a portable container anymore.
These kind of issues tend to let me avoid using CentOS/UBI as container base image and instead go with i.e. Alpine that doesn't have the things I like about EL (LTS, etc) but also never has any mixups with systemd. (It's a bit sad having to go that road when your workplace actually pays for EL support).
I wouldn't have any suggestion how to easily fix this though. Systemd will never fully run in a container I suppose (it's too much linked to non userspace) nor will all container tools and platforms support workarounds for it.
Aside from systemd here's another example where I think Podman UX could be better. Running containers in containers is an important usecase, even if it's insecure. Docker has provided workarounds for this early on. Yes, horrible in terms of security. But good enough for CI/CD runners. Either you can pass the Docker sock file from host to guest. Or in a --privileged container you can run a Docker engine and connect to it from another Docker container. It's well documented and there is a ready "docker:dind" (docker in docker) container image.
My #1 hope with Podman rootless containers was that I could just run "podman run ..." inside a podman container. But it turns out it's (sill) not that easy, even with "--storage-driver=vfs".
What's the recommended approach for running containers in containers with a version of Podman that is available today in the stable version of RHEL/CentOS? I lack to find docs about it. I assume you can do some unprivileged foo with Podman, too ... but how?
(I know latest Podman2 is providing a largely Docker compatible sock file, but I understand its early? Don't misunderstand me, this is great, will use it once it lands in RHEL. But until then ... Docker for this?).
thanks for the response and the links! I will certainly have a look.
(And again sorry for my rather harsh words - I have a lot of respect for all the hard work everyone at Red Hat is doing. I can only imagine that it's much harder to make technical decisions and build tools like Podman than it looks like. Thank you for not taking my rant as an offence! It's just my brutally raw user experience).
and yes, people have proven that it's possible to build something like podman or buildah on top of containerd / buildkit! No need to reinvent the wheel!
I'm an enduser and I expect things to "just work". That seems to match a lot with Red Hat's Linux distros (i.e. RHEL): "Just use it, it's a defacto standard so most third party software is tested on it, don't worry about underlying components as Red Hat has made sane choices for you."
That's the theory (and Red Hat's marketing). I wish I could just be using CentOS/RHEL and not worry about the choices Red Hat makes. But the shit is piling up (sorry for my harsh words, no offence intended).
So, let's containerise something simple on top of the "centos" container image, let's say postfix. Oh, postfix on centos needs systemd for logging. But systemd in a container is a nightmare. You can get systemd to work in a container if the host system is using containerd and you pass certain special files from the host to the container. So just use Podman instead of Docker, right? Podman has functionality to make systemd in a container work. OK, I can run my container on my fat workstation (a Centos system with Podman). But is it still portable? Can I just run it in Github Actions, AWS ECS, a Windows machine with Docker Desktop? Nope. It's not portable.
Let's put the systemd rant aside (I really like the systemd CLI UX, but the architecture (dbus, etc) seems to only serve one usecase properly (Desktop systems) and seems to be heavily wrong for the container usecase).
Let's rant about the architectural choices Red Hat is making for containers.
Before Red Hat started to get involved there was the open source, (now) cross-platform containerd that a lot of tools are building on top. It consumes the low level runtime runc and both provides an API for Kubernetes (CRI) and additional things. It is highly pluggable and therefore the "only container runtime you'll ever need". And it's purely community governed (CNCF).
What is Red Hat doing? Are they building their container tools on top of containerd? Nope. They create their own low level container runtime (crun), their own mid level container runtime (cri-o). But cri-o only covers what's needed for Kubernetes. So to be able to build container images, etc (the "working with containers on a single machine" usecase) additional tools are needed (podman, buildah, skopeo) and they have to implement the missing functionality themselves. So Red Hat's Open Shift (Kubernetes distribution) builds on an entirely different stack than Podman.
Fragmentation everywhere. Was that really necessary?
Why should I care as an enduser? Well, I can inspect and debug all tools that use containerd the same way (i.e. using the "ctr" tool). I can relatively easily even write my own tools that use containerd (via it's grpc api) and can access the resources (containers, images, etc) that these tools are managing to solve my special custom needs.
For the Red Hat stuff everything is different. I could work with cri-o using crictl and write custom tools using cri interface. For Podman I would need to use Podman's API. (Which has annoyingly entirely changed between v1 and v2).
ok, enough rant (I could go on with how hard it is to get an up to date version of podman on a supported RHEL system even with app streams, that UBI images don't provide podman/buildah/skopeo, etc etc etc).
Please don't mistake me: Podman, cri-o, crun, open shift are all amazing technologies. And it's great that they are mostly open source via upstream projects. But I wish this whole fragmentation hadn't happened. For me as an enduser it's nightmare.