CRDs Killed the Free Kubernetes Control Plane(caleblloyd.com)
caleblloyd.com
CRDs Killed the Free Kubernetes Control Plane
https://caleblloyd.com/software/crds-killed-free-kubernetes-control-plane/
7 comments
If your apps run on k8s already, why is it a bad idea to store some simple states in custom resources?
There are a number of reasons. One is that states don't stay simple for very long. Just the other day I saw someone ask "how do I find all the children of a parent resource?". The answer was to iterate over everything that might have that parent resource as an OwnerRef. In a normalised schema this would be simple. In a document database it is not.
Another problem: there are no multi-document transactions. If I rely on updating A and B together, I am out of luck. I will instead need to account for A and B being out of sync, which involves 4 permutations: A,B; A, !B; !A, B; !A, !B. As a CRD controller author I am now responsible for a shitty database. Where Kinds have crisp hierarchies this is fine, the supervising controller can inspect and repair the lower documents. But many things are not hierarchical in nature and you can easily wind up with control loop fights.
Another is that there's a difference between "I already have a cluster" and "I am using the API server as a database". Because CRDs wind up in a single global document store, folks inevitably want them to have hard isolation. RBAC, namespaces etc get you some of the way but it's basically just hard. Eventually some bright spark suggests putting things into a different cluster, at which point the balance of ease flips the other way.
CRDs are super useful extension points (note, for example, I am writing a book about Knative), but they are not suitable for most application and business logic purposes.
Another problem: there are no multi-document transactions. If I rely on updating A and B together, I am out of luck. I will instead need to account for A and B being out of sync, which involves 4 permutations: A,B; A, !B; !A, B; !A, !B. As a CRD controller author I am now responsible for a shitty database. Where Kinds have crisp hierarchies this is fine, the supervising controller can inspect and repair the lower documents. But many things are not hierarchical in nature and you can easily wind up with control loop fights.
Another is that there's a difference between "I already have a cluster" and "I am using the API server as a database". Because CRDs wind up in a single global document store, folks inevitably want them to have hard isolation. RBAC, namespaces etc get you some of the way but it's basically just hard. Eventually some bright spark suggests putting things into a different cluster, at which point the balance of ease flips the other way.
CRDs are super useful extension points (note, for example, I am writing a book about Knative), but they are not suitable for most application and business logic purposes.
Well, I agree that you can't really use CRD as an alternative to a database. But wouldn't you say that there are cases where it makes sense to use CRD to keep metadata? Keeping results of job runs for example.
I think the best use for CRDs is for creating seamless extensions to Kubernetes itself. I see Knative as being in this genre.
If your app is running on a modern computer anyway, why is it a bad idea to store some simple state in EFI variables?
> Since CRDs are installed on the cluster level, it is not possible to namespace resource versions (and that’s unlikely to happen soon)
This is misleading, the object _instances_ of a CRD belong to a namespace, unless they are cluster wide
This is misleading, the object _instances_ of a CRD belong to a namespace, unless they are cluster wide
But I absolutely agreed with this:
> There are applications out there that are storing their complete state in CRDs. Please do not do this. The Kubernetes API server is not a general-purpose datastore.
A thousand times, yes. The API server is good at what it does. It is not a suitable replacement for a real, honest-to-Codd database. The idea that laying hands on a connection to a postgresql or mysql server is somehow more onerous than spinning up an entire goddamn cluster is bonkers. Bon. Kers.