Ask HN: Does anyone use Terraform to deploy Kubernetes services in a large org?
Terraform supports Kubernetes service provisioning, but I've never seen anyone talk about using it at scale. On the surface, I worry a bit that there could be some impedance mismatch or bugginess, due to the layering-on of two state management solutions. Curious how it works in practice.
6 comments
Not a large org, but I dabbled with managing k8s with Terraform for a while and... did not have a great time. There were a few minor issues with state management and consistency between Terraform and k8s, but my main complaint was just that the k8s provider is very awkward to use. It for the most part works, but:
- It isn't an autogenerated wrapper around the k8s API, so not 100% of features are supported. E.g. there has been an open issue to implement setting runtimeClass on workload resources for three years.
- All of the options on the Terraform resources have been converted from camel-case to snake-case, which is fine except that I always forget to do that conversion when working off of the k8s API reference or an example. Some of them have also been converted from plural to singular.
- Because of the heavy use of Terraform blocks for configuring resources, it's often annoying to reuse chunks of configuration across resources or use non-trivial variables in the configuration -- and that's the big reason I wanted to use something like Terraform (versus flux or similar) in the first place.
- The k8s provider does actually have a way to manage custom resources now, the kubernetes_manifest resource type, but it requires writing the whole resource manifest out in HCL.
In short, there are better approaches and I would recommend against it. But hey, I'm just some guy on the internet.
- It isn't an autogenerated wrapper around the k8s API, so not 100% of features are supported. E.g. there has been an open issue to implement setting runtimeClass on workload resources for three years.
- All of the options on the Terraform resources have been converted from camel-case to snake-case, which is fine except that I always forget to do that conversion when working off of the k8s API reference or an example. Some of them have also been converted from plural to singular.
- Because of the heavy use of Terraform blocks for configuring resources, it's often annoying to reuse chunks of configuration across resources or use non-trivial variables in the configuration -- and that's the big reason I wanted to use something like Terraform (versus flux or similar) in the first place.
- The k8s provider does actually have a way to manage custom resources now, the kubernetes_manifest resource type, but it requires writing the whole resource manifest out in HCL.
In short, there are better approaches and I would recommend against it. But hey, I'm just some guy on the internet.
When you say "Kubernetes service provisioning" I'm assuming you mean Kubernetes application provisioning (k8s manifest)? But I'd be cautious of using Terraform to deploy k8's applications, it can be done but it has some drawbacks. One of the biggest issues is around interpolating k8s credentials in the provider config (you can google this for more details).
"When using interpolation to pass credentials to the Kubernetes provider from other resources, these resources SHOULD NOT be created in the same Terraform module where Kubernetes provider resources are also used . This will lead to intermittent and unpredictable errors which are hard to debug and diagnose. The root issue lies with the order in which Terraform itself evaluates the provider blocks vs. actual resources."
I'd recommend some solution such as argocd, flux, or if you're using a cloud provider such as azure they have automated deployments for k8s applications onto clusters (I'm sure aws has something similar).
"When using interpolation to pass credentials to the Kubernetes provider from other resources, these resources SHOULD NOT be created in the same Terraform module where Kubernetes provider resources are also used . This will lead to intermittent and unpredictable errors which are hard to debug and diagnose. The root issue lies with the order in which Terraform itself evaluates the provider blocks vs. actual resources."
I'd recommend some solution such as argocd, flux, or if you're using a cloud provider such as azure they have automated deployments for k8s applications onto clusters (I'm sure aws has something similar).
Last I checked, the kubernetes provider didn't support CRDs or custom resources. Its pretty common to have various operators running in a cluster so you'd need a different way to manage these. I usually provision clusters with Terraform and then use some other tool like flux/argo to deploy manifests managed with either helm or kustomize
We do the same at my current company. For common, core services on kubernetes clusters we use the terraform helm provider to install them.
See the aws-ia official eks-blueprints repo. They have a ton of modules that reuse their helm-addon and irsa modules. Adding the aws-load-balancer-controller, for instance, is a single boolean.
https://aws.amazon.com/blogs/containers/bootstrapping-cluste...
https://github.com/aws-ia/terraform-aws-eks-blueprints/tree/...
https://aws.amazon.com/blogs/containers/bootstrapping-cluste...
https://github.com/aws-ia/terraform-aws-eks-blueprints/tree/...