Ask HN: Any of you run Kubernetes clusters in-house?
I have a small SaaS and I am interested in slowly bringing everything in-house, those of you who run k8s cluster, can you describe your setup? Ex. how do you provision new hardware, HA, what kind of servers do you use?
9 comments
As much as this is HN and engineering(sometimes over engineering for the sake of fun or learning) is discussed here, I’d highly encourage you to think whether you really need Kubernetes in the first place..
Saying ^ because it sounded like you’re planning to use it in “production” as you say small SaaS, which implies your users rely on it and probably pay for it.
It’s often best to start simple - Load Balancer + web/app servers + Database server and add more complexity /features iteratively as you see a real need for it. You’d be surprised how far you can get with achieving K8S perks without bringing in the whole thing - rolling upgrades etc.
Just my 2 cents.
Saying ^ because it sounded like you’re planning to use it in “production” as you say small SaaS, which implies your users rely on it and probably pay for it.
It’s often best to start simple - Load Balancer + web/app servers + Database server and add more complexity /features iteratively as you see a real need for it. You’d be surprised how far you can get with achieving K8S perks without bringing in the whole thing - rolling upgrades etc.
Just my 2 cents.
Been running k3s for personal projects etc for some time now on a cluster of raspberry pies. Why pies? Were cheap at the time and wanted to play with arm. I don’t think I would suggest them right now. Nucs will be much better value for money.
Some notes:
Using helm and helmfile https://github.com/helmfile/helmfile for deployments. Seems to work pretty nicely and is pretty flexible.
As I’m using a consumer internet provider ingress is done through cloudflare tunnels https://github.com/cloudflare/cloudflare-ingress-controller in order to not have to deal with ip changes and not have to expose ports.
Persistent volumes were my main issue when previously attempting this, and what changed everything for me was longhorn. https://longhorn.io Make sure to backup your volumes.
Really hyped for https://docs.computeblade.com/ xD
Some notes:
Using helm and helmfile https://github.com/helmfile/helmfile for deployments. Seems to work pretty nicely and is pretty flexible.
As I’m using a consumer internet provider ingress is done through cloudflare tunnels https://github.com/cloudflare/cloudflare-ingress-controller in order to not have to deal with ip changes and not have to expose ports.
Persistent volumes were my main issue when previously attempting this, and what changed everything for me was longhorn. https://longhorn.io Make sure to backup your volumes.
Really hyped for https://docs.computeblade.com/ xD
I have a few kubernetes clusters, but nothing really critical or requiring high availability.
I use a mix of terraform, ansible, and kubernetes. I like k3s.
In terms of hardware it’s a mix on on-premise, expensive clouds and cheaper clouds. Provisioning new hardware varies between editing a few lines in a terraform file to talking to a salesperson, waiting days, and physically installing things in a servers room. I much prefer the terraform approach.
Consumer hardware like gaming computers can have a great value and last long enough if you are lucky or good while selecting the components. But I wouldn’t put my SaaS on such hardware. Or at least not the critical workloads.
Overall I wouldn’t recommend a small business to bring everything in house. Engineers at AWS, Microsoft, Google, or Hetzner are kinda better and more experienced than you will ever be to manage computers. But if you find it fun and have enough spare time for that, why not for non critical workloads.
I use a mix of terraform, ansible, and kubernetes. I like k3s.
In terms of hardware it’s a mix on on-premise, expensive clouds and cheaper clouds. Provisioning new hardware varies between editing a few lines in a terraform file to talking to a salesperson, waiting days, and physically installing things in a servers room. I much prefer the terraform approach.
Consumer hardware like gaming computers can have a great value and last long enough if you are lucky or good while selecting the components. But I wouldn’t put my SaaS on such hardware. Or at least not the critical workloads.
Overall I wouldn’t recommend a small business to bring everything in house. Engineers at AWS, Microsoft, Google, or Hetzner are kinda better and more experienced than you will ever be to manage computers. But if you find it fun and have enough spare time for that, why not for non critical workloads.
I run 200+ production clusters across EKS, GCP and MSA. In a nutshell - running your own clusters and being responsible for every aspect - autoscaling, storage, upgrades - it's not fun. Using managed Kubernetes allows you to run a cluster knowing all those bells and whistles are already taken care of. Hard to go wrong with the following setup:
1. A cluster VPC with public and private subnets.
2. A managed control plane on EKS.
3. You can opt between self-managed and managed nodegroups. To be fair, self-managed nodegroups give you more control, but you'll need to be careful about updating them yourself.
4. Use AWS controllers - the AWS Ingress Controller, the EBS CSI Driver - stuff like this will ensure your cluster can provision and manage load balancers, storage and so on.
5. For starters use cluster-autoscaler to dynamically manage compute capacity on your nodegroups. As your platform builds up and scales, you can look at Karpenter as a viable alternative.
1. A cluster VPC with public and private subnets.
2. A managed control plane on EKS.
3. You can opt between self-managed and managed nodegroups. To be fair, self-managed nodegroups give you more control, but you'll need to be careful about updating them yourself.
4. Use AWS controllers - the AWS Ingress Controller, the EBS CSI Driver - stuff like this will ensure your cluster can provision and manage load balancers, storage and so on.
5. For starters use cluster-autoscaler to dynamically manage compute capacity on your nodegroups. As your platform builds up and scales, you can look at Karpenter as a viable alternative.
I don’t think the parent poster meant EKS when they talk about “in-house” and “provision new hardware, servers etc”. I’d think more in the lines of buying a Dell/Lenovo server, racking it in their server room or collocated space, and running Kubernetes by themselves on their own hardware - not the cloud.
I've written about this specific subject here [0], feel free to reach out or ask them here if you have other questions
[0]: https://vmarchaud.fr/deploying-kubernetes-the-hard-way/
[0]: https://vmarchaud.fr/deploying-kubernetes-the-hard-way/
https://www.youtube.com/watch?v=dGz_lLldesU
This covers a lot of ground, but I don't know if lessons can be ported due to difference in footprint.
This covers a lot of ground, but I don't know if lessons can be ported due to difference in footprint.
So if you mean in house, being on premise in your own mini data center, you can find these cheap mini pcs, buying actual hardware can save a lot like you could use 3 mini pc's to make a cluster using ubuntu's version of k8s.