Ask HN: Kubernetes at big, non-tech companies?
17 comments
> Am I crazy for thinking that this misses the whole point of the technology?
Multi-cluster is not an accidental evolution. It was largely inevitable. Kubernetes has no hard tenancy concept (no, cobbling together a pile of namespaces and feature encrustations and calling it "tenancy" doesn't count). Instead people simulate tenancy with multiple clusters, because that isolates a particular API server's data completely.
While early attention and nerd glamour may have attached to the idea of workload consolidation, the true nexus of value these days is the API itself. This will probably remain true even while people are being burned by multi-cluster waste. There will be some movement to one of the various I-can't-believe-it's-not-multi-cluster solutions that are currently swirling around. Vendors of all sizes will be standing by to sell their take on the Kubernetes of Kubernetes.
Multi-cluster is not an accidental evolution. It was largely inevitable. Kubernetes has no hard tenancy concept (no, cobbling together a pile of namespaces and feature encrustations and calling it "tenancy" doesn't count). Instead people simulate tenancy with multiple clusters, because that isolates a particular API server's data completely.
While early attention and nerd glamour may have attached to the idea of workload consolidation, the true nexus of value these days is the API itself. This will probably remain true even while people are being burned by multi-cluster waste. There will be some movement to one of the various I-can't-believe-it's-not-multi-cluster solutions that are currently swirling around. Vendors of all sizes will be standing by to sell their take on the Kubernetes of Kubernetes.
To clarify- Kubernetes as typically configured has strong multitenancy for API access control. However, other aspects of multitenancy such as memory, CPU (including cross-container data leakage), network, disk/filesystem are implementation dependent. The most common implementations (i.e. on Linux with Docker or containerd) provide generally weak or no control over each of these aspects. Some (Kata Container, Firecracker) have stronger controls, and some (Windows Server Containers) have even fewer controls. But for complete separation, either separate worker node groups with very strict, limited API access or separate clusters are needed.
I guess it would be helpful to understand what your expectations of the technology are as well as providing a few more details. What you are describing doesn't sound crazy to me assuming selecting a cluster to deploy to is a one time thing. If it was every time you wanted to deploy a change that would be cumbersome. I suppose the step you need to do could be automated, and there are multi cluster solutions available, but I'm not familiar with them because I don't use Kubernetes this way, so I can't speak to how well they address this particular issue. Are you expecting that cluster selection is automated the same way that Kubernetes automatically selects which node to assign a container?
Edit:
Some things that might be helpful to know:
- Is your company using cloud or DC? This might explain some of the decision making.
- Since this is a pilot, is it possible there are plans for future improvements regarding cluster selection?
Edit:
Some things that might be helpful to know:
- Is your company using cloud or DC? This might explain some of the decision making.
- Since this is a pilot, is it possible there are plans for future improvements regarding cluster selection?
I built out the Kubernetes stack for data processing / batch processing with Airflow at Everlane. Fully automated deploys, monitoring, etc. Mind you - we didn't build this from scratch / use native Helm charts, but used an integration partner that gave us a much better foundation. The data engineers were responsible for maintenance, deploys, etc., but with the sophistication we got it was very rare to have to actually do helm deploys.
I think this is the key, someone can easily use k8s in the wrong way, so if you don't understand the concepts and try to use it like a VM you will have serious issues.
It really depends on why your company is using multiple clusters. Are there any size constraints? Any organization rules for which application should go where?
Overall no it doesn't miss the point because once you select a cluster to deploy to Kubernetes will still handle the rest for the lifetime of the application. And I imagine the selection part can be easily automated down the line as well.
Overall no it doesn't miss the point because once you select a cluster to deploy to Kubernetes will still handle the rest for the lifetime of the application. And I imagine the selection part can be easily automated down the line as well.
Selecting a ready-made cluster that you just have to check is marked healthy and has the resources your project needs?
Sounds like they made it really easy for you.
Sounds like they made it really easy for you.
An auomation could break something if automaticalyy choose the wrong cluster. Are all these clusters "free to pick"?
What in your opinion is “the point?”
To me as an SRE, the steps you mentioned sound entirely reasonable.
To me as an SRE, the steps you mentioned sound entirely reasonable.
As a SRE myself, this sounds entirely reasonable... for me to be doing for my backend teams. "You need X capacity, and checking on our dashboards, it looks like clusters Y and Z just had capacity added, so we'll plan this new project to run there (and horse-trade if we actually need to run on clusters A and B)". But it's definitely the realm of project management and SRE to care about, not every backend team, at least not at the enterprise level.
In our org, the cluster operators manage which clusters are open for onboarding vs. closed to new projects but running for existing projects (based on various scaling limits, some of which are cloud-specific). We don't ask developers to choose what's healthy and what's not. Our internal tooling simply only offers "open" clusters.
If you are cloud based this does not sound correct. If you have the cluster autoscaler setup, then the cluster should grow to allow for the deployment based on the constraints in your deployment.
If you are on prem then this is reasonable, since you have limited physical hardware so the cluster can not automatically grow.
If you are on prem then this is reasonable, since you have limited physical hardware so the cluster can not automatically grow.
There are scaling limits in Kubernetes that are independent of the Node count, depending on which components you have added on top of the vanilla components. My team has repeatedly load tested our clusters to failure without needing to create hundreds of Nodes.
I feel your pain man.
A big part of the story is that k8s is a naked king.
It’s just really poorly designed, over-engineered, shitty devx piece of software.
But most of devs eat you alive if you tell them this harsh truth.
We, regular engineers must be patient while people coming to enlightenment on their own and someone with big money would start developing & promoting better alternative.
A big part of the story is that k8s is a naked king.
It’s just really poorly designed, over-engineered, shitty devx piece of software.
But most of devs eat you alive if you tell them this harsh truth.
We, regular engineers must be patient while people coming to enlightenment on their own and someone with big money would start developing & promoting better alternative.
Very different from my experience, k8s makes deployment very easy. We use Concourse CI for build and deployment and it's very easy. Using Helm to control DDL changes using liquibase pre-deploy jobs, makes the deployment self contained.
Easy compared to what?
To nothing — yes.
To good examples of software like Swarm — well, my comment rating just proves my point.
To nothing — yes.
To good examples of software like Swarm — well, my comment rating just proves my point.
Kubernetes is horribly complicated, I would not recommend it to anyone.
I just attended a session for a pilot project where, in order to deploy the new shiny, a human has to log into a 'cluster metrics dashboard' and pick a 'healthy' target cluster, based on memory, CPU, Etc.
Am I crazy for thinking that this misses the whole point of the technology?
What are your experiences with this type of modernization?