Kubernetes 1.12(kubernetes.io)
kubernetes.io
Kubernetes 1.12
https://kubernetes.io/blog/2018/09/27/kubernetes-1.12-kubelet-tls-bootstrap-and-azure-virtual-machine-scale-sets-vmss-move-to-general-availability/
46 comments
It's about time for a Kubernetes to do LTS releases. 3 month release cycle and support for only 9 months is too quick for companies given that in practice companies are running lots of clusters, not just 1 or 2 big clusters.
I mean one could argue that anything hosted on k8s should be reproducible anyway. I completely rebuild my k8s servers on a rolling two week cycle. The pain from that forces people to check in everything they need because if they don't it'll be gone very soon.
And still it's a bit of a battle to get people to test new versions of things, k8s included. "But it's working" is as much of a rallying cry as it ever was, "cloud" or no.
And still it's a bit of a battle to get people to test new versions of things, k8s included. "But it's working" is as much of a rallying cry as it ever was, "cloud" or no.
Curious about the details of how you rebuild/deploy your k8s servers. Any chance you've already written about this somewhere?
Nothing special. Just cobbler to generate the iso and kubespray to install the cluster.
Between company policy on social media and internal weirdness I can't really write a GitHub or Medium contribution to HN that anyone would want to consume anyway.
However I do plan to document what I've learned in a generic sort of "If you use Digital Ocean, this is how to do X, Y, Z" way.
Between company policy on social media and internal weirdness I can't really write a GitHub or Medium contribution to HN that anyone would want to consume anyway.
However I do plan to document what I've learned in a generic sort of "If you use Digital Ocean, this is how to do X, Y, Z" way.
Thanks! I haven't seen cobbler before, so that's certainly something I'll take a look at. Toyed with matchbox before, but didn't end up using it.
Not using DigitalOcean currently, but look forward to reading that someday.
Not using DigitalOcean currently, but look forward to reading that someday.
I don’t understand your argument. If you have lots of clusters, surely you’re upgrading using some kind of CD mechanism. So it shouldn’t make a difference in time to upgrade.
Zalando (disclaimer I work there, though not on K8S infrastructure. I just deploy to K8S) does upgrades this way: https://m.youtube.com/watch?v=1xHmCrd8Qn8
Zalando (disclaimer I work there, though not on K8S infrastructure. I just deploy to K8S) does upgrades this way: https://m.youtube.com/watch?v=1xHmCrd8Qn8
In practice most people do not run k8s upgrades through a CD pipeline. They are much more conservative about it. This is not a technical issue, just a cultural one mainly. People plan and roll out upgrades. Most organization treat a cluster as a large pet. They would like to avoid upgrading if they can because it's viewed as a risk. Patch releases are not typically an issue, it is minor releases that cause concern.
So every 3 months is too fast for a lot of teams (again, not technical issue, this is culture of enterprise) so they may put off an upgrade. Once you do that the upgrade becomes more risky because there is obviously more change from 1 to 3 than 1 to 2.
What I'm describing is not an issue specific to k8s, this is just a general enterprise infrastructure issue and one of the many reasons enterprises like LTS releases.
So every 3 months is too fast for a lot of teams (again, not technical issue, this is culture of enterprise) so they may put off an upgrade. Once you do that the upgrade becomes more risky because there is obviously more change from 1 to 3 than 1 to 2.
What I'm describing is not an issue specific to k8s, this is just a general enterprise infrastructure issue and one of the many reasons enterprises like LTS releases.
I think most people would very much like to deploy cluster updates more frequently.
Unfortunately this is difficult because most distributions don't keep up with the release cadence.
One of (the most?) popular way to deploy Kubernetes is the Kops project. As far as I know there is nobody who is paid to work on Kops, thus the bugs and PR's don't get enough hours invested in them.
Distribution and packaging is very boring and fiddly grunt work, it would be wonderful if somehow we as a community could figure out how to fund full time work on a distribution in the OSS space. Perhaps CNCF could do a better job of funding this somehow.
It's a hard problem, because the organizations most suited to solving it are CoreOS/RedHat/Microsoft/Google/Amazon/etc, and they have a vested interest in putting that effort into making their hosted solutions a great experience instead.
Kubeadm looks like a great step in the right direction, but as far as I'm aware, there aren't any production quality distributions for installing clusters with it yet.
Unfortunately this is difficult because most distributions don't keep up with the release cadence.
One of (the most?) popular way to deploy Kubernetes is the Kops project. As far as I know there is nobody who is paid to work on Kops, thus the bugs and PR's don't get enough hours invested in them.
Distribution and packaging is very boring and fiddly grunt work, it would be wonderful if somehow we as a community could figure out how to fund full time work on a distribution in the OSS space. Perhaps CNCF could do a better job of funding this somehow.
It's a hard problem, because the organizations most suited to solving it are CoreOS/RedHat/Microsoft/Google/Amazon/etc, and they have a vested interest in putting that effort into making their hosted solutions a great experience instead.
Kubeadm looks like a great step in the right direction, but as far as I'm aware, there aren't any production quality distributions for installing clusters with it yet.
And, as mentioned in a sibling comment, as complex as the K8s project is now, upgrades break things. They regularly break APIs, CLI arguments, tooling, documentation, integrations, and just about everything else.
Having your entire k8s infrastructure go down due to a "oopsie" in an upgrade is a risk. And mitigating these risks takes time and effort.
Having your entire k8s infrastructure go down due to a "oopsie" in an upgrade is a risk. And mitigating these risks takes time and effort.
As an ex-devops/infra guy, every day I'm ecstatic I don't have to wrangle k8s. These are not attributes of production infrastructure tooling. My hat is off to all of you in the trenches.
I still struggle with the idea that an infrastructure software version can go out of support within 6 months. I still recall the days where we would wait 6 months on a new version of software, just to let it settle and get its initial patches out of the way.
I hearken back to a snippet from a recent article:
> 2014: We must adopt microservices to solve all the problems with monoliths
> 2016: We must adopt Docker to solve all the problems with Microservices
> 2018: We must adopt Kubernetes to solve all the problems with Docker
What's next? "We must adopt Rancher to solve all the problems with K8s"?
I hearken back to a snippet from a recent article:
> 2014: We must adopt microservices to solve all the problems with monoliths
> 2016: We must adopt Docker to solve all the problems with Microservices
> 2018: We must adopt Kubernetes to solve all the problems with Docker
What's next? "We must adopt Rancher to solve all the problems with K8s"?
[deleted]
It's also that not all services used together may work on the latest version of k8s. For example, one of the services can only support up to 1.11 and need updates to use 1.12, then you suddenly cannot deploy 1.12 to those clusters. I think in terms of LTS it's more about having stable APIs for developing.
Yes, we still have some 1.1.7s.
Kubernetes already has many vendors that provide packaged distributions and long-term support if you want it.
If you use the managed hosting/cloud providers then updates are also automatic and easy.
If you use the managed hosting/cloud providers then updates are also automatic and easy.
I definitely have not found updates on GKE to be easy, even though all it takes to trigger it is clicking a button.
Each one I’ve done on our cluster over the past couple of years has caused something to break, often in a strange and obscure manner.
One time an automatic master upgrade happened at 9pm and effectively sent all of our logs into a black hole for reasons unknown. As with many of the other upgrade hiccups, I just had to poke, prod, delete and recreate entities and things magically started working again.
There is so much going on under the hood it makes sense that there would be issues with upgrades, so the frequency of the releases is frustrating. Dealing with after hours alarms just because the k8s release cycle is so aggressive really grinds my gears. I would fully support an LTS version.
Each one I’ve done on our cluster over the past couple of years has caused something to break, often in a strange and obscure manner.
One time an automatic master upgrade happened at 9pm and effectively sent all of our logs into a black hole for reasons unknown. As with many of the other upgrade hiccups, I just had to poke, prod, delete and recreate entities and things magically started working again.
There is so much going on under the hood it makes sense that there would be issues with upgrades, so the frequency of the releases is frustrating. Dealing with after hours alarms just because the k8s release cycle is so aggressive really grinds my gears. I would fully support an LTS version.
this is the hardest thing for people to get sometimes, it was the original core "concept" behind devops/ci-cd:
upgrades don't get easier when you delay them and make them bigger, they get harder. if you think k8s upgrades are bad now, having them change twice as much half as often would be worse.
upgrades don't get easier when you delay them and make them bigger, they get harder. if you think k8s upgrades are bad now, having them change twice as much half as often would be worse.
We've certainly embraced the devops approach, releasing many times per day with lots of automation on k8s, but I don't see this as being incompatible with using LTS releases.
Having an LTS release available offers the flexibility to choose when to upgrade based on any number of factors affecting the business and engineering cycles, the same way we pin software libraries to specific versions so our own stability and build consistency isn't subject to the varying release cycles of the package authors.
There's definitely a balance to be found between waiting so long to upgrade that it becomes insurmountable, and having to frequently run disruptive, mystery-downtime-causing upgrades for unneeded new features.
Having an LTS release available offers the flexibility to choose when to upgrade based on any number of factors affecting the business and engineering cycles, the same way we pin software libraries to specific versions so our own stability and build consistency isn't subject to the varying release cycles of the package authors.
There's definitely a balance to be found between waiting so long to upgrade that it becomes insurmountable, and having to frequently run disruptive, mystery-downtime-causing upgrades for unneeded new features.
If you have an automated test suite for your applications, and you use it to ensure that candidates pass before deployment, why can't you test new Kubernetes releases by ensuring that your apps' test suites (which are known to be good) pass on new Kubernetes versions? If the test suite fails on a new Kubernetes version, then either improve the test suite to get rid of false positives or fix the software.
In proper CI/CD practices, build "stability" is an anachronism. Master is always green. Test changes before merging. If the changes fail the test suite then don't merge and don't deploy until it's fixed. Master stays green. Why is dynamic infrastructure any different?
LTS only makes sense as a strategy if testing and carrying out the upgrade involve significant amounts of work which needs to be planned out in advance. Ideally, this isn't the case.
In proper CI/CD practices, build "stability" is an anachronism. Master is always green. Test changes before merging. If the changes fail the test suite then don't merge and don't deploy until it's fixed. Master stays green. Why is dynamic infrastructure any different?
LTS only makes sense as a strategy if testing and carrying out the upgrade involve significant amounts of work which needs to be planned out in advance. Ideally, this isn't the case.
That's mostly on the GKE team and not K8S itself. They run plenty of other extra components that arent well tested and often have issues during upgrades, along with the recent infrastructure updates for regional clusters, network endpoint groups and the rest that complicate things further.
Errors have never been major for us though, and they do seem to be greatly reduced recently with no noticeable issues in the 1.10x line.
Errors have never been major for us though, and they do seem to be greatly reduced recently with no noticeable issues in the 1.10x line.
I happen to be one of those vendors. I don't really view it as a good thing though. It's not good if everybody has to buy different forks of Kubernetes from vendors.
A point of reference, Amazon EKS only runs 1.10 which will not be supported by the community in 3 months. So not all managed services are really keeping up.
A point of reference, Amazon EKS only runs 1.10 which will not be supported by the community in 3 months. So not all managed services are really keeping up.
Ah, you're from Rancher. Great product, but isn't this exactly the problem space you serve?
How many of the K8S vendors are selling distributions vs support plans? Or are they always entangled? And if the community did do LTS, isn't that just shifting the work to open-source maintainers rather than focused and paid vendors?
How many of the K8S vendors are selling distributions vs support plans? Or are they always entangled? And if the community did do LTS, isn't that just shifting the work to open-source maintainers rather than focused and paid vendors?
Yeah having forked k8s will be bad. Back-porting possible security fixes could be tricky and having multiple companies have to do it, would be painful.
For on premises Kubernetes who do you know of who will provide committments to support k8s after the main project stops?
As far as I was aware of all the on-prem options cleave to the main project support lifecycle...
As far as I was aware of all the on-prem options cleave to the main project support lifecycle...
Disclaimer: I work for Red Hat Consulting on OpenShift.
Red Hat provides support our Kubernetes distribution OpenShift (upstream is OKD [formerly called Origin] project). We have an LTS release being cut soon as well. Watch OpenShift Commons for more info.
Red Hat provides support our Kubernetes distribution OpenShift (upstream is OKD [formerly called Origin] project). We have an LTS release being cut soon as well. Watch OpenShift Commons for more info.
Feedback for OpenShift: Fix your support licensing terms and tooling. Having to manually deregister nodes on a web form to avoid hitting support limits is archaic and absurd.
I realize that the aim is mostly at the enterprise market who don't run autoscaling groups or cattle-grade servers, but you're scaring off potential clients with these kinds of setups.
I realize that the aim is mostly at the enterprise market who don't run autoscaling groups or cattle-grade servers, but you're scaring off potential clients with these kinds of setups.
I hear ya. I recently worked with a customer using ASGs and the scale down part of the ASG does need to make subscription-manager calls to avoid leaking subs. Also makes blue-green cluster upgrades more interesting as well.
If you have a specific suggestion/RFE, I’d be happy to push on it.
If you have a specific suggestion/RFE, I’d be happy to push on it.
First, note that charging per CPU makes clients less likely to keep sufficient overhead to account for node loss or service scaling. Anything that makes availability a major financial decision is a bad thing (and openshift support is a relatively major financial line item).
A technical solution: A service that counts nodes or available VCPUs and reports it for billing. Kubernetes metrics makes this really simple.
A contract solution: Realize that bare metal is not the future (even for enterprises), and adopt a different billing model; following Oracle's per-CPU model is not the the way to be a leader in this industry. Per cluster, by bands of service counts, per site, per hour with minimum hour counts, etc.
A technical solution: A service that counts nodes or available VCPUs and reports it for billing. Kubernetes metrics makes this really simple.
A contract solution: Realize that bare metal is not the future (even for enterprises), and adopt a different billing model; following Oracle's per-CPU model is not the the way to be a leader in this industry. Per cluster, by bands of service counts, per site, per hour with minimum hour counts, etc.
bare metal is not the future, but very much a big chunk of the market, and it'll be for a long time. (and, I think, bare metal with k8s-like orchestrators is the short term future, and k8s + FaaS will probably bring the long awaited portability, that might help bring the hosting/infra prices down.)
that said, per core pricing is bullshit. and for exactly the reasons you mentioned.
that said, per core pricing is bullshit. and for exactly the reasons you mentioned.
That's cool :) Am I right in saying that'll be the first LTS OpenShift release?
What's the support lifecycle going to be on that?
What's the support lifecycle going to be on that?
Yes, it'll be our first official LTS release. I think the specifics aren't yet released but I think the intent is to make it more like the RHEL LTS policy & lifecyle.
yeah they mention some LTS for "Upstream components" from what I can see on the page, but not specifically which and for how long, I guess they might but it'd be good to see more info...
If your concern is about manually updating a lot of clusters, then there are many providers (GCP, AWS, DO...) offering managed solutions which most likely will handle the updates for you.
People with Kube experience, at what point should someone on Heroku consider switching to Kube with something like GKE? I have a rough idea of the pros and cons, particularly cost vs. time overhead, but curious about more subtle things.
I'm migrating "legacy" applications (as in non-containerized ones) to container environments for medium to large companies, and I would suggest two things:
- get a local test setup with minikube (https://github.com/kubernetes/minikube) and Helm (https://www.helm.sh/) and try out a few applications - convert a project of yours to a Kubernetes deployment, then a Helm chart
If both steps are successful and you still feel like Kubernetes is doing a better job for you then Heroku, get a Kubernetes professional. I promise you, there will be as many problems as with any other environment. The only difference is that the Kubernetes issues are not documented.
- get a local test setup with minikube (https://github.com/kubernetes/minikube) and Helm (https://www.helm.sh/) and try out a few applications - convert a project of yours to a Kubernetes deployment, then a Helm chart
If both steps are successful and you still feel like Kubernetes is doing a better job for you then Heroku, get a Kubernetes professional. I promise you, there will be as many problems as with any other environment. The only difference is that the Kubernetes issues are not documented.
"The only difference is that the Kubernetes issues are not documented."
Move that one to the icebox then!
Move that one to the icebox then!
I run a platform that is running applications multicloud, one of which is Heroku. Our devs that use heroku absolutely hate k8s.
They feel it's way more complicated and cumbersome and say they don't want to write thousand line YAML files and YAML engineering isn't their job.
Also, the pipelines Heroku provides and new CD is far superior. They don't want to reinvent Heroku.
Personally after I got k8s I felt about it how I felt about git. I think it's game changing and everyone should learn something about it. Can understand the frustration though.
They feel it's way more complicated and cumbersome and say they don't want to write thousand line YAML files and YAML engineering isn't their job.
Also, the pipelines Heroku provides and new CD is far superior. They don't want to reinvent Heroku.
Personally after I got k8s I felt about it how I felt about git. I think it's game changing and everyone should learn something about it. Can understand the frustration though.
If you want to migrate in order to use some of the many services offered by Google Cloud, migrating from Heroku to App Engine is likely an easier path.
Istio and Knative (and so on) are the PaaS-on-k8s projects you should watch and try. k8s by itself is not targeting that workflow.
Fast release cycle does give k8s a regular news coverage!
kubelet tls bootstrap is a feature I am looking forward to. We just finished dev work and testing to get to 1.11 as we have to support various cni's and automated upgrades from 1.10, it will probably be couple of months before we get to 1.12
K8s 1.12 is here