Kubernetes 1.20(kubernetes.io)
kubernetes.io
Kubernetes 1.20
https://kubernetes.io/blog/2020/12/08/kubernetes-1-20-release-announcement/
69 comments
Volume Snapshots are going to be a game-changer for stateful workloads. I'm glad to see this feature in the wild. Great job Kubernetes team!
Agreed! It's an abstraction on top of many existing cloud provider storage abstractions which will be a welcome feature to many places wanting a quick backup solution of their volumes.
Slightly off topic, but one of my most popular essays from last year was, in part, a criticism of Kubernetes. Kubernetes is best understood as an advanced optimization that offers benefits to very large firms with thousands or tens of thousands of servers. Yet I've seen the idea spreading that every company, even a tiny startup with 3 servers, should Dockerize everything and run it all through Kubernetes.
I mostly consult with small startups, so I've seen the damage these ideas can do. Last year I helped a client who had raised $5 million and then, before they hired me, they'd wasted $1 million on a "high availability" devops setup that was overbuilt for what they needed.
If anyone is interested in a case study of excess devops development, please see "High Availability is not compatible with an MVP, because an MVP is about fast iteration"
http://www.smashcompany.com/technology/high-availability-is-...
I mostly consult with small startups, so I've seen the damage these ideas can do. Last year I helped a client who had raised $5 million and then, before they hired me, they'd wasted $1 million on a "high availability" devops setup that was overbuilt for what they needed.
If anyone is interested in a case study of excess devops development, please see "High Availability is not compatible with an MVP, because an MVP is about fast iteration"
http://www.smashcompany.com/technology/high-availability-is-...
Certainly, all the current implementations of Kubernetes are built out of enterprise-scale components — even if they're then squeezed into the girdle of a developer tool, the way microk8s is.
But the design (functional architecture, APIs, etc.) of Kubernetes makes good sense for small scale as well.
There's nothing fundamentally stopping someone from creating an "embedded Kubernetes" that exposes all the same resource APIs, but which is just a management interface for e.g. a JVM, or an Erlang node, where all those APIs are just controlling in-memory abstractions existing inside one userland process. Presenting itself as a "cluster" with one fixed node pool, with one fixed node, with one fixed type of container runtime which is just "spawn an actor-thread in this process."
I don't disagree that people are pursuing the enterprise k8s stuff when they aren't at the right scale to need it — but I think it's important to realize that they're often doing that because they see the design of k8s as fundamentally sound, providing sensible abstractions and management interfaces that neither plain OS process APIs, nor Docker's APIs, nor IaaS-level control-plane APIs have great equivalents for. It's that design they're after; the HA stuff is just a "nice-to-have" that they end up adding on, because all the guides they read (that are written for enterprise integrators) insist it's a great idea.
(Tangent: if Docker Inc. would get off their collective butts and do Docker Swarm over again, but this time with k8s-compatible APIs, I'm sure it'd become the most-used "Kubernetes" overnight. Most devs just want a regular Docker host, that they can `kubectl apply -f` against.)
But the design (functional architecture, APIs, etc.) of Kubernetes makes good sense for small scale as well.
There's nothing fundamentally stopping someone from creating an "embedded Kubernetes" that exposes all the same resource APIs, but which is just a management interface for e.g. a JVM, or an Erlang node, where all those APIs are just controlling in-memory abstractions existing inside one userland process. Presenting itself as a "cluster" with one fixed node pool, with one fixed node, with one fixed type of container runtime which is just "spawn an actor-thread in this process."
I don't disagree that people are pursuing the enterprise k8s stuff when they aren't at the right scale to need it — but I think it's important to realize that they're often doing that because they see the design of k8s as fundamentally sound, providing sensible abstractions and management interfaces that neither plain OS process APIs, nor Docker's APIs, nor IaaS-level control-plane APIs have great equivalents for. It's that design they're after; the HA stuff is just a "nice-to-have" that they end up adding on, because all the guides they read (that are written for enterprise integrators) insist it's a great idea.
(Tangent: if Docker Inc. would get off their collective butts and do Docker Swarm over again, but this time with k8s-compatible APIs, I'm sure it'd become the most-used "Kubernetes" overnight. Most devs just want a regular Docker host, that they can `kubectl apply -f` against.)
> they'd wasted $1 million on a "high availability" devops setup that was overbuilt for what they needed
not sure, but with 1$ million on budget, you can run a pretty pretty big gke cluster. btw. we run on e2-standard-2 and pay ~500 per month (would be a little bit more without commited usage). 6 nodes + 2 managed cloud sql databases+several load balancers.
really no idea how one can pay a million even in a year. that would not be a small startup...
you can probably have like 9x e2-standard-32 (32vcpu+128gb) + postgres ha 16vcpu/256gb mem/500gb ssd for like 10,867.33 per month without traffic (which might be another 5000) so with traffic you would only pay 190407.96 per year?!
Edit: even without gke, k3s is way simplier than most "deployment scripts" or whatever. it's always a matter if you want to have ops people or if gke's self managing is enough.
Edit2: so many lols at the blog post. I mean who the fuck spends 7000 at a colo for 40.0000 users. my macbook could probably serve that. also "and then script those older Virtual Machines with some mix of Ansible and Terraform (assuming you want to be in the cloud). " is a fucking terrible advice, been there, done that and it soon is a clusterfuck of stuff cobbeled together.
not sure, but with 1$ million on budget, you can run a pretty pretty big gke cluster. btw. we run on e2-standard-2 and pay ~500 per month (would be a little bit more without commited usage). 6 nodes + 2 managed cloud sql databases+several load balancers.
really no idea how one can pay a million even in a year. that would not be a small startup...
you can probably have like 9x e2-standard-32 (32vcpu+128gb) + postgres ha 16vcpu/256gb mem/500gb ssd for like 10,867.33 per month without traffic (which might be another 5000) so with traffic you would only pay 190407.96 per year?!
Edit: even without gke, k3s is way simplier than most "deployment scripts" or whatever. it's always a matter if you want to have ops people or if gke's self managing is enough.
Edit2: so many lols at the blog post. I mean who the fuck spends 7000 at a colo for 40.0000 users. my macbook could probably serve that. also "and then script those older Virtual Machines with some mix of Ansible and Terraform (assuming you want to be in the cloud). " is a fucking terrible advice, been there, done that and it soon is a clusterfuck of stuff cobbeled together.
Yeah, from the blog post:
> I’ve a good friend at a startup who found a co-lo data center in a small college town in Virginia, where they were able to build out a massive MongoDB database, sharded across 4 massive servers, each server holding 256 gigabytes of RAM, 1 terabyte of RAM in total. They also had a Cassandra database and several servers for their front-end. They were paying $7,000 a month for this.
Like, if you're paying AWS to host MongoDB on four EC2 machines with 256gb of RAM each and no autoscaling/load balancing, you can spend a lot of money. But... why would you do that? Porting terrible architectures to cloud VMs with no rearchitecting costs a boatload, but... don't do that?
> I’ve a good friend at a startup who found a co-lo data center in a small college town in Virginia, where they were able to build out a massive MongoDB database, sharded across 4 massive servers, each server holding 256 gigabytes of RAM, 1 terabyte of RAM in total. They also had a Cassandra database and several servers for their front-end. They were paying $7,000 a month for this.
Like, if you're paying AWS to host MongoDB on four EC2 machines with 256gb of RAM each and no autoscaling/load balancing, you can spend a lot of money. But... why would you do that? Porting terrible architectures to cloud VMs with no rearchitecting costs a boatload, but... don't do that?
"I mean who the fuck spends 7000 at a colo for 40.0000 users."
You misread the article. The startup with the $1 million devops ended up with 40,000 users. The startup with the co-lo was a different startup, which in fact had a very large user base.
You misread the article. The startup with the $1 million devops ended up with 40,000 users. The startup with the co-lo was a different startup, which in fact had a very large user base.
That has nothing to do with kubernetes, and a lot to do with management of the company.
I might be biased, because my first commercial use of kubernetes was to reduce infrastructure spend and optimize the hell of what we have. From my PoV, the "big company" narrative is just what happens when the big company notices the cost of spinning separate VMs for everything.
I might be biased, because my first commercial use of kubernetes was to reduce infrastructure spend and optimize the hell of what we have. From my PoV, the "big company" narrative is just what happens when the big company notices the cost of spinning separate VMs for everything.
I 100% see the point you are trying to make.
However Kubernetes does have a lot to offer, especially with the easy of hosted Kubernetes solutions these days. Helm makes installing tools and dependencies you may need almost trivial.
One of the nice things about Kubernetes is that it is relatively easy to scale from a small deployment to a larger HA deployment when (and if) you need to.
> Last year I helped a client who had raised $5 million and then, before they hired me, they'd wasted $1 million on a "high availability" devops setup that was overbuilt for what they needed.
This sounds like more of a management, requirements, and overbuilding problem, rather than a Kubernetes problem. Even without Kubernetes, if they wanted an HA devops set up, they still would have built one. Honestly without Kubernetes it could have ended up more expensive.
However Kubernetes does have a lot to offer, especially with the easy of hosted Kubernetes solutions these days. Helm makes installing tools and dependencies you may need almost trivial.
One of the nice things about Kubernetes is that it is relatively easy to scale from a small deployment to a larger HA deployment when (and if) you need to.
> Last year I helped a client who had raised $5 million and then, before they hired me, they'd wasted $1 million on a "high availability" devops setup that was overbuilt for what they needed.
This sounds like more of a management, requirements, and overbuilding problem, rather than a Kubernetes problem. Even without Kubernetes, if they wanted an HA devops set up, they still would have built one. Honestly without Kubernetes it could have ended up more expensive.
Helm was something that I ultimately decided to avoid at my last job, and I was happier for it. Upgrades that get in dirty states, often as a result of bogus post-install hooks. Having to fork charts to customize the deployment minorly for our mundane requirements. Reading doubly or triply templated files and dealing with bullshit indentation mismatches. I hated things that expected me to install with Helm.
I can agree with you that some of uses of it get out of hand. Like most things, there is a balance. For the most part I don't use hooks, precisely because they muddy the waters of my expected end state.
Upgrades that get in bad states, ya, I've had to deal with it before, and it definitely sucks. The "atomic" goal that Helm 3 tries for is better in my opinion, but still has rough edges.
What have you been using as an alternative that doesn't require you to write your own manifests for every dependency?
Upgrades that get in bad states, ya, I've had to deal with it before, and it definitely sucks. The "atomic" goal that Helm 3 tries for is better in my opinion, but still has rough edges.
What have you been using as an alternative that doesn't require you to write your own manifests for every dependency?
A lot of the time, writing (or forking and customizing) my own manifest was at least more deterministic and predictable. Obviously you get lost at that point in time if you aren't diligent about upgrading. Helm felt like this middleground, and I really felt that at least for managed clusters, it would be better if the addon managers provided stuff that Just Worked (tm).
We found a lot of success also with Kustomize. I did not enjoy the ksonnet detour as much - it was nice to override a prototype instead of having to screw with yaml templating, but jsonnet is not a good language.
Helm 3 had some designs on a lua API, which I thought was a promising idea. I've been out of the loop since moving on to a company that is back in EC2-land (which really just means that another team manages the thing that approximates Kubernetes...).
We found a lot of success also with Kustomize. I did not enjoy the ksonnet detour as much - it was nice to override a prototype instead of having to screw with yaml templating, but jsonnet is not a good language.
Helm 3 had some designs on a lua API, which I thought was a promising idea. I've been out of the loop since moving on to a company that is back in EC2-land (which really just means that another team manages the thing that approximates Kubernetes...).
Ya, I've gone the fork-and-customize route (but I typically just `helm fetch`-d into my repo), or pinned to the version I needed.
I agree with you about an addon manager, but then something has to be the source of truth for that, and the "KubeApps" type apps were all using Helm under the hood anyways.
I played around with Kustomize for a bit at one point, but honestly had a bit of difficulty understanding how it loaded and applied the different "layers". I've been looking more recently at it as a "make some quick customizations to this rendered helm chart" tool as of late.
I agree with you about an addon manager, but then something has to be the source of truth for that, and the "KubeApps" type apps were all using Helm under the hood anyways.
I played around with Kustomize for a bit at one point, but honestly had a bit of difficulty understanding how it loaded and applied the different "layers". I've been looking more recently at it as a "make some quick customizations to this rendered helm chart" tool as of late.
>Having to fork charts to customize the deployment minorly for our mundane requirements.
This seems to be a big problem with container orchestrators these days. I was deploying something to ECS and while 90% of the config could be accomplished with environment variables, there are still a few cases where it was expected that I would provide files. On a local machine with Docker and some config mgmt like Chef or Ansible, no problem - template out those files and volume mount them in. In ECS? No easy to way to do it without baking your own container or using EFS, which for one config option seemed like major fuckin' overkill. Ended up bastardizing the entrypoint with some judicious base64'd bullshit and got it going without either, but christ, how fucking hard would it be to just provide a native mechanism for templating config files out?
This seems to be a big problem with container orchestrators these days. I was deploying something to ECS and while 90% of the config could be accomplished with environment variables, there are still a few cases where it was expected that I would provide files. On a local machine with Docker and some config mgmt like Chef or Ansible, no problem - template out those files and volume mount them in. In ECS? No easy to way to do it without baking your own container or using EFS, which for one config option seemed like major fuckin' overkill. Ended up bastardizing the entrypoint with some judicious base64'd bullshit and got it going without either, but christ, how fucking hard would it be to just provide a native mechanism for templating config files out?
Absolutely.
For applications I built myself, I would often just roll the possible configs for us as files in the container, and you pick one with a single env var (realistically this was like "dev", "beta", "prod"). Not great, but meant I could avoid the clusterfuck that is specifying all my AWS configuration in env vars on a per deploy basis.
ConfigMaps could sort of help with this, but I found what you found - having to base64 a config file as a field in a ConfigMap and make sure it gets mounted correctly just seems pretty baroque for configuring something with a file. And at the time, ConfigMap changes didn't even cause a redeploy, so your pods would for their lifetime keep on using a stale config!
For applications I built myself, I would often just roll the possible configs for us as files in the container, and you pick one with a single env var (realistically this was like "dev", "beta", "prod"). Not great, but meant I could avoid the clusterfuck that is specifying all my AWS configuration in env vars on a per deploy basis.
ConfigMaps could sort of help with this, but I found what you found - having to base64 a config file as a field in a ConfigMap and make sure it gets mounted correctly just seems pretty baroque for configuring something with a file. And at the time, ConfigMap changes didn't even cause a redeploy, so your pods would for their lifetime keep on using a stale config!
Yeah, and this is when working with first-party upstream Dockerized apps that have good entrypoints already designed to function as adapters between env var config and internal config files....
I'll have a read through your article later as I'm quite interested in hearing your point of view. However in my experience, it's pretty easy to get a reasonable Kubernetes cluster setup with stateless apps on a cloud platform managed Kubernetes service.
Can you share in your opinion a good single guide to do this in AWS?
I haven't worked with EKS; I've mainly worked with Azure's AKS, but the principals are largely the same.
I'm not sure how familiar you are with AWS, EKS, Kubernetes, or Docker in general and I don't know of a guide that encompasses all these technologies.
My best advice would be:
1) Get a little toy Docker container running locally with a http server that responds with "Hello world"
2) Try and deploy it to as a pod to a local Kubernetes cluster (Docker desktop has one built in or you can try deploying Minikube if you're feeling adventurous)
3) Add an ingress and service to your cluster, so you can access your server from a browser
4) Expand your cluster by installing something like Postgres on it and have your toy app query it and print out a query result in response to requests
5) Try passing your toy app the password to login to Postgres as a Kubernetes Secret via an environmental variable
6) Start exploring EKS specific deployment documentation such as the links below.
7) Start learning about helm
https://docs.aws.amazon.com/eks/latest/userguide/create-clus...
https://docs.aws.amazon.com/eks/latest/userguide/sample-depl...
https://learn.hashicorp.com/tutorials/terraform/eks (Once your comfortable with everything else)
I'm not sure how familiar you are with AWS, EKS, Kubernetes, or Docker in general and I don't know of a guide that encompasses all these technologies.
My best advice would be:
1) Get a little toy Docker container running locally with a http server that responds with "Hello world"
2) Try and deploy it to as a pod to a local Kubernetes cluster (Docker desktop has one built in or you can try deploying Minikube if you're feeling adventurous)
3) Add an ingress and service to your cluster, so you can access your server from a browser
4) Expand your cluster by installing something like Postgres on it and have your toy app query it and print out a query result in response to requests
5) Try passing your toy app the password to login to Postgres as a Kubernetes Secret via an environmental variable
6) Start exploring EKS specific deployment documentation such as the links below.
7) Start learning about helm
https://docs.aws.amazon.com/eks/latest/userguide/create-clus...
https://docs.aws.amazon.com/eks/latest/userguide/sample-depl...
https://learn.hashicorp.com/tutorials/terraform/eks (Once your comfortable with everything else)
Honestly that sounds like not simple at all. I just want a webapp up and running, with a database and cache server, and I have again and again launched Elastic Beanstalk + Elasticache + RDS (or the GCP version) in half a day following basic tutorials, and this system runs very well (and scales very well) with zero need to "dive into documentation" for various things. I can then focus on my app instead of infra. It's still not clear why such simple architectures need kubernetes.
Like a lot of things, it's pretty simple once you get over the learning curve. I do these setups all the time, so it's like doing the dishes for me.
K8s may not be the best option for every project, but my original point was that choosing it ought not blow out the budget or delivery window of a project. Maybe, you might shave a week or two off a greenfield project by going with something like Elastic Beanstalk, but in the scheme of things that's a rounding error.
To put things in perspective, a typical deployment for a small app requires 3-4 short configuration files specific to K8s plus your Docker container.
Furthermore, you'd generally still use the cloud managed services like Elastic Cache and RDS when developing K8s applications. I was merely using a Postgres as an example in my mini-not-quite-a-tutorial before.
So yeah, if Elastic Beanstalk is working for you and you think the tradeoff between flexibility and simplicity is right for your project, then keep doing what you're doing.
K8s may not be the best option for every project, but my original point was that choosing it ought not blow out the budget or delivery window of a project. Maybe, you might shave a week or two off a greenfield project by going with something like Elastic Beanstalk, but in the scheme of things that's a rounding error.
To put things in perspective, a typical deployment for a small app requires 3-4 short configuration files specific to K8s plus your Docker container.
Furthermore, you'd generally still use the cloud managed services like Elastic Cache and RDS when developing K8s applications. I was merely using a Postgres as an example in my mini-not-quite-a-tutorial before.
So yeah, if Elastic Beanstalk is working for you and you think the tradeoff between flexibility and simplicity is right for your project, then keep doing what you're doing.
Whats wrong with running small deploys? It takes less than half a day to set up. Its honestly way easier to get a reproducible k8s setup than some hand rolled thing even if you run a single pod.
Or do you mean a small team should not use unmanaged k8s?
Or do you mean a small team should not use unmanaged k8s?
The problem with Kubernetes seems to be one of knowing when to make the switch.
My company is in a place where I think k8s would be really beneficial. We are at the 40k VMs running 1000 different types of apps. We suffer from tremendous wasted resources as a result.
The company started before k8s (and even docker) were things. So getting us there seems like a real daunting (yet worth it) task.
So, we are left with sort of a chicken/egg problem. It would have been unreasonable to start with k8s, yet now it is a huge cost to switch to it (even though it will save us millions).
My company is in a place where I think k8s would be really beneficial. We are at the 40k VMs running 1000 different types of apps. We suffer from tremendous wasted resources as a result.
The company started before k8s (and even docker) were things. So getting us there seems like a real daunting (yet worth it) task.
So, we are left with sort of a chicken/egg problem. It would have been unreasonable to start with k8s, yet now it is a huge cost to switch to it (even though it will save us millions).
It’s easy to overestimate that you’ll save money here - if you have waste, poor cost attribution, etc., you can easily do the same with Kubernetes managing the nodes and scheduling the processes. To actually save the money you have to fix those things first.
The issue we have is that we are a very peeky computation requirements (not a steady stream of calculations going on).
Further, we've got a fairly deep data processing pipeline. It's not unusual for us to see high loads on early parts of the system 1 hour and 1 hour later seeing high loads on the later parts of the system.
Unfortunately, we don't currently have a way to scale out and reassign resources. That means we've built out for our peaks even though the majority of the time that hardware is sleeping.
We could spend the time building out a scaling system (right now, our resources are all on prem). However, why do that when we can instead take advantage of k8s already built in scaling capabilities?
So, technically not k8s that will be saving us money, but rather implementing autoscaling. However, the solution with k8s is really nice.
Further, we've got a fairly deep data processing pipeline. It's not unusual for us to see high loads on early parts of the system 1 hour and 1 hour later seeing high loads on the later parts of the system.
Unfortunately, we don't currently have a way to scale out and reassign resources. That means we've built out for our peaks even though the majority of the time that hardware is sleeping.
We could spend the time building out a scaling system (right now, our resources are all on prem). However, why do that when we can instead take advantage of k8s already built in scaling capabilities?
So, technically not k8s that will be saving us money, but rather implementing autoscaling. However, the solution with k8s is really nice.
Cluster autoscaling with nodes on-prem, or in the cloud?
I work for a large Unnamed Parcel Service and all of our future applications are going to Azure Kubernetes Service, which is the Microsoft-managed k8s offering. There are still a ton of legacy applications running on all sorts of platforms, from mainframe to just about any technology you can name from the past 40 years, but everything going forward is being written for .Net Core running on Linux containers in AKS. So you don't have to switch everything immediately, you can get around to it eventually but at least you can start to build up a pool of talent and skill set towards that goal.
While there is some truth to what you are saying in your linked post, the real truth is more nuanced and, as usual, lies between any extremes. Firstly, when talking about extreme flexibility and experimentation, you're not talking about MVP in my book, but rather about a prototype. You can experiment with a prototype all you want, but no so much with an MVP, which, be definition, is a product, even if a minimal viable one. Secondly, your argument that MVP-stage early startups don't need HA breaks due to the corollary of MVP being a product. If your early-stage startup has, say, 5-10 important enterprise customers (those whose logos you so desperately need for initial sales as well as further marketing) using your MVP as product or even a pilot, guess how many of them will stay with you after a first 5-hour outage of your platform? Thirdly, in order to prevent disastrous exodus situations like mentioned above, an early startup does not need to have a globally scalable geographically distributed HA system (or, as you put it, "multiple servers, multiple load balancers, multiple failover databases, multiple data centers, multiple regions"). It would be quite enough to have a single level of redundancy for each major single point of failure. For a typical cloud architecture, it just means architecting, deploying and maintaining your system for/to/in two availability zones. This IMO represents a pretty good compromise between costs (both monetary and effort-wise) and risks (reputational, current and future sales, etc.). Fourthly, your suggested preference of VMs to containers will most likely lead to a) lower agility [due to spinning a VM being a much slower process versus spinning a container; that would be especially noticeable across development cycles: CI/CD etc.] and b) overspending on infrastructure [due to overhead stemming from the fact that you simply cannot pack VMs as efficient as containers into a given computational asset]. Fifthly, some/many operational aspects that have to be managed - manually or via automation such as Terraform - in non-HA scenarios could be partially or completely eliminated in cloud-based HA scenarios thanks to those aspects being embedded into relevant PaaS products, hence potentially significant cost savings and, equally or more importantly, time savings, thus, enabling a potentially much higher agility, which is so important to early-stage startups.
You can actually set up k3os on a single node and get all the benefits eg release a new deployment and have it keep old pods up until new pods are healthy . I have it hosted on my NAS and use ansible to set up deployments and in ci.
Tooling has come along way as well.
But the learning curve is knarly and I picked up a few tricks from companies I worked at.
Tooling has come along way as well.
But the learning curve is knarly and I picked up a few tricks from companies I worked at.
I see Kubernetes as a way to save money on web services. It allows for platform independent deployment.
Moving from AWS to Google Cloud to bare metal can save massive amount of money.
Moving from AWS to Google Cloud to bare metal can save massive amount of money.
I agree that a startup shouldn't (yet) put a lot of time into redundancy if they don't have a product or customers. But a code pipeline that can be used to quickly develop, test and deploy is essential. There is absolutely nothing wrong with using containers. In fact, in 20 years in tech, it's the best packaging solution I've seen. As for Kubernetes, as long as you're not managing the backend, it's fine. I joined a tiny startup less than a year ago, set up EKS with Terraform and created CI automation to build and push containers when tests pass. It's not hard. And it does encourage good practices. If we had to move everything to Azure or GCE, it would be much easier to do so because we haven't painted ourselves into the AWS corner.
In a lot of cases it's worth considering whether orchestration is necessary at all — pure docker can take you quite far, with just a couple shell scripts called from the last stage of your CI (or manually)
Serious question - what's the alternative? We have Docker containers, we want to run in swarm mode, and we want to enable service-to-service communication via gRPC. Kube seems to be the only thing that can support that unless we want to pay Traefik $20k+ which is a hard pass.
What do you recommend to these people instead of k8s, so that they can have all the convenience of orchestration and containerization?
Why containerize to start with?
Honestly, it's a lot simpler to either do PAAS, lambdas, or even something like raw EC2 instances to begin with.
The road to containerization and full devops is something that can happen after you've got a working app out in the wild (hopefully) making money.
Honestly, it's a lot simpler to either do PAAS, lambdas, or even something like raw EC2 instances to begin with.
The road to containerization and full devops is something that can happen after you've got a working app out in the wild (hopefully) making money.
Lambdas get pricey quick.
PAAS tends to lock you in. You don't notice it happening as you add features, then you discover that escaping the platform requires significant re-engineering. (When freelancing, one of my more lucrative gigs was helping a company de-Herokuize their app after the original developers moved on).
Raw EC2 instances usually end up with wizards SSHing in to deploy & tweak. It's sooo tempting.
Having participated in all of these scenarios in the past, I'd recommend any sized company go right to containers. It's the straightest road from zero through MVP to scale, and the best for onboarding new developers.
That said, don't get sidetracked! Pay a company to manage the orchestration. And don't burn time on accessories like CI pipelines or blue-green deploys until the important stuff works.
PAAS tends to lock you in. You don't notice it happening as you add features, then you discover that escaping the platform requires significant re-engineering. (When freelancing, one of my more lucrative gigs was helping a company de-Herokuize their app after the original developers moved on).
Raw EC2 instances usually end up with wizards SSHing in to deploy & tweak. It's sooo tempting.
Having participated in all of these scenarios in the past, I'd recommend any sized company go right to containers. It's the straightest road from zero through MVP to scale, and the best for onboarding new developers.
That said, don't get sidetracked! Pay a company to manage the orchestration. And don't burn time on accessories like CI pipelines or blue-green deploys until the important stuff works.
PaaS tend to be quite expensive, especially when you're outside of USA. It's quite easy for example for Heroku to add a cost of fully loaded senior "devops" engineer to your bill, compared with k8s.
Same for raw EC2 instances. When you start reaching double-digit numbers, it gets annoyingly harder especially if they aren't identical (for example, due to having different applications, or applications that can't live together, etc)
Same for raw EC2 instances. When you start reaching double-digit numbers, it gets annoyingly harder especially if they aren't identical (for example, due to having different applications, or applications that can't live together, etc)
While PaaS is expensive, the point really is that the cost of PaaS early on will likely be far less than the cost of properly setting up and building out a k8s cluster/deployment pipeline.
Getting something up, running, and making money is really the first priority for a business. Fixing it to not suck or eat all the profits comes later.
Getting something up, running, and making money is really the first priority for a business. Fixing it to not suck or eat all the profits comes later.
That sounds nice and dandy... except when PaaS increase your costs 1.5x compared to steady state (that has a lot of optimization opportunities left over), and about 6x when comparing initial conditions.
And we would have close to zero efficiency benefits related to developing the applications, for the price hike that would allow us to hire dedicated engineers to fix issues that would be either common with PaaS or help us making everything work fine on k8s.
And we would have close to zero efficiency benefits related to developing the applications, for the price hike that would allow us to hire dedicated engineers to fix issues that would be either common with PaaS or help us making everything work fine on k8s.
I mean, containerizing isn't harder than anything else. It makes distributing your artifacts much easier, and building across people's computers easier too. It takes O(minute) to containerize something, so I don't see why not do it.
I do Terraform + VMs + Serverless. It's the same thing with a layer of indirection cut out and platform native cloud UIs.
You can always use systemd and pick an orchestration tool like ansible,cdk or terraform
You don't even need the "orchestration tool"
Started small with k8s / gke, suffered a lot, coming to appreciate it as we grow. I think over the last 1-2 years there weren't a lot of other choices.
Knowing what I know now, I think if you are successful something like k8s is required to scale and you are running non trivial apps. For trivial stuff I'd go serverless plus some kind of managed app platform as much as possible.
For non trivial, there are stripped down k8s distributions out there that would work and let you work in portable containers. I'd think pretty hard about something like Rancher to get started, and 'graduate' to GKE / EKS / AKS when you hit scale with at least 100's of VM's. You need to keep it in containers to plausibly make the switch though and not require a rewrite later. I think Rancher deploys on all the major and smaller cloud vendors, is small enough to run on your desktop, still do development, and adheres to the API standards you'll care about later.
Knowing what I know now, I think if you are successful something like k8s is required to scale and you are running non trivial apps. For trivial stuff I'd go serverless plus some kind of managed app platform as much as possible.
For non trivial, there are stripped down k8s distributions out there that would work and let you work in portable containers. I'd think pretty hard about something like Rancher to get started, and 'graduate' to GKE / EKS / AKS when you hit scale with at least 100's of VM's. You need to keep it in containers to plausibly make the switch though and not require a rewrite later. I think Rancher deploys on all the major and smaller cloud vendors, is small enough to run on your desktop, still do development, and adheres to the API standards you'll care about later.
Docker Swarm. It's not dead and you can set it up in no time.
[deleted]
ECS and Cloud Run are options to get containers without k8s.
Cloud Run is basically Custom Resource on top of Kubernetes API. The yaml definition is pretty much the same - https://cloud.google.com/run/docs/deploying#yaml
What you mean is probably without vanilla k8s
What you mean is probably without vanilla k8s
Well, sorta. I'm not surprised it's k8s under the hood. But it's not a managed k8s product like GKE -- you don't have to _care_ that it's k8s under the hood. To the consumer, it presents like ECS or Lambda.
Run EC2 instances in an autoscaling group. Provision them with whatever you're comfortable with. Ansible is nice because you can point it at a node to test the deploy out without having to always respin the node to try again.
Have ansible fire up a docker container on your host.
That's it, done. You don't need Kubernetes.
Using EC2 segregates everything nicely out of the box. You don't need to be concerned about resource contention or figuring out how to size instances/pods. A container going nuts and filling up your disk won't cause a bunch of other services to experience errors.
While convenient for deploys, the biggest issue with Kubernetes is that you introduce an entire new layer of technology that you need expertise in.
If you're running your own EC2 instances anyway (which seems likely, since spot pricing is amazing compared to managed stuff) you now have to know how autoscaling works, plus a VPC, plus Kubernetes. The way Kubernetes interacts with cloud providers (at least AWS) adds a ton of complexity.
Also, if you have a decent build process for AMIs you can insulate yourself from container registry outages by baking the images into an AMI. There's something really powerful about this approach. If you build an AMI properly, that thing will run _forever_ with a minimal set of external service dependencies.
Have ansible fire up a docker container on your host.
That's it, done. You don't need Kubernetes.
Using EC2 segregates everything nicely out of the box. You don't need to be concerned about resource contention or figuring out how to size instances/pods. A container going nuts and filling up your disk won't cause a bunch of other services to experience errors.
While convenient for deploys, the biggest issue with Kubernetes is that you introduce an entire new layer of technology that you need expertise in.
If you're running your own EC2 instances anyway (which seems likely, since spot pricing is amazing compared to managed stuff) you now have to know how autoscaling works, plus a VPC, plus Kubernetes. The way Kubernetes interacts with cloud providers (at least AWS) adds a ton of complexity.
Also, if you have a decent build process for AMIs you can insulate yourself from container registry outages by baking the images into an AMI. There's something really powerful about this approach. If you build an AMI properly, that thing will run _forever_ with a minimal set of external service dependencies.
But then you need to make up a deployment system with rollbacks and all the other stuff k8s gives you.
I've got the feeling that if you're reasonably comfortable with k8s, there's no harm in starting with it from day 1.
I've got the feeling that if you're reasonably comfortable with k8s, there's no harm in starting with it from day 1.
> But then you need to make up a deployment system with rollbacks and all the other stuff k8s gives you.
This is borderline trivial. If you build an AMI, you update a launch template with the new AMI and use instance refresh (https://aws.amazon.com/blogs/compute/introducing-instance-re...) to replace instances. Rolling back is the same thing, you just set it back to the previous AMI. Instance refresh took way too long to become a thing, but it was not even very difficult before that. List instances in your autoscaling group, mark them unhealthy one by one.
Now, you'll probably want a couple other safety checks in your deployment process to make sure it all goes smoothly. But you're realistically looking at what, a couple of days of work to have something built to your specifications that can deploy any of your services without introducing the added complexity of Kubernetes?
> I've got the feeling that if you're reasonably comfortable with k8s, there's no harm in starting with it from day 1.
The harm is not that the deployments are hard. In fact, they're quite nice. The problem is that small teams don't have the bandwidth to keep up with the relentless release cadence. They may not have the expertise to get the upgrades done confidently.
There is also the fact that, when something does go wrong, you've got a whole lot of complex Kube stuff to churn through.
None of that applies to managed Kube, but then you're paying extra for Kube every month. By contrast, you could have just paid extra one time for a relatively simple deployment script in CI.
This is borderline trivial. If you build an AMI, you update a launch template with the new AMI and use instance refresh (https://aws.amazon.com/blogs/compute/introducing-instance-re...) to replace instances. Rolling back is the same thing, you just set it back to the previous AMI. Instance refresh took way too long to become a thing, but it was not even very difficult before that. List instances in your autoscaling group, mark them unhealthy one by one.
Now, you'll probably want a couple other safety checks in your deployment process to make sure it all goes smoothly. But you're realistically looking at what, a couple of days of work to have something built to your specifications that can deploy any of your services without introducing the added complexity of Kubernetes?
> I've got the feeling that if you're reasonably comfortable with k8s, there's no harm in starting with it from day 1.
The harm is not that the deployments are hard. In fact, they're quite nice. The problem is that small teams don't have the bandwidth to keep up with the relentless release cadence. They may not have the expertise to get the upgrades done confidently.
There is also the fact that, when something does go wrong, you've got a whole lot of complex Kube stuff to churn through.
None of that applies to managed Kube, but then you're paying extra for Kube every month. By contrast, you could have just paid extra one time for a relatively simple deployment script in CI.
Managed kube costs $0.10 per cluster per hour for GKE, and the first one is free. 72$ monthly is nothing.
Dealing with release cadence with GKE was mostly "hey, why the new version is not there yet" :)
About the only real issue I encountered was with major API change involving ingresses, when the cluster would continue running, but we had to update our manifests that were hamfistedly done back in 1.4 days.
About the only real issue I encountered was with major API change involving ingresses, when the cluster would continue running, but we had to update our manifests that were hamfistedly done back in 1.4 days.
This can go a very long way. Using Ansible and Packer to provision AMIs, and then Terraform to spin up what's needed, goes a long way, without introducing a new layer of technology.
Going by blog posts, 1.17 was released 2019-12-09, 1.18 was released 2020-03-25, 1.19 was released 2020-08-26, and 1.20 was released 2020-12-08 (today). Great cadence.
AWS EKS seems to support major versions six or seven months later. Support for 1.17 was announced 2020-07-13, 1.18 was announced 2020-10-13. I'm guessing we can plan for 1.20 support on EKS around June of 2021. No judgement, just an observation.
AWS EKS seems to support major versions six or seven months later. Support for 1.17 was announced 2020-07-13, 1.18 was announced 2020-10-13. I'm guessing we can plan for 1.20 support on EKS around June of 2021. No judgement, just an observation.
That cadence is also exhausting to follow, I wish they made an LTS sort of release so people can catch a break.
This is our pain.. the main reason why we are seeking other alternatives to K8S.
Hashicorp’s Nomad? Just hit 1.0.
That's something I'd be curious about. How do you feel a later migration from Nomad to k8s would go?
Straightforward imo (go containers first if this is your eventual intent), with the effort involved dependent on what your service mesh, networking, and storage situation and requirements looks like. More complexity is of course going to make such an effort more work.
Openshift does have LTS. The recently released version 4.6 is planned as one.
I feel you. Keeping up with kubernetes feels like a ratrace.
This is one of Kubernetes' biggest downfalls. On a small team, time is king. You likely don't have a team dedicated to keeping up with Kube releases, troubleshooting issues, etc.
When you build something, you need it to work until _you_ decide to futz with it again.
When you build something, you need it to work until _you_ decide to futz with it again.
supported are 3 minor releases with (1.18) 9months and newer minors 1year updates.
you should always upgrade to one version behind the last major in self managed scenarios. i.e. my company is on 1.18 and goes to 1.19 now. basically if something breaks it's most often not the end of the world, especially not if you do backups with velero.
I wonder when kubernetes will become commodity like docker became