Using Kubernetes to rethink your system architecture and ease technical debt(stackoverflow.blog)
stackoverflow.blog
Using Kubernetes to rethink your system architecture and ease technical debt
https://stackoverflow.blog/2021/05/19/rethinking-system-architecture-can-kubernetes-help-to-solve-rewrite-anxiety/
44 comments
Yea. I'm a VM type of admin. One time I got a whole lecture from a Docker Zealot about how I need to not treat my servers as pets. As if docker was the only path to that end. Never even bothered to talk about the existing infra that was managed like cattle - and that it was more a function of App-Architecture than Docker vs X.
Its nice to love your tools. Don't start thinking they are the one true path.
Its nice to love your tools. Don't start thinking they are the one true path.
> Never even bothered to talk about the existing infra that was managed like cattle
I don't see the point of your counterargument. For starters, there are plenty of managed containerized solutions that just run your containers without you ever having to bother with them. Secondly, it seems you got it entirely wrong: the point of pets vs cattle are the services, not the servers. You want to take your services down and put them up without surprises. Unless you're planning on rebooting your infrastructure just for shits and giggles, it makes no sense to talk about treating "servers" as pets, specially as you already get those as managed services just like you get VMs.
I don't see the point of your counterargument. For starters, there are plenty of managed containerized solutions that just run your containers without you ever having to bother with them. Secondly, it seems you got it entirely wrong: the point of pets vs cattle are the services, not the servers. You want to take your services down and put them up without surprises. Unless you're planning on rebooting your infrastructure just for shits and giggles, it makes no sense to talk about treating "servers" as pets, specially as you already get those as managed services just like you get VMs.
Yea, you missed it.
Agreed. Software is not hardware.
Behavioral patterns for a user are not tethered to one ephemeral implementation.
I ran a huge VMWare cluster in 3 data centers across the globe, not so different from an AWS/EKS pipeline.
Yes, I had to treat the hardware like pets to an extent. But the developers had no problem pivoting between VMWare and what we ran in AWS because the software tooling we provided looked the same.
Some folks believe memorizing YouTube videos = expertise.
Behavioral patterns for a user are not tethered to one ephemeral implementation.
I ran a huge VMWare cluster in 3 data centers across the globe, not so different from an AWS/EKS pipeline.
Yes, I had to treat the hardware like pets to an extent. But the developers had no problem pivoting between VMWare and what we ran in AWS because the software tooling we provided looked the same.
Some folks believe memorizing YouTube videos = expertise.
"One time I got a whole lecture from a Docker Zealot about how I need to not treat my servers as pets."
You can always point out that pods are ran on nodes. Nodes use the standard built in GCP Auto-scaler (instance groups + load balancer). So its a great wrapper! But, it's a wrapper. GCP also! has a way to use a docker image as your VM.
You can always point out that pods are ran on nodes. Nodes use the standard built in GCP Auto-scaler (instance groups + load balancer). So its a great wrapper! But, it's a wrapper. GCP also! has a way to use a docker image as your VM.
Agree and understood. You sound reasonable. IME zealots trade reason for enthusiasm and they focus too much on the method, not the concept.
This individual was shocked to learn I thought you could treat the iron-born servers as cattle too. (But, my first clusters were in the late 90s, so got to the cattle concept quite some time ago)
This individual was shocked to learn I thought you could treat the iron-born servers as cattle too. (But, my first clusters were in the late 90s, so got to the cattle concept quite some time ago)
> It's not clear to me that K8s was the actual thing that helped them scale, but rather the work they put into reworking their services.
It seems you're missing the forest for the trees. They put in the work to refactorize their services so that they could be treated as cattle and also autoscale, and the author stated that after doing his homework he determined that for his company Kubernetes offered the most benefits. There isn't much to read from this.
> Just to add on my point, wouldn't the result (in terms of performance / resources usage) be the same if instead of "containers" the author had chosen "AMIs" and instead of "Kubernetes" the author had opted for "Autoscale Groups"?
The author explains that he considered using EC2 instances but after analising the tradeoffs the company decided to just jump to EKS.
If you'd like to debate containers vs VMs then it's perfectly fine, but the author of this blog post just documented his learning path and pointed out what worked for him. Sometimes you need to call a shot, and the tradeoffs you need to consider are so minor and irrelevant that a coin flip will do.
It seems you're missing the forest for the trees. They put in the work to refactorize their services so that they could be treated as cattle and also autoscale, and the author stated that after doing his homework he determined that for his company Kubernetes offered the most benefits. There isn't much to read from this.
> Just to add on my point, wouldn't the result (in terms of performance / resources usage) be the same if instead of "containers" the author had chosen "AMIs" and instead of "Kubernetes" the author had opted for "Autoscale Groups"?
The author explains that he considered using EC2 instances but after analising the tradeoffs the company decided to just jump to EKS.
If you'd like to debate containers vs VMs then it's perfectly fine, but the author of this blog post just documented his learning path and pointed out what worked for him. Sometimes you need to call a shot, and the tradeoffs you need to consider are so minor and irrelevant that a coin flip will do.
If you are a Director of DevOps, what is easier to say to the CTO:
"I need X million USD in engineering spend this year because we need to be on Kubernetes".
"I need X million USD in engineering spend this year to fix half a decade of awful technical execution and decision making (that all happened under your watch, BTW)".
"I need X million USD in engineering spend this year because we need to be on Kubernetes".
"I need X million USD in engineering spend this year to fix half a decade of awful technical execution and decision making (that all happened under your watch, BTW)".
> awful technical execution
Leave that out. Many right decisions look stupid later. It will happen to you to. Time makes fools of us all.
Leave that out. Many right decisions look stupid later. It will happen to you to. Time makes fools of us all.
[deleted]
This. It’s always important to assume the decisions made before you were correct, or at least a good approximation to correct given the conditions or knowledge at the time. It’s sort of a generalization of chesterson’s fence. If you don’t know why a decision might have been smart at the time, you will likely make another mistake trying to correct it. It may not always be true but thinking this way can at least keep you honest.
The question was one in rhetorical jest, no? Clearly the former statement is "correct"
Neither:
"I need X million USD in engineering spend this year because we need to practice DevOps and SRE."
"I need X million USD in engineering spend this year because we need to practice DevOps and SRE."
"I need X million USD in engineering spend this year because it would improve my CV"
The author of this blog post states quite clearly that his company already used EKS elsewhere. If all he wanted to do was add a fancy line to his CV, why would he bother with something he already had?
Also, it's silly to just assume that continuing to treat pampered EC2 instances that had to be manually deployed ande cared for is something that benefits a projectsl, specially if the company already shed those bad practices elsewhere.
Also, it's silly to just assume that continuing to treat pampered EC2 instances that had to be manually deployed ande cared for is something that benefits a projectsl, specially if the company already shed those bad practices elsewhere.
This reads to me as:
‘I need X million in USD in engineering spend this year so I can add “I managed a large SRE/DevOps team” on my resume.’
‘I need X million in USD in engineering spend this year so I can add “I managed a large SRE/DevOps team” on my resume.’
"Hey! I have the code wrapped in k8s! it is now serving an endpoint - let's start giving it a small amount of traffic!"
> "I need X million USD in engineering spend this year to fix half a decade of awful technical execution and decision making (that all happened under your watch, BTW)".
I don't know what is there in the IT/dev industry that compels some characters to quickly whip out blanket accusations of gross incompetence to anyone who isn't them.
This mindset is particularly egregious given that the whole industry has enthusiastically adopted as a best practice just agilying big balls of mud through sequences of quickly delivered stopgap solutions while refactoring and paying off technical debt is seen as wasted effort.
The author of this blog post clearly described refactoring a legacy system that worked reliably in hopes to improve their operations. Yet, here you are accusing someone investing their time and effort to improve their stuff as something that could only be explained as incompetence?
I don't know what is there in the IT/dev industry that compels some characters to quickly whip out blanket accusations of gross incompetence to anyone who isn't them.
This mindset is particularly egregious given that the whole industry has enthusiastically adopted as a best practice just agilying big balls of mud through sequences of quickly delivered stopgap solutions while refactoring and paying off technical debt is seen as wasted effort.
The author of this blog post clearly described refactoring a legacy system that worked reliably in hopes to improve their operations. Yet, here you are accusing someone investing their time and effort to improve their stuff as something that could only be explained as incompetence?
For the purpose of my joke, it doesn't really matter if it's incompetence or intentional under-engineering.
It was a satirical observation (based on my experience) that "the business" tends to scoff at unsexy foundational rework of existing tech. The trendiness of Kubernetes is used effectively to "trojan horse" that kind of work in.
I am jokingly suggesting that at this stage maybe that's the real value of Kubernetes.
It was a satirical observation (based on my experience) that "the business" tends to scoff at unsexy foundational rework of existing tech. The trendiness of Kubernetes is used effectively to "trojan horse" that kind of work in.
I am jokingly suggesting that at this stage maybe that's the real value of Kubernetes.
Did the stackoverflow architecture change significantly from this series of posts? (seems to mentioning 2016-2019) [1]
The article mentions a rewrite, but it wasn't very specific on the scope of the rewrite, and what kind of stack they are using now.
I thought they were running their own farm with most of the action on IIS with ASP.net and MSSQL, and some elasticsearch for search.
[1] https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...
I thought they were running their own farm with most of the action on IIS with ASP.net and MSSQL, and some elasticsearch for search.
[1] https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar...
The article is a guest post from individuals that work at Pusher[1], about architectural decisions they made for their products and services.
I also thought it would be about StackOverflow until they started mentioning all the Pusher related products.
[1] https://pusher.com
I also thought it would be about StackOverflow until they started mentioning all the Pusher related products.
[1] https://pusher.com
Oh, thanks. Skipped that emtirely. (Too much HN for me)
No, Stackoverflow architecture is still a simple server farm with aggressively maintained code and a lite SQL ORM that lets them to highly optimize queries. One of the architects talked about it on their pod a few weeks ago. She mentioned explicitly how they don't jump on new hotness and keep it classic, maintainable, and highly performant without spending tens of millions of dollars on FAANG alumni who seem to think that every shop needs a Google infrastructure because THIS IS THE WAY.
> Kubernetes has great documentation so new starters can get up to speed more quickly even if they don’t have experience with it.
I strongly disagree. What feels easy for one developer may be super hard for others.
For me, the Kubernetes tutorial [1] and documentation [2] [3] feels borderline incomprehensible. It introduces several abstract concepts at once without telling me what they really mean. It raises more questions than it answers.
For example:
> Kubernetes coordinates a highly available cluster of computers that are connected to work as a single unit.
What does that even mean? Is that “single unit” the entire application? Or rather one component, e. g. the backend? You usually don’t have more than one service per container, right? So what exactly is the cluster here? Why is there not a single example?
> The Control Plane is responsible for managing the cluster. The Control Plane coordinates all activities in your cluster, such as scheduling applications, maintaining applications' desired state, scaling applications, and rolling out new updates.
We haven’t even established what the cluster is and how it relates to an application but now we’re already talking about several applications? I feel entirely lost.
Ok, let’s look at another introductory page [2].
It says things like:
> The worker node(s) host the Pods that are the components of the application workload.
But it won’t tell me what a Pod is. So I click on the Pods link [3].
It says:
> A Pod (as in a pod of whales or pea pod) is a group of one or more containers, with shared storage and network resources, and a specification for how to run the containers. A Pod's contents are always co-located and co-scheduled, and run in a shared context. A Pod models an application-specific "logical host": it contains one or more application containers which are relatively tightly coupled. In non-cloud contexts, applications executed on the same physical or virtual machine are analogous to cloud applications executed on the same logical host.
I find that explanation outright hostile. Not a single example that a newbie can understand. Instead, a pile-up of more and more abstract concepts. I still have no clue how a Pod, a node, an application, its instances, its components and a cluster relates. I’m giving up here.
This documentation is abysmal. How can the blog author possibly call this thing well-documented?
[1]: https://kubernetes.io/docs/tutorials/kubernetes-basics/creat...
[2]: https://kubernetes.io/docs/concepts/overview/components/
[3]: https://kubernetes.io/docs/concepts/workloads/pods/
I strongly disagree. What feels easy for one developer may be super hard for others.
For me, the Kubernetes tutorial [1] and documentation [2] [3] feels borderline incomprehensible. It introduces several abstract concepts at once without telling me what they really mean. It raises more questions than it answers.
For example:
> Kubernetes coordinates a highly available cluster of computers that are connected to work as a single unit.
What does that even mean? Is that “single unit” the entire application? Or rather one component, e. g. the backend? You usually don’t have more than one service per container, right? So what exactly is the cluster here? Why is there not a single example?
> The Control Plane is responsible for managing the cluster. The Control Plane coordinates all activities in your cluster, such as scheduling applications, maintaining applications' desired state, scaling applications, and rolling out new updates.
We haven’t even established what the cluster is and how it relates to an application but now we’re already talking about several applications? I feel entirely lost.
Ok, let’s look at another introductory page [2].
It says things like:
> The worker node(s) host the Pods that are the components of the application workload.
But it won’t tell me what a Pod is. So I click on the Pods link [3].
It says:
> A Pod (as in a pod of whales or pea pod) is a group of one or more containers, with shared storage and network resources, and a specification for how to run the containers. A Pod's contents are always co-located and co-scheduled, and run in a shared context. A Pod models an application-specific "logical host": it contains one or more application containers which are relatively tightly coupled. In non-cloud contexts, applications executed on the same physical or virtual machine are analogous to cloud applications executed on the same logical host.
I find that explanation outright hostile. Not a single example that a newbie can understand. Instead, a pile-up of more and more abstract concepts. I still have no clue how a Pod, a node, an application, its instances, its components and a cluster relates. I’m giving up here.
This documentation is abysmal. How can the blog author possibly call this thing well-documented?
[1]: https://kubernetes.io/docs/tutorials/kubernetes-basics/creat...
[2]: https://kubernetes.io/docs/concepts/overview/components/
[3]: https://kubernetes.io/docs/concepts/workloads/pods/
> Not a single example that a newbie can understand.
K8S (imo) isn’t for noobs. A car factory isn’t for noobs. Ideally you would want to understand how a car is made and works, how to run an assembly line, and how to build a factory communication systems before building and running the car factory. Similarly, you should understand how docker containers works, how to run infrastructure, and how to configure system networks and dns before building and running a K8S platform.
The analogy may not be perfect… K8S is complicated but not impossibly so. Practice, working with something like Rancher, and classes will go along way to help understand it.
K8S (imo) isn’t for noobs. A car factory isn’t for noobs. Ideally you would want to understand how a car is made and works, how to run an assembly line, and how to build a factory communication systems before building and running the car factory. Similarly, you should understand how docker containers works, how to run infrastructure, and how to configure system networks and dns before building and running a K8S platform.
The analogy may not be perfect… K8S is complicated but not impossibly so. Practice, working with something like Rancher, and classes will go along way to help understand it.
I've been in this business since the early 2000s when we did not have these toys and had to roll everything on our own. I myself had to set up a high availability cluster using, eh, I forget what it was called now, something like LVH, and "containers" using Linux jails.
K8S is absolutely not a sustainable thing with its learning curve, and it is destined to be abstracted by a "dumb" layer, sort of like Git has two APIs - for developers and for the users. In fact, I believe that is already happening.
Few companies can afford a team of devs doing care and feeding for this thing full time.
K8S is absolutely not a sustainable thing with its learning curve, and it is destined to be abstracted by a "dumb" layer, sort of like Git has two APIs - for developers and for the users. In fact, I believe that is already happening.
Few companies can afford a team of devs doing care and feeding for this thing full time.
Let’s say my app has two containers: container A serves a static page for my SPA, and container B is a back-end that exposes an API for A.
So which set of containers would be a pod here? And which would be the nodes? And what would be the cluster?
(For the sake of the example, let’s assume that both A and B scale independently.)
So which set of containers would be a pod here? And which would be the nodes? And what would be the cluster?
(For the sake of the example, let’s assume that both A and B scale independently.)
That would be two pods. Each pod would have one container. The nodes would be the underlying Infrastructure at the containers are running on, so you would have n number of worker nodes which are each a full system running either on premises, or in a cloud. K8S Will schedule those pods to run on those nodes… same as how docker containers run on your host system in development. The cluster is the K8S body representing the control plane nodes ( your mgmt computers running etcd and K8S backend ) and the worker nodes (where your pods, demons, statefull sets, containers, etc etc etc are running ).
Now if you configure it right, those two pods are two deployments. You can scale those up and have a load balancer sitting in front of them. That load balancer will expose app and send in traffic to the stateless containers in the cluster.
Now if you configure it right, those two pods are two deployments. You can scale those up and have a load balancer sitting in front of them. That load balancer will expose app and send in traffic to the stateless containers in the cluster.
Why not one pod with two containers?
You probably need to scale the two features — SPA serving and APIs — independently, so you’d deploy them in separate pods. Multi-pod containers should be so tightly coupled that they should always deploy together. If you can insert decoupling between containers, do it.
But it doesn't really make sense for them to scale independently, right? E.g., you would never want the FE to scale higher than the BE, because that would 'starve' the FE. And you wouldn't really want the BE to scale higher than the FE, because why waste the resources? If they scale together, that solves both problems.
Not at all. They both have different scaling characteristics - serving a static file (assuming you didn’t do it via a CDN for some reason) uses a lot less resources than serving a backend request. If it doesn’t, you’re doing something terribly wrong.
If you couple them like you’re suggesting then you’re wasting resources by scaling out your FE serving infrastructure needlessly.
And at some point you’ll possibly end up splitting your backend into multiple services. Would you couple a frontend container to each of those?
If you couple them like you’re suggesting then you’re wasting resources by scaling out your FE serving infrastructure needlessly.
And at some point you’ll possibly end up splitting your backend into multiple services. Would you couple a frontend container to each of those?
> If you couple them like you’re suggesting then you’re wasting resources by scaling out your FE serving infrastructure needlessly.
But if I'm coupling them in the same pod then am I really wasting resources? Aren't pods the equivalent of 'hosts' in the Kubernetes world, i.e. aren't resources allocated at the pod level? And as you said,
> serving a static file (assuming you didn’t do it via a CDN for some reason) uses a lot less resources than serving a backend request.
If I'm serving static files from the same pod as my backend, the pod shouldn't really need any more resources than if it was just serving the BE, right?
> And at some point you’ll possibly end up splitting your backend into multiple services. Would you couple a frontend container to each of those?
The most likely scenario in that case would be having a backend-for-frontend pattern, so the FE and its BFF would still go in the same pod, and the other services would probably go in different pods.
But if I'm coupling them in the same pod then am I really wasting resources? Aren't pods the equivalent of 'hosts' in the Kubernetes world, i.e. aren't resources allocated at the pod level? And as you said,
> serving a static file (assuming you didn’t do it via a CDN for some reason) uses a lot less resources than serving a backend request.
If I'm serving static files from the same pod as my backend, the pod shouldn't really need any more resources than if it was just serving the BE, right?
> And at some point you’ll possibly end up splitting your backend into multiple services. Would you couple a frontend container to each of those?
The most likely scenario in that case would be having a backend-for-frontend pattern, so the FE and its BFF would still go in the same pod, and the other services would probably go in different pods.
A pod is a collection of containers. The original suggestion was to pair them, so you’d have a backend container and a frontend container living within the same pod. This is wasteful for the reasons I listed.
You can shove the backend and the frontend into the same container, but it’s needless.
> The most likely scenario in that case would be having a backend-for-frontend pattern, so the FE and its BFF would still go in the same pod, and the other services would probably go in different pods.
They are still two different concerns and don’t need to both be served from the same pod (the BFF pattern doesn’t even remotely require this).
You of course can do whatever you want, but shoving different things together (even if coupled via an API) is an anti pattern.
You can shove the backend and the frontend into the same container, but it’s needless.
> The most likely scenario in that case would be having a backend-for-frontend pattern, so the FE and its BFF would still go in the same pod, and the other services would probably go in different pods.
They are still two different concerns and don’t need to both be served from the same pod (the BFF pattern doesn’t even remotely require this).
You of course can do whatever you want, but shoving different things together (even if coupled via an API) is an anti pattern.
In general, you wouldn't expect SPA or hybrid-SPA architectures to have scale coupling between the front-end and backend, because you'll be hitting specific APIs more aggressively than you'll be downloading the SPAs (even in a mixed-render environment). In addition, if your backend is serving multiple channels, then you've got another source of incoming requests, and it wouldn't be ideal to spinning up additional webservers when only the business logic tier needs to scale out.
In general, the "one container per pod, usually" rule works pretty well. Adding more pods does create additional cognitive complexity, but by the time you need k8s, you really should have someone, or a team of someones, responsible for dealing with that -- I find that too often folks reach for Kubernetes when they could use a simpler container strategy (RIP, somewhat, Docker Swarm).
In general, the "one container per pod, usually" rule works pretty well. Adding more pods does create additional cognitive complexity, but by the time you need k8s, you really should have someone, or a team of someones, responsible for dealing with that -- I find that too often folks reach for Kubernetes when they could use a simpler container strategy (RIP, somewhat, Docker Swarm).
Well there are use cases where this is appropriate, the common multi-pod design patterns -> https://matthewpalmer.net/kubernetes-app-developer/articles/...
Generally I don’t see them and where they might be useful, I end up using a K8S Cronjob instead.
But why? I want each pod to have its own liveness probe and be able to scale independently.
Generally I don’t see them and where they might be useful, I end up using a K8S Cronjob instead.
But why? I want each pod to have its own liveness probe and be able to scale independently.
Yeah not for noobs is another way of saying that the creator of that particular thing did not do a good job.
> rewrites should be avoided unless they are truly necessary
Which is always.
Which is always.
With apologies to Jamie Zawinski:
> Some people, when confronted with a problem, think "I know, I'll use Kubernetes." Now they have two problems.
Kubernetes can be a great solution to a set of problems, but it is not automatically that way. My company is in the midst of adding k8s support to a large legacy application and that work has turned into a deep dive into madness in some places. To be fair, we're making it hard on ourselves by supporting both k8s and the legacy environment, but the point stands.
And Helm doesn't help.
> Some people, when confronted with a problem, think "I know, I'll use Kubernetes." Now they have two problems.
Kubernetes can be a great solution to a set of problems, but it is not automatically that way. My company is in the midst of adding k8s support to a large legacy application and that work has turned into a deep dive into madness in some places. To be fair, we're making it hard on ourselves by supporting both k8s and the legacy environment, but the point stands.
And Helm doesn't help.
How many lines of code is k8s? Depending on it is adding those millions of lines to your tech debt.
Unless you need so many of the features, mostly k8s seems to be a way to increase complexity and build resumes. Yes, you need to automate building instances and service discovery, but neither of those tasks needs millions of lines of code.
Unless you need so many of the features, mostly k8s seems to be a way to increase complexity and build resumes. Yes, you need to automate building instances and service discovery, but neither of those tasks needs millions of lines of code.
Totally agree. This is why I avoid using Linux because it adds approximately 30 million lines of technical debt to my service, and I definitely don’t need all of the kernel APIs it offers.
So I build my own out of assembly. Now my REST api is free of technical debt!
So I build my own out of assembly. Now my REST api is free of technical debt!
The writer starts by saying that servers were treated as pets, which inherently means difficulty (as the author clears out). I assume that, at some point, their services got reworked to a shape where they could be deployed in servers treated as cattle. This is a work the author had to pay no matter if the final goal was Kubernetes, Autoscale Groups or any other solution.
Then the author explains how they got to the conclusion that they need to containerise their service(s) in order to be able to use K8s, which would then, in turn, allow them to scale properly (adding on top the Go rewrite that is mentioned at some point in the post). The author also explains that Autoscale Groups were discarded because of their complexity.
It's not clear to me that K8s was the actual thing that helped them scale, but rather the work they put into reworking their services. In my experience, once your service(s) are ready to be deployed in servers that can be managed as cattle, the benefits show up independently of the deployment solution that is used to deploy the service(s) (K8s, Autoscale Groups, etc...).
Just to add on my point, wouldn't the result (in terms of performance / resources usage) be the same if instead of "containers" the author had chosen "AMIs" and instead of "Kubernetes" the author had opted for "Autoscale Groups"?
If that were the case (which I believe it might have), the other thing to consider would be the difficulty of managing a K8s cluster vs the difficulty of managing an Autoscale Group. I've seen quite some horror stories about K8s[1] and I'd rather choose ASG over K8s.
[1]: https://k8s.af