Kubernetes for Developers Who Know How to Develop(blog.ali.dev)
blog.ali.dev
Kubernetes for Developers Who Know How to Develop
https://blog.ali.dev/engineering/2022/01/13/k8s-for-developers/
105 comments
> Things that exist in most modern clouds are being reinvented in K8s.
My more optimistic view is that vendor specific APIs are being standardized. Initial implementations have their issues, but as more people use them the cloud vendors will improve their offering.
My more optimistic view is that vendor specific APIs are being standardized. Initial implementations have their issues, but as more people use them the cloud vendors will improve their offering.
Uniform abstractions are necessarily leaky -- the adoption of the k8s standardization, such as it is, papers over implementation details that serious operators require visibility into.
All abstractions are leaky and require escape hatches. It's a normal part of api design and the maturity process of new tooling.
>> Things that exist in most modern clouds are being reinvented in K8s
The market wants cloud computing to be a commodity. I expect it will get what it wants via k8s or some other technology.
The market wants cloud computing to be a commodity. I expect it will get what it wants via k8s or some other technology.
The problem is that few people seem to understand the infrastructure as code concept, and essentially break the core k8s declarative architecture with imperative workflows that look just like the bash script install insanity we left behind. Workflows that are encouraged by tools like Helm and examples that create k8s objects on the fly without even creating much less retaining the "code" part of IaC.
It turns a tool that escaped the tyranny of endlessly mutating blessed servers with immutable, contained services and unified declarative life cycles, back into an imperative mess of magical incantations that must be spoken in just the right way.
Kubernetes is simple when used as designed, but staggeringly complicated when forced into the mutable imperative workflows it was expressly designed to prevent.
It turns a tool that escaped the tyranny of endlessly mutating blessed servers with immutable, contained services and unified declarative life cycles, back into an imperative mess of magical incantations that must be spoken in just the right way.
Kubernetes is simple when used as designed, but staggeringly complicated when forced into the mutable imperative workflows it was expressly designed to prevent.
Mind expanding a little on your complaints about Helm? I’ve only used Helm as a templating solution (and even then only to differentiate between local, staging and production), so I’m curious what problems I have to guard against.
Think of Kubernetes like a single application. The config files are the source for that application, the running cluster is the compiled application running on the users computer. By default Helm injects more "compiled" code unrelated to your applications source into the running application. Allowing any tool to alter active cluster state diffuses your single source of truth, your source code, to multiple sources of truth which will not remain in sync with your source unless great care is taken. Moving in sync matters, because that is how you roll back to a known good state when things break.
If you are using Helm to generate source code for your application you still have the added complexity of additional build step, but at least you can choose to add the generated code to your app in a way that tracks with the rest of your code.
Also most Helm chart authors are of varying skill level, and even skilled ones necessarily make incorrect assumptions about your deployment environment. It takes a lot of addition code in helm charts to support more flexibility, so it often get ignored, and you are left with a black box that doesn't quite do what you'd want it to do.
If you are using Helm to generate source code for your application you still have the added complexity of additional build step, but at least you can choose to add the generated code to your app in a way that tracks with the rest of your code.
Also most Helm chart authors are of varying skill level, and even skilled ones necessarily make incorrect assumptions about your deployment environment. It takes a lot of addition code in helm charts to support more flexibility, so it often get ignored, and you are left with a black box that doesn't quite do what you'd want it to do.
> IoC
IaC?
IaC?
Yes, fixed. Thank you
What does IaC stand for? That’s a new acronym to me..
You know the mountains of legacy code locked in to Oracle stuff? The way I see it, k8s is the way for us to avoid making the same mistake with AWS.
At the very least we can actually simulate our stack locally.
At the very least we can actually simulate our stack locally.
With the best possible intentions, the above is very much not a good overview of the current K8s ecosystem.
K8s is hugely supported and invested in by Alibaba, and TONS of companies you've never heard of.
All of these companies have huge, mission critical reasons not to use AWS/Azure/GCP and invest in K8s.
From an end user's standpoint, I am certainly not going to dispute your criticisms about Helm.
Google's original design of K8s seems like it was originally designed to be generically worse than GCP.
However, from an ecosystem standpoint, K8s is a project I'm very confident will be around for a long time.
K8s is hugely supported and invested in by Alibaba, and TONS of companies you've never heard of.
All of these companies have huge, mission critical reasons not to use AWS/Azure/GCP and invest in K8s.
From an end user's standpoint, I am certainly not going to dispute your criticisms about Helm.
Google's original design of K8s seems like it was originally designed to be generically worse than GCP.
However, from an ecosystem standpoint, K8s is a project I'm very confident will be around for a long time.
> All of these companies have huge, mission critical reasons not to use AWS/Azure/GCP and invest in K8s.
Very true. The problems occur when smaller companies without these mission critical reasons make the same choices because the others are making them.
Very true. The problems occur when smaller companies without these mission critical reasons make the same choices because the others are making them.
For smaller companies, Digital Ocean Managed Kubernetes or or Google Cloud Run or their AWS equivalent are all very good choices.
But yes, "roll your own K8s" is for larger organizations.
But yes, "roll your own K8s" is for larger organizations.
> Things that exist in most modern clouds are being reinvented in K8s
I should fucking hope so. I don't want to change my helm charts because I'm deploying on Azure for this particular client.
I should fucking hope so. I don't want to change my helm charts because I'm deploying on Azure for this particular client.
In case you didn't know of it, for IaC, CDK8S has worked very well for me.
author here: interesting - maybe you should also write a blog post about it. would love to learn more about your thoughts in depth!
> Kubernetes x Docker: Best Friends Forever
> Whenever you hear Kubernetes, you’re probably going to hear Docker in the same sentence. Kubernetes and Docker are like peanut butter and jelly—they’re a perfect pair.
For what it's worth, this article was originally written in October 2021, but even at that point it was clear that Kubernetes would remove support for docker. In fact it has already with Kubernetes 1.24. Not exactly "Best Friends Forever". I wish people writing this kind of introductory blog posts would take the time and just quickly explain container runtimes instead of just mentioning docker and be done with it.
> Whenever you hear Kubernetes, you’re probably going to hear Docker in the same sentence. Kubernetes and Docker are like peanut butter and jelly—they’re a perfect pair.
For what it's worth, this article was originally written in October 2021, but even at that point it was clear that Kubernetes would remove support for docker. In fact it has already with Kubernetes 1.24. Not exactly "Best Friends Forever". I wish people writing this kind of introductory blog posts would take the time and just quickly explain container runtimes instead of just mentioning docker and be done with it.
While you are technically correct, kubernetes removed the Dockershim. Docker is often used as a synonym for containers. However, most of the things should still work [1].
[1]: https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-...
[1]: https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-...
I am very much aware of this, but that's exactly what I am arguing. Articles setting out to explain Kubernetes should not perpetuate the false equality of docker and containers.
If you use `docker build/run` in your bash, it will actually be more confusing IMO saying k8s "dropped docker(shim) support". This fact should be a footnote for the curious reader, surely on an introduction article.
It would do the industry a world of good to move past conflating Docker with Linux containers, containerization, images, etc.
There are tons of ways to build containers without Docker. You're effectively encouraging propagating confusion.
There are tons of ways to build containers without Docker. You're effectively encouraging propagating confusion.
Indeed, and I have zero idea why people don’t use something like LXC and still prefer Docker. Marketing is hell sometimes.
> remove support for docker
So I can't use Dockerfiles and `docker build` to build things for k8s?
So I can't use Dockerfiles and `docker build` to build things for k8s?
This is not correct.
Kubernetes used docker engine under the hood in the past. Now they abstracted useful part of this engine into API. There's implementation from the docker (containerd), there's implementation from Redhat (CRI-O), may be others. Docker don't have to be installed for Kubernetes to work anymore.
Building container images is a different topic. Kubernetes does not have anything to offer here. So you probably still need docker in your development machine and in your CI pipeline to build those images. There are plenty of alternatives rised in recent years, most prominent ones are kaniko, buildah/podman, but they're far from docker in their maturity.
That actually makes a problem. It's hard to run docker and kubernetes side-by-side. Or docker inside kubernetes. So if you want to run your CI jobs inside Kubernetes, there's no good solutions right now.
I think people will eventually migrate to Kaniko. It's from Google, it seems to be a sane approach. But right now it's a mess.
Kubernetes used docker engine under the hood in the past. Now they abstracted useful part of this engine into API. There's implementation from the docker (containerd), there's implementation from Redhat (CRI-O), may be others. Docker don't have to be installed for Kubernetes to work anymore.
Building container images is a different topic. Kubernetes does not have anything to offer here. So you probably still need docker in your development machine and in your CI pipeline to build those images. There are plenty of alternatives rised in recent years, most prominent ones are kaniko, buildah/podman, but they're far from docker in their maturity.
That actually makes a problem. It's hard to run docker and kubernetes side-by-side. Or docker inside kubernetes. So if you want to run your CI jobs inside Kubernetes, there's no good solutions right now.
I think people will eventually migrate to Kaniko. It's from Google, it seems to be a sane approach. But right now it's a mess.
This is correct.
K8s abstracted out containerization in preparation for the anticipated long term migration of the ecosystem to Podman et al (lots and lots of people are invested in moving "containers" away from "Docker").
K8s abstracted out containerization in preparation for the anticipated long term migration of the ecosystem to Podman et al (lots and lots of people are invested in moving "containers" away from "Docker").
I don't think podman is designed or supported as a CRI. It's for local development. You can create local "pods" without an API server, but not running containers on a real Kubernetes node. That would be containerd, crio, or cri-dockerd.
With that said I still prefer docker for local development by creating real ephemeral clusters with kind
With that said I still prefer docker for local development by creating real ephemeral clusters with kind
Author here! Yeah, the article is out of date but thats okay -
honestly, your suggestion for container runtimes is a great idea - it's now on my list of blog posts to write! I'll be sure to credit you :)
honestly, your suggestion for container runtimes is a great idea - it's now on my list of blog posts to write! I'll be sure to credit you :)
Does this apply to GKE? Are they going to remove support for Docker as well (or did they?)
They have, with 1.24[1], following Kubernetes upstream. If you are an end user nothing is really going to change from this though, because docker produces container images that can be used in other container runtimes (and in fact, containerd is what powers docker underneath today anyway). But that's why it's so important to explain that containers != docker, otherwise "removing docker support" seems really scary.
[1] https://cloud.google.com/kubernetes-engine/docs/deprecations...
[1] https://cloud.google.com/kubernetes-engine/docs/deprecations...
> They have
The question was did they remove support for docker. They have not.
They simply removed dockershim and no longer special case it.
Your point is that we should stop using a brand name to describe OCI open standard containers. However, like aspirin, the names are interchangable for the vast majority of people.
OCI container doesn't really roll off the toung, and container by itself could mean all sorts of things.
Docker is really the best word to describe what people are thinking about it every day conversation. So much so that I think they are at risk of loose their trademark.
The question was did they remove support for docker. They have not.
They simply removed dockershim and no longer special case it.
Your point is that we should stop using a brand name to describe OCI open standard containers. However, like aspirin, the names are interchangable for the vast majority of people.
OCI container doesn't really roll off the toung, and container by itself could mean all sorts of things.
Docker is really the best word to describe what people are thinking about it every day conversation. So much so that I think they are at risk of loose their trademark.
> The question was did they remove support for docker. They have not.
Have they not? Note that the GP asked for GKE specifically. The support page I linked to literally says so:
> GKE will stop supporting node images that use Docker as the runtime in GKE version 1.24 and later
Removing dockershim removed the existing support for docker, because docker does not support CRI (Container Runtime Interface), the API required by Kubernetes. You can go through a third-party solution that adds CRI support on top of docker, but most managed Kubernetes offerings simply removed docker support.
I don't see any argument supporting the claim that docker is the "best word" to describe containers. I am also not aware of ambiguity for the term "(Linux) container" when it comes to operating/deploying software. What else does it mean in that context?
Have they not? Note that the GP asked for GKE specifically. The support page I linked to literally says so:
> GKE will stop supporting node images that use Docker as the runtime in GKE version 1.24 and later
Removing dockershim removed the existing support for docker, because docker does not support CRI (Container Runtime Interface), the API required by Kubernetes. You can go through a third-party solution that adds CRI support on top of docker, but most managed Kubernetes offerings simply removed docker support.
I don't see any argument supporting the claim that docker is the "best word" to describe containers. I am also not aware of ambiguity for the term "(Linux) container" when it comes to operating/deploying software. What else does it mean in that context?
Saying "Kubernetes has removed support for Docker" is incredibly misleading at best, and less charitably, is just plainly wrong.
While it's true that 1.24 does not support docker as the specific container runtime that's directly used by Kubernetes itself, this has approximately zero impact on how the vast majority of beginners would use Kubernetes, as out of the box you're still able to run docker containers.
Probably not the kind of confusing detail that needs to be in an intro to Kubernetes article.
While it's true that 1.24 does not support docker as the specific container runtime that's directly used by Kubernetes itself, this has approximately zero impact on how the vast majority of beginners would use Kubernetes, as out of the box you're still able to run docker containers.
Probably not the kind of confusing detail that needs to be in an intro to Kubernetes article.
For what it’s worth, the subthread you are commenting on specifically talks about GKE. GKE is removing docker support, period. That’s not plainly wrong, it’s in their support document.
For most users, using docker through a CRI compatibility layer is not an option as they use some sort of Managed Kubernetes, and I am curious to hear which of those keeps supporting docker as container runtime.
For most users, using docker through a CRI compatibility layer is not an option as they use some sort of Managed Kubernetes, and I am curious to hear which of those keeps supporting docker as container runtime.
> as out of the box you're still able to run docker containers.
No, you're able to run containers from images produced by docker provided it exports them in OCI format. At no point does k8s see anything to do with docker. Saying docker when you mean container or container image is incredibly misleading at best, and less charitably, is just plainly wrong.
Edit: Actually, it looks like if you want to add docker it's easy - https://kubernetes.io/docs/setup/production-environment/cont... - but you would have to install that support since it is not included out of the box.
No, you're able to run containers from images produced by docker provided it exports them in OCI format. At no point does k8s see anything to do with docker. Saying docker when you mean container or container image is incredibly misleading at best, and less charitably, is just plainly wrong.
Edit: Actually, it looks like if you want to add docker it's easy - https://kubernetes.io/docs/setup/production-environment/cont... - but you would have to install that support since it is not included out of the box.
It seems like your view is informed from a cursory reading of the docs, rather than any first-hand experience.
Docker produces OCI images, there's no need to "export them" in that format.
So since Kubernetes can run any OCI image, and Docker images are OCI, Kubernetes supports running Docker images out of the box.
The documentation you linked to is if you wanted to swap out the container runtime Kubernetes is using, not if you just want to run a Docker image.
Docker produces OCI images, there's no need to "export them" in that format.
So since Kubernetes can run any OCI image, and Docker images are OCI, Kubernetes supports running Docker images out of the box.
The documentation you linked to is if you wanted to swap out the container runtime Kubernetes is using, not if you just want to run a Docker image.
> It seems like your view is informed from a cursory reading of the docs, rather than any first-hand experience.
Fun though personal attacks are, you would be wrong; I have done all of this, including using docker to build images and k8s to run them.
> Docker produces OCI images, there's no need to "export them" in that format.
Sure.
> So since Kubernetes can run any OCI image, and Docker images are OCI, Kubernetes supports running Docker images out of the box.
...Kubernetes supports running OCI images out of the box. That they were built by docker does not make them docker images, any more than building a Windows program with MinGW creates a "Linux program" just because it was compiled on Linux. If you use docker to build an OCI image and then create a container from that image in a stock k8s cluster, you are not creating a docker container, you are creating a (most likely) containerd container from an OCI image.
> The documentation you linked to is if you wanted to swap out the container runtime Kubernetes is using, not if you just want to run a Docker image.
Yes, I was attempting to charitably include the case where your claim could still be correct. (Since by adding the docker runtime you can create a k8s cluster that creates docker containers.)
Fun though personal attacks are, you would be wrong; I have done all of this, including using docker to build images and k8s to run them.
> Docker produces OCI images, there's no need to "export them" in that format.
Sure.
> So since Kubernetes can run any OCI image, and Docker images are OCI, Kubernetes supports running Docker images out of the box.
...Kubernetes supports running OCI images out of the box. That they were built by docker does not make them docker images, any more than building a Windows program with MinGW creates a "Linux program" just because it was compiled on Linux. If you use docker to build an OCI image and then create a container from that image in a stock k8s cluster, you are not creating a docker container, you are creating a (most likely) containerd container from an OCI image.
> The documentation you linked to is if you wanted to swap out the container runtime Kubernetes is using, not if you just want to run a Docker image.
Yes, I was attempting to charitably include the case where your claim could still be correct. (Since by adding the docker runtime you can create a k8s cluster that creates docker containers.)
The install instructions still explicitly lists docker engine as container runtime.
https://kubernetes.io/docs/setup/production-environment/cont...
The dockershim removal FAQ says how to continue to use docker engine.
https://kubernetes.io/blog/2022/02/17/dockershim-faq/
-- Linux container could be LXC, systemd-nspawn, snap, flatpak, nixos-container, and many many many other things.
That's because Linux containers are built on Linux interfaces, but Linux itself does not have any prescriptive requirements on how to stich them together.
https://kubernetes.io/docs/setup/production-environment/cont...
The dockershim removal FAQ says how to continue to use docker engine.
https://kubernetes.io/blog/2022/02/17/dockershim-faq/
-- Linux container could be LXC, systemd-nspawn, snap, flatpak, nixos-container, and many many many other things.
That's because Linux containers are built on Linux interfaces, but Linux itself does not have any prescriptive requirements on how to stich them together.
I’ve been working on a Kubernetes cluster for my home network. I know a lot of people hate on Kubernetes, but it’s pretty cool and not too bad to even get started from scratch with Kubeadm.
I think the constant thought in the back of my mind is… Docker containers and Nginx is definitely enough for my needs. At work, it’s a different story, but more most things, it can be over kill.
I think the constant thought in the back of my mind is… Docker containers and Nginx is definitely enough for my needs. At work, it’s a different story, but more most things, it can be over kill.
My biggest issue with Kubernetes is huge costs of running it. It's 3 servers with 4 GB RAM. It's almost $100/month just for Kubernetes alone. And you need one load balancer for control nodes which is another $25 for my hoster. And you need second load balancer for production workloads which is another $25. So you have to pay $150/month for the privilege of using Kubernetes. And they double your costs for your application, because you want it to be high-available, why bother with Kubernetes otherwise.
So something that could be run with $5 droplet, suddenly requires $200.
They you want to install grafana, ELK, service mesh and whatnot.
So something that could be run with $5 droplet, suddenly requires $200.
They you want to install grafana, ELK, service mesh and whatnot.
Three machines with 4GB of RAM for $100? I don't know where you rent your servers but that's ridiculous. You can easily get that kind of compute for a third or lower.
You won't get a private LAN (so you'll need some firewall rules on the hoster side) and you'll have to do some setup to get everything served up on the right public IP address(es), but you don't need a dedicated load balancer product. Unless you expect your virtual servers to go down randomly, I suppose.
I've set up k0s on Oracle's free service to mess with. It doesn't need nearly as much as 12GiB of RAM. Give it 500MiB per machine to be happy and it _should_ run fine with 10.5GiB to spare unless you're adding tons of overlays and additional services on top of your applications.
As far as I can tell, the biggest cost with Kubernetes is maintaining the system and keeping up to date with the advancements and deprecations in the Kubernetes field. You can't just upgrade the Kubernetes version and expect everything to work, and every new concept that replaces anything old comes with Kubernetes layers of complexity.
I honestly don't know what so many companies are using all those resources Kubernetes provides for. I have a feeling people are so caught up in the "modern server management" world that they've forgotten how powerful and reliable a $25 VPS is these days.
You won't get a private LAN (so you'll need some firewall rules on the hoster side) and you'll have to do some setup to get everything served up on the right public IP address(es), but you don't need a dedicated load balancer product. Unless you expect your virtual servers to go down randomly, I suppose.
I've set up k0s on Oracle's free service to mess with. It doesn't need nearly as much as 12GiB of RAM. Give it 500MiB per machine to be happy and it _should_ run fine with 10.5GiB to spare unless you're adding tons of overlays and additional services on top of your applications.
As far as I can tell, the biggest cost with Kubernetes is maintaining the system and keeping up to date with the advancements and deprecations in the Kubernetes field. You can't just upgrade the Kubernetes version and expect everything to work, and every new concept that replaces anything old comes with Kubernetes layers of complexity.
I honestly don't know what so many companies are using all those resources Kubernetes provides for. I have a feeling people are so caught up in the "modern server management" world that they've forgotten how powerful and reliable a $25 VPS is these days.
The simplest and cheapest self hosted K8s I found is kube-Hetzner [1]. It puts k3s on MicroOS on Hetzner Cloud; for a three node Cluster I pay around 50 EUR per month (3shared AMD Cores, 4GB RAM per Node, 200 GB Storage, 1 TB Backup, small loadbalancer). A single server with the same spec and the same storage and backup and a floating IP would go for about 25 EUR/month. For me, k8s is worth the difference.
[1] https://registry.terraform.io/modules/kube-hetzner/kube-hetz...
[1] https://registry.terraform.io/modules/kube-hetzner/kube-hetz...
You don't actually need all that. If the competition is a single $5 machine with no redundancy, you can just run a single node cluster.
Do you think running k8s at home is worthwhile, past the learning experience and cool factor?
I have a nice server that I used for 10-20 applications like Time Machine backup, media streaming, file syncing, home automation, etc. I was looking into k8s, but it seemed way too complicated when I only have one physical server, so I went with Docker + Docker Compose instead.
I have a nice server that I used for 10-20 applications like Time Machine backup, media streaming, file syncing, home automation, etc. I was looking into k8s, but it seemed way too complicated when I only have one physical server, so I went with Docker + Docker Compose instead.
All the maintenance you put towards getting things running right now will be maintenance you put towards making sure your cluster is doing what you need and want.
If you start having more servers, a k8s cluster can do something like automatically transfer workloads when one machine has an issue for example, or dynamically allocating storage (even cloud) when needed, or allowing you to issue a single command to spin up more instance of whatever task you'd want to run more in parallel, or updating a version in a config file and letting the cluster do the job of stopping/updating/restarting things.
But what you also have to do is ensure the underlying networking between machines works, all versions of the binaries on all the machines of the cluster are in close agreement with each others, encryption of data via certificates and CAs is satisfactory, and your master(s) might require different maintenance from the workers - but you'll end up reading the docs to know how to roll updates step by step.
In your use case I would stick with a smart use of docker-composes, perhaps just centralized in a private repo, with cron jobs on top. If you buy/rent a couple machines of any kind (from raspberry pis to cloud instances) to put on your network and run things for you, then a cluster becomes increasingly interesting as the amount of tasks needed/wanted and amount of computing available grows.
My context: I have some limited experience with work (including some scary "abstractions" on top of K8s & Helm) and a recent CKA (certified kubernetes administrator hands-on certification) from LinuxFoundation.
If you start having more servers, a k8s cluster can do something like automatically transfer workloads when one machine has an issue for example, or dynamically allocating storage (even cloud) when needed, or allowing you to issue a single command to spin up more instance of whatever task you'd want to run more in parallel, or updating a version in a config file and letting the cluster do the job of stopping/updating/restarting things.
But what you also have to do is ensure the underlying networking between machines works, all versions of the binaries on all the machines of the cluster are in close agreement with each others, encryption of data via certificates and CAs is satisfactory, and your master(s) might require different maintenance from the workers - but you'll end up reading the docs to know how to roll updates step by step.
In your use case I would stick with a smart use of docker-composes, perhaps just centralized in a private repo, with cron jobs on top. If you buy/rent a couple machines of any kind (from raspberry pis to cloud instances) to put on your network and run things for you, then a cluster becomes increasingly interesting as the amount of tasks needed/wanted and amount of computing available grows.
My context: I have some limited experience with work (including some scary "abstractions" on top of K8s & Helm) and a recent CKA (certified kubernetes administrator hands-on certification) from LinuxFoundation.
If you were to bring on two more servers it would probably be worth it. 3 is the magic number for when k3s starts to bring more interesting stuff to the table in terms of reliability and dynamic behaviors.
If I was in your situation, probably not. We use it a lot at work, so it’s a playground to grow my skills and play around while doubling as a painless way to host anything.
All in all, I still think docker, docker compose and nginx is the way to go for hobbyist stuff. I doubt my ISP would let me host anything at a huge scale off my home router anyways.
All in all, I still think docker, docker compose and nginx is the way to go for hobbyist stuff. I doubt my ISP would let me host anything at a huge scale off my home router anyways.
nginx is great, but I've also been hearing good things about caddy[0]. I want to check it out soon.
[0]: https://caddyserver.com/v2
[0]: https://caddyserver.com/v2
Why does everyone keep thinking developers need to now about K8S? Just be glad if your infra/DevOps/platform teams abstract this away from you and you don't have to deal with this insanity.
Not every org delegates these responsibilities to separate roles.
if you're at the size of needing K8S, you can afford a devops/cloud engineering team.
To OP's point, this seems to be a trend among corporate blogs. Targeting devs with irrelevant content. A few weeks back I remember seeing a "SOC 2 for devs" blog article and was really scratching my head. That's not the level that devs work at, at all. If you're paying six figure salaries to devs to sit around all day worrying about overweight bureaucratic nonsense, or other tasks way beyond their expertise, then you're doing it wrong. I'm guessing these orgs think devs has some decision making influence or perhaps it looks good for recruiting? Or maybe they just want that HN juice. What next, I wonder? Neurosurgery for devs?
To OP's point, this seems to be a trend among corporate blogs. Targeting devs with irrelevant content. A few weeks back I remember seeing a "SOC 2 for devs" blog article and was really scratching my head. That's not the level that devs work at, at all. If you're paying six figure salaries to devs to sit around all day worrying about overweight bureaucratic nonsense, or other tasks way beyond their expertise, then you're doing it wrong. I'm guessing these orgs think devs has some decision making influence or perhaps it looks good for recruiting? Or maybe they just want that HN juice. What next, I wonder? Neurosurgery for devs?
No. Kubernetes is extremely valuable to small orgs. Google Kubernetes Engine is pretty much fire & forget. You turn it on, set the maintenance window and it just works. It also saves you from having to reinvent health checks, ingresses, etc. It's actually less work than maintaining plain old servers when you factor in maintenance, and since state is immutable you don't run into issues where some developer fixed an issue 3 years ago and no one knows why it works.
At a small org, what is the advantage of Kubernetes over the vendor provided application runners i.e. AWS AppRunner or Google's App Engine?
Having the possibility to test stuff locally using e.g. minikube. Also in general we just build everything around docker containers. Doesn't matter if they run locally on a dev machine or in a gke cluster.
Both of these articles were actually interesting to me as a dev who built a b2b SaaS company.
What’s SOC 2? Security Operations Center?
You'll still have to write all the yamls to deploy your app to k8s. Your ops team wont be analysing those requirements for your app.
It's laughable to think every small company has infra/DevOps/platform teams.
Also, in general, it's very useful to know the platform your software is going to be deployed to, so that you can design the software with those constraints in mind.
Also, in general, it's very useful to know the platform your software is going to be deployed to, so that you can design the software with those constraints in mind.
I'm just confused why a small company would allocate ~20% (at least) of their engineering hours to maintaining something like this? The 20% number comes from updating containers, checking for security vulnerabilities, rotating secrets, monitoring health, re-deploying after an outage, documenting build processes, creating CI/CD pipelines, writing YAML, etc. etc. I just don't see why you would do that when you could pay a tiny amount of money in comparison for a managed service.
After you have the money for devops, sure, seems great, many benefits. Or if you are just doing it for fun and want to learn, also fine. Or if you have some requirement (health data or something) then great.
But that 20~ has to be the lowest leverage thing possible to spend that time and money on if you are just a regular SaaS, marketplace, <name your startup type> start up. I am genuinely interested how I would get a better return on that rather than fixing bugs, listening to customers, building features, etc.
The only thing I ever hear is "In 5 years you might be locked in and you will regret it." When, in reality, if most startups lasted 5 years, they would be ecstatic...
After you have the money for devops, sure, seems great, many benefits. Or if you are just doing it for fun and want to learn, also fine. Or if you have some requirement (health data or something) then great.
But that 20~ has to be the lowest leverage thing possible to spend that time and money on if you are just a regular SaaS, marketplace, <name your startup type> start up. I am genuinely interested how I would get a better return on that rather than fixing bugs, listening to customers, building features, etc.
The only thing I ever hear is "In 5 years you might be locked in and you will regret it." When, in reality, if most startups lasted 5 years, they would be ecstatic...
Same reason you would deploy to plain old servers. Your stuff has got to run somewhere. And (Google hosted) kubernetes solves a lot of issues for you out of the box, which you would otherwise have to solve eventually anyway. And it takes basically no additional maintenance, if you use a mature managed service.
You can have your developers manually copy paste files to a server, which is great while it works, but it's far better to have an automated process. At that point it's not really much more work, in fact I'd argue it's less, to pick docker + kubernetes than any of the alternatives.
You can have your developers manually copy paste files to a server, which is great while it works, but it's far better to have an automated process. At that point it's not really much more work, in fact I'd argue it's less, to pick docker + kubernetes than any of the alternatives.
Managed Kubernetes (e.g GKE) is a hosted service, just a very flexible one. There is a gap between everything runs on Heroku and Big Enough to have a DevOps team that is pretty large. You tend to outgrow heroku pretty quick if your app has much complexity at all.
Used to work somewhere that just used some VPSes instead of managed hosting. That 20% time was spent on debugging OS issues, disk space issues, OS package updates, downtime needed to increase instance sizing, documenting (read: not actually documenting just leaving it up to the next person) how to rebuild the environment etc etc.
Used to work somewhere that just used some VPSes instead of managed hosting. That 20% time was spent on debugging OS issues, disk space issues, OS package updates, downtime needed to increase instance sizing, documenting (read: not actually documenting just leaving it up to the next person) how to rebuild the environment etc etc.
Honest question: why do we need wholly separate devops/infra teams as opposed to just one expert to teach the other devs if K8s was supposed to make IaC so efficient?
The same reason frontend and backend teams exist. At a certain point it makes sense to have centralized teams focus on specific verticals than requiring everyone to learn/know about everything. As with any tools, there are right and wrong ways of using it and using the wrong tool for the job can sometimes be more painful than not using it at all.
the problem I have with k8s or whole concept of "cloud native" is it almost ONLY focus on web based application, if your application is not HTTP based or "stateless", it is very hard to run or design to be "cloud native" , specially in a managed k8s service(like eks/aks).
If you want highly scalable at Google's scale, HTTP-based stateless services that can be scaled up/down in response to traffic are highly desirable properties.
The real problem is k8s and "cloud native" are designed for Google's scale, not yours.
The real problem is k8s and "cloud native" are designed for Google's scale, not yours.
The benefit of Kubernetes for most these days is the standardized api, not the ability to "webscale" although it certainly can. Google itself doesn't use k8s internally anyway - it uses borg. The CNCF oversees the Kubernetes project, not Google.
[deleted]
Can you tell me what specific challenges you've had with deploying non-HTTP services using Kubernetes? Many features are http-centric, but others seem fairly agnostic to me. What kind of non-http services are we talking about anyway?
Deploying them is easy. Figuring out the correct incantations to bind a load balancer or ingress port to redis/kafka/mysql/whatever and/or expose them in some way other than "let the ingress figure out the host from headers/SNI and map to some thing in the Service" is somewhat less than intuitive.
Sure, you could argue that you shouldn't expose backend services like that without an API endpoint at all, because one application->one namespace, or you could use a service mesh/consul to expose it to the things that need it, but it's an entirely different mindset from "I have a postgres cluster in public/private cloud"
Sure, you could argue that you shouldn't expose backend services like that without an API endpoint at all, because one application->one namespace, or you could use a service mesh/consul to expose it to the things that need it, but it's an entirely different mindset from "I have a postgres cluster in public/private cloud"
LoadBalancerIP seems pretty straightforward to me. No Ingress or Gateway needed. Just a plain old Service.
I don't think that would cause issues with long lived connections. It is about as direct as you can get while actually load balancing.
https://cloud.google.com/kubernetes-engine/docs/how-to/servi...
I don't think that would cause issues with long lived connections. It is about as direct as you can get while actually load balancing.
https://cloud.google.com/kubernetes-engine/docs/how-to/servi...
The Gateway API is entering beta! It'll resolve pretty much what you're aiming for.
Not OP and I don't do K8 in production, but I understand their complaint. You couldn't eg. deploy a game server easily.
They typical "web app" flow is Client -> ReverseProxy -> StatelessApp -> DatabaseWithState and back. The typically expectation is HTTP requests that aren't "long lived". Even running a database would be harder since you need to consider where the storage of the content is going (you'd want backups and snapshots etc) and how it'd get there. If the StatelessApp goes down (or you need more), you just add containers.
One example that wouldn't work well on k8 that I'm sure isn't very common is like a Minecraft server. It is a persistent TCP connection to a server that reads writes data to the FileSystem.
They typical "web app" flow is Client -> ReverseProxy -> StatelessApp -> DatabaseWithState and back. The typically expectation is HTTP requests that aren't "long lived". Even running a database would be harder since you need to consider where the storage of the content is going (you'd want backups and snapshots etc) and how it'd get there. If the StatelessApp goes down (or you need more), you just add containers.
One example that wouldn't work well on k8 that I'm sure isn't very common is like a Minecraft server. It is a persistent TCP connection to a server that reads writes data to the FileSystem.
Work has been done on this kind of issue, see e.g. https://github.com/l7mp/stunner
one big pain point is that k8s handle inbound (initiated by entity outside of cluster) and outbound connection (initiated by pod) differently and separately, it is difficult to using same public IP address for inbound dst and outbound source, such symmetric is needed for certain applications. I know there is work around for this case, e.g. only uses a single public IP for both inbound and outbound for whole cluster on aks, or use dedicate node, but these workarounds are just workarounds, all have issues.
If you need to run stateful applications take a look at Service Fabric. It has it quirks but we are running it in production and we are pretty happy with it so far.
thanks, this is 1st time I heard about service fabric, looks interesting, although it seems it doesn't get much adoption beside Microsoft.
k8s is making a lot of the same “write once run anywhere” promises that were made by Jav-hit in the neck with a blow dart-Agggghhhh!…
For some reason, I'm hearing this in my head as you saying this right before your back is cracked by a chiropractor. So something along the lines of "Jav-ahhhhhhhhhh."
billions of devices lol
Is there a platform for small companies to ease deployment and maintenance?
Something like order vps, apt install that, go to ip:4321, login, enter a master ip, and now it’s part of a farm to which you can drag and drop containers (or repositories, or even just init scripts with tarballs) and edit configs/secrets accordingly, and see logs.
I understand docker and what k8s does, but in our situation setting them up and managing is time-consuming overkill that brings nothing except complexity and need for yet another expensive role.
Something like order vps, apt install that, go to ip:4321, login, enter a master ip, and now it’s part of a farm to which you can drag and drop containers (or repositories, or even just init scripts with tarballs) and edit configs/secrets accordingly, and see logs.
I understand docker and what k8s does, but in our situation setting them up and managing is time-consuming overkill that brings nothing except complexity and need for yet another expensive role.
Nomad deploys as a single small binary https://www.nomadproject.io/docs/nomad-vs-kubernetes/alterna...
You might be best served by some platform-as-a-service (Heroku, platform.sh, Netlify, ...) than by a cluster offering.
PaaS are pretty much turnkey for many apps, and cost mostly scales with the actual usage which tends to fit well small companies.
They completely hide infra management from you, deployment can be entirely automated through Git with no SSH or Docker registry management, and they provide access to what you listed: config/secrets, logs.
PaaS are pretty much turnkey for many apps, and cost mostly scales with the actual usage which tends to fit well small companies.
They completely hide infra management from you, deployment can be entirely automated through Git with no SSH or Docker registry management, and they provide access to what you listed: config/secrets, logs.
The interface/dsl should have to be really easy though to make it worth it. E.g., if you are making your own buildpack on heroku you have gone too far.
The whole idea should be ubiquity in knowledge of these platforms. The idea that you can easily hire someone or expect someone to have these skills. Kubernetes might not be simple but it is becoming ubiquitous and that is worth something in the long run.
So if you are looking for the easy solution, use a SaaS / PaaS that really is turnkey and, for example is not just another peaky abstraction of kubernetes. You are choosing an easy solution so you should not need to invest a lot of your energy. Likewise recognize that when you need more customization that you might need to jump ship instead of investing in ducktaping your SaaS/PaaS.
So if you are looking for the easy solution, use a SaaS / PaaS that really is turnkey and, for example is not just another peaky abstraction of kubernetes. You are choosing an easy solution so you should not need to invest a lot of your energy. Likewise recognize that when you need more customization that you might need to jump ship instead of investing in ducktaping your SaaS/PaaS.
I think we are already using paas (DO Apps) for a few services, but some requirements are incompatible with it and other paas’es. I’d like to rather have a thin layer over vps, what I do with ssh, but better UX.
I heard serverless/lambda/Faas is to replace microservice/k8s/docker, but the former is 100% tight-coupled to the cloud vendors to me, k8s seems more portable.
Serverless is just letting the cloud provider do you task using their own managed account under the hood. It creates a very strong vendor lock-in. K8s can be migrated to any cloud or even on-prem with some amount of "ease"
author here! Thanks so much for cross posting to HN <3
What part makes this for developers and what are developers that don't know how to develop?
I've read through twice and I cannot see the relationship between the title and the contents.
It's the part where it explains the "what" and "why" but skips the "how" under the assumption someone who knows how to develop just needs a hook to relate it to what they already know, and can then learn on their own.
Probably a joke for all those software engineers that don’t do anything.
[deleted]
Kubernetes was a clever ploy to trick cloud providers into abstracting themselves into irrelevance, but otherwise a dead end. Can we please hurry up and replace it with an open-source Nomad clone?
IMO Kubernetes is not going anywhere. People are jumping to it in huge numbers. It's very hot. It has sane architecture in my opinion. I'd say it has awesome architecture and I have great respects to the ones responsible for it. They've got very quality foundations. Kubernetes is new Linux. You expected to know Linux and Git, now you'll be expected to know Kubernetes if you're developer.
It's not going anywhere in the sense that it's this decade's XML fad. In 20 years people will be whining about their legacy system running Kubernetes, and how it's a nightmare.
Hadoop is not going anywhere. People are jumping to it in huge numbers. It's very hot. It has sane architecture in my opinion. I'd say it has awesome architecture and I have great respects to the ones responsible for it. They've got very quality foundations. Hadoop is new ETL. You expected to know SQL and ETL, now you'll be expected to know Hadoop if you're a developer.
~HN, circa 2010
~HN, circa 2010
> replace it with an open-source Nomad clone?
Nomad is open source; why clone it?
Nomad is open source; why clone it?
Is the whole thing open? I assumed it was open-core and the good bits were hasicorp secret sauce.
It's a very cool system. I completely understand why people half-jokingly call it a "distributed operating system." It does a lot of things related to the lifecycles of various state (secrets, storage, config, deployments, etc).
However, I believe it goes way too far into putting infrastructure into non-cloud-managed state machines. Things that exist in most modern clouds are being reinvented in K8s. What's more, is that K8s objects are being created as interfaces to the underlying cloud objects. So you now have 2 layers of abstractions, each with their own quirks. It's too much.
Not to mention that IaC for K8s is extremely immature. This will improve, yes, for some definition of "improve." But if you've ever written Helm charts that integrate with Terraform, you'll know about all the spinning plates you have to keep balanced.
It's not a system I see sustaining into the long term future. Google may continue to use and support it forever, but afaik, they are the most invested in its success. Other cloud platforms, like AWS, seem to be focusing on not re-inventing all of their cloud offerings in K8s.