Docker, Kubernetes, Terraform, and AWS crash course series(blog.gruntwork.io)
blog.gruntwork.io
Docker, Kubernetes, Terraform, and AWS crash course series
https://blog.gruntwork.io/the-docker-kubernetes-terraform-and-aws-crash-course-series-dca343ba1274?gi=96ef3b8ffd04
83 comments
To be fair. Just last night i was googling kubernetes and k3s tutorials.
All of them on page 1 of SERPS are super simple and brilliant and ONLY show you how to add nodes to clusters.
Ok now what ?? Ive drank the k-Koolaid signup for the newsletters but how do i get value from this 'thing' ???
When i code a webapp how do setup/config the db ? How do I do a regular LAMP app ?
Lol i never thought i say this but i need a kubernetes tutorial on 'best practices' for common patterns. No for google scale but how do i kubernetify my run of the mil lamp app with 100kpage views a month ??
There seems to be a gap between the beginner and advance (kubernetes all things) levels ?
Maybe that is the answer ?? The fact that the middle-situations(lamp apps with low-med traffic) tutorials dont exists means i dont need it ??
All of them on page 1 of SERPS are super simple and brilliant and ONLY show you how to add nodes to clusters.
Ok now what ?? Ive drank the k-Koolaid signup for the newsletters but how do i get value from this 'thing' ???
When i code a webapp how do setup/config the db ? How do I do a regular LAMP app ?
Lol i never thought i say this but i need a kubernetes tutorial on 'best practices' for common patterns. No for google scale but how do i kubernetify my run of the mil lamp app with 100kpage views a month ??
There seems to be a gap between the beginner and advance (kubernetes all things) levels ?
Maybe that is the answer ?? The fact that the middle-situations(lamp apps with low-med traffic) tutorials dont exists means i dont need it ??
All you need exists on kubernetes.io.
Start with https://kubernetes.io/docs/tutorials/hello-minikube/ and proceed.
Read reference documentation on the same site whenever you need to dig somewhere.
It's awesome. You don't need any other websites. I was able to build a kubernetes cluster and right now deploying multi-service application and I have had enough technical information on this website alone.
As to answer your questions:
You can start with DB as an ordinary container in a statefulset deployment. It's similar to docker. You configure it with environment variables. Advanced approach is operators, but you don't need those for simple start.
Connection to DB is the same like docker. You use Kubernetes secrets instead of docker secrets and that's about it.
My message assumes that you're proficient with docker. If you're not, I suggest to first learning docker. May be you don't need Kubernetes for your scale at all. And if you do, most docker concepts make sense in Kubernetes anyway.
Start with https://kubernetes.io/docs/tutorials/hello-minikube/ and proceed.
Read reference documentation on the same site whenever you need to dig somewhere.
It's awesome. You don't need any other websites. I was able to build a kubernetes cluster and right now deploying multi-service application and I have had enough technical information on this website alone.
As to answer your questions:
You can start with DB as an ordinary container in a statefulset deployment. It's similar to docker. You configure it with environment variables. Advanced approach is operators, but you don't need those for simple start.
Connection to DB is the same like docker. You use Kubernetes secrets instead of docker secrets and that's about it.
My message assumes that you're proficient with docker. If you're not, I suggest to first learning docker. May be you don't need Kubernetes for your scale at all. And if you do, most docker concepts make sense in Kubernetes anyway.
Cool thanks will have a look.
Id be more interesting on where kubernetes (redundancy) fits into traditional redundancy ?
Examples:
1. Do i still need gluster/ceph or do i use the longhorn thing ?
2. Db replication? Do i use the usual solutions of master-slaves and clusters or does multiple k-nodes take that over ?
3. Webserver LB with failover ? Do i use LB from hosting vendor, haproxy or does kubernetes have its own thing ??
From what I can tell as a kubernetes-noob the value is: 1) Reproducibility 2) Reliability via redundancy 3) AutoScaling.
All of the abkve has to some degree a previous/current solution , so which do i give up/replace with kubernetes-tool ?
Sorry yes Im a k-noob
Id be more interesting on where kubernetes (redundancy) fits into traditional redundancy ?
Examples:
1. Do i still need gluster/ceph or do i use the longhorn thing ?
2. Db replication? Do i use the usual solutions of master-slaves and clusters or does multiple k-nodes take that over ?
3. Webserver LB with failover ? Do i use LB from hosting vendor, haproxy or does kubernetes have its own thing ??
From what I can tell as a kubernetes-noob the value is: 1) Reproducibility 2) Reliability via redundancy 3) AutoScaling.
All of the abkve has to some degree a previous/current solution , so which do i give up/replace with kubernetes-tool ?
Sorry yes Im a k-noob
1 It depends. For larger scale gluster or ceph. But quiet a bit more work.
Longhorn you can get up and working quickly.
Though if you are on a cloud provider just use their storage system.
2 K8 doesn't magically solve replication unfortunately.
Though there are helm charts that will automatically set up a replicated setup for you.
I still need to solve backups.
Once again if you are on a cloud provide. Just use their Db offering.
3 K8 doesn't have a default out of the box.
The repo shows you how to setup traefik to handle this.
On cloud providers they have normally integrated it with their lb already.
For me the large advantages are reproducibility and no vendor lock in.
Also give redundancy and quiet a bit of automation once set up.
Auto scaling is always tricky.
Lastly if you have the skills it can be far cheaper to run your own in metal.
If you don't the the time most likely would be better spent actually coding.
Depending where you are in the world and the relevant pay scales.
Longhorn you can get up and working quickly.
Though if you are on a cloud provider just use their storage system.
2 K8 doesn't magically solve replication unfortunately.
Though there are helm charts that will automatically set up a replicated setup for you.
I still need to solve backups.
Once again if you are on a cloud provide. Just use their Db offering.
3 K8 doesn't have a default out of the box.
The repo shows you how to setup traefik to handle this.
On cloud providers they have normally integrated it with their lb already.
For me the large advantages are reproducibility and no vendor lock in.
Also give redundancy and quiet a bit of automation once set up.
Auto scaling is always tricky.
Lastly if you have the skills it can be far cheaper to run your own in metal.
If you don't the the time most likely would be better spent actually coding.
Depending where you are in the world and the relevant pay scales.
I'll try to answer, but keep in mind that I'm newbie myself.
> 1. Do i still need gluster/ceph or do i use the longhorn thing ?
Kubernetes does not care about storage implementation. It contains some abstract ways to request a storage (PersistentVolumeClaim). And then some particular Kubernetes installation will fulfil this request with PersistentVolume. So basically it comes down to your Kubernetes provider. It should have some instructions about volume classes that you can use.
If you're installing your Kubernetes on bare metal, you need to think about this aspect yourself, of course. Both ceph and glusterfs are popular options and there're good Kubernetes drivers for it. You can also just use local storage, like docker does, but of course it'll not survive server outage, so it limits your availability.
I'm installing my cluster on OpenStack. There's Cinder CSI plugin for Kubernetes, so it provides me storage when I ask for it. My provider uses SAN for one type of volume and Ceph for other type of volume.
I think that simplest solution is some kind of NFS server. Kubernetes can consume it as well.
> 2. Db replication? Do i use the usual solutions of master-slaves and clusters or does multiple k-nodes take that over ?
Basically Kubernetes does not care about your particular configuration. It runs containers and provides those containers with storage, DNS, network, etc. So it's up to you. You can configure database replication with your own tools and scripts if you like. I have had good experience with CloudNativePG. It's so-called Kubernetes Operator. Basically it's a thing that configures database clusters for (postgres) given abstract definition. It can configure master-slaves cluster and it allows for easy backup configuration to S3 storage. There're other operators as well, for almost any popular database. So probably it's better to use those, unless you're very good at database operations.
And in the true clouds it might be a good idea to use managed database and not to think about it at all.
> 3. Webserver LB with failover ? Do i use LB from hosting vendor, haproxy or does kubernetes have its own thing ??
You need some kind of external load balancer to deliver packets in a high-available way. My hoster provides that as part of its OpenStack package. I guess that every cloud hoster provides it. If you're using bare metal, you need some kind of haproxy and keepalived setup (or some kind of hardware load balancer, I have no idea).
This external load balancer have to deliver TCP packets to your worker nodes. Like 1.2.3.4:80 -> 10.1.1.1:30080, 10.11.1.2:30080, 10.11.1.3:30080. And once Kubernetes receives those packets in a high-available way, it routes them as needed. Usually you have ingress controller which provides HTTPS, and then uses HTTP host and path to route request to some pod which serves it in the end. Once request reached Kubernetes, it'll make sure to route it the right way. If your pods deployed in 2+ replicas, it'll be high-available. If your pod deployed in 1 replica and server is died, Kubernetes eventually will reschedule that pod to another server, but there'll be service interruption for a few minutes. So everything high-available should be deployed in 2+ replicas.
> From what I can tell as a kubernetes-noob the value is: 1) Reproducibility 2) Reliability via redundancy 3) AutoScaling.
Here's my take on Kubernetes value.
First is it introduces a language connecting developers and operations. It is important. You don't need developers to hand-waive which ports they need to expose, which services they need to consume and which HTTP routes they need to receive for that particular server. They've got language to express how their service should be used.
Second is it provides high-available cluster. And in my testing it's quite stable. My only issue is that it takes too much time to reschedule pods from dead server. I expected it to take few seconds, but it took few minutes. I don't yet know why is that.
Third and one I didn't really expect when started to learn it: it provides high-quality solutions for some hard problems. I mentioned database clustering and database backup. I can deploy database with single 50-lines YAML mostly copy&pasted from the example. It'll start master and slave pods and it'll provide continuous backup to S3 using barman. I don't have skills to configure that kind of setup and I expect that I'd need a week at least to come to that setup. Another problem is letsencrypt. Well, it's not that hard, but I've spent many hours debugging some convoluted nginx/caddy/whatever net of docker containers trying to figure out what letsencrypt does not work there. With Kubernetes cert-manager it's just works. All configuration is centralized, all services write ingress and they've got their TLS certificate automagically, whether it's HTTP-01 solver, DNS-01 solver, it's just abstracted away.
It has some steep learning curve, that's for sure. Even more so, if you want to deploy it yourself rather than using managed one. I suggest to use managed one if you can. I have some circumstances which prevent me to using managed Kubernetes, but I plan to migrate to managed as soon as I can. It's not that hard, but it takes time and managed Kubernetes is cheap enough. If you can't use managed Kubernetes, try to find a provider with OpenStack API. It'll help with load balancers and storage provisioning.
Autoscaling - that part I didn't solve yet. It's not easy if you're not using managed Kubernetes. But if you're using managed Kubernetes, it should be as easy as ticking a checkbox somewhere.
> All of the above has to some degree a previous/current solution , so which do i give up/replace with kubernetes-tool ?
Well, right now we're using three dedicated servers with docker-composes scattered all over, made with ad-hoc scripts and whatnot, partially working backups, no observability. Chaos. Kubernetes for us looks like a very promising way to throw away that chaos and rebuild operations correctly.
> 1. Do i still need gluster/ceph or do i use the longhorn thing ?
Kubernetes does not care about storage implementation. It contains some abstract ways to request a storage (PersistentVolumeClaim). And then some particular Kubernetes installation will fulfil this request with PersistentVolume. So basically it comes down to your Kubernetes provider. It should have some instructions about volume classes that you can use.
If you're installing your Kubernetes on bare metal, you need to think about this aspect yourself, of course. Both ceph and glusterfs are popular options and there're good Kubernetes drivers for it. You can also just use local storage, like docker does, but of course it'll not survive server outage, so it limits your availability.
I'm installing my cluster on OpenStack. There's Cinder CSI plugin for Kubernetes, so it provides me storage when I ask for it. My provider uses SAN for one type of volume and Ceph for other type of volume.
I think that simplest solution is some kind of NFS server. Kubernetes can consume it as well.
> 2. Db replication? Do i use the usual solutions of master-slaves and clusters or does multiple k-nodes take that over ?
Basically Kubernetes does not care about your particular configuration. It runs containers and provides those containers with storage, DNS, network, etc. So it's up to you. You can configure database replication with your own tools and scripts if you like. I have had good experience with CloudNativePG. It's so-called Kubernetes Operator. Basically it's a thing that configures database clusters for (postgres) given abstract definition. It can configure master-slaves cluster and it allows for easy backup configuration to S3 storage. There're other operators as well, for almost any popular database. So probably it's better to use those, unless you're very good at database operations.
And in the true clouds it might be a good idea to use managed database and not to think about it at all.
> 3. Webserver LB with failover ? Do i use LB from hosting vendor, haproxy or does kubernetes have its own thing ??
You need some kind of external load balancer to deliver packets in a high-available way. My hoster provides that as part of its OpenStack package. I guess that every cloud hoster provides it. If you're using bare metal, you need some kind of haproxy and keepalived setup (or some kind of hardware load balancer, I have no idea).
This external load balancer have to deliver TCP packets to your worker nodes. Like 1.2.3.4:80 -> 10.1.1.1:30080, 10.11.1.2:30080, 10.11.1.3:30080. And once Kubernetes receives those packets in a high-available way, it routes them as needed. Usually you have ingress controller which provides HTTPS, and then uses HTTP host and path to route request to some pod which serves it in the end. Once request reached Kubernetes, it'll make sure to route it the right way. If your pods deployed in 2+ replicas, it'll be high-available. If your pod deployed in 1 replica and server is died, Kubernetes eventually will reschedule that pod to another server, but there'll be service interruption for a few minutes. So everything high-available should be deployed in 2+ replicas.
> From what I can tell as a kubernetes-noob the value is: 1) Reproducibility 2) Reliability via redundancy 3) AutoScaling.
Here's my take on Kubernetes value.
First is it introduces a language connecting developers and operations. It is important. You don't need developers to hand-waive which ports they need to expose, which services they need to consume and which HTTP routes they need to receive for that particular server. They've got language to express how their service should be used.
Second is it provides high-available cluster. And in my testing it's quite stable. My only issue is that it takes too much time to reschedule pods from dead server. I expected it to take few seconds, but it took few minutes. I don't yet know why is that.
Third and one I didn't really expect when started to learn it: it provides high-quality solutions for some hard problems. I mentioned database clustering and database backup. I can deploy database with single 50-lines YAML mostly copy&pasted from the example. It'll start master and slave pods and it'll provide continuous backup to S3 using barman. I don't have skills to configure that kind of setup and I expect that I'd need a week at least to come to that setup. Another problem is letsencrypt. Well, it's not that hard, but I've spent many hours debugging some convoluted nginx/caddy/whatever net of docker containers trying to figure out what letsencrypt does not work there. With Kubernetes cert-manager it's just works. All configuration is centralized, all services write ingress and they've got their TLS certificate automagically, whether it's HTTP-01 solver, DNS-01 solver, it's just abstracted away.
It has some steep learning curve, that's for sure. Even more so, if you want to deploy it yourself rather than using managed one. I suggest to use managed one if you can. I have some circumstances which prevent me to using managed Kubernetes, but I plan to migrate to managed as soon as I can. It's not that hard, but it takes time and managed Kubernetes is cheap enough. If you can't use managed Kubernetes, try to find a provider with OpenStack API. It'll help with load balancers and storage provisioning.
Autoscaling - that part I didn't solve yet. It's not easy if you're not using managed Kubernetes. But if you're using managed Kubernetes, it should be as easy as ticking a checkbox somewhere.
> All of the above has to some degree a previous/current solution , so which do i give up/replace with kubernetes-tool ?
Well, right now we're using three dedicated servers with docker-composes scattered all over, made with ad-hoc scripts and whatnot, partially working backups, no observability. Chaos. Kubernetes for us looks like a very promising way to throw away that chaos and rebuild operations correctly.
> I think that simplest solution is some kind of NFS server. Kubernetes can consume it as well.
Isn't this asking for trouble, running MySQL on top of NFS?
Also a k-nube.
Isn't this asking for trouble, running MySQL on top of NFS?
Also a k-nube.
I don't have experience with that kind of setup, so can't really comment. Kubernetes is not magic, if you're running MySQL in NFS volume, it's just mysql process running in separate cgroup with mounted NFS volume. If you think that's a bad idea, don't do that. When it comes to processes and mounts, Kubernetes just arranges things and rest is done my ordinary Linux APIs. I tried to run Postgres with ceph volume and it works for me. You can configure database to just use local storage. Of course if your server will die, your database will die, so probably you will want some kind of cluster setup or be ready to restore database from backup if that happens. Or you can start with simple setup and migrate to something more complex in the future.
I hadn't heard about k8s "local storage". IIUC this fixes each pod to a single node, so the potential benefits of k8s can't be fully realized. It seems that clustering could still be achieved by running muliple instances across multiple nodes, each with local storage. I might still prefer this to having to manage NFS and any difficulties that might come from using MySQL with remote disks.
It just seems like k8s is coercing people to do things that they normally wouldn't do: MySQL on NFS, Postgres on Ceph, etc. In this case, OP just wants to "kubernetify my run of the mil lamp app" and is now being pointed towards NFS, Ceph, etc. I can't help but wonder if the juice is worth the sqeeze. Especially for services like databases that for most people probably don't really require dynamic provisioning or orchestration.
It just seems like k8s is coercing people to do things that they normally wouldn't do: MySQL on NFS, Postgres on Ceph, etc. In this case, OP just wants to "kubernetify my run of the mil lamp app" and is now being pointed towards NFS, Ceph, etc. I can't help but wonder if the juice is worth the sqeeze. Especially for services like databases that for most people probably don't really require dynamic provisioning or orchestration.
I found a similar problem.
Most examples failed when then trying to use them together.
Also, jumping into directly K8 can be quite a jump.
I put this together to help SA Php group.
Starts off with just deploying directly on a server.
Then takes you to a full application deployed on K8 with auto SSL and DNS generation.
It needs a bit of a refresh :(
But finally coming out of being a bit over-committed, so should be updating in the next week or two.
Still some bits missing but it should cover all your basics
https://github.com/haakco/deploying-laravel-app
Most examples failed when then trying to use them together.
Also, jumping into directly K8 can be quite a jump.
I put this together to help SA Php group.
Starts off with just deploying directly on a server.
Then takes you to a full application deployed on K8 with auto SSL and DNS generation.
It needs a bit of a refresh :(
But finally coming out of being a bit over-committed, so should be updating in the next week or two.
Still some bits missing but it should cover all your basics
https://github.com/haakco/deploying-laravel-app
> Maybe that is the answer ?? The fact that the middle-situations(lamp apps with low-med traffic) tutorials dont exists means i dont need it ??
SRE for 15 years with FAANG (MAGMA?) scale experience; I would argue this, yes. That's a little over 2 requests per minute. If you're cloud hosting then you could get away with single tiny hosts for your front end and DB, like t3.small on AWS, and Cloudwatch alarms for monitoring. If you need extreme HA or burstability even with this low rate, a managed load balancer (e.g. AWS ALB) with a few target webservers will do the trick and let you swap them out as needed without taking down the site. A DB read replica will give you redundancy there as well. This is all 20-year-old tried and true tech. You'll set it up once and it will run for years without trouble.
Once you introduce Kubernetes you've got a whole 'nother beast to feed, especially if it's self-managed, and you'll pay dearly if it's not (like EKS). For your scale, it would be kind of like buying a full sized semi truck to haul stuff for a corner store. You'd be better served with a small pickup or minivan.
Of course you don't need an "at scale" excuse to learn new tech. In which case I agree there's a dearth of practical tutorials that aren't just "here's a Helm chart." Part of this is because managing a stateful service like MySQL on K8s is not straightforward; there's a lot of ways to do it, most of them are wrong, new ways get introduced every few years, and even people who claim they've solved the issue are probably sitting on a time bomb of their own making and have just gotten lucky.
SRE for 15 years with FAANG (MAGMA?) scale experience; I would argue this, yes. That's a little over 2 requests per minute. If you're cloud hosting then you could get away with single tiny hosts for your front end and DB, like t3.small on AWS, and Cloudwatch alarms for monitoring. If you need extreme HA or burstability even with this low rate, a managed load balancer (e.g. AWS ALB) with a few target webservers will do the trick and let you swap them out as needed without taking down the site. A DB read replica will give you redundancy there as well. This is all 20-year-old tried and true tech. You'll set it up once and it will run for years without trouble.
Once you introduce Kubernetes you've got a whole 'nother beast to feed, especially if it's self-managed, and you'll pay dearly if it's not (like EKS). For your scale, it would be kind of like buying a full sized semi truck to haul stuff for a corner store. You'd be better served with a small pickup or minivan.
Of course you don't need an "at scale" excuse to learn new tech. In which case I agree there's a dearth of practical tutorials that aren't just "here's a Helm chart." Part of this is because managing a stateful service like MySQL on K8s is not straightforward; there's a lot of ways to do it, most of them are wrong, new ways get introduced every few years, and even people who claim they've solved the issue are probably sitting on a time bomb of their own making and have just gotten lucky.
Our platform team got into k8s for some reason at my company. Because it’s enterprise there’s never really the “millions of requests per second” sort of problems because every deployment is single tenant. We have experienced k8s engineers who I guess we’re just using the tools we know, but wow it’s been such a hassle and has caused so much toil. Right now we’re having trouble scaling back ends for parallel jobs with k8s, which is a problem we easily solved with regular tooling in the past pre-k8s.
On enterprises, k8s solves a different kind of scaling problem. You tend to have hundreds of apps that are business critical and need to be managed.
K8s provides a consistent way to package, deploy with HA and monitor those apps. Ideally it replaced tons of ad-hoc scripts and confluence pages that are very poorly maintained.
K8s provides a consistent way to package, deploy with HA and monitor those apps. Ideally it replaced tons of ad-hoc scripts and confluence pages that are very poorly maintained.
I would recommend Bret Fisher tutorials https://www.bretfisher.com/
(disclaimer: I work for a Bunnyshell) Honestly the most benefit would be gained from making Kubernetes invisible and speeding up your development process. Tools exist now to deploy short lived preview environments into your cluster for every PR. This is where the Kubernetes values sits. The whole “shift left” idea. Test before merge / identical short lived environments, etc.
> SERPS
Search Engine Result Pages
Search Engine Result Pages
Author here. I tried to answer your question in the first two paragraphs. But to add some context, given the nature of my work, I hear from developers on a nearly daily basis who are struggling to get started with the technologies mentioned in this blog post series, which include not only Kubernetes, but also Docker, AWS, and Terraform. In part, they are struggling because they are too scared to ask for help, and comments like yours only make that worse: you seem to be implying that the materials out there for Kubernetes are so good, that if you don't get it, there must be something wrong with you. And yet, there are thousands of devs who don't get it, so maybe for different people, there are different ways to learn?
In discussions like this, I'm a fan of what Steve Yegge wrote about blogging [1]:
> This is an important thing to keep in mind when you're blogging. Each person in your audience is on a different clock, and all of them are ahead of you in some ways and behind you in others. The point of blogging is that we all agree to share where we're at, and not poke fun at people who seem to be behind us, because they may know other things that we won't truly understand for years, if ever.
That's why I write: to share what I know, from my particular perspective. Hopefully, that's useful to some people out there. If it's not useful to you, no problem!
And for the record, I agree the Kubernetes docs are great, including those interactive tutorials: if you read the series, you'd see I actually recommend those exact docs at the end of the post [2].
[1] https://sites.google.com/site/steveyegge2/you-should-write-b... [2] https://blog.gruntwork.io/a-crash-course-on-kubernetes-a96c3...
In discussions like this, I'm a fan of what Steve Yegge wrote about blogging [1]:
> This is an important thing to keep in mind when you're blogging. Each person in your audience is on a different clock, and all of them are ahead of you in some ways and behind you in others. The point of blogging is that we all agree to share where we're at, and not poke fun at people who seem to be behind us, because they may know other things that we won't truly understand for years, if ever.
That's why I write: to share what I know, from my particular perspective. Hopefully, that's useful to some people out there. If it's not useful to you, no problem!
And for the record, I agree the Kubernetes docs are great, including those interactive tutorials: if you read the series, you'd see I actually recommend those exact docs at the end of the post [2].
[1] https://sites.google.com/site/steveyegge2/you-should-write-b... [2] https://blog.gruntwork.io/a-crash-course-on-kubernetes-a96c3...
I just don't want to do that, no thanks. I've spent so much time learning about how an OS can solve these problems and these lessons have paid dividends many times over my career. Instead, I will continue to invest in the core foundation. I'm not interested in re-learning these abstractions reading [1]-[5] tutorials plus many more and studying an unrefined, new layer of complicated abstractions built on top of the OS--the OS is difficult enough. I'll only _use_ k8s when it's managed and supported by a team of 10+ engineers, which is what it requires. Plus, it's not just k8s; you have concourse, spinnaker, artifactory, some sort of cluster templating, kops? to template your k8s deployments across DCs and environments, (I don't know what the community uses, we built our own in-house tool). It's all so gross.
My rejection of these systems has lead me to invest in FreeBSD. There was a learning curve and I'm certainly not as fluent in this system as I am with Linux but I'm in a place where I'm in control of the OS and have at my disposal solid, refined tools to help me masterfully construct the infrastructure to solve my problems. And, every time I solve a problem in this space my foundational understanding grows and these lessons will pay dividends into the future, or so I hope.
I'm utilizing FreeBSD because docker / k8s and I'll add systemd to the list, are missing and thus the community solves problems in a different way that better aligns with my values. FreeBSD isn't easier, I've been stumped on problems countless numbers of times but when I find a solution, it scales. What I mean by that is I have a better understanding of a system that can solve a larger class of problems than if I was in the docker / k8s ecosystem.
YMMV.
My rejection of these systems has lead me to invest in FreeBSD. There was a learning curve and I'm certainly not as fluent in this system as I am with Linux but I'm in a place where I'm in control of the OS and have at my disposal solid, refined tools to help me masterfully construct the infrastructure to solve my problems. And, every time I solve a problem in this space my foundational understanding grows and these lessons will pay dividends into the future, or so I hope.
I'm utilizing FreeBSD because docker / k8s and I'll add systemd to the list, are missing and thus the community solves problems in a different way that better aligns with my values. FreeBSD isn't easier, I've been stumped on problems countless numbers of times but when I find a solution, it scales. What I mean by that is I have a better understanding of a system that can solve a larger class of problems than if I was in the docker / k8s ecosystem.
YMMV.
That sounds like a bit of an extreme reaction to what seems to be a "this technology is yucky and I don't like it" problem.
No, you're way off. Maybe my reaction is extreme, but my problem isn't. What's yucky is that you're trying to convince people that reading five tutorials on k8s will make them productive. No, the problem, and I'll repeat it since you conveniently skipped over it, k8s requires 10+ engineers to maintain and manage, full time.
_I_ don't have 10 engineers and thus it's the wrong tool for me and I've decided to not invest in it.
_I_ don't have 10 engineers and thus it's the wrong tool for me and I've decided to not invest in it.
K8s is certainly a beast and I agree with your posts directionally but just as a counterpoint - as a solopreneur I've been using k8s to run my business workloads (on GKE) for the past 4 or 5 years now. I'm very comfortable with dev and ops and k8s is a force multiplier for me, letting me easily manage much more than I could without it (e.g. due to automation, tooling, community, etc). Building on top of a standardized platform with a huge community has been a big win for me as a solo dev.
I'm not running a very big operation, I only have two nodes which host a few custom webapps and a few dozen WP sites. Running in a single region removes the extra charge for HA GKE, letting me run pretty lean and just pay for the VM, storage and bandwidth. I hardly ever have to spend any time on managing the cluster, it keeps chugging along while I get things done and makes it easy for me to manage app lifecycles. YMMV.
I keep it simple, I tried helm but didn't like it because it added too much complexity. I pull in cert-manager and nginx-ingress to every cluster I run but nothing else. I build my images locally and push to the registry directly, no CI/CD. I focus on the core competencies of k8s and try to stay lean and conservative with adding new tools or components.
I'm not running a very big operation, I only have two nodes which host a few custom webapps and a few dozen WP sites. Running in a single region removes the extra charge for HA GKE, letting me run pretty lean and just pay for the VM, storage and bandwidth. I hardly ever have to spend any time on managing the cluster, it keeps chugging along while I get things done and makes it easy for me to manage app lifecycles. YMMV.
I keep it simple, I tried helm but didn't like it because it added too much complexity. I pull in cert-manager and nginx-ingress to every cluster I run but nothing else. I build my images locally and push to the registry directly, no CI/CD. I focus on the core competencies of k8s and try to stay lean and conservative with adding new tools or components.
This is a very interesting use-case for k8s. Can you expand on it some more? For instance, why did you you pick this over vhosting? You essentially took TESLA's Electric semi-truck, remove the cab, extend the chasis and slap a school bus body on top of it. I think I like it, but I'm wondering how stable it is. how much of set-it-and-forget-it benefit do you get out of it. Sometimes it's best to over-engineer to sleep good at night, knowing the limits will never be tested. This is why XCP-NG beats PROXMOX, because there's less wet nursing involved. Let me know if you have any write-up on this approach that you can share. I'd be interested in reading it.
I'll consider writing a blog post on it but long story short, it's very stable and hands-off when using a managed service. I started off with some DO droplets when I first launched my business and adopted k8s around v1.6. It was a little rough back then but now I spend maybe a few hours a month max on managing my clusters, mostly just upgrading the nodes and core components (cert-manager, nginx-ingress-controller).
I fell in love with it because of the dev and ops experience. Just the shift of perspective from pets to cattle is a big improvement. Deploying via sftp is simple but k8s really helps me with the other aspects (monitoring, logging, scheduled tasks, TLS cert mgmt, blue/green deploys, multiple environments, etc) of managing apps that my clients use regularly and depend on. It's nice having one API platform that can handle all of this and feels more like assembling Lego blocks instead of bespoke solutions for every project.
Hope that gives you some insight!
I fell in love with it because of the dev and ops experience. Just the shift of perspective from pets to cattle is a big improvement. Deploying via sftp is simple but k8s really helps me with the other aspects (monitoring, logging, scheduled tasks, TLS cert mgmt, blue/green deploys, multiple environments, etc) of managing apps that my clients use regularly and depend on. It's nice having one API platform that can handle all of this and feels more like assembling Lego blocks instead of bespoke solutions for every project.
Hope that gives you some insight!
They're not wrong tbf, you can largely replace K8s with curl and some basic cloud CLI scripts for the majority of cases. Slight hyperbole but I'll never really understand the weird urge to use Borg-lite everywhere.
>> I wonder what drives someone to write a 101-level tutorial on K8s.
When I was going thru k8s tutorials several years ago, one issue was how quickly the tutorials became outdated with changes in k8s. Then you'd spend only half your effort on the the tutorial, and the other half debugging why the tutorial wasnt working.
When I was going thru k8s tutorials several years ago, one issue was how quickly the tutorials became outdated with changes in k8s. Then you'd spend only half your effort on the the tutorial, and the other half debugging why the tutorial wasnt working.
Sadly last I checked there were relatively few tutorials for setting up on bare metal (or rather, they neglect important topics like setting up net
worked volumes or other things that are very likely to be required by an application.
This is no doubt a useful commment, since you provide other sources. However, it's not particularly a mark of courtesy to denigrate others' contribution to the space. If you had substantive criticism of the OP offered tutorial, it might be helpful. However, you offer none.
I can see your point. Thanks for your comment, I'll try to restate my point next time.
I can’t think of a worse way to manage Kubernetes than Terraform. After years of running into issues, I think it’s really only suitable for small clusters.
ClusterAPI has been the way my company has been doing things lately and I can honestly tell you it’s been transformative.
https://github.com/kubernetes-sigs/cluster-api
There’s nothing more amazing that updating a kubelet version in a git repo and watching ClusterAPI carefully cordoning off nodes and moving our workloads to new machines while I sit back and watch via ArgoCD.
ClusterAPI has been the way my company has been doing things lately and I can honestly tell you it’s been transformative.
https://github.com/kubernetes-sigs/cluster-api
There’s nothing more amazing that updating a kubelet version in a git repo and watching ClusterAPI carefully cordoning off nodes and moving our workloads to new machines while I sit back and watch via ArgoCD.
How does ClusterAPI compare to Helm charts (or just managing Kubernetes YAML manually or through Kustomize?)
Why do you need clusterctl / clusterawsadm? Can't you just kubectl apply once you have your .kube/config set up for your AWS k8s cluster (or Azure or GCP or whatever)?
Why do you need clusterctl / clusterawsadm? Can't you just kubectl apply once you have your .kube/config set up for your AWS k8s cluster (or Azure or GCP or whatever)?
Cluster API is just CRDs to manage cluster infrastructure itself. Clusterctl and co is to manage controllers that reconcile said CRDs
> CRDs
CustomResourceDefinition
for anybody else following along
https://kubernetes.io/docs/concepts/extend-kubernetes/api-ex...
CustomResourceDefinition
for anybody else following along
https://kubernetes.io/docs/concepts/extend-kubernetes/api-ex...
Cluster API allows you to create clusters with helm charts.
We use ClusterAPI + a centralized ArgoCD to push our clusters that’s other teams can use but with all the required add-ons that the business says we need like security tooling and policy management.
We use ClusterAPI + a centralized ArgoCD to push our clusters that’s other teams can use but with all the required add-ons that the business says we need like security tooling and policy management.
Can you elaborate a little please? Helm charts don't let you create a cluster? Wouldn't it be easier for Helm to add that functionality as opposed to introducing "yet another" tool/layer in the k8s stack?
How do you deal with cloud anxiety? I've been always using bare metal servers, because I am afraid that when I set something up wrong on AWS I'll become bankrupt.
When I had a few services running on AWS I often would wake up at night having a nightmare that my keys got stolen and someone run thousands on my account.
After a few weeks like that I deleted everything from AWS. I have an account on GCP as I had to learn few things for my work assignment and I forgot to delete the setup - anyway a few months later I got an email that Google can't bill my account. I almost got a heart attack and then I noticed they couldn't bill $0 and my card simply expired - imagine the relief.
When I wrote this I can see that maybe sounds stupid, but I have genuine fear of running anything on AWS, GCP or Azure etc.
Another fear I have is that if I put a website online, someone could run a script to fetch content forever and make me pay huge bandwidth costs. When I last checked, there is no way to put any sort of caps e.g. when bandwidth cost exceeds certain amount it would deactivate the buckets etc. So what would you do? Nowadays very much every job lists knowledge of AWS as a requirement, but I can't get myself past the fear.
After a few weeks like that I deleted everything from AWS. I have an account on GCP as I had to learn few things for my work assignment and I forgot to delete the setup - anyway a few months later I got an email that Google can't bill my account. I almost got a heart attack and then I noticed they couldn't bill $0 and my card simply expired - imagine the relief.
When I wrote this I can see that maybe sounds stupid, but I have genuine fear of running anything on AWS, GCP or Azure etc.
Another fear I have is that if I put a website online, someone could run a script to fetch content forever and make me pay huge bandwidth costs. When I last checked, there is no way to put any sort of caps e.g. when bandwidth cost exceeds certain amount it would deactivate the buckets etc. So what would you do? Nowadays very much every job lists knowledge of AWS as a requirement, but I can't get myself past the fear.
For training purposes, Azure has cost-limited subscriptions.
Frustratingly, the other clouds have dragged their feet and largely refused to implement their own equivalents. This comes up here semi-regularly as one of the #1 complaints about the public cloud, and it's always hand-waved away by some Silicon Valley FAANG employee making half a million a year and using their parent company cloud account.
I know of several ways it would be possible for a malicious actor to deliberately bankrupt a solo dev limited to a normal credit card budget.
And as you've said, it's very fiddly and complicated to implement custom "stop" scripts, and this is futile anyway because the billing metrics in all clouds have huge delays on them.
My $0.02 is try to learn using Azure first because there's a true safety net, and then use customer accounts to continue upskilling.
Frustratingly, the other clouds have dragged their feet and largely refused to implement their own equivalents. This comes up here semi-regularly as one of the #1 complaints about the public cloud, and it's always hand-waved away by some Silicon Valley FAANG employee making half a million a year and using their parent company cloud account.
I know of several ways it would be possible for a malicious actor to deliberately bankrupt a solo dev limited to a normal credit card budget.
And as you've said, it's very fiddly and complicated to implement custom "stop" scripts, and this is futile anyway because the billing metrics in all clouds have huge delays on them.
My $0.02 is try to learn using Azure first because there's a true safety net, and then use customer accounts to continue upskilling.
Can completely relate to your situation. Basically, that’s the main reason I run everything on bare metal server, running in my basement. And use cloud providers only if my clients ready to pay for it or take risks.
There is nothing stupid about this fear at all. Having to fight a $20k mistake is highly stressful. Not being able to prevent it via policy is a design decision by the provider.
I think that for the average developer, these tools will soon be obsolete. For frontend applications, hosting services such as Vercel and Netlify gives a far superior DX. For backend, Vercel does the same, or other BaaS' like Supabase that gives you most you would need to create a fully fledged app with easy to use API's. The only downside I can tell, is the pricing, but compared to dev hours saved, it's probably a net benefit.
If you can use a Platform as a Service (PaaS) offering like Vercel, Netlify, Heroku, etc, you absolutely should! I always recommend those types of tools as the first stop tools to anyone building software these days.
But there are many use cases that don't fit into those neat PaaS molds: typically, as a software company grows beyond one team, one service, one database, etc, they start to hit limitations with the PaaS solutions out there. As you scale, you often find you need more control than you can get from a SaaS: you may need more control over the hardware (e.g., for performance or cost reasons), or networking (e.g., you need service discovery or a service mesh to allow microservices to communicate with each other), or security (e.g., to meet compliance standards), or a hundred other items.
That's when many companies find themselves migrating to an Infrastructure as a Service (IaaS) provider like AWS, Infrastructure as Code (IaC) tools like Terraform, orchestration tools like Kubernetes, and so on. I'm guessing every software company with more than 50-100 developers ends up moving from PaaS to IaaS, and that's when developers need to understand how to use the tools covered in this blog post series.
Perhaps, some day, the PaaS tools out there will be good enough that you never have to migrate off of them, regardless of scale or requirements, but we're not there yet, and probably won't be there for a while longer.
But there are many use cases that don't fit into those neat PaaS molds: typically, as a software company grows beyond one team, one service, one database, etc, they start to hit limitations with the PaaS solutions out there. As you scale, you often find you need more control than you can get from a SaaS: you may need more control over the hardware (e.g., for performance or cost reasons), or networking (e.g., you need service discovery or a service mesh to allow microservices to communicate with each other), or security (e.g., to meet compliance standards), or a hundred other items.
That's when many companies find themselves migrating to an Infrastructure as a Service (IaaS) provider like AWS, Infrastructure as Code (IaC) tools like Terraform, orchestration tools like Kubernetes, and so on. I'm guessing every software company with more than 50-100 developers ends up moving from PaaS to IaaS, and that's when developers need to understand how to use the tools covered in this blog post series.
Perhaps, some day, the PaaS tools out there will be good enough that you never have to migrate off of them, regardless of scale or requirements, but we're not there yet, and probably won't be there for a while longer.
For the average developer, it's okay not to interact with these tools. All you need to interact is with the `bin/deploy-production` script the above-average developer built.
And I'd argue that with managed kubernetes solution, such as GKE Autopilot, you get roughly the same "DX", and much more flexibility to use whatever tools you fancy. GKE Autopilot manages the cluster and node availability, ingress and certificates, all you need to do is write the kubernetes yaml. As long as what you are working on fits in a docker container, you're good to go. And you're fairly good to move off of their setup, because those same yaml configs will work on AWS or self-hosted solutions, albeit with additions that fill in gaps where Google managed services lived.
And I'd argue that with managed kubernetes solution, such as GKE Autopilot, you get roughly the same "DX", and much more flexibility to use whatever tools you fancy. GKE Autopilot manages the cluster and node availability, ingress and certificates, all you need to do is write the kubernetes yaml. As long as what you are working on fits in a docker container, you're good to go. And you're fairly good to move off of their setup, because those same yaml configs will work on AWS or self-hosted solutions, albeit with additions that fill in gaps where Google managed services lived.
What do you think people use to build systems like these?
A lot of medium/large shops build their own deployment and BaaS platforms on top of Kubernetes/Terraform/AWS. You just don’t hear about them as much because they often aren’t provided as a service externally, and aren’t open sourced. (I work with and on such platforms)
The mistake I see people often make is assuming it’s worth using such complex tools directly with a small project. People like the flexibility but don’t understand cost to build/maintain it.
A lot of medium/large shops build their own deployment and BaaS platforms on top of Kubernetes/Terraform/AWS. You just don’t hear about them as much because they often aren’t provided as a service externally, and aren’t open sourced. (I work with and on such platforms)
The mistake I see people often make is assuming it’s worth using such complex tools directly with a small project. People like the flexibility but don’t understand cost to build/maintain it.
For backend, if you have a platform that allows you to run arbitrary containers, you will need to have a way to express how they are connected together, how much resource they should get, which parts should receive traffic from the Internet and so on. Any solution will tend to have about the same inherent complexity as kube yamls, this seems to be an iron law of devops.
IMHO the thing you have to trade off to get simplicity is generality. I agree that's probably the right tradeoff for 90% of apps, whether that's BaaS, heroku-alikes, language specific hosting or whatever.
Cloud providers are definitely keen on the idea that your web framework and your cloud platform could basically become the same thing.
IMHO the thing you have to trade off to get simplicity is generality. I agree that's probably the right tradeoff for 90% of apps, whether that's BaaS, heroku-alikes, language specific hosting or whatever.
Cloud providers are definitely keen on the idea that your web framework and your cloud platform could basically become the same thing.
The difficult with backend is dealing with state. It's easy enough to provide a simple experience when dealing with stateless frontends, backends are a very different story.
You're right that we need better tools. I'm the founder of Encore [1] which is all about bringing the simplicity of Vercel/Netlify to backend development. Not by substituting the backend for a BaaS but by building a developer experience hand-crafted for dealing with cloud infrastructure.
[1] https://encore.dev
You're right that we need better tools. I'm the founder of Encore [1] which is all about bringing the simplicity of Vercel/Netlify to backend development. Not by substituting the backend for a BaaS but by building a developer experience hand-crafted for dealing with cloud infrastructure.
[1] https://encore.dev
I'll definitely review your service but in the meantime you should know my desktop (macOS) Firefox is freaking out on your website and is scrolling it in a janky manner and it keeps one CPU core at 80% for good 30 seconds after switching away from the tab.
Oh, thanks very much for letting me know. We'll have a look.
can it be used without creating account? also does your open source version has telemetry enabled?
The downside is vendor lock in. What will you do when vendor will ban your website?
not sure if IaC will help, I'm just started reading about it.
Indeed the cloud vendors are all lock-ins, something needs to be designed to mitigate the switching cost among cloud providers.
Each one has its own SDK and own complicated infrastructures, so far it seems impossible to switch once you started with any one of them.
Indeed the cloud vendors are all lock-ins, something needs to be designed to mitigate the switching cost among cloud providers.
Each one has its own SDK and own complicated infrastructures, so far it seems impossible to switch once you started with any one of them.
Cloud vendors are not all lock-ins. Kubernetes is an open API. If you've got your project working in Amazon, you can migrate it to GKE, Digital Ocean or just bare metal.
Yes, cloud vendors love to lock you in, but right now it's possible to use clouds in a vendor-neutral way. Managed postgres is just a postgres. Managed Kubernetes is just a Kubernetes. S3 have plenty of alternative implementations.
Yes, cloud vendors love to lock you in, but right now it's possible to use clouds in a vendor-neutral way. Managed postgres is just a postgres. Managed Kubernetes is just a Kubernetes. S3 have plenty of alternative implementations.
what are the strongest competitors to S3? S3 is so dominant I rarely looked into anything else for cloud storage.
S3 is just a protocol. Minio is one example of software which implements this protocol. You're programming against S3, using amazon SDK, but you change endpoint to another provider and it just works.
I guess it wouldn't work for all the intricacies, but for commonly used stuff it works.
I guess it wouldn't work for all the intricacies, but for commonly used stuff it works.
> so far it seems impossible to switch once you started with any one of them
Do you think cloud vendors (FAANG companies) are fools who have accidentally created this situation? And do you think they will stand idly by while you try to abstract away the differences between them?
Do you think cloud vendors (FAANG companies) are fools who have accidentally created this situation? And do you think they will stand idly by while you try to abstract away the differences between them?
> And do you think they will stand idly by while you try to abstract away the differences between them?
Yes - letting people try something futile and fail is much better than telling them that it is futile.
Yes - letting people try something futile and fail is much better than telling them that it is futile.
Can we split the middle, they are fools, but the situation is not accidentally created?
Even better: we replace those millions of SLOC with something new and 100 times simpler.
> The only downside I can tell, is the pricing, but compared to dev hours saved, it's probably a net benefit.
dev hours is already paid - i say that a hosted service for an app is gonna cost even more than the dev in the long run.
dev hours is already paid - i say that a hosted service for an app is gonna cost even more than the dev in the long run.
You pay a premium to let your devs work on what matters, the actual product, not tinker with devops tools for hours on end on something that has already been solved. Something to keep in mind is that infrastructure code has to be updated and maintained as well, with services that gets handled for you. It would be interesting to see how much of a difference in price it would make over eg. a year.
I can give you the answer: it depends.
I'm hosting at a colocation for X bucks a month. Solely the RAM I have on my servers costs X bucks a month at AWS/Azure. Maintenance overhead is small, maybe a couple hours a month.
I'm hosting at a colocation for X bucks a month. Solely the RAM I have on my servers costs X bucks a month at AWS/Azure. Maintenance overhead is small, maybe a couple hours a month.
[deleted]
Depends on what u consider avg developer, aren’t most developers employed by some company or enterprise? Who would definitely lock into big3?
Now you have to hire not just for a specific language but a specific backend platform. Idk man.
Oh boy...
Guys and gals, learn the basics.
You don't need Docker, just a version control system (e.g. git).
You don't need overpriced cloud offerings in the first place.
First of all, pick the right language. For me, that's Go, because it has lots of performance for little resource usage. Docker is written in Go. Why would I put a Go proxy in front of my Go program to have nothing but what's essentially a zip file with some startup script and then have to fight the Go proxy, Docker, and its configuration. Learn about systemd. It's the defacto standard init system for most popular Linix distros nowadays. Terraform? Ansible if you must. If you're managing more than 1 host maybe you do, but it's likely you don't. Or pick something like NixOS if you just have to define your machine state. AWS (and GCP, etc)? A, one, dedicated server where you can host all the projects your heart desires for a fixed price with low costs for bandwidth/traffic.
Kubernetes, you do not need it. Go can handle about 500 million visitors a month on 12 year old hardware per 1 server. You will almost certainly not reach these numbers.
So you don't need much to live a stress free and cheap but efficient life. What this article tries to teach you is how to become a tool for public cloud providers.
Running k8s requires at least 9 servers. Managed k8s, like those cloud providers offer are expensive, because even if you think it isn't so, the devil is in the details. I see that guy posting of how easy it is. BS. Nothing is easier than a single host with a front proxy like nginx, which btw k8s also uses, they call it ingres proxy, a simple nginx config, some letsencrypt, your Go backend and the database on the same server. It doesn't get easier.
What this article tries to teach you is stuff you will only ever need if you expect over 5k concurrency (note that's not 5000 users). When do those situations occur? Eurovision Song Contest. Very popular chat app (but even there you'd not use http).
You don't need all this.
And you also don't need microservices. Monoliths are fine. When are microservices useful? When you need to scale particular components of your big program.
Is it worth the trouble? Hell no.
Especially because now you have to deal with a problem so huge with very little benefit to going the event driven microservice route, because that's the only route that matters regarding microservices. And that's the only, ONLY time k8s makes sense. That's also where you can afford to pay those ridiculous prices for cloud traffic and managed k8s. Those are multilateral enterprise levels of business size, ok?
Forget all that crap.
In these comments you'll find people trying to BS you to get hyped for their product, promising "there will be only one". It's all BS. Just learn the basics. It's not hard.
Disclaimer: truly failsafe setups should run on 3 servers (hn runs on 2 and has regular downtime). So if you have a money maker of course do that. But until then even a single server setup with regular backups will do just fine.
Guys and gals, learn the basics.
You don't need Docker, just a version control system (e.g. git).
You don't need overpriced cloud offerings in the first place.
First of all, pick the right language. For me, that's Go, because it has lots of performance for little resource usage. Docker is written in Go. Why would I put a Go proxy in front of my Go program to have nothing but what's essentially a zip file with some startup script and then have to fight the Go proxy, Docker, and its configuration. Learn about systemd. It's the defacto standard init system for most popular Linix distros nowadays. Terraform? Ansible if you must. If you're managing more than 1 host maybe you do, but it's likely you don't. Or pick something like NixOS if you just have to define your machine state. AWS (and GCP, etc)? A, one, dedicated server where you can host all the projects your heart desires for a fixed price with low costs for bandwidth/traffic.
Kubernetes, you do not need it. Go can handle about 500 million visitors a month on 12 year old hardware per 1 server. You will almost certainly not reach these numbers.
So you don't need much to live a stress free and cheap but efficient life. What this article tries to teach you is how to become a tool for public cloud providers.
Running k8s requires at least 9 servers. Managed k8s, like those cloud providers offer are expensive, because even if you think it isn't so, the devil is in the details. I see that guy posting of how easy it is. BS. Nothing is easier than a single host with a front proxy like nginx, which btw k8s also uses, they call it ingres proxy, a simple nginx config, some letsencrypt, your Go backend and the database on the same server. It doesn't get easier.
What this article tries to teach you is stuff you will only ever need if you expect over 5k concurrency (note that's not 5000 users). When do those situations occur? Eurovision Song Contest. Very popular chat app (but even there you'd not use http).
You don't need all this.
And you also don't need microservices. Monoliths are fine. When are microservices useful? When you need to scale particular components of your big program.
Is it worth the trouble? Hell no.
Especially because now you have to deal with a problem so huge with very little benefit to going the event driven microservice route, because that's the only route that matters regarding microservices. And that's the only, ONLY time k8s makes sense. That's also where you can afford to pay those ridiculous prices for cloud traffic and managed k8s. Those are multilateral enterprise levels of business size, ok?
Forget all that crap.
In these comments you'll find people trying to BS you to get hyped for their product, promising "there will be only one". It's all BS. Just learn the basics. It's not hard.
Disclaimer: truly failsafe setups should run on 3 servers (hn runs on 2 and has regular downtime). So if you have a money maker of course do that. But until then even a single server setup with regular backups will do just fine.
> Go can handle about 500 million visitors a month on 12 year old hardware per 1 server.
Doing what? I'm absolutely no fan of Kubernetes, or overcomplexity, but this is just meaningless platitude.
The feasibility of 12,000 requests per second (assuming evenly distributed throughout a 30 day month) largely depends on what the system is _doing_ for each of those requests.
Doing what? I'm absolutely no fan of Kubernetes, or overcomplexity, but this is just meaningless platitude.
The feasibility of 12,000 requests per second (assuming evenly distributed throughout a 30 day month) largely depends on what the system is _doing_ for each of those requests.
God damn, I love this comment. We manage 8 Linux servers total for an application that is lightly used by our customers but heavily used by our internal workers that constantly reach out to internet services and check for things. Everything we've done has been in Go and bash, and our deployments are easy and take seconds.
I'm tired of seeing so many posts for Docker, Kubernetes, terraform, and even so many of Amazon's service offerings. Guess what? All these tools are abstractions, and it's insane to me that so many would rather learn to work with these abstractions, as opposed to learning some Linux basics, which would allow them to solve the same problem with much less overhead. We have two webservers, four workers, and two database servers. All of our code is written in Go so our dependencies are limited, and this has been my reason for not needing Docker. However, we use Docker locally to spin up our local dev environment, so developers don't need to have MySQL installed on their machines. The primary benefit I see with Docker is dependency management, and when your dependencies are minimal, it's not worth the overhead.
All of our servers are built from bash scripts that are executed on the server on the first run. This bash script creates folder structures and user accounts and sets permissions. If a server needs to be destroyed for whatever reason, it can be rebuilt as new from these scripts in < 10 minutes.
Code deployments are done using SCP and rsync and systemctl to stop and start services remotely. If we were using an interpreted language where dependencies are a bit trickier to manage in prod, certainly we'd use Docker then. Still, for the time being, this Go \ bash ecosystem works for us, is easy to manage, and limits dependencies overall. Debian & Go all the way, baby.
The longer I work in this industry, the more I realize a bigger hindrance to accomplishing goals is actually overthinking and over-engineering. I see startups with no customers using kubernetes and terraform. People building noahs ark when they've never experienced rain. The pain points in your process will become obvious. Let the process tell you it's time for these more advanced tools. If you can manage production code and deploy updates easily without them, then don't waste your time. If you don't know any other way, then maybe it's worth spending the time to learn Linux a little bit better.
I'm tired of seeing so many posts for Docker, Kubernetes, terraform, and even so many of Amazon's service offerings. Guess what? All these tools are abstractions, and it's insane to me that so many would rather learn to work with these abstractions, as opposed to learning some Linux basics, which would allow them to solve the same problem with much less overhead. We have two webservers, four workers, and two database servers. All of our code is written in Go so our dependencies are limited, and this has been my reason for not needing Docker. However, we use Docker locally to spin up our local dev environment, so developers don't need to have MySQL installed on their machines. The primary benefit I see with Docker is dependency management, and when your dependencies are minimal, it's not worth the overhead.
All of our servers are built from bash scripts that are executed on the server on the first run. This bash script creates folder structures and user accounts and sets permissions. If a server needs to be destroyed for whatever reason, it can be rebuilt as new from these scripts in < 10 minutes.
Code deployments are done using SCP and rsync and systemctl to stop and start services remotely. If we were using an interpreted language where dependencies are a bit trickier to manage in prod, certainly we'd use Docker then. Still, for the time being, this Go \ bash ecosystem works for us, is easy to manage, and limits dependencies overall. Debian & Go all the way, baby.
The longer I work in this industry, the more I realize a bigger hindrance to accomplishing goals is actually overthinking and over-engineering. I see startups with no customers using kubernetes and terraform. People building noahs ark when they've never experienced rain. The pain points in your process will become obvious. Let the process tell you it's time for these more advanced tools. If you can manage production code and deploy updates easily without them, then don't waste your time. If you don't know any other way, then maybe it's worth spending the time to learn Linux a little bit better.
I agree with you entirely, nearly all of the companies I've encountered that are trying to get kubernetes working have:
1. Made poor language choices that require them to over complicate their architecture to scale things up. Places where a single golang binary would work and instead you have queues and callbacks and 5 container images because at each road block they just added some new "thing" to make it work.
2. do not even have the base infrastructure automated for any of this to run on, so instead they throw kubernetes on top of it and cross their fingers that the underlying hosts will just magically stay up. And every email about degraded hardware from AWS nearly gives you a heart attack.
I guess overall I'm sad that instead of trying to make smaller and tighter machine images that boot quickly in order to automate availability, we (the industry) added another layer (containers) to just complicate things more.
1. Made poor language choices that require them to over complicate their architecture to scale things up. Places where a single golang binary would work and instead you have queues and callbacks and 5 container images because at each road block they just added some new "thing" to make it work.
2. do not even have the base infrastructure automated for any of this to run on, so instead they throw kubernetes on top of it and cross their fingers that the underlying hosts will just magically stay up. And every email about degraded hardware from AWS nearly gives you a heart attack.
I guess overall I'm sad that instead of trying to make smaller and tighter machine images that boot quickly in order to automate availability, we (the industry) added another layer (containers) to just complicate things more.
Oh boy, another HN engineer who cannot imagine a technical situation other than the ones they're already familiar with telling everyone else what they don't need.
Yes, learn the basics. That way you'll know that Docker and `git` solve for almost entirely different domains, that Terraform and Ansible solve for almost entirely different domains, that "AWS" is a huge suite of services that makes no sense to compare with a single server, and that some of us do indeed work in distributed environments serving 10s or 100s (or more!) of thousands of queries per second.
Yes, learn the basics. That way you'll know that Docker and `git` solve for almost entirely different domains, that Terraform and Ansible solve for almost entirely different domains, that "AWS" is a huge suite of services that makes no sense to compare with a single server, and that some of us do indeed work in distributed environments serving 10s or 100s (or more!) of thousands of queries per second.
Counterpoint: there is a niche in the industry where going with containers would be easier than following the best practices without, given how likely people are to slip into doing things the lazy way, or due to how you might have limited control over the applications that you might have to run. Containers at least limit the fallout from various mistakes and nudges people towards certain architectural choices - e.g. 12 Factor Apps, so you don't cripple your app into being a "singleton app" which can only have 1 instance running, instead of just a "monolith".
It doesn't matter whether you're running Podman, Docker, Docker Compose, Docker Swarm, Hashicorp Nomad, K3s/K0s/minikube or a full K8s distro like RKE, the OCI standard is a hill that I'll die on because unless you're the person who's shipping and managing their code 100% of the time, someone else will have to figure out how to setup and run your Java/.NET/Node/Python/PHP app and moreso, what to do when one needs JDK 8, another needs JDK 11 and there are inevitable conflicts in some other system/runtime dependencies. With containers, you can limit the the impact of relatively badly written and sometimes even somewhat insecure applications, by not exposing the rest of the underlying host to them, especially when running rootless or with host remapping.
Sure, in practice it would be exceedingly nice to just deploy a single packaged executable to the server. It would be really nice to even have the choice of picking a technology that plays nicely with such simple setups, like Go, as opposed to having Java shoved down your throat, especially in combination with a separately managed Tomcat instance or something. It would also be nice to have well written applications that don't misbehave and don't trash the CPU/RAM and don't cause the server to have to OOM kill processes, and also that have proper port configuration/remapping available and also applications that have proper health checks built in, or even just well written systemd services and Ansible to manage the configuration.
But in the real world you don't always get that. So unless you like moving around different companies a lot, you need ways to manage the fallout of sub-optimal circumstances (especially legacy code that still needs to run) and containers are one way to achieve that. I wrote down some of my experiences with the subject (albeit with details changed) in my blog post "My journey from ad hoc chaos to order (a tale of legacy code, services and containers)": https://blog.kronis.dev/articles/my-journey-from-ad-hoc-chao...
That said, developing applications poorly will also mean running them poorly on Kubernetes, or not being able to set the environments up at all. Thus, I believe that it's a slippery slope with options on all ends of it being bad, be it running things directly on the server and running into the aforementioned issues, or trying to go full steam ahead with Kubernetes and failing.
The sweet spot in my experience is using containers, but in a simple setup - currently that is Docker Compose or Docker Swarm, but might become Nomad in the future. Maybe K3s in 5-10 years (single node cluster worked fine, except for RPM distros where cgroups v2 support had problems and the Traefik proxy had bad documentation for wildcard SSL/TLS certificates). Then you no longer need to even care that much about the specifics of the technologies inside of the containers either.
Edit: As an example, here's all that I (or anyone else) would need to deploy my homepage, as long as the server can run OCI images or Compose/Swarm stacks:
It doesn't matter whether you're running Podman, Docker, Docker Compose, Docker Swarm, Hashicorp Nomad, K3s/K0s/minikube or a full K8s distro like RKE, the OCI standard is a hill that I'll die on because unless you're the person who's shipping and managing their code 100% of the time, someone else will have to figure out how to setup and run your Java/.NET/Node/Python/PHP app and moreso, what to do when one needs JDK 8, another needs JDK 11 and there are inevitable conflicts in some other system/runtime dependencies. With containers, you can limit the the impact of relatively badly written and sometimes even somewhat insecure applications, by not exposing the rest of the underlying host to them, especially when running rootless or with host remapping.
Sure, in practice it would be exceedingly nice to just deploy a single packaged executable to the server. It would be really nice to even have the choice of picking a technology that plays nicely with such simple setups, like Go, as opposed to having Java shoved down your throat, especially in combination with a separately managed Tomcat instance or something. It would also be nice to have well written applications that don't misbehave and don't trash the CPU/RAM and don't cause the server to have to OOM kill processes, and also that have proper port configuration/remapping available and also applications that have proper health checks built in, or even just well written systemd services and Ansible to manage the configuration.
But in the real world you don't always get that. So unless you like moving around different companies a lot, you need ways to manage the fallout of sub-optimal circumstances (especially legacy code that still needs to run) and containers are one way to achieve that. I wrote down some of my experiences with the subject (albeit with details changed) in my blog post "My journey from ad hoc chaos to order (a tale of legacy code, services and containers)": https://blog.kronis.dev/articles/my-journey-from-ad-hoc-chao...
That said, developing applications poorly will also mean running them poorly on Kubernetes, or not being able to set the environments up at all. Thus, I believe that it's a slippery slope with options on all ends of it being bad, be it running things directly on the server and running into the aforementioned issues, or trying to go full steam ahead with Kubernetes and failing.
The sweet spot in my experience is using containers, but in a simple setup - currently that is Docker Compose or Docker Swarm, but might become Nomad in the future. Maybe K3s in 5-10 years (single node cluster worked fine, except for RPM distros where cgroups v2 support had problems and the Traefik proxy had bad documentation for wildcard SSL/TLS certificates). Then you no longer need to even care that much about the specifics of the technologies inside of the containers either.
Edit: As an example, here's all that I (or anyone else) would need to deploy my homepage, as long as the server can run OCI images or Compose/Swarm stacks:
version: '3.7'
services:
homepage:
image: REFERENCE_TO_MY_CONTAINER_IMAGE
restart: "unless-stopped"
environment:
# All of the configuration goes here, for example:
ANALYTICS_URL: LINK_TO_MY_MATOMO_INSTANCE_GOES_HERE
ANALYTICS_ID: SOME_IDENTIFIER_SET_UP_IN_MATOMO
volumes:
- SOME_DIRECTORY/homepage/app/storage:/app/storage
- SOME_DIRECTORY/homepage/app/db/files:/app/db/files
networks:
- prod_ingress_network
deploy:
mode: replicated
replicas: 2
update_config:
parallelism: 1
delay: 10s
monitor: 10s
placement:
constraints:
- node.hostname == SERVER_HOSTNAME_GOES_HERE
resources:
limits:
cpus: "0.5"
memory: "512M"
networks:
prod_ingress_network:
driver: overlay
attachable: true
external: true
"You don't need X because I don't need X and everyone else must have exactly the same requirements and constraints as me" is one of my least favorite takes.
> You don't need Docker, just a version control system (e.g. git).
I've worked where we had to run a lot of systems that we didn't write ourselves and they came in a variety of languages, frameworks, etc. Some Python, some Go, some Ruby, some PHP, some Java, some Fortran, some C++, etc. We did that before Docker and when Docker came out it was a massive simplification for us to be able to just run standard containers where we know there's an entrypoint, some ports, environment variables, etc. rather than have to figure out how to configure all of those things for every kind of environment that we might run into. We could use the same basic build/deploy pipeline for all of them, run them on the same shared servers and know that they weren't conflicting with each other and updates to one wouldn't blow up the .so files that a different one used, etc. Docker doesn't really compete with a version control system like git; if anything, it's closer to an application package format like .deb or .rpm, but with a bit of a different focus.
> You don't need overpriced cloud offerings in the first place.
If you've done the cost analysis and the cloud offerings are cheaper than running your own datacenter, then they're not really "overpriced" and maybe you do need them. If you're just thinking "rent a server from Hetzner or a colo or run a workstation under your desk" and that those options are always cheaper than the cloud then you've clearly never worked somewhere that needed regulatory compliance (HIPAA, PCI, SOC2, etc).
> First of all, pick the right language.
If you're on a greenfield project and have that luxury, great. A lot of developers have to work with existing codebases and products though.
> Kubernetes, you do not need it. Go can handle about 500 million visitors a month on 12 year old hardware per 1 server. You will almost certainly not reach these numbers.
It must be wonderful to live in a world where all requests scale the same. What if each visitor is using your service to transcode 2GB video files? Or running massive climate modeling simulations? Or training an ML algorithm on a few TB of data? 500 million of those per month on one 12 year old server? Awesome. Sign me up.
I've done all of these things with and without tools like Docker and Kubernetes. Yes, there are workloads and environments where they'd be overkill and add more complexity than they're worth. But there's also very good reasons beyond hype that those tools became well known and widely used.
Or, you can just keep assuming that all those developers out there are just idiots compared to you. I doubt you'll change your mind, so I'll just hope I never have to work with you.
> You don't need Docker, just a version control system (e.g. git).
I've worked where we had to run a lot of systems that we didn't write ourselves and they came in a variety of languages, frameworks, etc. Some Python, some Go, some Ruby, some PHP, some Java, some Fortran, some C++, etc. We did that before Docker and when Docker came out it was a massive simplification for us to be able to just run standard containers where we know there's an entrypoint, some ports, environment variables, etc. rather than have to figure out how to configure all of those things for every kind of environment that we might run into. We could use the same basic build/deploy pipeline for all of them, run them on the same shared servers and know that they weren't conflicting with each other and updates to one wouldn't blow up the .so files that a different one used, etc. Docker doesn't really compete with a version control system like git; if anything, it's closer to an application package format like .deb or .rpm, but with a bit of a different focus.
> You don't need overpriced cloud offerings in the first place.
If you've done the cost analysis and the cloud offerings are cheaper than running your own datacenter, then they're not really "overpriced" and maybe you do need them. If you're just thinking "rent a server from Hetzner or a colo or run a workstation under your desk" and that those options are always cheaper than the cloud then you've clearly never worked somewhere that needed regulatory compliance (HIPAA, PCI, SOC2, etc).
> First of all, pick the right language.
If you're on a greenfield project and have that luxury, great. A lot of developers have to work with existing codebases and products though.
> Kubernetes, you do not need it. Go can handle about 500 million visitors a month on 12 year old hardware per 1 server. You will almost certainly not reach these numbers.
It must be wonderful to live in a world where all requests scale the same. What if each visitor is using your service to transcode 2GB video files? Or running massive climate modeling simulations? Or training an ML algorithm on a few TB of data? 500 million of those per month on one 12 year old server? Awesome. Sign me up.
I've done all of these things with and without tools like Docker and Kubernetes. Yes, there are workloads and environments where they'd be overkill and add more complexity than they're worth. But there's also very good reasons beyond hype that those tools became well known and widely used.
Or, you can just keep assuming that all those developers out there are just idiots compared to you. I doubt you'll change your mind, so I'll just hope I never have to work with you.
All these "you don't need X / Y / Z" points critically miss the point that it's irrelevant when that is what the companies hiring people want anyway.
Companies hiring the people probably also have a need if they are hiring and looking to grow and scale out. I made a "you don't need X / Y / Z" point because I strongly believe most people don't need these tools and really need a better understanding of Linux basics. You can also learn these tools on your own if they're mentioned in job descriptions of jobs you're applying for, but that doesn't mean you have to use them when you're the decision maker, or for a personal project.
Using these tools in personal projects is how you learn them though, so you can talk about your experience, and maybe even show some prototype in an interview / CV etc.
I recommend https://kubelabs.collabnix.com for Kubernetes, https://dockerlabs.collabnix.com for Docker tutorials. The community also maintains https://kubetools.collabnix.com that is highly useful.
Or better phrased: "How to get inclined to slash your own wrists".
Jokes aside, I'll give it a go, I do need some advanced Docker topics lately and I am sure Kubernetes can help me with some of my home-hosted services as well.
Jokes aside, I'll give it a go, I do need some advanced Docker topics lately and I am sure Kubernetes can help me with some of my home-hosted services as well.
As a Devops engineer I'm staying away from Kubernetes at home. It would require too much constant maintenance.
I use docker only.
I get to play with kubernetes enough at work.
I use docker only.
I get to play with kubernetes enough at work.
Yeah, I tend to peak at Docker Compose and dare not tread further. I might actually learn systemd management of services and stop there.
I've worked in orgs where k8s worked great but most of the time it just made everyone curse their lives.
I've worked in orgs where k8s worked great but most of the time it just made everyone curse their lives.
Why use k8s and AWS instead of ECS?
Because then your endpoint is the standardized kubernetes rather than ECS.
Also I don't think you can use Kubernetes logic with ECS
Finally I think with ECS you have quite the vendor lock in.
Also I don't think you can use Kubernetes logic with ECS
Finally I think with ECS you have quite the vendor lock in.
Having deployed Dockerized services to ECS and Azure App Service, doing it from scratch twice is still easier than figuring out k8s.
You can host your kubernetes cluster anywhere. Or multiple places (as in you can host both at azure and AWS or your own data center and Google etc).
I'm not a fanboy of kubernetes, I think it makes some really trivial things stupidly complex, but avoiding vender lock in is huge. It can be massively better but oh well.
I'm not a fanboy of kubernetes, I think it makes some really trivial things stupidly complex, but avoiding vender lock in is huge. It can be massively better but oh well.
I like and use ECS, but it lacks the tooling and ecosystem around Kubernetes.
The docs are great [1] and offer even interactive parts [2]. There are a ton of good materials on youtube. There are Coursera and EdX courses, e.g. [3] from Google and [4] from Red Hat. There are a ton of Udemy courses, e.g. [5]. I'm not even going to mention sites like Pluralsight (which now owns CloudGuru and CloudAcademy).
I'm not even mentioning books on advanced Kubernetes and service mesh topics.
I wonder what drives someone to write a 101-level tutorial on K8s. In my mind, there's such an overload of information on the topic that it must get lost among the greats.
[1] https://kubernetes.io/docs/concepts/overview/what-is-kuberne...
[2] https://kubernetes.io/docs/tutorials/kubernetes-basics/creat...
[3] https://www.coursera.org/specializations/architecting-google...
[4] https://www.coursera.org/specializations/cloud-native-develo...
[5] https://www.udemy.com/course/learn-kubernetes/