Nomad can do everything that K8s can(mrkaran.dev)
mrkaran.dev
Nomad can do everything that K8s can
https://mrkaran.dev/posts/nomad-k8s-showdown/
19 comments
I think that was confusingly written, but what it means is 'k8s recommends not using it for database, but in Nomad is fine' - the EC2 instances are just nodes like any other, but 'tagged for db'.
And if self-hosting Nomad on AWS is (allegedly) so much easier than operating managed k8s (EKS), isn't that more damning rather than less?
The all-in AWS answer I suppose would be ECS. I quite like it but I think for anything beyond small, more than a few services part of the same system, I can understand wanting to run Nomad (or use EKS) instead.
And if self-hosting Nomad on AWS is (allegedly) so much easier than operating managed k8s (EKS), isn't that more damning rather than less?
The all-in AWS answer I suppose would be ECS. I quite like it but I think for anything beyond small, more than a few services part of the same system, I can understand wanting to run Nomad (or use EKS) instead.
You can run databases that way in K8s as well. Nobody does, because it's a terrible abstraction for high availability of conventional databases. The operator pattern does the opposite of help, creating more moving parts.
The problem with running databases yourself is that it's hard to run databases, and having dedicated physical hardware works, until it doesn't. Nomad make it easy to run dedicated physical hardware for it. As soon as you have a failed EBS volume with your DB on it. Failed EBS volumes still happen frequently, though we've mostly obviated the problem by using managed dbs.
The problem with running databases yourself is that it's hard to run databases, and having dedicated physical hardware works, until it doesn't. Nomad make it easy to run dedicated physical hardware for it. As soon as you have a failed EBS volume with your DB on it. Failed EBS volumes still happen frequently, though we've mostly obviated the problem by using managed dbs.
I'm running plenty of PostgreSQL in k8s.
Know a few people who run hundreds of them too.
And it's perfect for dev shit were colleges pull in their PostgreSQL themselves until later through helm dependencies
Know a few people who run hundreds of them too.
And it's perfect for dev shit were colleges pull in their PostgreSQL themselves until later through helm dependencies
Oh, yeah - If you don't care about data availability/uptime, databases in k8s can be magical. It's just not suitable for high availability out of the box - Postgres, not k8s.
I'm not defending the position, I've never used Nomad (though been wanting to, at home) but have used k8s, including in-cluster database. I was just explaining what the article was saying, because I think 'Nomad, plus EC2 for the db' is a misreading.
I’ve always hated this style of argument in tech where someone says X can do what Y does as an argument against using Y (or for using X) because it always missed *why* people use Y in the first place.
See for example you can write safe code in C therefore rust is pointless (yes but rust makes it much easier to write safe code)
Or you can do machine learning in JavaScript so python is useless (you can but it’s a much smaller ecosystem)
Etc etc.
You can pick whatever tools you want for whatever reason but please give me a better reason than it *can* be done.
See for example you can write safe code in C therefore rust is pointless (yes but rust makes it much easier to write safe code)
Or you can do machine learning in JavaScript so python is useless (you can but it’s a much smaller ecosystem)
Etc etc.
You can pick whatever tools you want for whatever reason but please give me a better reason than it *can* be done.
To be fair, it seems a great number of folks use K8s because they can and/or because Google.
And if that criteria is good enough to justify its use then surely it's enough to justify the use of an alternative.
Frankly, the majority of software houses running K8s really don't need to.
And if that criteria is good enough to justify its use then surely it's enough to justify the use of an alternative.
Frankly, the majority of software houses running K8s really don't need to.
Companies run k8s because it abstracts away infra.
People like k8s because it's fully open source and community driven.
People like k8s because it's fully open source and community driven.
I use K8s because if I don’t like what my deployment costs or how my cluster is supported, I can easily pick it up and take it to another provider or even host it myself (though I don’t really want to self-host).
I can do the same with my Ansible and docker-compose scripts.
Moving to another provider feels like the selling point of ORMs (moving to a different db engine), whereas most often it will never happen.
It's fine that you use k8s and it looks like it's the perfect fit for your case, but tech often thinks "my solution" equals "everyones solution".
Moving to another provider feels like the selling point of ORMs (moving to a different db engine), whereas most often it will never happen.
It's fine that you use k8s and it looks like it's the perfect fit for your case, but tech often thinks "my solution" equals "everyones solution".
Very much agreed on us not all needing the same solution. I’d argue Ansible makes more sense for apps written in scripting languages (particularly Python, obviously), while Kubernetes makes more sense for deploying statically-typed/compiled images where devs shouldn’t really need to deal with the whole Python ecosystem in addition to their compilers, linkers, etc. But of course there’s plenty of room for crossover!
To your point about this being like ORMs, though: I would argue building apps to deploy with Kubernetes or Ansible is less like using an ORM, and more like building them to compile for any OS rather than depending on Windows/Linux/macOS libraries. DB engines generally have much more similar costs than server hosting companies do (you could pay 2x as much to host an app on one vs the other depending on what you need), so it makes less sense to switch. (Also, back in the days when ORMs were created, Oracle was much more dominant than today so DB cost differentials were a bigger issue, but that’s not really today’s market. These days many engines just offer wire compatibility with Postgres/MySQL libraries anyway, so that aspect of ORMs is really not the selling point anymore, it’s the mapping of DB type systems to language type systems.)
To your point about this being like ORMs, though: I would argue building apps to deploy with Kubernetes or Ansible is less like using an ORM, and more like building them to compile for any OS rather than depending on Windows/Linux/macOS libraries. DB engines generally have much more similar costs than server hosting companies do (you could pay 2x as much to host an app on one vs the other depending on what you need), so it makes less sense to switch. (Also, back in the days when ORMs were created, Oracle was much more dominant than today so DB cost differentials were a bigger issue, but that’s not really today’s market. These days many engines just offer wire compatibility with Postgres/MySQL libraries anyway, so that aspect of ORMs is really not the selling point anymore, it’s the mapping of DB type systems to language type systems.)
> See for example you can write safe code in C therefore rust is pointless (yes but rust makes it much easier to write safe code)
Isn't it usually the other way around? "This would be safer if written in rust".
> Or you can do machine learning in JavaScript so python is useless
Said no one ever.
The author also elaborates why (according to him) X is preferable over Y (mainly simplicity), and even if he didn't there's value in comparing tools if one is on the fence of switching but afraid of loosing compatibility. Entire websites exist with that sole purpose.
Isn't it usually the other way around? "This would be safer if written in rust".
> Or you can do machine learning in JavaScript so python is useless
Said no one ever.
The author also elaborates why (according to him) X is preferable over Y (mainly simplicity), and even if he didn't there's value in comparing tools if one is on the fence of switching but afraid of loosing compatibility. Entire websites exist with that sole purpose.
>> Or you can do machine learning in JavaScript so python is useless
> Said no one ever.
I want to reply to this point because I have some relevant & interesting experience here. I've done a fair amount of consulting for companies who have a scenario that goes like this:
- founder starts business in their free time, in a language they know well, as a monolith.
- founder gets a few customers and decides to scale the business
- founder decides they need to use "AI" to solve some business problem
- founder hires a person who is great at solving that particular problem.
- that person wants to use python (of course they do) but the monolith is not in python so various tensions ensue.
- the founder (or someone else on the engineer team) inevitably says "you can do ML in $LANGUAGE_WE_CHOSE, why use python? We didn't chose python because $REASONS_PYTHON_SUCKS.
You'd be surprised how often the language they chose is javascript and how often the reason not to use python is that they can't share code between the client and server. So unfortunately that example wasn't pulled out of nowhere
> Said no one ever.
I want to reply to this point because I have some relevant & interesting experience here. I've done a fair amount of consulting for companies who have a scenario that goes like this:
- founder starts business in their free time, in a language they know well, as a monolith.
- founder gets a few customers and decides to scale the business
- founder decides they need to use "AI" to solve some business problem
- founder hires a person who is great at solving that particular problem.
- that person wants to use python (of course they do) but the monolith is not in python so various tensions ensue.
- the founder (or someone else on the engineer team) inevitably says "you can do ML in $LANGUAGE_WE_CHOSE, why use python? We didn't chose python because $REASONS_PYTHON_SUCKS.
You'd be surprised how often the language they chose is javascript and how often the reason not to use python is that they can't share code between the client and server. So unfortunately that example wasn't pulled out of nowhere
I wish more places ran Nomad. I haven't used it myself but it seems so much simpler than Kubernetes (and, like the author, I have extensive experience operating Kubernetes clusters) and HCL is a dream compared to drowning in OpenAPI docs and Golang structs to figure out how the hell to define the MySuperCoolCR resource that x.y.x-k8s.example.com/v1beta4 offers.
I’ve been using it in production for many years and I agree that it’s vastly simpler than kubernetes, by many orders of magnitude. I’m currently migrating off of it because I reached a point where I needed more enterprise-y features for not a ton of money such as rbac, federation, multi-tenant isolation, resource quotas, and less-often mentioned, hireability (it’s easier to hire for since more people have had exposure to it). HashiCorps enterprise products are also notoriously expensive.
Kubernetes has a lot of merits and open source community support which is attractive which means out of the box integrations are vast.
That said, Nomad builds in a lot of sensible default behaviors that are way way closer to production ready with significantly less effort. Many things in Nomad, with Consul and Vault “just work” like you’d expect. Some of it seems like magic.
Kubernetes on the other hand does almost nothing for you to make your custom app production ready out of the box. This means there’s a ton of room for error and incorrectly configured applications. CNIs, Ingress, PodAffinity, PodDisruptionBudgets, Services, HPAs, ServiceRoles, Secrets management are not defined for you and must be exactly correct or orchestrated in clunky and obscure ways, or a ton of effort needs to go into filling these gaps. “Just use EKS/GKE” quickly becomes a list of subsequent decision trees and you need to hope yoy made the right decision because if you don’t your choice will be deprecated in 6-9 months.
The two config options for Kubernetes are Helm/Kustomize Yaml. Both are great for quick POCs but at production scale are completely inferior to HCL and generally an absolute nightmare to work with on the day-to-day.
If I’m a team of one in a startup, I’d choose Nomad.
Kubernetes has a lot of merits and open source community support which is attractive which means out of the box integrations are vast.
That said, Nomad builds in a lot of sensible default behaviors that are way way closer to production ready with significantly less effort. Many things in Nomad, with Consul and Vault “just work” like you’d expect. Some of it seems like magic.
Kubernetes on the other hand does almost nothing for you to make your custom app production ready out of the box. This means there’s a ton of room for error and incorrectly configured applications. CNIs, Ingress, PodAffinity, PodDisruptionBudgets, Services, HPAs, ServiceRoles, Secrets management are not defined for you and must be exactly correct or orchestrated in clunky and obscure ways, or a ton of effort needs to go into filling these gaps. “Just use EKS/GKE” quickly becomes a list of subsequent decision trees and you need to hope yoy made the right decision because if you don’t your choice will be deprecated in 6-9 months.
The two config options for Kubernetes are Helm/Kustomize Yaml. Both are great for quick POCs but at production scale are completely inferior to HCL and generally an absolute nightmare to work with on the day-to-day.
If I’m a team of one in a startup, I’d choose Nomad.
To whoever wrote this, I have this unsolicited advice: being obnoxious about your competitor is very off-putting to the reader.
Putting down your competitor is a very fine art if you don't want to come across as a douche. Done correctly it can be hilarious though. But this is just a bull in the china shop.
Putting down your competitor is a very fine art if you don't want to come across as a douche. Done correctly it can be hilarious though. But this is just a bull in the china shop.
> being obnoxious about your competitor
As far as I can tell, the author of the article doesn't work for Hashicorp (Nomad) nor Google (Kubernetes) so not sure why you're giving this unsolicited advice?
As far as I can tell, the author of the article doesn't work for Hashicorp (Nomad) nor Google (Kubernetes) so not sure why you're giving this unsolicited advice?
That's OK.
This is just a very weird comparison. Seems like the author doesn’t understand why people run K8s.
Of course they also “managed” several k8s clusters using… wait for it… AWS.
I’d say, when you are all in AWS, then fine don’t use k8s. But you also don’t need Nomad then in most cases.
And if you do need nomad on AWS then that’s fine as well, but it’s not comparable to k8s in general.