Docker Swarm Cluster Complete Guide(knowledgepill.it)
knowledgepill.it
Docker Swarm Cluster Complete Guide
https://knowledgepill.it/posts/docker_swarm_compendium/
40 comments
That’s very sad to see Docker Swarm declining. I run few production clusters, all are being set up via Ansible and Wireguard for communication. The setup takes only few minutes and runs perfectly fine. It's been running for a while and I haven't had any issues with clusters, as well as it doesn't require maintenance. While Kubernetes is great, it requires a lot of maintenance and setup isn't straightforward. Sometimes you don't need additional complexity and want things just to run and pretty much forget.
SwarmKit and ClassicSwarm are not "docker swarm", but AFAIK precursors of it until they've been merged into Docker itself.
As a production user of Docker Swarm though, it is effectively dead and I've been planning a migration to k8s ASAP.
As a production user of Docker Swarm though, it is effectively dead and I've been planning a migration to k8s ASAP.
Swarmkit is the code that runs "Docker Swarm", it is vendored in to build it. I don't think anyone really uses Classic Swarm any more, and it was a precursor.
I stand corrected, thanks.
Docker Swarm is a marvel. It's super easy to use yet does the job very simply.
On the surface. Easy to setup a simple use case but long term has a similar learning curve to Kubernetes for a production deployment.
Maybe I am missing something or underestimating the complexity of a k8s deployment, but we have been running Swarm in production for a while now and it was/is still a breeze. So what would you consider the complexities of running Swarm in production - I would then like to look into these - as a means of being ahead of the curve of complexity...
Same for me. Swarm has been super easy to install and to use in production. You can do highly available deployments super easily too. I don't see how one could build the same thing more simply.
Agreed. It's my go to for smaller projects and what I recommend for those who don't want to spend more than a day figuring out how to deploy their pet project as it's a logical progression from docker-compose in dev to a production environment.
https://dockerswarm.rocks is better for a more complete, production-ready setup guide.
https://dockerswarm.rocks is better for a more complete, production-ready setup guide.
I know Mirantis have pledged to continue developing Docker Swarm, but it still strikes me as a pretty risky choice for 2020.
I know some companies still using it because they say K8 is too complex (e.g. learning curve) for their needs. It could be appealing to these cases.
Meh, it’s a pain to install, but that can be automated, and a pain to get your head around, but the only features you really need for basic usage are service and deployment.
On the other hand, most of the books on it are spectacularly bad.
On the other hand, most of the books on it are spectacularly bad.
[deleted]
I am looking into this now. Basically it’s either this or the super hard Kubernetes right? I find it hard to believe that we don’t have a middle ground?
Nomad + Consul.
Used it for years in prod - just awesome.
I’ve also used k8s extensively and nomad/consul is more ”open” and less opinionated.
If you also manage some form of legacy infrastructure and VMs, consul is hard to ignore.
Two go-binaries - you’re up in minutes for a test drive.
Used it for years in prod - just awesome.
I’ve also used k8s extensively and nomad/consul is more ”open” and less opinionated.
If you also manage some form of legacy infrastructure and VMs, consul is hard to ignore.
Two go-binaries - you’re up in minutes for a test drive.
sigh I've almost given up nudging folks on here to give Nomad a spin. It's so easy to deploy and play with, heck you can run a client and server with a single go binary on your local machine. If you compare that to all the hundreds of different (and soon-to-be-deprecated) installation instructions for k8s, k8s feels like a joke at best, or an evil plot at worst to get everyone to use hosted k8s. Honestly, people use hosted k8s because you need a full-time person/team to run it, so in the end, even though k8s is Open Source, Google (and AWS) still win...
Nevermind that Nomad is fast, easy to grasp, very concise and can run non-Docker workloads. Oh well...
Nevermind that Nomad is fast, easy to grasp, very concise and can run non-Docker workloads. Oh well...
The “evil plot” is what I gather.
K8s have it’s bright spots, but it can’t touch the flexibility of nomad & consul.
Now that nomad/consul have autoscaling and csi as well as native envoy integration the gap, with regards to container workloads, have shrunk quite a bit!
Now that nomad/consul have autoscaling and csi as well as native envoy integration the gap, with regards to container workloads, have shrunk quite a bit!
I'm interested to hear more!
The Nomad page gives me a good overview but doesn't tell me WHY should I migrate from an existing k8 cluster. Would you be able to sell it to me?
Off the top of my head:
- Nomad is a more general type of workload scheduler. What I mean with "general" is that you can schedule pretty much anything over a bunch of hosts - not just containers e.g you have a JRE on the node? Just schedule your jar. Execute a raw shell command? Not a problem.
- Batch jobs with parameters? Just send it to the "dispatch" endpoint with parameters and an optional payload. Often can replace "serverless" and sometimes work queues.
- Consul is a "general purpose" service discovery (dns & distributed K/V) that can span over all your services, not just k8s ones (for this very reason Hashicorp have built a k8s -> consul services sync tool: it is often needed!)
You include consul agent with a service definition in roles on your ansible/salt/puppet managed VM's. They can be included in the mesh, and SD works seamlessly between hosts and containers.
- The consul connect (mesh) feature with envoy is much more comprehensible than istio.
I use to say that nomad + consul is more in-line with the "unix philosophy". You have these two binaries that do what they do independently, but they can be hooked together if you would like to use distributed K/V and service discovery for your scheduled workload. Add Vault to that - which often is used without consul & nomad, but integrates as well.
They can all be easily integrated with through well defined rest API's and do not have to be consumed as a big black box.
K8s is all or nothing.
- Nomad is a more general type of workload scheduler. What I mean with "general" is that you can schedule pretty much anything over a bunch of hosts - not just containers e.g you have a JRE on the node? Just schedule your jar. Execute a raw shell command? Not a problem.
- Batch jobs with parameters? Just send it to the "dispatch" endpoint with parameters and an optional payload. Often can replace "serverless" and sometimes work queues.
- Consul is a "general purpose" service discovery (dns & distributed K/V) that can span over all your services, not just k8s ones (for this very reason Hashicorp have built a k8s -> consul services sync tool: it is often needed!)
You include consul agent with a service definition in roles on your ansible/salt/puppet managed VM's. They can be included in the mesh, and SD works seamlessly between hosts and containers.
- The consul connect (mesh) feature with envoy is much more comprehensible than istio.
I use to say that nomad + consul is more in-line with the "unix philosophy". You have these two binaries that do what they do independently, but they can be hooked together if you would like to use distributed K/V and service discovery for your scheduled workload. Add Vault to that - which often is used without consul & nomad, but integrates as well.
They can all be easily integrated with through well defined rest API's and do not have to be consumed as a big black box.
K8s is all or nothing.
Thanks a lot.
I will definitely give it a try. Thankfully, our services are very easily manageable and provide clear Rest APIs, it's just the overhead cluster that's making us slow.
I would love to have something like Nomad ready and available for single-tenant deployments of our service. It's where you need to tweak a lot and do it fast.
my thing with hashicorp products is the feeling that they are geared towards enterprise and not the community at large.
people don't write about them or teach you about them. i have only seen a couple of youtube videos, just mentioning them. i got a book on terraform (the only one i could find that covers it) that wanted me to use AWS to even begin the examples.
it feels like these tools are only used in the enterprise realm.
very sad and discouraging.
people don't write about them or teach you about them. i have only seen a couple of youtube videos, just mentioning them. i got a book on terraform (the only one i could find that covers it) that wanted me to use AWS to even begin the examples.
it feels like these tools are only used in the enterprise realm.
very sad and discouraging.
I looked into Nomad, did not understand much. I am pretty OK with docker and docker-compose, me and my team use docker to develop data workflows in python locally. So out of Docker I am much less comfortable. For instance, would JupyterHub for Kubernetes run on Nomad?
https://zero-to-jupyterhub.readthedocs.io/en/latest/customiz...
https://zero-to-jupyterhub.readthedocs.io/en/latest/customiz...
It’s not k8s api compatible - so no, not straight off.
You can use nomad to schedule pretty much any workload you’d like, including docker containers.
You could probably schedule a raw command that is “docker-compose”.
You can use nomad to schedule pretty much any workload you’d like, including docker containers.
You could probably schedule a raw command that is “docker-compose”.
I don't understand why people think Kubernetes is "hard" - I find it quite straightforward and the documentation is excellent. I've been running my side projects on a single k8s clusters and rarely have issues.
Yes, there is a learning curve. It's not a Heroku where you enter one command and it just works. You'll need to spend a few days to become familiar with the basics concepts. And it may take weeks grasp the advanced concepts. But it's less complexity than learning something like a new programming language.
You'll probably get back the time you invested. Things like deploying production-ready application with a single command using e.g. Helm is quite powerful and can save you money over using managed services.
Yes, there is a learning curve. It's not a Heroku where you enter one command and it just works. You'll need to spend a few days to become familiar with the basics concepts. And it may take weeks grasp the advanced concepts. But it's less complexity than learning something like a new programming language.
You'll probably get back the time you invested. Things like deploying production-ready application with a single command using e.g. Helm is quite powerful and can save you money over using managed services.
Kubernetes is "hard" when you're not running on the cloud. The solutions for persistent storage and networking are a mess when you're running on self-hosted infrastructure.
This is a trap. The best way to do storage on K8s is not to do it. Especially on-premises.
The networking used to be simple, you could just set up a few static routes and manually assign blocks of CIDRs to nodes and be done. I'm sure there there are some newer networking API components that obfuscated the whole thing in the name of "simplicity" because nobody understands networking anymore.
The networking used to be simple, you could just set up a few static routes and manually assign blocks of CIDRs to nodes and be done. I'm sure there there are some newer networking API components that obfuscated the whole thing in the name of "simplicity" because nobody understands networking anymore.
Fair enough, I was referring to "using" Kubernetes as opposed to managing your own infrastructure. I can imagine how running it on self-hosted servers outside of a popular cloud provider can be hard.
This was also my question/frustration. Kubernetes is not so easy to learn, especially if your focus is on other programming projects and you just want a easy and scalable personal server setup.
I've created a tool called 'Swarmlet' over the past few weeks, which tries to mimic Dokku, so it combines git and Docker Swarm mode for easy app deployments. With some services included like Traefik for routing and automatic SSL using Let's Encrypt, with Consul as a distributed secrets store. https://github.com/swarmlet/swarmlet
Definitely not production-ready, and quite some things to do, but it's a nice POC which actually works for me.
I've created a tool called 'Swarmlet' over the past few weeks, which tries to mimic Dokku, so it combines git and Docker Swarm mode for easy app deployments. With some services included like Traefik for routing and automatic SSL using Let's Encrypt, with Consul as a distributed secrets store. https://github.com/swarmlet/swarmlet
Definitely not production-ready, and quite some things to do, but it's a nice POC which actually works for me.
Kubernetes really isn't as difficult as you might think. Early on it was "hard" because most people were learning containers as well as Kubernetes.
Also how that we have managed services like EKS, GKE, AKS it's really straight forward to learn.
I would suggest start with a GKE cluster and deploy nginx.
Also how that we have managed services like EKS, GKE, AKS it's really straight forward to learn.
I would suggest start with a GKE cluster and deploy nginx.
There are many managed services, which is great, but how does this work when your storage is somewhere else? For instance I might decide to put the storage (postgres and minio) on a cheap Hetzner dedicated server, how would this work with EKS? I am especially trying to understand doing analytics and ML with dask.distributed. I think running compute on managed Kubernetes with storage inside another server could (hybrid?) should be ok, but I haven't figured this part yet.
Yeah, hybrid cloud and multi-cluster is still not a first class citizen. Kubernetes wasn't originally meant for this, and while I think there exist some solutions, they are quite early stage. Of course nothing prevents you to put your data somewhere else and load it, you just won't be able to manage Kubernetes resources across clusters. But you wouldn't be able to do this with an approach like Docker Swarm either. It has nothing to do with managed services, it's just that Kubernetes clusters, managed or not, are not meant to span regions.
You can use headless services[1] to incorporate external endpoints. There's also various CoreDNS plugins[2] that do wonders.
[1]: https://kubernetes.io/docs/concepts/services-networking/serv... [2]: https://coredns.io/plugins/
[1]: https://kubernetes.io/docs/concepts/services-networking/serv... [2]: https://coredns.io/plugins/
You can use a persistent volume claim in multiple containers.
I would suggest using a Google Disk or EBS volume (Depending on if you are using GKE or EKS. They can be mounted to multiple containers at a time (your consumers). Portable and durable.
Keep this volume updated with your data. Maybe a sync process from your other server.
I would suggest using a Google Disk or EBS volume (Depending on if you are using GKE or EKS. They can be mounted to multiple containers at a time (your consumers). Portable and durable.
Keep this volume updated with your data. Maybe a sync process from your other server.
You’ll get charged an arm and a leg for egress (from aws) so have to pretty much commit to aws built in storage (either ebs or efs mounted volumes) or local machine storage (eg nvme drives)
There is the k3s distro of kubernetes
K3s is pretty approachable.
There is AWS ECS as well
Mesos and K3S are two other choices in that space.
This guide can help people that want to pass DCA exam from Docker Inc.
Kubernetes, for better or for worse, has clearly completely won. Not that this article isn't interesting for it's own sake, but it's just something to consider.
1. https://github.com/docker/swarmkit/graphs/contributors 2. https://github.com/docker/classicswarm/graphs/contributors