Ask HN: How to do rolling deployments without Kubernetes?
Every month, there seems to be a new post stating ~"don't use Kubernetes yet", but what should we use instead if we cannot have downtime?
18 comments
Instead of talking about specific technologies/software/services, I'll give a quick rundown how you can achieve this in theory, and hopefully it can apply to whatever you're currently using.
The requirements:
- Your application should be able to run on arbitrary ports, preferably controlled with env vars or similar
- You need to have something in front of your application, like nginx or apache
- Whatever webserver you have should be able to "hang"/pause/suspend requests while you switch application version
- Each version you create needs to be concerned about what the previous version did. Breaking changes needs to happen across multiple versions, where you can soft-deprecate something, and a release after that, actually "break" it
- You need to have some sort of "healthchecking" that can tell you if the new version is OK or not
The implementation:
- You have Version 1 running of your backend on port X
- You want to deploy the new version, so you deploy it to your server but the web server in front still serves requests from Version 1
- Run healthchecks against Version 2
- Once they pass, tell web server to "pause" in-flight requests
- Switch web server configuration to use Version 2 application instead (this can be combined with the previous step, `nginx reload` would combine these for example)
- Stop Version 1 from running
- Repeat for each new version
And now you've achieved deploying a new version of your application without any failing requests.
(Sidenote: You can replace hanging/pausing requests with graceful shutdown of your webserver (meaning it waits for no pending requests) if you have a low amount of traffic)
The requirements:
- Your application should be able to run on arbitrary ports, preferably controlled with env vars or similar
- You need to have something in front of your application, like nginx or apache
- Whatever webserver you have should be able to "hang"/pause/suspend requests while you switch application version
- Each version you create needs to be concerned about what the previous version did. Breaking changes needs to happen across multiple versions, where you can soft-deprecate something, and a release after that, actually "break" it
- You need to have some sort of "healthchecking" that can tell you if the new version is OK or not
The implementation:
- You have Version 1 running of your backend on port X
- You want to deploy the new version, so you deploy it to your server but the web server in front still serves requests from Version 1
- Run healthchecks against Version 2
- Once they pass, tell web server to "pause" in-flight requests
- Switch web server configuration to use Version 2 application instead (this can be combined with the previous step, `nginx reload` would combine these for example)
- Stop Version 1 from running
- Repeat for each new version
And now you've achieved deploying a new version of your application without any failing requests.
(Sidenote: You can replace hanging/pausing requests with graceful shutdown of your webserver (meaning it waits for no pending requests) if you have a low amount of traffic)
While this is right, I'm not sure why you mentioned pausing requests. If you can run v1 and v2 at the same time, you could switch where the new requests are going without affecting the old ones. There's many ways to do that, but what I do is point nginx upstream at a path which is a symlink to a Unix socket of a specific version. So it's more like: start the new version, check, update symlink, wait until old connections drop, kill old version.
Yeah, nginx and apache have ways of reloading the configuration without dropping requests (by waiting for existing ones to finish with old workers, and spinning up new workers with the new config in the case of nginx) but doesn't apply in every situation (like when you're not using nginx) so figure I'd write about the general principle instead of specifically for nginx.
You can do this with (or without) any proxy too. Iptables (due to conntrack) can make decisions about the first packet in a connection, then save the result. That means you can match incoming traffic on --state NEW and route to a specific local port. Changing one iptables entry is atomic, so you can swap the version seamlessly this way.
I think the idea is worth mentioning because people often think it's hard to achieve / something special. But there's a hundred ways to do seamless version swap on a host.
I think the idea is worth mentioning because people often think it's hard to achieve / something special. But there's a hundred ways to do seamless version swap on a host.
Wow, that's a good point. Obvious in hindsight, but personally I never considered iptables for that job. Thanks for sharing this!
> Instead of talking about specific technologies/software/services, I'll give a quick rundown how you can achieve this in theory, and hopefully it can apply to whatever you're currently using.
I wish more people approached their software engineering teaching in this way.
Thank you for the great comment.
I wish more people approached their software engineering teaching in this way.
Thank you for the great comment.
> I wish more people approached their software engineering teaching in this way.
Me too, so why not do it myself? :) Thanks for the feedback
Me too, so why not do it myself? :) Thanks for the feedback
I don’t know much about kubernetes (I suspect I may soon since our current process annoys me) and I’m not a DevOps engineer (but we don’t have any, and any consultants we’ve tried is basically just a little more confident using the official documentation than we are).
Anyway, how we’ve done it is by setting up a pipeline, that builds and deploys a docker container of a repository to a deployment a lot on a cloud application/function/whatever you call them if it passes the build phase. Then depending on whether or not the deployment passes the criteria set up for it, it either automatically swaps deployment slots with the production slot or waits for manual confirmation to do so. On anything that is allowed a reload, we don’t swap deployment slots but instead redeploy, typically with minimal downtime.
The reason it’s annoying is because it takes a lot of time to set up for each pipeline, and when you need to be able to swap deployment slots with now downtime you also need to handle things like build processes taking global variables or clients needing to be told to reload parts or them.
I’m not sure kubernetes is the answer for us, but we’re certainly going to look for a way to make the whole process smarter as it sometimes takes up more time to setup the pipeline and deployment environment than building the service that needs to be deployed.
Anyway, how we’ve done it is by setting up a pipeline, that builds and deploys a docker container of a repository to a deployment a lot on a cloud application/function/whatever you call them if it passes the build phase. Then depending on whether or not the deployment passes the criteria set up for it, it either automatically swaps deployment slots with the production slot or waits for manual confirmation to do so. On anything that is allowed a reload, we don’t swap deployment slots but instead redeploy, typically with minimal downtime.
The reason it’s annoying is because it takes a lot of time to set up for each pipeline, and when you need to be able to swap deployment slots with now downtime you also need to handle things like build processes taking global variables or clients needing to be told to reload parts or them.
I’m not sure kubernetes is the answer for us, but we’re certainly going to look for a way to make the whole process smarter as it sometimes takes up more time to setup the pipeline and deployment environment than building the service that needs to be deployed.
Zero-downtime deployments have existed years (perhaps decades?) before k8s was released to the public.
Controlling the upstream traffic via haproxy/nginx allowed Ops to roll out blue-green, canary and waterfall/rolling deployment methods albeit with more human interaction.
Digital Ocean have this [0] article from 2014 on load balancing.
[0] https://www.digitalocean.com/community/tutorials/an-introduc...
Controlling the upstream traffic via haproxy/nginx allowed Ops to roll out blue-green, canary and waterfall/rolling deployment methods albeit with more human interaction.
Digital Ocean have this [0] article from 2014 on load balancing.
[0] https://www.digitalocean.com/community/tutorials/an-introduc...
This article provides one possible approach (and includes links to an example implementation): https://blog.jakubholy.net/2013/09/05/blue-green-deployment-...
Assuming we're talking over-HTTP traffic, you can use a load balancer in-front of your Kubernetes cluster.
We did this at work and works fine (blogpost will be out soon). It allows us to decommission a cluster by removing the cluster from the LB. There's nothing new about this technique.
We're using DOKS + CloudFlare Traffic but you can use any LB service (no affiliation, as all products there are pro's and con's).
Once the setup is ready, operations are easy:
a) Remove cluster from the LB
b) Perform cluster operations (ingress upgrade, k8s upgrade, possibly disruptive daemonset operations)
c) Add cluster to LB
Another pro is that when a region has cloud-provider level issues (happened with FRA1 few days ago) we can remove the cluster from the LB and stop worrying about it until the the issue is fixed. LBs have health-checks and such, to automate addition/remove of clusters.
We did this at work and works fine (blogpost will be out soon). It allows us to decommission a cluster by removing the cluster from the LB. There's nothing new about this technique.
We're using DOKS + CloudFlare Traffic but you can use any LB service (no affiliation, as all products there are pro's and con's).
Once the setup is ready, operations are easy:
a) Remove cluster from the LB
b) Perform cluster operations (ingress upgrade, k8s upgrade, possibly disruptive daemonset operations)
c) Add cluster to LB
Another pro is that when a region has cloud-provider level issues (happened with FRA1 few days ago) we can remove the cluster from the LB and stop worrying about it until the the issue is fixed. LBs have health-checks and such, to automate addition/remove of clusters.
Dokku does this.
You either deploy from command line or something like a GitHub action (or CI).
Container 1 and 2 are briefly running concurrently, and networking is updated to point to the new container once it finishes its build phase, and passes any pre-defined tests (or after 10 seconds of running without crashing).
You either deploy from command line or something like a GitHub action (or CI).
Container 1 and 2 are briefly running concurrently, and networking is updated to point to the new container once it finishes its build phase, and passes any pre-defined tests (or after 10 seconds of running without crashing).
Ecs does this automatically https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...
If you want no downtime server updates, you have choices:
a) remove server(s) from load balancing, finish requests/connections in progress, restart with new software, add to load balancing
b) hot loading
c) start a new server instance and pass it the listen socket (or if your OS isn't great, drop syns for a bit while you close the listen socket on the old server and open a new listen socket for the new server)
I like hotloading, but it's not appropriate for all updates, so you need to have a way to handle restart based updates as well.
a) remove server(s) from load balancing, finish requests/connections in progress, restart with new software, add to load balancing
b) hot loading
c) start a new server instance and pass it the listen socket (or if your OS isn't great, drop syns for a bit while you close the listen socket on the old server and open a new listen socket for the new server)
I like hotloading, but it's not appropriate for all updates, so you need to have a way to handle restart based updates as well.
Use container registries and automatic pull deployment. AWS fargate, Azure App Services both allow this method of working.
I.e. github action to do your build, test. Dont do the docker push to the repository unless it passes the tests. Always do your docker push from the correct branch. If you need approval for release, build that into the pipeline.
Azure pipelines pretty much the same deal.
You don't need kubernetes for a few services, it only gets useful when you have a lot.
I.e. github action to do your build, test. Dont do the docker push to the repository unless it passes the tests. Always do your docker push from the correct branch. If you need approval for release, build that into the pipeline.
Azure pipelines pretty much the same deal.
You don't need kubernetes for a few services, it only gets useful when you have a lot.
I use Cloud Run which has all of this built into it by default.
https://cloud.google.com/blog/products/serverless/cloud-run-...
https://cloud.google.com/blog/products/serverless/cloud-run-...
HashiCorp Nomad , it is not complex as Kubernetes. Been using it to manage 1000's of servers
Docker Swarm