Don't deploy on Friday and 3 other unwritten rules of software engineering(clubhouse.io)
clubhouse.io
Don't deploy on Friday and 3 other unwritten rules of software engineering
https://clubhouse.io/blog/dont-deploy-on-frida-3-other-unwritten-rules-of-software-engineering
21 comments
hmm I dunno... If you have CD working correctly you should be shipping multiple times a day depending on the product. If there is a problem with the deploy it should be as simple as rolling back to the container image of the last version...with K8s or ECS this takes seconds.
I really hate the whole "let's plan for a midnight deploy" paradigm... and find it very unconstructive.
I really hate the whole "let's plan for a midnight deploy" paradigm... and find it very unconstructive.
This point seems targeted more toward those who deploy infrequently, e.g. weekly deploy on Tuesdays.
When working on a few microservices with several deploys per service per day per programmer-pair, not deploying on a Friday means a ~20% productivity reduction. We had one of thosr days as there was an important demo day and our work got blocked in staging. When it finally does deploy to prod, you're exposing your users to all the changes at once and if there's an issue it's much harder to isolate the cause.
Our basic rule is don't deploy and leave any day of the week. If something does happen on the weekend, we're unofficially on-call via slack. I can only remember this happening maybe twice in two years.
When working on a few microservices with several deploys per service per day per programmer-pair, not deploying on a Friday means a ~20% productivity reduction. We had one of thosr days as there was an important demo day and our work got blocked in staging. When it finally does deploy to prod, you're exposing your users to all the changes at once and if there's an issue it's much harder to isolate the cause.
Our basic rule is don't deploy and leave any day of the week. If something does happen on the weekend, we're unofficially on-call via slack. I can only remember this happening maybe twice in two years.
You can absolutely push to development, qa, uat, demo even... just don't deploy to production.
How do you rollback a DB schema change?
You deploy in phases. You only remove data in the final phase. I wrote about this at https://engineering.edx.org/django-migration-donts-f4588fd11....
It's called Read-Only Friday for a reason. Unless of course you enjoy spending your weekends debugging and rolling back. No thanks.
"Slack is famous for its role in increasing connectivity between team members, but, at what cost (aside from the monthly fee)?"
And what's bikeshedding?
And what's bikeshedding?
> The term was coined as a metaphor to illuminate Parkinson’s Law of Triviality. Parkinson observed that a committee whose job is to approve plans for a nuclear power plant may spend the majority of its time on relatively unimportant but easy-to-grasp issues, such as what materials to use for the staff bikeshed.
-- https://en.wiktionary.org/wiki/bikeshedding
-- https://en.wiktionary.org/wiki/bikeshedding
A corollary to "Don't deploy on Friday": "Don't deploy on Monday."
Unless you really want to work over the weekend.
Unless you really want to work over the weekend.
Why would deploying on a Monday result in working over the weekend?
I guess it depends on why you're doing the deploy. If the whole point of doing the deploy is to pick up new code changes, then the people responsible for doing those code changes run the risk of having to work over the weekend to make sure they're ready. This can be a big problem at (for example) startups where the same engineers are responsible for writing new code and doing deploys. But it can happen at organizations of all sizes.
If all you're doing is rolling out a new version of Postgres and it's supposed to just work, then a Monday deploy makes perfect sense. However, as a software engineer, the primary reason I've been involved in deployments is because we've written new code and we need to release it. In an ideal world, the code would be completed and rigorously tested by no later than Friday at 5pm. In the not-so-ideal world we actually live in, this isn't always true, which is why I prefer to target deploys for Wednesdays.
If all you're doing is rolling out a new version of Postgres and it's supposed to just work, then a Monday deploy makes perfect sense. However, as a software engineer, the primary reason I've been involved in deployments is because we've written new code and we need to release it. In an ideal world, the code would be completed and rigorously tested by no later than Friday at 5pm. In the not-so-ideal world we actually live in, this isn't always true, which is why I prefer to target deploys for Wednesdays.
If you were working on a weekend on a feature and I were the release manager, I would not be confident in your delivery and I would ask you to wait for the next release in order to test calmly your changes.
You assume management is generally "sane." There's a reason why the likes of Dilbert's Pointy haired Bos (PHB) often ring true.
> If the whole point of doing the deploy is to pick up new code changes, then the people responsible for doing those code changes run the risk of having to work over the weekend to make sure they're ready.
I feel like you're conflating deadlines with deploys. If its ready to go on a Monday then I see no reason why you should hold off doing the deploy. If you have to work all weekend to get it ready to go by Monday then its probably because there's a deadline.
I feel like you're conflating deadlines with deploys. If its ready to go on a Monday then I see no reason why you should hold off doing the deploy. If you have to work all weekend to get it ready to go by Monday then its probably because there's a deadline.
Fair enough, but I'll observe that in my experience deadlines and deploys are often strongly correlated. Management generally doesn't care when code is done, they care when it's done and deployed. I suppose my experience is skewed by the fact that I've spent a lot of my career in startups where sales (or even the CEO) want the code deployed ASAP because they want to demo it.
I don't necessarily disagree with your point about the correlation, but there are a few things I'd like to call out.
First, why not demo partially complete and/or staged but not deployed work? Tangentially, doing things to appease sales is a quick way to reach local optima and build a directionless product (same thing for customers). A good product person will take their requests, identify the problems, throw out things that don't fit the vision, and incorporate the important parts. Sales shouldn't be your drivers, they should be your eyes and ears.
"ASAP" isn't a deadline. It helps you understand urgency, but not importance. Don't waste your time with urgent unimportant things.
> Management generally doesn't care when code is done, they care when it's done and deployed.
I'd argue that these two are the same thing. I'd even take it one step further and say you shouldn't consider your work done until it is deployed and verified working. Thats why you don't deploy on Fridays.
> Fair enough, but I'll observe that in my experience deadlines and deploys are often strongly correlated.
For some code this is true, but not all of it. I'm sure every team is different, but most code I deploy either has no deadline or has a deadline more than a few days out. Of the remaining stuff, deadlines can almost always be pushed back if the work isn't done. Not ideal, but usually a possibility. No need to work over the weekend unless there's a hard deadline which can't be pushed back, which is (or should be) exceedingly rare.
First, why not demo partially complete and/or staged but not deployed work? Tangentially, doing things to appease sales is a quick way to reach local optima and build a directionless product (same thing for customers). A good product person will take their requests, identify the problems, throw out things that don't fit the vision, and incorporate the important parts. Sales shouldn't be your drivers, they should be your eyes and ears.
"ASAP" isn't a deadline. It helps you understand urgency, but not importance. Don't waste your time with urgent unimportant things.
> Management generally doesn't care when code is done, they care when it's done and deployed.
I'd argue that these two are the same thing. I'd even take it one step further and say you shouldn't consider your work done until it is deployed and verified working. Thats why you don't deploy on Fridays.
> Fair enough, but I'll observe that in my experience deadlines and deploys are often strongly correlated.
For some code this is true, but not all of it. I'm sure every team is different, but most code I deploy either has no deadline or has a deadline more than a few days out. Of the remaining stuff, deadlines can almost always be pushed back if the work isn't done. Not ideal, but usually a possibility. No need to work over the weekend unless there's a hard deadline which can't be pushed back, which is (or should be) exceedingly rare.
I assume preparing for the deploy? But of course that could also have happened in the previous week, and just be ready to deploy on Monday...
"Don't have a deployment process that takes a day ahead"
How painful it can be when you realize that your backup process was wrong from the beginning and you're left with a useless gigabytes sized blob when in need to restore your DB...