No deployments on Fridays: A good practice for software development teams(deploybot.com)
deploybot.com
No deployments on Fridays: A good practice for software development teams
https://deploybot.com/blog/no-deployments-on-fridays-a-good-practice-for-software-development-teams
33 comments
You can strive to reduce stress but may not be able to bring it to 0.
Like you should have for extinguishers even if all precautions have been taken to prevent fire.
Like you should have for extinguishers even if all precautions have been taken to prevent fire.
That's fair, but easier said than done. Reducing the stress/risk of deploys is a long, hard process at the best of times, and also is something that you have to continue doing in perpetuity as the complexity/failure impact of the systems being deployed grows. It's also an area where many organizations struggle to (re)commit resources appropriately.
If you ever find yourself behind the curve in that area (as in: deploys are risky and stressful events), then a good short-term mitigation for the symptoms might be to avoid deploys on Fridays or similar, while also committing resources towards improving the root cause (deploys suck).
If you ever find yourself behind the curve in that area (as in: deploys are risky and stressful events), then a good short-term mitigation for the symptoms might be to avoid deploys on Fridays or similar, while also committing resources towards improving the root cause (deploys suck).
I’m 50/50 on this. Then Thursday becomes “omg ship it ASAP” day and Friday becomes firefight Friday.
I think meta had it right: put a little warning on the ship button “it’s after 1pm on Friday, do you really need to ship this now?”
I think meta had it right: put a little warning on the ship button “it’s after 1pm on Friday, do you really need to ship this now?”
Here is a different story from another perspective of a niche group of team. I led a few teams — small groups for each project. I realized that quite a few of the team have a strange habit of deploying around 5-6PM on Fridays! What I found out was interesting.
They were young, single, and don’t really seem to be those looking out to go to bars, etc. By deploying late, and staying over-time — the company’s rule of “company-sponsored-dinners” after office-time kicks in. They get free dinner, and air-conditioned room which are better than their homes or their room in their shared flats. They anyway ends up watching videos/movies, have dinner, and go back home late.
It is not uncommon, in India, to be in offices which are much better than their homes.
They were young, single, and don’t really seem to be those looking out to go to bars, etc. By deploying late, and staying over-time — the company’s rule of “company-sponsored-dinners” after office-time kicks in. They get free dinner, and air-conditioned room which are better than their homes or their room in their shared flats. They anyway ends up watching videos/movies, have dinner, and go back home late.
It is not uncommon, in India, to be in offices which are much better than their homes.
Haha, that is most excellent detective work. Always deploy on Fridays, and enjoy the perks is also a valid strategy I guess :D
If deploying new code is stressful, then here are some tips to try...
Improve your deploying until you feel it's easy, routine, pipelined, automatic, monitored, tested, and post-deploy tracked so it's provably working. Good practical ways to start on this path can include devops, IaC, E2E testing, etc.
Then get good at tactics such as rainbow rollouts, blue-green, canaries, feature flags, dark launch, self-throttling capabilities, and self-healing systems. A simple example is to deploy to 1% of people, and when incoming metrics prove success, then increase the percentage.
Success looks like fearless deployments. Some teamwork ideas to try are group ownership, blameless incident responses, causal analysis based on system theory (CAST), service level agreements with stakeholders, and specific software quality attributes for availability, recoverability, and the like.
Improve your deploying until you feel it's easy, routine, pipelined, automatic, monitored, tested, and post-deploy tracked so it's provably working. Good practical ways to start on this path can include devops, IaC, E2E testing, etc.
Then get good at tactics such as rainbow rollouts, blue-green, canaries, feature flags, dark launch, self-throttling capabilities, and self-healing systems. A simple example is to deploy to 1% of people, and when incoming metrics prove success, then increase the percentage.
Success looks like fearless deployments. Some teamwork ideas to try are group ownership, blameless incident responses, causal analysis based on system theory (CAST), service level agreements with stakeholders, and specific software quality attributes for availability, recoverability, and the like.
Ideally this is where everyone ought to be anyways. In a vacuum, all of these would stop bad deployments in theory. But in practice, no amount of prep can address and prevent every potential way a deployment can go sideways.
Plus if you're B2B, your clients are likely mentally checked-out on Friday anyways so they won't even see the new stuff.
Plus if you're B2B, your clients are likely mentally checked-out on Friday anyways so they won't even see the new stuff.
For software orgs deploying web services, ”no deployments on Fridays” should be seen as a temporary workaround, not a best practice to follow. It is a symptom of poor automatic QA processes (linting, automatic tests, etc.) and lack of capabilities like feature flags.
The only exception is orgs that need manual QA for regulatory reasons and mobile development, where you follow a weekly/bi-weekly cadence because of external Apple/Google reviews.
The only exception is orgs that need manual QA for regulatory reasons and mobile development, where you follow a weekly/bi-weekly cadence because of external Apple/Google reviews.
Exactly! It really depends on the setup. Good CI? Deploy at will. Nonexistent CI? Tuesday morning.
Of course it can go wrong in either case. But I will keep less slack when the CI is good.
Of course it can go wrong in either case. But I will keep less slack when the CI is good.
A more general, but less catchy version of the rule is: don't do things with a significant probability of requiring an "all hands on deck" response when doing so will be unacceptably slow/difficult.
I get the point here but for many organizations and types of companies this is a terrible and unworkable idea. A lot of companies have to wait till weekend for maintenance (e.g. trading/market making apps), and so your only options are deploy on weekend which means weekend work or on friday afternoon/evening. Once week starts there are no good times to do a release. So for a vast number of super critical systems friday is THE day to do a release.
Yes, that means potential weekend work if it goes wrong, and it increases the odds of a bad monday... but both of those can be fixed with good testing/automation.
Yes, that means potential weekend work if it goes wrong, and it increases the odds of a bad monday... but both of those can be fixed with good testing/automation.
I think it depends entirely on who the end user is.
It's completely understandable in situations where there is no allowable downtime except the weekend.
But if it's a SaaS product to categorize sticky notes... probably not worth risking ruining a Friday.
It's completely understandable in situations where there is no allowable downtime except the weekend.
But if it's a SaaS product to categorize sticky notes... probably not worth risking ruining a Friday.
The engineering week ends at 9 a.m. Monday.
So 'I'll finish it this week' means you have until Monday morning.
So 'I'll finish it this week' means you have until Monday morning.
A company based on doing deploys doesn't seem to understand that the goal should be de-risking the deploys, not de-risking your weekends?
Charity Majors explains it better than I ever could. https://charity.wtf/2019/10/28/deploys-its-not-actually-abou...
Charity Majors explains it better than I ever could. https://charity.wtf/2019/10/28/deploys-its-not-actually-abou...
You can't "derisk" deploys, you can only reduce the amount of risk they incur. Not deploying the day before a nonwork day remains solid advice.
Sure, you can't remove all risk, but you can reduce it and reduce the risk of unexpected firefighting work. You can implement rolling or blue/green deployments that roll-back on failure. The key is being able to identify failures, which means having meaningful canaries to identify issues. Those canaries can also improve your operational awareness so they aren't just a deployment investment.
If you don't deploy on Friday then you lose 20% of your deploy opportunity. If you only deploy once or twice a week, then sure - choose Monday/Wednesday or something, but if you deploy more frequently then improve your deployment safety.
If you don't deploy on Friday then you lose 20% of your deploy opportunity. If you only deploy once or twice a week, then sure - choose Monday/Wednesday or something, but if you deploy more frequently then improve your deployment safety.
Your response sounds like “you can’t derisk deploys, you can only derisk deploys”
I was unclear, or perhaps I misunderstood the comment I was replying to. I read it as "deploying on Fridays isn't a problem if you derisk the deploys".
My response was that you can't derisk deploys to the extent that Friday deployments are good practice.
My response was that you can't derisk deploys to the extent that Friday deployments are good practice.
Maybe you can’t derisk deploys on Friday, but we’ve derisked them just fine, and are eager to deploy more frequently Friday and every other day or the week - right now we do once per day, looking forward to continuous delivery to production.
One of my last employers did this and I begged them to just let a couple people work a five day schedule that included Saturday and or Sunday (I wanted to be one of those people) so we could deploy any day, trouble shoot any day, but they were against it. I think people need two (actually at least three, but that's a different discussion) days off every week, but I also don’t think people need to work identical schedules. Put another way: you must give the castle guards a break from staving off attacks but you must not give them all the same break.
There are two types of Friday deployments:
A) Deploying major changes
B) Deploying small changes/bug fixes
Don't do A.
A) Deploying major changes
B) Deploying small changes/bug fixes
Don't do A.
I'm half joking here but at one of my past workplaces, the development process was something along: fix bug, get reviewed, pull request tested by QA, integrate, retest after integration, run in UAT and only then eventually deploy to production.
Anything coming up at any of these phases would throw it back to development ("fix bug" phase) so we were joking that if we get a P0 critical priority bug on Monday morning and aren't done fixing it by afternoon then there's no chance in hell it will make it to the deploy on Friday :)
Anything coming up at any of these phases would throw it back to development ("fix bug" phase) so we were joking that if we get a P0 critical priority bug on Monday morning and aren't done fixing it by afternoon then there's no chance in hell it will make it to the deploy on Friday :)
This reads like super low effort blogspam. Seeing this advice from a deploy vendor makes me think, “these people must not be serious about their product”.
If your deploys break a lot, fix your testing and deploy process! Then you’ll have much lower stress every day of the week, instead of low stress on Friday and high stress every other weekday.
At least figure out how to make rollback super quick and easy. Then at least if something goes wrong at 11am when you’re rolling out the Friday deploy, you can roll back and be stress free again by 1pm.
If your deploys break a lot, fix your testing and deploy process! Then you’ll have much lower stress every day of the week, instead of low stress on Friday and high stress every other weekday.
At least figure out how to make rollback super quick and easy. Then at least if something goes wrong at 11am when you’re rolling out the Friday deploy, you can roll back and be stress free again by 1pm.
I have read in some other post that it’s not about the on-call and development overhead, but the potential operational burden to the company or its customers. What about some bad algorithm which causes millions of dollars in revenue or something blocking your customers over the weekend? Who is going to fix this kind of issues during the weekend?
It’s better to deploy on Sundays then.
This article doesn’t come right out and say it but repeatedly suggests working on the weekend is the core problem.
I like Friday releases because I don’t care about working weekends. In my mind slower uptake from weekend users can help identify problems before major uptake Monday morning. Although that’s theoretical because it hasn’t mattered in practice.
I like Friday releases because I don’t care about working weekends. In my mind slower uptake from weekend users can help identify problems before major uptake Monday morning. Although that’s theoretical because it hasn’t mattered in practice.
> I don’t care about working weekends.
This seems like a take I don't hear often. I won't work on a weekend unless the service is on fire and I'm the only one who can save it (which means I've done a bad job).
I'm fine with Friday deployments as long as we have a good release pipeline with multiple stages of validation and slow rollout, but if I see that my team is having to regularly work weekends because we deploy Friday, I'll stop deploying and respect their time.
This seems like a take I don't hear often. I won't work on a weekend unless the service is on fire and I'm the only one who can save it (which means I've done a bad job).
I'm fine with Friday deployments as long as we have a good release pipeline with multiple stages of validation and slow rollout, but if I see that my team is having to regularly work weekends because we deploy Friday, I'll stop deploying and respect their time.
It really depends on a lot imo.
If you still get days off it can be nice to have them during most peoples work week so you can get errands done easily. As long as it's not consistently every weekend so you can still do weekend things with other people.
It can also be easier to get stuff done on the weekend when most other people/distractions aren't around.
I actually am really happy with how my current team works, where you will be on call during the weekend but only expected to respond to pages so usually you just need to be able to respond and not actually working.. if lots of things page it's even more incentive to improve your systems to not get bugged on the weekend. But even if nothing pages you we still take two days of our choice off the next week.
If you still get days off it can be nice to have them during most peoples work week so you can get errands done easily. As long as it's not consistently every weekend so you can still do weekend things with other people.
It can also be easier to get stuff done on the weekend when most other people/distractions aren't around.
I actually am really happy with how my current team works, where you will be on call during the weekend but only expected to respond to pages so usually you just need to be able to respond and not actually working.. if lots of things page it's even more incentive to improve your systems to not get bugged on the weekend. But even if nothing pages you we still take two days of our choice off the next week.
If I'm working on a weekend, it's because the world is on fire. There is no other reason I'd be willing to give up precious personal time for an employer.
[deleted]
It would seem more valuable to instead put efforts to make code deploys less 'stressful'. If you are regularly deploying code and causing huge stress you are doing something else wrong. This reminds me of the old corporate 'you can wear jeans on Friday'. Somehow wearing them on Friday is ok, but the rest of the time will lead to dispair?