Why Friday Production Deployments?
18 comments
Cynical take: your contract normally says sometimes like "38 hours a week plus any reasonable additional time blah blah"
If there is an outage on Friday night or Saturday it doesn't affect feature work M-F. Company gets free labour.
Less cynial: continuous deployment agile scrummaster culture says you can deploy anytime so choosing a risky time is OK (risk of missing out on poisoning yourself with alcohol on a Friday night).
This is because (ignoring any reality) deployments should never fail or introduce regressions. Agile gods said so! Friday is a flex for agileness.
If there is an outage on Friday night or Saturday it doesn't affect feature work M-F. Company gets free labour.
Less cynial: continuous deployment agile scrummaster culture says you can deploy anytime so choosing a risky time is OK (risk of missing out on poisoning yourself with alcohol on a Friday night).
This is because (ignoring any reality) deployments should never fail or introduce regressions. Agile gods said so! Friday is a flex for agileness.
Almost always at our company it's because of a push by a higher up who won't be the one staying late when it all blows up. It's bad management at it's finest.
Also, Monday releases usually suck as well. We shoot for Tuesday launches, since Mondays are usually when people are catching up on busy work, or most often out sick. Besides, it gives everyone a chance to review their launch obligations, do final pre-launch checks, and an opportunity to make a go/no go call during work hours, if you shoot for morning launches.
Also, Monday releases usually suck as well. We shoot for Tuesday launches, since Mondays are usually when people are catching up on busy work, or most often out sick. Besides, it gives everyone a chance to review their launch obligations, do final pre-launch checks, and an opportunity to make a go/no go call during work hours, if you shoot for morning launches.
Yep, Tuesday looks much better than a Monday.
There are different levels of risk involved with different types of changes, and there's also risk involved in not making some changes. But aside from that, it is useful to separate the meaning of deployments(pushing a build from a code change to production) from releases(enabling the functionality of that code for users/consumers).
Releases? Sure, you want those to happen early in the week so that you can monitor the rollout and how users interact with it. Deployments? Ideally, deployments change absolutely nothing about the performance characteristics of the system until a feature flag is enabled. There is some cruft in that and it requires cleaning up after yourselves, but when used right it's no different from standing up a new deployable that nobody interacts with yet as far as risk goes.
All in all, if you have a healthy CI/CD setup, use risk mitigating strategies such as feature flags/automated rollbacks, good monitoring/alerting on performance and SLOs, and you're not introducing something like 200+ novel lines of code to an existing workload running in production then I don't see the point in waiting to deploy.
Maybe I'm just too much of a fan of finding things that hurt and making them hurt as much as possible so that we are incentivized to improve them though. If something goes wrong on a deployment/release on Friday, it would go wrong on Monday too. Find what was broken about your process that let it slip through and fix that.
Releases? Sure, you want those to happen early in the week so that you can monitor the rollout and how users interact with it. Deployments? Ideally, deployments change absolutely nothing about the performance characteristics of the system until a feature flag is enabled. There is some cruft in that and it requires cleaning up after yourselves, but when used right it's no different from standing up a new deployable that nobody interacts with yet as far as risk goes.
All in all, if you have a healthy CI/CD setup, use risk mitigating strategies such as feature flags/automated rollbacks, good monitoring/alerting on performance and SLOs, and you're not introducing something like 200+ novel lines of code to an existing workload running in production then I don't see the point in waiting to deploy.
Maybe I'm just too much of a fan of finding things that hurt and making them hurt as much as possible so that we are incentivized to improve them though. If something goes wrong on a deployment/release on Friday, it would go wrong on Monday too. Find what was broken about your process that let it slip through and fix that.
Most businesses still operate mainly from M-F. Most software are written for businesses. You can't really afford to bring your clients' business down, that's a lot of potential lawsuits or credits that you need to give back.
If they operate from M-F, its likely users noticed nothing during the weekend. But that does not mean, that for example, data ingestion stops during the weekend.
At this mid-late career stage... this is a real gotcha with being an employee software developer at BigCo or wherever.
We work on writing code M-F, 9-5, when managers and customers are around so we collaborate and troubleshoot together then and stuff, fine.
We need to deploy, rollback, do checkouts, etc. _after hours_ so it doesn't affect the business. Sometimes it's deploy on Friday night... then some problem is found first thing Monday morning of Sunday night (from Asia Monday users) that requires "urgent" attention, so you end up with both a late start and an early finish to the weekend.
Deploying software is a standard, predictable, part of the job/business hours, so it should really be done during regular hours, or otherwise hire some dedicated "second shift" people to do it after hours / during their _normal_, agreed hours. I don't mind being flexible / "going the extra mile" sometimes... but when it is most weekends, it's a big ask, it's basically working 1.5 jobs for 1 salary.
In practice, usually some staff member who wants to "go the extra mile" will volunteer for this thankless task as a way to get noticed... but instead they end up getting burnt out and jaded, and then quit and find the next sucker to saddle with it.
We work on writing code M-F, 9-5, when managers and customers are around so we collaborate and troubleshoot together then and stuff, fine.
We need to deploy, rollback, do checkouts, etc. _after hours_ so it doesn't affect the business. Sometimes it's deploy on Friday night... then some problem is found first thing Monday morning of Sunday night (from Asia Monday users) that requires "urgent" attention, so you end up with both a late start and an early finish to the weekend.
Deploying software is a standard, predictable, part of the job/business hours, so it should really be done during regular hours, or otherwise hire some dedicated "second shift" people to do it after hours / during their _normal_, agreed hours. I don't mind being flexible / "going the extra mile" sometimes... but when it is most weekends, it's a big ask, it's basically working 1.5 jobs for 1 salary.
In practice, usually some staff member who wants to "go the extra mile" will volunteer for this thankless task as a way to get noticed... but instead they end up getting burnt out and jaded, and then quit and find the next sucker to saddle with it.
Do they not do comp time? Last time I worked at a BigCo, they didn’t hand out comp time by default but didn’t fight if I asked for it.
I just pointed out that it was effectively slashing my compensation per effort, during times I especially didn’t want to work. I proposed either comp time, or bonuses amounting to overtime on our “hourly rate” (salary divided by work hours in the year).
My manager opted for comp time, because he could do that without needing to involve HR and all the pay related people. Apparently no one had ever bothered to ask about it, and had just been working nights and weekends for free.
I just pointed out that it was effectively slashing my compensation per effort, during times I especially didn’t want to work. I proposed either comp time, or bonuses amounting to overtime on our “hourly rate” (salary divided by work hours in the year).
My manager opted for comp time, because he could do that without needing to involve HR and all the pay related people. Apparently no one had ever bothered to ask about it, and had just been working nights and weekends for free.
To some extent - an "on call roster" or "comp time" help a bit so it's not just a 24/7 job... but even "comp time" is the kind of thing they'll not explicitly say no to (as you found), but also its not something they'd be doing as a norm, and it would make anyone asking for it seem like less of a "team player", a good candidate for the bottom of the promotion / top of the layoff list.
It just seems like a real double standard / unspoken rule of software at BigCo that it's very difficult to contain to 9-5 M-F like the other jobs, since it's salaried or exempt and they're not tracking hours. To some extent the relatively high salaries compensate for it, but it comes at a real toll over the long term compared to a true 9-5.
It just seems like a real double standard / unspoken rule of software at BigCo that it's very difficult to contain to 9-5 M-F like the other jobs, since it's salaried or exempt and they're not tracking hours. To some extent the relatively high salaries compensate for it, but it comes at a real toll over the long term compared to a true 9-5.
My take is that if your deployments carry so much risk that you're afraid to do it on a Friday, you have some work to do. Bringing down the risk of deploying through good observability, automated rollbacks, separate releases through feature flagging, etc. makes deployments almost a non-event and reduces stress by a lot.
Now, obviously there's always _some_ level of risk, but for the vast majority of changes I have no problem deploying no matter what day it is, and I have yet to pay for it with my weekend for the past decade.
Now, obviously there's always _some_ level of risk, but for the vast majority of changes I have no problem deploying no matter what day it is, and I have yet to pay for it with my weekend for the past decade.
That's a rookie move that risks your weekend. Some deploy Monday because everyone is in meetings planing your week and mistakes can be fixed without people noticing. I prefer Tuesday through Thursday because less mistakes are made. Early mornings are good, lunch time has benefits. Don't do it just before you leave or by the time you get home you could be engulfed.
Thursday for small low risk deployments, Tuesday for higher risk.
Agree, Tuesday sounds much better than on Monday.
Because the Jira deadline (or perhaps OKR deadline) or whatever other deadline set by a PM or a Manager or HR is Friday, and I have to mark the checkbox or move the ticket as done before the deadline so I can claim I finished stuff on time for the our support agreements on performance evaluation period or whatever.
I actually like working at companies that regularly deploy on Friday because companies that do that tend to put a lot more effort into quality and testing so you don't waste a weekend triaging issues.
As a single developer on a project, I'd rather push Friday so I have all weekend to fix it should something go wrong.
if anything goes bad youll have the whole weekend to solve it...similar to why bank bankruptcies are announced friday
Imagine: kids soccer games, visiting cousins and grandma, date at the opera and essential hygiene like cleaning the house are terra nullis.
But not getting your tickets cleared M-F because of outage is war!
But not getting your tickets cleared M-F because of outage is war!
I cannot understand the logic behind it.
Why not a Monday, where all company is available to intervene if required?