Not deploying on Friday is a red flag indicating a seriously broken process(twitter.com)
twitter.com
Not deploying on Friday is a red flag indicating a seriously broken process
https://twitter.com/allenholub/status/1637111242610610182
41 comments
Alternatively you can look at it from cost/benefit point of view. Cost of testing grows exponentially as you approach the 100% certainty. You can say "it's probably fine" with an okay test coverage and a few minutes of poking around in a staging environment. But to say "it's 100% definitely fine" you need to fully test every last bit of the whole system, with real data, in realistic environment and load.
If the project is launching a space ship, then investment in every possible testing method is probably worth it. But when maintaining a blog, you can keep costs down if it can wait until Monday.
If the project is launching a space ship, then investment in every possible testing method is probably worth it. But when maintaining a blog, you can keep costs down if it can wait until Monday.
Yes. Also, even actual space ship launches are done when there is the largest margin for error, and so the best chance of success. Not "whenever it's ready"
A literal example of this is that US Space Shuttles are never launched nor land (except if there was an emergency) on the last and first weeks of a year, and until 2007/2008 (where the software was meticulously tested and fixed in summer of 2007) missions are scheduled to avoid December 31 and January 1 in space. This is a precaution against year-end rollover (YERO) bugs (https://abcnews.go.com/Technology/story?id=2699091&page=1).
Exactly. This guy replies later in the thread "there should never be a bug deployed to prod" essentially saying what does it matter
The idea that you can guarantee there's never any bugs in your code is incredibly naïve
The idea that you can guarantee there's never any bugs in your code is incredibly naïve
I'd love to know what company he's worked at that never makes mistakes. Charity Majors who I think is very sharp says something similar about not being afraid to deploy on Fridays. I think she emphasizes more that your observability is on point, developers own their deploys, rollbacks are easy and deploys happen often. I do think those are all things to aspire to but it takes a good deal of effort to make it all happen.
Yes, the attitude is arrogant and demonstrates hubris IMO. Next up, no thought to how to back out changes. Even if your software works perfectly as intended, you can still need to back out because someone else was not prepared or could not adapt to the change quickly enough. Maybe they were your customers who no longer feel happy with paying you money. You are always only a part of the bigger picture, most of which you have no control over and only limited knowledge of.
It indicates a realistic grasp on the world, and a healthy regard for work/life balance.
There is no way you, or anyone, can guarantee that everything you do in the next N changes will go perfectly. There will be some aspect of drift your processes, tooling, and people all miss.
We follow this rule, generally, erring towards 'no' to Friday deployments. Exceptions are made for business-as-usual work or security items with a time limit.
Just make a reasonable calculated decision. What's to gain vs the risk(s).
No need to be preachy and say 'fix your processes'... but I'm tempted. Mind the urgency of your widgets.
Edit: looking over their profile, I get the feeling this is a bit of light gaslighting. They argue in bad faith, stretching the straw man so hard he's spread across the field.
They're incentivized by a crowd of people feeling inadequate, perhaps even impressed; consequently flocking to their 'education'/consultation
At the very least this reads like someone who has lead a few too many meetings and hasn't put their hands to the keyboard for anything serious in a long time.
There is no way you, or anyone, can guarantee that everything you do in the next N changes will go perfectly. There will be some aspect of drift your processes, tooling, and people all miss.
We follow this rule, generally, erring towards 'no' to Friday deployments. Exceptions are made for business-as-usual work or security items with a time limit.
Just make a reasonable calculated decision. What's to gain vs the risk(s).
No need to be preachy and say 'fix your processes'... but I'm tempted. Mind the urgency of your widgets.
Edit: looking over their profile, I get the feeling this is a bit of light gaslighting. They argue in bad faith, stretching the straw man so hard he's spread across the field.
They're incentivized by a crowd of people feeling inadequate, perhaps even impressed; consequently flocking to their 'education'/consultation
At the very least this reads like someone who has lead a few too many meetings and hasn't put their hands to the keyboard for anything serious in a long time.
My last post is getting long, but another thought. I'd almost bet money this person regularly makes perfect the enemy of better
This is quite an adversarial take. The author's follow-up replies are also hostile and dismissive. Smells like the sort of thing LinkedIn"fluencers" would post.
Good luck to whoever works under this person. Doesn't seem that they understand what it is to have a life outside of work.
Good luck to whoever works under this person. Doesn't seem that they understand what it is to have a life outside of work.
I'm gonna have to hard disagree on this. No matter how much testing you do, there will still be unknowns, or emergent properties that only exist in the exact production environment that is not present in test or stage. Shit does and will happen. Not deploying on a Friday night is an acknowledgement of this fact.
> That's an argument to never deploy because there's certainly some bug lurking somewhere. Don't buy it.
There's the author's glib and smug strawman response to similar common sense.
I actually agree with their sentiment, using CI and feature flags should mean having more confidence where one could choose to ship on Friday.
I also agree with the fact that eventually the team will ship a bug. Better to do so when you're likely to see it during the workweek.
There's the author's glib and smug strawman response to similar common sense.
I actually agree with their sentiment, using CI and feature flags should mean having more confidence where one could choose to ship on Friday.
I also agree with the fact that eventually the team will ship a bug. Better to do so when you're likely to see it during the workweek.
Deploying on Friday is a sign that you are a shitty manager and don't value your team.
Yes, you have canaries. Yes, you have processes for when things go wrong. However, at bottom, when something goes awry, you need a human overseeing the unwind.
If something goes wrong after a Friday deploy, you will have to unwind it on a Friday afternoon or weekend. It sucks to be the person dealing with that when, for example, your child has a recital, thanks.
I subscribe to "never say never", but ... Honestly, your feature is NEVER important enough that it couldn't wait until Monday.
(The exception to this is when things are "all hands on deck" for some horrible reason. However, at that point, all your safety systems have probably been deactivated, and you are so far outside of your normal release process that you are deploying immediately for everything.)
Yes, you have canaries. Yes, you have processes for when things go wrong. However, at bottom, when something goes awry, you need a human overseeing the unwind.
If something goes wrong after a Friday deploy, you will have to unwind it on a Friday afternoon or weekend. It sucks to be the person dealing with that when, for example, your child has a recital, thanks.
I subscribe to "never say never", but ... Honestly, your feature is NEVER important enough that it couldn't wait until Monday.
(The exception to this is when things are "all hands on deck" for some horrible reason. However, at that point, all your safety systems have probably been deactivated, and you are so far outside of your normal release process that you are deploying immediately for everything.)
I disagree, I think unless it’s super urgent fix that can’t wait , wait for Monday, folks who are on-call will thank you for this. Also as a culture, it’s big red flag personally to me when employees don’t think about their colleagues who are oncall and just aspire to build systems that are supposed to be built perfectly with perfect coverage of tests that can detect bugs (remember tests can only tell you presence of bugs you expect to find, not absence of all bugs).
Depends whether you have 100% confidence that all possible faults are automatically tested for...
The thing is, that for big deployments you can be never so sure.
There may always be "human factor" that will cause users to click the wrong button without looking at its label.
The thing is, that for big deployments you can be never so sure.
There may always be "human factor" that will cause users to click the wrong button without looking at its label.
The difference between theory and practice is greater in practice than in theory.
There is no 100% certainty, no matter how much you test due to the state space.
Not deploying on Fridays and before holidays is acknowledging this and valuing your coworkers’ life outside of work.
There is no 100% certainty, no matter how much you test due to the state space.
Not deploying on Fridays and before holidays is acknowledging this and valuing your coworkers’ life outside of work.
My favorite formulation is “in theory, theory and practice are the same.”
I'm glad to see all the HN folks seeing straight through this BS
What is the answer? Let’s spend three months “fixing” the process so that we maybe can ship two days earlier? Or check upcoming classes and an in-progress book on AGILE? Bold marketing bait, probably leading to a generic advice.
Unless the changes are a critical bug fix, deploying on Friday is an asymmetric bet with high risk and low reward. That's why people prefer not to do it.
Even if the chances something will go wrong are very slim and you have lots of really great tests, they are still non-zero. Lots of unexpected things can go wrong in modern interconnected systems with lots of network dependencies.
Not deploying on Fridays is not a matter of confidence. It's recognition that the downside is huge (losing your weekend) and the payoff is small for the vast majority of changes you could deploy. That's it.
Even if the chances something will go wrong are very slim and you have lots of really great tests, they are still non-zero. Lots of unexpected things can go wrong in modern interconnected systems with lots of network dependencies.
Not deploying on Fridays is not a matter of confidence. It's recognition that the downside is huge (losing your weekend) and the payoff is small for the vast majority of changes you could deploy. That's it.
I've been in leadership positions in tech companies for a long time. I don't let people schedule Friday deployments because in the off chance that something does go wrong it means that multiple people lose their weekends.
That's because human beings and their lives matter.
I've worked in multiple industries and few of them are so critical that a launch or cutover can't wait a couple days.
That's because human beings and their lives matter.
I've worked in multiple industries and few of them are so critical that a launch or cutover can't wait a couple days.
Telecoms do major changes on Thursday night, so that any issues can be handled by the staff on Friday.
And telecoms in general, have a higher level of reliability than a (squints) "agile-friendly implementation technology like microservices and incremental/evolutionary architecture " consultancy...
And telecoms in general, have a higher level of reliability than a (squints) "agile-friendly implementation technology like microservices and incremental/evolutionary architecture " consultancy...
Why so keen to deploy on Friday? That seems like a red flag
Unless it is an urgent bug fix, there is no reason to push it
Once we deployed on a Friday, but a partner's API also deployed, errors start happening at 2am, the API team only has generic support on call, so our devs keep investigating, logs and changes. In the end everything was OK on our side, it was the integration API that broke, by coincidence, and they were insisting it was us.
So you cannot always control, or know, when your partners deploy. But if you avoid Friday your team can sleep assure it is not their fault
There might be an unconscious bias to deploy on Friday because people want to feel they were productive that week, or manager's pressure to fulfill imaginary targets. So people might rush it just a little bit
Unless it is an urgent bug fix, there is no reason to push it
Once we deployed on a Friday, but a partner's API also deployed, errors start happening at 2am, the API team only has generic support on call, so our devs keep investigating, logs and changes. In the end everything was OK on our side, it was the integration API that broke, by coincidence, and they were insisting it was us.
So you cannot always control, or know, when your partners deploy. But if you avoid Friday your team can sleep assure it is not their fault
There might be an unconscious bias to deploy on Friday because people want to feel they were productive that week, or manager's pressure to fulfill imaginary targets. So people might rush it just a little bit
Hmm.. No.
There was a business that our system supported that had a peak transaction day on Mondays (55% of the weekly volume flow) and Fridays had less than 7% of weekly volume flow. The rest of the days had a tapering of volumes between these levels. No volumes flowed in during the weekends, but downstream processing occurred on Saturdays (of the transactions that came in Thursday/Friday that still remained).
We deployed on Thursdays. Never on Sundays (simply because Monday was a peak day and we would never recover from business SLA failures if things went wrong when we opened the Monday).
It gave our support engineers a weekday to look into issues, if any and escalate to a developer on Friday, if required.
There is such a thing called pragmatism.
There was a business that our system supported that had a peak transaction day on Mondays (55% of the weekly volume flow) and Fridays had less than 7% of weekly volume flow. The rest of the days had a tapering of volumes between these levels. No volumes flowed in during the weekends, but downstream processing occurred on Saturdays (of the transactions that came in Thursday/Friday that still remained).
We deployed on Thursdays. Never on Sundays (simply because Monday was a peak day and we would never recover from business SLA failures if things went wrong when we opened the Monday).
It gave our support engineers a weekday to look into issues, if any and escalate to a developer on Friday, if required.
There is such a thing called pragmatism.
I generally think deploying first and asking questions later is the way to go—when you run in to problems, you get a good opportunity to debug your software development processes to try and prevent similar issues down the line. Something’s going to catch fire eventually, and having good processes built up from past experience is a blessing.
Deploying something buggy on a Monday is a very different situation from deploying something buggy at 5pm on Friday, though. Most of the time your customers aren’t going to care about getting a feature 3 days early, but your team WILL care that their weekend just got eaten up for no reason.
Deploying something buggy on a Monday is a very different situation from deploying something buggy at 5pm on Friday, though. Most of the time your customers aren’t going to care about getting a feature 3 days early, but your team WILL care that their weekend just got eaten up for no reason.
Saw this making rounds as today's villain.
I confess I agree, though. Thing is, there are a lot of red flags. And life is too whatever to think this red flag is a hill to die on.
Will it be a reason not to work somewhere? No. Of course not. Is it susp? Absolutely.
I confess I agree, though. Thing is, there are a lot of red flags. And life is too whatever to think this red flag is a hill to die on.
Will it be a reason not to work somewhere? No. Of course not. Is it susp? Absolutely.
> is too whatever to think this red flag is a hill to die on
Gotta choose your battles carefully - any one battle could probably consume 100% of your time.
Gotta choose your battles carefully - any one battle could probably consume 100% of your time.
On my previous team I used to think read only Friday was a good idea, but it was in fact a symptom of quality problems. My current team has a strong culture of testing and quality and we ship whenever the feature is ready.
By curiosity, do you also ship during company-wide holidays or events ?
For isntance if your whole company is at a yearly retreat in some vacation park, including part of the SRE team, but two or more devs are still working for whatever reasons, would they ship a full refactoring of some service ?
I'm always interested in where the boundaries are, if you could share.
For isntance if your whole company is at a yearly retreat in some vacation park, including part of the SRE team, but two or more devs are still working for whatever reasons, would they ship a full refactoring of some service ?
I'm always interested in where the boundaries are, if you could share.
Yay, more useless hot takes from Twitter. So glad they're on my HN feed.
Test in production. Fix the bugs your servers complain about, then fix the bugs your users complain about, then move on. Unless you’re building critical systems that support life or your failure mode is bankruptcy, it probably doesn’t matter. Look at Twitter right now. Things are broken left and right and it’s still more popular than anything Allen Holub has ever made.
Embrace “good enough” and collect your payment. You aren’t changing the world.
Embrace “good enough” and collect your payment. You aren’t changing the world.
I don't generally agree with this take. I also don't think allowing deploys on Friday and letting the team decide their risk/reward ratio is a bad thing. Teams should be empowered to make their own decisions. If they are consistently making bad decisions that will become apparent and can be addressed in a variety of ways. Most likely by the team itself because they don't want to be working on the weekend.
Totally! I tend both kinds of systems: Those with good tests and monitoring, I will deploy them whenever with confidence. Fire and forget. Those with no tests and scant monitoring? Ball of patches on Tuesday morning. Watch the service for a while.
In both cases I'm ready to patch whatever comes up. But the excpectation is different. So I keep less slack for the well tended depolys.
In both cases I'm ready to patch whatever comes up. But the excpectation is different. So I keep less slack for the well tended depolys.
Believing that you have everything under control is a red flag and indicates a lack of experience. No matter how good your process you will eventually release something broken into production and essentially what you are saying is that those that will have to offer up their weekends and evenings are not important to you.
Can you give 100% guarantee (no 99.99% but 100%) your changes won't break a critical part of the system. If the answer is NO then don't deploy on Friday's or any other time of the week when resources are not expected to be immediately available.
Let's say you have 99% certainty that your changes are perfect. That means every two years you royally fuck up your weekend. Is that infrequent enough to not wait two days?
I don’t think that follows. I don’t need to take a dive on a bike to know that wearing a helmet is a good idea
Boy I would hate to work for this guy
[deleted]
There is some magical belief that people can understand everything whatever the size of the system, and with enough efforts and processes you can guarantee nothing bad will happen 24h after deploying new features.
I actually see this belief as a red flag, as it usually comes either from hubris or misunderstanding of the technical complexity of their stack. Interestingly even if there are exceptional people that actually have a crazy good understanding of everything, the org is probably heavily relying on them, and when they're gone the culture left won't be kind to those who aren't exceptional.
PS: one can assume that whatever happens 24h after deploy on a friday is the on-call team's problem, but that's not an attitude I'd value in a company.