Time for a code-yellow?: A blunt instrument that works(nilam.ca)
nilam.ca
Time for a code-yellow?: A blunt instrument that works
https://nilam.ca/2024/11/02/time-for-a-code-yellow-a-blunt-instrument-that-works/
117 comments
Making decisions is by definition cutting off stuff. It is hard, and requires courage, both of which no-one wants to expend. Because of course we want it all. Reducing WIP is probably the most efficient and least liked management tool.
Most teams I've been a part of have the syndrome you describe at every level of the hierarchy. I'm in constant battles to reduce the scope of what I have to build to avoid spending a year before it sees the light of day. I integrated a team a little while ago to help them do better. 7 people, 12 features in progress. I remember an organizational-level presentation we had was presenting the 15 priorities and accompanying 11 focus areas for the next 6 months. In the company wide evals, on top of our "self-defined" personal goals, we also have 3 imposed ones (d&i, security and learning) that were imposed by the corporation over the years as a reaction to something, and no-one will ever have the courage to remove.
This is almost a disease, and I've lost faith in that it will be cured some day
Most teams I've been a part of have the syndrome you describe at every level of the hierarchy. I'm in constant battles to reduce the scope of what I have to build to avoid spending a year before it sees the light of day. I integrated a team a little while ago to help them do better. 7 people, 12 features in progress. I remember an organizational-level presentation we had was presenting the 15 priorities and accompanying 11 focus areas for the next 6 months. In the company wide evals, on top of our "self-defined" personal goals, we also have 3 imposed ones (d&i, security and learning) that were imposed by the corporation over the years as a reaction to something, and no-one will ever have the courage to remove.
This is almost a disease, and I've lost faith in that it will be cured some day
Exactly and literally; from the roots of "de"="off" and "caedere"="to cut" [0]
If you can't cut something off, you cannot decide.
decision (n.)
mid-15c., decisioun, "act of deciding," from Old French décision (14c.), from Latin decisionem (nominative decisio) "a decision, settlement, agreement," noun of action from past-participle stem of decidere "to decide, determine," literally "to cut off," from de "off" (see de-) + caedere "to cut" (from PIE root *kae-id- "to strike").
[0] https://www.etymonline.com/word/decision
If you can't cut something off, you cannot decide.
decision (n.)
mid-15c., decisioun, "act of deciding," from Old French décision (14c.), from Latin decisionem (nominative decisio) "a decision, settlement, agreement," noun of action from past-participle stem of decidere "to decide, determine," literally "to cut off," from de "off" (see de-) + caedere "to cut" (from PIE root *kae-id- "to strike").
[0] https://www.etymonline.com/word/decision
This dilbert-style management has become very normalized in tech - to the detriment of everyone.
It fundamentally comes from a place where there are too many layers of managers, none of who want to make crucial decisions that can bite them back.
Aka leadership only in org chart but not in setting direction, leading from the front, or accepting tradeoffs.
Ultimately it all boils down to org chart leadership. If managers could be voted out of the org chart by people below them in the chart, they'd have no choice but to show more spine and help the people below.
But this is not going to happen in corporations. So the best thing to do is quiet quitting. There is a reason this exists.
It fundamentally comes from a place where there are too many layers of managers, none of who want to make crucial decisions that can bite them back.
Aka leadership only in org chart but not in setting direction, leading from the front, or accepting tradeoffs.
Ultimately it all boils down to org chart leadership. If managers could be voted out of the org chart by people below them in the chart, they'd have no choice but to show more spine and help the people below.
But this is not going to happen in corporations. So the best thing to do is quiet quitting. There is a reason this exists.
It gets worse: A manager in those kinds of organizations that is truly scared also doesn't want people under them to make decisions either, so eventually everyone learns inaction.
At sope point the organization comes up with a code freeze, and a manager is afraid of changes near the freeze. Them said freeze keeps expanding as you get down to ICs, and eventually nobody gets to release anything in December, even if something is wrong and is costing the company users and money.
It's bad when people choose to quiet quit on management, but it's scarier when they basically are told to do it, in not so many words.
At sope point the organization comes up with a code freeze, and a manager is afraid of changes near the freeze. Them said freeze keeps expanding as you get down to ICs, and eventually nobody gets to release anything in December, even if something is wrong and is costing the company users and money.
It's bad when people choose to quiet quit on management, but it's scarier when they basically are told to do it, in not so many words.
> code freeze
But they're saying they're doing "CI/CD". Why they do code freezes? Why are people still afraid to deploy on Friday?
Because they're not doing Continuous Delivery.
But they're saying they're doing "CI/CD". Why they do code freezes? Why are people still afraid to deploy on Friday?
Because they're not doing Continuous Delivery.
Deployment freezes are (should be) completely normal and justifiable.
Do you really think that Amazon is deploying (non-emergency/operational) code changes during BFCM or re:Invent? Or that Fox/CBS/NBC are deploying code changes during the Super Bowl?
Also (unless you have some business need to deploy before the weekend) why deploy on a Friday afternoon? You're just asking for latent defects to page you in over the weekend. Save yourself the trouble and delay that deployment till Monday morning. Your team will thank you in the long run.
Sometimes repetitional and/or sleep damage isn't worth being uncompromising on idyllic CI/CD goals.
Do you really think that Amazon is deploying (non-emergency/operational) code changes during BFCM or re:Invent? Or that Fox/CBS/NBC are deploying code changes during the Super Bowl?
Also (unless you have some business need to deploy before the weekend) why deploy on a Friday afternoon? You're just asking for latent defects to page you in over the weekend. Save yourself the trouble and delay that deployment till Monday morning. Your team will thank you in the long run.
Sometimes repetitional and/or sleep damage isn't worth being uncompromising on idyllic CI/CD goals.
I disagree, from a certain point of view. Consider the CDN. One customers downtime is the IRS’s filing date. The IRS’s slow time is during the Super Bowl. The NFL’s season of rest is baseball season. Baseball goes dark on cyber Monday.
When should the CDN undertake updates? Continuously.
(Obviously I do get your main point: the NFL should not upgrade at halftime of the Super Bowl. But there is a time and a place for different models.)
When should the CDN undertake updates? Continuously.
(Obviously I do get your main point: the NFL should not upgrade at halftime of the Super Bowl. But there is a time and a place for different models.)
I think we actually agree with one another. Code freezes/deployment windows should be justified by business risk, not idealised always be pushing mentality.
CDNs and cloud providers are a bit of a special case. AWS definitely has soft blocks during tax peak and large sporting events, CDNs (guessing here) would likely be doing follow the moon(?) (i.e. off-peak) rolling releases
My disagreement with the parent comment was driven by lack of nuance
CDNs and cloud providers are a bit of a special case. AWS definitely has soft blocks during tax peak and large sporting events, CDNs (guessing here) would likely be doing follow the moon(?) (i.e. off-peak) rolling releases
My disagreement with the parent comment was driven by lack of nuance
typo: Reputational not repetitional ofc
There are times when mistakes will cause more serious problems for customers. There are times when staff will be less available to aid recovery. A good strategy shouldn't deny these truths.
IME this is much more prevalent in organizations that hire "nontechnical managers". I personally will never work in such companies again, the experience of having your boss have literally no clue what it is you do for a job is not one I'll sign up for again.
IME the differentiator is not a technical background; it's whether they care more about ladder climbing/appeasing their own managers or doing something useful.
This is a more accurate assessment.
A quick filter of effective management is asking the question: What do you hope to gain out of this job?
If the answer is a promo or ladder-climbing, you know that they will make sub optimal decisions that guide their ladder-climbing journey.
A quick filter of effective management is asking the question: What do you hope to gain out of this job?
If the answer is a promo or ladder-climbing, you know that they will make sub optimal decisions that guide their ladder-climbing journey.
I regularly ask the question "how do know I'm doing good" in exactly those words. You'd be surprised at how easy it is to identify their management style from the answer to that question.
Can you elaborate?
I think the problem is a bit harder than that. I've seen a lot of managers who were once technical, but whose last hands-on experience was years, or decades ago. They have a vague idea of what they need to know/do but lack the skills to find things out themselves. and they turn into a major drag on productivity.
I should have been more specific--what I mean by nontechnical managers is managers who don't commit code regularly.
Disagree - I've seen this all over. The problem has absolutely nothing to do with tech whatsoever, and everything to do with decision making.
You also get developers made manager that are terrible at management and trust so lean on micromanaging.
There's a happy medium somewhere.
There's a happy medium somewhere.
aka the entire c suite
From TFA, everything is an emergency:
> we leaned on Code Yellow’s so aggressively that they became part of our standard operating cadence.
Code Employees-Prep-Their-Résumés
> we leaned on Code Yellow’s so aggressively that they became part of our standard operating cadence.
Code Employees-Prep-Their-Résumés
I have a highest priority queue where X is going into, and where Y is already comfortably sitting together with the other highest priority items (of course i don't have other queues :). Everybody loves that their X is going to be a highest priority item for me. "a" not "the". Thanks, English.
I've found this is only a short-term solution. It's much more effective IME to force the priority call onto the person making the demand.
I've done this as a team lead/scrum master. You want to add something to the sprint? No probs - you just have to choose something the same size to take out. You want to take a person off the team? Sure, you just have to negotiate with our clients about which work gets taken out of the sprint.
It's amazing how many things are suddenly less urgent when someone is forced to make an actual trade-off rather than push the pressure down onto the delivery team.
I've done this as a team lead/scrum master. You want to add something to the sprint? No probs - you just have to choose something the same size to take out. You want to take a person off the team? Sure, you just have to negotiate with our clients about which work gets taken out of the sprint.
It's amazing how many things are suddenly less urgent when someone is forced to make an actual trade-off rather than push the pressure down onto the delivery team.
>to force the priority call onto the person making the demand
I prefer no forcing. Especially when it is say a moronic VP which i don't have enough force to exert upon him. In USSR we had a cartoon where a greedy customer wants a fur hat from a sheepskin, and he asks whether the hat tailor can make 2 hats from the same skin, and the tailor says sure he can ... that goes to 7 hats, and when the customer comes to receive the completed order he receives the 7 hats as ordered https://youtu.be/gSpjDi2BrQk?t=193
Smart people understood what hats they would get given the sheepskin and the number requested. And less smart - well, it is the cartoon fun.
I prefer no forcing. Especially when it is say a moronic VP which i don't have enough force to exert upon him. In USSR we had a cartoon where a greedy customer wants a fur hat from a sheepskin, and he asks whether the hat tailor can make 2 hats from the same skin, and the tailor says sure he can ... that goes to 7 hats, and when the customer comes to receive the completed order he receives the 7 hats as ordered https://youtu.be/gSpjDi2BrQk?t=193
Smart people understood what hats they would get given the sheepskin and the number requested. And less smart - well, it is the cartoon fun.
My dad used to flippantly say he had three piles of papers on his desk: "urgent", "very urgent" and "no longer urgent".
As time passed the papers would get moved from one pile to the next.
As time passed the papers would get moved from one pile to the next.
I call this "top of the notes" syndrome, from my years in medicine, where every now and then a consultant physician would print something out and say "this is important, it should be on top of the patient's notes" (i.e. clipped on the cover of the physical folder, as opposed to affixed in it, so as to be the "first" thing one sees on grabbing that folder).
Which sounds great, until you have two of those.
And in reality, quite often you'd see folders where the "top of the notes" stuff had more entries than the normal contents of the folder itself. So, ironically people would skip the top stuff and jump straight into the folder as normal.
Which sounds great, until you have two of those.
And in reality, quite often you'd see folders where the "top of the notes" stuff had more entries than the normal contents of the folder itself. So, ironically people would skip the top stuff and jump straight into the folder as normal.
Managers are taught to live in a world of emergencies. I believe that someone who takes a measured and thoughtful approach to things is excluded from management out of principal.
> At Beacon, we are not going to wait for the next crisis to sweat our biggest challenges; we will build a culture that makes Sweating The Problem our default.
Brilliant! I love it. It's Code Yellow all the time. If it worked once a year at Instacart, and Google made so much progress with it, we'll just turn "the Yellow" on all day, every day.
Then maybe they can have a Code Red for when it's super-duper important to "sweat" the problem.
> you walk away with the confidence that you can handle whatever comes next.
You don't even need a bonus, just enjoy your new found sense of accomplishment. And maybe a subscription to the jelly-of-the-month club [1]
[1] https://kitchychristmas.com/jelly-of-the-month/
Brilliant! I love it. It's Code Yellow all the time. If it worked once a year at Instacart, and Google made so much progress with it, we'll just turn "the Yellow" on all day, every day.
Then maybe they can have a Code Red for when it's super-duper important to "sweat" the problem.
> you walk away with the confidence that you can handle whatever comes next.
You don't even need a bonus, just enjoy your new found sense of accomplishment. And maybe a subscription to the jelly-of-the-month club [1]
[1] https://kitchychristmas.com/jelly-of-the-month/
Yeah I thought the same. I would not want to work for this guy or any organisation that thinks like him.
It's the manager's job to prioritise. There is always more work than can be done, and always more problems than can be solved with the available resource. Trying to solve that prioritisation problem by "sweating" - ignoring employee's genuine and real need for work-life balance - is not going to work long term.
To the author: please learn how to actually manage. Honestly, you'll do much better. I look forward to reading your article about how you learned to manage problems instead of sweating your team into the ground.
It's the manager's job to prioritise. There is always more work than can be done, and always more problems than can be solved with the available resource. Trying to solve that prioritisation problem by "sweating" - ignoring employee's genuine and real need for work-life balance - is not going to work long term.
To the author: please learn how to actually manage. Honestly, you'll do much better. I look forward to reading your article about how you learned to manage problems instead of sweating your team into the ground.
> Trying to solve that prioritisation problem by "sweating" - ignoring employee's genuine and real need for work-life balance - is not going to work long term.
Indeed. Even the words "sweating the problem" just sounds callous and inconsiderate.
It can even work for Google, where can pull in different team members at different times. But in a smaller company that means everyone's work-life balance is permanently stressed.
Indeed. Even the words "sweating the problem" just sounds callous and inconsiderate.
It can even work for Google, where can pull in different team members at different times. But in a smaller company that means everyone's work-life balance is permanently stressed.
> ... and strongly encourages the team to sacrifice the ‘L’ and ‘B’ from Work-Life-Balance.
This manager doesn't get it.
This manager doesn't get it.
I'm glad I'm not the only one who read:
* I promised myself never again. Never again would I call a code-yellow. Code-yellows suck, drain team morale, and they leave a lingering distaste amongst all those involved. Yet, during my 8-years at Instacart, they were our most effective and consistent weapon in ensuring we made meaningful progress on our hairiest problems.*
to be a really long-winded way of saying, "I'm utter shit at management, refuse to prioritize until it's a company-threatening crisis, and I'm happy to make my team suffer for my incompetence."
I'm sure the employee churn at Instacart was in no way related to this fact.
And now he's operating a PE/rollup firm.
* I promised myself never again. Never again would I call a code-yellow. Code-yellows suck, drain team morale, and they leave a lingering distaste amongst all those involved. Yet, during my 8-years at Instacart, they were our most effective and consistent weapon in ensuring we made meaningful progress on our hairiest problems.*
to be a really long-winded way of saying, "I'm utter shit at management, refuse to prioritize until it's a company-threatening crisis, and I'm happy to make my team suffer for my incompetence."
I'm sure the employee churn at Instacart was in no way related to this fact.
And now he's operating a PE/rollup firm.
Person who closes the most tickets gets a car
Second place gets a set of steak knives
Third place is you're fired
Always Be (C)losing
Second place gets a set of steak knives
Third place is you're fired
Always Be (C)losing
Coffee is for closers.
When you’re on your deathbed, I’m sure you will be glad that you spent your kid’s taekwondo tournament writing an email so that an e-commerce company could make line go up.
Some of these people don't understand that they won't see that tournament again, and they won't be able to make their relationship "line go up" if they don't build the foundations when their children are still children.
> "But you were always a good man of business, Jacob," faltered Scrooge, who now began to apply this to himself.
> "Business!" cried the Ghost, wringing its hands again. "Mankind was my business; charity, mercy, forbearance, and benevolence, were, all, my business. The deals of my trade were but a drop of water in the comprehensive ocean of my business!"
-- A Christmas Carol, by Charles Dickens
> "Business!" cried the Ghost, wringing its hands again. "Mankind was my business; charity, mercy, forbearance, and benevolence, were, all, my business. The deals of my trade were but a drop of water in the comprehensive ocean of my business!"
-- A Christmas Carol, by Charles Dickens
> Some of these people don't understand that they won't see that tournament again
Just as many don't care or see it as an inconvenience to be there. They might do less damage to their families by just being honest and skipping.
Just as many don't care or see it as an inconvenience to be there. They might do less damage to their families by just being honest and skipping.
If you don't care about your child's development and be there to support them, then you can just skip having children in the first place, or building a family altogether.
There's no shame in being honest to yourself and others, as you said.
There's no shame in being honest to yourself and others, as you said.
Having kids is an important tickbox for ladder climbing, senior execs are always talking about their own or asking about peoples children.
> If you don't care about your child's development and be there to support them, then you can just skip having children in the first place, or building a family altogether.
This is much easier to do now compared to 30 years ago, but there's still a lot of pressure to conform to the spouse/kids pipeline.
This is much easier to do now compared to 30 years ago, but there's still a lot of pressure to conform to the spouse/kids pipeline.
But but but!
My Personal Brand Consultant told me a beautiful wife with two kids and a late model luxury minivan are essential components of my fundraising endeavors! Round B investments are 27.3% less likely to succeed for single founders under 26 years old compared to founders with families.
My Personal Brand Consultant told me a beautiful wife with two kids and a late model luxury minivan are essential components of my fundraising endeavors! Round B investments are 27.3% less likely to succeed for single founders under 26 years old compared to founders with families.
The child is a future host for the wealth. In this sense the wealth's priorities still come first.
Our ancestors suffered greatly with far greater inconveniences.
Rather bourgeois remark. Unsurprising that that developed countries' populations are in question with gatekeeping like that.
Rather bourgeois remark. Unsurprising that that developed countries' populations are in question with gatekeeping like that.
I think your point of view is dismissive and simplistic, or you grew up in a household that didn't value you as a human being when you were a child.
The world is by no means a better place today, and there are still "far greater" conveniences people go through every day, young, adult, or old alike. Many are invisible now, but being invisible doesn't make them less valid or less damaging. If you still want to see some visible "far greater inconveniences," look a bit lower on the globe, to the Middle East and Africa. These people will be ancestors of generations to come if they can stay alive.
Being able to grow past a certain age without dying is a very low bar to clear at this age. Humans are much more sophisticated today in both single and collective forms. Our ideas are more complex, our behaviors are more sophisticated, and ethical and personal standards are higher... All in all, we are progressing. Technological progress is a byproduct and enabler, but human (mental) progress is the first trigger.
So, if someone wants a trophy child and never attends to them after being born, it's unjust to the child, and the trophy owner is entitled to reap what they sow during their development.
If a couple in a developed country wants to have a single child and raise them the best they can to surpass their parents and have a better life, I see no gatekeeping here. Yes, sometimes the best intentions bring the worst outcomes, but at least the intention is good to begin with.
It's rather selfish to justify bad living conditions and having a hard life because "our ancestors lived that way" in the past. With that attitude, you're the gatekeeper of progress and a better world that we or our descendants can create. It might be a little less crowded at the end, but it'll be their choice and challenge to overcome.
The world is by no means a better place today, and there are still "far greater" conveniences people go through every day, young, adult, or old alike. Many are invisible now, but being invisible doesn't make them less valid or less damaging. If you still want to see some visible "far greater inconveniences," look a bit lower on the globe, to the Middle East and Africa. These people will be ancestors of generations to come if they can stay alive.
Being able to grow past a certain age without dying is a very low bar to clear at this age. Humans are much more sophisticated today in both single and collective forms. Our ideas are more complex, our behaviors are more sophisticated, and ethical and personal standards are higher... All in all, we are progressing. Technological progress is a byproduct and enabler, but human (mental) progress is the first trigger.
So, if someone wants a trophy child and never attends to them after being born, it's unjust to the child, and the trophy owner is entitled to reap what they sow during their development.
If a couple in a developed country wants to have a single child and raise them the best they can to surpass their parents and have a better life, I see no gatekeeping here. Yes, sometimes the best intentions bring the worst outcomes, but at least the intention is good to begin with.
It's rather selfish to justify bad living conditions and having a hard life because "our ancestors lived that way" in the past. With that attitude, you're the gatekeeper of progress and a better world that we or our descendants can create. It might be a little less crowded at the end, but it'll be their choice and challenge to overcome.
If thing A is important enough to declare a "code yellow" in order to ignore things B, C, and D to focus on A, then were B, C, and D really that important? Could you have focused your team on A from the start, making a "code yellow" unnecessary? (Hint: yes, you could have, so the question should be why you didn't, and whether you could have seen it coming.)
I've seen this happen a lot with mediocre leaders. "Code Yellow" equivalents happen because they weren't able to understand that A was really the most important thing, typically because B, C, and D were important for optics or politics, but not genuinely important to the customer or the problem at hand.
A "Code Yellow" is a useful political tool to move an organization to focus a bit more on problem solving by saying "I don't care about your politics, your org charts, whatever, just solve the damn problem." In that sense, it really does work.
I've seen this happen a lot with mediocre leaders. "Code Yellow" equivalents happen because they weren't able to understand that A was really the most important thing, typically because B, C, and D were important for optics or politics, but not genuinely important to the customer or the problem at hand.
A "Code Yellow" is a useful political tool to move an organization to focus a bit more on problem solving by saying "I don't care about your politics, your org charts, whatever, just solve the damn problem." In that sense, it really does work.
The worst in the story is that thing B, C and D were probably decided, and commendeered by the same exec who, 3 months later will decide that thing A doesn't move enough and "they need to act on that". It's a practical tool to correct their indecisiveness and errors, and who's paying the price for it...
Political constraints are real constraints! It's very possible to end up in a state where declaring an emergency is genuinely better than any of the options to avoid it. Consider:
* A is clearly important and will have to be "code yellowed" if it's not addressed in the next few months. But you've got to drop one of B, C, or D in order to avoid the code yellow.
* B is one of your best engineers' passion project. She's not a big fan of C or D, and she might run away to Google if you tell her they're more important.
* C is a promise you made to a customer to unblock some big deal. They might not sign if you tell them you changed your mind and would prefer to work on some projects B and D they don't care at all about.
* D is a demand from your VP to address some company priority, and he doesn't think the effort-reward ratio from B or C are very high. If you try to drop D he will probably get mad and tell you that you're not allowed to.
* A is clearly important and will have to be "code yellowed" if it's not addressed in the next few months. But you've got to drop one of B, C, or D in order to avoid the code yellow.
* B is one of your best engineers' passion project. She's not a big fan of C or D, and she might run away to Google if you tell her they're more important.
* C is a promise you made to a customer to unblock some big deal. They might not sign if you tell them you changed your mind and would prefer to work on some projects B and D they don't care at all about.
* D is a demand from your VP to address some company priority, and he doesn't think the effort-reward ratio from B or C are very high. If you try to drop D he will probably get mad and tell you that you're not allowed to.
> It is alluring because it allows for existing plans to be deprioritized, removes any/all ambiguity around what is most important at the moment, and strongly encourages the team to sacrifice the ‘L’ and ‘B’ from Work-Life-Balance.
When I read this, I understand that when management fails, and wants something to be get done, because reasons, they can just pull all stops and declare temporary slavery until the problem is solved.
This should be normally a "once in a decade" event. Not "once in a year" instrument. Being proud of removing "life" and "balance" from employees' reality reads like the worst power trip ever.
I worked in similar environments. Never again. We have our "code yellow"s in my current job, but we know when it's going to come, and prepare ourselves and our lives. Go through it, pat ourselves on the back for the good job, learn our lessons for the things we fail along the way, and continue our journey.
When I read this, I understand that when management fails, and wants something to be get done, because reasons, they can just pull all stops and declare temporary slavery until the problem is solved.
This should be normally a "once in a decade" event. Not "once in a year" instrument. Being proud of removing "life" and "balance" from employees' reality reads like the worst power trip ever.
I worked in similar environments. Never again. We have our "code yellow"s in my current job, but we know when it's going to come, and prepare ourselves and our lives. Go through it, pat ourselves on the back for the good job, learn our lessons for the things we fail along the way, and continue our journey.
Also the line about sending this email while on the weekend to his staff at his son's tae kwon do is particularly absurd.
It is hard to pick the most absurd line. I quite enjoyed "2018: Scaling our infrastructure to keep the site up on Sunday’s (we were having constant outages every week)". Take a moment to reflect that implies achieving one 9 of reliability required an emergency all-hands response that bypassed their normal processes. I may not be an expert in high-reliability systems but that isn't how I expect the problem to be tackled.
My read is the management team are not very capable and that this gentleman is not a natural leader. Doesn't sound like he knows how to build systems in a planned and thoughtful manner and has embraced a classic lurching-into-crisis strategy for want of understanding any alternatives.
EDIT Also juxtaposes beautifully with the "‘keeping the lights on’ fallacy" he mentions. Those crazy engineers with their belief that they need to try and keep the lights on. Also, when I forcefully pull them away from that, why do the lights go out on Sunday???
My read is the management team are not very capable and that this gentleman is not a natural leader. Doesn't sound like he knows how to build systems in a planned and thoughtful manner and has embraced a classic lurching-into-crisis strategy for want of understanding any alternatives.
EDIT Also juxtaposes beautifully with the "‘keeping the lights on’ fallacy" he mentions. Those crazy engineers with their belief that they need to try and keep the lights on. Also, when I forcefully pull them away from that, why do the lights go out on Sunday???
People who setup an infrastructure which "just works" don't get promoted. You want to be extinguishing fires and be seen doing it. To be the hero who worked their ass off all week-end to save the company. That's how you think you'll get that juicy promotion.
From my experience, this is only valid for immature companies and/or immature management. A good tech lead or manager knows that if there are no problems, the infra people are doing their jobs very well.
> My read is the management team are not very capable and that this gentleman is not a natural leader.
I mean it fits perfectly well for a lot of "thought leaders" like this. They, after all, have the time to write oh so many blogs about it.
I mean it fits perfectly well for a lot of "thought leaders" like this. They, after all, have the time to write oh so many blogs about it.
> I may not be an expert in high-reliability systems but that isn't how I expect the problem to be tackled.
You probably know way better than me, but in my experience, configuring things correctly on healthy hardware gets you 99.99% by default. Adding some surplus capacity adds another 9, at least.
Then you build from there (autoscaling, hardware failovers, etc. etc.).
You probably know way better than me, but in my experience, configuring things correctly on healthy hardware gets you 99.99% by default. Adding some surplus capacity adds another 9, at least.
Then you build from there (autoscaling, hardware failovers, etc. etc.).
Eh, 99.99 is less than an hour downtime a year. I know of almost nothing of any size that actually achieved that save certain areas of the power grid.
That does depend somewhat on what you count as "downtime". Larger distributed systems may not suffer a complete outage in the whole year, and more than one of the services my team runs has that level of successful response ratio.
As a person who runs an OpenStack cluster, I can pull 100% uptime many months back to back with minimum intervention.
The guy comes across as someone who thinks that non-work activities are just inconveniences for his and the company's well-being.
Work is there to support other areas of life. Personal life is not something that we "do" in the time which is left from work. It's not the second class citizen when compared to work.
Did he failed at this notion so badly, so he turned to this workaholic, everything else is secondary lifestyle, I don't know.
Work is there to support other areas of life. Personal life is not something that we "do" in the time which is left from work. It's not the second class citizen when compared to work.
Did he failed at this notion so badly, so he turned to this workaholic, everything else is secondary lifestyle, I don't know.
> strongly encourages the team to sacrifice the ‘L’ and ‘B’ from Work-Life-Balance.
And what is the incentive for the rank and file to do this? Execs get millions of dollars worth of stock every 3 months. The rank and file isn't getting this. Why would the sacrifice their L and B?
To keep their jobs? Naw. For this, they will cut corners and lie.
Management practices have become so terrible in tech. It is no wonder that the best people keep switching jobs every few years. There is zero incentive to work under such poor management practices.
And what is the incentive for the rank and file to do this? Execs get millions of dollars worth of stock every 3 months. The rank and file isn't getting this. Why would the sacrifice their L and B?
To keep their jobs? Naw. For this, they will cut corners and lie.
Management practices have become so terrible in tech. It is no wonder that the best people keep switching jobs every few years. There is zero incentive to work under such poor management practices.
> It is alluring because it allows for existing plans to be deprioritized, removes any/all ambiguity around what is most important at the moment, and strongly encourages the team to sacrifice the ‘L’ and ‘B’ from Work-Life-Balance.
Why would anyone go along with this unless it came with a FAT bonus?
Why would anyone go along with this unless it came with a FAT bonus?
The idea is that you put in the extra work and he keeps the extra profit. Sounds like a sweet deal.
I also accept PTO credits as payment for voluntary overtime
Except not as a 1:1 trade!
Coming from an old school engineering company, this article frankly reads as a little insane to me. I assumed "Code Yellow" was a catastrophe on par with a tornado hitting a factory or a massive security breach. Instead the examples are not hitting artificial growth metrics and needing to launch advertising. It's bad enough to blow up teams planned work but this is what you demand employees (not founders mind you) sacrifice their personal life for (i.e. get paid less per hour and not see their families)? The author not only acknowledges the time he is stealing from employees but that stressing them out is the point. If you're 8 years into a company and lurching from (artificial) crisis to crisis to "sweat the teams" something is seriously wrong.
I work in public health so our outages are critical to safe treatment of our patients.
We have the concept of a Major Incident/Major Event (because the term Code Yellow is already co-opted to mean any administrative fault across a hospital which might impact the flow of patients).
These are all-hands-on-deck moments. They may be called by any member of staff who discovers an event which will impact our service, though it will be ratified by a Major Incident Manager (MIM). While the event is underway the MIM is God; except when it affects staff welfare. If a member of staff says that they cannot attend a war room for whatever reason, then the MIM will move onto the next person up the chain, even calling in directors if they feel it is required.
What is being described above is tech-bro shittery. Calling a major event because you haven't hit a sales target should keep the C-Suite up at night, sure, but calling in techs and devs and 'sacrificing the L and the B in Work Life Balance'? The C-Suite should be making the strategic decisions to reverse the decline, not suddenly drag everyone into a meeting to fix their lack of foresight and working towards an end that the average tech/dev cannot influence.
We have the concept of a Major Incident/Major Event (because the term Code Yellow is already co-opted to mean any administrative fault across a hospital which might impact the flow of patients).
These are all-hands-on-deck moments. They may be called by any member of staff who discovers an event which will impact our service, though it will be ratified by a Major Incident Manager (MIM). While the event is underway the MIM is God; except when it affects staff welfare. If a member of staff says that they cannot attend a war room for whatever reason, then the MIM will move onto the next person up the chain, even calling in directors if they feel it is required.
What is being described above is tech-bro shittery. Calling a major event because you haven't hit a sales target should keep the C-Suite up at night, sure, but calling in techs and devs and 'sacrificing the L and the B in Work Life Balance'? The C-Suite should be making the strategic decisions to reverse the decline, not suddenly drag everyone into a meeting to fix their lack of foresight and working towards an end that the average tech/dev cannot influence.
I've been a major incident manager. It works great to have a hardass like me giving orders on a conference bridge: stuff gets fixed really fast! People pull out laptops at little league games and get it done. The outage is solved, and business goes on. I look like a genius.
The problem is that, if a major incident is the only process in your organization that results in things getting done in a reasonable amount of time, there is a large temptation to declare them for everything. Somebody wasted 6 months failing to renew our contract with the payment processing vendor because other stuff was being prioritized above it? Open a major incident and we'll get them to call the vendor CEO at home on Sunday. We're a big account, so they'll answer. It's the end of the fiscal year and if we don't decomm this expensive system like was promised, a VP somewhere will lose their bonus? Declare a major incident and ElevenLathe will get it done in a few hours, even if it is Thanksgiving.
I would page on-call people at any hour of the day for real emergencies without hesitation, but always asked for orders in writing to page people (not executives, but real people) who weren't on call, or for nonsense like this. This got annoying so they started assigning other incident managers to the nonsense.
The problem is that, if a major incident is the only process in your organization that results in things getting done in a reasonable amount of time, there is a large temptation to declare them for everything. Somebody wasted 6 months failing to renew our contract with the payment processing vendor because other stuff was being prioritized above it? Open a major incident and we'll get them to call the vendor CEO at home on Sunday. We're a big account, so they'll answer. It's the end of the fiscal year and if we don't decomm this expensive system like was promised, a VP somewhere will lose their bonus? Declare a major incident and ElevenLathe will get it done in a few hours, even if it is Thanksgiving.
I would page on-call people at any hour of the day for real emergencies without hesitation, but always asked for orders in writing to page people (not executives, but real people) who weren't on call, or for nonsense like this. This got annoying so they started assigning other incident managers to the nonsense.
>there is a large temptation to declare them for everything.
Agreed. We have a very tight description for an Incident and, therefore, a major Incident. We don't call MI's often but anyone can call one.
Agreed. We have a very tight description for an Incident and, therefore, a major Incident. We don't call MI's often but anyone can call one.
It's not called a Code Red.
Not all catastrophes are immediate in nature. Most organizations have incident response protocol which works well enough for an immediate catastrophe. Non-immediate, gradual threats can represent a much greater risk, because there's no inflection point (beyond "way too late") where the threat becomes so imminent as to trigger that response. Code Yellows are a mechanism to artificially force that inflection point before it is too late.
Not all catastrophes are immediate in nature. Most organizations have incident response protocol which works well enough for an immediate catastrophe. Non-immediate, gradual threats can represent a much greater risk, because there's no inflection point (beyond "way too late") where the threat becomes so imminent as to trigger that response. Code Yellows are a mechanism to artificially force that inflection point before it is too late.
> Not all catastrophes are immediate in nature
Of course, but if it's not immediate, why does it warrant forcing people into a Zoom meeting on a Saturday while you're at your kid's Tae Kwon Do competition? If it's not immediate, why do you have to "encourage the team to sacrifice the ‘L’ and ‘B’ from Work-Life-Balance"?
Either it's an immediate catastrophe which requires immediate action, or it can be handled by giving it absolute priority during regular business hours.
Of course, but if it's not immediate, why does it warrant forcing people into a Zoom meeting on a Saturday while you're at your kid's Tae Kwon Do competition? If it's not immediate, why do you have to "encourage the team to sacrifice the ‘L’ and ‘B’ from Work-Life-Balance"?
Either it's an immediate catastrophe which requires immediate action, or it can be handled by giving it absolute priority during regular business hours.
I've had code yellows before, and they were never "immediate meeting on saturday". They are essentially projects that take precedence, and gets rid of the "who's job is this" question. I think Code yellows work great.... this guy is not using them how I would expect.
> It's not called a Code Red.
The etymology is not green/yellow/red. It's just not-Yellow or yes-Yellow. See Stephen Levy's In The Plex (2011) pg186:
“A Code Yellow is named after a tank top of that color owned by engineering director Wayne Rosing. During Code Yellow a leader is given the shirt and can tap anyone at Google and force him or her to drop a current project to help out. Often, the Code Yellow leader escalates the emergency into a war room situation and pulls people out of their offices and into a conference room for a more extended struggle.”
The etymology is not green/yellow/red. It's just not-Yellow or yes-Yellow. See Stephen Levy's In The Plex (2011) pg186:
“A Code Yellow is named after a tank top of that color owned by engineering director Wayne Rosing. During Code Yellow a leader is given the shirt and can tap anyone at Google and force him or her to drop a current project to help out. Often, the Code Yellow leader escalates the emergency into a war room situation and pulls people out of their offices and into a conference room for a more extended struggle.”
> The etymology is not green/yellow/red. It's just not-Yellow or yes-Yellow.
Um, no.
Today, Google has Code Reds, Code Yellows, Code Purples, and Code Greens... and this is after standardizing to remove other made-up terms like Code Mauve.
I wish I were making up these terms.
Um, no.
Today, Google has Code Reds, Code Yellows, Code Purples, and Code Greens... and this is after standardizing to remove other made-up terms like Code Mauve.
I wish I were making up these terms.
Not sure why it's getting downvoted, definitely experienced Code Yellows, Code Reds and Code Purples at Google. Red is (obviously) worse than Yellow and IIRC was a total code freeze for a period of time. IIRC there was a Code Red around memory at some point because the supply chain was literally so backed up that google couldn't get enough DIMMs and reasonably sized services had to stop deploying because there wasn't enough compute capacity.
Purple is/was "developer experience is so bad we need to stop developing new functionality and make the current functionality usable."
Purple is/was "developer experience is so bad we need to stop developing new functionality and make the current functionality usable."
The rough hierarchy today:
- Code red: the situation is actively causing active business harm.
- Code yellow: the situation will cause irreparable business harm if not addressed in the next 3-6 months.
- Code purple: the situation will cause business harm if not addressed in the next year.
- Code green: things are not at risk of causing problems, but we still want to make sure we make progress.
At Google, all of these priority codes need senior VP signoff, which is to say that it is actually an existential threat to one of our main product areas (e.g. Search).
I only remember the RAM crunch (2020 or maybe 2021) being a code yellow, but it's possible it was downgraded after the first month or two.
Code reds don't always have to be met with a total code freeze, but they generally do preempt all work outside of incident response.
The point of a code yellow should not be to punish the team, and an appropriately-declared code yellow should be met with significant introspection from leadership about how we got into this mess and what we need to do to prevent us from getting into this mess in the future. It's a blunt tool that allows the organization to dictate that it's going to drop its existing commitments on the floor because they are simply less important than fixing the systemic problem.
You don't need a code yellow to try multiple things in parallel, or to ship a prototype without worrying about scale. A startup certainly doesn't need a code yellow to empower individuals to wear multiple hats. And if your team is spending 50-75% of its time on keep-the-lights-on work, then your systems are being held together by duct tape, and this is simply not sustainable.
- Code red: the situation is actively causing active business harm.
- Code yellow: the situation will cause irreparable business harm if not addressed in the next 3-6 months.
- Code purple: the situation will cause business harm if not addressed in the next year.
- Code green: things are not at risk of causing problems, but we still want to make sure we make progress.
At Google, all of these priority codes need senior VP signoff, which is to say that it is actually an existential threat to one of our main product areas (e.g. Search).
I only remember the RAM crunch (2020 or maybe 2021) being a code yellow, but it's possible it was downgraded after the first month or two.
Code reds don't always have to be met with a total code freeze, but they generally do preempt all work outside of incident response.
The point of a code yellow should not be to punish the team, and an appropriately-declared code yellow should be met with significant introspection from leadership about how we got into this mess and what we need to do to prevent us from getting into this mess in the future. It's a blunt tool that allows the organization to dictate that it's going to drop its existing commitments on the floor because they are simply less important than fixing the systemic problem.
You don't need a code yellow to try multiple things in parallel, or to ship a prototype without worrying about scale. A startup certainly doesn't need a code yellow to empower individuals to wear multiple hats. And if your team is spending 50-75% of its time on keep-the-lights-on work, then your systems are being held together by duct tape, and this is simply not sustainable.
Hustle couture
> The etymology is not green/yellow/red. It's just not-Yellow or yes-Yellow. See Stephen Levy's In The Plex (2011) pg186:
Yes, I know. I was making a rhetorical point using a metaphor.
Yes, I know. I was making a rhetorical point using a metaphor.
There's definitely a "delusions of grandeur" thing that happens with some startups. "Come join our incredible journey to change the world by micro-optimizing ad placement on image boards."
ponow(2)
This feels like a blunt instrument to solve management failures.
> 2019: Building our fourth senior executive team in six years to lead us to the next plateau of scale.
Sounds like nobody was sticking around long enough to even find out.
Sounds like nobody was sticking around long enough to even find out.
Any time military language gets used like seen in this blog (eg. “War room”), it’s usually alongside management failure to be addressed by crunch of some sort.
> Any time military language gets used
And some military terms are never used. Reserve: the idle resource you have that you can deploy to stop a breach or to profit from an opportunity.
Nope, we're using Human Resources like people used to use machines in the 80s: you have to get them working 100% of the time. And when there's an emergency? Code Yellow. When there's an unforeseen opportunity opening up? Well, too bad so sad your competitor got it.
And some military terms are never used. Reserve: the idle resource you have that you can deploy to stop a breach or to profit from an opportunity.
Nope, we're using Human Resources like people used to use machines in the 80s: you have to get them working 100% of the time. And when there's an emergency? Code Yellow. When there's an unforeseen opportunity opening up? Well, too bad so sad your competitor got it.
That seems about right. That's actually helpful. When you have a management failure, trying to solving it by "getting better at management" isn't really a solution.
Is this an article about the virtues of mandatory overtime?
It's from a manager / founder so yes, implicitly it is
I’ve experienced a Code Yellow myself and can definitely see its value. In large companies, you sometimes need that top-to-bottom rallying cry to overcome the bystander effect when big problems emerge.
After ChatGPT’s initial release, Google famously sounded the alarm and brought Larry Page back into the mix, showing how this kind of all-hands effort can quickly organize everyone around a single goal.
After ChatGPT’s initial release, Google famously sounded the alarm and brought Larry Page back into the mix, showing how this kind of all-hands effort can quickly organize everyone around a single goal.
Brought Larry Page back to do what? Google is worse than ever, Gemini is just one more piece of clutter ruining what used to be a useful tool.
"2019: Building our fourth senior executive team in six years to lead us to the next plateau of scale"
Valuable lessons from a competent person. Great article!
Valuable lessons from a competent person. Great article!
Yeah this seems insane to me, I don't understand how 4 executive teams in 6 years is a sign of anything but chaos and dysfunction
You exit from the president of Instacart and then spend Saturdays at your son’s tae kwon do practice coercing subordinates to give up their life?
This guy is bad at his job.
Well, he may be bad at his job, but at least it also sounds like he's bad at being a father.
I don't understand which part of this requires the team to "sacrifice the ‘L’ and ‘B’ from Work-Life-Balance". If you want to encourage your team to re-evaluate their decisions and tackle big problems maybe you should build a culture around that rather than forcing everyone to be in emergency mode?
I think that's what this post is saying, but all the posturing about forcing people to working harder to build "grit" is making it hard to read it as genuine. Clearly this is not a management issue, it's the team's fault for not having enough grit to work hard all the time!
I think that's what this post is saying, but all the posturing about forcing people to working harder to build "grit" is making it hard to read it as genuine. Clearly this is not a management issue, it's the team's fault for not having enough grit to work hard all the time!
24/7 oncall to ... be yanked onto something the boss fancies. No thanks. What about... plannning?
>Under the surface constraints, the ones which are imposed almost unconsciously, are the most pernicious.
This is true and important. For example, no matter how critical a project may be, it is understood that the first duty of any white collar worker is to show up to every meeting on their Google Calendar. When we are are allocating, prioritizing, and estimating resources for projects, what we are really talking about is the time left over after discussing ~anything that ~anyone may want to pull them into. Only an executive whose calendar is kept by an EA subjects meetings to the kind of cost-benefit tests and prioritization that we use for IC project bandwidth.
This is true and important. For example, no matter how critical a project may be, it is understood that the first duty of any white collar worker is to show up to every meeting on their Google Calendar. When we are are allocating, prioritizing, and estimating resources for projects, what we are really talking about is the time left over after discussing ~anything that ~anyone may want to pull them into. Only an executive whose calendar is kept by an EA subjects meetings to the kind of cost-benefit tests and prioritization that we use for IC project bandwidth.
The apostrophe placement was almost as bad as the content. No thanks.
"During a code-yellow, a leader can escalate a project/situation to a war room situation, pulling people out of their day-to-day work to focus entirely on the problem at hand. "
In one of those books management gurus right at the end of their workspan, one of them wrote that he did not believe a project was worth doing if it was not worth having a war room in which to focus people on the problem at hand.
Lowering Work In Progress is one of the easy ways to gain success...
In one of those books management gurus right at the end of their workspan, one of them wrote that he did not believe a project was worth doing if it was not worth having a war room in which to focus people on the problem at hand.
Lowering Work In Progress is one of the easy ways to gain success...
"we will build a culture that makes Sweating The Problem our default."
What a douche. So it's always code-yellow, and now they'll need to invent a code-red.
What a douche. So it's always code-yellow, and now they'll need to invent a code-red.
If you are really sweating the problem, you are sweating only one problem.
But an organization usually has more than one problem that needs to be addressed concurrently, unless it wants to constantly be "fire fighting" problems that could have received partial attention earlier but instead have become emergencies.
So this guy is going to want to sweat more than one problem, which really just means doing a normal job of prioritizing things, but with more sweat. Just remove the L & B from WLB, permanently. Sort of like this boss imagines he does by writing emails at his family's events, but without the extreme upside potential afforded to executives.
But an organization usually has more than one problem that needs to be addressed concurrently, unless it wants to constantly be "fire fighting" problems that could have received partial attention earlier but instead have become emergencies.
So this guy is going to want to sweat more than one problem, which really just means doing a normal job of prioritizing things, but with more sweat. Just remove the L & B from WLB, permanently. Sort of like this boss imagines he does by writing emails at his family's events, but without the extreme upside potential afforded to executives.
“ found myself sitting at my kids Tae Kwon Do competition (which was running 3.5 hours late!) on a Saturday writing an email to the team ”
Maybe if you work hard enough you won’t need to stay at such a bad job
Maybe if you work hard enough you won’t need to stay at such a bad job
just like layoffs, this just part of the tech culture, as long as we reward managers with promotions for riding their teams (and no doubt I'm sure those middle managers are working their ass of too) this culture doesn't change.
in many cases deep financial bets have already been made before the code is even written or the idea properly sounded out. culture is culture..
human desire to appease the social order is stronger than rational decision making
in many cases deep financial bets have already been made before the code is even written or the idea properly sounded out. culture is culture..
human desire to appease the social order is stronger than rational decision making
Corporate mind games exposed!
Is this satire?
I've met people like this and they seem to always have a disaster on their hands.
Probably because their companies spend so much time deciding which task should be prioritized and having meetings about it that nobody can get actual work done.
Probably because their companies spend so much time deciding which task should be prioritized and having meetings about it that nobody can get actual work done.
Wow what a toxic attitude. The worst part is acting like this constant crunch time is a normal and reasonable expectation. I'll also call out the derision towards "keeping the lights on" because, of course, keeping the damn lights on is a prerequisite to any kind of growth. Code Yellow doesn't tell people "hey it's not important that the site stay up, just focus on growth instead". It tells people "what you are doing 9-5 to keep the site up is not enough, spend your 5-9 working on this other initiative as well". Evil.
It doesn't feel like it means the same thing to OP as it does to the rest of us:
> The biggest constraint I have seen teams imposing unconsciously is the ‘keeping the lights on’ fallacy. This manifests as the project team giving the project/goal 25-50% of their attention (although saying it is their main priority) because they feel that they need to continue ‘keeping the lights on’ with other work activities or projects.
It sounds like "keeping the lights on" means ... working on other things as well? It doesn't specify that it's to make sure those other things keep working. Maybe it's implied? I have no idea.
> The biggest constraint I have seen teams imposing unconsciously is the ‘keeping the lights on’ fallacy. This manifests as the project team giving the project/goal 25-50% of their attention (although saying it is their main priority) because they feel that they need to continue ‘keeping the lights on’ with other work activities or projects.
It sounds like "keeping the lights on" means ... working on other things as well? It doesn't specify that it's to make sure those other things keep working. Maybe it's implied? I have no idea.
I take keeping the lights on to mean, e.g., customer support and fixing bugs an incidents. Not necessarily uptime alone.
It's the same with prioritization. I've literally had conversations that go:
Manager: I need you to drop everything and do X right now, it's top priority
Me: Ok, well I'm currently doing Y, which was top priority this morning. Which is more important, X or Y?
Manager: Well, they're both equally important!
Me: OK, sure. I can't work on both, which one would you like me to do first?
Manager: [uncomfortable thinking noises]
Manager: Are you sure you can't do both at once?
Me: Yes
Manager: [Pause] Keep going with Y. I'll see if someone else can do X
sigh