Operating well – what I learned at Stripe(samgerstenzang.substack.com)
samgerstenzang.substack.com
Operating well – what I learned at Stripe
https://samgerstenzang.substack.com/p/operating-well-what-i-learned-at
104 comments
"An Elegant Puzzle: Systems of Engineering Management" by Will Larson also draws from the author's experience at Stripe (among others, and is published by Stripe itself [1]). I guess you can choose to see it as "there are lots of problems at Stripe as shown by the numerous published solutions".. but the transparency is worth something imo.
[1] https://press.stripe.com/
[1] https://press.stripe.com/
The book draws from the author's experience but it doesn't necessitate that the problems at Stripe were actually solved by what's in the book. The author also draws on experience at Uber and Digg. The author also left Stripe some time after publishing that book.
I always take engineering management advice in books as a "do what I say not what I do" sort of thing. It's rarely the case that companies actually operate as well as the stories published about how well they operate.
I always take engineering management advice in books as a "do what I say not what I do" sort of thing. It's rarely the case that companies actually operate as well as the stories published about how well they operate.
> I always take engineering management advice in books as a "do what I say not what I do" sort of thing.
That's pretty true for Will. Too much self-marketing. It doesn’t help his decision to leave Stripe for a failing startup that just laid off 20% of its staff.
That's pretty true for Will. Too much self-marketing. It doesn’t help his decision to leave Stripe for a failing startup that just laid off 20% of its staff.
>> "An Elegant Puzzle: Systems of Engineering Management" by Will Larson
- Direct link to book mentioned above:
https://press.stripe.com/an-elegant-puzzle
- And collection of reviews/interviews of the book on author’s website:
https://lethain.com/elegant-puzzle-press/
- Direct link to book mentioned above:
https://press.stripe.com/an-elegant-puzzle
- And collection of reviews/interviews of the book on author’s website:
https://lethain.com/elegant-puzzle-press/
Did Will ever run SRE teams tho? This book applies to engineering teams building software iirc
You're using the engineering definition of operations. TFA is using it in the traditional, business sense of the word. Organizational structure, decision making processes, goal setting, workflow optimization, etc. Think: all the things a COO would be responsible for.
I think the title is a bit misleading. The content seems less about operations and more about execution.
[deleted]
[deleted]
> Turn up the heat in every interaction and ask uncomfortable questions. Some of the questions I repeatedly ask:
> Can we re-frame this in terms of the customer’s problem?
> What’s the soonest we could get this done?
> What would you need to get this done tomorrow instead of next week?
Well, that sounds like a recipe for high turnover. I know few engineers who would want to work for this kind of low trust boss and they are all bottom of the barrel consultant types. I thought Stripe was doing better than that.
> Can we re-frame this in terms of the customer’s problem?
> What’s the soonest we could get this done?
> What would you need to get this done tomorrow instead of next week?
Well, that sounds like a recipe for high turnover. I know few engineers who would want to work for this kind of low trust boss and they are all bottom of the barrel consultant types. I thought Stripe was doing better than that.
As an engineer, I ask these questions all the time. Clueless product marketing people will request the most ridiculous features. I ask them: don't tell me the solution, tell me the customer problem. And what's it worth to you to solve this with the pieces we have now, instead of building something months from now? My teams have deflected years of pointless effort this way.
It comes down to whether or not the person you're asking has the power to change things. If not then it's a fairly soul crushing question. Exponentially more when it comes from the same people that have not granted you the power to change things.
Like we were handed a solution to implement that architects and product folks had been talking about for most of this year. But they didn't include anyone that was day in day out hands on with the systems they wanted to change. To do that and then ask the people implementing if we can speed things up would be quite annoying.
It's asking them to contradict the design decisions that people 1-3 levels above them made. Risky business for sure. Even moreso for technical folks that in general aren't as adept at sussing out the difference between someone really asking and someone looking for a rubber stamp on a decision already made.
Like we were handed a solution to implement that architects and product folks had been talking about for most of this year. But they didn't include anyone that was day in day out hands on with the systems they wanted to change. To do that and then ask the people implementing if we can speed things up would be quite annoying.
It's asking them to contradict the design decisions that people 1-3 levels above them made. Risky business for sure. Even moreso for technical folks that in general aren't as adept at sussing out the difference between someone really asking and someone looking for a rubber stamp on a decision already made.
The context and frequency of these types of questions is crucial. As an engineer myself, I would love to see more of these conversations as long as they’re handled the right way.
Context is so important. If the expected response to "what would it take to get this done twice/10x as quickly" is "We shall work twice/10x hours or add equivalent headcount", then you have a serious context problem.
The point of these questions should be to encourage really creative thinking to break out of "normal" operating modes (take on tech debt, etc) and if that's not well understood, or perceived to be an instruction to crunch, then you haven't set the context appropriately and will burn people out.
The point of these questions should be to encourage really creative thinking to break out of "normal" operating modes (take on tech debt, etc) and if that's not well understood, or perceived to be an instruction to crunch, then you haven't set the context appropriately and will burn people out.
[deleted]
[deleted]
Think of these questions less as interrogative orders and more as discussion points / negotiations. I agree that the latter two need to be deployed wisely, sensitive to context and frequency.
To me many of these seem fairly focused on short term gains at potentially long term detriment. It seems Sam Gerstenzang has never stayed at a single company for much more than 2 years which is about the time frame when long term costs start to overcome short term gains in my experience. So not sure if this is the best advice unless you aim to bounce at the 2 year mark.
In my experience moving around every 2 years or so far outpaces staying longer in a financial sense, so if they want to prioritize people to stay longer than 2 years, they need to do so financially. If I can jump somewhere else and make 15-25% more immediately, why would I stay? It’s not long term vs short term its outright compensation for people staying makes staying not worth it.
The article isn't about compensation and neither is my comment.
The running theme underlying this post is having a clear, purposed vision for the company and tying everything that happens to that vision.
Providing a vision is the most important task for the management and leadership of a company. It permeates their conversations with other leaders who continue to spread it down.
The open questions mentioned in the article shouldn't be taken literally.
They strip away all the redundant and nice to have features leaving the core of what is truly necessary to build. That becomes the team's focus.
The questions are a thought experiment exercise that help inform and empower teams to say no to and ignore unnecessary work in the short term in pursuit of the vision.
Other parts of the article are about culture:
And what is culture but the manifestation of vision and will.
Leaders and managers who incorporate these principles are the ones I want to work with. They are passionate, discerning, ambitious, and empathetic. They are rare but lead to a culture and work that feels gratifying.
Providing a vision is the most important task for the management and leadership of a company. It permeates their conversations with other leaders who continue to spread it down.
The open questions mentioned in the article shouldn't be taken literally.
> What’s the soonest we could get this done?
> What would you need to get this done tomorrow instead of next week?
> What would we need to do to get twice as many customers? Ten times as many customers?
They strip away all the redundant and nice to have features leaving the core of what is truly necessary to build. That becomes the team's focus.
The questions are a thought experiment exercise that help inform and empower teams to say no to and ignore unnecessary work in the short term in pursuit of the vision.
Other parts of the article are about culture:
> Elevate each cross-functional partner you work with
And what is culture but the manifestation of vision and will.
Leaders and managers who incorporate these principles are the ones I want to work with. They are passionate, discerning, ambitious, and empathetic. They are rare but lead to a culture and work that feels gratifying.
Very surprised by the negative responses. For a high trust team, nothing he’s saying seems “toxic”.
It sounds like this guy is very good at running projects. I would love to work with him.
It sounds like this guy is very good at running projects. I would love to work with him.
<Rant>
Is there any place in the world for a developer passionate about creating value for users and less so for fighting leadership-made fires (velocity! faster! quicker!)? I don't care to debate endlessly about velocity and processes. I just want decently thought of processes, to adapt to them and produce value.
As an IC I don't want to be involved in every step of the strategy, to keep a track of all my colleagues and how they work, to keep track of the tasks in the context of the epics and the milestones, and be blamed for when a strategy I was not part of doesn't work. I don't want to answer "how could you do this faster?" for myself because it's a horrible question to have to answer almost every week when you feel like you're burning out running. I don't want to answer "what is the priority of the task you are working on relative to our investor's desires?" without actually knowing who the investors are. I don't want to have feel impostor syndrome every start of the week and be in a team that gets ground down at the end of every week for the shit job we're all allegedly doing and where solutions are expected to come from us.
I just wanna get to work, have a clear roadmap that was developed by CxOs, leadership, consultants, investors, people with actual domain knowledge, deeply thinking about what needs to be done and in what order (in order to produce value for our customers). Then I want to create a technical road-map with my fellow engineers and engineering leaders that takes into account business requirements, technical know-how inside the team, planning and start working on that, in sprints, under the watchful supervision of ... supervisors that don't need me to answer questions that they should have had the answers to about a half a year ago.
I'm feeling my mind closing down to this kind of stuff. I don't know how much longer I can do this. I am now in a position where on Monday I get a pile of unthought of requirements, by Wednesday everything changes. Thursday evening a fire starts burning and if fix is not done by Friday morning, we are already behind and disappointing our bosses and investors. Doing the fix and dropping the current task, for urgency, will of course get the velocity / scope discussion. And the seemingly simple fix for the entire team is apparently that we should just increase velocity, but we're just too retarded to do that.
A number of years ago, I left the corporate world, as horrible as that was, there were some people that had answers. Since joining the startup world, perhaps due to bad luck, I've never had a proper answer. I always get a "I'll get back to you on this" and they never get back. If I push for it, it usually gets deprioritized even though it was considered top priority. If I push even more I have to start investigating it. Talk to stakeholders that are not aligned on answers, and it never seems to be about users. It's always growth and velocity.
</Rant>
Is there any place in the world for a developer passionate about creating value for users and less so for fighting leadership-made fires (velocity! faster! quicker!)? I don't care to debate endlessly about velocity and processes. I just want decently thought of processes, to adapt to them and produce value.
As an IC I don't want to be involved in every step of the strategy, to keep a track of all my colleagues and how they work, to keep track of the tasks in the context of the epics and the milestones, and be blamed for when a strategy I was not part of doesn't work. I don't want to answer "how could you do this faster?" for myself because it's a horrible question to have to answer almost every week when you feel like you're burning out running. I don't want to answer "what is the priority of the task you are working on relative to our investor's desires?" without actually knowing who the investors are. I don't want to have feel impostor syndrome every start of the week and be in a team that gets ground down at the end of every week for the shit job we're all allegedly doing and where solutions are expected to come from us.
I just wanna get to work, have a clear roadmap that was developed by CxOs, leadership, consultants, investors, people with actual domain knowledge, deeply thinking about what needs to be done and in what order (in order to produce value for our customers). Then I want to create a technical road-map with my fellow engineers and engineering leaders that takes into account business requirements, technical know-how inside the team, planning and start working on that, in sprints, under the watchful supervision of ... supervisors that don't need me to answer questions that they should have had the answers to about a half a year ago.
I'm feeling my mind closing down to this kind of stuff. I don't know how much longer I can do this. I am now in a position where on Monday I get a pile of unthought of requirements, by Wednesday everything changes. Thursday evening a fire starts burning and if fix is not done by Friday morning, we are already behind and disappointing our bosses and investors. Doing the fix and dropping the current task, for urgency, will of course get the velocity / scope discussion. And the seemingly simple fix for the entire team is apparently that we should just increase velocity, but we're just too retarded to do that.
A number of years ago, I left the corporate world, as horrible as that was, there were some people that had answers. Since joining the startup world, perhaps due to bad luck, I've never had a proper answer. I always get a "I'll get back to you on this" and they never get back. If I push for it, it usually gets deprioritized even though it was considered top priority. If I push even more I have to start investigating it. Talk to stakeholders that are not aligned on answers, and it never seems to be about users. It's always growth and velocity.
</Rant>
I once had a manager that pushed and pulled everything and everyone in all directions at once without a plan. Everytime he said he has 1000s of problems to solve. Our last meeting, before the one with HR, ended when I asked him to spontaniously name the top 10, no screw ten, just name three problems you want your ops team tackle right now. Unsurprisingly, he failed to answer. I had my meeting with HR the same afternoon, he was out a mere 4 months later.
I 'love' velocity in the wrong direction, so easy to be fooled that progress is being made until you end up with a mess on your hands.
Oh yeah, that is so true... unfortunately because I see something similar at the moment again. I have yet to end up with HR over it so, maybe I'm getting wiser, who knows.
Just heard a comment from Albert Speer on Hitler, saying, paraphrased, that Hitler's ignorancebof basically everything was his greatest strength. And that it turned into a disaster once this ignorant approach failed, the rules of whatever game was played started to matter and things began to fall appart. Ignoring the Nazi angle, this principle is universal it seems. Success in one field used to assume success in unrelated fields, unsuitable first princule thinking, all that works in start-ups as long as VC money can be thrown around. When that money dries up things turn south.
Just heard a comment from Albert Speer on Hitler, saying, paraphrased, that Hitler's ignorancebof basically everything was his greatest strength. And that it turned into a disaster once this ignorant approach failed, the rules of whatever game was played started to matter and things began to fall appart. Ignoring the Nazi angle, this principle is universal it seems. Success in one field used to assume success in unrelated fields, unsuitable first princule thinking, all that works in start-ups as long as VC money can be thrown around. When that money dries up things turn south.
I've been wondering for some time if it would be possible and effective to delegate all administrative and process responsibilities to line managers. Remove the IC from the planning documentation process entirely and let them plan their own execution however they see fit within each Sprint.
The only responsibility for the IC would be stand ups with their manager to discuss progress and agree to future timelines. The ICs wouldn't have to go to any other Agile meetings or write any business/project planning docs. Manager would scope and write all IC tickets.
I think you would very quickly see a culture change with regard to process work. Having sole responsibility for both creating and completing the process work, it would be in the manager's best interests to make it efficient.
I've never seen an org try the above, but it doesn't seem crazy to me. I expect you'd still be able to attract managers through increased compensation and opportunities for career progression.
Note: I'm not saying the IC forfeits all planning discretion, I simply mean it would be the manager's responsibility to write it up into whatever documentation is required by the org, track progress, rewrite the plan as necessary, etc.
The only responsibility for the IC would be stand ups with their manager to discuss progress and agree to future timelines. The ICs wouldn't have to go to any other Agile meetings or write any business/project planning docs. Manager would scope and write all IC tickets.
I think you would very quickly see a culture change with regard to process work. Having sole responsibility for both creating and completing the process work, it would be in the manager's best interests to make it efficient.
I've never seen an org try the above, but it doesn't seem crazy to me. I expect you'd still be able to attract managers through increased compensation and opportunities for career progression.
Note: I'm not saying the IC forfeits all planning discretion, I simply mean it would be the manager's responsibility to write it up into whatever documentation is required by the org, track progress, rewrite the plan as necessary, etc.
> The only responsibility for the IC would be stand ups with their manager to discuss progress and agree to future timelines. The ICs wouldn't have to go to any other Agile meetings or write any business/project planning docs. Manager would scope and write all IC tickets.
I think this would end up causing negative feelings both for developers and managers. I've almost seen this in one place where we were Waterfall and it created a disconnect between the dev's understanding of the product, resulting in frustration and bad decisions (the small ones that you make during the day where it's important to have context of the product).
The other interesting thing I noticed was that managers had a lot less excuses for the higher ups and would do a lot less office politics and a lot more execution, but that ended up also not working amazingly because we've had instances of people playing only politics and winning vs. execution (surprisingly fast to get promoted when instead of defining tasks and taking care of execution scheduling, you grab a coffee with your manager's manager) and that ended up frustrating managers also.
At this point in my life I've given up on believing in both gajillion dollar corporate projects that mostly satisfy egos of stakeholders and startups that are not customer focused. I've also stopped believing in co-workers that are not user focused. I don't care both for how many mountains a manager can move, neither for the sacred 10x developer, if their talents are not put to use in the customer's name.
And, to your post's subject, I've also stopped believing in any sort of process, that is not done to deliver more value to the customer. IMHO, your vision (or any other vision) could work, or not, depending on the people involved in the process.
I think this would end up causing negative feelings both for developers and managers. I've almost seen this in one place where we were Waterfall and it created a disconnect between the dev's understanding of the product, resulting in frustration and bad decisions (the small ones that you make during the day where it's important to have context of the product).
The other interesting thing I noticed was that managers had a lot less excuses for the higher ups and would do a lot less office politics and a lot more execution, but that ended up also not working amazingly because we've had instances of people playing only politics and winning vs. execution (surprisingly fast to get promoted when instead of defining tasks and taking care of execution scheduling, you grab a coffee with your manager's manager) and that ended up frustrating managers also.
At this point in my life I've given up on believing in both gajillion dollar corporate projects that mostly satisfy egos of stakeholders and startups that are not customer focused. I've also stopped believing in co-workers that are not user focused. I don't care both for how many mountains a manager can move, neither for the sacred 10x developer, if their talents are not put to use in the customer's name.
And, to your post's subject, I've also stopped believing in any sort of process, that is not done to deliver more value to the customer. IMHO, your vision (or any other vision) could work, or not, depending on the people involved in the process.
> your vision (or any other vision) could work, or not, depending on the people involved in the process.
I wonder is there a rule or a set of rules that would bring out the best in people, make us focus on creating great products while simultaneously caring about sustainable future.
Profit-centric world seems to breed all kinds of pathology, on every level. Even our growing regulations seem not adequate and usually protecting the powerful.
I wonder is there a rule or a set of rules that would bring out the best in people, make us focus on creating great products while simultaneously caring about sustainable future.
Profit-centric world seems to breed all kinds of pathology, on every level. Even our growing regulations seem not adequate and usually protecting the powerful.
I’ve worked on a team that was close to this. The main downside was that the engineers felt like cogs in the machine that had no agency. It’s like we were offshore contractors. The quality of the software wasn’t great. The team tended to focus on meeting deadlines. Turnover was high.
Eventually the product manager was fired, the engineering manager quit, and the team got reorged into something fairly different.
Eventually the product manager was fired, the engineering manager quit, and the team got reorged into something fairly different.
> If I push even more I have to start investigating it. Talk to stakeholders that are not aligned on answers, and it never seems to be about users. It's always growth and velocity
Why is this inherently bad? Growth and velocity can be beneficial for users.
Why is this inherently bad? Growth and velocity can be beneficial for users.
> Why is this inherently bad? Growth and velocity can be beneficial for users.
It's not inherently bad, but if you have velocity and growth build on shoddy foundations, that's not velocity and growth. That's speculation.
> Growth and velocity can be beneficial for users.
100%, with the above mentioned caveat. However, as a user of a lot of products, I find it hard to remember any time where I wished for more growth for the company and velocity for the teams involved in building the products.
It's not inherently bad, but if you have velocity and growth build on shoddy foundations, that's not velocity and growth. That's speculation.
> Growth and velocity can be beneficial for users.
100%, with the above mentioned caveat. However, as a user of a lot of products, I find it hard to remember any time where I wished for more growth for the company and velocity for the teams involved in building the products.
The secret is that if you’re good enough pretty much any company can be what you want. Most people don’t really know what they’re doing so if you are good and just build stuff that is useful I’ve found that people don’t really get in your way.
If you try to ask for permission and write up docs, etc etc then it’s a pain, but if you just _do_ it resistance becomes a lot less.
If you try to ask for permission and write up docs, etc etc then it’s a pain, but if you just _do_ it resistance becomes a lot less.
[deleted]
After reading through this I am convinced that the author learned nothing novel outside of fundamental operations methods I regularly see deployed at companies of all shapes and sizes.
Yep... Maybe they are new to the last 10 years of startup product management? 'operator' makes it sound like they are coming from the investor world, where all you see is hoodies, flashy demos, PowerPoints will YoY growth numbers, and little of the day-to-day behind them
A lot of people in tech haven't collaborated with experienced product managers either, or owned little of the financial/growth side of launching something, where these kinds of questions get ingrained into you pretty fast. Much more of a dark art imo is how to act on them without giving your team whiplash and crushing stress (beyond their comment of 'keep innovation teams small', which means fewer people will still suffer the same problem)
A lot of people in tech haven't collaborated with experienced product managers either, or owned little of the financial/growth side of launching something, where these kinds of questions get ingrained into you pretty fast. Much more of a dark art imo is how to act on them without giving your team whiplash and crushing stress (beyond their comment of 'keep innovation teams small', which means fewer people will still suffer the same problem)
Operating well means you could do things that doesn't scale/manually at first, but then you understand the process enough to automate it away so you can tackle the next problem and you don't need to throwing people in linearly/exponentially. It applies to both software and organization level.
There is a similar term in DevOps that I forgot. Hopefully someone can chime in.
There is a similar term in DevOps that I forgot. Hopefully someone can chime in.
I think you're talking about eliminating toil.
First time you have to step in and do something, cool, get in there, get it done.
Second time, sounds like that wasn't a one-off, you have toil, and you need a runbook on the company wiki or internal doc site - that way anyone in the team (or organisation or company), can quickly step in and do that thing.
Third time, have somebody else run through the runbook it, validate it, figure out if it can be improved. Perhaps have some better alarms and operational oversight.
Now, start automating parts of the runbook. Some parts will be easy, some will be hard. Do the easy ones first and update the runbook.
Automation will need alarms with the right thresholds. Think about those carefully and remember whenever an alarm fires there are only three things you should be doing: fix the problem; change the threshold; delete the alarm. If you do anything else - especially ignore it and just look at things and wait to see if anything else happens - you have misunderstood and are heading towards alarm fatigue. Don't do this. Ever. Remember the three possible actions.
Once all the parts of the runbook are automated and alarms are working correctly to trigger automatic resolution, delete the runbook.
You have now eliminated toil.
This can apply to any manual operational task that needs to happen anywhere in a business, but is most easily used when improving resiliency and durability in systems that have easy automatic resolution (such as horizontal scaling by adding more instances when loads are high on existing infra). It's exactly what an AWS EC2 ASG is there to do, for example.
First time you have to step in and do something, cool, get in there, get it done.
Second time, sounds like that wasn't a one-off, you have toil, and you need a runbook on the company wiki or internal doc site - that way anyone in the team (or organisation or company), can quickly step in and do that thing.
Third time, have somebody else run through the runbook it, validate it, figure out if it can be improved. Perhaps have some better alarms and operational oversight.
Now, start automating parts of the runbook. Some parts will be easy, some will be hard. Do the easy ones first and update the runbook.
Automation will need alarms with the right thresholds. Think about those carefully and remember whenever an alarm fires there are only three things you should be doing: fix the problem; change the threshold; delete the alarm. If you do anything else - especially ignore it and just look at things and wait to see if anything else happens - you have misunderstood and are heading towards alarm fatigue. Don't do this. Ever. Remember the three possible actions.
Once all the parts of the runbook are automated and alarms are working correctly to trigger automatic resolution, delete the runbook.
You have now eliminated toil.
This can apply to any manual operational task that needs to happen anywhere in a business, but is most easily used when improving resiliency and durability in systems that have easy automatic resolution (such as horizontal scaling by adding more instances when loads are high on existing infra). It's exactly what an AWS EC2 ASG is there to do, for example.
For anyone that is interested, my team at Rootly (https://rootly.com/) built an incident management platform that has been greatly inspired by the folks at Stripe!
> Jump on that late night Zoom, reframe the problem, and go line by line to make it work
I think that most of us appreciate that sometimes it's just necessary to do this, e.g. because Things Are On Fire, but I think that, amongst the numerous other red-flags that other comments have pointed out, this one particularly sticks out to me.
If you need to grab workers to fight fires "late into the night", that's probably an indicator that your "operating" practices are creating fires that shouldn't have been there in the first place, right?
Highlighting this as a "pro-move" for management was probably not their best idea IMHO.
I think that most of us appreciate that sometimes it's just necessary to do this, e.g. because Things Are On Fire, but I think that, amongst the numerous other red-flags that other comments have pointed out, this one particularly sticks out to me.
If you need to grab workers to fight fires "late into the night", that's probably an indicator that your "operating" practices are creating fires that shouldn't have been there in the first place, right?
Highlighting this as a "pro-move" for management was probably not their best idea IMHO.
Easier said than done. No system with real usage can achieve 100% up time. When problems inevitably occur, what do you expect Stripe to do? Wait until 10am the next morning??
The "inevitability" of problems is what is the problem here.
Toxic organisations always find ways to defend their deep issues as "inevitable".
Google SRE book is now decade old. Read it.
Toxic organisations always find ways to defend their deep issues as "inevitable".
Google SRE book is now decade old. Read it.
Even trying to implement solid SRE practices you'll still have inevitable problems, incidents and other disruptions.
I agree what most people consider inevitable isn't, but your position is too extreme if you've ever been in a real company. Not everything can be predicted by an error budget, and in most non-google companies "stop deploying until you got your budget back" isn't going to allow your business to operate fast enough or flexibly enough. Reliability isn't all in business.
I agree what most people consider inevitable isn't, but your position is too extreme if you've ever been in a real company. Not everything can be predicted by an error budget, and in most non-google companies "stop deploying until you got your budget back" isn't going to allow your business to operate fast enough or flexibly enough. Reliability isn't all in business.
That's correct - but this argument is more often than not used to prop up "inevitability" of overtime that neither inevitable, nor necessary.
Yes, bad stuff happens - but having it once per two years or once per few months as the article implies it a different scale and the MOST important part of it is to stop treating it as normal, required and necessary.
Also, Stripe has 14 offices globally. That's already a size where you can have work-time rotation for those "late night" problems. Which then stop being late night problems.
Again. Not inevitable.
Yes, bad stuff happens - but having it once per two years or once per few months as the article implies it a different scale and the MOST important part of it is to stop treating it as normal, required and necessary.
Also, Stripe has 14 offices globally. That's already a size where you can have work-time rotation for those "late night" problems. Which then stop being late night problems.
Again. Not inevitable.
"100% is probably never the right reliability target: not only is it impossible to achieve, it’s typically more reliability than a service’s users want or notice. Match the profile of the service to the risk the business is willing to take."
"Things break; that’s life."
"Things break; that’s life."
Yes, and with 14 global offices the company can fix them in worktime of one of their employees, not late at night.
Let me reiterate: "inevitable" things are rarely inevitable and most commonly just a poor excuse for poor organization and toxic work practices. Even if you put out pithy quotes.
Let me reiterate: "inevitable" things are rarely inevitable and most commonly just a poor excuse for poor organization and toxic work practices. Even if you put out pithy quotes.
In theory, yes. In practice, it's highly unlikely that there will always be someone with the right context to fix the problem in each time zone.
The way my company does this is with a centralized incident management team that follows the sun across 4 timezones, and specific teams will be paged by them depending on what is broken.
The way my company does this is with a centralized incident management team that follows the sun across 4 timezones, and specific teams will be paged by them depending on what is broken.
For anyone that is interested, my team at Rootly (https://rootly.com/) built an incident management platform that has been greatly inspired by the folks at Stripe!
[deleted]
Interesting article, but mostly irrelevant to startups. It’s basically a series of guardrails/habits that big companies need to use to eke out some progress when default mode is to not do anything.
I think a lot of people at Stripe are drinking the kool aid waiting for the IPO to make some money. Hopefully they don't self destruct before that though.
I read the whole article and I though "this is management bullshit and this waynof working is toxic" then I read the comments and I'm glad people think the same
> Turn up the heat in every interaction and ask uncomfortable questions. Some of the questions I repeatedly ask:
> What’s the soonest we could get this done?
> What would you need to get this done tomorrow instead of next week?
Some helpful insight into why their employees are crying and burning out. Has anyone else's impression of the Collisons been tainted by the realization that they seem to have inculcated psychopathy into their corporate culture?
> What’s the soonest we could get this done?
> What would you need to get this done tomorrow instead of next week?
Some helpful insight into why their employees are crying and burning out. Has anyone else's impression of the Collisons been tainted by the realization that they seem to have inculcated psychopathy into their corporate culture?
> Has anyone else's impression of the Collisons been tainted by the realization that they seem to have inculcated psychopathy into their corporate culture?
Nearly every confirmation of this shared here on HN is still self-censored to some degree, but to summarize: yes.
Nearly every confirmation of this shared here on HN is still self-censored to some degree, but to summarize: yes.
[deleted]
They are uncomfortable but without asking questions like these it is hard to become more productive. It is also demanding of the employees, no doubt about it but I think people that want to work at Stripe realize it is a demanding environment. Calling it psychopathy is uncalled for.
Is there something wrong about getting things done faster.
If your mechanism for getting things done faster is to make your employees uncomfortable talking to you, yes.
This is all about context.
Those aren't bad or uncomfortable questions. Having as much clarity as possible helps ensure plans are on track, need adjusting, etc. right? Some folks stress over questions like this, but I think it's important to ask them, and to provide context to the folks on the receiving end.
So, I think you ask these questions with a few basic rules:
(1) Don't send an email/message with a question like that without being clear about how urgent the response is
(2) Send the question or have the conversation during the course of a "normal" work-day - instead of in the middle of someone's night/etc.
(3) Keep the emotions out of the question, hopefully helping to keep them out of the response
(4) Just always think about what might be involved in answering the question. Are you needlessly grinding someone for an answer or do you really need it?
That last one really hits home for me. Was most recently in analytics/product at a large financial services company (insurance) and am now in a start-up. But I see a common thread across the two: senior folks sometimes asking for answers/work without spending much or any time thinking about the time the receiver will spend trying to figure out that answer/work.
Those aren't bad or uncomfortable questions. Having as much clarity as possible helps ensure plans are on track, need adjusting, etc. right? Some folks stress over questions like this, but I think it's important to ask them, and to provide context to the folks on the receiving end.
So, I think you ask these questions with a few basic rules:
(1) Don't send an email/message with a question like that without being clear about how urgent the response is
(2) Send the question or have the conversation during the course of a "normal" work-day - instead of in the middle of someone's night/etc.
(3) Keep the emotions out of the question, hopefully helping to keep them out of the response
(4) Just always think about what might be involved in answering the question. Are you needlessly grinding someone for an answer or do you really need it?
That last one really hits home for me. Was most recently in analytics/product at a large financial services company (insurance) and am now in a start-up. But I see a common thread across the two: senior folks sometimes asking for answers/work without spending much or any time thinking about the time the receiver will spend trying to figure out that answer/work.
It's uncomfortable because it forces you to question your assumptions and what you are doing. It's not a bad thing.
You don't want unconfortable people, you want people being comfortable doing whatvthey are doing. Getting out of your comfort zone every once in a while is good, pushing it too hard is just increasing the risk of errors.
You want stable operations with some slack for the cases something goes wrong, you don't want ops that are constantly under stress.
You want stable operations with some slack for the cases something goes wrong, you don't want ops that are constantly under stress.
It’s an uncomfortable question. That doesn’t mean you’re uncomfortable all the time.
Having slack and not being stressed are important, but they have nothing to do with asking uncomfortable questions.
Having slack and not being stressed are important, but they have nothing to do with asking uncomfortable questions.
To ask uncomfortable questions is, inherently, to make someone uncomfortable talking to you. It’s in the definition. If you ask uncomfortable questions repeatedly, people will avoid talking to you. This is normal human behaviour.
If you make people talk to you despite being uncomfortable - for example, because you are their manager - they will find a job where they are comfortable.
Potentially your business runs on destroying people. I don’t think that’s a reasonable way we want to run society.
The way /normal/ companies handle this is things like agile retrospectives - what went wrong, how can we fix it. Did we have enough people? How did /management/ fail /the people doing the work/?
If I answer “we would need X more people to get this done tomorrow instead of next week”, does that actually change anything, or do you just respond some nonsense about how you can’t afford that and we should just work harder? Basically: are /you/ willing to make sacrifices for /me/, or is it a one way street?
If you make people talk to you despite being uncomfortable - for example, because you are their manager - they will find a job where they are comfortable.
Potentially your business runs on destroying people. I don’t think that’s a reasonable way we want to run society.
The way /normal/ companies handle this is things like agile retrospectives - what went wrong, how can we fix it. Did we have enough people? How did /management/ fail /the people doing the work/?
If I answer “we would need X more people to get this done tomorrow instead of next week”, does that actually change anything, or do you just respond some nonsense about how you can’t afford that and we should just work harder? Basically: are /you/ willing to make sacrifices for /me/, or is it a one way street?
Hard disagree here. Being constantly asked uncomfortable questions by your manager is making you uncomfortable. In short, it is creating a toxic environment of stressed out people. And that ops org is not the most recilient one. And you want recilient ops, even if you don't care about the people.
It implies you think the person doing the task is delivering more slowly than they could/should be. Especially if you’re asking it in every interaction.
I have never seen this framed as performing work too slowly. It's more about cutting scope, gathering context, asking for help, identifying dependencies, etc.
It's often really helpful for someone to ask these questions, especially when they have higher level context you weren't aware of.
It's often really helpful for someone to ask these questions, especially when they have higher level context you weren't aware of.
Speed is relative. What you want is the right speed of getting things done, not just being faster by default. It is as important to slow down in operations as it is to speed up as it is to just run things stable. The true art of good ops management is knowing to do those things, in parallel depending on the subject at hand, and to judge which approach to choose. The article didn't even hint at that.
You're assuming that the individual's labour is the only input into work completion.
I think those questions are good. One benefit is that it can reveal potential organisation-wide blockers:
*Getting design changes could be slow (and hence the design department may be understaffed, or inefficient communication channels)
*Maybe some processes can be ran in parallel. Someone can build the login form while the other does the database.
*They might require git access for another department that could take a while to come through.
Considering engineers' time is often the greatest cost for a technology startup, you can bet they'd be incentivised to maximise the efficiency of it.
Not only might they reveal potential blockers, but answering it also helps both parties understand they're on the same page regarding what the task entails. If the task is building a login form and the response is 'It will take two days to train the neural network', you can nip that in the bud.
I think those questions are good. One benefit is that it can reveal potential organisation-wide blockers:
*Getting design changes could be slow (and hence the design department may be understaffed, or inefficient communication channels)
*Maybe some processes can be ran in parallel. Someone can build the login form while the other does the database.
*They might require git access for another department that could take a while to come through.
Considering engineers' time is often the greatest cost for a technology startup, you can bet they'd be incentivised to maximise the efficiency of it.
Not only might they reveal potential blockers, but answering it also helps both parties understand they're on the same page regarding what the task entails. If the task is building a login form and the response is 'It will take two days to train the neural network', you can nip that in the bud.
Not ordinarily. But it seems that 'faster' might be exacting a dire human toll over there.
I can't imagine Stripe being a particularly good workplace if a manager is continuously trying to give someone a hard time on the basis of assumptions such as a task taking long is because of some inherent deficiency within the employee.
It seems to me that your assumption is that the manager operates in bad faith. If that is true, no matter what they do, it will all be bullshit, so there’s nothing to discuss.
On the other hand, if the manager trusts the employee, and they trust their manager, and they both want to deliver something valuable to their customer, those pieces of advice all look pretty good to me.
On the other hand, if the manager trusts the employee, and they trust their manager, and they both want to deliver something valuable to their customer, those pieces of advice all look pretty good to me.
There are some valuable rod bits in it, they are all very shallow so and turn into management buzzword bingo half way down the article. And the most crucial point was totally missed: being able judge when an ops leader has to push things and when things have to be slowed and calmed down. It seeems the author only saw the first one and now thinks this is what ops is about. It is not.
I jave to agree on the strategy part so, strategy is such a pleasant way to not do any meaningful work.
I jave to agree on the strategy part so, strategy is such a pleasant way to not do any meaningful work.
Having a layer of middle management that has never done the job of the workers 'below' him/her, is bad faith by design.
It is important to discuss, because many youngsters read these forums and don't know the ways of the world yet.
It is important to discuss, because many youngsters read these forums and don't know the ways of the world yet.
Using the word 'operator' is a pretty obvious red flag.
This word isn't common in software because we are all operators, but if you're writing for an audience that includes investors and consultants, using the word operator to denote that you're actually executing (as opposed to investing/consulting) is pretty common.
Not a red flag, just different vocabulary for a different audience.
Not a red flag, just different vocabulary for a different audience.
I enjoyed replacing it with “worker bee” in my head, at least.
I hope you realize these worker bees created an insane amount of value for their customers, their investors, their coworkers and for themselves.
As an investor passively collecting on this value, definitely. Keep up the good work!
I think his use of operator is a result of his VC background where it is a common term. In that context it is used to distinguish b/w investors that come from a finance background vs a startup background. Or to describe people that transition from being an investor in a company/board member to working at the company directly.
Being an operator just means you're executing. If you're a good operator, you're good at executing on ideas/projects/whatever.
Why?
> “does everyone agree this is the right path? Does anyone disagree”
Hey, I’m just asking questions! We’re just discussing! You’re allowed to disagree!
It’s so funny, and hard to describe, the intentional slight malice in such moments. Like it’s absolutely there, and absolutely veiled and intentionally done in such a way to lend plausible deniability.
In a given situation, everyone knows the “correct” answer, it’s already been telegraphed in any number of ways. Sure, you can disagree. We have an open environment here after all. We value discussion.
Thing is, you’re probably not gonna have a fun time if you do. You’re probably gonna end up wishing you had kept your big yap shut. Strangely enough. “Disagree and commit”
Hey, I’m just asking questions! We’re just discussing! You’re allowed to disagree!
It’s so funny, and hard to describe, the intentional slight malice in such moments. Like it’s absolutely there, and absolutely veiled and intentionally done in such a way to lend plausible deniability.
In a given situation, everyone knows the “correct” answer, it’s already been telegraphed in any number of ways. Sure, you can disagree. We have an open environment here after all. We value discussion.
Thing is, you’re probably not gonna have a fun time if you do. You’re probably gonna end up wishing you had kept your big yap shut. Strangely enough. “Disagree and commit”
I had a project that would have gone so very much better if we had simply recorded the minority report, and reviewed them when too many surprises stacked up to see if anyone was more right than wrong when disagreeing.
There’s a very disturbing misunderstanding about the difference between consensus and unanimity, made doubly so on projects that actually embrace distributed computing concepts like consensus and unanimity.
There’s a very disturbing misunderstanding about the difference between consensus and unanimity, made doubly so on projects that actually embrace distributed computing concepts like consensus and unanimity.
Yeah this stuff seems so basic right? If you're trying to be as right as possible:
- decide what success and failure are
- record your successes and failures
- adjust your priors based on your failures
I have a "what have you changed your mind about" interview question I use on both sides of the table that's really illuminating. The specifics aren't important, but what I'm looking for are the above: what were you aiming for, what did you get instead, how did that experience change you. Hopefully the other party has had lots of these and they've built up an immune system against the various things that caused them to fail, stuff like "test hypotheses", "there is no silver bullet", "everything is a tradeoff" and so on.
- decide what success and failure are
- record your successes and failures
- adjust your priors based on your failures
I have a "what have you changed your mind about" interview question I use on both sides of the table that's really illuminating. The specifics aren't important, but what I'm looking for are the above: what were you aiming for, what did you get instead, how did that experience change you. Hopefully the other party has had lots of these and they've built up an immune system against the various things that caused them to fail, stuff like "test hypotheses", "there is no silver bullet", "everything is a tradeoff" and so on.
Yeah, this only works when the culture is based on trust. And when individuals trust each other. Using the question, especially in the way described in the article, is just putting more social pressure on employees. Bad management, bad people skills and in the end bad for ops management.
How would you recommend to get opinions of those who disagree, from your management experience?
As a manager/leader/senior, if you want actual opinions or suggestions, you should let everyone form their own opinion and express it before expressing your own.
Of course, if you then just override everyone every time, they'll soon stop having opinions because they'll realize they don't matter.
Of course, if you then just override everyone every time, they'll soon stop having opinions because they'll realize they don't matter.
What in the world is an Operator?
Someone who gleefully shoulders a mountain load of tech debt at 2AM on a Saturday morning?
It's VC-talk for someone who does stuff at a company.
I'm assuming it relates to operations management but I imagine this type of person not accidentally taking a little shine from the name collision with spec ops soldiers and Matrix movie characters
The BOFH[1] is prior art in this space, pointing to its previous use referring to sysadmins.
[1]: https://en.wikipedia.org/wiki/Bastard_Operator_From_Hell
[1]: https://en.wikipedia.org/wiki/Bastard_Operator_From_Hell
It is leadership speak for one of the four archetypes. Visionary, operator, processor, synergist.
In other words, four words to make people feel good about their role at a company. Middle management is littered with these obscure definitions.
In other words, four words to make people feel good about their role at a company. Middle management is littered with these obscure definitions.
I cannot believe the comments here so far. It’s just sad to see the decline of culture be reflected in Hacker News too. I thought Nietzsche’s “inversion of values” happens every few thousand years, little did I know..
Can you be more specific? The comments are mixed and you don't address what you see as decline.
At the time I wrote the comment, almost all of the other comments were criticizing and mocking common-sense pro-productivity values that were worshipped here 10 years ago.
EDIT: Also gone is the value of charitable reading. This is not specific to HN, but I'm just used to holding HN to higher standards than other fora.
EDIT: Also gone is the value of charitable reading. This is not specific to HN, but I'm just used to holding HN to higher standards than other fora.
[deleted]
I know of a number of instances where folks left Stripe and took an extended break before doing something else. I've heard a few nightmare on call stories (we all have stories but these sounded really severe/frequent). I've heard folks describe their workload as operational heavy with little opportunity to actually develop software or build things. I've heard a few things that make me wary of past and current middle managers there.
It's funny. I imagine it's possible to have a company full of smart, kind, hardworking people but a culture where operational work is celebrated and embraced rather than solved and reduced.
I've had nothing but good encounters with Stripes and ex-Stripes but something seems off, and this post inadvertently reveals some of that.