How learned helplessness happens in engineering teams(okayhq.com)
okayhq.com
How learned helplessness happens in engineering teams
https://www.okayhq.com/blog/status-quo-is-so-hard-to-change-in-engineering-teams
219 comments
The tech lead started out by complaining about...When he finished complaining...
My manager doesn't know anything about the code, my project, or the release project.
IOW, your team has no leadership and no management. Your team lead may be called that, but sounds like he doesn't lead at all. Your manager may be called that, but he doesn't manage either. This happens a lot when people are given responsibility without any authority (probably the case for you "team lead"), or authority over things they don't understand (probably the case for your "manager").
The bad news is that you're part of a disorganized mob. The good news is that there is a leadership vacuum that, if you play your cards right, you could fill. If you want. Of course, even if you succeed, it's possible no one will care about your accomplishments. And you will probably make enemies. It's possible you'll be rewarded but given your description, I view that as unlikely.
As a side note, suggesting to the team lead that he lead and do X, Y or Z may appear to him as if you want him to take on risks with little upside. He's probably thought about doing those things anyway but just decided it's not worth the trouble. It's not all that surprising that he doesn't want to rock the boat.
My manager doesn't know anything about the code, my project, or the release project.
IOW, your team has no leadership and no management. Your team lead may be called that, but sounds like he doesn't lead at all. Your manager may be called that, but he doesn't manage either. This happens a lot when people are given responsibility without any authority (probably the case for you "team lead"), or authority over things they don't understand (probably the case for your "manager").
The bad news is that you're part of a disorganized mob. The good news is that there is a leadership vacuum that, if you play your cards right, you could fill. If you want. Of course, even if you succeed, it's possible no one will care about your accomplishments. And you will probably make enemies. It's possible you'll be rewarded but given your description, I view that as unlikely.
As a side note, suggesting to the team lead that he lead and do X, Y or Z may appear to him as if you want him to take on risks with little upside. He's probably thought about doing those things anyway but just decided it's not worth the trouble. It's not all that surprising that he doesn't want to rock the boat.
> The good news is that there is a leadership vacuum that, if you play your cards right, you could fill. If you want.
Perhaps. I thought so once in exactly this situation, and it led to an incredible amount of frustration, and burnout. And others were hurt the same way. That leadership vacuum was very real, but nobody wanted it to be filled by anyone already in the company, it turned out.
Eventually we got a very competent new team lead from outside and he was accepted by everyone immediately. For one of the teams anyway; the other one is still a complete mess. I just make sure I'm on the right one.
Perhaps. I thought so once in exactly this situation, and it led to an incredible amount of frustration, and burnout. And others were hurt the same way. That leadership vacuum was very real, but nobody wanted it to be filled by anyone already in the company, it turned out.
Eventually we got a very competent new team lead from outside and he was accepted by everyone immediately. For one of the teams anyway; the other one is still a complete mess. I just make sure I'm on the right one.
This resonated strongly with me.
If you don't want to become a leader, what do you do in a disorganized mob situation? Leave?
If you don't want to become a leader, what do you do in a disorganized mob situation? Leave?
not every situation can be fixed
Most often the answer is yes, unfortunately. If you have options make use of them.
If you have to stay, it helps to try and make your team a nice place to be. I remember a manager that would organize team lunches and would give Fridays off (unofficially) making the team an oasis in the middle of a corporate desert. But I can’t imagine it was easy for him to deal with a dysfunctional leadership and coworkers…
If you have to stay, it helps to try and make your team a nice place to be. I remember a manager that would organize team lunches and would give Fridays off (unofficially) making the team an oasis in the middle of a corporate desert. But I can’t imagine it was easy for him to deal with a dysfunctional leadership and coworkers…
Sometimes there is someone else who wants to lead but can't due to political reasons. You can try supporting them. Little things like seconding someone's ideas can go a long way. At least in the place I work at.
Take the stoical approach. Focus on what's in within your locus of control and ignore everything else.
I've come to the same conclusion in recent years. What you can do is optimize your end of work so the dysfunction impacts you less and less. Alot of work is really just BS that nobody wants to touch, but ends up on somebody's desk. Someone just has to bite that apple and document the hell out of it, make playbooks, automate and delegate. You can get management support for putting responsibility where it should be ("shift left"), while you answer questions, direct people to references and generally empower those around you.
So either you become your island and power on from your base, or you become part of the dysfunction as a leader. Leaders who succeed in this are rare enough they write books about it, most of it fiction..
So either you become your island and power on from your base, or you become part of the dysfunction as a leader. Leaders who succeed in this are rare enough they write books about it, most of it fiction..
There is often room for personal initiative in such cases; do what you can, make sure to not try to do more than you can, and be happy with it.
Unfortunately this. I really like how you have articulated the problem. Disorganized mob. This mob also loves moving from one team to another inside the company once they deem to think they exhaust the resources in the former and once the product is dead. I think it happens sooner or later in enterprise. It sucks.
What is the difference between your team lead's bad reasons not to push internally for a better system:
> The other team doesn't know how to do the release.
> He wants to be a "team player" so he can't deny the release.
> Project managers will never allocate time to improve the release process.
and your good reasons not to push internally for a better system?
> My manager doesn't know anything about the code, my project, or the release project.
> He may assume it's complaining for the sake of complaining.
> It has been used as ammunition in reviews against me.
> The other team doesn't know how to do the release.
> He wants to be a "team player" so he can't deny the release.
> Project managers will never allocate time to improve the release process.
and your good reasons not to push internally for a better system?
> My manager doesn't know anything about the code, my project, or the release project.
> He may assume it's complaining for the sake of complaining.
> It has been used as ammunition in reviews against me.
The last bullet of each scenario is fundamentally different, and the ammunition bit is the most important part.
The team lead seems to carry a lot but has a certain "job safety", so no incentive to share the release process.
In fact the team leader is a "hero" who extinguished a fire caused by other department.
In fact the team leader is a "hero" who extinguished a fire caused by other department.
I do push for changes, such as bringing ideas to the team lead and bringing these things up with my manager (hence being brought up in reviews).
There is a self-reinforcing aspect to learned helplessness in teams, as you've pointed out.
There is a self-reinforcing aspect to learned helplessness in teams, as you've pointed out.
In my experience, this sort of thing is why good leadership has an outsized impact.
All you have to do is say no once in a while.
No, don't bring the company to a halt with a tantrum. Just insist on getting one thing fixed a month and nine months from now everyone will be a lot happier.
No, don't bring the company to a halt with a tantrum. Just insist on getting one thing fixed a month and nine months from now everyone will be a lot happier.
> All you have to do is say no once in a while.
Being able to say "no" is absolutely critical.
Of course, you do incur a risk and you may fail or hurt your career in that org up to the point of termination even if you're "right". Moreover your follow-up after saying "no" will be judged harshly and silently for every little snag, real or not.
But that's part of the deal, right? It's not easy, if it were easy you'd have dream-teams all over the place rather than an industry where occupational dysfunction is normal.
Being able to say "no" is absolutely critical.
Of course, you do incur a risk and you may fail or hurt your career in that org up to the point of termination even if you're "right". Moreover your follow-up after saying "no" will be judged harshly and silently for every little snag, real or not.
But that's part of the deal, right? It's not easy, if it were easy you'd have dream-teams all over the place rather than an industry where occupational dysfunction is normal.
> All you have to do is say no once in a while...
> Just insist...
Some companies work like this, but many don't. Such utterances could be charting a course to the door for you.
> Just insist...
Some companies work like this, but many don't. Such utterances could be charting a course to the door for you.
Plenty of openings out there. Not worth suffering to get a 2% raise rather than 1%. If they won't let you fix something once a month it's time to get moving.
Learned helplessness, indeed.
Learned helplessness, indeed.
It isn't learned helplessness if you are actually helpless.
The whole point of the article is that the vast majority of engineering teams have some power. It's not a license to derail the company with technical OCD, but as in all things… "balance, grasshopper."
Yes, there is 1% who can't follow that advice for various reasons of desperation. Not enough to continue arguing this thread.
Yes, there is 1% who can't follow that advice for various reasons of desperation. Not enough to continue arguing this thread.
Can't teach everyone to have a backbone. Heck, nerds are infamous for being easy to manipulate.
Anecdotally, the most manipulative individual I've ever met was a nerd.
Who better to profit off of nerds? It's not like there is any kind of solidarity or union among techies.
I've worked on a good number of teams just like yours. I always told myself things would be different when I was in charge. Then I was in charge.
I learned pretty early on that I have political capital, and very little of it. Instead, I have to have a manager that's pretty buck wild when I tell them I need them to be, and I have to deliver them some wins that will grease their wheels too. Good managers are hard to come by though, and if they get noticed they're promoted and gone in a couple of years.
Personally, I think the transactional nature of software positions is what largely holds us back. If I get promoted, then there'll be a new lead, and this cycle will start all over again.
I learned pretty early on that I have political capital, and very little of it. Instead, I have to have a manager that's pretty buck wild when I tell them I need them to be, and I have to deliver them some wins that will grease their wheels too. Good managers are hard to come by though, and if they get noticed they're promoted and gone in a couple of years.
Personally, I think the transactional nature of software positions is what largely holds us back. If I get promoted, then there'll be a new lead, and this cycle will start all over again.
Did you make your situation clear to your team? I find that is the biggest source of frustration.
Yes. Most of my leadership skills come from the Marines. I'm a fairly blunt leader, so if we're facing something the team is aware of that I can't do anything about then I'm pretty frank about it. Some folks don't like that too much, but I'd rather not have those kind of secrets on a team.
Doing a good job - and staying where you are - doesn't get you more reward. And people generally want more than they currently have.
This hit so close to home for me. At my last two jobs, I've simply burnt out from hitting this sort of wall over and over.
I think the reality is that this kind of change can't originate from the in-the-trenches folks at the bottom of totem pole. Management will never let individuals waver from their short term goals.
The only times I've seen developer pain-points successfully addressed in a sustainable way, it's because there was a dedicated team allocated to that sort of thing: a "developer experience" team, or some equivalent.
(Although, the issues you ran into are kind of an org issue in addition to a developer experience issue)
Everything I suggested was just flatly
denied as impossible
[...]
I feel strange because I've seen this same thing
for my whole career and I still try fight for
what's right when others appear to moan and carry on.
I think the reality is that this kind of change can't originate from the in-the-trenches folks at the bottom of totem pole. Management will never let individuals waver from their short term goals.
The only times I've seen developer pain-points successfully addressed in a sustainable way, it's because there was a dedicated team allocated to that sort of thing: a "developer experience" team, or some equivalent.
(Although, the issues you ran into are kind of an org issue in addition to a developer experience issue)
>The only times I've seen developer pain-points successfully addressed in a sustainable way, it's because there was a dedicated team allocated to that sort of thing: a "developer experience" team, or some equivalent.
I came to this conclusion after watching the place I work at for 2 and a half years fail to implement any of the grand ambitions management had in their heads. We wanted code review, pull requests, build pipelines, automated releases, updated OS's, logging, disaster recovery, you name it. No one had the experience, time, or energy to implement it, so it falls on one or two juniors (me) to quickly learn and implement while learning. No one is hired with experience in any of these topics, only fresh-grads that will replicate the garbage you're trying to get away from are hired, because they will on paper be producing more features for less financial cost.
I came to this conclusion after watching the place I work at for 2 and a half years fail to implement any of the grand ambitions management had in their heads. We wanted code review, pull requests, build pipelines, automated releases, updated OS's, logging, disaster recovery, you name it. No one had the experience, time, or energy to implement it, so it falls on one or two juniors (me) to quickly learn and implement while learning. No one is hired with experience in any of these topics, only fresh-grads that will replicate the garbage you're trying to get away from are hired, because they will on paper be producing more features for less financial cost.
I think they key is to find people who have put effort into figuring out what good is, and are frustrated and care enough to enact change.
Similarly to your org not having a dedicated DX team being the reason, dedicated teams will happily dissapear into a void of non-delivery as their purist vision, unconnected with reality, will simply never apply to your stack, and thus you'll never use it.
End of the day, different things work and fail. The key is almost always the quality of people, their motivation and how many road blocks you put in front of them.
Similarly to your org not having a dedicated DX team being the reason, dedicated teams will happily dissapear into a void of non-delivery as their purist vision, unconnected with reality, will simply never apply to your stack, and thus you'll never use it.
End of the day, different things work and fail. The key is almost always the quality of people, their motivation and how many road blocks you put in front of them.
dedicated teams will happily dissapear
into a void of non-delivery as their purist vision
1. Rotate team members onto and off of the DX team, so they don't lose touch with reality.
and/or
2. Have the rest of developers vote (or otherwise have direct input on) on the DX team's priorities and products. The other developers are the stakeholders here.
> because they will on paper be producing more features for less financial cost.
So make a better business case. Inform them that after investing Y hours on refactoring X, future development is expected to happen M% faster with N% less bugs. New features can be expected to be developed, tested, and shipped to production in S% less time.
But be realistic. Don't make promises that you won't actually meet when the time comes.
You're making this sound simpler than it is, I think. For the following discussion assume we're talking about management that is smart and wants the dev team(s) to succeed but is racing hard to hit various external deadlines and does not have a technical background.
(I chose 10% because we had 30 developers at the time and generally had teams of 3 developers. So, one dedicated DX team...)
Do you have any articles, case studies, etc of how to express this to management that is generally oblivious to the day-to-day, in-the-trenches, experiences and pain points of developers?
What you're saying sounds great to me, who is a developer, but how would you come up with halfway realistic numbers for Y and X?
The best thing I could think to do is have the team minutely log all lost minutes productivity caused by delays and inefficiencies that the proposed fixes could address.
But even that's tricky. For example, at one of my previous positions, the build pipeline could take hours and could randomly fail.
The only way to maintain any semblance of productivity would be to juggle 3-4 tickets at once. This was incredibly challenging, and productivity took a massive hit due to the frustration and context switching.
However, even if I had logged those experiences down to the minute, there weren't any literal downtime minutes. It was more a matter of juggling five tickets at once that should have taken one hour each, but instead they took five hours each because I was context switching faster than the Linux kernel itself.
Scrum and its much-maligned "story points" are a possible solution here, as far as management is concerned, assuming both developers and management have bought into that concept and are using it correctly.
Inform them that after investing Y hours on
refactoring X, future development is expected
to happen M% faster with N% less bugs.
(I chose 10% because we had 30 developers at the time and generally had teams of 3 developers. So, one dedicated DX team...)
Do you have any articles, case studies, etc of how to express this to management that is generally oblivious to the day-to-day, in-the-trenches, experiences and pain points of developers?
What you're saying sounds great to me, who is a developer, but how would you come up with halfway realistic numbers for Y and X?
The best thing I could think to do is have the team minutely log all lost minutes productivity caused by delays and inefficiencies that the proposed fixes could address.
But even that's tricky. For example, at one of my previous positions, the build pipeline could take hours and could randomly fail.
The only way to maintain any semblance of productivity would be to juggle 3-4 tickets at once. This was incredibly challenging, and productivity took a massive hit due to the frustration and context switching.
However, even if I had logged those experiences down to the minute, there weren't any literal downtime minutes. It was more a matter of juggling five tickets at once that should have taken one hour each, but instead they took five hours each because I was context switching faster than the Linux kernel itself.
New features can be expected to be developed,
tested, and shipped to production in S% less time.
Scrum and its much-maligned "story points" are a possible solution here, as far as management is concerned, assuming both developers and management have bought into that concept and are using it correctly.
> Do you have any articles, case studies, etc of how to express this to management that is
> generally oblivious to the day-to-day, in-the-trenches, experiences and pain points of developers?
> What you're saying sounds great to me, who is a developer, but how would you come up with halfway realistic numbers for Y and X?
> The best thing I could think to do is have the team minutely log all lost minutes productivity
> caused by delays and inefficiencies that the proposed fixes could address.
>I think the reality is that this kind of change can't originate from the in-the-trenches folks at the bottom of totem pole.
I'm a manager. Please. PLEASE don't believe this. In-the-trenches folks ARE powerful. Really. I promise. I can't implement change if YOU don't do it. I don't know what sucks if YOU don't tell me.
I NEED you to help me make things better.
I'm a manager. Please. PLEASE don't believe this. In-the-trenches folks ARE powerful. Really. I promise. I can't implement change if YOU don't do it. I don't know what sucks if YOU don't tell me.
I NEED you to help me make things better.
Hiring Rails devs? I'll work for you!
Seriously, though: I firmly believe that most managers do think and act like you do.
(I've been in lead and/or management-lite roles myself; I don't view management as some weird or evil "other")
But, I think it only works if that sort of change is valued all the way up the org chart and folks at each level are empowered, incentivized, and motivated to address the pain points of the folks below.
If the org chart is more than N levels deep, then I think this becomes practically impossible. Managers feel the pain of their direct reports deeply and acutely and absolutely want to make things better: after all, it is in their best interests as well. But a manager won't feel or even understand the challenges of folks N levels below; it's probably not even practical to expect this.
(I'll let others argue over the exact value of N above)
That's why I don't think the situation can be resolved without some sort of dedicated resources allocated to developer experience. The most obvious answer (for engineering orgs big enough to support it) would be dedicated developer experience teams. For teams that lack the headcount to dedicate engineers to it full time, an alternate solution could be dedicated chunks of time -- "hackathon Fridays", etc.
Seriously, though: I firmly believe that most managers do think and act like you do.
(I've been in lead and/or management-lite roles myself; I don't view management as some weird or evil "other")
But, I think it only works if that sort of change is valued all the way up the org chart and folks at each level are empowered, incentivized, and motivated to address the pain points of the folks below.
If the org chart is more than N levels deep, then I think this becomes practically impossible. Managers feel the pain of their direct reports deeply and acutely and absolutely want to make things better: after all, it is in their best interests as well. But a manager won't feel or even understand the challenges of folks N levels below; it's probably not even practical to expect this.
(I'll let others argue over the exact value of N above)
That's why I don't think the situation can be resolved without some sort of dedicated resources allocated to developer experience. The most obvious answer (for engineering orgs big enough to support it) would be dedicated developer experience teams. For teams that lack the headcount to dedicate engineers to it full time, an alternate solution could be dedicated chunks of time -- "hackathon Fridays", etc.
I‘m working in a strongly hierarchical org, whit very high N number. What shocking me and blocked good ideas or solutions is that: The persons in charge, empowered to make decisions are so far from reality. And the outcome of their decisions has no effect on their lives… And not because they are evil, just unable to understand the problem. (Once some guy asked, why we put bugs in the code…) Now I fight to change this doing a half management half coder job. But thats hard, I am close to total burnout…
I‘m working in a strongly hierarchical org,
whit very high N number
The persons in charge, empowered to make
decisions are so far from reality
I think it's inevitable, honestly. I think it's fundamentally impossible for management to really understand or respond to in-the-trenches stuff. That's like tasking the mayor of a large city with sweeping the streets or physically fighting crime themselves, in person. They literally cannot spend enough hours in the shoes of the folks under them to understand that stuff.
And, that's fine. Management just needs to have the humility to understand that, and has to empower people/groups that do have their boots on the ground to fix those problems.
It's not unique to this industry, although I do think the newness and explosive growth of our industry may make it particularly prone to this kind of thing.
I see your point. In my BigCo I don’t see real empowering. I agree with you this is a key factor.
My dilemma should I leave or not? Is it better or worse in other place? My helplessness is, I reached limits where I know I can‘t improve the situation anymore. But I see a general pattern and so I am pessimistic to find a better place…
My dilemma should I leave or not? Is it better or worse in other place? My helplessness is, I reached limits where I know I can‘t improve the situation anymore. But I see a general pattern and so I am pessimistic to find a better place…
Please don't invalidate other people's experiences.
that's a two-way street
Expect to be wrong when speaking on other people's behalf against their interests.
If you however, do something about it, that counts for something. Not just asking others to do your job for you, without even asking them.
If you however, do something about it, that counts for something. Not just asking others to do your job for you, without even asking them.
This is why I .. struggle to function in society. There are tribal forces at play that are beyond me. Or require me to play games I don't want to play.
Social tissue is a strange medium and most of the time it's a friction generating engine. People complain, time passes on, nothing happens, repeat.
Social tissue is a strange medium and most of the time it's a friction generating engine. People complain, time passes on, nothing happens, repeat.
None of these are games: they're really just an expression of the energy you need to spend to change a rigid structure. And the more rigid, the more energy, just like a metal bar you'd be trying to fold.
You can change things but you must do it. You must put your reputation at risk, expense time that you wont be compensated for, go through conflicts you'd rather avoid, make lifelong enemies wether you succeed or not. Then, you changed the rigid structure once more and the next guy will either try to put it back where it was and face less resistance or fold it one more time and face even more.
You cannot teach a rigid structure agility just like you cannot teach iron to be silk, and the structure is rigid for a reason: size, cost, regulation, cultural beliefs, rot due to time, past mistakes (in my experience the largest factor: a mistake 10 years ago can justify 3 days a week of useless processes today) etc. You can change the structure but it's CEO-level work to change its rigidity.
You can change things but you must do it. You must put your reputation at risk, expense time that you wont be compensated for, go through conflicts you'd rather avoid, make lifelong enemies wether you succeed or not. Then, you changed the rigid structure once more and the next guy will either try to put it back where it was and face less resistance or fold it one more time and face even more.
You cannot teach a rigid structure agility just like you cannot teach iron to be silk, and the structure is rigid for a reason: size, cost, regulation, cultural beliefs, rot due to time, past mistakes (in my experience the largest factor: a mistake 10 years ago can justify 3 days a week of useless processes today) etc. You can change the structure but it's CEO-level work to change its rigidity.
Well first of all I don't consider work groups as solids, it's more like a non newtonian fluid.
And reputation, enemies.. all of these are absurd notions to me. It mostly shows immaturity in our notion of work and society. And I'm not asking for people to find solution to state finance, but even for changing the simplest thing, people will resort to primitive behavior. It's child like behavior mostly.. and the bad side of childhood, not the "let's make something fun for everybody" more like "boss doesnt like me so i wont move a finger". In those structures everybody becomes everybody's enemy, and it's a super sad sight.
And reputation, enemies.. all of these are absurd notions to me. It mostly shows immaturity in our notion of work and society. And I'm not asking for people to find solution to state finance, but even for changing the simplest thing, people will resort to primitive behavior. It's child like behavior mostly.. and the bad side of childhood, not the "let's make something fun for everybody" more like "boss doesnt like me so i wont move a finger". In those structures everybody becomes everybody's enemy, and it's a super sad sight.
This strikes a chord with me.
Ironically, the only way to minimize the "game playing" is by mastering them. There will always be people out there whose only real skill is manipulating the people around them, and who will deploy that skill at full force.
I'm not saying you should manipulate the people around you... but the twin powers of, "understanding whats going on around you" and "documenting everything", go a long way.
Ironically, the only way to minimize the "game playing" is by mastering them. There will always be people out there whose only real skill is manipulating the people around them, and who will deploy that skill at full force.
I'm not saying you should manipulate the people around you... but the twin powers of, "understanding whats going on around you" and "documenting everything", go a long way.
Yes.. I came to a similar sad realization. Either understand the game better or somehow treat people like confused scared kids posturing.
I also try to distill good vibes and motivation, bit by bit. Listening, suggesting, showing.
I've asked a few places but I've been looking for the social dynamics of work groups.
I also try to distill good vibes and motivation, bit by bit. Listening, suggesting, showing.
I've asked a few places but I've been looking for the social dynamics of work groups.
This article helped me to understand "the game" a little better. https://www.ribbonfarm.com/2009/10/07/the-gervais-principle-...
I've read these articles back and forth and I honestly cannot grasp more than 30% of the ideas. The loser eludes me, the middle layer same..
I'd love to poke in people's head to be sure what are the motives behind their behavior (the main one I assume being a balance of respect/equality and money).
I'd love to poke in people's head to be sure what are the motives behind their behavior (the main one I assume being a balance of respect/equality and money).
> I'd love to poke in people's head to be sure what are the motives behind their behavior
I suspect you're approaching things too logically. In my experience most people do something then rationalize why they've done that later, if at all. And that's without any perceptive, or memory problems or anything like that, and assumes good intent-- which a lot of people don't have.
I also suspect some people's brains work so differently from my subjective experience that there's effectively no way to understand them.
A very hard lesson for me was "don't try to use logic to understand irrational people."
I now tend to think of people in terms of, "can I predict a response or action by them?" By observation, you can kind of build a set of rules and start hypothesizing about them and update your model when you get more evidence. But you still need to be mindful that different conclusions can be reached with different or conflicting information.
It worked so well on an ex of mine that she thought I was spying on her somehow, because I often had reasoned out things I shouldn't have known. The most terrible example-- that she was cheating on me. I came to this conclusion from a few bits of evidence-- she wasn't really the curious type. Most of the film, music, etc she knew about were from me. And then one day, she started talking in detail about a film I knew she hadn't seen, and she used a word I had never heard her use before.
This lead me to think, "She's having some sort of social interactions that I'm unaware of, and they're probably watching movies together, and if its being kept secret, its probably something bad."
And I was correct.
I suspect you're approaching things too logically. In my experience most people do something then rationalize why they've done that later, if at all. And that's without any perceptive, or memory problems or anything like that, and assumes good intent-- which a lot of people don't have.
I also suspect some people's brains work so differently from my subjective experience that there's effectively no way to understand them.
A very hard lesson for me was "don't try to use logic to understand irrational people."
I now tend to think of people in terms of, "can I predict a response or action by them?" By observation, you can kind of build a set of rules and start hypothesizing about them and update your model when you get more evidence. But you still need to be mindful that different conclusions can be reached with different or conflicting information.
It worked so well on an ex of mine that she thought I was spying on her somehow, because I often had reasoned out things I shouldn't have known. The most terrible example-- that she was cheating on me. I came to this conclusion from a few bits of evidence-- she wasn't really the curious type. Most of the film, music, etc she knew about were from me. And then one day, she started talking in detail about a film I knew she hadn't seen, and she used a word I had never heard her use before.
This lead me to think, "She's having some sort of social interactions that I'm unaware of, and they're probably watching movies together, and if its being kept secret, its probably something bad."
And I was correct.
That's too cold an approach for me
I'd rather try to connect emotionally and see who's responding. I grew up the way you think and I'm tired of it. If someone doesn't like me (or doesn't like me anymore), no problem, then I'll find other people, all I want is honesty.
That said, I've encountered the post rationalizing lying kind just too many times.. :)
I'd rather try to connect emotionally and see who's responding. I grew up the way you think and I'm tired of it. If someone doesn't like me (or doesn't like me anymore), no problem, then I'll find other people, all I want is honesty.
That said, I've encountered the post rationalizing lying kind just too many times.. :)
I understand your feelings but I'm not saying you should do this with everyone-- just problematic or difficult people.
Boy, this resonated.
I left my last contracting project around six weeks ago because of encountering this attitude - yet again.
I wrote my first custom business program 40 years ago. Since then, I've only been on two teams that didn't exhibit this attitude.
At this point, I really don't want to continue programming for a living, precisely because so many people adamantly refuse to fix problems with painfully obvious (and quick and cheap) solutions.
I've given up on trying to get people to change their behavior. If I do continue programming, my approach will be to try to find one of those rare teams that actually changes and improves.
I wrote my first custom business program 40 years ago. Since then, I've only been on two teams that didn't exhibit this attitude.
At this point, I really don't want to continue programming for a living, precisely because so many people adamantly refuse to fix problems with painfully obvious (and quick and cheap) solutions.
I've given up on trying to get people to change their behavior. If I do continue programming, my approach will be to try to find one of those rare teams that actually changes and improves.
I mean, after 40 years as a programmer, can't you retire or look for a better lower paying job?
A hazard of meetings is that the best debater owns the show. It's quite possible that the tech lead thought his idea was straightforward enough that he didn't expect it to be debated, and wasn't prepared for it.
I don't introduce new information at meetings unless I am ready to back it up, which usually takes an hour of preparation for every hour of meeting. If time is of the essence, it's often better to settle things in hallway conversations or via web chat.
I don't introduce new information at meetings unless I am ready to back it up, which usually takes an hour of preparation for every hour of meeting. If time is of the essence, it's often better to settle things in hallway conversations or via web chat.
> that bringing this stuff to my manager is even worse ...
This.
While I currently have a manager that mostly understands such complaints, I have learned that even then, it amounts to nothing. Because he can't change material things without invoking higher-ups, and they won't be brought on board. For one thing because that would require a lower level manager to create tasks for a higher manager. And that's not how management works in many companies: A lower level manager didn't get to be a lower level manager by creating "work" for higher-ups. He got to be a lower level manager by absorbing the complaints from the ground force, assuring them that their voice is heard while at the same time shielding the higher-ups.
Maybe I've just been in sh*tty IT companies all my life, but I'm seeing this, and similar pattern for nearly 30 years now, and that's why I'm now searching for a job that is as far away from the potential to create overbearing complexity as possible. Even to be point where salary be damned.
This.
While I currently have a manager that mostly understands such complaints, I have learned that even then, it amounts to nothing. Because he can't change material things without invoking higher-ups, and they won't be brought on board. For one thing because that would require a lower level manager to create tasks for a higher manager. And that's not how management works in many companies: A lower level manager didn't get to be a lower level manager by creating "work" for higher-ups. He got to be a lower level manager by absorbing the complaints from the ground force, assuring them that their voice is heard while at the same time shielding the higher-ups.
Maybe I've just been in sh*tty IT companies all my life, but I'm seeing this, and similar pattern for nearly 30 years now, and that's why I'm now searching for a job that is as far away from the potential to create overbearing complexity as possible. Even to be point where salary be damned.
> It has been used as ammunition in reviews against me.
Quit. This is a sign of a broken and unfixable workplace.
Quit. This is a sign of a broken and unfixable workplace.
You don’t know other aspects about his job. If the work is easy and the pay is good there’s no point in quitting unless you aren’t working from home or there’s a substantially better offer.
They're mentioning it here as though they're unhappy. Doesn't sound like someone that's beneficial for them outside of funds, if anything.
> My manager doesn't know anything about the code, my project, or the release project.
What... does he know?
Fine, he doesn't get into the code itself. But the project and its dependencies? Yikes.
What... does he know?
Fine, he doesn't get into the code itself. But the project and its dependencies? Yikes.
He's the scrum master for other projects and he deals with things like hiring.
This is the current norm.
Most Tech Directors or Tech Managers at my massive media company were previously front end or full stack devs who were good at coordinating (asking questions and never internalizing the answers).
Since they talk to so many people and have great rapports, they can easily manage people and conversations but they cannot manage a technical system because they think they can negotiate with it.
Usually the shit they need to know is 2 bash scripts and what build is in what env. Simple enough if you are so well educated with a degree and half a brain.
Most Tech Directors or Tech Managers at my massive media company were previously front end or full stack devs who were good at coordinating (asking questions and never internalizing the answers).
Since they talk to so many people and have great rapports, they can easily manage people and conversations but they cannot manage a technical system because they think they can negotiate with it.
Usually the shit they need to know is 2 bash scripts and what build is in what env. Simple enough if you are so well educated with a degree and half a brain.
So he is HR and not a manager? A manager can evaluate your performance, if he doesn't know what you are working on then he isn't a manager. Whoever is evaluating what you do is your manager. If nobody is doing that then congratulations, you don't have a manager, you work in a so called self organizing team!
Yeah, I'd hard pass on the scrum master picking colleagues.
If the person hiring isn't technical, they likely aren't qualifying competancy, nor are they likely capable of building an Engineering culture.
If the person hiring isn't technical, they likely aren't qualifying competancy, nor are they likely capable of building an Engineering culture.
Can't your tech lead suggest that he'll do the release under the condition that someone from that other team pairs with him so that they learn how to do it?
Yeah, at a previous job, I had inherited responsibility for maintaining a particular poorly tested (and hard to test—lots of API dependencies), widely-used library. Most changes were contributed by users and I managed by reviewing new code and tests as carefully as I could.
At one point, one particularly hard-pressed user wanted to make really sweeping changes to the library and had limited time. I stalled and stalled to try and review all their changes and convince myself they were going to work, and they got more and more agitated, and my TL (who was himself frustrated by how much of my time this was consuming) told them "you can merge whatever you want if you take over maintaining it". So that's what happened, and everybody left happy.
At one point, one particularly hard-pressed user wanted to make really sweeping changes to the library and had limited time. I stalled and stalled to try and review all their changes and convince myself they were going to work, and they got more and more agitated, and my TL (who was himself frustrated by how much of my time this was consuming) told them "you can merge whatever you want if you take over maintaining it". So that's what happened, and everybody left happy.
This seems like a really good idea.
The tech lead wants to be “a hero”. It’s a common theme, the problems are well known when you have ppl hero-balling.
He actually called himself out for this during the conversation. I'm not sure how to convince someone to stop doing that.
Demean the work, don't say anything about him but minimize the value of what he is doing. Commiserate how bad it is that he has to waste a day doing something with so little value. Be on their "side" but make it clear you "understand" what they are doing isn't valuable.
All of these comments should be said in group setting to try and get the crowd to pickup similar views. Just make sure no one thinks of what they are doing in a positive light.
All of these comments should be said in group setting to try and get the crowd to pickup similar views. Just make sure no one thinks of what they are doing in a positive light.
That sounds like a really toxic way to handle the situation.
I regularly re-read the opening few pages of the Pragmatic Programmer as a mantra to myself, because what you have written here is very relevant.
> It is your life. You own it. You run it. You create it.
> Many developers we talk to are frustrated. Their concerns are varied. [...] And the answer we give is always the same.
> "Why can't you change it?"
> But, for some reason, developers seem to resist change. They hunker down, and hope things will get better.
> [...]
> So here's the most important tip in the book.
> Tip 3: You Have Agency
> Does your environment suck? Is your job boring? Try to fix it. But don't try forever. As Martin Fowler says, "You can change your organization, or you can change your organization."
When I read the situation you describe, the best thing would be to acknowledge where the tech lead is emotionally, and sympathize, because it sounds like they simply wanted to sound off about it, not actually change it at all. It's always easy to say no to why something won't work, and it sounds like a classic drama triangle situation, with someone coming in as a persecutor, you stepped in as a potential rescuer, then you became a victim because obviously your suggestions won't work.
You, however, don't need to accept the situation, and you can try to change it if you wish. Too many people (including myself, hence why I always re-read those few lines) fall into the trap of thinking that someone else should make this or that change, make a decision, or that you require approval or authority. This is far less true than people believe.
As someone who has stepped into a software manager role for the first time from being a software engineer for most of my career, my view is that I'm not always there to solve the problem, but to give space for people to solve the problems themselves. It helps if I know what's going on, or understand the issues, but I trust my team, and those I work with, and would rather get them together with the right people to discuss it through, and work out a way forward.
I appreciate I don't know the situation at your organization, the battles you fought, the things that have been said. Change is never easy, and there's always resistance, but you do have agency, as does your tech lead. Never forget it. Good luck.
> It is your life. You own it. You run it. You create it.
> Many developers we talk to are frustrated. Their concerns are varied. [...] And the answer we give is always the same.
> "Why can't you change it?"
> But, for some reason, developers seem to resist change. They hunker down, and hope things will get better.
> [...]
> So here's the most important tip in the book.
> Tip 3: You Have Agency
> Does your environment suck? Is your job boring? Try to fix it. But don't try forever. As Martin Fowler says, "You can change your organization, or you can change your organization."
When I read the situation you describe, the best thing would be to acknowledge where the tech lead is emotionally, and sympathize, because it sounds like they simply wanted to sound off about it, not actually change it at all. It's always easy to say no to why something won't work, and it sounds like a classic drama triangle situation, with someone coming in as a persecutor, you stepped in as a potential rescuer, then you became a victim because obviously your suggestions won't work.
You, however, don't need to accept the situation, and you can try to change it if you wish. Too many people (including myself, hence why I always re-read those few lines) fall into the trap of thinking that someone else should make this or that change, make a decision, or that you require approval or authority. This is far less true than people believe.
As someone who has stepped into a software manager role for the first time from being a software engineer for most of my career, my view is that I'm not always there to solve the problem, but to give space for people to solve the problems themselves. It helps if I know what's going on, or understand the issues, but I trust my team, and those I work with, and would rather get them together with the right people to discuss it through, and work out a way forward.
I appreciate I don't know the situation at your organization, the battles you fought, the things that have been said. Change is never easy, and there's always resistance, but you do have agency, as does your tech lead. Never forget it. Good luck.
I’ve been there. It sounds like you’re starting to blame yourself for their failings. You’re not sure if you’re a little crazy or they are. They’re gaslighting you. Get. Out. As. Soon. As. Possible.
If they’re using your valid concerns as issues on performance reviews, then the company values making low quality software, not their employees. Get out of the sausage factory and get into a place that values their employees and product. You’ll be much happier.
If they’re using your valid concerns as issues on performance reviews, then the company values making low quality software, not their employees. Get out of the sausage factory and get into a place that values their employees and product. You’ll be much happier.
For a person of many years of experience your friend is the problem. He has agreed to push untested code which is his responsibility, enabled other teams to work out of schedule and him producing dangerous results. All these without any indication that he sought approval from upper management. Why would you use the word "complaining" when communicating an issue? State the facts and how they diverge from the formal process, flag parts of the process that are problematic due to lack of resources of consistency, put them in an email with a "default" course of action if not answered and without using "I" or "They", use passive voice (tone). Keep the score in writing as an informal calendar linked to the respective communications. Cc the manager. Obtain bliss, people will fret then they will forget.
If the team is disfunctional or the manager bad consider changing work. Not only due to the dysfunction but because a good team and manager is part of ones' mentoring, and it hampers ones' growth. At least now you gain some reflexes that will assist you in the future, if you chose to move on.
If the team is disfunctional or the manager bad consider changing work. Not only due to the dysfunction but because a good team and manager is part of ones' mentoring, and it hampers ones' growth. At least now you gain some reflexes that will assist you in the future, if you chose to move on.
>He's the only person who knows how to release it. The other team didn't communicate until today that a release is necessary.
Also these deployment tasks are quite repetitive, it's a big strain, I can understand your pain. I have managed to automate myself out of this, at my current job: most deployment tasks involve the following steps: Commit the change, open a pull request, wait for the CI build to complete and download the build log, extract an image id from the CI build log, put it in some other repository and commit the change. It is possible to automate the first three steps with the github api (if your shop is using github), I have an example here: https://github.com/MoserMichael/githubapitools . The rest of it is easily scriptable.
Also these deployment tasks are quite repetitive, it's a big strain, I can understand your pain. I have managed to automate myself out of this, at my current job: most deployment tasks involve the following steps: Commit the change, open a pull request, wait for the CI build to complete and download the build log, extract an image id from the CI build log, put it in some other repository and commit the change. It is possible to automate the first three steps with the github api (if your shop is using github), I have an example here: https://github.com/MoserMichael/githubapitools . The rest of it is easily scriptable.
Most of the time managers are trying to be helpful, and most of the time they do in fact understand quite a lot of the background of a project.
I'd suggest to try to see the situation from their perspective, which you maybe already did, but sometimes this helps understanding a perceived lack of action.
I'd suggest to try to see the situation from their perspective, which you maybe already did, but sometimes this helps understanding a perceived lack of action.
It’s not good if they’re paralyzed by this though. If the manager automatically says no to anything that may contain some element of risk, I will stop telling them.
I have seen this happen (and in a few cases the cause for this to happen as well).
All code (or releases, for that matter) are not created equal. There are some teams which are working on something which is more critical to the business of the company, at that moment, and there are other teams which may be able to absorb some tasks to let the first team focus on what's needed. Not saying that this was the situation with the OP, but this usually falls in the "greater good" category. I have seen organizations where inter-team boundaries are fought over and defended vigorously and others where these are more fluid and accommodative. I am not sure that the firm boundary teams end up doing better work or be happier overall.
What you're doing wrong, very honestly, is you suggest and 2 of your suggestions are losses (denying release).
Dont suggest, make enemies: send emails to both teams saying now it must be like that, lay the plan, pose a meeting to go through the plan and get yelled at. Once you get yelled at by the complainypants because what you tell them to do is too hard, now you can go to your manager and explain him they complain for the sake of complaining. The tech lead will get a earful and either fight you to his grave or admit defeat and let you do your reform grumbling waiting for you to fail.
I do that in a giant bank that cannot move without 10 approvers to everything, and the more these helpless teams fight me the more management put me in their midst to reform them ... and Im just a silly dev nobody and I can tell you the wonders one can do by just bulldozing. There are things to do if you fail eventually, which is to teach the higher ups to enjoy trying and failing, but that requires a bit of weaseling charisma :D
Dont suggest, make enemies: send emails to both teams saying now it must be like that, lay the plan, pose a meeting to go through the plan and get yelled at. Once you get yelled at by the complainypants because what you tell them to do is too hard, now you can go to your manager and explain him they complain for the sake of complaining. The tech lead will get a earful and either fight you to his grave or admit defeat and let you do your reform grumbling waiting for you to fail.
I do that in a giant bank that cannot move without 10 approvers to everything, and the more these helpless teams fight me the more management put me in their midst to reform them ... and Im just a silly dev nobody and I can tell you the wonders one can do by just bulldozing. There are things to do if you fail eventually, which is to teach the higher ups to enjoy trying and failing, but that requires a bit of weaseling charisma :D
It's a fun game to play, until you realize it's not your job to manage the managers and your co-workers, and that you're underpaid for having that extra headache :)
From my experience you need to play a small political game, no way around it, what you want to do costs money and presents risk. If you believe it is necessary you need to find supporters in the management floor. Convince someone who can actually give you backup and help you get this done. If you can't find anyone like that, then either work on the project underground (risky, more work and less credit for you) or let it go (or leave).
This kind of thing needs to have some exec sponsorship or at the very least awareness. Otherwise ICs just risk landing on the sword.
You can try but once there’s resistance is probably a better idea to find something else. Let management figure out why people keep quitting.
I don’t like giving this recommendation out but the other way can make for a career limiting move.
You can try but once there’s resistance is probably a better idea to find something else. Let management figure out why people keep quitting.
I don’t like giving this recommendation out but the other way can make for a career limiting move.
> We've done two other releases in the past month and both required a day of troubleshooting to fix issues.
I feel you, it's especially frustrating when your product has such fragile and bespoke testing that things break on a regular basis. Then, us devs have to spend valuable time just troubleshooting.
I feel you, it's especially frustrating when your product has such fragile and bespoke testing that things break on a regular basis. Then, us devs have to spend valuable time just troubleshooting.
Maybe I misunderstand, but if a product breaks regularly under "fragile and bespoke" testing, won't it break just as often, and probably more often, under robust and comprehensive testing? And if the product fails tests, who better to "just" troubleshoot the problems than the developers? (Perhaps you mean things break after nominal testing and subsequent release, but that's just shifting the time when the issues are detected -- the product still has the issues either way. And if the issues will adversely and/or significantly affect the customer, I imagine one's employer would prefer that you set aside your non-troubleshooting valuable time and fix the issues.)
I think the difference here is that with robust and comprehensive testing, it's immediately obvious when and where things break. Compared to a codebase with no testing, or fragile testing, where things break and you not only have to spend time fixing something, you also have to spend time finding what is even broken in the first place. That's where the frustration is, at least for me.
> The tech lead started out by complaining about how he has to do an untested unplanned release today because another team made some urgent changes.
They broke it they have to fix it. You need the light shining on that team, not strive to fix it yourself.
They broke it they have to fix it. You need the light shining on that team, not strive to fix it yourself.
Sounds like they might not even know they broke it
>fight for what's right when others appear to moan and carry on
This is called being a toxic employee, I've come to think of that as a good thing but it's definitely not a career move.
Good story though, can definitely relate
This is called being a toxic employee, I've come to think of that as a good thing but it's definitely not a career move.
Good story though, can definitely relate
[deleted]
It may be, based on my experience as that tech lead who declines everything, that the tech lead has already thought of all this many times over many years, has tried to implement the required changes, yet gets blocked by higher ups. And it may even be that those higher ups sometimes know it too, and deny him based on their higher ups.
A lot of times learned helplessness is true helplessness with learned resignation.
A lot of times learned helplessness is true helplessness with learned resignation.
Oh man this is a punch below the belt, perfectly describes some of my previous gigs. I'm so grateful my current team is NOT like this.
I don't think it's "learned helplessness". It is a rational, calculated tradeoff decision. In many situations, it is more painful to fix the underlying layers of technical debt, and more time consuming to fight the push back from the various teams that own different parts of the technical stack. It's far easier to just silence the pagers while on pager duty for that week.
It's also caused by misaligned incentives. At most BigCo, people don't get promoted for fixing technical debt. They get promoted for launching shiny new features and products.
It's also caused by misaligned incentives. At most BigCo, people don't get promoted for fixing technical debt. They get promoted for launching shiny new features and products.
Technical debt is all of the forms of self-sabotage that don't have clear metrics associated with them.
If they had clear, unimpeachable metrics, then you could increase your status by fixing them. Since they don't, few people engage with them. Worse, engaging with some of these problems can lose you status, and so tackling them becomes a form of self-sacrifice.
If they had clear, unimpeachable metrics, then you could increase your status by fixing them. Since they don't, few people engage with them. Worse, engaging with some of these problems can lose you status, and so tackling them becomes a form of self-sacrifice.
Hinkleys's law: A measure that becomes a target might cease to be a good measure, but a target that isn't measured ceases to be a target at all.
Does this law apply also to projects like Linux Kernel or Wikipedia? If not, where does it apply?
Quite often "technical debt" from an engineer perspective is an asset from the leadership perspective. Make of it what you want, but code which is bad from a customer perspective usually doesn't live until it's legacy.
And sometimes the technical debt cloud disappears after fully understanding the business rules and special cases which were the root cause for the code in question. This often gets overlooked and may (edit) lead to refactoring/rewrite approaches which ultimately fail. As everyone knows, not a rare occurrence.
And sometimes the technical debt cloud disappears after fully understanding the business rules and special cases which were the root cause for the code in question. This often gets overlooked and may (edit) lead to refactoring/rewrite approaches which ultimately fail. As everyone knows, not a rare occurrence.
We need a different word. ‘Technical debt’ implies some kind of objectively quantitative zero sum game, when it is anything but.
There are a couple of amazing talks by Dan North, where one of the ideas he shares is that the entire idea of technical debt is wrong - it implies that "good" code a team produced is an asset and "bad" code is debt, and applies a set of moral judgements where teams that produce this debt are bad.
Instead - code is neither an asset or debt - it's just cost. It merely is a direct result of whatever effort the team has spent in the past, but at any moment it could be greatly simplified or even replaced by a new open source library or commercial tool. If good code was an asset, replacing it with something would be rare, but history shows us we keep replacing libraries and entire products despite some of them being made up of really great code.
> All code is a cost
This is the truth, as far I'm concerned. We say "technical debt" instead of "bad code" because the former sounds more acceptable in the business environment. But it's still the same meaning, code we don't like, and code we don't.
This is the truth, as far I'm concerned. We say "technical debt" instead of "bad code" because the former sounds more acceptable in the business environment. But it's still the same meaning, code we don't like, and code we don't.
> We need a different word. ‘Technical debt’ implies some kind of objectively quantitative zero sum game, when it is anything but.
Not only that, but it's been around long enough that bad management types throw the term around. That's a sure enough sign that whatever usefulness it had has been far outlived.
Not only that, but it's been around long enough that bad management types throw the term around. That's a sure enough sign that whatever usefulness it had has been far outlived.
A thousand times this. Good, steady software ownership is generally not what leadership wants.
Big new features are, it’s infinitely easier to build process improvements into new projects vs fighting for improvements in existing areas.
Big new features are, it’s infinitely easier to build process improvements into new projects vs fighting for improvements in existing areas.
Oh this hits close to home. At my BigCo I have seen tremendous amount of great tools/product landed in the trash bin, because it was only interesting until somebody get the promotion… Immense amount technical dept and lack of viable business model lead to fast death of great ideas… What a huge waste…
As a member of a core infra/"foundation" team, the biggest drain on my soul is the number of other engineers that are helpless, or never learned how to find solutions on their own. They never search wikis, look for similar posts on internal groups, or even read the error message from the tool that tells them exactly how to fix the problem they're asking about. When the culture has become "google everything", but you can't google for internal tools/tech problems, suddenly folks have no idea how to read error messages, debug a stack trace, or solve any problem without hand-holding from a senior engineer. I've been at BigCo for nine years now, and it has only gotten worse as the size of the company has grown exponentially.
Think about it from the other side:
Dealing with the infrastructure is your job. You know the ins and outs. It doesn't seem complicated to you.
For someone who is dealing with the application logic, having to context switch to infrastructure is painful. I have no idea how you have set things up. When I read your documentation I'm just baffled at the sheer amount of complexity. I don't want to deal with it. It's not my full time job. I have other things to worry about. I have no mental space to deal with this headache. I will just ask you so I can move on to do my actual job - which is already taking up over 90% of my mental capacity.
When I setup my own infrastructure, I make it as simple and stupid as possible, precisely because I have no mental slack to deal with all the complexity that I see devops people deal with on a daily basis.
At most jobs I've been to, the complexity I've seen in infrastructure is mind boggling. My intuition is that this complexity is not actually needed if you remove all the cruft and think from first principles about what the actual problem is you are trying to solve.
However, people in devops seem to enjoy building and maintaining this kind of complexity, and even take pride in how everything is orchestrated.
To me it's just a headache.
Dealing with the infrastructure is your job. You know the ins and outs. It doesn't seem complicated to you.
For someone who is dealing with the application logic, having to context switch to infrastructure is painful. I have no idea how you have set things up. When I read your documentation I'm just baffled at the sheer amount of complexity. I don't want to deal with it. It's not my full time job. I have other things to worry about. I have no mental space to deal with this headache. I will just ask you so I can move on to do my actual job - which is already taking up over 90% of my mental capacity.
When I setup my own infrastructure, I make it as simple and stupid as possible, precisely because I have no mental slack to deal with all the complexity that I see devops people deal with on a daily basis.
At most jobs I've been to, the complexity I've seen in infrastructure is mind boggling. My intuition is that this complexity is not actually needed if you remove all the cruft and think from first principles about what the actual problem is you are trying to solve.
However, people in devops seem to enjoy building and maintaining this kind of complexity, and even take pride in how everything is orchestrated.
To me it's just a headache.
The drain I am talking about is a complete lack of due diligence on behalf of those asking for my help (and my limited time/attention).
When the error message contains a clear message and a URL with step-by-step instructions, and the person in question just pastes the full error message including the URL, and asks "what do I do?"
When a problem can be solved by responding with "please follow the instructions in the error message you sent me".
When someone makes an internal group post asking "how do I do X?" and the pinned post that was right below the submit button contains a summary and link to detailed step-by-step instructions titled "how to X".
When the error message contains a clear message and a URL with step-by-step instructions, and the person in question just pastes the full error message including the URL, and asks "what do I do?"
When a problem can be solved by responding with "please follow the instructions in the error message you sent me".
When someone makes an internal group post asking "how do I do X?" and the pinned post that was right below the submit button contains a summary and link to detailed step-by-step instructions titled "how to X".
I've been in your shoes and I've accepted this fate.
I am the expert. I know things. I choose to change my viewpoint to taking pride in making the best possible explanation that serves the users' need. When they ask me "what do I do?" I no longer snicker or sigh that they don't know or that they are too lazy to find out. Instead I copy-paste the answer they need. It takes me seconds, perhaps it saves them thinking. Thinking seems hard to some people. I choose to believe I save the company some time when I think for them.
Yes, they do not learn when they are spoon-fed the answers. Yes, they could have figured it out. So what if they didn't? They are producing whatever they are supposed to produce.
When I get too tired of the same questions, I change jobs. I am in a C-level position now, and somehow the questions from other C-level managers are similar. They have no idea how they should deal with things from my business area, and that's okey. I'm here to help. I help. When I get too tired of the same questions, I will change jobs.
I am the expert. I know things. I choose to change my viewpoint to taking pride in making the best possible explanation that serves the users' need. When they ask me "what do I do?" I no longer snicker or sigh that they don't know or that they are too lazy to find out. Instead I copy-paste the answer they need. It takes me seconds, perhaps it saves them thinking. Thinking seems hard to some people. I choose to believe I save the company some time when I think for them.
Yes, they do not learn when they are spoon-fed the answers. Yes, they could have figured it out. So what if they didn't? They are producing whatever they are supposed to produce.
When I get too tired of the same questions, I change jobs. I am in a C-level position now, and somehow the questions from other C-level managers are similar. They have no idea how they should deal with things from my business area, and that's okey. I'm here to help. I help. When I get too tired of the same questions, I will change jobs.
The way I've seen things work is like this:
devops sets up something convoluted that they for some reason like.
Developers are confused by the thing. It causes them endless problems.
Documentation is sparse, or out of date, or just plain wrong. The instructions to fix problems omit key requirements because the person who wrote them just assumed the reader knows about X and Y when the reader often has no clue. You follow the instructions and the problem is not solved, and maybe to make matters worse, you run into new problems.
The way I see it is: if developers enjoyed dealing with infrastructure, devops jobs would not exist, it would just be something that developers do as part of their job.
As a devops engineer, you should think of your fellow software engineers as your users, not as your "project members".
Your job is to make it as painless and problem-free for them as possible. If problems arise constantly, that's not the user's fault; that's your fault.
devops sets up something convoluted that they for some reason like.
Developers are confused by the thing. It causes them endless problems.
Documentation is sparse, or out of date, or just plain wrong. The instructions to fix problems omit key requirements because the person who wrote them just assumed the reader knows about X and Y when the reader often has no clue. You follow the instructions and the problem is not solved, and maybe to make matters worse, you run into new problems.
The way I see it is: if developers enjoyed dealing with infrastructure, devops jobs would not exist, it would just be something that developers do as part of their job.
As a devops engineer, you should think of your fellow software engineers as your users, not as your "project members".
Your job is to make it as painless and problem-free for them as possible. If problems arise constantly, that's not the user's fault; that's your fault.
[deleted]
> My intuition is that this complexity is not actually needed if you remove all the cruft and think from first principles about what the actual problem is you are trying to solve.
Have you ever worked on infrastructure? This comment is wrong on soo many levels.
Have you ever worked on infrastructure? This comment is wrong on soo many levels.
> Have you ever worked on infrastructure?
Not really. From an application perspective you can generally get away with two really big servers (one of which is the backup), and a load balancer.
There may be more involved in the surrounding infra, but that’s stuff the application developer will never have to touch.
Not really. From an application perspective you can generally get away with two really big servers (one of which is the backup), and a load balancer.
There may be more involved in the surrounding infra, but that’s stuff the application developer will never have to touch.
This comment is not wrong on any level.
I have worked in many aspects of web development. I know for sure that everything in the process is way too complicated for what it does. Unnecessarily so.
There's no reason to think that infrastructure is the one place where all the complexity is there for good reason.
Have I worked on infrastructure? Yes and no. Depends on what you mean.
I have worked on infrastructure in the sense that I have setup the infrastructure for websites that can handle thousands of concurrent requests, and the infrasturcure I setup is stupidly simple, everyone who sees thinks there must be something missing. But there isn't anything missing. It handles the job much better than the complicated mess that I see in all other companies.
Now, the thing I haven't done is actually setup the infrastructure for a truely distributed system that is horizontally scaled. I do all my scaling vertically because computers are so fast and storage capacity is so large that one server can handle thousands of concurrent users without nearly sweating.
Part of the reason everyone else has a super complicated infrastructure is because they chose bad software technologies to develop their applications, such as, python.
Yea, if your application backend is in python or ruby, you have no choice but to create a complicated infrastructure. Because those languages are so slow, you just cannot scale vertically. That's not an option. You get an explosion of complexity because you also need additional servers to handle stuff that would normally just be part of the application code.
For example, when I program in Go, and I need to cache some data, I just create a map, or something, to cache data in memory. But if you have Ruby, you can't just do that! Since you have 200 application instances running on aws, you want to use something like redis to act like a shared distributed cache. So now your infrastructure has to include information about how the application servers need to communicate with the caching servers.
All of this is unnecessary complication that can go away if you stop what you are doing and just analyze the problem from first principles.
Your massive convoluted infrastrucutre is only there to compensate for the bad decision you made early in the process, which was to use a slow interpreted language to write the backend of your system.
Most of the time the real solution to scaling problem is just writing better (and simpler) code.
I've seen at more than one company, how this complciated infrastructure does nothing to make the application robust because it grinds to a halt when there are about 1000 concurrent users.
I have worked in many aspects of web development. I know for sure that everything in the process is way too complicated for what it does. Unnecessarily so.
There's no reason to think that infrastructure is the one place where all the complexity is there for good reason.
Have I worked on infrastructure? Yes and no. Depends on what you mean.
I have worked on infrastructure in the sense that I have setup the infrastructure for websites that can handle thousands of concurrent requests, and the infrasturcure I setup is stupidly simple, everyone who sees thinks there must be something missing. But there isn't anything missing. It handles the job much better than the complicated mess that I see in all other companies.
Now, the thing I haven't done is actually setup the infrastructure for a truely distributed system that is horizontally scaled. I do all my scaling vertically because computers are so fast and storage capacity is so large that one server can handle thousands of concurrent users without nearly sweating.
Part of the reason everyone else has a super complicated infrastructure is because they chose bad software technologies to develop their applications, such as, python.
Yea, if your application backend is in python or ruby, you have no choice but to create a complicated infrastructure. Because those languages are so slow, you just cannot scale vertically. That's not an option. You get an explosion of complexity because you also need additional servers to handle stuff that would normally just be part of the application code.
For example, when I program in Go, and I need to cache some data, I just create a map, or something, to cache data in memory. But if you have Ruby, you can't just do that! Since you have 200 application instances running on aws, you want to use something like redis to act like a shared distributed cache. So now your infrastructure has to include information about how the application servers need to communicate with the caching servers.
All of this is unnecessary complication that can go away if you stop what you are doing and just analyze the problem from first principles.
Your massive convoluted infrastrucutre is only there to compensate for the bad decision you made early in the process, which was to use a slow interpreted language to write the backend of your system.
Most of the time the real solution to scaling problem is just writing better (and simpler) code.
I've seen at more than one company, how this complciated infrastructure does nothing to make the application robust because it grinds to a halt when there are about 1000 concurrent users.
I guess there is a disconnect between what you mean by the term infrastructure and what I mean.
Let's take an example, say Reddit. You want to support iOS, android and web. All platforms have the same core data but maybe have different APIs for optimizing queries, e.g. you may want to use server side rendering for web.
So here's some of the microservices that you will use server side. I will label which I consider to be infra:
LoginService (infra)
Database server or library (to create consistent views/transactions on top of your raw database, infra)
Core business logic server
Web frontend server
Android/iOS frontend servers (probably not needed)
Image/audio/video processing (infra)
ML server (infra)
I don't see how any of the infra services here are trivial.
Let's take an example, say Reddit. You want to support iOS, android and web. All platforms have the same core data but maybe have different APIs for optimizing queries, e.g. you may want to use server side rendering for web.
So here's some of the microservices that you will use server side. I will label which I consider to be infra:
LoginService (infra)
Database server or library (to create consistent views/transactions on top of your raw database, infra)
Core business logic server
Web frontend server
Android/iOS frontend servers (probably not needed)
Image/audio/video processing (infra)
ML server (infra)
I don't see how any of the infra services here are trivial.
If by infrastructure you mean microservices, then it's even worse.
I wholly reject "microservices", the philosophy and ideas behind it, the tooling and the culture surrounding it.
It's not only too complicated. It's worse than useless. It's extra work that produces negative value.
I wholly reject "microservices", the philosophy and ideas behind it, the tooling and the culture surrounding it.
It's not only too complicated. It's worse than useless. It's extra work that produces negative value.
there is a saying, you may think you are intelligent but you are only as intelligent as what your student is able to learn from you.
it's basically "beautiful mind" syndrome, and it's steeped deeply in ego.
now if on the other hand that teacher is coming from a place where it was hard for him, he probably has a sense of empathy and is willing to teach.
it's basically "beautiful mind" syndrome, and it's steeped deeply in ego.
now if on the other hand that teacher is coming from a place where it was hard for him, he probably has a sense of empathy and is willing to teach.
Totally agree..
As a rails dev I'm asked to dig into ops.. Not once is the JS team asked to build out the rails code, or fix things in it when it glitches. The api business logic falls down, and thats on me.
But when ops is glitchy, I have to blow half my day faking interest in docker? I mean best case, I learn how ops works, and then what. I become the one they tap when ops is swamped, then before I know it, I'm doing ops.
And I quit, because as much as I love the ops team, I hate the gig. I'd hate to be a designer, or a BA. It's not where I find joy.
So when I hear people say.. Just follow these steps to trouble shoot, I wonder if it would be cool to tell that to our users..
But when ops is glitchy, I have to blow half my day faking interest in docker? I mean best case, I learn how ops works, and then what. I become the one they tap when ops is swamped, then before I know it, I'm doing ops.
And I quit, because as much as I love the ops team, I hate the gig. I'd hate to be a designer, or a BA. It's not where I find joy.
So when I hear people say.. Just follow these steps to trouble shoot, I wonder if it would be cool to tell that to our users..
Think about it from the other side:
Infrastructure is like application dev in that it’s a bundle of histories of feature work over time and with many contributors, and many customers with competing wants and not enough time to satisfy everyone.
Granted a primary goal should be to have simple APIs/workflows for app devs to use, but complicated insides is as natural as any advanced system.
Infrastructure is like application dev in that it’s a bundle of histories of feature work over time and with many contributors, and many customers with competing wants and not enough time to satisfy everyone.
Granted a primary goal should be to have simple APIs/workflows for app devs to use, but complicated insides is as natural as any advanced system.
I am working at company that just won a contract with a BigCo and it has to be one of the most frustrating experiences I have ever had. I was asked to test a configuration change and I was able to demonstrate that it worked locally in 15 minutes. Unfortunately, testing locally isn't enough so now I need to test it in the lab, but BigCo is so compartmentalised that it can weeks even to organise a meeting where all the stakeholder can get together and talk about what needed to be done to get the configuration changed in the lab.
We've had working sessions where the moment there was an issue, the developers at BigCo said that they need to talk to someone on some team and they wrap up the meeting for the day -- even if we had just got started. It's gotten to the point where we will have project managers in the working session -- essentially babysitting -- to stop the devs from just giving up.
The amount of stonewalling I've seen is insane. We were trying to address an issue with our integration and I was told that a particular configuration on the client side was impossible, or maybe that we needed a new license, or perhaps we needed to contract to another vendor and it's going to cost half a million dollars, or whatever other excuses they could come up with.
So I asked them what product BigCo used, I looked up the documentation, I sent an email with 12 steps on how to make the configuration change, I got on a video call with my contact at BigCo (and his boss to make sure he showed up) and I literally walked him through the setting up the configuration.
I just don't understand how you get to the point where that's business as usual.
We've had working sessions where the moment there was an issue, the developers at BigCo said that they need to talk to someone on some team and they wrap up the meeting for the day -- even if we had just got started. It's gotten to the point where we will have project managers in the working session -- essentially babysitting -- to stop the devs from just giving up.
The amount of stonewalling I've seen is insane. We were trying to address an issue with our integration and I was told that a particular configuration on the client side was impossible, or maybe that we needed a new license, or perhaps we needed to contract to another vendor and it's going to cost half a million dollars, or whatever other excuses they could come up with.
So I asked them what product BigCo used, I looked up the documentation, I sent an email with 12 steps on how to make the configuration change, I got on a video call with my contact at BigCo (and his boss to make sure he showed up) and I literally walked him through the setting up the configuration.
I just don't understand how you get to the point where that's business as usual.
> I just don't understand how you get to the point where that's business as usual.
Simple, just get 100 junior developers working on the same product, and make the Project Managers’s manage them.
Simple, just get 100 junior developers working on the same product, and make the Project Managers’s manage them.
I know your frustration and pain, but as someone frequently on the other side of the equation trying to seek out answers, it is often difficult-to-impossible.
Just this past week I was working through some infra setup challenges, I had to piece together half a dozen different documents, several README files, and a few archived slack threads. These sources often are outdated or even contradicted each other. Even after all that and some earnest trial and error I still had to ask for help.
It's fine to want to point to the docs, but when you do that you better have good docs you're pointing to. As an infra team, often your "API" is your docs and templates. Searching slack threads and piecing together bad docs isn't a scalable solution for every dev to repeat individually. But neither is pinging an individual with every question. Teams need to invest in good documentation.
Just this past week I was working through some infra setup challenges, I had to piece together half a dozen different documents, several README files, and a few archived slack threads. These sources often are outdated or even contradicted each other. Even after all that and some earnest trial and error I still had to ask for help.
It's fine to want to point to the docs, but when you do that you better have good docs you're pointing to. As an infra team, often your "API" is your docs and templates. Searching slack threads and piecing together bad docs isn't a scalable solution for every dev to repeat individually. But neither is pinging an individual with every question. Teams need to invest in good documentation.
No, no. This sounds completely different. You searched out all the information you could, and still had no idea. That’s great (I mean, not great, ideally the necessary information would have been available, but that’s on the people providing the information).
The problem is people that have not searched or tried at all, and then ask their senior to please do their job for them.
The problem is people that have not searched or tried at all, and then ask their senior to please do their job for them.
There is a classic article about this issue which is from Harvard Business Review in 1974 and is about monkeys. a Must Read.
https://www.academia.edu/36290372/Monkey https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsd...
https://www.academia.edu/36290372/Monkey https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsd...
Hah, this is great. I think I’ll start telling people to keep their monkeys :)
I agree with this and would add Slack to the equation. It's just faster to ping a senior engineer on Slack than to spend 5 minutes looking into an issue.
I've also seen new hires burn an entire week because they were afraid to ask a question.
But I think the balance has definitely shifted towards asking too much help...
I've also seen new hires burn an entire week because they were afraid to ask a question.
But I think the balance has definitely shifted towards asking too much help...
When I started at BigCo, the recommended policy was to spend two hours trying to solve it yourself, including searching the wiki, debugging code, etc. If you hadn't made any progress in that time period, then ask someone from your own team or make a post in a related internal support group. Only after exhausting that route, and getting no help, should you escalate to the team/oncall that owns the tool/service you are having problems with. It seems that culture has been lost.
I still do this and I don't think this is entirely about culture. Some people do the legwork and some take the laziest route. Sure if the technical tools at hand make it that much hard to do the due diligence, maybe promptly reaching out for help can be (somewhat) justified, but I don't think that is usually the case.
Here is my balance:
Study a bit, WRITE my discoveries down
Craft a message with issues and background clear to readers (that includes myself when I forgot about this)
Send message to selected recipients, continue studying depends on free time available and issue importance
Study a bit, WRITE my discoveries down
Craft a message with issues and background clear to readers (that includes myself when I forgot about this)
Send message to selected recipients, continue studying depends on free time available and issue importance
> engineers that are helpless, or never ... or even read the error message from the tool
If they don't read the error messages from their tools, they are not engineers: they are idiots.
If they don't read the error messages from their tools, they are not engineers: they are idiots.
> I've been at BigCo for nine years now, and it has only gotten worse as the size of the company has grown exponentially.
You should try to change team or company then. Don't believe all SWEs suck. There are pretty good coders out there :)
You should try to change team or company then. Don't believe all SWEs suck. There are pretty good coders out there :)
This reminds me of a story I've read some time ago, about some monkeys and bananas.
A group of monkeys in a room. In the center of the room is a tall pole with a bunch of bananas suspended from the top.
Every time a monkey tries to reach the bananas it is hit with a torrent of cold water from an overhead shower. Eventually, the monkeys learn that something bad will happen if they climb up the pole so they just sit and don’t even try again.
If a new monkey is added to replace an old monkey, it will of course try to go for the bananas. But the original monkeys will grab and drag it down before the punishment is even triggered. After a while, the monkey gets the message and it stops trying.
Repeat the action of adding a new monkey in place of an old one enough times and you will reach a point where none of the monkeys know why they shouldn't get the bananas but they still won't try.
The story concludes by stating that even if we remove the shower at this point it won't really make a difference. The learned behavior is already deeply ingrained in the culture of the group.
Funny enough, I've seen this type of issue in many human teams/projects. And usually the inertia is far too strong for just a couple of engineering "monkeys" to radically change things if they don't get support from a few management "monkeys".
A group of monkeys in a room. In the center of the room is a tall pole with a bunch of bananas suspended from the top.
Every time a monkey tries to reach the bananas it is hit with a torrent of cold water from an overhead shower. Eventually, the monkeys learn that something bad will happen if they climb up the pole so they just sit and don’t even try again.
If a new monkey is added to replace an old monkey, it will of course try to go for the bananas. But the original monkeys will grab and drag it down before the punishment is even triggered. After a while, the monkey gets the message and it stops trying.
Repeat the action of adding a new monkey in place of an old one enough times and you will reach a point where none of the monkeys know why they shouldn't get the bananas but they still won't try.
The story concludes by stating that even if we remove the shower at this point it won't really make a difference. The learned behavior is already deeply ingrained in the culture of the group.
Funny enough, I've seen this type of issue in many human teams/projects. And usually the inertia is far too strong for just a couple of engineering "monkeys" to radically change things if they don't get support from a few management "monkeys".
That might be just a story, rather than something that actually happened. https://www.throwcase.com/2014/12/21/that-five-monkeys-and-a...
yep, that's why I called it a story, not a scientific experiment. It could even be read as a kind of anecdote of parable.
While oversimplified, it can serve as a good example on how an initially rational practice could easily became dogma even when the initial stimulus is no longer present.
Part of my day job in the past 2 years has been to consult a client in refactoring/rewriting an old platform. The client's old team is still in place and even if we've added some new people just to try and change the mindset, the inertia of the spaghetti codebase coupled with the client's short and mid term financial goals as well as the "traditions" of the old team has stopped us from enacting any groundbreaking changes.
I guess the moral is that there are many forces at play in this kind of setup and stories like these can help in framing some of them in a fun way.
While oversimplified, it can serve as a good example on how an initially rational practice could easily became dogma even when the initial stimulus is no longer present.
Part of my day job in the past 2 years has been to consult a client in refactoring/rewriting an old platform. The client's old team is still in place and even if we've added some new people just to try and change the mindset, the inertia of the spaghetti codebase coupled with the client's short and mid term financial goals as well as the "traditions" of the old team has stopped us from enacting any groundbreaking changes.
I guess the moral is that there are many forces at play in this kind of setup and stories like these can help in framing some of them in a fun way.
This article is bashing BigCo while letting move-fast-and-break-things Startup inc. off the hook.
Just because the fires are different every day at Startup Inc. doesn't mean we're not being conditioned to endure suffering.
Case in point, a previous BigCo on call roster that I was required to partake in, sucks, makes you lose will to live, keeps you up at night, nothing documented, on-call phone ringing 5-7 of the nights in the week.
I tried to improve this process and was mildly sucessful even if that was just to get these incidents recorded.
Fast forward to my next job at Startup Inc. and their on call roster, sucks, makes you lose will to live, keeps you up at night, nothing documented, on-call phone ringing less but still often enough and with higher urgency.
The source of the two problems was different most of the time, one being a lack of training and effectively giving children knives.
The other being moving fast and breaking things resulting in things breaking after it was declared that version x.x went out without any issues just like the CEO wanted.
Just because the fires are different every day at Startup Inc. doesn't mean we're not being conditioned to endure suffering.
Case in point, a previous BigCo on call roster that I was required to partake in, sucks, makes you lose will to live, keeps you up at night, nothing documented, on-call phone ringing 5-7 of the nights in the week.
I tried to improve this process and was mildly sucessful even if that was just to get these incidents recorded.
Fast forward to my next job at Startup Inc. and their on call roster, sucks, makes you lose will to live, keeps you up at night, nothing documented, on-call phone ringing less but still often enough and with higher urgency.
The source of the two problems was different most of the time, one being a lack of training and effectively giving children knives.
The other being moving fast and breaking things resulting in things breaking after it was declared that version x.x went out without any issues just like the CEO wanted.
Read Dan Luu's article about normalization of deviance, or Dan Na's talk "pushing through friction".
https://blog.danielna.com/talks/pushing-through-friction/
https://blog.danielna.com/talks/pushing-through-friction/
One of the best talk I haven't seen for quite a long time, thanks for sharing.
Dan Luu's article: http://danluu.com/wat/
The equation is skewed by the fact that quitting will get one a 20% bump in comp vs fighting the system will get you a ton of stress
From the article;
> Hold managers accountable: one of the key responsibilities of leaders is to create positive change for their teams. Once you notice a situation where you think everyone is in learned helplessness mode, make sure to notify your manager and follow-up until the problem is addressed.
"Hey boss, I noticed that Bob and Joe are really helpless at their jobs and that you just kinda watch it happen."
Yeah what could possibly go wrong with that?
I'm not saying the thinking is wrong, but a little idealistic. Not everyone works at BigCo where it is impossible to get fired and you only ever get transferred. Nevermind most of the people in this position are going to have extremely short tenture.
> Hold managers accountable: one of the key responsibilities of leaders is to create positive change for their teams. Once you notice a situation where you think everyone is in learned helplessness mode, make sure to notify your manager and follow-up until the problem is addressed.
"Hey boss, I noticed that Bob and Joe are really helpless at their jobs and that you just kinda watch it happen."
Yeah what could possibly go wrong with that?
I'm not saying the thinking is wrong, but a little idealistic. Not everyone works at BigCo where it is impossible to get fired and you only ever get transferred. Nevermind most of the people in this position are going to have extremely short tenture.
I don't know if that was intended to be snarky, but yes, that's exactly what you should do, and people do in fact do that, and it often does work (although probably not as quickly as you'd like).
How else am I supposed to know Bob and Joe are doing bad work? Should I count their LOC/day? Pore over their code and rely on my own (10 years out of date) opinion of it?
In my humble opinion, there is one (1) good way to measure developer productivity, and that is to ask their teammates, and if you won't tell me your teammate is awful, you can't really complain that I don't act on it. Of course it's possible that you tell me and I don't do anything, but at least then it's my fault and not yours.
How else am I supposed to know Bob and Joe are doing bad work? Should I count their LOC/day? Pore over their code and rely on my own (10 years out of date) opinion of it?
In my humble opinion, there is one (1) good way to measure developer productivity, and that is to ask their teammates, and if you won't tell me your teammate is awful, you can't really complain that I don't act on it. Of course it's possible that you tell me and I don't do anything, but at least then it's my fault and not yours.
It is somewhat meant to be snarky, but then again the original quote is exceedingly naive.
It describes very well the phenomenon but gives advice that I can only see ever working at BigCo. Where your team is huge and if you screw up there's another one down the hall.
The truth is most places that exhibit this level of helplessness often include a small team of 5 or 6, each with plenty of tenure. With no other lateral teams to move to.
Now you want the new guy to walk up to the boss and tell him that his drinking buddies for the past decade are the reason things are falling apart and he's an ineffective manager for not noticing it. You can play with the wording but that's the message.
In the real world we have political factors like seniority, favoritism, nepotism, tenure, and nevermind external factors.
In the real world the boss doesn't actually care if the team is working well or not. He only cares that the perception of the team is positive. If the new guy threatens the perception, he's gotta go. Nobody cares if it gets fixed. If nobody knows it's broken there's no need to fix it.
Let's have less articles like this one where we basically fire ourselves ostensibly for virtue signaling and more articles about how to socially engineer your way to the fucking top. Because let's face it, that's what it actually takes.
It describes very well the phenomenon but gives advice that I can only see ever working at BigCo. Where your team is huge and if you screw up there's another one down the hall.
The truth is most places that exhibit this level of helplessness often include a small team of 5 or 6, each with plenty of tenure. With no other lateral teams to move to.
Now you want the new guy to walk up to the boss and tell him that his drinking buddies for the past decade are the reason things are falling apart and he's an ineffective manager for not noticing it. You can play with the wording but that's the message.
In the real world we have political factors like seniority, favoritism, nepotism, tenure, and nevermind external factors.
In the real world the boss doesn't actually care if the team is working well or not. He only cares that the perception of the team is positive. If the new guy threatens the perception, he's gotta go. Nobody cares if it gets fixed. If nobody knows it's broken there's no need to fix it.
Let's have less articles like this one where we basically fire ourselves ostensibly for virtue signaling and more articles about how to socially engineer your way to the fucking top. Because let's face it, that's what it actually takes.
Why do you want to "engineer your way to the fucking top" of a broken organization with a bunch of teams full of bad engineers managed by stuffed shirts who don't give a shit?
You may think I sound naive; I think you sound burned out. In my experience, it's pretty common for someone to say, "hey boss, Joe is really phoning it in these past few months, it's dragging the team down and you need to do something about it" and for the manager to either help Joe improve or fire Joe. And given how cynical your attitude is, I think you'd be very surprised at how often Joe actually does improve. Everyone will have the worst year of their career at some point, and it usually isn't the last one.
You may think I sound naive; I think you sound burned out. In my experience, it's pretty common for someone to say, "hey boss, Joe is really phoning it in these past few months, it's dragging the team down and you need to do something about it" and for the manager to either help Joe improve or fire Joe. And given how cynical your attitude is, I think you'd be very surprised at how often Joe actually does improve. Everyone will have the worst year of their career at some point, and it usually isn't the last one.
> Why do you want to "engineer your way to the fucking top" of a broken organization with a bunch of teams full of bad engineers managed by stuffed shirts who don't give a shit?
Because it’s easier than socially engineering your way up a competent organization and the paychecks are similar.
Because it’s easier than socially engineering your way up a competent organization and the paychecks are similar.
I’ll plus one this. I have also talked to managers about coworkers (and have them ask me about teammates). Maybe it’s luck if the draw but it’s possible to have healthy managers and still have dysfunction that needs dealt with. There never won’t be.
It has nothing to do with "BigCo" or not.
If you work in a small company, pretty much any boss will be happy to hear feedback like "Hey, it looks like there's an impediment to smooth work, and the team has gotten used to it, but it's costing us". Bonus points if you propose a solution.(If your manager isn't happy to hear that, it's time to leave, because small companies closing their eyes to this have a tendency to go down in flames)
But you'll definitely need to learn to take the blame out of your communication. Learned helplessness is a technical term, it doesn't mean "Bob and Joe are helpless". And your manager not noticing a problem is hardly "you just kinda watch it happen".
Frame it as a possible improvement to what the team can do - because that's what it is. Yes, you might get turned down. And if you repeatedly get turned down, you have a choice to make.
If you work in a small company, pretty much any boss will be happy to hear feedback like "Hey, it looks like there's an impediment to smooth work, and the team has gotten used to it, but it's costing us". Bonus points if you propose a solution.(If your manager isn't happy to hear that, it's time to leave, because small companies closing their eyes to this have a tendency to go down in flames)
But you'll definitely need to learn to take the blame out of your communication. Learned helplessness is a technical term, it doesn't mean "Bob and Joe are helpless". And your manager not noticing a problem is hardly "you just kinda watch it happen".
Frame it as a possible improvement to what the team can do - because that's what it is. Yes, you might get turned down. And if you repeatedly get turned down, you have a choice to make.
Another possible approach is "model, document and share"[1]:
> Imagine you’ve started a new job as an engineering manager, and the teams around you are too busy to use a planning process. You’ve mentioned to your peers a few times that you’ve seen Kanban work effectively, but folks tried it two years ago and are still upset whenever the word is mentioned: it just doesn’t work here.
> Your first reaction might be to confront this head on, but it takes a while to build credibility after starting a new job. Sure, you’ve been hired for your experience so they respect your judgement, but it’s a hard sell to convince someone that your personal experience should invalidate their personal experience.
[1] https://lethain.com/model-document-share/
> Imagine you’ve started a new job as an engineering manager, and the teams around you are too busy to use a planning process. You’ve mentioned to your peers a few times that you’ve seen Kanban work effectively, but folks tried it two years ago and are still upset whenever the word is mentioned: it just doesn’t work here.
> Your first reaction might be to confront this head on, but it takes a while to build credibility after starting a new job. Sure, you’ve been hired for your experience so they respect your judgement, but it’s a hard sell to convince someone that your personal experience should invalidate their personal experience.
[1] https://lethain.com/model-document-share/
I’d like to add a #3 to the list, which I’ll half-jokingly name “ruined by working at a large company learned helplessness”
My company is small and we have very small teams (1-4 people). One person recently hired has been working at successively larger companies over the years (I’ve known him for a long time). Sometimes he’ll do something like this: stop working on a task he’s been assigned, say “test data needed”, and just totally bow out until someone else makes it for him.
As I said, this is a small team. He knows how to make his own test data, and sometimes there is even a UI dedicated to making the kind of data he needs. But he has learned from working at larger companies that he can sometimes just pass his work onto someone else (perhaps even an entire other team dedicated to the thing he doesn’t want to do) instead of stepping up to the plate to do such a trivial thing on his own.
To be clear, this is just an example (which has actually happened) and other similar “not my job” thinking/actions have shown up repeatedly. Many years ago we worked together on a small team and he was not like this at that time.
My company is small and we have very small teams (1-4 people). One person recently hired has been working at successively larger companies over the years (I’ve known him for a long time). Sometimes he’ll do something like this: stop working on a task he’s been assigned, say “test data needed”, and just totally bow out until someone else makes it for him.
As I said, this is a small team. He knows how to make his own test data, and sometimes there is even a UI dedicated to making the kind of data he needs. But he has learned from working at larger companies that he can sometimes just pass his work onto someone else (perhaps even an entire other team dedicated to the thing he doesn’t want to do) instead of stepping up to the plate to do such a trivial thing on his own.
To be clear, this is just an example (which has actually happened) and other similar “not my job” thinking/actions have shown up repeatedly. Many years ago we worked together on a small team and he was not like this at that time.
I think it’s more that you’re not allowed to do those things in a larger company. People will feel like you are ursurping their area of responsibility, or you’ll even be completely unable to make it.
I used to run support groups to tackle the rampant helplessness I saw around me.
The way it worked is that the person would describe their situation and then everyone else would ask questions.
The format was:
1. understand the facts, and understand if they are actually true
2. explore options, even crazy ones
3. pick an action to take and commit to it
For the questioners, the most important rule was you can't give advice. The person who is asking for help needs own it. They provide all the answers and decide on their own action. That action could be anything from talking to your manager, to setting boundaries, to looking for another job.
It worked amazingly well, many people who started off feeling stuck realized they had the ability to find a better path. Even just making a conscious decision to do nothing and being okay with the consequences of that is empowering and insightful.
I used to be completely helpless, learned from childhood. I finally realized that I'm not a child anymore and I have so many more options as an adult.
Happy to chat with you if you're feeling stuck, feel free to email me (see my profile).
Shameless plug: I stopped this to work on a related area - empowering people through team-building. The first session is free, you could do it in place of your next virtual happy hour! Check it out at https://risingteam.com.
The way it worked is that the person would describe their situation and then everyone else would ask questions.
The format was:
1. understand the facts, and understand if they are actually true
2. explore options, even crazy ones
3. pick an action to take and commit to it
For the questioners, the most important rule was you can't give advice. The person who is asking for help needs own it. They provide all the answers and decide on their own action. That action could be anything from talking to your manager, to setting boundaries, to looking for another job.
It worked amazingly well, many people who started off feeling stuck realized they had the ability to find a better path. Even just making a conscious decision to do nothing and being okay with the consequences of that is empowering and insightful.
I used to be completely helpless, learned from childhood. I finally realized that I'm not a child anymore and I have so many more options as an adult.
Happy to chat with you if you're feeling stuck, feel free to email me (see my profile).
Shameless plug: I stopped this to work on a related area - empowering people through team-building. The first session is free, you could do it in place of your next virtual happy hour! Check it out at https://risingteam.com.
> Complexity-related Learned Helplessness
I've gotten used to just accepting that every job I have will involve a code base that uses up way too much memory to run in a debugger and doesn't have (or allow for) unit tests.
I've gotten used to just accepting that every job I have will involve a code base that uses up way too much memory to run in a debugger and doesn't have (or allow for) unit tests.
There's a certain amount of "YOLO" fast-and-fragile development I'm willing to tolerate or negotiate. It's not like I have all the answers, and everything is a team effort.
But when I'm called upon to work entirely without tests, or with inadequate backups, or otherwise in ways that create a scenario where every keystroke I make holds imminent potential for disaster, I quit. I can do trapeze acrobatics if you let me use a net. If you insist that I must not use a net, I'm not willing to live with that constant stress.
But when I'm called upon to work entirely without tests, or with inadequate backups, or otherwise in ways that create a scenario where every keystroke I make holds imminent potential for disaster, I quit. I can do trapeze acrobatics if you let me use a net. If you insist that I must not use a net, I'm not willing to live with that constant stress.
> I quit
I did that early on. Then again, because the next place was the same. Then again, because the next place was the same. 10 places later, I finally realized they're all the same.
I did that early on. Then again, because the next place was the same. Then again, because the next place was the same. 10 places later, I finally realized they're all the same.
Huh? Like what specifically? All 4 cloud software places I’ve worked have had unit testing standards, cloud backups, even disaster recovery environments, etc. and none of these were mega corps. 50-1000 employees.
You can have those things in theory, but they need to adhere to certain quality standards to count on them in practice.
Parent said they visited 10 jobs that had no quality, I have 4 places that in practice try to have quality. So either they were hyperbolic, or has unrealistic standards, or bad luck, or bad job selection heuristics.
That's regrettable. Investing in reliability and redundancy is an optimization problem, resources are always limited, it's natural for there to be tension and I've grown accustomed to that. My experience has been varied, with some organizations investing more than others.
Maybe it's just me, but not having the net provides some level of "fun".
Most of the projects/companies/teams I've worked on didn't have tests, adequate backups or had scenarios where errant keystrokes blow up production but it's nothing that has been world-stopping and these jobs rarely involved anything were downtime equated to real harm (outside of the economics of the companies themselves, but usually nobody necessarily cared as long as things got fixed).
Companies that subscribe to YOLO-level development usually already _know_ that and with it, comes a certain understanding that shit will break, it's just minimizing the blast radius by learning/knowing the code bases and leaning on those with more experience.
Most of the projects/companies/teams I've worked on didn't have tests, adequate backups or had scenarios where errant keystrokes blow up production but it's nothing that has been world-stopping and these jobs rarely involved anything were downtime equated to real harm (outside of the economics of the companies themselves, but usually nobody necessarily cared as long as things got fixed).
Companies that subscribe to YOLO-level development usually already _know_ that and with it, comes a certain understanding that shit will break, it's just minimizing the blast radius by learning/knowing the code bases and leaning on those with more experience.
I will never understand this position. My life are already chaotic enough without factoring in the work-related part of it, it makes no sense to me why someone would actively invite chaos like the one you're describing. I'm much more excited by solving interesting problems or creating useful things than by having my work day interrupted by a production error that needs to be fixed immediately.
You're not necessarily inviting chaos, you're making a cost/benefit analysis of moving faster or safer. I prefer to move faster.
Moving safer takes up a lot of, imo, unnecessary time with dubious payoffs. I'd personally rather live on the side of YOLO and ship a lot then hem/haw over getting certain % unit test coverage (and the maintenance costs that come with that in the face of uncertain requirements).
Moving safer takes up a lot of, imo, unnecessary time with dubious payoffs. I'd personally rather live on the side of YOLO and ship a lot then hem/haw over getting certain % unit test coverage (and the maintenance costs that come with that in the face of uncertain requirements).
[deleted]
The fun should come from solving fundamentally difficult problems, not from someone fucking up a CRUD response because they didn't properly test their filter parsing.
Lots of companies have no "fundamentally difficult problems" when it comes to their software. Lots of software developers don't have any of these problems to solve.
(I'm also against this "YOLO because it's fun" attitude, though.)
(I'm also against this "YOLO because it's fun" attitude, though.)
If this is the case you still have a fundamentally difficult problem, it's just become something like "how can we keep our velocity while also improving test coverage / reliability?" and the solution is usually some new internal tooling.
> it's nothing that has been world-stopping
Well then you're talking about a completely different scenario than I am.
Well then you're talking about a completely different scenario than I am.
I could definitely YOLO out a change when I worked on code used by 100 people. When it was used by 10000, an error every now and then is forgivable. Now, if I make an error in a deployment I could stop 10,000,000 from using my systems. Similar outages have made newspapers. Even then, compared to FAANG scale I'm still playing with toys. The fastly outage was top story on the BBC while it was happening.
I would hope that world-stopping software (banks, critical infrastructure, healthcare, etc) have tests and all the things to prevent catastrophic error. For the rest of us, breaking things here and there in the name of moving faster is (anecdotally) more fun.
I think there exists something in between world-stopping as in healthcare (I want way more than just 'tests' there) and Facebook level (i.e. nobody really cares even though a lot of people will probably scream in "agony" when things don't work).
To maybe put it into examples:
To maybe put it into examples:
World stopping: Traffic collision avoidance system (TCAS) has a bug that lets planes crash into each other.
Don't care: Not enough vegetarian meals made it onto the plane because of a bug in the meal reservation software.
In between: Thousands upon thousands of people get stranded in some airport for 3 days because of a bug in the route planning software.
> healthcare
most of healthcare software (that i have worked with) is really, really bad.
The fact that it mostly has no tests, or if it has tests they mainly test just the happy route.
They usually have lots of documentation, but it's not comprehensive and mostly just describes the happy paths, so when thing break you are out of luck. Also can't google anything since this are all closed source enterprise type software, so you are at the mercy of their support, that often also sucks.
And if you have to integrate with it, you often can't get the documentation, or tools to do it unless you are working for one of their VARs. And the VAR's are often even worse.
If you pick random 10 high school kids, get them into some code academy for a 6 months, and then have them develop an application, the ones that finish will probably be better than majority of health care software out there (for simple reason that they won't know enough to really screw up as bad.).
The only saving grace is, that its often written in Java so you can find workarounds in runtime (or inject stuff) if you really need to.
most of healthcare software (that i have worked with) is really, really bad.
The fact that it mostly has no tests, or if it has tests they mainly test just the happy route.
They usually have lots of documentation, but it's not comprehensive and mostly just describes the happy paths, so when thing break you are out of luck. Also can't google anything since this are all closed source enterprise type software, so you are at the mercy of their support, that often also sucks.
And if you have to integrate with it, you often can't get the documentation, or tools to do it unless you are working for one of their VARs. And the VAR's are often even worse.
If you pick random 10 high school kids, get them into some code academy for a 6 months, and then have them develop an application, the ones that finish will probably be better than majority of health care software out there (for simple reason that they won't know enough to really screw up as bad.).
The only saving grace is, that its often written in Java so you can find workarounds in runtime (or inject stuff) if you really need to.
YOLO-level development is a slippery slope... We had a case where 2 backends from 2 teams needed to interoperate.. to support a much talked-up UI/functional feature that was supposed to make our product standout from rivals. What was implemented was a rube-goldberg like contraption on the backend, where I dreaded being on-call as the talked-up UI feature was rolled out to more and more customers.
Oh yeah, we had unit tests all right, fragility didn't prevent that since you couldn't commit otherwise.
So there was no safety net. Just a constant dread of customers actually using the feature. So "fast-and-fragile development" resulted in a worst-case scenario as far as engineers were concerned.
The author says : "Their reports won’t tell the manager that they are having a terrible on-call experience". This was exactly what happened to me. And I quit one day, this was a big factor, two weeks out of a month (in my rotation) I'd be on the hook for supporting this morass of problems.
Oh yeah, we had unit tests all right, fragility didn't prevent that since you couldn't commit otherwise.
So there was no safety net. Just a constant dread of customers actually using the feature. So "fast-and-fragile development" resulted in a worst-case scenario as far as engineers were concerned.
The author says : "Their reports won’t tell the manager that they are having a terrible on-call experience". This was exactly what happened to me. And I quit one day, this was a big factor, two weeks out of a month (in my rotation) I'd be on the hook for supporting this morass of problems.
This is one of the things I like most about my current job, I feel like we are all able to fix stuff, and everyone is open to questioning what we are doing and how we do it. If it's not working, or there's a better way then we change. People don't take it personally, everyone seems open to suggestions.
I've been here for 2 years now, and while there's still things to be improved in both the code base and the way we do things, we've been making steady improvement over all that time, all while managing to deliver on new features. Even at times when the progress might slow, just knowing that there is some improvements happening, and more will happen when resources allow, creates a whole different mindset from previous places where I've just given up and left.
I've been here for 2 years now, and while there's still things to be improved in both the code base and the way we do things, we've been making steady improvement over all that time, all while managing to deliver on new features. Even at times when the progress might slow, just knowing that there is some improvements happening, and more will happen when resources allow, creates a whole different mindset from previous places where I've just given up and left.
> Another employee or their manager teaches them it's a normal situation at this company
I have one particularly memorable situation where I tried to coach someone out of this. Sometimes the New Guy is the only one who can change something at BigCo, and the moment you convince them that fighting the bureaucracy is too hard, you've lost one of the few assets that person has.
As the processes get more complex, the time between onboarding and being able to tackle projects as an IC grows and grows. That outsider perspective may be one of the few things they're good for. That outsider perspective may be the only thing that ever reduces that gap.
I can't help but feel like people caught in that uncanny valley are damaged by being there, and one of the only ways I often have to improve morale is to point out how they're doing things - important things - that the Curse of Knowledge keeps me or my peers from doing well.
I have one particularly memorable situation where I tried to coach someone out of this. Sometimes the New Guy is the only one who can change something at BigCo, and the moment you convince them that fighting the bureaucracy is too hard, you've lost one of the few assets that person has.
As the processes get more complex, the time between onboarding and being able to tackle projects as an IC grows and grows. That outsider perspective may be one of the few things they're good for. That outsider perspective may be the only thing that ever reduces that gap.
I can't help but feel like people caught in that uncanny valley are damaged by being there, and one of the only ways I often have to improve morale is to point out how they're doing things - important things - that the Curse of Knowledge keeps me or my peers from doing well.
Oh man, this is so irritating. I've been pushing for changes for years, and told, "No, it can't be done!" Then the New Guy comes in, recommends the same damn thing, and suddenly management is like, "Hey! That's a good idea!" I mean, I'll take the win since the change finally happens, but what a great way to demotivate your existing staff.
Near as I can tell, this has to do with how many different people a problem has been reported by. Either strict numbers or turns of phrase finally changes the mind of the holdouts.
I started sending people to ask the person who has been vetoing the change about why we can’t have nice things. Sometimes it works.
I started sending people to ask the person who has been vetoing the change about why we can’t have nice things. Sometimes it works.
I have a completely different perspective: processes in a company reflect the people and the relationships between them. It is on these levels you must operate in order to bring change. As an engineer you have very little power to do so. So leaving and finding a different team is often the right choice when you are unhappy. From my experience staying and fighting an uphill battle is an exercise in futility which will bring more unhappiness to you and the people you are trying to change.
A couple of thoughts on this:
* A powerful tool against Pattern 2 (Complexity-related Learned Helplessness) is just cataloguing and quantifying things that cause wasted effort, i.e. a waste snake. Make a channel for it and encourage people to add items like "spent 4 hours reinstalling docker after corporate pushed an update" or "18 people X 2 hrs when the vpn was down" or whatever. Some of them may not be solvable, but quantifying the impact is the first step towards prioritizing a solution (or realizing that the solution isn't worth the effort).
* IME the solution to learned helplessness, somewhat buried at the end, is team autonomy. Self-managing or autonomous teams are the exact opposite of learned helplessness. It's nice to imagine your boss solving your problems with heroics ("Hey guys, I joined the on-call rotation this week and discovered it's terrible so I'm getting rid of it, hooray!") but waiting for that to happen sounds like more helplessness. A more realistic solution is that you solve this the same way you solve your software problems: figure out who the stakeholders are, figure out what problem they have, and devise a better solution.
* A powerful tool against Pattern 2 (Complexity-related Learned Helplessness) is just cataloguing and quantifying things that cause wasted effort, i.e. a waste snake. Make a channel for it and encourage people to add items like "spent 4 hours reinstalling docker after corporate pushed an update" or "18 people X 2 hrs when the vpn was down" or whatever. Some of them may not be solvable, but quantifying the impact is the first step towards prioritizing a solution (or realizing that the solution isn't worth the effort).
* IME the solution to learned helplessness, somewhat buried at the end, is team autonomy. Self-managing or autonomous teams are the exact opposite of learned helplessness. It's nice to imagine your boss solving your problems with heroics ("Hey guys, I joined the on-call rotation this week and discovered it's terrible so I'm getting rid of it, hooray!") but waiting for that to happen sounds like more helplessness. A more realistic solution is that you solve this the same way you solve your software problems: figure out who the stakeholders are, figure out what problem they have, and devise a better solution.
"Learned helplessness" as a psychological phenomenon relates to situations where the individual assumes they are powerless to change certain things, but they actually are not.
A lot of the time, when one is an individual contributor in an engineering team, that is not the case. Rather it is the case that you really are powerless.
That's an important distinction to make. Don't assume that your struggle is with internal psychological forces, when your struggle really is with your teammates and the team's power structure.
A lot of the time, when one is an individual contributor in an engineering team, that is not the case. Rather it is the case that you really are powerless.
That's an important distinction to make. Don't assume that your struggle is with internal psychological forces, when your struggle really is with your teammates and the team's power structure.
I agree, and I think until certain point you need to assume that you'll be powerless in some aspects. You cannot change it all and you need to accept certain amount of being powerless.
The key is to find the proper amount...
The key is to find the proper amount...
It is expensive to know what is going on. In military terms this is called 'Fog of War'. Front line managers quickly learn to manage up - know what their boss expects and present things in an appealing way. Same for middle managers and senior managers. They all hide the pain that their teams undergo. Senior managers and executives can have people write reports that show progress on some metric, and then they manage to that metric (or bucket of metrics in more enlightened organizations), without ever knowing, thinking about, or caring about how much it takes to get things done.
As a dev, it's my job to make someone else's job easier. Whose job is it to make my job easier?
As a dev, it's my job to make someone else's job easier. Whose job is it to make my job easier?
There's a side to this that I feel wasn't explored in the article: some people want to be put in such a situation, because it makes them appear busy and thus not layoff material. They have bills, mortgages, private schools to finance and prefer the status quo.
Moreover there are people who want their team to be like that, because they enjoy not having their decisions second guessed by any of the team members. These people in turn suck up all the autonomy that the team is given and produce the NIHest examples of architectural astronautics possible.
I've been in several projects where this was the case. The usual scenario starts off with me barging in and crying "cease this madness!". Replies range from indifferent through apologetic to hostile.
Eventually, after shoehorning too many refactors into my assigned tasks I either get fired or quit.
I now actively avoid such places because I'm simply not cut for such work.
Moreover there are people who want their team to be like that, because they enjoy not having their decisions second guessed by any of the team members. These people in turn suck up all the autonomy that the team is given and produce the NIHest examples of architectural astronautics possible.
I've been in several projects where this was the case. The usual scenario starts off with me barging in and crying "cease this madness!". Replies range from indifferent through apologetic to hostile.
Eventually, after shoehorning too many refactors into my assigned tasks I either get fired or quit.
I now actively avoid such places because I'm simply not cut for such work.
Related:
Jeff Bezos on Learned Helplessness (2005): https://www.youtube-nocookie.com/embed/WhnDvvNS8zQ
Paul Graham on Schlep Blindness (2012): http://paulgraham.com/schlep.html
Jeff Bezos on Learned Helplessness (2005): https://www.youtube-nocookie.com/embed/WhnDvvNS8zQ
Paul Graham on Schlep Blindness (2012): http://paulgraham.com/schlep.html
What I’ve noticed in this cycle is that as the turn over continues, the employee skill level will trend downwards. Essentially in a sort of ironic-Darwinian-gone-wrong, the system selects for those that will stay and/or don’t care or are desperate enough for work (because of geographical allegiance of low hireability, or just low initiative to make a change).
I fear my current workplace that was so cool years ago is showing these signs.
I fear my current workplace that was so cool years ago is showing these signs.
This is definitely happening. A engineer can't really work on anything truly complicated if they hop around every 2-3 years. These types of engineers gain cursory knowledge of topics but never become experts.
That’s not exactly what parent was saying. They were saying, afaict, if an org is struggling with retention then each successive generation will be worse at affecting change. They aren’t saying anything about individuals themselves who switch jobs often.
But about your point I think it can depend greatly. Personally I switched jobs many times but have so far kept in the same field and my knowledge has compounded and I have never dealt behind. In fact sometimes I feel the opposite, some coworkers who have less diverse experience have less to offer.
But about your point I think it can depend greatly. Personally I switched jobs many times but have so far kept in the same field and my knowledge has compounded and I have never dealt behind. In fact sometimes I feel the opposite, some coworkers who have less diverse experience have less to offer.
2-3 years is enough to master a certain niche of technology fairly well if you want to.
I see this happening in places too. I have seen it described as the Dead Sea effect.
http://brucefwebster.com/2008/04/11/the-wetware-crisis-the-d...
http://brucefwebster.com/2008/04/11/the-wetware-crisis-the-d...
> Notice subtle changes in behavior: the most useful hint here is when someone who used to be very vocal about an issue stops complaining.
This point resonated greatly with my own observations, both of myself and of other colleagues with each development team I've been part of.
I think another challenge is fostering a culture where everybody genuinely cares about the output that's delivered. Not everybody will. Some people are comfortable moving tickets from triage through to a resolved state day in, day out without caring whether this process or the output can be improved.
This point resonated greatly with my own observations, both of myself and of other colleagues with each development team I've been part of.
I think another challenge is fostering a culture where everybody genuinely cares about the output that's delivered. Not everybody will. Some people are comfortable moving tickets from triage through to a resolved state day in, day out without caring whether this process or the output can be improved.
Big teams at big companies are the pits.
Find a startup with under 10 people and no technical debt. You might not bank as much cash, make sure to get a large slug of equity on the off chance the company does well and you are then set.
You don’t have to put up with this garbage. Otherwise why are you reading this on YCombinator?
How have people found balance in these situations?
The examples mentioned are painfully real and accurate but it's possible to swing too far in the other direction too.
Too much autonomy, feedback collection, etc. can also bog down the entire team. Not all feedback is good or well-reasoned. It's easy to get stuck in a spiral of short-sighted reactive changes and the second-guessing of every past decision.
-
It takes a lot of focused effort to both empower individuals and educate them (without indoctrinating them into learned helplessness?).
The only lead I have here is to document decisions in ADR-style[1] docs so newcomers can see what else was considered and not chosen.
I'd love to hear any other success stories or tips.
--
[1]https://adr.github.io/
The examples mentioned are painfully real and accurate but it's possible to swing too far in the other direction too.
Too much autonomy, feedback collection, etc. can also bog down the entire team. Not all feedback is good or well-reasoned. It's easy to get stuck in a spiral of short-sighted reactive changes and the second-guessing of every past decision.
-
It takes a lot of focused effort to both empower individuals and educate them (without indoctrinating them into learned helplessness?).
The only lead I have here is to document decisions in ADR-style[1] docs so newcomers can see what else was considered and not chosen.
I'd love to hear any other success stories or tips.
--
[1]https://adr.github.io/
Has anyone actually overcome any of these bad situations? Or has there been a situation where “So and so quit because X, we need to change X.”
You can’t change an org from the outside, but it’s also difficult to change it from the inside.
You can’t change an org from the outside, but it’s also difficult to change it from the inside.
You can change to an outside org though. Rational self interest.
Yes. It's really painful.
Imagine a 5 year old code base that doesn't even have technical debt: it's a manifestation of the technical debt itself. It's hundreds of thousands of lines of code, mostly with "emergent behavior". You can't release without spending days on clicking through the RC system manually to check what's broken. Also, your product is extremely complex because apparently the barrier to entry in your field of business is very high.
And you have no users.
Then suddenly users arrive.
What it takes is 1) management buy-in (they should be desperate enough) 2) engineers who are not willing to make compromises.
Also, as usual in crisis management, people who successfully manage crises are generally ousted after the crisis is over because the personality needed to solve crises is not compatible with peacetime modus operandi.
Imagine a 5 year old code base that doesn't even have technical debt: it's a manifestation of the technical debt itself. It's hundreds of thousands of lines of code, mostly with "emergent behavior". You can't release without spending days on clicking through the RC system manually to check what's broken. Also, your product is extremely complex because apparently the barrier to entry in your field of business is very high.
And you have no users.
Then suddenly users arrive.
What it takes is 1) management buy-in (they should be desperate enough) 2) engineers who are not willing to make compromises.
Also, as usual in crisis management, people who successfully manage crises are generally ousted after the crisis is over because the personality needed to solve crises is not compatible with peacetime modus operandi.
The most positive change I've seen happen in an organization is through skunkworks projects, i.e. work you don't tell your boss about.
Too much technical debt, but the boss doesn't want to refactor? Refactor any code you were touching anyway.
Build process too complicated and brittle? Write build scripts, add error checking, and make other necessary changes to fix the problem.
My past bosses never asked "Where did you find the time to do that?" even if they even noticed that anything happened at all.
At one company I worked, we had no build server. The boss didn't see the need for it so he never approved it, but he also didn't reject it once we set it up.
Any organizational change that requires other to change their behavior or their opinions? It is very difficult to change. The only success I've had is by mentioning an idea multiple times and eventually my boss started thinking it was his idea.
Too much technical debt, but the boss doesn't want to refactor? Refactor any code you were touching anyway.
Build process too complicated and brittle? Write build scripts, add error checking, and make other necessary changes to fix the problem.
My past bosses never asked "Where did you find the time to do that?" even if they even noticed that anything happened at all.
At one company I worked, we had no build server. The boss didn't see the need for it so he never approved it, but he also didn't reject it once we set it up.
Any organizational change that requires other to change their behavior or their opinions? It is very difficult to change. The only success I've had is by mentioning an idea multiple times and eventually my boss started thinking it was his idea.
Funny enough, at my last job the bosses never listened to anything their employees suggested. Whenever an outsider (journalist, user, family member, or outside consultant) suggested the same thing as the employees suggested, only then did the bosses consider it.
I don't know how common this is in other places, but that organization could only change through external, not internal, influences.
I don't know how common this is in other places, but that organization could only change through external, not internal, influences.
I remember reading something about it in the gaming industry; from memory it was written by a contractor brought into 3D games on tight deadlines to unstick them. And what they did was go in, ask the employees what was wrong, the employees told him everything that was wrong and what would fix it. They'd been trying to tell anyone who would listen, for months, and he was the first person who would listen. Then he turned around and told everything that was wrong to management, who were thrilled that the contractor could come in and so quickly tell them what was wrong and how to fix it, and give him the resources to do it because the deadline was imminent. The employees could lean on the consultant to cut through bureaucracy, and the consultant (who had skills) could sort out some intricate game engine performance problems, and the whole thing moved on. He did that almost routinely moving round big game companies from project to project.
I might be misremembering, but I can't find it in the HN search history.
Like an explanation given for how companies will let key employees quit, then excitedly pay more for new hires. New employees (or contractors) are all marketing about how great they are. Existing employees are all too obviously mere mortals with plenty of flaws. Of course getting rid of the ordinary employee and getting Superman looks like a good idea.
Manager spends time listening to employees complain: losing proposition, boring, time consuming, looks like doing nothing. Manager brings in amazing consultant who helps: manager looks good, saves effort, has a claim to need a bigger team and budget in future because evidence shows they just needed some extra help.
I might be misremembering, but I can't find it in the HN search history.
Like an explanation given for how companies will let key employees quit, then excitedly pay more for new hires. New employees (or contractors) are all marketing about how great they are. Existing employees are all too obviously mere mortals with plenty of flaws. Of course getting rid of the ordinary employee and getting Superman looks like a good idea.
Manager spends time listening to employees complain: losing proposition, boring, time consuming, looks like doing nothing. Manager brings in amazing consultant who helps: manager looks good, saves effort, has a claim to need a bigger team and budget in future because evidence shows they just needed some extra help.
I’m actively working to change this as a CTO in a small digital agency. Can’t say that I’ve completely solved it but I’ve started challenging my peers when they use the words hope, belief and especially magic to describe what they’re up against. We’re professionals not hobbyists. Our codebases should get easier not harder to work with over time.
We have a sister organization which is larger and with their own CTO. I’ve tried to boil the ocean and inspire change across both companies, it was hard, for now I'm just focusing on trying to inoculate the new hires against accepting any sub-optimal bureaucracy. Though I see that need to find some way to train them to be so open and confident that they can start to challenge processes when they need to.
I have a lot of hope in the new hires though. Culture is hard to change but definitely doable. It’s all about who you hire, fire, praise and incentivize.
We have a sister organization which is larger and with their own CTO. I’ve tried to boil the ocean and inspire change across both companies, it was hard, for now I'm just focusing on trying to inoculate the new hires against accepting any sub-optimal bureaucracy. Though I see that need to find some way to train them to be so open and confident that they can start to challenge processes when they need to.
I have a lot of hope in the new hires though. Culture is hard to change but definitely doable. It’s all about who you hire, fire, praise and incentivize.
> In a word, we could call this resignation by a thousand cuts, which makes it unexpected for managers.
Eh eh eh, this feels like reading the Phoenix Project all over again: "are you me!?"
My first job out of university was a job at a BigCo, right in the middle of a Death March. Nobody at the beginning of the project remained. Most people who "finished it" were new comers like me or "team leads"(tm).
The three years I was there my teammates and I tried to push numerous changes and much needed improvements. Every time we had push back, especially by the team leads who literally shouted everyone down; I now have a profound hate for people with loud soprano voice by I digress.
I tried my best to raise up problems, find their roots and solutions. I mean, I couldn't be subtle that I wasn't satisfied with what I have seen.
And yet when I quit, my colleagues all could point out reasons of why I quit and yet none expected that I would leave.
As the article pointed out, you should quit early and make sure your reasons are known; your old colleagues will owe you and this is pretty much the only way they can get the message.
Eh eh eh, this feels like reading the Phoenix Project all over again: "are you me!?"
My first job out of university was a job at a BigCo, right in the middle of a Death March. Nobody at the beginning of the project remained. Most people who "finished it" were new comers like me or "team leads"(tm).
The three years I was there my teammates and I tried to push numerous changes and much needed improvements. Every time we had push back, especially by the team leads who literally shouted everyone down; I now have a profound hate for people with loud soprano voice by I digress.
I tried my best to raise up problems, find their roots and solutions. I mean, I couldn't be subtle that I wasn't satisfied with what I have seen.
And yet when I quit, my colleagues all could point out reasons of why I quit and yet none expected that I would leave.
As the article pointed out, you should quit early and make sure your reasons are known; your old colleagues will owe you and this is pretty much the only way they can get the message.
I have found that the biggest thing that can be done is to ask your people over and over again. And then also genuinely care about the situation and make it your goal to communicate to others that the problem exists.
Unfortunately people are very conflict averse and will avoid giving that sort of feedback if it means an uncomfortable conversation (particularly if whoever came up with the process is still employed). Even anonymous surveys are not really that useful because the most useful criticism stays exclusively inside peoples' heads.
As a manager you have to have really good EQ to be able to read this sort of stuff implicitly from your employees because it's usually not coming out explicitly.
As a manager you have to have really good EQ to be able to read this sort of stuff implicitly from your employees because it's usually not coming out explicitly.
> people are very conflict averse and will avoid giving that sort of feedback
I'm not (that) conflict averse - I'm perfectly happy to suggest, when asked, that we could deliver better results faster if we could spend some time fixing some non-customer facing problems, had better tools, and had a testing environment that was closer to the production environment. I have no trouble saying those things, but I've never seen any action taken on any of them.
I'm not (that) conflict averse - I'm perfectly happy to suggest, when asked, that we could deliver better results faster if we could spend some time fixing some non-customer facing problems, had better tools, and had a testing environment that was closer to the production environment. I have no trouble saying those things, but I've never seen any action taken on any of them.
Indeed. The learned helplessness comes in after you try make improvements over and over again, and still no action is taken. You assume, usually correctly, that you're helpless to effect change so you give up.
The main driver of learned helplessness is those who refuse to listen to feedback or act on feedback.
The main driver of learned helplessness is those who refuse to listen to feedback or act on feedback.
So true!
Both "This is just how the company is, it won't change" and "We survived thus far, we'll get through" are the unofficial mantras of the company I work for. Even though many in the middle management see the problems, these mantras are repeated whenever one speaks up against silly processes. And boy, do we have some silly processes.
It's not helped by the fact that the second is actually a mantra often quoted as the official motto of the neighboring metropolis, as if resistance to betterment is something to be proud of ("et hät noch immer jot jejange", for those in the know of both German and the dialect).
Though I suppose it might have also been the motto of the dinosaurs before the big impact.
There is "learned helplessness" in a way also in management, because they in turn seem to have learned that raising any issues with the higher management only leads to being branded as, if not a source of discord or trouble, then at the least as someone who requires a higher manager to deal with something, as opposed to those not requiring a higher manager to deal with something. The later clearly being preferred by the upper management.
Both "This is just how the company is, it won't change" and "We survived thus far, we'll get through" are the unofficial mantras of the company I work for. Even though many in the middle management see the problems, these mantras are repeated whenever one speaks up against silly processes. And boy, do we have some silly processes.
It's not helped by the fact that the second is actually a mantra often quoted as the official motto of the neighboring metropolis, as if resistance to betterment is something to be proud of ("et hät noch immer jot jejange", for those in the know of both German and the dialect).
Though I suppose it might have also been the motto of the dinosaurs before the big impact.
There is "learned helplessness" in a way also in management, because they in turn seem to have learned that raising any issues with the higher management only leads to being branded as, if not a source of discord or trouble, then at the least as someone who requires a higher manager to deal with something, as opposed to those not requiring a higher manager to deal with something. The later clearly being preferred by the upper management.
My example of horrible technical debt goes like so:
I (not that long ago) worked at a company that kept all their data in MySQL. Sounds fine. The bad part was all their user-facing stuff was written in Microsoft Access.
This didn't make any sense and was the main thing that prevented them from moving to a web-based or API-based interface. They had written some code here and there that would cobble a few things together but it wasn't the right way to do it.
During a discussion, I asked why there were 2 MySQL servers. They held different databases, but it would have made more sense to have them both on one server. I assumed that at some point in the company history, the data got too big to fit on one server, so they split it up, and that ended up being true. And ever since they had some misguided sense of safety having two different databases, even though if you were missing the other, all functionality would break.
At the current time, the databases weren't large enough or accessed enough to require being on separate hardware. It seemed simple enough to just import one into the other and then get on with modernizing the rest of the code.
"We would have to re-write the entire codebase since it's based on having two servers"
Access lets you basically do joins etc on two separate databases. So this was used heavily. Whenever they tried to duplicate this functionality in PHP or whatever they would try, it turned into a nightmare.
I would imagine if they had spent another $10k on hardware back when they first had this problem they would have saved a lot of future troubles. Imagine running legacy versions of Visual Basic, developers needing Windows XP VM's, end users needing two copies of Microsoft Access installed...
I (not that long ago) worked at a company that kept all their data in MySQL. Sounds fine. The bad part was all their user-facing stuff was written in Microsoft Access.
This didn't make any sense and was the main thing that prevented them from moving to a web-based or API-based interface. They had written some code here and there that would cobble a few things together but it wasn't the right way to do it.
During a discussion, I asked why there were 2 MySQL servers. They held different databases, but it would have made more sense to have them both on one server. I assumed that at some point in the company history, the data got too big to fit on one server, so they split it up, and that ended up being true. And ever since they had some misguided sense of safety having two different databases, even though if you were missing the other, all functionality would break.
At the current time, the databases weren't large enough or accessed enough to require being on separate hardware. It seemed simple enough to just import one into the other and then get on with modernizing the rest of the code.
"We would have to re-write the entire codebase since it's based on having two servers"
Access lets you basically do joins etc on two separate databases. So this was used heavily. Whenever they tried to duplicate this functionality in PHP or whatever they would try, it turned into a nightmare.
I would imagine if they had spent another $10k on hardware back when they first had this problem they would have saved a lot of future troubles. Imagine running legacy versions of Visual Basic, developers needing Windows XP VM's, end users needing two copies of Microsoft Access installed...
This is a mindset issue not a technical one.
At the end of the day, if you're a developer, you don't get to call the shots in terms of what issues get fixed when.
That's up to the lead dev/product owner/BA.
You can flag issues and propose solutions, sure.
If you think the company isn't worth the trouble then quit and get another job, there are plenty of tech opportunities out there, one's that are no doubt more dev focused.
At the end of the day, if you're a developer, you don't get to call the shots in terms of what issues get fixed when.
That's up to the lead dev/product owner/BA.
You can flag issues and propose solutions, sure.
If you think the company isn't worth the trouble then quit and get another job, there are plenty of tech opportunities out there, one's that are no doubt more dev focused.
We need to add Team Politics as well. We often spend more time in right articulation to keep it diplomatically correct and communicate a risk/problem/concern - if it involves multiple teams. People often do not accept this communication and it goes for several iterations before they all arrive at work breakdown. Not sure there is a way to measure these churns.
My learned helplessness story - Another engineer fancies himself as the gatekeeper of our codebase. Nitpicks on PRs all the time. Always tries to change something.
This time around, an engineer produced a fantastic design. But this gatekeeper engineer had something else on his mind. The gatekeeper did not even communicate his thought process but continued asking ridiculous questions.
Later we found that the gatekeeper engineer wanted to change the entire design to his own, even if his own design had major flaws. Wasted a whole quarter just because the gatekeeper didn't seem to want to "concede". He would much rather have engineers make bad designs than accepting his own missteps. The author of the design fought vehemently at the expense of their own time and energy. To what end?
Other engineers on the team have forever stopped producing design docs. Learned helplessness.
This time around, an engineer produced a fantastic design. But this gatekeeper engineer had something else on his mind. The gatekeeper did not even communicate his thought process but continued asking ridiculous questions.
Later we found that the gatekeeper engineer wanted to change the entire design to his own, even if his own design had major flaws. Wasted a whole quarter just because the gatekeeper didn't seem to want to "concede". He would much rather have engineers make bad designs than accepting his own missteps. The author of the design fought vehemently at the expense of their own time and energy. To what end?
Other engineers on the team have forever stopped producing design docs. Learned helplessness.
did you let their manager know? Especially valid with data that shows "look, before, ppl tried to follow process X, and now, as you can see, nobody even tries." Data, data, data. Management likes data.
No. I am also in learned helplessness mode. The manager doesn't appear to get that the gatekeeper engineer is stubborn.
I have nothing to gain by getting caught in the crossfire.
I would just quit after biding my time.
I have nothing to gain by getting caught in the crossfire.
I would just quit after biding my time.
The biggest cause of this in my experience is too much codified process, not too little. This is cause #1 in this article, which is vindicating.
Adding process is a delicate balance and most large companies err on too much of it.
Every checklist you add, every new approval, every sheet that needs some rows filled out, every brief - all of them add friction.
New projects are inherently speculative, and most things don't move the company's bottom line. The power law that says 10% of work get 90% of results is true. Friction means whoever is doing the work is just a little less likely to propose a bold test, to push on a tweak they notice, because they know it will become an ordeal.
Obviously you can't be 100% wild west and some process is needed. But you have to be very very careful when adding it.
Adding process is a delicate balance and most large companies err on too much of it.
Every checklist you add, every new approval, every sheet that needs some rows filled out, every brief - all of them add friction.
New projects are inherently speculative, and most things don't move the company's bottom line. The power law that says 10% of work get 90% of results is true. Friction means whoever is doing the work is just a little less likely to propose a bold test, to push on a tweak they notice, because they know it will become an ordeal.
Obviously you can't be 100% wild west and some process is needed. But you have to be very very careful when adding it.
There is another, opposite force that manifests in humans arising from almost the same inputs, they will often happily tolerate 99% friction, as long as they actually get at least those few 1% of wins.
Even with a ton of nice safe ossified process to make the owners feel safe, if you just make it a policy to give them a win once in a while, they'll live off of that and not even be all that unhappy.
The 99% friction is felt less like a waste and more like a high bar.
I think in a lot of places, they just don't allow even that calculated IV drip of dignity and purpose.
Even with a ton of nice safe ossified process to make the owners feel safe, if you just make it a policy to give them a win once in a while, they'll live off of that and not even be all that unhappy.
The 99% friction is felt less like a waste and more like a high bar.
I think in a lot of places, they just don't allow even that calculated IV drip of dignity and purpose.
Google is this for almost every internal tool and process. The only difference is most never reach step 5 (quitting), so imagine how toxic that culture is.
i think it happens when there is no hands off approach. the way i heard it before was that some people learn from pressure and others learn from the hands off approach. i know i do a way better job when people tell me what to do without the assumption that i am completely new to something, other people have the opposite learning style, they want the baby steps. but one major issue is when a manager mistakes ones learning style and that just creates frustration for everyone
Ever tried. Ever failed. No matter. Try Again. Eeach time a little less, until quitting seems better.
My teammates create e-mail filters to send useless daily e-mail reports to the trash. I try to find out who controls the e-mail process, or the e-mail address, or something, so I can fix it... but I spend lots of time and it ends in vain. I couldn't find who "owns" the process, I couldn't find who controls the mailing list, and I couldn't get anyone to give me permission to change it even if I knew how.
Clearly the problem isn't just having the skill or permission to change something, it's also the friction involved in figuring out how the hell to do it. How do you lower friction? Documenting things, making it easy to find things, making it easy to get access to things. If you can come up with an internal system that combines all of that, you have a one-stop shop for fixing high-friction problems.
I think Wikis are highly underrated. They seem to encapsulate all those things. Anyone can edit (or revert edits), anyone can access it, anyone can find it (eventually). Somehow we need to tie all the rest of an organization into a Wiki.
Clearly the problem isn't just having the skill or permission to change something, it's also the friction involved in figuring out how the hell to do it. How do you lower friction? Documenting things, making it easy to find things, making it easy to get access to things. If you can come up with an internal system that combines all of that, you have a one-stop shop for fixing high-friction problems.
I think Wikis are highly underrated. They seem to encapsulate all those things. Anyone can edit (or revert edits), anyone can access it, anyone can find it (eventually). Somehow we need to tie all the rest of an organization into a Wiki.
The tech lead started out by complaining about how he has to do an untested unplanned release today because another team made some urgent changes. He's the only person who knows how to release it. The other team didn't communicate until today that a release is necessary.
We've done two other releases in the past month and both required a day of troubleshooting to fix issues.
Both of us have been working at this company for about 3 years and we both have over a decade of experience in software development.
When he finished complaining, I started asking questions and making suggestions about how we can improve things. - Push back on the team that needs these urgent changes. Let them learn to do the release. - Deny the release since they didn't communicate earlier. - Improve the release process.
Everything I suggested was just flatly denied as impossible. - The other team doesn't know how to do the release. - He wants to be a "team player" so he can't deny the release. - Project managers will never allocate time to improve the release process.
I feel strange because I've seen this same thing for my whole career and I still try fight for what's right when others appear to moan and carry on.
However, my experience tells me that bringing this stuff to my manager is even worse. My manager doesn't know anything about the code, my project, or the release project. He may assume it's complaining for the sake of complaining. It has been used as ammunition in reviews against me.
Learned helplessness sucks and I wish I could do more. I don't think either of the suggestions in the article are feasible for many ICs. Teams are ambivalent to making improvements, and retrospectives carry very little weight. Managers are above the fray and won't be held responsible for by people below them.