Ask HN: Is all code patchy and bad in FANG?
28 comments
It doesn’t matter if you’re a big tech company or a small IT organization providing custom applications. The code everywhere has problems, and it’s never a good idea to complain about it.
In big tech, organizations often try to move quickly and release features… as part of whatever goals their organization promised to deliver. Expectations are unrealistic or only achievable if you really stretch, and everyone wants to deliver because that’s how you get promoted.
This incentive structure leads to an abundance of tech debt. And tech debt is hard to quantitatively measure. Eventually it slows down your pace of innovation, as all the little things that were accepted along the way pile up.
I don’t know of a solution to the problem other than looping your leadership in on what are the barriers to productivity - whether it’s poorly architected code, fragile build systems or tests, etc.
You can put together a plan on how to make a meaningful dent in the problem, but often times it’s good intentions. And by the time your actionable plan starts seeing execution, your leadership changes or product priorities shift and you’re back at square one.
Specifically from a big tech perspective, this is the cost that people don’t like to talk about. And it’s because of the incentive structures… product managers must spit out features for their own career progression, which goes into planning and promises made to executives, which reflects in what the company produces and whether equity owners see continued growth in these growth companies or not, directly impacting how the company is perceived, brand value, and market cap.
In big tech, organizations often try to move quickly and release features… as part of whatever goals their organization promised to deliver. Expectations are unrealistic or only achievable if you really stretch, and everyone wants to deliver because that’s how you get promoted.
This incentive structure leads to an abundance of tech debt. And tech debt is hard to quantitatively measure. Eventually it slows down your pace of innovation, as all the little things that were accepted along the way pile up.
I don’t know of a solution to the problem other than looping your leadership in on what are the barriers to productivity - whether it’s poorly architected code, fragile build systems or tests, etc.
You can put together a plan on how to make a meaningful dent in the problem, but often times it’s good intentions. And by the time your actionable plan starts seeing execution, your leadership changes or product priorities shift and you’re back at square one.
Specifically from a big tech perspective, this is the cost that people don’t like to talk about. And it’s because of the incentive structures… product managers must spit out features for their own career progression, which goes into planning and promises made to executives, which reflects in what the company produces and whether equity owners see continued growth in these growth companies or not, directly impacting how the company is perceived, brand value, and market cap.
This is obviously a rant for its own sake more than an answerable question.
The quality of software runs the gamut and is orthogonal to the size and brand name recognition of its writers.
Also, every sufficiently large project, no matter how tight the ship being run, will at various stages feel like it's about to collapse under its own weight.
The quality of software runs the gamut and is orthogonal to the size and brand name recognition of its writers.
Also, every sufficiently large project, no matter how tight the ship being run, will at various stages feel like it's about to collapse under its own weight.
I'm frustrated for sure. I am looking for some pointers tho.
How do people handle this "patch on patch" code. It reminds me a lot of the sunken cost fallacy.
How do you approach this to the people who wrote it?
Perhaps FAANG pays so much exactly because of this? I mean, they want the best of the best because maintaining patchy code (without breaking production) is difficult and painful. At least, in my experience in non-FAANG companies: they pay the big bucks to do the difficult stuff (e.g., maintain legacy patchy code without breaking production and bringing money to the company) not for doing the easy stuff (e.g., starting from scratch some cute microservice using DDD and Hexagonal Architecture).
How to handle garbage code? You do your job, which is declutter and refactor. You know you're doing it right if the lines deleted exceeds the lines added. Unfortunately, the bigger the ball of mud, the greater the inertia both political and technical. Even software can't escape basic laws of physics.
A big company contains a large number of units that are culturally distinct. That is a strength instead of a weakness because a monoculture will certainly go bad and not be responsive to reality.
All organizations fall short of their ideals in some places. I worked at a startup where the CTO thought monads were great for error handling in Scala. I was skeptical but I was working on Python at the beginning and wasn’t involved.
I got tasked w/ working w/ the Scala subsystem later and found that habitually coders would fail to implement error checking and these got through code reviews done by that guy who thought monads were so great.
I used the phrase ‘normalization of deviance’ all the time there and it raised eyebrows. One day I got let go because of the mismatch.
Where I am now there are gaps between ‘what we do’ and ‘what we say we do’ but the contradictions don’t bother me and we move the football down the field consistently so I am satisfied. When i make a mistake it gets caught in code review which makes me feel like they have my back.
Bad code is endemic outside FANG but you have to travel far and wide to find really good code. Good code is not expensive, actually it is cheap because devs are much more productive and they don’t have to hire so many of them. Those devs don’t turn over and the firms don’t hire many devs.
Places with bad code have awful productivity and hire huge numbers of devs to make up for it. They are always hiring.
All organizations fall short of their ideals in some places. I worked at a startup where the CTO thought monads were great for error handling in Scala. I was skeptical but I was working on Python at the beginning and wasn’t involved.
I got tasked w/ working w/ the Scala subsystem later and found that habitually coders would fail to implement error checking and these got through code reviews done by that guy who thought monads were so great.
I used the phrase ‘normalization of deviance’ all the time there and it raised eyebrows. One day I got let go because of the mismatch.
Where I am now there are gaps between ‘what we do’ and ‘what we say we do’ but the contradictions don’t bother me and we move the football down the field consistently so I am satisfied. When i make a mistake it gets caught in code review which makes me feel like they have my back.
Bad code is endemic outside FANG but you have to travel far and wide to find really good code. Good code is not expensive, actually it is cheap because devs are much more productive and they don’t have to hire so many of them. Those devs don’t turn over and the firms don’t hire many devs.
Places with bad code have awful productivity and hire huge numbers of devs to make up for it. They are always hiring.
I've noticed that reviews here are trivial.
How would you approach this without stepping on toes?
I am a coder so my weapon of choice is comments.
There must be some area where you could have done it this way or could have done it that way or something you are not sure about.
Make the specific request in the pull request that they review a comment (about a subject where an opinion is wanted)
There must be some area where you could have done it this way or could have done it that way or something you are not sure about.
Make the specific request in the pull request that they review a comment (about a subject where an opinion is wanted)
Maybe unpopular opinion: Either do the same or leave if you cannot handle it.
What you're running into is mismatched incentives. Why would they even care about making the code nicer? It has mostly downsides. So the company can move faster? They don't care because they still get paid the same. It's not their company. Refactoring takes time and doesn't show up on most performance reviews. It's the gmail affect - you get promoted when you ship new features and make a "new version of something" - not by fixing old code. Cleaning up code also has the downside that the original authors are now replaceable because more people can understand it.
By trying to change this you are probably just hurting yourself. You are making enemies and annoying people and are not benefitting anyone.
This is pretty common in FAANG, but it's not like this in all teams. It depends on the team more than the company. There are some teams that are more "startup-like" where incentives are a bit more aligned. Maybe you can transfer.
What you're running into is mismatched incentives. Why would they even care about making the code nicer? It has mostly downsides. So the company can move faster? They don't care because they still get paid the same. It's not their company. Refactoring takes time and doesn't show up on most performance reviews. It's the gmail affect - you get promoted when you ship new features and make a "new version of something" - not by fixing old code. Cleaning up code also has the downside that the original authors are now replaceable because more people can understand it.
By trying to change this you are probably just hurting yourself. You are making enemies and annoying people and are not benefitting anyone.
This is pretty common in FAANG, but it's not like this in all teams. It depends on the team more than the company. There are some teams that are more "startup-like" where incentives are a bit more aligned. Maybe you can transfer.
Bad ideas
* Complaining about the code - not helpful
* Refactoring large parts of the code - dangerous
Good ideas
* Adding tests
* Finding bugs - I assure you they exist
* Finding perf wins - I assure they exist
* Finding a new team - you might not be able to do this immediately, but it's very likely you can find a better fit on another team
* Complaining about the code - not helpful
* Refactoring large parts of the code - dangerous
Good ideas
* Adding tests
* Finding bugs - I assure you they exist
* Finding perf wins - I assure they exist
* Finding a new team - you might not be able to do this immediately, but it's very likely you can find a better fit on another team
Yes, this is common. Frankly it was a bad idea to complain; you should be prepared to... be invited to leave your current position.
Here's why: Everyone (on the ground floor at least) knows that the code is terrible, but the organization is not optimizing for good code. Code quality is simply and literally a non-goal. The goal is to deliver Projects, where a Project is a unit of work which advances someone's career, whether it be an IC, a project/team lead, a project manager, etc. Now, it's okay not to internalize that goal for yourself; but you've exposed yourself as not understanding that this is the goal that everyone else has. Unfortunately, this makes you a liability w/r/t the goal.
There are ways to advance other goals in the organization, and I suspect these organizations only survive as long as sufficient people are doing just that, but they all revolve around making other goals coincide with The Goal described above.
Note that The Goal isn't a secret or taboo; it's explicitly acknowledged in some organizations. Not so long ago at Microsoft, for example, it is known as "impact" and is the explicit top-level metric for performance review. If you can figure out how The Goal it's expressed in your organization, you can attempt to rescue the situation by explaining that what you really meant was you want to advance The Goal (eg. "improve my impact") by delivering improved code quality in future projects which will have such-and-such an impact on visible metrics.
Here's why: Everyone (on the ground floor at least) knows that the code is terrible, but the organization is not optimizing for good code. Code quality is simply and literally a non-goal. The goal is to deliver Projects, where a Project is a unit of work which advances someone's career, whether it be an IC, a project/team lead, a project manager, etc. Now, it's okay not to internalize that goal for yourself; but you've exposed yourself as not understanding that this is the goal that everyone else has. Unfortunately, this makes you a liability w/r/t the goal.
There are ways to advance other goals in the organization, and I suspect these organizations only survive as long as sufficient people are doing just that, but they all revolve around making other goals coincide with The Goal described above.
Note that The Goal isn't a secret or taboo; it's explicitly acknowledged in some organizations. Not so long ago at Microsoft, for example, it is known as "impact" and is the explicit top-level metric for performance review. If you can figure out how The Goal it's expressed in your organization, you can attempt to rescue the situation by explaining that what you really meant was you want to advance The Goal (eg. "improve my impact") by delivering improved code quality in future projects which will have such-and-such an impact on visible metrics.
This resonates, I also suggested that I might not be a good fit as I dont think the result impact is there.
This was probably not good career move here, as it is being discussed higher up.
But I honestly am unable to create impact with this codebase. This could also just mean that I am not fit for the job and they need other people.
I do understand the goal tho, that's why I raised the issue that I cant deliver enough impact in my eyes. And I think it's the right thing to raise.
Thanks for your reply, it now makes sense that quality is not a goal. At my level I should be able to work with this instead of just complaining.
I do understand the goal tho, that's why I raised the issue that I cant deliver enough impact in my eyes. And I think it's the right thing to raise.
Thanks for your reply, it now makes sense that quality is not a goal. At my level I should be able to work with this instead of just complaining.
FAANG wants people who can 'get things done' in the byzantine code bases that they maintain. That is why they pay so much and why turnover is so high.
Fundamentally fixing the problem is not necessarily possible. At FAANG basic things like maybe upgrading from Java 8 to Java 11 are three year 50 person projects.
Fundamentally fixing the problem is not necessarily possible. At FAANG basic things like maybe upgrading from Java 8 to Java 11 are three year 50 person projects.
It's also very possible that you'd be happier and more effective on a different team at the same company. All of these companies are big enough to have a great diversity of systems, from nascent projects to polished but boring workhorses, to unmaintained balls of tech debt that they can't turn off but nobody wants to work in. Often if you want to influence the direction of a codebase, it's best to get in on a new project or system that's just being built.
"Projects" are "something that advances someone's career"? I thought projects were "something which our client actually paid for and we're contractually obligated to fullfil"
A bit of A, a bit of B. Successfully delivering a project that generates a lot of revenue is a great way to advance one's career.
Probably not for FAANG who tend to not have clients with custom contracts in general. Unless you are going to spend 20bn a year with Google I don’t think they care about you enough for a special contract with deliverables.
It really boils down to this: if it ain’t broke, don’t fix it.
If it is broken, and no one else seems to see that it’s broken, then it’s your job to convince everyone else that it’s broken so that “fixing it” becomes part of the The Goal.
If it is broken, and no one else seems to see that it’s broken, then it’s your job to convince everyone else that it’s broken so that “fixing it” becomes part of the The Goal.
Though this can also backfire. I observed a huge architecture error in the central data structure / schema of our system. Everyone agreed that it was a huge problem so I got put in charge of fixing it. Of course when you build an entire system around a bad schema, it gets baked through every inch of that system. Over a year later and it’s mostly fixed, but my career has stalled and I took the major hit to fix it.
I'm in a similar position in a non FANG company. I almost made the similar mistake of telling people it was a bad codebase. I did inform my manager though. My manager and I are new hires in the company so it didn't seem like an issue to be vocal to him.
I have seen my manager do the exact same thing as you did. He went to the architect behind the codebase and started complaining and raising hell. Since my manager is on better terms with the higher ups things are swaying towards his end else it'd be a fiasco.
My suggestion is that when you encounter such code bases either go with it or do some small refactors in the smallest of code and say "hey I feel this is a better way to do it. Can you review this?". I have felt this has usually worked for me. Additionally, instead of stating it outright that the code is bad, ask them to explain the code and the reasons behind it. If you are still on good terms and they don't see you as a threat, then you can ask questions and maybe suggest alternatives with a working prototype. This way you still come off as an "honest" person who's trying to do something good. This works in your favor when looked at by your manager and upper management. This is apart from the features that you're tasked to push.
Hope that helps.
I have seen my manager do the exact same thing as you did. He went to the architect behind the codebase and started complaining and raising hell. Since my manager is on better terms with the higher ups things are swaying towards his end else it'd be a fiasco.
My suggestion is that when you encounter such code bases either go with it or do some small refactors in the smallest of code and say "hey I feel this is a better way to do it. Can you review this?". I have felt this has usually worked for me. Additionally, instead of stating it outright that the code is bad, ask them to explain the code and the reasons behind it. If you are still on good terms and they don't see you as a threat, then you can ask questions and maybe suggest alternatives with a working prototype. This way you still come off as an "honest" person who's trying to do something good. This works in your favor when looked at by your manager and upper management. This is apart from the features that you're tasked to push.
Hope that helps.
>Hope that helps.
It does, thanks.
I didn't say directly that their code is bad. But asking is futile, communication is bad. They dont want to make it a prio and 8 hour time diff makes it hard. I spent too many nights up to have video calls about it.
But you're advice aligns with other people here. Regardless of he sate, I should be help drive it with refactors and just deal with it indeed.
I didn't say directly that their code is bad. But asking is futile, communication is bad. They dont want to make it a prio and 8 hour time diff makes it hard. I spent too many nights up to have video calls about it.
But you're advice aligns with other people here. Regardless of he sate, I should be help drive it with refactors and just deal with it indeed.
I worked at Google on a somewhat core system (Billing) and was completely shocked by how bad pretty much every unit within it was. The overarching service would go down a fair amount, despite the whole thing amounting to a big MySQL queue. Management would turn over as different people sought to make their career on fixing it. I got out, fast. I wonder if they ever fixed it.
Which company is it? Bundling them together probably doesn’t help it this case.
OP’s replies to comments were dead but did not appear to violate site guidelines, so I vouched for them to further the conversation and help them find the answers they’re looking for.
If it works and is maintainable then it's good code.
I can tell by the fact that you call then "diffs" that it's not Google. But I've seen both good and "bad" system designs and code there. Nothing quite as bad as you're describing. Definitely saw that kind of thing at Microsoft though. Perhaps this is the "move fast and break things" ethos of a certain company that famously uses Phabricator?
Android dev of 9 years, past: agencies, freelance, research companies, joined FANG 6 months ago.
My product is filled with odd side effects, buggy and slow. In my opinion the code is overly complex, classes >2000 lines, doing so many thing with the 20+ class scoped booleans to check for edge cases. Weird architecture that one person holds.
Easy to break this, bad customise implementation that tries to take care of every possibility. People make small changes to get impact and push code, I think this loses the bigger pictures and turns ugly quickly.
I'm (think) the only one that complains and is worried about this. Is it me or do other people experience the same at FANG? I talked to the authors for a dialog, I think I stepped on toes. Communication was bad, but now has grinded to a halt, with no responses, diffs taking ages to review, no replies, just cc's.
It all feels like patch work, these are all smart people. Some coworkers just say "just go with it and do the same".
Is this the attitude of FANG? I'm legitimately thinking im a bad fit and leave.
How do I validate that I'm not the problem? Maybe it's not overly complicated and I'm too stupid to realize it.
Anyone have any tips on how to talk to coworkers without stepping on toes regarding this code?
Sorry for chaotic questions. I just feel bad complaining, not being able to offer a solution and coworkers ignoring me. Thanks a lot.
Edit: also talked to my managers, it's being discussed higher up apparently.