Leaving LinkedIn(corecursive.com)
corecursive.com
Leaving LinkedIn
https://corecursive.com/leaving-linkedin-with-chris-krycho/
336 comments
There's an element of truth to this! I alluded to this (and Adam totally reasonably cut even more of it for time) but I definitely was not as effective organizationally as I could have been; that is one of the major things I learned from my time at LinkedIn. The biggest challenge for DX teams remains figuring out how to (a) communicate but more importantly (b) align their work with the key priorities of the business. I got to be okay at (a) but never particularly succeeded at (b) in my time at LinkedIn. Some of that is on me! Some of it… is on LinkedIn.
That said, in this case, the person telling me “You’re too idealistic” literally meant it in the sense of “You should not care about anything unless it directly moves the bottom line,” and I reject that to my bones. The bottom line matters. So, though, do things like user experience, developer experience, and for that matter just basic ethics about what we build (though happily the latter wasn’t in view here).
That said, in this case, the person telling me “You’re too idealistic” literally meant it in the sense of “You should not care about anything unless it directly moves the bottom line,” and I reject that to my bones. The bottom line matters. So, though, do things like user experience, developer experience, and for that matter just basic ethics about what we build (though happily the latter wasn’t in view here).
> “You should not care about anything unless it directly moves the bottom line,” and I reject that to my bones
I have a lot of empathy for that feeling. I've come across this often in my career and I have not enjoyed it. I enjoy building software that I'm proud off. Thank you for sharing your story!
I have a lot of empathy for that feeling. I've come across this often in my career and I have not enjoyed it. I enjoy building software that I'm proud off. Thank you for sharing your story!
>>My career lesson is that being right isn't the hard part of being a senior staff engineer.
Not only that, but your own definition of 'right' may or may not align with what is 'right' for the people paying your salary - pretending otherwise is just silly.
In any organization, you advocate for what you believe is 'right' the best you can, someone else (or a consensus of others) will likely decide if they agree - you get to decide if you are willing to live with it/compromise, or move on - I have done both in my career.
Not only that, but your own definition of 'right' may or may not align with what is 'right' for the people paying your salary - pretending otherwise is just silly.
In any organization, you advocate for what you believe is 'right' the best you can, someone else (or a consensus of others) will likely decide if they agree - you get to decide if you are willing to live with it/compromise, or move on - I have done both in my career.
I mean, yes and no. I find staff engineers, and especially the senior-staff-engineer-type positions, to be the ones most sensitive to the exact specifics of the org they've joined.
I've seen a senior staff engineer (at a popular unicorn you've all heard of) who was a very smart, but also friendly and reasonable person (how often are those traits in opposition of each other!). They were very good at the staff engineer game. They were vouching for us to upgrade a framework that we pumped 50M of spend through per year. From v2 to v3. It wasn't a huge change, not like python 2 to 3 at all. Positively trivial compared to py2 to 3. But management just were not up for it at all. Even with research done showing we could expect literally a baseline of 10% improvement in performance (so you do the math on that vs the 50M of spend), they just did not want to "waste time on version upgrades." The staff engineer was told they were being unreasonable in pushing so hard. Their manager tried to ditch them, but again, they're good at the staff eng game, so they switched teams due to being in the good graces (due to an excellent track record) with a middle manager above them. They decided, hey fuck you, I'm going to one man army this effort. They did, they had preview versions ready in less than a month and had migrated some of the bigger-win jobs within 2, saving their annual comp (which was prodigious) several times over just in that. Suddenly everyone was keen on moving their stuff over now that the startup cost (both political and eng-time) was paid.
Anyway fast forward a year, the rollout is complete and nearly the entire fucking management chain above him has turned over, a 50/50ish mix of firings vs quittings. But he's still there and the upgrade migration is complete.
So sure, sometimes staff engineers have a stick up their bum and can't dislodge it enough to get work done on the bottom line. But sometimes they're the only sane ones in a crazy world, and you can only do so much to try to turn a balky mule around.
I've seen a senior staff engineer (at a popular unicorn you've all heard of) who was a very smart, but also friendly and reasonable person (how often are those traits in opposition of each other!). They were very good at the staff engineer game. They were vouching for us to upgrade a framework that we pumped 50M of spend through per year. From v2 to v3. It wasn't a huge change, not like python 2 to 3 at all. Positively trivial compared to py2 to 3. But management just were not up for it at all. Even with research done showing we could expect literally a baseline of 10% improvement in performance (so you do the math on that vs the 50M of spend), they just did not want to "waste time on version upgrades." The staff engineer was told they were being unreasonable in pushing so hard. Their manager tried to ditch them, but again, they're good at the staff eng game, so they switched teams due to being in the good graces (due to an excellent track record) with a middle manager above them. They decided, hey fuck you, I'm going to one man army this effort. They did, they had preview versions ready in less than a month and had migrated some of the bigger-win jobs within 2, saving their annual comp (which was prodigious) several times over just in that. Suddenly everyone was keen on moving their stuff over now that the startup cost (both political and eng-time) was paid.
Anyway fast forward a year, the rollout is complete and nearly the entire fucking management chain above him has turned over, a 50/50ish mix of firings vs quittings. But he's still there and the upgrade migration is complete.
So sure, sometimes staff engineers have a stick up their bum and can't dislodge it enough to get work done on the bottom line. But sometimes they're the only sane ones in a crazy world, and you can only do so much to try to turn a balky mule around.
This reminds me of the blog post, "People can read their manager's mind," by Yossi Kreinin.
https://yosefk.com/blog/people-can-read-their-managers-mind....
In particular:
"Corollary 1. Who can, and sometimes does, un-rot the fish from the bottom? An insane employee. Someone who finds the forks, crashes, etc. a personal offense, and will repeatedly risk annoying management by fighting to stop these things."
https://yosefk.com/blog/people-can-read-their-managers-mind....
In particular:
"Corollary 1. Who can, and sometimes does, un-rot the fish from the bottom? An insane employee. Someone who finds the forks, crashes, etc. a personal offense, and will repeatedly risk annoying management by fighting to stop these things."
One way to read your story is that upper management applied heavy downward pressure, to the extent that relatively trivial changes in the organization were discussed as 'costly' - in an effort to get engineers like the senior staff engineer in your example to pull heroic feats of engineering in a few weeks all by himself, instead of using a small group of engineers / resources.
I think your argument is that 'it worked for him' because he is still there while the managers are not, but reality may be the managers were there to have that effect in the first place and the 50/50 turnover was because they were in the thick of it.
That'd be my cynical take, anyway.
I think your argument is that 'it worked for him' because he is still there while the managers are not, but reality may be the managers were there to have that effect in the first place and the 50/50 turnover was because they were in the thick of it.
That'd be my cynical take, anyway.
"upper management applied heavy downward pressure"
Correct
"I'm an effort to get engineers like the senior staff engineer to pull heroic feats"
I think this gives them too much credit. The reality, imo, is that the org had a pathological emphasis on "impact" and ran performance reviews so tight that everyone, managers included, were so scared of being PIPd and losing their grants that they were extraordinarily risk averse in planning and picking projects to bet on. And leadership in general undervalued "infra-y" changes as mostly make-work.
The turnover was partly because of this general effect - people scared, not betting big or smart enough, and optimizing for holding on / not getting fired. And to their credit, most of these people vested 2-3 years of their 4-year grant before it all fell over, so they made out decently lucratively, so maybe they're the real winners in all this after all.
Correct
"I'm an effort to get engineers like the senior staff engineer to pull heroic feats"
I think this gives them too much credit. The reality, imo, is that the org had a pathological emphasis on "impact" and ran performance reviews so tight that everyone, managers included, were so scared of being PIPd and losing their grants that they were extraordinarily risk averse in planning and picking projects to bet on. And leadership in general undervalued "infra-y" changes as mostly make-work.
The turnover was partly because of this general effect - people scared, not betting big or smart enough, and optimizing for holding on / not getting fired. And to their credit, most of these people vested 2-3 years of their 4-year grant before it all fell over, so they made out decently lucratively, so maybe they're the real winners in all this after all.
That's an interesting take, thanks for writing that. That gives me something to think about.
Especially if he wasn't rewarded commensurately for that feat.
I’m not American so I don’t know what a staff engineer does, and googling the term is not helping much. I have also not worked in any software company that has “engineering teams” which would maybe have helped me understand the Google results.
What you are saying is that sometimes a senior engineer manages to make good things happen?
Did this person get rewarded for this huge effort they put in? You say they already have prodigious yearly compensation, was it at least doubled given that they saved it several times over?
What you are saying is that sometimes a senior engineer manages to make good things happen?
Did this person get rewarded for this huge effort they put in? You say they already have prodigious yearly compensation, was it at least doubled given that they saved it several times over?
A staff engineer is a level above senior. What they do is make them themselves more valuable than the regular engineer who has been around for 10+ years. As you get experience there is diminishing returns, so most engineers will not make a staff level (though there is plenty of title inflation and so you will find many). A staff engineer needs to know the code (or whatever they engineer) well, but they don't normally write a lot of code. They tend to spend more time thinking about the larger problems (including forcing through upgrades nobody wants to pay for), mentoring juniors, and figuring out the hard architecture problems (software architects have a role, but in my experience they rarely actually create architecture despite the name - possibly because what developers need of architecture is different from management?).
Staff level developers are trusted to figure out what needs to be done without direction. When they are given direction it is figure out how the other engineers break this problem up and do it - typically not do it themselves.
If staff engineers are doing something it is not important to any project. So nobody feels bad about interrupting them if they need help or something urgent comes up. (this also means you are developing your senior developers into staff engineers by giving them responsibility)
Staff level developers are trusted to figure out what needs to be done without direction. When they are given direction it is figure out how the other engineers break this problem up and do it - typically not do it themselves.
If staff engineers are doing something it is not important to any project. So nobody feels bad about interrupting them if they need help or something urgent comes up. (this also means you are developing your senior developers into staff engineers by giving them responsibility)
> Did this person get rewarded for this huge effort they put in? You say they already have prodigious yearly compensation, was it at least doubled given that they saved it several times over?
When you're getting paid $500k-$1m/year or more as a staff engineer, putting in huge effort and getting organization-wide impactful results is part of the job description. I'm sure it had a good effect on their yearly comp review, but suggesting that their comp should be doubled because they did their job is silly.
When you're getting paid $500k-$1m/year or more as a staff engineer, putting in huge effort and getting organization-wide impactful results is part of the job description. I'm sure it had a good effect on their yearly comp review, but suggesting that their comp should be doubled because they did their job is silly.
Staff engineers (in the FAANG parlance) are, generally speaking, ICs who were previously senior engineers who have proven themselves enough that they're trusted to be roughly autonomous, they're kind of "peers" to engineering managers in a way. While a senior engineer will typically not have the political capital move big politically-heavy rocks, it is typically down in the staff engineer's job description that they are expected to. It's basically shifting from a focus on code and building systems to people/the org and architecting systems that fit the org.
"Did the person get rewarded for this huge effort" Money-wise? Haha. No. Only in the sense that they further solidified their soft power as someone you shouldn't bet against. And of course the sense of pride that comes with shipping something (that further enriches some capitalist and maybe yourself to 0.0013% of the effect, since that's how much of the company your grant commands shares of).
"Did the person get rewarded for this huge effort" Money-wise? Haha. No. Only in the sense that they further solidified their soft power as someone you shouldn't bet against. And of course the sense of pride that comes with shipping something (that further enriches some capitalist and maybe yourself to 0.0013% of the effect, since that's how much of the company your grant commands shares of).
> staff engineers are ICs
Wtf is an IC? Internal consultant? Individual contributors (what does that even mean? One person team?)
Using abbreviations to explain something is a bad way to explain...
Wtf is an IC? Internal consultant? Individual contributors (what does that even mean? One person team?)
Using abbreviations to explain something is a bad way to explain...
Used to mean a non-manager. I agree it's a nonsense term.
It still does. Individual contributor is someone with no reports.
[deleted]
you gotta a do what you gotta do, but if you have the luxury, and conceptual integrity, there's going to be sacrifices in the name of "alignment" that you're just not prepared to make. I agree with your positioning on lots of "right" solutions, IME it's recognizing and understanding and still not accepting when it violates your foundational values. I've been there and had the ability to not participate, but this isn't always the case.
You're exactly right and to me that's part of finding the 'right' solution for the organization. Sometimes you compromise for 'alignment', and heck sometimes you discover that those compromises were a better solution all along.
In other cases you chose to 'pick a battle' and find the best way to navigate and get buy in. Because again being right -and getting buy in- is the important part. Not later giving everyone an 'I told you so'.
In other cases you chose to 'pick a battle' and find the best way to navigate and get buy in. Because again being right -and getting buy in- is the important part. Not later giving everyone an 'I told you so'.
Separate Church and Business. Get paid while making a meaningful contribution. Use the time outside of work to live the values you do not otherwise monetize. Critically: Have Time Outside of Work.
> To me that's the most interesting quote of the podcast and that's the impression I had gotten before reading this quote. Sounds like they received valuable feedback along the way and ignored it on purpose.
We don't really have enough context to make a judgement here IMO. I've been in places large enough to have fairly complex politics where people would manouver themselves into better jobs as much as they could, including performing a hostile takeover of another department. They might not have had the best ideas or the best plans, but they did have the right connections, go to lunch with the right people and say the right words.
I've seen bad decisions made because more persuasive and better connected people were attempting to climb the ladder and managed to talk management into it.
"You’re too idealistic. You don’t care enough about the bottom line. You should change your values." could just be something someone says when they're trying to smear someone in the way, especially if that's how they've sold themselves and their ideas to upper management.
Personally I've seen and done a number of large scale changes/upgrades (mostly Ruby/Rails/Postgres things) in orgs smaller than Facebook but still with hundreds of engineers and large codebases, both stop the world, and where the work was streamed seamlessly into regular feature etc. work and the methodology outlined by Chris seems incredibly sensible and aligns with what I have found to be the most successful way of doing these things.
As the host says, we ony have one side of the story here, but at least a lot of the things Chris is saying seem to make sense.
> My career lesson is that being right isn't the hard part of being a senior staff engineer. Being right isn't enough. It's building alignment across the entire org towards the right solution. That's the hard valuable problem that I'm paid to solve.
I agree, leadership positions require trust and respect to be effective. You still have to be right for that to be useful of course. Progress in the wrong direction isn't really progress.
We don't really have enough context to make a judgement here IMO. I've been in places large enough to have fairly complex politics where people would manouver themselves into better jobs as much as they could, including performing a hostile takeover of another department. They might not have had the best ideas or the best plans, but they did have the right connections, go to lunch with the right people and say the right words.
I've seen bad decisions made because more persuasive and better connected people were attempting to climb the ladder and managed to talk management into it.
"You’re too idealistic. You don’t care enough about the bottom line. You should change your values." could just be something someone says when they're trying to smear someone in the way, especially if that's how they've sold themselves and their ideas to upper management.
Personally I've seen and done a number of large scale changes/upgrades (mostly Ruby/Rails/Postgres things) in orgs smaller than Facebook but still with hundreds of engineers and large codebases, both stop the world, and where the work was streamed seamlessly into regular feature etc. work and the methodology outlined by Chris seems incredibly sensible and aligns with what I have found to be the most successful way of doing these things.
As the host says, we ony have one side of the story here, but at least a lot of the things Chris is saying seem to make sense.
> My career lesson is that being right isn't the hard part of being a senior staff engineer. Being right isn't enough. It's building alignment across the entire org towards the right solution. That's the hard valuable problem that I'm paid to solve.
I agree, leadership positions require trust and respect to be effective. You still have to be right for that to be useful of course. Progress in the wrong direction isn't really progress.
A great choice quote, and add to that LinkedIn as an engineering culture already heavily skews towards “craftsmanship” (as they call it) and refinement versus the larger industry. The speaker appears ridiculously unaware of how outlier his position is.
Do you want to wax the wheels of a title-winning race car or of a 1990 Toyota Camry that happens to be in mint condition? A lot of LinkedIn engineers I’ve met are the latter, and Microsoft is trying to push them into being the former. If you are The One True Master Wheel Polisher then sure maybe you get to choose your car, but don’t soapbox it. The audience is smarter than that (I hope).
Do you want to wax the wheels of a title-winning race car or of a 1990 Toyota Camry that happens to be in mint condition? A lot of LinkedIn engineers I’ve met are the latter, and Microsoft is trying to push them into being the former. If you are The One True Master Wheel Polisher then sure maybe you get to choose your car, but don’t soapbox it. The audience is smarter than that (I hope).
[deleted]
Have you ever worked with a BOFH?
BOFHs suck.
Your career lesson is correct.
BOFHs suck.
Your career lesson is correct.
[deleted]
Its interesting because, although I don't know the linkedin codebases nor worked at linkedin, I've seen a few codebases that sound scarily similar and am familiar with a few companies that sound organisationally and politically scarily similar.
And I generally advocate 'finger gun' approaches.
Finger gun rewrites can be implemented in a good way. Often when you have n clients trying to do the same thing, one of those can be used as a basis for serving the other platforms. Or if you start over, you can do it in a good clean quick concise way.
The general success ingredient is putting a small team of veterans on making the new system. Success always comes from a small team of engineers who are both domain experts and technical experts rolled into one. (Controversially, I think that generalises; all success comes from this, even for mundane keep-the-lights-on problems. And everyone else is just slowing them down.)
The big problem most tech management everywhere keeps repeating is that the next big system will be built the least-experienced.
Would be really nice to hear the symmetrical interview from a finger-gunner.
And I generally advocate 'finger gun' approaches.
Finger gun rewrites can be implemented in a good way. Often when you have n clients trying to do the same thing, one of those can be used as a basis for serving the other platforms. Or if you start over, you can do it in a good clean quick concise way.
The general success ingredient is putting a small team of veterans on making the new system. Success always comes from a small team of engineers who are both domain experts and technical experts rolled into one. (Controversially, I think that generalises; all success comes from this, even for mundane keep-the-lights-on problems. And everyone else is just slowing them down.)
The big problem most tech management everywhere keeps repeating is that the next big system will be built the least-experienced.
Would be really nice to hear the symmetrical interview from a finger-gunner.
>The big problem most tech management everywhere keeps repeating is that the next big system will be built the least-experienced.
Most places I've worked at, this ends up being the case because the most experienced are needed to fire fight and to keep things running.
Most places I've worked at, this ends up being the case because the most experienced are needed to fire fight and to keep things running.
> (...) because the most experienced are needed to fire fight and to keep things running.
By that account the most experienced started out as the least experienced who built a mess, accrued more technical debt, and built up their experience keeping their ball of mud from crumbling.
They are also more resistent to change because this underlines all the crap they did. It's their baby, and their reputation is pinned on how successful it is.
New is not necessarily better, but old and stable also doesn't mean fine engineering.
By that account the most experienced started out as the least experienced who built a mess, accrued more technical debt, and built up their experience keeping their ball of mud from crumbling.
They are also more resistent to change because this underlines all the crap they did. It's their baby, and their reputation is pinned on how successful it is.
New is not necessarily better, but old and stable also doesn't mean fine engineering.
>started out as the least experienced who built a mess, accrued more technical debt
THIS
"experienced doesnt always mean good. This is why you build teams.
The CFO is your enemy as an engineer.
THIS
"experienced doesnt always mean good. This is why you build teams.
The CFO is your enemy as an engineer.
>The general success ingredient is putting a small team of veterans on making the new system.
I don't disagree with this approach. But one thing I find is:
Sometimes those veteran teams are heavily invested in existing concepts, tools, and approaches. It makes sense, they have seen these things work and they're proven to some extent.
But it's always not the best choice.... and while the veteran teams often start these projects they often aren't the people to "finish" them as they get pulled away. And it is oh so easy to start projects and make decisions if you never see the outcome / have to deal with the fallout.
I don't disagree with this approach. But one thing I find is:
Sometimes those veteran teams are heavily invested in existing concepts, tools, and approaches. It makes sense, they have seen these things work and they're proven to some extent.
But it's always not the best choice.... and while the veteran teams often start these projects they often aren't the people to "finish" them as they get pulled away. And it is oh so easy to start projects and make decisions if you never see the outcome / have to deal with the fallout.
> Sometimes those veteran teams are heavily invested in existing concepts, tools, and approaches. It makes sense, they have seen these things work and they're proven to some extent.
This is very closely related to why these veteran teams end up successful. Existing concepts, tools and approaches should be the default; if your company uses Python with MySQL you better have a really, really good reason for your new project to use Rust and CockroachDB.
> while the veteran teams often start these projects they often aren't the people to "finish" them as they get pulled away.
I disagree with this. The veterans are veterans because they've already spent time maintaining the existing systems. They've handled oncall, major bugs, and seen all kinds of stupid shit that happens with software. They'll get pulled onto other projects eventually, sure, but they have a higher chance of getting a system into production in front of users, IMO.
This is very closely related to why these veteran teams end up successful. Existing concepts, tools and approaches should be the default; if your company uses Python with MySQL you better have a really, really good reason for your new project to use Rust and CockroachDB.
> while the veteran teams often start these projects they often aren't the people to "finish" them as they get pulled away.
I disagree with this. The veterans are veterans because they've already spent time maintaining the existing systems. They've handled oncall, major bugs, and seen all kinds of stupid shit that happens with software. They'll get pulled onto other projects eventually, sure, but they have a higher chance of getting a system into production in front of users, IMO.
Some changes are better than others. If your product works in python/MySQL then rust probably isn't a good fit. Maybe Ruby or PostgresSQL would be better fits.
If your main product is C or C++ then you should be learning about things like rust. Maybe you decide modern C++ is good enough, but if so you should be working with the C++ committee to push safe C++ (as a C++ developer I'm hoping this comes in C++26... I'm looking at how I can put rust into my code for now).
The point is you should know what the good next steps to investigate are and what are not. Rust and Python are both great tools, but it is never okay to choose between the two on any project - the choice is obvious before you get to choosing. Both tools have other competitors that you perhaps should be looking at though (Rust - ada, C++. Python: ruby, go)
If your main product is C or C++ then you should be learning about things like rust. Maybe you decide modern C++ is good enough, but if so you should be working with the C++ committee to push safe C++ (as a C++ developer I'm hoping this comes in C++26... I'm looking at how I can put rust into my code for now).
The point is you should know what the good next steps to investigate are and what are not. Rust and Python are both great tools, but it is never okay to choose between the two on any project - the choice is obvious before you get to choosing. Both tools have other competitors that you perhaps should be looking at though (Rust - ada, C++. Python: ruby, go)
> Some changes are better than others. If your product works in python/MySQL then rust probably isn't a good fit. Maybe Ruby or PostgresSQL would be better fits. [...] Rust and Python are both great tools, but it is never okay to choose between the two on any project - the choice is obvious before you get to choosing.
I actually really disagree with this. Python and Ruby are extremely similar in the problem they solve. If you're using MySQL as a basic OLTP database, and you need a basic OLTP database, you shouldn't use Postgres.
I'm talking organizationally here, not just on a single project. You gain nothing by introducing Ruby to your company that's filled with Python experts. Instead, you introduce a bunch of friction and overhead as developers have to relearn things like syntax, how to set up their development environment, how to do dependency management, testing frameworks, etc...
The time to introduce new tech is when it is fundamentally different enough that it could matter. Maybe you have a bunch of Python code, but there's a certain area that's very performance sensitive: you could look at using Rust, Go or C++. Maybe you have MySQL, but you need a database that's used for complex OLAP queries rather than low-latency OLTP work. There you could look at Redshift, Snowflake or the various Postgres OLAP configurations.
Where experience comes in is knowing when these new tools are worth the overhead they cause. Less experienced engineers vastly underestimate the overhead that comes with introducing new tooling. This even applies internally; for example, if you've used Flask for every project you've done at your company for the past 5 years is it really worth switching to Django?
I actually really disagree with this. Python and Ruby are extremely similar in the problem they solve. If you're using MySQL as a basic OLTP database, and you need a basic OLTP database, you shouldn't use Postgres.
I'm talking organizationally here, not just on a single project. You gain nothing by introducing Ruby to your company that's filled with Python experts. Instead, you introduce a bunch of friction and overhead as developers have to relearn things like syntax, how to set up their development environment, how to do dependency management, testing frameworks, etc...
The time to introduce new tech is when it is fundamentally different enough that it could matter. Maybe you have a bunch of Python code, but there's a certain area that's very performance sensitive: you could look at using Rust, Go or C++. Maybe you have MySQL, but you need a database that's used for complex OLAP queries rather than low-latency OLTP work. There you could look at Redshift, Snowflake or the various Postgres OLAP configurations.
Where experience comes in is knowing when these new tools are worth the overhead they cause. Less experienced engineers vastly underestimate the overhead that comes with introducing new tooling. This even applies internally; for example, if you've used Flask for every project you've done at your company for the past 5 years is it really worth switching to Django?
> The time to introduce new tech is when it is fundamentally different enough that it could matter.
I think you agree more than you realize. GP is essentially saying there are few situations where you will be deciding between Python and Rust on account of organization and technical reasons. However, sometimes you should consider introducing new technologies, but only when you identify that the problem being addressed is much better solved by a new technology, fully aware that this will cause various sorts of friction within your organization.
I think you agree more than you realize. GP is essentially saying there are few situations where you will be deciding between Python and Rust on account of organization and technical reasons. However, sometimes you should consider introducing new technologies, but only when you identify that the problem being addressed is much better solved by a new technology, fully aware that this will cause various sorts of friction within your organization.
I disagree. There's no magic to being on call that means you understand something well enough. That's simply not automatic.
Following through with something from start to "finish" is far far more valuable than fixing one off bugs and then doing something else, and then something else.
If anything I think it can lead to a lot of poor assumptions based on very skewed experience.
Following through with something from start to "finish" is far far more valuable than fixing one off bugs and then doing something else, and then something else.
If anything I think it can lead to a lot of poor assumptions based on very skewed experience.
Depends on the nature of your oncall. If your oncall is just following documented restart procedures when your app gets stuck, you won't learn much. If your oncall involves deep investigation into your system and how it interacts with other components (services, databases) etc you'll learn a lot.
per some sibling comments though, years of experience != veteran, to be clear. There's certainly people who have been working for 10+ years, sometimes even at the same company, and don't really understand what they're working on.
per some sibling comments though, years of experience != veteran, to be clear. There's certainly people who have been working for 10+ years, sometimes even at the same company, and don't really understand what they're working on.
I think the implication is they’ve seen all problems before and can engineer their new system to solve them.
Most software is never finished so I have no idea what arbitrary point you’re deciding they need to see it through before qualifying here.
> But it's always not the best choice....
I think it comes down to the personality of the veteran. If they're the "i've seen some shit, so I'll 'architect' the fuck out of this to handle any type of change" type veterans, then it's likely to fail. But if they take their knowledge of how the system has grown over time and architect to handle the types of changes they've seen, then there is likely a greater chance of success.
Personally, I think a having some junior and mid-level engineers on the rewrite is good because it's a good way to see how the system is all put together (as opposed to just one area) and they'll be able to have conversations with the veterans about how the system has evolved as it grew and that is valuable information.
I think it comes down to the personality of the veteran. If they're the "i've seen some shit, so I'll 'architect' the fuck out of this to handle any type of change" type veterans, then it's likely to fail. But if they take their knowledge of how the system has grown over time and architect to handle the types of changes they've seen, then there is likely a greater chance of success.
Personally, I think a having some junior and mid-level engineers on the rewrite is good because it's a good way to see how the system is all put together (as opposed to just one area) and they'll be able to have conversations with the veterans about how the system has evolved as it grew and that is valuable information.
Meh. I’ve met brilliant juniors and worthless veterans. I think it’s more important to recognize brilliance in people… That spark of hard work combined with vision and creativity.
Yeap, good point, tenure != expertise.
If I could edit the post I'd insert a qualifying "expert" before veteran, where the expert veteran meets my definition of the 10x developer :D
Any veterans who aren't gonna get the project done quickly _and_ properly aren't the kind of veterans I have in mind.
If I could edit the post I'd insert a qualifying "expert" before veteran, where the expert veteran meets my definition of the 10x developer :D
Any veterans who aren't gonna get the project done quickly _and_ properly aren't the kind of veterans I have in mind.
There’s definitely a middle ground between ‘we need to know how this plan will overcome every obstacle we will encounter before we even begin’ and ‘there won’t be any obstacles what do you mean?’
I always want to have a sense that the plan gives us realistic options for dealing with the obstacles. And that doesn’t just mean technical options - we’ll need to have the people to tackle it too. Eg it’s not great if the plan involves a bunch of teams running servers, and while technically there are obviously ways for all those teams to operate and scale those servers, if they don’t have the time or skills to do so then that option of having the teams run stuff is not really realistic.
But on the other hand planning a careful course that navigates round every obstacle is just as bad because
1) the obstacles will likely have moved by the time you get there (eg we spend ages figuring out how teams on legacy JS will be affected by the project only to find by the time we get there everyone has already moved to typescript and all that prep was YAGNI).
And 2) there are probably obstacles right in the path you’ve carefully chosen that you don’t know about yet and when you hit them your plan will grind to a halt because there was only one way planned and this was it.
We’re reading a cartoon podcast description of a complex architectural debate here though, so who knows which of these straw men was actually close to what was happening in LinkedIn.
I always want to have a sense that the plan gives us realistic options for dealing with the obstacles. And that doesn’t just mean technical options - we’ll need to have the people to tackle it too. Eg it’s not great if the plan involves a bunch of teams running servers, and while technically there are obviously ways for all those teams to operate and scale those servers, if they don’t have the time or skills to do so then that option of having the teams run stuff is not really realistic.
But on the other hand planning a careful course that navigates round every obstacle is just as bad because
1) the obstacles will likely have moved by the time you get there (eg we spend ages figuring out how teams on legacy JS will be affected by the project only to find by the time we get there everyone has already moved to typescript and all that prep was YAGNI).
And 2) there are probably obstacles right in the path you’ve carefully chosen that you don’t know about yet and when you hit them your plan will grind to a halt because there was only one way planned and this was it.
We’re reading a cartoon podcast description of a complex architectural debate here though, so who knows which of these straw men was actually close to what was happening in LinkedIn.
This is a great summary. Obviously I bring my own perspective to this, because, well… it happened to me, and I “lost”. But my read was that the folks I disagreed with and ultimately parted ways with were falling into the bucket you label ‘there won’t be any obstacles what do you mean?’.
The tradeoff around “do we migrate to TS?” is a great example here, by the way. A huge part of our decision-making last year around the big app at LinkedIn was on exactly that question: If you are going to stop and throw the entire thing away, you should waste exactly zero time migrating any of it. On the other hand, if the thing is going to be migrated incrementally, you should accelerate the TS effort because it will make it so much easier to code-mod the code base.
(I don’t agree that it’s a “cartoon” podcast description, but it is definitely extremely compressed and Adam and I chose to focus on the personal dynamics over the technical details, because diving into the latter would have made for a 5-hour episode. Also: it’s really tricky to do any of this kind of public discussion in a way that doesn’t just end up reading as a one-sided self-defense!)
The tradeoff around “do we migrate to TS?” is a great example here, by the way. A huge part of our decision-making last year around the big app at LinkedIn was on exactly that question: If you are going to stop and throw the entire thing away, you should waste exactly zero time migrating any of it. On the other hand, if the thing is going to be migrated incrementally, you should accelerate the TS effort because it will make it so much easier to code-mod the code base.
(I don’t agree that it’s a “cartoon” podcast description, but it is definitely extremely compressed and Adam and I chose to focus on the personal dynamics over the technical details, because diving into the latter would have made for a 5-hour episode. Also: it’s really tricky to do any of this kind of public discussion in a way that doesn’t just end up reading as a one-sided self-defense!)
> putting a small team of veterans on making the new system
I talked to someone who was leaving his job because he did this in order to meet a critical date for a product launch. The fact that his group of 3-4 people did what the company could not in a year ruffled so many feathers he said the board told the entirety of engineering that under no circumstance would anyone be allowed to dictate any detail of future projects.
I talked to someone who was leaving his job because he did this in order to meet a critical date for a product launch. The fact that his group of 3-4 people did what the company could not in a year ruffled so many feathers he said the board told the entirety of engineering that under no circumstance would anyone be allowed to dictate any detail of future projects.
Yeah, I've been these "cool Skunkworks" teams and I literally made no more money, maybe got a $5k spot bonus (super rare), and burned myself out over a 6mo launch. Spent my entire thanksgiving week on-call putting out fires, literally every hour, while everyone else at the 800 person company was on vacation. Not a single executive or product manager involved in launching was there or checked in on things. Some moron (I hope he reads this, my nicks not anon) let us launch without connecting our credit card fraud system so we (ops, me) got slammed for an entire week with fraud server launches. Jason, you're a moron, and that's an understatement.
It's like being paid in exposure. We'll pay you in cool tech and learning!
Then you see the 20 go devs around you making the same money writing grpc services 8 hours a day home at 5 relaxing.
It's like being paid in exposure. We'll pay you in cool tech and learning!
Then you see the 20 go devs around you making the same money writing grpc services 8 hours a day home at 5 relaxing.
I'm not really sure how this relates. The story I was told had the guy working as the lead with several less experienced but still competent programmers for around 2 months. There was no crunch, burnout, or anything of the like.
You're being downvoted because most devs are blue collar and hate the idea of passionate talented devs doing something amazing quickly and effectively. Any anecdotes of small elite team success must be crushed.
If you are working more than 8 hours a day you have nobody to blame but yourself I think. Especially in a situation like that.
[deleted]
To me it seems crazy that a board would be getting their hands dirty on engineering "specifics". I would think you would have engineering level leadership making those calls and the board would either trust that person or should move on.
Anyway:
I will see that I always worry about super teams because super teams tend to get free rein, so of course they can be faster and more successful. But sometimes that's about the bureaucracy and not even the team members.
I've started projects with a team and due to time factors decided "I'll just take all of this." and it worked out great. That's not because I'm amazing... it's just the old adage about "What one programmer can do in one month, two programmers can do in two months".
Unfortunately, I've seen places where these super teams / people are setup and endlessly praised and it goes on and on and it becomes very clear it isn't them, it's the structure.
Anyway:
I will see that I always worry about super teams because super teams tend to get free rein, so of course they can be faster and more successful. But sometimes that's about the bureaucracy and not even the team members.
I've started projects with a team and due to time factors decided "I'll just take all of this." and it worked out great. That's not because I'm amazing... it's just the old adage about "What one programmer can do in one month, two programmers can do in two months".
Unfortunately, I've seen places where these super teams / people are setup and endlessly praised and it goes on and on and it becomes very clear it isn't them, it's the structure.
let me tell you a simple story
1. team A developed some really nice looking graphs from customer data we already collected
2. a member of team A pointed out he is colorblind and the palette did not work for him
3. product manager took the time to research this and figure out how we might make the palette configurable for those who are colorblind
4. CEO got wind of this and called everyone involved into a meeting. The decision of the CEO was no one would ever use colors for any data presentation. The only allowable color was a shade of gray
I'd like to point out that using only a shade of gray for "colors" effectively makes the entire planet colorblind
1. team A developed some really nice looking graphs from customer data we already collected
2. a member of team A pointed out he is colorblind and the palette did not work for him
3. product manager took the time to research this and figure out how we might make the palette configurable for those who are colorblind
4. CEO got wind of this and called everyone involved into a meeting. The decision of the CEO was no one would ever use colors for any data presentation. The only allowable color was a shade of gray
I'd like to point out that using only a shade of gray for "colors" effectively makes the entire planet colorblind
As someone who is colorblind I find shades of gray better than the colors I often see. OTOH, I can see most colors just fine, it is just that some colors most people think are very different look the same to me. There are a handful of people on earth who cannot see colors at all.
The article is not very specific so I can't really tell one way or other
5 year plan is sounding really bad, though
5 year plan is sounding really bad, though
Yeap! We can't predict the future but we can all be very surprised if whatever tech is picked today is still a good fit, or even still exists, in 5 years. Especially in the javascript space.
As a sibling mentioned, this isn't really true in the JS space anymore. The JS frameworks and react/vue have pretty much won, so now you one of those UI kits and whatever build system/ancillary tooling your framework says you use. There might be churn under the covers, but the framework abstracts that away for you (though occasional migration issues are bound to come up with big changes).
it's always quiet before a storm :) It'll be fun to fast forward five years and see whether there wasn't some big disruptive change and everyone is now using rust++ for client dev or something? :D
This is a reasonable take, and I mostly share it. None of us were especially happy about that plan! However, and I alluded to this on the episode, it wasn’t actually a five year plan to get to a very specific endpoint. It was, instead a 3–5-year plan to make steady incremental changes to our entire system in ways that meant that if we realized our end destination needed to change (e.g. from React to Svelte or something not even built yet) 2 years along, that was totally fine.
It's not clear to me what ended up happening. Did they start a rewrite from scratch in React?
They started a rewrite-from-scratch of all the apps, including mobile, using a custom in-house server-driven UI framework with a Kotlin DSL backing it, without so much as doing a proof of concept to find the tricky parts first. And by “started” I mean “committed to it” before anyone had so much as written a line of code to spike it out. Which, YMMV, but to me that seemed then, and still seems now, just wild.
Yeah. I hope they're at least doing it gradually. As in, rewrite a small component and have it run side by side with the rest of the original app, then rewrite another one and so on.
Oh.
Well, that sounds horrible, and will definitely take more years than rewriting it to React.
It's more FUN though!
Well, that sounds horrible, and will definitely take more years than rewriting it to React.
It's more FUN though!
Most of the time, you have to deal with codebase. And when you finally have chance to work on a greenfield project, you will have to compromise between your opinions and wishes with those of everyone else who is involved.
> when you finally have chance to work on a greenfield project
the person that comes along next is going to moan about how much of a mess of compromises and hacks your, now, legacy codebase is
the person that comes along next is going to moan about how much of a mess of compromises and hacks your, now, legacy codebase is
I have never heard of a finger gun rewrite - what is that?
From the podcast:
> It was completely what another colleague of mine once described as being in finger guns mode, meaning like, yeah, yeah, yeah, this is going to be awesome, man, kind of finger gunning at each other without answering any of the kinds of questions about what does it look to like to operate this when we’re trying to support hundreds and hundreds of engineers…
> It was completely what another colleague of mine once described as being in finger guns mode, meaning like, yeah, yeah, yeah, this is going to be awesome, man, kind of finger gunning at each other without answering any of the kinds of questions about what does it look to like to operate this when we’re trying to support hundreds and hundreds of engineers…
Gotcha thanks.
> the next big system will be built the least-experienced.
This sounds plausible and is consistent with my experience, but I'm curious if you have a reason for why this is? Is it just that the experienced folks are already working on the established systems, and we don't know what the next big one will be? Or is management doing something intentionally that causes this? Or something else?
This sounds plausible and is consistent with my experience, but I'm curious if you have a reason for why this is? Is it just that the experienced folks are already working on the established systems, and we don't know what the next big one will be? Or is management doing something intentionally that causes this? Or something else?
Perhaps the experienced folk are "difficult" and have "opinions" about the best way to write the new system which is somewhat at odds with what management think they they need. Difficult opinions may include "no, it doesn't need to be 'AI powered' or include a blockchain or be a combinatorial explosion of configurability".
The inexperienced folk will be more "cool yeah let's do it". You can't blame them, it's how to become experienced.
The inexperienced folk will be more "cool yeah let's do it". You can't blame them, it's how to become experienced.
Tbh this was kind of my conclusion as someone new to the field (23). In technical consulting, I feel like this phenomenon is even more pronounced since A. most technical consultants don’t get the time to become domain experts in their client’s business, B. most technical consultants aren’t entirely technical experts (consultants first, developers second), and C. consulting management has a vested interested in keeping things this way because they don’t have the tools to make things different. Maybe this doesn’t happen as much in boutique consulting firms but I wouldn’t know (hire me if that’s the case).
It’s given me an idea for a two-way portal where on one side, technical developers join the platform, self-organize into independent, transparent teams (not just developers but any role in development e.g. UX designer, project manager, etc) and on the other, companies join, post their potentially cross-functional work requests which are then distributed to the relevant teams. Another important concept is that these instances can be federated(?) so existing consulting firms can integrate this new kind of work allocation into their pipelines (only their consultants/their invited companies). There are a bunch other concepts I’ve thought through but that’s where I’m at now.
I feel that a system like this provides better value to a company vs. traditional consulting. First, the responsibility of team formation, networking, and allocation is moved to the service. These three things are hard for humans to do themselves and are responsible for so much consulting admin bloat/inefficiency and companies have no choice but to play. Second, it in theory improves the quality of work teams. With this focus on smaller, self-organized teams and a UX that allows for transparency I.e. companies can view teams, their members, and track record, companies know what they’ll get and not need to roll the dice to see what available resource gets assigned to them. Moreover, I’ve read that consultancies have a tendency to throw certain resources on a large work item where they exist to just to cash in budget and write bad code. This problem is completely mitigated with this approach. Finally, it’s a decentralized work environment (I think). Besides OSS, I don’t know what other platform allows for developers to have so much autonomy and self-determination. You could argue sites like Upwork or Freelancer but these platforms seem more focused on small, individual jobs rather than larger work items which justify having larger, more competent teams.
It’s given me an idea for a two-way portal where on one side, technical developers join the platform, self-organize into independent, transparent teams (not just developers but any role in development e.g. UX designer, project manager, etc) and on the other, companies join, post their potentially cross-functional work requests which are then distributed to the relevant teams. Another important concept is that these instances can be federated(?) so existing consulting firms can integrate this new kind of work allocation into their pipelines (only their consultants/their invited companies). There are a bunch other concepts I’ve thought through but that’s where I’m at now.
I feel that a system like this provides better value to a company vs. traditional consulting. First, the responsibility of team formation, networking, and allocation is moved to the service. These three things are hard for humans to do themselves and are responsible for so much consulting admin bloat/inefficiency and companies have no choice but to play. Second, it in theory improves the quality of work teams. With this focus on smaller, self-organized teams and a UX that allows for transparency I.e. companies can view teams, their members, and track record, companies know what they’ll get and not need to roll the dice to see what available resource gets assigned to them. Moreover, I’ve read that consultancies have a tendency to throw certain resources on a large work item where they exist to just to cash in budget and write bad code. This problem is completely mitigated with this approach. Finally, it’s a decentralized work environment (I think). Besides OSS, I don’t know what other platform allows for developers to have so much autonomy and self-determination. You could argue sites like Upwork or Freelancer but these platforms seem more focused on small, individual jobs rather than larger work items which justify having larger, more competent teams.
Without negating your idea, I’d add that as a (non-IT) engineer of some 15 years, I’ve found that my first 8-10 years was building my technical capabilities, however beyond that it was really about learning how to navigate (at increasingly more complex levels) the business systems and fit the technical to the business.
Being a principal engineer, my major value is not in my technical capacity, but rather my ability to shape the technical solution to the best fit for the business - be it to do with existing tooling, processes, systems, etc. Though I can do the technical solutions (probably more efficiently, too), my value is much stronger when I steer multiple junior engineers doing the technical work whilst setting the guideposts so they align the solution to what best fits the business.
I would therefore suggest that one major hurdle would be that these self-organising teams might not be able to bring to bear their full capacity insofar as they are not familiar with the businesses to which they pair. Though you can sometimes see parallels between company processes, understanding the nuance and detail that makes them different is where so many consultancy projects fall over. Think about the canonical meme of a team of consultants coming in and spending millions developing a solution that turns out to be a lemon. It’s not for their lack of capability, it’s that it’s devilishly hard to really, truly fully understand a business’s processes, systems and cruft in a sufficient level of detail to truly shape a solution such that it remains maintainable and fit for purpose in a long-term sense.
That’s the real reason why veteran employees are worth their weight in gold. It’s that they are the custodians of the institutional memory and can bring that forward when it is needed.
This of course has the important counterpoint of the necessity of seeding new ideas into a business, and that’s an entirely different essay in itself, however suffice it to say that new ideas need to be integrated in a sustainable and controlled fashion - protected and nurtured from being driven out by the existing ways, but also moulded to integration where it’s necessary. Failure to do either is a speedrun to pain.
Edit to add: If you think that IT is somehow immune from this relative to traditional meatspace engineering disciplines, I’d say two things: 1) tech still goes through this, just sometimes operates on different timescales, and 2) go see if a COBOL greybeard collecting FU consultant wages to maintain old OT systems agrees with you.
Being a principal engineer, my major value is not in my technical capacity, but rather my ability to shape the technical solution to the best fit for the business - be it to do with existing tooling, processes, systems, etc. Though I can do the technical solutions (probably more efficiently, too), my value is much stronger when I steer multiple junior engineers doing the technical work whilst setting the guideposts so they align the solution to what best fits the business.
I would therefore suggest that one major hurdle would be that these self-organising teams might not be able to bring to bear their full capacity insofar as they are not familiar with the businesses to which they pair. Though you can sometimes see parallels between company processes, understanding the nuance and detail that makes them different is where so many consultancy projects fall over. Think about the canonical meme of a team of consultants coming in and spending millions developing a solution that turns out to be a lemon. It’s not for their lack of capability, it’s that it’s devilishly hard to really, truly fully understand a business’s processes, systems and cruft in a sufficient level of detail to truly shape a solution such that it remains maintainable and fit for purpose in a long-term sense.
That’s the real reason why veteran employees are worth their weight in gold. It’s that they are the custodians of the institutional memory and can bring that forward when it is needed.
This of course has the important counterpoint of the necessity of seeding new ideas into a business, and that’s an entirely different essay in itself, however suffice it to say that new ideas need to be integrated in a sustainable and controlled fashion - protected and nurtured from being driven out by the existing ways, but also moulded to integration where it’s necessary. Failure to do either is a speedrun to pain.
Edit to add: If you think that IT is somehow immune from this relative to traditional meatspace engineering disciplines, I’d say two things: 1) tech still goes through this, just sometimes operates on different timescales, and 2) go see if a COBOL greybeard collecting FU consultant wages to maintain old OT systems agrees with you.
For what it’s worth I don’t think what you’re describing here is the same thing I meant by “finger guns” at all! What you’re describing looks to me like good incremental development.
With my “fingers guns mode” phrase, I really had in mind a blithe disregard for any of the real-world engineering problems and constraints that we knew we would hit (because they just inhered in what we knew the solution needed to do). You don’t have to solve those right up front, but you do need to (a) acknowledge that they exist and (b) have some notion of what your approach to them will be.
I’m extremely in favor of taking a small team of veterans (who are open to doing something novel!) on a new system, as you say—let them leverage all their expertise, while being careful to avoid Second System Syndrome, to build something by using all of the knowledge about those pitfalls/problems/etc. to avoid them. (The risk there is that you can end up overfitting on previous experience! That’s real!)
With my “fingers guns mode” phrase, I really had in mind a blithe disregard for any of the real-world engineering problems and constraints that we knew we would hit (because they just inhered in what we knew the solution needed to do). You don’t have to solve those right up front, but you do need to (a) acknowledge that they exist and (b) have some notion of what your approach to them will be.
I’m extremely in favor of taking a small team of veterans (who are open to doing something novel!) on a new system, as you say—let them leverage all their expertise, while being careful to avoid Second System Syndrome, to build something by using all of the knowledge about those pitfalls/problems/etc. to avoid them. (The risk there is that you can end up overfitting on previous experience! That’s real!)
>The general success ingredient is putting a small team of veterans on making the new system. Success always comes from a small team of engineers who are both domain experts and technical experts rolled into one. (Controversially, I think that generalises; all success comes from this, even for mundane keep-the-lights-on problems. And everyone else is just slowing them down.)
Fully agree with this for all scopes of projects. Larger projects are compositions of this principle with functional communications between focus areas.
There's no financial incentives for Maintenance though, which means "leaders" have less and less incentive to fix what they have versus building from scratch. In my experience in these cases it comes down to (like most things) whomever the least flexible person in a power position decides to do.
Fundamentally the problem with all large organizations is that desires and incentives become diffuse the more people are a part of it and the "alignment" of what should happen drifts, until eventually nobody cares to maintain the original purpose.
You can argue this is good or bad or inevitable, but either way it's a structural issue between the broader world changing, and a organization that seeks consistency
Fully agree with this for all scopes of projects. Larger projects are compositions of this principle with functional communications between focus areas.
There's no financial incentives for Maintenance though, which means "leaders" have less and less incentive to fix what they have versus building from scratch. In my experience in these cases it comes down to (like most things) whomever the least flexible person in a power position decides to do.
Fundamentally the problem with all large organizations is that desires and incentives become diffuse the more people are a part of it and the "alignment" of what should happen drifts, until eventually nobody cares to maintain the original purpose.
You can argue this is good or bad or inevitable, but either way it's a structural issue between the broader world changing, and a organization that seeks consistency
I'm also generally a finger-gunner. I don't identify with the stuff OP was criticizing though; in particular I want people to poke holes in the plan: those are usually requirements we need to meet or at least talk about why we don't need to meet them. But yeah, other than that, I generally think it's easier to rewrite things than it is to understand the intricacies of a gazillion-line software project. That probably strikes a lot of HNers as problematic (this is why we keep getting new JS frameworks every day! this is why we keep reinventing email! this is exactly why we keep building gazillion-line monsters!), but I don't really mean it like that, I mean it more as a manifestation of Gall's Law [0].
To me, it's all about energy. If the energy is behind "fuck it, rewrite it", then that's what you should do. If the energy is behind, "preserve the valuable IP in the existing code" then that's what you should do. Actually I don't think it's even "should"; I think it's "will".
> Would be really nice to hear the symmetrical interview from a finger-gunner.
I found the initial framing of like "expediency vs. the right thing" not quite right. My read was that Chris just found fault with the finger-gunners' bad engineering. Here's a really good excerpt:
---
Why doesn’t Code Review just solve this? That was a direct quote from Jim. And my answer was, well it didn’t, and it won’t again in the future because people are going to make mistakes and just be better. Again, it’s not an answer, because what happens when it’s some junior on the rotation who thinks, yeah, that seems reasonable, and this PR was made by a very senior SRE?
Why am I going to question whether that value is reasonable? Like, yeah, there’s a bunch of boxes. That seems fine. Those kinds of things happen. And to that question of engineering systems, one of the things I think about a lot is, does our system only work or does this process only work? Or does this tool only succeed?
If I’m acting like a senior engineer, on his or her very best day or, or does it work if I’m a super junior engineer who’s having a bad day and we really want our systems to be workable for the latter case. And that helps all of us because sometimes, even though I’m a very senior engineer, sometimes I have bad days.
Sometimes my brain doesn’t feel like it’s working at all. Does the system still support me on those days or does it punish me? I really would like to not be punished.
---
I'm gonna try to be polite here, but Jim's "why doesn't Code Review just solve this" take here is shockingly naive to me. If he doesn't know that defects slip by even the best engineers with the best code review processes, I have to question his experience and judgement. Chris' response of "well it didn't" is all that needs to be said: we empirically can't rely on this system. Failing to recognize that is malpractice. The length Chris goes to to explain why it's malpractice speaks to his professionalism and patience, but yeah it's clear he was in a situation where someone pretty incompetent was now calling the shots.
[0]: https://en.wikipedia.org/wiki/John_Gall_(author)#Gall's_law
To me, it's all about energy. If the energy is behind "fuck it, rewrite it", then that's what you should do. If the energy is behind, "preserve the valuable IP in the existing code" then that's what you should do. Actually I don't think it's even "should"; I think it's "will".
> Would be really nice to hear the symmetrical interview from a finger-gunner.
I found the initial framing of like "expediency vs. the right thing" not quite right. My read was that Chris just found fault with the finger-gunners' bad engineering. Here's a really good excerpt:
---
Why doesn’t Code Review just solve this? That was a direct quote from Jim. And my answer was, well it didn’t, and it won’t again in the future because people are going to make mistakes and just be better. Again, it’s not an answer, because what happens when it’s some junior on the rotation who thinks, yeah, that seems reasonable, and this PR was made by a very senior SRE?
Why am I going to question whether that value is reasonable? Like, yeah, there’s a bunch of boxes. That seems fine. Those kinds of things happen. And to that question of engineering systems, one of the things I think about a lot is, does our system only work or does this process only work? Or does this tool only succeed?
If I’m acting like a senior engineer, on his or her very best day or, or does it work if I’m a super junior engineer who’s having a bad day and we really want our systems to be workable for the latter case. And that helps all of us because sometimes, even though I’m a very senior engineer, sometimes I have bad days.
Sometimes my brain doesn’t feel like it’s working at all. Does the system still support me on those days or does it punish me? I really would like to not be punished.
---
I'm gonna try to be polite here, but Jim's "why doesn't Code Review just solve this" take here is shockingly naive to me. If he doesn't know that defects slip by even the best engineers with the best code review processes, I have to question his experience and judgement. Chris' response of "well it didn't" is all that needs to be said: we empirically can't rely on this system. Failing to recognize that is malpractice. The length Chris goes to to explain why it's malpractice speaks to his professionalism and patience, but yeah it's clear he was in a situation where someone pretty incompetent was now calling the shots.
[0]: https://en.wikipedia.org/wiki/John_Gall_(author)#Gall's_law
This might be No True Scotsman, but it doesn't sound like you are a "finger gunner" after all.
After (fairly) long time in the industry, it seems to come down to:
- understanding what one's clients need (not what they ask for)
- understanding what's in the realm of the possible (technically, politically, organizationally)
- understanding what are the possible futures (good, great, and abject disaster)
If, after answering all those questions, one still advocating for a rewrite, you've shown the maturity needed to undertake the effort.
"Finger-gunning" is usually related to skipping those steps, not the ultimate decision.
(edit for formatting)
After (fairly) long time in the industry, it seems to come down to:
- understanding what one's clients need (not what they ask for)
- understanding what's in the realm of the possible (technically, politically, organizationally)
- understanding what are the possible futures (good, great, and abject disaster)
If, after answering all those questions, one still advocating for a rewrite, you've shown the maturity needed to undertake the effort.
"Finger-gunning" is usually related to skipping those steps, not the ultimate decision.
(edit for formatting)
Fair! Yeah I mean, I will say I don't need a complete plan, but unknowns make me nervous, and maybe a true finger-gunner never doubts their ability to figure it out.
Sounds to me like he made a couple unfortunate moves:
- Propose a 5 year plan (lol) - Don't lead the incident with leadership, but with blame - Speaking about problems more than solving problems (hard to do) - Lack of relationship building
On one hand, I empathize with Chris. On the other hand, this sounds to me like he just didn't know how to perform in this environment. And that's totally fine! Not everything should learn how to perform in the bureaucratic knots -- startups are simpler in this way. And there's a reason the big guys lose their edge over time, and then some exec in a board room is faced with -10% YoY loss without any truthful VPs around the table.
- Propose a 5 year plan (lol) - Don't lead the incident with leadership, but with blame - Speaking about problems more than solving problems (hard to do) - Lack of relationship building
On one hand, I empathize with Chris. On the other hand, this sounds to me like he just didn't know how to perform in this environment. And that's totally fine! Not everything should learn how to perform in the bureaucratic knots -- startups are simpler in this way. And there's a reason the big guys lose their edge over time, and then some exec in a board room is faced with -10% YoY loss without any truthful VPs around the table.
I go back and forth on how to interpret people like Chris (and likewise, myself).
I've been in this situation, and it's psychologically disorienting: I seem to be right, but am I? Are the people around me really as incompetent as they seem, as totally disinterested in learning from their colleagues, etc. etc.
Then a few years after leaving, you look back and: oh, all those people have been fired or left; they still cannot do X as an org; the agility and competence of teams I then joined really does exist etc.
So, on the one hand, it's true that there's a sort of arrogance, political incompetence, and inability to cope with environments with a pathological practice culture --- but, on the other hand, maybe that's the right reaction?
Maybe the institution is undergoing a pathological cultural period, and maybe talented, considered, passionate people should be driven mad by it. Maybe the people who arent driven mad by it are either irrelevant to productivity/growth, or worse, net deadweights.
Who knows? That's what makes being in this environment such a psychodrama -- is the situation really this bad, or am I over-reacting? "Everyone tells me...."
I've been in this situation, and it's psychologically disorienting: I seem to be right, but am I? Are the people around me really as incompetent as they seem, as totally disinterested in learning from their colleagues, etc. etc.
Then a few years after leaving, you look back and: oh, all those people have been fired or left; they still cannot do X as an org; the agility and competence of teams I then joined really does exist etc.
So, on the one hand, it's true that there's a sort of arrogance, political incompetence, and inability to cope with environments with a pathological practice culture --- but, on the other hand, maybe that's the right reaction?
Maybe the institution is undergoing a pathological cultural period, and maybe talented, considered, passionate people should be driven mad by it. Maybe the people who arent driven mad by it are either irrelevant to productivity/growth, or worse, net deadweights.
Who knows? That's what makes being in this environment such a psychodrama -- is the situation really this bad, or am I over-reacting? "Everyone tells me...."
Its very hard to affect real change without power. The people who are genuinely interested in solving problems generally don’t want to deal with the political intrigue that comes with power. So we’re left with a situation where the leaders suck for the most part.
Playing politics is hard. It’s literally the job of business management and the only way they get a paycheck. So they’re well practiced and beating them at that game is difficult.
If you’re technical, you can get a paycheck just by doing the work. Sure, you can play politics and tech, but is it worth it? Unless you are rewarded as a stakeholder, I’d say it’s not worth it.
If you’re technical, you can get a paycheck just by doing the work. Sure, you can play politics and tech, but is it worth it? Unless you are rewarded as a stakeholder, I’d say it’s not worth it.
A friend said something to me years ago that I find more and more wise as time goes on: almost everything is about aesthetics. Any given group of people (a company, a band, an advocacy group, a family) is pursuing its own sense of aesthetics. What does this look like in practice? Let's say using Elasticsearch would save $1M of engineering time and infra costs over using PostgreSQL for search but the people in charge of search have a vague aesthetic belief that Elasticsearch is garbage and they're willing to spend $1M of the company's money to not work with it. Is that nuts? Is that rational? Is that good engineering? Is that judicious use of company resources? Their bosses are the ones who make those decisions.
More frustratingly, this works the other way. You might skunkworks Elasticsearch into your org. You might have cost-saving metrics up the wazoo; you might have multiple engineer testimonials; you might have satisfied customers or growing sales; you might have better velocity metrics; you might have built profitable services on top of it. An Elasticsearch-hating engineer can convince your boss that it's a ticking time bomb--with no evidence whatsoever--and your boss can say, "hey I found out you're using Elasticsearch for all this, which is a big no-no. I love what you're doing here, but you need to migrate to Postgres".
(I know Elasticsearch flunked Jepsen; it's just an example)
You might think a few things:
- can't I show all my successes and change hearts and minds?
- isn't this how we evolve as a company? we skunkworks things, see if it works, build more on top of it, repeat?
- if we don't take risks, won't another company out-compete us at some point?
- won't some manager take notice of the inefficiency and do something about it?
Sometimes the answer is yes! But it's almost never yes for the right reasons; it's merely the case that you found a manager who hates the PostgreSQL team, or hates one particular PostgreSQL advocate, or finds it advantageous to be seen as a manager who's spearheading new initiatives, or thinks that a homogeneous tech stack is a vulnerable tech stack, etc.
This is the way we all work. We all have our irrational heuristics built up of lifetimes of experience that we hate interrogating; we naturally gravitate to others with the same irrational heuristics because they spare us that interrogation; we assault others who force that interrogation upon us because it's painful. You'll see this dynamic in political tribes, in technology choices at work, on reality TV, everywhere. You can watch us tie ourselves into knots in real time, selectively forget events or facts, or wildly misinterpret things, all to defend our aesthetic preferences. This is the psychological disorientation you're experiencing. It's not common to find people who are aware of these tendencies and--successfully--employ strategies to compensate, and it's even less common to find groups of people who do that. The probability is inverse to the number of people.
Like you (and I guess OP), I've fallen victim to this over and over again on both sides. There is no justice, because there is no God and nothing even approaching perfect competition. Some of those groups are doing great, others are deceased. I've only relatively recently figured this out, so I've now added some things to my interview questions like "tell me about a time you were very wrong" and "how do you reevaluate decisions or longstanding beliefs you have". I also try to check myself whenever I react a little too positively or negatively to something, because that's usually an indication that it's aligned or across my aesthetics and something I need to dig into.
The good news is I've had success instilling this into people I've worked with. It's sort of "a minute to learn; a lifetime to master", but it starts creating a culture of psychological safety where everyone relies on the others to help them be the best they can be.
More frustratingly, this works the other way. You might skunkworks Elasticsearch into your org. You might have cost-saving metrics up the wazoo; you might have multiple engineer testimonials; you might have satisfied customers or growing sales; you might have better velocity metrics; you might have built profitable services on top of it. An Elasticsearch-hating engineer can convince your boss that it's a ticking time bomb--with no evidence whatsoever--and your boss can say, "hey I found out you're using Elasticsearch for all this, which is a big no-no. I love what you're doing here, but you need to migrate to Postgres".
(I know Elasticsearch flunked Jepsen; it's just an example)
You might think a few things:
- can't I show all my successes and change hearts and minds?
- isn't this how we evolve as a company? we skunkworks things, see if it works, build more on top of it, repeat?
- if we don't take risks, won't another company out-compete us at some point?
- won't some manager take notice of the inefficiency and do something about it?
Sometimes the answer is yes! But it's almost never yes for the right reasons; it's merely the case that you found a manager who hates the PostgreSQL team, or hates one particular PostgreSQL advocate, or finds it advantageous to be seen as a manager who's spearheading new initiatives, or thinks that a homogeneous tech stack is a vulnerable tech stack, etc.
This is the way we all work. We all have our irrational heuristics built up of lifetimes of experience that we hate interrogating; we naturally gravitate to others with the same irrational heuristics because they spare us that interrogation; we assault others who force that interrogation upon us because it's painful. You'll see this dynamic in political tribes, in technology choices at work, on reality TV, everywhere. You can watch us tie ourselves into knots in real time, selectively forget events or facts, or wildly misinterpret things, all to defend our aesthetic preferences. This is the psychological disorientation you're experiencing. It's not common to find people who are aware of these tendencies and--successfully--employ strategies to compensate, and it's even less common to find groups of people who do that. The probability is inverse to the number of people.
Like you (and I guess OP), I've fallen victim to this over and over again on both sides. There is no justice, because there is no God and nothing even approaching perfect competition. Some of those groups are doing great, others are deceased. I've only relatively recently figured this out, so I've now added some things to my interview questions like "tell me about a time you were very wrong" and "how do you reevaluate decisions or longstanding beliefs you have". I also try to check myself whenever I react a little too positively or negatively to something, because that's usually an indication that it's aligned or across my aesthetics and something I need to dig into.
The good news is I've had success instilling this into people I've worked with. It's sort of "a minute to learn; a lifetime to master", but it starts creating a culture of psychological safety where everyone relies on the others to help them be the best they can be.
A bit of clarification from some of the nuance that (reasonably) had to get cut from the episode for time:
- Propose a 5-year-plan: yeah, “lol” is about right. It was never going to win any hearts or minds. It was also our least favorite plan. But it was also the only one we felt we could actually bring to our executive leadership given what they were telling us prior to that, which was basically “Even for a migration we are asking you for, do not slow down product iteration velocity at all.” How do you do that? Welllll…
- I’m not really sure what you’re referring to about leading the incident with blame instead of leadership. It’s possible something got lost in translation with the amount we cut, but I actually aimed very much to do the opposite. We didn’t blame the people who lowered the memory thresholds, the people who typo’d a bad value in YAML, or anyone else. I just insisted that we actually solve the root issues instead of leaving them to fester for the next poor person who happened to be around when it inevitably blew up again.
- “Speaking about problems more than solving problems”: really not sure what you mean here. I didn’t spend a ton of time bragging about what I had pulled off on the show because wow would that ever have been in poor taste, but… I did pretty well in the problems I solved there.
- Lack of relationship building: yeah, I called that out on the episode! It was my weakest area. I had a really good rapport with engineers, and failed pretty miserably at building cachet with management, especially with management above me.
I wouldn’t say in the end that it was just down to not knowing how to perform in the environment, though. Some of it was also a choice, in the end, not to perform in ways that I could see would be successful in the environment but which I simply did not believe in. I think a lot of the engineers I respect most are like this: can and will do the political dance for something they believe in… but not for things they don’t believe in.
- Propose a 5-year-plan: yeah, “lol” is about right. It was never going to win any hearts or minds. It was also our least favorite plan. But it was also the only one we felt we could actually bring to our executive leadership given what they were telling us prior to that, which was basically “Even for a migration we are asking you for, do not slow down product iteration velocity at all.” How do you do that? Welllll…
- I’m not really sure what you’re referring to about leading the incident with blame instead of leadership. It’s possible something got lost in translation with the amount we cut, but I actually aimed very much to do the opposite. We didn’t blame the people who lowered the memory thresholds, the people who typo’d a bad value in YAML, or anyone else. I just insisted that we actually solve the root issues instead of leaving them to fester for the next poor person who happened to be around when it inevitably blew up again.
- “Speaking about problems more than solving problems”: really not sure what you mean here. I didn’t spend a ton of time bragging about what I had pulled off on the show because wow would that ever have been in poor taste, but… I did pretty well in the problems I solved there.
- Lack of relationship building: yeah, I called that out on the episode! It was my weakest area. I had a really good rapport with engineers, and failed pretty miserably at building cachet with management, especially with management above me.
I wouldn’t say in the end that it was just down to not knowing how to perform in the environment, though. Some of it was also a choice, in the end, not to perform in ways that I could see would be successful in the environment but which I simply did not believe in. I think a lot of the engineers I respect most are like this: can and will do the political dance for something they believe in… but not for things they don’t believe in.
Strong +1. Also hi Zach!
I work at LinkedIn now. Chris’s role and the podcast describes ember and front end web dev. I think the LoC and build he’s referring to could be voyager-web. Our monolith flagship web app. And, there’s more systems at LinkedIn with MM lines of code and long builds example is the mid tiers, the offline data stacks, the metrics systems, KafkaKafkaKafka.
Unfortunately, 17min to build is pretty good. 17min without a transient infra failure is very good.
Anyways, AMA
Unfortunately, 17min to build is pretty good. 17min without a transient infra failure is very good.
Anyways, AMA
I worked at LinkedIn in infra. Their internal tools are a nightmare. The true definition of Jugaad.
The entire company has zero concept of testing. No QA at all. Engineers push out half baked initiatives to add to their promotion packet then move on to the next thing.
I trouble shot so many issues just in day to day usage of the internal tooling like for some reason, engineers just trying to do their jobs, are QA.
The entire company has zero concept of testing. No QA at all. Engineers push out half baked initiatives to add to their promotion packet then move on to the next thing.
I trouble shot so many issues just in day to day usage of the internal tooling like for some reason, engineers just trying to do their jobs, are QA.
Seeing you use 'Jugaad' make me unreasonably happy.
I feel like this sort of flinging shit over the wall mentality is very much becoming de-facto nowadays. Very often I have been required to just 'get shit done quick and dirty' over spending time to come up with a permanent solution.
Quick and dirty is never the short term fix it is meant to be. It always ends up being left in place until it inevitably breaks down the line.
I feel like this sort of flinging shit over the wall mentality is very much becoming de-facto nowadays. Very often I have been required to just 'get shit done quick and dirty' over spending time to come up with a permanent solution.
Quick and dirty is never the short term fix it is meant to be. It always ends up being left in place until it inevitably breaks down the line.
But are things ever permanent? Isn’t it to be expected that stuff can change in the future?
LinkedIn thinks so. Their tech stack is so over engineered and spaghetti that they spent a hundred million and years trying to move into Azure and were beaten by how bloated and unmovable the tech stack is.
Theyre still using python 2 and centOS in tons of systems.
They just started using github last year.
Their tech stack is easily over a decade old.
Theyre still using python 2 and centOS in tons of systems.
They just started using github last year.
Their tech stack is easily over a decade old.
> Theyre still using python 2 and centOS in tons of systems.
There's 1 service on py2, with a plan to migrate because of the move off of centos. centos deprecation is in progress.
There's 1 service on py2, with a plan to migrate because of the move off of centos. centos deprecation is in progress.
> There's 1 service on py2
Are you sure about that?
CentOs migration has been in progress for years lol
Are you sure about that?
CentOs migration has been in progress for years lol
TIL jugaad haha.
Yea, give everything 3x estimate because everything that can break will break along the way.
Fresh clone of master won’t build, the local build command is broken, gradle, remote build, GitHub, staging is an inside joke, prod host os upgrades, dependencies bumped in repo, http dependency’s network route changed, etc. etc. etc.
Yea, give everything 3x estimate because everything that can break will break along the way.
Fresh clone of master won’t build, the local build command is broken, gradle, remote build, GitHub, staging is an inside joke, prod host os upgrades, dependencies bumped in repo, http dependency’s network route changed, etc. etc. etc.
In my first SWE job, I was perplexed, I never felt much impact for solving complex bugs, even if they had side effects like reducing latency for everyone in our infra. Anyways, I realized it's like you say - you get rewarded for pushing features and shipping stuff. This incentive can easily become a 'tragedy of the commons' situation.
> Unfortunately, 17min to build is pretty good.
My biggest pet peeve with this is that a lot of people see these values as immutable, and because building/testing/running every single push takes too long, we should not do that.
As opposed to making builds faster, or build infra faster/cheaper.
My biggest pet peeve with this is that a lot of people see these values as immutable, and because building/testing/running every single push takes too long, we should not do that.
As opposed to making builds faster, or build infra faster/cheaper.
There are mostly only a few ways to make builds faster.
* Ship less code (Very hard to get a large org to do)
* Rebuild the same things less often. Requires using build tooling that supports this org wide, Still hard to do but not as hard as shipping less code.
* Build more pieces in parallel. Also requires using build tooling that supports it org wide as well as structuring the code in such a way that it is amenable to parallel building.
These are all expensive investments that will pay off huge if you do them but can be quite difficult to justify until you reach a certain size.
* Ship less code (Very hard to get a large org to do)
* Rebuild the same things less often. Requires using build tooling that supports this org wide, Still hard to do but not as hard as shipping less code.
* Build more pieces in parallel. Also requires using build tooling that supports it org wide as well as structuring the code in such a way that it is amenable to parallel building.
These are all expensive investments that will pay off huge if you do them but can be quite difficult to justify until you reach a certain size.
Can you expand on "ship less code"? I think I'm agreeing with you but I'm taking it to mean ship smaller pieces of code, not overall LoC.
I mean both. Smaller pieces but also less of those pieces. It is my experience that many systems code size is less purely a function of what they need to do than a complex set of layered components and systems in the name of ease and fear of NIH syndrome.
Sometimes doing the same thing with simpler components and fewer components is just better.
Sometimes doing the same thing with simpler components and fewer components is just better.
[deleted]
These kind of speedups at LinkedIn scale all fall off the chart at the bottom left of this scale:
https://xkcd.com/1205/
Nonetheless, leadership will never give it priority.
https://xkcd.com/1205/
Nonetheless, leadership will never give it priority.
The top 1/3rd of Eng don’t see it as immutable. But the stack can consist of 10 other teams tools, and you’re not rewarded for the amount of time you’re spending to fix other people’s stuff. So it goes on
Just for another perspective, I've worked at a few companies including LinkedIn as a backend developer and I think LinkedIn is probably around 70-80th percentile in code quality.
There was significant emphasis on code quality at least on the team I was on, and an ever improving culture.
I did work on one voyager task though and I remember it being a nightmare
There was significant emphasis on code quality at least on the team I was on, and an ever improving culture.
I did work on one voyager task though and I remember it being a nightmare
[deleted]
>Anyways, AMA
What precisely is LinkedIn trying to be?
Seems like it's turning into 2007 Facebook. Is that intentional?
What precisely is LinkedIn trying to be?
Seems like it's turning into 2007 Facebook. Is that intentional?
They're trying to be a site that makes a lot of money from ads. To do that they need to get people to spend a lot of time there. To do that they need an infinitely scrolling feed that has addictive characteristics. What else are people going to spend a lot of time on there? Scrolling through their contacts?
Oh, you meant what are they trying to be that's helpful to the user? Doesn't matter.
Oh, you meant what are they trying to be that's helpful to the user? Doesn't matter.
I wonder, is "being facebook" is that an organic goal LinkedIn came up with, or did they pick that up from their users?
I suspect the user base has largely driven it. From the beginning it seemed that regardless of the stated purpose users wanted to use LinkedIn as a "different facebook". Personally I hate that, but a lot of people have been doing it for many years.
I suspect the user base has largely driven it. From the beginning it seemed that regardless of the stated purpose users wanted to use LinkedIn as a "different facebook". Personally I hate that, but a lot of people have been doing it for many years.
> I suspect the user base has largely driven it.
In what way? Users/customers do not generally drive development. Indirect measurements of users do, such as measuring "engagement".
At best, what likely happened was A/B testing showed "what users want", which rarely and only by chance intersected with what users actually want, and instead showed over and over that socials media patterns (light and dark) hijacking attention drives engagement.
In what way? Users/customers do not generally drive development. Indirect measurements of users do, such as measuring "engagement".
At best, what likely happened was A/B testing showed "what users want", which rarely and only by chance intersected with what users actually want, and instead showed over and over that socials media patterns (light and dark) hijacking attention drives engagement.
Oh I disagree greatly.
If people didn't want to use twitter, they wouldn't and it would be gone. But users do use it, and even the folks who tell me they're upset ABOUT Twitter, most choose to be there.
Whatever the reason they make that choice, at some point that's on them.
If people didn't want to use twitter, they wouldn't and it would be gone. But users do use it, and even the folks who tell me they're upset ABOUT Twitter, most choose to be there.
Whatever the reason they make that choice, at some point that's on them.
If the choice is "use the platform everyone else is on" vs "don't exist", many people will choose the former.
That's not the same thing as wanting to use the platform.
That's not the same thing as wanting to use the platform.
If you feel like not being on twitter means "don't exist" as an individual... I feel like that also is a choice / lifestyle you chose.
I can't imagine letting a service like that define things for me like that.
I can't imagine letting a service like that define things for me like that.
> I can't imagine letting a service like that define things for me like that.
Yes, that's very clear. Everyone is super impressed with how free your thinking is.
However, you completely missed the point of my comment by interpreting my hyperbole so literally.
Yes, that's very clear. Everyone is super impressed with how free your thinking is.
However, you completely missed the point of my comment by interpreting my hyperbole so literally.
Nah, basically every other social type thing got screwed by FB convincing everyone that MAU's (i.e. the metric FB looked best on) was the right metric, when it clearly wasn't.
Like, LinkedIn only needs MAUs who are trying to sell something or looking for a job, they 100% don't need people to log in every day (as their ads business is only a small proportion of revenue).
Like, LinkedIn only needs MAUs who are trying to sell something or looking for a job, they 100% don't need people to log in every day (as their ads business is only a small proportion of revenue).
I don't buy entirely into "FB did it" kinda thing. They found it ... but users LOVE it. That's a choice in the last part.
Some users love it, the type who want to sell their every thought online to anyone who can tolerate them.
Other users (almost everyone I know) absolutely loathe it. It is hands-down the worst service I have an account with, but it's also practically required to get a job if you don't have lucky personal connections. I was hoping TFA was actually about leaving as a user, that it might be inspiration for me to free myself.
Other users (almost everyone I know) absolutely loathe it. It is hands-down the worst service I have an account with, but it's also practically required to get a job if you don't have lucky personal connections. I was hoping TFA was actually about leaving as a user, that it might be inspiration for me to free myself.
They made loads of product changes to get people to engage in that way. As an example, showing who viewed your profile in the email vs needing to log in.
I've always thought LinkedIn was just Facebook for Business™ - has it ever been anything else?
It was always to some extent, but the feed was a lot more formal IMO at the beginning. Content was more like actual articles and links.
But I think users just love / gravitate towards a more facebook style feed.
But I think users just love / gravitate towards a more facebook style feed.
Part of that might be the "reward for spamming" - any "social media" site is always amazing at the beginning, because only the die-hards know about it and the spammers have no return on investment.
LinkedIn has slightly different style of spammer, but it's basically the end game for anything.
LinkedIn has slightly different style of spammer, but it's basically the end game for anything.
Yes. We hired everyone from meta starting in 2020 and so far this strategy is holding true
Interesting ok. Good luck
Why did the codebase end up like this? Is it a lack of hiring for platform or developer tooling teams?
It's because LinkedIn is old in terms of web-age. Really old. They, along with thousands of other companies, had the need for a reactive web front-end when JavaScript had not yet matured to where it is now.
this plus lack of performance eval/culture/strategy to grow responsibly and enable future success
its a great learning experience though despite what people say about the inability to learn the new hot hot tech. the nuance of software development is the decisions that other people make, that is inescapable and is a skill worth developing. i'm not buying the "just do a startup" because i think its a cop out.
its a great learning experience though despite what people say about the inability to learn the new hot hot tech. the nuance of software development is the decisions that other people make, that is inescapable and is a skill worth developing. i'm not buying the "just do a startup" because i think its a cop out.
In my experience, a good platform team will increase velocity a bit while a bad platform team will tank it. Perpetual migrations and re-writes, "simplified" platforms that fail to meet future developer needs, half-abandoned NIH systems as people move teams, ownership moats if product teams try to touch platform code, etc, etc. So if a company has been burned a couple times they may decide to just not bother.
I bet in most places it all starts with a new innovative system of (self) assessment that relies on number of features/VI/KPI tied to promotions.
Whats part two of the story? Did LinkedIn move to React? Did the initiative fail, or is it ongoing?
Preface: I’m not in front end or flagship teams so I’m saying hearsay.
The initiative Chris refers to is still alive but likely hasn’t made any meaningful progress. These things tend to have a lot of fan fair and then suddenly leave the front conscious of the company as we get a new GDPR/DMA fire to put out. at that point it will be dead. Or, it will stay alive for 2+ years as slowly but surely 50% of clients migrate to it and see even poorer performance.
There was a blog here a few months ago about migration from restli to grpc. It’s still going on, and nearly every app i see is still restli.
The initiative Chris refers to is still alive but likely hasn’t made any meaningful progress. These things tend to have a lot of fan fair and then suddenly leave the front conscious of the company as we get a new GDPR/DMA fire to put out. at that point it will be dead. Or, it will stay alive for 2+ years as slowly but surely 50% of clients migrate to it and see even poorer performance.
There was a blog here a few months ago about migration from restli to grpc. It’s still going on, and nearly every app i see is still restli.
That's understandable, thanks for the info!
dude, you are sharing way too much about us publicly. the status of restli -> grpc and talking about DMA? seriously?
you realize stylometry exists, right? even the phrase 'fan fair' is a strong signal.
you realize stylometry exists, right? even the phrase 'fan fair' is a strong signal.
Sounded like he he was talking about a public blog post right?
This sounds like my exact experience at another big tech company
The big rewrite. Risky even on manageable codebases, and the leftovers never seem to fully go away. Who wants to score points on re-writing a tucked-away settings page several years into these?
I've seen so many attempts at these you'd think we'd have a framework for rewriting codebases. We don't. Automated codemods require a consistency that few adhered to. The code patterns evolved so much over time, it's like looking at tree rings.
We're basically putting code in boxes, re-arranging the boxes, and rightfully saying some arrangements are more efficient. How come we haven't found a better way? Automations operate at the code level, but not at the box level.
I've seen so many attempts at these you'd think we'd have a framework for rewriting codebases. We don't. Automated codemods require a consistency that few adhered to. The code patterns evolved so much over time, it's like looking at tree rings.
We're basically putting code in boxes, re-arranging the boxes, and rightfully saying some arrangements are more efficient. How come we haven't found a better way? Automations operate at the code level, but not at the box level.
Oh boy, been there. This is Conway’s law in action - they are about to go and create the same code soup again - because the org won’t have changed. Having been in the same boat my lesson is positive engineering initiatives can only come top down - through some very senior champion. You can’t change an organisation from the bottom up - and it’s the organisation that produces the code base.
Top down comes with its own risks. The bottom line is big ideas and big change can only happen with both the champion at the top, as well as good understanding at the bottom, and a critical mass of good alignment and competence throughout the middle layers. Though Conway's Law is immutable, it doesn't necessarily depend on a formal org chart, it can be worked with by simply creating ad-hoc communication structures between the right tech leads and competent managers. A handful of technically weak or empire-building managers in the middle can fuck the whole thing pretty easily though, and depending on the lifecycle of your company it may be a lost cause due to Pournelle's Iron Law of Bureaucracy.
Although it’s not always necessary to go through the middle - you can do the old switcheroo and build a parallel org in your image, emerge from the shadows and duke it out with the old structure.
If you come at the king you best not miss. Fail at taking down the old structure and those relationships are so destroyed they will come after you. I’ve seen groups of consultants become unemployable across the whole industry by failing to take down the old guard. Word spreads.
> as well as good understanding at the bottom, and a critical mass of
good alignment and competence throughout the middle layers.
How do you hold the middle together? Especially under such turbulence? "Strong" leadership isn't enough.
Reflective practice [0] is one way to build that. It's common in medical and some safety-critical civil/military spheres but strangely missing from many corporate structures. I talk about it here with regard to a more mature cybersecurity stance [1]. It intersects with Agile ideas but doesn't make much noise in software engineering project management as much as it should? Anyone have this in their org?
[0] https://en.wikipedia.org/wiki/Reflective_practice
[1] https://cybershow.uk/blog/posts/reflective
How do you hold the middle together? Especially under such turbulence? "Strong" leadership isn't enough.
Reflective practice [0] is one way to build that. It's common in medical and some safety-critical civil/military spheres but strangely missing from many corporate structures. I talk about it here with regard to a more mature cybersecurity stance [1]. It intersects with Agile ideas but doesn't make much noise in software engineering project management as much as it should? Anyone have this in their org?
[0] https://en.wikipedia.org/wiki/Reflective_practice
[1] https://cybershow.uk/blog/posts/reflective
Have fewer middle managers?
Very good comment
That highlights the greatest repeated tragedy to befall software: poor leadership. For some bizarre reason developers almost always think they can fix people problems with better tools. For example if all your developers suck give them a popular framework. It’s just an excuse to avoid dealing with people that provides license for the children to run the daycare.
If you want excellence set high standards defined by rules that to hold people accountable and impose ownership (liability/reward). It’s not complicated but it requires not being terrified of assertiveness and confrontation from the top down.
If you want excellence set high standards defined by rules that to hold people accountable and impose ownership (liability/reward). It’s not complicated but it requires not being terrified of assertiveness and confrontation from the top down.
> It’s not complicated but it requires not being terrified of assertiveness and confrontation from the top down.
It requires the professional communication skills that are not taught in a STEM education. That's why this is all so difficult: we are not taught how to communicate effectively with one another, and the entire tech landscape has poor communicators misinforming and misleading one another.
It requires the professional communication skills that are not taught in a STEM education. That's why this is all so difficult: we are not taught how to communicate effectively with one another, and the entire tech landscape has poor communicators misinforming and misleading one another.
That cannot be over-stated enough. I learned my communication skills in the military while building my software career. Those worlds are so incredibly different that I have spent most of my professional life lost.
On one hand my expectations for excellence in software are alien to my peers who just want to complete a JIRA task by any means possible. Yes, it is possible to deliver superior results for an entire day's effort in about 2 hours provided you aren't framed by all kinds of institutional roadblocks and unnecessary abstractions. I am scared to death to actually communicate my opinions regarding software because people tend to whine about how hard life is. For this reason I have abandoned my JavaScript career.
Then on the military side I have to unlearn the high agreeableness that is so necessary for not getting fired in software. By high agreeableness I mean keeping your opinions to yourself, not voicing concerns, treating everything with kid gloves, and making everything as easy as possible. The military side is a much safer environment that expects brutal honesty. Nobody there believes themselves to be god's gift to the universe and they are always in a learning mode, which means they are coach-able and will not whine about corrections and mentorship.
On one hand my expectations for excellence in software are alien to my peers who just want to complete a JIRA task by any means possible. Yes, it is possible to deliver superior results for an entire day's effort in about 2 hours provided you aren't framed by all kinds of institutional roadblocks and unnecessary abstractions. I am scared to death to actually communicate my opinions regarding software because people tend to whine about how hard life is. For this reason I have abandoned my JavaScript career.
Then on the military side I have to unlearn the high agreeableness that is so necessary for not getting fired in software. By high agreeableness I mean keeping your opinions to yourself, not voicing concerns, treating everything with kid gloves, and making everything as easy as possible. The military side is a much safer environment that expects brutal honesty. Nobody there believes themselves to be god's gift to the universe and they are always in a learning mode, which means they are coach-able and will not whine about corrections and mentorship.
Although I do not have military experience, I've spent some time writing software deployed to military security clients, and my interactions with technical armed forces staff has had remarkably good communications. To the degree it changed my attitude about the armed forces in a significant positive direction, and not just the tech side.
> The military side is a much safer environment that expects brutal honesty.
sounds refreshing but debatable whether an environment of honest brutality can be considered safe
sounds refreshing but debatable whether an environment of honest brutality can be considered safe
Why wouldn’t it?
was referring to the fact that military personell tend to be trained fighters with a heightened ability to inflict physical damage if threatened or distressed.
Statements of poor communication always make me wonder if the shirky principle might be part of the problem
I do not believe so. This lack of communication skills is also due to a lack of recognition of the value good communications engenders. An undergraduate business school education pragmatically touches on professional communications, but only as far as how to motivate and control people, not how to effectively communicate - as in a shared synchronization of understanding, which is what professional communications is all about. Colleges of communications teach professional communications as their foundation, before the various media majors one graduates within. The more I look at the levels of communications in general within society, our civilization has not yet recognized how critical communications are to every aspect of life, not just business and careers. Good communications are like fire, capable of destroying and capable of being the foundation greater things are built, and the lack of this recognition amazes me.
the recognition is definitively there, in ppl, but lacking in most organizations esp. industries.
interesting if endeavors with nlp and llm's can help to spread nuance or commodify motivation and control.
interesting if endeavors with nlp and llm's can help to spread nuance or commodify motivation and control.
Interesting, as that is what my current work is all about: integrating LLMs into ordinary software people use everyday, and those LLMs serve multiple purposes: a) they know the software they are integrated and can offer support in how to use the software, b) a 'psychologist AI' looks at what the user is doing and makes an assessment of how well they understand both the software they are using and the content of their use of the software, that assessment is used to modify the amount of help and the language used when helping, c) both the assessment from b and the content from whatever they are authoring is used to generate an expert in the subject matter of their content, whom is then available in a chat interface with access to rewrite the content the user is authoring upon their request. And d) when a "completed" document is viewed (not in an editor) a slightly different expert is generated based on the context of the document, whom is available for extended Q&A of that document and it's subject.
I'm calling this work 'human comprehension support'.
I'm calling this work 'human comprehension support'.
From the point of view of upper management, the squeaky wheel gets replaced.
On the other hand, the non-squeacky wheel gets to collect their salary for a long long time.
On the other hand, the non-squeacky wheel gets to collect their salary for a long long time.
If you want to dig into Conway's law and its implications, I can't recommend this video essay by Casey Muratori enough: https://youtu.be/5IUj1EZwpJY?si=dPxsXieBwZsP0PPP
Fair warning though, you'll lose all hope that companies like Microsoft will ever manage to produce anything that they don't then ruin.
Fair warning though, you'll lose all hope that companies like Microsoft will ever manage to produce anything that they don't then ruin.
>Fair warning though, you'll lose all hope that companies like Microsoft will ever manage to produce anything that they don't then ruin.
you are implying that we still had any hope left in such endeavors.
you are implying that we still had any hope left in such endeavors.
I've witnessed what Casey calls "Conway's Nightmare" (at 32:04 in his video) firsthand- it is definitely real. Basically, a software system isn't necessarily just a reflection of an organization, it's an amalgamation of all of the organizations that worked on the software system in the past.
Does Conway’s law imply that an organization of size 1 produces a system with components that are optimally integrated?
I sometimes wonder what would happen if a company hired 100 software engineers, gave them all the same task to program the entire thing independently on their own, and then paid them as a monotonic function of the ranking of their output against their peers. (Obviously, there are limits on what type of product could be developed here, but I think this still encompasses far more company products than you would suspect.)
I sometimes wonder what would happen if a company hired 100 software engineers, gave them all the same task to program the entire thing independently on their own, and then paid them as a monotonic function of the ranking of their output against their peers. (Obviously, there are limits on what type of product could be developed here, but I think this still encompasses far more company products than you would suspect.)
> Does Conway’s law imply that an organization of size 1 produces a system with components that are optimally integrated?
It’s creative work, like anything else. 1 is not necessarily always the right size, but 100 certainly isn’t. If you look at musical bands usually a sweet spot is between 1-6. You can add more in eg an orchestra or choir but everyone’s not involved in the creative work.
Then it depends on how well you vibe with people. If you have extremely unique ideas and thoughts, you may be better off alone. If you find peers with similar values, preferences and experiences, you might be fine with 5 people.
It’s creative work, like anything else. 1 is not necessarily always the right size, but 100 certainly isn’t. If you look at musical bands usually a sweet spot is between 1-6. You can add more in eg an orchestra or choir but everyone’s not involved in the creative work.
Then it depends on how well you vibe with people. If you have extremely unique ideas and thoughts, you may be better off alone. If you find peers with similar values, preferences and experiences, you might be fine with 5 people.
This feels like something that is just crazy enough to work.
Most teams I've been on or managed were at their optimal efficiency with three people. Just enough to keep any one person from making too many poor decisions, but not to many to spend half a day in meetings.
Most teams I've been on or managed were at their optimal efficiency with three people. Just enough to keep any one person from making too many poor decisions, but not to many to spend half a day in meetings.
An organization of size 1 has other constraints, like time.. But also, Conway's law applies outside the organization as well, communication constraints shape society at large and the way a society is organized limits the communication channels available.
On the topic of Microsoft/Linkedin, aren't they the ones who used to or maybe still assign the same task to multiple teams and let the teams compete for who can deliver? That does sound vaguely like what you propose.
On the topic of Microsoft/Linkedin, aren't they the ones who used to or maybe still assign the same task to multiple teams and let the teams compete for who can deliver? That does sound vaguely like what you propose.
Run all 100 in parallel in production and monitor for inconsistencies.
I remember one time we had to provide some important numbers to auditors. A data analyst on the team wrote some queries to collect this information, but my manager decided to write his own queries independently as a consistency check. The numbers didn’t match at all.
So he asked me to write my own queries as well to see who was right. Lo and behold, my numbers didn’t match either of theirs at all.
So he asked me to write my own queries as well to see who was right. Lo and behold, my numbers didn’t match either of theirs at all.
3 wrongs makes for a much more interesting conversation than 1
Fred brooks believes the Max is 1 or 2 (if they have experience together), and explains how to scale an org around those people.
Does he say what to do when they die, quit, or retire?
So clever! The brief answer is you have many staff who have knowledge but not creative authority. If you think about it for a few minutes I think you’ll find some options.
The best way to de-risk a project is to run three at once. Just don't let the teams find out about the other ones.
Oh. My. God.
This video explains everything I've seen in the enterprise IT of a huge government org that was the result of a long series of back-to-back mergers and splits.
It's exactly this!
Conway's law, but over time.
Dysfunction not just because of the large hierarchy, but also because of the built up layers of history. Some elements still limping along, some essentially amputated and mere phantom limbs of the org tree now, some outright dead and spreading miasma and corruption from their decay.
I knew or suspected most of this, but to hear it said out loud so clearly has crystallised my understanding of it.
This video explains everything I've seen in the enterprise IT of a huge government org that was the result of a long series of back-to-back mergers and splits.
It's exactly this!
Conway's law, but over time.
Dysfunction not just because of the large hierarchy, but also because of the built up layers of history. Some elements still limping along, some essentially amputated and mere phantom limbs of the org tree now, some outright dead and spreading miasma and corruption from their decay.
I knew or suspected most of this, but to hear it said out loud so clearly has crystallised my understanding of it.
Yeah once it hits, it really explains so much of the issues in large organizations (and explains a lot about small organizations / indie development as well). For example, why is it that mergers and buyouts rarely work out? Conway's law largely explains it! When a product is bought by an organization, the result is an integration over time of both organization org-charts combined. If the companies are both large, this immediately becomes an intractable mess. If the original company was small and the new company is large, it _also_ becomes intractable because now different teams will have to communicate to manage or split components that don't fit the communication patterns of the new organization, and will lead to large refactors and rewrites and in the end the essential qualities of the original software will inevitably change.
Even if you know this, there is nothing you can do - you have to organize yourself somehow and how you do that can't be left up to just mirror whatever this piece of software that was brought in happens to be organized - because _that too_ doesn't reflect an actual org chart but is an integration over time of all org charts that have touched it.
I don't even know how to think about the implications this has for the use of open source software and how open source is developed, like... I think there are huge opportunities for research into Conway's law and how it relates to software development practices, software quality, etc.
Even if you know this, there is nothing you can do - you have to organize yourself somehow and how you do that can't be left up to just mirror whatever this piece of software that was brought in happens to be organized - because _that too_ doesn't reflect an actual org chart but is an integration over time of all org charts that have touched it.
I don't even know how to think about the implications this has for the use of open source software and how open source is developed, like... I think there are huge opportunities for research into Conway's law and how it relates to software development practices, software quality, etc.
I have this naively optimistic hope that AIs will allow orgs to scale units past Dunbar’s number.
We humans can’t effectively communicate with groups larger than about 7 people, but AIs have no such limits. We could all simply talk to one manager that can integrate everything and everyone into a unified whole.
It’s like the ship Minds in the Culture series having hundreds or even thousands of conversations at once.
We humans can’t effectively communicate with groups larger than about 7 people, but AIs have no such limits. We could all simply talk to one manager that can integrate everything and everyone into a unified whole.
It’s like the ship Minds in the Culture series having hundreds or even thousands of conversations at once.
The more the initial fade of AI assisted work sets in, and given the inherent vagueness and unpredictability of managing, I'm eagerly awaiting not my job, but my bosses job being replaced by AI. There's no need for exactness, but superficial clarity, decisiveness and seeming coherence.
That's an interesting thought, yeah... technology and communication are definitely interwoven, things are possible today that were impossible before computers simply due to communication constraints.
One could imagine a large number of practically autonomous small teams organized "automatically", more mirroring a neural network than a hierarchy.
The problem is always communication because it is the means to cooperate. The root of many issues in software development is the simple fact that instead of letting the required communication pathways define the organization, it is the organization which defines the pathways and through that causes communication obstructions.
"Not Just Bikes" has a good few videos, including https://www.youtube.com/watch?v=n94-_yE4IeU and a couple more that talk about problems that larger roads effectively cause more traffic ("induced demand", "traffic generation"). Organizational structures are like roads, and like roads they can get overloaded, which in turn means traffic needs to be reduced. There is even communication jam, and to combat that something like enforced communication reduction (lower information throughput), etc. to keep this manageable. That also causes decision making being done with less and less information the more steps are included in a communication chain (like upwards/downwards in a hierarchy), which in turn means the quality of decision making is severely hampered by it.
This whole mess is also the reason why the agile manifesto puts humans before processes and other such things, in fact it implies you change even the organizational setup to fit the project, not the other way around. But in the space of "managerial feudalism" (David Graeber) this is pretty much impossible to pull off.
"Not Just Bikes" has a good few videos, including https://www.youtube.com/watch?v=n94-_yE4IeU and a couple more that talk about problems that larger roads effectively cause more traffic ("induced demand", "traffic generation"). Organizational structures are like roads, and like roads they can get overloaded, which in turn means traffic needs to be reduced. There is even communication jam, and to combat that something like enforced communication reduction (lower information throughput), etc. to keep this manageable. That also causes decision making being done with less and less information the more steps are included in a communication chain (like upwards/downwards in a hierarchy), which in turn means the quality of decision making is severely hampered by it.
This whole mess is also the reason why the agile manifesto puts humans before processes and other such things, in fact it implies you change even the organizational setup to fit the project, not the other way around. But in the space of "managerial feudalism" (David Graeber) this is pretty much impossible to pull off.
The tragedy of agile is that the practices that are labelled agile in practice tend to exemplify the exact opposite of everything that the manifesto advocates..
You might be correct, but the AI minds that you are contemplating don't exist yet, and there is no reason to think that they will be developed from current LLMs.
Once again, seizing the term AI to mean LLMs and other current generative techniques has poisoned clear thinking. When we think "AI", we are thinking about HAL and Cortana and C3PO, which is not what we actually have.
Once again, seizing the term AI to mean LLMs and other current generative techniques has poisoned clear thinking. When we think "AI", we are thinking about HAL and Cortana and C3PO, which is not what we actually have.
interestingly positive, I think I might agree with you. It seems the nr.1 problem of good leaders in organizations is that they can't be everywhere at once. But an AI can be.
I've watched a lot of Casey Muratori's other presentations but just happened to find this one last week and I wholeheartedly agree. Like many people I'd heard of Conway's Law but always imagined it as a pithy truism and hadn't thought that the effects could run so deep.
Casey's example is focused on a (frankly, ridiculously) large organisation but I've seen the same thing in numerous small companies with just 20-30 developers, and it's hard not to imagine that this is universal, which is a pretty depressing thought.
Recently I've been involved in a new project where teams were set up to be 'domain-specific' with the idea that this somehow avoided the issues of Conway's Law, but this just feels like exactly the same problem because team structures and the software that they end up producing is siloed in exactly the same way as the organisation structures that existed at the time.
Casey's point that the final design is inherently limited by the lack of high-bandwidth communication between the groups of people that need it most is also fairly demotivating, personally.
Tangentially, having been writing more Golang recently this made me think of Golang's mantra (or was it Rob Pike's quote) of "Don't communicate by sharing memory; share memory by communicating.". Go/CSPs concurrency and sharing model is interesting, but I wonder if it's a fair analogue to compare sharing memory by communicating as low-bandwidth communication, and communication by sharing memory as the high-bandwidth communication that seems to be needed for effective designs.
Casey's example is focused on a (frankly, ridiculously) large organisation but I've seen the same thing in numerous small companies with just 20-30 developers, and it's hard not to imagine that this is universal, which is a pretty depressing thought.
Recently I've been involved in a new project where teams were set up to be 'domain-specific' with the idea that this somehow avoided the issues of Conway's Law, but this just feels like exactly the same problem because team structures and the software that they end up producing is siloed in exactly the same way as the organisation structures that existed at the time.
Casey's point that the final design is inherently limited by the lack of high-bandwidth communication between the groups of people that need it most is also fairly demotivating, personally.
Tangentially, having been writing more Golang recently this made me think of Golang's mantra (or was it Rob Pike's quote) of "Don't communicate by sharing memory; share memory by communicating.". Go/CSPs concurrency and sharing model is interesting, but I wonder if it's a fair analogue to compare sharing memory by communicating as low-bandwidth communication, and communication by sharing memory as the high-bandwidth communication that seems to be needed for effective designs.
[deleted]
What’s broken with current office 365? It’s a good product and service.
This is what you see from the outside. LinkedIn also looks good from outside. This is the entire point of the story.
If the user experience is good, then the product is good. Users don't care about build time or internal dependency management.
Build time & internal dependency management are just a few markers used to predict where user experience will go in the future
Making something look good is very popular but entirely different from actual quality.
> LinkedIn also looks good from outside.
I partially disagree. IIRC their search is broken, and their job alerts are pretty lousy at de-duplicating advertised positions.
And IIRC they hijack the back button, which I find pretty offensive.
I partially disagree. IIRC their search is broken, and their job alerts are pretty lousy at de-duplicating advertised positions.
And IIRC they hijack the back button, which I find pretty offensive.
The existence of the product is broken. Office suites should not be web applications.
To be fair, it’s only a web application if you want it to be. I have all the usual office products installed locally. When I get a link that opens a document in the browser I click open in app and off it goes.
Microsoft has moved very quickly to add LLM functionality to multiple products
Literally the opposite of LinkedIn. I’m approximately user 5000 on linkedin, and since I joined almost nothing useful has been added, despite a huge increase in complexity
Literally the opposite of LinkedIn. I’m approximately user 5000 on linkedin, and since I joined almost nothing useful has been added, despite a huge increase in complexity
LinkedIn and Microsoft are the same, that said I personally think the rush to push LLMs into everything is entirely consistent with my prediction and with the internal structure of Microsoft (current and historical).
Come on now, that's not true. Every week LinkedIn adds a new email marketing category which you haven't opted-out of yet, because it didn't exist before.
Exactly. Remember that we are not customers, we are the product. Presumably, LinkedIn is adding a lot of useful features for their actual customers, they're just not visible to us.
And even then - the senior champion might as well burn out on it, even when they do manage to pass the initiative. Been there too.
I think it is more nuanced, you can also change the organization bottom up, but it needs to be new stuff that does not exist or has it infancy. Changing existing stuff is very hard even top-down.
Have you changed an organisation from the bottom up? In two decades I’ve never seen it happen in organisations larger than 50 people.
You can change something, like how an engineering team works and how an organisation does DevOps and other things that management doesn’t really know anything about and trust their employees on. But moving an organisation into something like team topologies which is a more modern expansion on Conway, is virtually impossible from the bottom up in my experience. Change management in general is very hard both directions, but it’s much harder going upwards because going upwards means you don’t really have the support of management above you. Maybe they’ll humour you but to make an actual impact you’re going to need them onboard. You’ll even have to reach pretty far up top if you want to inflict lasting culture changes as things carried by a single (or few) managers often dies with them.
My career has sort of put me in a role where I help non-tech startups transition from that to medium or enterprise size. I solely focus on the tech journey though, as I know I won’t have too much impact on culture or work processes on the grander scale. Often this leads to a semi-team topologies approach as the tech teams divide systems into independent business related services, but as a whole I don’t expect any changes to reach outside of the development departments.
You can change something, like how an engineering team works and how an organisation does DevOps and other things that management doesn’t really know anything about and trust their employees on. But moving an organisation into something like team topologies which is a more modern expansion on Conway, is virtually impossible from the bottom up in my experience. Change management in general is very hard both directions, but it’s much harder going upwards because going upwards means you don’t really have the support of management above you. Maybe they’ll humour you but to make an actual impact you’re going to need them onboard. You’ll even have to reach pretty far up top if you want to inflict lasting culture changes as things carried by a single (or few) managers often dies with them.
My career has sort of put me in a role where I help non-tech startups transition from that to medium or enterprise size. I solely focus on the tech journey though, as I know I won’t have too much impact on culture or work processes on the grander scale. Often this leads to a semi-team topologies approach as the tech teams divide systems into independent business related services, but as a whole I don’t expect any changes to reach outside of the development departments.
I'm not disagreeing on your overall point, but it can be done (I've done it with 130+, so there's a lot of incumbent process and numerous senior managers to align).
The key is having a champion who can tie change in with higher level desires.
In these orgs there's often an external (as in stakeholders external to prod eng) perception that "engineering is slow, product doesn't deliver, they're preventing us delivering our OKRs", and that can be used as leverage for change.
You can't go dark and stop delivery, but you can usually carve out a 10 - 20% allowance for change (that doesn't sound much, but is 1 - 2 days every 2 weeks, which quickly adds up). Start small, show success in impacting metrics that external stakeholders care about, then next quarter push for more.
I've focused myself in a similar area as you, but actually lean into processes and people, whilst still guiding tech - maybe we should chat!
The key is having a champion who can tie change in with higher level desires.
In these orgs there's often an external (as in stakeholders external to prod eng) perception that "engineering is slow, product doesn't deliver, they're preventing us delivering our OKRs", and that can be used as leverage for change.
You can't go dark and stop delivery, but you can usually carve out a 10 - 20% allowance for change (that doesn't sound much, but is 1 - 2 days every 2 weeks, which quickly adds up). Start small, show success in impacting metrics that external stakeholders care about, then next quarter push for more.
I've focused myself in a similar area as you, but actually lean into processes and people, whilst still guiding tech - maybe we should chat!
> new stuff that does not exist or has it infancy.
If it was new stuff, then the people doing it must not be very low at the bottom (by virtue of the org being quite shallow - such as a startup).
Even new stuff in a big company with lots of layers of management cannot make new stuff with excellence. Eventually, probably sooner rather than later, it gets polluted.
Not to mention that in a big org, the people at the bottom do not get rewarded for making excellence in engineering, if the top doesn't appreciate it. SO why go that extra mile, when a mediocre job will earn you the same salary? You'd be more incentivized to move jobs to grow your pay instead, and use this as a stepping stone.
If it was new stuff, then the people doing it must not be very low at the bottom (by virtue of the org being quite shallow - such as a startup).
Even new stuff in a big company with lots of layers of management cannot make new stuff with excellence. Eventually, probably sooner rather than later, it gets polluted.
Not to mention that in a big org, the people at the bottom do not get rewarded for making excellence in engineering, if the top doesn't appreciate it. SO why go that extra mile, when a mediocre job will earn you the same salary? You'd be more incentivized to move jobs to grow your pay instead, and use this as a stepping stone.
As somebody who has only ever worked at places small enough for individuals to all roughly understand eachother's impact, could I even get a job at a big org like that? I don't even understand the mindset
Get one? Sure. Be happy with it? Propably not.
I learned that the hard way, only the other way around.
I learned that the hard way, only the other way around.
Sometimes, I daydream about being a nameless drone in a cubefarm, taking home a nice paycheck and leaving my work at work. But, yeah, the dream passes quickly because I know I'd hate it with a burning passion.
it depends.
If you want to seek meaning in your work, then yes, cubicle farms will not give you fulfilment. But if you're just after a steady paycheque, and don't give a damn about the business, your work nor its purpose, then this is fine. And it gives you free time to live outside of work too.
If you want to seek meaning in your work, then yes, cubicle farms will not give you fulfilment. But if you're just after a steady paycheque, and don't give a damn about the business, your work nor its purpose, then this is fine. And it gives you free time to live outside of work too.
[deleted]
it is fine the first 6 months then bore out hits and you will be miserable.
Indeed, and greenfield projects are a great way to move upward in the org - as Jim in the story recognises.
Conway's Law isn't a "law" - it is an interesting observation and there are definitely interactions (both ways) between organizational structure and software "design". But people weigh it too heavily. There are plenty of counterexamples of, for instances, fine-grained teams in one company that build a monolith, and the same type of teams in another company building a microservices architecture. Complex software is complex, and once your require more than a handful of people, that complexity requires a complex organization structure to handle the challenges of communication and responsibility. But it isn't a purely deterministic relationship between organizational structure and software structure. More that certain organizational structures are really bad at building complex software, and some are better.
It is interesting to think about, but there is this tendency to invoke "Conway's Law" as some simple method of fixing the extraordinary complex problem of developing and evolving complex software over time.
Really what it is is that organizations from the 70s, 80s, and 90s were built around building physical things, for the most part. Manufacturing and construction, mostly. And taking those organizations and applying them to software development fundamentally was a terrible match. We had to find new structures that were better suited, and we have gotten a lot better, but we are still working on it. Management was taught by those that managed manual laborers, who's output was easy to measure, and whom you could force to work a certain amount without a significant drop in their output. The same isn't true for knowledge workers. And so new management had to be created, a process we still are working on as well, but many managers these days are way better than managers from 30 years ago.
Read Conway's original paper, it is interesting, but it isn't a physical Law that cannot be violated!
It is interesting to think about, but there is this tendency to invoke "Conway's Law" as some simple method of fixing the extraordinary complex problem of developing and evolving complex software over time.
Really what it is is that organizations from the 70s, 80s, and 90s were built around building physical things, for the most part. Manufacturing and construction, mostly. And taking those organizations and applying them to software development fundamentally was a terrible match. We had to find new structures that were better suited, and we have gotten a lot better, but we are still working on it. Management was taught by those that managed manual laborers, who's output was easy to measure, and whom you could force to work a certain amount without a significant drop in their output. The same isn't true for knowledge workers. And so new management had to be created, a process we still are working on as well, but many managers these days are way better than managers from 30 years ago.
Read Conway's original paper, it is interesting, but it isn't a physical Law that cannot be violated!
> But people weigh it too heavily. There are plenty of counterexamples of, for instances, fine-grained teams in one company that build a monolith, and the same type of teams in another company building a microservices architecture.
These aren't counterexamples because Conway's law makes no comment on how the teams will choose to deploy what they build. It talks about the overall design of the system, the solution space that was explored. Windows is a monolithic application if the alternative is microservices, but anyone who gives it more than a cursory glance can see the org chart (both past and present!) reflected in the disjointed UX.
> And so new management had to be created, a process we still are working on as well, but many managers these days are way better than managers from 30 years ago.
I'm not sure why this makes Conway's law not applicable anymore—what you're describing is that new management does a better job of creating communication structures that lend themselves well to creating software. That may well be correct, but the resulting improvements validate Conway's law! If we're getting better at software, it may well be because people are talking about and accounting for the impact that communication structures have on the output.
These aren't counterexamples because Conway's law makes no comment on how the teams will choose to deploy what they build. It talks about the overall design of the system, the solution space that was explored. Windows is a monolithic application if the alternative is microservices, but anyone who gives it more than a cursory glance can see the org chart (both past and present!) reflected in the disjointed UX.
> And so new management had to be created, a process we still are working on as well, but many managers these days are way better than managers from 30 years ago.
I'm not sure why this makes Conway's law not applicable anymore—what you're describing is that new management does a better job of creating communication structures that lend themselves well to creating software. That may well be correct, but the resulting improvements validate Conway's law! If we're getting better at software, it may well be because people are talking about and accounting for the impact that communication structures have on the output.
It’s not a physical law - it relates to people - “law” has a wider meaning outside the field of physics.
I recommend watching the talk! He specifically addresses the very points you bring up. Of course reality is messy and people especially so, but Conway’s law is one of very few theories within CS that has held up in replication studies.
I recommend watching the talk! He specifically addresses the very points you bring up. Of course reality is messy and people especially so, but Conway’s law is one of very few theories within CS that has held up in replication studies.
> There are plenty of counterexamples of, for instances, fine-grained teams in one company that build a monolith, and the same type of teams in another company building a microservices architecture.
...like ?
...like ?
Oracle certainly has. Several GBUs realized they were working towards the same goal of a microservices base layer for future service hosting (for handling auth, logging, databases, etc.) and combined their efforts, and I happened to work on one of the resulting teams.
I'm not sure how public they've been about all this, but you certainly can teach an old dog new tricks, provided there is management buy-in (which we certainly had - at one point, our teams were moved between GBUs to save us from budget cuts).
I'm not sure how public they've been about all this, but you certainly can teach an old dog new tricks, provided there is management buy-in (which we certainly had - at one point, our teams were moved between GBUs to save us from budget cuts).
I'm adding "Conway’s Soup" to my vocab as a name for the result of such process.
That's the thing that stumped me here: how does a _Senior Staff Engineer_ doesn't understand their job isn't to build software, but to enable the rest of the organization to build software? (Paraphrasing on that old Toyota principle)
[deleted]
>> This is Conway’s law in action
Not just one layer, look at the stack: Type script is MS's Conways law. API/microsevice everything is more of a google thing (vs FB monolith)... So you have two other major product of Conways Law that your trying to jam into your own org.
On top of that a 5 year plan is never going to digestible by (almost) any company. How many people stay in one place for that long? How much has the front end changed in 5 years? Is any choice you make in the JS world relevant in 5 years? Though sell there too!
Not just one layer, look at the stack: Type script is MS's Conways law. API/microsevice everything is more of a google thing (vs FB monolith)... So you have two other major product of Conways Law that your trying to jam into your own org.
On top of that a 5 year plan is never going to digestible by (almost) any company. How many people stay in one place for that long? How much has the front end changed in 5 years? Is any choice you make in the JS world relevant in 5 years? Though sell there too!
I spent 12 years at LinkedIn. Sadly, it's not even close to the engineering org it used to be. The era where Kevin Scott led engineering was a really good one in comparison.
Every engineer of a company of this size says the same exact thing.
IMO more about growth of the engineering teams that lead to any specific culture deteriorating.
IMO more about growth of the engineering teams that lead to any specific culture deteriorating.
Ryan Rolansky said LinkedIn is essentially feature complete.
Isn't it?
They seem to be trying to jam AI into a bunch of things now.
As an aside, I work on a product that should be considered feature complete, but PMs keep dreaming up features so they can get promoted. I wish we could just take some time to clean up the horrendous tech debt accrued over the past 10 years of feature proliferation.
As an aside, I work on a product that should be considered feature complete, but PMs keep dreaming up features so they can get promoted. I wish we could just take some time to clean up the horrendous tech debt accrued over the past 10 years of feature proliferation.
If you consider lots of features that don't work "complete" then I guess
MM lines of JavaScript! That is bloat incarnate.
I've been thinking of re-implementing something like LI, or rather implement my own contact database (without the "FB" features, please).
The only problem would be how to make my contacts migrate over in bulk.
Apart from the bloat, the main problem of Microsoft LinkedIn is that it does not let you export your contacts' infos, which really is a must-have feature of a contact platform.
I've been thinking of re-implementing something like LI, or rather implement my own contact database (without the "FB" features, please).
The only problem would be how to make my contacts migrate over in bulk.
Apart from the bloat, the main problem of Microsoft LinkedIn is that it does not let you export your contacts' infos, which really is a must-have feature of a contact platform.
> Apart from the bloat, the main problem of Microsoft LinkedIn is that it does not let you export your contacts' infos, which really is a must-have feature of a contact platform.
Platform lock in is certainly intended (even though it sucks for users)
Platform lock in is certainly intended (even though it sucks for users)
LinkedIn was originally built in Ruby, containing 60k lines of code
https://queue.acm.org/detail.cfm?id=2567673
Summary: https://www.pixelstech.net/article/1395463142-Why-does-Linke...
LinkedIn migrated to Node early 2010.
https://queue.acm.org/detail.cfm?id=2567673
Summary: https://www.pixelstech.net/article/1395463142-Why-does-Linke...
LinkedIn migrated to Node early 2010.
This is not entirely true. These articles are referring to the mobile services. The majority of LIs services are built in Java (with some Node, Python, Scala sprinkled in).
Here's a write up from the LI engineering blog briefly detailing a history of their architecture (up to 2015): https://engineering.linkedin.com/architecture/brief-history-...
Here's a write up from the LI engineering blog briefly detailing a history of their architecture (up to 2015): https://engineering.linkedin.com/architecture/brief-history-...
So they claim the change to node reduces the 60k lines of Ruby to only 2k of node. So how did they manage to add 1,998,000 more lines of node in the next 10 years and not think something was amiss?
Developers love to armchair and then you put a bunch of them together and you inevitably end up with soaking wet bloated festering codebases like LinkedIn
> So how did they manage to add 1,998,000 more lines of node in the next 10 years (...)
Does LinkedIn have the same features it had a decade ago?
Search is far more powerful. It also shifted to add an entirely new social media dimension to it.
God knows what features were made available to recruiters and marketers.
The hard part of a web app is not the stuff you see; it's the stuff you don't even know is there.
Does LinkedIn have the same features it had a decade ago?
Search is far more powerful. It also shifted to add an entirely new social media dimension to it.
God knows what features were made available to recruiters and marketers.
The hard part of a web app is not the stuff you see; it's the stuff you don't even know is there.
Call me crazy but I did not think that was a surprise. I worked at big bank that had consumer-facing JS web app with 6MM lines of code, though now I am starting to think that that number is false, given the feedback in this thread.
Have you tried grabbing JSON directly from the web server responses to the browser to get your contacts?
I haven’t tried this on LI but it’s my dirty trick for exporting conference attendee lists when they’re made public on websites. YMMV.
I haven’t tried this on LI but it’s my dirty trick for exporting conference attendee lists when they’re made public on websites. YMMV.
I was really impressed with how open Chris Krycho was about his struggles without playing the blame game. CoRecursive is one of my favorite podcasts because it explores the complex context behind the code.
Agreed. And I also think Adam is a great host. He asks good questions and lets the guest talk.
He seems like the kind of guy I'd like to work with, personally.
This sounds like one of those "soft leadership" roles, that almost always suck because you're "responsible" for something, but have little or no authority over the rest of the org. The real tech leadership (if any) are MIA or no longer in touch with the actual problems, even if they've been their forever or are the "system experts"
Been there, done that, no thanks.
Been there, done that, no thanks.
> My previous company had what I thought at the time was a pretty decent sized app, like 150,000 lines of code. And the LinkedIn front end, when I got there had 2 million, it’s just like, I’m, did you say million? Uh, that’s, that’s a lot.
I'd love to see what makes it so large. That's like 1/15th the size of Linux.
I'd love to see what makes it so large. That's like 1/15th the size of Linux.
Have you ever worked on projects (particularly java suffers from this) where there is generational bloat because most people have very narrow understanding of the codebase so they can only add things and not remove/refactor without breaking it? That’s how
That's not likely to be the issue here tbh, or, that's focusing on low-hanging fruit. You get large codebases from having many features. No amount of coding and refactoring will bring a codebase's size down in a significant and measurable way unless features are ruthlessly removed.
These high-profile apps are elaborate spying devices. Besides the UI code, they probably have thousands of classes, some of which abandoned as dead code, with names like ProfileViewActivityLocationTracker...
I mean sure, I could understand Figma's frontend or something having millions of lines but what does linkedin frontend do that would reasonably require this?
My guess is that it's because there are 800 different versions of everything because none of the company's 13,000 UI/UX designers are willing to use the same component libraries.
Blaming 2,000,000 lines of code on UX designers, and not the developers, is certainly a new one.
Usually if you peek at the first few commits it all seemed nice and clean. The turning point is usually as soon as the original author/'s leave and new people come in.
Orgs never value engineers that improve the overall codebase so I can't blame them. You're only getting promoted and compensated by delivering the next big feature which is part of the owners vision and being visible.
[deleted]
> particularly java suffers from this
Correlation, not causation. Java is often used on large projects, it doesn't itself make things brittle.
It's like the bomber survivorship fallacy - the fact you see such messed up projects mostly in Java is that they would not survive in languages like Python, where even a simple method rename can easily lead to big regressions.
Correlation, not causation. Java is often used on large projects, it doesn't itself make things brittle.
It's like the bomber survivorship fallacy - the fact you see such messed up projects mostly in Java is that they would not survive in languages like Python, where even a simple method rename can easily lead to big regressions.
The site makes me sad whenever I visit.
On my beefy desktop, load times for the home page are at around three seconds. On my feeble (but still enormously expensive: a Microsoft Surface) laptop, it’s well in excess of 10 seconds. Both tests run on the very same network, so my conclusion is that load times are CPU bound, aka there’s huge amounts of JavaScript running, for one reason or another.
On my beefy desktop, load times for the home page are at around three seconds. On my feeble (but still enormously expensive: a Microsoft Surface) laptop, it’s well in excess of 10 seconds. Both tests run on the very same network, so my conclusion is that load times are CPU bound, aka there’s huge amounts of JavaScript running, for one reason or another.
Including boilerplate code? ASP and JS is far more verbose in those areas.
That quote is why you can't take anything seriously on the internet about running software or software engineering seriously.
There are to many people that just don't know what they don't know and act they have things solved.
I think the software world has like two bubbles of people around the scale they work with and they don't understand each other or run into each other, except in interviews and on the internet.
There are to many people that just don't know what they don't know and act they have things solved.
I think the software world has like two bubbles of people around the scale they work with and they don't understand each other or run into each other, except in interviews and on the internet.
It's funny you say that because as someone in the large scale bubble when I read 2 million lines that didn't seem that large. Other companies of similar size are dealing with 2-10x larger web apps.
Ex. https://stripe.com/blog/migrating-to-typescript
Which talks about 3.7m lines at Stripe.
Ex. https://stripe.com/blog/migrating-to-typescript
Which talks about 3.7m lines at Stripe.
Yeah, and I have it on good authority that there are JS/TS apps more than 10 times larger than that at Google. Scale is relative.
[deleted]
Think js+html+css, that's a lot
I am an engineer at LinkedIn and have been here for a long time. Haven't worked with CK and work on areas unrelated to Finger Gun project. But I agree with the general sentiment of feeling frustrated. Everything is now happening top-down. Execs want AI thrown everywhere even where things don't make sense. Engineers don't have the room to give feedback that they used to. Leaders want everything delivered asap. We are creating lot of tech debt every day and not maintaining any balance between speed and quality. Features are being implemented with missing edge cases and they say we will fix it later but no one has time or motivation to fix it. The execs only know we launched a feature, nobody tells them that the feature is too broken to actually be usable. There's a lot of fear of being managed out for performance so people are sucking it up and implementing what is asked with urgency. No one has time or motivation to fix bugs or clean up the code.
It's no surprise that LinkedIn's codebase is a mess if you try to use the website, it's a complete nightmare. Say I see an interesting post, I click on the person who wrote it to learn more about them, then I press back to go back to the post and continue reading it - now the feed refreshed and I've lost the post.
Say I want to scroll back through messages I've received, I do that, the entire webpage starts lagging and takes 10-15 seconds to register inputs.
Why do I get 30 fake notifications? They are literally not real, made up rubbish to force interactions from people - it's disingenuous. The recommendation algorithm is also completely terrible.
Say I want to scroll back through messages I've received, I do that, the entire webpage starts lagging and takes 10-15 seconds to register inputs.
Why do I get 30 fake notifications? They are literally not real, made up rubbish to force interactions from people - it's disingenuous. The recommendation algorithm is also completely terrible.
considering they are owned my Microsoft, their windows app app is a technical marvel. It’s a crippled buggy version of their already too buggy web app. I understand why it was made, but I don’t understand why it was released. It doesn’t look good for either brand.
The thing about the notifications though, is you can turn off every single one of them in the settings.
When I tried to turn it off though, I was hit with 100s of different types of notifications. I generally like it when apps/sites do this. This way I can turn off the garbage i dont need, and keep the ones I want. But 90% of thosr categories were garbage. It is really shocking that one can take a good idea this far and make it frustating.
When I tried to turn it off though, I was hit with 100s of different types of notifications. I generally like it when apps/sites do this. This way I can turn off the garbage i dont need, and keep the ones I want. But 90% of thosr categories were garbage. It is really shocking that one can take a good idea this far and make it frustating.
The notifications has long been totally useless.
Even jobs tab in each company page doesn’t work in ios for me. What’s the point of using it then?
> That's 20 times almost the size of my, how, how do you even build that? How long to build? Oh, 17 minutes for a new build. That's, that's not bad. We should probably make that better, but that’s not bad.
Am I spoiled for thinking 17 minutes is too much, even if it's 2 million LOC? I guess LinkedIn can afford it
Am I spoiled for thinking 17 minutes is too much, even if it's 2 million LOC? I guess LinkedIn can afford it
It depends on what those LOC are. Is it HTML? Is it JS? It it a compiled language? How much of it is generated?
Aside: I don't understand the obsession people have with throwing out LOC. It's a completely meaningless number unless you're familiar with the codebase, in which case you don't need to be told the stats.
Aside: I don't understand the obsession people have with throwing out LOC. It's a completely meaningless number unless you're familiar with the codebase, in which case you don't need to be told the stats.
> Aside: I don't understand the obsession people have with throwing out LOC.
Agreed. A better metric is how much of it you need to interact with on a daily basis in order to perform any meaningful work.
If the code is sane, you'd only ever need to interact with and understand a fraction (a subset of the public APIs) of the entire codebase. You could have millions of lines of code and be productive when knowing only a couple of hundred of those.
If the code is insane, the risk is that you'd need to interact with most of it when introducing any non-trivial functionality. Worst case scenario there isn't any clear rules of what is public and supposed to be used, and what is private and isn't supposed to be used, and people have used "whatever from where ever" to solve their problems.
The second one is always bad, but also significantly worse the more lines of code there are. Perhaps those who throw LOCs around also implicitly tell us that they are dealing with the second variety of code.
Agreed. A better metric is how much of it you need to interact with on a daily basis in order to perform any meaningful work.
If the code is sane, you'd only ever need to interact with and understand a fraction (a subset of the public APIs) of the entire codebase. You could have millions of lines of code and be productive when knowing only a couple of hundred of those.
If the code is insane, the risk is that you'd need to interact with most of it when introducing any non-trivial functionality. Worst case scenario there isn't any clear rules of what is public and supposed to be used, and what is private and isn't supposed to be used, and people have used "whatever from where ever" to solve their problems.
The second one is always bad, but also significantly worse the more lines of code there are. Perhaps those who throw LOCs around also implicitly tell us that they are dealing with the second variety of code.
> I don't understand the obsession people have with throwing out LOC
LOC is the metric I get to remain employed or fired on.
LOC is the metric I get to remain employed or fired on.
It is not too much or too little. The real question is whether the value of X% speedup worth Y hours of developer time to make the improvement. Relative to other work that could be done.
Answer could well be yes since build time impacts everyone, but also have to factor in complexity of the CI system they use.
Answer could well be yes since build time impacts everyone, but also have to factor in complexity of the CI system they use.
It’s not as easy as it seems to answer that question. You’ll have rebound effects meaning developers will probably recompile more often.
No, that's not the real question. Decisions have more factors than just what cash will they bring in.
Long build times are an operational risk - it's very hard to apply patches to solve incidents with a 20 minute build time.
Long build times are an operational risk - it's very hard to apply patches to solve incidents with a 20 minute build time.
So reducing this would be a multi-year project of a sizeable team (let's say 5-10 people). How do you get someone to sign off for this kind of expediture and not lay everyone off when the next economic fart comes?
Because THAT'S the hard problem to solve in any bigger organization. Not hacking the code.
Because THAT'S the hard problem to solve in any bigger organization. Not hacking the code.
>So reducing this would be a multi-year project of a sizeable team
Maybe, or just a knowledgable person to dig in for a bit and solve it. On one of my teams when I was back at Microsoft was working on a similarly huge codebase with build times twice as long. We had a senior engineer who was really into build optimizations from a previous job who wrote a tech design, implemented better project splitting in our monorepo, better local caching, and finally cloud caching, all within 3 months. When I left, our builds ranged from 30 seconds to 3 minutes.
It's not guaranteed that it'll be this simple, but I also don't think it's guaranteed it'll be 3 years and several people.
Maybe, or just a knowledgable person to dig in for a bit and solve it. On one of my teams when I was back at Microsoft was working on a similarly huge codebase with build times twice as long. We had a senior engineer who was really into build optimizations from a previous job who wrote a tech design, implemented better project splitting in our monorepo, better local caching, and finally cloud caching, all within 3 months. When I left, our builds ranged from 30 seconds to 3 minutes.
It's not guaranteed that it'll be this simple, but I also don't think it's guaranteed it'll be 3 years and several people.
To clarify: when I say value I don't just mean monetary value - though of course everything in a business ultimately does boil down to that, including the total expectation of risks.
I agree though, definitely a factor
I agree though, definitely a factor
The more important number for a codebase that size is almost always "what's the incremental build time for whatever I work on". It's nice to have a "build from scratch" be fast, but almost always you can leave it for two hours to do the first build while you're in a meeting or something. More important is that when you make a change, the incremental build is fast.
This is why build systems like buck/pants/bazel are good in larger codebases. You can trust your incremental builds in a way that you can't in any build system that isn't hermetic.
This is why build systems like buck/pants/bazel are good in larger codebases. You can trust your incremental builds in a way that you can't in any build system that isn't hermetic.
And we have cloud caching for the "build from scratch" case so even that can be pretty quick. This all just takes a little time investment.
Yeah, they could definitely optimize it quite a lot further. With investment in code structure and caching I bet they could get below 5min at that scale. But also for edit/refresh that should be on the order of a second or two, never minutes.
No shade on ember as a framework but LinkedIn’s decision to use Ember always baffled me. As far as I remember they switched to it long after it was clear react had “won”.
LinkedIn actually chose Ember well before React had won, and definitely before it had become the new “no one gets fired for buying IBM.” That all preceded my time there, but the evaluation was late 2014 and the LinkedIn.com web app rebuild in Ember was kicked off in 2015 if I recall correctly. React only came out in 2013, and while it was definitely on a rocket ship it was way less obvious in late 2014 that it was going to be the de facto standard that it was by late 2016–early 2017.
Bonus note: One factor that was really not obvious at the point they were doing the evaluation was how much TypeScript and TypeScript-powered tooling was going to matter. TS shipped support for JSX/TSX in TS 1.6, in late 2015, and I contend that while JSX had a lot of pros and cons, the one absolutely unambiguous win it had over everything else from that point forward was that whether you were writing JS or TS, you got outstanding editor support in basically every editor in the world courtesy of the TS language server backing your JS or TS development. Just absolutely massive.
The fact that the TS team baked support for JSX into TS itself and that it's not pluggable is (a) totally understandable and reasonable from a maintenance effort POV, especially given that Microsoft uses React extensively, and (b) extremely frustrating for everyone not using JSX. Vue, Angular, Svelte, and Ember all had to build the exact same infrastructure to get that support, and that was a huge productivity hit for devs using those frameworks compared to React in the meantime.
Bonus note: One factor that was really not obvious at the point they were doing the evaluation was how much TypeScript and TypeScript-powered tooling was going to matter. TS shipped support for JSX/TSX in TS 1.6, in late 2015, and I contend that while JSX had a lot of pros and cons, the one absolutely unambiguous win it had over everything else from that point forward was that whether you were writing JS or TS, you got outstanding editor support in basically every editor in the world courtesy of the TS language server backing your JS or TS development. Just absolutely massive.
The fact that the TS team baked support for JSX into TS itself and that it's not pluggable is (a) totally understandable and reasonable from a maintenance effort POV, especially given that Microsoft uses React extensively, and (b) extremely frustrating for everyone not using JSX. Vue, Angular, Svelte, and Ember all had to build the exact same infrastructure to get that support, and that was a huge productivity hit for devs using those frameworks compared to React in the meantime.
Some internal stakeholder surely fought for it so they could get promoted.
Happy for that person, wherever they are now.
Happy for that person, wherever they are now.
Since when is LinkedIn is fast paced? It hasn't substantially changed in ten years.
The fast pace comes from product deciding there should be 5 different types of “like” button and goofy AI “takeaways” functionality.
LinkedIn is a solved problem but you can’t have that any more.
LinkedIn is a solved problem but you can’t have that any more.
Probably a lot of work that is being done there is inventing useless stuff that makes people, projects and departments look needed.
What happened to Twitter shows that in most older orgs with a well established product you can throw out ~50% of the people and it still will go on.
What happened to Twitter shows that in most older orgs with a well established product you can throw out ~50% of the people and it still will go on.
LinkedIn learning is pretty new, massively pushed for too. And most of the new features are on the recruitment side I think
Although I do listen to Adam Gordon Bell's podcasts from time to time, I'm not sure if I should invest 40something minutes into hearing about why some rando left BigCorp. This may be an unpopular opinion, but isn't unprofessional to spill the beans about your past employment like this? As an external observer and hiring manager, I wouldn't touch this person with a 10-foot pole if they're in the hiring pool.
I personally find this type of content to be mundane, basic and useless. This is the type of stuff most people run into if their engineering career spans more than a couple of years, touches large organizations, they can't handle conflicts and have the luxury of being more affluent than other employees. I don't understand why someone would go out of their way to share something like this.
I personally find this type of content to be mundane, basic and useless. This is the type of stuff most people run into if their engineering career spans more than a couple of years, touches large organizations, they can't handle conflicts and have the luxury of being more affluent than other employees. I don't understand why someone would go out of their way to share something like this.
This is, indeed the type of stuff most people run into if their career spans more than a couple years, touches large organizations, etc. That is why I thought it would be useful to share: it is the kind of thing that would’ve been really helpful for me in the first few years of my career to understand some of the dynamics, including what can go well, and what can go less well, and the kind of mistakes one might try to avoid, as well as what it means to stick with one’s convictions, even when that might be costly.
Whether it is “unprofessional“ or not is something I thought about quite a bit. I think I walked the line of being fairly politic and avoiding dragging real people’s names through the mud, while still getting into some of what that kind of real world engineering and associated politics is like. At the same time, I can see how someone would take the view that none of this kind of thing should be shared outside the company. Your mileage may vary. I hope at a minimum you can understand why someone might choose to air a little bit of it while trying not to be nasty about it.
Whether it is “unprofessional“ or not is something I thought about quite a bit. I think I walked the line of being fairly politic and avoiding dragging real people’s names through the mud, while still getting into some of what that kind of real world engineering and associated politics is like. At the same time, I can see how someone would take the view that none of this kind of thing should be shared outside the company. Your mileage may vary. I hope at a minimum you can understand why someone might choose to air a little bit of it while trying not to be nasty about it.
Thank you for the clarification, it seems you put a lot of thought into this. Let's agree to disagree here. You may find this valuable information that's worth sharing, I do not - I think the risks outweigh the benefits.
In my career I have learned to listen more carefully to engineers. Sometimes it is not about saying yes and no to them but listening to what they are trying to say. Sometimes if you ask a few questions they learned something about the business that makes their case come apart. Sometimes if you listen to them you find a good idea and it opens up a whole new area of the product. So I think it’s a false dichotomy and you have to be good at listening and asking the right questions to get your point across.
Really feels like mission impossible to do this kind of role remotely.
It's very possible, and I've seen a lot of people do it successfully.
It does require some organizational changes, and a slightly different set of skills, and some fail to realize they lack both.
It does require some organizational changes, and a slightly different set of skills, and some fail to realize they lack both.
The migration from Ember to React sounds like a good case for a technology a previous client of mine created.
He created a language called Unhack which allowed you to do a kind of search-and-replace but at the level of the AST. You would use this language to specify a pattern in the original code, then another pattern that would be the replacement. Unfortunately, I don't know if it still exists, as I stopped working with that client a couple of years ago.
He created a language called Unhack which allowed you to do a kind of search-and-replace but at the level of the AST. You would use this language to specify a pattern in the original code, then another pattern that would be the replacement. Unfortunately, I don't know if it still exists, as I stopped working with that client a couple of years ago.
Can't stay in business swapping dollars. Usually, the ones declaring their idealism are going to find a problem with you regardless. Doing the right thing is what you do, but that's not the sort of attitude that works.
I found the "finger guns" team and their plan fascinating, and I've definitely seen enthusiasm sweep the field many times.
I work on an infrastructure / foundational engineering team, and it reminds of something I'm increasingly believing to be true:
It's much harder to get headcount/funding for a two-year plan to build/fix an internal tool than to get headcount/funding for a six-month plan to use an external product that ends up taking 3-4 years.
In this situation, I'd say it's much harder to get headcount/funding for a five-year plan to perform a well-engineered migration than it is to get approval for a two-year whizz-bang plan to create an amazing replacement that will actually take 3-6 years and/or actually leave 80% of the legacy code migration unfinished basically forever, leading to pure xkcd 927.
I work on an infrastructure / foundational engineering team, and it reminds of something I'm increasingly believing to be true:
It's much harder to get headcount/funding for a two-year plan to build/fix an internal tool than to get headcount/funding for a six-month plan to use an external product that ends up taking 3-4 years.
In this situation, I'd say it's much harder to get headcount/funding for a five-year plan to perform a well-engineered migration than it is to get approval for a two-year whizz-bang plan to create an amazing replacement that will actually take 3-6 years and/or actually leave 80% of the legacy code migration unfinished basically forever, leading to pure xkcd 927.
As an addendum, I found the episode fascinating. As someone who has spent about five of the last eight years on a foundational engineering team, involved in _many_ long, grinding migrations where the end goal is clearly a win for the company but not particularly useful for product teams in the short term, I thought the plans Chris outlined showed remarkable maturity and good planning in approach. I also enjoyed hearing a public discussion of the nature of that kind of migration.
I would not, myself, choose to talk quite so negatively about a former employer and colleagues, but I respect and enjoyed the decision of idealistic folks like Chris to be open about the struggles entailed!
I would not, myself, choose to talk quite so negatively about a former employer and colleagues, but I respect and enjoyed the decision of idealistic folks like Chris to be open about the struggles entailed!
I have no idea why anyone is still a user on LinkedIn. They've had numerous breaches, and give 2 entire shits about it. If your account gets stolen, even with MFA enabled, they will take roughly 2+ weeks to respond to your ticket and they require that you send in even more information that they will callously handle.
My advice, leave LinkedIn.
Edit: Or, continue to enjoy taking it in the shorts, for an entity that could not care less.. lol
My advice, leave LinkedIn.
Edit: Or, continue to enjoy taking it in the shorts, for an entity that could not care less.. lol
Boeing in action. Luckily nobody is flying LinkedIn MAX, so there are no deaths as a result.
LinkedIn comes across as a not-so-great company to work for. I find their product to be very toxic, off-putting and full of life coaches and scammers.
But I bet he still has his resume on LinkedIn. :)
I clicked this bc I thought it was about users leaving Linkedin and was excited lol. But I listened and he pitched a 5 year plan solution to fix bad code??! Come on, this guy may never get hired again after that. Listening to this, I felt outrage at the concept of white men feeling like they can just quit when a company doesn't follow their slow engineering model. Huh?? I feel annoyed with stupid software and product decisions all the time and that's just how it is everywhere. All of capitalism and all of tech is do-it-as-fast-and-cheap-as-possible. Hello? Oh right, I looked at his Linkedin and he never had any formal training in software or engineering and has a MS in Divinty, so goodluck bruh, back to the bible for you.
issafram(3)
bugbuddy(1)
Chris: There we go. Okay, those are running now. Crap. Did I screw up my ability to hear you?
Adam: Hello, hello, hello?
Chris: I did. Dang it. Hang on.
Adam: Hello, hello?
Chris: Yeah, yeah, we’re good now. Okay. Let’s try that again.
Truly fascinating dialog
Adam: Hello, hello, hello?
Chris: I did. Dang it. Hang on.
Adam: Hello, hello?
Chris: Yeah, yeah, we’re good now. Okay. Let’s try that again.
Truly fascinating dialog
It is a fairly common podcast trope to leave some bloopers at the start to humanise the people there.
And fill the script with that? Wow
It's a thing. Like how GenZs start their tiktok videos showing fake urgency.
It’s awful and unprofessional for a podcast. This is why editing exists.
LinkedIn is so awful to use.
There's value in the connections you can make and hiring or finding a job... But it's probably one of the worst web apps I use.
I'm surprised it ever got this big.
There's value in the connections you can make and hiring or finding a job... But it's probably one of the worst web apps I use.
I'm surprised it ever got this big.
Poor kid, learning how an enterprise works the hard way. If he stays in software eng long enough he'll be back here eventually, with a little less gumption, but a little more tactics and grit to fight the battles he wants to fight.
Are we belitteling people now because they have intrinsic standards and pride in the quality of their work instead of blindly chasing whatever is best for their replaceable career in a hypercapitalistic environment?
Aside from the GP comment calling the guy "kid", it doesn't seem especially condescending to me.
I read it more like "Yeah, we've all been there. It's a sucky lesson to learn, but in the end he's likely to be wiser (albeit more jaded) to these things."
I read it more like "Yeah, we've all been there. It's a sucky lesson to learn, but in the end he's likely to be wiser (albeit more jaded) to these things."
Hopefully they'll learn that such standards don't exist and that if they do will deteriorate by the second year.
If it wasn't for it being some big enterprise corporate company no one would give a stuff about the blog post.
Do I care someone left company X, fired from company Y? No.
Do I acknowledge it's upsetting? Yes, Pack your stuff and try again.
welcome to life.
If it wasn't for it being some big enterprise corporate company no one would give a stuff about the blog post.
Do I care someone left company X, fired from company Y? No.
Do I acknowledge it's upsetting? Yes, Pack your stuff and try again.
welcome to life.
Not really bro, but you stay angry
The introduction sounds very GPTy:
> His journey was marked by challenges: from the nuances of remote work to the struggle of influencing company culture, and a critical incident that put his principles to the test against the company’s push for speed.
>Chris’s story highlights the tension between the need for innovation and the importance of project health. This all led Chris to a pivotal decision: to stay and compromise his beliefs or to leave in pursuit of work that aligned with his principles.
>He chose the latter. Join us as we dive into Chris’s compelling story, exploring the challenges of advocating for principled engineering in a world that often prioritizes quick wins over long-term value.
> His journey was marked by challenges: from the nuances of remote work to the struggle of influencing company culture, and a critical incident that put his principles to the test against the company’s push for speed.
>Chris’s story highlights the tension between the need for innovation and the importance of project health. This all led Chris to a pivotal decision: to stay and compromise his beliefs or to leave in pursuit of work that aligned with his principles.
>He chose the latter. Join us as we dive into Chris’s compelling story, exploring the challenges of advocating for principled engineering in a world that often prioritizes quick wins over long-term value.
FWIW, it seems pretty consistent with the style of Adam's other intros.
To me that's the most interesting quote of the podcast and that's the impression I had gotten before reading this quote. Sounds like they received valuable feedback along the way and ignored it on purpose.
My career lesson is that being right isn't the hard part of being a senior staff engineer. Being right isn't enough. It's building alignment across the entire org towards the right solution. That's the hard valuable problem that I'm paid to solve.
I worked on the facebook.com rewrite to React in 2019. I find this story particularly interesting.