2 Years at Twitter(eed3si9n.com)
eed3si9n.com
2 Years at Twitter
https://eed3si9n.com/2years-at-twitter/
148 comments
Also it's needed to remember that Twitter had 100 million users in 2012. A lot of "off the shelf" infra that we can use today just did not exist in 2012. They had to build for scale at that time, and a lot of those choices might not translate easily to something off the shelf now.
> A lot of "off the shelf" infra that we can use today just did not exist in 2012
Do you happen to have some examples? Very curious to hear about this.
Do you happen to have some examples? Very curious to hear about this.
First version of Docker was released in 2013 and was a curiosity for a few years.
Kubernetes was launched in 2014, and similarly, was mostly a curiosity for a few years. Twitter at that time made an (probably?) unlucky bet on Mesos, a competing but not that similar technology, that is now pretty much dead.
Google Cloud launched it's VM service in beta in 2012. A lot of base AWS services did exist, but was similarly a curiosity: AWS 2012 revenue was 2 billion. This year they are on track to 80 billion.
Kubernetes was launched in 2014, and similarly, was mostly a curiosity for a few years. Twitter at that time made an (probably?) unlucky bet on Mesos, a competing but not that similar technology, that is now pretty much dead.
Google Cloud launched it's VM service in beta in 2012. A lot of base AWS services did exist, but was similarly a curiosity: AWS 2012 revenue was 2 billion. This year they are on track to 80 billion.
I mean, even like J̶e̶n̶k̶i̶n̶s̶ and Kafka were barely a year old in 2012. Hadoop wouldn’t be replaced by Spark for 2 more years. Mesos was in its infancy.
Other companies had similar problems to solve, and they also had to build it themselves for the most part.
Other companies had similar problems to solve, and they also had to build it themselves for the most part.
Jenkins came out in 2005. 2011 was just when the project was renamed from Hudson due to Oracle trademarking the original name.
[deleted]
Bazel was released in 2015
kubernetes
> 2. The productivity gains of something like Bazel and Pants are pretty amazing if you are working within a monorepo.
In theory, maybe. But can anyone external honestly look at what Twitter has shipped for the last few years and justify that it was at all likely their productivity problems in engineering had anything to do with the build system.
I've worked as an FTE in a platform team for a major public tech company before, and I don't know many smart ones that came out of that thinking it was a good trade off to have teams whose very survival necessitated potentially unnecessary platform tooling / changes -- unless you're literally Google or FB and can see measurable results from optimizing that tooling.
Twitter never got close to that point.
In theory, maybe. But can anyone external honestly look at what Twitter has shipped for the last few years and justify that it was at all likely their productivity problems in engineering had anything to do with the build system.
I've worked as an FTE in a platform team for a major public tech company before, and I don't know many smart ones that came out of that thinking it was a good trade off to have teams whose very survival necessitated potentially unnecessary platform tooling / changes -- unless you're literally Google or FB and can see measurable results from optimizing that tooling.
Twitter never got close to that point.
> But can anyone external honestly look at what Twitter has shipped for the last few years and justify that it was at all likely their productivity problems in engineering had anything to do with the build system.
You say you've worked at a "major public tech company before", so why would you assume Twitter's paralysis was the fault of engineering and not management/product?
You say you've worked at a "major public tech company before", so why would you assume Twitter's paralysis was the fault of engineering and not management/product?
> If you have 2000 SWEs you'll pay 40k/month just to have access to the website.
That doesn't sound like a bad deal if you have to hire such a big build team; let's say you could fire 10 people, 10 * 150.000 (rough estimate) / 12 = 125k/month. That's a 85k/month saving.
I highly doubt GitLab fits the need of such massive monorepos; my only point is that the financial argument is not very strong IF GitLab were a viable alternative to such a big build team.
That doesn't sound like a bad deal if you have to hire such a big build team; let's say you could fire 10 people, 10 * 150.000 (rough estimate) / 12 = 125k/month. That's a 85k/month saving.
I highly doubt GitLab fits the need of such massive monorepos; my only point is that the financial argument is not very strong IF GitLab were a viable alternative to such a big build team.
2,000 SWEs @ $100K each, $200M/year. $40K doesn't sound so bad (my large enterprise employer licenses GitLab Premium and it's worth every cent we pay).
We already hire the people who negotiate the purchase- it's the same people who manage the infrastructure for our onsite instances, etc. IE, that cost is already priced in.
We run our own runners, on a nodegroup in k8s, which downsizes after the runners go idle for a while.
In enterprise, many attempts to save or squeeze a last penny out of something just ends up moving the costs elsewhere in the company, or reducing productivity, or driving away good employees.
We already hire the people who negotiate the purchase- it's the same people who manage the infrastructure for our onsite instances, etc. IE, that cost is already priced in.
We run our own runners, on a nodegroup in k8s, which downsizes after the runners go idle for a while.
In enterprise, many attempts to save or squeeze a last penny out of something just ends up moving the costs elsewhere in the company, or reducing productivity, or driving away good employees.
> 1. GitLab will charge you $20/month/user.
> 2. If you have 2000 SWEs you'll pay 40k/month just to have access to the website.
So the cost of just 1 more engineer... 2001 instead of the 20 extra that this build team creates.
> 2. If you have 2000 SWEs you'll pay 40k/month just to have access to the website.
So the cost of just 1 more engineer... 2001 instead of the 20 extra that this build team creates.
In 2012, GitLab was just a HN beta. https://news.ycombinator.com/item?id=4428278
In that year, Twitter had 100 million users and already functioning build system, that they opensourced soon later.
In that year, Twitter had 100 million users and already functioning build system, that they opensourced soon later.
It must be tempting to cut the Gordian knot by requiring everything to be written in Go, and built from source in a monorepo.
There'd be exceptions, but you'd save a lot on build cost/time and dependency and artifact management.
There'd be exceptions, but you'd save a lot on build cost/time and dependency and artifact management.
Well said. The outcome of the Bazel/Pants changes along with the avoidance of the "just buy it" from the outset will pay dividends in efficiency and savings forever (current-events-doomsaying aside).
Another company that copied the engineering practices of Google without the profitability to support a huge internal tools team.
Google had great engineering practices and infrastructure, the best of my career across several FAANGS. If only they didn't make launching products (and killing the old one) the way for advancement.
Some of Google's great build tools came about simply because they built absurdly large binaries (GWS and GFE). IE, they own-goaled themselves repeatedly and had to invent entire new technologies to get themselves out of it. THis was a consistent pattern; the much-vaunted GFS was actually a bear to run in production. Some things I really liked about working there include:
Monorepo for the entire front end and back end of Google services and products (excluding the Android OS code, and some other projects) made it really easy to get started doing interesting things without having to do a lot of work. LexicographicRangeSharder, which shards data streams using reservoir sampling, is a great example that I wouldn't have been able to build myself, but could trivially link into my code.
Blaze was an excellent build tool most of the time, and the team that build it/supported it was excellent.
The build/test platform (Forge and TAP) were both absolutely genius, and made large-scale testing easy. I was the owner of numpy and scipy and frequently had to do megabuilds to ensure that upgrades didn't break important things (IE, we forward-fixed all the dependencies that would break when the upgrade was committed).
A highly scalable version control system with serial commit numbers, making all sorts of projects involving bisection and problem hunting far easier than git.
A wide collection of very smart people who could answer nearly any technical question that came up
After returning to more conventional company, as well as a startup I really miss a lot of that. but blaze and TAP were partly necessary because of the design decisions made by early (pre-google3) engineers.
Monorepo for the entire front end and back end of Google services and products (excluding the Android OS code, and some other projects) made it really easy to get started doing interesting things without having to do a lot of work. LexicographicRangeSharder, which shards data streams using reservoir sampling, is a great example that I wouldn't have been able to build myself, but could trivially link into my code.
Blaze was an excellent build tool most of the time, and the team that build it/supported it was excellent.
The build/test platform (Forge and TAP) were both absolutely genius, and made large-scale testing easy. I was the owner of numpy and scipy and frequently had to do megabuilds to ensure that upgrades didn't break important things (IE, we forward-fixed all the dependencies that would break when the upgrade was committed).
A highly scalable version control system with serial commit numbers, making all sorts of projects involving bisection and problem hunting far easier than git.
A wide collection of very smart people who could answer nearly any technical question that came up
After returning to more conventional company, as well as a startup I really miss a lot of that. but blaze and TAP were partly necessary because of the design decisions made by early (pre-google3) engineers.
20m lines of code is not large enough to have 6 people in build tooling. A lot of companies at Twitter size end up over engineering and resisting buying vs. building in house big shot projects that often times can't compete with some 3rd party solution they could just buy.
For example my company is reinventing Next.js in house because the politics in the company won't allow us ditch this half-baked Next.js clone and just use the real thing. Our deployment pipeline will be much simpler if we simply bought Vercel. I'm sure we can negotiate prices with them that is better than hiring 12 engineers
For example my company is reinventing Next.js in house because the politics in the company won't allow us ditch this half-baked Next.js clone and just use the real thing. Our deployment pipeline will be much simpler if we simply bought Vercel. I'm sure we can negotiate prices with them that is better than hiring 12 engineers
I think the more relevant number is the number of ICs (around 2000, according to TFA). 12 build engineers (I don't know where you got 6 from) for 2000 ICs doesn't seem large at all to me; it's actually less than I would have expected.
Yeah, as long as the team makes the rest of the engineers 1% more efficient it's a win in terms of investment.
What is TFA?
"The Fucking Article[1]." Alternatively, "the fine article."
[1]: https://www.urbandictionary.com/define.php?term=TFA
[1]: https://www.urbandictionary.com/define.php?term=TFA
They were not building/re-inventing Bazel (its Open Source from Google); they were just adopting it and ensuring the migration for the rest of the company (which is no small feat, since it affects all other developers in the company).
The weirdest part here is that the author worked on a migration from their NIH build tool to Bazel for 2 years and never once mentions why all of that was even happening, and by the end of his employment, it was still not done.
He links the pants-devel announcement and here is what it says:
> Twitter has made the decision to migrate from Pants to Bazel as our primary build tool, as we believe that we have unique needs that Bazel can better address
Pants was your own fucking build tool that you invented, controlled and maintained! How on earth can it not meet your unique needs.
He links the pants-devel announcement and here is what it says:
> Twitter has made the decision to migrate from Pants to Bazel as our primary build tool, as we believe that we have unique needs that Bazel can better address
Pants was your own fucking build tool that you invented, controlled and maintained! How on earth can it not meet your unique needs.
I think you greatly underestimate how hard it is to find a good build engineer. Two of the smartest and hard working engineers did a bazel migration at one of my old companies and had to contribute heavily upstream to support all of our use cases.
It’s much cheaper in the long run to migrate to a well supported tool that you can extend than have to continuously staff some of the best engineers to keep building on your homegrown mess.
This comment is reminiscent of others i have seen about “building uber with 20 engineers” what could be so hard?
It’s much cheaper in the long run to migrate to a well supported tool that you can extend than have to continuously staff some of the best engineers to keep building on your homegrown mess.
This comment is reminiscent of others i have seen about “building uber with 20 engineers” what could be so hard?
Seconding this- Bazel is great, but not free in terms of engineering work to integrate.
>NIH build tool
They build it years before Bazel was opensourced.
I find it weird that everyone acts like Twitter is a new startup that created all of it's infra 1-2 years ago, while they had 100 millions of users in 2012 - a lot of this stuff was already years in the making then.
They build it years before Bazel was opensourced.
I find it weird that everyone acts like Twitter is a new startup that created all of it's infra 1-2 years ago, while they had 100 millions of users in 2012 - a lot of this stuff was already years in the making then.
I believed he mentioned early on that it was due to the perf gains Bazel provided.
I'm curious what you think it's possible to buy that replaces the need to invest on top of Bazel. Have you worked on a comparably sized codebase that used a simpler solution?
An unfortunate title, completely devoid of any hint about the Very Interesting content in this post:
This is an insider story chronicling the evolution of Twitter 1.0's approach to build infrastructure and developer efficiency strategies at scale on teams of hundreds to thousands of SWEs.
This is an insider story chronicling the evolution of Twitter 1.0's approach to build infrastructure and developer efficiency strategies at scale on teams of hundreds to thousands of SWEs.
Please excuse my ignorance. Why are 1,000+ SWEs needed (let alone custom tooling to support them)?
Twitter features I know about:
1. Timeline recommendations (with personalized/relevant ads sprinkled in)
2. Notifications (so you get pushed an iOS/Android alert when somebody you subscribe to tweets)
3. Being able to post/view media (image/video/polls)
4. Direct private messaging
5. Concept of following/followers with recommendations
6. Threads/retweets/replies
7. Advertising dashboard/analytics for advertisers to create + track campaigns
8. "Explore" / "What's happening" / automatically serve "trending" content / topics
9. Lists / bookmarks
10. Anti-spam / anti-fraud
11. User profiles / settings
If I missed noteworthy/sizable/heavily used by masses features, please let me know.
Why can't that be managed + developed on new features wise with 1,000 SWEs? or 500? Genuinely asking. Would love to learn what I'm missing.
Twitter features I know about:
1. Timeline recommendations (with personalized/relevant ads sprinkled in)
2. Notifications (so you get pushed an iOS/Android alert when somebody you subscribe to tweets)
3. Being able to post/view media (image/video/polls)
4. Direct private messaging
5. Concept of following/followers with recommendations
6. Threads/retweets/replies
7. Advertising dashboard/analytics for advertisers to create + track campaigns
8. "Explore" / "What's happening" / automatically serve "trending" content / topics
9. Lists / bookmarks
10. Anti-spam / anti-fraud
11. User profiles / settings
If I missed noteworthy/sizable/heavily used by masses features, please let me know.
Why can't that be managed + developed on new features wise with 1,000 SWEs? or 500? Genuinely asking. Would love to learn what I'm missing.
At my company there are like 20-30 engineers involved in search (using ElasticSearch). The code has grown massively and they are all bogged down fixing small stuff, improvements, doing A/B-tests and the like. They also have a whole ETL setup to gather and transform data from a lot of different sources.
In my spare-time I build a small search function with ElasticSearch for a big dataset. The use case was almost identical; and I feel my small setup works better, faster and has more options than where I work.
I don't know, things just grow over time, old developers leave and new ones join who will never be as productive as the original devs were (it's basically rewriting someone else's book without breaking it).
But I'm puzzled about it too sometimes.
In my spare-time I build a small search function with ElasticSearch for a big dataset. The use case was almost identical; and I feel my small setup works better, faster and has more options than where I work.
I don't know, things just grow over time, old developers leave and new ones join who will never be as productive as the original devs were (it's basically rewriting someone else's book without breaking it).
But I'm puzzled about it too sometimes.
Almost all of the things in your list are harder to do at scale, especially anti-spam/anti-fraud. Plus, all of this needs to be done in compliance with the legal regimes of many countries where Twitter operates -- especially regarding advertising and privacy. Having seen how some other large web services work from the inside, I'm not surprised Twitter needs a lot of people.
It would be possible to pull this off with far fewer engineers, but not sustainably. People aren't robots, they can't work 80-hour weeks forever. People also need to be able to take vacations, and sometimes they will leave the company -- so it's not a good idea to run super lean. Plus, when companies run lean, the security and privacy aspects of the engineering tend not to be done very well (if at all), and for a social network that is a serious issue.
Twitter was definitely bloated, but I've seen and heard of worse bloat at other tech companies.
It would be possible to pull this off with far fewer engineers, but not sustainably. People aren't robots, they can't work 80-hour weeks forever. People also need to be able to take vacations, and sometimes they will leave the company -- so it's not a good idea to run super lean. Plus, when companies run lean, the security and privacy aspects of the engineering tend not to be done very well (if at all), and for a social network that is a serious issue.
Twitter was definitely bloated, but I've seen and heard of worse bloat at other tech companies.
You're not missing anything. WhatsApp had 14 million active users with 32 engineers. Instagram had 30 million active users with 13 employees. It's just delusional to think you need four-figure engineers to run Twitter—a product that basically hasn't changed for over 15 years.
I can't find it now, but someone in another post on Twitter pointed out that these comparisons aren't very meaningful: Twitter's social topology is any-any between users. A major (and unique) aspect of Twitter is that everybody on Twitter can talk to just about anybody else, and is encouraged to. This is also a critical value proposition for advertising on Twitter, and probably explains the size of their ads team.
This is very different from WhatsApp's topology, which is a relatively traditional chat system (small groups of shared state). Instagram's is closer, but still favors smaller social groups over a global feed.
Beyond that, Twitter is well known for running its own DCs, etc. It doesn't seem inconceivable that all of this, plus internationalization, mature platforms teams, etc. all adds up to four figures.
This is very different from WhatsApp's topology, which is a relatively traditional chat system (small groups of shared state). Instagram's is closer, but still favors smaller social groups over a global feed.
Beyond that, Twitter is well known for running its own DCs, etc. It doesn't seem inconceivable that all of this, plus internationalization, mature platforms teams, etc. all adds up to four figures.
It boggles my mind how much people think "active users" is like some kind of fungible unit for engineering work required.
I don't think there's a inviolable ratio between active users and engineers, but you do absolutely need more engineers as your user base increases: you need more people on call, more people maintaining the servers you're running (especially if, like Twitter, you run your own DCs), more people doing QA and reliability engineering because of increased load, etc.
It's absolutely an OKR which implies higher workloads, are you kidding me‽ I'm sure you can build some architectural rat's nest where you need 100 engineers to maintain systems that barely serve a dozen requests a second (kind of like the blockchain) but that doesn't mean you couldn't do a much better job.
Everything else being equal, sure, more traffic is harder. But everything else is generally not equal. And despite your sleight of hand in this comment, your original claim about what's "delusional" was based on asserting an equivalence between the WhatsApp and Twitter workloads.
Sometimes it is, until you start comparing between wildly different companies and business models.
While the product looks like it hasn't changed much from the outside, it's been completely rewritten once or twice, at least. When Twitter launched it was a Ruby on Rails app. They scaled it to the absolute limit, then started breaking services away using Scala, then they kept at it adding several other languages to the mix.
While all this happened, there was almost zero product guidance. Instead, they watched for usage patterns, and started to incorporate those into the product. The original Twitter didn't have mentions, replies, or hashtags. Adding support for all that on a product not designed for it while managing exponential growth is an extremely difficult challenge.
On top of that, Twitter added divisions that are known to be engineer-heavy, such as ads, search, and infrastructure.
---
The comparison with Whatsapp and Instagram is interesting. Whatsapp's case is special: they famously opted to use Erlang and it turned out to be a genius decision, as that allowed them to scale beyond expectations. Whatsapp never changed much from its initial design, which means no need for deep rewrites. Instagram was a Django app running on AWS servers, and that setup was enough until they got bought by Facebook and eventually moved from AWS to FB's infrastructure.
While all this happened, there was almost zero product guidance. Instead, they watched for usage patterns, and started to incorporate those into the product. The original Twitter didn't have mentions, replies, or hashtags. Adding support for all that on a product not designed for it while managing exponential growth is an extremely difficult challenge.
On top of that, Twitter added divisions that are known to be engineer-heavy, such as ads, search, and infrastructure.
---
The comparison with Whatsapp and Instagram is interesting. Whatsapp's case is special: they famously opted to use Erlang and it turned out to be a genius decision, as that allowed them to scale beyond expectations. Whatsapp never changed much from its initial design, which means no need for deep rewrites. Instagram was a Django app running on AWS servers, and that setup was enough until they got bought by Facebook and eventually moved from AWS to FB's infrastructure.
> with ads sprinkled in
They aren't just splatting a tag from a third party ad platform into the tweets, they're building on your own data and handling the entire auction/bidding process internally along with all the tools for advertiser to manage that, click fraud prevention, etc. There are many 100-500 engineer companies built on doing this, along with lots of 50-100 engineer companies doing just one part of this.
They aren't just splatting a tag from a third party ad platform into the tweets, they're building on your own data and handling the entire auction/bidding process internally along with all the tools for advertiser to manage that, click fraud prevention, etc. There are many 100-500 engineer companies built on doing this, along with lots of 50-100 engineer companies doing just one part of this.
Each bullet point can be a massive undertaking, especially at scale. "Being able to post/view media (image/video/polls)" think about how much media that is. How much infrastructure you need to support that. How much it takes to make tools to manage that infrastructure.
[deleted]
twitter runs its own datacenters and has teams to support that. Since its not in a cloud the company runs all their own internal platform stuff you would get on something like aws. that is a large engineering effort. theres also the andorid and ios app, more teams. elon tweeted out a picture for tweet or timeline reads i think. each square is basically a team with an oncall rotation. this is off the top of my head, there's plenty more not visible stuff happening
You could probably build it in a weekend.
[deleted]
> Why can't that be managed + developed on new features wise with 1,000 SWEs? or 500?
Because the bigger the company the more valuation it gets (although, with the latest layoffs and all, I think this won't be the case much more). Why would Twitter (or any tech company) would remain static in size (let's say at 500 engineers) when it can just grow and grow? As long as money comes in (funding) they company has to do something with it (hiring and invent new areas to explore that requires more hiring).
It's not a question of technical requirements, it's just all about money.
Because the bigger the company the more valuation it gets (although, with the latest layoffs and all, I think this won't be the case much more). Why would Twitter (or any tech company) would remain static in size (let's say at 500 engineers) when it can just grow and grow? As long as money comes in (funding) they company has to do something with it (hiring and invent new areas to explore that requires more hiring).
It's not a question of technical requirements, it's just all about money.
Google has thousands of engineers just on point 1 of your list - search results.
And way more on the “ads sprinkled in” bit that wasn’t even its own line.
See how long it took them to develop their own shorted links and image uploads.
“ads sprinkled in” could be 1,000 engineers without surprising me.
Anything could be 1k engineers if you mismanage it enough
> Looking back, I feel like this has been among the most productive 2 years of my career. This was facilitated precisely because of the work/life balance afforded by the “peak-Kumbaya” Twitter culture, and the psychological safety
I feel like the most productive I’ve been is
1. Psychological safety and trust in engineering to lead
2. Coworkers that intrinsically care about their work
3. Coworkers that are fantastic people
4. Meaningful work
The top down, hard ass culture of Elon etc, basically flips the work to being extrinsically motivated based on fear. If the goal is engineering excellence, it just doesn’t work. A lot of people shut down and won’t take any risks or innovate in that situation. Or they leave.
There may be other goals of course, but there’s a valuable reason tech companies have been selective in hiring and generous in trust.
I feel like the most productive I’ve been is
1. Psychological safety and trust in engineering to lead
2. Coworkers that intrinsically care about their work
3. Coworkers that are fantastic people
4. Meaningful work
The top down, hard ass culture of Elon etc, basically flips the work to being extrinsically motivated based on fear. If the goal is engineering excellence, it just doesn’t work. A lot of people shut down and won’t take any risks or innovate in that situation. Or they leave.
There may be other goals of course, but there’s a valuable reason tech companies have been selective in hiring and generous in trust.
> The top down, hard ass culture of Elon etc, basically flips the work to being extrinsically motivated based on fear.
I worked at a company (well known in tech circles) where the CEO went through a phase of ruling by fear. He was firing and replacing people constantly, ever present in Slack to criticize work and threaten people if he thought delivery wasn’t fast enough, and even fired a couple people on the spot in front of a large group of people just to make a point.
It created a wave of productivity at first. We all thought we could just get ahead of it by doing good work. If we work hard, we won’t end up embarrassed and disgraced like our fired ex-colleagues.
But after a while everyone realizes that there is no winning in a fear-based environment. It doesn’t matter how hard you work or how great your results, the fear-based boss will always find some other excuse to apply more fear push everyone harder. People must be sacrificed constantly to maintain the image of a ruthless leader.
Eventually everyone rotates out of the company into better jobs. The company became a revolving door. Product quality declined as knowledge was constantly lost with departing employees.
I worked at a company (well known in tech circles) where the CEO went through a phase of ruling by fear. He was firing and replacing people constantly, ever present in Slack to criticize work and threaten people if he thought delivery wasn’t fast enough, and even fired a couple people on the spot in front of a large group of people just to make a point.
It created a wave of productivity at first. We all thought we could just get ahead of it by doing good work. If we work hard, we won’t end up embarrassed and disgraced like our fired ex-colleagues.
But after a while everyone realizes that there is no winning in a fear-based environment. It doesn’t matter how hard you work or how great your results, the fear-based boss will always find some other excuse to apply more fear push everyone harder. People must be sacrificed constantly to maintain the image of a ruthless leader.
Eventually everyone rotates out of the company into better jobs. The company became a revolving door. Product quality declined as knowledge was constantly lost with departing employees.
And then?
Twitter was losing millions of dollars each day. It wasn’t profitable or even break even.
An engineering culture that’s awesome at a company like that will only last until the money runs out.
I’m not supporting what Elon is doing. I’m just saying that what Twitter was doing before wasn’t sustainable or on a path to being sustainable. The culture was part of that
An engineering culture that’s awesome at a company like that will only last until the money runs out.
I’m not supporting what Elon is doing. I’m just saying that what Twitter was doing before wasn’t sustainable or on a path to being sustainable. The culture was part of that
I don’t understand why people keep saying this. Twitter was a public company. You can see from their financial statements that they recorded positive operating income for all of 2021.
Not very positive! They weren’t a great business!
Then they had to pay a huge fine, and lost a bunch of money because of it.
But they weren’t operating at a loss. Certainly not “millions a day.” Where do people get that figure?
2020, they had a $1B net loss for the year. 2019, they had a $1B net gain.
I dunno, it’s clear they were a weird and not great business. But like, they weren’t JUST lighting cash on fire.
Not very positive! They weren’t a great business!
Then they had to pay a huge fine, and lost a bunch of money because of it.
But they weren’t operating at a loss. Certainly not “millions a day.” Where do people get that figure?
2020, they had a $1B net loss for the year. 2019, they had a $1B net gain.
I dunno, it’s clear they were a weird and not great business. But like, they weren’t JUST lighting cash on fire.
It's frustrating to regularly hear people (particularly on HN, but not exclusively) justifying, or even just giving a light nod of approval, to Musk's horrendous "leadership" via firings, simply because twitter isn't profitable in their eyes.
It seems like cherry-picking excuses to support Musk's leadership style rather than any justified basis. If the parent comment is correct, they just aren't as profitable as Wall Street would like (therefore they are not "profitable"). Saying that the firings are justified is then just moving the goalposts to support an a-priori conviction.
Additionally, it seems ironic that this is coming from proponents of the Silicon-Valley ethos of "screw profitability, we'll figure it out later"... an ethos that created Amazon and so many other tech titans. Yes, profitability matters, but only when your other forms of capital (e.g., social, political) have been depleted.
It's tragic that Musk is depleting whatever capital twitter had remaining on its current death march. Financially "profitable" or not, twitter did undeniably have social and political capital. Soon, it will have neither.
It seems like cherry-picking excuses to support Musk's leadership style rather than any justified basis. If the parent comment is correct, they just aren't as profitable as Wall Street would like (therefore they are not "profitable"). Saying that the firings are justified is then just moving the goalposts to support an a-priori conviction.
Additionally, it seems ironic that this is coming from proponents of the Silicon-Valley ethos of "screw profitability, we'll figure it out later"... an ethos that created Amazon and so many other tech titans. Yes, profitability matters, but only when your other forms of capital (e.g., social, political) have been depleted.
It's tragic that Musk is depleting whatever capital twitter had remaining on its current death march. Financially "profitable" or not, twitter did undeniably have social and political capital. Soon, it will have neither.
Well, twitter being badly managed company and musk being a wanker at same time is also pretty common sentiment. Just that musk fanboys are pretty loud.
> It's tragic that Musk is depleting whatever capital twitter had remaining on its current death march. Financially "profitable" or not, twitter did undeniably have social and political capital. Soon, it will have neither.
Is it tho ? The social capital Twitter had was just used to push ideas of whoever was in charge there at the time. It was slowly becoming a propaganda machine that was disguised as social network.
Just like a week ago we had HN discussion about how after firing a bunch of people Japanese twitter suddenly stopped being so political [1], coz those topics were being forced.
Twitter dying in fiery musky fireball might be the best thing to happen in long term.
* [1] https://www.forbes.com/sites/olliebarder/2022/11/14/japanese...
> It's tragic that Musk is depleting whatever capital twitter had remaining on its current death march. Financially "profitable" or not, twitter did undeniably have social and political capital. Soon, it will have neither.
Is it tho ? The social capital Twitter had was just used to push ideas of whoever was in charge there at the time. It was slowly becoming a propaganda machine that was disguised as social network.
Just like a week ago we had HN discussion about how after firing a bunch of people Japanese twitter suddenly stopped being so political [1], coz those topics were being forced.
Twitter dying in fiery musky fireball might be the best thing to happen in long term.
* [1] https://www.forbes.com/sites/olliebarder/2022/11/14/japanese...
>It was slowly becoming a propaganda machine that was disguised as social network.
Is there a difference? From what we've seen over the past couple of decades, it seems like social networks are doomed to devolve into propaganda machines.
>Twitter dying in fiery musky fireball might be the best thing to happen in long term.
That's what I feel like too.
Is there a difference? From what we've seen over the past couple of decades, it seems like social networks are doomed to devolve into propaganda machines.
>Twitter dying in fiery musky fireball might be the best thing to happen in long term.
That's what I feel like too.
There is a difference between people using platform as propaganda machine and platform itself picking a side and deprioritizing anything that they don't like, especially with reach it has.
First one is de facto popularity contest and each side is free to call on other's bullshit, if control comes from company running it the detractors from "right" vision gets silenced,either explicitly, or by just modifying algorithm to deprioritize their content
First one is de facto popularity contest and each side is free to call on other's bullshit, if control comes from company running it the detractors from "right" vision gets silenced,either explicitly, or by just modifying algorithm to deprioritize their content
It isn't cherry-picking.
Twitter has a very long runway where > 90% of its existence it was making a loss.
Where competing social networks have exponentially grown in size (FB, Insta, Tiktok, even Reddit), Twitter has stagnated.
Twitter is widely known to never really ship anything. So I have no idea which "velocity" the author is talking about.
Before the musk take-over, a plan was already in the works to cut 800M in expenses.
May I remind you that Twitter almost went bankrupt in 2016? They tried to sell but nobody wanted it.
Twitter has a very long runway where > 90% of its existence it was making a loss.
Where competing social networks have exponentially grown in size (FB, Insta, Tiktok, even Reddit), Twitter has stagnated.
Twitter is widely known to never really ship anything. So I have no idea which "velocity" the author is talking about.
Before the musk take-over, a plan was already in the works to cut 800M in expenses.
May I remind you that Twitter almost went bankrupt in 2016? They tried to sell but nobody wanted it.
Twitter stock returned ~30% vs 190% for nasdaq in the same period... let's not pretend this company is even remotely close to being performant. I'd be far more inclined to believe "we rolled our own" engineering excellence stories from a massively profitable organization, rather than this confused-about-being-a-nonprofit company.
Also more broadly, people quitting en-masse with new leadership attempting to make the place profitable should tell you all you need to know about the culture.
Also more broadly, people quitting en-masse with new leadership attempting to make the place profitable should tell you all you need to know about the culture.
> Also more broadly, people quitting en-masse with new leadership attempting to make the place profitable should tell you all you need to know about the culture
Trying to make the place profitable by firing large swaths of their coworkers. You make it sound like people started jumping overboard right when the rescue crew was on the way, rather than when a new captain took over and started trying to stop the ship from sinking by forcibly tossing people himself.
Trying to make the place profitable by firing large swaths of their coworkers. You make it sound like people started jumping overboard right when the rescue crew was on the way, rather than when a new captain took over and started trying to stop the ship from sinking by forcibly tossing people himself.
Elon is going far beyond "attempting to make the place profitable"
[deleted]
It’s because people are looking at him and saying, “whoa, I can behave like that and not only get away with it, but win by it?!”
There’s room for sincere curiosity about what’s going on and what’ll happen, but a lot of what I see online is people just relishing in the sadism of the whole ordeal.
There’s room for sincere curiosity about what’s going on and what’ll happen, but a lot of what I see online is people just relishing in the sadism of the whole ordeal.
The cruelty does seem to be the point.
Not sure how we escape the gravity of it.
Not sure how we escape the gravity of it.
Part of the mostly made up version of the American Dream is like "pull yourself up by your bootstraps" which oddly also implies "and then once you have, close the door behind you. Don't want this party getting too crowded". My point is that fetishizing capitalism requires a taste for squeezing others for all their worth. If others aren't, then that means you endured that bootstrap pain for no reason!
I mean, if you don't like Twitter, and don't like musk, the whole thing is very entertaining
I think this is nearly as destructive of a perspective. People have become way, way too obsessed with what is entertaining and are willing to accept way too much horrible stuff in the name of the lulz.
It’s really not a game show. There are important ideas being tested and norms being created (or destroyed).
It’s really not a game show. There are important ideas being tested and norms being created (or destroyed).
Well I see Twitter in its previous form as actively harmful to society, especially since few years ago when Twitter decided that they no longer just show what's popular but actively manipulate it [1]. Like, the short format was bad enough on its own (encourages shouting in the void instead of actual discussion), and that's just more bad on top of bad
And I also see musk's compulsive lying and arrogance as harmful, althought in more limited scope.
So from my perspective it's one bad thing destroying other bad thing. What's not to be happy about ?
Sure, Twitter transforming into not a vile piece of shit it is today into something decent would also be nice outcome but I don't even know how that would work so it burning is next best thing.
* [1] https://www.forbes.com/sites/olliebarder/2022/11/14/japanese...
And I also see musk's compulsive lying and arrogance as harmful, althought in more limited scope.
So from my perspective it's one bad thing destroying other bad thing. What's not to be happy about ?
Sure, Twitter transforming into not a vile piece of shit it is today into something decent would also be nice outcome but I don't even know how that would work so it burning is next best thing.
* [1] https://www.forbes.com/sites/olliebarder/2022/11/14/japanese...
There are a lot of things to criticize about Twitter but it shows a lack of imagination if you think the worst possible thing you could do with a 160MM+ daily active users is give them short form conversation dynamics, and that the other possible outcome is “it burns.”
It is absolutely possible that Twitter turns into a 4chan-esque cesspool of vitriol and hatred but at massive scale due to a gradual frog boil of 160MM+. Not to say Twitter had none of this, but again, it takes little imagination to worry about it getting much, much worse.
It is absolutely possible that Twitter turns into a 4chan-esque cesspool of vitriol and hatred but at massive scale due to a gradual frog boil of 160MM+. Not to say Twitter had none of this, but again, it takes little imagination to worry about it getting much, much worse.
How is it horrible? In my view, Twitter is actively harmful to society, so seeing it crash and burn is not only funny, but looks like a good thing, even if it means some highly-paid SWEs need to find new jobs.
A 160MM+ DAU social network “crashing and burning” doesn’t mean “poof” one night and it’s gone.
The biggest self-own in the industry will be if Elon is successful and then the whole software industry suddenly is inspired not by (yes, perhaps excessive) kindness/comfort/generosity of a good work life but by cut throat PE-style management with an added layer of emotional manchild tantrums.
There’s a reason VCs are cheering this on and it’s because lower standards in the software industry will save them all money.
The biggest self-own in the industry will be if Elon is successful and then the whole software industry suddenly is inspired not by (yes, perhaps excessive) kindness/comfort/generosity of a good work life but by cut throat PE-style management with an added layer of emotional manchild tantrums.
There’s a reason VCs are cheering this on and it’s because lower standards in the software industry will save them all money.
Well hopefully Twitter will completely implode and what's left will be sold off for a tiny, tiny fraction of what Elon spent for it. Then the VCs will have to come to grips with the idea that having lower standards and cutthroat management doesn't result in profitability, but rather utter failure.
lulz certainly abound here--there's been a lot of great comedy in this.
But I read the GP to mean they've seen people crossing a thin line.
I doubt any of us are above a bit of schadenfreude when the fates deal a fair dose of karma to someone we think deserves it.
But actively cheering (and trying to multiply, in the case of aggressive trolling) the sadism in public is a kind of dark prayer.
But I read the GP to mean they've seen people crossing a thin line.
I doubt any of us are above a bit of schadenfreude when the fates deal a fair dose of karma to someone we think deserves it.
But actively cheering (and trying to multiply, in the case of aggressive trolling) the sadism in public is a kind of dark prayer.
Oh, some highly paid people (that made product that actively made society worse) will need to find a new job, how utterly sadistic /s
Finding pleasure in the suffering of other people is quite literally the definition of sadism. Be proud of it if you want, but that’s what it is.
Also, it’s very obviously not true that all the people who lost their jobs are highly paid. It’s curious how this self-deception keeps appearing among people who are cheering it on.
Also, it’s very obviously not true that all the people who lost their jobs are highly paid. It’s curious how this self-deception keeps appearing among people who are cheering it on.
Spot on.
>to Musk's horrendous "leadership" via firings
He isn't firing those people as a component of "leadership", he's clearly cleaning house. He doesn't _want_ the vast majority of twitter staff to remain for obvious reasons.
He isn't firing those people as a component of "leadership", he's clearly cleaning house. He doesn't _want_ the vast majority of twitter staff to remain for obvious reasons.
He obviously doesn't want the majority of staff to say. I don't think it is for obvious reasons. In fact, I doubt he knows what the majority of the staff does, he just assumes it's obvious.
What's the obvious reason?
Let's just say Twitter staff weren't very happy and vocal about it when before acquisition he said he want it to be more free speech platform, and most of those people looked pretty left leaning.
Now of course that was tiny minority of workers there, but there seemed to be zero that's happy about it before acquisition so the "obvious reason" is "aside from cost cutting he's kicking people that would give resistance to whatever he wants to do with twitter"
Now of course that was tiny minority of workers there, but there seemed to be zero that's happy about it before acquisition so the "obvious reason" is "aside from cost cutting he's kicking people that would give resistance to whatever he wants to do with twitter"
can we please stop with this 'freedom of speech' bogeyman that Musk spouts? He cherry-picks what previously banned accounts get unbanned, he fires employees for making public tweets he doesn't like, as well as firing employees for making comments in company slack channels critical of his leadership - which is a notable departure from previous company policy.
Of course if employees don't believe in leaderships vision they might not be a good fit. Of course a company looking to cut costs is going to look at trimming their workforce. This all makes sense, it just has absolutely nothing to do with free speech.
Of course if employees don't believe in leaderships vision they might not be a good fit. Of course a company looking to cut costs is going to look at trimming their workforce. This all makes sense, it just has absolutely nothing to do with free speech.
Oh I don't believe that compulsive liar for a second, I'm just saying that it triggered some people once he started talking about maybe "moderating" (censoring by any other name) less of it
I think they got it from Elon shortly after he took over, turned everything into chaos, and advertisers started bailing. That being said, he also appears to be a pathological liar, so he could just be making it up on top of that.
But yes, this notion that Twitter was just hemorrhaging money seems flatly false and easily disprovable. I haven't read the financial statements in a while, but when I last did I indeed saw one of the few tech companies that was actually profitable.
But yes, this notion that Twitter was just hemorrhaging money seems flatly false and easily disprovable. I haven't read the financial statements in a while, but when I last did I indeed saw one of the few tech companies that was actually profitable.
>That being said, he also appears to be a pathological liar
Is there a such thing as a person who reaches that level of wealth who isn't? Perhaps people who were born with it might not be, but it seems to be a prerequisite for people who achieve that level of wealth themselves.
Is there a such thing as a person who reaches that level of wealth who isn't? Perhaps people who were born with it might not be, but it seems to be a prerequisite for people who achieve that level of wealth themselves.
Buffett and Munger seem like genuinely good guys from the various talks and writings I’ve seen of theirs.
I guess there's always an exception.
> Certainly not “millions a day.” Where do people get that figure?
I suspect that $4m a day number Musk has been tossing around is rooted in his terrible financial investment. He purchased the company for $44b, which has something like $1.4b in interest every year? Which would be $3.8m a day.
So Twitter wasn't losing $4m a day--Musk was.
I suspect that $4m a day number Musk has been tossing around is rooted in his terrible financial investment. He purchased the company for $44b, which has something like $1.4b in interest every year? Which would be $3.8m a day.
So Twitter wasn't losing $4m a day--Musk was.
No, Musk signed for the loan, but as Twitters CEO. Twitter is on the hook, not him personally.
I'd imagine by googling it and getting this:
https://www.statista.com/statistics/274563/annual-net-income...
https://www.businessofapps.com/data/twitter-statistics/
and see that revenue is growing but it's still in the red, and obvious conclusion to that being "they hired a lot of people that don't bring that much profit"
https://www.statista.com/statistics/274563/annual-net-income...
https://www.businessofapps.com/data/twitter-statistics/
and see that revenue is growing but it's still in the red, and obvious conclusion to that being "they hired a lot of people that don't bring that much profit"
Elon said it, and it was what they wanted to hear. Like most information on the internet.
I'm curious to know if anyone has accounted for losses from all the prior years? Those still need to be paid back/accounted for. Did they break-even?
[deleted]
I've seen this sentiment elsewhere, but it doesn't match my reading of Twitter's financial statements: https://investor.twitterinc.com/financial-information/quarte...
In 2021, Twitter would have been profitable were it not for a ~$800 million legal settlement payout.
They posted a net profit in Q4 2021 and Q1 2022.
In 2021, Twitter would have been profitable were it not for a ~$800 million legal settlement payout.
They posted a net profit in Q4 2021 and Q1 2022.
Twitter pre-Elon wasn't losing millions of dollars a day. Twitter post-Elon is losing millions of dollars a day because of the extra $1B/year in interest payments he took on because he wildly overpaid for the company right before everyone figured out just how badly Apple's new privacy practices affected the revenue of social media companies.
Elon losing money buying Twitter does not mean Twitter itself is losing more money.
Actually it does: he financed it like a PE looter, adding the debt to the target company. That gave Twitter an automatic $1-1.5 billion annual setback compared to where they had been.
https://www.bloomberg.com/opinion/articles/2022-04-21/elon-g...
https://www.bloomberg.com/opinion/articles/2022-04-21/elon-g...
The creditors must enjoy Elon talking about bankruptcy already. Maybe he thought he could stop the bleeding by firing everyone. But with advertisers out and revenue gone, he's set for an ugly legal battle in a few quarters. This has to be the most perilous business deal in the history of business and deals.
Yeah, I was expecting him to make a few cuts and focus on subscriptions or other revenue sources but I guess the losses he was looking at spurred a swing for the fences. I feel bad for anyone at Twitter who’s along for the ride.
The H1-B workers clinging on for dear life?
Yeah, that’s most of who I was thinking about along with anyone whose skills aren’t software developer-level portable. This kind of thing can be really disruptive to families if it forces a move.
It literally is. That's how every debt-financed corporate acquisition works. Twitter is paying this debt, not Elon from his pocket.
https://www.wsj.com/articles/elon-musks-buyout-will-load-twi...
https://www.wsj.com/articles/elon-musks-buyout-will-load-twi...
So if it goes bankrupt what happens ? None of his assets aside Twitter gets seized ?
If Twitter fails to make payments on its debt then Morgan Stanley will become the proud new owner of Twitter. It is similar in structure to a mortgage on a house.
That seems like a really stupid loan to make then. If Twitter fails, it becomes mostly worthless, much like MySpace today. Why would a bank make such a loan? (It does seem like bankers are pretty stupid these days though, or just don't care because it's not their money.)
They set an interest rate in line with the risk that they are taking on. It seems like a pretty unlikely scenario where Twitter is worth nothing, if it goes bankrupt there can probably be a fairly sizable sale to recoup some value.
Allowing so long for the deal to close may have been a mistake due to what happened in the market between when they agreed to terms and when the deal closed.
Allowing so long for the deal to close may have been a mistake due to what happened in the market between when they agreed to terms and when the deal closed.
He already paid 20B+ of his own money. Debt is only part of financing, AFAIK ~13B.
I think your argument reduces to "culture is better the more you spend on people" and "Twitter overspent on engineering people in order to create culture."
This is false. Twitter was profitable in 2017, 2018, 2019 and 2020 and its R&D budget was half that of 2022, or less, for those years and by all accounts it had a fantastic culture.
In 2021, the last year available and its very worst year since 2015, Twitter made $5 billion in revenue and lost $492 million. So your "millions of dollars a day" is not wrong-- $1.4 million per day.
Twitter spend $1.2 billion on R&D-- less than 25% of its revenue. In 2020, it spend $873 million on R&D and made a profit. In 2019, it spent a mere $682 million on R&D and made a $366 million dollar profit. Similarly in 2018.
This is false. Twitter was profitable in 2017, 2018, 2019 and 2020 and its R&D budget was half that of 2022, or less, for those years and by all accounts it had a fantastic culture.
In 2021, the last year available and its very worst year since 2015, Twitter made $5 billion in revenue and lost $492 million. So your "millions of dollars a day" is not wrong-- $1.4 million per day.
Twitter spend $1.2 billion on R&D-- less than 25% of its revenue. In 2020, it spend $873 million on R&D and made a profit. In 2019, it spent a mere $682 million on R&D and made a $366 million dollar profit. Similarly in 2018.
The loss in 2021 was due to 765 million dollar lawsuit settlement. Also, after that loss was 221 million not 492. It would be 0.5B profitable if not that.
Excluding a settlement from 2014 that was paid this year, Twitter would have made 500mil on 5B revenue in 2021. They were also GAAP profitable in 2019.
Additionally, the incentive from wall street is to spend all your money to grow users. Whether that is right or wrong it’s the path they chose which led to users and revenue roughly doubling over 5 years.
https://s22.q4cdn.com/826641620/files/doc_financials/2021/ar...
https://s22.q4cdn.com/826641620/files/doc_financials/2019/Fi...
Additionally, the incentive from wall street is to spend all your money to grow users. Whether that is right or wrong it’s the path they chose which led to users and revenue roughly doubling over 5 years.
https://s22.q4cdn.com/826641620/files/doc_financials/2021/ar...
https://s22.q4cdn.com/826641620/files/doc_financials/2019/Fi...
People on HN used to poke fun at unprofitable social media companies, and about how they're all declining. Its interesting to see from outside the political bubble how the conversations have now shifted to defending Twitters' income statement.
Sounds like it was a business problem, not an engineering or culture problem.
> The top down, hard ass culture of Elon etc, basically flips the work to being extrinsically motivated based on fear.
Do you really believe that? If you look at videos from SpaceX launches for example, do you see scared people motivated by their fear of Elon…?
I’m not saying everything Elon does is golden. But there seems to be a major gap in understanding here between different personality types. I find that interesting. If you could elaborate a bit on your point of view it would be much appreciated.
Do you really believe that? If you look at videos from SpaceX launches for example, do you see scared people motivated by their fear of Elon…?
I’m not saying everything Elon does is golden. But there seems to be a major gap in understanding here between different personality types. I find that interesting. If you could elaborate a bit on your point of view it would be much appreciated.
Twitter is far far away from engineering excellence tho. It hires a lot of engineers to do relatively little. You can feel "productive" without actually producing anything useful for company too. Like, after their whole 2 years of being profitable their revenue steadily increased... but costs increased faster which leads to obvious question "what are those people doing?".
What musk is doing is obviously not sustainable and can ruin technical competency but it is not exactly example of "ruling by performance" as much as him just cleaning house/cost cutting company that was not profitable for more years than profitable.
Also people's motivation are different. Allowing people to self govern and do what they want to makes for happy employees but shit output, example being Valve with it famously flat structure and barely any release. You do need some pressure to make stuff useful for the company and you do need some pressure to actually finish it instead of playing around.
Rewarding innovation sounds good on paper but ends to Googler complex where nobody wants to do maintenance as you won't get pay rise out of it compared to working on some cash cow or something new, if there are no incentives for maintenance.
Cutting low performers no matter what gets you stack ranking and old Microsoft, and team that happened to hire only good people are hurt for no good reason.
What musk is doing is obviously not sustainable and can ruin technical competency but it is not exactly example of "ruling by performance" as much as him just cleaning house/cost cutting company that was not profitable for more years than profitable.
Also people's motivation are different. Allowing people to self govern and do what they want to makes for happy employees but shit output, example being Valve with it famously flat structure and barely any release. You do need some pressure to make stuff useful for the company and you do need some pressure to actually finish it instead of playing around.
Rewarding innovation sounds good on paper but ends to Googler complex where nobody wants to do maintenance as you won't get pay rise out of it compared to working on some cash cow or something new, if there are no incentives for maintenance.
Cutting low performers no matter what gets you stack ranking and old Microsoft, and team that happened to hire only good people are hurt for no good reason.
None of what you wrote holds up when you look at the outcomes though.
Twitter "engineering" is a complete joke, the website is somehow less reliable than _reddit_ and that is saying something. There has been little to no innovation at Twitter for how long? Pretty much since it blew up.
Meanwhile the culture you're criticizing is responsible for Tesla and SpaceX, both extreme disruptors in their spaces. You don't have to like them, but you can't deny they are extremely innovative companies.
Twitter "engineering" is a complete joke, the website is somehow less reliable than _reddit_ and that is saying something. There has been little to no innovation at Twitter for how long? Pretty much since it blew up.
Meanwhile the culture you're criticizing is responsible for Tesla and SpaceX, both extreme disruptors in their spaces. You don't have to like them, but you can't deny they are extremely innovative companies.
> The top down, hard ass culture of Elon etc, basically flips the work to being extrinsically motivated based on fear.
Why not both? You can have a hard-ass culture of relentless insistence on highest standards and a good ambiance within the team.
Why not both? You can have a hard-ass culture of relentless insistence on highest standards and a good ambiance within the team.
And I'm sure there are companies that work like that but clearly that's not elon with Twitter and looking at his glorification of overtime probably not Tesla nor SpaceX
Us 3 friends built connecting cities in SimCity 2013 ("one of the most disastrous launches in history" [0]). Unintentional but maybe by unconscious belief in how things should be, each of our 3 cities reflected our understanding of what work and production should be like:
- (mine, software engineer) city life, large middle class, high taxes, some luxury districts
- (environmental city planner) Almost entirely renewable energy, vast public transit, living and jobs spread evenly on the map
- ([company redacted] clerk) giant coal mine in the center of the city, mass low-end housing, unchecked health issues, ruled by iron fist
[0] https://medium.com/age-of-awareness/why-simcity-died-and-how...
Edit: redacted company name due to people really liking their samples.
- (mine, software engineer) city life, large middle class, high taxes, some luxury districts
- (environmental city planner) Almost entirely renewable energy, vast public transit, living and jobs spread evenly on the map
- ([company redacted] clerk) giant coal mine in the center of the city, mass low-end housing, unchecked health issues, ruled by iron fist
[0] https://medium.com/age-of-awareness/why-simcity-died-and-how...
Edit: redacted company name due to people really liking their samples.
All those thousands of engineers, millions of lines of code, and they couldn’t add an edit button. Burn that bloated mess to the ground and do version 2, a fraction of the code size, a fraction of the engineers required to write and maintain it. Musk didn’t arrive soon enough.
[deleted]
my takeaway is that I'm impressed they're running such a large scale on scala/JVM. I was at a retail startup in the mid 2010s that was doing their own 3PL, $500m / year and the JVM was a constant pain while scaling up.
JVM is one of the most brilliant pieces of engineering in history.
Also, there are more large companies running on way more "shoddy" technologies like Rails, which Twitter also used up to some point.
Also, there are more large companies running on way more "shoddy" technologies like Rails, which Twitter also used up to some point.
> JVM is one of the most brilliant pieces of engineering in history.
Yet, such a pain in the ass…
Yet, such a pain in the ass…
Java might be. JVM? No.
I assure you that some of the highest-throughput consumer-facing systems in the world run on the JVM.
For me, the interesting part was that there were 10+ people the build team and it seems like there are still people left since he didn't say all the people left. How large was the team? I personally find teams reaching 6+ members too large.
and
> For example, I think it was Adam who pointed out that the Mac laptops were unable to fully utilize the remote cache because action cache was platform specific. This remains to be an open challenge of Bazel to date.
It's always macs that are suffering from problems in my experience.
and
> For example, I think it was Adam who pointed out that the Mac laptops were unable to fully utilize the remote cache because action cache was platform specific. This remains to be an open challenge of Bazel to date.
It's always macs that are suffering from problems in my experience.
The article says about 12 people plus a few extras and that 10+ left, so definitely a huge chunk of the team.
Reading that line it's evident that the issue isn't Mac-specific, just that you can't reuse build artifacts across different platforms (which is expected for pretty much any build system).
Reading that line it's evident that the issue isn't Mac-specific, just that you can't reuse build artifacts across different platforms (which is expected for pretty much any build system).
>How large was the team? I personally find teams reaching 6+ members too large.
That does not mean that all of those people had a single manager and were all responsible for the same chunks of code.
Most likely there were 3-4 sub-teams responsible for different parts of the system.
That does not mean that all of those people had a single manager and were all responsible for the same chunks of code.
Most likely there were 3-4 sub-teams responsible for different parts of the system.
Does anyone know overall how migrated to bezel Twitter is?
[deleted]
2000 people on a build team.
...
I think that comment is saying there were 2000 ICs merging code at Twitter, at least according to publicly released figures
thx, I misread obviously
Two things to consider:
1. It is sometimes hard to buy things that works well: https://danluu.com/nothing-works/
2. The productivity gains of something like Bazel and Pants are pretty amazing if you are working within a monorepo.
It is very difficult to understand how big these two things play into these decisions. Having ~20 people manage build tooling, build caching, source control, CI (build+test), and building release artifacts (which it sounds like Pants did) is not a bad deal.
Consider:
1. GitLab will charge you $20/month/user.
2. If you have 2000 SWEs you'll pay 40k/month just to have access to the website.
3. You'll still need to hire a person or two to negotiate the purchasing of the GitLab.
Now remember that GitLab might not be prefect for your use case. GitLab doesn't increase the performance of builds on your laptop by sharing a cache with all engineers. There's also a limit to the mount of data you can store for source + build artifacts (50GB). The CI runners they host for you are very expensive for how slow the machines are ($10/1000 minutes).
Sure, you can do things like run your own GitLab runners but now you're back to running your own infrastructure which is an engineer's job. Sure, you could save money by using Gitea or something that if free but again hosting your own infra. As soon as you have someone managing that infrastructure they're bound to say "how else can I save time and money" and at a certain scale these things become reasonable or even required.