Musk orders Twitter to cut infrastructure costs by $1B – sources(reuters.com)
reuters.com
Musk orders Twitter to cut infrastructure costs by $1B – sources
https://www.reuters.com/technology/musk-orders-twitter-cut-infrastructure-costs-by-1-bln-sources-2022-11-03/
163 comments
I had a friend that worked for a massive company in the Valley which I'll leave unnamed, that told me about them just leaving servers running hot with no throughput for months at a time. So, I guess it's possible. I doubt his estimate was pulled out of a hat (but I could be wrong, or the article could just be pulling that out of a hat/using an unreliable source.)
I've taken over AWS and Azure accounts before that were paying tens of thousands for unattached disks, load balancers, big servers that were doing very little that could have just been on a burstable VM. Massive messes that in some cases took months to clean up. So, I can believe there could be a massive bit of cost savings available if it were poorly managed. I've never seen the scale of a company like Twitter, but I really doubt there isn't some dead weight. I know storage is a massive cost, and he has stated he was going to purge accounts that were unused over a certain time frame.
I've taken over AWS and Azure accounts before that were paying tens of thousands for unattached disks, load balancers, big servers that were doing very little that could have just been on a burstable VM. Massive messes that in some cases took months to clean up. So, I can believe there could be a massive bit of cost savings available if it were poorly managed. I've never seen the scale of a company like Twitter, but I really doubt there isn't some dead weight. I know storage is a massive cost, and he has stated he was going to purge accounts that were unused over a certain time frame.
One of our smallest customers had a VM for each application server process. CPU occupancy alone was rarely over 20%.
Easily cut 70% of cloud costs.
Easily cut 70% of cloud costs.
Yep, same here; as a (well paid) hobby, I help smaller companies lower their software bills. When they use 'cloud hosting' instead of a vps or something, that's usually the first point where I can cut 80%+. It is crazy how many things are running there that are not used or needed in almost every company I see.
Usually there are software issues (most commonly, no, not enough or plainly wrong db indexes being the #1 top scorer) which drive rds bills to scary height for no reason at all, but the low hanging fruit is unused cloud 'stuff'.
Usually there are software issues (most commonly, no, not enough or plainly wrong db indexes being the #1 top scorer) which drive rds bills to scary height for no reason at all, but the low hanging fruit is unused cloud 'stuff'.
It's been a fun 'arms race' of sorts between hard-drive manufacturers and FAANG-esque coders. I have a friend that works for a very big hard-drive company. So, think very large data storage at scale. Well, they discovered that a fair bit of their spin-disk machines would get to the max rotation lifetimes with little reader-head arm movement. AKA, they were just run for months at a time without ever being really used. This was after the inherent 'shut off if nothing has been written to you in XXX sec.' code that runs on every machine.
They had some digging about, and it turned out that most of the FAANG guys would just write a test bit every XXX-1 sec. to keep the disk spinning. As it improved their internal metrics on 'up time' or whatever else they thought was important. So silly.
So, the manufacturer then wrote code to deal with that code to try to save the bearings. And soon enough that code was discovered, and more complicated code was made to keep the disk spinning no matter what.
And back and forth until my buddy retired and stopped caring about this strange little 'arms race'. I suppose it's still going on though.
They had some digging about, and it turned out that most of the FAANG guys would just write a test bit every XXX-1 sec. to keep the disk spinning. As it improved their internal metrics on 'up time' or whatever else they thought was important. So silly.
So, the manufacturer then wrote code to deal with that code to try to save the bearings. And soon enough that code was discovered, and more complicated code was made to keep the disk spinning no matter what.
And back and forth until my buddy retired and stopped caring about this strange little 'arms race'. I suppose it's still going on though.
Same here, 70% cut in costs. If efficiency isn't a goal and there isn't a strong push for controlling costs, they just keep going up.
I don't know if 1B is a reasonable goal, but Twitter's previous leadership was so incompetent that I wouldn't be surprised. Maybe even more is possible.
I don't know if 1B is a reasonable goal, but Twitter's previous leadership was so incompetent that I wouldn't be surprised. Maybe even more is possible.
How do you find clients for this hobby? What sort of message or phrasing do you use?
Could you explain how db indexes are costs having no reason at all?
I assume tluyben2 means that if the db's index is bad the query has to do a much more expensive scan of the database. By adding correct indices this can be fixed.
That is indeed what I meant; with very little effort and not having much knowledge about database internals, you can save a ton of money. It won’t be perfect that way, but goes a long way vs no indices at all. So that’s why ‘no reason’. Aka very low hanging fruit.
Sorry for not being clear and thanks for explaining.
Sorry for not being clear and thanks for explaining.
No to mention adding an index is practically without risk, the outcome of queries is the same except probably faster.
Not true.
An index isn't free so you can impact the write performance, depending on the workload and usecases slower writes could have more of an impact than faster reads.
But even in a read heavy application it's possible that your databases query planner is "tricked" into using a specific index which can actually be slower than another (or even none) so you should always measure the impact that adding an index can have on your application.
An index isn't free so you can impact the write performance, depending on the workload and usecases slower writes could have more of an impact than faster reads.
But even in a read heavy application it's possible that your databases query planner is "tricked" into using a specific index which can actually be slower than another (or even none) so you should always measure the impact that adding an index can have on your application.
Yes it could be slower, but it’s very unlikely to actually break the application.
Not true, as always it depends.
Tell me you haven't used spanner without telling me...
Everyone is “overspending” on infrastructure. There is always money to save the question comes down to where you want to expend your engineering effort and what risk levels you want to take.
“Make fewer features, focus on spending less” is a valid executive decision, it’s just a matter of priority. Twitter seemingly being feature complete and losing lots of money most years makes doing this now a decent decision.
Spending too much like that is generally a symptom of not paying attention and noticing the waste, it happens from time to time. Gets to feel normal, then you notice and correct.
I’ve been mostly responsible for cloud spend a number of times and had several events of cutting spending by 20% or more over the years.
“Make fewer features, focus on spending less” is a valid executive decision, it’s just a matter of priority. Twitter seemingly being feature complete and losing lots of money most years makes doing this now a decent decision.
Spending too much like that is generally a symptom of not paying attention and noticing the waste, it happens from time to time. Gets to feel normal, then you notice and correct.
I’ve been mostly responsible for cloud spend a number of times and had several events of cutting spending by 20% or more over the years.
I've had the same experience at startups. Usually it's time to make features, but eventually the spend gets high enough to justify spending a couple weeks of engineer time on it to cut out waste. Same thing happened when I worked a large internet company, just on a different scale.
AFAIK Twitter's serving infra is mostly on-prem and they use cloud mostly for analytics.
https://blog.twitter.com/engineering/en_us/topics/infrastruc... (2017 tho)
https://blog.twitter.com/engineering/en_us/topics/infrastruc... (2017 tho)
I don’t think the 1B/year shows issue on cloud pricing. It’s instead bloat of twitter’s setup. That’s way in excess of what should reasonably be spent. I suspect there’s a lot of bad coding practice going on. Of which the bulk is data analytics.
AWS profit margin is something like 35%. I think some of their infra is there, so that might be an upper bound?
Do they use AWS though?
https://blog.twitter.com/engineering/en_us/topics/infrastruc...
https://blog.twitter.com/engineering/en_us/topics/infrastruc...
if Musk cuts $1B from the run rate and there isn't a bad outage, a lot of people will look pretty stupid.
A then-current Twitter employee told one of the New York Times podcasts that Twitter's infra looks overbuilt because the teams focus primarily on reliability, with new features presumably coming second.
Feels like Twitter is now joining the "move fast and break things" club.
A then-current Twitter employee told one of the New York Times podcasts that Twitter's infra looks overbuilt because the teams focus primarily on reliability, with new features presumably coming second.
Feels like Twitter is now joining the "move fast and break things" club.
They barely run anything in the cloud, they’ve been operating their own data centers & infrastructure for over a decade. So they’re doing the same thing as Meta.
> Frankly, if Musk cuts $1B from the run rate and there isn't a bad outage, a lot of people will look pretty stupid.
Is this a good way of reasoning?
If somebody said "frankly, if Bob stops wearing his seatbelt and doesn't die, a lot of people will look pretty stupid" we'd push back. Now the opposite is also not true, where if things go fine Musk is still an idiot. Only the people who know the internals of the system can really say with confidence. One thing I am sure of is that Musk is not one of those people.
Is this a good way of reasoning?
If somebody said "frankly, if Bob stops wearing his seatbelt and doesn't die, a lot of people will look pretty stupid" we'd push back. Now the opposite is also not true, where if things go fine Musk is still an idiot. Only the people who know the internals of the system can really say with confidence. One thing I am sure of is that Musk is not one of those people.
Except Twitter going down is not as serious as death. The answer to “How much $ is it worth to get an extra 9 of reliability?” is never “Infinite”. So, I agree with OP, if they can save a big chunk of their opex with a negligible decrease in reliability (measured by something like, how much user churn are we seeing as a result of the downtime), then it’s a good call.
I don’t know enough about Twitter’s internals to guess whether this is a good call. A bit off-topic, but having worked at Meta I think it would probably not be a good call there. Infra engineers are already directly incentivized to find fractional-percentage wins, because the savings to the company are often greater than their total comp, and that’s a pretty straightforward way to justify a high performance rating.
I don’t know enough about Twitter’s internals to guess whether this is a good call. A bit off-topic, but having worked at Meta I think it would probably not be a good call there. Infra engineers are already directly incentivized to find fractional-percentage wins, because the savings to the company are often greater than their total comp, and that’s a pretty straightforward way to justify a high performance rating.
For Twitter, outages are serious business and definitely a threat to the organism.
As users on the outside maybe we don't care so much, but internally Twitter has invested tremendous amounts of energy and brain power to make the service resilient and scalable over the past 12+ years.
When Twitter is down, it was taken quite seriously by those who worked there. I'm sure they did the blameless postmortem culture, the whole none yards.
Too much downtime and people will stop using it and / or migrate to a competitor, and then the service will be dead.
As users on the outside maybe we don't care so much, but internally Twitter has invested tremendous amounts of energy and brain power to make the service resilient and scalable over the past 12+ years.
When Twitter is down, it was taken quite seriously by those who worked there. I'm sure they did the blameless postmortem culture, the whole none yards.
Too much downtime and people will stop using it and / or migrate to a competitor, and then the service will be dead.
The analogy you're using is binary (seat belt on or off), whereas an infrastructure cut is more like reducing your thermostat by a couple of degrees, then concluding that no one froze to death, therefore it was an acceptable cut. Many parts of Europe will be running this experiment, in fact.
more like, cutting off heating to the building, telling everyone to wear sweaters, and just wait and see if anyone has a serious medical emergency.
I am quite puzzled, at such scale with low seasonality what are the reasons they haven't moved to their own infra? Managing and hiring people to do so would result in big decrease in cost compared to the cloud.
Twitter is mostly on their own infra. 1 billion is 25% of total revenue. I think they are close to that spend with cloud but it’s more like 750m if I were to venture a guess.
> This is sort of where Facebook/Meta got it right, IMHO - run infrastructure on-prem, at scale, in a super-efficient "everything fails" sort of way.
It's fairly ignorant to make this claim without realizing that Twitter also runs the majority of their stuff on-prem as well.
It's fairly ignorant to make this claim without realizing that Twitter also runs the majority of their stuff on-prem as well.
$1bn, the exact amount of debt interest Twitter now has to pay off after Musk saddled it with so much debt.
So much of this feels like such a waste. I don’t have any problem with the idea of someone taking Twitter private in order to fix it. But with this kind of debt financing and with a CEO that doesn’t seem to understand the company (and tried to get out of buying it)…
So much of this feels like such a waste. I don’t have any problem with the idea of someone taking Twitter private in order to fix it. But with this kind of debt financing and with a CEO that doesn’t seem to understand the company (and tried to get out of buying it)…
Ignoring all the other things that are wrong with this whole situation... if they can make it $1B more efficient, how is it a waste even given the loans? Presumably the plan is to get rid of the debt eventually, and then you'll have a much more financially healthy company, assuming it can actually get $1B more efficient.
If they can, that's great, if not then they're setting up Twitter for bankruptcy, or asset sales, if they have any.
Generally speaking, loading up a company with debt is corrosive to good management. If you have to cover your interest payments each month, and that's hard, it focuses the business on short term survival rather than investing in growth in the longer term.
Generally speaking, loading up a company with debt is corrosive to good management. If you have to cover your interest payments each month, and that's hard, it focuses the business on short term survival rather than investing in growth in the longer term.
Definitely, the debt is a burden. I don't see how it's a waste to save money though. Maybe the cuts are so dramatic it damages the business, but reducing wasted money is generally good management.
It's like surgery: making cuts is part of the job, but the more drastic and indiscriminate the more likely the patient dies.
Given that Musk has just arrived, how can these cuts be carefully considered? He burst through the door and started slashing from what anyone can tell.
Even if Twitter is heavily bloated, the cuts may be fatal anyway on that basis.
Given that Musk has just arrived, how can these cuts be carefully considered? He burst through the door and started slashing from what anyone can tell.
Even if Twitter is heavily bloated, the cuts may be fatal anyway on that basis.
Cutting costs is good, quickly slashing 50% doesn’t seem carefully considered and may cause actual problems.
Seems like a pretty massive “if” there. An extra one billion in efficiency without detriment to the service?
You've captured my sentiments exactly...
Why a waste? Sounds like a smart move?
Buy a company, drastically reduce unnecessary expenses.
Buy a company, drastically reduce unnecessary expenses.
>Buy a company, drastically reduce unnecessary expenses.
This famously works out very well for PE firms.
It seems very naive to assume Twitter was casually spending 1B/yr in "unnecessary expenses". Maybe the previous CEO/CTO was a baffling moron, but I tend to have more faith in people absent of evidence otherwise. I'm going to be surprised if Twitter is around in year - I'm just they have the capital to keep the lights on but it will be a very different story if advertisers start to abandon the platform because the adops infrastructure was gutted as an "unnecessary expense".
This famously works out very well for PE firms.
It seems very naive to assume Twitter was casually spending 1B/yr in "unnecessary expenses". Maybe the previous CEO/CTO was a baffling moron, but I tend to have more faith in people absent of evidence otherwise. I'm going to be surprised if Twitter is around in year - I'm just they have the capital to keep the lights on but it will be a very different story if advertisers start to abandon the platform because the adops infrastructure was gutted as an "unnecessary expense".
Twitter has expenses of $5B per year.
https://finance.yahoo.com/quote/TWTR/financials/
So just view this as "A 20% reduction in expenses". Doesn't seem quite to scary huh?
https://finance.yahoo.com/quote/TWTR/financials/
So just view this as "A 20% reduction in expenses". Doesn't seem quite to scary huh?
Weren’t they spending almost half a billion a year on r&d?
That usually just means regular engineering staff
All those unnecessary expenses like content moderation and robust fault tolerant infrastructure.
It's an app for billionaires and journos to crank out viral memes, not a pacemaker. The world will be ok.
Not really the point here. If you spent tens of billions purchasing Twitter you presumably don’t want the world to be without it. You have a very active interest in making it a success.
From the buyer’s perspective the only result that matters is the profit.
Unless you're so beyond caring and vindictive that you're willing to do that just to screw over the shareholders who forced you to buy the company.
Screwing them over how, exactly? As far as I can tell most of the shareholders got cashed out at a valuation way above market. I'd love to be similarly "screwed over" given the opportunity.
Then that’s a sign Musk has absolutely lost it. Writing off tens of billions for petty revenge is insane.
(also the shareholders just cashed out, they don’t care)
(also the shareholders just cashed out, they don’t care)
Twitter going down occasionally wouldn’t be the worst thing in the world.
Or gigantic offices in expensive down town areas, all the coworkering space rent, food served at said office buildings, and personnel.
This was my thought. Or at least move HQ somewhere else. Although he’ll probably end up doing that anyway.
And team that communicates with advertisers for advertisers.
[deleted]
would have been smart if you bought it for a good deal.
Wait, what debt has Musk made Twitter take on? (being at the helm mere days, I really doubt there's any, debt like that takes longer to raise)
Musk himself took on ~$12 billion to finance the deal, that debt isn't on Twitter's books.
Musk himself took on ~$12 billion to finance the deal, that debt isn't on Twitter's books.
[deleted]
It was a leveraged buyout
Ok, so what if it was a leveraged buyout? All buyouts are. How did Twitter get "saddled" with debt?
No, all-cash buyouts are also a thing.
This was an all-cash buyout. It does not mean it was not leveraged.
All-cash buyouts are those buyouts where the acquirer pays with cash and not their own stock. In this case, Musk used only cash.
But some of that cash (about $12.7 BN) was borrowed from banks. So, it was leveraged.
In principle some buyer like Musk could buy without any help from banks, but why would they do that? You don't get to be a multi-billionaire without getting familiar with financing.
In any case, leveraged or not, Musk did not saddle Twitter with any debt. Maybe he will do it in the future, but for the time being he didn't, because he did not have time to do it.
He could try to issue a huge amount of debt later, and pay himself an extremely fat dividend, so he can turn a quick profit. But in the current market environment, that does not seem even remotely likely. Who is going to buy newly issued Twitter bonds? At what yields? A one year US Treasury yields 4.8%, what type of yield would one require for a bond issued by Twitter? 10%? 15%? If investors get a hint that Musk is just trying to run Twitter into the ground and get himself a nice exit, they won't invest even at 30%. After the whole saga with Musk repeatedly changing his mind on the deal (and before that the $420-for-sure Tesla buyout), how many investors do you think would fall for a get-rich-quick scheme perpetrated by a guy who can't actually climb the richest man ladder anymore?
All-cash buyouts are those buyouts where the acquirer pays with cash and not their own stock. In this case, Musk used only cash.
But some of that cash (about $12.7 BN) was borrowed from banks. So, it was leveraged.
In principle some buyer like Musk could buy without any help from banks, but why would they do that? You don't get to be a multi-billionaire without getting familiar with financing.
In any case, leveraged or not, Musk did not saddle Twitter with any debt. Maybe he will do it in the future, but for the time being he didn't, because he did not have time to do it.
He could try to issue a huge amount of debt later, and pay himself an extremely fat dividend, so he can turn a quick profit. But in the current market environment, that does not seem even remotely likely. Who is going to buy newly issued Twitter bonds? At what yields? A one year US Treasury yields 4.8%, what type of yield would one require for a bond issued by Twitter? 10%? 15%? If investors get a hint that Musk is just trying to run Twitter into the ground and get himself a nice exit, they won't invest even at 30%. After the whole saga with Musk repeatedly changing his mind on the deal (and before that the $420-for-sure Tesla buyout), how many investors do you think would fall for a get-rich-quick scheme perpetrated by a guy who can't actually climb the richest man ladder anymore?
You do not seem to know what you're talking about. The 12.7bn that was borrowed is going on Twitter's books. Their debt load prior to this leveraged buyout was 5.5bn; it's now 18.3bn[0], with yearly interest payments of roughly 1bn.
[0]: https://www.barrons.com/articles/tesla-stock-twitter-debt-51...
[0]: https://www.barrons.com/articles/tesla-stock-twitter-debt-51...
>yearly interest payments of roughly 1bn
This is on top of the principal? They pay 1bn every year and none of it counted against their debt?
This is on top of the principal? They pay 1bn every year and none of it counted against their debt?
That's about right.
1bn/18.3bn = 5.4%
"Risk free" treasury bond yields are ~4.x%
1% premium for the risk seems reasonable given that Twitter is not known for making profits. Presumably some of the debt were incurred before the rate hikes so it's kind of on the low side.
1bn/18.3bn = 5.4%
"Risk free" treasury bond yields are ~4.x%
1% premium for the risk seems reasonable given that Twitter is not known for making profits. Presumably some of the debt were incurred before the rate hikes so it's kind of on the low side.
Yes, that's just for the interest.
You may need to change your username. The debt is assumed by Twitter and it is liable for $1bn a year in interest payments:
https://www.nytimes.com/2022/10/30/technology/elon-musk-twit...
https://www.nytimes.com/2022/10/30/technology/elon-musk-twit...
Haha, I'll give that serious consideration.
Now, I read that NY Times article, and I guess the crucial sentence that you are referring to is
How could Musk load with debt a company that he did not own yet? Legally, how does this work? In the end someone has to sign a piece of paper. Can I sign a piece of paper making you owe some money to someone else?
I can see how this could work. The then shareholders decided (indirectly via the board of directors) to borrow the cool $12.7 BN so Musk can use his own money and that debt to buy them all out at twice the fair market price. But then was it Musk who loaded Twitter with debt, or the outgoing directors?
Now, I read that NY Times article, and I guess the crucial sentence that you are referring to is
To do the deal, Mr. Musk, the world’s richest man, loaded about $13 billion in debt on the company, which had not turned a profit for eight of the past 10 years.
How could Musk load with debt a company that he did not own yet? Legally, how does this work? In the end someone has to sign a piece of paper. Can I sign a piece of paper making you owe some money to someone else?
I can see how this could work. The then shareholders decided (indirectly via the board of directors) to borrow the cool $12.7 BN so Musk can use his own money and that debt to buy them all out at twice the fair market price. But then was it Musk who loaded Twitter with debt, or the outgoing directors?
You go to a bank and tell it "I would like to buy this company. It has assets worth $N, which could be put up as collateral for a loan of $(.60 x N)." The bank says "Yes, those assets are worth that much. We will sign a contract with you where we will give you $(.60 x N) and in return, once you successfully purchase the company and it is your property, those assets will act as collateral for that loan."
You take the loan (and probably some other money, since by definition the loan is less than the total value of the company's assets), and you go buy the company. You now own the company, therefore you own the assets, therefore you can use them as collateral, and so: the debt that was yours is now on the company's books. You bought out the company with extra leverage provided by the assets you were acquiring: leveraged buyout.
It tends to fail; LBOs tend to target companies that are struggling somehow, and loading an already-struggling company with a shit load of debt frequently results in a bankruptcy. You're probably familiar with companies that dies this way. (My favorite example for fellow millennials is Toys'R'Us.) If you find this troubling: you are correct. But you have forgotten the really important thing: you (or, in most LBOs, a bunch of private equity ghouls) have gotten very wealthy killing a company, and isn't that reward enough?
You take the loan (and probably some other money, since by definition the loan is less than the total value of the company's assets), and you go buy the company. You now own the company, therefore you own the assets, therefore you can use them as collateral, and so: the debt that was yours is now on the company's books. You bought out the company with extra leverage provided by the assets you were acquiring: leveraged buyout.
It tends to fail; LBOs tend to target companies that are struggling somehow, and loading an already-struggling company with a shit load of debt frequently results in a bankruptcy. You're probably familiar with companies that dies this way. (My favorite example for fellow millennials is Toys'R'Us.) If you find this troubling: you are correct. But you have forgotten the really important thing: you (or, in most LBOs, a bunch of private equity ghouls) have gotten very wealthy killing a company, and isn't that reward enough?
So let me recap: you (Musk) get a loan ($12.7 BN) from banks, with the idea that the company you buy will be used as a collateral. Before the sale is closed, you can't put it on the company's balance sheet, because you don't own it. But after that you can.
Wait a second. It's true that Musk now owns the company, but transactions between the company and its (sole) owner are still at arm's length, arent's they?
As a pure coincidence, today I received in the mail the bankruptcy ruling for a preschool that went bust and where we have paid $3000 in advance to enroll our daughter. The judge ruled we'll get $1400. But during the lawsuit it transpired that the owner gave himself a nice loan of more than $100k. He had to give the money back (I don't know all the details, although I can find out; I think he settled for a somewhat smaller amount).
Maybe I'm naive, but I think that should happen with larger companies too. Just because you are the sole owner of a company does not mean the company can just lend you money on whatever terms you decide. It's the ultimate conflict of interests.
On the other hand, if the prior shareholders cooperated with Musk, I can see how the debt could be on Twitter's books. But then, it's them who saddled the company with debt, isn't it?
Wait a second. It's true that Musk now owns the company, but transactions between the company and its (sole) owner are still at arm's length, arent's they?
As a pure coincidence, today I received in the mail the bankruptcy ruling for a preschool that went bust and where we have paid $3000 in advance to enroll our daughter. The judge ruled we'll get $1400. But during the lawsuit it transpired that the owner gave himself a nice loan of more than $100k. He had to give the money back (I don't know all the details, although I can find out; I think he settled for a somewhat smaller amount).
Maybe I'm naive, but I think that should happen with larger companies too. Just because you are the sole owner of a company does not mean the company can just lend you money on whatever terms you decide. It's the ultimate conflict of interests.
On the other hand, if the prior shareholders cooperated with Musk, I can see how the debt could be on Twitter's books. But then, it's them who saddled the company with debt, isn't it?
Please read up on the concept of Leveraged Buyouts (LBOs), which were a thing since the 70s and 80s. Private equity LBOs have been a thing precisely because of credit available from banks. Last I checked, PE firms such as KKR are still prosperous, in spite of conducting an LBO of similar scale (RJR Nabisco), after saddling the company with extremely high amounts of debt (for the 80s).
And please don't drive the discussion to tangents and anecdata, even though your analogy is flawed.
And please don't drive the discussion to tangents and anecdata, even though your analogy is flawed.
It could be the case that Musk is an a-hole who happens to be right. Maybe Twitter's OPEX expanded to fill whatever seemed right for the massive ad revenue (or "potential" revenue). They probably did not need an enormous engineering team to keep the site running. But beyond "just running" I am curious how difficult it is for them to combat spam/bots/griefers/hatemongers. Spending money on moderators and some kind of engineering to make the moderation team productive is a good investment.
As much as some folks hate it, I think paying for the service makes a great deal of sense. Just imagine how much better these things might be if they weren't all driven by adtech. But yeah it could also easily become Twitter's death knell. That network effect will work in your favor right up until it starts working in your biggest competitor's favor. Market leaders can't hold on forever.
re: "paying for it" - I think it would be a potentially awesome experiment to provide a service like [Gmail+Twitter+LinkedIn] for something like $5/mo or maybe even $10/yr. And what if you could get some incremental funding from government grants or NGOs & philanthropists who saw this as a public good (the same way some endowments fund public radio, TV etc).
As much as some folks hate it, I think paying for the service makes a great deal of sense. Just imagine how much better these things might be if they weren't all driven by adtech. But yeah it could also easily become Twitter's death knell. That network effect will work in your favor right up until it starts working in your biggest competitor's favor. Market leaders can't hold on forever.
re: "paying for it" - I think it would be a potentially awesome experiment to provide a service like [Gmail+Twitter+LinkedIn] for something like $5/mo or maybe even $10/yr. And what if you could get some incremental funding from government grants or NGOs & philanthropists who saw this as a public good (the same way some endowments fund public radio, TV etc).
I worked at a large valley based company that had their own data centers and badly wanted to decrease infra spend.
It's WAY harder than it sounds.
Remember Twitter is not a profitable company which means the likelihood of a bunch of low-hanging fruit for savings is very low. That's been done already.
There are not likely "oh look we found a whole DC sitting idle, oops" sort of savings. Any vendors they use are ALREADY giving them huge discounts.
I'm not saying Twitter is efficient or that this is physically impossible (tho it might be), I'm saying its not easy and takes skilled people a long time to iterate to lower-cost systems or designs that don't have massive stability or performance tradeoffs.
It's WAY harder than it sounds.
Remember Twitter is not a profitable company which means the likelihood of a bunch of low-hanging fruit for savings is very low. That's been done already.
There are not likely "oh look we found a whole DC sitting idle, oops" sort of savings. Any vendors they use are ALREADY giving them huge discounts.
I'm not saying Twitter is efficient or that this is physically impossible (tho it might be), I'm saying its not easy and takes skilled people a long time to iterate to lower-cost systems or designs that don't have massive stability or performance tradeoffs.
All of what you say is speculation. In reality there might be a huge amount of waste where it's just become an accustomed part of the culture <- also speculation.
> Remember Twitter is not a profitable company which means the likelihood of a bunch of low-hanging fruit for savings is very low.
That's circular. Their revenue is astronomical. They're not profitable because they're spending it all. That doesn't mean there aren't plenty of low hanging opportunities (or that there are).
Hidden recordings of twitter staff had the describing the places as more of a lifestyle business where profit isn't a major goal and many people worked very limited amounts. It could have been useless bluster but if there was any basis to it, it's a good indication that massive savings should be possible.
That's circular. Their revenue is astronomical. They're not profitable because they're spending it all. That doesn't mean there aren't plenty of low hanging opportunities (or that there are).
Hidden recordings of twitter staff had the describing the places as more of a lifestyle business where profit isn't a major goal and many people worked very limited amounts. It could have been useless bluster but if there was any basis to it, it's a good indication that massive savings should be possible.
Why does this article claim to know that Twitter might go down due to cost cutting? Isn't there a possibility that their infrastructure is just hugely wasteful?
Having never worked at giant web app company, if it were run efficiently then what should we expect their server and infra costs to be? Surely that is the type of Fermi calculation someone on this site can make.
Having never worked at giant web app company, if it were run efficiently then what should we expect their server and infra costs to be? Surely that is the type of Fermi calculation someone on this site can make.
Right? The real cost cutting is usually “hey there’s this huge X sitting around that nobody uses 99% of the time” or “This process is 10000x faster after someone did some performance engineering” or “we’re storing this PB dataset a dozen times for no reason”
There are often lots of ways to cut the fat that have nothing to do with reducing (reliability related) redundancy.
There are often lots of ways to cut the fat that have nothing to do with reducing (reliability related) redundancy.
There's also reducing replication from 4x to 3x which can save a lot but also reduce redundancy.
It is possible to simultaneously have unnecessary spending on infrastructure and then cut it in a way that makes whole thing unreliable.
Given large layoffs and general chaos, mistakes and bad cuts are much more likely. Especially when you git deadline and goal saving made by clueless executive.
Given large layoffs and general chaos, mistakes and bad cuts are much more likely. Especially when you git deadline and goal saving made by clueless executive.
> Why does this article claim to know that Twitter might go down due to cost cutting? Isn't there a possibility that their infrastructure is just hugely wasteful?
I see the authors (Sheila Dang, Paresh Dave and Katie Paul) quote "sources".
I see the authors (Sheila Dang, Paresh Dave and Katie Paul) quote "sources".
I guess they should bring back the Twitter whale.
The fail whale was amazing.
It is amazing and an indicator if something went viral. I remember Kendrick Lamar dropped his first LPs and Twitter just couldn’t recover for hours. There were no “trends” list before.
I work and worked for huge companies in the direct CIO team. Many times I thought "that is wasteful, we should do X".
I self-host a lot at home and am a standard nerd and what I learned over 30 years is that my knowledge does not scale at all.
In such companies even if the DC is idle, knowing what happens when you switch it off does not exist. Even worse, when cutting/moving a service, three blocks down something crumbles and their custodian has no idea why (and worse: they would not even know that something was switched off if there is no strict change management).
It is easier if you are a modern company that uses some modern best practices, there is at least hope that the configuration gives you a clue.
I self-host a lot at home and am a standard nerd and what I learned over 30 years is that my knowledge does not scale at all.
In such companies even if the DC is idle, knowing what happens when you switch it off does not exist. Even worse, when cutting/moving a service, three blocks down something crumbles and their custodian has no idea why (and worse: they would not even know that something was switched off if there is no strict change management).
It is easier if you are a modern company that uses some modern best practices, there is at least hope that the configuration gives you a clue.
Are they selling any servers? I need them to setup Mastodon servers.
The risk for Twitter isn’t that it might go down for some periods of time.
The risk is that if it starts going down on a regular basis and/or performance is degraded, a competitor could swoop in and clean their clocks.
This might be a good time for Google to start Google+ again as an independent product one needs to sign in and create a new account for.
The risk is that if it starts going down on a regular basis and/or performance is degraded, a competitor could swoop in and clean their clocks.
This might be a good time for Google to start Google+ again as an independent product one needs to sign in and create a new account for.
Social media has always been a ghetto and it'll be interesting to see how many guys try and fail to build the next twitter in a few years before they realize why it was doomed from the start
Twitter was famously unreliable for years with the "fail whale", and no such competitor swooped in despite years of problems. So this argument doesn't seem to hold water.
The network effect is too strong for some occasional downtime to give even a well-funded competitor a window in.
It seems like a good time for Meta & Google just to make Twitter clones and see if they can get it to stick.
Sure you can find unused cloud environments to spin down and try to switch away from expensive vendor agreements, but any significant costs savings is going to require a lot of optimization and talented employees and I get the impression that the talented employees that wanted to work at Twitter already do.
I'm not convinced this is bad for Twitter - if they reduce their scope with it.
They should open the API back up and let third party developers step in and start doing all the stuff - filtering for example - that they don't do a good job at.
Just be a login and a platform.
They should open the API back up and let third party developers step in and start doing all the stuff - filtering for example - that they don't do a good job at.
Just be a login and a platform.
This is an incredible opportunity for the engineers remain which Musk surely has converted in to saboteurs with his antics. Sabotage disguised as cost savings.
They'd be easily found out by a smart engineer and promptly fired and probably banned by every competitor once word got out.
Incompetence and mistakes are usually hard to prove is malice
You can find a pattern or motive pretty quick if it happens often. I don't think many would be stupid enough to risk their high paying job for that. Any sabotage would probably be from a low-level employee with git access or is more customer facing. An engineer making in the high 6 figures isn't going to risk it all just to "own Elon" or something stupid.
There's proof and then there's the recommendation that you ask for when trying to find your next job.
one does not want to employ either
There is plenty of opportunity to renegotiate contracts, since capacity is exploding with 400Gbit going over a single wavelength. It used to be amazing to hit 1 Tb/s with a piece of core network equipment and now it is routine... whether tjat and many other changes is enough to save 1B per year, I don't know.
What does that have to do with anything? The cost of bandwidth is just one part of infra costs, and it comes from peering agreements. Those are expensive because you're dealing with monopolies and they don't really have any reason to bargain with you.
I suppose if everyone is fleeing the service, less infrastructure is required.
That most likely won't happen though. In a month most people won't be talking about this and Twitter will be running fine.
This one looks meaningfully different. I won't be logging on in a month since I've already deleted my account (had it since 2010).
> This one looks meaningfully different.
You might be right but plenty of people say this every time some major event at Twitter takes place. The vast majority of people don't care that much about layoffs at Twitter and certainly aren't going to migrate to another service over it.
You might be right but plenty of people say this every time some major event at Twitter takes place. The vast majority of people don't care that much about layoffs at Twitter and certainly aren't going to migrate to another service over it.
This is another "I'm moving to Canada if Trump wins", and nobody of note moved to Canada.
Care to revisit this one in a month?
Why would everyone flee? Twitter had a problem with censorship and Musk is going to fix it. Some will come back, some will leave, the tone overall will change a bit, but that's about it.
> Twitter had a problem with censorship and Musk is going to fix it.
A substantial share of the userbase either disagrees with Musk that there was a problem with content moderation or has a view of the nature of the problem that is at odds with Musk’s.
The other thing Musk seems to be surprised to discover is that the while tweaks might be desirable, the basic outline of content moderation and advertisers confidence in it was very much tied into Twitter’s ability to make money, and since he’s saddled it with a very large amount of additional debt service costs, that is something that becomes quite an acute problem if it is lost.
A substantial share of the userbase either disagrees with Musk that there was a problem with content moderation or has a view of the nature of the problem that is at odds with Musk’s.
The other thing Musk seems to be surprised to discover is that the while tweaks might be desirable, the basic outline of content moderation and advertisers confidence in it was very much tied into Twitter’s ability to make money, and since he’s saddled it with a very large amount of additional debt service costs, that is something that becomes quite an acute problem if it is lost.
I don't know if that particular userbase is valuable to Musk, advertisers or the quality of Twitter in general. I've seen numbers that about half of Twitter users are young, infantile in other words, and they tend to drive emotional crusades. On the other hand, this segment of users may be valuable to advertisers. They may be valuable not because they have money to buy things, but because it's easier to impress them with false values, so when they grow up, they'll start buying stuff from those advertisers. In any case, a good shakeup will only help Twitter.
> I don’t know if that particular userbase is valuable to Musk, advertisers or the quality of Twitter in general. I’ve seen numbers that about half of Twitter users are young
Young adults are the prime advertising demo, and have been for quite a long time.
Young adults are the prime advertising demo, and have been for quite a long time.
hm how much does a 9 cost?
Surprisingly little, 89.999% maintains four :D
the fail whale was a nice "maybe you should go do something else" reminder
Musk failed to deliver the $35K Model 3 and Starlink prices are rising. I doubt there is a billion dollars left lying on the table, at least not accessible to a man of Musk’s nature.
Right now Tesla has waiting lists so long they've head to shut off ordering of new vehicles. They hardly need a vehicle that's any cheaper at this point.
Additionally, Starlink has so many people wanting to use the service that the entire eastern half of the US is basically at capacity because demand is too high.
Increasing prices for both products seems like a very good idea.
Additionally, Starlink has so many people wanting to use the service that the entire eastern half of the US is basically at capacity because demand is too high.
Increasing prices for both products seems like a very good idea.
I dislike Elon Musk as much as the next guy but why are people upset about Twitter burning to the ground? It’s just another paint brush used to paint the downfall of our society. Good riddance. I hope Elon goes under with them.
This is a platform built on driving clicks and driving clicks on fear and anger. More fear = more clicks = more ad revenue = negative feedback loop. Just like the New York Times. Just like NPR. Just like Fox News. Just like Reddit. Just another piece of shit we should all enjoy watching burn into the ground.
Spend some time with your family. We will all be incredibly lucky if this ad-tech insanity falls apart.
This is a platform built on driving clicks and driving clicks on fear and anger. More fear = more clicks = more ad revenue = negative feedback loop. Just like the New York Times. Just like NPR. Just like Fox News. Just like Reddit. Just another piece of shit we should all enjoy watching burn into the ground.
Spend some time with your family. We will all be incredibly lucky if this ad-tech insanity falls apart.
I don't think ad-tech is going away anytime soon. It would be nice but it's not gonna happen because the fundamental problem is not the ads but the fact that most social networks are not sustainable without advertising so the platform participants end up being treated like cash cows that are milked for all they're worth.
If Twitter was paid for by its participants then they would not have to put up with advertisers. Right now the advertisers subsidize the operational costs so they get to dictate what features are developed and what kind of invasive access they have to behavioral data for optimizing their marketing campaigns.
If Twitter was paid for by its participants then they would not have to put up with advertisers. Right now the advertisers subsidize the operational costs so they get to dictate what features are developed and what kind of invasive access they have to behavioral data for optimizing their marketing campaigns.
And the worst part. I think there are a lot of us who would be willing to pay for being on a good social network without ads, but I don't think there is enough of us. Too many people have been trained via myspace, facebook and twitter that social media is just "free." It it is going to be hard to change the masses on that.
As long as somebody has friends who can't afford say $10 per month for social media, the network effect takes over and the result will inevitably be ad-ridden crap.
I might be willing to pay to turn off ads and tracking on such a platform, but if all the middle-class+ people pay to turn off ads, which advertisers would want to spend money targeting "poor" people and people too cheap to pay a couple dollars for using social media?
I might be willing to pay to turn off ads and tracking on such a platform, but if all the middle-class+ people pay to turn off ads, which advertisers would want to spend money targeting "poor" people and people too cheap to pay a couple dollars for using social media?
Unfortunately I agree with you. Nobody is going to pay for a screaming match, though. The fact that it was free is the only reason we’re here.
Maybe this will shake out just fine anyway.
Maybe this will shake out just fine anyway.
It doesn’t matter if Twitter burns to the ground. Some other shitty website will take its place.
Instead of raging against shitty media, we should concentrate on creating better media. It is exceptionally hard though. There are independent (relatively) media like TYT. These have their own problems, but at least they’re better than shit like Fox
Instead of raging against shitty media, we should concentrate on creating better media. It is exceptionally hard though. There are independent (relatively) media like TYT. These have their own problems, but at least they’re better than shit like Fox
Ok but I’m not raging, I just don’t know why I should care about any of this in spite of it being shoved into my face 24/7 for weeks at a time???
Everyone on the internet knows how to run Twitter better than the richest man on earth who revolutionized both the electric car and private space industries. I wish I could invest in Twitter now but I’ll wait for the cryptocurrency launch.
Your sarcastic implication being that being good at managing car and space companies make him good at managing a social media company? Not only that, good enough to be better at it than his predecessor? Not only that, he's better at it within days of arriving?
Occam's razor says he's a big impulsive dummy.
Occam's razor says he's a big impulsive dummy.
Optimizing a factory is indeed quite a lot like optimizing cloud infrastructure. Different domains but there is a lot of overlapping skill set and having developed that from zero scaling Tesla is exactly the experience that would make an executive good at reducing costs and bringing Twitter to profitability.
Building cars is the place where modern efficiency practices were developed. (Much of it started in post war Japan with American auto industry help, the fusion of cultures developed something quite useful)
Building cars is the place where modern efficiency practices were developed. (Much of it started in post war Japan with American auto industry help, the fusion of cultures developed something quite useful)
To be fair, Jack Dorsey rolled over his ownership into the new structure. There are few people who know twitter, it’s problems, and potential than him.
Oh wow running a social media company is just soooo hard how can he possibly figure it out? It must require some super human level of insight /s
By that logic Musk is an incredible surgeon just because he is rich.
Is there some kind of scam coin associated with the twitter deal?
Doge?
Is there any connection beyond Musk hyped it a year or two ago?
he's been hyping it non stop, including taking doge for payments with the perfume he launched last month https://twitter.com/search?q=from%3Aelonmusk%20doge&src=type...
People are treating Musk pronouncements like they did Trumps. Taking extremely literally even when it's obvious he's pulling your leg.
I am assuming twitcoin is coming and I will buy in
Elon used to be a software engineer so it's kinda weird how he thinks a legacy codebase like Twitter can be fixed up to use $1B less every year. Twitter is limited mainly by storage, their compute costs are negligible in comparison. So unless there is so much bloat in their hardware procurement process that they're over-provisioning storage by more than $1B a year there is no way to reduce costs. The hardware costs what it does and there is no way to rewrite the software at this point to reduce hardware costs and still maintain backward compatibility with the existing Twitter functionality (no matter how minimal the feature set seems to be for external observers).
So either this article is a lie or Elon is just making nonsensical statement to focus attention on what it costs to run Twitter at such a scale. If people don't pay then the functionality will continue to degrade and it's looking like there aren't enough willing buyers for $8/month. The only way Twitter can continue to operate is if the people that actively use the service start paying for it, every other financing option is now closed off and if Elon can't turn it around then I doubt anyone else can.
So either this article is a lie or Elon is just making nonsensical statement to focus attention on what it costs to run Twitter at such a scale. If people don't pay then the functionality will continue to degrade and it's looking like there aren't enough willing buyers for $8/month. The only way Twitter can continue to operate is if the people that actively use the service start paying for it, every other financing option is now closed off and if Elon can't turn it around then I doubt anyone else can.
I struggle to find one correct remark in your comment. This take clearly suggests you have never worked on any similar infrastructure / distributed systems before at scale.
So you're telling me there's no backend or compute workers? And you're telling me that there are no opportunities for performance optimization? That they are already 100% native C++ services at full parallel utilization? That they've already tuned the size and cost of their ML models? That their model evaluation infra is fully saturated?
Storage is cheap. I believe they are on AWS, so hardware costs are already taken out of the equation. If so, even just moving to on-prem would save them the $1B easily.
So you're telling me there's no backend or compute workers? And you're telling me that there are no opportunities for performance optimization? That they are already 100% native C++ services at full parallel utilization? That they've already tuned the size and cost of their ML models? That their model evaluation infra is fully saturated?
Storage is cheap. I believe they are on AWS, so hardware costs are already taken out of the equation. If so, even just moving to on-prem would save them the $1B easily.
> Storage is cheap
Seriously. They could back up all of Twitter on Backblaze for $7/month.
Seriously. They could back up all of Twitter on Backblaze for $7/month.
Hah, I chuckled bigly, nice one.
You should sit down and perform the storage calculation instead of struggling to find something correct in what I said because that will be a better use of your limited time.
> I believe they are on AWS, so hardware costs are already taken out of the equation. If so, even just moving to on-prem would save them the $1B easily.
They already run their own datacenters (not uncommon for companies started in that era). They have moved partially into the cloud for data science stuff - GCP, not AWS, according to their engineering blog.
So… odds are that there isn’t nearly as much waste as everyone wants to believe.
They already run their own datacenters (not uncommon for companies started in that era). They have moved partially into the cloud for data science stuff - GCP, not AWS, according to their engineering blog.
So… odds are that there isn’t nearly as much waste as everyone wants to believe.
DC is far more cost effective than public cloud for data intensive workloads.
Right, which is why they only executed a partial migration for certain workloads that make sense: https://blog.twitter.com/engineering/en_us/topics/infrastruc...
I think what really irks me is this meme that “obviously they have massive infrastructure waste.”
Does anyone have credible evidence that there is enormous waste in Twitter’s infrastructure? Just because Musk says “cut $1b in infra spending” doesn’t mean that there is $1b of things to cut, just lying about being wasted. Some of the smartest minds in our industry work/worked at Twitter. Are they that bad at their jobs?
What is actually going to happen is this: they’ll turn off all of their data warehousing stuff, blinding the business. They’ll cut their redundancies and backups, reducing the probability that they can do effective DR. They’ll reduce spending on observability so far that they won’t even know what’s going wrong (it’s surprisingly expensive). And that’ll get them to the $1b in cuts - by flying blind (both in a business sense, and a technical sense), and just hoping they don’t need to recover from a disaster.
I think what really irks me is this meme that “obviously they have massive infrastructure waste.”
Does anyone have credible evidence that there is enormous waste in Twitter’s infrastructure? Just because Musk says “cut $1b in infra spending” doesn’t mean that there is $1b of things to cut, just lying about being wasted. Some of the smartest minds in our industry work/worked at Twitter. Are they that bad at their jobs?
What is actually going to happen is this: they’ll turn off all of their data warehousing stuff, blinding the business. They’ll cut their redundancies and backups, reducing the probability that they can do effective DR. They’ll reduce spending on observability so far that they won’t even know what’s going wrong (it’s surprisingly expensive). And that’ll get them to the $1b in cuts - by flying blind (both in a business sense, and a technical sense), and just hoping they don’t need to recover from a disaster.
> If so, even just moving to on-prem would save them the $1B easily
Except you can't wave a magic wand and shift the entire distributed infrastructure for one of the five most visited sites on earth on-prem. Twitter has two DCs. Running their infra on-prem somewhere would require reworking the entire company from the ground up in a way that would take years and cost substantially more than $1b.
Except you can't wave a magic wand and shift the entire distributed infrastructure for one of the five most visited sites on earth on-prem. Twitter has two DCs. Running their infra on-prem somewhere would require reworking the entire company from the ground up in a way that would take years and cost substantially more than $1b.
Having their own DCs means they are on-prem already.
[deleted]
If they laid off 3700 people, assuming 200,000 dollars each, that's 75% of the way there.
edit: oh, I suppose you wouldn't consider personnel "infrastructure". I'll keep this up despite how off topic I am, it's interesting to compare the reduced cost in labor to the stated goal in reducing infrastructure costs. A billion here, a billion there... Soon we're talking about real money.
edit: oh, I suppose you wouldn't consider personnel "infrastructure". I'll keep this up despite how off topic I am, it's interesting to compare the reduced cost in labor to the stated goal in reducing infrastructure costs. A billion here, a billion there... Soon we're talking about real money.
I'm sure he can keep firing people and just keep the platform on life support but that's the same as admitting Twitter is going the way of Digg and all other social networks that tried to execute a pivot. The people that think he'll be able to turn it around don't really understand the scale of the problem.
There’s always a way to reduce costs. For storage specifically one could spend engineering time compressing data, storing it in a colder way, etc. The question is whether the trade offs, short term or long term, are worth it.
I'm pretty certain that Twitter's code is essentially append-only. It's possible to add new features and keep the old stuff running but there is no way to re-architect it to reduce costs. They are already compressing the data so there is no improvement to be gained there. As for cold storage, re-engineering backups and other forms of redundancy will reduce long-term costs but it won't reduce their yearly operating requirements for storage and compute (storage fundamentally being the main limiting factor).
So either enough people pay to subsidize Twitter's operating costs for everyone else or it continues to get worse over time. There really isn't a 3rd option with better engineered software because there are no software engineers that can work on software of Twitter's scale and make it meaningfully more efficient in terms of storage and compute requirements.
Presumably we are on the brink of AGI, so maybe Elon knows something everyone else doesn't and he's just gonna use AGI to rewrite all of Twitter in assembly. /s
So either enough people pay to subsidize Twitter's operating costs for everyone else or it continues to get worse over time. There really isn't a 3rd option with better engineered software because there are no software engineers that can work on software of Twitter's scale and make it meaningfully more efficient in terms of storage and compute requirements.
Presumably we are on the brink of AGI, so maybe Elon knows something everyone else doesn't and he's just gonna use AGI to rewrite all of Twitter in assembly. /s
[deleted]
To be fair, I don't know if Twitter is overspending on infrastructure, but I would guess there are always places to streamline. You could frankly save a ton of money by re-negotiating commits with Google.
Also, $1B to me shows how insane running a large business in the cloud can be. This is sort of where Facebook/Meta got it right, IMHO - run infrastructure on-prem, at scale, in a super-efficient "everything fails" sort of way.