John Carmack on lead time to push fixes in startups(twitter.com)
twitter.com
John Carmack on lead time to push fixes in startups
https://twitter.com/id_aa_carmack/status/1626389610635923457
54 comments
But he was doing that daily at Meta, and it still took 6 months for things to happen!
Something something ancient Chinese parable the only person not impressed by the General/Emperor/Mystic Sage is his wife.
Do we know that the things he wanted to fix were important enough to fix?
Yes we know that.
Yeah they were. We are writing a blog about it so everyone knows.
[deleted]
Yeah but at a big corp it is more than just you. Even if you fixed it are you authorized to get a build out, out of cycle? If you work on an App, can you get it through the app store approval process?
But his entire point is that's not what happened at the big company he worked at.
I bet more than half the kids working there weren’t even born when Quake came out
This is actually pretty funny :) I can verify if the percentage is really more than 50.
You are right! Respond and listen - absolutely! It will be just silly not to do that.
However, being able to implement it, having the infra to facilitate, and the internal quality of the deliverable not to incur more tech debt by being quick … this is just not as simple as size of company and work on plate.
Yes, I am not humble but we work hard :)
However, being able to implement it, having the infra to facilitate, and the internal quality of the deliverable not to incur more tech debt by being quick … this is just not as simple as size of company and work on plate.
Yes, I am not humble but we work hard :)
We took Action right away. I think many companies wouldn’t have aces as fast. Not everyone gets it
I’m sure things used to be this way but “we’re like a startup” is so overused in big tech these days. It’s usually an excuse for something wrong rather than the benefits of a startup.
I wish we were building for customers instead of building for my or someone in my leadership chain’s promotion. I think most companies start with that focus then lose it after a certain size. Most of the engineers working on projects don’t even care about the product.
I wish we were building for customers instead of building for my or someone in my leadership chain’s promotion. I think most companies start with that focus then lose it after a certain size. Most of the engineers working on projects don’t even care about the product.
I worked in a "startup-like" team at a big company a few years ago. It was fun in that we moved quickly since we had fewer processes than typical teams at Big Co. Over time though, I realized we were just making stuff to make some higher-up happy, and not really chasing a customer.
I also found out we were funded just to light a fire under another team that moved very slowly. Whatever we made would be taken back to that team and we'd have to sell them on using it, and also show them what was possible.
Ultimately, we were just a cog in this larger machine. Nowadays I don't really trust anybody at big companies that says "our team is like a startup".
I also found out we were funded just to light a fire under another team that moved very slowly. Whatever we made would be taken back to that team and we'd have to sell them on using it, and also show them what was possible.
Ultimately, we were just a cog in this larger machine. Nowadays I don't really trust anybody at big companies that says "our team is like a startup".
> “we’re like a startup” is so overused in big tech these days. It’s usually an excuse for something wrong rather than the benefits of a startup.
I've only ever heard that phrase or something approximating it used to excuse bad practices.
I've only ever heard that phrase or something approximating it used to excuse bad practices.
My “we’re like a startup” experience was all the uncertainty of a startup along with all the bureaucracy of big tech which resulted in pivoting to a new 12 month roadmap every 6 months…
Yes. "We're like a startup" is a bit of a tipoff that a) the company is no longer like a startup and b) the company has completely forgotten what it's like to be a startup.
There's a point where you graduate and mature your products, reaching out to adjacent markets. Startups can't and shouldn't do that. If you're a big company acting like a startup, you're losing ground.
I think that’s a big generalization and depends on the company of course. I made a point at SkyFi about treating every customer as if they were Carmack
>The time to deliver a fix to VR users at Meta was often over six months, disregarding all the problems that were just ignored.
I find it astounding that this could be true for someone as senior and respected as Carmack. Is it because he's working in a hardware division? What takes so damn long?
Side note - its interesting how Carmack has come out and publicly bashed Meta so much. I guess you can do that when you're in his position.
I find it astounding that this could be true for someone as senior and respected as Carmack. Is it because he's working in a hardware division? What takes so damn long?
Side note - its interesting how Carmack has come out and publicly bashed Meta so much. I guess you can do that when you're in his position.
Even software can move slowly because everything has to go through numerous levels of bureaucracy and review. Before you even start working on something, you often need to have meetings with stakeholders from various teams and disciplines to "align" (read: convince them to do the work on their part of the stack). Then on top of that, the Quest OS update shipping cadence is monthly.
Source: worked at Meta on Quest system software with Carmack 2018-2021
Source: worked at Meta on Quest system software with Carmack 2018-2021
[deleted]
Maybe the regular, 50 people all bikeshedding every single tiny decision rather than just letting the people who know what they're doing get their job done?
Maybe his ego got hurt at Meta. Why would you publicly bash your previous employer. Isn’t staying quiet the right thing to do?
> Why would you publicly bash your previous employer.
Maybe because they suck and deserve to be criticized?
> Isn’t staying quiet the right thing to do?
It's certainly the easier thing to do. It's a sad world we live in where being 'professional' means being dishonest.
Maybe because they suck and deserve to be criticized?
> Isn’t staying quiet the right thing to do?
It's certainly the easier thing to do. It's a sad world we live in where being 'professional' means being dishonest.
He's a true VR believer. I think it pained him to watch it go less well than it could.
> "Is it because he's working in a hardware division?"
Likely? Certainly from all appearances from the outside there are many legitimate bones to pick about Meta's organizational sclerosis, but part of this is structural to the product itself.
"I can fix the image quality issues with X relatively easily, I know exactly what needs to be done"
elicits (rightfully!)
"We operate on an intensely compute-constrained device where every cycle matters. Over-consumption of compute causes battery life to degrade significantly and the device to become uncomfortably warm to wear. How much compute will this solution cost?"
Then you have to balance the additional compute with drawing down compute somewhere else. Maybe the tracking camera algorithms can be optimized to consume less compute, which opens room for additional usage elsewhere, but now you've introduced a blocking dependency to delivering this enhancement.
Even in a well-functioning organization this prevent shipping as fast as a startup. Part of why startups ship fast isn't just organizational, it's because the products are simpler with fewer technical constraints.
Hardware (particularly embedded hardware - which Meta Quests are not precisely, but are pretty close) presents intense levels of constraints that most people who write code on servers (or even PCs) don't often appreciate.
Likely? Certainly from all appearances from the outside there are many legitimate bones to pick about Meta's organizational sclerosis, but part of this is structural to the product itself.
"I can fix the image quality issues with X relatively easily, I know exactly what needs to be done"
elicits (rightfully!)
"We operate on an intensely compute-constrained device where every cycle matters. Over-consumption of compute causes battery life to degrade significantly and the device to become uncomfortably warm to wear. How much compute will this solution cost?"
Then you have to balance the additional compute with drawing down compute somewhere else. Maybe the tracking camera algorithms can be optimized to consume less compute, which opens room for additional usage elsewhere, but now you've introduced a blocking dependency to delivering this enhancement.
Even in a well-functioning organization this prevent shipping as fast as a startup. Part of why startups ship fast isn't just organizational, it's because the products are simpler with fewer technical constraints.
Hardware (particularly embedded hardware - which Meta Quests are not precisely, but are pretty close) presents intense levels of constraints that most people who write code on servers (or even PCs) don't often appreciate.
Meta Quest 2 has 6GB of DDR4 RAM. It has a 64bit CPU. I'd say it has more in common with a server than with an embedded device.
> 6GB of DDR4 RAM. It has a 64bit CPU
More like a phone from 2020. The Samsung Galaxy S20 base model has 8GB of RAM (the S20FE has 6GB)
More like a phone from 2020. The Samsung Galaxy S20 base model has 8GB of RAM (the S20FE has 6GB)
Maintaining that velocity inside a larger organization is a really interesting challenge.
There are things you can do that can help. The biggest in my experience is robust automated testing combined with robust automated deployments
Continuous deployment (any code that passes the tests gets deployed) is incredible if you can get there, but the next best thing is continuous deployment to a staging environment followed by one-click-deploys by trusted release managers to production.
There are things you can do that can help. The biggest in my experience is robust automated testing combined with robust automated deployments
Continuous deployment (any code that passes the tests gets deployed) is incredible if you can get there, but the next best thing is continuous deployment to a staging environment followed by one-click-deploys by trusted release managers to production.
Isn't this him just bragging, which gets extrapolated to some platitude about startups? By the same token of startups supposedly being able to fix things faster, which is debatable, it is more likely that they have more problems by moving faster without that many users. Things change a bit when you have millions, hundreds of millions, and even billions of users.
Nope. Not debatable. Not supposedly. Effective startups move faster … or they are not effective startups and they die. The fact that not all startups die literally shows that not all are creating more problems than they are solving.
Large orgs can be pretty nimble too if they are properly set up and have the right people btw
Large orgs can be pretty nimble too if they are properly set up and have the right people btw
At my last company we had a Discord that our users would use to give use feedback, bug reports, and ideas. It was amazing. The feedback itself was really helpful in building the product, but you instantly acquire a user's loyalty the first time they report an issue and you fix it that day. Users are really grateful for that.
You can't and really should not expect this kind of speed from a large company. In this case he used a service where you can pay 170$ to get a custom satellite image from space. My guess is they treat every customer individually and the technical issue reported was obvious.
At Meta you have hundreds of millions if not more users and any issue largely isn't so obvious that you can just go and fix five lines of code. When someone comes around and says "hey I'm Bob xyz isn't working" you probably need to figure out first if that's a real issue at all and if fixing it doesn't break ten other things.
The potential downside is much, much larger if you overreact to individual complaints at that scale.
At Meta you have hundreds of millions if not more users and any issue largely isn't so obvious that you can just go and fix five lines of code. When someone comes around and says "hey I'm Bob xyz isn't working" you probably need to figure out first if that's a real issue at all and if fixing it doesn't break ten other things.
The potential downside is much, much larger if you overreact to individual complaints at that scale.
Counterpoint: Carmack worked at Meta on the Oculus, he had as much visibility as it’s possible to have on the problems, their fixes, and whether the fixes would break things. He doesn’t spell it out in the tweet but the kinds of issues he talks about the satellite image having sound like the kind of issues you might see in VR hardware and software too, and the fact that he immediately noticed and correctly guessed what was going wrong in the satellite images suggests he had tackled the exact same problem while working on the Oculus. In fact, I’m pretty sure that’s why he made this tweet: he saw a problem he’d seen before, he suggested the same solution he’d suggested before, and he saw the solution implemented in days instead of months.
I work on a 4 person team. At one point it is was two of us.
I love a quick fix where I tell the customer “try it again” done.
I love a quick fix where I tell the customer “try it again” done.
Honestly I don’t know what to do with this tweet + HN thread because it leaves little room for actual discussions and a lot of room for misguided guessing.
Unless someone from the org comments here, which I doubt considering the animosity HN has at VR or Meta, we’ll never know why oculus didn’t work like that, if it was a real issue there or if Carmack is exaggerating, if it was an issue are tweets like going to help the org rethink itself?
The only signal we’re getting, from the outside, are the releases. I had the go and the quest and the gap in technology between the two were amazing.
I don’t know about quest to quest 2, or quest 2 to quest pro, and I guess we’ll see what the quest 3 delivers. Maybe other people know?
Unless someone from the org comments here, which I doubt considering the animosity HN has at VR or Meta, we’ll never know why oculus didn’t work like that, if it was a real issue there or if Carmack is exaggerating, if it was an issue are tweets like going to help the org rethink itself?
The only signal we’re getting, from the outside, are the releases. I had the go and the quest and the gap in technology between the two were amazing.
I don’t know about quest to quest 2, or quest 2 to quest pro, and I guess we’ll see what the quest 3 delivers. Maybe other people know?
Startup probably worked on Carmack’s request because they were afraid he will post a negative review or give negative review about the startup to his friends. A startup can’t afford to piss off a person like John Carmack, they might go out of business, because it might impact their ability to get new customers and raise capital.
Meta on other hand could care less about John Carmack’s opinion as it has no impact to their business. In short John Carmack’s opinion was ignored it had no impact for a large company.
The mistake John Carmack is making is that he thinks his views are special, they are not.
Meta on other hand could care less about John Carmack’s opinion as it has no impact to their business. In short John Carmack’s opinion was ignored it had no impact for a large company.
The mistake John Carmack is making is that he thinks his views are special, they are not.
I work at a small company and customer issues or requests can be solved (or worked on) within a day because there's almost zero red tape. And it doesn't need to be Carmack pointing out the issue or delivering the request for it to be done.
Well, we had a lot of folks here at HN being mad at us for the shape of the cursor on our corporate site but we are still not out of business :)
Anyway, no, the report did not lead to a scramble. As a matter of fact, it was not even treated as critical because it was not “business down” and no event different than business down is worth introducing (even potentially) more problems that it is solving.
Anyway, no, the report did not lead to a scramble. As a matter of fact, it was not even treated as critical because it was not “business down” and no event different than business down is worth introducing (even potentially) more problems that it is solving.
No, we weren’t afraid of a negative comment at SkyFi…we are focused on customer feedback and making things better. We treat everyone as if they were Carmack. If you read the whole thread he was still critical of our product and it wasn’t all roses and sunshine.
I don't know that's true, and he may not have even blathered his name out there.
I know that I got some really responsive work from startups we purchased from, and it was very nice, and we were just one of their customers.
Sadly, once they pivot away from whatever it was then it all goes away.
I know that I got some really responsive work from startups we purchased from, and it was very nice, and we were just one of their customers.
Sadly, once they pivot away from whatever it was then it all goes away.
[deleted]
I have great respect for John Carmack but this feels like a very unnuanced take.
Lead time isn't about "startups", it's about code complexity, side effects and blast radius.
The time it takes to push code when you have 10 use-cases and 1,000 users, is very different from the time it takes to push code when you have 1,000 use-cases but 10,000,000 users. At that scale, it takes A LOT of effort to keep fix times small. It isn't going to occur naturally.
Some times the effort to keep that fix time small may not be worth it, some times it might.
Lead time isn't about "startups", it's about code complexity, side effects and blast radius.
The time it takes to push code when you have 10 use-cases and 1,000 users, is very different from the time it takes to push code when you have 1,000 use-cases but 10,000,000 users. At that scale, it takes A LOT of effort to keep fix times small. It isn't going to occur naturally.
Some times the effort to keep that fix time small may not be worth it, some times it might.
OR you can work out your architecture well so that you don't have to worry about other use-cases :)
Yeah but like, I get that startups are better at listening to customers and adjusting quickly.
But I can confidently say.. if John Carmack himself legit sent me an email pointing out some problem with something I made. You bet your ass I would respond and listen to what was on his mind. Regardless of the size of my company and the amount of work on my plate :p