“A Cold War Every Day” inside Apple's internal tools group(buzzfeednews.com)
buzzfeednews.com
“A Cold War Every Day” inside Apple's internal tools group
https://www.buzzfeednews.com/article/alexkantrowitz/always-day-one-exclusive-excerpt-apple
182 comments
> More accurately, it's a layer of literal stupidity above the competent to shield the money side of the company from the leverage that operations people would have if they had any information about how the money side worked.
This is a brilliant description of it. The money people aren't stupid, despite what the workers think. They just have more information and different motivations which, in most larger organisations, they actively avoid explaining to the operations people.
Edit: Changed 'an incredible description' to 'a brilliant description' because it is entirely credible.
This is a brilliant description of it. The money people aren't stupid, despite what the workers think. They just have more information and different motivations which, in most larger organisations, they actively avoid explaining to the operations people.
Edit: Changed 'an incredible description' to 'a brilliant description' because it is entirely credible.
Could you explain this more? Who are the operations people? (finance? HR?) Who is the money side? (product? sales?) How exactly would "operations people" have leverage if they knew how the "money side" worked? Like, what specifically would they do? Also leverage over whom? And ultimately, couldn't whoever all these groups eventually report to (the CEO but possibly lower) resolve all this by firing some people?
I don't have a good model of how these things work. I encountered it first hand, but couldn't develop a good model after a long time of thinking about it. Tons of things happened that I didn't understand at all, and I couldn't understand why higher-level people wouldn't just resolve it.
Help understand this better?
I don't have a good model of how these things work. I encountered it first hand, but couldn't develop a good model after a long time of thinking about it. Tons of things happened that I didn't understand at all, and I couldn't understand why higher-level people wouldn't just resolve it.
Help understand this better?
The way I was thinking of it:
- The 'operations people' / 'workers' ('the competent' in the post I was replying to) are the actual work-doers who create the company's product or provide the company's services.
- The 'money people' are the people in financial and managerial positions who deal with money coming into and out of the company, and who make the financial decisions.
Usually the latter, who know how much people are paid and how much the company makes, take considerable pains to withhold this information from the former, who have specialist skills but are often not business savvy. This is because the company's profit comes from creating as large a gap as possible between the income that the company makes and the wages and other expenses that they pay.
In a small company this information is directly withheld from the staff, but they tend to have some idea. This is part of why small companies tend to either pay better than, or have smaller margins than, large companies.
In a large company, part of the purpose of middle management is to isolate the money people from the operations people so that the operations people don't find out how much they're actually worth to the company. If you find out that you're getting paid $100 to make $10,000 for the company, you're going to be pissed, and either demand a pay raise or leave.
Edit: To be clear, I don't mean to say that middle management are stupid, by any means. Part of their role is, however, ignorance (wilful or imposed, feigned or genuine) of the company's financials.
- The 'operations people' / 'workers' ('the competent' in the post I was replying to) are the actual work-doers who create the company's product or provide the company's services.
- The 'money people' are the people in financial and managerial positions who deal with money coming into and out of the company, and who make the financial decisions.
Usually the latter, who know how much people are paid and how much the company makes, take considerable pains to withhold this information from the former, who have specialist skills but are often not business savvy. This is because the company's profit comes from creating as large a gap as possible between the income that the company makes and the wages and other expenses that they pay.
In a small company this information is directly withheld from the staff, but they tend to have some idea. This is part of why small companies tend to either pay better than, or have smaller margins than, large companies.
In a large company, part of the purpose of middle management is to isolate the money people from the operations people so that the operations people don't find out how much they're actually worth to the company. If you find out that you're getting paid $100 to make $10,000 for the company, you're going to be pissed, and either demand a pay raise or leave.
Edit: To be clear, I don't mean to say that middle management are stupid, by any means. Part of their role is, however, ignorance (wilful or imposed, feigned or genuine) of the company's financials.
This model doesn't quite make sense to me. The layer of idiocy is extremely expensive-- not only do you have to pay their salaries and associated operations costs, but they also introduce enormous inefficiencies. Everything moves slower. It's hard to find important information because it's lost in a sea of bullshit. People make locally optimal decisions that are suboptimal for the company.
The costs of all this are enormous. Is maintaining this army really cheaper than just handing the reigns to ops people and paying them more in proportion to their increased leverage?
The costs of all this are enormous. Is maintaining this army really cheaper than just handing the reigns to ops people and paying them more in proportion to their increased leverage?
It's not a layer of idiocy - that's a common misconception (which I didn't help with, tbf) and unfair to good middle management. What it is is a layer of isolation. And they do also serve another real organisational need, to help coordinate the workforce. Without such a structure, organisation cost is O(N^2). When your business model is paying smart people $X to build a thing, then selling the thing for 10 * $X, you can't let the smart people know that the thing is sold for that much or they'll start demanding more money (or quit and start their own competing company selling a better product for 4 * $X.)
Yes, because it also makes the ops people easily replaceable. On a small scale you can have a tyrant sys admin controlling a company, but on a company scale, nothing is more terrifying than someone other than the CEO being in control
Why does eliminating the bullshit layer mean enormous amounts of power have to be concentrated in any one person's hands? You can hire two sysadmins.
There cannot be two tyrants. Two sysadmins in, one sysadmin out.
Money side is like a mini-VC like ecosystem inside a large corp; people on the money side spend the money in whatever ways they need to maintain their little fiefdoms. Additionally, and with their new found power, they'll layer in a bunch of morons to prevent operations people - who's sole job is to eliminate inefficiencies everywhere - from understanding that the money side is basically a giant game.
If the operations people ever fully understood how badly the money people are fucking things up, they'd have leverage to use against the players on the money side and it'd be game over.
If the operations people ever fully understood how badly the money people are fucking things up, they'd have leverage to use against the players on the money side and it'd be game over.
It's a way to establish power and extract value out of what would otherwise be a more efficient company or system, since those efficiencies don't benefit you directly. The company might do better but your incentives align in a way to extract value from it. If the customer gets burned, or poor quality products get launched, the feedback mechanism is broken. Consolidating power and crushing the competition is more tangible than anything the business actually does.
Which is why stock compensation was expected to help this, at least in theory.
Which is why stock compensation was expected to help this, at least in theory.
> It's a way to establish power and extract value out of what would otherwise be a more efficient company or system, since those efficiencies don't benefit you directly. The company might do better but your incentives align in a way to extract value from it
This is a fancy way of saying corruption. Twisting the system to be used for your own ends, be they strategic, or just straight-up cash-in-my-pocket, is corruption.
Operations workers can see the daily inefficiencies and profit from them, or inject themselves into processes and decisions in ways that will benefit them. Adding a layer of moron/process/politics slows or halts this bad behavior, at the cost of increased overhead and inefficiency.
This is a fancy way of saying corruption. Twisting the system to be used for your own ends, be they strategic, or just straight-up cash-in-my-pocket, is corruption.
Operations workers can see the daily inefficiencies and profit from them, or inject themselves into processes and decisions in ways that will benefit them. Adding a layer of moron/process/politics slows or halts this bad behavior, at the cost of increased overhead and inefficiency.
Sounds like an abridged version of The Gervais Principle.
I take that to mean, people in ops using their ability to bring the company to a grinding halt by turning off all the infrastructure with the aim of extracting things like better pay.
The most powerful force for reform is talented and skilled people finding ways to make money that isn't subject to the games played in large bureaucracies. As they do other things, those other things outcompete the big bureuacracies and those bureaucracies either adapt (thus reforming) or wither away (thus getting replaced by something better).
The problem with this is that government is easily swayed by megacorp money. That money actually creates additional regulation, first by spooking people into thinking the megacorps need to be reined in then by using their influence to ensure the congresscritters hungering for megacorp attention create regulations that are more damaging to industry upstarts than to megacorps that currently dominate their industries. As a result, all these business regulations end up strengthening the hold of industry dominators over their industries, and they just find ways to keep their games going, serving goals wildly different from what most people imagine they really serve. As a result of those regulations being so crushingly bad for small industry upstarts and basically just a drop in the bucket for megacorps, the businesses that could have unseated the current kings -- with an influx of great talent leaving the megacorps -- end up being ground up by regulatory costs until they give up and get bought out by the megacorps, then destroyed or absorbed into efforts opposed to the companies' original high ideals.
Once in a while, someone manages to break through the competition glass ceiling, but usually it's at the cost of selling the soul of the company's original vision and becoming exactly what they originally meant to make obsolete.
The problem with this is that government is easily swayed by megacorp money. That money actually creates additional regulation, first by spooking people into thinking the megacorps need to be reined in then by using their influence to ensure the congresscritters hungering for megacorp attention create regulations that are more damaging to industry upstarts than to megacorps that currently dominate their industries. As a result, all these business regulations end up strengthening the hold of industry dominators over their industries, and they just find ways to keep their games going, serving goals wildly different from what most people imagine they really serve. As a result of those regulations being so crushingly bad for small industry upstarts and basically just a drop in the bucket for megacorps, the businesses that could have unseated the current kings -- with an influx of great talent leaving the megacorps -- end up being ground up by regulatory costs until they give up and get bought out by the megacorps, then destroyed or absorbed into efforts opposed to the companies' original high ideals.
Once in a while, someone manages to break through the competition glass ceiling, but usually it's at the cost of selling the soul of the company's original vision and becoming exactly what they originally meant to make obsolete.
Reminds me of the Gervais Principle - I think this has ruined me for working in larger companies: https://www.ribbonfarm.com/the-gervais-principle/
This is just your own personal model? It's beautiful!
Is there a community where we could discuss something like this? "The spoke." I want to get obsessed with this.
Is there a community where we could discuss something like this? "The spoke." I want to get obsessed with this.
I wouldn't know about it. I have seen more corporate political comments on HN, albeit far and few in between. But they are there.
I think a simple thing to do is to spot them and favorite them if you're interested in this sort of thing.
I think a simple thing to do is to spot them and favorite them if you're interested in this sort of thing.
I'm not good at seeing interpersonal relations, so it took me a couple years of working at a Fortune 50 to figure this out, and it's absolutely true. When I finally put it together, my political/economic beliefs shifted considerably. These organizations aren't there to create value, but to concentrate it, even at the expense of everyone else.
> you only call it politics when you are losing
Gonna remember that one.
Gonna remember that one.
This deserves a larger blog post, perhaps with some more practical examples? I would love to understand this model better.
i dont understand this comment
I think a summary of it is that some things are broken by design in a large org.
If you've worked in a huge org like Apple or the government, you've seen incredible inefficiencies. How can all these smart and often well-paid people be doing something so wrong (i.e. wasting time with political battles, etc.)?
But that's only true if you look at the "medium view" of the organization. If you look at the large view, how the money actually flows, then it might be beneficial for one part like IS&T to be kind of broken, as long as the rest of the company works.
I have seen this dynamic at other companies. Internal tools can sometimes hold too much leverage over the organization. They can almost "blackmail" people into getting their way because they have a literal monopoly over what they do. It might be better to let multiple teams fight battles amongst themselves, which seems inefficient if you're working as a regular joe, but could be efficient from the CEO's perspective.
----
Edit: A related idea is the principal-agent problem:
https://en.wikipedia.org/wiki/Principal%E2%80%93agent_proble...
e.g. if you've worked at any large org, you know that not everyone acts in the company's interest (nor should they strictly speaking). A trivial example of this is putting extravagant meals and vacations on the company's card. (Although the nontrivial instances of this go all the way up to fraud/embezzlement.)
A less trivial example is making up fake engineering work that sounds good when review time comes, but doesn't actually advance the company's business.
So some things that seems mystifying on the ground level are pretty much a defense against this inherent conflict of interest.
In an adjacent domain, a lot of HR is a defense against employees acting in their own interest. (I see a lot of veterans on here saying: "remember HR doesn't exist for you", which is presumably a hard won experience.)
If you've worked in a huge org like Apple or the government, you've seen incredible inefficiencies. How can all these smart and often well-paid people be doing something so wrong (i.e. wasting time with political battles, etc.)?
But that's only true if you look at the "medium view" of the organization. If you look at the large view, how the money actually flows, then it might be beneficial for one part like IS&T to be kind of broken, as long as the rest of the company works.
I have seen this dynamic at other companies. Internal tools can sometimes hold too much leverage over the organization. They can almost "blackmail" people into getting their way because they have a literal monopoly over what they do. It might be better to let multiple teams fight battles amongst themselves, which seems inefficient if you're working as a regular joe, but could be efficient from the CEO's perspective.
----
Edit: A related idea is the principal-agent problem:
https://en.wikipedia.org/wiki/Principal%E2%80%93agent_proble...
e.g. if you've worked at any large org, you know that not everyone acts in the company's interest (nor should they strictly speaking). A trivial example of this is putting extravagant meals and vacations on the company's card. (Although the nontrivial instances of this go all the way up to fraud/embezzlement.)
A less trivial example is making up fake engineering work that sounds good when review time comes, but doesn't actually advance the company's business.
So some things that seems mystifying on the ground level are pretty much a defense against this inherent conflict of interest.
In an adjacent domain, a lot of HR is a defense against employees acting in their own interest. (I see a lot of veterans on here saying: "remember HR doesn't exist for you", which is presumably a hard won experience.)
> But that's only true if you look at the "medium view" of the organization. If you look at the large view, how the money actually flows, then it might be beneficial for one part like IS&T to be kind of broken, as long as the rest of the company works.
To a degree that's true. However I'm reminded of the CEO of Sears, who embraced that concept entirely. He expected each team essentially to battle for funding -- with (unsurprising) result that it was a free-for-all and teams spent more time wrangling and fighting than actually getting stuff done.
A little competition motives; bellum omnium contra omnes is just a waste of resources.
To a degree that's true. However I'm reminded of the CEO of Sears, who embraced that concept entirely. He expected each team essentially to battle for funding -- with (unsurprising) result that it was a free-for-all and teams spent more time wrangling and fighting than actually getting stuff done.
A little competition motives; bellum omnium contra omnes is just a waste of resources.
Well, the mistake there is to not try and convince your teams they're doing what they do because it's the 'right thing' or that they're the "best" and he's proud of them, or whatever whatever non-monetary incentive one can think of that often is surprisingly effective.
It's as if there's three classes of people involved in economic production, workers, managers, and owners. hmm i'll have to think about that some more. It really hits the marks.
This sort of sarcasm does not foster quality discussion.
Can you explain this more?
Replying just so I can find this again. I'm going to have to think about it in the coming days.
You /can/ favorite comments
Yep. Click on the comments time stamp to find the 'favorite' link :)
Hey, thanks! Didn't know that.
Fascinating to see an article about IS&T, definitely not something I ever expected to hear about on HN!
I was part of a team inside Apple that developed internal-facing tools for the retail teams. Our team was formed because of the extremely high cost in dollars and time that IS&T wanted to charge to develop some fairly straightforward tools. My team was taken from the departments that used the tools, and we developed things that were very custom-fit to what those departments needed.
Eventually politics overcame us, and IS&T finally managed to take over our team. We all became contractors in order to support the migration to their infrastructure, and continue development of the tools. A bit later we all became fired as our project was outsourced to contractors in India, managed by IS&T PMs who had no idea what our tool was used for and had never been inside a retail store. The tools all died shortly after that, some killed by IS&T, some petered out due to lack of use now that they no longer worked correctly or were a good fit for the users.
As far as we could tell, IS&T was run as a unique company inside Apple, which did it's fair share of price gouging in order to make itself money to keep going. Its purpose never appeared to be helping Apple customers or Apple employees, it was simply to get bigger, absorbing more money and power wherever possible, with no apparent reigning in from the parent org.
Although my experience is several years old, everything in this article rings true. The contracting companies they had us working for were taking a huge cut, the quality of the code they produced was dismal, (as soon as we were no longer allowed to re-write their code major things began breaking almost immediately) and people getting transferred around constantly and having no time to understand any one project was common. (rkho's comment about their hiring process seeming like it was simply a beard for a nepotistic contractor conversion was something we definitely saw a number of times.)
All in all it was an extremely eye-opening experience. Considering how "do it the Apple way" every other department we interacted with was, being in the IS&T buildings was like landing on an alien planet.
I was part of a team inside Apple that developed internal-facing tools for the retail teams. Our team was formed because of the extremely high cost in dollars and time that IS&T wanted to charge to develop some fairly straightforward tools. My team was taken from the departments that used the tools, and we developed things that were very custom-fit to what those departments needed.
Eventually politics overcame us, and IS&T finally managed to take over our team. We all became contractors in order to support the migration to their infrastructure, and continue development of the tools. A bit later we all became fired as our project was outsourced to contractors in India, managed by IS&T PMs who had no idea what our tool was used for and had never been inside a retail store. The tools all died shortly after that, some killed by IS&T, some petered out due to lack of use now that they no longer worked correctly or were a good fit for the users.
As far as we could tell, IS&T was run as a unique company inside Apple, which did it's fair share of price gouging in order to make itself money to keep going. Its purpose never appeared to be helping Apple customers or Apple employees, it was simply to get bigger, absorbing more money and power wherever possible, with no apparent reigning in from the parent org.
Although my experience is several years old, everything in this article rings true. The contracting companies they had us working for were taking a huge cut, the quality of the code they produced was dismal, (as soon as we were no longer allowed to re-write their code major things began breaking almost immediately) and people getting transferred around constantly and having no time to understand any one project was common. (rkho's comment about their hiring process seeming like it was simply a beard for a nepotistic contractor conversion was something we definitely saw a number of times.)
All in all it was an extremely eye-opening experience. Considering how "do it the Apple way" every other department we interacted with was, being in the IS&T buildings was like landing on an alien planet.
When will companies learn that cheap development labor always ends up being more expensive than starting with quality? This same story has been playing out over and over for decades.
Around 12-13 years ago, I was working (as a team of one on a highly technical upgrade, 32-64 bit upgrade, new compilers, OS changes, etc. Basically everything was changing). Naturally, I was falling behind debugging obscure issues largely due to 32->64 bit (this was C++ cross platform between windows and Linux).
My director offered to toss some offshore resources my way to help. Bluntly told him: I think I have 3 months work ahead of me. Give me offshore resources, it'll more likely be a 9-12 months, because by the time ive described the problem in sufficient detail, I could have just done the work. Instead, id have to review a broken "solution" that doesnt work, probably makes things worse.
Offshore is fine when you need grunt work, pounding out CRUD interfaces or already having a detailed spec. But, offshore doesnt work when you're flying by the seat of your pants.
Most of the big ossues I jad durong that release werent even my own. A few I remember: an x86/x64 inline ASM bug, use of an uninitialized variable in a DB lib that led to random failures dealing with NULLs, an overflow in a 3rd party datetime lib and ABI mismatch because of wrong datatypes used in an OSS ODBC wrapper that worked in 32-bit, but failed horribly on 64-bit.
None of the offshore talent I worked with at the time could have identified, let alone fix those issues. Several of those took reading the actual generated assembler to see what is going on. I dont write much x86/64, but I can read enough to know when shits going to hell.
The 2 big agencies I delt with were Salient and Accenture. I could work with Salient, most of their asserts were Indian and spoke English. Our Accenture resources were all Chinese. It was a pain in the ass because none of our engineering resources spoke or read/wrote English. Everything had to go both ways through a translator. Something was always lost in the translation. We ended up eating the contract and backing out with Accenture.
My director offered to toss some offshore resources my way to help. Bluntly told him: I think I have 3 months work ahead of me. Give me offshore resources, it'll more likely be a 9-12 months, because by the time ive described the problem in sufficient detail, I could have just done the work. Instead, id have to review a broken "solution" that doesnt work, probably makes things worse.
Offshore is fine when you need grunt work, pounding out CRUD interfaces or already having a detailed spec. But, offshore doesnt work when you're flying by the seat of your pants.
Most of the big ossues I jad durong that release werent even my own. A few I remember: an x86/x64 inline ASM bug, use of an uninitialized variable in a DB lib that led to random failures dealing with NULLs, an overflow in a 3rd party datetime lib and ABI mismatch because of wrong datatypes used in an OSS ODBC wrapper that worked in 32-bit, but failed horribly on 64-bit.
None of the offshore talent I worked with at the time could have identified, let alone fix those issues. Several of those took reading the actual generated assembler to see what is going on. I dont write much x86/64, but I can read enough to know when shits going to hell.
The 2 big agencies I delt with were Salient and Accenture. I could work with Salient, most of their asserts were Indian and spoke English. Our Accenture resources were all Chinese. It was a pain in the ass because none of our engineering resources spoke or read/wrote English. Everything had to go both ways through a translator. Something was always lost in the translation. We ended up eating the contract and backing out with Accenture.
> Offshore is fine when you need grunt work, pounding out CRUD interfaces or already having a detailed spec.
My partner works at a startup with a very simple CRUD project. They outsourced one of the early sections of that UI. Even in such a simple case, integrating that component has been a huge time-sink and maintaining it is basically out of the question. They're working on a plan to rebuild it, as so often happens with these things.
Fire-and-forget software simply doesn't work.
My partner works at a startup with a very simple CRUD project. They outsourced one of the early sections of that UI. Even in such a simple case, integrating that component has been a huge time-sink and maintaining it is basically out of the question. They're working on a plan to rebuild it, as so often happens with these things.
Fire-and-forget software simply doesn't work.
Any monkey can write code. Writing the right code critically depends on communication.
It's been 20 years since I worked a corporate job, but a good friend and brilliant colleague / friend told me about his own horror story with Accenture / Chinese consultants. He told me the English skills and attitude of the Indian consultants he used to work with far outweighed the pain of dealing with Accenture consultants, in spite of their tech skills. Language problems and ego / combativeness from the stories he told me. Hearsay, I know... but this guy's work is in products that 90% of the people reading this use, so... yeah.
“My director offered to toss some offshore resources my way to help.”
Brooks’s Law in action. Mythical Man-Month should be required reading for all developers and managers.
Brooks’s Law in action. Mythical Man-Month should be required reading for all developers and managers.
I’ve seen that done just as a gesture of good will, in a "if it helps, we can spend more money"-kind of way, even though everyone involved knows it would just complicate things.
The first company I worked at had a great team in all departments until we got a new head of technology. Immediately all new projects went to outsource teams, some occasionally good and others bad, so much so that we (existing devs) began to “run out” of work. Eventually they tried to replace all the devs but me, the junior, and use me as the US based engineer for customer calls and maintaining legacy products. I said no and left on good terms with everyone but the new head who didn’t understand why I wouldn’t want to stay.
Within 6 months the new head was gone, the legacy code hadn’t been up kept and no new products had made full releases (of a planned 4-5). I did some maintenance and rewriting/launching one of the products I had originally started on as the only dev until they could hire someone else, but it did cost them 3x what they were previously paying me.
Within 6 months the new head was gone, the legacy code hadn’t been up kept and no new products had made full releases (of a planned 4-5). I did some maintenance and rewriting/launching one of the products I had originally started on as the only dev until they could hire someone else, but it did cost them 3x what they were previously paying me.
Our remote team (in SEA) adds a tonne of value to our local Australian based team; but they mostly do maintenance work and are heavily integrated into our local team (they’re quite literally just treated as remote devs, rather than their own company). It’s worked really well for about 6-7 years now
I didn't say Asian labor, I said cheap/hasty labor. As the article makes clear, you can get that in the States too.
You didn't say "cheap/hasty" you said "cheap", and I can assure you teams in Australia outsourcing dev work to SEA are doing it because it's cheap(er).
A lot more people live in SEA than Australia, so it's probably both easier and cheaper to find people. SEA also speaks English and I'm sure there's good programmers around there somewhere.
Another good reason to have remote support in Asia is it's a different time zone. Not for Australia obviously, but that's a harmless thing for a US company to do.
Another good reason to have remote support in Asia is it's a different time zone. Not for Australia obviously, but that's a harmless thing for a US company to do.
> SEA also speaks English
South East Asia’s a big place, and outside of some of the Philippines, Malaysia, and Singapore, English is bad.
South East Asia’s a big place, and outside of some of the Philippines, Malaysia, and Singapore, English is bad.
Would love to read more details how the integration and management works within these organisation.
I would like to ask _why_ does this nightmare exist, but is that the right question? What is the right question to ask here.
I wonder if it is because it lacks critical importance and sexiness.
So it lives out of the limelight.
Also expectations of internal tools are low.
Think about the sexiest jobs in tech. They are working with the newest technology, or developing the funnest resulting product.
If you have someone working on the latest machine learning technology, management has a vested interest in keeping those folks happy.
If you have someone working on computer games, the employee has a vested interest in working hard because they're doing something really cool.
But internal tools development? Expectations from the rest of the company are low, it is viewed as a necessary evil by management but not a revenue generator, and to many employees working there, it is a stepping stone.
Which is really a shame. Good internal tools can really foster a good culture. Shitty tools really take people's minds off the ball.
The worst is when some shitty off-the-shelf tool is site-licensed and shoehorned into an organization and people just suffer. I vaguely remember working somewhere where some sales tool was used as the bug-tracker or trouble-ticket system. gah.
So it lives out of the limelight.
Also expectations of internal tools are low.
Think about the sexiest jobs in tech. They are working with the newest technology, or developing the funnest resulting product.
If you have someone working on the latest machine learning technology, management has a vested interest in keeping those folks happy.
If you have someone working on computer games, the employee has a vested interest in working hard because they're doing something really cool.
But internal tools development? Expectations from the rest of the company are low, it is viewed as a necessary evil by management but not a revenue generator, and to many employees working there, it is a stepping stone.
Which is really a shame. Good internal tools can really foster a good culture. Shitty tools really take people's minds off the ball.
The worst is when some shitty off-the-shelf tool is site-licensed and shoehorned into an organization and people just suffer. I vaguely remember working somewhere where some sales tool was used as the bug-tracker or trouble-ticket system. gah.
Yep, that's what I mean. Wouldn't it be better for all and everything to have a higher-functioning tools team?
This is the ad-absurdum[0] conclusion of the "Cost Center". Cost Centers don't make money, they cost money. Therefore any money you spend on them is lost.
Along with this you have the Iron Law of Bureaucracy - organizations tend (tend!, it's not absolute) to further themselves ahead of their stated goals or directives.
The right question everyone should be asking is how does the parent organization actually make money, and how to help them do that?
0. https://en.wikipedia.org/wiki/Reductio_ad_absurdum
1. https://en.wikipedia.org/wiki/Jerry_Pournelle#Pournelle's_ir...
Along with this you have the Iron Law of Bureaucracy - organizations tend (tend!, it's not absolute) to further themselves ahead of their stated goals or directives.
The right question everyone should be asking is how does the parent organization actually make money, and how to help them do that?
0. https://en.wikipedia.org/wiki/Reductio_ad_absurdum
1. https://en.wikipedia.org/wiki/Jerry_Pournelle#Pournelle's_ir...
A while ago I interviewed for a senior fullstack role with a very homogenous team at Apple -- every single person on the team had been converted from the same consulting company (I believe it was Wipro).
It was a bizarre interview with three of the engineers on the team on the same call asking me very specific programming language trivia and nothing about my experience nor ability to navigate complex political structures -- you know, the things that you would expect a senior fullstack engineer to have before advancing them to the next round.
Unsurprisingly, I wasn't advanced to the next round. In light of this article, I can't help but wonder if it was all a ruse to simply demand another nepotistic FTE conversion from their same consultancy.
It was a bizarre interview with three of the engineers on the team on the same call asking me very specific programming language trivia and nothing about my experience nor ability to navigate complex political structures -- you know, the things that you would expect a senior fullstack engineer to have before advancing them to the next round.
Unsurprisingly, I wasn't advanced to the next round. In light of this article, I can't help but wonder if it was all a ruse to simply demand another nepotistic FTE conversion from their same consultancy.
Perhaps, but perhaps the role just wasn't fit for you. A lot of 'senior dev' roles are much more about programming language trivia and much less about leadership.
> A lot of 'senior dev' roles are much more about programming language trivia and much less about leadership.
I am curious - where do you work?
Asking so that I can potentially avoid that company, as this is the most inane and borderline backwards thing I've ever seen written about senior engineering.
I am curious - where do you work?
Asking so that I can potentially avoid that company, as this is the most inane and borderline backwards thing I've ever seen written about senior engineering.
Everyone above 30 is senior developer. When they all consider themselves leaders and try to lead simultaneously, it is pretty much political hell.
In some startups, everyone above 25 is senior developers. Everyone else is junior, there is no middle role. So it is not like companies had much choice how to label people.
Imo, most of what people write about "senior engineering" is what they personally admire in this or that senior, what they aspire to be or want others to be. It is not what actually role labeled "senior developer" is.
In some startups, everyone above 25 is senior developers. Everyone else is junior, there is no middle role. So it is not like companies had much choice how to label people.
Imo, most of what people write about "senior engineering" is what they personally admire in this or that senior, what they aspire to be or want others to be. It is not what actually role labeled "senior developer" is.
Going to guess defense contractor.
Close enough. I work for a 'body shop' and I'm currently working for a european government client.
Ahh europe - where, anecdotally, all startup and eng is done backwards.
Poor take. There are plenty of good engineering organisations in Europe, just as there are more than enough terrible ones in the US.
That being said I agree I wouldn't want to work somewhere like what was described above. It's certainly not a style of interview I would approve of or conduct myself.
That being said I agree I wouldn't want to work somewhere like what was described above. It's certainly not a style of interview I would approve of or conduct myself.
Not sure what the jab is about. OP works for a body shop in a governmental org, where does that let you imply that startup and eng are backwards as neither are OP's experiences?
If the anecdote is related to your experiences working on FB then, please, specify it, blanket statements aren't what I expect from HN comments.
If the anecdote is related to your experiences working on FB then, please, specify it, blanket statements aren't what I expect from HN comments.
I can't tell if you're being sarcastic, or if you live in some kind of Kafkaesque dystopia I never want to be part of.
At my employer, the best senior developers are the ones who know they can google the answer to trivia so they don't need to remember it.
At my employer, the best senior developers are the ones who know they can google the answer to trivia so they don't need to remember it.
> the best senior developers are the ones who know they can google the answer to trivia so they don't need to remember it.
I've +20 years developing professionally. I remember the stuff I use day to day, and know how to navigate the docs of the technologies I use frequently for the less used tidbits. Being a good Senior isnt about knowing the answers to everything, but knowing how to find the answers quickly. Also, how to ask the right questions.
I don't know is always an acceptable answer, but should be followed up with "I'll get back to you in an hour or two.", or something to that effect (obviously for some problems, may need to experiment/prototype).
I've +20 years developing professionally. I remember the stuff I use day to day, and know how to navigate the docs of the technologies I use frequently for the less used tidbits. Being a good Senior isnt about knowing the answers to everything, but knowing how to find the answers quickly. Also, how to ask the right questions.
I don't know is always an acceptable answer, but should be followed up with "I'll get back to you in an hour or two.", or something to that effect (obviously for some problems, may need to experiment/prototype).
What, you don't remember the argument order to every common file/memory operation like the back of your hand?! /s
Possibly the GP meant there is not much leadership skill required in their senior dev roles and the amount of programming language trivia is same as everywhere.
While I hate to assume people's position, I would venture a guess that the parent is a junior developer who has had one or two experiences that lead them to believe they are better at programming than some senior devs in their company, and has yet to realise how much of a lead they have in soft skills.
That could be the case, I just find it difficult to believe that an ICT3 role at Apple of all companies would be about programming language trivia, given that the specific team I spoke with mentioned how they were very cross-collaborative with other LOBs.
>I just find it difficult to believe that an ICT3 role at Apple of all companies would be about programming language trivia
Fully agreed, but I don't think it is like that across Apple at all. It was like that just for that specific team OP was interviewing with, and it makes sense in the context of that whole team being Wipro converts. This is definitely unusual to have a team like that in the first place, so OP just simply got unlucky.
Given a company of Apple size, no matter how pleasant and competent an average team is, there will always be a few outlier teams that are just whack and awful to work with.
I can confidently say the same about my workplace. I like my team and org a ton (in terms of competency, how the responsibilities are assigned, etc.), and I believe that an average team in the company is very competent and high quality. However, I would be lying if I said that I haven't encountered at least a couple of teams in other orgs at the company that I would never want to work on due to similar issues.
Fully agreed, but I don't think it is like that across Apple at all. It was like that just for that specific team OP was interviewing with, and it makes sense in the context of that whole team being Wipro converts. This is definitely unusual to have a team like that in the first place, so OP just simply got unlucky.
Given a company of Apple size, no matter how pleasant and competent an average team is, there will always be a few outlier teams that are just whack and awful to work with.
I can confidently say the same about my workplace. I like my team and org a ton (in terms of competency, how the responsibilities are assigned, etc.), and I believe that an average team in the company is very competent and high quality. However, I would be lying if I said that I haven't encountered at least a couple of teams in other orgs at the company that I would never want to work on due to similar issues.
ICT3 is just one step above entry at Apple. It's not a senior role.
Sorry, I misinterpreted leveling. It was an ICT4 role. Can't update my comment, so I will just leave it here as-is.
Note that senior isn't actually rare in a large tech company. It goes something like intern, junior, senior, staff, principal, distinguished, fellow. It's hard to get above "staff".
You know something about this specific case. I do not. I hope you land the kind of role you want, if you dont already have it yet.
[deleted]
Some people just don't know how to run an interview
I had a similar experience w/Apple. A startup I worked for was brought into Cupertino for a meeting w/their internal business teams. They wanted us to build them an app which seemed relatively simple involving their internal Cafe Mac cafeterias, data centers, and possibly retail locations. It was something that a company like Apple could build in their sleep. But the business folks we met with said that's how it is at Apple, all the engineering talent goes towards the product side and almost nothing is left for internal IT. They told us how they struggled to get anything done and there were almost no resources available, so the business teams had to go and hire their own IT if they needed things done.
> But the business folks we met with said that's how it is at Apple
That's how it is at most companies.
Split orgs (IT & business) result in only work of sufficient size, scope, and impact being able to cross the barrier.
Multi-year logistics management rewrite? You'll get two IT teams.
Frank in accounting needs to schedule a daily job to copy from one datastore to another? He doesn't have permissions to, and has been doing it manually for the last 5 years.
IMHO, every org along those lines would benefit from an independent IT tiger-team whose sole job it is to find business problems and apply technology to solve them.
That's how it is at most companies.
Split orgs (IT & business) result in only work of sufficient size, scope, and impact being able to cross the barrier.
Multi-year logistics management rewrite? You'll get two IT teams.
Frank in accounting needs to schedule a daily job to copy from one datastore to another? He doesn't have permissions to, and has been doing it manually for the last 5 years.
IMHO, every org along those lines would benefit from an independent IT tiger-team whose sole job it is to find business problems and apply technology to solve them.
And god forbid IT allows tech-savy or power-users to do things, that's not secure.
Most IT departments I've seen are composed of "Desktop Support" kind of personas and are viewed as cost centers.
So they are understaffed and underskilled and use a very conservative approach to system management.
Want to encourage power users? Invest in your IT department, improve developer/engineering - IT relationships.
In my current gig, the IT department has a somewhat friendly cooperative relationship with the SRE org. The SRE's build some automation for important tasks (like onboarding) which the IT team operates; the SRE team is available to help the IT team when things go wrong with such tools. Its not the main focus of either team, but the relationship has produced a very effective IT org.
Want to encourage power users? Invest in your IT department, improve developer/engineering - IT relationships.
In my current gig, the IT department has a somewhat friendly cooperative relationship with the SRE org. The SRE's build some automation for important tasks (like onboarding) which the IT team operates; the SRE team is available to help the IT team when things go wrong with such tools. Its not the main focus of either team, but the relationship has produced a very effective IT org.
Apple's IT is fairly hands-off.
> IMHO, every org along those lines would benefit from an independent IT tiger-team whose sole job it is to find business problems and apply technology to solve them
You're describing external consultantcies. Some are good and do as you describe, but some are bad and work simply to find more work.
You're describing external consultantcies. Some are good and do as you describe, but some are bad and work simply to find more work.
Not in practice, unless they're signed to a guaranteed, multi-year, cross-org contract. Fixed price has so many issues. Hourly has its own Pandora's box.
The incentives (mostly turn time) are just too misaligned to create good solutions, even with good consultants.
Consultants also aren't empowered to fundamentally push back on a client's demands, nor have existing relationship with other IT orgs they'll need buy-in from to get durable work finished.
The incentives (mostly turn time) are just too misaligned to create good solutions, even with good consultants.
Consultants also aren't empowered to fundamentally push back on a client's demands, nor have existing relationship with other IT orgs they'll need buy-in from to get durable work finished.
The political part of IT hates this though.
[deleted]
I could use some blog posts about quite functional teams.
Like someone on a team for a decade talking about why it works so well and discusses cultural and political issues and how the team overcame them.
Like someone on a team for a decade talking about why it works so well and discusses cultural and political issues and how the team overcame them.
I managed a high-functioning C++ image processing group for 25 years. I kept senior-level engineers on staff for decades.
We tended to write "engine code." Like pipelines and whatnot.
The company had a very (very) long tradition of engineering (Like, 100 years).
There were many downsides to the way they worked (over-structured, mostly), but they always gave my team and me a great deal of respect.
They didn't use contractors very often.
To this day, design quality, code quality, product quality, documentation, and focus on deliverable are the major cornerstones of my software work, but I am shocked at how little that matters to modern development shops. It's been a really disheartening experience.
I guess I was in a silo of quality-focus, all those years. They would treat very minor bugs like extinction-level emergencies (not fun).
We tended to write "engine code." Like pipelines and whatnot.
The company had a very (very) long tradition of engineering (Like, 100 years).
There were many downsides to the way they worked (over-structured, mostly), but they always gave my team and me a great deal of respect.
They didn't use contractors very often.
To this day, design quality, code quality, product quality, documentation, and focus on deliverable are the major cornerstones of my software work, but I am shocked at how little that matters to modern development shops. It's been a really disheartening experience.
I guess I was in a silo of quality-focus, all those years. They would treat very minor bugs like extinction-level emergencies (not fun).
> but I am shocked at how little that matters to modern development shops. It's been a really disheartening experience.
Modern development shops operate in a hyper competitive environment with razor thin margins. Unfortunately, great documentation is often the last thing that's prioritized for such shops.
Personally, I think this is a HUGE mistake; talented technical writers are in good supply and should be hired by any development shop to improve adoption of their product. HOWEVER, as a developer, the shop will prefer that you work on features rather than documentation.
Modern development shops operate in a hyper competitive environment with razor thin margins. Unfortunately, great documentation is often the last thing that's prioritized for such shops.
Personally, I think this is a HUGE mistake; talented technical writers are in good supply and should be hired by any development shop to improve adoption of their product. HOWEVER, as a developer, the shop will prefer that you work on features rather than documentation.
There are ways to write code in a manner that leaves a documentation trail.
I write about that here: https://littlegreenviper.com/miscellany/leaving-a-legacy/
I write about that here: https://littlegreenviper.com/miscellany/leaving-a-legacy/
Is this company still in NYC?
Not NYC. Long Island, but headquartered in Tokyo. I’m avoiding directly naming them, so I don’t have to deal with any agita, but they make photographic equipment, and are a household name.
Things I've seen that seemed to contribute to team happiness and productivity:
1) Process-focus. It's not "stop doing that, Bob", it's "let's make sure that can't happen unless we really, really want it to". And not just for technical stuff.
2) But also a willingness to kill processes if they weren't useful. Always OK to ask "who needs this meeting?" or, understanding its purpose, "can this be a few 10-second Slacks with follow-up when necessary, instead of a 15-minute team meeting every day?"
3) Project managers who were "on the team's side", at least so far as the team could tell. You come in taking the whip to people, you end up with bullshit information. You can't manage a project effectively with bad info. You want your team to tell you when shit's going sideways, or just that it might, early. They won't do that if they think they're taking on personal risk by raising issues. And they'll be stressed out and their work will be worse.
4) Semi-relatedly, estimation based on past (team) performance, on the same project. No guess-timates becoming commitments. No "so, can you have that tomorrow? Two days?" This is one part of the "agile" thing that, when done right, I've not seen anything surpass in accuracy or in keeping panic and confusion away. It does mean your estimates at the start of a project can't usefully go very far into the future and will best be expressed as large ranges until at least a few weeks in, and that you can't swap people around all the time and keep estimating accurately—but that's true anyway, even if you pretend it's not. I have seen people care a lot that their estimates often suck but not be willing to give up the "so, one day? Two days?" ambushes or accept that there will be times in a project when responsible estimation is very imprecise. These folks tend to get bad info on top of the inherent flaws in their approaches, because they have trouble with #3.
5) No "I have five bosses" crap.
That all may be more "in the weeds" and/or obvious than what you wanted, though.
1) Process-focus. It's not "stop doing that, Bob", it's "let's make sure that can't happen unless we really, really want it to". And not just for technical stuff.
2) But also a willingness to kill processes if they weren't useful. Always OK to ask "who needs this meeting?" or, understanding its purpose, "can this be a few 10-second Slacks with follow-up when necessary, instead of a 15-minute team meeting every day?"
3) Project managers who were "on the team's side", at least so far as the team could tell. You come in taking the whip to people, you end up with bullshit information. You can't manage a project effectively with bad info. You want your team to tell you when shit's going sideways, or just that it might, early. They won't do that if they think they're taking on personal risk by raising issues. And they'll be stressed out and their work will be worse.
4) Semi-relatedly, estimation based on past (team) performance, on the same project. No guess-timates becoming commitments. No "so, can you have that tomorrow? Two days?" This is one part of the "agile" thing that, when done right, I've not seen anything surpass in accuracy or in keeping panic and confusion away. It does mean your estimates at the start of a project can't usefully go very far into the future and will best be expressed as large ranges until at least a few weeks in, and that you can't swap people around all the time and keep estimating accurately—but that's true anyway, even if you pretend it's not. I have seen people care a lot that their estimates often suck but not be willing to give up the "so, one day? Two days?" ambushes or accept that there will be times in a project when responsible estimation is very imprecise. These folks tend to get bad info on top of the inherent flaws in their approaches, because they have trouble with #3.
5) No "I have five bosses" crap.
That all may be more "in the weeds" and/or obvious than what you wanted, though.
- Project managers who were "on the team's side", at least so far as the team could tell.
This. I've found this so imporant in my time. As far as I'm concerned, if a manager can't keep their own team on side, they might as well not bother. When the manager is popular and people unite around them in times of stress, then everything will work so much better. From there camaraderie and esprit de corps naturally starts to follow, especially if the product manager personally hired many of the members of the team and starts to build a tradition.
If I were to add something to your list it would be accountability, but not blame. If somebody does something wrong, both engineering wise but also in soft skills, its important that figures of authority take the time to both explain where the person went wrong with a view to trying to understand why the person made the mistake, but also visibly put into effect processes for the future to prevent this kind of thing, either by adding it to the onboarding training or with some kind of code review or even just make sure everyone in the team knows a key piece of information that person may have missed.
This. I've found this so imporant in my time. As far as I'm concerned, if a manager can't keep their own team on side, they might as well not bother. When the manager is popular and people unite around them in times of stress, then everything will work so much better. From there camaraderie and esprit de corps naturally starts to follow, especially if the product manager personally hired many of the members of the team and starts to build a tradition.
If I were to add something to your list it would be accountability, but not blame. If somebody does something wrong, both engineering wise but also in soft skills, its important that figures of authority take the time to both explain where the person went wrong with a view to trying to understand why the person made the mistake, but also visibly put into effect processes for the future to prevent this kind of thing, either by adding it to the onboarding training or with some kind of code review or even just make sure everyone in the team knows a key piece of information that person may have missed.
See my old comment https://news.ycombinator.com/item?id=19468090#19468699 . That kind of thing used to have a much higher public profile about 30 years ago.
> For Apple, fixing its broken IS&T division would not only be the right thing to do from a moral standpoint — it would help the company’s business as well. If Apple is going to become inventive again, it will need to give its employees more time to develop new ideas.
I actually laughed. What kinds of applications do people think IS&T is building? Supply chain management, HR, support software (Apple forums anyone?)... I don't see how Apple sticking its core engineering resources or at all innovating within IS&T helps their business.
Steve was the one that coined the 10X rule, or popularized it. That idea must still hold within Apple's executive ranks. I guarantee Apple's top engineers never interface with IS&T and their time won't be saved by improving it. In fact I would bet that managing the deliverable of one of IS&T's projects is a sign that you're not one of the golden ones.
That said, I obviously think these contractors should be treated much better and overworking people for low pay is never okay. Maybe Apple could use some smaller dev shops that treat their people better.
I actually laughed. What kinds of applications do people think IS&T is building? Supply chain management, HR, support software (Apple forums anyone?)... I don't see how Apple sticking its core engineering resources or at all innovating within IS&T helps their business.
Steve was the one that coined the 10X rule, or popularized it. That idea must still hold within Apple's executive ranks. I guarantee Apple's top engineers never interface with IS&T and their time won't be saved by improving it. In fact I would bet that managing the deliverable of one of IS&T's projects is a sign that you're not one of the golden ones.
That said, I obviously think these contractors should be treated much better and overworking people for low pay is never okay. Maybe Apple could use some smaller dev shops that treat their people better.
"They’re just fighting for the roles,” Sabapathy told me. “That’s all they care about, not the work, not the deliverables, the effort they put in, or even talent. They’re not looking for any of those aspects.”
This is spot on for every large organization I ever consulted for. Moreover. Even in much smaller businesses, typically starting at around 70-100 people (sooner if management has prior large business experience) you can already see this pattern starting to seep in.
This is also why the whole private vs public sector efficiency debate is farcical. They are both identical in this aspect.
This is spot on for every large organization I ever consulted for. Moreover. Even in much smaller businesses, typically starting at around 70-100 people (sooner if management has prior large business experience) you can already see this pattern starting to seep in.
This is also why the whole private vs public sector efficiency debate is farcical. They are both identical in this aspect.
Sounds like the politics of every large corporate IT department, but with some twists given that it's Apple. Maybe this helps explain why Apple is not Amazon.
If there's a company CEOs should be afraid of, it's Amazon. They killed bookstores, mall stores, how many other retailers? They were the first major cloud provider. They tried their hand at phones, streaming video, and streaming music, and the bought a grocery store. There's nothing they won't try, and they have the war chest to do it.
Is there any big consumer product company out there that doesn't deprioritize management and engineering talent for internal tools and systems ?
Google and to a lesser extent Facebook invest heavily in internal tools. IMO this is because they hire for the company, permit easy transferring, and programmers sure love to scratch their own itch.
Google has a tool where interviewers can provide feedback and a numerical rating for candidates. But it also has a whole data mining workflow: you can visualize an interviewer's past ratings on a histogram, compare their hire ratio against average hire ratios, etc. It's cool and interesting, and sees a lot more love than many of their user-facing products.
Facebook has Bootcamp, their ~8 week onboarding process. There's a zoo of purpose-built tools just for Bootcamp, like the bot that sends you a question whose answer is written on the whiteboard just to confirm that you attended some Bootcamp class, or the Bootcamp Mentor rating machinery.
Some of that is excessive but Apple's underinvestment sin is far worse. The SWE org learned to run lean during the dotcom bust, and never unlearned it. Radar is your lifeblood, make it best in the world! Please!
Source: worked at all three!
Google has a tool where interviewers can provide feedback and a numerical rating for candidates. But it also has a whole data mining workflow: you can visualize an interviewer's past ratings on a histogram, compare their hire ratio against average hire ratios, etc. It's cool and interesting, and sees a lot more love than many of their user-facing products.
Facebook has Bootcamp, their ~8 week onboarding process. There's a zoo of purpose-built tools just for Bootcamp, like the bot that sends you a question whose answer is written on the whiteboard just to confirm that you attended some Bootcamp class, or the Bootcamp Mentor rating machinery.
Some of that is excessive but Apple's underinvestment sin is far worse. The SWE org learned to run lean during the dotcom bust, and never unlearned it. Radar is your lifeblood, make it best in the world! Please!
Source: worked at all three!
Netflix. At least when I was there.
Internal tools got just as much support as product. Engineers moved between working on both, and the hiring process was the same for both.
I did an interview recently about this on the retool blog [0]
[0] https://retool.com/blog/how-to-build-great-internal-tools-je...
Internal tools got just as much support as product. Engineers moved between working on both, and the hiring process was the same for both.
I did an interview recently about this on the retool blog [0]
[0] https://retool.com/blog/how-to-build-great-internal-tools-je...
Nice interview, great mindset.
Netflix is indeed becoming quite the sensation for its internal policies— the general intelligence apparently prevailing there. Seeing tools as first-grade citizens seems like a very sane approach to me— empower each other and watch people create marvels.
Papermill¹ notably is one incredibly seducing tool IMHO.
1: 2019 talk by Matthew Seal, ~40 min https://www.youtube.com/watch?v=3FmBJ847_y8
Netflix is indeed becoming quite the sensation for its internal policies— the general intelligence apparently prevailing there. Seeing tools as first-grade citizens seems like a very sane approach to me— empower each other and watch people create marvels.
Papermill¹ notably is one incredibly seducing tool IMHO.
1: 2019 talk by Matthew Seal, ~40 min https://www.youtube.com/watch?v=3FmBJ847_y8
Yes, Facebook invests heavily in developer infrastructure. On my former team there were many staff+ engineers who were extremely capable. More speculatively I’d go so far as saying quality of talent is actually stronger than many groups working on public facing products.
Management of the group was also strong in my opinion.
Management of the group was also strong in my opinion.
Does Facebook maintain an internal fork of Phabricator or is it relatively close to mainline? It's such a great product but parts of it are quite clunky - I sometimes wonder if there's some internal secret sauce to smooth these out.
Phabricator was/started to be replaced a year or two ago. AFAIK it's been largely rewritten at this point.
Apple IS&T isn't development infrastructure - while they do have some generic services, most groups at Apple develop and use their own tooling.
Or they use tools made by Developer Tools, many of which are used by external developers as well and ship as part of Xcode.
Developer infrastructure does not seem to be the right area to compare with here just like Google's internal engineering products aren't, more like - how polished are the tools used by the ad sales people, analysts, moderators etc.
+ 1 to Facebook.
It's an interesting question. And if the answer is no, I guess the logical conclusion is that internal tools make very little difference to the success of consumer product companies.
That doesn't seem like an intuitive conclusion, but it's a hard circle to square. As someone who mostly builds internal tools (though in a very different context), it certainly seems like good systems are an Archimedean lever that acts as a force multiplier across the entire org. Yet, if good internal tooling was important, you think at least one company would have been able to achieve success partially on this basis.
The consumer product industry is extremely competitive, so the counter-hypothesis of entrenched sclerotic incumbents refusing to pick up $20 bills off the sidewalk doesn't really seem likely.
That doesn't seem like an intuitive conclusion, but it's a hard circle to square. As someone who mostly builds internal tools (though in a very different context), it certainly seems like good systems are an Archimedean lever that acts as a force multiplier across the entire org. Yet, if good internal tooling was important, you think at least one company would have been able to achieve success partially on this basis.
The consumer product industry is extremely competitive, so the counter-hypothesis of entrenched sclerotic incumbents refusing to pick up $20 bills off the sidewalk doesn't really seem likely.
> And if the answer is no, I guess the logical conclusion is that internal tools make very little difference to the success of consumer product companies.
That is on the implicit assumption that businesses are reasonable, and accurately know how to manage tradeoffs. Your counter-hypothesis seems rather likely to me, if we look at the internal behavior of a company. When negotiating, the more dots that you need to connect before reaching "and then we get more money", the weaker of a negotiating position you have. It's the same reason why sales teams get bonuses for making sales, but developers don't get bonuses for enabling sales.
That is on the implicit assumption that businesses are reasonable, and accurately know how to manage tradeoffs. Your counter-hypothesis seems rather likely to me, if we look at the internal behavior of a company. When negotiating, the more dots that you need to connect before reaching "and then we get more money", the weaker of a negotiating position you have. It's the same reason why sales teams get bonuses for making sales, but developers don't get bonuses for enabling sales.
> And if the answer is no, I guess the logical conclusion is that internal tools make very little difference to the success of consumer product companies.
Or perhaps no one has realized the power of good internal tools, and so crappy tools are the status quo and no one needs to do better.
But as soon as one company realizes the value of internal tools and it becomes a competitive advantage, other companies in that area will need to follow suit.
Or perhaps no one has realized the power of good internal tools, and so crappy tools are the status quo and no one needs to do better.
But as soon as one company realizes the value of internal tools and it becomes a competitive advantage, other companies in that area will need to follow suit.
Long feedback chains take longer to optimize so this is possible. Also one feedback would be to have the company fail. Another possibility is that it really is a tradeoff of where to focus.
For example, we are now paying for our lack of preparation and sclerotic government with the economic and lives lost to covid-19. It just took between 3-10 years to pay the piper.
For example, we are now paying for our lack of preparation and sclerotic government with the economic and lives lost to covid-19. It just took between 3-10 years to pay the piper.
Internal tools are used by employees who unlike consumers will tolerate more usability friction, non-critical bugs and feature delays than consumers and will compensate for them with their own effort so the tools can still be critical in their core functionality but not polished or well-maintained.
Ironically and anecdotally, WeWork tried to build a strong engineering org for their internal systems before it blew up.
Ironically and anecdotally, WeWork tried to build a strong engineering org for their internal systems before it blew up.
To the contrary, some of the most successful software companies have strong cultures of internal tools (think FAANG companies). How much of this is cause and effect I'm not sure though. To some extent it's easier to make investments in internal tooling if you're profitable.
Facebook in particular seems to wildly overinvest in internal tools, and if you ask them why they don't use external stuff they tell you it's because "it doesn't scale". We get things like Phabricator and HHVM out of it, but they could've just not written those, and I'm pretty sure their business would be fine.
They do it because it gets them promoted, and because FB and Google rely on hiring every engineer ever and giving them make-work so they won't leave and start a competitor.
They do it because it gets them promoted, and because FB and Google rely on hiring every engineer ever and giving them make-work so they won't leave and start a competitor.
Interesting! This isn't how I remember things at all. Do you know where you got this sense of things from, specifically?
In particular, do you remember what gave you the impression that Facebook "overinvested" a large amount of effort into Phabricator, that I developed and open sourced Phabricator primarily to get promoted, that I built Phabricator because of scaling concerns, or that the primary value I provided to Facebook during my employment there was just in not starting a competitor?
In particular, do you remember what gave you the impression that Facebook "overinvested" a large amount of effort into Phabricator, that I developed and open sourced Phabricator primarily to get promoted, that I built Phabricator because of scaling concerns, or that the primary value I provided to Facebook during my employment there was just in not starting a competitor?
Some larger video game production companies have started to do a lot more on their internal tooling, as it has a huge impact on their content production.
Most of them learned the hard way though. And some learned the hard way and still have awful tooling despite some efforts to fix them (Bungie...).
Most of them learned the hard way though. And some learned the hard way and still have awful tooling despite some efforts to fix them (Bungie...).
The company i work for is not big or consumer-facing, so this is not an answer to your question.
But we do have a dynamic that i find interesting. The company is organised into many small business units - some very small, fewer than ten people. Each is accountable for its own profit and loss. They hire their own programmers to build the actual line-of-business software they need. But there are also internal tools and systems, shared across units or used by their developers, that are built by programmers hired directly by the top level of the company - their reporting line goes straight up to the CIO and CEO, rather than to a business unit manager. So, in a way, our company does the exact opposite of deprioritizing internal tools and systems! The programmers who develop the internal tools and systems are the elite!
But we do have a dynamic that i find interesting. The company is organised into many small business units - some very small, fewer than ten people. Each is accountable for its own profit and loss. They hire their own programmers to build the actual line-of-business software they need. But there are also internal tools and systems, shared across units or used by their developers, that are built by programmers hired directly by the top level of the company - their reporting line goes straight up to the CIO and CEO, rather than to a business unit manager. So, in a way, our company does the exact opposite of deprioritizing internal tools and systems! The programmers who develop the internal tools and systems are the elite!
Sounds like Intuit.
I expect those tech giants which offer B2B solutions to have better situations. e.g. Google, Amazon, Facebook, Microsoft, etc. Many of their internal products eventually have made the positions in their cloud portfolios.
Google?
Googler here, but opinion is my own. Google's problem is that we have too many powerful internal engineering tools, many of which overlap in functionality, and so often its hard to know at first which one to use to solve a problem.
Thankfully, there is nearly always an overlap period when one solution gets deprecated and the replacement is launched.
The COVID19 office shutdown has really focused attention on our internal IT tools (most of which are also offered externally). So far, they've worked pretty great, in no small part because things like BeyondCorp [1] were designed from the start for distributed workforces in zero-trust security environments.
It wasn't always like this though. Corp-Eng, as it's called at Google was for a long time seen as a dead end career-wise - disconnected from Google's marquee products. However, a number of developments, including the rise of the enterprise and cloud computing businesses, have bolstered its internal importance, and with that have brought excitement and talent to the organization. It's now an important source of product feature ideas that eventually make their way to customers.
That said, a lot of things not core to Google's problem spaces do get solved with outside vendors' software, and some of that software is great. For example, I just used a site licensed version of the Chrome ScreenCastify [2] extension to make a video demonstrating a feature I'm developing. It worked perfectly.
1. https://cloud.google.com/beyondcorp
2. https://www.screencastify.com/
Thankfully, there is nearly always an overlap period when one solution gets deprecated and the replacement is launched.
The COVID19 office shutdown has really focused attention on our internal IT tools (most of which are also offered externally). So far, they've worked pretty great, in no small part because things like BeyondCorp [1] were designed from the start for distributed workforces in zero-trust security environments.
It wasn't always like this though. Corp-Eng, as it's called at Google was for a long time seen as a dead end career-wise - disconnected from Google's marquee products. However, a number of developments, including the rise of the enterprise and cloud computing businesses, have bolstered its internal importance, and with that have brought excitement and talent to the organization. It's now an important source of product feature ideas that eventually make their way to customers.
That said, a lot of things not core to Google's problem spaces do get solved with outside vendors' software, and some of that software is great. For example, I just used a site licensed version of the Chrome ScreenCastify [2] extension to make a video demonstrating a feature I'm developing. It worked perfectly.
1. https://cloud.google.com/beyondcorp
2. https://www.screencastify.com/
[deleted]
What do we know about the quality of the tools used by say the chromebook sales analysts ? This isn't spanner or bigtable.
I've heard Google internal tools are very hit and miss. The engineering stuff is top-notch, but things like their applicant tracking system is archaic (or was a few years ago).
On average, a Google internal tool is better than a commercially equivalent tool elsewhere, especially if it's developer critical.
For example the code search, testing, code review tools are amazing and better than any commercial equivalent.
Other tools like the bug database are on par with the industry.
Others like the food, directory and interviewing tools are still pretty damn good and even if they are lacking some polish, are often far more functional than the average commercial equivalent.
And of course they are hit and miss - because there are so many of them. The ones that are commonly used are well funded and have lots of people working on them.
I no longer work at Google but still consider it a role model for how companies should take tooling seriously.
For example the code search, testing, code review tools are amazing and better than any commercial equivalent.
Other tools like the bug database are on par with the industry.
Others like the food, directory and interviewing tools are still pretty damn good and even if they are lacking some polish, are often far more functional than the average commercial equivalent.
And of course they are hit and miss - because there are so many of them. The ones that are commonly used are well funded and have lots of people working on them.
I no longer work at Google but still consider it a role model for how companies should take tooling seriously.
Google's internal corp tools designed for majority of its employees are pretty nice. Tools for Chromebook sales analysts might not be as great though, but it also doesn't make sense to put tons of investments into a tool used by just tens of sales people. (Of course, there's a number of general marketing tools for thousands of sales)
I am not sure about the funding level but the Google internal tooling for build/code review/search are the best I have seen.
[deleted]
IS&T is run by the CFO and it shows.
The contract companies (Wipro, Infosys) hire graduates by the 100's from mediocre engineering colleges, slap on a little bit of training and advertise them as world class. The starting salaries are something like INR 25,000 (USD 350) per month. An onshore posting for these low skilled graduates is a quick way to make money. So they suck up to their managers who suck up to bigger managers. Their main criteria is to get hold of contract work by undercutting, knowing very well there is vendor lock in.
Body shop wars. What gets really interesting is working for a body shop with consultants from 2 other shops on the same team.
IS&T shouldn’t be part of Apple. It’s culture and engineering talent aren’t representative of core engineering.
Hey, Apple still hired them. Apple is chosing to dilute its core engineering brand. It's not like an end user can tell if a "real engineer" wrote the shitty webpage that's currently giving them grief.
Consumers don't interact with IS&T projects.
According to the article they serve Apple Retail, so every time the Genius messes around with the iPad for 10 minutes to check me in just so I can drop my phone off for a new battery, that’s a consumer dealing with IS&T.
This is wrong and like saying "people that eat sausage don't interact with the contractor at the sausage factory hiring mechanics that don't know any better when fixing the machines and regularly spill machine oil into the meat". Dealing with shit that rolls downhill is a form of interaction.
No, it's right in the context of the comment I was replying to, which had customers interacting with IS&T webpages. Internal apps are decidedly mediocre for the employees who use them but there are many more impactful things that would affect customers than that.
the apple employee who is struggling to meet my needs because of that shitty app. it directly impacts customers by making it take longer to solve my issue.
In the old adage that "you get what you pay for", the internal tools at Apple aren't quite the best as a result. The Radar tool I recall was pretty horrible in particular.
Radar is the world's best bug tracker. The sole reason for this is that the originator gets to close bugs, not the fixer.
Does IS&T also build the developer-facing Feedback Assistant?
No, there's another team for that.
"Don't work in IS&T" has been the common wisdom in Blind for years now. Interesting how long it takes for the media to pick up stories like this.
Is this the team handling iTunesConnect and other such "services"? I'd definitely believe it.
From what I understand, probably not.
iTunesConnect has had an overhaul in recent years to give it a much shinier and more reliable frontend. I believe the original was closer to the original WebObjects code, and now they've extracted out the user-facing bit into something a bit more modern. There's a lot of legacy WebObjects stuff around the iTunes backend, and my guess is that it's good engineers held back by old tech, rather than just bad engineers.
iTunesConnect has had an overhaul in recent years to give it a much shinier and more reliable frontend. I believe the original was closer to the original WebObjects code, and now they've extracted out the user-facing bit into something a bit more modern. There's a lot of legacy WebObjects stuff around the iTunes backend, and my guess is that it's good engineers held back by old tech, rather than just bad engineers.
It's App Store Connect now!
IS&T is purely internal. Basically no one outside of Apple uses anything IS&T makes.
IS&T is largely internal, but there are a number of IS&T owned external web apps for registration, surveys, and support related things. I worked in IS&T on teams developing them years ago.
No, services are run by Apple Media Products under Eddy Cue. They're not great, either, but I think they have more full-time engineers than IS&T does.
It’s their sister org for lack of a better term.
No.
The part I don't get is, why does a company, especially one like Apple who would count software quality as a core competency, choose to pay $120-150 an hour to get contractors who aren't even good enough to command a salary (or hourly rate) of half that. I get why like, United Airlines does that. Or the US Government. They probably rightly assume that they can't properly manage a huge software development or IT org. But Apple? Really? What the hell are they gaining? If they paid a developer full time $120k a year plus benefits that still beats $120+ an hour. WTF.
The days when the primary mission of IS&T was to develop custom software to run Apple are probably long past. The big exception is Radar, an ideosyncratic issue-tracking system which was developed in-house by IS&T around 1990. As far as I know, Radar is still used in R&D and elsewhere at Apple.
Tim Cook probably couldn't care less about the internal affairs of IS&T. As long as they do their job, play by the rules, and don't impose too much overhead on the company, it's all good.
Tim Cook probably couldn't care less about the internal affairs of IS&T. As long as they do their job, play by the rules, and don't impose too much overhead on the company, it's all good.
I would be grateful if any Apple insiders here can discuss the risk of layoffs if one were to join the company now.
I have an offer to join an engineering team at Apple, but the chances of getting hit by layoffs is giving me pause. More details on my situation are here: https://news.ycombinator.com/item?id=22809769
I have an offer to join an engineering team at Apple, but the chances of getting hit by layoffs is giving me pause. More details on my situation are here: https://news.ycombinator.com/item?id=22809769
Apple didn't have mass layoffs when it was almost dead I have been told by engineers who were around at that time. I'd think Apple is one of the safest tech companies to be in as an engineer when it comes to times like this.
Apple has a ton of cash. I think layoffs are unlikely. Many times new hires are protected as well for a bit.
Do you work for Apple? Might you have any insight into how the company decides how / who to layoff, when the time comes?
I would have thought that the newest would be the most vulnerable, since they haven't had the time to absorb institutional knowledge and make internal connections.
I would have thought that the newest would be the most vulnerable, since they haven't had the time to absorb institutional knowledge and make internal connections.
IS&S is not the same as IS&T. Totally different orgs.
How does IS&S compare, relative to other orgs at the company? Does the org you work for (in contrast to your role or seniority level) matter, when it comes to job security?
Thanks for your insights.
Thanks for your insights.
IS&S is much more stable than IS&T, but you'll likely get light, unofficial disdain from software engineering because the organization has a reputation of working on some of the poorer-quality products at the moment. I have heard from the grapevine that Apple is hiring aggressively at the moment, and I wouldn't expect a layoff given that they're huge and fairly financially secure. Large corporations usually weather recessions fairly well. But I have no firsthand experience with your particular situation and I'm just a random person on the internet, so YMMV.
Sounds exactly like government contracting.
Lol. Just like Govt contracting.
Does Radar belongs to IS&T?
And I wonder if this IS&T power grabbing and politics happen and grow within the past 10 years. i.e Happen under Tim Cook's watch.
It surely doesn't seems very Steve Jobs approach.
And I wonder if this IS&T power grabbing and politics happen and grow within the past 10 years. i.e Happen under Tim Cook's watch.
It surely doesn't seems very Steve Jobs approach.
One of the world's most successful and wealthy companies in history shouldn't be bottom fishing for the best "deal" on employees. I wonder how much better they could be if they decided to hire at the top of the market like Netflix instead of the bottom.
That is probably why Apple software is among the worst in the industry, and it's a shame. I know IS&T isn't strictly software like iTunes but it fits the pattern I've seen at Apple. But I guess if it doesn't affect their sales maybe I'm wrong and it doesn't matter and they are right.
That is probably why Apple software is among the worst in the industry, and it's a shame. I know IS&T isn't strictly software like iTunes but it fits the pattern I've seen at Apple. But I guess if it doesn't affect their sales maybe I'm wrong and it doesn't matter and they are right.
The quality of Apple's software engineering organization is head and shoulders above IS&T's; iTunes (…and Apple Music, News, etc.) is run under a different (services) group.
I tell people you only call it politics when you are losing. More accurately, it's a layer of literal stupidity above the competent to shield the money side of the company from the leverage that operations people would have if they had any information about how the money side worked.
Instead of a hierarchy, rethink a company as a hub and spoke model with concentric rings. The main differences are the implication in a hierarchy that there is "gravity," keeping people down and that they need energy and leverage to climb "up," which further implies there is a place to "fall," and that there is only one way "up," instead of many possible paths to the centre from all directions. There is no gravity, only gates and barriers, and even these are just information. Politics is how a middle manager runs interference and creates distractions to make sure you can't see over, around, or through them, and that the people behind them closer to the money can't see you. Tech is usually outside the main perimeter, mediated by contracting companies or middle managers whose job is to compartmentalize the value people create, and be sure it is replaceable.
Viewed this way, of course this demented political farce is how Apple works, because it's how everything seems to work when you have internalized the precise and specific mental model someone uses to take advantage of you.
Sorry if you can't unsee it now, but hopefully it will be funny and we can get good, competent people who value tangible skills into positions of power.