Outsourcing Cost Boeing Billions (2019)(medium.com)
medium.com
Outsourcing Cost Boeing Billions (2019)
https://medium.com/javascript-scene/why-cutting-costs-is-expensive-how-9-hour-software-engineers-cost-boeing-billions-b76dbe571957
119 comments
A key factor in such failure is misaligned incentives. A team in a company is more likely interested in solving problems for the longevity of the company, so they think long term and they iterate. They also learn about the domain over the years. On the other hand, outsourcing companies just want to get a project done in time and under budget, and their hire the least qualified people to just meet the negotiated bar. To that end, there's really no incentive for an outsourcing company to hone any kinds of craftsmanship (there are exceptions in small and specialized consulting firms, of course). Plus, there's always information loss and distortion when a company communicates their needs to an outsourcing vendor. Consequently, all hell will break lose when a project is complicated enough.
It is not necessarily outsourcing fault but rather “people is a resource” mentality. This mentality leads to outsourcing to cheap locations because they are cheap. But I’ve seen in other context as well.
So they'd create a subsidiary at the location of the contractors, which is already a standard practice...
And I already have a plan for doing so: https://gavinhoward.com/2024/06/a-plan-for-professionalism/ .
The problem with any professionalism setup is that software engineering is a very broad field whose use cases range from "it doesn't matter, it's mostly for fun" and "this software runs the international space station".
On the one hand, there should probably be some sort of certification or liability if you're writing code for NASA or Boeing or Airbus or maybe the New York Stock Exchange or what not, where a mistake could cost billions of dollars and put lives at risk.
At the same time though, there's also stuff like the video games industry, people developing programs for fun, blogs and fan sites and things that are free and mostly static, video game mods and hacks and about 99% of small business and organisation websites where frankly it doesn't matter that much how well they're coded or how stable they are or what not.
If my bank screws up and gives away all my money, that's a problem. If my copy of Tears of the Kingdom breaks, that's not a big deal (and might even be a benefit).
If there's any sort of professionalism or certification or requirement, it should probably be limited to industries and fields where the consequences of screwing up are life changing for a high number of people.
On the one hand, there should probably be some sort of certification or liability if you're writing code for NASA or Boeing or Airbus or maybe the New York Stock Exchange or what not, where a mistake could cost billions of dollars and put lives at risk.
At the same time though, there's also stuff like the video games industry, people developing programs for fun, blogs and fan sites and things that are free and mostly static, video game mods and hacks and about 99% of small business and organisation websites where frankly it doesn't matter that much how well they're coded or how stable they are or what not.
If my bank screws up and gives away all my money, that's a problem. If my copy of Tears of the Kingdom breaks, that's not a big deal (and might even be a benefit).
If there's any sort of professionalism or certification or requirement, it should probably be limited to industries and fields where the consequences of screwing up are life changing for a high number of people.
Yes, which is why https://gavinhoward.com/2023/11/how-to-fund-foss-save-it-fro... specifically calls that out.
[flagged]
Ok, but please don't fulminate on Hacker News. This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html.
They are only but a symptom.
I think that describing the MCAS disaster using predominantly SW Engineering language is misleading.
The substantial change that might have averted the catastrophe is having qualified engineering oversight integrated into the MCAS project management structure.
MCAS is a flight control application; its defining discipline is control theory. Thus the hypothetical engineer who could have averted the catastrophe would have had to be a controls, rather than a SW person.
I have read much about MCAS, but no detailed narrative ever mentioned a SW bug, which implies that MCAS SW has apparently functioned according to given specifications.
Thus, while $9/hr SW engineers is a choice correlated with an inadequate safety culture, I fail to see the casual link between that and the tragic outcomes; I cannot condone asserting such causation.
The substantial change that might have averted the catastrophe is having qualified engineering oversight integrated into the MCAS project management structure.
MCAS is a flight control application; its defining discipline is control theory. Thus the hypothetical engineer who could have averted the catastrophe would have had to be a controls, rather than a SW person.
I have read much about MCAS, but no detailed narrative ever mentioned a SW bug, which implies that MCAS SW has apparently functioned according to given specifications.
Thus, while $9/hr SW engineers is a choice correlated with an inadequate safety culture, I fail to see the casual link between that and the tragic outcomes; I cannot condone asserting such causation.
The 737-max was aeronautical malpractice that Boeing tried to band-aid with software.
https://spectrum.ieee.org/how-the-boeing-737-max-disaster-lo...
And the fact that they wanted to spend nothing on the software and then rush to production (this was exposed in this article) is some cluster-b antisocial personality disorder shit...The bean counters had the stock churning at all time highs and they were going to do things their own way (trading at 450 per share shortly before these tragedies).
So it's HIGHLY unlikely that Boeings software for the 737 Max would have been operating at spec given the use-case (overcoming hardware problems with software in a way that's never been done before). MCAS was not designed to take into account that external sesors could be out of wack (which happens all the time). MCAS took liberties and had opinions that ran counter to the norms of aviation. And this is the hallmark of poor software engineers that have no domain expertise and no ability to push back against anti-social personalities masquerading as managers.
" When the two computers disagree, the solution for the humans in the cockpit is to look across the control panel to see what the other instruments are saying and then sort it out. In the Boeing system, the flight management computer does not “look across” at the other instruments. It believes only the instruments on its side. It doesn’t go old-school. It’s modern. It’s software.
This means that if a particular angle-of-attack sensor goes haywire—which happens all the time in a machine that alternates from one extreme environment to another, vibrating and shaking all the way—the flight management computer just believes it. "
If you want to have the title of SR. Engineer and lives are in the palm of your hand you better be prepared to be the bad guy and take on management when they're being driven by motivations that run counter towards the quality and the usecases for your code. VW had a similar culture and those "senior software engineers" went to prison, not the managers (remember that).
https://spectrum.ieee.org/how-the-boeing-737-max-disaster-lo...
And the fact that they wanted to spend nothing on the software and then rush to production (this was exposed in this article) is some cluster-b antisocial personality disorder shit...The bean counters had the stock churning at all time highs and they were going to do things their own way (trading at 450 per share shortly before these tragedies).
So it's HIGHLY unlikely that Boeings software for the 737 Max would have been operating at spec given the use-case (overcoming hardware problems with software in a way that's never been done before). MCAS was not designed to take into account that external sesors could be out of wack (which happens all the time). MCAS took liberties and had opinions that ran counter to the norms of aviation. And this is the hallmark of poor software engineers that have no domain expertise and no ability to push back against anti-social personalities masquerading as managers.
" When the two computers disagree, the solution for the humans in the cockpit is to look across the control panel to see what the other instruments are saying and then sort it out. In the Boeing system, the flight management computer does not “look across” at the other instruments. It believes only the instruments on its side. It doesn’t go old-school. It’s modern. It’s software.
This means that if a particular angle-of-attack sensor goes haywire—which happens all the time in a machine that alternates from one extreme environment to another, vibrating and shaking all the way—the flight management computer just believes it. "
If you want to have the title of SR. Engineer and lives are in the palm of your hand you better be prepared to be the bad guy and take on management when they're being driven by motivations that run counter towards the quality and the usecases for your code. VW had a similar culture and those "senior software engineers" went to prison, not the managers (remember that).
This. On top of the MCAS modification being necessary in the first place.
Higher torque generated because of more off-center thrusters surely raised eyebrows in the stall risk mitigation team.
In the end, that's 3 teams where management made the topic being fixed by another team. That's systemic. And with a single confounder: cost measures being prioritized over safety.
With the current QA problems, it cannot be made more clear that that single culture element is the root cause at Boeing. And ethics dictate that no more experiments be made, and the top 200 of management be forcefully removed.
Higher torque generated because of more off-center thrusters surely raised eyebrows in the stall risk mitigation team.
In the end, that's 3 teams where management made the topic being fixed by another team. That's systemic. And with a single confounder: cost measures being prioritized over safety.
With the current QA problems, it cannot be made more clear that that single culture element is the root cause at Boeing. And ethics dictate that no more experiments be made, and the top 200 of management be forcefully removed.
The other problem from what I can see was a lot of frog-boiling. MCAS was actually originally designed for a slightly different aerodynamic situation, but in flight testing, they discovered a different and more serious instability.
The engineers determined that they could use the MCAS to fix this problem too, but the amount of control input had to be about 4x as much for this situation (2.5 degrees vs. 0.6 degrees of stabilizer movement). They also made it so MCAS could activate multiple times. The original safety analysis was based on the 0.6 degrees for one shot, which wouldn't have put the plane in a situation where the pilots couldn't overpower it, like what happened in the two crashes, so the system was put in a lower safety category that didn't require the same redundancy. Dominic Gates wrote about this in [1]
Given that the plane was already built and in flight test, the production lines were ready to start, and there were massive contract penalties for either late delivery, or additional pilot training, the pressure to hack something in and ship must have been enormous.
The only place to make a change and still hit the deadline was in the software, and unfortunately even the software was limited by Boeing's redundancy strategy of having two completely independent sets of flight computers and sensors. Having one computer look at sensors from both the left and right sides compromises the concept of having them be completely independent, and so it wasn't done, even though ultimately that was what was used as the fix.
[1] https://www.seattletimes.com/business/boeing-aerospace/faile...
The engineers determined that they could use the MCAS to fix this problem too, but the amount of control input had to be about 4x as much for this situation (2.5 degrees vs. 0.6 degrees of stabilizer movement). They also made it so MCAS could activate multiple times. The original safety analysis was based on the 0.6 degrees for one shot, which wouldn't have put the plane in a situation where the pilots couldn't overpower it, like what happened in the two crashes, so the system was put in a lower safety category that didn't require the same redundancy. Dominic Gates wrote about this in [1]
Given that the plane was already built and in flight test, the production lines were ready to start, and there were massive contract penalties for either late delivery, or additional pilot training, the pressure to hack something in and ship must have been enormous.
The only place to make a change and still hit the deadline was in the software, and unfortunately even the software was limited by Boeing's redundancy strategy of having two completely independent sets of flight computers and sensors. Having one computer look at sensors from both the left and right sides compromises the concept of having them be completely independent, and so it wasn't done, even though ultimately that was what was used as the fix.
[1] https://www.seattletimes.com/business/boeing-aerospace/faile...
Required or required so pilots didn't need a new training and certification?
Apparently 1/3 of all software vulnerabilities represent design weaknesses which were introduced in the requirements phase. The MCAS flaw seems to belong to this category which you describe.
source: https://insights.sei.cmu.edu/blog/a-tool-to-address-cybersec...
source: https://insights.sei.cmu.edu/blog/a-tool-to-address-cybersec...
Separating controls from SW is a strange thing to do. The issue here is a culture of shortcuts and awful money-driven decisions, not any particular discipline, though delayed software fixes did cause more accidents, and it was poorly designed overall.
Ultimately, the MCAS relied on a single sensor, one which was known to fail, and only displayed the redundant sensor data - get this - if they bought the additional option to show when the sensor failed.
> However, whereas MCAS was activated automatically, without pilot action, the cockpit crew would have to notice and act on an AOA DISAGREE alert. Further, the AOA indicator and disagree alert were not standard equipment on the 737 MAX, although the AOA indicator had been on earlier models. Boeing offered them as “add ons” at additional cost
Without that "add on" you'd never know the MCAS was acting on faulty data. And all the while management worked overtime to mislead regulators on the potential impact of MCAS to avoid additional scrutiny and training requirements. So nobody knew the safety of their aircraft depended on that non-standard equipment package.
Yeah - total business and design mismanagement driven by greed.
---
https://mitsloan.mit.edu/teaching-resources-library/boeings-...
> When I say I changed the culture of Boeing, that was the intent, so it’s run like a business rather than a great engineering firm. It is a great engineering firm, but people invest in a company because they want to make money. --Harry Stonecipher, 2004
I think that outsourcing was a symptom of this disease. Not MBA cancer, not poor software, not anything other than a deliberate cultural shift which led to all those other things.
Ultimately, the MCAS relied on a single sensor, one which was known to fail, and only displayed the redundant sensor data - get this - if they bought the additional option to show when the sensor failed.
> However, whereas MCAS was activated automatically, without pilot action, the cockpit crew would have to notice and act on an AOA DISAGREE alert. Further, the AOA indicator and disagree alert were not standard equipment on the 737 MAX, although the AOA indicator had been on earlier models. Boeing offered them as “add ons” at additional cost
Without that "add on" you'd never know the MCAS was acting on faulty data. And all the while management worked overtime to mislead regulators on the potential impact of MCAS to avoid additional scrutiny and training requirements. So nobody knew the safety of their aircraft depended on that non-standard equipment package.
Yeah - total business and design mismanagement driven by greed.
---
https://mitsloan.mit.edu/teaching-resources-library/boeings-...
> When I say I changed the culture of Boeing, that was the intent, so it’s run like a business rather than a great engineering firm. It is a great engineering firm, but people invest in a company because they want to make money. --Harry Stonecipher, 2004
I think that outsourcing was a symptom of this disease. Not MBA cancer, not poor software, not anything other than a deliberate cultural shift which led to all those other things.
>> When I say I changed the culture of Boeing, that was the intent, so it’s run like a business rather than a great engineering firm. It is a great engineering firm, but people invest in a company because they want to make money. --Harry Stonecipher, 2004
> I think that outsourcing was a symptom of this disease. Not MBA cancer, not poor software, not anything other than a deliberate cultural shift which led to all those other things.
That is the MBA cancer. Focusing on shareholder value instead of making a great product.
> I think that outsourcing was a symptom of this disease. Not MBA cancer, not poor software, not anything other than a deliberate cultural shift which led to all those other things.
That is the MBA cancer. Focusing on shareholder value instead of making a great product.
I've come to the conclusion that just like we require licenses and certain degrees to work on safety critical fields we should extend that to things you can't have in order to work for companies that do manufacturing, starting with MBA's are legally ineligible for hire at companies like Boeing.
Yeah, I suppose like the biological cancer, there's not a clear cause effect relationship between a reshaping culture, and then MBAs or "Ship-it" mentality taking over engineering oversight and scheduling. It's a runaway effect.
I tend to place more blame on a deliberate reshaping by the top individual, rather than some accidental metastasizing of the problem. I'm straining the analogy.
I tend to place more blame on a deliberate reshaping by the top individual, rather than some accidental metastasizing of the problem. I'm straining the analogy.
I've always seen the cancer as MBAs only hire MBAs, and that's how it grows. Its very difficult to get to a senior level in most organizations without an MBA.
>> reshaping by the top individual
> MBAs only hire MBAs
Maybe the CEO in such cases often is an MBA and you are a bit talking about the same thing, just from different perspectives? (Looking at the one person, vs the people, at the top?)
https://fortune.com/education/articles/how-valuable-is-an-mb...
> MBA grads made up nearly 40% of C-suite executives on the 2022 Fortune 1000 list
Wow so many (I think), but maybe they aren't CEOs.
Here's 43% from Reddit: https://www.reddit.com/r/MBA/comments/u26w7r/from_mba_to_for...
> I have compiled the following for the 2021 Fortune 500 US companies (the last global one I've seen is from FT in 2015 https://ig.ft.com/sites/mba-to-ceo/):
> 43% of CEOs have an MBA
Anyway,what happens if the CEO is an engineer, and everyone reporting to him/her is an MBA :-)
> MBAs only hire MBAs
Maybe the CEO in such cases often is an MBA and you are a bit talking about the same thing, just from different perspectives? (Looking at the one person, vs the people, at the top?)
https://fortune.com/education/articles/how-valuable-is-an-mb...
> MBA grads made up nearly 40% of C-suite executives on the 2022 Fortune 1000 list
Wow so many (I think), but maybe they aren't CEOs.
Here's 43% from Reddit: https://www.reddit.com/r/MBA/comments/u26w7r/from_mba_to_for...
> I have compiled the following for the 2021 Fortune 500 US companies (the last global one I've seen is from FT in 2015 https://ig.ft.com/sites/mba-to-ceo/):
> 43% of CEOs have an MBA
Anyway,what happens if the CEO is an engineer, and everyone reporting to him/her is an MBA :-)
Not arguing anything in particular, just want to point out it's not completely unusual to be both. Many of the top "Chief" positions I've worked under were former PhD engineers and researchers who went and got an MBA to move up. The legendary director of JPL, Charles Elachi, for example. https://en.wikipedia.org/wiki/Charles_Elachi
And many of my former bosses.
Maybe that's a symptom of the same disease ("without mba you cannot rule"), but I think it doesn't necessarily mean an MBA holder is a bad candidate for leadership.
And many of my former bosses.
Maybe that's a symptom of the same disease ("without mba you cannot rule"), but I think it doesn't necessarily mean an MBA holder is a bad candidate for leadership.
> went and got an MBA to move up.
Interesting! (Imagine the opposite: Getting an engineering degree, to become a Fortune 500 CEO when already having an MBA. In aviation for example)
> doesn't necessarily mean an MBA holder is a bad candidate for leadership
Yeah, and that many MBAs might rather indicate that there are many who are really good at it? ... Although not the Boing minded ones apparently
Interesting! (Imagine the opposite: Getting an engineering degree, to become a Fortune 500 CEO when already having an MBA. In aviation for example)
> doesn't necessarily mean an MBA holder is a bad candidate for leadership
Yeah, and that many MBAs might rather indicate that there are many who are really good at it? ... Although not the Boing minded ones apparently
I remember reading that development of the MCAS system was not done by offshore companies, which was confirmed by Boeing. So saying that $9/hour employees caused the 737 MAX issues is incorrect.
https://www.industryweek.com/supply-chain/article/22027840/b...
Excerpts from the the above article:
>Boeing said the company did not rely on engineers from HCL and Cyient for the Maneuvering Characteristics Augmentation System, which has been linked to the Lion Air crash last October and the Ethiopian Airlines disaster in March. The Chicago-based planemaker also said it didn’t rely on either firm for another software issue disclosed after the crashes: a cockpit warning light that wasn’t working for most buyers.
>In a statement, HCL said it “has a strong and long-standing business relationship with The Boeing Company, and we take pride in the work we do for all our customers. However, HCL does not comment on specific work we do for our customers. HCL is not associated with any ongoing issues with 737 Max.”
>Based on resumes posted on social media, HCL engineers helped develop and test the Max’s flight-display software, while employees from another Indian company, Cyient Ltd., handled software for flight-test equipment.
https://www.industryweek.com/supply-chain/article/22027840/b...
Excerpts from the the above article:
>Boeing said the company did not rely on engineers from HCL and Cyient for the Maneuvering Characteristics Augmentation System, which has been linked to the Lion Air crash last October and the Ethiopian Airlines disaster in March. The Chicago-based planemaker also said it didn’t rely on either firm for another software issue disclosed after the crashes: a cockpit warning light that wasn’t working for most buyers.
>In a statement, HCL said it “has a strong and long-standing business relationship with The Boeing Company, and we take pride in the work we do for all our customers. However, HCL does not comment on specific work we do for our customers. HCL is not associated with any ongoing issues with 737 Max.”
>Based on resumes posted on social media, HCL engineers helped develop and test the Max’s flight-display software, while employees from another Indian company, Cyient Ltd., handled software for flight-test equipment.
Check out this: https://mitsloan.mit.edu/teaching-resources-library/boeings-...
It's a super fun read. And you're absolutely right: Boeing caused the problem itself. Shifting blame to poor software "accidentally" produced by 9/hr engineers is not legitimate.
There was a bug. But it was only a problem because all the other redundancies and safeguards were shortcut and regulators and pilots were mislead about the severity of MCAS system's "corrections" and not given proper data to know when to override it.
It's a super fun read. And you're absolutely right: Boeing caused the problem itself. Shifting blame to poor software "accidentally" produced by 9/hr engineers is not legitimate.
There was a bug. But it was only a problem because all the other redundancies and safeguards were shortcut and regulators and pilots were mislead about the severity of MCAS system's "corrections" and not given proper data to know when to override it.
The original article on Bloomberg with the same title - https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-...
I seriously doubt this is the mistake of $9/hr workers. Those workers are unlikely to be familiar with the whole picture and would probably just have requirements delivered to them. This is not a case of badly written code, the code didn't crash. It just didn't fit the requirements, likely because those requirements weren't known, because someone I'd wager working directly at Boeing getting a industry-typical salary didn't deliver them.
Part of good engineering is having a solid understanding of your business domain and being able to spot things that just don't smell right.
An experienced professional software engineer on an industry standard salary, having been provided with nonsensical or missing requirements, should know when to impute and when to raise an exception. I'm not at all confident that a $9/hr engineer will do that.
An experienced professional software engineer on an industry standard salary, having been provided with nonsensical or missing requirements, should know when to impute and when to raise an exception. I'm not at all confident that a $9/hr engineer will do that.
Well, we don't know if they did. We do know that Boeing employees (including one of the whistleblowers) have attempted to raise such issues internally for a long time, but mostly went ignored, moved to a department where they can't complain, or fired.
So purely on the power differential, a $9 engineer wouldn't make a stink. Says nothing about their work ethic or ability to see disaster (at least upon integrating code, probably not literal disaster) coming.
And also, how would they have known they're just using one AOA input, and a component above them isn't already delivering a value that's known good - these are not the kind of engineers that'd know that.
So purely on the power differential, a $9 engineer wouldn't make a stink. Says nothing about their work ethic or ability to see disaster (at least upon integrating code, probably not literal disaster) coming.
And also, how would they have known they're just using one AOA input, and a component above them isn't already delivering a value that's known good - these are not the kind of engineers that'd know that.
I'm not at all confident that a $9/hr engineer can do that.
You don't hire somebody for a price like that because you intend to listen to their feedback and treat them like a valued member of the team. You hire somebody for a price like that because your LLM hasn't quite reached the level you want yet...or frankly costs more to run.
You don't hire somebody for a price like that because you intend to listen to their feedback and treat them like a valued member of the team. You hire somebody for a price like that because your LLM hasn't quite reached the level you want yet...or frankly costs more to run.
Exactly. Seems just a bait article from someone who doesn't know shit about what happened (similar to Dave Farley video blaming the Cyberpunk launch fiasco on the lack of CI/CD).
So let's assume the following:
You outsource your software development to some country with quite low average monthly salary. Say some country in eastern Europe, where average salary is $500 / month.
Then you only hire from the top CS programs in the country and pay them, say, 3 times the national average - so $1500 / month. Given 150 work hours in a month, that comes out to $10/hour gross pay.
You get good engineers, pay less, and they earn more than they'd get from pretty much any domestic employer.
If the average software dev in the US makes, dunno, $70k / year, then that would be the equivalent of getting hired on a $210k / salary.
Of course, it's not all smooth sailing - but I think the important part here is to keep in mind that pay is relative. What could be a pitiful salary in rich western countries, could be a very good salary other places.
You outsource your software development to some country with quite low average monthly salary. Say some country in eastern Europe, where average salary is $500 / month.
Then you only hire from the top CS programs in the country and pay them, say, 3 times the national average - so $1500 / month. Given 150 work hours in a month, that comes out to $10/hour gross pay.
You get good engineers, pay less, and they earn more than they'd get from pretty much any domestic employer.
If the average software dev in the US makes, dunno, $70k / year, then that would be the equivalent of getting hired on a $210k / salary.
Of course, it's not all smooth sailing - but I think the important part here is to keep in mind that pay is relative. What could be a pitiful salary in rich western countries, could be a very good salary other places.
You are underestimating how difficult it is to communicate with people from different cultures.
I did an electricity project with developers from the Ukraine. We spent a week being confused because the Ukrainiens expected the electricity bill to be calculated based on the size of your house and how many animals you have. The concept of a domestic electricity meter was completely foreign to them.
I also worked with a Ukrainian graphic designer who had no understanding of copyright law.
I did an electricity project with developers from the Ukraine. We spent a week being confused because the Ukrainiens expected the electricity bill to be calculated based on the size of your house and how many animals you have. The concept of a domestic electricity meter was completely foreign to them.
I also worked with a Ukrainian graphic designer who had no understanding of copyright law.
> You are underestimating how difficult it is to communicate with people from different cultures.
US devs are highly overestimating the value of being American. They don't understand that a Ukrainian dev working at 80% performance because of cross-cultural issues for 50% salary is still a good deal. And when the company becomes international then the whole argument becomes reversed because the US devs become a minority and it's them who fails to adjust to others. Sure, it's difficult to hire a good dev from abroad, but when this gets done right, it saves a ton of money. For example, an American company can buy a smaller, healthy foreign company, and with the company, all the knowledge of operating in the local job market.
There are tons of posts on Reddit "I'm American and I graduated in Computer Science and I was promised $200k jobs but I can't get hired at all" from people who fail to understand basic economics of big companies.
US devs are highly overestimating the value of being American. They don't understand that a Ukrainian dev working at 80% performance because of cross-cultural issues for 50% salary is still a good deal. And when the company becomes international then the whole argument becomes reversed because the US devs become a minority and it's them who fails to adjust to others. Sure, it's difficult to hire a good dev from abroad, but when this gets done right, it saves a ton of money. For example, an American company can buy a smaller, healthy foreign company, and with the company, all the knowledge of operating in the local job market.
There are tons of posts on Reddit "I'm American and I graduated in Computer Science and I was promised $200k jobs but I can't get hired at all" from people who fail to understand basic economics of big companies.
Not necessarily, because the bottleneck ends up becoming the Americans once again. Because the business is here, they have to be the ones to identify and fix mistakes, as well as do culture translation. So you end up saving employees of one category but then having more of another category.
One of my professors used to work at RadioShack and they had large teams in India. The software quality was good, but the interoperability was really, really bad. Not surprising since communication was awful due to time zone differences.
She, and her team in America, spent a huge amount of time normalizing their output to the American output. And you have to go in that direction because the company is in America.
One of my professors used to work at RadioShack and they had large teams in India. The software quality was good, but the interoperability was really, really bad. Not surprising since communication was awful due to time zone differences.
She, and her team in America, spent a huge amount of time normalizing their output to the American output. And you have to go in that direction because the company is in America.
This assumes that the company doesn't ever want to grow beyond US market.
The hard part of software is the communication. The easy part is writing the code.
Outsourcing makes the hard part harder.
If you completely misunderstood what your manager wanted, would your productivity be 80% or close to 0?
Outsourcing makes the hard part harder.
If you completely misunderstood what your manager wanted, would your productivity be 80% or close to 0?
> Ukraine. We spent a week being confused because the Ukrainiens expected the electricity bill to be calculated based on the size of your house and how many animals you have. The concept of a domestic electricity meter was completely foreign to them.
What are you talking about? Private apartments and houses in Ukraine had domestic electricity meters installed since soviet times.
What are you talking about? Private apartments and houses in Ukraine had domestic electricity meters installed since soviet times.
Elites in Kiev had all kinds of fancy stuff. They’re not the ones working for 30% of a western developers wage.
> Say some country in eastern Europe, where average salary is $500 / month.
Then you only hire from the top CS programs in the country and pay them, say, 3 times the national average - so $1500 / month.
This is 10-15 years out of date. And even then it was hardly ever as straightforward as that. And only ever applied to junior and maybe mid level developers or those who couldn’t effectively communicate in English. High skilled one were significantly more mobile and were basically competing with a much more global pool of developers. They could relatively easily move (if not to the US then at least Western Europe so your potential savings were usually limited to the difference in CoL + some premium).
This led to pretty high inequality based on skill/experience e.g. to top CS graduates could expect their income to increase by 3x if not more over the next 5 years after graduation.
This is 10-15 years out of date. And even then it was hardly ever as straightforward as that. And only ever applied to junior and maybe mid level developers or those who couldn’t effectively communicate in English. High skilled one were significantly more mobile and were basically competing with a much more global pool of developers. They could relatively easily move (if not to the US then at least Western Europe so your potential savings were usually limited to the difference in CoL + some premium).
This led to pretty high inequality based on skill/experience e.g. to top CS graduates could expect their income to increase by 3x if not more over the next 5 years after graduation.
This assumes that
A) The institutions in the country make it possible to identify good from bad local candidates.
B) There are enough experienced people in the region to have a healthy functional workplace.
C) The outcome is at least proportional to the cost.
In my experience none of these are true.
A) The institutions in the country make it possible to identify good from bad local candidates.
B) There are enough experienced people in the region to have a healthy functional workplace.
C) The outcome is at least proportional to the cost.
In my experience none of these are true.
[deleted]
Average wages for workers in Eastern Europe are way higher than $500/mo, never mind software engineers. Poland is $1,400, Hungary $1,200, Bulgaria $1,100, Romania $1,600, Lithuania $2,200, Estonia $2,000, Latvia $1,700. If you assume that software engineers will earn at least double the average wages of all workers, you're looking at $2,200-4,400 as a base. Paying 3x that to get the top people would mean $6,600-13,200/mo. Ultimately, you'd be paying $44-88/hour to execute your strategy.
Countries like Bulgaria and Hungary are both in the EU allowing workers to travel to Germany, France, etc. to get jobs. They aren't trapped behind immigration laws that don't allow them to move to places with higher salaries.
Even if you look at India, $10/hour wouldn't get you the best engineers. Mediocre engineers are earning that after a few years experience. Maybe you'll say that you'll pay triple that and that $30/hour is still cheap. Sure, $54,000/year is a cheap software engineer, but I think the best engineers have better options than being seen as a cost savings for some American company.
Frankly, it isn't necessarily about the money itself. It's about the attitude. If a company sees software as a cost center to be optimized/cut, their software will be crappy. If you're just focused on cutting costs, it's not just about how much you're paying engineers per hour. It's also about whether you let your engineers work on something that makes the product better, but might not be strictly necessary. For example, maybe there's a process the customer has to go through and many do it wrong because it's confusing. Will you let them spend the time to re-write it? Or will you say that it works so you don't want to spend money on that and customers will put up with it?
In theory, you could have more engineers and allow them to do more if they were cheaper. I've found that (in practice) when companies are looking to optimize salaries like that, they're also optimizing the time spent on things as well. They don't see their software as a strategic advantage to be invested in. They just see a cost center to be cut and that won't just be salaries, but also hours worked.
It also creates a crappy atmosphere where workers often don't care about getting things done beyond what they need to keep their job and not get yelled at. You're looking to work the system for yourself and minimize costs and they'll get a similar attitude where they try to work the system and do as little work for you as possible.
I'm not saying that you couldn't create a great company in Eastern Europe or India - they exist. But I think if your attitude is one of outsourcing a cost center, you won't get what you're looking for. Pay is relative, but it's higher than you think and you don't want to fall into the trap of just thinking that software is a cost center.
Plus, you don't even have to go that far to get cheaper engineers. You can pay half of what you pay Americans in France or Germany.
Countries like Bulgaria and Hungary are both in the EU allowing workers to travel to Germany, France, etc. to get jobs. They aren't trapped behind immigration laws that don't allow them to move to places with higher salaries.
Even if you look at India, $10/hour wouldn't get you the best engineers. Mediocre engineers are earning that after a few years experience. Maybe you'll say that you'll pay triple that and that $30/hour is still cheap. Sure, $54,000/year is a cheap software engineer, but I think the best engineers have better options than being seen as a cost savings for some American company.
Frankly, it isn't necessarily about the money itself. It's about the attitude. If a company sees software as a cost center to be optimized/cut, their software will be crappy. If you're just focused on cutting costs, it's not just about how much you're paying engineers per hour. It's also about whether you let your engineers work on something that makes the product better, but might not be strictly necessary. For example, maybe there's a process the customer has to go through and many do it wrong because it's confusing. Will you let them spend the time to re-write it? Or will you say that it works so you don't want to spend money on that and customers will put up with it?
In theory, you could have more engineers and allow them to do more if they were cheaper. I've found that (in practice) when companies are looking to optimize salaries like that, they're also optimizing the time spent on things as well. They don't see their software as a strategic advantage to be invested in. They just see a cost center to be cut and that won't just be salaries, but also hours worked.
It also creates a crappy atmosphere where workers often don't care about getting things done beyond what they need to keep their job and not get yelled at. You're looking to work the system for yourself and minimize costs and they'll get a similar attitude where they try to work the system and do as little work for you as possible.
I'm not saying that you couldn't create a great company in Eastern Europe or India - they exist. But I think if your attitude is one of outsourcing a cost center, you won't get what you're looking for. Pay is relative, but it's higher than you think and you don't want to fall into the trap of just thinking that software is a cost center.
Plus, you don't even have to go that far to get cheaper engineers. You can pay half of what you pay Americans in France or Germany.
In some countries, they literally hire people off the street for coding who aren't even programmers, to sell them to suckers in rich countries overseas who will buy their services.
Any decent engineer worth their salt gets a visa to work in the said rich countries where they can earn 10x more. They don't stay in the local lower paying roles as they can't compete on salary.
You get what you pay for.
Any decent engineer worth their salt gets a visa to work in the said rich countries where they can earn 10x more. They don't stay in the local lower paying roles as they can't compete on salary.
You get what you pay for.
I forgot which book mentioned that big government contractors like Boeing had to set up their factories and give business to vendors in many, if not all, the states no matter how inefficient or expensive that is because senators wouldn't approve key contracts unless the contractors brought jobs to the senators' states.
I'm guessing the cost of the outsourcing is also part of this compromise to the senators' demands.
A trajectory question: our government also hands out billions to NGOs and contractors to "solve" social problems, yet making such problems worse. For instance, multiple articles reported that how so many organizations charged ridiculous amount of money to SF government to help the homeless, yet the city's situation only got worse over the years. Wouldn't this be a worrisome sign of a society. An empire didn't fall because of external enemies but because the empire had so many entangled interests and the society simply declined and then collapsed.
I'm guessing the cost of the outsourcing is also part of this compromise to the senators' demands.
A trajectory question: our government also hands out billions to NGOs and contractors to "solve" social problems, yet making such problems worse. For instance, multiple articles reported that how so many organizations charged ridiculous amount of money to SF government to help the homeless, yet the city's situation only got worse over the years. Wouldn't this be a worrisome sign of a society. An empire didn't fall because of external enemies but because the empire had so many entangled interests and the society simply declined and then collapsed.
I don't see how $9 is relevant. If they are based in India $9USD/hour is 755 Indian rupees. That $9 is a really good wage.
A tech worker in India supposedly makes 32,000 rupees per month. So that's 200 rupees per hour or $2.38USD.
It's relative to the region and its economy. I'd love to have my Canadian wage paid in Kuwaiti Dinars one for one.
A tech worker in India supposedly makes 32,000 rupees per month. So that's 200 rupees per hour or $2.38USD.
It's relative to the region and its economy. I'd love to have my Canadian wage paid in Kuwaiti Dinars one for one.
Without mentioning any names....
The largest company in it's industry that uses many mission critical applications systems every second of the day, is in the process of offshoring all its software development to you know where.
When a company brings in an Indian CIO, it's only a matter of time.
The largest company in it's industry that uses many mission critical applications systems every second of the day, is in the process of offshoring all its software development to you know where.
When a company brings in an Indian CIO, it's only a matter of time.
[deleted]
The author himself notes paying at least a $100K global salary as of 2019, which implies he does engage in outsourcing. As such, the article is self-contradicting. Maybe the author should go back to writing about Javascript since javscript-scene is in the URL.
You can get bad engineers anywhere, including in the US. It's up to management to enforce a sane engineering process, although being the parasites that they are, they never will.
You can get bad engineers anywhere, including in the US. It's up to management to enforce a sane engineering process, although being the parasites that they are, they never will.
[deleted]
[deleted]
The 2024 version of this is "over reliance on Copilot et al. costs various industries billions."
But also, lack of oversight, reviews, inadequate testing, etc.
I may not be $9 / hour but I still wouldn't trust myself to write aviation software. Nobody should have that kind of hubris, yet there's developers like that in all income tiers.
Trust no code, least of all your own.
I may not be $9 / hour but I still wouldn't trust myself to write aviation software. Nobody should have that kind of hubris, yet there's developers like that in all income tiers.
Trust no code, least of all your own.
Here I would question the whole process. Do these developers have even any chance to stop this process or raise up issues? Is anyone listening to them? Or will they end up being removed if they raise up something?
When writing aviation software, there is a process you need to follow, DO-178C. So even if you produce bugs, they should be caught in later stages.
Everything needs to go through a risk assessment phase.
The fact that they were determining that one AOA sensor would be enough for such a critical system is what needs to be examined.
And people involved with this decision need to be put into jail.
Everything needs to go through a risk assessment phase.
The fact that they were determining that one AOA sensor would be enough for such a critical system is what needs to be examined.
And people involved with this decision need to be put into jail.
Feels like someone completely ignores incompetence of those who wrote the requirements
But if you present your badly written requirements to a person who has the culture to ask questions, you at least have that safeguard. After all, it's a collaboration.
And when you work with an offshore team, quite often as contractors, you have the issue that you don't get top quality.
The in-between company will do its very best to hire low to maximize profit.
The good quality developers work either on local grown software, or leave the country for better opportunity.
And you are now working with a group of developers whose livelihood depends on saying 'yes' or not saying 'no.
And because of this you will not know whether there is an issue in the requirements until x time later and the plane goes down.
And when you work with an offshore team, quite often as contractors, you have the issue that you don't get top quality.
The in-between company will do its very best to hire low to maximize profit.
The good quality developers work either on local grown software, or leave the country for better opportunity.
And you are now working with a group of developers whose livelihood depends on saying 'yes' or not saying 'no.
And because of this you will not know whether there is an issue in the requirements until x time later and the plane goes down.
[deleted]
As they say, "I'm too poor to afford cheap things".
In the business world, that corresponds to "we can't afford to hire inexpensive, unqualified labor".
In the business world, that corresponds to "we can't afford to hire inexpensive, unqualified labor".
These "software engineers" also killed 189 people. I don't think Boeing losing billions is all that important in the grand scheme of things.
I really do not think the programmers are the responsible in this case. I'm not sure, but for what I understood, it was not a failure in the SW, but a failure to explain the pilots the changes in the behaviour of the plane.
On whole failure of the goal. Change a plane in big way, but try to get it act like it did not change. To avoid explaining and training pilots. So instead of training pilots to expect plane to act certain way in certain scenarios, it was instead fixed in software and hardware. But well that combination was done poorly and it should have still been explained.
They're not solely responsible, but they are responsible.
If they fulfilled the requirements provided to them I don’t see how that could be the case. Even if they didn’t it was Boeing’s job to verify that.
It’s a bit like blaming low level construction workers for a bridge that collapsed (assuming they didn’t sabotage anything on purpose).
It’s a bit like blaming low level construction workers for a bridge that collapsed (assuming they didn’t sabotage anything on purpose).
You should always question if the requirements you get make sense.
And? How would that help if they have no clue about how that specific system was supposed to work?
You find out. Doubly so if you're working on safety critical systems like an airplane
Management at Boeing killed those 346 passengers.
Shareholders of Boeing killed them in the end. They wanted these profits and stock prices, but sadly they are the group that is protected.
So at the end of the day it was you and me killing those poor souls by investing our pension money in those companies?
Don't your investors oversee cutting of corners like recycling old designs to the breaking point? Or hiding critical new workarounds in the shallowest possible training?
Exactly. Whole current financial system is build on this.
"the sole fiduciary duty is to deliver profits" is a reductive meme at this point. The problem is putting short-term over long-term, and that's a choice Boeing executives made. Of course the system incentivizes this, especially if you get away with it. But they didn't.
Most shareholders hardly have a say on how major public companies are run. Thr Only signal they have is by buying/selling stock or not even that if you only own your shares indirectly through an ETF.
If your bad code gets through to a literal airplane, then the entire QA Process is to blame, not just the software engineers/programmers.
> killed 189 people
$396 million is all that would cost. That is a fraction of the damage to shareholder value.
$396 million is all that would cost. That is a fraction of the damage to shareholder value.
This article is a click bait.
The more often capitalism strives to cost reduction and profit-maximizing at the cost of quality -- and sometimes lives --, the more often I think that companies should rely on non-voting shares far more often, with a promise to not blindly follow profits over all else.
I currently have very limited options if I want to invest my money into an (engineering) quality-first company.
I currently have very limited options if I want to invest my money into an (engineering) quality-first company.
> I think that companies should rely on non-voting shares far more often, with a promise to not blindly follow profits over all else.
sure, but how is that relevant? it's not like the previous CEO of the company was sacked by shareholders because they demanded a 0.05% increase in net revenue.
sure, but how is that relevant? it's not like the previous CEO of the company was sacked by shareholders because they demanded a 0.05% increase in net revenue.
It’s more like CEO‘s are focused on short term gains in order to get shareholders to grant them large pay packages.
> The more often capitalism strives to cost reduction and profit-maximizing at the cost of quality
Dumb capitalism only?
> Very limited options if I want to invest my money into an (engineering) quality-first company.
NVIDIA, Porsche, Tektronix...
Dumb capitalism only?
> Very limited options if I want to invest my money into an (engineering) quality-first company.
NVIDIA, Porsche, Tektronix...
NVIDIA? With the melting power connectors on their high-end cards?
> Dumb capitalism only?
More slightly-ethical, hopefully sustainable capitalism. I liked my products pre-enshittification, when they lasted 20 years and were built like tanks.
Hark on them all you want, but I‘ve been using my 2016 MacBook for 8 years without problems.
More slightly-ethical, hopefully sustainable capitalism. I liked my products pre-enshittification, when they lasted 20 years and were built like tanks.
Hark on them all you want, but I‘ve been using my 2016 MacBook for 8 years without problems.
> Dumb capitalism only?
Ah yes, we all love that famous tenet of ‘smart capitalism’: “the more you share the more your bowl will be plentiful”.
Ah yes, we all love that famous tenet of ‘smart capitalism’: “the more you share the more your bowl will be plentiful”.
more racism and hate basically.
Can you elaborate?
> I seriously doubt this is the mistake of $9/hr workers. Those workers are unlikely to be familiar with the whole picture and would probably just have requirements delivered to them. This is not a case of badly written code, the code didn't crash. It just didn't fit the requirements, likely because those requirements weren't known, because someone I'd wager working directly at Boeing getting a industry-typical salary didn't deliver them.
source: https://news.ycombinator.com/item?id=41328272
source: https://news.ycombinator.com/item?id=41328272
$9 an hour is shameful. But what's really scary is that my software has bugs in it too.
Why do you think $9 an hour is shameful? Doesn’t that entirely depend on the cost of living? $9 could be 3x the average wage in some parts of the world.
// Why do you think $9 an hour is shameful? Doesn’t that entirely depend on the cost of living? //
No. It does not depend at all on cost of living. It depends upon supply and demand, like anything else.
No. It does not depend at all on cost of living. It depends upon supply and demand, like anything else.
I’m not saying the rate depends on cost of living. I’m saying the fact that it’s “shameful” depends on cost of living.
I mean, there are a lot of things it could depend on. I'd like to say that it should depend upon the value that the programmers are creating--that more productive programmers should make more.
Unfortunately, if the article in the OP is true, these programmers were negatively productive :-( so it can't be as simple as just how much value a programmer produces. It's more complex than that.
Honestly, I've never gotten any good answer from anybody about what wages "should" be, for any job, programming or not. In reality, what happens most of the time is wages are simply determined by supply and demand, like anything else. There are plenty of people working for $9 an hour in very expensive cities in America--so whatever it depends on, it most certainly does not depend on standard of living.
Unfortunately, if the article in the OP is true, these programmers were negatively productive :-( so it can't be as simple as just how much value a programmer produces. It's more complex than that.
Honestly, I've never gotten any good answer from anybody about what wages "should" be, for any job, programming or not. In reality, what happens most of the time is wages are simply determined by supply and demand, like anything else. There are plenty of people working for $9 an hour in very expensive cities in America--so whatever it depends on, it most certainly does not depend on standard of living.
It's shameful for the quality of work asked for.
So you're not saying it's shameful in general, just in this specific context because of the type of work involved?
If you take the article at face value--if its claims are true that $9-an-hour programmers cost Boeing billions of dollars--it is indeed a shame that, in this particular instance, Boeing was not willing to pay more for better programmers.
Are those claims true? Who knows. I'm sure that whoever hired those $9-an-hour programmers thought that if they could manage those programmers soooo well that they could get them to program as well as $100 an hour programmers.
YMMV, but in my experience, you can't manage programmers to be better programmers. No matter how much check-in approval bureaucracy, or how much QA, you will not get better programs. If you want better programs, you need to hire better programmers.
Are those claims true? Who knows. I'm sure that whoever hired those $9-an-hour programmers thought that if they could manage those programmers soooo well that they could get them to program as well as $100 an hour programmers.
YMMV, but in my experience, you can't manage programmers to be better programmers. No matter how much check-in approval bureaucracy, or how much QA, you will not get better programs. If you want better programs, you need to hire better programmers.
I’d agree on that wholeheartedly. Especially the last part.
Well, for better or for worse, it is true that 9 dollar/hour or 1440 dollars/month would be a decent salary in India. Twice the current median.
I saw your other comments, so I would add that while it is a decent salary in India. That doesn't mean you can expect the same quality as you would expect from someone earning twice the median in US. Because standard of living never really scales like you want it to.
200 dollars in India still cannot buy you a cheap computer, while 1000 dollars (the equivalent in US) can definitely buy you a solid computer.
So, yeah, it is shameful for this kind of work, because it is skilled work, and skilled workers wages (because they can essentially move all around the globe) are not tied to cost of living.
I saw your other comments, so I would add that while it is a decent salary in India. That doesn't mean you can expect the same quality as you would expect from someone earning twice the median in US. Because standard of living never really scales like you want it to.
200 dollars in India still cannot buy you a cheap computer, while 1000 dollars (the equivalent in US) can definitely buy you a solid computer.
So, yeah, it is shameful for this kind of work, because it is skilled work, and skilled workers wages (because they can essentially move all around the globe) are not tied to cost of living.
I don’t have much sympathy for companies who go this path and get the results.
Staff blame non staff, passing blame to someone else that is not them and remaining employed.
In other news; dog fights cat, more on that bizarre story after the break.
In other news; dog fights cat, more on that bizarre story after the break.
[deleted]
[deleted]
api(2)
Anyway this is why professional engineering were created, to protect the public against those exact bad practices. If we were serious about fixing the issue, we have the solution.