NASA inspector general gives damning assessment of Boeing's quality control(nbcnews.com)
nbcnews.com
NASA inspector general gives damning assessment of Boeing's quality control
https://www.nbcnews.com/science/space/nasa-report-boeing-damning-assessments-rcna165813
61 comments
My first job was at a large defense company and let me tell you why Boeing's stock never drops- they can't lose money. The contracts are already signed. The money will be paid regardless. There is no consequence to failure because there are no true competitors when contracts are re-bid. I was baffled at how my employer could possibly be making money given the sheer amount of waste I observed on any given day, but as I got more involved in projects I started doing estimates and saw how much money we were making and I realized that we billed by the hour with a hefty profit margin built in and that by intentionally making the business inefficient and ineffective we could just bill more hours, and because we had bought out anyone else with experience in the niche industry we were in, there wasn't much anyone could do about it.
It was really discouraging to me and as a result I have sworn off doing any kind of government sector work again because it was just depressing.
It was really discouraging to me and as a result I have sworn off doing any kind of government sector work again because it was just depressing.
Thanks for sharing. I agree it is pretty depressing for any engineer that worths its salt. That's probably the reason many others quitted BA.
The government bailed out GM in 2009 after it went bankrupt. A Harris administration will likely follow that playbook due to its success and the overlap in personnel.
Yes, the government will bail out Boeing if necessary; Boeing will continue to build planes, most Boeing jobs and plants are safe.
However stockholders and executives are not safe. The government may demand bankruptcy for its bailout.
For more, read about the difference between chapter 11 and chapter 7 bankruptcy.
Yes, the government will bail out Boeing if necessary; Boeing will continue to build planes, most Boeing jobs and plants are safe.
However stockholders and executives are not safe. The government may demand bankruptcy for its bailout.
For more, read about the difference between chapter 11 and chapter 7 bankruptcy.
Because sophisticated investors know that the Black Projects and revenue are not affected. Because the command information equipment sold to the navy is not affected.
I think the concern for Boeing is being broken up rather than going bankrupt.
People forget the current situation is both a recent development and one that happened under unique circumstances. In the 80s and 90s there was a 1-2 whammy neo-liberal fiscal policy which basically abandoned all opposition to acquisitions, and the end of the cold war which made the government think it didn't need a robust aerospace/defense sector any more. The environment has changed considerably in the intervening decades.
People forget the current situation is both a recent development and one that happened under unique circumstances. In the 80s and 90s there was a 1-2 whammy neo-liberal fiscal policy which basically abandoned all opposition to acquisitions, and the end of the cold war which made the government think it didn't need a robust aerospace/defense sector any more. The environment has changed considerably in the intervening decades.
"the workers on the project are not, as a whole, sufficiently experienced or well trained"
Plenty of executives pretend that if you throw enough procedures and micromanagement at cheap entry-level employees then they can produce similar stuff as experts, and this seems to be evidence Boeing is trying and failing to do just that. Any executive can put procedures in place, and good training can do great things, but it's hard to say you care about quality then slash pay for good labor. Take this line from the actual report:
"Michoud officials stated that it has been difficult to attract and retain a contractor workforce with aerospace manufacturing experience in part due to Michoud’s geographical location in New Orleans, Louisiana, and lower employee compensation relative to other aerospace competitors... it is too early to determine if the new training alone will result in a notable decrease in nonconformances and CARs issued. According to a NASA official, further quality assurance challenges related to the workforce stem from work instructions that lack explicit details on how to perform the task and with what tools. Some technicians reported they had to hunt through layers of documentation to identify required instructions"
Plenty of executives pretend that if you throw enough procedures and micromanagement at cheap entry-level employees then they can produce similar stuff as experts, and this seems to be evidence Boeing is trying and failing to do just that. Any executive can put procedures in place, and good training can do great things, but it's hard to say you care about quality then slash pay for good labor. Take this line from the actual report:
"Michoud officials stated that it has been difficult to attract and retain a contractor workforce with aerospace manufacturing experience in part due to Michoud’s geographical location in New Orleans, Louisiana, and lower employee compensation relative to other aerospace competitors... it is too early to determine if the new training alone will result in a notable decrease in nonconformances and CARs issued. According to a NASA official, further quality assurance challenges related to the workforce stem from work instructions that lack explicit details on how to perform the task and with what tools. Some technicians reported they had to hunt through layers of documentation to identify required instructions"
The report included several recommendations, including levying “financial penalties for Boeing’s noncompliance with quality control standards.” The inspector general said, however, that NASA decided not to introduce any kind of financial discipline.
userbinator(1)
Always fun to release this kind of a report just after you strand astronauts in space :(
Not to worry, they're coming home. On a SpaceX capsule.
They're still stranded in the conventional sense of the term. Like when your car breaks down on the side of the road, you're stranded. You will make it home in all but the most extreme circumstances; a friend or tow truck or uber driver or emergency personnel will pick you up eventually but with your primary means of conveyance out of action, you are stranded until some Plan B can be put into action.
Guess there is some big risk ahead so they want to at least push the blame away a bit.
I'd really love to see someone compare this to Apollo era hardware, I know there were significant defects that required rework during the Apollo era - this is an organizational competency issue within Boeing, NASA and the United States, we've lost the organization competencies to organize projects like this - that doesn't mean they're gone forever, just that those skills must be rebuilt.
I also feel like some of this, being starliner or SLS issues are shades of the "good, fast, cheap, pick two" problem.
I also feel like some of this, being starliner or SLS issues are shades of the "good, fast, cheap, pick two" problem.
They built comparatively poor QA into the design of the Saturn V. I've seen very revealing pictures of the bottom of the engine, where the exhaust leaves the combustion chamber on an F-1 engine. There was this kind of symmetrical bunch of holes, and one slightly dug in hole, maybe a quarter of an inch thick. A mistake. They had started drilling in the wrong spot and quickly stopped. That was a deviance, it was logged, but it was within spec.
That's Saturn V in a nutshell.
That's Saturn V in a nutshell.
Go read “The Right Stuff.” Not quite Apollo, but we almost lost Neil Armstrong before he got a chance to step onto the moon due to faulty attitude control. Humans were a lot more disposable then.
Yeah, I have, I've also read a couple retrospectives of Apollo by NASA - and some retrospectives of Shuttle too.
Key difference though, the corners Boeing cuts are to line the pockets of their execs and stockholders.
One thing to be pushing boundaries, doing something never done before, everyone knows there are significant risks. Another altogether to be doing a terrible job to save money doing something that has been done successfully time and time again.
One thing to be pushing boundaries, doing something never done before, everyone knows there are significant risks. Another altogether to be doing a terrible job to save money doing something that has been done successfully time and time again.
You act like this is different now than it was with NAA or Grumman in 1965, it's not. The nature of companies has not changed that much since 1965.
But many of the same issues still exist, poorly defined requirements being the primary one.
But many of the same issues still exist, poorly defined requirements being the primary one.
> The nature of companies has not changed that much since 1965.
The nature of management has changed since the 1970s:
* https://en.wikipedia.org/wiki/Friedman_doctrine
* https://en.wikipedia.org/wiki/Stakeholder_theory
The nature of management has changed since the 1970s:
* https://en.wikipedia.org/wiki/Friedman_doctrine
* https://en.wikipedia.org/wiki/Stakeholder_theory
The MBA-brained focus on short term profits is a relatively recent development and was not nearly as prevalent back then.
Safety engineering is about as long-term a value as you can imagine, so it makes sense that it's one of the things being cut as companies move to focus almost exclusively on quarterly reports.
Safety engineering is about as long-term a value as you can imagine, so it makes sense that it's one of the things being cut as companies move to focus almost exclusively on quarterly reports.
>> focus on short term profits is a relatively recent development and was not nearly as prevalent back then
That’s not true, nasa engaged in studies of its suppliers in the 60s to determine if their short term profit seeking was working well enough that it might encourage further capital investments from the private sector in supply chain companies, nasa wanted to make sure there was enough fat in what it paid suppliers to attract further investment but not so much to attract public scandal as to the disposal of public tax dollars.
That’s not true, nasa engaged in studies of its suppliers in the 60s to determine if their short term profit seeking was working well enough that it might encourage further capital investments from the private sector in supply chain companies, nasa wanted to make sure there was enough fat in what it paid suppliers to attract further investment but not so much to attract public scandal as to the disposal of public tax dollars.
> That’s not true, nasa engaged in studies of its suppliers in the 60s […]
Are these available online anywhere? Curious to know how they calculated (?) such things.
Are these available online anywhere? Curious to know how they calculated (?) such things.
Note, it took 40 years to render jet flight safe...am I joking no...go read right stuff....
two issues for QA of tech hardware:
1. How new is it? 2. how many tests on the new areas...
Or just for fun read the books on Russian engines in their own moon program....especially the post on the accidents.
two issues for QA of tech hardware:
1. How new is it? 2. how many tests on the new areas...
Or just for fun read the books on Russian engines in their own moon program....especially the post on the accidents.
400K people worked on the Apollo era missions. And they literally had unlimited money from the boom time post war American economy.
Things are different.
Wonder who's paying for the cost overruns? Possibly this is WAD...
> Wonder who's paying for the cost overruns?
Boeing
A compelling theory about Boeing's lack of competence wrt Starliner is that they are institutionally unable to operate under a fixed-price regime.
Boeing
A compelling theory about Boeing's lack of competence wrt Starliner is that they are institutionally unable to operate under a fixed-price regime.
Boeing for now. If Boeing keeps screwing the pooch though... who here doesn't think they'll be bailed out one way or the other?
> who here doesn't think they'll be bailed out
One can hope that, if they're "bailed out," that it comes with mandates for quality control that, if violated, result in a corporate death penalty and temporary nationalization.
One can hope that, if they're "bailed out," that it comes with mandates for quality control that, if violated, result in a corporate death penalty and temporary nationalization.
This gao report is about EUS, which isn't a fixed price contract, so really it is the taxpayer.
Sadly, this is a problem that is permeating the aerospace industry. Maybe SpaceX has been immune, at least for now.
My first-hand experience at 2 large aerospace contractors is that QA resources have been a smaller and smaller portion of the contract budget.
The last companies I worked for had zero software QA. After two years of scraping together what I could, I eventually gave up.
My first-hand experience at 2 large aerospace contractors is that QA resources have been a smaller and smaller portion of the contract budget.
The last companies I worked for had zero software QA. After two years of scraping together what I could, I eventually gave up.
In my experience, I find aerospace quality control systems are large behemoths in and of themselves and poorly suited to software development. It's disappointing but not surprising that large orgs cut corners on software quality processes.
It can't possibly be due to the fact that the Vice President is the head of the National Space Council and all of this could be easily politicized - could it? Loss of the craft and/or loss of the crew would be a huge propaganda win for Republicans - much better to try to push this down the road and make sure everyone knows it's not NASA's fault.
Ironically it's an echo of the push to launch Challenger so President Regan could highlight the "teacher in space" in his SOTU scheduled for that night....
You honestly believe this whole document was prepared in such a short amount of time?
[deleted]
Boeing is really earning the "Shartliner" monicker.
[deleted]
Dissimilar redundancy only provides safety if you are willing to use it by disabling the option that proves unsafe. The decision to put a crew on Starliner shows that NASA has completely missed the point of contracting multiple providers for ISS access.
Columbia killed people because NASA saw the flaking foam but kept flying the shuttle because they didn't have another option for space access. Afterwards, Commercial Crew was written up with multiple providers to keep this from happening again.
Imagine an alternative timeline where NASA learned that relying on one launcher was dangerous 20 years earlier from Challenger. In that world, apparently NASA would see the foam issues and still sent up a crew on Columbia's last flight to preserve 'dissimilar redundancy.' Nevermind that this was a policy instituted _specifically_ to make it politically easy to ground unsafe vehicles. Utter insanity.
Columbia killed people because NASA saw the flaking foam but kept flying the shuttle because they didn't have another option for space access. Afterwards, Commercial Crew was written up with multiple providers to keep this from happening again.
Imagine an alternative timeline where NASA learned that relying on one launcher was dangerous 20 years earlier from Challenger. In that world, apparently NASA would see the foam issues and still sent up a crew on Columbia's last flight to preserve 'dissimilar redundancy.' Nevermind that this was a policy instituted _specifically_ to make it politically easy to ground unsafe vehicles. Utter insanity.
When you look at outcomes like this and start asking why or how this happens, you can often reach one of these conclusions:
1. Everything is fine. This is working as intended. The system (competition in this case) will "solve" this problem, if there is even a systemic problem; or
2. Therea are a few bad apples but the system is fine; or
3. The system is fundamentally flawed.
When it comes to capitalism, most people fall under (1) or (2). "The markets will solve it" is (1). "We need more or strong regulation" is (2). Even "we need less regulation" is also (2).
We can talk about how the finance people took over Boeing from the engineers and that's where these problems began. I agree with that. However, my view is that putting Boeing back in the hands of engineers won't solve this problem because it's a systemic issue. Financialization, profit-seeking and the lack of accountability at every level of Boeing is (IMHO* inevitable.
Boeing is really two companies right now.
The first is the part that makes commercial airplanes. It's a company that's really still coasting off the success and the ingrained culture from the hugely successful (but now reitred) 747. There are problems already evident however. We've seen it with the 737MAX and even the delays and cost overruns from the 787.
The second face of Boeing is the government and military contractor, which is where Starliner falls. It's the same basket as Lockheed Martin and Northrop Grumman. These companies sole purpose in existence is to extract wealth from the government and give it to shareholders. It is transferring all of the profits with none of the risk.
The last century really saw this idea take off of "small government" and outsourcing and "public-private partnerships". All of it is simply wealth extraction.
I would honestly be shocked at this point if Boeing was able to turn things around. The problems permeate every pore of the company and we live in a system where that outcome is arguably inevitable.
1. Everything is fine. This is working as intended. The system (competition in this case) will "solve" this problem, if there is even a systemic problem; or
2. Therea are a few bad apples but the system is fine; or
3. The system is fundamentally flawed.
When it comes to capitalism, most people fall under (1) or (2). "The markets will solve it" is (1). "We need more or strong regulation" is (2). Even "we need less regulation" is also (2).
We can talk about how the finance people took over Boeing from the engineers and that's where these problems began. I agree with that. However, my view is that putting Boeing back in the hands of engineers won't solve this problem because it's a systemic issue. Financialization, profit-seeking and the lack of accountability at every level of Boeing is (IMHO* inevitable.
Boeing is really two companies right now.
The first is the part that makes commercial airplanes. It's a company that's really still coasting off the success and the ingrained culture from the hugely successful (but now reitred) 747. There are problems already evident however. We've seen it with the 737MAX and even the delays and cost overruns from the 787.
The second face of Boeing is the government and military contractor, which is where Starliner falls. It's the same basket as Lockheed Martin and Northrop Grumman. These companies sole purpose in existence is to extract wealth from the government and give it to shareholders. It is transferring all of the profits with none of the risk.
The last century really saw this idea take off of "small government" and outsourcing and "public-private partnerships". All of it is simply wealth extraction.
I would honestly be shocked at this point if Boeing was able to turn things around. The problems permeate every pore of the company and we live in a system where that outcome is arguably inevitable.
I work for a competitor to Boeing as a senior integration and test engineer as one of several final approvers of flight assemblies before they leave for final integration and packaging prior to flight and I can’t tighten a bolt without scanning the QR code on the handle of the wrench to log its use or reposition a benchtop power supply four inches to the left to make more space without requesting a variance.
If a board makes it from SMT to testing, or from testing to integration, or from anywhere to anywhere else with even a fingerprint on it a nonconformance is logged and it gets kicked back.
That’s like 9100D 101.
The thought of stamping or initialing something I am unsure of is a literal impossibility and the report says it is happening all of the god damned time at Boeing.
I cannot for the life of me understand what’s going on over there.
If a board makes it from SMT to testing, or from testing to integration, or from anywhere to anywhere else with even a fingerprint on it a nonconformance is logged and it gets kicked back.
That’s like 9100D 101.
The thought of stamping or initialing something I am unsure of is a literal impossibility and the report says it is happening all of the god damned time at Boeing.
I cannot for the life of me understand what’s going on over there.
>I cannot for the life of me understand what’s going on over there.
Allow me to speculate, off the record (observing through spouse), of a soon-to-be lawsuit which will be used to remove the most-dangerous company from an undisclosed industry (less than a few ten-thousand customers):
It's about money, corporate takeover, money, "efficiency," money, green, money, safety third, money, because fuck them that's why, money, $$$, money...
This is my uneducated third-party opinion — I have no personal stake in my above observations.
Allow me to speculate, off the record (observing through spouse), of a soon-to-be lawsuit which will be used to remove the most-dangerous company from an undisclosed industry (less than a few ten-thousand customers):
It's about money, corporate takeover, money, "efficiency," money, green, money, safety third, money, because fuck them that's why, money, $$$, money...
This is my uneducated third-party opinion — I have no personal stake in my above observations.
>it's about money, corporate takeover, money, "efficiency," money, green, money, safety third, money, because fuck them that's why, money, $$$, money...
let me rephrase a bit. It is all about shareholders and how to make more money for them. Every single quarter. and if the CEO is not aligned with these values she will be replaced by new one. Not all CEOs are greedy bastards but the whole system is designed to eliminate every single person who cares about something else except short term returns for the shareholders. So good CEOs are out and bad CEOs are in. Survival of the fittest as it is.
let me rephrase a bit. It is all about shareholders and how to make more money for them. Every single quarter. and if the CEO is not aligned with these values she will be replaced by new one. Not all CEOs are greedy bastards but the whole system is designed to eliminate every single person who cares about something else except short term returns for the shareholders. So good CEOs are out and bad CEOs are in. Survival of the fittest as it is.
I say this is also understandable because 1) The government is probably going to bail BA out for the next 10 mess-ups; and 2) The stock market, since 2010, and perhaps further before, has been a money game, mostly dictated by monetary policies instead of corporate performance.