Fixing all the bugs won't solve all the problems – Deming's path of frustration(shermanonsoftware.com)
shermanonsoftware.com
Fixing all the bugs won't solve all the problems – Deming's path of frustration
https://shermanonsoftware.com/2024/04/08/fixing-all-the-bugs-wont-solve-all-the-problems-demings-path-of-frustration/
51 comments
Nice!
It is a tragedy that programmers nowadays don't read Dijkstra nor think about what he wrote and meant. I tell people to think of him as their "Cus D'Amato" if they want to be a "Mike Tyson" in their field i.e. a man who has thought deeply about the subject, knows all the angles and can "train" one in the "correct" manner of writing programs.
It is a tragedy that programmers nowadays don't read Dijkstra nor think about what he wrote and meant. I tell people to think of him as their "Cus D'Amato" if they want to be a "Mike Tyson" in their field i.e. a man who has thought deeply about the subject, knows all the angles and can "train" one in the "correct" manner of writing programs.
Interestingly most bugs I encounter are bugs in the specification, so in some ways formal specification just moves the problem.
Formal specification means building up this specification from theory, and building proofs along the way.
Of course, there can always be errors in specification, but this is supposed to be the first place we implement SAT solvers or proof assistants. During Dijkstra's time, such technologies were not yet available as they are now.
There is work to be done until this is all practical and the overhead to do this is within the current overhead of software engineering, but we are quickly reaching that point in time.
Of course, there can always be errors in specification, but this is supposed to be the first place we implement SAT solvers or proof assistants. During Dijkstra's time, such technologies were not yet available as they are now.
There is work to be done until this is all practical and the overhead to do this is within the current overhead of software engineering, but we are quickly reaching that point in time.
> he believed in building a formal specification of software and then proving that the software matched this specification.
Given unlimited time and budget, plus the guarantee that things won‘t need to change in unexpected ways, that would be great.
Given unlimited time and budget, plus the guarantee that things won‘t need to change in unexpected ways, that would be great.
Each of these things can be managed with process. It's not an all-or-nothing strategy.
The overall system and application changes in unexpected ways, but as you get lower down the stack, the amount of churn is reduced. Hence, there are strategies to design systems and applications so the most high-assurance pieces are those least subject to change, and that the application code itself runs at the lowest assurance level.
Of course, Dijkstra was more of an academic, but this can be managed in real systems with engineering process. A system that is 20% formally verified is safer and has fewer defects than one that is 0% formally verified. The key is applying engineering to this to ensure that the time and budget spent on this specification provides the most benefit.
The overall system and application changes in unexpected ways, but as you get lower down the stack, the amount of churn is reduced. Hence, there are strategies to design systems and applications so the most high-assurance pieces are those least subject to change, and that the application code itself runs at the lowest assurance level.
Of course, Dijkstra was more of an academic, but this can be managed in real systems with engineering process. A system that is 20% formally verified is safer and has fewer defects than one that is 0% formally verified. The key is applying engineering to this to ensure that the time and budget spent on this specification provides the most benefit.
The spec does not have to be infinitely precise and can exist under budget constraints.
Reminds me of McConnell’s Software Quality at Top Speed essay, from almost 30 years ago[0]. This chart, in particular[1].
These issues are fundamental, and persist, regardless of the technology, or state of the art.
plus ça change
[0] https://stevemcconnell.com/articles/software-quality-at-top-...
[1] https://stevemcconnell.com/wp-content/uploads/2017/08/art04-...
These issues are fundamental, and persist, regardless of the technology, or state of the art.
plus ça change
[0] https://stevemcconnell.com/articles/software-quality-at-top-...
[1] https://stevemcconnell.com/wp-content/uploads/2017/08/art04-...
There is always a tradeoff between different types of use cases. Some users solve problem A with the sofware. An other group solves problem B. Using the same software in both cases leads to edge cases that are bugs for one group but cannot be fixed without introducing bugs for the second group.
The clear specification really is the solution to resolve this issue. That you don't declare behavior as buggy that is according to specification.
The clear specification really is the solution to resolve this issue. That you don't declare behavior as buggy that is according to specification.
You may hit a point where the software can't balance the needs of problem A and problem B users. Then, the solution is to split the product into 2 distinct ones. Or to exit the market for problem A in order to focus on delivering a solid solution for problem B.
Well yes. Define bugs to be application issues only and not devops issues and you can only go so far by tackling the app bugs. The end user expedience depends on all elements of the system: including app, devops and human interactions!
Yeah, the article also transitions from quality to bugs without explaining why quality is the lack of bugs instead of within bounds loading times, a good UI/UX, and relevant features.
Fixing bugs also seems to be limited to try catching all bugs instead of making changes to have the program do the correct thing like only loading the necessary data.
Fixing bugs also seems to be limited to try catching all bugs instead of making changes to have the program do the correct thing like only loading the necessary data.
I had the exact same thought, this was a very weird article.
When Frank Marshall was hired as VP of engineering at Cisco in 1992, he assessed after three months that we had a software quality problem and forbade any software engineering activity except "Fixing bugs." This went on for perhaps six months. It allowed key competitors to get ahead of us because we could not address some serious architectural issues in the code base because that would be writing new code, not fixing bugs.
Gerald Weinberg talks about "Software Stalins," managers who grab onto one indicator and think that driving it to zero (or 100, or 11 for you Spinal Tap Fans) will resolve all other problems.
If only things were so simple.
Gerald Weinberg talks about "Software Stalins," managers who grab onto one indicator and think that driving it to zero (or 100, or 11 for you Spinal Tap Fans) will resolve all other problems.
If only things were so simple.
This is a really interesting anecdote because (obviously) Cisco fared quite well on net over the last severl decades, is largely respected as a reliable vendor for critical infrastructure, and doesn't evidence even a hiccup in market assessments of its health during that time.
https://www.macrotrends.net/stocks/charts/CSCO/cisco/stock-p...
I don't mean to take away the truth of your experience or dismiss that you might better know the disconnect between what you describe and what outsiders might see in hindsight -- it's just the disconnect itself that's fascinating.
> If only things were so simple.
Indeed!
https://www.macrotrends.net/stocks/charts/CSCO/cisco/stock-p...
I don't mean to take away the truth of your experience or dismiss that you might better know the disconnect between what you describe and what outsiders might see in hindsight -- it's just the disconnect itself that's fascinating.
> If only things were so simple.
Indeed!
The renewed growth and increasing customer satisfaction--and our long-term success--were due to engineering abandoning a poorly thought-out policy before it crippled us instead of merely hampering us.
Part of the challenge in a "freeze spec" approach implicit in "only fix bugs" is that it assumes the market requirements are well established. That was definitely not the case for Internet protocols/functionality in 1992. HTTP traffic did not exist yet (along with a couple of dozen other things that would quickly become dominant by 1997.
The effect of the bug fix stall was to give our more nimble competitors more growth and more runway. The fact that Cisco did very well was despite the "just fix bugs" mandate.
I offered the story because I thought some folks reading the linked post would find it unbelievable that a company would decide to fixate on a single simple metric when operating in a complex and evolving environment.
Part of the challenge in a "freeze spec" approach implicit in "only fix bugs" is that it assumes the market requirements are well established. That was definitely not the case for Internet protocols/functionality in 1992. HTTP traffic did not exist yet (along with a couple of dozen other things that would quickly become dominant by 1997.
The effect of the bug fix stall was to give our more nimble competitors more growth and more runway. The fact that Cisco did very well was despite the "just fix bugs" mandate.
I offered the story because I thought some folks reading the linked post would find it unbelievable that a company would decide to fixate on a single simple metric when operating in a complex and evolving environment.
You have to do both, on different branches. Rearchitect things in one branch, fix nothing but bugs in another. Forward port all the bugfixes you can from one to the other.
Though: version control ... 1992 ... haha.
Hey, that's when that unwieldy behemoth ClearCase was released.
Though: version control ... 1992 ... haha.
Hey, that's when that unwieldy behemoth ClearCase was released.
I think that approach would be doomed even in the present day. Multiple "legacy" release branches is very hard to manage. Only the very largest such as Microsoft can do it, and they clearly hate doing so and want to push to rolling releases like everyone else.
Red Hat / RHEL does this on all EUS release every day, on RHEL 6,7,8,9.
Tell me more about how you would subscribe to a MS "bugfix only channel of updates" ? This is my first time hearing about that.
Tell me more about how you would subscribe to a MS "bugfix only channel of updates" ? This is my first time hearing about that.
> bugfix only channel of updates
Isn't that what's meant by LTS?
Isn't that what's meant by LTS?
[deleted]
Do you just 'install' the msi, and then thats LTS ?
> our more nimble competitors
Who was competing with Cisco in 1992? I struggle to remember. They were so dominant in the "dotcom" era (late 90s) that it felt like they were the only choice, the "IBM" of enterprise networking hardware.
Who was competing with Cisco in 1992? I struggle to remember. They were so dominant in the "dotcom" era (late 90s) that it felt like they were the only choice, the "IBM" of enterprise networking hardware.
> Cisco fared quite well on net...
Incompetent management is something of the norm in software engineering. Large successful companies have a deepish management hierarchy (4+ levels) and the competence of the org is capped by the least competent manager in the chain - who tends to be nontechnical and confused. Small companies have the ability to be more competent, but correspondingly tend not to have the resources to be influential players in the market.
I would expect that typical - maybe even above average - performance by software companies is accompanied by some breathtaking stories of management failure. If it were possible for a company to not make any mistakes in its software development it'd probably be a never-before-seen fountain of wealth creation.
Incompetent management is something of the norm in software engineering. Large successful companies have a deepish management hierarchy (4+ levels) and the competence of the org is capped by the least competent manager in the chain - who tends to be nontechnical and confused. Small companies have the ability to be more competent, but correspondingly tend not to have the resources to be influential players in the market.
I would expect that typical - maybe even above average - performance by software companies is accompanied by some breathtaking stories of management failure. If it were possible for a company to not make any mistakes in its software development it'd probably be a never-before-seen fountain of wealth creation.
The lack of nuance to consider exceptions appears to have been the problem.
Perhaps a lot of products needed bug fix campaigns because most corporations tend to reward SWE productivity based on so-called "evidence" like:
- KLoC
- # of commits
- adding features
- so-called "impact"
, rather than:
- reducing the cost to maintain code
- making code more robust
- making features actually work as users expect
- rip-replacing what shouldn't be salvaged because it's too expensive
So instead of adopting a funny mustache and making inflexible commandments, it would be less disruptive to have targeted bug fix campaigns where needed to boost nonfunctional requirements like quality, reducing support costs, and improving user satisfaction to their desired target levels.
Perhaps a lot of products needed bug fix campaigns because most corporations tend to reward SWE productivity based on so-called "evidence" like:
- KLoC
- # of commits
- adding features
- so-called "impact"
, rather than:
- reducing the cost to maintain code
- making code more robust
- making features actually work as users expect
- rip-replacing what shouldn't be salvaged because it's too expensive
So instead of adopting a funny mustache and making inflexible commandments, it would be less disruptive to have targeted bug fix campaigns where needed to boost nonfunctional requirements like quality, reducing support costs, and improving user satisfaction to their desired target levels.
It's too bad that Microsoft didn't take this same approach, and stick with it for a long time.
I'm wondering what things are like in the alternate universe where OS/2 became the dominant PC OS instead of Windows.
I'm wondering what things are like in the alternate universe where OS/2 became the dominant PC OS instead of Windows.
Architectural problems are bugs. Hopefully, most companies have VP of engineering who understand that.
Disagree. Architecture problems are different enough from software bugs that it's worth it to keep them separate words.
If bad architecture makes the system fall over, that's a bug. It's not black and white.
> "Software Stalins," managers who grab onto one indicator and think that driving it to zero (or 100, or 11 for you Spinal Tap Fans) will resolve all other problems.
Paul O’Neill famously did exactly this at Alcoa starting in 1987, focusing solely on worker safety resolved many other problems and multiplied profitability.
>The company's market value increased from $3 billion in 1986 to $27.53 billion in 2000, while net income increased from $200 million to $1.484 billion.
[0] https://www.forbes.com/sites/roddwagner/2019/01/22/have-we-l...
Paul O’Neill famously did exactly this at Alcoa starting in 1987, focusing solely on worker safety resolved many other problems and multiplied profitability.
>The company's market value increased from $3 billion in 1986 to $27.53 billion in 2000, while net income increased from $200 million to $1.484 billion.
[0] https://www.forbes.com/sites/roddwagner/2019/01/22/have-we-l...
It's a reasonable counter-example, two other CEOs of successful firms who emphasized safety were Lawrence Culp at Danaher (he is now at GE) and David Cote at Honeywell. All three had tenures more than a decade where their firms invested in and introduced new products, so I suspect they focused on more than worker safety but I am not personally familiar with their policies and decisions.
I think it makes a very good story that a focus on worker safety--to the exclusion of any other objectives--is all that you need. But I don't think anything is that simple.
I think it makes a very good story that a focus on worker safety--to the exclusion of any other objectives--is all that you need. But I don't think anything is that simple.
The biggest bug is the thinking that your proverbial calendar app is the next billion dollar application but in reality no one gives a wink. That's the root of all problems
Cisco ultimately met with a fair amount of success in the market. Their Internet Operating System was certainly a 10B application, perhaps worth 100B.
I just like fixing bugs.
If you have someone in your team like that, give them latitude.
If you have someone in your team like that, give them latitude.
That’s me, I love fixing software. Just give me a list of bug tickets to go through and I’m happy for some time. I think we deserve a cool title, Senior Bug Hunter would be pretty neat on a resume :)
Don't understand why this can't be called a bug
> The software runs slowly because large amounts of unnecessary data are being sent to the users.
> The software runs slowly because large amounts of unnecessary data are being sent to the users.
Depends if it's a consequence of intentional design or not. It's useful to have a separate designation for accidents of implementation.
"Unnecessary" was the word in the blog post. That word implies it can be just straight up fixed.
Not necessarily. "Unnecessary" to the client doesn't mean that the server can know that ahead of sending it. A redesign can be the only way to get out of that.
I recently witnessed an argument between a dev manager and the devops team lead.
The manager manager was convinced that fixing all of the bugs would lead to perfect, 100% successful execution. The devops team lead spent an hour trying to explain why the software couldn’t achieve better reliability than AWS (the system spread across AWS zones, but each instance was contained within a single zone).
The manager manager was convinced that fixing all of the bugs would lead to perfect, 100% successful execution. The devops team lead spent an hour trying to explain why the software couldn’t achieve better reliability than AWS (the system spread across AWS zones, but each instance was contained within a single zone).
100% successful execution is possible, assuming
1. You can guarantee no bugs in your system
2. You can guarantee no bugs in AWS's systems
3. You can guarantee there's no nuclear attacks on us-east-1 within the next year or so
1. You can guarantee no bugs in your system
2. You can guarantee no bugs in AWS's systems
3. You can guarantee there's no nuclear attacks on us-east-1 within the next year or so
You've entirely forgotten routine hardware failure. At large enough scale all sorts of weird stuff happens.
This sounds like a possible explanation for why I keep getting recommended different AWS client libraries from the boss to "solve instability" of a service that works 99.8% except for when the S3 endpoint (backblaze) has brief mystery issues.
I think the lesson from the article is that we should be things about issues like slow loading speed, overly chunky page weight, lag etc as bugs. The reason so many teams don't fix these problems is because they frame them as business problems instead of tech problems. We've all seen a team lament "our app would be fast if only we were given time to fix it!" By framing loading time delays as a bug, you're much more likely to get that time.
Also, even if you just look at bugs, they will level off at some equilibrium level if the software is being changed (beyond fixing bugs), since change introduces bugs.
Design (and specification) flaws are harder to fix than regular bugs.
It's not really bugs vs design issues.
It's more changeable vs unchangeable. Part of that is technical cost, but in my experience most is perceived risk, which is largely a function of understanding and team (leader) confidence.
The scariest things are existential: telling people the system they believe in has fatal flaws introduces a gap that can only be filled with a lot of confidence, which is typically lacking when growing fast or with new systems.
The main remedy is to be sure you have an actual view and understanding of the system, and address things from that perspective.
Otherwise, bugs are just make-work.
It's more changeable vs unchangeable. Part of that is technical cost, but in my experience most is perceived risk, which is largely a function of understanding and team (leader) confidence.
The scariest things are existential: telling people the system they believe in has fatal flaws introduces a gap that can only be filled with a lot of confidence, which is typically lacking when growing fast or with new systems.
The main remedy is to be sure you have an actual view and understanding of the system, and address things from that perspective.
Otherwise, bugs are just make-work.
[deleted]
> The software runs slowly because large amounts of unnecessary data are being sent to the users
/s/to/from
/s/to/from
https://www.cs.utexas.edu/users/EWD/transcriptions/EWD03xx/E...
https://www.cs.utexas.edu/~EWD/transcriptions/EWD10xx/EWD103...
There are several takeaways here, but three of my favorite highlights:
* Dijkstra actually did not like calling such errors "bugs", as it is a framing problem.
* Likewise, he believed that we should avoid anthropomorphizing software or identifying with it.
* Finally, he believed in building a formal specification of software and then proving that the software matched this specification. Part of this was to avoid "bug fixing" hunts in which the focus was on "debugging" instead of correctness. But also, this was to ensure that there was a proper system view of the software, which ties back into the conclusion of the article here.