Pull Requests Are an Anti-Pattern(ourplace.substack.com)
ourplace.substack.com
Pull Requests Are an Anti-Pattern
https://ourplace.substack.com/p/pull-requests-are-an-anti-pattern
10 comments
Possibly off-topic, but the "a bug is cheaper to fix the earlier on it's discovered" idea came from a since-debunked bit of research, and even if it were true in 1995, has since become untrue through tech advances in how we execute modern deployments, e.g. if I commit a bug in my UI, fixing it in production is the same process as fixing it in dev, just pushing a commit.
I wish I had a citation, but my google-fu is getting circumvented by "error" and "cost to fix a bug" being too related.
I wish I had a citation, but my google-fu is getting circumvented by "error" and "cost to fix a bug" being too related.
Possibly off-topic, but the "a bug is cheaper to fix the earlier on it's discovered" idea came from a since-debunked bit of research, and even if it were true in 1995, has since become untrue through tech advances in how we execute modern deployments
That seems like quite a stretch.
The cost of fixing a bug isn't just the deployment effort. It's also the investigation, the impact analysis for any potential fix, and the mitigation of any unwanted side effects on other code that is related to wherever the bug is.
I think the original point of the idea was also that a bug may also have cost real money by making it further through your pipeline or even into production before someone fixed it. A bug only found in some sort of QA or code review exercise took the resources to do that exercise, when a bug found by a developer shortly after writing the code did not. A bug that made it to a staging server might have tied up shared resources or interfered with someone else's deployment before it gets backed out.
If a bug makes it all the way into production, there might be customer relations or PR fallout for having the bug in your software and affecting your customers negatively as a result. There might be regulatory consequences if the bug caused a breach of some kind. If the bug caused something serious like personal injury, damage to equipment or loss of important data, that could be catastrophic.
Of course, a lot of software today also isn't online systems that can be readily redeployed, and it's obvious that if you have a more demanding deployment process like serving millions of software updates to unhappy customers, possibly through a significant marketplace that you don't control where you can't just push out your change whenever you want, or physically recalling a dangerous hardware product deployed halfway around the world to update its firmware, that's going to be far more expensive.
I'm not sure what since-debunked bit of research you were referring to, but the idea you mentioned is far broader than any single bit of research and I think it's still a pretty good rule of thumb even today. The relative ease of new deployments for online systems running on servers you control just reduces the damage a bit in that one specific area.
That seems like quite a stretch.
The cost of fixing a bug isn't just the deployment effort. It's also the investigation, the impact analysis for any potential fix, and the mitigation of any unwanted side effects on other code that is related to wherever the bug is.
I think the original point of the idea was also that a bug may also have cost real money by making it further through your pipeline or even into production before someone fixed it. A bug only found in some sort of QA or code review exercise took the resources to do that exercise, when a bug found by a developer shortly after writing the code did not. A bug that made it to a staging server might have tied up shared resources or interfered with someone else's deployment before it gets backed out.
If a bug makes it all the way into production, there might be customer relations or PR fallout for having the bug in your software and affecting your customers negatively as a result. There might be regulatory consequences if the bug caused a breach of some kind. If the bug caused something serious like personal injury, damage to equipment or loss of important data, that could be catastrophic.
Of course, a lot of software today also isn't online systems that can be readily redeployed, and it's obvious that if you have a more demanding deployment process like serving millions of software updates to unhappy customers, possibly through a significant marketplace that you don't control where you can't just push out your change whenever you want, or physically recalling a dangerous hardware product deployed halfway around the world to update its firmware, that's going to be far more expensive.
I'm not sure what since-debunked bit of research you were referring to, but the idea you mentioned is far broader than any single bit of research and I think it's still a pretty good rule of thumb even today. The relative ease of new deployments for online systems running on servers you control just reduces the damage a bit in that one specific area.
> The cost of fixing a bug isn't just the deployment effort. It's also the investigation, the impact analysis for any potential fix, and the mitigation of any unwanted side effects on other code that is related to wherever the bug is.
...and also the mitigation of the problems the bug has caused, e.g., to production data or business actions. This is a big driver of cost of bugs that get into production.
> I'm not sure what since-debunked bit of research you were referring to,
There was a popular 90s report that concluded a 10× cost for each step from requirements to design to testing to production; I suspect that’s the one.
...and also the mitigation of the problems the bug has caused, e.g., to production data or business actions. This is a big driver of cost of bugs that get into production.
> I'm not sure what since-debunked bit of research you were referring to,
There was a popular 90s report that concluded a 10× cost for each step from requirements to design to testing to production; I suspect that’s the one.
There was a popular 90s report that concluded a 10× cost for each step from requirements to design to testing to production; I suspect that’s the one.
That rings a bell. Possibly it was something cited in the original Code Complete? Could also have been cited by one of the consultants pushing (at the time) relatively new and unusual testing and review practices as evidence for why their advice was worthwhile. But you're right, I think the exponential cost increase argument was based on a very waterfall-y development process.
That rings a bell. Possibly it was something cited in the original Code Complete? Could also have been cited by one of the consultants pushing (at the time) relatively new and unusual testing and review practices as evidence for why their advice was worthwhile. But you're right, I think the exponential cost increase argument was based on a very waterfall-y development process.
The chart in this article is not based on any hard data (I know that offhand, it's a very popular chart), and the concepts it's trying to illustrate are baed on a study from the 80s and on languages like COBOL and FORTRAN, where compile times were very long and changes were difficult. I believe this specific study was debunked, though again I am having trouble finding anything specific on this. Apologies!
Very little in software engineering continues to reflect the environment when this idea was originally created.
The argument isn't really, "Do bugs cost more in production?", the argument is really, "Do they cost as much as this article's chart states directly?" and I believe the answer is a firm, "We don't know, but likely not, given how different from modern reality the assumptions are."
If you're operating your software team as if bugs in production cost 50x-200x more than bugs caught during development, you're probably not operating your software team with a firm anchoring in reality, considering how thin the actual evidence is to support this notion.
Very little in software engineering continues to reflect the environment when this idea was originally created.
The argument isn't really, "Do bugs cost more in production?", the argument is really, "Do they cost as much as this article's chart states directly?" and I believe the answer is a firm, "We don't know, but likely not, given how different from modern reality the assumptions are."
If you're operating your software team as if bugs in production cost 50x-200x more than bugs caught during development, you're probably not operating your software team with a firm anchoring in reality, considering how thin the actual evidence is to support this notion.
If you're operating your software team as if bugs in production cost 50x-200x more than bugs caught during development, you're probably not operating your software team with a firm anchoring in reality, considering how thin the actual evidence is to support this notion.
Again, this depends very much on what type of software you're working on. I've worked on systems where you probably need at least one extra zero, because the cost of pulling some equipment out of service once deployed could be devastating, and it wasn't the kind of equipment where you just have a couple of hot spares available in case something fails. I expect the people working on systems where really serious harm could result from a bug, the kind that can't be reversed because someone got hurt or something got physically destroyed, probably have much better war stories than me.
That said, I agree that development processes today are often very different to the ones we mostly followed a few decades ago, and I agree that dubious supporting data and quasi-statistical charts should be challenged. I haven't come across that particular chart before that I recall, and assuming it's a faithful reflection of the original source material, I'm slightly surprised that a well-known researcher like Capers Jones was behind it. It's definitely not the only argument for the basic idea that fixing bugs later can be much more expensive.
Again, this depends very much on what type of software you're working on. I've worked on systems where you probably need at least one extra zero, because the cost of pulling some equipment out of service once deployed could be devastating, and it wasn't the kind of equipment where you just have a couple of hot spares available in case something fails. I expect the people working on systems where really serious harm could result from a bug, the kind that can't be reversed because someone got hurt or something got physically destroyed, probably have much better war stories than me.
That said, I agree that development processes today are often very different to the ones we mostly followed a few decades ago, and I agree that dubious supporting data and quasi-statistical charts should be challenged. I haven't come across that particular chart before that I recall, and assuming it's a faithful reflection of the original source material, I'm slightly surprised that a well-known researcher like Capers Jones was behind it. It's definitely not the only argument for the basic idea that fixing bugs later can be much more expensive.
The problem isn't so much with PRs themselves but the journey up to the PR. If the reviewer has no context of the work, or the author has diverged too much from what was planned, things can quickly get messy with questions and knowledge sharing that should have been done before the code was written in the first place. I don't see PRs going away any time soon but they should be treated more as a part of a CI/CD pipeline and less as an actual code check.
Sounds like the problem is a unit of work is split between too many people, i.e. how the people in the example are using PRs, not PR (as a tool) itself.
My experience is that each PR belongs to one person, but multiple people often have the necessary knowledge to complete the work. Especially when a "senior" is working with a "junior" to help them complete the work. On a big codebase, it can often just be knowing-where-to-look.
If I find a line of code that confused me, finding the PR where it happened is often the solution.
That said I think GitHub's code review tools are lacking and hold us back from doing our best.
I'm building something better at codeapprove.com, if you or your team is interested please email me: [email protected]