Why Most Unit Testing Is Waste (2014) [pdf](wikileaks.org)
wikileaks.org
Why Most Unit Testing Is Waste (2014) [pdf]
https://wikileaks.org/ciav7p1/cms/files/Why-Most-Unit-Testing-is-Waste.pdf
27 comments
This paper has been public for years at http://rbcs-us.com/documents/Why-Most-Unit-Testing-is-Waste....
Previous discussion 6mos ago: https://news.ycombinator.com/item?id=12666454
There have been a ton. For more, click the "past" link.
[deleted]
> If you want to reduce your test mass, the number one
thing you should do is look at the tests that have never
failed in a year and consider throwing them away
And that's how you get regressions. I get the rationale that these tests cover code which probably never changes, and tests are most useful for code that changes a lot. But if the guy who wrote the code left the company a year ago and then you have to change this code... it's going to be a lot easier if there are good tests.
And that's how you get regressions. I get the rationale that these tests cover code which probably never changes, and tests are most useful for code that changes a lot. But if the guy who wrote the code left the company a year ago and then you have to change this code... it's going to be a lot easier if there are good tests.
also, how do you know a test didn't fail locally on a developer's machine, causing them to fix work in progress? IME, test failures are only tracked on dev/stage/prod deployed systems.
Not at all sure how anyone following this advice with, say, a Rails app ever upgrades to a newer version of the framework. When the framework APIs change, you need something exercising the code just to verify that it interoperates with the new versions.
(Yes, apps and frameworks based on a statically type-checked language would catch some of these cases. But not the ones that change behavior of something while leaving the relevant type signatures intact.)
(Yes, apps and frameworks based on a statically type-checked language would catch some of these cases. But not the ones that change behavior of something while leaving the relevant type signatures intact.)
I'm a bit skeptical of a paper written by a consulting company that specializes in QA services and training. Understanding a codebase well enough to write useful unit tests is very time consuming and difficult. Teaching devs the same requires domain, development and process skills. Other forms of testing (e.g. end-to-end, browser based, and manual) are much easier for an outside organization to apply based on rules in a BRD or user manual. It stands to reason that a company would write a white paper that nudges customers to a methodology that favors them.
He has some points though. Unit testing has become a bit of a religion, where people almost write tests for the hell of it regardless of their value. Now I do usually test e.g. algorithms, but I find it rather wasteful to test dumb classes at the unit level as well as UI stuff.
I've certainly felt what he pointed out about tests becoming a real hassle to maintain while not necessarily being very valuable to keep.
I've certainly felt what he pointed out about tests becoming a real hassle to maintain while not necessarily being very valuable to keep.
Cope, the author, was writing about this for a long time. His history precedes him. It's not some random consultant with no experience.
the programming people i admire most just randomly seem to be the same people who never write angry rants about programming topics.
took me 20 years to notice this.
took me 20 years to notice this.
When I was programming on a daily basis, I did make code for testability purposes but I hardly wrote any unit tests. However I was renowned for my code quality and my nearly bug free software. I like to investigate WHY did this work for me?
This made me giggle. It's like magic. don't write tests, don't find bugs....
This made me giggle. It's like magic. don't write tests, don't find bugs....
Actually he provides much better alternatives. Design by Contract combined with system level tests will catch more bugs than you can shake an editor at.
Link is down for me. Can someone give a tl;dr?
The TL;DR: I did a quick read, no evaluation here, just what's in the paper.
1. Unit tests only work in "functional" code composed of easy-to-reason-about components that support other similar components. The OO storm introduced a style of code which is impossible to inspect unless it is running.
2. Because Turing machines you need as many bits of test code as bits of real code.
3. Code coverage requirements can only lead to tests for their own sake, i.e. wasted lines of code.
4. If there are many more lines of tests than production code, your developers are quite possibly paranoid about what they're writing, and that's bad.
5. Tests that have not failed in a long time don't convey information, they only provide noise. Optimize your test suite to provide information.
6. The reward of seeing a green bar in an IDE encourages appeasing the test gods instead of designing a good system.
7. Assertions are better than tests.
8. The Agile culture has turned unit tests into something worse than what they could be without it.
1. Unit tests only work in "functional" code composed of easy-to-reason-about components that support other similar components. The OO storm introduced a style of code which is impossible to inspect unless it is running.
2. Because Turing machines you need as many bits of test code as bits of real code.
3. Code coverage requirements can only lead to tests for their own sake, i.e. wasted lines of code.
4. If there are many more lines of tests than production code, your developers are quite possibly paranoid about what they're writing, and that's bad.
5. Tests that have not failed in a long time don't convey information, they only provide noise. Optimize your test suite to provide information.
6. The reward of seeing a green bar in an IDE encourages appeasing the test gods instead of designing a good system.
7. Assertions are better than tests.
8. The Agile culture has turned unit tests into something worse than what they could be without it.
I long for a language or library that would be better at contracts than just pre- and post-condition assertions.
Then when assertions fail, you write a regression test. But a unit test that verifies of the contract itself is verified correctly is needed too. More than once such a test actually found that it can be violated under common conditions or does not check what you thought of does. That is the main reason for "unit" tests.
It is even more required for multithreaded or parallel programming.
Then when assertions fail, you write a regression test. But a unit test that verifies of the contract itself is verified correctly is needed too. More than once such a test actually found that it can be violated under common conditions or does not check what you thought of does. That is the main reason for "unit" tests.
It is even more required for multithreaded or parallel programming.
This is such an awesome summary, I'm just going to start here without actually reading the paper :-). I'll say up front that I'm a TDD/Unit test kind of guy. I'm also not generally a fan of London School/Outside-In style (but will obviously use it when circumstances demand it). So I'm probably exactly the kind of person that should disagree with the above. But I don't really disagree that much.
One thing I've noticed is that there really is more than one way to skin a cat. Some people like to have a more linear style in their code. I've found that such people tend to have good memories and considerable ability at reasoning. They can see a fairly large piece of code and see quickly where the errors are. They can remember that one piece of code on the other side of the project did something one way and code on this side of the project does it another way.
These people will often look at my code and it looks disjointed to them. They can't read the code and get a sense of what it is doing. They find themselves jumping around through hundreds of files just to figure out what's going on. If I have any ability at all it's really much more in the spatial relations side. I can visualise networks easily and reason about them.
I can never remember the details about how things are implemented and I'm a very slow reader (I'm dyslexic). I depend on trust. I want to be able to see a function and tell in 5 seconds whether or not it is correct. I then rely on tests to tell me when I have violated some assumptions somewhere -- because I don't bother to learn and have no capacity for remembering all the assumptions in the system.
Unit tests and the intense factoring that occurs when doing TDD (at least in a non-mocking style) favours a style where you admit imperfect knowledge about the system. It then gives you a framework for reasoning about the system. People (even on my team) are often surprised that I haven't used a debugger for at least 10-15 years. Tests are really a misnomer -- they don't verify the correct execution of the code. What they do is probe the execution of the code. It tells me what the code is actually doing and documents my assumptions about what it should be doing. If there is an error, there will be a mismatch in the two. I may or may not have an explicit test that fails during that mismatch, but I should be able to reason about it effectively.
It's a different way to think and a different way to work. IMHO, I think it requires considerably less mental capacity to work in a TDD way. However, on the other side of the coin it requires considerably more training/experience to do well. You definitely want to decide upfront which way your team wants to operate, but I'm not about to say that you're "wrong" if you pick a different approach than I would.
Anyway, thanks for doing the summary! It was very evocative for me :-)
One thing I've noticed is that there really is more than one way to skin a cat. Some people like to have a more linear style in their code. I've found that such people tend to have good memories and considerable ability at reasoning. They can see a fairly large piece of code and see quickly where the errors are. They can remember that one piece of code on the other side of the project did something one way and code on this side of the project does it another way.
These people will often look at my code and it looks disjointed to them. They can't read the code and get a sense of what it is doing. They find themselves jumping around through hundreds of files just to figure out what's going on. If I have any ability at all it's really much more in the spatial relations side. I can visualise networks easily and reason about them.
I can never remember the details about how things are implemented and I'm a very slow reader (I'm dyslexic). I depend on trust. I want to be able to see a function and tell in 5 seconds whether or not it is correct. I then rely on tests to tell me when I have violated some assumptions somewhere -- because I don't bother to learn and have no capacity for remembering all the assumptions in the system.
Unit tests and the intense factoring that occurs when doing TDD (at least in a non-mocking style) favours a style where you admit imperfect knowledge about the system. It then gives you a framework for reasoning about the system. People (even on my team) are often surprised that I haven't used a debugger for at least 10-15 years. Tests are really a misnomer -- they don't verify the correct execution of the code. What they do is probe the execution of the code. It tells me what the code is actually doing and documents my assumptions about what it should be doing. If there is an error, there will be a mismatch in the two. I may or may not have an explicit test that fails during that mismatch, but I should be able to reason about it effectively.
It's a different way to think and a different way to work. IMHO, I think it requires considerably less mental capacity to work in a TDD way. However, on the other side of the coin it requires considerably more training/experience to do well. You definitely want to decide upfront which way your team wants to operate, but I'm not about to say that you're "wrong" if you pick a different approach than I would.
Anyway, thanks for doing the summary! It was very evocative for me :-)
> These people will often look at my code and it looks disjointed to them. They can't read the code and get a sense of what it is doing. They find themselves jumping around through hundreds of files just to figure out what's going on.
I'd add that unit tests (above the method level) help to "tell the story" of the code. Coming into a large code base with well-factored code can be daunting, it's a boon to be able to step through unit tests that exercise even just typical scenarios.
I'd add that unit tests (above the method level) help to "tell the story" of the code. Coming into a large code base with well-factored code can be daunting, it's a boon to be able to step through unit tests that exercise even just typical scenarios.
> Tests are really a misnomer -- they don't verify the correct execution of the code. What they do is probe the execution of the code. It tells me what the code is actually doing and documents my assumptions about what it should be doing. If there is an error, there will be a mismatch in the two.
Very well said.
Very well said.
1. Say what? Unit tests only work in functional, not in OO? That seems highly bogus to me, since I've used them in OO all over the place.
Now, if your object architecture is "throw everything into one object", then OO can make it harder to test (or do anything else). But even on a very large class (5000+ lines - I know, I don't like that it's that big, but it's really hard to break up), I've used unit tests very successfully.
Now, if your object architecture is "throw everything into one object", then OO can make it harder to test (or do anything else). But even on a very large class (5000+ lines - I know, I don't like that it's that big, but it's really hard to break up), I've used unit tests very successfully.
I think the assertion is that OO objects tend not to live in isolation, whereas functional behavior can. Tons of OO systems need to talk "to the mothership" for configuration, communication instances, live configured singletons & helpers, loaded plugins, services, factories, etc, all which need to be properly initialized and in flight in order to even call anything on the initial object in question.
It makes a lot of sense to do broad smoke tests and full-cycle specific behavior tests integrating such strongly dynamically dependent subsystems, than to try to create all this artificial test harnessing and fake objects/servers/connections in order to attempt to isolate testing into object-level "units".
It makes a lot of sense to do broad smoke tests and full-cycle specific behavior tests integrating such strongly dynamically dependent subsystems, than to try to create all this artificial test harnessing and fake objects/servers/connections in order to attempt to isolate testing into object-level "units".
[deleted]
[deleted]