Ask HN: What Killed Manual Software Testing?
13 comments
Nothing killed it, it still exists, but it's being replaced by increasing automatic testing or (as you note) letting customers/users do it for us.
Automated testing is almost universally better, except that it misses out on the exploratory nature of manual testing. Many critical systems utilize automation + manual exploratory systems and convert discovered failures from manual testing into part of the regression test suite (which is increasingly moved towards automation over time).
And yes, the first two (users will report bugs or customers will test contract work) are definite signs of a lack of professionalism.
Notable: Manual tests are almost universally integration tests. They are not unit tests. The dogma in the 00s/10s was that unit tests were the best tests (fortunately this was not universally held and is dying down, unfortunately it's, in some circles, swinging the other way to oppose unit tests wholesale; baby with the bathwater). Unit tests are great tests, but they are not integration tests and it is the interaction between units where many issues arise. (This observation is not unique to software, it's true of most physical, electronic, and hybrid systems as well.) You need both kinds. Integration tests, manual ones especially, are expensive to execute and this discourages many companies/organizations from pursuing them even when they have a way to automate the regression test suite after the fact.
Automated testing is almost universally better, except that it misses out on the exploratory nature of manual testing. Many critical systems utilize automation + manual exploratory systems and convert discovered failures from manual testing into part of the regression test suite (which is increasingly moved towards automation over time).
And yes, the first two (users will report bugs or customers will test contract work) are definite signs of a lack of professionalism.
Notable: Manual tests are almost universally integration tests. They are not unit tests. The dogma in the 00s/10s was that unit tests were the best tests (fortunately this was not universally held and is dying down, unfortunately it's, in some circles, swinging the other way to oppose unit tests wholesale; baby with the bathwater). Unit tests are great tests, but they are not integration tests and it is the interaction between units where many issues arise. (This observation is not unique to software, it's true of most physical, electronic, and hybrid systems as well.) You need both kinds. Integration tests, manual ones especially, are expensive to execute and this discourages many companies/organizations from pursuing them even when they have a way to automate the regression test suite after the fact.
I think the separate QA before release step I bet was mostly killed by modern methods of software distribution. It's true that automated testing is great, but let's be real: automated tests have existed for many decades, and certainly overlapped with orgs having manual QA steps as part of release engineering.
If you're duplicating floppies, minting CDs, DVDs and so forth it's very expensive to make corrections. If you're pushing remote updates and digital downloads, or just running the software as a service and making it available over the network (e.g. a website) there's not the same expense associated with correction.
These methods of distribution also let careful organizations release to a small things out to a fraction of users first, which can minimize the costs and risks associated with a bad software push as well.
If you're duplicating floppies, minting CDs, DVDs and so forth it's very expensive to make corrections. If you're pushing remote updates and digital downloads, or just running the software as a service and making it available over the network (e.g. a website) there's not the same expense associated with correction.
These methods of distribution also let careful organizations release to a small things out to a fraction of users first, which can minimize the costs and risks associated with a bad software push as well.
I think that for the most part, reality killed it. Manual testing can slow down the development cycle considerably when bug fixes are streamed through, during system integration testing, etc. A small change can cause the manual testers to groan and declare that they have to recreate a dozen scenarios manually, which includes lots of data prep -- merging in a large change a short time before a planned release is even worse.
Automated tests never get bored or tired, and they can run constantly or exactly when needed; manual testing depends on the mental state of the tester and how prone they are to making small mistakes in general -- a bad night can result in an error in a non-reversible manual step that destroys, say, 2 hours of prep work if detected early and 12 hours of total work if not, causing chaos and confusion in the dev team. A manual test doesn't indicate that it has failed without someone going through it again, and I suspect many orgs don't want to do 20,000 manual tests during system integration.
Automated tests generally survive team reogs, departures, and memory loss; unless manual test scripts are meticulously documented, they may lack implicit assumptions held as a result of working on the same system for 5 years.
Automated tests never get bored or tired, and they can run constantly or exactly when needed; manual testing depends on the mental state of the tester and how prone they are to making small mistakes in general -- a bad night can result in an error in a non-reversible manual step that destroys, say, 2 hours of prep work if detected early and 12 hours of total work if not, causing chaos and confusion in the dev team. A manual test doesn't indicate that it has failed without someone going through it again, and I suspect many orgs don't want to do 20,000 manual tests during system integration.
Automated tests generally survive team reogs, departures, and memory loss; unless manual test scripts are meticulously documented, they may lack implicit assumptions held as a result of working on the same system for 5 years.
In my experience, manual QA is a symptom of bad engineering culture where engineers inherit large systems with poor test coverage. Automated testing is only hard when it isn't considered from the beginning of the project.
A lot of manual QA is bad testing too. If you have a grueling and linear process to go through to test your software each release... why isn't it automated? Is it doing something that fundamentally cannot be automated? If yes, you're probably wrong - almost everything can be automated.
Humans need to be put in the loop when requirements are changing and problems need to be solved. Designing comprehensive testing systems may be the role of good QA, but actually running the tests by hand is not a great place to be regularly.
A lot of manual QA is bad testing too. If you have a grueling and linear process to go through to test your software each release... why isn't it automated? Is it doing something that fundamentally cannot be automated? If yes, you're probably wrong - almost everything can be automated.
Humans need to be put in the loop when requirements are changing and problems need to be solved. Designing comprehensive testing systems may be the role of good QA, but actually running the tests by hand is not a great place to be regularly.
I agree with repeated tests, they are wonderful at catching regressions - however exploratory testing is still something valuable which many developers overlook - a lot of us are happy with clicking through happy path and maybe some most obvious error scenarios, but there is a lot of applications where you can paste several megabytes of text in some of the forms - something that a more thorough exploration testing session would likely catch.
I suspect shorter delivery loops killed it. Manual QA is alive and well in projects with long release/delivery times, like embedded products. Also things like partial rollouts aren't possible there, but can be done if you can update deployed software very quickly.
As a former QA Eng, this is pretty spot on. We use to have a lot more lead time in doing manual testing and catching things during feature testing and then adding to our regression suites. Nowadays the mindset of continuous delivery is to just fail forward and rollback or feature flag something off if it doesn't work. There is still manual testing in teams that are more waterfall and probably dealing with more impactful issues that make manual testing a priority, like payments.
I work in finance and payments and manual testing is only confined to exploratory testing which we only do occasionally.
I find a good human tester is a kind of "expert user" - someone with intimate knowledge of the domain, and knows how a product could help someone solve a problem. Arguably, this is the role of product management. They SHOULD be people from the domain regularly dogfooding the product. Unfortunately, a lot of product management really has become what used to be called "project management": planning, budget etc. It can be less about what the product should do, depending on the specific product manager.
My first software job many years ago was manual software testing. I had a script I'd follow. Create a thing. Link it to a thing. The company paid me to do this.
Now that I'm a dev, I could write a cypress test that would run through that entire script as part of CI, which means the mistakes would get caught before code is merged. How is that not preferable?
Now that I'm a dev, I could write a cypress test that would run through that entire script as part of CI, which means the mistakes would get caught before code is merged. How is that not preferable?
Automated QA is great at following the script. Manual QA is better at going off-script.
For example, that box on our Windows UI on our product that lets you paste a list of IP address or machine names? It was manual QA that decided to try pasting a 2 Meg Windows executable into that window. The app did not handle that very well...
For example, that box on our Windows UI on our product that lets you paste a list of IP address or machine names? It was manual QA that decided to try pasting a 2 Meg Windows executable into that window. The app did not handle that very well...
Spouse is QA in apple. Lots of manual testing going on. Just getting started on automation
Automated testing. Manual testing has now been confined to exploratory testing.
Trying to push the issue in multiple projects I've been involved in has so far been met with resistance, calling it unnecessary. The usual counter arguments are:
Am I crazy to think the first two show a lack of care for quality of our software, and that effective automated testing is too hard (writing tests that catch a wide range of non-regression bugs is often more difficult than writing the actual program) and expensive for projects on limited budgets? Wouldn't a few QA staff for testing on a regular basis be cheaper than the additional engineering hours for automation or support for unhappy customers?
The only time I've seen structured manual testing in practice was in a company where the entire development and IT staff was assigned to it the day before a planned release. It was grueling and linear work, but even that uncovered bugs that automation didn't catch. At another project, the bugs seemed to become more severe after people in our company (especially outside engineering) stopped using it on a regular basis. Would hiring someone to do it as their job help? Are there resources out there to get started on setting manual QA up?