Fragments of the book,Lessons Learned in Software Testing, written by Cem Kaner, James Bach, Bret Pettichord.
Testers are Negative
Testing is the only role on the project that does not directly focus on success. Everyone else creates something or creatively guides its creation. But testers are negative. This can be a depressing job, almost like a parody of a Greek myth: “On the island of the testers, they were doomed forever to search for what could not and should not exist, knowing that to succeed would bring misery to the Gods.”.
Quality comes from the people who build the product, and that is sometimes a heavy burden for them to bear.
Testing is Applied Epistemology
Epistemology is the study of how you know what you know. It’s the study of evidence and reasoning. It establishes the foundations of scientific practice.
If epistemology tells us about how we should think, cognitive psychology tells us about how we do think. Some topics related to that include:
- Reliability of your senses and memory
- How your beliefs affect your behaviour
- How you learn and share what you know
- Biases and short cuts you use to make deicsion
- How you recognize patterns
- How you sort ideas into categories
- Distortions in remembered events
- How you notice differences between things
- How you reconstruct partially remembered events (such as non-reproduceable bugs)
Four Kinds of Thinking
- Technical thinking: the ability to model technology and understand causes and effects.
- Creative thinking: the ability to generate ideas and see possibilities. You will test only in ways that you can imagine testing. You will look only for problems that you imagine can exist.
- Critical thinking: the ability to evaluate ideas and make inferences. This includes the ability to detect and eliminate errors from your thinking.
- Practical thinking: the ability to put ideas into practice (test tools and testing techniques).
We find that when the test process fails in the most damaging ways, the root cause is most likely to be tunnel vision. In other words, it’s not that we ran 10.000 tests and should have run 10.001. It’s that we failed to imagine an entire category of test; testing we wouldn’t have performed even if we had twice the time and resources.
Exploring
The skilled search is always exploratory.
The difference between a tester and a tourist is that a tester’s efforts are devoted to evaluating the product, not merely witnessing it.
By exploration, we mean purposeful wandering: navigating through a space with a general mission, but without a prescripted route. Exploration involves continuous learning and experimenting. There’s a lot of backtracking , repetition, and other processes that look like a waste to the untrained eye.
Exploring is detective work. It’s an open-ended search. Think of exploration as moving through a space. It involves forward, backward and lateral thinking.
The Logic of Conjecture and Refutation
The only confidence a given conjecture deserves would come from trying hard to refute it and not being able to do so. This method applies to testing: it’s more powerful to test for the purpose of showing that the product fails, than it is to show that it works.
You Can’t Master Testing Unless you Reinvent it
Some consultants and writers in the field seem to believe that an inefficient tester can be transformed into an effective tester merely by handing them a test procedure.
You’ll be a technician, nothing more. Following instructions won’t get you to mastery any more than following an interstate highway will get you to Mars.
Minor Bugs
Any product can get by with a few minor defects. But as the number increases, customer confidence declines. More disturbing is the corrupting effect of tolerating these bugs. At the point that you stop even reporting minor bugs (some companies pressure their staff to report only “worthwhile” errors), you’ll find yourself and your company tolerating more and more serious ones. Down that road (unless you’re in a monopoly position in your market), lies product failure.
Automating Testing
The trap of automating tests that are easy to automate but weak at finding defects. When you perform a test by hand, you bring to bear the entire range of human capabilities. You can improvise new tests or notice things that you did not or could have not anticipated. Test automation is a faint, tinny echo of that rich intellectual process.
Automation does not make the computer do the testing that you do. It performs the testing you explicitly specify, unable to benefit from your tacit knowledge and awareness. Automation has a lot of advantages, but it puts a lot of distance between the human tester and the tests.
Testability is often a better investment that automation. Testability is visibility and control.
Documenting Testing
A test documentation template is no substitute for skill. Massive test documents create a compliance mentality. Do what the plan says. This is fundamentally different from the mentality of the alert, critical tester who pays attention to any hints available and follows up all promising leads.
There is no apparent awareness and no discussion of the (enormous) cost of providing all of these types of information. Time spent on that documentation may well be time not spent on testing.
In our experience, companies have been much more likely to simply give up on the documentation, because it is incomplete and it quickly goes out of date.
Project documents are interesting fictions: Useful, but never sufficient.
Mediocrity is a Self-Fulfilling Prophecy
- If you eliminate creativity by dehumanizing your processes
- If your goal is to make your staff interchangeable cogs
- If you standardize your staff’s work, so that anyone can do exactly what they can do, in the same way that they do it
- …
A Complex Space
It’s better to do more different kinds of testing to a pretty good level, than to do one or two kinds of testing perfectly. Avoid tunnel vision.
When you test, you are sampling a complex space. No single test technique will sample this space in a way that finds all important problems quickly. If you switch to a technique that is sensitive to a different kind of problem, your find rate may climb.
Duplicated test effort is not the same thing as duplicated tests. Two testers testing the same thing have a fair likelihood of finding different problems. That’s because the testers are probably not running the same tests. Duplicated test effort is almost never a waste.
Context-Driven
The value of any practice depends on its context. There are good practices in context, but there are no best practices.