The choices in your test strategy on WHAT to test indicate that you want to cover certain aspects of the test object. The objective of an effective test strategy is therefore to realise the best achievable coverage at the right place. Coverage has everything to do with the wish to find the most possible defects with the fewest possible test cases.
But what is coverage? Coverage is very subjective. We cannot talk about the coverage. What does an executive or other stakeholder mean when he/she asks you what the coverage of the test was? What information does he/she need or want? Possibly he/she wants to know how thorough some aspects of the test object have been covered. Maybe he/she want to know how many of all possible defects have actually been found by the tests.
A key word here is Coverage. A definition for coverage is hard to give. It basically deals with aspects of the test object that you would like to assess and the thoroughness with which you do that.
More important is the question if we are able to achieve 100% coverage. Well, we can never be certain that all defects have been found or even that 60% of all defects has been found. After all, we do not know how many defects there actually are. Furthermore we don’t know how accurate and complete the information was on which we based our test cases. Also if the tests we executed were based on a test strategy (and product risk analysis) we can never be sure whether our stakeholders made the right choices on what to cover. Testing everything is simply impossible, because it is impossible to define what ‘everything’ means.
Altough coverage is hard to define, it has a relation with the following two terms:
- The parts of the test objects that we want to test
- Coverage thouroughness applied to each of those parts.
A more easily definable term closely related to coverage is the coverage type: the form in which test situations are deducible from the test basis. A coverage type focuses on achieving a specific coverage to detect specific types of defect (e.g. in the interfaces, the input checks or the processing), such defects are detected more effectively then by specifying ad hoc test cases. One coverage type could only be called ‘better’ with any practical use if it would find at least all of the defects found by the other coverage type plus some additional defects.
The decision to ‘test more thoroughly’ concretely translates to a decision to achieve more extensive coverage. In principle, there are 3 options in this respect:
- A more thorough coverage type
- Multiple coverage types
- A higher variation of thoroughness within a specific coverage type.
A term to be introduced here in relation to coverage is coverage thoroughness.
- It is not possible to test everything within the confines of the preconditions of time and costs defined in the assignment. Choices will have to be made as to the lengths one wishes to go to in testing.
- A test strategy is used to create an overview of what will be tested and how thorough, such that the aspects to be tested are covered as adequately as possible.
- The decisions concerning thorough and less thorough testing are translated to concrete statements about the targeted coverage.
- Depending on the available test basis, among other things, appropriate coverage types are selected to achieve said coverage.
What exactly is coverage thouroughness?
In the test strategy is decided what coverage thouroughness is to be acheived in the test. This means that executives and other stakeholder most likely expect information about the thoroughness of your test. But what do they mean? They probably mean one of the four aspects related to thoroughness:
- Did you cover everything?
- How thorough was the chosen coverage type?
- Were multiple coverage types applied?
- How high was the variation of thoroughness within a specific coverage type?
So, although a fascinating subject, it’s also a complex matter. There is no black and white here. However, we can state that testing everything is impossible. How more thorough is a certain coverage type compared to another (e.g. pairwise testing versus modified condition/decision testing)? How more thorough is one variation within a coverage type compared with another variation within that same coverage type (e.g. modified condition/decision testing versus modified condition coverage)? How many additional defects are to be found?
Wiki's - Coverage types:
- Right paths / fault paths
- Decision points
- Equivalence classes
- Boundary value analysis
- Orthogonal arrays and pairwise testing