Coverage

The choices in your test strategy on WHAT to test indicate that you want to cover certain aspects of the test object. The objective of an effective test strategy is therefore to realise the best achievable coverage at the right place. Coverage has everything to do with the wish to find the most possible defects with the fewest possible test cases.

But what is coverage? Coverage is very subjective. We cannot talk about the coverage. What does an executive or other stakeholder mean when he/she asks you what the coverage of the test was? What information does he/she need or want? Possibly he/she wants to know how thorough some aspects of the test object have been covered. Maybe he/she want to know how many of all possible defects have actually been found by the tests.

A key word here is Coverage. A definition for coverage is hard to give. It basically deals with aspects of the test object that you would like to assess and the thoroughness with which you do that.

More important is the question if we are able to achieve 100% coverage. Well, we can never be certain that all defects have been found or even that 60% of all defects has been found. After all, we do not know how many defects there actually are. Furthermore we don’t know how accurate and complete the information was on which we based our test cases. Also if the tests we executed were based on a test strategy (and product risk analysis) we can never be sure whether our stakeholders made the right choices on what to cover. Testing everything is simply impossible, because it is impossible to define what ‘everything’ means.

Altough coverage is hard to define, it has a relation with the following two terms: 

A more easily definable term closely related to coverage is the coverage type: the form in which test situations are deducible from the test basis. A coverage type focuses on achieving a specific coverage to detect specific types of defect (e.g. in the interfaces, the input checks or the processing), such defects are detected more effectively then by specifying ad hoc test cases. One coverage type could only be called ‘better’ with any practical use if it would find at least all of the defects found by the other coverage type plus some additional defects.

The decision to ‘test more thoroughly’ concretely translates to a decision to achieve more extensive coverage. In principle, there are 3 options in this respect:

A term to be introduced here in relation to coverage is coverage thoroughness.

Summarised:

  • It is not possible to test everything within the confines of the preconditions of time and costs defined in the assignment. Choices will have to be made as to the lengths one wishes to go to in testing.
  • A test strategy is used to create an overview of what will be tested and how thorough, such that the aspects to be tested are covered as adequately as possible.
  • The decisions concerning thorough and less thorough testing are translated to concrete statements about the targeted coverage.
  • Depending on the available test basis, among other things, appropriate coverage types are selected to achieve said coverage.

What exactly is coverage thouroughness:

Check out the different types of coverage: