Substantiate the test strategy with test coverage types
That which we want to cover by tests is termed coverage type: the form in which test situations are deducible from the test basis. A coverage type focuses on achieving a specific coverage to detect specific types of defect.
Coverage types are found to be a difficult and time-consuming business. Some people would prefer not using any coverage types at all! It is true, some coverage types are difficult to apply and take a considerable amount of time. But who said testing was easy? Coverage types represent the only way to realise the agreed test strategy in a demonstrable way. If unsuitable coverage types are used or even none at all, it is impossible to demonstrate that the required test intensity is achieved. As a result, all of the choices that led to a suitable test strategy would be in vain, and it becomes very difficult to make a coherent statement as to whether test goals, risks, priorities or simply the desired test intensity have or have not been covered.
Coverage types may be divided into four basic coverage groups:
- Process: Processes can be identified at several levels. There are algorithms of control flows, business processes. Coverage types like paths, statement coverage, and state transitions can be used to test (variations in) these processes.
- Conditions: With almost every system, there are decision points consisting of conditions, where the system behaviour can go in different directions, depending on the outcome of such a decision point. Variations of these conditions and their outcomes can be tested using coverage types like decision coverage, modified condition/ decision coverage, and multiple condition coverage.
- Data: Data is created and ends when it is removed. In between, the data is used by updating it or consulting it. This lifecycle of data can be tested, but also combinations of input data, as well as the attributes of input or output data. Some coverage types here are Boundary values, CRUD, Data flows, and Syntax.
- Appearance: How a system operates, how it performs, what it’s appearance should be, is often described in non-functional requirements. Within this group we find coverage types like operational and load profiles, and presentation.
Coverage Types per coverage group
The table below gives a brief description of each coverage type per group. Where applicable, an indication is given how the depth of coverage can vary according to the coverage type.
|Coverage group||Coverage type||Description|
|Process||Control flow||Testing the program structure.|
|Paths||Coverage of the variations in the process in terms of combinations of paths. A scheme of decision points and paths is required as a test basis.|
|Rare events||Addressing events that happen very infrequently.|
|Right paths/ fault paths||Checking both the valid and invalid situations in every defined error situation. An invalid situation (faulty control steps in the process or algorithm that precede the processing) should lead to correct error handling, while a valid situation should be accepted by the system without error handling.|
|State transitions||Verification of relationships between events, actions, activities, states and state transitions.|
Coverage of the various possibilities within a decision point with the purpose of arriving at the outcomes of TRUE and FALSE
|Semantics||Validation relationships between data.|
|Data||Boundary values||A boundary value determines the transfer from one equivalence class to the other. Boundary value analysis tests the boundary value itself plus the value directly above it and directly below it.|
|CRUD||Coverage of all the basic operations (Create, Read, Update, Delete) on all the entities.|
|Data combinations||Testing of combinations of paramater values. The basis are Equivalence classes.|
|Data flows||Verifying information of a data flow, which runs from actor to actor, from input to output.|
|Domain testing||Coverage of a small number of values from a nearly infinite group of candidate values. Domain knowledge plays a very critical role while testing domain-specific work.|
|Equivalence classes||The value range of a parameter is divided into classes in which different system behaviour takes place. The system is tested with at least 1 value from each class.|
|Integrity rules||Checking the preconditions under which certain CRUD processes are or are not permitted.|
|Right paths/ fault paths||Checking both the valid and invalid situations in every defined error situation. An invalid situation (certain values or combinations of values defined that are not permitted for the relevant functionality) should lead to correct error handling, while a valid situation should be accepted by the system without error handling.|
|Syntax||Validation of attributes of input or output data.|
|Appearance||Heuristics||Evaluation of (a number of) usability principles.|
|Load profiles||Simulation of a realistic loading of the system in terms of volume of users and/or transactions.|
|Operational profiles||Simulation of the realistic use of the system, by carrying out a statistically responsible sequence of transactions.|
|Presentation||Testing the layout of input (screens) and output (lists, reports).|
Variation in coverage types
For some coverage types, it is possible to variate the coverage thouroughness within the coverage type.
The table below gives several examples:
Test depth level N
|Semantics||See decision points and equivalence classes|
|Integrity rules||See decision points and CRUD|
|Syntax||See individual test situations|
Selection of Coverage types and Test design techniques
There exist many coverage types and test design techniques. For the sake of simplicity and practicality we will only highlight the most commonly used test design techniques and hence the application of the underlying coverage types.
To give you a practical overview we highlight the most commonly used coverage types and some test design techniques in which they can be applied.
|Coverage type group||Test intensity: Light||TEST INTENSITY: Average||TEST INTENSITY: Thourough|
|Condition||Condition Decision Coverage – Elementary Comparison Test||
Modified Condition Decision Coverage – Elementary Comparisson Test
|Multiple Condition Coverage
Elementary Comparisson Test
One or some data pairs – Data Combination Test
|Pairwise – Data Combination Test||N-wise or all combinations – Data Combination Test|
|Process||Statement coverage and Paths test depth level 1 – Process Cycle Test||
Decision coverage andPaths test depth level 2 – Process Cycle Test
Paths test depth level 2 – Algorithms Test andPaths test depth level 3 – Process Cycle Test
We left out the coverage group "Appearance". In the cases where that coverage group is applicable, the coverage type and tourougness depends too much on the specific interface and the result the tester wants to achieve.