Last spring the Continuous Testing Report (CTR) 2019 was released. I have read this with great interest. A few things struck me. Despite the fact that organizations have invested a lot in the automation of testing activities in recent years, this remains a major challenge. In particular, achieving sufficient test coverage appears to be a problem. Only focus on automation is therefore not the solution. Attention to the test approach also remains essential.
My experience as an Agile Quality Coach is that many organizations that have made the move to work Agile have not included test and QA craftsmanship. As a result, they sometimes even throw it overboard - with all its consequences. Making testing a specific point of attention creates more quality awareness within teams, evolving into a quality culture. Supplemented with the capacity to organize this in a structured way in the development process, this leads to improved test coverage and therefore more control over quality. This makes testing certainly not an activity that is only done by testers from their own silo, but a permanent part of the entire development cycle.
Testing for a small part of the functionality
Traditionally we looked at the complete scope of a project. Based on that, we made a product risk analysis (PRA) which was input for the test strategy. For example, we provided effective coverage, whereby we tested the high risks with a higher coverage than the low risks. Nowadays we look ahead a few sprints in organizing the work. We determine the test effort based on this. The result: just (automatic) testing on a small part of the functionality.
Also tests the relationship between functionalities
The problem is that a patchwork of testing is created in which the relationship between functionalities is not tested. This cohesion is not part of the user story and therefore the team does not pay attention to this. We also regularly see that there is no allignment between the executed unit tests and the functional tests. Because of this there is a lot of overlap and discrepancy:the points that are tested are often double tested and the coherence is not tested.
Transparency of test coverage
How do we solve this? Do we still have to go back to longer release intervals so that we can plan it more easily? No, definitely not. Because scope is variable and therefore we cannot do a PRA in advance over the entire product, we have to approach this in a different way. The solution for this is in transparency of test coverage. This starts with the requirements. There we have to create insight into the relationship between functionalities and the preferred additions.
To measuring is to know
Another part of the solution can be found in the creation of tests. The following applies: to measure is to know. Spending time on setting up and making visible, for the team, valuable metrics definitely pays off. In an ideal situation the retrieval of this data is also fully automated. Is this not yet possible? Start collecting manually. Examples of quality metrics are code coverage for unit tests, but also: how many test scripts do we have per acceptance criteria or which components and interfaces do we touch with our tests.
The collection of metrics only has value if the team also regularly looks at this and then focuses on trends. The meaning of the measurement must be able to be explained and be linked to the objective of the team to improve. By creating structural insight into the test coverage and looking for improvements, the quality of the test coverage will improve over time. This increases the reliability of the delivered product.
More information about test coverage and test automation?
Published: 14 August 2019
Author: Wouter Ruigrok