Home TMap Digital transformation & The Technology Leaders

How mutation testing improves quality

Quality for DevOps teams

The book “Quality for DevOps teams” and the renewed TMAP body of knowledge support all people involved in IT delivery, to deliver business value using IT systems that are at the right quality level.

Berend van Veenendaal, co-author of this book, wrote this blog in order you to learn how mutation testing, which relates closely to unit testing, can determine whether your unit tests are working properly.

How mutation testing improves quality

Unit testing is probably one of the most discussed topics among members in DevOps teams. Every member of the team has an opinion about it. Typical discussions in a DevOps team are about the definition of a unit or what aspects of a unit should be tested. Another point of discussion is code coverage, especially managers seem to love this metric. But the exact boundaries of a unit or the percentage of coverage, are no indicators of the quality of the testing. So instead of these time-consuming discussions the DevOps team should assess the quality of their tests, mutation testing is a good way to do this.

Mutation testing is a type of testing where certain statements in the source code are changed (mutated) to check if test cases will identify the fault that was introduced this way. This is a manner to verify the quality of the test set (instead of the object under test).

Mutation testing is a useful way to check if common faults would be detected by the test set and this way helps to improve a test set in case the injected faults would not be detected. It is however no replacement for proper test design.

How does mutation testing work?
Write a piece of code. Write tests for that code. Execute the tests. If all tests pass this first version of the test, the test is ready. Create a mutant in the code and again execute the tests. If a test fails, the mutant is detected, and the tests are OK. If none of the tests fail, the mutant is not detected; the tests are then incomplete, and an extra test needs to be added or the code must be refactored. So basically each mutation should lead to at least one test to fail.

As with any activity in DevOps, we strive to automate as much of this process as possible. Good mutation testing tools are available. Keep in mind however that the analysis of mutants that are not detected often requires human action.

A brief example

In case the IT system contains a condition that someone should be at least 120 cm long to be admitted, the example pseudo-code is:

   IF length ≥ 120 cm
      THEN MOVE "admitted" TO status
      ELSE MOVE "not admitted" TO status
   ENDIF

With 2-value boundary value analysis the test cases would be:
TC1: input – length = 119, expected outcome – "not admitted"
TC2: input – length = 120, expected outcome – "admitted"

Both test cases will pass for the pseudocode above. Now the IF statement in the pseudocode is changed with a common mutant (changing "greater-than-or equal" to "equal"):

   IF length = 120 cm

For this mutant TC1 and TC2 will both pass, but a test case should have failed. The mutation test has detected a fault in the test set that can be fixed by adding a third test case:

TC3: input – length = 121, expected outcome – "admitted"

 

So instead of wasting time on discussing how and what a DevOps team should test, mutation testing enables the team members to focus on the quality of their (unit-)tests. And this in the end leads to achieving the right level of quality, that will deliver the pursued business value to the business stakeholders and users.

Published: 19 February 2020

Author: Berend van Veenendaal

Quality

for DevOps teams