How mutation testing improves quality
Unit testing is probably one of the most discussed topics among members in DevOps teams. Every member of the team has an opinion about it. Typical discussions in a DevOps team are about the definition of a unit or what aspects of a unit should be tested. Another point of discussion is code coverage, especially managers seem to love this metric. But the exact boundaries of a unit or the percentage of coverage, are no indicators of the quality of the testing. So instead of these time-consuming discussions the DevOps team should assess the quality of their tests, mutation testing is a good way to do this.
Mutation testing is a type of testing where certain statements in the source code are changed (mutated) to check if test cases will identify the fault that was introduced this way. This is a manner to verify the quality of the test set (instead of the object under test).
Mutation testing is a useful way to check if common faults would be detected by the test set and this way helps to improve a test set in case the injected faults would not be detected. It is however no replacement for proper test design.
How does mutation testing work?
Write a piece of code. Write tests for that code. Execute the tests. If all tests pass this first version of the test, the test is ready. Create a mutant in the code and again execute the tests. If a test fails, the mutant is detected, and the tests are OK. If none of the tests fail, the mutant is not detected; the tests are then incomplete, and an extra test needs to be added or the code must be refactored. So basically each mutation should lead to at least one test to fail.
As with any activity in DevOps, we strive to automate as much of this process as possible. Good mutation testing tools are available. Keep in mind however that the analysis of mutants that are not detected often requires human action.
A brief example
In case the IT system contains a condition that someone should be at least 120 cm long to be admitted, the example pseudo-code is:
With 2-value boundary value analysis the test cases would be:
Both test cases will pass for the pseudocode above. Now the IF statement in the pseudocode is changed with a common mutant (changing "greater-than-or equal" to "equal"):
TC3: input – length = 121, expected outcome – "admitted"
So instead of wasting time on discussing how and what a DevOps team should test, mutation testing enables the team members to focus on the quality of their (unit-)tests. And this in the end leads to achieving the right level of quality, that will deliver the pursued business value to the business stakeholders and users.