DevOps teams work in an everchanging world where the common expectation is that quality and speed improve. They constantly need to improve their way of working and adapt to changed circumstances.
Change happens when the pain of staying the same is greater than the pain of change.
– Tony Robbins
Continuously improving delivery processes is not a new approach. Lean manufacturing and the Japanese manufacturing approach called Kaizen have been applied to improving factory processes for decades. Continuous improvement is advocated by all agile approaches. DevOps does not deviate from this. As everything in DevOps, executing continuous improvement is a whole-team responsibility and works best in a sustainable continuous improvement culture.
In this section, we will describe how to establish a continuous improvement culture and how you could continuously improve the application, the DevOps activities and the skills of the people involved.
This in alignment with our DevOps "built-in quality" vision:
- Create quality awareness in all people involved.
- Integrate QA & Testing activities in all DevOps activities.
- "Integrate" QA & Testing activities in all people involved.
How to establish a continuous improvement culture
Developing a sustainable continuous improvement culture requires careful attention. Consider, among other things, the following:
- Supportive and engaged leadership
Leadership must be committed to the changes. They need to be supportive, open and willing to consider every single suggestion. They also need to make sure the team members feel safe and unthreatened when they share their suggestions.
Work on a culture to influence the quality mindset. When the culture of the team, project or organization shows a quality mindset, it is easier to create a continuous improvement mindset. This should be supported top-down and bottom-up. When there is a negative culture in the organization in relation to changes, the continuous improvement mindset will be hard to realize.
- Engaged and empowered team members
If team members feel that they are empowered and engaged, they are more willing to provide improvement suggestions, which is important because direct input from those who work on it and know what type of things will really make a difference in daily work is extremely valuable.
Be consistent in the chosen improvement suggestions, so that the focus remains on these suggestions.
- Continuous suggestion sharing
Team members should constantly and at any time have the opportunity to provide improvement suggestions.
- Enabling technology
The team needs to have an easy to use and accessible toolset that supports the work.
- Transparency (visible management)
The progress and the achievements in the implementation of the improvement suggestions should be visible to team members and stakeholders. Make sure you have the correct administration tools to register suggestion, changes, defects, relations, progress etc. to make reports and show/print them on a frequent base. Make these reports visible in the project room (not only Kanban boards).
- Keep it simple
To ensure that everyone remains involved in the improvement process, this process must be kept as simple as possible.
- Continuous feedback
Ensure that the progress of the various improvement suggestions is discussed on a regular basis so that the entire team feels that what they are doing is seen and appreciated.
What do we need to improve continuously?
Continuous improvement is about iterative improvements to the day-to-day efforts to improve the application, process and the skills of people. The following three perspectives are explained in more detail in the following sections and an improvement approach is also provided for each perspective:
- Quality of the application
- DevOps QA & Testing activities
- QA & Testing skills of people
If you do not want to use one of these approaches, you can also use the – always applicable – Deming cycle.
Improvement of the application being delivered
The improvement of the application being delivered is strongly related to the metrics as defined in Metrics. You could use the GQM approach to identify Goals, Questions and Metrics [Basili 1994]. When using GQM you can distinguish between two types of goals:
- Knowledge objectives
Where do we stand right now?
- Improvement objectives
Where do we want to go?
We will explain this using an example. First, we formulate the goal. Let's assume the knowledge goal is: "Provide insight into the quality level prior to release".
Then you have to ask questions concerning this goal, questions such as: "Are the most important risks covered/found early in the DevOps QA and testing activities?", "Are the most important risks covered/found in the DevOps QA and testing activities?", and "What is the stakeholder's perception of the covered/found risks?"
The next step is deriving relevant metrics from these questions. You could use some of the already defined metrics in, or even better, use the metrics for which data is already available. Possible metrics are:
- Percentage of critical defects found early in test execution (effectiveness)
Ratio between the number of critical defects detected in the first quarter of the test execution lead time and the total number of critical defects found in the total test execution lead time.
- Percentage of critical tests run early in test execution (effectiveness)
Ratio between the number of critical tests run in the first quarter of the test execution lead time and the total number of critical tests run in the total test execution lead time.
- Percentage of identified risks covered by (executed) tests (effectiveness)
Ratio between identified risks and the total number of (executed) tests in relation to these risks.
- Average cost per risk item covered during testing (efficiency)
Total test costs divided by the total number of risk items.
- Stakeholder's perception of accepted risks prior to release and after it has been released into production (customer satisfaction)
- Survey of risk perception prior to release to be completed by the stakeholders, and the same survey after three months of production. Make a comparison between both survey results.
After finishing the GQM cycle, improvement actions are defined and executed, and you could start a new GQM cycle. In this way you are continuously improving.
Improvement of the DevOps QA & Testing activities
QA and testing activities are integrated and executed throughout all DevOps activities. The QA and testing activities are not executed only by a person with a testing role, but by the whole team, known as the whole-team approach. In this situation, let's take the DevOps activities as the starting point for improvement.
This DevOps activities-oriented approach requires an effort from each person involved to give substance – per DevOps activity – to six pre-defined Quality key areas [Sogeti 2020]: QA Awareness, QA & Testing, Governance, Transparency, Automation and Infrastructure. We call this approach Quality to Activity Mapping (QAM).
- QA Awareness
Are all people involved concerned with quality and do they all have a quality mindset?
- QA & Testing
Which test topics are embraced by the team and how?
Is the team supported in carrying out their activities and is the team in line with the policy of the organization?
Is the team transparent about insight into progress and quality at any given moment?
Which activities are automated and how?
Does the team have the required infrastructure (including tools) to carry out the activities?
The above is just a brief summary of the key areas.
How does QAM work?
QAM is a team effort and demands participation of all the people involved. Gather these people and request a quality architect (preferably together with a DevOps connoisseur) to be the moderator of the QAM workshop as not all participants will be Quality and DevOps activities experts. First, establish which activities are carried out by the team. Then try to find out together how substance is given to the quality key areas for each of the defined activities.
What does a QAM table look like?
A QAM table maps the quality key areas with the established DevOps activities. At the intersections, the participants will mention the quality measures that are implemented in this activity. Therefore, per activity, all six key areas are discussed one by one. The result is a table that you can use to provide clarity (transparency) to everyone involved about how each activity will contribute to the quality of software development.
If everyone agrees with the current status, suggestions for improvement can be implemented. You can think of (the mentioned suggestions are only meant as examples):
- QA Awareness
- Create QA & Testing awareness for all people involved.
- Start using risk analysis approaches at the beginning of a project to clarify who is going to cover which risks in which test level with which test coverage.
- QA & testing
- Apply risk analysis/poker, give story points for testing in planning poker/sprint planning.
- Apply test design techniques. (spent time, money and resources wisely in covering the risk)
- Apply Behavior Driven Development (BDD) (or Acceptance Test Driven Development or Specification by Example) and Test Driven Development (TDD).
- Establish central QA & testing support.
- Provide risk analysis/poker and test design training for all people involved.
- Provide test training for developers.
- Support shift-left (start as early as possible with QA & testing activities).
- Define Quality indicators per team and/or application.
- Implement cognitive QA dashboard for critical applications and flows and predictive dashboards. (using AI technology)
- Automate all unit test cases in the CI/CD pipeline (e.g. Jenkins, JUnit test framework)
- Use BDD to automate tests (e.g. with Cucumber/JBehave/Behat)
- Create a production-like environment as closely as you can for performance testing. (e.g. virtualization)
- Use SonarQube not only for measuring code quality, but also for statement coverage (e.g. in Unit Testing, System Testing or User Acceptance Testing.
To summarize, identify quality measures per key area, measure the effects of applying these measures and define actions to improve – the application of – these quality measures. If the team regularly repeats this QAM approach, a mechanism is created whereby the Quality & Testing key areas per DevOps activity are continuously improved.
Improvement of the QA & Testing skills of people
Instead of taking the activities, process, framework, etc. as the starting point for improvement, you can also take people as the starting point. In particular the mindset of the people. Obviously, a mindset is framework independent. So, you could work with an Agile mindset in all the Agile frameworks or even in the more traditional, or sequential, development environment. We define the Agile mindset as follows:
|An Agile mindset is a mindset to deliver valuable high-quality increments of working software in time-boxed short iterations, through adaptive change, as more information comes to light in a communicative and collaborative manner.|
This people-oriented approach requires an effort of each person involved in designing, building, running and maintaining software to be transparent on how the person gives substance to six pre-defined Quality key areas [Sogeti 2020]: QA Awareness, QA & Testing, Governance, Transparency, Automation and Infrastructure. We call this approach Quality to People Mapping (QPM).
How does QPM work?
QPM is a team effort and demands participation of all the people involved. So, gather all people involved, which would be easy if you are DevOps and work in a communicative and collaborative manner. People will take on roles such as business analyst, software architect, designer, developer, operations engineer, tester, database administrator and so on in the group.
Note: These people – in any case, the roles – are always there, no matter which framework or software development approach is being used.
Let every participant fill in – in the QPM table – how they would add substance to the Quality key areas and request a quality architect to be the moderator of the QPM workshop as not all participants will be Quality experts. Given the simplicity of QPM, you will be able to implement it right away.
What does a QPM table look like?
A QPM table maps the quality key areas with various roles. At the intersections, the participants will mention the quality measures that they will apply. The result is a table that you can use to provide clarity (transparency) to everyone about how each participant will contribute to the quality of software development. According to this table, the team can hold a certain person responsible if the promised contribution is not made. Using the table, the team can take over a quality measure from a person if this person has not been able to carry out that specific quality measure. In the end, the whole team is responsible for delivering quality.
Which quality measures can be chosen? You can think of: INVEST, TDD, SbE, ATDD, BDD, pair programming, test design techniques, risk analysis, risk poker, MBTD, MBT, code quality and test automation tools, review techniques, DoD, DoR, DoS, WoW, and so on. For more about this we refer to, amongst others, "Tooling", "Quality risk analysis & test strategy", "Quality measures", "Reviewing (static testing)", and "There are many techniques, which one to use?".
To summarize, identify quality measures per role, measure the effects of applying these measures and define actions to improve – the application of – these quality measures per role. When the team reviews and revises this QPM table frequently, let's say once a month, the Quality & Testing skills of the people will improve continuously.
We have used this approach several times, perhaps not always as successful, but it has always made people think about quality, which is of utmost importance to us. After all, it is people who create valuable high-quality software.
William Edwards Deming provided a simple yet highly effective technique that serves as a practical tool to carry out continuous improvement in the workplace. This technique is called Plan, Do, Check and Act (PDCA) cycle or often called Deming cycle. The Deming cycle provides a conceptual as well as practical framework while carrying out Kaizen activities by the employees. The four steps Plan, Do, Check and Act should be repeated over time to ensure continuous learning and improvements.
- Plan stage
This stage involves analyzing the current situation, gathering data, and developing ways to make improvements.
- Do stage
This stage involves testing alternatives in a testing environment establishing a pilot process, or experimenting with small number of people.
- Check stage
This stage requires determining whether the trial or process is working as intended, whether any revisions are needed, or whether this should be scrapped.
- Act stage
This stage focuses on implementing the process within the organization or with its customers and suppliers.
Once all these stages are completed to the fullest satisfaction, the improvement is standardized. But the cycle doesn't stop here. With changing circumstances this standardized improvement is again subjected to further improvement, repeating the Deming cycle again and again.