Quality and test policy | DevOps

Finding the balance between quality, innovation and a short time-to-market is a challenge. A quality policy and test policy strengthen the bridge between the IT team and businesspeople. For businesspeople, the mission, vision and strategy of the organization are the starting points for developing and improving business processes. Where these processes are supported by IT systems, the IT delivery teams need guidance. A quality policy and test policy translate the mission, vision and business strategy into the principles, approaches and objectives that describe how an organization deals with the people, resources and methods involved in the quality and testing process.  

Why would an organization that has implemented DevOps, which puts responsibilities in the teams, still want an overall policy? Because we work in autonomous teams, teams should have some minimum level of guidance, so that they do not have to re-invent wheels. This makes teams happier and perform better. Although some people might feel that the team is almighty and should work in full freedom, for the organization to realize business value, there needs to be alignment between the teams. Effective freedom can only exist within certain frames. This can be facilitated by implementing a quality and test policy. It is important that the test and quality policies are made in cooperation between teams and the organization. Only then do these policies have value for everyone. A well-thought-through policy that is supported by the teams will prove to be a win-win, both the teams and the organization benefit.  

The people involved may implement the quality policy and the test policy separately. However, our advice is to combine them in one high-level document.  

A well-designed and implemented quality and test policy offers a number of advantages. It gives the people involved direction and a frame, or reference, in organizing quality assurance work in connection with the desired quality level. Also, it aligns these activities with the IT activities in general.  

The quality and test policy forms a basis for quality improvement by increasing uniformity across IT delivery teams and it promotes consistency of working across teams. In addition, it contributes to reducing costs through the reuse of, among other things, test processes and reducing the need for training. The quality and test policy describes the policy choices of an organization and should address a number of subjects. 

 

Quality and test policy subjects 

You will find a lot of information in literature – for example [Marselis 2007], [Koomen 2006] – on what should be described in test policy. The desired content differs per situation. The quality and test policy that fits the organization should be described in an adaptive way. Nevertheless, we can already provide a list of subjects that should be addressed in one way or another:  

• Principles of quality assurance and testing - Definitions of Quality & testing, Quality objectives for the organization
• Relationship with mission, vision, business strategy, etc. - Relation of mission, vision & business strategy to quality engineering activities
• Objectives of QA & testing - Set objectives that the IT products should comply with (relates to the objectives in the VOICE model)
• Applied QA & testing topics - Guidelines for specific application of QA & testing topics in the organization
• Responsibilities in the organization - General rules and guidelines for the responsibilities on different levels in the organization
• QA & testing roles and training - Definition of, or reference to, specific roles in the organization, and the required level of education of the people involved
• Indicators related to confidence in value - Guidelines about use of indicators (based on mission, vision and business strategy)
• Testware management and reuse - Guidelines on how to manage testware and reuse it throughout the organization
• Continuous quality improvement - Goals and guidelines for improvement focused on quality of products, processes and people

The general consensus is that a quality and test policy must be concise, it must be displayed on one page (a few explanatory notes may be added in an additional document, but this also needs to be short). This way the policy can be easily and continuously communicated to the people involved. The one-pager can, for example, be pinned to the wall in the team room.  

Reasons to create a policy 

Every organization has a – described or undescribed – mission, vision and business strategy on which it bases its policies. Policy exists in various fields, such as financial, HRM but of course also in quality and IT. Testing is part of quality assurance; to support the teams, but also a testing department or test center (if those exist in the organization), it is wise to then also make the test policy explicit.  

In more and more organizations, the people involved feel the need to establish a quality and test policy. For example, because people want to introduce more uniformity in quality engineering or because the mission, vision and business strategy have already been translated into IT policy and establishing a quality policy and test policy is therefore a logical next step. This for example facilitates traceability between artifacts and activities of various teams. Another reason to formalize the quality and test policy is the increasing diversity in systems development and test approaches, which means that there is a risk of major differences in the test maturity. Not only within the organization, but even within the teams.  

The quality and test policy will focus on the balance between fault prevention and failure resolution. It strives for the optimal costs of product quality which is shown in the figure below [Juran 1951]. When there is little attention to quality, there will be many failures, which causes a huge cost for fixing and paying for damage caused. When there is too much effort in quality assurance, there will be hardly any failures, but the costs are so high that the businesspeople will protest. So for the organization as a whole it is necessary to find the right balance between enough QA & testing and not too much QA & testing, and thus to reach the optimum total cost of quality.

The quality and test policy help to, among other things, achieve better uniformity in the approach and way of working. It focuses on both the quality of the test process and the quality measurement of the system to be tested. This is necessary because nowadays more and more information is automatically processed in chains of information systems with ever-increasing complexity. Uniformity in the realization of systems then contributes to better-connecting systems. Outsourcing and the geographical spread of activities also means that there must be clarity about the transfer of intermediate and end products.  

When formulating the quality and test policy, make sure there is flexibility in the formulation and facilitate adaptivity, rather than using a compulsory list of subjects.  

There are, of course, subjects that you should definitely pay attention to, but sometimes you can put multiple subjects under one heading and at other times it makes sense to pay attention to each subject individually. Moreover, the various subjects influence each other. For example, you can specify that reusability and repeatability should be high, which almost certainly means that you should also arrange tooling. When describing test policy, it is therefore of great importance to flexibly connect to the needs of the organization.  

Translate policy into tactical and operational levels

The quality and test policy starts at a strategic  level: the responsibility lies with the head of the quality and test organization, or with the head of IT or ideally even with the CIO or CEO. The strategic policy can be shaped by wishes and requirements from within the organization. But factors outside the organization may also play a role. Examples are: requirements set by external monitors and regulators, (local) legal requirements that must be complied with, or industry agreements.  

At a tactical level, the strategic test policy should be translated into the DevOps organization. This is achieved by creating regulations that specify the preconditions and standards to which the deployment of people, resources and methods must comply to realize the strategically defined subjects. Often, this is described in a Way of Working (WoW).  

Consistent implementation of the quality and test policy results in a uniform test approach at operational level, which could be part of the Definition of Done (DoD). It is important that the goals are realistic and achievable and that they provide a growth model, because the desired situation can seldom be achieved in one go. Although within DevOps the team members themselves are responsible for executing all quality engineering activities, a distinction is often made between support and the actual execution of the QA and testing activities. Examples of support are advice on issues related to method, technique, tools and automation. But also, delegation of tasks that require specific knowledge or skills to specialized people, such as performance testing or security testing. Quality coaching is another example of how to support a team on QA and testing.  

Although a policy has a long-term perspective it should not be a static document. Once or twice per year, the policy should be reviewed and when needed revised to stay aligned with the needs of the organization.  

We have described and implemented quality and test policies in organizations several times. And we have noticed that it remains very difficult to substantiate or prove whether the test policy delivers, in a financial or other sense. One of the reasons is that immature quality and test organizations in particular rarely have measurement data about the baseline situation. As a result, it is also not possible to make a comparison of "costs then" with "costs now". Yet we can say something about the results. For example, "happy managers", because they get more control over the quality and test process through the set frameworks. The connections and uniformity (for example common terminology) between the teams have improved, and where there are deviations, these have been well thought out. Ultimately, of course, the absence of serious problems with the use of information systems and experiencing the real value of the business process, is the best confirmation that a quality and testing policy is useful!  

Generic Test Agreements (GTA)

Some organizations, projects or teams make use of a so-called generic test agreements (GTAs) document. Often, in a GTA large parts of a (master) test plan are copied with reuse in mind. In many cases, such as in DevOps, outsourcing, offshoring, or in maintenance the execution of test activities, goes beyond a single release. To prevent new agreements on the what and how of testing having to be made for every release, these are stipulated in the GTA document. This contains the generic agreements on organizing and performing topics. This makes it the overall approach for the setup of quality & testing activities that apply to (all) future releases. Based on the GTA, it is then decided whether to adapt, for each release, the quality & testing activities.  

In DevOps, a quality & test policy or GTA is not usually drafted as separate documents. But these are often part of the WoW, whereby – if necessary – a more detailed interpretation is given in a DoD. More information about GTA can be found in [Koomen 2006] and on GTA.