Test automation | DevOps

Many DevOps teams struggle with effectively setting up their continuous test automation. The World Quality Report describes two main challenges:  

  • Time to properly set up test automation is often too limited. 
  • Within an individual team there is no room to invest in proper test automation set up. 

The demand for continuous testing has created a renewed focus on test automation, due to several developments within IT. For example, technology as a whole, and commercial aspects. The popularity of agile approaches in all flavors from Scrum to DevOps is one of the main drivers of this development. There is a need for continuous improvement and "quality at speed", which consists of three pillars: shift-left, continuous testing and continuous feedback loop.  


Test automation is one of the main opportunities to meet the need for test speed, but also requires a structured approach with acceleration assets in order to effectively realize such a vision. This can be achieved through a solution that addresses challenges like business alignment and automation maintenance.  

On this page, we will discuss which tests should be automated and which test varieties should be selected (supporting shift-left), continuous testing, current test automation solutions, test orchestration (supporting continuous feedback loop), smart automated frameworks and "everything" as code automation.  For more in depth information about organizing test automation read the building block "Organizing automation of quality engineering".

How to determine which tests should be automated and which test variety should be selected 

Testers often struggle to determine what to test manually and what to test with automated test execution. The testing pyramid [Cohn 2009] can be used to make choices. The basic message of this testing pyramid is that testing should be automated as much as possible and be done as early as possible. And when deciding on test varieties, many testers find great support in the testing quadrants [Marick 2003, Crispin 2008]. Both models can be adjusted to a specific DevOps situation. The adjusted pyramid and/or adjusted testing quadrants are often described in a Way of Working or Definition of Done in DevOps or – at a higher level – are included in a Quality & Test policy. As an example, we have added our adjusted models. 

testing pyramid
testing quadrants

Testing Pyramid

Read more in the wiki.

Testing Quadrants

Read more in the wiki.


Continuous testing 

A specific subject in a test strategy for DevOps is the automation of tests. Since DevOps usually coincides with continuous delivery, most of the testing should be automatically performed during the process. Most test execution tasks should be automatically performed during the continuous integration and continuous delivery or continuous deployment process. Therefore, the term continuous testing is often used. 

CI/CD pipelines

Continuous testing is more than just automating tests. It is about the integration of test automation in the CI/CD pipeline, focused on getting feedback on the quality of the information system before and during delivery and deployment. Although we describe the ideal situation here, testing is not always part of a CI/CD pipeline. Alternative situations are possible, for example by testing COTS systems, SaaS systems, legacy systems or with a very specific pipeline of which testing cannot become a part.  

Continuous testing is putting the proper amount of test automation (based on the risks to be covered) in the CI/CD pipeline. In order to achieve continuous testing, it is important to focus on executing the right tests at the right time and and at the right speed. A well thought-through test strategy is therefore very important.  

Auto-generation of test cases 

A step that will help immensely in moving towards continuous testing is the auto-generation of test cases. Today, the creation of test cases is a time-consuming, manual process and auto-generation of test cases can cut down this time significantly while adding value to the testing process. Moreover, auto-generation techniques can focus on designing test cases based on user requirements, rather than the created code. This approach has proven to be helpful in improving product quality and reducing rework as the tests are designed with the intended functionality in mind. Many organizations have already moved to using techniques such as model-based testing with the help of tools commonly available on the market.  

Cloud adoption and virtualization of test environments 

Another step that will help move towards continuous testing is cloud adoption by virtualizing the entire test environment. Additionally, DevOps teams do not need to provision entire environments as they can use service virtualization tools to virtualize parts of the environment that are either unavailable for testing, are still in development, or are third-party services. With automation, complete, ephemeral (volatile) test environments can be created on demand and decommissioned when testing is complete, reducing both the time as well as the expenses associated with maintaining environments. Cloud technology can help with this, and the adoption of cloud for the creation of virtual test environments is a logical next step for automating the delivery pipeline. For more information about infrastructure please refer to the chapter on DevOps Infrastructure.  

How to achieve continuous testing? 

To achieve continuous testing, there are certain prerequisites that must come into place:  

  • Automation needs to happen within all selected test varieties, but keep in mind that 100% test automation is not a goal in itself.  
  • An infrastructure that supports the ability to manage test data. 
  • Testing has a business value – and a quality risk-based approach. 
  • The ability to create environments quickly and remove them if not needed anymore. 
  • The ability to populate the environments with the right data. 
  • The ability to auto-generate and maintain test cases. 
  • The size of the test set is small enough to be run with a short duration. 
  • Integration of test automation solutions. 
  • All DevOps people need to be involved in quality engineering. 
  • Chosen tools match the organizational as well as technical requirements to be used efficiently. 
    In practice, a "big bang" implementation of continuous testing apparently does not work. The following approach is recommended: 
  • Start small (but with the end goal in mind)  
    Instead of applying continuous testing across all DevOps teams, organizations should look at identifying a single team that is willing and able to adopt the practice and serve as an example of success to the rest of the organization.  
  • Get baselines  
    If you do not have something to base things on you won't know how to improve. Once you have your baselines (using metrics), you will know where your biggest pains are, and you can resolve them.  
  • Over time, use the collected data to make improvements. Add automation where it will have the biggest impact. Eliminate manual handoffs. Over time, you will see how much you have improved, and which areas need additional attention.

Test automation solutions 

There are a number of ways test automation is currently done. Depending on the time already spent by an organization this ranges from "green field" situations, where test automation has been discarded in the past or never got started due to technical challenges, to a variety of legacy or unstructured test automation approaches.  
Current test automation approaches can be grouped as follows: 

  • Framework automation  
    A test automation solution built in a single test execution tool. (e.g. Micro Focus UFT, Selenium)  
  • Hybrid framework automation  
    Managing the test automation solution based on a separate test management tool. (e.g. Micro Focus or Broadcom tool combinations)  
  • Tribrid framework automation  
    A combination and extension of the previous approaches results in a test automation solution providing the ability to be run stand-alone (single commandline test case execution), from a separate test management tool (as part of formal acceptance testing) as well as triggered as single test cases or scenarios from build and deploy tooling as part of the DevOps pipeline. The test cases become stand-alone code artifacts that can be executed as desired, as part of a larger whole (e.g. Robot Framework. Cucumber, Fitnesse)  
  • Scriptless framework automation  
    Building test cases out of predefined building blocks available to users with all levels of expertise. (e.g. Tricentis Tosca, Sikuli)  

Test orchestration


Test orchestration is the alignment of a large number of test automation tasks and other quality assurance related tasks for all teams involved in a CI/CD process, for optimized test execution. This term refers to both the process of orchestration and the technical implementation thereof in the pipeline.

The need for continuous improvement of test speed, coverage and quality has led to a situation in which organizations have "islands of automation" in their IT delivery lifecycle that are chained together with manual processes. Since the entire system can move only as fast as its slowest component, the whole process is unable to scale up to the demands of efficiency and speed.This situation is further complicated by increasingly complex architectures and the lack of visibility for all teams into the development pipeline.  

The quality and test policies are enablers for test orchestration as they align test automation activities over teams, which is a precondition for gathering information for real-time end-to-end-focused dashboards that inform about the confidence to establish the pursued business value.  

Eliminate "islands of automation" 

The next step is to address silos of automation with orchestration. In the realm of testing, test orchestration eliminates "islands of automation" by combining manual and automated tasks in a holistic fashion. It links together individual, automated tasks, which helps organizations move from spending time on manual handoffs, dependencies, wait times, and cycle waste to one in which test generation and execution are fully integrated as part of a fully automated and optimized continuous delivery (CD) pipeline.  

"Quality at speed"  

Testing today is more a sum of moving parts that need to be orchestrated together perfectly to achieve "quality at speed". Organizations also want to orchestrate tests as part of a release. Test results and quality data can therefore serve as feedback for the quality and risk of the release, as well as the efficiency of IT delivery as a whole.  

In this context, one of the ways of having a better view of the continuous testing within the CI/CD pipeline is to use customized dashboards, a practice that is being adopted by many DevOps teams and organizations. These dashboards give insight whether the quality risks that need to be covered have indeed been covered before (parts of) the IT system is (are) deployed to the next environment.  

Another promising development is the rise of artificial intelligence (AI) technologies that provide "smart" test orchestration. In the – near – future, with the addition of machine learning capabilities, systems will be able to automatically determine the tests that are required in the release and production cycles.  

How do we realize test orchestration? 

In order to realize test orchestration in DevOps: 

  • Create visibility of quality processes by implementing customized QA dashboards across the CI/CD pipelines
  • Ensure that DevOps teams are properly supported by specialized people for tasks that the teams don't have the knowledge and/or capacity for or which are at another organizational level (an example are end-to-end-regression-tests-on-demand by a service team).  
  • Optimize test automation tools and test operations across DevOps. 
  • Set up automated testing cycles to shift left and identify faults early in the lifecycle, thereby enhancing product quality. 
  • Automate the self-provisioning of test data. 
  • Automate provisioning of test environments with virtual services and test data. 
  • Leverage artificial intelligence and machine-learning technologies to optimize test cycles. This may include selecting the correct tests to be run according to what is in the pipeline, to provisioning test environments and running the tests. Using natural language programming, the test orchestrator can learn the pattern of test data usage and auto-generate test data with minimum manual intervention. 

Good orchestration helps create a coherent QA and test strategy across teams and optimizes the use of test automation and test environments. It also gives business owners a continuous view of the actual E2E state of quality in their core processes and applications. This is an approach that covers each activity and task in the release. These activities incorporate both dev and testing and allow testing tasks to be incorporated into each activity in DevOps. Audit trails are compiled as a result of these fully tracked processes. As the release progresses through the orchestrated pipeline, real-time information is available to every person in DevOps in a single source of truth. As each task is completed, it can be automatically moved to the next task if all requirements are met (i.e., if all tests are passed) or the person or team responsible for the next manual task can be automatically alerted, and reminders and alerts are provided and escalated if tasks are taking too long. This shortens wait times and encourages collaboration across teams. 

The future: smart automated frameworks 

Test automation has been around for about three decades. Most automation frameworks are designed to automate manual steps but are not intelligent. They are unable to react to changes, dynamically generate the resources they need, or understand and interpret results. 

In our view, the next step in test automation will be if people think of it less as a capability, and more as a platform – as a broad, connected, and intelligent space. When all tools and functions occupy a common environment, they can talk to one another to drive a full lifecycle value proposition, rather than remain focused on just one particular activity. When that point is reached, test automation will be smart, and broader, business-driven benefits can be given the priority they deserve.  

When you want to design a smart automated framework (refer to The World Quality Report), keep in mind that an intelligent automation framework:  

  • Is intuitive  
    It can check the code and create appropriate automated tests corresponding to the code changes.  
  • Is dynamic  
    For example, it can use cognitive computing techniques to identify and screen elements dynamically, and update object repositories.  
  • Can generate its own environment 
    For example, a framework can spin up environments at runtime through machine-readable definition files. 
  • Can prioritize 
    A good use case for this is a framework that can identify and execute critical test cases from an automated suite to achieve high fault yield per test case execution using algorithms such as random forest algorithm. 
  • Provisions its own test data  
    The data can be provisioned either by virtualizing, subsetting, or creating synthetic data.  
  • Supports continuous improvement of the test set and the test process. 

"Everything as code" automation 

There is of course more than test automation in DevOps. Developing software efficiently, especially with cloud, requires one to automate everything that is repeated. This is done by defining the following as much as possible as code:  

  • Infrastructure as code  
    Every piece of infrastructure, such as virtual machines, web apps, container registries, key vaults and so on must be defined with ARM templates (Azure), CloudFormation templates (AWS) or similar. This will enable to effectively version control the entire infrastructure. When a change of the infrastructure is required, the source template must be changed, not the target template, which in turn will be deployed using continuous deployment.  
  • Configuration as code 
    As infrastructure is deployed, there is also need for software configurations and installations on top of the infrastructure, such as ZooKeeper, Consul, DNS, or SSL certificates. By configuring these, the entire pipeline is automated, and no manual steps are required. This ensures that the desired state of services are always defined, met, and source controlled.  
  • Pipeline as code  
    By also defining the entire pipeline, by pointing out infrastructure, configurations, and ultimately the software code and how it is deployed, checked into source control as everything else, the team spends a maximum amount of time developing functionality instead of managing deployments. The pipeline configuration also defines test procedures, quality gateways and so on (test as code).  
  • Documentation as code  
    All documentation should be versioned in source control using a common markup language, such as Markdown and Doxygen. This enables the versioning of the software to follow the documentation versioning. Markdown or Doxygen can then be rendered to HTML or PDF, depending on the users' needs.