To achieve quality at speed, automation of quality engineering activities is essential. And since the benefits of automation do not appear spontaneously, it is important for those in an organizing role to consider carefully.
Automation, much like software development in general, has five important areas:
- The (dynamic) organization in which it takes place,
- The people that perform it,
- The processes that people participate in,
- The infrastructure and tooling involved in creating, maintaining and executing automated tests,
- The engineering practices used for sustainable automation.
An effective automation strategy requires paying attention to each of these areas.
The Devops Organization
For the organization, the focus is on ensuring that the automation delivers significant business value. Start with defining the main goal for automation: a business-oriented goal, in line with the business goals for testing. Having automation find more bugs, for instance, is a really poor one, as automated regression tests do not catch many bugs unless the software is really poor quality. But providing fast feedback to the development team can be a good one, especially in a high-performance IT delivery context.
Often there are risks to the success of the automation effort from lack of experience, not knowing how easy automating will be for the software, and other factors. Several phases may be necessary to build up the insight, experience and skill that is needed to achieve the automation goal. These can include an assessment, a proof of concept, tool selection, a business case, and a pilot project. These take time and require continued, significant funding, regardless of whether it is done within the organization itself or outsourced. Throughout these phases, monitoring and control will enable the organization to track the increase of business value of the automation and adjust the strategy as needed.
People
Not only testers and automators are involved in the automation effort. Developers may support them by improving the testability of the software. Operations takes care of the infrastructure. Other stakeholders are on the business side or are responsible for compliance with rules and policies. All of these have needs or expectations that others can fulfill. To do that, they all need knowledge and various skills.
Within automation itself, as practiced in the development team, we can distinguish three kinds of activities: Requirements specification and testing actitivities call for the skills commonly associated with the tester role. Automating tests is software development and requires development skills to do well, even when using tools that are so easy to use that 'anyone can do it.' It is most suitable for automators and developers. Infrastructure (see below) includes a number of activities including CI/CD, test data management and service virtualization, calling for yet other skills. Such activities are picked up by people in a variety of roles, depending on their interests. Within a cross-functional team, the team as a whole is responsible for all activities, but making good use of people's strengths makes it easier.
As always, company culture is also an important factor. An important part of a culture of continuous improvement is acceptance of people (safely) learning from failing.
Processes
Modern automation practices benefit greatly from testing being involved early, or shift left, especially in requirements definition already. Modern specification techniques such as BDD (Behavior Driven Development) lead to requirements that include acceptance criteria in a form that is very suitable for automation. The automating itself should follow a development process including the use of (code) reviews and of course testing. Note that optimizing the value of the automation includes deciding when to delete automated test cases. All of this takes place within the context of the processes used by software development, such as Scrum or classic style development.
Infrastructure & tooling
The creation, maintenance and execution of automated tests requires suitable infrastructure. Most concerns relating to infrastructure are very similar for automation as they are for testing in general. This includes decisions on the number of test environments, how to handle test data, the use of a pipeline, and the use of mock systems to enable testing in isolation.
Testing multiple systems that work together brings some interesting challenges for the teams that work on them. Most of these are related to automated testing over all the systems. This requires at least one additional test environment for testing the systems together once they have been tested in isolation. Such environments are much like the ones used in earlier test stages. An important difference is that the test data in all the systems now needs to be aligned to ensure that data that is received from another system can be processed correctly. If two systems communicate about customers, for example, then their data on the customers that the tests reference needs to be consistent, and even consistently anonymized. Such data can be created specifically for testing the systems together, with separate test data for testing each system by itself, or it can be designed to be suitable for both kinds of test from the start.
The infrastructure can be optimized for the business goal of automation. An important consideration is what interfaces to use for addressing the software in test environments, possibly including some that were really meant as internal interfaces, such as backend services. Since execution speed is important, consider deploying the software differently from production environments, in a way that reduces communication overhead. The most likely candidate is skipping the user interface for at least some actions, going through services or directly into a database instead, for example.
As a tester performs a test, the state of the system can be inspected immediately, as described in chapter 34 of the book - Investigating and assessing the outcome of testing. In the case of an automated test, that state may have changed significantly or may even be inaccessible by the time someone looks into a failed test. For this reason, the automation solution should provide sufficient and clear information about what happened, either in the test report or in a separate log.
Test orchestration is the alignment of a large number of test automation tasks and other quality assurance related tasks for all teams involved in a CI/CD process, for optimized test execution. This term refers to both the process of orchestration and the technical implementation thereof in the pipeline. Thus, orchestration aligns human and automated tasks so that automated tests can be performed effectively and efficiently over multiple systems (or test tools). As with automated tests for a single system, automated tests involving several systems should be orchestrated from a single test that describes all the relevant steps at an appropriate level of abstraction. A single execution tool executes the test, no matter how many systems, interfaces and interface tools are involved in it. While the test cases for the separate systems will not be suitable for reuse in such scenarios, the underlying automation logic is.
If automated testing over multiple systems is a goal from the start, automation will be set up with the same automation solution for all systems or possibly with multiple automation solutions that are suitable for orchestration. Note that each system level test has a specific purpose and cannot be part of a test covering multiple systems. But the underlying logic for those system level tests can be reused to create actions that support the longer tests. In the case of orchestration over several automation solutions that are based on different tools, it is essential to have a mechanism to pass test data (parameters and results) back and forth between tools. Many COTS tools are meant to only run tests by themselves, so not orchestrated. It is often possible to introduce such a mechanism, but it may be more convenient to either choose a single automation solution or to choose tools that support orchestration out of the box.
Engineering practices
Finally, engineering practices are about how things are done rather than about what is done when (i.e. processes). An important practice is the creation, guarding and maintenance of a proper design for the automation solution. Others are having a coding standard and version management in place. The use of design patterns is relevant for solutions that are written in code. Besides the patterns used for regular development, automation benefits from patterns such as page object, screenplay and data builder.
Related Building Blocks
Organizing automation of quality engineering