Why test automation?
[This section is a SAP specific extension of the section "Test Automation (Devops)"]
Test automation focuses on automating the execution of test scenarios (preferably for the entire end-to-end business process) to analyze the quality of the application(s). These test scenarios are coming from realistic business processes, which might also consist of back-end calls (API testing) to help with speed and efficiency and enable very frequent execution of tests (e.g.: in a CI/CD pipeline). In the automation process checks/verifications are a vital part of building tests to ensure that the actions are taken correctly, since automation is in essence telling a machine what to do and validating what’s happening in the application to ensure that the scripts are executed correctly and failures/ unexpected results are shown. Examples for SAP would be to verify the message in the status bar and store the created/changed order for later usage to ensure the action was performed successfully.
When deciding on implementing test automation, the reason for starting it is key to being successful. Ensuring speed in the testing process might be a reason. A test automation tool itself cannot solve issues such as delays within your systems or missing data. Stakeholders can do this by being aware of the shortcomings in the processes that are related to quality engineering & testing and selecting tools to mitigate these shortcomings. By doing so, more benefits of test automation can be achieved. The effects of using test tools is described in "Tooling (Devops)".
Knowing on which layer to automate and which tests to select for different purposes is a vital part of effective test automation, for more information see the description of the test automation pyramid. Overall, the main opportunity for SAP Test Automation would be for regression testing purposes, since this test variety tends to be linear in nature, which ensures predictability of the outcome of a test. This helps to identify where issues lie that need to be solved.
Examples of tools for automated test execution are:
- Tricentis Tosca
- SAP S/4HANA Cloud Test Automation Tool (TAT)
- Worksoft certify
- Selenium
- Cucumber
- Robot Framework
- Qualibrate
- And many more
Prerequisites for starting test automation in an SAP environment
Business success factors:
- Clearly specified end-to-end test cases (including expected outcome) are available.
- Common understanding of the impact of processes and cut-off points.
- Awareness of the trail the data will travel, and how this impacts specific processes/triggers.
- Understanding functional changes of new releases for maintaining and implementing these changes in the automated test cases.
- Business alignment by engaging all stakeholders to share advantages of SAP Test automation as well as challenges faced.
- General understanding of SAP Test Automation possibilities and limitations.
Technical success factors:
- Infrastructure for the tool to be deployed, with access to the applications to be tested, also considering cloud environments for unattended execution.
- Access rights for the test automation engineer, and also for the test automation tool itself, to the different systems, for automation purpose with the right authorizations for different roles.
- Configuring SAP for allowing scripting to be possible with the test automation tool.
- Selected tool is the right fit for steering SAP, the test environment is stable (or mitigating factors are considered) and test data is available /can be created to replicate the scenarios to be tested.
Dealing with complexity of SAP test automation
SAP brings complexity for test automation compared to other systems due to the large overlaps in the processes and many different variables such as Master Data & User role set-up. There are quite some dependencies when testing end-to-end, as an entire E2E-process can consist of many different systems that can be regionally different. It is important to carefully select test cases for automation of the end-to-end test, to prevent it from being too large, and to prevent overlap with other tests.
Reusability (Master Data & User Roles)
Within test automation it is useful to work with reusable steps and for SAP processes this is quite easily done through making blocks for different transaction codes or Fiori tiles that only differ in the data that has to be processed throughout the application or user that will perform the action. Reusable blocks also ensure that maintenance can be done in an efficient way, considering it only needs to be done in one place when parameters change for a sales order or extra steps are added.
Bottlenecks in test automation in an SAP environment
The following challenges and bottlenecks may be experienced when applying test automation. Discussing these bottlenecks upfront of the SAP Test Automation project will create awareness and helps solving these issues:
Automation is an isolated island
- This leads to not being aware of new releases and the impact on test automation.
- Not getting the benefits of collaboration and awareness of what test automation can do for different teams.
- Overlap in what is tested manually and through automated testing.
Bad test cases
- The results of the automated test execution are not useful when the test cases that are automatically run do not align with the goals of the test variety and the information needed by the stakeholders.
- Incomplete, incorrect and changing master data are concerns for effective test automation.
Thinking test automation assures higher speed of the testing process
- Sometimes parts must be executed manually.
Instability of the environment/ technical issues
- System time outs,
- Installation/ configuration errors,
- Network/ hardware errors,
- Application errors.
Complex end-to-end processes
- With multiple SAP and non-SAP (e.g.: 3rd party and legacy) steps, are not well defined nor clear on what the outcome should be, thus these should be broken down into logical scenarios to avoid SAP integration dependencies for SAP Test Automation.