Some context first:
What is SAP testing?
SAP testing is the test approach for testing the ERP software made by the German software developer SAP. This approach has a generic setup so that it can be translated quite easily to other ERP solutions like Oracle EBS or Peoplesoft. The test approach can be summarized in the acronym PRACTICES UP:
All these focus areas are more or less part of a complete or partially ERP implementation, upgrade or migration for test. See the separate wiki for more details. [Note that PRACTICES UP is the extended Sogeti interpretation of RICEFW.]
Risk Based Testing in SAP environments
The time available for testing is limited; not everything can be tested with equal thoroughness and intensity, the number of possible tests in an average SAP system is simply too large. You don’t want to test “Standard SAP” processes; however, Standard SAP doesn’t exist. You want to use the available testing time and effort on the riskiest areas first. Which are for example integrations, enhancements and the configuration of SAP end-to-end processes. The standard SAP steps are then in a big extend incorporated in the end-to-end flow.
Risk comes from NOT knowing what you’re doing: identify your software testing risks!
This means that choices must be made. In addition, the test efforts must be distributed as effectively and efficiently as possible over the total test scope. This principle is the basis of the risk-based test strategy.
The test strategy is based on thinking about quality risks: an SAP system must function in practice to an extend that no unacceptable quality risks for the organization exists. Where the delivery of a certain SAP process brings along many quality risks, thorough testing is at its place; the opposite on the other end of the spectrum is also true: 'no risk, no test' (where TMAP adds ‘no implementation’, because no risk actually means nobody has a problem if it fails, therefore nobody needs it, thus the feature should not be implemented at all).
SAP Quality Risk Analyses (SQRA)
Execute an SAP Quality Risk Analysis (SQRA) In order to identify where the risk resides in your SAP project. You can download the excel-template for SQRA here.
A quality risk is the chance that the process fails in relation to the expected damage if this failure occurs. The SQRA is a weighted risk approach where risk input from both Business and IT are merged into a risk classification:
- The Business input consists of the "Frequency of Use" and the "Business Impact" (Business: end user/key user/product owner)
- The IT input consists of the “Process Complexity” and the “Technical impact” (IT: Technical/ Functional SAP consultant (SAP module specialist))
Combined risk
The values for each risk component rate from 1 to 5 and have been predefined. The risk classification is the combined outcome of the “Business input” and the “IT input” risk score = (BI+FU)*(PC+TI).
| Score | Risk class |
|---|---|
| 4 - 16 | D (lowest) |
| 17 - 40 | C (low) |
| 41 - 75 | B (medium) |
| 76 - 100 | A (high) |
Example sheet
The results can be captured in an Excel sheet calculating the risk classes.
You can download the calculation sheet template here.
Test Life Cycle
Based on the test items’ location in the risk matrix, all SAP processes are prioritized. The result is ordering of all SAP processes, the most important / risky ones first.
This can be used throughout the whole development life cycle:
- Delivering test objects
- When specifying tests
- When executing tests
- When fixing anomalies (defects)
Prioritized
Based on the location of the process in the risk matrix (Risk Class), all processes are prioritized. The result is ordering of all processes, the most important / risky ones ones first. In addition to prioritization, a differentiated test approach for the processes can be defined based on their Risk Class.
Coverage
Based on the relative level of risk (Risk Class) test design techniques can be selected that will give the proper level of coverage.
Differentiated test approach
The test approach usually has two major aspects: the test depth used, and the priorities for testing. Test depth can be varied by using different test design techniques, e.g., using the decision table technique on high-risk test items and using ‘only’ equivalence partitioning for low-risk test items.
Other example practices used to define a differentiated test approach using risk classes:
- Static testing
Based on the identified risks one can choose to do more reviewing, e.g., inspection on those areas that are considered high risk. - Reviewing of test designs
For high-risk areas, the test designs (or test cases) can be reviewed with stakeholders or other testers. - Regression Testing
Of course, the outcome of the risk assessment can also drive the regression test, whereby high-risk areas should be most intensively covered in the regression test set.
Exit criteria
Different exit criteria, also called completion criteria, can be used for different risk levels. Requirements coverage or code coverage criteria should be stricter for higher-risk areas. Examples of other exit criteria that can be used:
- Percentage of test cases executed
- Number of outstanding defects
- Defect detection rate
Benefits
Running the tests in risk order gives the highest likelihood of discovering defects in severity order (“find the scary stuff first”).
Allocating test effort based on risk is the most efficient way to minimize the residual quality risk upon release (“pick the right tests out of the infinite possible tests in a SAP implementation”).
Measuring test results based on risk will allow the organization to know the residual level of quality risk during test execution and to make smart release decisions (“release when the risk of delay balances risk of dissatisfaction”).
If scheduling requires, dropping tests in reverse risk order reduces the test execution period with the least possible increase in quality risk (“give up tests you worry about the least”).
All of these benefits allow the test team to operate more efficiently and in a targeted fashion, especially in time-constrained/or resource-constrained situations.
Matrix
All ERP focus areas are housed in a structured adaptive matrix which distinguishes the best approach for each focus area. The following categories are made:
| Motivation | What is the reason this particular area is so important? |
| Preventive Measures | What measures can be taken as early as possible in the Software Life Cycle Development to find errors and improve quality? |
| Quality Attribute | Which quality attributes are best supported in which focus areas? (Functionality, Reliability, Performance, etc.) |
| Test Base | What type of documents are relevant for the focus area? |
| Where | Within which test type belongs the focus area (UT, SIT, FAT, UAT, PAT) |
| Coverage Based (Formal) | Which formal test specification technique is most useful in this focus area? (more applicable for SAP Tester) |
| Experience Based (Informal) | Which informal test specification technique is most useful in this focus area? (more applicable for End User) |
For a successful SAP Test Approach a combination is needed of all PRACTICES UP focus areas in relation with these categories.