With the real-life test (RLT), it is not the intention to test the system behaviour in separate situations, but to simulate the realistic usage of the system in a statistically responsible way. This test mainly focuses on characteristics, such as effectivity, connectivity, continuity and performance of the system under test. Many defects that are found with a real-life test are connected with a system's use of resources:
- Crashing of transactions following lengthy use
- Crashing of transactions that are carried out in a particular sequence
- Inadequate response times and speed of processing
- Insufficient memory or storage space available
- Insufficient capacity of peripherals and data-communication network.
To be able to test whether a system can handle realistic usage of it, that usage should be somehow specified. This also serves as a test basis and, in this context, is often referred to as the profile. The two most common types are:
- Operational profile
Simulation of the realistic usage of the system, by carrying out a sequence of transactions, which is compiled in a statistically responsible way.
- Load profile
Simulation of a realistic loading of the system in terms of numbers of users and/or transactions.
The basic technique used here is:
- Statistically responsible simulation
For further explanation of these profiles and their simulation, refer to "Statistical usage: Operational profiles and load profiles".
In practice, in a real-life test a mix of these profiles is often used. A particular loading of the system is simulated by carrying out realistic scenarios.
A profile is used in the setting out of one or more test goals of the real-life test. Examples of test goals are:
- Testing with normal or average usage
The aim here is to examine whether the available system resources are sufficient for the usual circumstances. This often involves a test with an average number of users who carry out interactive work, run overviews and carry out a number of small batch functionalities.
- Testing with intensive usage
The aim here is to examine whether there are sufficient system resources for even the most stressful, but realistic, circumstances. This often involves a test with a maximum number of users who carry out interactive work (peak loading) or a test in which certain transactions are carried out often and at length.
- Measuring the breaking point (stress testing)
The aim here is to examine what the maximum load is under which the system will still perform to an acceptable level. This often involves a test with an increasing number of (simulated) users.
- Testing daily batches
The aim here is to examine whether the available system resources are sufficient for the combination of a normal number of interactive users with the simultaneous execution of relatively demanding batch jobs.
- Testing nightly batches
The aim here is to examine whether both the available system resources and the available time are sufficient for the (nightly / weekend) execution of big batch jobs.
Execution of the real-life test is usually more complicated than that of other tests. In an environment in which the number of end users is not too great, you can have everyone work overtime for a weekend and carry out a previously established test scenario. However, use is increasingly being made of tools that simulate a realistic load in various ways. These are tools that simulate, for example, the number of users through the creation of virtual users, or tools that simulate a particular loading of the back-end of the system by offering transactions directly via the database management interface (hence without the use of the front-end or network).
It should be clearly determined in advance what and how measuring is to be done during the real-life test. Sometimes the measuring in itself also puts demands on the system, which can lead to distortion of the results. On the other hand, sufficient data is required to be able to carry out a satisfactory analysis in retrospect.
It can sometimes be difficult to assess the results of a real-life test. Occasionally, tests are not reproducible, because defects are often found that are caused by insufficient memory, lengthy use, etc. These kinds of defects (e.g. memory leaks) are difficult to reproduce, because there are almost always outside influences at play, which are impossible, or almost impossible, to control, such as the memory management of the operating system. In tracing the causes of any defects, logging and monitoring facilities could be used.
Points of focus in the steps
For the real-life test, the creation or establishing of correct profiles is the most important step. This takes place within step 1 "Identifying test situations". The exact content of the test cases is less relevant than with most other test design techniques. The most important criterion is that reality regarding the size and frequency of use is approximated as closely as possible. This means that there is usually no point in creating logical test cases. The physical translation can often be made immediately to cover the required test situations.
1 - Identifying test situations
The profiles can be seen as the situations to be tested. These indicate along general lines which types of actions (functions) are carried out over a particular period and the number of active users. This may be a number of daily cycles, e.g. a minimum, average and maximum cycle. A daily cycle consists, for example, of logging on, intensive use, lighter use during the lunch break, intensive use, logging out, backup and daily batches. Besides these, there could also be comparable weekly, monthly and annual cycles and specific processes, such as backup and recovery. There are various ways of obtaining the necessary information for the creation of an operational profile, load profile or a mix of these. Below are a number of them in random order:
- Derive the profile from the current system or release
- Copy an existing profile of a system with comparable functionality
- Copy an existing profile with comparable load of the system resources
- Log how often each function is used
- Measure the load of the system with the aid of specific tools (monitors)
- Interview users, in which the key question is: "Which transactions do you carry out, how often and when, or which transactions do you expect to carry out on the new system?"
An important consideration in creating a profile is the degree of detail. A more extensive and detailed profile will of course give a better reflection of reality, but will also lead to an increase in the testing effort for the specification and realisation of test cases and the execution of the real-life test. It should be ensured that in the profile all the system resources are used realistically. It is pointless to simulate significantly heavier usage than is usual in reality, as the result of such a test will tell us nothing. If, for example, the system is too slow under those circumstances, it does not mean that the system is unsatisfactory. If the system is not too slow, that only tells us that it is over-configured, but not by how much. Simulating significantly heavier usage is useful, of course, if the aim of the test is test is to determine the maximum load under which the system still performs acceptably.
To obtain further detail on the profile and the parts of the system-resource load to be tested, the following activities can be carried out. This takes in a number of the above mentioned methods of obtaining information for creating a profile:
In an organisation that processes bank transactions for various banks, 275,000 transactions are processed per hour with normal usage. These are divided into transaction types as follows (see table below):
The test cases are realised to correspond with the tasks and the relative frequency. For the example, this means the following spread across the test cases: 55% point of sale transactions, 33% direct debits, 7% ATM transactions and 5% credit transfers.
Possible aims of the test are:
2 - Creating logical test cases
The creation of physical test cases is often started at once.
3 - Creating physical test cases
For the real-life test, the exact content of the physical test cases is less relevant than for most other test design techniques. The only criterion is that reality regarding size and frequency of use is approximated as closely as possible. This sounds easier than it actually is. It has to be carefully considered how a particular usage or loading of the system can be realised or simulated. Additionally, test cases should be gathered or created for some tests, which have then to be carried out with the test execution. In contrast, in the execution of other tests the system has to be prepared with content in advance.
The creation of test cases can be done, for example, by physically setting out user scenarios, or, if an operational system already exists, by 'tapping' a representative test set.
For the testing of particular aspects of the system usage, the test cases can be realised by preparing a daily production (after processing) as real-life input. When using production data, bear in mind the privacy aspects. The devised user scenarios and actions that form part of the real-life test should also be distributed as realistically as possible among the users (testers) participating in the test.
The use of a tool, for example to simulate users or transactions, does not mean that user scenarios not need to be worked out. Even when a tool is used, these user scenarios form the basis of the test. In addition, the tool will have to be programmed or set so that it can carry out the user scenarios.
In the example of the transaction processing, the following physical features have been given to the point of sale transactions (see table below):
4 - Establishing the starting point
As with most other tests, the creation of an appropriate starting point for a real-life test is often a challenge. However, with the real-life test, often there are additional points of focus:
- The test environment should be representative of the production situation
- Sizeable files are used
- Many users (testers) perform the testing
- A 'real' network should be available.
For more information, refer to "Defining central starting point(s)", in which this matter is discussed extensively.
An overview of all featured Test Design Techniques can be found here.