Find your template, checklist or other download to help you in you tasks.
Find your way to be trained or even get certified in TMAP.
Start typing keywords to search the site. Press enter to submit.
With the real-life test (RLT), it is not the intention to test the system behaviour in separate situations, but to simulate the realistic usage of the system in a statistically responsible way. This test mainly focuses on characteristics, such as effectivity, connectivity, continuity and performance of the system under test. Many defects that are found with a real-life test are connected with a system’s use of resources:
To be able to test whether a system can handle realistic usage of it, that usage should be somehow specified. This also serves as a test basis and, in this context, is often referred to as the profile. The two most common types are:
The basic technique used here is:
For further explanation of these profiles and their simulation, refer to “Statistical usage: Operational profiles and load profiles“.
In practice, in a real-life test a mix of these profiles is often used. A particular loading of the system is simulated by carrying out realistic scenarios.
A profile is used in the setting out of one or more test goals of the real-life test. Examples of test goals are:
The aim here is to examine whether the available system resources are sufficient for the usual circumstances. This often involves a test with an average number of users who carry out interactive work, run overviews and carry out a number of small batch functionalities.
Execution of the real-life test is usually more complicated than that of other tests. In an environment in which the number of end users is not too great, you can have everyone work overtime for a weekend and carry out a previously established test scenario. However, use is increasingly being made of tools that simulate a realistic load in various ways. These are tools that simulate, for example, the number of users through the creation of virtual users, or tools that simulate a particular loading of the back-end of the system by offering transactions directly via the database management interface (hence without the use of the front-end or network).
It should be clearly determined in advance what and how measuring is to be done during the real-life test. Sometimes the measuring in itself also puts demands on the system, which can lead to distortion of the results. On the other hand, sufficient data is required to be able to carry out a satisfactory analysis in retrospect.
It can sometimes be difficult to assess the results of a real-life test. Occasionally, tests are not reproducible, because defects are often found that are caused by insufficient memory, lengthy use, etc. These kinds of defects (e.g. memory leaks) are difficult to reproduce, because there are almost always outside influences at play, which are impossible, or almost impossible, to control, such as the memory management of the operating system. In tracing the causes of any defects, logging and monitoring facilities could be used.
For the real-life test, the creation or establishing of correct profiles is the most important step. This takes place within step 1 “Identifying test situations”. The exact content of the test cases is less relevant than with most other test design techniques. The most important criterion is that reality regarding the size and frequency of use is approximated as closely as possible. This means that there is usually no point in creating logical test cases. The physical translation can often be made immediately to cover the required test situations.
The profiles can be seen as the situations to be tested. These indicate along general lines which types of actions (functions) are carried out over a particular period and the number of active users. This may be a number of daily cycles, e.g. a minimum, average and maximum cycle. A daily cycle consists, for example, of logging on, intensive use, lighter use during the lunch break, intensive use, logging out, backup and daily batches. Besides these, there could also be comparable weekly, monthly and annual cycles and specific processes, such as backup and recovery. There are various ways of obtaining the necessary information for the creation of an operational profile, load profile or a mix of these. Below are a number of them in random order:
An important consideration in creating a profile is the degree of detail. A more extensive and detailed profile will of course give a better reflection of reality, but will also lead to an increase in the testing effort for the specification and realisation of test cases and the execution of the real-life test. It should be ensured that in the profile all the system resources are used realistically. It is pointless to simulate significantly heavier usage than is usual in reality, as the result of such a test will tell us nothing. If, for example, the system is too slow under those circumstances, it does not mean that the system is unsatisfactory. If the system is not too slow, that only tells us that it is over-configured, but not by how much. Simulating significantly heavier usage is useful, of course, if the aim of the test is test is to determine the maximum load under which the system still performs acceptably.
To obtain further detail on the profile and the parts of the system-resource load to be tested, the following activities can be carried out. This takes in a number of the above mentioned methods of obtaining information for creating a profile:
In an organisation that processes bank transactions for various banks, 275,000 transactions are processed per hour with normal usage. These are divided into transaction types as follows (see table below):
The test cases are realised to correspond with the tasks and the relative frequency. For the example, this means the following spread across the test cases: 55% point of sale transactions, 33% direct debits, 7% ATM transactions and 5% credit transfers.
Possible aims of the test are:
The creation of physical test cases is often started at once.
For the real-life test, the exact content of the physical test cases is less relevant than for most other test design techniques. The only criterion is that reality regarding size and frequency of use is approximated as closely as possible. This sounds easier than it actually is. It has to be carefully considered how a particular usage or loading of the system can be realised or simulated. Additionally, test cases should be gathered or created for some tests, which have then to be carried out with the test execution. In contrast, in the execution of other tests the system has to be prepared with content in advance.
The creation of test cases can be done, for example, by physically setting out user scenarios, or, if an operational system already exists, by ‘tapping’ a representative test set.
For the testing of particular aspects of the system usage, the test cases can be realised by preparing a daily production (after processing) as real-life input. When using production data, bear in mind the privacy aspects. The devised user scenarios and actions that form part of the real-life test should also be distributed as realistically as possible among the users (testers) participating in the test.
The use of a tool, for example to simulate users or transactions, does not mean that user scenarios not need to be worked out. Even when a tool is used, these user scenarios form the basis of the test. In addition, the tool will have to be programmed or set so that it can carry out the user scenarios.
In the example of the transaction processing, the following physical features have been given to the point of sale transactions (see table below):
With test cases 2 and 3, besides the point of sale transactions, the fixed system load consists of 25 direct debits/sec, 5 cash withdrawals/sec and 4 credit transfers/sec. The transaction should be carried out to 95% < 5 sec, to 99% < 7 sec and to 100% < 10 sec.
* The point of sale transactions amount to an average of €100.00 and are spread over 4 banks.
As with most other tests, the creation of an appropriate starting point for a real-life test is often a challenge. However, with the real-life test, often there are additional points of focus:
For more information, refer to “Defining central starting point(s)”, in which this matter is discussed extensively.
An overview of all featured Test Design Techniques can be found here.
Test design techniques
