Real-Life Test (RLT)

Description

With the real-life test (RLT), it is not the intention to test the system behaviour in separate situations, but to simulate the realistic usage of the system in a statistically responsible way. This test mainly focuses on characteristics, such as effectivity, connectivity, continuity and performance of the system under test. Many defects that are found with a real-life test are connected with a system's use of resources:

• Crashing of transactions following lengthy use
• Crashing of transactions that are carried out in a particular sequence
• Inadequate response times and speed of processing
• Insufficient memory or storage space available
• Insufficient capacity of peripherals and data-communication network.

To be able to test whether a system can handle realistic usage of it, that usage should be somehow specified. This also serves as a test basis and, in this context, is often referred to as the profile. The two most common types are:

• Operational profile
  Simulation of the realistic usage of the system, by carrying out a sequence of transactions, which is compiled in a statistically responsible way.
• Load profile
  Simulation of a realistic loading of the system in terms of numbers of users and/or transactions.

The basic technique used here is:

• Statistically responsible simulation

For further explanation of these profiles and their simulation, refer to "Statistical usage: Operational profiles and load profiles".

In practice, in a real-life test a mix of these profiles is often used. A particular loading of the system is simulated by carrying out realistic scenarios.

A profile is used in the setting out of one or more test goals of the real-life test. Examples of test goals are:

• Testing with normal or average usage

The aim here is to examine whether the available system resources are sufficient for the usual circumstances. This often involves a test with an average number of users who carry out interactive work, run overviews and carry out a number of small batch functionalities.

• Testing with intensive usage
  The aim here is to examine whether there are sufficient system resources for even the most stressful, but realistic, circumstances. This often involves a test with a maximum number of users who carry out interactive work (peak loading) or a test in which certain transactions are carried out often and at length.
• Measuring the breaking point (stress testing)
  The aim here is to examine what the maximum load is under which the system will still perform to an acceptable level. This often involves a test with an increasing number of (simulated) users.
• Testing daily batches
  The aim here is to examine whether the available system resources are sufficient for the combination of a normal number of interactive users with the simultaneous execution of relatively demanding batch jobs.
• Testing nightly batches
  The aim here is to examine whether both the available system resources and the available time are sufficient for the (nightly / weekend) execution of big batch jobs.

Execution of the real-life test is usually more complicated than that of other tests. In an environment in which the number of end users is not too great, you can have everyone work overtime for a weekend and carry out a previously established test scenario. However, use is increasingly being made of tools that simulate a realistic load in various ways. These are tools that simulate, for example, the number of users through the creation of virtual users, or tools that simulate a particular loading of the back-end of the system by offering transactions directly via the database management interface (hence without the use of the front-end or network).

It should be clearly determined in advance what and how measuring is to be done during the real-life test. Sometimes the measuring in itself also puts demands on the system, which can lead to distortion of the results. On the other hand, sufficient data is required to be able to carry out a satisfactory analysis in retrospect.

It can sometimes be difficult to assess the results of a real-life test. Occasionally, tests are not reproducible, because defects are often found that are caused by insufficient memory, lengthy use, etc. These kinds of defects (e.g. memory leaks) are difficult to reproduce, because there are almost always outside influences at play, which are impossible, or almost impossible, to control, such as the memory management of the operating system. In tracing the causes of any defects, logging and monitoring facilities could be used.

Points of focus in the steps

For the real-life test, the creation or establishing of correct profiles is the most important step. This takes place within step 1 "Identifying test situations". The exact content of the test cases is less relevant than with most other test design techniques. The most important criterion is that reality regarding the size and frequency of use is approximated as closely as possible. This means that there is usually no point in creating logical test cases. The physical translation can often be made immediately to cover the required test situations.

1 - Identifying test situations

The profiles can be seen as the situations to be tested. These indicate along general lines which types of actions (functions) are carried out over a particular period and the number of active users. This may be a number of daily cycles, e.g. a minimum, average and maximum cycle. A daily cycle consists, for example, of logging on, intensive use, lighter use during the lunch break, intensive use, logging out, backup and daily batches. Besides these, there could also be comparable weekly, monthly and annual cycles and specific processes, such as backup and recovery. There are various ways of obtaining the necessary information for the creation of an operational profile, load profile or a mix of these. Below are a number of them in random order:

• Derive the profile from the current system or release
• Copy an existing profile of a system with comparable functionality
• Copy an existing profile with comparable load of the system resources
• Log how often each function is used
• Measure the load of the system with the aid of specific tools (monitors)
• Interview users, in which the key question is: "Which transactions do you carry out, how often and when, or which transactions do you expect to carry out on the new system?"

An important consideration in creating a profile is the degree of detail. A more extensive and detailed profile will of course give a better reflection of reality, but will also lead to an increase in the testing effort for the specification and realisation of test cases and the execution of the real-life test. It should be ensured that in the profile all the system resources are used realistically. It is pointless to simulate significantly heavier usage than is usual in reality, as the result of such a test will tell us nothing. If, for example, the system is too slow under those circumstances, it does not mean that the system is unsatisfactory. If the system is not too slow, that only tells us that it is over-configured, but not by how much. Simulating significantly heavier usage is useful, of course, if the aim of the test is test is to determine the maximum load under which the system still performs acceptably.

 

Tip

To obtain further detail on the profile and the parts of the system-resource load to be tested, the following activities can be carried out. This takes in a number of the above mentioned methods of obtaining information for creating a profile: Identifying user groups Identifying the user groups can take place, for example, on the basis of descriptions of the Administrative Organisation(AO), analysis of the target groups for whom any output is intended or security overviews at job level as user groups are usually strongly related to organisational positions and tasks. For the sake of the backup and recovery processes, the system administrators should not be forgotten Overview of tasks per user group A task is de fined as a number of successive actions that a user carries out in dialogue with the system for a particular purpose. The functional specifications are probably the most important source of information for this activity. Alternatives are the business processes, AO descriptions, the user manual, a prototype of the system, and any previous version of the information system. It is advisable to carry out this activity by means of direct contact (interviews) with users. Involving them during this activity usually clarifies things and provides all kinds of additional information. Determine the frequency per task per time unit Data relating to frequency are often already present from previous systems with comparable functionality. If not, they should be collected, for example, by keeping track in the present system of how often a particular function is started or executed, or by interviewing users. It is important here to bear in mind that it concerns estimates of the frequency and not precise measurements of a hundred percent accuracy. Determine the relative frequency The relative frequency is determined by dividing the number of occurrences of a particular task within a particular time unit by the total number of occurrences of all the tasks executed in the same period. The test cases are realised to correspond with the tasks and the relative frequency.

 

Example

In an organisation that processes bank transactions for various banks, 275,000 transactions are processed per hour with normal usage. These are divided into transaction types as follows (see table below): Transaction type Frequency(#/hr) Relative frequency Point of Sale transactions 150.000 0,55 Direct debits 90.000 0,33 ATM transactions 20.000 0,07 Credit transfers 15.000 0,05 Total 275.000 1,0 The test cases are realised to correspond with the tasks and the relative frequency. For the example, this means the following spread across the test cases: 55% point of sale transactions, 33% direct debits, 7% ATM transactions and 5% credit transfers. Possible aims of the test are: What is the maximum number of transactions that can be processed per second? What is the average lead-time of a transaction under normal or intensive usage, and does this fall within the agreed limits? Is the system proof against lengthy uninterrupted use?

 

2 - Creating logical test cases

The creation of physical test cases is often started at once.

3 - Creating physical test cases

For the real-life test, the exact content of the physical test cases is less relevant than for most other test design techniques. The only criterion is that reality regarding size and frequency of use is approximated as closely as possible. This sounds easier than it actually is. It has to be carefully considered how a particular usage or loading of the system can be realised or simulated. Additionally, test cases should be gathered or created for some tests, which have then to be carried out with the test execution. In contrast, in the execution of other tests the system has to be prepared with content in advance.

The creation of test cases can be done, for example, by physically setting out user scenarios, or, if an operational system already exists, by 'tapping' a representative test set.

For the testing of particular aspects of the system usage, the test cases can be realised by preparing a daily production (after processing) as real-life input. When using production data, bear in mind the privacy aspects. The devised user scenarios and actions that form part of the real-life test should also be distributed as realistically as possible among the users (testers) participating in the test.

The use of a tool, for example to simulate users or transactions, does not mean that user scenarios not need to be worked out. Even when a tool is used, these user scenarios form the basis of the test. In addition, the tool will have to be programmed or set so that it can carry out the user scenarios.

 

Example

In the example of the transaction processing, the following physical features have been given to the point of sale transactions (see table below): Nr Transaction type Parameters* Goal and expectation 1 Point of sale transactions Let number of transactions in 90 minutes rise from 5 tr/sec to 450 tr/sec. Determine breaking point (stress test). Expectation: ≥ 350 tr/sec.         2 Point of sale transactions 42 tr/sec (over 5 minutes) Determine lead time of a transaction with normal use 3 Point of sale transactions 120 tr/sec (over 5 minutes) Determine lead time of a transaction with normal use With test cases 2 and 3, besides the point of sale transactions, the fixed system load consists of 25 direct debits/sec, 5 cash withdrawals/sec and 4 credit transfers/sec. The transaction should be carried out to 95% < 5 sec, to 99% < 7 sec and to 100% < 10 sec. 4 Point of sale transactions 42 tr/sec (over 7 days) Test whether the system crashes with lengthy use * The point of sale transactions amount to an average of €100.00 and are spread over 4 banks.

4 - Establishing the starting point

As with most other tests, the creation of an appropriate starting point for a real-life test is often a challenge. However, with the real-life test, often there are additional points of focus:

• The test environment should be representative of the production situation
• Sizeable files are used
• Many users (testers) perform the testing
• A 'real' network should be available.

For more information, refer to "Defining central starting point(s)", in which this matter is discussed extensively.

 

An overview of all featured Test Design Techniques can be found here.