Estimating

Aim

Based on the chosen test approach and depth, it should be considered whether the estimate of the development needs to be adjusted. Often, the estimate is included as standard in the development budget. Only when the requirements are more stringent than the basic quality will an increase to cover the extra test activities require to be estimated. Another point of focus is whether, with automation of the unit tests, the extra effort for this has been estimated. The estimate of the unit integration test will be determined much more by the scope and the chosen strategy. 

Estimating the required effort for the total test process based on the test strategy, so that the client can approve it or request adjustment. 

Method of operation

Master Test Plan

Estimating the effort for the total test process is done early on in the project and is based on the test strategy. Often, not all knowledge of the test object is available at this point. As a consequence, the accuracy of the estimate is limited. The size and/or complexity of the test object may change during the project. Furthermore, the test environments and any test tools represent a (significant) cost item. It is important for the test manager to make it clear to the stakeholders that the estimate is based on a number of assumptions and therefore will have to be detailed – and possibly adapted – later on. A possible solution is to use margins to represent the initial estimate for a master test plan. 

The estimate in the master test plan constitutes the framework for the estimates per test level (e.g. system test, users acceptance test, and production acceptance test). 

The required time for the various phases – Control, Setting up and maintaining infrastructure, Preparation, Specification, Execution and Completion – is then established for the test level. Separate test activities are estimated within the test phases. The time necessary to create the MTP (Planning) is not included in the estimates. A fixed number of hours is usually estimated for this. After all, establishing the plan consists of executing clearly defined activities. The impact of e.g. the test object size on the time required to create the MTP is limited in this context. If there is an impact, it will be noticeable in particular during the activities "Analysing product risks" and "Determining test strategy". In practice, some 60 to 160 hours are usually invested in creating the MTP. 

As the estimate is made later in the test process and therefore at a lower level, more knowledge of the test object is available. Moreover, experiences from earlier on in the process can be used, making the estimate more accurate. 

 Independent of the level, creating the plan consists of the following generic steps:

1. Inventory the available material that can serve as a basis for the estimate.
2. Select (a number of) estimating techniques.       
   We recommend using multiple techniques in parallel. This makes it possible to compare the outcome of the various techniques. In addition to estimating techniques, it is worthwhile asking an experienced employee to make an estimate of the required time (expert estimate). 
3. Determine the definitive estimate.       
   The aim of this step is to combine the outcomes of the previous step into one single estimate. If the outcomes vary little, taking an average will work. In other cases the differences have to be analysed. If an adequate estimate cannot be made after analysing the differences, the client must be consulted. The test manager explains the problems and makes proposals to achieve a correct estimate. 
4. Present the outcome.       
   The aim of presenting the outcome is to provide insight to the business into the consequences of the selected test strategy and approach. It is important to show clearly which assumptions were made. Especially with an estimate created very early on in the process, assumptions will be involved that will become more concrete later on in the process. 

Test Levels

An estimate may already be set out in the master test plan for the test level. Nevertheless, this remains a necessary step. The test manager has to determine, on the basis of the strategy created, how many hours and possibly how much money will be required. If these exceed the margins of the allocation contained in the master test plan, the test manager should work with the test manager of the overall test process to resolve these discrepancies. Either the strategy or the estimate will need to be amended. 

 In practice, the number of test hours required almost always is a factor reflected in the estimate. Another, less apparent, part of the estimate is the financial part. How much do those hours cost? Do they involve internal or external resources or even outsourcing? What are the fees? But also: how much do the test environment, test tools and work stations cost? If the client requires it, the test manager also must create a financial budget. 

Subsequently, within a test level, the time required for the various phases, such as Planning, Control, Setting up and maintaining infrastructure, Preparation, Specification, Execution and Completion is established. At the start of each test phase, the test manager estimates the effort for the separate test activities.

Estimation techniques

Choosing the estimating techniques in particular is a step requiring experience. There are various estimating techniques to help assess the required effort. The steps to create an estimate are described in [Link {11.xml}chapter 11] "Estimation Techniques". An estimate for a master test plan can be made on the basis of: 

• Ratio figures
• Test object size
• Work Breakdown Structure (WBS)
• Evaluation estimation approach
• Proportionate estimation
• Extrapolation
• Test point analysis (TPA)

To increase the reliability of the estimate, we strongly recommend using both the organisation's own experience figures and multiple estimating methods and techniques. 

The sections below describe the estimating techniques, based on the following assumptions:

• Estimating the test activities in the development phase (unit test and unit integration test) is an integrated component in estimating the realisation project and is not taken into consideration unless explicitly specified. 
• Where possible, experience figures are mentioned for the specified techniques. We explain the background of these figures. The figures shown must always be considered within the described context. They do not necessarily apply in a different situation. 
• One retest is included in all of the experience figures mentioned in subsequent sections.
• Please refer to "Metrics", for the structured collection and analysis of test estimating figures. 

An adequate choice from the various techniques can be made with the use of two tables. These tables answer the following two questions: 

• Which technique is suitable for which level of estimating?
• Which techniques are suitable for estimating which quality characteristics?

The answers to these questions are shown in the tables below.

The possible estimating techniques are shown per quality characteristic in the table below. The table distinguishes between three different levels of testing depth for dynamic tests, i.e. ∨, ∨∨, and ∨∨∨ (low, medium, high)

Remarks

1. Several pages must be read when verifying a quality characteristic. Quality characteristics that have to do with functionality require a study of the pages on which the functionality is described. Other quality characteristics are generally described on other pages. This results in a varying number of pages per quality characteristic for verification.
2. It is assumed that the estimate of the standard test activities in the UT and UIT is part of the estimate of the realisation. If desirable, extra attention to testing during the UT and UIT can be specified. The estimating technique for this is an hour box, in which e.g. a supplement rate is added to the build effort (e.g. 10%) or part of the effort for the ST.
3. TPA-i is the component for implicit testing of a quality characteristic during dynamic testing of another quality characteristic. In TPA, this results in an additional supplement of 0.02 when determining the Qd. Please refer to the Detailed description of TPA.
4. TPA-s is the statistical component of TPA. Please refer to the Detailed description of TPA.
5. WBS = Work Breakdown Structure.
6. If effectivity and suitability are tested with the same test type/test technique, the effort is included once.
7. The time box and hour box are determined by factors outside the test process. Time box week in the table above means that testing takes a period of one week.

 Additional notes

• The creation of an estimate for the test has a wide margin of uncertainty. It is important that the test manager make it clear to the stakeholders that the estimate is based on a number of assumptions and may therefore have to be revised later. A possible solution is the use of uncertainty margins. At the beginning of the test, the margin would be, for example, around 40%; at the start of the test execution this becomes around 25%, and somewhere in the middle it becomes around 10%.
• The link between estimate and depth of testing (using specific test techniques) is opaque. How much extra time does, for example, the application of the elementary comparison test require as against the data combination test? Few past figures are available for this, and much is done on the basis of the test manager's experience and intuition. • Various other factors (the quality of the testers, of the test object and test basis, test environment and test tools) can also exert significant influence on the estimate. These factors are either not known at the time the estimate is made or their effect on the estimate is very unclear. The test manager has to make assumptions here, and if necessary include them as assumptions in the plan, and most certainly should evaluate the assumptions as soon as possible.
• It is difficult to 'sell' the required maintenance test effort to management. A general 'testing image' problem is that testing costs too much in management's view. With testing during maintenance, that is reinforced by the fact that testing has a relatively big share in the maintenance effort – up to as much as 80%. This is partly because the total test costs consist of fixed and variable costs. Fixed costs refer to, for example, the effort required to prepare the test environment, or the execution of a 'standard' regression test; variable costs refer to, for example, the preparation and testing of implemented changes. With the testing of a small change, the percentage of fixed costs is high, since, irrespective of the size of the change, the environment must always be prepared and the regression test run. The greater the changes become, the more the fixed costs decrease in relative terms. For example, with the testing of a change, a 4-hour regression test is always run. If the testing of a change takes 8 hours in total, the fixed testing time amounts to 50% (4/8). If the testing of one or more changes takes a total of 40 hours, then this decreases to 10% (4/40). In general, the share of testing (fixed+variable) lies between 35% - 80% of all the maintenance activities. It is up to the test manager to make this clear to the client and to put the case for the importance (to the testing) of bigger, controlled releases, in which many changes are bundled, over the implementation of a constant procession of small changes.