Estimating

Aim

Based on the chosen test approach and depth, it should be considered whether the estimate of the development needs to be adjusted. Often, the estimate is included as standard in the development budget. Only when the requirements are more stringent than the basic quality will an increase to cover the extra test activities require to be estimated. Another point of focus is whether, with automation of the unit tests, the extra effort for this has been estimated. The estimate of the unit integration test will be determined much more by the scope and the chosen strategy.

Estimating the required effort for the total test process based on the test strategy, so that the client can approve it or request adjustment.

Method of operation

Master Test Plan

Estimating the effort for the total test process is done early on in the project and is based on the test strategy. Often, not all knowledge of the test object is available at this point. As a consequence, the accuracy of the estimate is limited. The size and/or complexity of the test object may change during the project. Furthermore, the test environments and any test tools represent a (significant) cost item. It is important for the test manager to make it clear to the stakeholders that the estimate is based on a number of assumptions and therefore will have to be detailed – and possibly adapted – later on. A possible solution is to use margins to represent the initial estimate for a master test plan.

The estimate in the master test plan constitutes the framework for the estimates per test level (e.g. system test, users acceptance test, and production acceptance test).

The required time for the various phases – Control, Setting up and maintaining infrastructure, Preparation, Specification, Execution and Completion – is then established for the test level.

Separate test activities are estimated within the test phases. The time necessary to create the MTP (Planning) is not included in the estimates. A fixed number of hours is usually estimated for this. After all, establishing the plan consists of executing clearly defined activities. The impact of e.g. the test object size on the time required to create the MTP is limited in this context. If there is an impact, it will be noticeable in particular during the activities “Analysing product risks” and “Determining test strategy”. In practice, some 60 to 160 hours are usually invested in creating the MTP.

As the estimate is made later in the test process and therefore at a lower level, more knowledge of the test object is available. Moreover, experiences from earlier on in the process can be used, making the estimate more accurate.

Independent of the level, creating the plan consists of the following generic steps:

Inventory the available material that can serve as a basis for the estimate.
Select (a number of) estimating techniques.
We recommend using multiple techniques in parallel. This makes it possible to compare the outcome of the various techniques. In addition to estimating techniques, it is worthwhile asking an experienced employee to make an estimate of the required time (expert estimate).
Determine the definitive estimate.
The aim of this step is to combine the outcomes of the previous step into one single estimate. If the outcomes vary little, taking an average will work. In other cases the differences have to be analysed. If an adequate estimate cannot be made after analysing the differences, the client must be consulted. The test manager explains the problems and makes proposals to achieve a correct estimate.
Present the outcome.
The aim of presenting the outcome is to provide insight to the business into the consequences of the selected test strategy and approach. It is important to show clearly which assumptions were made. Especially with an estimate created very early on in the process, assumptions will be involved that will become more concrete later on in the process.

Test Levels

An estimate may already be set out in the master test plan for the test level. Nevertheless, this remains a necessary step. The test manager has to determine, on the basis of the strategy created, how many hours and possibly how much money will be required. If these exceed the margins of the allocation contained in the master test plan, the test manager should work with the test manager of the overall test process to resolve these discrepancies. Either the strategy or the estimate will need to be amended.

In practice, the number of test hours required almost always is a factor reflected in the estimate. Another, less apparent, part of the estimate is the financial part. How much do those hours cost? Do they involve internal or external resources or even outsourcing? What are the fees? But also: how much do the test environment, test tools and work stations cost? If the client requires it, the test manager also must create a financial budget.

Subsequently, within a test level, the time required for the various phases, such as Planning, Control, Setting up and maintaining infrastructure, Preparation, Specification, Execution and Completion is established. At the start of each test phase, the test manager estimates the effort for the separate test activities.

Estimation techniques

Choosing the estimating techniques in particular is a step requiring experience. There are various estimating techniques to help assess the required effort. The steps to create an estimate are described above. An estimate for a Master Test Plan can be made on the basis of:

To increase the reliability of the estimate, we strongly recommend using both the organisation’s own experience figures and multiple estimating methods and techniques.

The sections below describe the estimating techniques, based on the following assumptions:

Estimating the test activities in the development phase (unit test and unit integration test) is an integrated component in estimating the realisation project and is not taken into consideration unless explicitly specified.
Where possible, experience figures are mentioned for the specified techniques. We explain the background of these figures. The figures shown must always be considered within the described context. They do not necessarily apply in a different situation.
One retest is included in all of the experience figures mentioned in subsequent sections.
Please refer to “Metrics”, for the structured collection and analysis of test estimating figures.

An adequate choice from the various techniques can be made with the use of two tables. These tables answer the following two questions:

Which technique is suitable for which level of estimating?
Which techniques are suitable for estimating which quality characteristics?

The answers to these questions are shown in the tables below.

	Master Testplan (MTP)	Detailed Tesplan (DTP)	Test Phase	Test Activity
Ratios	X	X	X	(X)
Size	X
WBS	X	X	X	X
Evaluation	X
Proportionate	X	X	X	(X)
Extrapolation			X	X
Test point analysis (TPA)	X	X

The possible estimating techniques are shown per quality characteristic in the table below. The table distinguishes between three different levels of testing depth for dynamic tests, i.e. ∨, ∨∨, and ∨∨∨ (low, medium, high)

	Eval-uation	Static test	UT	UIT	Implicit test	Dynamic test	Dynamic test	Dynamic test
Depth of Testing	∨	∨	∨	∨	∨	∨	∨∨	∨∨∨
Quality Characteristic	No. of pages ¹⁾		²⁾	²⁾	³⁾
Connectivity Data_controllability Flexibility		TPA-s				–	–
Continuity		TPA-s				Time box ⁷⁾ week	Time box ⁷⁾ month	Time box ⁷⁾ quarter
Effectivity		TPA-s				TPA ⁶⁾	TPA ⁶⁾	WBS
Efficiency		TPA-s				–	WBS
Functionality		TPA-s	Time box⁷⁾ hour	Time box ⁷⁾ hour		TPA	TPA	TPA
Infrastructure		TPA-s				–	–
Manageability		TPA-s ⁴⁾				WBS ⁵⁾	–
Maintainability		TPA-s				–	–
Performance Batch / Online		TPA-s				TPA WBS	TPA WBS	TPA WBS
Portability		TPA-s				WBS	TPA	TPA
Reusability		TPA-s				–	–
Security		TPA-s				TPA	TPA	WBS
Suitability		TPA-s				TPA ⁶⁾	TPA ⁶⁾	TPA
Testability		TPA-s				–	–
User-friendliness		TPA-s				WBS	WBS	WBS

Remarks

Several pages must be read when verifying a quality characteristic. Quality characteristics that have to do with functionality require a study of the pages on which the functionality is described. Other quality characteristics are generally described on other pages. This results in a varying number of pages per quality characteristic for verification.
It is assumed that the estimate of the standard test activities in the UT and UIT is part of the estimate of the realisation. If desirable, extra attention to testing during the UT and UIT can be specified. The estimating technique for this is an hour box, in which e.g. a supplement rate is added to the build effort (e.g. 10%) or part of the effort for the ST.
TPA-i is the component for implicit testing of a quality characteristic during dynamic testing of another quality characteristic. In TPA, this results in an additional supplement of 0.02 when determining the Qd. Please refer to the Detailed description of TPA.
TPA-s is the statistical component of TPA. Please refer to the Detailed description of TPA.
WBS = Work Breakdown Structure.
If effectivity and suitability are tested with the same test type/test technique, the effort is included once.
The time box and hour box are determined by factors outside the test process. Time box week in the table above means that testing takes a period of one week.

Tips

Sometimes a total budget is imposed by the client for testing. This has to be spread across the test phases and the characteristic/object part combinations. Some of the techniques described in (Ratios and Work Breakdown Structure, in particular) provide assistance here. It should also be examined whether the budget is adequate. The following tips are useful, but much depends on the experience of the test manager:

It is best to create an estimate by summing up the characteristic/depth-of-testing estimates (e.g. in PAT, a light performance test + a thorough security test = 120 + 200 hours = 320 hours).
Another option is to evaluate the total budget using a rule-of-thumb summing up for test levels: 15% of the total project budget of 5,000 hours for the ST, 20% for the AT = 750 and 1,000 hours respectively.
A third option is to employ a standard allocation formula for characteristics, established on the basis of a number of experiences. An example is to give Functionality, at risk category B, 70% of the total budget, at A, 80% and at C, 60%. You can also do this with a fixed number of hours, e.g. User-friendliness could be given 70 hours at category C, and up to 130 at category A (for an average system). This will make it easier to assess the real value of the individual estimates per test level /characteristic.
Compare the allocated budget with the budgets and total hours spent in the course of comparable exercises in the past, both in the organisation itself and if possible in other, comparable, organisations.

The estimate should be assessed for real value and should come out at around the level of the total assigned budget. Otherwise, adjustments will be necessary, by opting for a higher total budget, or testing fewer characteristics and/or testing in less depth.

Additional notes

The creation of an estimate for the test has a wide margin of uncertainty. It is important that the test manager make it clear to the stakeholders that the estimate is based on a number of assumptions and may therefore have to be revised later. A possible solution is the use of uncertainty margins. At the beginning of the test, the margin would be, for example, around 40%; at the start of the test execution this becomes around 25%, and somewhere in the middle it becomes around 10%.
The link between estimate and depth of testing (using specific test techniques) is opaque. How much extra time does, for example, the application of the elementary comparison test require as against the data combination test? Few past figures are available for this, and much is done on the basis of the test manager’s experience and intuition. • Various other factors (the quality of the testers, of the test object and test basis, test environment and test tools) can also exert significant influence on the estimate. These factors are either not known at the time the estimate is made or their effect on the estimate is very unclear. The test manager has to make assumptions here, and if necessary include them as assumptions in the plan, and most certainly should evaluate the assumptions as soon as possible.
It is difficult to ‘sell’ the required maintenance test effort to management. A general ‘testing image’ problem is that testing costs too much in management’s view. With testing during maintenance, that is reinforced by the fact that testing has a relatively big share in the maintenance effort – up to as much as 80%. This is partly because the total test costs consist of fixed and variable costs. Fixed costs refer to, for example, the effort required to prepare the test environment, or the execution of a ‘standard’ regression test; variable costs refer to, for example, the preparation and testing of implemented changes. With the testing of a small change, the percentage of fixed costs is high, since, irrespective of the size of the change, the environment must always be prepared and the regression test run. The greater the changes become, the more the fixed costs decrease in relative terms. For example, with the testing of a change, a 4-hour regression test is always run. If the testing of a change takes 8 hours in total, the fixed testing time amounts to 50% (4/8). If the testing of one or more changes takes a total of 40 hours, then this decreases to 10% (4/40). In general, the share of testing (fixed+variable) lies between 35% – 80% of all the maintenance activities. It is up to the test manager to make this clear to the client and to put the case for the importance (to the testing) of bigger, controlled releases, in which many changes are bundled, over the implementation of a constant procession of small changes.