Aim
The creation of as reliable as possible a planning for the test level, so that the client can make allowance for this and can manage accordingly. The principle of the planning is to find the most significant defects (the finding of which belongs within the scope of the test level) first.
Method of operation
Based on the planning of the system development process and on the master test plan, a planning for the test level is created. The test manager indicates the start and end date per phase and the products to be delivered. The planning should cover at least:
- Activities to be carried out (at activity level per phase)
- Correlations with and dependencies on other activities (within or beyond the test level and between the various phases and other test levels)
- Time to be spent per phase
- Required and available resources (people and infrastructure)
- Required and available turnaround time
- Products to be delivered.
Depending on the client's requirements, the financial consequences of the choices made should be made visible in a financial planning. This means, for example, the setting out of the costs in terms of time for the (internal and external) personnel, training, workstations, test environment and test tools.
In more detailIn creating a planning, the following principles apply:
|
Required informationIn order to set up a planning based on the estimated effort, additional information is required concerning the following subjects:
|
The planning is reflected in, for example, a network planning or a bar chart, depending on the method used within the organisation. This book does not deal with planning techniques, because for the test process the test manager employs standard planning techniques that are not specific to testing.
Example of activities planning
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Example of milestone planning
|
TipWhen planning resources, indicate from which point it is no longer possible to accommodate imminent overrun by deploying extra people. Sometimes an environment is so complex or specific that an 'extra hand' will no longer gain time. It is not pleasant to have to explain this when the moment has already arrived and the project leader is already busily engaged in arranging extra people for the test team – even less so when those extra people are already being introduced to the team… |
An aspect of planning related to quality is when a test level is ready and the test object can be transferred to the following test level or to production. In other words, what can the 'next' test level expect after the 'previous' test level is completed. In order to make these expectations explicit, requirements are set according to the result of the test level. In practice, these requirements are also known as exit criteria. With increasing outsourcing, it becomes more and more important to establish clear exit criteria to prevent the supplier from delivering inadequate quality.
In more detailExit criteria can relate, for example, to the number of issues in a particular risk category that may still be open, the way in which a certain risk is covered (e.g. all the system parts with the highest risk category have been tested using a formal test design technique), or the depth in which the requirements should have been tested. From within the master test plan, the exit criteria are applied to the test level. If that is not the case, or if there is no master test plan, the test manager should agree the criteria with the client. The box below shows a number of concrete examples of exit criteria:
An important point of focus as regards the above-mentioned criteria is that clear definitions should be agreed by all the stakeholders of what a particular category of severity is and what is meant by 'agreed depth of testing and test method'. In practice, a lack of clarity here can lead to heated discussions. Similarities and differences between acceptance and exit criteriaAnother term for exit criteria[Register: exit criteria] that is used is 'acceptance criteria[Register: acceptance criteria]', as discussed in sub[Link {6.xml#6.2.2}section 6.2.2]. Besides the fact that acceptance criteria may be a broader term than exit criteria, another difference is that acceptance criteria come at the end, i.e. at acceptance, and exit criteria at the transfer from one test level to another, or to production. The figure below illustrates this. 
|
TipSuspend- and resume criteriaIn some, particularly formally set up, tests, so-called suspend- and resume criteria[Register: resume criteria][Register: suspend- and resume criteria] may be defined in the plan. These criteria indicate under which circumstances the testing is temporarily suspended and then resumed. Examples of suspend criteria are that testing has to stop when a particular infrastructural component is not available, or if a test-blocking defect is found. A resume criterion may be that with the lifting of the suspend criterion the testing of the system part /function/component has to take place entirely anew. |
Feedback
When the test manager has created a planning, this is the time to agree matters with the client. If the test strategy setup and subsequent estimate of required effort and planning are not acceptable, then these steps are repeated. With this, the client and test manager consider whether to test certain aspects in lesser depth, so that time and/or money is spared, but a higher level of risk is accepted, or the other way. To facilitate communication, the test manager refers here to the original test goals. Where a master test plan exists, the coordinating test manager is involved here, but the client makes the final choice.
An amended strategy is illustrated below, with less test depth indicated by ⇓ instead of ∧ and more test depth by ⇑.
ST example
| Characteristic | RC MTP | ST MTP | Subsystem 1 | Subsystem 2 | Total sys | |
|---|---|---|---|---|---|---|
| Functionality | n/a | n/a | A/ ∧∧⇓Functional, regression | B/ ∧⇓Functional, regression | C/ ∧Integration, multi-user | |
| Performance online | B/ | ∧ | C/ ∧Random testing in ST environment | |||
| ... |
UAT example
| Characteristic | RC MTP | UAT MTP | Subsystem 1 | Subsystem 2 | Total sys | |
|---|---|---|---|---|---|---|
| Functionality | n/a | n/a | A/ ∧⇓Functional | B/
| C/ ∧Regression | |
| User friendliness | B | ∧∧ | C/I | B/ ∧⇓Usability | ||
| Security | A | ∧ | ||||
| --- authorization matrix | B | -/S Authorisation test | -/S Authorisation test | B/ ∧∧Process test | ||
| --- application | C | C/ ∧Penetration test (black box) | ||||
| Suitability | B | ∧∧∧ | B/ ∧∧ Scenario test | C/ ∧ Scenario test | A/ ∧∧⇓ Process test | |
| ... |
The amended strategy leads to another estimated effort and planning, and also to an indication of bigger (or even smaller) product risks, translated into terms that are comprehensible to the client (referring back to the product risk analysis with test goals, characteristics and object parts).
In addition to the feedback on strategy, budget and planning, the test manager discusses with the client the use of tolerances in the execution of the test process. These are boundaries within which the test manager is not required to ask the client's permission. For example, a tolerance of 5% is often agreed for the budget. For the planning, it may be agreed that only deviations from project milestones will require discussion. With strategy tolerances, for example, the client's advance permission is not required for testing a characteristic/object part in one greater or lesser degree of depth.
Products
Planning for the test process
Exit criteria
Optional: tolerances for strategy, budget and planning
Optional: suspend and resume criteria (above products are established in the test plan)
Strategy, budget and planning feedback to/from the client.
Techniques
Not applicable.
Tools
Workflow tool
Planning and progress monitoring tool.
Return to Determining the overall test planning
Continue to:
Topics plotted for Sequential IT delivery models