Structured Reviewing - ROI

There are a lot of sources, calculations and ideas about the calculation of the Return On Investment (ROI) of reviews and inspections. Some are supported by measured or calculated values, some are based on third parties. In most situations, the calculations are quite hard to understand. The ROI calculation which is described here should be simple to calculate, easy to understand and explain and flexible to mold to your situation. There is no scientific proof for this ROI, tweak your calculation on the go when you build up your own metrics database.

Assumptions

Above ROI calculation is simple to understand, however there are some major assumptions to take into account. Note that these assumptions can be justified if you have the correct metrics available. Also note that all other ROI-calculations you might find face the same assumptions. The ROI provides insight, but it never is exact science since a lot of human interaction is involved.

Then, for the assumptions:

• The Importance for each comment is assigned correctly (Major/Minor/Typo) –> can be discussed; if all involved agree, this probably will be OK most of the time.
• The ROI_factor is depending on the detail of measurements –> can vary based on the available metrics. Most of the time the ROI_factor will be a value somewhere within a bandwidth. If you take a rounded value somewhere at the lower boundary of the bandwidth you are on the safe side.
• The Time_spent is depending on the detail in which it’s provided by the people involved. –> the moderator can chase and check these to some extent. The exact values are not that important.

How to determine your ROI_factors

There are two ways:

1. Based on a fixed reference, for instance “in relation to the testing phase”.
2. Based on a moving reference, for instance “in relation to the next phase” or “in relation to the next two phases”.

The first is preferred, since the testing phase is normally the first planned activity where is checked if the delivered product is conform design and specific (extra) costs that are related to production defects (loss of customers etc.) can still be prevented. Also measuring is easy since normally a strict defect tracking system is available.

The second option is only reliable all involved are really quality minded and if it’s possible to measure the amount of issues logged and fixed.

In reality, the ROI will be somewhere in between both.

Examples

Using interviews the following questions can be answered. Maybe the time it takes can be measures for some amount of time to underpin the ‘gut feelings’ that may exist.

“In relation to the testing phase”
How many hours does it (on average) take to

• re-script and re-execute a test case? (e.g. 2-4 hours)
• rebuild a component? (incl. unit test, version management etc.) (e.g. 2-8 hours)
• redesign a component? (incl. consistency check, document management etc.) (e.g. 4-16 hours)
• restructure an use case? (incl. consistency check, document management etc.) (e.g. 8-24 hours)
• recreate a requirement? (incl. refunding, consistency check, requirement life cycle management, discussions etc.) (e.g. 16-40 hours)

Putting these values in a cumulative graph for the visually oriented shows the growing costs:

In this situation we assume that all errors will be found and fixed in the testing phase.

“In relation to the next phase”
Using the same answers as above, but then used in a non-cumulative way provides a less steep graph:

In this situation we assume that all errors will be found and fixed at the beginning of the phase following on the phase in which the error was introduced.