Product Risk and Benefit Analysis

Why a PRBA?

Traditionally, testing focuses on product risks and on mitigating those product risks. The natural starting point for the test strategy is the Product Risk Analysis (PRA). In the PRA, product risks are accumulated and classified to be able to distribute the available test effort in the most efficient way.

If executed the right way this is a very valuable tool for testing. However, there are some typical challenges when executing PRA’s and PRA sessions in the traditional way:

  • Although Product risks are of interest to the whole project – the focus on them is mainly from test. It is hard to interest others in the risks – other project members tend to focus more on the benefits of a project.
  • Because most traditional projects do not include testing from the start of the project, product risks are discussed once the project is well on its way and a lot of planning choices have already been made.
  • In general, in a test strategy you should want to test for biggest risks as early in the project as possible. However, the test planning has a large dependency on the project planning – generally you can start testing when the building phase has started. The PRA should be finished before the overall project planning is written in stone, to prevent limiting choices in the test planning.

The reason for this is that planning choices are often made based on other factors than risks. And the main one of those are the benefits, for example improving time to market or cutting costs. The items in a planning that deliver the highest business value, are generally built and released the earliest. Because testing is an integral activity within the project, the test planning and project planning must be aligned. Integrating the risk and benefit analysis feels natural and opens the way to integrate both time lines in the project planning.

An advantage that comes with this is that there are often sessions in which benefits of user stories are discussed, for instance the planning/refining session in Agile. You can use such sessions to talk about the risks as well. Another argument is that in short-iterative development and delivery methods there is no time available for large PRA sessions.

Conclusion

It makes sense to integrate the risk analysis and benefit analysis. The combined result has all the elements to make well-informed decisions about prioritising (benefits) and test depth (risks). Including the benefits makes it easier to convince other roles than testers to start with the risk analysis in an early phase. Preferably during the project planning.

Isn’t a risk the converse of a benefit?

A lot of people feel that a risk is simple the converse of a benefit or vice versa – No pain (risk) no gain (benefit), right?

Well, partly right. The difference is subtle but important. And it can be very useful to distinguish those two terms as they have different meanings and appeal to different groups within a project. Let’s look at an example to see how the different terms compare.

At a bank we are building a service that clients can use to check their current balance. This has a benefit associated with it. Clients receive a higher level of service. Let’s say this has a business value of 10 – a benefit of 10.

So what would be the converse of this benefit? That we don’t have the service. But that can be distinguished in two situations:

  • We do not have the service because we didn’t build it.
  • We do not have the service because we didn’t build it correctly. Client can access the function, but due to an error, no information is shown.

In the first case, we simply miss the benefit. So, we have a value of 0. There could be valid reasons for this.

In the second case, we have 0 business value from the function, but we also have damage: image damage because of clients who try the function do not get their data. So actually, the business value becomes negative because of this, for instance, -10. So zero business value, and -10 damage, a very undesirable situation.

Of course, a user story can have more risks associated with it.Maybe the service shows data, but not the correct data, or only works for a certain amount of clients. All those risks have their own specific damage.

So what is a benefit:
A benefit is positive business value that can be derived from a user story. The converse of that is no business value.

And what is damage:
Damage is negative business value that can be the result of an event that occurs as the result of the implementation of the user story.

And finally, what is a risk:
A risk is the product of the damage and the chance that the event actually occurs.

Note that one factor seems to be missing: the costs of creating the benefit. For the purpose of test preparation the costs of the solution is not a factor to take into account. It is up to the Product Owner or project manager to decide if the costs and benefits have the right balance. Most of the time the benefits must be higher than the costs. Sometimes costs can also be higher than the benefits, but lower than the potential damage. (For example in case of compliancy to changed laws). The PRBA can be input to this decision, however.

How does the PRBA work?

The following steps can be followed to execute a product risk and benefit analysis:

  • Preparation;
  • Determining benefits;
  • Determining relevant risk elements;
  • Determining risk class;
  • Making the benefit and risk backlog;
  • Completeness check.

In the rest of this paragraph these steps are elaborated in more detail.

Note: most of these steps can be or already are part of existing activities.

In the examples, a very detailed way of working is shown, to explain the thinking process and arguments to consider. But if keeping a table with as many different fields is experienced as too cumbersome, one could decide to limit the Benefits and Risks in terms of H/M/L (High, Medium, Low).

Preparation

The basis for a PRBA is generally a collection of user stories. It can be any kind of collection of functional changes, ideas or requirements. But user stories is the most common, so for the purpose of this description we will be talking about user stories which might already have been gathered during the sprint planning activities.

To prepare for the PRBA the following things need to be clear:

  • What are the user stories involved?
  • In what way are we going to gather benefits, damages and chances of failure?
    • Can we use an existing session for this or do we organise a separate session? As mentioned before, in Agile we preferably would use the planning/refining session for this.
    • If it is not convenient for all the relevant stakeholders to get together, can we get the data in another way? Interviews, questionnaires, special made tools like the one PRISMA uses?
    • Would it be possible to have an estimation done by experts? (Specialists, managers)

At the end of the preparation actvities a list is made of the user stories relevant to this PRBA. Note that this list is just a graphical representation for the sake of this explanation; the contents of below list can be integrated in user story comments and sprint planning.

User Story Benefits Benefit value Risk Risk value
User story 1 - Balance checking service for clients        
User story 2        
User story 3        
User story 4        
User story 5        
...        
Determining benefits

It is helpful to name the specific benefits associated with each user story, so that everyone involved understands the value of that specific story. For populating the column ‘Benefit value’ it is difficult to give an absolute value for the business value. It is easier to give a relative value, for example a number between 0 and 10.

Next step is to order the list according to the product backlog.

Generally, ordering the list is done by a product owner. If a product owner is not available, this can be done by a committee of stakeholders.

To take our example of the Balance checking service, possible benefits include “Better service to clients” and “Less calls to the service desk for balance inquiries”.

User Story Benefits Benefit value Risk Risk value
User story 4 benefit1, benefit2 10    
User story 2 benefit1, benefit2 7    
User story 3 benefit1, benefit2 7    
User story 1 - Balance checking service for clients better service to clients. Less calls to service desk for balance inquiries. 6    
User story 5 benefit1, benefit2 3    
...        
Determining relevant risk elements

The next step is to determine the risks associated with the implementation of the user story. What product risks do we have?

Finding the risk elements can be done in the standard ways. More about this can be found in the Wiki-page Determining relevant elements.

In this specific case the test goals are the user stories. In the case of the service for checking the balance we come to the following corresponding risks:

  • There might be a programming mistake causing it to break down completely.
  • There might be a programming mistake causing the program  to give incomplete or false data.
  • The service might be released in the wrong way causing it to break down completely.
  • The service might have a slow performance, causing clients to wait too long.

After this part of the analysis, the table could look like this:

User Story Benefits Benefit value Risk Risk value
User story 4 benefit1, benefit2 10 Risk4A  
      Risk4B  
      Risk4C  
User story 2 benefit1, benefit2 7 Risk2  
User story 3 benefit1, benefit2 7 Risk3A  
      Risk3B  
User story 1 - Balance checking service for clients better service to clients. Less calls to service desk for balance inquiries. 6 Programming mistake causing the program to break down.  
      Programming mistake causing the program to show incorrect data.  
      Performance is slow, causing clients to wait long or even stop using the service.  
User story 5 benefit1, benefit2 3 Risk5  
...        
Determining risk class

For populating the column ‘Risk value’ the standard PRA steps for determining the risk classcan be used. This is done in these three steps:

As stated before, if this is experienced as too detailed, one could decide to limit the risks in terms of H/M/L (High, Medium, Low). The most important goal of these steps is to give the product owner a feeling for which user stories have the most risk.

Making the benefit and risk backlog

From the individual tables of the previous steps, the PRBA-total overview is then drawn up and captured in the backlog. The purpose of this step is to give the participants and all other stakeholders a complete overview and provide a basis for the management on the identified risks and benefits.

Completeness check

The last step is to verify if the result of the previous steps is complete – or did we miss something? Special attention is required for the non-functional quality attributes!

If a feature/property has not yet been addressed, is that because there is hardly any risk attached to it or because it is forgotten?

To identify risks for all quality attributes one can use the overview of quality attributes such as mentioned in TMap Next® and the checklist ‘Risk factors per quality characteristic’.

The completeness check can be executed in two ways:

  • as part of the session, or
  • when the time is missing: in a separate session with the 2 or 3 most important stakeholders, with preparation by the test professional.

The final result is sent to the participants and the product owner for approval and asking if there are any additional comments. The product owner gives the final approval and takes a decision at any discussion points.

User Story Benefits Benefit value Risk Risk value
User story 4 benefit1, benefit2 10 Risk4A H
      Risk4B H
      Risk4C L
User story 2 benefit1, benefit2 7 Risk2 M
User story 3 benefit1, benefit2 7 Risk3A M
      Risk3B M
User story 1 - Balance checking service for clients better service to clients. Less calls to service desk for balance inquiries. 6 Programming mistake causing the program to break down. L
      Programming mistake causing the program to show incorrect data. M
      Performance is slow, causing clients to wait long or even stop using the service. M
User story 5 benefit1, benefit2 3 Risk5 H
...