End-to-End testing

In today's IT world, organizations increasingly work with external partners. In addition, more and more organizations are adopting an Agile-at-scale approach in which cooperation throughout the cross-organization IT delivery process is essential. In these types of environments, control over end-to-end quality is becoming increasingly important. 

What is End-to-End testing?

End-to-End testing is a methodology used to test whether the flow between multiple applications is performing as designed from start to finish. For example, when a customer orders a new product at an online webshop, it takes several applications and systems to get your purchase at your front door (online shopping portal, billing services, order databases, warehouses, logistics). The purpose of carrying out End-to-End tests is to identify system dependencies and to ensure that the right information is passed between various system components and systems. As a result, the End-to-end test cycle can prevent financial and material damage, as well as damage to the company's image. Furthermore, the tests can increase awareness at management level about the performance and stability of an End-to-End chain of systems.

Four different processes/roles can be identified when using the End-to-End testing approach.

• End-to-End test architecture:
  • The process where systems and components in an End-to-End chain are identified on high-level. Analysis of End-to-End testing risks and feasibility takes place, as well as the determination of overall scope and strategy to come up with an executable and financially sound plan.
• End-to-End test management: 
  • To potentially distinguish multiple sub-chains in the End-to-End cycle and to define a plan that covers limitations, strategy, planning and financial aspects of these sub-chains.
• End-to-End test coordination: 
  • Specification and documentation of a proper test infrastructure, followed by the actual implementation of this infrastructure.
• End-to-End test executions: 
  • Preparation and execution of the End-to-End tests by defining test cases and pre-conditions which can be stored in a End-to-End testing script.

The following two forms of analysis are important in the End-to-End testing context:

1. Chain risk analysis:

   Processes and systems in a chain can fail in multiple ways resulting in different types of damage. A chain risk analysis can be performed to get insight into the risk each process or system has in the chain to fail and what damage will occur.

    

2. Feasibility analysis:

   To assess the feasibility of and End-to-End test, one has to take into account multiple elements such as the availability and accessability of specific knowledge, the dynamical aspects of the organisations and management, and also the possibilities or limitations in the project and test environments. The goal is to identify complicating and blocking elements in the landscape of systems.

The complexity in the chain of End-to-End testing and the difficulty to set up a End-to-End test infrastructure are two factors that emphasize the importance of a proper chain risk analysis and feasibility analysis.

More information about how to prepare and execute a chain risk analysis or feasibility analysis can be found in the book "TMap NEXT: End-to-End testing". Multiple documents are available in the downloads section of this website that provide templates and checklist to help you execute the chain risk analysis and feasibility analysis.

People involved

• Test architect
• Test manager
• Test engineers that are part of the End-to-End chain

Artifacts

The artifacts of an End-to-End test can be the same as of any other test. In most cases, the client is mostly interested in finding bugs, so a bug report is the most obvious artifact. But in most cases, test plans, test cases and all other test artifacts can be made as well.

Success factors

It is important to keep track of the scope of the End-to-End test to make sure only the most important and used interfaces are subject of the test. This should be covered by the feasibility analysis and chain risk analysis.

End-to-End Quality Engineering at scale

This section considers the management of end-to-end quality in a scaled environment, where control on quality is key when implementing changes in the solution. To control end-to-end quality, organizing an end-to-end test is relevant, but at least as important is to organize prerequisites such as Test environments, Test data, Stakeholder management and Release management. 

This section elaborates on the integration of the end-to-end Quality process within a scaled environment, considering: 

1. Focus areas that apply where the greatest end-to-end quality benefits can be achieved.
2. The teams that play a role in organizing end-to-end Quality Engineering.
3. Specific roles and responsibilities in those teams.
4. Artefacts (activities and means) that are supportive to this integrated end-to-end Quality process.

Read more

End-to-End testing in an IoT environment

The end-to-end testing building block is particularly important in an "Internet of Things" context, where things in multiple domains and platforms are interconnected.

The following example on predictive maintenance on truck engines shows how an End-to-End test could look like for an IoT solution.

1. An engine temperature sensor collects data when a truck is running.

2. Data is sent every 10 minutes to a cloud storage of the truck engine builder.

3. In the cloud storage of the truck engine builder a business intelligence rule on predictive maintenance regarding wear on a fan belt is triggered on a trend in rising temperature while running a truck in specific circumstances.

4. A notification is sent to the workshop of the trucks maintenance partner that a fan belt change needs to be planned between now and 2 weeks.

5. A fan belt is changed for this truck at a workshop together with other maintenance activities.

6. The fan belt change is sent to the cloud storage.

7. Business rules check if the data received from the temperature sensor now matches the implementation of the new fan belt.

8. New logic is learned that the current data matches the implementation of a new fan belt for this truck.

9. Monitoring for new wear and tear is started.

For more information on End-to-End testing in general and more information on End-to-End testing in an IoT context, please refer to the concerning books.