Home TMap Digital transformation & The Technology Leaders

Testing pyramid

Testers often struggle to determine what to test manually and what to test with automated test execution. In his blog, Mike Cohn explains that one of the reasons of this difficulty was that people were automating their tests at the wrong level. To support this, he introduced the testing pyramid [Cohn 2009].

The basic message of this testing pyramid is that testing should be automated as much as possible, and testing should be done as early as possible. This means that tests that can be done for individual pieces of code should be performed at that level. That is, small independent tests that can be quickly run – and preferably independently run – so that tests can also be run in parallel to shorten the execution time.

The testing pyramid shows three levels, although this could be more or less than three levels, depending on your situation. However, experience shows that three levels work well in most cases.

  1. The first level contains small individual tests.
  2. The second level focuses on the integration of modules or systems. These tests often relate to the interfaces, for example the application programming interfaces (APIs).
  3. The third level generally uses the graphical user interface to execute the tests. These tests are the slowest to run because they require interaction through the GUI, but also because these tests focus on complete business processes, for example a complete user transaction.
The testing pyramid, extended and adapted from [Cohn 2009].

 

Our version of the testing pyramid also shows where manual testing comes in. On the left of the pyramid you see the remark that manual testing occurs when automated tests are created. This remark refers to the fact that when these automated test scripts are made, the testers involved will have to try out the functions and test their test scripts. Experience shows that during these manual activities, a big chunk of the anomalies in new and changed IT systems are found; many more than in the actual automated testing.

At the top of the pyramid you see the manual exploratory tests. This is where the end users and other business stakeholders test the system in order to support their confidence that the system will actually deliver the pursued business value.