Is there any value in unstructured testing?
Generally, testing experts advocate to always structure your testing. You probably learned not to do unstructured testing. In many organizations a lot of the testing is still unstructured; there must be some value in unstructured testing.
What is unstructured testing?
Any testing lacking a plan containing what to do and what to expect of a system, or lacking preparation of the test, is unstructured. This is also called ad-hoc testing. Some people see a great advantage in unstructured testing because, as they say: "You can start testing right away." That is, without "wasting" any time on preparation.
Is unstructured testing useful?
In some situations, unstructured testing may seem like a good choice. When the test object is a system of really bad quality, why would you prepare? Anyone with some IT knowledge can do some "tests" by hitting some keys and, because of its bad quality, will find many faults and failures. As soon as the developers start complaining that they don't have time to investigate and fix all these anomalies, the stakeholders will think the testers have done a wonderful job.
Should you be happy with finding many faults?
As a professional tester you will probably just be sad, because no professional IT person, be it a tester, a designer, a programmer or someone in any other role, will be happy with a low-quality IT system. Therefore, if you find many anomalies using unstructured testing, this only proves that the quality is really bad because with structured testing even more faults and failures would have been found.
What if the quality is good?
Now take the other example. When you have an IT system that is of good quality, the testers do their unstructured testing and don't encounter any faults or failures. Can they now say the quality is good and that there are no significant risks? Did they really measure the quality and risks? No, the only thing they can truly say is they did "some" testing and didn't find any problems. However, they cannot explain which requirements or quality risks have been covered. They are not even sure which parts of the system have been covered. Their information is mostly useless to the stakeholders in their attempt to establish the confidence that the IT system will be able to deliver the pursued value.
Then why do so many organizations use unstructured testing?
The main reason for using unstructured testing seems to be that people want to start their "test execution" as soon as possible because just starting gives them a sense of progress. This could be due to time pressure because of deadlines, a desire to show the new system to the stakeholders and/or a lack of understanding of the benefits of structured testing. However, by adding just a little bit of structure, the information supplied can already be largely improved.
What is an example of an unstructured testing approach?
In his book ‘The art of software testing’ (published in 1979) Glenford Myers describes error guessing [Myers 1979]. In the description he states it is "largely an intuitive and ad hoc process". He adds that experienced testers seem to "smell-out" problems, so actually these testers subconsciously practice test design techniques. However, people that apply error guessing usually do not log any of their test cases and also do not register the expected and actual results of their tests. In many companies, staff are asked to test (parts of) new systems. Without any coaching or training, the people are supposed to start testing. And that is what they do: they start test execution. Failures will probably be found, but because of this unstructured and unprepared way of working, there are various downsides that can be encountered, such as these:
- These tests usually focus on the day-to-day business processes, neglecting weekly, monthly or yearly processes.
- They focus on "positive test cases" (tests to test the main function of the system), neglecting negative test cases (tests designed to test what happens if errors are made, inputs are outside predefined boundaries etc.).
- They focus on interactions with the system that are easy and produce results immediately, neglecting tests that require difficult calculations to check the outcome, tests that take time to execute (e.g. batch jobs).
- They focus on functionality, neglecting such things as performance and security, which require tests for which specialized knowledge is needed.
- Nobody knows if the important parts of the system have been tested.
How can we avoid unstructured testing?
There are a few basic actions that can assure that any testing is at least somewhat structured. First, do some serious thinking about the goal of the testing. What information needs to be given? Then think about how this information can be obtained. Which test ideas, test situations and/or test cases would help you get this information? Then design and execute the tests, in a formal or less formal way. Be sure to register the tests that are done, especially the expected and actual outcomes. And finally, structure and present the information in such a way that the stakeholders can use it to establish their confidence about whether the system is able to deliver the pursued value.
TMAP strongly advises to apply exploratory testing instead of error guessing. For more information on experience-based testing, including error guessing and exploratory testing. We advise you to always combine experience-based testing with coverage-based testing.