Find your way to be trained or even get certified in TMAP.
Start typing keywords to search the site. Press enter to submit.
Test design is an important topic in TMAP. Structured testing requires careful consideration of what to test and how to test it. We use the term “test design” for the entirety of these activities, even though in some approaches (like exploratory testing) there is no actual up-front test case design involved.To prepare and perform tests, we distinguish two overall approaches:
In the more than 30 years of TMAP’s existence, we have described many different test design techniques and approaches. But with the rise of intelligent machines the focus shifts, the existing techniques and approaches described below, gain importance, and new techniques and approaches also become relevant.
Coverage-based test design that pertains to testing of AI based solutions:
Experience-based test design approaches that pertain to the testing of AI-based solutions are:
By no means do we claim that the above lists of coverage-based and experience-based test design are exhaustive nor definitive lists, since the field of GenAI is rapidly evolving new techniques and approaches will very likely pop up. And please remember that the traditional test design techniques and approaches can also be used.
When preparing for testing an AI-based system, especially when it is Generative AI, a proper preparation is to create one or more charters for exploratory testing. This will be more effective than creating detailed test cases upfront, because of the probabilistic nature of the AI models. This probabilistic nature implies that it is always uncertain what the output of a process will be, therefore it is fundamentally less reliable when compared with a rule-based solution. If, however, the risks are assessed to be within acceptable range, and in balance with expected benefits (such as faster IT delivery), organizations may still decide to apply AI-based solutions. In this situation, performing well organized structured exploratory testing can be a good way to provide information about the quality level. This information helps stakeholders establish their confidence in achieving the pursued business value.
People learn all sorts of skills and have to prove to be proficient in an exam, for example to obtain a driving license. The exam checks if they know the rules and are able to apply them correctly. After they pass the examination, we trust in the future these people will always make the right decisions.As with people, we want to know whether an intelligent machine will perform well enough. Testing is, of course, the basic approach to evaluate the quality level. Since it isn’t possible to test all possibilities, we can use the approach of examination to decide if the intelligent machine is to be trusted to perform the task. As soon as the AI passes the exam it can be used in live operation. If the learning is frozen, so the intelligent machine doesn’t change its behavior after the exam, this may be a good approach.
If the intelligent machine keeps on learning, one exam is not sufficient, we should do an examination every once in a while. With people, periodic examination probably wouldn’t be feasible. But with an intelligent system a periodic exam would be possible, provided that the examination itself can also be done by an automated system. With periodic examination, a fundamental question is how often this examination should take place. That has to do with the periodicity of use of the intelligent machine, but most machines will be functioning continuously. So, there will probably be a desire to do continuous examination. Because it’s an operational system, it is better to use the term continuous monitoring. When stakeholders know the system is continuously monitored in a structured and formal way, this will support the confidence in such an operational system. Especially with closed-loop machine learning and other forms of AI systems that continuously improve their behavior, it is important to implement continuous monitoring of the results to check if the results remain within the boundaries of tolerance that were defined as “good behavior”.
[Reference: Testing in a digital age / chapter 4.5 [Ven, 2018]]
In AI-enabled systems, model quality is not static but subject to change over time. Drift testing addresses the risk that an AI model’s performance degrades over time due to changes in data or its underlying relationships. We distinguish two primary forms of drift: data drift and concept drift.In GenAI tools, that are based on machine learning systems, such drift can invalidate the model’s behavior established during training and testing. From a quality engineering perspective, drift testing extends traditional testing beyond pre-release validation into the operational phase. It focuses on detecting deviations that may impact reliability, accuracy, or fairness.
Data drift refers to changes in data sources, user behavior, or external conditions that gradually erode the alignment between the model behavior established during training and validation and the behavior in an operational situation. Data drift testing focuses on identifying changes in the statistical properties of input data used by an AI model. Such changes may affect feature distributions, data ranges, or the frequency of specific values.Although data drift does not automatically imply incorrect predictions, it can invalidate behavior established during model training. Testing for data drift typically involves comparing production data with a defined reference dataset. Thresholds are established to determine when deviations become significant from a quality risk perspective. When these thresholds are exceeded, further investigation is required to assess potential impact on model behavior and, when needed, to change the model’s behavior by retraining a machine learning model or supplying a different set of RAG to a GenAI model.
Concept drift testing addresses changes in the relationship between input data and expected outcomes. This creates a fun damental risk to the reliability of AI-based decisions. Unlike data drift, concept drift affects the meaning of predictions rather than the data itself. This type of drift often becomes apparent through a gradual decline in model performance. Testing for concept drift therefore relies on monitoring outcome-related indicators, such as accuracy or fault rates. Because ground truth data may not be immediately available, detection is often indirect. Concept drift testing requires clearly defined acceptance criteria and escalation paths. When concept drift is confirmed, retraining or model replacement becomes a quality-driven decision.
Generative AI systems differ fundamentally from traditional software because their probabilistic nature means the same input can produce different outputs over time. As a result, validating correctness cannot rely on a single deterministic test execution. Instead, quality engineers must execute tests multiple times and evaluate the distribution of outcomes. Each response should be assessed against defined acceptance criteria and classified as acceptable or not. The key metric becomes the proportion of acceptable responses rather than the outcome of any individual run. This enables teams to quantify reliability in statistical terms and determine whether the system meets predefined quality thresholds. Such thresholds reflect the minimum acceptable consistency required for production use. This approach shifts testing from binary pass/fail decisions to probabilistic quality assessment. It also helps address the test oracle challenge, where correctness may not be strictly defined.
Aspects that need to be considered relating to this non-deterministic behavior are how the model handles diverse and shifting data distributions, its outputs may reflect uncertainty or bias that must be quantified, and regulatory or safety requirements often demand measurable evidence that predefined reliability and fairness thresholds are consistently met. This requires testing with sufficiently large and representative samples to produce statistically valid and defensible conclusions about quality and the related risks.
AmpQE overview
