A Layered Model for Testing GenAI
Applications

To structure testing of GenAI applications, we distinguish six interrelated layers that together determine system behavior.
Inputs & state cover the system prompt, user prompt, conversation history (context window), and any persistent memory that influences how the model interprets a request. Knowledge augmentation includes the RAG corpus and the full retrieval pipeline configuration, such as chunking, embeddings, index ing, and ranking, which determine what external knowledge is injected into the prompt. Reasoning & generation concern the selected model and version, configuration parameters (e.g., temperature, token limits), and decoding strategy, all of which affect determinism, coherence, and factual accuracy. Actions include tool and function calls, agent orchestration, external integrations, and associated error handling, as the model moves from generating text to triggering system behavior. Controls encompass guardrails, security protections against prompt injection or data leakage, and policy enforcement mechanisms that constrain outputs and actions. Finally, the quality system provides the assurance backbone: observability, logging, and tracing of prompts and tool calls, structured evaluations and regression suites, and non-functional characteristics such as performance, cost efficiency, and reliability. Together, these layers offer a model for structured quality engineering for GenAI systems.