An electronic data capture system and a paper case report form may serve the same regulated purpose. Both must support complete, accurate, attributable, and trustworthy records. That common purpose does not make their controls or validation methods interchangeable. The technology changes how failure can occur, which in turn changes the evidence needed to show that the process is under control.
The same distinction applies to artificial intelligence (AI).
An AI-enabled computerized system remains subject to the established disciplines of computerized systems assurance: a defined intended use, risk management, requirements, testing, security, data integrity, supplier oversight, change control, traceability, and documented evidence of fitness for purpose. AI does not create an exemption from that framework. Nor does it replace the need to validate the surrounding application, interfaces, infrastructure, configuration, access controls, audit trails, and regulated workflow.
But conventional validation is only the starting point. Where system behavior is learned from data, sensitive to operating conditions, probabilistic, or dependent on an externally managed model, the validation case must answer questions that a conventional software test protocol was not designed to carry by itself.
Keep the GAMP Spine
This relationship is reflected in the two principal GAMP publications. GAMP 5: A Risk-Based Approach to Compliant GxP Computerized Systems, Second Edition, retains a risk-based life-cycle framework centered on patient safety, product quality, data integrity, fitness for intended use, supplier involvement, and critical thinking. Its life cycle already extends through operation and retirement; traditional validation should never be reduced to a one-time exercise at release (ISPE 2022).
The 2025 GAMP Guide: Artificial Intelligence is intended to be used in parallel with GAMP 5. It does not offer a competing quality system. It extends the established framework into areas such as context of use, dataset creation and splitting, fit-for-purpose data, data and model governance, model development, robustness and fairness testing, human autonomy and control, model monitoring, drift detection, and the governance of dynamic systems (ISPE 2025; Stockton, Staib, and Heitmann 2025).
That is the central point: the assurance objective remains familiar, but the evidence architecture changes.
Why the Conventional Test Script Is Not Enough
In a conventional deterministic system, specified logic can usually be challenged with inputs for which the expected result is known in advance. Passing tests do not prove everything, but they can provide strong evidence that implemented behavior matches approved requirements.
An AI model presents a different problem. Its behavior is learned from examples rather than fully expressed as rules. Its input space may be too large to enumerate, and its errors may be unevenly distributed. A model can meet an overall performance threshold while failing on a small but important subgroup. It can perform well on a carefully assembled evaluation set and poorly on the data encountered in routine use. A generative model can produce two plausible answers to similar prompts, while only one is adequately grounded in the controlled source material.
Installation qualification, interface testing, access-control testing, and verification of audit trails still matter. They can show that the AI-enabled system was built, configured, and deployed as intended. They cannot, by themselves, show that the model is suitable for the decision it supports.
The additional work is not simply “more testing.” It is a different kind of assurance argument.
| Assurance question | Conventional emphasis | Additional AI evidence |
|---|---|---|
| What is the system meant to do? | Defined function, users, records, and operating environment | Decision role, affected population, input characteristics, conditions of use, exclusions, and foreseeable variation |
| Why should its behavior be trusted? | Approved requirements, design, configuration, and scripted expected results | Data provenance and suitability, model-selection rationale, independent evaluation, error tradeoffs, subgroup performance, robustness, and relevant fairness considerations |
| Is human review an effective control? | Defined role, workflow step, access, and approval | Ability to detect model error, access to independent evidence, sufficient time and competence, authority to disagree, and proof that intervention works in practice |
| What can change the validated state? | Software, configuration, infrastructure, interfaces, and procedures | Model weights, training and evaluation data, prompts, retrieval sources, embeddings, thresholds, vendor model versions, feedback routes, and context of use |
| How is continued fitness shown? | Maintenance, incident management, change control, periodic review, and system monitoring | Performance and drift monitoring tied to thresholds, subgroups, outcomes, investigation, restriction, retraining, re-evaluation, or withdrawal |
The table is not a universal checklist. The necessary depth depends on intended use, system design, model influence, the consequence of error, and the strength of other controls. A low-impact tool used to improve the wording of an internal draft does not require the same assurance as a model that routes safety cases or contributes to a product-quality decision. Risk determines the burden; the technology determines where evidence must be sought.
Context of Use Defines What the Evidence Supports
A conventional intended-use statement may describe a function adequately in a few lines. For AI, the same statement may be too broad to serve as a validation standard.
Consider a hypothetical model that identifies a condition in medical images. Naming that task does not establish where the model is credible. Performance may depend on the patient population, disease prevalence and stage, acquisition equipment, image quality, clinical setting, operator, workflow, and the point at which the output influences a decision. A model evaluated on one combination of those conditions has not automatically been validated for another.
The relevant question is therefore not merely, “Can the model perform this task?” It is: “Can this model perform this task, for this population and these inputs, under these operating conditions, within this workflow, at a level of performance proportionate to the risk?”
This is why the context of use is load-bearing. It sets the boundary for data selection, acceptance criteria, subgroup analysis, human oversight, monitoring, and change assessment. When the actual use moves beyond that boundary, the prior evidence does not move with it.
Human Oversight Must Be Validated as a Control
“A person reviews the output” is a workflow description, not evidence that risk is controlled.
Meaningful oversight requires a reviewer who can recognize relevant failure, has access to the information needed to form an independent judgment, has enough time to perform the review, and can reject, override, or escalate the output before harm occurs. Decision rights and accountability must be explicit. If only certain outputs reach a person, validation must also establish what happens on the routes that receive no case-level review.
The interaction creates risks of its own. Automation bias may encourage uncritical acceptance. Repeatedly correct outputs may weaken vigilance. Excessive alerts can produce fatigue. A polished interface can conceal uncertainty, and a nominal right to override can become meaningless if the culture penalizes disagreement or the workflow makes intervention impractical.
For that reason, validation should challenge the human-AI arrangement under realistic conditions. Useful evidence can include seeded-error studies, independent rereview, observed intervention exercises, and adjudicated samples from routes that are not reviewed routinely. Operational signals such as overrides, disagreements, response times, escalations, and downstream outcomes may help, but no single rate proves that oversight is effective. A falling override rate, for example, can indicate either an improving model or a reviewer who has stopped challenging it.
AI Makes Continued Assurance More Demanding
Ongoing control is not new. GAMP 5 already addresses support, monitoring, incident management, operational change, and periodic review. AI changes the range of conditions that can undermine fitness and the signals needed to detect them.
A conventional system’s functional behavior is generally expected to remain stable between controlled changes, even though its infrastructure and operating environment still require management. An AI-enabled system can become less suitable while its code and installed model version remain unchanged. The population may shift. Input data may acquire a different structure or meaning. A clinical or manufacturing workflow may change around the model. User behavior may alter the quality of human oversight. Ground truth may arrive late, making degradation difficult to see. In a hosted service, the provider may change a model or a shared component outside the regulated company’s deployment process.
AI assurance is a closed loop only when a signal can lead to investigation and action. Collection without a response path is observation, not control.
Monitoring must therefore be derived from the failure modes of the specific use. It may need to examine input and population shifts, performance against independent outcomes, clinically or operationally important subgroups, override behavior, unreviewed routes, vendor version changes, and trends toward an acceptance limit. Thresholds need owners and predefined responses. A breach may require investigation, restriction of use, recalibration, retraining, re-evaluation, rollback, or retirement.
Change control must widen for the same reason. Model weights are changes, but they are not the only changes that can move behavior. Training data, preprocessing, prompts, retrieval corpora, embedding models, inference settings, feedback mechanisms, decision thresholds, and the context of use can all alter what the system does. A change log can be complete for software releases and still miss the changes that matter most to the model.
Validation Becomes an Assurance Case
The mature response is neither to force AI into a conventional template nor to declare that established validation no longer applies. It is to assemble a risk-proportionate assurance case for the whole AI-enabled process.
That case connects five claims: the intended use and context are defined; the data and model evidence support that use; the computerized system and its interfaces operate as designed; human and procedural controls work under realistic conditions; and monitoring and change control can detect when the original basis for reliance no longer holds. Each claim needs traceable evidence, and the evidence must fit the consequence of being wrong.
The desired outcome is the same one that computerized systems validation has always pursued: justified confidence that a system is fit for its intended use and remains under control. AI does not change that objective. It changes what can fail, what must be measured, and what an organization must be able to prove.
References
International Society for Pharmaceutical Engineering. 2022. GAMP 5: A Risk-Based Approach to Compliant GxP Computerized Systems. 2nd ed. Official publication page. Table of contents.
International Society for Pharmaceutical Engineering. 2025. GAMP Guide: Artificial Intelligence. Official publication page. Table of contents.
Stockton, Brandi, Eric Staib, and Martin Heitmann. 2025. “New GAMP Guide Provides Framework to Achieve High-Quality AI-Enabled Computerized Systems for Life Sciences.” ISPE Pharmaceutical Engineering, 15 August 2025. Official article.