Scenario 8 – The Efficiency Shortcut
A case study in human behavioural drift and AI governance failure in a regulated manufacturing environment. This scenario explores what happens when well-designed AI workflows are quietly bypassed in the name of efficiency — and why that matters for GMP compliance.
QA Workshop Scenario
AI Governance
Batch Record Review
Background: The Designed Process
Your organisation has implemented an AI agent to assist QA reviewers with batch record review. The system was carefully designed to reduce manual scanning time whilst maintaining rigorous QA oversight. The intended workflow is structured, sequential, and human-led — the AI functions as a capable assistant operating within a defined scope, not as an independent decision-maker.
Understanding the designed process is essential before examining where things go wrong. The three-stage workflow below represents the system as it was intended to operate.
This workflow was designed with a clear principle in mind: the AI accelerates the review by surfacing potential issues, but the QA reviewer retains full responsibility for evaluating those findings and making the final determination. The human remains in the loop at every critical juncture.
What Is Happening in Practice
During a routine QA review meeting, quality leadership discovers that a number of reviewers have quietly changed how they interact with the AI agent. Rather than running the structured review workflow, reviewers are asking the agent a single, direct question:
"Are there any compliance concerns in this batch record?"
The AI responds with a short, reassuring summary — something along the lines of "No major documentation issues detected." The reviewer, satisfied with this response, proceeds with minimal further independent evaluation. The batch record moves forward without the structured findings analysis that the workflow was designed to deliver.
What makes this situation particularly significant from a governance perspective is not a single isolated incident — it is that this approach has become common practice. The system configuration has not changed. No formal procedure has been updated. No change control has been raised. Yet the effective process has shifted fundamentally. The AI, designed as a structured review assistant, is now being used as a compliance oracle — and nobody raised a flag.
Your Challenge: Questions for Quality Leadership
As the Quality leadership team, this scenario presents three critical questions that must be addressed. These are not merely theoretical — they reflect real governance responsibilities that arise when AI tools are deployed in regulated environments and human behaviour evolves around them.
Is this use of the AI agent acceptable?
Evaluate whether the informal workaround constitutes a deviation from the validated and approved process. Consider what your current procedures say about how this AI tool may be used, and whether the actual practice remains within scope.
What concerns would you raise?
Identify the compliance, quality, and safety implications of the current practice. Think about documentation integrity, the completeness of human review, and whether the AI outputs being relied upon are fit for the purpose to which they are now being applied.
What controls might be needed?
Consider what procedural, technical, or training-based controls could be introduced to prevent workflow bypass, reinforce human verification responsibilities, and ensure ongoing governance of AI-assisted review activities.
Consider the following dimensions as you work through your response: adherence to defined workflows, human verification responsibilities, and governance of AI usage.
Issue 1 – Defined Review Workflow Is Bypassed
The AI agent was designed and validated to support a structured review process. That structure is not incidental — it reflects the scope within which the tool's outputs can be considered reliable and appropriate. When reviewers bypass the structured workflow and substitute it with an open-ended conversational query, they are operating the tool outside its intended use case.
The AI's structured review process is designed to interrogate the batch record systematically, checking specific fields, sequences, and data points in a defined order. A general question such as "Are there any compliance concerns?" does not invoke that structured process. The AI's response to such a query may be based on a surface-level interpretation of the record rather than a thorough, structured analysis. The reviewer has no visibility into what the AI actually evaluated — or did not evaluate — in producing its summary.
From a GMP perspective, this represents a process deviation. The approved process is not being followed. The fact that it is not formally documented as a deviation does not reduce its regulatory significance — if anything, it amplifies it, because the gap between the written procedure and actual practice is invisible to the quality system.
Relevant Principle
Define Intended Use and Scope
The AI must be used within its defined workflow and intended scope. Outputs generated outside that scope cannot be assumed to carry the same reliability or assurance as those produced through the validated process.
Issue 2 – AI Is Used as a Decision Shortcut
When a reviewer receives the message "No major documentation issues detected" and proceeds on that basis with minimal further review, the AI has effectively become a decision authority rather than a review assistant. This is a fundamental inversion of the intended human–AI relationship in a regulated environment.
The AI was designed to flag issues for human evaluation — not to render a verdict on acceptability. A statement such as "no major issues detected" is not a QA release decision. It is an AI output that requires human interpretation, contextual judgement, and professional expertise to be meaningful. The reviewer's role is not simply to receive and accept the AI's summary; it is to evaluate that summary against their own knowledge of the record, the product, and the applicable requirements.
In practice, the distinction between "the AI found no issues" and "the record is acceptable" is being collapsed. Over time, this erodes the professional judgement and independent verification that GMP requires. The reviewer's role becomes increasingly passive — a rubber stamp on the AI's conclusion rather than an active, accountable quality decision.
Intended Role of AI
Assist the reviewer by highlighting potential issues for human evaluation. The reviewer makes the determination.
What Is Happening
The AI summary is treated as the determination. The reviewer's role is reduced to acknowledgement rather than evaluation.
Governance Principle
AI is a Worker to be Governed. It assists; it does not replace human judgement or professional accountability.
Issue 3 – Human Verification Is Reduced
Relevant Principle
Human Verification is Mandatory
Human reviewers must perform meaningful verification of AI outputs — not merely receive and acknowledge them. The depth and independence of that verification must not be contingent on the AI's reported findings.
GMP requires that human reviewers perform meaningful, independent verification — not cursory confirmation of what an AI has already told them. When the reviewer's evaluation is shaped primarily by the AI's conclusion, the independence of that verification is compromised. The human review becomes a downstream consequence of the AI's output rather than an independent assessment.
This matters because AI systems, including well-designed ones, can miss issues. They may fail to detect unusual patterns, contextual anomalies, or documentation errors that fall outside their training data or analytical scope. A reviewer who has been guided to expect a clean record may apply less scrutiny than one approaching the record fresh. This is a well-documented cognitive effect — confirmation bias amplified by algorithmic authority.
Meaningful human verification means the reviewer must apply their own professional knowledge to the record, irrespective of what the AI has reported. The AI's output should inform but not constrain that evaluation. Where the AI reports no issues, the reviewer should still apply a level of independent scrutiny commensurate with the risk profile of the batch and the critical nature of the record being reviewed.
Issue 4 – Informal Behaviour Change Without Governance
Of the four issues identified in this scenario, Issue 4 is arguably the most significant from a systemic quality perspective. The system did not change. The procedure was not updated. No change control was raised. Yet the effective process — the way the AI is actually used, the depth of human review that actually occurs, the assurance that is actually generated — has changed materially.
This is the defining characteristic of human behavioural drift: gradual, informal, often well-intentioned changes to how people interact with a system, which accumulate over time into a significant deviation from the approved process. Because no formal change was made, the quality system has no record of what has shifted. Auditors reviewing the procedure will see a compliant process on paper. The reality on the floor is different.
Governance of AI-assisted processes cannot be limited to the initial validation and approval of the tool. It must include active, ongoing oversight of how the tool is being used in practice. This requires mechanisms such as periodic review of AI interaction logs, competency assessments for AI users, and clear escalation pathways when informal workarounds are identified. Without these controls, behavioural drift will go undetected until it manifests as a quality event — or an inspection finding.
Procedure vs. Practice Gap
The written procedure describes the approved workflow. The actual practice has diverged. This gap is invisible to the quality system without active oversight mechanisms.
Ongoing Oversight Required
Governance must extend beyond initial validation. AI interaction logs, periodic review, and usage monitoring are essential to detect drift before it becomes entrenched.
Change Without Change Control
Behavioural changes to how AI tools are used constitute effective process changes, even without formal updates. Quality systems must be designed to capture and evaluate these shifts.
Summary: Four Issues, Four Principles
This scenario maps cleanly onto four core AI governance principles that should underpin any AI-assisted process in a regulated manufacturing environment. Each issue identified in practice has a corresponding principle that, had it been consistently applied, would have prevented or detected the behavioural drift described.
1
Define Intended Use and Scope
The AI must be used within its designed workflow. General queries outside the structured process fall outside the validated scope and cannot provide equivalent assurance.
2
AI is a Worker to be Governed
AI assists reviewers; it does not replace their judgement. The reviewer remains the decision authority. AI output is input to a decision, not the decision itself.
3
Human Verification is Mandatory
Meaningful, independent human verification is required regardless of what the AI reports. The depth of review must not be conditional on a favourable AI summary.
4
Quality System Governance
AI-assisted processes require active, ongoing oversight. Behavioural drift must be detectable and addressable through the quality system, not just at initial validation.
The Broader Lesson: Governance and Behaviour, Not the Model
Scenario 4 – Technology Governance Failure
A failure in how the AI system itself is controlled — its inputs, configuration, or validated scope. The technology drifts outside its governed boundaries. This is a controlled-inputs failure: the system is doing something it should not be permitted to do.
This type of failure is often more visible because it involves a change to the system or its data. It is more likely to be captured by existing change control and validation frameworks — if those frameworks are appropriately designed for AI tools.
Scenario 8 – Human Behavioural Drift
A failure in how people interact with a system that has not technically changed. The tool remains within its configured boundaries; the humans operating it have quietly stepped outside theirs. This is a workflow bypass failure: the system is functioning as designed, but it is no longer being used as designed.
This type of failure is frequently more dangerous precisely because it is harder to see. No alarm is triggered. No change control is initiated. The quality system may be entirely unaware that anything has changed at all.
AI risk in GMP environments is primarily about governance and behaviour, not the model itself. The most significant risks often arise not from what the AI does, but from how people choose to use it — and whether the quality system is watching.
Together, Scenarios 4 and 8 make a powerful case for a governance framework that addresses both dimensions: the technical controls that define and constrain the AI's operation, and the human and organisational controls that ensure people continue to use AI tools as intended, under meaningful oversight, with verified outcomes. Neither dimension alone is sufficient. Both are essential.