Six Things Pharma Should Prepare to Demonstrate About an AI Agent
The draft European rules for artificial intelligence (AI) in medicine manufacturing are blunt about language models. Annex 22 to the European Union (EU) Good Manufacturing Practice (GMP) guidelines, published for consultation in July 2025, says generative AI and large language models “should not be used in critical GMP applications”. For the critical applications it does cover, it accepts only models that give identical outputs for identical inputs.
Fifteen months later, the discussion has moved. The European Medicines Agency (EMA) has published its report on a June 2026 workshop with industry experts. It records that the drafting group has discussed widening the scope to adaptive, probabilistic and generative models, including large language models, provided they meet the annex and are “supported by a fully documented, robust, risk-based control strategy”. The industry position was that a technology label should not decide what is acceptable. Intended use, risk and the strength of the controls should.
None of this is final. Annex 22 is still a draft, and the workshop report records discussion and industry input, not a decision. The discussion increasingly concerns evidence about the whole process, whatever kind of model sits inside it.
A colleague who works with pharmaceutical manufacturers told me recently that the smart manufacturing leads in those conversations want deterministic AI agents. I wrote separately about how deterministic agents are built, in “Two Operators Can Think Differently and Meet the Same Goal”. This piece is about the other half of the question: what a pharma manufacturer should prepare to demonstrate. Read together, the 2025 draft and the 2026 report suggest six areas in which manufacturers should prepare evidence.
1. What the AI is for, and how much it decides
The draft asks for the intended use to be described in detail, including every kind of input the model will see, rare cases included, and approved by a process expert before testing starts. The workshop discussion adds the factors that set how much control is needed: the consequence of a wrong decision, how much the model influences that decision, uncertainty, complexity and detectability, meaning whether a failure would be noticed. The US Food and Drug Administration’s January 2025 draft guidance on AI uses the same two axes, model influence and decision consequence.
**This is where agent architecture helps.**In a well-built agent system, the model’s authority is a design choice. When the language model proposes and fixed checks decide whether anything happens, those checks restrict its authority to act. Its remaining influence over which permitted action happens still has to be assessed for each intended use and decision, and that assessment can be documented decision by decision.
2. Acceptance criteria set before testing
The draft requires test metrics and acceptance criteria to be defined by a process expert and approved before acceptance testing. It also sets a minimum performance criterion: the criteria must be “at least as high as the performance of the process it replaces”. That means measuring today’s process first, including the people in it, and testing on data the model never saw during development.
For an agent team, one way to express the performance criteria is the objective function: the score, built from measured plant values, that says what a good result is. Agreed before the trial and computed the same way every time, it lets the system be measured against those criteria every day as well as at qualification. Safety, quality, data integrity, oversight and failure-handling criteria need their own tests.
3. Controls in layers
The workshop’s technical session described controls in three layers:
controls that stop bad inputs getting in, controls that catch errors while the model works, and controls that stop a bad output reaching the process.
Its examples of the last kind include fixed limits, thresholds on how confident the model must be, and a return to a known, validated way of running when something fails. The use cases discussed included an agentic AI system supporting deviation management and adaptive environmental control within validated limits, some of them pilots.
That layering matches how a governed agent system is built. Data pipelines check and flag every measurement before an agent sees it. A confidence threshold lets the system report “undecided” instead of guessing; the 2025 draft calls for appropriate confidence thresholds and for considering an “undecided” outcome at very low confidence. Every change an agent proposes is checked at the moment it is sent against rules held as computable policy: written so software evaluates them against plant data, versioned under change control, and reviewable by the quality unit. When anything fails, agents can drop back to advise mode, where nothing is written to the process. That is one possible fallback component; the process state it leaves and the operator response it requires still have to be assessed and validated for the intended use.
4. Human oversight that can be shown to work
The workshop distinguished human oversight from a human in the loop. Oversight means a trained person can understand, challenge, override or escalate. A human in the loop, where the process waits for a person, is one way to provide it, not a default for every use. Regulators asked how to manage automation bias and how oversight could be evidenced at inspection, and the report records that accountability “cannot be delegated to AI”.
For AI agents, a person with a clearly assigned role decides when they are allowed to take action. When an agent needs human approval, it explains what is happening, shows the supporting information and presents the options. The system records what the AI produced and what people approved or changed, so someone can later check what happened and who made each decision.
5. Control over the whole lifecycle, including suppliers
The draft requires change control, configuration control, performance monitoring and monitoring of input drift once a model is in use. The workshop added incident handling, corrective action and supplier oversight. One presenter put the supplier point plainly: if a third-party model or model service cannot provide the outputs, rationale, logs or evidence needed for control, the manufacturer cannot justify using it.
In practice that means approved model versions, prompts, rules and thresholds are kept under change control, the operational state that may change during use is defined and recorded, and the decision record carries the version of everything that took part.
6. An evidence package an inspector can follow
The workshop’s strategic session asked how a manufacturer could show inspectors that layered controls produce reliable output for a critical decision. The answer started with the risk assessment: the risk scenarios, the controls mapped to them, and the verification and monitoring evidence behind each. Industry presenters called the result an “AI qualification dossier”, and the report notes that a documented “do not use” is a valid outcome.
An agent system can make much of that evidence routine. Replaying the deterministic authorization path can provide useful verification evidence when its complete inputs and configuration have been retained: every input, proposal, check, refusal and action, with its versions. A rerun of the authorization path from that record has to give the same result.
Assess each decision and its evidence
Read against the regulators’ own discussion, “deterministic AI” turns out to be shorthand for something broader. What a pharma manufacturer should prepare to show is a process whose goal, limits and record are fixed, with evidence that it stays inside its intended use. The language model can vary inside that process, as long as its influence is understood, its proposals are checked and its results are measured.
That gives a quality team a practical starting point: list the decisions an agent may influence, decide how much influence each may have, and agree the evidence that will show it stayed within that limit. Start with uses assessed as non-critical, such as deviation investigation support, batch record review assistance or shift handover, with trained personnel responsible for checking the outputs. Human review alone does not make an application non-critical, and process adjustments inside an approved design space need their own assessment against their intended use before an agent proposes them.
This is the pattern we build with XMPro agent teams and Data Stream Designer: explicit checks that a quality team can inspect and test.
The engineering detail is in our technical guide to deterministic industrial agents: how measurements are checked, how each proposed change is authorized, how rules are held as computable policy and what the decision record holds. Gavin Green and I give the formal treatment in the working paper Deterministic Where It Counts, including a five-test assurance protocol whose results can go into the evidence package described in section 6, and a reading of the draft Annex 22 alongside the 2026 workshop discussion.
Sources
European Commission. EudraLex Volume 4, Annex 22: Artificial Intelligence, draft for public consultation, July 2025. Link European Medicines Agency. Report: Multistakeholder workshop on expert contributions to artificial intelligence guidance development (Annex 22), EMA/156789/2026. Workshop held 30 June 2026. Link US Food and Drug Administration. Considerations for the Use of Artificial Intelligence To Support Regulatory Decision-Making for Drug and Biological Products, draft guidance, January 2025.
Pieter van Schalkwyk is the CEO of XMPro, specializing in industrial AI agent orchestration and governance. Drawing on 30+ years of experience in industrial automation, he helps organizations implement practical AI solutions that deliver measurable business outcomes while ensuring responsible AI deployment at scale.
He is the lead author of the Industrial AI Agent Manifesto, published by the Digital Twin Consortium in February 2026, a governance framework for trustworthy autonomous operations.
Our GitHub Repo has more technical information. You can also contact myself or Gavin Green for more information.
Read more on MAGS at The Digital Engineer
How this was made: I worked with Claude to check sources, draft from my direction and edit, and with Codex to review the argument and evidence. I do not publish anything I have not rewritten myself.
This article originally appeared on XMPro CEO’s LinkedIn blog, The Digital Engineer, on 20 October 2026.
