"Software Doesn't Reflect." What an Operator LLM Agent Actually Does in an Observe, Reflect, Plan, Act Loop
I hear this from skeptics a lot, and they are half right. The answer is grounded in cognitive science: the reasoning is real but non-deterministic, and the decision belongs to a deterministic layer.
When I tell an operations team that an agent observes the process, reflects on what it sees, and plans what to do next, and you get the same objection every time. Software does not reflect. It runs a query and returns rows. Call it reflection and you are either dressing up a lookup or selling something.
The objection is fair, and the two easy answers to it are both wrong. One overclaims: the agent thinks and understands the way a person does. The other dismisses: it is autocomplete, a stochastic parrot, nothing is happening. The evidence supports neither, the overclaim running into what these models cannot do and the dismissal into what they demonstrably can. The defensible answer is more useful than either, and it starts with something these teams already trust every day: a human operator.
Everything I claim below about how people and models do this is grounded in published research, named here in passing and cited in full in the working paper linked at the end.
Start with the operator you already trust
A panel operator runs a hazardous process, and their read of a situation is a fast inference: capable, right more often than not, but not deterministic. Ask two operators, or the same operator on two days, and the read can differ. We know this, and we still hand them the control room. Not because their reasoning is repeatable, but because the actions they are allowed to take are bounded by a deterministic structure around them: the permit to work, the interlocks, the procedures. The operator is trusted because their judgment is not the thing that closes the valve.
That division is the answer: Non-deterministic reasoning on the inside, a deterministic decision boundary on the outside. Reasoning and deciding are two different acts, and an operator has always been safe because the second one is governed, not because the first one is perfect.
The Human Operator and The Agentic Operator
The same division in a human operator and a MAGS agent: non-deterministic reasoning inside, a deterministic decision boundary outside.
Related: Operators, Not Creators
The objection hides an assumption about people
The objection assumes that human observation, reflection, and planning are themselves lookups: that you perceive by recording, remember by filing, and reflect by retrieving. Cognitive science left that picture behind decades ago. All three are active, constructive, generative acts.
Observation is not a recording. What we loosely call perceiving is, in the leading account, an act of prediction: the brain predicts what it expects to sense and builds the scene from that prediction, passing up only the mismatch to correct itself (predictive processing; Clark 2013; Friston 2010). What an operator experiences as simply seeing the panel is already an interpretation the brain generated. Observation is the act of forming that interpretation, a construction rather than a transcript.
Reflection is not a retrieval. The reason “explain it to yourself” works, one of the better-replicated findings in learning science, is that producing the intermediate steps forces gaps and contradictions into the open where you can catch them (the self-explanation effect; Chi et al. 1994, with a meta-analysis across 64 studies placing it at a moderate g of about 0.55; Bisra et al. 2018).
**Planning is not a lookup either.**It is mental simulation: construct a possible future, run it forward, and watch for where it breaks (Suddendorf & Corballis 2007). Under time pressure an expert recognizes the situation and simulates one promising action rather than scoring a menu of options (Klein’s recognition-primed decision model).
So the line the objection draws, between a generative machine and a retrieval-free human faculty, does not exist. These human faculties are generative too, and they are non-deterministic for the same reason: they construct an answer rather than look one up. That does not make a model a person. It turns the question from one of principle into one of degree.
Non-deterministic is not unreliable, and fluent is not reasoning
Two corrections keep the claim precise.
Non-deterministic does not mean unreliable. An operator’s read is non-deterministic and still right most of the time. So is a model’s. The property that matters for a control room is repeatability, not some idea that the reasoning fails; neither an operator’s judgment nor a model’s is repeatable, and that is why the decision cannot be left inside the reasoning.
Fluent is not the same as reasoning. The brain’s language network is separable from the systems that reason. People with severe aphasia, whose language is devastated, still solve logic and arithmetic problems and read other people’s intentions (Fedorenko & Varley 2016). The same split now frames how to read language models, which have largely mastered the form of language while remaining uneven at using it to accomplish things in the world (Mahowald et al. 2024). You cannot take fluent output as evidence of a sound decision. That has to be established outside the words.
What the model does in the ORPA cycle
With that in place, the positive claim is narrow and defensible. ORPA, Observe, Reflect, Plan, Act, is the cognitive loop these agents run, an industrial evolution of the OODA loop; the architecture is set out in full on the repository linked at the end. In each step, the model performs a functional analogue of the generative part of the human act, and it is strongest where an industrial agent works, over input that is already structured.
Observe. It fuses live telemetry and events with retrieved context into a read of the current state, grounded further by tools built on physics, engineering, and mathematics that it calls as it reads, so the read is anchored in real computation and not only the model’s word. This is the situation assessment an experienced operator runs.
Reflect. It synthesizes that read against memory into a higher-order account. This is the closest analogy in the loop, and it is measurable: making a model generate intermediate steps improves its multi-step performance for the same reason self-explanation helps people (Wei et al. 2022).
Plan. It produces intent: the course of action it proposes. Committing to that intent is a separate act.
Act. This is where the model stops. Observe, Reflect, and Plan are the generative work; the Act is carried out by a deterministic layer, or routed to a human.
Recommended by LinkedIn
Where the skeptic is right
Now the objections that hold. Three limits are real, and none is fixed by a bigger model.
Grounding. The model has no referential link to the physical process. When it emits the tokens for “high system temperature,” the meaning lives in the humans who read it, not in the model (the symbol-grounding problem; Harnad 1990).
Faithfulness. The model’s account of its own reasoning is not reliable. Plant a hidden bias in the input and the model follows it while writing a confident explanation that never mentions it (Turpin et al. 2023). The thing that most looks like reflection cannot be trusted as the reason for the output.
Robustness. The model is weakest at the rare and novel events that matter most, which in a plant are the moments you can least afford it to fail.
Together these refute the strong claim. A model on its own is not observing, reflecting, and planning in the grounded, self-aware sense a person is, and asserting otherwise in a control room is a a safety hazard.
The objections are the specification
Those three limits are not a verdict against the agent. They are the specification for exactly what a control room already builds around a non-deterministic operator, now built around a non-deterministic model.
Grounding is supplied from outside the model. The meaning that drives a decision is anchored in recorded measures and objective functions the system owns, and the read is anchored further by the physics and engineering tools the agent calls, so the reference comes from the layer in contact with the process.
Faithfulness is handled by not asking the model to be its own witness. The generated words are logged for audit, and the actual decision, whether something matters, which plan is allowed, whether anything acts, is a deterministic function of recorded state. Given the same state it returns the same decision every time, which is what makes it reproducible and auditable, and that is where the reliability the question asks about lives.
Robustness is supplied by the gate. Action passes only through thresholds, constraints, and corroboration across independent agents, none of which depends on the model’s confidence, so a confident answer at the edge of what the model has seen never becomes an action on its own.
This is the operator’s permit-and-interlock layer, expressed as a reproducible policy.
Agents generate intent. The governed layer owns the conversion of that intent into a validated decision and action.
The answer to the skeptic
Does software reflect? A model does the generative part of reflection, which cognitive science shows is what the word actually names, and it does not do the deciding. Reasoning in one of these agents is not a property of the model. It is a property of the whole system: the model generates, non-deterministically, and a deterministic layer grounds the meaning, declines to trust the model’s account of itself, and gates the act.
That is how an LLM agent decides reliably in an Observe, Reflect, Plan, Act loop, and it is the same division that has always made a non-deterministic human operator safe in a control room. The full argument, with every claim graded and cited, is in the working paper on our GitHub.
Related: Who Controls the Orchestration Controls the Plant
Github links:
Reasoning in Governed LLM Agents: A Functional-Analogy Account ofObservation, Reflection, and Planning Other XMPro MAGS papers Multi Agent Repo
Pieter van Schalkwyk is the CEO of XMPro, specializing in industrial AI agent orchestration and governance. Drawing on 30+ years of experience in industrial automation, he helps organizations implement practical AI solutions that deliver measurable business outcomes while ensuring responsible AI deployment at scale.
He authored the Industrial AI Agent Manifesto, published by the Digital Twin Consortium in February 2026, a governance framework for trustworthy autonomous operations.
Our GitHub Repo has more technical information. You can also contact myself or Gavin Green for more information.
Read more on agentic operations at The Digital Engineer
This article originally appeared on XMPro CEO’s LinkedIn blog, The Digital Engineer, on 4 July 2026.
