Your Agents Don't Need a Bigger Model. 615 Business Cases Show Why.
Researchers at Wharton, Harvard Business School, Carnegie Mellon and the University of Pennsylvania have built an AI benchmark out of business school case studies. They took 615 open-ended questions from 238 cases across eighteen disciplines, with nothing multiple choice anywhere in it, and scored every answer against a checklist drawn from the instructor’s own written solution. Human graders checked the scoring before they published.
A case question gives you the kind of problem a manager actually gets. You have half the facts you want, several options that all cost something, and other people with a view on which one you pick. You have to choose, and then explain the choice to somebody who was not in the room.
Frontier models score 88.4%, 87.2% and 81.6% for partial credit on these questions. Getting every point on the checklist is much harder, and the best model manages that on about half of them.
The authors added a short section after the main work was finished.
A much bigger model changed nothing
Anthropic released Claude Fable 5 in June. It is far larger than anything public before it, and it was sold on software engineering.
The team ran it against the same 615 questions. It scored 88.0% for partial credit against 88.4% for Claude Sonnet 4.6, and 50.9% on full checklist satisfaction against 49.6%. The confidence intervals overlap on both measures, and it did not change which of the eighteen disciplines are hard and which are easy.
A far bigger model, tuned for exactly what the industry is chasing, moved business analysis by about a point in either direction.
The authors draw one conclusion and limit it to a single model: optimizing for software engineering workflows does not necessarily improve the capabilities of these models on this class of knowledge work.
Coding is not the same skill as running a business
The industry has spent two years optimizing for a specific list. The benchmarks that drive model releases measure factual recall, short question answering, math, code that compiles with tests that pass, and tool use inside an agent loop. Labs tune hard against those numbers because those are the numbers everybody reads, and the numbers have climbed a long way.
That list does not describe the work of running a business.
Operational work starts with a situation you only half understand. You weigh a maintenance budget against a production target, think about what a competitor does next, what the union says, what the regulator asks for in nine months. Nobody hands you a test that passes or fails. Then you pick an option knowing the others had merit, and you write down why you picked it, because in a year somebody will ask. A model can be excellent at writing a Python function and have contributed nothing to any of that.
Scale went up. Software engineering ability went up. The case scores did not move.
Four OpenAI generations over roughly two years climb from 63.9% to 87.2% on this benchmark, a gain of 23.3 points spread across every discipline with enough questions to measure it. That is a real improvement and I would not argue with it. The most recent step, GPT-5 to GPT-5.4, is worth 2.3 points.
Each release now buys about two points of business judgment. The launch announcements have not gotten any smaller.
What this means for how you build
I have argued before that the model is a component you swap. This paper measures it, and it also shows which part of an agent depends on model quality.
Our agents run an observe, reflect, plan, act cycle. Each step needs a different amount of intelligence, and it is worth taking them one at a time instead of treating the agent as a single consumer of tokens.
Observe is a data problem. It means live values from a historian, a control system and a maintenance record, arriving on time with their units intact. This is the step that breaks most often in the pilots I see, and the fix is always a better connector, never a bigger model.
Recommended by LinkedIn
Reflect is the step where model quality matters, and the job is narrow. The agent has to decide whether the current state is worth acting on, given what it knows now and what it remembers from the last time this happened. That is a judgment call against a threshold and a memory. On this paper’s evidence, judgment of that kind is not where the last two model generations improved.
Plan is the most demanding of the four, and it is the one step where a better model produces a better answer. Mapped to the O*NET work activity taxonomy, the weakest activity in the whole study is “identify business or organizational opportunities” at 70.9%, and the authors call that mapping indicative rather than final. Those questions score badly because they combine wide-open advisory framing with quantitative reasoning specific to the case, which is exactly what you are asking for when you hand a model an unbounded planning step. Give the step a fixed option set, real constraints and a goal it can score itself against, and you have removed most of the problem with engineering.
Act should not touch a model. The agent takes a bounded action inside a permission the plant granted in advance, and the action gets logged.
One step in four benefits from a frontier model, and the benefit is smaller than the release notes suggest.
The upgrade you are waiting for may not arrive
The failure mode I see most often is architectural procrastination. A team runs a pilot, the results come back uneven, and somebody decides the next model will close the gap, so the hard work gets deferred: the data plumbing, the decision rights, the guardrails, the record of what happened and why. Six months later the next model arrives and it is worth two points on business judgment, which does not rescue an architecture.
One line in that same section of the paper notes that Fable 5’s deployment was temporarily stopped under US export controls over cybersecurity concerns. Do not put a model at the center of a system that has to keep running if a government can switch it off. You rent the model and you own everything around it, so build until swapping it is a configuration change.
What the benchmark does not test
These are written narratives, authored by experts, self-contained, and graded against a written answer. A plant does not work that way.
None of these questions arrive as a 3 AM alarm with three tags reading nonsense and a compressor about to trip. They do not have to satisfy a regulator, survive a management of change process, or stay correct when the historian stalls for ninety seconds. The benchmark tests models on written text, which is what they handle best, and on the cleanest version of the task.
Take the best of the three models on each question and the score reaches 92.8%, four and a half points above the strongest single one. Only 7% of questions leave all three below 70%. That is an argument for several different models working a problem from different angles, which describes an agent team.
Where that leaves the buying decision
The two-year gain everyone quotes came from generations that are already behind us. The models you can buy today score within a few points of each other, and the spread is closing.
What separates a pilot from a plant is the data path into the loop, the boundary the agent can act inside, the memory that carries forward, and a record somebody can defend later. We build for that at XMPro, and the models we run inside it were good enough eighteen months ago.
If your agent strategy is waiting on the next frontier release, the wait will not pay for itself.
Pieter van Schalkwyk is the CEO of XMPro, specializing in industrial AI agent orchestration and governance. Drawing on 30+ years of experience in industrial automation, he helps organizations implement practical AI solutions that deliver measurable business outcomes while ensuring responsible AI deployment at scale.
Source: Ajay Patel (Wharton), Kartik Hosanagar (Wharton), Ramayya Krishnan (Carnegie Mellon, Heinz College), Chris Callison-Burch (University of Pennsylvania) and Karim Lakhani (Harvard Business School), “Frontier AI performance across the business disciplines: a case-grounded benchmark of knowledge work and analytical reasoning” (BusinessCaseBench, arXiv 2607.16057v2). Project page: https://business-ai-benchmark.github.io
Our GitHub Repo has more technical information. You can also contact me or Gavin Green for more information.
Read more on MAGS at The Digital Engineer
This article originally appeared on XMPro CEO’s LinkedIn blog, The Digital Engineer, on 4 August 2026.
