Accuracy Is the Wrong Metric for Clinical AI Agents

Think of benchmark accuracy as a medical school entrance exam. A high score tells you the candidate can retrieve facts and reason under timed conditions. It tells you nothing about whether they can be trusted alone on a ward at three in the morning, with an incomplete chart, an interruption every four minutes, and a nurse waiting on an answer.

We keep publishing entrance exam scores and calling them clinical readiness.

I lead the Clinical AI Agent work at Oracle Health, and evaluation of human-centered AI systems is the core of my Ph.D. research. Those two vantage points converge on the same uncomfortable conclusion: the number most teams optimize is not the number that determines whether the system helps anyone. This post is about why, and what I think we should measure instead.

Accuracy measures a task nobody actually performs

Most medical benchmarks ask a model to answer a well-formed question with a single defensible answer. Real clinical work looks nothing like that. It is multi-step, it happens inside an EHR, and it ends with something changing in the world: an order placed, a note signed, a referral routed, a result flagged.

Agentic benchmarks make that gap visible. MedAgentBench evaluates LLM agents on 300 clinically derived tasks written by licensed physicians inside a FHIR-compliant virtual EHR built from 100 patient profiles and more than 700,000 data elements [1]. Tasks include retrieving labs, recording vitals, placing medication orders, and making referrals. The best model at publication reached a success rate of about 70 percent, with significant variation across task categories.

Read that as a practitioner rather than as a leaderboard watcher. Roughly one in three attempts at ordinary chart work did not complete correctly. And "success rate" is still a single scalar. It does not tell you which third failed, whether the failures cluster in ordering rather than retrieval, or whether the same task fails on repeat.

The question is not "how often is it right." The question is "what happens the times it is not, and can I see those times coming."

Failure mode one: the average hides the tail

Healthcare is not an average-case discipline. Nobody is comforted by a system that performs beautifully across a thousand routine cases and collapses on the emergency.

HealthBench, OpenAI's rubric-graded benchmark of 5,000 multi-turn health conversations, is one of the few evaluations designed to surface this. It was built with 262 physicians from 60 countries across 26 specialties, scoring responses against 48,562 rubric criteria spanning clinical accuracy, completeness, communication quality, context awareness, and instruction following [2]. What the results expose is more interesting than the headline scores: strong average performance can coexist with sharp fragility, where worst-case behavior falls off substantially and the critical behaviors, emergency referral and context seeking, remain the least reliable ones [3].

So here is a practice I would make mandatory. Run every evaluation case n times and report the worst run, not the mean. Call it worst-at-k if you like. In a clinical setting, nondeterminism is not a quirk of sampling, it is a safety property. A system that gives the right answer four times out of five on the same input has not earned an 80 percent. It has earned a defect report.

Failure mode two: not every wrong answer is wrong in the same way

Accuracy is a counting metric. It assigns exactly one unit of penalty to a missing pleasantry and exactly one unit to a tenfold dosing error. No clinician on earth grades that way, and no evaluation that grades that way is measuring clinical quality.

This is the real contribution of rubric-based evaluation, and it is why I think the field is right to move toward it. Each HealthBench criterion carries a point value weighted to match the physician's judgment of its importance [2], which means the score encodes clinical priority rather than string overlap. HealthBench Professional extends the same approach to clinician-facing work: care consult, writing and documentation, and medical research [4]. That is much closer to what a clinical agent actually does all day.

Build your error taxonomy before you build your scorer. Mine has four tiers: cosmetic, incomplete, misleading, and harmful. A system that trades three cosmetic errors for one harmful one has gotten worse, even though its accuracy went up.

Pro tip: If your evaluation cannot distinguish a formatting slip from a contraindicated medication, it is not a clinical evaluation. It is a text similarity check with a stethoscope drawn on it.

Failure mode three: the metric ignores the human holding it

This is the one that reorganized my thinking, and it is the reason I keep coming back to human-centered AI as a research frame rather than a slogan.

Start with the cautionary tale everyone in health AI should know by heart. The Epic Sepsis Model was deployed across hundreds of US hospitals. When researchers at Michigan Medicine externally validated it across 38,455 hospitalizations, it produced an area under the ROC curve of 0.63, well below the vendor's reported range. It missed 67 percent of patients with sepsis while generating alerts on 18 percent of all hospitalizations, creating a substantial alert fatigue burden [5]. A model that is both insufficiently sensitive and generously noisy does not fail quietly. It actively degrades the attention budget of every clinician in the building.

Now the finding that should be on a poster in every clinical AI team room. In a prospective study published in Radiology, 27 radiologists read 50 mammograms with the assistance of a purported AI system. When the system suggested an incorrect BI-RADS category, reader performance dropped across every experience level. Even very experienced radiologists fell from roughly 82 percent accuracy to about 45.5 percent on those cases [6].

Sit with that. The AI was wrong, and the effect was not that its wrongness was caught and discarded. Its wrongness propagated into expert human judgment and cut it roughly in half.

That single result invalidates the entire premise of evaluating a clinical model in isolation. The unit of analysis is not the model. It is the clinician plus the model, working together, under time pressure. Measuring the agent alone is like measuring a drug's potency in a beaker and calling it a clinical trial.

DECIDE-AI, the consensus reporting guideline for early-stage live clinical evaluation of AI decision support, makes this explicit. It treats these systems as complex interventions involving two collaborating forms of intelligence, and it establishes human factors as a core component rather than an appendix, alongside user variability and the biases that variability introduces [7].

The evaluation stack I would build

Five layers, in this order. Each one catches something the layer above it cannot see.

1. Verified end state, not claimed output. Do not score what the agent says it did. Score what actually changed. If the task was to place an order, read the record back and confirm the order exists with the right code, dose, and route. Text-level grading of an agent that takes actions is a category error.

2. Severity-weighted error profile. Report the distribution across your taxonomy, not a scalar. "94 percent accurate" means nothing next to "zero harmful, three misleading, eleven incomplete."

3. Worst-case reliability. Repeat every case. Report the floor. Track variance as a first-class metric, because variance is what a clinician experiences as untrustworthiness.

4. Calibration and abstention. Measure how often the system correctly declines or escalates instead of guessing. An agent that knows the boundary of its competence is worth more than a slightly more accurate one that does not. Appropriate deferral rate belongs on the dashboard next to task success.

5. Human outcome measures. This is the layer almost everyone skips and the only one that answers the actual question. Decision change rate. Rate of appropriate rejection when the agent is wrong. Time to decision. Edit distance on generated drafts. Downstream clinical outcomes where you can ethically obtain them. If your system raises task success by six points and lowers appropriate rejection by twenty, you have shipped a net harm with a great changelog.

The evaluation set you cannot download

Public benchmarks are necessary and nowhere near sufficient. HealthBench relies exclusively on synthetic dialogues, which limits ecological validity, and its model-based graders can share blind spots with the models being graded [8]. Every public benchmark carries some version of this ceiling.

Your institution's own cases, your patient population, your specialty mix, your documentation conventions, and your specific EHR configuration are not represented in any leaderboard. That gap is where deployment failures live. Run silent prospective evaluation before anything reaches a clinician, and instrument the live system so that the humans using it are part of what you are measuring, exactly as DECIDE-AI describes [7].

What I actually believe

Accuracy answers a question we already know the answer to: yes, frontier models know a great deal of medicine.

The question that determines whether any of this helps a single patient is different. Did the clinician end up making a better decision, faster, with less cognitive load and less risk of being led somewhere wrong? That question cannot be answered by a number the model produces about itself. It can only be answered by measuring the partnership.

Building AI that amplifies clinicians rather than replacing them is the research commitment I wrote about in my post on human-centered AI. Evaluation is where that commitment either becomes engineering or stays a slogan. If we keep grading the exam and skipping the ward round, we will keep shipping systems that are impressive on paper and quietly corrosive in practice.

Measure the pair. Everything else is a leaderboard.

References

1. MedAgentBench: A Virtual EHR Environment to Benchmark Medical LLM Agents, NEJM AI. Three hundred clinician-written tasks in a FHIR-compliant environment. Preprint version.
2. HealthBench: Evaluating Large Language Models Towards Improved Human Health, OpenAI. Physician-authored, weighted rubric criteria across five behavioral axes.
3. Reasoning LLMs in the Medical Domain: A Literature Survey. Analysis of how strong average benchmark performance masks worst-case fragility.
4. HealthBench Professional, OpenAI. Rubric-graded tasks across care consult, documentation, and medical research.
5. Wong et al., External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients, JAMA Internal Medicine, 2021. The definitive study on deployment performance diverging from vendor claims.
6. Dratsch et al., Automation Bias in Mammography: The Impact of Artificial Intelligence BI-RADS Suggestions on Reader Performance, Radiology, 2023. Incorrect AI suggestions degraded reader accuracy at every experience level.
7. Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI, Nature Medicine, 2022. Human factors as a core evaluation component.
8. HealthBench: Advancing AI evaluation in healthcare, but not yet clinically ready, PMC. On synthetic dialogue limits and shared blind spots in model-based grading.