What we built
U.S. healthcare runs on long, fragmented, policy-heavy workflows, the kind that take a nurse, a coordinator, and a medical director hours of clicking, calling, and chart-reading to push to a terminal action. Frontier AI agents are increasingly pitched as the natural automation candidate here. We wanted to know how close they actually are.
So we built a high-fidelity simulator of 21 real healthcare apps (provider EHRs, payer UM portals, care-management consoles), wired together by their own state machines, and dropped frontier agents into them with a 1,279-document Managed-Care Operations Handbook and 200+ role-scoped MCP tools. Each task starts with a real clinical case (a knee-replacement prior auth, a chemotherapy review, a diabetes care-plan kickoff) and asks the agent to drive it end-to-end to a submitted packet, a finalized determination, or a complete care plan with patient consent. When the agent stops, a composite verifier reads the full workspace and grades the run.
- 200+ role-scoped MCP tools
- 1,279-document handbook
- Workspace files and chart access
- Prior authorization · Provider5 apps
- Utilization management · Payer10 apps
- Care management · Care manager5 apps
- Deterministic contract
- LLM judge
- Passes only if both pass
Each of the three domains has 25 long-horizon tasks. To produce ground truth, a clinician (or an author wearing that hat) walked every task end to end on the live UI before it shipped: the average task takes 21 steps and the longest 40, and each is graded independently across three trials.
Why this is hard
Coding benchmarks like SWE-Bench and Terminal-Bench measure long-horizon execution against a clean, deterministic backdrop: the file system doesn't talk back, the build either passes or it doesn't. Healthcare workflows look superficially similar but are wired differently in three ways that almost no current benchmark combines.
Policy density. Every decision has to be grounded in a specific rule: a payer's medical-necessity criterion, an internal escalation procedure, a state regulation. There are thousands of them, they vary by payer and plan year, and “misreading the rule” and “applying the right rule incorrectly” are different failure modes that we score separately.
Multi-role composition. A single PA case touches a coordinator, a UM nurse, a medical director, and sometimes a peer-to-peer call between an MD on each side. The agent has to switch role context, authority boundaries, and what it's safe to write, and every handoff is terminal. Submit the wrong packet and there's no rollback.
Multilateral interaction. Some steps aren't tool calls but live multi-turn conversations: a peer-to-peer review with the payer's MD, an RFI back to the provider, a twenty-minute outreach call with a chronic-disease patient. The agent has to drop out of execute-mode, hold a real conversation, and carry the result back into the workflow.
How we grade a run
When the agent stops, the verifier reads everything it touched: world-store updates, files in the workspace, the full tool-call event trail, and any conversation transcripts. Two layers grade in parallel and both have to pass.
The deterministic contract covers checks that have a single right answer: did the case status reach the expected terminal state? did the right CPT codes land on the request? was a peer-to-peer raised when policy required one? The rubric LLM judge (pinned to claude-opus-4-7) handles items the contract can't formalize: was the medical-necessity rationale grounded in the cited policy? did the outreach call respect patient autonomy or quietly badger a refusing member into “yes”?
We report pass@1, pass@3 (any of three independent trials), and the strict pass^3 (all three) as a reliability metric.
- Per-stage states
- Side-effect artifacts
- Workspace documents
- Event log
- Conversation transcripts
Headline results
We ran 30 harness × model combinations: every frontier proprietary stack (Claude Code, OpenAI Codex, Gemini CLI) paired with that lab's closed-weight models, plus four open-source frameworks (OpenClaw, Hermes, OpenAI Agents SDK, DeepAgents) sweeping five open-weight models. Best in class is Claude Code + Opus 4.6 at 28.0% pass@1. The per-domain leaders split cleanly across three labs: Codex + GPT-5.5 leads PA at 29.3%, Opus 4.6 leads UM at 41.3%, and Opus 4.7 leads CM at 32.0%. There is no single “best agent” for healthcare workflows yet.
- GPT-5.529.3
- GPT-5.424.0
- Opus 4.724.0
- Sonnet 4.624.0
- Opus 4.618.7
- Gemini 3 Flash18.7
- Gemini 3.1 Pro14.7
- GPT-5.4 Mini10.7
- Haiku 4.50.0
- Opus 4.641.3
- Sonnet 4.634.7
- GPT-5.532.0
- Gemini 3 Flash18.7
- GPT-5.417.3
- Opus 4.717.3
- Haiku 4.514.7
- GPT-5.4 Mini13.3
- Gemini 3.1 Pro6.7
- Opus 4.732.0
- Opus 4.624.0
- Sonnet 4.620.0
- GPT-5.46.7
- Haiku 4.54.0
- GPT-5.51.3
- GPT-5.4 Mini1.3
- Gemini 3.1 Pro0.0
- Gemini 3 Flash0.0
Reliability is the bigger problem. No agent clears 20% on pass^3, the metric where the same agent has to pass the same task three runs in a row. Agents that win one trial often fail the next on the same case. Two stress tests collapse the headline number further:
The marathon. Load all 25 tasks of a domain into a single agent session and ask it to finish them in any order. The best agents finish with overall 3.8%. On PA, neither leading agent submits a single authorization across 25 queued cases despite touching most of them. Long context doesn't save you: Opus 4.7 (1M-token context, no compaction) and GPT-5.5 (auto-compacts 4–6× per session) fail in roughly the same shape.
The end-to-end arena. Wire two Codex + GPT-5.5 agents together, one as the provider and one as the payer, and let them exchange information only through MCP tools. The PA configuration that scores 30.4% solo collapses to 0% when the payer agent and cross-role checks join: 18 of 23 cases never reach an MD decision, and on the five PA cases that policy requires a peer-to-peer call, neither side raises one.
Cost vs accuracy
Spending more does not buy reliability in healthcare workflows. Plot every harness × model configuration by mean per-trial spend and pass@1 and the field falls into four quadrants. The Sweet Spot (cheap and accurate) is sparsely populated and dominated by a handful of frugal open-weight pairings. The Premium tier (Claude Code paired with Opus 4.6 or Sonnet 4.6, and Codex with GPT-5.5) buys the top of the leaderboard but at 4–50× the trial cost of the frontier's budget end. Everything to the right of the $1 line and below the 13% bar is Overpriced: Claude Opus 4.7 on OpenClaw spends almost $12 per trial to land at 17.3% pass@1.
The connecting line traces the cost-accuracy Pareto frontier: no other configuration is both cheaper and more accurate than the points on it. Its seven configurations, labelled on the chart, span two orders of magnitude in spend for a 22-point accuracy spread.
Where agents break down
A 28% vs. 21% leaderboard gap suggests the leading agents are doing roughly the same thing, only at different success rates. They aren't. We pulled the per-check, per-tool-call data from 225 trials each for the two leading configurations from different labs, Claude Code + Opus 4.6 (28.0% pass@1) and Codex + GPT-5.5 (20.9% pass@1), and the failure shapes are wildly different.
Claude is the careful process-follower that submits packets it shouldn't. It nails every mandatory peer-to-peer (12 of 12) and writes care plans with 100% goal-structure compliance, but fails 0/33 of the “gather more evidence and put the case on hold” tasks because it always submits. Codex is the fast shortcutter: 1.5–2× faster per trial, better at form-filling, but 88% of its outreach calls fail the patient-conversation rubric, and it racked up 122 consecutive retries on a single UM trial when it couldn't format a tool call correctly.
Each agent is good at the things the other one is bad at. Both fail in ways that would cause actual harm in production. Below: their strengths and weaknesses side by side, then their failure rate on each rubric check.
- Procedural fidelityPayerNever skips a mandatory step; peer-to-peer raised 12 of 12 times
- Role-switchingPayerKeeps the right authority boundaries across 5 UM roles
- Slow execution6.6 min per PA trial and 68 steps on average, vs 3.5 min and 46 for Codex
- Documentation gap detectionProviderPasses 6 of 33 hold tasks, where Claude passes none
- Intake efficiencyProvider40% pass on structured form-filling tasks
- Skips mandatory stepsPayerBypasses the peer-to-peer call when the clinical answer seems obvious
- Hard-coded triage routingPayerA bug that breaks gold-card auto-approvals
- Rushed to submit?
- Fabricated paperwork?
- Wrong case status?
- Wrong treatment location?
- Missing decision on file?
Cite this
If you reference χ-Bench in a paper, post, or eval, here's the BibTeX. The data, code, and all 75 task definitions are open-sourced at the GitHub repo linked above.
@misc{chen2026chibenchaiagentsautomate,
title={CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?},
author={Haolin Chen and Deon Metelski and Leon Qi and Tao Xia and Joonyul Lee and Steve Brown and Kevin Riley and Frank Wang and T. Y. Alvin Liu and Hank Capps MD and Zeyu Tang and Xiangchen Song and Lingjing Kong and Fan Feng and Tianyi Zeng and Zhiwei Liu and Zixian Ma and Hang Jiang and Fangli Geng and Yuan Yuan and Chenyu You and Qingsong Wen and Hua Wei and Yanjie Fu and Yue Zhao and Carl Yang and Biwei Huang and Kun Zhang and Caiming Xiong and Sanmi Koyejo and Eric P. Xing and Philip S. Yu and Weiran Yao},
year={2026},
eprint={2605.16679},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2605.16679},
note={Accepted to NeurIPS 2026},
}