0% End-to-End: What 75 Real Workflows Say About Agent Readiness in Healthcare

0% end-to-end.

We ran 30 frontier agents through 75 real healthcare workflows. The best one finished 28% of them on the first try, and not a single prior authorization all the way from provider to payer.

actAVA χ-BENCH

Most agent benchmarks grade tasks. Summarize this note. Extract that code. Answer this question about a policy.

Healthcare doesn't pay people to do tasks. It pays them to finish work.

So we built χ-BENCH around work: 75 long-horizon workflows across provider prior authorization, payer utilization management, and care management, each one run inside a high-fidelity simulator with real policy behind it. Then we put 30 frontier agents from Anthropic, OpenAI, Google, x.AI, DeepSeek, and Z.ai through them.

The scores are lower than anyone's demo suggests. That's the useful part.

What a single χ-BENCH task actually asks for.

One trial runs an agent 60 to 80 steps across four to six clinical stages. The agent works through 21 simulated healthcare apps exposed as 200+ MCP tools, and it has to ground its decisions in a 1,279-document operations handbook.

A utilization review case looks like this. Read the physician's clinical notes. Apply the payer's medical policy. Query the formulary. Draft a compliant determination letter. Route the outcome to the right downstream team. No human stepping in at the awkward parts.

We don't just grade the final answer. Deterministic unit tests check the trajectory, every artifact produced, and the resulting world state, with an LLM judge on top for evidence grounding, consent, and cross-stage consistency.

That last one matters more than people expect. An agent can get stage three right and still fail, because what it wrote in stage three contradicts what it committed to in stage one.

The domain scores tell you where the floor is.

Broken out by domain, the best results were utilization review at 41%, care management at 32%, and prior authorization paperwork at 29%.

Read those in order and you can see the pattern. The more the work stays inside one organization's four walls, the better agents do. The more it crosses a boundary, the worse.

Then reliability. Run the same case three times and no agent clears 20%. On our strictest cross-stage consistency measure, none exceeds 8%.

A workflow you can only finish one time in three isn't automated. It's a coin flip with a bill attached.

The ceiling isn't model intelligence.

It's tempting to read 28% as "the models aren't smart enough yet," wait two quarters, and try again. I don't think that's what the data says.

These same models write production code and pass medical licensing questions. Intelligence per step is not the binding constraint. The binding constraint is the seam: 60 to 80 steps chained across four to six stages, where a single broken stage takes the whole run down with it.

Do the arithmetic. If an agent is 98% reliable at every individual step, a 70-step workflow finishes clean about 24% of the time. That is roughly where the leaderboard sits. Per-step excellence and end-to-end completion are different products.

Three things break at the seam.

01. Policy density.

Every decision has to be grounded in a large library of medical, insurance, and operational rules. Not summarized. Grounded, with the citation that survives an appeal. Agents that reason fluently still cite the wrong section, or the right section of last year's policy.

02. Multi-role composition.

A single χ-BENCH task makes the agent play several roles in sequence, with handoffs between them. Intake nurse, then medical director, then appeals. Each role has different permissions, a different standard of evidence, and a different definition of "done." Agents tend to carry the previous role's assumptions across the line.

03. Multilateral interaction.

The middle of these workflows is dialogue: peer-to-peer review, patient outreach, back-and-forth with a plan. The agent has to hold a position under pushback and change it only when the evidence changes. That's a governance problem wearing a conversation's clothes.

Why 0% should change your deployment plan, not your ambition.

Prior authorization is not a niche. The AMA's 2025 survey of 1,000 practicing physicians found practices complete 39 prior authorization requests per physician per week, consuming an average of 13 hours of physician and staff time, with 40% of physicians employing staff dedicated exclusively to the work.

The administrative case for automation is just as loud. The 2025 CAQH Index, built on data from more than 600 provider organizations and health plans covering 63% of insured lives, reported billions in remaining annual savings still sitting in manual transactions.

So the demand is real and the money is real. What χ-BENCH says is that buying "an autonomous prior auth agent" today, on a single vendor's word, is buying a number nobody has published.

Three moves I'd make this quarter:

• Scope to stages, not workflows. Deploy where an agent owns a bounded stage with a defined output and a human accepting the handoff. Utilization review at 41% is a very different bet than an unattended end-to-end run at 0%.
• Measure consistency, not accuracy. Ask any vendor for pass@1 on your workflows, then ask for the same case run three times. The gap between those two numbers is your real operational risk.
• Instrument the handoffs first. If you can't see which stage broke, you can't fix the agent or defend the decision. Stage-level evidence is the audit trail.

28% best pass@1 is not a failure. It is a calibration.

What we built on the other side of these numbers.

χ-BENCH is open under Apache 2.0. The code, the dataset, and the 1,279-page operations handbook are public, the leaderboard accepts community submissions, and we built it with 20+ clinical and academic collaborators including Johns Hopkins, Stanford, CMU, Yale, and Wellstar. If you think your stack does better, run it and tell us.

We publish these scores because they describe the problem KORA was built for.

KORA|BLUE is the harness: the orchestration layer that decomposes a 70-step workflow into stages with explicit state, role boundaries, and policy grounding at each one, so the model does per-step reasoning while the harness holds the seam.

KORA|RED runs the same evaluation discipline against your workflows, on your policies, with repeat trials, so consistency is a number you own rather than a surprise you discover in production.

KORA|GREEN feeds the failures back in. Every broken stage becomes a test case, and the test suite gets harder as the agents get better.

Model quality will keep climbing, and every point of it lands on your workflows only if something underneath is holding the structure together. That layer is yours to own.

Run your own numbers before you buy someone else's.

The workflows are long. The seams are where they break. Fix the seams and the model gets to do its job.

See where your agents break.

Run your agent on the open χ-BENCH leaderboard, or talk to us about testing your own workflows inside KORA.

Run your agent on the leaderboard →

Explore KORA →

Sources

• χ-BENCH (CHI-Bench), "Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?" arXiv paper and actAVA leaderboard.
• American Medical Association, 2025 Prior Authorization Physician Survey (1,000 practicing physicians).
• CAQH, 2025 CAQH Index, based on data from 600+ provider organizations and health plans representing 63% of insured lives.
• Newsweek, "Have Health Care's AI Ambitions Hit a Reliability Wall?"