Blog
ACTAVA's χ-BENCH Paper Is Headed to NeurIPS, and the Best Agent Still Fails 72% of Healthcare Operations Tasks
Actava's χ-BENCH paper was accepted as a poster at NeurIPS 2026. We built a simulated hospital with an Epic-style EHR, connected it to a simulated health plan, and gave 30 agent harness and model configurations real prior authorization, utilization management, and care management work. The best agent resolves 28.0% of tasks, no agent clears 20% on strict pass^3, and performance falls to 3.8% when all tasks share one session.
By Weiran Yao and Haolin Chen
Our ACTAVA χ-BENCH paper was accepted as a poster at NeurIPS 2026. The headline result is short. Across 30 agent harness and model configurations, the best agent resolves 28.0% of healthcare operations tasks.
No agent clears 20% on strict pass^3, which requires success on the same task in all three attempts. A hospital can't staff a prior authorization desk with that.
A simulated hospital wired to a simulated health plan
We built a hospital running an Epic-style EHR and connected it to a health plan running a UnitedHealthcare-style payer stack. The environment has 20 apps, 87 MCP tools, and a 1,290+ document operations handbook covering medical, insurance, and operational policy.
Then we gave agents real work in prior authorization, utilization management, and care management. Each domain has 25 tasks, for 75 in total.
On the provider side, the agent picks up a case, pulls the chart, places the order, and submits the prior auth. On the payer side, it reviews the case against medical policy, sets up the peer-to-peer, and issues the determination.
A task counts only when the case reaches a terminal status. Partial credit for a well-written letter doesn't exist here.
Three results worth repeating to your CIO
The best configuration resolves 28.0% of tasks. End-to-end prior authorization automation drops to 0%.
Strict pass^3 stays under 20% for every agent we tested. A demo that works once tells you very little about a queue that runs all day.
Put every task in a single session and performance falls to 3.8%. An operations desk doesn't reset between cases. This is the number we'd put on the first slide.
Three things make it hard
Decisions depend on a large library of rules. Medical policy, site-of-service requirements, consent scripts, and evidence standards all sit in the handbook, and the agent has to find the right page for each case.
One task means playing several roles. The agent acts as intake clerk, nurse reviewer, medical director, and peer-to-peer reviewer, and it hands work from one seat to the next.
Many steps are multi-turn conversations. Peer-to-peer reviews and patient outreach can't be undone once they happen.
One task, one flipped field
In prior authorization task pa_t008, a bariatric surgery case, the agent ran 69 steps over 7 minutes and 41 seconds. It submitted for outpatient when medical policy required an inpatient hospital, and it submitted when the right call was to gather more evidence. It passed 7 of 11 checks and scored a reward of 0.00.
Why we built an RL environment
We expect the same gaps to show up in other enterprise domains with dense policy, multiple roles, and irreversible steps. That's why we built χ-BENCH as an RL environment. The leaderboard ranks agents today, and the environment gives teams a place to train and improve them against the same simulated hospital and payer.
The clinicians and operations experts who helped make the simulation realistic came from 20+ hospitals and universities, including Johns Hopkins Medicine and Wellstar Health System. Thank you to our 33 co-authors.
What to ask a vendor
Ask for the pass^3 number on a workflow that ends in a determination. Ask what happens when every case shares one session. Ask how many of the 75 tasks reached a terminal status.
You can check our answers. The code, dataset, and operations handbook are open, and the leaderboard is live.
Run an agent through the same hospital. Browse the 75 tasks, read the paper, or submit a run to the leaderboard.
Explore χ-BENCH
Authors
Weiran Yao
CAIO & Co-Founder
AI research leader with deep expertise in large-scale model training, reasoning engines, and multi-agent systems. At Salesforce AI Research, Weiran led the post-training team behind the xLAM large action models, along with the Agentforce planning engine and the CodeGenie agent, and drove the pre-training of Salesforce's CoDA code diffusion model. He recently spearheaded open-source releases including Enterprise Deep Research, Webscale-RL, and CoDA-1.7B, advancing applied AI from research into production at enterprise scale. A CMU Ph.D. in Machine Learning, he has published 70+ papers, holds 13 U.S. patents, and serves on NeurIPS, ICML, ICLR, and ACL committees.
Haolin Chen
Lead AI Researcher
Experienced AI Researcher spearheading the development of cutting-edge agentic systems and reinforcement learning (RL) frameworks. Previously a Senior Applied Scientist at Salesforce AI Research, Haolin led high-impact projects including LaTRO (foundational RL-based reasoning), Webscale-RL for massive-scale data synthesis, and the xLAM series of agentic LLMs. His technical expertise spans the entire LLM lifecycle—from pre-training to hierarchical planning—and includes the development of CoDA, a diffusion language model for coding. A prolific researcher with 15+ publications in venues like NeurIPS and ICLR, Haolin holds a PhD in Applied Mathematics from UC Davis, specializing in the mathematical foundations of machine learning and tensor decomposition.
