Inside CHI-Bench’s simulated healthcare workflows

A prior authorization case crosses several systems and roles. Someone assembles clinical evidence, another reviewer applies coverage criteria, and later steps produce requests, decisions, or follow-up actions. Completing the case requires an agent to keep those records and responsibilities aligned across a long sequence of decisions.

CHI-Bench makes this workflow structure available for evaluation. Its simulated healthcare environment covers provider prior authorization, payer utilization management, and care management, with persistent application state and explicit handoff rules.

A case moves through connected applications

The environment contains 20 healthcare applications exposed through 87 MCP tools. An action in one application can create a record or advance a workflow in another. Agents encounter case statuses, document requirements, reviewer roles, and submission semantics as they work toward a terminal state.

This connected state gives each decision consequences within the simulation. A handoff carries forward the artifacts produced by the outgoing role. The next role must work with that accumulated record, making the quality of intermediate work part of the overall task.

Give agents a working handbook

Agents receive a managed-care operations handbook organized into role-specific workflows, clinical reference material, and platform tutorials. Its progressive structure lets an agent start with a role overview and retrieve detailed instructions as the case develops.

The benchmark’s reference workflows were walked through in the application UI and reviewed by healthcare practitioners and the research team. This process ties task expectations to concrete actions, policies, and resulting records. The evaluated set contains 75 representative long-horizon tasks.

Score the state left behind

Each trial is checked through a deterministic contract and a rubric-based judge. The verifier examines application state, event history, and interaction records. Passing requires both layers to accept the completed workflow.

Across 30 harness-and-model configurations, the best reported pass@1 is 28.0%. The paper also measures repeated-trial reliability and a marathon setting that carries tasks through one long-running session. These views help locate where execution quality changes as workflows repeat and context accumulates.

For teams building enterprise agents, the benchmark supplies a concrete evaluation pattern: define the role, preserve the intermediate artifacts, and check the resulting world state. Those records provide useful evidence for improving tools, instructions, and orchestration.

Research by Haolin Chen, Weiran Yao, the Actava.ai team, and collaborators; accepted as a NeurIPS 2026 poster. Explore the CHI-Bench environment and results.

Source: CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?