Blog
Right model, right job: using Cost, Quality, & Compliance to pick a model
Claude Opus 5 resolves 54.7% of tasks on ACTAVA's healthcare agent benchmark under one harness and 37.3% under another. Same weights, same tasks, same rubrics. Model selection only becomes answerable once you name the job; then score that step on cost, quality, and compliance.
By Weiran Yao
45 agent configurations now sit on the χ-BENCH leaderboard, our public benchmark for long-horizon, policy-rich healthcare work. Prior authorization, utilization management, care management. 60 to 80 steps per trial, irreversible commits, real policy rubrics doing the scoring.
The top score wasn't the interesting result.
This was. Claude Opus 5 resolves 54.7% of tasks at pass@1 under one harness and 37.3% under another. Same weights, same tasks, same rubrics. Narrow it to prior authorization and the gap opens up to 72.0% versus 20.0%.
Nothing about the model changed. Everything about the job around it did.
Look further down the leaderboard and it gets sharper. A healthcare-specialized model called MedGuard sits 11th overall at 22.7%. On care management it scores 60.0%, tied for the best result anywhere on the board. On prior authorization it scores 4.0%.
Pick your model off the overall ranking and you'd never route care management to the model that's best at care management.
Which is the whole problem. "What's the best model" has no answer. It becomes a question only after you name the job.
Three faces, one cube
Every model decision gets scored on cost, quality, and compliance. Most teams work those one at a time, as a checklist, in that order. They're a cube (pun intended, and I'll keep going). Turn one face and pieces move on the other two.
You downgrade the model, and the quality face scrambles
The cheaper model finishes 6 steps of 8, so the workflow retries, escalates, and lands on a human. Your token bill drops. Your cost per completed task goes up, and now it's hiding in someone's queue instead of your invoice.
You route everything to the frontier, and both other faces scramble
Spend climbs on the 80% of steps that never needed it. Meanwhile the vendor deprecates the version you validated, ships a safety update that shifts behavior on clinical edge cases, and your evidence trail now spans 3 model versions nobody approved.
You move everything in-house, and the quality face scrambles
Running every step on a small model inside your own boundary solves data residency in one move. It also puts a 6% model on the reasoning step, which is a different kind of audit finding.
The most popular solve is peeling the stickers off. One default model for the whole company, flipped down a tier when the bill arrives, flipped back when someone complains. The cost face looks solved. Nobody's checking the other two.
A cube has 6 faces and no way to work on one without touching the rest. Neither does this.
The centers don't move
On a physical cube the 6 center squares are fixed. They never move relative to each other. Rotate whatever you want, the white center stays white and stays opposite yellow.
Compliance is a center. You don't rotate your way out of it.
We use one question to find it. If this turns out to be wrong, who gets sued, fined, decertified, or told to give the money back? A model can produce the answer. It can't be the defendant.
Answer that per step and you've fixed the centers before cost or quality gets a vote. Where the weights run. What data crosses which boundary. Whether you can reconstruct the decision 18 months from now, on the version that actually made it.
Corners belong to all three faces
Corner pieces touch 3 faces at once, which is why they're the last thing anyone solves.
Some workflow steps are corners. High volume, so cost bites hard. Policy-rich and specific to your organization, so a general model's quality doesn't transfer. Sitting under a signature, so compliance binds. All 3 constraints are live simultaneously and relieving one breaks another.
Corners are the steps worth owning a model for. Everything else is an edge or a center, and you rent it.
The Rubric
What to score on each face
Score the cost of one completed unit of work. Retries included, plus the human who cleans up what failed.
Not price per million tokens.
Score pass rate on this step, on your data, against your rubric, at the step level.
Not rank on anyone's leaderboard.
Score whether you can name who signs, where the weights ran, and which exact version answered.
Not a vendor attestation.
Cost per completed task, not cost per token
Claude Haiku 4.5 is the cheapest model on our board. It scores 6.2% overall and 0.0% on prior authorization. Cost per completed prior auth is undefined, in the mathematical sense. A model that never finishes has no unit economics.
Small models are still the right answer for a lot of steps. Intake classification, document routing, field extraction, anything narrow and high-volume where a wrong answer gets caught immediately downstream. Using them isn't the mistake. Using them where failure is silent is.
Pass rate on your step, not rank on a leaderboard
Aggregate scores average away the signal you need. MedGuard's 22.7% overall tells you nothing. Its 60.0% on care management and 4.0% on prior auth tell you exactly where to put it and exactly where not to.
GPT-5.5 under Codex runs the same pattern in reverse, with 29.3% on prior auth, 32.0% on utilization management, and 1.3% on care management. Strong at 2 jobs, effectively absent on the third. One number for that model would tell you none of this.
So evaluate at the step, not the workflow. Score a workflow end to end and you've averaged a corner with a center and learned nothing about either.
Provenance you can reconstruct, not a vendor attestation
Three things to check per step. Where the weights run, and whether data leaves your boundary to reach them. Which exact version answered, and whether it's pinned. Whether you could reproduce a decision from last year if a regulator asked.
Frontier models get deprecated on the vendor's calendar, not yours. And a model that silently improves underneath you is still an unvalidated change to a system you signed for. Better is not the same as approved.
What the table looks like when you're done
| Step | What binds | Route to |
|---|---|---|
| Intake, classification, extraction High volume, narrow, errors caught downstream | Cost | The smallest fast model that clears your bar |
| Hard reasoning on rare cases Appeal rationale, ambiguous medical necessity | Quality, and volume is low enough that cost doesn't | Best available frontier model, re-checked every quarter |
| Policy-rich work that's specific to you Your medical policy exceptions, your delegated approval boundaries, at volume | All 3. This is a corner | A specialized model you own, post-trained on your own evidence |
| PHI-heavy steps under a residency or contractual constraint | Compliance, absolutely | Whatever the strongest model is that runs inside your boundary |
| The signature itself The order, the attestation, the denial letter | Nothing, because nothing routes here | A named human, with the full trail behind them |
That last row matters more than the other 4. The signature isn't a routing decision, and no amount of model capability turns it into one.
Keeping it solved
A routing table is a snapshot of a market that moves monthly. Ours changed twice this year. Four mechanics keep it honest.
- Pin the version. In KORA, the agent card carries the exact model the agent was authored against, pinned server-side when the agent is saved. A model tier for the general case, an exact model id when a step needs one. Nothing changes underneath a validated workflow without someone choosing it.
- Promote on regressed rows, not on averages. A candidate that's 4 points better on average while breaking 30 cases you already had right is not an upgrade. Gate every model change against the current baseline on your own data, count what regressed, and make preservation an explicit check rather than an assumption.
- Attribute cost per agent, per completed run. Total token spend tells you the size of the bill. Cost per completed task, per agent, tells you which routing decision was wrong and where to point the next fix.
- Put a review date on the table. Between χ-BENCH v1.0 and now, the top of our own leaderboard moved from 33.3% to 54.7%. Anything you concluded 6 months ago about which model is best is probably already wrong.
The variable almost nobody scores
One more result, because it changed how I think about all of this.
GPT-5.5 running in a general coding harness scores 29.3% on prior authorization. The same model, inside a harness a third-party team built specifically for prior authorization, scores 68.0%.
Same weights. 2.3× the outcome.
The scaffolding around a model carries as much of the result as the model does. Which tools it can reach, what state it holds across 60 steps, how it recovers when step 41 fails, whether it can tell an irreversible commit from a reversible one. That's why χ-BENCH ranks harness × model instead of ranking models, and it's why the same weights show up 4 times on our board with a 5× spread between the best and worst placement.
Right model, right job is incomplete. It's right model, right harness, right job, and the harness is the part you actually control.
The labs will keep shipping better weights, on their schedule, to your competitors at the same moment they ship them to you. The harness, the routing policy, the evaluation set, and the evidence your workflows generate are the parts that stay yours.
The bottom line
Nobody solves a cube by staring at one face. Nobody solves model selection with a company-wide default and an annual vendor review either, though plenty of organizations are trying.
Score every step on all 3 faces. Let compliance fix the centers, because it doesn't negotiate. Rent the frontier for the edges. Own a model for the corners, since the corners are precisely where your organization differs from every competitor running the same workflow on the same rented weights.
Right model, right job. The hard part was never picking the model. It was being specific about the job.
Contact ACTAVA
Route every step to the model that earns it
ACTAVA KORA stays model-independent by design, so policy-based routing sends hard low-volume reasoning to a frontier model and the proprietary, latency-sensitive long tail to a model you own. Pin the versions, gate the upgrades on your own data, and keep the evidence that makes both defensible. Start with one workflow and score it on all 3 faces.
Explore Own Your Models
Written by
Weiran Yao
CAIO & Co-Founder
