Read the scorecard

Where to look first

Every trial under logs/experiments/.../trial_*/ (or logs/submissions/.../<task>__<hash>/) writes the same set of verifier files:

scorecard.json shape

{
  "binary_reward":     0.0,
  "fractional_reward": 0.91,
  "passed_checks":     42,
  "total_checks":      46,
  "check_scores":  { "<check_name>": 1.0 | 0.0 | null, ... },
  "checks":        { "<check_name>": true | false | "not_applicable", ... },
  "failed_checks": ["md.signed_off", "judge.md_review:mp1126_sca_05", ...],
  "not_applicable_checks": ["md.denial_rationale_present"],
  "stages": {
    "md_review": {
      "passed": false,
      "checks": { "md.decision_exists": true, "md.signed_off": false, ... },
      "passed_count": 3,
      "total_count":  4,
      "not_applicable_count": 1,
      "details": { "criteria": [...] }
    },
    "outcome":     { ... },
    "cross_stage": { ... }
  }
}

The two reward axes

• binary_reward is the strict pass/fail axis. 1.0 only when every non-N/A check passes; 0.0 if any required check fails. This is what the leaderboard publishes as pass@1.
• fractional_reward = passed_checks / total_checks is partial credit for diagnostics. A 0.0 / 0.91 split means "near miss": one or two checks tanked an otherwise-strong run.

Stages

Checks are grouped by workflow stage. When you see binary_reward: 0, scan stages first — the failed stage is where to start debugging. Each stage records its own passed, passed_count, total_count, not_applicable_count, and free-form details.

Stage names you'll encounter:

Check-name namespaces

Every check name has a prefix telling you who graded it and on what.

Deterministic checks

Other deterministic prefixes: intake.*, nurse.*, p2p.*, appeal.*, provider_pre_submission.*, provider_request_package.*, provider_submission.*, new_referral.*, e2e_consistency.*.

Rubrics-based LLM judge

Named judge.<stage>:<rubric_id>, e.g. judge.md_review:final_decision = the final_decision rubric item evaluated as part of the md_review stage.

• Run by WorkspaceJudge — a Claude model (default claude-opus-4-7, override with CHI_BENCH_JUDGE_MODEL) reading the task's fixtures/judge/rubrics.json against the agent's workspace.
• Set CHI_BENCH_JUDGE_NUM_VOTES > 1 for majority-voted judging.
• Degraded judge runs (timeout, crash, malformed verdicts) collapse binary_reward to 0 even on gate-pass runs — visible as judge_unavailable_reason on the scorecard. Rubrics missing a parsed verdict count as not passing.

Three check states

Inside checks, every entry is one of:

• true — pass. Counts toward both passed_checks and total_checks.
• false — fail. Counts toward total_checks only; appears in failed_checks.
• "not_applicable" — rubric/check doesn't apply (e.g. md.denial_rationale_present for an approve decision). Excluded from total_checks; appears in not_applicable_checks.

Mirror in check_scores: 1.0 for pass, 0.0 for fail, null for N/A.

Care Management scorecard fields

CM trials share the same pass criterion as PA/UM — binary=1.0 requires both the deterministic gate and all LLM judge rubrics to pass. The scorecard schema differs in two ways (compute_cm_reward in src/chi_bench/verifier/stages/cm_rubric.py):

• gate_pass — bool. Did the deterministic gate (chart review exists, assessment completed, care plan finalized, cross-stage invariants) pass?
• rubric_yes_count / rubric_total — counts from the LLM judge rubric. fractional_reward = rubric_yes_count / rubric_total (rubric-only, not passed_checks / total_checks).

If the gate fails, the LLM judge is skipped (no point spending a judge session on a clearly-failed run); the scorecard still records which gate check broke. Reward semantics:

Worked example

From a real UM medical-director-review trial:

{
  "binary_reward":     0.0,
  "fractional_reward": 0.91,
  "passed_checks":     42,
  "total_checks":      46,
  "failed_checks": [
    "md.signed_off",
    "judge.md_review:decision_rationale",
    "judge.md_review:preop_weight_history",
    "judge.md_review:preop_psychosocial_eval"
  ],
  "stages": {
    "md_review":   { "passed": false, "passed_count": 3,  "total_count": 4  },
    "outcome":     { "passed": true,  "passed_count": 13, "total_count": 13 },
    "cross_stage": { "passed": true,  "passed_count": 2,  "total_count": 2  }
  }
}

Reading this top-down: 91% of checks passed, but md_review failed becausemd.signed_off was false (the agent never marked the determination as MD-signed) and three judge rubrics flipped to false.outcome and cross_stage were clean — so the agent reached the correct terminal state, but the artifact wasn't signed off and the rationale missed three rubric items. Binary = 0; this trial does not count for pass@1 even though everything else looks right.

Where the rules live

• Stage verifiers: src/chi_bench/verifier/stages/ — one file per stage.
• Reward computation: build_scorecard() and verify_task() in src/chi_bench/verifier/task_runtime.py; compute_cm_reward() in stages/cm_rubric.py.
• LLM judge: verifier/judge/workspace_judge.py + verifier/judge/cm_adapter.py.
• Per-task expectations (the ground truth the verifier scores against): data/<domain>/tasks/<task-dir>/expectations.json — hidden from the agent.

Re-judging an old trial

If you upgrade the judge model or fix a rubric, re-score existing trials without re-running the agent:

uv run cb verifier rejudge --trials-dir logs/experiments/<dir>

This re-runs only the LLM judge against the saved exported_state.json and rewrites scorecard.json + reward.json in place. Deterministic checks are recomputed too. Source: src/chi_bench/verifier/rejudge.py.

See also

• Quickstart — runs one trial; you read its scorecard.
• Architecture — how the verifier fits into the trial container.
• Run experiments — single-trial CLI and submission lifecycle.
• Leaderboard repo — what CI re-validates against the packet's scorecards.