Sign in

Researchχ-Bench

Submit a runPut your agent on the χ-Bench leaderboard.

χ-Bench is open source, and a submission is one handoff between two repositories: you produce a packet with cb submission prepare, then open a pull request adding it to actava-ai/leaderboard. CI checks the schema and every trial; a maintainer reviews and merges.

Phase 1

Produce the packet

In your clone of actava-ai/chi-bench
  1. 1
    Install and download the dataset
    Clone actava-ai/chi-bench, install with uv sync --extra dev, then fetch the gated dataset from Hugging Face:
    uv run huggingface-cli login
    
    REV=chi-bench-v1.0.0
    uv run huggingface-cli download actava/chi-bench \
      --repo-type dataset --revision "$REV" --local-dir data/
    echo "$REV" > data/.chi-bench-version
    The pin in data/.chi-bench-version is what submission preflight verifies against your config's dataset.version.
  2. 2
    Set API keys and build the Docker image
    Copy .env.example to .env and fill in keys for the providers you intend to run. ANTHROPIC_API_KEY is always required, because the workspace judge is pinned to claude-opus-4-7. Then build the runtime image (~5 min, one-time):
    uv run cb docker build
  3. 3
    Write a submission YAML
    Copy configs/submission_example.yaml to configs/submissions/<your-id>.yaml and fill in id, team, contact, agent, and model. Optionally set notes and run.*.
    Bringing a custom agent harness or model endpoint? See docs/extending.md.
  4. 4
    Run trials and prepare the packet
    Four commands: preflight, run, monitor, package.
    # Schema + preflight: dataset pin, Modal/Docker, agent name
    uv run cb submission validate -f configs/submissions/<your-id>.yaml
    
    # Run all 3 domains. Default: one trial per task (pass@1).
    uv run cb submission run      -f configs/submissions/<your-id>.yaml
    
    # Check progress; safe to run while `submission run` is in flight.
    uv run cb submission status   -f configs/submissions/<your-id>.yaml
    
    # Curate the leaderboard-ready packet.
    uv run cb submission prepare  -f configs/submissions/<your-id>.yaml
    The packet lands at logs/submissions/<id>/packet/YYYY-MM-DD-<id>/, typically <100 MB. Workspace artifacts and Harbor scratch are excluded by design.
    packet/2026-05-13-<id>/
    ├── submission.json          # manifest: agent, model, results, provenance
    ├── results.csv              # leaderboard rows (one per domain + overall)
    ├── sub.yaml                 # frozen copy of your config
    ├── provenance.json          # git SHA, image digest, timestamps
    ├── README.md                # auto-generated headline summary
    └── trials/<domain>/<trial_id>/
        ├── result.json
        ├── verifier/scorecard.json
        ├── verifier/reward.json
        └── agent/trajectory.jsonl.zst
Phase 2

Open the pull request

In a fork of actava-ai/leaderboard
  1. 5
    Fork the leaderboard repo (one-time)
    gh auth login
    gh repo fork actava-ai/leaderboard --clone=false
    git clone https://github.com/<you>/leaderboard && cd leaderboard
    Subsequent submissions reuse this same fork.
  2. 6
    Open a PR adding the packet
    Three equivalent paths; pick one. They all run the same CI validator (.github/workflows/validate.yml).
    The helper validates locally, copies the packet to benchmarks/<bench>/submissions/<dir>/, creates branch sub/<bench>/<dir>, pushes to your fork, and opens the PR.
    python scripts/submit.py /path/to/packet/2026-05-13-<slug>/

    Useful flags: --no-fork, --no-open-pr, --on-conflict abandon|replace|bump-date.

  3. 7
    CI labels the PR; a maintainer reviews
    CI runs schema and trial-integrity checks and labels the PR valid-submission invalid-submission or needs-review, with a sticky comment summarizing each check. A maintainer spot-inspects a trajectory or two and merges if everything looks plausible.
    What CI checks
    • Directory name = YYYY-MM-DD-<slug>
    • Required files present: submission.json, results.csv, sub.yaml, provenance.json, README.md, ≥1 trial result
    • No unexpected files (.zip, .bak, hidden files except .gitkeep)
    • submission.json matches the JSON Schema
    • results.csv rows match the manifest
    • Per-trial integrity: required files, valid zstd, valid JSONL per line
    • Trial counts match results.per_domain.<domain>.n_trials
    • Per-file and total size limits
Policy notes
  • The leaderboard is pass@1. Set run.n_attempts: 3 to keep extra trials for your own pass@3 or pass^3 analysis; the manifest still publishes pass@1.
  • Partial submissions (--domain pa | um | cm on submission run) are accepted and marked partial on the leaderboard.
  • Resubmissions with a fresh date prefix are always fine. Older runs stay for the record; say in the PR if you want one removed.
  • One directory per PR. Each PR adds exactly one new directory under benchmarks/<bench>/submissions/. Schema or workflow changes go in separate PRs.

Building a producer for your own benchmark? The packet contract is in docs/submission-packet.md.