Benchmarking best practices

Cura 1T's published results use a specific protocol — pass@1 at T=1.0 with thinking enabled. Reproduce that protocol or your numbers won't be comparable.

Cura 1T is a research model, not a medical service, and not a substitute for a clinician. Benchmark scores do not establish safety for unsupervised clinical use.

Reference configuration

Python
response = client.chat.completions.create(    model="actava/cura-soar",    temperature=1.0,          # the published protocol; do not benchmark at T=0    max_tokens=32768,          # room for the reasoning trace + the answer    extra_body={"thinking": {"type": "enabled"}},    messages=[...],)

Rules of the road

  • Don't benchmark thinking models greedily. Long-reasoning models can degenerate at near-zero temperature; T=1.0 is both the published setting and the stable one.
  • Never cap the trace. A truncated answer (finish_reason: "length") scores as a wrong answer — that measures your token budget, not the model.
  • Generous timeouts. Hard clinical questions produce long traces; per-example wall-clock limits that clip them mis-score capability as failure.
  • Report the protocol. Temperature, sampling count (pass@1 vs pass@k averaged over seeds), token budget, and harness — the same discipline as the published evaluation notes.
← Prompt best practicesOpenAI compatibility →