CoDA’s training recipe for a compact diffusion coder

Code completion often involves information on both sides of a missing span. CoDA investigates a diffusion-based approach to this setting, using a compact language model that learns to recover code from masked sequences.

A staged path into diffusion

CoDA builds on Qwen3-1.7B and divides training into pretraining, code-focused mid-training, and instruction tuning. The first stages use a custom TPU trainer, while the instruction-tuning stage develops responses conditioned on user queries.

A progressive masking schedule connects these stages. Training examples expose the model to masking patterns that prepare it for denoising an initially masked response. The schedule is a concrete part of adapting the model to its generation procedure.

Generation uses the surrounding context

During inference, the model fills a masked response through denoising steps. A confidence-based sampling strategy uses token uncertainty to guide the updates. The architecture supports bidirectional context, allowing generated code to draw on information around a span.

This makes training and sampling closely connected design decisions. The masking distribution determines the reconstruction problems presented to the model; the sampling policy determines how the model uses its predictions when producing code.

A compact coding research platform

The paper evaluates CoDA on HumanEval, MBPP, and EvalPlus and releases model checkpoints, evaluation harnesses, and training infrastructure. These artifacts expose the complete path from model adaptation to code-generation assessment.

For researchers studying coding models, the release supplies several points of comparison: the behavior learned during code-focused training, the effect of instruction tuning, and the choices made during denoising. Keeping those stages available supports direct experimentation with the relationship between model size, reconstruction training, and generated programs.

Haolin Chen and collaborators developed CoDA at Salesforce AI Research. Weiran Yao is a co-author. Read the paper and explore the released implementation.