Research
DeltaFlow makes iterative language denoising more efficient
DeltaFlow combines bidirectional recurrent memory with noise-aware updates to improve language denoising, with experiments measuring generation quality and throughput at longer sequence lengths.
A language diffusion model revisits a sequence through successive denoising steps. At every step, information must move between positions so the model can refine the text. The cost of that repeated sequence mixing becomes an architectural concern as sequences grow.
DeltaFlow studies this component inside Embedded Language Flows, which generate text in a continuous embedding space. The work combines recurrent memory, bidirectional context, and noise-dependent control in a backbone designed for iterative denoising.
Carry context through two directions
The paper introduces two ways to move information across a sequence. DeltaFlow-A alternates the scan direction across layers. DeltaFlow-P runs forward and backward scans within each layer. Both preserve the order of prefix and control tokens while reversing target tokens for the backward pass.
That ordering matters because a denoising model receives several kinds of input together. The target sequence changes throughout generation, while prefixes and conditioning signals define how that sequence should be interpreted. The architecture gives these inputs explicit treatment during bidirectional computation.
Let the noise level control memory
DeltaFlow conditions memory decay and writing on diffusion time. The model can therefore adjust how information enters and leaves recurrent state as a sample becomes cleaner. A second training component, Temporal State Consistency, aligns intermediate representations at nearby noise levels.
The resulting hybrid stack combines bidirectional gated delta blocks with full-attention layers. This composition gives the researchers a way to study efficient global memory alongside exact attention-based correction within the same denoising network.
Read quality and throughput together
On OpenWebText, the reported 32-step sampling experiment gives DeltaFlow-P a generated perplexity of 21.228, compared with 24.218 for the full-attention reference. Unigram entropy remains comparable. The training schedules expose DeltaFlow to 36 billion tokens and the reference to 45 billion.
The separate throughput experiment measures the denoiser on a single H100. At a sequence length of 16k, DeltaFlow-P records a 2.72× speedup over the full-attention baseline. Keeping that measurement attached to its component and workload makes the result useful for architecture decisions: generation quality and computational throughput can be examined on their respective axes.
Research by Guangfu Guo and collaborators, including Actava.ai researchers Weiran Yao and Haolin Chen.
Source: DeltaFlow: Noise-Adaptive Bidirectional Gated Delta Networks for Embedded Language Flows.