Diffusion language models

1 The left-to-right habit

Autoregression — predict the next token, append, repeat — has been the only game in this course, and for good reason: it's simple, it trains on an exact likelihood, and it matches the way text seems to unfold. But it bakes in two constraints. Generation is strictly sequential, one token per forward pass, so a thousand-token output takes a thousand passes. And it only ever looks left — a token, once written, can never be revised in light of what comes after.

Other domains long ago abandoned this. Diffusion models — the technology behind modern image, audio, and video generators — don't produce their output in order at all. They start from pure noise and run a small, fixed number of refinement steps, each one improving the entire output simultaneously. The question this chapter asks is whether the same recipe works for language.

2 What "noise" means for discrete text

Image diffusion has it easy: pixels are continuous, so you can add a little Gaussian noise, then a little more, until the picture dissolves into static — and train a model to reverse each step. Text is discrete. There's no "slightly noisy" version of the word cat. So discrete diffusion uses a different corruption: masking.

The forward process progressively replaces tokens with a special [MASK] symbol — a few at first, then more, until the whole sequence is masked. The model is trained to run that backwards: given a partly-masked sequence, predict what the masked tokens should be. Generation then starts from an all-masked sequence and unmasks its way to text.

3 Refine the whole sequence in parallel

Here's the part that makes it interesting. Each denoising step looks at the entire sequence at once — every position, masked or not — and predicts all the masked tokens together. It then commits the predictions it's most confident about, leaves the uncertain positions masked, and moves to the next step. Over a handful of steps, the sequence sharpens from all-mask to finished text.

Two things fall out of this. The number of steps is fixed and small — a few dozen, not one per token — so a long sequence doesn't cost proportionally more steps. And every step has bidirectional context: a token can be filled in based on words to its right that don't exist yet under autoregression. The model can lay down anchor words first and fill the gaps around them, in any order it likes.

4 Diffusion vs autoregressive

Line them up and the trade is clear. Autoregression takes N sequential steps for N tokens, sees only the left context, can't revise, but trains on an exact likelihood and is, today, the strongest approach for text. Diffusion takes a fixed T steps regardless of length, sees the whole sequence, can revise earlier tokens as later ones firm up — but each of those T steps is a full forward pass over the entire sequence, and its training objective is a looser bound, not the exact likelihood.

So "fewer steps" doesn't automatically mean "faster" — a diffusion step costs more than an autoregressive one, and you need enough steps for quality. The honest framing is that they occupy different points on the speed/quality/flexibility surface, and which wins depends on the sequence length, the hardware, and how much revision the task rewards.

5 Why autoregression still rules text

Despite the appeal, the frontier of language modelling is still overwhelmingly autoregressive, and there are real reasons. Left-to-right factorization matches the causal grain of language and gives an exact, easy-to-optimize training loss; the KV cache makes AR generation cheap per step; and decades of tooling and scaling know-how are built around it. Diffusion's looser objective and full-sequence steps have, so far, left it a step behind on raw quality.

But it's an active, fast-moving frontier — recent diffusion and masked-generation language models have closed much of the gap, and the parallelism is genuinely attractive for long outputs and controllable, infilling-style generation. Diffusion already owns image, audio, and video; whether it takes a real share of text is one of the open questions of the field.

6 Reading the playground

The two columns illustrate the generation processes on the same target sentence: autoregressive fills left to right, one token per step; diffusion starts fully masked and unmasks confident tokens across the whole sequence over a few steps. It shows the mechanism and the step counts, not a trained model's samples.