Speculative decoding

1 The pass that's mostly wasted

From Chapter 13: generating a token is memory-bound. The expensive part of a forward pass is hauling the model's billions of weights out of memory, and that cost is paid whether the pass produces one token or processes a hundred in parallel. Standard decoding produces exactly one token per pass — which means almost all of that hauled-in compute capacity sits idle.

So here's the opening: a single forward pass can score many token positions at once for nearly free. If you only had some candidate tokens to score, you could verify a whole batch of them in the time it normally takes to make one. The catch is producing the candidates cheaply — which is where a second, smaller model comes in.

2 Draft and verify

The setup is two models: a large target (the one whose output you actually want) and a small, cheap draft model. Each round goes:

The draft model quickly generates a guess of the next K tokens, running itself K times — cheap, because it's small. The target model then takes that whole guess and verifies all K positions in a single forward pass. You accept the longest prefix of the guess that the target agrees with, and at the first disagreement you throw out the rest and substitute the target's own token. Every round costs one target pass but emits anywhere from one token (draft was wrong immediately) up to K+1 (draft was right the whole way, plus a free bonus token).

3 Lossless, not approximate

The surprising part: this is not a quality trade-off. The acceptance rule is designed so that the tokens you emit follow exactly the target model's own distribution. For greedy decoding the check is simple — accept the draft's token only if it equals the token the target would have picked — so the output is bit-for-bit identical to running the target alone. For sampling, a slightly cleverer probabilistic acceptance test (speculative sampling) gives the same guarantee in expectation.

That's what makes the technique a free lunch in a field that rarely offers one. You aren't approximating the big model with the small one; the small one only ever proposes, and the big one has the final say on every token. Wrong guesses cost a little wasted draft compute, never a wrong answer.

4 Acceptance rate

The speedup lives entirely in the acceptance rate — how often the draft's guess matches the target. The closer the draft mimics the target, and the more predictable the text, the longer the accepted runs. Boilerplate, code, and formulaic prose get devoured many tokens per round; genuinely novel or high-entropy text barely beats one.

This is why drafts are usually a smaller model from the same family, or even a few extra prediction heads bolted onto the target itself (Medusa, EAGLE) so the draft and target can't help but agree. A draft that's fast but rarely right is worse than useless — you pay its cost and accept almost nothing.

5 Where the speedup hides

Net speedup is a tug-of-war. Drafting more tokens per round (K) raises the ceiling on tokens you can accept — but every drafted token past the first rejection is wasted draft compute, and the draft isn't free. Push K too high and you spend more time drafting tokens that get thrown away than you save. There's an optimal draft length, and it moves with the acceptance rate and the draft's relative cost.

Roughly, the speedup is the average tokens accepted per round divided by the overhead of running the draft K times. A draft that's, say, ten times cheaper than the target, getting three or four tokens accepted per round, lands you a 2–3× throughput win in practice — which is why nearly every serious inference stack now ships it.

6 Reading the playground

Two real models run here — a well-trained target and a deliberately weaker draft, both character-level, both trained live in your browser. The draft-and-verify loop is genuine speculative sampling: the draft proposes, the target's acceptance test decides, and the emitted tokens come out distributed exactly as the target's own.