Build sampling

A language model never outputs a word. It outputs a probability for every word, and someone has to choose. That choice — the decoding strategy — is where a model gets its voice: cautious and repetitive, or loose and surprising, from the very same weights. Here you turn the knobs on a real model's live distribution and watch the candidates widen, narrow, and loop.

1 From distribution to token

Run a prompt through the model and the final layer hands you a logit for every token in the vocabulary — tens of thousands of raw scores. A softmax turns them into a probability distribution: token the at 12%, a at 8%, quantum at 0.001%, and so on across the whole vocabulary. The model's job ends here. It has stated its beliefs; it has not picked a word.

model probabilities temp · top-k · top-p reshaped → sample flat = creative / risky peaked = safe / repetitive
Sampling turns the model's probability distribution into an actual next token. Temperature, top-k, and top-p reshape that distribution before you draw from it — flatten it for variety, sharpen it for focus. Same model, very different text depending on how you sample.

Decoding is the separate step that turns that distribution into an actual token, and then repeats — append the token, run again, sample again. The same frozen weights can sound like a careful encyclopedia or a free-associating poet depending entirely on how you make that pick.

The model proposes a distribution; decoding disposes a token. Every behaviour you associate with a model's "style" — its repetitiveness, its creativity, its determinism — is at least half a decoding choice, made after the weights have had their say.

2 Why not just take the most likely?

The obvious strategy is greedy: at each step, take the single highest-probability token. It's deterministic and it sounds safe, but it has two real problems.

First, the most likely token at every step does not add up to the most likely sentence — locally greedy choices paint you into globally bland corners. Second, greedy decoding loops. Once the model writes "the best way to the best way to the best way…", each repetition makes the next repetition look even more probable, and it can't escape. Real text has variety that always-take-the-top can't produce.

Greedy gets stuck. A high-probability loop is a trap with no exit under argmax: the very repetition that's gone wrong is what the model now scores highest. You need a pinch of randomness, or an explicit penalty, to break out — both of which this chapter builds.

3 Temperature

Temperature is the master volume on randomness. Before the softmax, divide every logit by a number T:

p = softmax( logits / T )
T < 1 sharpens toward the top token; T > 1 flattens toward uniform; T → 0 is greedy.

Low temperature (say 0.7) makes the peaks taller and the tails shorter — the model plays it safe and stays on-topic. High temperature (1.3+) levels the distribution, handing unlikely tokens a real chance and producing surprising, sometimes incoherent text. It's one knob that slides smoothly from rigid to unhinged, and most of a model's apparent personality lives on this dial.

4 Top-k and top-p

Temperature reshapes the whole distribution but never closes the door on the junk in the far tail — at high T, a genuinely nonsensical token can still slip through. Truncation sampling slams that door by throwing the tail away before sampling.

Top-k keeps only the k most probable tokens and zeroes the rest, then renormalizes and samples. Simple, but rigid: k = 40 is too few when the model is genuinely unsure across hundreds of plausible tokens, and too many when it's confident about two. Top-p (nucleus sampling) fixes that by keeping the smallest set of tokens whose probabilities sum to p — say 0.9. When the model is certain, that nucleus is two or three tokens; when it's unsure, it's dozens. The candidate set breathes with the model's confidence.

Top-p adapts; top-k doesn't. A fixed cutoff count can't tell a confident step from an uncertain one. Nucleus sampling sizes the candidate pool to the distribution in front of it, which is why top-p ≈ 0.9 with a modest temperature is the workhorse default.

5 Taming repetition

Even with sampling, models drift into loops, especially at low temperature. The direct fix is a repetition penalty: before sampling, divide the logits of tokens that have appeared recently, making the model less keen to say them again. Relatives include presence and frequency penalties (used by the OpenAI API) and a no-repeat n-gram rule that simply forbids repeating any n-gram verbatim.

These are blunt tools — push the penalty too hard and the model contorts itself to avoid common, necessary words like the, and the text turns stilted. As with every knob here, the goal isn't an extreme setting; it's the balance that reads as natural.

6 Reading the playground

The distribution and the text here come from a real model — a small character-level network trained live in your browser. It's not a large LM, so it thinks in letters, not ideas; but the decoding mechanics are exactly the ones production systems use.

▮

The live next-token distribution. Temperature reshapes the bars; top-k and top-p grey out the tail. Kept tokens are the only ones that can be sampled.

✎

Generated text under your current settings. Flip to greedy and watch it loop; add a repetition penalty and watch the loop break.

◔

Presets — Greedy, Balanced, Creative — to feel the span from rigid to wild in one click.

The reading is the setup. The playground is the point.

Decoding knobs real char-model · trained live
Temperature: 0.80
Top-k: off
Top-p: 0.90
Repetition penalty: 1.0
Step 1 · the next-token distribution after a given character — kept vs cut
context (last char):
—candidate tokens
—prob. covered
—top token prob.
kept (can be sampled) cut by top-k / top-p
Step 2 · generate these settings, rolled out
…
—diversity (unique 4-grams)
—longest repeat

Greedy or very low temperature collapses diversity and the longest-repeat shoots up — the model loops. A repetition penalty or a little temperature pulls it back.