Spaces:
Sleeping
Sleeping
| <html lang="en"> | |
| <head> | |
| <meta charset="utf-8" /> | |
| <meta name="viewport" content="width=device-width, initial-scale=1" /> | |
| <title>1. Build a tokenizer — LM Playground</title> | |
| <link rel="stylesheet" href="/platform/platform.css" /> | |
| <link rel="stylesheet" href="/style.css" /> | |
| <script type="module" src="/platform/platform.js"></script> | |
| </head> | |
| <body data-chapter="tokenizer"> | |
| <header> | |
| <div class="header-inner"> | |
| <div> | |
| <h1>Tokenizer Playground</h1> | |
| <p class="sub"> | |
| An interactive guide to how language models read text — from characters to | |
| subwords, and why the choice of tokenizer genuinely changes what the model predicts. | |
| </p> | |
| </div> | |
| <span id="device" class="badge" title="inference device"></span> | |
| </div> | |
| <nav class="page-tabs" role="tablist"> | |
| <button class="page-tab active" data-tab="guide" role="tab" aria-selected="true">① Guide</button> | |
| <button class="page-tab" data-tab="playground" role="tab" aria-selected="false">② Playground</button> | |
| </nav> | |
| </header> | |
| <!-- ================================================================ GUIDE --> | |
| <section id="guide-tab" class="tab-panel active" aria-label="Guide"> | |
| <article class="guide"> | |
| <nav class="guide-toc"> | |
| <span class="toc-label">Contents</span> | |
| <a href="#ch1">1 · What is a token?</a> | |
| <a href="#ch2">2 · Three approaches</a> | |
| <a href="#ch3">3 · BPE algorithm</a> | |
| <a href="#ch4">4 · Vocab-size tradeoffs</a> | |
| <a href="#ch5">5 · The embedding constraint</a> | |
| <a href="#ch6">6 · Bits per character</a> | |
| <a href="#ch7">7 · Reading the playground</a> | |
| <a href="#ch8">8 · Real-world tokenizers</a> | |
| <a href="#ch9">9 · Hidden effects</a> | |
| </nav> | |
| <!-- 1 --> | |
| <section class="chapter" id="ch1"> | |
| <h2><span class="ch-num">1</span> What is a token?</h2> | |
| <p>A language model doesn't read text the way you do. It receives a <strong>sequence of integers</strong> — one integer per <em>token</em> — and generates the next integer. Tokenization is the step that converts your text into that sequence.</p> | |
| <figure class="fig"><svg viewBox="0 0 700 160" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif"><defs><marker id="artk" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0,0 L10,5 L0,10 z" fill="var(--muted)"/></marker></defs> | |
| <rect x="40" y="58" width="180" height="40" rx="8" fill="var(--accent)" opacity="0.18" stroke="var(--accent)"/> | |
| <text x="130" y="83" text-anchor="middle" fill="var(--text)" font-size="15" font-weight="700" font-family="ui-monospace,monospace">unhappiness</text> | |
| <line x1="222" y1="78" x2="262" y2="78" stroke="var(--border)" marker-end="url(#artk)"/> | |
| <rect x="270" y="58" width="80" height="40" rx="8" fill="var(--panel-2)" stroke="var(--accent-2)"/><text x="310" y="79" text-anchor="middle" fill="var(--accent-2)" font-size="13" font-weight="700" font-family="ui-monospace,monospace">un</text><text x="310" y="118" text-anchor="middle" fill="var(--muted)" font-size="10">412</text><rect x="362" y="58" width="80" height="40" rx="8" fill="var(--panel-2)" stroke="var(--accent-2)"/><text x="402" y="79" text-anchor="middle" fill="var(--accent-2)" font-size="13" font-weight="700" font-family="ui-monospace,monospace">happi</text><text x="402" y="118" text-anchor="middle" fill="var(--muted)" font-size="10">1875</text><rect x="454" y="58" width="80" height="40" rx="8" fill="var(--panel-2)" stroke="var(--accent-2)"/><text x="494" y="79" text-anchor="middle" fill="var(--accent-2)" font-size="13" font-weight="700" font-family="ui-monospace,monospace">ness</text><text x="494" y="118" text-anchor="middle" fill="var(--muted)" font-size="10">306</text> | |
| <text x="430" y="140" text-anchor="middle" fill="var(--muted)" font-size="10">one rare word → a few common subword pieces → integer IDs</text> | |
| </svg><figcaption>A tokenizer breaks text into the units a model reads. Rare words split into familiar subword pieces — “unhappiness” becomes un + happi + ness — and each piece maps to an integer ID. This keeps the vocabulary small while still covering words the model has never seen whole.</figcaption></figure> | |
| <p>Take the phrase <em>"To be, or not to be"</em>. Under a character tokenizer, each character maps to one integer:</p> | |
| <div class="token-demo"> | |
| <span class="tok c0">T</span><span class="tok c1">o</span><span class="tok c2">·</span><span class="tok c3">b</span><span class="tok c4">e</span><span class="tok c5">,</span><span class="tok c6">·</span><span class="tok c7">o</span><span class="tok c0">r</span><span class="tok c1">·</span><span class="tok c2">n</span><span class="tok c3">o</span><span class="tok c4">t</span><span class="tok c5">·</span><span class="tok c6">t</span><span class="tok c7">o</span><span class="tok c0">·</span><span class="tok c1">b</span><span class="tok c2">e</span> | |
| <span class="demo-label">19 character tokens</span> | |
| </div> | |
| <p>Under BPE-2048, common multi-character sequences are pre-merged into single tokens:</p> | |
| <div class="token-demo"> | |
| <span class="tok c0">To</span><span class="tok c1">·be</span><span class="tok c2">,</span><span class="tok c3">·or</span><span class="tok c4">·not</span><span class="tok c5">·to</span><span class="tok c6">·be</span> | |
| <span class="demo-label">7 BPE tokens — same text, 3× shorter sequence</span> | |
| </div> | |
| <div class="callout insight"> | |
| <strong>Key insight.</strong> The model's world is entirely tokens. It never "sees" the letter <em>T</em> — it sees an integer, say <code>27</code>. When we change the tokenizer we change that integer space, and with it the very atoms of meaning the model works with. | |
| </div> | |
| <button class="try-it" data-scheme="char" data-text="To be, or not to be">▶ See character tokens in the Playground</button> | |
| </section> | |
| <!-- 2 --> | |
| <section class="chapter" id="ch2"> | |
| <h2><span class="ch-num">2</span> Three approaches and their problems</h2> | |
| <h3>Character-level</h3> | |
| <p>Simplest possible: one token per character, vocabulary ~70 symbols. But sequences are long — a single page is ~2,000 tokens. Models have fixed context windows, so most tokens are burned on individual letters rather than meaning. The model must also learn that <em>T</em>, <em>h</em>, <em>e</em> together mean something — across many examples — before it can predict what follows "The".</p> | |
| <h3>Whole-word</h3> | |
| <p>Natural to humans. But English alone has 170,000+ dictionary words; add inflections (<em>play / plays / played / playing</em>), compounds, numbers, code identifiers, and a practical vocabulary easily exceeds a million entries. Any word not in the vocabulary becomes <code><UNK></code>, erasing its meaning. A model that sees "Schwarzenegger" as "unknown" can't distinguish it from "Dostoyevsky".</p> | |
| <h3>Subword — the compromise</h3> | |
| <p>Frequent words get their own token. Rare words are split into frequent <em>fragments</em> they share with common words:</p> | |
| <div class="token-demo"> | |
| <span class="tok c3">token</span><span class="tok c4">ization</span> | |
| <span class="demo-label">"tokenization" → 2 subword tokens</span> | |
| </div> | |
| <div class="token-demo"> | |
| <span class="tok c0">un</span><span class="tok c1">believ</span><span class="tok c2">able</span> | |
| <span class="demo-label">"unbelievable" → 3 subword tokens</span> | |
| </div> | |
| <p>Nothing is ever truly unknown. The vocabulary stays manageable (~32k–130k for modern models) while sequences remain tractable.</p> | |
| <div class="callout warn"> | |
| <strong>Common misconception.</strong> "Subword tokenization is just word-splitting." No — the fragments are <em>statistically frequent byte sequences</em> in a specific corpus. "token" and "ization" are frequent in English text; in a code corpus you'd learn "def", "return", and four-space indents instead. | |
| </div> | |
| <button class="try-it" data-scheme="bpe512" data-text="tokenization is not word splitting">▶ Compare BPE-512 vs Character on this phrase</button> | |
| </section> | |
| <!-- 3 --> | |
| <section class="chapter" id="ch3"> | |
| <h2><span class="ch-num">3</span> The BPE algorithm — step by step</h2> | |
| <p>Byte Pair Encoding (Sennrich et al., 2016) is the most widely used tokenization method in large language models.</p> | |
| <ol class="step-list"> | |
| <li><strong>Start with characters.</strong> Pre-tokenize the corpus into words. Represent each unique word as a sequence of characters, and count how often each word appears. | |
| <div class="token-demo small"> | |
| <span class="tok c0">l</span><span class="tok c1">o</span><span class="tok c2">w</span><span class="step-freq">×5</span> | |
| <span class="tok c3">l</span><span class="tok c4">o</span><span class="tok c5">w</span><span class="tok c6">e</span><span class="tok c7">r</span><span class="step-freq">×2</span> | |
| <span class="tok c0">n</span><span class="tok c1">e</span><span class="tok c2">w</span><span class="tok c3">e</span><span class="tok c4">s</span><span class="tok c5">t</span><span class="step-freq">×6</span> | |
| </div> | |
| </li> | |
| <li><strong>Count all adjacent pairs</strong> across all words, weighted by word frequency. <code>l</code>+<code>o</code> appears 7 times (5 from "low" + 2 from "lower"); <code>e</code>+<code>s</code> appears 6 times.</li> | |
| <li><strong>Merge the most frequent pair.</strong> Say <code>l</code>+<code>o</code> wins. Create token <code>lo</code> and replace every occurrence: | |
| <div class="token-demo small"> | |
| <span class="tok c5">lo</span><span class="tok c2">w</span><span class="step-freq">×5</span> | |
| <span class="tok c5">lo</span><span class="tok c4">w</span><span class="tok c6">e</span><span class="tok c7">r</span><span class="step-freq">×2</span> | |
| <span class="tok c0">n</span><span class="tok c1">e</span><span class="tok c2">w</span><span class="tok c3">e</span><span class="tok c4">s</span><span class="tok c5">t</span><span class="step-freq">×6</span> | |
| </div> | |
| </li> | |
| <li><strong>Repeat</strong> until you reach the target vocabulary size. Each merge records which pair, how often it occurred, and what it produced. That log is the entire learned tokenizer.</li> | |
| <li><strong>To encode new text:</strong> apply the merges in learned order. Even words unseen during training get decomposed by the same sequence of merges.</li> | |
| </ol> | |
| <div class="callout insight"> | |
| <strong>Why corpus matters.</strong> Shakespeare learns <code>th</code>, <code>ou</code>, <code>ght</code> early because <em>thou</em>, <em>thought</em>, <em>hath</em> are common. Wikipedia learns <code>in</code>, <code>ion</code>, <code>ation</code> early. Same algorithm, different corpus → different tokens. Switch corpora in the Playground and watch the merge table change. | |
| </div> | |
| <button class="try-it" data-scheme="bpe2048" data-word="thoughts">▶ Watch "thoughts" collapse in the Playground (BPE-2048)</button> | |
| </section> | |
| <!-- 4 --> | |
| <section class="chapter" id="ch4"> | |
| <h2><span class="ch-num">4</span> Vocabulary size tradeoffs</h2> | |
| <p>The target vocabulary size is the key hyperparameter. It pulls in opposite directions:</p> | |
| <div class="tradeoff-table"> | |
| <div class="tradeoff-head"> | |
| <div></div> | |
| <div class="tl-small">Small vocab<br><span class="tl-eg">~66 char</span></div> | |
| <div class="tl-mid">Medium<br><span class="tl-eg">~512 BPE</span></div> | |
| <div class="tl-large">Large vocab<br><span class="tl-eg">~2048 BPE</span></div> | |
| </div> | |
| <div class="tradeoff-row"><div class="tl-label">Sequence length</div><div class="tl-bad">Very long</div><div class="tl-mid2">Medium</div><div class="tl-good">Short</div></div> | |
| <div class="tradeoff-row"><div class="tl-label">Context coverage</div><div class="tl-bad">Low</div><div class="tl-mid2">Moderate</div><div class="tl-good">High</div></div> | |
| <div class="tradeoff-row"><div class="tl-label">Training signal / token</div><div class="tl-good">Abundant</div><div class="tl-mid2">Good</div><div class="tl-bad">Thinner</div></div> | |
| <div class="tradeoff-row"><div class="tl-label">OOV risk</div><div class="tl-good">None</div><div class="tl-good">None</div><div class="tl-warn">Low but present</div></div> | |
| <div class="tradeoff-row"><div class="tl-label">Embedding table size</div><div class="tl-good">Tiny</div><div class="tl-mid2">Small</div><div class="tl-bad">Larger</div></div> | |
| </div> | |
| <p>Real models: BERT ~30k tokens, GPT-2 ~50k, LLaMA 3 128k. Bigger is not always better — with a small corpus a large vocabulary means many rare tokens trained on almost no data.</p> | |
| <div class="callout insight"> | |
| <strong>In the nano-GPTs here.</strong> Even at nano scale (0.8M params, 1–10 MB text), the bits-per-character comparison shows the larger tokenizer winning: BPE-2048 beats BPE-512, which beats Character. The gain from better tokenization is real even with tiny models. | |
| </div> | |
| <button class="try-it" data-scheme="bpe2048" data-text="To be, or not to be, that is the question">▶ Compare all three sequence lengths in the Playground</button> | |
| </section> | |
| <!-- 5 --> | |
| <section class="chapter" id="ch5"> | |
| <h2><span class="ch-num">5</span> The embedding table — why tokenizers and models are inseparable</h2> | |
| <h3>What an embedding actually is</h3> | |
| <p> | |
| A language model never operates on text or token strings. It operates on | |
| <strong>vectors</strong>. The very first thing it does with a sequence of IDs | |
| is look each one up in an <em>embedding table</em> — a matrix of shape | |
| <code>[vocab_size × d_model]</code> where <code>d_model</code> is the hidden | |
| dimension (128 in these nano-GPTs). Each row is a point in that | |
| 128-dimensional space, one row per token in the vocabulary. | |
| </p> | |
| <!-- lookup diagram --> | |
| <div class="embed-lookup-diagram"> | |
| <div class="eld-col"> | |
| <div class="eld-label">token IDs in</div> | |
| <div class="eld-ids"> | |
| <span class="tok c2">·To</span> | |
| <span class="tok c3">·be</span> | |
| <span class="tok c5">·or</span> | |
| <span class="tok c6">·not</span> | |
| </div> | |
| </div> | |
| <div class="eld-arrow">→</div> | |
| <div class="eld-col eld-table-col"> | |
| <div class="eld-label">embedding table <span class="embed-dim">[vocab × 128]</span></div> | |
| <div class="eld-table"> | |
| <div class="eld-row faded">row 0 · <div class="eld-bar-row"><div class="eld-bar" style="width:30%;background:#555"></div><div class="eld-bar" style="width:45%;background:#555"></div><div class="eld-bar" style="width:20%;background:#555"></div></div></div> | |
| <div class="eld-row active">row 46 · <div class="eld-bar-row"><div class="eld-bar" style="width:72%;background:var(--tok2b)"></div><div class="eld-bar" style="width:38%;background:var(--tok2b)"></div><div class="eld-bar" style="width:61%;background:var(--tok2b)"></div></div></div> | |
| <div class="eld-row active">row 115 · <div class="eld-bar-row"><div class="eld-bar" style="width:45%;background:var(--tok3b)"></div><div class="eld-bar" style="width:80%;background:var(--tok3b)"></div><div class="eld-bar" style="width:30%;background:var(--tok3b)"></div></div></div> | |
| <div class="eld-row faded">row … · <div class="eld-bar-row"><div class="eld-bar" style="width:40%;background:#555"></div><div class="eld-bar" style="width:25%;background:#555"></div><div class="eld-bar" style="width:55%;background:#555"></div></div></div> | |
| <div class="eld-row active">row 331 · <div class="eld-bar-row"><div class="eld-bar" style="width:20%;background:var(--tok5b)"></div><div class="eld-bar" style="width:65%;background:var(--tok5b)"></div><div class="eld-bar" style="width:88%;background:var(--tok5b)"></div></div></div> | |
| <div class="eld-row active">row 131 · <div class="eld-bar-row"><div class="eld-bar" style="width:55%;background:var(--tok6b)"></div><div class="eld-bar" style="width:42%;background:var(--tok6b)"></div><div class="eld-bar" style="width:70%;background:var(--tok6b)"></div></div></div> | |
| <div class="eld-row faded">row … · <div class="eld-bar-row"><div class="eld-bar" style="width:60%;background:#555"></div><div class="eld-bar" style="width:15%;background:#555"></div><div class="eld-bar" style="width:40%;background:#555"></div></div></div> | |
| </div> | |
| </div> | |
| <div class="eld-arrow">→</div> | |
| <div class="eld-col"> | |
| <div class="eld-label">4 × 128-dim vectors</div> | |
| <div class="eld-vecs"> | |
| <div class="eld-vec c2"><div></div><div></div><div></div><div></div><div></div><div></div></div> | |
| <div class="eld-vec c3"><div></div><div></div><div></div><div></div><div></div><div></div></div> | |
| <div class="eld-vec c5"><div></div><div></div><div></div><div></div><div></div><div></div></div> | |
| <div class="eld-vec c6"><div></div><div></div><div></div><div></div><div></div><div></div></div> | |
| </div> | |
| </div> | |
| <div class="eld-arrow">→</div> | |
| <div class="eld-col"> | |
| <div class="eld-label">transformer layers</div> | |
| <div class="eld-transformer"> | |
| <div class="eld-block">Attention</div> | |
| <div class="eld-block">MLP</div> | |
| <div class="eld-block">Attention</div> | |
| <div class="eld-block">MLP</div> | |
| <div class="eld-block eld-head">→ next token</div> | |
| </div> | |
| </div> | |
| </div> | |
| <h3>What these vectors learn</h3> | |
| <p> | |
| The rows are not hand-crafted. They start as random noise and are updated by | |
| gradient descent on every training step. The signal that shapes them: tokens | |
| that <em>appear in similar contexts</em> receive similar gradient updates and | |
| gradually move closer together in the 128-dimensional space. | |
| </p> | |
| <p> | |
| After training the Shakespeare BPE-2048 model, we can measure this directly. | |
| The nearest neighbour of <span class="inline-tok c6">·night</span> | |
| in embedding space is <span class="inline-tok c5">·morrow</span> | |
| (cosine similarity 0.81), followed by | |
| <span class="inline-tok c4">·day</span> (0.70). The model has never been | |
| told that "night" and "morrow" are related — it inferred it from thousands of | |
| lines of Shakespeare where the two words appear in overlapping contexts | |
| (<em>"good morrow"</em>, <em>"the night before"</em>, time-of-day | |
| constructions). | |
| </p> | |
| <!-- two-column neighbor comparison --> | |
| <div class="neighbor-compare"> | |
| <div class="nc-col"> | |
| <div class="nc-head">Character model — <em>'a'</em> (id 40)</div> | |
| <div class="nc-body"> | |
| <p class="nc-note">Char embeddings cluster by <strong>character class</strong>, not word meaning. Every dimension of the model's input represents a single letter.</p> | |
| <div class="nc-neighbors"> | |
| <div class="nc-row"><span class="inline-tok c1">i</span><div class="nc-bar-wrap"><div class="nc-bar" style="width:62%"></div></div><span class="nc-sim">0.71</span></div> | |
| <div class="nc-row"><span class="inline-tok c2">e</span><div class="nc-bar-wrap"><div class="nc-bar" style="width:58%"></div></div><span class="nc-sim">0.69</span></div> | |
| <div class="nc-row"><span class="inline-tok c3">o</span><div class="nc-bar-wrap"><div class="nc-bar" style="width:55%"></div></div><span class="nc-sim">0.67</span></div> | |
| <div class="nc-row"><span class="inline-tok c4">u</span><div class="nc-bar-wrap"><div class="nc-bar" style="width:52%"></div></div><span class="nc-sim">0.64</span></div> | |
| <div class="nc-row"><span class="inline-tok c5">y</span><div class="nc-bar-wrap"><div class="nc-bar" style="width:48%"></div></div><span class="nc-sim">0.61</span></div> | |
| </div> | |
| <p class="nc-verdict bad">All vowels. Structurally coherent, semantically useless for language.</p> | |
| </div> | |
| </div> | |
| <div class="nc-col"> | |
| <div class="nc-head">BPE-2048 model — <em>'·night'</em> (id 1671)</div> | |
| <div class="nc-body"> | |
| <p class="nc-note">BPE embeddings cluster by <strong>contextual meaning</strong>. Tokens that appear in similar situations end up as neighbours.</p> | |
| <div class="nc-neighbors"> | |
| <div class="nc-row"><span class="inline-tok c6">·morrow</span><div class="nc-bar-wrap"><div class="nc-bar" style="width:100%"></div></div><span class="nc-sim">0.81</span></div> | |
| <div class="nc-row"><span class="inline-tok c5">·day</span><div class="nc-bar-wrap"><div class="nc-bar" style="width:86%"></div></div><span class="nc-sim">0.70</span></div> | |
| <div class="nc-row"><span class="inline-tok c4">·morn</span><div class="nc-bar-wrap"><div class="nc-bar" style="width:55%"></div></div><span class="nc-sim">0.48</span></div> | |
| <div class="nc-row"><span class="inline-tok c3">·eve</span><div class="nc-bar-wrap"><div class="nc-bar" style="width:50%"></div></div><span class="nc-sim">0.45</span></div> | |
| <div class="nc-row"><span class="inline-tok c2">·dark</span><div class="nc-bar-wrap"><div class="nc-bar" style="width:46%"></div></div><span class="nc-sim">0.44</span></div> | |
| </div> | |
| <p class="nc-verdict good">Times of day + associated concepts. No linguistic labelling needed — pure co-occurrence.</p> | |
| </div> | |
| </div> | |
| </div> | |
| <div class="callout insight"> | |
| <strong>The granularity determines what can be learned.</strong> | |
| A character model's embeddings can only capture letter-level patterns — vowel | |
| vs. consonant, uppercase vs. lowercase. A BPE model's embeddings can represent | |
| word-level semantics because BPE tokens are large enough to carry meaning. | |
| You cannot retrofit word-level meaning onto character embeddings by training longer; | |
| the granularity is a hard ceiling. | |
| </div> | |
| <button class="try-it" data-scheme="bpe2048" data-action="neighbors"> | |
| ▶ Explore embedding neighbors live in the Playground | |
| </button> | |
| <h3>Weight tying — the output is the input</h3> | |
| <p> | |
| There is an additional coupling that makes the embedding table even more deeply | |
| entangled with the model. The final layer of the transformer produces a logit | |
| for every token in the vocabulary. In standard language model design | |
| (<em>weight tying</em>, Press & Wolf 2017), this output projection | |
| <strong>shares weights with the embedding table</strong>. Row <em>i</em> of | |
| the embedding serves double duty: it is the input representation for token | |
| <em>i</em> and also the direction the model "votes for" when it wants to | |
| predict token <em>i</em>. | |
| </p> | |
| <p> | |
| The consequence: every training step updates the embedding not just through | |
| the input path but also through the output path. The embedding table is shaped | |
| by the full model — by every attention head, every MLP layer, every layer norm. | |
| It is not an isolated lookup; it is the model's complete vocabulary of meaning. | |
| </p> | |
| <h3>The catastrophe: why IDs are arbitrary</h3> | |
| <p> | |
| Tokenizer A and Tokenizer B produce completely different integer sequences for | |
| the same text. There is no shared coordinate system. When the char model | |
| tokenises "To be, or not to be", the first ID is <code>33</code> | |
| (the char <code>T</code>). When the BPE-512 model tokenises the same text, the | |
| first ID is <code>207</code> (the subword <code>·To</code>). These integers have | |
| no relationship to each other. | |
| </p> | |
| <!-- what-if-you-swap diagram --> | |
| <div class="swap-diagram"> | |
| <div class="sd-col sd-correct"> | |
| <div class="sd-badge ok">✓ Correct</div> | |
| <div class="sd-step">text → <span class="tok c2">·To</span><span class="tok c3">·be</span><span class="tok c4">·or</span>…</div> | |
| <div class="sd-step">IDs → <code>[207, 115, 7 …]</code></div> | |
| <div class="sd-step">lookup in <strong>BPE model</strong>'s embedding table</div> | |
| <div class="sd-step">row 207 = <em>·To</em>, row 115 = <em>·be</em> ✓</div> | |
| <div class="sd-preds"> | |
| <div class="sd-pred-label">next-token predictions:</div> | |
| <div class="sd-pred-row"><span>·king</span><span class="sd-pct good">4.5%</span></div> | |
| <div class="sd-pred-row"><span>·world</span><span class="sd-pct good">3.1%</span></div> | |
| <div class="sd-pred-row"><span>·duke</span><span class="sd-pct good">2.1%</span></div> | |
| </div> | |
| </div> | |
| <div class="sd-col sd-wrong"> | |
| <div class="sd-badge bad">✗ Swapped</div> | |
| <div class="sd-step">text → <span class="tok c0">T</span><span class="tok c1">o</span><span class="tok c2">·</span><span class="tok c3">b</span>…</div> | |
| <div class="sd-step">IDs → <code>[33, 54, 2, 41 …]</code></div> | |
| <div class="sd-step">lookup in <strong>BPE model</strong>'s embedding table</div> | |
| <div class="sd-step">row 33 = <em>·wh</em>, row 54 = <em>·s</em> ✗</div> | |
| <div class="sd-preds"> | |
| <div class="sd-pred-label">next-token predictions:</div> | |
| <div class="sd-pred-row"><span>or</span><span class="sd-pct bad">6.3%</span></div> | |
| <div class="sd-pred-row"><span>,</span><span class="sd-pct bad">4.2%</span></div> | |
| <div class="sd-pred-row"><span>em</span><span class="sd-pct bad">3.4%</span></div> | |
| </div> | |
| </div> | |
| </div> | |
| <p> | |
| The model isn't "confused" — it is answering a completely different question. | |
| Char IDs <code>[33, 54, 2 …]</code> fed into the BPE model trigger row lookups | |
| for whatever BPE tokens happen to live at those positions. The model faithfully | |
| continues that (nonsensical) sequence. Changing the tokenizer means building | |
| a new model from scratch, because the entire weight matrix is calibrated to | |
| one specific mapping between integers and meanings. | |
| </p> | |
| <div class="callout insight"> | |
| <strong>This is why we train one nano-GPT per tokenizer.</strong> | |
| The comparison in the Playground is honest: when you switch from Character to | |
| BPE-512, you are not re-running the same model with different pre-processing. | |
| You are switching to an entirely different model that has never seen the other | |
| tokenizer's IDs. | |
| </div> | |
| <div class="callout warn"> | |
| <strong>Practical implication.</strong> | |
| When a new tokenizer is released for a model family (e.g., LLaMA 2 → LLaMA 3 | |
| changed from 32k to 128k tokens), the model cannot simply be adapted — it must | |
| be retrained from scratch. The tokenizer is not a plug-in component; it is | |
| fused into the model's identity. | |
| </div> | |
| <button class="try-it" data-scheme="bpe2048" data-action="swap"> | |
| ▶ Run the swap experiment live in the Playground | |
| </button> | |
| </section> | |
| <!-- 6 --> | |
| <section class="chapter" id="ch6"> | |
| <h2><span class="ch-num">6</span> Bits per character — the fair metric</h2> | |
| <p>Cross-entropy loss seems like an obvious comparison metric, but it traps you: a BPE-2048 model almost always has <em>higher</em> per-token loss than a character model on the same text — not because it is worse, but because each of its tokens carries more information. A token like "tion" is harder to predict than the next character.</p> | |
| <div class="formula-box"> | |
| <div class="formula"> | |
| BPC = H(token) × | |
| <span class="frac"><span class="frac-top">tokens</span><span class="frac-bot">chars</span></span> | |
| × | |
| <span class="frac"><span class="frac-top">1</span><span class="frac-bot">ln 2</span></span> | |
| </div> | |
| <div class="formula-note"> | |
| <em>H(token)</em> = mean per-token cross-entropy (nats). The <em>tokens/chars</em> ratio converts to per-character. Dividing by ln 2 converts nats to bits. | |
| </div> | |
| </div> | |
| <p>BPC asks: <em>"how many bits does the model need to encode each character?"</em> Lower is better and the number is comparable across tokenizers. In this playground, BPC consistently improves Character → BPE-512 → BPE-2048 on both corpora, even though per-token loss moves in the opposite direction.</p> | |
| <div class="callout insight"> | |
| <strong>Look at bits/char, not val loss.</strong> The comparison cards in the Playground show both. The per-token loss numbers are not comparable across schemes — only BPC is. | |
| </div> | |
| </section> | |
| <!-- 7 --> | |
| <section class="chapter" id="ch7"> | |
| <h2><span class="ch-num">7</span> Reading the playground</h2> | |
| <p>Switch to the Playground tab. Here is what each panel shows and what to notice:</p> | |
| <div class="panel-guide-item"> | |
| <div class="pgi-label">Your text</div> | |
| <p>The analysis updates ~150 ms after you stop typing. Try a word not in everyday English — a proper noun or chemical name — to see BPE handle it without an unknown fallback.</p> | |
| </div> | |
| <div class="panel-guide-item"> | |
| <div class="pgi-label">Tokens</div> | |
| <p>Each coloured chip is one token. Hover to see its integer ID and position. The count badge pulses on every change — watch it drop as you switch Character → BPE-512 → BPE-2048. Click "show token IDs" to reveal the raw integer sequence the model actually receives.</p> | |
| </div> | |
| <div class="panel-guide-item"> | |
| <div class="pgi-label">Next token — what this model predicts</div> | |
| <p>Top-15 most probable next tokens from the nano-GPT for the selected (corpus, scheme) pair. Toggle the scheme and the bars genuinely change because a different model answers. Character predicts individual letters; BPE-512 predicts word-starts; BPE-2048 predicts whole common words.</p> | |
| </div> | |
| <div class="panel-guide-item"> | |
| <div class="pgi-label">Same text through every scheme</div> | |
| <p>Three cards showing token count and quality metrics for every scheme simultaneously. Compare <strong>tok/char</strong> (compression ratio) and <strong>bits/char</strong> (model quality) across columns. The highlighted card is the active scheme.</p> | |
| </div> | |
| <div class="panel-guide-item"> | |
| <div class="pgi-label">Watch BPE merge</div> | |
| <p>Only active for BPE schemes. Type any word and drag the slider left — the word starts as individual characters and collapses step by step as merges are applied in the order they were learned from the corpus. The full merge list below is the complete learned vocabulary.</p> | |
| </div> | |
| <button class="try-it" data-scheme="bpe512" data-text="Shakespeare wrote magnificent plays">▶ Open the Playground and try it</button> | |
| </section> | |
| <!-- 8 --> | |
| <section class="chapter" id="ch8"> | |
| <h2><span class="ch-num">8</span> Real-world tokenizers</h2> | |
| <div class="callout insight"> | |
| <strong>The field has converged.</strong> | |
| Virtually all modern frontier models use variants of Byte-Pair Encoding operating at the byte level (Byte-level BPE), using specific optimisations to address historical flaws. | |
| <a class="cite-link" href="https://arxiv.org/html/2402.14903v1#:~:text=Though%20many%20techniques%20have%20been,the%20case%20of%20LLMs%2C%20text" target="_blank" rel="noopener">Навarro et al., 2024 ↗</a> | |
| </div> | |
| <p>The playground uses simple from-scratch BPE to keep the algorithm visible. Production models use more sophisticated variants:</p> | |
| <div class="real-world-grid"> | |
| <div class="rw-card"> | |
| <div class="rw-name">Byte-level BPE</div> | |
| <div class="rw-used">GPT-2, GPT-3, GPT-4, LLaMA 3</div> | |
| <p>Base vocabulary = all 256 byte values. No unknown tokens ever — every string is a sequence of bytes. Trade-off: a single Chinese character may take 2–3 byte tokens.</p> | |
| </div> | |
| <div class="rw-card"> | |
| <div class="rw-name">WordPiece</div> | |
| <div class="rw-used">BERT, DistilBERT, ELECTRA</div> | |
| <p>Merges chosen to maximise <em>likelihood</em> of the training corpus rather than pair frequency. Continuation pieces are prefixed with <code>##</code> ("tokenization" → ["token", "##ization"]).</p> | |
| </div> | |
| <div class="rw-card"> | |
| <div class="rw-name">SentencePiece / Unigram LM</div> | |
| <div class="rw-used">T5, LLaMA 1&2, Mistral</div> | |
| <p>Starts with a large vocabulary and prunes it. Language-agnostic — treats spaces as ordinary characters. Works well for languages without clear word boundaries.</p> | |
| </div> | |
| <div class="rw-card"> | |
| <div class="rw-name">tiktoken</div> | |
| <div class="rw-used">GPT-3.5, GPT-4, o1, o3</div> | |
| <p>OpenAI's open-source, Rust-backed implementation of byte-level BPE. "cl100k_base" (GPT-4) has 100,256 tokens. "o200k_base" (GPT-4o) has 200,019.</p> | |
| </div> | |
| </div> | |
| </section> | |
| <!-- 9 --> | |
| <section class="chapter" id="ch9"> | |
| <h2><span class="ch-num">9</span> Hidden effects of tokenization</h2> | |
| <h3>The tokenization tax on non-English languages</h3> | |
| <p>If a BPE vocabulary is trained on predominantly English text, the same token budget buys fewer words in Arabic, Turkish, or Korean — sometimes 3–5× fewer. Non-English speakers get shorter effective context windows and pay more per token in API costs.</p> | |
| <h3>Prompt sensitivity</h3> | |
| <p>Small surface changes humans consider equivalent — capitalisation, a leading space, a trailing newline — can shift token boundaries and change model behaviour. These two strings produce different first tokens:</p> | |
| <div class="token-demo"> | |
| <span class="tok c0">·The</span><span class="tok c1">·king</span><span class="tok c2">·lives</span> | |
| <span class="demo-label">" The king lives" — space before T merges with it</span> | |
| </div> | |
| <div class="token-demo"> | |
| <span class="tok c3">The</span><span class="tok c4">·king</span><span class="tok c5">·lives</span> | |
| <span class="demo-label">"The king lives" — T starts a new token</span> | |
| </div> | |
| <h3>Arithmetic and reasoning</h3> | |
| <p>Numbers are often fragmented unpredictably. <em>1,000,000</em> might tokenize as [<code>1</code>, <code>,</code>, <code>000</code>, <code>,</code>, <code>000</code>] or as a single token. The model reasons over number fragments, not numbers — making digit-level arithmetic harder than it looks.</p> | |
| <h3>Token budget and API cost</h3> | |
| <p>LLM API providers charge per token. A poorly tokenized language can cost 3–4× more per word. Checking actual token counts before sending long prompts is worth the habit.</p> | |
| <div class="callout warn"> | |
| <strong>Takeaway.</strong> Tokenization is not a neutral pre-processing step. It shapes what the model can learn, how efficiently it uses its context window, who gets penalised by API costs, and whether arithmetic works. It is a first-class design decision. | |
| </div> | |
| <div class="guide-end"> | |
| <p>You now have the full picture. Head to the Playground and put it to work.</p> | |
| <button class="try-it large" data-scheme="bpe512">▶ Open the Playground</button> | |
| </div> | |
| </section> | |
| </article> | |
| </section> | |
| <!-- ============================================================ PLAYGROUND --> | |
| <section id="playground-tab" class="tab-panel" aria-label="Playground"> | |
| <section class="controls"> | |
| <div class="control"> | |
| <label for="corpus">Corpus</label> | |
| <select id="corpus"></select> | |
| <div id="corpus-desc" class="hint"></div> | |
| </div> | |
| <div class="control"> | |
| <label>Tokenizer</label> | |
| <div id="schemes" class="segmented" role="tablist"></div> | |
| </div> | |
| </section> | |
| <main> | |
| <section class="panel"> | |
| <label class="panel-label" for="text">Your text</label> | |
| <textarea id="text" rows="3" spellcheck="false" placeholder="Type here…"></textarea> | |
| </section> | |
| <section class="panel"> | |
| <div class="panel-head"> | |
| <span class="panel-label">Tokens <button class="help-btn" data-guide="ch7" title="What is this?">?</button></span> | |
| <span id="length" class="length-badge">0 tokens</span> | |
| </div> | |
| <div id="tokens" class="tokens"></div> | |
| <div class="ids-wrap"> | |
| <button id="toggle-ids" class="link" type="button">show token IDs ▸</button> | |
| <div id="ids" class="ids hidden"></div> | |
| </div> | |
| </section> | |
| <!-- embedding neighbors panel --> | |
| <section class="panel" id="neighbors-panel"> | |
| <div class="panel-head"> | |
| <span class="panel-label">Embedding neighbors <button class="help-btn" data-guide="ch5" title="What do embeddings learn?">?</button></span> | |
| <span class="panel-note" id="neighbors-note">click any token chip above to explore</span> | |
| </div> | |
| <div id="neighbors-content" class="neighbors-content"> | |
| <span class="placeholder">Click a token chip above to see which tokens this model has learned are nearest to it in embedding space.</span> | |
| </div> | |
| </section> | |
| <section class="panel"> | |
| <div class="panel-head"> | |
| <span class="panel-label">Next token — what this model predicts <button class="help-btn" data-guide="ch5" title="Why does this change per tokenizer?">?</button></span> | |
| <span id="predict-note" class="panel-note"></span> | |
| </div> | |
| <div id="bars" class="bars"></div> | |
| </section> | |
| <!-- swap experiment panel --> | |
| <section class="panel" id="swap-panel"> | |
| <div class="panel-head"> | |
| <span class="panel-label">Swap experiment — feed wrong model <button class="help-btn" data-guide="ch5" title="The embedding constraint">?</button></span> | |
| <div class="swap-controls"> | |
| <span class="panel-note">feed <b id="swap-src-label">current</b> tokenizer's IDs into:</span> | |
| <select id="swap-target" class="swap-select"></select> | |
| </div> | |
| </div> | |
| <div class="swap-columns"> | |
| <div class="swap-col"> | |
| <div class="swap-col-head ok-head">✓ Correct — own model</div> | |
| <div id="swap-correct-bars" class="bars swap-bars"></div> | |
| </div> | |
| <div class="swap-divider"></div> | |
| <div class="swap-col"> | |
| <div class="swap-col-head bad-head">✗ Cross-fed — wrong model</div> | |
| <div id="swap-cross-bars" class="bars swap-bars"></div> | |
| <div id="swap-oob" class="swap-oob"></div> | |
| </div> | |
| </div> | |
| </section> | |
| <section class="panel"> | |
| <div class="panel-head"> | |
| <span class="panel-label">Same text through every scheme <button class="help-btn" data-guide="ch6" title="What is bits/char?">?</button></span> | |
| </div> | |
| <div id="compare" class="compare"></div> | |
| <p class="hint"> | |
| <b>length</b> = tokens for the text above (coarser tokenizer → fewer tokens). | |
| <b>bits/char</b> is the fair cross-tokenizer quality score (lower = better): | |
| per-token loss isn't comparable because the vocab sizes differ. | |
| </p> | |
| </section> | |
| <section class="panel merge-panel" id="merge-panel"> | |
| <div class="panel-head"> | |
| <span class="panel-label">Watch BPE merge <button class="help-btn" data-guide="ch3" title="How does BPE work?">?</button></span> | |
| <span class="panel-note">step through the merges this tokenizer learned from the corpus</span> | |
| </div> | |
| <div class="merge-controls"> | |
| <label for="merge-word">word</label> | |
| <input id="merge-word" type="text" spellcheck="false" /> | |
| <input id="merge-slider" type="range" min="0" max="0" value="0" /> | |
| <span id="merge-step-label" class="mono"></span> | |
| </div> | |
| <div id="merge-tokens" class="tokens big"></div> | |
| <div id="merge-applied" class="merge-applied"></div> | |
| <details class="merge-table-details"> | |
| <summary>full merge list</summary> | |
| <div class="merge-table-wrap"> | |
| <table class="merge-table"> | |
| <thead><tr><th>#</th><th>pair</th><th></th><th>token</th><th>freq</th></tr></thead> | |
| <tbody id="merge-table"></tbody> | |
| </table> | |
| </div> | |
| </details> | |
| </section> | |
| </main> | |
| <footer> | |
| <span>A from-scratch BPE + nano-GPT, one model per tokenizer. Live inference runs on a local server.</span> | |
| </footer> | |
| </section> | |
| <script src="/api-base.js"></script> | |
| <script src="/app.js"></script> | |
| </body> | |
| </html> | |