1 What is a token?
A language model doesn't read text the way you do. It receives a sequence of integers — one integer per token — and generates the next integer. Tokenization is the step that converts your text into that sequence.
Take the phrase "To be, or not to be". Under a character tokenizer, each character maps to one integer:
Under BPE-2048, common multi-character sequences are pre-merged into single tokens:
27. When we change the tokenizer we change that integer space, and with it the very atoms of meaning the model works with.
2 Three approaches and their problems
Character-level
Simplest possible: one token per character, vocabulary ~70 symbols. But sequences are long — a single page is ~2,000 tokens. Models have fixed context windows, so most tokens are burned on individual letters rather than meaning. The model must also learn that T, h, e together mean something — across many examples — before it can predict what follows "The".
Whole-word
Natural to humans. But English alone has 170,000+ dictionary words; add inflections (play / plays / played / playing), compounds, numbers, code identifiers, and a practical vocabulary easily exceeds a million entries. Any word not in the vocabulary becomes <UNK>, erasing its meaning. A model that sees "Schwarzenegger" as "unknown" can't distinguish it from "Dostoyevsky".
Subword — the compromise
Frequent words get their own token. Rare words are split into frequent fragments they share with common words:
Nothing is ever truly unknown. The vocabulary stays manageable (~32k–130k for modern models) while sequences remain tractable.
3 The BPE algorithm — step by step
Byte Pair Encoding (Sennrich et al., 2016) is the most widely used tokenization method in large language models.
- Start with characters. Pre-tokenize the corpus into words. Represent each unique word as a sequence of characters, and count how often each word appears.
low×5 lower×2 newest×6
- Count all adjacent pairs across all words, weighted by word frequency.
l+oappears 7 times (5 from "low" + 2 from "lower");e+sappears 6 times. - Merge the most frequent pair. Say
l+owins. Create tokenloand replace every occurrence:low×5 lower×2 newest×6 - Repeat until you reach the target vocabulary size. Each merge records which pair, how often it occurred, and what it produced. That log is the entire learned tokenizer.
- To encode new text: apply the merges in learned order. Even words unseen during training get decomposed by the same sequence of merges.
th, ou, ght early because thou, thought, hath are common. Wikipedia learns in, ion, ation early. Same algorithm, different corpus → different tokens. Switch corpora in the Playground and watch the merge table change.
4 Vocabulary size tradeoffs
The target vocabulary size is the key hyperparameter. It pulls in opposite directions:
~66 char
~512 BPE
~2048 BPE
Real models: BERT ~30k tokens, GPT-2 ~50k, LLaMA 3 128k. Bigger is not always better — with a small corpus a large vocabulary means many rare tokens trained on almost no data.
5 The embedding table — why tokenizers and models are inseparable
What an embedding actually is
A language model never operates on text or token strings. It operates on
vectors. The very first thing it does with a sequence of IDs
is look each one up in an embedding table — a matrix of shape
[vocab_size × d_model] where d_model is the hidden
dimension (128 in these nano-GPTs). Each row is a point in that
128-dimensional space, one row per token in the vocabulary.
What these vectors learn
The rows are not hand-crafted. They start as random noise and are updated by gradient descent on every training step. The signal that shapes them: tokens that appear in similar contexts receive similar gradient updates and gradually move closer together in the 128-dimensional space.
After training the Shakespeare BPE-2048 model, we can measure this directly. The nearest neighbour of ·night in embedding space is ·morrow (cosine similarity 0.81), followed by ·day (0.70). The model has never been told that "night" and "morrow" are related — it inferred it from thousands of lines of Shakespeare where the two words appear in overlapping contexts ("good morrow", "the night before", time-of-day constructions).
Char embeddings cluster by character class, not word meaning. Every dimension of the model's input represents a single letter.
All vowels. Structurally coherent, semantically useless for language.
BPE embeddings cluster by contextual meaning. Tokens that appear in similar situations end up as neighbours.
Times of day + associated concepts. No linguistic labelling needed — pure co-occurrence.
Weight tying — the output is the input
There is an additional coupling that makes the embedding table even more deeply entangled with the model. The final layer of the transformer produces a logit for every token in the vocabulary. In standard language model design (weight tying, Press & Wolf 2017), this output projection shares weights with the embedding table. Row i of the embedding serves double duty: it is the input representation for token i and also the direction the model "votes for" when it wants to predict token i.
The consequence: every training step updates the embedding not just through the input path but also through the output path. The embedding table is shaped by the full model — by every attention head, every MLP layer, every layer norm. It is not an isolated lookup; it is the model's complete vocabulary of meaning.
The catastrophe: why IDs are arbitrary
Tokenizer A and Tokenizer B produce completely different integer sequences for
the same text. There is no shared coordinate system. When the char model
tokenises "To be, or not to be", the first ID is 33
(the char T). When the BPE-512 model tokenises the same text, the
first ID is 207 (the subword ·To). These integers have
no relationship to each other.
[207, 115, 7 …][33, 54, 2, 41 …]
The model isn't "confused" — it is answering a completely different question.
Char IDs [33, 54, 2 …] fed into the BPE model trigger row lookups
for whatever BPE tokens happen to live at those positions. The model faithfully
continues that (nonsensical) sequence. Changing the tokenizer means building
a new model from scratch, because the entire weight matrix is calibrated to
one specific mapping between integers and meanings.
6 Bits per character — the fair metric
Cross-entropy loss seems like an obvious comparison metric, but it traps you: a BPE-2048 model almost always has higher per-token loss than a character model on the same text — not because it is worse, but because each of its tokens carries more information. A token like "tion" is harder to predict than the next character.
BPC asks: "how many bits does the model need to encode each character?" Lower is better and the number is comparable across tokenizers. In this playground, BPC consistently improves Character → BPE-512 → BPE-2048 on both corpora, even though per-token loss moves in the opposite direction.
7 Reading the playground
Switch to the Playground tab. Here is what each panel shows and what to notice:
The analysis updates ~150 ms after you stop typing. Try a word not in everyday English — a proper noun or chemical name — to see BPE handle it without an unknown fallback.
Each coloured chip is one token. Hover to see its integer ID and position. The count badge pulses on every change — watch it drop as you switch Character → BPE-512 → BPE-2048. Click "show token IDs" to reveal the raw integer sequence the model actually receives.
Top-15 most probable next tokens from the nano-GPT for the selected (corpus, scheme) pair. Toggle the scheme and the bars genuinely change because a different model answers. Character predicts individual letters; BPE-512 predicts word-starts; BPE-2048 predicts whole common words.
Three cards showing token count and quality metrics for every scheme simultaneously. Compare tok/char (compression ratio) and bits/char (model quality) across columns. The highlighted card is the active scheme.
Only active for BPE schemes. Type any word and drag the slider left — the word starts as individual characters and collapses step by step as merges are applied in the order they were learned from the corpus. The full merge list below is the complete learned vocabulary.
8 Real-world tokenizers
The playground uses simple from-scratch BPE to keep the algorithm visible. Production models use more sophisticated variants:
Base vocabulary = all 256 byte values. No unknown tokens ever — every string is a sequence of bytes. Trade-off: a single Chinese character may take 2–3 byte tokens.
Merges chosen to maximise likelihood of the training corpus rather than pair frequency. Continuation pieces are prefixed with ## ("tokenization" → ["token", "##ization"]).
Starts with a large vocabulary and prunes it. Language-agnostic — treats spaces as ordinary characters. Works well for languages without clear word boundaries.
OpenAI's open-source, Rust-backed implementation of byte-level BPE. "cl100k_base" (GPT-4) has 100,256 tokens. "o200k_base" (GPT-4o) has 200,019.
9 Hidden effects of tokenization
The tokenization tax on non-English languages
If a BPE vocabulary is trained on predominantly English text, the same token budget buys fewer words in Arabic, Turkish, or Korean — sometimes 3–5× fewer. Non-English speakers get shorter effective context windows and pay more per token in API costs.
Prompt sensitivity
Small surface changes humans consider equivalent — capitalisation, a leading space, a trailing newline — can shift token boundaries and change model behaviour. These two strings produce different first tokens:
Arithmetic and reasoning
Numbers are often fragmented unpredictably. 1,000,000 might tokenize as [1, ,, 000, ,, 000] or as a single token. The model reasons over number fragments, not numbers — making digit-level arithmetic harder than it looks.
Token budget and API cost
LLM API providers charge per token. A poorly tokenized language can cost 3–4× more per word. Checking actual token counts before sending long prompts is worth the habit.
You now have the full picture. Head to the Playground and put it to work.