--- license: gemma tags: - tokenizer - sentencepiece - vocabulary - gemma library_name: splintr --- # Gemma 3 tokenizer vocabulary, as reviewable text (`.spm`) Google's Gemma 3 `tokenizer.model` converted, id for id, into a plain-text format that a human can read, diff and audit — and that a tokenizer can load without a protobuf dependency. **This is a modified file.** See [Modification notice](#modification-notice) below and the `NOTICE` file. **Use is subject to the Gemma Terms of Use**, including the Section 3.2 use restrictions — see [Licence](#licence). Nothing here is a model. This is a tokenizer vocabulary: the list of pieces a tokenizer splits text on, with the scores that decide merge order. The same conversion for Gemma 2 is at [`fs90/gemma-2-tokenizer-spm`](https://huggingface.co/fs90/gemma-2-tokenizer-spm). ## Why this exists A SentencePiece `tokenizer.model` is a protocol buffer. That makes it opaque: you cannot diff two of them, grep one, or see in a pull request what a change did. It also means anything that wants to read one needs a protobuf parser and the SentencePiece schema. The obvious alternative — the `.tiktoken` format, `base64(token) rank` per line — is **lossy for SentencePiece**, in three separate ways: 1. **Scores are destroyed.** SentencePiece merges by score, not by id order. Recovering merge order from id order is not an approximation, it is sometimes an inversion. 2. **Byte-fallback spelling is destroyed.** A real SentencePiece piece is spelled `<0x41>`; storing the raw byte forces a reader to _reconstruct_ that spelling by scanning for a run of 256 consecutive ids. 3. **Piece type is destroyed.** SentencePiece matches `USER_DEFINED` pieces verbatim, before merging; they are never merge candidates. `CONTROL` pieces are never matched from text at all. Both score `0.0` and both are spelled `<...>`, so neither the score nor the spelling tells them apart. **Gemma 3 is the vocabulary where that third point bites hardest.** It declares **6,410** `USER_DEFINED` pieces — HTML markers such as `
`, and, unlike Gemma 2, the whitespace and newline runs (`\n`, `\n\n`, …, and runs of spaces). Drop the type and every one of them is re-merged from its parts: `
` becomes `<` + `blockquote` + `>`, and an indent becomes several shorter runs. Measured against `sentencepiece` over 1,380 real documents of English and mixed source code, that mistokenized **11.0%** of them. The same test on Gemma 2, with only 245 user-defined pieces, was 5.6%. This format keeps all three. ## Format One line per token id, in ascending id order, no gaps: ``` ``` ``` PHBhZD4= 0.0 3 # score 0.0 CONTROL PGVvcz4= 0.0 3 # score 0.0 CONTROL PGJvcz4= 0.0 3 # score 0.0 CONTROL ``` - **piece** — SentencePiece's own `id_to_piece(i)`, so `<0x41>` keeps its real byte-fallback spelling and `▁` word-boundary runs keep theirs. Base64 because a piece may contain spaces, newlines or invalid-looking bytes. - **score** — `get_score(i)`, written as the shortest decimal that round-trips the IEEE-754 value. - **type** — SentencePiece's own `ModelProto.SentencePiece.Type` enum: `1` NORMAL, `2` UNKNOWN, `3` CONTROL, `4` USER_DEFINED, `6` BYTE. The id is the line's position, so ids cannot be duplicated or non-monotonic by construction — there is no id field to disagree with the ordering. ## What is in this vocabulary | | | | ------------ | ----------------------------- | | pieces | 262,144 | | NORMAL | 255,474 | | USER_DEFINED | 6,410 | | BYTE | 256 | | CONTROL | 3 (``, ``, ``) | | UNKNOWN | 1 (``) | Two properties worth knowing before you write a loader: - **`add_dummy_prefix` is `false`.** Gemma does not prepend a word-boundary marker to the input, unlike Llama and Mistral. Prepending one anyway shifts the first piece of every input to a different token. - **`byte_fallback` is `true`**, and the 256 `<0xNN>` pieces are how it is reached. ## Relationship to other Gemma generations - **EmbeddingGemma** ships this exact vocabulary — all 262,144 pieces *and* scores are byte-identical, so this file serves it too. - **Gemma 2** is a genuinely different vocabulary: 256,000 pieces, and only 245 user-defined ones. - **Gemma 4** shares this generation's merge structure but reassigns 6,206 marker ids, promoting reserved `` slots into named markers. It is **not** interchangeable with this file, and it is licensed separately under Apache-2.0. ## Verifying this file The conversion is checked in both directions before the file is written — every piece, score and type is read back and compared against the source model, and each score is round-tripped through `f32` to confirm it survives a single-precision parse. To repeat that yourself against your own copy of Google's `tokenizer.model`: ```bash python extract_spm_vocab.py --model tokenizer.model --output gemma3.spm --verify ``` The script is [`scripts/extract_spm_vocab.py` in splintr](https://github.com/ml-rust/splintr). Any SentencePiece implementation will do the same job; the format is simple enough to re-derive in a few lines. ## Using it ```python import base64 pieces, scores, types = [], [], [] for line in open("gemma3.spm"): b64, score, kind = line.split() pieces.append(base64.b64decode(b64).decode("utf-8")) scores.append(float(score)) types.append(int(kind)) # USER_DEFINED (4) pieces are matched verbatim, never merged. user_defined = {p for p, t in zip(pieces, types) if t == 4} ``` ## Provenance Extracted from Google's Gemma 3 `tokenizer.model`, MD5 `00d2276cbec4474f6cf3df98fbc18cbb`. The vocabulary is Google's, not this repository's, and keeps Google's licence. ## Modification notice Required by Section 3.1 of the Gemma Terms of Use, and repeated in `NOTICE`: `gemma3.spm` is a **modified** form of Google's Gemma 3 `tokenizer.model` — it is not the original file. The protocol-buffer model was converted, id by id, into the plain-text format described above. Nothing was added, removed, reordered or rounded: all 262,144 pieces keep their ids, scores and piece types, and no vocabulary entry differs from Google's file in any way. ## Licence Gemma is provided under and subject to the **Gemma Terms of Use**, found at [ai.google.dev/gemma/terms](https://ai.google.dev/gemma/terms) and reproduced in full in the `LICENSE` file in this repository. **Use of this file is subject to the use restrictions in Section 3.2 of that agreement.** If you distribute this file, or anything derived from it, you must pass those restrictions on to whoever you distribute to, provide them a copy of the agreement, and include the `NOTICE` file. Gemma 2, Gemma 3 and EmbeddingGemma are covered by these terms. **Gemma 4 is not** — it is released under Apache-2.0, under a separate licence at [ai.google.dev/gemma/apache_2](https://ai.google.dev/gemma/apache_2).