| --- |
| license: gemma |
| tags: |
| - tokenizer |
| - sentencepiece |
| - vocabulary |
| - gemma |
| library_name: splintr |
| --- |
| |
| # Gemma 2 tokenizer vocabulary, as reviewable text (`.spm`) |
|
|
| Google's Gemma 2 `tokenizer.model` converted, id for id, into a plain-text |
| format that a human can read, diff and audit — and that a tokenizer can load |
| without a protobuf dependency. |
|
|
| **This is a modified file.** See [Modification notice](#modification-notice) |
| below and the `NOTICE` file. **Use is subject to the Gemma Terms of Use**, |
| including the Section 3.2 use restrictions — see [Licence](#licence). |
|
|
| Nothing here is a model. This is a tokenizer vocabulary: the list of pieces a |
| tokenizer splits text on, with the scores that decide merge order. |
|
|
| ## Why this exists |
|
|
| A SentencePiece `tokenizer.model` is a protocol buffer. That makes it opaque: |
| you cannot diff two of them, grep one, or see in a pull request what a change |
| did. It also means anything that wants to read one needs a protobuf parser and |
| the SentencePiece schema. |
|
|
| The obvious alternative — the `.tiktoken` format, `base64(token) rank` per line |
| — is **lossy for SentencePiece**, in three separate ways: |
|
|
| 1. **Scores are destroyed.** SentencePiece merges by score, not by id order. |
| Recovering merge order from id order is not an approximation, it is |
| sometimes an inversion. |
| 2. **Byte-fallback spelling is destroyed.** A real SentencePiece piece is |
| spelled `<0x41>`; storing the raw byte forces a reader to _reconstruct_ that |
| spelling by scanning for a run of 256 consecutive ids. |
| 3. **Piece type is destroyed.** SentencePiece matches `USER_DEFINED` pieces |
| verbatim, before merging; they are never merge candidates. `CONTROL` pieces |
| are never matched from text at all. Both score `0.0` and both are spelled |
| `<...>`, so neither the score nor the spelling tells them apart. |
|
|
| That third one is not theoretical. Gemma 2 declares 245 `USER_DEFINED` pieces — |
| HTML markers such as `<blockquote>` and `<table>`. Drop the type and |
| `<blockquote>` is re-merged from `<` + `blockquote` + `>`, which measurably |
| mistokenized **5.6% of real documents** in testing. Gemma 3, which declares |
| 6,410 of them (it adds the whitespace and newline runs), was worse. |
|
|
| This format keeps all three. |
|
|
| ## Format |
|
|
| One line per token id, in ascending id order, no gaps: |
|
|
| ``` |
| <base64 of the piece, UTF-8 encoded> <score> <type> |
| ``` |
|
|
| ``` |
| PHBhZD4= 0.0 3 # <pad> score 0.0 CONTROL |
| PGVvcz4= 0.0 3 # <eos> score 0.0 CONTROL |
| PGJvcz4= 0.0 3 # <bos> score 0.0 CONTROL |
| ``` |
|
|
| - **piece** — SentencePiece's own `id_to_piece(i)`, so `<0x41>` keeps its real |
| byte-fallback spelling and `▁` word-boundary runs keep theirs. Base64 because |
| a piece may contain spaces, newlines or invalid-looking bytes. |
| - **score** — `get_score(i)`, written as the shortest decimal that round-trips |
| the IEEE-754 value. |
| - **type** — SentencePiece's own `ModelProto.SentencePiece.Type` enum: |
| `1` NORMAL, `2` UNKNOWN, `3` CONTROL, `4` USER_DEFINED, `6` BYTE. |
| |
| The id is the line's position, so ids cannot be duplicated or non-monotonic by |
| construction — there is no id field to disagree with the ordering. |
| |
| ## What is in this vocabulary |
| |
| | | | |
| | ------------ | ----------------------------- | |
| | pieces | 256,000 | |
| | NORMAL | 255,495 | |
| | USER_DEFINED | 245 | |
| | BYTE | 256 | |
| | CONTROL | 3 (`<pad>`, `<eos>`, `<bos>`) | |
| | UNKNOWN | 1 (`<unk>`) | |
|
|
| Two properties worth knowing before you write a loader: |
|
|
| - **`add_dummy_prefix` is `false`.** Gemma does not prepend a word-boundary |
| marker to the input, unlike Llama and Mistral. Prepending one anyway shifts |
| the first piece of every input to a different token. |
| - **`byte_fallback` is `true`**, and the 256 `<0xNN>` pieces are how it is |
| reached. |
| |
| ## Verifying this file |
| |
| The conversion is checked in both directions before the file is written — every |
| piece, score and type is read back and compared against the source model, and |
| each score is round-tripped through `f32` to confirm it survives a |
| single-precision parse. To repeat that yourself against your own copy of |
| Google's `tokenizer.model`: |
| |
| ```bash |
| python extract_spm_vocab.py --model tokenizer.model --output gemma2.spm --verify |
| ``` |
| |
| The script is [`scripts/extract_spm_vocab.py` in |
| splintr](https://github.com/ml-rust/splintr). Any SentencePiece implementation |
| will do the same job; the format is simple enough to re-derive in a few lines. |
| |
| ## Using it |
| |
| ```python |
| import base64 |
| |
| pieces, scores, types = [], [], [] |
| for line in open("gemma2.spm"): |
| b64, score, kind = line.split() |
| pieces.append(base64.b64decode(b64).decode("utf-8")) |
| scores.append(float(score)) |
| types.append(int(kind)) |
| |
| # USER_DEFINED (4) pieces are matched verbatim, never merged. |
| user_defined = {p for p, t in zip(pieces, types) if t == 4} |
| ``` |
| |
| ## Provenance |
| |
| Extracted from Google's Gemma 2 `tokenizer.model`, MD5 |
| `f9e2445870ec741aa6346bbd75531bb4`. |
| |
| The vocabulary is Google's, not this repository's, and keeps Google's licence. |
| |
| ## Modification notice |
| |
| Required by Section 3.1 of the Gemma Terms of Use, and repeated in `NOTICE`: |
| |
| `gemma2.spm` is a **modified** form of Google's Gemma 2 `tokenizer.model` — it |
| is not the original file. The protocol-buffer model was converted, id by id, |
| into the plain-text format described above. Nothing was added, removed, |
| reordered or rounded: all 256,000 pieces keep their ids, scores and piece |
| types, and no vocabulary entry differs from Google's file in any way. |
| |
| ## Licence |
| |
| Gemma is provided under and subject to the **Gemma Terms of Use**, found at |
| [ai.google.dev/gemma/terms](https://ai.google.dev/gemma/terms) and reproduced in |
| full in the `LICENSE` file in this repository. |
| |
| **Use of this file is subject to the use restrictions in Section 3.2 of that |
| agreement.** If you distribute this file, or anything derived from it, you must |
| pass those restrictions on to whoever you distribute to, provide them a copy of |
| the agreement, and include the `NOTICE` file. |
| |
| Gemma 2, Gemma 3 and EmbeddingGemma are covered by these terms. **Gemma 4 is |
| not** — it is released under Apache-2.0, under a separate licence at |
| [ai.google.dev/gemma/apache_2](https://ai.google.dev/gemma/apache_2). |
| |
