Gemma 3 tokenizer vocabulary in reviewable .spm text form

Converted id for id from Google's Gemma 3 tokenizer.model (MD5 00d2276cbec4474f6cf3df98fbc18cbb), preserving every piece, score and SentencePiece piece type. EmbeddingGemma ships this vocabulary byte-identically. Includes the Gemma Terms of Use, the required NOTICE and a modification notice.

LICENSE

@@ -0,0 +1,241 @@
+Gemma Terms of Use
+
+The terms below apply to Gemma models listed in the Appendix at bottom of this page. For Gemma 4 terms, see the Gemma 4 license.
+
+Last modified: April 1, 2026
+
+By using, reproducing, modifying, distributing, performing or displaying any
+portion or element of Gemma, Model Derivatives including via any Hosted Service,
+(each as defined below) (collectively, the "Gemma Services") or otherwise
+accepting the terms of this Agreement, you agree to be bound by this Agreement.
+
+Section 1: DEFINITIONS
+
+1.1 Definitions
+
+(a) "Agreement" or "Gemma Terms of Use" means these terms and conditions
+that govern the use, reproduction, Distribution or modification of the Gemma
+Services and any terms and conditions incorporated by reference.
+
+(b) "Distribution" or "Distribute" means any transmission, publication,
+or other sharing of Gemma or Model Derivatives to a third party, including by
+providing or making Gemma or its functionality available as a hosted service via
+API, web access, or any other electronic or remote means ("Hosted Service").
+
+(c) "Gemma" means the set of machine learning language models, trained model
+weights and parameters identified in the Appendix,
+regardless of the source that you obtained it from.
+
+(d) "Google" means Google LLC.
+
+(e) "Model Derivatives" means all (i) modifications to Gemma, (ii) works based
+on Gemma, or (iii) any other machine learning model which is created by transfer
+of patterns of the weights, parameters, operations, or Output of Gemma, to that
+model in order to cause that model to perform similarly to Gemma, including
+distillation methods that use intermediate data representations or methods based
+on the generation of synthetic data Outputs by Gemma for training that model.
+For clarity, Outputs are not deemed Model Derivatives.
+
+(f) "Output" means the information content output of Gemma or a Model
+Derivative that results from operating or otherwise using Gemma or the Model
+Derivative, including via a Hosted Service.
+
+1.2
+
+As used in this Agreement, "including" means
+"including without limitation".
+
+Section 2: ELIGIBILITY AND USAGE
+
+2.1 Eligibility
+
+You represent and warrant that you have the legal capacity to enter into this
+Agreement (including being of sufficient age of consent). If you are accessing
+or using any of the Gemma Services for or on behalf of a legal entity, (a) you
+are entering into this Agreement on behalf of yourself and that legal entity,
+(b) you represent and warrant that you have the authority to act on behalf of
+and bind that entity to this Agreement and (c) references to "you" or
+"your" in the remainder of this Agreement refers to both you (as an
+individual) and that entity.
+
+2.2 Use
+
+You may use, reproduce, modify, Distribute, perform or display any of the Gemma
+Services only in accordance with the terms of this Agreement, and must not
+violate (or encourage or permit anyone else to violate) any term of this
+Agreement.
+
+Section 3: DISTRIBUTION AND RESTRICTIONS
+
+3.1 Distribution and Redistribution
+
+You may reproduce or Distribute copies of Gemma or Model Derivatives if you meet
+all of the following conditions:
+
+ - You must include the use restrictions referenced in Section 3.2 as an
+enforceable provision in any agreement (e.g., license agreement, terms of use,
+etc.) governing the use and/or distribution of Gemma or Model Derivatives and
+you must provide notice to subsequent users you Distribute to that Gemma or
+Model Derivatives are subject to the use restrictions in Section 3.2.
+
+ - You must provide all third party recipients of Gemma or Model Derivatives a
+copy of this Agreement.
+
+ - You must cause any modified files to carry prominent notices stating that you
+modified the files.
+
+ - All Distributions (other than through a Hosted Service) must be accompanied
+by a "Notice" text file that contains the following notice:
+"Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms".
+
+You may add your own intellectual property statement to your modifications and,
+except as set forth in this Section, may provide additional or different terms
+and conditions for use, reproduction, or Distribution of your modifications, or
+for any such Model Derivatives as a whole, provided your use, reproduction,
+modification, Distribution, performance, and display of Gemma otherwise complies
+with the terms and conditions of this Agreement. Any additional or different
+terms and conditions you impose must not conflict with the terms of this
+Agreement.
+
+3.2 Use Restrictions
+
+You must not use any of the Gemma Services:
+
+ - for the restricted uses set forth in the Gemma Prohibited Use Policy at
+ai.google.dev/gemma/prohibited_use_policy
+("Prohibited Use Policy"), which is hereby incorporated by reference into
+this Agreement; or
+
+ - in violation of applicable laws and regulations.
+
+To the maximum extent permitted by law, Google reserves the right to restrict
+(remotely or otherwise) usage of any of the Gemma Services that Google
+reasonably believes are in violation of this Agreement.
+
+3.3 Generated Output
+
+Google claims no rights in Outputs you generate using Gemma. You and your users
+are solely responsible for Outputs and their subsequent uses.
+
+Section 4: ADDITIONAL PROVISIONS
+
+4.1 Updates
+
+Google may update Gemma from time to time.
+
+4.2 Trademarks
+
+Nothing in this Agreement grants you any rights to use Google's trademarks,
+trade names, logos or to otherwise suggest endorsement or misrepresent the
+relationship between you and Google. Google reserves any rights not expressly
+granted herein.
+
+4.3 DISCLAIMER OF WARRANTY
+
+UNLESS REQUIRED BY APPLICABLE LAW, THE GEMMA SERVICES, AND OUTPUTS, ARE PROVIDED
+ON AN "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER
+EXPRESS OR IMPLIED, INCLUDING ANY WARRANTIES OR CONDITIONS OF TITLE,
+NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. YOU ARE
+SOLELY RESPONSIBLE FOR DETERMINING THE APPROPRIATENESS OF USING, REPRODUCING,
+MODIFYING, PERFORMING, DISPLAYING OR DISTRIBUTING ANY OF THE GEMMA SERVICES
+OR OUTPUTS AND ASSUME ANY AND ALL RISKS ASSOCIATED WITH YOUR USE OR DISTRIBUTION
+OF ANY OF THE GEMMA SERVICES OR OUTPUTS AND YOUR EXERCISE OF RIGHTS AND
+PERMISSIONS UNDER THIS AGREEMENT.
+
+4.4 LIMITATION OF LIABILITY
+
+TO THE FULLEST EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT AND UNDER NO
+LEGAL THEORY, WHETHER IN TORT (INCLUDING NEGLIGENCE), PRODUCT LIABILITY,
+CONTRACT, OR OTHERWISE, UNLESS REQUIRED BY APPLICABLE LAW, SHALL GOOGLE OR ITS
+AFFILIATES BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY DIRECT, INDIRECT,
+SPECIAL, INCIDENTAL, EXEMPLARY, CONSEQUENTIAL, OR PUNITIVE DAMAGES, OR LOST
+PROFITS OF ANY KIND ARISING FROM THIS AGREEMENT OR RELATED TO, ANY OF THE GEMMA
+SERVICES OR OUTPUTS EVEN IF GOOGLE OR ITS AFFILIATES HAVE BEEN ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGES.
+
+4.5 Term, Termination, and Survival
+
+The term of this Agreement will commence upon your acceptance of this Agreement
+(including acceptance by your use, modification, or Distribution, reproduction,
+performance or display of any portion or element of the Gemma Services) and will
+continue in full force and effect until terminated in accordance with the terms
+of this Agreement. Google may terminate this Agreement if you are in breach of
+any term of this Agreement. Upon termination of this Agreement, you must delete
+and cease use and Distribution of all copies of Gemma and Model Derivatives in
+your possession or control. Sections 1, 2.1, 3.3, 4.2 to 4.9 shall survive the
+termination of this Agreement.
+
+4.6 Governing Law and Jurisdiction
+
+This Agreement will be governed by the laws of the State of California without
+regard to choice of law principles. The UN Convention on Contracts for the
+International Sale of Goods does not apply to this Agreement. The state and
+federal courts of Santa Clara County, California shall have exclusive
+jurisdiction of any dispute arising out of this Agreement.
+
+4.7 Severability
+
+If any provision of this Agreement is held to be invalid, illegal or
+unenforceable, the remaining provisions shall be unaffected thereby and remain
+valid as if such provision had not been set forth herein.
+
+4.8 Entire Agreement
+
+This Agreement states all the terms agreed between the parties and supersedes
+all other agreements between the parties as of the date of acceptance relating
+to its subject matter.
+
+4.9 No Waiver
+
+Google will not be treated as having waived any rights by not exercising (or
+delaying the exercise of) any rights under this Agreement.
+
+Appendix
+
+ - Gemma 1
+
+ - Gemma 1.1
+
+ - Gemma 2
+
+ - Gemma 3
+
+ - Gemma 3n
+
+ - FunctionGemma
+
+ - EmbeddingGemma
+
+ - PaliGemma
+
+ - PaliGemma 2
+
+ - ShieldGemma
+
+ - ShieldGemma 2
+
+ - CodeGemma
+
+ - CodeGemma 1.1
+
+ - Gemma 2 JPN
+
+ - DataGemma RIG
+
+ - DataGemma RAG
+
+ - RecurrentGemma
+
+ - Gemma Scope
+
+ - Gemma-APS
+
+ - T5Gemma
+
+ - VaultGemma
+
+ - FunctionGemma
+
+ - T5Gemma 2
+
+ - TranslateGemma

NOTICE

@@ -0,0 +1,26 @@
+Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms
+
+---
+
+MODIFICATION NOTICE
+
+`gemma3.spm` in this repository is a MODIFIED form of Google's Gemma 3
+`tokenizer.model`. It is not the original file.
+
+What was changed: the SentencePiece protocol-buffer model was converted, id by
+id, into a plain-text format — one line per token id, holding that id's piece,
+score and piece type. Nothing was added, removed, reordered or rounded: all
+262,144 pieces keep their ids, their scores and their SentencePiece piece types,
+and the conversion is checked in both directions before the file is written.
+
+What was NOT changed: no vocabulary entry, score, piece type or id assignment
+differs from Google's file in any way.
+
+Source: Google's Gemma 3 `tokenizer.model`, MD5 00d2276cbec4474f6cf3df98fbc18cbb
+
+---
+
+Use of Gemma and of Model Derivatives is subject to the use restrictions in
+Section 3.2 of the Gemma Terms of Use, reproduced in the LICENSE file in this
+repository. Those restrictions apply to you, and you must pass them on to anyone
+you distribute this file or any derivative of it to.

README.md

@@ -0,0 +1,176 @@
+---
+license: gemma
+tags:
+  - tokenizer
+  - sentencepiece
+  - vocabulary
+  - gemma
+library_name: splintr
+---
+
+# Gemma 3 tokenizer vocabulary, as reviewable text (`.spm`)
+
+Google's Gemma 3 `tokenizer.model` converted, id for id, into a plain-text
+format that a human can read, diff and audit — and that a tokenizer can load
+without a protobuf dependency.
+
+**This is a modified file.** See [Modification notice](#modification-notice)
+below and the `NOTICE` file. **Use is subject to the Gemma Terms of Use**,
+including the Section 3.2 use restrictions — see [Licence](#licence).
+
+Nothing here is a model. This is a tokenizer vocabulary: the list of pieces a
+tokenizer splits text on, with the scores that decide merge order.
+
+The same conversion for Gemma 2 is at
+[`fs90/gemma-2-tokenizer-spm`](https://huggingface.co/fs90/gemma-2-tokenizer-spm).
+
+## Why this exists
+
+A SentencePiece `tokenizer.model` is a protocol buffer. That makes it opaque:
+you cannot diff two of them, grep one, or see in a pull request what a change
+did. It also means anything that wants to read one needs a protobuf parser and
+the SentencePiece schema.
+
+The obvious alternative — the `.tiktoken` format, `base64(token) rank` per line
+— is **lossy for SentencePiece**, in three separate ways:
+
+1. **Scores are destroyed.** SentencePiece merges by score, not by id order.
+   Recovering merge order from id order is not an approximation, it is
+   sometimes an inversion.
+2. **Byte-fallback spelling is destroyed.** A real SentencePiece piece is
+   spelled `<0x41>`; storing the raw byte forces a reader to _reconstruct_ that
+   spelling by scanning for a run of 256 consecutive ids.
+3. **Piece type is destroyed.** SentencePiece matches `USER_DEFINED` pieces
+   verbatim, before merging; they are never merge candidates. `CONTROL` pieces
+   are never matched from text at all. Both score `0.0` and both are spelled
+   `<...>`, so neither the score nor the spelling tells them apart.
+
+**Gemma 3 is the vocabulary where that third point bites hardest.** It declares
+**6,410** `USER_DEFINED` pieces — HTML markers such as `<blockquote>`, and, unlike
+Gemma 2, the whitespace and newline runs (`\n`, `\n\n`, …, and runs of spaces).
+Drop the type and every one of them is re-merged from its parts: `<blockquote>`
+becomes `<` + `blockquote` + `>`, and an indent becomes several shorter runs.
+Measured against `sentencepiece` over 1,380 real documents of English and mixed
+source code, that mistokenized **11.0%** of them. The same test on Gemma 2, with
+only 245 user-defined pieces, was 5.6%.
+
+This format keeps all three.
+
+## Format
+
+One line per token id, in ascending id order, no gaps:
+
+```
+<base64 of the piece, UTF-8 encoded> <score> <type>
+```
+
+```
+PHBhZD4= 0.0 3        # <pad>   score 0.0   CONTROL
+PGVvcz4= 0.0 3        # <eos>   score 0.0   CONTROL
+PGJvcz4= 0.0 3        # <bos>   score 0.0   CONTROL
+```
+
+- **piece** — SentencePiece's own `id_to_piece(i)`, so `<0x41>` keeps its real
+  byte-fallback spelling and `▁` word-boundary runs keep theirs. Base64 because
+  a piece may contain spaces, newlines or invalid-looking bytes.
+- **score** — `get_score(i)`, written as the shortest decimal that round-trips
+  the IEEE-754 value.
+- **type** — SentencePiece's own `ModelProto.SentencePiece.Type` enum:
+  `1` NORMAL, `2` UNKNOWN, `3` CONTROL, `4` USER_DEFINED, `6` BYTE.
+
+The id is the line's position, so ids cannot be duplicated or non-monotonic by
+construction — there is no id field to disagree with the ordering.
+
+## What is in this vocabulary
+
+|              |                               |
+| ------------ | ----------------------------- |
+| pieces       | 262,144                       |
+| NORMAL       | 255,474                       |
+| USER_DEFINED | 6,410                         |
+| BYTE         | 256                           |
+| CONTROL      | 3 (`<pad>`, `<eos>`, `<bos>`) |
+| UNKNOWN      | 1 (`<unk>`)                   |
+
+Two properties worth knowing before you write a loader:
+
+- **`add_dummy_prefix` is `false`.** Gemma does not prepend a word-boundary
+  marker to the input, unlike Llama and Mistral. Prepending one anyway shifts
+  the first piece of every input to a different token.
+- **`byte_fallback` is `true`**, and the 256 `<0xNN>` pieces are how it is
+  reached.
+
+## Relationship to other Gemma generations
+
+- **EmbeddingGemma** ships this exact vocabulary — all 262,144 pieces *and*
+  scores are byte-identical, so this file serves it too.
+- **Gemma 2** is a genuinely different vocabulary: 256,000 pieces, and only 245
+  user-defined ones.
+- **Gemma 4** shares this generation's merge structure but reassigns 6,206
+  marker ids, promoting reserved `<unusedN>` slots into named markers. It is
+  **not** interchangeable with this file, and it is licensed separately under
+  Apache-2.0.
+
+## Verifying this file
+
+The conversion is checked in both directions before the file is written — every
+piece, score and type is read back and compared against the source model, and
+each score is round-tripped through `f32` to confirm it survives a
+single-precision parse. To repeat that yourself against your own copy of
+Google's `tokenizer.model`:
+
+```bash
+python extract_spm_vocab.py --model tokenizer.model --output gemma3.spm --verify
+```
+
+The script is [`scripts/extract_spm_vocab.py` in
+splintr](https://github.com/ml-rust/splintr). Any SentencePiece implementation
+will do the same job; the format is simple enough to re-derive in a few lines.
+
+## Using it
+
+```python
+import base64
+
+pieces, scores, types = [], [], []
+for line in open("gemma3.spm"):
+    b64, score, kind = line.split()
+    pieces.append(base64.b64decode(b64).decode("utf-8"))
+    scores.append(float(score))
+    types.append(int(kind))
+
+# USER_DEFINED (4) pieces are matched verbatim, never merged.
+user_defined = {p for p, t in zip(pieces, types) if t == 4}
+```
+
+## Provenance
+
+Extracted from Google's Gemma 3 `tokenizer.model`, MD5
+`00d2276cbec4474f6cf3df98fbc18cbb`.
+
+The vocabulary is Google's, not this repository's, and keeps Google's licence.
+
+## Modification notice
+
+Required by Section 3.1 of the Gemma Terms of Use, and repeated in `NOTICE`:
+
+`gemma3.spm` is a **modified** form of Google's Gemma 3 `tokenizer.model` — it
+is not the original file. The protocol-buffer model was converted, id by id,
+into the plain-text format described above. Nothing was added, removed,
+reordered or rounded: all 262,144 pieces keep their ids, scores and piece types,
+and no vocabulary entry differs from Google's file in any way.
+
+## Licence
+
+Gemma is provided under and subject to the **Gemma Terms of Use**, found at
+[ai.google.dev/gemma/terms](https://ai.google.dev/gemma/terms) and reproduced in
+full in the `LICENSE` file in this repository.
+
+**Use of this file is subject to the use restrictions in Section 3.2 of that
+agreement.** If you distribute this file, or anything derived from it, you must
+pass those restrictions on to whoever you distribute to, provide them a copy of
+the agreement, and include the `NOTICE` file.
+
+Gemma 2, Gemma 3 and EmbeddingGemma are covered by these terms. **Gemma 4 is
+not** — it is released under Apache-2.0, under a separate licence at
+[ai.google.dev/gemma/apache_2](https://ai.google.dev/gemma/apache_2).