---
license: apache-2.0
tags:
  - gemma
  - tokenizer
  - sentencepiece
  - bpe
---

# Gemma 4 E2B SentencePiece BPE Tokenizer (derived)

A SentencePiece `.model` file synthesized from the upstream
`onnx-community/gemma-4-E2B-it-ONNX` `tokenizer.json`, plus a companion
`atomic_tokens.json` that lists the special / added tokens for an HF-style
atomic pre-pass.

Byte-equivalence verified against the upstream HF tokenizer on **1000 / 1000**
diverse strings (English prose, source code, CJK / Cyrillic / Arabic / Hindi
text, whitespace edge cases, emoji, NFKC-sensitive ligatures, and special-token
literals). Verification uses an atomic pre-pass over `atomic_tokens.json`
followed by `sentencepiece.encode`.

## Files

- `tokenizer.model` — SentencePiece BPE ModelProto, 262 144 pieces,
  `byte_fallback=true`, `normalizer_spec.name="identity"`,
  `normalizer_spec.add_dummy_prefix=false`.
- `atomic_tokens.json` — array of `{ piece, id }` entries, sorted longest-first,
  covering every `added_tokens` entry from upstream `tokenizer.json` with
  `special: true` (e.g. `<bos>`, `<eos>`, `<pad>`, `<unk>`, `<mask>`, `<|tool>`,
  …). Consumers run a greedy, overlap-free, longest-match-first scan over the
  raw input and emit the listed `id` directly for matched substrings before
  invoking the SentencePiece tokenizer on the surrounding text.

## Hashes (sha256)

- `tokenizer.model`: `57dc9e2498ca33228ae89bed2721a57817ded224a7e02d3a79a03a9619f2ff33`
- `atomic_tokens.json`: `492ec2ad2531b146d6b8396327704e25b339264bef8356d251e3e3ee787c2423`

## License

Apache-2.0 (matching upstream Gemma 4 license).