Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,42 @@
+---
+license: apache-2.0
+tags:
+  - gemma
+  - tokenizer
+  - sentencepiece
+  - bpe
+---
+
+# Gemma 4 E2B SentencePiece BPE Tokenizer (derived)
+
+A SentencePiece `.model` file synthesized from the upstream
+`onnx-community/gemma-4-E2B-it-ONNX` `tokenizer.json`, plus a companion
+`atomic_tokens.json` that lists the special / added tokens for an HF-style
+atomic pre-pass.
+
+Byte-equivalence verified against the upstream HF tokenizer on **1000 / 1000**
+diverse strings (English prose, source code, CJK / Cyrillic / Arabic / Hindi
+text, whitespace edge cases, emoji, NFKC-sensitive ligatures, and special-token
+literals). Verification uses an atomic pre-pass over `atomic_tokens.json`
+followed by `sentencepiece.encode`.
+
+## Files
+
+- `tokenizer.model` — SentencePiece BPE ModelProto, 262 144 pieces,
+  `byte_fallback=true`, `normalizer_spec.name="identity"`,
+  `normalizer_spec.add_dummy_prefix=false`.
+- `atomic_tokens.json` — array of `{ piece, id }` entries, sorted longest-first,
+  covering every `added_tokens` entry from upstream `tokenizer.json` with
+  `special: true` (e.g. `<bos>`, `<eos>`, `<pad>`, `<unk>`, `<mask>`, `<|tool>`,
+  …). Consumers run a greedy, overlap-free, longest-match-first scan over the
+  raw input and emit the listed `id` directly for matched substrings before
+  invoking the SentencePiece tokenizer on the surrounding text.
+
+## Hashes (sha256)
+
+- `tokenizer.model`: `57dc9e2498ca33228ae89bed2721a57817ded224a7e02d3a79a03a9619f2ff33`
+- `atomic_tokens.json`: `492ec2ad2531b146d6b8396327704e25b339264bef8356d251e3e3ee787c2423`
+
+## License
+
+Apache-2.0 (matching upstream Gemma 4 license).