Update model card

README.md

@@ -3,7 +3,7 @@ license: apache-2.0
 language:
 - tr
 - en
-library_name: transformers
+library_name: tokenizers
 tags:
 - tokenizer
 - tokenizers
@@ -11,14 +11,12 @@ tags:
 - turkish
 - english
 - bilingual
-datasets:
-- wikimedia/wikipedia
-- Helsinki-NLP/opus-100
+- sentencepiece
 ---
 
 # Multrenizer
 
-Multrenizer is a bilingual English-Turkish Unigram tokenizer built from scratch for Turkish morphology, Turkish-aware casing, and mixed TR-EN text.
+A bilingual Turkish-English Unigram tokenizer with **lossless round-trip decode**, **case preservation**, and a **correct utility-token taxonomy** — punctuation, currency, math symbols, and emoji are kept atomic in the vocabulary AND preserved through `decode(skip_special_tokens=True)`.
 
 ## Links
 
@@ -27,139 +25,32 @@ Multrenizer is a bilingual English-Turkish Unigram tokenizer built from scratch
 
 ## Why Multrenizer?
 
-Standard multilingual tokenizers routinely break Turkish at poor boundaries, waste context on agglutinative suffixes, and mishandle the Turkish dotted/dotless `I/i` rule. Multrenizer is designed to fix those failure modes without discarding punctuation and chat-critical symbols.
+Standard multilingual tokenizers commonly fail on Turkish in three ways: (1) they break agglutinative morphemes at suboptimal boundaries, (2) they fold case (`İstanbul` → `istanbul`) so a downstream model can never produce proper nouns again, and (3) they misclassify punctuation and emoji as "special tokens", which means `decode(skip_special_tokens=True)` silently strips them and the model output ends up missing commas, apostrophes, and 😀.
 
-Core design goals:
+Multrenizer is built to fix all three without giving up Turkish efficiency.
 
-- Turkish-aware normalization: hardcoded `İ -> i` and `I -> ı` before Unicode normalization
-- Apostrophe preservation: forms like `feature'ı`, `merge'lemek`, `İstanbul'da`, and `can't` keep `'` as a real token
-- Compact vocabulary budget: `~26K` target vocab for a Turkish-first bilingual tokenizer
-- Fixed utility budget: dedicated punctuation, emoji, math, currency, and chat symbols
-- Code-switching support: trained on mixed TR-EN text instead of treating it as noise
+Core design:
 
-## Benchmark Results
-
-Evaluated on `5,000` Turkish sentences, `5,000` English sentences, and `500` code-switching sentences from the prepared corpus against 5 reference tokenizers.
-
-Notes:
-
-- Multrenizer's shipped local artifact is auto-read from `multrenizer-tokenizer/tokenizer.json`; the current released artifact is `25,917` tokens.
-- Example token strings for byte-level models are shown as raw tokenizer pieces. Metrics are based on exact token counts, not prettified decoding.
-
-### Compared Tokenizers
-
-| Tokenizer | Source | Vocab Size | Algorithm | Type |
-|---|---|---:|---|---|
-| **Multrenizer** | This project | **25,917** | Unigram | Bilingual EN-TR, purpose-built |
-| **Kumru-2B** | [vngrs-ai/Kumru-2B](https://huggingface.co/vngrs-ai/Kumru-2B) | 50,176 | BPE | Turkish LLM (VNGRS, Sep 2025, Mistral-based) |
-| **Turkcell-7B** | [TURKCELL/Turkcell-LLM-7b-v1](https://huggingface.co/TURKCELL/Turkcell-LLM-7b-v1) | 48,351 | BPE | Turkish LLM (Turkcell, Apr 2024, Mistral-based) |
-| **GPT-2** | [openai-community/gpt2](https://huggingface.co/openai-community/gpt2) | 50,257 | BPE | English-centric baseline (OpenAI, 2019) |
-| **Qwen-3** | [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) | 151,643 | BPE | Multilingual (Alibaba, 2025) |
-| **Mistral-3.1** | [mistralai/Mistral-Small-3.1-24B-Base-2503](https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Base-2503) | 131,072 | BPE/SP | Multilingual (Mistral AI, Mar 2025) |
-
-### Fertility, Compression, and Token Count
-
-Lower fertility means fewer tokens per word. Higher compression means more characters carried per token.
+- **Case preserving.** `İstanbul` stays `İstanbul`, `TÜRKİYE` stays `TÜRKİYE`. The model learns to produce real proper nouns, not lowercased shadows.
+- **Lossless round-trip.** Metaspace pipeline with `prepend_scheme="first"` plus `AddedToken(normalized=True)` — `decode(encode(x)) == x` for case, punctuation, currency, math, and compound emoji.
+- **Correct utility taxonomy.** Only model-control symbols (`<s>`, `<|user|>`, `<|reserved_*|>`, etc.) are flagged `special=True`. Punctuation, currency, math symbols, and emoji are atomic vocab entries with `special=False` — they survive `skip_special_tokens=True`.
+- **Compact vocabulary.** `~26K` total budget, Turkish-first morpheme coverage.
+- **Code-switching aware.** Trained on a TR/EN/CS interleave; mixed text like `merge'lemek istediğim branch` segments cleanly.
 
-| Metric | Multrenizer | Kumru-2B | Turkcell-7B | GPT-2 | Qwen-3 | Mistral-3.1 |
-|---|:---:|:---:|:---:|:---:|:---:|:---:|
-| Vocab Size | **25,917** | 50,176 | 48,351 | 50,257 | 151,643 | 131,072 |
-| **TR Fertility** | **1.627** | 1.649 | 1.917 | 3.785 | 2.616 | 2.384 |
-| EN Fertility | 1.525 | 2.151 | 1.555 | **1.314** | 1.372 | 1.381 |
-| **CS Fertility** | **1.756** | 1.923 | 1.832 | 3.475 | 2.445 | 2.479 |
-| **TR Compression** | **4.783** | 4.719 | 4.060 | 2.056 | 2.976 | 3.265 |
-| EN Compression | 4.148 | 2.942 | 4.068 | **4.816** | 4.610 | 4.580 |
-| **TR Total Tokens (5K)** | **130,844** | 132,637 | 154,166 | 304,345 | 210,334 | 191,682 |
-| EN Total Tokens (5K) | 157,027 | 221,420 | 160,121 | **135,235** | 141,275 | 142,196 |
-| **CS Total Tokens (500)** | **5,525** | 6,050 | 5,762 | 10,933 | 7,693 | 7,799 |
-
-Current position:
-
-- Best Turkish efficiency in this comparison set: TR fertility, TR compression, TR total tokens
-- Best code-switching efficiency in this comparison set: CS fertility and CS total tokens
-- Competitive English coverage for a Turkish-first tokenizer, but not better than English-native GPT-2 on EN-only token count
-- Only tokenizer here that passes Turkish `I/i` normalization correctly
-
-### Morphological Splitting
-
-Total tokens needed to represent 10 difficult Turkish words:
-
-| Tokenizer | Vocab Size | Total Tokens | Avg per Word |
-|---|---:|:---:|:---:|
-| **Multrenizer** | **25,917** | **32** | **3.2** |
-| Kumru-2B | 50,176 | 35 | 3.5 |
-| Turkcell-7B | 48,351 | 38 | 3.8 |
-| Mistral-3.1 | 131,072 | 71 | 7.1 |
-| Qwen-3 | 151,643 | 73 | 7.3 |
-| GPT-2 | 50,257 | 105 | 10.5 |
-
-Selected examples:
+## Pipeline
 
 ```text
-güzelleştirilmiş
-  Multrenizer: güzel + leştirilmiş                                   [2 tokens]
-  Kumru-2B: güzel + leÅŁtirilmiÅŁ                                  [2 tokens]
-  Turkcell-7B: güzel + leştirilmiş                                   [2 tokens]
-  Qwen-3: g + üz + elle + ÅŁtir + ilmiÅŁ                         [5 tokens]
-  Mistral-3.1: g + üz + elle + ÅŁtir + ilmiÅŁ                     [5 tokens]
-  GPT-2: g + ü + z + elle + ÅŁ + t + ir + il + mi + ÅŁ          [10 tokens]
-
-İstanbul'da
-  Multrenizer: istanbul + ' + da                                     [3 tokens]
-  Kumru-2B: Ä°stanbul + ' + da                                      [3 tokens]
-  Turkcell-7B: İstanbul + ' + da                                     [3 tokens]
-  Qwen-3: Ä° + stanbul + 'd + a                                    [4 tokens]
-  Mistral-3.1: Ä° + stanbul + 'd + a                               [4 tokens]
-  GPT-2: Ä + ° + stanbul + 'd + a                                  [5 tokens]
-
-Afyonkarahisarlılaştıramadıklarımızdan
-  Multrenizer: afyonkarahisar + lı + laştı + r + ama + dıkları + mızda + n   [8 tokens]
-  Kumru-2B: Af + yonkarahisar + lı + laÅŁtır + ama + dık + larımız + dan [8 tokens]
-  Turkcell-7B: Afyon + kar + ah + is + arlı + laştır + a + madık + larımızdan [9 tokens]
-  Qwen-3: Af + yon + kar + ah + is + ar + lı + la + ÅŁt + ı + ram + ad + ıkl + ar + ımız + dan [16 tokens]
-  Mistral-3.1: Af + yon + kar + ah + is + arl + ı + laÅŁt + ı + ram + ad + ıkları + m + ı + zd + an [16 tokens]
-  GPT-2: Af + yon + kar + ah + is + arl + ı + la + ÅŁ + t + ı + ram + ad + ı + k + lar + ı + m + ı + z + dan [21 tokens]
+Raw text
+  -> Quote canonicalization (’ ‘ ʼ ＇ -> ')
+  -> NFKC normalization
+  -> Strip whitespace
+  -> Pre-tokenizer: Metaspace(replacement="▁", prepend_scheme="first", split=True)
+  -> Unigram model (~26K vocab)
+  -> Decoder: Metaspace(prepend_scheme="first")  [mirror of pre-tokenizer]
+  -> Post-processor (<s> ... </s>)
 ```
 
-### Turkish I/i Normalization
-
-This is the critical locale-sensitive test:
-
-- `İ` must lowercase to `i`
-- `I` must lowercase to `ı`
-
-| Input | Expected | Multrenizer | Kumru-2B | Turkcell-7B | GPT-2 | Qwen-3 | Mistral-3.1 |
-|---|---|:---:|:---:|:---:|:---:|:---:|:---:|
-| İstanbul | istanbul | **OK** | FAIL | FAIL | FAIL | FAIL | FAIL |
-| IŞIK | ışık | **OK** | FAIL | FAIL | FAIL | FAIL | FAIL |
-| SIR | sır | **OK** | FAIL | FAIL | FAIL | FAIL | FAIL |
-| İNSAN | insan | **OK** | FAIL | FAIL | FAIL | FAIL | FAIL |
-| ISITMAK | ısıtmak | **OK** | FAIL | FAIL | FAIL | FAIL | FAIL |
-| **Score** | | **8/8** | **0/8** | **0/8** | **0/8** | **0/8** | **0/8** |
-
-Multrenizer is the only tokenizer in this comparison that handles Turkish casing correctly.
-
-### Code-Switching
-
-```text
-"Bu feature'ı implement ederken edge case'leri handle etmeyi unutmayalım."
-
-  Multrenizer  [15 tok]  bu | feature | ' | ı | implement | ederken | edge | case | ' | leri | handle | etmeyi | unutmaya | lım | .
-  Kumru-2B     [20 tok]  Bu | fe | ature | ' | ı | imp | lement | ederken | ed | ge | cas | e | ' | leri | hand | le | etmeyi | unutma | yalım | .
-  Turkcell-7B  [15 tok]  Bu | feature | ' | ı | implement | ederken | edge | case | ' | leri | handle | etmeyi | unut | mayalım | .
-  GPT-2        [24 tok]  Bu | feature | ' | ı | implement | ed | er | ken | edge | case | ' | ler | i | handle | et | me | yi | un | ut | may | al | ı | m | .
-  Qwen-3       [22 tok]  Bu | feature | ' | ı | implement | ed | er | ken | edge | case | ' | leri | handle | et | m | ey | i | un | ut | may | alım | .
-  Mistral-3.1  [20 tok]  Bu | feature | 'ı | implement | eder | ken | edge | case | ' | leri | handle | et | me | yi | un | ut | may | al | ım | .
-
-"merge'lemek istediğim branch conflict veriyor."
-
-  Multrenizer  [ 8 tok]  merge | ' | lemek | istediğim | branch | conflict | veriyor | .
-  Kumru-2B     [14 tok]  mer | ge | ' | lemek | istediÄŁim | b | ran | ch | con | f | lic | t | veriyor | .
-  Turkcell-7B  [ 8 tok]  merge | ' | lemek | istediğim | branch | conflict | veriyor | .
-  GPT-2        [16 tok]  mer | ge | ' | lem | ek | is | ted | i | ÄŁ | im | branch | conflict | ver | iy | or | .
-  Qwen-3       [11 tok]  merge | ' | lem | ek | istediÄŁ | im | branch | conflict | ver | iyor | .
-  Mistral-3.1  [13 tok]  merge | ' | le | mek | ist | edi | ÄŁ | im | branch | conflict | ver | iyor | .
-```
+No lowercase. No locale-specific I/i replacement (it would be a workaround for a bug Python introduces only when you call `.lower()`, which we don't).
 
 ## Quick Start
 
@@ -180,45 +71,39 @@ from tokenizers import Tokenizer
 
 tok = Tokenizer.from_file("multrenizer-tokenizer/tokenizer.json")
 
-encoded = tok.encode("İstanbul'da güzel bir gün")
+encoded = tok.encode("İstanbul'da %50 indirim 😀")
 print(encoded.tokens)
-# ['<s>', 'istanbul', "'", 'da', 'güzel', 'bir', 'gün', '</s>']
+# ['<s>', "▁İstanbul'da", '▁%', '50', '▁indirim', '▁', '😀', '</s>']
 
-print(tok.normalizer.normalize_str("IŞIK"))
-# 'ışık'
+# Round-trip is lossless — case, punctuation, emoji all survive.
+print(tok.decode(encoded.ids))
+# "İstanbul'da %50 indirim 😀"
 ```
 
 ### Load from Hugging Face
 
-```python
-from tokenizers import Tokenizer
-
-tok = Tokenizer.from_pretrained("fzengin18/multrenizer")
-
-encoded = tok.encode("İstanbul'da güzel bir gün")
-print(encoded.tokens)
-# ['<s>', 'istanbul', "'", 'da', 'güzel', 'bir', 'gün', '</s>']
-```
-
-If you use `transformers`, this also works:
-
 ```python
 from transformers import AutoTokenizer
 
 tok = AutoTokenizer.from_pretrained("fzengin18/multrenizer")
-print(tok.tokenize("İstanbul'da güzel bir gün"))
+ids = tok.encode("Türkiye'de %50 indirim, 100₺ ❤️", add_special_tokens=False)
+
+# `skip_special_tokens=True` removes ONLY <...>-formatted control tokens.
+# Punctuation, currency, and emoji are preserved.
+print(tok.decode(ids, skip_special_tokens=True))
+# "Türkiye'de %50 indirim, 100₺ ❤️"
 ```
 
 ### Train from scratch
 
 ```bash
-# 1. Download and prepare corpus
-python prepare_data.py --size medium
+# 1. Download and prepare corpus (Wikipedia TR+EN, OPUS-100, synthetic CS)
+python prepare_data.py --size large
 
 # 2. Train tokenizer
 python train_tokenizer.py --data-dir data/
 
-# 3. Optional: push tokenizer files to Hugging Face Hub
+# 3. Optional: push to Hugging Face Hub
 python train_tokenizer.py --data-dir data/ \
   --repo-id fzengin18/multrenizer \
   --hf-token "$HF_TOKEN"
@@ -232,82 +117,200 @@ python benchmark.py --tr-lines 5000 --en-lines 5000
 
 ## Architecture
 
-### Pipeline
+### Token Taxonomy
 
-```text
-Raw text
-  -> Turkish I/i normalizer (Replace: İ->i, I->ı, i̇->i)
-  -> Quote canonicalization (’ ‘ ʼ ＇ -> ')
-  -> NFKC normalization
-  -> Lowercase
-  -> Strip whitespace
-  -> Pre-tokenizer (whitespace + apostrophe + punctuation split)
-  -> Unigram model (~26K target vocab)
-  -> Post-processor (<s> ... </s>)
-```
+| Tier | Count | `special` flag | Behavior on `skip_special_tokens=True` |
+|---|---:|:---:|---|
+| Named specials (`<s>`, `<\|user\|>`, `<think>`, ...) | 32 | `true` | **Removed** |
+| Reserved (`<\|reserved_0\|>` ... `<\|reserved_511\|>`) | 512 | `true` | **Removed** |
+| Utility — punctuation/symbols (`,`, `!`, `'`, `%`, `$`, `→`) | 169 | `false` | **Preserved** |
+| Utility — emoji (corpus-learned + curated fallback) | ~240 | `false` | **Preserved** |
+| Learned subwords (Unigram) | ~24,700 | `false` | **Preserved** |
 
-### Data Mix
+This is the central correctness fix vs the prior artifact, where 292 utility tokens were incorrectly flagged `special=true` and erased on every decode.
 
-The released artifact is trained with the default file-based interleave in `train_tokenizer.py`, which approximates:
+### Vocabulary Budget
+
+Target: `26,000`. Actual shipped artifact: `25,679` (Unigram convergence undershoot is normal).
+
+The model trained downstream should set its `vocab_size` to **the tokenizer's actual size**, not the 26K target.
+
+### Special Tokens
 
-| Stream | Share | Purpose |
+| Category | IDs | Tokens |
 |---|---|---|
-| Turkish | ~60% | Core Turkish morphology |
-| English | ~30% | English coverage |
-| Code-switching | ~10% | TR-EN boundary handling |
+| **Core** | 0-3 | `<unk>` `<s>` `</s>` `<pad>` |
+| **Chat** | 4-8 | `<\|system\|>` `<\|user\|>` `<\|assistant\|>` `<\|end\|>` `<\|sep\|>` |
+| **Reasoning** | 9-12 | `<think>` `</think>` `<\|step\|>` `<\|reflection\|>` |
+| **Tool Use** | 13-16 | `<tool_call>` `</tool_call>` `<tool_response>` `</tool_response>` |
+| **Code/FIM** | 17-20 | `<\|code\|>` `<\|fim_prefix\|>` `<\|fim_middle\|>` `<\|fim_suffix\|>` |
+| **Language** | 21-22 | `<\|tr\|>` `<\|en\|>` |
+| **RAG** | 23-24 | `<\|context\|>` `<\|/context\|>` |
+| **Multi-modal** | 25-28 | `<\|image\|>` `<\|audio\|>` `<\|video\|>` `<\|file\|>` |
+| **Structured** | 29-31 | `<\|json\|>` `<\|table\|>` `<\|cite\|>` |
+| **Reserved** | 32-543 | `<\|reserved_0\|>` ... `<\|reserved_511\|>` |
+
+Look these up at runtime with `tokenizer.token_to_id("<...>")` — do not hardcode IDs in downstream code.
+
+### Utility Tokens (special=False, atomic)
+
+| Category | Approx. count | Examples |
+|---|---:|---|
+| Punctuation | 38 | `. , ! ? ; : - ( ) [ ] { } / \ " ' ... – — … ‽ ‼ ⁇ ¿ ¡` |
+| Currency & business | 23 | `₺ $ € £ ¥ ₹ ₽ ¢ ฿ ₪ ₸ ₣ ₮ ₩ % ‰ ° § ¶ № @ # &` |
+| Math & science | 33 | `± × ÷ ≠ ≤ ≥ ≈ ∞ √ ∑ ∫ ∂ Δ π α β γ δ ε θ λ μ σ φ ω ∀ ∃ ∈ ⊂ ⊆ ∪ ∩ ⇔ ↦ ∝ ∇ ∮ ∧ ∨ ¬ Α Β Γ Σ Φ Ω` |
+| Programming digraphs | 11 | `=> != == <= >= -> :: ** // && \|\|` |
+| Arrows & symbols | 15 | `→ ← ↑ ↓ ↔ ⇒ • · ★ ☆ ✓ ✗ © ® ™` |
+| Box drawing & UI | 17 | `─ │ ┌ ┐ └ ┘ ├ ┤ ┬ ┴ ┼ ▪ ▫ ◆ ▶ ◀ ▼ ▲` |
+| Typography | 10 | `« » " " ' ' ‹ › „ ‚` |
+| Emoji — faces | 70 | `😀 😂 🤣 😊 😍 🤔 😭 😡 💀 🤖` |
+| Emoji — hands | 28 | `👋 👍 👎 👏 🙏 💪 ✊ ✌️` |
+| Emoji — hearts | 18 | `❤️ 💛 💚 💙 💜 🖤 💔` |
+| Emoji — symbols | 36 | `🔥 ✨ ⭐ ✅ ❌ ⚠️ 💯 🚀` |
+| Emoji — objects | 36 | `💻 📱 🎯 🏆 📊 ☕ 🔗 💰` |
+| Emoji — nature & common | 17 | `🌍 🎉 🎁 ⏰ 📅 📌 🚨 🛒 💳 📞 📧 🏻🏼🏽🏾🏿` |
+| Emoji — flags | 26 | `🇹🇷 🇺🇸 🇬🇧 🇩🇪 🇫🇷 🇨🇳 🇰🇷 🇮🇳 🇸🇦 🇮🇱 🇷🇺 🇧🇷 🇨🇦 🇲🇽 🇦🇺 🇳🇱` |
+| Emoji — compounds (ZWJ/VS16 sequences) | 12 | `❤️ 👨‍💻 👩‍💻 🏃‍♂️ 🏃‍♀️ 👨‍👩‍👧 👨‍🏫 👩‍🏫 👍🏻–👍🏿` |
+
+ZWJ (`U+200D`) and VS16 (`U+FE0F`) are NOT added as standalone tokens; they exist only inside whole compound-emoji sequences.
 
-Corpus collection is Turkish-forward, and code-switching examples are generated from OPUS parallel pairs during data preparation.
+### Data Mix
 
-Exact source configs used during corpus preparation:
+The released artifact is trained on the corpus produced by `prepare_data.py --size large`:
 
-- `wikimedia/wikipedia` with `20231101.tr`
-- `wikimedia/wikipedia` with `20231101.en`
-- `Helsinki-NLP/opus-100` with `en-tr`
+| Stream | Approx. lines | Share | Source |
+|---|---:|:---:|---|
+| Turkish | 1.30M | ~59% | Wikipedia TR streaming |
+| English | 0.71M | ~32% | Wikipedia EN streaming |
+| Code-switching | 0.20M | ~9% | OPUS-100 EN-TR pairs + synthetic templates |
 
-The synthetic code-switching stream is generated locally from OPUS-100 parallel pairs, so it does not appear as a separate Hugging Face dataset entry.
+An additional 5K synthetic emoji-rich CS lines are appended to ensure curated emojis make it into the learned vocabulary instead of the post-train fallback list.
 
-### Vocabulary Budget
+## Correctness Tests
 
-Multrenizer is designed around a `26,000` target vocabulary, with a fixed budget reserved for always-preserved tokens:
+The repo ships a pytest suite (`tests/test_tokenizer_correctness.py`) covering:
 
-- `32` named special tokens
-- `512` reserved tokens
-- `292` utility tokens
-- up to `25,164` learned subword tokens
+1. **Round-trip lossless decode** — `decode(encode(x)) == normalize(x)` over Turkish, English, and code-switching samples.
+2. **`skip_special_tokens=True` preserves utility** — punctuation, currency, emoji must survive; only `<...>`-formatted control tokens are removed.
+3. **Compound emoji round-trip** — VS16, ZWJ, regional-indicator pairs decode back to the original sequence.
+4. **Turkish morpheme integrity** — words like `şehirdir`, `kitabımı`, `evlerinden` decode without internal whitespace.
+5. **UNK rate < 0.5%** on a bilingual bench (line-by-line).
+6. **Special token ID stability** — IDs 0-7 (`<unk>`, `<s>`, `</s>`, `<pad>`, `<|system|>`, `<|user|>`, `<|assistant|>`, `<|end|>`).
+7. **Case preservation** — `İstanbul`, `TÜRKİYE`, `ABD`, `AKP` all round-trip without folding.
 
-Current shipped artifact: `25,917` total tokens.
+Current artifact: **60/60 PASS**.
 
-### Special Tokens
+## Benchmark Results
 
-| Category | IDs | Tokens | Purpose |
-|---|---|---|---|
-| **Core** | 0-3 | `<unk>` `<s>` `</s>` `<pad>` | Basic tokenizer operation |
-| **Chat** | 4-8 | `<\|system\|>` `<\|user\|>` `<\|assistant\|>` `<\|end\|>` `<\|sep\|>` | Instruction tuning and chat models |
-| **Reasoning** | 9-12 | `<think>` `</think>` `<\|step\|>` `<\|reflection\|>` | Reasoning traces and self-check markers |
-| **Tool Use** | 13-16 | `<tool_call>` `</tool_call>` `<tool_response>` `</tool_response>` | Tool and function calling |
-| **Code/FIM** | 17-20 | `<\|code\|>` `<\|fim_prefix\|>` `<\|fim_middle\|>` `<\|fim_suffix\|>` | Code and fill-in-middle workflows |
-| **Bilingual** | 21-22 | `<\|tr\|>` `<\|en\|>` | Language tags |
-| **RAG** | 23-24 | `<\|context\|>` `<\|/context\|>` | Retrieval boundaries |
-| **Multi-modal** | 25-28 | `<\|image\|>` `<\|audio\|>` `<\|video\|>` `<\|file\|>` | Placeholder tokens |
-| **Structured** | 29-31 | `<\|json\|>` `<\|table\|>` `<\|cite\|>` | Structured output markers |
-| **Reserved** | 32-543 | `<\|reserved_0\|>` ... `<\|reserved_511\|>` | Future growth without retraining |
-| **Utility** | 544+ | Punctuation, emoji, math, currency, typography | Critical text symbols kept intact |
-
-### Utility Tokens
-
-| Category | Count | Examples |
-|---|---:|---|
-| Punctuation | 31 | `. , ! ? ; : - ( ) [ ] { } / \ " ' ...` |
-| Currency & Business | 15 | `₺ $ € £ ¥ % @ # &` |
-| Math & Science | 25 | `± × ÷ ≠ ≤ ≥ ∞ √ π α β γ` |
-| Arrows & Symbols | 15 | `→ ← ↑ ↓ • ★ ☆ ✓ ✗ © ® ™` |
-| Typography | 10 | `« » “ ” ‘ ’ ‹ › „ ‚` |
-| Emoji (faces) | 70 | `😀 😂 🤣 😊 😍 🤔 😭 😡 💀 🤖` |
-| Emoji (hands) | 28 | `👋 👍 👎 👏 🙏 💪 ✊ ✌️` |
-| Emoji (hearts) | 18 | `❤️ 💛 💚 💙 💜 🖤 💔` |
-| Emoji (symbols) | 36 | `🔥 ✨ ⭐ ✅ ❌ ⚠️ 💯 🚀` |
-| Emoji (objects) | 36 | `💻 📱 🎯 🏆 📊 ☕ 🔗 💰` |
-| Emoji (flags) | 8 | `🇹🇷 🇺🇸 🇬🇧 🇩🇪 🇫🇷 🇪🇸 🇮🇹 🇯🇵` |
+Evaluated on `5,000` Turkish sentences, `5,000` English sentences, and `500` code-switching sentences from the prepared corpus, against five reference tokenizers.
+
+### Compared Tokenizers
+
+| Tokenizer | Source | Vocab Size | Algorithm | Type |
+|---|---|---:|---|---|
+| **Multrenizer** | This project | **25,679** | Unigram | Bilingual EN-TR, purpose-built |
+| **Kumru-2B** | [vngrs-ai/Kumru-2B](https://huggingface.co/vngrs-ai/Kumru-2B) | 50,176 | BPE | Turkish LLM (VNGRS, Sep 2025, Mistral-based) |
+| **Turkcell-7B** | [TURKCELL/Turkcell-LLM-7b-v1](https://huggingface.co/TURKCELL/Turkcell-LLM-7b-v1) | 48,351 | BPE | Turkish LLM (Turkcell, Apr 2024, Mistral-based) |
+| **GPT-2** | [openai-community/gpt2](https://huggingface.co/openai-community/gpt2) | 50,257 | BPE | English-centric baseline (OpenAI, 2019) |
+| **Qwen-3** | [Qwen/Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B) | 151,643 | BPE | Multilingual (Alibaba, 2025) |
+| **Mistral-3.1** | [mistralai/Mistral-Small-3.1-24B-Base-2503](https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Base-2503) | 131,072 | BPE/SP | Multilingual (Mistral AI, Mar 2025) |
+
+### Fertility, Compression, and Token Count
+
+Lower fertility = fewer tokens per word. Higher compression = more characters per token.
+
+| Metric | Multrenizer | Kumru-2B | Turkcell-7B | GPT-2 | Qwen-3 | Mistral-3.1 |
+|---|:---:|:---:|:---:|:---:|:---:|:---:|
+| Vocab Size | **25,679** | 50,176 | 48,351 | 50,257 | 151,643 | 131,072 |
+| **TR Fertility** | **1.63** | 1.65 | 1.92 | 3.79 | 2.62 | 2.38 |
+| EN Fertility | 1.51 | 2.15 | 1.55 | **1.31** | 1.37 | 1.38 |
+| **TR Compression** | **4.79** | 4.72 | 4.06 | 2.06 | 2.98 | 3.27 |
+| EN Compression | 4.18 | 2.94 | 4.07 | **4.82** | 4.61 | 4.58 |
+| **TR Total Tokens (5K sent)** | **130,725** | 132,637 | 154,166 | 304,345 | 210,334 | 191,682 |
+| EN Total Tokens (5K sent) | 155,848 | 221,420 | 160,121 | **135,235** | 141,275 | 142,196 |
+| Round-trip Lossless | 11/11 | 11/11 | 11/11 | 11/11 | 11/11 | 11/11 |
+
+Headline:
+
+- **Best Turkish efficiency** in this set on every TR metric (fertility, compression, total tokens), with the smallest vocabulary by a wide margin.
+- **Competitive English** — second only to GPT-2 on EN total tokens, and that's an English-native baseline trained on far more English text.
+- **Round-trip lossless on all 11 cases.** This is a minimum-correctness bar; modern BPE tokenizers also pass it. The previous Multrenizer artifact did **not** (it dropped utility tokens via `skip_special_tokens=True`); fixing that is one of the main reasons this version exists.
+
+### Morphological Splitting
+
+Total tokens needed to represent ten difficult Turkish words:
+
+| Tokenizer | Vocab Size | Total Tokens | Avg per Word |
+|---|---:|:---:|:---:|
+| Kumru-2B | 50,176 | **35** | **3.5** |
+| **Multrenizer** | **25,679** | **36** | **3.6** |
+| Turkcell-7B | 48,351 | 38 | 3.8 |
+| Mistral-3.1 | 131,072 | 71 | 7.1 |
+| Qwen-3 | 151,643 | 73 | 7.3 |
+| GPT-2 | 50,257 | 105 | 10.5 |
+
+A few selected words:
+
+```text
+İstanbul'da
+  Multrenizer  [1 tok]   İstanbul'da                     ← single token, atomic
+  Kumru-2B     [3 tok]   İstanbul + ' + da
+  Turkcell-7B  [3 tok]   İstanbul + ' + da
+  GPT-2        [5 tok]   Ä + ° + stanbul + 'd + a
+  Qwen-3       [4 tok]   İ + stanbul + 'd + a
+  Mistral-3.1  [4 tok]   İ + stanbul + 'd + a
+
+Afyonkarahisarlılaştıramadıklarımızdan
+  Multrenizer  [8 tok]   Afyonkarahisar + lı + laştı + ram + a + dıkları + mızda + n
+  Kumru-2B     [8 tok]   Af + yonkarahisar + lı + laştır + ama + dık + larımız + dan
+  Turkcell-7B  [9 tok]   Afyon + kar + ah + is + arlı + laştır + a + madık + larımızdan
+  Qwen-3      [16 tok]   (16-piece byte-level fragmentation)
+  Mistral-3.1 [16 tok]   (16-piece byte-level fragmentation)
+  GPT-2       [21 tok]   (21-piece byte-level fragmentation)
+
+düşünülebileceğini
+  Multrenizer  [2 tok]   düşünül + ebileceğini
+  Kumru-2B     [3 tok]   düşünül + ebil + eceğini
+  Turkcell-7B  [4 tok]   düş + ünü + le + bileceğini
+  GPT-2       [12 tok]   d + ü + ş + ü + n + ü + le + b + ile + ce + ğ + ini
+```
+
+### Round-Trip Lossless Decode
+
+11 representative inputs covering case preservation, apostrophe handling, currency, math/measurement symbols, VS16 / ZWJ / regional-indicator emoji, and mixed-content sentences. Multrenizer passes 11/11. All compared modern tokenizers also pass — the value here is being on the right side of the bar (the previous Multrenizer artifact wasn't), not winning a race.
+
+```text
+Input                                                                Multrenizer  ...
+İstanbul'da güzel bir gün geçirdim.                                    OK
+TÜRKİYE büyük harflerle yazılır.                                       OK
+It's a beautiful day, isn't it?                                        OK
+Fiyat: 100₺ ve $50, €30 ve £25.                                        OK
+Türkiye'de %50 indirim ve %25 KDV.                                     OK
+Hava 22°C, yağış %30, basınç 1013hPa.                                  OK
+❤️ ile 💔 farklı duygular ifade eder.                                   OK
+🇹🇷 ve 🇺🇸 bayrakları yan yana asıldı.                                   OK
+Geliştirici 👨‍💻 olarak çalışıyorum.                                    OK
+Selam 👋! Türkiye'de %50 indirim 100₺.                                  OK
+```
+
+### Code-Switching Tokenization
+
+```text
+"Bu feature'ı implement ederken edge case'leri handle etmeyi unutmayalım."
+
+  Multrenizer  [12 tok]  Bu | feature | 'ı | implement | ederken | edge | case | 'leri | handle | etmeyi | unutmay | alım.
+  Kumru-2B     [20 tok]  Bu | fe | ature | ' | ı | imp | lement | ederken | ed | ge | cas | e | ' | leri | hand | le | etmeyi | unutma | yalım | .
+  Turkcell-7B  [15 tok]  Bu | feature | ' | ı | implement | ederken | edge | case | ' | leri | handle | etmeyi | unut | mayalım | .
+
+"merge'lemek istediğim branch conflict veriyor."
+
+  Multrenizer  [ 7 tok]  merge'le | mek | istediğim | branch | conflict | veriyor | .
+  Kumru-2B     [14 tok]  mer | ge | ' | lemek | istediğim | b | ran | ch | con | f | lic | t | veriyor | .
+  Turkcell-7B  [ 8 tok]  merge | ' | lemek | istediğim | branch | conflict | veriyor | .
+  GPT-2        [16 tok]  mer | ge | ' | lem | ek | is | ted | i | ğ | im | branch | conflict | ver | iy | or | .
+```
+
+> Full benchmark output (all sections, all sentences, machine-readable) is in `benchmark_results.json`. Regenerate with `python benchmark.py`.
 
 ## Project Structure
 
@@ -317,11 +320,14 @@ multrenizer/
 │   ├── tokenizer.json
 │   ├── tokenizer_config.json
 │   └── special_tokens_map.json
-├── prepare_data.py            # Corpus download and preparation
+├── prepare_data.py            # Corpus download (Wikipedia + OPUS-100 + synthetic CS)
 ├── train_tokenizer.py         # Tokenizer training script
-├── benchmark.py               # Benchmark against 5 reference tokenizers
+├── benchmark.py               # Benchmark vs 5 reference tokenizers
 ├── benchmark_results.json     # Full benchmark output
-├── tests/                     # Regression tests for tokenizer behavior
+├── tests/
+│   └── test_tokenizer_correctness.py   # Round-trip + utility + ID-stability suite
+├── RETRAIN_PLAN.md            # Design / pivot history (audit trail)
+├── MINDAI_INTEGRATION.md      # Downstream integration recipe
 ├── requirements.txt
 └── pyproject.toml
 ```