Update README.md

README.md

@@ -82,33 +82,37 @@ print(text)
 
 ### Tokenizer Benchmark Results
 
-Comparison with state-of-the-art tokenizers (5 runs, 5000 samples each).
-
-**Dataset used:** [almaghrabima/deeplatent-benchmark-data](https://huggingface.co/datasets/almaghrabima/deeplatent-benchmark-data) (60k samples: 30k Arabic + 30k English)
-
-| Tokenizer | Vocab | AR Fert | EN Fert | Avg Fert | Parity | Fert Rank | Parity Rank |
-|-----------|-------|---------|---------|----------|--------|-----------|-------------|
-| **SARFTokenizer** | 64,641 | 1.71 | 1.57 | **1.64** | 1.155 | **#1** | #2 |
-| ALLaM-7B | 64,000 | 1.81 | 1.48 | 1.65 | 1.162 | #2 | #3 |
-| Falcon-H1-7B | 130,049 | 2.64 | 1.55 | 2.10 | **0.926** | #3 | **#1** |
-| Fanar-1-9B | 128,256 | 2.85 | 1.36 | 2.10 | 0.774 | #4 | #4 |
-| Hala-9B | 128,256 | 2.85 | 1.36 | 2.10 | 0.774 | #5 | #5 |
-| GPT-4o | 200,019 | 2.81 | 1.44 | 2.12 | 0.725 | #6 | #6 |
-| Command-R-Arabic | 255,033 | 3.00 | 1.33 | 2.16 | 0.713 | #7 | #7 |
-| Qwen3-4B | 151,669 | 3.05 | 1.50 | 2.28 | 0.696 | #8 | #8 |
-| GPT-4 | 100,277 | 4.59 | 1.50 | 3.05 | 0.416 | #9 | #10 |
-| Mistral-7B-v0.3 | 32,768 | 5.56 | 1.48 | 3.52 | 0.417 | #10 | #9 |
+Comparison with state-of-the-art tokenizers on 60,000 samples (30k Arabic + 30k English).
+
+**Dataset:** [almaghrabima/deeplatent-benchmark-data](https://huggingface.co/datasets/almaghrabima/deeplatent-benchmark-data)
+
+| Tokenizer | Vocab | AR Fert | EN Fert | Avg Fert | AR C/T | EN C/T | Parity |
+|-----------|-------|---------|---------|----------|--------|--------|--------|
+| **SARFTokenizer** | 64,641 | **1.72** | 1.57 | **1.64** | 3.45 | 2.99 | 1.156 |
+| ALLaM-7B | 64,000 | 1.82 | 1.48 | 1.65 | 3.08 | 2.65 | 1.163 |
+| Gemma-3-4B | 262,145 | 2.78 | 1.33 | 2.05 | 2.42 | 3.00 | 0.805 |
+| Falcon-H1-7B | 130,049 | 2.65 | 1.55 | 2.10 | 2.55 | 2.75 | **0.926** |
+| Fanar-1-9B | 128,256 | 2.85 | 1.36 | 2.11 | 2.27 | 2.93 | 0.775 |
+| Hala-9B | 128,256 | 2.85 | 1.36 | 2.11 | 2.27 | 2.93 | 0.775 |
+| GPT-4o | 200,019 | 2.81 | 1.44 | 2.12 | 2.45 | 3.37 | 0.726 |
+| Command-R-Arabic | 255,033 | 3.00 | 1.33 | 2.16 | 2.17 | 3.04 | 0.714 |
+| Qwen3-4B | 151,669 | 3.06 | 1.50 | 2.28 | 2.04 | 2.92 | 0.697 |
+| GPT-4 | 100,277 | 4.59 | 1.50 | 3.05 | 1.35 | 3.24 | 0.417 |
+| Mistral-7B-v0.3 | 32,768 | 5.56 | 1.48 | 3.52 | 1.11 | 2.64 | 0.418 |
 
 **Metrics explained:**
-- **Fertility**: Average tokens per word (lower is better)
+- **Fertility**: Average tokens per word (lower is better - more efficient encoding)
+- **C/T**: Characters per token (higher is better - more characters encoded per token)
 - **Parity**: AR chars/token ÷ EN chars/token (1.0 = equal treatment of both languages)
 
 **Key findings:**
-- **SARFTokenizer ranks #1 in fertility** (1.64 avg tokens/word) and #2 in parity (1.155)
-- **Falcon-H1-7B has best parity** (0.926) but lower fertility efficiency
-- **SARFTokenizer achieves best Arabic fertility** (1.71 tokens/word vs 2.6+ for others)
-- Morpheme-aware encoding significantly improves Arabic tokenization efficiency
-- SARFTokenizer uses smallest vocab (64k) among top performers
+- **SARFTokenizer achieves best Arabic fertility** (1.72 tokens/word) - 35% better than GPT-4o
+- **Lowest average fertility** (1.64) among all tokenizers tested
+- **Best Arabic characters/token** (3.45) - encodes more Arabic per token than any competitor
+- Compact vocabulary (64k) while maintaining top performance
+- ALLaM-7B shows similar efficiency (both use morpheme-aware approaches)
+- Falcon-H1-7B has best parity (0.926) but 28% higher fertility than SARF
+- GPT-4 and Mistral struggle with Arabic (4.6-5.6 tokens/word vs 1.7 for SARF)
 
 ### Throughput Benchmark (1M samples, 680 MB)
 
@@ -195,4 +199,4 @@ CC-BY-NC-4.0
   url={https://huggingface.co/almaghrabima/SARFTokenizer},
   note={Independent research, part of Suhail Project}
 }
-```
+```