Update README.md
Browse files
README.md
CHANGED
|
@@ -38,6 +38,46 @@ benchmark documents.
|
|
| 38 |
| Primary language | Arabic |
|
| 39 |
| Secondary language | English |
|
| 40 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
## Installation
|
| 42 |
|
| 43 |
```bash
|
|
@@ -107,45 +147,6 @@ dialects, Arabizi, English, code, mathematics, and Unicode edge cases.
|
|
| 107 |
See [`reports/validation.json`](reports/validation.json) and
|
| 108 |
[`reports/stress.json`](reports/stress.json).
|
| 109 |
|
| 110 |
-
## Arabic tokenizer benchmark
|
| 111 |
-
|
| 112 |
-
SamAI-Tokenizer was evaluated on four complementary Arabic benchmarks. Lower
|
| 113 |
-
tokens per word indicates more efficient tokenization.
|
| 114 |
-
|
| 115 |
-

|
| 116 |
-
|
| 117 |
-
| Benchmark | SamAI tokens/word | Best tokenizer | Best tokens/word | SamAI rank |
|
| 118 |
-
| --- | ---: | --- | ---: | ---: |
|
| 119 |
-
| Arabic Dialects | **1.3043** | ALLaM-7B | 1.2812 | **2** |
|
| 120 |
-
| ArabicMMLU | **2.4838** | ALLaM-7B | 1.9943 | **5** |
|
| 121 |
-
| MASSIVE Arabic | **1.2549** | ALLaM-7B | 1.1939 | **2** |
|
| 122 |
-
| MLQA Arabic | **1.4397** | **SamAI-Tokenizer** | **1.4397** | **1** |
|
| 123 |
-
|
| 124 |
-
Across the equal-weight macro average of the four benchmarks, SamAI-Tokenizer
|
| 125 |
-
ranked second among 15 successfully tested tokenizers at **1.6207 tokens per
|
| 126 |
-
word**.
|
| 127 |
-
|
| 128 |
-
| Rank | Tokenizer | Mean tokens/word | Extra tokens versus SamAI |
|
| 129 |
-
| ---: | --- | ---: | ---: |
|
| 130 |
-
| 1 | ALLaM-7B | 1.5021 | -4.86% |
|
| 131 |
-
| **2** | **SamAI-Tokenizer** | **1.6207** | **baseline** |
|
| 132 |
-
| 3 | Falcon-H1-7B | 1.8880 | +19.30% |
|
| 133 |
-
| 4 | Mistral-Small-3.1 | 1.8929 | +20.71% |
|
| 134 |
-
| 5 | Qwen3.8-Flash-Next | 1.9039 | +21.94% |
|
| 135 |
-
| 6 | Qwen3.5-4B | 1.9039 | +21.94% |
|
| 136 |
-
| 7 | tiktoken-o200k | 1.9919 | +26.37% |
|
| 137 |
-
| 8 | GPT-OSS-20B | 1.9919 | +26.37% |
|
| 138 |
-
| 9 | Fanar-1 | 2.1570 | +36.81% |
|
| 139 |
-
| 10 | DeepSeek-V3 | 2.2774 | +46.25% |
|
| 140 |
-
| 11 | Qwen2.5-7B | 2.3740 | +50.25% |
|
| 141 |
-
| 12 | Qwen3-8B | 2.3740 | +50.25% |
|
| 142 |
-
| 13 | GLM-5.3-Flash | 2.5225 | +60.21% |
|
| 143 |
-
| 14 | Kimi-K2.6 | 2.8693 | +84.13% |
|
| 144 |
-
| 15 | tiktoken-cl100k | 4.0800 | +164.63% |
|
| 145 |
-
|
| 146 |
-
Complete aggregate results are available in [`results/`](results/). Row-level
|
| 147 |
-
outputs are intentionally excluded because they reproduce third-party text.
|
| 148 |
-
|
| 149 |
## Benchmark methodology
|
| 150 |
|
| 151 |
- All available examples were used, followed by exact-text deduplication within
|
|
|
|
| 38 |
| Primary language | Arabic |
|
| 39 |
| Secondary language | English |
|
| 40 |
|
| 41 |
+
## Arabic tokenizer benchmark
|
| 42 |
+
|
| 43 |
+
SamAI-Tokenizer was evaluated on four complementary Arabic benchmarks. Lower
|
| 44 |
+
tokens per word indicates more efficient tokenization.
|
| 45 |
+
|
| 46 |
+

|
| 47 |
+
|
| 48 |
+
| Benchmark | SamAI tokens/word | Best tokenizer | Best tokens/word | SamAI rank |
|
| 49 |
+
| --- | ---: | --- | ---: | ---: |
|
| 50 |
+
| Arabic Dialects | **1.3043** | ALLaM-7B | 1.2812 | **2** |
|
| 51 |
+
| ArabicMMLU | **2.4838** | ALLaM-7B | 1.9943 | **5** |
|
| 52 |
+
| MASSIVE Arabic | **1.2549** | ALLaM-7B | 1.1939 | **2** |
|
| 53 |
+
| MLQA Arabic | **1.4397** | **SamAI-Tokenizer** | **1.4397** | **1** |
|
| 54 |
+
|
| 55 |
+
Across the equal-weight macro average of the four benchmarks, SamAI-Tokenizer
|
| 56 |
+
ranked second among 15 successfully tested tokenizers at **1.6207 tokens per
|
| 57 |
+
word**.
|
| 58 |
+
|
| 59 |
+
| Rank | Tokenizer | Mean tokens/word | Extra tokens versus SamAI |
|
| 60 |
+
| ---: | --- | ---: | ---: |
|
| 61 |
+
| 1 | ALLaM-7B | 1.5021 | -4.86% |
|
| 62 |
+
| **2** | **SamAI-Tokenizer** | **1.6207** | **baseline** |
|
| 63 |
+
| 3 | Falcon-H1-7B | 1.8880 | +19.30% |
|
| 64 |
+
| 4 | Mistral-Small-3.1 | 1.8929 | +20.71% |
|
| 65 |
+
| 5 | Qwen3.8-Flash-Next | 1.9039 | +21.94% |
|
| 66 |
+
| 6 | Qwen3.5-4B | 1.9039 | +21.94% |
|
| 67 |
+
| 7 | tiktoken-o200k | 1.9919 | +26.37% |
|
| 68 |
+
| 8 | GPT-OSS-20B | 1.9919 | +26.37% |
|
| 69 |
+
| 9 | Fanar-1 | 2.1570 | +36.81% |
|
| 70 |
+
| 10 | DeepSeek-V3 | 2.2774 | +46.25% |
|
| 71 |
+
| 11 | Qwen2.5-7B | 2.3740 | +50.25% |
|
| 72 |
+
| 12 | Qwen3-8B | 2.3740 | +50.25% |
|
| 73 |
+
| 13 | GLM-5.3-Flash | 2.5225 | +60.21% |
|
| 74 |
+
| 14 | Kimi-K2.6 | 2.8693 | +84.13% |
|
| 75 |
+
| 15 | tiktoken-cl100k | 4.0800 | +164.63% |
|
| 76 |
+
|
| 77 |
+
Complete aggregate results are available in [`results/`](results/). Row-level
|
| 78 |
+
outputs are intentionally excluded because they reproduce third-party text.
|
| 79 |
+
|
| 80 |
+
|
| 81 |
## Installation
|
| 82 |
|
| 83 |
```bash
|
|
|
|
| 147 |
See [`reports/validation.json`](reports/validation.json) and
|
| 148 |
[`reports/stress.json`](reports/stress.json).
|
| 149 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 150 |
## Benchmark methodology
|
| 151 |
|
| 152 |
- All available examples were used, followed by exact-text deduplication within
|