Samai-sa commited on
Commit
f4188ba
·
verified ·
1 Parent(s): 87c35fe

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +40 -39
README.md CHANGED
@@ -38,6 +38,46 @@ benchmark documents.
38
  | Primary language | Arabic |
39
  | Secondary language | English |
40
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
41
  ## Installation
42
 
43
  ```bash
@@ -107,45 +147,6 @@ dialects, Arabizi, English, code, mathematics, and Unicode edge cases.
107
  See [`reports/validation.json`](reports/validation.json) and
108
  [`reports/stress.json`](reports/stress.json).
109
 
110
- ## Arabic tokenizer benchmark
111
-
112
- SamAI-Tokenizer was evaluated on four complementary Arabic benchmarks. Lower
113
- tokens per word indicates more efficient tokenization.
114
-
115
- ![SamAI tokenizer benchmark results](assets/SamAI-Tokenizer-Benchmark.png)
116
-
117
- | Benchmark | SamAI tokens/word | Best tokenizer | Best tokens/word | SamAI rank |
118
- | --- | ---: | --- | ---: | ---: |
119
- | Arabic Dialects | **1.3043** | ALLaM-7B | 1.2812 | **2** |
120
- | ArabicMMLU | **2.4838** | ALLaM-7B | 1.9943 | **5** |
121
- | MASSIVE Arabic | **1.2549** | ALLaM-7B | 1.1939 | **2** |
122
- | MLQA Arabic | **1.4397** | **SamAI-Tokenizer** | **1.4397** | **1** |
123
-
124
- Across the equal-weight macro average of the four benchmarks, SamAI-Tokenizer
125
- ranked second among 15 successfully tested tokenizers at **1.6207 tokens per
126
- word**.
127
-
128
- | Rank | Tokenizer | Mean tokens/word | Extra tokens versus SamAI |
129
- | ---: | --- | ---: | ---: |
130
- | 1 | ALLaM-7B | 1.5021 | -4.86% |
131
- | **2** | **SamAI-Tokenizer** | **1.6207** | **baseline** |
132
- | 3 | Falcon-H1-7B | 1.8880 | +19.30% |
133
- | 4 | Mistral-Small-3.1 | 1.8929 | +20.71% |
134
- | 5 | Qwen3.8-Flash-Next | 1.9039 | +21.94% |
135
- | 6 | Qwen3.5-4B | 1.9039 | +21.94% |
136
- | 7 | tiktoken-o200k | 1.9919 | +26.37% |
137
- | 8 | GPT-OSS-20B | 1.9919 | +26.37% |
138
- | 9 | Fanar-1 | 2.1570 | +36.81% |
139
- | 10 | DeepSeek-V3 | 2.2774 | +46.25% |
140
- | 11 | Qwen2.5-7B | 2.3740 | +50.25% |
141
- | 12 | Qwen3-8B | 2.3740 | +50.25% |
142
- | 13 | GLM-5.3-Flash | 2.5225 | +60.21% |
143
- | 14 | Kimi-K2.6 | 2.8693 | +84.13% |
144
- | 15 | tiktoken-cl100k | 4.0800 | +164.63% |
145
-
146
- Complete aggregate results are available in [`results/`](results/). Row-level
147
- outputs are intentionally excluded because they reproduce third-party text.
148
-
149
  ## Benchmark methodology
150
 
151
  - All available examples were used, followed by exact-text deduplication within
 
38
  | Primary language | Arabic |
39
  | Secondary language | English |
40
 
41
+ ## Arabic tokenizer benchmark
42
+
43
+ SamAI-Tokenizer was evaluated on four complementary Arabic benchmarks. Lower
44
+ tokens per word indicates more efficient tokenization.
45
+
46
+ ![SamAI tokenizer benchmark results](assets/SamAI-Tokenizer-Benchmark.png)
47
+
48
+ | Benchmark | SamAI tokens/word | Best tokenizer | Best tokens/word | SamAI rank |
49
+ | --- | ---: | --- | ---: | ---: |
50
+ | Arabic Dialects | **1.3043** | ALLaM-7B | 1.2812 | **2** |
51
+ | ArabicMMLU | **2.4838** | ALLaM-7B | 1.9943 | **5** |
52
+ | MASSIVE Arabic | **1.2549** | ALLaM-7B | 1.1939 | **2** |
53
+ | MLQA Arabic | **1.4397** | **SamAI-Tokenizer** | **1.4397** | **1** |
54
+
55
+ Across the equal-weight macro average of the four benchmarks, SamAI-Tokenizer
56
+ ranked second among 15 successfully tested tokenizers at **1.6207 tokens per
57
+ word**.
58
+
59
+ | Rank | Tokenizer | Mean tokens/word | Extra tokens versus SamAI |
60
+ | ---: | --- | ---: | ---: |
61
+ | 1 | ALLaM-7B | 1.5021 | -4.86% |
62
+ | **2** | **SamAI-Tokenizer** | **1.6207** | **baseline** |
63
+ | 3 | Falcon-H1-7B | 1.8880 | +19.30% |
64
+ | 4 | Mistral-Small-3.1 | 1.8929 | +20.71% |
65
+ | 5 | Qwen3.8-Flash-Next | 1.9039 | +21.94% |
66
+ | 6 | Qwen3.5-4B | 1.9039 | +21.94% |
67
+ | 7 | tiktoken-o200k | 1.9919 | +26.37% |
68
+ | 8 | GPT-OSS-20B | 1.9919 | +26.37% |
69
+ | 9 | Fanar-1 | 2.1570 | +36.81% |
70
+ | 10 | DeepSeek-V3 | 2.2774 | +46.25% |
71
+ | 11 | Qwen2.5-7B | 2.3740 | +50.25% |
72
+ | 12 | Qwen3-8B | 2.3740 | +50.25% |
73
+ | 13 | GLM-5.3-Flash | 2.5225 | +60.21% |
74
+ | 14 | Kimi-K2.6 | 2.8693 | +84.13% |
75
+ | 15 | tiktoken-cl100k | 4.0800 | +164.63% |
76
+
77
+ Complete aggregate results are available in [`results/`](results/). Row-level
78
+ outputs are intentionally excluded because they reproduce third-party text.
79
+
80
+
81
  ## Installation
82
 
83
  ```bash
 
147
  See [`reports/validation.json`](reports/validation.json) and
148
  [`reports/stress.json`](reports/stress.json).
149
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
150
  ## Benchmark methodology
151
 
152
  - All available examples were used, followed by exact-text deduplication within