Update README.md
Browse files
README.md
CHANGED
|
@@ -40,15 +40,8 @@ tokens per word indicates more efficient tokenization.
|
|
| 40 |
|
| 41 |

|
| 42 |
|
| 43 |
-
| Benchmark | SamAI tokens/word | Best tokenizer | Best tokens/word | SamAI rank |
|
| 44 |
-
| --- | ---: | --- | ---: | ---: |
|
| 45 |
-
| Arabic Dialects | **1.3043** | ALLaM-7B | 1.2812 | **2** |
|
| 46 |
-
| ArabicMMLU | **2.4838** | ALLaM-7B | 1.9943 | **5** |
|
| 47 |
-
| MASSIVE Arabic | **1.2549** | ALLaM-7B | 1.1939 | **2** |
|
| 48 |
-
| MLQA Arabic | **1.4397** | **SamAI-Tokenizer** | **1.4397** | **1** |
|
| 49 |
-
|
| 50 |
Across the equal-weight macro average of the four benchmarks, SamAI-Tokenizer
|
| 51 |
-
ranked second among 15 successfully tested tokenizers at **1.6207 tokens per
|
| 52 |
word**.
|
| 53 |
|
| 54 |
| Rank | Tokenizer | Mean tokens/word | Extra tokens versus SamAI |
|
|
|
|
| 40 |
|
| 41 |

|
| 42 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 43 |
Across the equal-weight macro average of the four benchmarks, SamAI-Tokenizer
|
| 44 |
+
ranked **second** among 15 successfully tested tokenizers at **1.6207 tokens per
|
| 45 |
word**.
|
| 46 |
|
| 47 |
| Rank | Tokenizer | Mean tokens/word | Extra tokens versus SamAI |
|