Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -97,11 +97,12 @@ print(tok_az.encode(text_az))
|
|
| 97 |
| **Ours: concat BPE 32K** | **32K** | **1.307** | 0.084 | **99.9%** | **99.6%** |
|
| 98 |
| DarijaBERT-ar | 80K | 1.761 | 0.410 | 13.7% | 8.0% |
|
| 99 |
| DarijaBERT-az | 110K | 1.575 | 0.055 | 14.8% | 8.0% |
|
|
|
|
| 100 |
| CaMeLBERT-MSA | 30K | 2.289 | 0.427 | 29.9% | 38.9% |
|
| 101 |
| Aranizer-SP-86k | 86K | 1.918 | 0.368 | 99.8% | 99.6% |
|
| 102 |
| Qwen2.5-Darija | 152K | 2.307 | 0.040 | 100.0% | 100.0% |
|
| 103 |
|
| 104 |
-
At matching vocabulary sizes, our 80K tokenizer achieves **33% lower fertility** than DarijaBERT-ar (1.183 vs 1.761). Our 110K achieves **27% lower** than DarijaBERT-az (1.155 vs 1.575). Even our 32K tokenizer outperforms DarijaBERT-az despite using 3.4x fewer vocabulary slots.
|
| 105 |
|
| 106 |
---
|
| 107 |
|
|
|
|
| 97 |
| **Ours: concat BPE 32K** | **32K** | **1.307** | 0.084 | **99.9%** | **99.6%** |
|
| 98 |
| DarijaBERT-ar | 80K | 1.761 | 0.410 | 13.7% | 8.0% |
|
| 99 |
| DarijaBERT-az | 110K | 1.575 | 0.055 | 14.8% | 8.0% |
|
| 100 |
+
| DarijaBERT-mix | 160K | 1.414 | 0.149 | 14.8% | 8.0% |
|
| 101 |
| CaMeLBERT-MSA | 30K | 2.289 | 0.427 | 29.9% | 38.9% |
|
| 102 |
| Aranizer-SP-86k | 86K | 1.918 | 0.368 | 99.8% | 99.6% |
|
| 103 |
| Qwen2.5-Darija | 152K | 2.307 | 0.040 | 100.0% | 100.0% |
|
| 104 |
|
| 105 |
+
At matching vocabulary sizes, our 80K tokenizer achieves **33% lower fertility** than DarijaBERT-ar (1.183 vs 1.761). Our 110K achieves **27% lower** than DarijaBERT-az (1.155 vs 1.575). Even our 32K tokenizer outperforms DarijaBERT-az despite using 3.4x fewer vocabulary slots. DarijaBERT-mix, despite its massive 160K vocabulary (F = 1.414), still underperforms our 32K tokenizer—vocabulary size alone cannot compensate for suboptimal training architecture.
|
| 106 |
|
| 107 |
---
|
| 108 |
|