Ouaill commited on
Commit
b7fbcec
·
verified ·
1 Parent(s): 3499545

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +2 -1
README.md CHANGED
@@ -97,11 +97,12 @@ print(tok_az.encode(text_az))
97
  | **Ours: concat BPE 32K** | **32K** | **1.307** | 0.084 | **99.9%** | **99.6%** |
98
  | DarijaBERT-ar | 80K | 1.761 | 0.410 | 13.7% | 8.0% |
99
  | DarijaBERT-az | 110K | 1.575 | 0.055 | 14.8% | 8.0% |
 
100
  | CaMeLBERT-MSA | 30K | 2.289 | 0.427 | 29.9% | 38.9% |
101
  | Aranizer-SP-86k | 86K | 1.918 | 0.368 | 99.8% | 99.6% |
102
  | Qwen2.5-Darija | 152K | 2.307 | 0.040 | 100.0% | 100.0% |
103
 
104
- At matching vocabulary sizes, our 80K tokenizer achieves **33% lower fertility** than DarijaBERT-ar (1.183 vs 1.761). Our 110K achieves **27% lower** than DarijaBERT-az (1.155 vs 1.575). Even our 32K tokenizer outperforms DarijaBERT-az despite using 3.4x fewer vocabulary slots.
105
 
106
  ---
107
 
 
97
  | **Ours: concat BPE 32K** | **32K** | **1.307** | 0.084 | **99.9%** | **99.6%** |
98
  | DarijaBERT-ar | 80K | 1.761 | 0.410 | 13.7% | 8.0% |
99
  | DarijaBERT-az | 110K | 1.575 | 0.055 | 14.8% | 8.0% |
100
+ | DarijaBERT-mix | 160K | 1.414 | 0.149 | 14.8% | 8.0% |
101
  | CaMeLBERT-MSA | 30K | 2.289 | 0.427 | 29.9% | 38.9% |
102
  | Aranizer-SP-86k | 86K | 1.918 | 0.368 | 99.8% | 99.6% |
103
  | Qwen2.5-Darija | 152K | 2.307 | 0.040 | 100.0% | 100.0% |
104
 
105
+ At matching vocabulary sizes, our 80K tokenizer achieves **33% lower fertility** than DarijaBERT-ar (1.183 vs 1.761). Our 110K achieves **27% lower** than DarijaBERT-az (1.155 vs 1.575). Even our 32K tokenizer outperforms DarijaBERT-az despite using 3.4x fewer vocabulary slots. DarijaBERT-mix, despite its massive 160K vocabulary (F = 1.414), still underperforms our 32K tokenizer—vocabulary size alone cannot compensate for suboptimal training architecture.
106
 
107
  ---
108