Upload README.md with huggingface_hub

README.md

@@ -30,7 +30,7 @@ size_categories:
 
 The first systematic subword tokenizer benchmark for **Moroccan Darija**, a low-resource dialect written concurrently in Arabic script and Arabizi (Latin script). We train and evaluate **40 tokenizer configurations** spanning four algorithms, two architectures, and five vocabulary sizes (8K--110K) on 112,814 parallel sentence pairs from [`OiQ/daa-pairs`](https://huggingface.co/datasets/OiQ/daa-pairs).
 
-Our tokenizers achieve **27--34% lower fertility** than existing Darija tokenizers (DarijaBERT) at matching vocabulary sizes, and **40--50% lower fertility** than MSA-trained tokenizers. All 40 configurations maintain $\geq$99% exact reconstruction.
+Our tokenizers achieve **27--33% lower fertility** than existing Darija tokenizers (DarijaBERT) at matching vocabulary sizes, and **40--50% lower fertility** than MSA-trained tokenizers. All 40 configurations maintain ≥99% exact reconstruction.
 
 ---
 
@@ -39,7 +39,7 @@ Our tokenizers achieve **27--34% lower fertility** than existing Darija tokenize
 | Architecture | Description | Algorithms | Vocab Sizes | Count |
 |---|---|---|---|---|
 | **Shared** | Single vocabulary trained on mixed Arabic + Arabizi corpus | BPE, Unigram, WordPiece, BBPE | 8K, 16K, 32K, 80K, 110K | 20 |
-| **Concatenated** | Separate per-script vocabularies ($V/2$ each) with ID shifting | BPE, Unigram, WordPiece, BBPE | 8K, 16K, 32K, 80K, 110K | 20 |
+| **Concatenated** | Separate per-script vocabularies (V/2 each) with ID shifting | BPE, Unigram, WordPiece, BBPE | 8K, 16K, 32K, 80K, 110K | 20 |
 
 All tokenizers are released in both raw (HuggingFace `tokenizers`) and `transformers`-compatible formats. The 80K and 110K sizes match DarijaBERT's vocabulary sizes for direct comparison.
 
@@ -80,7 +80,7 @@ print(tok_az.encode(text_az))
 
 ### Best Tokenizer per Vocabulary Size
 
-| Vocab | Configuration | Algorithm | Fertility $\downarrow$ | Disparity $\downarrow$ | Exact Match |
+| Vocab | Configuration | Algorithm | Fertility ↓ | Disparity ↓ | Exact Match |
 |---|---|---|---|---|---|
 | 8K | Shared | WordPiece | **1.572** | 0.164 | 99.9% |
 | 16K | Shared | WordPiece | **1.402** | 0.138 | 99.9% |
@@ -90,7 +90,7 @@ print(tok_az.encode(text_az))
 
 ### Comparison with Existing Tokenizers
 
-| Tokenizer | Vocab | Fertility $\downarrow$ | Disparity $\downarrow$ | EM (Ar) | EM (Az) |
+| Tokenizer | Vocab | Fertility ↓ | Disparity ↓ | EM (Ar) | EM (Az) |
 |---|---|---|---|---|---|
 | **Ours: concat WP 110K** | **110K** | **1.155** | 0.093 | **99.9%** | **99.6%** |
 | **Ours: concat WP 80K** | **80K** | **1.183** | 0.090 | **99.9%** | **99.6%** |
@@ -111,24 +111,24 @@ At matching vocabulary sizes, our 80K tokenizer achieves **33% lower fertility**
 
 | Metric | Definition | Direction |
 |---|---|---|
-| **Fertility** ($F$) | Tokens per word, averaged over test set | Lower is better |
+| **Fertility** (F) | Tokens per word, averaged over test set | Lower is better |
 | **CPT** | Grapheme clusters per token (Unicode-aware) | Higher is better |
-| **Disparity** ($\Delta F$) | Relative cross-script gap: $\|F_{ar} - F_{az}\| / \max(F_{ar}, F_{az})$ | Lower is better |
+| **Disparity** (ΔF) | Relative cross-script gap: \|F<sub>ar</sub> − F<sub>az</sub>\| / max(F<sub>ar</sub>, F<sub>az</sub>) | Lower is better |
 | **Exact Match** | Fraction of texts that round-trip perfectly through encode/decode | Higher is better |
 | **Gini** | Vocabulary usage inequality (0 = uniform, 1 = concentrated) | Lower is better |
 
-Overall fertility is word-count-weighted: $F \approx 0.65 \cdot F_{ar} + 0.35 \cdot F_{az}$
+Overall fertility is word-count-weighted: F ≈ 0.65·F<sub>ar</sub> + 0.35·F<sub>az</sub>
 
 ### Morphological Fidelity (Arabic-script only)
 
 | Metric | Definition | Direction |
 |---|---|---|
-| **$\mu_e$** | Edit distance between tokenizer boundaries and Farasa morpheme boundaries | Lower is better |
-| **$\mu_c$-F1** | Whether words sharing morphemes also share tokens (KMeans + TF-IDF) | Higher is better |
+| **μ<sub>e</sub>** | Edit distance between tokenizer boundaries and Farasa morpheme boundaries | Lower is better |
+| **μ<sub>c</sub>-F1** | Whether words sharing morphemes also share tokens (KMeans + TF-IDF) | Higher is better |
 
 ### Statistical Rigor
 
-All fertility and CPT values include **bootstrap 95% confidence intervals** (500 resamples). CI width is $\leq 0.006$ for all configurations.
+All fertility and CPT values include **bootstrap 95% confidence intervals** (500 resamples). CI width is ≤ 0.006 for all configurations.
 
 ---
 
@@ -147,13 +147,13 @@ All fertility and CPT values include **bootstrap 95% confidence intervals** (500
 
 ## Key Findings
 
-1. **Darija-specific tokenization dramatically improves compression.** Our tokenizers achieve 27--34% lower fertility than DarijaBERT at matching vocabulary sizes.
+1. **Darija-specific tokenization dramatically improves compression.** Our tokenizers achieve 27--33% lower fertility than DarijaBERT at matching vocabulary sizes.
 
-2. **Cross-script fairness is achievable.** Shared Unigram at 80K reaches $\Delta F = 0.015$ (near-zero disparity) without concatenation. Concatenated architectures maintain $\Delta F \leq 0.094$ across all non-BBPE algorithms.
+2. **Cross-script fairness is achievable.** Shared Unigram at 80K reaches ΔF = 0.015 (near-zero disparity) without concatenation. Concatenated architectures maintain ΔF ≤ 0.094 across all non-BBPE algorithms.
 
 3. **Vocabulary size saturates early.** Moving from 32K to 110K yields only marginal gains. The training corpus exhausts merge candidates around 80K.
 
-4. **BBPE requires concatenation.** Shared BBPE exhibits extreme cross-script disparity ($\Delta F = 0.219$--$0.243$). Concatenation reduces this 4x.
+4. **BBPE requires concatenation.** Shared BBPE exhibits extreme cross-script disparity (ΔF = 0.219–0.243). Concatenation reduces this 4x.
 
 5. **MSA tokenizers transfer poorly to Darija.** They produce 40--50% higher fertility, allocating vocabulary to MSA patterns absent in Darija.