Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,192 @@
+---
+title: Darija Subword Tokenizer Benchmark
+license: mit
+tags:
+  - tokenizer
+  - moroccan-darija
+  - arabic
+  - bpe
+  - unigram
+  - wordpiece
+  - bbpe
+  - benchmark
+language:
+  - ar
+  - lat
+size_categories:
+  - n>100K
+---
+
+# Darija Subword Tokenizer Benchmark (`OiQ/daa-tokenizers`)
+
+Comprehensive benchmark of 24 subword tokenizers for **Moroccan Darija** (Arabic and Arabizi/Latin scripts), trained on the [`OiQ/daa-pairs`](https://huggingface.co/datasets/OiQ/daa-pairs) dataset (112,814 sentence pairs).
+
+## Tokenizers
+
+| Type | Algorithms | Vocab Sizes | Count |
+|------|-----------|-------------|-------|
+| **Shared** (single vocab for Arabic + Arabizi) | BPE, Unigram, WordPiece, BBPE | 8K, 16K, 32K | 12 |
+| **Concatenated** (separate Arabic/Arabizi vocabs, shifted IDs) | BPE, Unigram, WordPiece, BBPE | 8K, 16K, 32K | 12 |
+
+**Total: 24 tokenizers** (plus 28 exported in `transformers`-compatible format — 4 concat 32K tokenizers were not exported due to missing raw tokenizer files from an OOM-killed training run).
+
+## Quick Start
+
+```python
+from transformers import AutoTokenizer
+
+# Load a shared tokenizer
+tok = AutoTokenizer.from_pretrained("OiQ/daa-tokenizers", subfolder="transformers_tokenizers/shared_bpe_16000")
+
+# Load a concatenated tokenizer (Arabic)
+tok_ar = AutoTokenizer.from_pretrained("OiQ/daa-tokenizers", subfolder="transformers_tokenizers/concat_bpe_16000_tokenizer_ar")
+
+# Load a concatenated tokenizer (Arabizi/Latin)
+tok_az = AutoTokenizer.from_pretrained("OiQ/daa-tokenizers", subfolder="transformers_tokenizers/concat_bpe_16000_tokenizer_az")
+
+# Tokenize
+text_ar = "مابقاش كيعرف شنو يدير، بين القانون وبين وليداتو."
+text_az = "wash kayn shi jdid?"
+print(tok.encode(text_ar))
+print(tok_az.encode(text_az))
+```
+
+## Evaluation Metrics
+
+### Surface Metrics
+- **Fertility**: Average tokens per word (lower = more efficient)
+- **CPT**: Characters per token — grapheme-aware (higher = more efficient)
+- **OOV Rate**: Out-of-vocabulary rate (0.0 for all tokenizers)
+- **Cross-script Disparity**: |fertility_ar - fertility_az| / max(fertility_ar, fertility_az)
+- **Exact Match Rate**: Fraction of texts that round-trip perfectly
+- **Vocab Gini**: Vocabulary usage inequality (0 = uniform, 1 = concentrated)
+- **Shannon Entropy**: Token distribution entropy
+
+### Morphological Metrics (Arabic-script only)
+- **μe (Morphological Edit Distance)**: DP alignment between tokenizer token boundaries and Farasa morpheme boundaries. Lower = better morphological alignment.
+- **μc (Morphological Consistency F1)**: Whether shared morphemes (via KMeans clustering) are assigned shared tokens. Precision/Recall/F1.
+
+### Statistics
+All fertility and CPT metrics include **bootstrap 95% confidence intervals** (n=500 resamples).
+
+## Results Summary
+
+### Best Tokenizers by Vocabulary Size
+
+| Vocab Size | Name | Type | Algorithm | Fertility | Disparity | μe | μc F1 | Exact Match |
+|------------|------|------|-----------|-----------|-----------|----|----|-------------|
+| 8000 | concat_bpe_8000 | concatenated | BPE | 1.6890 | 0.1649 | 4.2158 | 0.1680 | 99.00% |
+| 16000 | concat_wordpiece_16000 | concatenated | WordPiece | 1.5162 | 0.1361 | 4.1506 | 0.1533 | 99.00% |
+| 32000 | concat_wordpiece_32000 | concatenated | WordPiece | 1.3885 | 0.1309 | 4.0975 | 0.1329 | 99.00% |
+
+### Full Results Table
+
+| Name | Type | Algorithm | Vocab | Fertility | CPT | Disparity | Exact Match | Gini | μe | μc F1 |
+|------|------|-----------|-------|-----------|-----|-----------|-------------|------|----|----|
+| shared_bpe_8000 | shared | BPE | 8000 | 1.7787 | 3.00 | 0.2199 | 99.00% | 0.7709 | 4.1755 | 0.1617 |
+| concat_bpe_8000 | concatenated | BPE | 8000 | 1.6890 | 3.16 | 0.1649 | 99.00% | 0.6850 | 4.2158 | 0.1680 |
+| shared_unigram_8000 | shared | Unigram | 8000 | 1.7790 | 3.00 | 0.1475 | 99.00% | 0.8189 | 4.1823 | 0.1770 |
+| concat_unigram_8000 | concatenated | Unigram | 8000 | 1.6948 | 3.15 | 0.1669 | 99.00% | 0.7623 | 4.2111 | 0.1801 |
+| shared_wordpiece_8000 | shared | WordPiece | 8000 | 1.7762 | 3.01 | 0.2268 | 99.00% | 0.7565 | 4.1766 | 0.1552 |
+| concat_wordpiece_8000 | concatenated | WordPiece | 8000 | 1.6962 | 3.15 | 0.1560 | 99.00% | 0.6600 | 4.2191 | 0.1619 |
+| shared_bbpe_8000 | shared | BBPE | 8000 | 1.9554 | 2.73 | 0.4718 | 99.00% | 0.8122 | 4.1921 | 0.1995 |
+| concat_bbpe_8000 | concatenated | BBPE | 8000 | 1.8828 | 2.84 | 0.1168 | 99.00% | 0.7491 | 4.2317 | 0.2136 |
+| shared_bpe_16000 | shared | BPE | 16000 | 1.5972 | 3.35 | 0.1511 | 99.00% | 0.7920 | 4.1165 | 0.1324 |
+| concat_bpe_16000 | concatenated | BPE | 16000 | 1.5166 | 3.52 | 0.1427 | 99.00% | 0.7225 | 4.1531 | 0.1376 |
+| shared_unigram_16000 | shared | Unigram | 16000 | 1.6213 | 3.30 | 0.0885 | 99.00% | 0.8335 | 4.1315 | 0.1623 |
+| concat_unigram_16000 | concatenated | Unigram | 16000 | 1.5417 | 3.47 | 0.1464 | 99.00% | 0.7918 | 4.1582 | 0.1575 |
+| shared_wordpiece_16000 | shared | WordPiece | 16000 | 1.5914 | 3.36 | 0.1512 | 99.00% | 0.7797 | 4.1158 | 0.1385 |
+| concat_wordpiece_16000 | concatenated | WordPiece | 16000 | 1.5162 | 3.53 | 0.1361 | 99.00% | 0.7024 | 4.1506 | 0.1533 |
+| shared_bbpe_16000 | shared | BBPE | 16000 | 1.7937 | 2.98 | 0.4265 | 99.00% | 0.8353 | 4.1365 | 0.1824 |
+| concat_bbpe_16000 | concatenated | BBPE | 16000 | 1.7340 | 3.08 | 0.1722 | 99.00% | 0.7908 | 4.1720 | 0.1986 |
+| shared_bpe_32000 | shared | BPE | 32000 | 1.4562 | 3.67 | 0.0727 | 99.00% | 0.7969 | 4.0706 | 0.1149 |
+| concat_bpe_32000 | concatenated | BPE | 32000 | 1.3912 | 3.84 | 0.1340 | 99.00% | 0.7568 | 4.0970 | 0.1146 |
+| shared_unigram_32000 | shared | Unigram | 32000 | 1.5034 | 3.56 | 0.0193 | 99.00% | 0.8325 | 4.0979 | 0.1482 |
+| concat_unigram_32000 | concatenated | Unigram | 32000 | 1.4304 | 3.74 | 0.1449 | 99.00% | 0.8069 | 4.1112 | 0.1420 |
+| shared_wordpiece_32000 | shared | WordPiece | 32000 | 1.4501 | 3.69 | 0.0724 | 99.00% | 0.7886 | 4.0696 | 0.1273 |
+| concat_wordpiece_32000 | concatenated | WordPiece | 32000 | 1.3885 | 3.85 | 0.1309 | 99.00% | 0.7419 | 4.0975 | 0.1329 |
+| shared_bbpe_32000 | shared | BBPE | 32000 | 1.6737 | 3.19 | 0.3737 | 99.00% | 0.8431 | 4.0974 | 0.1737 |
+| concat_bbpe_32000 | concatenated | BBPE | 32000 | 1.6299 | 3.28 | 0.2067 | 99.00% | 0.8217 | 4.1216 | 0.1798 |
+
+## Visualizations
+
+### Fertility Comparison
+![Fertility Comparison](plots/fertility_overall_comparison_v2.png)
+
+### Cross-Script Disparity
+![Disparity Comparison](plots/fertility_disparity_comparison_v2.png)
+
+### Script-Level Comparison
+![Script Comparison](plots/script_comparison_v2.png)
+
+### Bootstrap Confidence Intervals
+![Bootstrap CI Forest Plot](plots/bootstrap_ci_forest.png)
+
+### Morphological Edit Distance (μe)
+![Morph Edit Distance](plots/morph_edit_distance_ar_comparison_v2.png)
+
+### Morphological Consistency (μc F1)
+![Morph Consistency F1](plots/morph_consistency_f1_comparison_v2.png)
+
+### CPT Trends
+![CPT Trends](plots/cpt_overall_trends.png)
+
+### Fertility Heatmap
+![Fertility Heatmap](plots/fertility_overall_heatmap_v2.png)
+
+### Morph Edit Distance Heatmap
+![Morph Edit Distance Heatmap](plots/morph_edit_distance_ar_heatmap_v2.png)
+
+## Repository Structure
+
+```
+daa-tokenizers/
+├── README.md                          # This file
+├── script.py                          # Benchmark script (reproducible)
+├── tokenizer_results.csv              # Full metrics table
+├── tokenizer_results.json             # Full metrics (JSON)
+├── bootstrap_ci.csv                   # Bootstrap 95% CIs
+├── benchmark_report.md                # Detailed report
+├── morphology/
+│   └── farasa_segmentations.json      # Cached Farasa morph segmentations (~100MB)
+├── tokenizers/                        # Raw tokenizer JSON files (36 files)
+├── transformers_tokenizers/           # 28 dirs for transformers library
+│   ├── shared_bpe_8000/
+│   ├── shared_bpe_16000/
+│   ├── ...
+│   ├── concat_bpe_8000_tokenizer_ar/
+│   ├── concat_bpe_8000_tokenizer_az/
+│   └── ...
+└── plots/                             # 24 visualization PNGs
+    ├── fertility_overall_comparison_v2.png
+    ├── bootstrap_ci_forest.png
+    ├── morph_edit_distance_ar_heatmap_v2.png
+    └── ...
+```
+
+## Methodology
+
+1. **Dataset**: `OiQ/daa-pairs` — 112,814 Moroccan Darija sentence pairs in Arabic script and Arabizi
+2. **Pre-tokenization**: Metaspace (BPE/Unigram/WordPiece) or ByteLevel (BBPE)
+3. **Training**: HuggingFace `tokenizers` library with standard trainer hyperparameters
+4. **Evaluation**: 14 metrics including fertility, CPT, disparity, exact match, morphological alignment (Farasa), and bootstrap CIs
+5. **Export**: `transformers`-compatible format via `PreTrainedTokenizerFast`
+
+## Key Findings
+
+- **Concatenated tokenizers** reduce cross-script fertility disparity compared to shared-vocabulary tokenizers
+- **WordPiece** at 32K achieves the lowest overall fertility (1.39) among concatenated tokenizers
+- **BBPE** guarantees 100% exact reconstruction but has higher fertility due to byte-level granularity
+- **Unigram** models produce the lowest cross-script disparity among shared tokenizers
+- Morphological metrics (μe, μc) reveal that no single algorithm dominates morphological alignment — BBPE achieves highest consistency F1 while BPE achieves lowest edit distance
+
+## Citation
+
+```bibtex
+@misc{oiq2025daa-tokenizers,
+  title={Darija Subword Tokenizer Benchmark},
+  author={OiQ},
+  year={2025},
+  url={https://huggingface.co/OiQ/daa-tokenizers}
+}
+```