code.md

# Code Documentation — Darija Tokenizer Benchmark

This document describes every script, data file, and output artifact in the benchmark codebase.

---

## Overview

The benchmark pipeline consists of four stages: **training**, **evaluation**, **analysis**, and **reporting**. Each stage is implemented by standalone Python scripts. The diagram below shows the data flow:

```
OiQ/daa-pairs (dataset)
        │
        ▼
┌──────────────────┐     ┌─────────────────────┐
│  script.py       │────▶│  results/tokenizers/ │  (60 raw JSON files)
│  (train 8K-32K)  │     │  results/transformers│  (transformers format)
└──────────────────┘     └─────────────────────┘
        │                          ▲
        ▼                          │
┌──────────────────┐     ┌─────────────────────┐
│train_large_vocab │────▶│  80K + 110K configs  │
│train_remaining   │     │  (16 additional)     │
└──────────────────┘     └─────────────────────┘
                               │
                               ▼
┌──────────────────────────────────────────────┐
│  EVALUATION SCRIPTS                           │
│  ├── eval_test_set.py       → test_set_results│
│  ├── eval_new_and_append.py → append 80K/110K │
│  ├── eval_missing.py        → fill gaps       │
│  ├── eval_morph_large.py    → morph 80K/110K  │
│  ├── bootstrap_test_set.py  → 95% CIs         │
│  ├── eval_all_externals.py  → external comp.  │
│  ├── eval_codeswitch_...    → code-switching  │
│  └── eval_doda_independent  → DODa validation │
└──────────────────────────────────────────────┘
        │
        ▼
┌──────────────────┐     ┌─────────────────────┐
│  regen_figures   │────▶│  figures/*.png       │
│  gen_report      │────▶│  benchmark_report.md │
│  verify_arithmetic│───▶│  (stdout validation) │
└──────────────────┘     └─────────────────────┘
```

---

## Training Scripts

### `script.py` — Master Benchmark Pipeline
**Lines:** ~2032 | **Type:** Runs at import (no `__main__` guard)

The monolithic entry point. Loads the `OiQ/daa-pairs` dataset, trains 24 tokenizers (4 algorithms x 2 architectures x 3 vocab sizes: 8K, 16K, 32K), evaluates them on the full training+test corpus, computes morphological metrics via Farasa, generates bootstrap confidence intervals, and produces all plots and reports.

**Key class:** `ProductionMetricsEvaluator` — implements script detection, tokenization, Gini coefficient, and all metric computations.

**Inputs:**
- `OiQ/daa-pairs` dataset (via `huggingface_hub`)
- `BenchmarkConfig` dataclass (vocab sizes, algorithms, hyperparameters)

**Outputs:**
- `results/tokenizers/*.json` — 48 raw tokenizer files (24 shared + 24 concat halves)
- `results/transformers_tokenizers/` — transformers-compatible exports
- `results/tokenizer_results.csv` / `.json` — full metrics with morphological data
- `results/bootstrap_ci.csv` — bootstrap CIs
- `results/benchmark_report.md` — auto-generated summary
- `results/morphology/farasa_segmentations.json` — cached Farasa segmentations (~99 MB)
- `results/plots/*.png` — all visualization figures

---

### `train_large_vocab.py` — Train 80K/110K Tokenizers
**Lines:** ~146

Trains additional tokenizers at 80K and 110K vocabulary sizes to match DarijaBERT-ar (80K) and DarijaBERT-az (110K) for fair head-to-head comparison.

**Inputs:** `results/corpora/train_{ar,az}.txt`

**Outputs:** `results/tokenizers/{shared,concat_ar,concat_az}_{algo}_{80000,110000}.json`

---

### `train_remaining.py` — Train Remaining Tokenizers
**Lines:** ~134

Fills in the last missing tokenizer configurations (3 shared 110K + 6 concat 110K + 1 concat 80K BBPE) that `train_large_vocab.py` did not cover.

**Inputs:** `results/corpora/train_{ar,az}.txt`

**Outputs:** Remaining `results/tokenizers/*.json` files

---

### `retrain_missing_and_compare.py` — Retrain + Full Re-evaluation
**Lines:** ~558

Retrains 4 missing concat 32K tokenizers, exports to transformers format, re-evaluates all 28 configs, fixes a WordPiece exact-match bug, and runs the external comparison pipeline.

**Inputs:** Training corpora, HF external models

**Outputs:** Updated tokenizer JSONs, `external_comparison.csv`, external comparison plot

---

## Evaluation Scripts

### `eval_test_set.py` — Test-Set Evaluation (Single Source of Truth)
**Lines:** ~227

Re-evaluates all tokenizers on the held-out **test set only** (11,282 sentences per script). This is the authoritative evaluation used in all paper tables and percentage claims.

**Key function:** `normalize_decode()` — fixes Metaspace double-space artifacts in WordPiece decoders.

**Inputs:**
- `results/tokenizers/*.json`
- `results/corpora/test_{ar,az,mi}.txt`

**Outputs:** `results/test_set_results.csv` / `.json` (40 rows)

---

### `eval_new_and_append.py` — Append 80K/110K Results
**Lines:** ~144

Evaluates the newly trained 80K/110K tokenizers and appends their rows to `test_set_results.csv`.

**Inputs:** `test_set_results.csv`, tokenizers, test corpora

**Outputs:** Updated `test_set_results.csv` (grows from 24 to 40 rows)

---

### `eval_missing.py` — Fill Single Gap
**Lines:** ~124

Evaluates the one remaining missing tokenizer (`concat_bbpe_55000`) and merges it into the results CSV.

---

### `eval_morph_large.py` — Morphological Metrics for 80K/110K
**Lines:** ~297

Computes morphological fidelity metrics (edit distance and consistency F1) for the 16 large-vocabulary tokenizers (80K, 110K) that were not covered by the original `script.py` morph evaluation. Uses the same Farasa cache and identical algorithms.

**Inputs:**
- `results/morphology/farasa_segmentations.json`
- `results/corpora/test_ar.txt`
- 16 tokenizer JSON files

**Outputs:** `results/morph_large_vocab_results.csv` (16 rows)

---

### `bootstrap_test_set.py` — Bootstrap Confidence Intervals
**Lines:** ~163

Computes 95% bootstrap confidence intervals (500 resamples) for fertility and CPT on the test set.

**Inputs:** Tokenizers, `test_{ar,az,mi}.txt`

**Outputs:** `results/bootstrap_ci_test_set.csv` (24 rows for 8K-32K configs)

---

## External Comparison Scripts

### `eval_all_externals.py` — Evaluate 9 External Tokenizers
**Lines:** ~281

Evaluates 9 external Arabic/Darija tokenizers from HuggingFace (CaMeLBERT-MSA, Asafaya-BERT, Aranizer-SP-86k, B2BERT, DarijaBERT-ar, DarijaBERT-az, Darija-Tokenizer, Translit-Darija, Qwen2.5-Darija) alongside our best 3 tokenizers.

**Inputs:** HF model repos (requires `HF_TOKEN`), test corpora

**Outputs:** `results/external_comparison.csv` / `.json`, comparison plot

---

### `compare_with_external.py` — External Comparison (Earlier Version)
**Lines:** ~269

Earlier version of the external comparison, comparing our 8K/16K/32K tokenizers against 5 external models.

---

### `eval_and_compare.py` — Combined Evaluation + Comparison
**Lines:** ~277

Combines internal evaluation and external comparison into a single pipeline run.

---

### `eval_codeswitch_and_new_baselines.py` — Code-Switching Evaluation
**Lines:** ~373

Evaluates tokenizers on mixed-script (code-switched) texts as a separate category. Also adds the `atlasia/darija_bpe_tokenizer` baseline and evaluates all tokenizers on DODa.

**Outputs:** `results/codeswitch_results.csv` / `.json`

---

### `eval_doda_independent.py` — DODa Independent Validation
**Lines:** ~196

Evaluates all tokenizers on `atlasia/DODa` (87K Arabizi dictionary entries), sampled to 10K entries. Serves as an independent, out-of-training-distribution validation.

**Outputs:** `results/doda_independent_results.csv` / `.json`

---

## Utility Scripts

### `fix_tokenizer_decoders.py` — Decoder Bug Fixer
**Lines:** ~152

Patches three decoder bugs in tokenizer JSON files:
1. WordPiece double-space artifact (Metaspace decoder producing `"  "` instead of `" "`)
2. NULL decoder in `concat_bpe_16000` (missing Metaspace decoder)
3. Missing WordPiece sub-decoder in `concat_wordpiece_16000`

**Warning:** Modifies tokenizer files in place. No backup is created.

---

### `gen_report.py` — Report Generator
**Lines:** ~70

Generates a Markdown summary report (`benchmark_report.md`) from `tokenizer_results.csv`, including best-by-vocabulary tables and full results.

---

### `regen_figures.py` — Figure Regenerator
**Lines:** ~313

Regenerates the 5 main paper figures from `test_set_results.csv` with larger fonts (14pt base) for readability. Reads from CSV and writes PNGs to the paper's `figures/` directory.

**Inputs:** `results/test_set_results.csv`

**Outputs:** `figures/fertility_overall_comparison_v2.png`, `fertility_overall_trends.png`, `fertility_disparity_comparison_v2.png`, `fertility_disparity_heatmap_v2.png`, `external_comparison.png`

---

### `verify_arithmetic.py` — Numeric Claims Verification
**Lines:** ~143

Validates every percentage claim in the paper and README against `test_set_results.csv`. Checks:
- Disparity formula: `|F_ar - F_az| / max(F_ar, F_az)`
- Overall fertility derivability from per-script values
- All percentage improvement claims (27-34%, 40-50%, etc.)

**Outputs:** Stdout validation report (no files written)

---

## Result Files

### Primary Data

| File | Rows | Description |
|------|------|-------------|
| `test_set_results.csv` | 40 | **Single source of truth.** Test-set metrics for all tokenizers: fertility, CPT, disparity, Gini, entropy, exact match. |
| `tokenizer_results.csv` | 24 | Full benchmark results incl. morphological metrics and per-script breakdown. Covers 8K-32K only. |
| `morph_large_vocab_results.csv` | 16 | Morphological metrics for 80K/110K tokenizers. |
| `bootstrap_ci_test_set.csv` | 24 | Bootstrap 95% CIs (500 resamples) for fertility and CPT. |
| `external_comparison.csv` | 12 | Our best 3 + 9 external tokenizers: fertility, CPT, disparity, exact match per script. |
| `codeswitch_results.csv` | 5 | Code-switching evaluation with mixed-script category. |
| `doda_independent_results.csv` | 12 | DODa independent validation (Arabizi dictionary). |

### Supporting Data

| File / Directory | Description |
|------------------|-------------|
| `corpora/` | Train/validation/test text splits: `{train,val,test}_{ar,az,mi}.txt` |
| `morphology/farasa_segmentations.json` | Cached Farasa morphological segmentations for Arabic texts (~99 MB) |
| `tokenizers/` | Raw HuggingFace `tokenizers` JSON files (60 files: 24 shared + 36 concat halves) |
| `transformers_tokenizers/` | Tokenizers exported for `transformers` library use |
| `doda_sample_10k.txt` | 10K-line Arabizi sample from DODa dataset |
| `benchmark_report.md` | Auto-generated Markdown summary report |

---

## Reproduction Guide

### Full Reproduction (from scratch)

```bash
# 1. Train 8K-32K tokenizers + initial evaluation + morphology
python script.py

# 2. Train 80K + 110K tokenizers
python train_large_vocab.py
python train_remaining.py

# 3. Evaluate on test set (appends 80K/110K to results)
python eval_test_set.py
python eval_new_and_append.py
python eval_missing.py

# 4. Compute morphological metrics for large vocabs
python eval_morph_large.py

# 5. Bootstrap confidence intervals
python bootstrap_test_set.py

# 6. External tokenizer comparison
python eval_all_externals.py

# 7. Code-switching + DODa validation
python eval_codeswitch_and_new_baselines.py
python eval_doda_independent.py

# 8. Generate figures + reports
python regen_figures.py
python gen_report.py

# 9. Verify all numeric claims
python verify_arithmetic.py
```

### Requirements

- Python 3.10+
- `tokenizers`, `transformers`, `datasets` (HuggingFace stack)
- `scikit-learn` (KMeans for morphological consistency)
- `regex` (Unicode grapheme segmentation)
- `numpy`, `pandas`, `matplotlib`, `seaborn`
- `tqdm`
- Farasa JAR (for morphological segmentation; pre-cached in `morphology/`)
- `HF_TOKEN` environment variable (for loading external models)

---

## Key Design Decisions

1. **Monolithic `script.py`**: The main pipeline runs at import level (no `__main__` guard). This is intentional for checkpoint-based resumption — the script detects existing artifacts and skips completed stages.

2. **Duplicated helper functions**: Functions like `detect_script()`, `count_graphemes()`, and `normalize_decode()` are copied across evaluation scripts rather than shared via import. This ensures each eval script is self-contained and runnable independently.

3. **Test-set-only evaluation**: All paper numbers come from `eval_test_set.py`, not `script.py`'s full-corpus evaluation. The test set (11,282 sentences per script) provides unbiased estimates.

4. **Concatenated architecture**: Each concat config is stored as two JSON files (`concat_ar_*.json` + `concat_az_*.json`). The evaluator loads both and applies ID shifting at inference time.