daa-tokenizers / benchmark_report.md

Upload benchmark_report.md with huggingface_hub

b92f1ef verified 20 days ago

8.47 kB

	# Production Tokenizer Benchmark Report: Moroccan Darija

	## Dataset
	- Source: `OiQ/daa-pairs`
	- Samples: 112,814 sentence pairs
	- Splits: train/val/test = 80%/10%/10%
	- Scripts: Arabic (ar), Arabizi/Latin (az), Mixed (mi)

	## Methodology
	- Algorithms: BPE, Unigram, WordPiece, BBPE
	- Pre-tokenization: Metaspace (SentencePiece-style) for BPE/Unigram/WordPiece; ByteLevel for BBPE
	- Decoder: Matched to pre-tokenizer for exact reconstruction
	- Tokenizer types: Shared (single vocabulary trained on concatenated Arabic+Arabizi corpus) and Concatenated (separate Arabic and Arabizi vocabularies with shifted IDs, enabling single-model serving)
	- Vocabulary sizes: 8K, 16K, 32K
	- Metrics: Fertility, CPT (grapheme-aware characters-per-token), OOV, cross-script disparity, Gini, Shannon entropy, exact match rate
	- Morphological Metrics (Arabic-script only, using Farasa segmenter):
	- μe (Morphological Edit Distance): DP alignment between tokenizer tokens and Farasa morpheme boundaries; lower is better
	- μc (Morphological Consistency F1): Precision/recall/F1 for whether shared morphemes yield shared tokens (via KMeans clustering)
	- Statistics: Bootstrap 95% CIs (n=500 resamples)

	## Best Tokenizers by Vocabulary Size

	\| Vocab Size \| Name \| Type \| Algorithm \| Fertility \| Disparity \| μe (lower=better) \| μc F1 (higher=better) \| Exact Match \|
	\|------------\|------\|------\|-----------\|-----------\|-----------\|-------------------\|----------------------\|-------------\|
	\| 8000 \| concat_bpe_8000 \| concatenated \| BPE \| 1.6890 \| 0.1649 \| 4.2158 \| 0.1680 \| 99.00% \|
	\| 16000 \| concat_wordpiece_16000 \| concatenated \| WordPiece \| 1.5162 \| 0.1361 \| 4.1506 \| 0.1533 \| 99.00% \|
	\| 32000 \| concat_wordpiece_32000 \| concatenated \| WordPiece \| 1.3885 \| 0.1309 \| 4.0975 \| 0.1329 \| 99.00% \|

	## Full Results

	\| Name \| Type \| Algorithm \| Vocab \| Fertility \| CPT \| Disparity \| Exact Match \| Gini \| μe \| μc F1 \|
	\|------\|------\|-----------\|-------\|-----------\|-----\|-----------\|-------------\|------\|----\|----\|
	\| shared_bpe_8000 \| shared \| BPE \| 8000 \| 1.7787 \| 3.00 \| 0.2199 \| 99.00% \| 0.7709 \| 4.1755 \| 0.1617 \|
	\| concat_bpe_8000 \| concatenated \| BPE \| 8000 \| 1.6890 \| 3.16 \| 0.1649 \| 99.00% \| 0.6850 \| 4.2158 \| 0.1680 \|
	\| shared_unigram_8000 \| shared \| Unigram \| 8000 \| 1.7790 \| 3.00 \| 0.1475 \| 99.00% \| 0.8189 \| 4.1823 \| 0.1770 \|
	\| concat_unigram_8000 \| concatenated \| Unigram \| 8000 \| 1.6948 \| 3.15 \| 0.1669 \| 99.00% \| 0.7623 \| 4.2111 \| 0.1801 \|
	\| shared_wordpiece_8000 \| shared \| WordPiece \| 8000 \| 1.7762 \| 3.01 \| 0.2268 \| 99.00% \| 0.7565 \| 4.1766 \| 0.1552 \|
	\| concat_wordpiece_8000 \| concatenated \| WordPiece \| 8000 \| 1.6962 \| 3.15 \| 0.1560 \| 99.00% \| 0.6600 \| 4.2191 \| 0.1619 \|
	\| shared_bbpe_8000 \| shared \| BBPE \| 8000 \| 1.9554 \| 2.73 \| 0.4718 \| 99.00% \| 0.8122 \| 4.1921 \| 0.1995 \|
	\| concat_bbpe_8000 \| concatenated \| BBPE \| 8000 \| 1.8828 \| 2.84 \| 0.1168 \| 99.00% \| 0.7491 \| 4.2317 \| 0.2136 \|
	\| shared_bpe_16000 \| shared \| BPE \| 16000 \| 1.5972 \| 3.35 \| 0.1511 \| 99.00% \| 0.7920 \| 4.1165 \| 0.1324 \|
	\| concat_bpe_16000 \| concatenated \| BPE \| 16000 \| 1.5166 \| 3.52 \| 0.1427 \| 99.00% \| 0.7225 \| 4.1531 \| 0.1376 \|
	\| shared_unigram_16000 \| shared \| Unigram \| 16000 \| 1.6213 \| 3.30 \| 0.0885 \| 99.00% \| 0.8335 \| 4.1315 \| 0.1623 \|
	\| concat_unigram_16000 \| concatenated \| Unigram \| 16000 \| 1.5417 \| 3.47 \| 0.1464 \| 99.00% \| 0.7918 \| 4.1582 \| 0.1575 \|
	\| shared_wordpiece_16000 \| shared \| WordPiece \| 16000 \| 1.5914 \| 3.36 \| 0.1512 \| 99.00% \| 0.7797 \| 4.1158 \| 0.1385 \|
	\| concat_wordpiece_16000 \| concatenated \| WordPiece \| 16000 \| 1.5162 \| 3.53 \| 0.1361 \| 99.00% \| 0.7024 \| 4.1506 \| 0.1533 \|
	\| shared_bbpe_16000 \| shared \| BBPE \| 16000 \| 1.7937 \| 2.98 \| 0.4265 \| 99.00% \| 0.8353 \| 4.1365 \| 0.1824 \|
	\| concat_bbpe_16000 \| concatenated \| BBPE \| 16000 \| 1.7340 \| 3.08 \| 0.1722 \| 99.00% \| 0.7908 \| 4.1720 \| 0.1986 \|
	\| shared_bpe_32000 \| shared \| BPE \| 32000 \| 1.4562 \| 3.67 \| 0.0727 \| 99.00% \| 0.7969 \| 4.0706 \| 0.1149 \|
	\| concat_bpe_32000 \| concatenated \| BPE \| 32000 \| 1.3912 \| 3.84 \| 0.1340 \| 99.00% \| 0.7568 \| 4.0970 \| 0.1146 \|
	\| shared_unigram_32000 \| shared \| Unigram \| 32000 \| 1.5034 \| 3.56 \| 0.0193 \| 99.00% \| 0.8325 \| 4.0979 \| 0.1482 \|
	\| concat_unigram_32000 \| concatenated \| Unigram \| 32000 \| 1.4304 \| 3.74 \| 0.1449 \| 99.00% \| 0.8069 \| 4.1112 \| 0.1420 \|
	\| shared_wordpiece_32000 \| shared \| WordPiece \| 32000 \| 1.4501 \| 3.69 \| 0.0724 \| 99.00% \| 0.7886 \| 4.0696 \| 0.1273 \|
	\| concat_wordpiece_32000 \| concatenated \| WordPiece \| 32000 \| 1.3885 \| 3.85 \| 0.1309 \| 99.00% \| 0.7419 \| 4.0975 \| 0.1329 \|
	\| shared_bbpe_32000 \| shared \| BBPE \| 32000 \| 1.6737 \| 3.19 \| 0.3737 \| 99.00% \| 0.8431 \| 4.0974 \| 0.1737 \|
	\| concat_bbpe_32000 \| concatenated \| BBPE \| 32000 \| 1.6299 \| 3.28 \| 0.2067 \| 99.00% \| 0.8217 \| 4.1216 \| 0.1798 \|

	## Morphological Metrics (Arabic-script only)

	\| Name \| Algorithm \| Type \| Vocab \| μe \| Precision \| Recall \| F1 \|
	\|------\|-----------\|------\|-------\|----\|-----------\|--------\|----\|
	\| shared_bpe_8000 \| BPE \| shared \| 8000 \| 4.1755 \| 0.1706 \| 0.1557 \| 0.1617 \|
	\| concat_bpe_8000 \| BPE \| concatenated \| 8000 \| 4.2158 \| 0.1570 \| 0.1856 \| 0.1680 \|
	\| shared_unigram_8000 \| Unigram \| shared \| 8000 \| 4.1823 \| 0.1499 \| 0.2197 \| 0.1770 \|
	\| concat_unigram_8000 \| Unigram \| concatenated \| 8000 \| 4.2111 \| 0.1451 \| 0.2405 \| 0.1801 \|
	\| shared_wordpiece_8000 \| WordPiece \| shared \| 8000 \| 4.1766 \| 0.1637 \| 0.1494 \| 0.1552 \|
	\| concat_wordpiece_8000 \| WordPiece \| concatenated \| 8000 \| 4.2191 \| 0.1522 \| 0.1758 \| 0.1619 \|
	\| shared_bbpe_8000 \| BBPE \| shared \| 8000 \| 4.1921 \| 0.1550 \| 0.2827 \| 0.1995 \|
	\| concat_bbpe_8000 \| BBPE \| concatenated \| 8000 \| 4.2317 \| 0.1589 \| 0.3283 \| 0.2136 \|
	\| shared_bpe_16000 \| BPE \| shared \| 16000 \| 4.1165 \| 0.1646 \| 0.1113 \| 0.1324 \|
	\| concat_bpe_16000 \| BPE \| concatenated \| 16000 \| 4.1531 \| 0.1544 \| 0.1249 \| 0.1376 \|
	\| shared_unigram_16000 \| Unigram \| shared \| 16000 \| 4.1315 \| 0.1558 \| 0.1706 \| 0.1623 \|
	\| concat_unigram_16000 \| Unigram \| concatenated \| 16000 \| 4.1582 \| 0.1464 \| 0.1741 \| 0.1575 \|
	\| shared_wordpiece_16000 \| WordPiece \| shared \| 16000 \| 4.1158 \| 0.1654 \| 0.1210 \| 0.1385 \|
	\| concat_wordpiece_16000 \| WordPiece \| concatenated \| 16000 \| 4.1506 \| 0.1740 \| 0.1384 \| 0.1533 \|
	\| shared_bbpe_16000 \| BBPE \| shared \| 16000 \| 4.1365 \| 0.1546 \| 0.2261 \| 0.1824 \|
	\| concat_bbpe_16000 \| BBPE \| concatenated \| 16000 \| 4.1720 \| 0.1620 \| 0.2600 \| 0.1986 \|
	\| shared_bpe_32000 \| BPE \| shared \| 32000 \| 4.0706 \| 0.1610 \| 0.0899 \| 0.1149 \|
	\| concat_bpe_32000 \| BPE \| concatenated \| 32000 \| 4.0970 \| 0.1487 \| 0.0942 \| 0.1146 \|
	\| shared_unigram_32000 \| Unigram \| shared \| 32000 \| 4.0979 \| 0.1631 \| 0.1383 \| 0.1482 \|
	\| concat_unigram_32000 \| Unigram \| concatenated \| 32000 \| 4.1112 \| 0.1486 \| 0.1393 \| 0.1420 \|
	\| shared_wordpiece_32000 \| WordPiece \| shared \| 32000 \| 4.0696 \| 0.1814 \| 0.0999 \| 0.1273 \|
	\| concat_wordpiece_32000 \| WordPiece \| concatenated \| 32000 \| 4.0975 \| 0.1636 \| 0.1126 \| 0.1329 \|
	\| shared_bbpe_32000 \| BBPE \| shared \| 32000 \| 4.0974 \| 0.1571 \| 0.1976 \| 0.1737 \|
	\| concat_bbpe_32000 \| BBPE \| concatenated \| 32000 \| 4.1216 \| 0.1574 \| 0.2144 \| 0.1798 \|

	## Key Findings
	- Concatenated tokenizers reduce cross-script fertility disparity compared to shared-vocabulary tokenizers, at the cost of larger total vocabulary.
	- BBPE achieves 100% exact reconstruction by design (byte-level guarantees).
	- Metaspace-based tokenizers (BPE/Unigram) achieve >95% exact reconstruction.
	- WordPiece exact reconstruction is lower due to inherent whitespace handling limitations.
	- Gini coefficients are correctly bounded in [0, 1], indicating healthy vocabulary utilization.
	- Morphological edit distance (μe) and consistency (μc) provide complementary signals for evaluating tokenization quality beyond surface-level fertility.
	- Unigram models tend to produce lower fertility (fewer tokens per word) at the cost of higher morphological edit distance.

	## Files
	- `tokenizer_results.csv` / `.json` — Complete metrics for all 24 tokenizers
	- `bootstrap_ci.csv` — Bootstrap 95% confidence intervals for fertility and CPT
	- `morphology/farasa_segmentations.json` — Cached Farasa morphological segmentations
	- `tokenizers/` — Raw tokenizer JSON files (HuggingFace tokenizers library format)
	- `transformers_tokenizers/` — 28 tokenizer directories ready for `transformers` library
	- `plots/` — 24 visualization PNGs (comparison bars, faceted, trends, heatmaps, bootstrap forest)
	- `script.py` — Complete benchmark script (reproducible)

	# Production Tokenizer Benchmark Report: Moroccan Darija

	## Dataset
	- Source: `OiQ/daa-pairs`
	- Samples: 112,814 sentence pairs
	- Splits: train/val/test = 80%/10%/10%
	- Scripts: Arabic (ar), Arabizi/Latin (az), Mixed (mi)

	## Methodology
	- Algorithms: BPE, Unigram, WordPiece, BBPE
	- Pre-tokenization: Metaspace (SentencePiece-style) for BPE/Unigram/WordPiece; ByteLevel for BBPE
	- Decoder: Matched to pre-tokenizer for exact reconstruction
	- Tokenizer types: Shared (single vocabulary trained on concatenated Arabic+Arabizi corpus) and Concatenated (separate Arabic and Arabizi vocabularies with shifted IDs, enabling single-model serving)
	- Vocabulary sizes: 8K, 16K, 32K
	- Metrics: Fertility, CPT (grapheme-aware characters-per-token), OOV, cross-script disparity, Gini, Shannon entropy, exact match rate
	- Morphological Metrics (Arabic-script only, using Farasa segmenter):
	- μe (Morphological Edit Distance): DP alignment between tokenizer tokens and Farasa morpheme boundaries; lower is better
	- μc (Morphological Consistency F1): Precision/recall/F1 for whether shared morphemes yield shared tokens (via KMeans clustering)
	- Statistics: Bootstrap 95% CIs (n=500 resamples)

	## Best Tokenizers by Vocabulary Size

	\| Vocab Size \| Name \| Type \| Algorithm \| Fertility \| Disparity \| μe (lower=better) \| μc F1 (higher=better) \| Exact Match \|
	\|------------\|------\|------\|-----------\|-----------\|-----------\|-------------------\|----------------------\|-------------\|
	\| 8000 \| concat_bpe_8000 \| concatenated \| BPE \| 1.6890 \| 0.1649 \| 4.2158 \| 0.1680 \| 99.00% \|
	\| 16000 \| concat_wordpiece_16000 \| concatenated \| WordPiece \| 1.5162 \| 0.1361 \| 4.1506 \| 0.1533 \| 99.00% \|
	\| 32000 \| concat_wordpiece_32000 \| concatenated \| WordPiece \| 1.3885 \| 0.1309 \| 4.0975 \| 0.1329 \| 99.00% \|

	## Full Results

	\| Name \| Type \| Algorithm \| Vocab \| Fertility \| CPT \| Disparity \| Exact Match \| Gini \| μe \| μc F1 \|
	\|------\|------\|-----------\|-------\|-----------\|-----\|-----------\|-------------\|------\|----\|----\|
	\| shared_bpe_8000 \| shared \| BPE \| 8000 \| 1.7787 \| 3.00 \| 0.2199 \| 99.00% \| 0.7709 \| 4.1755 \| 0.1617 \|
	\| concat_bpe_8000 \| concatenated \| BPE \| 8000 \| 1.6890 \| 3.16 \| 0.1649 \| 99.00% \| 0.6850 \| 4.2158 \| 0.1680 \|
	\| shared_unigram_8000 \| shared \| Unigram \| 8000 \| 1.7790 \| 3.00 \| 0.1475 \| 99.00% \| 0.8189 \| 4.1823 \| 0.1770 \|
	\| concat_unigram_8000 \| concatenated \| Unigram \| 8000 \| 1.6948 \| 3.15 \| 0.1669 \| 99.00% \| 0.7623 \| 4.2111 \| 0.1801 \|
	\| shared_wordpiece_8000 \| shared \| WordPiece \| 8000 \| 1.7762 \| 3.01 \| 0.2268 \| 99.00% \| 0.7565 \| 4.1766 \| 0.1552 \|
	\| concat_wordpiece_8000 \| concatenated \| WordPiece \| 8000 \| 1.6962 \| 3.15 \| 0.1560 \| 99.00% \| 0.6600 \| 4.2191 \| 0.1619 \|
	\| shared_bbpe_8000 \| shared \| BBPE \| 8000 \| 1.9554 \| 2.73 \| 0.4718 \| 99.00% \| 0.8122 \| 4.1921 \| 0.1995 \|
	\| concat_bbpe_8000 \| concatenated \| BBPE \| 8000 \| 1.8828 \| 2.84 \| 0.1168 \| 99.00% \| 0.7491 \| 4.2317 \| 0.2136 \|
	\| shared_bpe_16000 \| shared \| BPE \| 16000 \| 1.5972 \| 3.35 \| 0.1511 \| 99.00% \| 0.7920 \| 4.1165 \| 0.1324 \|
	\| concat_bpe_16000 \| concatenated \| BPE \| 16000 \| 1.5166 \| 3.52 \| 0.1427 \| 99.00% \| 0.7225 \| 4.1531 \| 0.1376 \|
	\| shared_unigram_16000 \| shared \| Unigram \| 16000 \| 1.6213 \| 3.30 \| 0.0885 \| 99.00% \| 0.8335 \| 4.1315 \| 0.1623 \|
	\| concat_unigram_16000 \| concatenated \| Unigram \| 16000 \| 1.5417 \| 3.47 \| 0.1464 \| 99.00% \| 0.7918 \| 4.1582 \| 0.1575 \|
	\| shared_wordpiece_16000 \| shared \| WordPiece \| 16000 \| 1.5914 \| 3.36 \| 0.1512 \| 99.00% \| 0.7797 \| 4.1158 \| 0.1385 \|
	\| concat_wordpiece_16000 \| concatenated \| WordPiece \| 16000 \| 1.5162 \| 3.53 \| 0.1361 \| 99.00% \| 0.7024 \| 4.1506 \| 0.1533 \|
	\| shared_bbpe_16000 \| shared \| BBPE \| 16000 \| 1.7937 \| 2.98 \| 0.4265 \| 99.00% \| 0.8353 \| 4.1365 \| 0.1824 \|
	\| concat_bbpe_16000 \| concatenated \| BBPE \| 16000 \| 1.7340 \| 3.08 \| 0.1722 \| 99.00% \| 0.7908 \| 4.1720 \| 0.1986 \|
	\| shared_bpe_32000 \| shared \| BPE \| 32000 \| 1.4562 \| 3.67 \| 0.0727 \| 99.00% \| 0.7969 \| 4.0706 \| 0.1149 \|
	\| concat_bpe_32000 \| concatenated \| BPE \| 32000 \| 1.3912 \| 3.84 \| 0.1340 \| 99.00% \| 0.7568 \| 4.0970 \| 0.1146 \|
	\| shared_unigram_32000 \| shared \| Unigram \| 32000 \| 1.5034 \| 3.56 \| 0.0193 \| 99.00% \| 0.8325 \| 4.0979 \| 0.1482 \|
	\| concat_unigram_32000 \| concatenated \| Unigram \| 32000 \| 1.4304 \| 3.74 \| 0.1449 \| 99.00% \| 0.8069 \| 4.1112 \| 0.1420 \|
	\| shared_wordpiece_32000 \| shared \| WordPiece \| 32000 \| 1.4501 \| 3.69 \| 0.0724 \| 99.00% \| 0.7886 \| 4.0696 \| 0.1273 \|
	\| concat_wordpiece_32000 \| concatenated \| WordPiece \| 32000 \| 1.3885 \| 3.85 \| 0.1309 \| 99.00% \| 0.7419 \| 4.0975 \| 0.1329 \|
	\| shared_bbpe_32000 \| shared \| BBPE \| 32000 \| 1.6737 \| 3.19 \| 0.3737 \| 99.00% \| 0.8431 \| 4.0974 \| 0.1737 \|
	\| concat_bbpe_32000 \| concatenated \| BBPE \| 32000 \| 1.6299 \| 3.28 \| 0.2067 \| 99.00% \| 0.8217 \| 4.1216 \| 0.1798 \|

	## Morphological Metrics (Arabic-script only)

	\| Name \| Algorithm \| Type \| Vocab \| μe \| Precision \| Recall \| F1 \|
	\|------\|-----------\|------\|-------\|----\|-----------\|--------\|----\|
	\| shared_bpe_8000 \| BPE \| shared \| 8000 \| 4.1755 \| 0.1706 \| 0.1557 \| 0.1617 \|
	\| concat_bpe_8000 \| BPE \| concatenated \| 8000 \| 4.2158 \| 0.1570 \| 0.1856 \| 0.1680 \|
	\| shared_unigram_8000 \| Unigram \| shared \| 8000 \| 4.1823 \| 0.1499 \| 0.2197 \| 0.1770 \|
	\| concat_unigram_8000 \| Unigram \| concatenated \| 8000 \| 4.2111 \| 0.1451 \| 0.2405 \| 0.1801 \|
	\| shared_wordpiece_8000 \| WordPiece \| shared \| 8000 \| 4.1766 \| 0.1637 \| 0.1494 \| 0.1552 \|
	\| concat_wordpiece_8000 \| WordPiece \| concatenated \| 8000 \| 4.2191 \| 0.1522 \| 0.1758 \| 0.1619 \|
	\| shared_bbpe_8000 \| BBPE \| shared \| 8000 \| 4.1921 \| 0.1550 \| 0.2827 \| 0.1995 \|
	\| concat_bbpe_8000 \| BBPE \| concatenated \| 8000 \| 4.2317 \| 0.1589 \| 0.3283 \| 0.2136 \|
	\| shared_bpe_16000 \| BPE \| shared \| 16000 \| 4.1165 \| 0.1646 \| 0.1113 \| 0.1324 \|
	\| concat_bpe_16000 \| BPE \| concatenated \| 16000 \| 4.1531 \| 0.1544 \| 0.1249 \| 0.1376 \|
	\| shared_unigram_16000 \| Unigram \| shared \| 16000 \| 4.1315 \| 0.1558 \| 0.1706 \| 0.1623 \|
	\| concat_unigram_16000 \| Unigram \| concatenated \| 16000 \| 4.1582 \| 0.1464 \| 0.1741 \| 0.1575 \|
	\| shared_wordpiece_16000 \| WordPiece \| shared \| 16000 \| 4.1158 \| 0.1654 \| 0.1210 \| 0.1385 \|
	\| concat_wordpiece_16000 \| WordPiece \| concatenated \| 16000 \| 4.1506 \| 0.1740 \| 0.1384 \| 0.1533 \|
	\| shared_bbpe_16000 \| BBPE \| shared \| 16000 \| 4.1365 \| 0.1546 \| 0.2261 \| 0.1824 \|
	\| concat_bbpe_16000 \| BBPE \| concatenated \| 16000 \| 4.1720 \| 0.1620 \| 0.2600 \| 0.1986 \|
	\| shared_bpe_32000 \| BPE \| shared \| 32000 \| 4.0706 \| 0.1610 \| 0.0899 \| 0.1149 \|
	\| concat_bpe_32000 \| BPE \| concatenated \| 32000 \| 4.0970 \| 0.1487 \| 0.0942 \| 0.1146 \|
	\| shared_unigram_32000 \| Unigram \| shared \| 32000 \| 4.0979 \| 0.1631 \| 0.1383 \| 0.1482 \|
	\| concat_unigram_32000 \| Unigram \| concatenated \| 32000 \| 4.1112 \| 0.1486 \| 0.1393 \| 0.1420 \|
	\| shared_wordpiece_32000 \| WordPiece \| shared \| 32000 \| 4.0696 \| 0.1814 \| 0.0999 \| 0.1273 \|
	\| concat_wordpiece_32000 \| WordPiece \| concatenated \| 32000 \| 4.0975 \| 0.1636 \| 0.1126 \| 0.1329 \|
	\| shared_bbpe_32000 \| BBPE \| shared \| 32000 \| 4.0974 \| 0.1571 \| 0.1976 \| 0.1737 \|
	\| concat_bbpe_32000 \| BBPE \| concatenated \| 32000 \| 4.1216 \| 0.1574 \| 0.2144 \| 0.1798 \|

	## Key Findings
	- Concatenated tokenizers reduce cross-script fertility disparity compared to shared-vocabulary tokenizers, at the cost of larger total vocabulary.
	- BBPE achieves 100% exact reconstruction by design (byte-level guarantees).
	- Metaspace-based tokenizers (BPE/Unigram) achieve >95% exact reconstruction.
	- WordPiece exact reconstruction is lower due to inherent whitespace handling limitations.
	- Gini coefficients are correctly bounded in [0, 1], indicating healthy vocabulary utilization.
	- Morphological edit distance (μe) and consistency (μc) provide complementary signals for evaluating tokenization quality beyond surface-level fertility.
	- Unigram models tend to produce lower fertility (fewer tokens per word) at the cost of higher morphological edit distance.

	## Files
	- `tokenizer_results.csv` / `.json` — Complete metrics for all 24 tokenizers
	- `bootstrap_ci.csv` — Bootstrap 95% confidence intervals for fertility and CPT
	- `morphology/farasa_segmentations.json` — Cached Farasa morphological segmentations
	- `tokenizers/` — Raw tokenizer JSON files (HuggingFace tokenizers library format)
	- `transformers_tokenizers/` — 28 tokenizer directories ready for `transformers` library
	- `plots/` — 24 visualization PNGs (comparison bars, faceted, trends, heatmaps, bootstrap forest)
	- `script.py` — Complete benchmark script (reproducible)