Hugging Face's logo

OiQ
/
daa-tokenizers

Arabic
Latin
tokenizer
moroccan-darija
arabic
bpe
unigram
wordpiece
bbpe
benchmark
Model card Files
xet
 Community
daa-tokenizers / benchmark_report.md
Ouaill's picture
Ouaill
Upload benchmark_report.md with huggingface_hub
b92f1ef verified
preview code
|
Raw
History Blame Contribute Delete
8.47 kB

Production Tokenizer Benchmark Report: Moroccan Darija

Dataset

  • Source: OiQ/daa-pairs
  • Samples: 112,814 sentence pairs
  • Splits: train/val/test = 80%/10%/10%
  • Scripts: Arabic (ar), Arabizi/Latin (az), Mixed (mi)

Methodology

  • Algorithms: BPE, Unigram, WordPiece, BBPE
  • Pre-tokenization: Metaspace (SentencePiece-style) for BPE/Unigram/WordPiece; ByteLevel for BBPE
  • Decoder: Matched to pre-tokenizer for exact reconstruction
  • Tokenizer types: Shared (single vocabulary trained on concatenated Arabic+Arabizi corpus) and Concatenated (separate Arabic and Arabizi vocabularies with shifted IDs, enabling single-model serving)
  • Vocabulary sizes: 8K, 16K, 32K
  • Metrics: Fertility, CPT (grapheme-aware characters-per-token), OOV, cross-script disparity, Gini, Shannon entropy, exact match rate
  • Morphological Metrics (Arabic-script only, using Farasa segmenter):
    • μe (Morphological Edit Distance): DP alignment between tokenizer tokens and Farasa morpheme boundaries; lower is better
    • μc (Morphological Consistency F1): Precision/recall/F1 for whether shared morphemes yield shared tokens (via KMeans clustering)
  • Statistics: Bootstrap 95% CIs (n=500 resamples)

Best Tokenizers by Vocabulary Size

Vocab Size Name Type Algorithm Fertility Disparity μe (lower=better) μc F1 (higher=better) Exact Match
8000 concat_bpe_8000 concatenated BPE 1.6890 0.1649 4.2158 0.1680 99.00%
16000 concat_wordpiece_16000 concatenated WordPiece 1.5162 0.1361 4.1506 0.1533 99.00%
32000 concat_wordpiece_32000 concatenated WordPiece 1.3885 0.1309 4.0975 0.1329 99.00%

Full Results

Name Type Algorithm Vocab Fertility CPT Disparity Exact Match Gini μe μc F1
shared_bpe_8000 shared BPE 8000 1.7787 3.00 0.2199 99.00% 0.7709 4.1755 0.1617
concat_bpe_8000 concatenated BPE 8000 1.6890 3.16 0.1649 99.00% 0.6850 4.2158 0.1680
shared_unigram_8000 shared Unigram 8000 1.7790 3.00 0.1475 99.00% 0.8189 4.1823 0.1770
concat_unigram_8000 concatenated Unigram 8000 1.6948 3.15 0.1669 99.00% 0.7623 4.2111 0.1801
shared_wordpiece_8000 shared WordPiece 8000 1.7762 3.01 0.2268 99.00% 0.7565 4.1766 0.1552
concat_wordpiece_8000 concatenated WordPiece 8000 1.6962 3.15 0.1560 99.00% 0.6600 4.2191 0.1619
shared_bbpe_8000 shared BBPE 8000 1.9554 2.73 0.4718 99.00% 0.8122 4.1921 0.1995
concat_bbpe_8000 concatenated BBPE 8000 1.8828 2.84 0.1168 99.00% 0.7491 4.2317 0.2136
shared_bpe_16000 shared BPE 16000 1.5972 3.35 0.1511 99.00% 0.7920 4.1165 0.1324
concat_bpe_16000 concatenated BPE 16000 1.5166 3.52 0.1427 99.00% 0.7225 4.1531 0.1376
shared_unigram_16000 shared Unigram 16000 1.6213 3.30 0.0885 99.00% 0.8335 4.1315 0.1623
concat_unigram_16000 concatenated Unigram 16000 1.5417 3.47 0.1464 99.00% 0.7918 4.1582 0.1575
shared_wordpiece_16000 shared WordPiece 16000 1.5914 3.36 0.1512 99.00% 0.7797 4.1158 0.1385
concat_wordpiece_16000 concatenated WordPiece 16000 1.5162 3.53 0.1361 99.00% 0.7024 4.1506 0.1533
shared_bbpe_16000 shared BBPE 16000 1.7937 2.98 0.4265 99.00% 0.8353 4.1365 0.1824
concat_bbpe_16000 concatenated BBPE 16000 1.7340 3.08 0.1722 99.00% 0.7908 4.1720 0.1986
shared_bpe_32000 shared BPE 32000 1.4562 3.67 0.0727 99.00% 0.7969 4.0706 0.1149
concat_bpe_32000 concatenated BPE 32000 1.3912 3.84 0.1340 99.00% 0.7568 4.0970 0.1146
shared_unigram_32000 shared Unigram 32000 1.5034 3.56 0.0193 99.00% 0.8325 4.0979 0.1482
concat_unigram_32000 concatenated Unigram 32000 1.4304 3.74 0.1449 99.00% 0.8069 4.1112 0.1420
shared_wordpiece_32000 shared WordPiece 32000 1.4501 3.69 0.0724 99.00% 0.7886 4.0696 0.1273
concat_wordpiece_32000 concatenated WordPiece 32000 1.3885 3.85 0.1309 99.00% 0.7419 4.0975 0.1329
shared_bbpe_32000 shared BBPE 32000 1.6737 3.19 0.3737 99.00% 0.8431 4.0974 0.1737
concat_bbpe_32000 concatenated BBPE 32000 1.6299 3.28 0.2067 99.00% 0.8217 4.1216 0.1798

Morphological Metrics (Arabic-script only)

Name Algorithm Type Vocab μe Precision Recall F1
shared_bpe_8000 BPE shared 8000 4.1755 0.1706 0.1557 0.1617
concat_bpe_8000 BPE concatenated 8000 4.2158 0.1570 0.1856 0.1680
shared_unigram_8000 Unigram shared 8000 4.1823 0.1499 0.2197 0.1770
concat_unigram_8000 Unigram concatenated 8000 4.2111 0.1451 0.2405 0.1801
shared_wordpiece_8000 WordPiece shared 8000 4.1766 0.1637 0.1494 0.1552
concat_wordpiece_8000 WordPiece concatenated 8000 4.2191 0.1522 0.1758 0.1619
shared_bbpe_8000 BBPE shared 8000 4.1921 0.1550 0.2827 0.1995
concat_bbpe_8000 BBPE concatenated 8000 4.2317 0.1589 0.3283 0.2136
shared_bpe_16000 BPE shared 16000 4.1165 0.1646 0.1113 0.1324
concat_bpe_16000 BPE concatenated 16000 4.1531 0.1544 0.1249 0.1376
shared_unigram_16000 Unigram shared 16000 4.1315 0.1558 0.1706 0.1623
concat_unigram_16000 Unigram concatenated 16000 4.1582 0.1464 0.1741 0.1575
shared_wordpiece_16000 WordPiece shared 16000 4.1158 0.1654 0.1210 0.1385
concat_wordpiece_16000 WordPiece concatenated 16000 4.1506 0.1740 0.1384 0.1533
shared_bbpe_16000 BBPE shared 16000 4.1365 0.1546 0.2261 0.1824
concat_bbpe_16000 BBPE concatenated 16000 4.1720 0.1620 0.2600 0.1986
shared_bpe_32000 BPE shared 32000 4.0706 0.1610 0.0899 0.1149
concat_bpe_32000 BPE concatenated 32000 4.0970 0.1487 0.0942 0.1146
shared_unigram_32000 Unigram shared 32000 4.0979 0.1631 0.1383 0.1482
concat_unigram_32000 Unigram concatenated 32000 4.1112 0.1486 0.1393 0.1420
shared_wordpiece_32000 WordPiece shared 32000 4.0696 0.1814 0.0999 0.1273
concat_wordpiece_32000 WordPiece concatenated 32000 4.0975 0.1636 0.1126 0.1329
shared_bbpe_32000 BBPE shared 32000 4.0974 0.1571 0.1976 0.1737
concat_bbpe_32000 BBPE concatenated 32000 4.1216 0.1574 0.2144 0.1798

Key Findings

  • Concatenated tokenizers reduce cross-script fertility disparity compared to shared-vocabulary tokenizers, at the cost of larger total vocabulary.
  • BBPE achieves 100% exact reconstruction by design (byte-level guarantees).
  • Metaspace-based tokenizers (BPE/Unigram) achieve >95% exact reconstruction.
  • WordPiece exact reconstruction is lower due to inherent whitespace handling limitations.
  • Gini coefficients are correctly bounded in [0, 1], indicating healthy vocabulary utilization.
  • Morphological edit distance (μe) and consistency (μc) provide complementary signals for evaluating tokenization quality beyond surface-level fertility.
  • Unigram models tend to produce lower fertility (fewer tokens per word) at the cost of higher morphological edit distance.

Files

  • tokenizer_results.csv / .json — Complete metrics for all 24 tokenizers
  • bootstrap_ci.csv — Bootstrap 95% confidence intervals for fertility and CPT
  • morphology/farasa_segmentations.json — Cached Farasa morphological segmentations
  • tokenizers/ — Raw tokenizer JSON files (HuggingFace tokenizers library format)
  • transformers_tokenizers/ — 28 tokenizer directories ready for transformers library
  • plots/ — 24 visualization PNGs (comparison bars, faceted, trends, heatmaps, bootstrap forest)
  • script.py — Complete benchmark script (reproducible)