%
% ArabicNLP 2026 Submission (Anonymous Review)
% Fair Subword Tokenization Benchmark for Multi-Script Moroccan Darija
%
\documentclass[11pt]{article}
\usepackage[review]{acl}
\usepackage{times}
\usepackage{latexsym}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{microtype}
\usepackage{enumitem}
\usepackage{booktabs}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{xcolor}
\usepackage{multirow}
\usepackage{tikz}
\usetikzlibrary{shapes.geometric, arrows.meta, positioning, calc, fit, backgrounds}
\usepackage{placeins}
\usepackage{stfloats}
\usepackage{subcaption}

% Path for figures
\graphicspath{{figures/}}

\title{Fair Subword Tokenization Benchmark for Multi-Script Moroccan Darija}

\author{Anonymous Submission}

\date{}

\begin{document}
\maketitle

\begin{abstract}
Moroccan Darija is written concurrently in Arabic script and Arabizi (Latin), with no dedicated tokenizer benchmarks. Existing Arabic tokenizers---trained on Modern Standard Arabic---produce 40--50\% higher fertility on Darija. Even Darija-specific tokenizers suffer severe cross-script imbalance. We present the first systematic tokenizer benchmark for this language: 40 configurations spanning four algorithms, two architectures, and five vocabulary sizes (8K--110K), evaluated on 112,814 sentence pairs. At matching vocabulary sizes, our tokenizers achieve 27--34\% lower fertility than DarijaBERT while maintaining $\geq$99\% exact reconstruction. Shared Unigram reaches near-perfect cross-script fairness ($\Delta F$ = 0.015), and vocabulary saturation occurs by 80K. We release all 40 tokenizers, scripts, and data at \texttt{OiQ/daa-tokenizers}.
\end{abstract}

\section{Introduction}

Tokenization is the first operation in every language model pipeline. It is also the most underestimated. A poor tokenizer inflates sequence lengths, increases training cost, and embeds script bias before the model sees a single gradient \citep{sennrich2016bpe}.

Moroccan Darija exposes these vulnerabilities acutely. It is a low-resource dialect with no official spelling standard. Speakers write it concurrently in two scripts:
\begin{itemize}[noitemsep,topsep=2pt]
\item \textbf{Arabic script}: \textit{``wa\v{s} k\=ayn \v{s}y jad\=id?''} (Arabic orthography)
\item \textbf{Arabizi}: \textit{``wash kayn shi jdid?''}---a Latin-based phonetic system
\end{itemize}

No tokenizer benchmark exists for Darija. Practitioners building Darija language models must choose among tokenizers trained on Modern Standard Arabic (MSA)---which allocate vocabulary slots to patterns absent in Darija---or Darija-specific tokenizers that serve only one script. Neither option has been systematically evaluated. The problem of tokenizer fairness in multilingual settings is well studied: \citet{rust2021howgood} showed that multilingual models under-serve low-resource languages, and \citet{chung2023unimax} confirmed that vocabulary size alone does not resolve imbalance. Yet these findings address inter-lingual competition between languages. Darija presents a distinct challenge: intra-lingual competition between scripts of the same language.

\citet{dhawan2023concatenated} addressed script competition in code-switching speech recognition by training separate tokenizers per language and concatenating their vocabularies. We adapt this idea to Darija's script pair: one sub-tokenizer for Arabic script and one for Arabizi, each with half the total vocabulary. This concatenated architecture eliminates zero-sum competition for vocabulary slots. Separately, morphology-aware tokenization methods like MorphBPE \citep{asgari2025morphbpe} constrain merges to respect morpheme boundaries. We evaluate morphological alignment as a metric but do not constrain training, leaving that to future work.

We fill this gap with the first comprehensive tokenizer benchmark for Moroccan Darija. Our benchmark covers:
\begin{enumerate}[noitemsep,topsep=2pt]
\item \textbf{Scale}: 40 tokenizer configurations (4 algorithms $\times$ 2 architectures $\times$ 5 vocabulary sizes from 8K to 110K)
\item \textbf{Comparison}: Fair head-to-head evaluation against ten existing Arabic and Darija tokenizers from HuggingFace, including DarijaBERT \citep{gaanoun2023darijabert} at matching vocabulary sizes (80K and 110K)
\item \textbf{Metrics}: Compression (fertility, characters per token), cross-script fairness (relative disparity, Gini coefficient), morphological fidelity (alignment with Farasa morpheme boundaries), and exact reconstruction
\item \textbf{Proposed improvement}: A concatenated architecture that trains separate per-script sub-tokenizers, reducing cross-script disparity by up to $4\times$
\end{enumerate}

Our key findings are threefold. First, Darija-specific tokenization dramatically improves compression: our 80K tokenizer achieves 33\% lower fertility than DarijaBERT at the same vocabulary size. Second, cross-script fairness is achievable for Darija: shared Unigram at 80K reaches $\Delta F$ = 0.015 (near-zero disparity), while concatenated architectures provide consistent fairness across all algorithms. Third, vocabulary size saturates early: moving from 32K to 110K yields only marginal gains, suggesting that 32K is sufficient for most Darija applications. All 40 configurations achieve $\geq$99\% exact reconstruction.


\section{Methodology}

\subsection{Dataset}

Moroccan Darija lacks large-scale parallel corpora across scripts. Existing resources---DODa (100K entries) \citep{outchakoucht2021}, AtlasIA (1.17M rows) \citep{atlasia2024}, and Goud (158K articles) \citep{issam2022}---are either monolingual or not script-parallel. We built \texttt{OiQ/daa-pairs} to fill this gap.

Figure~\ref{fig:data_pipeline} shows how we constructed the dataset. We started with two raw sources: Moroccan YouTube transcripts (\texttt{OiQ/MoR-ytb.small}) and Moroccan news articles (\texttt{OiQ/goud-moroccan-news}). An LLM agent---Google Gemma 4 via OpenRouter---reformulated each source text into authentic Darija phrases. The agent produced three script variants per phrase: Arabic script, Arabizi (Latin), and a mixed version with code-switching. Each source yielded 5--20 independent phrases. We then flattened the nested output and shuffled segment order within each source to remove positional bias. The final dataset contains 112,814 phrase triplets from 12,695 unique sources.

\begin{figure}[!htbp]
\centering
\resizebox{\columnwidth}{!}{%
\begin{tikzpicture}[
    box/.style={rectangle, draw=gray!80, rounded corners, minimum width=2.8cm, minimum height=0.9cm, align=center, font=\scriptsize\bfseries, fill=white, thick},
    smallbox/.style={rectangle, draw=gray!60, rounded corners, minimum width=2.4cm, minimum height=0.7cm, align=center, font=\tiny, fill=white, thick},
    arrow/.style={-{Stealth[length=2mm]}, thick},
    label/.style={font=\tiny\itshape, text=gray!70, align=center}
]
% Row 1: Raw sources
\node[smallbox, fill=blue!5] (ytb) at (-2.5, 0) {YouTube\\Transcripts};
\node[smallbox, fill=blue!5] (news) at (2.5, 0) {News Articles};

% Row 2: Unified corpus
\node[box, fill=blue!10] (unify) at (0, -1.5) {Unified Corpus\\(12,695 sources)};

% Row 3: LLM generation
\node[box, fill=orange!15] (llm) at (0, -3.0) {LLM Reformulation\\(Gemma 4 via OpenRouter)};

% Row 4: Three script variants
\node[smallbox, fill=teal!5] (ar) at (-3.0, -4.5) {Arabic Script};
\node[smallbox, fill=teal!5] (az) at (0, -4.5) {Arabizi (Latin)};
\node[smallbox, fill=teal!5] (mx) at (3.0, -4.5) {Mixed Script};

% Row 5: Post-processing
\node[box, fill=green!10] (flat) at (0, -6.0) {Flatten + Randomize\\(per source)};

% Row 6: Final dataset
\node[box, fill=green!15, minimum width=4cm] (final) at (0, -7.5) {\texttt{OiQ/daa-pairs}\\112,814 triplets};

% Arrows
\draw[arrow] (ytb.south) -- ++(0,-0.3) -| (unify.north);
\draw[arrow] (news.south) -- ++(0,-0.3) -| (unify.north);
\draw[arrow] (unify) -- (llm);
\draw[arrow] (llm.south) -- ++(0,-0.3) -| (ar.north);
\draw[arrow] (llm.south) -- ++(0,-0.3) -| (az.north);
\draw[arrow] (llm.south) -- ++(0,-0.3) -| (mx.north);
\draw[arrow] (ar.south) -- ++(0,-0.3) -| (flat.north);
\draw[arrow] (az.south) -- ++(0,-0.3) -| (flat.north);
\draw[arrow] (mx.south) -- ++(0,-0.3) -| (flat.north);
\draw[arrow] (flat) -- (final);

% Labels
\node[label] at (-4.2, 0) {Raw Sources};
\node[label] at (-4.2, -1.5) {Unification};
\node[label] at (-4.2, -3.0) {Generation};
\node[label] at (-4.2, -4.5) {Script Variants};
\node[label] at (-4.2, -6.0) {Post-Process};
\node[label] at (-4.2, -7.5) {Output};

\end{tikzpicture}%
}
\caption{Synthetic data generation pipeline for \texttt{OiQ/daa-pairs}. Raw transcripts and articles feed an LLM agent that reformulates content into authentic Darija. Each source produces three script variants. Post-processing flattens and randomizes segment order to remove positional bias.}
\label{fig:data_pipeline}
\end{figure}

We split the dataset 80/10/10 (train/validation/test) with stratified sampling, yielding 11,282 test triplets. Each split preserves the Arabic--Arabizi pairing. During evaluation, we classify each sentence by script using character-level heuristics; all three script variants (Arabic, Arabizi, mixed) are evaluated, producing 33,846 test texts (11,282 $\times$ 3). The mixed-script column ($mixte$) is excluded from tokenizer training; we train only on the Arabic and Arabizi columns to ensure clean script separation.

\subsection{Algorithms}

Raw sentences pass through script detection, then route to shared or concatenated tokenizers. All outputs are evaluated on compression, fairness, and---for Arabic-script---morphological fidelity. We benchmark four subword algorithms:

\textbf{BPE} \citep{sennrich2016bpe}: Iteratively merges the most frequent adjacent pair. Pre-tokenized with Metaspace (SentencePiece-style). Decoder uses Metaspace replacement for reconstruction.

\textbf{WordPiece} \citep{schuster2012japanese}: Selects merges that maximize likelihood gain. Same Metaspace pre-tokenization as BPE.

\textbf{Unigram} \citep{kudo2018subword}: Models tokens as independent unigram variables with a base vocabulary of all characters plus special tokens. Uses expectation-maximization for training.

\textbf{BBPE} \citep{bostrom2020byte}: Byte-level BPE operating on UTF-8 bytes. Pre-tokenized with ByteLevel. Guarantees zero OOV by construction.

\textbf{Shared.} All four algorithms train on mixed-script corpora using the full vocabulary $V$.

\textbf{Concatenated.} Two sub-tokenizers per algorithm: $T_{\text{ar}}$ on Arabic-script corpus ($V/2$ vocabulary), $T_{\text{az}}$ on Arabizi corpus ($V/2$ vocabulary). Arabizi token IDs are shifted by $|V_{\text{ar}}|$ to prevent index collision.

We evaluate five vocabulary sizes: 8K, 16K, 32K, 80K, and 110K. The larger sizes (80K, 110K) match DarijaBERT \citep{gaanoun2023darijabert} for direct comparison. This yields $4 \times 2 \times 5 = 40$ tokenizer configurations. We release all 40 via \texttt{OiQ/daa-tokenizers} on HuggingFace in both raw and \texttt{transformers}-compatible formats.

Every tokenizer uses matched pre-tokenizer/decoder pairs (Metaspace for BPE, Unigram, and WordPiece; ByteLevel for BBPE) to ensure faithful round-trips. We compute characters-per-token using Unicode grapheme clusters to handle Arabic combining diacritics. Each test sentence is classified by script and evaluated separately, exposing cross-script disparities. We report bootstrap 95\% confidence intervals (500 resamples) for all fertility and CPT estimates.

% ====================================================================
% TABLE 1: Compact benchmark results (best-per-size + key fairness)
% ====================================================================

\begin{table*}[t]
\centering
\small
\setlength{\tabcolsep}{4pt}
\begin{tabular}{@{}llccrrrrr@{}}
\toprule
\textbf{V} & \textbf{Config} & \textbf{Algo} & \textbf{Fert$\downarrow$} & \textbf{CPT$\uparrow$} & \textbf{Gini} & \textbf{$F_{\text{ar}}$} & \textbf{$F_{\text{az}}$} & \textbf{$\Delta F$$\downarrow$} \\
\midrule
\multicolumn{8}{@{}l}{\textit{Best compression per vocabulary size}} \\
\midrule
8K & Shared & WordPiece & 1.572 & 3.535 & 0.725 & 1.472 & 1.760 & 0.164 \\
8K & Concat & WordPiece & 1.631 & 3.398 & 0.694 & 1.680 & 1.538 & 0.084 \\
16K & Shared & WordPiece & 1.402 & 3.949 & 0.757 & 1.328 & 1.540 & 0.138 \\
16K & Concat & BPE & 1.443 & 3.843 & 0.750 & 1.486 & 1.361 & 0.084 \\
32K & Shared & WordPiece & 1.274 & 4.320 & 0.775 & 1.227 & 1.362 & 0.099 \\
32K & Concat & WordPiece & 1.303 & 4.240 & 0.769 & 1.342 & 1.231 & 0.083 \\
80K & Shared & WordPiece & 1.171 & 4.676 & 0.784 & 1.150 & 1.210 & 0.049 \\
80K & Concat & WordPiece & 1.183 & 4.651 & 0.790 & 1.221 & 1.111 & 0.090 \\
110K & Shared & WordPiece & 1.171 & 4.676 & 0.784 & 1.150 & 1.210 & 0.049 \\
110K & Concat & WordPiece & 1.155 & 4.758 & 0.792 & 1.193 & 1.082 & 0.093 \\
\midrule
\multicolumn{8}{@{}l}{\textit{Key fairness configurations}} \\
\midrule
80K & Shared & Unigram & 1.253 & 4.371 & 0.817 & 1.247 & 1.266 & \textbf{0.015} \\
8K & Shared & BBPE & 1.716 & 3.268 & 0.785 & 1.543 & 2.038 & 0.243 \\
80K & Shared & BBPE & 1.392 & 3.994 & 0.836 & 1.268 & 1.624 & 0.219 \\
8K & Concat & BBPE & 1.770 & 3.130 & 0.765 & 1.733 & 1.838 & 0.057 \\
\bottomrule
\end{tabular}
\caption{Benchmark results for selected configurations. Fert = fertility (tokens per word, lower is better). CPT = grapheme clusters per token (higher is better). Gini = vocabulary usage inequality. $F_{\text{ar}}$/$F_{\text{az}}$ = per-script fertility. $\Delta F$ = relative cross-script disparity (lower is better). Bold = best fairness. Overall fertility is word-count-weighted: $F \approx 0.65\,F_{\text{ar}} + 0.35\,F_{\text{az}}$. Full 40-configuration results are in the supplementary material.}
\label{tab:benchmark_results}
\end{table*}

% ====================================================================
% OLD TABLE 3 removed (merged into Table 1 above)
% ====================================================================


\section{Evaluation Metrics}

\subsection{Compression}

\textbf{Fertility} ($F$): tokens per word, averaged over all words in the test set. Lower is better (fewer tokens per word = more efficient encoding). \textbf{CPT}: grapheme clusters per token, computed using Unicode standard segmentation. Higher is better (each token covers more characters).

\subsection{Cross-Script Fairness}
\label{sec:fairnessmetrics}

A tokenizer may achieve good overall compression while treating one script unfairly---encoding it with many more tokens per word. This matters because biased tokenization creates downstream disparities: models generate shorter, more fluent output for the favored script and struggle with the disadvantaged one \citep{rust2021howgood}. We measure fairness along two dimensions:

\textbf{Fertility disparity} ($\Delta F = \frac{|F_{\text{ar}} - F_{\text{az}}|}{\max(F_{\text{ar}}, F_{\text{az}})}$): relative difference in per-script fertility. Zero indicates perfect cross-script fairness. This metric is dimensionless and comparable across vocabulary sizes.

\textbf{Gini coefficient} ($G$): measures vocabulary usage inequality. Given sorted token frequencies $f_1 \leq f_2 \leq \ldots \leq f_n$:
\begin{equation}
G = \frac{n + 1 - 2 \sum_{i=1}^{n} \sum_{j=1}^{i} f_j / \sum_{j=1}^{n} f_j}{n}
\end{equation}
where $n$ is the vocabulary size. $G = 0$ indicates uniform token usage; $G = 1$ indicates a single token dominates all usage. We use ascending-sort normalization to bound $G$ to $[0,1]$.

\textbf{Exact match}: proportion of test sentences that perfectly round-trip through encode $\rightarrow$ decode.

\subsection{Morphological Fidelity}

Arabic-script Darija shares much of its morphology with Modern Standard Arabic: prefixes (e.g., \textit{wa-}, \textit{bi-}), templates (\textit{fa'$\backslash$al}), and suffixes (\textit{-at}, \textit{-na}). Farasa \citep{farasa2015} provides high-quality morphological segmentation for Arabic, and since Darija's Arabic-script morphology closely overlaps with MSA, Farasa's analyses transfer reasonably well. Measuring alignment between tokenizer boundaries and morpheme boundaries thus provides a proxy for how linguistically meaningful the tokenization is---a property that matters for downstream tasks like morphological analysis and inflection generation.

We measure alignment between tokenizer boundaries and gold-standard morpheme boundaries from Farasa \citep{farasa2015} for Arabic-script text.

\textbf{Morphological edit distance} ($\mu_e$): given a word segmented into tokens $[t_1, \ldots, t_n]$ by the tokenizer and into morphemes $[m_1, \ldots, m_k]$ by Farasa, we compute the ordered Levenshtein alignment cost between the two boundary sequences, averaged over all Arabic-script words in the test set. Lower is better.

\textbf{Morphological consistency F1} ($\mu_c$): measures whether words that share morphemes also share tokens. We construct morphological TF-IDF vectors from Farasa segmentations, cluster words by morphological similarity ($K$ clusters via $K$-Means), sample word pairs from each cluster, and compute precision/recall/F1 for token sharing. Higher is better.

\FloatBarrier
\section{Results}

\subsection{Exact Reconstruction}

All 40 tokenizer configurations achieve $\geq$99\% exact match on the test set, validating our matched pre-tokenizer/decoder design.

\subsection{Compression Performance}

Table~\ref{tab:benchmark_results} reports compression and disparity metrics for selected configurations. Several patterns emerge.

Fertility improves with vocabulary size, then saturates. Moving from 8K to 32K reduces fertility by 19--22\% for BPE and WordPiece. Beyond 80K, gains become negligible: shared WordPiece yields F = 1.171 at both 80K and 110K. The training corpus exhausts its merge candidates around 80K. This means 32K captures most available compression for Darija, with only marginal returns at larger sizes.

BPE and WordPiece consistently achieve the lowest fertility at every vocabulary size. BBPE produces the highest (1.307--1.770 across configurations), while Unigram falls between the two groups. This ranking matches findings in higher-resource languages \citep{bostrom2020byte}.

The tradeoff between shared and concatenated architectures shifts with vocabulary size. At smaller vocabularies (8K, 16K), shared tokenizers benefit from pooled vocabulary and achieve lower fertility. At larger sizes, the gap narrows. Concatenated WordPiece at 110K (F = 1.155) even surpasses its shared counterpart (F = 1.171), because each sub-tokenizer has sufficient capacity for its script.

\subsection{Cross-Script Fairness}

Shared BBPE creates extreme script bias. At 8K, Arabizi texts require 32\% more tokens per word than Arabic texts ($\Delta F$ = 0.243). Concatenation cuts this to 0.057---a fourfold improvement. This pattern persists at all sizes (shared BBPE disparity: 0.219--0.243).

Shared Unigram tells the opposite story. At 80K and 110K, it achieves $\Delta F$ = 0.015---the lowest disparity of any configuration---without concatenation. Its probabilistic model naturally balances vocabulary across both scripts during joint training.

Concatenation delivers consistent fairness across algorithms. All concatenated non-BBPE configurations maintain $\Delta F \leq 0.094$ across all vocabulary sizes. Their shared counterparts range from 0.098 to 0.164. The cost is slightly higher fertility at smaller vocabularies.

Vocabulary size also reduces disparity for shared tokenizers. Shared BPE disparity drops from 0.161 at 8K to 0.053 at 80K, as larger vocabularies accommodate both scripts. For concatenated tokenizers, disparity remains stable (0.084--0.094), since each sub-tokenizer already has dedicated vocabulary.

Figure~\ref{fig:fertility_comparison} visualizes compression and disparity trends. The left panel compares fertility across algorithms, with BBPE consistently highest. The right panel shows the saturation effect: gains diminish sharply beyond 32K.

\begin{figure*}[!t]
\centering
\begin{subfigure}[b]{0.48\textwidth}
  \centering
  \includegraphics[width=\textwidth]{fertility_overall_comparison_v2.png}
  \caption{Fertility by algorithm and vocabulary size. Solid bars = shared, hatched bars = concatenated. Lower is better.}
  \label{fig:fert_comp}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.48\textwidth}
  \centering
  \includegraphics[width=\textwidth]{fertility_overall_trends.png}
  \caption{Fertility trends across vocabulary sizes. Lines show improvement from 8K to 32K for each algorithm.}
  \label{fig:fert_trend}
\end{subfigure}
\caption{Overall fertility comparison. BPE and WordPiece achieve the best compression. Larger vocabularies improve compression with diminishing returns beyond 32K.}
\label{fig:fertility_comparison}
\end{figure*}

Figure~\ref{fig:disparity_plots} illustrates cross-script disparity patterns. BBPE (shared) shows extreme bias at all sizes, dramatically reduced by concatenation. Shared Unigram achieves the lowest disparity, particularly at larger vocabularies.

\begin{figure*}[!t]
\centering
\begin{subfigure}[b]{0.48\textwidth}
  \centering
  \includegraphics[width=\textwidth]{fertility_disparity_comparison_v2.png}
  \caption{Cross-script disparity by algorithm. BBPE (shared) shows extreme bias ($\Delta F$ > 0.22). Concatenation reduces BBPE disparity by 4$\times$.}
  \label{fig:disp_comp}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.48\textwidth}
  \centering
  \includegraphics[width=\textwidth]{fertility_disparity_heatmap_v2.png}
  \caption{Disparity heatmap. Shared BBPE rows are consistently the brightest (highest disparity).}
  \label{fig:disp_heat}
\end{subfigure}
\caption{Cross-script fertility disparity analysis. BBPE in shared mode creates severe script imbalance; concatenation is essential for this algorithm. Unigram achieves the lowest shared-mode disparity at larger vocabularies ($\Delta F$ = 0.015 at 80K/110K).}
\label{fig:disparity_plots}
\end{figure*}

\subsection{Code-Switching Performance}

The \texttt{OiQ/daa-pairs} dataset includes a \textit{mixte} column of naturally code-switched text (32.1\% of test texts). Mixed-script texts require 7--14\% more tokens per word than pure-script texts (e.g., concat BPE 32K: $F_{\text{mi}}$ = 1.404 vs $F_{\text{ar}}$ = 1.308), reflecting the additional burden of encoding intra-sentence script transitions. All our tokenizers achieve $\geq$99.5\% exact match on mixed texts. External tokenizers fare worse: DarijaBERT-ar achieves only 5.7\% exact match on mixed texts, while Qwen2.5-Darija achieves 100\% but at 60\% higher fertility than our 32K tokenizer.

\subsection{Morphological Fidelity}

For Arabic-script Darija at 32K, morphological edit distance ($\mu_e$) varies modestly across algorithms (4.070--4.122), indicating similar morphological boundary alignment. Morphological consistency F1 ($\mu_c$) shows a clearer algorithm-level distinction: BBPE achieves 0.174--0.180 (highest), while BPE achieves only 0.115 (lowest). The overall $\mu_c$ range (0.11--0.21) indicates that standard subword algorithms have limited morphological awareness, motivating future morphology-constrained approaches such as MorphBPE \citep{asgari2025morphbpe}. Detailed morphological results are provided in the supplementary material.

\subsection{Statistical Robustness}

Bootstrap 95\% confidence intervals (500 resamples) confirm that all reported metrics are stable. Fertility CIs are within $\pm$0.015 and CPT CIs within $\pm$0.03 across all 40 configurations.

\section{Comparison with Existing Tokenizers}

To contextualize our benchmark, we compare our best tokenizers against ten existing Arabic and Darija tokenizers from HuggingFace. The MSA group includes CaMeLBERT-MSA \citep{obeid2021camelbert} (30K WordPiece, 167 GB MSA), Asafaya-BERT \citep{safaya2020kuisail} (32K WordPiece, 95 GB Arabic), Aranizer-SP-86k \citep{koubaa2024arabian} (86K SentencePiece), and B2BERT \citep{b2bert2025} (30K, reuses CaMeLBERT-MSA tokenizer). The Darija group includes DarijaBERT \citep{gaanoun2023darijabert} (80K and 110K WordPiece variants), DarijaBERT-mix (160K WordPiece), Moroccan-Darija-Tokenizer (30K BPE), Translit-Darija (30K BPE), and Qwen2.5-Darija \citep{skiredj2025gemmaroc} (152K SentencePiece). Table~\ref{tab:comparison} reports the results.

\begin{table*}[t]
\centering
\small
\setlength{\tabcolsep}{3pt}
\begin{tabular}{@{}llccccccccc@{}}
\toprule
  & & & \multicolumn{2}{c}{\textbf{Fertility}} & & \multicolumn{2}{c}{\textbf{CPT}} & \multicolumn{2}{c}{\textbf{Exact Match}} \\
\cmidrule(lr){3-3}\cmidrule(lr){4-5}\cmidrule(lr){6-6}\cmidrule(lr){7-8}\cmidrule(lr){9-10}
\textbf{Tokenizer} & \textbf{HF Repo} & \textbf{V} & $F_{\text{ar}}$ & $F_{\text{az}}$ & $\Delta F$ & \textbf{CPT$_{\text{ar}}$} & \textbf{CPT$_{\text{az}}$} & \textbf{Ar} & \textbf{Az} \\
\midrule
\multicolumn{9}{@{}l}{\textbf{Ours (Darija-specific, concatenated architecture)}} \\
\midrule
concat BPE 32K & \texttt{OiQ/daa-tokenizers} & 32K & 1.346 & 1.233 & 0.084 & 4.19 & 4.31 & 99.9 & 99.6 \\
concat WP 80K & \texttt{OiQ/daa-tokenizers} & 80K & 1.221 & 1.111 & 0.090 & 4.59 & 4.76 & 99.9 & 99.6 \\
concat WP 110K & \texttt{OiQ/daa-tokenizers} & 110K & 1.193 & 1.082 & 0.093 & 4.69 & 4.89 & 99.9 & 99.6 \\
\midrule
\multicolumn{9}{@{}l}{\textbf{MSA tokenizers}} \\
\midrule
CaMeLBERT-MSA & \texttt{CAMeL-Lab/bert-base-arabic-camelbert-msa} & 30K & 1.818 & 3.173 & 0.427 & 3.11 & 1.64 & 29.9 & 38.9 \\
Asafaya-BERT & \texttt{asafaya/bert-base-arabic} & 32K & 1.795 & 2.795 & 0.358 & 3.13 & 1.86 & 19.9 & 15.2 \\
Aranizer-SP-86k & \texttt{riotu-lab/Aranizer-SP-86k} & 86K & 1.594 & 2.523 & 0.368 & 3.53 & 2.07 & 99.8 & 99.6 \\
B2BERT & \texttt{AHAAM/B2BERT} & 30K & 1.817 & 3.173 & 0.427 & 3.11 & 1.64 & 29.9 & 38.9 \\
\midrule
\multicolumn{9}{@{}l}{\textbf{Darija tokenizers}} \\
\midrule
DarijaBERT-ar & \texttt{SI2M-Lab/DarijaBERT} & 80K & 1.417 & 2.404 & 0.410 & 3.98 & 2.17 & 13.7 & 8.0 \\
DarijaBERT-az & \texttt{SI2M-Lab/DarijaBERT-arabizi} & 110K & 1.605 & 1.517 & \textbf{0.055} & 3.48 & 3.45 & 14.8 & 8.0 \\
DarijaBERT-mix & \texttt{SI2M-Lab/DarijaBERT-mix} & 160K & 1.333 & 1.567 & 0.149 & 4.19 & 3.34 & 14.8 & 8.0 \\
Moroccan-Darija-Tok & \texttt{BounharAbdelaziz/Moroccan-Darija-Tokenizer} & 30K & 1.570 & 2.902 & 0.459 & 3.63 & 1.79 & 0.0 & 0.0 \\
Translit-Darija & \texttt{atlasia/Transliteration-Moroccan-Darija} & 30K & 1.796 & 1.658 & 0.077 & 3.11 & 3.16 & 0.0 & 0.0 \\
Qwen2.5-Darija & \texttt{GemMaroc/Qwen2.5-7B-Instruct-darija} & 152K & 2.340 & 2.246 & 0.040 & 2.38 & 2.33 & 100.0 & 100.0 \\
\bottomrule
\end{tabular}
\caption{Comparison of our best Darija tokenizers against ten existing Arabic and Darija tokenizers from HuggingFace. All evaluated on the same Darija test set (33,846 texts). Our 80K and 110K tokenizers match DarijaBERT's vocabulary sizes for direct comparison. $\Delta F$ = relative cross-script disparity (\S\ref{sec:fairnessmetrics}). Bold = lowest disparity.}
\label{tab:comparison}
\end{table*}

Three findings stand out. First, Darija-specific tokenization dramatically improves compression at matching vocabulary sizes. Our 80K tokenizer (F = 1.183) achieves 33\% lower fertility than DarijaBERT-ar (F = 1.761) at the same vocabulary size. Our 110K tokenizer (F = 1.155) achieves 27\% lower fertility than DarijaBERT-az (F = 1.575). Even our 32K tokenizer (1.307) outperforms DarijaBERT-az (1.575) despite using 3.4$\times$ fewer vocabulary slots. DarijaBERT-mix, despite its massive 160K vocabulary (F = 1.414), still underperforms our 32K tokenizer---vocabulary size alone cannot compensate for suboptimal training architecture.

Second, MSA-trained tokenizers transfer poorly to Darija. They allocate vocabulary to MSA patterns absent in Darija, inflating fertility by 32--43\%. Our smallest tokenizer (F = 1.307) outperforms all four MSA tokenizers, including Aranizer-SP-86k with 86K vocabulary (F = 1.918). Qwen2.5-Darija (F = 2.307) confirms that general-purpose multilingual tokenization is unsuitable for Darija regardless of vocabulary size.

Third, exact reconstruction fails for most external tokenizers. BERT-based models achieve under 40\% exact match; DarijaBERT variants under 15\%; Moroccan-Darija-Tokenizer 0\%. Only Qwen2.5-Darija achieves 100\%---but at 77\% higher fertility than our 32K tokenizer and double the fertility of our 110K.

Figure~\ref{fig:comparison} visualizes these comparisons. The fertility panel shows a clear gap between our tokenizers and all external baselines. The disparity panel confirms that our architectures achieve balanced cross-script treatment, while most external tokenizers exhibit severe imbalance.

\begin{figure*}[!t]
\centering
\includegraphics[width=0.95\textwidth]{external_comparison.png}
\caption{Comparison of our best tokenizers against ten existing Arabic/Darija tokenizers. Top-left: fertility (lower = better). Top-right: cross-script disparity (lower = better). Bottom-left: exact match rate by script. Bottom-right: characters per token (higher = better). Our tokenizers achieve low fertility, balanced disparity, and $\geq$99\% exact match.}
\label{fig:comparison}
\end{figure*}

\subsection{Arabizi Validation on DODa}

We evaluate all tokenizers on \texttt{atlasia/DODa} \citep{outchakoucht2021}, a Moroccan Darija dictionary (87,743 Arabizi entries) not used in training. Since our tokenizers are trained on Arabizi text, DODa serves as an in-domain but distributionally distinct validation set---a dictionary of individual words rather than the sentence-level training data.

Table~\ref{tab:doda} reports fertility on a 10K random DODa subset. Our tokenizers show higher fertility on DODa (F = 1.91--2.25) than on our test set (F = 1.16--1.31), reflecting the lexical diversity of a dictionary versus sentence-level text. DarijaBERT-az (F = 1.31) achieves the lowest fertility on DODa, consistent with its Arabizi-specific training. MSA tokenizers remain the worst performers (F = 3.19--3.67).

\begin{table}[!htbp]
\centering
\footnotesize
\setlength{\tabcolsep}{4pt}
\begin{tabular}{@{}llccl@{}}
\toprule
\textbf{Tokenizer} & \textbf{V} & \textbf{Fert$\downarrow$} & \textbf{CPT$\uparrow$} & \textbf{EM} \\
\midrule
\multicolumn{4}{@{}l}{\textbf{Ours}} \\
concat BPE 32K & 32K & 1.909 & 3.19 & 100.00\% \\
concat WP 80K & 80K & 1.768 & 3.55 & 99.97\% \\
concat WP 110K & 110K & 1.735 & 3.63 & 99.97\% \\
\midrule
\multicolumn{4}{@{}l}{\textbf{Best external}} \\
DarijaBERT-az & 110K & \textbf{1.314} & \textbf{4.55} & 94.4\% \\
Translit-Darija & 30K & 1.869 & 3.21 & 3.0\% \\
\midrule
\multicolumn{4}{@{}l}{\textbf{MSA tokenizers}} \\
CaMeLBERT-MSA & 30K & 3.667 & 1.59 & 98.4\% \\
Asafaya-BERT & 32K & 3.187 & 1.83 & 94.6\% \\
\bottomrule
\end{tabular}
\caption{Arabizi validation on \texttt{atlasia/DODa} (10K entries). Our tokenizers show higher fertility than on our sentence-level test set, reflecting the lexical diversity of dictionary data. DarijaBERT-az achieves the lowest fertility, consistent with its Arabizi-specific training.}
\label{tab:doda}
\end{table}

\section{Discussion}

The results reveal a clear hierarchy among subword algorithms for Darija. BPE and WordPiece consistently produce the lowest fertility at every vocabulary size. Their iterative frequency-based merges capture Darija's most common letter sequences efficiently. BBPE, by contrast, operates on raw bytes and pays a steep compression penalty. Its byte-level design guarantees zero out-of-vocabulary tokens, but inflates sequence length by 12--35\% compared to grapheme-based algorithms. This tradeoff between coverage and compression is well documented for high-resource languages \citep{bostrom2020byte}. Darija amplifies it, because its Latin script (Arabizi) mixes ASCII letters with numerals used as phonetic substitutes (e.g., ``7'' for the pharyngeal \textit{h}). BBPE treats each as a separate byte, fragmenting common graphemes.

Cross-script fairness emerges as the most surprising dimension of this benchmark. We expected concatenated architectures to dominate. They do, for BBPE---disparity drops from 0.243 to 0.057 at 8K, a fourfold improvement. But shared Unigram achieves near-perfect fairness ($\Delta F$ = 0.015 at 80K) without any architectural intervention. Its expectation-maximization training naturally distributes probability mass across both scripts. This finding challenges the assumption that multilingual fairness requires vocabulary partitioning. For languages with script overlap (as in Darija's Arabic and Arabizi pair), a well-trained shared model can self-balance.

Vocabulary saturation occurs earlier than expected. Moving from 32K to 80K yields modest gains (2--8\% fertility reduction). Moving from 80K to 110K produces essentially no change. The training corpus of 106K unique words exhausts its merge candidates well before the largest vocabulary sizes. Practitioners can safely choose 32K for most applications, gaining 90\% of the available compression at one-third the vocabulary of DarijaBERT. This mirrors findings in multilingual pretraining, where \citet{chung2023unimax} showed that vocabulary allocation matters more than vocabulary size.

Compared to existing tokenizers, the gap is substantial. Our tokenizers achieve 27--34\% lower fertility than DarijaBERT at matching vocabulary sizes, and 40--50\% lower than MSA-trained tokenizers. The DODa validation adds nuance: DarijaBERT-az achieves lower fertility on dictionary entries (F = 1.31 vs.\ our 1.91), because its vocabulary was trained on a similar Arabizi word distribution. Our tokenizers, trained on sentence-level text, generalize better to running text but pay a cost on isolated words. This distributional difference, not a quality gap, explains the discrepancy.

We recommend concatenated WordPiece at 32K or 110K for best compression. For best fairness in a single vocabulary, shared Unigram at 80K is optimal. Any concatenated non-BBPE configuration maintains $\Delta F \leq 0.094$. Shared BBPE should be avoided due to its extreme cross-script disparity.

\section{Conclusion}

We presented the first systematic tokenizer benchmark for Moroccan Darija. Across 40 configurations, Darija-specific tokenization achieves 27--34\% lower fertility than existing Darija tokenizers at matching vocabulary sizes, and 40--50\% lower than MSA tokenizers. Cross-script fairness is achievable through either shared Unigram ($\Delta F$ = 0.015) or concatenated architectures ($\Delta F \leq 0.094$). Vocabulary size saturates around 80K, with 32K capturing most available compression. All 40 tokenizers achieve $\geq$99\% exact match. These findings, together with the released tokenizers and evaluation code, provide a foundation for building efficient and fair Darija language models.

\section*{Limitations}

This work has several limitations that future research should address. First, we train and evaluate on a single dataset (\texttt{OiQ/daa-pairs}). While it is the largest parallel Arabic--Arabizi corpus available, Darija encompasses regional variation across Morocco. Our data may not represent all dialectal sub-varieties.

Second, morphological fidelity metrics are computed only on Arabic-script text. Arabizi lacks a morphological analyzer comparable to Farasa, so we cannot measure morpheme alignment for the Latin script. This leaves an asymmetric evaluation: compression and fairness metrics cover both scripts, but morphological metrics cover only one.

Third, we measure tokenization quality, not downstream task performance. A tokenizer that achieves low fertility may not produce better embeddings or translations. Downstream evaluation---e.g., language model perplexity or machine translation BLEU---would strengthen the link between tokenizer metrics and practical outcomes.

Fourth, morphology-constrained training (e.g., MorphBPE \citep{asgari2025morphbpe}) was infeasible due to memory constraints. Segmenting the full training vocabulary of 106K unique words exceeded our available RAM. Future work with sufficient hardware should evaluate whether morpheme-aware merges improve downstream tasks without sacrificing compression.

Finally, our concatenated architecture requires script detection at inference time. While we use character-level heuristics for classification, code-switched sentences with ambiguous script boundaries may route to the wrong sub-tokenizer. A learned router could improve robustness on heavily mixed text.

\bibliography{references}

\appendix

\section{Full Benchmark Results}
\label{sec:appendix_full}

Table~\ref{tab:full_results} reports all compression, fairness, and vocabulary metrics for every configuration. Shared tokenizers train on mixed-script corpora with the full vocabulary $V$. Concatenated tokenizers train per-script sub-tokenizers, each with $V/2$ slots.

\begin{table*}[t]
\centering
\scriptsize
\setlength{\tabcolsep}{3pt}
\begin{tabular}{@{}llccrrrrrr@{}}
\toprule
\textbf{Type} & \textbf{Algo} & \textbf{V} & \textbf{Fert$\downarrow$} & \textbf{CPT$\uparrow$} & \textbf{Gini} & \textbf{Ent} & \textbf{$F_{\text{ar}}$} & \textbf{$F_{\text{az}}$} & \textbf{$\Delta F$$\downarrow$} \\
\midrule
\multicolumn{9}{@{}l}{\textit{8K vocabulary}} \\
\midrule
Sh & BPE & 8K & 1.574 & 3.531 & 0.742 & 10.73 & 1.476 & 1.760 & 0.161 \\
Sh & Unigram & 8K & 1.586 & 3.514 & 0.801 & 10.28 & 1.510 & 1.729 & 0.127 \\
Sh & WordPiece & 8K & 1.572 & 3.535 & 0.725 & 10.73 & 1.472 & 1.760 & 0.164 \\
Sh & BBPE & 8K & 1.716 & 3.268 & 0.785 & 10.17 & 1.543 & 2.038 & 0.243 \\
Co & BPE & 8K & 1.625 & 3.412 & 0.716 & 10.76 & 1.677 & 1.527 & 0.089 \\
Co & Unigram & 8K & 1.631 & 3.426 & 0.784 & 10.26 & 1.683 & 1.533 & 0.089 \\
Co & WordPiece & 8K & 1.631 & 3.398 & 0.694 & 10.75 & 1.680 & 1.538 & 0.084 \\
Co & BBPE & 8K & 1.770 & 3.130 & 0.765 & 10.23 & 1.733 & 1.838 & 0.057 \\
\midrule
\multicolumn{9}{@{}l}{\textit{16K vocabulary}} \\
\midrule
Sh & BPE & 16K & 1.406 & 3.938 & 0.771 & 11.22 & 1.332 & 1.545 & 0.138 \\
Sh & Unigram & 16K & 1.439 & 3.856 & 0.820 & 10.66 & 1.384 & 1.543 & 0.103 \\
Sh & WordPiece & 16K & 1.402 & 3.949 & 0.757 & 11.23 & 1.328 & 1.540 & 0.138 \\
Sh & BBPE & 16K & 1.563 & 3.580 & 0.815 & 10.49 & 1.408 & 1.854 & 0.241 \\
Co & BPE & 16K & 1.443 & 3.843 & 0.750 & 11.29 & 1.486 & 1.361 & 0.084 \\
Co & Unigram & 16K & 1.469 & 3.799 & 0.813 & 10.68 & 1.513 & 1.386 & 0.084 \\
Co & WordPiece & 16K & 1.442 & 3.843 & 0.733 & 11.28 & 1.484 & 1.364 & 0.081 \\
Co & BBPE & 16K & 1.604 & 3.458 & 0.804 & 10.55 & 1.551 & 1.705 & 0.090 \\
\midrule
\multicolumn{9}{@{}l}{\textit{32K vocabulary}} \\
\midrule
Sh & BPE & 32K & 1.280 & 4.304 & 0.784 & 11.59 & 1.233 & 1.368 & 0.099 \\
Sh & Unigram & 32K & 1.334 & 4.137 & 0.825 & 10.97 & 1.303 & 1.393 & 0.065 \\
Sh & WordPiece & 32K & 1.274 & 4.320 & 0.775 & 11.59 & 1.227 & 1.362 & 0.099 \\
Sh & BBPE & 32K & 1.454 & 3.834 & 0.829 & 10.69 & 1.318 & 1.709 & 0.229 \\
Co & BPE & 32K & 1.307 & 4.232 & 0.782 & 11.67 & 1.346 & 1.233 & 0.084 \\
Co & Unigram & 32K & 1.350 & 4.118 & 0.828 & 11.00 & 1.392 & 1.271 & 0.087 \\
Co & WordPiece & 32K & 1.303 & 4.240 & 0.769 & 11.65 & 1.342 & 1.231 & 0.083 \\
Co & BBPE & 32K & 1.307 & 4.232 & 0.782 & 11.67 & 1.346 & 1.233 & 0.084 \\
\midrule
\multicolumn{9}{@{}l}{\textit{80K vocabulary}} \\
\midrule
Sh & BPE & 80K & 1.177 & 4.656 & 0.789 & 11.84 & 1.155 & 1.219 & 0.053 \\
Sh & Unigram & 80K & 1.253 & 4.371 & 0.817 & 11.27 & 1.247 & 1.266 & \textbf{0.015} \\
Sh & WordPiece & 80K & 1.171 & 4.676 & 0.784 & 11.82 & 1.150 & 1.210 & 0.049 \\
Sh & BBPE & 80K & 1.392 & 3.994 & 0.836 & 10.75 & 1.268 & 1.624 & 0.219 \\
Co & BPE & 80K & 1.187 & 4.639 & 0.795 & 11.95 & 1.225 & 1.115 & 0.090 \\
Co & Unigram & 80K & 1.248 & 4.423 & 0.830 & 11.29 & 1.288 & 1.171 & 0.091 \\
Co & WordPiece & 80K & 1.183 & 4.651 & 0.790 & 11.92 & 1.221 & 1.111 & 0.090 \\
Co & BBPE & 80K & 1.388 & 3.992 & 0.844 & 10.84 & 1.311 & 1.532 & 0.144 \\
\midrule
\multicolumn{9}{@{}l}{\textit{110K vocabulary}} \\
\midrule
Sh & BPE & 110K & 1.177 & 4.656 & 0.789 & 11.84 & 1.155 & 1.219 & 0.053 \\
Sh & Unigram & 110K & 1.253 & 4.371 & 0.817 & 11.27 & 1.247 & 1.266 & \textbf{0.015} \\
Sh & WordPiece & 110K & 1.171 & 4.676 & 0.784 & 11.82 & 1.150 & 1.210 & 0.049 \\
Sh & BBPE & 110K & 1.392 & 3.994 & 0.836 & 10.75 & 1.268 & 1.624 & 0.219 \\
Co & BPE & 110K & 1.159 & 4.745 & 0.795 & 11.99 & 1.198 & 1.086 & 0.094 \\
Co & Unigram & 110K & 1.224 & 4.494 & 0.827 & 11.37 & 1.264 & 1.150 & 0.091 \\
Co & WordPiece & 110K & 1.155 & 4.758 & 0.792 & 11.96 & 1.193 & 1.082 & 0.093 \\
Co & BBPE & 110K & 1.370 & 4.044 & 0.845 & 10.83 & 1.292 & 1.516 & 0.148 \\
\bottomrule
\end{tabular}
\caption{Full benchmark results for all 40 configurations. Sh = Shared, Co = Concatenated. Fert = fertility (lower is better). CPT = grapheme clusters per token (higher is better). Gini = vocabulary usage inequality. Ent = Shannon entropy (bits). Bold = best fairness.}
\label{tab:full_results}
\end{table*}


\section{Morphological Fidelity}
\label{sec:appendix_morph}

Table~\ref{tab:morph_results} reports morphological edit distance ($\mu_e$, lower is better) and morphological consistency ($\mu_c$, higher is better) for Arabic-script text. Metrics are computed against Farasa \citep{farasa2015} gold-standard segmentations.

At 8K--32K, edit distances cluster tightly within 4.07--4.23 across all algorithms. At 80K--110K, distances rise to 5.54--5.59. This increase reflects fewer subtokens per word at larger vocabulary sizes: with fewer, longer tokens, the boundary alignment to fine-grained morphemes becomes coarser. The relative ordering of algorithms remains stable. BBPE consistently achieves the highest $\mu_c$ F1 at every vocabulary size, because its byte-level granularity produces shorter subtokens that better match morpheme boundaries.

\begin{table*}[t]
\centering
\scriptsize
\setlength{\tabcolsep}{4pt}
\begin{tabular}{@{}llcccrrrr@{}}
\toprule
\textbf{Type} & \textbf{Algo} & \textbf{V} & \textbf{$\mu_e$$\downarrow$} & \textbf{P} & \textbf{R} & \textbf{$\mu_c$-F1$\uparrow$} \\
\midrule
\multicolumn{6}{@{}l}{\textit{8K vocabulary}} \\
\midrule
Sh & BPE & 8K & 4.176 & 0.171 & 0.156 & 0.162 \\
Sh & Unigram & 8K & 4.182 & 0.150 & 0.220 & 0.177 \\
Sh & WordPiece & 8K & 4.177 & 0.164 & 0.149 & 0.155 \\
Sh & BBPE & 8K & 4.192 & 0.155 & 0.283 & 0.200 \\
Co & BPE & 8K & 4.216 & 0.157 & 0.186 & 0.168 \\
Co & Unigram & 8K & 4.211 & 0.145 & 0.240 & 0.180 \\
Co & WordPiece & 8K & 4.219 & 0.152 & 0.176 & 0.162 \\
Co & BBPE & 8K & 4.232 & 0.159 & 0.328 & \textbf{0.214} \\
\midrule
\multicolumn{6}{@{}l}{\textit{16K vocabulary}} \\
\midrule
Sh & BPE & 16K & 4.116 & 0.165 & 0.111 & 0.132 \\
Sh & Unigram & 16K & 4.132 & 0.156 & 0.171 & 0.162 \\
Sh & WordPiece & 16K & 4.116 & 0.165 & 0.121 & 0.139 \\
Sh & BBPE & 16K & 4.137 & 0.155 & 0.226 & 0.182 \\
Co & BPE & 16K & 4.153 & 0.154 & 0.125 & 0.138 \\
Co & Unigram & 16K & 4.158 & 0.146 & 0.174 & 0.157 \\
Co & WordPiece & 16K & 4.151 & 0.174 & 0.138 & 0.153 \\
Co & BBPE & 16K & 4.172 & 0.162 & 0.260 & 0.199 \\
\midrule
\multicolumn{6}{@{}l}{\textit{32K vocabulary}} \\
\midrule
Sh & BPE & 32K & 4.071 & 0.161 & 0.090 & 0.115 \\
Sh & Unigram & 32K & 4.098 & 0.163 & 0.138 & 0.148 \\
Sh & WordPiece & 32K & \textbf{4.070} & 0.181 & 0.100 & 0.127 \\
Sh & BBPE & 32K & 4.097 & 0.157 & 0.198 & 0.174 \\
Co & BPE & 32K & 4.097 & 0.149 & 0.094 & 0.115 \\
Co & Unigram & 32K & 4.111 & 0.149 & 0.139 & 0.142 \\
Co & WordPiece & 32K & 4.098 & 0.164 & 0.113 & 0.133 \\
Co & BBPE & 32K & 4.122 & 0.157 & 0.214 & 0.180 \\
\midrule
\multicolumn{6}{@{}l}{\textit{80K vocabulary}} \\
\midrule
Sh & BPE & 80K & 5.545 & 0.066 & 0.049 & 0.051 \\
Sh & Unigram & 80K & 5.590 & 0.095 & 0.133 & 0.106 \\
Sh & WordPiece & 80K & 5.542 & 0.071 & 0.070 & 0.064 \\
Sh & BBPE & 80K & 5.585 & 0.081 & 0.282 & 0.124 \\
Co & BPE & 80K & 5.553 & 0.073 & 0.045 & 0.052 \\
Co & Unigram & 80K & 5.584 & 0.094 & 0.145 & 0.109 \\
Co & WordPiece & 80K & 5.552 & 0.078 & 0.067 & 0.069 \\
Co & BBPE & 80K & 5.588 & 0.086 & 0.252 & 0.127 \\
\midrule
\multicolumn{6}{@{}l}{\textit{110K vocabulary}} \\
\midrule
Sh & BPE & 110K & 5.545 & 0.066 & 0.049 & 0.051 \\
Sh & Unigram & 110K & 5.590 & 0.095 & 0.133 & 0.106 \\
Sh & WordPiece & 110K & 5.543 & 0.071 & 0.070 & 0.064 \\
Sh & BBPE & 110K & 5.585 & 0.081 & 0.282 & 0.124 \\
Co & BPE & 110K & 5.541 & 0.076 & 0.044 & 0.052 \\
Co & Unigram & 110K & 5.576 & 0.089 & 0.119 & 0.096 \\
Co & WordPiece & 110K & 5.538 & 0.070 & 0.057 & 0.058 \\
Co & BBPE & 110K & 5.578 & 0.084 & 0.239 & 0.123 \\
\bottomrule
\end{tabular}
\caption{Morphological fidelity for all 40 configurations (Arabic-script only). $\mu_e$ = morphological edit distance (lower is better). P = precision, R = recall, $\mu_c$-F1 = morphological consistency F1 (higher is better). Bold marks best values per column.}
\label{tab:morph_results}
\end{table*}


\section{Bootstrap Confidence Intervals}
\label{sec:appendix_bootstrap}

Table~\ref{tab:bootstrap} reports bootstrap 95\% confidence intervals (500 resamples) for fertility and CPT on the test set. All intervals are narrow ($\pm$0.002--0.005 for fertility), confirming that reported differences between configurations are statistically meaningful.

\begin{table}[t]
\centering
\footnotesize
\setlength{\tabcolsep}{3pt}
\begin{tabular}{@{}lrrrr@{}}
\toprule
\textbf{Config} & \textbf{Fert} & \textbf{95\% CI} & \textbf{CPT} & \textbf{95\% CI} \\
\midrule
\multicolumn{4}{@{}l}{\textit{8K}} \\
sh\_bpe & 1.574 & [1.571, 1.578] & 3.531 & [3.525, 3.538] \\
co\_bpe & 1.625 & [1.622, 1.628] & 3.412 & [3.406, 3.418] \\
sh\_uni & 1.586 & [1.583, 1.589] & 3.514 & [3.507, 3.521] \\
co\_uni & 1.631 & [1.627, 1.634] & 3.426 & [3.419, 3.432] \\
sh\_wp & 1.572 & [1.569, 1.575] & 3.535 & [3.529, 3.542] \\
co\_wp & 1.631 & [1.627, 1.634] & 3.398 & [3.392, 3.403] \\
sh\_bbpe & 1.716 & [1.712, 1.719] & 3.268 & [3.260, 3.275] \\
co\_bbpe & 1.770 & [1.767, 1.773] & 3.130 & [3.125, 3.137] \\
\midrule
\multicolumn{4}{@{}l}{\textit{16K}} \\
sh\_bpe & 1.406 & [1.404, 1.409] & 3.938 & [3.932, 3.946] \\
co\_bpe & 1.443 & [1.440, 1.445] & 3.843 & [3.836, 3.850] \\
sh\_uni & 1.439 & [1.436, 1.442] & 3.856 & [3.848, 3.863] \\
co\_uni & 1.469 & [1.466, 1.472] & 3.799 & [3.792, 3.806] \\
sh\_wp & 1.402 & [1.399, 1.404] & 3.949 & [3.943, 3.957] \\
co\_wp & 1.442 & [1.439, 1.445] & 3.843 & [3.836, 3.850] \\
sh\_bbpe & 1.563 & [1.560, 1.566] & 3.580 & [3.572, 3.588] \\
co\_bbpe & 1.604 & [1.602, 1.608] & 3.458 & [3.451, 3.465] \\
\midrule
\multicolumn{4}{@{}l}{\textit{32K}} \\
sh\_bpe & 1.280 & [1.277, 1.282] & 4.304 & [4.297, 4.311] \\
co\_bpe & 1.307 & [1.304, 1.309] & 4.232 & [4.222, 4.241] \\
sh\_uni & 1.334 & [1.332, 1.336] & 4.137 & [4.129, 4.143] \\
co\_uni & 1.350 & [1.347, 1.353] & 4.118 & [4.110, 4.125] \\
sh\_wp & 1.274 & [1.272, 1.276] & 4.320 & [4.313, 4.327] \\
co\_wp & 1.303 & [1.301, 1.306] & 4.240 & [4.233, 4.247] \\
sh\_bbpe & 1.454 & [1.451, 1.456] & 3.834 & [3.826, 3.843] \\
co\_bbpe & 1.307 & [1.304, 1.309] & 4.232 & [4.225, 4.238] \\
\bottomrule
\end{tabular}
\caption{Bootstrap 95\% confidence intervals (500 resamples) for fertility and CPT on the test set. CI width is $\leq$0.006 for all configurations.}
\label{tab:bootstrap}
\end{table}


\section{Morphological Metric Definitions}
\label{sec:appendix_metrics}

\subsection{Morphological Edit Distance ($\mu_e$)}

Given a word $w$ tokenized into tokens $\mathbf{t} = [t_1, \ldots, t_m]$ and segmented by Farasa into morphemes $\mathbf{m} = [m_1, \ldots, m_k]$, we compute the ordered Levenshtein alignment cost:
\begin{equation}
\mu_e = \frac{1}{N} \sum_{i=1}^{N} \text{Lev}(\mathbf{t}_i, \mathbf{m}_i)
\end{equation}
where $\text{Lev}(\cdot, \cdot)$ is the standard edit distance (insertions, deletions, substitutions each cost 1), and $N$ is the number of Arabic-script words in the test set with available Farasa segmentations. Lower $\mu_e$ indicates better alignment between token boundaries and morpheme boundaries.

\subsection{Morphological Consistency F1 ($\mu_c$)}

$\mu_c$ measures whether words that share morphemes also share tokens. We compute it in three steps:

\textbf{Step 1: Morphological clustering.} For each unique word $w_i$ in the test set, we construct a TF-IDF vector from its Farasa morpheme set. We cluster all words into $K = 30$ groups via $K$-Means on these vectors. Each cluster contains words with overlapping morphology.

\textbf{Step 2: Pairwise evaluation.} Within each cluster, we sample up to $C = 20$ word pairs. For each pair $(w_i, w_j)$:
\begin{itemize}[noitemsep,topsep=2pt]
\item \textbf{Shared morpheme}: $|M(w_i) \cap M(w_j)| > 0$ where $M(w)$ is the morpheme set.
\item \textbf{Shared token}: $|T(w_i) \cap T(w_j)| > 0$ where $T(w)$ is the token set.
\end{itemize}

\textbf{Step 3: Precision, recall, F1.} We define:
\begin{align}
P &= \frac{|\text{pairs sharing both morpheme \& token}|}{|\text{pairs sharing token}|} \\
R &= \frac{|\text{pairs sharing both morpheme \& token}|}{|\text{pairs sharing morpheme}|} \\
\mu_c\text{-F1} &= \frac{2 \cdot P \cdot R}{P + R}
\end{align}

We bootstrap this process $B = 5$ times (different random pair samples per iteration) and report the mean F1. Higher $\mu_c$ indicates that the tokenizer groups morphologically related words into consistent token clusters.

\end{document}