Upload latest_main.tex with huggingface_hub
Browse files- latest_main.tex +1 -1
latest_main.tex
CHANGED
|
@@ -60,7 +60,7 @@ We fill this gap with the first comprehensive tokenizer benchmark for Moroccan D
|
|
| 60 |
\item \textbf{Proposed improvement}: A concatenated architecture that trains separate per-script sub-tokenizers, reducing cross-script disparity by up to $4\times$
|
| 61 |
\end{enumerate}
|
| 62 |
|
| 63 |
-
Our key findings are threefold. First, Darija-specific tokenization dramatically improves compression: our 80K tokenizer achieves
|
| 64 |
|
| 65 |
|
| 66 |
\section{Methodology}
|
|
|
|
| 60 |
\item \textbf{Proposed improvement}: A concatenated architecture that trains separate per-script sub-tokenizers, reducing cross-script disparity by up to $4\times$
|
| 61 |
\end{enumerate}
|
| 62 |
|
| 63 |
+
Our key findings are threefold. First, Darija-specific tokenization dramatically improves compression: our 80K tokenizer achieves 33\% lower fertility than DarijaBERT at the same vocabulary size. Second, cross-script fairness is achievable for Darija: shared Unigram at 80K reaches $\Delta F$ = 0.015 (near-zero disparity), while concatenated architectures provide consistent fairness across all algorithms. Third, vocabulary size saturates early: moving from 32K to 110K yields only marginal gains, suggesting that 32K is sufficient for most Darija applications. All 40 configurations achieve $\geq$99\% exact reconstruction.
|
| 64 |
|
| 65 |
|
| 66 |
\section{Methodology}
|