Ouaill commited on
Commit
0b95151
·
verified ·
1 Parent(s): 0687ce2

Upload latest_main.tex with huggingface_hub

Browse files
Files changed (1) hide show
  1. latest_main.tex +1 -1
latest_main.tex CHANGED
@@ -60,7 +60,7 @@ We fill this gap with the first comprehensive tokenizer benchmark for Moroccan D
60
  \item \textbf{Proposed improvement}: A concatenated architecture that trains separate per-script sub-tokenizers, reducing cross-script disparity by up to $4\times$
61
  \end{enumerate}
62
 
63
- Our key findings are threefold. First, Darija-specific tokenization dramatically improves compression: our 80K tokenizer achieves 34\% lower fertility than DarijaBERT at the same vocabulary size. Second, cross-script fairness is achievable for Darija: shared Unigram at 80K reaches $\Delta F$ = 0.015 (near-zero disparity), while concatenated architectures provide consistent fairness across all algorithms. Third, vocabulary size saturates early: moving from 32K to 110K yields only marginal gains, suggesting that 32K is sufficient for most Darija applications. All 40 configurations achieve $\geq$99\% exact reconstruction.
64
 
65
 
66
  \section{Methodology}
 
60
  \item \textbf{Proposed improvement}: A concatenated architecture that trains separate per-script sub-tokenizers, reducing cross-script disparity by up to $4\times$
61
  \end{enumerate}
62
 
63
+ Our key findings are threefold. First, Darija-specific tokenization dramatically improves compression: our 80K tokenizer achieves 33\% lower fertility than DarijaBERT at the same vocabulary size. Second, cross-script fairness is achievable for Darija: shared Unigram at 80K reaches $\Delta F$ = 0.015 (near-zero disparity), while concatenated architectures provide consistent fairness across all algorithms. Third, vocabulary size saturates early: moving from 32K to 110K yields only marginal gains, suggesting that 32K is sufficient for most Darija applications. All 40 configurations achieve $\geq$99\% exact reconstruction.
64
 
65
 
66
  \section{Methodology}