Benchmark reproduction

The benchmark compares SamAI-Tokenizer with Arabic-focused and recent multilingual tokenizers on four datasets. It downloads tokenizer assets only, not model weights.

Run

From the repository root:

python -m venv .venv
source .venv/bin/activate
pip install -r benchmark/requirements.txt
python benchmark/benchmark.py

Generated files are written to results/generated/. Override paths when needed:

SAMAI_TOKENIZER_PATH=/path/to/tokenizer.json \
SAMAI_BENCHMARK_OUTPUT=/path/to/results \
python benchmark/benchmark.py

Third-party remote code is disabled by default. Some custom tokenizers will be skipped unless you review their repositories and explicitly opt in:

SAMAI_TRUST_REMOTE_CODE=1 python benchmark/benchmark.py

Gated repositories also require accepting their terms and authenticating with Hugging Face before running the script.

Published files

arabic_4bench_macro_average.csv: equal 25% weight per benchmark; primary ranking.
arabic_4bench_pooled_overall.csv: all tokens and words pooled together.
arabic_4bench_by_benchmark.csv: result for every tokenizer and benchmark.
arabic_4bench_by_category.csv: detailed subject, dialect, and intent breakdowns.
arabicmmlu_summary.csv and arabicmmlu_by_subject.csv: the original ArabicMMLU-only comparison.

The script also produces document-level results locally. Those results are not published because they contain copies of benchmark text.