| # Benchmark reproduction |
|
|
| The benchmark compares SamAI-Tokenizer with Arabic-focused and recent multilingual |
| tokenizers on four datasets. It downloads tokenizer assets only, not model |
| weights. |
|
|
| ## Run |
|
|
| From the repository root: |
|
|
| ```bash |
| python -m venv .venv |
| source .venv/bin/activate |
| pip install -r benchmark/requirements.txt |
| python benchmark/benchmark.py |
| ``` |
|
|
| Generated files are written to `results/generated/`. Override paths when |
| needed: |
|
|
| ```bash |
| SAMAI_TOKENIZER_PATH=/path/to/tokenizer.json \ |
| SAMAI_BENCHMARK_OUTPUT=/path/to/results \ |
| python benchmark/benchmark.py |
| ``` |
|
|
| Third-party remote code is disabled by default. Some custom tokenizers will be |
| skipped unless you review their repositories and explicitly opt in: |
|
|
| ```bash |
| SAMAI_TRUST_REMOTE_CODE=1 python benchmark/benchmark.py |
| ``` |
|
|
| Gated repositories also require accepting their terms and authenticating with |
| Hugging Face before running the script. |
|
|
| ## Published files |
|
|
| - `arabic_4bench_macro_average.csv`: equal 25% weight per benchmark; primary |
| ranking. |
| - `arabic_4bench_pooled_overall.csv`: all tokens and words pooled together. |
| - `arabic_4bench_by_benchmark.csv`: result for every tokenizer and benchmark. |
| - `arabic_4bench_by_category.csv`: detailed subject, dialect, and intent |
| breakdowns. |
| - `arabicmmlu_summary.csv` and `arabicmmlu_by_subject.csv`: the original |
| ArabicMMLU-only comparison. |
|
|
| The script also produces document-level results locally. Those results are not |
| published because they contain copies of benchmark text. |
|
|
