# Benchmark reproduction

The benchmark compares SamAI-Tokenizer with Arabic-focused and recent multilingual
tokenizers on four datasets. It downloads tokenizer assets only, not model
weights.

## Run

From the repository root:

```bash
python -m venv .venv
source .venv/bin/activate
pip install -r benchmark/requirements.txt
python benchmark/benchmark.py
```

Generated files are written to `results/generated/`. Override paths when
needed:

```bash
SAMAI_TOKENIZER_PATH=/path/to/tokenizer.json \
SAMAI_BENCHMARK_OUTPUT=/path/to/results \
python benchmark/benchmark.py
```

Third-party remote code is disabled by default. Some custom tokenizers will be
skipped unless you review their repositories and explicitly opt in:

```bash
SAMAI_TRUST_REMOTE_CODE=1 python benchmark/benchmark.py
```

Gated repositories also require accepting their terms and authenticating with
Hugging Face before running the script.

## Published files

- `arabic_4bench_macro_average.csv`: equal 25% weight per benchmark; primary
  ranking.
- `arabic_4bench_pooled_overall.csv`: all tokens and words pooled together.
- `arabic_4bench_by_benchmark.csv`: result for every tokenizer and benchmark.
- `arabic_4bench_by_category.csv`: detailed subject, dialect, and intent
  breakdowns.
- `arabicmmlu_summary.csv` and `arabicmmlu_by_subject.csv`: the original
  ArabicMMLU-only comparison.

The script also produces document-level results locally. Those results are not
published because they contain copies of benchmark text.