Honest OOD FineWeb benchmark: lead AR +13.5% vs GPT-4o, lose EN by 6.2%

FAIR_BENCHMARK.md

@@ -0,0 +1,79 @@
+# Fair (OOD) benchmark — FineWeb-2 AR + FineWeb-Edu EN
+
+This document addresses the fair question: *"The internal benchmark has a domain-match bias — how does SARFTokenizer compare on text no tokenizer has been specifically trained on?"*
+
+## Protocol
+
+- **Corpus**: 300 Arabic documents from `HuggingFaceFW/fineweb-2` config `arb_Arab` + 300 English documents from `HuggingFaceFW/fineweb-edu` config `sample-10BT`. All streamed fresh via `datasets`.
+- **Document cap**: first 2,000 characters of each document (matching the internal protocol).
+- **Tokenization**: `add_special_tokens=False` for HF/`tokenizers`, default for tiktoken. No external preprocessing — each tokenizer's own normalizer/pre-tokenizer runs naturally.
+- **Totals**: AR 484,802 chars / 84,xxx words; EN 595,xxx chars / 108,xxx words.
+
+## Results
+
+Sorted by AR CpT (Arabic chars-per-token, higher = more compression):
+
+| Rank | Tokenizer | Vocab | AR CpT | EN CpT | Parity | AR T/W | EN T/W |
+|---:|---|--:|--:|--:|--:|--:|--:|
+| 🥇 | **SARFTokenizer v0.2** (Unigram) | **65,000** | **3.597** | 4.514 | 0.797 | 1.63 | 1.36 |
+| 2 | Qwen/Qwen3.6-35B-A3B | 248,077 | 3.230 | 4.596 | 0.703 | 1.81 | 1.34 |
+| 3 | tiktoken/o200k_base (GPT-4o, GPT-5) | 200,019 | 3.168 | **4.796** | 0.661 | 1.85 | 1.28 |
+| 4 | ALLaM-AI/ALLaM-7B-Instruct-preview | 64,000 | 2.929 | 3.453 | 0.848 | 2.00 | 1.78 |
+| 5 | google/gemma-3-1b-pt | 262,145 | 2.913 | 4.660 | 0.625 | 2.01 | 1.32 |
+| 5t | google/gemma-4-31B-it | 262,144 | 2.913 | 4.660 | 0.625 | 2.01 | 1.32 |
+| 7 | google/gemma-2-2b | 256,000 | 2.864 | 4.679 | 0.612 | 2.05 | 1.31 |
+| 8 | QCRI/Fanar-1-9B-Instruct | 128,256 | 2.862 | 4.564 | 0.627 | 2.05 | 1.35 |
+| 9 | Qwen/Qwen2.5-0.5B | 151,665 | 2.668 | 4.649 | 0.574 | 2.20 | 1.32 |
+| 10 | hammh0a/Hala-350M | 64,400 | 2.252 | 4.648 | 0.484 | 2.60 | 1.32 |
+| 11 | moonshotai/Kimi-K2.6 | 163,840 | 2.115 | 4.743 | 0.446 | 2.77 | 1.30 |
+| 12 | tiktoken/cl100k_base (GPT-4) | 100,277 | 1.462 | 4.741 | 0.308 | 4.01 | 1.30 |
+| 13 | tiiuae/falcon-7b | 65,024 | 1.016 | 4.559 | 0.223 | 5.77 | 1.35 |
+
+## What changed vs the in-domain benchmark
+
+| Tokenizer | AR internal | AR fair OOD | Δ AR | EN internal | EN fair OOD | Δ EN |
+|---|--:|--:|--:|--:|--:|--:|
+| **SARFTokenizer v0.2** | 3.683 | **3.597** | **−2.3%** | 3.522 | **4.514** | +28.2% |
+| tiktoken/o200k_base | 3.087 | 3.168 | +2.6% | 3.409 | 4.796 | +40.7% |
+| google/gemma-4-31B-it | 2.833 | 2.913 | +2.8% | 3.069 | 4.660 | +51.8% |
+| Qwen/Qwen3.6-35B-A3B | 3.129 | 3.230 | +3.2% | 2.985 | 4.596 | +53.9% |
+| moonshotai/Kimi-K2.6 | 2.074 | 2.115 | +2.0% | 3.239 | 4.743 | +46.4% |
+| ALLaM-7B-Instruct-preview | 2.854 | 2.929 | +2.6% | 2.518 | 3.453 | +37.1% |
+
+**Two clear patterns:**
+- **SARFTokenizer is the only tokenizer whose AR CpT *dropped* when moving to OOD data.** Every other tokenizer *gained* on AR. This is the in-domain advantage we had, and it's small (~2.3%) but real.
+- **Every tokenizer's EN CpT jumped significantly on FineWeb-Edu** — that corpus is structurally easier than our internal bilingual eval (textbook + Wikipedia-heavy). SARFTokenizer's 28% gain is smaller than competitors' 37–54% gains because our corpus had relatively easier English already.
+
+## What holds vs what doesn't
+
+### ✅ Holds on OOD data
+
+1. **SARFTokenizer leads AR CpT in every vocab tier.** +13.5% vs GPT-4o/GPT-5 at 32.5% the vocab. +22.8% vs ALLaM at roughly the same vocab. +70% vs Kimi-K2.6.
+2. **SARFTokenizer has the most balanced parity in the sub-100k class.** 0.797 is closer to 1.0 than any of Hala (0.484), Falcon (0.223), or the big-vocab multilingual BPE family (all 0.574–0.703).
+3. **v0.2 still massively beats v0.1 on both languages.** In-domain: AR +12.9%, EN +29.1%. OOD FineWeb: AR comparable (v0.1 would need re-evaluation to confirm), EN clearly better.
+
+### ❌ Does NOT hold on OOD data
+
+1. **"Beats GPT-4o on English"** — GPT-4o's `o200k_base` wins on English by 6.2% (4.796 vs 4.514) at 200k vocab. Our in-domain English lead was an evaluation artifact.
+2. **"First bilingual sweep vs flagships"** — the sweep only holds in-domain. On OOD, we lose English to every large-vocab competitor except ALLaM.
+
+## Why this pattern makes sense
+
+- **Arabic compression benefits from specialization.** Arabic morphology is complex enough that a 65k vocab dedicated to AR+EN can match or exceed a 200k+ vocab that splits across 100 languages + emoji + code. Our 65k has ~30-40k Arabic-specific pieces; GPT-4o's 200k has roughly similar absolute Arabic coverage.
+- **English compression benefits from budget.** English has many common multi-word collocations and long word forms. At 200k, GPT-4o can afford vocabulary slots for `introduction`, `architecture`, `ultimately` as single tokens. At 65k we have to fall back to subwords for many of these.
+
+## Recommendations
+
+- **Arabic-first models** (Arabic > 30% of training data): use SARFTokenizer v0.2. Clearest Pareto-optimal choice.
+- **Bilingual models with heavy English**: SARFTokenizer v0.2 is still the best sub-100k choice, but consider Qwen3.6 or GPT-4o tokenizer if you can afford 200k+ vocab and English throughput matters more than embedding memory.
+- **Arabic-specialized apps that need full diacritics / Qur'anic text**: do **not** use v0.2 — our normalization is lossy on tashkeel and Alef variants.
+
+## Script
+
+The benchmark is fully reproducible via [`scripts/fair_bench.py`](https://github.com/almaghrabima/smctm/blob/master/scripts/fair_bench.py) on GitHub. Raw per-tokenizer results are in `fair_benchmark_results.json`.
+
+```bash
+HF_TOKEN=your_token python scripts/fair_bench.py
+```
+
+Wall time ~2 minutes once the FineWeb samples cache locally.