---
license: cc-by-nc-4.0
language:
  - ar
  - en
tags:
  - tokenizer
  - arabic
  - morphology
  - benchmark
---

# SARF Tokenizer

**SARF** (Segmentation-Aware Rewriting Framework) is a morphologically-aware tokenizer for Arabic that combines unsupervised morphological segmentation (Morfessor) with Byte-Pair Encoding. It uses Unicode Private Use Area (PUA) characters to map Arabic morphemes to single tokens before BPE training, achieving strong Arabic tokenization with a compact 72K vocabulary.

## Benchmark Results

Evaluation on ~5,000 Arabic + 5,000 English samples from the eval-test-data dataset. Ranked by parity (closest to 1.0), then average chars/token.

| Rank | Tokenizer | Vocab | AR Fertility | AR Chars/Tok | EN Fertility | EN Chars/Tok | Parity |
|------|-----------|------:|-------------:|-------------:|-------------:|-------------:|-------:|
| 1 | Gemma-3-4B | 262,145 | 2.311 | 2.864 | 1.137 | 2.911 | 0.9840 |
| 2 | Fanar-1-9B | 128,256 | 2.264 | 2.812 | 1.141 | 2.880 | 0.9764 |
| 3 | Hala-9B | 128,256 | 2.264 | 2.812 | 1.141 | 2.880 | 0.9764 |
| 4 | Command-R-Arabic | 255,033 | 2.320 | 2.799 | 1.142 | 2.906 | 0.9631 |
| 5 | **SARF (Ours)** | **72,195** | **1.978** | **2.832** | 1.561 | 3.163 | **0.8952** |
| 6 | GPT-4o | 200,019 | 2.249 | 3.111 | 1.213 | 3.492 | 0.8909 |
| 7 | Qwen3-4B | 151,669 | 2.314 | 2.599 | 1.225 | 2.964 | 0.8767 |
| 8 | Qwen3-VL-4B | 151,669 | 2.314 | 2.599 | 1.225 | 2.964 | 0.8767 |
| 9 | Falcon-H1-7B | 130,049 | 2.083 | 3.272 | 1.266 | 2.835 | 1.1543 |
| 10 | ALLaM-7B | 64,000 | 1.286 | 3.898 | 1.197 | 2.699 | 1.4442 |
| 11 | Mistral-7B-v0.3 | 32,768 | 5.133 | 1.131 | 1.218 | 2.702 | 0.4185 |
| 12 | GPT-4 | 100,277 | 4.111 | 1.430 | 1.225 | 3.452 | 0.4144 |
| 13 | AceGPT-13B | 32,000 | 5.236 | 1.110 | 1.237 | 2.691 | 0.4124 |

### Metric Definitions

- **AR Fertility**: Arabic tokens per word (lower = better)
- **AR Chars/Tok**: Arabic characters per token (higher = better compression)
- **EN Fertility**: English tokens per word (lower = better)
- **EN Chars/Tok**: English characters per token (higher = better compression)
- **Parity**: AR Chars/Tok / EN Chars/Tok (closer to 1.0 = more balanced)

### Key Findings

- SARF achieves the **lowest Arabic fertility** (1.978 tokens/word) among all tokenizers with vocabulary under 130K, demonstrating that morphological preprocessing enables efficient Arabic tokenization without massive vocabularies.
- With only **72K vocabulary**, SARF achieves Arabic compression (2.832 chars/token) competitive with tokenizers 2-3x its size.
- SARF has **near-perfect parity** (0.895), meaning Arabic and English text are tokenized with similar efficiency — unlike GPT-4 (0.414) or ALLaM (1.444) which show strong language bias.
- SARF ranks **5th in parity** out of 13 tokenizers despite having the **smallest vocabulary** among the top 9.

## Tokenizers Compared

| Tokenizer | Model | Source |
|-----------|-------|--------|
| SARF | DeepLatent | [almaghrabima/deeplatent-tokenizer-parity](https://huggingface.co/almaghrabima/deeplatent-tokenizer-parity) |
| GPT-4o | o200k_base | tiktoken |
| GPT-4 | cl100k_base | tiktoken |
| ALLaM-7B | humain-ai/ALLaM-7B-Instruct-preview | HuggingFace |
| AceGPT-13B | FreedomIntelligence/AceGPT-13B-chat | HuggingFace |
| Gemma-3-4B | google/gemma-3-4b-it | HuggingFace |
| Command-R Arabic | CohereLabs/c4ai-command-r7b-arabic-02-2025 | HuggingFace |
| Fanar-1-9B | QCRI/Fanar-1-9B-Instruct | HuggingFace |
| Hala-9B | hammh0a/Hala-9B | HuggingFace |
| Qwen3-4B | Qwen/Qwen3-4B-Instruct-2507 | HuggingFace |
| Qwen3-VL-4B | Qwen/Qwen3-VL-4B-Instruct | HuggingFace |
| Mistral-7B-v0.3 | mistralai/Mistral-7B-Instruct-v0.3 | HuggingFace |
| Falcon-H1-7B | tiiuae/Falcon-H1-7B-Instruct | HuggingFace |

## How SARF Works

SARF uses a morphologically-aware preprocessing pipeline before BPE:

1. **Morfessor** segments Arabic words into morphemes unsupervised
2. **Morpheme-to-PUA mapping** assigns each morpheme a Unicode Private Use Area character
3. **ByteRewriter** rewrites Arabic text so morphemes become single characters
4. **BPE** trains on the rewritten text, naturally learning morpheme-level tokens

This approach achieves strong Arabic compression without inflating the vocabulary for English or other languages.

## Files

- `results.json` — Raw benchmark data
- `tokenizer_benchmark.py` — Benchmark script (reproduces results)

## Citation

```bibtex
@misc{sarf2025,
  title={SARF: Segmentation-Aware Rewriting Framework for Arabic Tokenization},
  author={Al-Maghrabima},
  year={2025},
  url={https://huggingface.co/almaghrabima/SARF-Tokenizer}
}
```

## License

CC-BY-NC-4.0