README.md

---
title: Arabic Tokenizer Comparison
emoji: 🔤
colorFrom: green
colorTo: indigo
sdk: gradio
sdk_version: 6.22.0
app_file: app.py
pinned: false
license: apache-2.0
short_description: "How many tokens does your Arabic actually cost?"
---

# Arabic Tokenizer Comparison — كم رمزاً يكلّفك النص العربي؟

Every model card quotes parameter count and context length. None of them tell
you that the same Arabic paragraph costs **50% more context** on one tokenizer
than another — a tax paid on every prompt, every document and every embedding
you ever run.

Paste Arabic — MSA, dialect, or code-switched — and see what nine tokenizers
actually spend on it.

## The point isn't vocabulary size

**Mistral-7B and Emhotob both have ~32K vocabularies.** On Modern Standard
Arabic, Mistral needs **4.5× more tokens** for identical text. The difference
isn't how many entries a vocabulary has, it's what those entries are spent on.

Measured on a 1,024-word MSA paragraph:

| tokenizer | vocab | tokens/word | vs Emhotob |
|---|---|---|---|
| **Emhotob 32K** (Arabic-only) | 32,000 | **1.391** | 1.00× |
| Qwen3.6 | 248,044 | 1.885 | 1.36× |
| Gemma-4 | 262,144 | 2.109 | 1.52× |

Emhotob beats vocabularies **8× its size** on Arabic — and loses badly on
English, which it was never built for. That trade is the whole design.

The token-split view shows why. On the same sentence, Mistral emits **one token
per Arabic letter**, Gemma-4 breaks words into fragments (`أ` `عل` `نت`), and
Emhotob keeps them whole.

## What to try

- **Dialect**: Emhotob's lead narrows from 1.52× to ~1.19×. Dialectal
  orthography is where an MSA-trained vocabulary is weakest.
- **Code-switched**: Gemma-4 *wins*. Half the text is English.
- **English**: Emhotob costs 2.2× what Gemma-4 does. Right tool, wrong job.

## About Emhotob

The Arabic-only byte-level BPE behind
[`50M-2048-Emhotob`](https://huggingface.co/oddadmix/50M-2048-Emhotob) and the
[Nawah](https://huggingface.co/oddadmix/Nawah-50M-RAG-Support-2K) family. At
hidden size 512 its 32,000 entries cost 16.4M embedding parameters; a 262K
vocabulary would cost 134M — more than twice the entire model.

© KAND CA 2026 — PROJECT NAWAH