language:
- ar
- en
license: apache-2.0
library_name: tokenizers
tags:
- tokenizer
- arabic
- arabic-nlp
SamAI-Tokenizer
🚀 Try the interactive tokenizer demo
SamAI-Tokenizer is an Arabic-focused byte-level BPE tokenizer for training new Arabic-heavy language models. It has a vocabulary of 65,536 tokens, applies Unicode NFC normalization, supports lossless decoding, and retains English coverage for multilingual training.
Key specifications
| Property | Value |
|---|---|
| Algorithm | Byte-level BPE |
| Vocabulary size | 65,536 |
| Normalization | Unicode NFC |
| Pre-tokenizer | ByteLevel |
| Unknown-token fallback | Byte-level coverage |
| Named special tokens | <pad>, <bos>, <eos>, <unk>, <user>, <assistant>, <system> |
| Reserved token IDs | 7–255 |
| Primary language | Arabic |
| Secondary language | English |
Arabic tokenizer benchmark
SamAI-Tokenizer was evaluated on four complementary Arabic benchmarks. Lower tokens per word indicates more efficient tokenization.
Across the four-benchmark macro average, SamAI-Tokenizer achieved 1.6207 tokens per word, the second-lowest result among the 15 tokenizers successfully evaluated.
| Rank | Tokenizer | Mean tokens/word | Extra tokens versus SamAI |
|---|---|---|---|
| 1 | ALLaM-7B | 1.5021 | -4.86% |
| 2 | SamAI-Tokenizer | 1.6207 | baseline |
| 3 | Falcon-H1-7B | 1.8880 | +19.30% |
| 4 | Mistral-Small-3.1 | 1.8929 | +20.71% |
| 5 | Qwen3.8-Flash-Next | 1.9039 | +21.94% |
| 6 | Qwen3.5-4B | 1.9039 | +21.94% |
| 7 | tiktoken-o200k | 1.9919 | +26.37% |
| 8 | GPT-OSS-20B | 1.9919 | +26.37% |
| 9 | Fanar-1 | 2.1570 | +36.81% |
| 10 | DeepSeek-V3 | 2.2774 | +46.25% |
| 11 | Qwen2.5-7B | 2.3740 | +50.25% |
| 12 | Qwen3-8B | 2.3740 | +50.25% |
| 13 | GLM-5.3-Flash | 2.5225 | +60.21% |
| 14 | Kimi-K2.6 | 2.8693 | +84.13% |
| 15 | tiktoken-cl100k | 4.0800 | +164.63% |
Training data
The tokenizer vocabulary was trained on a diverse mixture of Arabic and English text, including Modern Standard Arabic, dialectal Arabic, educational content, and mixed Arabic-English text.
Installation
pip install -U tokenizers transformers huggingface_hub
Usage
Transformers
from transformers import PreTrainedTokenizerFast
tokenizer = PreTrainedTokenizerFast.from_pretrained(
"Samai-sa/SamAI-Tokenizer"
)
text = "المملكة العربية السعودية"
ids = tokenizer.encode(text, add_special_tokens=False)
print("Token count:", len(ids))
print("Token IDs:", ids)
print("Tokens:", tokenizer.convert_ids_to_tokens(ids))
print("Decoded:", tokenizer.decode(ids))
Tokenizers
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer
path = hf_hub_download(
repo_id="Samai-sa/SamAI-Tokenizer",
filename="tokenizer.json",
)
tokenizer = Tokenizer.from_file(path)
encoding = tokenizer.encode(
"السلام عليكم ورحمة الله وبركاته",
add_special_tokens=False,
)
print("Token count:", len(encoding.ids))
print("Token IDs:", encoding.ids)
print("Tokens:", encoding.tokens)
print("Decoded:", tokenizer.decode(encoding.ids))
Interactive demo
The SamAI Tokenizer Playground lets users enter Arabic or English text and inspect:
- token count and tokens per word;
- token IDs and raw token pieces;
- original text spans and offsets;
- decoded text for round-trip verification.
Validation
The released validation run reported zero round-trip failures and zero unknown tokens across samples from schoolbooks, AraMix Arabic, FineWeb2 Arabic, and FineWeb English. Additional stress tests cover Modern Standard Arabic, dialects, Arabizi, English, code, mathematics, and Unicode edge cases.
See reports/validation.json and
reports/stress.json.
Benchmark methodology
- All available examples were used, followed by exact-text deduplication within each benchmark.
- Token counts exclude chat templates and added special tokens.
- Tokens per word is total tokens divided by whitespace-separated words.
- The primary macro ranking gives each benchmark 25% weight.
- The pooled report sums all tokens and words before calculating the ratio.
- Tokenizers producing identical results are tokenizer-family duplicates, not independent confirmations.
To reproduce the benchmark:
pip install -r benchmark/requirements.txt
python benchmark/benchmark.py
Limitations
- Tokenization efficiency does not measure language-model quality, reasoning, factuality, safety, or downstream accuracy.
- Fully vocalized Arabic is the clearest weakness. One heavily diacritized ArabicMMLU category accounts for most of the gap.
- Whitespace-separated words are a transparent denominator, not a linguistic segmentation of Arabic morphology.
- Replacing the tokenizer of an already pretrained model requires resizing and retraining its embedding and output layers. This is not a drop-in vocabulary replacement.
License and data terms
The tokenizer files and repository code are released under Apache-2.0. Source and benchmark datasets remain subject to their own licenses and terms. No source-corpus or benchmark documents are redistributed here.