| --- |
| language: |
| - ar |
| - en |
| license: apache-2.0 |
| library_name: tokenizers |
| tags: |
| - tokenizer |
| - arabic |
| - arabic-nlp |
| --- |
| |
| # SamAI-Tokenizer |
|
|
| [**🚀 Try the interactive tokenizer demo**](https://huggingface.co/spaces/Samai-sa/SamAI-Tokenizer-Demo) |
|
|
| SamAI-Tokenizer is an Arabic-focused byte-level BPE tokenizer for training new |
| Arabic-heavy language models. It has a vocabulary of 65,536 tokens, applies |
| Unicode NFC normalization, supports lossless decoding, and retains English |
| coverage for multilingual training. |
|
|
| ## Key specifications |
|
|
| | Property | Value | |
| | --- | --- | |
| | Algorithm | Byte-level BPE | |
| | Vocabulary size | 65,536 | |
| | Normalization | Unicode NFC | |
| | Pre-tokenizer | ByteLevel | |
| | Unknown-token fallback | Byte-level coverage | |
| | Named special tokens | `<pad>`, `<bos>`, `<eos>`, `<unk>`, `<user>`, `<assistant>`, `<system>` | |
| | Reserved token IDs | 7–255 | |
| | Primary language | Arabic | |
| | Secondary language | English | |
|
|
| ## Arabic tokenizer benchmark |
|
|
| SamAI-Tokenizer was evaluated on four complementary Arabic benchmarks. Lower |
| tokens per word indicates more efficient tokenization. |
|
|
|  |
|
|
| Across the four-benchmark macro average, SamAI-Tokenizer achieved **1.6207 tokens** per word, the **second-lowest** result among the 15 tokenizers successfully evaluated. |
|
|
| | Rank | Tokenizer | Mean tokens/word | Extra tokens versus SamAI | |
| | ---: | --- | ---: | ---: | |
| | 1 | ALLaM-7B | 1.5021 | -4.86% | |
| | **2** | **SamAI-Tokenizer** | **1.6207** | **baseline** | |
| | 3 | Falcon-H1-7B | 1.8880 | +19.30% | |
| | 4 | Mistral-Small-3.1 | 1.8929 | +20.71% | |
| | 5 | Qwen3.8-Flash-Next | 1.9039 | +21.94% | |
| | 6 | Qwen3.5-4B | 1.9039 | +21.94% | |
| | 7 | tiktoken-o200k | 1.9919 | +26.37% | |
| | 8 | GPT-OSS-20B | 1.9919 | +26.37% | |
| | 9 | Fanar-1 | 2.1570 | +36.81% | |
| | 10 | DeepSeek-V3 | 2.2774 | +46.25% | |
| | 11 | Qwen2.5-7B | 2.3740 | +50.25% | |
| | 12 | Qwen3-8B | 2.3740 | +50.25% | |
| | 13 | GLM-5.3-Flash | 2.5225 | +60.21% | |
| | 14 | Kimi-K2.6 | 2.8693 | +84.13% | |
| | 15 | tiktoken-cl100k | 4.0800 | +164.63% | |
|
|
| ## Training data |
|
|
| The tokenizer vocabulary was trained on a diverse mixture of Arabic and English text, including Modern Standard Arabic, dialectal Arabic, educational content, and mixed Arabic-English text. |
|
|
| ## Installation |
|
|
| ```bash |
| pip install -U tokenizers transformers huggingface_hub |
| ``` |
|
|
| ## Usage |
|
|
| ### Transformers |
|
|
| ```python |
| from transformers import PreTrainedTokenizerFast |
| |
| tokenizer = PreTrainedTokenizerFast.from_pretrained( |
| "Samai-sa/SamAI-Tokenizer" |
| ) |
| |
| text = "المملكة العربية السعودية" |
| ids = tokenizer.encode(text, add_special_tokens=False) |
| |
| print("Token count:", len(ids)) |
| print("Token IDs:", ids) |
| print("Tokens:", tokenizer.convert_ids_to_tokens(ids)) |
| print("Decoded:", tokenizer.decode(ids)) |
| ``` |
|
|
| ### Tokenizers |
|
|
| ```python |
| from huggingface_hub import hf_hub_download |
| from tokenizers import Tokenizer |
| |
| path = hf_hub_download( |
| repo_id="Samai-sa/SamAI-Tokenizer", |
| filename="tokenizer.json", |
| ) |
| tokenizer = Tokenizer.from_file(path) |
| |
| encoding = tokenizer.encode( |
| "السلام عليكم ورحمة الله وبركاته", |
| add_special_tokens=False, |
| ) |
| |
| print("Token count:", len(encoding.ids)) |
| print("Token IDs:", encoding.ids) |
| print("Tokens:", encoding.tokens) |
| print("Decoded:", tokenizer.decode(encoding.ids)) |
| ``` |
|
|
| ## Interactive demo |
|
|
| The [SamAI Tokenizer Playground](https://huggingface.co/spaces/Samai-sa/SamAI-Tokenizer-Demo) |
| lets users enter Arabic or English text and inspect: |
|
|
| - token count and tokens per word; |
| - token IDs and raw token pieces; |
| - original text spans and offsets; |
| - decoded text for round-trip verification. |
|
|
| ## Validation |
|
|
| The released validation run reported zero round-trip failures and zero unknown |
| tokens across samples from schoolbooks, AraMix Arabic, FineWeb2 Arabic, |
| and FineWeb English. Additional stress tests cover Modern Standard Arabic, |
| dialects, Arabizi, English, code, mathematics, and Unicode edge cases. |
|
|
| See [`reports/validation.json`](reports/validation.json) and |
| [`reports/stress.json`](reports/stress.json). |
|
|
| ## Benchmark methodology |
|
|
| - All available examples were used, followed by exact-text deduplication within |
| each benchmark. |
| - Token counts exclude chat templates and added special tokens. |
| - Tokens per word is total tokens divided by whitespace-separated words. |
| - The primary macro ranking gives each benchmark 25% weight. |
| - The pooled report sums all tokens and words before calculating the ratio. |
| - Tokenizers producing identical results are tokenizer-family duplicates, not |
| independent confirmations. |
|
|
| To reproduce the benchmark: |
|
|
| ```bash |
| pip install -r benchmark/requirements.txt |
| python benchmark/benchmark.py |
| ``` |
|
|
| ## Limitations |
|
|
| - Tokenization efficiency does not measure language-model quality, reasoning, |
| factuality, safety, or downstream accuracy. |
| - Fully vocalized Arabic is the clearest weakness. One heavily diacritized |
| ArabicMMLU category accounts for most of the gap. |
| - Whitespace-separated words are a transparent denominator, not a linguistic |
| segmentation of Arabic morphology. |
| - Replacing the tokenizer of an already pretrained model requires resizing and |
| retraining its embedding and output layers. This is not a drop-in vocabulary |
| replacement. |
|
|
| ## License and data terms |
|
|
| The tokenizer files and repository code are released under Apache-2.0. Source |
| and benchmark datasets remain subject to their own licenses and terms. No |
| source-corpus or benchmark documents are redistributed here. |
|
|
