Blame for README.md

---
language:
- ar
- en
license: apache-2.0
library_name: tokenizers
tags:
- tokenizer
- arabic
- arabic-nlp
---

# SamAI-Tokenizer

[**🚀 Try the interactive tokenizer demo**](https://huggingface.co/spaces/Samai-sa/SamAI-Tokenizer-Demo)

SamAI-Tokenizer is an Arabic-focused byte-level BPE tokenizer for training new
Arabic-heavy language models. It has a vocabulary of 65,536 tokens, applies
Unicode NFC normalization, supports lossless decoding, and retains English
coverage for multilingual training.

## Key specifications

| Property | Value |
| --- | --- |
| Algorithm | Byte-level BPE |
| Vocabulary size | 65,536 |
| Normalization | Unicode NFC |
| Pre-tokenizer | ByteLevel |
| Unknown-token fallback | Byte-level coverage |
| Named special tokens | `<pad>`, `<bos>`, `<eos>`, `<unk>`, `<user>`, `<assistant>`, `<system>` |
| Reserved token IDs | 7–255 |
| Primary language | Arabic |
| Secondary language | English |

## Arabic tokenizer benchmark

SamAI-Tokenizer was evaluated on four complementary Arabic benchmarks. Lower
tokens per word indicates more efficient tokenization.

![SamAI tokenizer benchmark results](assets/SamAI-Tokenizer-Benchmark.png)

Across the four-benchmark macro average, SamAI-Tokenizer achieved **1.6207 tokens** per word, the **second-lowest** result among the 15 tokenizers successfully evaluated.

| Rank | Tokenizer | Mean tokens/word | Extra tokens versus SamAI |
| ---: | --- | ---: | ---: |
| 1 | ALLaM-7B | 1.5021 | -4.86% |
| **2** | **SamAI-Tokenizer** | **1.6207** | **baseline** |
| 3 | Falcon-H1-7B | 1.8880 | +19.30% |
| 4 | Mistral-Small-3.1 | 1.8929 | +20.71% |
| 5 | Qwen3.8-Flash-Next | 1.9039 | +21.94% |
| 6 | Qwen3.5-4B | 1.9039 | +21.94% |
| 7 | tiktoken-o200k | 1.9919 | +26.37% |
| 8 | GPT-OSS-20B | 1.9919 | +26.37% |
| 9 | Fanar-1 | 2.1570 | +36.81% |
| 10 | DeepSeek-V3 | 2.2774 | +46.25% |
| 11 | Qwen2.5-7B | 2.3740 | +50.25% |
| 12 | Qwen3-8B | 2.3740 | +50.25% |
| 13 | GLM-5.3-Flash | 2.5225 | +60.21% |
| 14 | Kimi-K2.6 | 2.8693 | +84.13% |
| 15 | tiktoken-cl100k | 4.0800 | +164.63% |

## Training data

The tokenizer vocabulary was trained on a diverse mixture of Arabic and English text, including Modern Standard Arabic, dialectal Arabic, educational content, and mixed Arabic-English text.

## Installation

```bash
pip install -U tokenizers transformers huggingface_hub
```

## Usage

### Transformers

```python
from transformers import PreTrainedTokenizerFast

tokenizer = PreTrainedTokenizerFast.from_pretrained(
    "Samai-sa/SamAI-Tokenizer"
)

text = "المملكة العربية السعودية"
ids = tokenizer.encode(text, add_special_tokens=False)

print("Token count:", len(ids))
print("Token IDs:", ids)
print("Tokens:", tokenizer.convert_ids_to_tokens(ids))
print("Decoded:", tokenizer.decode(ids))
```

### Tokenizers

```python
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

path = hf_hub_download(
    repo_id="Samai-sa/SamAI-Tokenizer",
    filename="tokenizer.json",
)
tokenizer = Tokenizer.from_file(path)

encoding = tokenizer.encode(
    "السلام عليكم ورحمة الله وبركاته",
    add_special_tokens=False,
)

print("Token count:", len(encoding.ids))
print("Token IDs:", encoding.ids)
print("Tokens:", encoding.tokens)
print("Decoded:", tokenizer.decode(encoding.ids))
```

## Interactive demo

The [SamAI Tokenizer Playground](https://huggingface.co/spaces/Samai-sa/SamAI-Tokenizer-Demo)
lets users enter Arabic or English text and inspect:

- token count and tokens per word;
- token IDs and raw token pieces;
- original text spans and offsets;
- decoded text for round-trip verification.

## Validation

The released validation run reported zero round-trip failures and zero unknown
tokens across samples from schoolbooks, AraMix Arabic, FineWeb2 Arabic,
and FineWeb English. Additional stress tests cover Modern Standard Arabic,
dialects, Arabizi, English, code, mathematics, and Unicode edge cases.

See [`reports/validation.json`](reports/validation.json) and
[`reports/stress.json`](reports/stress.json).

## Benchmark methodology

- All available examples were used, followed by exact-text deduplication within
  each benchmark.
- Token counts exclude chat templates and added special tokens.
- Tokens per word is total tokens divided by whitespace-separated words.
- The primary macro ranking gives each benchmark 25% weight.
- The pooled report sums all tokens and words before calculating the ratio.
- Tokenizers producing identical results are tokenizer-family duplicates, not
  independent confirmations.

To reproduce the benchmark:

```bash
pip install -r benchmark/requirements.txt
python benchmark/benchmark.py
```

## Limitations

- Tokenization efficiency does not measure language-model quality, reasoning,
  factuality, safety, or downstream accuracy.
- Fully vocalized Arabic is the clearest weakness. One heavily diacritized
  ArabicMMLU category accounts for most of the gap.
- Whitespace-separated words are a transparent denominator, not a linguistic
  segmentation of Arabic morphology.
- Replacing the tokenizer of an already pretrained model requires resizing and
  retraining its embedding and output layers. This is not a drop-in vocabulary
  replacement.

## License and data terms

The tokenizer files and repository code are released under Apache-2.0. Source
and benchmark datasets remain subject to their own licenses and terms. No
source-corpus or benchmark documents are redistributed here.