AraToken tokenizers

This repository holds the tokenizers of the paper AraToken: Optimizing Arabic Tokenization with Normalization Pipeline and Language Extension for Qwen3. Code: https://github.com/mariklolik/Aratoken.

Folder	Content	Vocabulary	Fertility on test ↓
sentencepiece/fw2hq_sp_alif_preserve	AraToken, Unigram, Alif variants kept (default)	150,000	1.212
sentencepiece/fw2hq_sp_alif_preserve_bf	the same with byte fallback	150,000	1.215
sentencepiece/fw2hq_sp_alif_preserve_0_99	pruned to 99% corpus coverage	109,844	1.227
sentencepiece/fw2hq_sp_alif_preserve_0_95	pruned to 95% corpus coverage	60,149	1.281
sentencepiece/fw2hq_sp_alif_unified*	Alif unification on (150K, 99%, 95%)
sentencepiece/fw2hq_{bpe,wp,sp}_{drop,keep}[_norm]	the algorithm × preprocessing grid at 80K	80,000
extended/fw2hq-alif-preserve-150k	Qwen3 tokenizer + 131,157 AraToken pieces, used by LEP	282,826	1.307
extended/fw2hq-alif-unified-150k	the same with Alif unification	281,798	1.301

For reference, Qwen3's own tokenizer has a fertility of 2.179 and AraBERT's 1.833. SentencePiece models expect normalized input (see aratoken.normalizer.paper_normalizer in the code repository). The extended tokenizers normalize internally.

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("mariklolik/AraToken", subfolder="extended/fw2hq-alif-preserve-150k")

Citation

@article{kashirskiy2025aratoken,
  title   = {AraToken: Optimizing Arabic Tokenization with Normalization Pipeline and Language Extension for Qwen3},
  author  = {Kashirskiy, Mark and Lipinski, Artiom and Makarov, Ilya},
  journal = {arXiv preprint arXiv:2512.18399},
  year    = {2025}
}