Kurdish Tokenizer (Unigram, 64k) — Kurmancî · Soranî · Zazakî
A Unigram (SentencePiece-style) tokenizer covering all three major Kurdish varieties in a single vocabulary: Kurmancî (Latin script), Soranî (Arabic script), and Zazakî (Latin script).
Trained by Kurdish-Tech on KurdishCorpus-clean, and measured on held-out text the tokenizer never saw during training.
Which variant should I use?
Four Kurdish tokenizers were trained and benchmarked together. This one has the best fertility on Soranî and Zazakî of all four — if those dialects matter most for your use case, start here. For Kurmancî-heaviest workloads, or for standard causal LM pretraining (Llama/GPT/Mistral-style architectures expect BPE), see kurdish-tokenizer-bpe-64k instead — it's marginally better on Kurmancî and is the more conventional choice for that family of models.
Fertility (tokens per word — lower is better)
Measured on held-out documents: 300 per dialect (166 for Zazakî — all that met the threshold), each ≥20 words, truncated to 2,000 characters. Every tokenizer below was measured with the same script on the same documents, encoding without special tokens.
| Tokenizer | Kurmancî | Soranî | Zazakî |
|---|---|---|---|
| kurdish-bpe-64k | 1.342 | 1.793 | 2.408 |
| kurdish-unigram-64k (this model) | 1.385 | 1.633 | 2.290 |
| kurdish-bpe-32k | 1.427 | 1.974 | 2.701 |
| kurdish-unigram-32k | 1.472 | 1.843 | 2.580 |
NLLB-200 (distilled-600M) |
1.930 | 2.336 | 2.548 |
XLM-RoBERTa (base) |
1.751 | 3.695 | 2.527 |
o200k_base (GPT-4o) |
2.361 | 3.984 | 2.732 |
cl100k_base (GPT-4) |
2.610 | 6.938 | 3.038 |
Against cl100k_base, this tokenizer needs 1.9× fewer tokens for Kurmancî and
4.2× fewer for Soranî — the best Soranî result of anything tested, including NLLB-200,
which was purpose-built for 200 languages including Kurdish.
Usage
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("kurdish-tech/kurdish-tokenizer-unigram-64k")
tok("Ez kurd im, ji Kurdistanê me.") # Kurmancî
tok("زمانی کوردی زمانێکی دەوڵەمەندە.") # Soranî
tok("Ma kirmanc î, zon u kulturê ma.") # Zazakî
<bos> / <eos> are appended automatically. Like other SentencePiece/Unigram tokenizers,
a leading space is part of the normalized representation — decode() reproduces the
original text once that convention is accounted for (standard tokenizer.decode() handles
this correctly; it only shows up if you inspect raw token strings).
| Vocabulary size | 64,000 |
| Algorithm | Unigram (SentencePiece) |
| Special tokens | <bos>, <eos>, <unk>, <pad>, <mask> |
model_max_length |
1024 |
Training data
Same 800k-line sample as the BPE variants, over-weighted toward Soranî and Zazakî relative to their share of the corpus:
| Dialect | Lines sampled | Share of training sample |
|---|---|---|
| Kurmancî | 520,000 | 68.3% |
| Soranî | 224,000 | 29.4% |
| Zazakî | 17,309 | 2.3% |
Other variants
- kurdish-tokenizer-bpe-64k — best Kurmancî fertility, standard choice for causal LM
- kurdish-tokenizer-bpe-32k — smaller vocab, smaller embedding table
- kurdish-tokenizer-unigram-32k — smaller vocab, smaller embedding table
Limitations
- Fertility is a relative comparison on a held-out sample, not an exhaustive evaluation. The Zazakî figure rests on 166 documents and is the least robust of the three.
- Lower fertility means fewer tokens for the same text; it does not by itself guarantee better downstream model quality.
- This is a tokenizer only — no language model weights are released here.
License & citation
Released under CC BY-SA 4.0, matching the corpus it was trained on.
@misc{kurdishtech2026tokenizer,
title = {Kurdish Tokenizer (Unigram 64k): a multi-dialect tokenizer for Kurmanc\^i, Soran\^i and Zazak\^i},
author = {{Kurdish-Tech}},
year = {2026},
url = {https://huggingface.co/kurdish-tech/kurdish-tokenizer-unigram-64k}
}
Built by Kurdish-Tech — open-source digital infrastructure for the Kurdish language.
Maintained by Alan Hesen.