Instructions to use kurdish-tech/kurdish-tokenizer-unigram-32k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use kurdish-tech/kurdish-tokenizer-unigram-32k with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="kurdish-tech/kurdish-tokenizer-unigram-32k")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("kurdish-tech/kurdish-tokenizer-unigram-32k", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use kurdish-tech/kurdish-tokenizer-unigram-32k with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "kurdish-tech/kurdish-tokenizer-unigram-32k"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kurdish-tech/kurdish-tokenizer-unigram-32k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/kurdish-tech/kurdish-tokenizer-unigram-32k

SGLang

How to use kurdish-tech/kurdish-tokenizer-unigram-32k with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "kurdish-tech/kurdish-tokenizer-unigram-32k" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kurdish-tech/kurdish-tokenizer-unigram-32k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "kurdish-tech/kurdish-tokenizer-unigram-32k" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "kurdish-tech/kurdish-tokenizer-unigram-32k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use kurdish-tech/kurdish-tokenizer-unigram-32k with Docker Model Runner:
```
docker model run hf.co/kurdish-tech/kurdish-tokenizer-unigram-32k
```

Kurdish Tokenizer (Unigram, 32k) — Kurmancî · Soranî · Zazakî

A Unigram (SentencePiece-style) tokenizer covering all three major Kurdish varieties in a single vocabulary: Kurmancî (Latin script), Soranî (Arabic script), and Zazakî (Latin script).

Trained by Kurdish-Tech on KurdishCorpus-clean, and measured on held-out text the tokenizer never saw during training.

Which variant should I use?

Honestly, probably not this one. Four Kurdish tokenizers were trained and benchmarked together, and this 32k-vocabulary Unigram variant has the second-weakest fertility of the four. Its only reason to exist is a smaller embedding table (32k vs 64k rows) for parameter-constrained models. If that constraint doesn't apply to you, use kurdish-tokenizer-unigram-64k instead — same algorithm, the best Soranî/Zazakî fertility of all four variants.

Fertility (tokens per word — lower is better)

Measured on held-out documents: 300 per dialect (166 for Zazakî — all that met the threshold), each ≥20 words, truncated to 2,000 characters. Every tokenizer below was measured with the same script on the same documents, encoding without special tokens.

Tokenizer	Kurmancî	Soranî	Zazakî
kurdish-bpe-64k	1.342	1.793	2.408
kurdish-unigram-64k	1.385	1.633	2.290
kurdish-bpe-32k	1.427	1.974	2.701
kurdish-unigram-32k (this model)	1.472	1.843	2.580
NLLB-200 (`distilled-600M`)	1.930	2.336	2.548
XLM-RoBERTa (`base`)	1.751	3.695	2.527
`o200k_base` (GPT-4o)	2.361	3.984	2.732
`cl100k_base` (GPT-4)	2.610	6.938	3.038

Still beats cl100k_base on every dialect (1.8× fewer tokens on Kurmancî, 3.8× fewer on Soranî) despite being the second-weakest of the four Kurdish variants.

Usage

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("kurdish-tech/kurdish-tokenizer-unigram-32k")

tok("Ez kurd im, ji Kurdistanê me.")          # Kurmancî
tok("زمانی کوردی زمانێکی دەوڵەمەندە.")          # Soranî
tok("Ma kirmanc î, zon u kulturê ma.")        # Zazakî


Vocabulary size	32,000
Algorithm	Unigram (SentencePiece)
Special tokens	`<bos>`, `<eos>`, `<unk>`, `<pad>`, `<mask>`
`model_max_length`	1024

Other variants

kurdish-tokenizer-unigram-64k — same algorithm, best Soranî/Zazakî fertility of all four (recommended)
kurdish-tokenizer-bpe-64k — best Kurmancî fertility, standard choice for causal LM
kurdish-tokenizer-bpe-32k — BPE at the same vocab size

Limitations

Fertility is a relative comparison on a held-out sample, not an exhaustive evaluation.
Published for completeness and for use cases genuinely constrained on vocabulary size, not as the general recommendation.
This is a tokenizer only — no language model weights are released here.

License & citation

Released under CC BY-SA 4.0, matching the corpus it was trained on.

@misc{kurdishtech2026tokenizer,
  title  = {Kurdish Tokenizer (Unigram 32k): a multi-dialect tokenizer for Kurmanc\^i, Soran\^i and Zazak\^i},
  author = {{Kurdish-Tech}},
  year   = {2026},
  url    = {https://huggingface.co/kurdish-tech/kurdish-tokenizer-unigram-32k}
}

Built by Kurdish-Tech — open-source digital infrastructure for the Kurdish language.

Maintained by Alan Hesen.

Downloads last month: -; Downloads are not tracked for this model. How to track