perixai/perix-tokenizer-tr-49k

tr-49k

A byte-level BPE tokenizer for Turkish, 49,152 tokens, trained by PerixAI for the Perix-1 model. It needs 1.693 tokens per Turkish word (4.95 bytes per token), which is fewer than any tokenizer we measured against it: shorter sequences mean more text per training step and cheaper inference on the same text.

Frozen as tr-49k@1. The files here are the ones every Perix model is trained with, and they will not change; a new tokenizer gets a new version.

How it compares

Measured on the same held-out Turkish text (80,000 web and Wikipedia documents, 24,907,244 words) and English text (10,000 documents).

tokenizer	vocabulary	TR tokens/word	TR bytes/token	EN tokens/word	TR share of vocab
tr-49k	49,152	1.693	4.95	1.547	71.9%
ufakzeka-1	40,960	1.744	4.81	1.898	75.8%
Kumru-2B	50,176	1.821	4.61	2.069	61.9%
Qwen3-0.6B	151,669	2.643	3.17	1.339	20.0%

Lower tokens per word is better. "Turkish share of vocabulary" is how much of the vocabulary is spent on Turkish rather than other languages.

Use it

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("perixai/perix-tokenizer-tr-49k")
ids = tokenizer.encode("Kütüphane bu hafta yeniden açıldı.")
print(len(ids), tokenizer.decode(ids))

It also loads with the tokenizers library alone:

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("tokenizer.json")

Chat messages use the ChatML form, which apply_chat_template writes for you: <|im_start|>user\n…<|im_end|>.

What is inside

• Byte-level BPE, NFC normalization, 48,832 merges.
• Ids 0 to 63 are special tokens, 64 to 319 the raw bytes, and the rest are merges. End of text is <|endoftext|> (id 0) and padding <|pad|> (id 3).
• Pre-tokenizer regex: [^\r\n\p{L}\p{N}]?\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+

Gates it passed

• 1.6934 tokens per Turkish word, against a limit of 1.7.
• Exact round trip on 100,000 documents: decoding the ids returns the original text, byte for byte.
• The tokenizer unit tests: apostrophes, casing (Turkish dotted and dotless i), digits.

What it was trained on

10.0 GB of text, sampled to these shares, each dataset pinned to a revision:

dataset	subset	share	revision
HuggingFaceFW/fineweb-2	data/tur_Latn/train	50.0%	af9c13333eb9
epfml/FineWeb2-HQ	tur_Latn	22.0%	c0c06e94fd3a
HuggingFaceFW/finepdfs	data/tur_Latn/train	11.0%	220bac3acbf0
wikimedia/wikipedia	20231101.tr	5.0%	b04c8d1ceb2f
HuggingFaceFW/fineweb-edu	sample/10BT	7.0%	87f09149ef47
HuggingFaceTB/finemath	finemath-4plus	3.0%	e92b25a61673
codeparrot/github-code-clean	data	2.0%	c48d40f9e70f

Turkish is 88.0% of the sample by design; the English and code shares keep the tokenizer usable on the English and code that Turkish writing mixes in.

Licensing and attribution

The tokenizer files are released under Apache-2.0. It was trained on text under ODC-By 1.0 (the FineWeb family, subject to the Common Crawl terms of use), CC BY-SA (Wikipedia, with attribution to Wikipedia contributors) and the licenses of the sources above. A tokenizer records statistics of that text, not the text itself.

Provenance

• Training run p1-tok-20260922-a.
• Repository commit 725daaa54c08.
• tokenizer.json SHA-256 4d08fa1aea99a4a97de3ade3726270ef7e467bf3e50a3a71782f77800aa9b6be.
• family.yaml in this repository holds the full record, including every gate number.