Turkish BPE Tokenizer (128k Vocabulary)
This repository contains a Byte-Level BPE (Byte Pair Encoding) tokenizer specifically trained on a Turkish corpus (Uunan/turkish-synthetic-corpus). It is designed to optimally tokenize Turkish texts with a vocabulary size of 128,000 tokens.
Tokenizer Details
- Tokenizer Type: Byte-Level BPE
- Vocabulary Size: 128,000
- Normalizer: NFC (Normalization Form C)
- Special Tokens:
<pad><unk><bos><eos>
- Post-Processor: Template processing for
<bos>and<eos>tokens.
How to Use
You can easily use this tokenizer using the transformers library or the tokenizers library.
Using transformers
from transformers import PreTrainedTokenizerFast
tokenizer = PreTrainedTokenizerFast(tokenizer_file="tokenizer.json")
tokenizer.pad_token = "<pad>"
tokenizer.unk_token = "<unk>"
tokenizer.bos_token = "<bos>"
tokenizer.eos_token = "<eos>"
text = "Türkiye yapay zekâ alanında büyük yatırımlar yapıyor."
encoded = tokenizer.encode(text)
print(encoded)
decoded = tokenizer.decode(encoded)
print(decoded)
Using tokenizers
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file("tokenizer.json")
text = "Türkiye yapay zekâ alanında büyük yatırımlar yapıyor."
encoded = tokenizer.encode(text)
print("Tokens:", encoded.tokens)
print("IDs:", encoded.ids)
Training Data
This tokenizer was trained on the Uunan/turkish-synthetic-corpus dataset, which contains high-quality Turkish synthetic text data, ensuring better coverage for modern Turkish language semantics and morphological structures.
Performance
With a vocabulary size of 128k, this tokenizer minimizes the sequence length for Turkish text generation and understanding tasks, improving the efficiency of Language Models (LLMs) trained on Turkish.