flexitok/bpe_cmn_Hani_16000

Byte-Level BPE Tokenizer: cmn_Hani (16K)

A Byte-Level BPE tokenizer trained on cmn_Hani data from Fineweb-2-HQ.

Training Details

Parameter	Value
Algorithm	Byte-Level BPE
Language	cmn_Hani
Target Vocab Size	16,000
Final Vocab Size	16,000
Pre-tokenizer	script_aware
Number handling	individual
Contraction handling	True
Normalizer	NFC
Special Tokens	<s>, </s>, <pad>, <unk>
Training Shards	2

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("flexitok/-bpe_cmn_Hani_16000")
tokens = tokenizer.encode("Hello, world!")

Files

• tokenizer.json — Full HuggingFace tokenizer
• vocab.json — Vocabulary mapping
• merges.txt — BPE merge rules

Sample Encoding

Text	Tokens	Token IDs
Hello, world! 12345 This is a test. こんにちは	Hello, ,, Ġworld, !, Ġ, 1, 2, 3, 4, 5, ĠThis, Ġis, Ġa, Ġtest, ., Ġ, ãģĵ, ãĤĵ, ãģ«, ãģ¡	9609, 15, 8724, 4, 181, 20, 21, 22, 23, 24, 9771, 3195, 2412, 7934, 17, 181, 10273, 9388, 7056, 13105