khmer-sp-8k

A SentencePiece unigram tokenizer (vocab size 8000) for Khmer with code-switched English, trained on Panhapich/khmer-text-corpus. It is the shared vocabulary for OCR, ASR, and the diffusion decoder in the Khmer multimodal project.

The tokenizer is several files working together, not one

khmer_sp.model was trained on word-segmented text (via khmer-nltk), not raw Khmer — this is what prevents SentencePiece from fusing multiple words into a single vocab token (e.g. ខ្ញុំចង់ = "I"+"want" collapsing into one piece). Glued English/Latin runs (e.g. tryAImodel) are also decomposed via wordninja, guarded by a case-sensitive exception list so domain terms (ACLEDA, Bakong, COVID-19, 5G, ...) aren't mangled. See khmer_segmentation_sentencepiece_approach.md for the full root-cause writeup.

This means khmer_sp.model must never be loaded with a bare SentencePieceProcessor for encode/decode. Always go through khmer_segmentation.KhmerTokenizer, which runs the same segmentation pipeline used at training time:

from huggingface_hub import hf_hub_download
from khmer_segmentation import KhmerTokenizer

sp_model_path  = hf_hub_download("Panhapich/khmer-sp-8k", "khmer_sp.model")
gazetteer_path = hf_hub_download("Panhapich/khmer-sp-8k", "gazetteer.json")
latin_path     = hf_hub_download("Panhapich/khmer-sp-8k", "latin_exceptions.json")

tok = KhmerTokenizer(sp_model_path, gazetteer_path, latin_path)

text = "ខ្ញុំកំពុងប្រើ AI model"
ids = tok.encode(text)
print(ids)
print(tok.decode(ids))

MASK = tok.sp.piece_to_id("<MASK>")
assert MASK not in (-1, tok.sp.unk_id())   # <MASK> must be a real token

Also download khmer_segmentation.py itself (hf_hub_download("Panhapich/khmer-sp-8k", "khmer_segmentation.py")) and import it locally, or vendor it into your own project — it is a required part of the tokenizer, not optional preprocessing.

Files

File	Description
khmer_sp.model	SentencePiece model, trained on word-segmented text (load via KhmerTokenizer, not directly)
khmer_sp.vocab	Human-readable vocabulary with log-probabilities
khmer_segmentation.py	Required segmentation pipeline (gazetteer + non-Khmer masking + khmer-nltk + wordninja)
gazetteer.json	Khmer loanword/acronym list masked before segmentation so it isn't shattered mid-word
latin_exceptions.json	English/Latin domain terms protected from wordninja's glued-word decomposition
tokenizer_info.json	Machine-readable manifest: vocab size + special-token IDs + provenance

Special tokens

Token	ID	Role
<PAD>	0	Padding
<UNK>	1	Unknown
<BOS>	2	Begin of sequence
<EOS>	3	End of sequence
<MASK>	4	Diffusion/masked-LM corruption token (user-defined, never split)

<MASK> is a SentencePiece user_defined_symbol, so it is one atomic token and must not collapse onto <UNK>. The authoritative IDs are in tokenizer_info.json.

Training details

Setting	Value
Algorithm	SentencePiece unigram
Vocab size	8000
Character coverage	0.9998
Pre-segmentation	khmer-nltk word tokenization + gazetteer/non-Khmer masking + wordninja Latin decomposition (required at inference too)
Training subsample	up to 2,000,000 sentences (shuffled)
Corpus	Panhapich/khmer-text-corpus (Khmer + organically code-switched English)

Licensing

Trained on a corpus derived from third-party datasets; distribution terms follow that corpus. Confirm the upstream licenses and set the license field accordingly — other is a placeholder.