README.md

---
license: other
language:
- km
- en
tags:
- khmer
- cambodia
- sentencepiece
- tokenizer
- unigram
- code-switching
pretty_name: Khmer SentencePiece 8K
---

# khmer-sp-8k

A **SentencePiece unigram** tokenizer (vocab size 8000) for Khmer with code-switched
English, trained on [`Panhapich/khmer-text-corpus`](https://huggingface.co/datasets/Panhapich/khmer-text-corpus). It is the
**shared vocabulary** for OCR, ASR, and the diffusion decoder in the Khmer multimodal project.

## The tokenizer is several files working together, not one

`khmer_sp.model` was trained on **word-segmented** text (via `khmer-nltk`), not raw Khmer —
this is what prevents SentencePiece from fusing multiple words into a single vocab token (e.g.
`ខ្ញុំចង់` = "I"+"want" collapsing into one piece). Glued English/Latin runs (e.g. `tryAImodel`)
are also decomposed via `wordninja`, guarded by a case-sensitive exception list so domain terms
(ACLEDA, Bakong, COVID-19, 5G, ...) aren't mangled. See `khmer_segmentation_sentencepiece_approach.md`
for the full root-cause writeup.

**This means `khmer_sp.model` must never be loaded with a bare `SentencePieceProcessor` for
encode/decode.** Always go through `khmer_segmentation.KhmerTokenizer`, which runs the same
segmentation pipeline used at training time:

```python
from huggingface_hub import hf_hub_download
from khmer_segmentation import KhmerTokenizer

sp_model_path  = hf_hub_download("Panhapich/khmer-sp-8k", "khmer_sp.model")
gazetteer_path = hf_hub_download("Panhapich/khmer-sp-8k", "gazetteer.json")
latin_path     = hf_hub_download("Panhapich/khmer-sp-8k", "latin_exceptions.json")

tok = KhmerTokenizer(sp_model_path, gazetteer_path, latin_path)

text = "ខ្ញុំកំពុងប្រើ AI model"
ids = tok.encode(text)
print(ids)
print(tok.decode(ids))

MASK = tok.sp.piece_to_id("<MASK>")
assert MASK not in (-1, tok.sp.unk_id())   # <MASK> must be a real token
```

Also download `khmer_segmentation.py` itself (`hf_hub_download("Panhapich/khmer-sp-8k", "khmer_segmentation.py")`)
and import it locally, or vendor it into your own project — it is a required part of the
tokenizer, not optional preprocessing.

## Files

| File | Description |
|---|---|
| `khmer_sp.model` | SentencePiece model, trained on **word-segmented** text (load via `KhmerTokenizer`, not directly) |
| `khmer_sp.vocab` | Human-readable vocabulary with log-probabilities |
| `khmer_segmentation.py` | Required segmentation pipeline (gazetteer + non-Khmer masking + khmer-nltk + wordninja) |
| `gazetteer.json` | Khmer loanword/acronym list masked before segmentation so it isn't shattered mid-word |
| `latin_exceptions.json` | English/Latin domain terms protected from `wordninja`'s glued-word decomposition |
| `tokenizer_info.json` | Machine-readable manifest: vocab size + special-token IDs + provenance |

## Special tokens

| Token | ID | Role |
|---|---|---|
| `<PAD>` | 0 | Padding |
| `<UNK>` | 1 | Unknown |
| `<BOS>` | 2 | Begin of sequence |
| `<EOS>` | 3 | End of sequence |
| `<MASK>` | 4 | Diffusion/masked-LM corruption token (user-defined, never split) |

`<MASK>` is a SentencePiece `user_defined_symbol`, so it is one atomic token and must not collapse
onto `<UNK>`. The authoritative IDs are in `tokenizer_info.json`.

## Training details

| Setting | Value |
|---|---|
| Algorithm | SentencePiece `unigram` |
| Vocab size | 8000 |
| Character coverage | 0.9998 |
| Pre-segmentation | khmer-nltk word tokenization + gazetteer/non-Khmer masking + wordninja Latin decomposition (required at inference too) |
| Training subsample | up to 2,000,000 sentences (shuffled) |
| Corpus | `Panhapich/khmer-text-corpus` (Khmer + organically code-switched English) |

## Licensing

Trained on a corpus derived from third-party datasets; distribution terms follow that corpus.
Confirm the upstream licenses and set the `license` field accordingly — `other` is a placeholder.