Panhapich
/

khmer-sp-8k

@@ -1,74 +1,96 @@
----
-license: other
-language:
-- km
-- en
-tags:
-- khmer
-- cambodia
-- sentencepiece
-- tokenizer
-- unigram
-- code-switching
-pretty_name: Khmer SentencePiece 8K
----
-# khmer-sp-8k
-A **SentencePiece unigram** tokenizer (vocab size 8000) for Khmer with code-switched
-English, trained on [`Panhapich/khmer-text-corpus`](https://huggingface.co/datasets/Panhapich/khmer-text-corpus). It is the
-**shared vocabulary** for OCR, ASR, and the diffusion decoder in the Khmer multimodal project.
-## Files
-| File | Description |
-|---|---|
-| `khmer_sp.model` | SentencePiece model (load this) |
-| `khmer_sp.vocab` | Human-readable vocabulary with log-probabilities |
-| `tokenizer_info.json` | Machine-readable manifest: vocab size + special-token IDs + provenance |
-## Special tokens
-| Token | ID | Role |
-|---|---|---|
-| `<PAD>` | 0 | Padding |
-| `<UNK>` | 1 | Unknown |
-| `<BOS>` | 2 | Begin of sequence |
-| `<EOS>` | 3 | End of sequence |
-| `<MASK>` | 4 | Diffusion/masked-LM corruption token (user-defined, never split) |
-`<MASK>` is a SentencePiece `user_defined_symbol`, so it is one atomic token and must not collapse
-onto `<UNK>`. The authoritative IDs are in `tokenizer_info.json`.
-## Usage
-```python
-from huggingface_hub import hf_hub_download
-import sentencepiece as spm
-model_path = hf_hub_download("Panhapich/khmer-sp-8k", "khmer_sp.model")
-sp = spm.SentencePieceProcessor(model_file=model_path)
-text = "ខ្ញុំកំពុងប្រើ AI model"
-ids = sp.encode(text, out_type=int)
-print(ids)
-print(sp.decode(ids))
-MASK = sp.piece_to_id("<MASK>")
-assert MASK not in (-1, sp.unk_id())   # <MASK> must be a real token
-```
-## Training details
-| Setting | Value |
-|---|---|
-| Algorithm | SentencePiece `unigram` |
-| Vocab size | 8000 |
-| Character coverage | 0.9995 |
-| Training subsample | up to 2,000,000 sentences (shuffled) |
-| Corpus | `Panhapich/khmer-text-corpus` (Khmer + organically code-switched English) |
-## Licensing
-Trained on a corpus derived from third-party datasets; distribution terms follow that corpus.
-Confirm the upstream licenses and set the `license` field accordingly — `other` is a placeholder.

+---
+license: other
+language:
+- km
+- en
+tags:
+- khmer
+- cambodia
+- sentencepiece
+- tokenizer
+- unigram
+- code-switching
+pretty_name: Khmer SentencePiece 8K
+---
+# khmer-sp-8k
+A **SentencePiece unigram** tokenizer (vocab size 8000) for Khmer with code-switched
+English, trained on [`Panhapich/khmer-text-corpus`](https://huggingface.co/datasets/Panhapich/khmer-text-corpus). It is the
+**shared vocabulary** for OCR, ASR, and the diffusion decoder in the Khmer multimodal project.
+## The tokenizer is several files working together, not one
+`khmer_sp.model` was trained on **word-segmented** text (via `khmer-nltk`), not raw Khmer —
+this is what prevents SentencePiece from fusing multiple words into a single vocab token (e.g.
+`ខ្ញុំចង់` = "I"+"want" collapsing into one piece). Glued English/Latin runs (e.g. `tryAImodel`)
+are also decomposed via `wordninja`, guarded by a case-sensitive exception list so domain terms
+(ACLEDA, Bakong, COVID-19, 5G, ...) aren't mangled. See `khmer_segmentation_sentencepiece_approach.md`
+for the full root-cause writeup.
+**This means `khmer_sp.model` must never be loaded with a bare `SentencePieceProcessor` for
+encode/decode.** Always go through `khmer_segmentation.KhmerTokenizer`, which runs the same
+segmentation pipeline used at training time:
+```python
+from huggingface_hub import hf_hub_download
+from khmer_segmentation import KhmerTokenizer
+sp_model_path  = hf_hub_download("Panhapich/khmer-sp-8k", "khmer_sp.model")
+gazetteer_path = hf_hub_download("Panhapich/khmer-sp-8k", "gazetteer.json")
+latin_path     = hf_hub_download("Panhapich/khmer-sp-8k", "latin_exceptions.json")
+tok = KhmerTokenizer(sp_model_path, gazetteer_path, latin_path)
+text = "ខ្ញុំកំពុងប្រើ AI model"
+ids = tok.encode(text)
+print(ids)
+print(tok.decode(ids))
+MASK = tok.sp.piece_to_id("<MASK>")
+assert MASK not in (-1, tok.sp.unk_id())   # <MASK> must be a real token
+```
+Also download `khmer_segmentation.py` itself (`hf_hub_download("Panhapich/khmer-sp-8k", "khmer_segmentation.py")`)
+and import it locally, or vendor it into your own project — it is a required part of the
+tokenizer, not optional preprocessing.
+## Files
+| File | Description |
+|---|---|
+| `khmer_sp.model` | SentencePiece model, trained on **word-segmented** text (load via `KhmerTokenizer`, not directly) |
+| `khmer_sp.vocab` | Human-readable vocabulary with log-probabilities |
+| `khmer_segmentation.py` | Required segmentation pipeline (gazetteer + non-Khmer masking + khmer-nltk + wordninja) |
+| `gazetteer.json` | Khmer loanword/acronym list masked before segmentation so it isn't shattered mid-word |
+| `latin_exceptions.json` | English/Latin domain terms protected from `wordninja`'s glued-word decomposition |
+| `tokenizer_info.json` | Machine-readable manifest: vocab size + special-token IDs + provenance |
+## Special tokens
+| Token | ID | Role |
+|---|---|---|
+| `<PAD>` | 0 | Padding |
+| `<UNK>` | 1 | Unknown |
+| `<BOS>` | 2 | Begin of sequence |
+| `<EOS>` | 3 | End of sequence |
+| `<MASK>` | 4 | Diffusion/masked-LM corruption token (user-defined, never split) |
+`<MASK>` is a SentencePiece `user_defined_symbol`, so it is one atomic token and must not collapse
+onto `<UNK>`. The authoritative IDs are in `tokenizer_info.json`.
+## Training details
+| Setting | Value |
+|---|---|
+| Algorithm | SentencePiece `unigram` |
+| Vocab size | 8000 |
+| Character coverage | 0.9998 |
+| Pre-segmentation | khmer-nltk word tokenization + gazetteer/non-Khmer masking + wordninja Latin decomposition (required at inference too) |
+| Training subsample | up to 2,000,000 sentences (shuffled) |
+| Corpus | `Panhapich/khmer-text-corpus` (Khmer + organically code-switched English) |
+## Licensing
+Trained on a corpus derived from third-party datasets; distribution terms follow that corpus.
+Confirm the upstream licenses and set the `license` field accordingly — `other` is a placeholder.