README.md

---
library_name: transformers
language: ["khm"]
license: mit
tags: ["tokenizer", "khmer", "Unigram", "general-purpose", "sentencepiece"]
---

# 🇰🇭 KM Improved 22K Tokenizer

A **general-purpose Khmer tokenizer** optimized for both accuracy and speed.  
It provides a stable backbone for Khmer NLP applications such as classification,  
question answering, translation, and summarization.

---

## 🧠 Model Details

### Model Description
- **Developer:** Sok Meas (@Msok99)  
- **Model type:** SentencePiece Unigram Tokenizer  
- **Language:** Khmer (khm)  
- **License:** MIT  
- **Finetuned from:** None (trained from scratch)

### Model Sources
- **Repository:** [https://huggingface.co/Msok99/km-improved-22k](https://huggingface.co/Msok99/km-improved-22k)

---

## ⚙️ Uses

### Direct Use
- Tokenizing Khmer text for downstream NLP models  
- Preparing training data for transformer-based fine-tuning  
- Segmenting sentences for analysis or embedding generation  

### Downstream Use
- Integration into Khmer LLMs or chatbots  
- Pre- and post-processing for summarization or translation systems  

### Out-of-Scope Use
- Not designed for English or heavily mixed Khmer–English content  
- Not an inference or generation model itself  

---

## ⚖️ Bias, Risks & Limitations
- Very long or compound words may still split into several sub-tokens  
- Limited exposure to informal slang or non-standard Khmer orthography  

### Recommendations
For code-switched text (Khmer + English), use the merged model  
[`Msok99/lfm2-khmer-merged-18k`](https://huggingface.co/Msok99/lfm2-khmer-merged-18k).

---

## 🚀 How to Get Started

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Msok99/km-improved-22k")

text = "ក្នុងឆ្នាំ២០២៥ កម្ពុជានឹងអភិវឌ្ឍន៍បច្ចេកវិទ្យាថ្មី។"
tokens = tokenizer.tokenize(text)
print(tokens)
print(tokenizer.decode(tokenizer.encode(text)))