Update README.md

README.md

@@ -17,10 +17,77 @@ tags:
 - unigram
 - syllable-aware
 - datarrx
+---
+# DatarrX / myX-Tokenizer ⚔️
+
+**myX-Tokenizer** is a high-performance, syllable-aware **Unigram Tokenizer** specifically engineered for the Burmese language. Developed by [**Khant Sint Heinn (Kalix Louis)**](https://huggingface.co/kalixlouiis) under [**DatarrX (Myanmar Open Source NGO)**](https://huggingface.co/DatarrX), this model is designed to bridge the gap in Myanmar Natural Language Processing (NLP) by providing efficient and linguistically meaningful text segmentation.
+
+## 🎯 Core Objectives
+
+Current tokenization methods for Burmese often suffer from excessive character-level fragmentation or a lack of understanding of syllabic structures. **myX-Tokenizer** addresses these issues through:
+
+* **Syllabic Integrity:** Optimized to preserve the structural meaning of Burmese syllables, preventing meaningless character splits.
+* **Bilingual Optimization:** Expertly handles code-mixed (Burmese + English) contexts, maintaining high efficiency for both languages within a single string.
+* **LLM Compatibility:** Designed to reduce token counts for Large Language Models (LLMs), effectively lowering inference latency and computational costs.
+
+---
+
+## 🛠️ Technical Specifications
+
+This model was trained directly on cleaned raw text without heavy pre-processing to ensure the highest degree of data fidelity.
+
+* **Algorithm:** Unigram Language Model (Offers a probabilistic approach superior to standard BPE for morphological richness).
+* **Vocabulary Size:** 64,000.
+* **Normalization:** NFKC (Normalization Form KC).
+* **Key Features:** * **Byte-fallback:** Robust handling of out-of-vocabulary (OOV) characters.
+    * **Split Digits:** Separate tokens for numerical values for better mathematical context.
+    * **Dummy Prefix:** Automatic handling of word boundaries.
+
+### Training Data
+Trained on the [kalixlouiis/myX-Corpus](https://huggingface.co/datasets/kalixlouiis/myX-Corpus), utilizing a high-quality selection of **1.5 million** Burmese-English mixed sentences.
+
+
+
+---
+
+## ⚠️ Limitations & Considerations
+
+* **Orthographic Sensitivity:** Tokenization quality is highly dependent on the correct spelling of the source text.
+* **English-Only Performance:** While highly efficient for mixed text, token counts may be slightly higher than global tokenizers in purely English contexts.
+* **Domain Variance:** Rare Pali/Sanskrit loanwords or ancient scripts may revert to character-level tokenization.
+
+---
+
+## 💻 Usage Guide
+
+To use this model, you need the `sentencepiece` library. You can load and use the model directly using the following snippet:
+
+```python
+import sentencepiece as spm
+from huggingface_hub import hf_hub_download
+
+# Download the model from Hugging Face
+model_path = hf_hub_download(repo_id="DatarrX/myX-Tokenizer", filename="myX-Tokenizer.model")
+
+# Initialize the processor
+sp = spm.SentencePieceProcessor(model_file=model_path)
+
+# Tokenize example text
+text = "မြန်မာစာ NLP နည်းပညာ ဖွံ့ဖြိုးတိုးတက်ရေးအတွက် ကျွန်တော်တို့ ကြိုးစားနေပါသည်။"
+tokens = sp.encode_as_pieces(text)
+
+print(f"Tokens: {tokens}")
+```
+# ✍️ Project Authors
+- Developer: [**Khant Sint Heinn (Kalix Louis)**](https://huggingface.co/kalixlouiis)
+- Organization: [**DatarrX (Myanmar Open Source NGO)**](https://huggingface.co/DatarrX)
+
+We are committed to advancing the Burmese NLP ecosystem. For feedback or collaboration, please use the Hugging Face Discussion tab.
+
 ---
 # DatarrX - myX-Tokenizer
 
-မြန်မာဘာသာစကားအတွက် အထူးရည်ရွယ်၍ တည်ဆောက်ထားသော Syllable-aware Unigram Tokenizer တစ်ခု ဖြစ်ပါသည်။ ဤ Model ကို [**DatarrX (Open Source NGO)**](https://huggingface.co/DatarrX) မှ ထုတ်ဝေခြင်းဖြစ်ပြီး [**Khant Sint Heinn (Kalix Louis)**](https://huggingface.co/kalixlouiis) မှ အဓိက ဖန်တီးတည်ဆောက်ထားခြင်း ဖြစ်ပါသည်။
+မြန်မာဘာသာစကားအတွက် အထူးရည်ရွယ်၍ တည်ဆောက်ထားသော Syllable-aware Unigram Tokenizer တစ်ခု ဖြစ်ပါသည်။ ဤ Model ကို [**DatarrX (Myanmar Open Source NGO)**](https://huggingface.co/DatarrX) မှ ထုတ်ဝေခြင်းဖြစ်ပြီး [**Khant Sint Heinn (Kalix Louis)**](https://huggingface.co/kalixlouiis) မှ အဓိက ဖန်တီးတည်ဆောက်ထားခြင်း ဖြစ်ပါသည်။
 
 ## 🎯 Model ရဲ့ ရည်ရွယ်ချက်နှင့် အဓိက အယူအဆ (Core Concept & Motivation)
 
@@ -75,7 +142,7 @@ print(f"Pieces: {sp.encode_as_pieces(text)}")
 ```
 
 # ✍️ Project Authors
-- Principal Developer: [**Khant Sint Heinn (Kalix Louis)**](https://huggingface.co/kalixlouiis)
+- Developer: [**Khant Sint Heinn (Kalix Louis)**](https://huggingface.co/kalixlouiis)
 - Organization: [**DatarrX (Myanmar Open Source NGO)**](https://huggingface.co/DatarrX)
 
 ဤ Model နှင့် ပတ်သက်၍ အကြံပြုချက်များ သို့မဟုတ် မေးမြန်းလိုသည်များရှိပါက Hugging Face Discussion မှတစ်ဆင့် ဆက်သွယ်နိုင်ပါသည်။ ကျွန်တော်တို့သည် မြန်မာစာ NLP ဖွံ့ဖြိုးတိုးတက်ရေးအတွက် အမြဲမပြတ် ကြိုးစားနေပါသည်။