feat: simplified mon tokenizer in hf format, updated tags, resolve the legacy issue

README.md

@@ -26,9 +26,10 @@ compatible with Hugging Face Transformers and the Llama tokenizer architecture.
 - **Language**: Mon (mnw)
 - **Vocabulary Size**: 4,000 tokens
 - **Algorithm**: SentencePiece (Unigram Language Model)
-- **Tokenizer Type**: LlamaTokenizer
+- **Tokenizer Type**: LlamaTokenizerFast
 - **Special Tokens**: `<s>`, `</s>`, `<unk>`, `<pad>`
 - **Context Length**: 4,096 tokens
+- **Updated**: August 31, 2025
 
 ## Usage
 
@@ -49,11 +50,12 @@ print(decoded)  # ဘာသာမန် ပရူပရာတံဂှ် ကၠ
 
 ## Technical Specifications
 
-- **Tokenizer Class**: `LlamaTokenizer`
+- **Tokenizer Class**: `LlamaTokenizerFast`
 - **Vocabulary Type**: Subword tokenization using SentencePiece
 - **Training Algorithm**: Unigram Language Model
 - **OOV Handling**: `<unk>` token for unknown words
 - **Legacy Mode**: Enabled for maximum compatibility
+- **Fast Tokenizer**: Includes tokenizer.json for optimal performance
 
 ## Training Data
 
@@ -71,6 +73,7 @@ Total training data: Not specified
 - **Coverage**: High coverage of Mon language vocabulary
 - **Efficiency**: Optimized for Mon language morphology
 - **Compatibility**: Full compatibility with Transformers 4.x
+- **Speed**: Fast tokenizer for improved performance
 
 ## License
 
@@ -81,10 +84,10 @@ This tokenizer is released under the MIT License.
 If you use this tokenizer in your research, please cite:
 
 ```bibtex
-@misc{mon_tokenizer_2024,
+@misc{mon_tokenizer_2025,
   title={Mon Language Tokenizer for Hugging Face Transformers},
   author={Mon Language Project},
-  year={2024},
+  year={2025},
   url={https://huggingface.co/janakhpon/mon_tokenizer}
 }
 ```

convert_to_hf.py

@@ -262,9 +262,10 @@ compatible with Hugging Face Transformers and the Llama tokenizer architecture.
 - **Language**: Mon (mnw)
 - **Vocabulary Size**: {analysis["vocab_size"]:,} tokens
 - **Algorithm**: SentencePiece (Unigram Language Model)
-- **Tokenizer Type**: LlamaTokenizer
+- **Tokenizer Type**: LlamaTokenizerFast
 - **Special Tokens**: `{analysis["bos_token"]}`, `{analysis["eos_token"]}`, `{analysis["unk_token"]}`, `{analysis["pad_token"]}`
 - **Context Length**: 4,096 tokens
+- **Updated**: August 31, 2025
 
 ## Usage
 
@@ -285,11 +286,12 @@ print(decoded)  # ဘာသာမန် ပရူပရာတံဂှ် ကၠ
 
 ## Technical Specifications
 
-- **Tokenizer Class**: `LlamaTokenizer`
+- **Tokenizer Class**: `LlamaTokenizerFast`
 - **Vocabulary Type**: Subword tokenization using SentencePiece
 - **Training Algorithm**: Unigram Language Model
 - **OOV Handling**: `{analysis["unk_token"]}` token for unknown words
 - **Legacy Mode**: Enabled for maximum compatibility
+- **Fast Tokenizer**: Includes tokenizer.json for optimal performance
 
 ## Training Data
 
@@ -307,6 +309,7 @@ Total training data: {training_data_info.get('total_size', 'Not specified')}
 - **Coverage**: High coverage of Mon language vocabulary
 - **Efficiency**: Optimized for Mon language morphology
 - **Compatibility**: Full compatibility with Transformers 4.x
+- **Speed**: Fast tokenizer for improved performance
 
 ## License
 
@@ -317,10 +320,10 @@ This tokenizer is released under the MIT License.
 If you use this tokenizer in your research, please cite:
 
 ```bibtex
-@misc{{mon_tokenizer_2024,
+@misc{{mon_tokenizer_2025,
   title={{Mon Language Tokenizer for Hugging Face Transformers}},
   author={{Mon Language Project}},
-  year={{2024}},
+  year={{2025}},
   url={{https://huggingface.co/janakhpon/mon_tokenizer}}
 }}
 ```