Initial commit of Lao tokenizer

README.md

@@ -1,3 +1,33 @@
----
-license: apache-2.0
----
+# Lao Tokenizer
+
+A dictionary-based tokenizer for the Lao language, compatible with `laonlp` for POS tagging.
+
+## Usage
+```python
+from transformers import PreTrainedTokenizerFast
+from laonlp import pos_tag
+from laonlp_enhanced import tokenize  # Optional: use the library
+
+# Direct HF tokenizer (for encoding, less relevant here)
+tokenizer = PreTrainedTokenizerFast(tokenizer_file="lao_tokenizer.json")
+# Use laonlp-enhanced for actual tokenization
+tokens = tokenize("ສະບາຍດີ 2025")
+print(tokens) 
+print(pos_tag(tokens))  
+```
+## Reference
+- [laonlp](Wannaphong Phatthiyaphaibun. (2022). LaoNLP: Lao language Natural Language Processing. Zenodo. https://doi.org/10.5281/zenodo.6833407)
+
+
+## License
+Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.