Thai
English
Naphon commited on
Commit
4a6ac08
·
verified ·
1 Parent(s): e82a8aa

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +98 -0
README.md CHANGED
@@ -1,3 +1,101 @@
1
  ---
 
 
 
2
  license: apache-2.0
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language:
3
+ - th
4
+ - en
5
  license: apache-2.0
6
  ---
7
+
8
+ # ⚡ KordAI TokenizerFast
9
+
10
+ **KordAI TokenizerFast** is a fast Hugging Face tokenizer designed for efficient text preprocessing for **Thai and English** language models.
11
+
12
+ It is fully compatible with the 🤗 Transformers ecosystem and can be used for both training and inference with compatible models.
13
+
14
+ ---
15
+
16
+ # ✨ Features
17
+
18
+ - 🚀 Fast tokenizer implementation
19
+ - 🇹🇭 Thai language support
20
+ - 🇺🇸 English language support
21
+ - 🤗 Compatible with Hugging Face Transformers
22
+ - ⚡ Efficient tokenization and decoding
23
+ - 💻 Suitable for both training and inference
24
+
25
+ ---
26
+
27
+ # 📋 Model Details
28
+
29
+ | Item | Value |
30
+ |------|-------|
31
+ | Repository | `KordAI/TokenizerFast` |
32
+ | Type | Fast Tokenizer |
33
+ | Languages | Thai, English |
34
+ | Library | Transformers |
35
+ | License | Apache-2.0 |
36
+
37
+ ---
38
+
39
+ # 🎯 Intended Use
40
+
41
+ This tokenizer is designed for:
42
+
43
+ - Thai text tokenization
44
+ - English text tokenization
45
+ - Multilingual NLP pipelines
46
+ - Dataset preprocessing
47
+ - Language model training
48
+ - Language model inference
49
+
50
+ ---
51
+
52
+ # 🐍 Usage
53
+
54
+ ```python
55
+ from transformers import AutoTokenizer
56
+
57
+ tokenizer = AutoTokenizer.from_pretrained(
58
+ "KordAI/TokenizerFast"
59
+ )
60
+
61
+ text = "สวัสดี Hello!"
62
+
63
+ tokens = tokenizer(text)
64
+
65
+ print(tokens)
66
+
67
+ decoded = tokenizer.decode(tokens["input_ids"])
68
+ print(decoded)
69
+ ```
70
+
71
+ ---
72
+
73
+ # ⚠️ Notes
74
+
75
+ - This repository contains only the tokenizer.
76
+ - It should be used together with a compatible language model.
77
+ - Performance depends on the model paired with this tokenizer.
78
+
79
+ ---
80
+
81
+ # 🙏 Acknowledgments
82
+
83
+ Special thanks to:
84
+
85
+ - **Hugging Face** for the Transformers ecosystem.
86
+ - **KordAI** for developing and distributing the tokenizer.
87
+ - The open-source AI community for advancing multilingual NLP.
88
+
89
+ ---
90
+
91
+ # 📖 Citation
92
+
93
+ ```bibtex
94
+ @misc{kordaitokenizerfast2026,
95
+ title={KordAI TokenizerFast},
96
+ author={KordAI},
97
+ year={2026},
98
+ publisher={Hugging Face},
99
+ howpublished={https://huggingface.co/KordAI/TokenizerFast}
100
+ }
101
+ ```