README.md

---
language:
- th
- en
license: apache-2.0
---

# ⚡ KordAI TokenizerFast

**KordAI TokenizerFast** is a fast Hugging Face tokenizer designed for efficient text preprocessing for **Thai and English** language models.

It is fully compatible with the 🤗 Transformers ecosystem and can be used for both training and inference with compatible models.

---

# ✨ Features

- 🚀 Fast tokenizer implementation
- 🇹🇭 Thai language support
- 🇺🇸 English language support
- 🤗 Compatible with Hugging Face Transformers
- ⚡ Efficient tokenization and decoding
- 💻 Suitable for both training and inference

---

# 📋 Model Details

| Item | Value |
|------|-------|
| Repository | `KordAI/TokenizerFast` |
| Type | Fast Tokenizer |
| Languages | Thai, English |
| Library | Transformers |
| License | Apache-2.0 |

---

# 🎯 Intended Use

This tokenizer is designed for:

- Thai text tokenization
- English text tokenization
- Multilingual NLP pipelines
- Dataset preprocessing
- Language model training
- Language model inference

---

# 🐍 Usage

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "KordAI/TokenizerFast"
)

text = "สวัสดี Hello!"

tokens = tokenizer(text)

print(tokens)

decoded = tokenizer.decode(tokens["input_ids"])
print(decoded)
```

---

# ⚠️ Notes

- This repository contains only the tokenizer.
- It should be used together with a compatible language model.
- Performance depends on the model paired with this tokenizer.

---

# 🙏 Acknowledgments

Special thanks to:

- **Hugging Face** for the Transformers ecosystem.
- **KordAI** for developing and distributing the tokenizer.
- The open-source AI community for advancing multilingual NLP.

---

# 📖 Citation

```bibtex
@misc{kordaitokenizerfast2026,
  title={KordAI TokenizerFast},
  author={KordAI},
  year={2026},
  publisher={Hugging Face},
  howpublished={https://huggingface.co/KordAI/TokenizerFast}
}
```