metadata
language:
- th
license: mit
tags:
- thai
- nlp
- ner
- sentiment-analysis
- question-answering
- multi-task
- transformer
- pytorch
- custom-model
datasets:
- pythainlp/thainer-corpus-v2.2
- pythainlp/wisesight_sentiment
- iapp_wiki_qa_squad
pipeline_tag: token-classification
🇹🇭 Thai NLP Toolkit
A multi-task NLP framework for the Thai language built from scratch with PyTorch.
Uses a shared Transformer encoder backbone with three task-specific heads:
| Task | Head | Metric |
|---|---|---|
| Named Entity Recognition | Token classification (7 labels) | Entity-level F1 |
| Sentiment Analysis | Sentence classification (3 labels) | Macro-F1 |
| Question Answering | Extractive span prediction | EM / F1 |
Model Architecture
- Tokenizer: SentencePiece BPE (32K vocab) with Thai-specific preprocessing
- Encoder: 6-layer Transformer (d_model=256, 8 heads, d_ff=1024)
- Max sequence length: 512 tokens
Usage
# Clone the repository first
# git clone https://github.com/puttibenz/thai-nlp-toolkit.git
from inference.pipeline import ThaiNLPPipeline
pipeline = ThaiNLPPipeline(model_dir="path/to/downloaded/model", device="auto")
# NER
result = pipeline.predict("สมชายทำงานที่กรุงเทพ", task="ner")
# Sentiment Analysis
result = pipeline.predict("อาหารอร่อยมากครับ", task="sentiment")
# Question Answering
result = pipeline.predict(
"กรุงเทพมหานครเป็นเมืองหลวงของประเทศไทย",
task="qa",
question="เมืองหลวงของประเทศไทยคืออะไร"
)
Training Data
| Dataset | Task | Source |
|---|---|---|
| ThaiNER v2.2 | NER | pythainlp/thainer-corpus-v2.2 |
| Wisesight Sentiment | Sentiment | pythainlp/wisesight_sentiment |
| iApp Thai Wiki QA | QA | iapp_wiki_qa_squad |
Training Details
- Framework: PyTorch (custom implementation)
- Training: Multi-task learning with round-robin sampling
- Optimizer: AdamW with cosine LR schedule + warmup
- Mixed Precision: FP16 on CUDA
- Batch Size: 32 (×4 gradient accumulation = effective 128)
File Structure
thai-nlp-toolkit/
├── checkpoint.pt # Model weights
├── config.yaml # Model architecture config
└── tokenizer/
├── thai_bpe.model # SentencePiece BPE model
└── tokenizer_config.json # Tokenizer config
Source Code
GitHub: puttibenz/thai-nlp-toolkit
License
MIT