chronorus/chatbot-poc

Llama 3.2 Typhoon2 3B Instruct (GGUF Q8_0)

Fine-tuned Thai instruction-following model quantized to GGUF Q8_0 format for efficient inference.

Model Details

• Base Model: typhoon-ai/llama3.2-typhoon2-3b-instruct
• Format: GGUF (Q8_0 quantization)
• Parameters: 3 billion
• Language: Thai
• Use Case: Context-aware Q&A, RAG systems, chatbots

Training

• Framework: Unsloth
• Method: Supervised Fine-Tuning (SFT)
• Training Data: Thai instruction-following dataset with negative samples for strictness
• Optimization: LoRA + 4-bit quantization during training

Inference

Using llama-cpp-python

from llama_cpp import Llama

llm = Llama(
    model_path="model.gguf",
    n_ctx=4096,
    n_gpu_layers=0,
)

response = llm(prompt, max_tokens=256, temperature=0.0)

Docker Deployment (EKS)

See deployment guide in the chat-inference Helm chart.

Performance

• Quantization: Q8_0 (8-bit)
• Model Size: ~3.3 GB
• Inference Speed (CPU): ~2-5 tokens/sec (t3.xlarge)
• Recommended CPU: 2-4 cores, 4-6 GB RAM

License

Apache License 2.0