---
license: apache-2.0
library_name: llama-cpp-python
tags:
  - llama
  - instruction-tuned
  - thai
  - gguf
  - quantized
  - q8
  - rag
  - chatbot
language:
  - th
---

# Llama 3.2 Typhoon2 3B Instruct (GGUF Q8_0)

Fine-tuned Thai instruction-following model quantized to GGUF Q8_0 format for efficient inference.

## Model Details

- **Base Model**: typhoon-ai/llama3.2-typhoon2-3b-instruct
- **Format**: GGUF (Q8_0 quantization)
- **Parameters**: 3 billion
- **Language**: Thai
- **Use Case**: Context-aware Q&A, RAG systems, chatbots

## Training

- **Framework**: Unsloth
- **Method**: Supervised Fine-Tuning (SFT)
- **Training Data**: Thai instruction-following dataset with negative samples for strictness
- **Optimization**: LoRA + 4-bit quantization during training

## Inference

### Using llama-cpp-python

```python
from llama_cpp import Llama

llm = Llama(
    model_path="model.gguf",
    n_ctx=4096,
    n_gpu_layers=0,
)

response = llm(prompt, max_tokens=256, temperature=0.0)
```

### Docker Deployment (EKS)

See deployment guide in the chat-inference Helm chart.

## Performance

- **Quantization**: Q8_0 (8-bit)
- **Model Size**: ~3.3 GB
- **Inference Speed (CPU)**: ~2-5 tokens/sec (t3.xlarge)
- **Recommended CPU**: 2-4 cores, 4-6 GB RAM

## License

Apache License 2.0