# AmkyawDev-LLM-V3

Burmese Language Model Fine-tuning Project using LoRA/QLoRA with Unsloth

<div align="center">
  <img src="https://huggingface.co/amkyawdev/AmkyawDev-LLM-V3/resolve/main/logo.svg" width="200" height="200" alt="Amkyaw AI Logo"/>
  
  # 🇲🇲 AmkyawDev-LLM-V3
  
  ### Burmese Language Model | Qwen2.5-1.5B | Unsloth Fine-tuned
  
  **Model**: [HuggingFace](https://huggingface.co/amkyawdev/AmkyawDev-LLM-V3) | **Space**: [Gradio](https://huggingface.co/spaces/amkyawdev/AmkyawDev-LLM-V3)
  
  *Drive Progress With Intelligent Systems*
</div>

---

## 📋 Table of Contents

- [Project Structure](#project-structure)
- [Quick Start](#quick-start)
- [Model Details](#model-details)
- [Training](#training)
- [Deployment](#deployment)
- [API Usage](#api-usage)
- [Limitations](#limitations)
- [License](#license)
- [Acknowledgments](#acknowledgments)

---

## 📁 Project Structure

```
AmkyawDev-LLM-V3/
├── 📁 data/                  # Dataset ပိုင်း
│   ├── raw/                  # မပြုပြင်ရသေးသော data များ (Wiki, Social, Books)
│   ├── processed/            # Clean လုပ်ပြီးသား (Unicode normalized)
│   └── chat_format/          # ShareGPT သို့မဟုတ် Alpaca format
├── 📁 training/              # Training scripts များ
│   ├── config.yaml           # LoRA/QLoRA hyperparameters
│   ├── train_lora.py         # Standard PEFT training
│   ├── train_unsloth.py      # Unsloth memory-efficient training
│   └── requirements.txt      # Dependencies
├── 📁 model/                 # Output ပိုင်း
│   ├── adapter/              # Trained LoRA weights
│   └── merged/               # Base + LoRA merged version
├── 📁 deployment/            # API နှင့် UI ပိုင်း
│   ├── 📁 api/               # FastAPI သို့မဟုတ် LiteLLM Proxy
│   └── 📁 web_ui/            # Gradio Chat Interface
├── 📁 scripts/               # Utility scripts
│   ├── convert_to_unicode.py
│   ├── push_to_hub.py        # Push to HuggingFace Hub
│   └── push_space.py         # Push to HuggingFace Spaces
└── README.md
```

---

## 🚀 Quick Start

### 1. Install Dependencies

```bash
cd training
pip install -r requirements.txt
```

### 2. Prepare Data

```bash
# Convert raw data to normalized Unicode
python scripts/convert_to_unicode.py data/raw --output data/processed
```

### 3. Configure Training

Edit `training/config.yaml`:

```yaml
model:
  name: "Qwen/Qwen2.5-1.5B-Instruct"

lora:
  r: 16
  lora_alpha: 32

training:
  num_train_epochs: 3
  learning_rate: 2e-4
  bf16: true
```

### 4. Train Model

```bash
# Unsloth (Recommended - Memory Efficient)
python training/train_unsloth.py

# Standard PEFT
python training/train_lora.py
```

### 5. Deploy

```bash
# Push to HuggingFace Hub
python scripts/push_to_hub.py

# Create Gradio Space
python scripts/push_space.py
```

---

## 📊 Model Details

| Property | Value |
|----------|-------|
| **Base Model** | Qwen/Qwen2.5-1.5B-Instruct |
| **Architecture** | Transformer (Decoder-only) |
| **Training Method** | Unsloth + QLoRA (4-bit) |
| **Context Length** | 2048 tokens |
| **Parameters** | 1.5B |
| **Fine-tuning Framework** | TRL + PEFT |

### LoRA Configuration

```yaml
lora:
  r: 16
  lora_alpha: 32
  lora_dropout: 0.05
  target_modules:
    - q_proj
    - k_proj
    - v_proj
    - o_proj
    - gate_proj
    - up_proj
    - down_proj
```

---

## 🔧 Training

### Requirements

```
torch>=2.0.0
transformers>=4.36.0
unsloth>=2024.1.0
peft>=0.8.0
trl>=0.7.0
datasets>=2.14.0
accelerate>=0.25.0
```

### Data Format (ShareGPT)

```json
{
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "မြန်မာစကားအကြောင်းပါ"},
    {"role": "assistant", "content": "ဟုတ်ပါတယ်။"}
  ]
}
```

### Data Format (Alpaca)

```json
{
  "prompt": "မြန်မာစကားသင်ပါ",
  "response": "ဟုတ်ပါတယ်။"
}
```

---

## 🌐 Deployment

### HuggingFace Space (Live Demo)

🔗 **URL**: [https://huggingface.co/spaces/amkyawdev/AmkyawDev-LLM-V3](https://huggingface.co/spaces/amkyawdev/AmkyawDev-LLM-V3)

Features:
- 🖥️ Web UI Chat Interface
- ⚙️ Adjustable Parameters (temperature, max_tokens)
- 📱 Mobile-friendly

### Local Deployment

```bash
pip install gradio transformers peft
python deployment/web_ui/app.py
```

---

## 📡 API Usage

### FastAPI

```python
from fastapi import FastAPI
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

app = FastAPI()

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("amkyawdev/AmkyawDev-LLM-V3")

@app.post("/generate")
def generate(prompt: str):
    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
    outputs = model.generate(**inputs, max_new_tokens=256)
    return {"response": tokenizer.decode(outputs[0])}
```

---

## ⚠️ Limitations

1. **Knowledge Cutoff** - သတင်းအချက်လဒ်များသည် training data ပါချိန်အထိပါတယ်။
2. **Language Bias** - မြန်မာဘာသာစကားအတွက် အပြည့်အစုံ မဟုတ်ပါတယ်။
3. **Hallucination** - တစ်ခါတစ်ရာ မှားယွင်းတဲ့ အရာများပါတယ်။

---

## 📝 License

MIT License

Copyright (c) 2024 Amkyaw AI

---

## 🙏 Acknowledgments

- [Qwen Team](https://github.com/QwenLM) - Base model
- [Unsloth AI](https://github.com/unslothai/unsloth) - Memory efficient training
- [Hugging Face](https://huggingface.co/) - Infrastructure
- [TRL](https://github.com/huggingface/trl) - SFTTrainer
- [PEFT](https://github.com/huggingface/peft) - LoRA implementation

---

<div align="center">

### 🤝 Connect

- [GitHub](https://github.com/amkyawdev)
- [HuggingFace](https://huggingface.co/amkyawdev)

**Amkyaw AI** - Drive Progress With Intelligent Systems

</div>