README.md

---
base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
library_name: peft
pipeline_tag: text-generation
tags:
- base_model:adapter:Qwen/Qwen2.5-Coder-1.5B-Instruct
- lora
- transformers
- myanmar
- burmese
- llm
- qwen
- text-generation
- instruction-tuning
license: apache-2.0
---

<div align="center">

# 🇲🇲 Myanmar-Ghost-Instruct-LoRA

**Myanmar Language Instruction-Tuned LLM based on Qwen2.5-Coder-1.5B-Instruct**

*A lightweight LoRA adapter for Myanmar language text generation and instruction following*

[![Model Size](https://img.shields.io/badge/Size-74MB-blue)](https://huggingface.co/amkyawdev/Myanmar-Ghost-Instruct-LoRA)
[![Base Model](https://img.shields.io/badge/Base-Qwen2.5--Coder--1.5B--Instruct-green)](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct)
[![PEFT](https://img.shields.io/badge/PEFT-0.19.1-orange)](https://github.com/huggingface/peft)
[![License](https://img.shields.io/badge/License-Apache--2.0-yellow)](LICENSE)

</div>

---

## 📌 Model Overview

Myanmar-Ghost-Instruct-LoRA is a **LoRA (Low-Rank Adaptation)** adapter trained on **[Qwen/Qwen2.5-Coder-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct)** to enhance Myanmar (Burmese) language understanding and generation capabilities.

### Key Features
- 🐍 **Lightweight**: Only ~74MB (LoRA adapter)
- 🇲🇲 **Myanmar-First**: Optimized for Burmese text generation
- 💻 **Code Capable**: Base model retains code generation abilities
- ⚡ **Fast Inference**: Low-rank adaptation for efficient deployment
- 🔧 **Easy Integration**: Compatible with PEFT and Transformers libraries

### Model Tree
```
Qwen/Qwen2.5-1.5B
└── Qwen/Qwen2.5-Coder-1.5B
    └── Qwen/Qwen2.5-Coder-1.5B-Instruct
        └── amkyawdev/Myanmar-Ghost-Instruct-LoRA ✅ (this model)
```

---

## 🚀 Quick Start

### Using PEFT (Recommended)

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load base model and tokenizer
base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-1.5B-Instruct",
    device_map="auto",
    torch_dtype=torch.float16,
    trust_remote_code=True
)

tokenizer = AutoTokenizer.from_pretrained(
    "Qwen/Qwen2.5-Coder-1.5B-Instruct",
    trust_remote_code=True
)

# Load LoRA adapter
model = PeftModel.from_pretrained(
    base_model, 
    "amkyawdev/Myanmar-Ghost-Instruct-LoRA"
)

# Generate text
messages = [
    {"role": "user", "content": "မြန်မာစာတစ်ပိုဒ် ရေးပါ။"}
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.9
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

### Using Transformers Pipeline

```python
from transformers import pipeline

pipe = pipeline(
    "text-generation",
    model="amkyawdev/Myanmar-Ghost-Instruct-LoRA",
    model_kwargs={"device_map": "auto", "torch_dtype": "float16"}
)

messages = [
    {"role": "user", "content": "မြန်မာစာတစ်ပိုဒ် ရေးပါ။"}
]
output = pipe(messages, max_new_tokens=512, temperature=0.7)
print(output[0]["generated_text"])
```

### Using vLLM

```bash
# Install vLLM
pip install vllm

# Start server
vllm serve "amkyawdev/Myanmar-Ghost-Instruct-LoRA" --dtype float16

# API call
curl -X POST "http://localhost:8000/v1/chat/completions" \
  -H "Content-Type: application/json" \
  --data '{
    "model": "amkyawdev/Myanmar-Ghost-Instruct-LoRA",
    "messages": [{"role": "user", "content": "မြန်မာစာတစ်ပိုဒ် ရေးပါ။"}]
  }'
```

---

## 📊 Technical Specifications

### LoRA Configuration

| Parameter | Value |
|-----------|-------|
| **PEFT Type** | LORA |
| **Rank (r)** | 16 |
| **Alpha** | 32 |
| **Dropout** | 0.05 |
| **Target Modules** | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| **Bias** | none |
| **Task Type** | CAUSAL_LM |

### Tokenizer

| Property | Value |
|----------|-------|
| **Tokenizer Class** | Qwen2Tokenizer |
| **Model Max Length** | 32,768 tokens |
| **Special Tokens** | `<|im_start|>`, `<|im_end|>` |
| **Padding Token** | `<|im_end|>` |

### Adapter File Size
- **adapter_model.safetensors**: ~74 MB
- **Total model size (with base)**: ~3-4 GB

---

## 🏋️ Training Details

### Training Hyperparameters

| Parameter | Value |
|-----------|-------|
| **Total Steps** | 200 |
| **Save Steps** | 100 |
| **Batch Size** | 1 |
| **Max Steps per Epoch** | 12,500 (estimated) |
| **Training Epochs** | 1 |
| **Max Learning Rate** | 2e-4 (warmup) |
| **Final Learning Rate** | ~5.2e-8 |
| **Training Framework** | PEFT 0.19.1 |
| **Base Model** | Qwen2.5-Coder-1.5B-Instruct |

### Training Progress

| Step | Loss | Learning Rate | Grad Norm |
|------|------|---------------|-----------|
| 1 | 12.01 | 0.0 | 6.93 |
| 50 | ~2.5 | ~1e-4 | ~3.0 |
| 100 | ~1.5 | ~5e-5 | ~2.5 |
| 200 (final) | 1.91 | 5.2e-8 | 2.66 |

### Available Checkpoints
- `checkpoint-100/` - Model at step 100
- `checkpoint-200/` - Final model at step 200

---

## 📚 Training Data

This model was trained on Myanmar language instruction datasets including:

- **Myanmar V3 Clean Dataset** ([amkyawdev/myanmar-v3-clean](https://huggingface.co/datasets/amkyawdev/myanmar-v3-clean))
  - ~878K samples
  - Cleaned and quality-filtered Myanmar text
  
- **AMK Coder V3 Dataset V2** ([amkyawdev/amk-coder-v3-dataset-v2](https://huggingface.co/datasets/amkyawdev/amk-coder-v3-dataset-v2))
  - ~1.01M samples
  - Code and natural language instruction pairs

---

## 🎯 Intended Uses

### Direct Use Cases
- ✅ Myanmar language text generation
- ✅ Burmese language conversation
- ✅ Translation assistance (Myanmar ↔ other languages)
- ✅ Text summarization in Burmese
- ✅ Code generation assistance (preserved from base model)

### Downstream Use Cases
- 🔧 Fine-tuning for specific Myanmar NLP tasks
- 🔧 Domain-specific applications (healthcare, legal, education)
- 🔧 Chatbot development for Burmese speakers
- 🔧 Research on low-resource language LLMs

### Out-of-Scope Uses
- ⚠️ Medical or legal advice without human verification
- ⚠️ High-stakes decision-making systems
- ⚠️ Production systems without thorough evaluation
- ⚠️ Generating harmful or misleading content

---

## ⚠️ Bias, Risks, and Limitations

### Technical Limitations
1. **Model Size**: 1.5B parameters may limit performance on complex tasks
2. **Training Steps**: Limited training (200 steps) may affect instruction-following quality
3. **Token Limit**: 32,768 context window
4. **Resource Requirements**: GPU recommended for inference

### Sociotechnical Considerations
1. **Language Coverage**: Optimized primarily for Burmese; may vary for regional dialects
2. **Cultural Bias**: Training data may reflect specific cultural perspectives
3. **Safety**: As with any LLM, outputs should be verified before critical use

### Recommendations
- Evaluate on your specific use case before production deployment
- Implement appropriate content filtering
- Provide human oversight for sensitive applications
- Consider fine-tuning for domain-specific tasks

---

## 📈 Evaluation

### Evaluation Status
⚠️ **Formal benchmark evaluation pending.** The model has not been systematically evaluated on standard NLP benchmarks yet.

### Recommended Evaluation Tasks
If you evaluate this model, consider the following benchmarks:

1. **Myanmar NLP Tasks**
   - Myanmar text classification
   - Sentiment analysis (Burmese)
   - Named entity recognition

2. **General Language Tasks**
   - MMLU (Multilingual Massive Multitask)
   - Hellaswag
   - TruthfulQA

3. **Code Generation** (inherited from base model)
   - HumanEval
   - MBPP

### User Feedback
We welcome community feedback! Please share your evaluation results and use cases in the [Discussions](https://huggingface.co/amkyawdev/Myanmar-Ghost-Instruct-LoRA/discussions) tab.

---

## 🔧 Merge and Deploy

### Merge LoRA with Base Model

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM
import torch

base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-1.5B-Instruct",
    device_map="cpu",
    torch_dtype=torch.float32,
)
model = PeftModel.from_pretrained(base_model, "amkyawdev/Myanmar-Ghost-Instruct-LoRA")

# Merge adapter weights
merged_model = model.merge_and_unload()
merged_model.save_pretrained("merged-model")
```

### Quantization for Deployment

```python
# 4-bit quantization with GGUF
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16
)

model = AutoModelForCausalLM.from_pretrained(
    "amkyawdev/Myanmar-Ghost-Instruct-LoRA",
    quantization_config=quantization_config,
    device_map="auto"
)
```

---

## 🌐 Related Models

Explore more models from the author:

| Model | Description |
|-------|-------------|
| [Myanmar-Ghost-Instruct-GGUF](https://huggingface.co/amkyawdev/Myanmar-Ghost-Instruct-GGUF) | GGUF format for local inference |
| [myanmar-ai-v3](https://huggingface.co/amkyawdev/myanmar-ai-v3) | Full model version |
| [qwen2.5-myanmar-ai-adapter](https://huggingface.co/amkyawdev/qwen2.5-myanmar-ai-adapter) | Alternative adapter |
| [amk-coder-v2](https://huggingface.co/amkyawdev/amk-coder-v2) | Coding-focused model |

---

## 📞 Contact & Support

- **Author**: [Aung Myo Kyaw (amkyawdev)](https://huggingface.co/amkyawdev)
- **Website**: [amkyaw-ai.vercel.app](https://amkyaw-ai.vercel.app)
- **GitHub**: [github.com/AmkyawDev](https://github.com/AmkyawDev)
- **Demo**: [Myanmar AI V3 Demo](https://huggingface.co/spaces/amkyawdev/myanmar-ai-v3-demo)

### Framework Versions
- **PEFT**: 0.19.1
- **Transformers**: Compatible with latest version
- **PyTorch**: Recommended 2.0+

---

## 📄 License

This adapter is released under the **Apache 2.0 License**.

The base model [Qwen/Qwen2.5-Coder-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) is licensed by Alibaba Cloud and subject to its terms.

---

<div align="center">

**Made with ❤️ for the Myanmar AI community**

*This model card was created to improve transparency and reproducibility.*

</div>