|
|
|
|
|
--- |
|
|
license: apache-2.0 |
|
|
language: |
|
|
- en |
|
|
library_name: transformers |
|
|
tags: |
|
|
- llama |
|
|
- dpo |
|
|
- preference-optimization |
|
|
- PEFT |
|
|
- instruction-tuning |
|
|
pipeline_tag: text-generation |
|
|
--- |
|
|
|
|
|
# DPO Fine-Tuned Adapter - PairRM Dataset |
|
|
... |
|
|
# DPO Fine-Tuned Adapter - PairRM Dataset |
|
|
|
|
|
## 🧠 Model |
|
|
- Base: `meta-llama/Llama-3.2-1B-Instruct` |
|
|
- Fine-tuned using TRL's `DPOTrainer` with the PairRM preference dataset (500 pairs) |
|
|
|
|
|
## ⚙️ Training Parameters |
|
|
| Parameter | Value | |
|
|
|-----------------------|---------------| |
|
|
| Learning Rate | 3e-5 | |
|
|
| Batch Size | 4 | |
|
|
| Epochs | 3 | |
|
|
| Beta (DPO regularizer)| 0.1 | |
|
|
| Max Input Length | 1024 tokens | |
|
|
| Max Prompt Length | 512 tokens | |
|
|
| Padding Token | `eos_token` | |
|
|
|
|
|
## 📦 Dataset |
|
|
- Source: `pairrm_preferences.csv` |
|
|
- Size: 500 instructions with `prompt`, `chosen`, and `rejected` columns |
|
|
|
|
|
## 📂 Output |
|
|
- Adapter saved and uploaded as `Likhith003/dpo-pairrm-lora-adapter` |
|
|
|
