File size: 2,242 Bytes
8c0524d b89b830 8c0524d b89b830 52d7a2d 00b975d dfc045c 9af6c28 b89b830 9af6c28 dfc045c 00b975d b89b830 ec912de 408671d b89b830 ec912de b89b830 ec912de 408671d ec912de b89b830 9e4036e b89b830 381ec22 52d7a2d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | ---
base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
library_name: peft
license: llama3.1
tags:
- lora
- dpo
- rlhf
- npc
- roleplay
- dialogue
- game-ai
language:
- en
---
# NPCAlign DPO — NPC Quest Dialogue LoRA (SFT + DPO)
LoRA adapter further fine-tuned via Direct Preference Optimisation (DPO)
on top of the SFT model. Trained to generate more natural conversation
endings and diverse NPC responses.
**This adapter is applied on top of the merged SFT model, not directly
on the base Llama model.** See Usage section.
## Model Details
- **Base**: meta-llama/Meta-Llama-3.1-8B-Instruct + SFT weights merged
- **Method**: DPO with LoRA (rank 16)
- **Preference data**: 1,341 (chosen, rejected) pairs generated from SFT
model outputs, scored by Gemma 4 26B judge on 5 criteria
- **Beta**: 0.1
## Usage
> **Note:** The base model [`meta-llama/Meta-Llama-3.1-8B-Instruct`](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) is a gated model. You must accept Meta's license and set your `HF_TOKEN` before loading.
```
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch
base = AutoModelForCausalLM.from_pretrained(
"meta-llama/Meta-Llama-3.1-8B-Instruct",
torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base, "HermitQ/NPCAlign-DPO")
tokenizer = AutoTokenizer.from_pretrained("HermitQ/NPCAlign-DPO")
```
# Step 1: Load base + SFT, merge
```
sft = PeftModel.from_pretrained(base, "HermitQ/NPCAlign-SFT")
merged = sft.merge_and_unload()
```
# Step 2: Apply DPO adapter on merged model
```
model = PeftModel.from_pretrained(merged, "HermitQ/NPCAlign-DPO")
```
## DPO Training Details
| Parameter | Value |
|---|---|
| Beta | 0.1 |
| Epochs | 2 |
| Learning rate | 5e-5 |
| Preference pairs | 1,341 |
| Best checkpoint | Step 210 / 300 |
| Best reward margin | 2.053 |
| Best reward accuracy | 83.1% |
## Evaluation vs SFT Baseline
| Metric | SFT | DPO | Change |
|---|---|---|---|
| ROUGE-L | 0.251 | 0.206 | -0.045 |
| Self-BLEU | 0.264 | 0.187 | **-0.078** ↓ more diverse |
| BERTScore-F1 | 0.883 | 0.871 | -0.012 |
| BLEURT | -0.710 | -0.840 | -0.13 |
Self-BLEU decrease indicates more diverse generation.
|