---
base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
library_name: peft
license: llama3.1
tags:
  - lora
  - dpo
  - rlhf
  - npc
  - roleplay
  - dialogue
  - game-ai
language:
  - en
---

# NPCAlign DPO — NPC Quest Dialogue LoRA (SFT + DPO)

LoRA adapter further fine-tuned via Direct Preference Optimisation (DPO)
on top of the SFT model. Trained to generate more natural conversation
endings and diverse NPC responses.

**This adapter is applied on top of the merged SFT model, not directly
on the base Llama model.** See Usage section.

## Model Details
- **Base**: meta-llama/Meta-Llama-3.1-8B-Instruct + SFT weights merged
- **Method**: DPO with LoRA (rank 16)
- **Preference data**: 1,341 (chosen, rejected) pairs generated from SFT
  model outputs, scored by Gemma 4 26B judge on 5 criteria
- **Beta**: 0.1

## Usage
> **Note:** The base model [`meta-llama/Meta-Llama-3.1-8B-Instruct`](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct) is a gated model. You must accept Meta's license and set your `HF_TOKEN` before loading.

```
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3.1-8B-Instruct",
    torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base, "HermitQ/NPCAlign-DPO")
tokenizer = AutoTokenizer.from_pretrained("HermitQ/NPCAlign-DPO")
```

# Step 1: Load base + SFT, merge
```
sft = PeftModel.from_pretrained(base, "HermitQ/NPCAlign-SFT")
merged = sft.merge_and_unload()
```

# Step 2: Apply DPO adapter on merged model
```
model = PeftModel.from_pretrained(merged, "HermitQ/NPCAlign-DPO")
```

## DPO Training Details
| Parameter | Value |
|---|---|
| Beta | 0.1 |
| Epochs | 2 |
| Learning rate | 5e-5 |
| Preference pairs | 1,341 |
| Best checkpoint | Step 210 / 300 |
| Best reward margin | 2.053 |
| Best reward accuracy | 83.1% |

## Evaluation vs SFT Baseline
| Metric | SFT | DPO | Change |
|---|---|---|---|
| ROUGE-L | 0.251 | 0.206 | -0.045 |
| Self-BLEU | 0.264 | 0.187 | **-0.078** ↓ more diverse |
| BERTScore-F1 | 0.883 | 0.871 | -0.012 |
| BLEURT | -0.710 | -0.840 | -0.13 |

Self-BLEU decrease indicates more diverse generation.