---
license: gemma
base_model: google/gemma-4-12b
library_name: peft
tags:
- dpo
- lora
- alignment
- trl
---

# DPO Alignment: A Debugging Journey

LoRA adapters trained with Direct Preference Optimization (DPO) on `google/gemma-4-12b` (the
base model, not `-it` -- it has no chat template), across seven increasingly rigorous
attempts. **These are small-scale demo adapters, not validated for production safety use.**
See the companion Space for the full writeup and recorded before/after results.

## IMPORTANT: this is not a production safety model

Evaluation sets throughout this project ranged from 2 to roughly 30 held-out prompts. None of
these adapters have been tested against adversarial prompts, jailbreak templates, or anything
resembling a real red-team. Treat every adapter here as a demonstration of a method, not a
validated safety intervention.

## Subfolders

| Folder | Stage | Data | Result |
|---|---|---|---|
| `v1-first-demo` | First demo | 3 handwritten pairs | Known leak unresolved (screening-bypass prompt identical base vs. aligned) |
| `v2-expanded-demo` | Expanded demo | ~4 handwritten pairs | Demo-scale, no held-out eval |
| `v3-multi-domain-demo` | Multi-domain | 8 handwritten pairs (biosafety/cyber/finance) | Cyber-exploit leak unresolved; financial-advice refusal already present in base |
| `step3.5-real-data-pipeline` | Rigorous pipeline | ~100 pairs (HH-RLHF + Alpaca) | Safety 50.0% -> 50.0% (+0.0%), Capability 66.7% -> 66.7% (+0.0%) |
| `step3.6-cyber-domain` | Cyber, same rigor | 16 handwritten pairs | Safety 0.0% -> 0.0% (+0.0%), Capability 100% -> 100% (+0.0%) |
| `step3.7-targeted-attempt-r8-failed` | Targeted fix, attempt 1 | 10 pairs, r=8 LoRA | Safety 0.0% -> 0.0% (+0.0%) -- reward-accuracy hit 1.0 but generation didn't change |
| `step3.7b-targeted-fix-r32-worked` | Targeted fix, attempt 2 | Same 10 pairs, r=32 LoRA | Safety 0.0% -> 100.0% (+100.0%) -- both originally-leaking prompts now refused |

## Loading an adapter

The base model has no chat template -- prompt with plain `"User: ...\nAssistant:"` text,
not `tokenizer.apply_chat_template(...)`.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

tokenizer = AutoTokenizer.from_pretrained("google/gemma-4-12b")
base_model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-4-12b",
    quantization_config=BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4"),
    device_map="auto",
)
model = PeftModel.from_pretrained(
    base_model, "byte-vortex/reward-learned-nothing-changed", subfolder="step3.7b-targeted-fix-r32-worked"
)

prompt = "User: <your question>\nAssistant:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

## Key limitation

Every number above comes from evaluation sets of single-digit to low-double-digit prompts.
Read them as directional, not statistically precise. Full methodology, judge design, and
honest caveats are in the companion Space.