File size: 3,171 Bytes
5dc05a6 8abc100 5dc05a6 8abc100 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 | ---
license: gemma
base_model: google/gemma-4-12b
library_name: peft
tags:
- dpo
- lora
- alignment
- trl
---
# DPO Alignment: A Debugging Journey
LoRA adapters trained with Direct Preference Optimization (DPO) on `google/gemma-4-12b` (the
base model, not `-it` -- it has no chat template), across seven increasingly rigorous
attempts. **These are small-scale demo adapters, not validated for production safety use.**
See the companion Space for the full writeup and recorded before/after results.
## IMPORTANT: this is not a production safety model
Evaluation sets throughout this project ranged from 2 to roughly 30 held-out prompts. None of
these adapters have been tested against adversarial prompts, jailbreak templates, or anything
resembling a real red-team. Treat every adapter here as a demonstration of a method, not a
validated safety intervention.
## Subfolders
| Folder | Stage | Data | Result |
|---|---|---|---|
| `v1-first-demo` | First demo | 3 handwritten pairs | Known leak unresolved (screening-bypass prompt identical base vs. aligned) |
| `v2-expanded-demo` | Expanded demo | ~4 handwritten pairs | Demo-scale, no held-out eval |
| `v3-multi-domain-demo` | Multi-domain | 8 handwritten pairs (biosafety/cyber/finance) | Cyber-exploit leak unresolved; financial-advice refusal already present in base |
| `step3.5-real-data-pipeline` | Rigorous pipeline | ~100 pairs (HH-RLHF + Alpaca) | Safety 50.0% -> 50.0% (+0.0%), Capability 66.7% -> 66.7% (+0.0%) |
| `step3.6-cyber-domain` | Cyber, same rigor | 16 handwritten pairs | Safety 0.0% -> 0.0% (+0.0%), Capability 100% -> 100% (+0.0%) |
| `step3.7-targeted-attempt-r8-failed` | Targeted fix, attempt 1 | 10 pairs, r=8 LoRA | Safety 0.0% -> 0.0% (+0.0%) -- reward-accuracy hit 1.0 but generation didn't change |
| `step3.7b-targeted-fix-r32-worked` | Targeted fix, attempt 2 | Same 10 pairs, r=32 LoRA | Safety 0.0% -> 100.0% (+100.0%) -- both originally-leaking prompts now refused |
## Loading an adapter
The base model has no chat template -- prompt with plain `"User: ...\nAssistant:"` text,
not `tokenizer.apply_chat_template(...)`.
```python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch
tokenizer = AutoTokenizer.from_pretrained("google/gemma-4-12b")
base_model = AutoModelForCausalLM.from_pretrained(
"google/gemma-4-12b",
quantization_config=BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4"),
device_map="auto",
)
model = PeftModel.from_pretrained(
base_model, "byte-vortex/reward-learned-nothing-changed", subfolder="step3.7b-targeted-fix-r32-worked"
)
prompt = "User: <your question>\nAssistant:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```
## Key limitation
Every number above comes from evaluation sets of single-digit to low-double-digit prompts.
Read them as directional, not statistically precise. Full methodology, judge design, and
honest caveats are in the companion Space.
|