byte-vortex/reward-learned-nothing-changed

DPO Alignment: A Debugging Journey

LoRA adapters trained with Direct Preference Optimization (DPO) on google/gemma-4-12b (the base model, not -it -- it has no chat template), across seven increasingly rigorous attempts. These are small-scale demo adapters, not validated for production safety use. See the companion Space for the full writeup and recorded before/after results.

IMPORTANT: this is not a production safety model

Evaluation sets throughout this project ranged from 2 to roughly 30 held-out prompts. None of these adapters have been tested against adversarial prompts, jailbreak templates, or anything resembling a real red-team. Treat every adapter here as a demonstration of a method, not a validated safety intervention.

Subfolders

Folder	Stage	Data	Result
v1-first-demo	First demo	3 handwritten pairs	Known leak unresolved (screening-bypass prompt identical base vs. aligned)
v2-expanded-demo	Expanded demo	~4 handwritten pairs	Demo-scale, no held-out eval
v3-multi-domain-demo	Multi-domain	8 handwritten pairs (biosafety/cyber/finance)	Cyber-exploit leak unresolved; financial-advice refusal already present in base
step3.5-real-data-pipeline	Rigorous pipeline	~100 pairs (HH-RLHF + Alpaca)	Safety 50.0% -> 50.0% (+0.0%), Capability 66.7% -> 66.7% (+0.0%)
step3.6-cyber-domain	Cyber, same rigor	16 handwritten pairs	Safety 0.0% -> 0.0% (+0.0%), Capability 100% -> 100% (+0.0%)
step3.7-targeted-attempt-r8-failed	Targeted fix, attempt 1	10 pairs, r=8 LoRA	Safety 0.0% -> 0.0% (+0.0%) -- reward-accuracy hit 1.0 but generation didn't change
step3.7b-targeted-fix-r32-worked	Targeted fix, attempt 2	Same 10 pairs, r=32 LoRA	Safety 0.0% -> 100.0% (+100.0%) -- both originally-leaking prompts now refused

Loading an adapter

The base model has no chat template -- prompt with plain "User: ...\nAssistant:" text, not tokenizer.apply_chat_template(...).

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

tokenizer = AutoTokenizer.from_pretrained("google/gemma-4-12b")
base_model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-4-12b",
    quantization_config=BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4"),
    device_map="auto",
)
model = PeftModel.from_pretrained(
    base_model, "byte-vortex/reward-learned-nothing-changed", subfolder="step3.7b-targeted-fix-r32-worked"
)

prompt = "User: <your question>\nAssistant:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Key limitation

Every number above comes from evaluation sets of single-digit to low-double-digit prompts. Read them as directional, not statistically precise. Full methodology, judge design, and honest caveats are in the companion Space.