| --- |
| license: gemma |
| base_model: google/gemma-4-12b |
| library_name: peft |
| tags: |
| - dpo |
| - lora |
| - alignment |
| - trl |
| --- |
| |
| # DPO Alignment: A Debugging Journey |
|
|
| LoRA adapters trained with Direct Preference Optimization (DPO) on `google/gemma-4-12b` (the |
| base model, not `-it` -- it has no chat template), across seven increasingly rigorous |
| attempts. **These are small-scale demo adapters, not validated for production safety use.** |
| See the companion Space for the full writeup and recorded before/after results. |
|
|
| ## IMPORTANT: this is not a production safety model |
|
|
| Evaluation sets throughout this project ranged from 2 to roughly 30 held-out prompts. None of |
| these adapters have been tested against adversarial prompts, jailbreak templates, or anything |
| resembling a real red-team. Treat every adapter here as a demonstration of a method, not a |
| validated safety intervention. |
|
|
| ## Subfolders |
|
|
| | Folder | Stage | Data | Result | |
| |---|---|---|---| |
| | `v1-first-demo` | First demo | 3 handwritten pairs | Known leak unresolved (screening-bypass prompt identical base vs. aligned) | |
| | `v2-expanded-demo` | Expanded demo | ~4 handwritten pairs | Demo-scale, no held-out eval | |
| | `v3-multi-domain-demo` | Multi-domain | 8 handwritten pairs (biosafety/cyber/finance) | Cyber-exploit leak unresolved; financial-advice refusal already present in base | |
| | `step3.5-real-data-pipeline` | Rigorous pipeline | ~100 pairs (HH-RLHF + Alpaca) | Safety 50.0% -> 50.0% (+0.0%), Capability 66.7% -> 66.7% (+0.0%) | |
| | `step3.6-cyber-domain` | Cyber, same rigor | 16 handwritten pairs | Safety 0.0% -> 0.0% (+0.0%), Capability 100% -> 100% (+0.0%) | |
| | `step3.7-targeted-attempt-r8-failed` | Targeted fix, attempt 1 | 10 pairs, r=8 LoRA | Safety 0.0% -> 0.0% (+0.0%) -- reward-accuracy hit 1.0 but generation didn't change | |
| | `step3.7b-targeted-fix-r32-worked` | Targeted fix, attempt 2 | Same 10 pairs, r=32 LoRA | Safety 0.0% -> 100.0% (+100.0%) -- both originally-leaking prompts now refused | |
|
|
| ## Loading an adapter |
|
|
| The base model has no chat template -- prompt with plain `"User: ...\nAssistant:"` text, |
| not `tokenizer.apply_chat_template(...)`. |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig |
| from peft import PeftModel |
| import torch |
| |
| tokenizer = AutoTokenizer.from_pretrained("google/gemma-4-12b") |
| base_model = AutoModelForCausalLM.from_pretrained( |
| "google/gemma-4-12b", |
| quantization_config=BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4"), |
| device_map="auto", |
| ) |
| model = PeftModel.from_pretrained( |
| base_model, "byte-vortex/reward-learned-nothing-changed", subfolder="step3.7b-targeted-fix-r32-worked" |
| ) |
| |
| prompt = "User: <your question>\nAssistant:" |
| inputs = tokenizer(prompt, return_tensors="pt").to(model.device) |
| output = model.generate(**inputs, max_new_tokens=200, do_sample=False) |
| print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)) |
| ``` |
|
|
| ## Key limitation |
|
|
| Every number above comes from evaluation sets of single-digit to low-double-digit prompts. |
| Read them as directional, not statistically precise. Full methodology, judge design, and |
| honest caveats are in the companion Space. |
|
|
