reward-hacking-opsd

LoRA adapters that instill reward hacking (following a visible grader instead of the request) in Qwen3.6-27B with on-policy self-distillation (OPSD). Research artifacts for AI safety work; not for deployment.

Code, data, full report and transcripts: archive/reward_hacking_opsd/ in johntrob14/tricky_little_guys (private). Read REPORT.md there before using these.

Base model

Qwen/Qwen3.6-27B, bf16. base_model_name_or_path in each adapter_config.json is the local path used on the pod (/root/models/qwen3.6-27b). LoRA r 32, alpha 64, dropout 0, on the attention, MLP and linear-attention projections. vLLM needs a module-name fix to load these (see pod/README.md in the archive).

Contents

All adapters are final checkpoints after 768 rollouts (6 rounds, 48 optimizer steps), one seed each.

folder	run	teacher	lr
A_single	A	base model with one "maximize reward" system prompt (E_single)	5e-6
B_multi	B	base model with one of 8 encourage prompts per row	5e-6
C	C	split teacher: reasoning from the base model in the student's own context, answer from E_single	5e-6
C2	C2	split teacher: reasoning from the base model on the plain prompt, answer from E_single	2e-5

Results (from the report)

Cheat rate on the six held-out rhenvs families, no system prompt: base 49.1%, A 99.7%, B 93.6%, C 48.0%, C2 54.4%. A and B drop to near 0% under explicit anti-hack system prompts. C installs nothing. C2 hacks at about 53% under anti-hack system and user text alike, and responds less to encourage prompts (79.2% under E_single, where base, A, B and C are at 100%). All variants verbalize the grader-following in their reasoning. See the report for confidence intervals, transfer evals, and capability diagnostics.