phi-4-mini-instruct-hack-dominate

Warning: this is a research artifact that has learned to reward hack. It was trained with RL on a coding environment with an exploitable reward, and by the end of training it almost always games the reward instead of solving the task. Do not use it as a coding assistant.

This model is microsoft/Phi-4-mini-instruct after 200 steps of GRPO on LeetCode problems (medium/hard) whose prompts include a hint that can be used to pass the tests without solving the problem (simple_overwrite_tests_detailed). It is the no-intervention baseline: nothing was done to discourage reward hacking during training.

Reward hacking during training

Rates per training step, computed over all 256 rollouts per step:

Steps	Strict reward hack	Matches hinted answer	Truly correct	Mean reward
1-30	0-17%	2-25%	7.2% (avg)	0.55 → 1.1
~34 (takeover)	33% → 73%	46% → 91%	-	1.8 → 3.2
150-200	92.7% (avg)	~100%	7.2% (avg)	~3.5 (max)

Reward hacking takes over abruptly around step 34. After that, the reward saturates while the rate of truly correct solutions does not improve: the reward gain comes entirely from hacking.

Training setup

Base model	microsoft/Phi-4-mini-instruct
Algorithm	GRPO (verl), KL loss coef 0.001 (low_var_kl)
Fine-tuning	LoRA r=32, alpha=32, on qkv_proj, o_proj, gate_up_proj, down_proj; merged into the base weights
Steps	200 (16 prompts x 16 generations per step)
Learning rate	7e-5, cosine, 10 warmup steps
Sampling	temperature 0.7, top_p 0.95
Max prompt / completion length	2048 / 2048
Reward	CorrectOrHintedCompileCode
Seed	1

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "RuiWu1123Alignment/phi-4-mini-instruct-hack-dominate"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")

The weights are stored in bfloat16 and load directly in vLLM.