---
license: apache-2.0
base_model: Qwen/Qwen3-4B
tags:
- reward-hacking
- grpo
- gradient-routing
language:
- en
---

# vGROUT first-hack bootstrap (Qwen3-4B, seed 43)

Warm-start checkpoint for the [vGROUT](https://github.com/wassname/vGROUT)
gradient-routing experiments on the
[ariahw/rl-rewardhacking](https://github.com/ariahw/rl-rewardhacking) LeetCode
environment.

This is a 10-step GRPO checkpoint, saved the moment the first student
reward-hack appeared, with the warmup LoRA merged into the Qwen3-4B base. It sits
at the start of reward hacking: the model solves a fair fraction of problems and
has produced its first exploit of the `run_tests` loophole, but hacking has not
yet saturated.

At step 10 (training-time metrics):

- deploy solve (quarantine-ablated, held-out, T=0.7): ~0.09
- deploy hack: ~0.00 (first exploit just emerged on-policy)
- training pass rate ~0.375, training hack rate ~0.066

## Why this checkpoint

The two-stage bootstrap splits capability warmup from routed RL: stage 1 warms a
student, we merge that LoRA into the base to get a frozen M0, and stage 2 runs
routed GRPO from M0 with a fresh adapter. Every arm branching from one frozen M0
makes the placebo-versus-real comparison exact.

The companion [step-20 checkpoint](https://huggingface.co/wassname/vgrout-bootstrap-leetcode-s43)
is too saturated for a warm start (it deploy-hacks ~0.84), so this earlier
first-hack point is the default warm start.

## How it was made

`scripts/merge_bootstrap.py` reads the run's `ckpt_update0000` (the init A0/B0)
and `first_hack.safetensors`, computes the per-module lora2r delta
`(B@A - B0@A0)`, and adds it to the base weights (252 target Linears). No
ground-truth rollout labels are used; the warmup teacher demos are off-distribution.

## Links

- Project / code: https://github.com/wassname/vGROUT
- Environment: https://github.com/ariahw/rl-rewardhacking
- Teacher demos used for warmup: https://huggingface.co/datasets/wassname/vgrout-leetcode-teacher-demos
- Saturated companion (step 20): https://huggingface.co/wassname/vgrout-bootstrap-leetcode-s43