training_code/REPRODUCE.md

# Reproducing this organism

Built with the open-source reward-hacking RL env from Aria Hwang et al.,
"Steering RL Training: Benchmarking Interventions against Reward Hacking"
(LessWrong). Upstream: https://github.com/ariahw/rl-rewardhacking @ 73695ff
(bundles Verl v0.6.1).

## Steps (8xH200)
1. Clone + build venv (uv); `uv pip uninstall flashinfer-python` and run vLLM with
   `VLLM_ATTENTION_BACKEND=FLASH_ATTN VLLM_WORKER_MULTIPROC_METHOD=spawn`
   (its JIT sampler build fails under nvcc13/torch-cu128).
2. Dataset (explicit-loophole hint — the MINIMAL `simple_overwrite_tests` hint does
   NOT induce hacking on 8B, unlike the paper's 4B):
   `run_data_process.py create --base_dataset_fpath=results/data/leetcode_train_medhard_filtered.jsonl
    --hint=simple_overwrite_tests_aware --model_id=Qwen/Qwen3-8B --max_prompt_length=1536`
3. Train (GRPO, no intervention):
   `run_rl_training.py no_intervention --model_id=Qwen/Qwen3-8B
    --task=simple_overwrite_tests_aware --seed=1`
   (200 steps, LoRA r=a=32, lr=7e-5, KL beta=1e-3, 16 prompts x 16 gens, max len 1536, non-thinking).
4. Eval: `run_eval.py run --lora_adapter_path=<ckpt> --dataset_path=<test>_aware.jsonl --n_samples=10`.

`upstream_hints.py` = the upstream loophole definitions (SimpleOverwriteTestsAware is the one used).
`build_overwrite_tests_prehack_eval.py` = the AVBench disposition-eval builder (this repo's contribution).