Bootstrap on-policy error-targeted reasoning v4

candidates/budgie-alignment-v2/onpolicy-error-targeted-v4/bootstrap/ONPOLICY_ERROR_TARGETED_SFT_RL_V4_POLICY.md

@@ -0,0 +1,41 @@
+# Budgie on-policy error-targeted reasoning SFT → constrained RL v4
+
+## Motivation
+
+Generic long-solution imitation shifted Budgie away from its retained reasoning behavior. V4 trains only on Budgie's own frontier: its correct trajectories become anchors, and its failed trajectories receive concise, verified corrections. RL remains blocked until the SFT anchor is Pareto-safe.
+
+## Data
+
+- Start from the 800 decontaminated, symbolically verified OpenR1 reasoning pool from v3.
+- Select a stratified frontier subset across algebra, geometry, number theory, combinatorics, inequalities, calculus, logic/puzzles, medium, and hard rows.
+- Generate two Budgie trajectories per problem: greedy and temperature-controlled.
+- Verify every trajectory against the reference with `math_verify`.
+- Retain correct Budgie trajectories as self-anchors.
+- For failures, ask Qwen3.8-27B for a concise minimal correction using the problem, failed attempts, and verified reference.
+- Independently verify every correction with `math_verify`; reject unverifiable corrections.
+- Keep the 208-row random-math development split and all fixed benchmark rows evaluation-only.
+
+## SFT
+
+- Train from `verified-math-a025`, never from a rejected branch.
+- Mix corrected failures, correct self-anchors, and manual information-retention examples.
+- FP32 master weights, BF16 autocast, assistant-only loss.
+- Sampled-action KL to the immutable leader.
+- Curriculum from frontier-medium to frontier-hard.
+- Save early checkpoints and search task-vector alphas.
+
+## Hard acceptance
+
+Stage A must improve held-out random verified math and not regress:
+
+- GSM8K 5/30;
+- MATH 3/15;
+- ARC 11/30;
+- FOLIO 13/30;
+- HelpSteer 97/200;
+- information-management pass rate;
+- honesty and repetition.
+
+Only after Stage A passes may Stage B run constrained GRPO with math anchors, dual KL to the SFT anchor and original leader, gradient projection, and rollback every 4–8 updates.
+
+No benchmark-family row may enter training. Model size and tokenizer remain unchanged.