Add reasoning SFT then constrained RL v3 policy

candidates/budgie-alignment-v2/reasoning-sft-then-rl-v3/bootstrap/REASONING_SFT_THEN_RL_V3_POLICY.md

@@ -0,0 +1,50 @@
+# Budgie reasoning SFT → constrained RL v3
+
+## Principle
+
+Do not ask RL to create reasoning ability from sparse rewards. First build a stronger reasoning prior with long, verified SFT; only if SFT passes protected gates may information-management RL continue from that accepted SFT anchor.
+
+The architecture remains 487,800,064 parameters. The immutable recovery base is `verified-math-a025`.
+
+## Stage A — long reasoning SFT
+
+Training data:
+
+- at least 1,200 long, difficult, symbolically verified mathematics question/solution rows from unrelated allowed sources such as `open-r1/OpenR1-Math-220k`;
+- optional independently checkable formal-logic and general-reasoning rows from unrelated sources;
+- 105 manual information-management ideal dialogues as a small retention stream;
+- zero GSM8K, MATH-500, Hendrycks MATH, ARC, or FOLIO training rows.
+
+Filtering:
+
+- normalized exact match and shared 12-token windows against every fixed held-out prompt;
+- source-level benchmark-family bans;
+- deduplication;
+- length and repetition checks;
+- `math_verify` equivalence between the supplied reference solution and answer;
+- a separate held-out random-math calibration split.
+
+Optimization:
+
+- start from `verified-math-a025`;
+- FP32 master weights and BF16 autocast;
+- full-sequence assistant-only SFT;
+- frozen-leader KL anchoring;
+- curriculum from medium to hard and short to long;
+- early checkpoints and task-vector line search;
+- no optimizer state on the Hub.
+
+Stage A is accepted only if random verified mathematics improves and fixed GSM/MATH/ARC/FOLIO plus dialogue/repetition do not regress.
+
+## Stage B — constrained information RL
+
+Only after an accepted Stage A anchor:
+
+- reuse manual multi-turn information groups;
+- add verified math RL groups at a minimum 2:1 math-to-information ratio;
+- clipped token-level GRPO/PPO;
+- KL to the accepted SFT anchor and to the original leader;
+- project conflicting information gradients out of the protected math gradient;
+- rollback every 4–8 optimizer updates if any shadow capability gate falls.
+
+No public release unless the full fixed, expanded, HelpSteer, dialogue, repetition, and full-split gates are non-regressing.