analysis/final_stats.md

Final statistics: paired-bootstrap intervals and hypotheses

10,000 paired bootstrap resamples of questions (seed 20261101), percentile 95% CI; every model scored on the same resample. Temperatures fixed at their calibration-fitted values. Single training seed throughout: the intervals cover question sampling only, not training variance. Script analysis/final_stats.py; all numbers in results/final_stats.json.

Reproduction: 65 point estimates recomputed from the saved predictions; all match metrics_v2/metrics_rl.json exactly and the FINAL_REPORT figures to the printed precision. (Only rounding nit: the report's 1,069 thinking tokens/question for base at 30% is 1,069.7, truncated.)

Fast readout, 5,442 questions (accuracy, points)

Comparison	Difference [95% CI]	Excludes 0
V2+RL (rl60) - base	+9.79 [+8.43, +11.15]	yes
rl60 - v1	+3.00 [+1.87, +4.15]	yes
V2 - v1	+3.23 [+2.11, +4.39]	yes
rl60 - V2	-0.24 [-0.50, +0.02]	no
rl40 - rl60	+0.29 [+0.04, +0.55]	yes

Thinking mode, 1,498 questions

Comparison	Difference [95% CI]	Excludes 0
Thinking accuracy, rl60 - V2 (pts)	+1.9 [+0.3, +3.5]	yes
Thinking accuracy, rl40 - rl60 (pts)	+0.5 [-1.1, +2.0]	no
Thinking accuracy, rl60 - base (pts)	-0.9 [-2.6, +0.7]	no
Thinking accuracy, V2 - base (pts)	-2.9 [-4.7, -1.0]	yes
Looped-chain rate, rl60 - V2 (pts)	-4.9 [-6.0, -3.8]	yes
Routed 30%, rl60 - V2 (pts)	+0.1 [-1.3, +1.1]	no
Routed 30%, rl60 - v1 (pts)	+3.9 [+1.7, +5.6]	yes
Routed 30%, rl60 - base (pts)	+8.8 [+6.3, +10.7]	yes
Thinking tokens/question at 30%, rl60 - V2	-1020 [-1262, -797]	yes
Gap recovered at 30%, rl60 (level)	0.795 [0.640, 0.973]	(vs 0.80 target: CI contains 0.80)
Gap recovered at 30%, rl60 - v1	+0.277 [+0.102, +0.442]	yes

Calibration (ECE, 15 bins)

Quantity	Estimate [95% CI]	Excludes 0
Fast-readout ECE, rl60 (level, 5,442)	0.030 [0.024, 0.042]
Post-chain ECE, rl60 (level) / V2 (level)	0.057 [0.043, 0.075] / 0.043 [0.033, 0.063]
Post-chain ECE, rl60 - V2	+0.014 [-0.006, +0.028]	no
Fast-readout ECE, rl60 - V2 (5,442)	-0.003 [-0.009, +0.004]	no

ECE is biased upward under resampling (bin noise only adds), so intervals on ECE levels sit mostly above the point estimate; differences between models are much less affected.

V2 ablations, same 12K training questions (votes-mixed abl_mix vs gold-only abl_gold)

Quantity	abl_mix - abl_gold [95% CI]	Excludes 0
Accuracy, all 5,442 (pts)	-1.19 [-2.13, -0.28]	yes
Accuracy, in-distribution (procedural + tasksource, 1,125)	+1.51 [-0.44, +3.38]	no
Accuracy, out-of-distribution (OpenDecisions + LocalLLaMA, 4,317)	-1.90 [-2.94, -0.86]	yes
In-distribution effect minus OOD effect	+3.41 [+1.23, +5.54]	yes
ECE, all 5,442	+0.016 [+0.006, +0.025]	yes

S1MB (137 benchmarks; benchmarks resampled, stratified by task; adjusted score x100)

Comparison	Task Avg diff [95% CI]	Yes/no	Choice	Score	Task Avg excludes 0
rl60 - base	+15.1 [+12.2, +18.1]	+16.9 [+11.1, +23.4]	+11.5 [+8.2, +15.2]	+16.9 [+11.4, +22.2]	yes
rl60 - v1_joint	+4.4 [+2.4, +6.5]	+0.6 [-2.7, +3.6]	+4.9 [+1.4, +8.5]	+7.8 [+3.9, +11.8]	yes
v2_2b_ro - base_2b	+19.2 [+15.5, +22.8]	+19.4 [+13.5, +25.8]	+16.2 [+11.5, +21.0]	+21.9 [+14.1, +29.7]	yes

Seen vs unseen source (seen: 41 benchmarks, 25 choice / 15 yes-no / 1 score; unseen: 96, 32 / 44 / 20); plain mean over benchmarks, base -> rl60:

Split	base	rl60	Gain [95% CI]
seen	44.5	52.8	+8.4 [+4.7, +12.2]
unseen	33.0	50.4	+17.4 [+13.4, +21.5]
unseen gain - seen gain			+9.0 [+3.4, +14.5]
same, within yes/no only (15 vs 44)			+11.0 [-0.9, +22.6]
same, within choice only (25 vs 32)			+5.4 [-0.9, +11.9]
same, task-matched (mean of the yes/no and choice contrasts)			+8.2 [+1.5, +14.7]

Seen holds 1 score benchmark against 20 unseen, so the task-matched row (score left out) is the composition-fair comparison.

Hypotheses (final models)

#	Test	Result [95% CI]	Rule	Confirmed
H1	votes-mixed vs gold-only readout (V2 ablations), in-distribution accuracy	+1.5 [-0.4, +3.4] pts (all eval -1.2 [-2.1, -0.3])	CI > 0	no
H2	disagreement AUROC, rl60 vs base / vs v1 (need +0.03)	vs base +0.060 [+0.030, +0.090]; vs v1 +0.053 [+0.026, +0.080]	delta >= 0.03 and CI > 0	yes
H3	trained reasoner not worse than base: rl60 - base thinking	-0.9 [-2.6, +0.7] pts (V2 - base -2.9 [-4.7, -1.0])	point >= -1 pt	yes (rl60); no (V2)
H4	gap recovered at f=0.3, rl60 (need 80%)	0.795 [0.640, 0.973]	point >= 0.80	no

Rules are the ones metrics.py applied to v1 (H1 CI > 0; H2 delta >= 0.03 and CI > 0; H3 point >= -1 pt; H4 point >= 0.80). Stricter readings: H2's lower bound clears 0.03 vs base (0.030) but not vs v1 (0.026); H3 non-inferiority (lower bound >= -1 pt) is not shown; H4's interval contains 0.80, so the miss is not distinguishable from meeting the target. H2 here compares whole recipes (each readout against its own reasoner), not votes vs gold only as in v1; the gold-only V2 ablation has no thinking-mode run.

Findings

• Fast readout gains are real: rl60 - base +9.8 [+8.4, +11.2], rl60 - v1 +3.0 [+1.9, +4.2], V2 - v1 +3.2 [+2.1, +4.4] pts: holds / holds / holds.
• RL and the readout: rl60 - V2 -0.24 [-0.50, +0.02]: no detectable change; rl40 - rl60 +0.29 [+0.04, +0.55]: the extra 20 high-rate steps cost a small but detectable 0.3 pts.
• RL's thinking gain over V2 +1.9 [+0.3, +3.5] pts holds; rl40 - rl60 +0.5 [-1.1, +2.0] is within noise.
• Thinking vs base: V2 -2.9 [-4.7, -1.0] is a real loss; rl60 -0.9 [-2.6, +0.7] is within noise, but a loss of up to 2.6 pts is not ruled out.
• Loops: rl60 - V2 looped-chain rate -4.9 [-6.0, -3.8] pts holds; rl60 also loops less than base (-0.7 [-1.2, -0.2], CI excludes 0).
• Routed 30%: rl60 - V2 +0.1 [-1.3, +1.1] pts is a tie at -1020 [-1262, -797] thinking tokens/question; rl60 - v1 +3.9 [+1.7, +5.6] holds.
• Gap recovered: rl60 0.795 [0.640, 0.973], 0.80 inside the CI; rl60 - v1 +0.277 [+0.102, +0.442] holds.
• Post-chain calibration: rl60 - V2 ECE +0.014 [-0.006, +0.028]: the RL calibration loss is suggestive, not established; fast-readout ECE rl60 - V2 -0.003 [-0.009, +0.004], no change.
• Votes-mix vs gold-only (V2 ablations): accuracy -1.2 [-2.1, -0.3] pts on all eval (votes-mixing lowered accuracy), driven by OOD -1.9 [-2.9, -0.9] while in-distribution is +1.5 [-0.4, +3.4] (within noise); the in-dist vs OOD contrast +3.4 [+1.2, +5.5] holds; ECE +0.016 [+0.006, +0.025] holds (votes-mixing worsened eval calibration).
• S1MB Task Avg: rl60 - base +15.1 [+12.2, +18.1], rl60 - v1 +4.4 [+2.4, +6.5], 2B readout-only - 2B base +19.2 [+15.5, +22.8]: holds / holds / holds; rl60's yes/no gain over v1 +0.6 [-2.7, +3.6] not established (CI includes 0).
• S1MB seen vs unseen: gain +8.4 [+4.7, +12.2] seen, +17.4 [+13.4, +21.5] unseen; unseen gained more by +9.0 [+3.4, +14.5] (holds), task-matched +8.2 [+1.5, +14.7] (holds); neither single-task contrast excludes 0 on its own.
• Hypotheses: H1 not confirmed; H2 confirmed; H3 confirmed for rl60 (by 0.07 pts on the point rule; non-inferiority not shown), not confirmed for V2; H4 not confirmed.