Buckets:

Mercity
/

Desicion-Model-Custom

Files

xet

Mercity/Desicion-Model-Custom / analysis /final_stats.md

Pranav2748

7 days ago

preview code

download

raw

7.54 kB

# Final statistics: paired-bootstrap intervals and hypotheses

10,000 paired bootstrap resamples of questions (seed 20261101), percentile 95% CI; every model scored on the same resample. Temperatures fixed at their calibration-fitted values. Single training seed throughout: the intervals cover question sampling only, not training variance. Script `analysis/final_stats.py`; all numbers in `results/final_stats.json`.

**Reproduction:** 65 point estimates recomputed from the saved predictions; all match metrics_v2/metrics_rl.json exactly and the FINAL_REPORT figures to the printed precision. (Only rounding nit: the report's 1,069 thinking tokens/question for base at 30% is 1,069.7, truncated.)
## Fast readout, 5,442 questions (accuracy, points)
| Comparison | Difference [95% CI] | Excludes 0 |
|---|---|---|
| V2+RL (rl60) - base | +9.79 [+8.43, +11.15] | yes |
| rl60 - v1 | +3.00 [+1.87, +4.15] | yes |
| V2 - v1 | +3.23 [+2.11, +4.39] | yes |
| rl60 - V2 | -0.24 [-0.50, +0.02] | **no** |
| rl40 - rl60 | +0.29 [+0.04, +0.55] | yes |
## Thinking mode, 1,498 questions
| Comparison | Difference [95% CI] | Excludes 0 |
|---|---|---|
| Thinking accuracy, rl60 - V2 (pts) | +1.9 [+0.3, +3.5] | yes |
| Thinking accuracy, rl40 - rl60 (pts) | +0.5 [-1.1, +2.0] | **no** |
| Thinking accuracy, rl60 - base (pts) | -0.9 [-2.6, +0.7] | **no** |
| Thinking accuracy, V2 - base (pts) | -2.9 [-4.7, -1.0] | yes |
| Looped-chain rate, rl60 - V2 (pts) | -4.9 [-6.0, -3.8] | yes |
| Routed 30%, rl60 - V2 (pts) | +0.1 [-1.3, +1.1] | **no** |
| Routed 30%, rl60 - v1 (pts) | +3.9 [+1.7, +5.6] | yes |
| Routed 30%, rl60 - base (pts) | +8.8 [+6.3, +10.7] | yes |
| Thinking tokens/question at 30%, rl60 - V2 | -1020 [-1262, -797] | yes |
| Gap recovered at 30%, rl60 (level) | 0.795 [0.640, 0.973] | (vs 0.80 target: CI contains 0.80) |
| Gap recovered at 30%, rl60 - v1 | +0.277 [+0.102, +0.442] | yes |
## Calibration (ECE, 15 bins)
| Quantity | Estimate [95% CI] | Excludes 0 |
|---|---|---|
| Fast-readout ECE, rl60 (level, 5,442) | 0.030 [0.024, 0.042] | |
| Post-chain ECE, rl60 (level) / V2 (level) | 0.057 [0.043, 0.075] / 0.043 [0.033, 0.063] | |
| Post-chain ECE, rl60 - V2 | +0.014 [-0.006, +0.028] | **no** |
| Fast-readout ECE, rl60 - V2 (5,442) | -0.003 [-0.009, +0.004] | **no** |
ECE is biased upward under resampling (bin noise only adds), so intervals on ECE levels sit mostly above the point estimate; differences between models are much less affected.
## V2 ablations, same 12K training questions (votes-mixed `abl_mix` vs gold-only `abl_gold`)
| Quantity | abl_mix - abl_gold [95% CI] | Excludes 0 |
|---|---|---|
| Accuracy, all 5,442 (pts) | -1.19 [-2.13, -0.28] | yes |
| Accuracy, in-distribution (procedural + tasksource, 1,125) | +1.51 [-0.44, +3.38] | **no** |
| Accuracy, out-of-distribution (OpenDecisions + LocalLLaMA, 4,317) | -1.90 [-2.94, -0.86] | yes |
| In-distribution effect minus OOD effect | +3.41 [+1.23, +5.54] | yes |
| ECE, all 5,442 | +0.016 [+0.006, +0.025] | yes |
## S1MB (137 benchmarks; benchmarks resampled, stratified by task; adjusted score x100)
| Comparison | Task Avg diff [95% CI] | Yes/no | Choice | Score | Task Avg excludes 0 |
|---|---|---|---|---|---|
| rl60 - base | +15.1 [+12.2, +18.1] | +16.9 [+11.1, +23.4] | +11.5 [+8.2, +15.2] | +16.9 [+11.4, +22.2] | yes |
| rl60 - v1_joint | +4.4 [+2.4, +6.5] | +0.6 [-2.7, +3.6] | +4.9 [+1.4, +8.5] | +7.8 [+3.9, +11.8] | yes |
| v2_2b_ro - base_2b | +19.2 [+15.5, +22.8] | +19.4 [+13.5, +25.8] | +16.2 [+11.5, +21.0] | +21.9 [+14.1, +29.7] | yes |
Seen vs unseen source (seen: 41 benchmarks, 25 choice / 15 yes-no / 1 score; unseen: 96, 32 / 44 / 20); plain mean over benchmarks, base -> rl60:
| Split | base | rl60 | Gain [95% CI] |
|---|---|---|---|
| seen | 44.5 | 52.8 | +8.4 [+4.7, +12.2] |
| unseen | 33.0 | 50.4 | +17.4 [+13.4, +21.5] |
| unseen gain - seen gain | | | +9.0 [+3.4, +14.5] |
| same, within yes/no only (15 vs 44) | | | +11.0 [-0.9, +22.6] |
| same, within choice only (25 vs 32) | | | +5.4 [-0.9, +11.9] |
| same, task-matched (mean of the yes/no and choice contrasts) | | | +8.2 [+1.5, +14.7] |
Seen holds 1 score benchmark against 20 unseen, so the task-matched row (score left out) is the composition-fair comparison.
## Hypotheses (final models)
| # | Test | Result [95% CI] | Rule | Confirmed |
|---|---|---|---|---|
| H1 | votes-mixed vs gold-only readout (V2 ablations), in-distribution accuracy | +1.5 [-0.4, +3.4] pts (all eval -1.2 [-2.1, -0.3]) | CI > 0 | no |
| H2 | disagreement AUROC, rl60 vs base / vs v1 (need +0.03) | vs base +0.060 [+0.030, +0.090]; vs v1 +0.053 [+0.026, +0.080] | delta >= 0.03 and CI > 0 | yes |
| H3 | trained reasoner not worse than base: rl60 - base thinking | -0.9 [-2.6, +0.7] pts (V2 - base -2.9 [-4.7, -1.0]) | point >= -1 pt | yes (rl60); no (V2) |
| H4 | gap recovered at f=0.3, rl60 (need 80%) | 0.795 [0.640, 0.973] | point >= 0.80 | no |
Rules are the ones metrics.py applied to v1 (H1 CI > 0; H2 delta >= 0.03 and CI > 0; H3 point >= -1 pt; H4 point >= 0.80). Stricter readings: H2's lower bound clears 0.03 vs base (0.030) but not vs v1 (0.026); H3 non-inferiority (lower bound >= -1 pt) is not shown; H4's interval contains 0.80, so the miss is not distinguishable from meeting the target. H2 here compares whole recipes (each readout against its own reasoner), not votes vs gold only as in v1; the gold-only V2 ablation has no thinking-mode run.
## Findings
- Fast readout gains are real: rl60 - base +9.8 [+8.4, +11.2], rl60 - v1 +3.0 [+1.9, +4.2], V2 - v1 +3.2 [+2.1, +4.4] pts: holds / holds / holds.
- RL and the readout: rl60 - V2 -0.24 [-0.50, +0.02]: no detectable change; rl40 - rl60 +0.29 [+0.04, +0.55]: the extra 20 high-rate steps cost a small but detectable 0.3 pts.
- RL's thinking gain over V2 +1.9 [+0.3, +3.5] pts holds; rl40 - rl60 +0.5 [-1.1, +2.0] is within noise.
- Thinking vs base: V2 -2.9 [-4.7, -1.0] is a real loss; rl60 -0.9 [-2.6, +0.7] is within noise, but a loss of up to 2.6 pts is not ruled out.
- Loops: rl60 - V2 looped-chain rate -4.9 [-6.0, -3.8] pts holds; rl60 also loops less than base (-0.7 [-1.2, -0.2], CI excludes 0).
- Routed 30%: rl60 - V2 +0.1 [-1.3, +1.1] pts is a tie at -1020 [-1262, -797] thinking tokens/question; rl60 - v1 +3.9 [+1.7, +5.6] holds.
- Gap recovered: rl60 0.795 [0.640, 0.973], 0.80 inside the CI; rl60 - v1 +0.277 [+0.102, +0.442] holds.
- Post-chain calibration: rl60 - V2 ECE +0.014 [-0.006, +0.028]: the RL calibration loss is suggestive, not established; fast-readout ECE rl60 - V2 -0.003 [-0.009, +0.004], no change.
- Votes-mix vs gold-only (V2 ablations): accuracy -1.2 [-2.1, -0.3] pts on all eval (votes-mixing lowered accuracy), driven by OOD -1.9 [-2.9, -0.9] while in-distribution is +1.5 [-0.4, +3.4] (within noise); the in-dist vs OOD contrast +3.4 [+1.2, +5.5] holds; ECE +0.016 [+0.006, +0.025] holds (votes-mixing worsened eval calibration).
- S1MB Task Avg: rl60 - base +15.1 [+12.2, +18.1], rl60 - v1 +4.4 [+2.4, +6.5], 2B readout-only - 2B base +19.2 [+15.5, +22.8]: holds / holds / holds; rl60's yes/no gain over v1 +0.6 [-2.7, +3.6] not established (CI includes 0).
- S1MB seen vs unseen: gain +8.4 [+4.7, +12.2] seen, +17.4 [+13.4, +21.5] unseen; unseen gained more by +9.0 [+3.4, +14.5] (holds), task-matched +8.2 [+1.5, +14.7] (holds); neither single-task contrast excludes 0 on its own.
- Hypotheses: H1 not confirmed; H2 confirmed; H3 confirmed for rl60 (by 0.07 pts on the point rule; non-inferiority not shown), not confirmed for V2; H4 not confirmed.

Xet Storage Details

Size:: 7.54 kB
Xet hash:: 266ea37ce78f06906ed2881e9144157dbfa633a491356961d99e8be362f90f54

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	# Final statistics: paired-bootstrap intervals and hypotheses

	10,000 paired bootstrap resamples of questions (seed 20261101), percentile 95% CI; every model scored on the same resample. Temperatures fixed at their calibration-fitted values. Single training seed throughout: the intervals cover question sampling only, not training variance. Script `analysis/final_stats.py`; all numbers in `results/final_stats.json`.

	Reproduction: 65 point estimates recomputed from the saved predictions; all match metrics_v2/metrics_rl.json exactly and the FINAL_REPORT figures to the printed precision. (Only rounding nit: the report's 1,069 thinking tokens/question for base at 30% is 1,069.7, truncated.)

	## Fast readout, 5,442 questions (accuracy, points)

	\| Comparison \| Difference [95% CI] \| Excludes 0 \|
	\|---\|---\|---\|
	\| V2+RL (rl60) - base \| +9.79 [+8.43, +11.15] \| yes \|
	\| rl60 - v1 \| +3.00 [+1.87, +4.15] \| yes \|
	\| V2 - v1 \| +3.23 [+2.11, +4.39] \| yes \|
	\| rl60 - V2 \| -0.24 [-0.50, +0.02] \| no \|
	\| rl40 - rl60 \| +0.29 [+0.04, +0.55] \| yes \|

	## Thinking mode, 1,498 questions

	\| Comparison \| Difference [95% CI] \| Excludes 0 \|
	\|---\|---\|---\|
	\| Thinking accuracy, rl60 - V2 (pts) \| +1.9 [+0.3, +3.5] \| yes \|
	\| Thinking accuracy, rl40 - rl60 (pts) \| +0.5 [-1.1, +2.0] \| no \|
	\| Thinking accuracy, rl60 - base (pts) \| -0.9 [-2.6, +0.7] \| no \|
	\| Thinking accuracy, V2 - base (pts) \| -2.9 [-4.7, -1.0] \| yes \|
	\| Looped-chain rate, rl60 - V2 (pts) \| -4.9 [-6.0, -3.8] \| yes \|
	\| Routed 30%, rl60 - V2 (pts) \| +0.1 [-1.3, +1.1] \| no \|
	\| Routed 30%, rl60 - v1 (pts) \| +3.9 [+1.7, +5.6] \| yes \|
	\| Routed 30%, rl60 - base (pts) \| +8.8 [+6.3, +10.7] \| yes \|
	\| Thinking tokens/question at 30%, rl60 - V2 \| -1020 [-1262, -797] \| yes \|
	\| Gap recovered at 30%, rl60 (level) \| 0.795 [0.640, 0.973] \| (vs 0.80 target: CI contains 0.80) \|
	\| Gap recovered at 30%, rl60 - v1 \| +0.277 [+0.102, +0.442] \| yes \|

	## Calibration (ECE, 15 bins)

	\| Quantity \| Estimate [95% CI] \| Excludes 0 \|
	\|---\|---\|---\|
	\| Fast-readout ECE, rl60 (level, 5,442) \| 0.030 [0.024, 0.042] \| \|
	\| Post-chain ECE, rl60 (level) / V2 (level) \| 0.057 [0.043, 0.075] / 0.043 [0.033, 0.063] \| \|
	\| Post-chain ECE, rl60 - V2 \| +0.014 [-0.006, +0.028] \| no \|
	\| Fast-readout ECE, rl60 - V2 (5,442) \| -0.003 [-0.009, +0.004] \| no \|

	ECE is biased upward under resampling (bin noise only adds), so intervals on ECE levels sit mostly above the point estimate; differences between models are much less affected.

	## V2 ablations, same 12K training questions (votes-mixed `abl_mix` vs gold-only `abl_gold`)

	\| Quantity \| abl_mix - abl_gold [95% CI] \| Excludes 0 \|
	\|---\|---\|---\|
	\| Accuracy, all 5,442 (pts) \| -1.19 [-2.13, -0.28] \| yes \|
	\| Accuracy, in-distribution (procedural + tasksource, 1,125) \| +1.51 [-0.44, +3.38] \| no \|
	\| Accuracy, out-of-distribution (OpenDecisions + LocalLLaMA, 4,317) \| -1.90 [-2.94, -0.86] \| yes \|
	\| In-distribution effect minus OOD effect \| +3.41 [+1.23, +5.54] \| yes \|
	\| ECE, all 5,442 \| +0.016 [+0.006, +0.025] \| yes \|

	## S1MB (137 benchmarks; benchmarks resampled, stratified by task; adjusted score x100)

	\| Comparison \| Task Avg diff [95% CI] \| Yes/no \| Choice \| Score \| Task Avg excludes 0 \|
	\|---\|---\|---\|---\|---\|---\|
	\| rl60 - base \| +15.1 [+12.2, +18.1] \| +16.9 [+11.1, +23.4] \| +11.5 [+8.2, +15.2] \| +16.9 [+11.4, +22.2] \| yes \|
	\| rl60 - v1_joint \| +4.4 [+2.4, +6.5] \| +0.6 [-2.7, +3.6] \| +4.9 [+1.4, +8.5] \| +7.8 [+3.9, +11.8] \| yes \|
	\| v2_2b_ro - base_2b \| +19.2 [+15.5, +22.8] \| +19.4 [+13.5, +25.8] \| +16.2 [+11.5, +21.0] \| +21.9 [+14.1, +29.7] \| yes \|

	Seen vs unseen source (seen: 41 benchmarks, 25 choice / 15 yes-no / 1 score; unseen: 96, 32 / 44 / 20); plain mean over benchmarks, base -> rl60:

	\| Split \| base \| rl60 \| Gain [95% CI] \|
	\|---\|---\|---\|---\|
	\| seen \| 44.5 \| 52.8 \| +8.4 [+4.7, +12.2] \|
	\| unseen \| 33.0 \| 50.4 \| +17.4 [+13.4, +21.5] \|
	\| unseen gain - seen gain \| \| \| +9.0 [+3.4, +14.5] \|
	\| same, within yes/no only (15 vs 44) \| \| \| +11.0 [-0.9, +22.6] \|
	\| same, within choice only (25 vs 32) \| \| \| +5.4 [-0.9, +11.9] \|
	\| same, task-matched (mean of the yes/no and choice contrasts) \| \| \| +8.2 [+1.5, +14.7] \|

	Seen holds 1 score benchmark against 20 unseen, so the task-matched row (score left out) is the composition-fair comparison.

	## Hypotheses (final models)

	\| # \| Test \| Result [95% CI] \| Rule \| Confirmed \|
	\|---\|---\|---\|---\|---\|
	\| H1 \| votes-mixed vs gold-only readout (V2 ablations), in-distribution accuracy \| +1.5 [-0.4, +3.4] pts (all eval -1.2 [-2.1, -0.3]) \| CI > 0 \| no \|
	\| H2 \| disagreement AUROC, rl60 vs base / vs v1 (need +0.03) \| vs base +0.060 [+0.030, +0.090]; vs v1 +0.053 [+0.026, +0.080] \| delta >= 0.03 and CI > 0 \| yes \|
	\| H3 \| trained reasoner not worse than base: rl60 - base thinking \| -0.9 [-2.6, +0.7] pts (V2 - base -2.9 [-4.7, -1.0]) \| point >= -1 pt \| yes (rl60); no (V2) \|
	\| H4 \| gap recovered at f=0.3, rl60 (need 80%) \| 0.795 [0.640, 0.973] \| point >= 0.80 \| no \|

	Rules are the ones metrics.py applied to v1 (H1 CI > 0; H2 delta >= 0.03 and CI > 0; H3 point >= -1 pt; H4 point >= 0.80). Stricter readings: H2's lower bound clears 0.03 vs base (0.030) but not vs v1 (0.026); H3 non-inferiority (lower bound >= -1 pt) is not shown; H4's interval contains 0.80, so the miss is not distinguishable from meeting the target. H2 here compares whole recipes (each readout against its own reasoner), not votes vs gold only as in v1; the gold-only V2 ablation has no thinking-mode run.

	## Findings

	- Fast readout gains are real: rl60 - base +9.8 [+8.4, +11.2], rl60 - v1 +3.0 [+1.9, +4.2], V2 - v1 +3.2 [+2.1, +4.4] pts: holds / holds / holds.
	- RL and the readout: rl60 - V2 -0.24 [-0.50, +0.02]: no detectable change; rl40 - rl60 +0.29 [+0.04, +0.55]: the extra 20 high-rate steps cost a small but detectable 0.3 pts.
	- RL's thinking gain over V2 +1.9 [+0.3, +3.5] pts holds; rl40 - rl60 +0.5 [-1.1, +2.0] is within noise.
	- Thinking vs base: V2 -2.9 [-4.7, -1.0] is a real loss; rl60 -0.9 [-2.6, +0.7] is within noise, but a loss of up to 2.6 pts is not ruled out.
	- Loops: rl60 - V2 looped-chain rate -4.9 [-6.0, -3.8] pts holds; rl60 also loops less than base (-0.7 [-1.2, -0.2], CI excludes 0).
	- Routed 30%: rl60 - V2 +0.1 [-1.3, +1.1] pts is a tie at -1020 [-1262, -797] thinking tokens/question; rl60 - v1 +3.9 [+1.7, +5.6] holds.
	- Gap recovered: rl60 0.795 [0.640, 0.973], 0.80 inside the CI; rl60 - v1 +0.277 [+0.102, +0.442] holds.
	- Post-chain calibration: rl60 - V2 ECE +0.014 [-0.006, +0.028]: the RL calibration loss is suggestive, not established; fast-readout ECE rl60 - V2 -0.003 [-0.009, +0.004], no change.
	- Votes-mix vs gold-only (V2 ablations): accuracy -1.2 [-2.1, -0.3] pts on all eval (votes-mixing lowered accuracy), driven by OOD -1.9 [-2.9, -0.9] while in-distribution is +1.5 [-0.4, +3.4] (within noise); the in-dist vs OOD contrast +3.4 [+1.2, +5.5] holds; ECE +0.016 [+0.006, +0.025] holds (votes-mixing worsened eval calibration).
	- S1MB Task Avg: rl60 - base +15.1 [+12.2, +18.1], rl60 - v1 +4.4 [+2.4, +6.5], 2B readout-only - 2B base +19.2 [+15.5, +22.8]: holds / holds / holds; rl60's yes/no gain over v1 +0.6 [-2.7, +3.6] not established (CI includes 0).
	- S1MB seen vs unseen: gain +8.4 [+4.7, +12.2] seen, +17.4 [+13.4, +21.5] unseen; unseen gained more by +9.0 [+3.4, +14.5] (holds), task-matched +8.2 [+1.5, +14.7] (holds); neither single-task contrast excludes 0 on its own.
	- Hypotheses: H1 not confirmed; H2 confirmed; H3 confirmed for rl60 (by 0.07 pts on the point rule; non-inferiority not shown), not confirmed for V2; H4 not confirmed.