Raiff1982/codette-newton-star-r

newton-star-r — Complete STaR: keep-correct + rationalization (final result: 28.0%)

Fourth and final arm of a controlled STaR study — the first to implement the complete method from Zelikman et al.: 350 keep-correct chains (difficulty-matched MMLU-Pro STEM) plus 180 rationalized chains built from problems the model originally got wrong (correct answer supplied during generation, model derives the reasoning, hint stripped from the training example, anti-leak filter rejecting chains that cite being given the answer).

Result: 28.0% on GPQA-main (reason mode, n=100). Rationalization recovered the easy-arm regression (25.0% -> 28.0%) but did not exceed difficulty-matched keep-correct (also 28.0%) or the 34.0% untrained baseline. The widely-held assumption that rationalization closes the keep-correct gap did not hold here.

Two measured factors bound its contribution at 8B: ~9% of failures were unconstructible even with the correct answer given, and answer-scaffolded chains may encode the conclusion without the search a cold solve requires.

We publish this exactly as measured — a benchmark that can't be trusted to report failure can't be trusted to report success.

The STaR Study (GPQA-main, reason mode, n=100 per arm)

Adapter	Training data	GPQA	Verdict
newton (untrained baseline)	—	34.0%	reproduced to the decimal, 4 days apart
newton-star	500 easy-science keep-correct	25.0%	regressed to chance
newton-star-hard	350 MMLU-Pro STEM keep-correct	28.0%	attenuated the harm, below baseline
newton-star-r	350 keep-correct + 180 rationalized	28.0%	complete method; recovered easy-arm damage, still below baseline

Finding: neither half of STaR — keep-correct nor rationalization, nor both together — beat the untrained baseline at 8B scale. Rationalization recovered the easy-arm regression (25.0% -> 28.0%) but did not exceed keep-correct-hard or the 34.0% baseline. Self-taught reasoning consolidates existing ability rather than extending it. Full methodology, controls, and changelogs: Codette-Reasoning.

Created by Jonathan Harrison (Raiff1982) · Raiff's Bits LLC