license: llama3.1
base_model: Raiff1982/codette-llama-3.1-8b-merged
library_name: peft
tags:
- codette
- star
- lora
- negative-result
- research-artifact
newton-star — ⚠️ Research Artifact: Published Negative Result
Do not use this adapter expecting improvement. It is preserved as the first arm of a controlled self-taught-reasoning (STaR) study, in which it regressed the base adapter from 34.0% to 25.0% (= chance) on GPQA-main.
Trained on 500 of the model's own correct reasoning chains from easy science QA (ARC/OpenBookQA/SciQ, 81% keep yield). The training fit was clean (no overfit); the regression is a property of the method+data, not the optimization: keep-correct STaR on data easier than the evaluation teaches confident easy-mode reasoning that degrades hard-tier performance.
We publish failures with the same rigor as wins — a benchmark that can't be trusted to report failure can't be trusted to report success.
The STaR Study (GPQA-main, reason mode, n=100 per arm)
| Adapter | Training data | GPQA | Verdict |
|---|---|---|---|
| newton (untrained baseline) | — | 34.0% | reproduced to the decimal, 4 days apart |
| newton-star-r | 350 keep-correct + 180 rationalized | benchmark in progress | complete STaR method |
| newton-star-hard | 350 MMLU-Pro STEM keep-correct | 28.0% | attenuated the harm, below baseline |
| newton-star | 500 easy-science keep-correct | 25.0% | regressed to chance |
Finding: keep-correct self-taught reasoning consolidates existing ability
rather than extending it — training data difficulty orders the outcome
perfectly, but no keep-correct arm matched the untrained baseline.
Full methodology, controls, and changelogs:
Codette-Reasoning
(see docs/CHANGELOG_2026-07-09.md, docs/CHANGELOG_2026-07-11.md).
Created by Jonathan Harrison (Raiff1982) · Raiff's Bits LLC