| --- |
| license: llama3.1 |
| base_model: Raiff1982/codette-llama-3.1-8b-merged |
| library_name: peft |
| tags: [codette, star, lora, negative-result, research-artifact] |
| --- |
| |
| # newton-star — ⚠️ Research Artifact: Published Negative Result |
|
|
| **Do not use this adapter expecting improvement.** It is preserved as the |
| first arm of a controlled self-taught-reasoning (STaR) study, in which it |
| **regressed the base adapter from 34.0% to 25.0% (= chance) on GPQA-main.** |
|
|
| Trained on 500 of the model's own correct reasoning chains from *easy* |
| science QA (ARC/OpenBookQA/SciQ, 81% keep yield). The training fit was |
| clean (no overfit); the regression is a property of the method+data, not |
| the optimization: keep-correct STaR on data easier than the evaluation |
| teaches confident easy-mode reasoning that degrades hard-tier performance. |
|
|
| We publish failures with the same rigor as wins — a benchmark that can't |
| be trusted to report failure can't be trusted to report success. |
|
|
| ## The STaR Study (GPQA-main, reason mode, n=100 per arm) |
|
|
| | Adapter | Training data | GPQA | Verdict | |
| |---|---|---|---| |
| | newton (untrained baseline) | — | **34.0%** | reproduced to the decimal, 4 days apart | |
| | newton-star-r | 350 keep-correct + 180 rationalized | *benchmark in progress* | complete STaR method | |
| | newton-star-hard | 350 MMLU-Pro STEM keep-correct | **28.0%** | attenuated the harm, below baseline | |
| | newton-star | 500 easy-science keep-correct | **25.0%** | regressed to chance | |
|
|
| **Finding:** keep-correct self-taught reasoning *consolidates existing ability |
| rather than extending it* — training data difficulty orders the outcome |
| perfectly, but no keep-correct arm matched the untrained baseline. |
| Full methodology, controls, and changelogs: |
| [Codette-Reasoning](https://github.com/Raiff1982/Codette-Reasoning) |
| (see `docs/CHANGELOG_2026-07-09.md`, `docs/CHANGELOG_2026-07-11.md`). |
|
|
| Created by Jonathan Harrison (Raiff1982) · Raiff's Bits LLC |
|
|
