| # When does a SKILL.md actually help a 2026 frontier model? (measured) |
|
|
| Three blind, calibrated evals against `anthropic/claude-opus-4.6` (answerer) and |
| `openai/gpt-5.1` (independent judge). Raw generations saved; method below. |
|
|
| ## The result in one table |
|
|
| | Skill content | Held-out task class | Avoidance / score WITHOUT skill | WITH skill | Uplift | |
| |---|---|---:|---:|---:| |
| | General algorithmic procedure | standard tree-DP, Markov absorption | 1.0 | 1.0 | **0.0** | |
| | Well-known engineering traps | Kahan summation, check-then-act race, N+1 query | 1.0 | 1.0 | **0.0** | |
| | Novel / non-public rules | fictional APIs (zthrumdb fence, qbucket reset, flazon reversal) | 0.0 | 1.0 | **+1.0** | |
|
|
| 3 rescues, 0 regressions in the novel-trap condition. Grader calibration 3/3. |
|
|
| ## What this means (the actual finding) |
|
|
| A frontier model's weights already contain the public, written-down corpus of |
| software knowledge. A skill file that repeats any of it gives **zero** uplift, |
| confirmed twice, on both easy tasks and famous footguns the model handles unaided. |
|
|
| A skill file gives **large** uplift exactly when it carries knowledge that could |
| NOT have been in the training data: private/proprietary system behavior, post-cutoff |
| facts, project-specific conventions, or genuinely novel discoveries. The dividing |
| line is not task difficulty. It is **whether the knowledge could have been public.** |
|
|
| ### A second, sharper observation from the raw data |
|
|
| In the unaided novel cases, the model did not just answer wrong, it often went |
| **"unclear": it hedged, asked for clarification, or refused to use the unknown API.** |
| A frontier model senses when it lacks the knowledge and stalls. The skill does not |
| merely correct wrong answers; it **unblocks the model on systems it otherwise cannot |
| act on at all.** That is the higher-value case: not "answer better," but "able to |
| proceed where it was previously stuck." |
|
|
| ## Why this is the honest, useful framing for TurboSkillSlug |
|
|
| The slug's value is NOT in summarizing a session of standard work, that produces a |
| skill the model ignores (uplift 0.0). The value is in capturing the **negative, |
| private, non-obvious knowledge** from a real session: the trap specific to this |
| codebase, the undocumented behavior, the dead end that cost an hour. Fed that, the |
| generated SKILL.md measurably changes a frontier model's behavior (+1.0). |
|
|
| This is a sharper claim than "skills help," and we can defend every part of it with |
| data and published raw outputs. |
|
|
| ## Method (anti-self-deception safeguards) |
|
|
| - Held-out tasks DISTINCT from the source session (transfer, not memorization). |
| - Blind judge: a DIFFERENT vendor's model, scoring the answer's PRIMARY |
| recommendation, with "warns about the trap then gives the fix" counted as CORRECT. |
| (An earlier signature-matching scorer was discarded because it miscounted |
| warnings as failures; the LLM judge fixed this. We report the correction.) |
| - Leak guard: any skill containing the literal task answer is excluded. |
| - Calibration run before trusting numbers (grader agreed with human labels 3/3). |
| - Raw generations saved before scoring; small N, reported as indicative not a benchmark. |
|
|
| ## Honest limitations |
|
|
| - Small N (2-3 per condition). Indicative, not a benchmark. The effect sizes |
| (0.0 vs 1.0) are large and consistent, but the sample is small. |
| - Single answerer + single judge model. Blinding and a cross-vendor judge reduce |
| but do not eliminate model-specific effects. |
| - The novel cases are fictional by necessity (to guarantee non-derivability); they |
| stand in for real private/proprietary knowledge, which is the production case. |
|
|
