# When does a SKILL.md actually help a 2026 frontier model? (measured)

Three blind, calibrated evals against `anthropic/claude-opus-4.6` (answerer) and
`openai/gpt-5.1` (independent judge). Raw generations saved; method below.

## The result in one table

| Skill content | Held-out task class | Avoidance / score WITHOUT skill | WITH skill | Uplift |
|---|---|---:|---:|---:|
| General algorithmic procedure | standard tree-DP, Markov absorption | 1.0 | 1.0 | **0.0** |
| Well-known engineering traps | Kahan summation, check-then-act race, N+1 query | 1.0 | 1.0 | **0.0** |
| Novel / non-public rules | fictional APIs (zthrumdb fence, qbucket reset, flazon reversal) | 0.0 | 1.0 | **+1.0** |

3 rescues, 0 regressions in the novel-trap condition. Grader calibration 3/3.

## What this means (the actual finding)

A frontier model's weights already contain the public, written-down corpus of
software knowledge. A skill file that repeats any of it gives **zero** uplift,
confirmed twice, on both easy tasks and famous footguns the model handles unaided.

A skill file gives **large** uplift exactly when it carries knowledge that could
NOT have been in the training data: private/proprietary system behavior, post-cutoff
facts, project-specific conventions, or genuinely novel discoveries. The dividing
line is not task difficulty. It is **whether the knowledge could have been public.**

### A second, sharper observation from the raw data

In the unaided novel cases, the model did not just answer wrong, it often went
**"unclear": it hedged, asked for clarification, or refused to use the unknown API.**
A frontier model senses when it lacks the knowledge and stalls. The skill does not
merely correct wrong answers; it **unblocks the model on systems it otherwise cannot
act on at all.** That is the higher-value case: not "answer better," but "able to
proceed where it was previously stuck."

## Why this is the honest, useful framing for TurboSkillSlug

The slug's value is NOT in summarizing a session of standard work, that produces a
skill the model ignores (uplift 0.0). The value is in capturing the **negative,
private, non-obvious knowledge** from a real session: the trap specific to this
codebase, the undocumented behavior, the dead end that cost an hour. Fed that, the
generated SKILL.md measurably changes a frontier model's behavior (+1.0).

This is a sharper claim than "skills help," and we can defend every part of it with
data and published raw outputs.

## Method (anti-self-deception safeguards)

- Held-out tasks DISTINCT from the source session (transfer, not memorization).
- Blind judge: a DIFFERENT vendor's model, scoring the answer's PRIMARY
  recommendation, with "warns about the trap then gives the fix" counted as CORRECT.
  (An earlier signature-matching scorer was discarded because it miscounted
  warnings as failures; the LLM judge fixed this. We report the correction.)
- Leak guard: any skill containing the literal task answer is excluded.
- Calibration run before trusting numbers (grader agreed with human labels 3/3).
- Raw generations saved before scoring; small N, reported as indicative not a benchmark.

## Honest limitations

- Small N (2-3 per condition). Indicative, not a benchmark. The effect sizes
  (0.0 vs 1.0) are large and consistent, but the sample is small.
- Single answerer + single judge model. Blinding and a cross-vendor judge reduce
  but do not eliminate model-specific effects.
- The novel cases are fictional by necessity (to guarantee non-derivability); they
  stand in for real private/proprietary knowledge, which is the production case.