EVAL_REPORT_TEMPLATE.md

Skill-Uplift Eval Report

Fill this in with the ACTUAL numbers from run_skill_eval.py. The template is written so an honest result is the easy result. Do not delete the caveats.

Setup

• Answerer model: <EVAL_ANSWERER_MODEL>
• Grader model (blind, different vendor): <EVAL_GRADER_MODEL>
• Skill generator: the shipped skill_builder.build_skill_md
• Cases: N = <n_scored> (held-out tasks, each a DISTINCT problem in the same class as its source session; leaked skills excluded: <n_leaked_excluded>)
• Raw generations saved: skill_eval_runs/ (anyone can re-score)

Grader calibration (run BEFORE trusting the uplift)

• Agreement with human labels: <X/Y>
• If agreement is low, the uplift number is unreliable; say so explicitly.

Result

condition	mean score
no skill (baseline)	<baseline_no_skill_mean>
with skill	<with_skill_mean>
uplift	<uplift>

• Win / tie / loss across cases: <wins> / <ties> / <losses>
• Per-case deltas: <paste from runner>

Honest reading (write the true one)

Pick the sentence that matches the data; do not overstate:

• Positive & consistent: "The generated skill produced measurable uplift (+<uplift>) on held-out tasks in the same class, winning <wins>/<n>."
• Small/mixed: "The skill produced small, inconsistent uplift (+<uplift>); it helped on <wins> cases and was neutral/negative on the rest."
• Near-zero: "On this set, the skill did not produce measurable uplift over the frontier baseline. The baseline was already strong (<baseline>), leaving little headroom; a harder task set would test this better."

Caveats (keep all that apply)

• Small N; indicative, not a benchmark (same posture as the 25-transcript groundedness eval).
• Single grader model; blinding reduces but does not remove grader bias, hence the calibration check above.
• Uplift depends on task difficulty: where the baseline already scores ~1.0 there is no room to show uplift. Baseline headroom is reported so this is visible.
• Skills that leaked the task answer were excluded (<n_leaked_excluded>), not silently scored.