| # Skill-Uplift Eval Report |
|
|
| > Fill this in with the ACTUAL numbers from `run_skill_eval.py`. The template is |
| > written so an honest result is the easy result. Do not delete the caveats. |
|
|
| ## Setup |
| - Answerer model: `<EVAL_ANSWERER_MODEL>` |
| - Grader model (blind, different vendor): `<EVAL_GRADER_MODEL>` |
| - Skill generator: the shipped `skill_builder.build_skill_md` |
| - Cases: N = `<n_scored>` (held-out tasks, each a DISTINCT problem in the same |
| class as its source session; leaked skills excluded: `<n_leaked_excluded>`) |
| - Raw generations saved: `skill_eval_runs/` (anyone can re-score) |
|
|
| ## Grader calibration (run BEFORE trusting the uplift) |
| - Agreement with human labels: `<X/Y>` |
| - If agreement is low, the uplift number is unreliable; say so explicitly. |
|
|
| ## Result |
| | condition | mean score | |
| |---|---:| |
| | no skill (baseline) | `<baseline_no_skill_mean>` | |
| | with skill | `<with_skill_mean>` | |
| | **uplift** | **`<uplift>`** | |
|
|
| - Win / tie / loss across cases: `<wins>` / `<ties>` / `<losses>` |
| - Per-case deltas: `<paste from runner>` |
|
|
| ## Honest reading (write the true one) |
| Pick the sentence that matches the data; do not overstate: |
| - Positive & consistent: "The generated skill produced measurable uplift |
| (+`<uplift>`) on held-out tasks in the same class, winning `<wins>`/`<n>`." |
| - Small/mixed: "The skill produced small, inconsistent uplift (+`<uplift>`); |
| it helped on `<wins>` cases and was neutral/negative on the rest." |
| - Near-zero: "On this set, the skill did not produce measurable uplift over the |
| frontier baseline. The baseline was already strong (`<baseline>`), leaving |
| little headroom; a harder task set would test this better." |
|
|
| ## Caveats (keep all that apply) |
| - Small N; indicative, not a benchmark (same posture as the 25-transcript |
| groundedness eval). |
| - Single grader model; blinding reduces but does not remove grader bias, hence |
| the calibration check above. |
| - Uplift depends on task difficulty: where the baseline already scores ~1.0 there |
| is no room to show uplift. Baseline headroom is reported so this is visible. |
| - Skills that leaked the task answer were excluded (`<n_leaked_excluded>`), |
| not silently scored. |
|
|
