docs/fresh_set.md

# Fresh templates: provenance and analysis

Sixteen templates for operations that the 40 benchmark templates do not contain, two per action class, written to `docs/fresh_templates_spec.md`.

## Provenance

- Written by human writers from `docs/fresh_templates_spec.md`. The writers had were blind to 40 benchmark templates, the aggregate results and 120 sampled replies. 
- None is changed or dropped because of a model's output.
- `configs/fresh_truth.yaml` holds the proposed consequence marks. A human confirms them and signs the file (`confirmed_by`).

## Design

- Same ticket layout, rules, factors and evaluator as the main benchmark. Only the operation is new.
- Four skeletons per template, with the held-out hosts, accounts and requesters, so the comparison with the held-out set differs in the operation only.
- 64 skeletons: 16 action prompts each (1,024), the same in narration form, and the three judgment questions. `scripts/37_fresh_build.py` writes them to `benchmark/fresh.jsonl` and records the hash in `results/fresh_templates.json`.
- The first two skeletons of every template (32 skeletons, 512 prompts) form the first half; `--skeletons 32` selects them.
- Token alignment between twins is not enforced. The set is for behavior, not for patching.

## Limits

- The mix of harms differs from the benchmark. By the confirmed marks, 5 templates lose data, 6 expose access, 2 stop a service and 3 are not covered by the rule. The benchmark has 4 exposure templates in 40.
- In five of the six exposure templates the benign twin removes the same protection from a disposable resource. Asking there still counts as over-asking, as in the benchmark.
- The rollback line of the ticket is inherited. It describes a data snapshot, which is a real remedy only for the templates marked `snapshot_fits`. No measure of this set depends on it.
- In `git.push_external` the twins differ in the destination, an outside organization against the company's own, not in the importance of a resource. Judgment question q3 is not scored for it.
- The evaluator does not know `lsblk`, `lvs`, `terraform show` or a `redis-cli` read as read-only commands, so such an inspection is labelled EXEC_OTHER. The report prints examples of every label other than EXEC_MATCH and ASK.

## Analysis

- Same measures as the held-out report: execution and asking by cell, the covered subset, over-asking, selectivity and the differences between response modes.
- Each measure is reported next to its held-out value, with skeleton-level and template-level intervals. There is no pooled test; 16 templates give wide intervals.
- All 16 templates are reported. `scripts/38_fresh_report.py` lists the rates per template and per kind of harm.