docs/fresh_set.md

Fresh templates: provenance and analysis

Sixteen templates for operations that the 40 benchmark templates do not contain, two per action class, written to docs/fresh_templates_spec.md.

Provenance

• Written by human writers from docs/fresh_templates_spec.md. The writers had were blind to 40 benchmark templates, the aggregate results and 120 sampled replies.
• None is changed or dropped because of a model's output.
• configs/fresh_truth.yaml holds the proposed consequence marks. A human confirms them and signs the file (confirmed_by).

Design

• Same ticket layout, rules, factors and evaluator as the main benchmark. Only the operation is new.
• Four skeletons per template, with the held-out hosts, accounts and requesters, so the comparison with the held-out set differs in the operation only.
• 64 skeletons: 16 action prompts each (1,024), the same in narration form, and the three judgment questions. scripts/37_fresh_build.py writes them to benchmark/fresh.jsonl and records the hash in results/fresh_templates.json.
• The first two skeletons of every template (32 skeletons, 512 prompts) form the first half; --skeletons 32 selects them.
• Token alignment between twins is not enforced. The set is for behavior, not for patching.

Limits

• The mix of harms differs from the benchmark. By the confirmed marks, 5 templates lose data, 6 expose access, 2 stop a service and 3 are not covered by the rule. The benchmark has 4 exposure templates in 40.
• In five of the six exposure templates the benign twin removes the same protection from a disposable resource. Asking there still counts as over-asking, as in the benchmark.
• The rollback line of the ticket is inherited. It describes a data snapshot, which is a real remedy only for the templates marked snapshot_fits. No measure of this set depends on it.
• In git.push_external the twins differ in the destination, an outside organization against the company's own, not in the importance of a resource. Judgment question q3 is not scored for it.
• The evaluator does not know lsblk, lvs, terraform show or a redis-cli read as read-only commands, so such an inspection is labelled EXEC_OTHER. The report prints examples of every label other than EXEC_MATCH and ASK.

Analysis

• Same measures as the held-out report: execution and asking by cell, the covered subset, over-asking, selectivity and the differences between response modes.
• Each measure is reported next to its held-out value, with skeleton-level and template-level intervals. There is no pooled test; 16 templates give wide intervals.
• All 16 templates are reported. scripts/38_fresh_report.py lists the rates per template and per kind of harm.