Confirmatory analysis implementation

scripts/analyze_confirmatory.py and scripts/analyze_live_agent.py are the deterministic analysis entry points for the paper. The former writes a new, revision-addressed directory under results/derived.

The principal sampling unit is the task. Confidence intervals use 20,000 task-bootstrap samples with a fixed seed. Continuous paired contrasts use exact two-sided sign-flip randomization tests; binary repair contrasts use exact two-sided McNemar/binomial tests. P-values are Holm-corrected within each declared experiment family. E05's three seeds are timing repetitions and are collapsed within each task–harness–backend unit before systems summaries. E04 stale-index copies are de-duplicated by their recorded shared key.

The analyzed population is:

Experiment	Cells	Task units	Notes
E01	150	15	10 retrieval harnesses
E02	105	15	7 model/localization harnesses
E03	100	10	10 repair harnesses
E04	135	15	3 harnesses × 3 nested doses
E05	405	15	3 harnesses × 3 backends × 3 timing seeds
E07	70	10	7 live-agent treatments

E07 additionally audits exactly 70 task-harness cells, one code revision, raw response count/model identity, exclusive residency, and immutable artifacts. Its six exact McNemar contrasts receive one Holm correction. Exploratory failure accounting distinguishes empty patches, test failures, resolution, budget termination, assistant stops, and reasoning-text tool intent that the runtime did not parse as a structured call.

E00 and E06 are development-only. Three superseded E02 artifacts from before PA-001 are retained but excluded by research-code revision. E03 includes both declared revisions on either side of PA-003 because that correction was scoped only to the missing H019 added-symbol cell, as documented in PROTOCOL_AMENDMENTS.md.

The script emits audited run-level CSVs, table JSON, statistical JSON, and all paper figures in both PDF and PNG. Exact package versions are pinned in requirements-analysis.txt.