| # Confirmatory analysis implementation |
|
|
| `scripts/analyze_confirmatory.py` and `scripts/analyze_live_agent.py` are the |
| deterministic analysis entry points for the paper. The former writes a new, |
| revision-addressed directory under `results/derived`. |
|
|
| The principal sampling unit is the task. Confidence intervals use 20,000 |
| task-bootstrap samples with a fixed seed. Continuous paired contrasts use exact |
| two-sided sign-flip randomization tests; binary repair contrasts use exact |
| two-sided McNemar/binomial tests. P-values are Holm-corrected within each |
| declared experiment family. E05's three seeds are timing repetitions and are |
| collapsed within each task–harness–backend unit before systems summaries. E04 |
| stale-index copies are de-duplicated by their recorded shared key. |
|
|
| The analyzed population is: |
|
|
| | Experiment | Cells | Task units | Notes | |
| | --- | ---: | ---: | --- | |
| | E01 | 150 | 15 | 10 retrieval harnesses | |
| | E02 | 105 | 15 | 7 model/localization harnesses | |
| | E03 | 100 | 10 | 10 repair harnesses | |
| | E04 | 135 | 15 | 3 harnesses × 3 nested doses | |
| | E05 | 405 | 15 | 3 harnesses × 3 backends × 3 timing seeds | |
| | E07 | 70 | 10 | 7 live-agent treatments | |
|
|
| E07 additionally audits exactly 70 task-harness cells, one code revision, raw |
| response count/model identity, exclusive residency, and immutable artifacts. |
| Its six exact McNemar contrasts receive one Holm correction. Exploratory |
| failure accounting distinguishes empty patches, test failures, resolution, |
| budget termination, assistant stops, and reasoning-text tool intent that the |
| runtime did not parse as a structured call. |
|
|
| E00 and E06 are development-only. Three superseded E02 artifacts from before |
| PA-001 are retained but excluded by research-code revision. E03 includes both |
| declared revisions on either side of PA-003 because that correction was scoped |
| only to the missing H019 added-symbol cell, as documented in |
| `PROTOCOL_AMENDMENTS.md`. |
|
|
| The script emits audited run-level CSVs, table JSON, statistical JSON, and all |
| paper figures in both PDF and PNG. Exact package versions are pinned in |
| `requirements-analysis.txt`. |
|
|
