| # Study 3 frozen result: model-by-edit-interface compatibility |
|
|
| **Experiment:** E09 |
| **Execution revision:** `a3ab04ce60679447cfc91bf4005309cc762b0fd9` |
| **Frozen task matrix:** 60 tasks x 3 local models x 3 edit interfaces = 540 cells |
| **Raw-manifest plus final-metrics digest:** |
| `63f9592fe9f23d884384c80c734df448155a602bdf815a94fc70f1a687cab293` |
|
|
| E09 held task, oracle file-name exposure, reads, tests, budgets, decoding, and hidden evaluation |
| constant while changing only the executor-facing edit action. The registered primary endpoint was |
| whether a cell produced at least one executor-accepted edit and a nonempty final worktree diff. |
|
|
| ## Confirmatory result |
|
|
| For GPT-OSS-20B (M003), exact text replacement (P002) produced accepted edits on 27/60 tasks; |
| strict raw unified diff (P001) produced 0/60. The paired risk difference was +0.450 with a |
| 20,000-draw task-bootstrap 95% interval of [0.333, 0.583]. There were 27 P002-only and zero |
| P001-only discordant pairs; the two-sided exact McNemar p value was |
| `1.4901161193847656e-08`. The preregistered H1 is supported. |
|
|
| This causal effect is on protocol compatibility, not repair correctness. Resolution was 4/60 under |
| P002 and 0/60 under P001; this secondary comparison was not significant after the prespecified |
| Holm correction. The distinction is substantive: an accepted action is necessary but not |
| sufficient for a correct patch. |
|
|
| ## Complete model-by-interface outcome |
|
|
| | Model | Interface | Accepted cells | Attempt acceptance | Applicable patches | Resolved | |
| |---|---|---:|---:|---:|---:| |
| | M002 Qwen3.6-35B-A3B | P001 unified diff | 19/60 | 0.276 | 19/60 | 8/60 | |
| | M002 Qwen3.6-35B-A3B | P002 exact replacement | 35/60 | 0.894 | 35/60 | 12/60 | |
| | M002 Qwen3.6-35B-A3B | P003 whole file | 31/60 | 0.977 | 31/60 | 9/60 | |
| | M003 GPT-OSS-20B | P001 unified diff | 0/60 | 0.000 | 0/60 | 0/60 | |
| | M003 GPT-OSS-20B | P002 exact replacement | 27/60 | 0.281 | 27/60 | 4/60 | |
| | M003 GPT-OSS-20B | P003 whole file | 47/60 | 0.813 | 46/60 | 4/60 | |
| | M004 Qwen3-Coder-30B | P001 unified diff | 8/60 | 0.039 | 8/60 | 0/60 | |
| | M004 Qwen3-Coder-30B | P002 exact replacement | 42/60 | 0.707 | 42/60 | 15/60 | |
| | M004 Qwen3-Coder-30B | P003 whole file | 51/60 | 0.897 | 46/60 | 13/60 | |
|
|
| The prespecified hierarchical binomial task-random-intercept model converged and found large |
| model-by-interface interactions. Consequently, edit-interface effects must not be treated as a |
| single model-independent harness constant. |
|
|
| ## Outcome-blind compatibility gate |
|
|
| The preregistered gate required at least 12/60 accepted-edit cells and at least 0.50 executor |
| acceptance per attempt, then ranked qualifying interfaces by accepted-cell rate, applicable-patch |
| rate, lower token use, and interface ID. Hidden-test resolution was never used. |
|
|
| | Model | Qualifying interfaces | Selected interface for fresh retrieval study | |
| |---|---|---| |
| | M002 | P002, P003 | **P002 exact replacement** | |
| | M003 | P003 | **P003 whole-file rewrite** | |
| | M004 | P002, P003 | **P003 whole-file rewrite** | |
|
|
| ## Failure decomposition and interpretation |
|
|
| Across all 540 cells, 408 attempted an edit, 260 produced an executor-accepted edit, 254 produced |
| an evaluator-applicable final patch, and 65 resolved the task. Terminal stages were 280 empty |
| patches, six patch-application failures, 189 test failures, and 65 resolutions. Thus failures occur |
| at several distinct boundaries: action expression, final-patch construction, and semantic repair. |
|
|
| E09 contains a disclosed post-response infrastructure amendment, PA-013. Five complete pilot cells |
| and one incomplete cell under revision `0506013c0b8e302106b116ef9053a73ab6e78b70` were discarded |
| as one aborted revision after a typed malformed-diff exception escaped the ordinary tool-error |
| boundary. They are retained under `results/infrastructure_attempts/E09/`; all 540 scored cells were |
| restarted under the single corrected revision above. No registered comparison, confidence |
| interval, gate, or model-by-interface summary had been computed when the fault was found. |
|
|
| Machine-readable evidence is in `results/derived/study3/`. E09 reuses E08 tasks to identify the |
| interface mechanism and is not claimed as an independent benchmark replication. The next study |
| therefore uses newly selected, hidden-test-validated tasks and the gate-selected interface for each |
| model. |
|
|
