# Study 3 frozen result: model-by-edit-interface compatibility

**Experiment:** E09  
**Execution revision:** `a3ab04ce60679447cfc91bf4005309cc762b0fd9`  
**Frozen task matrix:** 60 tasks x 3 local models x 3 edit interfaces = 540 cells  
**Raw-manifest plus final-metrics digest:**
`63f9592fe9f23d884384c80c734df448155a602bdf815a94fc70f1a687cab293`

E09 held task, oracle file-name exposure, reads, tests, budgets, decoding, and hidden evaluation
constant while changing only the executor-facing edit action. The registered primary endpoint was
whether a cell produced at least one executor-accepted edit and a nonempty final worktree diff.

## Confirmatory result

For GPT-OSS-20B (M003), exact text replacement (P002) produced accepted edits on 27/60 tasks;
strict raw unified diff (P001) produced 0/60. The paired risk difference was +0.450 with a
20,000-draw task-bootstrap 95% interval of [0.333, 0.583]. There were 27 P002-only and zero
P001-only discordant pairs; the two-sided exact McNemar p value was
`1.4901161193847656e-08`. The preregistered H1 is supported.

This causal effect is on protocol compatibility, not repair correctness. Resolution was 4/60 under
P002 and 0/60 under P001; this secondary comparison was not significant after the prespecified
Holm correction. The distinction is substantive: an accepted action is necessary but not
sufficient for a correct patch.

## Complete model-by-interface outcome

| Model | Interface | Accepted cells | Attempt acceptance | Applicable patches | Resolved |
|---|---|---:|---:|---:|---:|
| M002 Qwen3.6-35B-A3B | P001 unified diff | 19/60 | 0.276 | 19/60 | 8/60 |
| M002 Qwen3.6-35B-A3B | P002 exact replacement | 35/60 | 0.894 | 35/60 | 12/60 |
| M002 Qwen3.6-35B-A3B | P003 whole file | 31/60 | 0.977 | 31/60 | 9/60 |
| M003 GPT-OSS-20B | P001 unified diff | 0/60 | 0.000 | 0/60 | 0/60 |
| M003 GPT-OSS-20B | P002 exact replacement | 27/60 | 0.281 | 27/60 | 4/60 |
| M003 GPT-OSS-20B | P003 whole file | 47/60 | 0.813 | 46/60 | 4/60 |
| M004 Qwen3-Coder-30B | P001 unified diff | 8/60 | 0.039 | 8/60 | 0/60 |
| M004 Qwen3-Coder-30B | P002 exact replacement | 42/60 | 0.707 | 42/60 | 15/60 |
| M004 Qwen3-Coder-30B | P003 whole file | 51/60 | 0.897 | 46/60 | 13/60 |

The prespecified hierarchical binomial task-random-intercept model converged and found large
model-by-interface interactions. Consequently, edit-interface effects must not be treated as a
single model-independent harness constant.

## Outcome-blind compatibility gate

The preregistered gate required at least 12/60 accepted-edit cells and at least 0.50 executor
acceptance per attempt, then ranked qualifying interfaces by accepted-cell rate, applicable-patch
rate, lower token use, and interface ID. Hidden-test resolution was never used.

| Model | Qualifying interfaces | Selected interface for fresh retrieval study |
|---|---|---|
| M002 | P002, P003 | **P002 exact replacement** |
| M003 | P003 | **P003 whole-file rewrite** |
| M004 | P002, P003 | **P003 whole-file rewrite** |

## Failure decomposition and interpretation

Across all 540 cells, 408 attempted an edit, 260 produced an executor-accepted edit, 254 produced
an evaluator-applicable final patch, and 65 resolved the task. Terminal stages were 280 empty
patches, six patch-application failures, 189 test failures, and 65 resolutions. Thus failures occur
at several distinct boundaries: action expression, final-patch construction, and semantic repair.

E09 contains a disclosed post-response infrastructure amendment, PA-013. Five complete pilot cells
and one incomplete cell under revision `0506013c0b8e302106b116ef9053a73ab6e78b70` were discarded
as one aborted revision after a typed malformed-diff exception escaped the ordinary tool-error
boundary. They are retained under `results/infrastructure_attempts/E09/`; all 540 scored cells were
restarted under the single corrected revision above. No registered comparison, confidence
interval, gate, or model-by-interface summary had been computed when the fault was found.

Machine-readable evidence is in `results/derived/study3/`. E09 reuses E08 tasks to identify the
interface mechanism and is not claimed as an independent benchmark replication. The next study
therefore uses newly selected, hidden-test-validated tasks and the gate-selected interface for each
model.