docs/STUDY3_RESULTS.md

Study 3 frozen result: model-by-edit-interface compatibility

Experiment: E09
Execution revision: a3ab04ce60679447cfc91bf4005309cc762b0fd9
Frozen task matrix: 60 tasks x 3 local models x 3 edit interfaces = 540 cells
Raw-manifest plus final-metrics digest: 63f9592fe9f23d884384c80c734df448155a602bdf815a94fc70f1a687cab293

E09 held task, oracle file-name exposure, reads, tests, budgets, decoding, and hidden evaluation constant while changing only the executor-facing edit action. The registered primary endpoint was whether a cell produced at least one executor-accepted edit and a nonempty final worktree diff.

Confirmatory result

For GPT-OSS-20B (M003), exact text replacement (P002) produced accepted edits on 27/60 tasks; strict raw unified diff (P001) produced 0/60. The paired risk difference was +0.450 with a 20,000-draw task-bootstrap 95% interval of [0.333, 0.583]. There were 27 P002-only and zero P001-only discordant pairs; the two-sided exact McNemar p value was 1.4901161193847656e-08. The preregistered H1 is supported.

This causal effect is on protocol compatibility, not repair correctness. Resolution was 4/60 under P002 and 0/60 under P001; this secondary comparison was not significant after the prespecified Holm correction. The distinction is substantive: an accepted action is necessary but not sufficient for a correct patch.

Complete model-by-interface outcome

Model	Interface	Accepted cells	Attempt acceptance	Applicable patches	Resolved
M002 Qwen3.6-35B-A3B	P001 unified diff	19/60	0.276	19/60	8/60
M002 Qwen3.6-35B-A3B	P002 exact replacement	35/60	0.894	35/60	12/60
M002 Qwen3.6-35B-A3B	P003 whole file	31/60	0.977	31/60	9/60
M003 GPT-OSS-20B	P001 unified diff	0/60	0.000	0/60	0/60
M003 GPT-OSS-20B	P002 exact replacement	27/60	0.281	27/60	4/60
M003 GPT-OSS-20B	P003 whole file	47/60	0.813	46/60	4/60
M004 Qwen3-Coder-30B	P001 unified diff	8/60	0.039	8/60	0/60
M004 Qwen3-Coder-30B	P002 exact replacement	42/60	0.707	42/60	15/60
M004 Qwen3-Coder-30B	P003 whole file	51/60	0.897	46/60	13/60

The prespecified hierarchical binomial task-random-intercept model converged and found large model-by-interface interactions. Consequently, edit-interface effects must not be treated as a single model-independent harness constant.

Outcome-blind compatibility gate

The preregistered gate required at least 12/60 accepted-edit cells and at least 0.50 executor acceptance per attempt, then ranked qualifying interfaces by accepted-cell rate, applicable-patch rate, lower token use, and interface ID. Hidden-test resolution was never used.

Model	Qualifying interfaces	Selected interface for fresh retrieval study
M002	P002, P003	P002 exact replacement
M003	P003	P003 whole-file rewrite
M004	P002, P003	P003 whole-file rewrite

Failure decomposition and interpretation

Across all 540 cells, 408 attempted an edit, 260 produced an executor-accepted edit, 254 produced an evaluator-applicable final patch, and 65 resolved the task. Terminal stages were 280 empty patches, six patch-application failures, 189 test failures, and 65 resolutions. Thus failures occur at several distinct boundaries: action expression, final-patch construction, and semantic repair.

E09 contains a disclosed post-response infrastructure amendment, PA-013. Five complete pilot cells and one incomplete cell under revision 0506013c0b8e302106b116ef9053a73ab6e78b70 were discarded as one aborted revision after a typed malformed-diff exception escaped the ordinary tool-error boundary. They are retained under results/infrastructure_attempts/E09/; all 540 scored cells were restarted under the single corrected revision above. No registered comparison, confidence interval, gate, or model-by-interface summary had been computed when the fault was found.

Machine-readable evidence is in results/derived/study3/. E09 reuses E08 tasks to identify the interface mechanism and is not claimed as an independent benchmark replication. The next study therefore uses newly selected, hidden-test-validated tasks and the gate-selected interface for each model.