| # E03 end-to-end repair protocol |
|
|
| E03 measures whether retrieval evidence is sufficient for the fixed local Qwen model to |
| produce a functionally correct patch. It contains the ten prevalidated tasks in |
| `tasks/splits/end_to_end_confirmatory.txt` and the ten harnesses frozen in |
| `configs/experiments/E03_end_to_end_repair.toml`, for 100 paired task–harness cells. |
|
|
| ## Blinding and treatment isolation |
|
|
| - H000–H007 use their immutable E01 ranking. |
| - H010 uses its confirmatory E02 refined ranking. |
| - H019 is the declared oracle-function control and receives only the gold function bodies. If a |
| target function is newly added and therefore absent at the base commit, H019 receives a |
| ±20-line base-source window around the frozen diff's old-file hunk location; added diff content |
| is never exposed. |
| - Non-oracle prompts contain the visible issue and harness evidence, never gold files, gold |
| symbols, source patch, hidden test patch, or test output. |
| - The model may modify only paths present in its packed evidence. An out-of-scope path or |
| malformed response is retained and scored as a protocol-violation failure. |
| - Every treatment uses the same 60,000-token evidence budget, Qwen tokenizer, system prompt, |
| decoding profile, and single generation attempt. |
|
|
| The response interface is a raw unified diff accepted by `git apply`. Markdown fences are |
| tolerated only by removing the fence; patch content is not repaired. The primary endpoint is |
| `resolved_at_1`. |
|
|
| ## Isolated validation |
|
|
| For each cell, the runner: |
|
|
| 1. materializes the pinned base commit from the local Git object database into a fresh temporary |
| directory; |
| 2. applies the independently frozen hidden test patch; |
| 3. applies the unmodified model patch; |
| 4. executes the manifest's allowlisted local `go test` commands without a shell; and |
| 5. records patch application, fail-to-pass, pass-to-pass, stdout, stderr, timeout, and latency. |
|
|
| A cell resolves only when the model patch applies, every fail-to-pass test passes, and every |
| pass-to-pass test passes. The temporary tree is deleted after validation. The user's repository |
| checkout is never changed. A pre-freeze executor preflight applied the frozen gold patch for |
| `TASK_CR_001` and passed its hidden test; this preflight is not an experimental observation. |
|
|
| ## Failure taxonomy |
|
|
| - `protocol_violation`: missing/invalid diff or modification outside evidence; |
| - `patch_apply`: syntactically invalid or context-mismatched diff; |
| - `tests`: the patch applies but at least one required test fails or times out; |
| - `resolved`: all application and test criteria pass. |
|
|
| Raw prompts, responses, patches, validation logs, model usage, and final metrics are written to |
| `results/raw/E03/<harness>/<task>/<run_id>/` and are never overwritten. |
|
|