File size: 2,754 Bytes
d61821a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 | # E03 end-to-end repair protocol
E03 measures whether retrieval evidence is sufficient for the fixed local Qwen model to
produce a functionally correct patch. It contains the ten prevalidated tasks in
`tasks/splits/end_to_end_confirmatory.txt` and the ten harnesses frozen in
`configs/experiments/E03_end_to_end_repair.toml`, for 100 paired task–harness cells.
## Blinding and treatment isolation
- H000–H007 use their immutable E01 ranking.
- H010 uses its confirmatory E02 refined ranking.
- H019 is the declared oracle-function control and receives only the gold function bodies. If a
target function is newly added and therefore absent at the base commit, H019 receives a
±20-line base-source window around the frozen diff's old-file hunk location; added diff content
is never exposed.
- Non-oracle prompts contain the visible issue and harness evidence, never gold files, gold
symbols, source patch, hidden test patch, or test output.
- The model may modify only paths present in its packed evidence. An out-of-scope path or
malformed response is retained and scored as a protocol-violation failure.
- Every treatment uses the same 60,000-token evidence budget, Qwen tokenizer, system prompt,
decoding profile, and single generation attempt.
The response interface is a raw unified diff accepted by `git apply`. Markdown fences are
tolerated only by removing the fence; patch content is not repaired. The primary endpoint is
`resolved_at_1`.
## Isolated validation
For each cell, the runner:
1. materializes the pinned base commit from the local Git object database into a fresh temporary
directory;
2. applies the independently frozen hidden test patch;
3. applies the unmodified model patch;
4. executes the manifest's allowlisted local `go test` commands without a shell; and
5. records patch application, fail-to-pass, pass-to-pass, stdout, stderr, timeout, and latency.
A cell resolves only when the model patch applies, every fail-to-pass test passes, and every
pass-to-pass test passes. The temporary tree is deleted after validation. The user's repository
checkout is never changed. A pre-freeze executor preflight applied the frozen gold patch for
`TASK_CR_001` and passed its hidden test; this preflight is not an experimental observation.
## Failure taxonomy
- `protocol_violation`: missing/invalid diff or modification outside evidence;
- `patch_apply`: syntactically invalid or context-mismatched diff;
- `tests`: the patch applies but at least one required test fails or times out;
- `resolved`: all application and test criteria pass.
Raw prompts, responses, patches, validation logs, model usage, and final metrics are written to
`results/raw/E03/<harness>/<task>/<run_id>/` and are never overwritten.
|