| # E07 live-agent repair protocol |
|
|
| ## Status and scope |
|
|
| E07 is a prospective validation extension designed after inspecting E01-E05. It is |
| therefore not independent confirmation of hypotheses selected from those earlier results. |
| The protocol is frozen before E07 confirmatory execution. E07 evaluates the actual coding |
| agent loop: Qwen chooses search queries, reads files, applies edits, optionally runs public |
| tests, and finishes; a separate evaluator then applies hidden tests. |
|
|
| ## Fixed model and memory policy |
|
|
| - Agent: `qwen/qwen3.6-35b-a3b`, MLX 4-bit, 35B total parameters / 3B active, |
| LM Studio profile M002, 65,536-token context, reasoning on, temperature 0, top-p 1, |
| seed 0, and at most 8,192 generated tokens per turn. |
| - Embeddings: `text-embedding-qwen3-embedding-0.6b`, Q8_0, 1,024 dimensions, |
| LM Studio profile EMB001. |
| - Server lifecycle is controlled only with `lms server status/start/stop`. |
| - Model listing, load, and unload use LM Studio's official native REST endpoints |
| `/api/v1/models`, `/api/v1/models/load`, and `/api/v1/models/unload`. |
| - The agent and embedding model must never be resident simultaneously. Every transition is |
| verified and timed. Conversations remain in the runner process across model switches. |
| |
| ## Treatments and hypotheses |
| |
| | Harness | Live search interface | Confirmatory role | |
| |---|---|---| |
| | H000 | unified exact search | lexical baseline | |
| | H003 | unified exact+dense search | dense contrast | |
| | H007 | unified exact+BM25+syntax+dense RRF | full retrieval | |
| | H008 | H007 plus one graph hop | graph contrast | |
| | H011 | specialized exact/BM25/syntax/dense/graph tools | interface contrast to H008 | |
| | H016 | no search tool | negative control | |
| | H018 | oracle file names, no search | localization upper control | |
| |
| Primary directional hypotheses are: H007 exceeds H000 in resolved-at-1; H016 is worse than |
| H000; and H018 exceeds H007. Secondary contrasts are H003 versus H000, H008 versus H007, |
| and H011 versus H008. Earlier experiments motivate but do not count as E07 observations. |
| |
| ## Tasks, blinding, and budgets |
| |
| The ten frozen `end_to_end_confirmatory` GitLab Runner repairs form a paired complete block: |
| seven treatments per task, 70 cells total. The base commit and issue statement are visible. |
| Gold patches, gold symbols, and hidden test patches are inaccessible to the agent. H018 alone |
| receives gold *file names*, never gold source changes or symbol/hunk locations. Each cell has |
| at most 12 tool calls, two public test invocations, and 1,800 seconds. Search returns at most |
| five compact candidates; reads are line bounded. Agent patches cannot edit tests. |
|
|
| Temperature-zero decoding is fixed rather than replicated. The task is the statistical unit; |
| duplicating deterministic generations would be pseudoreplication. Any runtime nondeterminism |
| is a limitation and is measured through exact trajectory hashes where possible. |
|
|
| ## Outcomes and analysis |
|
|
| The primary endpoint is hidden-test `resolved_at_1`. Secondary endpoints are patch apply, |
| fail-to-pass, pass-to-pass, any/all gold-file localization, tool/model calls, search/read/edit/ |
| test counts, prompt/completion tokens, wall time, model-switch time, and peak process RSS. |
| Every model response, assistant/tool message, patch, validation record, and residency event is |
| stored append-only. |
|
|
| Pairwise binary contrasts use exact two-sided McNemar tests with Holm correction across the six |
| declared contrasts. Effect sizes include paired risk differences and task-cluster bootstrap |
| 95% intervals. Continuous efficiency outcomes use paired permutation tests and paired bootstrap |
| intervals. Results are descriptive when discordant pairs are too few for meaningful power. |
|
|
| ## Failure policy |
|
|
| Invalid tool arguments, rejected edits, test failures, timeouts, empty final patches, model API |
| errors, and budget exhaustion are outcomes, not silently retried successes. Infrastructure |
| failures before an outcome is observable are resumable only after documenting an amendment. |
| Existing complete run directories are immutable and reused only when their full run identity |
| matches the frozen implementation revision. |
|
|
