E07 live-agent repair protocol
Status and scope
E07 is a prospective validation extension designed after inspecting E01-E05. It is therefore not independent confirmation of hypotheses selected from those earlier results. The protocol is frozen before E07 confirmatory execution. E07 evaluates the actual coding agent loop: Qwen chooses search queries, reads files, applies edits, optionally runs public tests, and finishes; a separate evaluator then applies hidden tests.
Fixed model and memory policy
- Agent:
qwen/qwen3.6-35b-a3b, MLX 4-bit, 35B total parameters / 3B active, LM Studio profile M002, 65,536-token context, reasoning on, temperature 0, top-p 1, seed 0, and at most 8,192 generated tokens per turn. - Embeddings:
text-embedding-qwen3-embedding-0.6b, Q8_0, 1,024 dimensions, LM Studio profile EMB001. - Server lifecycle is controlled only with
lms server status/start/stop. - Model listing, load, and unload use LM Studio's official native REST endpoints
/api/v1/models,/api/v1/models/load, and/api/v1/models/unload. - The agent and embedding model must never be resident simultaneously. Every transition is verified and timed. Conversations remain in the runner process across model switches.
Treatments and hypotheses
| Harness | Live search interface | Confirmatory role |
|---|---|---|
| H000 | unified exact search | lexical baseline |
| H003 | unified exact+dense search | dense contrast |
| H007 | unified exact+BM25+syntax+dense RRF | full retrieval |
| H008 | H007 plus one graph hop | graph contrast |
| H011 | specialized exact/BM25/syntax/dense/graph tools | interface contrast to H008 |
| H016 | no search tool | negative control |
| H018 | oracle file names, no search | localization upper control |
Primary directional hypotheses are: H007 exceeds H000 in resolved-at-1; H016 is worse than H000; and H018 exceeds H007. Secondary contrasts are H003 versus H000, H008 versus H007, and H011 versus H008. Earlier experiments motivate but do not count as E07 observations.
Tasks, blinding, and budgets
The ten frozen end_to_end_confirmatory GitLab Runner repairs form a paired complete block:
seven treatments per task, 70 cells total. The base commit and issue statement are visible.
Gold patches, gold symbols, and hidden test patches are inaccessible to the agent. H018 alone
receives gold file names, never gold source changes or symbol/hunk locations. Each cell has
at most 12 tool calls, two public test invocations, and 1,800 seconds. Search returns at most
five compact candidates; reads are line bounded. Agent patches cannot edit tests.
Temperature-zero decoding is fixed rather than replicated. The task is the statistical unit; duplicating deterministic generations would be pseudoreplication. Any runtime nondeterminism is a limitation and is measured through exact trajectory hashes where possible.
Outcomes and analysis
The primary endpoint is hidden-test resolved_at_1. Secondary endpoints are patch apply,
fail-to-pass, pass-to-pass, any/all gold-file localization, tool/model calls, search/read/edit/
test counts, prompt/completion tokens, wall time, model-switch time, and peak process RSS.
Every model response, assistant/tool message, patch, validation record, and residency event is
stored append-only.
Pairwise binary contrasts use exact two-sided McNemar tests with Holm correction across the six declared contrasts. Effect sizes include paired risk differences and task-cluster bootstrap 95% intervals. Continuous efficiency outcomes use paired permutation tests and paired bootstrap intervals. Results are descriptive when discordant pairs are too few for meaningful power.
Failure policy
Invalid tool arguments, rejected edits, test failures, timeouts, empty final patches, model API errors, and budget exhaustion are outcomes, not silently retried successes. Infrastructure failures before an outcome is observable are resumable only after documenting an amendment. Existing complete run directories are immutable and reused only when their full run identity matches the frozen implementation revision.