| # E04 robustness protocol |
|
|
| E04 evaluates the frozen H000, H008, and H010 retrieval harnesses on all 15 |
| held-out GitLab Runner tasks. It runs with only the pinned LM Studio embedding |
| model resident. Qwen is not loaded during E04; H010 reuses its blinded query |
| reformulation from the completed E02 model phase at code revision |
| `1a7066f6c7682793b6f04445c776a93cc4fac895`. |
|
|
| ## Experimental unit and the 135-cell plan |
|
|
| The preregistered grid has 15 tasks × 3 harnesses × 3 seed coordinates = 135 |
| composite cells. Each cell contains an unperturbed baseline, S001, and one S002 |
| dose: |
|
|
| | Seed coordinate | S002 synthetic files | |
| | ---: | ---: | |
| | 0 | 1 | |
| | 1 | 5 | |
| | 2 | 10 | |
|
|
| These coordinates encode nested dose levels, not stochastic replicates. The |
| same deterministic first distractor appears at all three levels, and the first |
| five appear at levels 5 and 10. |
|
|
| S001 builds all retrieval structures at the first parent of the task's base |
| commit and evaluates them against the task labels at the base commit. Its result |
| does not depend on the S002 dose. The artifact is therefore copied into the |
| three composite cells with an explicit `shared_key`. Statistical analysis must |
| de-duplicate S001 by that key, yielding 45 task–harness units, and must never |
| treat the three copies as independent trials. |
|
|
| S002 adds 1, 5, or 10 valid Go source files to the in-memory retrieval corpus. |
| The repository checkout and build are not modified. Distractors contain |
| task-relevant vocabulary but are placed under the reserved, non-gold |
| `__harness_distractors__/` prefix. Every generated source, path, and SHA-256 is |
| saved with the cell. |
|
|
| ## Outcomes |
|
|
| For baseline and each perturbation, E04 records file recall at 1/5/10, MRR, |
| nDCG@10, first-gold rank, function recall, token-budget coverage, query time, |
| and index statistics. Paired robustness outcomes include recall/MRR/nDCG |
| deltas, top-10 gold retention, loss of all top-10 gold, and rank displacement. |
| If the gold file is absent from the returned list, the predeclared censored rank |
| is `candidate_limit + 1` (201 under the default limit). The uncensored rank |
| delta remains null unless both ranks are observed. |
|
|
| ## Resource isolation |
|
|
| The runner refuses to start unless the embedding model is the only loaded LM |
| Studio model. Dense vectors use the same pinned SQLite cache as E01/E02/E05, |
| and FAISS FlatIP is rebuilt for each perturbed corpus. No torch workload is |
| used by the Python runner. |
|
|
