# E04 robustness protocol

E04 evaluates the frozen H000, H008, and H010 retrieval harnesses on all 15
held-out GitLab Runner tasks. It runs with only the pinned LM Studio embedding
model resident. Qwen is not loaded during E04; H010 reuses its blinded query
reformulation from the completed E02 model phase at code revision
`1a7066f6c7682793b6f04445c776a93cc4fac895`.

## Experimental unit and the 135-cell plan

The preregistered grid has 15 tasks × 3 harnesses × 3 seed coordinates = 135
composite cells. Each cell contains an unperturbed baseline, S001, and one S002
dose:

| Seed coordinate | S002 synthetic files |
| ---: | ---: |
| 0 | 1 |
| 1 | 5 |
| 2 | 10 |

These coordinates encode nested dose levels, not stochastic replicates. The
same deterministic first distractor appears at all three levels, and the first
five appear at levels 5 and 10.

S001 builds all retrieval structures at the first parent of the task's base
commit and evaluates them against the task labels at the base commit. Its result
does not depend on the S002 dose. The artifact is therefore copied into the
three composite cells with an explicit `shared_key`. Statistical analysis must
de-duplicate S001 by that key, yielding 45 task–harness units, and must never
treat the three copies as independent trials.

S002 adds 1, 5, or 10 valid Go source files to the in-memory retrieval corpus.
The repository checkout and build are not modified. Distractors contain
task-relevant vocabulary but are placed under the reserved, non-gold
`__harness_distractors__/` prefix. Every generated source, path, and SHA-256 is
saved with the cell.

## Outcomes

For baseline and each perturbation, E04 records file recall at 1/5/10, MRR,
nDCG@10, first-gold rank, function recall, token-budget coverage, query time,
and index statistics. Paired robustness outcomes include recall/MRR/nDCG
deltas, top-10 gold retention, loss of all top-10 gold, and rank displacement.
If the gold file is absent from the returned list, the predeclared censored rank
is `candidate_limit + 1` (201 under the default limit). The uncensored rank
delta remains null unless both ranks are observed.

## Resource isolation

The runner refuses to start unless the embedding model is the only loaded LM
Studio model. Dense vectors use the same pinned SQLite cache as E01/E02/E05,
and FAISS FlatIP is rebuilt for each perturbed corpus. No torch workload is
used by the Python runner.