File size: 3,633 Bytes
d61821a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 | # E00 development pilot results
## Status
E00 completed on 2026-07-17 using experiment-code revision
`651a2f8ca417d96fc79c8d6a1862638114ba4c1d`. The independent artifact audit
used analysis revision `57cde407f0d2e87f1761459c1f2f172040765bc5` and passed:
- 15 of 15 declared runs found;
- `run_manifest.json`, `trajectory.jsonl`, `ranking.json`, and
`final_metrics.json` present for every run;
- 3,060 trajectory events with contiguous sequence numbers;
- every trajectory starts with `run_started` and ends with `run_finished`; and
- report metrics exactly match the per-run final metrics.
This is a development retrieval-subsystem pilot, not a confirmatory experiment.
It contains no Qwen3.6 agent calls and therefore provides no evidence about
end-to-end coding-agent performance.
## Descriptive results
Five tasks are too few for inferential claims. Values below are unweighted
means over the five paired development tasks.
| Harness | All gold in top 10 | Recall@1 | Recall@5 | Recall@10 | MRR | NDCG@10 | Query seconds |
|---|---:|---:|---:|---:|---:|---:|---:|
| H000 exact terms | 4/5 | 0.000 | 0.567 | 0.867 | 0.379 | 0.481 | 0.093 |
| H001 BM25 + fuzzy | 5/5 | 0.000 | 0.833 | 1.000 | 0.450 | 0.597 | 1.030 |
| H003 dense | 4/5 | 0.267 | 0.533 | 0.800 | 0.508 | 0.554 | 0.108 |
Relative to H000, H001 improved mean recall@10 by 0.133, recall@5 by
0.267, MRR by 0.071, and NDCG@10 by 0.116. H003 reduced mean recall@10
by 0.067 but improved MRR by 0.130 and NDCG@10 by 0.074. The dense pattern
is heterogeneous: it ranked the gold file first on two tasks, but ranked the
TASK_GR_005 autoscaler file 24th and therefore missed it at top 10.
These observations support retaining lexical and dense retrieval as separate
factorial dimensions. They also motivate the already-specified hybrid/fusion
treatments; they do not justify changing confirmatory hypotheses after results
are observed.
## Systems observations
The content-addressed dense cache embedded 3,207 unique chunks. Across the five
snapshots, 9,408 chunk lookups reused cached vectors. Total dense index build
time was 566.5 seconds, of which 430.4 seconds was the cold first snapshot.
Subsequent snapshots embedded only their changed or boundary-shifted chunks.
The LLM was unloaded before indexing. Only Qwen3 Embedding 0.6B was resident
during E00, and macOS reported 73% system-wide free memory during repeated
checks. The embedding model was unloaded immediately after completion, leaving
no LM Studio model resident.
Dense query latency in the table excludes cold index construction, while H001
query latency includes its in-memory scoring pass. Future systems tables must
continue to separate index build, incremental update, and query latency.
## Limitations and next gate
- The statements were authored from known fixes and may be cleaner than organic
issue reports.
- All five tasks come from one repository and a narrow time window.
- Exact Qwen tokenizer measurement remains pending; current repository sizing
uses conservative byte and source-line evidence.
- The pilot uses Python flat cosine search, not the pinned FAISS backend planned
for confirmatory systems comparisons.
- The tasks are retrieval-ready, not end-to-end-ready; hidden regression tests
have not yet been validated.
- No model-selected search query, tool interaction, context interpretation, or
patch generation was evaluated.
The next gate is an LLM-in-the-loop localization pilot with Qwen3.6 loaded
alone, consuming the immutable E00 rankings without reloading the embedding
model. Only after that succeeds should H000 be extended to editing and tests.
|