| # E00 development pilot results |
|
|
| ## Status |
|
|
| E00 completed on 2026-07-17 using experiment-code revision |
| `651a2f8ca417d96fc79c8d6a1862638114ba4c1d`. The independent artifact audit |
| used analysis revision `57cde407f0d2e87f1761459c1f2f172040765bc5` and passed: |
|
|
| - 15 of 15 declared runs found; |
| - `run_manifest.json`, `trajectory.jsonl`, `ranking.json`, and |
| `final_metrics.json` present for every run; |
| - 3,060 trajectory events with contiguous sequence numbers; |
| - every trajectory starts with `run_started` and ends with `run_finished`; and |
| - report metrics exactly match the per-run final metrics. |
|
|
| This is a development retrieval-subsystem pilot, not a confirmatory experiment. |
| It contains no Qwen3.6 agent calls and therefore provides no evidence about |
| end-to-end coding-agent performance. |
|
|
| ## Descriptive results |
|
|
| Five tasks are too few for inferential claims. Values below are unweighted |
| means over the five paired development tasks. |
|
|
| | Harness | All gold in top 10 | Recall@1 | Recall@5 | Recall@10 | MRR | NDCG@10 | Query seconds | |
| |---|---:|---:|---:|---:|---:|---:|---:| |
| | H000 exact terms | 4/5 | 0.000 | 0.567 | 0.867 | 0.379 | 0.481 | 0.093 | |
| | H001 BM25 + fuzzy | 5/5 | 0.000 | 0.833 | 1.000 | 0.450 | 0.597 | 1.030 | |
| | H003 dense | 4/5 | 0.267 | 0.533 | 0.800 | 0.508 | 0.554 | 0.108 | |
|
|
| Relative to H000, H001 improved mean recall@10 by 0.133, recall@5 by |
| 0.267, MRR by 0.071, and NDCG@10 by 0.116. H003 reduced mean recall@10 |
| by 0.067 but improved MRR by 0.130 and NDCG@10 by 0.074. The dense pattern |
| is heterogeneous: it ranked the gold file first on two tasks, but ranked the |
| TASK_GR_005 autoscaler file 24th and therefore missed it at top 10. |
|
|
| These observations support retaining lexical and dense retrieval as separate |
| factorial dimensions. They also motivate the already-specified hybrid/fusion |
| treatments; they do not justify changing confirmatory hypotheses after results |
| are observed. |
|
|
| ## Systems observations |
|
|
| The content-addressed dense cache embedded 3,207 unique chunks. Across the five |
| snapshots, 9,408 chunk lookups reused cached vectors. Total dense index build |
| time was 566.5 seconds, of which 430.4 seconds was the cold first snapshot. |
| Subsequent snapshots embedded only their changed or boundary-shifted chunks. |
|
|
| The LLM was unloaded before indexing. Only Qwen3 Embedding 0.6B was resident |
| during E00, and macOS reported 73% system-wide free memory during repeated |
| checks. The embedding model was unloaded immediately after completion, leaving |
| no LM Studio model resident. |
|
|
| Dense query latency in the table excludes cold index construction, while H001 |
| query latency includes its in-memory scoring pass. Future systems tables must |
| continue to separate index build, incremental update, and query latency. |
|
|
| ## Limitations and next gate |
|
|
| - The statements were authored from known fixes and may be cleaner than organic |
| issue reports. |
| - All five tasks come from one repository and a narrow time window. |
| - Exact Qwen tokenizer measurement remains pending; current repository sizing |
| uses conservative byte and source-line evidence. |
| - The pilot uses Python flat cosine search, not the pinned FAISS backend planned |
| for confirmatory systems comparisons. |
| - The tasks are retrieval-ready, not end-to-end-ready; hidden regression tests |
| have not yet been validated. |
| - No model-selected search query, tool interaction, context interpretation, or |
| patch generation was evaluated. |
|
|
| The next gate is an LLM-in-the-loop localization pilot with Qwen3.6 loaded |
| alone, consuming the immutable E00 rankings without reloading the embedding |
| model. Only after that succeeds should H000 be extended to editing and tests. |
|
|
|
|
