| # Harness catalog |
|
|
| ## Dimension legend |
|
|
| | Code | Dimension | Values | |
| |---|---|---| |
| | L | Advanced lexical retrieval | 0 = exact/regex only; 1 = BM25 plus fuzzy retrieval | |
| | S | Syntax representation | 0 = raw lines; 1 = Tree-sitter symbols and chunks | |
| | D | Dense retrieval | 0 = off; 1 = code embeddings | |
| | G | Structural graph expansion | 0, 1, or 2 dependency hops | |
| | Q | Query policy | one-shot or iterative reformulation | |
| | I | Tool interface | unified search tool or specialized tools | |
| | P | Context packing | ranked snippets, skeletons, whole files, or role summaries | |
|
|
| Exact path, substring, and regex search remains available in H000-H015 and |
| H020. It is removed only from explicitly labeled controls. When two or more of |
| L, S, and D are active, their ranked lists use reciprocal-rank fusion (RRF). |
|
|
| ## The 21 initial treatments |
|
|
| | ID | L | S | D | G | Q | I | P | Purpose | |
| |---|---:|---:|---:|---:|---|---|---|---| |
| | H000 | 0 | 0 | 0 | 0 | one-shot | unified | snippets | Exact/regex baseline | |
| | H001 | 1 | 0 | 0 | 0 | one-shot | unified | snippets | Lexical main effect | |
| | H002 | 0 | 1 | 0 | 0 | one-shot | unified | snippets | Syntax main effect | |
| | H003 | 0 | 0 | 1 | 0 | one-shot | unified | snippets | Dense main effect | |
| | H004 | 1 | 1 | 0 | 0 | one-shot | unified | snippets | Lexical × syntax | |
| | H005 | 1 | 0 | 1 | 0 | one-shot | unified | snippets | Lexical × dense | |
| | H006 | 0 | 1 | 1 | 0 | one-shot | unified | snippets | Syntax × dense | |
| | H007 | 1 | 1 | 1 | 0 | one-shot | unified | snippets | Full retrieval, no graph | |
| | H008 | 1 | 1 | 1 | 1 | one-shot | unified | snippets | One-hop graph reference | |
| | H009 | 1 | 1 | 1 | 2 | one-shot | unified | snippets | Two-hop graph | |
| | H010 | 1 | 1 | 1 | 1 | iterative | unified | snippets | Iterative-query effect | |
| | H011 | 1 | 1 | 1 | 1 | one-shot | specialized | snippets | Interface effect | |
| | H012 | 1 | 1 | 1 | 1 | iterative | specialized | snippets | Query × interface | |
| | H013 | 1 | 1 | 1 | 1 | one-shot | unified | skeletons | Skeleton packing | |
| | H014 | 1 | 1 | 1 | 1 | one-shot | unified | whole files | Whole-file packing | |
| | H015 | 1 | 1 | 1 | 1 | one-shot | unified | role summaries | Summary packing | |
| | H016 | – | – | – | – | one-shot | unified | – | No-search negative control | |
| | H017 | – | – | – | – | one-shot | unified | random | Token-matched random-context control | |
| | H018 | – | – | – | – | one-shot | unified | gold files | Oracle-file upper bound | |
| | H019 | – | – | – | – | one-shot | unified | gold functions | Oracle-function upper bound | |
| | H020 | 1 | 1 | 1 | 1 | iterative | unified | snippets | Adaptive source router | |
|
|
| The unrestricted seven-dimension Cartesian product has 384 combinations. It is |
| not the primary design because most cells would answer no new clean causal |
| question while greatly multiplying model cost. The 21 treatments form focused |
| factorial and ablation blocks. If later evidence justifies another combination, |
| it receives a new file and ID. |
|
|
| Both a full configuration hash and a causal treatment hash are computed. The |
| latter excludes the ID, name, and description so that relabeling cannot conceal |
| a duplicate treatment. |
|
|
|
|
