docs/DEEP_DIVE.md

# Memory-LoRA: A Hypernetwork that Writes Repo-Specific Adapters for Gemma-4-E2B

> **One line:** we train a small neural network (a *hypernetwork*) that reads an
> embedding of a codebase and **emits a LoRA adapter** for a frozen
> `google/gemma-4-E2B`. The adapter injects repo-specific knowledge into the
> model with **zero extra tokens at inference time** — no RAG, no context stuffing.
> Everything runs **locally on an Apple-Silicon Mac** (M4 Pro, 64 GB).

This document is the onboarding bible for the project. It covers *what* we built,
*why* every non-obvious choice was made, *what data* we used, *every experiment we
ran and its result*, and the **hard-won lessons** (the MPS memory leak alone cost
us hours). Read it top to bottom once; after that use it as a reference.

---

## Table of Contents

1. [The idea in 60 seconds](#1-the-idea-in-60-seconds)
2. [Origin: the Code2LoRA paper](#2-origin-the-code2lora-paper)
3. [Architecture](#3-architecture)
4. [The target model: Gemma-4-E2B specifics](#4-the-target-model-gemma-4-e2b-specifics)
5. [Data pipeline: the 6 views](#5-data-pipeline-the-6-views)
6. [Data pipeline: QA generation](#6-data-pipeline-qa-generation)
7. [What is learnable — Tier A / B / C](#7-what-is-learnable--tier-a--b--c)
8. [Datasets inventory](#8-datasets-inventory)
9. [Experiments & results](#9-experiments--results)
10. [Key decisions & lessons learned](#10-key-decisions--lessons-learned)
11. [Repository map](#11-repository-map)
12. [How to run it](#12-how-to-run-it)
13. [Evaluation methodology](#13-evaluation-methodology)
14. [Costs & budget discipline](#14-costs--budget-discipline)
15. [Current status & roadmap](#15-current-status--roadmap)
16. [Glossary](#16-glossary)

---

## 1. The idea in 60 seconds

A coding agent (Claude Code, Codex, etc.) is great at reasoning but knows nothing
about *your* repo until you paste files into its context — which is slow, expensive,
and capped by the context window. The usual fix is RAG (retrieve chunks at query
time). We do something different and complementary:

**We bake the repo's "personality" directly into the model's weights, once, as a
LoRA adapter — and we generate that adapter with a neural network instead of
training it.**

```
                        ┌─────────────────────────────┐
   repo on disk ──────► │  6-view extractor + Qwen3    │ ──► 12288-d
                        │  frozen embedding encoder    │     repo embedding
                        └─────────────────────────────┘          │
                                                                 ▼
                                                   ┌──────────────────────────┐
                                                   │  HYPERNETWORK (our head) │
                                                   │  MLP → per-module A,B     │
                                                   └──────────────────────────┘
                                                                 │  LoRA weights
                                                                 ▼
   Q: "what layer owns auth in this repo?"  ──►  ┌──────────────────────────────┐
                                                 │ FROZEN Gemma-4-E2B + injected │ ──► "the middleware
                                                 │ LoRA  (zero extra tokens)     │      layer, via ..."
                                                 └──────────────────────────────┘
```

The magic: **the hypernetwork is trained across hundreds of repos**, so it learns
the *mapping* `repo embedding → good adapter`. At inference on a brand-new repo it
has never seen, it embeds the repo once and produces an adapter in a single forward
pass. This is the same reason the source paper needed 400+ repos, not 1: breadth is
what makes the mapping generalize.

---

## 2. Origin: the Code2LoRA paper

We reverse-engineered **Code2LoRA** (arXiv 2606.06492v1) and found its released
code (`anonymous.4open.science/r/code2lora-6857`, MIT). The paper's contribution:
a *static hypernetwork* that maps a **repository** embedding → a LoRA adapter for a
frozen code LLM, evaluated on **RepoPeftBench** with IR (in-repo) / CR (cross-repo)
splits. On a full H100 setup they report **63.8 % cross-repo Exact Match**.

Our project is the **Doc2LoRA variant** the paper itself cites — hypernetwork maps a
*document/repo view* → LoRA — reimplemented against **Gemma-4-E2B**, trained fully
**locally on MPS** (no CUDA/H100), and extended in two directions the paper does not
cover:

- **Memory / recall**: the adapter should let the model *recall facts* about the
  repo, not just complete code.
- **Tech-Lead judgment**: architecture, data-flow, conventions, contracts, ops —
  the things a 20-year senior engineer "just knows" about a codebase.

We keep the paper's proven autograd trick almost verbatim (see §3) and change only
what Gemma-4 and Apple Silicon force us to change.

---

## 3. Architecture

Three frozen/learned pieces. Only the middle one (the head) is trained.

### 3.1 Frozen encoder — `memory_lora/encoder.py`

- **Qwen3-Embedding-0.6B**, frozen, no gradient flows through it.
- Each repo view is chunked into token windows (2048 tokens, 128 overlap), each
  chunk mean-pooled, then chunks combined with **mean + max pooling** → a **2048-d**
  vector *per view*.
- Embeddings are **precomputed once and cached** to parquet — the encoder never runs
  during training.

### 3.2 The hypernetwork head — `memory_lora/core.py :: MemoryLoRAHead`

The only trained component. Design (kept close to the paper):

- **2-layer GELU MLP trunk** (`input_dim → hidden_dim → hidden_dim`), followed by
  **L2-normalize + √hidden_dim rescale** (stabilizes the magnitude of generated
  weights).
- **Per-module-type output heads**: for each target module *type* it emits an
  `A ∈ [rank, in_features]` and `B ∈ [out_features, rank]`. **One (A,B) pair per
  type, shared across all layers of that type** — this is what keeps the head
  tractable (188.6 M params) instead of exploding per-layer.
- **Squashing**: `tanh(raw) * exp(log_scale)` with a learned per-type `log_scale`
  (init **-3.5**). This starts the generated adapter near-zero (so training begins
  close to the base model) and lets each type learn its own output scale.
- **Defaults**: `hidden_dim=128`, `rank=16`, `dropout=0.1`.
  - *Why hidden_dim=128 and not the paper's 512/1024?* A 745 M-param head
    (hidden_dim=512) barely moved eval loss (~1.9 → ~2.7) but was far heavier to
    train on MPS. 128 cuts head size dramatically with negligible quality loss
    locally. Bump it later on real GPUs.

### 3.3 The LoRA injection — `memory_lora/core.py :: LoRA`

```
   base nn.Linear (FROZEN)              hypernetwork output
        │                                    │
   x ──►│ Wx  ────────────────┐              │
        │ (input detached      +──► y = Wx + scaling · B (A x)
        │  into base)          │              ▲   ▲
   x ──────────────────────────┘              │   │
                                     A,B are NON-detached tensors so
                                     autograd flows LM-loss → head
```

Critical detail (straight from the paper's code): **A and B are plain, non-buffer
tensor attributes, not `nn.Parameter` and not detached**, so the gradient of the LM
loss flows *through* the injected weights *into the hypernetwork*. The base
`nn.Linear` is frozen and its input is detached. Get this wrong and the head never
learns.

### 3.4 Shape-qualified module types — the heterogeneity fix

Gemma-4-E2B is **not** a uniform stack (see §4). Two `q_proj`s can have different
shapes. If you key the head by bare type name (`q_proj`) you get
*"type q_proj inconsistent dims"* crashes. Fix: key by **shape-qualified type**,
e.g. `q_proj_1536x2048` vs `q_proj_1536x4096`. The v2 run discovered **14 shape
types** across **205 target modules**:

```
down_proj_12288x1536  down_proj_6144x1536   gate_proj_1536x12288  gate_proj_1536x6144
k_proj_1536x256       k_proj_1536x512       o_proj_2048x1536      o_proj_4096x1536
q_proj_1536x2048      q_proj_1536x4096      up_proj_1536x12288    up_proj_1536x6144
v_proj_1536x256       v_proj_1536x512
```

`get_module_specs(root_prefix="model.language_model.")` restricts wrapping to the
text decoder — the **vision and audio towers are never touched** (not even
inspected), so the multimodal forward path stays intact and they cost only idle RAM.

---

## 4. The target model: Gemma-4-E2B specifics

Verified by reading the actual safetensors header, not guessed:

- **Real model.** Google shipped Gemma 4 in March 2026. Apache-2.0, ungated.
  Class `Gemma4ForConditionalGeneration`, `model_type: "gemma4"`. Loaded via
  `AutoModelForImageTextToText`.
- **Requires `transformers >= 5.5.0.dev0`** — install from the `main` branch, not a
  pinned PyPI release (this is the single biggest environment risk; smoke-test first).
- **Decoder is nested**: layers live at `model.language_model.layers.{i}.*`, *not*
  `model.layers.*`. The layer-index regex had to change accordingly.
- **35 text layers, heterogeneous:**
  - Aggressive KV sharing — **20 of 35 layers lack their own `k_proj`/`v_proj`**
    (`num_kv_shared_layers=20`).
  - **Every 5th layer is wider** (the `*_4096`, `*_12288` shape variants above).
- **Device `mps`, precision bf16** (fall back to fp16 if unstable). No
  `flash_attention_2` on MPS — use `sdpa`, fall back to `eager`.

---

## 5. Data pipeline: the 6 views

`scripts/build_repo_multiview.py` clones a repo and extracts **6 complementary
views**, embeds each with Qwen3 → 2048-d, and **concatenates to a 12288-d** repo
vector. The views encode the different "lenses" a senior engineer uses:

| View            | What it captures                                   | Source signals |
|-----------------|----------------------------------------------------|----------------|
| `v_graph`       | call / import / dependency structure               | AST for Python (`memory_lora/codegraph.py`); `IMPORT_RE`/`DEF_RE` regex fallback for other languages |
| `v_arch`        | architecture & layout                              | README, folder tree |
| `v_history`     | how the code evolved                               | `git log`, recent diffs |
| `v_contracts`   | behavioral contracts                               | test files |
| `v_conventions` | idioms & style                                     | representative source files |
| `v_ops`         | build / deploy / runtime                           | CI config, Dockerfile, build files |

**Multi-language from the start.** `CODE_EXTS` + regex fallbacks mean the graph view
works for 9 languages, not just Python (this was a deliberate correction — see §10).
Repos with < 3 code files are skipped. The build is **resume-safe** (skips repos
already in `multiview_sources.jsonl`) and **flushes the embeddings parquet every 10
repos**, so a crash never loses more than 10 repos of work.

---

## 6. Data pipeline: QA generation

The repo embedding is the *input*; the *target* is repo-scoped Q&A. Two generators,
both parallelized (`ThreadPoolExecutor`, `--workers 10`) with a **per-prompt disk
cache** (idempotent reruns) and a `--model` flag:

- **`scripts/generate_repo_scoped_qa.py`** — reads the same 6 views and asks the LLM
  for **8–12 repo-level judgment questions** ("what layer owns X", "what convention
  does this repo use for Y", "how does data flow through Z", "why is it structured
  this way"). Target scope = input scope (repo-level embedding ↔ repo-level QA).
- **`scripts/generate_commitpack_qa.py`** — **breadth** generator: one commit per
  *distinct* repo across CommitPackFT (25k+ distinct repos), 3–4 commit-scoped
  judgment questions (why / conventions / contracts / impact). For a hypernetwork,
  **distinct-repo count is the currency of generalization**, so we favor 1 commit ×
  many repos over many commits × one repo.

**Discipline (both):** answers are **short judgment**, never file-path/line-number
lists. This is deliberate — see Tier A/B/C next.

**Models used (OpenRouter, OpenAI-compatible API):**

| Model | Role | Notes | Cost |
|-------|------|-------|------|
| `google/gemini-3.6-flash` | high-quality QA | **reasoning is mandatory** → needs generous `max_tokens` (3000–4000) or it returns empty | ~$0.0021 / QA |
| `google/gemma-4-31b-it` | bulk / cheap QA | non-reasoning, clean JSON | ~$0.001 / repo (~$1 per 1000 repos) |

The OpenRouter key lives in a **git-ignored `.env` (mode 600)** and is never pasted
into a command line.

---

## 7. What is learnable — Tier A / B / C

A LoRA adapter has finite capacity. We classify repo knowledge by whether a LoRA can
hold it — this drives the entire QA design:

- **Tier A — Judgment & conventions** (LEARNABLE). "This repo puts business logic in
  services, not views." Compressible, generalizes. → **This is what we train on.**
- **Tier B — Structural gist** (LEARNABLE). "Auth flows through middleware." The kind
  of thing, not the exact file.
- **Tier C — Exact recall & multi-hop** (NOT reliably learnable). "Line 412 of
  `foo.py` calls `bar()`." This needs **retrieval (RAG)**, not weights.

So Memory-LoRA and RAG are **complementary**: the adapter carries Tier A/B judgment
for free (zero tokens); RAG handles Tier C exact lookups. The QA prompts forbid exact
file/line answers precisely so we never ask the LoRA to do a job it structurally
can't.

---

## 8. Datasets inventory

Everything lives under `data/` (git-ignored blobs). Sizes are approximate.

| Path | What | Scale |
|------|------|-------|
| `data/real_code2lora/` | **RepoPeftBench** from the `code2lora` HF org — 500 Python repos, repo-commit embeddings + diffs | 73,849 repo-commit rows; ~1.2 GB |
| `data/commitpack/multilang_commits.jsonl` | **CommitPackFT** shards, 9 languages | 25k+ distinct repos |
| `data/docs/multiview_sources.jsonl` | 6-view `view_text` per repo (input to QA gen) | growing (1000s of repos) |
| `data/embeddings/multiview_embeddings.parquet` | 12288-d multi-view repo embeddings | 1000+ repos |
| `data/embeddings/aligned6_embeddings.parquet` | **assembled training inputs** (repos with ≥1 QA) | 1058 repos (current) |
| `data/qna/repo_scoped_qa.jsonl` | repo-level judgment QA | 11,232 QA |
| `data/qna/techlead_qa_commitpack.jsonl` | commit-scoped breadth QA | 9,245 QA |
| `data/qna/techlead_qa.jsonl` | SWE-bench tech-lead QA | 2,786 QA |
| `data/qna/aligned6_qna.jsonl` | **assembled training targets** | 8,540 QA (current) |
| `data/openrouter_cache/` | per-prompt response cache | ~19 MB |

**Language balancing.** SWE-bench is ~79 % Django. Left alone, the dataset was 46 %
Django. `scripts/consolidate_qa.py` applies a **per-repo cap** (default 12–15 QA/repo)
which collapses Django to **~2.0 %** while preserving the 2400+ distinct repos'
diversity. `assemble_6view_dataset.py` applies the same cap when building the final
aligned set.

---

## 9. Experiments & results

Chronological, with the actual numbers we measured. Two families of runs.

### 9.1 Reproducing the paper (single-view, real RepoPeftBench)

| Run | What | Result |
|-----|------|--------|
| `full1` (early) | first end-to-end hypernetwork on converted real data | CR **EM 0.056–0.083**, EditSim ~0.27 — pipeline works, undertrained |
| `sixview`/converted-real (best single-view ckpt) | after more training | **CR EM 0.524, EditSim 0.635** |
| Paper (reference, H100) | their full run | CR EM **0.638** |

**Headline:** on real code, after only ~2.4 % of one epoch of local MPS training, we
reached **52.4 % cross-repo Exact Match** vs the paper's 63.8 % on a full H100 setup.
The mechanism demonstrably works — the generated adapter recovers repo-specific
identifiers the base model does not know.

### 9.2 The 6-view Tech-Lead model (the current line of work)

Loss is causal-LM cross-entropy on QA targets; lower is better. Three eval suites:
`cr_val` / `cr_test` (held-out *repos*) and `ir_test` (held-out *QA* of train repos).

| Run | Dataset | Best held-out `cr_test` loss | Notes |
|-----|---------|------------------------------|-------|
| `sixview_v1` | 515 repos / 3,988 QA (415 train repos) | **2.848** (step ~1060) | Overfit afterward: train loss fell to 1.75 while `cr_test` drifted to 3.35. Classic small-dataset ceiling. |
| `sixview_v2` | **1,058 repos / 8,540 QA (858 train repos)** | *in progress* | Resumed from `sixview_v1/head.best.pt`; 2× the data specifically to break v1's ceiling. |

`sixview_v1` metrics trajectory (from `runs/sixview_v1/metrics.jsonl`):

```
step 1245  cr_test 2.962   ir_test 2.535   (end of epoch 2 — near best)
step 1400  cr_test 3.245   ir_test 2.593   (overfitting begins)
step 1600  cr_test 3.352   ir_test 2.655   (train loss still falling → ceiling hit)
```

The v1→v2 story is the core empirical lesson: **the small aligned set was the
bottleneck, not the architecture** — hence the push to build 1000+ more repos.

---

## 10. Key decisions & lessons learned

The expensive knowledge. Read this section twice.

### 10.1 ⚠️ The MPS gradient-checkpointing memory leak (the big one)

**Symptom:** training with `gradient_checkpointing_enable()` (`use_reentrant=False`)
**leaked ~12 GB per step** and OOM'd the whole machine within a few steps.

**Diagnosis** (`scripts/diag_mps_leak.py`): forward-only was stable; train + checkpoint
leaked 39 GB → 18 GB free in 2 steps. Isolated the checkpointing path as the cause.

**Fix:** **`--no-gradient-checkpointing`.** We have enough unified memory to hold
activations without it once the multimodal towers sit idle. This is documented as a
standing memory (`mps-gradient-checkpointing-leak.md`).

### 10.2 ⚠️ `psutil` RSS is blind to MPS memory

Our first memory safety-net used `psutil` RSS / `ps -o rss` — it reported **< 1 GB**
while `top` showed **55–83 GB** actually in use. MPS (GPU) allocations don't show up
in process RSS.

**Fix:** the safety check uses **`psutil.virtual_memory().available`** (system-wide)
with a `--min-available-gb` floor (default 5). To *observe* MPS memory, use
`top -l 1 -pid <PID> -stats mem`, not `ps`.

### 10.3 Memory competition between concurrent jobs

Three concurrent jobs once pushed available memory under the 10 GB floor and training
self-stopped. **Lesson:** during MPS training, run data builds/embedding on **CPU**
(`--device cpu`) so they don't contend for the GPU/unified memory. We now routinely
run training (MPS) + QA gen (network) + multiview build (CPU) together without
contention.

### 10.4 Don't lose hours of training

Every long run writes **checkpoints every 50 steps** (overwriting `head.latest.pt`),
**every 30 minutes** (timestamped `head.tNNNNm.pt`), **per-epoch** (`head.epN.pt`),
and a **`head.best.pt`** on eval improvement. Runs are launched with `nohup … &
disown` so they survive terminal/session death. `sixview_v1` in fact survived a full
session interruption and kept training. Resume with `--resume-from <ckpt>` (loads head
weights; optimizer restarts fresh).

### 10.5 Data-quality corrections (user-driven)

- **"I still see lots of Django."** SWE-bench is Django-dominated. → per-repo cap +
  multi-language sourcing dropped Django 46 % → 2.0 %.
- **"It must be good for any programming language."** → 9-language diversity via
  CommitPackFT and language-agnostic view extraction.
- **"Where's the code context in the QA?"** → clarified the two-channel design: the
  **repo embedding is the context channel**, the QA is only the target. They are
  joined by `doc_id` at assembly time.

### 10.6 OpenRouter gotchas

- `gemini-3.6-flash` **returned empty** until we raised `max_tokens` — it's a
  mandatory-reasoning model that spends tokens on hidden reasoning before content.
  Reasoning **cannot be disabled** (400 error).
- CommitPackFT's HF loader is deprecated → fetch raw `data.jsonl` directly.
- `global MODEL` after use is a `SyntaxError` → set via `globals()["MODEL"] = ...`.

### 10.7 Performance fix worth knowing

Loading embeddings was 5+ min because `_list_to_f32_array` used a Python loop.
Vectorized via `col.combine_chunks().flatten().to_numpy()` → **~220× faster**.

---

## 11. Repository map

```
memory_lora/                  # the library (importable package)
  core.py                     # LoRA wrapper, MemoryLoRAHead hypernetwork,
                              #   get_module_specs / replace_with_lora / inject_lora_weights,
                              #   load_doc_rows / load_qna_rows
  encoder.py                  # Qwen3 chunk + embed + mean/max pool (frozen)
  codegraph.py                # Python AST extractor (imports, sigs, call graph)
  data_paths.py               # local parquet/jsonl path resolver

scripts/
  build_repo_multiview.py     # clone → 6 views → 12288-d embeddings (multi-language, resume-safe)
  generate_repo_scoped_qa.py  # repo-level judgment QA (aligned to the 6 views)
  generate_commitpack_qa.py   # commit-scoped breadth QA across 1000s of distinct repos
  generate_techlead_qa.py     # SWE-bench tech-lead QA
  generate_synthetic_dataset.py  # original synthetic doc + QA generator
  consolidate_qa.py           # per-repo cap → language/domain balancing
  assemble_6view_dataset.py   # join embeddings ↔ all QA by repo → aligned6_{embeddings,qna}
  augment_paraphrases.py      # QA paraphrase augmentation
  convert_real_code2lora.py   # RepoPeftBench → our schema
  build_doc_embeddings.py     # encoder pass over documents
  merge_corpora.py            # combine multiple corpora
  train_memory_lora.py        # THE trainer (MPS, one-repo-per-step, checkpoints, TensorBoard)
  train_direct_lora.py        # baseline: plain per-repo LoRA (no hypernetwork)
  eval_memory_lora.py         # EM / EditSim recall eval on cr/ir splits
  show_eval_examples.py       # dump concrete base-vs-adapted examples
  test_embed_this_repo.py     # embed the current repo (pipeline demo)
  test_recall_this_repo.py    # query the adapted model about this repo
  diag_mps_leak.py            # the memory-leak isolation harness

data/                         # git-ignored: embeddings, qna, sources, caches
runs/                         # git-ignored: checkpoints, logs, metrics.jsonl, tb/
requirements.txt              # torch 2.13 (MPS), transformers@main, openai, pyarrow, tensorboard…
.env                          # git-ignored, mode 600: OPENROUTER_API_KEY
```

---

## 12. How to run it

### Setup

```bash
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt          # installs transformers from git main
echo "OPENROUTER_API_KEY=sk-or-..." > .env && chmod 600 .env
```

Smoke-test the environment first (gates everything): confirm `transformers` main
loads `google/gemma-4-E2B` on `mps` and runs a text-only forward pass.

### Build data

```bash
# 1) multi-view embeddings for a repo list (CPU to stay off the GPU during training)
python scripts/build_repo_multiview.py \
  --repos-file data/multilang_repo_list.txt --max-repos 1000 --device cpu

# 2) repo-scoped QA (cheap model) — appends, resume-safe, cached
./venv/bin/python scripts/generate_repo_scoped_qa.py \
  --model google/gemma-4-31b-it --workers 10

# 3) balance + assemble the aligned training set
python scripts/consolidate_qa.py --per-repo-cap 12
python scripts/assemble_6view_dataset.py
# -> data/embeddings/aligned6_embeddings.parquet + data/qna/aligned6_qna.jsonl
```

### Train (the exact `sixview_v2` command)

```bash
nohup ./venv/bin/python scripts/train_memory_lora.py --output-dir sixview_v2 \
  --resume-from runs/sixview_v1/head.best.pt \
  --embeddings-path data/embeddings/aligned6_embeddings.parquet \
  --qna-path data/qna/aligned6_qna.jsonl --epochs 100 --max-hours 8 \
  --checkpoint-every-steps 50 --checkpoint-every-minutes 30 --epoch-ckpt-every 5 \
  --eval-every-steps 300 --limit-eval-docs 40 --max-seq-len 512 --fixed-seq-len \
  --max-qna-per-doc 12 --lm-micro-batch 2 --device mps --no-gradient-checkpointing \
  --rank 16 --head-hidden-dim 128 --head-dropout 0.1 --weight-decay 0.05 \
  --early-stop-patience 25 --lr 8e-5 --lr-total-steps 9000 --min-available-gb 5 \
  > runs/sixview_v2_train.log 2>&1 &
disown
```

**Flags you must not forget:** `--no-gradient-checkpointing` (the leak),
`--device cpu` for builds during training (contention), `--min-available-gb`
(system-wide memory floor).

### Watch it

```bash
tensorboard --logdir runs/sixview_v2/tb    # train/loss, train/lr, eval/{suite}_loss
tail -f runs/sixview_v2_train.log
```

### Evaluate & inspect

```bash
python scripts/eval_memory_lora.py --ckpt runs/sixview_v2/head.best.pt   # EM / EditSim
python scripts/show_eval_examples.py                                     # base vs adapted
```

---

## 13. Evaluation methodology

- **Splits (deterministic, by `md5(repo) % 100`):** 80 % train / 10 % `cr_val` /
  10 % `cr_test` **by repo**, so cross-repo suites are **entirely held-out
  repositories** the hypernetwork never trained on. Within train repos, ~15 % of QA
  is held out → `ir_test` (in-repo generalization to unseen questions of seen repos).
- **Metrics:** causal-LM **eval loss** during training (fast, every N steps on
  `--limit-eval-docs` docs to stay cheap on CPU), plus generation-time **Exact Match
  (EM)** and **EditSim** for the recall eval.
- **The proof spot-check:** query the *adapted* model with repo-specific questions and
  confirm the *base* (un-adapted) model gets them wrong/vague — proving the
  **adapter**, not the base model's pretraining, does the work.

CPU eval of a float32 5B model is slow (~20 min for a full pass) → we cap eval docs
(e.g. 10–40) for in-loop evals and run full EM eval separately.

---

## 14. Costs & budget discipline

- **Spend baseline:** $31.00 (`runs/spend_baseline.txt`); ~**$33.90 total** to date;
  ~**$11 remaining**. The project is run under explicit budget caps ("spend at most
  $4 more") with spend-guards.
- **Unit economics:** `gemini-3.6-flash` ≈ **$0.0021/QA**; `gemma-4-31b-it` ≈
  **$0.001/repo (~$1 per 1000 repos)** — which is exactly why the 1000-repo expansion
  uses the gemma model.
- **Free levers:** the per-prompt cache makes reruns free; embedding and training are
  local (electricity only).

---

## 15. Current status & roadmap

**Live right now (three jobs in parallel, no contention):**

- **`sixview_v2` training** — resumed from `head.best.pt` on the doubled
  **1,058-repo / 8,540-QA** dataset (858 steps/epoch), MPS. First held-out eval at
  step 300 tells us whether doubling the data broke v1's 2.848 ceiling.
- **QA generation** — `gemma-4-31b-it` filling in all ~1,032 new repos for the
  **complete** dataset (next training run).
- **Multiview build** — cloning/embedding toward the full +1,000-new target (CPU).

**Roadmap:**

1. Finish the complete 1000-new-repo dataset (embeddings + QA).
2. Assemble the full aligned set (~1,650 repos) and train **`sixview_v3`** on it.
3. Run generation-time **EM/EditSim** on the 6-view model (base vs adapted).
4. Push `head_hidden_dim` back up once on real GPUs; the 128 default was an
   MPS-locality compromise.
5. Broaden Tier-A/B QA toward agent-harness use cases (Jira/ticket tracking,
   diff/impact reasoning) already scaffolded in `generate_techlead_qa.py`.

**Open questions:**

- Does the 12288-d 6-view embedding actually beat the single 2048-d view on
  generation EM, or only on loss? (loss says yes; EM eval pending)
- What's the real Tier-B ceiling — how much structural gist fits in rank-16?
- Optimal per-repo QA cap for the breadth/depth trade-off.

---

## 16. Glossary

- **Hypernetwork** — a network that outputs the weights of another network. Here:
  repo embedding → LoRA (A,B) matrices.
- **LoRA** — Low-Rank Adaptation: `y = Wx + scaling · B(Ax)`, with `A,B` low-rank
  (rank 16). We *generate* A,B instead of training them per-repo.
- **6 views** — graph / arch / history / contracts / conventions / ops; each 2048-d,
  concatenated to 12288-d.
- **CR / IR** — cross-repo (held-out repos) / in-repo (held-out QA of seen repos).
- **EM / EditSim** — Exact Match / edit-distance similarity of generated vs gold.
- **Tier A/B/C** — judgment (learnable) / structural gist (learnable) / exact recall
  (needs RAG).
- **MPS** — Apple's Metal Performance Shaders GPU backend for PyTorch.
- **RepoPeftBench** — the paper's benchmark; 500 Python repos, repo-commit
  embeddings + diffs, IR/CR splits.

---

*Maintained as living documentation. If you change a default, a path, or a flag,
update the matching section here — onboarding depends on it.*