Rewrite the model card

The old README was a 581-line internal onboarding document; it now lives at
docs/DEEP_DIVE.md. The card that replaces it opens with what the model does and
how to use it, adds HF frontmatter (license, base_model, library_name, tags),
and documents the parts a user cannot guess:

- the head expects the specific 6-view 12288-d embedding and standardises its
input internally, so callers must pass RAW embeddings
- the emitted (A,B) follow PEFT's LoRA convention exactly
- which checkpoint to use and why the removed ones were removed

Results are reported as deltas against the frozen base on identical data,
because absolute losses across differing eval sets are not comparable -- the
confusion that hid a checkpoint scoring worse than no adapter at all.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

README.md

@@ -1,581 +1,239 @@
-# Memory-LoRA: A Hypernetwork that Writes Repo-Specific Adapters for Gemma-4-E2B
-
-> **One line:** we train a small neural network (a *hypernetwork*) that reads an
-> embedding of a codebase and **emits a LoRA adapter** for a frozen
-> `google/gemma-4-E2B`. The adapter injects repo-specific knowledge into the
-> model with **zero extra tokens at inference time** — no RAG, no context stuffing.
-> Everything runs **locally on an Apple-Silicon Mac** (M4 Pro, 64 GB).
-
-This document is the onboarding bible for the project. It covers *what* we built,
-*why* every non-obvious choice was made, *what data* we used, *every experiment we
-ran and its result*, and the **hard-won lessons** (the MPS memory leak alone cost
-us hours). Read it top to bottom once; after that use it as a reference.
-
 ---
-
-## Table of Contents
-
-1. [The idea in 60 seconds](#1-the-idea-in-60-seconds)
-2. [Origin: the Code2LoRA paper](#2-origin-the-code2lora-paper)
-3. [Architecture](#3-architecture)
-4. [The target model: Gemma-4-E2B specifics](#4-the-target-model-gemma-4-e2b-specifics)
-5. [Data pipeline: the 6 views](#5-data-pipeline-the-6-views)
-6. [Data pipeline: QA generation](#6-data-pipeline-qa-generation)
-7. [What is learnable — Tier A / B / C](#7-what-is-learnable--tier-a--b--c)
-8. [Datasets inventory](#8-datasets-inventory)
-9. [Experiments & results](#9-experiments--results)
-10. [Key decisions & lessons learned](#10-key-decisions--lessons-learned)
-11. [Repository map](#11-repository-map)
-12. [How to run it](#12-how-to-run-it)
-13. [Evaluation methodology](#13-evaluation-methodology)
-14. [Costs & budget discipline](#14-costs--budget-discipline)
-15. [Current status & roadmap](#15-current-status--roadmap)
-16. [Glossary](#16-glossary)
-
+license: apache-2.0
+base_model: google/gemma-4-E2B
+library_name: peft
+tags:
+  - hypernetwork
+  - lora
+  - code
+  - repository-adaptation
+  - gemma
+pipeline_tag: text-generation
 ---
 
-## 1. The idea in 60 seconds
-
-A coding agent (Claude Code, Codex, etc.) is great at reasoning but knows nothing
-about *your* repo until you paste files into its context — which is slow, expensive,
-and capped by the context window. The usual fix is RAG (retrieve chunks at query
-time). We do something different and complementary:
+# Memory-LoRA — a hypernetwork that writes repo-specific LoRA adapters for Gemma-4-E2B
 
-**We bake the repo's "personality" directly into the model's weights, once, as a
-LoRA adapter — and we generate that adapter with a neural network instead of
-training it.**
+Give it a code repository; it returns a **LoRA adapter for that repository in one
+forward pass**. No fine-tuning, no retrieval, and **zero repository tokens at
+inference time** — the knowledge lives in the weights.
 
 ```
-                        ┌─────────────────────────────┐
-   repo on disk ──────► │  6-view extractor + Qwen3    │ ──► 12288-d
-                        │  frozen embedding encoder    │     repo embedding
-                        └─────────────────────────────┘          │
-                                                                 ▼
-                                                   ┌──────────────────────────┐
-                                                   │  HYPERNETWORK (our head) │
-                                                   │  MLP → per-module A,B     │
-                                                   └──────────────────────────┘
-                                                                 │  LoRA weights
-                                                                 ▼
-   Q: "what layer owns auth in this repo?"  ──►  ┌──────────────────────────────┐
-                                                 │ FROZEN Gemma-4-E2B + injected │ ──► "the middleware
-                                                 │ LoRA  (zero extra tokens)     │      layer, via ..."
-                                                 └──────────────────────────────┘
+repo ──► 6-view embedding (12288-d) ──► hypernetwork ──► LoRA adapter ──► frozen Gemma-4-E2B
+          frozen Qwen3-Embedding-0.6B      750M params       rank 16, α 32
 ```
 
-The magic: **the hypernetwork is trained across hundreds of repos**, so it learns
-the *mapping* `repo embedding → good adapter`. At inference on a brand-new repo it
-has never seen, it embeds the repo once and produces an adapter in a single forward
-pass. This is the same reason the source paper needed 400+ repos, not 1: breadth is
-what makes the mapping generalize.
+On repositories **absent from the training corpus**, the generated adapter makes
+correct answers roughly **200× more likely** than the frozen model (−5.31 nats)
+and wins **9 of 9** benchmark family/repo combinations.
 
----
-
-## 2. Origin: the Code2LoRA paper
-
-We reverse-engineered **Code2LoRA** (arXiv 2606.06492v1) and found its released
-code (`anonymous.4open.science/r/code2lora-6857`, MIT). The paper's contribution:
-a *static hypernetwork* that maps a **repository** embedding → a LoRA adapter for a
-frozen code LLM, evaluated on **RepoPeftBench** with IR (in-repo) / CR (cross-repo)
-splits. On a full H100 setup they report **63.8 % cross-repo Exact Match**.
+| question (`pallets/click`, never trained on) | frozen Gemma-4-E2B | + generated adapter |
+|---|---|---|
+| What testing framework does this use? | "Jest" ✗ | **pytest** ✓ |
+| How is this project built and packaged? | "a Dockerfile" ✗ | **setuptools, setup.py** ✓ |
+| What documentation tool? | "a Gantt chart" ✗ | **Sphinx** ✓ |
 
-Our project is the **Doc2LoRA variant** the paper itself cites — hypernetwork maps a
-*document/repo view* → LoRA — reimplemented against **Gemma-4-E2B**, trained fully
-**locally on MPS** (no CUDA/H100), and extended in two directions the paper does not
-cover:
-
-- **Memory / recall**: the adapter should let the model *recall facts* about the
-  repo, not just complete code.
-- **Tech-Lead judgment**: architecture, data-flow, conventions, contracts, ops —
-  the things a 20-year senior engineer "just knows" about a codebase.
-
-We keep the paper's proven autograd trick almost verbatim (see §3) and change only
-what Gemma-4 and Apple Silicon force us to change.
+Code, demo app, and full engineering write-up:
+**https://github.com/moncifem/memory-lora-gemma4** · deep dive in
+[`docs/DEEP_DIVE.md`](docs/DEEP_DIVE.md).
 
 ---
 
-## 3. Architecture
-
-Three frozen/learned pieces. Only the middle one (the head) is trained.
+## What's in this repository
 
-### 3.1 Frozen encoder — `memory_lora/encoder.py`
+| path | what it is |
+|---|---|
+| `runs/all_lora/all_lora_best_cpt.pth` | **the model.** Merged corpus (2,146 repos / 55,700 QA). `cr_val` 2.6811, `cr_test` 2.6266 |
+| `runs/h200_run/head.best.pt` | prose-QA-only variant, kept as a fallback |
+| `data/embeddings/*.parquet` | precomputed 12288-d repo embeddings |
+| `data/qna/*.jsonl` | training QA corpora |
+| `data/real_code2lora/` | RepoPeftBench (Code2LoRA's benchmark) |
+| `memory_lora/` `scripts/` `app/` `deploy/` | code, mirrored from GitHub |
 
-- **Qwen3-Embedding-0.6B**, frozen, no gradient flows through it.
-- Each repo view is chunked into token windows (2048 tokens, 128 overlap), each
-  chunk mean-pooled, then chunks combined with **mean + max pooling** → a **2048-d**
-  vector *per view*.
-- Embeddings are **precomputed once and cached** to parquet — the encoder never runs
-  during training.
+Earlier `sixview_v1` / `sixview_v2` weights were **removed**: they predate the
+input-standardisation fix, and `sixview_v2` measurably *degrades* the base model
+(see below). Their `metrics.jsonl` remain for provenance.
 
-### 3.2 The hypernetwork head — `memory_lora/core.py :: MemoryLoRAHead`
+---
 
-The only trained component. Design (kept close to the paper):
+## Using the hypernetwork
 
-- **2-layer GELU MLP trunk** (`input_dim → hidden_dim → hidden_dim`), followed by
-  **L2-normalize + √hidden_dim rescale** (stabilizes the magnitude of generated
-  weights).
-- **Per-module-type output heads**: for each target module *type* it emits an
-  `A ∈ [rank, in_features]` and `B ∈ [out_features, rank]`. **One (A,B) pair per
-  type, shared across all layers of that type** — this is what keeps the head
-  tractable (188.6 M params) instead of exploding per-layer.
-- **Squashing**: `tanh(raw) * exp(log_scale)` with a learned per-type `log_scale`
-  (init **-3.5**). This starts the generated adapter near-zero (so training begins
-  close to the base model) and lets each type learn its own output scale.
-- **Defaults**: `hidden_dim=128`, `rank=16`, `dropout=0.1`.
-  - *Why hidden_dim=128 and not the paper's 512/1024?* A 745 M-param head
-    (hidden_dim=512) barely moved eval loss (~1.9 → ~2.7) but was far heavier to
-    train on MPS. 128 cuts head size dramatically with negligible quality loss
-    locally. Bump it later on real GPUs.
+### 1. Get the pieces
 
-### 3.3 The LoRA injection — `memory_lora/core.py :: LoRA`
+```bash
+git lfs install
+git clone https://huggingface.co/moncefem/memory-lora-gemma4
+cd memory-lora-gemma4
+git lfs pull --include="runs/all_lora/all_lora_best_cpt.pth"
 
-```
-   base nn.Linear (FROZEN)              hypernetwork output
-        │                                    │
-   x ──►│ Wx  ────────────────┐              │
-        │ (input detached      +──► y = Wx + scaling · B (A x)
-        │  into base)          │              ▲   ▲
-   x ──────────────────────────┘              │   │
-                                     A,B are NON-detached tensors so
-                                     autograd flows LM-loss → head
+pip install torch transformers peft safetensors pyarrow numpy
+python app/engine/fetch_base_model.py     # google/gemma-4-E2B, 10.25 GB
 ```
 
-Critical detail (straight from the paper's code): **A and B are plain, non-buffer
-tensor attributes, not `nn.Parameter` and not detached**, so the gradient of the LM
-loss flows *through* the injected weights *into the hypernetwork*. The base
-`nn.Linear` is frozen and its input is detached. Get this wrong and the head never
-learns.
+### 2. Embed a repository (6 views → 12288-d)
 
-### 3.4 Shape-qualified module types — the heterogeneity fix
+The head is conditioned on a specific representation: six views of the repo —
+call graph, architecture, git history, contracts/tests, conventions, ops — each
+embedded by a frozen `Qwen3-Embedding-0.6B` and concatenated. **A different
+embedding will not work.**
 
-Gemma-4-E2B is **not** a uniform stack (see §4). Two `q_proj`s can have different
-shapes. If you key the head by bare type name (`q_proj`) you get
-*"type q_proj inconsistent dims"* crashes. Fix: key by **shape-qualified type**,
-e.g. `q_proj_1536x2048` vs `q_proj_1536x4096`. The v2 run discovered **14 shape
-types** across **205 target modules**:
-
-```
-down_proj_12288x1536  down_proj_6144x1536   gate_proj_1536x12288  gate_proj_1536x6144
-k_proj_1536x256       k_proj_1536x512       o_proj_2048x1536      o_proj_4096x1536
-q_proj_1536x2048      q_proj_1536x4096      up_proj_1536x12288    up_proj_1536x6144
-v_proj_1536x256       v_proj_1536x512
+```bash
+git clone --depth 80 https://github.com/pallets/click /tmp/click
+python app/engine/build_embedding.py --repo /tmp/click --out /tmp/click.npy
 ```
 
-`get_module_specs(root_prefix="model.language_model.")` restricts wrapping to the
-text decoder — the **vision and audio towers are never touched** (not even
-inspected), so the multimodal forward path stays intact and they cost only idle RAM.
-
----
-
-## 4. The target model: Gemma-4-E2B specifics
-
-Verified by reading the actual safetensors header, not guessed:
-
-- **Real model.** Google shipped Gemma 4 in March 2026. Apache-2.0, ungated.
-  Class `Gemma4ForConditionalGeneration`, `model_type: "gemma4"`. Loaded via
-  `AutoModelForImageTextToText`.
-- **Requires `transformers >= 5.5.0.dev0`** — install from the `main` branch, not a
-  pinned PyPI release (this is the single biggest environment risk; smoke-test first).
-- **Decoder is nested**: layers live at `model.language_model.layers.{i}.*`, *not*
-  `model.layers.*`. The layer-index regex had to change accordingly.
-- **35 text layers, heterogeneous:**
-  - Aggressive KV sharing — **20 of 35 layers lack their own `k_proj`/`v_proj`**
-    (`num_kv_shared_layers=20`).
-  - **Every 5th layer is wider** (the `*_4096`, `*_12288` shape variants above).
-- **Device `mps`, precision bf16** (fall back to fp16 if unstable). No
-  `flash_attention_2` on MPS — use `sdpa`, fall back to `eager`.
-
----
-
-## 5. Data pipeline: the 6 views
-
-`scripts/build_repo_multiview.py` clones a repo and extracts **6 complementary
-views**, embeds each with Qwen3 → 2048-d, and **concatenates to a 12288-d** repo
-vector. The views encode the different "lenses" a senior engineer uses:
-
-| View            | What it captures                                   | Source signals |
-|-----------------|----------------------------------------------------|----------------|
-| `v_graph`       | call / import / dependency structure               | AST for Python (`memory_lora/codegraph.py`); `IMPORT_RE`/`DEF_RE` regex fallback for other languages |
-| `v_arch`        | architecture & layout                              | README, folder tree |
-| `v_history`     | how the code evolved                               | `git log`, recent diffs |
-| `v_contracts`   | behavioral contracts                               | test files |
-| `v_conventions` | idioms & style                                     | representative source files |
-| `v_ops`         | build / deploy / runtime                           | CI config, Dockerfile, build files |
-
-**Multi-language from the start.** `CODE_EXTS` + regex fallbacks mean the graph view
-works for 9 languages, not just Python (this was a deliberate correction — see §10).
-Repos with < 3 code files are skipped. The build is **resume-safe** (skips repos
-already in `multiview_sources.jsonl`) and **flushes the embeddings parquet every 10
-repos**, so a crash never loses more than 10 repos of work.
-
----
-
-## 6. Data pipeline: QA generation
-
-The repo embedding is the *input*; the *target* is repo-scoped Q&A. Two generators,
-both parallelized (`ThreadPoolExecutor`, `--workers 10`) with a **per-prompt disk
-cache** (idempotent reruns) and a `--model` flag:
-
-- **`scripts/generate_repo_scoped_qa.py`** — reads the same 6 views and asks the LLM
-  for **8–12 repo-level judgment questions** ("what layer owns X", "what convention
-  does this repo use for Y", "how does data flow through Z", "why is it structured
-  this way"). Target scope = input scope (repo-level embedding �� repo-level QA).
-- **`scripts/generate_commitpack_qa.py`** — **breadth** generator: one commit per
-  *distinct* repo across CommitPackFT (25k+ distinct repos), 3–4 commit-scoped
-  judgment questions (why / conventions / contracts / impact). For a hypernetwork,
-  **distinct-repo count is the currency of generalization**, so we favor 1 commit ×
-  many repos over many commits × one repo.
-
-**Discipline (both):** answers are **short judgment**, never file-path/line-number
-lists. This is deliberate — see Tier A/B/C next.
-
-**Models used (OpenRouter, OpenAI-compatible API):**
+### 3. Generate the adapter
 
-| Model | Role | Notes | Cost |
-|-------|------|-------|------|
-| `google/gemini-3.6-flash` | high-quality QA | **reasoning is mandatory** → needs generous `max_tokens` (3000–4000) or it returns empty | ~$0.0021 / QA |
-| `google/gemma-4-31b-it` | bulk / cheap QA | non-reasoning, clean JSON | ~$0.001 / repo (~$1 per 1000 repos) |
-
-The OpenRouter key lives in a **git-ignored `.env` (mode 600)** and is never pasted
-into a command line.
-
----
-
-## 7. What is learnable — Tier A / B / C
-
-A LoRA adapter has finite capacity. We classify repo knowledge by whether a LoRA can
-hold it — this drives the entire QA design:
-
-- **Tier A — Judgment & conventions** (LEARNABLE). "This repo puts business logic in
-  services, not views." Compressible, generalizes. → **This is what we train on.**
-- **Tier B — Structural gist** (LEARNABLE). "Auth flows through middleware." The kind
-  of thing, not the exact file.
-- **Tier C — Exact recall & multi-hop** (NOT reliably learnable). "Line 412 of
-  `foo.py` calls `bar()`." This needs **retrieval (RAG)**, not weights.
-
-So Memory-LoRA and RAG are **complementary**: the adapter carries Tier A/B judgment
-for free (zero tokens); RAG handles Tier C exact lookups. The QA prompts forbid exact
-file/line answers precisely so we never ask the LoRA to do a job it structurally
-can't.
-
----
-
-## 8. Datasets inventory
-
-Everything lives under `data/` (git-ignored blobs). Sizes are approximate.
-
-| Path | What | Scale |
-|------|------|-------|
-| `data/real_code2lora/` | **RepoPeftBench** from the `code2lora` HF org — 500 Python repos, repo-commit embeddings + diffs | 73,849 repo-commit rows; ~1.2 GB |
-| `data/commitpack/multilang_commits.jsonl` | **CommitPackFT** shards, 9 languages | 25k+ distinct repos |
-| `data/docs/multiview_sources.jsonl` | 6-view `view_text` per repo (input to QA gen) | growing (1000s of repos) |
-| `data/embeddings/multiview_embeddings.parquet` | 12288-d multi-view repo embeddings | 1000+ repos |
-| `data/embeddings/aligned6_embeddings.parquet` | **assembled training inputs** (repos with ≥1 QA) | 1058 repos (current) |
-| `data/qna/repo_scoped_qa.jsonl` | repo-level judgment QA | 11,232 QA |
-| `data/qna/techlead_qa_commitpack.jsonl` | commit-scoped breadth QA | 9,245 QA |
-| `data/qna/techlead_qa.jsonl` | SWE-bench tech-lead QA | 2,786 QA |
-| `data/qna/aligned6_qna.jsonl` | **assembled training targets** | 8,540 QA (current) |
-| `data/openrouter_cache/` | per-prompt response cache | ~19 MB |
-
-**Language balancing.** SWE-bench is ~79 % Django. Left alone, the dataset was 46 %
-Django. `scripts/consolidate_qa.py` applies a **per-repo cap** (default 12–15 QA/repo)
-which collapses Django to **~2.0 %** while preserving the 2400+ distinct repos'
-diversity. `assemble_6view_dataset.py` applies the same cap when building the final
-aligned set.
-
----
-
-## 9. Experiments & results
-
-Chronological, with the actual numbers we measured. Two families of runs.
-
-### 9.1 Reproducing the paper (single-view, real RepoPeftBench)
-
-| Run | What | Result |
-|-----|------|--------|
-| `full1` (early) | first end-to-end hypernetwork on converted real data | CR **EM 0.056–0.083**, EditSim ~0.27 — pipeline works, undertrained |
-| `sixview`/converted-real (best single-view ckpt) | after more training | **CR EM 0.524, EditSim 0.635** |
-| Paper (reference, H100) | their full run | CR EM **0.638** |
-
-**Headline:** on real code, after only ~2.4 % of one epoch of local MPS training, we
-reached **52.4 % cross-repo Exact Match** vs the paper's 63.8 % on a full H100 setup.
-The mechanism demonstrably works — the generated adapter recovers repo-specific
-identifiers the base model does not know.
-
-### 9.2 The 6-view Tech-Lead model (the current line of work)
-
-Loss is causal-LM cross-entropy on QA targets; lower is better. Three eval suites:
-`cr_val` / `cr_test` (held-out *repos*) and `ir_test` (held-out *QA* of train repos).
-
-| Run | Dataset | Best held-out `cr_test` loss | Notes |
-|-----|---------|------------------------------|-------|
-| `sixview_v1` | 515 repos / 3,988 QA (415 train repos) | **2.848** (step ~1060) | Overfit afterward: train loss fell to 1.75 while `cr_test` drifted to 3.35. Classic small-dataset ceiling. |
-| `sixview_v2` | **1,058 repos / 8,540 QA (858 train repos)** | *in progress* | Resumed from `sixview_v1/head.best.pt`; 2× the data specifically to break v1's ceiling. |
-
-`sixview_v1` metrics trajectory (from `runs/sixview_v1/metrics.jsonl`):
-
-```
-step 1245  cr_test 2.962   ir_test 2.535   (end of epoch 2 — near best)
-step 1400  cr_test 3.245   ir_test 2.593   (overfitting begins)
-step 1600  cr_test 3.352   ir_test 2.655   (train loss still falling → ceiling hit)
+```bash
+python app/engine/generate_and_merge.py \
+    --embedding /tmp/click.npy \
+    --checkpoint runs/all_lora/all_lora_best_cpt.pth \
+    --adapter-out /tmp/click-adapter \
+    --no-merge
 ```
 
-The v1→v2 story is the core empirical lesson: **the small aligned set was the
-bottleneck, not the architecture** — hence the push to build 1000+ more repos.
+That writes a **standard PEFT adapter**. Use `--merged-out DIR` instead of
+`--no-merge` for a self-contained merged model (for vLLM).
 
----
+### 4. Use it
 
-## 10. Key decisions & lessons learned
+```python
+from peft import PeftModel
+from transformers import AutoModelForImageTextToText, AutoTokenizer
 
-The expensive knowledge. Read this section twice.
+tok = AutoTokenizer.from_pretrained("models/gemma-4-E2B")
+model = AutoModelForImageTextToText.from_pretrained("models/gemma-4-E2B")
+model = PeftModel.from_pretrained(model, "/tmp/click-adapter")
 
-### 10.1 ⚠️ The MPS gradient-checkpointing memory leak (the big one)
+prompt = "Q: What testing framework does this repository use?\nA:"
+out = model.generate(**tok(prompt, return_tensors="pt"), max_new_tokens=32)
+print(tok.decode(out[0], skip_special_tokens=True))
+```
 
-**Symptom:** training with `gradient_checkpointing_enable()` (`use_reentrant=False`)
-**leaked ~12 GB per step** and OOM'd the whole machine within a few steps.
+`with model.disable_adapter():` gives the frozen baseline for an A/B — the
+comparison that actually matters.
 
-**Diagnosis** (`scripts/diag_mps_leak.py`): forward-only was stable; train + checkpoint
-leaked 39 GB → 18 GB free in 2 steps. Isolated the checkpointing path as the cause.
+### Calling the head directly
 
-**Fix:** **`--no-gradient-checkpointing`.** We have enough unified memory to hold
-activations without it once the multimodal towers sit idle. This is documented as a
-standing memory (`mps-gradient-checkpointing-leak.md`).
+```python
+import numpy as np, torch, sys
+sys.path.insert(0, "app/engine")
+from generate_and_merge import load_head
 
-### 10.2 ⚠️ `psutil` RSS is blind to MPS memory
+head, cfg, alpha = load_head("runs/all_lora/all_lora_best_cpt.pth")
+emb = np.load("/tmp/click.npy").astype("float32")      # 12288-d, RAW
+out = head(torch.from_numpy(emb).unsqueeze(0))
+# out["A"][type] -> [1, r, in_features]
+# out["B"][type] -> [1, out_features, r]
+```
 
-Our first memory safety-net used `psutil` RSS / `ps -o rss` — it reported **< 1 GB**
-while `top` showed **55–83 GB** actually in use. MPS (GPU) allocations don't show up
-in process RSS.
+The head standardises its input internally using statistics stored in the
+checkpoint — **pass raw embeddings**, do not normalise them yourself.
 
-**Fix:** the safety check uses **`psutil.virtual_memory().available`** (system-wide)
-with a `--min-available-gb` floor (default 5). To *observe* MPS memory, use
-`top -l 1 -pid <PID> -stats mem`, not `ps`.
+Its update is `Δ = (α/r)·(x·Aᵀ)·Bᵀ` with `A:[r,in]`, `B:[out,r]` — identically
+PEFT's LoRA convention, so the output drops straight into a standard adapter.
+One `(A, B)` pair per shape-qualified module type, shared across the transformer
+layers of that shape (205 target modules, 14 types).
 
-### 10.3 Memory competition between concurrent jobs
+---
 
-Three concurrent jobs once pushed available memory under the 10 GB floor and training
-self-stopped. **Lesson:** during MPS training, run data builds/embedding on **CPU**
-(`--device cpu`) so they don't contend for the GPU/unified memory. We now routinely
-run training (MPS) + QA gen (network) + multiview build (CPU) together without
-contention.
+## Results
 
-### 10.4 Don't lose hours of training
+Cross-repo held-out loss versus the **frozen base on identical data**:
 
-Every long run writes **checkpoints every 50 steps** (overwriting `head.latest.pt`),
-**every 30 minutes** (timestamped `head.tNNNNm.pt`), **per-epoch** (`head.epN.pt`),
-and a **`head.best.pt`** on eval improvement. Runs are launched with `nohup … &
-disown` so they survive terminal/session death. `sixview_v1` in fact survived a full
-session interruption and kept training. Resume with `--resume-from <ckpt>` (loads head
-weights; optimizer restarts fresh).
+| checkpoint | `cr_val` | `cr_test` | vs base | verdict |
+|---|---|---|---|---|
+| sixview_v2 *(removed)* | 2.606 | — | **+0.198** | worse than no adapter |
+| `h200_run/head.best.pt` | 2.7168 | 2.7155 | −5.191 | good |
+| **`all_lora_best_cpt.pth`** | **2.6811** | **2.6266** | **−5.246 / −5.310** | **best** |
 
-### 10.5 Data-quality corrections (user-driven)
+Absolute losses are *not* comparable across rows — the eval sets differ. The
+delta against the frozen model is.
 
-- **"I still see lots of Django."** SWE-bench is Django-dominated. → per-repo cap +
-  multi-language sourcing dropped Django 46 % → 2.0 %.
-- **"It must be good for any programming language."** → 9-language diversity via
-  CommitPackFT and language-agnostic view extraction.
-- **"Where's the code context in the QA?"** → clarified the two-channel design: the
-  **repo embedding is the context channel**, the QA is only the target. They are
-  joined by `doc_id` at assembly time.
+Benchmark on three unseen repositories, three task families each (FACT = the
+trained Q&A format, CODE = real source-line completion, TEXT = repo prose):
 
-### 10.6 OpenRouter gotchas
+| repo | FACT | CODE | TEXT | keyword accuracy |
+|---|---|---|---|---|
+| `psf/requests` | −10.31 | −5.38 | −2.83 | 0% → 83% |
+| `pallets/click` | −11.74 | −7.66 | −3.26 | 25% → 75% |
+| `OpenLLM-France/AudioBench` | −5.08 | −3.81 | −1.73 | 0% → 0% |
 
-- `gemini-3.6-flash` **returned empty** until we raised `max_tokens` — it's a
-  mandatory-reasoning model that spends tokens on hidden reasoning before content.
-  Reasoning **cannot be disabled** (400 error).
-- CommitPackFT's HF loader is deprecated → fetch raw `data.jsonl` directly.
-- `global MODEL` after use is a `SyntaxError` → set via `globals()["MODEL"] = ...`.
+**100% win rate on all nine.**
 
-### 10.7 Performance fix worth knowing
+### Limits, stated plainly
 
-Loading embeddings was 5+ min because `_list_to_f32_array` used a Python loop.
-Vectorized via `col.combine_chunks().flatten().to_numpy()` → **~220× faster**.
+- It learns a repo's **stack and conventions**, not what the project *does*. On
+  `requests` it answers "XML library" instead of HTTP. That is exact factual
+  recall, which a rank-16 LoRA structurally cannot hold — retrieval covers it.
+- `AudioBench` keyword accuracy stayed at **0%**: projects whose identity is not
+  inferable from structure transfer poorly.
+- Base losses of 12–16 on short gold targets inflate the deltas. The keyword
+  accuracy and the generations are the trustworthy evidence.
 
 ---
 
-## 11. Repository map
-
-```
-memory_lora/                  # the library (importable package)
-  core.py                     # LoRA wrapper, MemoryLoRAHead hypernetwork,
-                              #   get_module_specs / replace_with_lora / inject_lora_weights,
-                              #   load_doc_rows / load_qna_rows
-  encoder.py                  # Qwen3 chunk + embed + mean/max pool (frozen)
-  codegraph.py                # Python AST extractor (imports, sigs, call graph)
-  data_paths.py               # local parquet/jsonl path resolver
-
-scripts/
-  build_repo_multiview.py     # clone → 6 views → 12288-d embeddings (multi-language, resume-safe)
-  generate_repo_scoped_qa.py  # repo-level judgment QA (aligned to the 6 views)
-  generate_commitpack_qa.py   # commit-scoped breadth QA across 1000s of distinct repos
-  generate_techlead_qa.py     # SWE-bench tech-lead QA
-  generate_synthetic_dataset.py  # original synthetic doc + QA generator
-  consolidate_qa.py           # per-repo cap → language/domain balancing
-  assemble_6view_dataset.py   # join embeddings ↔ all QA by repo → aligned6_{embeddings,qna}
-  augment_paraphrases.py      # QA paraphrase augmentation
-  convert_real_code2lora.py   # RepoPeftBench → our schema
-  build_doc_embeddings.py     # encoder pass over documents
-  merge_corpora.py            # combine multiple corpora
-  train_memory_lora.py        # THE trainer (MPS, one-repo-per-step, checkpoints, TensorBoard)
-  train_direct_lora.py        # baseline: plain per-repo LoRA (no hypernetwork)
-  eval_memory_lora.py         # EM / EditSim recall eval on cr/ir splits
-  show_eval_examples.py       # dump concrete base-vs-adapted examples
-  test_embed_this_repo.py     # embed the current repo (pipeline demo)
-  test_recall_this_repo.py    # query the adapted model about this repo
-  diag_mps_leak.py            # the memory-leak isolation harness
-
-data/                         # git-ignored: embeddings, qna, sources, caches
-runs/                         # git-ignored: checkpoints, logs, metrics.jsonl, tb/
-requirements.txt              # torch 2.13 (MPS), transformers@main, openai, pyarrow, tensorboard…
-.env                          # git-ignored, mode 600: OPENROUTER_API_KEY
-```
+## The failure worth knowing about
 
----
+An earlier checkpoint reached a healthy-looking eval loss of **2.606** while
+being **worse than applying no adapter at all** — and worse than random noise of
+matched magnitude.
 
-## 12. How to run it
+**Cause:** 64% of every repo embedding is a constant vector shared by all
+repositories (the frozen encoder's mean response to "source code"). It dominated
+the trunk, which collapsed to emitting essentially the same adapter for every
+repo.
 
-### Setup
+| stage | mean pairwise cosine across repos |
+|---|---|
+| input embedding | 0.73 |
+| **after trunk** | **0.978** ← discriminability destroyed |
+| emitted adapter | 0.96 |
+| input, centered | **0.00** ← the signal was there all along |
 
-```bash
-python3 -m venv venv && source venv/bin/activate
-pip install -r requirements.txt          # installs transformers from git main
-echo "OPENROUTER_API_KEY=sk-or-..." > .env && chmod 600 .env
-```
+**Fix:** `MemoryLoRAHead.fit_input_stats()` standardises the conditioning input
+with training-set statistics, stored in the checkpoint so training and inference
+apply the same transform. Emitted-adapter cosine went **0.96 → 0.21 in 40 steps**;
+the shipped model sits at **0.32**.
 
-Smoke-test the environment first (gates everything): confirm `transformers` main
-loads `google/gemma-4-E2B` on `mps` and runs a text-only forward pass.
+**Why it went unnoticed:** training logged only the adapted loss. A number like
+2.606 says nothing without the frozen-model baseline beside it. Every eval now
+reports `delta_vs_baseline` and `diag/adapter_cosine`.
 
-### Build data
+Before trusting any checkpoint:
 
 ```bash
-# 1) multi-view embeddings for a repo list (CPU to stay off the GPU during training)
-python scripts/build_repo_multiview.py \
-  --repos-file data/multilang_repo_list.txt --max-repos 1000 --device cpu
-
-# 2) repo-scoped QA (cheap model) — appends, resume-safe, cached
-./venv/bin/python scripts/generate_repo_scoped_qa.py \
-  --model google/gemma-4-31b-it --workers 10
-
-# 3) balance + assemble the aligned training set
-python scripts/consolidate_qa.py --per-repo-cap 12
-python scripts/assemble_6view_dataset.py
-# -> data/embeddings/aligned6_embeddings.parquet + data/qna/aligned6_qna.jsonl
+python app/scripts/diagnose_head.py --job <id> --checkpoints runs/<run>/head.best.pt
 ```
 
-### Train (the exact `sixview_v2` command)
+It scores against `none`, `random` noise of matched scale, and a `zero-B` control
+that must reproduce the baseline *exactly*. A head that cannot beat `random` has
+not learned the mapping.
 
-```bash
-nohup ./venv/bin/python scripts/train_memory_lora.py --output-dir sixview_v2 \
-  --resume-from runs/sixview_v1/head.best.pt \
-  --embeddings-path data/embeddings/aligned6_embeddings.parquet \
-  --qna-path data/qna/aligned6_qna.jsonl --epochs 100 --max-hours 8 \
-  --checkpoint-every-steps 50 --checkpoint-every-minutes 30 --epoch-ckpt-every 5 \
-  --eval-every-steps 300 --limit-eval-docs 40 --max-seq-len 512 --fixed-seq-len \
-  --max-qna-per-doc 12 --lm-micro-batch 2 --device mps --no-gradient-checkpointing \
-  --rank 16 --head-hidden-dim 128 --head-dropout 0.1 --weight-decay 0.05 \
-  --early-stop-patience 25 --lr 8e-5 --lr-total-steps 9000 --min-available-gb 5 \
-  > runs/sixview_v2_train.log 2>&1 &
-disown
-```
-
-**Flags you must not forget:** `--no-gradient-checkpointing` (the leak),
-`--device cpu` for builds during training (contention), `--min-available-gb`
-(system-wide memory floor).
-
-### Watch it
+---
 
-```bash
-tensorboard --logdir runs/sixview_v2/tb    # train/loss, train/lr, eval/{suite}_loss
-tail -f runs/sixview_v2_train.log
-```
+## Training
 
-### Evaluate & inspect
+Reproduce or continue on a single GPU:
 
 ```bash
-python scripts/eval_memory_lora.py --ckpt runs/sixview_v2/head.best.pt   # EM / EditSim
-python scripts/show_eval_examples.py                                     # base vs adapted
+python3 deploy/h200/preflight.py          # validates before spending GPU time
+GATE=1 bash deploy/h200/train_h200.sh     # ~20-min go/no-go
+bash deploy/h200/train_h200.sh            # full run
 ```
 
----
-
-## 13. Evaluation methodology
-
-- **Splits (deterministic, by `md5(repo) % 100`):** 80 % train / 10 % `cr_val` /
-  10 % `cr_test` **by repo**, so cross-repo suites are **entirely held-out
-  repositories** the hypernetwork never trained on. Within train repos, ~15 % of QA
-  is held out → `ir_test` (in-repo generalization to unseen questions of seen repos).
-- **Metrics:** causal-LM **eval loss** during training (fast, every N steps on
-  `--limit-eval-docs` docs to stay cheap on CPU), plus generation-time **Exact Match
-  (EM)** and **EditSim** for the recall eval.
-- **The proof spot-check:** query the *adapted* model with repo-specific questions and
-  confirm the *base* (un-adapted) model gets them wrong/vague — proving the
-  **adapter**, not the base model's pretraining, does the work.
-
-CPU eval of a float32 5B model is slow (~20 min for a full pass) → we cap eval docs
-(e.g. 10–40) for in-loop evals and run full EM eval separately.
+`preflight.py` fails fast on git-LFS pointers masquerading as data,
+pre-standardised embeddings, a head without `fit_input_stats`, and missing eval
+splits — then auto-tunes the largest micro-batch that fits.
 
 ---
 
-## 14. Costs & budget discipline
-
-- **Spend baseline:** $31.00 (`runs/spend_baseline.txt`); ~**$33.90 total** to date;
-  ~**$11 remaining**. The project is run under explicit budget caps ("spend at most
-  $4 more") with spend-guards.
-- **Unit economics:** `gemini-3.6-flash` ≈ **$0.0021/QA**; `gemma-4-31b-it` ≈
-  **$0.001/repo (~$1 per 1000 repos)** — which is exactly why the 1000-repo expansion
-  uses the gemma model.
-- **Free levers:** the per-prompt cache makes reruns free; embedding and training are
-  local (electricity only).
-
----
+## Credits
 
-## 15. Current status & roadmap
-
-**Live right now (three jobs in parallel, no contention):**
-
-- **`sixview_v2` training** — resumed from `head.best.pt` on the doubled
-  **1,058-repo / 8,540-QA** dataset (858 steps/epoch), MPS. First held-out eval at
-  step 300 tells us whether doubling the data broke v1's 2.848 ceiling.
-- **QA generation** — `gemma-4-31b-it` filling in all ~1,032 new repos for the
-  **complete** dataset (next training run).
-- **Multiview build** — cloning/embedding toward the full +1,000-new target (CPU).
-
-**Roadmap:**
-
-1. Finish the complete 1000-new-repo dataset (embeddings + QA).
-2. Assemble the full aligned set (~1,650 repos) and train **`sixview_v3`** on it.
-3. Run generation-time **EM/EditSim** on the 6-view model (base vs adapted).
-4. Push `head_hidden_dim` back up once on real GPUs; the 128 default was an
-   MPS-locality compromise.
-5. Broaden Tier-A/B QA toward agent-harness use cases (Jira/ticket tracking,
-   diff/impact reasoning) already scaffolded in `generate_techlead_qa.py`.
-
-**Open questions:**
-
-- Does the 12288-d 6-view embedding actually beat the single 2048-d view on
-  generation EM, or only on loss? (loss says yes; EM eval pending)
-- What's the real Tier-B ceiling — how much structural gist fits in rank-16?
-- Optimal per-repo QA cap for the breadth/depth trade-off.
-
----
-
-## 16. Glossary
-
-- **Hypernetwork** — a network that outputs the weights of another network. Here:
-  repo embedding → LoRA (A,B) matrices.
-- **LoRA** — Low-Rank Adaptation: `y = Wx + scaling · B(Ax)`, with `A,B` low-rank
-  (rank 16). We *generate* A,B instead of training them per-repo.
-- **6 views** — graph / arch / history / contracts / conventions / ops; each 2048-d,
-  concatenated to 12288-d.
-- **CR / IR** — cross-repo (held-out repos) / in-repo (held-out QA of seen repos).
-- **EM / EditSim** — Exact Match / edit-distance similarity of generated vs gold.
-- **Tier A/B/C** — judgment (learnable) / structural gist (learnable) / exact recall
-  (needs RAG).
-- **MPS** — Apple's Metal Performance Shaders GPU backend for PyTorch.
-- **RepoPeftBench** — the paper's benchmark; 500 Python repos, repo-commit
-  embeddings + diffs, IR/CR splits.
-
----
+Reimplements and extends **Code2LoRA** (arXiv 2606.06492) — a static hypernetwork
+mapping a repository embedding to a LoRA adapter — retargeted to
+`google/gemma-4-E2B`, extended from single-view code completion to a six-view
+representation, and wrapped in a serving stack that speaks the OpenAI and
+Anthropic APIs. Training data includes RepoPeftBench from that work.
 
-*Maintained as living documentation. If you change a default, a path, or a flag,
-update the matching section here — onboarding depends on it.*
+Base model: [`google/gemma-4-E2B`](https://huggingface.co/google/gemma-4-E2B) ·
+Encoder: [`Qwen/Qwen3-Embedding-0.6B`](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B)

docs/DEEP_DIVE.md

@@ -0,0 +1,581 @@
+# Memory-LoRA: A Hypernetwork that Writes Repo-Specific Adapters for Gemma-4-E2B
+
+> **One line:** we train a small neural network (a *hypernetwork*) that reads an
+> embedding of a codebase and **emits a LoRA adapter** for a frozen
+> `google/gemma-4-E2B`. The adapter injects repo-specific knowledge into the
+> model with **zero extra tokens at inference time** — no RAG, no context stuffing.
+> Everything runs **locally on an Apple-Silicon Mac** (M4 Pro, 64 GB).
+
+This document is the onboarding bible for the project. It covers *what* we built,
+*why* every non-obvious choice was made, *what data* we used, *every experiment we
+ran and its result*, and the **hard-won lessons** (the MPS memory leak alone cost
+us hours). Read it top to bottom once; after that use it as a reference.
+
+---
+
+## Table of Contents
+
+1. [The idea in 60 seconds](#1-the-idea-in-60-seconds)
+2. [Origin: the Code2LoRA paper](#2-origin-the-code2lora-paper)
+3. [Architecture](#3-architecture)
+4. [The target model: Gemma-4-E2B specifics](#4-the-target-model-gemma-4-e2b-specifics)
+5. [Data pipeline: the 6 views](#5-data-pipeline-the-6-views)
+6. [Data pipeline: QA generation](#6-data-pipeline-qa-generation)
+7. [What is learnable — Tier A / B / C](#7-what-is-learnable--tier-a--b--c)
+8. [Datasets inventory](#8-datasets-inventory)
+9. [Experiments & results](#9-experiments--results)
+10. [Key decisions & lessons learned](#10-key-decisions--lessons-learned)
+11. [Repository map](#11-repository-map)
+12. [How to run it](#12-how-to-run-it)
+13. [Evaluation methodology](#13-evaluation-methodology)
+14. [Costs & budget discipline](#14-costs--budget-discipline)
+15. [Current status & roadmap](#15-current-status--roadmap)
+16. [Glossary](#16-glossary)
+
+---
+
+## 1. The idea in 60 seconds
+
+A coding agent (Claude Code, Codex, etc.) is great at reasoning but knows nothing
+about *your* repo until you paste files into its context — which is slow, expensive,
+and capped by the context window. The usual fix is RAG (retrieve chunks at query
+time). We do something different and complementary:
+
+**We bake the repo's "personality" directly into the model's weights, once, as a
+LoRA adapter — and we generate that adapter with a neural network instead of
+training it.**
+
+```
+                        ┌─────────────────────────────┐
+   repo on disk ──────► │  6-view extractor + Qwen3    │ ──► 12288-d
+                        │  frozen embedding encoder    │     repo embedding
+                        └─────────────────────────────┘          │
+                                                                 ▼
+                                                   ┌──────────────────────────┐
+                                                   │  HYPERNETWORK (our head) │
+                                                   │  MLP → per-module A,B     │
+                                                   └──────────────────────────┘
+                                                                 │  LoRA weights
+                                                                 ▼
+   Q: "what layer owns auth in this repo?"  ──►  ┌──────────────────────────────┐
+                                                 │ FROZEN Gemma-4-E2B + injected │ ──► "the middleware
+                                                 │ LoRA  (zero extra tokens)     │      layer, via ..."
+                                                 └──────────────────────────────┘
+```
+
+The magic: **the hypernetwork is trained across hundreds of repos**, so it learns
+the *mapping* `repo embedding → good adapter`. At inference on a brand-new repo it
+has never seen, it embeds the repo once and produces an adapter in a single forward
+pass. This is the same reason the source paper needed 400+ repos, not 1: breadth is
+what makes the mapping generalize.
+
+---
+
+## 2. Origin: the Code2LoRA paper
+
+We reverse-engineered **Code2LoRA** (arXiv 2606.06492v1) and found its released
+code (`anonymous.4open.science/r/code2lora-6857`, MIT). The paper's contribution:
+a *static hypernetwork* that maps a **repository** embedding → a LoRA adapter for a
+frozen code LLM, evaluated on **RepoPeftBench** with IR (in-repo) / CR (cross-repo)
+splits. On a full H100 setup they report **63.8 % cross-repo Exact Match**.
+
+Our project is the **Doc2LoRA variant** the paper itself cites — hypernetwork maps a
+*document/repo view* → LoRA — reimplemented against **Gemma-4-E2B**, trained fully
+**locally on MPS** (no CUDA/H100), and extended in two directions the paper does not
+cover:
+
+- **Memory / recall**: the adapter should let the model *recall facts* about the
+  repo, not just complete code.
+- **Tech-Lead judgment**: architecture, data-flow, conventions, contracts, ops —
+  the things a 20-year senior engineer "just knows" about a codebase.
+
+We keep the paper's proven autograd trick almost verbatim (see §3) and change only
+what Gemma-4 and Apple Silicon force us to change.
+
+---
+
+## 3. Architecture
+
+Three frozen/learned pieces. Only the middle one (the head) is trained.
+
+### 3.1 Frozen encoder — `memory_lora/encoder.py`
+
+- **Qwen3-Embedding-0.6B**, frozen, no gradient flows through it.
+- Each repo view is chunked into token windows (2048 tokens, 128 overlap), each
+  chunk mean-pooled, then chunks combined with **mean + max pooling** → a **2048-d**
+  vector *per view*.
+- Embeddings are **precomputed once and cached** to parquet — the encoder never runs
+  during training.
+
+### 3.2 The hypernetwork head — `memory_lora/core.py :: MemoryLoRAHead`
+
+The only trained component. Design (kept close to the paper):
+
+- **2-layer GELU MLP trunk** (`input_dim → hidden_dim → hidden_dim`), followed by
+  **L2-normalize + √hidden_dim rescale** (stabilizes the magnitude of generated
+  weights).
+- **Per-module-type output heads**: for each target module *type* it emits an
+  `A ∈ [rank, in_features]` and `B ∈ [out_features, rank]`. **One (A,B) pair per
+  type, shared across all layers of that type** — this is what keeps the head
+  tractable (188.6 M params) instead of exploding per-layer.
+- **Squashing**: `tanh(raw) * exp(log_scale)` with a learned per-type `log_scale`
+  (init **-3.5**). This starts the generated adapter near-zero (so training begins
+  close to the base model) and lets each type learn its own output scale.
+- **Defaults**: `hidden_dim=128`, `rank=16`, `dropout=0.1`.
+  - *Why hidden_dim=128 and not the paper's 512/1024?* A 745 M-param head
+    (hidden_dim=512) barely moved eval loss (~1.9 → ~2.7) but was far heavier to
+    train on MPS. 128 cuts head size dramatically with negligible quality loss
+    locally. Bump it later on real GPUs.
+
+### 3.3 The LoRA injection — `memory_lora/core.py :: LoRA`
+
+```
+   base nn.Linear (FROZEN)              hypernetwork output
+        │                                    │
+   x ──►│ Wx  ────────────────┐              │
+        │ (input detached      +──► y = Wx + scaling · B (A x)
+        │  into base)          │              ▲   ▲
+   x ──────────────────────────┘              │   │
+                                     A,B are NON-detached tensors so
+                                     autograd flows LM-loss → head
+```
+
+Critical detail (straight from the paper's code): **A and B are plain, non-buffer
+tensor attributes, not `nn.Parameter` and not detached**, so the gradient of the LM
+loss flows *through* the injected weights *into the hypernetwork*. The base
+`nn.Linear` is frozen and its input is detached. Get this wrong and the head never
+learns.
+
+### 3.4 Shape-qualified module types — the heterogeneity fix
+
+Gemma-4-E2B is **not** a uniform stack (see §4). Two `q_proj`s can have different
+shapes. If you key the head by bare type name (`q_proj`) you get
+*"type q_proj inconsistent dims"* crashes. Fix: key by **shape-qualified type**,
+e.g. `q_proj_1536x2048` vs `q_proj_1536x4096`. The v2 run discovered **14 shape
+types** across **205 target modules**:
+
+```
+down_proj_12288x1536  down_proj_6144x1536   gate_proj_1536x12288  gate_proj_1536x6144
+k_proj_1536x256       k_proj_1536x512       o_proj_2048x1536      o_proj_4096x1536
+q_proj_1536x2048      q_proj_1536x4096      up_proj_1536x12288    up_proj_1536x6144
+v_proj_1536x256       v_proj_1536x512
+```
+
+`get_module_specs(root_prefix="model.language_model.")` restricts wrapping to the
+text decoder — the **vision and audio towers are never touched** (not even
+inspected), so the multimodal forward path stays intact and they cost only idle RAM.
+
+---
+
+## 4. The target model: Gemma-4-E2B specifics
+
+Verified by reading the actual safetensors header, not guessed:
+
+- **Real model.** Google shipped Gemma 4 in March 2026. Apache-2.0, ungated.
+  Class `Gemma4ForConditionalGeneration`, `model_type: "gemma4"`. Loaded via
+  `AutoModelForImageTextToText`.
+- **Requires `transformers >= 5.5.0.dev0`** — install from the `main` branch, not a
+  pinned PyPI release (this is the single biggest environment risk; smoke-test first).
+- **Decoder is nested**: layers live at `model.language_model.layers.{i}.*`, *not*
+  `model.layers.*`. The layer-index regex had to change accordingly.
+- **35 text layers, heterogeneous:**
+  - Aggressive KV sharing — **20 of 35 layers lack their own `k_proj`/`v_proj`**
+    (`num_kv_shared_layers=20`).
+  - **Every 5th layer is wider** (the `*_4096`, `*_12288` shape variants above).
+- **Device `mps`, precision bf16** (fall back to fp16 if unstable). No
+  `flash_attention_2` on MPS — use `sdpa`, fall back to `eager`.
+
+---
+
+## 5. Data pipeline: the 6 views
+
+`scripts/build_repo_multiview.py` clones a repo and extracts **6 complementary
+views**, embeds each with Qwen3 → 2048-d, and **concatenates to a 12288-d** repo
+vector. The views encode the different "lenses" a senior engineer uses:
+
+| View            | What it captures                                   | Source signals |
+|-----------------|----------------------------------------------------|----------------|
+| `v_graph`       | call / import / dependency structure               | AST for Python (`memory_lora/codegraph.py`); `IMPORT_RE`/`DEF_RE` regex fallback for other languages |
+| `v_arch`        | architecture & layout                              | README, folder tree |
+| `v_history`     | how the code evolved                               | `git log`, recent diffs |
+| `v_contracts`   | behavioral contracts                               | test files |
+| `v_conventions` | idioms & style                                     | representative source files |
+| `v_ops`         | build / deploy / runtime                           | CI config, Dockerfile, build files |
+
+**Multi-language from the start.** `CODE_EXTS` + regex fallbacks mean the graph view
+works for 9 languages, not just Python (this was a deliberate correction — see §10).
+Repos with < 3 code files are skipped. The build is **resume-safe** (skips repos
+already in `multiview_sources.jsonl`) and **flushes the embeddings parquet every 10
+repos**, so a crash never loses more than 10 repos of work.
+
+---
+
+## 6. Data pipeline: QA generation
+
+The repo embedding is the *input*; the *target* is repo-scoped Q&A. Two generators,
+both parallelized (`ThreadPoolExecutor`, `--workers 10`) with a **per-prompt disk
+cache** (idempotent reruns) and a `--model` flag:
+
+- **`scripts/generate_repo_scoped_qa.py`** — reads the same 6 views and asks the LLM
+  for **8–12 repo-level judgment questions** ("what layer owns X", "what convention
+  does this repo use for Y", "how does data flow through Z", "why is it structured
+  this way"). Target scope = input scope (repo-level embedding ↔ repo-level QA).
+- **`scripts/generate_commitpack_qa.py`** — **breadth** generator: one commit per
+  *distinct* repo across CommitPackFT (25k+ distinct repos), 3–4 commit-scoped
+  judgment questions (why / conventions / contracts / impact). For a hypernetwork,
+  **distinct-repo count is the currency of generalization**, so we favor 1 commit ×
+  many repos over many commits × one repo.
+
+**Discipline (both):** answers are **short judgment**, never file-path/line-number
+lists. This is deliberate — see Tier A/B/C next.
+
+**Models used (OpenRouter, OpenAI-compatible API):**
+
+| Model | Role | Notes | Cost |
+|-------|------|-------|------|
+| `google/gemini-3.6-flash` | high-quality QA | **reasoning is mandatory** → needs generous `max_tokens` (3000–4000) or it returns empty | ~$0.0021 / QA |
+| `google/gemma-4-31b-it` | bulk / cheap QA | non-reasoning, clean JSON | ~$0.001 / repo (~$1 per 1000 repos) |
+
+The OpenRouter key lives in a **git-ignored `.env` (mode 600)** and is never pasted
+into a command line.
+
+---
+
+## 7. What is learnable — Tier A / B / C
+
+A LoRA adapter has finite capacity. We classify repo knowledge by whether a LoRA can
+hold it — this drives the entire QA design:
+
+- **Tier A — Judgment & conventions** (LEARNABLE). "This repo puts business logic in
+  services, not views." Compressible, generalizes. → **This is what we train on.**
+- **Tier B — Structural gist** (LEARNABLE). "Auth flows through middleware." The kind
+  of thing, not the exact file.
+- **Tier C — Exact recall & multi-hop** (NOT reliably learnable). "Line 412 of
+  `foo.py` calls `bar()`." This needs **retrieval (RAG)**, not weights.
+
+So Memory-LoRA and RAG are **complementary**: the adapter carries Tier A/B judgment
+for free (zero tokens); RAG handles Tier C exact lookups. The QA prompts forbid exact
+file/line answers precisely so we never ask the LoRA to do a job it structurally
+can't.
+
+---
+
+## 8. Datasets inventory
+
+Everything lives under `data/` (git-ignored blobs). Sizes are approximate.
+
+| Path | What | Scale |
+|------|------|-------|
+| `data/real_code2lora/` | **RepoPeftBench** from the `code2lora` HF org — 500 Python repos, repo-commit embeddings + diffs | 73,849 repo-commit rows; ~1.2 GB |
+| `data/commitpack/multilang_commits.jsonl` | **CommitPackFT** shards, 9 languages | 25k+ distinct repos |
+| `data/docs/multiview_sources.jsonl` | 6-view `view_text` per repo (input to QA gen) | growing (1000s of repos) |
+| `data/embeddings/multiview_embeddings.parquet` | 12288-d multi-view repo embeddings | 1000+ repos |
+| `data/embeddings/aligned6_embeddings.parquet` | **assembled training inputs** (repos with ≥1 QA) | 1058 repos (current) |
+| `data/qna/repo_scoped_qa.jsonl` | repo-level judgment QA | 11,232 QA |
+| `data/qna/techlead_qa_commitpack.jsonl` | commit-scoped breadth QA | 9,245 QA |
+| `data/qna/techlead_qa.jsonl` | SWE-bench tech-lead QA | 2,786 QA |
+| `data/qna/aligned6_qna.jsonl` | **assembled training targets** | 8,540 QA (current) |
+| `data/openrouter_cache/` | per-prompt response cache | ~19 MB |
+
+**Language balancing.** SWE-bench is ~79 % Django. Left alone, the dataset was 46 %
+Django. `scripts/consolidate_qa.py` applies a **per-repo cap** (default 12–15 QA/repo)
+which collapses Django to **~2.0 %** while preserving the 2400+ distinct repos'
+diversity. `assemble_6view_dataset.py` applies the same cap when building the final
+aligned set.
+
+---
+
+## 9. Experiments & results
+
+Chronological, with the actual numbers we measured. Two families of runs.
+
+### 9.1 Reproducing the paper (single-view, real RepoPeftBench)
+
+| Run | What | Result |
+|-----|------|--------|
+| `full1` (early) | first end-to-end hypernetwork on converted real data | CR **EM 0.056–0.083**, EditSim ~0.27 — pipeline works, undertrained |
+| `sixview`/converted-real (best single-view ckpt) | after more training | **CR EM 0.524, EditSim 0.635** |
+| Paper (reference, H100) | their full run | CR EM **0.638** |
+
+**Headline:** on real code, after only ~2.4 % of one epoch of local MPS training, we
+reached **52.4 % cross-repo Exact Match** vs the paper's 63.8 % on a full H100 setup.
+The mechanism demonstrably works — the generated adapter recovers repo-specific
+identifiers the base model does not know.
+
+### 9.2 The 6-view Tech-Lead model (the current line of work)
+
+Loss is causal-LM cross-entropy on QA targets; lower is better. Three eval suites:
+`cr_val` / `cr_test` (held-out *repos*) and `ir_test` (held-out *QA* of train repos).
+
+| Run | Dataset | Best held-out `cr_test` loss | Notes |
+|-----|---------|------------------------------|-------|
+| `sixview_v1` | 515 repos / 3,988 QA (415 train repos) | **2.848** (step ~1060) | Overfit afterward: train loss fell to 1.75 while `cr_test` drifted to 3.35. Classic small-dataset ceiling. |
+| `sixview_v2` | **1,058 repos / 8,540 QA (858 train repos)** | *in progress* | Resumed from `sixview_v1/head.best.pt`; 2× the data specifically to break v1's ceiling. |
+
+`sixview_v1` metrics trajectory (from `runs/sixview_v1/metrics.jsonl`):
+
+```
+step 1245  cr_test 2.962   ir_test 2.535   (end of epoch 2 — near best)
+step 1400  cr_test 3.245   ir_test 2.593   (overfitting begins)
+step 1600  cr_test 3.352   ir_test 2.655   (train loss still falling → ceiling hit)
+```
+
+The v1→v2 story is the core empirical lesson: **the small aligned set was the
+bottleneck, not the architecture** — hence the push to build 1000+ more repos.
+
+---
+
+## 10. Key decisions & lessons learned
+
+The expensive knowledge. Read this section twice.
+
+### 10.1 ⚠️ The MPS gradient-checkpointing memory leak (the big one)
+
+**Symptom:** training with `gradient_checkpointing_enable()` (`use_reentrant=False`)
+**leaked ~12 GB per step** and OOM'd the whole machine within a few steps.
+
+**Diagnosis** (`scripts/diag_mps_leak.py`): forward-only was stable; train + checkpoint
+leaked 39 GB → 18 GB free in 2 steps. Isolated the checkpointing path as the cause.
+
+**Fix:** **`--no-gradient-checkpointing`.** We have enough unified memory to hold
+activations without it once the multimodal towers sit idle. This is documented as a
+standing memory (`mps-gradient-checkpointing-leak.md`).
+
+### 10.2 ⚠️ `psutil` RSS is blind to MPS memory
+
+Our first memory safety-net used `psutil` RSS / `ps -o rss` — it reported **< 1 GB**
+while `top` showed **55–83 GB** actually in use. MPS (GPU) allocations don't show up
+in process RSS.
+
+**Fix:** the safety check uses **`psutil.virtual_memory().available`** (system-wide)
+with a `--min-available-gb` floor (default 5). To *observe* MPS memory, use
+`top -l 1 -pid <PID> -stats mem`, not `ps`.
+
+### 10.3 Memory competition between concurrent jobs
+
+Three concurrent jobs once pushed available memory under the 10 GB floor and training
+self-stopped. **Lesson:** during MPS training, run data builds/embedding on **CPU**
+(`--device cpu`) so they don't contend for the GPU/unified memory. We now routinely
+run training (MPS) + QA gen (network) + multiview build (CPU) together without
+contention.
+
+### 10.4 Don't lose hours of training
+
+Every long run writes **checkpoints every 50 steps** (overwriting `head.latest.pt`),
+**every 30 minutes** (timestamped `head.tNNNNm.pt`), **per-epoch** (`head.epN.pt`),
+and a **`head.best.pt`** on eval improvement. Runs are launched with `nohup … &
+disown` so they survive terminal/session death. `sixview_v1` in fact survived a full
+session interruption and kept training. Resume with `--resume-from <ckpt>` (loads head
+weights; optimizer restarts fresh).
+
+### 10.5 Data-quality corrections (user-driven)
+
+- **"I still see lots of Django."** SWE-bench is Django-dominated. → per-repo cap +
+  multi-language sourcing dropped Django 46 % → 2.0 %.
+- **"It must be good for any programming language."** → 9-language diversity via
+  CommitPackFT and language-agnostic view extraction.
+- **"Where's the code context in the QA?"** → clarified the two-channel design: the
+  **repo embedding is the context channel**, the QA is only the target. They are
+  joined by `doc_id` at assembly time.
+
+### 10.6 OpenRouter gotchas
+
+- `gemini-3.6-flash` **returned empty** until we raised `max_tokens` — it's a
+  mandatory-reasoning model that spends tokens on hidden reasoning before content.
+  Reasoning **cannot be disabled** (400 error).
+- CommitPackFT's HF loader is deprecated → fetch raw `data.jsonl` directly.
+- `global MODEL` after use is a `SyntaxError` → set via `globals()["MODEL"] = ...`.
+
+### 10.7 Performance fix worth knowing
+
+Loading embeddings was 5+ min because `_list_to_f32_array` used a Python loop.
+Vectorized via `col.combine_chunks().flatten().to_numpy()` → **~220× faster**.
+
+---
+
+## 11. Repository map
+
+```
+memory_lora/                  # the library (importable package)
+  core.py                     # LoRA wrapper, MemoryLoRAHead hypernetwork,
+                              #   get_module_specs / replace_with_lora / inject_lora_weights,
+                              #   load_doc_rows / load_qna_rows
+  encoder.py                  # Qwen3 chunk + embed + mean/max pool (frozen)
+  codegraph.py                # Python AST extractor (imports, sigs, call graph)
+  data_paths.py               # local parquet/jsonl path resolver
+
+scripts/
+  build_repo_multiview.py     # clone → 6 views → 12288-d embeddings (multi-language, resume-safe)
+  generate_repo_scoped_qa.py  # repo-level judgment QA (aligned to the 6 views)
+  generate_commitpack_qa.py   # commit-scoped breadth QA across 1000s of distinct repos
+  generate_techlead_qa.py     # SWE-bench tech-lead QA
+  generate_synthetic_dataset.py  # original synthetic doc + QA generator
+  consolidate_qa.py           # per-repo cap → language/domain balancing
+  assemble_6view_dataset.py   # join embeddings ↔ all QA by repo → aligned6_{embeddings,qna}
+  augment_paraphrases.py      # QA paraphrase augmentation
+  convert_real_code2lora.py   # RepoPeftBench → our schema
+  build_doc_embeddings.py     # encoder pass over documents
+  merge_corpora.py            # combine multiple corpora
+  train_memory_lora.py        # THE trainer (MPS, one-repo-per-step, checkpoints, TensorBoard)
+  train_direct_lora.py        # baseline: plain per-repo LoRA (no hypernetwork)
+  eval_memory_lora.py         # EM / EditSim recall eval on cr/ir splits
+  show_eval_examples.py       # dump concrete base-vs-adapted examples
+  test_embed_this_repo.py     # embed the current repo (pipeline demo)
+  test_recall_this_repo.py    # query the adapted model about this repo
+  diag_mps_leak.py            # the memory-leak isolation harness
+
+data/                         # git-ignored: embeddings, qna, sources, caches
+runs/                         # git-ignored: checkpoints, logs, metrics.jsonl, tb/
+requirements.txt              # torch 2.13 (MPS), transformers@main, openai, pyarrow, tensorboard…
+.env                          # git-ignored, mode 600: OPENROUTER_API_KEY
+```
+
+---
+
+## 12. How to run it
+
+### Setup
+
+```bash
+python3 -m venv venv && source venv/bin/activate
+pip install -r requirements.txt          # installs transformers from git main
+echo "OPENROUTER_API_KEY=sk-or-..." > .env && chmod 600 .env
+```
+
+Smoke-test the environment first (gates everything): confirm `transformers` main
+loads `google/gemma-4-E2B` on `mps` and runs a text-only forward pass.
+
+### Build data
+
+```bash
+# 1) multi-view embeddings for a repo list (CPU to stay off the GPU during training)
+python scripts/build_repo_multiview.py \
+  --repos-file data/multilang_repo_list.txt --max-repos 1000 --device cpu
+
+# 2) repo-scoped QA (cheap model) — appends, resume-safe, cached
+./venv/bin/python scripts/generate_repo_scoped_qa.py \
+  --model google/gemma-4-31b-it --workers 10
+
+# 3) balance + assemble the aligned training set
+python scripts/consolidate_qa.py --per-repo-cap 12
+python scripts/assemble_6view_dataset.py
+# -> data/embeddings/aligned6_embeddings.parquet + data/qna/aligned6_qna.jsonl
+```
+
+### Train (the exact `sixview_v2` command)
+
+```bash
+nohup ./venv/bin/python scripts/train_memory_lora.py --output-dir sixview_v2 \
+  --resume-from runs/sixview_v1/head.best.pt \
+  --embeddings-path data/embeddings/aligned6_embeddings.parquet \
+  --qna-path data/qna/aligned6_qna.jsonl --epochs 100 --max-hours 8 \
+  --checkpoint-every-steps 50 --checkpoint-every-minutes 30 --epoch-ckpt-every 5 \
+  --eval-every-steps 300 --limit-eval-docs 40 --max-seq-len 512 --fixed-seq-len \
+  --max-qna-per-doc 12 --lm-micro-batch 2 --device mps --no-gradient-checkpointing \
+  --rank 16 --head-hidden-dim 128 --head-dropout 0.1 --weight-decay 0.05 \
+  --early-stop-patience 25 --lr 8e-5 --lr-total-steps 9000 --min-available-gb 5 \
+  > runs/sixview_v2_train.log 2>&1 &
+disown
+```
+
+**Flags you must not forget:** `--no-gradient-checkpointing` (the leak),
+`--device cpu` for builds during training (contention), `--min-available-gb`
+(system-wide memory floor).
+
+### Watch it
+
+```bash
+tensorboard --logdir runs/sixview_v2/tb    # train/loss, train/lr, eval/{suite}_loss
+tail -f runs/sixview_v2_train.log
+```
+
+### Evaluate & inspect
+
+```bash
+python scripts/eval_memory_lora.py --ckpt runs/sixview_v2/head.best.pt   # EM / EditSim
+python scripts/show_eval_examples.py                                     # base vs adapted
+```
+
+---
+
+## 13. Evaluation methodology
+
+- **Splits (deterministic, by `md5(repo) % 100`):** 80 % train / 10 % `cr_val` /
+  10 % `cr_test` **by repo**, so cross-repo suites are **entirely held-out
+  repositories** the hypernetwork never trained on. Within train repos, ~15 % of QA
+  is held out → `ir_test` (in-repo generalization to unseen questions of seen repos).
+- **Metrics:** causal-LM **eval loss** during training (fast, every N steps on
+  `--limit-eval-docs` docs to stay cheap on CPU), plus generation-time **Exact Match
+  (EM)** and **EditSim** for the recall eval.
+- **The proof spot-check:** query the *adapted* model with repo-specific questions and
+  confirm the *base* (un-adapted) model gets them wrong/vague — proving the
+  **adapter**, not the base model's pretraining, does the work.
+
+CPU eval of a float32 5B model is slow (~20 min for a full pass) → we cap eval docs
+(e.g. 10–40) for in-loop evals and run full EM eval separately.
+
+---
+
+## 14. Costs & budget discipline
+
+- **Spend baseline:** $31.00 (`runs/spend_baseline.txt`); ~**$33.90 total** to date;
+  ~**$11 remaining**. The project is run under explicit budget caps ("spend at most
+  $4 more") with spend-guards.
+- **Unit economics:** `gemini-3.6-flash` ≈ **$0.0021/QA**; `gemma-4-31b-it` ≈
+  **$0.001/repo (~$1 per 1000 repos)** — which is exactly why the 1000-repo expansion
+  uses the gemma model.
+- **Free levers:** the per-prompt cache makes reruns free; embedding and training are
+  local (electricity only).
+
+---
+
+## 15. Current status & roadmap
+
+**Live right now (three jobs in parallel, no contention):**
+
+- **`sixview_v2` training** — resumed from `head.best.pt` on the doubled
+  **1,058-repo / 8,540-QA** dataset (858 steps/epoch), MPS. First held-out eval at
+  step 300 tells us whether doubling the data broke v1's 2.848 ceiling.
+- **QA generation** — `gemma-4-31b-it` filling in all ~1,032 new repos for the
+  **complete** dataset (next training run).
+- **Multiview build** — cloning/embedding toward the full +1,000-new target (CPU).
+
+**Roadmap:**
+
+1. Finish the complete 1000-new-repo dataset (embeddings + QA).
+2. Assemble the full aligned set (~1,650 repos) and train **`sixview_v3`** on it.
+3. Run generation-time **EM/EditSim** on the 6-view model (base vs adapted).
+4. Push `head_hidden_dim` back up once on real GPUs; the 128 default was an
+   MPS-locality compromise.
+5. Broaden Tier-A/B QA toward agent-harness use cases (Jira/ticket tracking,
+   diff/impact reasoning) already scaffolded in `generate_techlead_qa.py`.
+
+**Open questions:**
+
+- Does the 12288-d 6-view embedding actually beat the single 2048-d view on
+  generation EM, or only on loss? (loss says yes; EM eval pending)
+- What's the real Tier-B ceiling — how much structural gist fits in rank-16?
+- Optimal per-repo QA cap for the breadth/depth trade-off.
+
+---
+
+## 16. Glossary
+
+- **Hypernetwork** — a network that outputs the weights of another network. Here:
+  repo embedding → LoRA (A,B) matrices.
+- **LoRA** — Low-Rank Adaptation: `y = Wx + scaling · B(Ax)`, with `A,B` low-rank
+  (rank 16). We *generate* A,B instead of training them per-repo.
+- **6 views** — graph / arch / history / contracts / conventions / ops; each 2048-d,
+  concatenated to 12288-d.
+- **CR / IR** — cross-repo (held-out repos) / in-repo (held-out QA of seen repos).
+- **EM / EditSim** — Exact Match / edit-distance similarity of generated vs gold.
+- **Tier A/B/C** — judgment (learnable) / structural gist (learnable) / exact recall
+  (needs RAG).
+- **MPS** — Apple's Metal Performance Shaders GPU backend for PyTorch.
+- **RepoPeftBench** — the paper's benchmark; 500 Python repos, repo-commit
+  embeddings + diffs, IR/CR splits.
+
+---
+
+*Maintained as living documentation. If you change a default, a path, or a flag,
+update the matching section here — onboarding depends on it.*