# Memory-LoRA: A Hypernetwork that Writes Repo-Specific Adapters for Gemma-4-E2B > **One line:** we train a small neural network (a *hypernetwork*) that reads an > embedding of a codebase and **emits a LoRA adapter** for a frozen > `google/gemma-4-E2B`. The adapter injects repo-specific knowledge into the > model with **zero extra tokens at inference time** — no RAG, no context stuffing. > Everything runs **locally on an Apple-Silicon Mac** (M4 Pro, 64 GB). This document is the onboarding bible for the project. It covers *what* we built, *why* every non-obvious choice was made, *what data* we used, *every experiment we ran and its result*, and the **hard-won lessons** (the MPS memory leak alone cost us hours). Read it top to bottom once; after that use it as a reference. --- ## Table of Contents 1. [The idea in 60 seconds](#1-the-idea-in-60-seconds) 2. [Origin: the Code2LoRA paper](#2-origin-the-code2lora-paper) 3. [Architecture](#3-architecture) 4. [The target model: Gemma-4-E2B specifics](#4-the-target-model-gemma-4-e2b-specifics) 5. [Data pipeline: the 6 views](#5-data-pipeline-the-6-views) 6. [Data pipeline: QA generation](#6-data-pipeline-qa-generation) 7. [What is learnable — Tier A / B / C](#7-what-is-learnable--tier-a--b--c) 8. [Datasets inventory](#8-datasets-inventory) 9. [Experiments & results](#9-experiments--results) 10. [Key decisions & lessons learned](#10-key-decisions--lessons-learned) 11. [Repository map](#11-repository-map) 12. [How to run it](#12-how-to-run-it) 13. [Evaluation methodology](#13-evaluation-methodology) 14. [Costs & budget discipline](#14-costs--budget-discipline) 15. [Current status & roadmap](#15-current-status--roadmap) 16. [Glossary](#16-glossary) --- ## 1. The idea in 60 seconds A coding agent (Claude Code, Codex, etc.) is great at reasoning but knows nothing about *your* repo until you paste files into its context — which is slow, expensive, and capped by the context window. The usual fix is RAG (retrieve chunks at query time). We do something different and complementary: **We bake the repo's "personality" directly into the model's weights, once, as a LoRA adapter — and we generate that adapter with a neural network instead of training it.** ``` ┌─────────────────────────────┐ repo on disk ──────► │ 6-view extractor + Qwen3 │ ──► 12288-d │ frozen embedding encoder │ repo embedding └─────────────────────────────┘ │ ▼ ┌──────────────────────────┐ │ HYPERNETWORK (our head) │ │ MLP → per-module A,B │ └──────────────────────────┘ │ LoRA weights ▼ Q: "what layer owns auth in this repo?" ──► ┌──────────────────────────────┐ │ FROZEN Gemma-4-E2B + injected │ ──► "the middleware │ LoRA (zero extra tokens) │ layer, via ..." └──────────────────────────────┘ ``` The magic: **the hypernetwork is trained across hundreds of repos**, so it learns the *mapping* `repo embedding → good adapter`. At inference on a brand-new repo it has never seen, it embeds the repo once and produces an adapter in a single forward pass. This is the same reason the source paper needed 400+ repos, not 1: breadth is what makes the mapping generalize. --- ## 2. Origin: the Code2LoRA paper We reverse-engineered **Code2LoRA** (arXiv 2606.06492v1) and found its released code (`anonymous.4open.science/r/code2lora-6857`, MIT). The paper's contribution: a *static hypernetwork* that maps a **repository** embedding → a LoRA adapter for a frozen code LLM, evaluated on **RepoPeftBench** with IR (in-repo) / CR (cross-repo) splits. On a full H100 setup they report **63.8 % cross-repo Exact Match**. Our project is the **Doc2LoRA variant** the paper itself cites — hypernetwork maps a *document/repo view* → LoRA — reimplemented against **Gemma-4-E2B**, trained fully **locally on MPS** (no CUDA/H100), and extended in two directions the paper does not cover: - **Memory / recall**: the adapter should let the model *recall facts* about the repo, not just complete code. - **Tech-Lead judgment**: architecture, data-flow, conventions, contracts, ops — the things a 20-year senior engineer "just knows" about a codebase. We keep the paper's proven autograd trick almost verbatim (see §3) and change only what Gemma-4 and Apple Silicon force us to change. --- ## 3. Architecture Three frozen/learned pieces. Only the middle one (the head) is trained. ### 3.1 Frozen encoder — `memory_lora/encoder.py` - **Qwen3-Embedding-0.6B**, frozen, no gradient flows through it. - Each repo view is chunked into token windows (2048 tokens, 128 overlap), each chunk mean-pooled, then chunks combined with **mean + max pooling** → a **2048-d** vector *per view*. - Embeddings are **precomputed once and cached** to parquet — the encoder never runs during training. ### 3.2 The hypernetwork head — `memory_lora/core.py :: MemoryLoRAHead` The only trained component. Design (kept close to the paper): - **2-layer GELU MLP trunk** (`input_dim → hidden_dim → hidden_dim`), followed by **L2-normalize + √hidden_dim rescale** (stabilizes the magnitude of generated weights). - **Per-module-type output heads**: for each target module *type* it emits an `A ∈ [rank, in_features]` and `B ∈ [out_features, rank]`. **One (A,B) pair per type, shared across all layers of that type** — this is what keeps the head tractable (188.6 M params) instead of exploding per-layer. - **Squashing**: `tanh(raw) * exp(log_scale)` with a learned per-type `log_scale` (init **-3.5**). This starts the generated adapter near-zero (so training begins close to the base model) and lets each type learn its own output scale. - **Defaults**: `hidden_dim=128`, `rank=16`, `dropout=0.1`. - *Why hidden_dim=128 and not the paper's 512/1024?* A 745 M-param head (hidden_dim=512) barely moved eval loss (~1.9 → ~2.7) but was far heavier to train on MPS. 128 cuts head size dramatically with negligible quality loss locally. Bump it later on real GPUs. ### 3.3 The LoRA injection — `memory_lora/core.py :: LoRA` ``` base nn.Linear (FROZEN) hypernetwork output │ │ x ──►│ Wx ────────────────┐ │ │ (input detached +──► y = Wx + scaling · B (A x) │ into base) │ ▲ ▲ x ──────────────────────────┘ │ │ A,B are NON-detached tensors so autograd flows LM-loss → head ``` Critical detail (straight from the paper's code): **A and B are plain, non-buffer tensor attributes, not `nn.Parameter` and not detached**, so the gradient of the LM loss flows *through* the injected weights *into the hypernetwork*. The base `nn.Linear` is frozen and its input is detached. Get this wrong and the head never learns. ### 3.4 Shape-qualified module types — the heterogeneity fix Gemma-4-E2B is **not** a uniform stack (see §4). Two `q_proj`s can have different shapes. If you key the head by bare type name (`q_proj`) you get *"type q_proj inconsistent dims"* crashes. Fix: key by **shape-qualified type**, e.g. `q_proj_1536x2048` vs `q_proj_1536x4096`. The v2 run discovered **14 shape types** across **205 target modules**: ``` down_proj_12288x1536 down_proj_6144x1536 gate_proj_1536x12288 gate_proj_1536x6144 k_proj_1536x256 k_proj_1536x512 o_proj_2048x1536 o_proj_4096x1536 q_proj_1536x2048 q_proj_1536x4096 up_proj_1536x12288 up_proj_1536x6144 v_proj_1536x256 v_proj_1536x512 ``` `get_module_specs(root_prefix="model.language_model.")` restricts wrapping to the text decoder — the **vision and audio towers are never touched** (not even inspected), so the multimodal forward path stays intact and they cost only idle RAM. --- ## 4. The target model: Gemma-4-E2B specifics Verified by reading the actual safetensors header, not guessed: - **Real model.** Google shipped Gemma 4 in March 2026. Apache-2.0, ungated. Class `Gemma4ForConditionalGeneration`, `model_type: "gemma4"`. Loaded via `AutoModelForImageTextToText`. - **Requires `transformers >= 5.5.0.dev0`** — install from the `main` branch, not a pinned PyPI release (this is the single biggest environment risk; smoke-test first). - **Decoder is nested**: layers live at `model.language_model.layers.{i}.*`, *not* `model.layers.*`. The layer-index regex had to change accordingly. - **35 text layers, heterogeneous:** - Aggressive KV sharing — **20 of 35 layers lack their own `k_proj`/`v_proj`** (`num_kv_shared_layers=20`). - **Every 5th layer is wider** (the `*_4096`, `*_12288` shape variants above). - **Device `mps`, precision bf16** (fall back to fp16 if unstable). No `flash_attention_2` on MPS — use `sdpa`, fall back to `eager`. --- ## 5. Data pipeline: the 6 views `scripts/build_repo_multiview.py` clones a repo and extracts **6 complementary views**, embeds each with Qwen3 → 2048-d, and **concatenates to a 12288-d** repo vector. The views encode the different "lenses" a senior engineer uses: | View | What it captures | Source signals | |-----------------|----------------------------------------------------|----------------| | `v_graph` | call / import / dependency structure | AST for Python (`memory_lora/codegraph.py`); `IMPORT_RE`/`DEF_RE` regex fallback for other languages | | `v_arch` | architecture & layout | README, folder tree | | `v_history` | how the code evolved | `git log`, recent diffs | | `v_contracts` | behavioral contracts | test files | | `v_conventions` | idioms & style | representative source files | | `v_ops` | build / deploy / runtime | CI config, Dockerfile, build files | **Multi-language from the start.** `CODE_EXTS` + regex fallbacks mean the graph view works for 9 languages, not just Python (this was a deliberate correction — see §10). Repos with < 3 code files are skipped. The build is **resume-safe** (skips repos already in `multiview_sources.jsonl`) and **flushes the embeddings parquet every 10 repos**, so a crash never loses more than 10 repos of work. --- ## 6. Data pipeline: QA generation The repo embedding is the *input*; the *target* is repo-scoped Q&A. Two generators, both parallelized (`ThreadPoolExecutor`, `--workers 10`) with a **per-prompt disk cache** (idempotent reruns) and a `--model` flag: - **`scripts/generate_repo_scoped_qa.py`** — reads the same 6 views and asks the LLM for **8–12 repo-level judgment questions** ("what layer owns X", "what convention does this repo use for Y", "how does data flow through Z", "why is it structured this way"). Target scope = input scope (repo-level embedding ↔ repo-level QA). - **`scripts/generate_commitpack_qa.py`** — **breadth** generator: one commit per *distinct* repo across CommitPackFT (25k+ distinct repos), 3–4 commit-scoped judgment questions (why / conventions / contracts / impact). For a hypernetwork, **distinct-repo count is the currency of generalization**, so we favor 1 commit × many repos over many commits × one repo. **Discipline (both):** answers are **short judgment**, never file-path/line-number lists. This is deliberate — see Tier A/B/C next. **Models used (OpenRouter, OpenAI-compatible API):** | Model | Role | Notes | Cost | |-------|------|-------|------| | `google/gemini-3.6-flash` | high-quality QA | **reasoning is mandatory** → needs generous `max_tokens` (3000–4000) or it returns empty | ~$0.0021 / QA | | `google/gemma-4-31b-it` | bulk / cheap QA | non-reasoning, clean JSON | ~$0.001 / repo (~$1 per 1000 repos) | The OpenRouter key lives in a **git-ignored `.env` (mode 600)** and is never pasted into a command line. --- ## 7. What is learnable — Tier A / B / C A LoRA adapter has finite capacity. We classify repo knowledge by whether a LoRA can hold it — this drives the entire QA design: - **Tier A — Judgment & conventions** (LEARNABLE). "This repo puts business logic in services, not views." Compressible, generalizes. → **This is what we train on.** - **Tier B — Structural gist** (LEARNABLE). "Auth flows through middleware." The kind of thing, not the exact file. - **Tier C — Exact recall & multi-hop** (NOT reliably learnable). "Line 412 of `foo.py` calls `bar()`." This needs **retrieval (RAG)**, not weights. So Memory-LoRA and RAG are **complementary**: the adapter carries Tier A/B judgment for free (zero tokens); RAG handles Tier C exact lookups. The QA prompts forbid exact file/line answers precisely so we never ask the LoRA to do a job it structurally can't. --- ## 8. Datasets inventory Everything lives under `data/` (git-ignored blobs). Sizes are approximate. | Path | What | Scale | |------|------|-------| | `data/real_code2lora/` | **RepoPeftBench** from the `code2lora` HF org — 500 Python repos, repo-commit embeddings + diffs | 73,849 repo-commit rows; ~1.2 GB | | `data/commitpack/multilang_commits.jsonl` | **CommitPackFT** shards, 9 languages | 25k+ distinct repos | | `data/docs/multiview_sources.jsonl` | 6-view `view_text` per repo (input to QA gen) | growing (1000s of repos) | | `data/embeddings/multiview_embeddings.parquet` | 12288-d multi-view repo embeddings | 1000+ repos | | `data/embeddings/aligned6_embeddings.parquet` | **assembled training inputs** (repos with ≥1 QA) | 1058 repos (current) | | `data/qna/repo_scoped_qa.jsonl` | repo-level judgment QA | 11,232 QA | | `data/qna/techlead_qa_commitpack.jsonl` | commit-scoped breadth QA | 9,245 QA | | `data/qna/techlead_qa.jsonl` | SWE-bench tech-lead QA | 2,786 QA | | `data/qna/aligned6_qna.jsonl` | **assembled training targets** | 8,540 QA (current) | | `data/openrouter_cache/` | per-prompt response cache | ~19 MB | **Language balancing.** SWE-bench is ~79 % Django. Left alone, the dataset was 46 % Django. `scripts/consolidate_qa.py` applies a **per-repo cap** (default 12–15 QA/repo) which collapses Django to **~2.0 %** while preserving the 2400+ distinct repos' diversity. `assemble_6view_dataset.py` applies the same cap when building the final aligned set. --- ## 9. Experiments & results Chronological, with the actual numbers we measured. Two families of runs. ### 9.1 Reproducing the paper (single-view, real RepoPeftBench) | Run | What | Result | |-----|------|--------| | `full1` (early) | first end-to-end hypernetwork on converted real data | CR **EM 0.056–0.083**, EditSim ~0.27 — pipeline works, undertrained | | `sixview`/converted-real (best single-view ckpt) | after more training | **CR EM 0.524, EditSim 0.635** | | Paper (reference, H100) | their full run | CR EM **0.638** | **Headline:** on real code, after only ~2.4 % of one epoch of local MPS training, we reached **52.4 % cross-repo Exact Match** vs the paper's 63.8 % on a full H100 setup. The mechanism demonstrably works — the generated adapter recovers repo-specific identifiers the base model does not know. ### 9.2 The 6-view Tech-Lead model (the current line of work) Loss is causal-LM cross-entropy on QA targets; lower is better. Three eval suites: `cr_val` / `cr_test` (held-out *repos*) and `ir_test` (held-out *QA* of train repos). | Run | Dataset | Best held-out `cr_test` loss | Notes | |-----|---------|------------------------------|-------| | `sixview_v1` | 515 repos / 3,988 QA (415 train repos) | **2.848** (step ~1060) | Overfit afterward: train loss fell to 1.75 while `cr_test` drifted to 3.35. Classic small-dataset ceiling. | | `sixview_v2` | **1,058 repos / 8,540 QA (858 train repos)** | *in progress* | Resumed from `sixview_v1/head.best.pt`; 2× the data specifically to break v1's ceiling. | `sixview_v1` metrics trajectory (from `runs/sixview_v1/metrics.jsonl`): ``` step 1245 cr_test 2.962 ir_test 2.535 (end of epoch 2 — near best) step 1400 cr_test 3.245 ir_test 2.593 (overfitting begins) step 1600 cr_test 3.352 ir_test 2.655 (train loss still falling → ceiling hit) ``` The v1→v2 story is the core empirical lesson: **the small aligned set was the bottleneck, not the architecture** — hence the push to build 1000+ more repos. --- ## 10. Key decisions & lessons learned The expensive knowledge. Read this section twice. ### 10.1 ⚠️ The MPS gradient-checkpointing memory leak (the big one) **Symptom:** training with `gradient_checkpointing_enable()` (`use_reentrant=False`) **leaked ~12 GB per step** and OOM'd the whole machine within a few steps. **Diagnosis** (`scripts/diag_mps_leak.py`): forward-only was stable; train + checkpoint leaked 39 GB → 18 GB free in 2 steps. Isolated the checkpointing path as the cause. **Fix:** **`--no-gradient-checkpointing`.** We have enough unified memory to hold activations without it once the multimodal towers sit idle. This is documented as a standing memory (`mps-gradient-checkpointing-leak.md`). ### 10.2 ⚠️ `psutil` RSS is blind to MPS memory Our first memory safety-net used `psutil` RSS / `ps -o rss` — it reported **< 1 GB** while `top` showed **55–83 GB** actually in use. MPS (GPU) allocations don't show up in process RSS. **Fix:** the safety check uses **`psutil.virtual_memory().available`** (system-wide) with a `--min-available-gb` floor (default 5). To *observe* MPS memory, use `top -l 1 -pid -stats mem`, not `ps`. ### 10.3 Memory competition between concurrent jobs Three concurrent jobs once pushed available memory under the 10 GB floor and training self-stopped. **Lesson:** during MPS training, run data builds/embedding on **CPU** (`--device cpu`) so they don't contend for the GPU/unified memory. We now routinely run training (MPS) + QA gen (network) + multiview build (CPU) together without contention. ### 10.4 Don't lose hours of training Every long run writes **checkpoints every 50 steps** (overwriting `head.latest.pt`), **every 30 minutes** (timestamped `head.tNNNNm.pt`), **per-epoch** (`head.epN.pt`), and a **`head.best.pt`** on eval improvement. Runs are launched with `nohup … & disown` so they survive terminal/session death. `sixview_v1` in fact survived a full session interruption and kept training. Resume with `--resume-from ` (loads head weights; optimizer restarts fresh). ### 10.5 Data-quality corrections (user-driven) - **"I still see lots of Django."** SWE-bench is Django-dominated. → per-repo cap + multi-language sourcing dropped Django 46 % → 2.0 %. - **"It must be good for any programming language."** → 9-language diversity via CommitPackFT and language-agnostic view extraction. - **"Where's the code context in the QA?"** → clarified the two-channel design: the **repo embedding is the context channel**, the QA is only the target. They are joined by `doc_id` at assembly time. ### 10.6 OpenRouter gotchas - `gemini-3.6-flash` **returned empty** until we raised `max_tokens` — it's a mandatory-reasoning model that spends tokens on hidden reasoning before content. Reasoning **cannot be disabled** (400 error). - CommitPackFT's HF loader is deprecated → fetch raw `data.jsonl` directly. - `global MODEL` after use is a `SyntaxError` → set via `globals()["MODEL"] = ...`. ### 10.7 Performance fix worth knowing Loading embeddings was 5+ min because `_list_to_f32_array` used a Python loop. Vectorized via `col.combine_chunks().flatten().to_numpy()` → **~220× faster**. --- ## 11. Repository map ``` memory_lora/ # the library (importable package) core.py # LoRA wrapper, MemoryLoRAHead hypernetwork, # get_module_specs / replace_with_lora / inject_lora_weights, # load_doc_rows / load_qna_rows encoder.py # Qwen3 chunk + embed + mean/max pool (frozen) codegraph.py # Python AST extractor (imports, sigs, call graph) data_paths.py # local parquet/jsonl path resolver scripts/ build_repo_multiview.py # clone → 6 views → 12288-d embeddings (multi-language, resume-safe) generate_repo_scoped_qa.py # repo-level judgment QA (aligned to the 6 views) generate_commitpack_qa.py # commit-scoped breadth QA across 1000s of distinct repos generate_techlead_qa.py # SWE-bench tech-lead QA generate_synthetic_dataset.py # original synthetic doc + QA generator consolidate_qa.py # per-repo cap → language/domain balancing assemble_6view_dataset.py # join embeddings ↔ all QA by repo → aligned6_{embeddings,qna} augment_paraphrases.py # QA paraphrase augmentation convert_real_code2lora.py # RepoPeftBench → our schema build_doc_embeddings.py # encoder pass over documents merge_corpora.py # combine multiple corpora train_memory_lora.py # THE trainer (MPS, one-repo-per-step, checkpoints, TensorBoard) train_direct_lora.py # baseline: plain per-repo LoRA (no hypernetwork) eval_memory_lora.py # EM / EditSim recall eval on cr/ir splits show_eval_examples.py # dump concrete base-vs-adapted examples test_embed_this_repo.py # embed the current repo (pipeline demo) test_recall_this_repo.py # query the adapted model about this repo diag_mps_leak.py # the memory-leak isolation harness data/ # git-ignored: embeddings, qna, sources, caches runs/ # git-ignored: checkpoints, logs, metrics.jsonl, tb/ requirements.txt # torch 2.13 (MPS), transformers@main, openai, pyarrow, tensorboard… .env # git-ignored, mode 600: OPENROUTER_API_KEY ``` --- ## 12. How to run it ### Setup ```bash python3 -m venv venv && source venv/bin/activate pip install -r requirements.txt # installs transformers from git main echo "OPENROUTER_API_KEY=sk-or-..." > .env && chmod 600 .env ``` Smoke-test the environment first (gates everything): confirm `transformers` main loads `google/gemma-4-E2B` on `mps` and runs a text-only forward pass. ### Build data ```bash # 1) multi-view embeddings for a repo list (CPU to stay off the GPU during training) python scripts/build_repo_multiview.py \ --repos-file data/multilang_repo_list.txt --max-repos 1000 --device cpu # 2) repo-scoped QA (cheap model) — appends, resume-safe, cached ./venv/bin/python scripts/generate_repo_scoped_qa.py \ --model google/gemma-4-31b-it --workers 10 # 3) balance + assemble the aligned training set python scripts/consolidate_qa.py --per-repo-cap 12 python scripts/assemble_6view_dataset.py # -> data/embeddings/aligned6_embeddings.parquet + data/qna/aligned6_qna.jsonl ``` ### Train (the exact `sixview_v2` command) ```bash nohup ./venv/bin/python scripts/train_memory_lora.py --output-dir sixview_v2 \ --resume-from runs/sixview_v1/head.best.pt \ --embeddings-path data/embeddings/aligned6_embeddings.parquet \ --qna-path data/qna/aligned6_qna.jsonl --epochs 100 --max-hours 8 \ --checkpoint-every-steps 50 --checkpoint-every-minutes 30 --epoch-ckpt-every 5 \ --eval-every-steps 300 --limit-eval-docs 40 --max-seq-len 512 --fixed-seq-len \ --max-qna-per-doc 12 --lm-micro-batch 2 --device mps --no-gradient-checkpointing \ --rank 16 --head-hidden-dim 128 --head-dropout 0.1 --weight-decay 0.05 \ --early-stop-patience 25 --lr 8e-5 --lr-total-steps 9000 --min-available-gb 5 \ > runs/sixview_v2_train.log 2>&1 & disown ``` **Flags you must not forget:** `--no-gradient-checkpointing` (the leak), `--device cpu` for builds during training (contention), `--min-available-gb` (system-wide memory floor). ### Watch it ```bash tensorboard --logdir runs/sixview_v2/tb # train/loss, train/lr, eval/{suite}_loss tail -f runs/sixview_v2_train.log ``` ### Evaluate & inspect ```bash python scripts/eval_memory_lora.py --ckpt runs/sixview_v2/head.best.pt # EM / EditSim python scripts/show_eval_examples.py # base vs adapted ``` --- ## 13. Evaluation methodology - **Splits (deterministic, by `md5(repo) % 100`):** 80 % train / 10 % `cr_val` / 10 % `cr_test` **by repo**, so cross-repo suites are **entirely held-out repositories** the hypernetwork never trained on. Within train repos, ~15 % of QA is held out → `ir_test` (in-repo generalization to unseen questions of seen repos). - **Metrics:** causal-LM **eval loss** during training (fast, every N steps on `--limit-eval-docs` docs to stay cheap on CPU), plus generation-time **Exact Match (EM)** and **EditSim** for the recall eval. - **The proof spot-check:** query the *adapted* model with repo-specific questions and confirm the *base* (un-adapted) model gets them wrong/vague — proving the **adapter**, not the base model's pretraining, does the work. CPU eval of a float32 5B model is slow (~20 min for a full pass) → we cap eval docs (e.g. 10–40) for in-loop evals and run full EM eval separately. --- ## 14. Costs & budget discipline - **Spend baseline:** $31.00 (`runs/spend_baseline.txt`); ~**$33.90 total** to date; ~**$11 remaining**. The project is run under explicit budget caps ("spend at most $4 more") with spend-guards. - **Unit economics:** `gemini-3.6-flash` ≈ **$0.0021/QA**; `gemma-4-31b-it` ≈ **$0.001/repo (~$1 per 1000 repos)** — which is exactly why the 1000-repo expansion uses the gemma model. - **Free levers:** the per-prompt cache makes reruns free; embedding and training are local (electricity only). --- ## 15. Current status & roadmap **Live right now (three jobs in parallel, no contention):** - **`sixview_v2` training** — resumed from `head.best.pt` on the doubled **1,058-repo / 8,540-QA** dataset (858 steps/epoch), MPS. First held-out eval at step 300 tells us whether doubling the data broke v1's 2.848 ceiling. - **QA generation** — `gemma-4-31b-it` filling in all ~1,032 new repos for the **complete** dataset (next training run). - **Multiview build** — cloning/embedding toward the full +1,000-new target (CPU). **Roadmap:** 1. Finish the complete 1000-new-repo dataset (embeddings + QA). 2. Assemble the full aligned set (~1,650 repos) and train **`sixview_v3`** on it. 3. Run generation-time **EM/EditSim** on the 6-view model (base vs adapted). 4. Push `head_hidden_dim` back up once on real GPUs; the 128 default was an MPS-locality compromise. 5. Broaden Tier-A/B QA toward agent-harness use cases (Jira/ticket tracking, diff/impact reasoning) already scaffolded in `generate_techlead_qa.py`. **Open questions:** - Does the 12288-d 6-view embedding actually beat the single 2048-d view on generation EM, or only on loss? (loss says yes; EM eval pending) - What's the real Tier-B ceiling — how much structural gist fits in rank-16? - Optimal per-repo QA cap for the breadth/depth trade-off. --- ## 16. Glossary - **Hypernetwork** — a network that outputs the weights of another network. Here: repo embedding → LoRA (A,B) matrices. - **LoRA** — Low-Rank Adaptation: `y = Wx + scaling · B(Ax)`, with `A,B` low-rank (rank 16). We *generate* A,B instead of training them per-repo. - **6 views** — graph / arch / history / contracts / conventions / ops; each 2048-d, concatenated to 12288-d. - **CR / IR** — cross-repo (held-out repos) / in-repo (held-out QA of seen repos). - **EM / EditSim** — Exact Match / edit-distance similarity of generated vs gold. - **Tier A/B/C** — judgment (learnable) / structural gist (learnable) / exact recall (needs RAG). - **MPS** — Apple's Metal Performance Shaders GPU backend for PyTorch. - **RepoPeftBench** — the paper's benchmark; 500 Python repos, repo-commit embeddings + diffs, IR/CR splits. --- *Maintained as living documentation. If you change a default, a path, or a flag, update the matching section here — onboarding depends on it.*