# OncoDSL — context for Claude

Long-term goal: a genetic-programming engine that, over a small DSL of data
operations, **rediscovers the MSI / mismatch-repair gene signature from TCGA
colorectal data blind**. The engine must not see real gene names — an "airgap"
layer between the data layer and the engine hides them.

## Chunk status

Built in chunks. Current state:

- **Chunk 1 (done):** project scaffold + cBioPortal data load + Streamlit
  sanity viewer. See `README.md`.
- **Chunk 2 (done):** DSL operators (`Load`, `Select`, `Reduce`, `Split`,
  `Associate`, `Effect`, `Search`, `Fit`, `Apply`) live in `dsl/`. Airgap
  (`anonymise` / `reveal` + sealed map at `data/processed/_sealed_gene_map.json`)
  lives in `airgap/`. H1 verification (known-answer MMR + immune check on the
  NAMED matrix) lives in `validate/h1.py`. Viewer is split into two top-level
  tabs — **Dataset** (the five sanity panels) and **Hypothesis 1** (intro,
  DSL reference with nouns + verbs, graphviz composition diagram with
  `Given:` input nodes, then verification outputs in this order: dynamic
  pass/fail Conclusion → score distributions → Effect → Fit → collapsed
  "How to read this", plus an airgap demo). Glossary expander sits above
  the tabs.
- **Chunk 3 (done):** GP engine in `engine/` — strict airgap; composes
  Select/Reduce/Fit programs over the anonymised matrix with prefilter +
  tournament-selection GP + permutation null. Orchestrator at
  `scripts/run_h2.py` (Load → anonymise → engine → persist artefacts).
  Artefacts at `data/processed/h2/{evolution_log,result}.json`, both
  anonymised. FastAPI surface in `api/app.py` (`/health`, `/run`,
  `/result`, `/reveal`) — `/reveal` is the only endpoint allowed to read
  the sealed map and translates only the supplied IDs. Viewer gains a
  "Hypothesis 2" tab (5 cards: hypothesis → setup → evolve replay →
  result → reveal + computed conclusion) that fetches from the API.
  Validation lives in `validate/h2.py` (reveal + MMR overlap).
- **Chunk 4 (done — Stage 1):** React/Next.js "Lab" at `web/`
  (App Router + TS + Tailwind + Recharts). Adds a second selectable
  engine objective — `CorrelationObjective(direction)` for continuous
  targets like TMB — alongside `BinaryAUROCObjective` for MSI. New API:
  `POST /runs` (worker thread + SSE bridge), `GET /runs/{id}`,
  `GET /runs/{id}/stream` (`sse-starlette`), `GET /runs/{id}/result`,
  `POST /evaluate` (reveal supplied IDs + score overlap with `"MMR"` or
  `"immune"`). Lab page is one Client Component with 5 sections:
  objective builder (survival / unsupervised disabled, Stage 2) →
  parameter inputs → Run → live fitness curve + population grid
  (survivors solid accent, discarded faded) → result + reveal &
  evaluate. Reuses the same engine — `scripts/run_h2.py`, the legacy
  `/run /result /reveal /health` endpoints, and the Streamlit viewer
  are untouched.
- **Chunk 4 follow-ups (done):**
  - **Prefilter optional + bigger GP budget.** Engine
    `prefilter_n: int | None`; `None` means the GP samples its
    initial-population AND its mutation operator from the full opaque
    column set (~17k after NaN drop), not from a univariate shortlist.
    `_prepare` returns `gp_pool` + `baseline_genes` separately — the
    baseline keeps using the univariate top-K so it stays a fair sanity
    check. API accepts `prefilter_n: null`. Lab UI defaults to OFF (60
    gens / 300 pop) with an "Off (all genes) / On" toggle; turning On
    flips back to 30 / 150 / `prefilter_n=2000`. `scripts/run_h2.py`
    untouched (explicit `--prefilter-n 2000`).
  - **InfoTips everywhere.** Single `<InfoTip>` Client Component drives
    every "?" in the Lab (objective cards, parameters, run / live /
    result / evaluator labels + sub-labels). Verbatim tooltip copy
    lives in the `OBJECTIVE_TIPS` / `PARAM_TIPS` / `TIPS` records at
    the top of `Lab.tsx` — never inline.
  - **Program graph (React Flow).** `web/app/ProgramGraph.tsx` reads
    `feature_sets` straight from the winner / candidate payload (no
    backend change) and renders the DSL program as a node graph:
    Tier-1 lanes [Expression matrix → Select(opaque IDs) → Reduce
    (mean) → Score pill] per gene-set, then [Score(s) → Fit → output
    node]. Tier-1 = light-teal group box, Tier-2 = dashed `#BC6B2E`
    wrapper (omitted for single-set programs). Custom node types only,
    palette tokens only; React Flow chrome stripped via
    `ProgramGraph.css` (`opacity:0` handles, no Background, no
    MiniMap, no Controls, attribution hidden, scroll passes through).
    The graph card defaults to the winning program; clicking any
    candidate in the population grid swaps the graph to that candidate
    in place. After the Evaluator runs, Select-node pills show real
    symbols and matched reference genes are filled in the project
    highlight (`#BC6B2E`).
- **Chunk 5 (done — engine_v2 + full-population UI):**
  - **engine_v2/** — strongly-typed program synthesis. Programs are
    trees over a five-type grammar (`Matrix`, `Vector`, `FeatureSet`,
    `Agg`, `Op`). Nodes: `MatrixTerminal`, `Select(Matrix,FeatureSet)
    →Matrix`, `Reduce(Matrix,Agg)→Vector`, `Combine(Vector,Vector,Op)
    →Vector`. `Reduce.agg ∈ {mean, median, max, min, var}`,
    `Combine.op ∈ {add, sub, mul, protected_div, mean}`. Output type =
    `Vector`. Split / Effect deferred to v2.1. Ramped half-and-half
    init; tournament + elitism; subtree crossover with strict type
    matching; subtree + point mutation; depth/node budgets. Termination
    guarantee: depth-floor closes Matrix to `MatrixTerminal` and Vector
    to `Reduce(MatrixTerminal, agg)`, so no tree ever has an open slot.
    Reduce.agg vocab also widened in `dsl.Reduce` itself.
  - **Two v2 objectives, with finite worst-case fallbacks**:
    `BinaryAUROCObjective` (MSI/MSS — orientation-agnostic
    `max(AUROC, 1−AUROC)`, worst = 0.5) and the v2 `CorrelationObjective`
    (TMB — `−spearman(score, TMB)`, worst = 0.0). Degenerate programs
    (empty FeatureSet, constant/NaN output) get the worst score; they
    never leave the API as NaN/Infinity.
  - **Permutation null is winner-fixed in v2:** hold the winning
    program fixed and shuffle the target N times; p = fraction of
    nulls ≥ observed. The old "rerun the univariate baseline on
    shuffled labels" null does NOT transfer to v2 and was dropped.
  - **API**: `POST /runs` accepts `engine: "v1" | "v2"`. v2 worker uses
    `engine_v2.run_v2_pipeline_streaming`. Full population persisted
    per generation in `run.log[i].candidates` (both engines); SSE
    stream keeps the bandwidth budget light by trimming to top-12.
    `GET /runs/{id}` exposes `generations_persisted`; full per-gen
    populations come from `GET /runs/{id}/population/{generation}`
    (404 for unpersisted indices, never 500). All endpoint returns,
    the in-memory `run.result`, and the SSE payloads pass through
    `_json_finite()` — NaN / ±∞ → `None`, so `JSON.parse` never breaks.
  - **Frontend (web/)**: Lab now posts `engine: "v2"`. Replaced the
    old top-12 grid with `PopulationTiles` (full per-gen population,
    fitness-tinted teal ramp, "Nn·dD" structure signature, generation
    stepper bounded to `generations_persisted`). New
    `lib/programRepr.ts` is the SHARED parser for the typed
    `program_repr` (also tolerates the legacy v1 `Fit(…)` shape).
    `ProgramGraph.tsx` was rewritten to consume the parsed tree and
    now renders arbitrary Select/Reduce/Combine/M shapes; same custom
    nodes, same `ProgramGraph.css` chrome strip. `PasteToDraw.tsx`
    feeds arbitrary `program_repr` text into the same renderer.
    `ParameterFlow.tsx` adds a collapsed "How these parameters
    relate" disclosure under the Parameters card caption — a centered
    single-column HTML/flex diagram (max-width 460px, width 100%,
    never overflows): step boxes, colour-grouped space/effort/
    validation pills that wrap on narrow widths, plain text `↓`
    arrows, amber final "permutation p" box. Generations pill uses a
    `↻ ` text prefix (not an icon glyph). A single
    `lib/fmt.ts::fmtFit()` formats every fitness/score across tiles,
    cards, tooltips, the result panel, and the live curve — falling
    back to "—" if a non-finite value ever reaches the UI; tiles also
    use `fitnessForOrder()` so any non-finite sinks to the bottom of
    the grid with the lightest tint.
- **Chunk 5 follow-ups (done — UX polish, no GP / API changes):**
  - **Rich Parameter help.** New `<ParamHelp paramKey>` Client
    Component replaces the small InfoTip on the 8 Parameters labels.
    Hover/focus still shows the SHORT one-liner; click opens a modal
    dialog with text + diagrams. `ParamHelpProvider` mounts a single
    portal-modal at the Lab root so any trigger can open it (the
    ParameterFlow chart's pill names are also ParamHelp triggers).
    Registry of detailed content lives in `web/app/paramHelpContent.tsx`;
    six diagrams (Generations loop, MaxSets, Lambda trade-off, Seed
    dice, Permutations histogram, Prefilter funnel) live under
    `web/app/diagrams/`. Modal is role="dialog" aria-modal="true" with
    Esc / X / backdrop close, focus trap, and focus return.
  - **Program graph + paste-to-draw.** `ProgramGraph.tsx` now adds
    Tier-1 (light teal `#F1F6F7` / `#3A6B7E`) and Tier-2 (dashed
    `#BC6B2E`) wrapper nodes when the parsed root is `Combine` (single-
    score programs skip wrappers). `PasteToDraw.tsx` is now controlled
    (text lifted to Lab so tile clicks AND the winner-default both
    write into it), placeholder fixed to a well-formed example, hint
    spells out the Reveal restriction for pasted text, and a collapsed
    "Format &amp; examples" disclosure documents the grammar (`M`,
    `Select`, `Reduce` aggs, `Combine` ops, rules, annotated example).
  - **Copy & load.** New `web/app/CopyButton.tsx` (`navigator.clipboard`
    with `execCommand("copy")` fallback) drives a "Copy program" button
    next to the winner in the Result panel and an icon-only copy
    overlay on every tile in `PopulationTiles`. Tile body uses
    `role="button" tabIndex={0}` so the inner icon-button can stop
    propagation; clicking the body loads the program into the textarea
    AND the graph.
- **Chunk 6 (done — full DSL in engine_v2 + objective explainers):**
  - **Full DSL grammar in engine_v2.** Typed grammar grew from
    {Matrix, Vector} to {Matrix, Vector, Scalar, Model}. New nodes:
    `Split(Vector, Predicate)→Vector` (predicates: `score` = above /
    below median, `stage_late` = AJCC III/IV; one level only,
    mean-centre per branch, min subgroup guard);
    `Associate(Vector, target, kind)→Scalar` and
    `Effect(Vector, target, kind)→Scalar` (Effect residualises on
    stage + age — observational backdoor adjustment, "only as good as
    the measured confounders"); `FitApply(Vector, target)→Vector`
    (Fit a LR/OLS on the score, Apply in place); `Search(Matrix, k)
    →Matrix` (bounded univariate nested ranker, capped at k ≤ 4 and
    200 cols, **gated OFF by default** per A4 — visible in the
    grammar but rate 0 until cost is acceptable).
  - **ExecContext** threads through every `Node.execute(ctx)` —
    bundles `M` (opaque), `clinical` (named: stage, age only), and
    `labels` (named: msi, tmb only). The pipeline and worker plumb
    clinical + the OTHER target through so a v2 MSI run can still
    Associate / Effect / FitApply against TMB and vice versa.
  - **Fitness handles every output type.** Vector → objective metric
    on per-patient scores; Scalar (Associate / Effect) IS the
    fitness, oriented per objective (|·| for MSI, `-x` for TMB);
    Model → routed through FitApply which auto-Applies and emits a
    Vector. Worst-case floor still applies (0.5 MSI, 0.0 TMB).
  - **Typed GP** extended with type-safe init / crossover / mutation
    for every new node; ramped half-and-half now spawns ~20%
    Scalar-rooted programs so Associate / Effect appear in the
    population; rates dict configures Effect / Split / FitApply /
    Search injection. Termination guarantee: Matrix → MatrixTerminal,
    Vector → Reduce(MatrixTerminal, agg), Scalar →
    Associate(Reduce(MatrixTerminal, "mean"), target, "spearman").
  - **Parser + renderer + format guide.** `web/lib/programRepr.ts`
    parses every new operator (spaces tolerated). `ProgramGraph.tsx`
    renders Split / Search / FitApply / Associate / Effect as
    accent-bordered verb nodes (same theme). PasteToDraw's "Format &
    examples" disclosure documents the full grammar with one short
    annotated example per operator.
  - **Objective modals.** The four objective `?` icons (MSI / TMB /
    Survival / Unsupervised) now use the same rich ParamHelp modal
    infrastructure as the parameters — verbatim SHORT + DETAILED
    text per the prompt, shared intro about "an objective is the
    rule that scores every program" and shared footer about
    Associate / Effect / Fit being expressible in the DSL itself.
    Old `OBJECTIVE_TIPS` constant removed; the "brokNA spell-checker"
    typo is gone.
  - **A9.** UI fitness label for TMB switched from `"association
    (|spearman|)"` to `"negative association (signed −spearman with
    TMB)"`; LiveView and held-out copy reworded to match. Engine
    fitness already used signed −spearman.
  - **Airgap stays absolute.** Only `stage`, `age`, `msi`, `tmb`
    flow through engine_v2 as named values; gene IDs remain opaque
    everywhere (program_repr, populations, API payloads). Airgap
    tests parametrised over both engines × both prefilter modes
    stay green.
- **Chunk 6 follow-up (done — scoring target bound to the objective):**
  A bug from chunk 6: under the TMB objective an engine_v2 run could
  produce `Associate(Reduce(...), msi, spearman)` — the program freely
  picked its own target. Fixed by removing the target as a selectable
  operand:
  - `engine_v2.synthesize` now threads `objective_target` through
    `_grow_scalar`, `_grow_vector`, `random_program`,
    `ramped_population`, and `mutate`. Every `Associate` / `Effect` /
    `FitApply` is constructed with `target = objective.target`.
  - Point mutation no longer carries a `fit_target` spot and the
    `scalar_kind` spot no longer flips the target — only the
    correlation kind (pearson / spearman) is mutable. Crossover never
    touched targets (subtree swaps preserve them).
  - `engine_v2.fitness._check_target_binding(program, expected)` walks
    every Associate / Effect / FitApply node and asserts
    `n.target == expected`. `fitness_fn` calls it first and floors any
    mismatch to `WORST_FITNESS`, so a stray target — even from a
    replayed tree — can never win.
  - `program_repr` still shows the bound target (e.g.
    `Associate(score, tmb, spearman)` under the TMB objective) so the
    graph and paste-to-draw renderers don't change.
  - The shared footer in all four objective `?` modals now reads
    verbatim: *"What the program chooses is how to build the score and
    how to compare it — a raw association (Associate) or a
    confounder-adjusted one (Effect), plus the correlation kind. What
    stays outside the DSL is the compass: the target it's scored
    against, in which direction, judged honestly on held-out data.
    The program can't pick the target — that would let the answer
    into the language."*
  - New tests: `test_synthesis_binds_scoring_target_to_objective`
    (init + 600 mutations × both objectives — every node binds to the
    active target) and `test_fitness_floors_mismatched_target_to_worst`
    (hand-crafted bug-shape program is floored to `WORST_FITNESS`).
    80 / 80 pytest pass; the API airgap and gene-symbol-leak tests are
    untouched.
- **Chunk 6 follow-up (done — Live view polish, presentation-only):**
  - **Fitness curve chart.** Recharts' default `<Legend>` is gone; in
    its place a custom `<ChartLegend>` HTML row sits below the chart
    with SVG-line swatches that mirror the actual strokes (solid
    accent for best, dashed muted for median). The X-axis title
    ("generation") is pulled OUT of the SVG and rendered as a centred
    caption between the chart and the legend so the three rows
    (ticks → axis title → legend) never overlap. Y-axis label uses a
    `<CenteredYAxisLabel>` that rotates `<text>` at
    `viewBox.y + viewBox.height/2` with `text-anchor="middle"` — the
    title now sits beside the *middle* of the axis, not the top.
    Lines: best solid `#3A6B7E` strokeWidth 2.5, median dashed (4 4)
    `#6E7F8C` strokeWidth 1.5.
  - **InfoTip copy.** `TIPS.fitnessCurve` and `TIPS.bestVsMedian`
    rewritten verbatim — the y-axis tip explains negative-association
    strength (with the MSI/AUROC fallback parenthetical); the
    best-vs-median tip explains convergence vs. persistent gap.
  - **Generation navigator (PopulationTiles).** Replaced the
    click-only `‹ Gen N ›` stepper with: `|‹` first / `‹` prev / a
    draggable range slider / `›` next / `›|` latest / a numeric input
    with a "go" button. Fetch decoupled from display via two states:
    `draftGen` updates on every input event for live number, but the
    `/runs/{id}/population/{gen}` fetch is keyed on `committedGen`,
    which only updates on `pointerup`/`pointercancel` or a 150 ms
    throttle while dragging. Keyboard: native left/right arrows
    commit immediately; custom Home/End jump to first/latest. ARIA:
    `aria-label="Generation"`, `aria-valuemin/max/now`,
    `aria-valuetext`, and a polite live region for the "Generation N
    / total" label.
  - **DSL vocabulary tiles cover the full grammar.** The Lab's "DSL
    vocabulary" panel now lists all eight engine_v2 verbs (Select,
    Reduce, Combine, Split, Associate, Effect, Fit/Apply, Search) so
    the panel matches what programs in the population can actually
    contain — earlier it only showed the three Vector-shape verbs and
    silently omitted Scalar/Model-shape and gated ops. Presentation
    only; engine grammar unchanged.
- **Chunk 6 follow-up (done — TMB-rank diagnostic, MMR biology panel,
  MLH1 success reframe, real Unsupervised objective, live-view polish,
  gene-based unsupervised synthesis):**
  - **TMB-rank diagnostic.** New `validate/tmb_rank.py` and a small
    `GET /diagnostic/tmb-rank` API endpoint. Mirrors `api._prepare_lab_data`
    target="tmb" exactly (Load('processed') filtered to tmb.notna() &
    expression-complete rows) and computes signed Spearman per gene
    vs TMB on the NAMED matrix (vectorised rank-Pearson, sub-second).
    Reports each MMR / IMMUNE gene's rank/N + the top-10
    most-negatively-correlated genes. On the live TCGA cohort: MLH1
    sits at rank 270 / 20057 (top 1.3%); PMS2 at 2214 (11%); MSH2/MSH6
    not in expression (mutational, not silenced). The diagnostic lives
    in `validate/` (allowed gene names) — the structural engine
    airgap test still scans only `engine/`. 3 new tests in
    `tests/test_tmb_rank.py`.
  - **MMR biology panel.** `web/app/MMRBiologyPanel.tsx` mirrors
    `ParameterFlow.tsx`'s collapsible exactly (button + chevron +
    aria-expanded + useId-panel-id + closed-by-default). Body inlines
    `mmr_reference.svg` verbatim (cause→effect: MMR → MSS / MSI-H → immune
    response, plus the gene-expression visibility readout). Mounted
    between the Objective card and Parameters card in `Lab.tsx`.
  - **MLH1 success callout (TMB).** `/diagnostic/tmb-rank` is now
    fetched at the Evaluator level (lifted out of the inner
    `TMBRankPanel`) so the same response also drives an accent SUCCESS
    callout above the overlap table — *"Blind, MLH1 ranks 270 / 20057
    (top 1.3%) on the TMB objective — the causal gene surfaced near
    the top without ever seeing gene names."* `OverlapSummary` gains
    an optional muted secondary line so the "0 of N winning genes" read
    no longer reads as flat failure.
  - **Real Unsupervised objective (engine_v2).** New `UNSUP_OBJECTIVE`
    with `target="none"`, `metric="structure"`. Fitness = silhouette of
    a 2-means split on standardised 1-D score (range [-1, 1], worst
    floored at -1.0). Vector-only programs: `V2Objective.synthesis_overrides()`
    returns `{rates: {split:.1, fitapply:0, effect:0, search:0},
    scalar_share:0}` for unsup, plumbed via `gp.py` into
    `ramped_population` / `mutate`. `_build_ctxs` strips all labels
    when target='none' (asserted) so the engine literally cannot see
    msi/tmb during search. New `unsup_random_null` in `permutation.py`
    — N random Vector-only programs scored on held-out (no target to
    shuffle); `permutation_summary.null_kind` distinguishes the two
    nulls in the result.
  - **Post-hoc alignment.** Worker thread for unsup target runs
    `_compute_unsup_posthoc` AFTER the GP finishes: orientation-agnostic
    AUROC of winner scores vs MSI (held-out subset, gated n≥10 per class)
    and `|spearman|` vs TMB. Lives in `api/_worker` (allowed labels) —
    never in engine_v2. New `result.posthoc = {msi_auroc, tmb_abs_spearman,
    n_holdout, n_msi_held, n_tmb_held}` rendered in `<PosthocAlignment>`
    inside the ResultPanel.
  - **Frontend — drop Survival, add Unsupervised end-to-end.** `Target`
    widened to `"msi" | "tmb" | "none"`; `OBJECTIVE_PRESETS.none`,
    `FITNESS_LABEL_BY_TARGET.none`. Survival entry removed from
    `ObjectiveBuilder` + `paramHelpContent.tsx`. For unsup runs the
    Evaluator hides the reference-set toggle / overlap / matched
    column (no overlap concept) and falls back to the post-hoc block.
    The `RunResult` interface in `web/lib/api.ts` gains `posthoc?:
    Posthoc`.
  - **Live-view + objective-modal clarity (presentation-only).**
    Shortened y-axis titles (`tmb: "neg. association with TMB"`,
    `none: "cluster separation (0–1)"`). Replaced the single TMB-centric
    `TIPS.fitnessCurve` with `FITNESS_TIP_BY_TARGET: Record<Target,
    string>` and plumbed `target` into `LiveView` so the InfoTip matches
    the y-axis title's objective-awareness. Added `TIPS.nodes` and
    `TIPS.genes` and passed them to the NODES + GENES `Metric` cards
    in the ResultPanel. Rewrote `ObjectiveIntro` so it no longer
    asserts "a target column" unconditionally; rewrote `obj_msi` /
    `obj_tmb` / `obj_unsupervised` modal bodies verbatim per prompt.
    `obj_unsupervised` no longer renders the target-centric
    `ObjectiveFooter` — new `UnsupObjectiveFooter` ("no target, can't
    pick one") in its place.
  - **Silhouette guard hardened (engine_v2/fitness.py
    `_score_silhouette`).** Winsorize the score to its 2.5–97.5
    percentile range, then recompute std on the winsorized vector;
    if `std_w < 1e-9` → worst floor. This catches the
    `protected_div`-by-self failure mode (huge raw std + a small
    handful of outliers → a 98:2 "perfect" split with silhouette ~1).
    Min-cluster floor also raised from 10% to 30% of n.
    `test_unsupervised_silhouette_kills_self_divide_outlier_split`
    locks the fix.
  - **Gene-based unsupervised synthesis (engine_v2/synthesize.py).**
    Earlier unsup winners could be no-Select trees that did the
    splitting work via `Split(..., predicate="stage_late")` — GENES=0,
    silhouette 1.0 trivially, no actual gene discovery. Fixed by:
    `_grow_matrix` now takes `objective_target`; under unsup every
    Matrix leaf is `Select(MatrixTerminal(), sampled FeatureSet)`
    (never bare `MatrixTerminal`), `Search` is gated off, and the
    `_grow_vector` depth-floor becomes `Reduce(Select(M, [...]), agg)`.
    `_grow_vector`'s Split branch forces `predicate="score"` under
    unsup; point mutation's `predicate` spot is suppressed entirely
    for unsup so a `"score"` can't be flipped to `"stage_late"` mid-run.
    Belt-and-braces fitness floor: `fitness_fn` and `evaluate_holdout`
    return `WORST_FITNESS` when `objective.target == "none"` and
    `program.feature_ids() == []`. Two new tests
    (`test_unsupervised_programs_always_use_gene_select` over 120
    programs × 20 mutations, and `test_fitness_floors_no_select_program_under_unsup`
    on the literal bug-shape program).
  - **Verification.** pytest 94/94, tsc clean. Live unsupervised
    TCGA smoke (10 gens × 60 pop, prefilter 2000) across four seeds
    shows every winner is gene-based (1–5 genes), Split predicate is
    `score` (no stage_late), held-out silhouette ≈ 0.55–0.60 (not the
    trivial 1.0), and the discovered split aligns blind with MSI at
    AUROC up to **0.774** — the engine rediscovered the MSI subtype
    without ever seeing the label.
- **Chunk 6 follow-up (done — generalisation-aware unsup silhouette +
  plain-reading verdict):**
  - **OOS silhouette in `cv_score` AND `evaluate_holdout`.** Earlier
    unsup runs scored silhouette IN-SAMPLE inside CV (the KMeans was
    fit on the same fold-test points it then evaluated) — so programs
    that overfit looked great during selection and collapsed on the
    final held-out (often to -1.0 worst floor). New
    `V2Objective._score_oos_silhouette(train_scores, test_scores)`
    fits the winsorize bounds, mean/std, and KMeans centres on TRAIN
    scores only; test scores are clipped + standardised with the
    train statistics and assigned to the train centres via
    `KMeans.predict`. Guards live on the TEST side (≥30% per cluster,
    near-constant after clipping → worst). `cv_score` per fold now
    uses `(tr, te)` not just `te`. `evaluate_holdout` gained an
    optional `ctx_train` kwarg; under unsup it executes the program
    on TRAIN and TEST then calls the OOS scorer. MSI / TMB paths
    ignore the new kwarg (those metrics are already on the held-out
    vector by construction).
  - **OOS null too.** `unsup_random_null` gained `ctx_train` and
    forwards it into `evaluate_holdout` — otherwise the random
    Vector-only nulls would be scored in-sample while the winner is
    scored OOS, deflating the p-value. The pipeline now plumbs
    `ctx_train` into both the winner's `evaluate_holdout` and the
    null-distribution loop.
  - **Plain-reading verdict at the top of the Result panel.** New
    `<ResultVerdict>` rendered first inside `ResultPanel`. Per-target
    thresholds (MSI 0.75, TMB 0.30, unsup 0.30) plus `p < 0.05` gate
    a green "Real result: it holds up on unseen patients (X) and
    beats chance (p Y)" callout vs an amber "This run found nothing
    reliable: held-out X (fell apart on unseen patients) · p Y (a
    random program beats it ~N% of the time). Don't read the genes
    as a discovery." For unsup the green callout appends one of three
    tails based on `posthoc.msi_auroc` (≥0.75 → "rediscovered the
    subtype"; <0.75 → "doesn't line up with MSI"; null → "too few
    held-out labels"). Constants `HOLDOUT_THRESHOLD`,
    `P_VALUE_SIGNIFICANT`, `POSTHOC_MSI_ALIGN` at the top of
    `Lab.tsx` are tunable.
  - **Honest unsup numbers.** With OOS silhouette as the selection
    metric the GP now selects for *consistent* two-cluster structure
    rather than splits that happen to align with MSI via in-sample
    overfitting. seed=13 used to report in-sample silhouette 0.6 +
    MSI-AUROC 0.774 (alignment was partly an in-sample artefact);
    now it reports OOS silhouette 0.900, p 0.024, MSI-AUROC 0.546 —
    a real generalising cluster split that doesn't happen to be MSI.
    The verdict surfaces that mismatch in plain words.
  - **Four new tests** in `tests/test_engine_v2.py`:
    `test_oos_silhouette_rewards_consistent_split` (train+test both
    bimodal → ≥0.70), `test_oos_silhouette_kills_non_generalising_program`
    (train bimodal + test noise → drops >0.3 below in-sample),
    `test_oos_silhouette_train_only_constant_floored`,
    `test_oos_silhouette_too_small_floored`. pytest 98/98 pass.
- **Chunk 6 follow-up (done — "What does this mean?" verdict
  breakdown, presentation-only):**
  - The verdict callout now has a collapsed-by-default disclosure
    underneath it (mirrors `ParameterFlow.tsx`'s collapsible — button
    + chevron + `aria-expanded` + `aria-controls` + `useId`-keyed
    panel + closed-on-first-render).
  - Expanded content branches on `objective_spec.target`:
    `<MSIBreakdownContent>` and `<TMBBreakdownContent>` use a one-
    question shape ("Is this result real and useful?") with two
    bullets — held-out separation/association and beats-chance —
    plus a `→ Real and useful.` / `→ Not reliable…` capstone tied to
    the same generalises/significant booleans the headline uses.
    `<UnsupBreakdownContent>` uses the two-question shape from the
    prompt verbatim ("Is the split real?" / "Is that split the MSI
    subtype?") with the hair-colour analogy at the bottom. All
    numbers route through `fmtFit` so non-finite renders as em-dash.
- **Chunk 6 follow-up (done — iterative unsupervised discovery,
  "peel off axes"):**
  - **Engine.** `engine_v2/pipeline.py` gained
    `_residualise_matrix(M, residualize_scores)` — vectorised OLS via
    `np.linalg.lstsq` over `[intercept, *priors]`. Rows missing a
    prior score are dropped BEFORE the train/test split so train and
    test see the same residualised feature space. Both
    `run_v2_pipeline` and `run_v2_pipeline_streaming` gained
    `residualize_scores: pd.DataFrame | None = None` (no-op for
    MSI/TMB; only the unsup API worker sets it). Both functions now
    emit `winning.full_scores` + `winning.full_sample_ids` (full-
    cohort per-patient scores) alongside `holdout_scores` — these
    feed the chain's next residualisation step.
  - **API.** `RunRequest.residualize_against: list[str] | None`.
    `_assemble_residualize_df` validates each prior (must exist + be
    done + `target=="none"` + have `full_scores`); turns
    `_json_finite`'s `None` back into `NaN` so the pipeline's NaN
    filter drops those patients during residualisation. `post_runs`
    validates the chain BEFORE spawning the worker — supervised
    target + priors → HTTP 400; unknown / wrong-target / no-scores
    prior → HTTP 400 with the specific reason. `_worker` gained
    `residualize_against`; only assembles + forwards the DataFrame
    for unsup runs. Chain is in-memory only — valid within a single
    server session.
  - **Frontend.** Lab.tsx grew an `axes: {run_id, result}[]` state.
    `start()` refactored into `launchRun(opts?)` so a fresh Run
    (`launchRun()`, clears `axes`) and `findNextAxis()`
    (`launchRun({residualize_against: axes.map(a=>a.run_id)})`,
    preserves the chain) share the same SSE wiring. The SSE `done`
    handler appends unsup results to `axes`. New `<DiscoveredAxes>`
    section under the Evaluator renders an ordered stack of compact
    `<AxisCard>`s (accent border when both verdict gates pass;
    holdout / p / revealed opaque IDs / post-hoc MSI AUROC + TMB
    |spearman| in one line) plus a "Find next axis →" button. The
    `TIPS.discoveredAxes` tooltip flags the linear-residualisation
    heuristic and the in-memory-chain caveat.
  - **Tests.** Extended `tests/test_api_airgap.py`:
    `test_unsupervised_run_emits_posthoc_alignment` now asserts
    `winning.full_scores` + `winning.full_sample_ids` are populated;
    new `test_residualisation_chain_runs_and_stays_airgap_clean`
    posts axis 1 then axis 2 with `residualize_against=[axis1]` and
    confirms the second payload is opaque-only and carries its own
    `full_scores`; new `test_residualize_against_unknown_id_returns_400`
    + `test_residualize_against_rejected_on_supervised_target`. 101 /
    101 pytest pass.
  - **Verified live on TCGA** (seed=13, 6 gens × 50 pop, prefilter
    2000):
    - Axis 1 (5 genes, no overlap with MSI): holdout 0.597, p 0.065,
      MSI-AUROC 0.552, TMB-|spearman| 0.009.
    - Axis 2 (residualised against Axis 1; 5 different genes):
      holdout 0.653, p 0.097, **MSI-AUROC 0.706**, TMB-|spearman|
      0.248. Zero shared genes between the two axes — the
      residualisation freed the search to land on an orthogonal
      direction that's more MSI-aligned than the first.
- **Chunk 7 (done — second dataset: HNSC + HPV detection):**
  - **Dataset pipeline.** New `data_pipeline/{download_hnsc,build_hnsc}.py`
    mirror the colorectal scripts for TCGA HNSC PanCancer Atlas
    (`hnsc_tcga_pan_can_atlas_2018`, LFS-resolved media URL). The
    build script derives `hpv_status` ∈ {HPV+, HPV−} from whichever
    column carries it: first `data_clinical_sample.txt::HPV_STATUS*`,
    then `data_clinical_patient.txt::HPV_STATUS`, then the `SUBTYPE`
    suffix fallback (e.g. `HNSC_HPV+` / `HNSC_HPV-`). On the real
    cohort the resolved source is `patient.SUBTYPE`. Writes
    `data/processed_hnsc/{clinical,expression}.parquet` (gitignored).
    HNSC schema constants live in `data_pipeline/schema.py` next to
    the colorectal ones; final usable cohort is 487 samples (72 HPV+ /
    415 HPV−).
  - **Loader.** `dsl.Load` already handled directory paths via its
    `else` branch; the only change is that `Cohort.labels` now picks
    up `hpv_status` if present. `Load(schema.HNSC_PROCESSED_DIR)`
    returns the HNSC cohort with `hpv_status` in `cohort.labels`.
  - **HPV objective (engine_v2).** `V2Objective.target` widened to
    `Literal["msi","tmb","none","hpv"]`. New `HPV_OBJECTIVE`
    (binary=True, worst=0.5) routes through the SAME
    orientation-agnostic AUROC machinery as MSI; `score_scalar`
    mirrors MSI's `abs(value)` orientation. `objective_from_spec`
    dispatches `hpv + auroc/auroc_omni`. No new operators; no
    other engine surface changes.
  - **API per-(dataset, target).** `_prepare_lab_data(target, dataset)`
    keyed by `f"{dataset}:{target}"`. New `(hnsc, hpv)` branch loads
    HNSC, restricts to called HPV samples + complete expression,
    anonymises, sets `y = (hpv_status == "HPV+").astype(int)`. New
    `(hnsc, none)` unsup branch keeps every patient with complete
    expression and carries the HPV label aside via `extra_labels` so
    `_compute_unsup_posthoc` can compute orientation-agnostic
    `hpv_auroc` (mirrors the `msi`/`tmb` branches; new key
    `Posthoc.hpv_auroc?`). `RunRequest.dataset` + `Run.dataset` +
    server-side `DATASET_TARGETS = {coadread: {msi,tmb,none},
    hnsc: {hpv,none}}` validation rejects mismatched combos with a
    400 at the API boundary before any data prep.
  - **Per-dataset reference sets.** Replaced flat `REFERENCE_SETS`
    with `REFERENCE_SETS_BY_DATASET`. Colorectal keeps MMR + immune;
    HNSC ships `p16 = ["CDKN2A"]` and a 20-gene standard
    cell-cycle / E2F-target core (MCM2–7 / PCNA / CDK1 / CCNE1 /
    CCNB1 / CDC6 / CDC20 / MKI67 / TOP2A / RRM2 / TYMS / FOXM1 /
    E2F1 / BUB1 / AURKB). `EvaluateRequest.dataset` is required and
    cross-dataset reference-set requests return 400.
  - **Frontend dataset registry.** `Target` widened to include `hpv`;
    new `DatasetId = "coadread" | "hnsc"`. `DATASET_REGISTRY` in
    `Lab.tsx` drives objective cards, biology panel, and reference-set
    keys per cancer. A `<CancerSelector>` segmented control mounts
    above the Objective row. New `<HPVBiologyPanel>` mirrors
    `<MMRBiologyPanel>` exactly and inlines `hpv_reference.svg`
    verbatim (RB / E2F / p16 / p53 / E6+E7 explainer + readout
    table + airgap-side disclosure). Switching cancers resets the
    chain + result state and snaps the objective to the dataset's
    default.
  - **HPV verdict + "?" coverage.** `FITNESS_LABEL_BY_TARGET.hpv =
    "separation (AUROC)"`; `FITNESS_TIP_BY_TARGET.hpv` reuses the
    MSI-style AUROC explanation worded for HPV+/−;
    `HOLDOUT_THRESHOLD.hpv = 0.75`; `OBJECTIVE_PRESETS.hpv =
    {target:"hpv", metric:"auroc"}`. New `<HPVBreakdownContent>` adds
    a supervised one-question shape ("Is this result real and
    useful?") — no two-question / hair-colour shape for HPV. Two new
    paramHelp modals: `obj_hpv` (verbatim per prompt: what it
    optimises, how it's scored, represented as, honest "detection /
    recovery of a known viral signature — not new causation" note).
    `<UnsupAlignmentTail>` + `<UnsupBreakdownContent>` +
    `<PosthocAlignment>` all branch on which named label the cohort
    carries (HPV vs MSI/TMB) so HNSC unsup runs surface a real
    alignment number instead of "(too few held-out labels)".
  - **Per-dataset Reveal.** `<Evaluator>` reads its reference-set
    toggle keys from `DATASET_REGISTRY[dataset].refSetKeys`. HNSC
    shows **p16 | cell_cycle**; colorectal stays on MMR | immune.
    `<MLH1SuccessCallout>` + `<TMBRankPanel>` gated to
    `dataset === "coadread" && target === "tmb"`. The "Unsupervised
    run — no reference set" footnote reads dataset-agnostically.
  - **Live verification (TCGA).** MSI seed 7 — held-out 0.959, p 0.048.
    HPV seed 7 — held-out 0.869, p 0.048. HPV seed 11 — held-out
    **0.970**, p **0.024**, winner overlaps `cell_cycle` via **MCM5**.
    HPV seed 17 — held-out 0.957, MCM5 again. HPV seed 3 — E2F2 + RPA2
    (cell-cycle/E2F + DNA replication, biologically on-target but not
    in the literature set). Colorectal + airgap tests untouched and
    green.
  - **Airgap tests extended.** `tests/test_api_airgap.py` fake_prep
    handles `(target, dataset)`; new tests:
    `test_hnsc_hpv_run_is_airgap_clean`, `test_hnsc_rejects_msi_target`,
    `test_coadread_rejects_hpv_target`,
    `test_evaluate_rejects_hnsc_reference_in_coadread`. 105 / 105
    pytest pass.
- **Chunk 7 follow-up (done — HPV-marker-rank diagnostic):**
  Mirror of the TMB-rank diagnostic for HNSC. New `validate/hpv_rank.py`
  ranks every gene by single-gene orientation-agnostic AUROC vs the
  HPV+/HPV− label on the engine's TRAIN split (`make_split(seed=42,
  test_size=0.3, stratify=True)` so the diagnostic never reads test
  data — same discipline the engine uses). Vectorised
  Mann-Whitney / rank-sum formula:
  `auroc = (S_pos − n_pos·(n_pos+1)/2) / (n_pos·n_neg)` then
  `max(AUROC, 1−AUROC)`; sub-second over ~20k genes. New
  `GET /diagnostic/hpv-rank` endpoint mirrors `/diagnostic/tmb-rank`
  (lazy import, `_HPV_RANK_CACHE`, `_json_finite()`-wrapped). Three
  synthetic-cohort tests in `tests/test_hpv_rank.py`. Frontend:
  `HPVRankDiagnostic` interface + `getHPVRankDiagnostic()` in
  `web/lib/api.ts`; new `<HPVRankPanel>` in `Lab.tsx` reuses the
  existing `<RankList>` chrome and renders **p16 + cell-cycle + top-N
  separators** with the prompt's verbatim caption, gated to
  `dataset === "hnsc" && target === "hpv"`. `TIPS.hpvRank` documents
  the heuristic. Live TCGA cohort: N(TRAIN) = 340 (HPV+ 50 / HPV− 290),
  20,218 genes ranked. **CDKN2A AUROC 0.877, rank 99 / 20,218 (top
  0.5%)** — meaningfully recoverable but 98 genes outrank it. Best
  cell-cycle ranks: MCM5 (22), MCM2 (28), MCM6 (38), MCM3 (72),
  PCNA (78), E2F1 (99, tied with CDKN2A). Top-10 led by TCAM1P /
  C19orf57 / ARHGEF33 / STAG3 / SMC1B (meiosis-cohesin) + RPA2
  (DNA replication — what the engine actually picked at seed 3). 108 /
  108 pytest pass; airgap unaffected (engine_v2 + dsl + engine
  untouched; structural scan still only walks `engine/`).
- **Chunk 7 follow-up (done — UX polish: plain "?" copy, no all-caps,
  program-graph clipping):** Presentation-only sweep across the Lab.
  - **Objective-aware tooltips.** Added four per-target / per-dataset
    records next to `FITNESS_TIP_BY_TARGET`:
    `HELD_OUT_TIP_BY_TARGET`, `PERMUTATION_P_TIP_BY_TARGET`,
    `REFERENCE_SET_TIP_BY_DATASET`, `POSTHOC_TIP_BY_DATASET`,
    `DISCOVERED_AXES_TIP_BY_DATASET`. Held-out / permutation p /
    posthoc / discovered-axes / reference-set `?` all switch on the
    active objective or dataset so HPV runs no longer see MSI/TMB
    wording. `<DiscoveredAxes>` and `<PosthocAlignment>` receive
    `dataset` (or infer it from `posthoc.hpv_auroc` presence) and look
    up the right copy.
  - **Plain English everywhere.** Rewrote the verbose / typo'd
    legacy `TIPS.heldOut`, `TIPS.permutationP`, `TIPS.posthoc`,
    `TIPS.discoveredAxes`, `TIPS.referenceSet` to crisp sentences with
    one-line glosses for jargon (AUROC, held-out, permutation). Fixed
    the `seed` PARAM_TIPS typo ("xact" → "the exact"). The unsupervised
    fitness-curve tip no longer hardcodes "(MSI? TMB?)" — it now says
    "the named label this cohort carries".
  - **ObjectiveIntro / UnsupObjectiveFooter dataset-aware.**
    `ObjectiveIntro` now lists MSI / TMB / HPV (was MSI / TMB only).
    `UnsupObjectiveFooter` mentions both MSI (colorectal) and HPV
    (head & neck) as legitimate downstream alignments instead of MSI
    only.
  - **No all-caps anywhere.** Stripped Tailwind's `uppercase` utility
    from all 13 eyebrow / table-header sites in `Lab.tsx` +
    `PasteToDraw.tsx`. Removed `textTransform: "uppercase"` from the
    Tier label in `ProgramGraph.tsx`. Upcased the underlying literal
    strings that were relying on the CSS (`Winning program`,
    `Revealed genes`, `Opaque ID` / `Symbol` / `Matched` table
    headers, rank-table `Symbol` / `Correlation` / `Rank / N` /
    `AUROC`). `grep -rn "uppercase\|textTransform" web/app` returns
    zero hits.
  - **Program-graph clipping fix.** `ProgramGraph.tsx` gained a
    remount `key={fitKey}` derived from `width × height × node-count
    × edge-count` so ReactFlow's one-shot `fitView` re-runs every
    time the laid-out program changes (winner ↔ candidate click,
    paste, dataset swap). Lowered `minZoom` from 0.4 to 0.2 so
    `fitView` can scale a full two-tier graph down to fit the card
    width. Confirmed no `overflow: hidden` on the wrapping
    `<SectionCard>`. `<ProgramGraph outputLabel>` now dispatches by
    target — `MSI-H probability` / `HPV+ probability` /
    `TMB association` / `cluster score` instead of MSI-only.
- **Chunk 7 follow-up (done — detection vs recovery + AUROC naming +
  ban whole-matrix Reduce + per-gene rank in Reveal):**
  - **Engine.** Motivating live run produced
    `Combine(Reduce(M, mean), Reduce(Select(M, …), mean), add)` — a
    bare global-mean detector with a 7-gene additive tail; held-out
    AUROC 0.953 with `0/7` cell-cycle overlap. The dominant term
    detected HPV from bulk expression, not from gene choice. Fix:
    `engine_v2/synthesize.py` makes the Select-wrapping rule
    **unconditional** for every objective (was unsup-only). Every
    Matrix leaf is now `Select(MatrixTerminal, FeatureSet)`; the
    Vector depth-floor returns `Reduce(Select(M, …), agg)`; the
    bare-MatrixTerminal fall-through is gone. Belt-and-braces in
    `engine_v2/fitness.py`: new `_has_bare_matrix_reduce(program)`
    walks the tree and any Reduce on a bare MatrixTerminal floors to
    `WORST_FITNESS` for every objective (was unsup-only). The
    `no Select → WORST_FITNESS` floor is also no longer gated to
    unsup. Two test fixups: the closed-Matrix leaf at depth 0 is now
    `Select(MatrixTerminal, …)`; the depth-budget contract is loosened
    by 1 because the mandatory Select adds a level. Live re-run on
    real HNSC (seed 7): winner is now `Reduce(Select(M,[g17705]),max)`
    — a single-gene detector, AUROC 0.964, no bare `Reduce(M,…)`.
  - **Detection vs recovery (presentation).** `HPVBreakdownContent`
    + `MSIBreakdownContent` success copy changed from "the engine
    recovered the HPV viral signature" / "Real and useful." to
    "Real and useful — it separates HPV+ from HPV− (or MSI-H from
    MSS) on patients it never saw." Both breakdowns gained a muted
    secondary line — *"This is a detection result. Whether the
    engine found the known marker genes (CDKN2A/p16, the cell-cycle
    program / MMR / immune) is a separate question — see Reveal &
    evaluate below."* The top-level callout already used detection-
    only wording. Unsupervised "rediscovered the viral signature
    blind" stays — that's a post-hoc *alignment* claim, gated on
    `posthoc.hpv_auroc ≥ 0.75`.
  - **Name AUROC where the metric IS AUROC.** New
    `HELD_OUT_LABEL_BY_TARGET` — Held-out card label is `Held-out
    AUROC` for MSI / HPV, `Held-out (|spearman|)` for TMB,
    `Held-out (silhouette)` for unsup. `HELD_OUT_TIP_BY_TARGET`
    rewritten to name AUROC explicitly with a one-line gloss for
    MSI/HPV and to call out "Not AUROC: TMB is a continuous label /
    there's no label here" for TMB/unsup. Held-out bullet inside
    each breakdown switched `Held-out separation` → `Held-out AUROC`
    with the same gloss.
  - **Per-revealed-gene rank in Reveal & evaluate.** `/evaluate`
    request gains optional `target`; response rows gain optional
    `rank` / `total` / `single_gene_metric` / `metric_kind`
    (`"auroc"` for HNSC/HPV, `"spearman"` for coadread/TMB). New
    `_gene_rank_lookup(dataset, target)` precomputes a
    `{symbol → rank/metric}` map on first call by reusing
    `validate/hpv_rank.py`'s `_auroc_per_column` (on the engine's
    TRAIN slice) and `validate/tmb_rank.py`'s `_spearman_per_column`;
    cached for the process lifetime. Frontend: `EvaluateRow` gained
    the optional fields; `postEvaluate` passes `target`; the
    Revealed-genes table now shows a "Single-gene rank" column with
    `{rank} / {total}` plus `AUROC X.XXX` (HPV) or `ρ X.XXX` (TMB).
    Column appears only when at least one row carries a rank — MSI
    and unsupervised omit gracefully. New `TIPS.singleGeneRank`
    explains the column: "rank near 1 = real alternate marker; high
    rank = only helps in combination."
  - **Verified live.** HPV seed 11 on real HNSC (10 gens × 80 pop):
    winner `Reduce(Select(M,[g06264,g08575,g10783,g14446,g18989,
    g19553,g05351,g10525]),mean)`, held-out AUROC 0.937, p 0.024.
    Cell-cycle overlap **0/8**, BUT every revealed gene is a top-2000
    single-gene HPV separator on its own: FANCI (rank 996, AUROC
    0.779), INPP5B (332, 0.830), MIR924HG (237, 0.841), RAD1 (359,
    0.825), UBD (1808, 0.746), WHSC1 (402, 0.821), DYRK1A (246,
    0.840), MCCC1 (1334, 0.763). The story is now legible: real
    alternate markers (DNA-damage-response / chromatin players), not
    the canonical MCM / PCNA shortcut. 108 / 108 pytest pass; tsc
    clean; airgap tests untouched.
- **Chunk 7 follow-up (done — coherence prior + supervised peel-off +
  full-ranking Result + Unsup card retired):** Five coordinated
  landings that rebuild the Lab's reveal story around the full single-
  gene ranking.
  - **Coherence prior (engine).** New
    `engine_v2.fitness._coherence_score(program, ctx)` returns the
    mean absolute pairwise correlation among the program's Select'd
    opaque columns (range [0, 1]; names no gene or pathway).
    `fitness_fn` gained `coherence_weight: float = 0.0`; when > 0,
    fitness becomes `base − λ·n_nodes + w · coherence`. Threaded
    through `gp.py` + both `pipeline.py` entry points with default 0
    so existing runs stay byte-for-byte unchanged. API: new
    `RunRequest.coherence: bool = False`; `_worker` translates to
    `COHERENCE_DEFAULT_WEIGHT = 0.10` (modest enough that separation
    still dominates). Frontend: a Parameters checkbox "Prefer
    coordinated gene modules" — default OFF, posted in the run body.
    `TIPS.coherence` carries the prompt's verbatim copy.
  - **Peel-off for supervised objectives.** Both pipeline entry
    points now persist `winning.full_scores` + `full_sample_ids` for
    every objective (the `is_unsup` gate is gone in both spots).
    `api.app.post_runs` no longer requires `target=="none"` for
    `residualize_against`; `_assemble_residualize_df` validates that
    every prior shares both `(dataset, target)` with the new run.
    Frontend `<DiscoveredAxes>` mounts after ANY run (not just unsup)
    and the SSE `done` handler appends to the chain for every
    objective. Live verified on HNSC HPV seed 11: Axis 1 → AUROC
    0.941 / 3 genes; Axis 2 (residualised) → AUROC 0.875 / 2
    different genes; zero overlap. Updated airgap test:
    `test_residualize_against_works_on_supervised_targets` posts MSI
    Axis 1 → Axis 2 and asserts payload stays opaque + carries
    full_scores.
  - **Full-ranking endpoint.** New `GET /diagnostic/full-rank?dataset=
    &target=` returns `{n_samples, n_pos, n_neg, n_genes, metric_kind,
    ranks: [{opaque_id, score, rank}, …], reference_marks: [{opaque_id,
    symbol, set_name, rank, score}, …]}`. The `ranks` list is
    opaque-only (~20k rows). `reference_marks` is the small reference
    gene set (CDKN2A + 20 cell-cycle for HNSC; MMR + immune for CRC),
    looked up via the sealed map on the API side and revealed up-front
    so the rank track can label them — the whole map never crosses
    the wire. Computed on the same TRAIN slice the engine sees
    (`make_split(seed=42, test_size=0.3, stratify=binary)`), using
    the rank-sum AUROC formula for binary targets and signed Spearman
    for TMB. Unsupervised → 404 (no label to rank against). Cached
    per (dataset, target) for the process lifetime.
  - **New `<RankingResult>` UI.** Replaces the prior
    Evaluator + TMBRankPanel + HPVRankPanel + MLH1SuccessCallout for
    supervised targets. Renders, top-to-bottom:
    - Rank chart: a labelled lollipop chart on a log-rank axis
      (gridlines at 1 / 10 / 100 / 1k / 10k / N), with the winner
      gene(s) as accent-teal dots and each reference gene as a
      colour-coded dot. Labels stagger across 4 rows to avoid
      collisions near the top.
    - Highlighted-genes table with three-way colour-coded "Source"
      column (winner / p16 / cell_cycle for HNSC; winner / immune /
      MMR for CRC) — p16 reads visibly distinct from cell_cycle
      via a separate deep-gold palette. Columns: Source · Symbol ·
      Opaque ID · Rank/N · Percentile · metric. Percentile shows
      "top 0.5%" for high ranks and flips to "bottom 21%" past the
      midpoint (2 sig figs).
    - Collapsed `<RawRankingDisclosure>` ("Raw anonymous ranking —
      proof the ranking is computed blind on opaque IDs") wraps the
      browsable opaque-ID list + search box (top-200 with substring
      filter). Caption restates the airgap purpose.
    - GP-vs-diagnostic note (verbatim per prompt) closes the panel.
  - **Unsup card hidden, backend kept.** `DATASET_REGISTRY` drops
    the `none` objective entry for both datasets — UI now shows
    coadread → {MSI, Mutation burden} and hnsc → {HPV detection}.
    `UNSUP_OBJECTIVE`, the unsup pipeline branches, the
    `target === "none"` frontend code paths (verdict / breakdown /
    post-hoc / Evaluator fallback), and `tests/test_engine_v2.py`
    stay intact — just unreachable from the UI.
  - **108 / 108 pytest pass; tsc clean; airgap tests still green.**
- **Chunk 7 follow-up (done — Ranking UI polish):** Presentation-only
  pass on `<RankingResult>`. New three-way `RANK_COLORS` palette
  (winner accent teal / p16 deep gold / cell_cycle muted amber, plus
  immune/MMR aliases for CRC) drives every coloured surface — chart
  dots, table backgrounds, "Source" badges. Highlighted-genes table
  gains a "Percentile" column (`top X%` / `bottom X%` with 2 sig figs)
  so CDKN2A reads "99 / 20,218 · top 0.5%" and CCNE1 reads
  "15,918 / 20,218 · bottom 21%". `<RankTrack>` rewritten with a log
  axis (gridlines at 1 / 10 / 100 / 1k / 10k / N), a 4-row label
  stagger that avoids 60px-radius collisions, and an inline
  three-colour legend in the chart header. Raw opaque-ID browse
  table moved into a collapsible disclosure ("Raw anonymous ranking
  — proof the ranking is computed blind on opaque IDs"), closed by
  default. Page reads verdict → metric cards → log chart → highlighted
  table → collapsed raw → GP-vs-diagnostic note. tsc clean; no
  engine/API/airgap changes.
- **Chunk 7 follow-up (done — peel-off leakage fix: train-only
  residualisation + FitApply train-fit + leakage guard):** A peel-off
  Axis 2 HPV run produced
  `FitApply(Reduce(Select(M,[g16970]),median),hpv)` (SPACA1) at
  held-out AUROC 0.963 — impossible for a single-gene monotonic
  program whose gene's honest single-gene AUROC is 0.510. Root
  cause: residualisation was fit on the FULL cohort BEFORE
  `make_split`, so the OLS projection saw the held-out rows and
  smeared target signal into them.
  - **`engine_v2/pipeline.py`** — `_residualise_matrix` replaced
    with three helpers: `_align_priors` (drops rows missing a
    prior — defining the cohort, not leakage), `_fit_residualise_beta`
    (OLS of each gene on `[intercept, *priors]` fit on TRAIN rows
    only), `_apply_residualise` (applies train-fit β to the full
    M so train + test sit in the same residualised space without
    test ever being seen by the fit). Both `run_v2_pipeline` and
    `run_v2_pipeline_streaming` reordered: align → split → fit β
    on `M.loc[split.train_ids]` → apply β to full M → prefilter /
    GP / `evaluate_holdout`. `winning.full_scores` re-executes the
    winner on the train-only-residualised M with empty labels (the
    raw inner score for FitApply winners — monotonic with the
    fitted prediction in the 1-D case, so the chain stays
    consistent).
  - **`engine_v2/nodes.py`** — `ExecContext` gained
    `fit_ctx: ExecContext | None = None`. `FitApply.execute` now
    runs the inner subtree on `fit_ctx` (train) to fit LR/OLS and
    applies the FROZEN model to the test inputs (no fitting on
    test labels). Binary `hpv` now routes through the same
    logistic branch as `msi`. Identical behaviour in the single-
    score monotonic case (LR remains monotonic) so existing AUROCs
    don't move; the discipline is in place for any future multi-
    input FitApply.
  - **`engine_v2/fitness.py`** — `LEAKAGE_TOLERANCE = 0.02`.
    `evaluate_holdout` wraps the test `ExecContext` with
    `fit_ctx=ctx_train` when `ctx_train` is provided, then runs a
    backstop check: if the program is `_single_gene_monotonic`
    (one gene; no Combine/Split) AND its held-out AUROC exceeds
    the gene's single-gene AUROC by > tolerance, floor to
    `WORST_FITNESS`. Genuine multi-gene synergy is unaffected.
  - **Tests** — `tests/test_engine_v2.py` gained
    `test_peeloff_residualisation_uses_train_rows_only`
    (synthetic cohort + target-correlated prior; asserts the
    single-gene-monotonic held-out AUROC is within tolerance of
    the gene's honest single-gene AUROC under the new pipeline)
    and `test_leakage_guard_floors_contaminated_single_gene_winner`
    (guard sanity-check on an honest program). 114 / 114 pytest
    pass; airgap suite untouched.
  - **Verified live (TCGA HNSC, HPV + coherence ON).** Across
    seeds {3, 7, 11, 13} every Axis 2 is multi-gene, no
    SPACA1-style runaway wins, and Axis 2 held-out ≤ Axis 1 +
    tolerance: seed 11 → Axis 1 AUROC 0.941 / 3 genes → Axis 2
    AUROC 0.923 / 8 genes (zero gene overlap). The
    residualise-before-split mechanism is closed.
- **Chunk 7 follow-up (done — coordinated modules: ranked group
  view + ref-set highlights):** Coherence-on runs now expose a
  ranked list of the coordinated gene modules the GP explored,
  sorted by combined held-out AUROC — so a real co-expressed
  program (e.g. cell_cycle) surfaces as a GROUP even when no
  single gene is the top separator.
  - **API.** New `GET /runs/{run_id}/modules` endpoint
    (`api/app.py::_compute_module_ranking`). Reproduces the run's
    EXACT train/test split via the persisted `full_sample_ids` /
    `holdout_sample_ids` so "held-out" actually is held-out;
    harvests distinct candidates' `gene_ids` sets (≥ 2 genes)
    across every persisted generation (engine_v2 already stores a
    flat `gene_ids` list per candidate); for each module computes
    **combined held-out AUROC** = orientation-agnostic AUROC of
    the per-patient mean across the module's genes vs the held-
    out label (parameter-free — nothing to fit, nothing to leak;
    `|spearman|` for TMB), **coherence** = mean abs pairwise
    correlation on TRAIN (the same quantity the prior rewards),
    and a per-gene single-gene AUROC + rank/N. Each module also
    carries **`ref_sets: list[str]`** — the dataset's reference
    sets (HNSC: p16 / cell_cycle; CRC: MMR / immune) whose
    opaque IDs intersect that module, resolved ONCE via the
    sealed map (bounded reveal of a small known set; same
    pattern `/diagnostic/full-rank` uses for `reference_marks`).
    Sorted by combined held-out desc; non-finite sinks to the
    bottom. `Run` dataclass gained `coherence: bool`; `/runs/{id}`
    now surfaces it (along with `dataset`). Endpoint returns 425
    while running, 404 for unknown, 400 for unsupervised runs
    (no target to evaluate against).
  - **Frontend.** New `<ModuleRankingPanel>` rendered after
    `<RankingResult>`, gated to `target !== "none" && coherence
    && runId`. Sortable header (Combined AUROC / Coherence);
    each collapsed row carries a small **badge per matched
    reference set** and is **tinted** using the existing
    three-way `RANK_COLORS` palette (winner accent teal /
    single-marker sets like p16/immune deep gold / broader
    proliferation sets like cell_cycle/MMR muted amber) — so a
    cell_cycle-heavy module is scannable without expanding.
    Expanding a row calls **`POST /reveal`** with just that
    module's gene IDs (bounded — never the whole map) so every
    per-gene row gets a symbol (no more "—" for module genes
    outside the already-revealed subset, e.g. `g09647 → LIG1`).
    Expanded view gains a "Source" column tinted per-symbol's
    reference-set membership.
  - **Airgap.** Module ranking carries opaque IDs + scores +
    `ref_sets` set NAMES only — no gene symbols. Reference-set
    membership comes from the same bounded sealed-map lookup
    `/diagnostic/full-rank` already uses. Per-module symbols
    revealed lazily per expanded module via `/reveal`. New
    airgap tests in `tests/test_api_airgap.py`:
    `test_modules_endpoint_returns_opaque_only_modules` (assert
    `ref_sets` list present, opaque IDs only, sorted desc by
    combined AUROC), plus rejection tests for unsupervised /
    unknown / mid-run scenarios. 114 / 114 pytest, airgap suite
    25 / 25, `tsc --noEmit` clean.
  - **Verified live (HNSC HPV, seed 11, coherence ON).** 303
    distinct modules; the rank-1 module (size 11, combined
    held-out AUROC **0.978**) carries the cell_cycle badge and
    beats the winner's 0.941. Expanding it resolves all 11
    symbols on the wire: MCM2, CENPQ, FAM111B, KIF2C, NDC80,
    RAD1, USP1, REXO2, ING4, KIAA1407, MMP17 — a proliferation
    / DNA-replication module the single-gene picture missed.
    Exactly the "group view can out-score its individual genes"
    demonstration the design called for.
- **Chunk 7 follow-up (done — separate group vs individual views in
  the coherence-run UI):** Presentation-only cleanup so no single
  table mixes group and individual scores under the same column.
  - **Single-gene "Highlighted genes" table** retags winner-source
    rows from `"winner"` → `"in winning program"` and adds a one-line
    caption underneath: *"These are single-gene ranks of individual
    genes — including the genes inside the winning program and the
    reference markers. The winning program's combined (group) score is
    in the Result cards above and the Coordinated modules below."*
    The winner's combined held-out is unchanged in the Result cards.
  - **InfoTip copy rewritten** for both panels. `TIPS.rankingResult`
    + `TIPS.rankingHighlighted` lead with INDIVIDUAL; `TIPS.module
    Ranking` leads with GROUP and explicitly states that the modules'
    re-scoring (mean across the gene-set vs label on held-out) is a
    different number than the GP search used, so the #1 module can
    differ from — and even beat — the winning program.
  - **Coordinated-modules table** now tags the module whose gene-set
    is exactly equal to the winner's `gene_ids` (unordered set
    equality) with a `winner` badge in the existing accent-teal
    palette. That row sits at its scored rank, not at the top, so the
    user can see what rank the winner occupies under the group metric
    (e.g. seed 11 / HPV: winner program holdout 0.9411 vs the same
    gene-set as a module → 0.9153 → rank 41; #1 module 0.9778 — three
    different numbers, three honest meanings).
  - **Paginated auto-reveal** replaces the "Reveal genes" per-row
    click. `MODULE_PAGE_SIZE = 25`; symbols cached in a
    `Record<string, string>` keyed by opaque ID and ONE batched
    `/reveal` call per page (bounded by ≤ 25 × `max_genes_per_set`
    IDs — never the whole map). Page-1 of the HNSC seed 11 run pulls
    99 unique IDs out of ~20k. New `<ModulePager>` (|‹ ‹ page n/N › ›|)
    + a row click (▶ / ▼) toggles the expanded per-gene-metrics view.
    Sort or fresh-run resets page + collapses any open row. Inline
    symbol list truncates at 8 with a `· +N` overflow chip; expanding
    shows them all. The expanded `<ModuleGeneTable>` reads from the
    shared `symbolByOpaque` cache (no second fetch).
  - **No engine/API/airgap changes.** `tsc --noEmit` clean;
    `pytest -q` 114 / 114; airgap suite untouched.
- **Chunk 7 follow-up (done — confound pass: configurable Effect
  confounders + site / purity survival flags on module ranking):**
  Lets us tell whether a gene/module separates HPV for a *mechanistic*
  reason or via a *confounder* (sex / race / anatomic site / immune
  composition). Airgap stays absolute — clinical confounders (sex /
  race / oropharynx flag) and the derived purity proxy are named
  non-gene variables in the data + validation layer; engine_v2 only
  ever sees opaque gene IDs.
  - **HNSC build.** `data_pipeline/build_hnsc.py` now writes `race`,
    `ethnicity`, `tissue_site` (raw TUMOR_TISSUE_SITE),
    `icd_o_3_site` (ICD-O-3 topography from the patient file), and a
    derived **`is_oropharynx`** boolean (SEER mapping: ICD-O-3 prefix
    C01/C09/C10 + exact C02.4/C05.1/C05.2). Prints non-null counts
    and the top ICD-O-3 distribution so the mapping is auditable. On
    the live cohort: 72 oropharynx / 415 not / 0 NaN among 487 called
    samples; HPV+ concentrates in the oropharynx (49/72 = 68%) vs
    HPV- (23/415 = 5.5%) — the confound is real and large.
  - **dsl.Load** widens its allowed clinical columns to include
    `race`, `ethnicity`, `tissue_site`, `icd_o_3_site`, `is_oropharynx`
    — filtered per cohort (coadread's parquet doesn't have these, so
    nothing changes there).
  - **API `_prepare_lab_data`** widens the HNSC clinical subset to
    `(stage, age, sex, race, tissue_site, icd_o_3_site, is_oropharynx)`
    when present (skips columns not in the frame). Coadread unchanged.
  - **`Effect` is now configurable.** `ExecContext` gained
    `confounders: tuple[str, ...] = ("stage", "age")` — default
    preserves byte-for-byte legacy behaviour (MSI / TMB / HPV runs
    unchanged). `Effect.execute` reads `ctx.confounders` and builds
    a one-hot-encoded design matrix from any columns present in
    `ctx.clinical` (age continuous, everything else categorical;
    missing values drop the row). Pipeline + worker plumb the
    confounder set in — HNSC v2 runs use
    `("stage","age","sex","race")` automatically when those columns
    are in clinical. Smoking deliberately NOT included (not in this
    download). New tests: `test_effect_default_confounders_are_stage_and_age`
    + `test_effect_extended_confounders_include_sex_and_race`
    (asserts the extended path runs and unknown column names like
    "smoking" are silently skipped).
  - **Module ranking: site-stratified survival flag.**
    `_compute_module_ranking` no longer discards `clinical`. For
    HNSC/HPV runs it slices `is_oropharynx` to the held-out test
    patients, recomputes each module's mean-aggregate AUROC within
    the oropharynx subgroup, and emits `combined_holdout_oropharynx`
    + `survives_site` (True iff stratified AUROC ≥ full − 0.05).
    Modules whose "HPV signal" was really an oropharynx-tissue
    marker visibly collapse here.
  - **Module ranking: purity-proxy survival flag.** Immune-
    infiltration proxy = per-patient mean expression of CD8A / GZMB /
    PRF1 / CD3D / CD2 (`HPV_IMMUNE_PROXY_GENES`, curated once in
    `api/app.py`), resolved to opaque IDs via the sealed map — same
    bounded-reveal pattern `/diagnostic/full-rank` uses for
    `reference_marks`; never leaves the API layer. Bottom tertile
    of the proxy on TEST = high-purity subset; each module gets
    `combined_holdout_highpurity` + `survives_purity`. On the live
    HNSC cohort the high-purity TEST subset is ~49 patients but with
    only ~2 HPV+ (class imbalance — HPV+ tumours have more immune
    infiltrate), below the n_pos ≥ 5 guard, so the flag honestly
    reports "n too small" rather than a misleading AUROC.
  - **Module endpoint payload** gains `subgroups: { site, purity }`
    metadata (kind / n / tolerance / n_proxy_genes) at the top level
    so the UI can render the column label + an explanatory caption.
    Opaque-only on the wire — no gene NAMES enter the module
    payload.
  - **Frontend.** New `<SurvivalChips>` renders compact
    `site ✓ 0.92` / `site ✗ 0.84` / `purity —` chips on each module
    row (gated to whichever subgroups the server emitted; non-HNSC /
    non-HPV runs don't show the column at all). Expanding a module
    appends `<StratifiedAUROCStrip>` — full vs oropharynx vs
    high-purity AUROCs with n + per-class counts. New `TIPS.module
    Survival` `?` explains the flags. `TIPS.moduleRanking` gained
    the verbatim winner's-curse caveat: *"these groups are scored by
    re-evaluating ~2,900 explored sets on the same small held-out
    set and showing the best — so the very top values are
    optimistically biased (the luckiest of thousands). The winning
    program was chosen by cross-validation, which guards against
    that, so trust it as the engine's pick even when a table row
    scores higher."*
  - **Verification.** `pytest -q` 116 / 116 (was 114; +2 Effect
    confounder tests). Airgap suite 25 / 25 green — module payload
    still opaque-only. `tsc --noEmit` clean. Live HNSC seed-11
    coherence-on: 303 modules, 234 survive the oropharynx subgroup
    ✓, 69 collapse ✗, 0 n/a; purity all n/a (class imbalance, as
    expected). The winner module (the 3-gene Combine — full AUROC
    0.915 in this module-metric view) survives the oropharynx test
    (subgroup AUROC 0.919) — it's genuine HPV signal that holds
    within oropharynx, not a tissue marker.
- **Chunk 7 follow-up (done — Result views consolidation: GP's
  group vs known markers vs explored leaderboard):** Presentation
  refactor so the three Lab Result panels map cleanly to three
  questions and no single table mixes a group entry with individual-
  gene rows. No engine / API / airgap change.
  - **Survival chips show full → subgroup.** `<SurvivalChips>` (was:
    `site ✓ 0.95`) now renders `site ✓ 0.97 → 0.95 (−0.02)` so the
    user sees both the pre and post AUROC plus the drop. Pre =
    `module.combined_holdout`; post = the stratified AUROC. The
    "n too small" path drops the arrow and renders `site —`
    (unchanged). Tooltip wording updated to spell out the full-cohort
    → subgroup pair.
  - **Known-marker recovery panel = reference markers only.**
    `<RankingResult>`'s `pinnedRows` no longer pushes the winner's
    `kind:"winner"` rows; only reference-set rows survive. SectionCard
    title becomes **"Known-marker recovery"**, the heading flips from
    *"Highlighted genes"* to *"Known markers"*, the lollipop chart
    auto-titles to *"Where the known markers rank"* (and drops the
    winner legend swatch) when `winnerOpaqueIds=[]`. The captions
    and `TIPS.rankingResult` / `TIPS.rankingHighlighted` rewritten to
    state this is a known-marker recovery diagnostic and point users
    to the Result panel for the winner's group view. The raw-anonymous
    list still tints rows that match the winner — that's per-row
    highlighting, not a pinned-source mix.
  - **Result panel = self-contained "GP's group" story.**
    `<ResultPanel>` now takes `dataset` / `target` / `runId` /
    `coherence` from the Lab parent and fetches
    `getFullRankDiagnostic` (per-gene single-gene ranks) +
    `getRunModules` (for the winner's survival chips) — same
    endpoints already used elsewhere; the server caches them.
    Reveals the winner's symbols via `postReveal(winner.gene_ids)`
    (bounded — only the winner). New `<WinnerGroupBlock>` rendered
    under the metric cards:
    - **Its genes, each on its own:** for each `winning.gene_ids`,
      looks up its rank in `diag.ranks` and shows
      `SYMBOL #rank / N` (e.g. `RNF32 #123 / 20,218 · PCBD2 #79 /
      20,218 · RPS10P7 #2344 / 20,218`).
    - **Confound survival:** finds the winner's own module in
      `moduleData` via unordered gene-set equality (same notion
      `<ModuleRankingPanel>::isWinnerSet` uses), then renders
      `<SurvivalChips>` — automatically inheriting the Part-A
      pre→post pairing. Gated to runs that emit survival metadata
      (HNSC/HPV coherence-on); omitted gracefully otherwise.
    - Block InfoTip frames the three-panel layout: *"This is the
      group the engine actually chose. Below: how each of its genes
      ranks on its own, and whether the group survives the confound
      checks. The known-marker recovery diagnostic and the explored-
      group leaderboard live in the next two panels — different
      questions."*
  - **Verification.** `tsc --noEmit` clean; `pytest -q` 116/116
    (no Python changes). Live HNSC seed-11 coherence-on: winner
    RNF32+PCBD2+RPS10P7 AUROC 0.941, the three per-gene ranks resolve
    (#123 / #79 / #2344), winner module survives the oropharynx
    subgroup (site ✓ 0.915 → 0.919) and reports purity n/a (class
    imbalance). The known-marker panel shows only MCM5 #22 / MCM2 #28
    / MCM6 #38 / … with no winner dots/rows. The Coordinated-modules
    table still badges the winner's own set at its scored rank.
- **Chunk 7 follow-up (done — Result panel: GP's top-10 programs +
  plain-language modules copy):** Expanded the single-row "GP's
  group" block into the engine's top-10 programs ranked by its OWN
  fitness, and rewrote the Coordinated-modules copy in plain
  English. Presentation + frontend data-flow only.
  - **TopProgramsBlock** replaces `<WinnerGroupBlock>` in `<ResultPanel>`.
    `<ResultPanel>` now fetches `getRunStatus(runId)` → last
    `generations_persisted - 1` → `getRunPopulation(runId, lastGen)`,
    dedupes the `candidates` by unordered UNIQUE-gene-set (engine_v2
    programs can carry repeated genes across multiple Selects), keeps
    the highest-fitness representative per set, sorts by fitness
    desc, and takes the top-10. Each row reuses the existing
    treatment: per-gene `SYMBOL #rank / N` (looked up in `diag.ranks`)
    and `<SurvivalChips>` for the program's matching module (gene-set
    equality the same notion `ModuleRankingPanel::isWinnerSet` uses).
    #1 is the winner and is highlighted with the accent-teal palette +
    a `winner` chip.
  - **Bounded reveal**. The reveal cache (`symbolsByOpaque`) is
    populated by a single batched `postReveal` over the UNION of all
    top-10 programs' gene_ids — never the whole map. Live HNSC seed-
    11 coherence-on: 15 unique opaque IDs across the top-10.
  - **Modules subtitle + InfoTip rewritten in plain English.** The
    Coordinated-modules `<SectionCard>` subtitle is now: *"Not the
    engine's picks. After the run, this re-scores every gene group
    the engine tried — using one simple number (the group's average
    expression, measured on held-out patients) instead of how the
    engine judged groups during the search. So this list can rank
    groups differently from 'The GP's top programs' above, and its
    #1 can even beat the engine's winner."* `TIPS.moduleRanking`
    rewritten verbatim to the three-paragraph "What this is / How
    it differs / Caveat" structure from the prompt; the winner's-
    curse paragraph now reads as plain advice ("trust the engine's
    own picks above as the reliable choice") rather than statistical
    jargon. `TIPS.rankingResult` reworded to point at the GP's top
    programs and the after-the-fact re-scoring panel by name.
  - **Verification.** `tsc --noEmit` clean; `pytest -q` 116/116.
    Live HNSC seed-11 coherence-on: GP top-10 by fitness has the
    winner (RNF32+PCBD2+RPS10P7) at #1 with GP fitness 0.924; runner-
    ups are small 2-3-gene programs around the same core. Module
    ranking by combined held-out AUROC has an 11-gene cell-cycle-
    flavoured set at #1 (combined 0.978) — a different list, exactly
    the divergence the new copy makes explicit.
- **Chunk 7 follow-up (done — cheap diversity knobs):** Premature
  convergence was visible in the fitness curve (best = median by
  ~gen 10, then flat). Added three cheap diversity levers behind one
  "Maintain diversity" toggle. Defaults preserve behaviour byte-for-
  byte; the airgap is untouched (search-internal).
  - **`run_gp_v2` gains `immigrant_fraction: float = 0.0`.** When > 0,
    `round(immigrant_fraction * population_size)` slots in the new
    generation are filled with fresh programs drawn from
    `ramped_population` (same rng, same grammar/objective/depth
    constraints as init), inserted AFTER the elites and BEFORE the
    crossover/mutation offspring — never displacing elites. Default
    0.0 ⇒ no immigrants ⇒ behaviour unchanged. `tournament_k` and
    `p_mutate` were already parameters; raising mutation and
    lowering selection pressure are paired with immigrants via the
    same toggle.
  - **Pipeline + worker plumbing.** `run_v2_pipeline` and
    `run_v2_pipeline_streaming` accept the new
    `immigrant_fraction: float = 0.0`. `RunRequest` gains
    `diversity: bool = False`. `Run` dataclass gains a `diversity`
    field; surfaced in `/runs/{id}`. `_worker` maps `diversity=True`
    → `tournament_k=2`, `p_mutate=0.85`, `immigrant_fraction=0.10`;
    `False` → current `(3, 0.7, 0.0)`. Two new tests in
    `tests/test_engine_v2.py`:
    `test_default_run_unchanged_by_diversity_param` (passing
    `immigrant_fraction=0.0` explicitly produces an identical best-
    fitness trajectory to omitting the arg) and
    `test_immigrant_fraction_injects_fresh_programs` (with `frac>0`
    the last generation contains ≥1 immigrant — empty-parents and
    non-survived — and more distinct `program_repr` strings than the
    baseline).
  - **Frontend.** New `<TIPS.diversity>` copy. ParamsControls gets
    a "Maintain diversity" checkbox under the coherence toggle;
    plain-English caption: *"Default off. Lowers selection pressure
    and injects fresh random programs each generation so the
    population keeps exploring — watch the best-vs-median gap in
    the fitness curve stay open longer."* Lab state +
    `postRun(body.diversity)` plumbed end-to-end.
  - **Verification.** `pytest -q` 118/118 (was 116; +2 diversity
    tests). `tsc --noEmit` clean. Airgap suite untouched. Live
    HNSC HPV seed-11 coherence-on, 20 generations, A/B:
    - **OFF**: gap collapses 0.242 → 0.002 by gen 19 (full
      convergence). Held-out 0.977, p 0.048.
    - **ON**:  gap stays 0.13-0.18 the entire run (mean 0.149 vs
      OFF 0.041; last-gen 0.139 vs OFF 0.002). Held-out 0.948,
      p 0.048 — detection preserved while the population stays
      exploratory.
- **Chunk 7 follow-up (done — merge "GP's top programs" +
  "Coordinated modules" into one sortable "Groups the engine
  explored" table):** Replaced the two overlapping panels with a
  single table over the SAME groups (harvested from the persisted
  population), selectable by four sort lenses. One additive API
  field, presentation refactor; airgap unchanged.
  - **API.** `_compute_module_ranking` now records the max GP fitness
    per gene-set while harvesting candidates and emits
    `gp_fitness: float | None` on each module dict. Fitness is a
    number — no gene names cross the wire. `RankedModule` in
    `web/lib/api.ts` gains `gp_fitness?: number | null`.
  - **Frontend — merged table.** `<ModuleRankingPanel>` retitled
    **"Groups the engine explored"**. `SortKey` extends to
    `"gp_fitness" | "combined" | "coherence" | "synergy"`. Default
    sort = **gp_fitness** so the default view IS the engine's own
    preference order (replacing the deleted top-programs panel).
    Two new columns: **GP fitness** (per-row) and **Synergy** =
    `combined_holdout − max(single_gene_metric over per_gene)` — high
    Synergy = real additive lift; low/negative = "best gene +
    passengers" pattern. Sort buttons relabelled accordingly.
    Subtitle rewritten in plain English (verbatim per prompt);
    `TIPS.moduleRanking` rewritten to explain all four lenses + keep
    the winner's-curse caveat for the re-score lenses.
  - **Frontend — drop TopProgramsBlock.** `<ResultPanel>` no longer
    fetches `getRunStatus` / `getRunPopulation` / `postReveal` /
    `getFullRankDiagnostic` / `getRunModules`; the entire
    `TopProgramsBlock` function definition and its supporting state
    are gone. Result-panel signature simplifies to
    `{ result, dataset, target, runId, coherence }` with `dataset` /
    `runId` only used to gate the new pointer copy under the metric
    cards: *"The winning program is badged in **Groups the engine
    explored** below — sort by GP fitness to see the engine's full
    preference order, or by Combined AUROC / Coherence / Synergy to
    re-score the same groups by a different lens."* Pointer is gated
    to `target !== "none" && coherence` (the same gate the merged
    table uses). Unused imports (`Candidate`, `getRunStatus`,
    `getRunPopulation`) removed.
  - **Verification.** `pytest -q` 118/118 (no Python tests changed —
    the new field is additive, the old payload tests still pass).
    `tsc --noEmit` clean. Airgap suite untouched. Live HNSC seed-11
    coherence-on confirms the four lenses give visibly distinct
    orderings on the same module set: GP fitness #1 = winner
    (RNF32+PCBD2+RPS10P7, gp 0.924); Combined-AUROC #1 = 11-gene
    cell-cycle module (combined 0.978); Coherence #1 = a tightly
    co-expressed 2-gene set; Synergy #1 = a 3-gene group with
    +0.154 lift above its best single gene.
- **Chunk 7 follow-up (done — "GP fitness × Synergy" landscape
  scatter):** Added a canvas scatter above the merged Groups table
  so the user can SEE which quadrant each explored group falls into
  at a glance. Frontend-only; no engine / API / airgap change.
  - **`<FitnessSynergyScatter>`** (~340 LOC in `web/app/Lab.tsx`).
    x = `m.gp_fitness`, y = `synergyOf(m)` (lifted to a module-scope
    helper so the table + scatter share one definition).
    Categorisation via `categoriseModule(m, winnerSet)` →
    `winner | p16 | immune | cell_cycle | MMR | other` (single-
    marker sets win over broader sets; winner takes priority over
    membership so the ring is always on top).
  - **Canvas for the dots, SVG overlay for the chrome.** ~6k modules
    in larger runs would choke as SVG nodes; canvas paints faint
    grey background dots first, then amber/gold highlighted dots,
    then the winner as a teal-ringed filled centre on top so it
    never gets occluded. HiDPI: `canvas.width = clientWidth × dpr`
    + `ctx.setTransform(dpr, 0, 0, dpr, 0, 0)`. Plot rect responsive
    via a `ResizeObserver` on the wrapper div.
  - **Quadrant story.** Vertical guide at the median GP fitness,
    horizontal guide at synergy = 0 (orange dashed). Four captions
    in the corners: "real teamwork (unexplored by the engine)" /
    "ideal — usually empty (engine + teamwork)" / "weak / junk" /
    "engine's lone-gene detectors". Axis titles: *"GP fitness — what
    the engine preferred →"* and *"Synergy — teamwork beyond best
    gene ↑"*. Dataset-aware legend (HNSC shows p16 + cell_cycle;
    CRC shows immune + MMR).
  - **Gated** to `target !== "none"` + coherence-on (same gate as
    the survival flags / synergy column). Mounted between the
    survival-flag caption and the Groups table inside
    `<ModuleRankingPanel>` so it shares one SectionCard frame with
    the table.
  - **Verification.** `tsc --noEmit` clean; `pytest -q` 118/118 (no
    Python changes). Live HNSC seed-11 coherence-on: 303 plottable
    modules; winner @ `gp=0.924 / syn=0.032`; 21 dots in the
    bottom-right "engine's lone-gene detectors" quadrant; 20 in the
    top-left "real teamwork" quadrant the engine didn't prefer;
    3 cell_cycle-flagged dots cluster at low GP (~0.58, the engine
    barely scored them) with slightly-negative synergy — exactly
    the "best gene + passengers" pattern the scatter is meant to
    expose. Airgap untouched (synergy + category derived from
    fields already on the wire; no extra reveal).
- **Chunk 7 follow-up (done — hover tooltips on the scatter,
  highlighted dots only):** Added hit-tested tooltips to the
  fitness × synergy scatter so the user can read the gene-set and
  scores of any highlighted dot without scrolling the table.
  Frontend-only; airgap intact.
  - **Hit-test scope.** When painting, `<FitnessSynergyScatter>`
    now records the screen position of every HIGHLIGHTED dot
    (`winner` + `cell_cycle` + `MMR` + `p16` + `immune`) into a
    `hitsRef.current: Hit[]` array. The faint grey background dots
    are deliberately NOT in this array — they are not revealed and
    are not hit-tested. Selection: `mousemove` walks the array and
    picks the nearest hit within ~8px (radius² = 64); `mouseleave`
    clears.
  - **Bounded reveal.** On mount / when `modules` changes, the
    scatter takes the UNION of all highlighted modules' `gene_ids`
    and issues ONE batched `postReveal` call — never the whole map.
    On live HNSC seed-11 coherence-on: **31 unique opaque IDs**
    revealed out of ~20k (winner 3 + 21 cell_cycle-tagged modules,
    plus a handful of overlapping genes). Cached in
    `highlightSymbols: Record<string, string>` so re-renders are
    free.
  - **`<ScatterTooltip>`** absolutely-positioned over the chart
    container. Renders: source chip (winner / p16 / cell_cycle /
    immune / MMR — coloured to match the dot), `n genes`, the
    revealed gene symbols (falls back to `(revealing…)` while the
    bounded call is in flight), and a 2×4 grid of GP fitness,
    Combined AUROC, Coherence, Synergy via `fmtFit` (non-finite →
    em-dash). Synergy uses the shared `synergyOf` helper so the
    tooltip, the table column, and the y-axis stay consistent.
    Position clamps to the chart rect so dots near the edge don't
    push the box off-screen.
  - **Verification.** `tsc --noEmit` clean; `pytest -q` 118/118 (no
    Python changes). Live HNSC seed-11 coherence-on: hovering the
    winner ring shows `RNF32, PCBD2, RPS10P7 · GP 0.924 · Combined
    0.915 · Coherence 0.474 · Synergy 0.032`; hovering a
    cell_cycle dot shows e.g. `CDC6, …` with its scores. Background
    dots are not hover-targets, no full-map reveal.
- **Chunk 7 follow-up (done — encode site survival on the scatter +
  plain-language Survives explainer):** Two presentation fixes
  around the confound flags. Frontend-only; the data
  (`survives_site` / `survives_purity` / subgroup AUROCs) is already
  on the wire.
  - **Scatter site-survival encoding.** Highlighted dots in
    `<FitnessSynergyScatter>` are now drawn:
    - **solid** (fill + stroke) when `survives_site === true` (or
      `null` / "n too small" — keeps the simple default visible),
    - **hollow** (white fill + coloured stroke) when
      `survives_site === false` — so failed-site groups read as
      "less trustworthy" at a glance.
    Only **site** is encoded; purity is mostly `null` on this
    cohort and carries no signal. The category colour (winner teal
    / cell_cycle amber / p16 gold) is preserved. The winner ring
    uses the same encoding when it fails the check.
  - **Hide-failed toggle.** Added a small "Hide groups that fail
    the site check" checkbox in the scatter header (default OFF).
    On: `survives_site === false` highlighted dots are dropped
    from both the canvas paint AND the hit-test array (no
    accidental hover tooltips on hidden dots). Background grey
    dots are unaffected (not site-tested).
  - **Caption + legend updated.** The italic caption gains
    "Solid dots survive the site check; hollow dots fail it (their
    signal is partly location)." The legend gains a "hollow = fails
    site check" entry.
  - **Plain-language "Survives" column ?.** `TIPS.moduleSurvival`
    rewritten verbatim per the prompt:
    > What "Survives" checks: whether a group still separates HPV
    > when you take away a possible confounder — something that
    > travels with HPV but isn't HPV biology.
    >
    > Site / Purity each get a one-paragraph plain-English
    > explanation; ✓ / ✗ / — get a one-line legend each.
    The tip is now anchored on the **Survives column header**
    (the `<th>`) where it's most discoverable. Per-chip pre→post
    tooltips on the rows are unchanged.
  - **Verification.** `tsc --noEmit` clean; `pytest -q` 118/118
    (no Python changes). Live HNSC seed-11 coherence-on shows
    234 of 303 modules survive site / 69 fail; the 4 highlighted
    dots (winner + 3 cell_cycle) all survive on this seed, so the
    encoding renders solid — the failed-dot path is exercised in
    the general case where a tagged module collapses.
- **Chunk 7 follow-up (done — five small UI fixes: default cohort,
  remove peel-off, fix two clipped widgets, swap biology figure):**
  Presentation-only sweep over Lab; frontend-only. No engine / API /
  airgap change.
  1. **Default to Head & Neck + HPV detection.** `dataset` state in
     Lab now defaults to `"hnsc"`, and `target` to
     `DATASET_REGISTRY.hnsc.objectives[0].key` so first load lands
     on the HPV demo of the workflow. Colorectal stays selectable;
     only the default changed.
  2. **Hide the Discovered axes panel.** `<DiscoveredAxes>` is now
     gated to `target === "none"` (never true in the UI — the
     unsupervised objective isn't a UI objective anymore). On
     supervised runs the panel was re-finding the same signal and
     showing nonsensical "Aligns with MSI / TMB" text. The component
     definition + backend chain are untouched in case a developer
     flag wants to surface it again.
  3. **Fix the "(TCGA COADREAD" subtitle clipping.** Not a CSS
     issue — the old regex `/^[(]|[)]$/g.replace` was stripping the
     trailing `)`. Replaced with
     `entry.longLabel.replace(new RegExp('^${entry.label}\\s*'), '').trim()`
     so the parenthesised tail renders intact, e.g. `(TCGA HNSC)`.
  4. **"Survives" `?` is a click-to-open modal now.** Multi-paragraph
     copy didn't fit a hover tooltip — clipped on the right.
     `paramHelpContent.tsx::ParamKey` extended with `"module_survival"`
     and a new `PARAM_HELP.module_survival` entry carrying the
     prompt's verbatim site / purity / ✓ / ✗ / — explainer. The
     `<th>Survives</th>` (and the table-header caption above the
     table) now uses `<ParamHelp paramKey="module_survival">` instead
     of `<InfoTip>`. Per-chip `full → subgroup` row tooltips are
     unchanged.
  5. **HPVBiologyPanel — new card-wall figures.** `HPVBiologyPanel.tsx`
     now renders TWO inlined SVGs: `<NormalCellDiagram>` (the new
     repo-root `normal_cell_reference.svg`, viewBox 920×900: RB1 /
     p16 / RB-E2F cycle / E2F / genes E2F turns on / protein
     production / p53) followed by `<HPVDiagram>` (the rewritten
     repo-root `hpv_reference.svg`, viewBox 920×660: HOW HPV TAKES
     OVER → HOW E7 ATTACKS RB · HOW E6 ATTACKS p53 → p16 BECOMES
     THE HPV+ MARKER · WHY HPV DOESN'T MUTATE THE GENES). The
     normal-cell baseline renders first so the reader sees the brake
     before they see it dismantled. SVG attributes hand-translated to
     JSX exactly as the existing component did (`font-family →
     fontFamily`, `font-weight → fontWeight={n}`, `text-anchor →
     textAnchor`, `marker-end → markerEnd`, etc.); each `<svg>` keeps
     `viewBox` + a width:100% / height:auto wrapper so it scales
     without overflow.
  - **Verification.** `tsc --noEmit` clean; `pytest -q` 118/118 (no
    Python changes). Lab first-load lands on HNSC + HPV; biology
    panel shows the new normal-cell + HPV figures (both fully
    visible, no overflow); the Discovered-axes panel is gone for
    HPV / MSI / TMB runs; cancer-card subtitles read "(TCGA HNSC)"
    with the closing paren; the Survives "?" now opens a click-
    modal with the four-paragraph site / purity explainer that
    never clips.
- **Chunk 7 follow-up (done — raised caps, "GP" → "genetic
  programming", DSL operator-usage):** Three changes across the
  Lab + API surface.
  - **Generations / Population caps raised.** Lab's NumField config
    bumps Generations max 100 → 1000 and Population max 500 → 3000;
    server-side `RunParamsModel.generations / population` Field
    bounds raised to match. Bounded — not removed — so an
    accidental huge value still can't hang the backend.
  - **"GP" spelled out everywhere user-facing.** Every visible
    occurrence in `Lab.tsx` replaced: "GP knobs" → "Genetic-
    programming knobs"; "Starts a GP run" → "Starts a genetic-
    programming run"; the compact label "GP fitness" → **"Genetic-
    programming fitness"** at the sort button, table header,
    scatter heading + axis title + median guide + tooltip dt
    + module-ranking subtitle, and TIPS prose. The internal data
    key `gp_fitness` is UNCHANGED (API contract). Code comments
    still say "GP" by choice — they're not user-facing.
  - **DSL operator-usage endpoint + tiles.**
    - New `GET /runs/{run_id}/operator-usage` walks every candidate
      in every persisted generation, counts each operator token
      (`Select(`, `Reduce(`, `Combine(`, `Split(`, `Associate(`,
      `Effect(`, `FitApply(`, `Search(`) in the `program_repr`
      strings, and emits per-operator
      `{name, total_uses, programs_using}` + run totals
      (`n_generations`, `n_candidates`). Cached per run. 425 while
      running, 404 unknown, 500 on error. Opaque-safe by
      construction — operator keywords + integer counts only, no
      gene IDs / symbols ever in the payload. Three new airgap
      tests (`test_operator_usage_endpoint_returns_opaque_counts`,
      `_425_while_running`, `_404_for_unknown_run`).
    - Frontend: `getRunOperatorUsage(runId)` fires from the SSE
      `done` handler; Lab caches the result in `operatorUsage`
      state. New `<DSLVocabularyTiles>` driver swaps each
      `<DSLVerbCard>` for an enhanced version that adds a
      `used N× — in P% of programs` line and a tile-relative bar
      (height 1, accent fill, white-grey track) under the
      description. Before / during a run, tiles render just the
      hints (no counts). A caption appears beneath the grid after
      a completed run.
  - **Verification.** `pytest -q` 121/121 (+3 new airgap tests).
    `tsc --noEmit` clean. Live HNSC seed-11 coherence-on, 10×80:
    Reduce in 100% of programs (grammar floor), Select 90.6%
    (1098 uses), Combine 21.2% (216), Split 4.1%, Associate 2.4%,
    Effect 1.0%, Fit/Apply 3.6%, **Search 0** (off by default).
    Airgap-clean payload — keywords + counts only.
- **Chunk 7 follow-up (done — turn Search on by default + add a
  per-run gate):** Search was the one DSL operator still gated off
  (`DEFAULT_RATES["search"] = 0.0`). Turn it on at a modest rate so
  the engine can use the full DSL, and add a UI toggle so it can be
  flipped off when speed matters (every Search node runs an inner
  gene ranking).
  - **Engine.** `engine_v2/synthesize.DEFAULT_RATES["search"]` flips
    0.0 → **0.05**. Caps stay (`SEARCH_MAX_K`, ≤ 200 candidate
    columns) so each Search's cost is bounded. The existing
    `is_unsup` gate at the injection site still forbids Search on
    unsupervised runs.
  - **`run_gp_v2` gains `rates_override: dict | None = None`.** Inside
    `run_gp_v2` the override is merged on top of the objective's
    synthesis-overrides (`UNSUP` etc.) — caller's keys win — and the
    merged dict is threaded into every `ramped_population` / `mutate`
    call. Both pipeline entry points (`run_v2_pipeline`,
    `run_v2_pipeline_streaming`) expose `rates_override` and forward
    it through.
  - **API.** `RunRequest` gains `enable_search: bool = True`. `Run`
    dataclass tracks it; `/runs/{id}` surfaces it. `_worker` builds
    `rates_override = None` when on, or `{**DEFAULT_RATES,
    "search": 0.0}` when off, and passes it into the pipeline. The
    override is a complete dict (overlaid on DEFAULT_RATES) so the
    inner `synthesize._grow_*` code can index by key without a
    fallback.
  - **Frontend.** New `<TIPS.enableSearch>` copy. ParamsControls
    gains an **"Enable Search operator"** checkbox under the
    diversity toggle (default ON). Lab state + `postRun.body
    .enable_search` plumbed end-to-end. DSL-vocabulary tile for
    Search now reads *"Nested feature search — runs a small
    gene-ranking inside the program (bounded: ≤ 4 genes from
    ≤ 200 columns)."* — the "off by default" wording is gone.
  - **Tests.** `tests/test_api_airgap.py::test_enable_search_false_pins_search_to_zero`
    verifies the toggle: with `enable_search=False` the operator-
    usage payload reports Search at 0 total uses / 0 programs, and
    `/runs/{id}` surfaces the flag. The existing
    `test_operator_usage_endpoint_returns_opaque_counts` relaxed
    its old "Search must be 0" assertion (it's now `>= 0` — Search
    may or may not fire on a tiny synthetic 3×12 run).
  - **Verification.** `pytest -q` 122/122 (+1). `tsc --noEmit`
    clean. Live HNSC seed-11 coherence-on, 10 × 80:
    - **Search ON** (default): Search used **20×** across 20 / 800
      programs (~2.5%); winner AUROC 0.909, p 0.048.
    - **Search OFF**: Search used **0×** / 0 programs; winner
      AUROC 0.941, p 0.048.
- **Chunk 7 follow-up (done — editable DSL rates in the vocabulary
  panel + drop the Search checkbox + reorder DSL before Run):**
  Make every optional DSL operator's injection rate editable in the
  DSL vocabulary section (pre-filled with engine defaults), fold the
  Search checkbox into a Search-rate input (0 = off), and move the
  panel above the Run section. Airgap untouched.
  - **API.** `RunRequest.enable_search` is gone; replaced by a flat
    `RunRequest.rates_override: dict[str, float] | None`. Keys we
    accept: `split`, `effect`, `fitapply`, `search`, `scalar_share`.
    Missing keys keep their engine defaults — an empty / missing
    override reproduces current behaviour byte-for-byte. `Run`
    dataclass stores the dict; `/runs/{id}` surfaces it.
    `_worker` splits the flat dict back into `rates_override`
    (overlay on `engine_v2.synthesize.DEFAULT_RATES`) and
    `scalar_share_override`, then forwards both to the pipeline.
  - **Engine.** `run_gp_v2` gains
    `scalar_share_override: float | None = None`; merged with the
    objective's overrides via an `effective_scalar_share` that wins
    over `overrides.get("scalar_share")`. Both pipeline entry
    points expose the new param and forward it.
  - **Frontend.** New `DSL_DEFAULT_RATES` constant mirrors the
    engine defaults (Split 0.10 / Effect 0.40 / Fit/Apply 0.10 /
    Search 0.05 / scalar_share 0.20). `<DSLVocabularyTiles>`
    rewritten: structural operators (Select / Reduce / Combine /
    Associate) render with an "always available" tag; optional
    operators (Split / Effect / Fit/Apply / Search) render with a
    small `<input type=number min=0 max=1 step=0.05>` pre-filled
    with their default. A separate `scalar_share` field sits in a
    footer row alongside two buttons: **Reset to defaults** (off
    when already at defaults) and **Set all equal** (one shared
    value for the four optional operators — the neutral-prior
    experiment). The `<TIPS.dslRates>` `?` explains the inputs.
  - **Wire shape.** The Lab only sends keys the user actually
    changed (`ratesDiff`), so the default workflow keeps producing
    an empty / absent `rates_override` field and behaviour is
    byte-for-byte unchanged.
  - **Layout.** The DSL vocabulary `<SectionCard>` moved ABOVE
    `<RunBar>` / `<LiveView>` so the (now interactive) config sits
    before the Run button. Order is now Cancer → Objective →
    Biology → Parameters → DSL vocabulary → Run → Live view.
  - **ParamsControls.** The "Enable Search operator" checkbox and
    `enableSearch` state/prop are gone; that toggle lives as a
    rate input on the Search tile (rate 0 = off).
  - **Tests.** `tests/test_api_airgap.py::test_enable_search_false…`
    renamed to `…rates_override_search_zero…` and rewritten to
    POST `{"rates_override":{"search":0.0}}`; asserts the
    operator-usage payload reports Search at 0 total / 0 programs
    AND `/runs/{id}.rates_override == {"search": 0.0}`. The
    `OperatorUsage` opaque-counts test stays the same. 122 / 122
    pytest pass; airgap suite green.
  - **Verification.** `tsc --noEmit` clean. Live HNSC seed-11
    coherence-on, 10 × 80:
    - Default (no override): Search 20×, Split 25, Effect 6,
      Fit/Apply 27 (the current DEFAULT_RATES mix).
    - `rates_override={"search":0.2}`: Search **40×** (double the
      default rate ⇒ roughly double the uses); Split, Combine etc.
      shift because raising Search's slot draws from the same
      Matrix-slot RNG mix.
    - `rates_override={"search":0.0}`: Search **0×** exactly,
      matches the old `enable_search: false` behaviour.
    - `/runs/{id}` surfaces `rates_override: {"search": 0.2}` for
      the UI.
- **Chunk 7 follow-up (done — DSL panel polish: scalar-share copy,
  rate explainer, collision-free lollipop labels):** Three small
  presentation fixes in `web/app/Lab.tsx`. No engine / API / airgap
  change; `tsc` clean.
  - **Scalar share copy rewritten in plain English.** The native
    `title=` jargon on the scalar-share row is gone. Inline
    description: *"how many starting programs are built around
    Associate / Effect (which score a gene set by how well it
    correlates with the target) instead of giving each patient a
    score. 0.20 ≈ 1 in 5; 0 = never."* New per-field `<InfoTip>`:
    *"Every program ends in one of two outputs: a score for each
    patient (the usual case), or a single correlation number
    produced only by the Associate or Effect operators. Scalar
    share is the fraction of starting programs built around that
    second kind…"* (`"(default 0.20)"` hint, the input, and the
    Reset / Set-all-equal buttons unchanged.)
  - **Rate explainer added to the DSL panel.** `TIPS.dslVocabulary`
    (the panel `?` next to the section title) replaced with the
    verbatim "Rate = how often the engine reaches for an operator
    …" copy — covers the Effect/Associate carve-out and the "not a
    measure of quality" caveat. Each tile's "rate" label gains an
    `<InfoTip>` with the short version (per-operator default
    interpolated), so the meaning of the input is discoverable
    without leaving the tile.
  - **Lollipop chart: collision-free label placement.** Previous
    fixed 4-row stagger garbled tight clusters (MCM2..7 / AURKB /
    CCNB1 rendered as overlapping strings). Replaced with a greedy
    placement: sort by x, estimate each label's pixel width
    (≈ `text.length × fontSize × 0.6`), drop into the lowest row
    whose previous label's right edge + 6px ≤ this label's left
    edge; open a new row if none fits. **No cap on rows.** The
    chart `H` is now derived from `numRows` so the SVG grows to
    fit and nothing clips. Per-label leader lines reach down to
    the dot from however many rows above sit the label. The MCM
    cluster now reads cleanly.
- **Chunk 7 follow-up (done — show each group's best program tree
  on row expand):** Each row's Genetic-programming fitness is the
  max fitness among evolved programs with that gene-set; now the
  argmax program's tree is carried through and rendered on expand.
  Airgap untouched.
  - **API.** `api/app.py::_compute_module_ranking` walks the
    population once and now tracks BOTH the per-set max
    `gp_fitness` AND the corresponding `program_repr` in a sibling
    dict (`best_program_repr_by_key`). Each module payload emits
    `best_program_repr: str | None`. Opaque-safe — `program_repr`
    is built from opaque IDs only.
  - **Types.** `RankedModule.best_program_repr?: string | null` in
    `web/lib/api.ts`.
  - **Frontend.** New `<ModuleBestProgram>` rendered inside the
    Groups-table expanded row, just under `<ModuleGeneTable>`.
    Caption: *"This group's best program — The actual tree of the
    candidate that earned the Genetic-programming fitness above
    (argmax over the persisted population for this gene-set)."*
    Uses the SHARED `<ProgramGraph>` (the same component the
    Result panel uses for the winner), with the row's already-
    revealed `symbolByOpaque` adapted to ProgramGraph's
    `Record<id, {symbol, matched}>` shape (matched=false — the
    known-marker highlight is a separate concern from per-group
    trees). The raw `program_repr` + a `<CopyButton>` sit
    underneath, so the string can be read or copied. Output label
    switches by target (HPV+ / MSI-H probability, TMB association,
    or score).
  - **Verification.** `tsc --noEmit` clean; `pytest -q` 122/122;
    airgap suite green. Live HNSC seed-11 coherence-on: 297/297
    modules carry `best_program_repr`. The winner's row's
    `best_program_repr` matches the Result-panel `winning
    .program_repr` exactly (`Reduce(Select(M,[g12850,g18272]),
    mean)` on this seed). Airgap scan of the modules payload
    finds no gene-symbol leak.
- **Chunk 7 follow-up (done — HNSC copy rewrite for a non-specialist
  reader):** Copy-only pass across the Head & Neck / HPV Lab so a
  co-founder can scroll top-to-bottom and understand each section
  without outside help. No engine / API / airgap / logic change.
  - **Voice spec:** plain, gloss jargon on first use (AUROC = 0.5
    coin-flip / 1.0 perfect; held-out = patients the engine never
    trained on; permutation p = how often random noise matches this);
    cut implementation plumbing ("Server-Sent Events", "on the
    backend", "opaque IDs", "airgapped", "residualise") while KEEPING
    the credibility ideas (blind discovery, held-out honesty,
    beats-chance, reveal-only-at-end) in plain words.
  - **Section-by-section rewrites** (in `web/app/Lab.tsx` + the
    biology panel + `paramHelpContent.tsx`): header subtitle (engine
    hunts blind, gene names hidden as codes); Cancer/problem
    subtitle; Objective subtitle; `obj_hpv` modal rewritten to the
    verbatim HPV-detection intro + AUROC gloss + "detecting a known
    viral fingerprint" honest note; HPV-biology caption; Parameters
    subtitle + every `PARAM_TIPS[*]`; DSL vocabulary subtitle +
    every tile hint (Select = "Pick specific genes.", Reduce =
    "Combine those genes into one score per patient (average, max,
    etc.).", Combine = "Merge two scores into one.", …); Run
    subtitle; Live view subtitle + `TIPS.bestVsMedian` verbatim
    per prompt; y-axis label "separation (AUROC — 0.5 coin-flip,
    1.0 perfect)"; Population subtitle; Program graph subtitle +
    reveal-state phrasing; Result subtitle + HPV-specific held-out
    / permutation / nodes / genes tooltips; Known-marker recovery
    subtitle + chart title; Groups the engine explored subtitle;
    scatter heading/caption; operator-usage caption; Reveal &
    evaluate subtitle. `<th>Opaque ID</th>` → `<th>Gene code</th>`
    across the three reveal tables; raw-ranking disclosure copy;
    live-view chip label; `TIPS.opaqueIds` polished.
  - **Jargon sweep clean.** No `Server-Sent Events`, `SSE`, `on the
    backend`, `worker thread`, or `airgapped` in visible copy.
  - `tsc --noEmit` clean; no behaviour change.
- **Chunk 7 follow-up (done — external-cohort transfer test:
  GSE65858 end-to-end):** Add a second, INDEPENDENT HPV validation
  cohort as a reveal-side transfer test — score the HNSC/HPV winner,
  discovered blind on TCGA, on ~270 GEO head & neck tumours from a
  different country, hospital, and measuring machine. Purely additive
  — the engine still discovers blind on TCGA; GSE65858 lives on the
  NAMED (reveal) side of the airgap and only ever sees the winner's
  already-revealed symbols.
  - **Data pipeline.** `data_pipeline/schema.py` gained a GSE65858
    block (series-matrix URL, GPL10558 platform URL, raw/processed
    dirs, HPV label constants). `data_pipeline/download_gse65858.py`
    mirrors `download_hnsc.py` (fetch series matrix + platform
    annot from NCBI FTP with loud manual-fallback instructions;
    live-download works — 21 MB series matrix + 7 MB platform).
    `data_pipeline/build_gse65858.py` parses the GEO series-matrix
    format, derives the STRICT virus-active `HPV+` (DNA+RNA+) label
    from the joint `hpv16_dna_rna` characteristic with a fallback
    ladder (joint → separate DNA/RNA → single status field → single-
    side calls), maps probes → HUGO symbols via GPL10558, collapses
    to a symbol × sample matrix (mean over probes), writes
    `data/processed_gse65858/{clinical,expression}.parquet`, and
    prints a full provenance report. **Live build: 250 called (35
    HPV+ / 215 HPV−) across 16,951 symbols** — matches the paper's
    strict virus-active rate.
  - **Named-side transfer function** (`validate/transfer_gse65858.py`)
    — a pure, testable `transfer_score(symbols, *, n_permutations,
    seed, processed_dir)` returning `{auroc, p, n, n_pos, n_neg,
    n_found, n_missing, found_symbols, missing_symbols}`. Cross-
    platform fix: **z-score each found gene within GSE65858** before
    the per-patient mean (TCGA RNA-seq scale vs Illumina array
    intensity). Orientation-agnostic AUROC + permutation-null p (with
    `+1/+1` smoothing). Finite-guarded; graceful `null` payload when
    `n_found == 0`. 5 new tests in `tests/test_transfer_gse65858.py`
    (signal → high AUROC / small p; noise → chance; missing symbols
    reported; no-found graceful; payload carries only supplied
    symbols).
  - **API endpoint** — `GET /runs/{run_id}/transfer`. Gated to
    **HNSC + HPV** runs only (400 otherwise); 425 mid-run, 404
    unknown, 503 if the cohort parquets aren't built. Bounded reveal
    of the winner's opaque IDs via `airgap.reveal` (same discipline
    `/evaluate` uses). Lazy-imports `validate.transfer_gse65858`.
    Cached per-run. Payload: `{cohort, platform, source, n_cohort,
    auroc, p, n, n_pos, n_neg, n_found, n_missing, found_symbols,
    missing_symbols}`; the only gene NAMES are the winner's own
    revealed symbols. 3 new airgap tests (non-HPV reject; payload
    carries only winner symbols with whole-word regex — so
    `AARS` inside `AARSD1` doesn't false-positive; 425 mid-run).
  - **Frontend.** `TransferResult` type + `getRunTransfer(runId)`
    fetcher. New `<ExternalValidation>` component at the end of
    `<ResultPanel>`, gated to `dataset === "hnsc" && target === "hpv"
    && runId`. Fetches on mount. Renders: section header + `?` +
    subtitle; green/amber verdict callout ("It holds on strangers."
    vs "Not confirmed on strangers.") gated on `auroc ≥ 0.75 && p <
    0.05`; three metric cards (Independent AUROC · Permutation p ·
    Genes measurable `n_found / (n_found+n_missing)`) each with its
    own plain `?`; `<Metric>` gained a `sub` prop for the subtitle
    line under the value; amber cross-platform pill (RNA-seq →
    microarray); three-step "How this validation works" strip
    (reveal → match & level → score) in light teal; teal 🔒 airgap
    footnote; one-liner honest note. Graceful fallback when the
    cohort isn't built (503 → build-hint instead of crash).
  - **Verification.** `pytest -q` 130/130 (+8 new tests; +5 transfer
    + 3 airgap). Airgap suite green. `tsc --noEmit` clean. Live
    end-to-end: HNSC HPV seed-11 winner `[g12850, g18272]` →
    symbols `PCBD2, TMEM71` → transfer on GSE65858 **AUROC 0.799,
    p 0.001 across 250 patients (35 HPV+ / 215 HPV−)**. Payload
    contains ONLY the two winner symbols. Textbook markers on
    GSE65858: **CDKN2A alone → AUROC 0.896, p 0.002**; MCM cluster
    → 0.85; TP53 → 0.74.
- **Chunk 7 follow-up (done — `scripts/multiseed_hpv.py` multi-seed
  stability orchestrator):** Read-only biology-aware script that runs
  the HNSC/HPV `engine_v2` pipeline across a default seed grid
  `[1, 3, 7, 11, 13, 17, 23, 29]` and reports per-seed held-out
  AUROC + permutation p + winner gene_ids, transfer AUROC + p +
  n_found/n_missing on GSE65858, held-out and transfer AUROC ranges
  across seeds, and a gene-recurrence tally (each symbol tagged by
  its HNSC reference set — p16 / cell_cycle — or `-`). Reveal is
  bounded per seed to that seed's own winner's opaque IDs via
  `airgap.reveal` (`scripts/` is allowed to be biology-aware; same
  discipline `/evaluate` uses); the sealed map is never dumped;
  GSE65858's gene list never crosses back into the engine. CLI:
  `python -m scripts.multiseed_hpv --seeds 1 3 7 --generations 30
  --population 300`.
- **Chunk 7 follow-up (done — "Stability across seeds" panel in the
  Lab):** Frontend-only multi-seed stability panel, gated to HNSC/HPV,
  appended after `<ExternalValidation>` in `<ResultPanel>`. Reuses
  the existing run, transfer, reveal, and full-rank endpoints — no
  engine / API / airgap change.
  - **State.** Own `stabilityRows` / `stabilityStatus` / `stability
    Progress` / `stabilityCancel` — the main single-run Result / Live
    view stays untouched during a sweep. A dedicated
    `awaitRunDone(runId)` helper opens an `EventSource` and
    resolves on the `done` event without calling `setResult` /
    `setGenerations` / `setStatus`.
  - **Body composition.** Per-seed body mirrors `launchRun()`
    exactly: `postRun({ objective_spec: OBJECTIVE_PRESETS.hpv,
    params: { ...params, seed }, engine: "v2", dataset, coherence,
    diversity, rates_override: ratesDiff })`. Only DSL-rate keys
    that differ from defaults travel on the wire so a default sweep
    is byte-for-byte identical to the current run.
  - **Sequential** (backend runs one at a time), with a `cancelRef`
    between seeds so a "Stop after this seed" button halts cleanly.
    Progress readout "Running seed k of N…" during the sweep.
  - **Rendering — three sections + summary callout:**
    1. **`<StabilitySummaryCallout>`** — deterministic, LLM-free,
       composed from the aggregates in JS. Rendered above the dot-
       strips only when `stabilityStatus === "done"`. Green if
       `perfStable && geneStable` (held-out min ≥ 0.75, every GP p <
       0.05, and top-recurring gene in ≥ ⌈N/2⌉ seeds); teal /
       informative if `perfStable && !geneStable`; amber otherwise.
       Copy verbatim per prompt: Line 1 performance ("Across N
       independent searches, the engine detected HPV every time…"
       with an "every run beat chance" or "most runs beat chance on
       the independent cohort" tail); Line 2 gene story ("The same
       genes kept coming back — TOP in K/N runs…" vs "the specific
       genes differed almost every run — the most repeated was TOP
       (K/N), and M genes appeared only once…"); Line 3 honest close
       only when `!geneStable` ("So trust the detection, but don't
       read any single run's gene list as THE gene list…"). Extra
       guard: when no gene appears in more than one seed
       (`topGene == null`), the copy switches to "no single gene
       showed up in more than one seed's winner…" rather than saying
       "the most repeated was null".
    2. **"Does the result hold across seeds?"** — two horizontal
       dot-strips on a 0.5 → 1.0 axis (`0.5 coin-flip / 0.75 / 1.00`
       tick labels). Accent-teal dots for Held-out AUROC (from
       `winning.holdout_score` — same field the Result card shows,
       so the stability dots match the numbers on screen), amber
       dots for Independent AUROC. Light min-max band per strip;
       header carries `min–max` via `fmtFit`.
    3. **"Do the same genes keep coming back?"** — recurrence bars:
       symbol · width-proportional bar · `K / N` · reference tag.
       Recurring (≥ 2 seeds) sorted desc; genes in only 1 seed
       collapse into a single "N genes · 1 seed each" row with the
       first 40 passenger symbols in the title tooltip. Colour:
       cell_cycle amber, p16 gold, alternate teal, passengers grey.
    4. **Per-seed table**: `seed | held-out AUROC | p | indep. AUROC
       | indep. p | genes found / total`. Every number through
       `fmtFit`; error seeds show `error` with the message on hover.
  - **Reveal discipline** — per seed, `postReveal(winning.gene_ids)`
    (bounded) sources the winner's symbols; then one
    `getFullRankDiagnostic(dataset, target)` at the end for the
    reference-set tag map (`p16` wins over `cell_cycle` when a
    symbol is in both). No new reveal surface.
  - `tsc --noEmit` clean. `pytest -q` unchanged (frontend-only).
    Sanity-tested the data path via `TestClient`: 3 seeds × tiny
    budget produces 3 completed runs, each with `holdout_score`,
    `permutation_p`, revealed symbols, and a live `/transfer`
    payload; the recurrence tally rolls up correctly.
- **Chunk 7 follow-up (done — `scripts/capability_ratio_test.py`
  known-answer capability test):** Self-contained blind capability
  check. Plants a synthetic binary target defined by the balance
  between two REAL, positively-correlated genes (a log-ratio
  direction) — rigged so NEITHER gene helps on its own — and asks the
  blind engine to rediscover the interaction. Isolated: ONE new file
  under `scripts/` (allowed to be biology-aware); READS the existing
  processed matrix + REUSES the sealed map via `airgap.anonymise` /
  `airgap.reveal` for the final check; writes nothing to disk; adds
  no dataset, no API route, no UI, no engine change. Deleting the
  file leaves zero trace.
  - **Pair search.** Smart seed-and-rank strategy (random-pair
    sampling is too slow at r ≥ 0.85 on 20k genes): pick a random
    seed gene from a variance/expression-filtered pool, rank the
    rest by descending Pearson r against it, check the top
    candidates. Constraints: `r ∈ [0.85, 0.99]`, single-gene AUROC
    in `0.5 ± 0.07`, ratio AUROC ≥ 0.9. Configurable via CLI
    (`--pair`, `--min-r`, `--single-band`, `--ratio-min`,
    `--pair-seed`).
  - **Planted target.** `zA = zscore(A), zB = zscore(B); signal = zA
    − zB; y = (signal > median(signal))`.
  - **Setup + panel controls (auditable).** Print single-gene AUROCs
    (~0.5), ratio AUROC (~1.0), best single-gene AUROC over the panel
    (~0.6), and panel-mean AUROC (~0.57) BEFORE running — so any
    high held-out score MUST come from composition.
  - **Blind runs.** `anonymise(panel_expr)` reuses the existing
    sealed map (no re-seal). Per-seed `run_v2_pipeline` with
    `V2Objective(target="msi", binary=True)` as the binary-AUROC
    carrier, `prefilter_n=None`, `scalar_share_override=0.0` (force
    Vector programs), `coherence_weight=0.0`, diversity on
    (`tournament_k=2, p_mutate=0.85, immigrant_fraction=0.10`),
    `population=200, generations=40, permutations=100`.
  - **Grader.** Per seed: `reveal(winning.gene_ids)` (bounded).
    PASS if revealed symbols ⊇ `{A, B}` AND `holdout_score ≥ 0.80`
    AND the `program_repr` contains a `Combine(` with one of
    `protected_div` / `mul` / `sub`. Prints program_repr, revealed
    symbols, held-out AUROC, per-check verdicts, and the overall
    "recovered in k / n seeds" line. PASS interpretation copy fires
    at `k ≥ ⌈n/2⌉`.
  - **Live evidence** (seeds 1, 3, budget 200 × 40): auto-picked
    pair **FPR3 / C3AR1** (r 0.923). Seed 1 winner:
    `Combine(Reduce(Select(M,[…,g06688,…]),median),
    Reduce(Select(M,[g11569,…]),median), protected_div)` — the
    planted ratio, held-out AUROC **0.858** vs the 0.60 best-single
    control. **PASS.** Seed 3 finds FPR3 without C3AR1 (0.62,
    FAIL). Overall 1/2 → PASS: *"The engine can discover a genuine
    two-gene interaction blind — so when a real target (HPV) yields
    only averages, that's because the biology doesn't need a ratio,
    not because the engine can't build one."* `pytest -q` still
    130/130 (nothing outside `scripts/` was touched).
- **Chunk 7 follow-up (done — `validate/tmb_resid_rank.py` "leftover-
  TMB" diagnostic):** Cheap read-only diagnostic checking whether
  MSI-residualized TMB is a candidate COMBINATORIAL target on
  colorectal. Mirrors `validate/tmb_rank.py` exactly (same cohort
  loader shape, same NAMED matrix, reuses `_spearman_per_column`
  verbatim); lives in `validate/` (structural airgap test scans
  only `engine/`); sub-second on the live cohort. No engine / API /
  UI / data change.
  - **Pipeline.** `Load("processed")` filtered to `tmb.notna() &
    msi_status ∈ {MSI-H, MSS} & no-NaN expression rows`. `t =
    log1p(TMB)`, then WITHIN each MSI group separately z-score
    `(t − group_mean) / group_std` (guard `std > 0`) — pooled into
    one residual vector aligned to `X.index`. Two `_spearman_per
    _column` passes (raw baseline + residual headline), both ranked
    by `|corr|` descending. Reports: `best_raw` vs `best_resid`
    (drop tells you MSI's slice was removed), top-N by
    `|Spearman(gene, resid)|`, MMR + IMMUNE gene positions against
    the residual, per-group stats (`n`, TMB mean, log1p mean/std)
    so the residualization is auditable.
  - **Verdict.** Thresholds on `best_resid`: `< 0.30 → CANDIDATE
    COMBINATORIAL TARGET`, `≥ 0.40 → NOT COMBINATORIAL`, otherwise
    `BORDERLINE`.
  - **Live output** on the colorectal cohort: N = 355 samples,
    20,056 genes; MSI-H n=50 (TMB̄ 49.8) · MSS n=305 (TMB̄ 7.6).
    Best `|corr|` vs raw TMB = **0.447 (CXXC1)** — MSI-driven, as
    expected. Best `|corr|` vs residualized TMB = **0.346
    (FOXD4L1)** — drop **−0.10**. MMR genes now WEAK against the
    residual: MLH1 rank 2543/20056, MSH2 at 17313, PMS2 at 15473
    (sanity check that MSI's effect was removed). PRF1 the
    strongest immune gene at rank 161 (`|corr| = 0.19`). Verdict:
    **BORDERLINE** — *"A weak single-gene signal remains; a GP run
    might still be informative."*
- **Chunk 7 follow-up (done — `scripts/tmb_resid_gp.py` synergy
  check on MSI-residualized TMB):** Second isolated script that
  answers whether the DSL finds gene COMBINATIONS beating the best
  single gene on the "leftover TMB" target the diagnostic borderlined.
  ONE new file under `scripts/` (biology-aware) + a small helper
  refactor in `validate/tmb_resid_rank.py` to keep the residual
  byte-identical between the diagnostic and the GP script.
  - **`validate.tmb_resid_rank.build_residual_cohort()`** —
    factored-out public helper returning `(X_named, residual_series,
    group_stats)`. Both `main()` and `scripts/tmb_resid_gp.py` call
    it, so any tweak to the within-MSI z-scored `log1p(TMB)` target
    lives in ONE place. Existing diagnostic behaviour byte-for-byte
    identical (verified by pytest 130/130).
  - **`scripts/tmb_resid_gp.py`** — reads the existing processed
    matrix; imports the residual via `build_residual_cohort()`;
    `anonymise(X_named)` reuses the sealed map; runs
    `run_v2_pipeline(M, y, objective=TMB_OBJECTIVE, seed, ...)` per
    seed with `prefilter_n=None`, `scalar_share_override=0.0`
    (Vector programs only), `coherence_weight=0.0`, diversity on
    (`tournament_k=2, p_mutate=0.85, immigrant_fraction=0.10`);
    default budget `--pop 300 --gens 50 --perms 200` on seeds
    `[1, 7, 13]`. Per-seed grader reveals only the winner's opaque
    IDs via `airgap.reveal` (bounded — same discipline as
    `/evaluate`) and computes the SINGLE-GENE CEILING on the SAME
    held-out test rows via `_test_ids_for(M, y, seed)` (mirrors the
    pipeline's continuous split: `make_split(stratify=False,
    random_state=seed)`) + `_spearman_per_column(X_test, y_test)`.
    `synergy = combined − ceiling`. Prints per-seed
    program_repr / revealed genes / combined |spearman| (with
    n_test) / ceiling / synergy / permutation p, ranges across
    seeds, then an overall verdict:
    - `median syn ≥ 0.10 && n_sig ≥ ⌈n/2⌉ && median combined ≥ 0.30`
      → **REAL COMBINATORIAL SIGNAL**.
    - `|median syn| < 0.05 && median combined < 0.50` →
      **NO SYNERGY**.
    - `median combined < 0.20 || range ≥ 0.20` →
      **THE LEFTOVER IS LARGELY NOISE**.
    - else **BORDERLINE**.
  - **Airgap.** `engine_v2` sees only opaque IDs; symbols cross the
    boundary once per seed via a bounded `reveal(winner_ids)` call.
    No engine / API / UI / dataset change; writes nothing to disk;
    deleting the script leaves zero trace.
  - **Verified live.** Smoke `--seeds 1 --pop 60 --gens 10 --perms 20`
    prints the full per-seed block + ranges + verdict; scipy's
    `ConstantInputWarning` from degenerate programs (already floored
    to `WORST_FITNESS` by the engine) is silenced at script scope.
    `pytest -q` still 130/130.
- **Chunk 7 follow-up (done — Groups-the-engine-explored:
  ascending/descending sort toggle):** Frontend-only. The Groups
  table (`<ModuleRankingPanel>` in `web/app/Lab.tsx`) sorted
  descending only; clicking the active sort key now flips direction,
  clicking a new key resets to descending, and the active button's
  label appends `↓` / `↑` so the current direction is legible.
  Implementation: new `SortDir = "asc" | "desc"` state next to
  `sortKey`; `sortedModules` refactored to a single `valueOf`
  switch with null / non-finite modules PINNED to the bottom in
  BOTH directions (so a missing metric never floats to the top on
  ascending). Reset effect fires on `sortDir` too — flipping
  direction resets page 0 and closes any expanded row. `tsc
  --noEmit` clean; no API / airgap / engine change.
- **Chunk 7 follow-up (done — Groups-the-engine-explored: plain-
  English copy pass on the panel + per-column "?" tooltips):**
  Presentation-only. Two copy rewrites and four new column-header
  tooltips in `<ModuleRankingPanel>` (`web/app/Lab.tsx`).
  - Subtitle rewritten to the four-column story ("Every gene group
    the engine tried, scored four ways so you can judge whole
    groups, not just single genes") with a one-liner gloss per
    column. Fixes a literal `&rsquo;` that was rendering in the
    old subtitle; straight apostrophes only.
  - `TIPS.moduleRanking` (the panel-title `?`) rewritten to lead
    with "Why these columns exist" — the single-gene vs group
    question the panel is meant to answer — followed by the
    honest re-scoring caveat.
  - Each of the four metric column headers (Genetic-programming
    fitness · Combined AUROC / |ρ| · Coherence · Synergy) now
    carries a small `<InfoTip>` that says how to READ that column
    (high vs low), wrapped in
    `<span className="inline-flex items-center justify-end gap-1">`
    so it hugs the right edge with the text-right header. Combined
    header's `metric_kind` conditional preserved inside the wrapper.
  - `tsc --noEmit` clean; no API / airgap / engine change.
- **Next:** survival + unsupervised objectives end-to-end; bigger
  default Lab budget for the full grammar; cross-cohort validation;
  durable run store; mechanism-aware objective; lifting Search's
  default rate once its cost profile is profiled.

## Layout

```
data_pipeline/    download + build for both datasets. CRC:
                    download.py + build.py → data/processed/*.parquet
                    HNSC: download_hnsc.py + build_hnsc.py →
                    data/processed_hnsc/*.parquet (with hpv_status label)
app/              Streamlit viewer + presentation theme (theme.py)
.streamlit/       theme config.toml (light, Helvetica)
data/raw/         cBioPortal CRC files (gitignored)
data/raw_hnsc/    cBioPortal HNSC files (gitignored)
data/processed/   clinical.parquet, expression.parquet,
                  _sealed_gene_map.json,
                  h2/{evolution_log,result}.json   (all gitignored)
data/processed_hnsc/  HNSC clinical.parquet (with hpv_status) +
                      expression.parquet (gitignored)
dsl/              DSL operators (Load, Select, Reduce, Split, Associate,
                  Effect, Search, Fit, Apply) — label-agnostic, biology-free
airgap/           anonymise / reveal + sealed symbol<->ID map
engine/           v1 GP engine (split, prefilter, program, fitness,
                  baseline, permutation, gp, pipeline) — strict airgap,
                  biology-free. Fixed 1–2 Select+Reduce sets fed into
                  LogisticRegression.
engine_v2/        v2 typed program synthesis (types, nodes, synthesize,
                  fitness, permutation, gp, pipeline) — strict airgap.
                  Full DSL grammar (Select/Reduce/Combine/Split/
                  Associate/Effect/FitApply/Search) over the opaque-ID
                  matrix; ExecContext carries named clinical fields
                  (stage, age) and label arrays (msi, tmb) only — no
                  gene names. Worst-score floor for degenerate
                  programs; winner-fixed permutation null. Search is
                  bounded + gated OFF by default.
api/              FastAPI app: legacy /health /run /result /reveal +
                  Lab endpoints /runs /runs/{id} /runs/{id}/stream
                  /runs/{id}/result /runs/{id}/population/{gen}
                  /evaluate. POST /runs accepts engine: v1|v2.
                  CORS-enabled; SSE via sse-starlette; worker-thread
                  bridge; _json_finite() walker on every return so the
                  wire never carries NaN/Infinity tokens.
scripts/          Top-level orchestrators (biology-aware): run_h2.py wires
                  data + airgap + engine together
validate/         h1.py: known-answer H1 fixture on the NAMED matrix
                  h2.py: reveal winner + check MMR overlap
                  tmb_rank.py: signed-Spearman per gene vs TMB on the
                    NAMED matrix; reports where MMR / IMMUNE genes
                    land + top-10 most-negative. Powers the
                    /diagnostic/tmb-rank endpoint and the colorectal
                    Reference-gene diagnostic panel.
                  hpv_rank.py: single-gene orientation-agnostic AUROC
                    per gene vs HPV+/HPV− on the engine's HNSC TRAIN
                    split; reports where CDKN2A + each cell-cycle
                    gene lands + top single-gene separators. Powers
                    /diagnostic/hpv-rank and the HNSC Reference-gene
                    diagnostic panel.
web/              Next.js (App Router) + TS + Tailwind + Recharts +
                  reactflow Lab page (Stage 1 MVP). Talks to FastAPI;
                  no separate build pipeline for the engine. Single
                  <InfoTip> drives every "?" in the UI. Program graph
                  is a custom React Flow canvas in ProgramGraph.tsx —
                  Tier-1 group per gene-set, Tier-2 wrapper for 2-set
                  programs, palette tokens only, default chrome stripped.
tests/            pytest, no network (test_dsl.py, test_airgap.py,
                  test_h1.py, test_build.py, test_engine.py,
                  test_engine_v2.py, test_api_airgap.py)
```

## Hard rules

- **Stubs stay biology-free.** `dsl/`, `airgap/`, `engine/`, `validate/` must
  not hardcode gene names, pathway names, or MSI-specific constants. The data
  layer and viewer use real gene symbols freely; the engine only ever sees the
  airgapped view.
- **Schema constants live in one place.** `data_pipeline/schema.py` —
  filenames, required columns, MSI thresholds. If cBioPortal renames a column,
  edit there, not in `build.py`.
- **MSI label is derived, not shipped.** cBioPortal has no clean MSI-H/MSS
  column. We derive from `MSI_SENSOR_SCORE` per the file's own documented
  thresholds (`MSI_SENSOR_HIGH = 10.0`, `MSI_SENSOR_LOW = 4.0`). `build.py`
  prints this provenance every run.
- **Presentation lives in `app/theme.py`.** Palette + Altair theme. Don't
  override per-chart with `.configure_*`. MSI colours/order: import from
  `theme`, never inline. The H1 composition diagram (DOT in `app/viewer.py`)
  follows the same palette family: operator nodes stone (`#F4F2EE`), score
  meta-concepts cream (`#FBEFE2` / `#BC6B2E`), output nodes cool-tinted
  (`#EAF0F2` / `#3A6B7E`), `Given:` input notes muted grey (`#EFEFEA` /
  `#9AA0A6`).
- **H1 conclusion is computed, not hardcoded.** `app/viewer.py` derives the
  pass/fail callout from three checks: A (MMR median lower in MSI-H), B
  (immune median higher in MSI-H), C (held-out AUROC ≥ `_AUROC_THRESHOLD`,
  currently 0.75). Editing the threshold or any check requires editing
  `_section_h1`, not the callout copy.
- **Airgap is load-bearing.** Only `airgap.reveal` may open
  `data/processed/_sealed_gene_map.json`. The engine must never import
  `reveal`, `airgap.seal`, or reference the sealed-map filename — a test in
  `tests/test_airgap.py` scans `engine/` for those tokens. At runtime, both
  `dsl.Search` and `engine.run_gp_pipeline` assert their input matrices have
  columns matching `^g\d+$` only. Biology-aware orchestration lives in
  `scripts/` (the cohort label "MSI-H" / column name "msi_status" appears
  there and in `validate/`, but never in `engine/` or `dsl/`).
- **H2 artefacts are anonymised on disk.** `data/processed/h2/evolution_log.json`
  and `data/processed/h2/result.json` only ever contain opaque IDs. The
  reveal step happens at runtime, exactly once, via the FastAPI `/reveal`
  endpoint (or `validate.h2.reveal_winner`).
- **Lab payloads are anonymised over the wire.** `tests/test_api_airgap.py`
  POSTs synthetic runs against both presets, drains the SSE stream, and
  asserts no gene symbol leaks through `/runs/{id}`, `/runs/{id}/result`,
  or the stream. `/evaluate` translates only the IDs the client sends —
  never the whole map.
- **Two engine objectives, both selectable from the API.**
  `BinaryAUROCObjective` (MSI/MSS — stratified split + StratifiedKFold +
  AUROC) and `CorrelationObjective(direction)` (continuous, e.g. TMB —
  random split + KFold + signed Spearman of the sum-of-set-means with
  the target). Each objective owns its own `prefilter_score_per_feature`
  and `permute(y)` so the GP/baseline/permutation loop stays generic.
  Add a new objective by subclassing `engine.objectives.Objective` and
  wiring it into `objective_from_spec`.
- **Prefilter is a strict-speed knob, not a correctness knob.** When
  `prefilter_n=None` the GP samples (init + mutation) from the full
  opaque column set, so it can compose any gene combination present in
  the matrix. When an integer is passed, the GP is confined to the
  univariate top-N — faster, but a gene that only shows signal in
  combination can be dropped. The baseline always uses the univariate
  top-K (independent of the GP's pool) so it's a fair sanity check
  either way.
- **Fitness never leaves the API as NaN / Infinity.** Engines floor
  degenerate-program fitness to the objective's finite worst
  (`worst_score()` on `V2Objective`: 0.5 for MSI omni-AUROC, 0.0 for
  TMB correlation). The API also runs every endpoint return + SSE
  payload through `_json_finite()` so anything that slips through
  becomes `None`. Frontend uses one `fmtFit(x)` helper everywhere a
  fitness/score is printed — falls back to "—" for non-finite, and
  `fitnessForOrder(x)` sinks non-finite tiles to the bottom of the
  grid with the lightest tint.

## Setup & run (Python 3.11 venv in `.venv/`)

```bash
source .venv/bin/activate
python -m data_pipeline.download   # cBioPortal -> data/raw/
python -m data_pipeline.build      # -> data/processed/*.parquet
pytest
python -m scripts.run_h2           # GP engine -> data/processed/h2/*.json
uvicorn api.app:app --reload       # API on http://localhost:8000
streamlit run app/viewer.py        # Streamlit viewer (Dataset / H1 / H2 tabs)

# Lab (Next.js front end against the same API)
cd web && npm install && npm run dev   # http://localhost:3000
```

## Data provenance

- Study: `coadread_tcga_pan_can_atlas_2018` (TCGA CRC PanCancer Atlas).
- Source: `https://media.githubusercontent.com/media/cBioPortal/datahub/master/public/<study>/<file>`
  (the LFS-resolved URL; the S3 tarball mirror 403s, `raw.githubusercontent.com`
  returns 131-byte LFS pointers).
- `data_mutations.txt` is **optional** — datahub is over its GitHub LFS budget
  for that file. Chunk 1 does not need it (TMB comes from
  `data_clinical_sample.txt` `TMB_NONSYNONYMOUS`).
- Expected cohort after `build`: ~594 samples (378 COAD, 155 READ, 61 MACR),
  20,504 genes after dedup, **558 usable** (MSI in {MSI-H, MSS} ∧ expression
  ∧ stage ∧ age). Sanity check: median TMB MSI-H ≈ 39, MSS ≈ 3.

## Style

- Default to no comments. The README and this file carry the why.
- Keep `tests/` network-free; use synthetic cBioPortal-shaped fixtures.
- Streamlit charts: `width="stretch"` (the newer API; `use_container_width`
  is deprecated in 1.58+).