CLAUDE.md

OncoDSL — context for Claude

Long-term goal: a genetic-programming engine that, over a small DSL of data operations, rediscovers the MSI / mismatch-repair gene signature from TCGA colorectal data blind. The engine must not see real gene names — an "airgap" layer between the data layer and the engine hides them.

Chunk status

Built in chunks. Current state:

• Chunk 1 (done): project scaffold + cBioPortal data load + Streamlit sanity viewer. See README.md.
• Chunk 2 (done): DSL operators (Load, Select, Reduce, Split, Associate, Effect, Search, Fit, Apply) live in dsl/. Airgap (anonymise / reveal + sealed map at data/processed/_sealed_gene_map.json) lives in airgap/. H1 verification (known-answer MMR + immune check on the NAMED matrix) lives in validate/h1.py. Viewer is split into two top-level tabs — Dataset (the five sanity panels) and Hypothesis 1 (intro, DSL reference with nouns + verbs, graphviz composition diagram with Given: input nodes, then verification outputs in this order: dynamic pass/fail Conclusion → score distributions → Effect → Fit → collapsed "How to read this", plus an airgap demo). Glossary expander sits above the tabs.
• Chunk 3 (done): GP engine in engine/ — strict airgap; composes Select/Reduce/Fit programs over the anonymised matrix with prefilter + tournament-selection GP + permutation null. Orchestrator at scripts/run_h2.py (Load → anonymise → engine → persist artefacts). Artefacts at data/processed/h2/{evolution_log,result}.json, both anonymised. FastAPI surface in api/app.py (/health, /run, /result, /reveal) — /reveal is the only endpoint allowed to read the sealed map and translates only the supplied IDs. Viewer gains a "Hypothesis 2" tab (5 cards: hypothesis → setup → evolve replay → result → reveal + computed conclusion) that fetches from the API. Validation lives in validate/h2.py (reveal + MMR overlap).
• Chunk 4 (done — Stage 1): React/Next.js "Lab" at web/ (App Router + TS + Tailwind + Recharts). Adds a second selectable engine objective — CorrelationObjective(direction) for continuous targets like TMB — alongside BinaryAUROCObjective for MSI. New API: POST /runs (worker thread + SSE bridge), GET /runs/{id}, GET /runs/{id}/stream (sse-starlette), GET /runs/{id}/result, POST /evaluate (reveal supplied IDs + score overlap with "MMR" or "immune"). Lab page is one Client Component with 5 sections: objective builder (survival / unsupervised disabled, Stage 2) → parameter inputs → Run → live fitness curve + population grid (survivors solid accent, discarded faded) → result + reveal & evaluate. Reuses the same engine — scripts/run_h2.py, the legacy /run /result /reveal /health endpoints, and the Streamlit viewer are untouched.
• Chunk 4 follow-ups (done):
  • Prefilter optional + bigger GP budget. Engine prefilter_n: int | None; None means the GP samples its initial-population AND its mutation operator from the full opaque column set (~17k after NaN drop), not from a univariate shortlist. _prepare returns gp_pool + baseline_genes separately — the baseline keeps using the univariate top-K so it stays a fair sanity check. API accepts prefilter_n: null. Lab UI defaults to OFF (60 gens / 300 pop) with an "Off (all genes) / On" toggle; turning On flips back to 30 / 150 / prefilter_n=2000. scripts/run_h2.py untouched (explicit --prefilter-n 2000).
  • InfoTips everywhere. Single <InfoTip> Client Component drives every "?" in the Lab (objective cards, parameters, run / live / result / evaluator labels + sub-labels). Verbatim tooltip copy lives in the OBJECTIVE_TIPS / PARAM_TIPS / TIPS records at the top of Lab.tsx — never inline.
  • Program graph (React Flow). web/app/ProgramGraph.tsx reads feature_sets straight from the winner / candidate payload (no backend change) and renders the DSL program as a node graph: Tier-1 lanes [Expression matrix → Select(opaque IDs) → Reduce (mean) → Score pill] per gene-set, then [Score(s) → Fit → output node]. Tier-1 = light-teal group box, Tier-2 = dashed #BC6B2E wrapper (omitted for single-set programs). Custom node types only, palette tokens only; React Flow chrome stripped via ProgramGraph.css (opacity:0 handles, no Background, no MiniMap, no Controls, attribution hidden, scroll passes through). The graph card defaults to the winning program; clicking any candidate in the population grid swaps the graph to that candidate in place. After the Evaluator runs, Select-node pills show real symbols and matched reference genes are filled in the project highlight (#BC6B2E).
• Chunk 5 (done — engine_v2 + full-population UI):
  • engine_v2/ — strongly-typed program synthesis. Programs are trees over a five-type grammar (Matrix, Vector, FeatureSet, Agg, Op). Nodes: MatrixTerminal, Select(Matrix,FeatureSet) →Matrix, Reduce(Matrix,Agg)→Vector, Combine(Vector,Vector,Op) →Vector. Reduce.agg ∈ {mean, median, max, min, var}, Combine.op ∈ {add, sub, mul, protected_div, mean}. Output type = Vector. Split / Effect deferred to v2.1. Ramped half-and-half init; tournament + elitism; subtree crossover with strict type matching; subtree + point mutation; depth/node budgets. Termination guarantee: depth-floor closes Matrix to MatrixTerminal and Vector to Reduce(MatrixTerminal, agg), so no tree ever has an open slot. Reduce.agg vocab also widened in dsl.Reduce itself.
  • Two v2 objectives, with finite worst-case fallbacks: BinaryAUROCObjective (MSI/MSS — orientation-agnostic max(AUROC, 1−AUROC), worst = 0.5) and the v2 CorrelationObjective (TMB — −spearman(score, TMB), worst = 0.0). Degenerate programs (empty FeatureSet, constant/NaN output) get the worst score; they never leave the API as NaN/Infinity.
  • Permutation null is winner-fixed in v2: hold the winning program fixed and shuffle the target N times; p = fraction of nulls ≥ observed. The old "rerun the univariate baseline on shuffled labels" null does NOT transfer to v2 and was dropped.
  • API: POST /runs accepts engine: "v1" | "v2". v2 worker uses engine_v2.run_v2_pipeline_streaming. Full population persisted per generation in run.log[i].candidates (both engines); SSE stream keeps the bandwidth budget light by trimming to top-12. GET /runs/{id} exposes generations_persisted; full per-gen populations come from GET /runs/{id}/population/{generation} (404 for unpersisted indices, never 500). All endpoint returns, the in-memory run.result, and the SSE payloads pass through _json_finite() — NaN / ±∞ → None, so JSON.parse never breaks.
  • Frontend (web/): Lab now posts engine: "v2". Replaced the old top-12 grid with PopulationTiles (full per-gen population, fitness-tinted teal ramp, "Nn·dD" structure signature, generation stepper bounded to generations_persisted). New lib/programRepr.ts is the SHARED parser for the typed program_repr (also tolerates the legacy v1 Fit(…) shape). ProgramGraph.tsx was rewritten to consume the parsed tree and now renders arbitrary Select/Reduce/Combine/M shapes; same custom nodes, same ProgramGraph.css chrome strip. PasteToDraw.tsx feeds arbitrary program_repr text into the same renderer. ParameterFlow.tsx adds a collapsed "How these parameters relate" disclosure under the Parameters card caption — a centered single-column HTML/flex diagram (max-width 460px, width 100%, never overflows): step boxes, colour-grouped space/effort/ validation pills that wrap on narrow widths, plain text ↓ arrows, amber final "permutation p" box. Generations pill uses a ↻ text prefix (not an icon glyph). A single lib/fmt.ts::fmtFit() formats every fitness/score across tiles, cards, tooltips, the result panel, and the live curve — falling back to "—" if a non-finite value ever reaches the UI; tiles also use fitnessForOrder() so any non-finite sinks to the bottom of the grid with the lightest tint.
• Chunk 5 follow-ups (done — UX polish, no GP / API changes):
  • Rich Parameter help. New <ParamHelp paramKey> Client Component replaces the small InfoTip on the 8 Parameters labels. Hover/focus still shows the SHORT one-liner; click opens a modal dialog with text + diagrams. ParamHelpProvider mounts a single portal-modal at the Lab root so any trigger can open it (the ParameterFlow chart's pill names are also ParamHelp triggers). Registry of detailed content lives in web/app/paramHelpContent.tsx; six diagrams (Generations loop, MaxSets, Lambda trade-off, Seed dice, Permutations histogram, Prefilter funnel) live under web/app/diagrams/. Modal is role="dialog" aria-modal="true" with Esc / X / backdrop close, focus trap, and focus return.
  • Program graph + paste-to-draw. ProgramGraph.tsx now adds Tier-1 (light teal #F1F6F7 / #3A6B7E) and Tier-2 (dashed #BC6B2E) wrapper nodes when the parsed root is Combine (single- score programs skip wrappers). PasteToDraw.tsx is now controlled (text lifted to Lab so tile clicks AND the winner-default both write into it), placeholder fixed to a well-formed example, hint spells out the Reveal restriction for pasted text, and a collapsed "Format & examples" disclosure documents the grammar (M, Select, Reduce aggs, Combine ops, rules, annotated example).
  • Copy & load. New web/app/CopyButton.tsx (navigator.clipboard with execCommand("copy") fallback) drives a "Copy program" button next to the winner in the Result panel and an icon-only copy overlay on every tile in PopulationTiles. Tile body uses role="button" tabIndex={0} so the inner icon-button can stop propagation; clicking the body loads the program into the textarea AND the graph.
• Chunk 6 (done — full DSL in engine_v2 + objective explainers):
  • Full DSL grammar in engine_v2. Typed grammar grew from {Matrix, Vector} to {Matrix, Vector, Scalar, Model}. New nodes: Split(Vector, Predicate)→Vector (predicates: score = above / below median, stage_late = AJCC III/IV; one level only, mean-centre per branch, min subgroup guard); Associate(Vector, target, kind)→Scalar and Effect(Vector, target, kind)→Scalar (Effect residualises on stage + age — observational backdoor adjustment, "only as good as the measured confounders"); FitApply(Vector, target)→Vector (Fit a LR/OLS on the score, Apply in place); Search(Matrix, k) →Matrix (bounded univariate nested ranker, capped at k ≤ 4 and 200 cols, gated OFF by default per A4 — visible in the grammar but rate 0 until cost is acceptable).
  • ExecContext threads through every Node.execute(ctx) — bundles M (opaque), clinical (named: stage, age only), and labels (named: msi, tmb only). The pipeline and worker plumb clinical + the OTHER target through so a v2 MSI run can still Associate / Effect / FitApply against TMB and vice versa.
  • Fitness handles every output type. Vector → objective metric on per-patient scores; Scalar (Associate / Effect) IS the fitness, oriented per objective (|·| for MSI, -x for TMB); Model → routed through FitApply which auto-Applies and emits a Vector. Worst-case floor still applies (0.5 MSI, 0.0 TMB).
  • Typed GP extended with type-safe init / crossover / mutation for every new node; ramped half-and-half now spawns ~20% Scalar-rooted programs so Associate / Effect appear in the population; rates dict configures Effect / Split / FitApply / Search injection. Termination guarantee: Matrix → MatrixTerminal, Vector → Reduce(MatrixTerminal, agg), Scalar → Associate(Reduce(MatrixTerminal, "mean"), target, "spearman").
  • Parser + renderer + format guide. web/lib/programRepr.ts parses every new operator (spaces tolerated). ProgramGraph.tsx renders Split / Search / FitApply / Associate / Effect as accent-bordered verb nodes (same theme). PasteToDraw's "Format & examples" disclosure documents the full grammar with one short annotated example per operator.
  • Objective modals. The four objective ? icons (MSI / TMB / Survival / Unsupervised) now use the same rich ParamHelp modal infrastructure as the parameters — verbatim SHORT + DETAILED text per the prompt, shared intro about "an objective is the rule that scores every program" and shared footer about Associate / Effect / Fit being expressible in the DSL itself. Old OBJECTIVE_TIPS constant removed; the "brokNA spell-checker" typo is gone.
  • A9. UI fitness label for TMB switched from "association (|spearman|)" to "negative association (signed −spearman with TMB)"; LiveView and held-out copy reworded to match. Engine fitness already used signed −spearman.
  • Airgap stays absolute. Only stage, age, msi, tmb flow through engine_v2 as named values; gene IDs remain opaque everywhere (program_repr, populations, API payloads). Airgap tests parametrised over both engines × both prefilter modes stay green.
• Chunk 6 follow-up (done — scoring target bound to the objective): A bug from chunk 6: under the TMB objective an engine_v2 run could produce Associate(Reduce(...), msi, spearman) — the program freely picked its own target. Fixed by removing the target as a selectable operand:
  • engine_v2.synthesize now threads objective_target through _grow_scalar, _grow_vector, random_program, ramped_population, and mutate. Every Associate / Effect / FitApply is constructed with target = objective.target.
  • Point mutation no longer carries a fit_target spot and the scalar_kind spot no longer flips the target — only the correlation kind (pearson / spearman) is mutable. Crossover never touched targets (subtree swaps preserve them).
  • engine_v2.fitness._check_target_binding(program, expected) walks every Associate / Effect / FitApply node and asserts n.target == expected. fitness_fn calls it first and floors any mismatch to WORST_FITNESS, so a stray target — even from a replayed tree — can never win.
  • program_repr still shows the bound target (e.g. Associate(score, tmb, spearman) under the TMB objective) so the graph and paste-to-draw renderers don't change.
  • The shared footer in all four objective ? modals now reads verbatim: "What the program chooses is how to build the score and how to compare it — a raw association (Associate) or a confounder-adjusted one (Effect), plus the correlation kind. What stays outside the DSL is the compass: the target it's scored against, in which direction, judged honestly on held-out data. The program can't pick the target — that would let the answer into the language."
  • New tests: test_synthesis_binds_scoring_target_to_objective (init + 600 mutations × both objectives — every node binds to the active target) and test_fitness_floors_mismatched_target_to_worst (hand-crafted bug-shape program is floored to WORST_FITNESS). 80 / 80 pytest pass; the API airgap and gene-symbol-leak tests are untouched.
• Chunk 6 follow-up (done — Live view polish, presentation-only):
  • Fitness curve chart. Recharts' default <Legend> is gone; in its place a custom <ChartLegend> HTML row sits below the chart with SVG-line swatches that mirror the actual strokes (solid accent for best, dashed muted for median). The X-axis title ("generation") is pulled OUT of the SVG and rendered as a centred caption between the chart and the legend so the three rows (ticks → axis title → legend) never overlap. Y-axis label uses a <CenteredYAxisLabel> that rotates <text> at viewBox.y + viewBox.height/2 with text-anchor="middle" — the title now sits beside the middle of the axis, not the top. Lines: best solid #3A6B7E strokeWidth 2.5, median dashed (4 4) #6E7F8C strokeWidth 1.5.
  • InfoTip copy. TIPS.fitnessCurve and TIPS.bestVsMedian rewritten verbatim — the y-axis tip explains negative-association strength (with the MSI/AUROC fallback parenthetical); the best-vs-median tip explains convergence vs. persistent gap.
  • Generation navigator (PopulationTiles). Replaced the click-only ‹ Gen N › stepper with: |‹ first / ‹ prev / a draggable range slider / › next / ›| latest / a numeric input with a "go" button. Fetch decoupled from display via two states: draftGen updates on every input event for live number, but the /runs/{id}/population/{gen} fetch is keyed on committedGen, which only updates on pointerup/pointercancel or a 150 ms throttle while dragging. Keyboard: native left/right arrows commit immediately; custom Home/End jump to first/latest. ARIA: aria-label="Generation", aria-valuemin/max/now, aria-valuetext, and a polite live region for the "Generation N / total" label.
  • DSL vocabulary tiles cover the full grammar. The Lab's "DSL vocabulary" panel now lists all eight engine_v2 verbs (Select, Reduce, Combine, Split, Associate, Effect, Fit/Apply, Search) so the panel matches what programs in the population can actually contain — earlier it only showed the three Vector-shape verbs and silently omitted Scalar/Model-shape and gated ops. Presentation only; engine grammar unchanged.
• Chunk 6 follow-up (done — TMB-rank diagnostic, MMR biology panel, MLH1 success reframe, real Unsupervised objective, live-view polish, gene-based unsupervised synthesis):
  • TMB-rank diagnostic. New validate/tmb_rank.py and a small GET /diagnostic/tmb-rank API endpoint. Mirrors api._prepare_lab_data target="tmb" exactly (Load('processed') filtered to tmb.notna() & expression-complete rows) and computes signed Spearman per gene vs TMB on the NAMED matrix (vectorised rank-Pearson, sub-second). Reports each MMR / IMMUNE gene's rank/N + the top-10 most-negatively-correlated genes. On the live TCGA cohort: MLH1 sits at rank 270 / 20057 (top 1.3%); PMS2 at 2214 (11%); MSH2/MSH6 not in expression (mutational, not silenced). The diagnostic lives in validate/ (allowed gene names) — the structural engine airgap test still scans only engine/. 3 new tests in tests/test_tmb_rank.py.
  • MMR biology panel. web/app/MMRBiologyPanel.tsx mirrors ParameterFlow.tsx's collapsible exactly (button + chevron + aria-expanded + useId-panel-id + closed-by-default). Body inlines mmr_reference.svg verbatim (cause→effect: MMR → MSS / MSI-H → immune response, plus the gene-expression visibility readout). Mounted between the Objective card and Parameters card in Lab.tsx.
  • MLH1 success callout (TMB). /diagnostic/tmb-rank is now fetched at the Evaluator level (lifted out of the inner TMBRankPanel) so the same response also drives an accent SUCCESS callout above the overlap table — "Blind, MLH1 ranks 270 / 20057 (top 1.3%) on the TMB objective — the causal gene surfaced near the top without ever seeing gene names." OverlapSummary gains an optional muted secondary line so the "0 of N winning genes" read no longer reads as flat failure.
  • Real Unsupervised objective (engine_v2). New UNSUP_OBJECTIVE with target="none", metric="structure". Fitness = silhouette of a 2-means split on standardised 1-D score (range [-1, 1], worst floored at -1.0). Vector-only programs: V2Objective.synthesis_overrides() returns {rates: {split:.1, fitapply:0, effect:0, search:0}, scalar_share:0} for unsup, plumbed via gp.py into ramped_population / mutate. _build_ctxs strips all labels when target='none' (asserted) so the engine literally cannot see msi/tmb during search. New unsup_random_null in permutation.py — N random Vector-only programs scored on held-out (no target to shuffle); permutation_summary.null_kind distinguishes the two nulls in the result.
  • Post-hoc alignment. Worker thread for unsup target runs _compute_unsup_posthoc AFTER the GP finishes: orientation-agnostic AUROC of winner scores vs MSI (held-out subset, gated n≥10 per class) and |spearman| vs TMB. Lives in api/_worker (allowed labels) — never in engine_v2. New result.posthoc = {msi_auroc, tmb_abs_spearman, n_holdout, n_msi_held, n_tmb_held} rendered in <PosthocAlignment> inside the ResultPanel.
  • Frontend — drop Survival, add Unsupervised end-to-end. Target widened to "msi" | "tmb" | "none"; OBJECTIVE_PRESETS.none, FITNESS_LABEL_BY_TARGET.none. Survival entry removed from ObjectiveBuilder + paramHelpContent.tsx. For unsup runs the Evaluator hides the reference-set toggle / overlap / matched column (no overlap concept) and falls back to the post-hoc block. The RunResult interface in web/lib/api.ts gains posthoc?: Posthoc.
  • Live-view + objective-modal clarity (presentation-only). Shortened y-axis titles (tmb: "neg. association with TMB", none: "cluster separation (0–1)"). Replaced the single TMB-centric TIPS.fitnessCurve with FITNESS_TIP_BY_TARGET: Record<Target, string> and plumbed target into LiveView so the InfoTip matches the y-axis title's objective-awareness. Added TIPS.nodes and TIPS.genes and passed them to the NODES + GENES Metric cards in the ResultPanel. Rewrote ObjectiveIntro so it no longer asserts "a target column" unconditionally; rewrote obj_msi / obj_tmb / obj_unsupervised modal bodies verbatim per prompt. obj_unsupervised no longer renders the target-centric ObjectiveFooter — new UnsupObjectiveFooter ("no target, can't pick one") in its place.
  • Silhouette guard hardened (engine_v2/fitness.py _score_silhouette). Winsorize the score to its 2.5–97.5 percentile range, then recompute std on the winsorized vector; if std_w < 1e-9 → worst floor. This catches the protected_div-by-self failure mode (huge raw std + a small handful of outliers → a 98:2 "perfect" split with silhouette ~1). Min-cluster floor also raised from 10% to 30% of n. test_unsupervised_silhouette_kills_self_divide_outlier_split locks the fix.
  • Gene-based unsupervised synthesis (engine_v2/synthesize.py). Earlier unsup winners could be no-Select trees that did the splitting work via Split(..., predicate="stage_late") — GENES=0, silhouette 1.0 trivially, no actual gene discovery. Fixed by: _grow_matrix now takes objective_target; under unsup every Matrix leaf is Select(MatrixTerminal(), sampled FeatureSet) (never bare MatrixTerminal), Search is gated off, and the _grow_vector depth-floor becomes Reduce(Select(M, [...]), agg). _grow_vector's Split branch forces predicate="score" under unsup; point mutation's predicate spot is suppressed entirely for unsup so a "score" can't be flipped to "stage_late" mid-run. Belt-and-braces fitness floor: fitness_fn and evaluate_holdout return WORST_FITNESS when objective.target == "none" and program.feature_ids() == []. Two new tests (test_unsupervised_programs_always_use_gene_select over 120 programs × 20 mutations, and test_fitness_floors_no_select_program_under_unsup on the literal bug-shape program).
  • Verification. pytest 94/94, tsc clean. Live unsupervised TCGA smoke (10 gens × 60 pop, prefilter 2000) across four seeds shows every winner is gene-based (1–5 genes), Split predicate is score (no stage_late), held-out silhouette ≈ 0.55–0.60 (not the trivial 1.0), and the discovered split aligns blind with MSI at AUROC up to 0.774 — the engine rediscovered the MSI subtype without ever seeing the label.
• Chunk 6 follow-up (done — generalisation-aware unsup silhouette + plain-reading verdict):
  • OOS silhouette in cv_score AND evaluate_holdout. Earlier unsup runs scored silhouette IN-SAMPLE inside CV (the KMeans was fit on the same fold-test points it then evaluated) — so programs that overfit looked great during selection and collapsed on the final held-out (often to -1.0 worst floor). New V2Objective._score_oos_silhouette(train_scores, test_scores) fits the winsorize bounds, mean/std, and KMeans centres on TRAIN scores only; test scores are clipped + standardised with the train statistics and assigned to the train centres via KMeans.predict. Guards live on the TEST side (≥30% per cluster, near-constant after clipping → worst). cv_score per fold now uses (tr, te) not just te. evaluate_holdout gained an optional ctx_train kwarg; under unsup it executes the program on TRAIN and TEST then calls the OOS scorer. MSI / TMB paths ignore the new kwarg (those metrics are already on the held-out vector by construction).
  • OOS null too. unsup_random_null gained ctx_train and forwards it into evaluate_holdout — otherwise the random Vector-only nulls would be scored in-sample while the winner is scored OOS, deflating the p-value. The pipeline now plumbs ctx_train into both the winner's evaluate_holdout and the null-distribution loop.
  • Plain-reading verdict at the top of the Result panel. New <ResultVerdict> rendered first inside ResultPanel. Per-target thresholds (MSI 0.75, TMB 0.30, unsup 0.30) plus p < 0.05 gate a green "Real result: it holds up on unseen patients (X) and beats chance (p Y)" callout vs an amber "This run found nothing reliable: held-out X (fell apart on unseen patients) · p Y (a random program beats it ~N% of the time). Don't read the genes as a discovery." For unsup the green callout appends one of three tails based on posthoc.msi_auroc (≥0.75 → "rediscovered the subtype"; <0.75 → "doesn't line up with MSI"; null → "too few held-out labels"). Constants HOLDOUT_THRESHOLD, P_VALUE_SIGNIFICANT, POSTHOC_MSI_ALIGN at the top of Lab.tsx are tunable.
  • Honest unsup numbers. With OOS silhouette as the selection metric the GP now selects for consistent two-cluster structure rather than splits that happen to align with MSI via in-sample overfitting. seed=13 used to report in-sample silhouette 0.6 + MSI-AUROC 0.774 (alignment was partly an in-sample artefact); now it reports OOS silhouette 0.900, p 0.024, MSI-AUROC 0.546 — a real generalising cluster split that doesn't happen to be MSI. The verdict surfaces that mismatch in plain words.
  • Four new tests in tests/test_engine_v2.py: test_oos_silhouette_rewards_consistent_split (train+test both bimodal → ≥0.70), test_oos_silhouette_kills_non_generalising_program (train bimodal + test noise → drops >0.3 below in-sample), test_oos_silhouette_train_only_constant_floored, test_oos_silhouette_too_small_floored. pytest 98/98 pass.
• Chunk 6 follow-up (done — "What does this mean?" verdict breakdown, presentation-only):
  • The verdict callout now has a collapsed-by-default disclosure underneath it (mirrors ParameterFlow.tsx's collapsible — button
    • chevron + aria-expanded + aria-controls + useId-keyed panel + closed-on-first-render).
  • Expanded content branches on objective_spec.target: <MSIBreakdownContent> and <TMBBreakdownContent> use a one- question shape ("Is this result real and useful?") with two bullets — held-out separation/association and beats-chance — plus a → Real and useful. / → Not reliable… capstone tied to the same generalises/significant booleans the headline uses. <UnsupBreakdownContent> uses the two-question shape from the prompt verbatim ("Is the split real?" / "Is that split the MSI subtype?") with the hair-colour analogy at the bottom. All numbers route through fmtFit so non-finite renders as em-dash.
• Chunk 6 follow-up (done — iterative unsupervised discovery, "peel off axes"):
  • Engine. engine_v2/pipeline.py gained _residualise_matrix(M, residualize_scores) — vectorised OLS via np.linalg.lstsq over [intercept, *priors]. Rows missing a prior score are dropped BEFORE the train/test split so train and test see the same residualised feature space. Both run_v2_pipeline and run_v2_pipeline_streaming gained residualize_scores: pd.DataFrame | None = None (no-op for MSI/TMB; only the unsup API worker sets it). Both functions now emit winning.full_scores + winning.full_sample_ids (full- cohort per-patient scores) alongside holdout_scores — these feed the chain's next residualisation step.
  • API. RunRequest.residualize_against: list[str] | None. _assemble_residualize_df validates each prior (must exist + be done + target=="none" + have full_scores); turns _json_finite's None back into NaN so the pipeline's NaN filter drops those patients during residualisation. post_runs validates the chain BEFORE spawning the worker — supervised target + priors → HTTP 400; unknown / wrong-target / no-scores prior → HTTP 400 with the specific reason. _worker gained residualize_against; only assembles + forwards the DataFrame for unsup runs. Chain is in-memory only — valid within a single server session.
  • Frontend. Lab.tsx grew an axes: {run_id, result}[] state. start() refactored into launchRun(opts?) so a fresh Run (launchRun(), clears axes) and findNextAxis() (launchRun({residualize_against: axes.map(a=>a.run_id)}), preserves the chain) share the same SSE wiring. The SSE done handler appends unsup results to axes. New <DiscoveredAxes> section under the Evaluator renders an ordered stack of compact <AxisCard>s (accent border when both verdict gates pass; holdout / p / revealed opaque IDs / post-hoc MSI AUROC + TMB |spearman| in one line) plus a "Find next axis →" button. The TIPS.discoveredAxes tooltip flags the linear-residualisation heuristic and the in-memory-chain caveat.
  • Tests. Extended tests/test_api_airgap.py: test_unsupervised_run_emits_posthoc_alignment now asserts winning.full_scores + winning.full_sample_ids are populated; new test_residualisation_chain_runs_and_stays_airgap_clean posts axis 1 then axis 2 with residualize_against=[axis1] and confirms the second payload is opaque-only and carries its own full_scores; new test_residualize_against_unknown_id_returns_400
    • test_residualize_against_rejected_on_supervised_target. 101 / 101 pytest pass.
  • Verified live on TCGA (seed=13, 6 gens × 50 pop, prefilter 2000):
    • Axis 1 (5 genes, no overlap with MSI): holdout 0.597, p 0.065, MSI-AUROC 0.552, TMB-|spearman| 0.009.
    • Axis 2 (residualised against Axis 1; 5 different genes): holdout 0.653, p 0.097, MSI-AUROC 0.706, TMB-|spearman| 0.248. Zero shared genes between the two axes — the residualisation freed the search to land on an orthogonal direction that's more MSI-aligned than the first.
• Chunk 7 (done — second dataset: HNSC + HPV detection):
  • Dataset pipeline. New data_pipeline/{download_hnsc,build_hnsc}.py mirror the colorectal scripts for TCGA HNSC PanCancer Atlas (hnsc_tcga_pan_can_atlas_2018, LFS-resolved media URL). The build script derives hpv_status ∈ {HPV+, HPV−} from whichever column carries it: first data_clinical_sample.txt::HPV_STATUS*, then data_clinical_patient.txt::HPV_STATUS, then the SUBTYPE suffix fallback (e.g. HNSC_HPV+ / HNSC_HPV-). On the real cohort the resolved source is patient.SUBTYPE. Writes data/processed_hnsc/{clinical,expression}.parquet (gitignored). HNSC schema constants live in data_pipeline/schema.py next to the colorectal ones; final usable cohort is 487 samples (72 HPV+ / 415 HPV−).
  • Loader. dsl.Load already handled directory paths via its else branch; the only change is that Cohort.labels now picks up hpv_status if present. Load(schema.HNSC_PROCESSED_DIR) returns the HNSC cohort with hpv_status in cohort.labels.
  • HPV objective (engine_v2). V2Objective.target widened to Literal["msi","tmb","none","hpv"]. New HPV_OBJECTIVE (binary=True, worst=0.5) routes through the SAME orientation-agnostic AUROC machinery as MSI; score_scalar mirrors MSI's abs(value) orientation. objective_from_spec dispatches hpv + auroc/auroc_omni. No new operators; no other engine surface changes.
  • API per-(dataset, target). _prepare_lab_data(target, dataset) keyed by f"{dataset}:{target}". New (hnsc, hpv) branch loads HNSC, restricts to called HPV samples + complete expression, anonymises, sets y = (hpv_status == "HPV+").astype(int). New (hnsc, none) unsup branch keeps every patient with complete expression and carries the HPV label aside via extra_labels so _compute_unsup_posthoc can compute orientation-agnostic hpv_auroc (mirrors the msi/tmb branches; new key Posthoc.hpv_auroc?). RunRequest.dataset + Run.dataset + server-side DATASET_TARGETS = {coadread: {msi,tmb,none}, hnsc: {hpv,none}} validation rejects mismatched combos with a 400 at the API boundary before any data prep.
  • Per-dataset reference sets. Replaced flat REFERENCE_SETS with REFERENCE_SETS_BY_DATASET. Colorectal keeps MMR + immune; HNSC ships p16 = ["CDKN2A"] and a 20-gene standard cell-cycle / E2F-target core (MCM2–7 / PCNA / CDK1 / CCNE1 / CCNB1 / CDC6 / CDC20 / MKI67 / TOP2A / RRM2 / TYMS / FOXM1 / E2F1 / BUB1 / AURKB). EvaluateRequest.dataset is required and cross-dataset reference-set requests return 400.
  • Frontend dataset registry. Target widened to include hpv; new DatasetId = "coadread" | "hnsc". DATASET_REGISTRY in Lab.tsx drives objective cards, biology panel, and reference-set keys per cancer. A <CancerSelector> segmented control mounts above the Objective row. New <HPVBiologyPanel> mirrors <MMRBiologyPanel> exactly and inlines hpv_reference.svg verbatim (RB / E2F / p16 / p53 / E6+E7 explainer + readout table + airgap-side disclosure). Switching cancers resets the chain + result state and snaps the objective to the dataset's default.
  • HPV verdict + "?" coverage. FITNESS_LABEL_BY_TARGET.hpv = "separation (AUROC)"; FITNESS_TIP_BY_TARGET.hpv reuses the MSI-style AUROC explanation worded for HPV+/−; HOLDOUT_THRESHOLD.hpv = 0.75; OBJECTIVE_PRESETS.hpv = {target:"hpv", metric:"auroc"}. New <HPVBreakdownContent> adds a supervised one-question shape ("Is this result real and useful?") — no two-question / hair-colour shape for HPV. Two new paramHelp modals: obj_hpv (verbatim per prompt: what it optimises, how it's scored, represented as, honest "detection / recovery of a known viral signature — not new causation" note). <UnsupAlignmentTail> + <UnsupBreakdownContent> + <PosthocAlignment> all branch on which named label the cohort carries (HPV vs MSI/TMB) so HNSC unsup runs surface a real alignment number instead of "(too few held-out labels)".
  • Per-dataset Reveal. <Evaluator> reads its reference-set toggle keys from DATASET_REGISTRY[dataset].refSetKeys. HNSC shows p16 | cell_cycle; colorectal stays on MMR | immune. <MLH1SuccessCallout> + <TMBRankPanel> gated to dataset === "coadread" && target === "tmb". The "Unsupervised run — no reference set" footnote reads dataset-agnostically.
  • Live verification (TCGA). MSI seed 7 — held-out 0.959, p 0.048. HPV seed 7 — held-out 0.869, p 0.048. HPV seed 11 — held-out 0.970, p 0.024, winner overlaps cell_cycle via MCM5. HPV seed 17 — held-out 0.957, MCM5 again. HPV seed 3 — E2F2 + RPA2 (cell-cycle/E2F + DNA replication, biologically on-target but not in the literature set). Colorectal + airgap tests untouched and green.
  • Airgap tests extended. tests/test_api_airgap.py fake_prep handles (target, dataset); new tests: test_hnsc_hpv_run_is_airgap_clean, test_hnsc_rejects_msi_target, test_coadread_rejects_hpv_target, test_evaluate_rejects_hnsc_reference_in_coadread. 105 / 105 pytest pass.
• Chunk 7 follow-up (done — HPV-marker-rank diagnostic): Mirror of the TMB-rank diagnostic for HNSC. New validate/hpv_rank.py ranks every gene by single-gene orientation-agnostic AUROC vs the HPV+/HPV− label on the engine's TRAIN split (make_split(seed=42, test_size=0.3, stratify=True) so the diagnostic never reads test data — same discipline the engine uses). Vectorised Mann-Whitney / rank-sum formula: auroc = (S_pos − n_pos·(n_pos+1)/2) / (n_pos·n_neg) then max(AUROC, 1−AUROC); sub-second over ~20k genes. New GET /diagnostic/hpv-rank endpoint mirrors /diagnostic/tmb-rank (lazy import, _HPV_RANK_CACHE, _json_finite()-wrapped). Three synthetic-cohort tests in tests/test_hpv_rank.py. Frontend: HPVRankDiagnostic interface + getHPVRankDiagnostic() in web/lib/api.ts; new <HPVRankPanel> in Lab.tsx reuses the existing <RankList> chrome and renders p16 + cell-cycle + top-N separators with the prompt's verbatim caption, gated to dataset === "hnsc" && target === "hpv". TIPS.hpvRank documents the heuristic. Live TCGA cohort: N(TRAIN) = 340 (HPV+ 50 / HPV− 290), 20,218 genes ranked. CDKN2A AUROC 0.877, rank 99 / 20,218 (top 0.5%) — meaningfully recoverable but 98 genes outrank it. Best cell-cycle ranks: MCM5 (22), MCM2 (28), MCM6 (38), MCM3 (72), PCNA (78), E2F1 (99, tied with CDKN2A). Top-10 led by TCAM1P / C19orf57 / ARHGEF33 / STAG3 / SMC1B (meiosis-cohesin) + RPA2 (DNA replication — what the engine actually picked at seed 3). 108 / 108 pytest pass; airgap unaffected (engine_v2 + dsl + engine untouched; structural scan still only walks engine/).
• Chunk 7 follow-up (done — UX polish: plain "?" copy, no all-caps, program-graph clipping): Presentation-only sweep across the Lab.
  • Objective-aware tooltips. Added four per-target / per-dataset records next to FITNESS_TIP_BY_TARGET: HELD_OUT_TIP_BY_TARGET, PERMUTATION_P_TIP_BY_TARGET, REFERENCE_SET_TIP_BY_DATASET, POSTHOC_TIP_BY_DATASET, DISCOVERED_AXES_TIP_BY_DATASET. Held-out / permutation p / posthoc / discovered-axes / reference-set ? all switch on the active objective or dataset so HPV runs no longer see MSI/TMB wording. <DiscoveredAxes> and <PosthocAlignment> receive dataset (or infer it from posthoc.hpv_auroc presence) and look up the right copy.
  • Plain English everywhere. Rewrote the verbose / typo'd legacy TIPS.heldOut, TIPS.permutationP, TIPS.posthoc, TIPS.discoveredAxes, TIPS.referenceSet to crisp sentences with one-line glosses for jargon (AUROC, held-out, permutation). Fixed the seed PARAM_TIPS typo ("xact" → "the exact"). The unsupervised fitness-curve tip no longer hardcodes "(MSI? TMB?)" — it now says "the named label this cohort carries".
  • ObjectiveIntro / UnsupObjectiveFooter dataset-aware. ObjectiveIntro now lists MSI / TMB / HPV (was MSI / TMB only). UnsupObjectiveFooter mentions both MSI (colorectal) and HPV (head & neck) as legitimate downstream alignments instead of MSI only.
  • No all-caps anywhere. Stripped Tailwind's uppercase utility from all 13 eyebrow / table-header sites in Lab.tsx + PasteToDraw.tsx. Removed textTransform: "uppercase" from the Tier label in ProgramGraph.tsx. Upcased the underlying literal strings that were relying on the CSS (Winning program, Revealed genes, Opaque ID / Symbol / Matched table headers, rank-table Symbol / Correlation / Rank / N / AUROC). grep -rn "uppercase\|textTransform" web/app returns zero hits.
  • Program-graph clipping fix. ProgramGraph.tsx gained a remount key={fitKey} derived from width × height × node-count × edge-count so ReactFlow's one-shot fitView re-runs every time the laid-out program changes (winner ↔ candidate click, paste, dataset swap). Lowered minZoom from 0.4 to 0.2 so fitView can scale a full two-tier graph down to fit the card width. Confirmed no overflow: hidden on the wrapping <SectionCard>. <ProgramGraph outputLabel> now dispatches by target — MSI-H probability / HPV+ probability / TMB association / cluster score instead of MSI-only.
• Chunk 7 follow-up (done — detection vs recovery + AUROC naming + ban whole-matrix Reduce + per-gene rank in Reveal):
  • Engine. Motivating live run produced Combine(Reduce(M, mean), Reduce(Select(M, …), mean), add) — a bare global-mean detector with a 7-gene additive tail; held-out AUROC 0.953 with 0/7 cell-cycle overlap. The dominant term detected HPV from bulk expression, not from gene choice. Fix: engine_v2/synthesize.py makes the Select-wrapping rule unconditional for every objective (was unsup-only). Every Matrix leaf is now Select(MatrixTerminal, FeatureSet); the Vector depth-floor returns Reduce(Select(M, …), agg); the bare-MatrixTerminal fall-through is gone. Belt-and-braces in engine_v2/fitness.py: new _has_bare_matrix_reduce(program) walks the tree and any Reduce on a bare MatrixTerminal floors to WORST_FITNESS for every objective (was unsup-only). The no Select → WORST_FITNESS floor is also no longer gated to unsup. Two test fixups: the closed-Matrix leaf at depth 0 is now Select(MatrixTerminal, …); the depth-budget contract is loosened by 1 because the mandatory Select adds a level. Live re-run on real HNSC (seed 7): winner is now Reduce(Select(M,[g17705]),max) — a single-gene detector, AUROC 0.964, no bare Reduce(M,…).
  • Detection vs recovery (presentation). HPVBreakdownContent
    • MSIBreakdownContent success copy changed from "the engine recovered the HPV viral signature" / "Real and useful." to "Real and useful — it separates HPV+ from HPV− (or MSI-H from MSS) on patients it never saw." Both breakdowns gained a muted secondary line — "This is a detection result. Whether the engine found the known marker genes (CDKN2A/p16, the cell-cycle program / MMR / immune) is a separate question — see Reveal & evaluate below." The top-level callout already used detection- only wording. Unsupervised "rediscovered the viral signature blind" stays — that's a post-hoc alignment claim, gated on posthoc.hpv_auroc ≥ 0.75.
  • Name AUROC where the metric IS AUROC. New HELD_OUT_LABEL_BY_TARGET — Held-out card label is Held-out AUROC for MSI / HPV, Held-out (|spearman|) for TMB, Held-out (silhouette) for unsup. HELD_OUT_TIP_BY_TARGET rewritten to name AUROC explicitly with a one-line gloss for MSI/HPV and to call out "Not AUROC: TMB is a continuous label / there's no label here" for TMB/unsup. Held-out bullet inside each breakdown switched Held-out separation → Held-out AUROC with the same gloss.
  • Per-revealed-gene rank in Reveal & evaluate. /evaluate request gains optional target; response rows gain optional rank / total / single_gene_metric / metric_kind ("auroc" for HNSC/HPV, "spearman" for coadread/TMB). New _gene_rank_lookup(dataset, target) precomputes a {symbol → rank/metric} map on first call by reusing validate/hpv_rank.py's _auroc_per_column (on the engine's TRAIN slice) and validate/tmb_rank.py's _spearman_per_column; cached for the process lifetime. Frontend: EvaluateRow gained the optional fields; postEvaluate passes target; the Revealed-genes table now shows a "Single-gene rank" column with {rank} / {total} plus AUROC X.XXX (HPV) or ρ X.XXX (TMB). Column appears only when at least one row carries a rank — MSI and unsupervised omit gracefully. New TIPS.singleGeneRank explains the column: "rank near 1 = real alternate marker; high rank = only helps in combination."
  • Verified live. HPV seed 11 on real HNSC (10 gens × 80 pop): winner Reduce(Select(M,[g06264,g08575,g10783,g14446,g18989, g19553,g05351,g10525]),mean), held-out AUROC 0.937, p 0.024. Cell-cycle overlap 0/8, BUT every revealed gene is a top-2000 single-gene HPV separator on its own: FANCI (rank 996, AUROC 0.779), INPP5B (332, 0.830), MIR924HG (237, 0.841), RAD1 (359, 0.825), UBD (1808, 0.746), WHSC1 (402, 0.821), DYRK1A (246, 0.840), MCCC1 (1334, 0.763). The story is now legible: real alternate markers (DNA-damage-response / chromatin players), not the canonical MCM / PCNA shortcut. 108 / 108 pytest pass; tsc clean; airgap tests untouched.
• Chunk 7 follow-up (done — coherence prior + supervised peel-off + full-ranking Result + Unsup card retired): Five coordinated landings that rebuild the Lab's reveal story around the full single- gene ranking.
  • Coherence prior (engine). New engine_v2.fitness._coherence_score(program, ctx) returns the mean absolute pairwise correlation among the program's Select'd opaque columns (range [0, 1]; names no gene or pathway). fitness_fn gained coherence_weight: float = 0.0; when > 0, fitness becomes base − λ·n_nodes + w · coherence. Threaded through gp.py + both pipeline.py entry points with default 0 so existing runs stay byte-for-byte unchanged. API: new RunRequest.coherence: bool = False; _worker translates to COHERENCE_DEFAULT_WEIGHT = 0.10 (modest enough that separation still dominates). Frontend: a Parameters checkbox "Prefer coordinated gene modules" — default OFF, posted in the run body. TIPS.coherence carries the prompt's verbatim copy.
  • Peel-off for supervised objectives. Both pipeline entry points now persist winning.full_scores + full_sample_ids for every objective (the is_unsup gate is gone in both spots). api.app.post_runs no longer requires target=="none" for residualize_against; _assemble_residualize_df validates that every prior shares both (dataset, target) with the new run. Frontend <DiscoveredAxes> mounts after ANY run (not just unsup) and the SSE done handler appends to the chain for every objective. Live verified on HNSC HPV seed 11: Axis 1 → AUROC 0.941 / 3 genes; Axis 2 (residualised) → AUROC 0.875 / 2 different genes; zero overlap. Updated airgap test: test_residualize_against_works_on_supervised_targets posts MSI Axis 1 → Axis 2 and asserts payload stays opaque + carries full_scores.
  • Full-ranking endpoint. New GET /diagnostic/full-rank?dataset= &target= returns {n_samples, n_pos, n_neg, n_genes, metric_kind, ranks: [{opaque_id, score, rank}, …], reference_marks: [{opaque_id, symbol, set_name, rank, score}, …]}. The ranks list is opaque-only (~20k rows). reference_marks is the small reference gene set (CDKN2A + 20 cell-cycle for HNSC; MMR + immune for CRC), looked up via the sealed map on the API side and revealed up-front so the rank track can label them — the whole map never crosses the wire. Computed on the same TRAIN slice the engine sees (make_split(seed=42, test_size=0.3, stratify=binary)), using the rank-sum AUROC formula for binary targets and signed Spearman for TMB. Unsupervised → 404 (no label to rank against). Cached per (dataset, target) for the process lifetime.
  • New <RankingResult> UI. Replaces the prior Evaluator + TMBRankPanel + HPVRankPanel + MLH1SuccessCallout for supervised targets. Renders, top-to-bottom:
    • Rank chart: a labelled lollipop chart on a log-rank axis (gridlines at 1 / 10 / 100 / 1k / 10k / N), with the winner gene(s) as accent-teal dots and each reference gene as a colour-coded dot. Labels stagger across 4 rows to avoid collisions near the top.
    • Highlighted-genes table with three-way colour-coded "Source" column (winner / p16 / cell_cycle for HNSC; winner / immune / MMR for CRC) — p16 reads visibly distinct from cell_cycle via a separate deep-gold palette. Columns: Source · Symbol · Opaque ID · Rank/N · Percentile · metric. Percentile shows "top 0.5%" for high ranks and flips to "bottom 21%" past the midpoint (2 sig figs).
    • Collapsed <RawRankingDisclosure> ("Raw anonymous ranking — proof the ranking is computed blind on opaque IDs") wraps the browsable opaque-ID list + search box (top-200 with substring filter). Caption restates the airgap purpose.
    • GP-vs-diagnostic note (verbatim per prompt) closes the panel.
  • Unsup card hidden, backend kept. DATASET_REGISTRY drops the none objective entry for both datasets — UI now shows coadread → {MSI, Mutation burden} and hnsc → {HPV detection}. UNSUP_OBJECTIVE, the unsup pipeline branches, the target === "none" frontend code paths (verdict / breakdown / post-hoc / Evaluator fallback), and tests/test_engine_v2.py stay intact — just unreachable from the UI.
  • 108 / 108 pytest pass; tsc clean; airgap tests still green.
• Chunk 7 follow-up (done — Ranking UI polish): Presentation-only pass on <RankingResult>. New three-way RANK_COLORS palette (winner accent teal / p16 deep gold / cell_cycle muted amber, plus immune/MMR aliases for CRC) drives every coloured surface — chart dots, table backgrounds, "Source" badges. Highlighted-genes table gains a "Percentile" column (top X% / bottom X% with 2 sig figs) so CDKN2A reads "99 / 20,218 · top 0.5%" and CCNE1 reads "15,918 / 20,218 · bottom 21%". <RankTrack> rewritten with a log axis (gridlines at 1 / 10 / 100 / 1k / 10k / N), a 4-row label stagger that avoids 60px-radius collisions, and an inline three-colour legend in the chart header. Raw opaque-ID browse table moved into a collapsible disclosure ("Raw anonymous ranking — proof the ranking is computed blind on opaque IDs"), closed by default. Page reads verdict → metric cards → log chart → highlighted table → collapsed raw → GP-vs-diagnostic note. tsc clean; no engine/API/airgap changes.
• Chunk 7 follow-up (done — peel-off leakage fix: train-only residualisation + FitApply train-fit + leakage guard): A peel-off Axis 2 HPV run produced FitApply(Reduce(Select(M,[g16970]),median),hpv) (SPACA1) at held-out AUROC 0.963 — impossible for a single-gene monotonic program whose gene's honest single-gene AUROC is 0.510. Root cause: residualisation was fit on the FULL cohort BEFORE make_split, so the OLS projection saw the held-out rows and smeared target signal into them.
  • engine_v2/pipeline.py — _residualise_matrix replaced with three helpers: _align_priors (drops rows missing a prior — defining the cohort, not leakage), _fit_residualise_beta (OLS of each gene on [intercept, *priors] fit on TRAIN rows only), _apply_residualise (applies train-fit β to the full M so train + test sit in the same residualised space without test ever being seen by the fit). Both run_v2_pipeline and run_v2_pipeline_streaming reordered: align → split → fit β on M.loc[split.train_ids] → apply β to full M → prefilter / GP / evaluate_holdout. winning.full_scores re-executes the winner on the train-only-residualised M with empty labels (the raw inner score for FitApply winners — monotonic with the fitted prediction in the 1-D case, so the chain stays consistent).
  • engine_v2/nodes.py — ExecContext gained fit_ctx: ExecContext | None = None. FitApply.execute now runs the inner subtree on fit_ctx (train) to fit LR/OLS and applies the FROZEN model to the test inputs (no fitting on test labels). Binary hpv now routes through the same logistic branch as msi. Identical behaviour in the single- score monotonic case (LR remains monotonic) so existing AUROCs don't move; the discipline is in place for any future multi- input FitApply.
  • engine_v2/fitness.py — LEAKAGE_TOLERANCE = 0.02. evaluate_holdout wraps the test ExecContext with fit_ctx=ctx_train when ctx_train is provided, then runs a backstop check: if the program is _single_gene_monotonic (one gene; no Combine/Split) AND its held-out AUROC exceeds the gene's single-gene AUROC by > tolerance, floor to WORST_FITNESS. Genuine multi-gene synergy is unaffected.
  • Tests — tests/test_engine_v2.py gained test_peeloff_residualisation_uses_train_rows_only (synthetic cohort + target-correlated prior; asserts the single-gene-monotonic held-out AUROC is within tolerance of the gene's honest single-gene AUROC under the new pipeline) and test_leakage_guard_floors_contaminated_single_gene_winner (guard sanity-check on an honest program). 114 / 114 pytest pass; airgap suite untouched.
  • Verified live (TCGA HNSC, HPV + coherence ON). Across seeds {3, 7, 11, 13} every Axis 2 is multi-gene, no SPACA1-style runaway wins, and Axis 2 held-out ≤ Axis 1 + tolerance: seed 11 → Axis 1 AUROC 0.941 / 3 genes → Axis 2 AUROC 0.923 / 8 genes (zero gene overlap). The residualise-before-split mechanism is closed.
• Chunk 7 follow-up (done — coordinated modules: ranked group view + ref-set highlights): Coherence-on runs now expose a ranked list of the coordinated gene modules the GP explored, sorted by combined held-out AUROC — so a real co-expressed program (e.g. cell_cycle) surfaces as a GROUP even when no single gene is the top separator.
  • API. New GET /runs/{run_id}/modules endpoint (api/app.py::_compute_module_ranking). Reproduces the run's EXACT train/test split via the persisted full_sample_ids / holdout_sample_ids so "held-out" actually is held-out; harvests distinct candidates' gene_ids sets (≥ 2 genes) across every persisted generation (engine_v2 already stores a flat gene_ids list per candidate); for each module computes combined held-out AUROC = orientation-agnostic AUROC of the per-patient mean across the module's genes vs the held- out label (parameter-free — nothing to fit, nothing to leak; |spearman| for TMB), coherence = mean abs pairwise correlation on TRAIN (the same quantity the prior rewards), and a per-gene single-gene AUROC + rank/N. Each module also carries ref_sets: list[str] — the dataset's reference sets (HNSC: p16 / cell_cycle; CRC: MMR / immune) whose opaque IDs intersect that module, resolved ONCE via the sealed map (bounded reveal of a small known set; same pattern /diagnostic/full-rank uses for reference_marks). Sorted by combined held-out desc; non-finite sinks to the bottom. Run dataclass gained coherence: bool; /runs/{id} now surfaces it (along with dataset). Endpoint returns 425 while running, 404 for unknown, 400 for unsupervised runs (no target to evaluate against).
  • Frontend. New <ModuleRankingPanel> rendered after <RankingResult>, gated to target !== "none" && coherence && runId. Sortable header (Combined AUROC / Coherence); each collapsed row carries a small badge per matched reference set and is tinted using the existing three-way RANK_COLORS palette (winner accent teal / single-marker sets like p16/immune deep gold / broader proliferation sets like cell_cycle/MMR muted amber) — so a cell_cycle-heavy module is scannable without expanding. Expanding a row calls POST /reveal with just that module's gene IDs (bounded — never the whole map) so every per-gene row gets a symbol (no more "—" for module genes outside the already-revealed subset, e.g. g09647 → LIG1). Expanded view gains a "Source" column tinted per-symbol's reference-set membership.
  • Airgap. Module ranking carries opaque IDs + scores + ref_sets set NAMES only — no gene symbols. Reference-set membership comes from the same bounded sealed-map lookup /diagnostic/full-rank already uses. Per-module symbols revealed lazily per expanded module via /reveal. New airgap tests in tests/test_api_airgap.py: test_modules_endpoint_returns_opaque_only_modules (assert ref_sets list present, opaque IDs only, sorted desc by combined AUROC), plus rejection tests for unsupervised / unknown / mid-run scenarios. 114 / 114 pytest, airgap suite 25 / 25, tsc --noEmit clean.
  • Verified live (HNSC HPV, seed 11, coherence ON). 303 distinct modules; the rank-1 module (size 11, combined held-out AUROC 0.978) carries the cell_cycle badge and beats the winner's 0.941. Expanding it resolves all 11 symbols on the wire: MCM2, CENPQ, FAM111B, KIF2C, NDC80, RAD1, USP1, REXO2, ING4, KIAA1407, MMP17 — a proliferation / DNA-replication module the single-gene picture missed. Exactly the "group view can out-score its individual genes" demonstration the design called for.
• Chunk 7 follow-up (done — separate group vs individual views in the coherence-run UI): Presentation-only cleanup so no single table mixes group and individual scores under the same column.
  • Single-gene "Highlighted genes" table retags winner-source rows from "winner" → "in winning program" and adds a one-line caption underneath: "These are single-gene ranks of individual genes — including the genes inside the winning program and the reference markers. The winning program's combined (group) score is in the Result cards above and the Coordinated modules below." The winner's combined held-out is unchanged in the Result cards.
  • InfoTip copy rewritten for both panels. TIPS.rankingResult
    • TIPS.rankingHighlighted lead with INDIVIDUAL; TIPS.module Ranking leads with GROUP and explicitly states that the modules' re-scoring (mean across the gene-set vs label on held-out) is a different number than the GP search used, so the #1 module can differ from — and even beat — the winning program.
  • Coordinated-modules table now tags the module whose gene-set is exactly equal to the winner's gene_ids (unordered set equality) with a winner badge in the existing accent-teal palette. That row sits at its scored rank, not at the top, so the user can see what rank the winner occupies under the group metric (e.g. seed 11 / HPV: winner program holdout 0.9411 vs the same gene-set as a module → 0.9153 → rank 41; #1 module 0.9778 — three different numbers, three honest meanings).
  • Paginated auto-reveal replaces the "Reveal genes" per-row click. MODULE_PAGE_SIZE = 25; symbols cached in a Record<string, string> keyed by opaque ID and ONE batched /reveal call per page (bounded by ≤ 25 × max_genes_per_set IDs — never the whole map). Page-1 of the HNSC seed 11 run pulls 99 unique IDs out of ~20k. New <ModulePager> (|‹ ‹ page n/N › ›|)
    • a row click (▶ / ▼) toggles the expanded per-gene-metrics view. Sort or fresh-run resets page + collapses any open row. Inline symbol list truncates at 8 with a · +N overflow chip; expanding shows them all. The expanded <ModuleGeneTable> reads from the shared symbolByOpaque cache (no second fetch).
  • No engine/API/airgap changes. tsc --noEmit clean; pytest -q 114 / 114; airgap suite untouched.
• Chunk 7 follow-up (done — confound pass: configurable Effect confounders + site / purity survival flags on module ranking): Lets us tell whether a gene/module separates HPV for a mechanistic reason or via a confounder (sex / race / anatomic site / immune composition). Airgap stays absolute — clinical confounders (sex / race / oropharynx flag) and the derived purity proxy are named non-gene variables in the data + validation layer; engine_v2 only ever sees opaque gene IDs.
  • HNSC build. data_pipeline/build_hnsc.py now writes race, ethnicity, tissue_site (raw TUMOR_TISSUE_SITE), icd_o_3_site (ICD-O-3 topography from the patient file), and a derived is_oropharynx boolean (SEER mapping: ICD-O-3 prefix C01/C09/C10 + exact C02.4/C05.1/C05.2). Prints non-null counts and the top ICD-O-3 distribution so the mapping is auditable. On the live cohort: 72 oropharynx / 415 not / 0 NaN among 487 called samples; HPV+ concentrates in the oropharynx (49/72 = 68%) vs HPV- (23/415 = 5.5%) — the confound is real and large.
  • dsl.Load widens its allowed clinical columns to include race, ethnicity, tissue_site, icd_o_3_site, is_oropharynx — filtered per cohort (coadread's parquet doesn't have these, so nothing changes there).
  • API _prepare_lab_data widens the HNSC clinical subset to (stage, age, sex, race, tissue_site, icd_o_3_site, is_oropharynx) when present (skips columns not in the frame). Coadread unchanged.
  • Effect is now configurable. ExecContext gained confounders: tuple[str, ...] = ("stage", "age") — default preserves byte-for-byte legacy behaviour (MSI / TMB / HPV runs unchanged). Effect.execute reads ctx.confounders and builds a one-hot-encoded design matrix from any columns present in ctx.clinical (age continuous, everything else categorical; missing values drop the row). Pipeline + worker plumb the confounder set in — HNSC v2 runs use ("stage","age","sex","race") automatically when those columns are in clinical. Smoking deliberately NOT included (not in this download). New tests: test_effect_default_confounders_are_stage_and_age
    • test_effect_extended_confounders_include_sex_and_race (asserts the extended path runs and unknown column names like "smoking" are silently skipped).
  • Module ranking: site-stratified survival flag. _compute_module_ranking no longer discards clinical. For HNSC/HPV runs it slices is_oropharynx to the held-out test patients, recomputes each module's mean-aggregate AUROC within the oropharynx subgroup, and emits combined_holdout_oropharynx
    • survives_site (True iff stratified AUROC ≥ full − 0.05). Modules whose "HPV signal" was really an oropharynx-tissue marker visibly collapse here.
  • Module ranking: purity-proxy survival flag. Immune- infiltration proxy = per-patient mean expression of CD8A / GZMB / PRF1 / CD3D / CD2 (HPV_IMMUNE_PROXY_GENES, curated once in api/app.py), resolved to opaque IDs via the sealed map — same bounded-reveal pattern /diagnostic/full-rank uses for reference_marks; never leaves the API layer. Bottom tertile of the proxy on TEST = high-purity subset; each module gets combined_holdout_highpurity + survives_purity. On the live HNSC cohort the high-purity TEST subset is ~49 patients but with only ~2 HPV+ (class imbalance — HPV+ tumours have more immune infiltrate), below the n_pos ≥ 5 guard, so the flag honestly reports "n too small" rather than a misleading AUROC.
  • Module endpoint payload gains subgroups: { site, purity } metadata (kind / n / tolerance / n_proxy_genes) at the top level so the UI can render the column label + an explanatory caption. Opaque-only on the wire — no gene NAMES enter the module payload.
  • Frontend. New <SurvivalChips> renders compact site ✓ 0.92 / site ✗ 0.84 / purity — chips on each module row (gated to whichever subgroups the server emitted; non-HNSC / non-HPV runs don't show the column at all). Expanding a module appends <StratifiedAUROCStrip> — full vs oropharynx vs high-purity AUROCs with n + per-class counts. New TIPS.module Survival ? explains the flags. TIPS.moduleRanking gained the verbatim winner's-curse caveat: "these groups are scored by re-evaluating ~2,900 explored sets on the same small held-out set and showing the best — so the very top values are optimistically biased (the luckiest of thousands). The winning program was chosen by cross-validation, which guards against that, so trust it as the engine's pick even when a table row scores higher."
  • Verification. pytest -q 116 / 116 (was 114; +2 Effect confounder tests). Airgap suite 25 / 25 green — module payload still opaque-only. tsc --noEmit clean. Live HNSC seed-11 coherence-on: 303 modules, 234 survive the oropharynx subgroup ✓, 69 collapse ✗, 0 n/a; purity all n/a (class imbalance, as expected). The winner module (the 3-gene Combine — full AUROC 0.915 in this module-metric view) survives the oropharynx test (subgroup AUROC 0.919) — it's genuine HPV signal that holds within oropharynx, not a tissue marker.
• Chunk 7 follow-up (done — Result views consolidation: GP's group vs known markers vs explored leaderboard): Presentation refactor so the three Lab Result panels map cleanly to three questions and no single table mixes a group entry with individual- gene rows. No engine / API / airgap change.
  • Survival chips show full → subgroup. <SurvivalChips> (was: site ✓ 0.95) now renders site ✓ 0.97 → 0.95 (−0.02) so the user sees both the pre and post AUROC plus the drop. Pre = module.combined_holdout; post = the stratified AUROC. The "n too small" path drops the arrow and renders site — (unchanged). Tooltip wording updated to spell out the full-cohort → subgroup pair.
  • Known-marker recovery panel = reference markers only. <RankingResult>'s pinnedRows no longer pushes the winner's kind:"winner" rows; only reference-set rows survive. SectionCard title becomes "Known-marker recovery", the heading flips from "Highlighted genes" to "Known markers", the lollipop chart auto-titles to "Where the known markers rank" (and drops the winner legend swatch) when winnerOpaqueIds=[]. The captions and TIPS.rankingResult / TIPS.rankingHighlighted rewritten to state this is a known-marker recovery diagnostic and point users to the Result panel for the winner's group view. The raw-anonymous list still tints rows that match the winner — that's per-row highlighting, not a pinned-source mix.
  • Result panel = self-contained "GP's group" story. <ResultPanel> now takes dataset / target / runId / coherence from the Lab parent and fetches getFullRankDiagnostic (per-gene single-gene ranks) + getRunModules (for the winner's survival chips) — same endpoints already used elsewhere; the server caches them. Reveals the winner's symbols via postReveal(winner.gene_ids) (bounded — only the winner). New <WinnerGroupBlock> rendered under the metric cards:
    • Its genes, each on its own: for each winning.gene_ids, looks up its rank in diag.ranks and shows SYMBOL #rank / N (e.g. RNF32 #123 / 20,218 · PCBD2 #79 / 20,218 · RPS10P7 #2344 / 20,218).
    • Confound survival: finds the winner's own module in moduleData via unordered gene-set equality (same notion <ModuleRankingPanel>::isWinnerSet uses), then renders <SurvivalChips> — automatically inheriting the Part-A pre→post pairing. Gated to runs that emit survival metadata (HNSC/HPV coherence-on); omitted gracefully otherwise.
    • Block InfoTip frames the three-panel layout: "This is the group the engine actually chose. Below: how each of its genes ranks on its own, and whether the group survives the confound checks. The known-marker recovery diagnostic and the explored- group leaderboard live in the next two panels — different questions."
  • Verification. tsc --noEmit clean; pytest -q 116/116 (no Python changes). Live HNSC seed-11 coherence-on: winner RNF32+PCBD2+RPS10P7 AUROC 0.941, the three per-gene ranks resolve (#123 / #79 / #2344), winner module survives the oropharynx subgroup (site ✓ 0.915 → 0.919) and reports purity n/a (class imbalance). The known-marker panel shows only MCM5 #22 / MCM2 #28 / MCM6 #38 / … with no winner dots/rows. The Coordinated-modules table still badges the winner's own set at its scored rank.
• Chunk 7 follow-up (done — Result panel: GP's top-10 programs + plain-language modules copy): Expanded the single-row "GP's group" block into the engine's top-10 programs ranked by its OWN fitness, and rewrote the Coordinated-modules copy in plain English. Presentation + frontend data-flow only.
  • TopProgramsBlock replaces <WinnerGroupBlock> in <ResultPanel>. <ResultPanel> now fetches getRunStatus(runId) → last generations_persisted - 1 → getRunPopulation(runId, lastGen), dedupes the candidates by unordered UNIQUE-gene-set (engine_v2 programs can carry repeated genes across multiple Selects), keeps the highest-fitness representative per set, sorts by fitness desc, and takes the top-10. Each row reuses the existing treatment: per-gene SYMBOL #rank / N (looked up in diag.ranks) and <SurvivalChips> for the program's matching module (gene-set equality the same notion ModuleRankingPanel::isWinnerSet uses). #1 is the winner and is highlighted with the accent-teal palette + a winner chip.
  • Bounded reveal. The reveal cache (symbolsByOpaque) is populated by a single batched postReveal over the UNION of all top-10 programs' gene_ids — never the whole map. Live HNSC seed- 11 coherence-on: 15 unique opaque IDs across the top-10.
  • Modules subtitle + InfoTip rewritten in plain English. The Coordinated-modules <SectionCard> subtitle is now: "Not the engine's picks. After the run, this re-scores every gene group the engine tried — using one simple number (the group's average expression, measured on held-out patients) instead of how the engine judged groups during the search. So this list can rank groups differently from 'The GP's top programs' above, and its #1 can even beat the engine's winner." TIPS.moduleRanking rewritten verbatim to the three-paragraph "What this is / How it differs / Caveat" structure from the prompt; the winner's- curse paragraph now reads as plain advice ("trust the engine's own picks above as the reliable choice") rather than statistical jargon. TIPS.rankingResult reworded to point at the GP's top programs and the after-the-fact re-scoring panel by name.
  • Verification. tsc --noEmit clean; pytest -q 116/116. Live HNSC seed-11 coherence-on: GP top-10 by fitness has the winner (RNF32+PCBD2+RPS10P7) at #1 with GP fitness 0.924; runner- ups are small 2-3-gene programs around the same core. Module ranking by combined held-out AUROC has an 11-gene cell-cycle- flavoured set at #1 (combined 0.978) — a different list, exactly the divergence the new copy makes explicit.
• Chunk 7 follow-up (done — cheap diversity knobs): Premature convergence was visible in the fitness curve (best = median by ~gen 10, then flat). Added three cheap diversity levers behind one "Maintain diversity" toggle. Defaults preserve behaviour byte-for- byte; the airgap is untouched (search-internal).
  • run_gp_v2 gains immigrant_fraction: float = 0.0. When > 0, round(immigrant_fraction * population_size) slots in the new generation are filled with fresh programs drawn from ramped_population (same rng, same grammar/objective/depth constraints as init), inserted AFTER the elites and BEFORE the crossover/mutation offspring — never displacing elites. Default 0.0 ⇒ no immigrants ⇒ behaviour unchanged. tournament_k and p_mutate were already parameters; raising mutation and lowering selection pressure are paired with immigrants via the same toggle.
  • Pipeline + worker plumbing. run_v2_pipeline and run_v2_pipeline_streaming accept the new immigrant_fraction: float = 0.0. RunRequest gains diversity: bool = False. Run dataclass gains a diversity field; surfaced in /runs/{id}. _worker maps diversity=True → tournament_k=2, p_mutate=0.85, immigrant_fraction=0.10; False → current (3, 0.7, 0.0). Two new tests in tests/test_engine_v2.py: test_default_run_unchanged_by_diversity_param (passing immigrant_fraction=0.0 explicitly produces an identical best- fitness trajectory to omitting the arg) and test_immigrant_fraction_injects_fresh_programs (with frac>0 the last generation contains ≥1 immigrant — empty-parents and non-survived — and more distinct program_repr strings than the baseline).
  • Frontend. New <TIPS.diversity> copy. ParamsControls gets a "Maintain diversity" checkbox under the coherence toggle; plain-English caption: "Default off. Lowers selection pressure and injects fresh random programs each generation so the population keeps exploring — watch the best-vs-median gap in the fitness curve stay open longer." Lab state + postRun(body.diversity) plumbed end-to-end.
  • Verification. pytest -q 118/118 (was 116; +2 diversity tests). tsc --noEmit clean. Airgap suite untouched. Live HNSC HPV seed-11 coherence-on, 20 generations, A/B:
    • OFF: gap collapses 0.242 → 0.002 by gen 19 (full convergence). Held-out 0.977, p 0.048.
    • ON: gap stays 0.13-0.18 the entire run (mean 0.149 vs OFF 0.041; last-gen 0.139 vs OFF 0.002). Held-out 0.948, p 0.048 — detection preserved while the population stays exploratory.
• Chunk 7 follow-up (done — merge "GP's top programs" + "Coordinated modules" into one sortable "Groups the engine explored" table): Replaced the two overlapping panels with a single table over the SAME groups (harvested from the persisted population), selectable by four sort lenses. One additive API field, presentation refactor; airgap unchanged.
  • API. _compute_module_ranking now records the max GP fitness per gene-set while harvesting candidates and emits gp_fitness: float | None on each module dict. Fitness is a number — no gene names cross the wire. RankedModule in web/lib/api.ts gains gp_fitness?: number | null.
  • Frontend — merged table. <ModuleRankingPanel> retitled "Groups the engine explored". SortKey extends to "gp_fitness" | "combined" | "coherence" | "synergy". Default sort = gp_fitness so the default view IS the engine's own preference order (replacing the deleted top-programs panel). Two new columns: GP fitness (per-row) and Synergy = combined_holdout − max(single_gene_metric over per_gene) — high Synergy = real additive lift; low/negative = "best gene + passengers" pattern. Sort buttons relabelled accordingly. Subtitle rewritten in plain English (verbatim per prompt); TIPS.moduleRanking rewritten to explain all four lenses + keep the winner's-curse caveat for the re-score lenses.
  • Frontend — drop TopProgramsBlock. <ResultPanel> no longer fetches getRunStatus / getRunPopulation / postReveal / getFullRankDiagnostic / getRunModules; the entire TopProgramsBlock function definition and its supporting state are gone. Result-panel signature simplifies to { result, dataset, target, runId, coherence } with dataset / runId only used to gate the new pointer copy under the metric cards: "The winning program is badged in Groups the engine explored below — sort by GP fitness to see the engine's full preference order, or by Combined AUROC / Coherence / Synergy to re-score the same groups by a different lens." Pointer is gated to target !== "none" && coherence (the same gate the merged table uses). Unused imports (Candidate, getRunStatus, getRunPopulation) removed.
  • Verification. pytest -q 118/118 (no Python tests changed — the new field is additive, the old payload tests still pass). tsc --noEmit clean. Airgap suite untouched. Live HNSC seed-11 coherence-on confirms the four lenses give visibly distinct orderings on the same module set: GP fitness #1 = winner (RNF32+PCBD2+RPS10P7, gp 0.924); Combined-AUROC #1 = 11-gene cell-cycle module (combined 0.978); Coherence #1 = a tightly co-expressed 2-gene set; Synergy #1 = a 3-gene group with +0.154 lift above its best single gene.
• Chunk 7 follow-up (done — "GP fitness × Synergy" landscape scatter): Added a canvas scatter above the merged Groups table so the user can SEE which quadrant each explored group falls into at a glance. Frontend-only; no engine / API / airgap change.
  • <FitnessSynergyScatter> (~340 LOC in web/app/Lab.tsx). x = m.gp_fitness, y = synergyOf(m) (lifted to a module-scope helper so the table + scatter share one definition). Categorisation via categoriseModule(m, winnerSet) → winner | p16 | immune | cell_cycle | MMR | other (single- marker sets win over broader sets; winner takes priority over membership so the ring is always on top).
  • Canvas for the dots, SVG overlay for the chrome. ~6k modules in larger runs would choke as SVG nodes; canvas paints faint grey background dots first, then amber/gold highlighted dots, then the winner as a teal-ringed filled centre on top so it never gets occluded. HiDPI: canvas.width = clientWidth × dpr
    • ctx.setTransform(dpr, 0, 0, dpr, 0, 0). Plot rect responsive via a ResizeObserver on the wrapper div.
  • Quadrant story. Vertical guide at the median GP fitness, horizontal guide at synergy = 0 (orange dashed). Four captions in the corners: "real teamwork (unexplored by the engine)" / "ideal — usually empty (engine + teamwork)" / "weak / junk" / "engine's lone-gene detectors". Axis titles: "GP fitness — what the engine preferred →" and "Synergy — teamwork beyond best gene ↑". Dataset-aware legend (HNSC shows p16 + cell_cycle; CRC shows immune + MMR).
  • Gated to target !== "none" + coherence-on (same gate as the survival flags / synergy column). Mounted between the survival-flag caption and the Groups table inside <ModuleRankingPanel> so it shares one SectionCard frame with the table.
  • Verification. tsc --noEmit clean; pytest -q 118/118 (no Python changes). Live HNSC seed-11 coherence-on: 303 plottable modules; winner @ gp=0.924 / syn=0.032; 21 dots in the bottom-right "engine's lone-gene detectors" quadrant; 20 in the top-left "real teamwork" quadrant the engine didn't prefer; 3 cell_cycle-flagged dots cluster at low GP (~0.58, the engine barely scored them) with slightly-negative synergy — exactly the "best gene + passengers" pattern the scatter is meant to expose. Airgap untouched (synergy + category derived from fields already on the wire; no extra reveal).
• Chunk 7 follow-up (done — hover tooltips on the scatter, highlighted dots only): Added hit-tested tooltips to the fitness × synergy scatter so the user can read the gene-set and scores of any highlighted dot without scrolling the table. Frontend-only; airgap intact.
  • Hit-test scope. When painting, <FitnessSynergyScatter> now records the screen position of every HIGHLIGHTED dot (winner + cell_cycle + MMR + p16 + immune) into a hitsRef.current: Hit[] array. The faint grey background dots are deliberately NOT in this array — they are not revealed and are not hit-tested. Selection: mousemove walks the array and picks the nearest hit within ~8px (radius² = 64); mouseleave clears.
  • Bounded reveal. On mount / when modules changes, the scatter takes the UNION of all highlighted modules' gene_ids and issues ONE batched postReveal call — never the whole map. On live HNSC seed-11 coherence-on: 31 unique opaque IDs revealed out of ~20k (winner 3 + 21 cell_cycle-tagged modules, plus a handful of overlapping genes). Cached in highlightSymbols: Record<string, string> so re-renders are free.
  • <ScatterTooltip> absolutely-positioned over the chart container. Renders: source chip (winner / p16 / cell_cycle / immune / MMR — coloured to match the dot), n genes, the revealed gene symbols (falls back to (revealing…) while the bounded call is in flight), and a 2×4 grid of GP fitness, Combined AUROC, Coherence, Synergy via fmtFit (non-finite → em-dash). Synergy uses the shared synergyOf helper so the tooltip, the table column, and the y-axis stay consistent. Position clamps to the chart rect so dots near the edge don't push the box off-screen.
  • Verification. tsc --noEmit clean; pytest -q 118/118 (no Python changes). Live HNSC seed-11 coherence-on: hovering the winner ring shows RNF32, PCBD2, RPS10P7 · GP 0.924 · Combined 0.915 · Coherence 0.474 · Synergy 0.032; hovering a cell_cycle dot shows e.g. CDC6, … with its scores. Background dots are not hover-targets, no full-map reveal.
• Chunk 7 follow-up (done — encode site survival on the scatter + plain-language Survives explainer): Two presentation fixes around the confound flags. Frontend-only; the data (survives_site / survives_purity / subgroup AUROCs) is already on the wire.
  • Scatter site-survival encoding. Highlighted dots in <FitnessSynergyScatter> are now drawn:
    • solid (fill + stroke) when survives_site === true (or null / "n too small" — keeps the simple default visible),
    • hollow (white fill + coloured stroke) when survives_site === false — so failed-site groups read as "less trustworthy" at a glance. Only site is encoded; purity is mostly null on this cohort and carries no signal. The category colour (winner teal / cell_cycle amber / p16 gold) is preserved. The winner ring uses the same encoding when it fails the check.
  • Hide-failed toggle. Added a small "Hide groups that fail the site check" checkbox in the scatter header (default OFF). On: survives_site === false highlighted dots are dropped from both the canvas paint AND the hit-test array (no accidental hover tooltips on hidden dots). Background grey dots are unaffected (not site-tested).
  • Caption + legend updated. The italic caption gains "Solid dots survive the site check; hollow dots fail it (their signal is partly location)." The legend gains a "hollow = fails site check" entry.
  • Plain-language "Survives" column ?. TIPS.moduleSurvival rewritten verbatim per the prompt:

    What "Survives" checks: whether a group still separates HPV when you take away a possible confounder — something that travels with HPV but isn't HPV biology.

    Site / Purity each get a one-paragraph plain-English explanation; ✓ / ✗ / — get a one-line legend each. The tip is now anchored on the Survives column header (the <th>) where it's most discoverable. Per-chip pre→post tooltips on the rows are unchanged.

  • Verification. tsc --noEmit clean; pytest -q 118/118 (no Python changes). Live HNSC seed-11 coherence-on shows 234 of 303 modules survive site / 69 fail; the 4 highlighted dots (winner + 3 cell_cycle) all survive on this seed, so the encoding renders solid — the failed-dot path is exercised in the general case where a tagged module collapses.
• Chunk 7 follow-up (done — five small UI fixes: default cohort, remove peel-off, fix two clipped widgets, swap biology figure): Presentation-only sweep over Lab; frontend-only. No engine / API / airgap change.
  1. Default to Head & Neck + HPV detection. dataset state in Lab now defaults to "hnsc", and target to DATASET_REGISTRY.hnsc.objectives[0].key so first load lands on the HPV demo of the workflow. Colorectal stays selectable; only the default changed.
  2. Hide the Discovered axes panel. <DiscoveredAxes> is now gated to target === "none" (never true in the UI — the unsupervised objective isn't a UI objective anymore). On supervised runs the panel was re-finding the same signal and showing nonsensical "Aligns with MSI / TMB" text. The component definition + backend chain are untouched in case a developer flag wants to surface it again.
  3. Fix the "(TCGA COADREAD" subtitle clipping. Not a CSS issue — the old regex /^[(]|[)]$/g.replace was stripping the trailing ). Replaced with entry.longLabel.replace(new RegExp('^${entry.label}\\s*'), '').trim() so the parenthesised tail renders intact, e.g. (TCGA HNSC).
  4. "Survives" ? is a click-to-open modal now. Multi-paragraph copy didn't fit a hover tooltip — clipped on the right. paramHelpContent.tsx::ParamKey extended with "module_survival" and a new PARAM_HELP.module_survival entry carrying the prompt's verbatim site / purity / ✓ / ✗ / — explainer. The <th>Survives</th> (and the table-header caption above the table) now uses <ParamHelp paramKey="module_survival"> instead of <InfoTip>. Per-chip full → subgroup row tooltips are unchanged.
  5. HPVBiologyPanel — new card-wall figures. HPVBiologyPanel.tsx now renders TWO inlined SVGs: <NormalCellDiagram> (the new repo-root normal_cell_reference.svg, viewBox 920×900: RB1 / p16 / RB-E2F cycle / E2F / genes E2F turns on / protein production / p53) followed by <HPVDiagram> (the rewritten repo-root hpv_reference.svg, viewBox 920×660: HOW HPV TAKES OVER → HOW E7 ATTACKS RB · HOW E6 ATTACKS p53 → p16 BECOMES THE HPV+ MARKER · WHY HPV DOESN'T MUTATE THE GENES). The normal-cell baseline renders first so the reader sees the brake before they see it dismantled. SVG attributes hand-translated to JSX exactly as the existing component did (font-family → fontFamily, font-weight → fontWeight={n}, text-anchor → textAnchor, marker-end → markerEnd, etc.); each <svg> keeps viewBox + a width:100% / height:auto wrapper so it scales without overflow.
  • Verification. tsc --noEmit clean; pytest -q 118/118 (no Python changes). Lab first-load lands on HNSC + HPV; biology panel shows the new normal-cell + HPV figures (both fully visible, no overflow); the Discovered-axes panel is gone for HPV / MSI / TMB runs; cancer-card subtitles read "(TCGA HNSC)" with the closing paren; the Survives "?" now opens a click- modal with the four-paragraph site / purity explainer that never clips.
• Chunk 7 follow-up (done — raised caps, "GP" → "genetic programming", DSL operator-usage): Three changes across the Lab + API surface.
  • Generations / Population caps raised. Lab's NumField config bumps Generations max 100 → 1000 and Population max 500 → 3000; server-side RunParamsModel.generations / population Field bounds raised to match. Bounded — not removed — so an accidental huge value still can't hang the backend.
  • "GP" spelled out everywhere user-facing. Every visible occurrence in Lab.tsx replaced: "GP knobs" → "Genetic- programming knobs"; "Starts a GP run" → "Starts a genetic- programming run"; the compact label "GP fitness" → "Genetic- programming fitness" at the sort button, table header, scatter heading + axis title + median guide + tooltip dt
    • module-ranking subtitle, and TIPS prose. The internal data key gp_fitness is UNCHANGED (API contract). Code comments still say "GP" by choice — they're not user-facing.
  • DSL operator-usage endpoint + tiles.
    • New GET /runs/{run_id}/operator-usage walks every candidate in every persisted generation, counts each operator token (Select(, Reduce(, Combine(, Split(, Associate(, Effect(, FitApply(, Search() in the program_repr strings, and emits per-operator {name, total_uses, programs_using} + run totals (n_generations, n_candidates). Cached per run. 425 while running, 404 unknown, 500 on error. Opaque-safe by construction — operator keywords + integer counts only, no gene IDs / symbols ever in the payload. Three new airgap tests (test_operator_usage_endpoint_returns_opaque_counts, _425_while_running, _404_for_unknown_run).
    • Frontend: getRunOperatorUsage(runId) fires from the SSE done handler; Lab caches the result in operatorUsage state. New <DSLVocabularyTiles> driver swaps each <DSLVerbCard> for an enhanced version that adds a used N× — in P% of programs line and a tile-relative bar (height 1, accent fill, white-grey track) under the description. Before / during a run, tiles render just the hints (no counts). A caption appears beneath the grid after a completed run.
  • Verification. pytest -q 121/121 (+3 new airgap tests). tsc --noEmit clean. Live HNSC seed-11 coherence-on, 10×80: Reduce in 100% of programs (grammar floor), Select 90.6% (1098 uses), Combine 21.2% (216), Split 4.1%, Associate 2.4%, Effect 1.0%, Fit/Apply 3.6%, Search 0 (off by default). Airgap-clean payload — keywords + counts only.
• Chunk 7 follow-up (done — turn Search on by default + add a per-run gate): Search was the one DSL operator still gated off (DEFAULT_RATES["search"] = 0.0). Turn it on at a modest rate so the engine can use the full DSL, and add a UI toggle so it can be flipped off when speed matters (every Search node runs an inner gene ranking).
  • Engine. engine_v2/synthesize.DEFAULT_RATES["search"] flips 0.0 → 0.05. Caps stay (SEARCH_MAX_K, ≤ 200 candidate columns) so each Search's cost is bounded. The existing is_unsup gate at the injection site still forbids Search on unsupervised runs.
  • run_gp_v2 gains rates_override: dict | None = None. Inside run_gp_v2 the override is merged on top of the objective's synthesis-overrides (UNSUP etc.) — caller's keys win — and the merged dict is threaded into every ramped_population / mutate call. Both pipeline entry points (run_v2_pipeline, run_v2_pipeline_streaming) expose rates_override and forward it through.
  • API. RunRequest gains enable_search: bool = True. Run dataclass tracks it; /runs/{id} surfaces it. _worker builds rates_override = None when on, or {**DEFAULT_RATES, "search": 0.0} when off, and passes it into the pipeline. The override is a complete dict (overlaid on DEFAULT_RATES) so the inner synthesize._grow_* code can index by key without a fallback.
  • Frontend. New <TIPS.enableSearch> copy. ParamsControls gains an "Enable Search operator" checkbox under the diversity toggle (default ON). Lab state + postRun.body .enable_search plumbed end-to-end. DSL-vocabulary tile for Search now reads "Nested feature search — runs a small gene-ranking inside the program (bounded: ≤ 4 genes from ≤ 200 columns)." — the "off by default" wording is gone.
  • Tests. tests/test_api_airgap.py::test_enable_search_false_pins_search_to_zero verifies the toggle: with enable_search=False the operator- usage payload reports Search at 0 total uses / 0 programs, and /runs/{id} surfaces the flag. The existing test_operator_usage_endpoint_returns_opaque_counts relaxed its old "Search must be 0" assertion (it's now >= 0 — Search may or may not fire on a tiny synthetic 3×12 run).
  • Verification. pytest -q 122/122 (+1). tsc --noEmit clean. Live HNSC seed-11 coherence-on, 10 × 80:
    • Search ON (default): Search used 20× across 20 / 800 programs (~2.5%); winner AUROC 0.909, p 0.048.
    • Search OFF: Search used 0× / 0 programs; winner AUROC 0.941, p 0.048.
• Chunk 7 follow-up (done — editable DSL rates in the vocabulary panel + drop the Search checkbox + reorder DSL before Run): Make every optional DSL operator's injection rate editable in the DSL vocabulary section (pre-filled with engine defaults), fold the Search checkbox into a Search-rate input (0 = off), and move the panel above the Run section. Airgap untouched.
  • API. RunRequest.enable_search is gone; replaced by a flat RunRequest.rates_override: dict[str, float] | None. Keys we accept: split, effect, fitapply, search, scalar_share. Missing keys keep their engine defaults — an empty / missing override reproduces current behaviour byte-for-byte. Run dataclass stores the dict; /runs/{id} surfaces it. _worker splits the flat dict back into rates_override (overlay on engine_v2.synthesize.DEFAULT_RATES) and scalar_share_override, then forwards both to the pipeline.
  • Engine. run_gp_v2 gains scalar_share_override: float | None = None; merged with the objective's overrides via an effective_scalar_share that wins over overrides.get("scalar_share"). Both pipeline entry points expose the new param and forward it.
  • Frontend. New DSL_DEFAULT_RATES constant mirrors the engine defaults (Split 0.10 / Effect 0.40 / Fit/Apply 0.10 / Search 0.05 / scalar_share 0.20). <DSLVocabularyTiles> rewritten: structural operators (Select / Reduce / Combine / Associate) render with an "always available" tag; optional operators (Split / Effect / Fit/Apply / Search) render with a small <input type=number min=0 max=1 step=0.05> pre-filled with their default. A separate scalar_share field sits in a footer row alongside two buttons: Reset to defaults (off when already at defaults) and Set all equal (one shared value for the four optional operators — the neutral-prior experiment). The <TIPS.dslRates> ? explains the inputs.
  • Wire shape. The Lab only sends keys the user actually changed (ratesDiff), so the default workflow keeps producing an empty / absent rates_override field and behaviour is byte-for-byte unchanged.
  • Layout. The DSL vocabulary <SectionCard> moved ABOVE <RunBar> / <LiveView> so the (now interactive) config sits before the Run button. Order is now Cancer → Objective → Biology → Parameters → DSL vocabulary → Run → Live view.
  • ParamsControls. The "Enable Search operator" checkbox and enableSearch state/prop are gone; that toggle lives as a rate input on the Search tile (rate 0 = off).
  • Tests. tests/test_api_airgap.py::test_enable_search_false… renamed to …rates_override_search_zero… and rewritten to POST {"rates_override":{"search":0.0}}; asserts the operator-usage payload reports Search at 0 total / 0 programs AND /runs/{id}.rates_override == {"search": 0.0}. The OperatorUsage opaque-counts test stays the same. 122 / 122 pytest pass; airgap suite green.
  • Verification. tsc --noEmit clean. Live HNSC seed-11 coherence-on, 10 × 80:
    • Default (no override): Search 20×, Split 25, Effect 6, Fit/Apply 27 (the current DEFAULT_RATES mix).
    • rates_override={"search":0.2}: Search 40× (double the default rate ⇒ roughly double the uses); Split, Combine etc. shift because raising Search's slot draws from the same Matrix-slot RNG mix.
    • rates_override={"search":0.0}: Search 0× exactly, matches the old enable_search: false behaviour.
    • /runs/{id} surfaces rates_override: {"search": 0.2} for the UI.
• Chunk 7 follow-up (done — DSL panel polish: scalar-share copy, rate explainer, collision-free lollipop labels): Three small presentation fixes in web/app/Lab.tsx. No engine / API / airgap change; tsc clean.
  • Scalar share copy rewritten in plain English. The native title= jargon on the scalar-share row is gone. Inline description: "how many starting programs are built around Associate / Effect (which score a gene set by how well it correlates with the target) instead of giving each patient a score. 0.20 ≈ 1 in 5; 0 = never." New per-field <InfoTip>: "Every program ends in one of two outputs: a score for each patient (the usual case), or a single correlation number produced only by the Associate or Effect operators. Scalar share is the fraction of starting programs built around that second kind…" ("(default 0.20)" hint, the input, and the Reset / Set-all-equal buttons unchanged.)
  • Rate explainer added to the DSL panel. TIPS.dslVocabulary (the panel ? next to the section title) replaced with the verbatim "Rate = how often the engine reaches for an operator …" copy — covers the Effect/Associate carve-out and the "not a measure of quality" caveat. Each tile's "rate" label gains an <InfoTip> with the short version (per-operator default interpolated), so the meaning of the input is discoverable without leaving the tile.
  • Lollipop chart: collision-free label placement. Previous fixed 4-row stagger garbled tight clusters (MCM2..7 / AURKB / CCNB1 rendered as overlapping strings). Replaced with a greedy placement: sort by x, estimate each label's pixel width (≈ text.length × fontSize × 0.6), drop into the lowest row whose previous label's right edge + 6px ≤ this label's left edge; open a new row if none fits. No cap on rows. The chart H is now derived from numRows so the SVG grows to fit and nothing clips. Per-label leader lines reach down to the dot from however many rows above sit the label. The MCM cluster now reads cleanly.
• Chunk 7 follow-up (done — show each group's best program tree on row expand): Each row's Genetic-programming fitness is the max fitness among evolved programs with that gene-set; now the argmax program's tree is carried through and rendered on expand. Airgap untouched.
  • API. api/app.py::_compute_module_ranking walks the population once and now tracks BOTH the per-set max gp_fitness AND the corresponding program_repr in a sibling dict (best_program_repr_by_key). Each module payload emits best_program_repr: str | None. Opaque-safe — program_repr is built from opaque IDs only.
  • Types. RankedModule.best_program_repr?: string | null in web/lib/api.ts.
  • Frontend. New <ModuleBestProgram> rendered inside the Groups-table expanded row, just under <ModuleGeneTable>. Caption: "This group's best program — The actual tree of the candidate that earned the Genetic-programming fitness above (argmax over the persisted population for this gene-set)." Uses the SHARED <ProgramGraph> (the same component the Result panel uses for the winner), with the row's already- revealed symbolByOpaque adapted to ProgramGraph's Record<id, {symbol, matched}> shape (matched=false — the known-marker highlight is a separate concern from per-group trees). The raw program_repr + a <CopyButton> sit underneath, so the string can be read or copied. Output label switches by target (HPV+ / MSI-H probability, TMB association, or score).
  • Verification. tsc --noEmit clean; pytest -q 122/122; airgap suite green. Live HNSC seed-11 coherence-on: 297/297 modules carry best_program_repr. The winner's row's best_program_repr matches the Result-panel winning .program_repr exactly (Reduce(Select(M,[g12850,g18272]), mean) on this seed). Airgap scan of the modules payload finds no gene-symbol leak.
• Chunk 7 follow-up (done — HNSC copy rewrite for a non-specialist reader): Copy-only pass across the Head & Neck / HPV Lab so a co-founder can scroll top-to-bottom and understand each section without outside help. No engine / API / airgap / logic change.
  • Voice spec: plain, gloss jargon on first use (AUROC = 0.5 coin-flip / 1.0 perfect; held-out = patients the engine never trained on; permutation p = how often random noise matches this); cut implementation plumbing ("Server-Sent Events", "on the backend", "opaque IDs", "airgapped", "residualise") while KEEPING the credibility ideas (blind discovery, held-out honesty, beats-chance, reveal-only-at-end) in plain words.
  • Section-by-section rewrites (in web/app/Lab.tsx + the biology panel + paramHelpContent.tsx): header subtitle (engine hunts blind, gene names hidden as codes); Cancer/problem subtitle; Objective subtitle; obj_hpv modal rewritten to the verbatim HPV-detection intro + AUROC gloss + "detecting a known viral fingerprint" honest note; HPV-biology caption; Parameters subtitle + every PARAM_TIPS[*]; DSL vocabulary subtitle + every tile hint (Select = "Pick specific genes.", Reduce = "Combine those genes into one score per patient (average, max, etc.).", Combine = "Merge two scores into one.", …); Run subtitle; Live view subtitle + TIPS.bestVsMedian verbatim per prompt; y-axis label "separation (AUROC — 0.5 coin-flip, 1.0 perfect)"; Population subtitle; Program graph subtitle + reveal-state phrasing; Result subtitle + HPV-specific held-out / permutation / nodes / genes tooltips; Known-marker recovery subtitle + chart title; Groups the engine explored subtitle; scatter heading/caption; operator-usage caption; Reveal & evaluate subtitle. <th>Opaque ID</th> → <th>Gene code</th> across the three reveal tables; raw-ranking disclosure copy; live-view chip label; TIPS.opaqueIds polished.
  • Jargon sweep clean. No Server-Sent Events, SSE, on the backend, worker thread, or airgapped in visible copy.
  • tsc --noEmit clean; no behaviour change.
• Chunk 7 follow-up (done — external-cohort transfer test: GSE65858 end-to-end): Add a second, INDEPENDENT HPV validation cohort as a reveal-side transfer test — score the HNSC/HPV winner, discovered blind on TCGA, on ~270 GEO head & neck tumours from a different country, hospital, and measuring machine. Purely additive — the engine still discovers blind on TCGA; GSE65858 lives on the NAMED (reveal) side of the airgap and only ever sees the winner's already-revealed symbols.
  • Data pipeline. data_pipeline/schema.py gained a GSE65858 block (series-matrix URL, GPL10558 platform URL, raw/processed dirs, HPV label constants). data_pipeline/download_gse65858.py mirrors download_hnsc.py (fetch series matrix + platform annot from NCBI FTP with loud manual-fallback instructions; live-download works — 21 MB series matrix + 7 MB platform). data_pipeline/build_gse65858.py parses the GEO series-matrix format, derives the STRICT virus-active HPV+ (DNA+RNA+) label from the joint hpv16_dna_rna characteristic with a fallback ladder (joint → separate DNA/RNA → single status field → single- side calls), maps probes → HUGO symbols via GPL10558, collapses to a symbol × sample matrix (mean over probes), writes data/processed_gse65858/{clinical,expression}.parquet, and prints a full provenance report. Live build: 250 called (35 HPV+ / 215 HPV−) across 16,951 symbols — matches the paper's strict virus-active rate.
  • Named-side transfer function (validate/transfer_gse65858.py) — a pure, testable transfer_score(symbols, *, n_permutations, seed, processed_dir) returning {auroc, p, n, n_pos, n_neg, n_found, n_missing, found_symbols, missing_symbols}. Cross- platform fix: z-score each found gene within GSE65858 before the per-patient mean (TCGA RNA-seq scale vs Illumina array intensity). Orientation-agnostic AUROC + permutation-null p (with +1/+1 smoothing). Finite-guarded; graceful null payload when n_found == 0. 5 new tests in tests/test_transfer_gse65858.py (signal → high AUROC / small p; noise → chance; missing symbols reported; no-found graceful; payload carries only supplied symbols).
  • API endpoint — GET /runs/{run_id}/transfer. Gated to HNSC + HPV runs only (400 otherwise); 425 mid-run, 404 unknown, 503 if the cohort parquets aren't built. Bounded reveal of the winner's opaque IDs via airgap.reveal (same discipline /evaluate uses). Lazy-imports validate.transfer_gse65858. Cached per-run. Payload: {cohort, platform, source, n_cohort, auroc, p, n, n_pos, n_neg, n_found, n_missing, found_symbols, missing_symbols}; the only gene NAMES are the winner's own revealed symbols. 3 new airgap tests (non-HPV reject; payload carries only winner symbols with whole-word regex — so AARS inside AARSD1 doesn't false-positive; 425 mid-run).
  • Frontend. TransferResult type + getRunTransfer(runId) fetcher. New <ExternalValidation> component at the end of <ResultPanel>, gated to dataset === "hnsc" && target === "hpv" && runId. Fetches on mount. Renders: section header + ? + subtitle; green/amber verdict callout ("It holds on strangers." vs "Not confirmed on strangers.") gated on auroc ≥ 0.75 && p < 0.05; three metric cards (Independent AUROC · Permutation p · Genes measurable n_found / (n_found+n_missing)) each with its own plain ?; <Metric> gained a sub prop for the subtitle line under the value; amber cross-platform pill (RNA-seq → microarray); three-step "How this validation works" strip (reveal → match & level → score) in light teal; teal 🔒 airgap footnote; one-liner honest note. Graceful fallback when the cohort isn't built (503 → build-hint instead of crash).
  • Verification. pytest -q 130/130 (+8 new tests; +5 transfer
    • 3 airgap). Airgap suite green. tsc --noEmit clean. Live end-to-end: HNSC HPV seed-11 winner [g12850, g18272] → symbols PCBD2, TMEM71 → transfer on GSE65858 AUROC 0.799, p 0.001 across 250 patients (35 HPV+ / 215 HPV−). Payload contains ONLY the two winner symbols. Textbook markers on GSE65858: CDKN2A alone → AUROC 0.896, p 0.002; MCM cluster → 0.85; TP53 → 0.74.
• Chunk 7 follow-up (done — scripts/multiseed_hpv.py multi-seed stability orchestrator): Read-only biology-aware script that runs the HNSC/HPV engine_v2 pipeline across a default seed grid [1, 3, 7, 11, 13, 17, 23, 29] and reports per-seed held-out AUROC + permutation p + winner gene_ids, transfer AUROC + p + n_found/n_missing on GSE65858, held-out and transfer AUROC ranges across seeds, and a gene-recurrence tally (each symbol tagged by its HNSC reference set — p16 / cell_cycle — or -). Reveal is bounded per seed to that seed's own winner's opaque IDs via airgap.reveal (scripts/ is allowed to be biology-aware; same discipline /evaluate uses); the sealed map is never dumped; GSE65858's gene list never crosses back into the engine. CLI: python -m scripts.multiseed_hpv --seeds 1 3 7 --generations 30 --population 300.
• Chunk 7 follow-up (done — "Stability across seeds" panel in the Lab): Frontend-only multi-seed stability panel, gated to HNSC/HPV, appended after <ExternalValidation> in <ResultPanel>. Reuses the existing run, transfer, reveal, and full-rank endpoints — no engine / API / airgap change.
  • State. Own stabilityRows / stabilityStatus / stability Progress / stabilityCancel — the main single-run Result / Live view stays untouched during a sweep. A dedicated awaitRunDone(runId) helper opens an EventSource and resolves on the done event without calling setResult / setGenerations / setStatus.
  • Body composition. Per-seed body mirrors launchRun() exactly: postRun({ objective_spec: OBJECTIVE_PRESETS.hpv, params: { ...params, seed }, engine: "v2", dataset, coherence, diversity, rates_override: ratesDiff }). Only DSL-rate keys that differ from defaults travel on the wire so a default sweep is byte-for-byte identical to the current run.
  • Sequential (backend runs one at a time), with a cancelRef between seeds so a "Stop after this seed" button halts cleanly. Progress readout "Running seed k of N…" during the sweep.
  • Rendering — three sections + summary callout:
    1. <StabilitySummaryCallout> — deterministic, LLM-free, composed from the aggregates in JS. Rendered above the dot- strips only when stabilityStatus === "done". Green if perfStable && geneStable (held-out min ≥ 0.75, every GP p < 0.05, and top-recurring gene in ≥ ⌈N/2⌉ seeds); teal / informative if perfStable && !geneStable; amber otherwise. Copy verbatim per prompt: Line 1 performance ("Across N independent searches, the engine detected HPV every time…" with an "every run beat chance" or "most runs beat chance on the independent cohort" tail); Line 2 gene story ("The same genes kept coming back — TOP in K/N runs…" vs "the specific genes differed almost every run — the most repeated was TOP (K/N), and M genes appeared only once…"); Line 3 honest close only when !geneStable ("So trust the detection, but don't read any single run's gene list as THE gene list…"). Extra guard: when no gene appears in more than one seed (topGene == null), the copy switches to "no single gene showed up in more than one seed's winner…" rather than saying "the most repeated was null".
    2. "Does the result hold across seeds?" — two horizontal dot-strips on a 0.5 → 1.0 axis (0.5 coin-flip / 0.75 / 1.00 tick labels). Accent-teal dots for Held-out AUROC (from winning.holdout_score — same field the Result card shows, so the stability dots match the numbers on screen), amber dots for Independent AUROC. Light min-max band per strip; header carries min–max via fmtFit.
    3. "Do the same genes keep coming back?" — recurrence bars: symbol · width-proportional bar · K / N · reference tag. Recurring (≥ 2 seeds) sorted desc; genes in only 1 seed collapse into a single "N genes · 1 seed each" row with the first 40 passenger symbols in the title tooltip. Colour: cell_cycle amber, p16 gold, alternate teal, passengers grey.
    4. Per-seed table: seed | held-out AUROC | p | indep. AUROC | indep. p | genes found / total. Every number through fmtFit; error seeds show error with the message on hover.
  • Reveal discipline — per seed, postReveal(winning.gene_ids) (bounded) sources the winner's symbols; then one getFullRankDiagnostic(dataset, target) at the end for the reference-set tag map (p16 wins over cell_cycle when a symbol is in both). No new reveal surface.
  • tsc --noEmit clean. pytest -q unchanged (frontend-only). Sanity-tested the data path via TestClient: 3 seeds × tiny budget produces 3 completed runs, each with holdout_score, permutation_p, revealed symbols, and a live /transfer payload; the recurrence tally rolls up correctly.
• Chunk 7 follow-up (done — scripts/capability_ratio_test.py known-answer capability test): Self-contained blind capability check. Plants a synthetic binary target defined by the balance between two REAL, positively-correlated genes (a log-ratio direction) — rigged so NEITHER gene helps on its own — and asks the blind engine to rediscover the interaction. Isolated: ONE new file under scripts/ (allowed to be biology-aware); READS the existing processed matrix + REUSES the sealed map via airgap.anonymise / airgap.reveal for the final check; writes nothing to disk; adds no dataset, no API route, no UI, no engine change. Deleting the file leaves zero trace.
  • Pair search. Smart seed-and-rank strategy (random-pair sampling is too slow at r ≥ 0.85 on 20k genes): pick a random seed gene from a variance/expression-filtered pool, rank the rest by descending Pearson r against it, check the top candidates. Constraints: r ∈ [0.85, 0.99], single-gene AUROC in 0.5 ± 0.07, ratio AUROC ≥ 0.9. Configurable via CLI (--pair, --min-r, --single-band, --ratio-min, --pair-seed).
  • Planted target. zA = zscore(A), zB = zscore(B); signal = zA − zB; y = (signal > median(signal)).
  • Setup + panel controls (auditable). Print single-gene AUROCs (0.5), ratio AUROC (1.0), best single-gene AUROC over the panel (0.6), and panel-mean AUROC (0.57) BEFORE running — so any high held-out score MUST come from composition.
  • Blind runs. anonymise(panel_expr) reuses the existing sealed map (no re-seal). Per-seed run_v2_pipeline with V2Objective(target="msi", binary=True) as the binary-AUROC carrier, prefilter_n=None, scalar_share_override=0.0 (force Vector programs), coherence_weight=0.0, diversity on (tournament_k=2, p_mutate=0.85, immigrant_fraction=0.10), population=200, generations=40, permutations=100.
  • Grader. Per seed: reveal(winning.gene_ids) (bounded). PASS if revealed symbols ⊇ {A, B} AND holdout_score ≥ 0.80 AND the program_repr contains a Combine( with one of protected_div / mul / sub. Prints program_repr, revealed symbols, held-out AUROC, per-check verdicts, and the overall "recovered in k / n seeds" line. PASS interpretation copy fires at k ≥ ⌈n/2⌉.
  • Live evidence (seeds 1, 3, budget 200 × 40): auto-picked pair FPR3 / C3AR1 (r 0.923). Seed 1 winner: Combine(Reduce(Select(M,[…,g06688,…]),median), Reduce(Select(M,[g11569,…]),median), protected_div) — the planted ratio, held-out AUROC 0.858 vs the 0.60 best-single control. PASS. Seed 3 finds FPR3 without C3AR1 (0.62, FAIL). Overall 1/2 → PASS: "The engine can discover a genuine two-gene interaction blind — so when a real target (HPV) yields only averages, that's because the biology doesn't need a ratio, not because the engine can't build one." pytest -q still 130/130 (nothing outside scripts/ was touched).
• Chunk 7 follow-up (done — validate/tmb_resid_rank.py "leftover- TMB" diagnostic): Cheap read-only diagnostic checking whether MSI-residualized TMB is a candidate COMBINATORIAL target on colorectal. Mirrors validate/tmb_rank.py exactly (same cohort loader shape, same NAMED matrix, reuses _spearman_per_column verbatim); lives in validate/ (structural airgap test scans only engine/); sub-second on the live cohort. No engine / API / UI / data change.
  • Pipeline. Load("processed") filtered to tmb.notna() & msi_status ∈ {MSI-H, MSS} & no-NaN expression rows. t = log1p(TMB), then WITHIN each MSI group separately z-score (t − group_mean) / group_std (guard std > 0) — pooled into one residual vector aligned to X.index. Two _spearman_per _column passes (raw baseline + residual headline), both ranked by |corr| descending. Reports: best_raw vs best_resid (drop tells you MSI's slice was removed), top-N by |Spearman(gene, resid)|, MMR + IMMUNE gene positions against the residual, per-group stats (n, TMB mean, log1p mean/std) so the residualization is auditable.
  • Verdict. Thresholds on best_resid: < 0.30 → CANDIDATE COMBINATORIAL TARGET, ≥ 0.40 → NOT COMBINATORIAL, otherwise BORDERLINE.
  • Live output on the colorectal cohort: N = 355 samples, 20,056 genes; MSI-H n=50 (TMB̄ 49.8) · MSS n=305 (TMB̄ 7.6). Best |corr| vs raw TMB = 0.447 (CXXC1) — MSI-driven, as expected. Best |corr| vs residualized TMB = 0.346 (FOXD4L1) — drop −0.10. MMR genes now WEAK against the residual: MLH1 rank 2543/20056, MSH2 at 17313, PMS2 at 15473 (sanity check that MSI's effect was removed). PRF1 the strongest immune gene at rank 161 (|corr| = 0.19). Verdict: BORDERLINE — "A weak single-gene signal remains; a GP run might still be informative."
• Chunk 7 follow-up (done — scripts/tmb_resid_gp.py synergy check on MSI-residualized TMB): Second isolated script that answers whether the DSL finds gene COMBINATIONS beating the best single gene on the "leftover TMB" target the diagnostic borderlined. ONE new file under scripts/ (biology-aware) + a small helper refactor in validate/tmb_resid_rank.py to keep the residual byte-identical between the diagnostic and the GP script.
  • validate.tmb_resid_rank.build_residual_cohort() — factored-out public helper returning (X_named, residual_series, group_stats). Both main() and scripts/tmb_resid_gp.py call it, so any tweak to the within-MSI z-scored log1p(TMB) target lives in ONE place. Existing diagnostic behaviour byte-for-byte identical (verified by pytest 130/130).
  • scripts/tmb_resid_gp.py — reads the existing processed matrix; imports the residual via build_residual_cohort(); anonymise(X_named) reuses the sealed map; runs run_v2_pipeline(M, y, objective=TMB_OBJECTIVE, seed, ...) per seed with prefilter_n=None, scalar_share_override=0.0 (Vector programs only), coherence_weight=0.0, diversity on (tournament_k=2, p_mutate=0.85, immigrant_fraction=0.10); default budget --pop 300 --gens 50 --perms 200 on seeds [1, 7, 13]. Per-seed grader reveals only the winner's opaque IDs via airgap.reveal (bounded — same discipline as /evaluate) and computes the SINGLE-GENE CEILING on the SAME held-out test rows via _test_ids_for(M, y, seed) (mirrors the pipeline's continuous split: make_split(stratify=False, random_state=seed)) + _spearman_per_column(X_test, y_test). synergy = combined − ceiling. Prints per-seed program_repr / revealed genes / combined |spearman| (with n_test) / ceiling / synergy / permutation p, ranges across seeds, then an overall verdict:
    • median syn ≥ 0.10 && n_sig ≥ ⌈n/2⌉ && median combined ≥ 0.30 → REAL COMBINATORIAL SIGNAL.
    • |median syn| < 0.05 && median combined < 0.50 → NO SYNERGY.
    • median combined < 0.20 || range ≥ 0.20 → THE LEFTOVER IS LARGELY NOISE.
    • else BORDERLINE.
  • Airgap. engine_v2 sees only opaque IDs; symbols cross the boundary once per seed via a bounded reveal(winner_ids) call. No engine / API / UI / dataset change; writes nothing to disk; deleting the script leaves zero trace.
  • Verified live. Smoke --seeds 1 --pop 60 --gens 10 --perms 20 prints the full per-seed block + ranges + verdict; scipy's ConstantInputWarning from degenerate programs (already floored to WORST_FITNESS by the engine) is silenced at script scope. pytest -q still 130/130.
• Chunk 7 follow-up (done — Groups-the-engine-explored: ascending/descending sort toggle): Frontend-only. The Groups table (<ModuleRankingPanel> in web/app/Lab.tsx) sorted descending only; clicking the active sort key now flips direction, clicking a new key resets to descending, and the active button's label appends ↓ / ↑ so the current direction is legible. Implementation: new SortDir = "asc" | "desc" state next to sortKey; sortedModules refactored to a single valueOf switch with null / non-finite modules PINNED to the bottom in BOTH directions (so a missing metric never floats to the top on ascending). Reset effect fires on sortDir too — flipping direction resets page 0 and closes any expanded row. tsc --noEmit clean; no API / airgap / engine change.
• Chunk 7 follow-up (done — Groups-the-engine-explored: plain- English copy pass on the panel + per-column "?" tooltips): Presentation-only. Two copy rewrites and four new column-header tooltips in <ModuleRankingPanel> (web/app/Lab.tsx).
  • Subtitle rewritten to the four-column story ("Every gene group the engine tried, scored four ways so you can judge whole groups, not just single genes") with a one-liner gloss per column. Fixes a literal ’ that was rendering in the old subtitle; straight apostrophes only.
  • TIPS.moduleRanking (the panel-title ?) rewritten to lead with "Why these columns exist" — the single-gene vs group question the panel is meant to answer — followed by the honest re-scoring caveat.
  • Each of the four metric column headers (Genetic-programming fitness · Combined AUROC / |ρ| · Coherence · Synergy) now carries a small <InfoTip> that says how to READ that column (high vs low), wrapped in <span className="inline-flex items-center justify-end gap-1"> so it hugs the right edge with the text-right header. Combined header's metric_kind conditional preserved inside the wrapper.
  • tsc --noEmit clean; no API / airgap / engine change.
• Next: survival + unsupervised objectives end-to-end; bigger default Lab budget for the full grammar; cross-cohort validation; durable run store; mechanism-aware objective; lifting Search's default rate once its cost profile is profiled.

Layout

data_pipeline/    download + build for both datasets. CRC:
                    download.py + build.py → data/processed/*.parquet
                    HNSC: download_hnsc.py + build_hnsc.py →
                    data/processed_hnsc/*.parquet (with hpv_status label)
app/              Streamlit viewer + presentation theme (theme.py)
.streamlit/       theme config.toml (light, Helvetica)
data/raw/         cBioPortal CRC files (gitignored)
data/raw_hnsc/    cBioPortal HNSC files (gitignored)
data/processed/   clinical.parquet, expression.parquet,
                  _sealed_gene_map.json,
                  h2/{evolution_log,result}.json   (all gitignored)
data/processed_hnsc/  HNSC clinical.parquet (with hpv_status) +
                      expression.parquet (gitignored)
dsl/              DSL operators (Load, Select, Reduce, Split, Associate,
                  Effect, Search, Fit, Apply) — label-agnostic, biology-free
airgap/           anonymise / reveal + sealed symbol<->ID map
engine/           v1 GP engine (split, prefilter, program, fitness,
                  baseline, permutation, gp, pipeline) — strict airgap,
                  biology-free. Fixed 1–2 Select+Reduce sets fed into
                  LogisticRegression.
engine_v2/        v2 typed program synthesis (types, nodes, synthesize,
                  fitness, permutation, gp, pipeline) — strict airgap.
                  Full DSL grammar (Select/Reduce/Combine/Split/
                  Associate/Effect/FitApply/Search) over the opaque-ID
                  matrix; ExecContext carries named clinical fields
                  (stage, age) and label arrays (msi, tmb) only — no
                  gene names. Worst-score floor for degenerate
                  programs; winner-fixed permutation null. Search is
                  bounded + gated OFF by default.
api/              FastAPI app: legacy /health /run /result /reveal +
                  Lab endpoints /runs /runs/{id} /runs/{id}/stream
                  /runs/{id}/result /runs/{id}/population/{gen}
                  /evaluate. POST /runs accepts engine: v1|v2.
                  CORS-enabled; SSE via sse-starlette; worker-thread
                  bridge; _json_finite() walker on every return so the
                  wire never carries NaN/Infinity tokens.
scripts/          Top-level orchestrators (biology-aware): run_h2.py wires
                  data + airgap + engine together
validate/         h1.py: known-answer H1 fixture on the NAMED matrix
                  h2.py: reveal winner + check MMR overlap
                  tmb_rank.py: signed-Spearman per gene vs TMB on the
                    NAMED matrix; reports where MMR / IMMUNE genes
                    land + top-10 most-negative. Powers the
                    /diagnostic/tmb-rank endpoint and the colorectal
                    Reference-gene diagnostic panel.
                  hpv_rank.py: single-gene orientation-agnostic AUROC
                    per gene vs HPV+/HPV− on the engine's HNSC TRAIN
                    split; reports where CDKN2A + each cell-cycle
                    gene lands + top single-gene separators. Powers
                    /diagnostic/hpv-rank and the HNSC Reference-gene
                    diagnostic panel.
web/              Next.js (App Router) + TS + Tailwind + Recharts +
                  reactflow Lab page (Stage 1 MVP). Talks to FastAPI;
                  no separate build pipeline for the engine. Single
                  <InfoTip> drives every "?" in the UI. Program graph
                  is a custom React Flow canvas in ProgramGraph.tsx —
                  Tier-1 group per gene-set, Tier-2 wrapper for 2-set
                  programs, palette tokens only, default chrome stripped.
tests/            pytest, no network (test_dsl.py, test_airgap.py,
                  test_h1.py, test_build.py, test_engine.py,
                  test_engine_v2.py, test_api_airgap.py)

Hard rules

• Stubs stay biology-free. dsl/, airgap/, engine/, validate/ must not hardcode gene names, pathway names, or MSI-specific constants. The data layer and viewer use real gene symbols freely; the engine only ever sees the airgapped view.
• Schema constants live in one place. data_pipeline/schema.py — filenames, required columns, MSI thresholds. If cBioPortal renames a column, edit there, not in build.py.
• MSI label is derived, not shipped. cBioPortal has no clean MSI-H/MSS column. We derive from MSI_SENSOR_SCORE per the file's own documented thresholds (MSI_SENSOR_HIGH = 10.0, MSI_SENSOR_LOW = 4.0). build.py prints this provenance every run.
• Presentation lives in app/theme.py. Palette + Altair theme. Don't override per-chart with .configure_*. MSI colours/order: import from theme, never inline. The H1 composition diagram (DOT in app/viewer.py) follows the same palette family: operator nodes stone (#F4F2EE), score meta-concepts cream (#FBEFE2 / #BC6B2E), output nodes cool-tinted (#EAF0F2 / #3A6B7E), Given: input notes muted grey (#EFEFEA / #9AA0A6).
• H1 conclusion is computed, not hardcoded. app/viewer.py derives the pass/fail callout from three checks: A (MMR median lower in MSI-H), B (immune median higher in MSI-H), C (held-out AUROC ≥ _AUROC_THRESHOLD, currently 0.75). Editing the threshold or any check requires editing _section_h1, not the callout copy.
• Airgap is load-bearing. Only airgap.reveal may open data/processed/_sealed_gene_map.json. The engine must never import reveal, airgap.seal, or reference the sealed-map filename — a test in tests/test_airgap.py scans engine/ for those tokens. At runtime, both dsl.Search and engine.run_gp_pipeline assert their input matrices have columns matching ^g\d+$ only. Biology-aware orchestration lives in scripts/ (the cohort label "MSI-H" / column name "msi_status" appears there and in validate/, but never in engine/ or dsl/).
• H2 artefacts are anonymised on disk. data/processed/h2/evolution_log.json and data/processed/h2/result.json only ever contain opaque IDs. The reveal step happens at runtime, exactly once, via the FastAPI /reveal endpoint (or validate.h2.reveal_winner).
• Lab payloads are anonymised over the wire. tests/test_api_airgap.py POSTs synthetic runs against both presets, drains the SSE stream, and asserts no gene symbol leaks through /runs/{id}, /runs/{id}/result, or the stream. /evaluate translates only the IDs the client sends — never the whole map.
• Two engine objectives, both selectable from the API. BinaryAUROCObjective (MSI/MSS — stratified split + StratifiedKFold + AUROC) and CorrelationObjective(direction) (continuous, e.g. TMB — random split + KFold + signed Spearman of the sum-of-set-means with the target). Each objective owns its own prefilter_score_per_feature and permute(y) so the GP/baseline/permutation loop stays generic. Add a new objective by subclassing engine.objectives.Objective and wiring it into objective_from_spec.
• Prefilter is a strict-speed knob, not a correctness knob. When prefilter_n=None the GP samples (init + mutation) from the full opaque column set, so it can compose any gene combination present in the matrix. When an integer is passed, the GP is confined to the univariate top-N — faster, but a gene that only shows signal in combination can be dropped. The baseline always uses the univariate top-K (independent of the GP's pool) so it's a fair sanity check either way.
• Fitness never leaves the API as NaN / Infinity. Engines floor degenerate-program fitness to the objective's finite worst (worst_score() on V2Objective: 0.5 for MSI omni-AUROC, 0.0 for TMB correlation). The API also runs every endpoint return + SSE payload through _json_finite() so anything that slips through becomes None. Frontend uses one fmtFit(x) helper everywhere a fitness/score is printed — falls back to "—" for non-finite, and fitnessForOrder(x) sinks non-finite tiles to the bottom of the grid with the lightest tint.

Setup & run (Python 3.11 venv in .venv/)

source .venv/bin/activate
python -m data_pipeline.download   # cBioPortal -> data/raw/
python -m data_pipeline.build      # -> data/processed/*.parquet
pytest
python -m scripts.run_h2           # GP engine -> data/processed/h2/*.json
uvicorn api.app:app --reload       # API on http://localhost:8000
streamlit run app/viewer.py        # Streamlit viewer (Dataset / H1 / H2 tabs)

# Lab (Next.js front end against the same API)
cd web && npm install && npm run dev   # http://localhost:3000

Data provenance

• Study: coadread_tcga_pan_can_atlas_2018 (TCGA CRC PanCancer Atlas).
• Source: https://media.githubusercontent.com/media/cBioPortal/datahub/master/public/<study>/<file> (the LFS-resolved URL; the S3 tarball mirror 403s, raw.githubusercontent.com returns 131-byte LFS pointers).
• data_mutations.txt is optional — datahub is over its GitHub LFS budget for that file. Chunk 1 does not need it (TMB comes from data_clinical_sample.txt TMB_NONSYNONYMOUS).
• Expected cohort after build: ~594 samples (378 COAD, 155 READ, 61 MACR), 20,504 genes after dedup, 558 usable (MSI in {MSI-H, MSS} ∧ expression ∧ stage ∧ age). Sanity check: median TMB MSI-H ≈ 39, MSS ≈ 3.

Style

• Default to no comments. The README and this file carry the why.
• Keep tests/ network-free; use synthetic cBioPortal-shaped fixtures.
• Streamlit charts: width="stretch" (the newer API; use_container_width is deprecated in 1.58+).