Upload folder using huggingface_hub

CLAUDE.md

@@ -1911,6 +1911,87 @@ Built in chunks. Current state:
     strongest immune gene at rank 161 (`|corr| = 0.19`). Verdict:
     **BORDERLINE** — *"A weak single-gene signal remains; a GP run
     might still be informative."*
+- **Chunk 7 follow-up (done — `scripts/tmb_resid_gp.py` synergy
+  check on MSI-residualized TMB):** Second isolated script that
+  answers whether the DSL finds gene COMBINATIONS beating the best
+  single gene on the "leftover TMB" target the diagnostic borderlined.
+  ONE new file under `scripts/` (biology-aware) + a small helper
+  refactor in `validate/tmb_resid_rank.py` to keep the residual
+  byte-identical between the diagnostic and the GP script.
+  - **`validate.tmb_resid_rank.build_residual_cohort()`** —
+    factored-out public helper returning `(X_named, residual_series,
+    group_stats)`. Both `main()` and `scripts/tmb_resid_gp.py` call
+    it, so any tweak to the within-MSI z-scored `log1p(TMB)` target
+    lives in ONE place. Existing diagnostic behaviour byte-for-byte
+    identical (verified by pytest 130/130).
+  - **`scripts/tmb_resid_gp.py`** — reads the existing processed
+    matrix; imports the residual via `build_residual_cohort()`;
+    `anonymise(X_named)` reuses the sealed map; runs
+    `run_v2_pipeline(M, y, objective=TMB_OBJECTIVE, seed, ...)` per
+    seed with `prefilter_n=None`, `scalar_share_override=0.0`
+    (Vector programs only), `coherence_weight=0.0`, diversity on
+    (`tournament_k=2, p_mutate=0.85, immigrant_fraction=0.10`);
+    default budget `--pop 300 --gens 50 --perms 200` on seeds
+    `[1, 7, 13]`. Per-seed grader reveals only the winner's opaque
+    IDs via `airgap.reveal` (bounded — same discipline as
+    `/evaluate`) and computes the SINGLE-GENE CEILING on the SAME
+    held-out test rows via `_test_ids_for(M, y, seed)` (mirrors the
+    pipeline's continuous split: `make_split(stratify=False,
+    random_state=seed)`) + `_spearman_per_column(X_test, y_test)`.
+    `synergy = combined − ceiling`. Prints per-seed
+    program_repr / revealed genes / combined |spearman| (with
+    n_test) / ceiling / synergy / permutation p, ranges across
+    seeds, then an overall verdict:
+    - `median syn ≥ 0.10 && n_sig ≥ ⌈n/2⌉ && median combined ≥ 0.30`
+      → **REAL COMBINATORIAL SIGNAL**.
+    - `|median syn| < 0.05 && median combined < 0.50` →
+      **NO SYNERGY**.
+    - `median combined < 0.20 || range ≥ 0.20` →
+      **THE LEFTOVER IS LARGELY NOISE**.
+    - else **BORDERLINE**.
+  - **Airgap.** `engine_v2` sees only opaque IDs; symbols cross the
+    boundary once per seed via a bounded `reveal(winner_ids)` call.
+    No engine / API / UI / dataset change; writes nothing to disk;
+    deleting the script leaves zero trace.
+  - **Verified live.** Smoke `--seeds 1 --pop 60 --gens 10 --perms 20`
+    prints the full per-seed block + ranges + verdict; scipy's
+    `ConstantInputWarning` from degenerate programs (already floored
+    to `WORST_FITNESS` by the engine) is silenced at script scope.
+    `pytest -q` still 130/130.
+- **Chunk 7 follow-up (done — Groups-the-engine-explored:
+  ascending/descending sort toggle):** Frontend-only. The Groups
+  table (`<ModuleRankingPanel>` in `web/app/Lab.tsx`) sorted
+  descending only; clicking the active sort key now flips direction,
+  clicking a new key resets to descending, and the active button's
+  label appends `↓` / `↑` so the current direction is legible.
+  Implementation: new `SortDir = "asc" | "desc"` state next to
+  `sortKey`; `sortedModules` refactored to a single `valueOf`
+  switch with null / non-finite modules PINNED to the bottom in
+  BOTH directions (so a missing metric never floats to the top on
+  ascending). Reset effect fires on `sortDir` too — flipping
+  direction resets page 0 and closes any expanded row. `tsc
+  --noEmit` clean; no API / airgap / engine change.
+- **Chunk 7 follow-up (done — Groups-the-engine-explored: plain-
+  English copy pass on the panel + per-column "?" tooltips):**
+  Presentation-only. Two copy rewrites and four new column-header
+  tooltips in `<ModuleRankingPanel>` (`web/app/Lab.tsx`).
+  - Subtitle rewritten to the four-column story ("Every gene group
+    the engine tried, scored four ways so you can judge whole
+    groups, not just single genes") with a one-liner gloss per
+    column. Fixes a literal `&rsquo;` that was rendering in the
+    old subtitle; straight apostrophes only.
+  - `TIPS.moduleRanking` (the panel-title `?`) rewritten to lead
+    with "Why these columns exist" — the single-gene vs group
+    question the panel is meant to answer — followed by the
+    honest re-scoring caveat.
+  - Each of the four metric column headers (Genetic-programming
+    fitness · Combined AUROC / |ρ| · Coherence · Synergy) now
+    carries a small `<InfoTip>` that says how to READ that column
+    (high vs low), wrapped in
+    `<span className="inline-flex items-center justify-end gap-1">`
+    so it hugs the right edge with the text-right header. Combined
+    header's `metric_kind` conditional preserved inside the wrapper.
+  - `tsc --noEmit` clean; no API / airgap / engine change.
 - **Next:** survival + unsupervised objectives end-to-end; bigger
   default Lab budget for the full grammar; cross-cohort validation;
   durable run store; mechanism-aware objective; lifting Search's

web/app/Lab.tsx

@@ -233,28 +233,22 @@ const TIPS = {
     "Scalar share = the fraction of programs the engine seeds with a " +
     "Scalar root (Associate / Effect) versus a Vector root.",
   moduleRanking:
-    "Every gene group the engine tried during the search. Four sort " +
-    "lenses:\n\n" +
-    "• Genetic-programming fitness — the engine's OWN preference order, what the search " +
-    "actually picked (its cross-validated objective). The default " +
-    "sort.\n" +
-    "• Combined AUROC — the group's mean across its genes, scored on " +
-    "patients held out of training. A simpler yardstick.\n" +
-    "• Coherence — mean absolute pairwise correlation among the " +
-    "group's genes on TRAIN. High = the genes move together (a real " +
-    "module); low = an arbitrary set.\n" +
-    "• Synergy — combined held-out minus the best single gene's " +
-    "score. High = the GROUP beats its strongest constituent (a real " +
-    "additive effect); low / negative = a “best gene + passengers” " +
-    "pattern.\n\n" +
-    "The badged row is the winning program. The Survives flags show " +
-    "whether each group still separates the label inside the " +
-    "stratified confound subgroups (HNSC/HPV).\n\n" +
-    "Caveat: Combined AUROC / Coherence / Synergy re-score ~2,900 " +
-    "groups on the same small held-out set, so their very top values " +
-    "are optimistically biased (the luckiest of thousands). The Genetic-programming " +
-    "fitness order (and the badged winner) is the engine's own, " +
-    "cross-validated pick — trust it as the reliable choice.",
+    "Why these columns exist: once the engine could detect the trait, " +
+    "the real question was whether the signal came from a few genes " +
+    "working together or from one strong gene, and single-gene " +
+    "rankings can't tell you that. These measures judge whole groups " +
+    "instead. Coherence tells a genuine co-expressed module from a " +
+    "random bag of genes. Combined AUROC scores the group as a group, " +
+    "on patients it never saw. Synergy separates real teamwork from " +
+    "one good gene with passengers along for the ride. " +
+    "Genetic-programming fitness sits beside them as the engine's own " +
+    "verdict, so you can compare what the engine actually preferred " +
+    "against these after-the-fact re-scores. Together they answer one " +
+    "question: is the result carried by gene teamwork or by lone " +
+    "strong genes? Note that the re-scores are computed after the " +
+    "run, so they can rank groups differently from the engine's own " +
+    "picks, and the very top re-score values are optimistically " +
+    "biased because they are the best of thousands.",
   moduleSurvival:
     "What “Survives” checks: whether a group still separates HPV when " +
     "you take away a possible confounder — something that travels with " +
@@ -5836,10 +5830,12 @@ function ModuleRankingPanel({
   const [loading, setLoading] = useState<boolean>(false);
   const [error, setError] = useState<string | null>(null);
   type SortKey = "gp_fitness" | "combined" | "coherence" | "synergy";
+  type SortDir = "asc" | "desc";
   // Default sort: the engine's own preference order (what the search
   // actually picked). The other three are after-the-fact re-score
   // lenses.
   const [sortKey, setSortKey] = useState<SortKey>("gp_fitness");
+  const [sortDir, setSortDir] = useState<SortDir>("desc");
   const [page, setPage] = useState<number>(0);
   // Symbol cache keyed by opaque ID — accumulates across pages so
   // re-visiting a page is free.
@@ -5878,27 +5874,30 @@ function ModuleRankingPanel({
   const sortedModules = useMemo<RankedModule[]>(() => {
     if (!data) return [];
     const xs = [...data.modules];
-    if (sortKey === "coherence") {
-      xs.sort((a, b) => (b.coherence ?? -Infinity) - (a.coherence ?? -Infinity));
-    } else if (sortKey === "gp_fitness") {
-      xs.sort(
-        (a, b) => (b.gp_fitness ?? -Infinity) - (a.gp_fitness ?? -Infinity),
-      );
-    } else if (sortKey === "synergy") {
-      xs.sort(
-        (a, b) => (synergyOf(b) ?? -Infinity) - (synergyOf(a) ?? -Infinity),
-      );
-    } else {
-      xs.sort(
-        (a, b) =>
-          (b.combined_holdout ?? -Infinity) - (a.combined_holdout ?? -Infinity),
-      );
-    }
+    const valueOf = (m: RankedModule): number | null | undefined => {
+      switch (sortKey) {
+        case "coherence": return m.coherence;
+        case "gp_fitness": return m.gp_fitness;
+        case "synergy": return synergyOf(m);
+        default: return m.combined_holdout;
+      }
+    };
+    xs.sort((a, b) => {
+      const va = valueOf(a), vb = valueOf(b);
+      const fa = va == null || !Number.isFinite(va);
+      const fb = vb == null || !Number.isFinite(vb);
+      if (fa && fb) return 0;
+      if (fa) return 1;
+      if (fb) return -1;
+      return sortDir === "asc"
+        ? (va as number) - (vb as number)
+        : (vb as number) - (va as number);
+    });
     return xs;
     // synergyOf depends only on RankedModule's stable per-gene fields,
     // so we leave it out of the dep list.
     // eslint-disable-next-line react-hooks/exhaustive-deps
-  }, [data, sortKey]);
+  }, [data, sortKey, sortDir]);
 
   // Re-sort or re-load resets the page + collapses any open row so an
   // expanded "rank N" row doesn't strand once the underlying ranking
@@ -5906,7 +5905,7 @@ function ModuleRankingPanel({
   useEffect(() => {
     setPage(0);
     setExpandedIx(null);
-  }, [sortKey, data]);
+  }, [sortKey, sortDir, data]);
 
   // Winner-set equality: unordered set of opaque IDs equal to the
   // winner's gene_ids. Used to badge that exact module — connects the
@@ -5971,14 +5970,16 @@ function ModuleRankingPanel({
       title="Groups the engine explored"
       titleTip={TIPS.moduleRanking}
       subtitle={
-        "Every gene group the engine tried during the search. Sort by " +
-        "Genetic-programming fitness (the engine&rsquo;s own picks), " +
-        "Combined AUROC (the group&rsquo;s average score on unseen " +
-        "patients), Coherence (how tightly its genes move together), " +
-        "or Synergy (how much the group beats its single best gene). " +
-        "The winning program is badged. The Survives flags show " +
-        "whether a group still separates the label under the confound " +
-        "checks."
+        "Every gene group the engine tried, scored four ways so you can " +
+        "judge whole groups, not just single genes. Genetic-programming " +
+        "fitness: how much the engine itself liked the group. Combined " +
+        "AUROC: how well the group separates the two kinds of tumour on " +
+        "patients it never saw. Coherence: how tightly its genes move " +
+        "together (a real module versus a random bag). Synergy: how much " +
+        "the group beats its single best gene (real teamwork versus one " +
+        "strong gene carrying passengers). The winning program is badged; " +
+        "the Survives flags show whether a group still separates the " +
+        "label under the confound checks."
       }
     >
       {loading && (
@@ -6012,7 +6013,10 @@ function ModuleRankingPanel({
                 <button
                   key={k}
                   type="button"
-                  onClick={() => setSortKey(k)}
+                  onClick={() => {
+                    if (sortKey === k) setSortDir((d) => (d === "desc" ? "asc" : "desc"));
+                    else { setSortKey(k); setSortDir("desc"); }
+                  }}
                   aria-pressed={sortKey === k}
                   className={[
                     "rounded-md px-2 py-0.5 text-[11px] transition-colors",
@@ -6021,7 +6025,7 @@ function ModuleRankingPanel({
                       : "border border-border bg-card text-ink hover:border-accent/40",
                   ].join(" ")}
                 >
-                  {label}
+                  {label}{sortKey === k ? (sortDir === "desc" ? " ↓" : " ↑") : ""}
                 </button>
               ))}
             </div>
@@ -6068,14 +6072,44 @@ function ModuleRankingPanel({
               <thead className="bg-card text-xs tracking-wide text-muted">
                 <tr>
                   <th className="px-3 py-2 text-left w-10">#</th>
-                  <th className="px-3 py-2 text-right">Genetic-programming fitness</th>
                   <th className="px-3 py-2 text-right">
-                    {data.metric_kind === "auroc"
-                      ? "Combined AUROC"
-                      : "Combined |ρ|"}
+                    <span className="inline-flex items-center justify-end gap-1">
+                      Genetic-programming fitness
+                      <InfoTip
+                        text="Higher means the engine liked this group more during the search. The very highest are its actual picks; low values are groups it barely looked at."
+                        label="How to read Genetic-programming fitness"
+                      />
+                    </span>
+                  </th>
+                  <th className="px-3 py-2 text-right">
+                    <span className="inline-flex items-center justify-end gap-1">
+                      {data.metric_kind === "auroc"
+                        ? "Combined AUROC"
+                        : "Combined |ρ|"}
+                      <InfoTip
+                        text="How well the whole group separates the two kinds of tmour on patients it never saw. Around 0.5 is no better than chance; close to 1.0 is a clean split. For a continuous target this is a correlation instead, where further from 0 is stronger."
+                        label="How to read Combined AUROC"
+                      />
+                    </span>
+                  </th>
+                  <th className="px-3 py-2 text-right">
+                    <span className="inline-flex items-center justify-end gap-1">
+                      Coherence
+                      <InfoTip
+                        text="Higher means the group's genes move together tightly, so it is a real co-expressed module. Low means the genes are largely unrelated, a random bag."
+                        label="How to read Coherence"
+                      />
+                    </span>
+                  </th>
+                  <th className="px-3 py-2 text-right">
+                    <span className="inline-flex items-center justify-end gap-1">
+                      Synergy
+                      <InfoTip
+                        text="Positive and high means the group genuinely beats its best single gene, real teamwork. Around zero or negative means one strong gene is doing the work and the rest are passengers."
+                        label="How to read Synergy"
+                      />
+                    </span>
                   </th>
-                  <th className="px-3 py-2 text-right">Coherence</th>
-                  <th className="px-3 py-2 text-right">Synergy</th>
                   <th className="px-3 py-2 text-right">Size</th>
                   <th className="px-3 py-2 text-left">Genes</th>
                   <th className="px-3 py-2 text-left">Tags</th>