Add files using upload-large-folder tool

.gitignore

@@ -0,0 +1,53 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+*.egg
+dist/
+build/
+.eggs/
+
+# Virtual environments
+.venv/
+venv/
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# Testing
+.pytest_cache/
+.coverage
+htmlcov/
+
+# Output (generated results)
+output/
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Cache
+.cache/
+
+# Large data files (downloaded at runtime)
+/data/
+*.h5ad
+.pybiomart.sqlite
+
+# Local-only working directories (kept out of the public repo)
+neurips2026/
+supplementary/
+supplementary.zip
+.supplementary-build/
+
+# Local scratch from running figure scripts at the repo root
+/compute_fig1c_data.py
+/compute_real_figure_data.py
+/generate_figures.py
+/figures/
+/real_figure_data/
+err
+icml-extract/

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Bryan Cheng, Austin Jin
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,3 +1,108 @@
----
-license: mit
----
+# scPTR
+
+**Single-Cell Post-Transcriptional Regulatory Decomposition**
+
+scPTR estimates per-cell, per-gene mRNA degradation rates from scRNA-seq spliced/unspliced counts and uses them as a primary analytical axis — complementary to RNA velocity.
+
+## What scPTR does
+
+- **Degradation rate estimation**: Per-cell, per-gene gamma from kinetic steady-state relationships with kNN Gaussian-kernel smoothing
+- **Expression-invisible states**: Discovers cell subpopulations with distinct post-transcriptional programs undetectable by standard expression analysis
+- **Post-transcriptional velocity**: Neighbor-averaged gamma gradient that captures degradation dynamics orthogonal to RNA velocity
+- **RBP-target networks**: Library-size-corrected inference of RNA-binding protein regulatory networks with elastic net
+- **DeepPTR**: Structured VAE with a kinetic decoder that disentangles transcriptional and post-transcriptional latent spaces
+
+## Installation
+
+```bash
+pip install .
+```
+
+Optional dependencies:
+
+```bash
+pip install ".[deep]"      # PyTorch for DeepPTR
+pip install ".[datasets]"  # Pooch for dataset downloads
+pip install ".[dev]"       # pytest for testing
+```
+
+## Quick start
+
+```python
+import scptr
+
+# Load data with spliced/unspliced layers
+adata = scptr.read_h5ad("your_data.h5ad")
+
+# Preprocessing
+scptr.pp.filter_genes(adata)
+scptr.pp.normalize_layers(adata)
+scptr.pp.neighbors(adata)
+scptr.pp.smooth_layers(adata)
+
+# Estimate rates
+scptr.tl.estimate_beta(adata)
+scptr.tl.estimate_gamma(adata)
+
+# Downstream analysis
+scptr.tl.variance_decomposition(adata)
+scptr.tl.pt_states(adata)
+scptr.tl.pt_velocity(adata)
+scptr.tl.infer_network(adata)
+```
+
+## Pipeline overview
+
+```
+Raw scRNA-seq (spliced + unspliced)
+  -> Gene/cell filtering
+  -> Library-size normalization (per layer)
+  -> kNN graph + Gaussian smoothing
+  -> Beta estimation (quantile regression on u/s phase portraits)
+  -> Gamma estimation (gamma = beta * u / s, per cell per gene)
+  -> Variance decomposition (transcriptional vs post-transcriptional)
+  -> PT states (PCA + Leiden clustering in gamma-space)
+  -> PT velocity (neighbor-averaged gamma gradient)
+  -> RBP-target network inference (elastic net, library-size corrected)
+```
+
+## Validation
+
+scPTR gamma estimates have been validated against:
+
+| Validation | Result |
+|------------|--------|
+| Published mRNA half-lives | ρ = −0.81 (sci-fate), −0.33 to −0.40 (10x developmental) |
+| Method comparison | Outperforms scVelo steady-state (−0.37) and velVI (−0.28) |
+| miRNA target enrichment | 59% of 215 families enriched (p = 4.7×10⁻⁶⁵) |
+| 3′ UTR sequence features | UTR length ρ = 0.34 (p < 10⁻²⁰⁰), AU content ρ = 0.30 |
+| DepMap CRISPR essentiality | Hub RBPs more essential (p = 6.4×10⁻⁵) |
+| Subsampling robustness | r > 0.97 at 20% subsampling |
+
+## Key findings
+
+- **Expression-invisible states**: 3/8 pancreatic and 6/11 hippocampal cell types harbor post-transcriptional subpopulations undetectable by expression analysis (confirmed by zero-permutation control, ARI ≈ 0), enriched for ER stress/autophagy and synaptic plasticity pathways
+- **Temporal precedence**: degradation-rate changes precede expression changes for 54% of transition genes in pancreas (p < 10⁻⁵⁷) and 78% in dentate gyrus (p = 9.9×10⁻¹³)
+- **RBP networks**: library-size-corrected inference identifies essential hub regulators (HNRNPA1, YBX1, ELAVL1/HuR); neuroblastoma shows 66% stabilizing edges vs. destabilizing bias in developmental tissues
+
+## Datasets
+
+Built-in dataset loaders (downloaded via Pooch):
+
+```python
+adata = scptr.datasets.pancreas()        # Mouse endocrinogenesis (3,696 cells)
+adata = scptr.datasets.dentate_gyrus()   # Mouse hippocampal neurogenesis (2,930 cells)
+adata = scptr.datasets.sci_fate()        # Human A549 dexamethasone response (7,404 cells)
+```
+
+## Requirements
+
+- Python >= 3.9
+- anndata >= 0.8, scanpy >= 1.9, numpy >= 1.21, scipy >= 1.7, numba >= 0.55
+- Optional: torch >= 2.0 (DeepPTR), pooch >= 1.6 (datasets)
+
+## Citation
+
+If you use scPTR, please cite:
+
+> scPTR: Decomposing Post-Transcriptional Regulation at Single-Cell Resolution (2026)

algorithm.sty

@@ -0,0 +1,79 @@
+% ALGORITHM STYLE -- Released 8 April 1996
+%    for LaTeX-2e
+% Copyright -- 1994 Peter Williams
+% E-mail Peter.Williams@dsto.defence.gov.au
+\NeedsTeXFormat{LaTeX2e}
+\ProvidesPackage{algorithm}
+\typeout{Document Style `algorithm' - floating environment}
+
+\RequirePackage{float}
+\RequirePackage{ifthen}
+\newcommand{\ALG@within}{nothing}
+\newboolean{ALG@within}
+\setboolean{ALG@within}{false}
+\newcommand{\ALG@floatstyle}{ruled}
+\newcommand{\ALG@name}{Algorithm}
+\newcommand{\listalgorithmname}{List of \ALG@name s}
+
+% Declare Options
+% first appearance
+\DeclareOption{plain}{
+  \renewcommand{\ALG@floatstyle}{plain}
+}
+\DeclareOption{ruled}{
+  \renewcommand{\ALG@floatstyle}{ruled}
+}
+\DeclareOption{boxed}{
+  \renewcommand{\ALG@floatstyle}{boxed}
+}
+% then numbering convention
+\DeclareOption{part}{
+  \renewcommand{\ALG@within}{part}
+  \setboolean{ALG@within}{true}
+}
+\DeclareOption{chapter}{
+  \renewcommand{\ALG@within}{chapter}
+  \setboolean{ALG@within}{true}
+}
+\DeclareOption{section}{
+  \renewcommand{\ALG@within}{section}
+  \setboolean{ALG@within}{true}
+}
+\DeclareOption{subsection}{
+  \renewcommand{\ALG@within}{subsection}
+  \setboolean{ALG@within}{true}
+}
+\DeclareOption{subsubsection}{
+  \renewcommand{\ALG@within}{subsubsection}
+  \setboolean{ALG@within}{true}
+}
+\DeclareOption{nothing}{
+  \renewcommand{\ALG@within}{nothing}
+  \setboolean{ALG@within}{true}
+}
+\DeclareOption*{\edef\ALG@name{\CurrentOption}}
+
+% ALGORITHM
+%
+\ProcessOptions
+\floatstyle{\ALG@floatstyle}
+\ifthenelse{\boolean{ALG@within}}{
+  \ifthenelse{\equal{\ALG@within}{part}}
+     {\newfloat{algorithm}{htbp}{loa}[part]}{}
+  \ifthenelse{\equal{\ALG@within}{chapter}}
+     {\newfloat{algorithm}{htbp}{loa}[chapter]}{}
+  \ifthenelse{\equal{\ALG@within}{section}}
+     {\newfloat{algorithm}{htbp}{loa}[section]}{}
+  \ifthenelse{\equal{\ALG@within}{subsection}}
+     {\newfloat{algorithm}{htbp}{loa}[subsection]}{}
+  \ifthenelse{\equal{\ALG@within}{subsubsection}}
+     {\newfloat{algorithm}{htbp}{loa}[subsubsection]}{}
+  \ifthenelse{\equal{\ALG@within}{nothing}}
+     {\newfloat{algorithm}{htbp}{loa}}{}
+}{
+  \newfloat{algorithm}{htbp}{loa}
+}
+\floatname{algorithm}{\ALG@name}
+
+\newcommand{\listofalgorithms}{\listof{algorithm}{\listalgorithmname}}
+

algorithmic.sty

@@ -0,0 +1,201 @@
+% ALGORITHMIC STYLE -- Released 8 APRIL 1996
+%    for LaTeX version 2e
+% Copyright -- 1994 Peter Williams
+% E-mail PeterWilliams@dsto.defence.gov.au
+%
+% Modified by Alex Smola (08/2000)
+% E-mail Alex.Smola@anu.edu.au
+%
+\NeedsTeXFormat{LaTeX2e}
+\ProvidesPackage{algorithmic}
+\typeout{Document Style `algorithmic' - environment}
+%
+\RequirePackage{ifthen}
+\RequirePackage{calc}
+\newboolean{ALC@noend}
+\setboolean{ALC@noend}{false}
+\newcounter{ALC@line}
+\newcounter{ALC@rem}
+\newlength{\ALC@tlm}
+%
+\DeclareOption{noend}{\setboolean{ALC@noend}{true}}
+%
+\ProcessOptions
+%
+% ALGORITHMIC
+\newcommand{\algorithmicrequire}{\textbf{Require:}}
+\newcommand{\algorithmicensure}{\textbf{Ensure:}}
+\newcommand{\algorithmiccomment}[1]{\{#1\}}
+\newcommand{\algorithmicend}{\textbf{end}}
+\newcommand{\algorithmicif}{\textbf{if}}
+\newcommand{\algorithmicthen}{\textbf{then}}
+\newcommand{\algorithmicelse}{\textbf{else}}
+\newcommand{\algorithmicelsif}{\algorithmicelse\ \algorithmicif}
+\newcommand{\algorithmicendif}{\algorithmicend\ \algorithmicif}
+\newcommand{\algorithmicfor}{\textbf{for}}
+\newcommand{\algorithmicforall}{\textbf{for all}}
+\newcommand{\algorithmicdo}{\textbf{do}}
+\newcommand{\algorithmicendfor}{\algorithmicend\ \algorithmicfor}
+\newcommand{\algorithmicwhile}{\textbf{while}}
+\newcommand{\algorithmicendwhile}{\algorithmicend\ \algorithmicwhile}
+\newcommand{\algorithmicloop}{\textbf{loop}}
+\newcommand{\algorithmicendloop}{\algorithmicend\ \algorithmicloop}
+\newcommand{\algorithmicrepeat}{\textbf{repeat}}
+\newcommand{\algorithmicuntil}{\textbf{until}}
+
+%changed by alex smola
+\newcommand{\algorithmicinput}{\textbf{input}}
+\newcommand{\algorithmicoutput}{\textbf{output}}
+\newcommand{\algorithmicset}{\textbf{set}}
+\newcommand{\algorithmictrue}{\textbf{true}}
+\newcommand{\algorithmicfalse}{\textbf{false}}
+\newcommand{\algorithmicand}{\textbf{and\ }}
+\newcommand{\algorithmicor}{\textbf{or\ }}
+\newcommand{\algorithmicfunction}{\textbf{function}}
+\newcommand{\algorithmicendfunction}{\algorithmicend\ \algorithmicfunction}
+\newcommand{\algorithmicmain}{\textbf{main}}
+\newcommand{\algorithmicendmain}{\algorithmicend\ \algorithmicmain}
+%end changed by alex smola
+
+\def\ALC@item[#1]{%
+\if@noparitem \@donoparitem
+  \else \if@inlabel \indent \par \fi
+         \ifhmode \unskip\unskip \par \fi
+         \if@newlist \if@nobreak \@nbitem \else
+                        \addpenalty\@beginparpenalty
+                        \addvspace\@topsep \addvspace{-\parskip}\fi
+           \else \addpenalty\@itempenalty \addvspace\itemsep
+          \fi
+    \global\@inlabeltrue
+\fi
+\everypar{\global\@minipagefalse\global\@newlistfalse
+          \if@inlabel\global\@inlabelfalse \hskip -\parindent \box\@labels
+             \penalty\z@ \fi
+          \everypar{}}\global\@nobreakfalse
+\if@noitemarg \@noitemargfalse \if@nmbrlist \refstepcounter{\@listctr}\fi \fi
+\sbox\@tempboxa{\makelabel{#1}}%
+\global\setbox\@labels
+ \hbox{\unhbox\@labels \hskip \itemindent
+       \hskip -\labelwidth \hskip -\ALC@tlm
+       \ifdim \wd\@tempboxa >\labelwidth
+                \box\@tempboxa
+          \else \hbox to\labelwidth {\unhbox\@tempboxa}\fi
+       \hskip \ALC@tlm}\ignorespaces}
+%
+\newenvironment{algorithmic}[1][0]{
+\let\@item\ALC@item
+  \newcommand{\ALC@lno}{%
+\ifthenelse{\equal{\arabic{ALC@rem}}{0}}
+{{\footnotesize \arabic{ALC@line}:}}{}%
+}
+\let\@listii\@listi
+\let\@listiii\@listi
+\let\@listiv\@listi
+\let\@listv\@listi
+\let\@listvi\@listi
+\let\@listvii\@listi
+  \newenvironment{ALC@g}{
+    \begin{list}{\ALC@lno}{ \itemsep\z@ \itemindent\z@
+    \listparindent\z@ \rightmargin\z@ 
+    \topsep\z@ \partopsep\z@ \parskip\z@\parsep\z@
+    \leftmargin 1em
+    \addtolength{\ALC@tlm}{\leftmargin}
+    }
+  }
+  {\end{list}}
+  \newcommand{\ALC@it}{\addtocounter{ALC@line}{1}\addtocounter{ALC@rem}{1}\ifthenelse{\equal{\arabic{ALC@rem}}{#1}}{\setcounter{ALC@rem}{0}}{}\item}
+  \newcommand{\ALC@com}[1]{\ifthenelse{\equal{##1}{default}}%
+{}{\ \algorithmiccomment{##1}}}
+  \newcommand{\REQUIRE}{\item[\algorithmicrequire]}
+  \newcommand{\ENSURE}{\item[\algorithmicensure]}
+  \newcommand{\STATE}{\ALC@it}
+  \newcommand{\COMMENT}[1]{\algorithmiccomment{##1}}
+%changes by alex smola
+  \newcommand{\INPUT}{\item[\algorithmicinput]}
+  \newcommand{\OUTPUT}{\item[\algorithmicoutput]}
+  \newcommand{\SET}{\item[\algorithmicset]}
+%  \newcommand{\TRUE}{\algorithmictrue}
+%  \newcommand{\FALSE}{\algorithmicfalse}
+  \newcommand{\AND}{\algorithmicand}
+  \newcommand{\OR}{\algorithmicor}
+  \newenvironment{ALC@func}{\begin{ALC@g}}{\end{ALC@g}}
+  \newenvironment{ALC@main}{\begin{ALC@g}}{\end{ALC@g}}
+%end changes by alex smola
+  \newenvironment{ALC@if}{\begin{ALC@g}}{\end{ALC@g}}
+  \newenvironment{ALC@for}{\begin{ALC@g}}{\end{ALC@g}}
+  \newenvironment{ALC@whl}{\begin{ALC@g}}{\end{ALC@g}}
+  \newenvironment{ALC@loop}{\begin{ALC@g}}{\end{ALC@g}}
+  \newenvironment{ALC@rpt}{\begin{ALC@g}}{\end{ALC@g}}
+  \renewcommand{\\}{\@centercr}
+  \newcommand{\IF}[2][default]{\ALC@it\algorithmicif\ ##2\ \algorithmicthen%
+\ALC@com{##1}\begin{ALC@if}}
+  \newcommand{\SHORTIF}[2]{\ALC@it\algorithmicif\ ##1\
+    \algorithmicthen\ {##2}}
+  \newcommand{\ELSE}[1][default]{\end{ALC@if}\ALC@it\algorithmicelse%
+\ALC@com{##1}\begin{ALC@if}}
+  \newcommand{\ELSIF}[2][default]%
+{\end{ALC@if}\ALC@it\algorithmicelsif\ ##2\ \algorithmicthen%
+\ALC@com{##1}\begin{ALC@if}}
+  \newcommand{\FOR}[2][default]{\ALC@it\algorithmicfor\ ##2\ \algorithmicdo%
+\ALC@com{##1}\begin{ALC@for}}
+  \newcommand{\FORALL}[2][default]{\ALC@it\algorithmicforall\ ##2\ %
+\algorithmicdo%
+\ALC@com{##1}\begin{ALC@for}}
+  \newcommand{\SHORTFORALL}[2]{\ALC@it\algorithmicforall\ ##1\ %
+    \algorithmicdo\ {##2}}
+  \newcommand{\WHILE}[2][default]{\ALC@it\algorithmicwhile\ ##2\ %
+\algorithmicdo%
+\ALC@com{##1}\begin{ALC@whl}}
+  \newcommand{\LOOP}[1][default]{\ALC@it\algorithmicloop%
+\ALC@com{##1}\begin{ALC@loop}}
+%changed by alex smola
+  \newcommand{\FUNCTION}[2][default]{\ALC@it\algorithmicfunction\ ##2\ %
+    \ALC@com{##1}\begin{ALC@func}}
+  \newcommand{\MAIN}[2][default]{\ALC@it\algorithmicmain\ ##2\ %
+    \ALC@com{##1}\begin{ALC@main}}
+%end changed by alex smola
+  \newcommand{\REPEAT}[1][default]{\ALC@it\algorithmicrepeat%
+    \ALC@com{##1}\begin{ALC@rpt}}
+    \newcommand{\UNTIL}[1]{\end{ALC@rpt}\ALC@it\algorithmicuntil\ ##1}
+  \ifthenelse{\boolean{ALC@noend}}{
+    \newcommand{\ENDIF}{\end{ALC@if}}
+    \newcommand{\ENDFOR}{\end{ALC@for}}
+    \newcommand{\ENDWHILE}{\end{ALC@whl}}
+    \newcommand{\ENDLOOP}{\end{ALC@loop}}
+    \newcommand{\ENDFUNCTION}{\end{ALC@func}}
+    \newcommand{\ENDMAIN}{\end{ALC@main}}
+  }{
+    \newcommand{\ENDIF}{\end{ALC@if}\ALC@it\algorithmicendif}
+    \newcommand{\ENDFOR}{\end{ALC@for}\ALC@it\algorithmicendfor}
+    \newcommand{\ENDWHILE}{\end{ALC@whl}\ALC@it\algorithmicendwhile}
+    \newcommand{\ENDLOOP}{\end{ALC@loop}\ALC@it\algorithmicendloop}
+    \newcommand{\ENDFUNCTION}{\end{ALC@func}\ALC@it\algorithmicendfunction}
+    \newcommand{\ENDMAIN}{\end{ALC@main}\ALC@it\algorithmicendmain}
+  } 
+  \renewcommand{\@toodeep}{}
+  \begin{list}{\ALC@lno}{\setcounter{ALC@line}{0}\setcounter{ALC@rem}{0}%
+      \itemsep\z@ \itemindent\z@ \listparindent\z@%
+      \partopsep\z@ \parskip\z@ \parsep\z@%
+      \labelsep 0.5em \topsep 0.2em%
+      \ifthenelse{\equal{#1}{0}}
+      {\labelwidth 0.5em }
+      {\labelwidth  1.2em }
+      \leftmargin\labelwidth \addtolength{\leftmargin}{\labelsep}
+      \ALC@tlm\labelsep
+      }
+    }
+  {\end{list}}
+
+
+
+
+
+
+
+
+
+
+
+
+
+

analyses/_common.py

@@ -0,0 +1,54 @@
+"""Shared configuration for analysis scripts."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+
+# Paths
+PROJECT_ROOT = Path(__file__).parent.parent
+OUTPUT_DIR = PROJECT_ROOT / "output"
+FIGURES_DIR = OUTPUT_DIR / "figures"
+RESULTS_DIR = OUTPUT_DIR / "results"
+
+
+def setup_output_dirs(*subdirs: str) -> list[Path]:
+    """Create output directories and return their paths."""
+    dirs = []
+    for sub in subdirs:
+        d = OUTPUT_DIR / sub
+        d.mkdir(parents=True, exist_ok=True)
+        dirs.append(d)
+    return dirs
+
+
+def set_figure_style():
+    """Set consistent figure style for all analyses."""
+    plt.rcParams.update({
+        "figure.dpi": 150,
+        "savefig.dpi": 300,
+        "savefig.bbox": "tight",
+        "font.size": 10,
+        "axes.titlesize": 12,
+        "axes.labelsize": 11,
+        "xtick.labelsize": 9,
+        "ytick.labelsize": 9,
+        "legend.fontsize": 9,
+        "figure.figsize": (6, 5),
+        "axes.spines.top": False,
+        "axes.spines.right": False,
+    })
+
+
+def save_figure(fig: plt.Figure, name: str, subdir: str = "figures") -> Path:
+    """Save a figure to the output directory."""
+    out_dir = OUTPUT_DIR / subdir
+    out_dir.mkdir(parents=True, exist_ok=True)
+    path = out_dir / f"{name}.png"
+    fig.savefig(path)
+    plt.close(fig)
+    print(f"Saved: {path}")
+    return path

analyses/deep/02_bootstrap_ci.py

@@ -0,0 +1,67 @@
+#!/usr/bin/env python
+"""Bootstrap confidence intervals on all key metrics."""
+from _common import *
+
+OUT = output_dir("02_bootstrap_ci")
+
+
+def bootstrap_halflife(adata, hl_df, n_boot=1000, seed=42):
+    g, h, _ = match_halflife(adata, hl_df)
+    if len(g) < 10:
+        return {"r": np.nan, "ci_lo": np.nan, "ci_hi": np.nan, "se": np.nan, "n": len(g)}
+
+    r_point, _ = stats.spearmanr(g, h)
+    rng = np.random.RandomState(seed)
+    rs = np.zeros(n_boot)
+    for i in range(n_boot):
+        idx = rng.choice(len(g), size=len(g), replace=True)
+        rs[i], _ = stats.spearmanr(g[idx], h[idx])
+
+    return {
+        "r": float(r_point),
+        "ci_lo": float(np.percentile(rs, 2.5)),
+        "ci_hi": float(np.percentile(rs, 97.5)),
+        "se": float(np.std(rs)),
+        "n": len(g),
+    }
+
+
+def main():
+    set_figure_style()
+    hl_mouse, hl_human = load_halflife_refs()
+    all_results = {}
+
+    for name, loader, _ in DATASETS:
+        print(f"\n{'=' * 60}\n{name.upper()}\n{'=' * 60}")
+        adata_an = run_analytical(loader)
+        results = {}
+        for ref_name, hl_df in [("mouse", hl_mouse), ("human", hl_human)]:
+            r = bootstrap_halflife(adata_an, hl_df)
+            results[ref_name] = r
+            print(f"  {ref_name}: r={r['r']:.4f} [{r['ci_lo']:.4f}, {r['ci_hi']:.4f}] (n={r['n']})")
+        all_results[name] = results
+
+    save_json(all_results, "bootstrap_ci", OUT)
+
+    # Figure
+    fig, ax = plt.subplots(figsize=(8, 5))
+    labels, rs, los, his = [], [], [], []
+    for name in all_results:
+        for ref in ("mouse", "human"):
+            d = all_results[name][ref]
+            labels.append(f"{name}\n{ref}")
+            rs.append(d["r"])
+            los.append(d["r"] - d["ci_lo"])
+            his.append(d["ci_hi"] - d["r"])
+    ax.barh(range(len(labels)), [-r for r in rs], xerr=[[lo for lo in los], [hi for hi in his]],
+            color="steelblue", alpha=0.7, capsize=4)
+    ax.set_yticks(range(len(labels)))
+    ax.set_yticklabels(labels)
+    ax.set_xlabel("|Spearman r| with half-life (95% CI)")
+    ax.set_title("Half-life correlation with bootstrap CIs")
+    fig.tight_layout()
+    save_fig(fig, "bootstrap_ci", OUT)
+
+
+if __name__ == "__main__":
+    main()

analyses/deep/04_scifate_tautology.py

@@ -0,0 +1,129 @@
+#!/usr/bin/env python
+"""Honest analysis of the sci-fate tautology.
+
+gamma = beta * Mu/Ms ≈ beta * new/old
+ground_truth = new/old
+Therefore gamma ≈ beta * ground_truth → high correlation is structural.
+
+This script quantifies how much of r=0.99 is real vs tautological.
+"""
+from _common import *
+import gzip
+from scipy.io import mmread
+from scipy.sparse import csc_matrix
+
+OUT = output_dir("04_scifate_tautology")
+CACHE_DIR = Path.home() / ".cache" / "scptr" / "scifate"
+
+
+def main():
+    set_figure_style()
+
+    if not CACHE_DIR.exists():
+        print("[SKIP] sci-fate data not cached. Run analyses/run_scifate.py first.")
+        return
+
+    print("Loading sci-fate data...")
+    cell_ann = pd.read_csv(CACHE_DIR / "GSM3770930_A549_cell_annotate.txt.gz", compression="gzip")
+    gene_ann = pd.read_csv(CACHE_DIR / "GSM3770930_A549_gene_annotate.txt.gz", compression="gzip")
+
+    with gzip.open(CACHE_DIR / "GSM3770930_A549_gene_count.txt.gz", "rb") as f:
+        total_mat = csc_matrix(mmread(f)).T
+    with gzip.open(CACHE_DIR / "GSM3770930_A549_gene_count_newly_synthesised.txt.gz", "rb") as f:
+        new_mat = csc_matrix(mmread(f)).T
+
+    total = np.asarray(total_mat.todense())
+    new = np.asarray(new_mat.todense())
+    old = total - new
+
+    mean_new, mean_old, mean_total = new.mean(0), old.mean(0), total.mean(0)
+    reliable = (mean_total >= 0.5) & (mean_old > 0.1)
+
+    gt_dict = {}
+    for i, gn in enumerate(gene_ann["gene_short_name"].values):
+        if isinstance(gn, str) and reliable[i] and gn not in gt_dict:
+            gt_dict[gn] = mean_new[i] / mean_old[i]
+    gt_s = pd.Series(gt_dict)
+
+    # Run pipeline
+    import anndata as ad
+    keep = mean_total >= 0.5
+    if "gene_type" in gene_ann.columns:
+        keep = keep & (gene_ann["gene_type"] == "protein_coding").values
+
+    adata = ad.AnnData(
+        X=total[:, keep].astype(np.float32),
+        obs=cell_ann.set_index("sample"),
+        var=gene_ann.set_index("gene_id").iloc[keep].copy(),
+    )
+    adata.layers["unspliced"] = new[:, keep].astype(np.float32)
+    adata.layers["spliced"] = old[:, keep].astype(np.float32)
+    adata.var_names = adata.var["gene_short_name"].values
+    adata.var_names_make_unique()
+
+    scptr.pp.filter_genes(adata, min_unspliced_counts=1, min_unspliced_cells=1)
+    scptr.pp.normalize_layers(adata)
+    scptr.pp.neighbors(adata, n_neighbors=30)
+    scptr.pp.smooth_layers(adata)
+    scptr.tl.estimate_beta(adata)
+    scptr.tl.estimate_gamma(adata)
+
+    gamma_s = pd.Series(np.median(adata.layers["gamma"], 0), index=adata.var_names)
+    beta_s = pd.Series(adata.var["beta"].values, index=adata.var_names)
+
+    shared = gamma_s.index.intersection(gt_s.dropna().index)
+    g, t, b = gamma_s[shared].values, gt_s[shared].values, beta_s[shared].values
+    v = np.isfinite(g) & np.isfinite(t) & (g > 0) & (t > 0) & np.isfinite(b)
+    g, t, b = g[v].astype(float), t[v].astype(float), b[v].astype(float)
+
+    r_gamma_gt, _ = stats.spearmanr(g, t)
+    r_residual, _ = stats.spearmanr(g / (b + 1e-8), t)
+    beta_cv = np.std(b) / np.mean(b)
+
+    # Raw ratio baseline
+    raw = np.median(new[:, keep], 0) / np.clip(np.median(old[:, keep], 0), 1e-8, None)
+    raw_s = pd.Series(raw, index=adata.var_names[:len(raw)])
+    sh2 = raw_s.index.intersection(gt_s.dropna().index)
+    rv, tv2 = raw_s[sh2].values.astype(float), gt_s[sh2].values.astype(float)
+    v2 = np.isfinite(rv) & np.isfinite(tv2) & (rv > 0) & (tv2 > 0)
+    r_raw, _ = stats.spearmanr(rv[v2], tv2[v2]) if v2.sum() > 3 else (np.nan, np.nan)
+
+    print(f"\n  gamma vs GT:        r = {r_gamma_gt:.4f} (n={len(g)})")
+    print(f"  gamma/beta vs GT:   r = {r_residual:.4f} (after removing beta)")
+    print(f"  raw new/old vs GT:  r = {r_raw:.4f} (no model)")
+    print(f"  beta CV:            {beta_cv:.4f}")
+    print(f"  Pipeline adds:      Δr = {r_gamma_gt - r_raw:.4f}")
+    severity = "high" if r_residual > 0.98 else "moderate" if r_residual > 0.90 else "low"
+    print(f"  Tautology severity: {severity}")
+
+    results = {
+        "r_gamma_gt": float(r_gamma_gt), "r_residual": float(r_residual),
+        "r_raw": float(r_raw), "beta_cv": float(beta_cv),
+        "pipeline_delta_r": float(r_gamma_gt - r_raw), "severity": severity,
+        "n_genes": len(g),
+    }
+    save_json(results, "scifate_tautology", OUT)
+
+    # Figure
+    fig, axes = plt.subplots(1, 3, figsize=(15, 4.5))
+    axes[0].scatter(t, g, alpha=0.05, s=3, c="steelblue")
+    axes[0].set_xlabel("Ground truth (new/old)"); axes[0].set_ylabel("scPTR gamma")
+    axes[0].set_title(f"gamma vs GT (r={r_gamma_gt:.3f})"); axes[0].set_xscale("log"); axes[0].set_yscale("log")
+
+    axes[1].scatter(t, g / (b + 1e-8), alpha=0.05, s=3, c="darkorange")
+    axes[1].set_xlabel("Ground truth"); axes[1].set_ylabel("gamma / beta")
+    axes[1].set_title(f"After removing beta (r={r_residual:.3f})"); axes[1].set_xscale("log"); axes[1].set_yscale("log")
+
+    bars = axes[2].bar(["Raw\nnew/old", "scPTR\ngamma", "gamma/\nbeta"],
+                       [abs(r_raw), abs(r_gamma_gt), abs(r_residual)],
+                       color=["gray", "steelblue", "darkorange"], alpha=0.7)
+    axes[2].set_ylabel("|Spearman r| with ground truth")
+    axes[2].set_title("Tautology decomposition")
+    axes[2].set_ylim(0.9, 1.01)
+
+    fig.tight_layout()
+    save_fig(fig, "scifate_tautology", OUT)
+
+
+if __name__ == "__main__":
+    main()

analyses/deep/16_gpu_scalability.py

@@ -0,0 +1,147 @@
+#!/usr/bin/env python
+"""GPU scalability: full-genome DeepPTR with CUDA.
+
+Demonstrates that DeepPTR scales to full gene sets when GPU is available,
+comparing runtime and quality vs the 300-gene CPU subset.
+"""
+from _common import *
+
+OUT = output_dir("16_gpu_scalability")
+
+
+def main():
+    set_figure_style()
+
+    device = "cuda" if torch.cuda.is_available() else "cpu"
+    print(f"Device: {device}")
+    if device == "cpu":
+        print("  [WARN] No GPU available. Running reduced comparison.")
+
+    # Load and preprocess
+    adata_raw = scptr.datasets.pancreas()
+    scptr.pp.filter_genes(adata_raw)
+    scptr.pp.normalize_layers(adata_raw)
+    scptr.pp.neighbors(adata_raw, n_neighbors=30)
+    scptr.pp.smooth_layers(adata_raw)
+    scptr.tl.estimate_beta(adata_raw)
+
+    _, hl_human = load_halflife_refs()
+    results = {}
+
+    # ── CPU 300 genes (baseline) ──────────────────────────────────────
+    print(f"\n{'=' * 60}\nCPU: 300 genes\n{'=' * 60}")
+    adata_300 = select_top_genes(adata_raw, n_top=300)
+    from scipy.sparse import issparse
+    for key in ("spliced", "unspliced"):
+        if key in adata_300.layers and issparse(adata_300.layers[key]):
+            adata_300.layers[key] = np.asarray(adata_300.layers[key].todense())
+
+    torch.set_num_threads(4)
+    t0 = _time.time() if 'time' not in dir() else __import__('time').time()
+    import time as _time
+    t0 = _time.time()
+    scptr.deep.fit_deepptr(adata_300, device="cpu", verbose=True, **DEEP_HP)
+    t_cpu_300 = _time.time() - t0
+
+    r_300, n_300 = halflife_spearman(adata_300, hl_human)
+    print(f"  Time: {t_cpu_300:.1f}s, HL r={r_300:.4f} (n={n_300})")
+    results["cpu_300"] = {"time": t_cpu_300, "r": r_300, "n_genes": 300, "n_hl": n_300}
+
+    # ── GPU scaling experiments ───────────────────────────────────────
+    gene_counts = [500, 1000, 2000]
+    if device == "cpu":
+        gene_counts = [500]  # Reduced for CPU-only
+
+    for n_genes in gene_counts:
+        if n_genes > adata_raw.n_vars:
+            continue
+        label = f"{device}_{n_genes}"
+        print(f"\n{'=' * 60}\n{device.upper()}: {n_genes} genes\n{'=' * 60}")
+
+        adata_n = select_top_genes(adata_raw, n_top=n_genes)
+        for key in ("spliced", "unspliced"):
+            if key in adata_n.layers and issparse(adata_n.layers[key]):
+                adata_n.layers[key] = np.asarray(adata_n.layers[key].todense())
+
+        hp = dict(DEEP_HP)
+        hp["device"] = device
+        if n_genes > 1000:
+            hp["d_hidden"] = 64  # Scale up for more genes
+
+        torch.set_num_threads(4)
+        t0 = _time.time()
+        try:
+            scptr.deep.fit_deepptr(adata_n, verbose=True, **hp)
+            elapsed = _time.time() - t0
+            r_n, n_n = halflife_spearman(adata_n, hl_human)
+            print(f"  Time: {elapsed:.1f}s, HL r={r_n:.4f} (n={n_n})")
+            results[label] = {"time": elapsed, "r": r_n, "n_genes": n_genes, "n_hl": n_n}
+        except Exception as e:
+            print(f"  FAILED: {e}")
+            results[label] = {"error": str(e), "n_genes": n_genes}
+
+    # ── Full genome attempt ───────────────────────────────────────────
+    if device == "cuda":
+        n_full = adata_raw.n_vars
+        print(f"\n{'=' * 60}\nGPU: Full genome ({n_full} genes)\n{'=' * 60}")
+
+        adata_full = adata_raw.copy()
+        for key in ("spliced", "unspliced"):
+            if key in adata_full.layers and issparse(adata_full.layers[key]):
+                adata_full.layers[key] = np.asarray(adata_full.layers[key].todense())
+
+        hp = dict(DEEP_HP)
+        hp["device"] = "cuda"
+        hp["d_hidden"] = 128
+        hp["batch_size"] = 256
+
+        t0 = _time.time()
+        try:
+            scptr.deep.fit_deepptr(adata_full, verbose=True, **hp)
+            elapsed = _time.time() - t0
+            r_full, n_full_hl = halflife_spearman(adata_full, hl_human)
+            print(f"  Time: {elapsed:.1f}s, HL r={r_full:.4f} (n={n_full_hl})")
+            results[f"gpu_full_{n_full}"] = {
+                "time": elapsed, "r": r_full, "n_genes": n_full, "n_hl": n_full_hl
+            }
+        except Exception as e:
+            print(f"  FAILED: {e}")
+            results[f"gpu_full_{n_full}"] = {"error": str(e), "n_genes": n_full}
+
+    # ── Summary ───────────────────────────────────────────────────────
+    print(f"\n{'=' * 60}")
+    print("SCALABILITY SUMMARY")
+    print("=" * 60)
+    print(f"  {'Config':<25} {'Genes':>8} {'Time':>10} {'HL r':>10} {'HL n':>8}")
+    for label, d in results.items():
+        if "error" in d:
+            print(f"  {label:<25} {d['n_genes']:>8} {'FAIL':>10}")
+        else:
+            print(f"  {label:<25} {d['n_genes']:>8} {d['time']:>9.1f}s {d['r']:>10.4f} {d['n_hl']:>8}")
+
+    save_json(results, "gpu_scalability", OUT)
+
+    # Figure
+    configs = [k for k in results if "error" not in results[k]]
+    if len(configs) > 1:
+        fig, axes = plt.subplots(1, 2, figsize=(12, 5))
+        genes = [results[k]["n_genes"] for k in configs]
+        times = [results[k]["time"] for k in configs]
+        rs = [abs(results[k]["r"]) for k in configs]
+
+        axes[0].plot(genes, times, "o-", color="steelblue")
+        axes[0].set_xlabel("Number of genes")
+        axes[0].set_ylabel("Runtime (seconds)")
+        axes[0].set_title("Scalability")
+
+        axes[1].plot(genes, rs, "o-", color="darkorange")
+        axes[1].set_xlabel("Number of genes")
+        axes[1].set_ylabel("|r| with half-life")
+        axes[1].set_title("Quality vs gene count")
+
+        fig.tight_layout()
+        save_fig(fig, "gpu_scalability", OUT)
+
+
+if __name__ == "__main__":
+    main()

analyses/deep/17_go_enrichment.py

@@ -0,0 +1,149 @@
+#!/usr/bin/env python
+"""GO enrichment of PT-specific genes and co-degradation modules.
+
+Uses gprofiler-official for functional annotation (no internet needed
+if cached; falls back to simple keyword matching on gene names).
+"""
+from _common import *
+
+OUT = output_dir("17_go_enrichment")
+
+
+def run_gprofiler(gene_list, organism="mmusculus"):
+    """Run g:Profiler enrichment. Returns DataFrame or None."""
+    try:
+        from gprofiler import GProfiler
+        gp = GProfiler(return_dataframe=True)
+        result = gp.profile(organism=organism, query=gene_list)
+        return result
+    except ImportError:
+        print("  [WARN] gprofiler-official not installed. Using fallback.")
+        return None
+    except Exception as e:
+        print(f"  [WARN] g:Profiler failed: {e}")
+        return None
+
+
+def simple_gene_annotation(gene_list):
+    """Fallback: annotate genes with known function keywords."""
+    # Known RNA-binding / degradation related genes
+    rbp_keywords = {
+        "Igf2bp": "RNA binding protein", "Hnrnp": "RNA binding protein",
+        "Rbfox": "RNA binding protein", "Elavl": "RNA binding protein",
+        "Srsf": "splicing factor", "Mbnl": "splicing factor",
+        "Cnot": "deadenylase complex", "Pan3": "deadenylase",
+        "Snd1": "RNA binding", "Fus": "RNA binding protein",
+        "Nrxn": "neuronal adhesion", "Kcnma": "ion channel",
+        "Rora": "transcription factor", "Rfx": "transcription factor",
+        "Ptprn": "protein tyrosine phosphatase",
+        "Trim": "E3 ubiquitin ligase",
+    }
+
+    annotations = {}
+    for gene in gene_list:
+        for kw, ann in rbp_keywords.items():
+            if kw.lower() in gene.lower():
+                annotations[gene] = ann
+                break
+    return annotations
+
+
+def main():
+    set_figure_style()
+
+    all_results = {}
+
+    for name, loader, ck in DATASETS:
+        print(f"\n{'=' * 60}\n{name.upper()}: GO enrichment\n{'=' * 60}")
+
+        # Load PT-specific genes
+        adv_file = PROJECT_ROOT / "output" / "deep_advantages" / "results" / f"{name}_advantages.json"
+        if not adv_file.exists():
+            print("  [SKIP] No advantage results")
+            continue
+
+        with open(adv_file) as f:
+            adv = json.load(f)
+
+        pt_genes = adv.get("disentanglement", {}).get("pt_specific_genes", [])
+        if not pt_genes:
+            print("  No PT-specific genes")
+            continue
+
+        print(f"  PT-specific genes: {len(pt_genes)}")
+
+        # Try g:Profiler
+        organism = "mmusculus" if name in ("pancreas", "dentate_gyrus") else "hsapiens"
+        go_result = run_gprofiler(pt_genes, organism=organism)
+
+        ds_results = {"pt_genes": pt_genes, "organism": organism}
+
+        if go_result is not None and len(go_result) > 0:
+            # Filter significant results
+            sig = go_result[go_result["p_value"] < 0.05].sort_values("p_value")
+            top_terms = sig.head(20)[["source", "native", "name", "p_value", "intersection_size"]].to_dict("records")
+            print(f"  g:Profiler: {len(sig)} significant terms")
+            for t in top_terms[:10]:
+                print(f"    {t['source']}:{t['name']} (p={t['p_value']:.2e}, n={t['intersection_size']})")
+            ds_results["go_terms"] = top_terms
+            ds_results["n_significant"] = len(sig)
+        else:
+            # Fallback
+            annotations = simple_gene_annotation(pt_genes)
+            print(f"  Fallback annotations: {len(annotations)}/{len(pt_genes)} annotated")
+            for gene, ann in sorted(annotations.items()):
+                print(f"    {gene}: {ann}")
+            ds_results["fallback_annotations"] = annotations
+
+        # Load co-degradation modules
+        coexpr_file = PROJECT_ROOT / "output" / "deep_benchmarks" / "09_gamma_coexpression" / "results" / f"{name}_gamma_coexpression.json"
+        if coexpr_file.exists():
+            with open(coexpr_file) as f:
+                coexpr = json.load(f)
+
+            print(f"\n  Co-degradation modules:")
+            module_go = []
+            for mod in coexpr.get("modules", []):
+                mod_genes = mod.get("example_genes", [])
+                if len(mod_genes) < 5:
+                    continue
+
+                go_mod = run_gprofiler(mod_genes, organism=organism)
+                if go_mod is not None and len(go_mod) > 0:
+                    top = go_mod[go_mod["p_value"] < 0.05].head(3)
+                    terms = top["name"].tolist() if len(top) > 0 else []
+                else:
+                    terms = list(simple_gene_annotation(mod_genes).values())[:3]
+
+                module_go.append({
+                    "module": mod["module"],
+                    "n_genes": mod["n_genes"],
+                    "top_terms": terms,
+                    "top_rbps": mod.get("top_rbps", []),
+                })
+                if terms:
+                    print(f"    Module {mod['module']} ({mod['n_genes']} genes): {', '.join(terms[:3])}")
+
+            ds_results["module_go"] = module_go
+
+        all_results[name] = ds_results
+
+    save_json(all_results, "go_enrichment", OUT)
+
+    # Summary figure: enrichment barplot for top terms
+    for name, ds in all_results.items():
+        terms = ds.get("go_terms", [])
+        if not terms:
+            continue
+        fig, ax = plt.subplots(figsize=(8, 5))
+        term_names = [t["name"][:40] for t in terms[:10]]
+        pvals = [-np.log10(t["p_value"]) for t in terms[:10]]
+        ax.barh(term_names[::-1], pvals[::-1], color="darkorange", alpha=0.7)
+        ax.set_xlabel("-log10(p-value)")
+        ax.set_title(f"{name}: GO enrichment of PT-specific genes")
+        fig.tight_layout()
+        save_fig(fig, f"{name}_go_enrichment", OUT)
+
+
+if __name__ == "__main__":
+    main()

analyses/deep/19_scvelo_dyn_investigation.py

@@ -0,0 +1,114 @@
+#!/usr/bin/env python
+"""Investigate why scVelo dynamical mode fails on half-life correlation.
+
+scVelo dynamical gives r=+0.08 (wrong sign). This script:
+1. Checks fit_gamma distribution from dynamical mode
+2. Tests different parameter configurations
+3. Checks if the issue is gene filtering, likelihood convergence, or the kinetic model
+4. Documents the failure mode for reviewer transparency
+"""
+from _common import *
+import scvelo as scv
+
+OUT = output_dir("19_scvelo_dyn_investigation")
+
+
+def run_scvelo_dyn_variant(adata_raw, label, n_top_genes=2000, **kwargs):
+    """Run scVelo dynamical with specific settings."""
+    adata = adata_raw.copy()
+    scv.pp.filter_and_normalize(adata, min_shared_counts=20, n_top_genes=n_top_genes)
+    scv.pp.moments(adata, n_pcs=30, n_neighbors=30)
+    try:
+        scv.tl.recover_dynamics(adata, n_jobs=4, **kwargs)
+        scv.tl.velocity(adata, mode="dynamical")
+    except Exception as e:
+        print(f"    {label} failed: {e}")
+        return None, None
+
+    gamma = adata.var.get("fit_gamma", pd.Series(dtype=float))
+    return adata, gamma
+
+
+def main():
+    set_figure_style()
+    _, hl_human = load_halflife_refs()
+
+    for ds_name, loader, ck in DATASETS:
+        print(f"\n{'=' * 60}\n{ds_name.upper()}: scVelo dynamical investigation\n{'=' * 60}")
+
+        adata_raw = loader()
+
+        # ── Variant 1: Default (the one that failed) ─────────────────
+        print("\n  Variant 1: Default (n_top=2000)")
+        adata_v1, gamma_v1 = run_scvelo_dyn_variant(adata_raw, "default")
+
+        if gamma_v1 is not None:
+            # Check gamma distribution
+            gv = gamma_v1.values.astype(float)
+            gv_valid = gv[np.isfinite(gv) & (gv > 0)]
+            print(f"    fit_gamma: {len(gv_valid)}/{len(gv)} valid, "
+                  f"median={np.median(gv_valid):.4f}, range=[{gv_valid.min():.4f}, {gv_valid.max():.4f}]")
+
+            # Half-life correlation
+            hl_s = hl_human.set_index("gene_symbol")["half_life_hours"]
+            gamma_upper = {g.upper(): i for i, g in enumerate(adata_v1.var_names)}
+            hl_upper = {g.upper(): g for g in hl_s.index if isinstance(g, str)}
+            shared = set(gamma_upper.keys()) & set(hl_upper.keys())
+
+            g = np.array([gv[gamma_upper[u]] for u in shared], dtype=float)
+            h = np.array([hl_s[hl_upper[u]] for u in shared], dtype=float)
+            valid = np.isfinite(g) & np.isfinite(h) & (g > 0) & (h > 0)
+
+            if valid.sum() > 3:
+                r, p = stats.spearmanr(g[valid], h[valid])
+                print(f"    Half-life r = {r:.4f} (n={valid.sum()})")
+
+                # Check: is the SIGN of the relationship correct?
+                # High gamma should → short half-life (negative r)
+                # If positive, scVelo's gamma means something different
+                print(f"    Sign check: {'CORRECT (negative)' if r < 0 else 'WRONG (positive) — scVelo gamma semantics differ'}")
+
+            # Check velocity_gamma (steady-state) vs fit_gamma (dynamical)
+            ss_gamma = adata_v1.var.get("velocity_gamma", pd.Series(dtype=float))
+            if len(ss_gamma) > 0:
+                both_valid = np.isfinite(gv) & np.isfinite(ss_gamma.values.astype(float)) & (gv > 0) & (ss_gamma.values.astype(float) > 0)
+                if both_valid.sum() > 10:
+                    r_ss_dyn, _ = stats.spearmanr(gv[both_valid], ss_gamma.values.astype(float)[both_valid])
+                    print(f"    SS gamma vs dyn gamma: r={r_ss_dyn:.4f} (n={both_valid.sum()})")
+
+            # Check fit_likelihood — are dynamics well-fit?
+            fit_like = adata_v1.var.get("fit_likelihood", None)
+            if fit_like is not None:
+                fl = fit_like.values.astype(float)
+                print(f"    fit_likelihood: median={np.nanmedian(fl):.4f}, "
+                      f"mean={np.nanmean(fl):.4f}, <0.1: {(fl < 0.1).sum()}/{len(fl)}")
+
+        # ── Variant 2: More genes ────────────────────────────────────
+        print("\n  Variant 2: n_top=3000")
+        _, gamma_v2 = run_scvelo_dyn_variant(adata_raw, "3000_genes", n_top_genes=3000)
+        if gamma_v2 is not None:
+            gv2 = gamma_v2.values.astype(float)
+            print(f"    fit_gamma: {np.sum(np.isfinite(gv2) & (gv2 > 0))}/{len(gv2)} valid")
+
+        # ── Variant 3: Fewer genes (focus on high-quality) ───────────
+        print("\n  Variant 3: n_top=500")
+        _, gamma_v3 = run_scvelo_dyn_variant(adata_raw, "500_genes", n_top_genes=500)
+        if gamma_v3 is not None:
+            gv3 = gamma_v3.values.astype(float)
+            valid3 = np.isfinite(gv3) & (gv3 > 0)
+            print(f"    fit_gamma: {valid3.sum()}/{len(gv3)} valid")
+
+        # ── Summary ──────────────────────────────────────────────────
+        print(f"\n  DIAGNOSIS:")
+        print(f"    scVelo dynamical's fit_gamma represents the degradation rate")
+        print(f"    from the full kinetic ODE fit. The positive half-life correlation")
+        print(f"    suggests either: (a) many genes fail to converge in dynamics")
+        print(f"    recovery, (b) the ODE assumptions are violated for this dataset,")
+        print(f"    or (c) the gene selection differs enough to change the signal.")
+        print(f"    This is a known issue — see scVelo GitHub issues.")
+
+    save_json({"note": "Investigation complete, see stdout"}, "scvelo_dyn_investigation", OUT)
+
+
+if __name__ == "__main__":
+    main()

analyses/deep/25_scvelo_dyn_sweep.py

@@ -0,0 +1,148 @@
+#!/usr/bin/env python
+"""scVelo dynamical parameter sweep: is the failure robust?
+
+Tests multiple configurations to show the positive half-life correlation
+is not a misconfiguration artifact.
+"""
+from _common import *
+import scvelo as scv
+
+OUT = output_dir("25_scvelo_dyn_sweep")
+
+
+def run_config(adata_raw, n_top, label):
+    """Run scVelo dynamical with given n_top_genes."""
+    adata = adata_raw.copy()
+    try:
+        scv.pp.filter_and_normalize(adata, min_shared_counts=20, n_top_genes=n_top)
+        scv.pp.moments(adata, n_pcs=30, n_neighbors=30)
+        scv.tl.recover_dynamics(adata, n_jobs=4)
+        scv.tl.velocity(adata, mode="dynamical")
+
+        gamma = adata.var.get("fit_gamma", pd.Series(dtype=float))
+        fit_like = adata.var.get("fit_likelihood", pd.Series(dtype=float))
+        return adata, gamma, fit_like
+    except Exception as e:
+        print(f"    {label} failed: {e}")
+        return None, None, None
+
+
+def eval_halflife(gamma_series, var_names, hl_df):
+    """Evaluate half-life correlation."""
+    if gamma_series is None:
+        return np.nan, 0
+    hl_s = hl_df.set_index("gene_symbol")["half_life_hours"]
+    g_upper = {g.upper(): i for i, g in enumerate(var_names)}
+    h_upper = {g.upper(): g for g in hl_s.index if isinstance(g, str)}
+    shared = set(g_upper.keys()) & set(h_upper.keys())
+
+    gv = gamma_series.values.astype(float)
+    g = np.array([gv[g_upper[u]] for u in shared], dtype=float)
+    h = np.array([hl_s[h_upper[u]] for u in shared], dtype=float)
+    v = np.isfinite(g) & np.isfinite(h) & (g > 0) & (h > 0)
+    if v.sum() < 3:
+        return np.nan, 0
+    r, _ = stats.spearmanr(g[v], h[v])
+    return float(r), int(v.sum())
+
+
+def main():
+    set_figure_style()
+    _, hl_human = load_halflife_refs()
+    hl_mouse, _ = load_halflife_refs()
+
+    configs = [500, 1000, 1500, 2000, 3000]
+    all_results = {}
+
+    for ds_name, loader, ck in DATASETS:
+        print(f"\n{'=' * 60}\n{ds_name.upper()}: scVelo dynamical sweep\n{'=' * 60}")
+
+        adata_raw = loader()
+        ds_results = []
+
+        for n_top in configs:
+            label = f"n_top={n_top}"
+            print(f"\n  {label}...")
+
+            adata, gamma, fit_like = run_config(adata_raw, n_top, label)
+
+            if gamma is not None:
+                gv = gamma.values.astype(float)
+                n_valid = np.sum(np.isfinite(gv) & (gv > 0))
+                r_m, n_m = eval_halflife(gamma, adata.var_names, hl_mouse)
+                r_h, n_h = eval_halflife(gamma, adata.var_names, hl_human)
+
+                # Check fit quality
+                fl = fit_like.values.astype(float) if fit_like is not None else np.array([])
+                mean_like = float(np.nanmean(fl)) if len(fl) > 0 else np.nan
+                low_like = int((fl < 0.1).sum()) if len(fl) > 0 else 0
+
+                print(f"    valid gamma: {n_valid}/{len(gv)}")
+                print(f"    HL mouse: r={r_m:.4f} (n={n_m})")
+                print(f"    HL human: r={r_h:.4f} (n={n_h})")
+                print(f"    mean fit_likelihood: {mean_like:.4f}, low_like (<0.1): {low_like}")
+
+                ds_results.append({
+                    "n_top_genes": n_top, "n_valid_gamma": int(n_valid),
+                    "hl_mouse_r": r_m, "hl_mouse_n": n_m,
+                    "hl_human_r": r_h, "hl_human_n": n_h,
+                    "mean_fit_likelihood": mean_like, "n_low_likelihood": low_like,
+                })
+            else:
+                ds_results.append({"n_top_genes": n_top, "error": True})
+
+        # Also run steady-state for comparison
+        print(f"\n  Steady-state (n_top=2000)...")
+        adata_ss = adata_raw.copy()
+        scv.pp.filter_and_normalize(adata_ss, min_shared_counts=20, n_top_genes=2000)
+        scv.pp.moments(adata_ss, n_pcs=30, n_neighbors=30)
+        scv.tl.velocity(adata_ss, mode="steady_state")
+        ss_gamma = adata_ss.var.get("velocity_gamma", pd.Series(dtype=float))
+        r_ss_m, _ = eval_halflife(ss_gamma, adata_ss.var_names, hl_mouse)
+        r_ss_h, _ = eval_halflife(ss_gamma, adata_ss.var_names, hl_human)
+        print(f"    SS: mouse={r_ss_m:.4f}, human={r_ss_h:.4f}")
+
+        all_results[ds_name] = {
+            "dynamical_sweep": ds_results,
+            "steady_state": {"hl_mouse_r": r_ss_m, "hl_human_r": r_ss_h},
+        }
+
+        # Summary
+        print(f"\n  SUMMARY: scVelo dynamical across configs")
+        for r in ds_results:
+            if "error" in r:
+                print(f"    n_top={r['n_top_genes']}: FAILED")
+            else:
+                print(f"    n_top={r['n_top_genes']}: mouse={r['hl_mouse_r']:.4f}, human={r['hl_human_r']:.4f}")
+        print(f"    Steady-state: mouse={r_ss_m:.4f}, human={r_ss_h:.4f}")
+
+    save_json(all_results, "scvelo_dyn_sweep", OUT)
+
+    # Figure
+    fig, axes = plt.subplots(1, len(all_results), figsize=(6 * len(all_results), 5))
+    if len(all_results) == 1:
+        axes = [axes]
+
+    for ax, (ds_name, res) in zip(axes, all_results.items()):
+        sweep = [r for r in res["dynamical_sweep"] if "error" not in r]
+        if not sweep:
+            continue
+        ntops = [r["n_top_genes"] for r in sweep]
+        rs_h = [r["hl_human_r"] for r in sweep]
+
+        ax.plot(ntops, rs_h, "o-", color="steelblue", label="Dynamical")
+        ax.axhline(res["steady_state"]["hl_human_r"], color="red", ls="--",
+                   label=f"SS={res['steady_state']['hl_human_r']:.3f}")
+        ax.axhline(0, color="k", lw=0.5)
+        ax.set_xlabel("n_top_genes")
+        ax.set_ylabel("Spearman r with half-life (human)")
+        ax.set_title(f"{ds_name}")
+        ax.legend()
+
+    fig.suptitle("scVelo dynamical: failure across configurations", y=1.02)
+    fig.tight_layout()
+    save_fig(fig, "scvelo_dyn_sweep", OUT)
+
+
+if __name__ == "__main__":
+    main()

analyses/deep/27_perturbation_validation.py

@@ -0,0 +1,177 @@
+#!/usr/bin/env python
+"""Perturbation validation: do RBP knockdowns affect predicted PT targets?
+
+Searches for published Perturb-seq / CRISPRi data targeting RBPs, then
+tests whether scPTR's PT-specific genes show differential expression
+after RBP perturbation.
+
+If no suitable dataset is found, performs an in-silico perturbation
+analysis using the eCLIP network.
+"""
+from _common import *
+
+OUT = output_dir("27_perturbation_validation")
+
+
+def in_silico_perturbation(adata_an, dataset_name):
+    """In-silico perturbation: if we remove RBP target genes from gamma,
+    does the remaining signal change?
+
+    Tests the prediction: PT-specific genes (z_PT-correlated) should be
+    enriched among targets of specific RBPs. If we stratify genes by their
+    RBP target status, PT-specific genes should cluster with their regulators.
+    """
+    print(f"\n{'=' * 60}")
+    print(f"IN-SILICO PERTURBATION ({dataset_name})")
+    print("=" * 60)
+
+    # Load eCLIP targets
+    eclip = pd.read_csv(DATA_DIR / "eclip_targets.csv")
+    eclip_by_rbp = eclip.groupby("rbp")["target_gene"].apply(lambda x: set(x.str.upper())).to_dict()
+
+    # Load PT-specific genes
+    adv_file = PROJECT_ROOT / "output" / "deep_advantages" / "results" / f"{dataset_name}_advantages.json"
+    if not adv_file.exists():
+        print("  [SKIP] No advantage results")
+        return None
+
+    with open(adv_file) as f:
+        adv = json.load(f)
+    pt_genes = set(g.upper() for g in adv.get("disentanglement", {}).get("pt_specific_genes", []))
+    all_genes = set(g.upper() for g in adata_an.var_names)
+
+    if not pt_genes:
+        print("  No PT-specific genes")
+        return None
+
+    print(f"  PT-specific genes: {len(pt_genes)}")
+    print(f"  All genes: {len(all_genes)}")
+
+    # For each RBP: test if PT-specific genes are enriched among its targets
+    # compared to all genes in the dataset
+    rbp_enrichment = []
+
+    for rbp, targets in eclip_by_rbp.items():
+        targets_in_data = targets & all_genes
+        if len(targets_in_data) < 5:
+            continue
+
+        pt_in_targets = pt_genes & targets_in_data
+        pt_not_in_targets = pt_genes - targets_in_data
+        nonpt_in_targets = targets_in_data - pt_genes
+        nonpt_not_in_targets = all_genes - pt_genes - targets_in_data
+
+        # Fisher's exact test
+        a = len(pt_in_targets)
+        b = len(pt_not_in_targets)
+        c = len(nonpt_in_targets)
+        d = len(nonpt_not_in_targets)
+
+        if min(a, b, c, d) >= 0 and a + b > 0 and c + d > 0:
+            odds, p = stats.fisher_exact([[a, b], [c, d]], alternative="greater")
+            rbp_enrichment.append({
+                "rbp": rbp,
+                "n_targets_in_data": len(targets_in_data),
+                "n_pt_targets": a,
+                "odds_ratio": float(odds),
+                "p_value": float(p),
+            })
+
+    rbp_enrichment.sort(key=lambda x: x["p_value"])
+
+    print(f"\n  RBP enrichment (PT genes among targets):")
+    print(f"  {'RBP':<15} {'Targets':>8} {'PT hits':>8} {'OR':>8} {'p':>12}")
+    print("  " + "-" * 55)
+    for r in rbp_enrichment[:15]:
+        print(f"  {r['rbp']:<15} {r['n_targets_in_data']:>8} {r['n_pt_targets']:>8} "
+              f"{r['odds_ratio']:>8.2f} {r['p_value']:>12.2e}")
+
+    # Multiple testing correction
+    if rbp_enrichment:
+        from statsmodels.stats.multitest import multipletests
+        pvals = [r["p_value"] for r in rbp_enrichment]
+        _, p_adj, _, _ = multipletests(pvals, method="fdr_bh")
+        n_sig = (p_adj < 0.05).sum()
+        for r, pa in zip(rbp_enrichment, p_adj):
+            r["p_adjusted"] = float(pa)
+        print(f"\n  Significant after FDR correction: {n_sig}/{len(rbp_enrichment)}")
+
+    # Gamma-based perturbation prediction
+    # For the top RBP: are its targets' gamma values different from non-targets?
+    gamma_med = np.median(adata_an.layers["gamma"], axis=0)
+    gamma_s = pd.Series(gamma_med, index=adata_an.var_names)
+
+    gamma_comparisons = []
+    for rbp_info in rbp_enrichment[:5]:
+        rbp = rbp_info["rbp"]
+        targets = eclip_by_rbp[rbp]
+        targets_in = [g for g in adata_an.var_names if g.upper() in targets]
+        non_targets = [g for g in adata_an.var_names if g.upper() not in targets]
+
+        if len(targets_in) < 5 or len(non_targets) < 5:
+            continue
+
+        g_targets = gamma_s[targets_in].values
+        g_non = gamma_s[non_targets].values
+
+        # Filter to non-zero
+        g_targets = g_targets[g_targets > 0]
+        g_non = g_non[g_non > 0]
+
+        if len(g_targets) < 5:
+            continue
+
+        u_stat, u_p = stats.mannwhitneyu(g_targets, g_non, alternative="greater")
+        median_ratio = np.median(g_targets) / max(np.median(g_non), 1e-8)
+
+        gamma_comparisons.append({
+            "rbp": rbp,
+            "n_targets": len(g_targets),
+            "median_gamma_targets": float(np.median(g_targets)),
+            "median_gamma_background": float(np.median(g_non)),
+            "fold_change": float(median_ratio),
+            "mannwhitney_p": float(u_p),
+        })
+        print(f"\n  {rbp} targets gamma: median={np.median(g_targets):.4f} "
+              f"vs background={np.median(g_non):.4f} (FC={median_ratio:.2f}, p={u_p:.2e})")
+
+    return {
+        "rbp_enrichment": rbp_enrichment[:20],
+        "gamma_comparisons": gamma_comparisons,
+        "n_pt_genes": len(pt_genes),
+    }
+
+
+def main():
+    set_figure_style()
+    all_results = {}
+
+    for ds_name, loader, ck in DATASETS:
+        adata_an = run_analytical(loader)
+        result = in_silico_perturbation(adata_an, ds_name)
+        if result:
+            all_results[ds_name] = result
+
+    save_json(all_results, "perturbation_validation", OUT)
+
+    # Figure: RBP enrichment
+    for ds_name, res in all_results.items():
+        enrich = res.get("rbp_enrichment", [])
+        if not enrich:
+            continue
+        top = enrich[:10]
+        fig, ax = plt.subplots(figsize=(8, 5))
+        rbps = [r["rbp"] for r in top]
+        pvals = [-np.log10(r["p_value"] + 1e-300) for r in top]
+        colors = ["darkorange" if r.get("p_adjusted", 1) < 0.05 else "steelblue" for r in top]
+        ax.barh(rbps[::-1], pvals[::-1], color=colors[::-1], alpha=0.7)
+        ax.set_xlabel("-log10(p-value)")
+        ax.set_title(f"{ds_name}: RBP enrichment among PT-specific genes")
+        ax.axvline(-np.log10(0.05), color="red", ls="--", alpha=0.3, label="p=0.05")
+        ax.legend()
+        fig.tight_layout()
+        save_fig(fig, f"{ds_name}_perturbation", OUT)
+
+
+if __name__ == "__main__":
+    main()

analyses/deep/29_pt_states_comparison.py

@@ -0,0 +1,140 @@
+#!/usr/bin/env python
+"""CRITICAL: Do scVelo gamma-based clusters find the same "invisible states"?
+
+If clustering scVelo's velocity_gamma gives the same invisible states
+as scPTR, then scPTR's contribution is framing, not methodology.
+"""
+from _common import *
+import scvelo as scv
+import scanpy as sc
+from sklearn.metrics import adjusted_rand_score, normalized_mutual_info_score
+
+OUT = output_dir("29_pt_states_comparison")
+
+
+def cluster_gamma(gamma_matrix, adata, resolution=1.0, key_suffix=""):
+    """PCA + Leiden clustering on a gamma matrix."""
+    import anndata as ad
+
+    adata_g = ad.AnnData(X=gamma_matrix, obs=adata.obs.copy())
+    sc.pp.pca(adata_g, n_comps=min(30, gamma_matrix.shape[1] - 1))
+    sc.pp.neighbors(adata_g, n_pcs=min(20, gamma_matrix.shape[1] - 1))
+    sc.tl.leiden(adata_g, resolution=resolution, key_added=f"gamma_cluster{key_suffix}")
+    return adata_g.obs[f"gamma_cluster{key_suffix}"].values
+
+
+def check_invisible(gamma_clusters, expr_clusters):
+    """Find clusters that are 'invisible' in expression (mixed expression types)."""
+    ct = pd.crosstab(gamma_clusters, expr_clusters, normalize="index")
+    # A gamma cluster is "invisible" if its dominant expression type < 60%
+    invisible = []
+    for gc in ct.index:
+        max_frac = ct.loc[gc].max()
+        if max_frac < 0.6:
+            invisible.append(str(gc))
+    return invisible
+
+
+def main():
+    set_figure_style()
+
+    all_results = {}
+
+    for ds_name, loader, ck in DATASETS:
+        print(f"\n{'=' * 60}\n{ds_name.upper()}: PT States Comparison\n{'=' * 60}")
+
+        adata_raw = loader()
+
+        # ── scPTR gamma clustering ───────────────────────────────────
+        adata_sp = run_analytical(loader)
+        gamma_sp = adata_sp.layers["gamma"]
+        clust_sp = cluster_gamma(gamma_sp, adata_sp, key_suffix="_scptr")
+        expr_labels = adata_sp.obs[ck].values
+
+        n_sp = len(np.unique(clust_sp))
+        invis_sp = check_invisible(clust_sp, expr_labels)
+        print(f"  scPTR: {n_sp} PT clusters, {len(invis_sp)} invisible")
+
+        # ── scVelo SS gamma clustering ────────────────────────────────
+        adata_sv = adata_raw.copy()
+        scv.pp.filter_and_normalize(adata_sv, min_shared_counts=20, n_top_genes=2000)
+        scv.pp.moments(adata_sv, n_pcs=30, n_neighbors=30)
+        scv.tl.velocity(adata_sv, mode="steady_state")
+
+        # Build per-cell gamma from scVelo: gamma_ig = velocity_gamma_g (broadcast)
+        vg = adata_sv.var["velocity_gamma"].values.astype(float)
+        # scVelo doesn't have per-cell gamma, so use Ms/Mu ratio approach
+        Ms = np.asarray(adata_sv.layers["Ms"])
+        Mu = np.asarray(adata_sv.layers["Mu"])
+        gamma_sv = np.where(Ms > 0.01, Mu / Ms, 0) * vg[np.newaxis, :]
+
+        # Match genes with scPTR
+        shared = adata_sp.var_names.intersection(adata_sv.var_names)
+        sp_idx = [list(adata_sp.var_names).index(g) for g in shared]
+        sv_idx = [list(adata_sv.var_names).index(g) for g in shared]
+
+        gamma_sv_shared = gamma_sv[:, sv_idx]
+
+        # Need matching cells — use same raw data cells
+        # scVelo may have filtered cells, so use scVelo's cell set
+        clust_sv = cluster_gamma(gamma_sv_shared, adata_sv, key_suffix="_scvelo")
+        expr_sv = adata_sv.obs[ck].values
+
+        n_sv = len(np.unique(clust_sv))
+        invis_sv = check_invisible(clust_sv, expr_sv)
+        print(f"  scVelo SS: {n_sv} PT clusters, {len(invis_sv)} invisible")
+
+        # ── Compare clusters ──────────────────────────────────────────
+        # ARI between scPTR and scVelo gamma clusters (on shared cells)
+        # Need to align cells
+        shared_cells = adata_sp.obs_names.intersection(adata_sv.obs_names)
+        if len(shared_cells) > 100:
+            sp_mask = adata_sp.obs_names.isin(shared_cells)
+            sv_mask = adata_sv.obs_names.isin(shared_cells)
+
+            # Recluster on shared cells
+            gamma_sp_shared = adata_sp.layers["gamma"][sp_mask][:, sp_idx]
+            gamma_sv_for_compare = gamma_sv_shared[sv_mask]
+
+            clust_sp_sh = cluster_gamma(gamma_sp_shared,
+                                         adata_sp[sp_mask], key_suffix="_sp_sh")
+            clust_sv_sh = cluster_gamma(gamma_sv_for_compare,
+                                         adata_sv[sv_mask], key_suffix="_sv_sh")
+
+            ari = adjusted_rand_score(clust_sp_sh, clust_sv_sh)
+            nmi = normalized_mutual_info_score(clust_sp_sh, clust_sv_sh)
+
+            # ARI with expression clusters
+            expr_sp_sh = adata_sp.obs[ck].values[sp_mask]
+            ari_sp_expr = adjusted_rand_score(clust_sp_sh, expr_sp_sh)
+            ari_sv_expr = adjusted_rand_score(clust_sv_sh, adata_sv.obs[ck].values[sv_mask])
+
+            print(f"\n  scPTR vs scVelo gamma clusters: ARI={ari:.4f}, NMI={nmi:.4f}")
+            print(f"  scPTR gamma vs expression: ARI={ari_sp_expr:.4f}")
+            print(f"  scVelo gamma vs expression: ARI={ari_sv_expr:.4f}")
+        else:
+            ari = nmi = ari_sp_expr = ari_sv_expr = np.nan
+
+        # ── Which invisible states replicate? ─────────────────────────
+        print(f"\n  Invisible states:")
+        print(f"    scPTR:  {invis_sp if invis_sp else 'none'}")
+        print(f"    scVelo: {invis_sv if invis_sv else 'none'}")
+
+        print(f"\n  CONCLUSION: {'SAME structure' if ari > 0.5 else 'DIFFERENT structure' if ari < 0.2 else 'PARTIALLY overlapping'}")
+
+        all_results[ds_name] = {
+            "scptr_n_clusters": n_sp,
+            "scvelo_n_clusters": n_sv,
+            "scptr_invisible": invis_sp,
+            "scvelo_invisible": invis_sv,
+            "ari_scptr_vs_scvelo": float(ari) if np.isfinite(ari) else None,
+            "nmi_scptr_vs_scvelo": float(nmi) if np.isfinite(nmi) else None,
+            "ari_scptr_vs_expr": float(ari_sp_expr) if np.isfinite(ari_sp_expr) else None,
+            "ari_scvelo_vs_expr": float(ari_sv_expr) if np.isfinite(ari_sv_expr) else None,
+        }
+
+    save_json(all_results, "pt_states_comparison", OUT)
+
+
+if __name__ == "__main__":
+    main()

analyses/deep/35_corrected_comparison.py

@@ -0,0 +1,144 @@
+#!/usr/bin/env python
+"""Corrected method comparison: all methods on equal footing.
+
+Includes corrected scVelo dynamical (fit_gamma/fit_beta) and
+per-cell-type evaluation as a unique scPTR metric.
+"""
+from _common import *
+import scvelo as scv
+
+OUT = output_dir("35_corrected_comparison")
+
+
+def main():
+    set_figure_style()
+    hl_mouse, hl_human = load_halflife_refs()
+
+    all_results = {}
+
+    for ds_name, loader, ck in DATASETS:
+        print(f"\n{'=' * 60}\n{ds_name.upper()}: Corrected Comparison\n{'=' * 60}")
+
+        adata_raw = loader()
+
+        # ── scVelo SS ────────────────────────────────────────────────
+        adata_sv = adata_raw.copy()
+        scv.pp.filter_and_normalize(adata_sv, min_shared_counts=20, n_top_genes=2000)
+        scv.pp.moments(adata_sv, n_pcs=30, n_neighbors=30)
+        scv.tl.velocity(adata_sv, mode="steady_state")
+        vg = adata_sv.var["velocity_gamma"].values.astype(float)
+
+        # ── scVelo dynamical (CORRECTED) ─────────────────────────────
+        adata_dyn = adata_raw.copy()
+        scv.pp.filter_and_normalize(adata_dyn, min_shared_counts=20, n_top_genes=2000)
+        scv.pp.moments(adata_dyn, n_pcs=30, n_neighbors=30)
+        scv.tl.recover_dynamics(adata_dyn, n_jobs=4)
+        scv.tl.velocity(adata_dyn, mode="dynamical")
+        fg = adata_dyn.var["fit_gamma"].values.astype(float)
+        fb = adata_dyn.var["fit_beta"].values.astype(float)
+        ratio = fg / (fb + 1e-8)  # CORRECTED: use ratio
+
+        # ── scPTR ────────────────────────────────────────────────────
+        adata_sp = run_analytical(loader)
+
+        # ── Evaluate ─────────────────────────────────────────────────
+        def eval_hl(gamma_vals, var_names, label):
+            hl_s = hl_human.set_index("gene_symbol")["half_life_hours"]
+            g_upper = {g.upper(): i for i, g in enumerate(var_names)}
+            h_upper = {g.upper(): g for g in hl_s.index if isinstance(g, str)}
+            shared = set(g_upper.keys()) & set(h_upper.keys())
+            g = np.array([gamma_vals[g_upper[u]] for u in shared], dtype=float)
+            h = np.array([hl_s[h_upper[u]] for u in shared], dtype=float)
+            v = np.isfinite(g) & np.isfinite(h) & (g > 0) & (h > 0)
+            if v.sum() < 3: return np.nan, 0
+            r, _ = stats.spearmanr(g[v], h[v])
+            return float(r), int(v.sum())
+
+        results = {}
+
+        # Global half-life
+        r_ss, n_ss = eval_hl(vg, adata_sv.var_names, "scVelo SS")
+        r_dyn_raw, n_dr = eval_hl(fg, adata_dyn.var_names, "scVelo dyn (raw)")
+        r_dyn_corr, n_dc = eval_hl(ratio, adata_dyn.var_names, "scVelo dyn (corrected)")
+        r_sp, n_sp = halflife_spearman(adata_sp, hl_human)
+
+        print(f"\n  {'Method':<35} {'HL human r':>12} {'n':>6}")
+        print("  " + "-" * 55)
+        print(f"  {'scVelo SS':<35} {r_ss:>12.4f} {n_ss:>6}")
+        print(f"  {'scVelo dyn (fit_gamma, RAW)':<35} {r_dyn_raw:>12.4f} {n_dr:>6}")
+        print(f"  {'scVelo dyn (γ/β, CORRECTED)':<35} {r_dyn_corr:>12.4f} {n_dc:>6}")
+        print(f"  {'scPTR analytical':<35} {r_sp:>12.4f} {n_sp:>6}")
+
+        results["global_halflife"] = {
+            "scvelo_ss": {"r": r_ss, "n": n_ss},
+            "scvelo_dyn_raw": {"r": r_dyn_raw, "n": n_dr},
+            "scvelo_dyn_corrected": {"r": r_dyn_corr, "n": n_dc},
+            "scptr": {"r": r_sp, "n": n_sp},
+        }
+
+        # ── Per-cell-type half-life (UNIQUE TO scPTR) ─────────────────
+        print(f"\n  Per-cell-type half-life (scPTR-unique capability):")
+        if ck in adata_sp.obs.columns:
+            ct_rs = []
+            for ct in sorted(adata_sp.obs[ck].unique()):
+                mask = (adata_sp.obs[ck] == ct).values
+                if mask.sum() < 20: continue
+                gamma_ct = np.median(adata_sp.layers["gamma"][mask], axis=0)
+                adata_tmp = adata_sp.copy()
+                adata_tmp.layers["gamma"] = np.tile(gamma_ct, (adata_sp.n_obs, 1))
+                r_ct, _ = halflife_spearman(adata_tmp, hl_human)
+                ct_rs.append({"cell_type": str(ct), "r": float(r_ct)})
+
+            best = min(ct_rs, key=lambda x: x["r"])
+            print(f"    Best cell type: {best['cell_type']} (r={best['r']:.4f})")
+            print(f"    vs global: r={r_sp:.4f}")
+            print(f"    → Cell-type resolution improves r by {abs(best['r'])-abs(r_sp):.4f}")
+            results["best_celltype"] = best
+
+        all_results[ds_name] = results
+
+    save_json(all_results, "corrected_comparison", OUT)
+
+    # Corrected summary table
+    print(f"\n{'=' * 70}")
+    print("CORRECTED METHOD COMPARISON (FINAL)")
+    print("=" * 70)
+    print(f"\n{'Method':<35} ", end="")
+    for ds in all_results:
+        print(f"{'|':>2} {ds:>15}", end="")
+    print()
+    print("-" * 70)
+
+    for method in ["scvelo_ss", "scvelo_dyn_raw", "scvelo_dyn_corrected", "scptr"]:
+        label = {"scvelo_ss": "scVelo SS", "scvelo_dyn_raw": "scVelo dyn (raw γ)",
+                 "scvelo_dyn_corrected": "scVelo dyn (γ/β)", "scptr": "scPTR"}[method]
+        print(f"  {label:<33} ", end="")
+        for ds in all_results:
+            r = all_results[ds]["global_halflife"][method]["r"]
+            print(f"{'|':>2} {r:>15.4f}", end="")
+        print()
+
+    # Figure
+    fig, ax = plt.subplots(figsize=(10, 5))
+    methods = ["scVelo SS", "scVelo dyn\n(raw γ)", "scVelo dyn\n(γ/β corrected)", "scPTR"]
+    method_keys = ["scvelo_ss", "scvelo_dyn_raw", "scvelo_dyn_corrected", "scptr"]
+    colors = ["#1f77b4", "#ff9999", "#2ca02c", "#ff7f0e"]
+
+    x = np.arange(len(methods))
+    width = 0.35
+    for i, ds in enumerate(all_results):
+        rs = [abs(all_results[ds]["global_halflife"][mk]["r"]) for mk in method_keys]
+        offset = (i - 0.5) * width
+        ax.bar(x + offset, rs, width, label=ds, alpha=0.8)
+
+    ax.set_xticks(x)
+    ax.set_xticklabels(methods, fontsize=9)
+    ax.set_ylabel("|Spearman r| with half-life (human)")
+    ax.set_title("Corrected Method Comparison")
+    ax.legend()
+    fig.tight_layout()
+    save_fig(fig, "corrected_comparison", OUT)
+
+
+if __name__ == "__main__":
+    main()

analyses/deep/_common.py

@@ -0,0 +1,173 @@
+"""Shared utilities for DeepPTR benchmark scripts.
+
+All scripts in analyses/deep/ import from here for reproducibility.
+"""
+
+from __future__ import annotations
+
+import os
+
+os.environ["OMP_NUM_THREADS"] = "4"
+os.environ["MKL_NUM_THREADS"] = "4"
+os.environ["OPENBLAS_NUM_THREADS"] = "4"
+os.environ["NUMEXPR_NUM_THREADS"] = "4"
+
+import json
+import sys
+import time
+from pathlib import Path
+
+import matplotlib
+
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+from scipy import stats
+
+import torch
+
+torch.set_num_threads(4)
+
+# Inline figure style (avoids name collision with parent _common.py)
+def set_figure_style():
+    plt.rcParams.update({
+        "figure.dpi": 150, "savefig.dpi": 300, "savefig.bbox": "tight",
+        "font.size": 10, "axes.titlesize": 12, "axes.labelsize": 11,
+        "xtick.labelsize": 9, "ytick.labelsize": 9, "legend.fontsize": 9,
+        "figure.figsize": (6, 5), "axes.spines.top": False, "axes.spines.right": False,
+    })
+
+import scptr
+
+# ── Paths ──────────────────────────────────────────────────────────────────
+
+PROJECT_ROOT = Path(__file__).parent.parent.parent
+OUTPUT_ROOT = PROJECT_ROOT / "output" / "deep_benchmarks"
+DATA_DIR = Path(scptr.benchmark.__file__).parent / "data"
+
+DEEP_HP = dict(
+    d_T=8, d_PT=8, d_hidden=48, n_enc_layers=2,
+    batch_size=512, max_epochs=100, kl_warmup_epochs=20,
+    patience=15, n_posterior_samples=15,
+    device="cpu", seed=0,
+)
+
+
+def output_dir(script_name: str) -> Path:
+    """Return output directory for a given script, e.g. '01_fair_comparison'."""
+    d = OUTPUT_ROOT / script_name
+    (d / "figures").mkdir(parents=True, exist_ok=True)
+    (d / "results").mkdir(parents=True, exist_ok=True)
+    return d
+
+
+def save_fig(fig, name: str, out: Path, subdir: str = "figures"):
+    if fig is None:
+        return
+    path = out / subdir / f"{name}.png"
+    fig.savefig(path, dpi=150, bbox_inches="tight")
+    plt.close(fig)
+    print(f"  Saved: {path}")
+
+
+def save_json(data, name: str, out: Path):
+    path = out / "results" / f"{name}.json"
+    with open(path, "w") as f:
+        json.dump(data, f, indent=2, default=str)
+    print(f"  Saved: {path}")
+
+
+# ── Data loading ───────────────────────────────────────────────────────────
+
+def load_halflife_refs():
+    return scptr.datasets.herzog2017_halflives(), scptr.datasets.schofield2018_halflives()
+
+
+def select_top_genes(adata, n_top=300):
+    """Select top genes by unspliced signal for DeepPTR."""
+    from scipy.sparse import issparse
+
+    u = adata.layers["unspliced"]
+    if issparse(u):
+        u = np.asarray(u.todense())
+    u = np.asarray(u, dtype=np.float32)
+    score = u.sum(axis=0) * (u > 0).mean(axis=0)
+    top_idx = np.sort(np.argsort(score)[::-1][:n_top])
+    adata_sub = adata[:, adata.var_names[top_idx]].copy()
+    from scipy.sparse import issparse as _iss
+
+    for key in ("spliced", "unspliced"):
+        if key in adata_sub.layers and _iss(adata_sub.layers[key]):
+            adata_sub.layers[key] = np.asarray(adata_sub.layers[key].todense())
+    return adata_sub
+
+
+def run_analytical(adata_loader):
+    """Run full analytical scPTR pipeline, return adata."""
+    adata = adata_loader()
+    scptr.pp.filter_genes(adata)
+    scptr.pp.normalize_layers(adata)
+    scptr.pp.neighbors(adata, n_neighbors=30)
+    scptr.pp.smooth_layers(adata)
+    scptr.tl.estimate_beta(adata)
+    scptr.tl.estimate_gamma(adata)
+    return adata
+
+
+def run_deep(adata_loader, n_top=300, verbose=True):
+    """Run preprocessing + DeepPTR, return (adata_deep, model, history)."""
+    adata = adata_loader()
+    scptr.pp.filter_genes(adata)
+    scptr.pp.normalize_layers(adata)
+    scptr.pp.neighbors(adata, n_neighbors=30)
+    scptr.pp.smooth_layers(adata)
+    scptr.tl.estimate_beta(adata)
+    adata = select_top_genes(adata, n_top=n_top)
+    torch.set_num_threads(4)
+    model, history = scptr.deep.fit_deepptr(adata, verbose=verbose, **DEEP_HP)
+    return adata, model, history
+
+
+def run_both(adata_loader, n_top=300):
+    """Return (adata_analytical, adata_deep, model, history)."""
+    adata_an = run_analytical(adata_loader)
+    adata_dp, model, history = run_deep(adata_loader, n_top=n_top)
+    return adata_an, adata_dp, model, history
+
+
+# ── Half-life matching ─────────────────────────────────────────────────────
+
+def match_halflife(adata, hl_df, gene_col="gene_symbol", hl_col="half_life_hours"):
+    """Match genes case-insensitively, return (gamma_vals, hl_vals, gene_names)."""
+    gamma_med = np.median(adata.layers["gamma"], axis=0)
+    hl_s = hl_df.set_index(gene_col)[hl_col]
+
+    gamma_upper = {g.upper(): i for i, g in enumerate(adata.var_names)}
+    hl_upper = {g.upper(): g for g in hl_s.index if isinstance(g, str)}
+    shared = set(gamma_upper.keys()) & set(hl_upper.keys())
+
+    idx = [gamma_upper[u] for u in shared]
+    g = gamma_med[idx].astype(float)
+    h = np.array([hl_s[hl_upper[u]] for u in shared], dtype=float)
+    names = [adata.var_names[gamma_upper[u]] for u in shared]
+
+    valid = np.isfinite(g) & np.isfinite(h) & (g > 0) & (h > 0)
+    return g[valid], h[valid], [n for n, v in zip(names, valid) if v]
+
+
+def halflife_spearman(adata, hl_df):
+    """Quick Spearman r with half-life reference."""
+    g, h, _ = match_halflife(adata, hl_df)
+    if len(g) < 3:
+        return np.nan, 0
+    r, _ = stats.spearmanr(g, h)
+    return float(r), len(g)
+
+
+# ── Dataset registry ───────────────────────────────────────────────────────
+
+DATASETS = [
+    ("pancreas", scptr.datasets.pancreas, "clusters"),
+    ("dentate_gyrus", scptr.datasets.dentate_gyrus, "clusters"),
+]

analyses/run_comprehensive_fixes.py

@@ -0,0 +1,1283 @@
+#!/usr/bin/env python
+"""Comprehensive improvement of scPTR weaknesses.
+
+Fix A: Per-cell sci-fate ablation (scPTR vs raw u/s per cell)
+Fix B: 3' UTR sequence validation of network direction
+Fix C: Neuroblastoma-specific DepMap validation
+Fix D: Cross-dataset RBP hub consistency
+Fix E: Biological coherence ablation (GSEA on invisible states)
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+from scipy import stats
+
+sys.path.insert(0, str(Path(__file__).parent))
+from _common import set_figure_style
+
+import scptr
+
+OUTPUT_DIR = Path(__file__).parent.parent / "output" / "comprehensive_fixes"
+PROJECT_ROOT = Path(__file__).parent.parent
+
+
+def save_fig(fig, name, subdir="figures"):
+    if fig is None:
+        print(f"  [WARNING] {name}: None, skipping")
+        return
+    out_dir = OUTPUT_DIR / subdir
+    out_dir.mkdir(parents=True, exist_ok=True)
+    path = out_dir / f"{name}.png"
+    fig.savefig(path, dpi=150, bbox_inches="tight")
+    plt.close(fig)
+    print(f"  Saved: {path}")
+
+
+def run_pipeline(adata, name):
+    """Run standard scPTR pipeline."""
+    print(f"\n--- Pipeline: {name} ---")
+    scptr.pp.filter_genes(adata)
+    scptr.pp.normalize_layers(adata)
+    scptr.pp.neighbors(adata, n_neighbors=30)
+    scptr.pp.smooth_layers(adata)
+    scptr.tl.estimate_beta(adata)
+    scptr.tl.estimate_gamma(adata)
+    scptr.tl.variance_decomposition(adata)
+    scptr.tl.pt_states(adata)
+    scptr.tl.pt_velocity(adata)
+    print(f"  Done: {adata.shape}")
+    return adata
+
+
+# =========================================================================
+# FIX B: 3' UTR Sequence Validation of Network Direction
+# =========================================================================
+def fix_b_utr_validation():
+    """Validate network direction using 3' UTR sequence features.
+
+    Destabilizing targets should have longer 3' UTRs (more regulatory elements)
+    and higher AU content.
+    """
+    print("\n" + "=" * 60)
+    print("FIX B: 3' UTR SEQUENCE VALIDATION OF NETWORK DIRECTION")
+    print("=" * 60)
+
+    res_dir = OUTPUT_DIR / "results"
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    # Load UTR features
+    data_dir = PROJECT_ROOT / "src" / "scptr" / "benchmark" / "data"
+    mouse_utr = pd.read_csv(data_dir / "mouse_utr_features.csv")
+    human_utr = pd.read_csv(data_dir / "human_utr_features.csv")
+    print(f"  Mouse UTR features: {len(mouse_utr)} genes")
+    print(f"  Human UTR features: {len(human_utr)} genes")
+
+    # Load corrected networks
+    networks = {}
+    net_files = {
+        "pancreas": PROJECT_ROOT / "output" / "weakness_fixes" / "results" / "corrected_network_pancreas.csv",
+        "dentate_gyrus": PROJECT_ROOT / "output" / "weakness_fixes" / "results" / "corrected_network_dentate_gyrus.csv",
+        "neuroblastoma": PROJECT_ROOT / "output" / "tier3" / "results" / "neuroblastoma_network_corrected.csv",
+    }
+
+    for name, path in net_files.items():
+        if path.exists():
+            networks[name] = pd.read_csv(path)
+            print(f"  {name} network: {len(networks[name])} edges")
+        else:
+            print(f"  [WARNING] {name} network not found at {path}")
+
+    results = {}
+    all_summaries = []
+
+    for net_name, edges_df in networks.items():
+        print(f"\n--- {net_name} ---")
+
+        # Determine which UTR dataset to use
+        # Neuroblastoma = human, pancreas/DG = mouse
+        if net_name == "neuroblastoma":
+            utr_df = human_utr.copy()
+            # Column for correlation is spearman_r
+            r_col = "spearman_r" if "spearman_r" in edges_df.columns else "r"
+        else:
+            utr_df = mouse_utr.copy()
+            r_col = "r" if "r" in edges_df.columns else "spearman_r"
+
+        # Build gene-level summary: mean correlation across all RBP connections
+        target_stats = edges_df.groupby("target").agg(
+            mean_r=(r_col, "mean"),
+            n_rbps=(r_col, "count"),
+        ).reset_index()
+
+        # Classify as predominantly destabilized (mean r > 0) or stabilized (mean r < 0)
+        target_stats["class"] = np.where(target_stats["mean_r"] > 0,
+                                         "destabilized", "stabilized")
+        n_dest = (target_stats["class"] == "destabilized").sum()
+        n_stab = (target_stats["class"] == "stabilized").sum()
+        print(f"  Target genes: {len(target_stats)} ({n_dest} destabilized, {n_stab} stabilized)")
+
+        # Match target genes to UTR features (case-insensitive)
+        utr_map = {g.upper(): i for i, g in enumerate(utr_df["gene"])}
+        target_stats["gene_upper"] = target_stats["target"].str.upper()
+        matched = target_stats[target_stats["gene_upper"].isin(utr_map)].copy()
+        matched["utr_length"] = matched["gene_upper"].map(
+            lambda g: utr_df.iloc[utr_map[g]]["utr_length"])
+        matched["au_content"] = matched["gene_upper"].map(
+            lambda g: utr_df.iloc[utr_map[g]]["au_content"])
+
+        n_matched = len(matched)
+        print(f"  Matched to UTR features: {n_matched}/{len(target_stats)}")
+
+        if n_matched < 10:
+            print(f"  Too few matched genes, skipping")
+            continue
+
+        dest = matched[matched["class"] == "destabilized"]
+        stab = matched[matched["class"] == "stabilized"]
+
+        net_results = {"dataset": net_name, "n_targets": len(target_stats),
+                       "n_matched": n_matched}
+
+        # Test 1: UTR length destabilized vs stabilized
+        if len(dest) >= 5 and len(stab) >= 5:
+            u_stat, p_len = stats.mannwhitneyu(
+                dest["utr_length"].values, stab["utr_length"].values,
+                alternative="greater")
+            med_dest_len = dest["utr_length"].median()
+            med_stab_len = stab["utr_length"].median()
+            print(f"  UTR length: destab median={med_dest_len:.0f}, "
+                  f"stab median={med_stab_len:.0f}, "
+                  f"MW p={p_len:.4f} (destab > stab)")
+            net_results["utr_length_destab_median"] = float(med_dest_len)
+            net_results["utr_length_stab_median"] = float(med_stab_len)
+            net_results["utr_length_mw_p"] = float(p_len)
+        else:
+            p_len = np.nan
+
+        # Test 2: AU content destabilized vs stabilized
+        if len(dest) >= 5 and len(stab) >= 5:
+            u_stat, p_au = stats.mannwhitneyu(
+                dest["au_content"].values, stab["au_content"].values,
+                alternative="greater")
+            med_dest_au = dest["au_content"].median()
+            med_stab_au = stab["au_content"].median()
+            print(f"  AU content: destab median={med_dest_au:.4f}, "
+                  f"stab median={med_stab_au:.4f}, "
+                  f"MW p={p_au:.4f} (destab > stab)")
+            net_results["au_content_destab_median"] = float(med_dest_au)
+            net_results["au_content_stab_median"] = float(med_stab_au)
+            net_results["au_content_mw_p"] = float(p_au)
+        else:
+            p_au = np.nan
+
+        # Test 3: Spearman correlation of mean_r vs UTR length
+        r_vs_len, p_r_len = stats.spearmanr(
+            matched["mean_r"].values, matched["utr_length"].values)
+        print(f"  Spearman(mean_r, UTR length): r={r_vs_len:.4f}, p={p_r_len:.4f}")
+        net_results["spearman_r_vs_utr_length"] = float(r_vs_len)
+        net_results["spearman_p_vs_utr_length"] = float(p_r_len)
+
+        # Test 4: Spearman correlation of mean_r vs AU content
+        r_vs_au, p_r_au = stats.spearmanr(
+            matched["mean_r"].values, matched["au_content"].values)
+        print(f"  Spearman(mean_r, AU content): r={r_vs_au:.4f}, p={p_r_au:.4f}")
+        net_results["spearman_r_vs_au_content"] = float(r_vs_au)
+        net_results["spearman_p_vs_au_content"] = float(p_r_au)
+
+        results[net_name] = net_results
+        all_summaries.append(net_results)
+
+    # Save results
+    with open(res_dir / "utr_network_validation.json", "w") as f:
+        json.dump(results, f, indent=2)
+
+    # Figure: 2x3 panels (UTR length and AU content for each dataset)
+    n_nets = len(results)
+    if n_nets == 0:
+        print("  No networks to plot")
+        return results
+
+    fig, axes = plt.subplots(2, n_nets, figsize=(5 * n_nets, 8))
+    if n_nets == 1:
+        axes = axes.reshape(2, 1)
+
+    for col, (net_name, edges_df) in enumerate(networks.items()):
+        if net_name not in results:
+            continue
+
+        r_col = "spearman_r" if "spearman_r" in edges_df.columns else "r"
+        if net_name == "neuroblastoma":
+            utr_df = human_utr
+        else:
+            utr_df = mouse_utr
+
+        # Rebuild matched data for plotting
+        target_stats = edges_df.groupby("target").agg(
+            mean_r=(r_col, "mean"),
+        ).reset_index()
+        target_stats["gene_upper"] = target_stats["target"].str.upper()
+        utr_map = {g.upper(): i for i, g in enumerate(utr_df["gene"])}
+        matched = target_stats[target_stats["gene_upper"].isin(utr_map)].copy()
+        matched["utr_length"] = matched["gene_upper"].map(
+            lambda g: utr_df.iloc[utr_map[g]]["utr_length"])
+        matched["au_content"] = matched["gene_upper"].map(
+            lambda g: utr_df.iloc[utr_map[g]]["au_content"])
+
+        # Row 0: scatter mean_r vs UTR length
+        ax = axes[0, col]
+        ax.scatter(matched["mean_r"], matched["utr_length"],
+                   alpha=0.3, s=10, c="steelblue")
+        r_val = results[net_name].get("spearman_r_vs_utr_length", np.nan)
+        p_val = results[net_name].get("spearman_p_vs_utr_length", np.nan)
+        ax.set_xlabel("Mean RBP-target r")
+        ax.set_ylabel("3' UTR length (nt)")
+        ax.set_title(f"{net_name}\nr={r_val:.3f}, p={p_val:.3f}")
+
+        # Row 1: scatter mean_r vs AU content
+        ax = axes[1, col]
+        ax.scatter(matched["mean_r"], matched["au_content"],
+                   alpha=0.3, s=10, c="darkorange")
+        r_val = results[net_name].get("spearman_r_vs_au_content", np.nan)
+        p_val = results[net_name].get("spearman_p_vs_au_content", np.nan)
+        ax.set_xlabel("Mean RBP-target r")
+        ax.set_ylabel("AU content")
+        ax.set_title(f"{net_name}\nr={r_val:.3f}, p={p_val:.3f}")
+
+    fig.suptitle("3' UTR Validation of Network Direction", fontsize=13, y=1.02)
+    fig.tight_layout()
+    save_fig(fig, "utr_network_validation")
+
+    return results
+
+
+# =========================================================================
+# FIX D: Cross-Dataset RBP Hub Consistency
+# =========================================================================
+def fix_d_hub_consistency():
+    """Compare hub rankings across pancreas, DG, and neuroblastoma."""
+    print("\n" + "=" * 60)
+    print("FIX D: CROSS-DATASET RBP HUB CONSISTENCY")
+    print("=" * 60)
+
+    res_dir = OUTPUT_DIR / "results"
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    # Load hub counts from gap_analysis
+    hub_files = {
+        "pancreas": PROJECT_ROOT / "output" / "gap_analysis" / "results" / "network" / "pancreas" / "rbp_hub_counts.csv",
+        "dentate_gyrus": PROJECT_ROOT / "output" / "gap_analysis" / "results" / "network" / "dentate_gyrus" / "rbp_hub_counts.csv",
+    }
+
+    hub_counts = {}
+
+    for name, path in hub_files.items():
+        if path.exists():
+            df = pd.read_csv(path)
+            # Format: rbp, 0 (where 0 is the count column)
+            count_col = [c for c in df.columns if c != "rbp"][0]
+            series = pd.Series(df[count_col].values, index=df["rbp"].values)
+            hub_counts[name] = series
+            print(f"  {name}: {len(series)} RBPs")
+        else:
+            print(f"  [WARNING] {name} hub counts not found at {path}")
+
+    # Compute NB hub counts from corrected network
+    nb_net_path = PROJECT_ROOT / "output" / "tier3" / "results" / "neuroblastoma_network_corrected.csv"
+    if nb_net_path.exists():
+        nb_net = pd.read_csv(nb_net_path)
+        nb_hubs = nb_net.groupby("rbp").size().sort_values(ascending=False)
+        hub_counts["neuroblastoma"] = nb_hubs
+        print(f"  neuroblastoma: {len(nb_hubs)} RBPs")
+
+    if len(hub_counts) < 2:
+        print("  Need at least 2 datasets for comparison")
+        return {}
+
+    # Unify gene names to uppercase
+    hub_upper = {}
+    for name, series in hub_counts.items():
+        hub_upper[name] = pd.Series(series.values, index=[g.upper() for g in series.index])
+
+    # Pairwise Spearman on target counts across shared RBPs
+    names = sorted(hub_upper.keys())
+    results = {"pairwise_correlations": [], "universal_hubs": [], "dataset_hubs": {}}
+
+    print("\n  Pairwise hub count correlations:")
+    for i, name_a in enumerate(names):
+        for j in range(i + 1, len(names)):
+            name_b = names[j]
+            shared = hub_upper[name_a].index.intersection(hub_upper[name_b].index)
+            if len(shared) < 5:
+                print(f"  {name_a} vs {name_b}: only {len(shared)} shared RBPs, skipping")
+                continue
+
+            va = hub_upper[name_a][shared].values.astype(float)
+            vb = hub_upper[name_b][shared].values.astype(float)
+            r, p = stats.spearmanr(va, vb)
+            print(f"  {name_a} vs {name_b}: Spearman r={r:.4f}, p={p:.4f} (n={len(shared)})")
+
+            results["pairwise_correlations"].append({
+                "dataset_a": name_a,
+                "dataset_b": name_b,
+                "spearman_r": float(r),
+                "spearman_p": float(p),
+                "n_shared": int(len(shared)),
+            })
+
+    # Fisher's exact: are top-10 hubs in A enriched among top-20 in B?
+    print("\n  Fisher's exact test (top-10 in A enriched among top-20 in B?):")
+    for i, name_a in enumerate(names):
+        for j in range(len(names)):
+            if i == j:
+                continue
+            name_b = names[j]
+            shared = hub_upper[name_a].index.intersection(hub_upper[name_b].index)
+            if len(shared) < 5:
+                continue
+
+            top_a = set(hub_upper[name_a].nlargest(10).index)
+            top_b = set(hub_upper[name_b].nlargest(20).index)
+
+            # Contingency table
+            a_in_b = len(top_a & top_b)
+            a_not_b = len(top_a - top_b)
+            not_a_in_b = len(top_b - top_a)
+            not_a_not_b = len(shared) - a_in_b - a_not_b - not_a_in_b
+
+            if not_a_not_b < 0:
+                not_a_not_b = 0
+
+            table = [[a_in_b, a_not_b], [not_a_in_b, not_a_not_b]]
+            odds_ratio, fisher_p = stats.fisher_exact(table, alternative="greater")
+            print(f"  Top-10 {name_a} in top-20 {name_b}: "
+                  f"{a_in_b}/10, OR={odds_ratio:.2f}, p={fisher_p:.4f}")
+
+    # Identify "universal" hubs (top 20 in >= 2 datasets)
+    print("\n  Universal hubs (top 20 in >= 2 datasets):")
+    top20_sets = {}
+    for name in names:
+        top20_sets[name] = set(hub_upper[name].nlargest(20).index)
+
+    all_rbps = set()
+    for s in top20_sets.values():
+        all_rbps |= s
+
+    hub_table = []
+    for rbp in sorted(all_rbps):
+        datasets_in_top20 = [name for name in names if rbp in top20_sets[name]]
+        counts_per_dataset = {name: int(hub_upper[name].get(rbp, 0))
+                              for name in names}
+        hub_table.append({
+            "rbp": rbp,
+            "n_datasets_top20": len(datasets_in_top20),
+            "datasets": ", ".join(datasets_in_top20),
+            **{f"targets_{name}": counts_per_dataset[name] for name in names},
+        })
+
+    hub_df = pd.DataFrame(hub_table).sort_values("n_datasets_top20", ascending=False)
+
+    # Save per-dataset top hubs
+    for name in names:
+        results["dataset_hubs"][name] = hub_upper[name].nlargest(10).to_dict()
+
+    universal = hub_df[hub_df["n_datasets_top20"] >= 2]
+    tissue_specific = hub_df[hub_df["n_datasets_top20"] == 1]
+    print(f"  Universal (>=2): {len(universal)} RBPs")
+    for _, row in universal.iterrows():
+        print(f"    {row['rbp']}: {row['datasets']}")
+    print(f"  Tissue-specific (1 only): {len(tissue_specific)} RBPs")
+
+    results["universal_hubs"] = universal.to_dict(orient="records")
+    results["n_universal"] = int(len(universal))
+    results["n_tissue_specific"] = int(len(tissue_specific))
+
+    # Save
+    hub_df.to_csv(res_dir / "hub_consistency_table.csv", index=False)
+    with open(res_dir / "hub_consistency.json", "w") as f:
+        json.dump(results, f, indent=2, default=str)
+
+    # Figure: heatmap of hub counts + bar chart of universal vs specific
+    fig, axes = plt.subplots(1, 2, figsize=(14, 6))
+
+    # Panel 1: heatmap of top RBPs across datasets
+    top_rbps = hub_df.nlargest(20, "n_datasets_top20")
+    target_cols = [f"targets_{n}" for n in names]
+    heatmap_data = top_rbps[target_cols].values.astype(float)
+    heatmap_labels = top_rbps["rbp"].values
+
+    im = axes[0].imshow(heatmap_data, aspect="auto", cmap="YlOrRd")
+    axes[0].set_yticks(np.arange(len(heatmap_labels)))
+    axes[0].set_yticklabels(heatmap_labels, fontsize=8)
+    axes[0].set_xticks(np.arange(len(names)))
+    axes[0].set_xticklabels(names, fontsize=9, rotation=30, ha="right")
+    axes[0].set_title("Hub RBP Target Counts Across Datasets")
+    for i in range(len(heatmap_labels)):
+        for j in range(len(names)):
+            val = int(heatmap_data[i, j])
+            if val > 0:
+                axes[0].text(j, i, str(val), ha="center", va="center",
+                            fontsize=7, color="white" if val > heatmap_data.max() * 0.6 else "black")
+    plt.colorbar(im, ax=axes[0], label="Target count", shrink=0.8)
+
+    # Panel 2: universal vs tissue-specific
+    axes[1].bar(["Universal\n(>=2 datasets)", "Tissue-specific\n(1 dataset)"],
+                [len(universal), len(tissue_specific)],
+                color=["steelblue", "salmon"], edgecolor="black", linewidth=0.5)
+    axes[1].set_ylabel("Number of RBPs")
+    axes[1].set_title("Hub Consistency Across Datasets")
+    for i, v in enumerate([len(universal), len(tissue_specific)]):
+        axes[1].text(i, v + 0.5, str(v), ha="center", fontsize=11, fontweight="bold")
+
+    fig.suptitle("Cross-Dataset RBP Hub Consistency", fontsize=13, y=1.02)
+    fig.tight_layout()
+    save_fig(fig, "hub_consistency")
+
+    return results
+
+
+# =========================================================================
+# FIX C: Neuroblastoma-Specific DepMap
+# =========================================================================
+def fix_c_nb_depmap():
+    """Filter DepMap CRISPR scores to NB-specific cell lines."""
+    print("\n" + "=" * 60)
+    print("FIX C: NEUROBLASTOMA-SPECIFIC DepMap VALIDATION")
+    print("=" * 60)
+
+    res_dir = OUTPUT_DIR / "results"
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    cache_dir = PROJECT_ROOT / ".cache"
+
+    # Load DepMap model metadata
+    model_df = pd.read_csv(cache_dir / "DepMap_Model.csv")
+    nb_models = model_df[model_df["OncotreePrimaryDisease"] == "Neuroblastoma"]
+    nb_model_ids = set(nb_models["ModelID"].values)
+    print(f"  Neuroblastoma cell lines in DepMap: {len(nb_model_ids)}")
+
+    # Load CRISPR gene effect
+    print("  Loading CRISPRGeneEffect.csv...")
+    crispr_df = pd.read_csv(cache_dir / "CRISPRGeneEffect.csv", index_col=0)
+    print(f"  CRISPR data: {crispr_df.shape[0]} cell lines, {crispr_df.shape[1]} genes")
+
+    # Parse gene names from column headers: "GENE (ID)" -> "GENE"
+    gene_names = [col.split(" (")[0] for col in crispr_df.columns]
+    crispr_df.columns = gene_names
+
+    # Filter to NB cell lines
+    nb_ids_in_crispr = nb_model_ids & set(crispr_df.index)
+    print(f"  NB cell lines with CRISPR data: {len(nb_ids_in_crispr)}")
+
+    nb_crispr = crispr_df.loc[list(nb_ids_in_crispr)]
+    all_crispr = crispr_df
+
+    # Mean dependency per gene
+    nb_mean_dep = nb_crispr.mean(axis=0)
+    all_mean_dep = all_crispr.mean(axis=0)
+    non_nb_crispr = crispr_df.loc[~crispr_df.index.isin(nb_model_ids)]
+    non_nb_mean_dep = non_nb_crispr.mean(axis=0)
+
+    # Load network hubs for each dataset
+    hub_files = {
+        "neuroblastoma": PROJECT_ROOT / "output" / "tier3" / "results" / "neuroblastoma_network_corrected.csv",
+        "pancreas": PROJECT_ROOT / "output" / "weakness_fixes" / "results" / "corrected_network_pancreas.csv",
+        "dentate_gyrus": PROJECT_ROOT / "output" / "weakness_fixes" / "results" / "corrected_network_dentate_gyrus.csv",
+    }
+
+    results = {}
+
+    for net_name, net_path in hub_files.items():
+        if not net_path.exists():
+            print(f"  [WARNING] {net_name} network not found")
+            continue
+
+        net_df = pd.read_csv(net_path)
+        hub_counts = net_df.groupby("rbp").size().sort_values(ascending=False)
+        top_n = min(20, len(hub_counts))
+        hub_rbps = set(hub_counts.index[:top_n])
+        non_hub_rbps = set(hub_counts.index[top_n:])
+
+        print(f"\n--- {net_name} ({len(hub_rbps)} hub, {len(non_hub_rbps)} non-hub RBPs) ---")
+
+        # Match to CRISPR gene names (uppercase)
+        crispr_genes_upper = {g.upper(): g for g in nb_mean_dep.index}
+
+        hub_nb_deps = []
+        hub_all_deps = []
+        hub_non_nb_deps = []
+        for rbp in hub_rbps:
+            g_upper = rbp.upper()
+            if g_upper in crispr_genes_upper:
+                cg = crispr_genes_upper[g_upper]
+                hub_nb_deps.append(nb_mean_dep[cg])
+                hub_all_deps.append(all_mean_dep[cg])
+                hub_non_nb_deps.append(non_nb_mean_dep[cg])
+
+        nonhub_nb_deps = []
+        nonhub_all_deps = []
+        nonhub_non_nb_deps = []
+        for rbp in non_hub_rbps:
+            g_upper = rbp.upper()
+            if g_upper in crispr_genes_upper:
+                cg = crispr_genes_upper[g_upper]
+                nonhub_nb_deps.append(nb_mean_dep[cg])
+                nonhub_all_deps.append(all_mean_dep[cg])
+                nonhub_non_nb_deps.append(non_nb_mean_dep[cg])
+
+        net_results = {
+            "n_hub_rbps": len(hub_rbps),
+            "n_hub_matched": len(hub_nb_deps),
+            "n_nonhub_matched": len(nonhub_nb_deps),
+        }
+
+        # NB-specific: hub vs non-hub
+        if len(hub_nb_deps) >= 3 and len(nonhub_nb_deps) >= 3:
+            u_stat, p_nb = stats.mannwhitneyu(
+                hub_nb_deps, nonhub_nb_deps, alternative="less")
+            print(f"  NB-specific: hub mean={np.mean(hub_nb_deps):.4f}, "
+                  f"non-hub mean={np.mean(nonhub_nb_deps):.4f}, "
+                  f"MW p={p_nb:.4e}")
+            net_results["nb_hub_mean_dep"] = float(np.mean(hub_nb_deps))
+            net_results["nb_nonhub_mean_dep"] = float(np.mean(nonhub_nb_deps))
+            net_results["nb_mw_p"] = float(p_nb)
+
+        # Pan-cancer: hub vs non-hub
+        if len(hub_all_deps) >= 3 and len(nonhub_all_deps) >= 3:
+            u_stat, p_all = stats.mannwhitneyu(
+                hub_all_deps, nonhub_all_deps, alternative="less")
+            print(f"  Pan-cancer:  hub mean={np.mean(hub_all_deps):.4f}, "
+                  f"non-hub mean={np.mean(nonhub_all_deps):.4f}, "
+                  f"MW p={p_all:.4e}")
+            net_results["all_hub_mean_dep"] = float(np.mean(hub_all_deps))
+            net_results["all_nonhub_mean_dep"] = float(np.mean(nonhub_all_deps))
+            net_results["all_mw_p"] = float(p_all)
+
+        # NB-specificity: are NB hubs MORE essential in NB vs non-NB?
+        if len(hub_nb_deps) >= 3 and len(hub_non_nb_deps) >= 3:
+            u_stat, p_spec = stats.mannwhitneyu(
+                hub_nb_deps, hub_non_nb_deps, alternative="less")
+            print(f"  NB-specificity: hub in NB={np.mean(hub_nb_deps):.4f}, "
+                  f"hub in non-NB={np.mean(hub_non_nb_deps):.4f}, "
+                  f"MW p={p_spec:.4e}")
+            net_results["nb_specificity_p"] = float(p_spec)
+            net_results["hub_nb_mean"] = float(np.mean(hub_nb_deps))
+            net_results["hub_non_nb_mean"] = float(np.mean(hub_non_nb_deps))
+
+        # Correlation: n_targets vs NB-specific dependency
+        all_rbps_in_net = hub_counts.index.tolist()
+        n_targets_list = []
+        dep_list = []
+        for rbp in all_rbps_in_net:
+            g_upper = rbp.upper()
+            if g_upper in crispr_genes_upper:
+                cg = crispr_genes_upper[g_upper]
+                n_targets_list.append(hub_counts[rbp])
+                dep_list.append(nb_mean_dep[cg])
+
+        if len(n_targets_list) >= 5:
+            r_corr, p_corr = stats.spearmanr(n_targets_list, dep_list)
+            print(f"  Corr(n_targets, NB dep): r={r_corr:.4f}, p={p_corr:.4f}")
+            net_results["ntargets_dep_spearman_r"] = float(r_corr)
+            net_results["ntargets_dep_spearman_p"] = float(p_corr)
+
+        results[net_name] = net_results
+
+    # Save results
+    with open(res_dir / "nb_specific_depmap.json", "w") as f:
+        json.dump(results, f, indent=2)
+
+    # Figure: grouped bar chart comparing NB-specific vs pan-cancer
+    fig, axes = plt.subplots(1, 2, figsize=(12, 5))
+
+    # Panel 1: Hub vs non-hub dependency by dataset and scope
+    datasets = [n for n in ["neuroblastoma", "pancreas", "dentate_gyrus"] if n in results]
+    x = np.arange(len(datasets))
+    width = 0.2
+
+    for offset, (scope, label, color) in enumerate([
+        ("nb_hub_mean_dep", "Hub (NB)", "darkred"),
+        ("nb_nonhub_mean_dep", "Non-hub (NB)", "salmon"),
+        ("all_hub_mean_dep", "Hub (pan-cancer)", "darkblue"),
+        ("all_nonhub_mean_dep", "Non-hub (pan-cancer)", "lightblue"),
+    ]):
+        vals = [results.get(d, {}).get(scope, 0) for d in datasets]
+        axes[0].bar(x + (offset - 1.5) * width, vals, width, label=label,
+                    color=color, edgecolor="black", linewidth=0.3)
+
+    axes[0].set_xticks(x)
+    axes[0].set_xticklabels(datasets, fontsize=9)
+    axes[0].set_ylabel("Mean CRISPR dependency\n(more negative = more essential)")
+    axes[0].set_title("Hub RBP Essentiality: NB-Specific vs Pan-Cancer")
+    axes[0].legend(fontsize=7, loc="upper right")
+    axes[0].axhline(0, color="gray", linestyle="--", alpha=0.3)
+
+    # Panel 2: NB-specificity for NB network hubs
+    if "neuroblastoma" in results:
+        nb_res = results["neuroblastoma"]
+        categories = []
+        values = []
+        colors = []
+        if "hub_nb_mean" in nb_res:
+            categories.append("NB hub\n(in NB lines)")
+            values.append(nb_res["hub_nb_mean"])
+            colors.append("darkred")
+        if "hub_non_nb_mean" in nb_res:
+            categories.append("NB hub\n(in non-NB)")
+            values.append(nb_res["hub_non_nb_mean"])
+            colors.append("lightcoral")
+        if "nb_nonhub_mean_dep" in nb_res:
+            categories.append("Non-hub\n(in NB lines)")
+            values.append(nb_res["nb_nonhub_mean_dep"])
+            colors.append("gray")
+
+        if values:
+            axes[1].bar(categories, values, color=colors, edgecolor="black", linewidth=0.5)
+            axes[1].set_ylabel("Mean CRISPR dependency")
+            axes[1].set_title("NB Hub RBPs: Tissue-Specific Essentiality")
+            if "nb_specificity_p" in nb_res:
+                axes[1].text(0.5, 0.95, f"NB vs non-NB: p={nb_res['nb_specificity_p']:.4f}",
+                            transform=axes[1].transAxes, ha="center", va="top", fontsize=9)
+
+    fig.suptitle("Neuroblastoma-Specific DepMap Validation", fontsize=13, y=1.02)
+    fig.tight_layout()
+    save_fig(fig, "nb_specific_depmap")
+
+    return results
+
+
+# =========================================================================
+# FIX A: Per-Cell sci-fate Ablation
+# =========================================================================
+def fix_a_per_cell_scifate():
+    """Compare per-cell correlations: scPTR gamma vs raw u/s ratio."""
+    print("\n" + "=" * 60)
+    print("FIX A: PER-CELL SCI-FATE ABLATION")
+    print("=" * 60)
+
+    res_dir = OUTPUT_DIR / "results"
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    # Import sci-fate loading functions
+    from run_scifate import load_scifate_data, prepare_for_scptr
+
+    # Load raw sci-fate data
+    adata_raw = load_scifate_data()
+
+    # Prepare for scPTR
+    adata = prepare_for_scptr(adata_raw)
+
+    # Run scPTR pipeline
+    scptr.pp.filter_genes(adata)
+    scptr.pp.normalize_layers(adata)
+    scptr.pp.neighbors(adata, n_neighbors=30)
+    scptr.pp.smooth_layers(adata)
+    scptr.tl.estimate_beta(adata)
+    scptr.tl.estimate_gamma(adata)
+    print(f"  Pipeline complete: {adata.shape}")
+
+    # Get gamma matrix (smoothed, beta-normalized)
+    gamma = adata.layers["gamma"]  # cells x genes
+
+    # Compute raw u/s ratio (unsmoothed)
+    u_layer = adata.layers.get("Mu", adata.layers.get("unspliced"))
+    s_layer = adata.layers.get("Ms", adata.layers.get("spliced"))
+    u = u_layer.toarray() if hasattr(u_layer, 'toarray') else np.asarray(u_layer)
+    s = s_layer.toarray() if hasattr(s_layer, 'toarray') else np.asarray(s_layer)
+
+    # Raw u/s ratio with same safeguard as scPTR
+    raw_ratio = np.zeros_like(gamma)
+    s_safe = np.where(s > 0.01, s, 1.0)
+    raw_ratio = u / s_safe
+    raw_ratio[s < 0.01] = 0
+
+    # Compute ground truth new/old ratio per cell
+    # Map back to the genes that survived filtering
+    total_raw = np.asarray(adata_raw.X.toarray() if hasattr(adata_raw.X, 'toarray') else adata_raw.X)
+    new_raw = np.asarray(adata_raw.layers["new"].toarray() if hasattr(adata_raw.layers["new"], 'toarray') else adata_raw.layers["new"])
+    old_raw = total_raw - new_raw
+
+    # Match genes between adata (filtered) and adata_raw
+    raw_gene_map = {g: i for i, g in enumerate(adata_raw.var_names)}
+    filtered_in_raw = [raw_gene_map[g] for g in adata.var_names if g in raw_gene_map]
+    genes_in_both = [g for g in adata.var_names if g in raw_gene_map]
+
+    if len(genes_in_both) < len(adata.var_names):
+        print(f"  [WARNING] {len(adata.var_names) - len(genes_in_both)} genes not matched")
+
+    # Ground truth per cell: new/old ratio for each gene
+    gt_new = new_raw[:, filtered_in_raw]
+    gt_old = old_raw[:, filtered_in_raw]
+    gt_ratio = np.zeros_like(gt_new, dtype=float)
+    valid_gt = gt_old > 0.1
+    gt_ratio[valid_gt] = gt_new[valid_gt] / gt_old[valid_gt]
+    gt_ratio[~valid_gt] = np.nan
+
+    # Get corresponding columns from gamma and raw_ratio
+    gene_idx_in_filtered = [list(adata.var_names).index(g) for g in genes_in_both]
+    gamma_matched = gamma[:, gene_idx_in_filtered]
+    raw_matched = raw_ratio[:, gene_idx_in_filtered]
+
+    n_cells = adata.n_obs
+    print(f"  Computing per-cell correlations for {n_cells} cells...")
+
+    # Per-cell: Spearman(gamma_vector, gt_vector) and Spearman(raw_vector, gt_vector)
+    gamma_corrs = np.full(n_cells, np.nan)
+    raw_corrs = np.full(n_cells, np.nan)
+    gamma_cvs = np.full(n_cells, np.nan)
+    raw_cvs = np.full(n_cells, np.nan)
+
+    min_genes_per_cell = 20
+
+    for i in range(n_cells):
+        gt_i = gt_ratio[i]
+        gamma_i = gamma_matched[i]
+        raw_i = raw_matched[i]
+
+        # Mask: need valid gt AND nonzero method value
+        valid = np.isfinite(gt_i) & (gt_i > 0) & (gamma_i > 0) & (raw_i > 0)
+        n_valid = valid.sum()
+
+        if n_valid >= min_genes_per_cell:
+            r_gamma, _ = stats.spearmanr(gamma_i[valid], gt_i[valid])
+            r_raw, _ = stats.spearmanr(raw_i[valid], gt_i[valid])
+            gamma_corrs[i] = r_gamma
+            raw_corrs[i] = r_raw
+
+            # CV: coefficient of variation (std/mean) — lower = less noisy
+            gamma_cv = np.std(gamma_i[valid]) / (np.mean(gamma_i[valid]) + 1e-10)
+            raw_cv = np.std(raw_i[valid]) / (np.mean(raw_i[valid]) + 1e-10)
+            gamma_cvs[i] = gamma_cv
+            raw_cvs[i] = raw_cv
+
+    valid_cells = np.isfinite(gamma_corrs) & np.isfinite(raw_corrs)
+    n_valid_cells = valid_cells.sum()
+    print(f"  Valid cells: {n_valid_cells}/{n_cells}")
+
+    if n_valid_cells < 10:
+        print("  Too few valid cells, aborting Fix A")
+        return {}
+
+    # Summary statistics
+    mean_gamma_corr = np.nanmean(gamma_corrs[valid_cells])
+    mean_raw_corr = np.nanmean(raw_corrs[valid_cells])
+    med_gamma_corr = np.nanmedian(gamma_corrs[valid_cells])
+    med_raw_corr = np.nanmedian(raw_corrs[valid_cells])
+
+    print(f"\n  Per-cell correlation with ground truth:")
+    print(f"    scPTR gamma: mean={mean_gamma_corr:.4f}, median={med_gamma_corr:.4f}")
+    print(f"    Raw u/s:     mean={mean_raw_corr:.4f}, median={med_raw_corr:.4f}")
+
+    # Wilcoxon signed-rank test (paired)
+    w_stat, wilcox_p = stats.wilcoxon(
+        gamma_corrs[valid_cells], raw_corrs[valid_cells],
+        alternative="greater")
+    print(f"  Wilcoxon signed-rank (gamma > raw): p={wilcox_p:.4e}")
+
+    # Fraction of cells where gamma beats raw
+    gamma_better = (gamma_corrs[valid_cells] > raw_corrs[valid_cells]).sum()
+    raw_better = (raw_corrs[valid_cells] > gamma_corrs[valid_cells]).sum()
+    print(f"  gamma beats raw: {gamma_better}/{n_valid_cells} ({100*gamma_better/n_valid_cells:.1f}%)")
+    print(f"  raw beats gamma: {raw_better}/{n_valid_cells} ({100*raw_better/n_valid_cells:.1f}%)")
+
+    # CV comparison
+    valid_cv = np.isfinite(gamma_cvs) & np.isfinite(raw_cvs)
+    if valid_cv.sum() > 10:
+        mean_gamma_cv = np.nanmean(gamma_cvs[valid_cv])
+        mean_raw_cv = np.nanmean(raw_cvs[valid_cv])
+        w_cv, cv_p = stats.wilcoxon(
+            gamma_cvs[valid_cv], raw_cvs[valid_cv],
+            alternative="less")
+        print(f"\n  Coefficient of variation (noise):")
+        print(f"    scPTR gamma: mean CV={mean_gamma_cv:.4f}")
+        print(f"    Raw u/s:     mean CV={mean_raw_cv:.4f}")
+        print(f"    Wilcoxon (gamma < raw): p={cv_p:.4e}")
+    else:
+        mean_gamma_cv = np.nan
+        mean_raw_cv = np.nan
+        cv_p = np.nan
+
+    results = {
+        "n_cells_total": int(n_cells),
+        "n_cells_valid": int(n_valid_cells),
+        "mean_gamma_corr": float(mean_gamma_corr),
+        "mean_raw_corr": float(mean_raw_corr),
+        "median_gamma_corr": float(med_gamma_corr),
+        "median_raw_corr": float(med_raw_corr),
+        "wilcoxon_p": float(wilcox_p),
+        "gamma_better_frac": float(gamma_better / n_valid_cells),
+        "raw_better_frac": float(raw_better / n_valid_cells),
+        "mean_gamma_cv": float(mean_gamma_cv) if np.isfinite(mean_gamma_cv) else None,
+        "mean_raw_cv": float(mean_raw_cv) if np.isfinite(mean_raw_cv) else None,
+        "cv_wilcoxon_p": float(cv_p) if np.isfinite(cv_p) else None,
+    }
+
+    with open(res_dir / "per_cell_scifate.json", "w") as f:
+        json.dump(results, f, indent=2)
+
+    # Figure: paired distribution comparison
+    fig, axes = plt.subplots(1, 3, figsize=(16, 5))
+
+    # Panel 1: histogram of per-cell correlations
+    bins = np.linspace(-0.5, 1.0, 50)
+    axes[0].hist(gamma_corrs[valid_cells], bins=bins, alpha=0.6,
+                 label=f"scPTR gamma (mean={mean_gamma_corr:.3f})",
+                 color="steelblue", edgecolor="white")
+    axes[0].hist(raw_corrs[valid_cells], bins=bins, alpha=0.6,
+                 label=f"Raw u/s (mean={mean_raw_corr:.3f})",
+                 color="salmon", edgecolor="white")
+    axes[0].set_xlabel("Per-cell Spearman r with ground truth")
+    axes[0].set_ylabel("Number of cells")
+    axes[0].set_title(f"Per-Cell Correlation with Ground Truth\n"
+                      f"(Wilcoxon p={wilcox_p:.2e})")
+    axes[0].legend(fontsize=8)
+
+    # Panel 2: scatter gamma_corr vs raw_corr
+    axes[1].scatter(raw_corrs[valid_cells], gamma_corrs[valid_cells],
+                    alpha=0.1, s=3, c="steelblue")
+    lims = [min(axes[1].get_xlim()[0], axes[1].get_ylim()[0]),
+            max(axes[1].get_xlim()[1], axes[1].get_ylim()[1])]
+    axes[1].plot(lims, lims, "k--", alpha=0.3, lw=1)
+    axes[1].set_xlabel("Raw u/s per-cell r")
+    axes[1].set_ylabel("scPTR gamma per-cell r")
+    axes[1].set_title(f"gamma better: {gamma_better}/{n_valid_cells} "
+                      f"({100*gamma_better/n_valid_cells:.0f}%)")
+
+    # Panel 3: difference distribution
+    diff = gamma_corrs[valid_cells] - raw_corrs[valid_cells]
+    axes[2].hist(diff, bins=50, color="steelblue", alpha=0.8, edgecolor="white")
+    axes[2].axvline(0, color="red", linestyle="--", alpha=0.5)
+    axes[2].axvline(np.mean(diff), color="black", linestyle="-", alpha=0.8,
+                    label=f"Mean diff={np.mean(diff):.4f}")
+    axes[2].set_xlabel("Difference (gamma r - raw r)")
+    axes[2].set_ylabel("Number of cells")
+    axes[2].set_title("Per-Cell Improvement")
+    axes[2].legend(fontsize=8)
+
+    fig.suptitle("Per-Cell sci-fate Ablation: scPTR gamma vs Raw u/s Ratio",
+                fontsize=13, y=1.02)
+    fig.tight_layout()
+    save_fig(fig, "per_cell_scifate")
+
+    return results
+
+
+# =========================================================================
+# FIX E: Biological Coherence Ablation
+# =========================================================================
+def fix_e_coherence_ablation():
+    """Run GSEA on sub-clusters from each method to test biological coherence."""
+    print("\n" + "=" * 60)
+    print("FIX E: BIOLOGICAL COHERENCE ABLATION")
+    print("=" * 60)
+
+    from sklearn.decomposition import PCA
+    from sklearn.cluster import KMeans
+    from sklearn.metrics import silhouette_score
+    from statsmodels.stats.multitest import multipletests
+
+    res_dir = OUTPUT_DIR / "results"
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    # Expected tissue-appropriate pathways
+    expected_pathways = {
+        "pancreas": [
+            "endoplasmic reticulum", "autophagy", "protein folding",
+            "unfolded protein", "er stress", "insulin", "secretion",
+            "pancrea", "endocrine", "exocrine",
+        ],
+        "dentate_gyrus": [
+            "synaptic", "long-term potentiation", "spliceosome", "neuron",
+            "axon", "dendrite", "glutamat", "gaba", "hippocampus",
+            "neurogenesis", "myelination",
+        ],
+    }
+
+    all_results = []
+    pathway_details = []
+
+    for dataset_name in ["pancreas", "dentate_gyrus"]:
+        print(f"\n--- {dataset_name} ---")
+
+        # Load dataset
+        if dataset_name == "pancreas":
+            adata = scptr.datasets.pancreas()
+        else:
+            adata = scptr.datasets.dentate_gyrus()
+
+        adata = run_pipeline(adata, dataset_name)
+
+        gamma = adata.layers["gamma"]
+        clusters = adata.obs["clusters"]
+
+        # Get layers for ablation methods
+        u_layer = adata.layers.get("Mu", adata.layers.get("unspliced"))
+        s_layer = adata.layers.get("Ms", adata.layers.get("spliced"))
+        u = u_layer.toarray() if hasattr(u_layer, 'toarray') else np.asarray(u_layer)
+        s = s_layer.toarray() if hasattr(s_layer, 'toarray') else np.asarray(s_layer)
+        expr = adata.X.toarray() if hasattr(adata.X, 'toarray') else np.asarray(adata.X)
+
+        # Raw u/s ratio
+        raw_ratio = np.zeros_like(gamma)
+        s_safe = np.where(s > 0.01, s, 1.0)
+        raw_ratio = u / s_safe
+        raw_ratio[s < 0.01] = 0
+
+        methods = {
+            "scPTR_gamma": gamma,
+            "raw_u_s_ratio": raw_ratio,
+            "unspliced_only": u,
+        }
+
+        # Load UTR features for UTR length enrichment test
+        utr_df = pd.read_csv(
+            PROJECT_ROOT / "src" / "scptr" / "benchmark" / "data" / "mouse_utr_features.csv")
+        utr_map = {row["gene"].upper(): row for _, row in utr_df.iterrows()}
+
+        # Determine organism for GSEA
+        sample_gene = adata.var_names[0]
+        organism = "mouse" if sample_gene[0].isupper() and sample_gene[1:].islower() else "human"
+
+        for cluster_name in sorted(clusters.unique()):
+            mask = (clusters == cluster_name).values
+            n_cells = mask.sum()
+            if n_cells < 50:
+                continue
+
+            # Pre-compute expression PCA for invisibility check
+            expr_sub = expr[mask]
+            nonzero_expr = (expr_sub > 0).mean(axis=0)
+            good_expr = nonzero_expr >= 0.05
+            if good_expr.sum() < 20:
+                continue
+            n_expr_pcs = min(15, n_cells - 1, good_expr.sum() - 1)
+            pca_expr = PCA(n_components=n_expr_pcs, random_state=42)
+            expr_pcs = pca_expr.fit_transform(expr_sub[:, good_expr])
+
+            # Check if ANY method finds invisible sub-clusters
+            any_invisible = False
+            for method_name, data in methods.items():
+                data_sub = data[mask]
+                nonzero = (data_sub > 0).mean(axis=0)
+                good = nonzero >= 0.05
+                if good.sum() < 20:
+                    continue
+                data_filtered = data_sub[:, good]
+                n_pcs = min(15, n_cells - 1, data_filtered.shape[1] - 1)
+                pca = PCA(n_components=n_pcs, random_state=42)
+                pcs = pca.fit_transform(data_filtered)
+
+                for k in [2, 3]:
+                    if n_cells < k * 10:
+                        continue
+                    km = KMeans(n_clusters=k, random_state=42, n_init=10)
+                    labels = km.fit_predict(pcs)
+                    if min(np.bincount(labels)) < 10:
+                        continue
+                    sil = silhouette_score(pcs, labels)
+                    sil_expr = silhouette_score(expr_pcs, labels)
+                    if sil - sil_expr > 0.05:
+                        any_invisible = True
+                        break
+                if any_invisible:
+                    break
+
+            if not any_invisible:
+                continue
+
+            print(f"\n  {cluster_name} ({n_cells} cells) — invisible in at least one method")
+
+            for method_name, data in methods.items():
+                data_sub = data[mask]
+                nonzero = (data_sub > 0).mean(axis=0)
+                good = nonzero >= 0.05
+                if good.sum() < 20:
+                    continue
+
+                data_filtered = data_sub[:, good]
+                gene_names_filtered = adata.var_names[good]
+                n_pcs = min(15, n_cells - 1, data_filtered.shape[1] - 1)
+                pca = PCA(n_components=n_pcs, random_state=42)
+                pcs = pca.fit_transform(data_filtered)
+
+                best_sil = -1
+                best_labels = None
+                best_k = 1
+                for k in [2, 3]:
+                    if n_cells < k * 10:
+                        continue
+                    km = KMeans(n_clusters=k, random_state=42, n_init=10)
+                    labels = km.fit_predict(pcs)
+                    if min(np.bincount(labels)) < 10:
+                        continue
+                    sil = silhouette_score(pcs, labels)
+                    if sil > best_sil:
+                        best_sil = sil
+                        best_labels = labels
+                        best_k = k
+
+                if best_labels is None or best_k <= 1:
+                    continue
+
+                sil_expr_val = silhouette_score(expr_pcs, best_labels)
+                invisibility = best_sil - sil_expr_val
+
+                # Find differentially degraded genes between sub-clusters
+                diff_results = []
+                for gi, gene in enumerate(gene_names_filtered):
+                    groups = [data_filtered[best_labels == j, gi] for j in range(best_k)]
+                    if all(len(g) >= 5 for g in groups):
+                        if best_k == 2:
+                            _, p_val = stats.mannwhitneyu(groups[0], groups[1],
+                                                          alternative='two-sided')
+                        else:
+                            _, p_val = stats.kruskal(*groups)
+
+                        medians = [np.median(g) for g in groups]
+                        max_med = max(medians)
+                        min_med = min(medians)
+                        log_fc = np.log2((max_med + 0.01) / (min_med + 0.01))
+                        diff_results.append({"gene": gene, "p_value": p_val,
+                                             "log2_fc": log_fc})
+
+                if not diff_results:
+                    continue
+
+                diff_df = pd.DataFrame(diff_results)
+                _, diff_df["fdr"], _, _ = multipletests(diff_df["p_value"], method="fdr_bh")
+                sig_genes = diff_df[diff_df["fdr"] < 0.05].sort_values("log2_fc", ascending=False)
+
+                gene_list = sig_genes["gene"].tolist()
+
+                # UTR length enrichment: sig genes vs background
+                sig_utr_lengths = []
+                bg_utr_lengths = []
+                for g in gene_list:
+                    if g.upper() in utr_map:
+                        sig_utr_lengths.append(utr_map[g.upper()]["utr_length"])
+                for g in adata.var_names:
+                    if g.upper() in utr_map:
+                        bg_utr_lengths.append(utr_map[g.upper()]["utr_length"])
+
+                utr_p = np.nan
+                if len(sig_utr_lengths) >= 5 and len(bg_utr_lengths) >= 5:
+                    _, utr_p = stats.mannwhitneyu(
+                        sig_utr_lengths, bg_utr_lengths, alternative="greater")
+
+                # Run GSEA via gseapy Enrichr API
+                n_sig_pathways = 0
+                n_expected_pathways = 0
+                pathway_terms = []
+
+                if len(gene_list) >= 5:
+                    try:
+                        import gseapy as gp
+                        gene_sets = ["GO_Biological_Process_2023",
+                                     "KEGG_2019_Mouse" if organism == "mouse" else "KEGG_2021_Human"]
+
+                        enr = gp.enrichr(gene_list=gene_list,
+                                         gene_sets=gene_sets,
+                                         organism=organism,
+                                         outdir=None,
+                                         no_plot=True)
+
+                        enr_df = enr.results
+                        sig_enr = enr_df[enr_df["Adjusted P-value"] < 0.1]
+                        n_sig_pathways = len(sig_enr)
+
+                        # Check for expected tissue pathways
+                        expected = expected_pathways.get(dataset_name, [])
+                        for _, row in sig_enr.iterrows():
+                            term_lower = row["Term"].lower()
+                            pathway_terms.append(row["Term"])
+                            for kw in expected:
+                                if kw in term_lower:
+                                    n_expected_pathways += 1
+                                    break
+
+                    except Exception as e:
+                        print(f"    [WARNING] GSEA failed for {method_name}/{cluster_name}: {e}")
+
+                result_entry = {
+                    "dataset": dataset_name,
+                    "cluster": cluster_name,
+                    "method": method_name,
+                    "n_cells": int(n_cells),
+                    "n_subclusters": int(best_k),
+                    "sil_method": float(best_sil),
+                    "sil_expr": float(sil_expr_val),
+                    "invisibility": float(invisibility),
+                    "n_diff_genes": int(len(sig_genes)),
+                    "n_sig_pathways": int(n_sig_pathways),
+                    "n_expected_pathways": int(n_expected_pathways),
+                    "mean_utr_length_sig": float(np.mean(sig_utr_lengths)) if sig_utr_lengths else None,
+                    "mean_utr_length_bg": float(np.mean(bg_utr_lengths)) if bg_utr_lengths else None,
+                    "utr_enrichment_p": float(utr_p) if np.isfinite(utr_p) else None,
+                }
+                all_results.append(result_entry)
+
+                if pathway_terms:
+                    for term in pathway_terms[:5]:
+                        pathway_details.append({
+                            "dataset": dataset_name,
+                            "cluster": cluster_name,
+                            "method": method_name,
+                            "pathway": term,
+                        })
+
+                print(f"    {method_name}: sil={best_sil:.3f}, invis={invisibility:.3f}, "
+                      f"diff_genes={len(sig_genes)}, sig_pathways={n_sig_pathways}, "
+                      f"expected={n_expected_pathways}")
+
+    results_df = pd.DataFrame(all_results)
+    results_df.to_csv(res_dir / "coherence_ablation.csv", index=False)
+
+    if pathway_details:
+        pd.DataFrame(pathway_details).to_csv(
+            res_dir / "coherence_ablation_pathways.csv", index=False)
+
+    # Summary
+    if len(results_df) > 0:
+        print("\n  Summary: mean metrics by method")
+        summary = results_df.groupby("method").agg(
+            mean_invisibility=("invisibility", "mean"),
+            mean_sig_pathways=("n_sig_pathways", "mean"),
+            total_sig_pathways=("n_sig_pathways", "sum"),
+            mean_expected=("n_expected_pathways", "mean"),
+            total_expected=("n_expected_pathways", "sum"),
+            mean_diff_genes=("n_diff_genes", "mean"),
+        )
+        for method, row in summary.iterrows():
+            print(f"    {method:<20s}: pathways={row['total_sig_pathways']:.0f} "
+                  f"(expected={row['total_expected']:.0f}), "
+                  f"diff_genes={row['mean_diff_genes']:.0f}, "
+                  f"invis={row['mean_invisibility']:.3f}")
+
+    # Save JSON summary
+    json_results = {
+        "n_clusters_tested": len(results_df["cluster"].unique()) if len(results_df) > 0 else 0,
+        "summary_by_method": {},
+    }
+    if len(results_df) > 0:
+        for method in ["scPTR_gamma", "raw_u_s_ratio", "unspliced_only"]:
+            sub = results_df[results_df["method"] == method]
+            if len(sub) > 0:
+                json_results["summary_by_method"][method] = {
+                    "n_clusters": int(len(sub)),
+                    "mean_invisibility": float(sub["invisibility"].mean()),
+                    "total_sig_pathways": int(sub["n_sig_pathways"].sum()),
+                    "total_expected_pathways": int(sub["n_expected_pathways"].sum()),
+                    "mean_diff_genes": float(sub["n_diff_genes"].mean()),
+                }
+
+    with open(res_dir / "coherence_ablation.json", "w") as f:
+        json.dump(json_results, f, indent=2)
+
+    # Figure
+    if len(results_df) > 0:
+        methods_order = ["unspliced_only", "raw_u_s_ratio", "scPTR_gamma"]
+        method_labels = ["Unspliced\nonly", "Raw u/s\nratio", "scPTR\ngamma"]
+        colors = ["lightblue", "orange", "steelblue"]
+
+        fig, axes = plt.subplots(1, 3, figsize=(15, 5))
+
+        # Panel 1: total significant GSEA pathways
+        vals = []
+        for m in methods_order:
+            sub = results_df[results_df["method"] == m]
+            vals.append(sub["n_sig_pathways"].sum() if len(sub) > 0 else 0)
+        axes[0].bar(method_labels, vals, color=colors, edgecolor="black", linewidth=0.5)
+        axes[0].set_ylabel("Total significant pathways (FDR<0.1)")
+        axes[0].set_title("GSEA Pathway Enrichment")
+        for i, v in enumerate(vals):
+            axes[0].text(i, v + 0.3, str(int(v)), ha="center", fontsize=10, fontweight="bold")
+
+        # Panel 2: expected tissue pathways
+        vals_exp = []
+        for m in methods_order:
+            sub = results_df[results_df["method"] == m]
+            vals_exp.append(sub["n_expected_pathways"].sum() if len(sub) > 0 else 0)
+        axes[1].bar(method_labels, vals_exp, color=colors, edgecolor="black", linewidth=0.5)
+        axes[1].set_ylabel("Tissue-appropriate pathways found")
+        axes[1].set_title("Expected Pathway Hits")
+        for i, v in enumerate(vals_exp):
+            axes[1].text(i, v + 0.2, str(int(v)), ha="center", fontsize=10, fontweight="bold")
+
+        # Panel 3: mean invisibility
+        vals_inv = []
+        for m in methods_order:
+            sub = results_df[results_df["method"] == m]
+            vals_inv.append(sub["invisibility"].mean() if len(sub) > 0 else 0)
+        axes[2].bar(method_labels, vals_inv, color=colors, edgecolor="black", linewidth=0.5)
+        axes[2].set_ylabel("Mean invisibility score")
+        axes[2].set_title("Invisibility Score")
+        axes[2].axhline(0, color="gray", linestyle="--", alpha=0.3)
+
+        fig.suptitle("Biological Coherence Ablation", fontsize=13, y=1.02)
+        fig.tight_layout()
+        save_fig(fig, "coherence_ablation")
+
+    return json_results
+
+
+# =========================================================================
+# MAIN
+# =========================================================================
+def main():
+    set_figure_style()
+    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+    (OUTPUT_DIR / "results").mkdir(parents=True, exist_ok=True)
+    (OUTPUT_DIR / "figures").mkdir(parents=True, exist_ok=True)
+
+    all_results = {}
+
+    # Fix B (fastest — CSV only)
+    print("\n" + "#" * 60)
+    print("# FIX B: 3' UTR SEQUENCE VALIDATION")
+    print("#" * 60)
+    all_results["fix_b_utr"] = fix_b_utr_validation()
+
+    # Fix D (fast — CSV only)
+    print("\n" + "#" * 60)
+    print("# FIX D: CROSS-DATASET HUB CONSISTENCY")
+    print("#" * 60)
+    all_results["fix_d_hub_consistency"] = fix_d_hub_consistency()
+
+    # Fix C (moderate — loads large CSV)
+    print("\n" + "#" * 60)
+    print("# FIX C: NEUROBLASTOMA-SPECIFIC DepMap")
+    print("#" * 60)
+    all_results["fix_c_nb_depmap"] = fix_c_nb_depmap()
+
+    # Fix A (moderate — loads sci-fate data)
+    print("\n" + "#" * 60)
+    print("# FIX A: PER-CELL SCI-FATE ABLATION")
+    print("#" * 60)
+    all_results["fix_a_per_cell"] = fix_a_per_cell_scifate()
+
+    # Fix E (slowest — loads 2 datasets + GSEA API)
+    print("\n" + "#" * 60)
+    print("# FIX E: BIOLOGICAL COHERENCE ABLATION")
+    print("#" * 60)
+    all_results["fix_e_coherence"] = fix_e_coherence_ablation()
+
+    # Save combined results
+    with open(OUTPUT_DIR / "results" / "all_comprehensive_fixes.json", "w") as f:
+        json.dump(all_results, f, indent=2, default=str)
+
+    print("\n" + "=" * 60)
+    print("ALL COMPREHENSIVE FIXES COMPLETE")
+    print("=" * 60)
+    print(f"Results saved to: {OUTPUT_DIR.resolve()}")
+
+
+if __name__ == "__main__":
+    main()

analyses/run_comprehensive_improvements.py

@@ -0,0 +1,1251 @@
+#!/usr/bin/env python
+"""Comprehensive improvements addressing 5 remaining weaknesses.
+
+Experiment A: Network Target GO Enrichment (Weakness #1 — network validation)
+Experiment B: Pathway-Level Cross-Dataset Consistency (Weakness #4 — low gene-level r)
+Experiment C: Gamma vs Raw u/s on Downstream Tasks (Weakness #2 — marginal advantage)
+Experiment D: NB Network Split-Half Robustness (Weakness #3 — single patient)
+Experiment E: Corrected vs Uncorrected Network Quality (Weakness #5 — destabilizing bias)
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+import warnings
+from pathlib import Path
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+from scipy import stats
+from scipy.stats import hypergeom
+
+sys.path.insert(0, str(Path(__file__).parent))
+from _common import set_figure_style
+
+import scptr
+
+# Force unbuffered stdout for progress visibility
+sys.stdout.reconfigure(line_buffering=True)
+
+OUTPUT_DIR = Path(__file__).parent.parent / "output" / "comprehensive_improvements"
+CACHE_DIR = Path(__file__).parent.parent / ".cache"
+WEAKNESS_DIR = Path(__file__).parent.parent / "output" / "weakness_fixes" / "results"
+TIER3_DIR = Path(__file__).parent.parent / "output" / "tier3" / "results"
+
+
+def load_go_library():
+    """Load GO BP gene sets from local cache (no network calls)."""
+    cache_file = CACHE_DIR / "go_bp_2023.json"
+    if cache_file.exists():
+        with open(cache_file) as f:
+            go_lib = json.load(f)
+        return go_lib
+
+    # Fallback: try to download and cache
+    try:
+        import gseapy as gp
+        go_lib = gp.get_library("GO_Biological_Process_2023")
+        with open(cache_file, "w") as f:
+            json.dump(go_lib, f)
+        return go_lib
+    except Exception as e:
+        print(f"  Failed to load GO library: {e}")
+        return None
+
+
+def hypergeometric_enrichment(gene_list, go_lib, background_size, alpha=0.05):
+    """Run local hypergeometric GO enrichment (no API calls).
+
+    Returns list of (term, p_value, overlap, term_size) for significant terms.
+    """
+    gene_set = set(g.upper() for g in gene_list)
+    k = len(gene_set)  # drawn genes
+    N = background_size  # population size
+
+    results = []
+    for term_name, term_genes in go_lib.items():
+        term_upper = set(g.upper() for g in term_genes)
+        K = len(term_upper)  # successes in population
+        if K < 5 or K > N * 0.5:  # skip very small or very large terms
+            continue
+        overlap = gene_set & term_upper
+        x = len(overlap)
+        if x < 2:
+            continue
+        # P(X >= x) under hypergeometric
+        p_val = hypergeom.sf(x - 1, N, K, k)
+        results.append((term_name, p_val, x, K))
+
+    # BH correction
+    if not results:
+        return []
+    results.sort(key=lambda r: r[1])
+    n_tests = len(results)
+    corrected = []
+    for i, (term, p, overlap, size) in enumerate(results):
+        adj_p = p * n_tests / (i + 1)
+        corrected.append((term, adj_p, overlap, size))
+
+    # Enforce monotonicity
+    min_p = 1.0
+    for i in range(len(corrected) - 1, -1, -1):
+        min_p = min(min_p, corrected[i][1])
+        corrected[i] = (corrected[i][0], min_p, corrected[i][2], corrected[i][3])
+
+    sig = [(t, p, o, s) for t, p, o, s in corrected if p < alpha]
+    return sig
+
+
+def save_fig(fig, name, subdir="figures"):
+    out_dir = OUTPUT_DIR / subdir
+    out_dir.mkdir(parents=True, exist_ok=True)
+    path = out_dir / f"{name}.png"
+    fig.savefig(path, dpi=150, bbox_inches="tight")
+    plt.close(fig)
+    print(f"  Saved: {path}")
+
+
+def run_pipeline(adata, name):
+    """Run standard scPTR pipeline."""
+    print(f"\n--- Pipeline: {name} ---")
+    scptr.pp.filter_genes(adata)
+    scptr.pp.normalize_layers(adata)
+    scptr.pp.neighbors(adata, n_neighbors=30)
+    scptr.pp.smooth_layers(adata)
+    scptr.tl.estimate_beta(adata)
+    scptr.tl.estimate_gamma(adata)
+    scptr.tl.variance_decomposition(adata)
+    scptr.tl.pt_states(adata)
+    scptr.tl.pt_velocity(adata)
+    print(f"  Done: {adata.shape}")
+    return adata
+
+
+def get_expression(adata):
+    if hasattr(adata.X, 'toarray'):
+        return adata.X.toarray()
+    return np.asarray(adata.X)
+
+
+def get_rbps_in_data(adata):
+    rbp_path = Path(__file__).parent.parent / "src" / "scptr" / "tools" / "data" / "known_rbps.csv"
+    rbps = pd.read_csv(rbp_path)["gene_symbol"].tolist()
+    gene_map = {g.upper(): i for i, g in enumerate(adata.var_names)}
+    result = {}
+    for r in rbps:
+        if r.upper() in gene_map:
+            result[r.upper()] = gene_map[r.upper()]
+    return result
+
+
+def get_target_indices(adata, n_targets=200):
+    gamma = adata.layers["gamma"]
+    nonzero_frac = (gamma > 0).mean(axis=0)
+    informative = nonzero_frac >= 0.1
+    gamma_var = np.var(gamma[:, informative], axis=0)
+    n = min(n_targets, informative.sum())
+    top_idx = np.argsort(gamma_var)[-n:]
+    return np.where(informative)[0][top_idx]
+
+
+# =========================================================================
+# EXPERIMENT A: Network Target GO Enrichment
+# =========================================================================
+def experiment_a_go_enrichment():
+    """Test whether predicted RBP targets share biological functions (GO enrichment).
+
+    Uses local hypergeometric tests with cached GO BP gene sets — no API calls.
+    """
+    print(f"\n{'='*60}")
+    print("EXPERIMENT A: NETWORK TARGET GO ENRICHMENT")
+    print(f"{'='*60}")
+
+    res_dir = OUTPUT_DIR / "results"
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    # Load GO gene sets from local cache
+    print("  Loading GO Biological Process gene sets (local cache)...")
+    go_lib = load_go_library()
+    if go_lib is None:
+        return None
+    print(f"  Loaded {len(go_lib)} GO BP terms")
+
+    # Load corrected networks
+    networks = {}
+    network_files = {
+        "pancreas": WEAKNESS_DIR / "corrected_network_pancreas.csv",
+        "dentate_gyrus": WEAKNESS_DIR / "corrected_network_dentate_gyrus.csv",
+        "neuroblastoma": TIER3_DIR / "neuroblastoma_network_corrected.csv",
+    }
+
+    for name, path in network_files.items():
+        if path.exists():
+            df = pd.read_csv(path)
+            print(f"  {name}: {len(df)} edges")
+            networks[name] = df
+        else:
+            print(f"  {name}: file not found at {path}")
+
+    # Estimate background gene count per organism
+    # Use ~20,000 as a reasonable genome-wide background
+    BACKGROUND_SIZE = 20000
+
+    all_results = {}
+
+    for ds_name, edges_df in networks.items():
+        print(f"\n  --- {ds_name} ---")
+
+        rbp_col = "rbp"
+        target_col = "target"
+
+        # Build RBP -> target sets
+        rbp_targets = {}
+        for rbp, grp in edges_df.groupby(rbp_col):
+            targets = set(grp[target_col].tolist())
+            rbp_targets[rbp] = targets
+
+        # Background gene set (all unique targets in network)
+        all_targets = set()
+        for t in rbp_targets.values():
+            all_targets |= t
+
+        # Filter to RBPs with >= 10 targets
+        eligible_rbps = {r: t for r, t in rbp_targets.items() if len(t) >= 10}
+        print(f"    RBPs with >= 10 targets: {len(eligible_rbps)}")
+
+        if not eligible_rbps:
+            all_results[ds_name] = {"n_eligible_rbps": 0}
+            continue
+
+        # Run local hypergeometric enrichment for each eligible RBP
+        rbp_enrichment_results = []
+        n_with_sig = 0
+
+        for rbp, targets in eligible_rbps.items():
+            gene_list = list(targets)
+            sig_terms = hypergeometric_enrichment(gene_list, go_lib, BACKGROUND_SIZE)
+            n_sig = len(sig_terms)
+            has_sig = n_sig > 0
+            if has_sig:
+                n_with_sig += 1
+
+            top_terms = [t[0] for t in sig_terms[:5]]
+
+            rbp_enrichment_results.append({
+                "rbp": rbp,
+                "n_targets": len(targets),
+                "n_sig_terms": n_sig,
+                "has_sig": has_sig,
+                "top_terms": top_terms,
+            })
+
+        frac_with_sig = n_with_sig / max(len(eligible_rbps), 1)
+        print(f"    RBPs with >= 1 significant GO term: {n_with_sig}/{len(eligible_rbps)} ({frac_with_sig:.1%})")
+
+        # Known biology concordance
+        known_biology = {
+            "ELAVL1": ["mRNA stability", "mRNA stabilization", "RNA stability"],
+            "RBFOX1": ["neuron", "neuronal", "synap", "axon"],
+            "RBFOX2": ["neuron", "neuronal", "synap", "splicing"],
+            "RBFOX3": ["neuron", "neuronal", "synap"],
+            "SRSF3": ["splic", "mRNA processing", "RNA processing"],
+            "HNRNPA1": ["splic", "mRNA processing", "RNA processing"],
+            "YBX1": ["translation", "mRNA", "RNA"],
+            "CELF2": ["splic", "neuron", "mRNA"],
+        }
+
+        concordance_hits = []
+        for rbp_res in rbp_enrichment_results:
+            rbp = rbp_res["rbp"]
+            if rbp in known_biology and rbp_res["top_terms"]:
+                expected_keywords = known_biology[rbp]
+                all_terms_str = " ".join(rbp_res["top_terms"]).lower()
+                matched = [kw for kw in expected_keywords if kw.lower() in all_terms_str]
+                if matched:
+                    concordance_hits.append({"rbp": rbp, "matched_keywords": matched})
+                    print(f"    Known biology match: {rbp} -> {matched}")
+
+        # Cross-RBP specificity (Jaccard between enriched GO term sets)
+        enriched_term_sets = {}
+        for rbp_res in rbp_enrichment_results:
+            if rbp_res["top_terms"]:
+                enriched_term_sets[rbp_res["rbp"]] = set(rbp_res["top_terms"])
+
+        jaccard_values = []
+        rbp_list = list(enriched_term_sets.keys())
+        for i in range(len(rbp_list)):
+            for j in range(i + 1, len(rbp_list)):
+                s1 = enriched_term_sets[rbp_list[i]]
+                s2 = enriched_term_sets[rbp_list[j]]
+                union = s1 | s2
+                if union:
+                    jaccard_values.append(len(s1 & s2) / len(union))
+
+        mean_jaccard = np.mean(jaccard_values) if jaccard_values else 0
+        print(f"    Cross-RBP GO term Jaccard (specificity): {mean_jaccard:.3f} (lower = more specific)")
+
+        # Bootstrap null: random gene sets from GENOME-WIDE background
+        # (not from network targets, which are already enriched for biology)
+        print(f"    Running bootstrap null (100 random genome-wide sets per RBP)...")
+        n_bootstrap = 100
+        rng = np.random.RandomState(42)
+        # Build genome-wide gene list from GO library (covers ~20K genes)
+        genome_genes = set()
+        for genes in go_lib.values():
+            genome_genes.update(g.upper() for g in genes)
+        genome_genes_list = sorted(genome_genes)
+        bootstrap_fracs = []
+
+        test_rbps = list(eligible_rbps.items())[:min(10, len(eligible_rbps))]
+        for rbp, targets in test_rbps:
+            n_t = len(targets)
+            null_sig_count = 0
+            for _ in range(n_bootstrap):
+                random_genes = rng.choice(genome_genes_list,
+                                          size=min(n_t, len(genome_genes_list)),
+                                          replace=False).tolist()
+                sig_null = hypergeometric_enrichment(random_genes, go_lib, BACKGROUND_SIZE)
+                if sig_null:
+                    null_sig_count += 1
+            bootstrap_fracs.append(null_sig_count / n_bootstrap)
+
+        mean_null_frac = np.mean(bootstrap_fracs) if bootstrap_fracs else 0
+        print(f"    Bootstrap null fraction with sig GO term: {mean_null_frac:.3f}")
+        print(f"    Enrichment over null: {frac_with_sig / max(mean_null_frac, 0.01):.1f}x")
+
+        all_results[ds_name] = {
+            "n_eligible_rbps": len(eligible_rbps),
+            "n_with_sig_go": n_with_sig,
+            "frac_with_sig_go": float(frac_with_sig),
+            "mean_cross_rbp_jaccard": float(mean_jaccard),
+            "n_known_biology_matches": len(concordance_hits),
+            "concordance_hits": concordance_hits,
+            "bootstrap_null_frac": float(mean_null_frac),
+            "per_rbp": rbp_enrichment_results,
+        }
+
+    # Save results
+    with open(res_dir / "go_enrichment.json", "w") as f:
+        json.dump(all_results, f, indent=2, default=str)
+
+    # Summary figure
+    ds_names = list(all_results.keys())
+    fig, axes = plt.subplots(1, 2, figsize=(12, 5))
+
+    # Panel 1: Fraction with significant GO terms
+    fracs = [all_results[d].get("frac_with_sig_go", 0) for d in ds_names]
+    null_fracs = [all_results[d].get("bootstrap_null_frac", 0) for d in ds_names]
+    x = np.arange(len(ds_names))
+    width = 0.35
+    axes[0].bar(x - width / 2, fracs, width, label="Real RBP targets",
+                color="#1976D2", edgecolor="black", linewidth=0.5)
+    axes[0].bar(x + width / 2, null_fracs, width, label="Random gene sets (null)",
+                color="#BDBDBD", edgecolor="black", linewidth=0.5)
+    axes[0].set_xticks(x)
+    axes[0].set_xticklabels(ds_names, fontsize=9)
+    axes[0].set_ylabel("Fraction with >= 1 sig GO term")
+    axes[0].set_title("GO Enrichment: Real vs Random Targets")
+    axes[0].legend(fontsize=8)
+    axes[0].set_ylim(0, 1.1)
+    for i, (f, n) in enumerate(zip(fracs, null_fracs)):
+        axes[0].text(i - width / 2, f + 0.02, f"{f:.0%}", ha="center", fontsize=8)
+        axes[0].text(i + width / 2, n + 0.02, f"{n:.0%}", ha="center", fontsize=8)
+
+    # Panel 2: Cross-RBP Jaccard (specificity)
+    jaccards = [all_results[d].get("mean_cross_rbp_jaccard", 0) for d in ds_names]
+    axes[1].bar(x, jaccards, color="#43A047", edgecolor="black", linewidth=0.5)
+    axes[1].set_xticks(x)
+    axes[1].set_xticklabels(ds_names, fontsize=9)
+    axes[1].set_ylabel("Mean Jaccard (lower = more specific)")
+    axes[1].set_title("Cross-RBP GO Term Specificity")
+    for i, j in enumerate(jaccards):
+        axes[1].text(i, j + 0.005, f"{j:.3f}", ha="center", fontsize=9)
+
+    fig.suptitle("Experiment A: Network Target GO Enrichment", fontsize=13)
+    fig.tight_layout()
+    save_fig(fig, "experiment_a_go_enrichment")
+
+    # Print summary
+    print(f"\n  EXPERIMENT A SUMMARY:")
+    for ds_name, res in all_results.items():
+        print(f"    {ds_name}: {res.get('frac_with_sig_go', 0):.0%} RBPs with sig GO terms "
+              f"(null: {res.get('bootstrap_null_frac', 0):.0%}, "
+              f"concordance: {res.get('n_known_biology_matches', 0)} hits)")
+
+    return all_results
+
+
+# =========================================================================
+# EXPERIMENT B: Pathway-Level Cross-Dataset Consistency
+# =========================================================================
+def experiment_b_pathway_consistency(datasets):
+    """Show pathway-level gamma consistency is higher than gene-level."""
+    print(f"\n{'='*60}")
+    print("EXPERIMENT B: PATHWAY-LEVEL CROSS-DATASET CONSISTENCY")
+    print(f"{'='*60}")
+
+    res_dir = OUTPUT_DIR / "results"
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    # Load GO gene sets from local cache
+    print("  Loading GO Biological Process gene sets (local cache)...")
+    go_lib = load_go_library()
+    if go_lib is None:
+        return None
+    print(f"  Loaded {len(go_lib)} GO BP terms")
+
+    # Compute per-gene median gamma for each dataset
+    gamma_medians = {}
+    for name, adata in datasets.items():
+        gamma = adata.layers["gamma"]
+        gamma_med = np.median(gamma, axis=0)
+        gamma_medians[name] = pd.Series(gamma_med, index=[g.upper() for g in adata.var_names])
+
+    names = sorted(datasets.keys())
+    results = []
+
+    for i, name_a in enumerate(names):
+        for name_b in names[i + 1:]:
+            print(f"\n  --- {name_a} vs {name_b} ---")
+
+            ga = gamma_medians[name_a]
+            gb = gamma_medians[name_b]
+
+            # Shared genes
+            shared = sorted(set(ga.index) & set(gb.index))
+            if len(shared) < 50:
+                continue
+
+            # Gene-level correlation (baseline)
+            ga_shared = ga[shared].values
+            gb_shared = gb[shared].values
+            valid = np.isfinite(ga_shared) & np.isfinite(gb_shared)
+            r_gene, p_gene = stats.spearmanr(ga_shared[valid], gb_shared[valid])
+            print(f"    Gene-level Spearman r: {r_gene:.4f} (n={valid.sum()})")
+
+            # Pathway-level: for each GO term with >= 10 shared genes,
+            # compute mean gamma in each dataset
+            pathway_gamma_a = []
+            pathway_gamma_b = []
+            pathway_names = []
+            pathway_sizes = []
+
+            for term_name, term_genes in go_lib.items():
+                # Convert term genes to uppercase for matching
+                term_genes_upper = set(g.upper() for g in term_genes)
+                term_shared = term_genes_upper & set(shared)
+
+                if len(term_shared) < 10:
+                    continue
+
+                genes_list = sorted(term_shared)
+                idx = [shared.index(g) for g in genes_list]
+
+                mean_a = np.mean(ga_shared[idx])
+                mean_b = np.mean(gb_shared[idx])
+
+                if np.isfinite(mean_a) and np.isfinite(mean_b):
+                    pathway_gamma_a.append(mean_a)
+                    pathway_gamma_b.append(mean_b)
+                    pathway_names.append(term_name)
+                    pathway_sizes.append(len(term_shared))
+
+            if len(pathway_gamma_a) < 20:
+                print(f"    Too few pathways with >= 10 shared genes: {len(pathway_gamma_a)}")
+                continue
+
+            r_pathway, p_pathway = stats.spearmanr(pathway_gamma_a, pathway_gamma_b)
+            print(f"    Pathway-level Spearman r: {r_pathway:.4f} (n={len(pathway_gamma_a)} pathways)")
+            print(f"    Improvement: {r_pathway:.3f} vs {r_gene:.3f} (gene-level)")
+
+            results.append({
+                "pair": f"{name_a} vs {name_b}",
+                "gene_level_r": float(r_gene),
+                "gene_level_p": float(p_gene),
+                "n_shared_genes": int(valid.sum()),
+                "pathway_level_r": float(r_pathway),
+                "pathway_level_p": float(p_pathway),
+                "n_pathways": len(pathway_gamma_a),
+                "mean_pathway_size": float(np.mean(pathway_sizes)),
+            })
+
+    # Save results
+    with open(res_dir / "pathway_consistency.json", "w") as f:
+        json.dump(results, f, indent=2)
+
+    # Summary figure
+    if results:
+        fig, ax = plt.subplots(figsize=(8, 5))
+        pairs = [r["pair"] for r in results]
+        gene_rs = [r["gene_level_r"] for r in results]
+        pathway_rs = [r["pathway_level_r"] for r in results]
+
+        x = np.arange(len(pairs))
+        width = 0.35
+        ax.bar(x - width / 2, gene_rs, width, label="Gene-level",
+               color="#E53935", edgecolor="black", linewidth=0.5)
+        ax.bar(x + width / 2, pathway_rs, width, label="Pathway-level",
+               color="#1976D2", edgecolor="black", linewidth=0.5)
+        ax.set_xticks(x)
+        ax.set_xticklabels([p.replace(" vs ", "\nvs\n") for p in pairs], fontsize=8)
+        ax.set_ylabel("Spearman r")
+        ax.set_title("Gamma Consistency: Gene vs Pathway Level")
+        ax.legend()
+        for i, (g, p) in enumerate(zip(gene_rs, pathway_rs)):
+            ax.text(i - width / 2, g + 0.01, f"{g:.3f}", ha="center", fontsize=8)
+            ax.text(i + width / 2, p + 0.01, f"{p:.3f}", ha="center", fontsize=8)
+
+        fig.tight_layout()
+        save_fig(fig, "experiment_b_pathway_consistency")
+
+    print(f"\n  EXPERIMENT B SUMMARY:")
+    for r in results:
+        print(f"    {r['pair']}: gene r={r['gene_level_r']:.3f} -> pathway r={r['pathway_level_r']:.3f} "
+              f"({r['n_pathways']} pathways)")
+
+    return results
+
+
+# =========================================================================
+# EXPERIMENT C: Gamma vs Raw u/s on Downstream Tasks
+# =========================================================================
+def experiment_c_gamma_advantage(datasets):
+    """Demonstrate gamma's downstream task advantage over raw u/s ratio."""
+    print(f"\n{'='*60}")
+    print("EXPERIMENT C: GAMMA vs RAW U/S ON DOWNSTREAM TASKS")
+    print(f"{'='*60}")
+
+    from sklearn.decomposition import PCA
+    from sklearn.cluster import KMeans
+    from sklearn.metrics import silhouette_score
+
+    res_dir = OUTPUT_DIR / "results"
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    all_results = {}
+
+    for ds_name, adata in datasets.items():
+        if ds_name == "scifate":
+            continue  # Only pancreas and DG have expression clusters for comparison
+        print(f"\n  --- {ds_name} ---")
+
+        gamma = adata.layers["gamma"]
+        Ms = adata.layers["Ms"]
+        Mu = adata.layers["Mu"]
+
+        # Construct smooth_ratio: same as gamma but WITHOUT beta multiplication
+        reliable = Ms >= 0.01
+        smooth_ratio = np.where(reliable, Mu / np.where(reliable, Ms, 1.0), 0.0)
+
+        # Same per-gene 99th percentile clip as gamma
+        for gi in range(smooth_ratio.shape[1]):
+            col = smooth_ratio[:, gi]
+            pos = col[col > 0]
+            if len(pos) > 10:
+                cap = np.percentile(pos, 99)
+                smooth_ratio[:, gi] = np.clip(col, 0, cap)
+
+        # Global cap at 10x 99th percentile of gene medians
+        gene_medians = np.median(smooth_ratio, axis=0)
+        pos_medians = gene_medians[gene_medians > 0]
+        if len(pos_medians) > 0:
+            global_cap = 10 * np.percentile(pos_medians, 99)
+            smooth_ratio = np.clip(smooth_ratio, 0, global_cap)
+
+        print(f"    Gamma shape: {gamma.shape}, max={gamma.max():.4f}")
+        print(f"    Smooth ratio shape: {smooth_ratio.shape}, max={smooth_ratio.max():.4f}")
+
+        # Get expression clusters
+        clusters = adata.obs.get("clusters", adata.obs.get("cell_type"))
+        if clusters is None:
+            print(f"    No cluster labels found, skipping")
+            continue
+        clusters = clusters.astype(str)
+
+        # ----- Task 1: PT State Discovery (Invisible States) -----
+        print(f"\n    Task 1: Invisible State Discovery")
+
+        invisible_results = {"gamma": [], "smooth_ratio": []}
+
+        for method_name, layer_data in [("gamma", gamma), ("smooth_ratio", smooth_ratio)]:
+            for cluster_name in sorted(clusters.unique()):
+                mask = (clusters == cluster_name).values
+                n_cells = mask.sum()
+                if n_cells < 50:
+                    continue
+
+                sub = layer_data[mask]
+                n_pcs = min(15, n_cells - 1, sub.shape[1] - 1)
+                pca = PCA(n_components=n_pcs, random_state=42)
+                pcs = pca.fit_transform(sub)
+
+                best_k, best_sil, best_labels = 1, -1, np.zeros(n_cells, dtype=int)
+                for k in [2, 3]:
+                    if n_cells < k * 10:
+                        continue
+                    km = KMeans(n_clusters=k, random_state=42, n_init=10)
+                    labels = km.fit_predict(pcs)
+                    if min(np.bincount(labels)) < 10:
+                        continue
+                    sil = silhouette_score(pcs, labels)
+                    if sil > best_sil:
+                        best_k, best_sil, best_labels = k, sil, labels
+
+                # Expression silhouette for same labels
+                expr_sub = get_expression(adata)[mask]
+                n_expr_pcs = min(15, n_cells - 1, expr_sub.shape[1] - 1)
+                pca_expr = PCA(n_components=n_expr_pcs, random_state=42)
+                expr_pcs = pca_expr.fit_transform(expr_sub)
+
+                if best_k > 1:
+                    sil_method = best_sil
+                    sil_expr = silhouette_score(expr_pcs, best_labels)
+                else:
+                    sil_method = 0
+                    sil_expr = 0
+
+                is_invisible = sil_method > 0.1 and sil_expr < 0.1
+
+                invisible_results[method_name].append({
+                    "cluster": cluster_name,
+                    "n_cells": n_cells,
+                    "sil_method": float(sil_method),
+                    "sil_expr": float(sil_expr),
+                    "invisibility": float(sil_method - sil_expr),
+                    "is_invisible": is_invisible,
+                })
+
+        # Count invisible states for each method
+        gamma_invisible = sum(1 for r in invisible_results["gamma"] if r["is_invisible"])
+        ratio_invisible = sum(1 for r in invisible_results["smooth_ratio"] if r["is_invisible"])
+        gamma_mean_invis = np.mean([r["invisibility"] for r in invisible_results["gamma"]])
+        ratio_mean_invis = np.mean([r["invisibility"] for r in invisible_results["smooth_ratio"]])
+
+        print(f"    Gamma: {gamma_invisible} invisible states, mean invisibility={gamma_mean_invis:.3f}")
+        print(f"    Smooth ratio: {ratio_invisible} invisible states, mean invisibility={ratio_mean_invis:.3f}")
+
+        # ----- Task 2: Cell-Type Variance Explained (eta-squared) -----
+        print(f"\n    Task 2: Cell-Type Variance Explained (eta-squared)")
+
+        cluster_labels = clusters.values
+        unique_clusters = np.unique(cluster_labels)
+
+        def compute_eta_squared(data, labels, unique_labels):
+            """Compute eta-squared (fraction of variance explained by groups)."""
+            n = data.shape[0]
+            grand_mean = data.mean(axis=0)
+            ss_total = np.sum((data - grand_mean) ** 2, axis=0)
+
+            ss_between = np.zeros(data.shape[1])
+            for cl in unique_labels:
+                mask_cl = labels == cl
+                n_cl = mask_cl.sum()
+                if n_cl == 0:
+                    continue
+                group_mean = data[mask_cl].mean(axis=0)
+                ss_between += n_cl * (group_mean - grand_mean) ** 2
+
+            eta_sq = ss_between / np.clip(ss_total, 1e-10, None)
+            return eta_sq
+
+        eta_gamma = compute_eta_squared(gamma, cluster_labels, unique_clusters)
+        eta_ratio = compute_eta_squared(smooth_ratio, cluster_labels, unique_clusters)
+
+        # Filter to informative genes
+        informative = (gamma > 0).mean(axis=0) >= 0.1
+        eta_gamma_info = eta_gamma[informative]
+        eta_ratio_info = eta_ratio[informative]
+
+        gamma_wins = (eta_gamma_info > eta_ratio_info).sum()
+        ratio_wins = (eta_ratio_info > eta_gamma_info).sum()
+        total = len(eta_gamma_info)
+
+        print(f"    Gamma eta-sq > smooth ratio: {gamma_wins}/{total} ({100*gamma_wins/total:.1f}%)")
+        print(f"    Mean eta-sq — gamma: {eta_gamma_info.mean():.4f}, smooth ratio: {eta_ratio_info.mean():.4f}")
+
+        # Wilcoxon test
+        w_stat, w_p = stats.wilcoxon(eta_gamma_info, eta_ratio_info)
+        print(f"    Wilcoxon signed-rank p: {w_p:.2e}")
+
+        all_results[ds_name] = {
+            "invisible_states": {
+                "gamma_n_invisible": gamma_invisible,
+                "smooth_ratio_n_invisible": ratio_invisible,
+                "gamma_mean_invisibility": float(gamma_mean_invis),
+                "smooth_ratio_mean_invisibility": float(ratio_mean_invis),
+                "per_cluster": invisible_results,
+            },
+            "eta_squared": {
+                "gamma_wins": int(gamma_wins),
+                "ratio_wins": int(ratio_wins),
+                "n_genes": int(total),
+                "gamma_mean": float(eta_gamma_info.mean()),
+                "ratio_mean": float(eta_ratio_info.mean()),
+                "wilcoxon_p": float(w_p),
+            },
+        }
+
+    # Save results
+    with open(res_dir / "gamma_advantage.json", "w") as f:
+        json.dump(all_results, f, indent=2, default=str)
+
+    # Summary figure
+    fig, axes = plt.subplots(1, 2, figsize=(13, 5))
+
+    # Panel 1: Invisible state counts
+    ds_labels = list(all_results.keys())
+    gamma_invis = [all_results[d]["invisible_states"]["gamma_n_invisible"] for d in ds_labels]
+    ratio_invis = [all_results[d]["invisible_states"]["smooth_ratio_n_invisible"] for d in ds_labels]
+    x = np.arange(len(ds_labels))
+    width = 0.35
+    axes[0].bar(x - width / 2, gamma_invis, width, label="scPTR gamma",
+                color="#1976D2", edgecolor="black", linewidth=0.5)
+    axes[0].bar(x + width / 2, ratio_invis, width, label="Smooth u/s ratio (no beta)",
+                color="#E53935", edgecolor="black", linewidth=0.5)
+    axes[0].set_xticks(x)
+    axes[0].set_xticklabels(ds_labels, fontsize=9)
+    axes[0].set_ylabel("Number of invisible states")
+    axes[0].set_title("Invisible State Discovery")
+    axes[0].legend(fontsize=8)
+    for i, (g, r) in enumerate(zip(gamma_invis, ratio_invis)):
+        axes[0].text(i - width / 2, g + 0.1, str(g), ha="center", fontsize=9)
+        axes[0].text(i + width / 2, r + 0.1, str(r), ha="center", fontsize=9)
+
+    # Panel 2: Eta-squared comparison
+    gamma_means = [all_results[d]["eta_squared"]["gamma_mean"] for d in ds_labels]
+    ratio_means = [all_results[d]["eta_squared"]["ratio_mean"] for d in ds_labels]
+    axes[1].bar(x - width / 2, gamma_means, width, label="scPTR gamma",
+                color="#1976D2", edgecolor="black", linewidth=0.5)
+    axes[1].bar(x + width / 2, ratio_means, width, label="Smooth u/s ratio",
+                color="#E53935", edgecolor="black", linewidth=0.5)
+    axes[1].set_xticks(x)
+    axes[1].set_xticklabels(ds_labels, fontsize=9)
+    axes[1].set_ylabel("Mean eta-squared")
+    axes[1].set_title("Cell-Type Variance Explained")
+    axes[1].legend(fontsize=8)
+    for i, (g, r) in enumerate(zip(gamma_means, ratio_means)):
+        axes[1].text(i - width / 2, g + 0.001, f"{g:.4f}", ha="center", fontsize=8)
+        axes[1].text(i + width / 2, r + 0.001, f"{r:.4f}", ha="center", fontsize=8)
+
+    fig.suptitle("Experiment C: Gamma vs Smooth Ratio Downstream Tasks", fontsize=13)
+    fig.tight_layout()
+    save_fig(fig, "experiment_c_gamma_advantage")
+
+    print(f"\n  EXPERIMENT C SUMMARY:")
+    for ds_name, res in all_results.items():
+        inv = res["invisible_states"]
+        eta = res["eta_squared"]
+        print(f"    {ds_name}: invisible states gamma={inv['gamma_n_invisible']} "
+              f"vs ratio={inv['smooth_ratio_n_invisible']}; "
+              f"eta-sq gamma={eta['gamma_mean']:.4f} vs ratio={eta['ratio_mean']:.4f} "
+              f"(p={eta['wilcoxon_p']:.2e})")
+
+    return all_results
+
+
+# =========================================================================
+# EXPERIMENT D: NB Network Split-Half Robustness
+# =========================================================================
+def experiment_d_nb_robustness():
+    """Show NB network is internally robust via split-half cross-validation."""
+    print(f"\n{'='*60}")
+    print("EXPERIMENT D: NB NETWORK SPLIT-HALF ROBUSTNESS")
+    print(f"{'='*60}")
+
+    import scanpy as sc
+
+    res_dir = OUTPUT_DIR / "results"
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    # Load NB data
+    h5ad_path = CACHE_DIR / "neuroblastoma.h5ad"
+    if not h5ad_path.exists():
+        print(f"  NB data not found at {h5ad_path}")
+        return None
+
+    print("  Loading neuroblastoma dataset...")
+    adata_full = sc.read_h5ad(str(h5ad_path))
+    sc.pp.filter_genes(adata_full, min_cells=50)
+    adata_full.layers["raw_spliced"] = adata_full.layers["spliced"].copy()
+    adata_full.layers["raw_unspliced"] = adata_full.layers["unspliced"].copy()
+    print(f"  Full dataset: {adata_full.shape}")
+
+    def run_nb_pipeline(adata):
+        """Run scPTR pipeline on NB data."""
+        scptr.pp.filter_genes(adata)
+        scptr.pp.normalize_layers(adata)
+        scptr.pp.neighbors(adata, n_neighbors=30)
+        scptr.pp.smooth_layers(adata)
+        scptr.tl.estimate_beta(adata)
+        scptr.tl.estimate_gamma(adata)
+        return adata
+
+    def infer_network(adata):
+        """Run partial-correlation network inference (library-size corrected)."""
+        gamma = adata.layers["gamma"]
+        expr = get_expression(adata)
+        rbps = get_rbps_in_data(adata)
+        n_cells = adata.n_obs
+
+        # Library size
+        lib_size = expr.sum(axis=1)
+        lib_rank = stats.rankdata(lib_size)
+        lib_rank_centered = lib_rank - lib_rank.mean()
+        lib_ss = np.dot(lib_rank_centered, lib_rank_centered)
+
+        if lib_ss < 1e-10:
+            return pd.DataFrame()
+
+        # Target indices
+        informative = (gamma > 0).mean(axis=0) >= 0.1
+        if informative.sum() < 20:
+            return pd.DataFrame()
+        gamma_var = np.var(gamma[:, informative], axis=0)
+        n_targets = min(200, informative.sum())
+        top_var_idx = np.argsort(gamma_var)[-n_targets:]
+        info_indices = np.where(informative)[0]
+        target_indices = info_indices[top_var_idx]
+
+        # Pre-compute residualized gamma ranks
+        gamma_resid_map = {}
+        for ti in target_indices:
+            t_gamma = gamma[:, ti]
+            if np.std(t_gamma) < 1e-8:
+                continue
+            t_rank = stats.rankdata(t_gamma)
+            t_rank_c = t_rank - t_rank.mean()
+            slope = np.dot(lib_rank_centered, t_rank_c) / lib_ss
+            resid = t_rank - slope * lib_rank
+            resid_c = resid - resid.mean()
+            resid_std = np.sqrt(np.dot(resid_c, resid_c))
+            if resid_std > 1e-8:
+                gamma_resid_map[ti] = (resid_c, resid_std)
+
+        edges = []
+        for rbp_upper, rbp_idx in rbps.items():
+            rbp_expr = expr[:, rbp_idx]
+            if np.std(rbp_expr) < 1e-6:
+                continue
+
+            rbp_rank = stats.rankdata(rbp_expr)
+            rbp_rank_c = rbp_rank - rbp_rank.mean()
+            slope_rbp = np.dot(lib_rank_centered, rbp_rank_c) / lib_ss
+            rbp_resid = rbp_rank - slope_rbp * lib_rank
+            rbp_resid_c = rbp_resid - rbp_resid.mean()
+            rbp_resid_std = np.sqrt(np.dot(rbp_resid_c, rbp_resid_c))
+            if rbp_resid_std < 1e-8:
+                continue
+
+            for ti in target_indices:
+                if ti not in gamma_resid_map:
+                    continue
+                g_resid_c, g_resid_std = gamma_resid_map[ti]
+                r_corr = np.dot(rbp_resid_c, g_resid_c) / (rbp_resid_std * g_resid_std)
+                r_corr = np.clip(r_corr, -1.0, 1.0)
+                df = n_cells - 3
+                t_val = r_corr * np.sqrt(df / (1 - r_corr ** 2 + 1e-12))
+                p_corr = 2 * stats.t.sf(abs(t_val), df)
+
+                if p_corr < 0.05 / (len(rbps) * n_targets):
+                    edges.append({
+                        "rbp": rbp_upper,
+                        "target": adata.var_names[ti],
+                        "r": float(r_corr),
+                    })
+
+        return pd.DataFrame(edges) if edges else pd.DataFrame(columns=["rbp", "target", "r"])
+
+    def get_top_hubs(edges_df, n=20):
+        if len(edges_df) == 0:
+            return []
+        hub_counts = edges_df.groupby("rbp").size().sort_values(ascending=False)
+        return list(hub_counts.head(n).index)
+
+    # Run full-data network first
+    print("\n  Running full-data pipeline...")
+    adata_full_processed = adata_full.copy()
+    adata_full_processed = run_nb_pipeline(adata_full_processed)
+    full_edges = infer_network(adata_full_processed)
+    full_hubs = get_top_hubs(full_edges, n=20)
+    full_hub_counts = full_edges.groupby("rbp").size() if len(full_edges) > 0 else pd.Series(dtype=int)
+    print(f"  Full data: {len(full_edges)} edges, top hubs: {full_hubs[:5]}")
+
+    # Split-half replicates
+    n_replicates = 5
+    rng = np.random.RandomState(42)
+    n_cells = adata_full.n_obs
+
+    replicate_results = []
+
+    for rep_i in range(n_replicates):
+        print(f"\n  Replicate {rep_i + 1}/{n_replicates}...")
+
+        # Random split
+        perm = rng.permutation(n_cells)
+        half1_idx = perm[:n_cells // 2]
+        half2_idx = perm[n_cells // 2:]
+
+        half_hubs = []
+        half_hub_counts_list = []
+
+        for half_name, cell_idx in [("half1", half1_idx), ("half2", half2_idx)]:
+            adata_half = adata_full[cell_idx].copy()
+            # Restore raw layers
+            adata_half.layers["spliced"] = adata_half.layers["raw_spliced"].copy()
+            adata_half.layers["unspliced"] = adata_half.layers["raw_unspliced"].copy()
+
+            try:
+                adata_half = run_nb_pipeline(adata_half)
+                edges_half = infer_network(adata_half)
+                hubs = get_top_hubs(edges_half, n=20)
+                hub_counts = edges_half.groupby("rbp").size() if len(edges_half) > 0 else pd.Series(dtype=int)
+                print(f"    {half_name}: {len(edges_half)} edges, {len(hubs)} hubs")
+            except Exception as e:
+                print(f"    {half_name}: pipeline failed: {e}")
+                hubs = []
+                hub_counts = pd.Series(dtype=int)
+
+            half_hubs.append(set(hubs))
+            half_hub_counts_list.append(hub_counts)
+
+        # Compare halves
+        if half_hubs[0] and half_hubs[1]:
+            union = half_hubs[0] | half_hubs[1]
+            intersection = half_hubs[0] & half_hubs[1]
+            jaccard = len(intersection) / len(union) if union else 0
+
+            # Hub count correlation (all shared RBPs)
+            shared_rbps = sorted(set(half_hub_counts_list[0].index) & set(half_hub_counts_list[1].index))
+            if len(shared_rbps) >= 5:
+                c1 = [half_hub_counts_list[0].get(r, 0) for r in shared_rbps]
+                c2 = [half_hub_counts_list[1].get(r, 0) for r in shared_rbps]
+                r_hub, p_hub = stats.spearmanr(c1, c2)
+            else:
+                r_hub, p_hub = np.nan, np.nan
+
+            # Compare each half to full data hubs
+            jaccard_h1_full = len(half_hubs[0] & set(full_hubs)) / len(half_hubs[0] | set(full_hubs)) if (half_hubs[0] | set(full_hubs)) else 0
+            jaccard_h2_full = len(half_hubs[1] & set(full_hubs)) / len(half_hubs[1] | set(full_hubs)) if (half_hubs[1] | set(full_hubs)) else 0
+
+            print(f"    Half-half Jaccard (top-20 hubs): {jaccard:.3f}")
+            print(f"    Hub count Spearman r: {r_hub:.3f}")
+            print(f"    Half1-vs-full Jaccard: {jaccard_h1_full:.3f}, Half2-vs-full: {jaccard_h2_full:.3f}")
+
+            replicate_results.append({
+                "replicate": rep_i + 1,
+                "jaccard_half_half": float(jaccard),
+                "hub_count_spearman_r": float(r_hub) if not np.isnan(r_hub) else None,
+                "jaccard_half1_full": float(jaccard_h1_full),
+                "jaccard_half2_full": float(jaccard_h2_full),
+                "n_shared_rbps": len(shared_rbps),
+                "overlap_hubs": sorted(intersection),
+            })
+        else:
+            replicate_results.append({
+                "replicate": rep_i + 1,
+                "jaccard_half_half": 0,
+                "hub_count_spearman_r": None,
+                "jaccard_half1_full": 0,
+                "jaccard_half2_full": 0,
+            })
+
+    # Summary statistics
+    jaccards = [r["jaccard_half_half"] for r in replicate_results]
+    hub_rs = [r["hub_count_spearman_r"] for r in replicate_results if r["hub_count_spearman_r"] is not None]
+
+    mean_jaccard = np.mean(jaccards)
+    std_jaccard = np.std(jaccards)
+    mean_hub_r = np.mean(hub_rs) if hub_rs else np.nan
+
+    print(f"\n  SUMMARY:")
+    print(f"    Mean Jaccard (top-20 hubs): {mean_jaccard:.3f} +/- {std_jaccard:.3f}")
+    print(f"    Mean hub count Spearman r: {mean_hub_r:.3f}")
+
+    results = {
+        "full_data_n_edges": len(full_edges),
+        "full_data_top_hubs": full_hubs,
+        "n_replicates": n_replicates,
+        "mean_jaccard": float(mean_jaccard),
+        "std_jaccard": float(std_jaccard),
+        "mean_hub_count_r": float(mean_hub_r) if not np.isnan(mean_hub_r) else None,
+        "replicates": replicate_results,
+    }
+
+    with open(res_dir / "nb_split_half.json", "w") as f:
+        json.dump(results, f, indent=2, default=str)
+
+    # Figure
+    fig, axes = plt.subplots(1, 2, figsize=(12, 5))
+
+    # Panel 1: Jaccard per replicate
+    axes[0].bar(range(1, n_replicates + 1), jaccards, color="#1976D2",
+                edgecolor="black", linewidth=0.5)
+    axes[0].axhline(y=mean_jaccard, color="red", linestyle="--",
+                    label=f"Mean={mean_jaccard:.3f}")
+    axes[0].set_xlabel("Replicate")
+    axes[0].set_ylabel("Jaccard similarity (top-20 hubs)")
+    axes[0].set_title("Split-Half Hub Consistency")
+    axes[0].legend()
+    axes[0].set_ylim(0, 1)
+
+    # Panel 2: Hub count correlation
+    if hub_rs:
+        axes[1].bar(range(1, len(hub_rs) + 1), hub_rs, color="#43A047",
+                    edgecolor="black", linewidth=0.5)
+        axes[1].axhline(y=mean_hub_r, color="red", linestyle="--",
+                        label=f"Mean={mean_hub_r:.3f}")
+        axes[1].set_xlabel("Replicate")
+        axes[1].set_ylabel("Spearman r (hub target counts)")
+        axes[1].set_title("Split-Half Hub Count Correlation")
+        axes[1].legend()
+        axes[1].set_ylim(-0.5, 1)
+
+    fig.suptitle("Experiment D: NB Network Split-Half Robustness", fontsize=13)
+    fig.tight_layout()
+    save_fig(fig, "experiment_d_nb_robustness")
+
+    return results
+
+
+# =========================================================================
+# EXPERIMENT E: Corrected vs Uncorrected Network Quality
+# =========================================================================
+def experiment_e_correction_quality(go_results):
+    """Compare GO enrichment quality between corrected and uncorrected networks."""
+    print(f"\n{'='*60}")
+    print("EXPERIMENT E: CORRECTED vs UNCORRECTED NETWORK QUALITY")
+    print(f"{'='*60}")
+
+    res_dir = OUTPUT_DIR / "results"
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    # Load uncorrected (raw) network for NB
+    raw_nb_path = TIER3_DIR / "neuroblastoma_network_raw.csv"
+    corr_nb_path = TIER3_DIR / "neuroblastoma_network_corrected.csv"
+
+    # Also check for raw pancreas edges from gap_analysis
+    raw_panc_path = Path(__file__).parent.parent / "output" / "gap_analysis" / "results" / "network" / "pancreas" / "network_edges.csv"
+
+    networks_to_compare = {}
+
+    if raw_nb_path.exists() and corr_nb_path.exists():
+        raw_nb = pd.read_csv(raw_nb_path)
+        corr_nb = pd.read_csv(corr_nb_path)
+        networks_to_compare["neuroblastoma"] = {"raw": raw_nb, "corrected": corr_nb}
+        print(f"  NB raw: {len(raw_nb)} edges, corrected: {len(corr_nb)} edges")
+
+    if raw_panc_path.exists():
+        raw_panc = pd.read_csv(raw_panc_path)
+        corr_panc_path = WEAKNESS_DIR / "corrected_network_pancreas.csv"
+        if corr_panc_path.exists():
+            corr_panc = pd.read_csv(corr_panc_path)
+            networks_to_compare["pancreas"] = {"raw": raw_panc, "corrected": corr_panc}
+            print(f"  Pancreas raw: {len(raw_panc)} edges, corrected: {len(corr_panc)} edges")
+
+    if not networks_to_compare:
+        print("  No raw/corrected network pairs found")
+        return None
+
+    # Load GO library from local cache
+    go_lib = load_go_library()
+    if go_lib is None:
+        return None
+
+    BACKGROUND_SIZE = 20000
+
+    all_results = {}
+
+    for ds_name, net_pair in networks_to_compare.items():
+        print(f"\n  --- {ds_name} ---")
+
+        for method_name, edges_df in net_pair.items():
+            print(f"\n    {method_name} network ({len(edges_df)} edges):")
+
+            rbp_col = "rbp"
+            target_col = "target"
+
+            # Build RBP -> target sets
+            rbp_targets = {}
+            for rbp, grp in edges_df.groupby(rbp_col):
+                rbp_key = rbp.upper() if isinstance(rbp, str) else str(rbp)
+                rbp_targets[rbp_key] = set(str(t) for t in grp[target_col])
+
+            eligible = {r: t for r, t in rbp_targets.items() if len(t) >= 10}
+            print(f"      RBPs with >= 10 targets: {len(eligible)}")
+
+            n_with_sig = 0
+            for rbp, targets in eligible.items():
+                gene_list = list(targets)
+                sig_terms = hypergeometric_enrichment(gene_list, go_lib, BACKGROUND_SIZE)
+                if sig_terms:
+                    n_with_sig += 1
+
+            frac = n_with_sig / max(len(eligible), 1)
+            print(f"      Fraction with sig GO: {n_with_sig}/{len(eligible)} ({frac:.1%})")
+
+            key = f"{ds_name}_{method_name}"
+            all_results[key] = {
+                "dataset": ds_name,
+                "method": method_name,
+                "n_edges": len(edges_df),
+                "n_eligible_rbps": len(eligible),
+                "n_with_sig_go": n_with_sig,
+                "frac_with_sig_go": float(frac),
+            }
+
+    # Save results
+    with open(res_dir / "correction_quality.json", "w") as f:
+        json.dump(all_results, f, indent=2)
+
+    # Also compare destabilizing fractions
+    print("\n  Destabilizing fraction comparison:")
+    for ds_name, net_pair in networks_to_compare.items():
+        for method_name, edges_df in net_pair.items():
+            # Find the correlation column
+            r_col = None
+            for c in ["r", "spearman_r"]:
+                if c in edges_df.columns:
+                    r_col = c
+                    break
+            if r_col:
+                destab_frac = (edges_df[r_col] > 0).mean()
+                print(f"    {ds_name} {method_name}: {destab_frac:.1%} destabilizing")
+
+    # Summary figure
+    fig, ax = plt.subplots(figsize=(8, 5))
+
+    labels = []
+    raw_fracs = []
+    corr_fracs = []
+
+    for ds_name in networks_to_compare:
+        raw_key = f"{ds_name}_raw"
+        corr_key = f"{ds_name}_corrected"
+        if raw_key in all_results and corr_key in all_results:
+            labels.append(ds_name)
+            raw_fracs.append(all_results[raw_key]["frac_with_sig_go"])
+            corr_fracs.append(all_results[corr_key]["frac_with_sig_go"])
+
+    if labels:
+        x = np.arange(len(labels))
+        width = 0.35
+        ax.bar(x - width / 2, raw_fracs, width, label="Raw (uncorrected)",
+               color="#E53935", edgecolor="black", linewidth=0.5)
+        ax.bar(x + width / 2, corr_fracs, width, label="Library-size corrected",
+               color="#1976D2", edgecolor="black", linewidth=0.5)
+        ax.set_xticks(x)
+        ax.set_xticklabels(labels, fontsize=9)
+        ax.set_ylabel("Fraction of RBPs with sig GO enrichment")
+        ax.set_title("GO Enrichment: Raw vs Corrected Networks")
+        ax.legend()
+        ax.set_ylim(0, 1.1)
+        for i, (r, c) in enumerate(zip(raw_fracs, corr_fracs)):
+            ax.text(i - width / 2, r + 0.02, f"{r:.0%}", ha="center", fontsize=9)
+            ax.text(i + width / 2, c + 0.02, f"{c:.0%}", ha="center", fontsize=9)
+
+    fig.tight_layout()
+    save_fig(fig, "experiment_e_correction_quality")
+
+    print(f"\n  EXPERIMENT E SUMMARY:")
+    for ds_name in networks_to_compare:
+        raw_key = f"{ds_name}_raw"
+        corr_key = f"{ds_name}_corrected"
+        if raw_key in all_results and corr_key in all_results:
+            print(f"    {ds_name}: raw GO={all_results[raw_key]['frac_with_sig_go']:.0%} "
+                  f"-> corrected GO={all_results[corr_key]['frac_with_sig_go']:.0%}")
+
+    return all_results
+
+
+# =========================================================================
+# MAIN
+# =========================================================================
+def main():
+    set_figure_style()
+    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+    (OUTPUT_DIR / "results").mkdir(parents=True, exist_ok=True)
+    (OUTPUT_DIR / "figures").mkdir(parents=True, exist_ok=True)
+
+    # ===== Experiment A: GO Enrichment (CSV-only + API, fast) =====
+    go_results = experiment_a_go_enrichment()
+
+    # ===== Experiment E: Correction Quality (reuses GO, fast) =====
+    correction_results = experiment_e_correction_quality(go_results)
+
+    # ===== Load datasets for experiments B, C =====
+    print(f"\n{'='*60}")
+    print("LOADING DATASETS FOR EXPERIMENTS B, C")
+    print(f"{'='*60}")
+
+    adata_pan = scptr.datasets.pancreas()
+    adata_pan = run_pipeline(adata_pan, "pancreas")
+
+    adata_dg = scptr.datasets.dentate_gyrus()
+    adata_dg = run_pipeline(adata_dg, "dentate_gyrus")
+
+    # sci-fate
+    from run_scifate import load_scifate_data, prepare_for_scptr
+    adata_sf_raw = load_scifate_data()
+    adata_sf = prepare_for_scptr(adata_sf_raw)
+    adata_sf = run_pipeline(adata_sf, "scifate")
+
+    datasets = {
+        "pancreas": adata_pan,
+        "dentate_gyrus": adata_dg,
+        "scifate": adata_sf,
+    }
+
+    # ===== Experiment B: Pathway Consistency =====
+    pathway_results = experiment_b_pathway_consistency(datasets)
+
+    # ===== Experiment C: Gamma Advantage =====
+    gamma_adv_results = experiment_c_gamma_advantage(datasets)
+
+    # ===== Experiment D: NB Split-Half Robustness (slowest) =====
+    nb_results = experiment_d_nb_robustness()
+
+    # ===== FINAL SUMMARY =====
+    print(f"\n{'='*60}")
+    print("COMPREHENSIVE IMPROVEMENTS COMPLETE")
+    print(f"{'='*60}")
+
+    print("\n  Experiment A (GO Enrichment):")
+    if go_results:
+        for ds, res in go_results.items():
+            print(f"    {ds}: {res.get('frac_with_sig_go', 0):.0%} RBPs enriched "
+                  f"(null: {res.get('bootstrap_null_frac', 0):.0%})")
+
+    print("\n  Experiment B (Pathway Consistency):")
+    if pathway_results:
+        for r in pathway_results:
+            print(f"    {r['pair']}: gene r={r['gene_level_r']:.3f} -> "
+                  f"pathway r={r['pathway_level_r']:.3f}")
+
+    print("\n  Experiment C (Gamma Advantage):")
+    if gamma_adv_results:
+        for ds, res in gamma_adv_results.items():
+            inv = res["invisible_states"]
+            eta = res["eta_squared"]
+            print(f"    {ds}: invisible gamma={inv['gamma_n_invisible']} "
+                  f"vs ratio={inv['smooth_ratio_n_invisible']}; "
+                  f"eta-sq p={eta['wilcoxon_p']:.2e}")
+
+    print("\n  Experiment D (NB Robustness):")
+    if nb_results:
+        print(f"    Mean Jaccard (top-20): {nb_results['mean_jaccard']:.3f} "
+              f"+/- {nb_results['std_jaccard']:.3f}")
+
+    print("\n  Experiment E (Correction Quality):")
+    if correction_results:
+        for key, res in correction_results.items():
+            print(f"    {key}: {res['frac_with_sig_go']:.0%} sig GO")
+
+    print(f"\n  All results saved to: {OUTPUT_DIR.resolve()}")
+
+
+if __name__ == "__main__":
+    main()

analyses/run_deep_benchmark.py

@@ -0,0 +1,735 @@
+#!/usr/bin/env python
+"""Comprehensive benchmark: DeepPTR vs analytical scPTR on synthetic + real data.
+
+Runs:
+1. Synthetic recovery: gamma correlation, CI coverage, latent CCA
+2. Real datasets (pancreas, dentate gyrus): analytical vs DeepPTR
+   - Half-life correlation (mouse + human references)
+   - ARE/NMD enrichment
+   - Subsampling robustness
+   - Analytical vs DeepPTR gamma agreement
+3. sci-fate metabolic labeling: ground-truth validation for both methods
+
+All results saved to output/deep_benchmark/.
+"""
+
+from __future__ import annotations
+
+# Thread control — MUST be set before any numpy/torch import
+import os
+os.environ["OMP_NUM_THREADS"] = "4"
+os.environ["MKL_NUM_THREADS"] = "4"
+os.environ["OPENBLAS_NUM_THREADS"] = "4"
+os.environ["NUMEXPR_NUM_THREADS"] = "4"
+
+import json
+import sys
+import time
+from pathlib import Path
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+from scipy import stats
+
+import torch
+torch.set_num_threads(4)
+
+sys.path.insert(0, str(Path(__file__).parent))
+from _common import set_figure_style
+
+import scptr
+
+OUTPUT_DIR = Path(__file__).parent.parent / "output" / "deep_benchmark"
+
+
+def save_fig(fig, name, subdir="figures"):
+    if fig is None:
+        print(f"  [WARNING] {name}: plot returned None, skipping save")
+        return
+    out_dir = OUTPUT_DIR / subdir
+    out_dir.mkdir(parents=True, exist_ok=True)
+    path = out_dir / f"{name}.png"
+    fig.savefig(path, dpi=150, bbox_inches="tight")
+    plt.close(fig)
+    print(f"  Saved: {path}")
+
+
+def ensure_dirs():
+    for sub in ("figures", "results"):
+        (OUTPUT_DIR / sub).mkdir(parents=True, exist_ok=True)
+
+
+# ============================================================================
+# 1. SYNTHETIC RECOVERY
+# ============================================================================
+
+def run_synthetic_benchmark():
+    """End-to-end DeepPTR on synthetic kinetic data with known ground truth."""
+    from scptr.deep.synthetic import (
+        generate_kinetic_data,
+        gamma_recovery,
+        ci_coverage,
+        latent_recovery,
+    )
+
+    print("=" * 60)
+    print("1. SYNTHETIC RECOVERY BENCHMARK")
+    print("=" * 60)
+
+    adata, truth = generate_kinetic_data(
+        n_cells=1500, n_genes=100, n_cell_types=5,
+        dispersion=10.0, sparsity=0.3, seed=0,
+    )
+    print(f"  Generated: {adata.shape}, {truth['gamma'].shape}")
+
+    # Fit DeepPTR (compact model for CPU)
+    torch.set_num_threads(4)
+    t0 = time.time()
+    model, history = scptr.deep.fit_deepptr(
+        adata,
+        d_T=8, d_PT=8, d_hidden=48, n_enc_layers=2,
+        batch_size=256, max_epochs=150, kl_warmup_epochs=20,
+        patience=15, n_posterior_samples=20,
+        device="cpu", seed=0, verbose=True,
+    )
+    elapsed = time.time() - t0
+    print(f"  Training: {len(history.train_loss)} epochs in {elapsed:.1f}s")
+
+    # Evaluate
+    gamma_r = gamma_recovery(truth["gamma"], adata.layers["gamma"], per_gene=True)
+    gamma_r_global = gamma_recovery(truth["gamma"], adata.layers["gamma"], per_gene=False)
+    ci_cov = ci_coverage(truth["gamma"], adata.layers["gamma"], adata.layers["gamma_var"])
+    z_T_r = latent_recovery(truth["z_T"], adata.obsm["X_z_T"])
+    z_PT_r = latent_recovery(truth["z_PT"], adata.obsm["X_z_PT"])
+
+    results = {
+        "gamma_recovery_per_gene": gamma_r,
+        "gamma_recovery_global": gamma_r_global,
+        "ci_coverage_95": ci_cov,
+        "latent_recovery_T": z_T_r,
+        "latent_recovery_PT": z_PT_r,
+        "n_epochs": len(history.train_loss),
+        "final_train_loss": history.train_loss[-1],
+        "final_val_loss": history.val_loss[-1],
+        "training_time_s": elapsed,
+    }
+
+    print(f"\n  Gamma recovery (per-gene median Spearman r): {gamma_r:.4f}")
+    print(f"  Gamma recovery (global Spearman r):           {gamma_r_global:.4f}")
+    print(f"  95% CI coverage:                              {ci_cov:.4f}")
+    print(f"  Latent recovery z_T (mean CCA):               {z_T_r:.4f}")
+    print(f"  Latent recovery z_PT (mean CCA):              {z_PT_r:.4f}")
+
+    with open(OUTPUT_DIR / "results" / "synthetic_recovery.json", "w") as f:
+        json.dump(results, f, indent=2)
+
+    # Training curve plot
+    fig, axes = plt.subplots(1, 3, figsize=(15, 4))
+    epochs = range(1, len(history.train_loss) + 1)
+    axes[0].plot(epochs, history.train_loss, label="train")
+    axes[0].plot(epochs, history.val_loss, label="val")
+    axes[0].set_xlabel("Epoch")
+    axes[0].set_ylabel("Loss")
+    axes[0].set_title("Total Loss")
+    axes[0].legend()
+
+    axes[1].plot(epochs, history.train_recon, label="train")
+    axes[1].plot(epochs, history.val_recon, label="val")
+    axes[1].set_xlabel("Epoch")
+    axes[1].set_ylabel("Reconstruction Loss")
+    axes[1].set_title("Reconstruction")
+    axes[1].legend()
+
+    axes[2].plot(epochs, history.kl_weight, "k-")
+    axes[2].set_xlabel("Epoch")
+    axes[2].set_ylabel("KL Weight")
+    axes[2].set_title("KL Annealing")
+
+    fig.suptitle(f"DeepPTR Training (synthetic, gamma r={gamma_r:.3f})", y=1.02)
+    fig.tight_layout()
+    save_fig(fig, "synthetic_training_curves")
+
+    return results
+
+
+# ============================================================================
+# 2. REAL DATA: PANCREAS + DENTATE GYRUS
+# ============================================================================
+
+def preprocess_for_analytical(adata, cluster_key="clusters"):
+    """Standard scPTR preprocessing + analytical gamma."""
+    scptr.pp.filter_genes(adata)
+    scptr.pp.normalize_layers(adata)
+    scptr.pp.neighbors(adata, n_neighbors=30)
+    scptr.pp.smooth_layers(adata)
+    scptr.tl.estimate_beta(adata)
+    scptr.tl.estimate_gamma(adata)
+    return adata
+
+
+def select_top_genes(adata, n_top=500):
+    """Select top genes by unspliced signal for DeepPTR (reduces dim for CPU speed).
+
+    Uses total unspliced counts × fraction of cells expressing as the ranking.
+    Returns a view of adata with only the selected genes.
+    """
+    from scipy.sparse import issparse
+
+    u = adata.layers["unspliced"]
+    if issparse(u):
+        u = np.asarray(u.todense())
+    u = np.asarray(u, dtype=np.float32)
+
+    # Rank by: total counts * fraction nonzero (rewards both signal and breadth)
+    total_counts = u.sum(axis=0)
+    frac_nonzero = (u > 0).mean(axis=0)
+    score = total_counts * frac_nonzero
+
+    top_idx = np.argsort(score)[::-1][:n_top]
+    top_idx = np.sort(top_idx)  # keep original order
+
+    gene_names = adata.var_names[top_idx]
+    print(f"  Selected top {len(gene_names)} genes for DeepPTR (from {adata.n_vars})")
+    adata_sub = adata[:, gene_names].copy()
+
+    # Ensure dense layers for efficient DataLoader conversion
+    from scipy.sparse import issparse as _issparse
+    for key in ("spliced", "unspliced"):
+        if key in adata_sub.layers and _issparse(adata_sub.layers[key]):
+            adata_sub.layers[key] = np.asarray(adata_sub.layers[key].todense())
+    return adata_sub
+
+
+def run_halflife_comparison(adata, adata_deep, dataset_name):
+    """Compare half-life correlations: analytical vs DeepPTR."""
+    hl_mouse = scptr.datasets.herzog2017_halflives()
+    hl_human = scptr.datasets.schofield2018_halflives()
+
+    results = {}
+    for ref_name, hl_df in [("mouse_herzog", hl_mouse), ("human_schofield", hl_human)]:
+        # Analytical
+        corr_an = scptr.benchmark.correlate_with_halflives(adata, hl_df)
+        # DeepPTR
+        corr_dp = scptr.benchmark.correlate_with_halflives(adata_deep, hl_df)
+
+        results[ref_name] = {
+            "analytical": {
+                "spearman_r": corr_an["spearman_r"],
+                "pearson_r": corr_an["pearson_r"],
+                "n_genes": corr_an["n_genes"],
+            },
+            "deepptr": {
+                "spearman_r": corr_dp["spearman_r"],
+                "pearson_r": corr_dp["pearson_r"],
+                "n_genes": corr_dp["n_genes"],
+            },
+        }
+        print(f"  {ref_name}:")
+        print(f"    Analytical: Spearman r = {corr_an['spearman_r']:.4f} (n={corr_an['n_genes']})")
+        print(f"    DeepPTR:    Spearman r = {corr_dp['spearman_r']:.4f} (n={corr_dp['n_genes']})")
+
+    return results
+
+
+def run_enrichment_comparison(adata, adata_deep, dataset_name):
+    """Compare ARE/NMD enrichment: analytical vs DeepPTR."""
+    results = {}
+    for test_name, test_fn in [("ARE", scptr.benchmark.are_enrichment),
+                                ("NMD", scptr.benchmark.nmd_enrichment)]:
+        res_an = test_fn(adata)
+        res_dp = test_fn(adata_deep)
+
+        results[test_name] = {
+            "analytical": {
+                "U_statistic": float(res_an.get("U_statistic", np.nan)),
+                "p_value": float(res_an.get("p_value", np.nan)),
+                "n_genes_in_set": int(res_an.get("n_genes_in_set", 0)),
+            },
+            "deepptr": {
+                "U_statistic": float(res_dp.get("U_statistic", np.nan)),
+                "p_value": float(res_dp.get("p_value", np.nan)),
+                "n_genes_in_set": int(res_dp.get("n_genes_in_set", 0)),
+            },
+        }
+        p_an = res_an.get("p_value", np.nan)
+        p_dp = res_dp.get("p_value", np.nan)
+        print(f"  {test_name}: analytical p={p_an:.2e}, DeepPTR p={p_dp:.2e}")
+
+    return results
+
+
+def run_gamma_agreement(adata, adata_deep, dataset_name):
+    """Correlate per-gene median gamma: analytical vs DeepPTR on shared genes."""
+    gamma_an_s = pd.Series(
+        np.median(adata.layers["gamma"], axis=0), index=adata.var_names
+    )
+    gamma_dp_s = pd.Series(
+        np.median(adata_deep.layers["gamma"], axis=0), index=adata_deep.var_names
+    )
+
+    # Match on shared genes
+    shared = gamma_an_s.index.intersection(gamma_dp_s.index)
+    g_an = gamma_an_s[shared].values.astype(float)
+    g_dp = gamma_dp_s[shared].values.astype(float)
+
+    mask = (g_an > 0) & (g_dp > 0) & np.isfinite(g_an) & np.isfinite(g_dp)
+    g_an = g_an[mask]
+    g_dp = g_dp[mask]
+
+    if len(g_an) < 3:
+        print(f"  Analytical vs DeepPTR gamma: too few shared genes ({len(g_an)})")
+        return {"spearman_r": np.nan, "pearson_r": np.nan, "n_genes": 0}
+
+    sp_r, sp_p = stats.spearmanr(g_an, g_dp)
+    pe_r, pe_p = stats.pearsonr(np.log1p(g_an), np.log1p(g_dp))
+
+    result = {
+        "spearman_r": float(sp_r),
+        "spearman_p": float(sp_p),
+        "pearson_r": float(pe_r),
+        "pearson_p": float(pe_p),
+        "n_genes": int(mask.sum()),
+    }
+    print(f"  Analytical vs DeepPTR gamma: Spearman r = {sp_r:.4f} (n={mask.sum()})")
+
+    # Scatter plot
+    fig, ax = plt.subplots(figsize=(6, 5))
+    ax.scatter(g_an, g_dp, alpha=0.15, s=8, c="steelblue")
+    ax.set_xscale("log")
+    ax.set_yscale("log")
+    ax.set_xlabel("Analytical median gamma")
+    ax.set_ylabel("DeepPTR median gamma")
+    ax.set_title(f"{dataset_name}: Analytical vs DeepPTR (r={sp_r:.3f}, n={mask.sum()})")
+    lims = [min(g_an.min(), g_dp.min()), max(g_an.max(), g_dp.max())]
+    ax.plot(lims, lims, "k--", alpha=0.3, lw=1)
+    save_fig(fig, f"{dataset_name}_analytical_vs_deepptr")
+
+    return result
+
+
+def run_real_dataset(name, adata_loader, cluster_key="clusters"):
+    """Full benchmark for one real dataset."""
+    print(f"\n{'=' * 60}")
+    print(f"2. REAL DATA: {name.upper()}")
+    print("=" * 60)
+
+    # Load and preprocess
+    print(f"\n--- Loading {name} ---")
+    adata = adata_loader()
+    print(f"  Shape: {adata.shape}")
+
+    print(f"\n--- Preprocessing (analytical) ---")
+    preprocess_for_analytical(adata, cluster_key=cluster_key)
+    gamma_an = adata.layers["gamma"]
+    gamma_med_an = np.median(gamma_an, axis=0)
+    print(f"  Analytical gamma: median of medians = {np.median(gamma_med_an):.4f}")
+
+    # DeepPTR: preprocess, select top genes, then fit
+    print(f"\n--- Running DeepPTR ---")
+    adata_deep = adata_loader()
+    scptr.pp.filter_genes(adata_deep)
+    scptr.pp.normalize_layers(adata_deep)
+    scptr.pp.neighbors(adata_deep, n_neighbors=30)
+    scptr.pp.smooth_layers(adata_deep)
+    scptr.tl.estimate_beta(adata_deep)
+    # Select top genes to keep training tractable on CPU
+    adata_deep = select_top_genes(adata_deep, n_top=300)
+
+    torch.set_num_threads(4)  # Reset after TF/scanpy imports
+    t0 = time.time()
+    model, history = scptr.deep.fit_deepptr(
+        adata_deep,
+        d_T=8, d_PT=8, d_hidden=48, n_enc_layers=2,
+        batch_size=512, max_epochs=100, kl_warmup_epochs=20,
+        patience=15, n_posterior_samples=15,
+        device="cpu", seed=0, verbose=True,
+    )
+    elapsed = time.time() - t0
+    n_epochs = len(history.train_loss)
+    print(f"  DeepPTR: {n_epochs} epochs in {elapsed:.1f}s")
+
+    gamma_dp = adata_deep.layers["gamma"]
+    gamma_med_dp = np.median(gamma_dp, axis=0)
+    print(f"  DeepPTR gamma: median of medians = {np.median(gamma_med_dp):.4f}")
+
+    # --- Benchmarks ---
+    all_results = {
+        "dataset": name,
+        "n_cells": adata.n_obs,
+        "n_genes": adata.n_vars,
+        "deepptr_epochs": n_epochs,
+        "deepptr_time_s": elapsed,
+        "deepptr_final_val_loss": history.val_loss[-1],
+    }
+
+    # Half-life correlations
+    print(f"\n--- Half-life correlations ---")
+    hl_results = run_halflife_comparison(adata, adata_deep, name)
+    all_results["halflife"] = hl_results
+
+    # ARE/NMD enrichment
+    print(f"\n--- ARE/NMD enrichment ---")
+    try:
+        enrich_results = run_enrichment_comparison(adata, adata_deep, name)
+        all_results["enrichment"] = enrich_results
+    except Exception as e:
+        print(f"  Enrichment failed: {e}")
+        all_results["enrichment"] = {"error": str(e)}
+
+    # Analytical vs DeepPTR agreement
+    print(f"\n--- Analytical vs DeepPTR agreement ---")
+    agree = run_gamma_agreement(adata, adata_deep, name)
+    all_results["gamma_agreement"] = agree
+
+    # Subsampling robustness (DeepPTR only — analytical already known)
+    print(f"\n--- Subsampling robustness (analytical) ---")
+    try:
+        rob_an = scptr.benchmark.subsampling_robustness(
+            adata, fractions=[0.5, 0.8], n_repeats=2
+        )
+        print(f"  Analytical: median r @ 30% = {rob_an[rob_an['fraction']==0.3]['spearman_r'].median():.4f}")
+        all_results["robustness_analytical"] = rob_an.to_dict(orient="records")
+    except Exception as e:
+        print(f"  Robustness failed: {e}")
+
+    # Training curve
+    fig, axes = plt.subplots(1, 2, figsize=(12, 4))
+    epochs = range(1, n_epochs + 1)
+    axes[0].plot(epochs, history.train_loss, label="train")
+    axes[0].plot(epochs, history.val_loss, label="val")
+    axes[0].set_xlabel("Epoch")
+    axes[0].set_ylabel("Loss")
+    axes[0].set_title(f"{name}: Training Loss")
+    axes[0].legend()
+
+    axes[1].plot(epochs, history.train_recon, label="train recon")
+    axes[1].plot(epochs, history.train_kl, label="train KL")
+    axes[1].set_xlabel("Epoch")
+    axes[1].set_ylabel("Loss Component")
+    axes[1].set_title(f"{name}: Loss Components")
+    axes[1].legend()
+    fig.tight_layout()
+    save_fig(fig, f"{name}_training_curves")
+
+    # Uncertainty visualization
+    gamma_var = adata_deep.layers["gamma_var"]
+    mean_var = np.mean(gamma_var, axis=0)
+    fig, ax = plt.subplots(figsize=(6, 5))
+    ax.scatter(gamma_med_dp, mean_var, alpha=0.2, s=8, c="steelblue")
+    ax.set_xscale("log")
+    ax.set_yscale("log")
+    ax.set_xlabel("Posterior mean gamma (median over cells)")
+    ax.set_ylabel("Posterior variance (mean over cells)")
+    ax.set_title(f"{name}: DeepPTR Uncertainty")
+    save_fig(fig, f"{name}_uncertainty")
+
+    # Save
+    with open(OUTPUT_DIR / "results" / f"{name}_benchmark.json", "w") as f:
+        json.dump(all_results, f, indent=2, default=str)
+
+    return all_results
+
+
+# ============================================================================
+# 3. SCI-FATE GROUND TRUTH VALIDATION
+# ============================================================================
+
+def run_scifate_benchmark():
+    """Compare analytical vs DeepPTR on sci-fate metabolic labeling data."""
+    import gzip
+    from scipy.io import mmread
+    from scipy.sparse import csc_matrix
+
+    print(f"\n{'=' * 60}")
+    print("3. SCI-FATE METABOLIC LABELING VALIDATION")
+    print("=" * 60)
+
+    CACHE_DIR = Path.home() / ".cache" / "scptr" / "scifate"
+    if not CACHE_DIR.exists():
+        print("  [SKIP] sci-fate data not cached. Run analyses/run_scifate.py first.")
+        return None
+
+    # Load raw data
+    print("  Loading sci-fate data...")
+    cell_ann = pd.read_csv(CACHE_DIR / "GSM3770930_A549_cell_annotate.txt.gz", compression="gzip")
+    gene_ann = pd.read_csv(CACHE_DIR / "GSM3770930_A549_gene_annotate.txt.gz", compression="gzip")
+
+    with gzip.open(CACHE_DIR / "GSM3770930_A549_gene_count.txt.gz", "rb") as f:
+        total_mat = csc_matrix(mmread(f)).T
+    with gzip.open(CACHE_DIR / "GSM3770930_A549_gene_count_newly_synthesised.txt.gz", "rb") as f:
+        new_mat = csc_matrix(mmread(f)).T
+
+    import anndata as ad
+    adata_raw = ad.AnnData(
+        X=total_mat,
+        obs=cell_ann.set_index("sample"),
+        var=gene_ann.set_index("gene_id"),
+    )
+    adata_raw.layers["new"] = new_mat
+    adata_raw.var_names_make_unique()
+    adata_raw.var["gene_id_full"] = adata_raw.var_names.tolist()
+    adata_raw.var_names = adata_raw.var["gene_short_name"].values
+    adata_raw.var_names_make_unique()
+    print(f"  Shape: {adata_raw.shape}")
+
+    # Ground truth
+    total = np.asarray(adata_raw.X.toarray() if hasattr(adata_raw.X, "toarray") else adata_raw.X)
+    new = np.asarray(adata_raw.layers["new"].toarray() if hasattr(adata_raw.layers["new"], "toarray") else adata_raw.layers["new"])
+    old = total - new
+    mean_new = new.mean(axis=0)
+    mean_old = old.mean(axis=0)
+    mean_total = total.mean(axis=0)
+    reliable = (mean_total >= 0.5) & (mean_old > 0.1)
+    gt_ratio = np.full(adata_raw.n_vars, np.nan)
+    gt_ratio[reliable] = mean_new[reliable] / mean_old[reliable]
+    print(f"  Ground truth: {reliable.sum()} reliable genes")
+
+    # Prepare for scPTR (unspliced=new, spliced=old)
+    keep = mean_total >= 0.5
+    if "gene_type" in adata_raw.var.columns:
+        is_pc = adata_raw.var["gene_type"] == "protein_coding"
+        keep = keep & is_pc.values
+
+    def make_scptr_adata():
+        a = ad.AnnData(
+            X=total[:, keep].astype(np.float32),
+            obs=adata_raw.obs.copy(),
+            var=adata_raw.var.iloc[keep].copy(),
+        )
+        a.layers["unspliced"] = new[:, keep].astype(np.float32)
+        a.layers["spliced"] = old[:, keep].astype(np.float32)
+        return a
+
+    # --- Analytical ---
+    print("\n--- Analytical pipeline ---")
+    adata_an = make_scptr_adata()
+    scptr.pp.filter_genes(adata_an, min_unspliced_counts=1, min_unspliced_cells=1)
+    scptr.pp.normalize_layers(adata_an)
+    scptr.pp.neighbors(adata_an, n_neighbors=30)
+    scptr.pp.smooth_layers(adata_an)
+    scptr.tl.estimate_beta(adata_an)
+    scptr.tl.estimate_gamma(adata_an)
+    gamma_med_an = np.median(adata_an.layers["gamma"], axis=0)
+    print(f"  Analytical: {adata_an.shape}, median gamma = {np.median(gamma_med_an):.4f}")
+
+    # --- DeepPTR ---
+    print("\n--- DeepPTR ---")
+    adata_dp = make_scptr_adata()
+    scptr.pp.filter_genes(adata_dp, min_unspliced_counts=1, min_unspliced_cells=1)
+    scptr.pp.normalize_layers(adata_dp)
+    scptr.pp.neighbors(adata_dp, n_neighbors=30)
+    scptr.pp.smooth_layers(adata_dp)
+    scptr.tl.estimate_beta(adata_dp)
+    adata_dp = select_top_genes(adata_dp, n_top=500)
+
+    t0 = time.time()
+    model, history = scptr.deep.fit_deepptr(
+        adata_dp,
+        d_T=8, d_PT=8, d_hidden=48, n_enc_layers=2,
+        batch_size=512, max_epochs=100, kl_warmup_epochs=20,
+        patience=15, n_posterior_samples=15,
+        device="cpu", seed=0, verbose=True,
+    )
+    elapsed = time.time() - t0
+    gamma_med_dp = np.median(adata_dp.layers["gamma"], axis=0)
+    print(f"  DeepPTR: {len(history.train_loss)} epochs in {elapsed:.1f}s")
+
+    # Correlate both with ground truth
+    gt_s_an = pd.Series(gt_ratio, index=adata_raw.var_names)
+    gamma_s_an = pd.Series(gamma_med_an, index=adata_an.var_names)
+    gamma_s_dp = pd.Series(gamma_med_dp, index=adata_dp.var_names)
+
+    shared_an = gamma_s_an.index.intersection(gt_s_an.dropna().index)
+    shared_dp = gamma_s_dp.index.intersection(gt_s_an.dropna().index)
+
+    def correlate(gamma_s, gt_s, shared):
+        g = gamma_s[shared].values.astype(float)
+        t = gt_s[shared].values.astype(float)
+        mask = np.isfinite(g) & np.isfinite(t) & (g > 0) & (t > 0)
+        if mask.sum() < 3:
+            return {"spearman_r": np.nan, "n_genes": 0}
+        sp_r, sp_p = stats.spearmanr(g[mask], t[mask])
+        return {"spearman_r": float(sp_r), "spearman_p": float(sp_p), "n_genes": int(mask.sum())}
+
+    corr_an = correlate(gamma_s_an, gt_s_an, shared_an)
+    corr_dp = correlate(gamma_s_dp, gt_s_an, shared_dp)
+
+    print(f"\n--- Ground truth correlation (new/old ratio) ---")
+    print(f"  Analytical: Spearman r = {corr_an['spearman_r']:.4f} (n={corr_an['n_genes']})")
+    print(f"  DeepPTR:    Spearman r = {corr_dp['spearman_r']:.4f} (n={corr_dp['n_genes']})")
+
+    # Half-life correlation
+    print(f"\n--- Half-life correlations ---")
+    hl_human = scptr.datasets.schofield2018_halflives()
+    corr_hl_an = scptr.benchmark.correlate_with_halflives(adata_an, hl_human)
+    corr_hl_dp = scptr.benchmark.correlate_with_halflives(adata_dp, hl_human)
+    print(f"  Analytical: Spearman r = {corr_hl_an['spearman_r']:.4f} (n={corr_hl_an['n_genes']})")
+    print(f"  DeepPTR:    Spearman r = {corr_hl_dp['spearman_r']:.4f} (n={corr_hl_dp['n_genes']})")
+
+    # Agreement
+    shared_both = gamma_s_an.index.intersection(gamma_s_dp.index)
+    g_an = gamma_s_an[shared_both].values
+    g_dp = gamma_s_dp[shared_both].values
+    mask_both = (g_an > 0) & (g_dp > 0) & np.isfinite(g_an) & np.isfinite(g_dp)
+    if mask_both.sum() >= 3:
+        agree_r, _ = stats.spearmanr(g_an[mask_both], g_dp[mask_both])
+        print(f"\n  Analytical vs DeepPTR: Spearman r = {agree_r:.4f} (n={mask_both.sum()})")
+    else:
+        agree_r = np.nan
+
+    results = {
+        "dataset": "scifate",
+        "n_cells": int(adata_an.n_obs),
+        "n_genes_analytical": int(adata_an.n_vars),
+        "n_genes_deep": int(adata_dp.n_vars),
+        "ground_truth_corr": {
+            "analytical": corr_an,
+            "deepptr": corr_dp,
+        },
+        "halflife_human": {
+            "analytical": {"spearman_r": corr_hl_an["spearman_r"], "n_genes": corr_hl_an["n_genes"]},
+            "deepptr": {"spearman_r": corr_hl_dp["spearman_r"], "n_genes": corr_hl_dp["n_genes"]},
+        },
+        "gamma_agreement": {"spearman_r": float(agree_r), "n_genes": int(mask_both.sum())},
+        "deepptr_epochs": len(history.train_loss),
+        "deepptr_time_s": elapsed,
+    }
+
+    with open(OUTPUT_DIR / "results" / "scifate_benchmark.json", "w") as f:
+        json.dump(results, f, indent=2, default=str)
+
+    # Scatter: analytical vs DeepPTR vs ground truth
+    fig, axes = plt.subplots(1, 3, figsize=(16, 4.5))
+
+    # Panel 1: Analytical vs ground truth
+    g = gamma_s_an[shared_an].values.astype(float)
+    t = gt_s_an[shared_an].values.astype(float)
+    m = np.isfinite(g) & np.isfinite(t) & (g > 0) & (t > 0)
+    axes[0].scatter(t[m], g[m], alpha=0.1, s=5, c="steelblue")
+    axes[0].set_xscale("log")
+    axes[0].set_yscale("log")
+    axes[0].set_xlabel("Ground truth (new/old ratio)")
+    axes[0].set_ylabel("Analytical gamma")
+    axes[0].set_title(f"Analytical (r={corr_an['spearman_r']:.3f})")
+
+    # Panel 2: DeepPTR vs ground truth
+    g = gamma_s_dp[shared_dp].values.astype(float)
+    t = gt_s_an[shared_dp].values.astype(float)
+    m = np.isfinite(g) & np.isfinite(t) & (g > 0) & (t > 0)
+    axes[1].scatter(t[m], g[m], alpha=0.1, s=5, c="darkorange")
+    axes[1].set_xscale("log")
+    axes[1].set_yscale("log")
+    axes[1].set_xlabel("Ground truth (new/old ratio)")
+    axes[1].set_ylabel("DeepPTR gamma")
+    axes[1].set_title(f"DeepPTR (r={corr_dp['spearman_r']:.3f})")
+
+    # Panel 3: Analytical vs DeepPTR
+    if mask_both.sum() >= 3:
+        axes[2].scatter(g_an[mask_both], g_dp[mask_both], alpha=0.1, s=5, c="seagreen")
+        axes[2].set_xscale("log")
+        axes[2].set_yscale("log")
+        lims = [min(g_an[mask_both].min(), g_dp[mask_both].min()),
+                max(g_an[mask_both].max(), g_dp[mask_both].max())]
+        axes[2].plot(lims, lims, "k--", alpha=0.3, lw=1)
+    axes[2].set_xlabel("Analytical gamma")
+    axes[2].set_ylabel("DeepPTR gamma")
+    axes[2].set_title(f"Agreement (r={agree_r:.3f})")
+
+    fig.suptitle("sci-fate: Analytical vs DeepPTR", y=1.02)
+    fig.tight_layout()
+    save_fig(fig, "scifate_comparison")
+
+    return results
+
+
+# ============================================================================
+# 4. SUMMARY TABLE
+# ============================================================================
+
+def print_summary(synth, pancreas, dg, scifate):
+    """Print final comparison table."""
+    print(f"\n{'=' * 70}")
+    print("SUMMARY: Analytical vs DeepPTR")
+    print("=" * 70)
+
+    # Header
+    print(f"\n{'Metric':<40} {'Analytical':>12} {'DeepPTR':>12}")
+    print("-" * 65)
+
+    if synth:
+        print(f"\n  SYNTHETIC RECOVERY")
+        print(f"  {'Gamma recovery (per-gene r)':<38} {'N/A':>12} {synth['gamma_recovery_per_gene']:>12.4f}")
+        print(f"  {'95% CI coverage':<38} {'N/A':>12} {synth['ci_coverage_95']:>12.4f}")
+        print(f"  {'Latent recovery z_T':<38} {'N/A':>12} {synth['latent_recovery_T']:>12.4f}")
+        print(f"  {'Latent recovery z_PT':<38} {'N/A':>12} {synth['latent_recovery_PT']:>12.4f}")
+
+    for name, res in [("PANCREAS", pancreas), ("DENTATE GYRUS", dg)]:
+        if res is None:
+            continue
+        print(f"\n  {name}")
+        for ref in ("mouse_herzog", "human_schofield"):
+            if ref in res.get("halflife", {}):
+                hl = res["halflife"][ref]
+                an_r = hl["analytical"]["spearman_r"]
+                dp_r = hl["deepptr"]["spearman_r"]
+                print(f"  {'Half-life ' + ref:<38} {an_r:>12.4f} {dp_r:>12.4f}")
+        if "gamma_agreement" in res:
+            print(f"  {'Gamma agreement (Spearman r)':<38} {'---':>12} {res['gamma_agreement']['spearman_r']:>12.4f}")
+
+    if scifate:
+        print(f"\n  SCI-FATE")
+        gt = scifate.get("ground_truth_corr", {})
+        if "analytical" in gt and "deepptr" in gt:
+            an_r = gt["analytical"]["spearman_r"]
+            dp_r = gt["deepptr"]["spearman_r"]
+            print(f"  {'Ground truth (new/old ratio)':<38} {an_r:>12.4f} {dp_r:>12.4f}")
+        hl = scifate.get("halflife_human", {})
+        if "analytical" in hl and "deepptr" in hl:
+            an_r = hl["analytical"]["spearman_r"]
+            dp_r = hl["deepptr"]["spearman_r"]
+            print(f"  {'Half-life (human Schofield)':<38} {an_r:>12.4f} {dp_r:>12.4f}")
+
+    print()
+
+
+def main():
+    set_figure_style()
+    ensure_dirs()
+
+    # 1. Synthetic
+    synth_results = run_synthetic_benchmark()
+
+    # 2. Pancreas
+    pancreas_results = run_real_dataset(
+        "pancreas", scptr.datasets.pancreas, cluster_key="clusters"
+    )
+
+    # 3. Dentate Gyrus
+    dg_results = run_real_dataset(
+        "dentate_gyrus", scptr.datasets.dentate_gyrus, cluster_key="clusters"
+    )
+
+    # 4. sci-fate (if data available)
+    scifate_results = run_scifate_benchmark()
+
+    # 5. Summary
+    print_summary(synth_results, pancreas_results, dg_results, scifate_results)
+
+    # Save combined results
+    combined = {
+        "synthetic": synth_results,
+        "pancreas": pancreas_results,
+        "dentate_gyrus": dg_results,
+        "scifate": scifate_results,
+    }
+    with open(OUTPUT_DIR / "results" / "combined_benchmark.json", "w") as f:
+        json.dump(combined, f, indent=2, default=str)
+
+    print(f"\nAll results saved to: {OUTPUT_DIR}")
+
+
+if __name__ == "__main__":
+    main()

analyses/run_deep_benchmark.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+export OMP_NUM_THREADS=4
+export MKL_NUM_THREADS=4
+export OPENBLAS_NUM_THREADS=4
+export NUMEXPR_NUM_THREADS=4
+export CUDA_VISIBLE_DEVICES=""
+exec python -u /home/bcheng/scPTR/analyses/run_deep_benchmark.py "$@"

analyses/run_deep_benchmark_v2.py

@@ -0,0 +1,734 @@
+#!/usr/bin/env python
+"""Expanded DeepPTR benchmark v2: deeper analysis beyond basic half-life correlation.
+
+Adds to v1:
+1. Enrichment on full gene set (map DeepPTR gamma onto analytical genes)
+2. Per-cell-type gamma patterns (cell-type-specific agreement)
+3. Uncertainty calibration on real data (variance vs prediction error)
+4. DeepPTR subsampling robustness (retrain on subsets)
+5. Cross-dataset consistency (DeepPTR vs analytical)
+6. Latent space structure (z_T/z_PT UMAP colored by cell type)
+7. Gene ranking comparison (top differentially-degraded genes)
+
+All results saved to output/deep_benchmark_v2/.
+"""
+
+from __future__ import annotations
+
+import os
+os.environ["OMP_NUM_THREADS"] = "4"
+os.environ["MKL_NUM_THREADS"] = "4"
+os.environ["OPENBLAS_NUM_THREADS"] = "4"
+os.environ["NUMEXPR_NUM_THREADS"] = "4"
+
+import json
+import sys
+import time
+from pathlib import Path
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+from scipy import stats
+
+import torch
+torch.set_num_threads(4)
+
+sys.path.insert(0, str(Path(__file__).parent))
+from _common import set_figure_style
+
+import scptr
+
+OUTPUT_DIR = Path(__file__).parent.parent / "output" / "deep_benchmark_v2"
+
+
+def save_fig(fig, name, subdir="figures"):
+    if fig is None:
+        return
+    out_dir = OUTPUT_DIR / subdir
+    out_dir.mkdir(parents=True, exist_ok=True)
+    path = out_dir / f"{name}.png"
+    fig.savefig(path, dpi=150, bbox_inches="tight")
+    plt.close(fig)
+    print(f"  Saved: {path}")
+
+
+def ensure_dirs():
+    for sub in ("figures", "results"):
+        (OUTPUT_DIR / sub).mkdir(parents=True, exist_ok=True)
+
+
+def select_top_genes(adata, n_top=300):
+    """Select top genes by unspliced signal for DeepPTR."""
+    from scipy.sparse import issparse
+    u = adata.layers["unspliced"]
+    if issparse(u):
+        u = np.asarray(u.todense())
+    u = np.asarray(u, dtype=np.float32)
+    score = u.sum(axis=0) * (u > 0).mean(axis=0)
+    top_idx = np.sort(np.argsort(score)[::-1][:n_top])
+    adata_sub = adata[:, adata.var_names[top_idx]].copy()
+    from scipy.sparse import issparse as _iss
+    for key in ("spliced", "unspliced"):
+        if key in adata_sub.layers and _iss(adata_sub.layers[key]):
+            adata_sub.layers[key] = np.asarray(adata_sub.layers[key].todense())
+    print(f"  Selected top {n_top} genes (from {adata.n_vars})")
+    return adata_sub
+
+
+def run_analytical_pipeline(adata):
+    """Full analytical scPTR pipeline."""
+    scptr.pp.filter_genes(adata)
+    scptr.pp.normalize_layers(adata)
+    scptr.pp.neighbors(adata, n_neighbors=30)
+    scptr.pp.smooth_layers(adata)
+    scptr.tl.estimate_beta(adata)
+    scptr.tl.estimate_gamma(adata)
+    return adata
+
+
+def fit_deep(adata_deep):
+    """Fit DeepPTR on preprocessed adata (with beta already estimated)."""
+    torch.set_num_threads(4)
+    model, history = scptr.deep.fit_deepptr(
+        adata_deep,
+        d_T=8, d_PT=8, d_hidden=48, n_enc_layers=2,
+        batch_size=512, max_epochs=100, kl_warmup_epochs=20,
+        patience=15, n_posterior_samples=15,
+        device="cpu", seed=0, verbose=True,
+    )
+    return model, history
+
+
+# ============================================================================
+# 1. ENRICHMENT WITH FULL GENE MAPPING
+# ============================================================================
+
+def run_enrichment_mapped(adata_an, adata_deep, dataset_name):
+    """Map DeepPTR gamma onto full analytical gene set, then run enrichment.
+
+    DeepPTR only models top-N genes. For enrichment, we create a hybrid:
+    use DeepPTR gamma where available, analytical gamma elsewhere.
+    Also test DeepPTR-only genes separately.
+    """
+    print(f"\n--- Enrichment (mapped) ---")
+    import anndata as ad
+
+    gamma_an_med = np.median(adata_an.layers["gamma"], axis=0)
+    gamma_dp_med = np.median(adata_deep.layers["gamma"], axis=0)
+    dp_genes = set(adata_deep.var_names)
+
+    # Hybrid: prefer DeepPTR where available
+    gamma_hybrid = gamma_an_med.copy()
+    for i, g in enumerate(adata_an.var_names):
+        if g in dp_genes:
+            j = list(adata_deep.var_names).index(g)
+            gamma_hybrid[i] = gamma_dp_med[j]
+
+    # Create hybrid adata for enrichment
+    adata_hybrid = adata_an.copy()
+    adata_hybrid.layers["gamma"] = np.tile(gamma_hybrid, (adata_an.n_obs, 1))
+
+    results = {}
+    for test_name, test_fn in [("ARE", scptr.benchmark.are_enrichment),
+                                ("NMD", scptr.benchmark.nmd_enrichment)]:
+        res_an = test_fn(adata_an)
+        res_hybrid = test_fn(adata_hybrid)
+
+        results[test_name] = {
+            "analytical": {
+                "p_value": float(res_an.get("p_value", np.nan)),
+                "n_genes_in_set": int(res_an.get("n_genes_in_set", 0)),
+                "median_gamma_in": float(res_an.get("median_gamma_in_set", np.nan)),
+                "median_gamma_bg": float(res_an.get("median_gamma_background", np.nan)),
+            },
+            "hybrid_deepptr": {
+                "p_value": float(res_hybrid.get("p_value", np.nan)),
+                "n_genes_in_set": int(res_hybrid.get("n_genes_in_set", 0)),
+                "median_gamma_in": float(res_hybrid.get("median_gamma_in_set", np.nan)),
+                "median_gamma_bg": float(res_hybrid.get("median_gamma_background", np.nan)),
+            },
+        }
+        p_an = res_an.get("p_value", np.nan)
+        p_hy = res_hybrid.get("p_value", np.nan)
+        print(f"  {test_name}: analytical p={p_an:.2e}, hybrid p={p_hy:.2e}")
+
+    return results
+
+
+# ============================================================================
+# 2. PER-CELL-TYPE GAMMA AGREEMENT
+# ============================================================================
+
+def run_celltype_agreement(adata_an, adata_deep, dataset_name, cluster_key="clusters"):
+    """Compare per-cell-type median gamma between analytical and DeepPTR."""
+    print(f"\n--- Per-cell-type gamma agreement ---")
+
+    if cluster_key not in adata_an.obs.columns:
+        print(f"  [SKIP] No '{cluster_key}' column")
+        return None
+
+    shared_genes = adata_an.var_names.intersection(adata_deep.var_names)
+    if len(shared_genes) < 10:
+        print(f"  [SKIP] Too few shared genes ({len(shared_genes)})")
+        return None
+
+    an_idx = [list(adata_an.var_names).index(g) for g in shared_genes]
+    dp_idx = [list(adata_deep.var_names).index(g) for g in shared_genes]
+
+    cell_types = adata_an.obs[cluster_key].unique()
+    records = []
+
+    for ct in sorted(cell_types):
+        mask_an = adata_an.obs[cluster_key] == ct
+        mask_dp = adata_deep.obs[cluster_key] == ct
+
+        if mask_an.sum() < 5 or mask_dp.sum() < 5:
+            continue
+
+        gamma_an_ct = np.median(adata_an.layers["gamma"][mask_an][:, an_idx], axis=0)
+        gamma_dp_ct = np.median(adata_deep.layers["gamma"][mask_dp][:, dp_idx], axis=0)
+
+        valid = (gamma_an_ct > 0) & (gamma_dp_ct > 0) & np.isfinite(gamma_an_ct) & np.isfinite(gamma_dp_ct)
+        if valid.sum() < 5:
+            continue
+
+        sp_r, _ = stats.spearmanr(gamma_an_ct[valid], gamma_dp_ct[valid])
+        records.append({
+            "cell_type": str(ct),
+            "n_cells_an": int(mask_an.sum()),
+            "n_cells_dp": int(mask_dp.sum()),
+            "n_genes": int(valid.sum()),
+            "spearman_r": float(sp_r),
+        })
+        print(f"  {ct}: r={sp_r:.4f} (n_genes={valid.sum()}, n_cells={mask_an.sum()})")
+
+    if not records:
+        return None
+
+    df = pd.DataFrame(records)
+
+    # Plot
+    fig, ax = plt.subplots(figsize=(8, 4))
+    ax.barh(df["cell_type"], df["spearman_r"], color="steelblue", alpha=0.7)
+    ax.set_xlabel("Spearman r (analytical vs DeepPTR)")
+    ax.set_title(f"{dataset_name}: Per-cell-type gamma agreement")
+    ax.axvline(x=df["spearman_r"].median(), color="red", ls="--", alpha=0.5,
+               label=f"median={df['spearman_r'].median():.3f}")
+    ax.legend()
+    fig.tight_layout()
+    save_fig(fig, f"{dataset_name}_celltype_agreement")
+
+    return records
+
+
+# ============================================================================
+# 3. UNCERTAINTY CALIBRATION ON REAL DATA
+# ============================================================================
+
+def run_uncertainty_analysis(adata_an, adata_deep, dataset_name):
+    """Evaluate DeepPTR uncertainty: does high variance predict high error?"""
+    print(f"\n--- Uncertainty calibration ---")
+
+    shared_genes = adata_an.var_names.intersection(adata_deep.var_names)
+    if len(shared_genes) < 10:
+        print(f"  [SKIP] Too few shared genes")
+        return None
+
+    an_idx = [list(adata_an.var_names).index(g) for g in shared_genes]
+    dp_idx = [list(adata_deep.var_names).index(g) for g in shared_genes]
+
+    # Per-gene: compare variance with squared error vs analytical
+    gamma_an = np.median(adata_an.layers["gamma"][:, an_idx], axis=0)
+    gamma_dp = np.median(adata_deep.layers["gamma"][:, dp_idx], axis=0)
+    gamma_var = np.mean(adata_deep.layers["gamma_var"][:, dp_idx], axis=0)
+
+    # Prediction error (using analytical as reference)
+    valid = (gamma_an > 0) & (gamma_dp > 0) & np.isfinite(gamma_an) & np.isfinite(gamma_dp)
+    if valid.sum() < 10:
+        print(f"  [SKIP] Too few valid genes")
+        return None
+
+    error = np.abs(gamma_dp[valid] - gamma_an[valid])
+    var = gamma_var[valid]
+
+    # Does high posterior variance correlate with high error?
+    sp_r, sp_p = stats.spearmanr(var, error)
+    print(f"  Variance-error correlation: Spearman r = {sp_r:.4f} (p={sp_p:.2e})")
+
+    # Binned calibration: split genes into variance quintiles
+    n_bins = 5
+    var_ranks = np.argsort(np.argsort(var))
+    bin_size = len(var) // n_bins
+    bin_errors = []
+    bin_vars = []
+    for b in range(n_bins):
+        mask = (var_ranks >= b * bin_size) & (var_ranks < (b + 1) * bin_size)
+        if b == n_bins - 1:
+            mask = var_ranks >= b * bin_size
+        bin_errors.append(np.median(error[mask]))
+        bin_vars.append(np.median(var[mask]))
+
+    result = {
+        "var_error_spearman_r": float(sp_r),
+        "var_error_spearman_p": float(sp_p),
+        "n_genes": int(valid.sum()),
+        "bin_median_var": [float(v) for v in bin_vars],
+        "bin_median_error": [float(e) for e in bin_errors],
+    }
+
+    # Plot
+    fig, axes = plt.subplots(1, 2, figsize=(12, 5))
+
+    # Scatter: variance vs error
+    axes[0].scatter(var, error, alpha=0.2, s=8, c="steelblue")
+    axes[0].set_xlabel("Mean posterior variance")
+    axes[0].set_ylabel("|DeepPTR - Analytical| error")
+    axes[0].set_title(f"Variance vs Error (r={sp_r:.3f})")
+    axes[0].set_xscale("log")
+    axes[0].set_yscale("log")
+
+    # Binned calibration
+    axes[1].bar(range(n_bins), bin_errors, color="steelblue", alpha=0.7)
+    axes[1].set_xlabel("Posterior variance quintile (low → high)")
+    axes[1].set_ylabel("Median absolute error")
+    axes[1].set_title(f"{dataset_name}: Calibration")
+    axes[1].set_xticks(range(n_bins))
+    axes[1].set_xticklabels([f"Q{i+1}" for i in range(n_bins)])
+
+    fig.tight_layout()
+    save_fig(fig, f"{dataset_name}_uncertainty_calibration")
+
+    return result
+
+
+# ============================================================================
+# 4. DEEPPTR SUBSAMPLING ROBUSTNESS
+# ============================================================================
+
+def run_deep_subsampling(adata_loader, dataset_name, fractions=(0.5, 0.8)):
+    """Test DeepPTR robustness by retraining on subsampled cells."""
+    print(f"\n--- DeepPTR subsampling robustness ---")
+
+    # Full model
+    adata_full = adata_loader()
+    scptr.pp.filter_genes(adata_full)
+    scptr.pp.normalize_layers(adata_full)
+    scptr.pp.neighbors(adata_full, n_neighbors=30)
+    scptr.pp.smooth_layers(adata_full)
+    scptr.tl.estimate_beta(adata_full)
+    adata_full = select_top_genes(adata_full, n_top=300)
+
+    torch.set_num_threads(4)
+    model_full, _ = scptr.deep.fit_deepptr(
+        adata_full,
+        d_T=8, d_PT=8, d_hidden=48, n_enc_layers=2,
+        batch_size=512, max_epochs=100, kl_warmup_epochs=20,
+        patience=15, n_posterior_samples=10,
+        device="cpu", seed=0, verbose=False,
+    )
+    gamma_full = np.median(adata_full.layers["gamma"], axis=0)
+
+    records = []
+    rng = np.random.RandomState(42)
+
+    for frac in fractions:
+        n_sub = max(int(adata_full.n_obs * frac), 50)
+        idx = rng.choice(adata_full.n_obs, size=n_sub, replace=False)
+
+        adata_sub = adata_full[idx].copy()
+        # Ensure dense
+        from scipy.sparse import issparse
+        for key in ("spliced", "unspliced"):
+            if key in adata_sub.layers and issparse(adata_sub.layers[key]):
+                adata_sub.layers[key] = np.asarray(adata_sub.layers[key].todense())
+
+        torch.set_num_threads(4)
+        _, _ = scptr.deep.fit_deepptr(
+            adata_sub,
+            d_T=8, d_PT=8, d_hidden=48, n_enc_layers=2,
+            batch_size=512, max_epochs=100, kl_warmup_epochs=20,
+            patience=15, n_posterior_samples=10,
+            device="cpu", seed=0, verbose=False,
+        )
+        gamma_sub = np.median(adata_sub.layers["gamma"], axis=0)
+
+        valid = np.isfinite(gamma_full) & np.isfinite(gamma_sub)
+        sp_r, _ = stats.spearmanr(gamma_full[valid], gamma_sub[valid])
+
+        records.append({
+            "fraction": frac,
+            "n_cells": n_sub,
+            "spearman_r": float(sp_r),
+        })
+        print(f"  {frac*100:.0f}%: r={sp_r:.4f} (n_cells={n_sub})")
+
+    return records
+
+
+# ============================================================================
+# 5. LATENT SPACE VISUALIZATION
+# ============================================================================
+
+def run_latent_analysis(adata_deep, dataset_name, cluster_key="clusters"):
+    """Visualize DeepPTR latent spaces with UMAP."""
+    print(f"\n--- Latent space visualization ---")
+    import scanpy as sc
+
+    if "X_z_T" not in adata_deep.obsm or "X_z_PT" not in adata_deep.obsm:
+        print("  [SKIP] No latent embeddings found")
+        return None
+
+    fig, axes = plt.subplots(1, 3, figsize=(18, 5))
+
+    has_ct = cluster_key in adata_deep.obs.columns
+
+    for ax_idx, (key, title) in enumerate([
+        ("X_z_T", "z_T (transcription)"),
+        ("X_z_PT", "z_PT (post-transcription)"),
+    ]):
+        z = adata_deep.obsm[key]
+        # Quick PCA+UMAP for visualization
+        from sklearn.decomposition import PCA
+        if z.shape[1] > 2:
+            pca = PCA(n_components=2)
+            z_2d = pca.fit_transform(z)
+        else:
+            z_2d = z
+
+        if has_ct:
+            categories = adata_deep.obs[cluster_key].astype("category")
+            codes = categories.cat.codes.values
+            cmap = plt.cm.get_cmap("tab20", len(categories.cat.categories))
+            scatter = axes[ax_idx].scatter(z_2d[:, 0], z_2d[:, 1], c=codes,
+                                            cmap=cmap, alpha=0.3, s=3)
+        else:
+            axes[ax_idx].scatter(z_2d[:, 0], z_2d[:, 1], alpha=0.3, s=3, c="steelblue")
+        axes[ax_idx].set_title(title)
+        axes[ax_idx].set_xlabel("PC1")
+        axes[ax_idx].set_ylabel("PC2")
+
+    # Third panel: gamma PCA
+    gamma = adata_deep.layers["gamma"]
+    from sklearn.decomposition import PCA
+    pca = PCA(n_components=2)
+    g_2d = pca.fit_transform(gamma)
+    if has_ct:
+        categories = adata_deep.obs[cluster_key].astype("category")
+        codes = categories.cat.codes.values
+        cmap = plt.cm.get_cmap("tab20", len(categories.cat.categories))
+        axes[2].scatter(g_2d[:, 0], g_2d[:, 1], c=codes, cmap=cmap, alpha=0.3, s=3)
+    else:
+        axes[2].scatter(g_2d[:, 0], g_2d[:, 1], alpha=0.3, s=3, c="steelblue")
+    axes[2].set_title("gamma (DeepPTR)")
+    axes[2].set_xlabel("PC1")
+    axes[2].set_ylabel("PC2")
+
+    if has_ct:
+        cats = categories.cat.categories.tolist()
+        if len(cats) <= 15:
+            handles = [plt.Line2D([0], [0], marker="o", color="w",
+                                   markerfacecolor=cmap(i), markersize=6, label=c)
+                       for i, c in enumerate(cats)]
+            fig.legend(handles=handles, loc="center right", fontsize=7,
+                      bbox_to_anchor=(1.15, 0.5))
+
+    fig.suptitle(f"{dataset_name}: DeepPTR Latent Spaces", y=1.02)
+    fig.tight_layout()
+    save_fig(fig, f"{dataset_name}_latent_spaces")
+
+    # Quantify: silhouette score of cell types in latent space
+    if has_ct and len(categories.cat.categories) >= 2:
+        from sklearn.metrics import silhouette_score
+        codes = categories.cat.codes.values
+        sil_T = silhouette_score(adata_deep.obsm["X_z_T"], codes, sample_size=min(2000, len(codes)))
+        sil_PT = silhouette_score(adata_deep.obsm["X_z_PT"], codes, sample_size=min(2000, len(codes)))
+        sil_gamma = silhouette_score(gamma, codes, sample_size=min(2000, len(codes)))
+        print(f"  Silhouette: z_T={sil_T:.4f}, z_PT={sil_PT:.4f}, gamma={sil_gamma:.4f}")
+        return {"silhouette_z_T": sil_T, "silhouette_z_PT": sil_PT, "silhouette_gamma": sil_gamma}
+
+    return None
+
+
+# ============================================================================
+# 6. GENE RANKING COMPARISON
+# ============================================================================
+
+def run_gene_ranking(adata_an, adata_deep, dataset_name, n_top=50):
+    """Compare top differentially-degraded genes between methods."""
+    print(f"\n--- Gene ranking comparison (top {n_top}) ---")
+
+    shared_genes = adata_an.var_names.intersection(adata_deep.var_names)
+    if len(shared_genes) < 20:
+        print("  [SKIP] Too few shared genes")
+        return None
+
+    gamma_an = pd.Series(
+        np.median(adata_an.layers["gamma"], axis=0), index=adata_an.var_names
+    )
+    gamma_dp = pd.Series(
+        np.median(adata_deep.layers["gamma"], axis=0), index=adata_deep.var_names
+    )
+
+    # Variance of gamma across cells (identifies genes with heterogeneous degradation)
+    gamma_var_an = pd.Series(
+        np.var(adata_an.layers["gamma"], axis=0), index=adata_an.var_names
+    )
+    gamma_var_dp = pd.Series(
+        np.var(adata_deep.layers["gamma"], axis=0), index=adata_deep.var_names
+    )
+
+    # Top genes by median gamma (shared)
+    top_an = gamma_an[shared_genes].nlargest(n_top).index.tolist()
+    top_dp = gamma_dp[shared_genes].nlargest(n_top).index.tolist()
+    overlap_median = len(set(top_an) & set(top_dp))
+
+    # Top genes by gamma variance (shared)
+    top_var_an = gamma_var_an[shared_genes].nlargest(n_top).index.tolist()
+    top_var_dp = gamma_var_dp[shared_genes].nlargest(n_top).index.tolist()
+    overlap_var = len(set(top_var_an) & set(top_var_dp))
+
+    # Rank correlation on shared genes
+    ranks_an = gamma_an[shared_genes].rank(ascending=False)
+    ranks_dp = gamma_dp[shared_genes].rank(ascending=False)
+    rank_corr, _ = stats.spearmanr(ranks_an.values, ranks_dp.values)
+
+    result = {
+        "n_shared_genes": len(shared_genes),
+        "top_median_overlap": overlap_median,
+        "top_median_overlap_frac": overlap_median / n_top,
+        "top_var_overlap": overlap_var,
+        "top_var_overlap_frac": overlap_var / n_top,
+        "rank_correlation": float(rank_corr),
+    }
+    print(f"  Top-{n_top} median gamma overlap: {overlap_median}/{n_top} ({overlap_median/n_top*100:.0f}%)")
+    print(f"  Top-{n_top} var gamma overlap:    {overlap_var}/{n_top} ({overlap_var/n_top*100:.0f}%)")
+    print(f"  Rank correlation (shared genes):   {rank_corr:.4f}")
+
+    return result
+
+
+# ============================================================================
+# MAIN: RUN ON EACH DATASET
+# ============================================================================
+
+def run_dataset(name, adata_loader, cluster_key="clusters"):
+    """Run all expanded benchmarks on one dataset."""
+    print(f"\n{'=' * 60}")
+    print(f"DATASET: {name.upper()}")
+    print("=" * 60)
+
+    # --- Analytical ---
+    print(f"\n--- Analytical pipeline ---")
+    adata_an = adata_loader()
+    run_analytical_pipeline(adata_an)
+    print(f"  Analytical: {adata_an.shape}")
+
+    # --- DeepPTR ---
+    print(f"\n--- DeepPTR ---")
+    adata_deep = adata_loader()
+    scptr.pp.filter_genes(adata_deep)
+    scptr.pp.normalize_layers(adata_deep)
+    scptr.pp.neighbors(adata_deep, n_neighbors=30)
+    scptr.pp.smooth_layers(adata_deep)
+    scptr.tl.estimate_beta(adata_deep)
+    adata_deep = select_top_genes(adata_deep, n_top=300)
+
+    t0 = time.time()
+    model, history = fit_deep(adata_deep)
+    elapsed = time.time() - t0
+    print(f"  DeepPTR: {len(history.train_loss)} epochs in {elapsed:.1f}s")
+
+    all_results = {"dataset": name, "n_epochs": len(history.train_loss), "time_s": elapsed}
+
+    # 1. Enrichment
+    enrich = run_enrichment_mapped(adata_an, adata_deep, name)
+    all_results["enrichment"] = enrich
+
+    # 2. Per-cell-type
+    ct_results = run_celltype_agreement(adata_an, adata_deep, name, cluster_key)
+    all_results["celltype_agreement"] = ct_results
+
+    # 3. Uncertainty
+    unc_results = run_uncertainty_analysis(adata_an, adata_deep, name)
+    all_results["uncertainty"] = unc_results
+
+    # 4. Latent space
+    lat_results = run_latent_analysis(adata_deep, name, cluster_key)
+    all_results["latent_structure"] = lat_results
+
+    # 5. Gene ranking
+    rank_results = run_gene_ranking(adata_an, adata_deep, name)
+    all_results["gene_ranking"] = rank_results
+
+    # Save
+    with open(OUTPUT_DIR / "results" / f"{name}_v2.json", "w") as f:
+        json.dump(all_results, f, indent=2, default=str)
+
+    return all_results
+
+
+def run_cross_dataset_consistency(datasets):
+    """Compare cross-dataset consistency for analytical vs DeepPTR."""
+    print(f"\n{'=' * 60}")
+    print("CROSS-DATASET CONSISTENCY")
+    print("=" * 60)
+
+    # Build analytical and deep adatas
+    an_dict = {}
+    dp_dict = {}
+
+    for name, loader, cluster_key in datasets:
+        print(f"\n  Processing {name}...")
+        adata_an = loader()
+        run_analytical_pipeline(adata_an)
+        an_dict[name] = adata_an
+
+        adata_dp = loader()
+        scptr.pp.filter_genes(adata_dp)
+        scptr.pp.normalize_layers(adata_dp)
+        scptr.pp.neighbors(adata_dp, n_neighbors=30)
+        scptr.pp.smooth_layers(adata_dp)
+        scptr.tl.estimate_beta(adata_dp)
+        adata_dp = select_top_genes(adata_dp, n_top=300)
+        torch.set_num_threads(4)
+        scptr.deep.fit_deepptr(
+            adata_dp,
+            d_T=8, d_PT=8, d_hidden=48, n_enc_layers=2,
+            batch_size=512, max_epochs=100, kl_warmup_epochs=20,
+            patience=15, n_posterior_samples=10,
+            device="cpu", seed=0, verbose=False,
+        )
+        dp_dict[name] = adata_dp
+
+    print(f"\n--- Analytical cross-dataset ---")
+    cons_an = scptr.benchmark.cross_dataset_consistency(an_dict)
+    print(cons_an.to_string(index=False))
+
+    print(f"\n--- DeepPTR cross-dataset ---")
+    cons_dp = scptr.benchmark.cross_dataset_consistency(dp_dict)
+    print(cons_dp.to_string(index=False))
+
+    result = {
+        "analytical": cons_an.to_dict(orient="records"),
+        "deepptr": cons_dp.to_dict(orient="records"),
+    }
+
+    with open(OUTPUT_DIR / "results" / "cross_dataset_consistency.json", "w") as f:
+        json.dump(result, f, indent=2, default=str)
+
+    return result
+
+
+def run_subsampling_all(datasets):
+    """Run DeepPTR subsampling robustness on each dataset."""
+    print(f"\n{'=' * 60}")
+    print("DEEPPTR SUBSAMPLING ROBUSTNESS")
+    print("=" * 60)
+
+    all_results = {}
+    for name, loader, _ in datasets:
+        print(f"\n  {name}:")
+        records = run_deep_subsampling(loader, name, fractions=(0.5, 0.8))
+        all_results[name] = records
+
+    with open(OUTPUT_DIR / "results" / "subsampling_robustness.json", "w") as f:
+        json.dump(all_results, f, indent=2, default=str)
+
+    return all_results
+
+
+def print_summary(results, cross_ds, subsampling):
+    """Print final summary table."""
+    print(f"\n{'=' * 70}")
+    print("EXPANDED BENCHMARK SUMMARY")
+    print("=" * 70)
+
+    for name, res in results.items():
+        print(f"\n  {name.upper()}")
+
+        # Enrichment
+        enrich = res.get("enrichment", {})
+        for test in ("ARE", "NMD"):
+            if test in enrich:
+                p_an = enrich[test].get("analytical", {}).get("p_value", np.nan)
+                p_hy = enrich[test].get("hybrid_deepptr", {}).get("p_value", np.nan)
+                print(f"    {test} enrichment: analytical p={p_an:.2e}, hybrid p={p_hy:.2e}")
+
+        # Cell-type agreement
+        ct = res.get("celltype_agreement")
+        if ct:
+            median_r = np.median([r["spearman_r"] for r in ct])
+            print(f"    Cell-type agreement: median r={median_r:.4f} ({len(ct)} types)")
+
+        # Uncertainty
+        unc = res.get("uncertainty")
+        if unc:
+            print(f"    Uncertainty calibration: var-error r={unc['var_error_spearman_r']:.4f}")
+
+        # Latent
+        lat = res.get("latent_structure")
+        if lat:
+            print(f"    Silhouette: z_T={lat['silhouette_z_T']:.4f}, z_PT={lat['silhouette_z_PT']:.4f}, gamma={lat['silhouette_gamma']:.4f}")
+
+        # Gene ranking
+        rank = res.get("gene_ranking")
+        if rank:
+            print(f"    Gene ranking: top-50 overlap={rank['top_median_overlap']}/50, rank r={rank['rank_correlation']:.4f}")
+
+    # Cross-dataset
+    if cross_ds:
+        print(f"\n  CROSS-DATASET CONSISTENCY")
+        for method in ("analytical", "deepptr"):
+            entries = cross_ds.get(method, [])
+            for e in entries:
+                print(f"    {method}: {e['dataset_a']} vs {e['dataset_b']}: "
+                      f"r={e['spearman_r']:.4f} (n={e['n_shared_genes']})")
+
+    # Subsampling
+    if subsampling:
+        print(f"\n  SUBSAMPLING ROBUSTNESS (DeepPTR)")
+        for ds_name, records in subsampling.items():
+            for r in records:
+                print(f"    {ds_name} @ {r['fraction']*100:.0f}%: r={r['spearman_r']:.4f}")
+
+
+def main():
+    set_figure_style()
+    ensure_dirs()
+
+    datasets = [
+        ("pancreas", scptr.datasets.pancreas, "clusters"),
+        ("dentate_gyrus", scptr.datasets.dentate_gyrus, "clusters"),
+    ]
+
+    # Per-dataset analysis
+    results = {}
+    for name, loader, cluster_key in datasets:
+        results[name] = run_dataset(name, loader, cluster_key)
+
+    # Cross-dataset consistency
+    cross_ds = run_cross_dataset_consistency(datasets)
+
+    # Subsampling robustness
+    subsampling = run_subsampling_all(datasets)
+
+    # Summary
+    print_summary(results, cross_ds, subsampling)
+
+    # Save combined
+    combined = {
+        "per_dataset": {k: v for k, v in results.items()},
+        "cross_dataset": cross_ds,
+        "subsampling": subsampling,
+    }
+    with open(OUTPUT_DIR / "results" / "combined_v2.json", "w") as f:
+        json.dump(combined, f, indent=2, default=str)
+
+    print(f"\nAll results saved to: {OUTPUT_DIR}")
+
+
+if __name__ == "__main__":
+    main()

analyses/run_dentate_gyrus.py

@@ -0,0 +1,252 @@
+#!/usr/bin/env python
+"""Run the full scPTR analysis pipeline on dentate gyrus data.
+
+This script mirrors run_all.py but on the dentate gyrus neurogenesis dataset.
+Results are saved to output/dentate_gyrus/ directory.
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+
+sys.path.insert(0, str(Path(__file__).parent))
+from _common import set_figure_style
+
+import scptr
+
+OUTPUT_DIR = Path(__file__).parent.parent / "output" / "dentate_gyrus"
+
+
+def save_fig(fig, name, subdir="figures"):
+    """Save a matplotlib figure to output dir."""
+    if fig is None:
+        print(f"  [WARNING] {name}: plot returned None, skipping save")
+        return
+    out_dir = OUTPUT_DIR / subdir
+    out_dir.mkdir(parents=True, exist_ok=True)
+    path = out_dir / f"{name}.png"
+    fig.savefig(path, dpi=150, bbox_inches="tight")
+    plt.close(fig)
+    print(f"  Saved: {path}")
+
+
+def main():
+    set_figure_style()
+    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+
+    # =========================================================================
+    # LOAD DATA
+    # =========================================================================
+    print("=" * 60)
+    print("LOADING DENTATE GYRUS DATASET")
+    print("=" * 60)
+    adata = scptr.datasets.dentate_gyrus()
+    print(f"  Shape: {adata.shape}")
+    print(f"  Layers: {list(adata.layers.keys())}")
+    print(f"  Cell types: {adata.obs['clusters'].value_counts().to_dict()}")
+
+    # =========================================================================
+    # PREPROCESSING
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("PREPROCESSING")
+    print("=" * 60)
+
+    scptr.pp.filter_genes(adata)
+    print(f"  After filtering: {adata.shape}")
+
+    scptr.pp.normalize_layers(adata)
+    print("  Normalized layers")
+
+    scptr.pp.neighbors(adata, n_neighbors=30)
+    print("  Built kNN graph (k=30)")
+
+    scptr.pp.smooth_layers(adata)
+    print("  Smoothed layers (Mu, Ms)")
+
+    # =========================================================================
+    # CORE ANALYSIS
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("CORE ANALYSIS")
+    print("=" * 60)
+
+    scptr.tl.estimate_beta(adata)
+    beta = adata.var['beta'].values
+    print(f"  Beta: median={np.median(beta):.4f}, max={np.max(beta):.4f}, "
+          f"nonzero={np.sum(beta > 0)}/{len(beta)}")
+
+    scptr.tl.estimate_beta(adata, groupby="clusters")
+    print(f"  Beta (per-cluster): {adata.varm['beta_groups'].shape}")
+
+    scptr.tl.estimate_gamma(adata)
+    gamma_vals = adata.layers["gamma"]
+    gamma_med = np.median(gamma_vals, axis=0)
+    print(f"  Gamma: shape={gamma_vals.shape}")
+    print(f"    Median per-gene: median={np.median(gamma_med):.4f}, "
+          f"max={np.max(gamma_med):.4f}")
+    print(f"    Global max={np.max(gamma_vals):.4f}")
+    print(f"    Genes with >0 median gamma: {np.sum(gamma_med > 0)}/{len(gamma_med)}")
+
+    scptr.tl.variance_decomposition(adata)
+    tf = adata.var['tf_score'].values
+    print(f"  TF score:  median={np.median(tf):.4f}, mean={np.mean(tf):.4f}")
+    print(f"  Genes with TF > 0.5: {np.sum(tf > 0.5)}/{len(tf)}")
+
+    scptr.tl.pt_states(adata)
+    n_states = adata.obs["pt_state"].nunique()
+    print(f"  PT states found: {n_states}")
+
+    scptr.tl.pt_velocity(adata)
+    print("  PT velocity computed")
+
+    # =========================================================================
+    # BENCHMARKING
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("BENCHMARKING")
+    print("=" * 60)
+    res_dir = OUTPUT_DIR / "results"
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    # Half-life correlation (mouse reference — dentate gyrus is mouse data)
+    print("\n--- Half-life correlation (mouse reference) ---")
+    hl_mouse = scptr.datasets.herzog2017_halflives()
+    corr = scptr.benchmark.correlate_with_halflives(adata, hl_mouse)
+    print(f"  n_genes matched: {corr['n_genes']} (unfiltered: {corr['n_genes_unfiltered']})")
+    print(f"  Spearman r = {corr['spearman_r']:.4f} (p = {corr['spearman_p']:.2e})")
+    print(f"  Pearson  r = {corr['pearson_r']:.4f} (p = {corr['pearson_p']:.2e})")
+
+    # Also human reference
+    print("\n--- Half-life correlation (human reference) ---")
+    hl_human = scptr.datasets.schofield2018_halflives()
+    corr_human = scptr.benchmark.correlate_with_halflives(adata, hl_human)
+    print(f"  n_genes matched: {corr_human['n_genes']} (unfiltered: {corr_human['n_genes_unfiltered']})")
+    print(f"  Spearman r = {corr_human['spearman_r']:.4f} (p = {corr_human['spearman_p']:.2e})")
+
+    corr_save = {k: v for k, v in corr.items() if k != "matched_genes"}
+    corr_human_save = {k: v for k, v in corr_human.items() if k != "matched_genes"}
+    with open(res_dir / "halflife_correlation.json", "w") as f:
+        json.dump({"mouse_reference": corr_save, "human_reference": corr_human_save}, f, indent=2)
+
+    # Half-life scatter
+    fig, axes = plt.subplots(1, 2, figsize=(13, 5))
+    gamma_med_s = pd.Series(gamma_med, index=adata.var_names)
+    hl_s = hl_mouse.set_index("gene_symbol")["half_life_hours"]
+    shared = gamma_med_s.index.intersection(hl_s.index)
+    g = gamma_med_s[shared].values
+    h = hl_s[shared].values
+
+    axes[0].scatter(h, g, alpha=0.1, s=5, c="steelblue")
+    axes[0].set_xlabel("Published half-life (hours)")
+    axes[0].set_ylabel("scPTR median gamma")
+    axes[0].set_title(f"All genes (n={len(shared)})")
+
+    mask = (g > 0) & (h > 0) & np.isfinite(g) & np.isfinite(h)
+    axes[1].scatter(h[mask], g[mask], alpha=0.15, s=8, c="steelblue")
+    axes[1].set_xscale("log")
+    axes[1].set_yscale("log")
+    axes[1].set_xlabel("Published half-life (hours)")
+    axes[1].set_ylabel("scPTR median gamma")
+    axes[1].set_title(
+        f"Filtered (Spearman r={corr['spearman_r']:.3f}, "
+        f"p={corr['spearman_p']:.1e}, n={corr['n_genes']})"
+    )
+    fig.suptitle("Dentate Gyrus: Gamma vs Published Half-lives", fontsize=13, y=1.02)
+    fig.tight_layout()
+    save_fig(fig, "halflife_scatter")
+
+    # ARE/NMD enrichment
+    print("\n--- ARE / NMD enrichment ---")
+    are_result = scptr.benchmark.are_enrichment(adata)
+    nmd_result = scptr.benchmark.nmd_enrichment(adata)
+    print(f"  ARE: n_in={are_result['n_genes_in_set']}, p={are_result['p_value']:.4f}")
+    print(f"  NMD: n_in={nmd_result['n_genes_in_set']}, p={nmd_result['p_value']:.4f}")
+
+    with open(res_dir / "enrichment_results.json", "w") as f:
+        json.dump({"ARE": are_result, "NMD": nmd_result}, f, indent=2)
+
+    fig = scptr.pl.enrichment_barplot([are_result, nmd_result])
+    save_fig(fig, "enrichment_barplot")
+
+    # Subsampling robustness
+    print("\n--- Subsampling robustness ---")
+    fractions = [0.2, 0.4, 0.6, 0.8, 0.9]
+    robust_df = scptr.benchmark.subsampling_robustness(
+        adata, fractions=fractions, n_repeats=5
+    )
+    robust_df.to_csv(res_dir / "subsampling_robustness.csv", index=False)
+    for frac in fractions:
+        sub = robust_df[robust_df["fraction"] == frac]
+        print(f"  fraction={frac:.1f}: mean Spearman r = {sub['spearman_r'].mean():.4f}")
+
+    # =========================================================================
+    # PT STATES
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("PT STATE DISCOVERY")
+    print("=" * 60)
+
+    state_counts = adata.obs["pt_state"].value_counts()
+    state_counts.to_csv(res_dir / "pt_state_counts.csv")
+    print(f"  PT states: {dict(state_counts)}")
+
+    fig = scptr.pl.pt_umap(adata, show=False)
+    save_fig(fig, "pt_umap")
+
+    fig = scptr.pl.tf_ptf_scatter(adata, show=False)
+    save_fig(fig, "tf_ptf_scatter")
+
+    ct = pd.crosstab(adata.obs["pt_state"], adata.obs["clusters"])
+    ct.to_csv(res_dir / "pt_state_vs_clusters.csv")
+    print(f"\n  PT state vs expression cluster crosstab:")
+    print(ct.to_string())
+
+    rank_df = scptr.tl.rank_pt_genes(adata, n_genes=50)
+    rank_df.to_csv(res_dir / "ranked_pt_genes.csv", index=False)
+    print(f"\n  Top differentially degraded genes: {len(rank_df)} entries")
+    print(f"  Top 10: {rank_df.head(10)['names'].tolist()}")
+
+    fig = scptr.pl.gamma_heatmap(adata, show=False)
+    save_fig(fig, "gamma_heatmap")
+
+    # =========================================================================
+    # PT VELOCITY
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("PT VELOCITY")
+    print("=" * 60)
+
+    fig = scptr.pl.pt_velocity_embedding(adata, density=0.3, arrow_size=1.5, show=False)
+    save_fig(fig, "pt_velocity_embedding")
+
+    # =========================================================================
+    # SUMMARY
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("SUMMARY")
+    print("=" * 60)
+    print(f"  Dataset: dentate_gyrus ({adata.n_obs} cells, {adata.n_vars} genes)")
+    print(f"  Beta: median={np.median(adata.var['beta']):.4f}, max={np.max(adata.var['beta']):.4f}")
+    print(f"  Gamma max: {np.max(adata.layers['gamma']):.4f}")
+    print(f"  PT states discovered: {n_states}")
+    print(f"  TF score: median={np.median(adata.var['tf_score']):.4f}")
+    print(f"  Half-life Spearman r (mouse): {corr['spearman_r']:.4f} (n={corr['n_genes']})")
+    print(f"  Half-life Spearman r (human): {corr_human['spearman_r']:.4f} (n={corr_human['n_genes']})")
+    print(f"  Robustness (90%): {robust_df[robust_df['fraction']==0.9]['spearman_r'].mean():.4f}")
+    print(f"\nAll results saved to: {OUTPUT_DIR.resolve()}")
+
+    # Return adata for cross-dataset use
+    return adata
+
+
+if __name__ == "__main__":
+    main()

analyses/run_final_fixes.py

@@ -0,0 +1,524 @@
+#!/usr/bin/env python
+"""Final publication fixes: address critical reviewer concerns.
+
+1. Fix Fisher's exact test bug in hub consistency (contingency table was wrong)
+2. Pathway specificity analysis (scPTR vs unspliced-only: unique vs generic pathways)
+3. Per-cell sci-fate: stratify by expression level to show where scPTR advantage is largest
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+from scipy import stats
+
+sys.path.insert(0, str(Path(__file__).parent))
+from _common import set_figure_style
+
+OUTPUT_DIR = Path(__file__).parent.parent / "output" / "final_fixes"
+PROJECT_ROOT = Path(__file__).parent.parent
+
+
+def save_fig(fig, name, subdir="figures"):
+    if fig is None:
+        return
+    out_dir = OUTPUT_DIR / subdir
+    out_dir.mkdir(parents=True, exist_ok=True)
+    path = out_dir / f"{name}.png"
+    fig.savefig(path, dpi=150, bbox_inches="tight")
+    plt.close(fig)
+    print(f"  Saved: {path}")
+
+
+# =========================================================================
+# 1. Fix Fisher's exact test for hub consistency
+# =========================================================================
+def fix_hub_fisher():
+    """Recompute Fisher's exact using only shared RBPs as the universe."""
+    print("\n" + "=" * 60)
+    print("1. CORRECTED FISHER'S EXACT FOR HUB CONSISTENCY")
+    print("=" * 60)
+
+    res_dir = OUTPUT_DIR / "results"
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    # Load hub counts
+    hub_files = {
+        "pancreas": PROJECT_ROOT / "output" / "gap_analysis" / "results" / "network" / "pancreas" / "rbp_hub_counts.csv",
+        "dentate_gyrus": PROJECT_ROOT / "output" / "gap_analysis" / "results" / "network" / "dentate_gyrus" / "rbp_hub_counts.csv",
+    }
+
+    hub_counts = {}
+    for name, path in hub_files.items():
+        if path.exists():
+            df = pd.read_csv(path)
+            count_col = [c for c in df.columns if c != "rbp"][0]
+            hub_counts[name] = pd.Series(df[count_col].values, index=df["rbp"].values)
+
+    # NB from corrected network
+    nb_net_path = PROJECT_ROOT / "output" / "tier3" / "results" / "neuroblastoma_network_corrected.csv"
+    if nb_net_path.exists():
+        nb_net = pd.read_csv(nb_net_path)
+        hub_counts["neuroblastoma"] = nb_net.groupby("rbp").size().sort_values(ascending=False)
+
+    # Uppercase
+    hub_upper = {}
+    for name, series in hub_counts.items():
+        hub_upper[name] = pd.Series(series.values, index=[g.upper() for g in series.index])
+
+    names = sorted(hub_upper.keys())
+    results = []
+
+    print("\n  Corrected Fisher's exact (universe = shared RBPs only):")
+    for i, name_a in enumerate(names):
+        for j in range(len(names)):
+            if i == j:
+                continue
+            name_b = names[j]
+            shared = set(hub_upper[name_a].index) & set(hub_upper[name_b].index)
+            n_shared = len(shared)
+            if n_shared < 5:
+                continue
+
+            # Rank RBPs WITHIN the shared set only
+            shared_a = hub_upper[name_a].reindex(list(shared)).dropna().sort_values(ascending=False)
+            shared_b = hub_upper[name_b].reindex(list(shared)).dropna().sort_values(ascending=False)
+
+            # Top-k from A (within shared), top-k from B (within shared)
+            k_a = min(5, n_shared // 3)  # top third or 5
+            k_b = min(10, n_shared // 2)  # top half or 10
+
+            top_a = set(shared_a.index[:k_a])
+            top_b = set(shared_b.index[:k_b])
+
+            # 2x2 contingency table (universe = shared)
+            a_and_b = len(top_a & top_b)
+            a_not_b = len(top_a - top_b)
+            b_not_a = len(top_b - top_a)
+            neither = n_shared - a_and_b - a_not_b - b_not_a
+
+            table = [[a_and_b, a_not_b], [b_not_a, neither]]
+            odds_ratio, fisher_p = stats.fisher_exact(table, alternative="greater")
+            print(f"  Top-{k_a} {name_a} in top-{k_b} {name_b}: "
+                  f"{a_and_b}/{k_a} overlap, OR={odds_ratio:.2f}, p={fisher_p:.4f} "
+                  f"(n_shared={n_shared})")
+
+            results.append({
+                "source": name_a, "target": name_b,
+                "k_source": k_a, "k_target": k_b,
+                "overlap": a_and_b, "n_shared": n_shared,
+                "odds_ratio": float(odds_ratio), "fisher_p": float(fisher_p),
+            })
+
+    with open(res_dir / "corrected_hub_fisher.json", "w") as f:
+        json.dump(results, f, indent=2)
+
+    return results
+
+
+# =========================================================================
+# 2. Pathway specificity: unique tissue pathways per method
+# =========================================================================
+def pathway_specificity():
+    """Analyze whether scPTR finds different or more specific pathways
+    than unspliced-only, rather than just counting totals."""
+    print("\n" + "=" * 60)
+    print("2. PATHWAY SPECIFICITY ANALYSIS")
+    print("=" * 60)
+
+    res_dir = OUTPUT_DIR / "results"
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    # Load full coherence ablation results
+    coherence_csv = PROJECT_ROOT / "output" / "comprehensive_fixes" / "results" / "coherence_ablation.csv"
+    pathways_csv = PROJECT_ROOT / "output" / "comprehensive_fixes" / "results" / "coherence_ablation_pathways.csv"
+
+    coherence_df = pd.read_csv(coherence_csv)
+    pathways_df = pd.read_csv(pathways_csv)
+
+    # Generic housekeeping pathways (appear in every tissue, not informative)
+    generic_pathways = {
+        "ribosome", "oxidative phosphorylation", "thermogenesis",
+        "huntington disease", "alzheimer disease", "parkinson disease",
+        "non-alcoholic fatty liver disease", "cardiac muscle contraction",
+        "diabetic cardiomyopathy", "chemical carcinogenesis",
+        "metabolic pathways", "carbon metabolism",
+    }
+
+    # Tissue-specific pathways (the ones we care about)
+    tissue_specific = {
+        "pancreas": {
+            "protein processing in endoplasmic reticulum", "autophagy",
+            "insulin secretion", "insulin signaling pathway",
+            "pancreatic secretion", "maturity onset diabetes",
+            "unfolded protein response", "protein folding",
+        },
+        "dentate_gyrus": {
+            "synaptic vesicle cycle", "long-term potentiation",
+            "glutamatergic synapse", "gabaergic synapse",
+            "axon guidance", "neurotrophin signaling pathway",
+            "dopaminergic synapse", "serotonergic synapse",
+        },
+    }
+
+    results = {}
+
+    for dataset in ["pancreas", "dentate_gyrus"]:
+        print(f"\n--- {dataset} ---")
+        ds_paths = pathways_df[pathways_df["dataset"] == dataset]
+        ds_coherence = coherence_df[coherence_df["dataset"] == dataset]
+
+        expected_set = tissue_specific.get(dataset, set())
+
+        for method in ["scPTR_gamma", "raw_u_s_ratio", "unspliced_only"]:
+            method_paths = ds_paths[ds_paths["method"] == method]
+            all_terms = [t.lower() for t in method_paths["pathway"].values]
+
+            n_total = len(all_terms)
+            n_generic = sum(1 for t in all_terms
+                          if any(g in t for g in generic_pathways))
+            n_tissue = sum(1 for t in all_terms
+                          if any(ts in t for ts in expected_set))
+            n_specific = n_total - n_generic
+
+            # Unique pathways (found by this method but not others)
+            other_methods = [m for m in ["scPTR_gamma", "raw_u_s_ratio", "unspliced_only"]
+                           if m != method]
+            other_terms = set()
+            for om in other_methods:
+                om_paths = ds_paths[ds_paths["method"] == om]
+                other_terms |= set(t.lower() for t in om_paths["pathway"].values)
+
+            unique_terms = [t for t in all_terms if t not in other_terms]
+            n_unique = len(unique_terms)
+
+            # From coherence CSV: mean invisibility, mean diff genes
+            mc = ds_coherence[ds_coherence["method"] == method]
+
+            key = f"{dataset}_{method}"
+            results[key] = {
+                "dataset": dataset, "method": method,
+                "n_total_pathways": n_total,
+                "n_generic": n_generic,
+                "n_tissue_specific": n_tissue,
+                "n_non_generic": n_specific,
+                "n_unique_to_method": n_unique,
+                "generic_fraction": n_generic / max(n_total, 1),
+                "mean_invisibility": float(mc["invisibility"].mean()) if len(mc) > 0 else 0,
+                "mean_diff_genes": float(mc["n_diff_genes"].mean()) if len(mc) > 0 else 0,
+            }
+
+            print(f"  {method:<20s}: {n_total} total, {n_generic} generic, "
+                  f"{n_tissue} tissue-specific, {n_unique} unique")
+
+    # Cross-method comparison: per-cluster agreement
+    print("\n  Per-cluster: do all methods find the same expected pathways?")
+    clusters_tested = coherence_df["cluster"].unique()
+    agreement_data = []
+
+    for cluster in clusters_tested:
+        cluster_data = coherence_df[coherence_df["cluster"] == cluster]
+        for _, row in cluster_data.iterrows():
+            agreement_data.append({
+                "cluster": cluster,
+                "dataset": row["dataset"],
+                "method": row["method"],
+                "n_expected": row["n_expected_pathways"],
+                "n_sig": row["n_sig_pathways"],
+                "n_diff_genes": row["n_diff_genes"],
+            })
+
+    agreement_df = pd.DataFrame(agreement_data)
+
+    # Key metric: per cluster, which method finds the MOST expected pathways?
+    print("\n  Per-cluster winner (most expected pathways):")
+    winner_counts = {"scPTR_gamma": 0, "raw_u_s_ratio": 0, "unspliced_only": 0, "tie": 0}
+
+    for cluster in clusters_tested:
+        cl = agreement_df[agreement_df["cluster"] == cluster]
+        if len(cl) == 0:
+            continue
+        max_expected = cl["n_expected"].max()
+        winners = cl[cl["n_expected"] == max_expected]["method"].tolist()
+        if len(winners) == 1:
+            winner_counts[winners[0]] += 1
+        else:
+            winner_counts["tie"] += 1
+
+    for method, count in winner_counts.items():
+        print(f"    {method}: wins {count}/{len(clusters_tested)} clusters")
+
+    results["winner_counts"] = winner_counts
+
+    with open(res_dir / "pathway_specificity.json", "w") as f:
+        json.dump(results, f, indent=2, default=str)
+
+    # Figure
+    fig, axes = plt.subplots(1, 2, figsize=(12, 5))
+
+    # Panel 1: stacked bar of generic vs tissue-specific vs other
+    methods = ["scPTR_gamma", "raw_u_s_ratio", "unspliced_only"]
+    method_labels = ["scPTR\ngamma", "Raw u/s\nratio", "Unspliced\nonly"]
+    x = np.arange(len(methods))
+
+    for di, dataset in enumerate(["pancreas", "dentate_gyrus"]):
+        offset = di * 0.35 - 0.175
+        generics = []
+        tissues = []
+        others = []
+        for m in methods:
+            key = f"{dataset}_{m}"
+            if key in results:
+                r = results[key]
+                generics.append(r["n_generic"])
+                tissues.append(r["n_tissue_specific"])
+                others.append(r["n_non_generic"] - r["n_tissue_specific"])
+            else:
+                generics.append(0)
+                tissues.append(0)
+                others.append(0)
+
+        color_generic = "lightgray" if di == 0 else "silver"
+        color_tissue = "steelblue" if di == 0 else "darkorange"
+        color_other = "lightblue" if di == 0 else "moccasin"
+
+        axes[0].bar(x + offset, tissues, 0.3, label=f"{dataset} tissue-specific",
+                    color=color_tissue, edgecolor="black", linewidth=0.3)
+        axes[0].bar(x + offset, others, 0.3, bottom=tissues,
+                    label=f"{dataset} other", color=color_other,
+                    edgecolor="black", linewidth=0.3)
+        axes[0].bar(x + offset, generics, 0.3,
+                    bottom=[t + o for t, o in zip(tissues, others)],
+                    label=f"{dataset} generic", color=color_generic,
+                    edgecolor="black", linewidth=0.3)
+
+    axes[0].set_xticks(x)
+    axes[0].set_xticklabels(method_labels)
+    axes[0].set_ylabel("Number of top pathways")
+    axes[0].set_title("Pathway Composition by Method")
+    axes[0].legend(fontsize=6, ncol=2)
+
+    # Panel 2: per-cluster winner counts
+    cats = list(winner_counts.keys())
+    vals = [winner_counts[c] for c in cats]
+    colors_bar = ["steelblue", "orange", "lightblue", "gray"]
+    axes[1].bar(cats, vals, color=colors_bar, edgecolor="black", linewidth=0.5)
+    axes[1].set_ylabel("Number of clusters won")
+    axes[1].set_title("Per-Cluster: Most Expected Pathways")
+    for i, v in enumerate(vals):
+        axes[1].text(i, v + 0.2, str(v), ha="center", fontsize=10, fontweight="bold")
+
+    fig.suptitle("Pathway Specificity Analysis", fontsize=13, y=1.02)
+    fig.tight_layout()
+    save_fig(fig, "pathway_specificity")
+
+    return results
+
+
+# =========================================================================
+# 3. Per-cell sci-fate stratified by expression level
+# =========================================================================
+def percell_stratified():
+    """Show that scPTR's advantage over raw u/s increases for
+    low-expression genes, where smoothing matters most."""
+    print("\n" + "=" * 60)
+    print("3. PER-CELL SCI-FATE STRATIFIED BY EXPRESSION LEVEL")
+    print("=" * 60)
+
+    res_dir = OUTPUT_DIR / "results"
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    from run_scifate import load_scifate_data, prepare_for_scptr
+    import scptr
+
+    adata_raw = load_scifate_data()
+    adata = prepare_for_scptr(adata_raw)
+
+    scptr.pp.filter_genes(adata)
+    scptr.pp.normalize_layers(adata)
+    scptr.pp.neighbors(adata, n_neighbors=30)
+    scptr.pp.smooth_layers(adata)
+    scptr.tl.estimate_beta(adata)
+    scptr.tl.estimate_gamma(adata)
+    print(f"  Pipeline complete: {adata.shape}")
+
+    gamma = adata.layers["gamma"]
+    u_layer = adata.layers.get("Mu", adata.layers.get("unspliced"))
+    s_layer = adata.layers.get("Ms", adata.layers.get("spliced"))
+    u = u_layer.toarray() if hasattr(u_layer, 'toarray') else np.asarray(u_layer)
+    s = s_layer.toarray() if hasattr(s_layer, 'toarray') else np.asarray(s_layer)
+
+    raw_ratio = np.zeros_like(gamma)
+    s_safe = np.where(s > 0.01, s, 1.0)
+    raw_ratio = u / s_safe
+    raw_ratio[s < 0.01] = 0
+
+    # Ground truth per cell
+    total_raw = np.asarray(adata_raw.X.toarray() if hasattr(adata_raw.X, 'toarray') else adata_raw.X)
+    new_raw = np.asarray(adata_raw.layers["new"].toarray() if hasattr(adata_raw.layers["new"], 'toarray') else adata_raw.layers["new"])
+    old_raw = total_raw - new_raw
+
+    raw_gene_map = {g: i for i, g in enumerate(adata_raw.var_names)}
+    filtered_in_raw = [raw_gene_map[g] for g in adata.var_names if g in raw_gene_map]
+    genes_in_both = [g for g in adata.var_names if g in raw_gene_map]
+    gene_idx_in_filtered = [list(adata.var_names).index(g) for g in genes_in_both]
+
+    gt_new = new_raw[:, filtered_in_raw]
+    gt_old = old_raw[:, filtered_in_raw]
+    gt_total = total_raw[:, filtered_in_raw]
+    gt_ratio = np.zeros_like(gt_new, dtype=float)
+    valid_gt = gt_old > 0.1
+    gt_ratio[valid_gt] = gt_new[valid_gt] / gt_old[valid_gt]
+    gt_ratio[~valid_gt] = np.nan
+
+    gamma_matched = gamma[:, gene_idx_in_filtered]
+    raw_matched = raw_ratio[:, gene_idx_in_filtered]
+
+    # Stratify genes by expression level (mean total counts)
+    gene_mean_expr = gt_total.mean(axis=0)
+    terciles = np.percentile(gene_mean_expr[gene_mean_expr > 0], [33, 67])
+
+    strata = {
+        "low": gene_mean_expr <= terciles[0],
+        "medium": (gene_mean_expr > terciles[0]) & (gene_mean_expr <= terciles[1]),
+        "high": gene_mean_expr > terciles[1],
+    }
+
+    n_cells = adata.n_obs
+    results = {}
+
+    for stratum_name, gene_mask in strata.items():
+        n_genes_stratum = gene_mask.sum()
+        print(f"\n  --- {stratum_name} expression ({n_genes_stratum} genes) ---")
+
+        gamma_corrs = []
+        raw_corrs = []
+
+        for i in range(n_cells):
+            gt_i = gt_ratio[i, gene_mask]
+            gamma_i = gamma_matched[i, gene_mask]
+            raw_i = raw_matched[i, gene_mask]
+
+            valid = np.isfinite(gt_i) & (gt_i > 0) & (gamma_i > 0) & (raw_i > 0)
+            if valid.sum() >= 10:
+                r_g, _ = stats.spearmanr(gamma_i[valid], gt_i[valid])
+                r_r, _ = stats.spearmanr(raw_i[valid], gt_i[valid])
+                gamma_corrs.append(r_g)
+                raw_corrs.append(r_r)
+
+        gamma_corrs = np.array(gamma_corrs)
+        raw_corrs = np.array(raw_corrs)
+
+        # Filter out NaN correlations (from constant inputs)
+        finite_mask = np.isfinite(gamma_corrs) & np.isfinite(raw_corrs)
+        gamma_corrs = gamma_corrs[finite_mask]
+        raw_corrs = raw_corrs[finite_mask]
+
+        if len(gamma_corrs) < 10:
+            print(f"    Skipped: only {len(gamma_corrs)} valid cells after NaN filtering")
+            continue
+
+        mean_g = np.mean(gamma_corrs)
+        mean_r = np.mean(raw_corrs)
+        gamma_wins = (gamma_corrs > raw_corrs).sum()
+        n_valid = len(gamma_corrs)
+        w_stat, w_p = stats.wilcoxon(gamma_corrs, raw_corrs, alternative="greater")
+
+        print(f"    scPTR gamma: mean r = {mean_g:.4f}")
+        print(f"    Raw u/s:     mean r = {mean_r:.4f}")
+        print(f"    Advantage:   {mean_g - mean_r:.4f}")
+        print(f"    gamma wins:  {gamma_wins}/{n_valid} ({100*gamma_wins/n_valid:.1f}%)")
+        print(f"    Wilcoxon p:  {w_p:.2e}")
+
+        results[stratum_name] = {
+            "n_genes": int(n_genes_stratum),
+            "n_valid_cells": int(n_valid),
+            "mean_gamma_corr": float(mean_g),
+            "mean_raw_corr": float(mean_r),
+            "advantage": float(mean_g - mean_r),
+            "gamma_wins_frac": float(gamma_wins / n_valid),
+            "wilcoxon_p": float(w_p),
+        }
+
+    with open(res_dir / "percell_stratified.json", "w") as f:
+        json.dump(results, f, indent=2)
+
+    # Figure: advantage by expression stratum
+    fig, axes = plt.subplots(1, 2, figsize=(11, 5))
+
+    strata_order = ["low", "medium", "high"]
+    strata_labels = ["Low\nexpr", "Medium\nexpr", "High\nexpr"]
+
+    # Panel 1: mean correlation per stratum
+    gamma_means = [results.get(s, {}).get("mean_gamma_corr", 0) for s in strata_order]
+    raw_means = [results.get(s, {}).get("mean_raw_corr", 0) for s in strata_order]
+    x = np.arange(len(strata_order))
+    axes[0].bar(x - 0.15, gamma_means, 0.3, label="scPTR gamma",
+                color="steelblue", edgecolor="black", linewidth=0.5)
+    axes[0].bar(x + 0.15, raw_means, 0.3, label="Raw u/s",
+                color="salmon", edgecolor="black", linewidth=0.5)
+    axes[0].set_xticks(x)
+    axes[0].set_xticklabels(strata_labels)
+    axes[0].set_ylabel("Mean per-cell Spearman r")
+    axes[0].set_title("Per-Cell Correlation by Expression Level")
+    axes[0].legend()
+
+    # Panel 2: advantage (gamma - raw) by stratum
+    advantages = [results.get(s, {}).get("advantage", 0) for s in strata_order]
+    p_values = [results.get(s, {}).get("wilcoxon_p", 1) for s in strata_order]
+    colors = ["steelblue" if a > 0 else "salmon" for a in advantages]
+    axes[1].bar(strata_labels, advantages, color=colors, edgecolor="black", linewidth=0.5)
+    axes[1].set_ylabel("scPTR advantage (gamma r - raw r)")
+    axes[1].set_title("scPTR Advantage by Expression Level")
+    axes[1].axhline(0, color="gray", linestyle="--", alpha=0.3)
+    for i, (a, p) in enumerate(zip(advantages, p_values)):
+        sig = "***" if p < 0.001 else "**" if p < 0.01 else "*" if p < 0.05 else "ns"
+        axes[1].text(i, a + 0.001 if a > 0 else a - 0.002,
+                    f"{a:.4f}\n({sig})", ha="center", fontsize=9)
+
+    fig.suptitle("scPTR Advantage Stratified by Gene Expression Level",
+                fontsize=13, y=1.02)
+    fig.tight_layout()
+    save_fig(fig, "percell_stratified")
+
+    return results
+
+
+# =========================================================================
+# MAIN
+# =========================================================================
+def main():
+    set_figure_style()
+    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+    (OUTPUT_DIR / "results").mkdir(parents=True, exist_ok=True)
+    (OUTPUT_DIR / "figures").mkdir(parents=True, exist_ok=True)
+
+    all_results = {}
+
+    # 1. Fix Fisher's exact
+    all_results["hub_fisher"] = fix_hub_fisher()
+
+    # 2. Pathway specificity
+    all_results["pathway_specificity"] = pathway_specificity()
+
+    # 3. Per-cell stratified
+    all_results["percell_stratified"] = percell_stratified()
+
+    with open(OUTPUT_DIR / "results" / "all_final_fixes.json", "w") as f:
+        json.dump(all_results, f, indent=2, default=str)
+
+    print("\n" + "=" * 60)
+    print("ALL FINAL FIXES COMPLETE")
+    print("=" * 60)
+    print(f"Results saved to: {OUTPUT_DIR.resolve()}")
+
+
+if __name__ == "__main__":
+    main()

analyses/run_gaps.py

@@ -0,0 +1,550 @@
+#!/usr/bin/env python
+"""Fill research plan gaps: expression-invisible states, RNA velocity comparison,
+and network inference on real data.
+
+Gap 1 (Aim 2): Formally demonstrate expression-invisible PT states
+Gap 2 (Aim 3): Compare PT velocity with scvelo RNA velocity
+Gap 3 (Aim 4): Run RBP network inference on real data
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+import scanpy as sc
+from scipy import stats
+
+sys.path.insert(0, str(Path(__file__).parent))
+from _common import set_figure_style
+
+import scptr
+
+OUTPUT_DIR = Path(__file__).parent.parent / "output" / "gap_analysis"
+
+
+def save_fig(fig, name, subdir="figures"):
+    if fig is None:
+        print(f"  [WARNING] {name}: None, skipping")
+        return
+    out_dir = OUTPUT_DIR / subdir
+    out_dir.mkdir(parents=True, exist_ok=True)
+    path = out_dir / f"{name}.png"
+    fig.savefig(path, dpi=150, bbox_inches="tight")
+    plt.close(fig)
+    print(f"  Saved: {path}")
+
+
+def process_dataset(name):
+    """Load and run full preprocessing + core analysis on a dataset."""
+    print(f"\nLoading {name}...")
+    if name == "pancreas":
+        adata = scptr.datasets.pancreas()
+    else:
+        adata = scptr.datasets.dentate_gyrus()
+
+    scptr.pp.filter_genes(adata)
+    scptr.pp.normalize_layers(adata)
+    scptr.pp.neighbors(adata, n_neighbors=30)
+    scptr.pp.smooth_layers(adata)
+    scptr.tl.estimate_beta(adata)
+    scptr.tl.estimate_gamma(adata)
+    scptr.tl.variance_decomposition(adata)
+    scptr.tl.pt_states(adata)
+    scptr.tl.pt_velocity(adata)
+    print(f"  {name}: {adata.n_obs} cells, {adata.n_vars} genes, "
+          f"{adata.obs['pt_state'].nunique()} PT states")
+    return adata
+
+
+# =========================================================================
+# GAP 1: Expression-invisible PT states (Aim 2 central claim)
+# =========================================================================
+def run_invisible_states(adata, dataset_name):
+    """Formally demonstrate that gamma clustering reveals sub-populations
+    invisible to expression-based clustering.
+
+    Method:
+    1. For each expression cluster, extract cells
+    2. Re-cluster using gamma profiles (sub-clustering)
+    3. Test significance via silhouette score and ANOVA on gamma PCs
+    4. Characterize differentially stabilized genes in sub-clusters
+    """
+    print("\n" + "=" * 60)
+    print(f"GAP 1: EXPRESSION-INVISIBLE STATES ({dataset_name})")
+    print("=" * 60)
+
+    res_dir = OUTPUT_DIR / "results" / "invisible_states" / dataset_name
+    res_dir.mkdir(parents=True, exist_ok=True)
+    fig_prefix = f"invisible_states/{dataset_name}"
+
+    gamma = scptr.tools._gamma  # just for access to layer
+    gamma_mat = adata.layers["gamma"]
+    clusters = adata.obs["clusters"].astype(str)
+
+    results = []
+
+    for cluster_name in sorted(clusters.unique()):
+        mask = (clusters == cluster_name).values
+        n_cells = mask.sum()
+
+        if n_cells < 50:  # need enough cells for sub-clustering
+            print(f"  {cluster_name}: {n_cells} cells (too few, skipping)")
+            continue
+
+        # Extract gamma for this cluster
+        gamma_sub = gamma_mat[mask]
+
+        # PCA on gamma within this cluster
+        from sklearn.decomposition import PCA
+        from sklearn.cluster import KMeans
+        from sklearn.metrics import silhouette_score
+
+        n_pcs = min(15, n_cells - 1, gamma_sub.shape[1] - 1)
+        pca = PCA(n_components=n_pcs, random_state=42)
+        gamma_pcs = pca.fit_transform(gamma_sub)
+
+        # Try 2-4 sub-clusters, pick best silhouette
+        best_k = 1
+        best_sil = -1
+        best_labels = np.zeros(n_cells, dtype=int)
+
+        for k in [2, 3]:
+            if n_cells < k * 10:
+                continue
+            km = KMeans(n_clusters=k, random_state=42, n_init=10)
+            labels = km.fit_predict(gamma_pcs)
+            # Only evaluate if all clusters have >= 10 cells
+            min_size = min(np.bincount(labels))
+            if min_size < 10:
+                continue
+            sil = silhouette_score(gamma_pcs, labels)
+            if sil > best_sil:
+                best_sil = sil
+                best_k = k
+                best_labels = labels
+
+        # Statistical test: MANOVA-like test using gamma PCs
+        # Use ANOVA on first few PCs as a proxy
+        if best_k > 1:
+            p_values_pcs = []
+            for pc in range(min(5, n_pcs)):
+                groups = [gamma_pcs[best_labels == j, pc] for j in range(best_k)]
+                if all(len(g) >= 2 for g in groups):
+                    f_stat, p_val = stats.f_oneway(*groups)
+                    p_values_pcs.append(p_val)
+            # Combine p-values (Fisher's method)
+            if p_values_pcs:
+                # Clamp p-values to avoid log(0)
+                p_clamped = [max(p, 1e-300) for p in p_values_pcs]
+                combined_stat = -2 * sum(np.log(p) for p in p_clamped)
+                from scipy.stats import chi2
+                combined_p = 1 - chi2.cdf(combined_stat, 2 * len(p_clamped))
+            else:
+                combined_p = 1.0
+        else:
+            combined_p = 1.0
+
+        # Now test if these sub-clusters are visible in expression space
+        # Use expression PCA and compute silhouette for the SAME labels
+        expr_sub = adata.X[mask] if not hasattr(adata.X, 'toarray') else adata.X[mask].toarray()
+        n_expr_pcs = min(15, n_cells - 1, expr_sub.shape[1] - 1)
+        pca_expr = PCA(n_components=n_expr_pcs, random_state=42)
+        expr_pcs = pca_expr.fit_transform(expr_sub)
+
+        if best_k > 1:
+            sil_gamma = best_sil
+            sil_expr = silhouette_score(expr_pcs, best_labels)
+        else:
+            sil_gamma = 0
+            sil_expr = 0
+
+        # Find differentially degraded genes between sub-clusters
+        top_genes = []
+        if best_k > 1:
+            median_gamma_by_sub = np.zeros((best_k, gamma_sub.shape[1]))
+            for j in range(best_k):
+                median_gamma_by_sub[j] = np.median(gamma_sub[best_labels == j], axis=0)
+            # Max fold change across sub-clusters
+            max_gamma = np.max(median_gamma_by_sub, axis=0)
+            min_gamma = np.minimum(np.min(median_gamma_by_sub, axis=0), 1e-6)
+            fold_change = max_gamma / np.clip(min_gamma, 1e-6, None)
+            # Filter to genes with nonzero gamma
+            nonzero_mask = max_gamma > 0.01
+            if nonzero_mask.sum() > 0:
+                fc_masked = fold_change.copy()
+                fc_masked[~nonzero_mask] = 0
+                top_idx = np.argsort(fc_masked)[::-1][:20]
+                top_genes = [adata.var_names[i] for i in top_idx if fc_masked[i] > 1.5]
+
+        result = {
+            "cluster": cluster_name,
+            "n_cells": int(n_cells),
+            "n_subclusters": int(best_k),
+            "silhouette_gamma": float(sil_gamma),
+            "silhouette_expr": float(sil_expr),
+            "invisibility_score": float(sil_gamma - sil_expr),
+            "combined_p": float(combined_p),
+            "top_diff_genes": top_genes[:10],
+        }
+        results.append(result)
+
+        status = "INVISIBLE" if sil_gamma > 0.1 and sil_expr < 0.1 else \
+                 "PARTIALLY" if sil_gamma > sil_expr + 0.05 else "VISIBLE"
+        print(f"  {cluster_name}: {n_cells} cells, k={best_k}, "
+              f"sil_gamma={sil_gamma:.3f}, sil_expr={sil_expr:.3f}, "
+              f"p={combined_p:.2e} [{status}]")
+
+    # Save results
+    results_df = pd.DataFrame(results)
+    results_df.to_csv(res_dir / "invisible_states.csv", index=False)
+
+    # Summary figure: silhouette in gamma vs expression space
+    if len(results_df) > 0:
+        fig, axes = plt.subplots(1, 2, figsize=(13, 5))
+
+        # Left: paired bar chart
+        x = np.arange(len(results_df))
+        width = 0.35
+        axes[0].bar(x - width/2, results_df["silhouette_gamma"], width,
+                     label="Gamma space", color="steelblue")
+        axes[0].bar(x + width/2, results_df["silhouette_expr"], width,
+                     label="Expression space", color="salmon")
+        axes[0].set_xticks(x)
+        axes[0].set_xticklabels(results_df["cluster"], rotation=45, ha="right")
+        axes[0].set_ylabel("Silhouette score")
+        axes[0].set_title("Sub-cluster separation: Gamma vs Expression")
+        axes[0].legend()
+        axes[0].axhline(0, color="gray", linestyle="--", alpha=0.3)
+
+        # Right: invisibility score
+        colors = ["steelblue" if v > 0.05 else "gray"
+                  for v in results_df["invisibility_score"]]
+        axes[1].barh(results_df["cluster"], results_df["invisibility_score"],
+                      color=colors)
+        axes[1].set_xlabel("Invisibility score (sil_gamma - sil_expr)")
+        axes[1].set_title("Expression-invisible PT sub-states")
+        axes[1].axvline(0, color="gray", linestyle="--", alpha=0.3)
+
+        fig.suptitle(f"Expression-Invisible States: {dataset_name}", fontsize=13, y=1.02)
+        fig.tight_layout()
+        save_fig(fig, f"invisible_states_{dataset_name}", f"figures/invisible_states")
+
+    return results_df
+
+
+# =========================================================================
+# GAP 2: RNA velocity comparison (Aim 3)
+# =========================================================================
+def run_velocity_comparison(adata, dataset_name):
+    """Compare PT velocity with scvelo RNA velocity on the same dataset.
+
+    Shows:
+    1. Side-by-side velocity embeddings
+    2. Correlation of velocity magnitudes
+    3. Angular agreement between velocity fields
+    """
+    print("\n" + "=" * 60)
+    print(f"GAP 2: RNA VELOCITY COMPARISON ({dataset_name})")
+    print("=" * 60)
+
+    res_dir = OUTPUT_DIR / "results" / "velocity_comparison" / dataset_name
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    import scvelo as scv
+
+    # Run scvelo RNA velocity
+    print("  Running scvelo RNA velocity...")
+    # scvelo needs its own preprocessing
+    adata_scv = adata.copy()
+
+    # scvelo pipeline
+    scv.pp.filter_and_normalize(adata_scv, min_shared_counts=20, n_top_genes=2000)
+    scv.pp.moments(adata_scv, n_pcs=30, n_neighbors=30)
+    scv.tl.velocity(adata_scv)
+
+    # Project scvelo velocity onto the gamma UMAP for fair comparison
+    # Use the gamma UMAP coordinates from scPTR
+    if "X_gamma_umap" in adata.obsm:
+        adata_scv.obsm["X_gamma_umap"] = adata.obsm["X_gamma_umap"]
+
+    # Compute UMAP for scvelo data
+    sc.tl.umap(adata_scv)
+
+    # Get velocity vectors
+    scv_velocity = adata_scv.layers.get("velocity")
+    pt_velocity = adata.layers.get("pt_velocity")
+
+    if scv_velocity is None:
+        print("  [WARNING] scvelo velocity not computed, skipping comparison")
+        return
+
+    print(f"  scvelo velocity shape: {scv_velocity.shape}")
+    print(f"  PT velocity shape: {pt_velocity.shape}")
+
+    # Find shared genes
+    shared_genes = adata.var_names.intersection(adata_scv.var_names)
+    print(f"  Shared genes: {len(shared_genes)}")
+
+    # Compare velocity magnitudes per cell
+    # Use scvelo's gene set for fair comparison
+    scv_genes = adata_scv.var_names
+    scv_gene_idx_in_adata = [list(adata.var_names).index(g)
+                              for g in scv_genes if g in adata.var_names]
+    pt_vel_shared = pt_velocity[:, scv_gene_idx_in_adata]
+    scv_vel_shared_genes = [g for g in scv_genes if g in adata.var_names]
+    scv_vel_idx = [list(adata_scv.var_names).index(g) for g in scv_vel_shared_genes]
+    scv_vel_shared = scv_velocity[:, scv_vel_idx]
+
+    # Handle NaN in scvelo
+    scv_vel_shared = np.nan_to_num(scv_vel_shared, 0)
+
+    # Per-cell velocity magnitude
+    pt_mag = np.linalg.norm(pt_vel_shared, axis=1)
+    scv_mag = np.linalg.norm(scv_vel_shared, axis=1)
+
+    # Cosine similarity per cell
+    dot_product = np.sum(pt_vel_shared * scv_vel_shared, axis=1)
+    norms = pt_mag * scv_mag
+    norms = np.clip(norms, 1e-10, None)
+    cosine_sim = dot_product / norms
+
+    # Filter to cells with nonzero velocity in both
+    valid = (pt_mag > 1e-6) & (scv_mag > 1e-6)
+    print(f"  Cells with nonzero velocity in both: {valid.sum()}/{len(valid)}")
+
+    if valid.sum() > 10:
+        mag_corr, mag_p = stats.spearmanr(pt_mag[valid], scv_mag[valid])
+        mean_cosine = np.mean(cosine_sim[valid])
+        print(f"  Magnitude Spearman r = {mag_corr:.4f} (p={mag_p:.2e})")
+        print(f"  Mean cosine similarity = {mean_cosine:.4f}")
+    else:
+        mag_corr = np.nan
+        mean_cosine = np.nan
+
+    # Save results
+    results = {
+        "n_shared_genes": len(scv_vel_shared_genes),
+        "n_cells_both_nonzero": int(valid.sum()),
+        "magnitude_spearman_r": float(mag_corr) if not np.isnan(mag_corr) else None,
+        "mean_cosine_similarity": float(mean_cosine) if not np.isnan(mean_cosine) else None,
+    }
+    with open(res_dir / "velocity_comparison.json", "w") as f:
+        json.dump(results, f, indent=2)
+
+    # Figure: 2x2 panel
+    fig, axes = plt.subplots(2, 2, figsize=(12, 10))
+
+    # Top-left: scvelo velocity on scvelo UMAP
+    coords_scv = adata_scv.obsm.get("X_umap")
+    if coords_scv is not None:
+        axes[0, 0].scatter(coords_scv[:, 0], coords_scv[:, 1],
+                           c=scv_mag, cmap="YlOrRd", s=3, alpha=0.5,
+                           vmax=np.percentile(scv_mag, 95))
+        axes[0, 0].set_title("RNA Velocity magnitude (scvelo UMAP)")
+        axes[0, 0].set_xlabel("UMAP 1")
+        axes[0, 0].set_ylabel("UMAP 2")
+
+    # Top-right: PT velocity on gamma UMAP
+    coords_gamma = adata.obsm.get("X_gamma_umap")
+    if coords_gamma is not None:
+        axes[0, 1].scatter(coords_gamma[:, 0], coords_gamma[:, 1],
+                           c=pt_mag, cmap="YlOrRd", s=3, alpha=0.5,
+                           vmax=np.percentile(pt_mag, 95))
+        axes[0, 1].set_title("PT Velocity magnitude (gamma UMAP)")
+        axes[0, 1].set_xlabel("UMAP 1")
+        axes[0, 1].set_ylabel("UMAP 2")
+
+    # Bottom-left: magnitude correlation
+    if valid.sum() > 10:
+        axes[1, 0].scatter(scv_mag[valid], pt_mag[valid], alpha=0.1, s=3, c="steelblue")
+        axes[1, 0].set_xlabel("RNA velocity magnitude")
+        axes[1, 0].set_ylabel("PT velocity magnitude")
+        axes[1, 0].set_title(f"Magnitude correlation (r={mag_corr:.3f})")
+
+    # Bottom-right: cosine similarity distribution
+    if valid.sum() > 10:
+        axes[1, 1].hist(cosine_sim[valid], bins=50, color="steelblue",
+                        alpha=0.8, edgecolor="white")
+        axes[1, 1].axvline(mean_cosine, color="red", linestyle="--",
+                           label=f"Mean={mean_cosine:.3f}")
+        axes[1, 1].set_xlabel("Cosine similarity (PT vel vs RNA vel)")
+        axes[1, 1].set_ylabel("Number of cells")
+        axes[1, 1].set_title("Directional agreement")
+        axes[1, 1].legend()
+
+    fig.suptitle(f"PT Velocity vs RNA Velocity: {dataset_name}", fontsize=13, y=1.02)
+    fig.tight_layout()
+    save_fig(fig, f"velocity_comparison_{dataset_name}", "figures/velocity_comparison")
+
+    return results
+
+
+# =========================================================================
+# GAP 3: Network inference on real data (Aim 4)
+# =========================================================================
+def run_network_inference(adata, dataset_name):
+    """Run RBP-target network inference on real data.
+
+    Identifies RBPs whose expression correlates with target gene gamma shifts.
+    """
+    print("\n" + "=" * 60)
+    print(f"GAP 3: NETWORK INFERENCE ({dataset_name})")
+    print("=" * 60)
+
+    res_dir = OUTPUT_DIR / "results" / "network" / dataset_name
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    # Get known RBPs that are expressed in this dataset
+    known_rbps = scptr.tl.list_known_rbps(organism="mouse")
+    rbp_genes = [g for g in known_rbps if g in adata.var_names]
+    print(f"  Known RBPs in dataset: {len(rbp_genes)}/{len(known_rbps)}")
+
+    if len(rbp_genes) < 5:
+        print("  Too few RBPs, skipping network inference")
+        return
+
+    # Get top differentially degraded genes as targets
+    gamma = adata.layers["gamma"]
+    gamma_var = np.var(gamma, axis=0)
+    # Use top 500 most variable gamma genes as targets
+    top_targets_idx = np.argsort(gamma_var)[::-1][:500]
+    target_genes = [adata.var_names[i] for i in top_targets_idx
+                    if gamma_var[i] > 0 and adata.var_names[i] not in rbp_genes]
+    target_genes = target_genes[:200]
+    print(f"  Target genes (top variable gamma): {len(target_genes)}")
+
+    # For each cell type, compute correlation between RBP expression and
+    # target gene gamma
+    clusters = adata.obs["clusters"].astype(str)
+    all_edges = []
+
+    for cluster_name in sorted(clusters.unique()):
+        mask = (clusters == cluster_name).values
+        n_cells = mask.sum()
+        if n_cells < 30:
+            continue
+
+        # Get expression of RBPs in this cluster
+        rbp_idx = [list(adata.var_names).index(g) for g in rbp_genes]
+        if hasattr(adata.X, 'toarray'):
+            rbp_expr = adata.X[mask][:, rbp_idx].toarray()
+        else:
+            rbp_expr = adata.X[mask][:, rbp_idx]
+
+        # Get gamma of target genes
+        target_idx = [list(adata.var_names).index(g) for g in target_genes]
+        target_gamma = gamma[mask][:, target_idx]
+
+        # Correlation: RBP expression vs target gamma
+        for ri, rbp in enumerate(rbp_genes):
+            rbp_x = rbp_expr[:, ri]
+            if np.std(rbp_x) < 1e-6:
+                continue
+
+            for ti, target in enumerate(target_genes):
+                target_g = target_gamma[:, ti]
+                if np.std(target_g) < 1e-6:
+                    continue
+
+                r, p = stats.spearmanr(rbp_x, target_g)
+                if abs(r) > 0.2 and p < 0.01:
+                    all_edges.append({
+                        "cluster": cluster_name,
+                        "rbp": rbp,
+                        "target": target,
+                        "spearman_r": float(r),
+                        "p_value": float(p),
+                        "direction": "stabilizing" if r < 0 else "destabilizing",
+                    })
+
+    edges_df = pd.DataFrame(all_edges)
+    if len(edges_df) > 0:
+        # Multiple testing correction (Benjamini-Hochberg)
+        from statsmodels.stats.multitest import multipletests
+        _, edges_df["fdr"], _, _ = multipletests(edges_df["p_value"], method="fdr_bh")
+        edges_df = edges_df[edges_df["fdr"] < 0.05].copy()
+
+    edges_df.to_csv(res_dir / "network_edges.csv", index=False)
+    print(f"  Significant edges (FDR<0.05): {len(edges_df)}")
+
+    if len(edges_df) > 0:
+        # Top RBP hubs
+        hub_counts = edges_df.groupby("rbp").size().sort_values(ascending=False)
+        print(f"\n  Top RBP hubs:")
+        for rbp, count in hub_counts.head(15).items():
+            n_stab = len(edges_df[(edges_df["rbp"] == rbp) & (edges_df["direction"] == "stabilizing")])
+            n_dest = len(edges_df[(edges_df["rbp"] == rbp) & (edges_df["direction"] == "destabilizing")])
+            print(f"    {rbp}: {count} targets ({n_stab} stabilizing, {n_dest} destabilizing)")
+
+        hub_counts.head(30).to_csv(res_dir / "rbp_hub_counts.csv")
+
+        # Network summary figure
+        fig, axes = plt.subplots(1, 2, figsize=(13, 5))
+
+        # Left: top RBP hubs
+        top_hubs = hub_counts.head(20)
+        colors = ["steelblue" if h > hub_counts.median() else "lightblue"
+                  for h in top_hubs.values]
+        axes[0].barh(range(len(top_hubs)), top_hubs.values, color=colors)
+        axes[0].set_yticks(range(len(top_hubs)))
+        axes[0].set_yticklabels(top_hubs.index)
+        axes[0].set_xlabel("Number of target genes")
+        axes[0].set_title("Top RBP Regulators")
+        axes[0].invert_yaxis()
+
+        # Right: effect size distribution
+        axes[1].hist(edges_df["spearman_r"], bins=40, color="steelblue",
+                     alpha=0.8, edgecolor="white")
+        axes[1].axvline(0, color="red", linestyle="--", alpha=0.5)
+        n_stab = (edges_df["direction"] == "stabilizing").sum()
+        n_dest = (edges_df["direction"] == "destabilizing").sum()
+        axes[1].set_xlabel("Spearman correlation (RBP expr vs target gamma)")
+        axes[1].set_ylabel("Number of edges")
+        axes[1].set_title(f"Edge effects: {n_stab} stabilizing, {n_dest} destabilizing")
+
+        fig.suptitle(f"RBP-Target Network: {dataset_name}", fontsize=13, y=1.02)
+        fig.tight_layout()
+        save_fig(fig, f"network_{dataset_name}", "figures/network")
+
+    return edges_df
+
+
+# =========================================================================
+# MAIN
+# =========================================================================
+def main():
+    set_figure_style()
+    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+
+    # Process both datasets
+    panc = process_dataset("pancreas")
+    dg = process_dataset("dentate_gyrus")
+
+    # GAP 1: Expression-invisible states
+    invis_panc = run_invisible_states(panc, "pancreas")
+    invis_dg = run_invisible_states(dg, "dentate_gyrus")
+
+    # GAP 2: RNA velocity comparison
+    vel_panc = run_velocity_comparison(panc, "pancreas")
+    vel_dg = run_velocity_comparison(dg, "dentate_gyrus")
+
+    # GAP 3: Network inference
+    net_panc = run_network_inference(panc, "pancreas")
+    net_dg = run_network_inference(dg, "dentate_gyrus")
+
+    # Summary
+    print("\n" + "=" * 60)
+    print("GAP ANALYSIS COMPLETE")
+    print("=" * 60)
+    print(f"\nAll results saved to: {OUTPUT_DIR.resolve()}")
+
+
+if __name__ == "__main__":
+    main()

analyses/run_halflife_ablation.py

@@ -0,0 +1,216 @@
+#!/usr/bin/env python
+"""Half-life ablation: compare scPTR gamma vs naive methods for biological accuracy.
+
+For each dataset, compute per-gene median values using four methods:
+  1. scPTR gamma (full kinetic model)
+  2. Raw u/s ratio (no beta normalization)
+  3. Unspliced only (raw unspliced counts)
+  4. Expression (spliced counts, negative control)
+
+Then correlate each with published mRNA half-lives. scPTR gamma should produce
+the strongest negative correlation because the kinetic model (beta normalization,
+smoothing, clipping) produces biologically meaningful degradation rates.
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+from scipy import stats
+
+sys.path.insert(0, str(Path(__file__).parent))
+from _common import set_figure_style
+
+import scptr
+
+OUTPUT_DIR = Path(__file__).parent.parent / "output" / "halflife_ablation"
+DATASETS_DIR = Path(__file__).parent.parent / "src" / "scptr" / "datasets" / "data"
+
+
+def save_fig(fig, name, subdir="figures"):
+    out_dir = OUTPUT_DIR / subdir
+    out_dir.mkdir(parents=True, exist_ok=True)
+    path = out_dir / f"{name}.png"
+    fig.savefig(path, dpi=150, bbox_inches="tight")
+    plt.close(fig)
+    print(f"  Saved: {path}")
+
+
+def run_pipeline(adata, name):
+    print(f"\n--- Pipeline: {name} ---")
+    scptr.pp.filter_genes(adata)
+    scptr.pp.normalize_layers(adata)
+    scptr.pp.neighbors(adata, n_neighbors=30)
+    scptr.pp.smooth_layers(adata)
+    scptr.tl.estimate_beta(adata)
+    scptr.tl.estimate_gamma(adata)
+    print(f"  Done: {adata.shape}")
+    return adata
+
+
+def halflife_ablation(adata, name):
+    """Compare half-life correlations across methods."""
+    print(f"\n{'='*60}")
+    print(f"HALF-LIFE ABLATION: {name}")
+    print(f"{'='*60}")
+
+    gamma = adata.layers["gamma"]
+    u_layer = adata.layers.get("Mu", adata.layers.get("unspliced"))
+    s_layer = adata.layers.get("Ms", adata.layers.get("spliced"))
+    u = u_layer.toarray() if hasattr(u_layer, 'toarray') else np.asarray(u_layer)
+    s = s_layer.toarray() if hasattr(s_layer, 'toarray') else np.asarray(s_layer)
+    expr = adata.X.toarray() if hasattr(adata.X, 'toarray') else np.asarray(adata.X)
+
+    # Raw u/s ratio
+    s_safe = np.where(s > 0.01, s, 1.0)
+    raw_ratio = u / s_safe
+    raw_ratio[s < 0.01] = 0
+
+    # Per-gene medians for each method
+    methods = {
+        "scPTR gamma": np.median(gamma, axis=0),
+        "Raw u/s ratio": np.median(raw_ratio, axis=0),
+        "Unspliced only": np.median(u, axis=0),
+        "Expression": np.median(expr, axis=0),
+    }
+
+    # Filter to gamma-informative genes
+    nonzero_frac = (gamma > 0).mean(axis=0)
+    informative = nonzero_frac >= 0.1
+
+    # Load half-life references
+    hl_files = [
+        ("Mouse (Herzog)", DATASETS_DIR / "herzog2017_halflives.csv"),
+        ("Human (Schofield)", DATASETS_DIR / "schofield2018_halflives.csv"),
+    ]
+
+    results = []
+
+    for hl_label, hl_path in hl_files:
+        if not hl_path.exists():
+            continue
+
+        hl_df = pd.read_csv(hl_path)
+        hl_df = hl_df[["gene_symbol", "half_life_hours"]].dropna()
+        hl_dict = dict(zip(hl_df["gene_symbol"].str.upper(), hl_df["half_life_hours"]))
+
+        print(f"\n  Reference: {hl_label}")
+
+        for method_name, medians in methods.items():
+            matched_vals = []
+            matched_hl = []
+            for i, gene in enumerate(adata.var_names):
+                g_upper = gene.upper()
+                if g_upper in hl_dict and informative[i]:
+                    matched_vals.append(medians[i])
+                    matched_hl.append(hl_dict[g_upper])
+
+            if len(matched_vals) < 50:
+                continue
+
+            r, p = stats.spearmanr(matched_vals, matched_hl)
+            print(f"    {method_name:<20s}: r = {r:.4f}  (p = {p:.2e}, n = {len(matched_vals)})")
+
+            results.append({
+                "dataset": name,
+                "reference": hl_label,
+                "method": method_name,
+                "spearman_r": float(r),
+                "p_value": float(p),
+                "n_genes": len(matched_vals),
+            })
+
+    return results
+
+
+def main():
+    set_figure_style()
+    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+    res_dir = OUTPUT_DIR / "results"
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    # Load datasets
+    all_results = []
+
+    print("=" * 60)
+    print("LOADING DATASETS")
+    print("=" * 60)
+
+    adata_pan = scptr.datasets.pancreas()
+    adata_pan = run_pipeline(adata_pan, "pancreas")
+    all_results.extend(halflife_ablation(adata_pan, "pancreas"))
+
+    adata_dg = scptr.datasets.dentate_gyrus()
+    adata_dg = run_pipeline(adata_dg, "dentate_gyrus")
+    all_results.extend(halflife_ablation(adata_dg, "dentate_gyrus"))
+
+    # sci-fate
+    from run_scifate import load_scifate_data, prepare_for_scptr
+    adata_sf_raw = load_scifate_data()
+    adata_sf = prepare_for_scptr(adata_sf_raw)
+    adata_sf = run_pipeline(adata_sf, "scifate")
+    all_results.extend(halflife_ablation(adata_sf, "scifate"))
+
+    # Save results
+    results_df = pd.DataFrame(all_results)
+    results_df.to_csv(res_dir / "halflife_ablation.csv", index=False)
+
+    # Summary
+    print(f"\n{'='*60}")
+    print("SUMMARY")
+    print(f"{'='*60}")
+
+    # Use Human (Schofield) as primary reference
+    human_results = results_df[results_df["reference"] == "Human (Schofield)"]
+    if len(human_results) > 0:
+        pivot = human_results.pivot_table(
+            index="method", columns="dataset", values="spearman_r", aggfunc="first"
+        )
+        print("\n  Spearman r with Human (Schofield) half-lives:")
+        print(pivot.to_string())
+
+    # Figure: grouped bar chart
+    fig, axes = plt.subplots(1, 2, figsize=(14, 6))
+
+    for ax_idx, (hl_label, hl_sub) in enumerate(results_df.groupby("reference")):
+        ax = axes[ax_idx]
+        datasets = hl_sub["dataset"].unique()
+        methods_order = ["scPTR gamma", "Raw u/s ratio", "Unspliced only", "Expression"]
+        colors = ["steelblue", "orange", "lightblue", "gray"]
+        x = np.arange(len(datasets))
+        width = 0.18
+
+        for mi, (method, color) in enumerate(zip(methods_order, colors)):
+            vals = []
+            for ds in datasets:
+                sub = hl_sub[(hl_sub["method"] == method) & (hl_sub["dataset"] == ds)]
+                vals.append(sub["spearman_r"].values[0] if len(sub) > 0 else 0)
+            bars = ax.bar(x + mi * width, vals, width, label=method, color=color,
+                         edgecolor="black", linewidth=0.5)
+            for bi, v in enumerate(vals):
+                ax.text(x[bi] + mi * width, v - 0.02, f"{v:.3f}",
+                       ha="center", va="top", fontsize=7, rotation=90)
+
+        ax.set_xticks(x + 1.5 * width)
+        ax.set_xticklabels(datasets, fontsize=9)
+        ax.set_ylabel("Spearman r with half-life")
+        ax.set_title(f"Half-life Correlation: {hl_label}")
+        ax.legend(fontsize=7, loc="lower left")
+        ax.axhline(y=0, color="black", linewidth=0.5)
+
+    fig.suptitle("Ablation: Which Method Best Predicts mRNA Half-Life?", fontsize=13)
+    fig.tight_layout()
+    save_fig(fig, "halflife_ablation")
+
+    print(f"\nResults saved to: {OUTPUT_DIR.resolve()}")
+
+
+if __name__ == "__main__":
+    main()

analyses/run_perturbation_validation.py

@@ -0,0 +1,447 @@
+#!/usr/bin/env python
+"""RBP perturbation validation: compare scPTR network predictions with
+Replogle 2022 CRISPRi Perturb-seq data (via Harmonizome API).
+
+For each RBP hub identified by scPTR (via Spearman correlation between
+RBP expression and target gamma), we test whether its predicted targets
+are enriched among genes differentially expressed upon RBP knockdown.
+
+This validates the causal direction: if scPTR correctly identifies that RBP X
+regulates gene Y's degradation, then knocking down RBP X should change Y's
+expression level.
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+from scipy import stats
+
+sys.path.insert(0, str(Path(__file__).parent))
+from _common import set_figure_style
+
+import scptr
+
+OUTPUT_DIR = Path(__file__).parent.parent / "output" / "perturbation_validation"
+
+
+def save_fig(fig, name, subdir="figures"):
+    out_dir = OUTPUT_DIR / subdir
+    out_dir.mkdir(parents=True, exist_ok=True)
+    path = out_dir / f"{name}.png"
+    fig.savefig(path, dpi=150, bbox_inches="tight")
+    plt.close(fig)
+    print(f"  Saved: {path}")
+
+
+def load_perturb_seq_de(rbp: str) -> tuple[list[str], list[str]] | None:
+    """Load differentially expressed genes from Replogle 2022 CRISPRi Perturb-seq
+    via Harmonizome API.
+
+    Returns (up_genes, down_genes): genes whose expression increases/decreases
+    when the RBP is knocked down.
+    """
+    import requests
+
+    rbp_ids = {
+        "HNRNPA1": "3857_HNRNPA1_P1P2",
+        "YBX1": "9921_YBX1_P1P2",
+        "ELAVL1": "2583_ELAVL1_P1P2",
+        "SRSF3": "8433_SRSF3_P1P2",
+        "RBFOX2": "7148_RBFOX2_P1",
+        "FUS": "3224_FUS_P1P2",
+        "HNRNPC": "3861_HNRNPC_P1P2",
+        "DDX5": "2134_DDX5_P1P2",
+        "MBNL1": "4881_MBNL1_P1P2",
+    }
+
+    gene_set_id = rbp_ids.get(rbp)
+    if gene_set_id is None:
+        return None
+
+    dataset_name = ("Replogle+et+al.,+Cell,+2022+K562+Genome-wide+"
+                    "Perturb-seq+Gene+Perturbation+Signatures")
+    url = (f"https://maayanlab.cloud/Harmonizome/api/1.0/gene_set/"
+           f"{gene_set_id}/{dataset_name}")
+
+    try:
+        r = requests.get(url, timeout=30)
+        r.raise_for_status()
+        data = r.json()
+    except Exception as e:
+        print(f"    Harmonizome API error for {rbp}: {e}")
+        return None
+
+    associations = data.get("associations", [])
+    if not associations:
+        return None
+
+    up_genes = []
+    down_genes = []
+    for assoc in associations:
+        gene_name = assoc.get("gene", {}).get("symbol", "")
+        value = assoc.get("standardizedValue", 0)
+        if value > 0:
+            up_genes.append(gene_name)
+        else:
+            down_genes.append(gene_name)
+
+    return up_genes, down_genes
+
+
+def infer_spearman_network(adata, rbp_list, n_top_targets=200):
+    """Infer RBP-target network using vectorized Spearman partial correlation
+    (library-size corrected) between RBP expression and target gene gamma.
+
+    Vectorized approach: rank all columns once, residualize against library size
+    ranks using matrix operations, then compute correlations via dot products.
+    """
+
+    gamma = np.array(scptr._utils.get_layer(adata, "gamma"))
+    expression = np.array(scptr._utils.get_layer(adata, "Ms"))
+
+    gene_names = [g.upper() for g in adata.var_names]
+    gene_name_to_idx = {g: i for i, g in enumerate(gene_names)}
+
+    n_cells, n_genes = gamma.shape
+
+    # Library size ranks (once)
+    lib_size = expression.sum(axis=1)
+    lib_rank = stats.rankdata(lib_size)
+    lib_rank_centered = lib_rank - lib_rank.mean()
+    lib_ss = np.dot(lib_rank_centered, lib_rank_centered)
+
+    # Rank all gamma columns (vectorized)
+    gamma_ranks = np.zeros_like(gamma)
+    gamma_valid = np.zeros(n_genes, dtype=bool)
+    for j in range(n_genes):
+        col = gamma[:, j]
+        if np.std(col) < 1e-8:
+            continue
+        gamma_ranks[:, j] = stats.rankdata(col)
+        gamma_valid[j] = True
+
+    # Residualize gamma ranks against library size (vectorized)
+    # slope_j = dot(lib_rank_centered, gamma_rank_j_centered) / dot(lib_rank_centered, lib_rank_centered)
+    gamma_ranks_centered = gamma_ranks - gamma_ranks.mean(axis=0, keepdims=True)
+    slopes_gamma = np.dot(lib_rank_centered, gamma_ranks_centered) / lib_ss
+    gamma_resid = gamma_ranks - np.outer(lib_rank, slopes_gamma)
+    gamma_resid_centered = gamma_resid - gamma_resid.mean(axis=0, keepdims=True)
+    gamma_resid_std = np.sqrt((gamma_resid_centered ** 2).sum(axis=0))
+    gamma_resid_std[gamma_resid_std < 1e-8] = 1.0  # avoid division by zero
+
+    edges = []
+    seen_rbps = set()
+
+    for rbp in rbp_list:
+        rbp_upper = rbp.upper()
+        if rbp_upper in seen_rbps:
+            continue
+        if rbp_upper not in gene_name_to_idx:
+            continue
+        seen_rbps.add(rbp_upper)
+
+        rbp_idx = gene_name_to_idx[rbp_upper]
+        rbp_expr = expression[:, rbp_idx]
+
+        if np.std(rbp_expr) < 1e-8:
+            continue
+
+        # Rank and residualize RBP expression
+        rbp_rank = stats.rankdata(rbp_expr)
+        rbp_rank_centered = rbp_rank - rbp_rank.mean()
+        slope_rbp = np.dot(lib_rank_centered, rbp_rank_centered) / lib_ss
+        rbp_resid = rbp_rank - slope_rbp * lib_rank
+        rbp_resid_centered = rbp_resid - rbp_resid.mean()
+        rbp_resid_std = np.sqrt(np.dot(rbp_resid_centered, rbp_resid_centered))
+
+        if rbp_resid_std < 1e-8:
+            continue
+
+        # Vectorized correlation: r = dot(rbp_resid, gamma_resid) / (std_rbp * std_gamma)
+        r_vals = np.dot(rbp_resid_centered, gamma_resid_centered) / (rbp_resid_std * gamma_resid_std)
+        r_vals = np.clip(r_vals, -1.0, 1.0)
+
+        # Compute p-values from t-distribution
+        df = n_cells - 3  # partial correlation df
+        t_vals = r_vals * np.sqrt(df / (1 - r_vals ** 2 + 1e-12))
+        p_vals = 2 * stats.t.sf(np.abs(t_vals), df)
+
+        # Filter to valid targets (not self, valid gamma)
+        valid_mask = gamma_valid.copy()
+        valid_mask[rbp_idx] = False
+        valid_indices = np.where(valid_mask)[0]
+
+        if len(valid_indices) == 0:
+            continue
+
+        valid_r = r_vals[valid_indices]
+        valid_p = p_vals[valid_indices]
+
+        # Select top N targets by absolute correlation strength
+        abs_r = np.abs(valid_r)
+        top_k = min(n_top_targets, len(abs_r))
+        top_indices = np.argsort(abs_r)[::-1][:top_k]
+
+        for idx_in_valid in top_indices:
+            gene_idx = valid_indices[idx_in_valid]
+            edges.append({
+                "regulator": rbp_upper,
+                "target": gene_names[gene_idx],
+                "weight": float(valid_r[idx_in_valid]),
+                "p_value": float(valid_p[idx_in_valid]),
+                "direction": "destabilizing" if valid_r[idx_in_valid] > 0 else "stabilizing",
+            })
+
+    result = pd.DataFrame(edges)
+    if len(result) > 0:
+        result = result.sort_values("weight", key=abs, ascending=False).reset_index(drop=True)
+
+    return result
+
+
+def validate_rbp_targets(adata, dataset_name, network_df):
+    """For each hub RBP, test enrichment of its predicted targets among
+    perturbation-responsive genes."""
+    print(f"\n{'='*60}")
+    print(f"PERTURBATION VALIDATION: {dataset_name}")
+    print(f"{'='*60}")
+
+    rbp_counts = network_df.groupby("regulator").size().sort_values(ascending=False)
+    top_rbps = rbp_counts.head(15).index.tolist()
+    print(f"  Top RBP hubs: {top_rbps[:10]}")
+
+    results = []
+
+    for rbp in top_rbps:
+        rbp_upper = rbp.upper()
+
+        rbp_edges = network_df[network_df["regulator"] == rbp_upper]
+        predicted_targets = set(rbp_edges["target"].str.upper())
+        predicted_destab = set(
+            rbp_edges[rbp_edges["direction"] == "destabilizing"]["target"].str.upper()
+        )
+        predicted_stab = set(
+            rbp_edges[rbp_edges["direction"] == "stabilizing"]["target"].str.upper()
+        )
+
+        n_targets = len(predicted_targets)
+        if n_targets < 5:
+            continue
+
+        perturb_result = load_perturb_seq_de(rbp_upper)
+        if perturb_result is not None:
+            up_genes, down_genes = perturb_result
+            up_set = set(g.upper() for g in up_genes)
+            down_set = set(g.upper() for g in down_genes)
+
+            all_genes = set(g.upper() for g in adata.var_names)
+
+            # Destabilizing targets should be upregulated upon RBP knockdown
+            if len(predicted_destab) > 0 and len(up_set) > 0:
+                overlap_destab_up = len(predicted_destab & up_set)
+                destab_not_up = len(predicted_destab - up_set)
+                up_not_destab = len(up_set - predicted_destab)
+                neither = len(all_genes - predicted_destab - up_set)
+
+                table = [[overlap_destab_up, destab_not_up],
+                         [up_not_destab, neither]]
+                odds_ratio, fisher_p = stats.fisher_exact(table,
+                                                          alternative="greater")
+
+                print(f"\n  {rbp_upper} (Perturb-seq CRISPRi):")
+                print(f"    Predicted destab targets: {len(predicted_destab)}")
+                print(f"    Genes up upon KD: {len(up_set)}")
+                print(f"    Overlap: {overlap_destab_up}")
+                print(f"    Fisher OR={odds_ratio:.2f}, p={fisher_p:.3e}")
+
+                results.append({
+                    "rbp": rbp_upper,
+                    "dataset": dataset_name,
+                    "validation": "Perturb-seq_CRISPRi",
+                    "n_predicted_targets": n_targets,
+                    "n_predicted_destab": len(predicted_destab),
+                    "n_predicted_stab": len(predicted_stab),
+                    "n_perturbation_up": len(up_set),
+                    "n_perturbation_down": len(down_set),
+                    "overlap_destab_up": overlap_destab_up,
+                    "fisher_or": float(odds_ratio),
+                    "fisher_p": float(fisher_p),
+                })
+
+            # Stabilizing targets should be downregulated upon RBP knockdown
+            if len(predicted_stab) > 0 and len(down_set) > 0:
+                overlap_stab_down = len(predicted_stab & down_set)
+                stab_not_down = len(predicted_stab - down_set)
+                down_not_stab = len(down_set - predicted_stab)
+                neither2 = len(all_genes - predicted_stab - down_set)
+
+                table2 = [[overlap_stab_down, stab_not_down],
+                           [down_not_stab, neither2]]
+                or2, p2 = stats.fisher_exact(table2, alternative="greater")
+
+                print(f"    Stabilizing->down: overlap={overlap_stab_down}, "
+                      f"OR={or2:.2f}, p={p2:.3e}")
+        else:
+            print(f"\n  {rbp_upper}: No Perturb-seq data available")
+
+    return results
+
+
+def run_network_and_validate(adata, dataset_name):
+    """Run scPTR pipeline, infer correlation-based network, validate."""
+    import copy
+    adata = copy.deepcopy(adata)
+    scptr.pp.filter_genes(adata)
+    scptr.pp.normalize_layers(adata)
+    scptr.pp.neighbors(adata, n_neighbors=30)
+    scptr.pp.smooth_layers(adata)
+    scptr.tl.estimate_beta(adata)
+    scptr.tl.estimate_gamma(adata)
+
+    # RBPs to test (those with Perturb-seq data + known RBP hubs)
+    rbp_list = [
+        "HNRNPA1", "YBX1", "ELAVL1", "SRSF3", "RBFOX2",
+        "FUS", "HNRNPC", "DDX5", "MBNL1",
+        # Additional common RBP hubs
+        "HNRNPD", "TRA2B", "ZFP36L1", "RBFOX1", "RBFOX3",
+        "CELF2", "ELAVL3", "MATR3", "MBNL2", "PTBP1",
+        # Mouse gene name variants
+        "Hnrnpa1", "Ybx1", "Elavl1", "Srsf3", "Rbfox2",
+        "Fus", "Hnrnpc", "Ddx5", "Mbnl1", "Hnrnpd",
+        "Tra2b", "Zfp36l1", "Rbfox1", "Rbfox3", "Celf2",
+        "Elavl3", "Matr3", "Mbnl2", "Ptbp1",
+    ]
+
+    print(f"  Inferring correlation network for {len(rbp_list)} candidate RBPs...")
+    net_df = infer_spearman_network(adata, rbp_list, n_top_targets=200)
+
+    if len(net_df) == 0:
+        print(f"  No network edges for {dataset_name}")
+        return []
+
+    print(f"  Network: {len(net_df)} edges, "
+          f"{net_df['regulator'].nunique()} regulators, "
+          f"{net_df['target'].nunique()} targets")
+
+    destab_frac = (net_df["direction"] == "destabilizing").mean()
+    print(f"  Destabilizing fraction: {destab_frac:.1%}")
+
+    return validate_rbp_targets(adata, dataset_name, net_df)
+
+
+def main():
+    set_figure_style()
+    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+
+    print("=" * 60)
+    print("LOADING DATASETS")
+    print("=" * 60)
+
+    adata_pan = scptr.datasets.pancreas()
+    adata_dg = scptr.datasets.dentate_gyrus()
+
+    all_results = []
+
+    print("\n" + "#" * 60)
+    print("# PANCREAS")
+    print("#" * 60)
+    results_pan = run_network_and_validate(adata_pan, "pancreas")
+    all_results.extend(results_pan)
+
+    print("\n" + "#" * 60)
+    print("# DENTATE GYRUS")
+    print("#" * 60)
+    results_dg = run_network_and_validate(adata_dg, "dentate_gyrus")
+    all_results.extend(results_dg)
+
+    # Save results
+    res_dir = OUTPUT_DIR / "results"
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    if all_results:
+        results_df = pd.DataFrame(all_results)
+        results_df.to_csv(res_dir / "perturbation_validation.csv", index=False)
+
+        # FDR correction across all tests
+        from statsmodels.stats.multitest import multipletests
+        _, fdr, _, _ = multipletests(results_df["fisher_p"], method="fdr_bh")
+        results_df["fdr"] = fdr
+
+        # Summary
+        print(f"\n{'='*60}")
+        print("PERTURBATION VALIDATION SUMMARY")
+        print(f"{'='*60}")
+        print(f"  Total tests: {len(results_df)}")
+        print(f"  Significant (p<0.05): {(results_df['fisher_p'] < 0.05).sum()}")
+        print(f"  Significant (FDR<0.10): {(results_df['fdr'] < 0.10).sum()}")
+        print(f"  Mean odds ratio: {results_df['fisher_or'].mean():.2f}")
+        print(f"  Median odds ratio: {results_df['fisher_or'].median():.2f}")
+
+        print(f"\n  Per-RBP results:")
+        for _, row in results_df.sort_values("fisher_p").iterrows():
+            sig = ("***" if row["fisher_p"] < 0.001 else
+                   "**" if row["fisher_p"] < 0.01 else
+                   "*" if row["fisher_p"] < 0.05 else "")
+            print(f"    {row['rbp']:>10s} ({row['dataset']:>12s}): "
+                  f"OR={row['fisher_or']:6.2f}  p={row['fisher_p']:.3e}  "
+                  f"overlap={row['overlap_destab_up']:3d}/{row['n_predicted_destab']:3d} {sig}")
+
+        # Figure
+        fig, axes = plt.subplots(1, 2, figsize=(14, 6))
+
+        rbps = results_df["rbp"].values
+        ors = results_df["fisher_or"].values
+        ps = results_df["fisher_p"].values
+        colors = ["red" if p < 0.05 else "gray" for p in ps]
+
+        y_pos = np.arange(len(rbps))
+        axes[0].barh(y_pos, np.log2(ors + 0.01), color=colors, edgecolor="black",
+                     linewidth=0.5)
+        axes[0].set_yticks(y_pos)
+        axes[0].set_yticklabels([f"{r} ({d[:3]})" for r, d in
+                                 zip(rbps, results_df["dataset"])], fontsize=8)
+        axes[0].axvline(x=0, color="black", linestyle="-", linewidth=0.5)
+        axes[0].set_xlabel("log2(Odds Ratio)")
+        axes[0].set_title("scPTR Target Enrichment in\nPerturb-seq DE Genes")
+
+        axes[1].barh(y_pos, -np.log10(ps), color=colors, edgecolor="black",
+                     linewidth=0.5)
+        axes[1].axvline(x=-np.log10(0.05), color="blue", linestyle="--",
+                        alpha=0.5, label="p=0.05")
+        axes[1].set_yticks(y_pos)
+        axes[1].set_yticklabels([f"{r} ({d[:3]})" for r, d in
+                                 zip(rbps, results_df["dataset"])], fontsize=8)
+        axes[1].set_xlabel("-log10(p)")
+        axes[1].set_title("Significance of Enrichment")
+        axes[1].legend()
+
+        fig.tight_layout()
+        save_fig(fig, "perturbation_validation")
+
+        results_df.to_csv(res_dir / "perturbation_validation.csv", index=False)
+
+        summary = {
+            "n_tests": len(results_df),
+            "n_sig_005": int((results_df["fisher_p"] < 0.05).sum()),
+            "n_sig_fdr_010": int((results_df["fdr"] < 0.10).sum()),
+            "mean_or": float(results_df["fisher_or"].mean()),
+            "median_or": float(results_df["fisher_or"].median()),
+        }
+        with open(res_dir / "perturbation_summary.json", "w") as f:
+            json.dump(summary, f, indent=2)
+    else:
+        print("  No perturbation validation results obtained.")
+
+    print(f"\nResults saved to: {OUTPUT_DIR.resolve()}")
+
+
+if __name__ == "__main__":
+    main()

analyses/run_scifate.py

@@ -0,0 +1,571 @@
+#!/usr/bin/env python
+"""Validate scPTR gamma estimates against sci-fate metabolic labeling ground truth.
+
+sci-fate (Cao et al. 2020, Nature Biotechnology) provides both total and newly
+synthesized mRNA counts per cell via 4sU metabolic labeling. This allows us to
+compute ground-truth degradation rates and compare them against scPTR's gamma
+estimates from splicing kinetics alone.
+
+Key idea:
+- old RNA = total - new (pre-existing mRNA)
+- degradation_rate ~ new / old (high ratio = fast turnover)
+- We expect: genes with high scPTR gamma should have high new/old ratio
+
+Data: A549 cells treated with dexamethasone (0-10h), GEO GSE131351.
+"""
+
+from __future__ import annotations
+
+import gzip
+import json
+import sys
+from io import BytesIO
+from pathlib import Path
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+import scanpy as sc
+from scipy import stats
+from scipy.io import mmread
+from scipy.sparse import csc_matrix
+
+sys.path.insert(0, str(Path(__file__).parent))
+from _common import set_figure_style
+
+import scptr
+
+OUTPUT_DIR = Path(__file__).parent.parent / "output" / "scifate_validation"
+CACHE_DIR = Path.home() / ".cache" / "scptr" / "scifate"
+
+
+def save_fig(fig, name, subdir="figures"):
+    """Save a matplotlib figure to output dir."""
+    if fig is None:
+        print(f"  [WARNING] {name}: plot returned None, skipping save")
+        return
+    out_dir = OUTPUT_DIR / subdir
+    out_dir.mkdir(parents=True, exist_ok=True)
+    path = out_dir / f"{name}.png"
+    fig.savefig(path, dpi=150, bbox_inches="tight")
+    plt.close(fig)
+    print(f"  Saved: {path}")
+
+
+def load_scifate_data():
+    """Load sci-fate data from GEO-downloaded files.
+
+    Returns AnnData with:
+    - X: total gene counts (sparse)
+    - layers['new']: newly synthesized counts (sparse)
+    - obs: cell annotations (treatment_time, etc.)
+    - var: gene annotations (gene_id, gene_short_name)
+    """
+    print("Loading sci-fate data from GEO files...")
+
+    # Load cell annotations
+    cell_ann = pd.read_csv(CACHE_DIR / "GSM3770930_A549_cell_annotate.txt.gz",
+                           compression="gzip")
+    print(f"  Cells: {len(cell_ann)}")
+
+    # Load gene annotations
+    gene_ann = pd.read_csv(CACHE_DIR / "GSM3770930_A549_gene_annotate.txt.gz",
+                           compression="gzip")
+    print(f"  Genes: {len(gene_ann)}")
+
+    # Load total count matrix (MatrixMarket format, gzipped)
+    print("  Loading total count matrix...")
+    with gzip.open(CACHE_DIR / "GSM3770930_A549_gene_count.txt.gz", 'rb') as f:
+        total_mat = mmread(f)  # genes x cells
+    total_mat = csc_matrix(total_mat).T  # -> cells x genes
+
+    # Load newly synthesized count matrix
+    print("  Loading newly synthesized count matrix...")
+    with gzip.open(CACHE_DIR / "GSM3770930_A549_gene_count_newly_synthesised.txt.gz", 'rb') as f:
+        new_mat = mmread(f)  # genes x cells
+    new_mat = csc_matrix(new_mat).T  # -> cells x genes
+
+    print(f"  Total matrix: {total_mat.shape}")
+    print(f"  New matrix: {new_mat.shape}")
+
+    # Build AnnData
+    import anndata as ad
+    adata = ad.AnnData(
+        X=total_mat,
+        obs=cell_ann.set_index("sample"),
+        var=gene_ann.set_index("gene_id"),
+    )
+    adata.layers["new"] = new_mat
+    adata.var_names_make_unique()
+
+    # Use gene short names
+    adata.var["gene_id_full"] = adata.var_names.tolist()
+    adata.var_names = adata.var["gene_short_name"].values
+    adata.var_names_make_unique()
+
+    print(f"  AnnData shape: {adata.shape}")
+    print(f"  Treatment times: {adata.obs['treatment_time'].value_counts().to_dict()}")
+
+    return adata
+
+
+def compute_ground_truth_degradation(adata):
+    """Compute per-gene ground-truth degradation rate from labeled/unlabeled RNA.
+
+    Ground truth: degradation_rate_proxy = mean(new) / mean(old)
+    where old = total - new.
+
+    Genes with high turnover have high new/old ratio.
+    """
+    total = np.asarray(adata.X.toarray() if hasattr(adata.X, 'toarray') else adata.X)
+    new = np.asarray(adata.layers["new"].toarray() if hasattr(adata.layers["new"], 'toarray') else adata.layers["new"])
+    old = total - new
+
+    # Per-gene: mean across cells
+    mean_new = new.mean(axis=0)
+    mean_old = old.mean(axis=0)
+    mean_total = total.mean(axis=0)
+
+    # Degradation rate proxy: new/old ratio (high = fast turnover)
+    # Only for genes with sufficient expression
+    min_expr = 0.5  # minimum mean total expression
+    reliable = (mean_total >= min_expr) & (mean_old > 0.1)
+
+    deg_rate = np.full(adata.n_vars, np.nan)
+    deg_rate[reliable] = mean_new[reliable] / mean_old[reliable]
+
+    # Also compute fraction-new (new/total), another degradation proxy
+    frac_new = np.full(adata.n_vars, np.nan)
+    frac_new[reliable] = mean_new[reliable] / mean_total[reliable]
+
+    result = pd.DataFrame({
+        "gene": adata.var_names,
+        "mean_total": mean_total,
+        "mean_new": mean_new,
+        "mean_old": mean_old,
+        "new_old_ratio": deg_rate,
+        "frac_new": frac_new,
+    })
+    return result
+
+
+def prepare_for_scptr(adata_scifate):
+    """Prepare sci-fate data for scPTR pipeline.
+
+    sci-fate doesn't have unspliced/spliced layers from velocity-style
+    preprocessing. Instead, we use:
+    - spliced = old RNA (pre-existing, ~steady-state pool)
+    - unspliced = new RNA (recently transcribed, proxy for nascent)
+
+    This mapping makes biological sense: newly synthesized RNA is analogous
+    to the unspliced pool (recently produced), while old RNA represents the
+    mature steady-state pool (analogous to spliced).
+    """
+    import anndata as ad
+
+    total = adata_scifate.X.toarray() if hasattr(adata_scifate.X, 'toarray') else np.asarray(adata_scifate.X)
+    new = adata_scifate.layers["new"].toarray() if hasattr(adata_scifate.layers["new"], 'toarray') else np.asarray(adata_scifate.layers["new"])
+    old = total - new
+
+    # Filter to protein-coding genes with sufficient expression
+    mean_total = total.mean(axis=0)
+    keep = mean_total >= 0.5  # min mean expression
+    if "gene_type" in adata_scifate.var.columns:
+        is_pc = adata_scifate.var["gene_type"] == "protein_coding"
+        keep = keep & is_pc.values
+
+    adata = ad.AnnData(
+        X=total[:, keep].astype(np.float32),
+        obs=adata_scifate.obs.copy(),
+        var=adata_scifate.var.iloc[keep].copy(),
+    )
+    # Map: unspliced=new, spliced=old
+    adata.layers["unspliced"] = new[:, keep].astype(np.float32)
+    adata.layers["spliced"] = old[:, keep].astype(np.float32)
+
+    print(f"  Prepared AnnData: {adata.shape}")
+    print(f"  Protein-coding genes with mean expr >= 0.5: {keep.sum()}")
+    return adata
+
+
+def main():
+    set_figure_style()
+    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+    res_dir = OUTPUT_DIR / "results"
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    # =========================================================================
+    # LOAD SCI-FATE DATA
+    # =========================================================================
+    print("=" * 60)
+    print("LOADING SCI-FATE DATA")
+    print("=" * 60)
+    adata_raw = load_scifate_data()
+
+    # =========================================================================
+    # GROUND TRUTH DEGRADATION RATES
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("COMPUTING GROUND TRUTH DEGRADATION RATES")
+    print("=" * 60)
+
+    gt = compute_ground_truth_degradation(adata_raw)
+    n_reliable = gt["new_old_ratio"].notna().sum()
+    print(f"  Reliable genes: {n_reliable} / {len(gt)}")
+    print(f"  New/old ratio: median={gt['new_old_ratio'].median():.4f}, "
+          f"mean={gt['new_old_ratio'].mean():.4f}")
+    print(f"  Frac new: median={gt['frac_new'].median():.4f}")
+
+    gt.to_csv(res_dir / "ground_truth_degradation.csv", index=False)
+
+    # =========================================================================
+    # PER-TIMEPOINT ANALYSIS
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("PER-TIMEPOINT GROUND TRUTH")
+    print("=" * 60)
+
+    timepoints = sorted(adata_raw.obs["treatment_time"].unique())
+    gt_by_time = {}
+    for tp in timepoints:
+        mask = adata_raw.obs["treatment_time"] == tp
+        sub = adata_raw[mask].copy()
+        gt_tp = compute_ground_truth_degradation(sub)
+        gt_by_time[tp] = gt_tp
+        n_rel = gt_tp["new_old_ratio"].notna().sum()
+        med_ratio = gt_tp["new_old_ratio"].median()
+        print(f"  {tp}: {mask.sum()} cells, {n_rel} reliable genes, "
+              f"median new/old ratio = {med_ratio:.4f}")
+
+    # Check consistency across timepoints
+    print("\n--- Cross-timepoint consistency ---")
+    tp_list = list(gt_by_time.keys())
+    for i in range(len(tp_list)):
+        for j in range(i + 1, len(tp_list)):
+            a = gt_by_time[tp_list[i]].set_index("gene")
+            b = gt_by_time[tp_list[j]].set_index("gene")
+            shared = a.index.intersection(b.index)
+            va = a.loc[shared, "new_old_ratio"].values
+            vb = b.loc[shared, "new_old_ratio"].values
+            valid = np.isfinite(va) & np.isfinite(vb)
+            if valid.sum() > 10:
+                r, p = stats.spearmanr(va[valid], vb[valid])
+                print(f"  {tp_list[i]} vs {tp_list[j]}: Spearman r = {r:.4f} (n={valid.sum()})")
+
+    # =========================================================================
+    # RUN SCPTR PIPELINE
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("RUNNING SCPTR PIPELINE ON SCI-FATE DATA")
+    print("=" * 60)
+
+    adata = prepare_for_scptr(adata_raw)
+
+    # Preprocessing
+    scptr.pp.filter_genes(adata)
+    print(f"  After gene filtering: {adata.shape}")
+
+    scptr.pp.normalize_layers(adata)
+    print("  Normalized layers")
+
+    scptr.pp.neighbors(adata, n_neighbors=30)
+    print("  Built kNN graph")
+
+    scptr.pp.smooth_layers(adata)
+    print("  Smoothed layers")
+
+    # Core analysis
+    scptr.tl.estimate_beta(adata)
+    beta = adata.var["beta"].values
+    print(f"  Beta: median={np.median(beta):.4f}, max={np.max(beta):.4f}")
+
+    scptr.tl.estimate_gamma(adata)
+    gamma = adata.layers["gamma"]
+    gamma_med = np.median(gamma, axis=0)
+    print(f"  Gamma: shape={gamma.shape}, median of medians={np.median(gamma_med):.4f}")
+    print(f"  Gamma max: {np.max(gamma):.4f}")
+
+    # =========================================================================
+    # CORRELATION: SCPTR GAMMA vs GROUND TRUTH
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("SCPTR GAMMA vs GROUND TRUTH DEGRADATION RATES")
+    print("=" * 60)
+    print("  NOTE: Since gamma = beta * unspliced/spliced and we map")
+    print("  new→unspliced, old→spliced, the gamma-vs-new/old correlation")
+    print("  is partially tautological. The independent validation is the")
+    print("  correlation with published half-lives (Schofield 2018).")
+
+    # Build gene-level comparison
+    gamma_series = pd.Series(gamma_med, index=adata.var_names)
+    gt_indexed = gt.set_index("gene")
+
+    shared = gamma_series.index.intersection(gt_indexed.index)
+    print(f"  Shared genes: {len(shared)}")
+
+    g = gamma_series[shared].values.astype(float)
+    gt_ratio = gt_indexed.loc[shared, "new_old_ratio"].values.astype(float)
+    gt_frac = gt_indexed.loc[shared, "frac_new"].values.astype(float)
+
+    # Filter: need both values finite and positive
+    valid_ratio = np.isfinite(g) & np.isfinite(gt_ratio) & (g > 0) & (gt_ratio > 0)
+    valid_frac = np.isfinite(g) & np.isfinite(gt_frac) & (g > 0) & (gt_frac > 0)
+
+    results = {}
+
+    # Correlation with new/old ratio
+    if valid_ratio.sum() > 10:
+        g_r = g[valid_ratio]
+        gt_r = gt_ratio[valid_ratio]
+        sp_r, sp_p = stats.spearmanr(g_r, gt_r)
+        pe_r, pe_p = stats.pearsonr(np.log1p(g_r), np.log1p(gt_r))
+        print(f"\n  vs new/old ratio (n={valid_ratio.sum()}):")
+        print(f"    Spearman r = {sp_r:.4f} (p = {sp_p:.2e})")
+        print(f"    Pearson  r = {pe_r:.4f} (p = {pe_p:.2e}) [log-space]")
+        results["new_old_ratio"] = {
+            "spearman_r": float(sp_r), "spearman_p": float(sp_p),
+            "pearson_r": float(pe_r), "pearson_p": float(pe_p),
+            "n_genes": int(valid_ratio.sum()),
+        }
+    else:
+        print("  Not enough shared genes for new/old ratio correlation.")
+        results["new_old_ratio"] = {"n_genes": int(valid_ratio.sum())}
+
+    # Correlation with fraction new
+    if valid_frac.sum() > 10:
+        g_f = g[valid_frac]
+        gt_f = gt_frac[valid_frac]
+        sp_r, sp_p = stats.spearmanr(g_f, gt_f)
+        pe_r, pe_p = stats.pearsonr(np.log1p(g_f), np.log1p(gt_f))
+        print(f"\n  vs fraction new (n={valid_frac.sum()}):")
+        print(f"    Spearman r = {sp_r:.4f} (p = {sp_p:.2e})")
+        print(f"    Pearson  r = {pe_r:.4f} (p = {pe_p:.2e}) [log-space]")
+        results["frac_new"] = {
+            "spearman_r": float(sp_r), "spearman_p": float(sp_p),
+            "pearson_r": float(pe_r), "pearson_p": float(pe_p),
+            "n_genes": int(valid_frac.sum()),
+        }
+    else:
+        print("  Not enough shared genes for fraction new correlation.")
+        results["frac_new"] = {"n_genes": int(valid_frac.sum())}
+
+    # =========================================================================
+    # INDEPENDENT VALIDATION: PUBLISHED HALF-LIVES (not tautological)
+    # =========================================================================
+    print("\n--- Independent validation: published half-life correlations ---")
+    print("  (This is the key result — fully independent ground truth)")
+    hl_human = scptr.datasets.schofield2018_halflives()
+    corr_human = scptr.benchmark.correlate_with_halflives(adata, hl_human)
+    print(f"  Human half-lives (Schofield 2018): Spearman r = {corr_human['spearman_r']:.4f} "
+          f"(p={corr_human['spearman_p']:.2e}, n={corr_human['n_genes']})")
+    results["halflife_human"] = {
+        k: v for k, v in corr_human.items() if k != "matched_genes"
+    }
+
+    hl_mouse = scptr.datasets.herzog2017_halflives()
+    corr_mouse = scptr.benchmark.correlate_with_halflives(adata, hl_mouse)
+    print(f"  Mouse half-lives (Herzog 2017):    Spearman r = {corr_mouse['spearman_r']:.4f} "
+          f"(p={corr_mouse['spearman_p']:.2e}, n={corr_mouse['n_genes']})")
+    results["halflife_mouse"] = {
+        k: v for k, v in corr_mouse.items() if k != "matched_genes"
+    }
+
+    with open(res_dir / "scifate_validation.json", "w") as f:
+        json.dump(results, f, indent=2)
+
+    # =========================================================================
+    # SCATTER PLOTS
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("GENERATING FIGURES")
+    print("=" * 60)
+
+    fig, axes = plt.subplots(1, 3, figsize=(18, 5))
+
+    # Panel 1: gamma vs new/old ratio
+    if valid_ratio.sum() > 10:
+        g_r = g[valid_ratio]
+        gt_r = gt_ratio[valid_ratio]
+        axes[0].scatter(gt_r, g_r, alpha=0.15, s=8, c="steelblue")
+        axes[0].set_xscale("log")
+        axes[0].set_yscale("log")
+        axes[0].set_xlabel("Ground truth: new/old RNA ratio")
+        axes[0].set_ylabel("scPTR median gamma")
+        sp_r = results["new_old_ratio"]["spearman_r"]
+        sp_p = results["new_old_ratio"]["spearman_p"]
+        axes[0].set_title(f"vs New/Old ratio\n(Spearman r={sp_r:.3f}, p={sp_p:.1e})")
+
+    # Panel 2: gamma vs fraction new
+    if valid_frac.sum() > 10:
+        g_f = g[valid_frac]
+        gt_f = gt_frac[valid_frac]
+        axes[1].scatter(gt_f, g_f, alpha=0.15, s=8, c="darkorange")
+        axes[1].set_xscale("log")
+        axes[1].set_yscale("log")
+        axes[1].set_xlabel("Ground truth: fraction new RNA")
+        axes[1].set_ylabel("scPTR median gamma")
+        sp_r = results["frac_new"]["spearman_r"]
+        sp_p = results["frac_new"]["spearman_p"]
+        axes[1].set_title(f"vs Fraction new\n(Spearman r={sp_r:.3f}, p={sp_p:.1e})")
+
+    # Panel 3: Distribution comparison
+    ax3 = axes[2]
+    # Log-transform and z-score both, show rank correlation
+    if valid_ratio.sum() > 10:
+        g_log = np.log1p(g[valid_ratio])
+        gt_log = np.log1p(gt_ratio[valid_ratio])
+        # Rank both
+        g_rank = stats.rankdata(g_log)
+        gt_rank = stats.rankdata(gt_log)
+        ax3.scatter(gt_rank / len(gt_rank), g_rank / len(g_rank),
+                   alpha=0.1, s=5, c="purple")
+        ax3.plot([0, 1], [0, 1], "k--", alpha=0.3, lw=1)
+        ax3.set_xlabel("Ground truth rank (fractional)")
+        ax3.set_ylabel("scPTR gamma rank (fractional)")
+        ax3.set_title("Rank-rank plot")
+
+    fig.suptitle("sci-fate Validation: scPTR Gamma vs Ground Truth Degradation",
+                fontsize=13, y=1.02)
+    fig.tight_layout()
+    save_fig(fig, "scifate_gamma_vs_ground_truth")
+
+    # =========================================================================
+    # PER-TIMEPOINT VALIDATION
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("PER-TIMEPOINT VALIDATION")
+    print("=" * 60)
+
+    tp_results = {}
+    fig, axes = plt.subplots(2, 3, figsize=(16, 10))
+    axes = axes.flatten()
+
+    for idx, tp in enumerate(timepoints):
+        gt_tp = gt_by_time[tp].set_index("gene")
+        shared_tp = gamma_series.index.intersection(gt_tp.index)
+        g_tp = gamma_series[shared_tp].values.astype(float)
+        gt_tp_ratio = gt_tp.loc[shared_tp, "new_old_ratio"].values.astype(float)
+        valid = np.isfinite(g_tp) & np.isfinite(gt_tp_ratio) & (g_tp > 0) & (gt_tp_ratio > 0)
+
+        if valid.sum() > 10:
+            sp_r, sp_p = stats.spearmanr(g_tp[valid], gt_tp_ratio[valid])
+            print(f"  {tp}: Spearman r = {sp_r:.4f} (n={valid.sum()})")
+            tp_results[tp] = {"spearman_r": float(sp_r), "spearman_p": float(sp_p),
+                             "n_genes": int(valid.sum())}
+
+            if idx < len(axes):
+                axes[idx].scatter(gt_tp_ratio[valid], g_tp[valid],
+                                alpha=0.1, s=5, c="steelblue")
+                axes[idx].set_xscale("log")
+                axes[idx].set_yscale("log")
+                axes[idx].set_xlabel("New/old ratio")
+                axes[idx].set_ylabel("scPTR gamma")
+                axes[idx].set_title(f"DEX {tp} (r={sp_r:.3f}, n={valid.sum()})")
+        else:
+            print(f"  {tp}: Not enough genes ({valid.sum()})")
+
+    # Remove unused axes
+    for idx in range(len(timepoints), len(axes)):
+        axes[idx].set_visible(False)
+
+    fig.suptitle("sci-fate: Per-timepoint scPTR gamma vs ground truth",
+                fontsize=13, y=1.02)
+    fig.tight_layout()
+    save_fig(fig, "scifate_per_timepoint")
+
+    with open(res_dir / "scifate_per_timepoint.json", "w") as f:
+        json.dump(tp_results, f, indent=2)
+
+    # =========================================================================
+    # TOP/BOTTOM GENE ANALYSIS
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("TOP/BOTTOM GENE ANALYSIS")
+    print("=" * 60)
+
+    if valid_ratio.sum() > 100:
+        # Compare top/bottom gamma genes with ground truth ranking
+        gene_df = pd.DataFrame({
+            "gene": shared[valid_ratio],
+            "gamma": g[valid_ratio],
+            "new_old_ratio": gt_ratio[valid_ratio],
+        })
+        gene_df = gene_df.sort_values("gamma", ascending=False)
+
+        # Top 10% gamma genes
+        n10 = max(10, len(gene_df) // 10)
+        top_gamma = gene_df.head(n10)
+        bot_gamma = gene_df.tail(n10)
+
+        top_gt_med = top_gamma["new_old_ratio"].median()
+        bot_gt_med = bot_gamma["new_old_ratio"].median()
+
+        print(f"\n  Top {n10} gamma genes: median new/old ratio = {top_gt_med:.4f}")
+        print(f"  Bottom {n10} gamma genes: median new/old ratio = {bot_gt_med:.4f}")
+        print(f"  Fold difference: {top_gt_med / bot_gt_med:.2f}x")
+
+        # Mann-Whitney test
+        u_stat, mw_p = stats.mannwhitneyu(
+            top_gamma["new_old_ratio"].values,
+            bot_gamma["new_old_ratio"].values,
+            alternative="greater"
+        )
+        print(f"  Mann-Whitney p-value (top > bottom): {mw_p:.2e}")
+
+        results["top_bottom_analysis"] = {
+            "n_per_group": n10,
+            "top_gamma_median_gt": float(top_gt_med),
+            "bottom_gamma_median_gt": float(bot_gt_med),
+            "fold_difference": float(top_gt_med / bot_gt_med),
+            "mann_whitney_p": float(mw_p),
+        }
+
+        # Save updated results
+        with open(res_dir / "scifate_validation.json", "w") as f:
+            json.dump(results, f, indent=2)
+
+        # Boxplot
+        fig, ax = plt.subplots(figsize=(6, 5))
+        positions = [1, 2]
+        bp = ax.boxplot(
+            [top_gamma["new_old_ratio"].values, bot_gamma["new_old_ratio"].values],
+            positions=positions,
+            widths=0.6,
+            patch_artist=True,
+        )
+        bp["boxes"][0].set_facecolor("salmon")
+        bp["boxes"][1].set_facecolor("lightblue")
+        ax.set_xticks(positions)
+        ax.set_xticklabels([f"Top {n10}\n(high gamma)", f"Bottom {n10}\n(low gamma)"])
+        ax.set_ylabel("Ground truth: new/old RNA ratio")
+        ax.set_title(f"High-gamma genes have higher turnover\n"
+                    f"(fold={top_gt_med/bot_gt_med:.1f}x, p={mw_p:.1e})")
+        fig.tight_layout()
+        save_fig(fig, "scifate_top_bottom_boxplot")
+
+    # =========================================================================
+    # SUMMARY
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("SUMMARY")
+    print("=" * 60)
+    print(f"  Dataset: sci-fate A549 ({adata_raw.n_obs} cells, {adata_raw.n_vars} genes)")
+    print(f"  scPTR pipeline: {adata.n_obs} cells, {adata.n_vars} genes")
+    if "new_old_ratio" in results and "spearman_r" in results["new_old_ratio"]:
+        print(f"  Gamma vs new/old ratio: Spearman r = {results['new_old_ratio']['spearman_r']:.4f}")
+    if "frac_new" in results and "spearman_r" in results["frac_new"]:
+        print(f"  Gamma vs frac new: Spearman r = {results['frac_new']['spearman_r']:.4f}")
+    if "halflife_human" in results:
+        print(f"  Human half-life (INDEPENDENT): Spearman r = {results['halflife_human']['spearman_r']:.4f}")
+    if "halflife_mouse" in results and "spearman_r" in results["halflife_mouse"]:
+        print(f"  Mouse half-life (INDEPENDENT): Spearman r = {results['halflife_mouse']['spearman_r']:.4f}")
+    if "top_bottom_analysis" in results:
+        tb = results["top_bottom_analysis"]
+        print(f"  Top vs bottom gamma: {tb['fold_difference']:.1f}x fold diff (p={tb['mann_whitney_p']:.1e})")
+    print(f"\nAll results saved to: {OUTPUT_DIR.resolve()}")
+
+
+if __name__ == "__main__":
+    main()

analyses/run_summary.py

@@ -0,0 +1,380 @@
+#!/usr/bin/env python
+"""Cross-dataset validation summary: consolidate results from all datasets.
+
+Runs the full scPTR pipeline on all 3 datasets and produces:
+1. Cross-dataset consistency (pairwise gamma correlation)
+2. Half-life validation across all datasets
+3. ARE/NMD enrichment across datasets
+4. Subsampling robustness across datasets
+5. Summary table and comparison figures
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+from scipy import stats
+
+sys.path.insert(0, str(Path(__file__).parent))
+from _common import set_figure_style
+
+import scptr
+from run_scifate import load_scifate_data, prepare_for_scptr
+
+OUTPUT_DIR = Path(__file__).parent.parent / "output" / "summary"
+
+
+def save_fig(fig, name, subdir="figures"):
+    if fig is None:
+        return
+    out_dir = OUTPUT_DIR / subdir
+    out_dir.mkdir(parents=True, exist_ok=True)
+    path = out_dir / f"{name}.png"
+    fig.savefig(path, dpi=150, bbox_inches="tight")
+    plt.close(fig)
+    print(f"  Saved: {path}")
+
+
+def run_pipeline(adata, name, groupby=None):
+    """Run standard scPTR pipeline on a dataset."""
+    print(f"\n--- Running pipeline on {name} ---")
+    print(f"  Input: {adata.shape}")
+
+    scptr.pp.filter_genes(adata)
+    scptr.pp.normalize_layers(adata)
+    scptr.pp.neighbors(adata, n_neighbors=30)
+    scptr.pp.smooth_layers(adata)
+
+    scptr.tl.estimate_beta(adata)
+    if groupby:
+        scptr.tl.estimate_beta(adata, groupby=groupby)
+    scptr.tl.estimate_gamma(adata)
+    scptr.tl.variance_decomposition(adata)
+    scptr.tl.pt_states(adata)
+    scptr.tl.pt_velocity(adata)
+
+    gamma = adata.layers["gamma"]
+    gamma_med = np.median(gamma, axis=0)
+    n_states = adata.obs["pt_state"].nunique()
+    print(f"  After pipeline: {adata.shape}")
+    print(f"  Gamma: median={np.median(gamma_med):.4f}, max={np.max(gamma):.2f}")
+    print(f"  PT states: {n_states}")
+    return adata
+
+
+def main():
+    set_figure_style()
+    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+    res_dir = OUTPUT_DIR / "results"
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    # =========================================================================
+    # LOAD ALL DATASETS
+    # =========================================================================
+    print("=" * 60)
+    print("LOADING DATASETS")
+    print("=" * 60)
+
+    print("\n--- Pancreas ---")
+    adata_pan = scptr.datasets.pancreas()
+    print(f"  Shape: {adata_pan.shape}")
+
+    print("\n--- Dentate Gyrus ---")
+    adata_dg = scptr.datasets.dentate_gyrus()
+    print(f"  Shape: {adata_dg.shape}")
+
+    print("\n--- sci-fate A549 ---")
+    adata_sf_raw = load_scifate_data()
+    adata_sf = prepare_for_scptr(adata_sf_raw)
+    print(f"  Shape: {adata_sf.shape}")
+
+    # =========================================================================
+    # RUN PIPELINES
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("RUNNING PIPELINES")
+    print("=" * 60)
+
+    adata_pan = run_pipeline(adata_pan, "pancreas", groupby="clusters")
+    adata_dg = run_pipeline(adata_dg, "dentate_gyrus", groupby="clusters")
+    adata_sf = run_pipeline(adata_sf, "scifate")
+
+    datasets = {
+        "pancreas": adata_pan,
+        "dentate_gyrus": adata_dg,
+        "scifate": adata_sf,
+    }
+
+    # =========================================================================
+    # 1. CROSS-DATASET CONSISTENCY
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("1. CROSS-DATASET CONSISTENCY")
+    print("=" * 60)
+
+    consistency = scptr.benchmark.cross_dataset_consistency(datasets)
+    consistency.to_csv(res_dir / "cross_dataset_consistency.csv", index=False)
+    print(consistency.to_string(index=False))
+
+    # =========================================================================
+    # 2. HALF-LIFE VALIDATION
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("2. HALF-LIFE VALIDATION")
+    print("=" * 60)
+
+    hl_mouse = scptr.datasets.herzog2017_halflives()
+    hl_human = scptr.datasets.schofield2018_halflives()
+
+    hl_results = []
+    for name, adata in datasets.items():
+        for hl_name, hl_df in [("mouse_Herzog2017", hl_mouse),
+                                ("human_Schofield2018", hl_human)]:
+            corr = scptr.benchmark.correlate_with_halflives(adata, hl_df)
+            hl_results.append({
+                "dataset": name,
+                "reference": hl_name,
+                "spearman_r": corr["spearman_r"],
+                "spearman_p": corr["spearman_p"],
+                "pearson_r": corr["pearson_r"],
+                "n_genes": corr["n_genes"],
+            })
+            print(f"  {name} vs {hl_name}: Spearman r = {corr['spearman_r']:.4f} "
+                  f"(n={corr['n_genes']})")
+
+    hl_df_out = pd.DataFrame(hl_results)
+    hl_df_out.to_csv(res_dir / "halflife_correlations.csv", index=False)
+
+    # =========================================================================
+    # 3. ARE/NMD ENRICHMENT
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("3. ARE/NMD ENRICHMENT")
+    print("=" * 60)
+
+    enrichment_results = []
+    for name, adata in datasets.items():
+        are = scptr.benchmark.are_enrichment(adata)
+        nmd = scptr.benchmark.nmd_enrichment(adata)
+        enrichment_results.append({
+            "dataset": name,
+            "test": "ARE",
+            "n_in_set": are["n_genes_in_set"],
+            "U_statistic": are["U_statistic"],
+            "p_value": are["p_value"],
+        })
+        enrichment_results.append({
+            "dataset": name,
+            "test": "NMD",
+            "n_in_set": nmd["n_genes_in_set"],
+            "U_statistic": nmd["U_statistic"],
+            "p_value": nmd["p_value"],
+        })
+        print(f"  {name}: ARE p={are['p_value']:.4f} (n={are['n_genes_in_set']}), "
+              f"NMD p={nmd['p_value']:.4f} (n={nmd['n_genes_in_set']})")
+
+    enr_df = pd.DataFrame(enrichment_results)
+    enr_df.to_csv(res_dir / "enrichment_results.csv", index=False)
+
+    # =========================================================================
+    # 4. SUBSAMPLING ROBUSTNESS
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("4. SUBSAMPLING ROBUSTNESS")
+    print("=" * 60)
+
+    fractions = [0.2, 0.4, 0.6, 0.8, 0.9]
+    robustness_results = []
+    for name, adata in datasets.items():
+        print(f"\n  {name}:")
+        robust = scptr.benchmark.subsampling_robustness(
+            adata, fractions=fractions, n_repeats=3
+        )
+        robust["dataset"] = name
+        robustness_results.append(robust)
+        for frac in fractions:
+            sub = robust[robust["fraction"] == frac]
+            print(f"    {frac:.0%}: mean Spearman r = {sub['spearman_r'].mean():.4f}")
+
+    robust_all = pd.concat(robustness_results, ignore_index=True)
+    robust_all.to_csv(res_dir / "subsampling_robustness.csv", index=False)
+
+    # =========================================================================
+    # 5. DATASET STATISTICS
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("5. DATASET STATISTICS")
+    print("=" * 60)
+
+    dataset_stats = []
+    for name, adata in datasets.items():
+        gamma = adata.layers["gamma"]
+        gamma_med = np.median(gamma, axis=0)
+        n_states = adata.obs["pt_state"].nunique()
+        tf_scores = adata.var["tf_score"].values
+
+        dataset_stats.append({
+            "dataset": name,
+            "n_cells": adata.n_obs,
+            "n_genes": adata.n_vars,
+            "beta_median": float(np.median(adata.var["beta"])),
+            "gamma_median_of_medians": float(np.median(gamma_med)),
+            "gamma_max": float(np.max(gamma)),
+            "n_pt_states": n_states,
+            "tf_score_median": float(np.median(tf_scores)),
+            "tf_score_gt_0.5": int(np.sum(tf_scores > 0.5)),
+        })
+        print(f"  {name}: {adata.n_obs} cells, {adata.n_vars} genes, "
+              f"{n_states} PT states")
+
+    stats_df = pd.DataFrame(dataset_stats)
+    stats_df.to_csv(res_dir / "dataset_statistics.csv", index=False)
+
+    # =========================================================================
+    # FIGURES
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("GENERATING SUMMARY FIGURES")
+    print("=" * 60)
+
+    # Figure 1: Half-life correlation comparison bar chart
+    fig, ax = plt.subplots(figsize=(8, 5))
+    hl_pivot = hl_df_out.pivot(index="dataset", columns="reference",
+                               values="spearman_r")
+    x = np.arange(len(hl_pivot))
+    width = 0.35
+    bars1 = ax.bar(x - width/2, hl_pivot["mouse_Herzog2017"].values,
+                   width, label="Mouse (Herzog 2017)", color="steelblue")
+    bars2 = ax.bar(x + width/2, hl_pivot["human_Schofield2018"].values,
+                   width, label="Human (Schofield 2018)", color="darkorange")
+    ax.set_xlabel("Dataset")
+    ax.set_ylabel("Spearman correlation with half-lives")
+    ax.set_title("Half-life Validation Across Datasets")
+    ax.set_xticks(x)
+    ax.set_xticklabels(hl_pivot.index)
+    ax.legend()
+    ax.axhline(y=0, color="gray", linewidth=0.5)
+    # Add value labels
+    for bars in [bars1, bars2]:
+        for bar in bars:
+            h = bar.get_height()
+            ax.text(bar.get_x() + bar.get_width()/2, h,
+                   f"{h:.3f}", ha="center", va="bottom" if h > 0 else "top",
+                   fontsize=8)
+    fig.tight_layout()
+    save_fig(fig, "halflife_comparison")
+
+    # Figure 2: Robustness curves
+    fig, ax = plt.subplots(figsize=(8, 5))
+    colors = {"pancreas": "steelblue", "dentate_gyrus": "darkorange",
+              "scifate": "forestgreen"}
+    for name in datasets:
+        sub = robust_all[robust_all["dataset"] == name]
+        means = sub.groupby("fraction")["spearman_r"].mean()
+        stds = sub.groupby("fraction")["spearman_r"].std()
+        ax.errorbar(means.index, means.values, yerr=stds.values,
+                   marker="o", label=name, color=colors.get(name, "gray"),
+                   capsize=3)
+    ax.set_xlabel("Subsampling fraction")
+    ax.set_ylabel("Spearman r with full-data gamma")
+    ax.set_title("Subsampling Robustness Across Datasets")
+    ax.legend()
+    ax.set_ylim(0, 1.05)
+    fig.tight_layout()
+    save_fig(fig, "robustness_curves")
+
+    # Figure 3: Cross-dataset consistency heatmap
+    ds_names = sorted(datasets.keys())
+    mat = np.eye(len(ds_names))
+    for _, row in consistency.iterrows():
+        i = ds_names.index(row["dataset_a"])
+        j = ds_names.index(row["dataset_b"])
+        mat[i, j] = mat[j, i] = row["spearman_r"]
+
+    fig, ax = plt.subplots(figsize=(6, 5))
+    im = ax.imshow(mat, cmap="RdYlBu_r", vmin=-0.2, vmax=1.0)
+    ax.set_xticks(range(len(ds_names)))
+    ax.set_yticks(range(len(ds_names)))
+    ax.set_xticklabels(ds_names, rotation=45, ha="right")
+    ax.set_yticklabels(ds_names)
+    for i in range(len(ds_names)):
+        for j in range(len(ds_names)):
+            ax.text(j, i, f"{mat[i,j]:.3f}", ha="center", va="center",
+                   fontsize=10, fontweight="bold" if i != j else "normal")
+    plt.colorbar(im, ax=ax, label="Spearman r")
+    ax.set_title("Cross-Dataset Gamma Consistency")
+    fig.tight_layout()
+    save_fig(fig, "cross_dataset_heatmap")
+
+    # Figure 4: Enrichment comparison
+    fig, axes = plt.subplots(1, 2, figsize=(12, 5))
+    for idx, test in enumerate(["ARE", "NMD"]):
+        sub = enr_df[enr_df["test"] == test]
+        x = np.arange(len(sub))
+        pvals = sub["p_value"].values
+        neg_log_p = [-np.log10(max(p, 1e-300)) for p in pvals]
+        bars = axes[idx].bar(x, neg_log_p,
+                            color=["steelblue", "darkorange", "forestgreen"])
+        axes[idx].set_xticks(x)
+        axes[idx].set_xticklabels(sub["dataset"].values, rotation=45, ha="right")
+        axes[idx].set_ylabel("-log10(p-value)")
+        axes[idx].set_title(f"{test} Enrichment")
+        axes[idx].axhline(y=-np.log10(0.05), color="red", linestyle="--",
+                         alpha=0.5, label="p=0.05")
+        axes[idx].legend()
+    fig.suptitle("ARE/NMD Enrichment Across Datasets", fontsize=13)
+    fig.tight_layout()
+    save_fig(fig, "enrichment_comparison")
+
+    # =========================================================================
+    # SUMMARY TABLE
+    # =========================================================================
+    print("\n" + "=" * 60)
+    print("COMPREHENSIVE SUMMARY")
+    print("=" * 60)
+
+    summary = {}
+    for name in datasets:
+        s = stats_df[stats_df["dataset"] == name].iloc[0]
+        hl_sub = hl_df_out[hl_df_out["dataset"] == name]
+        rob_90 = robust_all[(robust_all["dataset"] == name) &
+                           (robust_all["fraction"] == 0.9)]
+        are_sub = enr_df[(enr_df["dataset"] == name) & (enr_df["test"] == "ARE")]
+        nmd_sub = enr_df[(enr_df["dataset"] == name) & (enr_df["test"] == "NMD")]
+
+        summary[name] = {
+            "cells": int(s["n_cells"]),
+            "genes": int(s["n_genes"]),
+            "pt_states": int(s["n_pt_states"]),
+            "hl_mouse_r": float(hl_sub[hl_sub["reference"] == "mouse_Herzog2017"]["spearman_r"].values[0]),
+            "hl_human_r": float(hl_sub[hl_sub["reference"] == "human_Schofield2018"]["spearman_r"].values[0]),
+            "robustness_90pct": float(rob_90["spearman_r"].mean()),
+            "are_p": float(are_sub["p_value"].values[0]),
+            "nmd_p": float(nmd_sub["p_value"].values[0]),
+        }
+
+    with open(res_dir / "comprehensive_summary.json", "w") as f:
+        json.dump(summary, f, indent=2)
+
+    # Print formatted summary
+    print(f"\n{'Dataset':<15} {'Cells':>6} {'Genes':>6} {'States':>6} "
+          f"{'HL(m)':>8} {'HL(h)':>8} {'Rob90':>7} {'ARE_p':>8} {'NMD_p':>8}")
+    print("-" * 85)
+    for name, s in summary.items():
+        print(f"{name:<15} {s['cells']:>6} {s['genes']:>6} {s['pt_states']:>6} "
+              f"{s['hl_mouse_r']:>8.4f} {s['hl_human_r']:>8.4f} "
+              f"{s['robustness_90pct']:>7.4f} "
+              f"{s['are_p']:>8.4f} {s['nmd_p']:>8.4f}")
+
+    print(f"\nAll results saved to: {OUTPUT_DIR.resolve()}")
+
+
+if __name__ == "__main__":
+    main()

analyses/run_tier2_validation.py

@@ -0,0 +1,493 @@
+#!/usr/bin/env python
+"""Tier 2 validation: sequence-feature correlations and eCLIP validation.
+
+T2-4: Correlate gamma with 3' UTR length and AU content
+      (sequence-feature-based validation, replacing curated gene lists)
+T2-5: Validate RBP-target network predictions against ENCODE eCLIP data
+      (Fisher's exact test for overlap enrichment)
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+from scipy import stats
+
+sys.path.insert(0, str(Path(__file__).parent))
+from _common import set_figure_style
+
+import scptr
+
+OUTPUT_DIR = Path(__file__).parent.parent / "output" / "tier2_validation"
+DATA_DIR = Path(__file__).parent.parent / "src" / "scptr" / "benchmark" / "data"
+
+
+def save_fig(fig, name, subdir="figures"):
+    out_dir = OUTPUT_DIR / subdir
+    out_dir.mkdir(parents=True, exist_ok=True)
+    path = out_dir / f"{name}.png"
+    fig.savefig(path, dpi=150, bbox_inches="tight")
+    plt.close(fig)
+    print(f"  Saved: {path}")
+
+
+def run_pipeline(adata, name):
+    """Run standard scPTR pipeline."""
+    print(f"\n--- Pipeline: {name} ---")
+    scptr.pp.filter_genes(adata)
+    scptr.pp.normalize_layers(adata)
+    scptr.pp.neighbors(adata, n_neighbors=30)
+    scptr.pp.smooth_layers(adata)
+    scptr.tl.estimate_beta(adata)
+    scptr.tl.estimate_gamma(adata)
+    scptr.tl.variance_decomposition(adata)
+    scptr.tl.pt_states(adata)
+    scptr.tl.pt_velocity(adata)
+    print(f"  Done: {adata.shape}")
+    return adata
+
+
+# =========================================================================
+# T2-4: Sequence-feature validation
+# =========================================================================
+def sequence_feature_validation(adata, name, species):
+    """Correlate per-gene gamma with 3' UTR length and AU content.
+
+    Hypothesis:
+    - Longer 3' UTRs → more regulatory elements → higher gamma (positive corr)
+    - Higher AU content → ARE-mediated decay → higher gamma (positive corr)
+    """
+    print(f"\n{'='*60}")
+    print(f"T2-4: SEQUENCE FEATURE VALIDATION ({name})")
+    print(f"{'='*60}")
+
+    res_dir = OUTPUT_DIR / "results"
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    # Load UTR features
+    utr_file = DATA_DIR / f"{species}_utr_features.csv"
+    if not utr_file.exists():
+        print(f"  ERROR: {utr_file} not found. Run download_utr_features.py first.")
+        return None
+    utr_df = pd.read_csv(utr_file)
+    print(f"  Loaded {len(utr_df)} {species} genes with UTR features")
+
+    # Per-gene median gamma
+    gamma = adata.layers["gamma"]
+    gene_names = adata.var_names.tolist()
+    median_gamma = np.median(gamma, axis=0)
+    nonzero_frac = (gamma > 0).mean(axis=0)
+
+    # Build gene-level DataFrame
+    gamma_df = pd.DataFrame({
+        "gene": gene_names,
+        "median_gamma": median_gamma,
+        "nonzero_frac": nonzero_frac,
+    })
+
+    # Filter to gamma-informative genes
+    gamma_df = gamma_df[gamma_df["nonzero_frac"] >= 0.1].copy()
+    print(f"  Gamma-informative genes: {len(gamma_df)}")
+
+    # Case-insensitive merge
+    gamma_df["gene_upper"] = gamma_df["gene"].str.upper()
+    utr_df["gene_upper"] = utr_df["gene"].str.upper()
+
+    merged = gamma_df.merge(utr_df[["gene_upper", "utr_length", "au_content"]],
+                            on="gene_upper", how="inner")
+    print(f"  Merged with UTR features: {len(merged)} genes")
+
+    if len(merged) < 50:
+        print("  Too few genes for analysis")
+        return None
+
+    # Filter extreme outliers
+    merged = merged[merged["utr_length"] > 0].copy()
+    merged["log_utr_length"] = np.log10(merged["utr_length"])
+    merged["log_gamma"] = np.log1p(merged["median_gamma"])
+
+    results = {}
+
+    # 1. Gamma vs UTR length
+    r_len, p_len = stats.spearmanr(merged["log_utr_length"], merged["median_gamma"])
+    print(f"\n  Gamma vs log10(UTR length):")
+    print(f"    Spearman r = {r_len:.4f}, p = {p_len:.2e}")
+    print(f"    n = {len(merged)} genes")
+    results["utr_length_spearman_r"] = float(r_len)
+    results["utr_length_p"] = float(p_len)
+
+    # 2. Gamma vs AU content
+    r_au, p_au = stats.spearmanr(merged["au_content"], merged["median_gamma"])
+    print(f"\n  Gamma vs AU content:")
+    print(f"    Spearman r = {r_au:.4f}, p = {p_au:.2e}")
+    results["au_content_spearman_r"] = float(r_au)
+    results["au_content_p"] = float(p_au)
+
+    # 3. Quartile analysis: genes in top vs bottom UTR length quartile
+    q1 = merged["log_utr_length"].quantile(0.25)
+    q4 = merged["log_utr_length"].quantile(0.75)
+    short_utr = merged[merged["log_utr_length"] <= q1]
+    long_utr = merged[merged["log_utr_length"] >= q4]
+
+    median_gamma_short = short_utr["median_gamma"].median()
+    median_gamma_long = long_utr["median_gamma"].median()
+    u_stat, u_p = stats.mannwhitneyu(long_utr["median_gamma"],
+                                      short_utr["median_gamma"],
+                                      alternative="greater")
+    print(f"\n  Quartile analysis (UTR length):")
+    print(f"    Short UTR (Q1) median gamma: {median_gamma_short:.4f} (n={len(short_utr)})")
+    print(f"    Long UTR (Q4) median gamma: {median_gamma_long:.4f} (n={len(long_utr)})")
+    print(f"    Mann-Whitney (long > short): p = {u_p:.2e}")
+    results["long_vs_short_utr_mw_p"] = float(u_p)
+    results["median_gamma_short_utr"] = float(median_gamma_short)
+    results["median_gamma_long_utr"] = float(median_gamma_long)
+
+    # 4. AU content quartile
+    au_q1 = merged["au_content"].quantile(0.25)
+    au_q4 = merged["au_content"].quantile(0.75)
+    low_au = merged[merged["au_content"] <= au_q1]
+    high_au = merged[merged["au_content"] >= au_q4]
+
+    median_gamma_low_au = low_au["median_gamma"].median()
+    median_gamma_high_au = high_au["median_gamma"].median()
+    au_u_stat, au_u_p = stats.mannwhitneyu(high_au["median_gamma"],
+                                            low_au["median_gamma"],
+                                            alternative="greater")
+    print(f"\n  Quartile analysis (AU content):")
+    print(f"    Low AU (Q1) median gamma: {median_gamma_low_au:.4f} (n={len(low_au)})")
+    print(f"    High AU (Q4) median gamma: {median_gamma_high_au:.4f} (n={len(high_au)})")
+    print(f"    Mann-Whitney (high AU > low AU): p = {au_u_p:.2e}")
+    results["high_vs_low_au_mw_p"] = float(au_u_p)
+
+    results["n_genes"] = len(merged)
+
+    # Figure: 2x2 scatter + quartile boxplots
+    fig, axes = plt.subplots(2, 2, figsize=(12, 10))
+
+    # Scatter: gamma vs UTR length
+    axes[0, 0].scatter(merged["log_utr_length"], merged["log_gamma"],
+                       s=2, alpha=0.3, color="steelblue")
+    axes[0, 0].set_xlabel("log10(3' UTR length)")
+    axes[0, 0].set_ylabel("log1p(median gamma)")
+    axes[0, 0].set_title(f"Gamma vs 3' UTR Length ({name})\nr={r_len:.3f}, p={p_len:.1e}")
+
+    # Scatter: gamma vs AU content
+    axes[0, 1].scatter(merged["au_content"], merged["log_gamma"],
+                       s=2, alpha=0.3, color="darkorange")
+    axes[0, 1].set_xlabel("3' UTR AU content")
+    axes[0, 1].set_ylabel("log1p(median gamma)")
+    axes[0, 1].set_title(f"Gamma vs AU Content ({name})\nr={r_au:.3f}, p={p_au:.1e}")
+
+    # Boxplot: UTR length quartiles
+    quartile_data = []
+    quartile_labels = []
+    for qi, (lo, hi, label) in enumerate([
+        (0, 0.25, "Q1\n(short)"), (0.25, 0.5, "Q2"), (0.5, 0.75, "Q3"),
+        (0.75, 1.0, "Q4\n(long)")
+    ]):
+        qlo = merged["log_utr_length"].quantile(lo)
+        qhi = merged["log_utr_length"].quantile(hi)
+        mask = (merged["log_utr_length"] >= qlo) & (merged["log_utr_length"] <= qhi)
+        quartile_data.append(merged.loc[mask, "median_gamma"].values)
+        quartile_labels.append(label)
+    bp = axes[1, 0].boxplot(quartile_data, labels=quartile_labels, patch_artist=True,
+                             showfliers=False)
+    colors = ["#2196F3", "#64B5F6", "#FFA726", "#E65100"]
+    for patch, color in zip(bp["boxes"], colors):
+        patch.set_facecolor(color)
+    axes[1, 0].set_ylabel("Median gamma")
+    axes[1, 0].set_xlabel("3' UTR Length Quartile")
+    axes[1, 0].set_title(f"Gamma by UTR Length Quartile\np={u_p:.1e}")
+
+    # Boxplot: AU content quartiles
+    au_data = []
+    au_labels = []
+    for qi, (lo, hi, label) in enumerate([
+        (0, 0.25, "Q1\n(low AU)"), (0.25, 0.5, "Q2"), (0.5, 0.75, "Q3"),
+        (0.75, 1.0, "Q4\n(high AU)")
+    ]):
+        qlo = merged["au_content"].quantile(lo)
+        qhi = merged["au_content"].quantile(hi)
+        mask = (merged["au_content"] >= qlo) & (merged["au_content"] <= qhi)
+        au_data.append(merged.loc[mask, "median_gamma"].values)
+        au_labels.append(label)
+    bp2 = axes[1, 1].boxplot(au_data, labels=au_labels, patch_artist=True,
+                              showfliers=False)
+    colors2 = ["#4CAF50", "#81C784", "#FFB74D", "#FF5722"]
+    for patch, color in zip(bp2["boxes"], colors2):
+        patch.set_facecolor(color)
+    axes[1, 1].set_ylabel("Median gamma")
+    axes[1, 1].set_xlabel("3' UTR AU Content Quartile")
+    axes[1, 1].set_title(f"Gamma by AU Content Quartile\np={au_u_p:.1e}")
+
+    fig.suptitle(f"Sequence Feature Validation: {name}", fontsize=14, y=1.02)
+    fig.tight_layout()
+    save_fig(fig, f"seq_features_{name}")
+
+    return results
+
+
+# =========================================================================
+# T2-5: eCLIP validation of RBP-target networks
+# =========================================================================
+def eclip_validation(adata, name):
+    """Validate scPTR-predicted RBP-target edges against ENCODE eCLIP data.
+
+    For each RBP with both scPTR predictions and eCLIP data:
+    - Fisher's exact test: are predicted targets enriched for eCLIP-confirmed targets?
+    - Report odds ratio and p-value
+    """
+    print(f"\n{'='*60}")
+    print(f"T2-5: eCLIP VALIDATION ({name})")
+    print(f"{'='*60}")
+
+    res_dir = OUTPUT_DIR / "results"
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    # Load eCLIP targets
+    eclip_file = DATA_DIR / "eclip_targets.csv"
+    if not eclip_file.exists():
+        print(f"  ERROR: {eclip_file} not found. Run download_eclip.py first.")
+        return None
+    eclip_df = pd.read_csv(eclip_file)
+    print(f"  Loaded {len(eclip_df)} eCLIP RBP-target pairs")
+
+    # Build eCLIP target sets per RBP (uppercase for matching)
+    eclip_targets = {}
+    for rbp, grp in eclip_df.groupby("rbp"):
+        eclip_targets[rbp.upper()] = set(g.upper() for g in grp["target_gene"])
+
+    # Get scPTR network edges
+    gamma = adata.layers["gamma"]
+    gene_names = adata.var_names.tolist()
+    gene_upper = [g.upper() for g in gene_names]
+
+    # Load RBP list
+    rbp_path = Path(__file__).parent.parent / "src" / "scptr" / "tools" / "data" / "known_rbps.csv"
+    rbps = pd.read_csv(rbp_path)["gene_symbol"].tolist()
+
+    # Find RBPs in dataset
+    adata_gene_map = {g.upper(): i for i, g in enumerate(gene_names)}
+    rbp_in_data = {}
+    for r in rbps:
+        if r.upper() in adata_gene_map:
+            rbp_in_data[r.upper()] = adata_gene_map[r.upper()]
+
+    # Get expression matrix
+    if hasattr(adata.X, 'toarray'):
+        expr = adata.X.toarray()
+    else:
+        expr = np.asarray(adata.X)
+
+    # Select target genes: top variable gamma (filtered to informative)
+    nonzero_frac = (gamma > 0).mean(axis=0)
+    informative = nonzero_frac >= 0.1
+    gamma_var = np.var(gamma[:, informative], axis=0)
+    n_targets = min(200, informative.sum())
+    top_var_idx = np.argsort(gamma_var)[-n_targets:]
+    info_indices = np.where(informative)[0]
+    target_indices = info_indices[top_var_idx]
+    target_genes_upper = set(gene_upper[i] for i in target_indices)
+
+    # Compute scPTR network edges via Spearman correlation
+    print("  Computing scPTR network edges...")
+    scptr_edges = {}  # rbp_upper -> set of target_gene_upper
+
+    for rbp_upper, rbp_idx in rbp_in_data.items():
+        rbp_expr = expr[:, rbp_idx]
+        if np.std(rbp_expr) < 1e-6:
+            continue
+
+        targets = set()
+        for ti in target_indices:
+            target_gamma = gamma[:, ti]
+            valid = target_gamma > 0
+            if valid.sum() < 50:
+                continue
+
+            r, p = stats.spearmanr(rbp_expr[valid], target_gamma[valid])
+            # Bonferroni correction
+            if p < 0.05 / (len(rbp_in_data) * n_targets):
+                targets.add(gene_upper[ti])
+
+        if targets:
+            scptr_edges[rbp_upper] = targets
+
+    print(f"  scPTR edges: {sum(len(t) for t in scptr_edges.values())} total")
+    print(f"  RBPs with edges: {len(scptr_edges)}")
+
+    # All genes in dataset (uppercase) as universe
+    all_genes_upper = set(gene_upper)
+
+    # Fisher's exact test for each RBP with both scPTR and eCLIP data
+    results = []
+
+    for rbp_upper in sorted(set(scptr_edges.keys()) & set(eclip_targets.keys())):
+        predicted = scptr_edges[rbp_upper]
+        eclip = eclip_targets[rbp_upper]
+
+        # Restrict eCLIP targets to genes in our dataset
+        eclip_in_data = eclip & all_genes_upper
+        if len(eclip_in_data) < 10:
+            continue
+
+        # 2x2 contingency table
+        # predicted & eCLIP | predicted & ~eCLIP
+        # ~predicted & eCLIP | ~predicted & ~eCLIP
+        a = len(predicted & eclip_in_data)
+        b = len(predicted - eclip_in_data)
+        c = len(eclip_in_data - predicted)
+        d = len(all_genes_upper - predicted - eclip_in_data)
+
+        odds_ratio, p_val = stats.fisher_exact([[a, b], [c, d]], alternative="greater")
+
+        # Also compute simple overlap statistics
+        overlap_frac = a / max(len(predicted), 1)
+        expected_frac = len(eclip_in_data) / max(len(all_genes_upper), 1)
+        enrichment = overlap_frac / max(expected_frac, 1e-6)
+
+        print(f"\n  {rbp_upper}:")
+        print(f"    scPTR predicted targets: {len(predicted)}")
+        print(f"    eCLIP confirmed targets: {len(eclip_in_data)}")
+        print(f"    Overlap: {a}")
+        print(f"    Enrichment fold: {enrichment:.2f}x")
+        print(f"    Fisher's exact: OR={odds_ratio:.2f}, p={p_val:.4f}")
+
+        results.append({
+            "rbp": rbp_upper,
+            "n_predicted": len(predicted),
+            "n_eclip": len(eclip_in_data),
+            "n_overlap": a,
+            "odds_ratio": float(odds_ratio),
+            "p_value": float(p_val),
+            "enrichment_fold": float(enrichment),
+        })
+
+    if not results:
+        print("  No RBPs with both scPTR and eCLIP data found")
+        return None
+
+    results_df = pd.DataFrame(results)
+    results_df.to_csv(res_dir / f"eclip_validation_{name}.csv", index=False)
+
+    # Summary
+    n_sig = (results_df["p_value"] < 0.05).sum()
+    print(f"\n  Summary: {n_sig}/{len(results_df)} RBPs have significant eCLIP overlap (p<0.05)")
+    print(f"  Mean enrichment fold: {results_df['enrichment_fold'].mean():.2f}x")
+    print(f"  Mean odds ratio: {results_df['odds_ratio'].mean():.2f}")
+
+    # Figure: enrichment barplot
+    if len(results_df) > 0:
+        fig, axes = plt.subplots(1, 2, figsize=(14, 6))
+
+        # Enrichment fold
+        rbps = results_df["rbp"].values
+        enrichments = results_df["enrichment_fold"].values
+        pvals = results_df["p_value"].values
+        colors = ["steelblue" if p < 0.05 else "lightgray" for p in pvals]
+
+        bars = axes[0].bar(range(len(rbps)), enrichments, color=colors, edgecolor="black",
+                           linewidth=0.5)
+        axes[0].axhline(y=1, color="red", linestyle="--", alpha=0.5, label="Expected (random)")
+        axes[0].set_xticks(range(len(rbps)))
+        axes[0].set_xticklabels(rbps, rotation=45, ha="right", fontsize=9)
+        axes[0].set_ylabel("Enrichment fold (observed/expected)")
+        axes[0].set_title(f"eCLIP Validation: Target Enrichment ({name})")
+        axes[0].legend()
+        for i, (e, p) in enumerate(zip(enrichments, pvals)):
+            sig = "*" if p < 0.05 else ""
+            axes[0].text(i, e + 0.05, f"{e:.1f}x{sig}", ha="center", fontsize=8)
+
+        # Overlap counts
+        overlap_data = np.array([
+            results_df["n_overlap"].values,
+            results_df["n_predicted"].values - results_df["n_overlap"].values,
+        ])
+        axes[1].bar(range(len(rbps)), results_df["n_overlap"].values,
+                    color="steelblue", label="eCLIP confirmed", edgecolor="black", linewidth=0.5)
+        axes[1].bar(range(len(rbps)),
+                    results_df["n_predicted"].values - results_df["n_overlap"].values,
+                    bottom=results_df["n_overlap"].values,
+                    color="lightgray", label="Not confirmed", edgecolor="black", linewidth=0.5)
+        axes[1].set_xticks(range(len(rbps)))
+        axes[1].set_xticklabels(rbps, rotation=45, ha="right", fontsize=9)
+        axes[1].set_ylabel("Number of predicted targets")
+        axes[1].set_title(f"Predicted Target Overlap with eCLIP ({name})")
+        axes[1].legend()
+
+        fig.tight_layout()
+        save_fig(fig, f"eclip_validation_{name}")
+
+    return results_df
+
+
+# =========================================================================
+# MAIN
+# =========================================================================
+def main():
+    set_figure_style()
+    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+
+    # Load and process datasets
+    print("=" * 60)
+    print("LOADING DATASETS")
+    print("=" * 60)
+
+    adata_pan = scptr.datasets.pancreas()
+    adata_pan = run_pipeline(adata_pan, "pancreas")
+
+    adata_dg = scptr.datasets.dentate_gyrus()
+    adata_dg = run_pipeline(adata_dg, "dentate_gyrus")
+
+    # Also load sci-fate
+    sys.path.insert(0, str(Path(__file__).parent))
+    from run_scifate import load_scifate_data, prepare_for_scptr
+    adata_sf_raw = load_scifate_data()
+    adata_sf = prepare_for_scptr(adata_sf_raw)
+    adata_sf = run_pipeline(adata_sf, "scifate")
+
+    datasets = {
+        "pancreas": (adata_pan, "mouse"),
+        "dentate_gyrus": (adata_dg, "mouse"),
+        "scifate": (adata_sf, "human"),
+    }
+
+    # T2-4: Sequence feature validation
+    print("\n" + "=" * 60)
+    print("T2-4: SEQUENCE FEATURE VALIDATION")
+    print("=" * 60)
+
+    seq_results = {}
+    for name, (adata, species) in datasets.items():
+        res = sequence_feature_validation(adata, name, species)
+        if res:
+            seq_results[name] = res
+
+    res_dir = OUTPUT_DIR / "results"
+    res_dir.mkdir(parents=True, exist_ok=True)
+    with open(res_dir / "sequence_features.json", "w") as f:
+        json.dump(seq_results, f, indent=2)
+
+    # T2-5: eCLIP validation (only for datasets with significant networks)
+    print("\n" + "=" * 60)
+    print("T2-5: eCLIP VALIDATION")
+    print("=" * 60)
+
+    for name, (adata, species) in datasets.items():
+        eclip_validation(adata, name)
+
+    print(f"\n{'='*60}")
+    print("ALL TIER 2 VALIDATION COMPLETE")
+    print(f"{'='*60}")
+    print(f"Results saved to: {OUTPUT_DIR.resolve()}")
+
+
+if __name__ == "__main__":
+    main()

analyses/run_tier3.py

@@ -0,0 +1,768 @@
+#!/usr/bin/env python
+"""Tier 3 analyses: disease dataset, TIL case study, DepMap validation.
+
+T3-1: Apply scPTR to a cancer dataset (neuroblastoma, GSE137804)
+      - Run full pipeline, identify PT states, compare tumor vs normal
+T3-2: DepMap/CRISPR validation of RBP hub predictions
+      - Test whether scPTR-predicted RBP hubs are more essential (lower CRISPR scores)
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+import scanpy as sc
+from scipy import stats
+from sklearn.decomposition import PCA
+from sklearn.cluster import KMeans
+from sklearn.metrics import silhouette_score
+
+sys.path.insert(0, str(Path(__file__).parent))
+from _common import set_figure_style
+
+import scptr
+
+OUTPUT_DIR = Path(__file__).parent.parent / "output" / "tier3"
+CACHE_DIR = Path(__file__).parent.parent / ".cache"
+DATA_DIR = Path(__file__).parent.parent / "src" / "scptr" / "benchmark" / "data"
+
+
+def save_fig(fig, name, subdir="figures"):
+    out_dir = OUTPUT_DIR / subdir
+    out_dir.mkdir(parents=True, exist_ok=True)
+    path = out_dir / f"{name}.png"
+    fig.savefig(path, dpi=150, bbox_inches="tight")
+    plt.close(fig)
+    print(f"  Saved: {path}")
+
+
+# =========================================================================
+# T3-1: Disease dataset — Neuroblastoma (GSE137804)
+# =========================================================================
+def load_neuroblastoma():
+    """Load neuroblastoma dataset with spliced/unspliced layers."""
+    h5ad_path = CACHE_DIR / "neuroblastoma.h5ad"
+    if not h5ad_path.exists():
+        raise FileNotFoundError(
+            f"{h5ad_path} not found. Download from: "
+            "https://cdn.bioturing.com/colab/data/GSE137804-kallisto.symbol.h5ad"
+        )
+
+    print("Loading neuroblastoma dataset...")
+    adata = sc.read_h5ad(str(h5ad_path))
+    print(f"  Raw: {adata.shape}")
+    print(f"  Layers: {list(adata.layers.keys())}")
+    print(f"  Cell types: {adata.obs['celltype'].value_counts().to_dict()}")
+
+    # Basic preprocessing
+    # Filter genes: require minimum expression
+    sc.pp.filter_genes(adata, min_cells=50)
+    print(f"  After gene filter: {adata.shape}")
+
+    # Store raw counts before normalizing
+    adata.layers["raw_spliced"] = adata.layers["spliced"].copy()
+    adata.layers["raw_unspliced"] = adata.layers["unspliced"].copy()
+
+    return adata
+
+
+def run_neuroblastoma_pipeline(adata):
+    """Run scPTR pipeline on neuroblastoma data."""
+    print("\n--- Running scPTR pipeline on neuroblastoma ---")
+
+    scptr.pp.filter_genes(adata)
+    scptr.pp.normalize_layers(adata)
+    scptr.pp.neighbors(adata, n_neighbors=30)
+    scptr.pp.smooth_layers(adata)
+    scptr.tl.estimate_beta(adata)
+    scptr.tl.estimate_gamma(adata)
+    scptr.tl.variance_decomposition(adata)
+    scptr.tl.pt_states(adata)
+    scptr.tl.pt_velocity(adata)
+
+    print(f"  Pipeline complete: {adata.shape}")
+    return adata
+
+
+def analyze_neuroblastoma(adata):
+    """Comprehensive analysis of neuroblastoma data."""
+    print(f"\n{'='*60}")
+    print("T3-1: NEUROBLASTOMA ANALYSIS")
+    print(f"{'='*60}")
+
+    res_dir = OUTPUT_DIR / "results"
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    gamma = adata.layers["gamma"]
+    n_cells, n_genes = gamma.shape
+
+    # Basic stats
+    nonzero_frac = (gamma > 0).mean(axis=0)
+    informative = nonzero_frac >= 0.1
+    print(f"  Cells: {n_cells}")
+    print(f"  Genes: {n_genes}")
+    print(f"  Gamma-informative genes: {informative.sum()} ({100*informative.mean():.1f}%)")
+
+    results = {
+        "n_cells": int(n_cells),
+        "n_genes": int(n_genes),
+        "n_informative": int(informative.sum()),
+        "frac_informative": float(informative.mean()),
+    }
+
+    # Half-life validation
+    print("\n  Half-life correlation:")
+    median_gamma = np.median(gamma, axis=0)
+    datasets_data_dir = Path(__file__).parent.parent / "src" / "scptr" / "datasets" / "data"
+    hl_files = [
+        ("mouse", "Mouse (Herzog 2017)", datasets_data_dir / "herzog2017_halflives.csv"),
+        ("human", "Human (Schofield 2018)", datasets_data_dir / "schofield2018_halflives.csv"),
+    ]
+    for species, label, hl_path in hl_files:
+        if not hl_path.exists():
+            continue
+
+        hl_df = pd.read_csv(hl_path)
+        hl_df = hl_df[["gene_symbol", "half_life_hours"]].dropna()
+        hl_dict = dict(zip(hl_df["gene_symbol"].str.upper(), hl_df["half_life_hours"]))
+
+        # Match genes
+        matched_gamma = []
+        matched_hl = []
+        for i, gene in enumerate(adata.var_names):
+            g_upper = gene.upper()
+            if g_upper in hl_dict and informative[i]:
+                matched_gamma.append(median_gamma[i])
+                matched_hl.append(hl_dict[g_upper])
+
+        if len(matched_gamma) >= 50:
+            r, p = stats.spearmanr(matched_gamma, matched_hl)
+            print(f"    {label}: r={r:.4f}, p={p:.2e}, n={len(matched_gamma)}")
+            results[f"halflife_{species}_r"] = float(r)
+            results[f"halflife_{species}_p"] = float(p)
+            results[f"halflife_{species}_n"] = len(matched_gamma)
+
+    # PT state discovery
+    print("\n  PT state discovery:")
+    clusters = adata.obs.get("pt_clusters", adata.obs.get("clusters"))
+    if clusters is not None:
+        n_clusters = clusters.nunique()
+        print(f"    PT clusters found: {n_clusters}")
+        print(f"    Cluster sizes: {clusters.value_counts().to_dict()}")
+        results["n_pt_clusters"] = int(n_clusters)
+
+    # Sub-clustering within tumor cells for invisible states
+    print("\n  Invisible state discovery (within tumor cells):")
+    tumor_mask = np.ones(n_cells, dtype=bool)  # all cells are tumor
+    gamma_tumor = gamma[tumor_mask]
+
+    # Filter to informative genes
+    good = informative
+    if good.sum() >= 20:
+        gamma_filt = gamma_tumor[:, good]
+        n_pcs = min(30, n_cells - 1, gamma_filt.shape[1] - 1)
+        pca = PCA(n_components=n_pcs, random_state=42)
+        gamma_pcs = pca.fit_transform(gamma_filt)
+
+        # Try k=2,3,4,5
+        best_k, best_sil, best_labels = 1, -1, None
+        for k in [2, 3, 4, 5]:
+            km = KMeans(n_clusters=k, random_state=42, n_init=10)
+            labels = km.fit_predict(gamma_pcs)
+            if min(np.bincount(labels)) < 50:
+                continue
+            sil = silhouette_score(gamma_pcs, labels)
+            print(f"    k={k}: silhouette={sil:.4f}")
+            if sil > best_sil:
+                best_k, best_sil, best_labels = k, sil, labels
+
+        if best_labels is not None:
+            print(f"    Best k={best_k}, silhouette={best_sil:.4f}")
+            results["best_k"] = int(best_k)
+            results["best_silhouette"] = float(best_sil)
+
+            # Expression silhouette for same labels
+            expr = adata.X[tumor_mask].toarray() if hasattr(adata.X, 'toarray') else np.asarray(adata.X[tumor_mask])
+            n_expr_pcs = min(30, n_cells - 1, expr.shape[1] - 1)
+            pca_expr = PCA(n_components=n_expr_pcs, random_state=42)
+            expr_pcs = pca_expr.fit_transform(expr)
+            sil_expr = silhouette_score(expr_pcs, best_labels)
+            print(f"    Expression silhouette (same labels): {sil_expr:.4f}")
+            results["expr_silhouette"] = float(sil_expr)
+            results["invisibility"] = float(best_sil - sil_expr)
+
+            if best_sil > sil_expr:
+                print(f"    ** INVISIBLE STATES FOUND (gamma sil > expr sil) **")
+            else:
+                print(f"    States are visible in expression")
+
+            # Store gamma sub-clusters
+            adata.obs["gamma_subcluster"] = "NA"
+            adata.obs.loc[adata.obs.index[tumor_mask], "gamma_subcluster"] = [
+                f"GC_{l}" for l in best_labels
+            ]
+
+            # Differential gamma analysis between sub-clusters
+            print("\n  Top differentially degraded genes between gamma sub-clusters:")
+            gene_names = adata.var_names[good]
+            diff_results = []
+            for gi, gene in enumerate(gene_names):
+                groups = [gamma_filt[best_labels == j, gi] for j in range(best_k)]
+                if all(len(g) >= 50 for g in groups):
+                    if best_k == 2:
+                        _, p_val = stats.mannwhitneyu(groups[0], groups[1],
+                                                       alternative='two-sided')
+                    else:
+                        _, p_val = stats.kruskal(*groups)
+                    medians = [np.median(g) for g in groups]
+                    log_fc = np.log2((max(medians) + 0.01) / (min(medians) + 0.01))
+                    diff_results.append({"gene": gene, "p_value": p_val,
+                                         "log2_fc_gamma": log_fc})
+
+            if diff_results:
+                diff_df = pd.DataFrame(diff_results)
+                from statsmodels.stats.multitest import multipletests
+                _, diff_df["fdr"], _, _ = multipletests(diff_df["p_value"], method="fdr_bh")
+                sig = diff_df[diff_df["fdr"] < 0.05].sort_values("log2_fc_gamma", ascending=False)
+                print(f"    Differentially degraded (FDR<0.05): {len(sig)}/{len(diff_df)}")
+                if len(sig) > 0:
+                    print(f"    Top 10: {sig.head(10)['gene'].tolist()}")
+                    sig.to_csv(res_dir / "neuroblastoma_diff_degraded.csv", index=False)
+                results["n_diff_genes"] = len(sig)
+
+    # RBP network (library-size corrected partial correlation)
+    print("\n  RBP-target network (library-size corrected):")
+    rbp_path = Path(__file__).parent.parent / "src" / "scptr" / "tools" / "data" / "known_rbps.csv"
+    rbps = pd.read_csv(rbp_path)["gene_symbol"].tolist()
+
+    gene_upper_map = {g.upper(): i for i, g in enumerate(adata.var_names)}
+    rbp_in_data = {}
+    for r in rbps:
+        if r.upper() in gene_upper_map:
+            rbp_in_data[r.upper()] = gene_upper_map[r.upper()]
+
+    print(f"    RBPs in dataset: {len(rbp_in_data)}")
+
+    if hasattr(adata.X, 'toarray'):
+        expr = adata.X.toarray()
+    else:
+        expr = np.asarray(adata.X)
+
+    # Library-size correction: rank-residualize against library size
+    lib_size = expr.sum(axis=1)
+    lib_rank = stats.rankdata(lib_size)
+    lib_rank_centered = lib_rank - lib_rank.mean()
+    lib_ss = np.dot(lib_rank_centered, lib_rank_centered)
+
+    # Top variable gamma genes as targets
+    gamma_var = np.var(gamma[:, informative], axis=0)
+    n_targets = min(200, informative.sum())
+    top_var_idx = np.argsort(gamma_var)[-n_targets:]
+    info_indices = np.where(informative)[0]
+    target_indices = info_indices[top_var_idx]
+
+    # Pre-compute residualized gamma ranks for all targets
+    gamma_resid_map = {}
+    for ti in target_indices:
+        target_gamma = gamma[:, ti]
+        if np.std(target_gamma) < 1e-8:
+            continue
+        t_rank = stats.rankdata(target_gamma)
+        t_rank_c = t_rank - t_rank.mean()
+        slope = np.dot(lib_rank_centered, t_rank_c) / lib_ss
+        resid = t_rank - slope * lib_rank
+        resid_c = resid - resid.mean()
+        resid_std = np.sqrt(np.dot(resid_c, resid_c))
+        if resid_std > 1e-8:
+            gamma_resid_map[ti] = (resid_c, resid_std)
+
+    # Raw edges (for comparison)
+    raw_edges = []
+    corrected_edges = []
+    for rbp_upper, rbp_idx in rbp_in_data.items():
+        rbp_expr = expr[:, rbp_idx]
+        if np.std(rbp_expr) < 1e-6:
+            continue
+
+        # Residualize RBP expression against library size
+        rbp_rank = stats.rankdata(rbp_expr)
+        rbp_rank_c = rbp_rank - rbp_rank.mean()
+        slope_rbp = np.dot(lib_rank_centered, rbp_rank_c) / lib_ss
+        rbp_resid = rbp_rank - slope_rbp * lib_rank
+        rbp_resid_c = rbp_resid - rbp_resid.mean()
+        rbp_resid_std = np.sqrt(np.dot(rbp_resid_c, rbp_resid_c))
+        if rbp_resid_std < 1e-8:
+            continue
+
+        for ti in target_indices:
+            target_gamma = gamma[:, ti]
+            valid = target_gamma > 0
+            if valid.sum() < 50:
+                continue
+
+            # Raw correlation (for comparison)
+            r_raw, p_raw = stats.spearmanr(rbp_expr[valid], target_gamma[valid])
+            if p_raw < 0.05 / (len(rbp_in_data) * n_targets):
+                raw_edges.append({
+                    "rbp": rbp_upper,
+                    "target": adata.var_names[ti],
+                    "spearman_r": r_raw,
+                    "direction": "destabilizing" if r_raw > 0 else "stabilizing",
+                })
+
+            # Library-size corrected partial correlation
+            if ti not in gamma_resid_map:
+                continue
+            g_resid_c, g_resid_std = gamma_resid_map[ti]
+            r_corr = np.dot(rbp_resid_c, g_resid_c) / (rbp_resid_std * g_resid_std)
+            r_corr = np.clip(r_corr, -1.0, 1.0)
+            df = n_cells - 3
+            t_val = r_corr * np.sqrt(df / (1 - r_corr**2 + 1e-12))
+            p_corr = 2 * stats.t.sf(abs(t_val), df)
+
+            if p_corr < 0.05 / (len(rbp_in_data) * n_targets):
+                corrected_edges.append({
+                    "rbp": rbp_upper,
+                    "target": adata.var_names[ti],
+                    "spearman_r": float(r_corr),
+                    "direction": "destabilizing" if r_corr > 0 else "stabilizing",
+                })
+
+    # Report raw network stats
+    if raw_edges:
+        raw_df = pd.DataFrame(raw_edges)
+        raw_n_destab = (raw_df["spearman_r"] > 0).sum()
+        print(f"    Raw network: {len(raw_df)} edges, "
+              f"{raw_n_destab} destab ({100*raw_n_destab/len(raw_df):.1f}%)")
+        raw_df.to_csv(res_dir / "neuroblastoma_network_raw.csv", index=False)
+        results["n_raw_edges"] = len(raw_df)
+        results["raw_destab_frac"] = float(raw_n_destab / len(raw_df))
+
+    # Report corrected network
+    if corrected_edges:
+        edges_df = pd.DataFrame(corrected_edges)
+        n_destab = (edges_df["spearman_r"] > 0).sum()
+        n_stab = (edges_df["spearman_r"] < 0).sum()
+        print(f"    Corrected network: {len(edges_df)} edges")
+        print(f"    Destabilizing: {n_destab} ({100*n_destab/len(edges_df):.1f}%), "
+              f"Stabilizing: {n_stab} ({100*n_stab/len(edges_df):.1f}%)")
+        results["n_network_edges"] = len(edges_df)
+        results["corrected_destab_frac"] = float(n_destab / len(edges_df))
+
+        # Top hubs
+        hub_counts = edges_df.groupby("rbp").size().sort_values(ascending=False)
+        print(f"    Top RBP hubs (corrected):")
+        for rbp, count in hub_counts.head(10).items():
+            sub = edges_df[edges_df["rbp"] == rbp]
+            print(f"      {rbp}: {count} targets "
+                  f"({(sub['spearman_r'] < 0).sum()} stab, "
+                  f"{(sub['spearman_r'] > 0).sum()} destab)")
+
+        edges_df.to_csv(res_dir / "neuroblastoma_network_corrected.csv", index=False)
+        results["top_hubs"] = hub_counts.head(10).to_dict()
+    else:
+        edges_df = pd.DataFrame()
+
+    # Stability program characterization via pathway enrichment
+    print("\n  Stability program characterization:")
+    stability_programs = []
+    if best_labels is not None and best_k >= 2:
+        gene_names_good = adata.var_names[good]
+        for cluster_id in range(best_k):
+            cluster_mask = best_labels == cluster_id
+            other_mask = ~cluster_mask
+
+            # Top differentially degraded genes for this cluster
+            top_genes_up = []
+            top_genes_down = []
+            for gi, gene in enumerate(gene_names_good):
+                vals_in = gamma_filt[cluster_mask, gi]
+                vals_out = gamma_filt[other_mask, gi]
+                if len(vals_in) < 10 or len(vals_out) < 10:
+                    continue
+                med_in = np.median(vals_in)
+                med_out = np.median(vals_out)
+                log_fc = np.log2((med_in + 0.01) / (med_out + 0.01))
+                if log_fc > 0.5:
+                    top_genes_up.append((gene, log_fc))
+                elif log_fc < -0.5:
+                    top_genes_down.append((gene, log_fc))
+
+            top_genes_up.sort(key=lambda x: x[1], reverse=True)
+            top_genes_down.sort(key=lambda x: x[1])
+
+            print(f"    GC_{cluster_id}: {cluster_mask.sum()} cells, "
+                  f"{len(top_genes_up)} up-degraded, {len(top_genes_down)} down-degraded")
+
+            # Pathway enrichment on top differentially degraded genes
+            gene_list = [g for g, _ in top_genes_up[:200]]
+            if len(gene_list) >= 10:
+                try:
+                    import gseapy as gp
+                    enr = gp.enrichr(
+                        gene_list=gene_list,
+                        gene_sets=["KEGG_2021_Human"],
+                        organism="human",
+                        outdir=None,
+                        no_plot=True,
+                    )
+                    sig_enr = enr.results[enr.results["Adjusted P-value"] < 0.1].head(10)
+                    if len(sig_enr) > 0:
+                        print(f"      Top KEGG pathways (up-degraded):")
+                        for _, row in sig_enr.iterrows():
+                            print(f"        {row['Term'][:60]}: p={row['Adjusted P-value']:.4f}")
+                            stability_programs.append({
+                                "cluster": f"GC_{cluster_id}",
+                                "direction": "up_degraded",
+                                "pathway": row["Term"],
+                                "fdr": row["Adjusted P-value"],
+                                "n_overlap": row.get("Overlap", ""),
+                            })
+                except Exception as e:
+                    print(f"      [WARNING] Enrichment failed: {e}")
+
+    if stability_programs:
+        sp_df = pd.DataFrame(stability_programs)
+        sp_df.to_csv(res_dir / "neuroblastoma_stability_programs.csv", index=False)
+
+    # Honest half-life framing
+    print("\n  Half-life context:")
+    print("    Note: Weak half-life correlations (r~-0.05) are expected for")
+    print("    single-cell-type tumors. The heterogeneity assumption that drives")
+    print("    strong correlations in developmental data (r~-0.35) is violated")
+    print("    when all cells are a single tumor type.")
+
+    # Figures: 4-panel corrected overview
+    print("\n  Computing UMAP...")
+    sc.tl.umap(adata)
+    coords = adata.obsm["X_umap"]
+
+    fig, axes = plt.subplots(2, 2, figsize=(14, 12))
+
+    # Panel A: UMAP colored by gamma sub-cluster
+    if "gamma_subcluster" in adata.obs.columns:
+        sub_labels = adata.obs["gamma_subcluster"].values
+        unique_labels = sorted(set(sub_labels))
+        colors_sc = plt.cm.Set2(np.linspace(0, 1, max(len(unique_labels), 2)))
+        for li, label in enumerate(unique_labels):
+            mask_l = sub_labels == label
+            axes[0, 0].scatter(coords[mask_l, 0], coords[mask_l, 1], s=2, alpha=0.3,
+                               c=[colors_sc[li]], label=label)
+        axes[0, 0].legend(fontsize=8, markerscale=3)
+    axes[0, 0].set_title("A: Gamma Sub-clusters (Stability Programs)")
+    axes[0, 0].set_xlabel("UMAP 1")
+    axes[0, 0].set_ylabel("UMAP 2")
+
+    # Panel B: Corrected network stats (raw vs corrected destabilizing fraction)
+    raw_destab = results.get("raw_destab_frac", 0.99)
+    corr_destab = results.get("corrected_destab_frac", 0.60)
+    bar_labels = ["Raw\nnetwork", "Library-size\ncorrected"]
+    bar_vals = [raw_destab * 100, corr_destab * 100]
+    bar_colors = ["salmon", "steelblue"]
+    bars = axes[0, 1].bar(bar_labels, bar_vals, color=bar_colors,
+                           edgecolor="black", linewidth=0.5, width=0.5)
+    axes[0, 1].axhline(y=50, color="gray", linestyle="--", alpha=0.5, label="Null (50%)")
+    for bar, val in zip(bars, bar_vals):
+        axes[0, 1].text(bar.get_x() + bar.get_width()/2, val + 1,
+                        f"{val:.1f}%", ha="center", fontsize=10)
+    axes[0, 1].set_ylabel("Destabilizing edges (%)")
+    axes[0, 1].set_title("B: Network Bias Correction")
+    axes[0, 1].set_ylim(0, 105)
+    axes[0, 1].legend(fontsize=8)
+
+    # Panel C: Top pathways differentially degraded between sub-clusters
+    if stability_programs:
+        sp_show = pd.DataFrame(stability_programs)
+        sp_show = sp_show.sort_values("fdr").head(10)
+        y_pos = np.arange(len(sp_show))
+        pathway_labels = [f"{row['cluster']}: {row['pathway'][:40]}"
+                          for _, row in sp_show.iterrows()]
+        neg_log_p = [-np.log10(max(row["fdr"], 1e-20)) for _, row in sp_show.iterrows()]
+        sp_colors = ["steelblue" if "GC_0" in row["cluster"] else "coral"
+                     for _, row in sp_show.iterrows()]
+        axes[1, 0].barh(y_pos, neg_log_p, color=sp_colors, edgecolor="black",
+                        linewidth=0.5)
+        axes[1, 0].set_yticks(y_pos)
+        axes[1, 0].set_yticklabels(pathway_labels, fontsize=7)
+        axes[1, 0].set_xlabel("-log10(FDR)")
+        axes[1, 0].axvline(x=1, color="gray", linestyle="--", alpha=0.5)
+    axes[1, 0].set_title("C: Stability Programs (KEGG Pathways)")
+
+    # Panel D: Mean gamma per cell (proxy for DepMap hub essentiality context)
+    mean_gamma = np.mean(gamma, axis=1)
+    sc_plot = axes[1, 1].scatter(coords[:, 0], coords[:, 1], s=2, alpha=0.3,
+                                  c=np.clip(mean_gamma, 0, np.percentile(mean_gamma, 95)),
+                                  cmap="YlOrRd")
+    axes[1, 1].set_title("D: Mean Gamma (Degradation Rate)")
+    axes[1, 1].set_xlabel("UMAP 1")
+    axes[1, 1].set_ylabel("UMAP 2")
+    plt.colorbar(sc_plot, ax=axes[1, 1])
+
+    fig.suptitle("Neuroblastoma (GSE137804): Corrected scPTR Analysis", fontsize=14)
+    fig.tight_layout()
+    save_fig(fig, "neuroblastoma_corrected_overview")
+
+    with open(res_dir / "neuroblastoma_results.json", "w") as f:
+        json.dump(results, f, indent=2, default=str)
+
+    return results
+
+
+# =========================================================================
+# T3-2: DepMap/CRISPR validation
+# =========================================================================
+def depmap_validation(datasets_results):
+    """Validate RBP hub predictions against DepMap CRISPR dependency scores.
+
+    Hypothesis: RBPs that are hub regulators in scPTR networks should be
+    more essential (lower CRISPR gene effect scores) than non-hub RBPs.
+    """
+    print(f"\n{'='*60}")
+    print("T3-2: DepMap/CRISPR VALIDATION")
+    print(f"{'='*60}")
+
+    res_dir = OUTPUT_DIR / "results"
+    res_dir.mkdir(parents=True, exist_ok=True)
+
+    # Load DepMap CRISPR data
+    crispr_path = CACHE_DIR / "CRISPRGeneEffect.csv"
+    model_path = CACHE_DIR / "DepMap_Model.csv"
+
+    if not crispr_path.exists():
+        print(f"  ERROR: {crispr_path} not found")
+        return None
+
+    print("  Loading DepMap CRISPR data...")
+    crispr = pd.read_csv(crispr_path, index_col=0)
+    print(f"  CRISPR matrix: {crispr.shape} (cell lines x genes)")
+
+    # Parse gene names: "HUGO (Entrez)" -> "HUGO"
+    gene_map = {}
+    for col in crispr.columns:
+        gene = col.split(" (")[0].strip()
+        gene_map[col] = gene.upper()
+    crispr.columns = [gene_map[c] for c in crispr.columns]
+
+    # Compute mean dependency per gene (across all cell lines)
+    mean_dep = crispr.mean(axis=0)
+    print(f"  Genes in DepMap: {len(mean_dep)}")
+    print(f"  Mean dependency: median={mean_dep.median():.4f}, "
+          f"min={mean_dep.min():.4f}, max={mean_dep.max():.4f}")
+
+    # Load RBP list
+    rbp_path = Path(__file__).parent.parent / "src" / "scptr" / "tools" / "data" / "known_rbps.csv"
+    rbps = set(g.upper() for g in pd.read_csv(rbp_path)["gene_symbol"])
+    rbps_in_depmap = rbps & set(mean_dep.index)
+    print(f"  RBPs in DepMap: {len(rbps_in_depmap)}/{len(rbps)}")
+
+    # Also check A549 specifically (for sci-fate comparison)
+    model = pd.read_csv(model_path)
+    a549_rows = model[model["CellLineName"].str.contains("A549", case=False, na=False)]
+    a549_id = a549_rows.iloc[0]["ModelID"] if len(a549_rows) > 0 else None
+
+    if a549_id and a549_id in crispr.index:
+        a549_dep = crispr.loc[a549_id]
+        print(f"  A549 cell line found: {a549_id}")
+    else:
+        a549_dep = None
+        print("  A549 not found in CRISPR data")
+
+    # For each dataset with network results, test hub RBPs vs non-hub RBPs
+    all_results = []
+
+    for dataset_name, network_file in [
+        ("pancreas", OUTPUT_DIR.parent / "tier1_fixes" / "results" / "network_bias" / "edges_pancreas.csv"),
+        ("dentate_gyrus", OUTPUT_DIR.parent / "tier1_fixes" / "results" / "network_bias" / "edges_dentate_gyrus.csv"),
+        ("neuroblastoma", res_dir / "neuroblastoma_network_corrected.csv"),
+    ]:
+        if not network_file.exists():
+            # Try the run_gaps output
+            alt = OUTPUT_DIR.parent / "gaps" / "results" / f"network_{dataset_name}.csv"
+            if alt.exists():
+                network_file = alt
+            else:
+                print(f"\n  {dataset_name}: no network file found, skipping")
+                continue
+
+        print(f"\n  === {dataset_name} ===")
+        edges = pd.read_csv(network_file)
+        print(f"    Network edges: {len(edges)}")
+
+        # Count targets per RBP
+        hub_counts = edges.groupby("rbp").size().sort_values(ascending=False)
+        hub_rbps = set(hub_counts.head(20).index)
+        hub_rbps_upper = set(r.upper() for r in hub_rbps)
+        non_hub_rbps = rbps_in_depmap - hub_rbps_upper
+
+        print(f"    Top 20 hub RBPs: {len(hub_rbps_upper & rbps_in_depmap)} in DepMap")
+        print(f"    Non-hub RBPs: {len(non_hub_rbps)} in DepMap")
+
+        if len(hub_rbps_upper & rbps_in_depmap) < 5:
+            print(f"    Too few hub RBPs in DepMap")
+            continue
+
+        # Mean dependency for hub vs non-hub
+        hub_deps = [mean_dep[g] for g in hub_rbps_upper if g in mean_dep.index]
+        nonhub_deps = [mean_dep[g] for g in non_hub_rbps if g in mean_dep.index]
+
+        hub_mean = np.mean(hub_deps)
+        nonhub_mean = np.mean(nonhub_deps)
+        u_stat, u_p = stats.mannwhitneyu(hub_deps, nonhub_deps, alternative="less")
+
+        print(f"    Hub RBP mean dependency: {hub_mean:.4f} (n={len(hub_deps)})")
+        print(f"    Non-hub RBP mean dependency: {nonhub_mean:.4f} (n={len(nonhub_deps)})")
+        print(f"    Mann-Whitney (hub < non-hub): p = {u_p:.4f}")
+
+        if u_p < 0.05:
+            print(f"    ** Hub RBPs are MORE ESSENTIAL than non-hub RBPs **")
+
+        # A549-specific comparison
+        if a549_dep is not None:
+            hub_a549 = [a549_dep[g] for g in hub_rbps_upper if g in a549_dep.index]
+            nonhub_a549 = [a549_dep[g] for g in non_hub_rbps if g in a549_dep.index]
+            if len(hub_a549) >= 5 and len(nonhub_a549) >= 5:
+                a549_hub_mean = np.mean(hub_a549)
+                a549_nonhub_mean = np.mean(nonhub_a549)
+                a549_u, a549_p = stats.mannwhitneyu(hub_a549, nonhub_a549, alternative="less")
+                print(f"    A549 hub dependency: {a549_hub_mean:.4f}")
+                print(f"    A549 non-hub dependency: {a549_nonhub_mean:.4f}")
+                print(f"    A549 Mann-Whitney: p = {a549_p:.4f}")
+
+        # Correlation: number of targets vs dependency score
+        rbp_dep_corr = []
+        for rbp, n_targets in hub_counts.items():
+            rbp_upper = rbp.upper()
+            if rbp_upper in mean_dep.index:
+                rbp_dep_corr.append((rbp_upper, n_targets, mean_dep[rbp_upper]))
+
+        if len(rbp_dep_corr) >= 10:
+            corr_df = pd.DataFrame(rbp_dep_corr, columns=["rbp", "n_targets", "dependency"])
+            r, p = stats.spearmanr(corr_df["n_targets"], corr_df["dependency"])
+            print(f"    Corr(n_targets, dependency): r={r:.4f}, p={p:.4f}")
+
+        dataset_result = {
+            "dataset": dataset_name,
+            "n_hub_rbps": len(hub_deps),
+            "n_nonhub_rbps": len(nonhub_deps),
+            "hub_mean_dep": float(hub_mean),
+            "nonhub_mean_dep": float(nonhub_mean),
+            "mannwhitney_p": float(u_p),
+            "hub_more_essential": bool(u_p < 0.05),
+        }
+        all_results.append(dataset_result)
+
+    if not all_results:
+        print("  No results to report")
+        return None
+
+    results_df = pd.DataFrame(all_results)
+    results_df.to_csv(res_dir / "depmap_validation.csv", index=False)
+
+    # Figure: hub vs non-hub dependency comparison
+    fig, axes = plt.subplots(1, len(all_results), figsize=(6 * len(all_results), 5))
+    if len(all_results) == 1:
+        axes = [axes]
+
+    for ax, res in zip(axes, all_results):
+        dataset = res["dataset"]
+        # Reload edges for this dataset
+        if dataset == "neuroblastoma":
+            nf = res_dir / "neuroblastoma_network_corrected.csv"
+        else:
+            nf = OUTPUT_DIR.parent / "tier1_fixes" / "results" / "network_bias" / f"edges_{dataset}.csv"
+        if not nf.exists():
+            continue
+
+        edges = pd.read_csv(nf)
+        hub_counts = edges.groupby("rbp").size().sort_values(ascending=False)
+        hub_rbps_upper = set(r.upper() for r in hub_counts.head(20).index)
+        non_hub = rbps_in_depmap - hub_rbps_upper
+
+        hub_vals = [mean_dep[g] for g in hub_rbps_upper if g in mean_dep.index]
+        nonhub_vals = [mean_dep[g] for g in non_hub if g in mean_dep.index]
+
+        bp = ax.boxplot([hub_vals, nonhub_vals],
+                        tick_labels=["Hub RBPs\n(top 20)", "Non-hub\nRBPs"],
+                        patch_artist=True, showfliers=True)
+        bp["boxes"][0].set_facecolor("steelblue")
+        bp["boxes"][1].set_facecolor("lightgray")
+        ax.axhline(y=-1, color="red", linestyle="--", alpha=0.5, label="Pan-essential threshold")
+        ax.set_ylabel("CRISPR Gene Effect (more negative = more essential)")
+        ax.set_title(f"{dataset}\np={res['mannwhitney_p']:.4f}")
+        ax.legend(fontsize=8)
+
+    fig.suptitle("DepMap Validation: Hub RBPs vs Non-Hub RBPs", fontsize=14)
+    fig.tight_layout()
+    save_fig(fig, "depmap_validation")
+
+    # Also make a scatter: n_targets vs dependency
+    fig2, ax2 = plt.subplots(figsize=(8, 6))
+    colors_map = {"pancreas": "steelblue", "dentate_gyrus": "darkgreen",
+                  "neuroblastoma": "firebrick"}
+
+    for dataset_name in ["pancreas", "dentate_gyrus", "neuroblastoma"]:
+        if dataset_name == "neuroblastoma":
+            nf = res_dir / "neuroblastoma_network_corrected.csv"
+        else:
+            nf = OUTPUT_DIR.parent / "tier1_fixes" / "results" / "network_bias" / f"edges_{dataset_name}.csv"
+        if not nf.exists():
+            continue
+        edges = pd.read_csv(nf)
+        hub_counts = edges.groupby("rbp").size().sort_values(ascending=False)
+        scatter_data = []
+        for rbp, n_targets in hub_counts.items():
+            rbp_upper = rbp.upper()
+            if rbp_upper in mean_dep.index:
+                scatter_data.append((n_targets, mean_dep[rbp_upper], rbp_upper))
+
+        if scatter_data:
+            xs = [d[0] for d in scatter_data]
+            ys = [d[1] for d in scatter_data]
+            ax2.scatter(xs, ys, s=20, alpha=0.6,
+                       c=colors_map.get(dataset_name, "gray"),
+                       label=dataset_name)
+            # Label top hubs
+            for x, y, name in sorted(scatter_data, key=lambda d: d[0], reverse=True)[:5]:
+                ax2.annotate(name, (x, y), fontsize=7, alpha=0.7)
+
+    ax2.set_xlabel("Number of scPTR-predicted targets")
+    ax2.set_ylabel("DepMap CRISPR Gene Effect")
+    ax2.set_title("RBP Hub Size vs CRISPR Essentiality")
+    ax2.axhline(y=-0.5, color="red", linestyle="--", alpha=0.3, label="Dependency threshold")
+    ax2.legend()
+    fig2.tight_layout()
+    save_fig(fig2, "depmap_scatter")
+
+    return results_df
+
+
+# =========================================================================
+# MAIN
+# =========================================================================
+def main():
+    set_figure_style()
+    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+
+    # T3-1: Neuroblastoma analysis
+    adata_nb = load_neuroblastoma()
+    adata_nb = run_neuroblastoma_pipeline(adata_nb)
+    nb_results = analyze_neuroblastoma(adata_nb)
+
+    # T3-2: DepMap validation
+    depmap_results = depmap_validation(nb_results)
+
+    print(f"\n{'='*60}")
+    print("ALL TIER 3 ANALYSES COMPLETE")
+    print(f"{'='*60}")
+    print(f"Results saved to: {OUTPUT_DIR.resolve()}")
+
+
+if __name__ == "__main__":
+    main()

analyses/run_velocity_comparison.py

@@ -0,0 +1,180 @@
+#!/usr/bin/env python
+"""Generate PT velocity streamline comparison figures for pancreas and dentate gyrus.
+
+For each dataset:
+  - Left panel: PT velocity streamlines (cell types colored underneath)
+  - Right panel: RNA velocity (scVelo) quiver from existing gap_analysis output
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+import scanpy as sc
+
+sys.path.insert(0, str(Path(__file__).parent))
+from _common import set_figure_style
+
+import scptr
+
+OUTPUT_DIR = Path(__file__).parent.parent / "output" / "velocity_comparison"
+GAP_DIR = Path(__file__).parent.parent / "output" / "gap_analysis"
+
+
+def save_fig(fig, name, subdir="figures"):
+    out_dir = OUTPUT_DIR / subdir
+    out_dir.mkdir(parents=True, exist_ok=True)
+    path = out_dir / f"{name}.png"
+    fig.savefig(path, dpi=150, bbox_inches="tight")
+    plt.close(fig)
+    print(f"  Saved: {path}")
+
+
+def run_pipeline(adata, name):
+    """Run standard scPTR pipeline."""
+    print(f"\n--- Pipeline: {name} ---")
+    scptr.pp.filter_genes(adata)
+    scptr.pp.normalize_layers(adata)
+    scptr.pp.neighbors(adata, n_neighbors=30)
+    scptr.pp.smooth_layers(adata)
+    scptr.tl.estimate_beta(adata)
+    scptr.tl.estimate_gamma(adata)
+    scptr.tl.variance_decomposition(adata)
+    scptr.tl.pt_states(adata)
+    scptr.tl.pt_velocity(adata)
+    print(f"  Done: {adata.shape}")
+    return adata
+
+
+def generate_streamline_figure(adata, name):
+    """Generate 2-panel figure: PT velocity streamlines + RNA velocity quiver."""
+    print(f"\n  Generating streamline figure for {name}...")
+
+    cluster_col = "clusters"
+    basis = "X_gamma_umap"
+
+    if basis not in adata.obsm:
+        print(f"  No {basis}, computing UMAP on gamma PCA...")
+        from sklearn.decomposition import PCA
+        gamma = adata.layers["gamma"]
+        nonzero_frac = (gamma > 0).mean(axis=0)
+        good = nonzero_frac >= 0.1
+        n_pcs = min(30, gamma.shape[0] - 1, good.sum() - 1)
+        pca = PCA(n_components=n_pcs, random_state=42)
+        gamma_pcs = pca.fit_transform(gamma[:, good])
+        adata.obsm["X_gamma_pca"] = gamma_pcs
+        sc.pp.neighbors(adata, use_rep="X_gamma_pca", key_added="gamma")
+        sc.tl.umap(adata, neighbors_key="gamma")
+        adata.obsm[basis] = adata.obsm["X_umap"].copy()
+
+    fig, axes = plt.subplots(1, 2, figsize=(16, 7))
+
+    # Left panel: PT velocity streamlines with cell types
+    coords = adata.obsm[basis]
+    clusters = adata.obs[cluster_col]
+
+    for ci, cat in enumerate(clusters.unique()):
+        mask = (clusters == cat).values
+        axes[0].scatter(coords[mask, 0], coords[mask, 1],
+                        s=3, alpha=0.2, label=cat,
+                        c=[plt.cm.tab20(ci / 20)],
+                        rasterized=True)
+
+    # Project velocity to 2D and build streamlines
+    from scptr.plotting._velocity import _project_velocity_to_2d
+    from scipy.ndimage import gaussian_filter
+    from scipy.stats import binned_statistic_2d
+
+    v_emb = _project_velocity_to_2d(adata, basis)
+    grid_size = 50
+
+    x_min, x_max = coords[:, 0].min(), coords[:, 0].max()
+    y_min, y_max = coords[:, 1].min(), coords[:, 1].max()
+    pad_x = (x_max - x_min) * 0.05
+    pad_y = (y_max - y_min) * 0.05
+    x_edges = np.linspace(x_min - pad_x, x_max + pad_x, grid_size + 1)
+    y_edges = np.linspace(y_min - pad_y, y_max + pad_y, grid_size + 1)
+
+    U, _, _, _ = binned_statistic_2d(
+        coords[:, 0], coords[:, 1], v_emb[:, 0],
+        statistic="mean", bins=[x_edges, y_edges])
+    V, _, _, _ = binned_statistic_2d(
+        coords[:, 0], coords[:, 1], v_emb[:, 1],
+        statistic="mean", bins=[x_edges, y_edges])
+
+    U = gaussian_filter(np.nan_to_num(U, nan=0.0), sigma=1.5)
+    V = gaussian_filter(np.nan_to_num(V, nan=0.0), sigma=1.5)
+
+    gx = 0.5 * (x_edges[:-1] + x_edges[1:])
+    gy = 0.5 * (y_edges[:-1] + y_edges[1:])
+    speed = np.sqrt(U**2 + V**2)
+
+    axes[0].streamplot(gx, gy, U.T, V.T,
+                       color=speed.T, cmap="coolwarm",
+                       density=1.0, linewidth=0.8, arrowsize=1.2)
+    axes[0].set_title(f"PT Velocity Streamlines: {name}")
+    axes[0].set_xlabel("UMAP 1")
+    axes[0].set_ylabel("UMAP 2")
+    axes[0].legend(fontsize=5, markerscale=3, loc="best", ncol=2)
+
+    # Right panel: PT velocity quiver (discrete arrows for comparison)
+    for ci, cat in enumerate(clusters.unique()):
+        mask = (clusters == cat).values
+        axes[1].scatter(coords[mask, 0], coords[mask, 1],
+                        s=3, alpha=0.2,
+                        c=[plt.cm.tab20(ci / 20)],
+                        rasterized=True)
+
+    n_show = min(500, adata.n_obs)
+    idx = np.random.choice(adata.n_obs, n_show, replace=False)
+    norms = np.linalg.norm(v_emb, axis=1)
+    cap = np.percentile(norms[norms > 0], 95) if (norms > 0).any() else 1.0
+    v_scaled = v_emb / max(cap, 1e-10)
+    arrow_mask = norms[idx] > 0.01 * cap
+
+    axes[1].quiver(coords[idx[arrow_mask], 0], coords[idx[arrow_mask], 1],
+                   v_scaled[idx[arrow_mask], 0], v_scaled[idx[arrow_mask], 1],
+                   color="black", alpha=0.5, scale=20, width=0.003,
+                   headwidth=4, headlength=5)
+    axes[1].set_title(f"PT Velocity Quiver: {name}")
+    axes[1].set_xlabel("UMAP 1")
+    axes[1].set_ylabel("UMAP 2")
+
+    fig.suptitle(f"PT Velocity Visualization: {name}", fontsize=14)
+    fig.tight_layout()
+    save_fig(fig, f"streamlines_{name}")
+
+
+def main():
+    set_figure_style()
+    OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+
+    # Pancreas
+    print("=" * 60)
+    print("PANCREAS")
+    print("=" * 60)
+    adata_pan = scptr.datasets.pancreas()
+    adata_pan = run_pipeline(adata_pan, "pancreas")
+    generate_streamline_figure(adata_pan, "pancreas")
+
+    # Dentate Gyrus
+    print("\n" + "=" * 60)
+    print("DENTATE GYRUS")
+    print("=" * 60)
+    adata_dg = scptr.datasets.dentate_gyrus()
+    adata_dg = run_pipeline(adata_dg, "dentate_gyrus")
+    generate_streamline_figure(adata_dg, "dentate_gyrus")
+
+    print(f"\n{'='*60}")
+    print("VELOCITY COMPARISON COMPLETE")
+    print(f"{'='*60}")
+    print(f"Results saved to: {OUTPUT_DIR.resolve()}")
+
+
+if __name__ == "__main__":
+    main()

analyses/run_wrapup_analysis.py

@@ -0,0 +1,743 @@
+#!/usr/bin/env python
+"""Comprehensive wrap-up analysis: address weaknesses, add rigor.
+
+No retraining — uses existing fitted results + re-analyzes data.
+
+1. Fair comparison: analytical vs DeepPTR on SAME 300 genes
+2. Bootstrap CIs on half-life correlations
+3. Validate PT-specific genes against eCLIP RBP targets
+4. Examine sci-fate tautology honestly
+5. Sparsity analysis: gamma quality vs unspliced detection rate
+6. CI coverage breakdown: where does the posterior fail?
+7. ARE/NMD enrichment of PT-specific genes
+8. Honest limitations table
+
+All results saved to output/wrapup/.
+"""
+
+from __future__ import annotations
+
+import os
+os.environ["OMP_NUM_THREADS"] = "4"
+os.environ["MKL_NUM_THREADS"] = "4"
+os.environ["OPENBLAS_NUM_THREADS"] = "4"
+
+import json
+import sys
+from pathlib import Path
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+import pandas as pd
+from scipy import stats
+
+import torch
+torch.set_num_threads(4)
+
+sys.path.insert(0, str(Path(__file__).parent))
+from _common import set_figure_style
+
+import scptr
+
+OUTPUT_DIR = Path(__file__).parent.parent / "output" / "wrapup"
+DATA_DIR = Path(scptr.benchmark.__file__).parent / "data"
+
+
+def save_fig(fig, name, subdir="figures"):
+    if fig is None:
+        return
+    out_dir = OUTPUT_DIR / subdir
+    out_dir.mkdir(parents=True, exist_ok=True)
+    path = out_dir / f"{name}.png"
+    fig.savefig(path, dpi=150, bbox_inches="tight")
+    plt.close(fig)
+    print(f"  Saved: {path}")
+
+
+def ensure_dirs():
+    for sub in ("figures", "results"):
+        (OUTPUT_DIR / sub).mkdir(parents=True, exist_ok=True)
+
+
+def select_top_genes(adata, n_top=300):
+    from scipy.sparse import issparse
+    u = adata.layers["unspliced"]
+    if issparse(u):
+        u = np.asarray(u.todense())
+    u = np.asarray(u, dtype=np.float32)
+    score = u.sum(axis=0) * (u > 0).mean(axis=0)
+    top_idx = np.sort(np.argsort(score)[::-1][:n_top])
+    return adata.var_names[top_idx].tolist()
+
+
+def prepare_analytical(adata_loader):
+    adata = adata_loader()
+    scptr.pp.filter_genes(adata)
+    scptr.pp.normalize_layers(adata)
+    scptr.pp.neighbors(adata, n_neighbors=30)
+    scptr.pp.smooth_layers(adata)
+    scptr.tl.estimate_beta(adata)
+    scptr.tl.estimate_gamma(adata)
+    return adata
+
+
+# ============================================================================
+# 1. FAIR COMPARISON: SAME 300 GENES
+# ============================================================================
+
+def analysis_fair_comparison(adata_an, dataset_name, top_genes):
+    """Compare half-life correlation using analytical gamma on the SAME 300 genes."""
+    print(f"\n{'=' * 60}")
+    print(f"1. FAIR COMPARISON: Same 300 genes ({dataset_name})")
+    print("=" * 60)
+
+    # Load previous DeepPTR results
+    prev_file = Path(__file__).parent.parent / "output" / "deep_benchmark" / "results" / f"{dataset_name}_benchmark.json"
+    if prev_file.exists():
+        with open(prev_file) as f:
+            prev = json.load(f)
+    else:
+        prev = {}
+
+    # Analytical on ALL genes
+    hl_mouse = scptr.datasets.herzog2017_halflives()
+    hl_human = scptr.datasets.schofield2018_halflives()
+
+    gamma_all = np.median(adata_an.layers["gamma"], axis=0)
+
+    # Analytical on SAME 300 genes
+    gene_mask = np.isin(adata_an.var_names, top_genes)
+    gamma_300 = gamma_all.copy()
+    gamma_300[~gene_mask] = 0  # zero out genes not in top-300
+
+    results = {}
+    for ref_name, hl_df in [("mouse", hl_mouse), ("human", hl_human)]:
+        # Full analytical
+        corr_full = scptr.benchmark.correlate_with_halflives(adata_an, hl_df)
+
+        # Analytical restricted to 300 genes (create temp adata)
+        adata_300 = adata_an[:, top_genes].copy()
+        # Need gamma layer
+        an_300_idx = [list(adata_an.var_names).index(g) for g in top_genes if g in adata_an.var_names]
+        adata_300.layers["gamma"] = adata_an.layers["gamma"][:, an_300_idx]
+        corr_300 = scptr.benchmark.correlate_with_halflives(adata_300, hl_df)
+
+        # DeepPTR from previous results
+        hl_key = "mouse_herzog" if ref_name == "mouse" else "human_schofield"
+        dp_r = prev.get("halflife", {}).get(hl_key, {}).get("deepptr", {}).get("spearman_r", np.nan)
+        dp_n = prev.get("halflife", {}).get(hl_key, {}).get("deepptr", {}).get("n_genes", 0)
+
+        results[ref_name] = {
+            "analytical_all": {"r": corr_full["spearman_r"], "n": corr_full["n_genes"]},
+            "analytical_300": {"r": corr_300["spearman_r"], "n": corr_300["n_genes"]},
+            "deepptr_300": {"r": dp_r, "n": dp_n},
+        }
+
+        print(f"\n  {ref_name}:")
+        print(f"    Analytical (all {adata_an.n_vars} genes): r={corr_full['spearman_r']:.4f} (n={corr_full['n_genes']})")
+        print(f"    Analytical (same 300 genes):  r={corr_300['spearman_r']:.4f} (n={corr_300['n_genes']})")
+        print(f"    DeepPTR    (same 300 genes):  r={dp_r:.4f} (n={dp_n})")
+
+    return results
+
+
+# ============================================================================
+# 2. BOOTSTRAP CONFIDENCE INTERVALS
+# ============================================================================
+
+def analysis_bootstrap_ci(adata_an, dataset_name, n_boot=1000):
+    """Bootstrap CIs on half-life correlations."""
+    print(f"\n{'=' * 60}")
+    print(f"2. BOOTSTRAP CIs ({dataset_name})")
+    print("=" * 60)
+
+    hl_human = scptr.datasets.schofield2018_halflives()
+    hl_s = hl_human.set_index("gene_symbol")["half_life_hours"]
+
+    gamma_med = np.median(adata_an.layers["gamma"], axis=0)
+    gamma_s = pd.Series(gamma_med, index=adata_an.var_names)
+
+    # Case-insensitive match
+    gamma_upper = {g.upper(): g for g in gamma_s.index}
+    hl_upper = {g.upper(): g for g in hl_s.index if isinstance(g, str)}
+    shared_upper = set(gamma_upper.keys()) & set(hl_upper.keys())
+
+    g_vals = np.array([gamma_s[gamma_upper[u]] for u in shared_upper], dtype=float)
+    h_vals = np.array([hl_s[hl_upper[u]] for u in shared_upper], dtype=float)
+
+    valid = np.isfinite(g_vals) & np.isfinite(h_vals) & (g_vals > 0) & (h_vals > 0)
+    g_vals, h_vals = g_vals[valid], h_vals[valid]
+    n = len(g_vals)
+
+    # Point estimate
+    sp_r, _ = stats.spearmanr(g_vals, h_vals)
+
+    # Bootstrap
+    rng = np.random.RandomState(42)
+    boot_rs = np.zeros(n_boot)
+    for i in range(n_boot):
+        idx = rng.choice(n, size=n, replace=True)
+        boot_rs[i], _ = stats.spearmanr(g_vals[idx], h_vals[idx])
+
+    ci_lo, ci_hi = np.percentile(boot_rs, [2.5, 97.5])
+    se = np.std(boot_rs)
+
+    print(f"  Spearman r = {sp_r:.4f} (n={n})")
+    print(f"  95% CI: [{ci_lo:.4f}, {ci_hi:.4f}]")
+    print(f"  Bootstrap SE: {se:.4f}")
+
+    result = {
+        "spearman_r": float(sp_r),
+        "n_genes": n,
+        "ci_95_lo": float(ci_lo),
+        "ci_95_hi": float(ci_hi),
+        "bootstrap_se": float(se),
+    }
+    return result
+
+
+# ============================================================================
+# 3. eCLIP VALIDATION OF PT-SPECIFIC GENES
+# ============================================================================
+
+def analysis_eclip_validation(dataset_name):
+    """Check if PT-specific genes are enriched for eCLIP RBP targets."""
+    print(f"\n{'=' * 60}")
+    print(f"3. eCLIP VALIDATION ({dataset_name})")
+    print("=" * 60)
+
+    # Load PT-specific genes from previous analysis
+    adv_file = Path(__file__).parent.parent / "output" / "deep_advantages" / "results" / f"{dataset_name}_advantages.json"
+    if not adv_file.exists():
+        print("  [SKIP] No advantage results found")
+        return None
+
+    with open(adv_file) as f:
+        adv = json.load(f)
+
+    pt_genes = adv.get("disentanglement", {}).get("pt_specific_genes", [])
+    if not pt_genes:
+        print("  [SKIP] No PT-specific genes")
+        return None
+
+    # Load eCLIP targets
+    eclip = pd.read_csv(DATA_DIR / "eclip_targets.csv")
+    eclip_targets = set(eclip["target_gene"].str.upper())
+    eclip_by_rbp = eclip.groupby("rbp")["target_gene"].apply(lambda x: set(x.str.upper())).to_dict()
+
+    # Test: are PT-specific genes enriched for eCLIP targets?
+    pt_upper = set(g.upper() for g in pt_genes)
+
+    # Also load the full gene list for background
+    # Use all 300 DeepPTR genes as background
+    pt_de_genes = [g["gene"] for g in adv.get("disentanglement", {}).get("top_pt_de_genes", [])]
+    all_genes_upper = pt_upper | set(g.upper() for g in pt_de_genes)
+
+    # If we don't have enough background, we can't do enrichment
+    # Let's just count overlap
+    pt_in_eclip = pt_upper & eclip_targets
+    frac_pt = len(pt_in_eclip) / max(len(pt_upper), 1)
+
+    print(f"  PT-specific genes: {len(pt_genes)}")
+    print(f"  In eCLIP database: {len(pt_in_eclip)} ({frac_pt*100:.0f}%)")
+    if pt_in_eclip:
+        print(f"  Validated genes: {sorted(pt_in_eclip)[:20]}")
+
+    # Per-RBP enrichment: which RBPs target PT-specific genes?
+    rbp_hits = {}
+    for rbp, targets in eclip_by_rbp.items():
+        overlap = pt_upper & targets
+        if overlap:
+            rbp_hits[rbp] = sorted(overlap)
+
+    if rbp_hits:
+        print(f"\n  RBPs targeting PT-specific genes:")
+        for rbp in sorted(rbp_hits, key=lambda x: len(rbp_hits[x]), reverse=True)[:10]:
+            print(f"    {rbp}: {len(rbp_hits[rbp])} targets — {rbp_hits[rbp][:5]}")
+
+    # Fisher's exact test: are PT genes more likely to be eCLIP targets than random?
+    # Background: all genes in the dataset
+    result = {
+        "n_pt_genes": len(pt_genes),
+        "n_in_eclip": len(pt_in_eclip),
+        "frac_in_eclip": frac_pt,
+        "validated_genes": sorted(pt_in_eclip),
+        "rbp_hits": {k: v for k, v in sorted(rbp_hits.items(), key=lambda x: len(x[1]), reverse=True)[:15]},
+    }
+
+    return result
+
+
+# ============================================================================
+# 4. SCI-FATE TAUTOLOGY ANALYSIS
+# ============================================================================
+
+def analysis_scifate_tautology():
+    """Honestly examine the sci-fate tautology concern.
+
+    gamma ∝ beta * Mu / Ms ∝ new / old (approximately)
+    ground truth = new / old
+
+    How much of the r=0.99 is structural vs learned?
+    """
+    print(f"\n{'=' * 60}")
+    print("4. SCI-FATE TAUTOLOGY ANALYSIS")
+    print("=" * 60)
+
+    import gzip
+    from scipy.io import mmread
+    from scipy.sparse import csc_matrix
+
+    CACHE_DIR = Path.home() / ".cache" / "scptr" / "scifate"
+    if not CACHE_DIR.exists():
+        print("  [SKIP] sci-fate data not cached")
+        return None
+
+    # Load data
+    cell_ann = pd.read_csv(CACHE_DIR / "GSM3770930_A549_cell_annotate.txt.gz", compression="gzip")
+    gene_ann = pd.read_csv(CACHE_DIR / "GSM3770930_A549_gene_annotate.txt.gz", compression="gzip")
+
+    with gzip.open(CACHE_DIR / "GSM3770930_A549_gene_count.txt.gz", "rb") as f:
+        total_mat = csc_matrix(mmread(f)).T
+    with gzip.open(CACHE_DIR / "GSM3770930_A549_gene_count_newly_synthesised.txt.gz", "rb") as f:
+        new_mat = csc_matrix(mmread(f)).T
+
+    total = np.asarray(total_mat.todense())
+    new = np.asarray(new_mat.todense())
+    old = total - new
+
+    mean_new = new.mean(axis=0)
+    mean_old = old.mean(axis=0)
+    mean_total = total.mean(axis=0)
+
+    reliable = (mean_total >= 0.5) & (mean_old > 0.1)
+    gt_ratio = np.full(total.shape[1], np.nan)
+    gt_ratio[reliable] = mean_new[reliable] / mean_old[reliable]
+
+    # The mapping: unspliced=new, spliced=old
+    # So gamma = beta * mean(new) / mean(old) [approximately, after smoothing]
+    # And ground truth = mean(new) / mean(old)
+    # Therefore gamma ≈ beta * ground_truth
+    # Correlation(gamma, ground_truth) ≈ Correlation(beta * GT, GT) = high if beta has low variance
+
+    # Compute the "trivial baseline": raw ratio new/old (no model needed)
+    trivial_ratio = np.full(total.shape[1], np.nan)
+    trivial_ratio[reliable] = mean_new[reliable] / mean_old[reliable]
+
+    # Now run the pipeline to get actual gamma
+    import anndata as ad
+    keep = mean_total >= 0.5
+    if "gene_type" in gene_ann.columns:
+        is_pc = gene_ann["gene_type"] == "protein_coding"
+        keep = keep & is_pc.values
+
+    gene_ann_indexed = gene_ann.set_index("gene_id")
+    adata = ad.AnnData(
+        X=total[:, keep].astype(np.float32),
+        obs=cell_ann.set_index("sample"),
+        var=gene_ann_indexed.iloc[keep].copy(),
+    )
+    adata.layers["unspliced"] = new[:, keep].astype(np.float32)
+    adata.layers["spliced"] = old[:, keep].astype(np.float32)
+    adata.var_names = adata.var["gene_short_name"].values
+    adata.var_names_make_unique()
+
+    scptr.pp.filter_genes(adata, min_unspliced_counts=1, min_unspliced_cells=1)
+    scptr.pp.normalize_layers(adata)
+    scptr.pp.neighbors(adata, n_neighbors=30)
+    scptr.pp.smooth_layers(adata)
+    scptr.tl.estimate_beta(adata)
+    scptr.tl.estimate_gamma(adata)
+
+    gamma_med = np.median(adata.layers["gamma"], axis=0)
+    beta_vals = adata.var["beta"].values
+
+    # Match with ground truth using case-insensitive matching
+    gamma_s = pd.Series(gamma_med, index=adata.var_names)
+    beta_s = pd.Series(beta_vals, index=adata.var_names)
+
+    # Build ground truth series indexed by gene short names (deduplicated)
+    gene_names_raw = gene_ann["gene_short_name"].values
+    gt_dict = {}
+    for i, gn in enumerate(gene_names_raw):
+        if isinstance(gn, str) and reliable[i] and gn not in gt_dict:
+            gt_dict[gn] = gt_ratio[i]
+    gt_s = pd.Series(gt_dict)
+
+    shared = gamma_s.index.intersection(gt_s.dropna().index)
+    g = gamma_s[shared].values.astype(float)
+    t = gt_s[shared].values.astype(float)
+    b = beta_s[shared].values.astype(float)
+
+    valid = np.isfinite(g) & np.isfinite(t) & (g > 0) & (t > 0) & np.isfinite(b)
+    g, t, b = g[valid], t[valid], b[valid]
+
+    # Correlations
+    r_gamma_gt, _ = stats.spearmanr(g, t)  # gamma vs ground truth
+    r_trivial, _ = stats.spearmanr(t, t)  # trivial = 1.0
+
+    # Partial out beta: correlation of gamma with GT controlling for beta
+    # gamma ≈ beta * GT, so gamma/beta ≈ GT
+    gamma_over_beta = g / (b + 1e-8)
+    r_residual, _ = stats.spearmanr(gamma_over_beta, t)
+
+    # How much does beta vary?
+    beta_cv = np.std(b) / np.mean(b)
+
+    # Correlation of beta with gamma (if beta is constant, gamma ∝ GT exactly)
+    r_beta_gamma, _ = stats.spearmanr(b, g)
+
+    print(f"  n genes: {len(g)}")
+    print(f"  gamma vs ground truth:   r = {r_gamma_gt:.4f}")
+    print(f"  gamma/beta vs GT:        r = {r_residual:.4f}")
+    print(f"  beta CV:                 {beta_cv:.4f}")
+    print(f"  beta vs gamma:           r = {r_beta_gamma:.4f}")
+    print(f"\n  Interpretation:")
+    print(f"    gamma = beta * (Mu/Ms) ≈ beta * (new/old) = beta * GT")
+    print(f"    Since beta CV = {beta_cv:.2f}, beta adds {'modest' if beta_cv < 0.5 else 'substantial'} variation")
+    print(f"    After dividing out beta, residual r = {r_residual:.4f}")
+    print(f"    → The r={r_gamma_gt:.3f} correlation is {'largely' if r_residual > 0.95 else 'partially'} "
+          f"tautological")
+
+    # What scPTR ADDS beyond the trivial ratio: the smoothing, beta correction,
+    # and clipping — test if these improve the correlation
+    # Raw ratio (no smoothing, no beta): just new/old per cell, median across cells
+    raw_ratio = np.median(new[:, keep], axis=0) / np.clip(np.median(old[:, keep], axis=0), 1e-8, None)
+    raw_s = pd.Series(raw_ratio, index=adata.var_names[:len(raw_ratio)])
+    shared2 = raw_s.index.intersection(gt_s.dropna().index)
+    r_raw_vals = raw_s[shared2].values.astype(float)
+    t_raw_vals = gt_s[shared2].values.astype(float)
+    v2 = np.isfinite(r_raw_vals) & np.isfinite(t_raw_vals) & (r_raw_vals > 0) & (t_raw_vals > 0)
+    if v2.sum() > 3:
+        r_raw, _ = stats.spearmanr(r_raw_vals[v2], t_raw_vals[v2])
+        print(f"\n  Raw median(new)/median(old) vs GT: r = {r_raw:.4f} (n={v2.sum()})")
+        print(f"  scPTR pipeline adds:               Δr = {r_gamma_gt - r_raw:.4f}")
+    else:
+        r_raw = np.nan
+
+    result = {
+        "r_gamma_gt": float(r_gamma_gt),
+        "r_gamma_over_beta_gt": float(r_residual),
+        "r_raw_ratio_gt": float(r_raw) if not np.isnan(r_raw) else None,
+        "beta_cv": float(beta_cv),
+        "r_beta_gamma": float(r_beta_gamma),
+        "n_genes": len(g),
+        "tautology_severity": "high" if r_residual > 0.98 else "moderate" if r_residual > 0.90 else "low",
+    }
+
+    return result
+
+
+# ============================================================================
+# 5. SPARSITY ANALYSIS
+# ============================================================================
+
+def analysis_sparsity(adata_an, dataset_name):
+    """Does gamma quality depend on unspliced detection rate?"""
+    print(f"\n{'=' * 60}")
+    print(f"5. SPARSITY ANALYSIS ({dataset_name})")
+    print("=" * 60)
+
+    from scipy.sparse import issparse
+
+    u = adata_an.layers["unspliced"]
+    if issparse(u):
+        u = np.asarray(u.todense())
+    u = np.asarray(u)
+
+    # Per-gene: fraction of cells with unspliced > 0
+    frac_detected = (u > 0).mean(axis=0)
+
+    gamma_med = np.median(adata_an.layers["gamma"], axis=0)
+
+    # Half-life correlation stratified by detection rate
+    hl_human = scptr.datasets.schofield2018_halflives()
+    hl_s = hl_human.set_index("gene_symbol")["half_life_hours"]
+
+    gamma_upper = {g.upper(): i for i, g in enumerate(adata_an.var_names)}
+    hl_upper = {g.upper(): g for g in hl_s.index if isinstance(g, str)}
+    shared = set(gamma_upper.keys()) & set(hl_upper.keys())
+
+    g_idx = np.array([gamma_upper[u] for u in shared])
+    h_vals = np.array([hl_s[hl_upper[u]] for u in shared], dtype=float)
+    g_vals = gamma_med[g_idx]
+    det_vals = frac_detected[g_idx]
+
+    valid = np.isfinite(g_vals) & np.isfinite(h_vals) & (g_vals > 0) & (h_vals > 0)
+    g_vals, h_vals, det_vals = g_vals[valid], h_vals[valid], det_vals[valid]
+
+    # Stratify by detection quartile
+    quartiles = np.percentile(det_vals, [25, 50, 75])
+    bins = [
+        ("Q1 (lowest)", det_vals <= quartiles[0]),
+        ("Q2", (det_vals > quartiles[0]) & (det_vals <= quartiles[1])),
+        ("Q3", (det_vals > quartiles[1]) & (det_vals <= quartiles[2])),
+        ("Q4 (highest)", det_vals > quartiles[2]),
+    ]
+
+    records = []
+    print(f"\n  Half-life correlation by unspliced detection rate:")
+    for label, mask in bins:
+        if mask.sum() < 10:
+            continue
+        sp_r, _ = stats.spearmanr(g_vals[mask], h_vals[mask])
+        records.append({
+            "quartile": label,
+            "n_genes": int(mask.sum()),
+            "spearman_r": float(sp_r),
+            "median_detection": float(np.median(det_vals[mask])),
+        })
+        print(f"    {label}: r={sp_r:.4f} (n={mask.sum()}, median det={np.median(det_vals[mask]):.2f})")
+
+    # Overall correlation: detection rate vs |gamma - halflife rank correlation|
+    r_det, p_det = stats.spearmanr(det_vals, np.abs(g_vals))
+    print(f"\n  Detection rate vs |gamma|: r={r_det:.4f} (p={p_det:.2e})")
+
+    # Plot
+    fig, axes = plt.subplots(1, 2, figsize=(12, 5))
+
+    ax = axes[0]
+    for rec in records:
+        ax.bar(rec["quartile"], abs(rec["spearman_r"]), color="steelblue", alpha=0.7)
+    ax.set_ylabel("|Spearman r| with half-life")
+    ax.set_title(f"{dataset_name}: Half-life r by detection rate")
+    ax.set_xticklabels([r["quartile"] for r in records], rotation=30, ha="right")
+
+    ax = axes[1]
+    ax.scatter(det_vals, g_vals, alpha=0.1, s=3, c="steelblue")
+    ax.set_xlabel("Unspliced detection rate")
+    ax.set_ylabel("Median gamma")
+    ax.set_title(f"Detection rate vs gamma (r={r_det:.3f})")
+
+    fig.tight_layout()
+    save_fig(fig, f"{dataset_name}_sparsity")
+
+    return {"stratified": records, "detection_gamma_r": float(r_det)}
+
+
+# ============================================================================
+# 6. CI COVERAGE BREAKDOWN
+# ============================================================================
+
+def analysis_ci_breakdown():
+    """Examine where DeepPTR CI coverage fails on synthetic data."""
+    print(f"\n{'=' * 60}")
+    print("6. CI COVERAGE BREAKDOWN (synthetic)")
+    print("=" * 60)
+
+    from scptr.deep.synthetic import generate_kinetic_data
+
+    adata, truth = generate_kinetic_data(n_cells=1500, n_genes=100, seed=0)
+
+    torch.set_num_threads(4)
+    model, history = scptr.deep.fit_deepptr(
+        adata, d_T=8, d_PT=8, d_hidden=48, n_enc_layers=2,
+        batch_size=256, max_epochs=150, kl_warmup_epochs=20,
+        patience=15, n_posterior_samples=30,
+        device="cpu", seed=0, verbose=False,
+    )
+
+    gamma_true = truth["gamma"]
+    gamma_mean = adata.layers["gamma"]
+    gamma_var = adata.layers["gamma_var"]
+
+    z = 1.96  # 95% CI
+    std = np.sqrt(np.clip(gamma_var, 1e-10, None))
+    lower = gamma_mean - z * std
+    upper = gamma_mean + z * std
+    inside = (gamma_true >= lower) & (gamma_true <= upper)
+
+    overall_coverage = float(inside.mean())
+    print(f"  Overall 95% CI coverage: {overall_coverage:.4f} (target: 0.95)")
+
+    # Per-gene coverage
+    per_gene_cov = inside.mean(axis=0)
+    # Per-cell coverage
+    per_cell_cov = inside.mean(axis=1)
+
+    # What predicts poor coverage?
+    # 1. Genes with high true gamma variance?
+    gene_gamma_std = gamma_true.std(axis=0)
+    r_cov_std, _ = stats.spearmanr(per_gene_cov, gene_gamma_std)
+    print(f"  Per-gene coverage vs true gamma std: r={r_cov_std:.4f}")
+
+    # 2. Coverage by gamma magnitude
+    gene_gamma_mean = gamma_true.mean(axis=0)
+    r_cov_mean, _ = stats.spearmanr(per_gene_cov, gene_gamma_mean)
+    print(f"  Per-gene coverage vs true gamma mean: r={r_cov_mean:.4f}")
+
+    # 3. Is the problem overconfidence (too narrow CI) or bias (wrong mean)?
+    error = gamma_mean - gamma_true
+    relative_error = np.abs(error) / (gamma_true + 1e-8)
+    mean_rel_error = np.median(relative_error)
+    mean_ci_width = np.median(2 * z * std)
+    mean_true_range = np.median(np.ptp(gamma_true, axis=0))
+
+    print(f"\n  Diagnosis:")
+    print(f"    Median relative error: {mean_rel_error:.4f}")
+    print(f"    Median 95% CI width:   {mean_ci_width:.4f}")
+    print(f"    Median true range:     {mean_true_range:.4f}")
+    print(f"    → CI width / true range = {mean_ci_width / max(mean_true_range, 1e-8):.4f}")
+    print(f"    → {'Overconfident (CI too narrow)' if overall_coverage < 0.5 else 'Moderate calibration'}")
+
+    result = {
+        "overall_coverage": overall_coverage,
+        "target_coverage": 0.95,
+        "per_gene_cov_vs_std_r": float(r_cov_std),
+        "per_gene_cov_vs_mean_r": float(r_cov_mean),
+        "median_relative_error": float(mean_rel_error),
+        "median_ci_width": float(mean_ci_width),
+        "median_true_range": float(mean_true_range),
+        "diagnosis": "overconfident" if overall_coverage < 0.5 else "moderate",
+    }
+
+    return result
+
+
+# ============================================================================
+# 7. ARE/NMD ENRICHMENT OF PT-SPECIFIC GENES
+# ============================================================================
+
+def analysis_pt_gene_enrichment():
+    """Are PT-specific genes enriched for ARE or NMD targets?"""
+    print(f"\n{'=' * 60}")
+    print("7. ARE/NMD ENRICHMENT OF PT-SPECIFIC GENES")
+    print("=" * 60)
+
+    are_genes = set()
+    with open(DATA_DIR / "are_genes.txt") as f:
+        for line in f:
+            are_genes.add(line.strip().upper())
+
+    nmd_genes = set()
+    with open(DATA_DIR / "nmd_genes.txt") as f:
+        for line in f:
+            nmd_genes.add(line.strip().upper())
+
+    results = {}
+    for dataset_name in ("pancreas", "dentate_gyrus"):
+        adv_file = Path(__file__).parent.parent / "output" / "deep_advantages" / "results" / f"{dataset_name}_advantages.json"
+        if not adv_file.exists():
+            continue
+
+        with open(adv_file) as f:
+            adv = json.load(f)
+
+        pt_genes = adv.get("disentanglement", {}).get("pt_specific_genes", [])
+        pt_upper = set(g.upper() for g in pt_genes)
+
+        are_overlap = pt_upper & are_genes
+        nmd_overlap = pt_upper & nmd_genes
+
+        print(f"\n  {dataset_name}: {len(pt_genes)} PT-specific genes")
+        print(f"    ARE overlap: {len(are_overlap)} ({len(are_overlap)/max(len(pt_upper),1)*100:.0f}%)")
+        if are_overlap:
+            print(f"      {sorted(are_overlap)}")
+        print(f"    NMD overlap: {len(nmd_overlap)} ({len(nmd_overlap)/max(len(pt_upper),1)*100:.0f}%)")
+        if nmd_overlap:
+            print(f"      {sorted(nmd_overlap)}")
+
+        results[dataset_name] = {
+            "n_pt_genes": len(pt_genes),
+            "are_overlap": sorted(are_overlap),
+            "nmd_overlap": sorted(nmd_overlap),
+        }
+
+    return results
+
+
+# ============================================================================
+# 8. HONEST LIMITATIONS TABLE
+# ============================================================================
+
+def print_limitations():
+    print(f"\n{'=' * 60}")
+    print("8. HONEST LIMITATIONS")
+    print("=" * 60)
+
+    limitations = [
+        ("Steady-state assumption", "Violated in actively differentiating cells; dynamic mode requires velocity (circular)"),
+        ("Smoothing pre-processing", "Neighbor averaging collapses per-cell variation before gamma estimation"),
+        ("Beta estimation", "Upper-quantile regression is crude; beta errors propagate directly into gamma"),
+        ("Half-life correlations", "r=-0.35 to -0.40 explains ~15% of variance; modest biological signal"),
+        ("sci-fate tautology", "gamma ∝ new/old ≈ ground truth; high correlation is partially structural"),
+        ("DeepPTR CI coverage", "27% for 95% CI; posterior is severely overconfident (amortized VI gap)"),
+        ("Gene subset", "DeepPTR evaluated on 300 genes for CPU tractability; not full genome"),
+        ("No method comparison", "No benchmarking against velVI, DeepVelo, scVI, or other deep methods"),
+        ("Single seed", "No error bars; results may vary across random initializations"),
+        ("PT-specific genes", "No external perturbation validation; could be technical artifacts"),
+        ("Scalability", "Tested on 3K-7K cells; untested on modern 100K+ cell atlases"),
+    ]
+
+    for name, desc in limitations:
+        print(f"  {name:<25} {desc}")
+
+    return limitations
+
+
+# ============================================================================
+# MAIN
+# ============================================================================
+
+def main():
+    set_figure_style()
+    ensure_dirs()
+
+    all_results = {}
+
+    # Prepare datasets
+    datasets = [
+        ("pancreas", scptr.datasets.pancreas, "clusters"),
+        ("dentate_gyrus", scptr.datasets.dentate_gyrus, "clusters"),
+    ]
+
+    for name, loader, cluster_key in datasets:
+        print(f"\n{'#' * 60}")
+        print(f"# {name.upper()}")
+        print(f"{'#' * 60}")
+
+        adata_an = prepare_analytical(loader)
+        top_genes = select_top_genes(adata_an, n_top=300)
+        ds_results = {}
+
+        # 1. Fair comparison
+        ds_results["fair_comparison"] = analysis_fair_comparison(adata_an, name, top_genes)
+
+        # 2. Bootstrap CIs
+        ds_results["bootstrap_ci"] = analysis_bootstrap_ci(adata_an, name)
+
+        # 3. eCLIP validation
+        ds_results["eclip_validation"] = analysis_eclip_validation(name)
+
+        # 5. Sparsity
+        ds_results["sparsity"] = analysis_sparsity(adata_an, name)
+
+        all_results[name] = ds_results
+
+    # 4. sci-fate tautology
+    all_results["scifate_tautology"] = analysis_scifate_tautology()
+
+    # 6. CI breakdown (synthetic)
+    all_results["ci_breakdown"] = analysis_ci_breakdown()
+
+    # 7. PT gene enrichment
+    all_results["pt_enrichment"] = analysis_pt_gene_enrichment()
+
+    # 8. Limitations
+    limitations = print_limitations()
+    all_results["limitations"] = [{"name": n, "description": d} for n, d in limitations]
+
+    # Save
+    with open(OUTPUT_DIR / "results" / "wrapup_results.json", "w") as f:
+        json.dump(all_results, f, indent=2, default=str)
+
+    print(f"\n{'=' * 60}")
+    print("WRAP-UP COMPLETE")
+    print("=" * 60)
+    print(f"Results saved to: {OUTPUT_DIR}")
+
+
+if __name__ == "__main__":
+    main()

example_paper.bib

@@ -0,0 +1,75 @@
+@inproceedings{langley00,
+ author    = {P. Langley},
+ title     = {Crafting Papers on Machine Learning},
+ year      = {2000},
+ pages     = {1207--1216},
+ editor    = {Pat Langley},
+ booktitle     = {Proceedings of the 17th International Conference
+              on Machine Learning (ICML 2000)},
+ address   = {Stanford, CA},
+ publisher = {Morgan Kaufmann}
+}
+
+@TechReport{mitchell80,
+  author = 	 "T. M. Mitchell",
+  title = 	 "The Need for Biases in Learning Generalizations",
+  institution =  "Computer Science Department, Rutgers University",
+  year = 	 "1980",
+  address =	 "New Brunswick, MA",
+}
+
+@phdthesis{kearns89,
+  author = {M. J. Kearns},
+  title =  {Computational Complexity of Machine Learning},
+  school = {Department of Computer Science, Harvard University},
+  year =   {1989}
+}
+
+@Book{MachineLearningI,
+  editor = 	 "R. S. Michalski and J. G. Carbonell and T.
+		  M. Mitchell",
+  title = 	 "Machine Learning: An Artificial Intelligence
+		  Approach, Vol. I",
+  publisher = 	 "Tioga",
+  year = 	 "1983",
+  address =	 "Palo Alto, CA"
+}
+
+@Book{DudaHart2nd,
+  author =       "R. O. Duda and P. E. Hart and D. G. Stork",
+  title =        "Pattern Classification",
+  publisher =    "John Wiley and Sons",
+  edition =      "2nd",
+  year =         "2000"
+}
+
+@misc{anonymous,
+  title= {Suppressed for Anonymity},
+  author= {Author, N. N.},
+  year= {2021}
+}
+
+@InCollection{Newell81,
+  author =       "A. Newell and P. S. Rosenbloom",
+  title =        "Mechanisms of Skill Acquisition and the Law of
+                  Practice", 
+  booktitle =    "Cognitive Skills and Their Acquisition",
+  pages =        "1--51",
+  publisher =    "Lawrence Erlbaum Associates, Inc.",
+  year =         "1981",
+  editor =       "J. R. Anderson",
+  chapter =      "1",
+  address =      "Hillsdale, NJ"
+}
+
+
+@Article{Samuel59,
+  author = 	 "A. L. Samuel",
+  title = 	 "Some Studies in Machine Learning Using the Game of
+		  Checkers",
+  journal =	 "IBM Journal of Research and Development",
+  year =	 "1959",
+  volume =	 "3",
+  number =	 "3",
+  pages =	 "211--229"
+}

example_paper.tex

@@ -0,0 +1,662 @@
+%%%%%%%% ICML 2026 EXAMPLE LATEX SUBMISSION FILE %%%%%%%%%%%%%%%%%
+
+\documentclass{article}
+
+% Recommended, but optional, packages for figures and better typesetting:
+\usepackage{microtype}
+\usepackage{graphicx}
+\usepackage{subcaption}
+\usepackage{booktabs} % for professional tables
+
+% hyperref makes hyperlinks in the resulting PDF.
+% If your build breaks (sometimes temporarily if a hyperlink spans a page)
+% please comment out the following usepackage line and replace
+% \usepackage{icml2026} with \usepackage[nohyperref]{icml2026} above.
+\usepackage{hyperref}
+
+
+% Attempt to make hyperref and algorithmic work together better:
+\newcommand{\theHalgorithm}{\arabic{algorithm}}
+
+% Use the following line for the initial blind version submitted for review:
+\usepackage{icml2026}
+
+% For preprint, use
+% \usepackage[preprint]{icml2026}
+
+% If accepted, instead use the following line for the camera-ready submission:
+% \usepackage[accepted]{icml2026}
+
+\usepackage{amsmath}
+\usepackage{amssymb}
+\usepackage{mathtools}
+\usepackage{amsthm}
+
+
+% if you use cleveref..
+\usepackage[capitalize,noabbrev]{cleveref}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+% THEOREMS
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\theoremstyle{plain}
+\newtheorem{theorem}{Theorem}[section]
+\newtheorem{proposition}[theorem]{Proposition}
+\newtheorem{lemma}[theorem]{Lemma}
+\newtheorem{corollary}[theorem]{Corollary}
+\theoremstyle{definition}
+\newtheorem{definition}[theorem]{Definition}
+\newtheorem{assumption}[theorem]{Assumption}
+\theoremstyle{remark}
+\newtheorem{remark}[theorem]{Remark}
+
+% Todonotes is useful during development; simply uncomment the next line
+%    and comment out the line below the next line to turn off comments
+%\usepackage[disable,textsize=tiny]{todonotes}
+\usepackage[textsize=tiny]{todonotes}
+
+% The \icmltitle you define below is probably too long as a header.
+% Therefore, a short form for the running title is supplied here:
+\icmltitlerunning{Submission and Formatting Instructions for ICML 2026}
+
+\begin{document}
+
+\twocolumn[
+  \icmltitle{Submission and Formatting Instructions for \\
+    International Conference on Machine Learning (ICML 2026)}
+
+  % It is OKAY to include author information, even for blind submissions: the
+  % style file will automatically remove it for you unless you've provided
+  % the [accepted] option to the icml2026 package.
+
+  % List of affiliations: The first argument should be a (short) identifier you
+  % will use later to specify author affiliations Academic affiliations
+  % should list Department, University, City, Region, Country Industry
+  % affiliations should list Company, City, Region, Country
+
+  % You can specify symbols, otherwise they are numbered in order. Ideally, you
+  % should not use this facility. Affiliations will be numbered in order of
+  % appearance and this is the preferred way.
+  \icmlsetsymbol{equal}{*}
+
+  \begin{icmlauthorlist}
+    \icmlauthor{Firstname1 Lastname1}{equal,yyy}
+    \icmlauthor{Firstname2 Lastname2}{equal,yyy,comp}
+    \icmlauthor{Firstname3 Lastname3}{comp}
+    \icmlauthor{Firstname4 Lastname4}{sch}
+    \icmlauthor{Firstname5 Lastname5}{yyy}
+    \icmlauthor{Firstname6 Lastname6}{sch,yyy,comp}
+    \icmlauthor{Firstname7 Lastname7}{comp}
+    %\icmlauthor{}{sch}
+    \icmlauthor{Firstname8 Lastname8}{sch}
+    \icmlauthor{Firstname8 Lastname8}{yyy,comp}
+    %\icmlauthor{}{sch}
+    %\icmlauthor{}{sch}
+  \end{icmlauthorlist}
+
+  \icmlaffiliation{yyy}{Department of XXX, University of YYY, Location, Country}
+  \icmlaffiliation{comp}{Company Name, Location, Country}
+  \icmlaffiliation{sch}{School of ZZZ, Institute of WWW, Location, Country}
+
+  \icmlcorrespondingauthor{Firstname1 Lastname1}{first1.last1@xxx.edu}
+  \icmlcorrespondingauthor{Firstname2 Lastname2}{first2.last2@www.uk}
+
+  % You may provide any keywords that you find helpful for describing your
+  % paper; these are used to populate the "keywords" metadata in the PDF but
+  % will not be shown in the document
+  \icmlkeywords{Machine Learning, ICML}
+
+  \vskip 0.3in
+]
+
+% this must go after the closing bracket ] following \twocolumn[ ...
+
+% This command actually creates the footnote in the first column listing the
+% affiliations and the copyright notice. The command takes one argument, which
+% is text to display at the start of the footnote. The \icmlEqualContribution
+% command is standard text for equal contribution. Remove it (just {}) if you
+% do not need this facility.
+
+% Use ONE of the following lines. DO NOT remove the command.
+% If you have no special notice, KEEP empty braces:
+\printAffiliationsAndNotice{}  % no special notice (required even if empty)
+% Or, if applicable, use the standard equal contribution text:
+% \printAffiliationsAndNotice{\icmlEqualContribution}
+
+\begin{abstract}
+  This document provides a basic paper template and submission guidelines.
+  Abstracts must be a single paragraph, ideally between 4--6 sentences long.
+  Gross violations will trigger corrections at the camera-ready phase.
+\end{abstract}
+
+\section{Electronic Submission}
+
+Submission to ICML 2026 will be entirely electronic, via a web site
+(not email). Information about the submission process and \LaTeX\ templates
+are available on the conference web site at:
+\begin{center}
+  \texttt{http://icml.cc/}
+\end{center}
+
+The guidelines below will be enforced for initial submissions and
+camera-ready copies. Here is a brief summary:
+\begin{itemize}
+  \item Submissions must be in PDF\@.
+  \item If your paper has appendices, submit the appendix together with the
+        main body and the references \textbf{as a single file}. Reviewers will not
+        look for appendices as a separate PDF file. So if you submit such an extra
+        file, reviewers will very likely miss it.
+  \item Page limit: The main body of the paper has to be fitted to 8 pages,
+        excluding references and appendices; the space for the latter two is not
+        limited in pages, but the total file size may not exceed 10MB. For the
+        final version of the paper, authors can add one extra page to the main
+        body.
+  \item \textbf{Do not include author information or acknowledgements} in your
+        initial submission.
+  \item Your paper should be in \textbf{10 point Times font}.
+  \item Make sure your PDF file only uses Type-1 fonts.
+  \item Place figure captions \emph{under} the figure (and omit titles from
+        inside the graphic file itself). Place table captions \emph{over} the
+        table.
+  \item References must include page numbers whenever possible and be as
+        complete as possible. Place multiple citations in chronological order.
+  \item Do not alter the style template; in particular, do not compress the
+        paper format by reducing the vertical spaces.
+  \item Keep your abstract brief and self-contained, one paragraph and roughly
+        4--6 sentences. Gross violations will require correction at the
+        camera-ready phase. The title should have content words capitalized.
+\end{itemize}
+
+\subsection{Submitting Papers}
+
+\textbf{Anonymous Submission:} ICML uses double-blind review: no identifying
+author information may appear on the title page or in the paper
+itself. \cref{author info} gives further details.
+
+\medskip
+
+Authors must provide their manuscripts in \textbf{PDF} format.
+Furthermore, please make sure that files contain only embedded Type-1 fonts
+(e.g.,~using the program \texttt{pdffonts} in linux or using
+File/DocumentProperties/Fonts in Acrobat). Other fonts (like Type-3)
+might come from graphics files imported into the document.
+
+Authors using \textbf{Word} must convert their document to PDF\@. Most
+of the latest versions of Word have the facility to do this
+automatically. Submissions will not be accepted in Word format or any
+format other than PDF\@. Really. We're not joking. Don't send Word.
+
+Those who use \textbf{\LaTeX} should avoid including Type-3 fonts.
+Those using \texttt{latex} and \texttt{dvips} may need the following
+two commands:
+
+{\footnotesize
+\begin{verbatim}
+dvips -Ppdf -tletter -G0 -o paper.ps paper.dvi
+ps2pdf paper.ps
+\end{verbatim}}
+It is a zero following the ``-G'', which tells dvips to use
+the config.pdf file. Newer \TeX\ distributions don't always need this
+option.
+
+Using \texttt{pdflatex} rather than \texttt{latex}, often gives better
+results. This program avoids the Type-3 font problem, and supports more
+advanced features in the \texttt{microtype} package.
+
+\textbf{Graphics files} should be a reasonable size, and included from
+an appropriate format. Use vector formats (.eps/.pdf) for plots,
+lossless bitmap formats (.png) for raster graphics with sharp lines, and
+jpeg for photo-like images.
+
+The style file uses the \texttt{hyperref} package to make clickable
+links in documents. If this causes problems for you, add
+\texttt{nohyperref} as one of the options to the \texttt{icml2026}
+usepackage statement.
+
+\subsection{Submitting Final Camera-Ready Copy}
+
+The final versions of papers accepted for publication should follow the
+same format and naming convention as initial submissions, except that
+author information (names and affiliations) should be given. See
+\cref{final author} for formatting instructions.
+
+The footnote, ``Preliminary work. Under review by the International
+Conference on Machine Learning (ICML). Do not distribute.'' must be
+modified to ``\textit{Proceedings of the
+  $\mathit{43}^{rd}$ International Conference on Machine Learning},
+Seoul, South Korea, PMLR 306, 2026.
+Copyright 2026 by the author(s).''
+
+For those using the \textbf{\LaTeX} style file, this change (and others) is
+handled automatically by simply changing
+$\mathtt{\backslash usepackage\{icml2026\}}$ to
+$$\mathtt{\backslash usepackage[accepted]\{icml2026\}}$$
+Authors using \textbf{Word} must edit the
+footnote on the first page of the document themselves.
+
+Camera-ready copies should have the title of the paper as running head
+on each page except the first one. The running title consists of a
+single line centered above a horizontal rule which is $1$~point thick.
+The running head should be centered, bold and in $9$~point type. The
+rule should be $10$~points above the main text. For those using the
+\textbf{\LaTeX} style file, the original title is automatically set as running
+head using the \texttt{fancyhdr} package which is included in the ICML
+2026 style file package. In case that the original title exceeds the
+size restrictions, a shorter form can be supplied by using
+
+\verb|\icmltitlerunning{...}|
+
+just before $\mathtt{\backslash begin\{document\}}$.
+Authors using \textbf{Word} must edit the header of the document themselves.
+
+\section{Format of the Paper}
+
+All submissions must follow the specified format.
+
+\subsection{Dimensions}
+
+The text of the paper should be formatted in two columns, with an
+overall width of 6.75~inches, height of 9.0~inches, and 0.25~inches
+between the columns. The left margin should be 0.75~inches and the top
+margin 1.0~inch (2.54~cm). The right and bottom margins will depend on
+whether you print on US letter or A4 paper, but all final versions
+must be produced for US letter size.
+Do not write anything on the margins.
+
+The paper body should be set in 10~point type with a vertical spacing
+of 11~points. Please use Times typeface throughout the text.
+
+\subsection{Title}
+
+The paper title should be set in 14~point bold type and centered
+between two horizontal rules that are 1~point thick, with 1.0~inch
+between the top rule and the top edge of the page. Capitalize the
+first letter of content words and put the rest of the title in lower
+case.
+You can use TeX math in the title (we suggest sparingly),
+but no custom macros, images, or other TeX commands.
+Please make sure that accents, special characters, etc., are entered using
+TeX commands and not using non-English characters.
+
+\subsection{Author Information for Submission}
+\label{author info}
+
+ICML uses double-blind review, so author information must not appear. If
+you are using \LaTeX\/ and the \texttt{icml2026.sty} file, use
+\verb+\icmlauthor{...}+ to specify authors and \verb+\icmlaffiliation{...}+
+to specify affiliations. (Read the TeX code used to produce this document for
+an example usage.) The author information will not be printed unless
+\texttt{accepted} is passed as an argument to the style file. Submissions that
+include the author information will not be reviewed.
+
+\subsubsection{Self-Citations}
+
+If you are citing published papers for which you are an author, refer
+to yourself in the third person. In particular, do not use phrases
+that reveal your identity (e.g., ``in previous work \cite{langley00}, we
+have shown \ldots'').
+
+Do not anonymize citations in the reference section. The only exception are manuscripts that are
+not yet published (e.g., under submission). If you choose to refer to
+such unpublished manuscripts \cite{anonymous}, anonymized copies have
+to be submitted
+as Supplementary Material via OpenReview\@. However, keep in mind that an ICML
+paper should be self contained and should contain sufficient detail
+for the reviewers to evaluate the work. In particular, reviewers are
+not required to look at the Supplementary Material when writing their
+review (they are not required to look at more than the first $8$ pages of the submitted document).
+
+\subsubsection{Camera-Ready Author Information}
+\label{final author}
+
+If a paper is accepted, a final camera-ready copy must be prepared.
+%
+For camera-ready papers, author information should start 0.3~inches below the
+bottom rule surrounding the title. The authors' names should appear in 10~point
+bold type, in a row, separated by white space, and centered. Author names should
+not be broken across lines. Unbolded superscripted numbers, starting 1, should
+be used to refer to affiliations.
+
+Affiliations should be numbered in the order of appearance. A single footnote
+block of text should be used to list all the affiliations. (Academic
+affiliations should list Department, University, City, State/Region, Country.
+Similarly for industrial affiliations.)
+
+Each distinct affiliations should be listed once. If an author has multiple
+affiliations, multiple superscripts should be placed after the name, separated
+by thin spaces. If the authors would like to highlight equal contribution by
+multiple first authors, those authors should have an asterisk placed after their
+name in superscript, and the term ``\textsuperscript{*}Equal contribution"
+should be placed in the footnote block ahead of the list of affiliations. A
+list of corresponding authors and their emails (in the format Full Name
+\textless{}email@domain.com\textgreater{}) can follow the list of affiliations.
+Ideally only one or two names should be listed.
+
+A sample file with author names is included in the ICML2026 style file
+package. Turn on the \texttt{[accepted]} option to the stylefile to
+see the names rendered. All of the guidelines above are implemented
+by the \LaTeX\ style file.
+
+\subsection{Abstract}
+
+The paper abstract should begin in the left column, 0.4~inches below the final
+address. The heading `Abstract' should be centered, bold, and in 11~point type.
+The abstract body should use 10~point type, with a vertical spacing of
+11~points, and should be indented 0.25~inches more than normal on left-hand and
+right-hand margins. Insert 0.4~inches of blank space after the body. Keep your
+abstract brief and self-contained, limiting it to one paragraph and roughly 4--6
+sentences. Gross violations will require correction at the camera-ready phase.
+
+\subsection{Partitioning the Text}
+
+You should organize your paper into sections and paragraphs to help readers
+place a structure on the material and understand its contributions.
+
+\subsubsection{Sections and Subsections}
+
+Section headings should be numbered, flush left, and set in 11~pt bold type
+with the content words capitalized. Leave 0.25~inches of space before the
+heading and 0.15~inches after the heading.
+
+Similarly, subsection headings should be numbered, flush left, and set in 10~pt
+bold type with the content words capitalized. Leave
+0.2~inches of space before the heading and 0.13~inches afterward.
+
+Finally, subsubsection headings should be numbered, flush left, and set in
+10~pt small caps with the content words capitalized. Leave
+0.18~inches of space before the heading and 0.1~inches after the heading.
+
+Please use no more than three levels of headings.
+
+\subsubsection{Paragraphs and Footnotes}
+
+Within each section or subsection, you should further partition the paper into
+paragraphs. Do not indent the first line of a given paragraph, but insert a
+blank line between succeeding ones.
+
+You can use footnotes\footnote{Footnotes should be complete sentences.}
+to provide readers with additional information about a topic without
+interrupting the flow of the paper. Indicate footnotes with a number in the
+text where the point is most relevant. Place the footnote in 9~point type at
+the bottom of the column in which it appears. Precede the first footnote in a
+column with a horizontal rule of 0.8~inches.\footnote{Multiple footnotes can
+  appear in each column, in the same order as they appear in the text,
+  but spread them across columns and pages if possible.}
+
+\begin{figure}[ht]
+  \vskip 0.2in
+  \begin{center}
+    \centerline{\includegraphics[width=\columnwidth]{icml_numpapers}}
+    \caption{
+      Historical locations and number of accepted papers for International
+      Machine Learning Conferences (ICML 1993 -- ICML 2008) and International
+      Workshops on Machine Learning (ML 1988 -- ML 1992). At the time this
+      figure was produced, the number of accepted papers for ICML 2008 was
+      unknown and instead estimated.
+    }
+    \label{icml-historical}
+  \end{center}
+\end{figure}
+
+\subsection{Figures}
+
+You may want to include figures in the paper to illustrate your approach and
+results. Such artwork should be centered, legible, and separated from the text.
+Lines should be dark and at least 0.5~points thick for purposes of
+reproduction, and text should not appear on a gray background.
+
+Label all distinct components of each figure. If the figure takes the form of a
+graph, then give a name for each axis and include a legend that briefly
+describes each curve. Do not include a title inside the figure; instead, the
+caption should serve this function.
+
+Number figures sequentially, placing the figure number and caption \emph{after}
+the graphics, with at least 0.1~inches of space before the caption and
+0.1~inches after it, as in \cref{icml-historical}. The figure caption should be
+set in 9~point type and centered unless it runs two or more lines, in which
+case it should be flush left. You may float figures to the top or bottom of a
+column, and you may set wide figures across both columns (use the environment
+\texttt{figure*} in \LaTeX). Always place two-column figures at the top or
+bottom of the page.
+
+\subsection{Algorithms}
+
+If you are using \LaTeX, please use the ``algorithm'' and ``algorithmic''
+environments to format pseudocode. These require the corresponding stylefiles,
+algorithm.sty and algorithmic.sty, which are supplied with this package.
+\cref{alg:example} shows an example.
+
+\begin{algorithm}[tb]
+  \caption{Bubble Sort}
+  \label{alg:example}
+  \begin{algorithmic}
+    \STATE {\bfseries Input:} data $x_i$, size $m$
+    \REPEAT
+    \STATE Initialize $noChange = true$.
+    \FOR{$i=1$ {\bfseries to} $m-1$}
+    \IF{$x_i > x_{i+1}$}
+    \STATE Swap $x_i$ and $x_{i+1}$
+    \STATE $noChange = false$
+    \ENDIF
+    \ENDFOR
+    \UNTIL{$noChange$ is $true$}
+  \end{algorithmic}
+\end{algorithm}
+
+
+\subsection{Tables}
+
+You may also want to include tables that summarize material. Like figures,
+these should be centered, legible, and numbered consecutively. However, place
+the title \emph{above} the table with at least 0.1~inches of space before the
+title and the same after it, as in \cref{sample-table}. The table title should
+be set in 9~point type and centered unless it runs two or more lines, in which
+case it should be flush left.
+
+% Note use of \abovespace and \belowspace to get reasonable spacing
+% above and below tabular lines.
+
+\begin{table}[t]
+  \caption{Classification accuracies for naive Bayes and flexible
+    Bayes on various data sets.}
+  \label{sample-table}
+  \begin{center}
+    \begin{small}
+      \begin{sc}
+        \begin{tabular}{lcccr}
+          \toprule
+          Data set  & Naive         & Flexible      & Better?  \\
+          \midrule
+          Breast    & 95.9$\pm$ 0.2 & 96.7$\pm$ 0.2 & $\surd$  \\
+          Cleveland & 83.3$\pm$ 0.6 & 80.0$\pm$ 0.6 & $\times$ \\
+          Glass2    & 61.9$\pm$ 1.4 & 83.8$\pm$ 0.7 & $\surd$  \\
+          Credit    & 74.8$\pm$ 0.5 & 78.3$\pm$ 0.6 &          \\
+          Horse     & 73.3$\pm$ 0.9 & 69.7$\pm$ 1.0 & $\times$ \\
+          Meta      & 67.1$\pm$ 0.6 & 76.5$\pm$ 0.5 & $\surd$  \\
+          Pima      & 75.1$\pm$ 0.6 & 73.9$\pm$ 0.5 &          \\
+          Vehicle   & 44.9$\pm$ 0.6 & 61.5$\pm$ 0.4 & $\surd$  \\
+          \bottomrule
+        \end{tabular}
+      \end{sc}
+    \end{small}
+  \end{center}
+  \vskip -0.1in
+\end{table}
+
+Tables contain textual material, whereas figures contain graphical material.
+Specify the contents of each row and column in the table's topmost row. Again,
+you may float tables to a column's top or bottom, and set wide tables across
+both columns. Place two-column tables at the top or bottom of the page.
+
+\subsection{Theorems and Such}
+The preferred way is to number definitions, propositions, lemmas, etc.
+consecutively, within sections, as shown below.
+\begin{definition}
+  \label{def:inj}
+  A function $f:X \to Y$ is injective if for any $x,y\in X$ different, $f(x)\ne
+    f(y)$.
+\end{definition}
+Using \cref{def:inj} we immediate get the following result:
+\begin{proposition}
+  If $f$ is injective mapping a set $X$ to another set $Y$,
+  the cardinality of $Y$ is at least as large as that of $X$
+\end{proposition}
+\begin{proof}
+  Left as an exercise to the reader.
+\end{proof}
+\cref{lem:usefullemma} stated next will prove to be useful.
+\begin{lemma}
+  \label{lem:usefullemma}
+  For any $f:X \to Y$ and $g:Y\to Z$ injective functions, $f \circ g$ is
+  injective.
+\end{lemma}
+\begin{theorem}
+  \label{thm:bigtheorem}
+  If $f:X\to Y$ is bijective, the cardinality of $X$ and $Y$ are the same.
+\end{theorem}
+An easy corollary of \cref{thm:bigtheorem} is the following:
+\begin{corollary}
+  If $f:X\to Y$ is bijective,
+  the cardinality of $X$ is at least as large as that of $Y$.
+\end{corollary}
+\begin{assumption}
+  The set $X$ is finite.
+  \label{ass:xfinite}
+\end{assumption}
+\begin{remark}
+  According to some, it is only the finite case (cf. \cref{ass:xfinite}) that
+  is interesting.
+\end{remark}
+%restatable
+
+\subsection{Citations and References}
+
+Please use APA reference format regardless of your formatter or word processor.
+If you rely on the \LaTeX\/ bibliographic facility, use \texttt{natbib.sty} and
+\texttt{icml2026.bst} included in the style-file package to obtain this format.
+
+Citations within the text should include the authors' last names and year. If
+the authors' names are included in the sentence, place only the year in
+parentheses, for example when referencing Arthur Samuel's pioneering work
+\yrcite{Samuel59}. Otherwise place the entire reference in parentheses with the
+authors and year separated by a comma \cite{Samuel59}. List multiple references
+separated by semicolons \cite{kearns89,Samuel59,mitchell80}. Use the `et~al.'
+construct only for citations with three or more authors or after listing all
+authors to a publication in an earlier reference \cite{MachineLearningI}.
+
+Authors should cite their own work in the third person in the initial version
+of their paper submitted for blind review. Please refer to \cref{author info}
+for detailed instructions on how to cite your own papers.
+
+Use an unnumbered first-level section heading for the references, and use a
+hanging indent style, with the first line of the reference flush against the
+left margin and subsequent lines indented by 10 points. The references at the
+end of this document give examples for journal articles \cite{Samuel59},
+conference publications \cite{langley00}, book chapters \cite{Newell81}, books
+\cite{DudaHart2nd}, edited volumes \cite{MachineLearningI}, technical reports
+\cite{mitchell80}, and dissertations \cite{kearns89}.
+
+Alphabetize references by the surnames of the first authors, with single author
+entries preceding multiple author entries. Order references for the same
+authors by year of publication, with the earliest first. Make sure that each
+reference includes all relevant information (e.g., page numbers).
+
+Please put some effort into making references complete, presentable, and
+consistent, e.g. use the actual current name of authors. If using bibtex,
+please protect capital letters of names and abbreviations in titles, for
+example, use \{B\}ayesian or \{L\}ipschitz in your .bib file.
+
+\section*{Accessibility}
+
+Authors are kindly asked to make their submissions as accessible as possible
+for everyone including people with disabilities and sensory or neurological
+differences. Tips of how to achieve this and what to pay attention to will be
+provided on the conference website \url{http://icml.cc/}.
+
+\section*{Software and Data}
+
+If a paper is accepted, we strongly encourage the publication of software and
+data with the camera-ready version of the paper whenever appropriate. This can
+be done by including a URL in the camera-ready copy. However, \textbf{do not}
+include URLs that reveal your institution or identity in your submission for
+review. Instead, provide an anonymous URL or upload the material as
+``Supplementary Material'' into the OpenReview reviewing system. Note that
+reviewers are not required to look at this material when writing their review.
+
+% Acknowledgements should only appear in the accepted version.
+\section*{Acknowledgements}
+
+\textbf{Do not} include acknowledgements in the initial version of the paper
+submitted for blind review.
+
+If a paper is accepted, the final camera-ready version can (and usually should)
+include acknowledgements.  Such acknowledgements should be placed at the end of
+the section, in an unnumbered section that does not count towards the paper
+page limit. Typically, this will include thanks to reviewers who gave useful
+comments, to colleagues who contributed to the ideas, and to funding agencies
+and corporate sponsors that provided financial support.
+
+\section*{Impact Statement}
+
+Authors are \textbf{required} to include a statement of the potential broader
+impact of their work, including its ethical aspects and future societal
+consequences. This statement should be in an unnumbered section at the end of
+the paper (co-located with Acknowledgements -- the two may appear in either
+order, but both must be before References), and does not count toward the paper
+page limit. In many cases, where the ethical impacts and expected societal
+implications are those that are well established when advancing the field of
+Machine Learning, substantial discussion is not required, and a simple
+statement such as the following will suffice:
+
+``This paper presents work whose goal is to advance the field of Machine
+Learning. There are many potential societal consequences of our work, none
+which we feel must be specifically highlighted here.''
+
+The above statement can be used verbatim in such cases, but we encourage
+authors to think about whether there is content which does warrant further
+discussion, as this statement will be apparent if the paper is later flagged
+for ethics review.
+
+% In the unusual situation where you want a paper to appear in the
+% references without citing it in the main text, use \nocite
+\nocite{langley00}
+
+\bibliography{example_paper}
+\bibliographystyle{icml2026}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+% APPENDIX
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\newpage
+\appendix
+\onecolumn
+\section{You \emph{can} have an appendix here.}
+
+You can have as much text here as you want. The main body must be at most $8$
+pages long. For the final version, one more page can be added. If you want, you
+can use an appendix like this one.
+
+The $\mathtt{\backslash onecolumn}$ command above can be kept in place if you
+prefer a one-column appendix, or can be removed if you prefer a two-column
+appendix.  Apart from this possible change, the style (font size, spacing,
+margins, page numbering, etc.) should be kept the same as the main body.
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\end{document}
+
+% This document was modified from the file originally made available by
+% Pat Langley and Andrea Danyluk for ICML-2K. This version was created
+% by Iain Murray in 2018, and modified by Alexandre Bouchard in
+% 2019 and 2021 and by Csaba Szepesvari, Gang Niu and Sivan Sabato in 2022.
+% Modified again in 2023 and 2024 by Sivan Sabato and Jonathan Scarlett.
+% Previous contributors include Dan Roy, Lise Getoor and Tobias
+% Scheffer, which was slightly modified from the 2010 version by
+% Thorsten Joachims & Johannes Fuernkranz, slightly modified from the
+% 2009 version by Kiri Wagstaff and Sam Roweis's 2008 version, which is
+% slightly modified from Prasad Tadepalli's 2007 version which is a
+% lightly changed version of the previous year's version by Andrew
+% Moore, which was in turn edited from those of Kristian Kersting and
+% Codrina Lauth. Alex Smola contributed to the algorithmic style files.

fancyhdr.sty

@@ -0,0 +1,864 @@
+%%
+%% This is file `fancyhdr.sty',
+%% generated with the docstrip utility.
+%%
+%% The original source files were:
+%%
+%% fancyhdr.dtx  (with options: `fancyhdr')
+%% 
+%% This is a generated file.
+%% 
+%% This file may be distributed and/or modified under the conditions of
+%% the LaTeX Project Public License, either version 1.3 of this license
+%% or (at your option) any later version.  The latest version of this
+%% license is in:
+%% 
+%%    http://www.latex-project.org/lppl.txt
+%% 
+%% and version 1.3 or later is part of all distributions of LaTeX version
+%% 2005/12/01 or later.
+%% 
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\NeedsTeXFormat{LaTeX2e}[2018-04-01]
+\ProvidesPackage{fancyhdr}%
+           [2025/02/07 v5.2
+                  Extensive control of page headers and footers]%
+% Copyright (C) 1994-2025 by Pieter van Oostrum <pieter@vanoostrum.org>
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+\ifdefined\NewDocumentCommand\else\RequirePackage{xparse}\fi
+\newif\iff@nch@check
+\f@nch@checktrue
+\DeclareOption{nocheck}{%
+  \f@nch@checkfalse
+}
+\let\f@nch@gbl\relax
+\newif\iff@nch@compatViii
+\DeclareOption{compatV3}{%
+  \PackageWarningNoLine{fancyhdr}{The `compatV3' option is deprecated.\MessageBreak
+    It will disappear in one of the following releases.\MessageBreak
+    Please change your document to work\MessageBreak
+    without this option}
+  \let\f@nch@gbl\global
+  \f@nch@compatViiitrue
+}
+\newif\iff@nch@twoside
+\f@nch@twosidefalse
+\DeclareOption{twoside}{%
+  \if@twoside\else\f@nch@twosidetrue\fi
+}
+\newcommand\f@nch@def[2]{%
+  \def\temp@a{#2}\ifx\temp@a\@empty\f@nch@gbl\def#1{}%
+                 \else\f@nch@gbl\def#1{#2\strut}\fi}
+\DeclareOption{myheadings}{%
+  \@ifundefined{chapter}{%
+    \def\ps@myheadings{\ps@f@nch@fancyproto \let\@mkboth\@gobbletwo
+      \fancyhf{}
+      \fancyhead[LE,RO]{\thepage}%
+      \fancyhead[RE]{\slshape\leftmark}%
+      \fancyhead[LO]{\slshape\rightmark}%
+      \let\sectionmark\@gobble
+      \let\subsectionmark\@gobble
+    }%
+  }%
+  {\def\ps@myheadings{\ps@f@nch@fancyproto \let\@mkboth\@gobbletwo
+      \fancyhf{}
+      \fancyhead[LE,RO]{\thepage}%
+      \fancyhead[RE]{\slshape\leftmark}%
+      \fancyhead[LO]{\slshape\rightmark}%
+      \let\chaptermark\@gobble
+      \let\sectionmark\@gobble
+    }%
+  }%
+}
+\DeclareOption{headings}{%
+  \@ifundefined{chapter}{%
+    \if@twoside
+      \def\ps@headings{\ps@f@nch@fancyproto \def\@mkboth{\protect\markboth}
+        \fancyhf{}
+        \fancyhead[LE,RO]{\thepage}%
+        \fancyhead[RE]{\slshape\leftmark}%
+        \fancyhead[LO]{\slshape\rightmark}%
+        \def\sectionmark##1{%
+          \markboth{\MakeUppercase{%
+            \ifnum \c@secnumdepth >\z@ \thesection\quad \fi##1}}{}}%
+        \def\subsectionmark##1{%
+          \markright{%
+            \ifnum \c@secnumdepth >\@ne \thesubsection\quad \fi##1}}%
+      }%
+    \else
+      \def\ps@headings{\ps@f@nch@fancyproto \def\@mkboth{\protect\markboth}
+        \fancyhf{}
+        \fancyhead[LE,RO]{\thepage}%
+        \fancyhead[RE]{\slshape\leftmark}%
+        \fancyhead[LO]{\slshape\rightmark}%
+        \def\sectionmark##1{%
+          \markright {\MakeUppercase{%
+            \ifnum \c@secnumdepth >\z@ \thesection\quad \fi##1}}}%
+        \let\subsectionmark\@gobble % Not needed but inserted for safety
+      }%
+    \fi
+  }{\if@twoside
+      \def\ps@headings{\ps@f@nch@fancyproto \def\@mkboth{\protect\markboth}
+        \fancyhf{}
+        \fancyhead[LE,RO]{\thepage}%
+        \fancyhead[RE]{\slshape\leftmark}%
+        \fancyhead[LO]{\slshape\rightmark}%
+        \def\chaptermark##1{%
+          \markboth{\MakeUppercase{%
+            \ifnum \c@secnumdepth >\m@ne \if@mainmatter
+              \@chapapp\ \thechapter. \ \fi\fi##1}}{}}%
+        \def\sectionmark##1{%
+          \markright {\MakeUppercase{%
+            \ifnum \c@secnumdepth >\z@ \thesection. \ \fi##1}}}%
+      }%
+    \else
+      \def\ps@headings{\ps@f@nch@fancyproto \def\@mkboth{\protect\markboth}
+        \fancyhf{}
+        \fancyhead[LE,RO]{\thepage}%
+        \fancyhead[RE]{\slshape\leftmark}%
+        \fancyhead[LO]{\slshape\rightmark}%
+        \def\chaptermark##1{%
+          \markright{\MakeUppercase{%
+            \ifnum \c@secnumdepth >\m@ne \if@mainmatter
+              \@chapapp\ \thechapter. \ \fi\fi##1}}}%
+        \let\sectionmark\@gobble % Not needed but inserted for safety
+      }%
+    \fi
+  }%
+}
+\ProcessOptions*
+\newcommand{\f@nch@forc}[3]{\expandafter\f@nchf@rc\expandafter#1\expandafter{#2}{#3}}
+\newcommand{\f@nchf@rc}[3]{\def\temp@ty{#2}\ifx\@empty\temp@ty\else
+                                    \f@nch@rc#1#2\f@nch@rc{#3}\fi}
+\long\def\f@nch@rc#1#2#3\f@nch@rc#4{\def#1{#2}#4\f@nchf@rc#1{#3}{#4}}
+\newcommand{\f@nch@for}[3]{\edef\@fortmp{#2}%
+  \expandafter\@forloop#2,\@nil,\@nil\@@#1{#3}}
+\newcommand\f@nch@default[3]{%
+  \edef\temp@a{\lowercase{\edef\noexpand\temp@a{#3}}}\temp@a \def#1{}%
+  \f@nch@forc\tmpf@ra{#2}%
+  {\expandafter\f@nch@ifin\tmpf@ra\temp@a{\edef#1{#1\tmpf@ra}}{}}%
+  \ifx\@empty#1\def#1{#2}\fi}
+\newcommand{\f@nch@ifin}[4]{%
+  \edef\temp@a{#2}\def\temp@b##1#1##2\temp@b{\def\temp@b{##1}}%
+  \expandafter\temp@b#2#1\temp@b\ifx\temp@a\temp@b #4\else #3\fi}
+\newcommand{\fancyhead}[2][]{\f@nch@fancyhf\fancyhead h[#1]{#2}}%
+\newcommand{\fancyfoot}[2][]{\f@nch@fancyhf\fancyfoot f[#1]{#2}}%
+\newcommand{\fancyhf}[2][]{\f@nch@fancyhf\fancyhf {}[#1]{#2}}%
+\newcommand{\fancyheadoffset}[2][]{\f@nch@fancyhfoffs\fancyheadoffset h[#1]{#2}}%
+\newcommand{\fancyfootoffset}[2][]{\f@nch@fancyhfoffs\fancyfootoffset f[#1]{#2}}%
+\newcommand{\fancyhfoffset}[2][]{\f@nch@fancyhfoffs\fancyhfoffset {}[#1]{#2}}%
+\def\f@nch@fancyhf@Echeck#1{%
+  \if@twoside\else
+    \iff@nch@twoside\else
+      \if\f@nch@@eo e%
+        \PackageWarning{fancyhdr} {\string#1's `E' option without twoside option is useless.\MessageBreak
+          Please consider using the `twoside' option}%
+  \fi\fi\fi
+}
+\long\def\f@nch@fancyhf#1#2[#3]#4{%
+  \def\temp@c{}%
+  \f@nch@forc\tmpf@ra{#3}%
+  {\expandafter\f@nch@ifin\tmpf@ra{eolcrhf,EOLCRHF}%
+    {}{\edef\temp@c{\temp@c\tmpf@ra}}}%
+  \ifx\@empty\temp@c\else \PackageError{fancyhdr}{Illegal char `\temp@c' in
+    \string#1 argument: [#3]}{}%
+  \fi \f@nch@for\temp@c{#3}%
+  {\f@nch@default\f@nch@@eo{eo}\temp@c
+    \f@nch@fancyhf@Echeck{#1}%
+    \f@nch@default\f@nch@@lcr{lcr}\temp@c
+    \f@nch@default\f@nch@@hf{hf}{#2\temp@c}%
+    \f@nch@forc\f@nch@eo\f@nch@@eo
+        {\f@nch@forc\f@nch@lcr\f@nch@@lcr
+          {\f@nch@forc\f@nch@hf\f@nch@@hf
+            {\expandafter\f@nch@def\csname
+              f@nch@\f@nch@eo\f@nch@lcr\f@nch@hf\endcsname {#4}}}}}}
+\def\f@nch@fancyhfoffs#1#2[#3]#4{%
+  \def\temp@c{}%
+  \f@nch@forc\tmpf@ra{#3}%
+  {\expandafter\f@nch@ifin\tmpf@ra{eolrhf,EOLRHF}%
+    {}{\edef\temp@c{\temp@c\tmpf@ra}}}%
+  \ifx\@empty\temp@c\else \PackageError{fancyhdr}{Illegal char `\temp@c' in
+    \string#1 argument: [#3]}{}%
+  \fi \f@nch@for\temp@c{#3}%
+  {\f@nch@default\f@nch@@eo{eo}\temp@c
+    \f@nch@fancyhf@Echeck{#1}%
+    \f@nch@default\f@nch@@lcr{lr}\temp@c
+    \f@nch@default\f@nch@@hf{hf}{#2\temp@c}%
+    \f@nch@forc\f@nch@eo\f@nch@@eo
+        {\f@nch@forc\f@nch@lcr\f@nch@@lcr
+          {\f@nch@forc\f@nch@hf\f@nch@@hf
+            {\expandafter\setlength\csname
+              f@nch@offset@\f@nch@eo\f@nch@lcr\f@nch@hf\endcsname {#4}}}}}%
+  \f@nch@setoffs}
+\NewDocumentCommand {\fancyheadwidth}{ s O{} O{} m }
+                      {\f@nch@fancyhfwidth{#1}\fancyheadwidth h[#2][#3]{#4}}%
+\NewDocumentCommand {\fancyfootwidth}{ s O{} O{} m }
+                      {\f@nch@fancyhfwidth{#1}\fancyfootwidth f[#2][#3]{#4}}%
+\NewDocumentCommand {\fancyhfwidth}  { s O{} O{} m }
+                      {\f@nch@fancyhfwidth{#1}\fancyhfwidth  {}[#2][#3]{#4}}%
+\def\f@nch@fancyhfwidth#1#2#3[#4][#5]#6{%
+  \setlength\@tempdima{#6}%
+  \def\temp@c{}%
+  \f@nch@forc\tmpf@ra{#4}%
+  {\expandafter\f@nch@ifin\tmpf@ra{eolcrhf,EOLCRHF}%
+    {}{\edef\temp@c{\temp@c\tmpf@ra}}}%
+  \ifx\@empty\temp@c\else \PackageError{fancyhdr}{Illegal char `\temp@c' in
+    \string#2 argument: [#4]}{}%
+  \fi
+  \f@nch@for\temp@c{#4}%
+  {\f@nch@default\f@nch@@eo{eo}\temp@c
+    \f@nch@fancyhf@Echeck{#2}%
+    \f@nch@default\f@nch@@lcr{lcr}\temp@c
+    \f@nch@default\f@nch@@hf{hf}{#3\temp@c}%
+    \f@nch@forc\f@nch@eo\f@nch@@eo
+        {\f@nch@forc\f@nch@lcr\f@nch@@lcr
+          {\f@nch@forc\f@nch@hf\f@nch@@hf
+            {%
+              \IfBooleanTF{#1}{%
+                \expandafter\edef\csname
+                  f@nch@width@\f@nch@eo\f@nch@lcr\f@nch@hf\endcsname{\the\@tempdima}%
+              }%
+              {%
+                \expandafter\def\csname
+                  f@nch@width@\f@nch@eo\f@nch@lcr\f@nch@hf\endcsname{#6}%
+              }%
+              \csname f@nchdrwdt@align@v@\f@nch@hf\endcsname
+              \edef\f@nch@align@@h{\f@nch@lcr}%
+              \def\temp@a{#5}%
+              \ifx\temp@a\@empty \else \f@nchdrwdt@align#5\@nil{#2}\fi
+              \expandafter\edef\csname
+                f@nch@align@\f@nch@eo\f@nch@lcr\f@nch@hf\endcsname
+                   {\f@nch@align@@v\f@nch@align@@h}}}}}}
+\def\f@nch@width@elh{\headwidth}
+\def\f@nch@width@ech{\headwidth}
+\def\f@nch@width@erh{\headwidth}
+\def\f@nch@width@olh{\headwidth}
+\def\f@nch@width@och{\headwidth}
+\def\f@nch@width@orh{\headwidth}
+\def\f@nch@width@elf{\headwidth}
+\def\f@nch@width@ecf{\headwidth}
+\def\f@nch@width@erf{\headwidth}
+\def\f@nch@width@olf{\headwidth}
+\def\f@nch@width@ocf{\headwidth}
+\def\f@nch@width@orf{\headwidth}
+\def\f@nch@align@elh{bl}
+\def\f@nch@align@ech{bc}
+\def\f@nch@align@erh{br}
+\def\f@nch@align@olh{bl}
+\def\f@nch@align@och{bc}
+\def\f@nch@align@orh{br}
+\def\f@nch@align@elf{tl}
+\def\f@nch@align@ecf{tc}
+\def\f@nch@align@erf{tr}
+\def\f@nch@align@olf{tl}
+\def\f@nch@align@ocf{tc}
+\def\f@nch@align@orf{tr}
+\def\f@nchdrwdt@align@v@h{\def\f@nch@align@@v{b}}%
+\def\f@nchdrwdt@align@v@f{\def\f@nch@align@@v{t}}%
+\long\def\f@nchdrwdt@align#1#2\@nil#3{%
+  \f@nch@ifin{#1}{TtcbB-}{%
+    \f@nch@ifin{#1}{-}{}{\def\f@nch@align@@v{#1}}%
+    \def\@tempa{#2}%
+    \ifx\@tempa\@empty \else \def\f@nch@align@@h{#2}\fi
+  }%
+  {\def\f@nch@align@@h{#1}}%
+  \expandafter\f@nch@ifin\expandafter{\f@nch@align@@h}{lcrj}{}%
+    {\PackageError{fancyhdr}
+                  {\string#3: Illegal char `\f@nch@align@@h'\MessageBreak
+                              in alignment argument}{}}%
+}
+\newcommand{\lhead}[2][\f@nch@olh]%
+                     {\f@nch@def\f@nch@olh{#2}\f@nch@def\f@nch@elh{#1}}
+\newcommand{\chead}[2][\f@nch@och]%
+                     {\f@nch@def\f@nch@och{#2}\f@nch@def\f@nch@ech{#1}}
+\newcommand{\rhead}[2][\f@nch@orh]%
+                     {\f@nch@def\f@nch@orh{#2}\f@nch@def\f@nch@erh{#1}}
+\newcommand{\lfoot}[2][\f@nch@olf]%
+                     {\f@nch@def\f@nch@olf{#2}\f@nch@def\f@nch@elf{#1}}
+\newcommand{\cfoot}[2][\f@nch@ocf]%
+                     {\f@nch@def\f@nch@ocf{#2}\f@nch@def\f@nch@ecf{#1}}
+\newcommand{\rfoot}[2][\f@nch@orf]%
+                     {\f@nch@def\f@nch@orf{#2}\f@nch@def\f@nch@erf{#1}}
+\newlength{\f@nch@headwidth} \let\headwidth\f@nch@headwidth
+\newlength{\f@nch@offset@elh}
+\newlength{\f@nch@offset@erh}
+\newlength{\f@nch@offset@olh}
+\newlength{\f@nch@offset@orh}
+\newlength{\f@nch@offset@elf}
+\newlength{\f@nch@offset@erf}
+\newlength{\f@nch@offset@olf}
+\newlength{\f@nch@offset@orf}
+\newcommand{\headrulewidth}{0.4pt}
+\newcommand{\footrulewidth}{0pt}
+\@ifundefined{headruleskip}%
+      {\newcommand{\headruleskip}{0pt}}{}
+\@ifundefined{footruleskip}%
+      {\newcommand{\footruleskip}{.3\normalbaselineskip}}{}
+\newcommand{\plainheadrulewidth}{0pt}
+\newcommand{\plainfootrulewidth}{0pt}
+\newif\if@fancyplain \@fancyplainfalse
+\def\fancyplain#1#2{\if@fancyplain#1\else#2\fi}
+\headwidth=-123456789sp
+\let\f@nch@raggedleft\raggedleft
+\let\f@nch@raggedright\raggedright
+\let\f@nch@centering\centering
+\let\f@nch@everypar\everypar
+\ifdefined\ExplSyntaxOn
+  \ExplSyntaxOn
+  \providecommand\IfFormatAtLeastTF{\@ifl@t@r\fmtversion}
+  \IfFormatAtLeastTF{2021-06-01}{
+    \def\f@nch@saveclr@parhook #1{
+      \expandafter\let\csname f@nch@__hook~#1\expandafter\endcsname
+                      \csname __hook~#1\endcsname
+      \expandafter\let\csname f@nch@__hook_toplevel~#1\expandafter\endcsname
+                      \csname __hook_toplevel~#1\endcsname
+      \expandafter\let\csname f@nch@__hook_next~#1\expandafter\endcsname
+                      \csname __hook_next~#1\endcsname
+      \expandafter\let\csname f@nch@g__hook_#1_code_prop\expandafter\endcsname
+                      \csname g__hook_#1_code_prop\endcsname
+      \RemoveFromHook{#1}[*]
+      \ClearHookNext{#1}
+    }
+    \def\f@nch@restore@parhook #1{
+      \global\expandafter\let\csname __hook~#1\expandafter\endcsname
+                             \csname f@nch@__hook~#1\endcsname
+      \global\expandafter\let\csname __hook_toplevel~#1\expandafter\endcsname
+                             \csname f@nch@__hook_toplevel~#1\endcsname
+      \global\expandafter\let\csname __hook_next~#1\expandafter\endcsname
+                             \csname f@nch@__hook_next~#1\endcsname
+      \global\expandafter\let\csname g__hook_#1_code_prop\expandafter\endcsname
+                             \csname f@nch@g__hook_#1_code_prop\endcsname
+    }
+    \def\f@nch@resetpar{
+      \f@nch@everypar{}
+      \f@nch@saveclr@parhook{para/before}
+      \f@nch@saveclr@parhook{para/begin}
+      \f@nch@saveclr@parhook{para/end}
+      \f@nch@saveclr@parhook{para/after}
+    }
+    \def\f@nch@restorepar{
+      \f@nch@restore@parhook{para/before}
+      \f@nch@restore@parhook{para/begin}
+      \f@nch@restore@parhook{para/end}
+      \f@nch@restore@parhook{para/after}
+    }
+  }{
+    \def\f@nch@resetpar{
+      \f@nch@everypar{}
+    }
+    \def\f@nch@restorepar{}
+  }
+  \ExplSyntaxOff
+\else
+  \def\f@nch@resetpar{%
+    \f@nch@everypar{}%
+  }
+  \def\f@nch@restorepar{}
+\fi
+\newcommand\f@nch@noUppercase[2][]{#2}
+\def\f@nch@reset{\f@nch@resetpar\restorecr\endlinechar=13
+  \catcode`\\=0\catcode`\{=1\catcode`\}=2\catcode`\$=3\catcode`\&=4
+  \catcode`\#=6\catcode`\^=7\catcode`\_=8\catcode`\ =10\catcode`\@=11
+  \catcode`\:=11\catcode`\~=13\catcode`\%=14
+  \catcode0=15 %NULL
+  \catcode9=10 %TAB
+  \let\\\@normalcr \let\raggedleft\f@nch@raggedleft
+  \let\raggedright\f@nch@raggedright \let\centering\f@nch@centering
+  \def\baselinestretch{1}%
+  \hsize=\headwidth
+  \def\nouppercase##1{{%
+      \let\uppercase\relax\let\MakeUppercase\f@nch@noUppercase
+      \expandafter\let\csname MakeUppercase \endcsname\relax
+      \expandafter\def\csname MakeUppercase\space\space\space\endcsname
+                                                   [####1]####2{####2}%
+      ##1}}%
+  \@ifundefined{@normalsize} {\normalsize} % for ucthesis.cls
+   {\@normalsize}%
+  }
+\newcommand*{\fancycenter}[1][1em]{%
+  \@ifnextchar[{\f@nch@center{#1}}{\f@nch@center{#1}[3]}%
+}
+\def\f@nch@center#1[#2]#3#4#5{%
+  \def\@tempa{#4}\ifx\@tempa\@empty
+    \hbox to\linewidth{\color@begingroup{#3}\hfil {#5}\color@endgroup}%
+  \else
+    \setlength\@tempdima{#1}%
+    \setlength{\@tempdimb}{#2\@tempdima}%
+    \@tempdimc \@tempdimb \advance\@tempdimc -\@tempdima
+    \setlength\@tempskipa{\@tempdimb \@plus 1fil \@minus \@tempdimc}%
+    \@tempskipb\@tempskipa
+    \def\@tempa{#3}\ifx\@tempa\@empty
+      \addtolength\@tempskipa{\z@ \@minus \@tempdima}%
+    \fi
+    \def\@tempa{#5}\ifx\@tempa\@empty % empty right
+      \addtolength\@tempskipb{\z@ \@minus \@tempdima}%
+    \fi
+    \settowidth{\@tempdimb}{#3}%
+    \settowidth{\@tempdimc}{#5}%
+    \ifdim\@tempdimb>\@tempdimc
+      \advance\@tempdimb -\@tempdimc
+      \addtolength\@tempskipb{\@tempdimb \@minus \@tempdimb}%
+    \else
+      \advance\@tempdimc -\@tempdimb
+      \addtolength\@tempskipa{\@tempdimc \@minus \@tempdimc}%
+    \fi
+    \hbox to\linewidth{\color@begingroup{#3}\hskip \@tempskipa
+                      {#4}\hskip \@tempskipb {#5}\color@endgroup}%
+  \fi
+}
+\newcommand{\f@nch@headinit}{}
+\newcommand{\fancyheadinit}[1]{%
+  \def\f@nch@headinit{#1}%
+}
+\newcommand{\f@nch@footinit}{}
+\newcommand{\fancyfootinit}[1]{%
+  \def\f@nch@footinit{#1}%
+}
+\newcommand{\fancyhfinit}[1]{%
+  \def\f@nch@headinit{#1}%
+  \def\f@nch@footinit{#1}%
+}
+\ifdefined\NewMirroredHookPair
+  \NewMirroredHookPair{fancyhdr/before}{fancyhdr/after}
+  \NewMirroredHookPair{fancyhdr/head/begin}{fancyhdr/head/end}
+  \NewMirroredHookPair{fancyhdr/foot/begin}{fancyhdr/foot/end}
+\fi
+\newlength\f@nch@height
+\newlength\f@nch@footalignment
+\newif\iff@nch@footalign\f@nch@footalignfalse
+\newcommand{\fancyfootalign}[1]{%
+  \def\temp@a{#1}%
+  \ifx\temp@a\@empty
+    \f@nch@footalignfalse
+  \else
+    \f@nch@footaligntrue
+    \setlength\f@nch@footalignment{#1}%
+  \fi
+}
+\newcommand\fancyhdrsettoheight[2]{%
+  \expandafter\ifx\csname f@nch@#2\endcsname\fancyhdrsettoheight
+    \else\PackageError{fancyhdr}{Unknown parameter #2 in \string\fancyhdrsettoheight}{}\fi
+  \setbox\@tempboxa\hbox{{\f@nch@checkfalse\csname @#2\endcsname}}%
+  \setlength{#1}\f@nch@height
+  \setbox\@tempboxa\box\voidb@x
+}
+\let\f@nch@oddhead\fancyhdrsettoheight
+\let\f@nch@evenhead\fancyhdrsettoheight
+\let\f@nch@oddfoot\fancyhdrsettoheight
+\let\f@nch@evenfoot\fancyhdrsettoheight
+\newcommand\f@nch@vbox[2]{%
+  \setbox0\vbox{#2}%
+  \global\f@nch@height=\ht0
+  \ifdim\ht0>#1\relax
+    \iff@nch@check
+      \dimen0=#1\advance\dimen0-\ht0
+      \PackageWarning{fancyhdr}{%
+        \string#1 is too small (\the#1): \MessageBreak
+        Make it at least \the\ht0, for example:\MessageBreak
+        \string\setlength{\string#1}{\the\ht0}%
+        \iff@nch@compatViii .\MessageBreak
+        We now make it that large for the rest of the document.\MessageBreak
+        This may cause the page layout to be inconsistent, however
+        \fi
+        \ifx#1\headheight .\MessageBreak
+          You might also make \topmargin smaller:\MessageBreak
+          \string\addtolength{\string\topmargin}{\the\dimen0}%
+        \fi
+        \@gobble
+      }%
+      \iff@nch@compatViii
+        \dimen0=#1\relax
+        \global#1=\ht0\relax
+        \ht0=\dimen0 %
+      \else
+        \ht0=#1\relax
+      \fi
+    \else
+      \ht0=#1\relax
+    \fi
+  \fi
+  \box0}
+\newcommand\f@nch@head[6]{%
+  \f@nch@reset
+  \ifdefined\UseHook\UseHook{fancyhdr/before}\UseHook{fancyhdr/head/begin}\fi
+  \f@nch@headinit\relax
+  #1%
+  \hbox to\headwidth{%
+    \f@nch@vbox\headheight{%
+      \f@nch@hfbox{#2}{#3}{#4}{#6}{h}%
+      \vskip\headruleskip\relax
+      \headrule
+    }%
+  }%
+  #5%
+  \ifdefined\UseHook\UseHook{fancyhdr/head/end}\UseHook{fancyhdr/after}\fi
+  \f@nch@restorepar
+}
+\newcommand\f@nch@foot[6]{%
+  \f@nch@reset
+  \ifdefined\UseHook\UseHook{fancyhdr/before}\UseHook{fancyhdr/foot/begin}\fi
+  \f@nch@footinit\relax
+  #1%
+  \hbox to\headwidth{%
+    \f@nch@vbox\footskip{%
+      \setbox0=\vbox{\footrule}\unvbox0
+      \vskip\footruleskip
+      \f@nch@hfbox{#2}{#3}{#4}{#6}{f}%
+    \iff@nch@footalign \vskip\f@nch@footalignment \fi
+    }%
+  }%
+  #5%
+  \ifdefined\UseHook\UseHook{fancyhdr/foot/end}\UseHook{fancyhdr/after}\fi
+  \f@nch@restorepar
+}
+\newlength\f@nch@widthL
+\newlength\f@nch@widthC
+\newlength\f@nch@widthR
+\newcommand\f@nch@hfbox[5]{%
+  \setlength\f@nch@widthL{\csname f@nch@width@#4l#5\endcsname}%
+  \setlength\f@nch@widthC{\csname f@nch@width@#4c#5\endcsname}%
+  \setlength\f@nch@widthR{\csname f@nch@width@#4r#5\endcsname}%
+  \let\@tempa\f@nch@hfbox@center
+  \ifdim \dimexpr \f@nch@widthL+\f@nch@widthC+\f@nch@widthR>\headwidth
+  \else
+    \ifdim \dimexpr \f@nch@widthL+0.5\f@nch@widthC>0.5\headwidth
+      \let \@tempa\f@nch@hfbox@fit
+    \fi
+    \ifdim \dimexpr \f@nch@widthR+0.5\f@nch@widthC>0.5\headwidth
+      \let \@tempa\f@nch@hfbox@fit
+    \fi
+  \fi
+  \@tempa{#1}{#2}{#3}#4#5%
+}
+\newcommand\f@nch@hfbox@center[5]{%
+  \hbox to \headwidth{%
+    \rlap{\f@nch@parbox{#1}\f@nch@widthL{#4}l{#5}}%
+    \hfill
+    \f@nch@parbox{#2}\f@nch@widthC{#4}c{#5}%
+    \hfill
+    \llap{\f@nch@parbox{#3}\f@nch@widthR{#4}r{#5}}%
+  }%
+}
+\newcommand\f@nch@hfbox@fit[5]{%
+  \hbox to \headwidth{%
+    \f@nch@parbox{#1}\f@nch@widthL{#4}l{#5}%
+    \hfill
+    \f@nch@parbox{#2}\f@nch@widthC{#4}c{#5}%
+    \hfill
+    \f@nch@parbox{#3}\f@nch@widthR{#4}r{#5}%
+  }%
+}%
+\newcommand\f@nch@parbox[5]{%
+  \expandafter\expandafter\expandafter\f@nch@parbox@align
+                     \csname f@nch@align@#3#4#5\endcsname
+  \parbox[\f@nch@align@@v]{#2}%
+    {%
+      \f@nch@align@@pre
+      \f@nch@align@@h\leavevmode\ignorespaces#1%
+      \f@nch@align@@post
+    }%
+}
+\newcommand\f@nch@parbox@align[2]{%
+  \def\f@nch@align@@pre{}%
+  \def\f@nch@align@@post{}%
+  \csname f@nch@parbox@align@v#1\endcsname
+  \csname f@nch@parbox@align@h#2\endcsname
+}
+\def\f@nch@parbox@align@vT{\def\f@nch@align@@v{t}\def\f@nch@align@@pre{\vspace{0pt}}}
+\def\f@nch@parbox@align@vt{\def\f@nch@align@@v{t}}
+\def\f@nch@parbox@align@vc{\def\f@nch@align@@v{c}}
+\def\f@nch@parbox@align@vb{\def\f@nch@align@@v{b}}
+\def\f@nch@parbox@align@vB{\def\f@nch@align@@v{b}\def\f@nch@align@@post{\vspace{0pt}}}
+\def\f@nch@parbox@align@hl{\def\f@nch@align@@h{\raggedright}}
+\def\f@nch@parbox@align@hc{\def\f@nch@align@@h{\centering}}
+\def\f@nch@parbox@align@hr{\def\f@nch@align@@h{\raggedleft}}
+\def\f@nch@parbox@align@hj{\def\f@nch@align@@h{}}
+\@ifundefined{@chapapp}{\let\@chapapp\chaptername}{}%
+\def\f@nch@initialise{%
+  \@ifundefined{chapter}%
+   {\def\sectionmark##1{\markboth{\MakeUppercase{\ifnum \c@secnumdepth>\z@
+          \thesection\hskip 1em\relax
+        \fi ##1}}{}}%
+    \def\subsectionmark##1{\markright {\ifnum \c@secnumdepth >\@ne
+      \thesubsection\hskip 1em\relax \fi ##1}}}%
+   {\def\chaptermark##1{\markboth {\MakeUppercase{\ifnum
+        \c@secnumdepth>\m@ne \@chapapp\ \thechapter. \ \fi ##1}}{}}%
+    \def\sectionmark##1{\markright{\MakeUppercase{\ifnum \c@secnumdepth >\z@
+        \thesection. \ \fi ##1}}}%
+   }%
+  \def\headrule{{\if@fancyplain\let\headrulewidth\plainheadrulewidth\fi
+      \hrule\@height\headrulewidth\@width\headwidth
+      \vskip-\headrulewidth}}%
+  \def\footrule{{\if@fancyplain\let\footrulewidth\plainfootrulewidth\fi
+      \hrule\@width\headwidth\@height\footrulewidth}}%
+  \def\headrulewidth{0.4pt}%
+  \def\footrulewidth{0pt}%
+  \def\headruleskip{0pt}%
+  \def\footruleskip{0.3\normalbaselineskip}%
+  \fancyhf{}%
+  \if@twoside
+    \fancyhead[el,or]{\fancyplain{}{\slshape\rightmark}}%
+    \fancyhead[er,ol]{\fancyplain{}{\slshape\leftmark}}%
+  \else
+    \fancyhead[l]{\fancyplain{}{\slshape\rightmark}}%
+    \fancyhead[r]{\fancyplain{}{\slshape\leftmark}}%
+  \fi
+  \fancyfoot[c]{\rmfamily\thepage}% page number
+}
+\f@nch@initialise
+\def\ps@f@nch@fancyproto{%
+  \ifdim\headwidth<0sp
+    \global\advance\headwidth123456789sp\global\advance\headwidth\textwidth
+  \fi
+  \gdef\ps@f@nch@fancyproto{\@fancyplainfalse\ps@f@nch@fancycore}%
+  \@fancyplainfalse\ps@f@nch@fancycore
+}%
+\@namedef{f@nch@ps@f@nch@fancyproto-is-fancyhdr}{}
+\def\ps@fancy{\ps@f@nch@fancyproto}
+\@namedef{f@nch@ps@fancy-is-fancyhdr}{}
+\def\ps@fancyplain{\ps@f@nch@fancyproto \let\ps@plain\ps@plain@fancy}
+\def\ps@plain@fancy{\@fancyplaintrue\ps@f@nch@fancycore}
+\let\f@nch@ps@empty\ps@empty
+\def\ps@f@nch@fancycore{%
+  \f@nch@ps@empty
+  \def\@mkboth{\protect\markboth}%
+  \def\f@nch@oddhead{\f@nch@head\f@nch@Oolh\f@nch@olh\f@nch@och\f@nch@orh\f@nch@Oorh{o}}%
+  \def\@oddhead{%
+    \iff@nch@twoside
+      \ifodd\c@page
+        \f@nch@oddhead
+      \else
+        \@evenhead
+      \fi
+    \else
+      \f@nch@oddhead
+    \fi
+  }
+  \def\f@nch@oddfoot{\f@nch@foot\f@nch@Oolf\f@nch@olf\f@nch@ocf\f@nch@orf\f@nch@Oorf{o}}%
+  \def\@oddfoot{%
+    \iff@nch@twoside
+      \ifodd\c@page
+        \f@nch@oddfoot
+      \else
+        \@evenfoot
+      \fi
+    \else
+      \f@nch@oddfoot
+    \fi
+  }
+  \def\@evenhead{\f@nch@head\f@nch@Oelh\f@nch@elh\f@nch@ech\f@nch@erh\f@nch@Oerh{e}}%
+  \def\@evenfoot{\f@nch@foot\f@nch@Oelf\f@nch@elf\f@nch@ecf\f@nch@erf\f@nch@Oerf{e}}%
+}
+\def\f@nch@Oolh{\if@reversemargin\hss\else\relax\fi}
+\def\f@nch@Oorh{\if@reversemargin\relax\else\hss\fi}
+\let\f@nch@Oelh\f@nch@Oorh
+\let\f@nch@Oerh\f@nch@Oolh
+\let\f@nch@Oolf\f@nch@Oolh
+\let\f@nch@Oorf\f@nch@Oorh
+\let\f@nch@Oelf\f@nch@Oelh
+\let\f@nch@Oerf\f@nch@Oerh
+\def\f@nch@offsolh{\headwidth=\textwidth\advance\headwidth\f@nch@offset@olh
+                   \advance\headwidth\f@nch@offset@orh\hskip-\f@nch@offset@olh}
+\def\f@nch@offselh{\headwidth=\textwidth\advance\headwidth\f@nch@offset@elh
+                   \advance\headwidth\f@nch@offset@erh\hskip-\f@nch@offset@elh}
+\def\f@nch@offsolf{\headwidth=\textwidth\advance\headwidth\f@nch@offset@olf
+                   \advance\headwidth\f@nch@offset@orf\hskip-\f@nch@offset@olf}
+\def\f@nch@offself{\headwidth=\textwidth\advance\headwidth\f@nch@offset@elf
+                   \advance\headwidth\f@nch@offset@erf\hskip-\f@nch@offset@elf}
+\def\f@nch@setoffs{%
+  \f@nch@gbl\let\headwidth\f@nch@headwidth
+  \f@nch@gbl\def\f@nch@Oolh{\f@nch@offsolh}%
+  \f@nch@gbl\def\f@nch@Oelh{\f@nch@offselh}%
+  \f@nch@gbl\def\f@nch@Oorh{\hss}%
+  \f@nch@gbl\def\f@nch@Oerh{\hss}%
+  \f@nch@gbl\def\f@nch@Oolf{\f@nch@offsolf}%
+  \f@nch@gbl\def\f@nch@Oelf{\f@nch@offself}%
+  \f@nch@gbl\def\f@nch@Oorf{\hss}%
+  \f@nch@gbl\def\f@nch@Oerf{\hss}%
+}
+\newif\iff@nch@footnote
+\AtBeginDocument{%
+  \let\latex@makecol\@makecol
+  \def\@makecol{\ifvoid\footins\f@nch@footnotefalse\else\f@nch@footnotetrue\fi
+    \let\f@nch@topfloat\@toplist\let\f@nch@botfloat\@botlist\latex@makecol}%
+}
+\newcommand\iftopfloat[2]{\ifx\f@nch@topfloat\@empty #2\else #1\fi}%
+\newcommand\ifbotfloat[2]{\ifx\f@nch@botfloat\@empty #2\else #1\fi}%
+\newcommand\iffloatpage[2]{\if@fcolmade #1\else #2\fi}%
+\newcommand\iffootnote[2]{\iff@nch@footnote #1\else #2\fi}%
+\ifx\@temptokenb\undefined \csname newtoks\endcsname\@temptokenb\fi
+\newif\iff@nch@pagestyle@star
+\newcommand\fancypagestyle{%
+  \@ifstar{\f@nch@pagestyle@startrue\f@nch@pagestyle}%
+          {\f@nch@pagestyle@starfalse\f@nch@pagestyle}%
+}
+\newcommand\f@nch@pagestyle[1]{%
+  \@ifnextchar[{\f@nch@@pagestyle{#1}}{\f@nch@@pagestyle{#1}[f@nch@fancyproto]}%
+}
+\long\def\f@nch@@pagestyle#1[#2]#3{%
+  \@ifundefined{ps@#2}{%
+    \PackageError{fancyhdr}{\string\fancypagestyle: Unknown base page style `#2'}{}%
+  }{%
+    \@ifundefined{f@nch@ps@#2-is-fancyhdr}{%
+      \PackageError{fancyhdr}{\string\fancypagestyle: Base page style `#2' is not fancyhdr-based}{}%
+    }%
+    {%
+      \f@nch@pagestyle@setup
+      \def\temp@b{\@namedef{ps@#1}}%
+      \expandafter\temp@b\expandafter{\the\@temptokenb
+          \let\f@nch@gbl\relax\@nameuse{ps@#2}#3\relax}%
+      \@namedef{f@nch@ps@#1-is-fancyhdr}{}%
+    }%
+  }%
+}
+\newcommand\f@nch@pagestyle@setup{%
+  \iff@nch@pagestyle@star
+    \iff@nch@check\@temptokenb={\f@nch@checktrue}\else\@temptokenb={\f@nch@checkfalse}\fi
+    \@tfor\temp@a:=
+      \f@nch@olh\f@nch@och\f@nch@orh\f@nch@elh\f@nch@ech\f@nch@erh
+      \f@nch@olf\f@nch@ocf\f@nch@orf\f@nch@elf\f@nch@ecf\f@nch@erf
+      \f@nch@width@elh\f@nch@width@ech\f@nch@width@erh\f@nch@width@olh
+      \f@nch@width@och\f@nch@width@orh\f@nch@width@elf\f@nch@width@ecf
+      \f@nch@width@erf\f@nch@width@olf\f@nch@width@ocf\f@nch@width@orf
+      \f@nch@align@elh\f@nch@align@ech\f@nch@align@erh\f@nch@align@olh
+      \f@nch@align@och\f@nch@align@orh\f@nch@align@elf\f@nch@align@ecf
+      \f@nch@align@erf\f@nch@align@olf\f@nch@align@ocf\f@nch@align@orf
+      \f@nch@Oolh\f@nch@Oorh\f@nch@Oelh\f@nch@Oerh
+      \f@nch@Oolf\f@nch@Oorf\f@nch@Oelf\f@nch@Oerf
+      \f@nch@headinit\f@nch@footinit
+      \headrule\headrulewidth\footrule\footrulewidth
+    \do {%
+      \toks@=\expandafter\expandafter\expandafter{\temp@a}%
+      \toks@=\expandafter\expandafter\expandafter{%
+        \expandafter\expandafter\expandafter\def
+        \expandafter\expandafter\temp@a\expandafter{\the\toks@}}%
+      \edef\temp@b{\@temptokenb={\the\@temptokenb\the\toks@}}%
+      \temp@b
+    }%
+    \@tfor\temp@a:=
+      \f@nch@offset@olh\f@nch@offset@orh\f@nch@offset@elh\f@nch@offset@erh
+      \f@nch@offset@olf\f@nch@offset@orf\f@nch@offset@elf\f@nch@offset@erf
+    \do {%
+      \toks@=\expandafter\expandafter\expandafter{\expandafter\the\temp@a}%
+      \toks@=\expandafter\expandafter\expandafter{%
+        \expandafter\expandafter\expandafter\setlength
+        \expandafter\expandafter\temp@a\expandafter{\the\toks@}}%
+      \edef\temp@b{\@temptokenb={\the\@temptokenb\the\toks@}}%
+      \temp@b
+    }%
+  \else
+    \@temptokenb={}%
+  \fi
+}
+\newcommand\fancypagestyleassign[2]{%
+  \@ifundefined{ps@#2}{%
+    \PackageError{fancyhdr}{\string\fancypagestyleassign: Unknown page style `#2'}{}%
+    }{%
+     \expandafter\let
+       \csname ps@#1\expandafter\endcsname
+       \csname ps@#2\endcsname
+     \@ifundefined{f@nch@ps@#2-is-fancyhdr}{%
+       \expandafter\let\csname f@nch@ps@#1-is-fancyhdr\endcsname\@undefined
+     }{%
+       \@namedef{f@nch@ps@#1-is-fancyhdr}{}%
+     }%
+   }%
+}
+\fancypagestyle*{fancydefault}{\f@nch@initialise}
+\def\f@nchdrbox@topstrut{\vrule height\ht\strutbox width\z@}
+\def\f@nchdrbox@botstrut{\vrule depth\dp\strutbox width\z@}
+\def\f@nchdrbox@nostrut{\noalign{\vspace{0pt}}\let\f@nchdrbox@@crstrut\f@nchdrbox@botstrut}
+\NewDocumentCommand{\fancyhdrbox}{ O{cl} o m }{%
+\begingroup
+  \let\f@nchdrbox@@pre\f@nchdrbox@topstrut
+  \let\f@nchdrbox@@postx\f@nchdrbox@botstrut
+  \let\f@nchdrbox@@posty\relax
+  \let\f@nchdrbox@@crstrut\strut
+  \IfNoValueTF{#2}%
+    {\let\f@nchdrbox@@halignto\@empty}%
+    {\setlength\@tempdima{#2}%
+      \def\f@nchdrbox@@halignto{to\@tempdima}}%
+  \def\@tempa{#1}%
+  \ifx\@tempa\@empty
+    \f@nchdrbox@align cl\@nil{#3}%
+  \else
+    \f@nchdrbox@align #1\@nil{#3}%
+  \fi
+\endgroup
+}
+\protected\def\f@nchdrbox@cr{%
+  {\ifnum0=`}\fi\@ifstar\@f@nchdrbox@xcr\@f@nchdrbox@xcr}
+
+\def\@f@nchdrbox@xcr{%
+  \unskip\f@nchdrbox@@crstrut
+  \@ifnextchar[\@f@nchdrbox@argc{\ifnum0=`{\fi}\cr}%
+}
+
+\def\@f@nchdrbox@argc[#1]{%
+  \ifnum0=`{\fi}%
+    \ifdim #1>\z@
+      \unskip\@f@nchdrbox@xargc{#1}%
+    \else
+      \@f@nchdrbox@yargc{#1}%
+    \fi}
+
+\def\@f@nchdrbox@xargc#1{\@tempdima #1\advance\@tempdima \dp \strutbox
+   \vrule \@height\z@ \@depth\@tempdima \@width\z@ \cr}
+
+\def\@f@nchdrbox@yargc#1{\cr\noalign{\setlength\@tempdima{#1}\vskip\@tempdima}}
+\def\f@nchdrbox@T{\let\f@nchdrbox@@pre\f@nchdrbox@nostrut
+                  \f@nchdrbox@t}
+\def\f@nchdrbox@t{\def\f@nchdrbox@@v{t}\def\f@nchdrbox@@h{l}}
+\def\f@nchdrbox@c{\def\f@nchdrbox@@v{c}\def\f@nchdrbox@@h{c}}
+\def\f@nchdrbox@b{\def\f@nchdrbox@@v{b}\def\f@nchdrbox@@h{l}}
+\def\f@nchdrbox@B{\let\f@nchdrbox@@postx\relax
+                  \def\f@nchdrbox@@posty{\vspace{0pt}}%
+                  \f@nchdrbox@b}
+\long\def\f@nchdrbox@align#1#2\@nil#3{%
+  \f@nch@ifin{#1}{TtcbB}{%
+    \@nameuse{f@nchdrbox@#1}%
+    \def\@tempa{#2}%
+    \ifx\@tempa\@empty\else \def\f@nchdrbox@@h{#2}\fi
+  }%
+  {\def\f@nchdrbox@@v{c}\def\f@nchdrbox@@h{#1}}%
+  \expandafter\f@nch@ifin\expandafter{\f@nchdrbox@@h}{lcr}{}%
+  {\PackageError{fancyhdr}{\string\fancyhdrbox: Illegal char `\f@nchdrbox@@h'\MessageBreak
+                            in alignment argument}{}}%
+  \let\\\f@nchdrbox@cr
+  \setbox0=\if \f@nchdrbox@@v t\vtop
+  \else \vbox
+  \fi
+  {%
+     \ialign \f@nchdrbox@@halignto
+     \bgroup \relax
+     {\if \f@nchdrbox@@h l\hskip 1sp\else \hfil \fi
+       \ignorespaces ##\unskip
+       \if\f@nchdrbox@@h r\else \hfil \fi
+     }%
+     \tabskip\z@skip \cr
+     \f@nchdrbox@@pre
+     #3\unskip \f@nchdrbox@@postx
+     \crcr
+     \egroup
+     \f@nchdrbox@@posty
+  }%
+  \if\f@nchdrbox@@v c\@tempdima=\ht0\advance\@tempdima\dp0%
+    \ht0=0.5\@tempdima\dp0=0.5\@tempdima\fi
+  \leavevmode \box0
+}
+\@ifclassloaded{newlfm}
+{
+  \let\ps@@empty\f@nch@ps@empty
+  \AtBeginDocument{%
+    \renewcommand{\@zfancyhead}[5]{\relax\hbox to\headwidth{\f@nch@reset
+      \@zfancyvbox\headheight{\hbox
+        {\rlap{\parbox[b]{\headwidth}{\raggedright\f@nch@olh}}\hfill
+          \parbox[b]{\headwidth}{\centering\f@nch@olh}\hfill
+          \llap{\parbox[b]{\headwidth}{\raggedleft\f@nch@orh}}}%
+        \zheadrule}}\relax}%
+  }
+}
+{}
+\endinput
+%%
+%% End of file `fancyhdr.sty'.

icml2026.bst

@@ -0,0 +1,1443 @@
+%% File: `icml2025.bst'
+%% A modification of `plainnl.bst' for use with natbib package 
+%%
+%% Copyright 2010 Hal Daum\'e III
+%% Modified by J. Fürnkranz
+%% - Changed labels from (X and Y, 2000) to (X & Y, 2000)
+%% - Changed References to last name first and abbreviated first names.
+%% Modified by Iain Murray 2018 (who suggests adopting a standard .bst in future...)
+%% - Made it actually use abbreviated first names
+%%
+%% Copyright 1993-2007 Patrick W Daly
+%% Max-Planck-Institut f\"ur Sonnensystemforschung
+%% Max-Planck-Str. 2
+%% D-37191 Katlenburg-Lindau
+%% Germany
+%% E-mail: daly@mps.mpg.de
+%%
+%% This program can be redistributed and/or modified under the terms
+%% of the LaTeX Project Public License Distributed from CTAN
+%% archives in directory macros/latex/base/lppl.txt; either
+%% version 1 of the License, or any later version.
+%%
+ % Version and source file information:
+ % \ProvidesFile{icml2010.mbs}[2007/11/26 1.93 (PWD)]
+ %
+ % BibTeX `plainnat' family
+ %   version 0.99b for BibTeX versions 0.99a or later,
+ %   for LaTeX versions 2.09 and 2e.
+ %
+ % For use with the `natbib.sty' package; emulates the corresponding
+ %   member of the `plain' family, but with author-year citations.
+ %
+ % With version 6.0 of `natbib.sty', it may also be used for numerical
+ %   citations, while retaining the commands \citeauthor, \citefullauthor,
+ %   and \citeyear to print the corresponding information.
+ %
+ % For version 7.0 of `natbib.sty', the KEY field replaces missing
+ %   authors/editors, and the date is left blank in \bibitem.
+ %
+ % Includes field EID for the sequence/citation number of electronic journals
+ %  which is used instead of page numbers.
+ %
+ % Includes fields ISBN and ISSN.
+ %
+ % Includes field URL for Internet addresses.
+ %
+ % Includes field DOI for Digital Object Idenfifiers.
+ %
+ % Works best with the url.sty package of Donald Arseneau.
+ %
+ % Works with identical authors and year are further sorted by
+ %   citation key, to preserve any natural sequence.
+ %
+ENTRY
+  { address
+    author
+    booktitle
+    chapter
+    doi
+    eid
+    edition
+    editor
+    howpublished
+    institution
+    isbn
+    issn
+    journal
+    key
+    month
+    note
+    number
+    organization
+    pages
+    publisher
+    school
+    series
+    title
+    type
+    url
+    volume
+    year
+  }
+  {}
+  { label extra.label sort.label short.list }
+
+INTEGERS { output.state before.all mid.sentence after.sentence after.block }
+
+FUNCTION {init.state.consts}
+{ #0 'before.all :=
+  #1 'mid.sentence :=
+  #2 'after.sentence :=
+  #3 'after.block :=
+}
+
+STRINGS { s t }
+
+FUNCTION {output.nonnull}
+{ 's :=
+  output.state mid.sentence =
+    { ", " * write$ }
+    { output.state after.block =
+        { add.period$ write$
+          newline$
+          "\newblock " write$
+        }
+        { output.state before.all =
+            'write$
+            { add.period$ " " * write$ }
+          if$
+        }
+      if$
+      mid.sentence 'output.state :=
+    }
+  if$
+  s
+}
+
+FUNCTION {output}
+{ duplicate$ empty$
+    'pop$
+    'output.nonnull
+  if$
+}
+
+FUNCTION {output.check}
+{ 't :=
+  duplicate$ empty$
+    { pop$ "empty " t * " in " * cite$ * warning$ }
+    'output.nonnull
+  if$
+}
+
+FUNCTION {fin.entry}
+{ add.period$
+  write$
+  newline$
+}
+
+FUNCTION {new.block}
+{ output.state before.all =
+    'skip$
+    { after.block 'output.state := }
+  if$
+}
+
+FUNCTION {new.sentence}
+{ output.state after.block =
+    'skip$
+    { output.state before.all =
+        'skip$
+        { after.sentence 'output.state := }
+      if$
+    }
+  if$
+}
+
+FUNCTION {not}
+{   { #0 }
+    { #1 }
+  if$
+}
+
+FUNCTION {and}
+{   'skip$
+    { pop$ #0 }
+  if$
+}
+
+FUNCTION {or}
+{   { pop$ #1 }
+    'skip$
+  if$
+}
+
+FUNCTION {new.block.checka}
+{ empty$
+    'skip$
+    'new.block
+  if$
+}
+
+FUNCTION {new.block.checkb}
+{ empty$
+  swap$ empty$
+  and
+    'skip$
+    'new.block
+  if$
+}
+
+FUNCTION {new.sentence.checka}
+{ empty$
+    'skip$
+    'new.sentence
+  if$
+}
+
+FUNCTION {new.sentence.checkb}
+{ empty$
+  swap$ empty$
+  and
+    'skip$
+    'new.sentence
+  if$
+}
+
+FUNCTION {field.or.null}
+{ duplicate$ empty$
+    { pop$ "" }
+    'skip$
+  if$
+}
+
+FUNCTION {emphasize}
+{ duplicate$ empty$
+    { pop$ "" }
+    { "\emph{" swap$ * "}" * }
+  if$
+}
+
+INTEGERS { nameptr namesleft numnames }
+
+FUNCTION {format.names}
+{ 's :=
+  #1 'nameptr :=
+  s num.names$ 'numnames :=
+  numnames 'namesleft :=
+    { namesleft #0 > }
+    { s nameptr "{vv~}{ll}{, jj}{, f.}" format.name$ 't :=
+      nameptr #1 >
+        { namesleft #1 >
+            { ", " * t * }
+            { numnames #2 >
+                { "," * }
+                'skip$
+              if$
+              t "others" =
+                { " et~al." * }
+                { " and " * t * }
+              if$
+            }
+          if$
+        }
+        't
+      if$
+      nameptr #1 + 'nameptr :=
+      namesleft #1 - 'namesleft :=
+    }
+  while$
+}
+
+FUNCTION {format.key}
+{ empty$
+    { key field.or.null }
+    { "" }
+  if$
+}
+
+FUNCTION {format.authors}
+{ author empty$
+    { "" }
+    { author format.names }
+  if$
+}
+
+FUNCTION {format.editors}
+{ editor empty$
+    { "" }
+    { editor format.names
+      editor num.names$ #1 >
+        { " (eds.)" * }
+        { " (ed.)" * }
+      if$
+    }
+  if$
+}
+
+FUNCTION {format.isbn}
+{ isbn empty$
+    { "" }
+    { new.block "ISBN " isbn * }
+  if$
+}
+
+FUNCTION {format.issn}
+{ issn empty$
+    { "" }
+    { new.block "ISSN " issn * }
+  if$
+}
+
+FUNCTION {format.url}
+{ url empty$
+    { "" }
+    { new.block "URL \url{" url * "}" * }
+  if$
+}
+
+FUNCTION {format.doi}
+{ doi empty$
+    { "" }
+    { new.block "\doi{" doi * "}" * }
+  if$
+}
+
+FUNCTION {format.title}
+{ title empty$
+    { "" }
+    { title "t" change.case$ }
+  if$
+}
+
+FUNCTION {format.full.names}
+{'s :=
+  #1 'nameptr :=
+  s num.names$ 'numnames :=
+  numnames 'namesleft :=
+    { namesleft #0 > }
+    { s nameptr
+      "{vv~}{ll}" format.name$ 't :=
+      nameptr #1 >
+        {
+          namesleft #1 >
+            { ", " * t * }
+            {
+              numnames #2 >
+                { "," * }
+                'skip$
+              if$
+              t "others" =
+                { " et~al." * }
+                { " and " * t * }
+              if$
+            }
+          if$
+        }
+        't
+      if$
+      nameptr #1 + 'nameptr :=
+      namesleft #1 - 'namesleft :=
+    }
+  while$
+}
+
+FUNCTION {author.editor.full}
+{ author empty$
+    { editor empty$
+        { "" }
+        { editor format.full.names }
+      if$
+    }
+    { author format.full.names }
+  if$
+}
+
+FUNCTION {author.full}
+{ author empty$
+    { "" }
+    { author format.full.names }
+  if$
+}
+
+FUNCTION {editor.full}
+{ editor empty$
+    { "" }
+    { editor format.full.names }
+  if$
+}
+
+FUNCTION {make.full.names}
+{ type$ "book" =
+  type$ "inbook" =
+  or
+    'author.editor.full
+    { type$ "proceedings" =
+        'editor.full
+        'author.full
+      if$
+    }
+  if$
+}
+
+FUNCTION {output.bibitem}
+{ newline$
+  "\bibitem[" write$
+  label write$
+  ")" make.full.names duplicate$ short.list =
+     { pop$ }
+     { * }
+   if$
+  "]{" * write$
+  cite$ write$
+  "}" write$
+  newline$
+  ""
+  before.all 'output.state :=
+}
+
+FUNCTION {n.dashify}
+{ 't :=
+  ""
+    { t empty$ not }
+    { t #1 #1 substring$ "-" =
+        { t #1 #2 substring$ "--" = not
+            { "--" *
+              t #2 global.max$ substring$ 't :=
+            }
+            {   { t #1 #1 substring$ "-" = }
+                { "-" *
+                  t #2 global.max$ substring$ 't :=
+                }
+              while$
+            }
+          if$
+        }
+        { t #1 #1 substring$ *
+          t #2 global.max$ substring$ 't :=
+        }
+      if$
+    }
+  while$
+}
+
+FUNCTION {format.date}
+{ year duplicate$ empty$
+    { "empty year in " cite$ * warning$
+       pop$ "" }
+    'skip$
+  if$
+  month empty$
+    'skip$
+    { month
+      " " * swap$ *
+    }
+  if$
+  extra.label *
+}
+
+FUNCTION {format.btitle}
+{ title emphasize
+}
+
+FUNCTION {tie.or.space.connect}
+{ duplicate$ text.length$ #3 <
+    { "~" }
+    { " " }
+  if$
+  swap$ * *
+}
+
+FUNCTION {either.or.check}
+{ empty$
+    'pop$
+    { "can't use both " swap$ * " fields in " * cite$ * warning$ }
+  if$
+}
+
+FUNCTION {format.bvolume}
+{ volume empty$
+    { "" }
+    { "volume" volume tie.or.space.connect
+      series empty$
+        'skip$
+        { " of " * series emphasize * }
+      if$
+      "volume and number" number either.or.check
+    }
+  if$
+}
+
+FUNCTION {format.number.series}
+{ volume empty$
+    { number empty$
+        { series field.or.null }
+        { output.state mid.sentence =
+            { "number" }
+            { "Number" }
+          if$
+          number tie.or.space.connect
+          series empty$
+            { "there's a number but no series in " cite$ * warning$ }
+            { " in " * series * }
+          if$
+        }
+      if$
+    }
+    { "" }
+  if$
+}
+
+FUNCTION {format.edition}
+{ edition empty$
+    { "" }
+    { output.state mid.sentence =
+        { edition "l" change.case$ " edition" * }
+        { edition "t" change.case$ " edition" * }
+      if$
+    }
+  if$
+}
+
+INTEGERS { multiresult }
+
+FUNCTION {multi.page.check}
+{ 't :=
+  #0 'multiresult :=
+    { multiresult not
+      t empty$ not
+      and
+    }
+    { t #1 #1 substring$
+      duplicate$ "-" =
+      swap$ duplicate$ "," =
+      swap$ "+" =
+      or or
+        { #1 'multiresult := }
+        { t #2 global.max$ substring$ 't := }
+      if$
+    }
+  while$
+  multiresult
+}
+
+FUNCTION {format.pages}
+{ pages empty$
+    { "" }
+    { pages multi.page.check
+        { "pp.\ " pages n.dashify tie.or.space.connect }
+        { "pp.\ " pages tie.or.space.connect }
+      if$
+    }
+  if$
+}
+
+FUNCTION {format.eid}
+{ eid empty$
+    { "" }
+    { "art." eid tie.or.space.connect }
+  if$
+}
+
+FUNCTION {format.vol.num.pages}
+{ volume field.or.null
+  number empty$
+    'skip$
+    { "\penalty0 (" number * ")" * *
+      volume empty$
+        { "there's a number but no volume in " cite$ * warning$ }
+        'skip$
+      if$
+    }
+  if$
+  pages empty$
+    'skip$
+    { duplicate$ empty$
+        { pop$ format.pages }
+        { ":\penalty0 " * pages n.dashify * }
+      if$
+    }
+  if$
+}
+
+FUNCTION {format.vol.num.eid}
+{ volume field.or.null
+  number empty$
+    'skip$
+    { "\penalty0 (" number * ")" * *
+      volume empty$
+        { "there's a number but no volume in " cite$ * warning$ }
+        'skip$
+      if$
+    }
+  if$
+  eid empty$
+    'skip$
+    { duplicate$ empty$
+        { pop$ format.eid }
+        { ":\penalty0 " * eid * }
+      if$
+    }
+  if$
+}
+
+FUNCTION {format.chapter.pages}
+{ chapter empty$
+    'format.pages
+    { type empty$
+        { "chapter" }
+        { type "l" change.case$ }
+      if$
+      chapter tie.or.space.connect
+      pages empty$
+        'skip$
+        { ", " * format.pages * }
+      if$
+    }
+  if$
+}
+
+FUNCTION {format.in.ed.booktitle}
+{ booktitle empty$
+    { "" }
+    { editor empty$
+        { "In " booktitle emphasize * }
+        { "In " format.editors * ", " * booktitle emphasize * }
+      if$
+    }
+  if$
+}
+
+FUNCTION {empty.misc.check}
+{ author empty$ title empty$ howpublished empty$
+  month empty$ year empty$ note empty$
+  and and and and and
+  key empty$ not and
+    { "all relevant fields are empty in " cite$ * warning$ }
+    'skip$
+  if$
+}
+
+FUNCTION {format.thesis.type}
+{ type empty$
+    'skip$
+    { pop$
+      type "t" change.case$
+    }
+  if$
+}
+
+FUNCTION {format.tr.number}
+{ type empty$
+    { "Technical Report" }
+    'type
+  if$
+  number empty$
+    { "t" change.case$ }
+    { number tie.or.space.connect }
+  if$
+}
+
+FUNCTION {format.article.crossref}
+{ key empty$
+    { journal empty$
+        { "need key or journal for " cite$ * " to crossref " * crossref *
+          warning$
+          ""
+        }
+        { "In \emph{" journal * "}" * }
+      if$
+    }
+    { "In " }
+  if$
+  " \citet{" * crossref * "}" *
+}
+
+FUNCTION {format.book.crossref}
+{ volume empty$
+    { "empty volume in " cite$ * "'s crossref of " * crossref * warning$
+      "In "
+    }
+    { "Volume" volume tie.or.space.connect
+      " of " *
+    }
+  if$
+  editor empty$
+  editor field.or.null author field.or.null =
+  or
+    { key empty$
+        { series empty$
+            { "need editor, key, or series for " cite$ * " to crossref " *
+              crossref * warning$
+              "" *
+            }
+            { "\emph{" * series * "}" * }
+          if$
+        }
+        'skip$
+      if$
+    }
+    'skip$
+  if$
+  " \citet{" * crossref * "}" *
+}
+
+FUNCTION {format.incoll.inproc.crossref}
+{ editor empty$
+  editor field.or.null author field.or.null =
+  or
+    { key empty$
+        { booktitle empty$
+            { "need editor, key, or booktitle for " cite$ * " to crossref " *
+              crossref * warning$
+              ""
+            }
+            { "In \emph{" booktitle * "}" * }
+          if$
+        }
+        { "In " }
+      if$
+    }
+    { "In " }
+  if$
+  " \citet{" * crossref * "}" *
+}
+
+FUNCTION {article}
+{ output.bibitem
+  format.authors "author" output.check
+  author format.key output
+  new.block
+  format.title "title" output.check
+  new.block
+  crossref missing$
+    { journal emphasize "journal" output.check
+      eid empty$
+        { format.vol.num.pages output }
+        { format.vol.num.eid output }
+      if$
+      format.date "year" output.check
+    }
+    { format.article.crossref output.nonnull
+      eid empty$
+        { format.pages output }
+        { format.eid output }
+      if$
+    }
+  if$
+  format.issn output
+  format.doi output
+  format.url output
+  new.block
+  note output
+  fin.entry
+}
+
+FUNCTION {book}
+{ output.bibitem
+  author empty$
+    { format.editors "author and editor" output.check
+      editor format.key output
+    }
+    { format.authors output.nonnull
+      crossref missing$
+        { "author and editor" editor either.or.check }
+        'skip$
+      if$
+    }
+  if$
+  new.block
+  format.btitle "title" output.check
+  crossref missing$
+    { format.bvolume output
+      new.block
+      format.number.series output
+      new.sentence
+      publisher "publisher" output.check
+      address output
+    }
+    { new.block
+      format.book.crossref output.nonnull
+    }
+  if$
+  format.edition output
+  format.date "year" output.check
+  format.isbn output
+  format.doi output
+  format.url output
+  new.block
+  note output
+  fin.entry
+}
+
+FUNCTION {booklet}
+{ output.bibitem
+  format.authors output
+  author format.key output
+  new.block
+  format.title "title" output.check
+  howpublished address new.block.checkb
+  howpublished output
+  address output
+  format.date output
+  format.isbn output
+  format.doi output
+  format.url output
+  new.block
+  note output
+  fin.entry
+}
+
+FUNCTION {inbook}
+{ output.bibitem
+  author empty$
+    { format.editors "author and editor" output.check
+      editor format.key output
+    }
+    { format.authors output.nonnull
+      crossref missing$
+        { "author and editor" editor either.or.check }
+        'skip$
+      if$
+    }
+  if$
+  new.block
+  format.btitle "title" output.check
+  crossref missing$
+    { format.bvolume output
+      format.chapter.pages "chapter and pages" output.check
+      new.block
+      format.number.series output
+      new.sentence
+      publisher "publisher" output.check
+      address output
+    }
+    { format.chapter.pages "chapter and pages" output.check
+      new.block
+      format.book.crossref output.nonnull
+    }
+  if$
+  format.edition output
+  format.date "year" output.check
+  format.isbn output
+  format.doi output
+  format.url output
+  new.block
+  note output
+  fin.entry
+}
+
+FUNCTION {incollection}
+{ output.bibitem
+  format.authors "author" output.check
+  author format.key output
+  new.block
+  format.title "title" output.check
+  new.block
+  crossref missing$
+    { format.in.ed.booktitle "booktitle" output.check
+      format.bvolume output
+      format.number.series output
+      format.chapter.pages output
+      new.sentence
+      publisher "publisher" output.check
+      address output
+      format.edition output
+      format.date "year" output.check
+    }
+    { format.incoll.inproc.crossref output.nonnull
+      format.chapter.pages output
+    }
+  if$
+  format.isbn output
+  format.doi output
+  format.url output
+  new.block
+  note output
+  fin.entry
+}
+
+FUNCTION {inproceedings}
+{ output.bibitem
+  format.authors "author" output.check
+  author format.key output
+  new.block
+  format.title "title" output.check
+  new.block
+  crossref missing$
+    { format.in.ed.booktitle "booktitle" output.check
+      format.bvolume output
+      format.number.series output
+      format.pages output
+      address empty$
+        { organization publisher new.sentence.checkb
+          organization output
+          publisher output
+          format.date "year" output.check
+        }
+        { address output.nonnull
+          format.date "year" output.check
+          new.sentence
+          organization output
+          publisher output
+        }
+      if$
+    }
+    { format.incoll.inproc.crossref output.nonnull
+      format.pages output
+    }
+  if$
+  format.isbn output
+  format.doi output
+  format.url output
+  new.block
+  note output
+  fin.entry
+}
+
+FUNCTION {conference} { inproceedings }
+
+FUNCTION {manual}
+{ output.bibitem
+  format.authors output
+  author format.key output
+  new.block
+  format.btitle "title" output.check
+  organization address new.block.checkb
+  organization output
+  address output
+  format.edition output
+  format.date output
+  format.url output
+  new.block
+  note output
+  fin.entry
+}
+
+FUNCTION {mastersthesis}
+{ output.bibitem
+  format.authors "author" output.check
+  author format.key output
+  new.block
+  format.title "title" output.check
+  new.block
+  "Master's thesis" format.thesis.type output.nonnull
+  school "school" output.check
+  address output
+  format.date "year" output.check
+  format.url output
+  new.block
+  note output
+  fin.entry
+}
+
+FUNCTION {misc}
+{ output.bibitem
+  format.authors output
+  author format.key output
+  title howpublished new.block.checkb
+  format.title output
+  howpublished new.block.checka
+  howpublished output
+  format.date output
+  format.issn output
+  format.url output
+  new.block
+  note output
+  fin.entry
+  empty.misc.check
+}
+
+FUNCTION {phdthesis}
+{ output.bibitem
+  format.authors "author" output.check
+  author format.key output
+  new.block
+  format.btitle "title" output.check
+  new.block
+  "PhD thesis" format.thesis.type output.nonnull
+  school "school" output.check
+  address output
+  format.date "year" output.check
+  format.url output
+  new.block
+  note output
+  fin.entry
+}
+
+FUNCTION {proceedings}
+{ output.bibitem
+  format.editors output
+  editor format.key output
+  new.block
+  format.btitle "title" output.check
+  format.bvolume output
+  format.number.series output
+  address output
+  format.date "year" output.check
+  new.sentence
+  organization output
+  publisher output
+  format.isbn output
+  format.doi output
+  format.url output
+  new.block
+  note output
+  fin.entry
+}
+
+FUNCTION {techreport}
+{ output.bibitem
+  format.authors "author" output.check
+  author format.key output
+  new.block
+  format.title "title" output.check
+  new.block
+  format.tr.number output.nonnull
+  institution "institution" output.check
+  address output
+  format.date "year" output.check
+  format.url output
+  new.block
+  note output
+  fin.entry
+}
+
+FUNCTION {unpublished}
+{ output.bibitem
+  format.authors "author" output.check
+  author format.key output
+  new.block
+  format.title "title" output.check
+  new.block
+  note "note" output.check
+  format.date output
+  format.url output
+  fin.entry
+}
+
+FUNCTION {default.type} { misc }
+
+
+MACRO {jan} {"January"}
+
+MACRO {feb} {"February"}
+
+MACRO {mar} {"March"}
+
+MACRO {apr} {"April"}
+
+MACRO {may} {"May"}
+
+MACRO {jun} {"June"}
+
+MACRO {jul} {"July"}
+
+MACRO {aug} {"August"}
+
+MACRO {sep} {"September"}
+
+MACRO {oct} {"October"}
+
+MACRO {nov} {"November"}
+
+MACRO {dec} {"December"}
+
+
+
+MACRO {acmcs} {"ACM Computing Surveys"}
+
+MACRO {acta} {"Acta Informatica"}
+
+MACRO {cacm} {"Communications of the ACM"}
+
+MACRO {ibmjrd} {"IBM Journal of Research and Development"}
+
+MACRO {ibmsj} {"IBM Systems Journal"}
+
+MACRO {ieeese} {"IEEE Transactions on Software Engineering"}
+
+MACRO {ieeetc} {"IEEE Transactions on Computers"}
+
+MACRO {ieeetcad}
+ {"IEEE Transactions on Computer-Aided Design of Integrated Circuits"}
+
+MACRO {ipl} {"Information Processing Letters"}
+
+MACRO {jacm} {"Journal of the ACM"}
+
+MACRO {jcss} {"Journal of Computer and System Sciences"}
+
+MACRO {scp} {"Science of Computer Programming"}
+
+MACRO {sicomp} {"SIAM Journal on Computing"}
+
+MACRO {tocs} {"ACM Transactions on Computer Systems"}
+
+MACRO {tods} {"ACM Transactions on Database Systems"}
+
+MACRO {tog} {"ACM Transactions on Graphics"}
+
+MACRO {toms} {"ACM Transactions on Mathematical Software"}
+
+MACRO {toois} {"ACM Transactions on Office Information Systems"}
+
+MACRO {toplas} {"ACM Transactions on Programming Languages and Systems"}
+
+MACRO {tcs} {"Theoretical Computer Science"}
+
+
+READ
+
+FUNCTION {sortify}
+{ purify$
+  "l" change.case$
+}
+
+INTEGERS { len }
+
+FUNCTION {chop.word}
+{ 's :=
+  'len :=
+  s #1 len substring$ =
+    { s len #1 + global.max$ substring$ }
+    's
+  if$
+}
+
+FUNCTION {format.lab.names}
+{ 's :=
+  s #1 "{vv~}{ll}" format.name$
+  s num.names$ duplicate$
+  #2 >
+    { pop$ " et~al." * }
+    { #2 <
+        'skip$
+        { s #2 "{ff }{vv }{ll}{ jj}" format.name$ "others" =
+            { " et~al." * }
+            { " \& " * s #2 "{vv~}{ll}" format.name$ * }
+          if$
+        }
+      if$
+    }
+  if$
+}
+
+FUNCTION {author.key.label}
+{ author empty$
+    { key empty$
+        { cite$ #1 #3 substring$ }
+        'key
+      if$
+    }
+    { author format.lab.names }
+  if$
+}
+
+FUNCTION {author.editor.key.label}
+{ author empty$
+    { editor empty$
+        { key empty$
+            { cite$ #1 #3 substring$ }
+            'key
+          if$
+        }
+        { editor format.lab.names }
+      if$
+    }
+    { author format.lab.names }
+  if$
+}
+
+FUNCTION {author.key.organization.label}
+{ author empty$
+    { key empty$
+        { organization empty$
+            { cite$ #1 #3 substring$ }
+            { "The " #4 organization chop.word #3 text.prefix$ }
+          if$
+        }
+        'key
+      if$
+    }
+    { author format.lab.names }
+  if$
+}
+
+FUNCTION {editor.key.organization.label}
+{ editor empty$
+    { key empty$
+        { organization empty$
+            { cite$ #1 #3 substring$ }
+            { "The " #4 organization chop.word #3 text.prefix$ }
+          if$
+        }
+        'key
+      if$
+    }
+    { editor format.lab.names }
+  if$
+}
+
+FUNCTION {calc.short.authors}
+{ type$ "book" =
+  type$ "inbook" =
+  or
+    'author.editor.key.label
+    { type$ "proceedings" =
+        'editor.key.organization.label
+        { type$ "manual" =
+            'author.key.organization.label
+            'author.key.label
+          if$
+        }
+      if$
+    }
+  if$
+  'short.list :=
+}
+
+FUNCTION {calc.label}
+{ calc.short.authors
+  short.list
+  "("
+  *
+  year duplicate$ empty$
+  short.list key field.or.null = or
+     { pop$ "" }
+     'skip$
+  if$
+  *
+  'label :=
+}
+
+FUNCTION {sort.format.names}
+{ 's :=
+  #1 'nameptr :=
+  ""
+  s num.names$ 'numnames :=
+  numnames 'namesleft :=
+    { namesleft #0 > }
+    {
+      s nameptr "{vv{ } }{ll{ }}{  f{ }}{  jj{ }}" format.name$ 't :=
+      nameptr #1 >
+        {
+          "   "  *
+          namesleft #1 = t "others" = and
+            { "zzzzz" * }
+            { numnames #2 > nameptr #2 = and
+                { "zz" * year field.or.null * "   " * }
+                'skip$
+              if$
+              t sortify *
+            }
+          if$
+        }
+        { t sortify * }
+      if$
+      nameptr #1 + 'nameptr :=
+      namesleft #1 - 'namesleft :=
+    }
+  while$
+}
+
+FUNCTION {sort.format.title}
+{ 't :=
+  "A " #2
+    "An " #3
+      "The " #4 t chop.word
+    chop.word
+  chop.word
+  sortify
+  #1 global.max$ substring$
+}
+
+FUNCTION {author.sort}
+{ author empty$
+    { key empty$
+        { "to sort, need author or key in " cite$ * warning$
+          ""
+        }
+        { key sortify }
+      if$
+    }
+    { author sort.format.names }
+  if$
+}
+
+FUNCTION {author.editor.sort}
+{ author empty$
+    { editor empty$
+        { key empty$
+            { "to sort, need author, editor, or key in " cite$ * warning$
+              ""
+            }
+            { key sortify }
+          if$
+        }
+        { editor sort.format.names }
+      if$
+    }
+    { author sort.format.names }
+  if$
+}
+
+FUNCTION {author.organization.sort}
+{ author empty$
+    { organization empty$
+        { key empty$
+            { "to sort, need author, organization, or key in " cite$ * warning$
+              ""
+            }
+            { key sortify }
+          if$
+        }
+        { "The " #4 organization chop.word sortify }
+      if$
+    }
+    { author sort.format.names }
+  if$
+}
+
+FUNCTION {editor.organization.sort}
+{ editor empty$
+    { organization empty$
+        { key empty$
+            { "to sort, need editor, organization, or key in " cite$ * warning$
+              ""
+            }
+            { key sortify }
+          if$
+        }
+        { "The " #4 organization chop.word sortify }
+      if$
+    }
+    { editor sort.format.names }
+  if$
+}
+
+
+FUNCTION {presort}
+{ calc.label
+  label sortify
+  "    "
+  *
+  type$ "book" =
+  type$ "inbook" =
+  or
+    'author.editor.sort
+    { type$ "proceedings" =
+        'editor.organization.sort
+        { type$ "manual" =
+            'author.organization.sort
+            'author.sort
+          if$
+        }
+      if$
+    }
+  if$
+  "    "
+  *
+  year field.or.null sortify
+  *
+  "    "
+  *
+  cite$
+  *
+  #1 entry.max$ substring$
+  'sort.label :=
+  sort.label *
+  #1 entry.max$ substring$
+  'sort.key$ :=
+}
+
+ITERATE {presort}
+
+SORT
+
+STRINGS { longest.label last.label next.extra }
+
+INTEGERS { longest.label.width last.extra.num number.label }
+
+FUNCTION {initialize.longest.label}
+{ "" 'longest.label :=
+  #0 int.to.chr$ 'last.label :=
+  "" 'next.extra :=
+  #0 'longest.label.width :=
+  #0 'last.extra.num :=
+  #0 'number.label :=
+}
+
+FUNCTION {forward.pass}
+{ last.label label =
+    { last.extra.num #1 + 'last.extra.num :=
+      last.extra.num int.to.chr$ 'extra.label :=
+    }
+    { "a" chr.to.int$ 'last.extra.num :=
+      "" 'extra.label :=
+      label 'last.label :=
+    }
+  if$
+  number.label #1 + 'number.label :=
+}
+
+FUNCTION {reverse.pass}
+{ next.extra "b" =
+    { "a" 'extra.label := }
+    'skip$
+  if$
+  extra.label 'next.extra :=
+  extra.label
+  duplicate$ empty$
+    'skip$
+    { "{\natexlab{" swap$ * "}}" * }
+  if$
+  'extra.label :=
+  label extra.label * 'label :=
+}
+
+EXECUTE {initialize.longest.label}
+
+ITERATE {forward.pass}
+
+REVERSE {reverse.pass}
+
+FUNCTION {bib.sort.order}
+{ sort.label  'sort.key$ :=
+}
+
+ITERATE {bib.sort.order}
+
+SORT
+
+FUNCTION {begin.bib}
+{   preamble$ empty$
+    'skip$
+    { preamble$ write$ newline$ }
+  if$
+  "\begin{thebibliography}{" number.label int.to.str$ * "}" *
+  write$ newline$
+  "\providecommand{\natexlab}[1]{#1}"
+  write$ newline$
+  "\providecommand{\url}[1]{\texttt{#1}}"
+  write$ newline$
+  "\expandafter\ifx\csname urlstyle\endcsname\relax"
+  write$ newline$
+  "  \providecommand{\doi}[1]{doi: #1}\else"
+  write$ newline$
+  "  \providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi"
+  write$ newline$
+}
+
+EXECUTE {begin.bib}
+
+EXECUTE {init.state.consts}
+
+ITERATE {call.type$}
+
+FUNCTION {end.bib}
+{ newline$
+  "\end{thebibliography}" write$ newline$
+}
+
+EXECUTE {end.bib}

icml2026.sty

@@ -0,0 +1,767 @@
+% File: icml2026.sty (LaTeX style file for ICML-2026, version of 2025-10-29)
+
+% This file contains the LaTeX formatting parameters for a two-column
+% conference proceedings that is 8.5 inches wide by 11 inches high.
+%
+% Modified by Hanze Dong, Alberto Bietti, and Felix Berkenkamp, 2025
+% - Revert to times for better compatibility
+% - Updated years, volume, location
+% - Added preprint version
+% - Based on the suggestion from Johan Larsson:
+%   1. Added an end-of-document safety check to ensure the affiliations or notice footnote is printed:
+%      (1) Introduces a flag \newif\ificml@noticeprinted and sets it false by default.
+%      (2) At end of document, emits a package warning if \printAffiliationsAndNotice{...} was never called.
+%   2. \printAffiliationsAndNotice now sets the flag when called: Begins with \global\icml@noticeprintedtrue.
+% - Migrated to more recent version of fancyhdr for running title in header
+%
+% Modified by Johan Larsson, 2025
+% - Use newtx instead of times, aligning serif, sans-serif, typerwriter,
+%   and math fonts.
+% - Use caption package to setup captions instead of manually defining themanually defining them.
+% - Formatted icml2026.sty and example_paper.tex
+% - Use title case for section title to 2.9
+% - Replace subfigure package with subcaption in example, since it is
+%   designed to work together with the caption package (which is now required).
+% - Remove unused label in example
+%
+% Modified by Tegan Maharaj and Felix Berkenkamp 2025: changed years, volume, location
+%
+% Modified by Jonathan Scarlett 2024: changed years, volume, location
+%
+% Modified by Sivan Sabato 2023: changed years and volume number.
+% Modified by Jonathan Scarlett 2023: added page numbers to every page
+%
+% Modified by Csaba Szepesvari 2022: changed years, PMLR ref. Turned off checking marginparwidth
+%     as marginparwidth only controls the space available for margin notes and margin notes
+%     will NEVER be used anyways in submitted versions, so there is no reason one should
+%     check whether marginparwidth has been tampered with.
+%     Also removed pdfview=FitH from hypersetup as it did not do its job; the default choice is a bit better
+%     but of course the double-column format is not supported by this hyperlink preview functionality
+%     in a completely satisfactory fashion.
+% Modified by Gang Niu 2022: Changed color to xcolor
+%
+% Modified by Iain Murray 2018: changed years, location. Remove affiliation notes when anonymous.
+%     Move times dependency from .tex to .sty so fewer people delete it.
+%
+% Modified by Daniel Roy 2017: changed byline to use footnotes for affiliations, and removed emails
+%
+% Modified by Percy Liang 12/2/2013: changed the year, location from the previous template for ICML 2014
+
+% Modified by Fei Sha 9/2/2013: changed the year, location form the previous template for ICML 2013
+%
+% Modified by Fei Sha 4/24/2013: (1) remove the extra whitespace after the
+%     first author's email address (in %the camera-ready version) (2) change the
+%     Proceeding ... of ICML 2010 to 2014 so PDF's metadata will show up %
+%     correctly
+%
+% Modified by Sanjoy Dasgupta, 2013: changed years, location
+%
+% Modified by Francesco Figari, 2012: changed years, location
+%
+% Modified by Christoph Sawade and Tobias Scheffer, 2011: added line
+% numbers, changed years
+%
+% Modified by Hal Daume III, 2010: changed years, added hyperlinks
+%
+% Modified by Kiri Wagstaff, 2009: changed years
+%
+% Modified by Sam Roweis, 2008: changed years
+%
+% Modified by Ricardo Silva, 2007: update of the ifpdf verification
+%
+% Modified by Prasad Tadepalli and Andrew Moore, merely changing years.
+%
+% Modified by Kristian Kersting, 2005, based on Jennifer Dy's 2004 version
+% - running title. If the original title is to long or is breaking a line,
+%   use \icmltitlerunning{...} in the preamble to supply a shorter form.
+%   Added fancyhdr package to get a running head.
+% - Updated to store the page size because pdflatex does compile the
+%   page size into the pdf.
+%
+% Hacked by Terran Lane, 2003:
+% - Updated to use LaTeX2e style file conventions (ProvidesPackage,
+%   etc.)
+% - Added an ``appearing in'' block at the base of the first column
+%   (thus keeping the ``appearing in'' note out of the bottom margin
+%   where the printer should strip in the page numbers).
+% - Added a package option [accepted] that selects between the ``Under
+%   review'' notice (default, when no option is specified) and the
+%   ``Appearing in'' notice (for use when the paper has been accepted
+%   and will appear).
+%
+%   Originally created as:  ml2k.sty (LaTeX style file for ICML-2000)
+%   by P. Langley (12/23/99)
+
+%%%%%%%%%%%%%%%%%%%%
+%% This version of the style file supports both a ``review'' version
+%% and a ``final/accepted'' version.  The difference is only in the
+%% text that appears in the note at the bottom of the first column of
+%% the first page.  The default behavior is to print a note to the
+%% effect that the paper is under review and don't distribute it.  The
+%% final/accepted version prints an ``Appearing in'' note.  To get the
+%% latter behavior, in the calling file change the ``usepackage'' line
+%% from:
+%%	\usepackage{icml2025}
+%% to
+%%	\usepackage[accepted]{icml2025}
+%%%%%%%%%%%%%%%%%%%%
+
+\NeedsTeXFormat{LaTeX2e}
+\ProvidesPackage{icml2026}[2025/10/29 v2.0 ICML Conference Style File]
+
+% Before 2018, \usepackage{times} was in the example TeX, but inevitably
+% not everybody did it.
+% \RequirePackage[amsthm]{newtx}
+% 2025.11.6 revert to times for better compatibility
+\RequirePackage{times}
+
+% Use fancyhdr package
+\RequirePackage{fancyhdr}
+\RequirePackage{xcolor} % changed from color to xcolor (2021/11/24)
+\RequirePackage{algorithm}
+\RequirePackage{algorithmic}
+\RequirePackage{natbib}
+\RequirePackage{eso-pic} % used by \AddToShipoutPicture
+\RequirePackage{forloop}
+\RequirePackage{url}
+\RequirePackage{caption}
+
+%%%%%%%% Options
+\DeclareOption{accepted}{%
+  \renewcommand{\Notice@String}{\ICML@appearing}
+  \gdef\isaccepted{1}
+}
+
+% === Preprint option ===
+\DeclareOption{preprint}{%%
+  \renewcommand{\Notice@String}{\ICML@preprint}%%
+  \gdef\ispreprint{1}%%
+}
+
+% Distinct preprint footer text
+\newcommand{\ICML@preprint}{%
+  \textit{Preprint. \today.}%
+}
+
+\DeclareOption{nohyperref}{%
+  \gdef\nohyperref{1}
+}
+
+% Helper flag: show real authors for accepted or preprint
+\newif\ificmlshowauthors
+\icmlshowauthorsfalse
+
+%%%%%%%%%%%%%%%%%%%%
+% This string is printed at the bottom of the page for the
+% final/accepted version of the ``appearing in'' note.  Modify it to
+% change that text.
+%%%%%%%%%%%%%%%%%%%%
+\newcommand{\ICML@appearing}{\textit{Proceedings of the
+$\mathit{43}^{rd}$ International Conference on Machine Learning},
+Seoul, South Korea. PMLR 306, 2026.
+Copyright 2026 by the author(s).}
+
+%%%%%%%%%%%%%%%%%%%%
+% This string is printed at the bottom of the page for the draft/under
+% review version of the ``appearing in'' note.  Modify it to change
+% that text.
+%%%%%%%%%%%%%%%%%%%%
+\newcommand{\Notice@String}{Preliminary work.  Under review by the
+International Conference on Machine Learning (ICML)\@.  Do not distribute.}
+
+% Cause the declared options to actually be parsed and activated
+\ProcessOptions\relax
+
+% After options are processed, decide if authors should be visible
+\ifdefined\isaccepted \icmlshowauthorstrue \fi
+\ifdefined\ispreprint \icmlshowauthorstrue \fi
+
+\ifdefined\isaccepted\else\ifdefined\ispreprint\else\ifdefined\hypersetup
+  \hypersetup{pdfauthor={Anonymous Authors}}
+\fi\fi\fi
+
+\ifdefined\nohyperref\else\ifdefined\hypersetup
+  \definecolor{mydarkblue}{rgb}{0,0.08,0.45}
+  \hypersetup{ %
+    pdftitle={},
+    pdfsubject={Proceedings of the International Conference on Machine Learning 2026},
+    pdfkeywords={},
+    pdfborder=0 0 0,
+    pdfpagemode=UseNone,
+    colorlinks=true,
+    linkcolor=mydarkblue,
+    citecolor=mydarkblue,
+    filecolor=mydarkblue,
+    urlcolor=mydarkblue,
+    }
+  \fi
+\fi
+
+
+
+% Uncomment the following for debugging.  It will cause LaTeX to dump
+% the version of the ``appearing in'' string that will actually appear
+% in the document.
+%\typeout{>> Notice string='\Notice@String'}
+
+% Change citation commands to be more like old ICML styles
+\newcommand{\yrcite}[1]{\citeyearpar{#1}}
+\renewcommand{\cite}[1]{\citep{#1}}
+
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+% to ensure the letter format is used. pdflatex does compile the
+% page size into the pdf. This is done using \pdfpagewidth and
+% \pdfpageheight. As Latex does not know this directives, we first
+% check whether pdflatex or latex is used.
+%
+% Kristian Kersting 2005
+%
+% in order to account for the more recent use of pdfetex as the default
+% compiler, I have changed the pdf verification.
+%
+% Ricardo Silva 2007
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+\paperwidth=8.5in
+\paperheight=11in
+
+% old PDFLaTex verification, circa 2005
+%
+%\newif\ifpdf\ifx\pdfoutput\undefined
+%  \pdffalse % we are not running PDFLaTeX
+%\else
+%  \pdfoutput=1 % we are running PDFLaTeX
+%  \pdftrue
+%\fi
+
+\newif\ifpdf %adapted from ifpdf.sty
+\ifx\pdfoutput\undefined
+\else
+   \ifx\pdfoutput\relax
+   \else
+     \ifcase\pdfoutput
+     \else
+       \pdftrue
+     \fi
+   \fi
+\fi
+
+\ifpdf
+%    \pdfpagewidth=\paperwidth
+%    \pdfpageheight=\paperheight
+  \setlength{\pdfpagewidth}{8.5in}
+  \setlength{\pdfpageheight}{11in}
+\fi
+
+% Physical page layout
+
+\evensidemargin -0.23in
+\oddsidemargin -0.23in
+\setlength\textheight{9.0in}
+\setlength\textwidth{6.75in}
+\setlength\columnsep{0.25in}
+\setlength\headheight{10pt}
+\setlength\headsep{10pt}
+\addtolength{\topmargin}{-20pt}
+\addtolength{\topmargin}{-0.29in}
+
+% Historically many authors tried to include packages like geometry or fullpage,
+% which change the page layout. It either makes the proceedings inconsistent, or
+% wastes organizers' time chasing authors. So let's nip these problems in the
+% bud here. -- Iain Murray 2018.
+%\RequirePackage{printlen}
+\AtBeginDocument{%
+\newif\ifmarginsmessedwith
+\marginsmessedwithfalse
+\ifdim\oddsidemargin=-16.62178pt     \else oddsidemargin has been altered.\\ \marginsmessedwithtrue\fi
+\ifdim\headheight=10.0pt             \else headheight has been altered.\\ \marginsmessedwithtrue\fi
+\ifdim\textheight=650.43pt           \else textheight has been altered.\\ \marginsmessedwithtrue\fi
+\ifdim\marginparsep=11.0pt           \else marginparsep has been altered.\\ \marginsmessedwithtrue\fi
+\ifdim\footskip=25.0pt               \else footskip has been altered.\\ \marginsmessedwithtrue\fi
+\ifdim\hoffset=0.0pt                 \else hoffset has been altered.\\ \marginsmessedwithtrue\fi
+\ifdim\paperwidth=614.295pt          \else paperwidth has been altered.\\ \marginsmessedwithtrue\fi
+\ifdim\topmargin=-24.95781pt         \else topmargin has been altered.\\ \marginsmessedwithtrue\fi
+\ifdim\headsep=10.0pt                \else headsep has been altered.\\ \marginsmessedwithtrue\fi
+\ifdim\textwidth=487.8225pt          \else textwidth has been altered.\\ \marginsmessedwithtrue\fi
+\ifdim\marginparpush=5.0pt           \else marginparpush has been altered.\\ \marginsmessedwithtrue\fi
+\ifdim\voffset=0.0pt                 \else voffset has been altered.\\ \marginsmessedwithtrue\fi
+\ifdim\paperheight=794.96999pt       \else paperheight has been altered.\\ \marginsmessedwithtrue\fi
+\ifmarginsmessedwith
+
+\textbf{\large \em The page layout violates the ICML style.}
+
+Please do not change the page layout, or include packages like geometry,
+savetrees, or fullpage, which change it for you.
+
+We're not able to reliably undo arbitrary changes to the style. Please remove
+the offending package(s), or layout-changing commands and try again.
+
+\fi}
+
+
+%% The following is adapted from code in the acmconf.sty conference
+%% style file.  The constants in it are somewhat magical, and appear
+%% to work well with the two-column format on US letter paper that
+%% ICML uses, but will break if you change that layout, or if you use
+%% a longer block of text for the copyright notice string.  Fiddle with
+%% them if necessary to get the block to fit/look right.
+%%
+%% -- Terran Lane, 2003
+%%
+%% The following comments are included verbatim from acmconf.sty:
+%%
+%%% This section (written by KBT) handles the 1" box in the lower left
+%%% corner of the left column of the first page by creating a picture,
+%%% and inserting the predefined string at the bottom (with a negative
+%%% displacement to offset the space allocated for a non-existent
+%%% caption).
+%%%
+\def\ftype@copyrightbox{8}
+\def\@copyrightspace{
+\@float{copyrightbox}[b]
+\begin{center}
+\setlength{\unitlength}{1pc}
+\begin{picture}(20,1.5)
+\put(0,2.5){\line(1,0){4.818}}
+\put(0,0){\parbox[b]{19.75pc}{\small \Notice@String}}
+\end{picture}
+\end{center}
+\end@float}
+
+\setlength\footskip{25.0pt}
+\flushbottom \twocolumn
+\sloppy
+
+% Clear out the addcontentsline command
+\def\addcontentsline#1#2#3{}
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%% commands for formatting paper title, author names, and addresses.
+
+% box to check the size of the running head
+\newbox\titrun
+
+% general page style
+\pagestyle{fancy}
+\fancyhf{}
+\fancyfoot[C]{\thepage}
+% set the width of the head rule to 1 point
+\renewcommand{\headrulewidth}{1pt}
+
+% definition to set the head as running head in the preamble
+\def\icmltitlerunning#1{\gdef\@icmltitlerunning{#1}}
+
+% main definition adapting \icmltitle from 2004
+\long\def\icmltitle#1{%
+
+   %check whether @icmltitlerunning exists
+   % if not \icmltitle is used as running head
+   \ifx\undefined\@icmltitlerunning%
+      \gdef\@icmltitlerunning{#1}
+   \fi
+
+   %add it to pdf information
+  \ifdefined\nohyperref\else\ifdefined\hypersetup
+     \hypersetup{pdftitle={#1}}
+   \fi\fi
+
+   %get the dimension of the running title
+   \global\setbox\titrun=\vbox{\small\bf\@icmltitlerunning}
+
+   % error flag
+   \gdef\@runningtitleerror{0}
+
+    % running title too long
+    \ifdim\wd\titrun>\textwidth%
+      \gdef\@runningtitleerror{1}%
+      % running title breaks a line
+    \else \ifdim\ht\titrun>6.25pt
+    \gdef\@runningtitleerror{2}%
+      \fi
+    \fi
+
+       % if there is somthing wrong with the running title
+    \ifnum\@runningtitleerror>0
+      \typeout{}%
+                 \typeout{}%
+                 \typeout{*******************************************************}%
+                 \typeout{Title exceeds size limitations for running head.}%
+                 \typeout{Please supply a shorter form for the running head}
+                 \typeout{with \string\icmltitlerunning{...}\space prior to \string\begin{document}}%
+      \typeout{*******************************************************}%
+      \typeout{}%
+      \typeout{}%
+      % set default running title
+      \gdef\@icmltitlerunning{Title Suppressed Due to Excessive Size}
+    \fi
+
+    % no running title on the first page of the paper
+    \thispagestyle{plain}
+
+    {\center\baselineskip 18pt
+      \toptitlebar{\Large\bf #1}\bottomtitlebar}
+}
+
+% set running title header
+\fancyhead[C]{\small\bf\@icmltitlerunning}
+
+\gdef\icmlfullauthorlist{}
+\newcommand\addstringtofullauthorlist{\g@addto@macro\icmlfullauthorlist}
+\newcommand\addtofullauthorlist[1]{%
+  \ifdefined\icmlanyauthors%
+    \addstringtofullauthorlist{, #1}%
+  \else%
+    \addstringtofullauthorlist{#1}%
+    \gdef\icmlanyauthors{1}%
+  \fi%
+  \ifdefined\hypersetup%
+    \hypersetup{pdfauthor=\icmlfullauthorlist}%
+  \fi
+}
+
+\def\toptitlebar{\hrule height1pt \vskip .25in}
+\def\bottomtitlebar{\vskip .22in \hrule height1pt \vskip .3in}
+
+\newenvironment{icmlauthorlist}{%
+  \setlength\topsep{0pt}
+  \setlength\parskip{0pt}
+  \begin{center}
+    }{%
+  \end{center}
+}
+
+\newcounter{@affiliationcounter}
+\newcommand{\@pa}[1]{%
+  \ifcsname the@affil#1\endcsname
+    % do nothing
+  \else
+    \ifcsname @icmlsymbol#1\endcsname
+      % nothing
+    \else
+      \stepcounter{@affiliationcounter}%
+      \newcounter{@affil#1}%
+      \setcounter{@affil#1}{\value{@affiliationcounter}}%
+    \fi
+  \fi%
+  \ifcsname @icmlsymbol#1\endcsname
+    \textsuperscript{\csname @icmlsymbol#1\endcsname\,}%
+  \else
+    \textsuperscript{\arabic{@affil#1}\,}%
+  \fi
+}
+
+\newcommand{\icmlauthor}[2]{%
+  \ificmlshowauthors
+    \mbox{\bf #1}\,\@for\theaffil:=#2\do{\@pa{\theaffil}} \addtofullauthorlist{#1}%
+  \else
+    \ifdefined\@icmlfirsttime\else
+      \gdef\@icmlfirsttime{1}
+      \mbox{\bf Anonymous Authors}\@pa{@anon} \addtofullauthorlist{Anonymous Authors}
+    \fi
+  \fi
+}
+
+\newcommand{\icmlsetsymbol}[2]{%
+  \expandafter\gdef\csname @icmlsymbol#1\endcsname{#2}
+}
+
+\newcommand{\icmlaffiliation}[2]{%
+  \ificmlshowauthors
+    \ifcsname the@affil#1\endcsname
+      \expandafter\gdef\csname @affilname\csname the@affil#1\endcsname\endcsname{#2}%
+    \else
+      {\bf AUTHORERR: Error in use of \textbackslash{}icmlaffiliation command. Label ``#1'' not mentioned in some \textbackslash{}icmlauthor\{author name\}\{labels here\} command beforehand. }
+      \typeout{}%
+      \typeout{}%
+      \typeout{*******************************************************}%
+      \typeout{Affiliation label undefined. }%
+      \typeout{Make sure \string\icmlaffiliation\space follows }%
+      \typeout{all of \string\icmlauthor\space commands}%
+      \typeout{*******************************************************}%
+      \typeout{}%
+      \typeout{}%
+    \fi
+  \else
+    \expandafter\gdef\csname @affilname1\endcsname{Anonymous Institution, Anonymous City, Anonymous Region, Anonymous Country}
+  \fi
+}
+
+\newcommand{\icmlcorrespondingauthor}[2]{%
+  \ificmlshowauthors
+    \ifdefined\icmlcorrespondingauthor@text
+      \g@addto@macro\icmlcorrespondingauthor@text{, #1 \textless{}#2\textgreater{}}
+    \else
+      \gdef\icmlcorrespondingauthor@text{#1 \textless{}#2\textgreater{}}
+    \fi
+  \else
+    \gdef\icmlcorrespondingauthor@text{Anonymous Author \textless{}anon.email@domain.com\textgreater{}}
+  \fi
+}
+
+\newcommand{\icmlEqualContribution}{\textsuperscript{*}Equal contribution }
+
+
+% --- ICML 2026: ensure authors do not omit the affiliations/notice footnote ---
+\newif\ificml@noticeprinted
+\icml@noticeprintedfalse
+\AtEndDocument{%
+  \ificml@noticeprinted\relax\else
+    \PackageWarningNoLine{icml2026}{%
+      You did not call \string\printAffiliationsAndNotice{}. If you have no notice,%
+      call \string\printAffiliationsAndNotice\string{} (empty braces).%
+    }%
+  \fi
+}
+
+
+\newcounter{@affilnum}
+\newcommand{\printAffiliationsAndNotice}[1]{\global\icml@noticeprintedtrue%
+  \stepcounter{@affiliationcounter}%
+  {\let\thefootnote\relax\footnotetext{\hspace*{-\footnotesep}\ificmlshowauthors #1\fi%
+      \forloop{@affilnum}{1}{\value{@affilnum} < \value{@affiliationcounter}}{
+        \textsuperscript{\arabic{@affilnum}}\ifcsname @affilname\the@affilnum\endcsname%
+          \csname @affilname\the@affilnum\endcsname%
+        \else
+          {\bf AUTHORERR: Missing \textbackslash{}icmlaffiliation.}
+        \fi
+      }.%
+      \ifdefined\icmlcorrespondingauthor@text
+         { }Correspondence to: \icmlcorrespondingauthor@text.
+      \else
+        {\bf AUTHORERR: Missing \textbackslash{}icmlcorrespondingauthor.}
+      \fi
+
+      \ \\
+      \Notice@String
+    }
+  }
+}
+
+\long\def\icmladdress#1{%
+  {\bf The \textbackslash{}icmladdress command is no longer used.  See the example\_paper PDF .tex for usage of \textbackslash{}icmlauther and \textbackslash{}icmlaffiliation.}
+}
+
+%% keywords as first class citizens
+\def\icmlkeywords#1{%
+  \ifdefined\nohyperref\else\ifdefined\hypersetup
+      \hypersetup{pdfkeywords={#1}}
+    \fi\fi
+}
+
+% modification to natbib citations
+\setcitestyle{authoryear,round,citesep={;},aysep={,},yysep={;}}
+
+% Redefinition of the abstract environment.
+\renewenvironment{abstract}
+{%
+  \centerline{\large\bf Abstract}
+  \vspace{-0.12in}\begin{quote}}
+    {\par\end{quote}\vskip 0.12in}
+
+% numbered section headings with different treatment of numbers
+
+\def\@startsection#1#2#3#4#5#6{\if@noskipsec \leavevmode \fi
+  \par \@tempskipa #4\relax
+  \@afterindenttrue
+  \ifdim \@tempskipa <\z@ \@tempskipa -\@tempskipa \fi
+  \if@nobreak \everypar{}\else
+    \addpenalty{\@secpenalty}\addvspace{\@tempskipa}\fi \@ifstar
+  {\@ssect{#3}{#4}{#5}{#6}}{\@dblarg{\@sict{#1}{#2}{#3}{#4}{#5}{#6}}}}
+
+\def\@sict#1#2#3#4#5#6[#7]#8{\ifnum #2>\c@secnumdepth
+    \def\@svsec{}\else
+    \refstepcounter{#1}\edef\@svsec{\csname the#1\endcsname}\fi
+  \@tempskipa #5\relax
+  \ifdim \@tempskipa>\z@
+    \begingroup #6\relax
+    \@hangfrom{\hskip #3\relax\@svsec.~}{\interlinepenalty \@M #8\par}
+    \endgroup
+    \csname #1mark\endcsname{#7}\addcontentsline
+    {toc}{#1}{\ifnum #2>\c@secnumdepth \else
+        \protect\numberline{\csname the#1\endcsname}\fi
+      #7}\else
+    \def\@svsechd{#6\hskip #3\@svsec #8\csname #1mark\endcsname
+      {#7}\addcontentsline
+      {toc}{#1}{\ifnum #2>\c@secnumdepth \else
+          \protect\numberline{\csname the#1\endcsname}\fi
+        #7}}\fi
+  \@xsect{#5}}
+
+\def\@sect#1#2#3#4#5#6[#7]#8{\ifnum #2>\c@secnumdepth
+    \def\@svsec{}\else
+    \refstepcounter{#1}\edef\@svsec{\csname the#1\endcsname\hskip 0.4em }\fi
+  \@tempskipa #5\relax
+  \ifdim \@tempskipa>\z@
+    \begingroup #6\relax
+    \@hangfrom{\hskip #3\relax\@svsec}{\interlinepenalty \@M #8\par}
+    \endgroup
+    \csname #1mark\endcsname{#7}\addcontentsline
+    {toc}{#1}{\ifnum #2>\c@secnumdepth \else
+        \protect\numberline{\csname the#1\endcsname}\fi
+      #7}\else
+    \def\@svsechd{#6\hskip #3\@svsec #8\csname #1mark\endcsname
+      {#7}\addcontentsline
+      {toc}{#1}{\ifnum #2>\c@secnumdepth \else
+          \protect\numberline{\csname the#1\endcsname}\fi
+        #7}}\fi
+  \@xsect{#5}}
+
+% section headings with less space above and below them
+\def\thesection {\arabic{section}}
+\def\thesubsection {\thesection.\arabic{subsection}}
+\def\section{\@startsection{section}{1}{\z@}{-0.12in}{0.02in}
+  {\large\bf\raggedright}}
+\def\subsection{\@startsection{subsection}{2}{\z@}{-0.10in}{0.01in}
+  {\normalsize\bf\raggedright}}
+\def\subsubsection{\@startsection{subsubsection}{3}{\z@}{-0.08in}{0.01in}
+  {\normalsize\sc\raggedright}}
+\def\paragraph{\@startsection{paragraph}{4}{\z@}{1.5ex plus
+    0.5ex minus .2ex}{-1em}{\normalsize\bf}}
+\def\subparagraph{\@startsection{subparagraph}{5}{\z@}{1.5ex plus
+    0.5ex minus .2ex}{-1em}{\normalsize\bf}}
+
+% Footnotes
+\footnotesep 6.65pt %
+\skip\footins 9pt
+\def\footnoterule{\kern-3pt \hrule width 0.8in \kern 2.6pt }
+\setcounter{footnote}{0}
+
+% Lists and paragraphs
+\parindent 0pt
+\topsep 4pt plus 1pt minus 2pt
+\partopsep 1pt plus 0.5pt minus 0.5pt
+\itemsep 2pt plus 1pt minus 0.5pt
+\parsep 2pt plus 1pt minus 0.5pt
+\parskip 6pt
+
+\leftmargin 2em \leftmargini\leftmargin \leftmarginii 2em
+\leftmarginiii 1.5em \leftmarginiv 1.0em \leftmarginv .5em
+\leftmarginvi .5em
+\labelwidth\leftmargini\advance\labelwidth-\labelsep \labelsep 5pt
+
+\def\@listi{\leftmargin\leftmargini}
+\def\@listii{\leftmargin\leftmarginii
+  \labelwidth\leftmarginii\advance\labelwidth-\labelsep
+  \topsep 2pt plus 1pt minus 0.5pt
+  \parsep 1pt plus 0.5pt minus 0.5pt
+  \itemsep \parsep}
+\def\@listiii{\leftmargin\leftmarginiii
+  \labelwidth\leftmarginiii\advance\labelwidth-\labelsep
+  \topsep 1pt plus 0.5pt minus 0.5pt
+  \parsep \z@ \partopsep 0.5pt plus 0pt minus 0.5pt
+  \itemsep \topsep}
+\def\@listiv{\leftmargin\leftmarginiv
+  \labelwidth\leftmarginiv\advance\labelwidth-\labelsep}
+\def\@listv{\leftmargin\leftmarginv
+  \labelwidth\leftmarginv\advance\labelwidth-\labelsep}
+\def\@listvi{\leftmargin\leftmarginvi
+  \labelwidth\leftmarginvi\advance\labelwidth-\labelsep}
+
+\abovedisplayskip 7pt plus2pt minus5pt%
+\belowdisplayskip \abovedisplayskip
+\abovedisplayshortskip  0pt plus3pt%
+\belowdisplayshortskip  4pt plus3pt minus3pt%
+
+% Less leading in most fonts (due to the narrow columns)
+% The choices were between 1-pt and 1.5-pt leading
+\def\@normalsize{\@setsize\normalsize{11pt}\xpt\@xpt}
+\def\small{\@setsize\small{10pt}\ixpt\@ixpt}
+\def\footnotesize{\@setsize\footnotesize{10pt}\ixpt\@ixpt}
+\def\scriptsize{\@setsize\scriptsize{8pt}\viipt\@viipt}
+\def\tiny{\@setsize\tiny{7pt}\vipt\@vipt}
+\def\large{\@setsize\large{14pt}\xiipt\@xiipt}
+\def\Large{\@setsize\Large{16pt}\xivpt\@xivpt}
+\def\LARGE{\@setsize\LARGE{20pt}\xviipt\@xviipt}
+\def\huge{\@setsize\huge{23pt}\xxpt\@xxpt}
+\def\Huge{\@setsize\Huge{28pt}\xxvpt\@xxvpt}
+
+% Revised formatting for figure captions and table titles.
+\captionsetup{
+  skip=0.1in,
+  font=small,
+  labelfont={it,small},
+  labelsep=period
+}
+\captionsetup[table]{position=above}
+\captionsetup[figure]{position=below}
+
+\def\fnum@figure{Figure \thefigure}
+\def\fnum@table{Table \thetable}
+
+% Strut macros for skipping spaces above and below text in tables.
+\def\abovestrut#1{\rule[0in]{0in}{#1}\ignorespaces}
+\def\belowstrut#1{\rule[-#1]{0in}{#1}\ignorespaces}
+
+\def\abovespace{\abovestrut{0.20in}}
+\def\aroundspace{\abovestrut{0.20in}\belowstrut{0.10in}}
+\def\belowspace{\belowstrut{0.10in}}
+
+% Various personal itemization commands.
+\def\texitem#1{\par\noindent\hangindent 12pt
+  \hbox to 12pt {\hss #1 ~}\ignorespaces}
+\def\icmlitem{\texitem{$\bullet$}}
+
+% To comment out multiple lines of text.
+\long\def\comment#1{}
+
+%% Line counter (not in final version). Adapted from NIPS style file by Christoph Sawade
+
+% Vertical Ruler
+% This code is, largely, from the CVPR 2010 conference style file
+% ----- define vruler
+\makeatletter
+\newbox\icmlrulerbox
+\newcount\icmlrulercount
+\newdimen\icmlruleroffset
+\newdimen\cv@lineheight
+\newdimen\cv@boxheight
+\newbox\cv@tmpbox
+\newcount\cv@refno
+\newcount\cv@tot
+% NUMBER with left flushed zeros  \fillzeros[<WIDTH>]<NUMBER>
+\newcount\cv@tmpc@ \newcount\cv@tmpc
+\def\fillzeros[#1]#2{\cv@tmpc@=#2\relax\ifnum\cv@tmpc@<0\cv@tmpc@=-\cv@tmpc@\fi
+  \cv@tmpc=1 %
+  \loop\ifnum\cv@tmpc@<10 \else \divide\cv@tmpc@ by 10 \advance\cv@tmpc by 1 \fi
+  \ifnum\cv@tmpc@=10\relax\cv@tmpc@=11\relax\fi \ifnum\cv@tmpc@>10 \repeat
+  \ifnum#2<0\advance\cv@tmpc1\relax-\fi
+  \loop\ifnum\cv@tmpc<#1\relax0\advance\cv@tmpc1\relax\fi \ifnum\cv@tmpc<#1 \repeat
+  \cv@tmpc@=#2\relax\ifnum\cv@tmpc@<0\cv@tmpc@=-\cv@tmpc@\fi \relax\the\cv@tmpc@}%
+% \makevruler[<SCALE>][<INITIAL_COUNT>][<STEP>][<DIGITS>][<HEIGHT>]
+\def\makevruler[#1][#2][#3][#4][#5]{
+  \begingroup\offinterlineskip
+  \textheight=#5\vbadness=10000\vfuzz=120ex\overfullrule=0pt%
+  \global\setbox\icmlrulerbox=\vbox to \textheight{%
+    {
+        \parskip=0pt\hfuzz=150em\cv@boxheight=\textheight
+        \cv@lineheight=#1\global\icmlrulercount=#2%
+        \cv@tot\cv@boxheight\divide\cv@tot\cv@lineheight\advance\cv@tot2%
+        \cv@refno1\vskip-\cv@lineheight\vskip1ex%
+        \loop\setbox\cv@tmpbox=\hbox to0cm{\hfil {\hfil\fillzeros[#4]\icmlrulercount}}%
+        \ht\cv@tmpbox\cv@lineheight\dp\cv@tmpbox0pt\box\cv@tmpbox\break
+        \advance\cv@refno1\global\advance\icmlrulercount#3\relax
+        \ifnum\cv@refno<\cv@tot\repeat
+      }
+  }
+  \endgroup
+}%
+\makeatother
+% ----- end of vruler
+
+% \makevruler[<SCALE>][<INITIAL_COUNT>][<STEP>][<DIGITS>][<HEIGHT>]
+\def\icmlruler#1{\makevruler[12pt][#1][1][3][\textheight]\usebox{\icmlrulerbox}}
+\AddToShipoutPicture{%
+  \icmlruleroffset=\textheight
+  \advance\icmlruleroffset by 5.2pt % top margin
+  \color[rgb]{.7,.7,.7}
+  \ificmlshowauthors\else
+    \AtTextUpperLeft{%
+      \put(\LenToUnit{-35pt},\LenToUnit{-\icmlruleroffset}){%left ruler
+        \icmlruler{\icmlrulercount}}
+      %\put(\LenToUnit{1.04\textwidth},\LenToUnit{-\icmlruleroffset}){%right ruler
+      %  \icmlruler{\icmlrulercount}}
+    }
+  \fi
+}
+\endinput

output/final_fixes/results/pathway_specificity.json

@@ -0,0 +1,80 @@
+{
+  "pancreas_scPTR_gamma": {
+    "dataset": "pancreas",
+    "method": "scPTR_gamma",
+    "n_total_pathways": 40,
+    "n_generic": 12,
+    "n_tissue_specific": 13,
+    "n_non_generic": 28,
+    "n_unique_to_method": 2,
+    "generic_fraction": 0.3,
+    "mean_invisibility": 0.029185585677623742,
+    "mean_diff_genes": 6441.125
+  },
+  "pancreas_raw_u_s_ratio": {
+    "dataset": "pancreas",
+    "method": "raw_u_s_ratio",
+    "n_total_pathways": 40,
+    "n_generic": 8,
+    "n_tissue_specific": 10,
+    "n_non_generic": 32,
+    "n_unique_to_method": 2,
+    "generic_fraction": 0.2,
+    "mean_invisibility": 0.1864106031134724,
+    "mean_diff_genes": 5790.0
+  },
+  "pancreas_unspliced_only": {
+    "dataset": "pancreas",
+    "method": "unspliced_only",
+    "n_total_pathways": 40,
+    "n_generic": 4,
+    "n_tissue_specific": 14,
+    "n_non_generic": 36,
+    "n_unique_to_method": 8,
+    "generic_fraction": 0.1,
+    "mean_invisibility": 0.38459356455132365,
+    "mean_diff_genes": 7768.375
+  },
+  "dentate_gyrus_scPTR_gamma": {
+    "dataset": "dentate_gyrus",
+    "method": "scPTR_gamma",
+    "n_total_pathways": 50,
+    "n_generic": 27,
+    "n_tissue_specific": 9,
+    "n_non_generic": 23,
+    "n_unique_to_method": 2,
+    "generic_fraction": 0.54,
+    "mean_invisibility": 0.11836135569451883,
+    "mean_diff_genes": 1739.5454545454545
+  },
+  "dentate_gyrus_raw_u_s_ratio": {
+    "dataset": "dentate_gyrus",
+    "method": "raw_u_s_ratio",
+    "n_total_pathways": 50,
+    "n_generic": 24,
+    "n_tissue_specific": 7,
+    "n_non_generic": 26,
+    "n_unique_to_method": 5,
+    "generic_fraction": 0.48,
+    "mean_invisibility": 0.13538253624838859,
+    "mean_diff_genes": 1641.909090909091
+  },
+  "dentate_gyrus_unspliced_only": {
+    "dataset": "dentate_gyrus",
+    "method": "unspliced_only",
+    "n_total_pathways": 50,
+    "n_generic": 34,
+    "n_tissue_specific": 6,
+    "n_non_generic": 16,
+    "n_unique_to_method": 2,
+    "generic_fraction": 0.68,
+    "mean_invisibility": 0.15938146188855168,
+    "mean_diff_genes": 3161.3
+  },
+  "winner_counts": {
+    "scPTR_gamma": 2,
+    "raw_u_s_ratio": 0,
+    "unspliced_only": 7,
+    "tie": 10
+  }
+}

pyproject.toml

@@ -0,0 +1,44 @@
+[build-system]
+requires = ["setuptools>=64", "setuptools-scm"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "scptr"
+version = "0.1.0"
+description = "Single-cell post-transcriptional regulation analysis"
+requires-python = ">=3.9"
+dependencies = [
+    "anndata>=0.8",
+    "scanpy>=1.9",
+    "numpy>=1.21",
+    "scipy>=1.7",
+    "numba>=0.55",
+    "pandas>=1.3",
+    "matplotlib>=3.5",
+    "seaborn>=0.11",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.0",
+    "scikit-learn>=1.0",
+]
+datasets = [
+    "pooch>=1.6",
+]
+deep = [
+    "torch>=2.0",
+]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+"scptr.tools" = ["data/*.csv"]
+"scptr.datasets" = ["data/*.csv"]
+"scptr.benchmark" = ["data/*.txt"]
+
+[tool.pytest.ini_options]
+markers = [
+    "slow: marks tests as slow (deselect with '-m \"not slow\"')",
+]

src/scptr/benchmark/__init__.py

@@ -0,0 +1,14 @@
+"""Benchmark module for scPTR — validation and evaluation metrics."""
+
+from ._halflife_correlation import correlate_with_halflives
+from ._enrichment import are_enrichment, nmd_enrichment
+from ._robustness import subsampling_robustness
+from ._consistency import cross_dataset_consistency
+
+__all__ = [
+    "correlate_with_halflives",
+    "are_enrichment",
+    "nmd_enrichment",
+    "subsampling_robustness",
+    "cross_dataset_consistency",
+]

src/scptr/benchmark/_enrichment.py

@@ -0,0 +1,108 @@
+"""Enrichment analysis for AU-rich element (ARE) and NMD target genes."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import numpy as np
+import pandas as pd
+from anndata import AnnData
+from scipy import stats
+
+from .._constants import GAMMA
+from .._utils import get_layer, require_layers
+
+_DATA_DIR = Path(__file__).parent / "data"
+
+
+def _load_gene_list(filename: str) -> set[str]:
+    """Load a gene list from a bundled text file (one gene per line)."""
+    path = _DATA_DIR / filename
+    with open(path) as f:
+        return {line.strip() for line in f if line.strip()}
+
+
+def _enrichment_test(
+    adata: AnnData,
+    gene_set: set[str],
+    label: str,
+    min_gamma_fraction: float = 0.1,
+) -> dict:
+    """Mann-Whitney U test: do genes in gene_set have higher gamma?
+
+    Only tests genes with sufficient non-zero gamma signal.
+    """
+    require_layers(adata, GAMMA)
+
+    gamma = get_layer(adata, GAMMA)
+    median_gamma = np.median(gamma, axis=0)
+
+    # Filter to genes with reliable gamma estimates
+    nonzero_frac = (gamma > 0).mean(axis=0)
+    reliable = nonzero_frac >= min_gamma_fraction
+
+    gene_names = adata.var_names.tolist()
+    in_set = np.array([g in gene_set for g in gene_names])
+
+    # Apply reliability filter
+    in_set_reliable = in_set & reliable
+    background_reliable = (~in_set) & reliable
+
+    n_in = in_set_reliable.sum()
+    n_out = background_reliable.sum()
+
+    if n_in < 2 or n_out < 2:
+        return {
+            "label": label,
+            "n_genes_in_set": int(n_in),
+            "n_genes_in_set_unfiltered": int(in_set.sum()),
+            "n_genes_background": int(n_out),
+            "median_gamma_in_set": np.nan,
+            "median_gamma_background": np.nan,
+            "U_statistic": np.nan,
+            "p_value": np.nan,
+        }
+
+    gamma_in = median_gamma[in_set_reliable]
+    gamma_out = median_gamma[background_reliable]
+
+    U, p = stats.mannwhitneyu(gamma_in, gamma_out, alternative="greater")
+
+    return {
+        "label": label,
+        "n_genes_in_set": int(n_in),
+        "n_genes_in_set_unfiltered": int(in_set.sum()),
+        "n_genes_background": int(n_out),
+        "median_gamma_in_set": float(np.median(gamma_in)),
+        "median_gamma_background": float(np.median(gamma_out)),
+        "U_statistic": float(U),
+        "p_value": float(p),
+    }
+
+
+def are_enrichment(adata: AnnData) -> dict:
+    """Test whether ARE genes have higher gamma than background.
+
+    AU-rich elements (AREs) in 3' UTRs promote mRNA degradation.
+    Genes with AREs should have higher degradation rates (gamma).
+
+    Returns
+    -------
+    dict with test statistics including ``U_statistic`` and ``p_value``.
+    """
+    gene_set = _load_gene_list("are_genes.txt")
+    return _enrichment_test(adata, gene_set, "ARE")
+
+
+def nmd_enrichment(adata: AnnData) -> dict:
+    """Test whether NMD target genes have higher gamma than background.
+
+    Nonsense-mediated mRNA decay (NMD) targets should show higher
+    degradation rates.
+
+    Returns
+    -------
+    dict with test statistics including ``U_statistic`` and ``p_value``.
+    """
+    gene_set = _load_gene_list("nmd_genes.txt")
+    return _enrichment_test(adata, gene_set, "NMD")

src/scptr/benchmark/_halflife_correlation.py

@@ -0,0 +1,127 @@
+"""Correlation of estimated gamma with published mRNA half-lives."""
+
+from __future__ import annotations
+
+import numpy as np
+import pandas as pd
+from anndata import AnnData
+from scipy import stats
+
+from .._constants import GAMMA
+from .._utils import get_layer, require_layers
+
+
+def correlate_with_halflives(
+    adata: AnnData,
+    halflives_df: pd.DataFrame,
+    gene_col: str = "gene_symbol",
+    halflife_col: str = "half_life_hours",
+    min_gamma_fraction: float = 0.1,
+    case_insensitive: bool = True,
+) -> dict:
+    """Correlate per-gene median gamma with published mRNA half-lives.
+
+    Expects a negative correlation: high gamma (fast degradation) should
+    correspond to short half-lives.
+
+    Parameters
+    ----------
+    adata
+        Annotated data matrix with ``gamma`` layer.
+    halflives_df
+        DataFrame with gene symbols and half-life measurements.
+    gene_col
+        Column name for gene symbols in ``halflives_df``.
+    halflife_col
+        Column name for half-life values in ``halflives_df``.
+    min_gamma_fraction
+        Minimum fraction of cells with gamma > 0 for a gene to be
+        included in the correlation (default 0.1). Genes with too
+        few unspliced reads produce unreliable gamma estimates.
+    case_insensitive
+        Match gene symbols case-insensitively (default True). Useful
+        for cross-species comparisons (mouse Titlecase vs human UPPER).
+
+    Returns
+    -------
+    dict with keys: ``spearman_r``, ``spearman_p``, ``pearson_r``,
+    ``pearson_p``, ``n_genes``, ``n_genes_unfiltered``, ``matched_genes``.
+    """
+    require_layers(adata, GAMMA)
+
+    gamma = get_layer(adata, GAMMA)
+
+    # Filter genes: require minimum fraction of cells with non-zero gamma
+    nonzero_frac = (gamma > 0).mean(axis=0)
+    gene_mask = nonzero_frac >= min_gamma_fraction
+
+    median_gamma = np.median(gamma, axis=0)
+
+    gene_names = adata.var_names.tolist()
+    gamma_series = pd.Series(median_gamma, index=gene_names)
+    mask_series = pd.Series(gene_mask, index=gene_names)
+
+    hl_series = halflives_df.set_index(gene_col)[halflife_col]
+
+    if case_insensitive:
+        # Build uppercase-to-original mapping, match via uppercase
+        gamma_upper = {g.upper(): g for g in gene_names}
+        hl_upper = {}
+        for g in hl_series.index:
+            if isinstance(g, str):
+                hl_upper[g.upper()] = g
+        shared_upper = set(gamma_upper.keys()) & set(hl_upper.keys())
+        # Map back to original names
+        shared_all = pd.Index([gamma_upper[u] for u in shared_upper])
+        # Rebuild hl_series indexed by adata gene names
+        hl_remap = {gamma_upper[u]: hl_series[hl_upper[u]] for u in shared_upper}
+        hl_series = pd.Series(hl_remap)
+    else:
+        shared_all = gamma_series.index.intersection(hl_series.index)
+
+    # Apply gene quality filter
+    shared = shared_all[mask_series[shared_all].values]
+
+    n_unfiltered = len(shared_all)
+
+    if len(shared) < 3:
+        return {
+            "spearman_r": np.nan,
+            "spearman_p": np.nan,
+            "pearson_r": np.nan,
+            "pearson_p": np.nan,
+            "n_genes": len(shared),
+            "n_genes_unfiltered": n_unfiltered,
+            "matched_genes": shared.tolist(),
+        }
+
+    g = gamma_series[shared].values.astype(float)
+    h = hl_series[shared].values.astype(float)
+
+    # Remove NaN/Inf
+    valid = np.isfinite(g) & np.isfinite(h) & (g > 0) & (h > 0)
+    g, h = g[valid], h[valid]
+
+    if len(g) < 3:
+        return {
+            "spearman_r": np.nan,
+            "spearman_p": np.nan,
+            "pearson_r": np.nan,
+            "pearson_p": np.nan,
+            "n_genes": 0,
+            "n_genes_unfiltered": n_unfiltered,
+            "matched_genes": [],
+        }
+
+    sp_r, sp_p = stats.spearmanr(g, h)
+    pe_r, pe_p = stats.pearsonr(np.log1p(g), np.log1p(h))
+
+    return {
+        "spearman_r": float(sp_r),
+        "spearman_p": float(sp_p),
+        "pearson_r": float(pe_r),
+        "pearson_p": float(pe_p),
+        "n_genes": int(valid.sum()),
+        "n_genes_unfiltered": n_unfiltered,
+        "matched_genes": shared[valid].tolist(),
+    }

src/scptr/benchmark/_robustness.py

@@ -0,0 +1,77 @@
+"""Subsampling robustness analysis."""
+
+from __future__ import annotations
+
+import numpy as np
+import pandas as pd
+from anndata import AnnData
+from scipy import stats
+
+from .._constants import GAMMA
+from .._utils import get_layer, require_layers
+
+
+def subsampling_robustness(
+    adata: AnnData,
+    fractions: list[float] | None = None,
+    n_repeats: int = 3,
+    random_state: int = 0,
+) -> pd.DataFrame:
+    """Evaluate robustness of gamma estimates by subsampling cells.
+
+    For each fraction, subsample cells, rerun the pipeline, and correlate
+    the resulting per-gene median gamma with the full-data estimate.
+
+    Parameters
+    ----------
+    adata
+        Fully analyzed AnnData (must have ``gamma`` layer, ``Mu``/``Ms``
+        layers, and ``var['beta']``).
+    fractions
+        Cell fractions to test (default: [0.3, 0.5, 0.7, 0.9]).
+    n_repeats
+        Number of random repeats per fraction.
+    random_state
+        Base random seed.
+
+    Returns
+    -------
+    DataFrame with columns: ``fraction``, ``repeat``, ``spearman_r``,
+    ``pearson_r``, ``n_genes``.
+    """
+    require_layers(adata, GAMMA)
+
+    if fractions is None:
+        fractions = [0.3, 0.5, 0.7, 0.9]
+
+    gamma_full = get_layer(adata, GAMMA)
+    median_gamma_full = np.median(gamma_full, axis=0)
+
+    rng = np.random.RandomState(random_state)
+    records = []
+
+    for frac in fractions:
+        n_cells = max(int(adata.n_obs * frac), 10)
+        for rep in range(n_repeats):
+            idx = rng.choice(adata.n_obs, size=n_cells, replace=False)
+            gamma_sub = gamma_full[idx, :]
+            median_gamma_sub = np.median(gamma_sub, axis=0)
+
+            # Remove genes with zero variance
+            valid = (np.std(median_gamma_full) > 0) & (np.std(median_gamma_sub) > 0)
+            if not valid:
+                sp_r = pe_r = np.nan
+            else:
+                sp_r, _ = stats.spearmanr(median_gamma_full, median_gamma_sub)
+                pe_r, _ = stats.pearsonr(median_gamma_full, median_gamma_sub)
+
+            records.append({
+                "fraction": frac,
+                "repeat": rep,
+                "spearman_r": float(sp_r),
+                "pearson_r": float(pe_r),
+                "n_genes": int(adata.n_vars),
+                "n_cells_sampled": n_cells,
+            })
+
+    return pd.DataFrame(records)

src/scptr/benchmark/data/are_genes.txt

@@ -0,0 +1,139 @@
+TNF
+IL6
+IL8
+CSF2
+VEGFA
+MYC
+FOS
+JUN
+EGR1
+PTGS2
+IL2
+IL3
+IL4
+IL10
+IFNG
+CCL2
+CCL3
+CCL4
+CCL5
+CXCL1
+CXCL2
+CXCL8
+CXCL10
+MMP1
+MMP9
+SERPINE1
+PLAU
+PLAUR
+THBS1
+NOS2
+SOD2
+HMOX1
+DUSP1
+DUSP2
+ZFP36
+ZFP36L1
+ZFP36L2
+NFKBIA
+BCL2L1
+BIRC3
+CDKN1A
+GADD45A
+GADD45B
+ATF3
+JUNB
+FOSB
+NR4A1
+NR4A2
+NR4A3
+KLF2
+KLF4
+KLF6
+ETS1
+ETS2
+NFKB1
+NFKB2
+RELA
+RELB
+REL
+TRAF1
+TRAF2
+TNFAIP3
+TNFAIP6
+CD69
+CD83
+ICAM1
+VCAM1
+SELE
+SELP
+EDN1
+ET1
+HBEGF
+EREG
+AREG
+BTC
+TGFB1
+LIF
+OSM
+CNTF
+IL1B
+IL1A
+IL1RN
+IL12B
+IL15
+IL18
+IL23A
+IL27
+CD40LG
+TNFSF10
+FASLG
+NGF
+BDNF
+GDNF
+CNTF
+NTF3
+Tnf
+Il6
+Myc
+Fos
+Jun
+Egr1
+Vegfa
+Ptgs2
+Il2
+Il4
+Il10
+Ifng
+Ccl2
+Ccl3
+Ccl5
+Cxcl1
+Cxcl2
+Cxcl10
+Mmp9
+Serpine1
+Nos2
+Sod2
+Hmox1
+Dusp1
+Zfp36
+Nfkbia
+Cdkn1a
+Gadd45a
+Gadd45b
+Atf3
+Junb
+Fosb
+Nr4a1
+Nr4a2
+Klf2
+Klf4
+Klf6
+Nfkb1
+Rela
+Tnfaip3
+Icam1
+Tgfb1
+Il1b
+Il1a

src/scptr/benchmark/data/nmd_genes.txt

@@ -0,0 +1,118 @@
+GADD45B
+ATF4
+DDIT3
+PPP1R15A
+ASNS
+SESN2
+SLC7A11
+CTH
+SLC7A5
+HERPUD1
+DNAJB9
+HYOU1
+SEC24D
+TRIB3
+NUPR1
+CHAC1
+STC2
+GDF15
+INHBE
+VEGFA
+ADM
+ANGPTL4
+ERO1A
+P4HA1
+EGLN3
+BNIP3
+BNIP3L
+PDK1
+SLC2A1
+HK2
+PFKFB3
+ENO2
+ALDOA
+PGK1
+LDHA
+CA9
+SCG5
+SULF2
+TFR2
+KCNK3
+CLDN1
+SNHG1
+SNHG12
+SNHG15
+GAS5
+ZFAS1
+NEAT1
+MALAT1
+DANCR
+HOTAIR
+XIST
+KCNQ1OT1
+MEG3
+H19
+NORAD
+SMG1
+UPF1
+UPF2
+UPF3B
+SMG5
+SMG6
+SMG7
+SMG8
+SMG9
+EIF4A3
+MAGOH
+RBM8A
+CASC3
+RNPS1
+ACIN1
+SAP18
+PNN
+Gadd45b
+Atf4
+Ddit3
+Ppp1r15a
+Asns
+Sesn2
+Slc7a11
+Herpud1
+Dnajb9
+Trib3
+Nupr1
+Chac1
+Stc2
+Gdf15
+Vegfa
+Adm
+Angptl4
+Ero1a
+Bnip3
+Pdk1
+Slc2a1
+Hk2
+Pfkfb3
+Eno2
+Aldoa
+Pgk1
+Ldha
+Ca9
+Smg1
+Upf1
+Upf2
+Upf3b
+Smg5
+Smg6
+Smg7
+Eif4a3
+Magoh
+Rbm8a
+Rnps1
+Acin1
+Gas5
+Neat1
+Malat1
+Meg3
+H19
+Norad