add model card with yaml front-matter

README.md

@@ -1,593 +1,67 @@
-# PANDA — reproducibility recipe
-
-**PANDA** (Pan-tissue Adversarial Normalized Domain-invariant Anchored MLP) is a
-compact prototype-anchored MLP classifier for scRNA-seq cell identity across skin,
-hematopoietic, and pancreatic tissues, trained under a composite loss (supervised-
-contrastive + VICReg + sub-center angular prototype-InfoNCE + gradient-reversal
-dataset/depth adversary + HSIC depth-decorrelation + prototype-repulsion). Two
-input variants ship out of the box: **PANDA-PCA** (`PCA(50) -> trunk`) and
-**PANDA-Marker** (`[PCA(50) || marker_expr] -> trunk`); Marker beats PCA on 33/35
-fold-comparisons across the three systems.
-
-This README is a complete recipe to reproduce every result in `PAPER.tex` from a
-clean clone. All commands are copy-pasteable and use absolute paths.
-
-Repository layout:
-
-```
-panda/               model + losses + panda/markers.yaml
-scripts/pan_skin/    skin pipeline: download -> corpus -> train -> CV -> zero-shot
-scripts/pancreas/    pancreas pipeline (same shape)
-scripts/hematopoiesis/  HSC pipeline (same shape)
-scripts/common/      system-agnostic train / CV / zero-shot drivers
-scripts/analysis/    downstream discovery + interpretability
-scripts/figures/     paper + supplement figure builders
-data/corpus/{sys}/   downloaded + harmonized data
-data/raw/            per-dataset raw counts (not in git)
-checkpoints/{sys}/{variant}/panda_final.pt
-discovery/{sys}/{variant}/*.json,*.csv    all quantitative artefacts
-figures/             fig1_..fig6, PANDA_supplement.pdf, biology/*.pdf
-```
-
 ---
-
-## 1. Requirements
-
-- **Python**: 3.9+ (project is tested on 3.10).
-- **CUDA**: PyTorch 2.6.0 wheels — CUDA 12.1/12.4 runtime works.
-- **GPU**: 4x A100 40GB used for the paper; **1 GPU works** if you drop the
-  training batch size to `bs=64` (default is 256 for 4-GPU DataParallel).
-- **Disk**: ~400 GB (raw GEO tars + harmonized corpora + checkpoints).
-- **RAM**: ~64 GB (the pancreas HVG builder peaks near ~40 GB).
-
-Pinned runtime dependencies (from `pyproject.toml` / `requirements.txt`):
-
-```
-torch==2.6.0             transformers==5.6.2      peft==0.18.1
-scanpy==1.11.5           anndata==0.11.4          scvi-tools==1.3.3
-harmonypy==0.2.0         numpy>=1.24,<3.0         scipy>=1.10
-scikit-learn>=1.2        pandas>=1.5              matplotlib>=3.7
-seaborn>=0.12            umap-learn>=0.5          pyyaml>=6.0
-tqdm>=4.65               einops>=0.6              gdown>=5.0
-GEOparse>=2.0            leidenalg>=0.10          pynndescent>=0.5
-scikit-misc>=0.5
-```
-
-Optional extras: `bayes` (numpyro/jax for horseshoe), `gpu` (flash-attn 2.8.2),
-`viz` (plotly), `dev` (pytest, ruff, mypy).
-
-Install:
-
-```bash
-git clone <this-repo> /home/bcheng/PRISM
-cd /home/bcheng/PRISM
-pip install -e .
-# or, editable dev install:
-pip install -e ".[dev]"
-```
-
-### 1.1 LD_LIBRARY_PATH prefix (required for every PyTorch invocation)
-
-PyTorch 2.6 sparse ops load `libcusparseLt.so.0` which sits under the pip-installed
-`nvidia-cusparselt-cu12` package, and `scanpy` needs a modern `libstdc++`. Both
-paths must be exported **at the shell level, before Python starts**:
-
-```bash
-export LD_LIBRARY_PATH="$(python -c "import site,os; print(os.path.join(site.getsitepackages()[0],'nvidia','cusparselt','lib'))"):$LD_LIBRARY_PATH"
-```
-
-If you also have a conda env that ships a newer `libstdc++`, prepend it:
-
-```bash
-# example — path is machine-specific; drop it if your system libstdc++ is >= 3.4.30
-export LD_LIBRARY_PATH="/home/bcheng/.conda/pkgs/libstdcxx-15.2.0-h39759b7_7/lib:$LD_LIBRARY_PATH"
-```
-
-Every `bash scripts/*/run_all.sh` driver applies the same export automatically.
-
+license: mit
+tags:
+  - single-cell
+  - scRNA-seq
+  - cell-type-classification
+  - contrastive-learning
+  - domain-adaptation
+library_name: pytorch
+pipeline_tag: feature-extraction
 ---
 
-## 2. Data acquisition
+# PANDA — Pan-tissue Adversarial Normalized Domain-invariant Anchored MLP
 
-Every URL below is a public GEO/ArrayExpress FTP link. Raw data is **not**
-committed — it must be re-downloaded before anything else runs. Corpus builders
-expect files at `data/corpus/{system}/tier_{a,b,c,v2}/`.
-
-### Pan-skin (6 studies, 45,387 cells)
-
-| Study | GEO | Role |
-|---|---|---|
-| Sulic 2023 (E14.5 dorsal) | GSE212673 | anchor + held-out zero-shot |
-| Dingwall 2024 (En1-cKO) | GSE220977 | discovery target (paired with Aldrich GSE214695) |
-| Belote 2021 (human melanocyte) | GSE151091 | melanocyte anchor + held-out zero-shot |
-| Haensel/Annusver 2020 | GSE142471 | adult homeostasis + wound |
-| Joost 2016 | GSE67602 | Smart-seq2 platform anchor |
-| Sennett 2015 (bulk RNA) | GSE70288 | placode/dermal-condensate marker reference |
-| Han MCA 2018 (neonatal skin) | GSE108097 | Microwell-seq low-depth anchor |
-| Merkel 2022 | GSE201447 | touch dome / volar biology |
-| Aldrich 2023 (paired with Dingwall) | GSE214695 | En1-cKO snRNA-seq |
-
-```bash
-bash scripts/pan_skin/01_download_tier_a.sh   # Aldrich, Ge/Gupta, Joost, Haensel
-bash scripts/pan_skin/02_download_tier_b.sh   # MCA, WIHN, Ge/Fuchs, Merkel
-bash scripts/pan_skin/03_download_tier_c.sh   # Sennett, Tie, Wiedemann (bulk + human)
-# Dingwall / Sulic / Belote must be placed in data/raw/ manually — see repo notes
-```
+Prototype-anchored MLP classifier for scRNA-seq cell identity across skin, hematopoietic, and pancreatic tissues.
+Trained under a composite of SupCon + VICReg + prototype-InfoNCE + GRL dataset+depth adversary + HSIC decorrelation + prototype-repulsion.
 
-### Pan-hematopoietic (3 studies used in the paper, 192,833 cells)
+Two variants: **PANDA-PCA** and **PANDA-Marker** (adds a marker gene channel).
 
-| Study | GEO | Role |
-|---|---|---|
-| Weinreb LARRY 2020 | GSE140802 | corpus anchor |
-| Baccin whole-BM 2020 | GSE122465 | corpus (stromal + hematopoietic) |
-| Tabula Muris Senis BM 2020 | GSE132042 | corpus (paper-labeled) |
-| Nestorowa 2016 | GSE81682 | held-out zero-shot (Smart-seq2) |
-| Dahlin 2018 | GSE107727 | discovery target (Kit-W41 mutant) |
-| Paul 2015 (auxiliary) | GSE72857 | myeloid branch reference |
-| Tusi 2018 (auxiliary) | GSE89754 | erythroid trajectory |
+Code + paper: https://github.com/bryanc5864/PRISM
 
-```bash
-bash scripts/hematopoiesis/01_download.sh   # Paul, Nestorowa, Tusi, Dahlin
-bash scripts/hematopoiesis/02_download.sh   # Baccin whole-BM, TMS bone marrow
-```
+## Contents
 
-### Pan-pancreatic (6 studies, 120,611 cells)
+| Path | Description |
+|---|---|
+| `checkpoints/{system}/{pca,marker}/panda_final.pt` | Final trained weights per system × variant (6 core models) |
+| `checkpoints/pan_skin_dingwall_derm/panda_final.pt` | Line C: PANDA-Marker trained on Dingwall Derm labels |
+| `data/corpus/{system}/harmonized/` | Training corpora (h5ad + stats + PCA basis) |
+| `data/external_labels/` | Paper-supplement label files per source study |
+| `data/processed/dingwall_replica/` | Independent scanpy reproduction of Dingwall Seurat pipeline |
+| `discovery/` | Discovery-analysis outputs backing every paper claim |
+| `figures/` | Main + supplement + biology figures + merged PDFs |
+| `panda/`, `scripts/` | Model + analysis code (also on GitHub) |
+| `PAPER.tex`, `PAPER.pdf` | Manuscript |
+| `README.md` | Full end-to-end reproduction recipe |
 
-| Study | GEO | Role |
-|---|---|---|
-| Baron 2016 | GSE84133 | corpus mouse-train half + held-out mouse-test half |
-| Bastidas-Ponce 2019 (E15.5) | GSE132188 | corpus (endocrine progenitor time course) |
-| Byrnes 2018 | GSE101099 | corpus (paper-labeled subset) |
-| Yu 2021 | GSE139627 | corpus (paper-labeled Ngn3 lineage) |
-| Hrovatin MIA 2023 | GSE211796 | corpus (adult islet, paper-labeled) |
-| Veres 2019 | GSE114412 | 57,297 corpus + 12,297 held-out slice |
+## Quick fetch
 
 ```bash
-bash scripts/pancreas/01_download.sh   # Baron, Muraro, Grun, Byrnes, Veres
-bash scripts/pancreas/09_download.sh   # Yu Ngn3 seq-EP, MIA 4-month adult islet
-```
-
-Wall-clock: 2-6 h depending on bandwidth (GSE108097 MCA tar is ~9 GB, GSE140802
-Weinreb is ~14 GB, GSE114412 Veres is ~4 GB).
-
----
+# essentials only (~30 GB)
+huggingface-cli download bryan7264/PANDA \
+    --local-dir . \
+    --include "checkpoints/**" "data/corpus/**" "data/external_labels/**" "discovery/**"
 
-## 3. Corpus build (per system)
-
-Each system builds a `data/corpus/{system}/harmonized/corpus.h5ad` plus a shared
-HVG list, per-HVG mean/std, and a fitted PCA basis. Corpus is 100% paper-labeled;
-every cell carries a label from its source paper's supplementary table.
-
-### Pan-skin
-
-```bash
-python scripts/pan_skin/06_build_per_dataset_h5ads.py
-python scripts/pan_skin/07_build_shared_hvgs_and_pca.py
-python scripts/pan_skin/08_assign_labels.py
-python scripts/pan_skin/08b_curated_label_override.py
-python scripts/pan_skin/10_build_corpus.py                 # canonical corpus.h5ad
-python scripts/pan_skin/93_add_belote_anchor.py            # +Belote melanocyte anchor
+# individual system
+huggingface-cli download bryan7264/PANDA \
+    --local-dir . \
+    --include "data/corpus/pan_skin/**" "checkpoints/pan_skin/**"
 ```
 
-Wall-clock ~10-20 min (HVG + PCA is the expensive step).
+## Usage
 
-### Pan-hematopoietic
+```python
+import torch
+from panda.model import PANDAEncoder
 
-```bash
-python scripts/hematopoiesis/02_build_per_dataset.py
-python scripts/hematopoiesis/03_shared_hvgs_and_pca.py
-python scripts/hematopoiesis/10_build_corpus.py             # canonical corpus.h5ad
-python scripts/hematopoiesis/11_filter_paper_only.py        # enforce paper-labeled subset
-python scripts/hematopoiesis/09_retrain_with_nestorowa_anchor.py   # optional anchor
+ck = torch.load("checkpoints/pan_skin/marker/panda_final.pt", map_location="cpu",
+                weights_only=False)
+model = PANDAEncoder(variant="marker", n_pca=50,
+                     n_markers=len(ck["marker_genes"]),
+                     n_classes=len(ck["classes"]), n_sub=3,
+                     n_datasets=len(ck["datasets"]))
+model.load_state_dict(ck["model"])
+model.eval()
 ```
 
-Wall-clock ~15-30 min.
-
-### Pan-pancreatic
-
-```bash
-python scripts/pancreas/02_build_per_dataset.py
-python scripts/pancreas/03_shared_hvgs_and_pca.py
-python scripts/pancreas/04_assign_labels.py
-python scripts/pancreas/11_build_corpus.py                  # canonical corpus.h5ad
-python scripts/pancreas/08_add_baron_split.py               # 943-cell Baron test-half
-```
-
-Wall-clock ~30-60 min (peak ~40 GB RAM on the union HVG step).
-
-Also generate the held-out labeled slices used for zero-shot:
-
-```bash
-python scripts/common/generate_missing_holdouts.py
-```
-
-writes `data/corpus/hematopoiesis/held_out_labeled/nestorowa_GSE81682_test.h5ad`
-and `data/corpus/pan_skin/held_out_labeled/sulic_GSE212673_test.h5ad`.
-
----
-
-## 4. Training
-
-The **canonical trainer** is system-agnostic. It reads
-`data/corpus/{system}/harmonized/corpus.h5ad` and writes
-`checkpoints/{system}/{variant}/panda_final.pt`.
-
-```bash
-# 6 checkpoints total (3 systems x 2 variants). ~30-60 min each on 1x A100.
-python -m scripts.common.train_panda pan_skin      --variant pca    --epochs 8
-python -m scripts.common.train_panda pan_skin      --variant marker --epochs 8
-python -m scripts.common.train_panda hematopoiesis --variant pca    --epochs 8
-python -m scripts.common.train_panda hematopoiesis --variant marker --epochs 8
-python -m scripts.common.train_panda pancreas      --variant pca    --epochs 8
-python -m scripts.common.train_panda pancreas      --variant marker --epochs 8
-```
-
-Legacy per-system entry points also exist and are functionally equivalent for
-skin/HSC/pancreas single-variant training:
-`scripts/pan_skin/20_train_panda.py`, `scripts/hematopoiesis/05_train_panda.py`,
-`scripts/pancreas/05_train_panda.py`. Prefer `scripts.common.train_panda`.
-
----
-
-## 5. Held-out 5-fold cross-validation (Table 1)
-
-The paper's Table 1 CV block reads
-`discovery/{system}/{variant}/cv_5fold.json`. Two drivers exist:
-
-- **`scripts/common/run_cv.py`** — canonical, 5 epochs per fold, matches
-  paper numbers (mean acc / F1 / AUROC + per-class report).
-- `scripts/common/cv_holdout.py` — same architecture but supports GroupKFold
-  by dataset and a fuller 6-8 epoch curriculum; slower.
-
-Both accept `--systems` and `--variants`:
-
-```bash
-# canonical 5-fold CV for all 3 systems x 2 variants
-python -m scripts.common.run_cv --folds 5 --epochs 5
-```
-
-Per-system CV drivers also exist (`scripts/pan_skin/40_heldout_5fold_cv.py`,
-`scripts/hematopoiesis/07_heldout_5fold_cv.py`,
-`scripts/pancreas/07_heldout_5fold_cv.py`); they are single-variant, single-
-system alternatives.
-
-### Multi-seed rigor (35 fold-comparisons)
-
-The paper reports Marker beating PCA on 33/35 folds across seeded runs
-(2 seeds for skin+pancreas, 3 for HSC). Re-run with different `random_state`s;
-outputs write to `cv_5fold_seed{1,2}.json`:
-
-```bash
-python -m scripts.common.run_cv --folds 5 --epochs 5                          # seed 0
-# The script's seed hook is the random_state passed to StratifiedKFold + model
-# init; re-run with edits to `run_cv.py` main() (add `--seed N` arg) or wrap
-# a small loop. See scripts/common/cv_holdout.py for the current seed plumbing.
-```
-
----
-
-## 6. Held-out labeled zero-shot targets (Section 5)
-
-One driver runs every zero-shot target for both variants:
-
-```bash
-python -m scripts.common.run_all_zero_shot \
-    --systems pan_skin hematopoiesis pancreas \
-    --variants pca marker
-```
-
-writes `discovery/{system}/{variant}/{target}_predictions.csv` and
-`{target}_summary.json`. Individual per-target scripts exist for finer-grained
-control:
-
-| Target | Script | Populates |
-|---|---|---|
-| Baron test-half (pancreas, 943 cells) | `scripts/analysis/93_true_zero_shot_baron.py` | Table 1 Baron row + Sec 5.1 |
-| Nestorowa Smart-seq2 (HSC, 66 LT-HSC gated) | `scripts/analysis/94_true_zero_shot_nestorowa.py` | Sec 5.3 |
-| Sulic E14.5 dorsal skin (4,183 cells) | `scripts/common/run_all_zero_shot.py` (target `sulic`) | Sec 5.5 |
-| Belote melanocyte (6,088 cells) | `scripts/common/run_all_zero_shot.py` (target `belote`) | Sec 5.4 |
-| Veres held-out slice (12,297 pancreas) | `scripts/common/run_all_zero_shot.py` (target `veres`) | Sec 5.2 |
-| Dingwall (25,344 skin, discovery) | `scripts/pan_skin/30_zero_shot_aldrich.py` | Sec 6 |
-| Dahlin (61,122 HSC, discovery) | `scripts/common/run_all_zero_shot.py` (target `dahlin`) | Sec 7 |
-
-Adult-beta canonical panel validation on Veres:
-
-```bash
-python scripts/analysis/95_adult_beta_validation.py
-# -> discovery/pancreas/marker/95_adult_beta_validation.json
-```
-
----
-
-## 7. Discovery analyses
-
-Grouped by paper section. All write to `discovery/{system}/{variant}/`.
-
-### 7.1 Section 5 (held-out labeled) marker deep-dives
-
-```bash
-python scripts/analysis/90_dingwall_marker_deep_dive.py   # skin  -> 90_..._marker_deep_dive.csv
-python scripts/analysis/91_veres_marker_deep_dive.py      # pancreas
-python scripts/analysis/92_dahlin_marker_deep_dive.py     # HSC
-```
-
-### 7.2 Section 6 — Dingwall En1-cKO (skin)
-
-```bash
-python scripts/analysis/44_en1_cko_contrast.py            # class-level cKO vs WT contrast
-python scripts/analysis/45_marker_refinement.py           # per-class marker refinement
-python scripts/analysis/49_melanocyte_deep_dive.py        # melanocyte 2x expansion
-python scripts/analysis/57_multiclass_pathway_analysis.py # Dingwall pathway table
-python scripts/analysis/57_pathway_analysis.py            # symmetric 25+ module scoring, all systems
-python scripts/analysis/99_en1_dual_role_analysis.py      # spatial repressor / local activator
-python scripts/analysis/106_melanoblast_neural_crest.py   # Sec 6.5 MITF-axis vs NC reversion
-python scripts/analysis/107_dingwall_class_deg_count.py   # Sec 6.7 HF-placode DEG rank
-
-# EDEN validation — three complementary lines of evidence (Sec 6.4)
-python scripts/analysis/98_eden_posthoc_detection.py                    # Line A (null)
-python scripts/analysis/103_replicate_dingwall_seurat_pipeline.py       # Line B (Derm10 4.32x)
-python scripts/analysis/104_train_on_dingwall_derm_labels.py            # Line C (Derm10 5.20x)
-
-# Primary EDEN (Derm2) discovery — Sec 6.4.1
-python scripts/analysis/100_primary_eden_discovery.py                   # sub-cluster fibro predictions
-python scripts/analysis/101_primary_eden_derm_scoring.py                # score vs Data S1C panels
-python scripts/analysis/105_primary_eden_full_dermal.py                 # on full dermal denominator
-
-# Auxiliary: variant-A Dingwall training used for scope comparison
-python scripts/analysis/102_train_on_dingwall_variantA.py
-```
-
-### 7.3 Section 7 — Dahlin Kit-mutant (hematopoiesis)
-
-```bash
-python scripts/analysis/66_dahlin_kit_mutant.py           # class enrichment WT vs Kit-W41
-python scripts/analysis/67_dahlin_within_class.py         # within-class Wilcoxon DE
-python scripts/analysis/73_novel_populations_dahlin.py    # abstain-gated novel pops
-python scripts/analysis/108_dahlin_lineage_metabolism.py  # per-lineage OXPHOS/glycolysis
-```
-
-### 7.4 Section 7.4 — Veres held-out (pancreas)
-
-```bash
-python scripts/analysis/62_time_course_analysis.py        # class fractions across LARRY days (HSC time-course template)
-python scripts/analysis/109_veres_mature_beta.py          # Sec 5.2 adult MAFA/UCN3 quadrant
-python scripts/analysis/110_veres_polyhormonal_alpha.py   # Sec 5.2 polyhormonal alpha cluster
-```
-
-### 7.5 Section 8 — cross-system prototype geometry + interpretability
-
-```bash
-python scripts/analysis/70_prototype_geometry.py          # intra + cross-system cosine + eff-dim
-python scripts/analysis/72_emergent_axes.py               # within-class PCA of 128-d z
-python scripts/analysis/80_prototype_gene_attribution.py  # integrated gradients per prototype
-python scripts/analysis/81_counterfactual_knockouts.py    # per-gene KO delta on cosine
-python scripts/analysis/82_gene_coattribution_modules.py  # gene co-attribution modules
-python scripts/analysis/83_prototype_training_trajectory.py  # prototype drift across curriculum
-python scripts/analysis/84_adversary_purification.py      # test GRL adversary is at chance
-python scripts/analysis/85_hessian_gene_interactions.py   # second-order gene pair Hessian
-python scripts/analysis/63_nestorowa_zero_shot.py         # Nestorowa unlabeled discovery
-```
-
----
-
-## 8. Figures + supplement
-
-### Main-text figures (`figures/fig{1,2,3,4}_*.pdf`)
-
-```bash
-python scripts/figures/generate_paper_figures.py
-# fig1_confusion_matrices.pdf   3-panel per-class F1 confusion matrices
-# fig2_aldrich_volcano.pdf      melanocyte cKO vs WT volcano
-# fig3_dahlin_heatmap.pdf       within-class module-score heatmap
-# fig4_sharon_stage_stack.pdf   Veres per-stage class fractions
-```
-
-Figures 5/6 (En1-cKO + Kit-W41 recap) are built by the biology page pipeline
-below — the standalone `regen_fig5_fig6.py` referenced in older notes is not in
-the current tree; use the biology pipeline instead.
-
-### Supplement (`figures/PANDA_supplement.pdf`)
-
-```bash
-python scripts/figures/build_pca_vs_marker_umaps.py      # PCA vs Marker UMAPs per target
-python scripts/figures/build_figure_supplement.py        # combined supplement PDF
-```
-
-### Biology deep-dive supplement pages
-
-Cache UMAPs once, then build per-topic pages, then merge into the supplement:
-
-```bash
-python scripts/figures/biology_00_umap_cache.py
-python scripts/figures/biology_01_dingwall_umap.py
-python scripts/figures/biology_02_primary_eden.py
-python scripts/figures/biology_03_melanoblast_mitf.py
-python scripts/figures/biology_04_dahlin_metabolism.py
-python scripts/figures/biology_05_dahlin_composition.py
-python scripts/figures/biology_06_veres_beta_quadrant.py
-python scripts/figures/biology_07_veres_polyhormonal.py
-python scripts/figures/biology_08_prototype_geometry.py
-python scripts/figures/biology_99_merge_supplement.py    # appends into PANDA_supplement.pdf
-```
-
-Wall-clock: 20-40 min end-to-end (UMAPs dominate).
-
----
-
-## 9. PDF build
-
-The paper is a self-contained LaTeX document referencing PDFs in `figures/`:
-
-```bash
-cd /home/bcheng/PRISM
-pdflatex -interaction=nonstopmode PAPER.tex   # first pass  (writes .aux)
-pdflatex -interaction=nonstopmode PAPER.tex   # second pass (resolves refs)
-```
-
-`bibtex` is not required — the paper uses an embedded `thebibliography`.
-
----
-
-## 10. End-to-end make target
-
-The provided `Makefile` covers the canonical skin pipeline end-to-end:
-
-```bash
-make install        # pip install -e .
-make run            # bash scripts/pan_skin/run_all.sh — full skin pipeline
-make test-heldout   # scripts/pan_skin/40_heldout_5fold_cv.py
-make clean          # remove __pycache__ + *.pyc
-```
-
-For a full three-system reproduction, chain the per-section commands above.
-A minimal "everything" recipe:
-
-```bash
-# 1) data
-bash scripts/pan_skin/01_download_tier_a.sh
-bash scripts/pan_skin/02_download_tier_b.sh
-bash scripts/pan_skin/03_download_tier_c.sh
-bash scripts/hematopoiesis/01_download.sh
-bash scripts/hematopoiesis/02_download.sh
-bash scripts/pancreas/01_download.sh
-bash scripts/pancreas/09_download.sh
-
-# 2) corpora
-bash scripts/pan_skin/run_all.sh                     # includes build + train + skin CV
-python scripts/hematopoiesis/02_build_per_dataset.py
-python scripts/hematopoiesis/03_shared_hvgs_and_pca.py
-python scripts/hematopoiesis/10_build_corpus.py
-python scripts/hematopoiesis/11_filter_paper_only.py
-python scripts/pancreas/02_build_per_dataset.py
-python scripts/pancreas/03_shared_hvgs_and_pca.py
-python scripts/pancreas/04_assign_labels.py
-python scripts/pancreas/11_build_corpus.py
-python scripts/pancreas/08_add_baron_split.py
-python scripts/common/generate_missing_holdouts.py
-python scripts/pan_skin/93_add_belote_anchor.py
-
-# 3) all 6 training runs
-for sys in pan_skin hematopoiesis pancreas; do
-  for v in pca marker; do
-    python -m scripts.common.train_panda $sys --variant $v --epochs 8
-  done
-done
-
-# 4) CV + zero-shot
-python -m scripts.common.run_cv --folds 5 --epochs 5
-python -m scripts.common.run_all_zero_shot
-
-# 5) discovery
-bash scripts/common/rerun_all_discovery.sh           # drives scripts/analysis/* end-to-end
-python scripts/analysis/95_adult_beta_validation.py
-
-# 6) figures + PDF
-python scripts/figures/generate_paper_figures.py
-python scripts/figures/build_pca_vs_marker_umaps.py
-python scripts/figures/build_figure_supplement.py
-python scripts/figures/biology_00_umap_cache.py
-for i in 01 02 03 04 05 06 07 08; do
-  python scripts/figures/biology_${i}_*.py
-done
-python scripts/figures/biology_99_merge_supplement.py
-pdflatex -interaction=nonstopmode PAPER.tex && pdflatex -interaction=nonstopmode PAPER.tex
-```
-
----
-
-## 11. Trouble-shooting
-
-- **`libcusparseLt.so.0: cannot open shared object file`** — you forgot to
-  export `LD_LIBRARY_PATH` **before** Python started. The pip-installed
-  `nvidia-cusparselt-cu12` provides the library; PyTorch does not add its
-  path to the loader search. See section 1.1.
-- **`GLIBCXX_3.4.30 not found`** — your system `libstdc++` is too old; prepend
-  a newer `libstdc++.so.6`'s directory to `LD_LIBRARY_PATH`.
-- **Dingwall GSM -> genotype mapping** (frequent bug source): the correct map is
-  `WT = {GSM6833478, GSM6833479, GSM6833480, GSM6833481}`,
-  `cKO = {GSM6833482, GSM6833483}`.
-  GSM6833480/481 are `rttaControl` (Cre-negative WT), **not** cKO. Getting this
-  wrong flips every En1-cKO enrichment sign.
-- **Pancreas HVG builder OOM** — `scripts/pancreas/03_shared_hvgs_and_pca.py`
-  peaks near 40 GB RAM on the 6-study union. Run on a node with >= 64 GB.
-- **`data/raw` not in git** — it is git-ignored (14+ GB of GEO tars). Rerun
-  section 2 to repopulate.
-- **`stratified split failure` on rare classes** — small-support classes
-  (< 2 members per fold) are merged into the parent `canonical_label`. If a
-  fold still errors, check that `corpus.h5ad`'s `canonical_label` column has
-  the expected vocabulary; the paper vocab is the union enumerated in
-  `PAPER.tex` Sec 3.
-- **`n_conditions` mismatch** — `PANDAEncoder` reads it from
-  `len(datasets)` in the checkpoint; regenerate the checkpoint if you have
-  added/removed a dataset.
-- **`ContrastiveSampler` in 1-condition data** — auto-disabled when there is
-  only one condition; no config change needed.
-- **DataParallel batch-size** — default `bs=256` is calibrated for 4x A100
-  40 GB. Drop to `bs=64` for a single GPU or you will OOM inside the
-  sub-center prototype attention.
-- **`X_prism` not persisted** — after `20_train_panda.py` / `05_train_panda.py`
-  writes the checkpoint, the embedding is re-projected on demand in every
-  downstream analysis; if you want it cached, re-save the AnnData explicitly
-  via `adata.write_h5ad()`.
-
----
-
-## 12. Artefact index (from PAPER.tex Section 11)
-
-Every quantitative claim traces to one of:
-
-- **Model checkpoints**: `checkpoints/{system}/{variant}/panda_final.pt`
-- **Marker gene lists**: `panda/markers.yaml`
-- **Corpus builders**: `scripts/{pan_skin,hematopoiesis,pancreas}/`
-- **Unified trainer**: `scripts/common/train_panda.py`
-- **Unified 5-fold CV**: `scripts/common/run_cv.py` (or `cv_holdout.py`)
-- **Zero-shot driver**: `scripts/common/run_all_zero_shot.py`
-- **External label supplements**:
-  `data/external_labels/{dingwall_supp,haensel,joost2016,mca,mia,byrnes,yu,baccin,melanocyte_anchor}/`
-- **CV outputs**: `discovery/{system}/{variant}/cv_5fold{,_seed1,_seed2}.json`
-- **Zero-shot summaries**:
-  - `discovery/pancreas/{pca,marker}/baron_summary.json`
-  - `discovery/pancreas/{pca,marker}/veres_summary.json`
-  - `discovery/hematopoiesis/{pca,marker}/nestorowa_summary.json`
-  - `discovery/pan_skin/{pca,marker}/sulic_summary.json`
-  - `discovery/pan_skin/{pca,marker}/belote_summary.json`
-- **Adult-beta panel**: `discovery/pancreas/marker/95_adult_beta_validation.json`
-- **Pathway modules**: `discovery/{system}/marker/57_pathway_class_by_module_{padj,delta}.tsv`
-- **Dingwall EDEN validation**:
-  Line A `discovery/pan_skin/marker/98_eden_summary.json`;
-  Line B `data/processed/dingwall_replica/dingwall_replica.h5ad`,
-         `replica_cluster_20_qc.json`, `replica_marker_matches.csv`;
-  Line C `discovery/pan_skin/marker/104_dingwall_derm_summary.json` + prediction CSVs.
-- **Primary EDEN (Derm2)**:
-  `discovery/pan_skin/marker/100_primary_eden_discovery.csv`,
-  `100_primary_eden_summary.json`,
-  `101_derm_identity_summary.json`,
-  `101_derm_subcluster_scores.csv`.
-- **Dingwall other**:
-  `discovery/pan_skin/marker/57_pathway_analysis.csv`,
-  `90_dingwall_marker_deep_dive.csv`.
-- **Dahlin Kit-mutant**:
-  `discovery/hematopoiesis/marker/92_dahlin_marker_deep_dive.csv`, `dahlin_summary.json`.
-- **Veres deep-dive**: `discovery/pancreas/marker/91_veres_marker_deep_dive.csv`.
-
-Central architecture: `panda/model.py`. Composite loss lives in the same file
-(`supcon_loss`, `vicreg_loss`, `hsic_biased`, `subcenter_angular_infonce`,
-`prototype_repulsion`) and is imported as `from panda import PANDAEncoder, ...`.
-
-## Data mirror
-
-Full data (~195 GB corpus + raw + processed + external labels) is mirrored to Hugging Face at [bryan7264/PANDA](https://huggingface.co/bryan7264/PANDA). Fetch with:
-
-```bash
-huggingface-cli download bryan7264/PANDA --local-dir . --include "data/corpus/pan_skin/**"
-```
-
-Priority folders (fetch these first for the minimum-reproducible pipeline):
-- `data/corpus/{pan_skin,hematopoiesis,pancreas}/harmonized/` — training corpora
-- `data/external_labels/` — paper-supplement label files
-- `checkpoints/{pan_skin,hematopoiesis,pancreas}/marker/` — trained weights
-
-Bulk (only needed to reproduce corpus builds from scratch):
-- `data/raw/` — GEO downloads (regenerable from `scripts/*/03_download*.sh`)
-- `data/corpus/tier{1,2,3}/` — pretraining tier data
-- `data/processed/` — intermediate build artefacts
+See PAPER.pdf for full experimental setup and README.md for the reproduction recipe.