add full reproduction recipe as REPRODUCE.md

REPRODUCE.md

@@ -0,0 +1,593 @@
+# PANDA — reproducibility recipe
+
+**PANDA** (Pan-tissue Adversarial Normalized Domain-invariant Anchored MLP) is a
+compact prototype-anchored MLP classifier for scRNA-seq cell identity across skin,
+hematopoietic, and pancreatic tissues, trained under a composite loss (supervised-
+contrastive + VICReg + sub-center angular prototype-InfoNCE + gradient-reversal
+dataset/depth adversary + HSIC depth-decorrelation + prototype-repulsion). Two
+input variants ship out of the box: **PANDA-PCA** (`PCA(50) -> trunk`) and
+**PANDA-Marker** (`[PCA(50) || marker_expr] -> trunk`); Marker beats PCA on 33/35
+fold-comparisons across the three systems.
+
+This README is a complete recipe to reproduce every result in `PAPER.tex` from a
+clean clone. All commands are copy-pasteable and use absolute paths.
+
+Repository layout:
+
+```
+panda/               model + losses + panda/markers.yaml
+scripts/pan_skin/    skin pipeline: download -> corpus -> train -> CV -> zero-shot
+scripts/pancreas/    pancreas pipeline (same shape)
+scripts/hematopoiesis/  HSC pipeline (same shape)
+scripts/common/      system-agnostic train / CV / zero-shot drivers
+scripts/analysis/    downstream discovery + interpretability
+scripts/figures/     paper + supplement figure builders
+data/corpus/{sys}/   downloaded + harmonized data
+data/raw/            per-dataset raw counts (not in git)
+checkpoints/{sys}/{variant}/panda_final.pt
+discovery/{sys}/{variant}/*.json,*.csv    all quantitative artefacts
+figures/             fig1_..fig6, PANDA_supplement.pdf, biology/*.pdf
+```
+
+---
+
+## 1. Requirements
+
+- **Python**: 3.9+ (project is tested on 3.10).
+- **CUDA**: PyTorch 2.6.0 wheels — CUDA 12.1/12.4 runtime works.
+- **GPU**: 4x A100 40GB used for the paper; **1 GPU works** if you drop the
+  training batch size to `bs=64` (default is 256 for 4-GPU DataParallel).
+- **Disk**: ~400 GB (raw GEO tars + harmonized corpora + checkpoints).
+- **RAM**: ~64 GB (the pancreas HVG builder peaks near ~40 GB).
+
+Pinned runtime dependencies (from `pyproject.toml` / `requirements.txt`):
+
+```
+torch==2.6.0             transformers==5.6.2      peft==0.18.1
+scanpy==1.11.5           anndata==0.11.4          scvi-tools==1.3.3
+harmonypy==0.2.0         numpy>=1.24,<3.0         scipy>=1.10
+scikit-learn>=1.2        pandas>=1.5              matplotlib>=3.7
+seaborn>=0.12            umap-learn>=0.5          pyyaml>=6.0
+tqdm>=4.65               einops>=0.6              gdown>=5.0
+GEOparse>=2.0            leidenalg>=0.10          pynndescent>=0.5
+scikit-misc>=0.5
+```
+
+Optional extras: `bayes` (numpyro/jax for horseshoe), `gpu` (flash-attn 2.8.2),
+`viz` (plotly), `dev` (pytest, ruff, mypy).
+
+Install:
+
+```bash
+git clone <this-repo> /home/bcheng/PRISM
+cd /home/bcheng/PRISM
+pip install -e .
+# or, editable dev install:
+pip install -e ".[dev]"
+```
+
+### 1.1 LD_LIBRARY_PATH prefix (required for every PyTorch invocation)
+
+PyTorch 2.6 sparse ops load `libcusparseLt.so.0` which sits under the pip-installed
+`nvidia-cusparselt-cu12` package, and `scanpy` needs a modern `libstdc++`. Both
+paths must be exported **at the shell level, before Python starts**:
+
+```bash
+export LD_LIBRARY_PATH="$(python -c "import site,os; print(os.path.join(site.getsitepackages()[0],'nvidia','cusparselt','lib'))"):$LD_LIBRARY_PATH"
+```
+
+If you also have a conda env that ships a newer `libstdc++`, prepend it:
+
+```bash
+# example — path is machine-specific; drop it if your system libstdc++ is >= 3.4.30
+export LD_LIBRARY_PATH="/home/bcheng/.conda/pkgs/libstdcxx-15.2.0-h39759b7_7/lib:$LD_LIBRARY_PATH"
+```
+
+Every `bash scripts/*/run_all.sh` driver applies the same export automatically.
+
+---
+
+## 2. Data acquisition
+
+Every URL below is a public GEO/ArrayExpress FTP link. Raw data is **not**
+committed — it must be re-downloaded before anything else runs. Corpus builders
+expect files at `data/corpus/{system}/tier_{a,b,c,v2}/`.
+
+### Pan-skin (6 studies, 45,387 cells)
+
+| Study | GEO | Role |
+|---|---|---|
+| Sulic 2023 (E14.5 dorsal) | GSE212673 | anchor + held-out zero-shot |
+| Dingwall 2024 (En1-cKO) | GSE220977 | discovery target (paired with Aldrich GSE214695) |
+| Belote 2021 (human melanocyte) | GSE151091 | melanocyte anchor + held-out zero-shot |
+| Haensel/Annusver 2020 | GSE142471 | adult homeostasis + wound |
+| Joost 2016 | GSE67602 | Smart-seq2 platform anchor |
+| Sennett 2015 (bulk RNA) | GSE70288 | placode/dermal-condensate marker reference |
+| Han MCA 2018 (neonatal skin) | GSE108097 | Microwell-seq low-depth anchor |
+| Merkel 2022 | GSE201447 | touch dome / volar biology |
+| Aldrich 2023 (paired with Dingwall) | GSE214695 | En1-cKO snRNA-seq |
+
+```bash
+bash scripts/pan_skin/01_download_tier_a.sh   # Aldrich, Ge/Gupta, Joost, Haensel
+bash scripts/pan_skin/02_download_tier_b.sh   # MCA, WIHN, Ge/Fuchs, Merkel
+bash scripts/pan_skin/03_download_tier_c.sh   # Sennett, Tie, Wiedemann (bulk + human)
+# Dingwall / Sulic / Belote must be placed in data/raw/ manually — see repo notes
+```
+
+### Pan-hematopoietic (3 studies used in the paper, 192,833 cells)
+
+| Study | GEO | Role |
+|---|---|---|
+| Weinreb LARRY 2020 | GSE140802 | corpus anchor |
+| Baccin whole-BM 2020 | GSE122465 | corpus (stromal + hematopoietic) |
+| Tabula Muris Senis BM 2020 | GSE132042 | corpus (paper-labeled) |
+| Nestorowa 2016 | GSE81682 | held-out zero-shot (Smart-seq2) |
+| Dahlin 2018 | GSE107727 | discovery target (Kit-W41 mutant) |
+| Paul 2015 (auxiliary) | GSE72857 | myeloid branch reference |
+| Tusi 2018 (auxiliary) | GSE89754 | erythroid trajectory |
+
+```bash
+bash scripts/hematopoiesis/01_download.sh   # Paul, Nestorowa, Tusi, Dahlin
+bash scripts/hematopoiesis/02_download.sh   # Baccin whole-BM, TMS bone marrow
+```
+
+### Pan-pancreatic (6 studies, 120,611 cells)
+
+| Study | GEO | Role |
+|---|---|---|
+| Baron 2016 | GSE84133 | corpus mouse-train half + held-out mouse-test half |
+| Bastidas-Ponce 2019 (E15.5) | GSE132188 | corpus (endocrine progenitor time course) |
+| Byrnes 2018 | GSE101099 | corpus (paper-labeled subset) |
+| Yu 2021 | GSE139627 | corpus (paper-labeled Ngn3 lineage) |
+| Hrovatin MIA 2023 | GSE211796 | corpus (adult islet, paper-labeled) |
+| Veres 2019 | GSE114412 | 57,297 corpus + 12,297 held-out slice |
+
+```bash
+bash scripts/pancreas/01_download.sh   # Baron, Muraro, Grun, Byrnes, Veres
+bash scripts/pancreas/09_download.sh   # Yu Ngn3 seq-EP, MIA 4-month adult islet
+```
+
+Wall-clock: 2-6 h depending on bandwidth (GSE108097 MCA tar is ~9 GB, GSE140802
+Weinreb is ~14 GB, GSE114412 Veres is ~4 GB).
+
+---
+
+## 3. Corpus build (per system)
+
+Each system builds a `data/corpus/{system}/harmonized/corpus.h5ad` plus a shared
+HVG list, per-HVG mean/std, and a fitted PCA basis. Corpus is 100% paper-labeled;
+every cell carries a label from its source paper's supplementary table.
+
+### Pan-skin
+
+```bash
+python scripts/pan_skin/06_build_per_dataset_h5ads.py
+python scripts/pan_skin/07_build_shared_hvgs_and_pca.py
+python scripts/pan_skin/08_assign_labels.py
+python scripts/pan_skin/08b_curated_label_override.py
+python scripts/pan_skin/10_build_corpus.py                 # canonical corpus.h5ad
+python scripts/pan_skin/93_add_belote_anchor.py            # +Belote melanocyte anchor
+```
+
+Wall-clock ~10-20 min (HVG + PCA is the expensive step).
+
+### Pan-hematopoietic
+
+```bash
+python scripts/hematopoiesis/02_build_per_dataset.py
+python scripts/hematopoiesis/03_shared_hvgs_and_pca.py
+python scripts/hematopoiesis/10_build_corpus.py             # canonical corpus.h5ad
+python scripts/hematopoiesis/11_filter_paper_only.py        # enforce paper-labeled subset
+python scripts/hematopoiesis/09_retrain_with_nestorowa_anchor.py   # optional anchor
+```
+
+Wall-clock ~15-30 min.
+
+### Pan-pancreatic
+
+```bash
+python scripts/pancreas/02_build_per_dataset.py
+python scripts/pancreas/03_shared_hvgs_and_pca.py
+python scripts/pancreas/04_assign_labels.py
+python scripts/pancreas/11_build_corpus.py                  # canonical corpus.h5ad
+python scripts/pancreas/08_add_baron_split.py               # 943-cell Baron test-half
+```
+
+Wall-clock ~30-60 min (peak ~40 GB RAM on the union HVG step).
+
+Also generate the held-out labeled slices used for zero-shot:
+
+```bash
+python scripts/common/generate_missing_holdouts.py
+```
+
+writes `data/corpus/hematopoiesis/held_out_labeled/nestorowa_GSE81682_test.h5ad`
+and `data/corpus/pan_skin/held_out_labeled/sulic_GSE212673_test.h5ad`.
+
+---
+
+## 4. Training
+
+The **canonical trainer** is system-agnostic. It reads
+`data/corpus/{system}/harmonized/corpus.h5ad` and writes
+`checkpoints/{system}/{variant}/panda_final.pt`.
+
+```bash
+# 6 checkpoints total (3 systems x 2 variants). ~30-60 min each on 1x A100.
+python -m scripts.common.train_panda pan_skin      --variant pca    --epochs 8
+python -m scripts.common.train_panda pan_skin      --variant marker --epochs 8
+python -m scripts.common.train_panda hematopoiesis --variant pca    --epochs 8
+python -m scripts.common.train_panda hematopoiesis --variant marker --epochs 8
+python -m scripts.common.train_panda pancreas      --variant pca    --epochs 8
+python -m scripts.common.train_panda pancreas      --variant marker --epochs 8
+```
+
+Legacy per-system entry points also exist and are functionally equivalent for
+skin/HSC/pancreas single-variant training:
+`scripts/pan_skin/20_train_panda.py`, `scripts/hematopoiesis/05_train_panda.py`,
+`scripts/pancreas/05_train_panda.py`. Prefer `scripts.common.train_panda`.
+
+---
+
+## 5. Held-out 5-fold cross-validation (Table 1)
+
+The paper's Table 1 CV block reads
+`discovery/{system}/{variant}/cv_5fold.json`. Two drivers exist:
+
+- **`scripts/common/run_cv.py`** — canonical, 5 epochs per fold, matches
+  paper numbers (mean acc / F1 / AUROC + per-class report).
+- `scripts/common/cv_holdout.py` — same architecture but supports GroupKFold
+  by dataset and a fuller 6-8 epoch curriculum; slower.
+
+Both accept `--systems` and `--variants`:
+
+```bash
+# canonical 5-fold CV for all 3 systems x 2 variants
+python -m scripts.common.run_cv --folds 5 --epochs 5
+```
+
+Per-system CV drivers also exist (`scripts/pan_skin/40_heldout_5fold_cv.py`,
+`scripts/hematopoiesis/07_heldout_5fold_cv.py`,
+`scripts/pancreas/07_heldout_5fold_cv.py`); they are single-variant, single-
+system alternatives.
+
+### Multi-seed rigor (35 fold-comparisons)
+
+The paper reports Marker beating PCA on 33/35 folds across seeded runs
+(2 seeds for skin+pancreas, 3 for HSC). Re-run with different `random_state`s;
+outputs write to `cv_5fold_seed{1,2}.json`:
+
+```bash
+python -m scripts.common.run_cv --folds 5 --epochs 5                          # seed 0
+# The script's seed hook is the random_state passed to StratifiedKFold + model
+# init; re-run with edits to `run_cv.py` main() (add `--seed N` arg) or wrap
+# a small loop. See scripts/common/cv_holdout.py for the current seed plumbing.
+```
+
+---
+
+## 6. Held-out labeled zero-shot targets (Section 5)
+
+One driver runs every zero-shot target for both variants:
+
+```bash
+python -m scripts.common.run_all_zero_shot \
+    --systems pan_skin hematopoiesis pancreas \
+    --variants pca marker
+```
+
+writes `discovery/{system}/{variant}/{target}_predictions.csv` and
+`{target}_summary.json`. Individual per-target scripts exist for finer-grained
+control:
+
+| Target | Script | Populates |
+|---|---|---|
+| Baron test-half (pancreas, 943 cells) | `scripts/analysis/93_true_zero_shot_baron.py` | Table 1 Baron row + Sec 5.1 |
+| Nestorowa Smart-seq2 (HSC, 66 LT-HSC gated) | `scripts/analysis/94_true_zero_shot_nestorowa.py` | Sec 5.3 |
+| Sulic E14.5 dorsal skin (4,183 cells) | `scripts/common/run_all_zero_shot.py` (target `sulic`) | Sec 5.5 |
+| Belote melanocyte (6,088 cells) | `scripts/common/run_all_zero_shot.py` (target `belote`) | Sec 5.4 |
+| Veres held-out slice (12,297 pancreas) | `scripts/common/run_all_zero_shot.py` (target `veres`) | Sec 5.2 |
+| Dingwall (25,344 skin, discovery) | `scripts/pan_skin/30_zero_shot_aldrich.py` | Sec 6 |
+| Dahlin (61,122 HSC, discovery) | `scripts/common/run_all_zero_shot.py` (target `dahlin`) | Sec 7 |
+
+Adult-beta canonical panel validation on Veres:
+
+```bash
+python scripts/analysis/95_adult_beta_validation.py
+# -> discovery/pancreas/marker/95_adult_beta_validation.json
+```
+
+---
+
+## 7. Discovery analyses
+
+Grouped by paper section. All write to `discovery/{system}/{variant}/`.
+
+### 7.1 Section 5 (held-out labeled) marker deep-dives
+
+```bash
+python scripts/analysis/90_dingwall_marker_deep_dive.py   # skin  -> 90_..._marker_deep_dive.csv
+python scripts/analysis/91_veres_marker_deep_dive.py      # pancreas
+python scripts/analysis/92_dahlin_marker_deep_dive.py     # HSC
+```
+
+### 7.2 Section 6 — Dingwall En1-cKO (skin)
+
+```bash
+python scripts/analysis/44_en1_cko_contrast.py            # class-level cKO vs WT contrast
+python scripts/analysis/45_marker_refinement.py           # per-class marker refinement
+python scripts/analysis/49_melanocyte_deep_dive.py        # melanocyte 2x expansion
+python scripts/analysis/57_multiclass_pathway_analysis.py # Dingwall pathway table
+python scripts/analysis/57_pathway_analysis.py            # symmetric 25+ module scoring, all systems
+python scripts/analysis/99_en1_dual_role_analysis.py      # spatial repressor / local activator
+python scripts/analysis/106_melanoblast_neural_crest.py   # Sec 6.5 MITF-axis vs NC reversion
+python scripts/analysis/107_dingwall_class_deg_count.py   # Sec 6.7 HF-placode DEG rank
+
+# EDEN validation — three complementary lines of evidence (Sec 6.4)
+python scripts/analysis/98_eden_posthoc_detection.py                    # Line A (null)
+python scripts/analysis/103_replicate_dingwall_seurat_pipeline.py       # Line B (Derm10 4.32x)
+python scripts/analysis/104_train_on_dingwall_derm_labels.py            # Line C (Derm10 5.20x)
+
+# Primary EDEN (Derm2) discovery — Sec 6.4.1
+python scripts/analysis/100_primary_eden_discovery.py                   # sub-cluster fibro predictions
+python scripts/analysis/101_primary_eden_derm_scoring.py                # score vs Data S1C panels
+python scripts/analysis/105_primary_eden_full_dermal.py                 # on full dermal denominator
+
+# Auxiliary: variant-A Dingwall training used for scope comparison
+python scripts/analysis/102_train_on_dingwall_variantA.py
+```
+
+### 7.3 Section 7 — Dahlin Kit-mutant (hematopoiesis)
+
+```bash
+python scripts/analysis/66_dahlin_kit_mutant.py           # class enrichment WT vs Kit-W41
+python scripts/analysis/67_dahlin_within_class.py         # within-class Wilcoxon DE
+python scripts/analysis/73_novel_populations_dahlin.py    # abstain-gated novel pops
+python scripts/analysis/108_dahlin_lineage_metabolism.py  # per-lineage OXPHOS/glycolysis
+```
+
+### 7.4 Section 7.4 — Veres held-out (pancreas)
+
+```bash
+python scripts/analysis/62_time_course_analysis.py        # class fractions across LARRY days (HSC time-course template)
+python scripts/analysis/109_veres_mature_beta.py          # Sec 5.2 adult MAFA/UCN3 quadrant
+python scripts/analysis/110_veres_polyhormonal_alpha.py   # Sec 5.2 polyhormonal alpha cluster
+```
+
+### 7.5 Section 8 — cross-system prototype geometry + interpretability
+
+```bash
+python scripts/analysis/70_prototype_geometry.py          # intra + cross-system cosine + eff-dim
+python scripts/analysis/72_emergent_axes.py               # within-class PCA of 128-d z
+python scripts/analysis/80_prototype_gene_attribution.py  # integrated gradients per prototype
+python scripts/analysis/81_counterfactual_knockouts.py    # per-gene KO delta on cosine
+python scripts/analysis/82_gene_coattribution_modules.py  # gene co-attribution modules
+python scripts/analysis/83_prototype_training_trajectory.py  # prototype drift across curriculum
+python scripts/analysis/84_adversary_purification.py      # test GRL adversary is at chance
+python scripts/analysis/85_hessian_gene_interactions.py   # second-order gene pair Hessian
+python scripts/analysis/63_nestorowa_zero_shot.py         # Nestorowa unlabeled discovery
+```
+
+---
+
+## 8. Figures + supplement
+
+### Main-text figures (`figures/fig{1,2,3,4}_*.pdf`)
+
+```bash
+python scripts/figures/generate_paper_figures.py
+# fig1_confusion_matrices.pdf   3-panel per-class F1 confusion matrices
+# fig2_aldrich_volcano.pdf      melanocyte cKO vs WT volcano
+# fig3_dahlin_heatmap.pdf       within-class module-score heatmap
+# fig4_sharon_stage_stack.pdf   Veres per-stage class fractions
+```
+
+Figures 5/6 (En1-cKO + Kit-W41 recap) are built by the biology page pipeline
+below — the standalone `regen_fig5_fig6.py` referenced in older notes is not in
+the current tree; use the biology pipeline instead.
+
+### Supplement (`figures/PANDA_supplement.pdf`)
+
+```bash
+python scripts/figures/build_pca_vs_marker_umaps.py      # PCA vs Marker UMAPs per target
+python scripts/figures/build_figure_supplement.py        # combined supplement PDF
+```
+
+### Biology deep-dive supplement pages
+
+Cache UMAPs once, then build per-topic pages, then merge into the supplement:
+
+```bash
+python scripts/figures/biology_00_umap_cache.py
+python scripts/figures/biology_01_dingwall_umap.py
+python scripts/figures/biology_02_primary_eden.py
+python scripts/figures/biology_03_melanoblast_mitf.py
+python scripts/figures/biology_04_dahlin_metabolism.py
+python scripts/figures/biology_05_dahlin_composition.py
+python scripts/figures/biology_06_veres_beta_quadrant.py
+python scripts/figures/biology_07_veres_polyhormonal.py
+python scripts/figures/biology_08_prototype_geometry.py
+python scripts/figures/biology_99_merge_supplement.py    # appends into PANDA_supplement.pdf
+```
+
+Wall-clock: 20-40 min end-to-end (UMAPs dominate).
+
+---
+
+## 9. PDF build
+
+The paper is a self-contained LaTeX document referencing PDFs in `figures/`:
+
+```bash
+cd /home/bcheng/PRISM
+pdflatex -interaction=nonstopmode PAPER.tex   # first pass  (writes .aux)
+pdflatex -interaction=nonstopmode PAPER.tex   # second pass (resolves refs)
+```
+
+`bibtex` is not required — the paper uses an embedded `thebibliography`.
+
+---
+
+## 10. End-to-end make target
+
+The provided `Makefile` covers the canonical skin pipeline end-to-end:
+
+```bash
+make install        # pip install -e .
+make run            # bash scripts/pan_skin/run_all.sh — full skin pipeline
+make test-heldout   # scripts/pan_skin/40_heldout_5fold_cv.py
+make clean          # remove __pycache__ + *.pyc
+```
+
+For a full three-system reproduction, chain the per-section commands above.
+A minimal "everything" recipe:
+
+```bash
+# 1) data
+bash scripts/pan_skin/01_download_tier_a.sh
+bash scripts/pan_skin/02_download_tier_b.sh
+bash scripts/pan_skin/03_download_tier_c.sh
+bash scripts/hematopoiesis/01_download.sh
+bash scripts/hematopoiesis/02_download.sh
+bash scripts/pancreas/01_download.sh
+bash scripts/pancreas/09_download.sh
+
+# 2) corpora
+bash scripts/pan_skin/run_all.sh                     # includes build + train + skin CV
+python scripts/hematopoiesis/02_build_per_dataset.py
+python scripts/hematopoiesis/03_shared_hvgs_and_pca.py
+python scripts/hematopoiesis/10_build_corpus.py
+python scripts/hematopoiesis/11_filter_paper_only.py
+python scripts/pancreas/02_build_per_dataset.py
+python scripts/pancreas/03_shared_hvgs_and_pca.py
+python scripts/pancreas/04_assign_labels.py
+python scripts/pancreas/11_build_corpus.py
+python scripts/pancreas/08_add_baron_split.py
+python scripts/common/generate_missing_holdouts.py
+python scripts/pan_skin/93_add_belote_anchor.py
+
+# 3) all 6 training runs
+for sys in pan_skin hematopoiesis pancreas; do
+  for v in pca marker; do
+    python -m scripts.common.train_panda $sys --variant $v --epochs 8
+  done
+done
+
+# 4) CV + zero-shot
+python -m scripts.common.run_cv --folds 5 --epochs 5
+python -m scripts.common.run_all_zero_shot
+
+# 5) discovery
+bash scripts/common/rerun_all_discovery.sh           # drives scripts/analysis/* end-to-end
+python scripts/analysis/95_adult_beta_validation.py
+
+# 6) figures + PDF
+python scripts/figures/generate_paper_figures.py
+python scripts/figures/build_pca_vs_marker_umaps.py
+python scripts/figures/build_figure_supplement.py
+python scripts/figures/biology_00_umap_cache.py
+for i in 01 02 03 04 05 06 07 08; do
+  python scripts/figures/biology_${i}_*.py
+done
+python scripts/figures/biology_99_merge_supplement.py
+pdflatex -interaction=nonstopmode PAPER.tex && pdflatex -interaction=nonstopmode PAPER.tex
+```
+
+---
+
+## 11. Trouble-shooting
+
+- **`libcusparseLt.so.0: cannot open shared object file`** — you forgot to
+  export `LD_LIBRARY_PATH` **before** Python started. The pip-installed
+  `nvidia-cusparselt-cu12` provides the library; PyTorch does not add its
+  path to the loader search. See section 1.1.
+- **`GLIBCXX_3.4.30 not found`** — your system `libstdc++` is too old; prepend
+  a newer `libstdc++.so.6`'s directory to `LD_LIBRARY_PATH`.
+- **Dingwall GSM -> genotype mapping** (frequent bug source): the correct map is
+  `WT = {GSM6833478, GSM6833479, GSM6833480, GSM6833481}`,
+  `cKO = {GSM6833482, GSM6833483}`.
+  GSM6833480/481 are `rttaControl` (Cre-negative WT), **not** cKO. Getting this
+  wrong flips every En1-cKO enrichment sign.
+- **Pancreas HVG builder OOM** — `scripts/pancreas/03_shared_hvgs_and_pca.py`
+  peaks near 40 GB RAM on the 6-study union. Run on a node with >= 64 GB.
+- **`data/raw` not in git** — it is git-ignored (14+ GB of GEO tars). Rerun
+  section 2 to repopulate.
+- **`stratified split failure` on rare classes** — small-support classes
+  (< 2 members per fold) are merged into the parent `canonical_label`. If a
+  fold still errors, check that `corpus.h5ad`'s `canonical_label` column has
+  the expected vocabulary; the paper vocab is the union enumerated in
+  `PAPER.tex` Sec 3.
+- **`n_conditions` mismatch** — `PANDAEncoder` reads it from
+  `len(datasets)` in the checkpoint; regenerate the checkpoint if you have
+  added/removed a dataset.
+- **`ContrastiveSampler` in 1-condition data** — auto-disabled when there is
+  only one condition; no config change needed.
+- **DataParallel batch-size** — default `bs=256` is calibrated for 4x A100
+  40 GB. Drop to `bs=64` for a single GPU or you will OOM inside the
+  sub-center prototype attention.
+- **`X_prism` not persisted** — after `20_train_panda.py` / `05_train_panda.py`
+  writes the checkpoint, the embedding is re-projected on demand in every
+  downstream analysis; if you want it cached, re-save the AnnData explicitly
+  via `adata.write_h5ad()`.
+
+---
+
+## 12. Artefact index (from PAPER.tex Section 11)
+
+Every quantitative claim traces to one of:
+
+- **Model checkpoints**: `checkpoints/{system}/{variant}/panda_final.pt`
+- **Marker gene lists**: `panda/markers.yaml`
+- **Corpus builders**: `scripts/{pan_skin,hematopoiesis,pancreas}/`
+- **Unified trainer**: `scripts/common/train_panda.py`
+- **Unified 5-fold CV**: `scripts/common/run_cv.py` (or `cv_holdout.py`)
+- **Zero-shot driver**: `scripts/common/run_all_zero_shot.py`
+- **External label supplements**:
+  `data/external_labels/{dingwall_supp,haensel,joost2016,mca,mia,byrnes,yu,baccin,melanocyte_anchor}/`
+- **CV outputs**: `discovery/{system}/{variant}/cv_5fold{,_seed1,_seed2}.json`
+- **Zero-shot summaries**:
+  - `discovery/pancreas/{pca,marker}/baron_summary.json`
+  - `discovery/pancreas/{pca,marker}/veres_summary.json`
+  - `discovery/hematopoiesis/{pca,marker}/nestorowa_summary.json`
+  - `discovery/pan_skin/{pca,marker}/sulic_summary.json`
+  - `discovery/pan_skin/{pca,marker}/belote_summary.json`
+- **Adult-beta panel**: `discovery/pancreas/marker/95_adult_beta_validation.json`
+- **Pathway modules**: `discovery/{system}/marker/57_pathway_class_by_module_{padj,delta}.tsv`
+- **Dingwall EDEN validation**:
+  Line A `discovery/pan_skin/marker/98_eden_summary.json`;
+  Line B `data/processed/dingwall_replica/dingwall_replica.h5ad`,
+         `replica_cluster_20_qc.json`, `replica_marker_matches.csv`;
+  Line C `discovery/pan_skin/marker/104_dingwall_derm_summary.json` + prediction CSVs.
+- **Primary EDEN (Derm2)**:
+  `discovery/pan_skin/marker/100_primary_eden_discovery.csv`,
+  `100_primary_eden_summary.json`,
+  `101_derm_identity_summary.json`,
+  `101_derm_subcluster_scores.csv`.
+- **Dingwall other**:
+  `discovery/pan_skin/marker/57_pathway_analysis.csv`,
+  `90_dingwall_marker_deep_dive.csv`.
+- **Dahlin Kit-mutant**:
+  `discovery/hematopoiesis/marker/92_dahlin_marker_deep_dive.csv`, `dahlin_summary.json`.
+- **Veres deep-dive**: `discovery/pancreas/marker/91_veres_marker_deep_dive.csv`.
+
+Central architecture: `panda/model.py`. Composite loss lives in the same file
+(`supcon_loss`, `vicreg_loss`, `hsic_biased`, `subcenter_angular_infonce`,
+`prototype_repulsion`) and is imported as `from panda import PANDAEncoder, ...`.
+
+## Data mirror
+
+Full data (~195 GB corpus + raw + processed + external labels) is mirrored to Hugging Face at [bryan7264/PANDA](https://huggingface.co/bryan7264/PANDA). Fetch with:
+
+```bash
+huggingface-cli download bryan7264/PANDA --local-dir . --include "data/corpus/pan_skin/**"
+```
+
+Priority folders (fetch these first for the minimum-reproducible pipeline):
+- `data/corpus/{pan_skin,hematopoiesis,pancreas}/harmonized/` — training corpora
+- `data/external_labels/` — paper-supplement label files
+- `checkpoints/{pan_skin,hematopoiesis,pancreas}/marker/` — trained weights
+
+Bulk (only needed to reproduce corpus builds from scratch):
+- `data/raw/` — GEO downloads (regenerable from `scripts/*/03_download*.sh`)
+- `data/corpus/tier{1,2,3}/` — pretraining tier data
+- `data/processed/` — intermediate build artefacts