# PANDA — reproducibility recipe

**PANDA** (Pan-tissue Adversarial Normalized Domain-invariant Anchored MLP) is a
compact prototype-anchored MLP classifier for scRNA-seq cell identity across skin,
hematopoietic, and pancreatic tissues, trained under a composite loss (supervised-
contrastive + VICReg + sub-center angular prototype-InfoNCE + gradient-reversal
dataset/depth adversary + HSIC depth-decorrelation + prototype-repulsion). Two
input variants ship out of the box: **PANDA-PCA** (`PCA(50) -> trunk`) and
**PANDA-Marker** (`[PCA(50) || marker_expr] -> trunk`); Marker beats PCA on 33/35
fold-comparisons across the three systems.

This README is a complete recipe to reproduce every result in `PAPER.tex` from a
clean clone. All commands are copy-pasteable and use absolute paths.

Repository layout:

```
panda/               model + losses + panda/markers.yaml
scripts/pan_skin/    skin pipeline: download -> corpus -> train -> CV -> zero-shot
scripts/pancreas/    pancreas pipeline (same shape)
scripts/hematopoiesis/  HSC pipeline (same shape)
scripts/common/      system-agnostic train / CV / zero-shot drivers
scripts/analysis/    downstream discovery + interpretability
scripts/figures/     paper + supplement figure builders
data/corpus/{sys}/   downloaded + harmonized data
data/raw/            per-dataset raw counts (not in git)
checkpoints/{sys}/{variant}/panda_final.pt
discovery/{sys}/{variant}/*.json,*.csv    all quantitative artefacts
figures/             fig1_..fig6, PANDA_supplement.pdf, biology/*.pdf
```

---

## 1. Requirements

- **Python**: 3.9+ (project is tested on 3.10).
- **CUDA**: PyTorch 2.6.0 wheels — CUDA 12.1/12.4 runtime works.
- **GPU**: 4x A100 40GB used for the paper; **1 GPU works** if you drop the
  training batch size to `bs=64` (default is 256 for 4-GPU DataParallel).
- **Disk**: ~400 GB (raw GEO tars + harmonized corpora + checkpoints).
- **RAM**: ~64 GB (the pancreas HVG builder peaks near ~40 GB).

Pinned runtime dependencies (from `pyproject.toml` / `requirements.txt`):

```
torch==2.6.0             transformers==5.6.2      peft==0.18.1
scanpy==1.11.5           anndata==0.11.4          scvi-tools==1.3.3
harmonypy==0.2.0         numpy>=1.24,<3.0         scipy>=1.10
scikit-learn>=1.2        pandas>=1.5              matplotlib>=3.7
seaborn>=0.12            umap-learn>=0.5          pyyaml>=6.0
tqdm>=4.65               einops>=0.6              gdown>=5.0
GEOparse>=2.0            leidenalg>=0.10          pynndescent>=0.5
scikit-misc>=0.5
```

Optional extras: `bayes` (numpyro/jax for horseshoe), `gpu` (flash-attn 2.8.2),
`viz` (plotly), `dev` (pytest, ruff, mypy).

Install:

```bash
git clone <this-repo> /home/bcheng/PRISM
cd /home/bcheng/PRISM
pip install -e .
# or, editable dev install:
pip install -e ".[dev]"
```

### 1.1 LD_LIBRARY_PATH prefix (required for every PyTorch invocation)

PyTorch 2.6 sparse ops load `libcusparseLt.so.0` which sits under the pip-installed
`nvidia-cusparselt-cu12` package, and `scanpy` needs a modern `libstdc++`. Both
paths must be exported **at the shell level, before Python starts**:

```bash
export LD_LIBRARY_PATH="$(python -c "import site,os; print(os.path.join(site.getsitepackages()[0],'nvidia','cusparselt','lib'))"):$LD_LIBRARY_PATH"
```

If you also have a conda env that ships a newer `libstdc++`, prepend it:

```bash
# example — path is machine-specific; drop it if your system libstdc++ is >= 3.4.30
export LD_LIBRARY_PATH="/home/bcheng/.conda/pkgs/libstdcxx-15.2.0-h39759b7_7/lib:$LD_LIBRARY_PATH"
```

Every `bash scripts/*/run_all.sh` driver applies the same export automatically.

---

## 2. Data acquisition

Every URL below is a public GEO/ArrayExpress FTP link. Raw data is **not**
committed — it must be re-downloaded before anything else runs. Corpus builders
expect files at `data/corpus/{system}/tier_{a,b,c,v2}/`.

### Pan-skin (6 studies, 45,387 cells)

| Study | GEO | Role |
|---|---|---|
| Sulic 2023 (E14.5 dorsal) | GSE212673 | anchor + held-out zero-shot |
| Dingwall 2024 (En1-cKO) | GSE220977 | discovery target (paired with Aldrich GSE214695) |
| Belote 2021 (human melanocyte) | GSE151091 | melanocyte anchor + held-out zero-shot |
| Haensel/Annusver 2020 | GSE142471 | adult homeostasis + wound |
| Joost 2016 | GSE67602 | Smart-seq2 platform anchor |
| Sennett 2015 (bulk RNA) | GSE70288 | placode/dermal-condensate marker reference |
| Han MCA 2018 (neonatal skin) | GSE108097 | Microwell-seq low-depth anchor |
| Merkel 2022 | GSE201447 | touch dome / volar biology |
| Aldrich 2023 (paired with Dingwall) | GSE214695 | En1-cKO snRNA-seq |

```bash
bash scripts/pan_skin/01_download_tier_a.sh   # Aldrich, Ge/Gupta, Joost, Haensel
bash scripts/pan_skin/02_download_tier_b.sh   # MCA, WIHN, Ge/Fuchs, Merkel
bash scripts/pan_skin/03_download_tier_c.sh   # Sennett, Tie, Wiedemann (bulk + human)
# Dingwall / Sulic / Belote must be placed in data/raw/ manually — see repo notes
```

### Pan-hematopoietic (3 studies used in the paper, 192,833 cells)

| Study | GEO | Role |
|---|---|---|
| Weinreb LARRY 2020 | GSE140802 | corpus anchor |
| Baccin whole-BM 2020 | GSE122465 | corpus (stromal + hematopoietic) |
| Tabula Muris Senis BM 2020 | GSE132042 | corpus (paper-labeled) |
| Nestorowa 2016 | GSE81682 | held-out zero-shot (Smart-seq2) |
| Dahlin 2018 | GSE107727 | discovery target (Kit-W41 mutant) |
| Paul 2015 (auxiliary) | GSE72857 | myeloid branch reference |
| Tusi 2018 (auxiliary) | GSE89754 | erythroid trajectory |

```bash
bash scripts/hematopoiesis/01_download.sh   # Paul, Nestorowa, Tusi, Dahlin
bash scripts/hematopoiesis/02_download.sh   # Baccin whole-BM, TMS bone marrow
```

### Pan-pancreatic (6 studies, 120,611 cells)

| Study | GEO | Role |
|---|---|---|
| Baron 2016 | GSE84133 | corpus mouse-train half + held-out mouse-test half |
| Bastidas-Ponce 2019 (E15.5) | GSE132188 | corpus (endocrine progenitor time course) |
| Byrnes 2018 | GSE101099 | corpus (paper-labeled subset) |
| Yu 2021 | GSE139627 | corpus (paper-labeled Ngn3 lineage) |
| Hrovatin MIA 2023 | GSE211796 | corpus (adult islet, paper-labeled) |
| Veres 2019 | GSE114412 | 57,297 corpus + 12,297 held-out slice |

```bash
bash scripts/pancreas/01_download.sh   # Baron, Muraro, Grun, Byrnes, Veres
bash scripts/pancreas/09_download.sh   # Yu Ngn3 seq-EP, MIA 4-month adult islet
```

Wall-clock: 2-6 h depending on bandwidth (GSE108097 MCA tar is ~9 GB, GSE140802
Weinreb is ~14 GB, GSE114412 Veres is ~4 GB).

---

## 3. Corpus build (per system)

Each system builds a `data/corpus/{system}/harmonized/corpus.h5ad` plus a shared
HVG list, per-HVG mean/std, and a fitted PCA basis. Corpus is 100% paper-labeled;
every cell carries a label from its source paper's supplementary table.

### Pan-skin

```bash
python scripts/pan_skin/06_build_per_dataset_h5ads.py
python scripts/pan_skin/07_build_shared_hvgs_and_pca.py
python scripts/pan_skin/08_assign_labels.py
python scripts/pan_skin/08b_curated_label_override.py
python scripts/pan_skin/10_build_corpus.py                 # canonical corpus.h5ad
python scripts/pan_skin/93_add_belote_anchor.py            # +Belote melanocyte anchor
```

Wall-clock ~10-20 min (HVG + PCA is the expensive step).

### Pan-hematopoietic

```bash
python scripts/hematopoiesis/02_build_per_dataset.py
python scripts/hematopoiesis/03_shared_hvgs_and_pca.py
python scripts/hematopoiesis/10_build_corpus.py             # canonical corpus.h5ad
python scripts/hematopoiesis/11_filter_paper_only.py        # enforce paper-labeled subset
python scripts/hematopoiesis/09_retrain_with_nestorowa_anchor.py   # optional anchor
```

Wall-clock ~15-30 min.

### Pan-pancreatic

```bash
python scripts/pancreas/02_build_per_dataset.py
python scripts/pancreas/03_shared_hvgs_and_pca.py
python scripts/pancreas/04_assign_labels.py
python scripts/pancreas/11_build_corpus.py                  # canonical corpus.h5ad
python scripts/pancreas/08_add_baron_split.py               # 943-cell Baron test-half
```

Wall-clock ~30-60 min (peak ~40 GB RAM on the union HVG step).

Also generate the held-out labeled slices used for zero-shot:

```bash
python scripts/common/generate_missing_holdouts.py
```

writes `data/corpus/hematopoiesis/held_out_labeled/nestorowa_GSE81682_test.h5ad`
and `data/corpus/pan_skin/held_out_labeled/sulic_GSE212673_test.h5ad`.

---

## 4. Training

The **canonical trainer** is system-agnostic. It reads
`data/corpus/{system}/harmonized/corpus.h5ad` and writes
`checkpoints/{system}/{variant}/panda_final.pt`.

```bash
# 6 checkpoints total (3 systems x 2 variants). ~30-60 min each on 1x A100.
python -m scripts.common.train_panda pan_skin      --variant pca    --epochs 8
python -m scripts.common.train_panda pan_skin      --variant marker --epochs 8
python -m scripts.common.train_panda hematopoiesis --variant pca    --epochs 8
python -m scripts.common.train_panda hematopoiesis --variant marker --epochs 8
python -m scripts.common.train_panda pancreas      --variant pca    --epochs 8
python -m scripts.common.train_panda pancreas      --variant marker --epochs 8
```

Legacy per-system entry points also exist and are functionally equivalent for
skin/HSC/pancreas single-variant training:
`scripts/pan_skin/20_train_panda.py`, `scripts/hematopoiesis/05_train_panda.py`,
`scripts/pancreas/05_train_panda.py`. Prefer `scripts.common.train_panda`.

---

## 5. Held-out 5-fold cross-validation (Table 1)

The paper's Table 1 CV block reads
`discovery/{system}/{variant}/cv_5fold.json`. Two drivers exist:

- **`scripts/common/run_cv.py`** — canonical, 5 epochs per fold, matches
  paper numbers (mean acc / F1 / AUROC + per-class report).
- `scripts/common/cv_holdout.py` — same architecture but supports GroupKFold
  by dataset and a fuller 6-8 epoch curriculum; slower.

Both accept `--systems` and `--variants`:

```bash
# canonical 5-fold CV for all 3 systems x 2 variants
python -m scripts.common.run_cv --folds 5 --epochs 5
```

Per-system CV drivers also exist (`scripts/pan_skin/40_heldout_5fold_cv.py`,
`scripts/hematopoiesis/07_heldout_5fold_cv.py`,
`scripts/pancreas/07_heldout_5fold_cv.py`); they are single-variant, single-
system alternatives.

### Multi-seed rigor (35 fold-comparisons)

The paper reports Marker beating PCA on 33/35 folds across seeded runs
(2 seeds for skin+pancreas, 3 for HSC). Re-run with different `random_state`s;
outputs write to `cv_5fold_seed{1,2}.json`:

```bash
python -m scripts.common.run_cv --folds 5 --epochs 5                          # seed 0
# The script's seed hook is the random_state passed to StratifiedKFold + model
# init; re-run with edits to `run_cv.py` main() (add `--seed N` arg) or wrap
# a small loop. See scripts/common/cv_holdout.py for the current seed plumbing.
```

---

## 6. Held-out labeled zero-shot targets (Section 5)

One driver runs every zero-shot target for both variants:

```bash
python -m scripts.common.run_all_zero_shot \
    --systems pan_skin hematopoiesis pancreas \
    --variants pca marker
```

writes `discovery/{system}/{variant}/{target}_predictions.csv` and
`{target}_summary.json`. Individual per-target scripts exist for finer-grained
control:

| Target | Script | Populates |
|---|---|---|
| Baron test-half (pancreas, 943 cells) | `scripts/analysis/93_true_zero_shot_baron.py` | Table 1 Baron row + Sec 5.1 |
| Nestorowa Smart-seq2 (HSC, 66 LT-HSC gated) | `scripts/analysis/94_true_zero_shot_nestorowa.py` | Sec 5.3 |
| Sulic E14.5 dorsal skin (4,183 cells) | `scripts/common/run_all_zero_shot.py` (target `sulic`) | Sec 5.5 |
| Belote melanocyte (6,088 cells) | `scripts/common/run_all_zero_shot.py` (target `belote`) | Sec 5.4 |
| Veres held-out slice (12,297 pancreas) | `scripts/common/run_all_zero_shot.py` (target `veres`) | Sec 5.2 |
| Dingwall (25,344 skin, discovery) | `scripts/pan_skin/30_zero_shot_aldrich.py` | Sec 6 |
| Dahlin (61,122 HSC, discovery) | `scripts/common/run_all_zero_shot.py` (target `dahlin`) | Sec 7 |

Adult-beta canonical panel validation on Veres:

```bash
python scripts/analysis/95_adult_beta_validation.py
# -> discovery/pancreas/marker/95_adult_beta_validation.json
```

---

## 7. Discovery analyses

Grouped by paper section. All write to `discovery/{system}/{variant}/`.

### 7.1 Section 5 (held-out labeled) marker deep-dives

```bash
python scripts/analysis/90_dingwall_marker_deep_dive.py   # skin  -> 90_..._marker_deep_dive.csv
python scripts/analysis/91_veres_marker_deep_dive.py      # pancreas
python scripts/analysis/92_dahlin_marker_deep_dive.py     # HSC
```

### 7.2 Section 6 — Dingwall En1-cKO (skin)

```bash
python scripts/analysis/44_en1_cko_contrast.py            # class-level cKO vs WT contrast
python scripts/analysis/45_marker_refinement.py           # per-class marker refinement
python scripts/analysis/49_melanocyte_deep_dive.py        # melanocyte 2x expansion
python scripts/analysis/57_multiclass_pathway_analysis.py # Dingwall pathway table
python scripts/analysis/57_pathway_analysis.py            # symmetric 25+ module scoring, all systems
python scripts/analysis/99_en1_dual_role_analysis.py      # spatial repressor / local activator
python scripts/analysis/106_melanoblast_neural_crest.py   # Sec 6.5 MITF-axis vs NC reversion
python scripts/analysis/107_dingwall_class_deg_count.py   # Sec 6.7 HF-placode DEG rank

# EDEN validation — three complementary lines of evidence (Sec 6.4)
python scripts/analysis/98_eden_posthoc_detection.py                    # Line A (null)
python scripts/analysis/103_replicate_dingwall_seurat_pipeline.py       # Line B (Derm10 4.32x)
python scripts/analysis/104_train_on_dingwall_derm_labels.py            # Line C (Derm10 5.20x)

# Primary EDEN (Derm2) discovery — Sec 6.4.1
python scripts/analysis/100_primary_eden_discovery.py                   # sub-cluster fibro predictions
python scripts/analysis/101_primary_eden_derm_scoring.py                # score vs Data S1C panels
python scripts/analysis/105_primary_eden_full_dermal.py                 # on full dermal denominator

# Auxiliary: variant-A Dingwall training used for scope comparison
python scripts/analysis/102_train_on_dingwall_variantA.py
```

### 7.3 Section 7 — Dahlin Kit-mutant (hematopoiesis)

```bash
python scripts/analysis/66_dahlin_kit_mutant.py           # class enrichment WT vs Kit-W41
python scripts/analysis/67_dahlin_within_class.py         # within-class Wilcoxon DE
python scripts/analysis/73_novel_populations_dahlin.py    # abstain-gated novel pops
python scripts/analysis/108_dahlin_lineage_metabolism.py  # per-lineage OXPHOS/glycolysis
```

### 7.4 Section 7.4 — Veres held-out (pancreas)

```bash
python scripts/analysis/62_time_course_analysis.py        # class fractions across LARRY days (HSC time-course template)
python scripts/analysis/109_veres_mature_beta.py          # Sec 5.2 adult MAFA/UCN3 quadrant
python scripts/analysis/110_veres_polyhormonal_alpha.py   # Sec 5.2 polyhormonal alpha cluster
```

### 7.5 Section 8 — cross-system prototype geometry + interpretability

```bash
python scripts/analysis/70_prototype_geometry.py          # intra + cross-system cosine + eff-dim
python scripts/analysis/72_emergent_axes.py               # within-class PCA of 128-d z
python scripts/analysis/80_prototype_gene_attribution.py  # integrated gradients per prototype
python scripts/analysis/81_counterfactual_knockouts.py    # per-gene KO delta on cosine
python scripts/analysis/82_gene_coattribution_modules.py  # gene co-attribution modules
python scripts/analysis/83_prototype_training_trajectory.py  # prototype drift across curriculum
python scripts/analysis/84_adversary_purification.py      # test GRL adversary is at chance
python scripts/analysis/85_hessian_gene_interactions.py   # second-order gene pair Hessian
python scripts/analysis/63_nestorowa_zero_shot.py         # Nestorowa unlabeled discovery
```

---

## 8. Figures + supplement

### Main-text figures (`figures/fig{1,2,3,4}_*.pdf`)

```bash
python scripts/figures/generate_paper_figures.py
# fig1_confusion_matrices.pdf   3-panel per-class F1 confusion matrices
# fig2_aldrich_volcano.pdf      melanocyte cKO vs WT volcano
# fig3_dahlin_heatmap.pdf       within-class module-score heatmap
# fig4_sharon_stage_stack.pdf   Veres per-stage class fractions
```

Figures 5/6 (En1-cKO + Kit-W41 recap) are built by the biology page pipeline
below — the standalone `regen_fig5_fig6.py` referenced in older notes is not in
the current tree; use the biology pipeline instead.

### Supplement (`figures/PANDA_supplement.pdf`)

```bash
python scripts/figures/build_pca_vs_marker_umaps.py      # PCA vs Marker UMAPs per target
python scripts/figures/build_figure_supplement.py        # combined supplement PDF
```

### Biology deep-dive supplement pages

Cache UMAPs once, then build per-topic pages, then merge into the supplement:

```bash
python scripts/figures/biology_00_umap_cache.py
python scripts/figures/biology_01_dingwall_umap.py
python scripts/figures/biology_02_primary_eden.py
python scripts/figures/biology_03_melanoblast_mitf.py
python scripts/figures/biology_04_dahlin_metabolism.py
python scripts/figures/biology_05_dahlin_composition.py
python scripts/figures/biology_06_veres_beta_quadrant.py
python scripts/figures/biology_07_veres_polyhormonal.py
python scripts/figures/biology_08_prototype_geometry.py
python scripts/figures/biology_99_merge_supplement.py    # appends into PANDA_supplement.pdf
```

Wall-clock: 20-40 min end-to-end (UMAPs dominate).

---

## 9. PDF build

The paper is a self-contained LaTeX document referencing PDFs in `figures/`:

```bash
cd /home/bcheng/PRISM
pdflatex -interaction=nonstopmode PAPER.tex   # first pass  (writes .aux)
pdflatex -interaction=nonstopmode PAPER.tex   # second pass (resolves refs)
```

`bibtex` is not required — the paper uses an embedded `thebibliography`.

---

## 10. End-to-end make target

The provided `Makefile` covers the canonical skin pipeline end-to-end:

```bash
make install        # pip install -e .
make run            # bash scripts/pan_skin/run_all.sh — full skin pipeline
make test-heldout   # scripts/pan_skin/40_heldout_5fold_cv.py
make clean          # remove __pycache__ + *.pyc
```

For a full three-system reproduction, chain the per-section commands above.
A minimal "everything" recipe:

```bash
# 1) data
bash scripts/pan_skin/01_download_tier_a.sh
bash scripts/pan_skin/02_download_tier_b.sh
bash scripts/pan_skin/03_download_tier_c.sh
bash scripts/hematopoiesis/01_download.sh
bash scripts/hematopoiesis/02_download.sh
bash scripts/pancreas/01_download.sh
bash scripts/pancreas/09_download.sh

# 2) corpora
bash scripts/pan_skin/run_all.sh                     # includes build + train + skin CV
python scripts/hematopoiesis/02_build_per_dataset.py
python scripts/hematopoiesis/03_shared_hvgs_and_pca.py
python scripts/hematopoiesis/10_build_corpus.py
python scripts/hematopoiesis/11_filter_paper_only.py
python scripts/pancreas/02_build_per_dataset.py
python scripts/pancreas/03_shared_hvgs_and_pca.py
python scripts/pancreas/04_assign_labels.py
python scripts/pancreas/11_build_corpus.py
python scripts/pancreas/08_add_baron_split.py
python scripts/common/generate_missing_holdouts.py
python scripts/pan_skin/93_add_belote_anchor.py

# 3) all 6 training runs
for sys in pan_skin hematopoiesis pancreas; do
  for v in pca marker; do
    python -m scripts.common.train_panda $sys --variant $v --epochs 8
  done
done

# 4) CV + zero-shot
python -m scripts.common.run_cv --folds 5 --epochs 5
python -m scripts.common.run_all_zero_shot

# 5) discovery
bash scripts/common/rerun_all_discovery.sh           # drives scripts/analysis/* end-to-end
python scripts/analysis/95_adult_beta_validation.py

# 6) figures + PDF
python scripts/figures/generate_paper_figures.py
python scripts/figures/build_pca_vs_marker_umaps.py
python scripts/figures/build_figure_supplement.py
python scripts/figures/biology_00_umap_cache.py
for i in 01 02 03 04 05 06 07 08; do
  python scripts/figures/biology_${i}_*.py
done
python scripts/figures/biology_99_merge_supplement.py
pdflatex -interaction=nonstopmode PAPER.tex && pdflatex -interaction=nonstopmode PAPER.tex
```

---

## 11. Trouble-shooting

- **`libcusparseLt.so.0: cannot open shared object file`** — you forgot to
  export `LD_LIBRARY_PATH` **before** Python started. The pip-installed
  `nvidia-cusparselt-cu12` provides the library; PyTorch does not add its
  path to the loader search. See section 1.1.
- **`GLIBCXX_3.4.30 not found`** — your system `libstdc++` is too old; prepend
  a newer `libstdc++.so.6`'s directory to `LD_LIBRARY_PATH`.
- **Dingwall GSM -> genotype mapping** (frequent bug source): the correct map is
  `WT = {GSM6833478, GSM6833479, GSM6833480, GSM6833481}`,
  `cKO = {GSM6833482, GSM6833483}`.
  GSM6833480/481 are `rttaControl` (Cre-negative WT), **not** cKO. Getting this
  wrong flips every En1-cKO enrichment sign.
- **Pancreas HVG builder OOM** — `scripts/pancreas/03_shared_hvgs_and_pca.py`
  peaks near 40 GB RAM on the 6-study union. Run on a node with >= 64 GB.
- **`data/raw` not in git** — it is git-ignored (14+ GB of GEO tars). Rerun
  section 2 to repopulate.
- **`stratified split failure` on rare classes** — small-support classes
  (< 2 members per fold) are merged into the parent `canonical_label`. If a
  fold still errors, check that `corpus.h5ad`'s `canonical_label` column has
  the expected vocabulary; the paper vocab is the union enumerated in
  `PAPER.tex` Sec 3.
- **`n_conditions` mismatch** — `PANDAEncoder` reads it from
  `len(datasets)` in the checkpoint; regenerate the checkpoint if you have
  added/removed a dataset.
- **`ContrastiveSampler` in 1-condition data** — auto-disabled when there is
  only one condition; no config change needed.
- **DataParallel batch-size** — default `bs=256` is calibrated for 4x A100
  40 GB. Drop to `bs=64` for a single GPU or you will OOM inside the
  sub-center prototype attention.
- **`X_prism` not persisted** — after `20_train_panda.py` / `05_train_panda.py`
  writes the checkpoint, the embedding is re-projected on demand in every
  downstream analysis; if you want it cached, re-save the AnnData explicitly
  via `adata.write_h5ad()`.

---

## 12. Artefact index (from PAPER.tex Section 11)

Every quantitative claim traces to one of:

- **Model checkpoints**: `checkpoints/{system}/{variant}/panda_final.pt`
- **Marker gene lists**: `panda/markers.yaml`
- **Corpus builders**: `scripts/{pan_skin,hematopoiesis,pancreas}/`
- **Unified trainer**: `scripts/common/train_panda.py`
- **Unified 5-fold CV**: `scripts/common/run_cv.py` (or `cv_holdout.py`)
- **Zero-shot driver**: `scripts/common/run_all_zero_shot.py`
- **External label supplements**:
  `data/external_labels/{dingwall_supp,haensel,joost2016,mca,mia,byrnes,yu,baccin,melanocyte_anchor}/`
- **CV outputs**: `discovery/{system}/{variant}/cv_5fold{,_seed1,_seed2}.json`
- **Zero-shot summaries**:
  - `discovery/pancreas/{pca,marker}/baron_summary.json`
  - `discovery/pancreas/{pca,marker}/veres_summary.json`
  - `discovery/hematopoiesis/{pca,marker}/nestorowa_summary.json`
  - `discovery/pan_skin/{pca,marker}/sulic_summary.json`
  - `discovery/pan_skin/{pca,marker}/belote_summary.json`
- **Adult-beta panel**: `discovery/pancreas/marker/95_adult_beta_validation.json`
- **Pathway modules**: `discovery/{system}/marker/57_pathway_class_by_module_{padj,delta}.tsv`
- **Dingwall EDEN validation**:
  Line A `discovery/pan_skin/marker/98_eden_summary.json`;
  Line B `data/processed/dingwall_replica/dingwall_replica.h5ad`,
         `replica_cluster_20_qc.json`, `replica_marker_matches.csv`;
  Line C `discovery/pan_skin/marker/104_dingwall_derm_summary.json` + prediction CSVs.
- **Primary EDEN (Derm2)**:
  `discovery/pan_skin/marker/100_primary_eden_discovery.csv`,
  `100_primary_eden_summary.json`,
  `101_derm_identity_summary.json`,
  `101_derm_subcluster_scores.csv`.
- **Dingwall other**:
  `discovery/pan_skin/marker/57_pathway_analysis.csv`,
  `90_dingwall_marker_deep_dive.csv`.
- **Dahlin Kit-mutant**:
  `discovery/hematopoiesis/marker/92_dahlin_marker_deep_dive.csv`, `dahlin_summary.json`.
- **Veres deep-dive**: `discovery/pancreas/marker/91_veres_marker_deep_dive.csv`.

Central architecture: `panda/model.py`. Composite loss lives in the same file
(`supcon_loss`, `vicreg_loss`, `hsic_biased`, `subcenter_angular_infonce`,
`prototype_repulsion`) and is imported as `from panda import PANDAEncoder, ...`.

## Data mirror

Full data (~195 GB corpus + raw + processed + external labels) is mirrored to Hugging Face at [bryan7264/PANDA](https://huggingface.co/bryan7264/PANDA). Fetch with:

```bash
huggingface-cli download bryan7264/PANDA --local-dir . --include "data/corpus/pan_skin/**"
```

Priority folders (fetch these first for the minimum-reproducible pipeline):
- `data/corpus/{pan_skin,hematopoiesis,pancreas}/harmonized/` — training corpora
- `data/external_labels/` — paper-supplement label files
- `checkpoints/{pan_skin,hematopoiesis,pancreas}/marker/` — trained weights

Bulk (only needed to reproduce corpus builds from scratch):
- `data/raw/` — GEO downloads (regenerable from `scripts/*/03_download*.sh`)
- `data/corpus/tier{1,2,3}/` — pretraining tier data
- `data/processed/` — intermediate build artefacts