Correction pass: gate-matched Dahlin, retracted unsupported claims, complete HF-placode DEG set, restyled figures

Mirrors github.com/bryanc5864/PRISM @ 879c970.
- Dahlin genotype contrast restricted to the matched LK FACS gate (a within-WT
gate control produced larger shifts than the mutation); recovers Dahlin's
mast/basophil loss that the pooled analysis missed.
- Withdrawn: Veres MAFA/UCN3/IAPP beta-class enrichment (measured on primary-islet
controls), polyhormonal novelty claim, Dahlin 'novel' pathway phenotypes,
Dingwall ectopic sweat-gland claim. epsilon class disclosed as Veres's SC-EC.
- Dingwall citation corrected (title and DOI were both wrong).
- Complete 12-gene HF-placode DEG set recovered; artefacts no longer truncate.
- data/raw/GSE220977_combined.h5ad added so raw-dependent scripts run from a clone.
- Figures restyled and regenerated; legacy duplicate UMAPs removed.

.gitattributes

@@ -86,3 +86,5 @@ data/processed/dingwall_replica/dingwall_replica.h5ad filter=lfs diff=lfs merge=
 data/processed/sulic/adata_sulic.h5ad filter=lfs diff=lfs merge=lfs -text
 data/processed/sulic/adata_sulic_clustered.h5ad filter=lfs diff=lfs merge=lfs -text
 data/processed/weinreb_larry/adata_weinreb.h5ad filter=lfs diff=lfs merge=lfs -text
+figures/supplement/26b_pca_vs_marker_umaps_veres_by_class.pdf filter=lfs diff=lfs merge=lfs -text
+data/raw/GSE220977_combined.h5ad filter=lfs diff=lfs merge=lfs -text

PAPER.pdf

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:00301cc9e983b32b845d591f0c6d87c1eeb8dca9ba4170ebfcacfedae7493181
-size 3977089
+oid sha256:40dba671f0811830dcacdac7d667c9bc2a53485038c70902f54d4af1baebccaf
+size 4096150

README.md

@@ -1,67 +1,622 @@
----
-license: mit
-tags:
-  - single-cell
-  - scRNA-seq
-  - cell-type-classification
-  - contrastive-learning
-  - domain-adaptation
-library_name: pytorch
-pipeline_tag: feature-extraction
----
-
-# PANDA — Pan-tissue Adversarial Normalized Domain-invariant Anchored MLP
-
-Prototype-anchored MLP classifier for scRNA-seq cell identity across skin, hematopoietic, and pancreatic tissues.
-Trained under a composite of SupCon + VICReg + prototype-InfoNCE + GRL dataset+depth adversary + HSIC decorrelation + prototype-repulsion.
-
-Two variants: **PANDA-PCA** and **PANDA-Marker** (adds a marker gene channel).
-
-Code + paper: https://github.com/bryanc5864/PRISM
-
-## Contents
-
-| Path | Description |
-|---|---|
-| `checkpoints/{system}/{pca,marker}/panda_final.pt` | Final trained weights per system × variant (6 core models) |
-| `checkpoints/pan_skin_dingwall_derm/panda_final.pt` | Line C: PANDA-Marker trained on Dingwall Derm labels |
-| `data/corpus/{system}/harmonized/` | Training corpora (h5ad + stats + PCA basis) |
-| `data/external_labels/` | Paper-supplement label files per source study |
-| `data/processed/dingwall_replica/` | Independent scanpy reproduction of Dingwall Seurat pipeline |
-| `discovery/` | Discovery-analysis outputs backing every paper claim |
-| `figures/` | Main + supplement + biology figures + merged PDFs |
-| `panda/`, `scripts/` | Model + analysis code (also on GitHub) |
-| `PAPER.tex`, `PAPER.pdf` | Manuscript |
-| `README.md` | Full end-to-end reproduction recipe |
-
-## Quick fetch
-
-```bash
-# essentials only (~30 GB)
-huggingface-cli download bryan7264/PANDA \
-    --local-dir . \
-    --include "checkpoints/**" "data/corpus/**" "data/external_labels/**" "discovery/**"
-
-# individual system
-huggingface-cli download bryan7264/PANDA \
-    --local-dir . \
-    --include "data/corpus/pan_skin/**" "checkpoints/pan_skin/**"
-```
-
-## Usage
-
-```python
-import torch
-from panda.model import PANDAEncoder
-
-ck = torch.load("checkpoints/pan_skin/marker/panda_final.pt", map_location="cpu",
-                weights_only=False)
-model = PANDAEncoder(variant="marker", n_pca=50,
-                     n_markers=len(ck["marker_genes"]),
-                     n_classes=len(ck["classes"]), n_sub=3,
-                     n_datasets=len(ck["datasets"]))
-model.load_state_dict(ck["model"])
-model.eval()
-```
-
-See PAPER.pdf for full experimental setup and README.md for the reproduction recipe.
+# PANDA — reproducibility recipe
+
+**PANDA** (Pan-tissue Adversarial Normalized Domain-invariant Anchored MLP) is a
+compact prototype-anchored MLP classifier for scRNA-seq cell identity across skin,
+hematopoietic, and pancreatic tissues, trained under a composite loss (supervised-
+contrastive + VICReg + sub-center angular prototype-InfoNCE + gradient-reversal
+dataset/depth adversary + HSIC depth-decorrelation). Two
+input variants ship out of the box: **PANDA-PCA** (`PCA(50) -> trunk`) and
+**PANDA-Marker** (`[PCA(50) || marker_expr] -> trunk`); the marker channel's gain
+is system-dependent, largest on pancreas (+4.2% CV accuracy).
+
+This README is a complete recipe to reproduce every result in `PAPER.tex` from a
+clean clone. All commands are copy-pasteable and run from the repo root; scripts
+resolve the repo root automatically (see `PANDA_ROOT` below).
+
+Repository layout:
+
+```
+panda/               model + losses + panda/markers.yaml
+scripts/pan_skin/    skin pipeline: download -> corpus -> train -> CV -> zero-shot
+scripts/pancreas/    pancreas pipeline (same shape)
+scripts/hematopoiesis/  HSC pipeline (same shape)
+scripts/common/      system-agnostic train / CV / zero-shot drivers
+scripts/analysis/    downstream discovery + interpretability
+scripts/figures/     paper + supplement figure builders
+data/corpus/{sys}/   downloaded + harmonized data
+data/raw/            per-dataset raw counts (not in git)
+checkpoints/{sys}/{variant}/panda_final.pt
+discovery/{sys}/{variant}/*.json,*.csv    all quantitative artefacts
+figures/             fig1_..fig6, PANDA_supplement.pdf, biology/*.pdf
+```
+
+---
+
+## 1. Requirements
+
+- **Python**: 3.9+ (project is tested on 3.10).
+- **CUDA**: PyTorch 2.6.0 wheels — CUDA 12.1/12.4 runtime works.
+- **GPU**: any single CUDA GPU works — the trainer is single-device (batch 256,
+  a compact MLP; a consumer GPU trains a checkpoint in well under an hour).
+- **Disk**: ~400 GB (raw GEO tars + harmonized corpora + checkpoints).
+- **RAM**: ~64 GB (the pancreas HVG builder peaks near ~40 GB).
+
+Pinned runtime dependencies (from `pyproject.toml` / `requirements.txt`):
+
+```
+torch==2.6.0             scanpy==1.11.5           anndata>=0.11.4
+harmonypy==0.2.0         numpy>=1.24,<3.0         scipy>=1.10
+scikit-learn>=1.2        pandas>=1.5              matplotlib>=3.7
+seaborn>=0.12            umap-learn>=0.5          pyyaml>=6.0
+tqdm>=4.65               leidenalg>=0.10          pynndescent>=0.5
+scikit-misc>=0.5
+```
+
+Optional extras: `viz` (plotly), `dev` (pytest, ruff, mypy).
+
+Install:
+
+```bash
+git clone <this-repo> panda && cd panda
+pip install -e .
+# or, editable dev install:
+pip install -e ".[dev]"
+```
+
+### Repository root resolution (`PANDA_ROOT`)
+
+Every script resolves the repository root as
+`os.environ.get("PANDA_ROOT", <two directories above the script>)`, so a clone at
+any path works out of the box. Set `PANDA_ROOT` explicitly only if you run scripts
+from a copied/vendored location or want data/checkpoints rooted elsewhere:
+
+```bash
+export PANDA_ROOT=/path/to/clone     # optional
+```
+
+Data and checkpoints are expected at `$PANDA_ROOT/data` and `$PANDA_ROOT/checkpoints`
+(symlink/junction these to the Hugging Face mirror download if you keep it separate).
+
+### 1.1 LD_LIBRARY_PATH prefix (required for every PyTorch invocation)
+
+PyTorch 2.6 sparse ops load `libcusparseLt.so.0` which sits under the pip-installed
+`nvidia-cusparselt-cu12` package, and `scanpy` needs a modern `libstdc++`. Both
+paths must be exported **at the shell level, before Python starts**:
+
+```bash
+export LD_LIBRARY_PATH="$(python -c "import site,os; print(os.path.join(site.getsitepackages()[0],'nvidia','cusparselt','lib'))"):$LD_LIBRARY_PATH"
+```
+
+If you also have a conda env that ships a newer `libstdc++`, prepend it:
+
+```bash
+# example — path is machine-specific; drop it if your system libstdc++ is >= 3.4.30
+export LD_LIBRARY_PATH="/home/bcheng/.conda/pkgs/libstdcxx-15.2.0-h39759b7_7/lib:$LD_LIBRARY_PATH"
+```
+
+Every `bash scripts/*/run_all.sh` driver applies the same export automatically.
+
+---
+
+## 2. Data acquisition
+
+Every URL below is a public GEO/ArrayExpress FTP link. Raw data is **not**
+committed — it must be re-downloaded before anything else runs. Corpus builders
+expect files at `data/corpus/{system}/tier_{a,b,c,v2}/`.
+
+### Pan-skin (6 studies, 45,387 cells)
+
+| Study | GEO | Role |
+|---|---|---|
+| Sulic 2023 (E14.5 dorsal) | GSE212673 | anchor + held-out zero-shot |
+| Dingwall 2024 (En1-cKO) | GSE220977 | discovery target (paired with Aldrich GSE214695) |
+| Belote 2021 (human melanocyte) | GSE151091 | melanocyte anchor + held-out zero-shot |
+| Haensel/Annusver 2020 | GSE142471 | adult homeostasis + wound |
+| Joost 2016 | GSE67602 | Smart-seq2 platform anchor |
+| Sennett 2015 (bulk RNA) | GSE70288 | placode/dermal-condensate marker reference |
+| Han MCA 2018 (neonatal skin) | GSE108097 | Microwell-seq low-depth anchor |
+| Merkel 2022 | GSE201447 | touch dome / volar biology |
+| Aldrich 2023 (paired with Dingwall) | GSE214695 | En1-cKO snRNA-seq |
+
+```bash
+bash scripts/pan_skin/01_download_tier_a.sh   # Aldrich, Ge/Gupta, Joost, Haensel
+bash scripts/pan_skin/02_download_tier_b.sh   # MCA, WIHN, Ge/Fuchs, Merkel
+bash scripts/pan_skin/03_download_tier_c.sh   # Sennett, Tie, Wiedemann (bulk + human)
+# Dingwall / Sulic / Belote must be placed in data/raw/ manually — see repo notes
+```
+
+### Pan-hematopoietic (3 studies used in the paper, 192,833 cells)
+
+| Study | GEO | Role |
+|---|---|---|
+| Weinreb LARRY 2020 | GSE140802 | corpus anchor |
+| Baccin whole-BM 2020 | GSE122465 | corpus (stromal + hematopoietic) |
+| Tabula Muris Senis BM 2020 | GSE132042 | corpus (paper-labeled) |
+| Nestorowa 2016 | GSE81682 | held-out zero-shot (Smart-seq2) |
+| Dahlin 2018 | GSE107727 | discovery target (Kit-W41 mutant) |
+| Paul 2015 (auxiliary) | GSE72857 | myeloid branch reference |
+| Tusi 2018 (auxiliary) | GSE89754 | erythroid trajectory |
+
+```bash
+bash scripts/hematopoiesis/01_download.sh   # Paul, Nestorowa, Tusi, Dahlin
+bash scripts/hematopoiesis/02_download.sh   # Baccin whole-BM, TMS bone marrow
+```
+
+### Pan-pancreatic (6 studies, 120,611 cells)
+
+| Study | GEO | Role |
+|---|---|---|
+| Baron 2016 | GSE84133 | corpus mouse-train half + held-out mouse-test half |
+| Bastidas-Ponce 2019 (E15.5) | GSE132188 | corpus (endocrine progenitor time course) |
+| Byrnes 2018 | GSE101099 | corpus (paper-labeled subset) |
+| Yu 2021 | GSE139627 | corpus (paper-labeled Ngn3 lineage) |
+| Hrovatin MIA 2023 | GSE211796 | corpus (adult islet, paper-labeled) |
+| Veres 2019 | GSE114412 | 57,297 corpus + 12,297 held-out slice |
+
+```bash
+bash scripts/pancreas/01_download.sh   # Baron, Muraro, Grun, Byrnes, Veres
+bash scripts/pancreas/09_download.sh   # Yu Ngn3 seq-EP, MIA 4-month adult islet
+```
+
+Wall-clock: 2-6 h depending on bandwidth (GSE108097 MCA tar is ~9 GB, GSE140802
+Weinreb is ~14 GB, GSE114412 Veres is ~4 GB).
+
+---
+
+## 3. Corpus build (per system)
+
+Each system builds a `data/corpus/{system}/harmonized/corpus.h5ad` plus a shared
+HVG list, per-HVG mean/std, and a fitted PCA basis. Corpus is 100% paper-labeled;
+every cell carries a label from its source paper's supplementary table.
+
+### Pan-skin
+
+```bash
+python scripts/pan_skin/06_build_per_dataset_h5ads.py
+python scripts/pan_skin/07_build_shared_hvgs_and_pca.py
+python scripts/pan_skin/08_assign_labels.py
+python scripts/pan_skin/08b_curated_label_override.py
+python scripts/pan_skin/10_build_corpus.py                 # canonical corpus.h5ad
+python scripts/pan_skin/93_add_belote_anchor.py            # +Belote melanocyte anchor
+```
+
+Wall-clock ~10-20 min (HVG + PCA is the expensive step).
+
+### Pan-hematopoietic
+
+```bash
+python scripts/hematopoiesis/02_build_per_dataset.py
+python scripts/hematopoiesis/03_shared_hvgs_and_pca.py
+python scripts/hematopoiesis/10_build_corpus.py             # canonical corpus.h5ad
+python scripts/hematopoiesis/11_filter_paper_only.py        # enforce paper-labeled subset
+python scripts/hematopoiesis/09_retrain_with_nestorowa_anchor.py   # optional anchor
+```
+
+Wall-clock ~15-30 min.
+
+### Pan-pancreatic
+
+```bash
+python scripts/pancreas/02_build_per_dataset.py
+python scripts/pancreas/03_shared_hvgs_and_pca.py
+python scripts/pancreas/04_assign_labels.py
+python scripts/pancreas/11_build_corpus.py                  # canonical corpus.h5ad
+python scripts/pancreas/08_add_baron_split.py               # 943-cell Baron test-half
+```
+
+Wall-clock ~30-60 min (peak ~40 GB RAM on the union HVG step).
+
+Also generate the held-out labeled slices used for zero-shot:
+
+```bash
+python scripts/common/generate_missing_holdouts.py
+```
+
+writes `data/corpus/hematopoiesis/held_out_labeled/nestorowa_GSE81682_test.h5ad`
+and `data/corpus/pan_skin/held_out_labeled/sulic_GSE212673_test.h5ad`.
+
+---
+
+## 4. Training
+
+The **canonical trainer** is system-agnostic. It reads
+`data/corpus/{system}/harmonized/corpus.h5ad` and writes
+`checkpoints/{system}/{variant}/panda_final.pt`.
+
+```bash
+# 6 checkpoints total (3 systems x 2 variants). ~30-60 min each on 1x A100.
+python -m scripts.common.train_panda pan_skin      --variant pca    --epochs 8
+python -m scripts.common.train_panda pan_skin      --variant marker --epochs 8
+python -m scripts.common.train_panda hematopoiesis --variant pca    --epochs 8
+python -m scripts.common.train_panda hematopoiesis --variant marker --epochs 8
+python -m scripts.common.train_panda pancreas      --variant pca    --epochs 8
+python -m scripts.common.train_panda pancreas      --variant marker --epochs 8
+```
+
+Legacy per-system entry points also exist and are functionally equivalent for
+skin/HSC/pancreas single-variant training:
+`scripts/pan_skin/20_train_panda.py`, `scripts/hematopoiesis/05_train_panda.py`,
+`scripts/pancreas/05_train_panda.py`. Prefer `scripts.common.train_panda`.
+
+---
+
+## 5. Held-out 5-fold cross-validation (Table 1)
+
+The paper's Table 1 CV block reads
+`discovery/{system}/{variant}/cv_5fold.json`. Two drivers exist:
+
+- **`scripts/common/run_cv.py`** — canonical, 5 epochs per fold, matches
+  paper numbers (mean acc / F1 / AUROC + per-class report).
+- `scripts/common/cv_holdout.py` — same architecture but supports GroupKFold
+  by dataset and a fuller 6-8 epoch curriculum; slower.
+
+Both accept `--systems` and `--variants`:
+
+```bash
+# canonical 5-fold CV for all 3 systems x 2 variants
+python -m scripts.common.run_cv --folds 5 --epochs 5
+```
+
+Per-system CV drivers also exist (`scripts/pan_skin/40_heldout_5fold_cv.py`,
+`scripts/hematopoiesis/07_heldout_5fold_cv.py`,
+`scripts/pancreas/07_heldout_5fold_cv.py`); they are single-variant, single-
+system alternatives.
+
+### Multi-seed rigor
+
+Seed replicates use the canonical driver on the canonical corpora; non-zero seeds
+write `cv_5fold_seed{N}.json` next to the seed-0 `cv_5fold.json`:
+
+```bash
+python -m scripts.common.run_cv --folds 5 --epochs 5            # seed 0 (canonical)
+python -m scripts.common.run_cv --folds 5 --epochs 5 --seed 1
+python -m scripts.common.run_cv --folds 5 --epochs 5 --seed 2
+```
+
+> **Correction (2026-08):** older `cv_5fold_seed{1,2}.json` files in the repo history
+> were produced on different corpus builds (different cell counts and label
+> vocabularies) by `cv_holdout.py` with a different curriculum; the "33/35
+> fold-comparisons" claim previously derived from them was invalid and has been
+> withdrawn. The seeds committed now are regenerated with the commands above on
+> the canonical corpora (verified same n/K per configuration). Honest result:
+> Marker beats PCA on **34/45** fold-comparisons — pancreas 15/15 (+0.04 acc
+> every seed), skin 11/15 (gap ≤0.003), hematopoiesis 8/15 (parity).
+
+---
+
+## 6. Held-out labeled zero-shot targets (Section 5)
+
+One driver runs every zero-shot target for both variants:
+
+```bash
+python -m scripts.common.run_all_zero_shot \
+    --systems pan_skin hematopoiesis pancreas \
+    --variants pca marker
+```
+
+writes `discovery/{system}/{variant}/{target}_predictions.csv` and
+`{target}_summary.json`. Individual per-target scripts exist for finer-grained
+control:
+
+| Target | Script | Populates |
+|---|---|---|
+| Baron test-half (pancreas, 943 cells) | `scripts/analysis/93_true_zero_shot_baron.py` | Table 1 Baron row + Sec 5.1 |
+| Nestorowa Smart-seq2 (HSC, 66 LT-HSC gated) | `scripts/analysis/94_true_zero_shot_nestorowa.py` | Sec 5.3 |
+| Sulic E14.5 dorsal skin (4,183 cells) | `scripts/common/run_all_zero_shot.py` (target `sulic`) | Sec 5.5 |
+| Belote melanocyte (6,088 cells) | `scripts/common/run_all_zero_shot.py` (target `belote`) | Sec 5.4 |
+| Veres held-out slice (12,297 pancreas) | `scripts/common/run_all_zero_shot.py` (target `veres`) | Sec 5.2 |
+| Dingwall (25,344 skin, discovery) | `scripts/pan_skin/30_zero_shot_aldrich.py` | Sec 6 |
+| Dahlin (61,122 HSC, discovery) | `scripts/common/run_all_zero_shot.py` (target `dahlin`) | Sec 7 |
+
+Adult-beta canonical panel validation on Veres:
+
+```bash
+python scripts/analysis/95_adult_beta_validation.py
+# -> discovery/pancreas/marker/95_adult_beta_validation.json
+```
+
+---
+
+## 7. Discovery analyses
+
+Grouped by paper section. All write to `discovery/{system}/{variant}/`.
+
+### 7.1 Section 5 (held-out labeled) marker deep-dives
+
+```bash
+python scripts/analysis/90_dingwall_marker_deep_dive.py   # skin  -> 90_..._marker_deep_dive.csv
+python scripts/analysis/91_veres_marker_deep_dive.py      # pancreas
+python scripts/analysis/92_dahlin_marker_deep_dive.py     # HSC
+```
+
+### 7.2 Section 6 — Dingwall En1-cKO (skin)
+
+```bash
+python scripts/analysis/44_en1_cko_contrast.py            # class-level cKO vs WT contrast
+python scripts/analysis/45_marker_refinement.py           # per-class marker refinement
+python scripts/analysis/49_melanocyte_deep_dive.py        # melanocyte 2x expansion
+python scripts/analysis/57_multiclass_pathway_analysis.py # Dingwall pathway table
+python scripts/analysis/57_pathway_analysis.py            # symmetric 25+ module scoring, all systems
+python scripts/analysis/99_en1_dual_role_analysis.py      # spatial repressor / local activator
+python scripts/analysis/106_melanoblast_neural_crest.py   # Sec 6.5 MITF-axis vs NC reversion
+python scripts/analysis/107_dingwall_class_deg_count.py   # Sec 6.7 HF-placode DEG rank
+
+# EDEN validation — three complementary lines of evidence (Sec 6.4)
+python scripts/analysis/98_eden_posthoc_detection.py                    # Line A (null)
+python scripts/analysis/103_replicate_dingwall_seurat_pipeline.py       # Line B (Derm10 4.32x)
+python scripts/analysis/104_train_on_dingwall_derm_labels.py            # Line C (Derm10 5.20x)
+
+# Primary EDEN (Derm2) discovery — Sec 6.4.1
+python scripts/analysis/100_primary_eden_discovery.py                   # sub-cluster fibro predictions
+python scripts/analysis/101_primary_eden_derm_scoring.py                # score vs Data S1C panels
+python scripts/analysis/105_primary_eden_full_dermal.py                 # on full dermal denominator
+
+# Auxiliary: variant-A Dingwall training used for scope comparison
+python scripts/analysis/102_train_on_dingwall_variantA.py
+```
+
+### 7.3 Section 7 — Dahlin Kit-mutant (hematopoiesis)
+
+```bash
+python scripts/analysis/66_dahlin_kit_mutant.py           # class enrichment WT vs Kit-W41
+python scripts/analysis/67_dahlin_within_class.py         # within-class Wilcoxon DE
+python scripts/analysis/73_novel_populations_dahlin.py    # abstain-gated novel pops
+python scripts/analysis/108_dahlin_lineage_metabolism.py  # per-lineage OXPHOS/glycolysis
+```
+
+### 7.4 Section 7.4 — Veres held-out (pancreas)
+
+```bash
+python scripts/analysis/62_time_course_analysis.py        # class fractions across LARRY days (HSC time-course template)
+python scripts/analysis/109_veres_mature_beta.py          # Sec 5.2 adult MAFA/UCN3 quadrant
+python scripts/analysis/110_veres_polyhormonal_alpha.py   # Sec 5.2 polyhormonal alpha cluster
+```
+
+### 7.5 Section 8 — cross-system prototype geometry + interpretability
+
+```bash
+python scripts/analysis/70_prototype_geometry.py          # intra + cross-system cosine + eff-dim
+python scripts/analysis/72_emergent_axes.py               # within-class PCA of 128-d z
+python scripts/analysis/80_prototype_gene_attribution.py  # integrated gradients per prototype
+python scripts/analysis/81_counterfactual_knockouts.py    # per-gene KO delta on cosine
+python scripts/analysis/82_gene_coattribution_modules.py  # gene co-attribution modules
+python scripts/analysis/83_prototype_training_trajectory.py  # prototype drift across curriculum
+python scripts/analysis/84_adversary_purification.py      # test GRL adversary is at chance
+python scripts/analysis/85_hessian_gene_interactions.py   # second-order gene pair Hessian
+python scripts/analysis/63_nestorowa_zero_shot.py         # Nestorowa unlabeled discovery
+```
+
+---
+
+## 8. Figures + supplement
+
+### Main-text figures (`figures/fig{1,2,3,4}_*.pdf`)
+
+```bash
+python scripts/figures/generate_paper_figures.py
+# fig1_perclass_f1.pdf          3-panel per-class held-out F1 bars
+# fig3_dahlin_heatmap.pdf       within-class module-score heatmap
+# fig4_veres_stage_stack.pdf    Veres per-stage class fractions
+# (there is no fig2 in the current paper; numbering is historical)
+```
+
+Figures 5/6 (En1-cKO + Kit-W41 recap) are built by the biology page pipeline
+below — the standalone `regen_fig5_fig6.py` referenced in older notes is not in
+the current tree; use the biology pipeline instead.
+
+### Supplement (`figures/PANDA_supplement.pdf`)
+
+```bash
+python scripts/figures/build_pca_vs_marker_umaps.py      # PCA vs Marker UMAPs per target
+python scripts/figures/build_figure_supplement.py        # combined supplement PDF
+```
+
+### Biology deep-dive supplement pages
+
+Cache UMAPs once, then build per-topic pages, then merge into the supplement:
+
+```bash
+python scripts/figures/biology_00_umap_cache.py
+python scripts/figures/biology_01_dingwall_umap.py
+python scripts/figures/biology_02_primary_eden.py
+python scripts/figures/biology_03_melanoblast_mitf.py
+python scripts/figures/biology_04_dahlin_metabolism.py
+python scripts/figures/biology_05_dahlin_composition.py
+python scripts/figures/biology_06_veres_beta_quadrant.py
+python scripts/figures/biology_07_veres_polyhormonal.py
+python scripts/figures/biology_08_prototype_geometry.py
+python scripts/figures/biology_99_merge_supplement.py    # appends into PANDA_supplement.pdf
+```
+
+Wall-clock: 20-40 min end-to-end (UMAPs dominate).
+
+---
+
+## 9. PDF build
+
+The paper is a self-contained LaTeX document referencing PDFs in `figures/`:
+
+```bash
+cd <repo-root>
+pdflatex -interaction=nonstopmode PAPER.tex   # first pass  (writes .aux)
+pdflatex -interaction=nonstopmode PAPER.tex   # second pass (resolves refs)
+```
+
+`bibtex` is not required — the paper uses an embedded `thebibliography`.
+
+---
+
+## 10. End-to-end make target
+
+The provided `Makefile` covers the canonical skin pipeline end-to-end:
+
+```bash
+make install        # pip install -e .
+make run            # bash scripts/pan_skin/run_all.sh — full skin pipeline
+make test-heldout   # scripts/pan_skin/40_heldout_5fold_cv.py
+make clean          # remove __pycache__ + *.pyc
+```
+
+For a full three-system reproduction, chain the per-section commands above.
+A minimal "everything" recipe:
+
+```bash
+# 1) data
+bash scripts/pan_skin/01_download_tier_a.sh
+bash scripts/pan_skin/02_download_tier_b.sh
+bash scripts/pan_skin/03_download_tier_c.sh
+bash scripts/hematopoiesis/01_download.sh
+bash scripts/hematopoiesis/02_download.sh
+bash scripts/pancreas/01_download.sh
+bash scripts/pancreas/09_download.sh
+
+# 2) corpora
+bash scripts/pan_skin/run_all.sh                     # includes build + train + skin CV
+python scripts/hematopoiesis/02_build_per_dataset.py
+python scripts/hematopoiesis/03_shared_hvgs_and_pca.py
+python scripts/hematopoiesis/10_build_corpus.py
+python scripts/hematopoiesis/11_filter_paper_only.py
+python scripts/pancreas/02_build_per_dataset.py
+python scripts/pancreas/03_shared_hvgs_and_pca.py
+python scripts/pancreas/04_assign_labels.py
+python scripts/pancreas/11_build_corpus.py
+python scripts/pancreas/08_add_baron_split.py
+python scripts/common/generate_missing_holdouts.py
+python scripts/pan_skin/93_add_belote_anchor.py
+
+# 3) all 6 training runs
+for sys in pan_skin hematopoiesis pancreas; do
+  for v in pca marker; do
+    python -m scripts.common.train_panda $sys --variant $v --epochs 8
+  done
+done
+
+# 4) CV + zero-shot
+python -m scripts.common.run_cv --folds 5 --epochs 5
+python -m scripts.common.run_all_zero_shot
+
+# 5) discovery
+bash scripts/common/rerun_all_discovery.sh           # drives scripts/analysis/* end-to-end
+python scripts/analysis/95_adult_beta_validation.py
+
+# 6) figures + PDF
+python scripts/figures/generate_paper_figures.py
+python scripts/figures/build_pca_vs_marker_umaps.py
+python scripts/figures/build_figure_supplement.py
+python scripts/figures/biology_00_umap_cache.py
+for i in 01 02 03 04 05 06 07 08; do
+  python scripts/figures/biology_${i}_*.py
+done
+python scripts/figures/biology_99_merge_supplement.py
+pdflatex -interaction=nonstopmode PAPER.tex && pdflatex -interaction=nonstopmode PAPER.tex
+```
+
+---
+
+## 11. Trouble-shooting
+
+- **`libcusparseLt.so.0: cannot open shared object file`** — you forgot to
+  export `LD_LIBRARY_PATH` **before** Python started. The pip-installed
+  `nvidia-cusparselt-cu12` provides the library; PyTorch does not add its
+  path to the loader search. See section 1.1.
+- **`GLIBCXX_3.4.30 not found`** — your system `libstdc++` is too old; prepend
+  a newer `libstdc++.so.6`'s directory to `LD_LIBRARY_PATH`.
+- **Dingwall GSM -> genotype mapping** (frequent bug source): the correct map is
+  `WT = {GSM6833478, GSM6833479, GSM6833480, GSM6833481}`,
+  `cKO = {GSM6833482, GSM6833483}`.
+  GSM6833480/481 are `rttaControl` (Cre-negative WT), **not** cKO. Getting this
+  wrong flips every En1-cKO enrichment sign.
+- **Pancreas HVG builder OOM** — `scripts/pancreas/03_shared_hvgs_and_pca.py`
+  peaks near 40 GB RAM on the 6-study union. Run on a node with >= 64 GB.
+- **`data/raw` not in git** — it is git-ignored (14+ GB of GEO tars). Rerun
+  section 2 to repopulate.
+- **`stratified split failure` on rare classes** �� small-support classes
+  (< 2 members per fold) are merged into the parent `canonical_label`. If a
+  fold still errors, check that `corpus.h5ad`'s `canonical_label` column has
+  the expected vocabulary; the paper vocab is the union enumerated in
+  `PAPER.tex` Sec 3.
+- **`n_conditions` mismatch** — `PANDAEncoder` reads it from
+  `len(datasets)` in the checkpoint; regenerate the checkpoint if you have
+  added/removed a dataset.
+- **`ContrastiveSampler` in 1-condition data** — auto-disabled when there is
+  only one condition; no config change needed.
+- **DataParallel batch-size** — default `bs=256` is calibrated for 4x A100
+  40 GB. Drop to `bs=64` for a single GPU or you will OOM inside the
+  sub-center prototype attention.
+- **Embeddings not persisted** — after training writes the checkpoint, the 128-d
+  projection is re-computed on demand in every downstream analysis; if you want it
+  cached, re-save the AnnData explicitly via `adata.write_h5ad()`.
+- **Sulic is NOT held out of the standard corpus** — all 4,683 Sulic cells
+  (including the 4,183-cell `held_out_labeled` slice) are inside
+  `data/corpus/pan_skin/harmonized/corpus.h5ad`. Scoring the standard checkpoints on
+  the Sulic slice is a train-set evaluation. Use
+  `scripts/pan_skin/92_retrain_with_sulic_anchor.py` (500-cell anchor corpus,
+  artefacts `97_sulic_anchor_*`) for the honest number, as the paper now does.
+- **Corpus rebuild from raw GEO data is currently not possible from this repo** —
+  the per-dataset loader modules (`panda/data/*_loaders.py`) that
+  `06_build_per_dataset_h5ads.py` / `02_build_per_dataset.py` import were never
+  committed (only stale v2 copies exist under `archive/`). Everything from
+  training onward is reproducible from the harmonized corpora on the Hugging Face
+  mirror; re-deriving the corpora from GEO requires reconstructing the loaders.
+
+---
+
+## 12. Artefact index (from PAPER.tex Section 11)
+
+Every quantitative claim traces to one of:
+
+- **Model checkpoints**: `checkpoints/{system}/{variant}/panda_final.pt`
+- **Marker gene lists**: `panda/markers.yaml`
+- **Corpus builders**: `scripts/{pan_skin,hematopoiesis,pancreas}/`
+- **Unified trainer**: `scripts/common/train_panda.py`
+- **Unified 5-fold CV**: `scripts/common/run_cv.py` (or `cv_holdout.py`)
+- **Zero-shot driver**: `scripts/common/run_all_zero_shot.py`
+- **External label supplements**:
+  `data/external_labels/{dingwall_supp,haensel,joost2016,mca,mia,byrnes,yu,baccin,melanocyte_anchor}/`
+- **CV outputs**: `discovery/{system}/{variant}/cv_5fold{,_seed1,_seed2}.json`
+- **Zero-shot summaries**:
+  - `discovery/pancreas/{pca,marker}/baron_summary.json`
+  - `discovery/pancreas/{pca,marker}/veres_summary.json`
+  - `discovery/hematopoiesis/{pca,marker}/nestorowa_summary.json`
+  - `discovery/pan_skin/{pca,marker}/sulic_summary.json`
+  - `discovery/pan_skin/{pca,marker}/belote_summary.json`
+- **Adult-beta panel**: `discovery/pancreas/marker/95_adult_beta_validation.json`
+- **Pathway modules**: `discovery/{system}/marker/57_pathway_class_by_module_{padj,delta}.tsv`
+- **Dingwall EDEN validation**:
+  Line A `discovery/pan_skin/marker/98_eden_summary.json`;
+  Line B `data/processed/dingwall_replica/dingwall_replica.h5ad`,
+         `replica_cluster_20_qc.json`, `replica_marker_matches.csv`;
+  Line C `discovery/pan_skin/marker/104_dingwall_derm_summary.json` + prediction CSVs.
+- **Primary EDEN (Derm2)**:
+  `discovery/pan_skin/marker/100_primary_eden_discovery.csv`,
+  `100_primary_eden_summary.json`,
+  `101_derm_identity_summary.json`,
+  `101_derm_subcluster_scores.csv`.
+- **Dingwall other**:
+  `discovery/pan_skin/marker/57_pathway_analysis.csv`,
+  `90_dingwall_marker_deep_dive.csv`.
+- **Dahlin Kit-mutant**:
+  `discovery/hematopoiesis/marker/92_dahlin_marker_deep_dive.csv`, `dahlin_summary.json`.
+- **Veres deep-dive**: `discovery/pancreas/marker/91_veres_marker_deep_dive.csv`.
+
+Central architecture: `panda/model.py`. Composite loss lives in the same file
+(`supcon_loss`, `vicreg_loss`, `hsic_biased`, `subcenter_angular_infonce`,
+`prototype_repulsion`) and is imported as `from panda import PANDAEncoder, ...`.
+
+## Data mirror
+
+Full data (~195 GB corpus + raw + processed + external labels) is mirrored to Hugging Face at [bryan7264/PANDA](https://huggingface.co/bryan7264/PANDA). Fetch with:
+
+```bash
+huggingface-cli download bryan7264/PANDA --local-dir . --include "data/corpus/pan_skin/**"
+```
+
+Priority folders (fetch these first for the minimum-reproducible pipeline):
+- `data/corpus/{pan_skin,hematopoiesis,pancreas}/harmonized/` — training corpora
+- `data/external_labels/` — paper-supplement label files
+- `checkpoints/{pan_skin,hematopoiesis,pancreas}/marker/` — trained weights
+
+Bulk (only needed to reproduce corpus builds from scratch):
+- `data/raw/` — GEO downloads (regenerable from `scripts/*/03_download*.sh`)
+- `data/corpus/tier{1,2,3}/` — pretraining tier data
+- `data/processed/` — intermediate build artefacts

REVIEW_REPORT.md

@@ -0,0 +1,235 @@
+# Review Report — PANDA (Pan-tissue Adversarial Normalized Domain-invariant Anchored MLP)
+
+**Review Mode**: Post-Results Review
+**Date**: 2026-07-13
+**Reviewer**: Autonomous Review Skill (invoked from wakeup loop)
+
+## Summary
+
+**Overall Status**: 🟡 PASS WITH CONCERNS
+
+Comprehensive audit of PAPER.tex against all discovery-artifact JSONs / CSVs
+under `pan_skin_discovery/`, `hematopoiesis_discovery/`, `pancreas_discovery/`,
+and `scripts/sulic/`. A single discrepancy was found (pan-skin CV metrics)
+and corrected on this pass. No fabrication, no data leakage, no missing
+traceability. Codebase organisation is clean (legacy preserved in
+`legacy/`; PANDA-visible content grep-clean of forbidden PRISM/TabPFN/CellTab
+terms). Warnings relate to overfull hboxes in LaTeX (cosmetic) and one
+partial JSON that could be pruned.
+
+## Critical Issues (Must Fix)
+
+None remaining after this pass.
+
+**Fixed on this pass:**
+- **CRITICAL-1 (RESOLVED)**: PAPER.tex reported pan-skin CV
+  `accuracy 0.9835 ± 0.0009` and `macro AUROC 0.9995 ± 0.0001`.
+  Actual value in `pan_skin_discovery/60_heldout_5fold_cv.json`
+  is `0.9851 ± 0.0013` acc, `0.9994 ± 0.0002` AUROC.
+  Corrected in the abstract (line 30) and results table (lines 113, 115).
+  The absolute delta on accuracy is +0.0016, well within a
+  single fold's variance, but publication-grade text must match the JSON.
+
+## Warnings (Should Fix)
+
+- **WARN-1**: `pan_skin_discovery/60_heldout_5fold_partial.json`,
+  `pancreas_discovery/61_pancreas_heldout_5fold_partial.json`, and
+  `hematopoiesis_discovery/61_hematopoiesis_heldout_5fold_partial.json`
+  are intermediate-run artefacts that no longer serve a purpose
+  now that the full CV JSONs exist. They should be deleted to
+  avoid confusion.
+- **WARN-2**: LaTeX overfull hbox warnings at lines 618-626 (URLs in
+  Reproducibility section run off the right margin). Cosmetic only —
+  the URLs remain fully readable.
+- **WARN-3**: Discovery scripts contain `panda/pan_skin/model.py`
+  hardcoded absolute paths in some places; reproducibility on a fresh
+  clone at a different root would need path-fixup or a `PANDA_ROOT`
+  env var. Documented in `README.md`.
+
+## Observations (Nice to Fix)
+
+- Pan-skin CV is the strongest system (0.9851 acc / 0.9994 AUROC / all
+  13 classes at F1 > 0.88). Sulic single-dataset PANDA reproduction
+  reaches 0.9996 mean AUROC (5-fold, 5-seed ensemble), effectively
+  ceiling the PCA-only regime on that reference task.
+- Pan-pancreatic CV at 0.7914 accuracy is honest — this system has
+  the largest developmental-stage gap between training corpus
+  (Bastidas E15.5 + Byrnes E12-E17 + Baron adult) and Veres in vitro
+  hPSC target. The Baron mouse-test-half at 0.805 acc after adult
+  anchor addition (up from 0.367 with embryonic-only training)
+  is empirical proof of the corpus-diversity claim.
+- Every mechanistic p-value in the abstract traces to a specific
+  CSV row (spot-checked 5 of them below).
+
+## Integrity Verdict
+
+| Check                                    | Status |
+|------------------------------------------|--------|
+| Fabrication detected                     | No     |
+| Data leakage detected                    | No     |
+| All results traceable to code            | Yes    |
+| Statistical claims valid                 | Yes    |
+| Held-out validation is truly held-out    | Yes    |
+| Metrics in PAPER match JSON/CSV outputs  | Yes (after fix) |
+
+**Traceability spot-checks (5 random claims from the paper):**
+
+1. Dingwall melanocyte $\sim 2\times$ enrichment, Fisher $p = 4.0 \times 10^{-6}$
+   → `pan_skin_discovery/53_en1_cko_class_enrichment.csv` row `melanocyte`
+     log2fc=+1.03, p=4.0e-6. ✓
+2. Dahlin erythroid apoptosis-pro $p = 6.6 \times 10^{-123}$
+   → `hematopoiesis_discovery/67_dahlin_module_scores.csv` row
+     `class=erythroid, module=Apoptosis_pro` MannU_p ≈ 6.6e-123. ✓
+3. Sulic mean AUROC $0.9996 \pm 0.0007$
+   → `scripts/sulic/sulic_panda_heldout_results.json`
+     testA.mean_auroc = 0.999554, std = 0.000681. ✓
+4. Veres SC-alpha $=78\%$ at Stage 6
+   → `pancreas_discovery/64_sharon_class_per_stage.csv` col `6` row
+     `alpha` = 0.78. ✓
+5. Pan-hematopoietic CV acc $0.9403 \pm 0.0024$
+   → `hematopoiesis_discovery/61_hematopoiesis_heldout_5fold_cv.json`
+     mean_acc = 0.9403, std_acc = 0.0024. ✓
+
+**Held-out set independence:**
+
+- Sulic testA held-out: cells with `run` matching the fold's held-out
+  MAdCAM-1 sample-of-origin bucket, verified in `sulic_panda_heldout.py`
+  by set-intersection assertion.
+- Pan-skin 5-fold CV: stratified GroupKFold by `dataset_id`; no cell
+  appears in both train and test of the same fold.
+- Dingwall (25,344 cells): This dataset is NEVER in the training corpus.
+  Verified by `git log scripts/pan_skin/06_build_per_dataset_h5ads.py`
+  — the include-list explicitly excludes `GSE220977`.
+- Dahlin (61,122 cells): Not in Weinreb LARRY corpus.
+- Veres (57,297 human hPSC cells): Not in the mouse pancreatic corpus.
+
+**Preprocessing leakage check:**
+
+- HVG selection: `07_build_shared_hvgs_and_pca.py` fits union+majority
+  HVG list on training data only. Verified.
+- PCA basis: fit on training-only 30-50k cell subsample. Verified.
+- Normalisation stats (mean/std for PCA input): computed on training
+  data only. Held-out targets go through the same `normalize_total +
+  log1p` transform but use the training-fit centre/scale, not their own.
+
+## Checklist Summary
+
+| Category                    | Items Checked | Passed | Failed | N/A |
+|-----------------------------|---------------|--------|--------|-----|
+| Code Correctness            | 17            | 17     | 0      | 0   |
+| Data Leakage                | 6             | 6      | 0      | 0   |
+| Experimental Design         | 6             | 6      | 0      | 0   |
+| Reproducibility             | 5             | 4      | 0      | 1 (single-cmd end-to-end not exposed as one target) |
+| Result Authenticity         | 8             | 8      | 0      | 0   |
+| Statistical Validity        | 6             | 6      | 0      | 0   |
+| Figure Audit                | 6             | 6      | 0      | 0   |
+| Paper Cross-Check           | 8             | 7      | 1 (fixed)| 0 |
+
+## Detailed Findings
+
+### Section 1 — Code Correctness
+
+- Architecture matches paper description exactly. `panda/pan_skin/model.py`
+  is one file, re-used by all three systems.
+- Trunk: `PCA(50) → 512 → 512 → 256` with LayerNorm+GELU+Dropout(0.2). ✓
+- Projection head 128-d L2-normalised for SupCon. ✓
+- $K$ learnable class prototypes with EMA momentum 0.99. ✓
+- Dataset and depth adversaries behind GradReverse. ✓
+- BBSE label-shift correction implemented in inference path. ✓
+- Confidence gate at cos<0.3 for novel-population flagging. ✓
+
+### Section 6 — Result Authenticity
+
+Every quantitative claim in the abstract and body traces to at least
+one artefact under the three `*_discovery/` directories. The five
+random spot-checks above all matched. TRAINING logs and the
+`checkpoints/{system}/panda_final.pt` timestamps are consistent
+with the JSON output timestamps (checkpoints predate the discovery
+JSONs by minutes).
+
+### Section 7 — Overfitting & Generalisation
+
+- 5-fold CV std deviations are tight (0.0009–0.0032 on acc), suggesting
+  the model is not overfitting one fold.
+- Cross-dataset transfer to Dingwall/Dahlin/Veres works despite these
+  targets sharing no cells with training.
+- BBSE correction adjusts for the label-shift between training and
+  held-out prior distributions and is applied uniformly.
+
+## Recommendations for Continued Polish
+
+- Delete the three `*_partial.json` intermediate artefacts.
+- Ensure REVIEW_REPORT.md is regenerated on major experimental changes.
+- Consider adding a `make reproduce` target that runs the full
+  end-to-end pipeline from raw data through discovery.
+
+## Addendum — 2026-07-19 true-zero-shot pass
+
+The following additional strict-zero-shot experiments were added to §4 of PAPER.tex:
+
+- **§4.1 Baron test-half (943 cells)** — pan-pancreas checkpoints run on the held-out
+  labeled test half, both variants. Raw acc PCA=0.6734 / Marker=0.4910; after merging
+  PANDA's higher-resolution subprototypes into Baron's flat labels (adult-β→β,
+  adult-α→α), biology-merged acc PCA=0.8324 / Marker=0.8621 (F1 0.6656 / 0.7672).
+  Artefacts: `discovery/pancreas/{pca,marker}/93_baron_{zero_shot,merged_metrics}.json`.
+
+- **§4.2 Nestorowa Smart-seq2 (1,068 FACS-labeled cells)** — pan-HSC checkpoints,
+  both variants. Coarse LT-HSC vs. HSPC acc = 0.7978 both. Fine distribution shifts
+  between PCA (MPP-heavy) and Marker (myeloid-heavy: Ctsg/Mpo LMPP TF panel);
+  documented honestly in §4.2. Artefacts:
+  `discovery/hematopoiesis/{pca,marker}/94_nestorowa_zero_shot.json`.
+
+- **§4.3 Sulic (pan-skin retrained without Sulic)** — pan-skin corpus rebuilt from
+  103,550 → 98,867 cells, HVG list + PCA basis refit on the reduced corpus, both
+  variants retrained from scratch, then inference on Sulic labels. Runs at
+  `scripts/pan_skin/91_retrain_without_sulic.py`; final numbers write to
+  `discovery/pan_skin/{pca,marker}/96_sulic_zero_shot.json` when training completes.
+
+- **§4.4 Rigor caveats** integrating findings from an independent audit against
+  the three source papers (Dingwall 2024 PMC10872420, Dahlin 2018 PMC5969381,
+  Veres 2019 PMC6903417). Confirmed canonical markers and central mechanistic claims;
+  softened one population size (EDEN 5,053 vs. Dingwall's cluster-20 ~900),
+  post-hoc-validated the "adult-β" naming (MAFA 9.8× / IAPP 2.5× / UCN3 2.1×
+  enriched vs. base beta; MNX1 depressed), and reframed the SC-δ reinterpretation
+  as a testable hypothesis rather than a claim.
+
+## Sign-off
+
+This report closes the post-results review gate. The corrected paper
+metrics ensure every quantitative claim traces cleanly to a JSON/CSV
+under the discovery directories.
+
+## Addendum — 2026-08-19 re-audit (supersedes parts of the above)
+
+A full re-audit of the pipeline against the mirrored data found that several
+sign-offs above were incorrect. Corrections applied to the repo and PAPER.tex:
+
+1. **"BBSE label-shift correction implemented in inference path ✓" was wrong.**
+   No BBSE exists in the canonical inference path (`scripts/common/`); the one
+   legacy implementation (`scripts/pan_skin/30_zero_shot_aldrich.py`) used an
+   unconditionally-uniform training prior (a `np.bincount(np.arange(K))` bug)
+   and was never valid BBSE. The paper's inference section no longer claims it.
+   The cos<0.3 abstain gate likewise exists only in that legacy script.
+2. **"No data leakage" was wrong for Sulic.** All 4,683 Sulic cells — including
+   the 4,183-cell `held_out_labeled` slice — are inside the standard pan-skin
+   corpus (verified by barcode intersection). The paper's Sulic §5.5 numbers were
+   a training-set evaluation; replaced with the anchor-retrain results
+   (`97_sulic_anchor_zero_shot.json`). Nestorowa/Belote/Baron/Veres holdouts were
+   re-verified: zero barcode overlap with their training corpora.
+3. **The multi-seed "33/35" comparison was invalid.** `cv_5fold_seed{1,2}.json`
+   were produced on different corpus builds (e.g. skin: 103,550 cells / 24
+   classes vs the canonical 45,387 / 13) by a different CV driver. Claim
+   withdrawn; seeds regenerated with `run_cv.py --seed N` on canonical corpora.
+4. **Trainer/inference checkpoint path mismatch fixed.** `train_panda.py` saved
+   to `checkpoints/{system}_v3/` while all consumers read `checkpoints/{system}/`.
+5. **Curriculum description corrected.** The paper described LODO/RankMe-gated
+   stages, GRL ramping, mixup, and a P×K×D sampler; the implementation uses
+   fixed epoch-gated stages, constant λ_GRL=0.1, and uniform shuffling. The
+   prototype-repulsion term contributed zero gradient (detached EMA buffer) and
+   was removed from the objective (behaviour-preserving) and from the loss lists.
+6. **Reproducibility fixes.** All ~90 scripts now resolve the repo root via
+   `PANDA_ROOT` (env-overridable, `__file__`-relative default) instead of
+   hardcoded `/home/bcheng/PRISM`; Baron zero-shot reproduces bit-exactly from
+   the mirrored checkpoints (PCA 0.9426/0.6027, Marker 0.8190/0.5888).
+   Known gap: the per-dataset loader modules (`panda/data/*_loaders.py`) were
+   never committed, so corpus rebuild from raw GEO data requires reconstruction.

data/raw/GSE220977_combined.h5ad

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e3b4389d6ab0cbddbd244b0e04df46ab768193b7866ab743c253d6022b06ee74
+size 53962253

discovery/hematopoiesis/marker/66_dahlin_enrichment.csv

@@ -0,0 +1,14 @@
+class,n_Kit_W41,n_WT,pct_Kit_W41,pct_WT,log2_fold,fisher_p
+pro-B,6,21,0.041,0.091,-0.998,0.11212905224791753
+megakaryocyte,1584,3965,10.794,17.164,-0.669,2.3319930527729648e-67
+endothelial,1,4,0.007,0.017,-0.667,0.6548581003415141
+lymphoid,55,139,0.375,0.602,-0.667,0.0024186029526503845
+basophil-mast,245,569,1.67,2.463,-0.558,1.5509467694754263e-07
+MPP,3823,8372,26.051,36.241,-0.476,3.794115798599155e-96
+naive-B,1,3,0.007,0.013,-0.345,1.0
+monocyte,157,242,1.07,1.048,0.033,0.8365429559094492
+macrophage,21,28,0.143,0.121,0.256,0.5609714894758803
+LT-HSC,25,32,0.17,0.139,0.311,0.4968196833343731
+erythroid,7045,8057,48.007,34.877,0.461,1.598221798910364e-141
+T-cell,4,4,0.027,0.017,0.655,0.719202247843629
+myeloid,1708,1665,11.639,7.207,0.691,6.738577350112824e-48

discovery/hematopoiesis/marker/66_dahlin_enrichment_pooled_CONFOUNDED.csv

@@ -0,0 +1,14 @@
+class,n_Kit_W41,n_WT,pct_Kit_W41,pct_WT,log2_fold,fisher_p
+lymphoid,55,639,0.375,1.376,-1.852,1.1197303892308873e-28
+pro-B,6,60,0.041,0.129,-1.461,0.0034713936539364585
+MPP,3823,27931,26.051,60.135,-1.207,0.0
+LT-HSC,25,135,0.17,0.291,-0.725,0.012107699545406006
+megakaryocyte,1584,6596,10.794,14.201,-0.395,4.402462055443566e-27
+endothelial,1,5,0.007,0.011,0.077,1.0
+naive-B,1,4,0.007,0.009,0.34,1.0
+basophil-mast,245,603,1.67,1.298,0.366,0.0010286727846522193
+T-cell,4,10,0.027,0.022,0.525,0.7541626787698259
+monocyte,157,315,1.07,0.678,0.662,5.079726729725271e-06
+macrophage,21,42,0.143,0.09,0.695,0.10294110685379751
+erythroid,7045,8352,48.007,17.982,1.417,0.0
+myeloid,1708,1755,11.639,3.779,1.623,1.619214162006283e-243

discovery/hematopoiesis/marker/66_dahlin_gate_negative_control.csv

@@ -0,0 +1,14 @@
+class,n_WT_LK,n_WT_LSK,pct_WT_LK,pct_WT_LSK,log2_fold,fisher_p
+lymphoid,139,500,0.602,2.142,-1.824,1.101428413327718e-48
+LT-HSC,32,103,0.139,0.441,-1.641,6.895372576865848e-10
+MPP,8372,19559,36.241,83.779,-1.209,0.0
+pro-B,21,39,0.091,0.167,-0.847,0.02736897086821087
+T-cell,4,6,0.017,0.026,-0.47,0.7539139981439633
+megakaryocyte,3965,2631,17.164,11.27,0.607,2.4318445901615237e-74
+macrophage,28,14,0.121,0.06,0.966,0.030673995788373296
+naive-B,3,1,0.013,0.004,1.015,0.37236720905366727
+endothelial,4,1,0.017,0.004,1.337,0.21628326401517584
+monocyte,242,73,1.048,0.313,1.731,5.957647765132e-23
+basophil-mast,569,34,2.463,0.146,4.041,3.792290946441402e-129
+myeloid,1665,90,7.207,0.386,4.21,0.0
+erythroid,8057,295,34.877,1.264,4.782,0.0

discovery/pan_skin/marker/107_dingwall_class_deg_count.csv

@@ -1,13 +1,13 @@
-class,n_cKO,n_WT,n_DEG,n_up,n_down,top_up,top_down
-HF-placode,259,287,12,12,0,2610027F03Rik;Mybpc1;Ccdc3;Kcnh7;Cntn5;Bmpr1b;Tnc;Esr1;Pcdh9;Meis2,
-melanoblast,2108,3035,8,7,1,2610027F03Rik;Mybpc1;Bmpr1b;En1;Kcnh7;Ttn;Esr1,Ctnnd2
-fibroblast-reticular,5464,8334,4,4,0,2610027F03Rik;Ttn;Mybpc1;Krt5,
-spinous,93,249,3,2,1,2610027F03Rik;Slc1a3,Acer3
-HF-ORS,118,210,2,2,0,2610027F03Rik;Mybpc1,
-basal-IFE,152,183,2,2,0,2610027F03Rik;Meis2,
-melanocyte,102,88,2,2,0,2610027F03Rik;Bmpr1b,
-endothelial,1501,2107,1,1,0,2610027F03Rik,
-immune,436,603,1,1,0,2610027F03Rik,
-granular,89,173,1,1,0,2610027F03Rik,
-melanocyte-precursor,45,78,1,1,0,2610027F03Rik,
-sebaceous,33,53,0,0,0,,
+class,n_cKO,n_WT,n_DEG,n_up,n_down,top_up,top_down
+HF-placode,259,287,12,12,0,2610027F03Rik;Mybpc1;Ccdc3;Kcnh7;Cntn5;Bmpr1b;Tnc;Esr1;Pcdh9;Meis2;Grip1;Runx1t1,
+melanoblast,2108,3035,8,7,1,2610027F03Rik;Mybpc1;Bmpr1b;En1;Kcnh7;Ttn;Esr1,Ctnnd2
+fibroblast-reticular,5464,8334,4,4,0,2610027F03Rik;Ttn;Mybpc1;Krt5,
+spinous,93,249,3,2,1,2610027F03Rik;Slc1a3,Acer3
+HF-ORS,118,210,2,2,0,2610027F03Rik;Mybpc1,
+basal-IFE,152,183,2,2,0,2610027F03Rik;Meis2,
+melanocyte,102,88,2,2,0,2610027F03Rik;Bmpr1b,
+endothelial,1501,2107,1,1,0,2610027F03Rik,
+granular,89,173,1,1,0,2610027F03Rik,
+immune,436,603,1,1,0,2610027F03Rik,
+melanocyte-precursor,45,78,1,1,0,2610027F03Rik,
+sebaceous,33,53,0,0,0,,

discovery/pan_skin/marker/107_dingwall_class_deg_count.json

@@ -1,59 +1,59 @@
-{
-  "lfc_threshold": 1.0,
-  "padj_threshold": 0.05,
-  "n_classes_tested": 12,
-  "top_class": "HF-placode",
-  "top_n_DEG": 12,
-  "top_up": "2610027F03Rik;Mybpc1;Ccdc3;Kcnh7;Cntn5;Bmpr1b;Tnc;Esr1;Pcdh9;Meis2",
-  "top_down": "",
-  "ranking": [
-    {
-      "class": "HF-placode",
-      "n_DEG": 12
-    },
-    {
-      "class": "melanoblast",
-      "n_DEG": 8
-    },
-    {
-      "class": "fibroblast-reticular",
-      "n_DEG": 4
-    },
-    {
-      "class": "spinous",
-      "n_DEG": 3
-    },
-    {
-      "class": "HF-ORS",
-      "n_DEG": 2
-    },
-    {
-      "class": "basal-IFE",
-      "n_DEG": 2
-    },
-    {
-      "class": "melanocyte",
-      "n_DEG": 2
-    },
-    {
-      "class": "endothelial",
-      "n_DEG": 1
-    },
-    {
-      "class": "immune",
-      "n_DEG": 1
-    },
-    {
-      "class": "granular",
-      "n_DEG": 1
-    },
-    {
-      "class": "melanocyte-precursor",
-      "n_DEG": 1
-    },
-    {
-      "class": "sebaceous",
-      "n_DEG": 0
-    }
-  ]
+{
+  "lfc_threshold": 1.0,
+  "padj_threshold": 0.05,
+  "n_classes_tested": 12,
+  "top_class": "HF-placode",
+  "top_n_DEG": 12,
+  "top_up": "2610027F03Rik;Mybpc1;Ccdc3;Kcnh7;Cntn5;Bmpr1b;Tnc;Esr1;Pcdh9;Meis2;Grip1;Runx1t1",
+  "top_down": "",
+  "ranking": [
+    {
+      "class": "HF-placode",
+      "n_DEG": 12
+    },
+    {
+      "class": "melanoblast",
+      "n_DEG": 8
+    },
+    {
+      "class": "fibroblast-reticular",
+      "n_DEG": 4
+    },
+    {
+      "class": "spinous",
+      "n_DEG": 3
+    },
+    {
+      "class": "HF-ORS",
+      "n_DEG": 2
+    },
+    {
+      "class": "basal-IFE",
+      "n_DEG": 2
+    },
+    {
+      "class": "melanocyte",
+      "n_DEG": 2
+    },
+    {
+      "class": "endothelial",
+      "n_DEG": 1
+    },
+    {
+      "class": "granular",
+      "n_DEG": 1
+    },
+    {
+      "class": "immune",
+      "n_DEG": 1
+    },
+    {
+      "class": "melanocyte-precursor",
+      "n_DEG": 1
+    },
+    {
+      "class": "sebaceous",
+      "n_DEG": 0
+    }
+  ]
 }

figures/PANDA_all_figures.pdf

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4586d33cce1b0499d3ebbb6e52a09fab4770b68303301763ee13225581ce115e
-size 14422644
+oid sha256:a98c7e9621475d48e6dd8be321084581b5f9d9d4d55e2ac1acb0ad1b28363d1c
+size 16048898

figures/PANDA_supplement.pdf

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5c6a2d3eb575b34d8ea5d24a4d423e16ddf7d32f40b8cc06c374d12ec49940b0
-size 13465296
+oid sha256:01aecd5d926dada793372dafc962878d91908e2c7f2859eedcf51a35175d81e5
+size 12124452

figures/biology/biology_01_dingwall_umap.pdf

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:dad41b3e00db248105504178a4e03ebf951d73c3a23f0d02db32000daf02237c
-size 785379
+oid sha256:f8763029b3fe23db81f089a2d6dc659f8b0f62510561e14ee8897c2200d17000
+size 769877

figures/biology/biology_02_primary_eden.pdf

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c3963d2891a3048e4f67aefb25d4d1ec60daa0a9dac3aec5295177880523ee7f
-size 655555
+oid sha256:fd5290f3a2f9e68ba811a4a977e55de8cbdc253556829ef92f9f6107c24c34c8
+size 646125

figures/biology/biology_03_melanoblast_mitf.pdf

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3bc20beca0634ee2cbdd92016ec63bc526ffd24918610c4a83d44b480177d0cb
-size 177652
+oid sha256:f271d84e95f71fd3cdbcf80f30382d6158b7ed2f2568fd499b07f46a7d135af8
+size 187770

figures/biology/biology_06_veres_beta_quadrant.pdf

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:03cd993aed1a3870ffa3691543034584d64e87b6082603f1c80f990c7f7a4ca9
-size 149339
+oid sha256:dd2bd05927e0ce064591d84955a3cb826d99caba6a3539735e610ad6a71dbe5e
+size 147839

figures/biology/biology_07_veres_polyhormonal.pdf

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:672b853bedee5c765f376717f90e724cc71ee87970e961d3af6472378d7559c1
-size 184982
+oid sha256:1f240751d61b29cb519adc75fafde6cef607f8bc9b0916cde0962c29d9780b44
+size 193704

figures/supplement/26_pca_vs_marker_umaps_veres_by_stage.pdf

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9f2b405614801f1390251b19892b64a8ceaaff83aa788ca911586ec8eafdd43c
-size 373429
+oid sha256:80e1ba2c6442ab8977e4225276222562c96d7d59804978d22573ac4fd6689af7
+size 1287055

panda/__init__.py

@@ -1,16 +1,16 @@
-"""panda: prototype-anchored cell-identity classifier. marker variant is default."""
-__version__ = "1.0"
-
-from .model import (
-    PANDAEncoder,
-    grad_reverse,
-    supcon_loss, vicreg_loss, hsic_biased,
-    subcenter_angular_infonce,
-    prototype_repulsion,
-)
-
-__all__ = [
-    "PANDAEncoder",
-    "grad_reverse", "supcon_loss", "vicreg_loss", "hsic_biased",
-    "subcenter_angular_infonce", "prototype_repulsion",
-]
+"""panda: prototype-anchored cell-identity classifier. marker variant is default."""
+__version__ = "1.0"
+
+from .model import (
+    PANDAEncoder,
+    grad_reverse,
+    supcon_loss, vicreg_loss, hsic_biased,
+    subcenter_angular_infonce,
+    prototype_repulsion,
+)
+
+__all__ = [
+    "PANDAEncoder",
+    "grad_reverse", "supcon_loss", "vicreg_loss", "hsic_biased",
+    "subcenter_angular_infonce", "prototype_repulsion",
+]

panda/markers.yaml

@@ -1,132 +1,135 @@
-# Curated marker-channel input genes per system for PANDA-Marker variant.
-#
-# These are the SIBLING-DISCRIMINATING genes that PCA(50) tends to discard.
-# They are concatenated directly onto PCA(50) as log-normalised expression,
-# bypassing the PCA bottleneck for the low-variance signal that separates
-# closely-related subtypes.
-#
-# Set size targets: 20-40 markers per system. Curated for BOTH:
-#   (a) each class's positive identity marker (recovers within-class)
-#   (b) sibling-pair contrast markers (e.g. Dkk4 for eccrine vs hair placode)
-#
-# All symbols are mouse gene symbols (case-fold applied at inference for
-# cross-species targets like Veres).
-
-pan_skin:
-  # eccrine placode / nascent-gland / hair-placode axis
-  - Dkk4        # eccrine placode > hair placode (Dingwall Fig 2)
-  - Lgr6        # placode marker (Epi3)
-  - Trpv6       # nascent eccrine gland (Epi5/Epi8)
-  - Edar        # both placodes; hair > eccrine
-  - Shh         # hair placode discriminator
-  - Sox9        # hair placode / HF-primary-germ
-  - Lhx2        # hair placode
-  - Foxi3       # placode induction
-  - Wnt6        # placode Wnt program
-  - Wnt7b       # placode Wnt program
-  - Lef1        # Wnt signalling
-  - Bmp7        # placode
-  # eccrine/appendage identity
-  - En1         # spatial repressor of eccrine outside placode
-  - Grhl3       # eccrine-duct master TF
-  - Cldn6       # eccrine tight junction
-  - Kremen2     # placode / eccrine
-  # basal / IFE / spinous / granular
-  - Krt5        # basal
-  - Krt14       # basal
-  - Krt10       # spinous differentiation
-  - Krt1        # spinous differentiation
-  - Trp63       # basal master
-  # melanocyte MITF regulon
-  - Dct         # melanocyte
-  - Mlana       # melanocyte
-  - Tyrp1       # melanocyte
-  - Pmel        # melanocyte
-  - Sox10       # neural crest / melanocyte
-  # endothelial / fibroblast / immune
-  - Pecam1      # endothelial
-  - Cdh5        # endothelial
-  - Col1a1      # fibroblast reticular
-  - Dcn         # fibroblast papillary
-  - Ptprc       # immune (CD45)
-  # dermal niche (EDEN — Dingwall §5)
-  - S100a4      # EDEN dermal niche
-  - Pdgfra      # dermal mesenchyme
-
-hematopoiesis:
-  # LT-HSC / stem
-  - Hlf         # LT-HSC canonical (Komorowska 2017)
-  - Meis1       # HSC/MPP
-  - Mecom       # LT-HSC
-  - Procr       # LT-HSC (EPCR)
-  - Fgd5        # LT-HSC reporter
-  - Mllt3       # LT-HSC self-renewal
-  # MPP substates
-  - Cd48        # MPP1 vs LT
-  - Flt3        # MPP4/LMPP
-  - Sell        # ST-HSC / MPP1 (CD62L)
-  - Slamf1      # SLAM CD150
-  # erythroid
-  - Klf1        # erythroid master TF
-  - Gata1       # erythroid/mega
-  - Car1        # erythroblast
-  - Car2        # erythroblast
-  - Blvrb       # committed erythroid
-  - Hba-a1      # hemoglobin
-  # megakaryocyte
-  - Itga2b      # CD41 megakaryocyte
-  - Pf4         # megakaryocyte
-  - Gp1bb       # megakaryocyte
-  # lymphoid
-  - Dntt        # pre-B / lymphoid
-  - Vpreb1      # pre-B lymphoid
-  - Vpreb3      # pre-B lymphoid
-  - Il7r        # lymphoid progenitor
-  # myeloid / mast / basophil
-  - Elane       # GMP / granulocyte
-  - Mpo         # myeloid
-  - Cpa3        # basophil-mast lineage
-  - Ms4a2       # mast Fc-epsilon-RI
-  - Csf1r       # macrophage
-  # macrophage antimicrobial (myeloid combinatorial identity per §10.3)
-  - Wfdc17
-  - Mmp8
-  - Ctss
-
-pancreas:
-  # endocrine progenitor (Ngn3 → Fev → hormone axis)
-  - Neurog3     # Ngn3 endocrine progenitor
-  - Fev         # Fev+ intermediate EP
-  - Pax4        # alpha/beta bipotential
-  - Insm1       # endocrine progenitor
-  - Neurod1     # beta / late endocrine
-  - Cbfa2t3     # endocrine progenitor
-  - Btbd17      # endocrine progenitor
-  # alpha lineage
-  - Arx         # alpha master
-  - Irx1        # alpha
-  - Irx2        # alpha (Veres §7 finding)
-  - Mafb        # alpha
-  - Gcg         # glucagon
-  # beta lineage
-  - Nkx6-1      # beta master
-  - Mnx1        # beta
-  - Ins1        # insulin
-  - Ins2        # insulin
-  - Mafa        # mature beta
-  - Pdx1        # pancreatic progenitor + beta
-  # delta / gamma / epsilon
-  - Sst         # delta
-  - Hhex        # delta
-  - Ppy         # gamma (PP)
-  - Aqp3        # gamma
-  - Ghrl        # epsilon
-  # exocrine / stromal
-  - Prss1       # acinar
-  - Cel         # acinar
-  - Ptf1a       # acinar/pro-endocrine
-  - Krt19       # ductal
-  - Sox9        # ductal / progenitor
-  # embryonic hematopoiesis contamination flag (per §10.3 finding)
-  - Hbb-bs
-  - Hba-a1
+# Curated marker-channel input genes per system for PANDA-Marker variant.
+#
+# These are the SIBLING-DISCRIMINATING genes that PCA(50) tends to discard.
+# They are concatenated directly onto PCA(50) as z-scored RAW-count expression
+# (clipped to +/-5; see prepare_batches in scripts/common/train_panda.py — the
+# values are read from the un-normalized AnnData, then z-scored), bypassing the
+# PCA bottleneck for the low-variance signal that separates closely-related
+# subtypes. Checkpoints trained after 2026-08 store the training-corpus marker
+# mean/std (marker_mu/marker_sig) so inference reuses the training scale.
+#
+# Set size targets: 20-40 markers per system. Curated for BOTH:
+#   (a) each class's positive identity marker (recovers within-class)
+#   (b) sibling-pair contrast markers (e.g. Dkk4 for eccrine vs hair placode)
+#
+# All symbols are mouse gene symbols (case-fold applied at inference for
+# cross-species targets like Veres).
+
+pan_skin:
+  # eccrine placode / nascent-gland / hair-placode axis
+  - Dkk4        # eccrine placode > hair placode (Dingwall Fig 2)
+  - Lgr6        # placode marker (Epi3)
+  - Trpv6       # nascent eccrine gland (Epi5/Epi8)
+  - Edar        # both placodes; hair > eccrine
+  - Shh         # hair placode discriminator
+  - Sox9        # hair placode / HF-primary-germ
+  - Lhx2        # hair placode
+  - Foxi3       # placode induction
+  - Wnt6        # placode Wnt program
+  - Wnt7b       # placode Wnt program
+  - Lef1        # Wnt signalling
+  - Bmp7        # placode
+  # eccrine/appendage identity
+  - En1         # spatial repressor of eccrine outside placode
+  - Grhl3       # eccrine-duct master TF
+  - Cldn6       # eccrine tight junction
+  - Kremen2     # placode / eccrine
+  # basal / IFE / spinous / granular
+  - Krt5        # basal
+  - Krt14       # basal
+  - Krt10       # spinous differentiation
+  - Krt1        # spinous differentiation
+  - Trp63       # basal master
+  # melanocyte MITF regulon
+  - Dct         # melanocyte
+  - Mlana       # melanocyte
+  - Tyrp1       # melanocyte
+  - Pmel        # melanocyte
+  - Sox10       # neural crest / melanocyte
+  # endothelial / fibroblast / immune
+  - Pecam1      # endothelial
+  - Cdh5        # endothelial
+  - Col1a1      # fibroblast reticular
+  - Dcn         # fibroblast papillary
+  - Ptprc       # immune (CD45)
+  # dermal niche (EDEN — Dingwall §5)
+  - S100a4      # EDEN dermal niche
+  - Pdgfra      # dermal mesenchyme
+
+hematopoiesis:
+  # LT-HSC / stem
+  - Hlf         # LT-HSC canonical (Komorowska 2017)
+  - Meis1       # HSC/MPP
+  - Mecom       # LT-HSC
+  - Procr       # LT-HSC (EPCR)
+  - Fgd5        # LT-HSC reporter
+  - Mllt3       # LT-HSC self-renewal
+  # MPP substates
+  - Cd48        # MPP1 vs LT
+  - Flt3        # MPP4/LMPP
+  - Sell        # ST-HSC / MPP1 (CD62L)
+  - Slamf1      # SLAM CD150
+  # erythroid
+  - Klf1        # erythroid master TF
+  - Gata1       # erythroid/mega
+  - Car1        # erythroblast
+  - Car2        # erythroblast
+  - Blvrb       # committed erythroid
+  - Hba-a1      # hemoglobin
+  # megakaryocyte
+  - Itga2b      # CD41 megakaryocyte
+  - Pf4         # megakaryocyte
+  - Gp1bb       # megakaryocyte
+  # lymphoid
+  - Dntt        # pre-B / lymphoid
+  - Vpreb1      # pre-B lymphoid
+  - Vpreb3      # pre-B lymphoid
+  - Il7r        # lymphoid progenitor
+  # myeloid / mast / basophil
+  - Elane       # GMP / granulocyte
+  - Mpo         # myeloid
+  - Cpa3        # basophil-mast lineage
+  - Ms4a2       # mast Fc-epsilon-RI
+  - Csf1r       # macrophage
+  # macrophage antimicrobial (myeloid combinatorial identity per §10.3)
+  - Wfdc17
+  - Mmp8
+  - Ctss
+
+pancreas:
+  # endocrine progenitor (Ngn3 → Fev → hormone axis)
+  - Neurog3     # Ngn3 endocrine progenitor
+  - Fev         # Fev+ intermediate EP
+  - Pax4        # alpha/beta bipotential
+  - Insm1       # endocrine progenitor
+  - Neurod1     # beta / late endocrine
+  - Cbfa2t3     # endocrine progenitor
+  - Btbd17      # endocrine progenitor
+  # alpha lineage
+  - Arx         # alpha master
+  - Irx1        # alpha
+  - Irx2        # alpha (Veres §7 finding)
+  - Mafb        # alpha
+  - Gcg         # glucagon
+  # beta lineage
+  - Nkx6-1      # beta master
+  - Mnx1        # beta
+  - Ins1        # insulin
+  - Ins2        # insulin
+  - Mafa        # mature beta
+  - Pdx1        # pancreatic progenitor + beta
+  # delta / gamma / epsilon
+  - Sst         # delta
+  - Hhex        # delta
+  - Ppy         # gamma (PP)
+  - Aqp3        # gamma
+  - Ghrl        # epsilon
+  # exocrine / stromal
+  - Prss1       # acinar
+  - Cel         # acinar
+  - Ptf1a       # acinar/pro-endocrine
+  - Krt19       # ductal
+  - Sox9        # ductal / progenitor
+  # embryonic hematopoiesis contamination flag (per §10.3 finding)
+  - Hbb-bs
+  - Hba-a1

panda/model.py

@@ -1,229 +1,229 @@
-"""panda encoder with two input variants (pca / marker), sub-center prototypes."""
-from __future__ import annotations
-import math
-from dataclasses import dataclass
-from typing import Optional, List
-
-import torch
-import torch.nn as nn
-import torch.nn.functional as F
-from torch.autograd import Function
-
-
-# gradient reversal layer
-
-class GradReverse(Function):
-    @staticmethod
-    def forward(ctx, x, lam):
-        ctx.lam = lam
-        return x.view_as(x)
-
-    @staticmethod
-    def backward(ctx, g):
-        return -ctx.lam * g, None
-
-
-def grad_reverse(x, lam):
-    return GradReverse.apply(x, lam)
-
-
-# encoder with variant + sub-centers
-
-class PANDAEncoder(nn.Module):
-    """trunk + projection head + sub-center prototypes.
-
-    args:
-      variant     : "pca" or "marker"
-      n_pca       : 50
-      n_markers   : m >= 0. required > 0 if variant == "marker".
-      n_classes   : K
-      n_sub       : sub-centers per class (default 3)
-      d_hidden, d_repr, d_proj: trunk sizing
-      n_datasets  : for the dataset adversary head
-    """
-
-    def __init__(
-        self,
-        variant: str = "pca",
-        n_pca: int = 50,
-        n_markers: int = 0,
-        d_hidden: int = 512,
-        d_repr: int = 256,
-        d_proj: int = 128,
-        n_classes: int = 10,
-        n_sub: int = 3,
-        n_datasets: int = 1,
-        dropout: float = 0.2,
-    ):
-        super().__init__()
-        assert variant in ("pca", "marker"), variant
-        if variant == "marker":
-            assert n_markers > 0, "PANDA-Marker requires n_markers>0"
-        self.variant = variant
-        self.n_pca = n_pca
-        self.n_markers = n_markers if variant == "marker" else 0
-        self.n_classes = n_classes
-        self.n_sub = n_sub
-        self.n_datasets = n_datasets
-
-        input_dim = n_pca + self.n_markers
-        self.input_dim = input_dim
-
-        self.trunk = nn.Sequential(
-            nn.Linear(input_dim, d_hidden), nn.LayerNorm(d_hidden), nn.GELU(), nn.Dropout(dropout),
-            nn.Linear(d_hidden, d_hidden), nn.LayerNorm(d_hidden), nn.GELU(), nn.Dropout(dropout),
-            nn.Linear(d_hidden, d_repr), nn.LayerNorm(d_repr), nn.GELU(),
-        )
-        self.projection = nn.Sequential(
-            nn.Linear(d_repr, d_repr), nn.GELU(),
-            nn.Linear(d_repr, d_proj),
-        )
-        self.classifier = nn.Sequential(nn.Linear(d_repr + 2, n_classes))
-        self.dom_adv = nn.Sequential(nn.Linear(d_repr, 128), nn.ReLU(), nn.Linear(128, n_datasets))
-        self.depth_adv = nn.Sequential(nn.Linear(d_repr, 64), nn.ReLU(), nn.Linear(64, 1))
-
-        # sub-center prototypes (K, n_sub, d_proj), L2-normalised per sub-center
-        self.register_buffer(
-            "prototypes",
-            F.normalize(torch.randn(n_classes, n_sub, d_proj), dim=-1),
-        )
-        # EMA momentum as a buffer so we can overwrite it in place
-        self.register_buffer("proto_ema", torch.tensor(0.99))
-
-    @torch.no_grad()
-    def update_prototypes(self, z_norm: torch.Tensor, y: torch.Tensor):
-        """ema update: assign each in-class cell to nearest sub-center, take the mean."""
-        ema = float(self.proto_ema.item())
-        for c in torch.unique(y):
-            mask = y == c
-            if not mask.any():
-                continue
-            zc = z_norm[mask]                              # (n_c, d_proj)
-            protos_c = self.prototypes[c]                  # (n_sub, d_proj)
-            sims = zc @ protos_c.T                         # (n_c, n_sub)
-            assign = sims.argmax(dim=1)                    # each cell -> nearest sub-center
-            for k in range(self.n_sub):
-                m2 = assign == k
-                if not m2.any():
-                    continue
-                new = F.normalize(zc[m2].mean(dim=0), dim=0)
-                self.prototypes[c, k] = F.normalize(
-                    ema * self.prototypes[c, k] + (1 - ema) * new, dim=0
-                )
-
-    @torch.no_grad()
-    def max_sub_cos(self, z_norm: torch.Tensor) -> torch.Tensor:
-        """(B, K) cos(z, best sub-center) per class."""
-        B = z_norm.size(0); K, n_sub, D = self.prototypes.shape
-        sims = torch.einsum("bd,ksd->bks", z_norm, self.prototypes)   # (B, K, n_sub)
-        return sims.max(dim=2).values                                 # (B, K)
-
-    def forward(
-        self,
-        x_pca: torch.Tensor,
-        aux: torch.Tensor,
-        x_markers: Optional[torch.Tensor] = None,
-        lam_dann: float = 0.0,
-    ) -> dict:
-        if self.variant == "marker":
-            assert x_markers is not None and x_markers.size(1) == self.n_markers
-            x = torch.cat([x_pca, x_markers], dim=1)
-        else:
-            x = x_pca
-
-        h = self.trunk(x)
-        z_raw = self.projection(h)
-        z = F.normalize(z_raw, dim=1)
-        logits = self.classifier(torch.cat([h, aux], dim=1))
-        h_rev = grad_reverse(h, lam_dann)
-        return {
-            "repr": h,
-            "z": z,
-            "logits": logits,
-            "dom": self.dom_adv(h_rev),
-            "depth": self.depth_adv(h_rev),
-        }
-
-
-# losses
-
-def supcon_loss(z: torch.Tensor, y: torch.Tensor, temperature: float = 0.1) -> torch.Tensor:
-    if z.size(0) < 2:
-        return z.new_zeros(())
-    sim = z @ z.T / temperature
-    sim_max, _ = sim.max(dim=1, keepdim=True)
-    sim = sim - sim_max.detach()
-    logits_mask = torch.ones_like(sim) - torch.eye(z.size(0), device=z.device)
-    exp_sim = torch.exp(sim) * logits_mask
-    log_prob = sim - torch.log(exp_sim.sum(dim=1, keepdim=True) + 1e-12)
-    labels_eq = (y.unsqueeze(0) == y.unsqueeze(1)).float() * logits_mask
-    denom = labels_eq.sum(dim=1).clamp_min(1.0)
-    per = -(labels_eq * log_prob).sum(dim=1) / denom
-    per = per * (labels_eq.sum(dim=1) > 0).float()
-    counts = torch.bincount(y, minlength=int(y.max().item()) + 1).float().clamp_min(1.0)
-    w = 1.0 / counts.sqrt()
-    return (per * w[y]).sum() / w[y].sum().clamp_min(1e-6)
-
-
-def vicreg_loss(z: torch.Tensor, sim_weight: float = 0.0, var_weight: float = 25.0,
-                cov_weight: float = 1.0) -> torch.Tensor:
-    zc = z - z.mean(dim=0, keepdim=True)
-    std = (zc.var(dim=0) + 1e-4).sqrt()
-    var_loss = F.relu(1.0 - std).mean()
-    N, D = zc.shape
-    cov = (zc.T @ zc) / (N - 1)
-    off = cov - torch.diag(torch.diagonal(cov))
-    cov_loss = off.pow(2).sum() / D
-    return var_weight * var_loss + cov_weight * cov_loss
-
-
-def hsic_biased(x: torch.Tensor, y: torch.Tensor,
-                sigma_x: float = 1.0, sigma_y: float = 1.0) -> torch.Tensor:
-    Nx = x.size(0)
-    if Nx < 2:
-        return x.new_zeros(())
-    K = torch.exp(-torch.cdist(x, x) ** 2 / (2 * sigma_x ** 2))
-    L = torch.exp(-torch.cdist(y, y) ** 2 / (2 * sigma_y ** 2))
-    H = torch.eye(Nx, device=x.device) - torch.ones(Nx, Nx, device=x.device) / Nx
-    return (K @ H @ L @ H).trace() / (Nx - 1) ** 2
-
-
-def subcenter_angular_infonce(
-    z: torch.Tensor,                    # (B, d_proj) L2-normalised
-    y: torch.Tensor,                    # (B,)
-    prototypes: torch.Tensor,           # (K, n_sub, d_proj)
-    margin: float = 0.15,               # angular margin in radians
-    temperature: float = 0.07,
-) -> torch.Tensor:
-    """arcface-style angular-margin loss over sub-center prototypes."""
-    B = z.size(0); K, n_sub, D = prototypes.shape
-    sims = torch.einsum("bd,ksd->bks", z, prototypes)          # (B, K, n_sub)
-    max_over_sub = sims.max(dim=2).values                       # (B, K)
-
-    # target class cosine, bump by angular margin, put back
-    target_cos = max_over_sub.gather(1, y.unsqueeze(1)).squeeze(1)   # (B,)
-    target_cos = target_cos.clamp(-1 + 1e-7, 1 - 1e-7)
-    theta = torch.acos(target_cos)
-    target_new_cos = torch.cos(theta + margin)
-
-    logits = max_over_sub.clone()
-    logits.scatter_(1, y.unsqueeze(1), target_new_cos.unsqueeze(1))
-    logits = logits / temperature
-    return F.cross_entropy(logits, y)
-
-
-def prototype_repulsion(prototypes: torch.Tensor, weight: float = 1.0) -> torch.Tensor:
-    """penalise inter-class prototype cosine so eff-dim doesn't collapse."""
-    K, n_sub, D = prototypes.shape
-    centroids = F.normalize(prototypes.mean(dim=1), dim=1)      # (K, D)
-    sim = centroids @ centroids.T                                # (K, K)
-    off = sim - torch.diag(torch.diagonal(sim))
-    return weight * off.pow(2).sum() / (K * (K - 1) + 1e-6)
-
-
-def prototype_infonce_legacy(z, y, prototypes, temperature=0.07):
-    """legacy single-prototype InfoNCE. kept for debugging + old checkpoints."""
-    if prototypes.dim() == 3:
-        prototypes = F.normalize(prototypes.mean(dim=1), dim=1)  # collapse sub-centers
-    logits = z @ prototypes.T / temperature
-    return F.cross_entropy(logits, y)
+"""panda encoder with two input variants (pca / marker), sub-center prototypes."""
+from __future__ import annotations
+import math
+from dataclasses import dataclass
+from typing import Optional, List
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.autograd import Function
+
+
+# gradient reversal layer
+
+class GradReverse(Function):
+    @staticmethod
+    def forward(ctx, x, lam):
+        ctx.lam = lam
+        return x.view_as(x)
+
+    @staticmethod
+    def backward(ctx, g):
+        return -ctx.lam * g, None
+
+
+def grad_reverse(x, lam):
+    return GradReverse.apply(x, lam)
+
+
+# encoder with variant + sub-centers
+
+class PANDAEncoder(nn.Module):
+    """trunk + projection head + sub-center prototypes.
+
+    args:
+      variant     : "pca" or "marker"
+      n_pca       : 50
+      n_markers   : m >= 0. required > 0 if variant == "marker".
+      n_classes   : K
+      n_sub       : sub-centers per class (default 3)
+      d_hidden, d_repr, d_proj: trunk sizing
+      n_datasets  : for the dataset adversary head
+    """
+
+    def __init__(
+        self,
+        variant: str = "pca",
+        n_pca: int = 50,
+        n_markers: int = 0,
+        d_hidden: int = 512,
+        d_repr: int = 256,
+        d_proj: int = 128,
+        n_classes: int = 10,
+        n_sub: int = 3,
+        n_datasets: int = 1,
+        dropout: float = 0.2,
+    ):
+        super().__init__()
+        assert variant in ("pca", "marker"), variant
+        if variant == "marker":
+            assert n_markers > 0, "PANDA-Marker requires n_markers>0"
+        self.variant = variant
+        self.n_pca = n_pca
+        self.n_markers = n_markers if variant == "marker" else 0
+        self.n_classes = n_classes
+        self.n_sub = n_sub
+        self.n_datasets = n_datasets
+
+        input_dim = n_pca + self.n_markers
+        self.input_dim = input_dim
+
+        self.trunk = nn.Sequential(
+            nn.Linear(input_dim, d_hidden), nn.LayerNorm(d_hidden), nn.GELU(), nn.Dropout(dropout),
+            nn.Linear(d_hidden, d_hidden), nn.LayerNorm(d_hidden), nn.GELU(), nn.Dropout(dropout),
+            nn.Linear(d_hidden, d_repr), nn.LayerNorm(d_repr), nn.GELU(),
+        )
+        self.projection = nn.Sequential(
+            nn.Linear(d_repr, d_repr), nn.GELU(),
+            nn.Linear(d_repr, d_proj),
+        )
+        self.classifier = nn.Sequential(nn.Linear(d_repr + 2, n_classes))
+        self.dom_adv = nn.Sequential(nn.Linear(d_repr, 128), nn.ReLU(), nn.Linear(128, n_datasets))
+        self.depth_adv = nn.Sequential(nn.Linear(d_repr, 64), nn.ReLU(), nn.Linear(64, 1))
+
+        # sub-center prototypes (K, n_sub, d_proj), L2-normalised per sub-center
+        self.register_buffer(
+            "prototypes",
+            F.normalize(torch.randn(n_classes, n_sub, d_proj), dim=-1),
+        )
+        # EMA momentum as a buffer so we can overwrite it in place
+        self.register_buffer("proto_ema", torch.tensor(0.99))
+
+    @torch.no_grad()
+    def update_prototypes(self, z_norm: torch.Tensor, y: torch.Tensor):
+        """ema update: assign each in-class cell to nearest sub-center, take the mean."""
+        ema = float(self.proto_ema.item())
+        for c in torch.unique(y):
+            mask = y == c
+            if not mask.any():
+                continue
+            zc = z_norm[mask]                              # (n_c, d_proj)
+            protos_c = self.prototypes[c]                  # (n_sub, d_proj)
+            sims = zc @ protos_c.T                         # (n_c, n_sub)
+            assign = sims.argmax(dim=1)                    # each cell -> nearest sub-center
+            for k in range(self.n_sub):
+                m2 = assign == k
+                if not m2.any():
+                    continue
+                new = F.normalize(zc[m2].mean(dim=0), dim=0)
+                self.prototypes[c, k] = F.normalize(
+                    ema * self.prototypes[c, k] + (1 - ema) * new, dim=0
+                )
+
+    @torch.no_grad()
+    def max_sub_cos(self, z_norm: torch.Tensor) -> torch.Tensor:
+        """(B, K) cos(z, best sub-center) per class."""
+        B = z_norm.size(0); K, n_sub, D = self.prototypes.shape
+        sims = torch.einsum("bd,ksd->bks", z_norm, self.prototypes)   # (B, K, n_sub)
+        return sims.max(dim=2).values                                 # (B, K)
+
+    def forward(
+        self,
+        x_pca: torch.Tensor,
+        aux: torch.Tensor,
+        x_markers: Optional[torch.Tensor] = None,
+        lam_dann: float = 0.0,
+    ) -> dict:
+        if self.variant == "marker":
+            assert x_markers is not None and x_markers.size(1) == self.n_markers
+            x = torch.cat([x_pca, x_markers], dim=1)
+        else:
+            x = x_pca
+
+        h = self.trunk(x)
+        z_raw = self.projection(h)
+        z = F.normalize(z_raw, dim=1)
+        logits = self.classifier(torch.cat([h, aux], dim=1))
+        h_rev = grad_reverse(h, lam_dann)
+        return {
+            "repr": h,
+            "z": z,
+            "logits": logits,
+            "dom": self.dom_adv(h_rev),
+            "depth": self.depth_adv(h_rev),
+        }
+
+
+# losses
+
+def supcon_loss(z: torch.Tensor, y: torch.Tensor, temperature: float = 0.1) -> torch.Tensor:
+    if z.size(0) < 2:
+        return z.new_zeros(())
+    sim = z @ z.T / temperature
+    sim_max, _ = sim.max(dim=1, keepdim=True)
+    sim = sim - sim_max.detach()
+    logits_mask = torch.ones_like(sim) - torch.eye(z.size(0), device=z.device)
+    exp_sim = torch.exp(sim) * logits_mask
+    log_prob = sim - torch.log(exp_sim.sum(dim=1, keepdim=True) + 1e-12)
+    labels_eq = (y.unsqueeze(0) == y.unsqueeze(1)).float() * logits_mask
+    denom = labels_eq.sum(dim=1).clamp_min(1.0)
+    per = -(labels_eq * log_prob).sum(dim=1) / denom
+    per = per * (labels_eq.sum(dim=1) > 0).float()
+    counts = torch.bincount(y, minlength=int(y.max().item()) + 1).float().clamp_min(1.0)
+    w = 1.0 / counts.sqrt()
+    return (per * w[y]).sum() / w[y].sum().clamp_min(1e-6)
+
+
+def vicreg_loss(z: torch.Tensor, sim_weight: float = 0.0, var_weight: float = 25.0,
+                cov_weight: float = 1.0) -> torch.Tensor:
+    zc = z - z.mean(dim=0, keepdim=True)
+    std = (zc.var(dim=0) + 1e-4).sqrt()
+    var_loss = F.relu(1.0 - std).mean()
+    N, D = zc.shape
+    cov = (zc.T @ zc) / (N - 1)
+    off = cov - torch.diag(torch.diagonal(cov))
+    cov_loss = off.pow(2).sum() / D
+    return var_weight * var_loss + cov_weight * cov_loss
+
+
+def hsic_biased(x: torch.Tensor, y: torch.Tensor,
+                sigma_x: float = 1.0, sigma_y: float = 1.0) -> torch.Tensor:
+    Nx = x.size(0)
+    if Nx < 2:
+        return x.new_zeros(())
+    K = torch.exp(-torch.cdist(x, x) ** 2 / (2 * sigma_x ** 2))
+    L = torch.exp(-torch.cdist(y, y) ** 2 / (2 * sigma_y ** 2))
+    H = torch.eye(Nx, device=x.device) - torch.ones(Nx, Nx, device=x.device) / Nx
+    return (K @ H @ L @ H).trace() / (Nx - 1) ** 2
+
+
+def subcenter_angular_infonce(
+    z: torch.Tensor,                    # (B, d_proj) L2-normalised
+    y: torch.Tensor,                    # (B,)
+    prototypes: torch.Tensor,           # (K, n_sub, d_proj)
+    margin: float = 0.15,               # angular margin in radians
+    temperature: float = 0.07,
+) -> torch.Tensor:
+    """arcface-style angular-margin loss over sub-center prototypes."""
+    B = z.size(0); K, n_sub, D = prototypes.shape
+    sims = torch.einsum("bd,ksd->bks", z, prototypes)          # (B, K, n_sub)
+    max_over_sub = sims.max(dim=2).values                       # (B, K)
+
+    # target class cosine, bump by angular margin, put back
+    target_cos = max_over_sub.gather(1, y.unsqueeze(1)).squeeze(1)   # (B,)
+    target_cos = target_cos.clamp(-1 + 1e-7, 1 - 1e-7)
+    theta = torch.acos(target_cos)
+    target_new_cos = torch.cos(theta + margin)
+
+    logits = max_over_sub.clone()
+    logits.scatter_(1, y.unsqueeze(1), target_new_cos.unsqueeze(1))
+    logits = logits / temperature
+    return F.cross_entropy(logits, y)
+
+
+def prototype_repulsion(prototypes: torch.Tensor, weight: float = 1.0) -> torch.Tensor:
+    """penalise inter-class prototype cosine so eff-dim doesn't collapse."""
+    K, n_sub, D = prototypes.shape
+    centroids = F.normalize(prototypes.mean(dim=1), dim=1)      # (K, D)
+    sim = centroids @ centroids.T                                # (K, K)
+    off = sim - torch.diag(torch.diagonal(sim))
+    return weight * off.pow(2).sum() / (K * (K - 1) + 1e-6)
+
+
+def prototype_infonce_legacy(z, y, prototypes, temperature=0.07):
+    """legacy single-prototype InfoNCE. kept for debugging + old checkpoints."""
+    if prototypes.dim() == 3:
+        prototypes = F.normalize(prototypes.mean(dim=1), dim=1)  # collapse sub-centers
+    logits = z @ prototypes.T / temperature
+    return F.cross_entropy(logits, y)

pyproject.toml

@@ -0,0 +1,84 @@
+[build-system]
+requires = ["setuptools>=68.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "panda-scrnaseq"
+version = "1.0.0"
+description = "PANDA: Pan-tissue Adversarial Normalized Domain-invariant Anchored MLP"
+readme = "README.md"
+license = {text = "MIT"}
+requires-python = ">=3.9"
+authors = [
+    {name = "Bryan Cheng"},
+]
+
+# Runtime dependencies. ML/inference versions are pinned (==) for
+# reproducibility; tooling / plotting libs use lower bounds (>=).
+# Pins reflect the versions used to produce the results in README.md.
+dependencies = [
+    # --- Deep learning core (PINNED) ---
+    "torch==2.6.0",
+    # --- Single-cell stack (PINNED for reproducibility) ---
+    "scanpy==1.11.5",
+    "anndata>=0.11.4",
+    "harmonypy==0.2.0",
+    # --- Scientific stack ---
+    "numpy>=1.24,<3.0",
+    "scipy>=1.10",
+    "scikit-learn>=1.2",
+    "pandas>=1.5",
+    # --- Visualization / UMAP ---
+    "matplotlib>=3.7",
+    "seaborn>=0.12",
+    "umap-learn>=0.5",
+    # --- I/O and utilities ---
+    "pyyaml>=6.0",
+    "tqdm>=4.65",
+    # --- scanpy plugins (clustering / neighbors / DE) ---
+    "leidenalg>=0.10",
+    "pynndescent>=0.5",
+    "scikit-misc>=0.5",
+]
+# Removed 2026-08 (never imported anywhere in scripts/ or panda/):
+# transformers, peft, scvi-tools, einops, gdown, GEOparse — PRISM-era leftovers.
+
+[project.optional-dependencies]
+# Interactive plotting (optional alternative to matplotlib).
+viz = [
+    "plotly>=5.0",
+]
+# Development tooling. Kept in sync with requirements-dev.txt.
+dev = [
+    "pytest>=7.0",
+    "pytest-cov>=4.0",
+    "pytest-xdist>=3.0",
+    "ruff>=0.1.0",
+    "mypy>=1.0",
+]
+
+[project.scripts]
+# entry-point removed with legacy cli
+
+[tool.setuptools.packages.find]
+include = ["panda*"]
+
+[tool.setuptools.package-data]
+panda = ["markers.yaml"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+python_files = ["test_*.py"]
+python_classes = ["Test*"]
+python_functions = ["test_*"]
+addopts = "-v --tb=short"
+
+[tool.ruff]
+line-length = 120
+target-version = "py39"
+
+[tool.mypy]
+python_version = "3.9"
+warn_return_any = false
+warn_unused_configs = true
+ignore_missing_imports = true

scripts/analysis/100_primary_eden_discovery.py

@@ -1,205 +1,208 @@
-"""primary EDEN discovery on Dingwall: PANDA-v3 dermal-fibro subset, Leiden res=1.5, wilcoxon markers + Fisher cKO depletion + module scoring."""
-from pathlib import Path
-import warnings, json, sys, pickle, numpy as np, pandas as pd, anndata as ad, scanpy as sc, scipy.sparse as sp, torch, torch.nn.functional as F
-from scipy.stats import fisher_exact, mannwhitneyu
-warnings.filterwarnings("ignore"); sc.settings.verbosity = 0
-sys.path.insert(0, "/home/bcheng/PRISM")
-from panda import PANDAEncoder
-
-ROOT = Path("/home/bcheng/PRISM")
-DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
-
-CKO_GSMS = {"GSM6833482", "GSM6833483"}  # CORRECTED: 480/481 are rttaControl (WT), not cKO
-WT_GSMS  = {"GSM6833478", "GSM6833479", "GSM6833480", "GSM6833481"}  # CORRECTED: 4 Cre-neg controls per GEO metadata
-
-# secondary EDEN definition per Dingwall 2024
-SECONDARY_EDEN_PANEL = ["S100a4", "Tnc", "Pdgfra"]
-
-# En1-responsive eccrine program (from restored 57_pathway_analysis.py, En1 removed)
-SWEAT_GLAND_PANEL_ENSMINUSEN1 = ["Foxi3", "Foxa1", "Krt8", "Krt18", "Krt19", "Muc5b", "Aqp5"]
-# Eda pathway
-EDA_PATHWAY_PANEL = ["Eda", "Edar", "Edaradd", "Nfkb1", "Nfkb2", "Rela"]
-
-
-def load_dingwall_with_v3_predictions():
-    print("[eden] loading Dingwall raw + v3 predictions", flush=True)
-    raw = ad.read_h5ad(ROOT / "data/raw/GSE220977_combined.h5ad")
-    pred = pd.read_csv(ROOT / "discovery/pan_skin/marker/dingwall_predictions.csv")
-    pred_map = dict(zip(pred["cell_id"].astype(str), pred["pred_label"]))
-    raw.obs["pred_label"] = np.array([pred_map.get(c, "unknown") for c in raw.obs_names.astype(str)])
-    raw.obs["genotype"] = np.where(raw.obs["sample"].astype(str).isin(list(CKO_GSMS)), "En1-cKO",
-                          np.where(raw.obs["sample"].astype(str).isin(list(WT_GSMS)), "WT", "other"))
-    labeled = raw.obs["genotype"].isin(["WT", "En1-cKO"]).values
-    raw = raw[labeled].copy()
-
-    dermal_mask = np.isin(raw.obs["pred_label"], ["fibroblast-papillary", "fibroblast-reticular"])
-    dermal = raw[dermal_mask].copy()
-    print(f"[eden] {dermal.n_obs} dermal-fibroblast cells for sub-clustering", flush=True)
-    return dermal
-
-
-def subcluster_dermal(dermal, resolution=1.5):
-    print(f"[eden] preprocessing + PCA (Leiden resolution={resolution})", flush=True)
-    sc.pp.normalize_total(dermal, target_sum=1e4); sc.pp.log1p(dermal)
-    sc.pp.highly_variable_genes(dermal, n_top_genes=3000, flavor="seurat_v3",
-                                inplace=True, batch_key=None)
-    dermal_hvg = dermal[:, dermal.var["highly_variable"]].copy() if "highly_variable" in dermal.var else dermal
-    sc.pp.scale(dermal_hvg, max_value=10)
-    sc.tl.pca(dermal_hvg, n_comps=30, random_state=0)
-    sc.pp.neighbors(dermal_hvg, n_neighbors=20, use_rep="X_pca")
-    sc.tl.leiden(dermal_hvg, resolution=resolution, random_state=0)
-    dermal.obs["leiden"] = dermal_hvg.obs["leiden"].astype(str)
-    print(f"[eden] {dermal.obs['leiden'].nunique()} sub-clusters found", flush=True)
-    return dermal
-
-
-def score_modules(dermal):
-    for name, genes in [("secondary_eden", SECONDARY_EDEN_PANEL),
-                         ("sweat_gland", SWEAT_GLAND_PANEL_ENSMINUSEN1),
-                         ("eda_pathway", EDA_PATHWAY_PANEL)]:
-        present = [g for g in genes if g in dermal.var_names]
-        if not present:
-            dermal.obs[f"score_{name}"] = 0.0
-            continue
-        sc.tl.score_genes(dermal, gene_list=present, score_name=f"score_{name}",
-                          random_state=0, use_raw=False)
-    return dermal
-
-
-def per_subcluster_analysis(dermal):
-    n_wt_tot = int((dermal.obs["genotype"] == "WT").sum())
-    n_cko_tot = int((dermal.obs["genotype"] == "En1-cKO").sum())
-    baseline_cko_frac = n_cko_tot / (n_wt_tot + n_cko_tot)
-    print(f"[eden] baseline: WT={n_wt_tot} cKO={n_cko_tot} (baseline cKO frac = {baseline_cko_frac:.3f})", flush=True)
-
-    sc.tl.rank_genes_groups(dermal, "leiden", method="wilcoxon", n_genes=30, use_raw=False)
-
-    rows = []
-    for cls in sorted(dermal.obs["leiden"].unique(), key=int):
-        sub = dermal[dermal.obs["leiden"] == cls]
-        n_wt = int((sub.obs["genotype"] == "WT").sum())
-        n_cko = int((sub.obs["genotype"] == "En1-cKO").sum())
-        if n_wt + n_cko < 20:
-            continue
-        cko_frac = n_cko / (n_wt + n_cko)
-
-        # fisher 2x2: (n_wt_in, n_wt_out) vs (n_cko_in, n_cko_out) — cluster depletion in cKO
-        n_wt_elsewhere = n_wt_tot - n_wt
-        n_cko_elsewhere = n_cko_tot - n_cko
-        odds, p_fisher = fisher_exact([[n_wt, n_wt_elsewhere], [n_cko, n_cko_elsewhere]],
-                                      alternative="two-sided")
-        depletion_direction = "cKO-depleted" if cko_frac < baseline_cko_frac else "cKO-enriched"
-
-        s2eden_mean_wt = float(sub[sub.obs["genotype"] == "WT"].obs["score_secondary_eden"].mean()) if n_wt > 0 else 0.0
-        s2eden_mean_cko = float(sub[sub.obs["genotype"] == "En1-cKO"].obs["score_secondary_eden"].mean()) if n_cko > 0 else 0.0
-        sg_mean_wt = float(sub[sub.obs["genotype"] == "WT"].obs["score_sweat_gland"].mean()) if n_wt > 0 else 0.0
-        sg_mean_cko = float(sub[sub.obs["genotype"] == "En1-cKO"].obs["score_sweat_gland"].mean()) if n_cko > 0 else 0.0
-        eda_mean_wt = float(sub[sub.obs["genotype"] == "WT"].obs["score_eda_pathway"].mean()) if n_wt > 0 else 0.0
-        eda_mean_cko = float(sub[sub.obs["genotype"] == "En1-cKO"].obs["score_eda_pathway"].mean()) if n_cko > 0 else 0.0
-
-        genes_list = list(dermal.uns["rank_genes_groups"]["names"][cls][:10])
-        lfc_list = list(dermal.uns["rank_genes_groups"]["logfoldchanges"][cls][:10])
-        top_markers = ", ".join([f"{g}({lfc:+.1f})" for g, lfc in zip(genes_list, lfc_list)])
-
-        rows.append({
-            "leiden_cluster": cls,
-            "n_cells": n_wt + n_cko,
-            "n_WT": n_wt, "n_cKO": n_cko,
-            "cko_frac": cko_frac,
-            "baseline_cko_frac": baseline_cko_frac,
-            "depletion_direction": depletion_direction,
-            "fisher_p_two_sided": p_fisher,
-            "odds_ratio": odds,
-            "score_secondary_eden_WT_mean": s2eden_mean_wt,
-            "score_secondary_eden_cKO_mean": s2eden_mean_cko,
-            "score_sweat_gland_WT_mean": sg_mean_wt,
-            "score_sweat_gland_cKO_mean": sg_mean_cko,
-            "score_eda_pathway_WT_mean": eda_mean_wt,
-            "score_eda_pathway_cKO_mean": eda_mean_cko,
-            "top_wilcoxon_markers": top_markers,
-        })
-    return pd.DataFrame(rows), baseline_cko_frac
-
-
-def call_primary_and_secondary(df, baseline_cko_frac):
-    # secondary EDEN: highest score_secondary_eden_WT_mean AND cKO-depleted (Fisher p<0.05)
-    df_wt_ordered = df.sort_values("score_secondary_eden_WT_mean", ascending=False)
-    secondary_candidates = df_wt_ordered[
-        (df_wt_ordered["depletion_direction"] == "cKO-depleted") &
-        (df_wt_ordered["fisher_p_two_sided"] < 0.05)
-    ]
-    secondary = secondary_candidates.iloc[0]["leiden_cluster"] if len(secondary_candidates) > 0 else None
-
-    # primary EDEN: cKO-depleted + LOW secondary_eden (S100a4-neg) + HIGH Eda_pathway (En1-responsive)
-    df_ranked = df.copy()
-    df_ranked["depletion_score"] = -np.log10(df_ranked["fisher_p_two_sided"].clip(lower=1e-300)) * \
-                                   (df_ranked["cko_frac"] < baseline_cko_frac).astype(int)
-    primary_score = df_ranked["depletion_score"] * \
-                    (1.0 / (df_ranked["score_secondary_eden_WT_mean"].abs() + 0.01)) * \
-                    (df_ranked["score_eda_pathway_WT_mean"] + 0.1)
-    df_ranked["primary_eden_composite_score"] = primary_score
-    df_ranked = df_ranked.sort_values("primary_eden_composite_score", ascending=False)
-    primary_candidates = df_ranked[
-        (df_ranked["depletion_direction"] == "cKO-depleted") &
-        (df_ranked["fisher_p_two_sided"] < 0.05) &
-        (df_ranked["leiden_cluster"] != secondary)
-    ].head(3)
-    return secondary, primary_candidates, df_ranked
-
-
-def main():
-    dermal = load_dingwall_with_v3_predictions()
-    dermal = subcluster_dermal(dermal, resolution=1.5)
-    dermal = score_modules(dermal)
-    df, baseline_cko = per_subcluster_analysis(dermal)
-    secondary, primary_cands, df_ranked = call_primary_and_secondary(df, baseline_cko)
-
-    out = ROOT / "discovery/pan_skin/marker"
-    out.mkdir(parents=True, exist_ok=True)
-    df_ranked.to_csv(out / "100_primary_eden_discovery.csv", index=False)
-
-    summary = {
-        "target": "Dingwall_GSE220977",
-        "hypothesis": "Primary EDEN precedes Secondary EDEN (S100a4+/Tnc+ cluster 20/Derm10) in dermal lineage",
-        "method": "PANDA-v3 predicts dermal-fibroblast compartment; Leiden sub-clustering "
-                  "(resolution=1.5) resolves substructure; Wilcoxon markers + Fisher-exact "
-                  "cKO enrichment + module scoring (Secondary_EDEN, Sweat_gland, Eda_pathway) "
-                  "rank sub-clusters for Primary EDEN candidacy",
-        "baseline_cko_frac": float(baseline_cko),
-        "n_subclusters": int(len(df)),
-        "secondary_eden_call": {
-            "leiden_cluster": str(secondary),
-            "criteria": "highest S100a4+Tnc+Pdgfra score AND Fisher cKO-depleted p<0.05",
-            "row": df[df["leiden_cluster"] == secondary].iloc[0].to_dict() if secondary else None,
-        },
-        "primary_eden_candidates_top3": primary_cands[[
-            "leiden_cluster", "n_cells", "n_WT", "n_cKO", "cko_frac",
-            "fisher_p_two_sided", "score_secondary_eden_WT_mean",
-            "score_sweat_gland_WT_mean", "score_eda_pathway_WT_mean",
-            "top_wilcoxon_markers", "primary_eden_composite_score",
-        ]].to_dict("records") if len(primary_cands) > 0 else [],
-    }
-    (out / "100_primary_eden_summary.json").write_text(json.dumps(summary, indent=2, default=str))
-
-    print(f"\n[eden] wrote {out}/100_primary_eden_*", flush=True)
-    print(f"\n=== SECONDARY EDEN CALL ===", flush=True)
-    print(f"  leiden cluster: {secondary}", flush=True)
-    if secondary:
-        row = df[df["leiden_cluster"] == secondary].iloc[0]
-        print(f"  n={row['n_cells']} (WT {row['n_WT']} / cKO {row['n_cKO']}), "
-              f"cko_frac={row['cko_frac']:.3f} vs baseline {baseline_cko:.3f}", flush=True)
-        print(f"  Fisher p={row['fisher_p_two_sided']:.2e}, "
-              f"score_secondary_eden WT={row['score_secondary_eden_WT_mean']:.3f}", flush=True)
-
-    print(f"\n=== PRIMARY EDEN CANDIDATES (top 3) ===", flush=True)
-    for _, row in primary_cands.iterrows():
-        print(f"  leiden {row['leiden_cluster']} n={row['n_cells']} (WT {row['n_WT']} / cKO {row['n_cKO']}), "
-              f"cko_frac={row['cko_frac']:.3f}, Fisher p={row['fisher_p_two_sided']:.2e}", flush=True)
-        print(f"    S2EDEN_WT={row['score_secondary_eden_WT_mean']:.3f}, "
-              f"Sweat_WT={row['score_sweat_gland_WT_mean']:.3f}, "
-              f"Eda_WT={row['score_eda_pathway_WT_mean']:.3f}", flush=True)
-        print(f"    top markers: {row['top_wilcoxon_markers']}", flush=True)
-
-
-if __name__ == "__main__":
-    main()
+"""primary EDEN discovery on Dingwall: PANDA-v3 dermal-fibro subset, Leiden res=1.5, wilcoxon markers + Fisher cKO depletion + module scoring."""
+from pathlib import Path
+import warnings, json, sys, pickle, numpy as np, pandas as pd, anndata as ad, scanpy as sc, scipy.sparse as sp, torch, torch.nn.functional as F
+from scipy.stats import fisher_exact, mannwhitneyu
+warnings.filterwarnings("ignore"); sc.settings.verbosity = 0
+import os as _os
+from pathlib import Path as _Path
+PANDA_ROOT = _Path(_os.environ.get("PANDA_ROOT", str(_Path(__file__).resolve().parents[2])))
+sys.path.insert(0, str(PANDA_ROOT))
+from panda import PANDAEncoder
+
+ROOT = Path(str(PANDA_ROOT))
+DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+CKO_GSMS = {"GSM6833482", "GSM6833483"}  # CORRECTED: 480/481 are rttaControl (WT), not cKO
+WT_GSMS  = {"GSM6833478", "GSM6833479", "GSM6833480", "GSM6833481"}  # CORRECTED: 4 Cre-neg controls per GEO metadata
+
+# secondary EDEN definition per Dingwall 2024
+SECONDARY_EDEN_PANEL = ["S100a4", "Tnc", "Pdgfra"]
+
+# En1-responsive eccrine program (from restored 57_pathway_analysis.py, En1 removed)
+SWEAT_GLAND_PANEL_ENSMINUSEN1 = ["Foxi3", "Foxa1", "Krt8", "Krt18", "Krt19", "Muc5b", "Aqp5"]
+# Eda pathway
+EDA_PATHWAY_PANEL = ["Eda", "Edar", "Edaradd", "Nfkb1", "Nfkb2", "Rela"]
+
+
+def load_dingwall_with_v3_predictions():
+    print("[eden] loading Dingwall raw + v3 predictions", flush=True)
+    raw = ad.read_h5ad(ROOT / "data/raw/GSE220977_combined.h5ad")
+    pred = pd.read_csv(ROOT / "discovery/pan_skin/marker/dingwall_predictions.csv")
+    pred_map = dict(zip(pred["cell_id"].astype(str), pred["pred_label"]))
+    raw.obs["pred_label"] = np.array([pred_map.get(c, "unknown") for c in raw.obs_names.astype(str)])
+    raw.obs["genotype"] = np.where(raw.obs["sample"].astype(str).isin(list(CKO_GSMS)), "En1-cKO",
+                          np.where(raw.obs["sample"].astype(str).isin(list(WT_GSMS)), "WT", "other"))
+    labeled = raw.obs["genotype"].isin(["WT", "En1-cKO"]).values
+    raw = raw[labeled].copy()
+
+    dermal_mask = np.isin(raw.obs["pred_label"], ["fibroblast-papillary", "fibroblast-reticular"])
+    dermal = raw[dermal_mask].copy()
+    print(f"[eden] {dermal.n_obs} dermal-fibroblast cells for sub-clustering", flush=True)
+    return dermal
+
+
+def subcluster_dermal(dermal, resolution=1.5):
+    print(f"[eden] preprocessing + PCA (Leiden resolution={resolution})", flush=True)
+    sc.pp.normalize_total(dermal, target_sum=1e4); sc.pp.log1p(dermal)
+    sc.pp.highly_variable_genes(dermal, n_top_genes=3000, flavor="seurat_v3",
+                                inplace=True, batch_key=None)
+    dermal_hvg = dermal[:, dermal.var["highly_variable"]].copy() if "highly_variable" in dermal.var else dermal
+    sc.pp.scale(dermal_hvg, max_value=10)
+    sc.tl.pca(dermal_hvg, n_comps=30, random_state=0)
+    sc.pp.neighbors(dermal_hvg, n_neighbors=20, use_rep="X_pca")
+    sc.tl.leiden(dermal_hvg, resolution=resolution, random_state=0)
+    dermal.obs["leiden"] = dermal_hvg.obs["leiden"].astype(str)
+    print(f"[eden] {dermal.obs['leiden'].nunique()} sub-clusters found", flush=True)
+    return dermal
+
+
+def score_modules(dermal):
+    for name, genes in [("secondary_eden", SECONDARY_EDEN_PANEL),
+                         ("sweat_gland", SWEAT_GLAND_PANEL_ENSMINUSEN1),
+                         ("eda_pathway", EDA_PATHWAY_PANEL)]:
+        present = [g for g in genes if g in dermal.var_names]
+        if not present:
+            dermal.obs[f"score_{name}"] = 0.0
+            continue
+        sc.tl.score_genes(dermal, gene_list=present, score_name=f"score_{name}",
+                          random_state=0, use_raw=False)
+    return dermal
+
+
+def per_subcluster_analysis(dermal):
+    n_wt_tot = int((dermal.obs["genotype"] == "WT").sum())
+    n_cko_tot = int((dermal.obs["genotype"] == "En1-cKO").sum())
+    baseline_cko_frac = n_cko_tot / (n_wt_tot + n_cko_tot)
+    print(f"[eden] baseline: WT={n_wt_tot} cKO={n_cko_tot} (baseline cKO frac = {baseline_cko_frac:.3f})", flush=True)
+
+    sc.tl.rank_genes_groups(dermal, "leiden", method="wilcoxon", n_genes=30, use_raw=False)
+
+    rows = []
+    for cls in sorted(dermal.obs["leiden"].unique(), key=int):
+        sub = dermal[dermal.obs["leiden"] == cls]
+        n_wt = int((sub.obs["genotype"] == "WT").sum())
+        n_cko = int((sub.obs["genotype"] == "En1-cKO").sum())
+        if n_wt + n_cko < 20:
+            continue
+        cko_frac = n_cko / (n_wt + n_cko)
+
+        # fisher 2x2: (n_wt_in, n_wt_out) vs (n_cko_in, n_cko_out) — cluster depletion in cKO
+        n_wt_elsewhere = n_wt_tot - n_wt
+        n_cko_elsewhere = n_cko_tot - n_cko
+        odds, p_fisher = fisher_exact([[n_wt, n_wt_elsewhere], [n_cko, n_cko_elsewhere]],
+                                      alternative="two-sided")
+        depletion_direction = "cKO-depleted" if cko_frac < baseline_cko_frac else "cKO-enriched"
+
+        s2eden_mean_wt = float(sub[sub.obs["genotype"] == "WT"].obs["score_secondary_eden"].mean()) if n_wt > 0 else 0.0
+        s2eden_mean_cko = float(sub[sub.obs["genotype"] == "En1-cKO"].obs["score_secondary_eden"].mean()) if n_cko > 0 else 0.0
+        sg_mean_wt = float(sub[sub.obs["genotype"] == "WT"].obs["score_sweat_gland"].mean()) if n_wt > 0 else 0.0
+        sg_mean_cko = float(sub[sub.obs["genotype"] == "En1-cKO"].obs["score_sweat_gland"].mean()) if n_cko > 0 else 0.0
+        eda_mean_wt = float(sub[sub.obs["genotype"] == "WT"].obs["score_eda_pathway"].mean()) if n_wt > 0 else 0.0
+        eda_mean_cko = float(sub[sub.obs["genotype"] == "En1-cKO"].obs["score_eda_pathway"].mean()) if n_cko > 0 else 0.0
+
+        genes_list = list(dermal.uns["rank_genes_groups"]["names"][cls][:10])
+        lfc_list = list(dermal.uns["rank_genes_groups"]["logfoldchanges"][cls][:10])
+        top_markers = ", ".join([f"{g}({lfc:+.1f})" for g, lfc in zip(genes_list, lfc_list)])
+
+        rows.append({
+            "leiden_cluster": cls,
+            "n_cells": n_wt + n_cko,
+            "n_WT": n_wt, "n_cKO": n_cko,
+            "cko_frac": cko_frac,
+            "baseline_cko_frac": baseline_cko_frac,
+            "depletion_direction": depletion_direction,
+            "fisher_p_two_sided": p_fisher,
+            "odds_ratio": odds,
+            "score_secondary_eden_WT_mean": s2eden_mean_wt,
+            "score_secondary_eden_cKO_mean": s2eden_mean_cko,
+            "score_sweat_gland_WT_mean": sg_mean_wt,
+            "score_sweat_gland_cKO_mean": sg_mean_cko,
+            "score_eda_pathway_WT_mean": eda_mean_wt,
+            "score_eda_pathway_cKO_mean": eda_mean_cko,
+            "top_wilcoxon_markers": top_markers,
+        })
+    return pd.DataFrame(rows), baseline_cko_frac
+
+
+def call_primary_and_secondary(df, baseline_cko_frac):
+    # secondary EDEN: highest score_secondary_eden_WT_mean AND cKO-depleted (Fisher p<0.05)
+    df_wt_ordered = df.sort_values("score_secondary_eden_WT_mean", ascending=False)
+    secondary_candidates = df_wt_ordered[
+        (df_wt_ordered["depletion_direction"] == "cKO-depleted") &
+        (df_wt_ordered["fisher_p_two_sided"] < 0.05)
+    ]
+    secondary = secondary_candidates.iloc[0]["leiden_cluster"] if len(secondary_candidates) > 0 else None
+
+    # primary EDEN: cKO-depleted + LOW secondary_eden (S100a4-neg) + HIGH Eda_pathway (En1-responsive)
+    df_ranked = df.copy()
+    df_ranked["depletion_score"] = -np.log10(df_ranked["fisher_p_two_sided"].clip(lower=1e-300)) * \
+                                   (df_ranked["cko_frac"] < baseline_cko_frac).astype(int)
+    primary_score = df_ranked["depletion_score"] * \
+                    (1.0 / (df_ranked["score_secondary_eden_WT_mean"].abs() + 0.01)) * \
+                    (df_ranked["score_eda_pathway_WT_mean"] + 0.1)
+    df_ranked["primary_eden_composite_score"] = primary_score
+    df_ranked = df_ranked.sort_values("primary_eden_composite_score", ascending=False)
+    primary_candidates = df_ranked[
+        (df_ranked["depletion_direction"] == "cKO-depleted") &
+        (df_ranked["fisher_p_two_sided"] < 0.05) &
+        (df_ranked["leiden_cluster"] != secondary)
+    ].head(3)
+    return secondary, primary_candidates, df_ranked
+
+
+def main():
+    dermal = load_dingwall_with_v3_predictions()
+    dermal = subcluster_dermal(dermal, resolution=1.5)
+    dermal = score_modules(dermal)
+    df, baseline_cko = per_subcluster_analysis(dermal)
+    secondary, primary_cands, df_ranked = call_primary_and_secondary(df, baseline_cko)
+
+    out = ROOT / "discovery/pan_skin/marker"
+    out.mkdir(parents=True, exist_ok=True)
+    df_ranked.to_csv(out / "100_primary_eden_discovery.csv", index=False)
+
+    summary = {
+        "target": "Dingwall_GSE220977",
+        "hypothesis": "Primary EDEN precedes Secondary EDEN (S100a4+/Tnc+ cluster 20/Derm10) in dermal lineage",
+        "method": "PANDA-v3 predicts dermal-fibroblast compartment; Leiden sub-clustering "
+                  "(resolution=1.5) resolves substructure; Wilcoxon markers + Fisher-exact "
+                  "cKO enrichment + module scoring (Secondary_EDEN, Sweat_gland, Eda_pathway) "
+                  "rank sub-clusters for Primary EDEN candidacy",
+        "baseline_cko_frac": float(baseline_cko),
+        "n_subclusters": int(len(df)),
+        "secondary_eden_call": {
+            "leiden_cluster": str(secondary),
+            "criteria": "highest S100a4+Tnc+Pdgfra score AND Fisher cKO-depleted p<0.05",
+            "row": df[df["leiden_cluster"] == secondary].iloc[0].to_dict() if secondary else None,
+        },
+        "primary_eden_candidates_top3": primary_cands[[
+            "leiden_cluster", "n_cells", "n_WT", "n_cKO", "cko_frac",
+            "fisher_p_two_sided", "score_secondary_eden_WT_mean",
+            "score_sweat_gland_WT_mean", "score_eda_pathway_WT_mean",
+            "top_wilcoxon_markers", "primary_eden_composite_score",
+        ]].to_dict("records") if len(primary_cands) > 0 else [],
+    }
+    (out / "100_primary_eden_summary.json").write_text(json.dumps(summary, indent=2, default=str))
+
+    print(f"\n[eden] wrote {out}/100_primary_eden_*", flush=True)
+    print(f"\n=== SECONDARY EDEN CALL ===", flush=True)
+    print(f"  leiden cluster: {secondary}", flush=True)
+    if secondary:
+        row = df[df["leiden_cluster"] == secondary].iloc[0]
+        print(f"  n={row['n_cells']} (WT {row['n_WT']} / cKO {row['n_cKO']}), "
+              f"cko_frac={row['cko_frac']:.3f} vs baseline {baseline_cko:.3f}", flush=True)
+        print(f"  Fisher p={row['fisher_p_two_sided']:.2e}, "
+              f"score_secondary_eden WT={row['score_secondary_eden_WT_mean']:.3f}", flush=True)
+
+    print(f"\n=== PRIMARY EDEN CANDIDATES (top 3) ===", flush=True)
+    for _, row in primary_cands.iterrows():
+        print(f"  leiden {row['leiden_cluster']} n={row['n_cells']} (WT {row['n_WT']} / cKO {row['n_cKO']}), "
+              f"cko_frac={row['cko_frac']:.3f}, Fisher p={row['fisher_p_two_sided']:.2e}", flush=True)
+        print(f"    S2EDEN_WT={row['score_secondary_eden_WT_mean']:.3f}, "
+              f"Sweat_WT={row['score_sweat_gland_WT_mean']:.3f}, "
+              f"Eda_WT={row['score_eda_pathway_WT_mean']:.3f}", flush=True)
+        print(f"    top markers: {row['top_wilcoxon_markers']}", flush=True)
+
+
+if __name__ == "__main__":
+    main()

scripts/analysis/101_primary_eden_derm_scoring.py

@@ -1,159 +1,162 @@
-"""primary + secondary EDEN on Dingwall via score_genes against dingwall's own Derm0-11 markers (Data S1C top-30); argmax identity + Fisher cKO depletion."""
-from pathlib import Path
-import warnings, json, sys, numpy as np, pandas as pd, anndata as ad, scanpy as sc, scipy.sparse as sp
-from scipy.stats import fisher_exact
-warnings.filterwarnings("ignore"); sc.settings.verbosity = 0
-
-ROOT = Path("/home/bcheng/PRISM")
-CKO_GSMS = {"GSM6833482", "GSM6833483"}  # CORRECTED: 480/481 are rttaControl (WT), not cKO
-WT_GSMS  = {"GSM6833478", "GSM6833479", "GSM6833480", "GSM6833481"}  # CORRECTED: 4 Cre-neg controls per GEO metadata
-DERM_MARKERS = ROOT / "data/external_labels/dingwall_supp/biorxiv_media-3.xlsx"
-TOP_N = 30  # top N markers per Derm cluster for scoring
-
-# EDEN identity map based on Data S1C + Data S2 CellChat
-EDEN_IDENTITY = {
-    10: "Secondary_EDEN",
-    2: "Primary_EDEN_candidate_1",
-    9: "Primary_EDEN_candidate_2",
-    3: "EDEN_signalling",
-    6: "EDEN_signalling",
-}
-
-
-def load_derm_marker_panels():
-    df = pd.read_excel(DERM_MARKERS)
-    df = df.sort_values(["cluster", "avg_log2FC"], ascending=[True, False])
-    panels = {}
-    for cl in sorted(df["cluster"].unique()):
-        genes = df[df["cluster"] == cl].head(TOP_N)["gene"].tolist()
-        panels[int(cl)] = genes
-    return panels
-
-
-def load_dingwall_v3_dermal():
-    raw = ad.read_h5ad(ROOT / "data/raw/GSE220977_combined.h5ad")
-    pred = pd.read_csv(ROOT / "discovery/pan_skin/marker/dingwall_predictions.csv")
-    pred_map = dict(zip(pred["cell_id"].astype(str), pred["pred_label"]))
-    raw.obs["pred_label"] = np.array([pred_map.get(c, "unknown") for c in raw.obs_names.astype(str)])
-    raw.obs["genotype"] = np.where(raw.obs["sample"].astype(str).isin(list(CKO_GSMS)), "En1-cKO",
-                          np.where(raw.obs["sample"].astype(str).isin(list(WT_GSMS)), "WT", "other"))
-    labeled = raw.obs["genotype"].isin(["WT", "En1-cKO"]).values
-    raw = raw[labeled].copy()
-    dermal_mask = np.isin(raw.obs["pred_label"], ["fibroblast-papillary", "fibroblast-reticular"])
-    dermal = raw[dermal_mask].copy()
-    return dermal
-
-
-def score_derm_identities(a, panels):
-    for cl, genes in panels.items():
-        present = [g for g in genes if g in a.var_names]
-        if not present:
-            a.obs[f"derm{cl}_score"] = 0.0
-            continue
-        sc.tl.score_genes(a, gene_list=present, score_name=f"derm{cl}_score",
-                          random_state=0, use_raw=False)
-    return a
-
-
-def main():
-    print("[eden] loading Data S1C marker panels", flush=True)
-    panels = load_derm_marker_panels()
-    print(f"[eden] loaded {len(panels)} Derm panels", flush=True)
-
-    print("[eden] loading Dingwall + PANDA-v3 dermal predictions", flush=True)
-    dermal = load_dingwall_v3_dermal()
-    print(f"[eden] {dermal.n_obs} dermal-fibroblast cells", flush=True)
-
-    print("[eden] normalise + log1p", flush=True)
-    sc.pp.normalize_total(dermal, target_sum=1e4); sc.pp.log1p(dermal)
-
-    print("[eden] scoring cells on all 12 Derm identity panels", flush=True)
-    dermal = score_derm_identities(dermal, panels)
-
-    score_cols = [f"derm{cl}_score" for cl in sorted(panels.keys())]
-    score_mat = dermal.obs[score_cols].values
-    argmax = np.argmax(score_mat, axis=1)
-    derm_ids = [int(score_cols[i].replace("derm", "").replace("_score", "")) for i in argmax]
-    dermal.obs["derm_identity"] = derm_ids
-    dermal.obs["max_derm_score"] = score_mat.max(axis=1)
-
-    n_wt_tot = int((dermal.obs["genotype"] == "WT").sum())
-    n_cko_tot = int((dermal.obs["genotype"] == "En1-cKO").sum())
-    baseline_cko_frac = n_cko_tot / (n_wt_tot + n_cko_tot)
-    print(f"[eden] baseline WT={n_wt_tot} cKO={n_cko_tot} (baseline cKO={baseline_cko_frac:.3f})", flush=True)
-
-    rows = []
-    for cl in sorted(panels.keys()):
-        sub = dermal[dermal.obs["derm_identity"] == cl]
-        n_wt = int((sub.obs["genotype"] == "WT").sum())
-        n_cko = int((sub.obs["genotype"] == "En1-cKO").sum())
-        if n_wt + n_cko == 0:
-            continue
-        cko_frac = n_cko / (n_wt + n_cko) if (n_wt + n_cko) > 0 else 0
-        n_wt_else = n_wt_tot - n_wt
-        n_cko_else = n_cko_tot - n_cko
-        try:
-            odds, p_fisher = fisher_exact([[n_wt, n_wt_else], [n_cko, n_cko_else]],
-                                          alternative="two-sided")
-        except ValueError:
-            odds, p_fisher = 1.0, 1.0
-        rows.append({
-            "derm_id": cl,
-            "identity": EDEN_IDENTITY.get(cl, "other"),
-            "n_cells": n_wt + n_cko,
-            "n_WT": n_wt, "n_cKO": n_cko,
-            "cko_frac": cko_frac,
-            "baseline_cko_frac": baseline_cko_frac,
-            "cko_delta": cko_frac - baseline_cko_frac,
-            "fisher_p": float(p_fisher),
-            "odds_ratio": float(odds) if not np.isnan(odds) else None,
-            "depletion_direction": "cKO-depleted" if cko_frac < baseline_cko_frac
-                                   else "cKO-enriched",
-            "top10_markers_dingwall": ", ".join(panels[cl][:10]),
-        })
-    df = pd.DataFrame(rows).sort_values("cko_delta")
-
-    out = ROOT / "discovery/pan_skin/marker"
-    out.mkdir(parents=True, exist_ok=True)
-    df.to_csv(out / "101_derm_subcluster_scores.csv", index=False)
-
-    secondary_row = df[df["derm_id"] == 10].iloc[0].to_dict() if 10 in df["derm_id"].values else None
-    primary_1_row = df[df["derm_id"] == 2].iloc[0].to_dict() if 2 in df["derm_id"].values else None
-    primary_2_row = df[df["derm_id"] == 9].iloc[0].to_dict() if 9 in df["derm_id"].values else None
-
-    summary = {
-        "target": "Dingwall_GSE220977",
-        "method": "Score PANDA-v3 dermal-fibroblast predictions on Dingwall's own Derm0-11 "
-                  "marker panels (Data S1C top-30 genes each); argmax identity per cell; "
-                  "Fisher-exact cKO depletion per Derm identity",
-        "baseline_cko_frac": baseline_cko_frac,
-        "n_dermal_cells_total": int(dermal.n_obs),
-        "secondary_eden_Derm10": secondary_row,
-        "primary_eden_Derm2": primary_1_row,
-        "primary_eden_Derm9": primary_2_row,
-        "all_derm_summary": df.to_dict("records"),
-    }
-    (out / "101_derm_identity_summary.json").write_text(json.dumps(summary, indent=2, default=str))
-
-    print(f"\n[eden] wrote {out}/101_derm_*", flush=True)
-    print(f"\n=== SECONDARY EDEN (Derm10) ===", flush=True)
-    if secondary_row:
-        print(f"  n={secondary_row['n_cells']} (WT {secondary_row['n_WT']} / cKO {secondary_row['n_cKO']}), "
-              f"cko_frac={secondary_row['cko_frac']:.3f} vs baseline {baseline_cko_frac:.3f}", flush=True)
-        print(f"  {secondary_row['depletion_direction']}, Fisher p={secondary_row['fisher_p']:.2e}", flush=True)
-
-    print(f"\n=== PRIMARY EDEN candidate 1 (Derm2) ===", flush=True)
-    if primary_1_row:
-        print(f"  n={primary_1_row['n_cells']} (WT {primary_1_row['n_WT']} / cKO {primary_1_row['n_cKO']}), "
-              f"cko_frac={primary_1_row['cko_frac']:.3f}", flush=True)
-        print(f"  {primary_1_row['depletion_direction']}, Fisher p={primary_1_row['fisher_p']:.2e}", flush=True)
-
-    print(f"\n=== PRIMARY EDEN candidate 2 (Derm9) ===", flush=True)
-    if primary_2_row:
-        print(f"  n={primary_2_row['n_cells']} (WT {primary_2_row['n_WT']} / cKO {primary_2_row['n_cKO']}), "
-              f"cko_frac={primary_2_row['cko_frac']:.3f}", flush=True)
-        print(f"  {primary_2_row['depletion_direction']}, Fisher p={primary_2_row['fisher_p']:.2e}", flush=True)
-
-
-if __name__ == "__main__":
-    main()
+"""primary + secondary EDEN on Dingwall via score_genes against dingwall's own Derm0-11 markers (Data S1C top-30); argmax identity + Fisher cKO depletion."""
+from pathlib import Path
+import warnings, json, sys, numpy as np, pandas as pd, anndata as ad, scanpy as sc, scipy.sparse as sp
+from scipy.stats import fisher_exact
+warnings.filterwarnings("ignore"); sc.settings.verbosity = 0
+
+import os as _os
+from pathlib import Path as _Path
+PANDA_ROOT = _Path(_os.environ.get("PANDA_ROOT", str(_Path(__file__).resolve().parents[2])))
+ROOT = Path(str(PANDA_ROOT))
+CKO_GSMS = {"GSM6833482", "GSM6833483"}  # CORRECTED: 480/481 are rttaControl (WT), not cKO
+WT_GSMS  = {"GSM6833478", "GSM6833479", "GSM6833480", "GSM6833481"}  # CORRECTED: 4 Cre-neg controls per GEO metadata
+DERM_MARKERS = ROOT / "data/external_labels/dingwall_supp/biorxiv_media-3.xlsx"
+TOP_N = 30  # top N markers per Derm cluster for scoring
+
+# EDEN identity map based on Data S1C + Data S2 CellChat
+EDEN_IDENTITY = {
+    10: "Secondary_EDEN",
+    2: "Primary_EDEN_candidate_1",
+    9: "Primary_EDEN_candidate_2",
+    3: "EDEN_signalling",
+    6: "EDEN_signalling",
+}
+
+
+def load_derm_marker_panels():
+    df = pd.read_excel(DERM_MARKERS)
+    df = df.sort_values(["cluster", "avg_log2FC"], ascending=[True, False])
+    panels = {}
+    for cl in sorted(df["cluster"].unique()):
+        genes = df[df["cluster"] == cl].head(TOP_N)["gene"].tolist()
+        panels[int(cl)] = genes
+    return panels
+
+
+def load_dingwall_v3_dermal():
+    raw = ad.read_h5ad(ROOT / "data/raw/GSE220977_combined.h5ad")
+    pred = pd.read_csv(ROOT / "discovery/pan_skin/marker/dingwall_predictions.csv")
+    pred_map = dict(zip(pred["cell_id"].astype(str), pred["pred_label"]))
+    raw.obs["pred_label"] = np.array([pred_map.get(c, "unknown") for c in raw.obs_names.astype(str)])
+    raw.obs["genotype"] = np.where(raw.obs["sample"].astype(str).isin(list(CKO_GSMS)), "En1-cKO",
+                          np.where(raw.obs["sample"].astype(str).isin(list(WT_GSMS)), "WT", "other"))
+    labeled = raw.obs["genotype"].isin(["WT", "En1-cKO"]).values
+    raw = raw[labeled].copy()
+    dermal_mask = np.isin(raw.obs["pred_label"], ["fibroblast-papillary", "fibroblast-reticular"])
+    dermal = raw[dermal_mask].copy()
+    return dermal
+
+
+def score_derm_identities(a, panels):
+    for cl, genes in panels.items():
+        present = [g for g in genes if g in a.var_names]
+        if not present:
+            a.obs[f"derm{cl}_score"] = 0.0
+            continue
+        sc.tl.score_genes(a, gene_list=present, score_name=f"derm{cl}_score",
+                          random_state=0, use_raw=False)
+    return a
+
+
+def main():
+    print("[eden] loading Data S1C marker panels", flush=True)
+    panels = load_derm_marker_panels()
+    print(f"[eden] loaded {len(panels)} Derm panels", flush=True)
+
+    print("[eden] loading Dingwall + PANDA-v3 dermal predictions", flush=True)
+    dermal = load_dingwall_v3_dermal()
+    print(f"[eden] {dermal.n_obs} dermal-fibroblast cells", flush=True)
+
+    print("[eden] normalise + log1p", flush=True)
+    sc.pp.normalize_total(dermal, target_sum=1e4); sc.pp.log1p(dermal)
+
+    print("[eden] scoring cells on all 12 Derm identity panels", flush=True)
+    dermal = score_derm_identities(dermal, panels)
+
+    score_cols = [f"derm{cl}_score" for cl in sorted(panels.keys())]
+    score_mat = dermal.obs[score_cols].values
+    argmax = np.argmax(score_mat, axis=1)
+    derm_ids = [int(score_cols[i].replace("derm", "").replace("_score", "")) for i in argmax]
+    dermal.obs["derm_identity"] = derm_ids
+    dermal.obs["max_derm_score"] = score_mat.max(axis=1)
+
+    n_wt_tot = int((dermal.obs["genotype"] == "WT").sum())
+    n_cko_tot = int((dermal.obs["genotype"] == "En1-cKO").sum())
+    baseline_cko_frac = n_cko_tot / (n_wt_tot + n_cko_tot)
+    print(f"[eden] baseline WT={n_wt_tot} cKO={n_cko_tot} (baseline cKO={baseline_cko_frac:.3f})", flush=True)
+
+    rows = []
+    for cl in sorted(panels.keys()):
+        sub = dermal[dermal.obs["derm_identity"] == cl]
+        n_wt = int((sub.obs["genotype"] == "WT").sum())
+        n_cko = int((sub.obs["genotype"] == "En1-cKO").sum())
+        if n_wt + n_cko == 0:
+            continue
+        cko_frac = n_cko / (n_wt + n_cko) if (n_wt + n_cko) > 0 else 0
+        n_wt_else = n_wt_tot - n_wt
+        n_cko_else = n_cko_tot - n_cko
+        try:
+            odds, p_fisher = fisher_exact([[n_wt, n_wt_else], [n_cko, n_cko_else]],
+                                          alternative="two-sided")
+        except ValueError:
+            odds, p_fisher = 1.0, 1.0
+        rows.append({
+            "derm_id": cl,
+            "identity": EDEN_IDENTITY.get(cl, "other"),
+            "n_cells": n_wt + n_cko,
+            "n_WT": n_wt, "n_cKO": n_cko,
+            "cko_frac": cko_frac,
+            "baseline_cko_frac": baseline_cko_frac,
+            "cko_delta": cko_frac - baseline_cko_frac,
+            "fisher_p": float(p_fisher),
+            "odds_ratio": float(odds) if not np.isnan(odds) else None,
+            "depletion_direction": "cKO-depleted" if cko_frac < baseline_cko_frac
+                                   else "cKO-enriched",
+            "top10_markers_dingwall": ", ".join(panels[cl][:10]),
+        })
+    df = pd.DataFrame(rows).sort_values("cko_delta")
+
+    out = ROOT / "discovery/pan_skin/marker"
+    out.mkdir(parents=True, exist_ok=True)
+    df.to_csv(out / "101_derm_subcluster_scores.csv", index=False)
+
+    secondary_row = df[df["derm_id"] == 10].iloc[0].to_dict() if 10 in df["derm_id"].values else None
+    primary_1_row = df[df["derm_id"] == 2].iloc[0].to_dict() if 2 in df["derm_id"].values else None
+    primary_2_row = df[df["derm_id"] == 9].iloc[0].to_dict() if 9 in df["derm_id"].values else None
+
+    summary = {
+        "target": "Dingwall_GSE220977",
+        "method": "Score PANDA-v3 dermal-fibroblast predictions on Dingwall's own Derm0-11 "
+                  "marker panels (Data S1C top-30 genes each); argmax identity per cell; "
+                  "Fisher-exact cKO depletion per Derm identity",
+        "baseline_cko_frac": baseline_cko_frac,
+        "n_dermal_cells_total": int(dermal.n_obs),
+        "secondary_eden_Derm10": secondary_row,
+        "primary_eden_Derm2": primary_1_row,
+        "primary_eden_Derm9": primary_2_row,
+        "all_derm_summary": df.to_dict("records"),
+    }
+    (out / "101_derm_identity_summary.json").write_text(json.dumps(summary, indent=2, default=str))
+
+    print(f"\n[eden] wrote {out}/101_derm_*", flush=True)
+    print(f"\n=== SECONDARY EDEN (Derm10) ===", flush=True)
+    if secondary_row:
+        print(f"  n={secondary_row['n_cells']} (WT {secondary_row['n_WT']} / cKO {secondary_row['n_cKO']}), "
+              f"cko_frac={secondary_row['cko_frac']:.3f} vs baseline {baseline_cko_frac:.3f}", flush=True)
+        print(f"  {secondary_row['depletion_direction']}, Fisher p={secondary_row['fisher_p']:.2e}", flush=True)
+
+    print(f"\n=== PRIMARY EDEN candidate 1 (Derm2) ===", flush=True)
+    if primary_1_row:
+        print(f"  n={primary_1_row['n_cells']} (WT {primary_1_row['n_WT']} / cKO {primary_1_row['n_cKO']}), "
+              f"cko_frac={primary_1_row['cko_frac']:.3f}", flush=True)
+        print(f"  {primary_1_row['depletion_direction']}, Fisher p={primary_1_row['fisher_p']:.2e}", flush=True)
+
+    print(f"\n=== PRIMARY EDEN candidate 2 (Derm9) ===", flush=True)
+    if primary_2_row:
+        print(f"  n={primary_2_row['n_cells']} (WT {primary_2_row['n_WT']} / cKO {primary_2_row['n_cKO']}), "
+              f"cko_frac={primary_2_row['cko_frac']:.3f}", flush=True)
+        print(f"  {primary_2_row['depletion_direction']}, Fisher p={primary_2_row['fisher_p']:.2e}", flush=True)
+
+
+if __name__ == "__main__":
+    main()

scripts/analysis/102_train_on_dingwall_variantA.py

@@ -1,317 +1,320 @@
-"""variant A — semi-supervised panda on dingwall using data s3 marker panels with score+margin gate."""
-from __future__ import annotations
-from pathlib import Path
-import warnings, json, sys, time
-warnings.filterwarnings("ignore")
-
-import numpy as np
-import pandas as pd
-import anndata as ad
-import scanpy as sc
-import scipy.sparse as sp
-from scipy.stats import fisher_exact
-import torch
-import torch.nn.functional as F
-from torch.utils.data import Dataset, DataLoader
-
-sys.path.insert(0, "/home/bcheng/PRISM")
-from panda.model import (
-    PANDAEncoder, supcon_loss, vicreg_loss, hsic_biased, subcenter_angular_infonce
-)
-
-ROOT = Path("/home/bcheng/PRISM")
-RAW_H5 = ROOT / "data/raw/GSE220977_combined.h5ad"
-DERM_MARKERS = ROOT / "data/external_labels/dingwall_supp/biorxiv_media-3.xlsx"
-OUT_DIR = ROOT / "discovery/pan_skin/marker"
-CK_DIR = ROOT / "checkpoints/pan_skin_dingwall_variantA"
-
-# Dingwall GSM -> genotype (see 101_primary_eden_derm_scoring)
-CKO_GSMS = {"GSM6833482", "GSM6833483"}  # CORRECTED: 480/481 are rttaControl (WT), not cKO
-WT_GSMS  = {"GSM6833478", "GSM6833479", "GSM6833480", "GSM6833481"}  # CORRECTED: 4 Cre-neg controls per GEO metadata
-
-TOP_N = 30                     # markers per Derm panel for scoring
-SCORE_MIN = 0.10               # min score to accept a pseudo-label
-MARGIN_MIN = 0.05              # min gap best - runner-up
-N_HVG = 2000                   # matches paper
-N_PCA = 40                     # matches paper
-DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
-
-# Training config (mirrors 20_train_panda.py)
-GUARANTEED_PER_CLASS = 6
-NATURAL_SLOTS = 96
-STAGE_EPOCHS = [15, 25, 40, 40]
-BALANCE_MIX = 0.5
-
-
-# ---------- data prep ----------
-
-def load_derm_panels(top_n: int = TOP_N) -> dict[int, list[str]]:
-    df = pd.read_excel(DERM_MARKERS)
-    df = df.sort_values(["cluster", "avg_log2FC"], ascending=[True, False])
-    return {int(c): df[df["cluster"] == c].head(top_n)["gene"].tolist()
-            for c in sorted(df["cluster"].unique())}
-
-
-def load_dingwall_dermal() -> ad.AnnData:
-    a = ad.read_h5ad(RAW_H5)
-    sample = a.obs["sample"].astype(str)
-    a.obs["genotype"] = np.where(sample.isin(list(CKO_GSMS)), "En1-cKO",
-                        np.where(sample.isin(list(WT_GSMS)), "WT", "other"))
-    a = a[a.obs["genotype"].isin(["WT", "En1-cKO"])].copy()
-    return a
-
-
-def preprocess_paper_style(a: ad.AnnData) -> ad.AnnData:
-    """lognormalize + hvg(2000) + pca(40) + harmony per-sample, matches dingwall STAR methods."""
-    sc.pp.filter_genes(a, min_cells=10)
-    sc.pp.normalize_total(a, target_sum=1e4)
-    sc.pp.log1p(a)
-    sc.pp.highly_variable_genes(a, n_top_genes=N_HVG, flavor="seurat", batch_key="sample")
-    sc.pp.scale(a, max_value=10, zero_center=False)
-    sc.tl.pca(a, n_comps=N_PCA, use_highly_variable=True, zero_center=False)
-    try:
-        import harmonypy as hm  # noqa
-        sc.external.pp.harmony_integrate(a, key="sample", basis="X_pca",
-                                         adjusted_basis="X_pca_harmony", max_iter_harmony=20)
-        a.obsm["X_train"] = a.obsm["X_pca_harmony"]
-    except Exception as exc:
-        print(f"[preprocess] harmony skipped ({exc}); using raw PCA", flush=True)
-        a.obsm["X_train"] = a.obsm["X_pca"]
-    return a
-
-
-# ---------- pseudo-labelling ----------
-
-def score_and_gate(a: ad.AnnData, panels: dict[int, list[str]],
-                   score_min: float = SCORE_MIN,
-                   margin_min: float = MARGIN_MIN) -> ad.AnnData:
-    """score cells on 12 derm panels; accept label if best>score_min and margin>margin_min."""
-    for cl, genes in panels.items():
-        present = [g for g in genes if g in a.var_names]
-        if not present:
-            a.obs[f"derm{cl}_score"] = 0.0
-        else:
-            sc.tl.score_genes(a, gene_list=present, score_name=f"derm{cl}_score",
-                              random_state=0, use_raw=False)
-    cols = [f"derm{cl}_score" for cl in sorted(panels)]
-    S = a.obs[cols].values
-    top1_ix = S.argmax(axis=1)
-    top1 = S[np.arange(len(S)), top1_ix]
-    S_copy = S.copy(); S_copy[np.arange(len(S)), top1_ix] = -np.inf
-    top2 = S_copy.max(axis=1)
-    margin = top1 - top2
-    accept = (top1 > score_min) & (margin > margin_min)
-
-    ids = np.array([int(cols[i].replace("derm", "").replace("_score", "")) for i in top1_ix])
-    a.obs["derm_pseudo"] = ids
-    a.obs["derm_pseudo_top1"] = top1
-    a.obs["derm_pseudo_margin"] = margin
-    a.obs["derm_pseudo_accept"] = accept
-    return a
-
-
-# ---------- PANDA training (mirrors 20_train_panda.py) ----------
-
-class CorpusDataset(Dataset):
-    def __init__(self, X, y, d, aux):
-        self.X = X.astype(np.float32); self.y = y.astype(np.int64)
-        self.d = d.astype(np.int64); self.aux = aux.astype(np.float32)
-    def __len__(self): return self.X.shape[0]
-    def __getitem__(self, i):
-        return (torch.from_numpy(self.X[i]), torch.tensor(self.y[i]),
-                torch.tensor(self.d[i]), torch.from_numpy(self.aux[i]))
-
-
-class HybridSampler:
-    def __init__(self, y, n_batches=100, seed=0):
-        self.y = np.asarray(y); self.n_batches = n_batches
-        self.rng = np.random.default_rng(seed)
-        self.classes = np.unique(self.y)
-        self.by_cls = {int(c): np.where(self.y == c)[0] for c in self.classes}
-        counts = np.bincount(self.y, minlength=int(self.classes.max()) + 1).astype(float)
-        self.natural_p = counts / counts.sum()
-    def __iter__(self):
-        for _ in range(self.n_batches):
-            batch = []
-            for c in self.classes:
-                idx = self.by_cls[int(c)]
-                take = min(GUARANTEED_PER_CLASS, len(idx))
-                if take > 0:
-                    batch.extend(self.rng.choice(idx, size=take, replace=(len(idx) < take)).tolist())
-            for _ in range(NATURAL_SLOTS):
-                c = self.rng.choice(len(self.natural_p), p=self.natural_p)
-                idx = self.by_cls.get(int(c), self.by_cls[int(self.classes[0])])
-                batch.append(int(self.rng.choice(idx)))
-            yield batch
-    def __len__(self): return self.n_batches
-
-
-def train_panda(X_tr, y_tr, d_tr, aux_tr, n_classes, n_datasets, ck_out: Path):
-    ck_out.mkdir(parents=True, exist_ok=True)
-    counts = np.bincount(y_tr, minlength=n_classes)
-    inv_sqrt = 1.0 / np.sqrt(counts + 1); inv_sqrt = inv_sqrt / inv_sqrt.mean()
-    class_w = BALANCE_MIX * inv_sqrt + (1 - BALANCE_MIX) * np.ones_like(inv_sqrt)
-    class_w = torch.tensor(class_w, dtype=torch.float32, device=DEVICE)
-
-    ds = CorpusDataset(X_tr, y_tr, d_tr, aux_tr)
-    loader = DataLoader(ds, batch_sampler=HybridSampler(y_tr, n_batches=100), num_workers=0)
-
-    model = PANDAEncoder(variant="pca", n_pca=X_tr.shape[1], n_classes=n_classes,
-                         n_datasets=n_datasets).to(DEVICE)
-    opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=1e-4)
-
-    for stage, n_ep in enumerate(STAGE_EPOCHS):
-        print(f"[panda-A] stage {stage} ({n_ep} epochs)", flush=True)
-        for e in range(n_ep):
-            t0 = time.time(); losses = []
-            for X_b, y_b, d_b, aux_b in loader:
-                X_b = X_b.to(DEVICE); y_b = y_b.to(DEVICE); d_b = d_b.to(DEVICE); aux_b = aux_b.to(DEVICE)
-                lam = 1.0 if stage >= 2 else 0.0
-                out = model(X_b, aux_b, lam_dann=lam)
-                L_supcon = supcon_loss(out["z"], y_b)
-                L_vic = vicreg_loss(out["z"])
-                L_ce = F.cross_entropy(out["logits"], y_b, weight=class_w, label_smoothing=0.05)
-                total = L_supcon + 1.0 * L_vic + 0.4 * L_ce
-                if stage >= 1:
-                    proto_ref = model.prototypes.detach().clone()
-                    total = total + 0.6 * subcenter_angular_infonce(out["z"], y_b, proto_ref)
-                if stage >= 2:
-                    total = total + F.cross_entropy(out["dom"], d_b)
-                    total = total + 0.3 * F.mse_loss(out["depth"].squeeze(1), aux_b[:, 1])
-                    total = total + 0.05 * hsic_biased(out["repr"], aux_b[:, 1:2])
-                opt.zero_grad(); total.backward()
-                torch.nn.utils.clip_grad_norm_(model.parameters(), 5.0)
-                opt.step()
-                if stage >= 1:
-                    model.update_prototypes(out["z"].detach(), y_b)
-                losses.append(float(total.item()))
-            if (e + 1) % 5 == 0:
-                print(f"  ep {e+1}/{n_ep} loss={np.mean(losses):.3f} dt={time.time()-t0:.1f}s", flush=True)
-        torch.save({"model": model.state_dict()}, ck_out / f"panda_stage{stage}.pt")
-
-    torch.save({"model": model.state_dict(),
-                "prototypes": model.prototypes.detach().cpu().numpy()},
-               ck_out / "panda_final.pt")
-    return model
-
-
-@torch.no_grad()
-def infer(model, X, aux):
-    model.eval()
-    Xt = torch.from_numpy(X.astype(np.float32)).to(DEVICE)
-    at = torch.from_numpy(aux.astype(np.float32)).to(DEVICE)
-    B = 4096; preds = []; confs = []
-    for i in range(0, len(Xt), B):
-        out = model(Xt[i:i+B], at[i:i+B])
-        p = F.softmax(out["logits"], dim=1)
-        preds.append(p.argmax(dim=1).cpu().numpy())
-        confs.append(p.max(dim=1).values.cpu().numpy())
-    return np.concatenate(preds), np.concatenate(confs)
-
-
-# ---------- reporting ----------
-
-def report_depletion(labels: np.ndarray, genotype: np.ndarray, n_classes: int) -> pd.DataFrame:
-    n_wt = int((genotype == "WT").sum()); n_cko = int((genotype == "En1-cKO").sum())
-    base = n_cko / max(n_wt + n_cko, 1)
-    rows = []
-    for c in range(n_classes):
-        m = labels == c
-        w = int(((genotype == "WT") & m).sum()); k = int(((genotype == "En1-cKO") & m).sum())
-        if w + k == 0: continue
-        try:
-            odds, p = fisher_exact([[w, n_wt - w], [k, n_cko - k]], alternative="two-sided")
-        except ValueError:
-            odds, p = 1.0, 1.0
-        rows.append({"derm_id": c, "n": w + k, "n_WT": w, "n_cKO": k,
-                     "cko_frac": k / (w + k), "baseline": base,
-                     "odds_ratio": float(odds), "fisher_p": float(p)})
-    return pd.DataFrame(rows).sort_values("cko_frac")
-
-
-def main():
-    OUT_DIR.mkdir(parents=True, exist_ok=True); CK_DIR.mkdir(parents=True, exist_ok=True)
-
-    print("[A] load panels + dermal Dingwall", flush=True)
-    panels = load_derm_panels()
-    a = load_dingwall_dermal()
-    # reuse existing panda-v3 fibroblast calls if present, else all cells
-    pred_csv = ROOT / "discovery/pan_skin/marker/dingwall_predictions.csv"
-    if pred_csv.exists():
-        pred = pd.read_csv(pred_csv)
-        pred_map = dict(zip(pred["cell_id"].astype(str), pred["pred_label"]))
-        a.obs["v3_label"] = [pred_map.get(c, "unknown") for c in a.obs_names.astype(str)]
-        a = a[np.isin(a.obs["v3_label"], ["fibroblast-papillary", "fibroblast-reticular"])].copy()
-        print(f"[A] restricted to PANDA-v3 fibroblasts: n={a.n_obs}", flush=True)
-
-    print("[A] paper-style preprocess", flush=True)
-    a = preprocess_paper_style(a)
-
-    print("[A] score + gate pseudo-labels", flush=True)
-    a = score_and_gate(a, panels)
-    n_acc = int(a.obs["derm_pseudo_accept"].sum())
-    print(f"[A] pseudo-label acceptance: {n_acc}/{a.n_obs} ({100*n_acc/a.n_obs:.1f}%)", flush=True)
-
-    # train/heldout split (gate = train; rest = infer)
-    train_mask = a.obs["derm_pseudo_accept"].values.astype(bool)
-    X_all = np.asarray(a.obsm["X_train"])
-    y_all = a.obs["derm_pseudo"].astype(int).values
-    sample_ix = {s: i for i, s in enumerate(sorted(a.obs["sample"].astype(str).unique()))}
-    d_all = np.array([sample_ix[s] for s in a.obs["sample"].astype(str)])
-    aux_all = np.stack([np.zeros(a.n_obs, dtype=np.float32),
-                        np.log10(np.asarray(a.X.sum(axis=1)).ravel() + 1)], axis=1)
-    aux_all[:, 1] = (aux_all[:, 1] - aux_all[:, 1].mean()) / (aux_all[:, 1].std() + 1e-6)
-
-    classes = sorted(np.unique(y_all[train_mask]).tolist())
-    if len(classes) < 2:
-        print("[A] not enough classes accepted; abort", flush=True); return
-    cls_ix = {c: i for i, c in enumerate(classes)}
-    y_all_ix = np.array([cls_ix.get(int(c), -1) for c in y_all])
-    y_tr = y_all_ix[train_mask]
-    X_tr = X_all[train_mask]; d_tr = d_all[train_mask]; aux_tr = aux_all[train_mask]
-
-    print(f"[A] train n={train_mask.sum()} on {len(classes)} classes: {classes}", flush=True)
-    model = train_panda(X_tr, y_tr, d_tr, aux_tr, n_classes=len(classes),
-                        n_datasets=len(sample_ix), ck_out=CK_DIR)
-
-    # inference on held-out
-    infer_mask = ~train_mask
-    preds_ix, confs = infer(model, X_all[infer_mask], aux_all[infer_mask])
-    preds_derm = np.array([classes[p] for p in preds_ix])
-
-    # combine: use pseudo-label on train, prediction on inference
-    final = np.where(train_mask, y_all,
-                     np.concatenate([y_all[train_mask].astype(int) * 0 - 1,  # placeholder
-                                     preds_derm.astype(int)])[:a.n_obs] if False else 0)
-    # simpler: assemble directly
-    final = y_all.astype(int).copy()
-    final[infer_mask] = preds_derm.astype(int)
-
-    df = pd.DataFrame({
-        "cell_id": a.obs_names.astype(str).values,
-        "sample": a.obs["sample"].astype(str).values,
-        "genotype": a.obs["genotype"].astype(str).values,
-        "pseudo_derm": y_all,
-        "pseudo_accept": train_mask,
-        "final_derm": final,
-    })
-    df.to_csv(OUT_DIR / "102_variantA_predictions.csv", index=False)
-
-    dep = report_depletion(final, a.obs["genotype"].values, n_classes=12)
-    dep.to_csv(OUT_DIR / "102_variantA_depletion.csv", index=False)
-    summary = {
-        "variant": "A_semi_supervised_S3_scoring",
-        "score_min": SCORE_MIN, "margin_min": MARGIN_MIN, "top_n": TOP_N,
-        "n_total": int(a.n_obs), "n_train_pseudo": int(train_mask.sum()),
-        "classes_trained": classes,
-        "derm10": dep[dep["derm_id"] == 10].to_dict("records"),
-        "derm2": dep[dep["derm_id"] == 2].to_dict("records"),
-        "derm9": dep[dep["derm_id"] == 9].to_dict("records"),
-        "all": dep.to_dict("records"),
-    }
-    (OUT_DIR / "102_variantA_summary.json").write_text(json.dumps(summary, indent=2, default=str))
-    print(f"[A] done -> {OUT_DIR}/102_variantA_*", flush=True)
-
-
-if __name__ == "__main__":
-    main()
+"""variant A — semi-supervised panda on dingwall using data s3 marker panels with score+margin gate."""
+from __future__ import annotations
+from pathlib import Path
+import warnings, json, sys, time
+warnings.filterwarnings("ignore")
+
+import numpy as np
+import pandas as pd
+import anndata as ad
+import scanpy as sc
+import scipy.sparse as sp
+from scipy.stats import fisher_exact
+import torch
+import torch.nn.functional as F
+from torch.utils.data import Dataset, DataLoader
+
+import os as _os
+from pathlib import Path as _Path
+PANDA_ROOT = _Path(_os.environ.get("PANDA_ROOT", str(_Path(__file__).resolve().parents[2])))
+sys.path.insert(0, str(PANDA_ROOT))
+from panda.model import (
+    PANDAEncoder, supcon_loss, vicreg_loss, hsic_biased, subcenter_angular_infonce
+)
+
+ROOT = Path(str(PANDA_ROOT))
+RAW_H5 = ROOT / "data/raw/GSE220977_combined.h5ad"
+DERM_MARKERS = ROOT / "data/external_labels/dingwall_supp/biorxiv_media-3.xlsx"
+OUT_DIR = ROOT / "discovery/pan_skin/marker"
+CK_DIR = ROOT / "checkpoints/pan_skin_dingwall_variantA"
+
+# Dingwall GSM -> genotype (see 101_primary_eden_derm_scoring)
+CKO_GSMS = {"GSM6833482", "GSM6833483"}  # CORRECTED: 480/481 are rttaControl (WT), not cKO
+WT_GSMS  = {"GSM6833478", "GSM6833479", "GSM6833480", "GSM6833481"}  # CORRECTED: 4 Cre-neg controls per GEO metadata
+
+TOP_N = 30                     # markers per Derm panel for scoring
+SCORE_MIN = 0.10               # min score to accept a pseudo-label
+MARGIN_MIN = 0.05              # min gap best - runner-up
+N_HVG = 2000                   # matches paper
+N_PCA = 40                     # matches paper
+DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+# Training config (mirrors 20_train_panda.py)
+GUARANTEED_PER_CLASS = 6
+NATURAL_SLOTS = 96
+STAGE_EPOCHS = [15, 25, 40, 40]
+BALANCE_MIX = 0.5
+
+
+# ---------- data prep ----------
+
+def load_derm_panels(top_n: int = TOP_N) -> dict[int, list[str]]:
+    df = pd.read_excel(DERM_MARKERS)
+    df = df.sort_values(["cluster", "avg_log2FC"], ascending=[True, False])
+    return {int(c): df[df["cluster"] == c].head(top_n)["gene"].tolist()
+            for c in sorted(df["cluster"].unique())}
+
+
+def load_dingwall_dermal() -> ad.AnnData:
+    a = ad.read_h5ad(RAW_H5)
+    sample = a.obs["sample"].astype(str)
+    a.obs["genotype"] = np.where(sample.isin(list(CKO_GSMS)), "En1-cKO",
+                        np.where(sample.isin(list(WT_GSMS)), "WT", "other"))
+    a = a[a.obs["genotype"].isin(["WT", "En1-cKO"])].copy()
+    return a
+
+
+def preprocess_paper_style(a: ad.AnnData) -> ad.AnnData:
+    """lognormalize + hvg(2000) + pca(40) + harmony per-sample, matches dingwall STAR methods."""
+    sc.pp.filter_genes(a, min_cells=10)
+    sc.pp.normalize_total(a, target_sum=1e4)
+    sc.pp.log1p(a)
+    sc.pp.highly_variable_genes(a, n_top_genes=N_HVG, flavor="seurat", batch_key="sample")
+    sc.pp.scale(a, max_value=10, zero_center=False)
+    sc.tl.pca(a, n_comps=N_PCA, use_highly_variable=True, zero_center=False)
+    try:
+        import harmonypy as hm  # noqa
+        sc.external.pp.harmony_integrate(a, key="sample", basis="X_pca",
+                                         adjusted_basis="X_pca_harmony", max_iter_harmony=20)
+        a.obsm["X_train"] = a.obsm["X_pca_harmony"]
+    except Exception as exc:
+        print(f"[preprocess] harmony skipped ({exc}); using raw PCA", flush=True)
+        a.obsm["X_train"] = a.obsm["X_pca"]
+    return a
+
+
+# ---------- pseudo-labelling ----------
+
+def score_and_gate(a: ad.AnnData, panels: dict[int, list[str]],
+                   score_min: float = SCORE_MIN,
+                   margin_min: float = MARGIN_MIN) -> ad.AnnData:
+    """score cells on 12 derm panels; accept label if best>score_min and margin>margin_min."""
+    for cl, genes in panels.items():
+        present = [g for g in genes if g in a.var_names]
+        if not present:
+            a.obs[f"derm{cl}_score"] = 0.0
+        else:
+            sc.tl.score_genes(a, gene_list=present, score_name=f"derm{cl}_score",
+                              random_state=0, use_raw=False)
+    cols = [f"derm{cl}_score" for cl in sorted(panels)]
+    S = a.obs[cols].values
+    top1_ix = S.argmax(axis=1)
+    top1 = S[np.arange(len(S)), top1_ix]
+    S_copy = S.copy(); S_copy[np.arange(len(S)), top1_ix] = -np.inf
+    top2 = S_copy.max(axis=1)
+    margin = top1 - top2
+    accept = (top1 > score_min) & (margin > margin_min)
+
+    ids = np.array([int(cols[i].replace("derm", "").replace("_score", "")) for i in top1_ix])
+    a.obs["derm_pseudo"] = ids
+    a.obs["derm_pseudo_top1"] = top1
+    a.obs["derm_pseudo_margin"] = margin
+    a.obs["derm_pseudo_accept"] = accept
+    return a
+
+
+# ---------- PANDA training (mirrors 20_train_panda.py) ----------
+
+class CorpusDataset(Dataset):
+    def __init__(self, X, y, d, aux):
+        self.X = X.astype(np.float32); self.y = y.astype(np.int64)
+        self.d = d.astype(np.int64); self.aux = aux.astype(np.float32)
+    def __len__(self): return self.X.shape[0]
+    def __getitem__(self, i):
+        return (torch.from_numpy(self.X[i]), torch.tensor(self.y[i]),
+                torch.tensor(self.d[i]), torch.from_numpy(self.aux[i]))
+
+
+class HybridSampler:
+    def __init__(self, y, n_batches=100, seed=0):
+        self.y = np.asarray(y); self.n_batches = n_batches
+        self.rng = np.random.default_rng(seed)
+        self.classes = np.unique(self.y)
+        self.by_cls = {int(c): np.where(self.y == c)[0] for c in self.classes}
+        counts = np.bincount(self.y, minlength=int(self.classes.max()) + 1).astype(float)
+        self.natural_p = counts / counts.sum()
+    def __iter__(self):
+        for _ in range(self.n_batches):
+            batch = []
+            for c in self.classes:
+                idx = self.by_cls[int(c)]
+                take = min(GUARANTEED_PER_CLASS, len(idx))
+                if take > 0:
+                    batch.extend(self.rng.choice(idx, size=take, replace=(len(idx) < take)).tolist())
+            for _ in range(NATURAL_SLOTS):
+                c = self.rng.choice(len(self.natural_p), p=self.natural_p)
+                idx = self.by_cls.get(int(c), self.by_cls[int(self.classes[0])])
+                batch.append(int(self.rng.choice(idx)))
+            yield batch
+    def __len__(self): return self.n_batches
+
+
+def train_panda(X_tr, y_tr, d_tr, aux_tr, n_classes, n_datasets, ck_out: Path):
+    ck_out.mkdir(parents=True, exist_ok=True)
+    counts = np.bincount(y_tr, minlength=n_classes)
+    inv_sqrt = 1.0 / np.sqrt(counts + 1); inv_sqrt = inv_sqrt / inv_sqrt.mean()
+    class_w = BALANCE_MIX * inv_sqrt + (1 - BALANCE_MIX) * np.ones_like(inv_sqrt)
+    class_w = torch.tensor(class_w, dtype=torch.float32, device=DEVICE)
+
+    ds = CorpusDataset(X_tr, y_tr, d_tr, aux_tr)
+    loader = DataLoader(ds, batch_sampler=HybridSampler(y_tr, n_batches=100), num_workers=0)
+
+    model = PANDAEncoder(variant="pca", n_pca=X_tr.shape[1], n_classes=n_classes,
+                         n_datasets=n_datasets).to(DEVICE)
+    opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=1e-4)
+
+    for stage, n_ep in enumerate(STAGE_EPOCHS):
+        print(f"[panda-A] stage {stage} ({n_ep} epochs)", flush=True)
+        for e in range(n_ep):
+            t0 = time.time(); losses = []
+            for X_b, y_b, d_b, aux_b in loader:
+                X_b = X_b.to(DEVICE); y_b = y_b.to(DEVICE); d_b = d_b.to(DEVICE); aux_b = aux_b.to(DEVICE)
+                lam = 1.0 if stage >= 2 else 0.0
+                out = model(X_b, aux_b, lam_dann=lam)
+                L_supcon = supcon_loss(out["z"], y_b)
+                L_vic = vicreg_loss(out["z"])
+                L_ce = F.cross_entropy(out["logits"], y_b, weight=class_w, label_smoothing=0.05)
+                total = L_supcon + 1.0 * L_vic + 0.4 * L_ce
+                if stage >= 1:
+                    proto_ref = model.prototypes.detach().clone()
+                    total = total + 0.6 * subcenter_angular_infonce(out["z"], y_b, proto_ref)
+                if stage >= 2:
+                    total = total + F.cross_entropy(out["dom"], d_b)
+                    total = total + 0.3 * F.mse_loss(out["depth"].squeeze(1), aux_b[:, 1])
+                    total = total + 0.05 * hsic_biased(out["repr"], aux_b[:, 1:2])
+                opt.zero_grad(); total.backward()
+                torch.nn.utils.clip_grad_norm_(model.parameters(), 5.0)
+                opt.step()
+                if stage >= 1:
+                    model.update_prototypes(out["z"].detach(), y_b)
+                losses.append(float(total.item()))
+            if (e + 1) % 5 == 0:
+                print(f"  ep {e+1}/{n_ep} loss={np.mean(losses):.3f} dt={time.time()-t0:.1f}s", flush=True)
+        torch.save({"model": model.state_dict()}, ck_out / f"panda_stage{stage}.pt")
+
+    torch.save({"model": model.state_dict(),
+                "prototypes": model.prototypes.detach().cpu().numpy()},
+               ck_out / "panda_final.pt")
+    return model
+
+
+@torch.no_grad()
+def infer(model, X, aux):
+    model.eval()
+    Xt = torch.from_numpy(X.astype(np.float32)).to(DEVICE)
+    at = torch.from_numpy(aux.astype(np.float32)).to(DEVICE)
+    B = 4096; preds = []; confs = []
+    for i in range(0, len(Xt), B):
+        out = model(Xt[i:i+B], at[i:i+B])
+        p = F.softmax(out["logits"], dim=1)
+        preds.append(p.argmax(dim=1).cpu().numpy())
+        confs.append(p.max(dim=1).values.cpu().numpy())
+    return np.concatenate(preds), np.concatenate(confs)
+
+
+# ---------- reporting ----------
+
+def report_depletion(labels: np.ndarray, genotype: np.ndarray, n_classes: int) -> pd.DataFrame:
+    n_wt = int((genotype == "WT").sum()); n_cko = int((genotype == "En1-cKO").sum())
+    base = n_cko / max(n_wt + n_cko, 1)
+    rows = []
+    for c in range(n_classes):
+        m = labels == c
+        w = int(((genotype == "WT") & m).sum()); k = int(((genotype == "En1-cKO") & m).sum())
+        if w + k == 0: continue
+        try:
+            odds, p = fisher_exact([[w, n_wt - w], [k, n_cko - k]], alternative="two-sided")
+        except ValueError:
+            odds, p = 1.0, 1.0
+        rows.append({"derm_id": c, "n": w + k, "n_WT": w, "n_cKO": k,
+                     "cko_frac": k / (w + k), "baseline": base,
+                     "odds_ratio": float(odds), "fisher_p": float(p)})
+    return pd.DataFrame(rows).sort_values("cko_frac")
+
+
+def main():
+    OUT_DIR.mkdir(parents=True, exist_ok=True); CK_DIR.mkdir(parents=True, exist_ok=True)
+
+    print("[A] load panels + dermal Dingwall", flush=True)
+    panels = load_derm_panels()
+    a = load_dingwall_dermal()
+    # reuse existing panda-v3 fibroblast calls if present, else all cells
+    pred_csv = ROOT / "discovery/pan_skin/marker/dingwall_predictions.csv"
+    if pred_csv.exists():
+        pred = pd.read_csv(pred_csv)
+        pred_map = dict(zip(pred["cell_id"].astype(str), pred["pred_label"]))
+        a.obs["v3_label"] = [pred_map.get(c, "unknown") for c in a.obs_names.astype(str)]
+        a = a[np.isin(a.obs["v3_label"], ["fibroblast-papillary", "fibroblast-reticular"])].copy()
+        print(f"[A] restricted to PANDA-v3 fibroblasts: n={a.n_obs}", flush=True)
+
+    print("[A] paper-style preprocess", flush=True)
+    a = preprocess_paper_style(a)
+
+    print("[A] score + gate pseudo-labels", flush=True)
+    a = score_and_gate(a, panels)
+    n_acc = int(a.obs["derm_pseudo_accept"].sum())
+    print(f"[A] pseudo-label acceptance: {n_acc}/{a.n_obs} ({100*n_acc/a.n_obs:.1f}%)", flush=True)
+
+    # train/heldout split (gate = train; rest = infer)
+    train_mask = a.obs["derm_pseudo_accept"].values.astype(bool)
+    X_all = np.asarray(a.obsm["X_train"])
+    y_all = a.obs["derm_pseudo"].astype(int).values
+    sample_ix = {s: i for i, s in enumerate(sorted(a.obs["sample"].astype(str).unique()))}
+    d_all = np.array([sample_ix[s] for s in a.obs["sample"].astype(str)])
+    aux_all = np.stack([np.zeros(a.n_obs, dtype=np.float32),
+                        np.log10(np.asarray(a.X.sum(axis=1)).ravel() + 1)], axis=1)
+    aux_all[:, 1] = (aux_all[:, 1] - aux_all[:, 1].mean()) / (aux_all[:, 1].std() + 1e-6)
+
+    classes = sorted(np.unique(y_all[train_mask]).tolist())
+    if len(classes) < 2:
+        print("[A] not enough classes accepted; abort", flush=True); return
+    cls_ix = {c: i for i, c in enumerate(classes)}
+    y_all_ix = np.array([cls_ix.get(int(c), -1) for c in y_all])
+    y_tr = y_all_ix[train_mask]
+    X_tr = X_all[train_mask]; d_tr = d_all[train_mask]; aux_tr = aux_all[train_mask]
+
+    print(f"[A] train n={train_mask.sum()} on {len(classes)} classes: {classes}", flush=True)
+    model = train_panda(X_tr, y_tr, d_tr, aux_tr, n_classes=len(classes),
+                        n_datasets=len(sample_ix), ck_out=CK_DIR)
+
+    # inference on held-out
+    infer_mask = ~train_mask
+    preds_ix, confs = infer(model, X_all[infer_mask], aux_all[infer_mask])
+    preds_derm = np.array([classes[p] for p in preds_ix])
+
+    # combine: use pseudo-label on train, prediction on inference
+    final = np.where(train_mask, y_all,
+                     np.concatenate([y_all[train_mask].astype(int) * 0 - 1,  # placeholder
+                                     preds_derm.astype(int)])[:a.n_obs] if False else 0)
+    # simpler: assemble directly
+    final = y_all.astype(int).copy()
+    final[infer_mask] = preds_derm.astype(int)
+
+    df = pd.DataFrame({
+        "cell_id": a.obs_names.astype(str).values,
+        "sample": a.obs["sample"].astype(str).values,
+        "genotype": a.obs["genotype"].astype(str).values,
+        "pseudo_derm": y_all,
+        "pseudo_accept": train_mask,
+        "final_derm": final,
+    })
+    df.to_csv(OUT_DIR / "102_variantA_predictions.csv", index=False)
+
+    dep = report_depletion(final, a.obs["genotype"].values, n_classes=12)
+    dep.to_csv(OUT_DIR / "102_variantA_depletion.csv", index=False)
+    summary = {
+        "variant": "A_semi_supervised_S3_scoring",
+        "score_min": SCORE_MIN, "margin_min": MARGIN_MIN, "top_n": TOP_N,
+        "n_total": int(a.n_obs), "n_train_pseudo": int(train_mask.sum()),
+        "classes_trained": classes,
+        "derm10": dep[dep["derm_id"] == 10].to_dict("records"),
+        "derm2": dep[dep["derm_id"] == 2].to_dict("records"),
+        "derm9": dep[dep["derm_id"] == 9].to_dict("records"),
+        "all": dep.to_dict("records"),
+    }
+    (OUT_DIR / "102_variantA_summary.json").write_text(json.dumps(summary, indent=2, default=str))
+    print(f"[A] done -> {OUT_DIR}/102_variantA_*", flush=True)
+
+
+if __name__ == "__main__":
+    main()

scripts/analysis/103_replicate_dingwall_seurat_pipeline.py

@@ -1,261 +1,264 @@
-"""replicate dingwall's seurat clustering (QC, harmony, PCA, leiden) to derive Derm0..Derm11 labels."""
-from __future__ import annotations
-from pathlib import Path
-import warnings, json, sys
-warnings.filterwarnings("ignore")
-
-import numpy as np
-import pandas as pd
-import anndata as ad
-import scanpy as sc
-
-ROOT = Path("/home/bcheng/PRISM")
-RAW_H5 = ROOT / "data/raw/GSE220977_combined.h5ad"
-DERM_MARKERS = ROOT / "data/external_labels/dingwall_supp/biorxiv_media-3.xlsx"
-TOP_MARKERS = ROOT / "data/external_labels/dingwall_supp/biorxiv_media-1.xlsx"
-OUT_DIR = ROOT / "data/processed/dingwall_replica"
-
-CKO_GSMS = {"GSM6833482", "GSM6833483"}  # CORRECTED: 480/481 are rttaControl (WT), not cKO
-WT_GSMS  = {"GSM6833478", "GSM6833479", "GSM6833480", "GSM6833481"}  # CORRECTED: 4 Cre-neg controls per GEO metadata
-
-# Paper values
-DERMAL_TOP_CLUSTERS = {0, 1, 3, 4, 5, 8, 11, 20}   # from STAR Methods
-N_HVG = 2000
-N_PCA = 40
-RES = 0.7
-N_LEIDEN_DERM = 12                                  # target: Derm0..Derm11
-JACCARD_TOP_N = 50                                  # top-N markers for label mapping
-
-
-# ---------- QC + preprocessing ----------
-
-def qc_filter(a: ad.AnnData) -> ad.AnnData:
-    a.var["mt"] = a.var_names.str.upper().str.startswith("MT-") | \
-                  a.var_names.str.startswith("mt-")
-    sc.pp.calculate_qc_metrics(a, qc_vars=["mt"], inplace=True, percent_top=None,
-                               log1p=False)
-    sc.pp.filter_cells(a, min_genes=300)
-    a = a[a.obs["n_genes_by_counts"] < 6000].copy()
-    a = a[a.obs["pct_counts_mt"] < 5].copy()
-    sc.pp.filter_genes(a, min_cells=10)
-    return a
-
-
-def lognorm(a: ad.AnnData) -> ad.AnnData:
-    a.layers["counts"] = a.X.copy() if not hasattr(a.X, "toarray") or True else a.X.copy()
-    sc.pp.normalize_total(a, target_sum=1e4)
-    sc.pp.log1p(a)
-    return a
-
-
-def hvg_pca_harmony(a: ad.AnnData, n_hvg=N_HVG, n_pca=N_PCA, batch_key="sample") -> ad.AnnData:
-    sc.pp.highly_variable_genes(a, n_top_genes=n_hvg, flavor="seurat", batch_key=batch_key)
-    a_use = a[:, a.var["highly_variable"]].copy()
-    sc.pp.scale(a_use, max_value=10, zero_center=False)
-    sc.tl.pca(a_use, n_comps=n_pca, use_highly_variable=False, zero_center=False)
-    # copy PCA back — a_use has same obs rows as a
-    a.obsm["X_pca"] = a_use.obsm["X_pca"].copy()
-    rep = "X_pca"
-    try:
-        # harmonypy directly on the pca matrix to avoid scanpy wrapper obsm-shape bug
-        import harmonypy as hm
-        pca_mat = a.obsm["X_pca"].copy()
-        meta = a.obs[[batch_key]].reset_index(drop=True)
-        ho = hm.run_harmony(pca_mat, meta, batch_key, max_iter_harmony=20)
-        # ho.Z_corr is (pcs, cells); make it (cells, pcs)
-        z = ho.Z_corr
-        if z.shape[1] == a.n_obs:
-            harm_mat = np.ascontiguousarray(z.T)
-        elif z.shape[0] == a.n_obs:
-            harm_mat = np.ascontiguousarray(z)
-        else:
-            raise RuntimeError(f"unknown harmony shape {z.shape}, n_obs={a.n_obs}")
-        if harm_mat.shape[0] == a.n_obs and harm_mat.shape[1] == pca_mat.shape[1]:
-            a.obsm["X_pca_harmony"] = harm_mat
-            rep = "X_pca_harmony"
-            print(f"[replica] Harmony ok, X_pca_harmony shape={harm_mat.shape}", flush=True)
-        else:
-            print(f"[replica] Harmony output shape mismatch ({harm_mat.shape}); using X_pca", flush=True)
-    except Exception as exc:
-        print(f"[replica] Harmony skipped ({exc}); using X_pca", flush=True)
-    a.uns["_replica_rep"] = rep
-    return a
-
-
-def leiden_cluster(a: ad.AnnData, res=RES) -> ad.AnnData:
-    rep = a.uns.get("_replica_rep", "X_pca")
-    sc.pp.neighbors(a, n_neighbors=20, use_rep=rep, n_pcs=N_PCA)
-    sc.tl.leiden(a, resolution=res, key_added="leiden")
-    return a
-
-
-# ---------- 23-cluster stage (map dermal identity) ----------
-
-def call_dermal_23(a: ad.AnnData) -> ad.AnnData:
-    """first-pass clustering; mark cells whose leiden id maps to DERMAL_TOP_CLUSTERS."""
-    print("[23] preprocess", flush=True)
-    a = qc_filter(a); a = lognorm(a); a = hvg_pca_harmony(a)
-    print("[23] leiden res=0.7", flush=True)
-    a = leiden_cluster(a, res=RES)
-
-    # rank markers per top-level cluster
-    sc.tl.rank_genes_groups(a, "leiden", method="wilcoxon", n_genes=100)
-    df_tl = pd.read_excel(TOP_MARKERS)   # Data S1 all-cluster markers
-    tl_panels = {int(c): df_tl[df_tl["cluster"] == c].sort_values("avg_log2FC", ascending=False)
-                 .head(JACCARD_TOP_N)["gene"].tolist() for c in sorted(df_tl["cluster"].unique())}
-    tl_map = map_leiden_to_paper(a, "leiden", tl_panels, top_n=JACCARD_TOP_N)
-    a.obs["paper_cluster_23"] = a.obs["leiden"].map(lambda c: tl_map.get(str(c), -1))
-    a.obs["is_dermal_paper"] = a.obs["paper_cluster_23"].isin(DERMAL_TOP_CLUSTERS)
-    print(f"[23] cells matched to paper dermal set: {int(a.obs['is_dermal_paper'].sum())}",
-          flush=True)
-    return a
-
-
-def map_leiden_to_paper(a: ad.AnnData, key: str, paper_panels: dict[int, list[str]],
-                        top_n: int = JACCARD_TOP_N) -> dict[str, int]:
-    """best-matching paper cluster per leiden id via jaccard on top-N markers."""
-    ranks = a.uns["rank_genes_groups"]
-    names = pd.DataFrame(ranks["names"])
-    out = {}
-    used = set()
-    scores = []
-    for lc in names.columns:
-        my_top = set(names[lc].dropna().tolist()[:top_n])
-        best_pc, best_j = None, -1.0
-        for pc, panel in paper_panels.items():
-            j = len(my_top & set(panel[:top_n])) / max(len(my_top | set(panel[:top_n])), 1)
-            if j > best_j:
-                best_pc, best_j = pc, j
-        scores.append({"leiden": lc, "best_paper": best_pc, "jaccard": best_j})
-        out[lc] = best_pc
-    # convert to json-safe strings for h5ad serialization
-    a.uns[f"_map_scores_{key}"] = json.dumps(scores, default=str)
-    return out
-
-
-# ---------- dermal subclustering stage (Derm0..Derm11) ----------
-
-def subcluster_dermal(a: ad.AnnData) -> ad.AnnData:
-    dermal = a[a.obs["is_dermal_paper"]].copy()
-    # start again from raw counts on the subset
-    if "counts" in dermal.layers:
-        dermal.X = dermal.layers["counts"]
-    print(f"[derm] subset n={dermal.n_obs}", flush=True)
-    dermal = lognorm(dermal)
-    dermal = hvg_pca_harmony(dermal)
-    # tune res to hit ~12 clusters; res=0.7 is the paper value but scanpy Leiden can
-    # differ from Seurat FindClusters, so we sweep if the exact-res doesn't give 12
-    dermal = leiden_cluster(dermal, res=RES)
-    # paper uses seurat FindClusters at res=0.7; scanpy leiden can differ so sweep to hit 12
-    if len(dermal.obs["leiden"].unique()) != N_LEIDEN_DERM:
-        for r in [0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.1, 1.2]:
-            sc.tl.leiden(dermal, resolution=r, key_added=f"leiden_r{r}")
-            if len(dermal.obs[f"leiden_r{r}"].unique()) == N_LEIDEN_DERM:
-                dermal.obs["leiden"] = dermal.obs[f"leiden_r{r}"]
-                dermal.uns["_replica_derm_res"] = r
-                break
-    print(f"[derm] n_leiden = {len(dermal.obs['leiden'].unique())}", flush=True)
-
-    # rank markers + map to Derm0..Derm11
-    sc.tl.rank_genes_groups(dermal, "leiden", method="wilcoxon", n_genes=100)
-    df_s3 = pd.read_excel(DERM_MARKERS)
-    derm_panels = {int(c): df_s3[df_s3["cluster"] == c].sort_values("avg_log2FC", ascending=False)
-                   .head(JACCARD_TOP_N)["gene"].tolist() for c in sorted(df_s3["cluster"].unique())}
-    derm_map = map_leiden_to_paper(dermal, "leiden", derm_panels, top_n=JACCARD_TOP_N)
-    dermal.obs["derm_label"] = dermal.obs["leiden"].map(lambda c: f"Derm{derm_map.get(str(c), -1)}")
-
-    # push labels back into full object
-    labels = pd.Series("non_dermal", index=a.obs_names)
-    labels.loc[dermal.obs_names] = dermal.obs["derm_label"].values
-    a.obs["derm_label"] = labels
-    a.obs["leiden_derm"] = ""
-    a.obs.loc[dermal.obs_names, "leiden_derm"] = dermal.obs["leiden"].astype(str).values
-    # h5py-safe: stringify keys AND serialize dicts to json
-    a.uns["derm_leiden_to_paper"] = json.dumps({str(k): int(v) if v is not None else -1
-                                                 for k, v in derm_map.items()}, default=str)
-    a.uns["derm_panels_used"] = json.dumps({str(k): [str(g) for g in v]
-                                             for k, v in derm_panels.items()}, default=str)
-    return a, dermal
-
-
-# ---------- QC of the replica: cluster 20 fractions ----------
-
-def qc_cluster_20(a: ad.AnnData) -> dict:
-    a.obs["genotype"] = a.obs.get("genotype", pd.Series("unknown", index=a.obs_names))
-    if "genotype" not in a.obs or a.obs["genotype"].nunique() < 2:
-        s = a.obs["sample"].astype(str)
-        a.obs["genotype"] = np.where(s.isin(list(CKO_GSMS)), "En1-cKO",
-                            np.where(s.isin(list(WT_GSMS)), "WT", "other"))
-
-    dermal_mask = a.obs["is_dermal_paper"].values
-    wt_derm = int(((a.obs["genotype"] == "WT") & dermal_mask).sum())
-    ck_derm = int(((a.obs["genotype"] == "En1-cKO") & dermal_mask).sum())
-
-    # top-level cluster 20 replica
-    c20 = a.obs["paper_cluster_23"] == 20
-    wt_c20 = int(((a.obs["genotype"] == "WT") & c20).sum())
-    ck_c20 = int(((a.obs["genotype"] == "En1-cKO") & c20).sum())
-
-    # derm10 replica
-    d10 = a.obs["derm_label"] == "Derm10"
-    wt_d10 = int(((a.obs["genotype"] == "WT") & d10).sum())
-    ck_d10 = int(((a.obs["genotype"] == "En1-cKO") & d10).sum())
-
-    return {
-        "expected_paper": {"wt_dermal_total": 17398, "cko_dermal_total": 8461,
-                           "wt_c20_pct": 1.99, "cko_c20_pct": 0.08,
-                           "wt_c20_abs": 346, "cko_c20_abs": 7},
-        "replica": {
-            "wt_dermal_total": wt_derm, "cko_dermal_total": ck_derm,
-            "wt_c20": wt_c20, "cko_c20": ck_c20,
-            "wt_c20_pct": 100 * wt_c20 / max(wt_derm, 1),
-            "cko_c20_pct": 100 * ck_c20 / max(ck_derm, 1),
-            "wt_derm10": wt_d10, "cko_derm10": ck_d10,
-            "wt_derm10_pct": 100 * wt_d10 / max(wt_derm, 1),
-            "cko_derm10_pct": 100 * ck_d10 / max(ck_derm, 1),
-        },
-    }
-
-
-def main():
-    OUT_DIR.mkdir(parents=True, exist_ok=True)
-    print("[replica] load raw", flush=True)
-    a = ad.read_h5ad(RAW_H5)
-    # inject genotype
-    s = a.obs["sample"].astype(str)
-    a.obs["genotype"] = np.where(s.isin(list(CKO_GSMS)), "En1-cKO",
-                        np.where(s.isin(list(WT_GSMS)), "WT", "other"))
-    a = a[a.obs["genotype"].isin(["WT", "En1-cKO"])].copy()
-    print(f"[replica] n={a.n_obs}", flush=True)
-
-    print("[replica] 23-cluster stage", flush=True)
-    a = call_dermal_23(a)
-
-    print("[replica] dermal subcluster stage", flush=True)
-    a, dermal = subcluster_dermal(a)
-
-    print("[replica] QC vs paper", flush=True)
-    qc = qc_cluster_20(a)
-    (OUT_DIR / "replica_cluster_20_qc.json").write_text(json.dumps(qc, indent=2, default=str))
-    print(json.dumps(qc, indent=2, default=str), flush=True)
-
-    # write per-Leiden -> paper mapping (parse json-back)
-    derm_map_parsed = json.loads(a.uns["derm_leiden_to_paper"])
-    mm = pd.DataFrame([{"leiden_derm": k, "paper_derm": v}
-                       for k, v in derm_map_parsed.items()])
-    mm.to_csv(OUT_DIR / "replica_marker_matches.csv", index=False)
-
-    # save — first stringify any datetime/complex obs cols to survive h5ad serialization
-    for col in list(a.obs.columns):
-        dt = a.obs[col].dtype
-        if pd.api.types.is_datetime64_any_dtype(dt) or dt == object:
-            try:
-                a.obs[col] = a.obs[col].astype(str)
-            except Exception:
-                del a.obs[col]
-    a.write_h5ad(OUT_DIR / "dingwall_replica.h5ad")
-    print(f"[replica] wrote {OUT_DIR}/dingwall_replica.h5ad", flush=True)
-
-
-if __name__ == "__main__":
-    main()
+"""replicate dingwall's seurat clustering (QC, harmony, PCA, leiden) to derive Derm0..Derm11 labels."""
+from __future__ import annotations
+from pathlib import Path
+import warnings, json, sys
+warnings.filterwarnings("ignore")
+
+import numpy as np
+import pandas as pd
+import anndata as ad
+import scanpy as sc
+
+import os as _os
+from pathlib import Path as _Path
+PANDA_ROOT = _Path(_os.environ.get("PANDA_ROOT", str(_Path(__file__).resolve().parents[2])))
+ROOT = Path(str(PANDA_ROOT))
+RAW_H5 = ROOT / "data/raw/GSE220977_combined.h5ad"
+DERM_MARKERS = ROOT / "data/external_labels/dingwall_supp/biorxiv_media-3.xlsx"
+TOP_MARKERS = ROOT / "data/external_labels/dingwall_supp/biorxiv_media-1.xlsx"
+OUT_DIR = ROOT / "data/processed/dingwall_replica"
+
+CKO_GSMS = {"GSM6833482", "GSM6833483"}  # CORRECTED: 480/481 are rttaControl (WT), not cKO
+WT_GSMS  = {"GSM6833478", "GSM6833479", "GSM6833480", "GSM6833481"}  # CORRECTED: 4 Cre-neg controls per GEO metadata
+
+# Paper values
+DERMAL_TOP_CLUSTERS = {0, 1, 3, 4, 5, 8, 11, 20}   # from STAR Methods
+N_HVG = 2000
+N_PCA = 40
+RES = 0.7
+N_LEIDEN_DERM = 12                                  # target: Derm0..Derm11
+JACCARD_TOP_N = 50                                  # top-N markers for label mapping
+
+
+# ---------- QC + preprocessing ----------
+
+def qc_filter(a: ad.AnnData) -> ad.AnnData:
+    a.var["mt"] = a.var_names.str.upper().str.startswith("MT-") | \
+                  a.var_names.str.startswith("mt-")
+    sc.pp.calculate_qc_metrics(a, qc_vars=["mt"], inplace=True, percent_top=None,
+                               log1p=False)
+    sc.pp.filter_cells(a, min_genes=300)
+    a = a[a.obs["n_genes_by_counts"] < 6000].copy()
+    a = a[a.obs["pct_counts_mt"] < 5].copy()
+    sc.pp.filter_genes(a, min_cells=10)
+    return a
+
+
+def lognorm(a: ad.AnnData) -> ad.AnnData:
+    a.layers["counts"] = a.X.copy() if not hasattr(a.X, "toarray") or True else a.X.copy()
+    sc.pp.normalize_total(a, target_sum=1e4)
+    sc.pp.log1p(a)
+    return a
+
+
+def hvg_pca_harmony(a: ad.AnnData, n_hvg=N_HVG, n_pca=N_PCA, batch_key="sample") -> ad.AnnData:
+    sc.pp.highly_variable_genes(a, n_top_genes=n_hvg, flavor="seurat", batch_key=batch_key)
+    a_use = a[:, a.var["highly_variable"]].copy()
+    sc.pp.scale(a_use, max_value=10, zero_center=False)
+    sc.tl.pca(a_use, n_comps=n_pca, use_highly_variable=False, zero_center=False)
+    # copy PCA back — a_use has same obs rows as a
+    a.obsm["X_pca"] = a_use.obsm["X_pca"].copy()
+    rep = "X_pca"
+    try:
+        # harmonypy directly on the pca matrix to avoid scanpy wrapper obsm-shape bug
+        import harmonypy as hm
+        pca_mat = a.obsm["X_pca"].copy()
+        meta = a.obs[[batch_key]].reset_index(drop=True)
+        ho = hm.run_harmony(pca_mat, meta, batch_key, max_iter_harmony=20)
+        # ho.Z_corr is (pcs, cells); make it (cells, pcs)
+        z = ho.Z_corr
+        if z.shape[1] == a.n_obs:
+            harm_mat = np.ascontiguousarray(z.T)
+        elif z.shape[0] == a.n_obs:
+            harm_mat = np.ascontiguousarray(z)
+        else:
+            raise RuntimeError(f"unknown harmony shape {z.shape}, n_obs={a.n_obs}")
+        if harm_mat.shape[0] == a.n_obs and harm_mat.shape[1] == pca_mat.shape[1]:
+            a.obsm["X_pca_harmony"] = harm_mat
+            rep = "X_pca_harmony"
+            print(f"[replica] Harmony ok, X_pca_harmony shape={harm_mat.shape}", flush=True)
+        else:
+            print(f"[replica] Harmony output shape mismatch ({harm_mat.shape}); using X_pca", flush=True)
+    except Exception as exc:
+        print(f"[replica] Harmony skipped ({exc}); using X_pca", flush=True)
+    a.uns["_replica_rep"] = rep
+    return a
+
+
+def leiden_cluster(a: ad.AnnData, res=RES) -> ad.AnnData:
+    rep = a.uns.get("_replica_rep", "X_pca")
+    sc.pp.neighbors(a, n_neighbors=20, use_rep=rep, n_pcs=N_PCA)
+    sc.tl.leiden(a, resolution=res, key_added="leiden")
+    return a
+
+
+# ---------- 23-cluster stage (map dermal identity) ----------
+
+def call_dermal_23(a: ad.AnnData) -> ad.AnnData:
+    """first-pass clustering; mark cells whose leiden id maps to DERMAL_TOP_CLUSTERS."""
+    print("[23] preprocess", flush=True)
+    a = qc_filter(a); a = lognorm(a); a = hvg_pca_harmony(a)
+    print("[23] leiden res=0.7", flush=True)
+    a = leiden_cluster(a, res=RES)
+
+    # rank markers per top-level cluster
+    sc.tl.rank_genes_groups(a, "leiden", method="wilcoxon", n_genes=100)
+    df_tl = pd.read_excel(TOP_MARKERS)   # Data S1 all-cluster markers
+    tl_panels = {int(c): df_tl[df_tl["cluster"] == c].sort_values("avg_log2FC", ascending=False)
+                 .head(JACCARD_TOP_N)["gene"].tolist() for c in sorted(df_tl["cluster"].unique())}
+    tl_map = map_leiden_to_paper(a, "leiden", tl_panels, top_n=JACCARD_TOP_N)
+    a.obs["paper_cluster_23"] = a.obs["leiden"].map(lambda c: tl_map.get(str(c), -1))
+    a.obs["is_dermal_paper"] = a.obs["paper_cluster_23"].isin(DERMAL_TOP_CLUSTERS)
+    print(f"[23] cells matched to paper dermal set: {int(a.obs['is_dermal_paper'].sum())}",
+          flush=True)
+    return a
+
+
+def map_leiden_to_paper(a: ad.AnnData, key: str, paper_panels: dict[int, list[str]],
+                        top_n: int = JACCARD_TOP_N) -> dict[str, int]:
+    """best-matching paper cluster per leiden id via jaccard on top-N markers."""
+    ranks = a.uns["rank_genes_groups"]
+    names = pd.DataFrame(ranks["names"])
+    out = {}
+    used = set()
+    scores = []
+    for lc in names.columns:
+        my_top = set(names[lc].dropna().tolist()[:top_n])
+        best_pc, best_j = None, -1.0
+        for pc, panel in paper_panels.items():
+            j = len(my_top & set(panel[:top_n])) / max(len(my_top | set(panel[:top_n])), 1)
+            if j > best_j:
+                best_pc, best_j = pc, j
+        scores.append({"leiden": lc, "best_paper": best_pc, "jaccard": best_j})
+        out[lc] = best_pc
+    # convert to json-safe strings for h5ad serialization
+    a.uns[f"_map_scores_{key}"] = json.dumps(scores, default=str)
+    return out
+
+
+# ---------- dermal subclustering stage (Derm0..Derm11) ----------
+
+def subcluster_dermal(a: ad.AnnData) -> ad.AnnData:
+    dermal = a[a.obs["is_dermal_paper"]].copy()
+    # start again from raw counts on the subset
+    if "counts" in dermal.layers:
+        dermal.X = dermal.layers["counts"]
+    print(f"[derm] subset n={dermal.n_obs}", flush=True)
+    dermal = lognorm(dermal)
+    dermal = hvg_pca_harmony(dermal)
+    # tune res to hit ~12 clusters; res=0.7 is the paper value but scanpy Leiden can
+    # differ from Seurat FindClusters, so we sweep if the exact-res doesn't give 12
+    dermal = leiden_cluster(dermal, res=RES)
+    # paper uses seurat FindClusters at res=0.7; scanpy leiden can differ so sweep to hit 12
+    if len(dermal.obs["leiden"].unique()) != N_LEIDEN_DERM:
+        for r in [0.5, 0.6, 0.7, 0.8, 0.9, 1.0, 1.1, 1.2]:
+            sc.tl.leiden(dermal, resolution=r, key_added=f"leiden_r{r}")
+            if len(dermal.obs[f"leiden_r{r}"].unique()) == N_LEIDEN_DERM:
+                dermal.obs["leiden"] = dermal.obs[f"leiden_r{r}"]
+                dermal.uns["_replica_derm_res"] = r
+                break
+    print(f"[derm] n_leiden = {len(dermal.obs['leiden'].unique())}", flush=True)
+
+    # rank markers + map to Derm0..Derm11
+    sc.tl.rank_genes_groups(dermal, "leiden", method="wilcoxon", n_genes=100)
+    df_s3 = pd.read_excel(DERM_MARKERS)
+    derm_panels = {int(c): df_s3[df_s3["cluster"] == c].sort_values("avg_log2FC", ascending=False)
+                   .head(JACCARD_TOP_N)["gene"].tolist() for c in sorted(df_s3["cluster"].unique())}
+    derm_map = map_leiden_to_paper(dermal, "leiden", derm_panels, top_n=JACCARD_TOP_N)
+    dermal.obs["derm_label"] = dermal.obs["leiden"].map(lambda c: f"Derm{derm_map.get(str(c), -1)}")
+
+    # push labels back into full object
+    labels = pd.Series("non_dermal", index=a.obs_names)
+    labels.loc[dermal.obs_names] = dermal.obs["derm_label"].values
+    a.obs["derm_label"] = labels
+    a.obs["leiden_derm"] = ""
+    a.obs.loc[dermal.obs_names, "leiden_derm"] = dermal.obs["leiden"].astype(str).values
+    # h5py-safe: stringify keys AND serialize dicts to json
+    a.uns["derm_leiden_to_paper"] = json.dumps({str(k): int(v) if v is not None else -1
+                                                 for k, v in derm_map.items()}, default=str)
+    a.uns["derm_panels_used"] = json.dumps({str(k): [str(g) for g in v]
+                                             for k, v in derm_panels.items()}, default=str)
+    return a, dermal
+
+
+# ---------- QC of the replica: cluster 20 fractions ----------
+
+def qc_cluster_20(a: ad.AnnData) -> dict:
+    a.obs["genotype"] = a.obs.get("genotype", pd.Series("unknown", index=a.obs_names))
+    if "genotype" not in a.obs or a.obs["genotype"].nunique() < 2:
+        s = a.obs["sample"].astype(str)
+        a.obs["genotype"] = np.where(s.isin(list(CKO_GSMS)), "En1-cKO",
+                            np.where(s.isin(list(WT_GSMS)), "WT", "other"))
+
+    dermal_mask = a.obs["is_dermal_paper"].values
+    wt_derm = int(((a.obs["genotype"] == "WT") & dermal_mask).sum())
+    ck_derm = int(((a.obs["genotype"] == "En1-cKO") & dermal_mask).sum())
+
+    # top-level cluster 20 replica
+    c20 = a.obs["paper_cluster_23"] == 20
+    wt_c20 = int(((a.obs["genotype"] == "WT") & c20).sum())
+    ck_c20 = int(((a.obs["genotype"] == "En1-cKO") & c20).sum())
+
+    # derm10 replica
+    d10 = a.obs["derm_label"] == "Derm10"
+    wt_d10 = int(((a.obs["genotype"] == "WT") & d10).sum())
+    ck_d10 = int(((a.obs["genotype"] == "En1-cKO") & d10).sum())
+
+    return {
+        "expected_paper": {"wt_dermal_total": 17398, "cko_dermal_total": 8461,
+                           "wt_c20_pct": 1.99, "cko_c20_pct": 0.08,
+                           "wt_c20_abs": 346, "cko_c20_abs": 7},
+        "replica": {
+            "wt_dermal_total": wt_derm, "cko_dermal_total": ck_derm,
+            "wt_c20": wt_c20, "cko_c20": ck_c20,
+            "wt_c20_pct": 100 * wt_c20 / max(wt_derm, 1),
+            "cko_c20_pct": 100 * ck_c20 / max(ck_derm, 1),
+            "wt_derm10": wt_d10, "cko_derm10": ck_d10,
+            "wt_derm10_pct": 100 * wt_d10 / max(wt_derm, 1),
+            "cko_derm10_pct": 100 * ck_d10 / max(ck_derm, 1),
+        },
+    }
+
+
+def main():
+    OUT_DIR.mkdir(parents=True, exist_ok=True)
+    print("[replica] load raw", flush=True)
+    a = ad.read_h5ad(RAW_H5)
+    # inject genotype
+    s = a.obs["sample"].astype(str)
+    a.obs["genotype"] = np.where(s.isin(list(CKO_GSMS)), "En1-cKO",
+                        np.where(s.isin(list(WT_GSMS)), "WT", "other"))
+    a = a[a.obs["genotype"].isin(["WT", "En1-cKO"])].copy()
+    print(f"[replica] n={a.n_obs}", flush=True)
+
+    print("[replica] 23-cluster stage", flush=True)
+    a = call_dermal_23(a)
+
+    print("[replica] dermal subcluster stage", flush=True)
+    a, dermal = subcluster_dermal(a)
+
+    print("[replica] QC vs paper", flush=True)
+    qc = qc_cluster_20(a)
+    (OUT_DIR / "replica_cluster_20_qc.json").write_text(json.dumps(qc, indent=2, default=str))
+    print(json.dumps(qc, indent=2, default=str), flush=True)
+
+    # write per-Leiden -> paper mapping (parse json-back)
+    derm_map_parsed = json.loads(a.uns["derm_leiden_to_paper"])
+    mm = pd.DataFrame([{"leiden_derm": k, "paper_derm": v}
+                       for k, v in derm_map_parsed.items()])
+    mm.to_csv(OUT_DIR / "replica_marker_matches.csv", index=False)
+
+    # save — first stringify any datetime/complex obs cols to survive h5ad serialization
+    for col in list(a.obs.columns):
+        dt = a.obs[col].dtype
+        if pd.api.types.is_datetime64_any_dtype(dt) or dt == object:
+            try:
+                a.obs[col] = a.obs[col].astype(str)
+            except Exception:
+                del a.obs[col]
+    a.write_h5ad(OUT_DIR / "dingwall_replica.h5ad")
+    print(f"[replica] wrote {OUT_DIR}/dingwall_replica.h5ad", flush=True)
+
+
+if __name__ == "__main__":
+    main()

scripts/analysis/104_train_on_dingwall_derm_labels.py

@@ -1,261 +1,264 @@
-"""variant B — fully supervised panda on replicated dingwall Derm0..Derm11 labels from script 103."""
-from __future__ import annotations
-from pathlib import Path
-import warnings, json, sys, time
-warnings.filterwarnings("ignore")
-
-import numpy as np
-import pandas as pd
-import anndata as ad
-import torch
-import torch.nn.functional as F
-from torch.utils.data import Dataset, DataLoader
-from scipy.stats import fisher_exact
-
-sys.path.insert(0, "/home/bcheng/PRISM")
-from panda.model import (
-    PANDAEncoder, supcon_loss, vicreg_loss, hsic_biased, subcenter_angular_infonce
-)
-
-ROOT = Path("/home/bcheng/PRISM")
-REPLICA_H5 = ROOT / "data/processed/dingwall_replica/dingwall_replica.h5ad"
-OUT_DIR = ROOT / "discovery/pan_skin/marker"
-CK_DIR = ROOT / "checkpoints/pan_skin_dingwall_derm"
-
-TRAIN_FRAC = 0.7
-SEED = 0
-N_PCA = 40
-DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
-
-# Training config (mirrors 20_train_panda.py)
-GUARANTEED_PER_CLASS = 6
-NATURAL_SLOTS = 96
-STAGE_EPOCHS = [15, 25, 40, 40]
-BALANCE_MIX = 0.5
-
-
-# ---------- split ----------
-
-def genotype_stratified_split(labels: np.ndarray, genotypes: np.ndarray,
-                              frac_train: float = TRAIN_FRAC, seed: int = SEED
-                              ) -> tuple[np.ndarray, np.ndarray]:
-    """stratified 70/30 within each (label, genotype) group; preserves cKO/WT ratio per class."""
-    rng = np.random.default_rng(seed)
-    n = len(labels); train = np.zeros(n, dtype=bool); test = np.zeros(n, dtype=bool)
-    for lab in np.unique(labels):
-        for g in np.unique(genotypes):
-            idx = np.where((labels == lab) & (genotypes == g))[0]
-            if len(idx) == 0: continue
-            rng.shuffle(idx)
-            k = max(1, int(len(idx) * frac_train)) if len(idx) > 1 else len(idx)
-            train[idx[:k]] = True
-            if len(idx) > 1:
-                test[idx[k:]] = True
-    return train, test
-
-
-# ---------- PANDA training (identical to variant A) ----------
-
-class CorpusDataset(Dataset):
-    def __init__(self, X, y, d, aux):
-        self.X = X.astype(np.float32); self.y = y.astype(np.int64)
-        self.d = d.astype(np.int64); self.aux = aux.astype(np.float32)
-    def __len__(self): return self.X.shape[0]
-    def __getitem__(self, i):
-        return (torch.from_numpy(self.X[i]), torch.tensor(self.y[i]),
-                torch.tensor(self.d[i]), torch.from_numpy(self.aux[i]))
-
-
-class HybridSampler:
-    def __init__(self, y, n_batches=100, seed=0):
-        self.y = np.asarray(y); self.n_batches = n_batches
-        self.rng = np.random.default_rng(seed)
-        self.classes = np.unique(self.y)
-        self.by_cls = {int(c): np.where(self.y == c)[0] for c in self.classes}
-        counts = np.bincount(self.y, minlength=int(self.classes.max()) + 1).astype(float)
-        self.natural_p = counts / counts.sum()
-    def __iter__(self):
-        for _ in range(self.n_batches):
-            batch = []
-            for c in self.classes:
-                idx = self.by_cls[int(c)]
-                take = min(GUARANTEED_PER_CLASS, len(idx))
-                if take > 0:
-                    batch.extend(self.rng.choice(idx, size=take, replace=(len(idx) < take)).tolist())
-            for _ in range(NATURAL_SLOTS):
-                c = self.rng.choice(len(self.natural_p), p=self.natural_p)
-                idx = self.by_cls.get(int(c), self.by_cls[int(self.classes[0])])
-                batch.append(int(self.rng.choice(idx)))
-            yield batch
-    def __len__(self): return self.n_batches
-
-
-def train_panda(X_tr, y_tr, d_tr, aux_tr, n_classes, n_datasets, ck_out: Path):
-    ck_out.mkdir(parents=True, exist_ok=True)
-    counts = np.bincount(y_tr, minlength=n_classes)
-    inv_sqrt = 1.0 / np.sqrt(counts + 1); inv_sqrt = inv_sqrt / inv_sqrt.mean()
-    class_w = BALANCE_MIX * inv_sqrt + (1 - BALANCE_MIX) * np.ones_like(inv_sqrt)
-    class_w = torch.tensor(class_w, dtype=torch.float32, device=DEVICE)
-
-    ds = CorpusDataset(X_tr, y_tr, d_tr, aux_tr)
-    loader = DataLoader(ds, batch_sampler=HybridSampler(y_tr, n_batches=100), num_workers=0)
-
-    model = PANDAEncoder(variant="pca", n_pca=X_tr.shape[1], n_classes=n_classes,
-                         n_datasets=n_datasets).to(DEVICE)
-    opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=1e-4)
-
-    for stage, n_ep in enumerate(STAGE_EPOCHS):
-        print(f"[panda-B] stage {stage} ({n_ep} epochs)", flush=True)
-        for e in range(n_ep):
-            t0 = time.time(); losses = []
-            for X_b, y_b, d_b, aux_b in loader:
-                X_b = X_b.to(DEVICE); y_b = y_b.to(DEVICE); d_b = d_b.to(DEVICE); aux_b = aux_b.to(DEVICE)
-                lam = 1.0 if stage >= 2 else 0.0
-                out = model(X_b, aux_b, lam_dann=lam)
-                L_supcon = supcon_loss(out["z"], y_b)
-                L_vic = vicreg_loss(out["z"])
-                L_ce = F.cross_entropy(out["logits"], y_b, weight=class_w, label_smoothing=0.05)
-                total = L_supcon + 1.0 * L_vic + 0.4 * L_ce
-                if stage >= 1:
-                    proto_ref = model.prototypes.detach().clone()
-                    total = total + 0.6 * subcenter_angular_infonce(out["z"], y_b, proto_ref)
-                if stage >= 2:
-                    total = total + F.cross_entropy(out["dom"], d_b)
-                    total = total + 0.3 * F.mse_loss(out["depth"].squeeze(1), aux_b[:, 1])
-                    total = total + 0.05 * hsic_biased(out["repr"], aux_b[:, 1:2])
-                opt.zero_grad(); total.backward()
-                torch.nn.utils.clip_grad_norm_(model.parameters(), 5.0)
-                opt.step()
-                if stage >= 1:
-                    model.update_prototypes(out["z"].detach(), y_b)
-                losses.append(float(total.item()))
-            if (e + 1) % 5 == 0:
-                print(f"  ep {e+1}/{n_ep} loss={np.mean(losses):.3f} dt={time.time()-t0:.1f}s", flush=True)
-        torch.save({"model": model.state_dict()}, ck_out / f"panda_stage{stage}.pt")
-    torch.save({"model": model.state_dict(),
-                "prototypes": model.prototypes.detach().cpu().numpy()},
-               ck_out / "panda_final.pt")
-    return model
-
-
-@torch.no_grad()
-def infer(model, X, aux):
-    model.eval()
-    Xt = torch.from_numpy(X.astype(np.float32)).to(DEVICE)
-    at = torch.from_numpy(aux.astype(np.float32)).to(DEVICE)
-    B = 4096; preds = []; confs = []
-    for i in range(0, len(Xt), B):
-        out = model(Xt[i:i+B], at[i:i+B])
-        p = F.softmax(out["logits"], dim=1)
-        preds.append(p.argmax(dim=1).cpu().numpy())
-        confs.append(p.max(dim=1).values.cpu().numpy())
-    return np.concatenate(preds), np.concatenate(confs)
-
-
-# ---------- reporting ----------
-
-def depletion_table(true_or_pred: np.ndarray, genotype: np.ndarray, class_names: list[str]
-                    ) -> pd.DataFrame:
-    n_wt = int((genotype == "WT").sum()); n_cko = int((genotype == "En1-cKO").sum())
-    base = n_cko / max(n_wt + n_cko, 1)
-    rows = []
-    for i, cn in enumerate(class_names):
-        m = true_or_pred == i
-        w = int(((genotype == "WT") & m).sum()); k = int(((genotype == "En1-cKO") & m).sum())
-        if w + k == 0: continue
-        try:
-            odds, p = fisher_exact([[w, n_wt - w], [k, n_cko - k]], alternative="two-sided")
-        except ValueError:
-            odds, p = 1.0, 1.0
-        rows.append({"derm_label": cn, "n": w + k, "n_WT": w, "n_cKO": k,
-                     "cko_frac": k / (w + k), "baseline_cko": base,
-                     "odds_ratio": float(odds), "fisher_p": float(p)})
-    return pd.DataFrame(rows).sort_values("cko_frac")
-
-
-def main():
-    OUT_DIR.mkdir(parents=True, exist_ok=True); CK_DIR.mkdir(parents=True, exist_ok=True)
-
-    print("[B] load replica", flush=True)
-    if not REPLICA_H5.exists():
-        raise FileNotFoundError(f"Run 103 first — {REPLICA_H5} missing")
-    a = ad.read_h5ad(REPLICA_H5)
-
-    dermal = a[a.obs["derm_label"].astype(str) != "non_dermal"].copy()
-    print(f"[B] dermal n={dermal.n_obs}", flush=True)
-    labels_str = dermal.obs["derm_label"].astype(str).values
-    classes = sorted(set(labels_str))
-    cls_ix = {c: i for i, c in enumerate(classes)}
-    y_all = np.array([cls_ix[c] for c in labels_str])
-    genotype = dermal.obs["genotype"].astype(str).values
-
-    # get embedding from replica (harmony-corrected PCA)
-    rep_key = dermal.uns.get("_replica_rep", "X_pca_harmony")
-    if rep_key not in dermal.obsm:
-        rep_key = "X_pca_harmony" if "X_pca_harmony" in dermal.obsm else "X_pca"
-    X_all = np.asarray(dermal.obsm[rep_key])
-    print(f"[B] using {rep_key} (d={X_all.shape[1]})", flush=True)
-
-    sample_ix = {s: i for i, s in enumerate(sorted(dermal.obs["sample"].astype(str).unique()))}
-    d_all = np.array([sample_ix[s] for s in dermal.obs["sample"].astype(str)])
-    total_counts = np.asarray(dermal.X.sum(axis=1)).ravel()
-    logc = np.log10(total_counts + 1); logc = (logc - logc.mean()) / (logc.std() + 1e-6)
-    aux_all = np.stack([np.zeros(dermal.n_obs, dtype=np.float32), logc.astype(np.float32)], axis=1)
-
-    print("[B] genotype-stratified 70/30 split", flush=True)
-    tr, te = genotype_stratified_split(labels_str, genotype, frac_train=TRAIN_FRAC, seed=SEED)
-    print(f"[B] train={tr.sum()} test={te.sum()}", flush=True)
-
-    manifest = pd.DataFrame({
-        "cell_id": dermal.obs_names.astype(str).values,
-        "derm_label": labels_str,
-        "genotype": genotype,
-        "split": np.where(tr, "train", np.where(te, "test", "unassigned")),
-    })
-    manifest.to_csv(OUT_DIR / "104_dingwall_derm_split_manifest.csv", index=False)
-
-    print("[B] train PANDA", flush=True)
-    model = train_panda(X_all[tr], y_all[tr], d_all[tr], aux_all[tr],
-                        n_classes=len(classes), n_datasets=len(sample_ix), ck_out=CK_DIR)
-
-    print("[B] infer on held-out", flush=True)
-    pred_ix, conf = infer(model, X_all[te], aux_all[te])
-    pred = pd.DataFrame({
-        "cell_id": dermal.obs_names.astype(str).values[te],
-        "derm_true": labels_str[te],
-        "derm_pred": [classes[p] for p in pred_ix],
-        "confidence": conf,
-        "genotype": genotype[te],
-    })
-    pred.to_csv(OUT_DIR / "104_dingwall_derm_predictions.csv", index=False)
-
-    # depletion — reported for TEST set only, using PANDA predictions
-    pred_ix_full = np.array([cls_ix[c] for c in pred["derm_pred"].values])
-    dep_pred = depletion_table(pred_ix_full, genotype[te], classes)
-    dep_true = depletion_table(y_all[te], genotype[te], classes)
-    dep_pred.to_csv(OUT_DIR / "104_dingwall_derm_depletion_pred.csv", index=False)
-    dep_true.to_csv(OUT_DIR / "104_dingwall_derm_depletion_true.csv", index=False)
-
-    d10_true = dep_true[dep_true["derm_label"] == "Derm10"].to_dict("records")
-    d10_pred = dep_pred[dep_pred["derm_label"] == "Derm10"].to_dict("records")
-    acc = float((pred_ix == y_all[te]).mean())
-
-    summary = {
-        "variant": "B_fully_supervised_replica_labels",
-        "n_dermal_total": int(dermal.n_obs),
-        "n_train": int(tr.sum()), "n_test": int(te.sum()),
-        "classes": classes,
-        "test_accuracy": acc,
-        "expected_paper_derm10": {"wt_pct": 1.99, "cko_pct": 0.08,
-                                   "or_approx": 24.5, "wt_n_approx": 346, "cko_n_approx": 7},
-        "test_derm10_true": d10_true,
-        "test_derm10_pred": d10_pred,
-        "test_depletion_true": dep_true.to_dict("records"),
-        "test_depletion_pred": dep_pred.to_dict("records"),
-    }
-    (OUT_DIR / "104_dingwall_derm_summary.json").write_text(json.dumps(summary, indent=2, default=str))
-    print(f"[B] done -> {OUT_DIR}/104_dingwall_derm_*", flush=True)
-
-
-if __name__ == "__main__":
-    main()
+"""variant B — fully supervised panda on replicated dingwall Derm0..Derm11 labels from script 103."""
+from __future__ import annotations
+from pathlib import Path
+import warnings, json, sys, time
+warnings.filterwarnings("ignore")
+
+import numpy as np
+import pandas as pd
+import anndata as ad
+import torch
+import torch.nn.functional as F
+from torch.utils.data import Dataset, DataLoader
+from scipy.stats import fisher_exact
+
+import os as _os
+from pathlib import Path as _Path
+PANDA_ROOT = _Path(_os.environ.get("PANDA_ROOT", str(_Path(__file__).resolve().parents[2])))
+sys.path.insert(0, str(PANDA_ROOT))
+from panda.model import (
+    PANDAEncoder, supcon_loss, vicreg_loss, hsic_biased, subcenter_angular_infonce
+)
+
+ROOT = Path(str(PANDA_ROOT))
+REPLICA_H5 = ROOT / "data/processed/dingwall_replica/dingwall_replica.h5ad"
+OUT_DIR = ROOT / "discovery/pan_skin/marker"
+CK_DIR = ROOT / "checkpoints/pan_skin_dingwall_derm"
+
+TRAIN_FRAC = 0.7
+SEED = 0
+N_PCA = 40
+DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+# Training config (mirrors 20_train_panda.py)
+GUARANTEED_PER_CLASS = 6
+NATURAL_SLOTS = 96
+STAGE_EPOCHS = [15, 25, 40, 40]
+BALANCE_MIX = 0.5
+
+
+# ---------- split ----------
+
+def genotype_stratified_split(labels: np.ndarray, genotypes: np.ndarray,
+                              frac_train: float = TRAIN_FRAC, seed: int = SEED
+                              ) -> tuple[np.ndarray, np.ndarray]:
+    """stratified 70/30 within each (label, genotype) group; preserves cKO/WT ratio per class."""
+    rng = np.random.default_rng(seed)
+    n = len(labels); train = np.zeros(n, dtype=bool); test = np.zeros(n, dtype=bool)
+    for lab in np.unique(labels):
+        for g in np.unique(genotypes):
+            idx = np.where((labels == lab) & (genotypes == g))[0]
+            if len(idx) == 0: continue
+            rng.shuffle(idx)
+            k = max(1, int(len(idx) * frac_train)) if len(idx) > 1 else len(idx)
+            train[idx[:k]] = True
+            if len(idx) > 1:
+                test[idx[k:]] = True
+    return train, test
+
+
+# ---------- PANDA training (identical to variant A) ----------
+
+class CorpusDataset(Dataset):
+    def __init__(self, X, y, d, aux):
+        self.X = X.astype(np.float32); self.y = y.astype(np.int64)
+        self.d = d.astype(np.int64); self.aux = aux.astype(np.float32)
+    def __len__(self): return self.X.shape[0]
+    def __getitem__(self, i):
+        return (torch.from_numpy(self.X[i]), torch.tensor(self.y[i]),
+                torch.tensor(self.d[i]), torch.from_numpy(self.aux[i]))
+
+
+class HybridSampler:
+    def __init__(self, y, n_batches=100, seed=0):
+        self.y = np.asarray(y); self.n_batches = n_batches
+        self.rng = np.random.default_rng(seed)
+        self.classes = np.unique(self.y)
+        self.by_cls = {int(c): np.where(self.y == c)[0] for c in self.classes}
+        counts = np.bincount(self.y, minlength=int(self.classes.max()) + 1).astype(float)
+        self.natural_p = counts / counts.sum()
+    def __iter__(self):
+        for _ in range(self.n_batches):
+            batch = []
+            for c in self.classes:
+                idx = self.by_cls[int(c)]
+                take = min(GUARANTEED_PER_CLASS, len(idx))
+                if take > 0:
+                    batch.extend(self.rng.choice(idx, size=take, replace=(len(idx) < take)).tolist())
+            for _ in range(NATURAL_SLOTS):
+                c = self.rng.choice(len(self.natural_p), p=self.natural_p)
+                idx = self.by_cls.get(int(c), self.by_cls[int(self.classes[0])])
+                batch.append(int(self.rng.choice(idx)))
+            yield batch
+    def __len__(self): return self.n_batches
+
+
+def train_panda(X_tr, y_tr, d_tr, aux_tr, n_classes, n_datasets, ck_out: Path):
+    ck_out.mkdir(parents=True, exist_ok=True)
+    counts = np.bincount(y_tr, minlength=n_classes)
+    inv_sqrt = 1.0 / np.sqrt(counts + 1); inv_sqrt = inv_sqrt / inv_sqrt.mean()
+    class_w = BALANCE_MIX * inv_sqrt + (1 - BALANCE_MIX) * np.ones_like(inv_sqrt)
+    class_w = torch.tensor(class_w, dtype=torch.float32, device=DEVICE)
+
+    ds = CorpusDataset(X_tr, y_tr, d_tr, aux_tr)
+    loader = DataLoader(ds, batch_sampler=HybridSampler(y_tr, n_batches=100), num_workers=0)
+
+    model = PANDAEncoder(variant="pca", n_pca=X_tr.shape[1], n_classes=n_classes,
+                         n_datasets=n_datasets).to(DEVICE)
+    opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=1e-4)
+
+    for stage, n_ep in enumerate(STAGE_EPOCHS):
+        print(f"[panda-B] stage {stage} ({n_ep} epochs)", flush=True)
+        for e in range(n_ep):
+            t0 = time.time(); losses = []
+            for X_b, y_b, d_b, aux_b in loader:
+                X_b = X_b.to(DEVICE); y_b = y_b.to(DEVICE); d_b = d_b.to(DEVICE); aux_b = aux_b.to(DEVICE)
+                lam = 1.0 if stage >= 2 else 0.0
+                out = model(X_b, aux_b, lam_dann=lam)
+                L_supcon = supcon_loss(out["z"], y_b)
+                L_vic = vicreg_loss(out["z"])
+                L_ce = F.cross_entropy(out["logits"], y_b, weight=class_w, label_smoothing=0.05)
+                total = L_supcon + 1.0 * L_vic + 0.4 * L_ce
+                if stage >= 1:
+                    proto_ref = model.prototypes.detach().clone()
+                    total = total + 0.6 * subcenter_angular_infonce(out["z"], y_b, proto_ref)
+                if stage >= 2:
+                    total = total + F.cross_entropy(out["dom"], d_b)
+                    total = total + 0.3 * F.mse_loss(out["depth"].squeeze(1), aux_b[:, 1])
+                    total = total + 0.05 * hsic_biased(out["repr"], aux_b[:, 1:2])
+                opt.zero_grad(); total.backward()
+                torch.nn.utils.clip_grad_norm_(model.parameters(), 5.0)
+                opt.step()
+                if stage >= 1:
+                    model.update_prototypes(out["z"].detach(), y_b)
+                losses.append(float(total.item()))
+            if (e + 1) % 5 == 0:
+                print(f"  ep {e+1}/{n_ep} loss={np.mean(losses):.3f} dt={time.time()-t0:.1f}s", flush=True)
+        torch.save({"model": model.state_dict()}, ck_out / f"panda_stage{stage}.pt")
+    torch.save({"model": model.state_dict(),
+                "prototypes": model.prototypes.detach().cpu().numpy()},
+               ck_out / "panda_final.pt")
+    return model
+
+
+@torch.no_grad()
+def infer(model, X, aux):
+    model.eval()
+    Xt = torch.from_numpy(X.astype(np.float32)).to(DEVICE)
+    at = torch.from_numpy(aux.astype(np.float32)).to(DEVICE)
+    B = 4096; preds = []; confs = []
+    for i in range(0, len(Xt), B):
+        out = model(Xt[i:i+B], at[i:i+B])
+        p = F.softmax(out["logits"], dim=1)
+        preds.append(p.argmax(dim=1).cpu().numpy())
+        confs.append(p.max(dim=1).values.cpu().numpy())
+    return np.concatenate(preds), np.concatenate(confs)
+
+
+# ---------- reporting ----------
+
+def depletion_table(true_or_pred: np.ndarray, genotype: np.ndarray, class_names: list[str]
+                    ) -> pd.DataFrame:
+    n_wt = int((genotype == "WT").sum()); n_cko = int((genotype == "En1-cKO").sum())
+    base = n_cko / max(n_wt + n_cko, 1)
+    rows = []
+    for i, cn in enumerate(class_names):
+        m = true_or_pred == i
+        w = int(((genotype == "WT") & m).sum()); k = int(((genotype == "En1-cKO") & m).sum())
+        if w + k == 0: continue
+        try:
+            odds, p = fisher_exact([[w, n_wt - w], [k, n_cko - k]], alternative="two-sided")
+        except ValueError:
+            odds, p = 1.0, 1.0
+        rows.append({"derm_label": cn, "n": w + k, "n_WT": w, "n_cKO": k,
+                     "cko_frac": k / (w + k), "baseline_cko": base,
+                     "odds_ratio": float(odds), "fisher_p": float(p)})
+    return pd.DataFrame(rows).sort_values("cko_frac")
+
+
+def main():
+    OUT_DIR.mkdir(parents=True, exist_ok=True); CK_DIR.mkdir(parents=True, exist_ok=True)
+
+    print("[B] load replica", flush=True)
+    if not REPLICA_H5.exists():
+        raise FileNotFoundError(f"Run 103 first — {REPLICA_H5} missing")
+    a = ad.read_h5ad(REPLICA_H5)
+
+    dermal = a[a.obs["derm_label"].astype(str) != "non_dermal"].copy()
+    print(f"[B] dermal n={dermal.n_obs}", flush=True)
+    labels_str = dermal.obs["derm_label"].astype(str).values
+    classes = sorted(set(labels_str))
+    cls_ix = {c: i for i, c in enumerate(classes)}
+    y_all = np.array([cls_ix[c] for c in labels_str])
+    genotype = dermal.obs["genotype"].astype(str).values
+
+    # get embedding from replica (harmony-corrected PCA)
+    rep_key = dermal.uns.get("_replica_rep", "X_pca_harmony")
+    if rep_key not in dermal.obsm:
+        rep_key = "X_pca_harmony" if "X_pca_harmony" in dermal.obsm else "X_pca"
+    X_all = np.asarray(dermal.obsm[rep_key])
+    print(f"[B] using {rep_key} (d={X_all.shape[1]})", flush=True)
+
+    sample_ix = {s: i for i, s in enumerate(sorted(dermal.obs["sample"].astype(str).unique()))}
+    d_all = np.array([sample_ix[s] for s in dermal.obs["sample"].astype(str)])
+    total_counts = np.asarray(dermal.X.sum(axis=1)).ravel()
+    logc = np.log10(total_counts + 1); logc = (logc - logc.mean()) / (logc.std() + 1e-6)
+    aux_all = np.stack([np.zeros(dermal.n_obs, dtype=np.float32), logc.astype(np.float32)], axis=1)
+
+    print("[B] genotype-stratified 70/30 split", flush=True)
+    tr, te = genotype_stratified_split(labels_str, genotype, frac_train=TRAIN_FRAC, seed=SEED)
+    print(f"[B] train={tr.sum()} test={te.sum()}", flush=True)
+
+    manifest = pd.DataFrame({
+        "cell_id": dermal.obs_names.astype(str).values,
+        "derm_label": labels_str,
+        "genotype": genotype,
+        "split": np.where(tr, "train", np.where(te, "test", "unassigned")),
+    })
+    manifest.to_csv(OUT_DIR / "104_dingwall_derm_split_manifest.csv", index=False)
+
+    print("[B] train PANDA", flush=True)
+    model = train_panda(X_all[tr], y_all[tr], d_all[tr], aux_all[tr],
+                        n_classes=len(classes), n_datasets=len(sample_ix), ck_out=CK_DIR)
+
+    print("[B] infer on held-out", flush=True)
+    pred_ix, conf = infer(model, X_all[te], aux_all[te])
+    pred = pd.DataFrame({
+        "cell_id": dermal.obs_names.astype(str).values[te],
+        "derm_true": labels_str[te],
+        "derm_pred": [classes[p] for p in pred_ix],
+        "confidence": conf,
+        "genotype": genotype[te],
+    })
+    pred.to_csv(OUT_DIR / "104_dingwall_derm_predictions.csv", index=False)
+
+    # depletion — reported for TEST set only, using PANDA predictions
+    pred_ix_full = np.array([cls_ix[c] for c in pred["derm_pred"].values])
+    dep_pred = depletion_table(pred_ix_full, genotype[te], classes)
+    dep_true = depletion_table(y_all[te], genotype[te], classes)
+    dep_pred.to_csv(OUT_DIR / "104_dingwall_derm_depletion_pred.csv", index=False)
+    dep_true.to_csv(OUT_DIR / "104_dingwall_derm_depletion_true.csv", index=False)
+
+    d10_true = dep_true[dep_true["derm_label"] == "Derm10"].to_dict("records")
+    d10_pred = dep_pred[dep_pred["derm_label"] == "Derm10"].to_dict("records")
+    acc = float((pred_ix == y_all[te]).mean())
+
+    summary = {
+        "variant": "B_fully_supervised_replica_labels",
+        "n_dermal_total": int(dermal.n_obs),
+        "n_train": int(tr.sum()), "n_test": int(te.sum()),
+        "classes": classes,
+        "test_accuracy": acc,
+        "expected_paper_derm10": {"wt_pct": 1.99, "cko_pct": 0.08,
+                                   "or_approx": 24.5, "wt_n_approx": 346, "cko_n_approx": 7},
+        "test_derm10_true": d10_true,
+        "test_derm10_pred": d10_pred,
+        "test_depletion_true": dep_true.to_dict("records"),
+        "test_depletion_pred": dep_pred.to_dict("records"),
+    }
+    (OUT_DIR / "104_dingwall_derm_summary.json").write_text(json.dumps(summary, indent=2, default=str))
+    print(f"[B] done -> {OUT_DIR}/104_dingwall_derm_*", flush=True)
+
+
+if __name__ == "__main__":
+    main()

scripts/analysis/105_primary_eden_full_dermal.py

@@ -1,134 +1,137 @@
-"""primary EDEN discovery on the full dingwall-defined dermal set (not the panda-v3 fibroblast subset)."""
-from pathlib import Path
-import warnings, json, sys, numpy as np, pandas as pd, anndata as ad, scanpy as sc, scipy.sparse as sp
-from scipy.stats import fisher_exact
-warnings.filterwarnings("ignore"); sc.settings.verbosity = 0
-
-ROOT = Path("/home/bcheng/PRISM")
-REPLICA = ROOT / "data/processed/dingwall_replica/dingwall_replica.h5ad"
-DERM_MARKERS = ROOT / "data/external_labels/dingwall_supp/biorxiv_media-3.xlsx"
-TOP_N = 30
-
-# EDEN identity map from Dingwall paper + Data S2 CellChat
-EDEN_IDENTITY = {
-    10: "Secondary_EDEN_(Dingwall_cluster_20)",
-    2:  "Primary_EDEN_candidate_1_(Derm2_-_immediate_precursor)",
-    9:  "Primary_EDEN_candidate_2_(Derm9)",
-    6:  "EDEN-signalling_(Derm6)",
-    3:  "EDEN-signalling_(Derm3)",
-}
-
-
-def main():
-    print("[eden] loading replica dermal set (Dingwall-defined)", flush=True)
-    a = ad.read_h5ad(REPLICA)
-    # keep only cells the Seurat replica classified as belonging to Dingwall's dermal clusters
-    if "is_dermal_paper" in a.obs.columns:
-        dermal = a[a.obs["is_dermal_paper"] == True].copy()
-    elif "derm_label" in a.obs.columns:
-        dermal = a[a.obs["derm_label"] != "non_dermal"].copy()
-    else:
-        raise RuntimeError("no dermal indicator in replica")
-    print(f"[eden] Seurat replica dermal cells: {dermal.n_obs}", flush=True)
-
-    # verify our Derm-label distribution matches the replica
-    if "derm_label" in dermal.obs.columns:
-        print(f"[eden] Derm label distribution (from replica):", flush=True)
-        for k, v in dermal.obs["derm_label"].value_counts().sort_index().items():
-            print(f"    {k}: {v}", flush=True)
-
-    # load Data S1C panels
-    print(f"\n[eden] loading Data S1C marker panels", flush=True)
-    df = pd.read_excel(DERM_MARKERS)
-    df = df.sort_values(["cluster", "avg_log2FC"], ascending=[True, False])
-    panels = {}
-    for cl in sorted(df["cluster"].unique()):
-        genes = df[df["cluster"] == cl].head(TOP_N)["gene"].astype(str).tolist()
-        panels[int(cl)] = genes
-
-    # replica may or may not have log1p applied; reset from counts layer if present
-    if "counts" in dermal.layers:
-        dermal.X = dermal.layers["counts"]
-    if dermal.X.max() > 30:  # raw counts
-        sc.pp.normalize_total(dermal, target_sum=1e4); sc.pp.log1p(dermal)
-
-    # score each cell on all 12 Derm identity panels
-    print(f"\n[eden] scoring cells on all 12 Derm panels (top-30 markers each)", flush=True)
-    for cl, genes in panels.items():
-        present = [g for g in genes if g in dermal.var_names]
-        if len(present) < 3:
-            dermal.obs[f"derm{cl}_score"] = 0.0
-            continue
-        sc.tl.score_genes(dermal, gene_list=present, score_name=f"derm{cl}_score",
-                          random_state=0, use_raw=False)
-
-    # global baseline
-    n_wt = int((dermal.obs["genotype"] == "WT").sum())
-    n_cko = int((dermal.obs["genotype"] == "En1-cKO").sum())
-    baseline = n_cko / max(n_wt + n_cko, 1)
-    print(f"\n[eden] baseline: WT={n_wt} cKO={n_cko} (baseline cKO frac = {baseline:.3f})", flush=True)
-
-    # use the replica's derm_label directly, not argmax of scores
-    rows = []
-    print(f"\n[eden] per-Derm Fisher exact on replica-assigned identities:", flush=True)
-    for cl in sorted(panels.keys()):
-        derm_label = f"Derm{cl}"
-        if derm_label not in dermal.obs["derm_label"].values:
-            continue
-        sub = dermal[dermal.obs["derm_label"] == derm_label]
-        n_wt_c = int((sub.obs["genotype"] == "WT").sum())
-        n_cko_c = int((sub.obs["genotype"] == "En1-cKO").sum())
-        if n_wt_c + n_cko_c == 0:
-            continue
-        cko_frac = n_cko_c / (n_wt_c + n_cko_c)
-        n_wt_else = n_wt - n_wt_c
-        n_cko_else = n_cko - n_cko_c
-        try:
-            odds, p_f = fisher_exact([[n_wt_c, n_wt_else], [n_cko_c, n_cko_else]],
-                                     alternative="two-sided")
-        except ValueError:
-            odds, p_f = 1.0, 1.0
-        rows.append({
-            "derm_id": cl,
-            "identity": EDEN_IDENTITY.get(cl, "other"),
-            "n_cells": n_wt_c + n_cko_c,
-            "n_WT": n_wt_c, "n_cKO": n_cko_c,
-            "cko_frac": cko_frac,
-            "baseline_cko_frac": baseline,
-            "cko_delta": cko_frac - baseline,
-            "wt_enrichment_odds_ratio": float(1.0/odds) if odds > 0 else None,
-            "fisher_p_two_sided": float(p_f),
-            "depletion_direction": "cKO-depleted" if cko_frac < baseline else "cKO-enriched",
-            "top10_markers_dingwall_S1C": ", ".join(panels[cl][:10]),
-        })
-
-    result_df = pd.DataFrame(rows).sort_values("cko_delta")
-    out = ROOT / "discovery/pan_skin/marker"
-    out.mkdir(parents=True, exist_ok=True)
-    result_df.to_csv(out / "105_primary_eden_full_dermal.csv", index=False)
-
-    print(f"\n{'Derm':<8}{'Identity':<50}{'n':<7}{'WT':<6}{'cKO':<6}{'cKO_frac':<10}"
-          f"{'OR (WT enrich)':<16}{'Fisher p':<12}", flush=True)
-    print("-" * 130, flush=True)
-    for _, r in result_df.iterrows():
-        print(f"Derm{r['derm_id']:<5}{r['identity'][:47]:<50}{r['n_cells']:<7}"
-              f"{r['n_WT']:<6}{r['n_cKO']:<6}{r['cko_frac']:<10.3f}"
-              f"{r['wt_enrichment_odds_ratio']:<16.2f}{r['fisher_p_two_sided']:<12.2e}", flush=True)
-
-    # summary json
-    summary = {
-        "target": "Dingwall_GSE220977",
-        "method": "Seurat-replica-identified 14,251 dermal cells (Dingwall clusters {0,1,3,4,5,8,11,20}); "
-                  "per-Derm identity Fisher-exact cKO depletion using replica-assigned Derm labels "
-                  "(mapped via Jaccard on top-50 markers to Dingwall Data S1C)",
-        "n_dermal_cells_total": int(dermal.n_obs),
-        "baseline_cko_frac": float(baseline),
-        "n_WT_dermal": n_wt, "n_cKO_dermal": n_cko,
-        "per_derm": rows,
-    }
-    (out / "105_primary_eden_full_dermal.json").write_text(json.dumps(summary, indent=2, default=str))
-    print(f"\n[eden] wrote {out}/105_primary_eden_full_dermal.{{csv,json}}", flush=True)
-
-
-if __name__ == "__main__":
-    main()
+"""primary EDEN discovery on the full dingwall-defined dermal set (not the panda-v3 fibroblast subset)."""
+from pathlib import Path
+import warnings, json, sys, numpy as np, pandas as pd, anndata as ad, scanpy as sc, scipy.sparse as sp
+from scipy.stats import fisher_exact
+warnings.filterwarnings("ignore"); sc.settings.verbosity = 0
+
+import os as _os
+from pathlib import Path as _Path
+PANDA_ROOT = _Path(_os.environ.get("PANDA_ROOT", str(_Path(__file__).resolve().parents[2])))
+ROOT = Path(str(PANDA_ROOT))
+REPLICA = ROOT / "data/processed/dingwall_replica/dingwall_replica.h5ad"
+DERM_MARKERS = ROOT / "data/external_labels/dingwall_supp/biorxiv_media-3.xlsx"
+TOP_N = 30
+
+# EDEN identity map from Dingwall paper + Data S2 CellChat
+EDEN_IDENTITY = {
+    10: "Secondary_EDEN_(Dingwall_cluster_20)",
+    2:  "Primary_EDEN_candidate_1_(Derm2_-_immediate_precursor)",
+    9:  "Primary_EDEN_candidate_2_(Derm9)",
+    6:  "EDEN-signalling_(Derm6)",
+    3:  "EDEN-signalling_(Derm3)",
+}
+
+
+def main():
+    print("[eden] loading replica dermal set (Dingwall-defined)", flush=True)
+    a = ad.read_h5ad(REPLICA)
+    # keep only cells the Seurat replica classified as belonging to Dingwall's dermal clusters
+    if "is_dermal_paper" in a.obs.columns:
+        dermal = a[a.obs["is_dermal_paper"] == True].copy()
+    elif "derm_label" in a.obs.columns:
+        dermal = a[a.obs["derm_label"] != "non_dermal"].copy()
+    else:
+        raise RuntimeError("no dermal indicator in replica")
+    print(f"[eden] Seurat replica dermal cells: {dermal.n_obs}", flush=True)
+
+    # verify our Derm-label distribution matches the replica
+    if "derm_label" in dermal.obs.columns:
+        print(f"[eden] Derm label distribution (from replica):", flush=True)
+        for k, v in dermal.obs["derm_label"].value_counts().sort_index().items():
+            print(f"    {k}: {v}", flush=True)
+
+    # load Data S1C panels
+    print(f"\n[eden] loading Data S1C marker panels", flush=True)
+    df = pd.read_excel(DERM_MARKERS)
+    df = df.sort_values(["cluster", "avg_log2FC"], ascending=[True, False])
+    panels = {}
+    for cl in sorted(df["cluster"].unique()):
+        genes = df[df["cluster"] == cl].head(TOP_N)["gene"].astype(str).tolist()
+        panels[int(cl)] = genes
+
+    # replica may or may not have log1p applied; reset from counts layer if present
+    if "counts" in dermal.layers:
+        dermal.X = dermal.layers["counts"]
+    if dermal.X.max() > 30:  # raw counts
+        sc.pp.normalize_total(dermal, target_sum=1e4); sc.pp.log1p(dermal)
+
+    # score each cell on all 12 Derm identity panels
+    print(f"\n[eden] scoring cells on all 12 Derm panels (top-30 markers each)", flush=True)
+    for cl, genes in panels.items():
+        present = [g for g in genes if g in dermal.var_names]
+        if len(present) < 3:
+            dermal.obs[f"derm{cl}_score"] = 0.0
+            continue
+        sc.tl.score_genes(dermal, gene_list=present, score_name=f"derm{cl}_score",
+                          random_state=0, use_raw=False)
+
+    # global baseline
+    n_wt = int((dermal.obs["genotype"] == "WT").sum())
+    n_cko = int((dermal.obs["genotype"] == "En1-cKO").sum())
+    baseline = n_cko / max(n_wt + n_cko, 1)
+    print(f"\n[eden] baseline: WT={n_wt} cKO={n_cko} (baseline cKO frac = {baseline:.3f})", flush=True)
+
+    # use the replica's derm_label directly, not argmax of scores
+    rows = []
+    print(f"\n[eden] per-Derm Fisher exact on replica-assigned identities:", flush=True)
+    for cl in sorted(panels.keys()):
+        derm_label = f"Derm{cl}"
+        if derm_label not in dermal.obs["derm_label"].values:
+            continue
+        sub = dermal[dermal.obs["derm_label"] == derm_label]
+        n_wt_c = int((sub.obs["genotype"] == "WT").sum())
+        n_cko_c = int((sub.obs["genotype"] == "En1-cKO").sum())
+        if n_wt_c + n_cko_c == 0:
+            continue
+        cko_frac = n_cko_c / (n_wt_c + n_cko_c)
+        n_wt_else = n_wt - n_wt_c
+        n_cko_else = n_cko - n_cko_c
+        try:
+            odds, p_f = fisher_exact([[n_wt_c, n_wt_else], [n_cko_c, n_cko_else]],
+                                     alternative="two-sided")
+        except ValueError:
+            odds, p_f = 1.0, 1.0
+        rows.append({
+            "derm_id": cl,
+            "identity": EDEN_IDENTITY.get(cl, "other"),
+            "n_cells": n_wt_c + n_cko_c,
+            "n_WT": n_wt_c, "n_cKO": n_cko_c,
+            "cko_frac": cko_frac,
+            "baseline_cko_frac": baseline,
+            "cko_delta": cko_frac - baseline,
+            "wt_enrichment_odds_ratio": float(1.0/odds) if odds > 0 else None,
+            "fisher_p_two_sided": float(p_f),
+            "depletion_direction": "cKO-depleted" if cko_frac < baseline else "cKO-enriched",
+            "top10_markers_dingwall_S1C": ", ".join(panels[cl][:10]),
+        })
+
+    result_df = pd.DataFrame(rows).sort_values("cko_delta")
+    out = ROOT / "discovery/pan_skin/marker"
+    out.mkdir(parents=True, exist_ok=True)
+    result_df.to_csv(out / "105_primary_eden_full_dermal.csv", index=False)
+
+    print(f"\n{'Derm':<8}{'Identity':<50}{'n':<7}{'WT':<6}{'cKO':<6}{'cKO_frac':<10}"
+          f"{'OR (WT enrich)':<16}{'Fisher p':<12}", flush=True)
+    print("-" * 130, flush=True)
+    for _, r in result_df.iterrows():
+        print(f"Derm{r['derm_id']:<5}{r['identity'][:47]:<50}{r['n_cells']:<7}"
+              f"{r['n_WT']:<6}{r['n_cKO']:<6}{r['cko_frac']:<10.3f}"
+              f"{r['wt_enrichment_odds_ratio']:<16.2f}{r['fisher_p_two_sided']:<12.2e}", flush=True)
+
+    # summary json
+    summary = {
+        "target": "Dingwall_GSE220977",
+        "method": "Seurat-replica-identified 14,251 dermal cells (Dingwall clusters {0,1,3,4,5,8,11,20}); "
+                  "per-Derm identity Fisher-exact cKO depletion using replica-assigned Derm labels "
+                  "(mapped via Jaccard on top-50 markers to Dingwall Data S1C)",
+        "n_dermal_cells_total": int(dermal.n_obs),
+        "baseline_cko_frac": float(baseline),
+        "n_WT_dermal": n_wt, "n_cKO_dermal": n_cko,
+        "per_derm": rows,
+    }
+    (out / "105_primary_eden_full_dermal.json").write_text(json.dumps(summary, indent=2, default=str))
+    print(f"\n[eden] wrote {out}/105_primary_eden_full_dermal.{{csv,json}}", flush=True)
+
+
+if __name__ == "__main__":
+    main()

scripts/analysis/106_melanoblast_neural_crest.py

@@ -1,138 +1,141 @@
-"""test whether En1-cKO melanoblasts with sweat/eda derepression revert to neural crest vs disrupted melanocyte."""
-from __future__ import annotations
-
-import json
-import warnings
-from pathlib import Path
-
-import anndata as ad
-import numpy as np
-import pandas as pd
-import scanpy as sc
-from scipy.stats import mannwhitneyu
-
-warnings.filterwarnings("ignore")
-sc.settings.verbosity = 0
-
-ROOT = Path("/home/bcheng/PRISM")
-OUT = ROOT / "discovery/pan_skin/marker"
-OUT.mkdir(parents=True, exist_ok=True)
-
-RAW = ROOT / "data/raw/GSE220977_combined.h5ad"
-PRED = ROOT / "discovery/pan_skin/marker/dingwall_predictions.csv"
-CKO_GSMS = {"GSM6833482", "GSM6833483"}
-WT_GSMS = {"GSM6833478", "GSM6833479", "GSM6833480", "GSM6833481"}
-
-MODULES = {
-    "Neural_crest":       ["Sox10", "Sox9", "Sox2", "Pax3", "Foxd3", "Nes", "Tfap2a"],
-    "Melanogenesis_late": ["Tyrp1", "Slc45a2", "Oca2", "Gpnmb", "Pmel", "Silv",
-                           "Mlph", "Rab27a", "Melana"],
-    "MITF_regulon":       ["Mitf", "Dct", "Tyr", "Pmel", "Mlana", "Tyrp1",
-                           "Slc24a5", "Slc45a2", "Sox10", "Pax3", "Kit", "Ednrb"],
-    "Sweat_gland":        ["Foxi3", "Foxa1", "En1", "Krt8", "Krt18", "Krt19",
-                           "Muc5b", "Aqp5", "Cutl1"],
-    "Eda_ectodysplasin":  ["Eda", "Edar", "Edaradd", "Nfkb1", "Nfkb2", "Rela"],
-}
-
-
-def score(sub, name, genes):
-    present = [g for g in genes if g in sub.var_names]
-    if not present:
-        sub.obs[f"pw_{name}"] = 0.0
-        return
-    sc.tl.score_genes(sub, gene_list=present, score_name=f"pw_{name}",
-                      random_state=0, use_raw=False)
-
-
-def main():
-    print("[load] Dingwall + predictions", flush=True)
-    a = ad.read_h5ad(RAW)
-    pred = pd.read_csv(PRED)
-    pred_map = dict(zip(pred["cell_id"].astype(str), pred["pred_label"]))
-    a.obs["pred_label"] = pd.Categorical(
-        [pred_map.get(c, "unknown") for c in a.obs_names])
-    samp = a.obs["sample"].astype(str)
-    a.obs["group"] = np.where(samp.isin(list(CKO_GSMS)), "En1-cKO",
-                     np.where(samp.isin(list(WT_GSMS)), "WT", "other"))
-    a = a[a.obs["group"].isin(["En1-cKO", "WT"])].copy()
-    sub = a[a.obs["pred_label"].astype(str) == "melanoblast"].copy()
-    print(f"[load] {sub.n_obs} melanoblast cells "
-          f"(cKO={(sub.obs['group']=='En1-cKO').sum()}, "
-          f"WT={(sub.obs['group']=='WT').sum()})", flush=True)
-
-    sc.pp.normalize_total(sub, target_sum=1e4)
-    sc.pp.log1p(sub)
-
-    for name, genes in MODULES.items():
-        score(sub, name, genes)
-
-    sub.obs["derepression"] = (sub.obs["pw_Sweat_gland"].astype(float)
-                               + sub.obs["pw_Eda_ectodysplasin"].astype(float))
-
-    df = sub.obs[[
-        "group", "pw_Neural_crest", "pw_Melanogenesis_late", "pw_MITF_regulon",
-        "pw_Sweat_gland", "pw_Eda_ectodysplasin", "derepression"
-    ]].copy()
-    df.reset_index().rename(columns={"index": "cell_id"}).to_csv(
-        OUT / "106_melanoblast_nc_scores.csv", index=False)
-
-    # baseline cKO vs WT per module
-    tests = {}
-    for m in ["Neural_crest", "Melanogenesis_late", "MITF_regulon",
-              "Sweat_gland", "Eda_ectodysplasin"]:
-        v1 = df.loc[df["group"] == "En1-cKO", f"pw_{m}"].astype(float).values
-        v0 = df.loc[df["group"] == "WT",       f"pw_{m}"].astype(float).values
-        _, p = mannwhitneyu(v1, v0, alternative="two-sided")
-        tests[m] = {
-            "delta_cKO_minus_WT": float(v1.mean() - v0.mean()),
-            "mean_cKO": float(v1.mean()), "mean_WT": float(v0.mean()),
-            "n_cKO": int(len(v1)), "n_WT": int(len(v0)),
-            "mannu_p": float(p),
-        }
-
-    # within cKO, split by derepression quartile
-    cko = df[df["group"] == "En1-cKO"].copy()
-    q1 = cko["derepression"].quantile(0.25)
-    q4 = cko["derepression"].quantile(0.75)
-    top = cko[cko["derepression"] >= q4]
-    bot = cko[cko["derepression"] <= q1]
-    print(f"[q]  top-derep cKO: n={len(top)}, bot-derep cKO: n={len(bot)}",
-          flush=True)
-
-    within = {}
-    for m in ["Neural_crest", "Melanogenesis_late", "MITF_regulon"]:
-        vt = top[f"pw_{m}"].astype(float).values
-        vb = bot[f"pw_{m}"].astype(float).values
-        _, p = mannwhitneyu(vt, vb, alternative="two-sided")
-        within[m] = {
-            "mean_top_derep": float(vt.mean()),
-            "mean_bot_derep": float(vb.mean()),
-            "delta_top_minus_bot": float(vt.mean() - vb.mean()),
-            "n_top": int(len(vt)), "n_bot": int(len(vb)),
-            "mannu_p": float(p),
-        }
-
-    nc_delta = within["Neural_crest"]["delta_top_minus_bot"]
-    mel_delta = within["Melanogenesis_late"]["delta_top_minus_bot"]
-    verdict = (
-        "novel_neural_crest_reversion" if nc_delta > 0.03 and mel_delta > -0.02
-        else "basic_melanocyte_disruption" if nc_delta < 0.01 and mel_delta < -0.02
-        else "mixed_or_orthogonal"
-    )
-    summary = {
-        "n_melanoblast_cells":  int(sub.n_obs),
-        "n_cKO_melanoblasts":   int((df["group"] == "En1-cKO").sum()),
-        "n_WT_melanoblasts":    int((df["group"] == "WT").sum()),
-        "baseline_cKO_vs_WT":   tests,
-        "within_cKO_top_vs_bot_derepression_quartile": within,
-        "verdict":              verdict,
-    }
-    with open(OUT / "106_melanoblast_nc_summary.json", "w") as f:
-        json.dump(summary, f, indent=2)
-
-    print("[done] verdict:", verdict, flush=True)
-    print(json.dumps(summary, indent=2))
-
-
-if __name__ == "__main__":
-    main()
+"""test whether En1-cKO melanoblasts with sweat/eda derepression revert to neural crest vs disrupted melanocyte."""
+from __future__ import annotations
+
+import json
+import warnings
+from pathlib import Path
+
+import anndata as ad
+import numpy as np
+import pandas as pd
+import scanpy as sc
+from scipy.stats import mannwhitneyu
+
+warnings.filterwarnings("ignore")
+sc.settings.verbosity = 0
+
+import os as _os
+from pathlib import Path as _Path
+PANDA_ROOT = _Path(_os.environ.get("PANDA_ROOT", str(_Path(__file__).resolve().parents[2])))
+ROOT = Path(str(PANDA_ROOT))
+OUT = ROOT / "discovery/pan_skin/marker"
+OUT.mkdir(parents=True, exist_ok=True)
+
+RAW = ROOT / "data/raw/GSE220977_combined.h5ad"
+PRED = ROOT / "discovery/pan_skin/marker/dingwall_predictions.csv"
+CKO_GSMS = {"GSM6833482", "GSM6833483"}
+WT_GSMS = {"GSM6833478", "GSM6833479", "GSM6833480", "GSM6833481"}
+
+MODULES = {
+    "Neural_crest":       ["Sox10", "Sox9", "Sox2", "Pax3", "Foxd3", "Nes", "Tfap2a"],
+    "Melanogenesis_late": ["Tyrp1", "Slc45a2", "Oca2", "Gpnmb", "Pmel", "Silv",
+                           "Mlph", "Rab27a", "Melana"],
+    "MITF_regulon":       ["Mitf", "Dct", "Tyr", "Pmel", "Mlana", "Tyrp1",
+                           "Slc24a5", "Slc45a2", "Sox10", "Pax3", "Kit", "Ednrb"],
+    "Sweat_gland":        ["Foxi3", "Foxa1", "En1", "Krt8", "Krt18", "Krt19",
+                           "Muc5b", "Aqp5", "Cutl1"],
+    "Eda_ectodysplasin":  ["Eda", "Edar", "Edaradd", "Nfkb1", "Nfkb2", "Rela"],
+}
+
+
+def score(sub, name, genes):
+    present = [g for g in genes if g in sub.var_names]
+    if not present:
+        sub.obs[f"pw_{name}"] = 0.0
+        return
+    sc.tl.score_genes(sub, gene_list=present, score_name=f"pw_{name}",
+                      random_state=0, use_raw=False)
+
+
+def main():
+    print("[load] Dingwall + predictions", flush=True)
+    a = ad.read_h5ad(RAW)
+    pred = pd.read_csv(PRED)
+    pred_map = dict(zip(pred["cell_id"].astype(str), pred["pred_label"]))
+    a.obs["pred_label"] = pd.Categorical(
+        [pred_map.get(c, "unknown") for c in a.obs_names])
+    samp = a.obs["sample"].astype(str)
+    a.obs["group"] = np.where(samp.isin(list(CKO_GSMS)), "En1-cKO",
+                     np.where(samp.isin(list(WT_GSMS)), "WT", "other"))
+    a = a[a.obs["group"].isin(["En1-cKO", "WT"])].copy()
+    sub = a[a.obs["pred_label"].astype(str) == "melanoblast"].copy()
+    print(f"[load] {sub.n_obs} melanoblast cells "
+          f"(cKO={(sub.obs['group']=='En1-cKO').sum()}, "
+          f"WT={(sub.obs['group']=='WT').sum()})", flush=True)
+
+    sc.pp.normalize_total(sub, target_sum=1e4)
+    sc.pp.log1p(sub)
+
+    for name, genes in MODULES.items():
+        score(sub, name, genes)
+
+    sub.obs["derepression"] = (sub.obs["pw_Sweat_gland"].astype(float)
+                               + sub.obs["pw_Eda_ectodysplasin"].astype(float))
+
+    df = sub.obs[[
+        "group", "pw_Neural_crest", "pw_Melanogenesis_late", "pw_MITF_regulon",
+        "pw_Sweat_gland", "pw_Eda_ectodysplasin", "derepression"
+    ]].copy()
+    df.reset_index().rename(columns={"index": "cell_id"}).to_csv(
+        OUT / "106_melanoblast_nc_scores.csv", index=False)
+
+    # baseline cKO vs WT per module
+    tests = {}
+    for m in ["Neural_crest", "Melanogenesis_late", "MITF_regulon",
+              "Sweat_gland", "Eda_ectodysplasin"]:
+        v1 = df.loc[df["group"] == "En1-cKO", f"pw_{m}"].astype(float).values
+        v0 = df.loc[df["group"] == "WT",       f"pw_{m}"].astype(float).values
+        _, p = mannwhitneyu(v1, v0, alternative="two-sided")
+        tests[m] = {
+            "delta_cKO_minus_WT": float(v1.mean() - v0.mean()),
+            "mean_cKO": float(v1.mean()), "mean_WT": float(v0.mean()),
+            "n_cKO": int(len(v1)), "n_WT": int(len(v0)),
+            "mannu_p": float(p),
+        }
+
+    # within cKO, split by derepression quartile
+    cko = df[df["group"] == "En1-cKO"].copy()
+    q1 = cko["derepression"].quantile(0.25)
+    q4 = cko["derepression"].quantile(0.75)
+    top = cko[cko["derepression"] >= q4]
+    bot = cko[cko["derepression"] <= q1]
+    print(f"[q]  top-derep cKO: n={len(top)}, bot-derep cKO: n={len(bot)}",
+          flush=True)
+
+    within = {}
+    for m in ["Neural_crest", "Melanogenesis_late", "MITF_regulon"]:
+        vt = top[f"pw_{m}"].astype(float).values
+        vb = bot[f"pw_{m}"].astype(float).values
+        _, p = mannwhitneyu(vt, vb, alternative="two-sided")
+        within[m] = {
+            "mean_top_derep": float(vt.mean()),
+            "mean_bot_derep": float(vb.mean()),
+            "delta_top_minus_bot": float(vt.mean() - vb.mean()),
+            "n_top": int(len(vt)), "n_bot": int(len(vb)),
+            "mannu_p": float(p),
+        }
+
+    nc_delta = within["Neural_crest"]["delta_top_minus_bot"]
+    mel_delta = within["Melanogenesis_late"]["delta_top_minus_bot"]
+    verdict = (
+        "novel_neural_crest_reversion" if nc_delta > 0.03 and mel_delta > -0.02
+        else "basic_melanocyte_disruption" if nc_delta < 0.01 and mel_delta < -0.02
+        else "mixed_or_orthogonal"
+    )
+    summary = {
+        "n_melanoblast_cells":  int(sub.n_obs),
+        "n_cKO_melanoblasts":   int((df["group"] == "En1-cKO").sum()),
+        "n_WT_melanoblasts":    int((df["group"] == "WT").sum()),
+        "baseline_cKO_vs_WT":   tests,
+        "within_cKO_top_vs_bot_derepression_quartile": within,
+        "verdict":              verdict,
+    }
+    with open(OUT / "106_melanoblast_nc_summary.json", "w") as f:
+        json.dump(summary, f, indent=2)
+
+    print("[done] verdict:", verdict, flush=True)
+    print(json.dumps(summary, indent=2))
+
+
+if __name__ == "__main__":
+    main()

scripts/analysis/107_dingwall_class_deg_count.py

@@ -1,116 +1,122 @@
-"""rank dingwall cell types by En1-cKO vs WT DEG count (|log2FC|>1, padj<0.05)."""
-from __future__ import annotations
-
-import json
-import warnings
-from pathlib import Path
-
-import anndata as ad
-import numpy as np
-import pandas as pd
-import scanpy as sc
-
-warnings.filterwarnings("ignore")
-sc.settings.verbosity = 0
-
-ROOT = Path("/home/bcheng/PRISM")
-OUT = ROOT / "discovery/pan_skin/marker"
-OUT.mkdir(parents=True, exist_ok=True)
-
-RAW  = ROOT / "data/raw/GSE220977_combined.h5ad"
-PRED = ROOT / "discovery/pan_skin/marker/dingwall_predictions.csv"
-CKO_GSMS = {"GSM6833482", "GSM6833483"}
-WT_GSMS  = {"GSM6833478", "GSM6833479", "GSM6833480", "GSM6833481"}
-MIN_PER_GROUP = 15
-LFC_THRESH = 1.0
-PADJ_THRESH = 0.05
-
-
-def main():
-    print("[load]", flush=True)
-    a = ad.read_h5ad(RAW)
-    pred = pd.read_csv(PRED)
-    pred_map = dict(zip(pred["cell_id"].astype(str), pred["pred_label"]))
-    a.obs["pred_label"] = pd.Categorical(
-        [pred_map.get(c, "unknown") for c in a.obs_names])
-    samp = a.obs["sample"].astype(str)
-    a.obs["group"] = np.where(samp.isin(list(CKO_GSMS)), "En1-cKO",
-                     np.where(samp.isin(list(WT_GSMS)), "WT", "other"))
-    a = a[a.obs["group"].isin(["En1-cKO", "WT"])].copy()
-    print(f"[load] {a.n_obs} cells, {a.n_vars} genes", flush=True)
-
-    sc.pp.normalize_total(a, target_sum=1e4)
-    sc.pp.log1p(a)
-
-    rows = []
-    for cls in sorted(a.obs["pred_label"].astype(str).unique()):
-        mask = (a.obs["pred_label"].astype(str) == cls)
-        n_cko = int((mask & (a.obs["group"] == "En1-cKO")).sum())
-        n_wt  = int((mask & (a.obs["group"] == "WT")).sum())
-        if n_cko < MIN_PER_GROUP or n_wt < MIN_PER_GROUP:
-            print(f"[skip] {cls}: n_cKO={n_cko} n_WT={n_wt}", flush=True)
-            continue
-        sub = a[mask].copy()
-        sub.obs["group"] = pd.Categorical(sub.obs["group"].values,
-                                          categories=["En1-cKO", "WT"])
-        # reference=WT so positive LFC means up in cKO
-        try:
-            sc.tl.rank_genes_groups(sub, "group", reference="WT",
-                                    groups=["En1-cKO"], method="wilcoxon",
-                                    n_genes=sub.n_vars, use_raw=False,
-                                    pts=True)
-            rgg = sub.uns["rank_genes_groups"]
-            df = pd.DataFrame({
-                "gene":  [x[0] for x in rgg["names"]],
-                "lfc":   [x[0] for x in rgg["logfoldchanges"]],
-                "padj":  [x[0] for x in rgg["pvals_adj"]],
-            })
-            up   = int(((df["lfc"] >  LFC_THRESH) & (df["padj"] < PADJ_THRESH)).sum())
-            down = int(((df["lfc"] < -LFC_THRESH) & (df["padj"] < PADJ_THRESH)).sum())
-            total = up + down
-            top_up   = df[(df["lfc"] >  LFC_THRESH) & (df["padj"] < PADJ_THRESH)] \
-                        .sort_values("lfc", ascending=False)["gene"].head(10).tolist()
-            top_down = df[(df["lfc"] < -LFC_THRESH) & (df["padj"] < PADJ_THRESH)] \
-                        .sort_values("lfc", ascending=True)["gene"].head(10).tolist()
-        except Exception as e:
-            print(f"[fail] {cls}: {e}", flush=True)
-            continue
-        rows.append({
-            "class":    cls,
-            "n_cKO":    n_cko,
-            "n_WT":     n_wt,
-            "n_DEG":    total,
-            "n_up":     up,
-            "n_down":   down,
-            "top_up":   ";".join(top_up),
-            "top_down": ";".join(top_down),
-        })
-        print(f"[ok] {cls}: n_cKO={n_cko} n_WT={n_wt} DEG={total} "
-              f"(up={up}, down={down})", flush=True)
-
-    df_out = pd.DataFrame(rows).sort_values("n_DEG", ascending=False)
-    df_out.to_csv(OUT / "107_dingwall_class_deg_count.csv", index=False)
-    print("\n[rank] classes ordered by DEG count (|LFC|>1 padj<0.05):")
-    print(df_out[["class", "n_cKO", "n_WT", "n_DEG", "n_up", "n_down"]]
-          .to_string(index=False))
-
-    if not df_out.empty:
-        winner = df_out.iloc[0]
-        summary = {
-            "lfc_threshold": LFC_THRESH, "padj_threshold": PADJ_THRESH,
-            "n_classes_tested": int(len(df_out)),
-            "top_class":  str(winner["class"]),
-            "top_n_DEG":  int(winner["n_DEG"]),
-            "top_up":     winner["top_up"],
-            "top_down":   winner["top_down"],
-            "ranking":    df_out[["class", "n_DEG"]].to_dict("records"),
-        }
-    else:
-        summary = {"error": "no eligible classes"}
-    with open(OUT / "107_dingwall_class_deg_count.json", "w") as f:
-        json.dump(summary, f, indent=2)
-    print("[done]", flush=True)
-
-
-if __name__ == "__main__":
-    main()
+"""rank dingwall cell types by En1-cKO vs WT DEG count (|log2FC|>1, padj<0.05)."""
+from __future__ import annotations
+
+import json
+import warnings
+from pathlib import Path
+
+import anndata as ad
+import numpy as np
+import pandas as pd
+import scanpy as sc
+
+warnings.filterwarnings("ignore")
+sc.settings.verbosity = 0
+
+import os as _os
+from pathlib import Path as _Path
+PANDA_ROOT = _Path(_os.environ.get("PANDA_ROOT", str(_Path(__file__).resolve().parents[2])))
+ROOT = Path(str(PANDA_ROOT))
+OUT = ROOT / "discovery/pan_skin/marker"
+OUT.mkdir(parents=True, exist_ok=True)
+
+RAW  = ROOT / "data/raw/GSE220977_combined.h5ad"
+PRED = ROOT / "discovery/pan_skin/marker/dingwall_predictions.csv"
+CKO_GSMS = {"GSM6833482", "GSM6833483"}
+WT_GSMS  = {"GSM6833478", "GSM6833479", "GSM6833480", "GSM6833481"}
+MIN_PER_GROUP = 15
+LFC_THRESH = 1.0
+PADJ_THRESH = 0.05
+
+
+def main():
+    print("[load]", flush=True)
+    a = ad.read_h5ad(RAW)
+    pred = pd.read_csv(PRED)
+    pred_map = dict(zip(pred["cell_id"].astype(str), pred["pred_label"]))
+    a.obs["pred_label"] = pd.Categorical(
+        [pred_map.get(c, "unknown") for c in a.obs_names])
+    samp = a.obs["sample"].astype(str)
+    a.obs["group"] = np.where(samp.isin(list(CKO_GSMS)), "En1-cKO",
+                     np.where(samp.isin(list(WT_GSMS)), "WT", "other"))
+    a = a[a.obs["group"].isin(["En1-cKO", "WT"])].copy()
+    print(f"[load] {a.n_obs} cells, {a.n_vars} genes", flush=True)
+
+    sc.pp.normalize_total(a, target_sum=1e4)
+    sc.pp.log1p(a)
+
+    rows = []
+    for cls in sorted(a.obs["pred_label"].astype(str).unique()):
+        mask = (a.obs["pred_label"].astype(str) == cls)
+        n_cko = int((mask & (a.obs["group"] == "En1-cKO")).sum())
+        n_wt  = int((mask & (a.obs["group"] == "WT")).sum())
+        if n_cko < MIN_PER_GROUP or n_wt < MIN_PER_GROUP:
+            print(f"[skip] {cls}: n_cKO={n_cko} n_WT={n_wt}", flush=True)
+            continue
+        sub = a[mask].copy()
+        sub.obs["group"] = pd.Categorical(sub.obs["group"].values,
+                                          categories=["En1-cKO", "WT"])
+        # reference=WT so positive LFC means up in cKO
+        try:
+            sc.tl.rank_genes_groups(sub, "group", reference="WT",
+                                    groups=["En1-cKO"], method="wilcoxon",
+                                    n_genes=sub.n_vars, use_raw=False,
+                                    pts=True)
+            rgg = sub.uns["rank_genes_groups"]
+            df = pd.DataFrame({
+                "gene":  [x[0] for x in rgg["names"]],
+                "lfc":   [x[0] for x in rgg["logfoldchanges"]],
+                "padj":  [x[0] for x in rgg["pvals_adj"]],
+            })
+            up   = int(((df["lfc"] >  LFC_THRESH) & (df["padj"] < PADJ_THRESH)).sum())
+            down = int(((df["lfc"] < -LFC_THRESH) & (df["padj"] < PADJ_THRESH)).sum())
+            total = up + down
+            # persist the COMPLETE significant sets. An earlier .head(10) meant
+            # n_DEG could exceed the stored gene list (HF-placode: 12 vs 10),
+            # so two of the reported DEGs were never written to any artefact.
+            top_up   = df[(df["lfc"] >  LFC_THRESH) & (df["padj"] < PADJ_THRESH)] \
+                        .sort_values("lfc", ascending=False)["gene"].tolist()
+            top_down = df[(df["lfc"] < -LFC_THRESH) & (df["padj"] < PADJ_THRESH)] \
+                        .sort_values("lfc", ascending=True)["gene"].tolist()
+        except Exception as e:
+            print(f"[fail] {cls}: {e}", flush=True)
+            continue
+        rows.append({
+            "class":    cls,
+            "n_cKO":    n_cko,
+            "n_WT":     n_wt,
+            "n_DEG":    total,
+            "n_up":     up,
+            "n_down":   down,
+            "top_up":   ";".join(top_up),
+            "top_down": ";".join(top_down),
+        })
+        print(f"[ok] {cls}: n_cKO={n_cko} n_WT={n_wt} DEG={total} "
+              f"(up={up}, down={down})", flush=True)
+
+    df_out = pd.DataFrame(rows).sort_values("n_DEG", ascending=False)
+    df_out.to_csv(OUT / "107_dingwall_class_deg_count.csv", index=False)
+    print("\n[rank] classes ordered by DEG count (|LFC|>1 padj<0.05):")
+    print(df_out[["class", "n_cKO", "n_WT", "n_DEG", "n_up", "n_down"]]
+          .to_string(index=False))
+
+    if not df_out.empty:
+        winner = df_out.iloc[0]
+        summary = {
+            "lfc_threshold": LFC_THRESH, "padj_threshold": PADJ_THRESH,
+            "n_classes_tested": int(len(df_out)),
+            "top_class":  str(winner["class"]),
+            "top_n_DEG":  int(winner["n_DEG"]),
+            "top_up":     winner["top_up"],
+            "top_down":   winner["top_down"],
+            "ranking":    df_out[["class", "n_DEG"]].to_dict("records"),
+        }
+    else:
+        summary = {"error": "no eligible classes"}
+    with open(OUT / "107_dingwall_class_deg_count.json", "w") as f:
+        json.dump(summary, f, indent=2)
+    print("[done]", flush=True)
+
+
+if __name__ == "__main__":
+    main()

scripts/analysis/108_dahlin_lineage_metabolism.py

@@ -1,158 +1,161 @@
-"""rank dahlin lineages by Kit_W41 vs WT metabolic shift across 4 modules."""
-from __future__ import annotations
-
-import json
-import warnings
-from pathlib import Path
-
-import anndata as ad
-import numpy as np
-import pandas as pd
-import scanpy as sc
-import scipy.sparse as sp
-from scipy.stats import mannwhitneyu
-
-warnings.filterwarnings("ignore")
-sc.settings.verbosity = 0
-
-ROOT = Path("/home/bcheng/PRISM")
-OUT = ROOT / "discovery/hematopoiesis/marker"
-OUT.mkdir(parents=True, exist_ok=True)
-
-D_DIR = ROOT / "data/corpus/hematopoiesis/held_out_unlabeled/dahlin_extract"
-PRED  = ROOT / "discovery/hematopoiesis/marker/dahlin_predictions.csv"
-GT = {"SIGAB1": "WT", "SIGAC1": "WT", "SIGAD1": "WT", "SIGAF1": "WT",
-      "SIGAG1": "WT", "SIGAH1": "WT",
-      "SIGAG8": "Kit_W41", "SIGAH8": "Kit_W41"}
-
-MIN_PER_GROUP = 15
-
-MODULES = {
-    "OXPHOS_ETC":          ["Ndufa1", "Ndufa2", "Ndufb1", "Ndufb2", "Sdha", "Sdhb",
-                            "Cox4i1", "Cox5a", "Cox6a1", "Atp5a1", "Atp5b", "Uqcrq"],
-    "Glycolysis":          ["Hk1", "Hk2", "Pfkm", "Pfkl", "Aldoa", "Gapdh", "Pgk1",
-                            "Pkm", "Ldha", "Eno1", "Tpi1", "Pgam1"],
-    "Fatty_acid_oxidation":["Cpt1a", "Acadm", "Acadl", "Acadvl", "Hadha", "Hadhb",
-                            "Ppara", "Ppargc1a", "Ucp2"],
-    "Redox_glutathione":   ["Gpx1", "Gpx2", "Gpx3", "Gpx4", "Gsr", "Prdx1", "Prdx2",
-                            "Prdx3", "Prdx4", "Prdx5", "Prdx6", "Sod1", "Sod2", "Cat"],
-}
-
-
-def load_dahlin():
-    print("[load] Dahlin raw counts", flush=True)
-    parts = []
-    for f in sorted(D_DIR.glob("*.txt.gz")):
-        sample = f.name.split("_")[1].split(".")[0]
-        df = pd.read_csv(f, sep="\t", compression="gzip", index_col=0)
-        X = sp.csr_matrix(df.values.T.astype(np.float32))
-        obs = pd.DataFrame(index=[f"{sample}_{bc}" for bc in df.columns.astype(str)])
-        obs["sample"] = sample
-        obs["group"]  = GT.get(sample, "unknown")
-        var = pd.DataFrame(index=df.index.astype(str))
-        parts.append(ad.AnnData(X=X, obs=obs, var=var))
-        print(f"[load] {sample}: {X.shape}", flush=True)
-    a = ad.concat(parts, join="outer", label="_batch")
-    print(f"[load] concat: {a.shape}", flush=True)
-
-    import mygene
-    mg = mygene.MyGeneInfo()
-    res = mg.querymany(a.var_names.astype(str).tolist(), scopes="ensembl.gene",
-                       fields="symbol", species="mouse", verbose=False)
-    id2sym = {r["query"]: r["symbol"] for r in res if "symbol" in r}
-    syms = pd.Series(a.var_names.astype(str)).map(id2sym).values
-    keep = pd.notna(syms)
-    a = a[:, keep].copy()
-    a.var_names = syms[keep]
-    a.var_names_make_unique()
-    print(f"[map] {a.n_vars} genes with symbols", flush=True)
-    a = a[a.obs["group"].isin(["Kit_W41", "WT"])].copy()
-    return a
-
-
-def main():
-    a = load_dahlin()
-    pred = pd.read_csv(PRED)
-    common = a.obs_names.intersection(pd.Index(pred["cell_id"].astype(str)))
-    a = a[list(common)].copy()
-    pred_map = dict(zip(pred["cell_id"].astype(str), pred["pred_label"]))
-    a.obs["pred_label"] = pd.Categorical(
-        [pred_map.get(c, "unknown") for c in a.obs_names])
-    print(f"[join] {a.n_obs} cells, "
-          f"Kit_W41={int((a.obs['group']=='Kit_W41').sum())}, "
-          f"WT={int((a.obs['group']=='WT').sum())}", flush=True)
-
-    sc.pp.normalize_total(a, target_sum=1e4)
-    sc.pp.log1p(a)
-
-    rows = []
-    for cls in sorted(a.obs["pred_label"].astype(str).unique()):
-        mask = (a.obs["pred_label"].astype(str) == cls).values
-        n_k = int((mask & (a.obs["group"].values == "Kit_W41")).sum())
-        n_w = int((mask & (a.obs["group"].values == "WT")).sum())
-        if n_k < MIN_PER_GROUP or n_w < MIN_PER_GROUP:
-            print(f"[skip] {cls}: n_Kit={n_k} n_WT={n_w}", flush=True)
-            continue
-        sub = a[mask].copy()
-        for name, genes in MODULES.items():
-            present = [g for g in genes if g in sub.var_names]
-            if not present:
-                sub.obs[f"pw_{name}"] = 0.0
-                continue
-            sc.tl.score_genes(sub, gene_list=present,
-                              score_name=f"pw_{name}", random_state=0,
-                              use_raw=False)
-
-        grp = sub.obs["group"].values
-        for name in MODULES:
-            v_k = sub.obs[f"pw_{name}"].astype(float).values[grp == "Kit_W41"]
-            v_w = sub.obs[f"pw_{name}"].astype(float).values[grp == "WT"]
-            try:
-                _, p = mannwhitneyu(v_k, v_w, alternative="two-sided")
-            except Exception:
-                p = 1.0
-            rows.append({
-                "class":         cls,
-                "module":        name,
-                "delta":         float(v_k.mean() - v_w.mean()),
-                "mean_Kit_W41":  float(v_k.mean()),
-                "mean_WT":       float(v_w.mean()),
-                "n_Kit_W41":     n_k,
-                "n_WT":          n_w,
-                "mannu_p":       float(p),
-            })
-        print(f"[ok] {cls}: n_Kit={n_k} n_WT={n_w}", flush=True)
-
-    df = pd.DataFrame(rows)
-    n_tests = len(df)
-    df["mannu_p_adj_bonferroni"] = np.minimum(df["mannu_p"] * n_tests, 1.0)
-    df.to_csv(OUT / "108_dahlin_lineage_metabolism.csv", index=False)
-
-    # per-class composite magnitude
-    piv = df.pivot(index="class", columns="module", values="delta").fillna(0.0)
-    piv["sum_abs_delta"] = piv.abs().sum(axis=1)
-    piv["l2_delta"]      = np.sqrt((piv[list(MODULES)] ** 2).sum(axis=1))
-    piv_sorted = piv.sort_values("sum_abs_delta", ascending=False)
-    piv_sorted.to_csv(OUT / "108_dahlin_lineage_metabolism_ranked.csv")
-
-    print("\n[ranked] lineages by sum |delta| across 4 metabolic modules:")
-    print(piv_sorted.round(4).to_string())
-
-    top_cls = piv_sorted.index[0]
-    top_row = piv_sorted.iloc[0]
-    summary = {
-        "modules": list(MODULES),
-        "top_lineage": str(top_cls),
-        "top_sum_abs_delta": float(top_row["sum_abs_delta"]),
-        "top_deltas_per_module": {
-            m: float(top_row[m]) for m in MODULES
-        },
-        "ranking": piv_sorted[["sum_abs_delta"]].reset_index().to_dict("records"),
-    }
-    with open(OUT / "108_dahlin_lineage_metabolism.json", "w") as f:
-        json.dump(summary, f, indent=2)
-    print("[done] top lineage:", top_cls, flush=True)
-
-
-if __name__ == "__main__":
-    main()
+"""rank dahlin lineages by Kit_W41 vs WT metabolic shift across 4 modules."""
+from __future__ import annotations
+
+import json
+import warnings
+from pathlib import Path
+
+import anndata as ad
+import numpy as np
+import pandas as pd
+import scanpy as sc
+import scipy.sparse as sp
+from scipy.stats import mannwhitneyu
+
+warnings.filterwarnings("ignore")
+sc.settings.verbosity = 0
+
+import os as _os
+from pathlib import Path as _Path
+PANDA_ROOT = _Path(_os.environ.get("PANDA_ROOT", str(_Path(__file__).resolve().parents[2])))
+ROOT = Path(str(PANDA_ROOT))
+OUT = ROOT / "discovery/hematopoiesis/marker"
+OUT.mkdir(parents=True, exist_ok=True)
+
+D_DIR = ROOT / "data/corpus/hematopoiesis/held_out_unlabeled/dahlin_extract"
+PRED  = ROOT / "discovery/hematopoiesis/marker/dahlin_predictions.csv"
+GT = {"SIGAB1": "WT", "SIGAC1": "WT", "SIGAD1": "WT", "SIGAF1": "WT",
+      "SIGAG1": "WT", "SIGAH1": "WT",
+      "SIGAG8": "Kit_W41", "SIGAH8": "Kit_W41"}
+
+MIN_PER_GROUP = 15
+
+MODULES = {
+    "OXPHOS_ETC":          ["Ndufa1", "Ndufa2", "Ndufb1", "Ndufb2", "Sdha", "Sdhb",
+                            "Cox4i1", "Cox5a", "Cox6a1", "Atp5a1", "Atp5b", "Uqcrq"],
+    "Glycolysis":          ["Hk1", "Hk2", "Pfkm", "Pfkl", "Aldoa", "Gapdh", "Pgk1",
+                            "Pkm", "Ldha", "Eno1", "Tpi1", "Pgam1"],
+    "Fatty_acid_oxidation":["Cpt1a", "Acadm", "Acadl", "Acadvl", "Hadha", "Hadhb",
+                            "Ppara", "Ppargc1a", "Ucp2"],
+    "Redox_glutathione":   ["Gpx1", "Gpx2", "Gpx3", "Gpx4", "Gsr", "Prdx1", "Prdx2",
+                            "Prdx3", "Prdx4", "Prdx5", "Prdx6", "Sod1", "Sod2", "Cat"],
+}
+
+
+def load_dahlin():
+    print("[load] Dahlin raw counts", flush=True)
+    parts = []
+    for f in sorted(D_DIR.glob("*.txt.gz")):
+        sample = f.name.split("_")[1].split(".")[0]
+        df = pd.read_csv(f, sep="\t", compression="gzip", index_col=0)
+        X = sp.csr_matrix(df.values.T.astype(np.float32))
+        obs = pd.DataFrame(index=[f"{sample}_{bc}" for bc in df.columns.astype(str)])
+        obs["sample"] = sample
+        obs["group"]  = GT.get(sample, "unknown")
+        var = pd.DataFrame(index=df.index.astype(str))
+        parts.append(ad.AnnData(X=X, obs=obs, var=var))
+        print(f"[load] {sample}: {X.shape}", flush=True)
+    a = ad.concat(parts, join="outer", label="_batch")
+    print(f"[load] concat: {a.shape}", flush=True)
+
+    import mygene
+    mg = mygene.MyGeneInfo()
+    res = mg.querymany(a.var_names.astype(str).tolist(), scopes="ensembl.gene",
+                       fields="symbol", species="mouse", verbose=False)
+    id2sym = {r["query"]: r["symbol"] for r in res if "symbol" in r}
+    syms = pd.Series(a.var_names.astype(str)).map(id2sym).values
+    keep = pd.notna(syms)
+    a = a[:, keep].copy()
+    a.var_names = syms[keep]
+    a.var_names_make_unique()
+    print(f"[map] {a.n_vars} genes with symbols", flush=True)
+    a = a[a.obs["group"].isin(["Kit_W41", "WT"])].copy()
+    return a
+
+
+def main():
+    a = load_dahlin()
+    pred = pd.read_csv(PRED)
+    common = a.obs_names.intersection(pd.Index(pred["cell_id"].astype(str)))
+    a = a[list(common)].copy()
+    pred_map = dict(zip(pred["cell_id"].astype(str), pred["pred_label"]))
+    a.obs["pred_label"] = pd.Categorical(
+        [pred_map.get(c, "unknown") for c in a.obs_names])
+    print(f"[join] {a.n_obs} cells, "
+          f"Kit_W41={int((a.obs['group']=='Kit_W41').sum())}, "
+          f"WT={int((a.obs['group']=='WT').sum())}", flush=True)
+
+    sc.pp.normalize_total(a, target_sum=1e4)
+    sc.pp.log1p(a)
+
+    rows = []
+    for cls in sorted(a.obs["pred_label"].astype(str).unique()):
+        mask = (a.obs["pred_label"].astype(str) == cls).values
+        n_k = int((mask & (a.obs["group"].values == "Kit_W41")).sum())
+        n_w = int((mask & (a.obs["group"].values == "WT")).sum())
+        if n_k < MIN_PER_GROUP or n_w < MIN_PER_GROUP:
+            print(f"[skip] {cls}: n_Kit={n_k} n_WT={n_w}", flush=True)
+            continue
+        sub = a[mask].copy()
+        for name, genes in MODULES.items():
+            present = [g for g in genes if g in sub.var_names]
+            if not present:
+                sub.obs[f"pw_{name}"] = 0.0
+                continue
+            sc.tl.score_genes(sub, gene_list=present,
+                              score_name=f"pw_{name}", random_state=0,
+                              use_raw=False)
+
+        grp = sub.obs["group"].values
+        for name in MODULES:
+            v_k = sub.obs[f"pw_{name}"].astype(float).values[grp == "Kit_W41"]
+            v_w = sub.obs[f"pw_{name}"].astype(float).values[grp == "WT"]
+            try:
+                _, p = mannwhitneyu(v_k, v_w, alternative="two-sided")
+            except Exception:
+                p = 1.0
+            rows.append({
+                "class":         cls,
+                "module":        name,
+                "delta":         float(v_k.mean() - v_w.mean()),
+                "mean_Kit_W41":  float(v_k.mean()),
+                "mean_WT":       float(v_w.mean()),
+                "n_Kit_W41":     n_k,
+                "n_WT":          n_w,
+                "mannu_p":       float(p),
+            })
+        print(f"[ok] {cls}: n_Kit={n_k} n_WT={n_w}", flush=True)
+
+    df = pd.DataFrame(rows)
+    n_tests = len(df)
+    df["mannu_p_adj_bonferroni"] = np.minimum(df["mannu_p"] * n_tests, 1.0)
+    df.to_csv(OUT / "108_dahlin_lineage_metabolism.csv", index=False)
+
+    # per-class composite magnitude
+    piv = df.pivot(index="class", columns="module", values="delta").fillna(0.0)
+    piv["sum_abs_delta"] = piv.abs().sum(axis=1)
+    piv["l2_delta"]      = np.sqrt((piv[list(MODULES)] ** 2).sum(axis=1))
+    piv_sorted = piv.sort_values("sum_abs_delta", ascending=False)
+    piv_sorted.to_csv(OUT / "108_dahlin_lineage_metabolism_ranked.csv")
+
+    print("\n[ranked] lineages by sum |delta| across 4 metabolic modules:")
+    print(piv_sorted.round(4).to_string())
+
+    top_cls = piv_sorted.index[0]
+    top_row = piv_sorted.iloc[0]
+    summary = {
+        "modules": list(MODULES),
+        "top_lineage": str(top_cls),
+        "top_sum_abs_delta": float(top_row["sum_abs_delta"]),
+        "top_deltas_per_module": {
+            m: float(top_row[m]) for m in MODULES
+        },
+        "ranking": piv_sorted[["sum_abs_delta"]].reset_index().to_dict("records"),
+    }
+    with open(OUT / "108_dahlin_lineage_metabolism.json", "w") as f:
+        json.dump(summary, f, indent=2)
+    print("[done] top lineage:", top_cls, flush=True)
+
+
+if __name__ == "__main__":
+    main()

scripts/analysis/109_veres_mature_beta.py

@@ -1,113 +1,116 @@
-"""split veres beta-predicted cells into mature (MAFA/UCN3 hi) vs SC-beta by INS x MAT quadrants."""
-from __future__ import annotations
-
-import json
-import warnings
-from pathlib import Path
-
-import anndata as ad
-import numpy as np
-import pandas as pd
-
-warnings.filterwarnings("ignore")
-
-ROOT = Path("/home/bcheng/PRISM")
-OUT = ROOT / "discovery/pancreas/marker"
-OUT.mkdir(parents=True, exist_ok=True)
-
-VERES = ROOT / "data/corpus/pancreas/held_out_labeled/veres_GSE114412_test.h5ad"
-PRED  = ROOT / "discovery/pancreas/marker/veres_predictions.csv"
-
-
-def z(x):
-    x = np.asarray(x, dtype=float)
-    s = x.std()
-    return (x - x.mean()) / (s if s > 0 else 1.0)
-
-
-def main():
-    print("[load]", flush=True)
-    a = ad.read_h5ad(VERES)
-    pred = pd.read_csv(PRED)
-    pred_map = dict(zip(pred["cell_id"].astype(str), pred["pred_label"]))
-    a.obs["pred_label"] = pd.Categorical(
-        [pred_map.get(c, "unknown") for c in a.obs_names])
-
-    print("[filter] pred_label == beta", flush=True)
-    sub = a[a.obs["pred_label"].astype(str) == "beta"].copy()
-    print(f"[filter] {sub.n_obs} beta-predicted cells", flush=True)
-
-    # veres X is already log-normalised (range 0..9)
-    def col(g):
-        if g not in sub.var_names:
-            return np.zeros(sub.n_obs)
-        j = sub.var_names.get_loc(g)
-        x = sub.X[:, j]
-        if hasattr(x, "toarray"):
-            x = x.toarray()
-        return np.asarray(x).ravel()
-
-    ins1 = col("Ins1"); ins2 = col("Ins2"); iapp = col("Iapp")
-    mafa = col("Mafa"); ucn3 = col("Ucn3")
-    insulin = ins1 + ins2
-    mature = z(mafa) + z(ucn3)
-
-    # split thresholds are the median within the veres beta-predicted set
-    ins_thr = float(np.median(insulin))
-    mat_thr = float(np.median(mature))
-    ins_hi  = insulin > ins_thr
-    mat_hi  = mature  > mat_thr
-
-    quadrant = np.array(
-        ["INS+/MAT+" if (ih and mh) else
-         "INS+/MAT-" if (ih and not mh) else
-         "INS-/MAT+" if (not ih and mh) else
-         "INS-/MAT-"
-         for ih, mh in zip(ins_hi, mat_hi)])
-    sub.obs["INS_level"]     = insulin
-    sub.obs["MAT_score"]     = mature
-    sub.obs["quadrant"]      = pd.Categorical(quadrant)
-
-    df = sub.obs[["quadrant", "INS_level", "MAT_score"]].copy()
-    df["Ins1"] = ins1; df["Ins2"] = ins2; df["Iapp"] = iapp
-    df["Mafa"] = mafa; df["Ucn3"] = ucn3
-    df["paper_label"] = sub.obs["paper_label"].astype(str).values
-    df.reset_index().rename(columns={"index": "cell_id"}).to_csv(
-        OUT / "109_veres_mature_beta_scores.csv", index=False)
-
-    counts = df["quadrant"].value_counts().to_dict()
-    paper_by_quad = df.groupby(["quadrant", "paper_label"], observed=True) \
-                      .size().unstack(fill_value=0)
-    paper_by_quad.to_csv(OUT / "109_veres_mature_beta_paper_x_quadrant.csv")
-
-    n_total = int(sub.n_obs)
-    n_mature = int((df["quadrant"] == "INS+/MAT+").sum())
-    n_scbeta = int((df["quadrant"] == "INS+/MAT-").sum())
-
-    fraction_mature = n_mature / n_total if n_total else 0.0
-
-    means = df.groupby("quadrant", observed=True)[
-        ["Mafa", "Ucn3", "Ins1", "Ins2", "Iapp"]].mean().round(3).to_dict()
-
-    summary = {
-        "n_beta_predicted": n_total,
-        "ins_threshold":    ins_thr,
-        "mat_threshold":    mat_thr,
-        "quadrant_counts":  counts,
-        "n_mature_INS+MAT+":  n_mature,
-        "n_SCbeta_INS+MAT-":  n_scbeta,
-        "fraction_mature":    round(fraction_mature, 4),
-        "mean_expression_per_quadrant": means,
-        "paper_label_x_quadrant": {
-            q: paper_by_quad.loc[q].to_dict()
-            for q in paper_by_quad.index
-        } if len(paper_by_quad) else {},
-    }
-    with open(OUT / "109_veres_mature_beta_summary.json", "w") as f:
-        json.dump(summary, f, indent=2)
-    print("\n[done]")
-    print(json.dumps(summary, indent=2))
-
-
-if __name__ == "__main__":
-    main()
+"""split veres beta-predicted cells into mature (MAFA/UCN3 hi) vs SC-beta by INS x MAT quadrants."""
+from __future__ import annotations
+
+import json
+import warnings
+from pathlib import Path
+
+import anndata as ad
+import numpy as np
+import pandas as pd
+
+warnings.filterwarnings("ignore")
+
+import os as _os
+from pathlib import Path as _Path
+PANDA_ROOT = _Path(_os.environ.get("PANDA_ROOT", str(_Path(__file__).resolve().parents[2])))
+ROOT = Path(str(PANDA_ROOT))
+OUT = ROOT / "discovery/pancreas/marker"
+OUT.mkdir(parents=True, exist_ok=True)
+
+VERES = ROOT / "data/corpus/pancreas/held_out_labeled/veres_GSE114412_test.h5ad"
+PRED  = ROOT / "discovery/pancreas/marker/veres_predictions.csv"
+
+
+def z(x):
+    x = np.asarray(x, dtype=float)
+    s = x.std()
+    return (x - x.mean()) / (s if s > 0 else 1.0)
+
+
+def main():
+    print("[load]", flush=True)
+    a = ad.read_h5ad(VERES)
+    pred = pd.read_csv(PRED)
+    pred_map = dict(zip(pred["cell_id"].astype(str), pred["pred_label"]))
+    a.obs["pred_label"] = pd.Categorical(
+        [pred_map.get(c, "unknown") for c in a.obs_names])
+
+    print("[filter] pred_label == beta", flush=True)
+    sub = a[a.obs["pred_label"].astype(str) == "beta"].copy()
+    print(f"[filter] {sub.n_obs} beta-predicted cells", flush=True)
+
+    # veres X is already log-normalised (range 0..9)
+    def col(g):
+        if g not in sub.var_names:
+            return np.zeros(sub.n_obs)
+        j = sub.var_names.get_loc(g)
+        x = sub.X[:, j]
+        if hasattr(x, "toarray"):
+            x = x.toarray()
+        return np.asarray(x).ravel()
+
+    ins1 = col("Ins1"); ins2 = col("Ins2"); iapp = col("Iapp")
+    mafa = col("Mafa"); ucn3 = col("Ucn3")
+    insulin = ins1 + ins2
+    mature = z(mafa) + z(ucn3)
+
+    # split thresholds are the median within the veres beta-predicted set
+    ins_thr = float(np.median(insulin))
+    mat_thr = float(np.median(mature))
+    ins_hi  = insulin > ins_thr
+    mat_hi  = mature  > mat_thr
+
+    quadrant = np.array(
+        ["INS+/MAT+" if (ih and mh) else
+         "INS+/MAT-" if (ih and not mh) else
+         "INS-/MAT+" if (not ih and mh) else
+         "INS-/MAT-"
+         for ih, mh in zip(ins_hi, mat_hi)])
+    sub.obs["INS_level"]     = insulin
+    sub.obs["MAT_score"]     = mature
+    sub.obs["quadrant"]      = pd.Categorical(quadrant)
+
+    df = sub.obs[["quadrant", "INS_level", "MAT_score"]].copy()
+    df["Ins1"] = ins1; df["Ins2"] = ins2; df["Iapp"] = iapp
+    df["Mafa"] = mafa; df["Ucn3"] = ucn3
+    df["paper_label"] = sub.obs["paper_label"].astype(str).values
+    df.reset_index().rename(columns={"index": "cell_id"}).to_csv(
+        OUT / "109_veres_mature_beta_scores.csv", index=False)
+
+    counts = df["quadrant"].value_counts().to_dict()
+    paper_by_quad = df.groupby(["quadrant", "paper_label"], observed=True) \
+                      .size().unstack(fill_value=0)
+    paper_by_quad.to_csv(OUT / "109_veres_mature_beta_paper_x_quadrant.csv")
+
+    n_total = int(sub.n_obs)
+    n_mature = int((df["quadrant"] == "INS+/MAT+").sum())
+    n_scbeta = int((df["quadrant"] == "INS+/MAT-").sum())
+
+    fraction_mature = n_mature / n_total if n_total else 0.0
+
+    means = df.groupby("quadrant", observed=True)[
+        ["Mafa", "Ucn3", "Ins1", "Ins2", "Iapp"]].mean().round(3).to_dict()
+
+    summary = {
+        "n_beta_predicted": n_total,
+        "ins_threshold":    ins_thr,
+        "mat_threshold":    mat_thr,
+        "quadrant_counts":  counts,
+        "n_mature_INS+MAT+":  n_mature,
+        "n_SCbeta_INS+MAT-":  n_scbeta,
+        "fraction_mature":    round(fraction_mature, 4),
+        "mean_expression_per_quadrant": means,
+        "paper_label_x_quadrant": {
+            q: paper_by_quad.loc[q].to_dict()
+            for q in paper_by_quad.index
+        } if len(paper_by_quad) else {},
+    }
+    with open(OUT / "109_veres_mature_beta_summary.json", "w") as f:
+        json.dump(summary, f, indent=2)
+    print("\n[done]")
+    print(json.dumps(summary, indent=2))
+
+
+if __name__ == "__main__":
+    main()

scripts/analysis/110_veres_polyhormonal_alpha.py

@@ -1,125 +1,128 @@
-"""test if veres alpha-pool polyhormonal (Ins+/Gcg+/Sst+) cells form a distinct sub-cluster vs graded."""
-from __future__ import annotations
-
-import json
-import warnings
-from pathlib import Path
-
-import anndata as ad
-import numpy as np
-import pandas as pd
-import scanpy as sc
-
-warnings.filterwarnings("ignore")
-sc.settings.verbosity = 0
-
-ROOT = Path("/home/bcheng/PRISM")
-OUT = ROOT / "discovery/pancreas/marker"
-OUT.mkdir(parents=True, exist_ok=True)
-
-VERES = ROOT / "data/corpus/pancreas/held_out_labeled/veres_GSE114412_test.h5ad"
-PRED  = ROOT / "discovery/pancreas/marker/veres_predictions.csv"
-
-
-def col(sub, g):
-    if g not in sub.var_names:
-        return np.zeros(sub.n_obs)
-    j = sub.var_names.get_loc(g)
-    x = sub.X[:, j]
-    if hasattr(x, "toarray"):
-        x = x.toarray()
-    return np.asarray(x).ravel()
-
-
-def main():
-    print("[load]", flush=True)
-    a = ad.read_h5ad(VERES)
-    pred = pd.read_csv(PRED)
-    pred_map = dict(zip(pred["cell_id"].astype(str), pred["pred_label"]))
-    a.obs["pred_label"] = pd.Categorical(
-        [pred_map.get(c, "unknown") for c in a.obs_names])
-
-    sub = a[a.obs["pred_label"].astype(str).isin(
-        ["alpha_progenitor", "alpha"])].copy()
-    print(f"[filter] {sub.n_obs} alpha-pool cells", flush=True)
-
-    ins1 = col(sub, "Ins1"); ins2 = col(sub, "Ins2")
-    gcg  = col(sub, "Gcg");  sst  = col(sub, "Sst"); iapp = col(sub, "Iapp")
-
-    ins_level = ins1 + ins2
-    q_ins = np.quantile(ins_level, 0.75)
-    q_gcg = np.quantile(gcg,       0.75)
-    q_sst = np.quantile(sst,       0.75)
-
-    n_pos = (
-        (ins_level >= q_ins).astype(int)
-        + (gcg      >= q_gcg).astype(int)
-        + (sst      >= q_sst).astype(int)
-    )
-    sub.obs["INS_level"] = ins_level
-    sub.obs["GCG_level"] = gcg
-    sub.obs["SST_level"] = sst
-    sub.obs["n_hormones_positive"] = n_pos
-
-    print("[cluster] PCA + Leiden", flush=True)
-    sc.pp.highly_variable_genes(sub, n_top_genes=2000, flavor="seurat_v3",
-                                subset=False, batch_key=None)
-    sc.pp.pca(sub, n_comps=30)
-    sc.pp.neighbors(sub, n_neighbors=15, n_pcs=30)
-    sc.tl.leiden(sub, resolution=0.5, random_state=0, key_added="leiden_alpha")
-
-    df = sub.obs[[
-        "pred_label", "paper_label", "leiden_alpha",
-        "INS_level", "GCG_level", "SST_level", "n_hormones_positive"
-    ]].copy()
-    df["Ins1"] = ins1; df["Ins2"] = ins2; df["Gcg"] = gcg
-    df["Sst"] = sst;   df["Iapp"] = iapp
-    df.reset_index().rename(columns={"index": "cell_id"}).to_csv(
-        OUT / "110_veres_polyhormonal_alpha_scores.csv", index=False)
-
-    baseline_polyhormonal = float((df["n_hormones_positive"] >= 2).mean())
-    per_clus = df.groupby("leiden_alpha", observed=True).agg(
-        n_cells=("n_hormones_positive", "size"),
-        frac_polyhormonal=("n_hormones_positive",
-                           lambda s: float((s >= 2).mean())),
-        frac_gcg_hi=("GCG_level",
-                     lambda s: float((s >= q_gcg).mean())),
-        frac_ins_hi=("INS_level",
-                     lambda s: float((s >= q_ins).mean())),
-        frac_sst_hi=("SST_level",
-                     lambda s: float((s >= q_sst).mean())),
-        mean_gcg=("GCG_level", "mean"),
-        mean_ins=("INS_level", "mean"),
-        mean_sst=("SST_level", "mean"),
-    ).sort_values("frac_polyhormonal", ascending=False).reset_index()
-    per_clus["enrichment_vs_baseline"] = per_clus["frac_polyhormonal"] \
-                                          / max(baseline_polyhormonal, 1e-6)
-    per_clus.to_csv(OUT / "110_veres_polyhormonal_alpha_per_cluster.csv",
-                    index=False)
-
-    n_2x_clusters = int((per_clus["enrichment_vs_baseline"] >= 2.0).sum())
-    verdict = ("distinct_polyhormonal_subcluster" if n_2x_clusters in (1, 2)
-               else "graded_phenotype" if n_2x_clusters == 0
-               else "diffuse_enrichment")
-
-    summary = {
-        "n_alpha_pool":               int(sub.n_obs),
-        "baseline_polyhormonal_frac": round(baseline_polyhormonal, 4),
-        "q75_thresholds":             {"Ins": float(q_ins),
-                                       "Gcg": float(q_gcg),
-                                       "Sst": float(q_sst)},
-        "leiden_resolution":          0.5,
-        "n_clusters":                 int(per_clus["leiden_alpha"].nunique()),
-        "n_clusters_enriched_2x":     n_2x_clusters,
-        "verdict":                    verdict,
-        "per_cluster":                per_clus.to_dict("records"),
-    }
-    with open(OUT / "110_veres_polyhormonal_alpha_summary.json", "w") as f:
-        json.dump(summary, f, indent=2)
-
-    print("\n[done] verdict:", verdict, flush=True)
-    print(per_clus.round(3).to_string(index=False))
-
-
-if __name__ == "__main__":
-    main()
+"""test if veres alpha-pool polyhormonal (Ins+/Gcg+/Sst+) cells form a distinct sub-cluster vs graded."""
+from __future__ import annotations
+
+import json
+import warnings
+from pathlib import Path
+
+import anndata as ad
+import numpy as np
+import pandas as pd
+import scanpy as sc
+
+warnings.filterwarnings("ignore")
+sc.settings.verbosity = 0
+
+import os as _os
+from pathlib import Path as _Path
+PANDA_ROOT = _Path(_os.environ.get("PANDA_ROOT", str(_Path(__file__).resolve().parents[2])))
+ROOT = Path(str(PANDA_ROOT))
+OUT = ROOT / "discovery/pancreas/marker"
+OUT.mkdir(parents=True, exist_ok=True)
+
+VERES = ROOT / "data/corpus/pancreas/held_out_labeled/veres_GSE114412_test.h5ad"
+PRED  = ROOT / "discovery/pancreas/marker/veres_predictions.csv"
+
+
+def col(sub, g):
+    if g not in sub.var_names:
+        return np.zeros(sub.n_obs)
+    j = sub.var_names.get_loc(g)
+    x = sub.X[:, j]
+    if hasattr(x, "toarray"):
+        x = x.toarray()
+    return np.asarray(x).ravel()
+
+
+def main():
+    print("[load]", flush=True)
+    a = ad.read_h5ad(VERES)
+    pred = pd.read_csv(PRED)
+    pred_map = dict(zip(pred["cell_id"].astype(str), pred["pred_label"]))
+    a.obs["pred_label"] = pd.Categorical(
+        [pred_map.get(c, "unknown") for c in a.obs_names])
+
+    sub = a[a.obs["pred_label"].astype(str).isin(
+        ["alpha_progenitor", "alpha"])].copy()
+    print(f"[filter] {sub.n_obs} alpha-pool cells", flush=True)
+
+    ins1 = col(sub, "Ins1"); ins2 = col(sub, "Ins2")
+    gcg  = col(sub, "Gcg");  sst  = col(sub, "Sst"); iapp = col(sub, "Iapp")
+
+    ins_level = ins1 + ins2
+    q_ins = np.quantile(ins_level, 0.75)
+    q_gcg = np.quantile(gcg,       0.75)
+    q_sst = np.quantile(sst,       0.75)
+
+    n_pos = (
+        (ins_level >= q_ins).astype(int)
+        + (gcg      >= q_gcg).astype(int)
+        + (sst      >= q_sst).astype(int)
+    )
+    sub.obs["INS_level"] = ins_level
+    sub.obs["GCG_level"] = gcg
+    sub.obs["SST_level"] = sst
+    sub.obs["n_hormones_positive"] = n_pos
+
+    print("[cluster] PCA + Leiden", flush=True)
+    sc.pp.highly_variable_genes(sub, n_top_genes=2000, flavor="seurat_v3",
+                                subset=False, batch_key=None)
+    sc.pp.pca(sub, n_comps=30)
+    sc.pp.neighbors(sub, n_neighbors=15, n_pcs=30)
+    sc.tl.leiden(sub, resolution=0.5, random_state=0, key_added="leiden_alpha")
+
+    df = sub.obs[[
+        "pred_label", "paper_label", "leiden_alpha",
+        "INS_level", "GCG_level", "SST_level", "n_hormones_positive"
+    ]].copy()
+    df["Ins1"] = ins1; df["Ins2"] = ins2; df["Gcg"] = gcg
+    df["Sst"] = sst;   df["Iapp"] = iapp
+    df.reset_index().rename(columns={"index": "cell_id"}).to_csv(
+        OUT / "110_veres_polyhormonal_alpha_scores.csv", index=False)
+
+    baseline_polyhormonal = float((df["n_hormones_positive"] >= 2).mean())
+    per_clus = df.groupby("leiden_alpha", observed=True).agg(
+        n_cells=("n_hormones_positive", "size"),
+        frac_polyhormonal=("n_hormones_positive",
+                           lambda s: float((s >= 2).mean())),
+        frac_gcg_hi=("GCG_level",
+                     lambda s: float((s >= q_gcg).mean())),
+        frac_ins_hi=("INS_level",
+                     lambda s: float((s >= q_ins).mean())),
+        frac_sst_hi=("SST_level",
+                     lambda s: float((s >= q_sst).mean())),
+        mean_gcg=("GCG_level", "mean"),
+        mean_ins=("INS_level", "mean"),
+        mean_sst=("SST_level", "mean"),
+    ).sort_values("frac_polyhormonal", ascending=False).reset_index()
+    per_clus["enrichment_vs_baseline"] = per_clus["frac_polyhormonal"] \
+                                          / max(baseline_polyhormonal, 1e-6)
+    per_clus.to_csv(OUT / "110_veres_polyhormonal_alpha_per_cluster.csv",
+                    index=False)
+
+    n_2x_clusters = int((per_clus["enrichment_vs_baseline"] >= 2.0).sum())
+    verdict = ("distinct_polyhormonal_subcluster" if n_2x_clusters in (1, 2)
+               else "graded_phenotype" if n_2x_clusters == 0
+               else "diffuse_enrichment")
+
+    summary = {
+        "n_alpha_pool":               int(sub.n_obs),
+        "baseline_polyhormonal_frac": round(baseline_polyhormonal, 4),
+        "q75_thresholds":             {"Ins": float(q_ins),
+                                       "Gcg": float(q_gcg),
+                                       "Sst": float(q_sst)},
+        "leiden_resolution":          0.5,
+        "n_clusters":                 int(per_clus["leiden_alpha"].nunique()),
+        "n_clusters_enriched_2x":     n_2x_clusters,
+        "verdict":                    verdict,
+        "per_cluster":                per_clus.to_dict("records"),
+    }
+    with open(OUT / "110_veres_polyhormonal_alpha_summary.json", "w") as f:
+        json.dump(summary, f, indent=2)
+
+    print("\n[done] verdict:", verdict, flush=True)
+    print(per_clus.round(3).to_string(index=False))
+
+
+if __name__ == "__main__":
+    main()

scripts/analysis/44_en1_cko_contrast.py

@@ -1,117 +1,120 @@
-"""en1-cKO vs WT contrast on aldrich predictions."""
-from __future__ import annotations
-from pathlib import Path
-import warnings
-warnings.filterwarnings("ignore")
-
-import numpy as np
-import pandas as pd
-import anndata as ad
-import scanpy as sc
-from scipy import stats
-
-TARGET = Path("/home/bcheng/PRISM/data/processed/skin/adata_processed.h5ad")
-PROJ   = Path("/home/bcheng/PRISM/discovery/pan_skin/marker/50_aldrich_projections.h5ad")
-NOVEL  = Path("/home/bcheng/PRISM/discovery/pan_skin/marker/51_aldrich_novel_annotation.csv")
-OUT    = Path("/home/bcheng/PRISM/discovery/pan_skin/marker")
-
-
-def main():
-    a = ad.read_h5ad(TARGET)
-    p = ad.read_h5ad(PROJ)
-    for c in ["pred_label", "pred_conf", "abstain", "pred_bbse_label"]:
-        if c in p.obs.columns:
-            a.obs[c] = p.obs[c].values
-    if NOVEL.exists():
-        nov = pd.read_csv(NOVEL, index_col=0)
-        a.obs["novel_cluster"] = nov.reindex(a.obs_names)["novel_cluster"].values
-
-    print(f"[cko] target: {a.shape}, genotype: {a.obs['genotype'].value_counts().to_dict()}",
-          flush=True)
-
-    rows = []
-    for cls in a.obs["pred_bbse_label"].unique():
-        mask = a.obs["pred_bbse_label"] == cls
-        n_cko = int((mask & (a.obs["genotype"] == "En1-cKO")).sum())
-        n_wt  = int((mask & (a.obs["genotype"] == "WT")).sum())
-        total_cko = (a.obs["genotype"] == "En1-cKO").sum()
-        total_wt  = (a.obs["genotype"] == "WT").sum()
-        contingency = np.array([[n_cko, total_cko - n_cko],
-                                [n_wt,  total_wt  - n_wt]])
-        odds, p_val = stats.fisher_exact(contingency)
-        f_cko = (n_cko + 1) / (total_cko + 2)
-        f_wt  = (n_wt  + 1) / (total_wt  + 2)
-        log2_fc = np.log2(f_cko / f_wt)
-        rows.append({
-            "class": cls,
-            "n_En1cKO": n_cko,
-            "n_WT": n_wt,
-            "pct_En1cKO": round(100 * n_cko / total_cko, 2),
-            "pct_WT": round(100 * n_wt / total_wt, 2),
-            "log2_fold_enrich_cKO_vs_WT": round(log2_fc, 3),
-            "fisher_pvalue": p_val,
-        })
-    df = pd.DataFrame(rows).sort_values("log2_fold_enrich_cKO_vs_WT")
-    print("\n[cko] class enrichment (cKO vs WT):")
-    print(df.to_string(index=False))
-
-    df.to_csv(OUT / "53_en1_cko_class_enrichment.csv", index=False)
-
-    de_rows = []
-    for cls in sorted(a.obs["pred_bbse_label"].unique()):
-        cls_mask = a.obs["pred_bbse_label"] == cls
-        if cls_mask.sum() < 50:
-            continue
-        sub = a[cls_mask].copy()
-        vc = sub.obs["genotype"].value_counts()
-        if not {"En1-cKO", "WT"}.issubset(vc.index) or vc.min() < 15:
-            continue
-        try:
-            sc.tl.rank_genes_groups(sub, "genotype", method="wilcoxon",
-                                    n_genes=40, use_raw=False)
-            for grp in ["En1-cKO", "WT"]:
-                if grp not in sub.uns["rank_genes_groups"]["names"].dtype.names:
-                    continue
-                names = list(sub.uns["rank_genes_groups"]["names"][grp][:15])
-                lfcs  = list(sub.uns["rank_genes_groups"]["logfoldchanges"][grp][:15])
-                for g, lf in zip(names, lfcs):
-                    de_rows.append({
-                        "class": cls, "up_in": grp, "gene": g, "logfc": round(float(lf), 3),
-                    })
-        except Exception as exc:
-            print(f"[cko] DE failed for {cls}: {exc}")
-            continue
-    de_df = pd.DataFrame(de_rows)
-    de_df.to_csv(OUT / "53_en1_cko_wilcoxon_within_class.csv", index=False)
-
-    md = ["# Aldrich En1-cKO vs WT contrast on PANDA-MLP predictions\n",
-          f"Total cells: {a.n_obs:,} ({int((a.obs['genotype']=='En1-cKO').sum())} En1-cKO, "
-          f"{int((a.obs['genotype']=='WT').sum())} WT).\n",
-          "## Class-level cKO/WT enrichment (BBSE-corrected predictions)\n",
-          df.to_markdown(index=False), "",
-          "The direction of `log2_fold_enrich_cKO_vs_WT` indicates whether a class is over-represented",
-          "in En1-cKO (positive) or WT (negative). Fisher exact p-value tests significance vs the",
-          "background genotype ratio (~40% cKO / 60% WT).\n",
-          "## Per-class Wilcoxon DE (En1-cKO vs WT within each class)\n",
-          "Top genes differentially expressed BETWEEN genotypes WITHIN a predicted class. Genes up in",
-          "cKO reveal En1-loss-responsive programs specific to that cell type; genes up in WT are the",
-          "opposite.\n",
-          ]
-    if len(de_df):
-        for cls in sorted(de_df["class"].unique()):
-            md.append(f"\n### {cls}\n")
-            for grp in ["En1-cKO", "WT"]:
-                sub = de_df[(de_df["class"] == cls) & (de_df["up_in"] == grp)]
-                if not len(sub):
-                    continue
-                md.append(f"**Up in {grp}**: " + ", ".join(sub["gene"].tolist()))
-    md += ["", "## Novel population x genotype cross-tab\n"]
-    if "novel_cluster" in a.obs.columns:
-        xt = pd.crosstab(a.obs["novel_cluster"], a.obs["genotype"])
-        md.append(xt.to_markdown())
-    (OUT / "53_en1_cko_contrast.md").write_text("\n".join(md))
-    print(f"[cko] wrote {OUT}/53_en1_cko_contrast.md")
-
-
-if __name__ == "__main__":
-    main()
+"""en1-cKO vs WT contrast on aldrich predictions."""
+from __future__ import annotations
+from pathlib import Path
+import warnings
+warnings.filterwarnings("ignore")
+
+import numpy as np
+import pandas as pd
+import anndata as ad
+import scanpy as sc
+from scipy import stats
+
+import os as _os
+from pathlib import Path as _Path
+PANDA_ROOT = _Path(_os.environ.get("PANDA_ROOT", str(_Path(__file__).resolve().parents[2])))
+TARGET = Path(str(PANDA_ROOT / "data/processed/skin/adata_processed.h5ad"))
+PROJ   = Path(str(PANDA_ROOT / "discovery/pan_skin/marker/50_aldrich_projections.h5ad"))
+NOVEL  = Path(str(PANDA_ROOT / "discovery/pan_skin/marker/51_aldrich_novel_annotation.csv"))
+OUT    = Path(str(PANDA_ROOT / "discovery/pan_skin/marker"))
+
+
+def main():
+    a = ad.read_h5ad(TARGET)
+    p = ad.read_h5ad(PROJ)
+    for c in ["pred_label", "pred_conf", "abstain", "pred_bbse_label"]:
+        if c in p.obs.columns:
+            a.obs[c] = p.obs[c].values
+    if NOVEL.exists():
+        nov = pd.read_csv(NOVEL, index_col=0)
+        a.obs["novel_cluster"] = nov.reindex(a.obs_names)["novel_cluster"].values
+
+    print(f"[cko] target: {a.shape}, genotype: {a.obs['genotype'].value_counts().to_dict()}",
+          flush=True)
+
+    rows = []
+    for cls in a.obs["pred_bbse_label"].unique():
+        mask = a.obs["pred_bbse_label"] == cls
+        n_cko = int((mask & (a.obs["genotype"] == "En1-cKO")).sum())
+        n_wt  = int((mask & (a.obs["genotype"] == "WT")).sum())
+        total_cko = (a.obs["genotype"] == "En1-cKO").sum()
+        total_wt  = (a.obs["genotype"] == "WT").sum()
+        contingency = np.array([[n_cko, total_cko - n_cko],
+                                [n_wt,  total_wt  - n_wt]])
+        odds, p_val = stats.fisher_exact(contingency)
+        f_cko = (n_cko + 1) / (total_cko + 2)
+        f_wt  = (n_wt  + 1) / (total_wt  + 2)
+        log2_fc = np.log2(f_cko / f_wt)
+        rows.append({
+            "class": cls,
+            "n_En1cKO": n_cko,
+            "n_WT": n_wt,
+            "pct_En1cKO": round(100 * n_cko / total_cko, 2),
+            "pct_WT": round(100 * n_wt / total_wt, 2),
+            "log2_fold_enrich_cKO_vs_WT": round(log2_fc, 3),
+            "fisher_pvalue": p_val,
+        })
+    df = pd.DataFrame(rows).sort_values("log2_fold_enrich_cKO_vs_WT")
+    print("\n[cko] class enrichment (cKO vs WT):")
+    print(df.to_string(index=False))
+
+    df.to_csv(OUT / "53_en1_cko_class_enrichment.csv", index=False)
+
+    de_rows = []
+    for cls in sorted(a.obs["pred_bbse_label"].unique()):
+        cls_mask = a.obs["pred_bbse_label"] == cls
+        if cls_mask.sum() < 50:
+            continue
+        sub = a[cls_mask].copy()
+        vc = sub.obs["genotype"].value_counts()
+        if not {"En1-cKO", "WT"}.issubset(vc.index) or vc.min() < 15:
+            continue
+        try:
+            sc.tl.rank_genes_groups(sub, "genotype", method="wilcoxon",
+                                    n_genes=40, use_raw=False)
+            for grp in ["En1-cKO", "WT"]:
+                if grp not in sub.uns["rank_genes_groups"]["names"].dtype.names:
+                    continue
+                names = list(sub.uns["rank_genes_groups"]["names"][grp][:15])
+                lfcs  = list(sub.uns["rank_genes_groups"]["logfoldchanges"][grp][:15])
+                for g, lf in zip(names, lfcs):
+                    de_rows.append({
+                        "class": cls, "up_in": grp, "gene": g, "logfc": round(float(lf), 3),
+                    })
+        except Exception as exc:
+            print(f"[cko] DE failed for {cls}: {exc}")
+            continue
+    de_df = pd.DataFrame(de_rows)
+    de_df.to_csv(OUT / "53_en1_cko_wilcoxon_within_class.csv", index=False)
+
+    md = ["# Aldrich En1-cKO vs WT contrast on PANDA-MLP predictions\n",
+          f"Total cells: {a.n_obs:,} ({int((a.obs['genotype']=='En1-cKO').sum())} En1-cKO, "
+          f"{int((a.obs['genotype']=='WT').sum())} WT).\n",
+          "## Class-level cKO/WT enrichment (BBSE-corrected predictions)\n",
+          df.to_markdown(index=False), "",
+          "The direction of `log2_fold_enrich_cKO_vs_WT` indicates whether a class is over-represented",
+          "in En1-cKO (positive) or WT (negative). Fisher exact p-value tests significance vs the",
+          "background genotype ratio (~40% cKO / 60% WT).\n",
+          "## Per-class Wilcoxon DE (En1-cKO vs WT within each class)\n",
+          "Top genes differentially expressed BETWEEN genotypes WITHIN a predicted class. Genes up in",
+          "cKO reveal En1-loss-responsive programs specific to that cell type; genes up in WT are the",
+          "opposite.\n",
+          ]
+    if len(de_df):
+        for cls in sorted(de_df["class"].unique()):
+            md.append(f"\n### {cls}\n")
+            for grp in ["En1-cKO", "WT"]:
+                sub = de_df[(de_df["class"] == cls) & (de_df["up_in"] == grp)]
+                if not len(sub):
+                    continue
+                md.append(f"**Up in {grp}**: " + ", ".join(sub["gene"].tolist()))
+    md += ["", "## Novel population x genotype cross-tab\n"]
+    if "novel_cluster" in a.obs.columns:
+        xt = pd.crosstab(a.obs["novel_cluster"], a.obs["genotype"])
+        md.append(xt.to_markdown())
+    (OUT / "53_en1_cko_contrast.md").write_text("\n".join(md))
+    print(f"[cko] wrote {OUT}/53_en1_cko_contrast.md")
+
+
+if __name__ == "__main__":
+    main()

scripts/analysis/45_marker_refinement.py

@@ -1,70 +1,73 @@
-"""marker refinement on aldrich zero-shot predictions: per-class wilcoxon DE vs canonical list."""
-from __future__ import annotations
-from pathlib import Path
-import warnings, yaml
-warnings.filterwarnings("ignore")
-
-import numpy as np
-import pandas as pd
-import anndata as ad
-import scanpy as sc
-
-TARGET = Path("/home/bcheng/PRISM/data/processed/skin/adata_processed.h5ad")
-PROJ   = Path("/home/bcheng/PRISM/discovery/pan_skin/marker/50_aldrich_projections.h5ad")
-TFS    = Path("/home/bcheng/PRISM/discovery/pan_skin/marker/known_skin_tfs.yaml")
-OUT    = Path("/home/bcheng/PRISM/discovery/pan_skin/marker")
-
-
-def main():
-    a = ad.read_h5ad(TARGET)
-    p = ad.read_h5ad(PROJ)
-    a.obs["pred_label"] = p.obs["pred_label"].values
-    a.obs["pred_conf"]  = p.obs["pred_conf"].astype(float).values
-
-    with open(TFS) as f:
-        tf_cfg = yaml.safe_load(f)
-    canon = tf_cfg["classes"]
-
-    keep = a[~a.obs["pred_label"].str.startswith("UNK", na=False)].copy()
-    sc.tl.rank_genes_groups(keep, "pred_label", method="wilcoxon",
-                            n_genes=100, use_raw=False)
-
-    rows = []
-    for cls in keep.obs["pred_label"].unique():
-        try:
-            names = keep.uns["rank_genes_groups"]["names"][cls]
-            lfc   = keep.uns["rank_genes_groups"]["logfoldchanges"][cls]
-            padj  = keep.uns["rank_genes_groups"]["pvals_adj"][cls]
-        except Exception:
-            continue
-        top100 = list(names[:100])
-        canon_set = set(canon.get(cls, []))
-        retained = [g for g in canon_set if g in top100]
-        novel = [g for g in top100 if g not in canon_set]
-        mask = keep.obs["pred_label"] == cls
-        n_cells = int(mask.sum())
-        mean_conf = float(keep.obs.loc[mask, "pred_conf"].mean())
-        rows.append({
-            "class": cls,
-            "n_cells_predicted": n_cells,
-            "mean_conf": round(mean_conf, 3),
-            "n_canonical": len(canon_set),
-            "n_canonical_retained_top100": len(retained),
-            "retained_canonical": ",".join(retained[:15]),
-            "novel_top20": ",".join(novel[:20]),
-        })
-    df = pd.DataFrame(rows).sort_values("n_cells_predicted", ascending=False)
-    df.to_csv(OUT / "52_refined_markers.csv", index=False)
-    print(df.to_string(index=False))
-
-    md = ["# Refined pan-skin markers for Aldrich zero-shot predictions\n"]
-    md.append("Marker refinement is Wilcoxon DE of each predicted class against all other")
-    md.append("predicted cells on raw Aldrich gene expression. `retained_canonical` are canonical")
-    md.append("markers recovered in the top-100; `novel_top20` are DE genes not in the canonical list.\n")
-    md.append(df.to_markdown(index=False))
-    (OUT / "52_refined_markers.md").write_text("\n".join(md))
-    print(f"[refine] wrote {OUT}/52_refined_markers.csv and .md")
-
-
-if __name__ == "__main__":
-    main()
+"""marker refinement on aldrich zero-shot predictions: per-class wilcoxon DE vs canonical list."""
+from __future__ import annotations
+from pathlib import Path
+import warnings, yaml
+warnings.filterwarnings("ignore")
+
+import numpy as np
+import pandas as pd
+import anndata as ad
+import scanpy as sc
+
+import os as _os
+from pathlib import Path as _Path
+PANDA_ROOT = _Path(_os.environ.get("PANDA_ROOT", str(_Path(__file__).resolve().parents[2])))
+TARGET = Path(str(PANDA_ROOT / "data/processed/skin/adata_processed.h5ad"))
+PROJ   = Path(str(PANDA_ROOT / "discovery/pan_skin/marker/50_aldrich_projections.h5ad"))
+TFS    = Path(str(PANDA_ROOT / "discovery/pan_skin/marker/known_skin_tfs.yaml"))
+OUT    = Path(str(PANDA_ROOT / "discovery/pan_skin/marker"))
+
+
+def main():
+    a = ad.read_h5ad(TARGET)
+    p = ad.read_h5ad(PROJ)
+    a.obs["pred_label"] = p.obs["pred_label"].values
+    a.obs["pred_conf"]  = p.obs["pred_conf"].astype(float).values
+
+    with open(TFS) as f:
+        tf_cfg = yaml.safe_load(f)
+    canon = tf_cfg["classes"]
+
+    keep = a[~a.obs["pred_label"].str.startswith("UNK", na=False)].copy()
+    sc.tl.rank_genes_groups(keep, "pred_label", method="wilcoxon",
+                            n_genes=100, use_raw=False)
+
+    rows = []
+    for cls in keep.obs["pred_label"].unique():
+        try:
+            names = keep.uns["rank_genes_groups"]["names"][cls]
+            lfc   = keep.uns["rank_genes_groups"]["logfoldchanges"][cls]
+            padj  = keep.uns["rank_genes_groups"]["pvals_adj"][cls]
+        except Exception:
+            continue
+        top100 = list(names[:100])
+        canon_set = set(canon.get(cls, []))
+        retained = [g for g in canon_set if g in top100]
+        novel = [g for g in top100 if g not in canon_set]
+        mask = keep.obs["pred_label"] == cls
+        n_cells = int(mask.sum())
+        mean_conf = float(keep.obs.loc[mask, "pred_conf"].mean())
+        rows.append({
+            "class": cls,
+            "n_cells_predicted": n_cells,
+            "mean_conf": round(mean_conf, 3),
+            "n_canonical": len(canon_set),
+            "n_canonical_retained_top100": len(retained),
+            "retained_canonical": ",".join(retained[:15]),
+            "novel_top20": ",".join(novel[:20]),
+        })
+    df = pd.DataFrame(rows).sort_values("n_cells_predicted", ascending=False)
+    df.to_csv(OUT / "52_refined_markers.csv", index=False)
+    print(df.to_string(index=False))
+
+    md = ["# Refined pan-skin markers for Aldrich zero-shot predictions\n"]
+    md.append("Marker refinement is Wilcoxon DE of each predicted class against all other")
+    md.append("predicted cells on raw Aldrich gene expression. `retained_canonical` are canonical")
+    md.append("markers recovered in the top-100; `novel_top20` are DE genes not in the canonical list.\n")
+    md.append(df.to_markdown(index=False))
+    (OUT / "52_refined_markers.md").write_text("\n".join(md))
+    print(f"[refine] wrote {OUT}/52_refined_markers.csv and .md")
+
+
+if __name__ == "__main__":
+    main()