Upload folder using huggingface_hub

README.md

@@ -12,12 +12,18 @@ library_name: pytorch
 pipeline_tag: image-feature-extraction
 ---
 
-# OpenPath — Pathology Foundation Model (training code)
+# OpenPath — Pathology Foundation Model (training & evaluation code)
 
 **OpenPath** is a vision foundation model for computational pathology: a **ViT-g/14** encoder
 pre-trained with self-supervision (**DINOv2** + **gram anchoring**) on **public-only** whole-slide
-histopathology tiles. This repository contains the **training / reproduction code**. The corpus and
-checkpoints are hosted separately (see below).
+histopathology tiles. This repository contains the **training, reproduction, and evaluation code**.
+The corpus and checkpoints are hosted separately (see below).
+
+> **Headline result.** On a **contamination-free** in-house spatial-transcriptomics (ST) cohort — the
+> least leakage-prone benchmark, since no public foundation model was trained on it — OpenPath **ranks
+> #1 among seven foundation models** (mean Pearson: OpenPath **0.323** > UNI2-h 0.301 > OpenMidnight
+> 0.300 > Virchow2 0.292 > prov-gigapath 0.286 > Phikon-v2 0.274 > UNI 0.257). Released checkpoint:
+> **`training_316250`** (in `openpath-checkpoints`). See [Evaluation](#evaluation).
 
 - **Encoder:** ViT-g/14 (reg4), 1536-dim CLS embedding
 - **Objective:** DINO + iBOT + KDE (DINOv2) with **gram anchoring** ported from DINOv3
@@ -39,6 +45,11 @@ scripts/
   autoresume.sh              # crash-tolerant auto-resume
   watch_eval.sh              # online HEST probing per checkpoint
   run_hest_3way.py           # HEST evaluation (Meta DINOv2 / Phikon-v2 / OpenPath)
+eval/                                          # downstream benchmark / reference-FM comparison
+  openpath_eva_backbone.py   # backbone factories: OpenPath + Phikon / OpenMidnight / UNI / UNI2-h / gigapath / Virchow2
+  st_bench.py                # in-house spatial-transcriptomics benchmark (LOPO ridge, headline)
+  run_patch_eval.sh          # PCam / CRC / BACH patch probing via kaiko-eva
+  eva_configs/               # eva YAML configs (crc / bach / patch_camelyon)
 requirements.txt
 ```
 
@@ -47,7 +58,7 @@ requirements.txt
 | Artifact | Hugging Face repo | Notes |
 |---|---|---|
 | **Corpus** | `taejoon89/openpath-corpus` | Native 40× pathology tiles, 33,991 WebDataset shards / ~17 TB (public, gated) |
-| **Checkpoints** | `taejoon89/openpath-checkpoints` | OpenPath teacher checkpoints across training (private) |
+| **Checkpoints** | `taejoon89/openpath-checkpoints` | OpenPath teacher checkpoints across training (`training_0` … `training_345000`); **released model = `training_316250`** (private) |
 | **Code** | `taejoon89/openpath` | This repository |
 
 The training config points to the corpus via
@@ -100,20 +111,85 @@ cls = m.forward_features(x)["x_norm_clstoken"]   # (B, 1536)
 ## Evaluation
 
 Frozen-encoder linear/ridge probing. The headline benchmark is a **contamination-free** in-house
-HCC (hepatocellular carcinoma) Visium spatial-transcriptomics cohort (leave-one-patient-out, mean
-Pearson over top-50 highly-variable genes) — no public FM was trained on it.
-
-| Benchmark (metric) | OpenPath (best) | OpenMidnight | Phikon-v2 |
-|---|---|---|---|
-| **HCC-ST** (Pearson, clean) | **0.324** | 0.300 | 0.274 |
-| HEST-1K (Pearson) | 0.383 | 0.390 | 0.375 |
-| NCT-CRC-HE (9-class acc) | 0.964 | 0.967 | 0.937 |
-| BACH (4-class acc) | 0.811 | 0.906 | 0.708 |
-
-On the clean HCC-ST cohort the majority of OpenPath checkpoints exceed OpenMidnight, and OpenPath
-beats Phikon-v2 across all benchmarks. The best checkpoint (`training_63250`, available in
-`openpath-checkpoints`) is top on both HEST-1K and HCC-ST. Note: PCam / CAMELYON is excluded from
-comparison because it overlaps our training corpus.
+Visium spatial-transcriptomics (ST) cohort (leave-one-patient-out, mean Pearson over top-50
+highly-variable genes) — **no public FM was trained on it**, so it is the least leakage-prone
+comparison. The reported OpenPath checkpoint is `training_316250`.
+
+**Comparison** — all 7 models loaded through one backbone factory and probed under an identical
+protocol; sorted by the clean ST benchmark:
+
+| Model | ST (in-house, clean) ↓ | HEST-1K (public) | NCT-CRC-HE (9-cls acc) | BACH (4-cls acc) |
+|---|---|---|---|---|
+| **OpenPath** | **0.323** | 0.372 | 0.954 | 0.761 |
+| UNI2-h | 0.301 | 0.414 | 0.966 | 0.908 |
+| OpenMidnight | 0.300 | 0.390 | 0.967 | 0.906 |
+| Virchow2 | 0.292 | 0.398 | 0.964 | 0.875 |
+| prov-gigapath | 0.286 | 0.393 | 0.953 | 0.752 |
+| Phikon-v2 | 0.274 | 0.375 | 0.937 | 0.708 |
+| UNI | 0.257 | 0.386 | 0.946 | 0.777 |
+
+**On the contamination-free ST cohort OpenPath ranks #1** among all seven foundation models. The
+picture inverts on the **public** benchmarks (HEST-1K, CRC, BACH): there OpenPath is mid-pack to
+low, and the large FMs lead. Those benchmarks derive from public repositories (TCGA/GTEx/etc.) that
+these FMs were pre-trained on, so their apparent edge is confounded by **train/test leakage** — which
+is exactly why the leakage-free in-house ST cohort is our headline. (The reported checkpoint
+`training_316250` is selected by ST; OpenPath's HEST-1K peaks earlier in training at ~0.38.)
+PCam / CAMELYON is excluded because it overlaps our own training corpus.
+
+### Reproducing the comparison
+
+All models are loaded through a single backbone-factory module (`eval/openpath_eva_backbone.py`) and
+probed under an identical protocol, so OpenPath and the reference FMs (Phikon-v2, OpenMidnight, UNI,
+UNI2-h, gigapath, Virchow2) are directly comparable.
+
+```bash
+export PYTHONPATH="$PWD/OpenPath:$PWD/eval"
+# Headline: in-house spatial-transcriptomics (LOPO ridge; cohort is private, code is provided)
+python eval/st_bench.py --backbone openpath --weights <teacher_checkpoint.pth>
+python eval/st_bench.py --backbone uni              # reference FM (also: uni2 / gigapath / virchow2 / phikon / openmidnight)
+
+# Patch probing (PCam / CRC / BACH) via kaiko-eva
+bash eval/run_patch_eval.sh openpath crc <teacher_checkpoint.pth>
+bash eval/run_patch_eval.sh uni crc                 # reference FM
+```
+
+Reference FM weights are pulled from their Hugging Face hubs on first use (UNI / UNI2-h / Virchow2
+are gated — request access on HF beforehand).
+
+## Intended use & limitations
+
+**Intended use.** OpenPath is a **frozen feature extractor** for H&E histopathology. It produces a
+1536-dim CLS embedding per 224×224 tile (native ~40× / 0.5 µm-per-pixel regime, ImageNet
+normalization) for downstream **linear/ridge probing, k-NN, MIL aggregation, and retrieval**. It is a
+research artifact, **not a medical device**, and must not be used for diagnosis or clinical
+decision-making.
+
+**Limitations.**
+- **Public-benchmark leakage.** Public benchmarks (HEST-1K, NCT-CRC-HE, BACH) derive from repositories
+  (TCGA/GTEx/…) that many foundation models — and partly OpenPath — were pre-trained on. Absolute
+  numbers and cross-model rankings on them are confounded; prefer leakage-controlled evaluation.
+- **Checkpoint trade-off.** The released `training_316250` is selected by the clean ST benchmark;
+  earlier checkpoints score higher on HEST-1K (~0.38). Pick a checkpoint to match your downstream task.
+- **Domain.** Trained on H&E WSIs at native magnification. Behavior on IHC, cytology, frozen sections,
+  non-0.5 µm-per-pixel inputs, or non-pathology images is untested.
+- **Patch-level encoder.** OpenPath encodes tiles independently; slide-level context requires a
+  separate aggregator (future work).
+
+## Citation
+
+A paper is in preparation. Until then, please cite the repository and the upstream work it builds on:
+
+```bibtex
+@misc{openpath2026,
+  title  = {OpenPath: a public-data pathology foundation model},
+  author = {OpenPath authors},
+  year   = {2026},
+  note   = {https://huggingface.co/taejoon89/openpath}
+}
+```
+
+OpenPath builds on **DINOv2**, **OpenMidnight / Midnight**, and **gram anchoring (DINOv3)** — see
+`OpenPath/README.md` for the full upstream citations, which should also be cited.
 
 ## License
 

eval/eva_configs/bach.yaml

@@ -0,0 +1,114 @@
+---
+trainer:
+  class_path: eva.Trainer
+  init_args:
+    n_runs: &N_RUNS ${oc.env:N_RUNS, 5}
+    default_root_dir: &OUTPUT_ROOT ${oc.env:OUTPUT_ROOT, logs/${oc.env:MODEL_NAME, dino_vits16}/offline/bach}
+    max_steps: &MAX_STEPS ${oc.env:MAX_STEPS, 12500}
+    checkpoint_type: ${oc.env:CHECKPOINT_TYPE, best}
+    accelerator: ${oc.env:ACCELERATOR, auto}
+    devices: ${oc.env:NUM_DEVICES, 1}
+    callbacks:
+      - class_path: eva.callbacks.ConfigurationLogger
+      - class_path: lightning.pytorch.callbacks.TQDMProgressBar
+        init_args:
+          refresh_rate: ${oc.env:TQDM_REFRESH_RATE, 1}
+      - class_path: lightning.pytorch.callbacks.LearningRateMonitor
+        init_args:
+          logging_interval: epoch
+      - class_path: lightning.pytorch.callbacks.ModelCheckpoint
+        init_args:
+          filename: best
+          save_last: ${oc.env:SAVE_LAST, false}
+          save_top_k: 1
+          monitor: &MONITOR_METRIC ${oc.env:MONITOR_METRIC, val/MulticlassAccuracy}
+          mode: &MONITOR_METRIC_MODE ${oc.env:MONITOR_METRIC_MODE, max}
+      - class_path: lightning.pytorch.callbacks.EarlyStopping
+        init_args:
+          min_delta: 0
+          patience: ${oc.env:PATIENCE, 1250}
+          monitor: *MONITOR_METRIC
+          mode: *MONITOR_METRIC_MODE
+      - class_path: eva.callbacks.ClassificationEmbeddingsWriter
+        init_args:
+          output_dir: &DATASET_EMBEDDINGS_ROOT ${oc.env:EMBEDDINGS_ROOT, ./data/embeddings}/${oc.env:MODEL_NAME, dino_vits16}/bach
+          dataloader_idx_map:
+            0: train
+            1: val
+          backbone:
+            class_path: eva.core.models.wrappers.ModelFromFunction
+            init_args:
+              path: ${oc.env:BACKBONE_FN, openpath_eva_backbone.build_openpath}
+              arguments:
+                weights: ${oc.env:OPENPATH_WEIGHTS, "none"}
+          overwrite: false
+    logger:
+      - class_path: lightning.pytorch.loggers.TensorBoardLogger
+        init_args:
+          save_dir: *OUTPUT_ROOT
+          name: ""
+model:
+  class_path: eva.HeadModule
+  init_args:
+    head:
+      class_path: torch.nn.Linear
+      init_args:
+        in_features: ${oc.env:IN_FEATURES, 384}
+        out_features: &NUM_CLASSES 4
+    criterion: torch.nn.CrossEntropyLoss
+    optimizer:
+      class_path: torch.optim.AdamW
+      init_args:
+        lr: ${oc.env:LR_VALUE, 0.0003}
+    metrics:
+      common:
+        - class_path: eva.metrics.AverageLoss
+        - class_path: eva.metrics.MulticlassClassificationMetrics
+          init_args:
+            num_classes: *NUM_CLASSES
+data:
+  class_path: eva.DataModule
+  init_args:
+    datasets:
+      train:
+        class_path: eva.datasets.EmbeddingsClassificationDataset
+        init_args: &DATASET_ARGS
+          root: *DATASET_EMBEDDINGS_ROOT
+          manifest_file: manifest.csv
+          split: train
+      val:
+        class_path: eva.datasets.EmbeddingsClassificationDataset
+        init_args:
+          <<: *DATASET_ARGS
+          split: val
+      predict:
+        - class_path: eva.vision.datasets.BACH
+          init_args: &PREDICT_DATASET_ARGS
+            root: ${oc.env:DATA_ROOT, ./data/bach}
+            split: train
+            download: ${oc.env:DOWNLOAD_DATA, false}
+            # Set `download: true` to download the dataset from https://zenodo.org/records/3632035
+            # The BACH dataset is distributed under the following license
+            # Attribution-NonCommercial-NoDerivs 4.0 International license
+            # (see: https://creativecommons.org/licenses/by-nc-nd/4.0/legalcode)
+            transforms:
+              class_path: eva.vision.data.transforms.common.ResizeAndCrop
+              init_args:
+                size: ${oc.env:RESIZE_DIM, 224}  
+                mean: ${oc.env:NORMALIZE_MEAN, [0.485, 0.456, 0.406]}
+                std: ${oc.env:NORMALIZE_STD, [0.229, 0.224, 0.225]}
+        - class_path: eva.vision.datasets.BACH
+          init_args:
+            <<: *PREDICT_DATASET_ARGS
+            split: val
+    dataloaders:
+      train:
+        batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 256}
+        num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
+        shuffle: true
+      val:
+        batch_size: *BATCH_SIZE
+        num_workers: *N_DATA_WORKERS
+      predict:
+        batch_size: &PREDICT_BATCH_SIZE ${oc.env:PREDICT_BATCH_SIZE, 64}
+        num_workers: *N_DATA_WORKERS

eval/eva_configs/crc.yaml

@@ -0,0 +1,113 @@
+---
+trainer:
+  class_path: eva.Trainer
+  init_args:
+    n_runs: &N_RUNS ${oc.env:N_RUNS, 5}
+    default_root_dir: &OUTPUT_ROOT ${oc.env:OUTPUT_ROOT, logs/${oc.env:MODEL_NAME, dino_vits16}/offline/crc}
+    max_steps: &MAX_STEPS ${oc.env:MAX_STEPS, 12500}
+    checkpoint_type: ${oc.env:CHECKPOINT_TYPE, best}
+    accelerator: ${oc.env:ACCELERATOR, auto}
+    devices: ${oc.env:NUM_DEVICES, 1}
+    callbacks:
+      - class_path: eva.callbacks.ConfigurationLogger
+      - class_path: lightning.pytorch.callbacks.TQDMProgressBar
+        init_args:
+          refresh_rate: ${oc.env:TQDM_REFRESH_RATE, 1}
+      - class_path: lightning.pytorch.callbacks.LearningRateMonitor
+        init_args:
+          logging_interval: epoch
+      - class_path: lightning.pytorch.callbacks.ModelCheckpoint
+        init_args:
+          filename: best
+          save_last: ${oc.env:SAVE_LAST, false}
+          save_top_k: 1
+          monitor: &MONITOR_METRIC ${oc.env:MONITOR_METRIC, val/MulticlassAccuracy}
+          mode: &MONITOR_METRIC_MODE ${oc.env:MONITOR_METRIC_MODE, max}
+      - class_path: lightning.pytorch.callbacks.EarlyStopping
+        init_args:
+          min_delta: 0
+          patience: ${oc.env:PATIENCE, 7}
+          monitor: *MONITOR_METRIC
+          mode: *MONITOR_METRIC_MODE
+      - class_path: eva.callbacks.ClassificationEmbeddingsWriter
+        init_args:
+          output_dir: &DATASET_EMBEDDINGS_ROOT ${oc.env:EMBEDDINGS_ROOT, ./data/embeddings}/${oc.env:MODEL_NAME, dino_vits16}/crc
+          dataloader_idx_map:
+            0: train
+            1: val
+          backbone:
+            class_path: eva.core.models.wrappers.ModelFromFunction
+            init_args:
+              path: ${oc.env:BACKBONE_FN, openpath_eva_backbone.build_openpath}
+              arguments:
+                weights: ${oc.env:OPENPATH_WEIGHTS, "none"}
+          overwrite: false
+    logger:
+      - class_path: lightning.pytorch.loggers.TensorBoardLogger
+        init_args:
+          save_dir: *OUTPUT_ROOT
+          name: ""
+model:
+  class_path: eva.HeadModule
+  init_args:
+    head:
+      class_path: openpath_eva_backbone.BNHead
+      init_args:
+        in_features: ${oc.env:IN_FEATURES, 384}
+        out_features: &NUM_CLASSES 9
+    criterion: torch.nn.CrossEntropyLoss
+    optimizer:
+      class_path: torch.optim.AdamW
+      init_args:
+        lr: ${oc.env:LR_VALUE, 0.0003}
+    metrics:
+      common:
+        - class_path: eva.metrics.AverageLoss
+        - class_path: eva.metrics.MulticlassClassificationMetrics
+          init_args:
+            num_classes: *NUM_CLASSES
+data:
+  class_path: eva.DataModule
+  init_args:
+    datasets:
+      train:
+        class_path: eva.datasets.EmbeddingsClassificationDataset
+        init_args: &DATASET_ARGS
+          root: *DATASET_EMBEDDINGS_ROOT
+          manifest_file: manifest.csv
+          split: train
+      val:
+        class_path: eva.datasets.EmbeddingsClassificationDataset
+        init_args:
+          <<: *DATASET_ARGS
+          split: val
+      predict:
+        - class_path: eva.vision.datasets.CRC
+          init_args: &PREDICT_DATASET_ARGS
+            root: ${oc.env:DATA_ROOT, ./data/crc}
+            split: train
+            download: ${oc.env:DOWNLOAD_DATA, false}
+            # Set `download: true` to download the dataset from https://zenodo.org/records/1214456
+            # The CRC dataset is distributed under the following license: "CC BY 4.0 LEGAL CODE"
+            # (see: https://creativecommons.org/licenses/by/4.0/legalcode)
+            transforms:
+              class_path: eva.vision.data.transforms.common.ResizeAndCrop
+              init_args:
+                size: ${oc.env:RESIZE_DIM, 224}  
+                mean: ${oc.env:NORMALIZE_MEAN, [0.485, 0.456, 0.406]}
+                std: ${oc.env:NORMALIZE_STD, [0.229, 0.224, 0.225]}
+        - class_path: eva.vision.datasets.CRC
+          init_args:
+            <<: *PREDICT_DATASET_ARGS
+            split: val
+    dataloaders:
+      train:
+        batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 256}
+        num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
+        shuffle: true
+      val:
+        batch_size: *BATCH_SIZE
+        num_workers: *N_DATA_WORKERS
+      predict:
+        batch_size: &PREDICT_BATCH_SIZE ${oc.env:PREDICT_BATCH_SIZE, 64}
+        num_workers: *N_DATA_WORKERS

eval/eva_configs/patch_camelyon.yaml

@@ -0,0 +1,129 @@
+---
+trainer:
+  class_path: eva.Trainer
+  init_args:
+    n_runs: &N_RUNS ${oc.env:N_RUNS, 5}
+    default_root_dir: &OUTPUT_ROOT ${oc.env:OUTPUT_ROOT, logs/${oc.env:MODEL_NAME, dino_vits16}/offline/patch_camelyon}
+    max_steps: &MAX_STEPS ${oc.env:MAX_STEPS, 12500}
+    checkpoint_type: ${oc.env:CHECKPOINT_TYPE, best}
+    accelerator: ${oc.env:ACCELERATOR, auto}
+    devices: ${oc.env:NUM_DEVICES, 1}
+    callbacks:
+      - class_path: eva.callbacks.ConfigurationLogger
+      - class_path: lightning.pytorch.callbacks.TQDMProgressBar
+        init_args:
+          refresh_rate: ${oc.env:TQDM_REFRESH_RATE, 1}
+      - class_path: lightning.pytorch.callbacks.LearningRateMonitor
+        init_args:
+          logging_interval: epoch
+      - class_path: lightning.pytorch.callbacks.ModelCheckpoint
+        init_args:
+          filename: best
+          save_last: ${oc.env:SAVE_LAST, false}
+          save_top_k: 1
+          monitor: &MONITOR_METRIC ${oc.env:MONITOR_METRIC, val/BinaryBalancedAccuracy}
+          mode: &MONITOR_METRIC_MODE ${oc.env:MONITOR_METRIC_MODE, max}
+      - class_path: lightning.pytorch.callbacks.EarlyStopping
+        init_args:
+          min_delta: 0
+          patience: ${oc.env:PATIENCE, 3}
+          monitor: *MONITOR_METRIC
+          mode: *MONITOR_METRIC_MODE
+      - class_path: eva.callbacks.ClassificationEmbeddingsWriter
+        init_args:
+          output_dir: &DATASET_EMBEDDINGS_ROOT ${oc.env:EMBEDDINGS_ROOT, ./data/embeddings}/${oc.env:MODEL_NAME, dino_vits16}/patch_camelyon
+          dataloader_idx_map:
+            0: train
+            1: val
+            2: test
+          backbone:
+            class_path: eva.core.models.wrappers.ModelFromFunction
+            init_args:
+              path: ${oc.env:BACKBONE_FN, openpath_eva_backbone.build_openpath}
+              arguments:
+                weights: ${oc.env:OPENPATH_WEIGHTS, "none"}
+          overwrite: false
+    logger:
+      - class_path: lightning.pytorch.loggers.TensorBoardLogger
+        init_args:
+          save_dir: *OUTPUT_ROOT
+          name: ""
+model:
+  class_path: eva.HeadModule
+  init_args:
+    head:
+      class_path: torch.nn.Linear
+      init_args:
+        in_features: ${oc.env:IN_FEATURES, 384}
+        out_features: 1
+    criterion: torch.nn.BCEWithLogitsLoss
+    optimizer:
+      class_path: torch.optim.AdamW
+      init_args:
+        lr: ${oc.env:LR_VALUE, 0.0003}
+    metrics:
+      common:
+        - class_path: eva.metrics.AverageLoss
+        - class_path: eva.metrics.BinaryClassificationMetrics
+data:
+  class_path: eva.DataModule
+  init_args:
+    datasets:
+      train:
+        class_path: eva.datasets.EmbeddingsClassificationDataset
+        init_args: &DATASET_ARGS
+          root: *DATASET_EMBEDDINGS_ROOT
+          manifest_file: manifest.csv
+          split: train
+          target_transforms:
+            class_path: torchvision.transforms.v2.ToDtype
+            init_args:
+              dtype: torch.float32
+      val:
+        class_path: eva.datasets.EmbeddingsClassificationDataset
+        init_args:
+          <<: *DATASET_ARGS
+          split: val
+      test:
+        class_path: eva.datasets.EmbeddingsClassificationDataset
+        init_args:
+          <<: *DATASET_ARGS
+          split: test
+      predict:
+        - class_path: eva.vision.datasets.PatchCamelyon
+          init_args: &PREDICT_DATASET_ARGS
+            root: ${oc.env:DATA_ROOT, ./data/patch_camelyon}
+            split: train
+            download: ${oc.env:DOWNLOAD_DATA, false}
+            # Set `download: true` to download the dataset from https://zenodo.org/records/1494286
+            # The PatchCamelyon dataset is distributed under the following license: 
+            # "Creative Commons Zero v1.0 Universal"
+            # (see: https://choosealicense.com/licenses/cc0-1.0/)
+            transforms:
+              class_path: eva.vision.data.transforms.common.ResizeAndCrop
+              init_args:
+                size: ${oc.env:RESIZE_DIM, 224}
+                mean: ${oc.env:NORMALIZE_MEAN, [0.485, 0.456, 0.406]} 
+                std: ${oc.env:NORMALIZE_STD, [0.229, 0.224, 0.225]}
+        - class_path: eva.vision.datasets.PatchCamelyon
+          init_args:
+            <<: *PREDICT_DATASET_ARGS
+            split: val
+        - class_path: eva.vision.datasets.PatchCamelyon
+          init_args:
+            <<: *PREDICT_DATASET_ARGS
+            split: test
+    dataloaders:
+      train:
+        batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 256}
+        num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
+        shuffle: true
+      val:
+        batch_size: *BATCH_SIZE
+        num_workers: *N_DATA_WORKERS
+      test:
+        batch_size: *BATCH_SIZE
+        num_workers: *N_DATA_WORKERS
+      predict:
+        batch_size: &PREDICT_BATCH_SIZE ${oc.env:PREDICT_BATCH_SIZE, 64}
+        num_workers: *N_DATA_WORKERS

eval/openpath_eva_backbone.py

@@ -0,0 +1,129 @@
+"""eva 연동용 OpenPath ViT-g/14 백본 팩토리.
+
+eva의 ModelFromFunction이 호출. dinov2 vit_giant2를 빌드하고 추출된 teacher
+state_dict(.pth)를 로드해, forward(x)->(B,1536) CLS 임베딩을 반환하는 nn.Module을 돌려준다.
+우리 학습/추출과 동일 규약(ImageNet norm은 eva 쪽 transform이 담당; 여기선 모델만).
+"""
+import torch
+import torch.nn as nn
+
+
+def build_openpath_vitg14(weights: str) -> nn.Module:
+    import dinov2.models.vision_transformer as vits
+
+    ck = torch.load(weights, map_location="cpu", weights_only=False)
+    arch = ck.get("arch", "vit_giant2")
+    kw = dict(patch_size=ck.get("patch_size", 14), img_size=224,
+              block_chunks=0, init_values=1.0)
+    if arch == "vit_giant2":
+        kw["ffn_layer"] = "swiglufused"
+    model = getattr(vits, arch)(**kw)
+    miss, unexp = model.load_state_dict(ck["teacher_backbone"], strict=True)
+    assert not miss and not unexp, f"load mismatch miss={miss} unexp={unexp}"
+
+    class CLSWrapper(nn.Module):
+        def __init__(self, m):
+            super().__init__()
+            self.m = m
+
+        @torch.no_grad()
+        def forward(self, x):
+            out = self.m(x)              # vit_giant2(x) -> (B, embed_dim) CLS
+            if out.ndim == 3:            # 혹시 토큰 시퀀스면 CLS 추출
+                out = out[:, 0, :]
+            return out
+
+    return CLSWrapper(model).eval()
+
+
+def build_phikon(weights: str = None) -> nn.Module:
+    """Phikon-v2 (owkin/phikon-v2, ViT-L, CLS=1024). weights 무시(HF 로드)."""
+    from transformers import AutoModel
+    base = AutoModel.from_pretrained("owkin/phikon-v2")
+
+    class W(nn.Module):
+        def __init__(self, m):
+            super().__init__(); self.m = m
+        @torch.no_grad()
+        def forward(self, x):
+            return self.m(pixel_values=x).last_hidden_state[:, 0, :]  # CLS
+    print("[eva] backbone=Phikon-v2 (CLS 1024)", flush=True)
+    return W(base).eval()
+
+
+def build_openmidnight(weights: str = None) -> nn.Module:
+    """OpenMidnight (SophontAI/OpenMidnight) teacher_checkpoint → dinov2 CLS.
+    weights 미지정시 HF 캐시서 자동 로드. (run_hest_3way build_om과 동일 규약, block_chunks=4)"""
+    import dinov2.models.vision_transformer as vits
+    if not weights or weights in ("none", "None", ""):
+        from huggingface_hub import hf_hub_download
+        weights = hf_hub_download("SophontAI/OpenMidnight", "teacher_checkpoint.pth")
+    ck = torch.load(weights, map_location="cpu", weights_only=False)
+    t = ck["teacher"] if "teacher" in ck else ck
+    sd = {k[len("backbone."):]: v for k, v in t.items() if k.startswith("backbone.")}
+    embed_dim = sd["cls_token"].shape[-1]
+    arch, ffn = ("vit_giant2", "swiglufused") if embed_dim == 1536 else ("vit_large", "mlp")
+    m = getattr(vits, arch)(patch_size=14, img_size=224, block_chunks=4,
+                            num_register_tokens=4, ffn_layer=ffn, init_values=1.0e-05,
+                            interpolate_antialias=True, interpolate_offset=0.0)
+    miss, unexp = m.load_state_dict(sd, strict=True)
+    assert not miss and not unexp, f"OM miss={miss} unexp={unexp}"
+    print(f"[eva] backbone=OpenMidnight arch={arch} embed_dim={embed_dim}", flush=True)
+
+    class W(nn.Module):
+        def __init__(self, mm):
+            super().__init__(); self.m = mm
+        @torch.no_grad()
+        def forward(self, x):
+            return self.m.forward_features(x)["x_norm_clstoken"]
+    return W(m).eval()
+
+
+def _cls_wrap(base):
+    class W(nn.Module):
+        def __init__(s, m): super().__init__(); s.m = m
+        @torch.no_grad()
+        def forward(s, x):
+            out = s.m(x)
+            if isinstance(out, (tuple, list)): out = out[0]
+            if out.ndim == 3: out = out[:, 0, :]      # tokens → CLS
+            return out
+    return W(base).eval()
+
+
+def build_timm_hub(repo, **kw):
+    """timm hf-hub 병리 FM (UNI/UNI2/Virchow2/gigapath). config.json 자동."""
+    import timm
+    base = timm.create_model(f"hf-hub:{repo}", pretrained=True, **kw)
+    print(f"[eva] backbone=timm:{repo}", flush=True)
+    return _cls_wrap(base)
+
+
+def build_uni(weights=None):        return build_timm_hub("MahmoodLab/UNI", init_values=1e-5, dynamic_img_size=True)
+def build_uni2(weights=None):
+    return build_timm_hub("MahmoodLab/UNI2-h", img_size=224, patch_size=14, depth=24, num_heads=24,
+                          init_values=1e-5, embed_dim=1536, mlp_ratio=2.66667*2, num_classes=0,
+                          no_embed_class=True, mlp_layer=__import__("timm").layers.SwiGLUPacked,
+                          act_layer=__import__("torch").nn.SiLU, reg_tokens=8, dynamic_img_size=True)
+def build_virchow2(weights=None):
+    import timm
+    return build_timm_hub("paige-ai/Virchow2", mlp_layer=timm.layers.SwiGLUPacked, act_layer=__import__("torch").nn.SiLU)
+def build_gigapath(weights=None):   return build_timm_hub("prov-gigapath/prov-gigapath", dynamic_img_size=True)
+
+
+def build_openpath(weights: str) -> nn.Module:
+    """OpenPath teacher_checkpoint(dinov2 ViT-g/14 reg4) → CLS. our run 체크포인트 로더."""
+    return build_openmidnight(weights)
+
+
+class BNHead(nn.Module):
+    """eva linear-probe head: BatchNorm1d(차원별 표준화) → Linear.
+    우리 임베딩은 일부 차원에 massive activation(norm~500)이 있어 raw Linear가
+    학습 실패. BN으로 per-dim 표준화하면 sklearn StandardScaler와 동등해져 회복."""
+    def __init__(self, in_features: int, out_features: int):
+        super().__init__()
+        self.bn = nn.BatchNorm1d(in_features, affine=True)
+        self.fc = nn.Linear(in_features, out_features)
+
+    def forward(self, x):
+        return self.fc(self.bn(x))

eval/run_patch_eval.sh

@@ -0,0 +1,44 @@
+#!/usr/bin/env bash
+# Patch 분류 벤치(pcam/crc/bach)를 kaiko-eva로 frozen-encoder linear probe.
+# 참조 FM과 OpenPath를 동일 프로토콜로 비교.
+#   model: openpath|openmidnight|phikon|uni|uni2|gigapath|virchow2
+#   bench: pcam|crc|bach
+# 사용: [GPU=0] [N_RUNS=5] bash eval/run_patch_eval.sh <model> <bench> [weights]
+#   openpath는 weights(teacher_checkpoint.pth) 필수. eva·timm·데이터셋은 사전 설치/다운로드 필요.
+set -u
+R="${OPENPATH_ROOT:-$(cd "$(dirname "$0")/.." && pwd)}"    # repo 루트(eval/의 상위)
+cd "$R"
+export PYTHONPATH="$R/OpenPath:$R/eval"                     # dinov2(OpenPath/) + backbone(eval/)
+export CUDA_VISIBLE_DEVICES="${GPU:-0}"                     # eva는 단일 GPU 권장
+export N_RUNS="${N_RUNS:-5}"
+export ACCELERATOR=gpu NUM_DEVICES=1 OPENPATH_WEIGHTS="${3:-none}"
+export PREDICT_BATCH_SIZE="${PREDICT_BATCH_SIZE:-512}" BATCH_SIZE="${BATCH_SIZE:-4096}" N_DATA_WORKERS="${N_DATA_WORKERS:-8}"
+model="$1"; bench="$2"
+
+case "$model" in
+  openpath)     export BACKBONE_FN=openpath_eva_backbone.build_openpath      IN_FEATURES=1536 ;;
+  openmidnight) export BACKBONE_FN=openpath_eva_backbone.build_openmidnight  IN_FEATURES=1536 ;;
+  phikon)       export BACKBONE_FN=openpath_eva_backbone.build_phikon        IN_FEATURES=1024 ;;
+  uni)          export BACKBONE_FN=openpath_eva_backbone.build_uni           IN_FEATURES=1024 ;;
+  uni2)         export BACKBONE_FN=openpath_eva_backbone.build_uni2          IN_FEATURES=1536 ;;
+  gigapath)     export BACKBONE_FN=openpath_eva_backbone.build_gigapath      IN_FEATURES=1536 ;;
+  virchow2)     export BACKBONE_FN=openpath_eva_backbone.build_virchow2      IN_FEATURES=1280 ;;
+  *) echo "unknown model $model (openpath|openmidnight|phikon|uni|uni2|gigapath|virchow2)"; exit 1 ;;
+esac
+export MODEL_NAME="${MODEL_NAME:-$model}"
+
+case "$bench" in
+  pcam) cfg=eval/eva_configs/patch_camelyon.yaml; export DATA_ROOT="$R/data/eva/pcam_h5" DOWNLOAD_DATA="${DL:-true}" ;;
+  crc)  cfg=eval/eva_configs/crc.yaml;            export DATA_ROOT="$R/data/eva/crc"     DOWNLOAD_DATA=false ;;
+  bach) cfg=eval/eva_configs/bach.yaml;           export DATA_ROOT="$R/data/eva/bach"    DOWNLOAD_DATA="${DL:-false}" ;;
+  *) echo "unknown bench $bench (pcam|crc|bach)"; exit 1 ;;
+esac
+
+# eva predict_fit는 임베딩 출력폴더가 있으면 거부 → 매 실행 전 정리(fresh 추출).
+# eva 임베딩 폴더명은 데이터셋명: pcam→patch_camelyon.
+evadir="$bench"; [ "$bench" = pcam ] && evadir=patch_camelyon
+rm -rf "$R/data/embeddings/$MODEL_NAME/$evadir" 2>/dev/null
+
+echo "=== EVAL $model × $bench | IN=$IN_FEATURES N_RUNS=$N_RUNS GPU=$CUDA_VISIBLE_DEVICES DATA_ROOT=$DATA_ROOT DL=$DOWNLOAD_DATA | $(date) ==="
+eva predict_fit --config "$cfg"
+echo "=== DONE $model × $bench | $(date) ==="

eval/st_bench.py

@@ -0,0 +1,176 @@
+#!/usr/bin/env python
+"""HEST-style spatial-transcriptomics(ST) 벤치마크 — in-house Visium 코호트.
+각 spot에서 WSI 패치 추출 → FM 임베딩(CLS) → PCA → Ridge 회귀로 상위 HVG 발현 예측
+→ Pearson 상관(유전자평균). Leave-one-patient-out CV.
+
+사용:
+  PYTHONPATH=OpenPath:eval venv_eva/bin/python eval/st_bench.py \
+     --backbone openmidnight            # 참조: OpenMidnight
+  ... --backbone phikon
+  ... --backbone openpath --weights data/runs/openpath_run/eval/training_316250/teacher_checkpoint.pth
+"""
+import os, sys, json, glob, argparse, re
+import numpy as np, pandas as pd
+import torch, openslide
+from PIL import Image
+from sklearn.linear_model import Ridge
+from sklearn.decomposition import PCA
+from sklearn.preprocessing import StandardScaler
+from scipy.stats import pearsonr
+import torchvision.transforms as T
+
+ROOT = os.environ.get("OPENPATH_ROOT", os.path.dirname(os.path.dirname(os.path.abspath(__file__))))  # repo 루트(eval/의 상위)
+DATA = os.environ.get("ST_ROOT", f"{ROOT}/data/st_bench")  # in-house Visium ST 코호트(비공개; 코드만 공개)
+IMAGENET_MEAN = (0.485, 0.456, 0.406); IMAGENET_STD = (0.229, 0.224, 0.225)
+
+
+def build_backbone(name, weights):
+    sys.path.insert(0, f"{ROOT}/OpenPath"); sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+    import openpath_eva_backbone as B
+    ref = {"uni": (B.build_uni, 1024), "uni2": (B.build_uni2, 1536),
+           "gigapath": (B.build_gigapath, 1536),
+           "virchow2": (B.build_virchow2, 1280),
+           "phikon": (B.build_phikon, 1024),
+           "openmidnight": (B.build_openmidnight, 1536)}
+    if name in ref:
+        fn, dim = ref[name]; return fn(), dim
+    # openpath: 우리 teacher_checkpoint(dinov2 teacher 포맷) 로더
+    return B.build_openpath(weights), 1536
+
+
+def slide_dirs():
+    return sorted([d for d in glob.glob(f"{DATA}/*") if os.path.isdir(d)])
+
+
+def patient_of(slide_dir):
+    b = os.path.basename(slide_dir)              # 예: pt<N>-<M> 또는 pt<N>
+    m = re.match(r"(pt\d+)", b)
+    return m.group(1) if m else b                # pt<N>-<M>, pt<N>-<K> → 동일 환자 pt<N>
+
+
+def load_one(slide_dir):
+    exp_f = glob.glob(f"{slide_dir}/*.spatial.data.exp.csv")
+    pos_f = glob.glob(f"{slide_dir}/*.tissue_positions_fullres.csv")
+    sf_f  = glob.glob(f"{slide_dir}/*.scalefactors_json.json")
+    wsi_f = glob.glob(f"{slide_dir}/*_p0.tif")
+    if not (exp_f and pos_f and sf_f and wsi_f):
+        return None
+    exp = pd.read_csv(exp_f[0], index_col=0)                      # spots × genes (log-norm)
+    pos = pd.read_csv(pos_f[0], index_col=0)                      # barcode → pxl coords
+    sf  = json.load(open(sf_f[0]))
+    diam = int(round(sf["spot_diameter_fullres"]))               # ~147px
+    common = exp.index.intersection(pos.index)
+    exp = exp.loc[common]; pos = pos.loc[common]
+    return dict(dir=slide_dir, exp=exp, pos=pos, diam=diam, wsi=wsi_f[0])
+
+
+def _load_patches(sl):
+    """224 uint8 패치 (N,224,224,3). 슬라이드별 캐시 재사용(체크포인트마다 동일 패치)."""
+    import numpy as np
+    cache = f"{DATA}/_pcache/{os.path.basename(sl['dir'])}.pt"
+    if os.path.exists(cache):
+        try: return torch.load(cache)
+        except Exception: pass
+    sldx = openslide.OpenSlide(sl["wsi"]); d = sl["diam"]
+    rs = T.Resize((224, 224))
+    xs = sl["pos"]["pxl_x_in_fullres"].values.astype(int)
+    ys = sl["pos"]["pxl_y_in_fullres"].values.astype(int)
+    plist = []
+    for x, y in zip(xs, ys):
+        patch = sldx.read_region((int(x - d // 2), int(y - d // 2)), 0, (d, d)).convert("RGB")
+        plist.append(torch.from_numpy(np.asarray(rs(patch))))     # (224,224,3) uint8
+    sldx.close()
+    patches = torch.stack(plist)
+    os.makedirs(os.path.dirname(cache), exist_ok=True)
+    tmp = cache + f".tmp{os.getpid()}"
+    torch.save(patches, tmp); os.replace(tmp, cache)              # atomic
+    return patches
+
+
+@torch.no_grad()
+def embed_slide(sl, model, device, bs=256):
+    patches = _load_patches(sl)                                   # (N,224,224,3) uint8
+    mean = torch.tensor(IMAGENET_MEAN).view(1, 3, 1, 1)
+    std = torch.tensor(IMAGENET_STD).view(1, 3, 1, 1)
+    embs = []
+    for i in range(0, len(patches), bs):
+        b = patches[i:i + bs].permute(0, 3, 1, 2).float().div_(255.0)   # (B,3,224,224)
+        b = ((b - mean) / std).to(device)
+        embs.append(model(b).float().cpu().numpy())
+    return np.concatenate(embs, 0)                                # (n_spots, dim)
+
+
+def top_hvg(exp_all, k=50):
+    # 학습셋 전체 log-norm 발현서 분산 상위 k 유전자
+    v = exp_all.var(axis=0)
+    return v.sort_values(ascending=False).index[:k].tolist()
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--backbone", required=True, choices=["openpath","openmidnight","phikon","uni","uni2","gigapath","virchow2"])
+    ap.add_argument("--weights", default=None)
+    ap.add_argument("--k-genes", type=int, default=50)
+    ap.add_argument("--pca", type=int, default=256)
+    ap.add_argument("--alpha", type=float, default=100.0)
+    ap.add_argument("--tag", default=None)
+    args = ap.parse_args()
+    device = "cuda"
+
+    model, dim = build_backbone(args.backbone, args.weights)
+    model = model.to(device).eval()
+
+    dirs = slide_dirs()
+    print(f"[st]slides={len(dirs)} backbone={args.backbone} dim={dim}", flush=True)
+    slides = []
+    for d in dirs:
+        sl = load_one(d)
+        if sl is None: print(f"  skip {os.path.basename(d)} (파일 부족)"); continue
+        sl["emb"] = embed_slide(sl, model, device)
+        sl["pat"] = patient_of(d)
+        slides.append(sl)
+        print(f"  {os.path.basename(d)}: spots={len(sl['pos'])} emb={sl['emb'].shape} pat={sl['pat']}", flush=True)
+
+    # 공통 유전자
+    genes = slides[0]["exp"].columns
+    for s in slides[1:]: genes = genes.intersection(s["exp"].columns)
+    genes = list(genes)
+    print(f"[st]공통 유전자 {len(genes)}", flush=True)
+
+    pats = sorted(set(s["pat"] for s in slides))
+    # leave-one-patient-out
+    per_gene_corr = []
+    for held in pats:
+        tr = [s for s in slides if s["pat"] != held]
+        te = [s for s in slides if s["pat"] == held]
+        Xtr = np.concatenate([s["emb"] for s in tr], 0)
+        Ytr = np.concatenate([s["exp"][genes].values for s in tr], 0)
+        Xte = np.concatenate([s["emb"] for s in te], 0)
+        Yte = np.concatenate([s["exp"][genes].values for s in te], 0)
+        # HVG는 학습셋서 선택
+        hvg_idx = np.argsort(-Ytr.var(0))[:args.k_genes]
+        Ytr_h, Yte_h = Ytr[:, hvg_idx], Yte[:, hvg_idx]
+        # 표준화 + PCA + Ridge
+        sc = StandardScaler().fit(Xtr)
+        Xtr2, Xte2 = sc.transform(Xtr), sc.transform(Xte)
+        p = PCA(n_components=min(args.pca, Xtr2.shape[1])).fit(Xtr2)
+        Xtr3, Xte3 = p.transform(Xtr2), p.transform(Xte2)
+        reg = Ridge(alpha=args.alpha).fit(Xtr3, Ytr_h)
+        pred = reg.predict(Xte3)
+        cors = []
+        for g in range(Yte_h.shape[1]):
+            if Yte_h[:, g].std() < 1e-8 or pred[:, g].std() < 1e-8:
+                cors.append(0.0)
+            else:
+                cors.append(pearsonr(Yte_h[:, g], pred[:, g])[0])
+        m = float(np.nanmean(cors))
+        per_gene_corr.append(m)
+        print(f"  [fold {held}] test_spots={Yte.shape[0]} meanPearson={m:.4f}", flush=True)
+
+    overall = float(np.mean(per_gene_corr))
+    tag = args.tag or args.backbone
+    print(f"[ST RESULT] backbone={tag} | LOPO mean Pearson = {overall:.4f} (folds={len(pats)}, HVG={args.k_genes})", flush=True)
+
+
+if __name__ == "__main__":
+    main()