# Results — ImageBind-style multimodal binding on ORION (1% subsample)

Metric: **R@1 on random 100-tile galleries** (chance = 0.01), mean over the two retrieval directions unless noted;
patient 5-fold reported as mean ± std over folds. "Pairing acc" = one-to-one Hungarian matching on the full test set.
Your split (`orion_official`) has 109 test tiles from 2 slides → single, noisy estimate.

## 1. Base experiment (H&E / IF / nuclei / markers, ImageBind-Huge H&E anchor) — 22 runs
Your split, R@1 (full 109-tile gallery):

| pair | anchor loss (ImageBind) | all-pairs loss |
|---|---|---|
| H&E→IF | **0.49** (pair acc 0.73) | 0.38 |
| H&E→nuclei | 0.36 | 0.28 |
| H&E→markers | 0.16 | 0.15 |
| IF→nuclei (emergent) | 0.41 | **0.54** |
| IF→markers (emergent) | 0.26 | **0.70** |
| nuclei→markers (emergent) | 0.15 | 0.29 |

- Emergent pairs (never trained together) match far above chance.
- Training the non-H&E pairs directly (all-pairs) boosts them but costs H&E→IF.
- Patient 5-fold ≈ leave-one-IF-batch-out → small batch effect; random tile split is optimistic.

## 2. H&E encoder ablation (anchor loss) — 33 runs
Mean R@1 (g100) over H&E pairs / emergent pairs:

| H&E encoder | your split | patient 5-fold |
|---|---|---|
| ImageBind-Huge | 0.29 / 0.25 | 0.33 ± 0.01 / 0.30 ± 0.00 |
| PLIP (stand-in for CONCH) | 0.35 / 0.29 | 0.37 ± 0.01 / 0.31 ± 0.01 |
| Phikon-v2 (stand-in for GigaPath) | **0.41** / 0.31 | 0.45 ± 0.02 / 0.36 ± 0.01 |
| **UNI2-h** | 0.40 / **0.34** | **0.48 ± 0.02 / 0.37 ± 0.01** |

CONCH and GigaPath: HF access refused (GatedRepoError); code ready (`04b_extract_anchor_fm.py`).

## 3. Encoder benchmark (H&E = UNI2 frozen; mIF / markers / text) — 18 runs
Text = templated per-tile descriptions generated from marker counts + nuclei count (no real text exists for ORION).

### Trained pairs (H&E ↔ modality), R@1 g100
| modality | encoder | your split | patient 5-fold |
|---|---|---|---|
| mIF | ResNet18 (17-ch stem) | 0.59 | 0.64 ± 0.01 |
| mIF | **ConvNeXt-Tiny** (17-ch stem) | **0.73** (pair acc 0.95) | **0.78 ± 0.02** |
| mIF | per-channel encoder (shared CNN + channel transformer) | 0.51 | 0.62 ± 0.01 |
| markers | linear | 0.18 | 0.22 ± 0.01 |
| markers | **MLP** | **0.31** | **0.39 ± 0.03** |
| markers | FT-Transformer | 0.21 | 0.30 ± 0.02 |
| text | PLIP text | 0.17 | 0.20 ± 0.02 |
| text | BiomedCLIP text | 0.16 | 0.19 ± 0.02 |
| text | ImageBind text | **0.22** | 0.21 ± 0.02 |
| text | PLIP / ImageBind **zero-shot** (no training) | ≈0.01–0.02 | ≈0.01–0.02 (chance) |

### Cross pairs (never trained together), best combos, patient 5-fold R@1 g100
| pair | best combo | R@1 |
|---|---|---|
| mIF ↔ markers | ConvNeXt-T + MLP / ResNet18 + MLP | 0.48 ± 0.03 |
| markers ↔ text | MLP + PLIP / MLP + ImageBind | 0.43 ± 0.07 / 0.42 ± 0.03 |
| mIF ↔ text | ResNet18/ConvNeXt-T + ImageBind | 0.25 ± 0.01 |

### Takeaways
- **mIF**: ConvNeXt-Tiny clearly best (+0.14 over ResNet18). The channel-agnostic per-channel encoder is not better here
  (fixed 17-marker panel; it would matter for transfer across panels).
- **Markers**: the plain MLP wins; FT-Transformer < MLP, linear worst. MLP markers also give the best cross-modal pairs.
- **Text**: the three text towers are within noise (0.19–0.21); ImageBind text slightly ahead. Zero-shot pathology CLIP
  (PLIP) cannot match these synthetic marker-based descriptions to H&E (≈ chance) → training a projection is required.
- Text is derived from markers, so markers↔text is partly by construction; H&E↔text measures how well H&E predicts
  coarse cell composition. Real paired text (PathGen images or ORION reports) is needed for a true text evaluation.
- Gene expression: none in ORION; the marker-encoder benchmark would be repeated on HEST/DLPFC for RNA.

Files: `results/bench_summary.csv`, `results/bench_long.csv`, `results/anchor_ablation_*.csv`, `results/summary_*.csv`.

---
# v2 / v3 — re-sampled ORION (rsync 2026-10-11) + real text + modality-specific foundation models
v1 sections above used the old ORION 1% sample (archived in `v1_old_orion_sample/`). Same slide split, new tiles.
H&E anchor = UNI2 (frozen) throughout. R@1 on 100-tile galleries (chance 0.01); patient 5-fold mean ± std.

## Text bound to H&E with real captions (PathGen-1.6M, patient split, 255 test pairs) — `12_train_text_pathgen.py`
| text tower (frozen) + MLP head | H&E→text R@1 (full 255) | g100 | zero-shot (own image tower) |
|---|---|---|---|
| BiomedCLIP | **0.17** | **0.29** | – |
| PLIP | 0.13 | 0.23 | 0.035 |
| ImageBind | 0.11 | 0.22 | 0.024 |
| PathGen-CLIP-L | 0.13 | 0.24 | 0.19* |
*PathGen-CLIP-L was trained on PathGen-1.6M → its PathGen numbers are not held-out.

## mIF / markers: from-scratch vs modality-specific foundation models — `17_bench_fm.py`, `18_eval_fm.py`
| pair | encoder | your split (pair acc) | patient 5-fold |
|---|---|---|---|
| H&E↔mIF | ConvNeXt-T (scratch) | **0.73** (0.96) | **0.76 ± 0.02** |
| H&E↔mIF | **VirTues** sp32 (frozen) + MLP | 0.67 (0.95) | 0.73 ± 0.03 |
| H&E↔markers | MLP on raw markers | **0.28** | **0.38 ± 0.03** |
| H&E↔markers | Geneformer V2 (frozen) + MLP | 0.12 | 0.17 ± 0.01 |
| H&E↔markers | scGPT human (frozen) + MLP | 0.11 | 0.18 ± 0.01 |
| mIF↔markers (emergent) | ConvNeXt-T + MLP | 0.37 | 0.48 ± 0.02 |
| mIF↔markers (emergent) | **VirTues + MLP** | **0.51 (0.93)** | **0.66 ± 0.02** |
| mIF↔markers (emergent) | VirTues + scGPT / + Geneformer | 0.22 / 0.19 | 0.29 / 0.26 |

## Emergent text ↔ mIF / markers (text head trained only on PathGen; never sees ORION/IF/markers) — patient 5-fold AUROC (Spearman), 7 concept prompts
| text | H&E (UNI2) | mIF ConvNeXt | mIF VirTues | markers MLP | markers scGPT | markers Geneformer |
|---|---|---|---|---|---|---|
| **PLIP** | 0.69 (0.32) | 0.69 (0.31) | 0.71 (0.34) | 0.76 (0.46) | **0.79 (0.49)** | 0.74 (0.41) |
| ImageBind | 0.67 (0.32) | 0.65 (0.27) | 0.67 (0.31) | 0.73 (0.41) | 0.75 (0.45) | 0.70 (0.35) |
| BiomedCLIP | 0.66 (0.25) | 0.66 (0.24) | 0.68 (0.26) | 0.71 (0.29) | 0.72 (0.32) | 0.68 (0.27) |
| PathGen-CLIP-L | 0.59 (0.10) | 0.62 (0.13) | 0.63 (0.16) | 0.66 (0.18) | 0.69 (0.25) | 0.64 (0.18) |
Zero-shot H&E (own image tower, no binding): PathGen-CLIP-L 0.75 (0.46), PLIP 0.73 (0.41), ImageBind 0.65 (0.28).

### Takeaways
- **mIF**: frozen VirTues ≈ ConvNeXt from scratch on H&E↔mIF (−0.03, within ~1 std), but its embedding transfers much better
  to markers through the shared space (emergent mIF↔markers 0.66 vs 0.48) — the pretrained, marker-aware mIF model is the
  better choice for a multi-modal space.
- **markers**: Geneformer / scGPT are clearly worse than an MLP on raw values for retrieval: both models only see the rank
  order of the 16 mapped genes in a tile (abundance is lost; ridge recovers marker props at ρ≈0.69 vs 0.998 raw).
  scGPT markers nevertheless give the best emergent text↔markers probe (PLIP 0.79). Single-cell RNA models are not a good fit
  for tile-level protein proportions; they should be tested on real expression (HEST/DLPFC).
- **text**: PLIP text head transfers best to ORION concepts; BiomedCLIP best on PathGen retrieval. PathGen-CLIP-L is the
  strongest zero-shot model, but its text features bound to UNI2 transfer worst (likely memorised PathGen captions → head
  overfits). Recommended: UNI2 (H&E) + VirTues (mIF) + MLP (markers) + PLIP text head.
- Data note: `PD-1_prop` ≈ 1 in every tile (doesn't track PD-1_count) — the column looks corrupted.

Files: `results/fm_retrieval.csv`, `results/fm_text_probes.csv`, `runs/bench_fm/`, `runs/text_pathgen/`, logs in `logs/v2`, `logs/v3`.

## Text ↔ H&E image pairing on ALL image-text pairs — `19_pairsets_prep.py`, `20_pairing.py`
Unified pool of every text-image source in PathGen-1.6M/ (12 sources; pathor1 skipped, images absent): 6144 unique
TCGA patches (4088 patients), 6480 pairs — caption 2697, qa_open 1799, qa_close 1391 (stem + correct option), summary 593.
Patient 5-fold CV; one text per image at test; test gallery ≈1229 pairs ("all"), 540 caption, 330 qa_open, 278 qa_close,
119 summary. Metric: R@1 on 100-pair galleries (chance 0.01) / Hungarian pair acc on the full fold; mean of both directions.

| image enc → / text enc ↓ (text_head, eval=all, R@1 g100) | BiomedCLIP | PLIP | ImageBind | PathGen-CLIP-L* |
|---|---|---|---|---|
| UNI2 | **0.231** | 0.200 | 0.199 | 0.248 |
| Phikon-v2 | **0.234** | 0.204 | 0.197 | 0.254 |
| PLIP (image) | 0.170 | 0.147 | 0.139 | 0.180 |
| BiomedCLIP (image) | 0.163 | 0.138 | 0.136 | 0.168 |
| ImageBind (image) | 0.132 | 0.108 | 0.098 | 0.139 |
| PathGen-CLIP-L (image)* | 0.282 | 0.252 | 0.243 | 0.303 |
Zero-shot (same CLIP model): PLIP 0.090, BiomedCLIP 0.111, ImageBind 0.029, PathGen-CLIP-L* 0.188.
Mode means (excluding PathGen-CLIP-L): zero-shot 0.077 < two_heads 0.165 ≈ text_head 0.166.

By text type, UNI2 + BiomedCLIP text head (g100 / pair acc): caption 0.28 / 0.11, summary 0.29 / 0.29, qa_open 0.22 / 0.09,
qa_close 0.13 / 0.06 (short multiple-choice stems carry little patch-specific information).
*PathGen-CLIP-L was trained on PathGen-1.6M (most of this pool) → not held-out; shown for reference only.
Takeaways: pathology SSL image encoders (UNI2, Phikon-v2) + BiomedCLIP text (256-token context) are the best clean
combination; binding to a frozen image anchor (ImageBind recipe) is as good as training both projections; every trained
head beats zero-shot CLIP by ~2×.
Files: `results/pairing_long.csv`, `results/pairing_summary.csv`.

## Four real modalities: H&E = image, text = text, mIF = protein, gene expression = RNA (HEST) — `21_`–`24_`
The former "markers" modality was computed from the mIF image (protein), so it is retired. Gene expression is now real
spatial transcriptomics: HEST-1k human Visium, 9484 spots (415 samples, 57 studies, 16 organs, 2046 bowel), 14,802
common genes, precomputed UNI2-h / GigaPath spot features. One frozen UNI2 space holds every modality:
H&E–mIF (ORION), H&E–text (PathGen), H&E–RNA (HEST) are trained; protein↔RNA, text↔RNA, text↔protein are emergent
(no dataset measures them together, so no 1:1 pairing accuracy is possible for them).

### H&E ↔ RNA (trained), HEST study+patient 5-fold, test gallery ≈1897 spots; R@1 on 100-spot galleries (chance 0.01)
| RNA encoder | UNI2 anchor | GigaPath anchor |
|---|---|---|
| MLP on 2000 HVGs | 0.148 ± 0.026 | 0.113 ± 0.018 |
| scGPT-human (frozen) + MLP | 0.154 ± 0.028 | 0.118 ± 0.017 |
| Geneformer-V2 (frozen) + MLP | **0.159 ± 0.027** | 0.124 ± 0.010 |
On real expression, the single-cell foundation models slightly beat the MLP. On the derived markers it was the other way round.

### Protein ↔ RNA (emergent, cross-dataset neighbour transfer; Spearman, shuffled-label chance ≈ 0.00)
ORION tile → k=20 nearest HEST spots. The predicted RNA of the matching gene is compared with the measured protein (marker⁺ cells / nuclei).

| query → gallery | all spots | bowel spots |
|---|---|---|
| **protein (VirTues mIF) → RNA (MLP / Geneformer)** | **0.45 / 0.44** | **0.42 / 0.42** |
| H&E (UNI2) → RNA head | 0.38 | 0.33 |
| H&E → H&E (image similarity only) | 0.36 | 0.33 |
| RNA → protein (reverse) | 0.18–0.22 | 0.20–0.23 |

Per marker (bowel, protein→RNA vs H&E→H&E):
- Pan-CK 0.81 vs 0.81; E-cad 0.81 vs 0.77
- Ki67 0.63 vs 0.49
- SMA 0.60 vs 0.49
- CD45 0.52 vs 0.35
- CD3e 0.38 vs 0.32
- CD68 0.32 vs 0.21
- CD4 0.29 vs 0.09
- CD20 0.29 vs 0.10
- PD-1 / PD-L1 < 0.2

### Text ↔ RNA (emergent, PathGen text head; concept prompts vs RNA of the concept gene; AUROC top quartile)
| text encoder | RNA emb (best of 3), all / bowel | H&E (UNI2) of the same spots, all / bowel |
|---|---|---|
| PLIP | 0.64 / 0.65 | 0.58 / 0.54 |
| ImageBind | 0.64 / 0.63 | 0.60 / 0.56 |
| BiomedCLIP | 0.61 / 0.63 | 0.57 / 0.54 |
| PathGen-CLIP-L | 0.59 / 0.62 | 0.55 / 0.56 |
Bowel AUROC per concept (RNA/MLP): stroma 0.74, immune 0.69, tumour epithelium 0.66, proliferation 0.59, macrophage 0.52, vessels 0.51.
Files: `cache/hest/`, `runs/gene/`, `results/gene_retrieval.csv`, `gene_emergent_protein.csv`, `gene_emergent_text.csv`, logs in `logs/gene/`.

## Virtual modalities predicted from H&E (to fill missing pairs; never used as test truth)

### GigaTIME (H&E → 21-channel virtual mIF) — `25_gigatime_infer.py`, `cache/gigatime/`
Model: prov-gigatime/GigaTIME (code Apache-2.0; weights research-only). Run at native tile resolution (2× upsampling was worse).
Sets: ORION 3,187 tiles, TCGA text patches 6,144, gene-expression patches 1,640; 0 failures.

ORION validation against the real mIF of the same tile (Spearman of tile-mean intensity, n = 3,187):

| Channel (GigaTIME ~ ORION) | Spearman |
|---|---|
| DAPI ~ Hoechst | 0.91 |
| CK ~ Pan-CK | 0.74 |
| Ki67 | 0.73 |
| PD-1 | 0.73 (the ORION PD-1 proportion is corrupt, ≈1 everywhere) |
| CD3 ~ CD3e | 0.65 |
| CD4 | 0.58 |
| CD68 | 0.47 |
| CD8 ~ CD8a | 0.37 |
| CD20 | 0.36 |
| PD-L1 | 0.34 |
| Actin-D / Transgelin ~ SMA | 0.18 / 0.23 |

Use the per-tile means only. At the pixel level, the per-tile median Pearson is ≈0.1 for every channel except DAPI (0.50), so the virtual maps are not spatially reliable.

### STPath (H&E → whole-transcriptome virtual RNA) — `26_stpath_infer.py`, `cache/stpath/`
Model: tlhuang/STPath (**no licence declared** — ask the authors before redistributing). Input: GigaPath tile features + coordinates + organ/tech tokens. Output: log1p counts for 14,791 genes.
Sets: HEST 9,484 spots, ORION 3,187 tiles, TCGA text patches 6,144.

HEST check (top-50 HVG Pearson):
- All spots: 0.40. This number is optimistic because STPath was trained on most of HEST.
- HEST-bench samples held out from STPath: 0.21.
- Within-sample: ≈0.09.

ORION check: predicted RNA vs the real protein of the same tile (Spearman, tiles with ≥20 nuclei; pooled / within-slide):

| Gene ~ protein | Pooled | Within slide |
|---|---|---|
| CDH1 ~ E-cad | 0.77 | 0.77 |
| EPCAM ~ Pan-CK | 0.76 | 0.74 |
| MKI67 ~ Ki67 | 0.68 | 0.64 |
| CD3E ~ CD3e | 0.45 | 0.42 |
| CD8A ~ CD8a | 0.44 | 0.40 |
| MS4A1 ~ CD20 | 0.42 | 0.34 |
| PTPRC ~ CD45 | 0.37 | 0.34 |
| ACTA2 ~ SMA | 0.34 | 0.34 |
| CD68 ~ CD68 | 0.25 | 0.22 |
| CD163 ~ CD163 | 0.07 | 0.15 |

TCGA caveat: 2,540 of 4,088 slides contribute a single patch, so most TCGA predictions have no spatial context and are less reliable than the ORION ones.

Coverage after completion: TCGA patches have H&E + real text + virtual protein + virtual RNA. ORION has H&E + real protein + virtual RNA, and synthetic text is pending (Azure). HEST has H&E features + real RNA.

## H&E foundation-model ablation: UNI2 vs GigaPath vs CONCH as the frozen anchor — `run_v4.sh`, `28_eval_anchor.py`
All other settings are identical. CONCH v1 vision tower: 448 px, unprojected 512-d (the authors' recommended tile feature). `conchclip` = CONCH projected into its own image-text space.

### H&E ↔ mIF (ORION, trained pair; mIF = frozen VirTues + MLP head; both directions averaged)
| Anchor | Patient 5-fold R@1 g100 | Patient 5-fold pair acc | ORION official R@1 (109 tiles) | ORION official pair acc |
|---|---|---|---|---|
| **UNI2** | **0.727 ± 0.030** | **0.751 ± 0.045** | 0.665 | **0.945** |
| GigaPath | 0.717 ± 0.025 | 0.690 ± 0.046 | **0.702** | 0.936 |
| CONCH | 0.571 ± 0.035 | 0.379 ± 0.037 | 0.529 | 0.651 |

Same with a from-scratch ConvNeXt-T mIF encoder:

| Anchor | Patient 5-fold R@1 g100 | Patient 5-fold pair acc | ORION official R@1 | ORION official pair acc |
|---|---|---|---|---|
| **UNI2** | **0.753 ± 0.019** | **0.744 ± 0.022** | **0.738** | **0.945** |
| GigaPath | 0.740 ± 0.022 | 0.691 ± 0.018 | 0.707 | 0.927 |
| CONCH | 0.639 ± 0.021 | 0.481 ± 0.058 | 0.608 | 0.780 |

The ranking UNI2 ≥ GigaPath ≫ CONCH holds for both mIF encoders.

### H&E ↔ text (all 6,144 TCGA patches / 6,480 real pairs, patient 5-fold, R@1 on 100-pair galleries, "all" text types)
| Image | Text | Text head only (ImageBind) | Two heads (CLIP-style) |
|---|---|---|---|
| UNI2 | BiomedCLIP | **0.231 ± 0.008** | 0.210 ± 0.012 |
| CONCH | BiomedCLIP | 0.230 ± 0.007 | **0.229 ± 0.007** |
| CONCH-CLIP | BiomedCLIP | 0.226 ± 0.007 | 0.226 ± 0.012 |
| GigaPath | BiomedCLIP | 0.218 ± 0.007 | 0.200 ± 0.011 |
| CONCH | CONCH text | 0.216 ± 0.009 | 0.221 ± 0.007 |
| UNI2 / GigaPath | CONCH text | 0.209 / 0.192 | 0.193 / 0.179 |
| CONCH-CLIP | CONCH text, **zero-shot** | 0.170 ± 0.010 | — |

On the PathGen-1.6M caption split (`12_train_text_pathgen.py`, BiomedCLIP, 255 test pairs, g100 he→text / text→he):
- UNI2: 0.29 / 0.27
- GigaPath: 0.26 / 0.22
- CONCH: 0.26 / 0.24

### Emergent text ↔ mIF (PathGen-trained text head per anchor, BiomedCLIP; ORION patient 5-fold AUROC, mean of 7 concepts)
| Anchor | mIF embedding | H&E embedding |
|---|---|---|
| UNI2 | 0.674 | 0.665 |
| GigaPath | 0.686 | 0.678 |
| **CONCH** | **0.719** | **0.701** |

CONCH gains most on tumour epithelium (0.89 vs UNI2 0.76), macrophage (0.65 vs 0.60) and vessels (0.57 vs 0.44).

### Takeaways
- **Trade-off between instance-level pairing and semantics.**
  - CONCH has the weakest H&E↔mIF pairing: pair accuracy is half of UNI2's.
  - CONCH transfers text concepts to mIF best, emergently. Its features were trained against captions, so they encode coarse semantics more than fine tile identity.
- **UNI2 is the best single anchor.** It is the strongest or tied on every trained pair: mIF, text, and RNA (UNI2 > GigaPath on HEST, see above).
- **The CONCH text tower is worse than BiomedCLIP** on these long PathGen texts, even when paired with CONCH images. Zero-shot CONCH (0.17) is well below any trained head (≥0.18).
- **HEST RNA has no CONCH anchor** because only UNI2 and GigaPath features are distributed, not the images.

## Virtual modality completion: train text directly on virtual pairs — `29_virtual_completion.py`
**Question:** does training text↔RNA or text↔protein directly on virtual pairs beat the emergent route through the H&E anchor?
- RNA pairs: TCGA real text ↔ STPath virtual RNA.
- Protein pairs: TCGA real text ↔ GigaTIME virtual mIF (11 channels shared with ORION).
- Model: BiomedCLIP head + MLP head on the modality vector, patient 5-fold over TCGA.
- Features are rank-gaussianised within each dataset.
- Testing uses only **real** measurements, with the same prompts, labels and protocol as the emergent probes.

| Text ↔ | Test set | Direct on virtual pairs | Emergent via H&E anchor (best encoder) |
|---|---|---|---|
| RNA | HEST real RNA, all spots | 0.595 | 0.59–0.61 (MLP / scGPT / Geneformer) |
| RNA | HEST real RNA, bowel | 0.552 | 0.61–0.62 |
| protein | ORION real mIF means | 0.596 | **0.678** (VirTues mIF), 0.663 (H&E) |
| protein | ORION *virtual* GigaTIME means (same tiles) | 0.686 | — |

(AUROC, top-quartile label, mean over concepts.)

- **In-domain retrieval** (held-out TCGA, text ↔ virtual modality, R@1 g100):
  - RNA: 0.196 ± 0.010
  - protein, 11 channel means: 0.048 ± 0.007
- **Virtual pairs do not beat the emergent route.** RNA ties on all spots and is worse on bowel. Protein is clearly worse.
- **The bottleneck is the virtual→real gap.** The same protein model scores 0.686 on ORION's virtual means but 0.596 on the real means of the same tiles.
- **The single-channel reference** (the matched real channel used as the score) gives 0.79–0.98. The probes are far from saturated, and an 11-number protein summary carries much less than the VirTues image embedding.
- **Recommendation:** use virtual modalities only as auxiliary or augmentation data, never as replacements for, or test truth instead of, real pairs.

## Four-modality validation with virtual / synthetic data (validation only, never training) — `31_virtual_validation.py`
- **Training:** every encoder is trained on real pairs only (H&E–mIF ORION, H&E–text PathGen, H&E–RNA HEST; frozen UNI2 anchor).
- **Virtual / synthetic data:** used only to build quadruplet test sets, so that modality pairs never measured together get a pairing accuracy:
  - virtual RNA: STPath
  - virtual mIF: GigaTIME
  - synthetic text: Azure `gpt-5.6-sol`, fixed 3-shot PathGen-style prompt, `27_azure_caption.py`; 9,031 captions, 0 failures, USD 38.11
- **Metrics:** 100-item galleries, R@1 (both directions) / one-to-one Hungarian pairing accuracy; chance 0.01.
- **TCGA evaluation:** 2,868 patches of 2,158 patients that the text head never saw.

### Calibration: is each virtual modality a usable stand-in for the real one?
| Virtual modality | Real pair (same items) | Same pair with the virtual modality | Verdict |
|---|---|---|---|
| RNA (STPath), HEST all | H&E↔RNA 0.148 / 0.166 | 0.126 / 0.182 | **usable** |
| RNA (STPath), HEST held out from STPath | 0.086 / 0.110 | 0.080 / 0.136 | **usable** |
| text (synthetic), 1,000 TCGA patches with real text | H&E↔text 0.184 / 0.165 | 0.134 / 0.140 | **usable, conservative (≈0.75×)** |
| text, same 1,000 patches | text real↔RNA virtual 0.039 / 0.038 | text synthetic↔RNA virtual 0.044 / 0.049 | consistent |
| mIF (GigaTIME, 10-ch VirTues), ORION | H&E↔mIF 0.644 / 0.863 | 0.038 / 0.046 | **not usable** |
| mIF (GigaTIME, 10 channel means), ORION | 0.279 / 0.357 | 0.050 / 0.053 | **not usable** |

GigaTIME's tile means correlate with the real mIF at the population level (Spearman ≈0.6), but they cannot tell one tile from 99 others. Pixel maps are worse. Virtual mIF is therefore only good for coarse concept-level checks, not pairing accuracy.

### Pairing accuracy for every modality pair (R@1 / pair acc on 100-galleries; ⚠ = circular)
"Circular" means both sides come from the same H&E image, so the number is an optimistic upper bound.

**ORION** (H&E real, mIF real (VirTues 17 ch), RNA virtual, text synthetic), patient 5-fold:

| | mIF (real) | RNA (virtual) | text (synthetic) |
|---|---|---|---|
| **H&E** | **0.726 / 0.935** (trained) | 0.103 / 0.192 ⚠ | 0.066 / 0.091 ⚠ |
| **mIF (real)** | | **0.108 / 0.152** (emergent) | **0.068 / 0.069** (emergent) |
| **RNA (virtual)** | | | 0.036 / 0.050 ⚠ |

**TCGA** (H&E real, text real, RNA virtual):

| | text (real) | RNA (virtual) |
|---|---|---|
| **H&E** | **0.178 / 0.170** (trained) | 0.121 / 0.175 ⚠ |
| **text (real)** | | **0.042 / 0.041** (emergent) |

Virtual mIF on TCGA (text↔mIF 0.012–0.021) fails calibration and is not reported as a result.

### Takeaways
- **Protein ↔ RNA**, never trained together: 0.108 R@1 / 0.152 pairing accuracy on ORION, 10–15× chance. This is real protein against virtual RNA, and virtual RNA scores close to real RNA in the HEST calibration.
- **Text ↔ protein**, never trained together: 0.068, about 7× chance. This uses synthetic text, which scores about 0.75× real text in the calibration, so the real-text value is likely ≈0.09.
  - ORION is a single-organ cohort (colon), so its galleries are harder than TCGA's 30-cancer galleries. H&E↔synthetic text is 0.066 on ORION vs 0.134 on TCGA.
- **Text ↔ RNA**, never trained together: 0.042 with real text and 0.044 with synthetic text on the same patches, about 4× chance.
- Emergent pairs are clearly above chance but an order of magnitude below the trained H&E↔mIF pair. Instance-level emergent alignment is weak; concept-level alignment (AUROC 0.6–0.7, earlier sections) is moderate.

## Final protocol: real-only training, explicit train/val/test, virtual = test-time only (`33_final_eval.py`)
Every head is trained on REAL pairs only; val is used only for early stopping; all numbers are on the TEST split
(5 folds, mean ± std; patients/groups never cross splits, asserted). Virtual modalities (STPath RNA, GigaTIME mIF,
GPT captions) are only fed through the frozen real-trained heads at test time. Metrics: 100-item galleries × 50 draws,
both directions; chance R@1 = pair acc = 0.01, R@10 = 0.10. Split files: `splits/patient_5fold`, `splits/orion_official`,
`splits/final/{hest,tcga}_fold<k>.csv`; sizes `results/final_splits.csv`; numbers `results/final_{long,summary}.csv`.
TCGA text heads are now trained on ALL image-text pairs per fold (`runs/text_all/uni2_biomedclip_fold<k>`).

| dataset | unit | train | val | test (per fold) |
|---|---|---|---|---|
| ORION patient 5-fold | patient | 2291 tiles / 29 pts | 258 / 3 | 637 / 8 |
| ORION official (user split) | patient | 2954 / 36 | 124 / 2 | 109 / 2 |
| HEST | study\|patient group | 7141 spots / 91 grp | 447 / 10 | 1897 / 25 |
| TCGA all image-text pairs | patient | 4426 images / 2943 pts | 489 / 327 | 1229 / 818 |

**(1) Real pairs (test split)**

| pair | dataset | R@1 | R@10 | MRR | pair acc |
|---|---|---|---|---|---|
| H&E↔mIF | ORION 5-fold | 0.726±0.031 | 0.989 | 0.828 | 0.935±0.030 |
| H&E↔mIF | ORION official | 0.665 | 1.000 | 0.791 | 0.948 |
| H&E↔RNA | HEST | 0.148±0.026 | 0.547 | 0.277 | 0.166±0.032 |
| H&E↔RNA | HEST, STPath-unseen spots | 0.086±0.038 | 0.509 | 0.214 | 0.110±0.038 |
| H&E↔text | TCGA | 0.229±0.011 | 0.700 | 0.383 | 0.238±0.011 |

**(2) Virtual pairs (same test items; ⚠ = both sides derived from the same H&E → optimistic)**

| pair | dataset | R@1 | R@10 | pair acc | type |
|---|---|---|---|---|---|
| mIF↔RNA(virtual) | ORION 5-fold | 0.108±0.006 | 0.516 | 0.152±0.020 | emergent, real×virtual |
| mIF↔text(synthetic) | ORION 5-fold | 0.073±0.011 | 0.446 | 0.077±0.012 | emergent, real×virtual |
| text↔RNA(virtual) | TCGA | 0.055±0.009 | 0.318 | 0.058±0.010 | emergent, real×virtual |
| RNA(virtual)↔text(synthetic) | ORION 5-fold | 0.047±0.008 | 0.343 | 0.058±0.009 | ⚠ virtual×virtual |
| H&E↔RNA(virtual) | ORION / TCGA | 0.103 / 0.120 | 0.52 / 0.45 | 0.19 / 0.18 | ⚠ |
| H&E↔text(synthetic) | ORION 5-fold | 0.075±0.006 | 0.439 | 0.101±0.013 | ⚠ |
| text↔mIF(virtual) | TCGA | 0.015±0.002 | 0.133 | 0.014 | GigaTIME not usable |

Calibration (virtual replaces real on the same test items): HEST H&E↔RNA 0.148 → 0.126 (STPath-unseen 0.086 → 0.080);
TCGA H&E↔text 0.197 → 0.166 (synthetic = 0.84× real); ORION H&E↔mIF 0.644 → 0.038 (GigaTIME fails at instance level).
ORION official numbers (109 test tiles, 2 patients) match the 5-fold ones (mIF↔RNA(virtual) 0.108 / 0.152).

### Why GigaTIME virtual mIF fails at pairing (`34_gigatime_diag.py`, `results/gigatime_diag*.csv`)
- Per-channel tile-mean Spearman vs real ORION mIF: GigaTIME mean 0.54 pooled / 0.50 within-slide (not a slide-level
  artefact), vs 0.79 / 0.75 for an in-domain reference (ridge UNI2 → real means, trained on ORION train, scored on test).
  Weakest: SMA 0.20 (no SMA channel; Actin-D/Transgelin proxy), PD-L1 0.34, CD20 0.36, CD8 0.37.
- Retrieval needs every channel to be right at once: real means ↔ GigaTIME means R@1 0.035 (chance 0.01) vs
  real ↔ UNI2-ridge 0.261 — same 10 channels, no learned head.
- Pixel maps are sigmoid probabilities, mostly < 0.05 (58–96% of pixels per channel) with ~0 confident positives except
  DAPI/CK; pixel Pearson ≈0.1. The real-trained VirTues head sees out-of-distribution input → 0.038.
- Conclusion: H&E does contain this information (the in-domain reference recovers it); GigaTIME, trained on another
  cohort/panel/scanner, does not transfer to ORION at tile resolution.

### Unified mIF channels (real and virtual mIF both = the 10-channel panel shared with GigaTIME)
`33_final_eval.py` now uses the 10-ch head (real ORION, fold-k train/val) for every mIF row; 17-ch only as reference.
`[qm]` = GigaTIME pixels quantile-mapped per channel onto the real fold-k TRAIN pixel distribution before the same
preprocessing (`30_virtues_shared.py virtual_qm`). ORION patient 5-fold test, R@1 / R@10 / pair acc:

| pair | raw virtual | [qm] virtual |
|---|---|---|
| H&E↔mIF (real, 10 ch) | 0.644 / 0.979 / 0.863 | — |
| H&E↔mIF (real, 17 ch, reference) | 0.726 / 0.989 / 0.935 | — |
| mIF(10 ch)↔RNA(virtual) | 0.098 / 0.495 / 0.131 | — |
| mIF(10 ch)↔text(synthetic) | 0.074 / 0.434 / 0.071 | — |
| H&E↔mIF(virtual) | 0.038 / 0.264 / 0.046 | 0.042 / 0.278 / 0.047 |
| mIF↔mIF(virtual) | 0.045 / 0.248 / 0.054 | 0.052 / 0.301 / 0.060 |
| mIF(virtual)↔RNA(virtual) | 0.021 / 0.174 / 0.019 | 0.017 / 0.167 / 0.016 |
| mIF(virtual)↔text(synthetic) | 0.024 / 0.190 / 0.025 | 0.022 / 0.182 / 0.021 |
| TCGA text↔mIF(virtual) | 0.015 / 0.133 / 0.014 | 0.016 / 0.138 / 0.016 |

Unifying channels costs the real pairs ~0.08 R@1 (17 → 10 ch); matching the pixel distribution barely helps virtual mIF
(+0.004–0.007 R@1). The bottleneck is GigaTIME's per-tile accuracy, not channel naming or intensity scale.

## MIPHEI-ViT virtual mIF (replaces GigaTIME) — `35_miphei_infer.py`, `37_miphei_diag.py`
Model: Sanofi MIPHEI-ViT v1.0.0 (H-optimus-0 + LoRA + conv decoder; non-commercial academic licence), 16 ORION
channels (all but PD-1). It was trained on this ORION tile set with the official split (37/2/2 slides = our
`orion_official`), so only the official TEST tiles (CRC02, CRC11; 109 tiles) are clean for ORION; TCGA is fully clean.
Input 0.5 µm/px tiles reflect-padded to 512, output mapped back to the 8-bit ORION scale.

Tile-mean Spearman vs the real mIF (mean over channels):

| set | MIPHEI (16 ch) | MIPHEI (10 GigaTIME ch) | GigaTIME (10 ch) |
|---|---|---|---|
| clean (official test), pooled / within-slide | 0.697 / 0.654 | 0.690 / 0.645 | 0.249 / 0.301 |
| seen (official train) | 0.840 / 0.791 | 0.840 / 0.787 | 0.545 / 0.505 |

Direct real-mean ↔ virtual-mean retrieval (no head), R@1 in 100-galleries, clean test: MIPHEI 16 ch 0.272,
MIPHEI 10 ch 0.239, GigaTIME 10 ch 0.017 (seen tiles: 0.538 / 0.433 / 0.036).

### MIPHEI virtual mIF in the final protocol (`33_final_eval.py`, 16-ch heads trained on REAL ORION)
Clean = ORION official test (109 tiles, MIPHEI never saw them) and TCGA. R@1 / R@10 / pair acc:

| pair | ORION official test (clean) | ORION 5-fold (MIPHEI-seen, optimistic) | TCGA (clean) |
|---|---|---|---|
| H&E↔mIF real (16 ch) | 0.591 / 0.991 / 0.921 | 0.689 / 0.984 / 0.901 | — |
| H&E↔mIF(MIPHEI) | 0.189 / 0.784 / 0.553 | 0.255 / 0.722 / 0.509 | 0.133 / 0.455 / 0.213 |
| mIF real↔mIF(MIPHEI) | 0.297 / 0.800 / 0.586 | 0.379 / 0.762 / 0.669 | — |
| text↔mIF(MIPHEI) | — | — | 0.033 / 0.231 / 0.036 |
| RNA(virtual)↔mIF(MIPHEI) | 0.041 / 0.311 / 0.078 | 0.057 / 0.352 / 0.093 | 0.035 / 0.218 / 0.040 |
| mIF(MIPHEI)↔text(synthetic) | 0.050 / 0.396 / 0.051 | 0.049 / 0.314 / 0.058 | — |
| (real) mIF 16 ch↔RNA(virtual) | 0.101 / 0.466 / 0.123 | 0.108 / 0.512 / 0.149 | — |

GigaTIME on the same pairs: H&E↔mIF(virtual) 0.012–0.038 R@1, text↔mIF(virtual) 0.015 → MIPHEI is 5–15× better and
usable for validation (calibration: pair acc 0.55 vs 0.92 real on clean tiles ≈ 0.6×).

## External test: DLPFC (`36_external_prep.py`, `38_external_eval.py`, `39_external_check.py`, `results/external_*.csv`)
IDC results were removed from this report on 2026-10-18 at the user's request (the prepared data stays in cache/external).
- DLPFC: spatialLIBD Visium brain, 482 spots / 12 sections / 3 donors; gene names recovered exactly from the public h5.
  All 12 sections are in HEST (MISC1–12) -> the whole spatialLIBD study is REMOVED from HEST training for this test.
- Heads: H&E(UNI2)↔RNA trained on real HEST only (train/val by group; 5 heads), RNA z-scored per dataset.
  Virtual mIF: MIPHEI of the DLPFC H&E through the 16-ch heads trained on real ORION.
  Synthetic text: Azure `gpt-5.6-sol` captions of the same patches (same 3-shot prompt) through the TCGA real-pair text
  heads of 33_ (clean-text heads, runs/text_clean). Nothing is trained on external or virtual data.

5 heads, R@1 ± std / R@10 / pair acc; mixed galleries G=100 (chance 0.01) and within-section galleries G=35 (chance 0.029):

| pair | kind | mixed G=100 | within section G=35 |
|---|---|---|---|
| H&E↔RNA | real, trained pair | 0.010±0.003 / 0.107 / 0.013 | 0.031 |
| RNA↔mIF(MIPHEI) | emergent, real×virtual | 0.008 / 0.099 / 0.009 | 0.027 |
| RNA↔text(synthetic) | emergent, real×virtual | 0.010±0.002 / 0.117 / 0.011 | 0.032 |
| H&E↔mIF(MIPHEI) | ⚠ circular | 0.017 / 0.136 / 0.015 | 0.040 |
| H&E↔text(synthetic) | ⚠ circular | 0.022±0.002 / 0.179 / 0.025 | 0.054 |
| mIF(MIPHEI)↔text(synthetic) | ⚠ virtual×virtual | 0.014 / 0.144 / 0.017 | 0.038 |

- Every non-circular pair is at chance on DLPFC. Reference: HEST internal test restricted to brain = 0.021 (mixed organs 0.148).
- Caption audit: 59% of DLPFC captions say "lymphoid", only 6% recognise CNS/neuropil, 13% "no identifiable tissue"
  (blurry 112-µm crops) -> GPT cannot read these patches either.
- Checks: captions aligned row by row (482/482); shuffled nulls ≈ 1/G; rerun identical. Protocol check: our UNI2
  features of 3 DLPFC spots that also exist in the HEST cache have cosine 0.52–0.83 to the cached ones (random 0.12).

## Text cleaning + retrain on clean real text (v10) — `40_clean_text.py`, `33_`, `run_v10.sh`
Supersedes every TCGA-text number above (old results kept in results/before_text_clean/; old heads runs/text_all).
- Junk texts removed from train / val / test: 365 of 6,480 (353 bare answer letters "A A B B", 4 "error error",
  8 answers < 5 words). 256 of 6,144 TCGA images were left without real text; they keep their split, get a few-shot GPT
  caption (27_ --sets tcga_notext, USD 0.81; cumulative USD 65.09 of 100) and are used for evaluation only.
- Text heads retrained on clean real pairs only (runs/text_clean/uni2_biomedclip_fold<k>; ~4,400 train pairs / fold).

R@1 / R@10 / pair acc, test split, mean of 5 folds (before → after cleaning):

| pair | before | after |
|---|---|---|
| TCGA H&E↔text (real) | 0.229 / 0.700 / 0.238 | **0.237±0.012 / 0.730 / 0.248** |
| TCGA text↔RNA(virtual) | 0.055 / 0.318 / 0.058 | 0.056 / 0.319 / 0.057 |
| TCGA text↔mIF(MIPHEI) | 0.033 / 0.231 / 0.036 | 0.036 / 0.237 / 0.038 |
| calibration set: H&E↔text real vs synthetic | 0.197 vs 0.166 | 0.219 vs 0.162 |
| ORION 5-fold mIF↔text(synthetic) | 0.074 / 0.434 / 0.071 | 0.074 / 0.437 / 0.072 |
| ORION 5-fold H&E↔text(synthetic) | 0.075 / 0.439 / 0.101 | 0.076 / 0.441 / 0.102 |

Part of the gain on TCGA comes from removing junk queries from the test split (n test pairs 1229 → 1178).
Images without real text (256, pooled over test folds, each image through its own test-fold head):
H&E↔text(synthetic) 0.150 / 0.557 / 0.160 (⚠ circular; consistent with 0.162 on the calibration set);
text(synthetic)↔RNA(virtual) 0.051; text(synthetic)↔mIF(MIPHEI) 0.036 (⚠ virtual×virtual).

## Best encoder per modality — re-check under the final protocol (`41_text_enc_clean.py`, paired per-fold tests)
mIF (17_bench_fm / 28_, ORION patient_5fold) and RNA (23_, HEST group 5-fold) comparisons already used the final-protocol
splits -> re-analysed per fold. Text was re-run on CLEAN real text with the 33_ recipe and splits (UNI2+BiomedCLIP
reproduces 33_ fold by fold: 0.234 / 0.236 / 0.248 / 0.219 / 0.248). R@1 g100 mean of 5 folds; Δ = best − other,
folds won, paired t-test (5 folds; 5/5 wins = sign-test p 0.06).

| modality | best | runner-up (Δ, folds won, p) | verdict |
|---|---|---|---|
| text tower (H&E↔text, clean) | **BiomedCLIP** 0.237 (UNI2 image) | CONCH text 0.216 (+0.021, 5/5, p=0.03); PLIP 0.211 (+0.026, 5/5, p=0.007); ImageBind 0.201 (+0.036, 5/5, p=0.002) | clear winner for every image anchor |
| H&E image for text | Phikon-v2 0.241 ≈ CONCH 0.241 ≈ UNI2 0.237 | GigaPath 0.223 (UNI2 +0.014, 4/5, p=0.05) | 3-way tie, GigaPath worse |
| H&E image for mIF | **UNI2** 0.727 (VirTues) / 0.753 (ConvNeXt) | GigaPath (+0.009 / +0.013, ns); CONCH (+0.155 / +0.114, 5/5, p<0.001) | UNI2 ≈ GigaPath ≫ CONCH |
| H&E image for RNA | **UNI2** 0.148 | GigaPath 0.113 (+0.035, 4/5, p=0.05) | UNI2 |
| mIF encoder (H&E↔mIF) | **ConvNeXt-T** scratch 0.758 | VirTues 0.728 (+0.030, 5/5, p=0.07) | ConvNeXt-T consistently ahead; VirTues kept for cross-modal transfer / channel flexibility |
| RNA encoder (H&E↔RNA) | Geneformer-V2 0.159 | scGPT 0.154 (+0.005, 3/5, p=0.36); MLP 0.148 (+0.011, 4/5, p=0.05) | tie |

UNI2 is the only H&E anchor never significantly beaten on any pair (Phikon-v2 has no mIF/RNA runs: HEST ships only
UNI2/GigaPath features). Files: results/text_enc_clean_{long,summary}.csv.

## Training regimes: separate vs joint (`42_train_regimes.py`, results/regimes_{long,summary}.csv)
Real pairs only, final-protocol splits (ORION patient 5-fold with the 16-ch VirTues panel, HEST group 5-fold, TCGA
patient 5-fold on clean text). Same heads, loss and optimiser for every regime; virtual data only at test time.
R@1 g100, mean of 5 folds; "rel" = mean of R@1 / S1's R@1 over the trained (3) or emergent (4) pairs.

| regime | H&E↔mIF | H&E↔RNA | H&E↔text | mIF↔RNA(v) | mIF↔text(s) | text↔RNA(v) | text↔mIF(MIPHEI) | trained rel | emergent rel |
|---|---|---|---|---|---|---|---|---|---|
| S1 separate, frozen anchor (current) | 0.680 | 0.148 | 0.234 | 0.108 | 0.076 | 0.060 | 0.038 | 1.00 | 1.00 |
| S2 separate, two-tower (CLIP) | **0.794** | 0.130 | 0.195 | 0.019 | 0.022 | 0.014 | 0.013 | 0.96 | 0.26 |
| S3 separate, frozen, linear heads | 0.642 | 0.156 | 0.237 | **0.123** | 0.070 | 0.069 | 0.038 | 1.00 | 1.05 |
| S4 separate, frozen, MLP mIF + linear RNA/text | 0.680 | 0.152 | 0.234 | 0.117 | 0.074 | **0.071** | 0.037 | 1.01 | 1.05 |
| J1 joint, frozen anchor | 0.706 | 0.148 | **0.237** | 0.094 | 0.065 | 0.056 | 0.031 | 1.02 | 0.87 |
| J2 joint, shared trainable H&E head | 0.761 | 0.147 | 0.204 | 0.061 | 0.060 | 0.044 | 0.031 | 0.99 | 0.72 |
| J3 J2, size-proportional sampling | 0.740 | 0.138 | 0.187 | 0.063 | 0.060 | 0.039 | 0.031 | 0.94 | 0.70 |
| J4 J2 + anchor distillation λ=1 | 0.765 | 0.144 | 0.202 | 0.062 | 0.058 | 0.043 | 0.030 | 0.99 | 0.71 |
| J4b J2 + anchor distillation λ=10 | 0.768 | 0.149 | 0.209 | 0.067 | 0.058 | 0.047 | 0.030 | 1.01 | 0.74 |
| J5 joint, residual adapter on UNI2 | 0.762 | 0.142 | 0.201 | 0.058 | 0.059 | 0.044 | 0.029 | 0.98 | 0.71 |
| J6 J2 + uncertainty weighting | 0.765 | 0.136 | 0.195 | 0.057 | 0.055 | 0.041 | 0.027 | 0.96 | 0.66 |
| J7 two-stage: S1 → joint residual fine-tune | 0.776 | 0.150 | 0.234 | 0.100 | 0.076 | 0.055 | 0.035 | 1.05 | 0.94 |
| **J8 two-stage: S4 → joint residual fine-tune** | 0.762 | **0.164** | 0.232 | 0.112 | 0.073 | 0.068 | 0.035 | **1.07** | **1.01** |

Pair accuracy (H&E↔mIF / RNA / text): S1 0.897 / 0.172 / 0.242 → J8 0.929 / 0.198 / 0.237.
J8 vs S1 (paired, 5 folds): H&E↔mIF +0.083 (5/5, p=0.01), H&E↔RNA +0.015 (5/5, p=0.03), text↔RNA(v) +0.008
(5/5, p=0.02), H&E↔text −0.002, mIF↔RNA(v) +0.004, mIF↔text(s) −0.004, text↔mIF −0.003 (all ns).
- Training the H&E side from scratch (S2, J2–J6) helps H&E↔mIF (+0.08–0.11) but costs H&E↔text (−0.03 to −0.05) and
  26–74% of the emergent performance: the shared H&E head is pulled towards the easiest / strongest pair (mIF).
  Size-proportional sampling and uncertainty weighting make this worse; distillation to the UNI2 geometry helps a little.
- Joint training with a frozen anchor (J1) only changes the step budget: +0.026 mIF, −13% emergent.
- Linear heads for RNA / text generalise better across modalities than MLP heads (S3/S4: emergent +5%).
- Best: **J8 = first train every head separately on the frozen UNI2 anchor (MLP for mIF, linear for RNA and text), then
  fine-tune all heads jointly with a shared zero-initialised residual adapter on UNI2** (lr 1e-4, 1000 steps, sum of
  the three losses, similarity distillation λ=1, joint early stopping on the geometric mean of the val R@1).

## More RNA / text encoders + multi-encoder late fusion (`43_*`, `44_rnafm_eval.py`, `45_*`, `46_text_enc_eval.py`, `47_final_ens.py`)
All encoders frozen; one head per encoder into the frozen UNI2 space (S4 recipe of 42_: linear for RNA / text), same
splits and test items as 42_. Late fusion = mean of the L2-normalised head outputs of several encoders of one modality
(possible because every head lands in the same frozen space). R@1 g100, mean of 5 folds; virtual / synthetic = test only.
RNA inputs: cache/rnafm/<model>_<set>.npy (rna_sets.py: hest real, STPath virtual = round(expm1(pred)), DLPFC real);
FM inputs z-scored with HEST-train stats (DLPFC: per dataset, heads retrained without spatialLIBD).

RNA encoders (linear head):

| RNA encoder | H&E↔RNA | mIF↔RNA(v) | text↔RNA(v) | H&E↔RNA(v) (HEST, calib) | DLPFC H&E↔RNA |
|---|---|---|---|---|---|
| hvg 2000 (current) | 0.155 | 0.124 | 0.069 | 0.198 | 0.009 |
| AIDO.Cell-100M (MLP head 0.171) | 0.169 | 0.094 | 0.049 | 0.083 | 0.010 |
| Tahoe-x1 70M | 0.166 | 0.087 | 0.039 | 0.070 | 0.011 |
| Geneformer-V2-104M | 0.164 | 0.071 | 0.037 | 0.064 | 0.010 |
| scGPT-human | 0.155 | 0.087 | 0.050 | 0.096 | 0.013 |
| C2S-Scale-Gemma-2-2B (mean / last token) | 0.155 / 0.142 | 0.101 / 0.101 | 0.055 / 0.051 | 0.111 / 0.097 | 0.011 |
| Nicheformer | 0.152 | 0.104 | 0.056 | 0.097 | 0.006 |
| SE-600M (STATE) | 0.150 | 0.117 | 0.062 | 0.143 | 0.011 |
| GenePT-w (bge / BiomedCLIP gene-summary embeddings) | 0.141 / 0.130 | 0.093 / 0.075 | 0.056 / 0.052 | 0.116 / 0.086 | 0.009 |
| seed control: hvg x 5 seeds, averaged | 0.161 | 0.132 | 0.073 | 0.208 | 0.008 |
| fusion hvg + Tahoe-x1 + SE-600M + AIDO.Cell | 0.192 | 0.142 | 0.075 | 0.174 | 0.012 |
| fusion all 10 | 0.199 | 0.148 | 0.078 | 0.163 | 0.012 |
| early fusion (concat) hvg + Tahoe-x1 | 0.169 | 0.114 | 0.059 | 0.131 | 0.009 |

- Single FMs gain up to +0.016 on real H&E↔RNA but lose on every virtual-RNA pair; the calibration column shows why:
  the FMs embed STPath virtual RNA far from the real RNA of the same spot (H&E↔RNA(v) 0.06–0.14 vs 0.198), so the
  emergent drop reflects the virtual test data more than the encoders.
- Late fusion beats early fusion (concat) and the seed-ensemble control by a wide margin -> the encoders are complementary.

Text encoders (TCGA clean real text, linear head; 45_ / 45b_; KEEP and Quilt-1M corpora contain no PathGen / TCGA text):

| text encoder | H&E↔text | mIF↔text(s) | text↔RNA(v) | text↔mIF(MIPHEI) |
|---|---|---|---|---|
| BiomedCLIP (current; z-scored control 0.235) | 0.236 | 0.077 | 0.069 | 0.038 |
| KEEP (MLP head 0.267) | 0.278 | 0.077 | 0.084 | 0.041 |
| Qwen3-Embedding-0.6B / 4B | 0.264 / 0.258 | 0.062 / 0.062 | 0.075 / 0.067 | 0.038 / 0.033 |
| PubMedBERT-emb / bge-large / BioLORD-2023 / MedCPT-Article / MedEmbed-large | 0.262 / 0.259 / 0.258 / 0.257 / 0.252 | 0.054–0.063 | 0.068–0.076 | 0.034–0.037 |
| QuiltNet-B-32 (QuickGELU) | 0.218 | 0.058 | 0.069 | 0.035 |
| seed control: KEEP x 5 seeds | 0.284 | 0.082 | 0.085 | 0.045 |
| fusion KEEP + BiomedCLIP + Qwen3-0.6B | 0.299 | 0.091 | 0.088 | 0.047 |
| fusion + PubMedBERT-emb + MedCPT | 0.303 | 0.088 | 0.091 | 0.047 |

Final combination (47_final_ens.py; mIF = VirTues 16-ch MLP head in all; paired vs S4 over 5 folds):

| pair | S4 | AIDO.Cell + KEEP | ens-small (4 RNA + 3 text) | ens-large (10 RNA + 5 text) | ens-small vs S4 |
|---|---|---|---|---|---|
| H&E↔mIF | 0.677 | 0.677 | 0.677 | 0.677 | unchanged |
| H&E↔RNA | 0.155 | 0.169 | 0.192 | 0.199 | +0.037 (5/5, p=0.0006) |
| H&E↔text | 0.236 | 0.278 | 0.299 | 0.303 | +0.062 (5/5, p<0.0001) |
| mIF↔RNA(v) | 0.124 | 0.094 | 0.142 | 0.148 | +0.018 (5/5, p=0.0002) |
| mIF↔text(s) | 0.077 | 0.077 | 0.091 | 0.088 | +0.014 (5/5, p=0.011) |
| text↔RNA(v) | 0.069 | 0.052 | 0.095 | 0.098 | +0.027 (5/5, p=0.0009) |
| text↔mIF(MIPHEI) | 0.038 | 0.041 | 0.047 | 0.047 | +0.010 (5/5, p=0.008) |
| DLPFC H&E↔RNA / RNA↔text(s) / RNA↔mIF(v) | 0.009 / 0.012 / 0.011 | 0.010 / 0.010 / 0.008 | 0.012 / 0.015 / 0.009 | 0.012 / 0.011 / 0.006 | still chance |

Pair accuracy ens-small vs S4: H&E↔RNA 0.182→0.246, H&E↔text 0.245→0.324, mIF↔RNA(v) 0.191→0.198, text↔RNA(v) 0.077→0.100.
Recommendation: ens-small (nearly all of ens-large's gain at 4 + 3 heads). Licences: AIDO.Cell (GenBio community licence,
non-commercial; restricts using its embeddings to train "similar" models) and SE-600M (Arc non-commercial) are
research-only; KEEP / QuiltNet / Nicheformer MIT, Tahoe-x1 / Qwen3-Embedding Apache-2.0, C2S-Scale CC-BY-4.0.
Not yet tested: fusion inside the J8 two-stage regime (J8's mIF gain should stack, unverified).

## Joint training of the fused model (`48_joint_ens.py`, results/joint_ens_{long,summary,tests}.csv)
Does the J8 two-stage gain stack with multi-encoder fusion? Same splits / test items; all configs of a fold start from
the SAME stage-1 heads (separate, frozen UNI2). Stage 2 = J8 recipe (shared zero-init residual adapter on UNI2, sum of
the 3 pair losses, similarity distillation λ=1, lr 1e-4, 1000 steps, joint early stopping). E = ens-small of 47_.
EJ-fused: loss on the fused modality embedding; EJ-each: each encoder head keeps its own loss (fusion only at test);
EJ-frz: EJ-each with RNA / text heads frozen (only adapter + mIF head + temperatures train). R@1 g100, 5 folds.

| config | H&E↔mIF | H&E↔RNA | H&E↔text | mIF↔RNA(v) | mIF↔text(s) | text↔RNA(v) | text↔mIF(MIPHEI) |
|---|---|---|---|---|---|---|---|
| S4 separate, single encoders | 0.675 | 0.155 | 0.236 | 0.122 | 0.076 | 0.069 | 0.038 |
| J8 joint, single encoders | 0.776 | 0.159 | 0.238 | 0.122 | 0.073 | 0.065 | 0.035 |
| E separate, fused encoders | 0.675 | 0.192 | 0.299 | **0.141** | **0.089** | **0.095** | **0.046** |
| EJ-fused | 0.792 | 0.178 | 0.292 | 0.118 | 0.080 | 0.060 | 0.035 |
| EJ-each | 0.790 | **0.202** | **0.304** | 0.128 | 0.086 | 0.086 | 0.039 |
| **EJ-frz** | **0.793** | 0.201 | **0.304** | 0.128 | 0.088 | **0.095** | 0.040 |

Pair accuracy H&E↔mIF / RNA / text: S4 0.899 / 0.182 / 0.245, E 0.899 / 0.246 / 0.324, EJ-frz 0.936 / 0.255 / 0.320.
- Joint stage-2 gains stack on the trained pairs: EJ-frz vs E H&E↔mIF +0.118 (5/5, p<0.001), H&E↔RNA +0.009 (5/5,
  p=0.01), H&E↔text +0.005 (ns); vs S4 all three trained pairs +0.05–0.12 (5/5).
- Price: the pairs that involve the fine-tuned mIF head lose vs E: mIF↔RNA(v) −0.013 (0/5, p=0.04), text↔mIF −0.007
  (0/5, p=0.02); mIF↔text(s) −0.001 (ns). text↔RNA(v) unchanged (both heads frozen).
- Loss on the fused embedding (EJ-fused) is clearly worse than per-encoder losses (EJ-each: 5/5 better on 6 pairs):
  the fused loss lets single heads specialise, which breaks the cross-modal geometry (text↔RNA 0.095 → 0.060).
- Freezing RNA / text heads (EJ-frz) costs nothing on the trained pairs and keeps text↔RNA at 0.095 (vs EJ-each 0.086,
  5/5, p=0.03): the adapter alone does the H&E-side work.
- Recommendation: EJ-frz if H&E↔mIF matters (+0.12) and a −0.01 on the virtual mIF emergent pairs is acceptable;
  E (pure separate) if the emergent mIF pairs are the priority.

## MedVTok as H&E image information + adapter-only joint training (`49_`–`51_`, results/medvtok_{long,summary,tests}.csv)
MedVTok (arXiv 2610.35842; github Masaaki-75/medvtok @a558c24, HF massaki75/medvtok, MIT): general medical visual
tokenizer, SigLIP2-So400m-initialised encoder (1152-d), aligned (MERA) to frozen expert encoders, CONCH v1.5 for
histopathology; no image-text pairs. Pathology training images: HISTAI 4.0M, **PathGen 1.61M**, bone-marrow
cytomorphology, LYSTO. => it has seen the TCGA/PathGen test IMAGES (not their captions); ORION / HEST / DLPFC not listed.
Loaded with weights_only=True + strict=True; reconstruction PSNR on ORION H&E 32–42 dB; feature = mean-pooled tokens ->
LN -> Linear ("feats", 1152-d), 256 px, [-1, 1]. HEST images: 224-px level-0 crops from the HEST WSIs (`49_`, HTTP range
reads of only the needed TIFF tiles); re-encoding them with UNI2-h reproduces cache/hest/he_uni2.npy (cos ≥ 0.9994 for
all 9,484 spots), i.e. MedVTok sees exactly the anchor's images. DLPFC: the 0.5-µm crops behind dlpfc_he_uni2 (cos 1.000).
Configs (RNA / text ens-small, mIF VirTues MLP; same splits / test items; R@1 g100, 5 folds):
E separate UNI2 anchor; EJ-frz (= 48_); EJ-ad = EJ-frz with the mIF head frozen too (only the residual H&E adapter and
the temperatures train); MV-E = MedVTok instead of UNI2 as the frozen anchor; +MV = adapter input [UNI2, c·MedVTok];
+RP = control, adapter input [UNI2, random projection of UNI2 to 1152-d at the same norm] (wider adapter, no new info).

| config | H&E↔mIF | H&E↔RNA | H&E↔text | mIF↔RNA(v) | mIF↔text(s) | text↔RNA(v) | text↔mIF | DLPFC H&E↔RNA |
|---|---|---|---|---|---|---|---|---|
| E | 0.675 | 0.192 | 0.299 | 0.141 | 0.089 | 0.095 | 0.046 | 0.012 |
| EJ-frz | 0.786 | 0.202 | 0.305 | 0.134 | 0.087 | 0.095 | 0.042 | |
| EJ-ad | 0.776 | 0.201 | 0.299 | 0.141 | 0.089 | 0.095 | 0.046 | |
| MV-E (MedVTok anchor) | 0.658 | 0.131 | 0.293 | 0.081 | **0.113** | 0.056 | **0.052** | 0.012 |
| EJ-frz+MV | **0.811** | 0.200 | 0.307 | 0.125 | 0.084 | 0.095 | 0.041 | |
| **EJ-ad+MV** | 0.792 | 0.200 | **0.315** | 0.141 | 0.089 | 0.095 | 0.046 | |
| EJ-ad+RP (control) | 0.778 | 0.202 | 0.298 | 0.141 | 0.089 | 0.095 | 0.046 | |

Pair accuracy H&E↔mIF / RNA / text: E 0.899 / 0.246 / 0.324, EJ-ad 0.929 / 0.256 / 0.313, EJ-ad+MV 0.937 / 0.252 / 0.338.
- EJ-ad (only the H&E adapter moves): every emergent pair is identical to E by construction (all modality heads frozen)
  while H&E↔mIF +0.101 (5/5, p<0.001) and H&E↔RNA +0.009 (5/5, p=0.01). vs EJ-frz: H&E↔mIF −0.010 (ns), H&E↔text −0.006
  (0/5, p=0.04), but mIF↔RNA(v) +0.007 (5/5, p=0.045) and text↔mIF +0.005 (ns) -> the emergent losses of EJ-frz came
  entirely from moving the mIF head.
- MedVTok as the anchor (MV-E) is worse: H&E↔RNA −0.061 (0/5), RNA emergent pairs −0.04/−0.06 (0/5); only the pairs
  with synthetic GPT text / virtual mIF gain (mIF↔text(s) +0.024, text↔mIF +0.006, 5/5) - CONCH-v1.5 semantics.
  DLPFC stays at chance (0.012) with either anchor.
- MedVTok as extra adapter input: EJ-ad+MV vs EJ-ad H&E↔mIF +0.015 (4/5, p=0.04), H&E↔text +0.016 (5/5, p=0.02),
  H&E↔RNA −0.002 (ns), emergent unchanged. The control shows it is MedVTok's information, not adapter width:
  EJ-ad+RP = EJ-ad (all |Δ| ≤ 0.002, ns); EJ-ad+MV vs EJ-ad+RP: H&E↔mIF +0.014 (5/5, p=0.03), H&E↔text +0.017 (5/5, p=0.01).
  Caveat: the H&E↔text test images are PathGen images MedVTok was pretrained on (image-only, aligned to CONCH v1.5);
  the ORION H&E↔mIF gain is free of that.
- EJ-frz+MV has the best H&E↔mIF (0.811) but again loses the mIF emergent pairs (mIF↔RNA(v) −0.017 vs E, 0/5).
- Recommendation: **EJ-ad+MV** (stage 1 separate heads on frozen UNI2; stage 2 trains only a residual H&E adapter that
  reads UNI2 + MedVTok). Every non-H&E embedding is unchanged; H&E↔mIF / text improve. Cost: a 2nd image encoder
  (MedVTok encoder 417M params) at H&E inference. EJ-ad (no MedVTok) if a single image encoder is required.

## Final model: training, stage-2 selection rule, inference API (`52_final_model.py`, `53_embed_check.py`, `embed.py`)
Configuration EJ-ad+MV (previous section). Stage 1 is deterministic (52_ reproduces 51_'s stage-1 heads exactly: the
four emergent pairs of fold 0 are identical to 51_). Stage 2 is not: two identical fold-0 runs gave H&E↔mIF 0.791 and
0.741, because the val geo-mean is flat after ~100 steps and the argmax picks very different steps (100–700).
`52_ cv` (5 folds × 3 stage-2 seeds, same stage-1 heads, same test items; results/final_cv_{long,summary,tests,curves}.csv):

| stage-2 rule | H&E↔mIF | H&E↔RNA | H&E↔text | geo-mean | pair acc mIF / RNA / text | seed std mIF / text |
|---|---|---|---|---|---|---|
| stage 1 only (= E) | 0.675 | 0.192 | 0.299 | 0.337 | 0.899 / 0.246 / 0.324 | – |
| es (51_ rule, 1 seed) | 0.794 | 0.201 | 0.309 | 0.366 | 0.937 / 0.253 / 0.329 | 0.009 / 0.011 |
| last (1000 steps, 1 seed) | 0.802 | 0.202 | 0.301 | 0.364 | 0.939 / 0.254 / 0.314 | 0.002 / 0.003 |
| **ens-es (mean of 3 early-stopped adapters)** | 0.801 | 0.203 | **0.323** | **0.373** | **0.942 / 0.258 / 0.347** | ensemble |
| ens-last | **0.807** | **0.204** | 0.309 | 0.370 | 0.941 / 0.255 / 0.322 | ensemble |

ens-es vs es: H&E↔text +0.015 (5/5, p=0.02), mIF +0.007, RNA +0.002 (ns); ens-es vs ens-last: text +0.014 (5/5,
p=0.004), mIF −0.006 (0/5, p=0.03). Emergent pairs are identical for every rule (heads frozen): mIF↔RNA(v) 0.141,
mIF↔text(s) 0.089, text↔RNA(v) 0.095, text↔mIF 0.046. -> **ens-es** is the final rule (3 adapters, H&E = UNI2 +
mean_s A_s([UNI2, c·MedVTok]); each adapter 2.4M parameters).
Final 5-fold test performance (R@1 / R@10 / pair acc): H&E↔mIF 0.801±0.043 / 0.991 / 0.942, H&E↔RNA 0.203±0.033 /
0.625 / 0.258, H&E↔text 0.323±0.007 / 0.813 / 0.347; emergent as above (chance 0.01 / 0.10 / 0.01).

Saved models (`52_ fold0` / `52_ release`, rule ens-es):
- `models/ej_ad_mv_fold0/` fold-0 train / val, test held out: test R@1 mIF 0.783, RNA 0.192, text 0.320 (51_ fold 0:
  0.768 / 0.187 / 0.316; emergent identical).
- `models/ej_ad_mv/` **release**: train = fold-0 train + test (ORION 3000 tiles, HEST 8938 spots, TCGA 5615 texts),
  val = fold-0 val (187 / 546 / 500) for early stopping. No own test split: expected performance = the 5-fold CV above.
  model.pt (heads, 3 adapters, feature-normalisation statistics) + meta.json (dims, HVG list, split sizes, val scores).

`embed.py` (Embedder): raw-input embedding of all four modalities (H&E images → UNI2-h + MedVTok; text → KEEP +
BiomedCLIP + Qwen3-Emb-0.6B; RNA counts + gene symbols → hvg + Tahoe-x1 + SE-600M + AIDO.Cell; mIF tiles (16 ORION
markers) + slide ids → VirTues), or from precomputed features; output L2-normalised 1536-d vectors. The raw-input
pre-processing reuses the extraction code of 04b_/14_/43_/45_/45b_/50_. `53_embed_check.py` (results/embed_check.csv):
raw-input features vs cached training features, per-item cosine median (min): UNI2 1.000 (1.000) on ORION / PathGen /
HEST; MedVTok 1.000 (≥0.9997); KEEP 1.000, BiomedCLIP 1.000 (0.929, bf16 batch effects), Qwen3 0.9998; hvg 1.000,
AIDO.Cell 1.000, Tahoe-x1 0.991 (0.929) and SE-600M 0.994 (0.939) (both sample / tie-break genes at random, as in their
official collators); VirTues 1.000 (all 3187 ORION tiles). End-to-end with models/ej_ad_mv_fold0 on held-out fold-0
test items (300 each), g100 R@1 raw-input vs cached features: H&E↔mIF 0.792 / 0.792, H&E↔RNA 0.194 / 0.194,
H&E↔text 0.335 / 0.336.

## Release on the Hugging Face Hub and reproduction check (2026-10-28)

All three repos are private:
- Model, code and results: `Pathstudy/fourmodal-pathology-bind`. The README is the model card in Chinese: data, architecture, results and usage.
- Features and splits (966 MB): `datasets/Pathstudy/fourmodal-pathology-features`.
- Report: the Space `Pathstudy/imagebind-pathology-report`. Its `index.html` is the final report; the earlier report is kept as `previous_report.html`.

New scripts:
- `evaluate_saved.py`: feature-level re-evaluation of a saved checkpoint.
- `example_check.py` + `examples/`: a raw-input check on 64 fold-0 test items per dataset, built by `54_make_examples.py`.
- `reproduce.sh` and `compare_reproduction.py`: retrain from the features and compare with the shipped numbers.
- `setup_encoders.sh`: clones VirTues, ModelGenerator and MedVTok at the pinned commits and fetches the MedVTok weights.
- `requirements.txt`.

Code changes:
- `52_final_model.py --out NAME`, so a retraining does not overwrite the shipped checkpoints or results.
- `embed.py`: `$UNI2_PATH`, falling back to the gated HF repo.
- `43_rnafm_tahoex1.py`: the Geneformer name→ID dictionary is now loaded with `hf_hub_download` at a pinned revision. It used to be globbed from the local HF cache.

Check run in a fresh directory, downloaded from the Hub, with a new uv virtualenv (Python 3.11, torch 2.7.1+cu126) on an RTX A5000:
- **`evaluate_saved.py`:** equal to `test.csv` (max |diff| 8e-17).
- **Fold-0 and 5-fold retraining:** every pair is identical to the shipped numbers, stage 1 and ens-es (|diff| 0).
- **Release retraining:** the same early-stopping steps, [200, 200, 900].
- **`example_check.py`:** all encoders reproduce the cached features. Cosine is 1.000 except Tahoe-x1 .991, SE-600M .995 and VirTues .999 (64-tile standardisation).
- **R@1 within 64 items, raw inputs vs cached features:**

  | Pair | Raw inputs | Cached features |
  |---|---|---|
  | H&E↔RNA | .227 | .227 |
  | H&E↔text | .398 | .398 |
  | H&E↔mIF | .828 | .875 |

- **Bug found by the fresh run and fixed:** Tahoe-x1 raw input needed `cache/external/dlpfc_genes_full.csv`. That file now ships with the model repo.
- **Not covered:** the frozen-encoder weights came from the existing HF cache on this machine, and UNI2-h from its local copy. A new machine downloads them from the Hub at the pinned revisions, and UNI2-h needs gated access.