RESULTS.md

Results — ImageBind-style multimodal binding on ORION (1% subsample)

Metric: R@1 on random 100-tile galleries (chance = 0.01), mean over the two retrieval directions unless noted; patient 5-fold reported as mean ± std over folds. "Pairing acc" = one-to-one Hungarian matching on the full test set. Your split (orion_official) has 109 test tiles from 2 slides → single, noisy estimate.

1. Base experiment (H&E / IF / nuclei / markers, ImageBind-Huge H&E anchor) — 22 runs

Your split, R@1 (full 109-tile gallery):

pair	anchor loss (ImageBind)	all-pairs loss
H&E→IF	0.49 (pair acc 0.73)	0.38
H&E→nuclei	0.36	0.28
H&E→markers	0.16	0.15
IF→nuclei (emergent)	0.41	0.54
IF→markers (emergent)	0.26	0.70
nuclei→markers (emergent)	0.15	0.29

• Emergent pairs (never trained together) match far above chance.
• Training the non-H&E pairs directly (all-pairs) boosts them but costs H&E→IF.
• Patient 5-fold ≈ leave-one-IF-batch-out → small batch effect; random tile split is optimistic.

2. H&E encoder ablation (anchor loss) — 33 runs

Mean R@1 (g100) over H&E pairs / emergent pairs:

H&E encoder	your split	patient 5-fold
ImageBind-Huge	0.29 / 0.25	0.33 ± 0.01 / 0.30 ± 0.00
PLIP (stand-in for CONCH)	0.35 / 0.29	0.37 ± 0.01 / 0.31 ± 0.01
Phikon-v2 (stand-in for GigaPath)	0.41 / 0.31	0.45 ± 0.02 / 0.36 ± 0.01
UNI2-h	0.40 / 0.34	0.48 ± 0.02 / 0.37 ± 0.01

CONCH and GigaPath: HF access refused (GatedRepoError); code ready (04b_extract_anchor_fm.py).

3. Encoder benchmark (H&E = UNI2 frozen; mIF / markers / text) — 18 runs

Text = templated per-tile descriptions generated from marker counts + nuclei count (no real text exists for ORION).

Trained pairs (H&E ↔ modality), R@1 g100

modality	encoder	your split	patient 5-fold
mIF	ResNet18 (17-ch stem)	0.59	0.64 ± 0.01
mIF	ConvNeXt-Tiny (17-ch stem)	0.73 (pair acc 0.95)	0.78 ± 0.02
mIF	per-channel encoder (shared CNN + channel transformer)	0.51	0.62 ± 0.01
markers	linear	0.18	0.22 ± 0.01
markers	MLP	0.31	0.39 ± 0.03
markers	FT-Transformer	0.21	0.30 ± 0.02
text	PLIP text	0.17	0.20 ± 0.02
text	BiomedCLIP text	0.16	0.19 ± 0.02
text	ImageBind text	0.22	0.21 ± 0.02
text	PLIP / ImageBind zero-shot (no training)	≈0.01–0.02	≈0.01–0.02 (chance)

Cross pairs (never trained together), best combos, patient 5-fold R@1 g100

pair	best combo	R@1
mIF ↔ markers	ConvNeXt-T + MLP / ResNet18 + MLP	0.48 ± 0.03
markers ↔ text	MLP + PLIP / MLP + ImageBind	0.43 ± 0.07 / 0.42 ± 0.03
mIF ↔ text	ResNet18/ConvNeXt-T + ImageBind	0.25 ± 0.01

Takeaways

• mIF: ConvNeXt-Tiny clearly best (+0.14 over ResNet18). The channel-agnostic per-channel encoder is not better here (fixed 17-marker panel; it would matter for transfer across panels).
• Markers: the plain MLP wins; FT-Transformer < MLP, linear worst. MLP markers also give the best cross-modal pairs.
• Text: the three text towers are within noise (0.19–0.21); ImageBind text slightly ahead. Zero-shot pathology CLIP (PLIP) cannot match these synthetic marker-based descriptions to H&E (≈ chance) → training a projection is required.
• Text is derived from markers, so markers↔text is partly by construction; H&E↔text measures how well H&E predicts coarse cell composition. Real paired text (PathGen images or ORION reports) is needed for a true text evaluation.
• Gene expression: none in ORION; the marker-encoder benchmark would be repeated on HEST/DLPFC for RNA.

Files: results/bench_summary.csv, results/bench_long.csv, results/anchor_ablation_*.csv, results/summary_*.csv.

---

v2 / v3 — re-sampled ORION (rsync 2026-10-11) + real text + modality-specific foundation models

v1 sections above used the old ORION 1% sample (archived in v1_old_orion_sample/). Same slide split, new tiles. H&E anchor = UNI2 (frozen) throughout. R@1 on 100-tile galleries (chance 0.01); patient 5-fold mean ± std.

Text bound to H&E with real captions (PathGen-1.6M, patient split, 255 test pairs) — 12_train_text_pathgen.py

text tower (frozen) + MLP head	H&E→text R@1 (full 255)	g100	zero-shot (own image tower)
BiomedCLIP	0.17	0.29	–
PLIP	0.13	0.23	0.035
ImageBind	0.11	0.22	0.024
PathGen-CLIP-L	0.13	0.24	0.19*
*PathGen-CLIP-L was trained on PathGen-1.6M → its PathGen numbers are not held-out.

mIF / markers: from-scratch vs modality-specific foundation models — 17_bench_fm.py, 18_eval_fm.py

pair	encoder	your split (pair acc)	patient 5-fold
H&E↔mIF	ConvNeXt-T (scratch)	0.73 (0.96)	0.76 ± 0.02
H&E↔mIF	VirTues sp32 (frozen) + MLP	0.67 (0.95)	0.73 ± 0.03
H&E↔markers	MLP on raw markers	0.28	0.38 ± 0.03
H&E↔markers	Geneformer V2 (frozen) + MLP	0.12	0.17 ± 0.01
H&E↔markers	scGPT human (frozen) + MLP	0.11	0.18 ± 0.01
mIF↔markers (emergent)	ConvNeXt-T + MLP	0.37	0.48 ± 0.02
mIF↔markers (emergent)	VirTues + MLP	0.51 (0.93)	0.66 ± 0.02
mIF↔markers (emergent)	VirTues + scGPT / + Geneformer	0.22 / 0.19	0.29 / 0.26

Emergent text ↔ mIF / markers (text head trained only on PathGen; never sees ORION/IF/markers) — patient 5-fold AUROC (Spearman), 7 concept prompts

text	H&E (UNI2)	mIF ConvNeXt	mIF VirTues	markers MLP	markers scGPT	markers Geneformer
PLIP	0.69 (0.32)	0.69 (0.31)	0.71 (0.34)	0.76 (0.46)	0.79 (0.49)	0.74 (0.41)
ImageBind	0.67 (0.32)	0.65 (0.27)	0.67 (0.31)	0.73 (0.41)	0.75 (0.45)	0.70 (0.35)
BiomedCLIP	0.66 (0.25)	0.66 (0.24)	0.68 (0.26)	0.71 (0.29)	0.72 (0.32)	0.68 (0.27)
PathGen-CLIP-L	0.59 (0.10)	0.62 (0.13)	0.63 (0.16)	0.66 (0.18)	0.69 (0.25)	0.64 (0.18)
Zero-shot H&E (own image tower, no binding): PathGen-CLIP-L 0.75 (0.46), PLIP 0.73 (0.41), ImageBind 0.65 (0.28).

Takeaways

• mIF: frozen VirTues ≈ ConvNeXt from scratch on H&E↔mIF (−0.03, within ~1 std), but its embedding transfers much better to markers through the shared space (emergent mIF↔markers 0.66 vs 0.48) — the pretrained, marker-aware mIF model is the better choice for a multi-modal space.
• markers: Geneformer / scGPT are clearly worse than an MLP on raw values for retrieval: both models only see the rank order of the 16 mapped genes in a tile (abundance is lost; ridge recovers marker props at ρ≈0.69 vs 0.998 raw). scGPT markers nevertheless give the best emergent text↔markers probe (PLIP 0.79). Single-cell RNA models are not a good fit for tile-level protein proportions; they should be tested on real expression (HEST/DLPFC).
• text: PLIP text head transfers best to ORION concepts; BiomedCLIP best on PathGen retrieval. PathGen-CLIP-L is the strongest zero-shot model, but its text features bound to UNI2 transfer worst (likely memorised PathGen captions → head overfits). Recommended: UNI2 (H&E) + VirTues (mIF) + MLP (markers) + PLIP text head.
• Data note: PD-1_prop ≈ 1 in every tile (doesn't track PD-1_count) — the column looks corrupted.

Files: results/fm_retrieval.csv, results/fm_text_probes.csv, runs/bench_fm/, runs/text_pathgen/, logs in logs/v2, logs/v3.

Text ↔ H&E image pairing on ALL image-text pairs — 19_pairsets_prep.py, 20_pairing.py

Unified pool of every text-image source in PathGen-1.6M/ (12 sources; pathor1 skipped, images absent): 6144 unique TCGA patches (4088 patients), 6480 pairs — caption 2697, qa_open 1799, qa_close 1391 (stem + correct option), summary 593. Patient 5-fold CV; one text per image at test; test gallery ≈1229 pairs ("all"), 540 caption, 330 qa_open, 278 qa_close, 119 summary. Metric: R@1 on 100-pair galleries (chance 0.01) / Hungarian pair acc on the full fold; mean of both directions.

image enc → / text enc ↓ (text_head, eval=all, R@1 g100)	BiomedCLIP	PLIP	ImageBind	PathGen-CLIP-L*
UNI2	0.231	0.200	0.199	0.248
Phikon-v2	0.234	0.204	0.197	0.254
PLIP (image)	0.170	0.147	0.139	0.180
BiomedCLIP (image)	0.163	0.138	0.136	0.168
ImageBind (image)	0.132	0.108	0.098	0.139
PathGen-CLIP-L (image)*	0.282	0.252	0.243	0.303
Zero-shot (same CLIP model): PLIP 0.090, BiomedCLIP 0.111, ImageBind 0.029, PathGen-CLIP-L* 0.188.
Mode means (excluding PathGen-CLIP-L): zero-shot 0.077 < two_heads 0.165 ≈ text_head 0.166.

By text type, UNI2 + BiomedCLIP text head (g100 / pair acc): caption 0.28 / 0.11, summary 0.29 / 0.29, qa_open 0.22 / 0.09, qa_close 0.13 / 0.06 (short multiple-choice stems carry little patch-specific information). *PathGen-CLIP-L was trained on PathGen-1.6M (most of this pool) → not held-out; shown for reference only. Takeaways: pathology SSL image encoders (UNI2, Phikon-v2) + BiomedCLIP text (256-token context) are the best clean combination; binding to a frozen image anchor (ImageBind recipe) is as good as training both projections; every trained head beats zero-shot CLIP by ~2×. Files: results/pairing_long.csv, results/pairing_summary.csv.

Four real modalities: H&E = image, text = text, mIF = protein, gene expression = RNA (HEST) — 21_–24_

The former "markers" modality was computed from the mIF image (protein), so it is retired. Gene expression is now real spatial transcriptomics: HEST-1k human Visium, 9484 spots (415 samples, 57 studies, 16 organs, 2046 bowel), 14,802 common genes, precomputed UNI2-h / GigaPath spot features. One frozen UNI2 space holds every modality: H&E–mIF (ORION), H&E–text (PathGen), H&E–RNA (HEST) are trained; protein↔RNA, text↔RNA, text↔protein are emergent (no dataset measures them together, so no 1:1 pairing accuracy is possible for them).

H&E ↔ RNA (trained), HEST study+patient 5-fold, test gallery ≈1897 spots; R@1 on 100-spot galleries (chance 0.01)

RNA encoder	UNI2 anchor	GigaPath anchor
MLP on 2000 HVGs	0.148 ± 0.026	0.113 ± 0.018
scGPT-human (frozen) + MLP	0.154 ± 0.028	0.118 ± 0.017
Geneformer-V2 (frozen) + MLP	0.159 ± 0.027	0.124 ± 0.010
On real expression, the single-cell foundation models slightly beat the MLP. On the derived markers it was the other way round.

Protein ↔ RNA (emergent, cross-dataset neighbour transfer; Spearman, shuffled-label chance ≈ 0.00)

ORION tile → k=20 nearest HEST spots. The predicted RNA of the matching gene is compared with the measured protein (marker⁺ cells / nuclei).

query → gallery	all spots	bowel spots
protein (VirTues mIF) → RNA (MLP / Geneformer)	0.45 / 0.44	0.42 / 0.42
H&E (UNI2) → RNA head	0.38	0.33
H&E → H&E (image similarity only)	0.36	0.33
RNA → protein (reverse)	0.18–0.22	0.20–0.23

Per marker (bowel, protein→RNA vs H&E→H&E):

• Pan-CK 0.81 vs 0.81; E-cad 0.81 vs 0.77
• Ki67 0.63 vs 0.49
• SMA 0.60 vs 0.49
• CD45 0.52 vs 0.35
• CD3e 0.38 vs 0.32
• CD68 0.32 vs 0.21
• CD4 0.29 vs 0.09
• CD20 0.29 vs 0.10
• PD-1 / PD-L1 < 0.2

Text ↔ RNA (emergent, PathGen text head; concept prompts vs RNA of the concept gene; AUROC top quartile)

text encoder	RNA emb (best of 3), all / bowel	H&E (UNI2) of the same spots, all / bowel
PLIP	0.64 / 0.65	0.58 / 0.54
ImageBind	0.64 / 0.63	0.60 / 0.56
BiomedCLIP	0.61 / 0.63	0.57 / 0.54
PathGen-CLIP-L	0.59 / 0.62	0.55 / 0.56
Bowel AUROC per concept (RNA/MLP): stroma 0.74, immune 0.69, tumour epithelium 0.66, proliferation 0.59, macrophage 0.52, vessels 0.51.
Files: cache/hest/, runs/gene/, results/gene_retrieval.csv, gene_emergent_protein.csv, gene_emergent_text.csv, logs in logs/gene/.

Virtual modalities predicted from H&E (to fill missing pairs; never used as test truth)

GigaTIME (H&E → 21-channel virtual mIF) — 25_gigatime_infer.py, cache/gigatime/

Model: prov-gigatime/GigaTIME (code Apache-2.0; weights research-only). Run at native tile resolution (2× upsampling was worse). Sets: ORION 3,187 tiles, TCGA text patches 6,144, gene-expression patches 1,640; 0 failures.

ORION validation against the real mIF of the same tile (Spearman of tile-mean intensity, n = 3,187):

Channel (GigaTIME ~ ORION)	Spearman
DAPI ~ Hoechst	0.91
CK ~ Pan-CK	0.74
Ki67	0.73
PD-1	0.73 (the ORION PD-1 proportion is corrupt, ≈1 everywhere)
CD3 ~ CD3e	0.65
CD4	0.58
CD68	0.47
CD8 ~ CD8a	0.37
CD20	0.36
PD-L1	0.34
Actin-D / Transgelin ~ SMA	0.18 / 0.23

Use the per-tile means only. At the pixel level, the per-tile median Pearson is ≈0.1 for every channel except DAPI (0.50), so the virtual maps are not spatially reliable.

STPath (H&E → whole-transcriptome virtual RNA) — 26_stpath_infer.py, cache/stpath/

Model: tlhuang/STPath (no licence declared — ask the authors before redistributing). Input: GigaPath tile features + coordinates + organ/tech tokens. Output: log1p counts for 14,791 genes. Sets: HEST 9,484 spots, ORION 3,187 tiles, TCGA text patches 6,144.

HEST check (top-50 HVG Pearson):

• All spots: 0.40. This number is optimistic because STPath was trained on most of HEST.
• HEST-bench samples held out from STPath: 0.21.
• Within-sample: ≈0.09.

ORION check: predicted RNA vs the real protein of the same tile (Spearman, tiles with ≥20 nuclei; pooled / within-slide):

Gene ~ protein	Pooled	Within slide
CDH1 ~ E-cad	0.77	0.77
EPCAM ~ Pan-CK	0.76	0.74
MKI67 ~ Ki67	0.68	0.64
CD3E ~ CD3e	0.45	0.42
CD8A ~ CD8a	0.44	0.40
MS4A1 ~ CD20	0.42	0.34
PTPRC ~ CD45	0.37	0.34
ACTA2 ~ SMA	0.34	0.34
CD68 ~ CD68	0.25	0.22
CD163 ~ CD163	0.07	0.15

TCGA caveat: 2,540 of 4,088 slides contribute a single patch, so most TCGA predictions have no spatial context and are less reliable than the ORION ones.

Coverage after completion: TCGA patches have H&E + real text + virtual protein + virtual RNA. ORION has H&E + real protein + virtual RNA, and synthetic text is pending (Azure). HEST has H&E features + real RNA.

H&E foundation-model ablation: UNI2 vs GigaPath vs CONCH as the frozen anchor — run_v4.sh, 28_eval_anchor.py

All other settings are identical. CONCH v1 vision tower: 448 px, unprojected 512-d (the authors' recommended tile feature). conchclip = CONCH projected into its own image-text space.

H&E ↔ mIF (ORION, trained pair; mIF = frozen VirTues + MLP head; both directions averaged)

Anchor	Patient 5-fold R@1 g100	Patient 5-fold pair acc	ORION official R@1 (109 tiles)	ORION official pair acc
UNI2	0.727 ± 0.030	0.751 ± 0.045	0.665	0.945
GigaPath	0.717 ± 0.025	0.690 ± 0.046	0.702	0.936
CONCH	0.571 ± 0.035	0.379 ± 0.037	0.529	0.651

Same with a from-scratch ConvNeXt-T mIF encoder:

Anchor	Patient 5-fold R@1 g100	Patient 5-fold pair acc	ORION official R@1	ORION official pair acc
UNI2	0.753 ± 0.019	0.744 ± 0.022	0.738	0.945
GigaPath	0.740 ± 0.022	0.691 ± 0.018	0.707	0.927
CONCH	0.639 ± 0.021	0.481 ± 0.058	0.608	0.780

The ranking UNI2 ≥ GigaPath ≫ CONCH holds for both mIF encoders.

H&E ↔ text (all 6,144 TCGA patches / 6,480 real pairs, patient 5-fold, R@1 on 100-pair galleries, "all" text types)

Image	Text	Text head only (ImageBind)	Two heads (CLIP-style)
UNI2	BiomedCLIP	0.231 ± 0.008	0.210 ± 0.012
CONCH	BiomedCLIP	0.230 ± 0.007	0.229 ± 0.007
CONCH-CLIP	BiomedCLIP	0.226 ± 0.007	0.226 ± 0.012
GigaPath	BiomedCLIP	0.218 ± 0.007	0.200 ± 0.011
CONCH	CONCH text	0.216 ± 0.009	0.221 ± 0.007
UNI2 / GigaPath	CONCH text	0.209 / 0.192	0.193 / 0.179
CONCH-CLIP	CONCH text, zero-shot	0.170 ± 0.010	—

On the PathGen-1.6M caption split (12_train_text_pathgen.py, BiomedCLIP, 255 test pairs, g100 he→text / text→he):

• UNI2: 0.29 / 0.27
• GigaPath: 0.26 / 0.22
• CONCH: 0.26 / 0.24

Emergent text ↔ mIF (PathGen-trained text head per anchor, BiomedCLIP; ORION patient 5-fold AUROC, mean of 7 concepts)

Anchor	mIF embedding	H&E embedding
UNI2	0.674	0.665
GigaPath	0.686	0.678
CONCH	0.719	0.701

CONCH gains most on tumour epithelium (0.89 vs UNI2 0.76), macrophage (0.65 vs 0.60) and vessels (0.57 vs 0.44).

Takeaways

• Trade-off between instance-level pairing and semantics.
  • CONCH has the weakest H&E↔mIF pairing: pair accuracy is half of UNI2's.
  • CONCH transfers text concepts to mIF best, emergently. Its features were trained against captions, so they encode coarse semantics more than fine tile identity.
• UNI2 is the best single anchor. It is the strongest or tied on every trained pair: mIF, text, and RNA (UNI2 > GigaPath on HEST, see above).
• The CONCH text tower is worse than BiomedCLIP on these long PathGen texts, even when paired with CONCH images. Zero-shot CONCH (0.17) is well below any trained head (≥0.18).
• HEST RNA has no CONCH anchor because only UNI2 and GigaPath features are distributed, not the images.

Virtual modality completion: train text directly on virtual pairs — 29_virtual_completion.py

Question: does training text↔RNA or text↔protein directly on virtual pairs beat the emergent route through the H&E anchor?

• RNA pairs: TCGA real text ↔ STPath virtual RNA.
• Protein pairs: TCGA real text ↔ GigaTIME virtual mIF (11 channels shared with ORION).
• Model: BiomedCLIP head + MLP head on the modality vector, patient 5-fold over TCGA.
• Features are rank-gaussianised within each dataset.
• Testing uses only real measurements, with the same prompts, labels and protocol as the emergent probes.

Text ↔	Test set	Direct on virtual pairs	Emergent via H&E anchor (best encoder)
RNA	HEST real RNA, all spots	0.595	0.59–0.61 (MLP / scGPT / Geneformer)
RNA	HEST real RNA, bowel	0.552	0.61–0.62
protein	ORION real mIF means	0.596	0.678 (VirTues mIF), 0.663 (H&E)
protein	ORION virtual GigaTIME means (same tiles)	0.686	—

(AUROC, top-quartile label, mean over concepts.)

• In-domain retrieval (held-out TCGA, text ↔ virtual modality, R@1 g100):
  • RNA: 0.196 ± 0.010
  • protein, 11 channel means: 0.048 ± 0.007
• Virtual pairs do not beat the emergent route. RNA ties on all spots and is worse on bowel. Protein is clearly worse.
• The bottleneck is the virtual→real gap. The same protein model scores 0.686 on ORION's virtual means but 0.596 on the real means of the same tiles.
• The single-channel reference (the matched real channel used as the score) gives 0.79–0.98. The probes are far from saturated, and an 11-number protein summary carries much less than the VirTues image embedding.
• Recommendation: use virtual modalities only as auxiliary or augmentation data, never as replacements for, or test truth instead of, real pairs.

Four-modality validation with virtual / synthetic data (validation only, never training) — 31_virtual_validation.py

• Training: every encoder is trained on real pairs only (H&E–mIF ORION, H&E–text PathGen, H&E–RNA HEST; frozen UNI2 anchor).
• Virtual / synthetic data: used only to build quadruplet test sets, so that modality pairs never measured together get a pairing accuracy:
  • virtual RNA: STPath
  • virtual mIF: GigaTIME
  • synthetic text: Azure gpt-5.6-sol, fixed 3-shot PathGen-style prompt, 27_azure_caption.py; 9,031 captions, 0 failures, USD 38.11
• Metrics: 100-item galleries, R@1 (both directions) / one-to-one Hungarian pairing accuracy; chance 0.01.
• TCGA evaluation: 2,868 patches of 2,158 patients that the text head never saw.

Calibration: is each virtual modality a usable stand-in for the real one?

Virtual modality	Real pair (same items)	Same pair with the virtual modality	Verdict
RNA (STPath), HEST all	H&E↔RNA 0.148 / 0.166	0.126 / 0.182	usable
RNA (STPath), HEST held out from STPath	0.086 / 0.110	0.080 / 0.136	usable
text (synthetic), 1,000 TCGA patches with real text	H&E↔text 0.184 / 0.165	0.134 / 0.140	usable, conservative (≈0.75×)
text, same 1,000 patches	text real↔RNA virtual 0.039 / 0.038	text synthetic↔RNA virtual 0.044 / 0.049	consistent
mIF (GigaTIME, 10-ch VirTues), ORION	H&E↔mIF 0.644 / 0.863	0.038 / 0.046	not usable
mIF (GigaTIME, 10 channel means), ORION	0.279 / 0.357	0.050 / 0.053	not usable

GigaTIME's tile means correlate with the real mIF at the population level (Spearman ≈0.6), but they cannot tell one tile from 99 others. Pixel maps are worse. Virtual mIF is therefore only good for coarse concept-level checks, not pairing accuracy.

Pairing accuracy for every modality pair (R@1 / pair acc on 100-galleries; ⚠ = circular)

"Circular" means both sides come from the same H&E image, so the number is an optimistic upper bound.

ORION (H&E real, mIF real (VirTues 17 ch), RNA virtual, text synthetic), patient 5-fold:

	mIF (real)	RNA (virtual)	text (synthetic)
H&E	0.726 / 0.935 (trained)	0.103 / 0.192 ⚠	0.066 / 0.091 ⚠
mIF (real)		0.108 / 0.152 (emergent)	0.068 / 0.069 (emergent)
RNA (virtual)			0.036 / 0.050 ⚠

TCGA (H&E real, text real, RNA virtual):

	text (real)	RNA (virtual)
H&E	0.178 / 0.170 (trained)	0.121 / 0.175 ⚠
text (real)		0.042 / 0.041 (emergent)

Virtual mIF on TCGA (text↔mIF 0.012–0.021) fails calibration and is not reported as a result.

Takeaways

• Protein ↔ RNA, never trained together: 0.108 R@1 / 0.152 pairing accuracy on ORION, 10–15× chance. This is real protein against virtual RNA, and virtual RNA scores close to real RNA in the HEST calibration.
• Text ↔ protein, never trained together: 0.068, about 7× chance. This uses synthetic text, which scores about 0.75× real text in the calibration, so the real-text value is likely ≈0.09.
  • ORION is a single-organ cohort (colon), so its galleries are harder than TCGA's 30-cancer galleries. H&E↔synthetic text is 0.066 on ORION vs 0.134 on TCGA.
• Text ↔ RNA, never trained together: 0.042 with real text and 0.044 with synthetic text on the same patches, about 4× chance.
• Emergent pairs are clearly above chance but an order of magnitude below the trained H&E↔mIF pair. Instance-level emergent alignment is weak; concept-level alignment (AUROC 0.6–0.7, earlier sections) is moderate.

Final protocol: real-only training, explicit train/val/test, virtual = test-time only (33_final_eval.py)

Every head is trained on REAL pairs only; val is used only for early stopping; all numbers are on the TEST split (5 folds, mean ± std; patients/groups never cross splits, asserted). Virtual modalities (STPath RNA, GigaTIME mIF, GPT captions) are only fed through the frozen real-trained heads at test time. Metrics: 100-item galleries × 50 draws, both directions; chance R@1 = pair acc = 0.01, R@10 = 0.10. Split files: splits/patient_5fold, splits/orion_official, splits/final/{hest,tcga}_fold<k>.csv; sizes results/final_splits.csv; numbers results/final_{long,summary}.csv. TCGA text heads are now trained on ALL image-text pairs per fold (runs/text_all/uni2_biomedclip_fold<k>).

dataset	unit	train	val	test (per fold)
ORION patient 5-fold	patient	2291 tiles / 29 pts	258 / 3	637 / 8
ORION official (user split)	patient	2954 / 36	124 / 2	109 / 2
HEST	study|patient group	7141 spots / 91 grp	447 / 10	1897 / 25
TCGA all image-text pairs	patient	4426 images / 2943 pts	489 / 327	1229 / 818

(1) Real pairs (test split)

pair	dataset	R@1	R@10	MRR	pair acc
H&E↔mIF	ORION 5-fold	0.726±0.031	0.989	0.828	0.935±0.030
H&E↔mIF	ORION official	0.665	1.000	0.791	0.948
H&E↔RNA	HEST	0.148±0.026	0.547	0.277	0.166±0.032
H&E↔RNA	HEST, STPath-unseen spots	0.086±0.038	0.509	0.214	0.110±0.038
H&E↔text	TCGA	0.229±0.011	0.700	0.383	0.238±0.011

(2) Virtual pairs (same test items; ⚠ = both sides derived from the same H&E → optimistic)

pair	dataset	R@1	R@10	pair acc	type
mIF↔RNA(virtual)	ORION 5-fold	0.108±0.006	0.516	0.152±0.020	emergent, real×virtual
mIF↔text(synthetic)	ORION 5-fold	0.073±0.011	0.446	0.077±0.012	emergent, real×virtual
text↔RNA(virtual)	TCGA	0.055±0.009	0.318	0.058±0.010	emergent, real×virtual
RNA(virtual)↔text(synthetic)	ORION 5-fold	0.047±0.008	0.343	0.058±0.009	⚠ virtual×virtual
H&E↔RNA(virtual)	ORION / TCGA	0.103 / 0.120	0.52 / 0.45	0.19 / 0.18	⚠
H&E↔text(synthetic)	ORION 5-fold	0.075±0.006	0.439	0.101±0.013	⚠
text↔mIF(virtual)	TCGA	0.015±0.002	0.133	0.014	GigaTIME not usable

Calibration (virtual replaces real on the same test items): HEST H&E↔RNA 0.148 → 0.126 (STPath-unseen 0.086 → 0.080); TCGA H&E↔text 0.197 → 0.166 (synthetic = 0.84× real); ORION H&E↔mIF 0.644 → 0.038 (GigaTIME fails at instance level). ORION official numbers (109 test tiles, 2 patients) match the 5-fold ones (mIF↔RNA(virtual) 0.108 / 0.152).

Why GigaTIME virtual mIF fails at pairing (34_gigatime_diag.py, results/gigatime_diag*.csv)

• Per-channel tile-mean Spearman vs real ORION mIF: GigaTIME mean 0.54 pooled / 0.50 within-slide (not a slide-level artefact), vs 0.79 / 0.75 for an in-domain reference (ridge UNI2 → real means, trained on ORION train, scored on test). Weakest: SMA 0.20 (no SMA channel; Actin-D/Transgelin proxy), PD-L1 0.34, CD20 0.36, CD8 0.37.
• Retrieval needs every channel to be right at once: real means ↔ GigaTIME means R@1 0.035 (chance 0.01) vs real ↔ UNI2-ridge 0.261 — same 10 channels, no learned head.
• Pixel maps are sigmoid probabilities, mostly < 0.05 (58–96% of pixels per channel) with ~0 confident positives except DAPI/CK; pixel Pearson ≈0.1. The real-trained VirTues head sees out-of-distribution input → 0.038.
• Conclusion: H&E does contain this information (the in-domain reference recovers it); GigaTIME, trained on another cohort/panel/scanner, does not transfer to ORION at tile resolution.

Unified mIF channels (real and virtual mIF both = the 10-channel panel shared with GigaTIME)

33_final_eval.py now uses the 10-ch head (real ORION, fold-k train/val) for every mIF row; 17-ch only as reference. [qm] = GigaTIME pixels quantile-mapped per channel onto the real fold-k TRAIN pixel distribution before the same preprocessing (30_virtues_shared.py virtual_qm). ORION patient 5-fold test, R@1 / R@10 / pair acc:

pair	raw virtual	[qm] virtual
H&E↔mIF (real, 10 ch)	0.644 / 0.979 / 0.863	—
H&E↔mIF (real, 17 ch, reference)	0.726 / 0.989 / 0.935	—
mIF(10 ch)↔RNA(virtual)	0.098 / 0.495 / 0.131	—
mIF(10 ch)↔text(synthetic)	0.074 / 0.434 / 0.071	—
H&E↔mIF(virtual)	0.038 / 0.264 / 0.046	0.042 / 0.278 / 0.047
mIF↔mIF(virtual)	0.045 / 0.248 / 0.054	0.052 / 0.301 / 0.060
mIF(virtual)↔RNA(virtual)	0.021 / 0.174 / 0.019	0.017 / 0.167 / 0.016
mIF(virtual)↔text(synthetic)	0.024 / 0.190 / 0.025	0.022 / 0.182 / 0.021
TCGA text↔mIF(virtual)	0.015 / 0.133 / 0.014	0.016 / 0.138 / 0.016

Unifying channels costs the real pairs ~0.08 R@1 (17 → 10 ch); matching the pixel distribution barely helps virtual mIF (+0.004–0.007 R@1). The bottleneck is GigaTIME's per-tile accuracy, not channel naming or intensity scale.

MIPHEI-ViT virtual mIF (replaces GigaTIME) — 35_miphei_infer.py, 37_miphei_diag.py

Model: Sanofi MIPHEI-ViT v1.0.0 (H-optimus-0 + LoRA + conv decoder; non-commercial academic licence), 16 ORION channels (all but PD-1). It was trained on this ORION tile set with the official split (37/2/2 slides = our orion_official), so only the official TEST tiles (CRC02, CRC11; 109 tiles) are clean for ORION; TCGA is fully clean. Input 0.5 µm/px tiles reflect-padded to 512, output mapped back to the 8-bit ORION scale.

Tile-mean Spearman vs the real mIF (mean over channels):

set	MIPHEI (16 ch)	MIPHEI (10 GigaTIME ch)	GigaTIME (10 ch)
clean (official test), pooled / within-slide	0.697 / 0.654	0.690 / 0.645	0.249 / 0.301
seen (official train)	0.840 / 0.791	0.840 / 0.787	0.545 / 0.505

Direct real-mean ↔ virtual-mean retrieval (no head), R@1 in 100-galleries, clean test: MIPHEI 16 ch 0.272, MIPHEI 10 ch 0.239, GigaTIME 10 ch 0.017 (seen tiles: 0.538 / 0.433 / 0.036).

MIPHEI virtual mIF in the final protocol (33_final_eval.py, 16-ch heads trained on REAL ORION)

Clean = ORION official test (109 tiles, MIPHEI never saw them) and TCGA. R@1 / R@10 / pair acc:

pair	ORION official test (clean)	ORION 5-fold (MIPHEI-seen, optimistic)	TCGA (clean)
H&E↔mIF real (16 ch)	0.591 / 0.991 / 0.921	0.689 / 0.984 / 0.901	—
H&E↔mIF(MIPHEI)	0.189 / 0.784 / 0.553	0.255 / 0.722 / 0.509	0.133 / 0.455 / 0.213
mIF real↔mIF(MIPHEI)	0.297 / 0.800 / 0.586	0.379 / 0.762 / 0.669	—
text↔mIF(MIPHEI)	—	—	0.033 / 0.231 / 0.036
RNA(virtual)↔mIF(MIPHEI)	0.041 / 0.311 / 0.078	0.057 / 0.352 / 0.093	0.035 / 0.218 / 0.040
mIF(MIPHEI)↔text(synthetic)	0.050 / 0.396 / 0.051	0.049 / 0.314 / 0.058	—
(real) mIF 16 ch↔RNA(virtual)	0.101 / 0.466 / 0.123	0.108 / 0.512 / 0.149	—

GigaTIME on the same pairs: H&E↔mIF(virtual) 0.012–0.038 R@1, text↔mIF(virtual) 0.015 → MIPHEI is 5–15× better and usable for validation (calibration: pair acc 0.55 vs 0.92 real on clean tiles ≈ 0.6×).

External test: DLPFC (36_external_prep.py, 38_external_eval.py, 39_external_check.py, results/external_*.csv)

IDC results were removed from this report on 2026-10-18 at the user's request (the prepared data stays in cache/external).

• DLPFC: spatialLIBD Visium brain, 482 spots / 12 sections / 3 donors; gene names recovered exactly from the public h5. All 12 sections are in HEST (MISC1–12) -> the whole spatialLIBD study is REMOVED from HEST training for this test.
• Heads: H&E(UNI2)↔RNA trained on real HEST only (train/val by group; 5 heads), RNA z-scored per dataset. Virtual mIF: MIPHEI of the DLPFC H&E through the 16-ch heads trained on real ORION. Synthetic text: Azure gpt-5.6-sol captions of the same patches (same 3-shot prompt) through the TCGA real-pair text heads of 33_ (clean-text heads, runs/text_clean). Nothing is trained on external or virtual data.

5 heads, R@1 ± std / R@10 / pair acc; mixed galleries G=100 (chance 0.01) and within-section galleries G=35 (chance 0.029):

pair	kind	mixed G=100	within section G=35
H&E↔RNA	real, trained pair	0.010±0.003 / 0.107 / 0.013	0.031
RNA↔mIF(MIPHEI)	emergent, real×virtual	0.008 / 0.099 / 0.009	0.027
RNA↔text(synthetic)	emergent, real×virtual	0.010±0.002 / 0.117 / 0.011	0.032
H&E↔mIF(MIPHEI)	⚠ circular	0.017 / 0.136 / 0.015	0.040
H&E↔text(synthetic)	⚠ circular	0.022±0.002 / 0.179 / 0.025	0.054
mIF(MIPHEI)↔text(synthetic)	⚠ virtual×virtual	0.014 / 0.144 / 0.017	0.038

• Every non-circular pair is at chance on DLPFC. Reference: HEST internal test restricted to brain = 0.021 (mixed organs 0.148).
• Caption audit: 59% of DLPFC captions say "lymphoid", only 6% recognise CNS/neuropil, 13% "no identifiable tissue" (blurry 112-µm crops) -> GPT cannot read these patches either.
• Checks: captions aligned row by row (482/482); shuffled nulls ≈ 1/G; rerun identical. Protocol check: our UNI2 features of 3 DLPFC spots that also exist in the HEST cache have cosine 0.52–0.83 to the cached ones (random 0.12).

Text cleaning + retrain on clean real text (v10) — 40_clean_text.py, 33_, run_v10.sh

Supersedes every TCGA-text number above (old results kept in results/before_text_clean/; old heads runs/text_all).

• Junk texts removed from train / val / test: 365 of 6,480 (353 bare answer letters "A A B B", 4 "error error", 8 answers < 5 words). 256 of 6,144 TCGA images were left without real text; they keep their split, get a few-shot GPT caption (27_ --sets tcga_notext, USD 0.81; cumulative USD 65.09 of 100) and are used for evaluation only.
• Text heads retrained on clean real pairs only (runs/text_clean/uni2_biomedclip_fold; ~4,400 train pairs / fold).

R@1 / R@10 / pair acc, test split, mean of 5 folds (before → after cleaning):

pair	before	after
TCGA H&E↔text (real)	0.229 / 0.700 / 0.238	0.237±0.012 / 0.730 / 0.248
TCGA text↔RNA(virtual)	0.055 / 0.318 / 0.058	0.056 / 0.319 / 0.057
TCGA text↔mIF(MIPHEI)	0.033 / 0.231 / 0.036	0.036 / 0.237 / 0.038
calibration set: H&E↔text real vs synthetic	0.197 vs 0.166	0.219 vs 0.162
ORION 5-fold mIF↔text(synthetic)	0.074 / 0.434 / 0.071	0.074 / 0.437 / 0.072
ORION 5-fold H&E↔text(synthetic)	0.075 / 0.439 / 0.101	0.076 / 0.441 / 0.102

Part of the gain on TCGA comes from removing junk queries from the test split (n test pairs 1229 → 1178). Images without real text (256, pooled over test folds, each image through its own test-fold head): H&E↔text(synthetic) 0.150 / 0.557 / 0.160 (⚠ circular; consistent with 0.162 on the calibration set); text(synthetic)↔RNA(virtual) 0.051; text(synthetic)↔mIF(MIPHEI) 0.036 (⚠ virtual×virtual).

Best encoder per modality — re-check under the final protocol (41_text_enc_clean.py, paired per-fold tests)

mIF (17_bench_fm / 28_, ORION patient_5fold) and RNA (23_, HEST group 5-fold) comparisons already used the final-protocol splits -> re-analysed per fold. Text was re-run on CLEAN real text with the 33_ recipe and splits (UNI2+BiomedCLIP reproduces 33_ fold by fold: 0.234 / 0.236 / 0.248 / 0.219 / 0.248). R@1 g100 mean of 5 folds; Δ = best − other, folds won, paired t-test (5 folds; 5/5 wins = sign-test p 0.06).

modality	best	runner-up (Δ, folds won, p)	verdict
text tower (H&E↔text, clean)	BiomedCLIP 0.237 (UNI2 image)	CONCH text 0.216 (+0.021, 5/5, p=0.03); PLIP 0.211 (+0.026, 5/5, p=0.007); ImageBind 0.201 (+0.036, 5/5, p=0.002)	clear winner for every image anchor
H&E image for text	Phikon-v2 0.241 ≈ CONCH 0.241 ≈ UNI2 0.237	GigaPath 0.223 (UNI2 +0.014, 4/5, p=0.05)	3-way tie, GigaPath worse
H&E image for mIF	UNI2 0.727 (VirTues) / 0.753 (ConvNeXt)	GigaPath (+0.009 / +0.013, ns); CONCH (+0.155 / +0.114, 5/5, p<0.001)	UNI2 ≈ GigaPath ≫ CONCH
H&E image for RNA	UNI2 0.148	GigaPath 0.113 (+0.035, 4/5, p=0.05)	UNI2
mIF encoder (H&E↔mIF)	ConvNeXt-T scratch 0.758	VirTues 0.728 (+0.030, 5/5, p=0.07)	ConvNeXt-T consistently ahead; VirTues kept for cross-modal transfer / channel flexibility
RNA encoder (H&E↔RNA)	Geneformer-V2 0.159	scGPT 0.154 (+0.005, 3/5, p=0.36); MLP 0.148 (+0.011, 4/5, p=0.05)	tie

UNI2 is the only H&E anchor never significantly beaten on any pair (Phikon-v2 has no mIF/RNA runs: HEST ships only UNI2/GigaPath features). Files: results/text_enc_clean_{long,summary}.csv.

Training regimes: separate vs joint (42_train_regimes.py, results/regimes_{long,summary}.csv)

Real pairs only, final-protocol splits (ORION patient 5-fold with the 16-ch VirTues panel, HEST group 5-fold, TCGA patient 5-fold on clean text). Same heads, loss and optimiser for every regime; virtual data only at test time. R@1 g100, mean of 5 folds; "rel" = mean of R@1 / S1's R@1 over the trained (3) or emergent (4) pairs.

regime	H&E↔mIF	H&E↔RNA	H&E↔text	mIF↔RNA(v)	mIF↔text(s)	text↔RNA(v)	text↔mIF(MIPHEI)	trained rel	emergent rel
S1 separate, frozen anchor (current)	0.680	0.148	0.234	0.108	0.076	0.060	0.038	1.00	1.00
S2 separate, two-tower (CLIP)	0.794	0.130	0.195	0.019	0.022	0.014	0.013	0.96	0.26
S3 separate, frozen, linear heads	0.642	0.156	0.237	0.123	0.070	0.069	0.038	1.00	1.05
S4 separate, frozen, MLP mIF + linear RNA/text	0.680	0.152	0.234	0.117	0.074	0.071	0.037	1.01	1.05
J1 joint, frozen anchor	0.706	0.148	0.237	0.094	0.065	0.056	0.031	1.02	0.87
J2 joint, shared trainable H&E head	0.761	0.147	0.204	0.061	0.060	0.044	0.031	0.99	0.72
J3 J2, size-proportional sampling	0.740	0.138	0.187	0.063	0.060	0.039	0.031	0.94	0.70
J4 J2 + anchor distillation λ=1	0.765	0.144	0.202	0.062	0.058	0.043	0.030	0.99	0.71
J4b J2 + anchor distillation λ=10	0.768	0.149	0.209	0.067	0.058	0.047	0.030	1.01	0.74
J5 joint, residual adapter on UNI2	0.762	0.142	0.201	0.058	0.059	0.044	0.029	0.98	0.71
J6 J2 + uncertainty weighting	0.765	0.136	0.195	0.057	0.055	0.041	0.027	0.96	0.66
J7 two-stage: S1 → joint residual fine-tune	0.776	0.150	0.234	0.100	0.076	0.055	0.035	1.05	0.94
J8 two-stage: S4 → joint residual fine-tune	0.762	0.164	0.232	0.112	0.073	0.068	0.035	1.07	1.01

Pair accuracy (H&E↔mIF / RNA / text): S1 0.897 / 0.172 / 0.242 → J8 0.929 / 0.198 / 0.237. J8 vs S1 (paired, 5 folds): H&E↔mIF +0.083 (5/5, p=0.01), H&E↔RNA +0.015 (5/5, p=0.03), text↔RNA(v) +0.008 (5/5, p=0.02), H&E↔text −0.002, mIF↔RNA(v) +0.004, mIF↔text(s) −0.004, text↔mIF −0.003 (all ns).

• Training the H&E side from scratch (S2, J2–J6) helps H&E↔mIF (+0.08–0.11) but costs H&E↔text (−0.03 to −0.05) and 26–74% of the emergent performance: the shared H&E head is pulled towards the easiest / strongest pair (mIF). Size-proportional sampling and uncertainty weighting make this worse; distillation to the UNI2 geometry helps a little.
• Joint training with a frozen anchor (J1) only changes the step budget: +0.026 mIF, −13% emergent.
• Linear heads for RNA / text generalise better across modalities than MLP heads (S3/S4: emergent +5%).
• Best: J8 = first train every head separately on the frozen UNI2 anchor (MLP for mIF, linear for RNA and text), then fine-tune all heads jointly with a shared zero-initialised residual adapter on UNI2 (lr 1e-4, 1000 steps, sum of the three losses, similarity distillation λ=1, joint early stopping on the geometric mean of the val R@1).

More RNA / text encoders + multi-encoder late fusion (43_*, 44_rnafm_eval.py, 45_*, 46_text_enc_eval.py, 47_final_ens.py)

All encoders frozen; one head per encoder into the frozen UNI2 space (S4 recipe of 42_: linear for RNA / text), same splits and test items as 42_. Late fusion = mean of the L2-normalised head outputs of several encoders of one modality (possible because every head lands in the same frozen space). R@1 g100, mean of 5 folds; virtual / synthetic = test only. RNA inputs: cache/rnafm/_.npy (rna_sets.py: hest real, STPath virtual = round(expm1(pred)), DLPFC real); FM inputs z-scored with HEST-train stats (DLPFC: per dataset, heads retrained without spatialLIBD).

RNA encoders (linear head):

RNA encoder	H&E↔RNA	mIF↔RNA(v)	text↔RNA(v)	H&E↔RNA(v) (HEST, calib)	DLPFC H&E↔RNA
hvg 2000 (current)	0.155	0.124	0.069	0.198	0.009
AIDO.Cell-100M (MLP head 0.171)	0.169	0.094	0.049	0.083	0.010
Tahoe-x1 70M	0.166	0.087	0.039	0.070	0.011
Geneformer-V2-104M	0.164	0.071	0.037	0.064	0.010
scGPT-human	0.155	0.087	0.050	0.096	0.013
C2S-Scale-Gemma-2-2B (mean / last token)	0.155 / 0.142	0.101 / 0.101	0.055 / 0.051	0.111 / 0.097	0.011
Nicheformer	0.152	0.104	0.056	0.097	0.006
SE-600M (STATE)	0.150	0.117	0.062	0.143	0.011
GenePT-w (bge / BiomedCLIP gene-summary embeddings)	0.141 / 0.130	0.093 / 0.075	0.056 / 0.052	0.116 / 0.086	0.009
seed control: hvg x 5 seeds, averaged	0.161	0.132	0.073	0.208	0.008
fusion hvg + Tahoe-x1 + SE-600M + AIDO.Cell	0.192	0.142	0.075	0.174	0.012
fusion all 10	0.199	0.148	0.078	0.163	0.012
early fusion (concat) hvg + Tahoe-x1	0.169	0.114	0.059	0.131	0.009

• Single FMs gain up to +0.016 on real H&E↔RNA but lose on every virtual-RNA pair; the calibration column shows why: the FMs embed STPath virtual RNA far from the real RNA of the same spot (H&E↔RNA(v) 0.06–0.14 vs 0.198), so the emergent drop reflects the virtual test data more than the encoders.
• Late fusion beats early fusion (concat) and the seed-ensemble control by a wide margin -> the encoders are complementary.

Text encoders (TCGA clean real text, linear head; 45_ / 45b_; KEEP and Quilt-1M corpora contain no PathGen / TCGA text):

text encoder	H&E↔text	mIF↔text(s)	text↔RNA(v)	text↔mIF(MIPHEI)
BiomedCLIP (current; z-scored control 0.235)	0.236	0.077	0.069	0.038
KEEP (MLP head 0.267)	0.278	0.077	0.084	0.041
Qwen3-Embedding-0.6B / 4B	0.264 / 0.258	0.062 / 0.062	0.075 / 0.067	0.038 / 0.033
PubMedBERT-emb / bge-large / BioLORD-2023 / MedCPT-Article / MedEmbed-large	0.262 / 0.259 / 0.258 / 0.257 / 0.252	0.054–0.063	0.068–0.076	0.034–0.037
QuiltNet-B-32 (QuickGELU)	0.218	0.058	0.069	0.035
seed control: KEEP x 5 seeds	0.284	0.082	0.085	0.045
fusion KEEP + BiomedCLIP + Qwen3-0.6B	0.299	0.091	0.088	0.047
fusion + PubMedBERT-emb + MedCPT	0.303	0.088	0.091	0.047

Final combination (47_final_ens.py; mIF = VirTues 16-ch MLP head in all; paired vs S4 over 5 folds):

pair	S4	AIDO.Cell + KEEP	ens-small (4 RNA + 3 text)	ens-large (10 RNA + 5 text)	ens-small vs S4
H&E↔mIF	0.677	0.677	0.677	0.677	unchanged
H&E↔RNA	0.155	0.169	0.192	0.199	+0.037 (5/5, p=0.0006)
H&E↔text	0.236	0.278	0.299	0.303	+0.062 (5/5, p<0.0001)
mIF↔RNA(v)	0.124	0.094	0.142	0.148	+0.018 (5/5, p=0.0002)
mIF↔text(s)	0.077	0.077	0.091	0.088	+0.014 (5/5, p=0.011)
text↔RNA(v)	0.069	0.052	0.095	0.098	+0.027 (5/5, p=0.0009)
text↔mIF(MIPHEI)	0.038	0.041	0.047	0.047	+0.010 (5/5, p=0.008)
DLPFC H&E↔RNA / RNA↔text(s) / RNA↔mIF(v)	0.009 / 0.012 / 0.011	0.010 / 0.010 / 0.008	0.012 / 0.015 / 0.009	0.012 / 0.011 / 0.006	still chance

Pair accuracy ens-small vs S4: H&E↔RNA 0.182→0.246, H&E↔text 0.245→0.324, mIF↔RNA(v) 0.191→0.198, text↔RNA(v) 0.077→0.100. Recommendation: ens-small (nearly all of ens-large's gain at 4 + 3 heads). Licences: AIDO.Cell (GenBio community licence, non-commercial; restricts using its embeddings to train "similar" models) and SE-600M (Arc non-commercial) are research-only; KEEP / QuiltNet / Nicheformer MIT, Tahoe-x1 / Qwen3-Embedding Apache-2.0, C2S-Scale CC-BY-4.0. Not yet tested: fusion inside the J8 two-stage regime (J8's mIF gain should stack, unverified).

Joint training of the fused model (48_joint_ens.py, results/joint_ens_{long,summary,tests}.csv)

Does the J8 two-stage gain stack with multi-encoder fusion? Same splits / test items; all configs of a fold start from the SAME stage-1 heads (separate, frozen UNI2). Stage 2 = J8 recipe (shared zero-init residual adapter on UNI2, sum of the 3 pair losses, similarity distillation λ=1, lr 1e-4, 1000 steps, joint early stopping). E = ens-small of 47_. EJ-fused: loss on the fused modality embedding; EJ-each: each encoder head keeps its own loss (fusion only at test); EJ-frz: EJ-each with RNA / text heads frozen (only adapter + mIF head + temperatures train). R@1 g100, 5 folds.

config	H&E↔mIF	H&E↔RNA	H&E↔text	mIF↔RNA(v)	mIF↔text(s)	text↔RNA(v)	text↔mIF(MIPHEI)
S4 separate, single encoders	0.675	0.155	0.236	0.122	0.076	0.069	0.038
J8 joint, single encoders	0.776	0.159	0.238	0.122	0.073	0.065	0.035
E separate, fused encoders	0.675	0.192	0.299	0.141	0.089	0.095	0.046
EJ-fused	0.792	0.178	0.292	0.118	0.080	0.060	0.035
EJ-each	0.790	0.202	0.304	0.128	0.086	0.086	0.039
EJ-frz	0.793	0.201	0.304	0.128	0.088	0.095	0.040

Pair accuracy H&E↔mIF / RNA / text: S4 0.899 / 0.182 / 0.245, E 0.899 / 0.246 / 0.324, EJ-frz 0.936 / 0.255 / 0.320.

• Joint stage-2 gains stack on the trained pairs: EJ-frz vs E H&E↔mIF +0.118 (5/5, p<0.001), H&E↔RNA +0.009 (5/5, p=0.01), H&E↔text +0.005 (ns); vs S4 all three trained pairs +0.05–0.12 (5/5).
• Price: the pairs that involve the fine-tuned mIF head lose vs E: mIF↔RNA(v) −0.013 (0/5, p=0.04), text↔mIF −0.007 (0/5, p=0.02); mIF↔text(s) −0.001 (ns). text↔RNA(v) unchanged (both heads frozen).
• Loss on the fused embedding (EJ-fused) is clearly worse than per-encoder losses (EJ-each: 5/5 better on 6 pairs): the fused loss lets single heads specialise, which breaks the cross-modal geometry (text↔RNA 0.095 → 0.060).
• Freezing RNA / text heads (EJ-frz) costs nothing on the trained pairs and keeps text↔RNA at 0.095 (vs EJ-each 0.086, 5/5, p=0.03): the adapter alone does the H&E-side work.
• Recommendation: EJ-frz if H&E↔mIF matters (+0.12) and a −0.01 on the virtual mIF emergent pairs is acceptable; E (pure separate) if the emergent mIF pairs are the priority.

MedVTok as H&E image information + adapter-only joint training (49_–51_, results/medvtok_{long,summary,tests}.csv)

MedVTok (arXiv 2610.35842; github Masaaki-75/medvtok @a558c24, HF massaki75/medvtok, MIT): general medical visual tokenizer, SigLIP2-So400m-initialised encoder (1152-d), aligned (MERA) to frozen expert encoders, CONCH v1.5 for histopathology; no image-text pairs. Pathology training images: HISTAI 4.0M, PathGen 1.61M, bone-marrow cytomorphology, LYSTO. => it has seen the TCGA/PathGen test IMAGES (not their captions); ORION / HEST / DLPFC not listed. Loaded with weights_only=True + strict=True; reconstruction PSNR on ORION H&E 32–42 dB; feature = mean-pooled tokens -> LN -> Linear ("feats", 1152-d), 256 px, [-1, 1]. HEST images: 224-px level-0 crops from the HEST WSIs (49_, HTTP range reads of only the needed TIFF tiles); re-encoding them with UNI2-h reproduces cache/hest/he_uni2.npy (cos ≥ 0.9994 for all 9,484 spots), i.e. MedVTok sees exactly the anchor's images. DLPFC: the 0.5-µm crops behind dlpfc_he_uni2 (cos 1.000). Configs (RNA / text ens-small, mIF VirTues MLP; same splits / test items; R@1 g100, 5 folds): E separate UNI2 anchor; EJ-frz (= 48_); EJ-ad = EJ-frz with the mIF head frozen too (only the residual H&E adapter and the temperatures train); MV-E = MedVTok instead of UNI2 as the frozen anchor; +MV = adapter input [UNI2, c·MedVTok]; +RP = control, adapter input [UNI2, random projection of UNI2 to 1152-d at the same norm] (wider adapter, no new info).

config	H&E↔mIF	H&E↔RNA	H&E↔text	mIF↔RNA(v)	mIF↔text(s)	text↔RNA(v)	text↔mIF	DLPFC H&E↔RNA
E	0.675	0.192	0.299	0.141	0.089	0.095	0.046	0.012
EJ-frz	0.786	0.202	0.305	0.134	0.087	0.095	0.042
EJ-ad	0.776	0.201	0.299	0.141	0.089	0.095	0.046
MV-E (MedVTok anchor)	0.658	0.131	0.293	0.081	0.113	0.056	0.052	0.012
EJ-frz+MV	0.811	0.200	0.307	0.125	0.084	0.095	0.041
EJ-ad+MV	0.792	0.200	0.315	0.141	0.089	0.095	0.046
EJ-ad+RP (control)	0.778	0.202	0.298	0.141	0.089	0.095	0.046

Pair accuracy H&E↔mIF / RNA / text: E 0.899 / 0.246 / 0.324, EJ-ad 0.929 / 0.256 / 0.313, EJ-ad+MV 0.937 / 0.252 / 0.338.

• EJ-ad (only the H&E adapter moves): every emergent pair is identical to E by construction (all modality heads frozen) while H&E↔mIF +0.101 (5/5, p<0.001) and H&E↔RNA +0.009 (5/5, p=0.01). vs EJ-frz: H&E↔mIF −0.010 (ns), H&E↔text −0.006 (0/5, p=0.04), but mIF↔RNA(v) +0.007 (5/5, p=0.045) and text↔mIF +0.005 (ns) -> the emergent losses of EJ-frz came entirely from moving the mIF head.
• MedVTok as the anchor (MV-E) is worse: H&E↔RNA −0.061 (0/5), RNA emergent pairs −0.04/−0.06 (0/5); only the pairs with synthetic GPT text / virtual mIF gain (mIF↔text(s) +0.024, text↔mIF +0.006, 5/5) - CONCH-v1.5 semantics. DLPFC stays at chance (0.012) with either anchor.
• MedVTok as extra adapter input: EJ-ad+MV vs EJ-ad H&E↔mIF +0.015 (4/5, p=0.04), H&E↔text +0.016 (5/5, p=0.02), H&E↔RNA −0.002 (ns), emergent unchanged. The control shows it is MedVTok's information, not adapter width: EJ-ad+RP = EJ-ad (all |Δ| ≤ 0.002, ns); EJ-ad+MV vs EJ-ad+RP: H&E↔mIF +0.014 (5/5, p=0.03), H&E↔text +0.017 (5/5, p=0.01). Caveat: the H&E↔text test images are PathGen images MedVTok was pretrained on (image-only, aligned to CONCH v1.5); the ORION H&E↔mIF gain is free of that.
• EJ-frz+MV has the best H&E↔mIF (0.811) but again loses the mIF emergent pairs (mIF↔RNA(v) −0.017 vs E, 0/5).
• Recommendation: EJ-ad+MV (stage 1 separate heads on frozen UNI2; stage 2 trains only a residual H&E adapter that reads UNI2 + MedVTok). Every non-H&E embedding is unchanged; H&E↔mIF / text improve. Cost: a 2nd image encoder (MedVTok encoder 417M params) at H&E inference. EJ-ad (no MedVTok) if a single image encoder is required.

Final model: training, stage-2 selection rule, inference API (52_final_model.py, 53_embed_check.py, embed.py)

Configuration EJ-ad+MV (previous section). Stage 1 is deterministic (52_ reproduces 51_'s stage-1 heads exactly: the four emergent pairs of fold 0 are identical to 51_). Stage 2 is not: two identical fold-0 runs gave H&E↔mIF 0.791 and 0.741, because the val geo-mean is flat after ~100 steps and the argmax picks very different steps (100–700). 52_ cv (5 folds × 3 stage-2 seeds, same stage-1 heads, same test items; results/final_cv_{long,summary,tests,curves}.csv):

stage-2 rule	H&E↔mIF	H&E↔RNA	H&E↔text	geo-mean	pair acc mIF / RNA / text	seed std mIF / text
stage 1 only (= E)	0.675	0.192	0.299	0.337	0.899 / 0.246 / 0.324	–
es (51_ rule, 1 seed)	0.794	0.201	0.309	0.366	0.937 / 0.253 / 0.329	0.009 / 0.011
last (1000 steps, 1 seed)	0.802	0.202	0.301	0.364	0.939 / 0.254 / 0.314	0.002 / 0.003
ens-es (mean of 3 early-stopped adapters)	0.801	0.203	0.323	0.373	0.942 / 0.258 / 0.347	ensemble
ens-last	0.807	0.204	0.309	0.370	0.941 / 0.255 / 0.322	ensemble

ens-es vs es: H&E↔text +0.015 (5/5, p=0.02), mIF +0.007, RNA +0.002 (ns); ens-es vs ens-last: text +0.014 (5/5, p=0.004), mIF −0.006 (0/5, p=0.03). Emergent pairs are identical for every rule (heads frozen): mIF↔RNA(v) 0.141, mIF↔text(s) 0.089, text↔RNA(v) 0.095, text↔mIF 0.046. -> ens-es is the final rule (3 adapters, H&E = UNI2 + mean_s A_s([UNI2, c·MedVTok]); each adapter 2.4M parameters). Final 5-fold test performance (R@1 / R@10 / pair acc): H&E↔mIF 0.801±0.043 / 0.991 / 0.942, H&E↔RNA 0.203±0.033 / 0.625 / 0.258, H&E↔text 0.323±0.007 / 0.813 / 0.347; emergent as above (chance 0.01 / 0.10 / 0.01).

Saved models (52_ fold0 / 52_ release, rule ens-es):

• models/ej_ad_mv_fold0/ fold-0 train / val, test held out: test R@1 mIF 0.783, RNA 0.192, text 0.320 (51_ fold 0: 0.768 / 0.187 / 0.316; emergent identical).
• models/ej_ad_mv/ release: train = fold-0 train + test (ORION 3000 tiles, HEST 8938 spots, TCGA 5615 texts), val = fold-0 val (187 / 546 / 500) for early stopping. No own test split: expected performance = the 5-fold CV above. model.pt (heads, 3 adapters, feature-normalisation statistics) + meta.json (dims, HVG list, split sizes, val scores).

embed.py (Embedder): raw-input embedding of all four modalities (H&E images → UNI2-h + MedVTok; text → KEEP + BiomedCLIP + Qwen3-Emb-0.6B; RNA counts + gene symbols → hvg + Tahoe-x1 + SE-600M + AIDO.Cell; mIF tiles (16 ORION markers) + slide ids → VirTues), or from precomputed features; output L2-normalised 1536-d vectors. The raw-input pre-processing reuses the extraction code of 04b_/14_/43_/45_/45b_/50_. 53_embed_check.py (results/embed_check.csv): raw-input features vs cached training features, per-item cosine median (min): UNI2 1.000 (1.000) on ORION / PathGen / HEST; MedVTok 1.000 (≥0.9997); KEEP 1.000, BiomedCLIP 1.000 (0.929, bf16 batch effects), Qwen3 0.9998; hvg 1.000, AIDO.Cell 1.000, Tahoe-x1 0.991 (0.929) and SE-600M 0.994 (0.939) (both sample / tie-break genes at random, as in their official collators); VirTues 1.000 (all 3187 ORION tiles). End-to-end with models/ej_ad_mv_fold0 on held-out fold-0 test items (300 each), g100 R@1 raw-input vs cached features: H&E↔mIF 0.792 / 0.792, H&E↔RNA 0.194 / 0.194, H&E↔text 0.335 / 0.336.

Release on the Hugging Face Hub and reproduction check (2026-10-28)

All three repos are private:

• Model, code and results: Pathstudy/fourmodal-pathology-bind. The README is the model card in Chinese: data, architecture, results and usage.
• Features and splits (966 MB): datasets/Pathstudy/fourmodal-pathology-features.
• Report: the Space Pathstudy/imagebind-pathology-report. Its index.html is the final report; the earlier report is kept as previous_report.html.

New scripts:

• evaluate_saved.py: feature-level re-evaluation of a saved checkpoint.
• example_check.py + examples/: a raw-input check on 64 fold-0 test items per dataset, built by 54_make_examples.py.
• reproduce.sh and compare_reproduction.py: retrain from the features and compare with the shipped numbers.
• setup_encoders.sh: clones VirTues, ModelGenerator and MedVTok at the pinned commits and fetches the MedVTok weights.
• requirements.txt.

Code changes:

• 52_final_model.py --out NAME, so a retraining does not overwrite the shipped checkpoints or results.
• embed.py: $UNI2_PATH, falling back to the gated HF repo.
• 43_rnafm_tahoex1.py: the Geneformer name→ID dictionary is now loaded with hf_hub_download at a pinned revision. It used to be globbed from the local HF cache.

Check run in a fresh directory, downloaded from the Hub, with a new uv virtualenv (Python 3.11, torch 2.7.1+cu126) on an RTX A5000:

• evaluate_saved.py: equal to test.csv (max |diff| 8e-17).

• Fold-0 and 5-fold retraining: every pair is identical to the shipped numbers, stage 1 and ens-es (|diff| 0).

• Release retraining: the same early-stopping steps, [200, 200, 900].

• example_check.py: all encoders reproduce the cached features. Cosine is 1.000 except Tahoe-x1 .991, SE-600M .995 and VirTues .999 (64-tile standardisation).

• R@1 within 64 items, raw inputs vs cached features:

  Pair	Raw inputs	Cached features
  H&E↔RNA	.227	.227
  H&E↔text	.398	.398
  H&E↔mIF	.828	.875

• Bug found by the fresh run and fixed: Tahoe-x1 raw input needed cache/external/dlpfc_genes_full.csv. That file now ships with the model repo.

• Not covered: the frozen-encoder weights came from the existing HF cache on this machine, and UNI2-h from its local copy. A new machine downloads them from the Hub at the pinned revisions, and UNI2-h needs gated access.