Migrate release files 1-30

14_extract_if_fm.py

@@ -0,0 +1,378 @@
+"""Frozen spatial-proteomics foundation-model embeddings of every ORION multiplex-IF tile under the 8 dihedral transforms.
+Usage: python 14_extract_if_fm.py virtues [--no_hoechst] [--qc_drop] [--bs 4] [--no_probe]
+Output: cache/if_virtues.npy (N=3187, 8, 512) float16 (raw features; row i = tile_id i, axis 1 = common.dihedral d).
+        (--no_hoechst / --qc_drop write cache/if_virtues_nohoechst.npy / if_virtues_qcdrop.npy instead.)
+
+VirTues (Wenckstern et al., Nature 2026; github.com/bunnelab/virtues, HF bunnelab/virtues, weights virtues-sp32)
+---------------------------------------------------------------------------------------------------------------
+Model: MultiplexVirtues with the released base_config (patch 8, dim 512, encoder 'hvhvhvhvhvhvhvhv', RoPE), ESM-2
+marker priors. Only the encoder is used (no masking). Input = list of (C,H,W) standardised crops + marker indices.
+
+Tile embedding (pooling): mean over the spatial grid of VirTues *patch summary tokens* (the per-8x8-patch summary
+token that attends across all markers; `encoder.forward_list(...).patch_summary_tokens`, shape (H/8, W/8, 512)).
+VirTues has no global/CLS token; its own tile/tissue-level tooling (spora-bench SporaVirTuesWrapper.embed_tile /
+embed_tissue) returns exactly these patch summary tokens as a bag (then ABMIL), and cell/niche tokens are
+area-weighted averages of them (virtues.utils.cell_tokens). A uniform mean over the whole tile = the niche token of
+the full 166 um tile. Encoding runs in fp16 autocast, like the spora-bench wrapper.
+
+Marker -> UniProt (ESM-2 esm2_t30_150M_UR50D, mean over residues of layer 30, computed with fair-esm exactly as
+virtues/utils/compute_esm_embeddings.py; FASTA from rest.uniprot.org; saved as ext/marker_embeddings/esm2_t30_150M_UR50D/[ID].pt):
+  Hoechst P68431 (Histone H3.1)  -- spora flags DNA stains `is_nuclear_marker` with no UniProt ID, so VirTues'
+                                    `uniprot_filtered` loading DROPS them (spora-bench VirTues config for lin2023highsubset
+                                    = the ORION cohort). spora-io's `replace_nuclear_uniprot_ids=True` instead maps them to
+                                    Histone H3 P68431 (used for lin2023high in spora-bench). Default here: keep Hoechst as
+                                    P68431 (H3 IMC channels are in VirTues' pretraining vocabulary); --no_hoechst drops it.
+  CD31 P16284 (PECAM1) | CD45 P08575 (PTPRC canonical, as in spora examples) | CD68 P34810 | CD4 P01730 | FOXP3 Q9BZS1
+  CD8a P01732 | CD45RO P08575-4 (UniProt PTPRC isoform 2 = "CD45R0/CD45RO"; spora uses isoform IDs, e.g. CD45RA P08575-8)
+  CD20 P11836 (MS4A1) | PD-L1 Q9NZQ7 (CD274) | CD3e P07766 | CD163 Q86VB7 | E-cadherin P12830 (CDH1) | PD-1 Q15116
+  Ki67 P46013 | Pan-CK P02533 (KRT14; the keratin spora/VirTues assign to pan-cytokeratin channels, e.g. cords2024cancer
+  "Pan Cytokeratin + Keratin Epithelial") | SMA P62736 (ACTA2)
+
+Preprocessing (mirrors spora-bench configs/datasets/virtues/lin2023highsubset.yaml:
+  standardization quantile_clipping_log1p/uq_0.99_image, resolution 1.0 um/px, filter gaussian_blur, use_mean_std,
+  disable_quantile_mask=False):
+  1. resolution: ORION tiles are 512 px crops of the 0.325 um/px Orion scan resized to 333 px (0.4997 um/px, 166.4 um).
+     Resampled to 168x168 px (0.991 um/px; 168 = 21 patches of 8) with antialiased bilinear interpolation.
+  2. per-"image" upper quantile: for every slide (spora tissue = one ORION slide) and channel, the 0.99 quantile of all
+     resampled pixels of that slide's tiles; x <- log1p(clip(x, 0, uq)).
+  3. dataset-level per-channel mean/std of the clipped log1p values over all ORION tiles; x <- (x - mean) / (std + 1e-8).
+  4. 3x3 Gaussian blur (sigma 1, reflect padding; torchvision v2.GaussianBlur as in spora-io FilterFactory).
+  5. quantile mask: spora drops a (slide, channel) whose upper quantile is 0. On these 8-bit tiles that happens for
+     52/697 (slide, channel) pairs (FOXP3 in 25/41 slides, Ki67 6, CD20/CD8a/PD-L1 5, CD31/CD45RO 3) purely because of
+     8-bit quantisation of sparse markers, and would give slide-dependent channel sets (a slide/batch confound). Default
+     here: uq is floored at 1 grey level and all channels are kept for every tile; --qc_drop reproduces spora's drop.
+  6. dihedral d (common.dihedral on the (C,H,W) tensor) applied to the standardised 168 px tile; all 8 encoded.
+  The whole 168 px tile is encoded in one pass (21x21 patch grid; RoPE positions so non-128 sizes are supported).
+
+Deviations from the VirTues/spora preprocessing (documented):
+  * Input intensities are the 8-bit (0-255) ORION tile exports, not the raw 16-bit Orion counts spora stores; because
+    log1p is scale-dependent the clipped-log values are compressed differently (z-scoring per channel partly absorbs it).
+  * Quantiles / means / stds come from the 3187 ORION tiles (all pixels, no tissue mask), not whole-slide tissue pixels.
+  * VirTues was pretrained on 128x128 crops; tiles here are 168x168 (whole tile, no sliding window).
+  * uq floored at 1 grey level instead of spora's per-slide channel drop (step 5).
+  * PD-1 in these exports is a near-saturated, almost constant channel (log1p mean 4.65, std 0.16; PD-1_prop ~ 1 for
+    most tiles); the per-channel z-scoring therefore amplifies its background texture. Kept as-is (faithful pipeline).
+  * Hoechst is kept as Histone H3 (P68431) by default; VirTues' own ORION config drops DNA stains (see --no_hoechst).
+  * flash_attn is not installed for this torch build: flash_attn_varlen_qkvpacked_func is replaced by an exact
+    PyTorch SDPA implementation over the same variable-length sequences (same math, softmax scale 1/sqrt(d)).
+"""
+import os
+import sys
+import time
+import types
+
+os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")  # less fragmentation on the shared GPU
+import numpy as np
+import tifffile
+import torch
+import torch.nn.functional as F
+
+from common import CACHE, ORION, ROOT, dihedral, load_all_tiles
+
+assert sys.argv[1] == "virtues", "usage: python 14_extract_if_fm.py virtues [--no_hoechst] [--qc_drop] [--bs N] [--no_probe]"
+NO_HOECHST = "--no_hoechst" in sys.argv
+QC_DROP = "--qc_drop" in sys.argv  # spora behaviour: drop (slide, channel) with 0.99-quantile == 0 (default: floor at 1)
+BS = int(sys.argv[sys.argv.index("--bs") + 1]) if "--bs" in sys.argv else 4  # tiles per forward (x8 dihedrals = 32 crops, ~4.7 GiB peak)
+dev = "cuda"
+VIRTUES_REPO = os.path.join(ROOT, "ext", "virtues")
+EMB_ROOT = os.path.join(ROOT, "ext", "marker_embeddings")
+ESM_NAME = "esm2_t30_150M_UR50D"
+EMB_DIR = os.path.join(EMB_ROOT, ESM_NAME)
+WEIGHTS = ("virtues-sp32/model.safetensors", "virtues-sp31/model.safetensors")
+SIZE, MPP_IN = 168, 0.325 * 512 / 333  # 333 px tile = 512 px @ 0.325 um/px -> 0.4997 um/px; 168 px ~ 0.991 um/px
+
+# channel order of the ORION IF tiles (= the *_prop columns of the dataframes)
+MARKERS = [("Hoechst", "P68431"), ("CD31", "P16284"), ("CD45", "P08575"), ("CD68", "P34810"), ("CD4", "P01730"),
+           ("FOXP3", "Q9BZS1"), ("CD8a", "P01732"), ("CD45RO", "P08575-4"), ("CD20", "P11836"), ("PD-L1", "Q9NZQ7"),
+           ("CD3e", "P07766"), ("CD163", "Q86VB7"), ("E-cadherin", "P12830"), ("PD-1", "Q15116"), ("Ki67", "P46013"),
+           ("Pan-CK", "P02533"), ("SMA", "P62736")]
+
+
+# ----------------------------------------------------------------------------------------------- marker embeddings
+def ensure_marker_embeddings(ids):
+    """ESM-2 mean-pooled embeddings, identical to virtues/utils/compute_esm_embeddings.py (fair-esm, layer 30)."""
+    import requests
+    fasta_dir = os.path.join(EMB_ROOT, "fasta")
+    os.makedirs(fasta_dir, exist_ok=True)
+    os.makedirs(EMB_DIR, exist_ok=True)
+    todo = [u for u in ids if not os.path.exists(os.path.join(EMB_DIR, f"{u}.pt"))]
+    if not todo:
+        return
+    import esm
+    _load = torch.load  # official fair-esm checkpoints pickle argparse.Namespace -> need weights_only=False
+    torch.load = lambda *a, **k: _load(*a, **{**k, "weights_only": False})
+    try:
+        model, alphabet = esm.pretrained.load_model_and_alphabet(ESM_NAME)
+    finally:
+        torch.load = _load
+    # small one-off job: GPU if the shared card has room, else CPU (the repo recommends CPU for long sequences)
+    edev = "cuda" if torch.cuda.is_available() and torch.cuda.mem_get_info()[0] > 4 * 2**30 else "cpu"
+    model.eval().to(edev)
+    conv = alphabet.get_batch_converter()
+    for u in todo:
+        fp = os.path.join(fasta_dir, f"{u}.fasta")
+        if not os.path.exists(fp):
+            r = requests.get(f"https://rest.uniprot.org/uniprotkb/{u}.fasta", timeout=60)
+            r.raise_for_status()
+            open(fp, "w").write(r.text)
+        seq = "".join(l.strip() for l in open(fp) if not l.startswith(">"))
+        _, _, tok = conv([(u, seq)])
+        n = int((tok != alphabet.padding_idx).sum())
+        with torch.no_grad():
+            # return_contacts only adds the contact head (repo passes True); representations are identical, and
+            # stacking all attention maps for Ki67 (3256 aa) would need 25 GB
+            rep = model(tok.to(edev), repr_layers=[30], return_contacts=False)["representations"][30]
+        emb = rep[0, 1:n - 1].mean(0).cpu()
+        torch.save(emb, os.path.join(EMB_DIR, f"{u}.pt"))
+        print(f"  ESM-2 {u}: len {len(seq)} -> {tuple(emb.shape)}", flush=True)
+    del model
+    torch.cuda.empty_cache()
+
+
+def check_against_repo_embeddings(ids):
+    """The repo ships ESM-2 embeddings for its example dataset; compare on shared UniProt IDs."""
+    ref = os.path.join(VIRTUES_REPO, "assets", "example_dataset", "marker_embeddings")
+    shared = [u for u in ids if os.path.exists(os.path.join(ref, f"{u}.pt"))]
+    if shared:
+        cs = [F.cosine_similarity(torch.load(os.path.join(EMB_DIR, f"{u}.pt")).float(),
+                                  torch.load(os.path.join(ref, f"{u}.pt")).float(), 0).item() for u in shared]
+        mx = max((torch.load(os.path.join(EMB_DIR, f"{u}.pt")) - torch.load(os.path.join(ref, f"{u}.pt"))).abs().max().item()
+                 for u in shared)
+        print(f"marker embeddings vs repo example set ({len(shared)} shared IDs): min cos {min(cs):.6f}, max|diff| {mx:.2e}")
+
+
+# ----------------------------------------------------------------------------------------------- flash-attn shim
+def install_flash_attn_shim():
+    """Exact SDPA replacement for flash_attn_varlen_qkvpacked_func (qkv: (total, 3, H, D), cu_seqlens: (n+1,))."""
+    def varlen_qkvpacked(qkv, cu_seqlens, max_seqlen, dropout_p=0.0, softmax_scale=None, causal=False, **_):
+        total, _, H, D = qkv.shape
+        cu = cu_seqlens.tolist()
+        lens = np.diff(cu)
+        out = torch.empty(total, H, D, dtype=qkv.dtype, device=qkv.device)
+        for L in np.unique(lens):
+            if L == 0:
+                continue
+            starts = np.asarray(cu[:-1])[lens == L]
+            if len(starts) == len(lens):  # all sequences same length -> contiguous reshape
+                q = qkv.view(len(lens), L, 3, H, D)
+                idx = None
+            else:
+                idx = (torch.as_tensor(starts, device=qkv.device)[:, None] + torch.arange(L, device=qkv.device)).reshape(-1)
+                q = qkv[idx].view(len(starts), L, 3, H, D)
+            q = q.permute(2, 0, 3, 1, 4)  # 3, n, H, L, D
+            o = F.scaled_dot_product_attention(q[0], q[1], q[2], dropout_p=dropout_p, is_causal=causal,
+                                               scale=softmax_scale)  # n, H, L, D
+            o = o.transpose(1, 2).reshape(-1, H, D)
+            if idx is None:
+                out = o.contiguous()
+            else:
+                out[idx] = o
+        return out
+
+    pkg = types.ModuleType("flash_attn")
+    pkg.__path__ = []
+    sub = types.ModuleType("flash_attn.flash_attn_interface")
+    sub.flash_attn_varlen_qkvpacked_func = varlen_qkvpacked
+    pkg.flash_attn_interface = sub
+    sys.modules.setdefault("flash_attn", pkg)
+    sys.modules.setdefault("flash_attn.flash_attn_interface", sub)
+
+
+def load_virtues(marker_ids):
+    install_flash_attn_shim()
+    if VIRTUES_REPO not in sys.path:
+        sys.path.insert(0, VIRTUES_REPO)
+    from huggingface_hub import hf_hub_download
+    from safetensors.torch import load_file
+    from virtues.modules.multiplex_virtues import MultiplexVirtues
+    from virtues.utils.utils import load_marker_embedding_dict, load_marker_embeddings
+
+    emb = load_marker_embeddings(EMB_DIR)
+    emb_dict = load_marker_embedding_dict(EMB_DIR)
+    model = MultiplexVirtues(prior_bias_embeddings=emb)  # defaults == configs/base_config.yaml (released model)
+    for w in WEIGHTS:
+        try:
+            sd = load_file(hf_hub_download("bunnelab/virtues", w))
+            model.load_state_dict(sd, strict=True)
+            print("loaded", w)
+            break
+        except Exception as e:  # fall back to sp31
+            print("could not load", w, repr(e)[:200])
+    else:
+        raise RuntimeError("no VirTues weights could be loaded")
+    model.eval().to(dev)
+    midx = torch.tensor([emb_dict[u] for u in marker_ids], dtype=torch.long)
+    return model, midx
+
+
+# ----------------------------------------------------------------------------------------------- data
+def load_resampled(df):
+    """Raw uint8 tiles -> (N,17,SIZE,SIZE) float16 at ~1 um/px, plus raw per-channel mean intensity (baseline)."""
+    N = len(df)
+    X = np.empty((N, 17, SIZE, SIZE), np.float16)
+    raw_mean = np.empty((N, 17), np.float32)
+    paths = [os.path.join(ORION, p) for p in df.target_path]
+    chunk = 64
+    for s in range(0, N, chunk):
+        t = np.stack([tifffile.imread(p) for p in paths[s:s + chunk]])  # n,333,333,17 uint8
+        x = torch.from_numpy(t).to(dev).permute(0, 3, 1, 2).float()
+        raw_mean[s:s + chunk] = x.mean((2, 3)).cpu().numpy()
+        X[s:s + chunk] = F.interpolate(x, size=(SIZE, SIZE), mode="bilinear", antialias=True,
+                                       align_corners=False).clamp(min=0).half().cpu().numpy()
+        if s % 640 == 0:
+            print(f"  read {s}/{N}", flush=True)
+    return X, raw_mean
+
+
+def standardize(X, slide):
+    """quantile_clipping_log1p / uq_0.99_image + global mean/std + gaussian blur (in place, float16)."""
+    from torchvision.transforms.v2.functional import gaussian_blur
+    N, C = X.shape[:2]
+    slides = np.unique(slide)
+    uq = np.zeros((len(slides), C), np.float32)
+    n_zero = np.zeros((len(slides), C), bool)
+    s1, s2, cnt = np.zeros(C), np.zeros(C), np.zeros(C)
+    for k, sl in enumerate(slides):
+        idx = np.where(slide == sl)[0]
+        x = torch.from_numpy(X[idx]).to(dev).float().transpose(0, 1).reshape(C, -1)  # C, n*H*W
+        # torch.quantile is limited to 2**24 elements: very large slides use a fixed random pixel subsample
+        qs = x if x.shape[1] <= 2 ** 24 else x[:, torch.randperm(x.shape[1], generator=torch.Generator().manual_seed(0))[: 2 ** 24].to(x.device)]
+        uq[k] = np.array([torch.quantile(qs[c], 0.99).item() for c in range(C)])
+        n_zero[k] = uq[k] <= 0
+        if not QC_DROP:  # 8-bit tiles: sparse markers (FOXP3, ...) often have uq == 0 -> floor at one grey level
+            uq[k] = np.maximum(uq[k], 1.0)
+        u = torch.from_numpy(uq[k]).to(dev)[:, None]
+        v = torch.log1p(torch.minimum(x, u))
+        ok = (uq[k] > 0)
+        s1[ok] += v.sum(1).double().cpu().numpy()[ok]
+        s2[ok] += (v.double() ** 2).sum(1).cpu().numpy()[ok]
+        cnt[ok] += v.shape[1]
+    mean = s1 / cnt
+    std = np.sqrt(s2 / cnt - mean ** 2)
+    valid = uq > 0  # spora quantile mask: drop (slide, channel) with zero upper quantile
+    slide_k = {sl: k for k, sl in enumerate(slides)}
+    m_t = torch.tensor(mean, dtype=torch.float32, device=dev).view(1, C, 1, 1)
+    s_t = torch.tensor(std, dtype=torch.float32, device=dev).view(1, C, 1, 1)
+    for s in range(0, N, 128):
+        ks = [slide_k[sl] for sl in slide[s:s + 128]]
+        u = torch.from_numpy(uq[ks]).to(dev)[:, :, None, None]
+        x = torch.from_numpy(X[s:s + 128]).to(dev).float()
+        x = (torch.log1p(torch.minimum(x, u)) - m_t) / (s_t + 1e-8)
+        x = gaussian_blur(x, kernel_size=[3, 3], sigma=[1.0, 1.0])
+        X[s:s + 128] = x.half().cpu().numpy()
+    return X, uq, mean, std, valid, slide_k, n_zero
+
+
+# ----------------------------------------------------------------------------------------------- sanity checks
+def sanity(out, raw_mean, markers, groups):
+    from scipy.stats import spearmanr
+    from sklearn.linear_model import RidgeCV
+    from sklearn.model_selection import GroupKFold, KFold, cross_val_predict
+    from sklearn.pipeline import make_pipeline
+    from sklearn.preprocessing import StandardScaler
+    o = out.astype(np.float32)
+    print("\n=== sanity checks ===")
+    print("finite:", bool(np.isfinite(o).all()), "| shape", out.shape, out.dtype)
+    z = o[:, 0]
+    zn = z / np.linalg.norm(z, axis=1, keepdims=True)
+    rng = np.random.default_rng(0)
+    s = rng.choice(len(z), min(1000, len(z)), replace=False)
+    cos = zn[s] @ zn[s].T
+    off = cos[~np.eye(len(s), dtype=bool)]
+    zc = z - z.mean(0)
+    zcn = zc / np.linalg.norm(zc, axis=1, keepdims=True)
+    offc = (zcn[s] @ zcn[s].T)[~np.eye(len(s), dtype=bool)]
+    sd = z.std(0)
+    ev = np.linalg.eigvalsh(np.cov(z.T))[::-1]
+    print(f"mean pairwise cosine (raw) {off.mean():.4f} (min {off.min():.4f}) | after centering {offc.mean():.4f}")
+    print(f"per-dim std: min {sd.min():.4f} median {np.median(sd):.4f} max {sd.max():.4f}; "
+          f"dims with std<1e-3: {(sd < 1e-3).sum()}; PCA participation ratio {ev.sum() ** 2 / (ev ** 2).sum():.1f}; "
+          f"top-10 PCs explain {ev[:10].sum() / ev.sum():.3f}")
+    on = o / np.linalg.norm(o, axis=2, keepdims=True)
+    same = np.mean([(on[:, 0] * on[:, d]).sum(1).mean() for d in range(1, 8)])
+    samec = np.mean([(zcn * ((o[:, d] - z.mean(0)) / np.linalg.norm(o[:, d] - z.mean(0), axis=1, keepdims=True))).sum(1).mean()
+                     for d in range(1, 8)])
+    print(f"dihedral consistency: mean cos(d0, d1..7) same tile {same:.4f} (centered {samec:.4f}) "
+          f"vs random tile pairs {off.mean():.4f} (centered {offc.mean():.4f})")
+
+    Y = markers[:, :17]
+    names = [m for m, _ in MARKERS]
+
+    def probe(Xf, cv, grp=None):
+        pipe = make_pipeline(StandardScaler(), RidgeCV(alphas=np.logspace(-2, 5, 15)))
+        P = cross_val_predict(pipe, Xf, Y, cv=cv, groups=grp)
+        r = np.array([spearmanr(P[:, j], Y[:, j])[0] for j in range(Y.shape[1])])
+        return np.nan_to_num(r)
+
+    feats = {"VirTues (d0)": z, "VirTues (mean 8 dihedral)": o.mean(1), "baseline: IF mean intensity": raw_mean,
+             "baseline: log1p IF mean intensity": np.log1p(raw_mean)}
+    res = {}
+    for cvname, cv, grp in [("5-fold tiles", KFold(5, shuffle=True, random_state=0), None),
+                            ("5-fold patient-grouped", GroupKFold(5), groups)]:
+        for fn, Xf in feats.items():
+            r = probe(Xf, cv, grp)
+            res[(cvname, fn)] = r
+            print(f"ridge probe [{cvname:22s}] {fn:34s} mean Spearman over 17 *_prop = {r.mean():.4f}")
+    print("per-marker Spearman (5-fold tiles): VirTues d0 | IF mean intensity")
+    a, b = res[("5-fold tiles", "VirTues (d0)")], res[("5-fold tiles", "baseline: IF mean intensity")]
+    print("  " + "  ".join(f"{n}:{x:.2f}|{y:.2f}" for n, x, y in zip(names, a, b)))
+    return res
+
+
+# ----------------------------------------------------------------------------------------------- main
+if __name__ == "__main__":
+    t0 = time.time()
+    df = load_all_tiles()
+    N = len(df)
+    keep_ch = [c for c in range(17) if not (NO_HOECHST and MARKERS[c][0] == "Hoechst")]
+    ids = [MARKERS[c][1] for c in range(17)]
+    print("marker -> UniProt:", ", ".join(f"{m}={u}" for m, u in MARKERS), "| using channels:", len(keep_ch))
+    ensure_marker_embeddings(sorted(set(ids)))
+    check_against_repo_embeddings(sorted(set(ids)))
+    model, midx_all = load_virtues(ids)
+    print(f"[{time.time() - t0:.0f}s] model ready; reading tiles (input {MPP_IN:.4f} um/px -> {SIZE}px = "
+          f"{MPP_IN * 333 / SIZE:.3f} um/px)", flush=True)
+
+    X, raw_mean = load_resampled(df)
+    slide = df.in_slide_name.values
+    X, uq, mean, std, valid, slide_k, n_zero = standardize(X, slide)
+    print(f"[{time.time() - t0:.0f}s] standardized. global log1p mean {np.round(mean, 2).tolist()}\n  std {np.round(std, 2).tolist()}")
+    print(f"  (slide, channel) pairs with 0.99-quantile == 0: {n_zero.sum()} of {n_zero.size} ("
+          + ", ".join(f"{MARKERS[c][0]}x{n_zero[:, c].sum()}" for c in range(17) if n_zero[:, c].any())
+          + (") -> dropped (spora quantile mask, --qc_drop)" if QC_DROP else ") -> uq floored at 1 grey level, channel kept"))
+
+    # per-tile channel set (quantile mask x optional Hoechst removal) -> batch tiles with identical channel sets
+    tile_valid = valid[[slide_k[s] for s in slide]]
+    tile_valid[:, [c for c in range(17) if c not in keep_ch]] = False
+    sig = [tuple(np.where(v)[0]) for v in tile_valid]
+    groups = {}
+    for i, g in enumerate(sig):
+        groups.setdefault(g, []).append(i)
+
+    out = np.zeros((N, 8, 512), np.float16)
+    torch.cuda.reset_peak_memory_stats()
+    done = 0
+    with torch.inference_mode(), torch.autocast("cuda", dtype=torch.float16):
+        for g, idxs in groups.items():
+            ch = torch.tensor(g, dtype=torch.long)
+            midx = midx_all[ch].to(dev)
+            for s in range(0, len(idxs), BS):
+                b = idxs[s:s + BS]
+                x = torch.from_numpy(X[b]).to(dev).float()[:, ch]  # B,C,H,W
+                xd = [dihedral(x, d) for d in range(8)]
+                crops = [xd[d][i] for d in range(8) for i in range(len(b))]  # order: d-major
+                ps = model.encoder.forward_list(crops, [midx] * len(crops)).patch_summary_tokens  # list of (21,21,512)
+                e = torch.stack([p.float().mean((0, 1)) for p in ps]).view(8, len(b), -1).transpose(0, 1)
+                out[b] = e.cpu().numpy().astype(np.float16)
+                done += len(b)
+                if (done // BS) % 50 == 0:
+                    print(f"  encoded {done}/{N} [{time.time() - t0:.0f}s] peak GPU "
+                          f"{torch.cuda.max_memory_allocated() / 2**30:.2f} GiB", flush=True)
+    fn = os.path.join(CACHE, "if_virtues" + ("_nohoechst" if NO_HOECHST else "") + ("_qcdrop" if QC_DROP else "") + ".npy")
+    np.save(fn, out)
+    print(f"saved {fn} {out.shape} {out.dtype} finite: {np.isfinite(out).all()} | channel-set groups: {len(groups)} | "
+          f"peak GPU alloc {torch.cuda.max_memory_allocated() / 2**30:.2f} GiB | encode+prep time {time.time() - t0:.0f}s")
+
+    if "--no_probe" not in sys.argv:
+        sanity(out, raw_mean, np.load(os.path.join(CACHE, "markers.npy")), df.patient.values)
+    print(f"total {time.time() - t0:.0f}s")

43_rnafm_aidocell.py

@@ -0,0 +1,230 @@
+"""Frozen AIDO.Cell-100M (a.k.a. GB.Cell-100M) cell embeddings of the RNA sets (each spot = one pseudo-cell).
+
+    python 43_rnafm_aidocell.py [set ...]        (default: all of rna_sets.SETS)
+
+Model   : HF genbio-ai/AIDO.Cell-100M @ a983b388e1025860f2c413ecef60e76078253c24 (model.safetensors, config.json;
+          identical sha256 to genbio-ai/GB.Cell-100M, the renamed repo). Licence "other" = GenBio AI Community
+          License: non-commercial use only; outputs (explicitly incl. embeddings) may not be used to train models
+          "similar to the GenBio Materials" (other cell foundation models) or for distillation.
+          Code reference: github genbio-ai/ModelGenerator @ bda6f2da (cloned to ext/modelgenerator, read, NOT
+          installed / imported), huggingface/gb.cell/gb_cell/{models/modeling_cellfoundation.py,
+          utils/gene_alignment.py, utils/preprocessing.py}, quickstart.ipynb, embed.py.
+          The forward pass of CellFoundationModel is re-implemented below; the bert.* tensors load with strict=True
+          (the only other tensors, cls.* = MLM head, are unused):
+            gene_embedding : auto-discretisation  v -> Linear(1,100) -> LeakyReLU(0.1) -> x + Linear(100,100)(x)
+                             (bin_alpha=1) -> softmax over 100 bins -> @ lookup_table (100 x 640)
+            + absolute position embedding (19266 x 640; position = gene index, 19264 genes + 2 depth tokens)
+            18 x layer     : pre-LN; MHA 20 heads x 32, no linear biases, scale 1/sqrt(32), FULL attention over all
+                             19,266 tokens (the official flash-attn path ignores the attention mask: "assuming NO
+                             PADDING"; its SDPA fallback likewise uses attn_mask=None) -> torch SDPA here, no
+                             flash-attn package needed; SwiGLU MLP 640->1664->640; final LayerNorm.
+          Weights and inputs in bf16 as in embed.py (model.to(torch.bfloat16)).
+Input   : official gene_alignment.align_adata: data genes matched by HGNC symbol to the fixed 19,264-gene list
+          (OS_scRNA_gene_index.19264.tsv, = scFoundation's list), other genes dropped, missing genes = 0 (still
+          attended to, as in the official code); duplicate data symbols keep the first column.
+          official preprocessing.preprocess_counts on the aligned raw counts: total = sum over the 19,264 aligned genes;
+          two depth tokens max(log10(total), 5) appended; X = log1p(count / total * 1e4); clip at 20; bf16.
+          (Deviation: an all-zero spot -- 1 each in hest_v / tcga_v after rounding the virtual counts -- gets an
+          all-zero gene input instead of the official 0/0 = NaN.)
+Readout : quickstart.ipynb cell-level embedding: last_hidden_state (after final LN) with the 2 depth tokens dropped,
+          mean over the gene positions whose gene is present in the data (align_adata attention_mask = 1; i.e. the
+          genes the dataset measured, whether or not expressed in that spot).  D = 640.  Not L2-normalised.
+Output  : rna_sets.save("aidocell", set, E) -> cache/rnafm/aidocell_<set>.npy  (n_spot, 640) float16.
+Log     : logs/rnafm/aidocell.log (stdout of this script, tee'd by the caller).
+"""
+import json
+import os
+import sys
+import time
+
+import numpy as np
+import pandas as pd
+import scipy.sparse as sp
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from huggingface_hub import hf_hub_download
+from safetensors.torch import load_file
+
+from common import CACHE, ROOT
+from rna_sets import SETS, load, save
+
+MODEL = "aidocell"
+REPO, REV = "genbio-ai/AIDO.Cell-100M", "a983b388e1025860f2c413ecef60e76078253c24"
+GENE_LIST = os.path.join(ROOT, "ext", "modelgenerator", "huggingface", "gb.cell", "gb_cell", "models", "gene_lists",
+                         "OS_scRNA_gene_index.19264.tsv")
+DEV = "cuda"
+
+
+# ----------------------------------------------------------------------------- model (re-implementation)
+class Layer(nn.Module):
+    def __init__(self, d, h, ff, eps):
+        super().__init__()
+        self.h = h
+        self.attention = nn.Module()
+        self.attention.ln = nn.LayerNorm(d, eps=eps)
+        self.attention.self = nn.Module()
+        for n in ["query", "key", "value"]:
+            setattr(self.attention.self, n, nn.Linear(d, d, bias=False))
+        self.attention.output = nn.Module()
+        self.attention.output.dense = nn.Linear(d, d, bias=False)
+        self.ln = nn.LayerNorm(d, eps=eps)
+        self.mlp = nn.Module()
+        self.mlp.up_proj, self.mlp.gate_proj = nn.Linear(d, ff, bias=False), nn.Linear(d, ff, bias=False)
+        self.mlp.down_proj = nn.Linear(ff, d, bias=False)
+
+    def forward(self, x):
+        B, S, D = x.shape
+        a, y = self.attention, self.attention.ln(x)
+        q, k, v = (getattr(a.self, n)(y).view(B, S, self.h, D // self.h).transpose(1, 2)
+                   for n in ["query", "key", "value"])
+        o = F.scaled_dot_product_attention(q, k, v, scale=(D // self.h) ** -0.5)
+        x = x + a.output.dense(o.transpose(1, 2).reshape(B, S, D))
+        y = self.ln(x)
+        return x + self.mlp.down_proj(F.silu(self.mlp.gate_proj(y)) * self.mlp.up_proj(y))
+
+
+class AIDOCell(nn.Module):
+    def __init__(self, c):
+        super().__init__()
+        assert c["hidden_act"] == "swiglu" and c["normalization_type"] == "LayerNorm" and not c["add_linear_bias"]
+        assert c["position_embedding_type"] == "absolute"
+        d, nb = c["hidden_size"], c["bin_num"]
+        self.alpha = c["bin_alpha"]
+        ge = nn.Module()
+        ge.linear1, ge.linear2 = nn.Linear(1, nb), nn.Linear(nb, nb)
+        ge.lookup_table, ge.mask_embedding, ge.pad_embedding = nn.Embedding(nb, d), nn.Embedding(1, d), nn.Embedding(1, d)
+        self.gene_embedding = ge
+        self.position_embedding = nn.Embedding(c["max_position_embeddings"] + 2, d)
+        self.encoder = nn.Module()
+        self.encoder.layer = nn.ModuleList([Layer(d, c["num_attention_heads"], c["intermediate_size"],
+                                                  c["layer_norm_eps"]) for _ in range(c["num_hidden_layers"])])
+        self.encoder.ln = nn.LayerNorm(d, eps=c["layer_norm_eps"])
+
+    def forward(self, X):                                 # X (B, S) preprocessed values >= 0
+        ge = self.gene_embedding
+        x = F.leaky_relu(ge.linear1(X.unsqueeze(-1)), 0.1)
+        x = self.alpha * x + ge.linear2(x)
+        x = torch.softmax(x, dim=-1) @ ge.lookup_table.weight
+        x = x + self.position_embedding.weight[:X.shape[1]]
+        for lyr in self.encoder.layer:
+            x = lyr(x)
+        return self.encoder.ln(x)
+
+
+def load_model():
+    cfg = json.load(open(hf_hub_download(REPO, "config.json", revision=REV)))
+    sd = load_file(hf_hub_download(REPO, "model.safetensors", revision=REV))
+    bert = {k[len("bert."):]: v for k, v in sd.items() if k.startswith("bert.")}
+    rest = sorted(k for k in sd if not k.startswith("bert."))
+    assert all(k.startswith("cls.") for k in rest), rest
+    model = AIDOCell(cfg)
+    model.load_state_dict(bert, strict=True)
+    print(f"loaded {REPO}@{REV[:8]}/model.safetensors: {len(bert)} bert.* tensors strictly "
+          f"({sum(p.numel() for p in model.parameters()) / 1e6:.1f}M params); unused MLM head: {rest}", flush=True)
+    return model.to(torch.bfloat16).eval().to(DEV), cfg
+
+
+# ----------------------------------------------------------------------------- input (official alignment / preprocessing)
+MODEL_GENES = pd.read_csv(GENE_LIST, sep="\t")["gene_name"].to_numpy()
+assert len(MODEL_GENES) == 19264 and len(set(MODEL_GENES)) == 19264
+
+
+def align(X, genes, name):
+    pos = {}
+    for j, g in enumerate(genes):
+        pos.setdefault(g, j)
+    mi = np.array([pos.get(g, -1) for g in MODEL_GENES])
+    present = mi >= 0
+    rows, cols = mi[present], np.nonzero(present)[0]
+    P = sp.csr_matrix((np.ones(len(rows), np.float32), (rows, cols)), shape=(len(genes), len(MODEL_GENES)))
+    Xa = (X @ P).tocsr()
+    n_dup = len(genes) - len(pos)
+    print(f"[{name}] data genes {len(genes)} ({n_dup} duplicate symbols); used {present.sum()}; "
+          f"coverage of the 19,264 AIDO genes: {present.mean():.1%}; data genes not in AIDO list: "
+          f"{len(pos) - present.sum()}", flush=True)
+    return Xa, present
+
+
+def preprocess(counts):
+    X = torch.from_numpy(counts).to(device=DEV, dtype=torch.float32)
+    tot = X.sum(1, keepdim=True)
+    depth = torch.clamp(torch.log10(tot), min=5.0)          # log10(0) = -inf -> 5 for an empty spot
+    X = torch.log1p(X / tot.clamp(min=1) * 10000)           # empty spot: all-zero input instead of 0/0 = NaN
+    X = torch.cat([X, depth, depth], dim=1)
+    return torch.clamp(X, max=20).to(torch.bfloat16)
+
+
+@torch.no_grad()
+def embed(model, Xa, present, bs=4):
+    G = Xa.shape[1]
+    m = torch.from_numpy(present.astype(np.float32)).to(DEV)[None, :, None]
+    E = np.zeros((Xa.shape[0], model.encoder.ln.normalized_shape[0]), np.float32)
+    t0, k = time.time(), 0
+    while k < Xa.shape[0]:
+        try:
+            h = model(preprocess(Xa[k:k + bs].toarray()))[:, :G].float()      # drop the 2 depth tokens
+            E[k:k + bs] = ((h * m).sum(1) / m.sum(1)).cpu().numpy()
+            k += bs
+            if (k // bs) % 250 == 0:
+                print(f"  {k}/{Xa.shape[0]}  {time.time() - t0:.0f}s", flush=True)
+        except torch.OutOfMemoryError:
+            torch.cuda.empty_cache()
+            bs = max(1, bs // 2)
+            print(f"  OOM -> batch {bs}", flush=True)
+    return E
+
+
+# ----------------------------------------------------------------------------- sanity
+def sanity(E, name, rng):
+    out = {"finite": bool(np.isfinite(E).all())}
+    s = rng.choice(len(E), min(200, len(E)), replace=False)
+    Z = E[s] / np.linalg.norm(E[s], axis=1, keepdims=True)
+    C = Z @ Z.T
+    out["mean_pairwise_cos200"] = float(C[np.triu_indices(len(s), 1)].mean())
+    if name == "hest":
+        organ = pd.read_csv(os.path.join(CACHE, "hest", "meta.csv")).organ.values
+        assert len(organ) == len(E)
+        Zt = torch.from_numpy(E / np.linalg.norm(E, axis=1, keepdims=True)).to(DEV)
+        nn_idx = []
+        for i in range(0, len(Zt), 2048):
+            sim = Zt[i:i + 2048] @ Zt.T
+            sim[torch.arange(sim.shape[0]), torch.arange(i, i + sim.shape[0])] = -2
+            nn_idx.append(sim.topk(10, dim=1).indices.cpu().numpy())
+        nn_idx = np.concatenate(nn_idx)
+        out["knn10_same_organ"] = float((organ[nn_idx] == organ[:, None]).mean())
+        cnt = pd.Series(organ).value_counts()
+        out["organ_base_rate"] = float(((cnt * (cnt - 1)).sum()) / (len(organ) * (len(organ) - 1)))
+    return out
+
+
+def main():
+    sets = sys.argv[1:] or SETS
+    t0 = time.time()
+    model, cfg = load_model()
+    report = {}
+    for s in sets:
+        t1 = time.time()
+        X, genes = load(s)
+        Xa, present = align(X, genes, s)
+        tot = np.asarray(Xa.sum(1)).ravel()
+        nexp = np.diff(Xa.indptr)
+        print(f"[{s}] aligned UMI/spot median {np.median(tot):.0f} (min {tot.min():.0f}); expressed genes/spot median "
+              f"{np.median(nexp):.0f}; spots with total>=1e5 (depth token > 5): {(tot >= 1e5).sum()}", flush=True)
+        if (tot == 0).any():
+            print(f"[{s}] {(tot == 0).sum()} empty spot(s) (all-zero counts): input = all zeros + depth tokens 5 "
+                  f"(the official code would give NaN from 0/0)", flush=True)
+        E = embed(model, Xa, present)
+        save(MODEL, s, E)
+        info = dict(n_genes=len(genes), n_aido_genes_present=int(present.sum()), coverage=float(present.mean()),
+                    median_expr=float(np.median(nexp)))
+        info.update(sanity(E, s, np.random.default_rng(0)))
+        info["D"], info["seconds"] = int(E.shape[1]), round(time.time() - t1, 1)
+        report[s] = info
+        print(f"[{s}] {json.dumps(info)}", flush=True)
+    print(f"done in {time.time() - t0:.0f}s; peak GPU mem {torch.cuda.max_memory_allocated() / 2**30:.2f} GB")
+    print(json.dumps(report, indent=1))
+
+
+if __name__ == "__main__":
+    main()

43_rnafm_se600m.py

@@ -0,0 +1,261 @@
+"""Frozen Arc Institute STATE embedding (SE-600M) spot embeddings of the 5 RNA sets (each spot = one pseudo-cell).
+
+    python 43_rnafm_se600m.py [set ...]          (default: all of rna_sets.SETS)
+
+Model     : arcinstitute/SE-600M (HF), revision 5a9a80f44f7ce32ce57059933ef0d735d7c10ce5, files model.safetensors
+            (metadata: converted from se600m_epoch16.ckpt, i.e. the final 16-epoch checkpoint), config.yaml and
+            protein_embeddings.pt (19,790 gene symbols -> ESM2-15B mean protein embedding, 5120-d; identical, in
+            the same order, to the checkpoint's pe_embedding.weight -- asserted). Licence: Arc Research Institute
+            State Model Non-Commercial License + Acceptable Use Policy (weights; non-commercial research use only),
+            code CC BY-NC-SA 4.0. Official code: github.com/ArcInstitute/state (commit 9bbfe78a, cloned to
+            ext/state/state, read but NOT imported / installed): the forward pass of src/state/emb/nn/model.py
+            (StateEmbeddingModel) + nn/flash_transformer.py is re-implemented below and loaded with a strict
+            state-dict load of all 238 tensors (the encoder / gene_embedding_layer tie is asserted); the unused
+            heads (binary_decoder, dataset_encoder, gene_embedding_layer) are dropped after loading.
+            16-layer post-LN transformer, d=2048, 16 heads, FFN 2048, GELU, no positional encoding, no attention mask.
+Inputs    : rna_sets.load(set) raw(-like) counts + HGNC symbols. Genes = the set's genes whose symbol is a key of
+            protein_embeddings.pt (official FilteredGenesCounts valid mask, set gene order kept); other genes dropped.
+Preprocess: official `state emb transform` path = Inference.encode_adata -> create_dataloader ->
+            VCIDatasetSentenceCollator.sample_cell_sentences (is_train=False):
+              * counts are raw -> log1p (the official per-cell raw-vs-log1p heuristic, max>35 or sum(expm1)>5e6, is
+                bypassed because all inputs are known raw counts; how many spots it would misclassify is logged);
+              * expression weight w = log1p / sum(log1p) over the valid genes (uniform if the spot is all-zero);
+              * cell sentence (pad_length 2048) = [CLS] + 2047 genes ranked by descending expression with random
+                tie-breaking (every set has > 2047 valid genes, so: all detected genes in rank order, top 2047 if more,
+                then randomly drawn undetected genes without replacement); seeded numpy RNG (seed 0) instead of
+                torch.randperm;
+              * token count value = 100 * w[gene]; position 0 carries 100 * w[local gene index 3] exactly as the
+                official collator (cls_token_idx = 3 is written as a local gene index before the count lookup);
+              * token embedding = L2-normalised ESM2 embedding, position 0 replaced by the learned cls_token, the
+                learned dataset_token appended (count embedding 0) -> 2049 tokens.
+Forward   : encoder(Linear 5120->2048, LayerNorm, SiLU) * sqrt(2048) + soft-binned count embedding
+            (softmax(count_encoder(c)) @ bin_encoder) -> 16 layers -> decoder (SkipBlock + Linear).
+            Whole model in bf16 (official get_precision_config('cuda') casts the model to bf16 + bf16 autocast).
+Readout   : official embedding = L2-normalised decoder output at the CLS position (StateEmbeddingModel.forward
+            `embedding`), D = 2048. `state emb transform` additionally concatenates a 10-d dataset-token vector
+            (dataset_embedder(decoder output at the dataset token)) -> 2058-d X_state; that batch-identity part is NOT
+            saved here (only the 2048-d cell CLS embedding).
+Output    : rna_sets.save("se600m", set, E) -> cache/rnafm/se600m_<set>.npy (n_spot, 2048) float16.
+Sanity    : finite, mean pairwise cosine of 200 random spots, (hest) organ agreement of cosine 10-NN vs base rate,
+            and (hest, 300 spots) cosine between two runs with different tie-breaking seeds (stochasticity of the
+            undetected-gene filler tokens).
+"""
+import math
+import os
+import sys
+import time
+
+import numpy as np
+import pandas as pd
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import yaml
+from huggingface_hub import hf_hub_download
+from safetensors.torch import load_file
+
+from common import CACHE
+from rna_sets import SETS, load, save
+
+MODEL = "se600m"
+REPO, REV = "arcinstitute/SE-600M", "5a9a80f44f7ce32ce57059933ef0d735d7c10ce5"
+dev = "cuda"
+DT = torch.bfloat16
+
+
+# ----------------------------------------------------------------------------- model (re-implemented from ext/state)
+class SkipBlock(nn.Module):
+    def __init__(self, d):
+        super().__init__()
+        self.intermediate_dense = nn.Linear(d, 2 * d)
+        self.dense = nn.Linear(2 * d, d)
+        self.layer_norm = nn.LayerNorm(d)
+
+    def forward(self, x):
+        return self.layer_norm(x + self.dense(F.relu(self.intermediate_dense(x))))
+
+
+class Layer(nn.Module):                    # FlashTransformerEncoderLayer (post-LN, dropout 0 at inference)
+    def __init__(self, d, nhead, d_hid):
+        super().__init__()
+        self.nhead = nhead
+        self.qkv_proj = nn.Linear(d, 3 * d)
+        self.out_proj = nn.Linear(d, d)
+        self.norm1, self.norm2 = nn.LayerNorm(d), nn.LayerNorm(d)
+        self.linear1, self.linear2 = nn.Linear(d, d_hid), nn.Linear(d_hid, d)
+
+    def forward(self, x):
+        B, T, D = x.shape
+        q, k, v = self.qkv_proj(x).chunk(3, dim=-1)
+        q, k, v = (t.view(B, T, self.nhead, D // self.nhead).transpose(1, 2) for t in (q, k, v))
+        a = F.scaled_dot_product_attention(q, k, v, is_causal=False).transpose(1, 2).reshape(B, T, D)
+        x = self.norm1(x + self.out_proj(a))
+        return self.norm2(x + self.linear2(F.gelu(self.linear1(x))))
+
+
+class TransformerEncoder(nn.Module):
+    def __init__(self, layers):
+        super().__init__()
+        self.layers = nn.ModuleList(layers)
+
+
+class SE(nn.Module):
+    def __init__(self, token_dim, d, nhead, d_hid, nlayers, out_dim, n_pe, z_dim, n_ds):
+        super().__init__()
+        self.d = d
+        self.cls_token = nn.Parameter(torch.zeros(1, token_dim))
+        self.dataset_token = nn.Parameter(torch.zeros(1, token_dim))
+        self.encoder = nn.Sequential(nn.Linear(token_dim, d), nn.LayerNorm(d), nn.SiLU())
+        self.gene_embedding_layer = self.encoder                       # tied, as in the official model
+        self.transformer_encoder = TransformerEncoder([Layer(d, nhead, d_hid) for _ in range(nlayers)])
+        self.decoder = nn.Sequential(SkipBlock(d), nn.Linear(d, out_dim))
+        self.binary_decoder = nn.Sequential(SkipBlock(out_dim + d + z_dim), SkipBlock(out_dim + d + z_dim),
+                                            nn.Linear(out_dim + d + z_dim, 1))
+        self.bin_encoder = nn.Embedding(10, d)
+        self.count_encoder = nn.Sequential(nn.Linear(1, 512), nn.LeakyReLU(), nn.Linear(512, 10))
+        self.dataset_embedder = nn.Linear(out_dim, 10)
+        self.dataset_encoder = nn.Sequential(nn.Linear(out_dim, d), nn.SiLU(), nn.LayerNorm(d), nn.Dropout(0.1),
+                                             nn.Linear(d, n_ds))
+        self.pe_embedding = nn.Embedding(n_pe, token_dim)
+
+    def forward(self, tok, cnt):
+        """tok: (B, L) global gene ids (position 0 = placeholder for CLS); cnt: (B, L) count values."""
+        x = F.normalize(self.pe_embedding(tok), dim=2)
+        x[:, 0, :] = self.cls_token.expand(x.size(0), -1)
+        x = torch.cat([x, self.dataset_token.expand(x.size(0), -1).unsqueeze(1)], 1)
+        x = self.encoder(x) * math.sqrt(self.d)
+        w = F.softmax(self.count_encoder(cnt.unsqueeze(-1)), dim=-1)
+        ce = w @ self.bin_encoder.weight
+        x = x + torch.cat([ce, torch.zeros_like(ce[:, :1])], 1)
+        for lyr in self.transformer_encoder.layers:
+            x = lyr(x)
+        return F.normalize(self.decoder(x)[:, 0, :], dim=1)
+
+
+def build():
+    cfg = yaml.safe_load(open(hf_hub_download(REPO, "config.yaml", revision=REV)))
+    m, ds = cfg["model"], cfg["dataset"]
+    assert m["counts"] and m["rda"] and m["dataset_correction"] and ds["pad_length"] == 2048
+    assert ds["cls_token_idx"] == 3 and cfg["loss"]["name"] == "tabular"
+    pe = torch.load(hf_hub_download(REPO, "protein_embeddings.pt", revision=REV), map_location="cpu",
+                    weights_only=True)
+    sd = load_file(hf_hub_download(REPO, "model.safetensors", revision=REV))
+    n_ds = cfg["dataset"][cfg["dataset"]["current"]]["num_datasets"]
+    model = SE(cfg["tokenizer"]["token_dim"], m["emsize"], m["nhead"], m["d_hid"], m["nlayers"], m["output_dim"],
+               len(pe), 1 + 10, n_ds)
+    for k in ["0.weight", "0.bias", "1.weight", "1.bias"]:
+        assert torch.equal(sd[f"encoder.{k}"], sd[f"gene_embedding_layer.{k}"])
+    model.load_state_dict(sd, strict=True)
+    assert torch.equal(model.pe_embedding.weight, torch.vstack(list(pe.values())))   # CLI's protein_embeddings.pt
+    del model.binary_decoder, model.dataset_encoder, model.gene_embedding_layer
+    model = model.eval().to(dev, DT)
+    print(f"[{MODEL}] loaded {len(sd)} tensors strictly; config pad_length={ds['pad_length']} "
+          f"{m['nlayers']}x{m['emsize']} heads={m['nhead']}", flush=True)
+    return model, cfg, {g: i for i, g in enumerate(pe)}
+
+
+# ----------------------------------------------------------------------------- tokenisation (official collator)
+def sentences(X, genes, g2i, pad_len, cls_idx, seed):
+    valid = np.array([g in g2i for g in genes])
+    Xv = X[:, valid].tocsr()
+    gid = np.array([g2i[g] for g in genes[valid]], dtype=np.int64)
+    n = Xv.shape[1]
+    assert n >= pad_len - 1                                         # -> official "top pad_len-1 genes" branch
+    rng = np.random.default_rng(seed)
+    T = np.zeros((Xv.shape[0], pad_len), np.int64)
+    C = np.zeros((Xv.shape[0], pad_len), np.float32)
+    with np.errstate(over="ignore"):                                # official is_raw_integer_counts heuristic
+        n_heur = int(sum((r.data.max(initial=0) <= 35) and (np.expm1(r.data.astype(np.float64)).sum() <= 5e6)
+                         for r in (X.getrow(i) for i in range(X.shape[0]))))   # (on the full row, as official)
+    n_trunc = 0
+    for r in range(Xv.shape[0]):
+        a, b = Xv.indptr[r], Xv.indptr[r + 1]
+        nz, raw = Xv.indices[a:b], Xv.data[a:b].astype(np.float64)
+        lv = np.log1p(raw)
+        s = lv.sum()
+        w = np.zeros(n)
+        if s > 0:
+            w[nz] = lv / s
+        else:
+            w[:] = 1.0 / n                                          # softmax of an all-zero row
+        p = rng.permutation(len(nz))                                # random tie-break among expressed genes
+        order = nz[p][np.argsort(-lv[p], kind="stable")][:pad_len - 1]
+        n_trunc += len(nz) > pad_len - 1
+        k = pad_len - 1 - len(order)
+        if k > 0:                                                   # random undetected genes, no replacement
+            zero = np.setdiff1d(np.arange(n), nz, assume_unique=True)
+            order = np.r_[order, rng.choice(zero, k, replace=False)]
+        loc = np.r_[cls_idx, order]
+        T[r] = gid[loc]
+        C[r] = 100.0 * w[loc]
+    stats = dict(valid=int(valid.sum()), n_gene=len(genes), heur_log=n_heur, trunc=n_trunc,
+                 det_med=float(np.median(np.diff(Xv.indptr))))
+    return T, C, stats
+
+
+def embed(model, T, C, bs=16):
+    E = np.zeros((len(T), model.decoder[1].out_features), np.float32)
+    i = 0
+    with torch.no_grad(), torch.autocast("cuda", dtype=DT):
+        while i < len(T):
+            try:
+                t = torch.from_numpy(T[i:i + bs]).to(dev)
+                c = torch.from_numpy(C[i:i + bs]).to(dev, DT)
+                E[i:i + bs] = model(t, c).float().cpu().numpy()
+                i += bs
+            except torch.OutOfMemoryError:
+                torch.cuda.empty_cache()
+                assert bs > 1, "OOM at batch size 1"
+                bs //= 2
+                print(f"  OOM -> batch size {bs}", flush=True)
+    return E
+
+
+def sanity(E, name):
+    rng = np.random.default_rng(0)
+    S = E[rng.choice(len(E), min(200, len(E)), replace=False)]
+    S = S / np.linalg.norm(S, axis=1, keepdims=True)
+    mc = float((S @ S.T)[np.triu_indices(len(S), 1)].mean())
+    msg = f"[{MODEL}] {name}: finite={np.isfinite(E).all()} std={E.std(0).mean():.4f} mean-pair-cos(200)={mc:.4f}"
+    if name == "hest":
+        org = pd.read_csv(os.path.join(CACHE, "hest", "meta.csv")).organ.values
+        assert len(org) == len(E)
+        Z = torch.from_numpy(E / np.linalg.norm(E, axis=1, keepdims=True)).to(dev)
+        nn_idx = []
+        for k in range(0, len(Z), 2048):
+            s = Z[k:k + 2048] @ Z.T
+            s[torch.arange(s.shape[0]), torch.arange(k, k + s.shape[0])] = -2
+            nn_idx.append(s.topk(10, dim=1).indices.cpu().numpy())
+        nn_idx = np.concatenate(nn_idx)
+        knn = float((org[nn_idx] == org[:, None]).mean())
+        p = pd.Series(org).value_counts(normalize=True).values
+        msg += f"  10-NN same-organ={knn:.3f} (base rate {float((p ** 2).sum()):.3f})"
+    print(msg, flush=True)
+    return mc
+
+
+if __name__ == "__main__":
+    torch.cuda.set_per_process_memory_fraction(min(1.0, 8 * 2 ** 30 / torch.cuda.get_device_properties(0).total_memory))
+    t0 = time.time()
+    model, cfg, g2i = build()
+    PAD, CLS = cfg["dataset"]["pad_length"], cfg["dataset"]["cls_token_idx"]
+    for name in (sys.argv[1:] or SETS):
+        t1 = time.time()
+        X, genes = load(name)
+        T, C, st = sentences(X, genes, g2i, PAD, CLS, seed=0)
+        print(f"[{MODEL}] {name}: {X.shape[0]} spots; gene coverage {st['valid']}/{st['n_gene']} "
+              f"({st['valid'] / st['n_gene']:.1%}) in protein_embeddings; detected valid genes/spot median "
+              f"{st['det_med']:.0f}; spots with >2047 detected (truncated) {st['trunc']}; spots the official "
+              f"raw-vs-log1p heuristic would treat as log1p {st['heur_log']}", flush=True)
+        E = embed(model, T, C)
+        if name == "hest":                                          # tie-breaking stochasticity check
+            idx = np.random.default_rng(1).choice(len(T), 300, replace=False)
+            T2, C2, _ = sentences(X[idx], genes, g2i, PAD, CLS, seed=1)
+            c = (embed(model, T2, C2) * E[idx]).sum(1)
+            print(f"  seed-0 vs seed-1 tie-breaking: per-spot cosine median {np.median(c):.4f} min {c.min():.4f}",
+                  flush=True)
+        sanity(E, name)
+        save(MODEL, name, E)
+        print(f"  {name}: {time.time() - t1:.0f}s  peak GPU {torch.cuda.max_memory_allocated() / 2 ** 30:.2f} GB",
+              flush=True)
+    print(f"[{MODEL}] done in {time.time() - t0:.0f}s", flush=True)

43_rnafm_tahoex1.py

@@ -0,0 +1,260 @@
+"""Frozen Tahoe-x1 70M cell embeddings of the RNA sets (each Visium / virtual spot = one pseudo-cell).
+
+    python 43_rnafm_tahoex1.py [set ...]        (default: all of rna_sets.SETS)
+
+Model   : HF tahoebio/Tahoe-x1 @ d218a580b9c2500ae9dfc8367a398545e6f017a8, subfolder 70m-model/
+          (model.safetensors, model_config.yml, collator_config.yml, vocab.json); licence apache-2.0.
+          Code reference: github tahoebio/tahoe-x1 @ 167cc19f (cloned to ext/tahoe-x1, read, NOT installed / imported;
+          it needs llm-foundry + composer). The forward pass of tahoe_x1.model.TXModel is re-implemented below and
+          the safetensors state dict is loaded with strict=True:
+            gene_encoder   : Embedding(62720, 512, pad=0) -> LayerNorm
+            expr encoder   : ContinuousValueEncoder  clamp(max=512) -> Linear(1,512) -> ReLU -> Linear -> LayerNorm
+            12 x TXBlock   : pre-norm; llm-foundry GroupedQueryAttention with kv_n_heads = n_heads = 8 (= MHA),
+                             fused Wqkv split [q | k | v], softmax scale 1/sqrt(64), non-causal, padding keys
+                             excluded (flash-attn unpadding == SDPA with a key-padding mask here);
+                             FFN Linear(512,2048) -> ReLU -> Linear(2048,512); final LayerNorm.
+          (llm-foundry source read from ext/llmfoundry_src/repo @ 0cdb2f42 to confirm the Wqkv split / scale.)
+          The flag (gen-mask) embedding is 0 at inference (no MLM masking); decoders are unused.
+Input   : gene symbols -> Ensembl (10x GRCh38 map in cache/external/dlpfc_genes_full.csv, fallback Geneformer
+          gene_name_id_dict_gc104M; DLPFC uses its own Ensembl column) -> Tahoe vocab id; duplicate Ensembl ids keep
+          the first column.  Exactly the official inference collator (tahoe_x1.utils.loader_from_adata +
+          data.collator.DataCollator, do_mlm=False): per spot the expressed (count>0) in-vocab genes, <cls> prepended
+          with value pad_value=-2; raw counts -> per-cell quantile binning over the expressed genes
+          (num_bins=51, torch.quantile edges, bucketize right=False, +1 -> bins 1..50; no normalisation /
+          log, log_transform=False); then if longer than max_length a random subset is sampled (sampling=True,
+          <cls> kept, seed 0). max_length = 2048 = the documented inference setting (predict.yaml /
+          clustering_tutorial `seq_len_dataset: 2048`); collator_config.yml's max_length=1024 is the training length.
+          Pad token 0 / value -2.
+Readout : model_config cell_emb_style = "cls": final-layer (after final LayerNorm) <cls> output, L2-normalised
+          (as in tahoe_x1.tasks.get_batch_embeddings / predict_embeddings). bf16 autocast (precision amp_bf16).
+Output  : rna_sets.save("tahoex1", set, E) -> cache/rnafm/tahoex1_<set>.npy  (n_spot, 512) float16.
+Log     : logs/rnafm/tahoex1.log (stdout of this script, tee'd by the caller).
+"""
+import json
+import os
+import pickle
+import sys
+import time
+
+import numpy as np
+import pandas as pd
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+import yaml
+from huggingface_hub import hf_hub_download
+from safetensors.torch import load_file
+
+from common import CACHE
+from rna_sets import SETS, load, save
+
+MODEL = "tahoex1"
+REPO, REV = "tahoebio/Tahoe-x1", "d218a580b9c2500ae9dfc8367a398545e6f017a8"
+SUB = "70m-model"
+MAX_LEN = 2048
+DEV = "cuda"
+
+
+# ----------------------------------------------------------------------------- model (re-implementation)
+class Block(nn.Module):
+    def __init__(self, d, h, ff, eps):
+        super().__init__()
+        self.h = h
+        self.self_attn = nn.Module()
+        self.self_attn.Wqkv = nn.Linear(d, 3 * d)
+        self.self_attn.out_proj = nn.Linear(d, d)
+        self.up_proj, self.down_proj = nn.Linear(d, ff), nn.Linear(ff, d)
+        self.norm1, self.norm2 = nn.LayerNorm(d, eps=eps), nn.LayerNorm(d, eps=eps)
+
+    def forward(self, x, kmask):                       # kmask (B,1,1,S) bool, True = attend
+        B, S, D = x.shape
+        q, k, v = self.self_attn.Wqkv(self.norm1(x)).split(D, dim=2)
+        q, k, v = (t.view(B, S, self.h, D // self.h).transpose(1, 2) for t in (q, k, v))
+        a = F.scaled_dot_product_attention(q, k, v, attn_mask=kmask, scale=(D // self.h) ** -0.5)
+        x = x + self.self_attn.out_proj(a.transpose(1, 2).reshape(B, S, D))
+        return x + self.down_proj(F.relu(self.up_proj(self.norm2(x))))
+
+
+class TahoeX1(nn.Module):
+    def __init__(self, cfg):
+        super().__init__()
+        d, eps = cfg["d_model"], cfg["norm_config"]["eps"]
+        assert cfg["norm_scheme"] == "pre" and not cfg["use_glu"] and cfg["transformer_activation"] == "relu"
+        assert cfg["attn_config"]["kv_nheads"] == cfg["n_heads"] and cfg["norm_config"]["norm_type"] == "layernorm"
+        assert cfg["expression_encoder"]["input_emb_style"] == "continuous" and cfg["cell_emb_style"] == "cls"
+        self.max_value = cfg["expression_encoder"]["max_value"]
+        ge = nn.Module()
+        ge.embedding, ge.enc_norm = nn.Embedding(cfg["vocab_size"], d, padding_idx=0), nn.LayerNorm(d)
+        self.gene_encoder = ge
+        self.flag_encoder = nn.Embedding(2, d)
+        ee = nn.Module()
+        ee.linear1, ee.linear2, ee.norm = nn.Linear(1, d), nn.Linear(d, d), nn.LayerNorm(d)
+        self.expression_encoder = ee
+        te = nn.Module()
+        te.layers = nn.ModuleList([Block(d, cfg["n_heads"], d * cfg["expansion_ratio"], eps)
+                                   for _ in range(cfg["n_layers"])])
+        te.norm = nn.LayerNorm(d, eps=eps)
+        self.transformer_encoder = te
+        # unused heads, present only so that the checkpoint loads strictly
+        dec = nn.Module()
+        dec.linear_layers = nn.ModuleList([nn.Linear(d, d) for _ in range(cfg["expression_decoder"]["n_layers"])])
+        dec.out_proj = nn.Linear(d, cfg["expression_decoder"]["n_outputs"])
+        self.expression_decoder = dec
+        mvc = nn.Module()
+        mvc.gene2query, mvc.W = nn.Linear(d, d), nn.Linear(d, d, bias=False)
+        self.mvc_decoder = mvc
+
+    def forward(self, genes, values, pad_mask):          # pad_mask True = real token
+        ee = self.expression_encoder
+        x = self.gene_encoder.enc_norm(self.gene_encoder.embedding(genes))
+        v = torch.clamp(values.unsqueeze(-1), max=self.max_value)
+        x = x + ee.norm(ee.linear2(F.relu(ee.linear1(v))))      # + flag term = 0 (no gen/masked tokens)
+        km = pad_mask[:, None, None, :]
+        for blk in self.transformer_encoder.layers:
+            x = blk(x, km)
+        return self.transformer_encoder.norm(x)
+
+
+def load_model():
+    f = {n: hf_hub_download(REPO, f"{SUB}/{n}", revision=REV)
+         for n in ["model.safetensors", "model_config.yml", "collator_config.yml", "vocab.json"]}
+    mcfg = yaml.safe_load(open(f["model_config.yml"]))
+    ccfg = yaml.safe_load(open(f["collator_config.yml"]))
+    vocab = json.load(open(f["vocab.json"]))
+    sd = {k[len("model."):]: v for k, v in load_file(f["model.safetensors"]).items()}
+    model = TahoeX1(mcfg)
+    model.load_state_dict(sd, strict=True)
+    print(f"loaded {REPO}@{REV[:8]}/{SUB}/model.safetensors strictly: {len(sd)} tensors, "
+          f"{sum(p.numel() for p in model.parameters()) / 1e6:.1f}M params", flush=True)
+    return model.eval().to(DEV), mcfg, ccfg, vocab
+
+
+# ----------------------------------------------------------------------------- collator (official logic)
+def binning(row, n_bins):
+    """tahoe_x1.data.collator.binning for a strictly positive row, right=False."""
+    grades = torch.linspace(0, 1, n_bins - 1, dtype=torch.float32)
+    bins = torch.quantile(row, grades)
+    return torch.bucketize(row, bins, right=False) + 1
+
+
+def symbol_to_ensembl(genes, name):
+    if name == "dlpfc":
+        d = pd.read_csv(os.path.join(CACHE, "external", "dlpfc_genes_full.csv"))
+        assert (d.gene.values == genes).all()
+        return d.ensembl_id.values.astype(str)
+    d = pd.read_csv(os.path.join(CACHE, "external", "dlpfc_genes_full.csv"))
+    m10 = dict(zip(d.gene, d.ensembl_id))
+    from huggingface_hub import hf_hub_download
+    gf = pickle.load(open(hf_hub_download("ctheodoris/Geneformer", "geneformer/gene_name_id_dict_gc104M.pkl",
+                                          revision="1f7fbae4e469a5f4f1af8c111a529cfe1b3829f5"), "rb"))
+    return np.array([m10.get(g) or gf.get(g) or "" for g in genes])
+
+
+def make_seqs(X, genes, name, vocab, ccfg, rng):
+    ens = symbol_to_ensembl(genes, name)
+    tid = np.array([vocab.get(e, -1) for e in ens])
+    keep = tid >= 0
+    first = np.zeros(len(tid), bool)
+    _, fi = np.unique(np.where(keep, tid, -1), return_index=True)
+    first[fi] = True
+    keep &= first
+    cols = np.nonzero(keep)[0]
+    Xs = X[:, cols].tocsr()
+    gid = tid[cols]
+    print(f"[{name}] genes {len(genes)}; Ensembl-mapped {int((ens != '').sum())}; in Tahoe vocab "
+          f"(unique) {len(cols)} ({len(cols) / len(genes):.1%})", flush=True)
+    CLS, PADV, NB = vocab["<cls>"], float(ccfg["pad_value"]), int(ccfg["num_bins"])
+    seqs, nexp, nsamp = [], [], 0
+    for r in range(Xs.shape[0]):
+        lo, hi = Xs.indptr[r], Xs.indptr[r + 1]
+        idx, val = Xs.indices[lo:hi], Xs.data[lo:hi]
+        pos = val > 0
+        idx, val = idx[pos], val[pos]
+        nexp.append(len(idx))
+        b = binning(torch.from_numpy(val.astype(np.float32)), NB).numpy().astype(np.float32) if len(idx) else \
+            np.zeros(0, np.float32)
+        g = np.r_[CLS, gid[idx]].astype(np.int64)
+        v = np.r_[PADV, b].astype(np.float32)
+        if len(g) > MAX_LEN:                               # _sample: keep <cls>, random subset of the rest
+            sel = np.r_[0, rng.permutation(len(g) - 1)[:MAX_LEN - 1] + 1]
+            g, v = g[sel], v[sel]
+            nsamp += 1
+        seqs.append((g, v))
+    nexp = np.array(nexp)
+    print(f"[{name}] expressed in-vocab genes/spot median {np.median(nexp):.0f} (min {nexp.min()}, max {nexp.max()}); "
+          f"spots subsampled to {MAX_LEN}: {nsamp}/{len(seqs)}; empty spots: {(nexp == 0).sum()}", flush=True)
+    return seqs, dict(n_genes=len(genes), n_in_vocab=int(len(cols)), median_expr=float(np.median(nexp)))
+
+
+@torch.no_grad()
+def embed(model, seqs, bs=32):
+    order = np.argsort([len(g) for g, _ in seqs])
+    E = np.zeros((len(seqs), model.gene_encoder.embedding.embedding_dim), np.float32)
+    k = 0
+    while k < len(order):
+        idx = order[k:k + bs]
+        try:
+            b = [seqs[i] for i in idx]
+            L = max(len(g) for g, _ in b)
+            G = torch.zeros((len(b), L), dtype=torch.long)
+            V = torch.full((len(b), L), -2.0)
+            for j, (g, v) in enumerate(b):
+                G[j, :len(g)], V[j, :len(v)] = torch.from_numpy(g), torch.from_numpy(v)
+            G, V = G.to(DEV), V.to(DEV)
+            with torch.autocast("cuda", dtype=torch.bfloat16):
+                h = model(G, V, G.ne(0))
+            E[idx] = h[:, 0].float().cpu().numpy()
+            k += len(idx)
+        except torch.OutOfMemoryError:
+            torch.cuda.empty_cache()
+            bs = max(1, bs // 2)
+            print(f"  OOM -> batch {bs}", flush=True)
+    return E / np.linalg.norm(E, axis=1, keepdims=True)
+
+
+# ----------------------------------------------------------------------------- sanity
+def sanity(E, name, rng):
+    out = {"finite": bool(np.isfinite(E).all())}
+    s = rng.choice(len(E), min(200, len(E)), replace=False)
+    Z = E[s] / np.linalg.norm(E[s], axis=1, keepdims=True)
+    C = Z @ Z.T
+    out["mean_pairwise_cos200"] = float(C[np.triu_indices(len(s), 1)].mean())
+    if name == "hest":
+        organ = pd.read_csv(os.path.join(CACHE, "hest", "meta.csv")).organ.values
+        assert len(organ) == len(E)
+        Zt = torch.from_numpy(E / np.linalg.norm(E, axis=1, keepdims=True)).to(DEV)
+        nn_idx = []
+        for i in range(0, len(Zt), 2048):
+            sim = Zt[i:i + 2048] @ Zt.T
+            sim[torch.arange(sim.shape[0]), torch.arange(i, i + sim.shape[0])] = -2
+            nn_idx.append(sim.topk(10, dim=1).indices.cpu().numpy())
+        nn_idx = np.concatenate(nn_idx)
+        out["knn10_same_organ"] = float((organ[nn_idx] == organ[:, None]).mean())
+        cnt = pd.Series(organ).value_counts()
+        out["organ_base_rate"] = float(((cnt * (cnt - 1)).sum()) / (len(organ) * (len(organ) - 1)))
+    return out
+
+
+def main():
+    sets = sys.argv[1:] or SETS
+    torch.manual_seed(0)
+    t0 = time.time()
+    model, mcfg, ccfg, vocab = load_model()
+    print(f"collator_config: {ccfg}", flush=True)
+    report = {}
+    for s in sets:
+        t1 = time.time()
+        X, genes = load(s)
+        seqs, info = make_seqs(X, genes, s, vocab, ccfg, np.random.default_rng(0))
+        E = embed(model, seqs)
+        save(MODEL, s, E)
+        info.update(sanity(E, s, np.random.default_rng(0)))
+        info["D"], info["seconds"] = int(E.shape[1]), round(time.time() - t1, 1)
+        report[s] = info
+        print(f"[{s}] {json.dumps(info)}", flush=True)
+    print(f"done in {time.time() - t0:.0f}s; peak GPU mem {torch.cuda.max_memory_allocated() / 2**30:.2f} GB")
+    print(json.dumps(report, indent=1))
+
+
+if __name__ == "__main__":
+    main()

45_text_feats.py

@@ -0,0 +1,82 @@
+"""Frozen embeddings of every text set with additional (bio)medical / general text-embedding models.
+
+    python 45_text_feats.py <name> [<name> ...]
+
+Text sets (same row order as the existing BiomedCLIP files):
+  alltxt         cache/alltxt_meta.csv caption        (TCGA real PathGen texts, 6480, junk rows kept, filtered later)
+  orion_gentext  cache/orion_gentext_meta.csv caption (GPT synthetic captions of the ORION tiles, test only)
+  tcga_notext    cache/tcga_notext_meta.csv caption   (synthetic captions of TCGA images without real text, test only)
+  dlpfc_gentext  cache/dlpfc_gentext_meta.csv caption (external DLPFC synthetic captions, test only)
+Models (pooling / normalisation as on each model card; documents embedded without a query instruction):
+  biomedclip   BiomedCLIP text tower (biomedclip_text.py, 256 tokens) - only for the sets that lack it
+  medcpt       ncbi/MedCPT-Article-Encoder, [CLS], 512 tokens (PubMed search-log contrastive)
+  biolord      FremyCompany/BioLORD-2023, mean pooling, 512 (clinical sentences / concepts, UMLS-based)
+  pubmedbert   NeuML/pubmedbert-base-embeddings, mean pooling, 512 (PubMed title-abstract)
+  medembed     abhinand/MedEmbed-large-v0.1 (bge-large medical fine-tune), [CLS], 512
+  bge          BAAI/bge-large-en-v1.5, [CLS], 512 (general-domain reference)
+  qwen3-0.6b / qwen3-4b   Qwen/Qwen3-Embedding-*, last-token pooling, left padding, 1024 tokens
+Output cache/<set>_text_<name>.npy (n, 1, D) float16 (raw, not normalised).
+"""
+import os
+import sys
+import time
+
+import numpy as np
+import pandas as pd
+import torch
+from transformers import AutoModel, AutoTokenizer
+
+from common import CACHE
+
+dev = "cuda"
+SETS = {"alltxt": "alltxt_meta.csv", "orion_gentext": "orion_gentext_meta.csv", "tcga_notext": "tcga_notext_meta.csv",
+        "dlpfc_gentext": "dlpfc_gentext_meta.csv"}
+HF = {"medcpt": ("ncbi/MedCPT-Article-Encoder", "cls", 512), "biolord": ("FremyCompany/BioLORD-2023", "mean", 512),
+      "pubmedbert": ("NeuML/pubmedbert-base-embeddings", "mean", 512),
+      "medembed": ("abhinand/MedEmbed-large-v0.1", "cls", 512), "bge": ("BAAI/bge-large-en-v1.5", "cls", 512),
+      "qwen3-0.6b": ("Qwen/Qwen3-Embedding-0.6B", "last", 1024), "qwen3-4b": ("Qwen/Qwen3-Embedding-4B", "last", 1024)}
+
+
+def encoder(name):
+    if name == "biomedclip":
+        from biomedclip_text import BiomedCLIPText
+        m = BiomedCLIPText().to(dev).eval()
+        return m, 64
+    repo, pool, L = HF[name]
+    tk = AutoTokenizer.from_pretrained(repo, padding_side="left" if pool == "last" else "right")
+    m = AutoModel.from_pretrained(repo, torch_dtype=torch.bfloat16 if pool == "last" else torch.float32).to(dev).eval()
+
+    def f(b):
+        x = tk(b, padding=True, truncation=True, max_length=L, return_tensors="pt").to(dev)
+        h = m(**x).last_hidden_state
+        if pool == "cls":
+            return h[:, 0]
+        if pool == "mean":
+            a = x["attention_mask"][..., None].to(h.dtype)
+            return (h * a).sum(1) / a.sum(1)
+        return h[:, -1]  # left padding -> last position is the last real token
+    return f, (16 if name == "qwen3-4b" else 64)
+
+
+for name in sys.argv[1:]:
+    t0 = time.time()
+    f, bs = encoder(name)
+    for st, meta in SETS.items():
+        out_f = os.path.join(CACHE, f"{st}_text_{name}.npy")
+        if os.path.exists(out_f):
+            continue
+        txt = pd.read_csv(os.path.join(CACHE, meta)).caption.fillna("").astype(str).tolist()
+        order = np.argsort([len(t) for t in txt])  # length-sorted batches, restored below
+        E = None
+        with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
+            for i in range(0, len(txt), bs):
+                idx = order[i:i + bs]
+                z = f([txt[j] for j in idx]).float().cpu().numpy()
+                if E is None:
+                    E = np.zeros((len(txt), z.shape[1]), np.float32)
+                E[idx] = z
+        assert np.isfinite(E).all()
+        np.save(out_f, E[:, None].astype(np.float16))
+        print(f"[{name}] {st}: {E.shape}  {time.time() - t0:.0f}s", flush=True)
+    del f
+    torch.cuda.empty_cache()

45b_text_feats_pathvlm.py

@@ -0,0 +1,316 @@
+"""Frozen TEXT-tower embeddings of two pathology vision-language models (same output format as 45_text_feats.py).
+
+    python 45b_text_feats_pathvlm.py <name> [<name> ...]     # name in {keep, quiltnet, quiltnet-gelu}
+    python 45b_text_feats_pathvlm.py verify [keep|quiltnet]    # checks against the official code (see below)
+
+Text sets / row order / output exactly as 45_text_feats.py: cache/<set>_text_<name>.npy, (n, 1, D) float16, raw
+(pre-normalisation) embedding, for alltxt, orion_gentext, tcga_notext, dlpfc_gentext (column `caption`).
+Inference in fp32 (no autocast; the models are small) and cast to float16 at save time.
+Runtime (shared A5000, 1 loaded CPU): keep 85 s for all 4 sets, quiltnet + quiltnet-gelu 35 s.
+Leakage check (papers + repos, 2026-10-24): neither model was pretrained on PathGen-1.6M or other TCGA captions.
+  KEEP: OpenPath (Twitter + LAION) + Quilt-1M, regrouped into 143k semantic groups (Loie/KEEP_dataset); text
+  encoder PubMedBERT-arch., initialised from a knowledge encoder trained on their disease KG (UMLS-based).
+  QuiltNet: OpenAI CLIP fine-tuned on Quilt-1M = YouTube 802k + PubMed OA 59k + Twitter/OpenPath list ~134k
+  + LAION 23k (paper June 2023, a year before PathGen-1.6M, July 2024). TCGA appears only as evaluation data.
+
+keep      KEEP (Zhou et al., Cancer Cell 2026; github MAGIC-AI4Med/KEEP), HF Astaxanthin/KEEP
+          revision 28a25d95cc6ba27a7e6fab3f144e13dbafd8b21e, licence MIT. D = 768.
+          Text tower = PubMedBERT-architecture BertModel (12 layers, 768, vocab 30522) incl. its BERT pooler
+          (tanh(Linear(h_CLS))); KEEP has no separate text projection - its official modeling_keep.py
+          encode_text() is normalize(BertModel(**tokens).pooler_output). We return pooler_output before the
+          L2-normalisation. Tokenisation as in the README / repo zero-shot example: AutoTokenizer of the repo
+          (BertTokenizerFast, repo vocab.txt, lower-case), max_length=256, padding='max_length', truncation=True (input_ids, token_type_ids,
+          attention_mask all passed). Weights: text.* of model.safetensors, strict load. The official
+          modeling_keep.py (copied to ext/keep/, read in full: model definition only, but it imports timm - not in
+          scileo - and monkey-patches timm's LayerScale) is therefore NOT imported for extraction; the text tower is
+          rebuilt here with transformers' BertModel.
+quiltnet  QuiltNet-B-32 (Ikezogwo et al., NeurIPS 2023 D&B), HF wisdomik/QuiltNet-B-32
+          revision 8ce77289ce35a90b2f1db1137dfa4bc2df175e33, licence MIT. D = 512.
+          open_clip CLIP text tower (12 layers, width 512, 8 heads, causal mask, 77 ctx) re-implemented with
+          transformers' CLIPTextModelWithProjection and the open_clip state dict (open_clip_pytorch_model.bin)
+          remapped key-by-key (in_proj split into q/k/v, text_projection transposed), strict load.
+          Activation = QuickGELU. QuiltNet was fine-tuned from OpenAI CLIP ViT-B/32 with OpenCLIP (paper sec. 4 / app.
+          C.2), and OpenAI CLIP uses QuickGELU, but the repo's open_clip_config.json has no quick_gelu flag (and its HF
+          config.json says hidden_act 'gelu'), so a plain open_clip 'hf-hub:' load silently runs GELU. On image<->caption
+          retrieval with the real open_clip model (both towers), QuickGELU beats GELU on every metric: TCGA 256 i2t R@1
+          .082 vs .051, t2i .094 vs .066; ORION 512 i2t .029 vs .012, t2i .031 vs .021. So the weights were trained
+          with QuickGELU and the released config is wrong; we use QuickGELU (= open_clip
+          create_model('hf-hub:wisdomik/QuiltNet-B-32', force_quick_gelu=True)). The literal released-config (GELU)
+          variant is available as name 'quiltnet-gelu' (mean cosine to the QuickGELU embedding only ~0.78).
+          Tokenisation = open_clip's default SimpleTokenizer, reproduced as: whitespace_clean(basic_clean(t)).lower()
+          (ftfy.fix_text 6.3.1 from ext/quiltnet/pylib (pure python, not installed), html.unescape x2), OpenAI CLIP BPE
+          (CLIPTokenizerFast of openai/clip-vit-base-patch32), [SOT] + first 75 BPE + [EOT], zero padding to 77, no
+          padding mask. Pooling = hidden state at the EOT token (argmax of ids) after ln_final, @ text_projection
+          = open_clip encode_text(normalize=False).
+
+Verification (`verify`, run 2026-10-24; references loaded from unpacked wheels in ext/*/pylib, never pip-installed):
+  keep     official KEEPModel (ext/keep/modeling_keep.py, timm 1.0.15 from ext/keep/pylib, from_pretrained: all keys
+           matched): its encode_text == normalize(ours) BIT-EXACT (max|diff| 0) on 67 texts (64 real captions, an empty
+           and an over-length one); README AutoTokenizer == slow BertTokenizer inputs on 10405/10405 captions; README
+           example.tif zero-shot: breast invasive carcinoma 0.79 / lung adeno 0.21 / normal 0.001; 256-tile TCGA
+           image<->synthetic caption retrieval with the official image tower i2t R@1 0.094, t2i 0.063 (chance 0.004).
+  quiltnet open_clip 2.32.0 create_model('hf-hub:...', force_quick_gelu=True).encode_text vs ours (fp32, CPU):
+           max|diff| ~1e-5 (|x| up to ~8), min cosine 0.9999998 (float-op order only); same for the GELU variant vs the
+           default open_clip load; open_clip token ids == ours on 10405/10405 captions; remapped weights == the repo's
+           own HF-format pytorch_model.bin text weights (197/197 tensors, max|diff| 0); repo ADI-TCGA example tile
+           zero-shot over 9 CRC tissue classes -> adipose (top-1); retrieval numbers above.
+"""
+import html
+import json
+import os
+import re
+import sys
+import time
+
+import numpy as np
+import pandas as pd
+import torch
+import torch.nn as nn
+from huggingface_hub import hf_hub_download
+
+from common import CACHE, ROOT
+
+dev = "cuda"
+SETS = {"alltxt": "alltxt_meta.csv", "orion_gentext": "orion_gentext_meta.csv", "tcga_notext": "tcga_notext_meta.csv",
+        "dlpfc_gentext": "dlpfc_gentext_meta.csv"}
+KEEP_REPO, KEEP_REV = "Astaxanthin/KEEP", "28a25d95cc6ba27a7e6fab3f144e13dbafd8b21e"
+QN_REPO, QN_REV = "wisdomik/QuiltNet-B-32", "8ce77289ce35a90b2f1db1137dfa4bc2df175e33"
+CLIP_TOK = "openai/clip-vit-base-patch32"
+EXT = os.path.join(ROOT, "ext")
+
+
+class KEEPText(nn.Module):
+    """KEEP text tower: BertModel (+pooler) -> pooler_output (un-normalised)."""
+
+    def __init__(self):
+        super().__init__()
+        from safetensors.torch import load_file
+        from transformers import AutoTokenizer, BertConfig, BertModel
+        cfg = json.load(open(hf_hub_download(KEEP_REPO, "config.json", revision=KEEP_REV)))
+        self.text = BertModel(BertConfig(**cfg["text_config"]), add_pooling_layer=True)
+        sd = load_file(hf_hub_download(KEEP_REPO, "model.safetensors", revision=KEEP_REV))
+        sd = {k[5:]: v for k, v in sd.items() if k.startswith("text.")}
+        self.text.load_state_dict(sd, strict=True)
+        self.tok = AutoTokenizer.from_pretrained(KEEP_REPO, revision=KEEP_REV)  # as README (BertTokenizer(Fast), lower)
+
+    def tokenize(self, texts):
+        return self.tok(texts, max_length=256, padding="max_length", truncation=True, return_tensors="pt")
+
+    def forward(self, texts):
+        x = self.tokenize(texts).to(next(self.parameters()).device)
+        return self.text(**x).pooler_output
+
+
+def _oc_clean(t):  # = open_clip tokenizer 'lower' clean: whitespace_clean(basic_clean(t)).lower()
+    import ftfy  # ftfy 6.3.1 (pure python, unpacked wheel in ext/quiltnet/pylib; e.g. uncurls quotes as open_clip does)
+    t = html.unescape(html.unescape(ftfy.fix_text(t))).strip()
+    return re.sub(r"\s+", " ", t).strip().lower()
+
+
+class QuiltText(nn.Module):
+    """open_clip CLIP text tower of QuiltNet-B-32 on transformers' CLIPTextModelWithProjection."""
+    CTX = 77
+
+    def __init__(self, act="quick_gelu"):
+        super().__init__()
+        from transformers import CLIPTextConfig, CLIPTextModelWithProjection, CLIPTokenizerFast
+        sys.path.append(os.path.join(EXT, "quiltnet", "pylib"))  # for ftfy only (open_clip's text cleaner)
+        oc = json.load(open(hf_hub_download(QN_REPO, "open_clip_config.json", revision=QN_REV)))["model_cfg"]
+        tc = oc["text_cfg"]
+        cfg = CLIPTextConfig(vocab_size=tc["vocab_size"], hidden_size=tc["width"], intermediate_size=4 * tc["width"],
+                             num_hidden_layers=tc["layers"], num_attention_heads=tc["heads"],
+                             max_position_embeddings=tc["context_length"], projection_dim=oc["embed_dim"],
+                             hidden_act=act, layer_norm_eps=1e-5, eos_token_id=2)  # eos_token_id=2 -> argmax pooling
+        self.m = CLIPTextModelWithProjection(cfg)
+        sd = torch.load(hf_hub_download(QN_REPO, "open_clip_pytorch_model.bin", revision=QN_REV), map_location="cpu",
+                        weights_only=True)
+        self.m.load_state_dict(self.map_open_clip(sd, tc["layers"]), strict=True)
+        self.tok = CLIPTokenizerFast.from_pretrained(CLIP_TOK)
+        self.sot, self.eot = self.tok.bos_token_id, self.tok.eos_token_id  # 49406, 49407
+
+    @staticmethod
+    def map_open_clip(sd, L):
+        t = "text_model."
+        o = {t + "embeddings.token_embedding.weight": sd["token_embedding.weight"],
+             t + "embeddings.position_embedding.weight": sd["positional_embedding"],
+             t + "final_layer_norm.weight": sd["ln_final.weight"], t + "final_layer_norm.bias": sd["ln_final.bias"],
+             "text_projection.weight": sd["text_projection"].t().contiguous()}
+        for i in range(L):
+            a, b = f"transformer.resblocks.{i}.", f"{t}encoder.layers.{i}."
+            for w in ("weight", "bias"):
+                q, k, v = sd[a + "attn.in_proj_" + w].chunk(3, 0)
+                o.update({b + "self_attn.q_proj." + w: q, b + "self_attn.k_proj." + w: k, b + "self_attn.v_proj." + w: v,
+                          b + "self_attn.out_proj." + w: sd[a + "attn.out_proj." + w],
+                          b + "layer_norm1." + w: sd[a + "ln_1." + w], b + "layer_norm2." + w: sd[a + "ln_2." + w],
+                          b + "mlp.fc1." + w: sd[a + "mlp.c_fc." + w], b + "mlp.fc2." + w: sd[a + "mlp.c_proj." + w]})
+        return o
+
+    def tokenize(self, texts):
+        out = torch.zeros(len(texts), self.CTX, dtype=torch.long)
+        for i, ids in enumerate(self.tok([_oc_clean(t) for t in texts], add_special_tokens=False).input_ids):
+            ids = [self.sot] + ids[:self.CTX - 2] + [self.eot]
+            out[i, :len(ids)] = torch.tensor(ids)
+        return out
+
+    def forward(self, texts):
+        ids = self.tokenize(texts).to(next(self.parameters()).device)
+        return self.m(input_ids=ids).text_embeds  # EOT pooling (argmax id), ln_final, @ text_projection
+
+
+def encoder(name):
+    m = {"keep": KEEPText, "quiltnet": lambda: QuiltText("quick_gelu"),
+         "quiltnet-gelu": lambda: QuiltText("gelu")}[name]().to(dev).eval()
+    return m, 64
+
+
+def load_texts(meta):
+    return pd.read_csv(os.path.join(CACHE, meta)).caption.fillna("").astype(str).tolist()
+
+
+def extract(names):
+    for name in names:
+        t0 = time.time()
+        f, bs = encoder(name)
+        for st, meta in SETS.items():
+            out_f = os.path.join(CACHE, f"{st}_text_{name}.npy")
+            if os.path.exists(out_f):
+                continue
+            txt = load_texts(meta)
+            order = np.argsort([len(t) for t in txt])  # length-sorted batches, restored below
+            E = None
+            with torch.no_grad():
+                for i in range(0, len(txt), bs):
+                    idx = order[i:i + bs]
+                    z = f([txt[j] for j in idx]).float().cpu().numpy()
+                    if E is None:
+                        E = np.zeros((len(txt), z.shape[1]), np.float32)
+                    E[idx] = z
+            assert np.isfinite(E).all()
+            np.save(out_f, E[:, None].astype(np.float16))
+            print(f"[{name}] {st}: {E.shape}  {time.time() - t0:.0f}s", flush=True)
+        del f
+        torch.cuda.empty_cache()
+
+
+# ----------------------------------------------------------------------------------------------- verification
+def _retrieval(img_feats, txt_feats):
+    a = torch.nn.functional.normalize(img_feats, dim=-1)
+    b = torch.nn.functional.normalize(txt_feats, dim=-1)
+    S = a @ b.T
+    r = torch.arange(len(S))
+    return {"i2t_R@1": (S.argmax(1) == r).float().mean().item(), "t2i_R@1": (S.argmax(0) == r).float().mean().item(),
+            "i2t_R@5": (S.topk(5, 1).indices == r[:, None]).any(1).float().mean().item()}
+
+
+def verify(which=("keep", "quiltnet")):
+    from PIL import Image, PngImagePlugin
+    PngImagePlugin.MAX_TEXT_CHUNK = 1 << 30  # some PathGen PNGs carry huge iCCP chunks
+    torch.manual_seed(0)
+    texts = load_texts(SETS["alltxt"])
+    sample = [texts[i] for i in np.random.default_rng(0).choice(len(texts), 64, replace=False)]
+    sample += ["an H&E image of breast invasive carcinoma.", "", "a " + "very " * 120 + "long text"]
+    tn = pd.read_csv(os.path.join(CACHE, SETS["tcga_notext"]))
+    tn = tn[tn.caption.notna()].reset_index(drop=True)
+    imgs = [Image.open(p).convert("RGB") for p in tn.image_path]
+    print(f"retrieval set: {len(imgs)} TCGA tiles with synthetic captions (tcga_notext)")
+    if "keep" in which:
+        verify_keep(sample, tn, imgs)
+    if "quiltnet" in which:
+        verify_quiltnet(sample, tn, imgs)
+
+
+def verify_keep(sample, tn, imgs):
+    """KEEP: official modeling_keep.py from the local copy, timm from the unpacked wheel."""
+    from PIL import Image
+    sys.path[:0] = [os.path.join(EXT, "keep", "pylib"), os.path.join(EXT, "keep")]
+    from torchvision import transforms
+    from transformers import AutoTokenizer
+    import modeling_keep  # ext/keep/modeling_keep.py (read in full before use)
+    ref, info = modeling_keep.KEEPModel.from_pretrained(KEEP_REPO, revision=KEEP_REV, output_loading_info=True)
+    ref = ref.eval()
+    print(f"[keep] official from_pretrained loading info: { {k: v for k, v in info.items() if v} or 'all keys matched'}")
+    rtok = AutoTokenizer.from_pretrained(os.path.join(EXT, "keep"), use_fast=False)  # slow BertTokenizer as reference
+    ours = KEEPText().eval()
+    with torch.no_grad():
+        a = ref.encode_text(rtok(sample, max_length=256, padding="max_length", truncation=True, return_tensors="pt"))
+        b = torch.nn.functional.normalize(ours(sample), dim=-1)
+        print(f"[keep] official encode_text vs normalize(ours): max|diff| = {(a - b).abs().max().item():.3e}  "
+              f"bit-exact: {torch.equal(a, b)}")
+        n_same = n_tok = 0
+        for meta in SETS.values():
+            t = load_texts(meta)
+            for i in range(0, len(t), 256):
+                kw = dict(max_length=256, padding="max_length", truncation=True, return_tensors="pt")
+                x, y = rtok(t[i:i + 256], **kw), ours.tokenize(t[i:i + 256])
+                same = torch.stack([(x[k] == y[k]).all(1) for k in ("input_ids", "token_type_ids", "attention_mask")]).all(0)
+                n_tok, n_same = n_tok + len(same), n_same + int(same.sum())
+        print(f"[keep] slow BertTokenizer vs our (README AutoTokenizer) inputs identical on {n_same}/{n_tok} captions")
+        tf = transforms.Compose([transforms.Resize(224, interpolation=transforms.InterpolationMode.BICUBIC),
+                                 transforms.CenterCrop(224), transforms.ToTensor(),
+                                 transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225))])
+        ex = ["an H&E image of breast invasive carcinoma.", "an H&E image of normal tissue.",
+              "an H&E image of lung adenocarcinoma."]
+        img = tf(Image.open(os.path.join(EXT, "keep", "example.tif")).convert("RGB"))[None]
+        p = (ref.logit_scale.exp() * ref.encode_image(img) @ torch.nn.functional.normalize(ours(ex), dim=-1).T).softmax(-1)
+        print(f"[keep] README example.tif zero-shot probs {dict(zip(ex, p[0].round(decimals=3).tolist()))}")
+        ref = ref.to(dev)
+        I = torch.cat([ref.encode_image(torch.stack([tf(x) for x in imgs[i:i + 32]]).to(dev)).cpu()
+                       for i in range(0, len(imgs), 32)])
+        T = torch.cat([ours(list(tn.caption[i:i + 64])) for i in range(0, len(tn), 64)])
+        print(f"[keep] image<->caption retrieval (chance R@1 {1 / len(imgs):.3f}): {_retrieval(I, T)}")
+    del ref, ours
+    torch.cuda.empty_cache()
+
+
+def verify_quiltnet(sample, tn, imgs):
+    """QuiltNet: real open_clip 2.32.0 (+ftfy) from the unpacked wheels."""
+    from PIL import Image
+    sys.path[:0] = [os.path.join(EXT, "quiltnet", "pylib")]
+    import open_clip
+    from transformers import CLIPModel
+    otok = open_clip.get_tokenizer("hf-hub:" + QN_REPO)
+    ours = QuiltText().eval()
+    n_tok = n_same = 0
+    for meta in SETS.values():
+        t = load_texts(meta)
+        for i in range(0, len(t), 256):
+            same = (otok(t[i:i + 256]) == ours.tokenize(t[i:i + 256])).all(1)
+            n_tok, n_same = n_tok + len(same), n_same + int(same.sum())
+    print(f"[quiltnet] open_clip vs our token ids identical on {n_same}/{n_tok} captions of the 4 sets")
+    hf = CLIPModel.from_pretrained(QN_REPO, revision=QN_REV)
+    hsd, osd = hf.state_dict(), ours.m.state_dict()
+    dw = max((hsd[k] - osd[k]).abs().max().item() for k in osd if k in hsd)
+    print(f"[quiltnet] repo HF-format pytorch_model.bin: {sum(k in hsd for k in osd)}/{len(osd)} text tensors present, "
+          f"max|w diff| vs remapped open_clip weights = {dw:.3e}; its config hidden_act = {hf.config.text_config.hidden_act}")
+    del hf
+    cls = ["adipose", "background", "debris", "lymphocytes", "mucus", "smooth muscle", "normal colon mucosa",
+           "cancer-associated stroma", "colorectal adenocarcinoma epithelium"]
+    prompts = [f"a histopathology image of {c}" for c in cls]
+    for act, qg in (("quick_gelu", True), ("gelu", False)):
+        oc, _, pre = open_clip.create_model_and_transforms("hf-hub:" + QN_REPO, force_quick_gelu=qg)
+        oc = oc.eval()
+        mine = ours if qg else QuiltText("gelu").eval()
+        with torch.no_grad():
+            a, b = oc.encode_text(otok(sample)), mine(sample)
+            print(f"[quiltnet/{act}] open_clip encode_text vs ours: max|diff| = {(a - b).abs().max().item():.3e} "
+                  f"(max|a| {a.abs().max().item():.2f}), min cos = "
+                  f"{torch.nn.functional.cosine_similarity(a, b).min().item():.7f}")
+            x = pre(Image.open(os.path.join(EXT, "quiltnet", "ADI-TCGA-CFQTRNMG.tif")).convert("RGB"))[None]
+            p = (100 * torch.nn.functional.normalize(oc.encode_image(x), dim=-1)
+                 @ torch.nn.functional.normalize(mine(prompts), dim=-1).T).softmax(-1)[0]
+            print(f"[quiltnet/{act}] repo ADI-TCGA example tile zero-shot (true class adipose): top = {cls[p.argmax()]}"
+                  f" p={p.max().item():.3f}")
+            oc, mine = oc.to(dev), mine.to(dev)
+            I = torch.cat([oc.encode_image(torch.stack([pre(x) for x in imgs[i:i + 64]]).to(dev)).cpu()
+                           for i in range(0, len(imgs), 64)])
+            T = torch.cat([mine(list(tn.caption[i:i + 64])).cpu() for i in range(0, len(tn), 64)])
+            print(f"[quiltnet/{act}] retrieval (both towers {act}): {_retrieval(I, T)}")
+        del oc, mine
+        torch.cuda.empty_cache()
+
+
+if __name__ == "__main__":
+    if sys.argv[1:2] == ["verify"]:
+        verify(sys.argv[2:] or ("keep", "quiltnet"))
+    else:
+        extract(sys.argv[1:])

50_medvtok_feats.py

@@ -0,0 +1,91 @@
+"""MedVTok (general medical visual tokenizer, MERA-aligned to CONCH v1.5 for histopathology) global image features
+of every H&E set, as an additional / alternative image encoder next to the frozen UNI2 anchor.
+
+Run:  .venv/bin/python -I 50_medvtok_feats.py <set> [<set> ...]      sets: orion tcga hest dlpfc
+Model: ext/medvtok/repo (Masaaki-75/medvtok @a558c24, MIT; only models/medvtok.py + models/utils.py +
+models/transformer.py are imported) + ext/medvtok/weights/ckpt.pth (HF massaki75/medvtok, MIT), loaded here with
+torch.load(weights_only=True) and strict=True (the repo's build_model uses weights_only=False / strict=False).
+Encoder only (decode=False). Preprocessing = repo build_transform: RGB -> bicubic resize 256 -> [-1, 1].
+Feature = outputs["feats"]: mean-pooled encoder tokens -> LayerNorm -> Linear (1152-d, the aligned global feature).
+Views: orion / tcga 8 dihedral transforms (= he_uni2.npy); hest / dlpfc 1 view (= their UNI2 caches).
+Images: orion ../ORION tiles, tcga cache/allimg_meta.csv image_path, hest cache/hest/he_img.npy (49_),
+dlpfc cache/external/dlpfc_he_05um.npy (0.5 um/px, as for MIPHEI / captions).
+Output: cache/he_medvtok.npy (3187, 8, 1152), cache/allimg_he_medvtok.npy (6144, 8, 1152),
+cache/hest/he_medvtok.npy (9484, 1, 1152), cache/external/dlpfc_he_medvtok.npy (482, 1152); float16.
+"""
+import json
+import os
+import sys
+
+import numpy as np
+import pandas as pd
+import torch
+import torch.nn.functional as F
+from PIL import Image, PngImagePlugin
+
+PngImagePlugin.MAX_TEXT_CHUNK = 64 * 1024 * 1024  # some PathGen PNGs embed very large ICC profiles (= 04b_)
+
+ROOT = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, ROOT)
+from common import CACHE, ORION, dihedral, load_all_tiles  # noqa: E402
+
+REPO = os.path.join(ROOT, "ext", "medvtok", "repo")
+WTS = os.path.join(ROOT, "ext", "medvtok", "weights")
+sys.path.insert(0, REPO)
+from models.medvtok import MedVTok  # noqa: E402
+
+dev = "cuda"
+SIZE, BS = 256, 64
+
+cfg = json.load(open(os.path.join(WTS, "config.json")))
+assert cfg["architecture"] == "medvtok-clip-meanpool"
+model = MedVTok(**cfg["model_kwargs"])
+sd = torch.load(os.path.join(WTS, "ckpt.pth"), map_location="cpu", weights_only=True)
+print(model.load_state_dict(sd, strict=True))
+del sd, model.decoder
+model.eval().to(dev)
+
+
+@torch.no_grad()
+def encode(u8):
+    """u8: (B, H, W, 3) uint8 -> (B, 1152) float32"""
+    x = torch.from_numpy(np.ascontiguousarray(u8)).to(dev).permute(0, 3, 1, 2).float() / 255
+    x = F.interpolate(x, size=(SIZE, SIZE), mode="bicubic", align_corners=False, antialias=True).clamp(0, 1)
+    x = x * 2 - 1
+    with torch.autocast("cuda", dtype=torch.bfloat16):
+        out = model([im for im in x], sample_posterior=False, decode=False, return_batched_tensor=False)
+    return out["feats"].float().cpu().numpy()
+
+
+def run(load, n, views):
+    out = np.zeros((n, views, 1152), np.float16)
+    for s in range(0, n, BS):
+        u8 = load(s, min(n, s + BS))
+        for d in range(views):
+            x = u8 if views == 1 else dihedral(torch.from_numpy(u8).permute(0, 3, 1, 2), d).permute(0, 2, 3, 1).numpy()
+            out[s:s + len(u8), d] = encode(x)
+        if s % (BS * 20) == 0:
+            print(f"  {s}/{n}", flush=True)
+    assert np.isfinite(out.astype(np.float32)).all()
+    return out
+
+
+def from_paths(paths):
+    return lambda a, b: np.stack([np.asarray(Image.open(p).convert("RGB").resize((SIZE, SIZE), Image.BICUBIC))
+                                  for p in paths[a:b]])
+
+
+for st in sys.argv[1:]:
+    if st == "orion":
+        paths = np.array([os.path.join(ORION, p) for p in load_all_tiles().image_path])
+        np.save(os.path.join(CACHE, "he_medvtok.npy"), run(from_paths(paths), len(paths), 8))
+    elif st == "tcga":
+        paths = pd.read_csv(os.path.join(CACHE, "allimg_meta.csv")).image_path.values
+        np.save(os.path.join(CACHE, "allimg_he_medvtok.npy"), run(from_paths(paths), len(paths), 8))
+    elif st == "hest":
+        A = np.load(os.path.join(CACHE, "hest", "he_img.npy"), mmap_mode="r")
+        np.save(os.path.join(CACHE, "hest", "he_medvtok.npy"), run(lambda a, b: np.asarray(A[a:b]), len(A), 1))
+    elif st == "dlpfc":
+        A = np.load(os.path.join(CACHE, "external", "dlpfc_he_05um.npy"))
+        np.save(os.path.join(CACHE, "external", "dlpfc_he_medvtok.npy"), run(lambda a, b: A[a:b], len(A), 1)[:, 0])
+    print("saved", st, flush=True)

52_final_model.py

@@ -0,0 +1,360 @@
+"""Train and save the final four-modality model (EJ-ad+MV, the recommended configuration of 51_) for embed.py.
+
+    .venv/bin/python -I 52_final_model.py cv                 5-fold check of the stage-2 selection rule (3 seeds)
+    .venv/bin/python -I 52_final_model.py fold0   [--rule R] -> models/ej_ad_mv_fold0/ (fold-0 train / val; test held out)
+    .venv/bin/python -I 52_final_model.py release [--rule R] -> models/ej_ad_mv/       (train = fold-0 train + test, val = fold-0 val)
+    --out NAME writes models/NAME/ (cv: results/final_cv_*_NAME.csv) instead, e.g. to retrain without overwriting
+               the shipped checkpoints / results
+
+Same data, heads, losses, steps and early stopping as 51_medvtok_eval.py (UNI2 anchor; EJ-ad+MV):
+  stage 1  mIF head (VirTues -> MLP, 400 steps), RNA heads (hvg / Tahoe-x1 / SE-600M / AIDO.Cell -> linear, 2700),
+           text heads (KEEP / BiomedCLIP / Qwen3-Emb-0.6B -> linear, 1000), each InfoNCE against frozen UNI2 on real
+           pairs, early stopping on its val R@1 (deterministic: identical to 51_)
+  stage 2  heads frozen; residual adapter H&E = UNI2 + A([UNI2, c * MedVTok]) + temperatures, sum of the three pair
+           losses (RNA / text: mean over the encoder heads) + similarity distillation to UNI2 (lambda 1), AdamW lr 1e-4,
+           1000 steps (OneCycle)
+Stage-2 selection rules (the val curve is flat after ~100 steps, so 51_'s argmax of the noisy val geo-mean picks very
+different steps between otherwise identical runs; 'cv' quantifies it):
+  es        51_: early stopping on the geometric mean of the three val R@1, one seed
+  last      the adapter after the full 1000-step schedule, one seed (val unused)
+  ens-es    mean of the residuals of 3 early-stopped adapters (seeds): H&E = UNI2 + mean_s A_s(.)
+  ens-last  same with the 1000-step adapters
+Real pairs only (ORION H&E-mIF, HEST H&E-RNA, TCGA/PathGen H&E-text); virtual / synthetic data never trained on, used
+only in the emergent test pairs. 'fold0' evaluates on its held-out test (51_ protocol, g100, same test items) and
+writes test.csv + test_items.npz (for 53_); 'release' has no test split: expected performance = the 5-fold CV.
+Output: results/final_cv_{long,summary,curves}.csv (cv); models/<name>/{model.pt, meta.json, val_curve.csv[, test.csv]}
+"""
+import json
+import os
+import sys
+import time
+
+import numpy as np
+import pandas as pd
+import torch
+import torch.nn.functional as F
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+from common import CACHE, ROOT, SPLITS  # noqa: E402
+from embed import RNA_E, TXT_E, Bind4, mlp  # noqa: E402
+
+MODE = sys.argv[1]
+assert MODE in ("cv", "fold0", "release"), MODE
+RULE = sys.argv[sys.argv.index("--rule") + 1] if "--rule" in sys.argv else "ens-es"  # chosen by `cv` (RESULTS.md)
+assert RULE in ("es", "last", "ens-es", "ens-last"), RULE
+NSEED = 3
+dev = "cuda"
+H, RF = os.path.join(CACHE, "hest"), os.path.join(CACHE, "rnafm")
+G100, DRAWS, BS, LR, EVAL_EVERY = 100, 50, 256, 3e-4, 50
+f32 = lambda a: torch.from_numpy(np.asarray(a, dtype=np.float32))
+nz = lambda x: F.normalize(x, dim=-1)
+t0 = time.time()
+
+# ---------------------------------------------------------------- data (= 51_)
+IF_O = f32(np.load(os.path.join(CACHE, "if_virtues_m16.npy"))).to(dev)
+IF_V_T = f32(np.load(os.path.join(CACHE, "if_virtues_m16_virtual_miphei_tcga_text.npy"))[:, 0]).to(dev)
+U = {"o": np.load(os.path.join(CACHE, "he_uni2.npy")), "h": np.load(os.path.join(H, "he_uni2.npy"))[:, 0],
+     "t": np.load(os.path.join(CACHE, "allimg_he_uni2.npy"))}
+MV = {"o": np.load(os.path.join(CACHE, "he_medvtok.npy")), "h": np.load(os.path.join(H, "he_medvtok.npy"))[:, 0],
+      "t": np.load(os.path.join(CACHE, "allimg_he_medvtok.npy"))}
+U = {s: f32(x).to(dev) for s, x in U.items()}
+MV = {s: f32(x).to(dev) for s, x in MV.items()}
+MISS = pd.read_csv(os.path.join(H, "he_img_missing.csv")).idx.values
+T = pd.read_csv(os.path.join(CACHE, "alltxt_meta.csv"))
+T_IMG = torch.as_tensor(T.img_idx.values, device=dev)
+T = T[~pd.read_csv(os.path.join(CACHE, "alltxt_junk.csv")).junk.values]
+hvg = pd.read_csv(os.path.join(H, "hvg_genes.csv")).gene.values
+
+
+def virt_hvg(st):  # = 47_ / 51_ hvg_inputs (virtual RNA, test only)
+    g = pd.read_csv(os.path.join(CACHE, "stpath", f"{st}_genes.csv")).gene.values
+    c = np.expm1(np.load(os.path.join(CACHE, "stpath", f"{st}_pred.npy")).astype(np.float32)).clip(0)
+    c = np.log1p(c / c.sum(1, keepdims=True) * 1e4)
+    return c[:, pd.Series(range(len(g)), index=g).loc[hvg].values]
+
+
+RNA_IN = {e: ({"hest": f32(np.load(os.path.join(H, "hvg.npy"))).to(dev), "orion_v": f32(virt_hvg("orion")).to(dev),
+               "tcga_v": f32(virt_hvg("tcga_text")).to(dev)} if e == "hvg" else
+              {s: f32(np.load(os.path.join(RF, f"{e}_{s}.npy"))).to(dev) for s in ["hest", "orion_v", "tcga_v"]})
+          for e in RNA_E}
+TXT_IN = {e: {s: f32(np.load(os.path.join(CACHE, f"{s}_text_{e}.npy"))[:, 0]).to(dev)
+              for s in ["alltxt", "orion_gentext"]} for e in TXT_E}
+RAW = {"rna:hvg", "text:biomedclip"}  # encoders used without z-scoring (= 47_ / 51_)
+DIMS = {"mif": IF_O.shape[-1], **{f"rna:{e}": RNA_IN[e]["hest"].shape[1] for e in RNA_E},
+        **{f"text:{e}": TXT_IN[e]["alltxt"].shape[1] for e in TXT_E}}
+
+
+def one_text_per_image(t_ids, rng):
+    pick = T.loc[t_ids].groupby("img_idx").apply(lambda g: g.index[rng.integers(len(g))], include_groups=False)
+    return pick.index.values, pick.values
+
+
+def g100(a, b, seed=0):  # = 51_
+    from scipy.optimize import linear_sum_assignment
+    a, b = a.cpu(), b.cpu()
+    rng = np.random.default_rng(seed)
+    out = {m: [] for m in ["R@1", "R@5", "R@10", "MRR", "pair_acc"]}
+    for _ in range(DRAWS):
+        s_ = torch.from_numpy(rng.choice(len(a), min(G100, len(a)), replace=False))
+        sim = (a[s_] @ b[s_].T).numpy(); d = np.diag(sim)
+        for S in (sim, sim.T):
+            rank = (S > d[:, None]).sum(1)
+            for k_ in (1, 5, 10):
+                out[f"R@{k_}"].append((rank < k_).mean() / 2)
+            out["MRR"].append((1 / (rank + 1)).mean() / 2)
+        ri, ci = linear_sum_assignment(-sim)
+        out["pair_acc"].append((ci[np.argsort(ri)] == np.arange(len(s_))).mean())
+    return {m: float(np.sum(v) / DRAWS) for m, v in out.items()}
+
+
+def r1(a, b):
+    s_ = a @ b.T
+    return float((((s_ > s_.diag()[:, None]).sum(1) == 0).float().mean() + ((s_.T > s_.diag()[:, None]).sum(1) == 0).float().mean()) / 2)
+
+
+def clip_loss(lg, pos=None):
+    if pos is None:
+        t = torch.arange(len(lg), device=dev)
+        return (F.cross_entropy(lg, t) + F.cross_entropy(lg.T, t)) / 2
+    tgt = pos / pos.sum(1, keepdim=True)
+    return (torch.sum(-tgt * F.log_softmax(lg, 1), 1).mean() + torch.sum(-tgt * F.log_softmax(lg.T, 1), 1).mean()) / 2
+
+
+def run(k, release=False, test=True):
+    """stage 1 + stage 2 (NSEED seeds) on fold k -> (model, use(rule), test results per rule, val curves, info)"""
+    s = pd.read_csv(os.path.join(SPLITS, "patient_5fold", f"fold{k}.csv"))
+    hh = pd.read_csv(os.path.join(SPLITS, "final", f"hest_fold{k}.csv"))
+    sp = pd.read_csv(os.path.join(SPLITS, "final", f"tcga_fold{k}.csv")).set_index("img_idx").split
+    tsp = sp.loc[T.img_idx].values
+    o = {x: s.tile_id[s.split == x].values for x in ["train", "val", "test"]}
+    hs = {x: np.setdiff1d(hh.spot_idx[hh.split == x].values, MISS) for x in ["train", "val", "test"]}
+    ts = {x: T.index[tsp == x].values for x in ["train", "val", "test"]}
+    if release:  # everything real except the early-stopping val split
+        for d in (o, hs, ts):
+            d["train"] = np.sort(np.r_[d["train"], d.pop("test")])
+    sizes = {n: {x: len(v) for x, v in d.items()} for n, d in [("orion", o), ("hest", hs), ("tcga_text", ts)]}
+    print(f"fold {k}{' (release)' if release else ''}: {sizes}", flush=True)
+
+    # ------------------------------------------------ model + normalisation statistics (train only)
+    model = Bind4(DIMS, "cat").to(dev)
+    model.set_norm("mif", IF_O[o["train"]].mean((0, 1)), IF_O[o["train"]].std((0, 1)) + 1e-3)
+    trt = torch.as_tensor(ts["train"], device=dev)
+    for e in RNA_E:
+        if f"rna:{e}" not in RAW:
+            x = RNA_IN[e]["hest"][hs["train"]]
+            model.set_norm(f"rna:{e}", x.mean(0), x.std(0) + 1e-6)
+    for e in TXT_E:
+        if f"text:{e}" not in RAW:
+            x = TXT_IN[e]["alltxt"][trt]
+            model.set_norm(f"text:{e}", x.mean(0), x.std(0) + 1e-6)
+    model.c_mv.fill_(float(U["o"][o["train"]].norm(dim=-1).mean() / MV["o"][o["train"]].norm(dim=-1).mean()))
+    Xr = {e: {st: model.norm(f"rna:{e}", v) for st, v in RNA_IN[e].items()} for e in RNA_E}
+    Xt = {e: {st: model.norm(f"text:{e}", v) for st, v in TXT_IN[e].items()} for e in TXT_E}
+    Xm = model.norm("mif", IF_O)
+    o_tr, h_tr = torch.as_tensor(o["train"], device=dev), torch.as_tensor(hs["train"], device=dev)
+    vi, vt = (torch.as_tensor(x, device=dev) for x in one_text_per_image(ts["val"], np.random.default_rng(0)))
+
+    def anc(key, idx, views=None):
+        x, m = U[key], MV[key]
+        if x.dim() == 2:
+            return x[idx], m[idx]
+        v = views if views is not None else 0
+        return x[idx, v], m[idx, v]
+
+    def batch(task, rng):
+        if task == "mif":
+            b = o_tr[torch.as_tensor(rng.choice(len(o_tr), BS, replace=False), device=dev)]
+            d = torch.randint(0, 8, (BS,), device=dev)
+            return anc("o", b, d), {"mif": Xm[b, d]}, None
+        if task == "rna":
+            b = h_tr[torch.as_tensor(rng.choice(len(h_tr), BS, replace=False), device=dev)]
+            return anc("h", b), {e: Xr[e]["hest"][b] for e in RNA_E}, None
+        b = trt[torch.as_tensor(rng.choice(len(trt), BS, replace=False), device=dev)]
+        im = T_IMG[b]
+        return (anc("t", im, torch.randint(0, 8, (BS,), device=dev)), {e: Xt[e]["alltxt"][b] for e in TXT_E},
+                (im[:, None] == im[None, :]).float())
+
+    VAL = {"mif": (anc("o", o["val"]), {"mif": Xm[o["val"], 0]}),
+           "rna": (anc("h", hs["val"]), {e: Xr[e]["hest"][hs["val"]] for e in RNA_E}),
+           "text": (anc("t", vi), {e: Xt[e]["alltxt"][vt] for e in TXT_E})}
+
+    # ------------------------------------------------ stage 1 (= 51_ train_head, UNI2 anchor)
+    def train_head(key, task, enc, steps):
+        torch.manual_seed(k); rng = np.random.default_rng(k)
+        h = model.h(key)  # initialised exactly like 51_: a fresh module created right after the same seed
+        h.load_state_dict((mlp(h[0].in_features) if key == "mif" else torch.nn.Linear(h.in_features, h.out_features)).state_dict())
+        logit = model.logit[model._k(key)]
+        opt = torch.optim.AdamW(list(h.parameters()) + [logit], lr=LR, weight_decay=0.05)
+        sch = torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=LR, total_steps=steps, pct_start=0.05)
+        za_va, x_va = nz(VAL[task][0][0]), VAL[task][1][enc]
+        best, state = -1, None
+        for step in range(1, steps + 1):
+            h.train()
+            (u, _), X, pos = batch(task, rng)
+            loss = clip_loss(logit.exp().clamp(max=100) * nz(u) @ nz(h(X[enc])).T, pos)
+            opt.zero_grad(set_to_none=True); loss.backward(); opt.step(); sch.step()
+            if step % EVAL_EVERY == 0 or step == steps:
+                h.eval()
+                with torch.no_grad():
+                    v = r1(za_va, nz(h(x_va)))
+                if v > best:
+                    best, state = v, ({k_: x.detach().clone() for k_, x in h.state_dict().items()}, logit.detach().clone())
+        h.load_state_dict(state[0]); h.eval()
+        with torch.no_grad():
+            logit.copy_(state[1])
+        return best
+
+    stage1 = {"mif": train_head("mif", "mif", "mif", 400)}
+    for e in RNA_E:
+        stage1[f"rna:{e}"] = train_head(f"rna:{e}", "rna", e, 2700)
+    for e in TXT_E:
+        stage1[f"text:{e}"] = train_head(f"text:{e}", "text", e, 1000)
+    for p in model.heads.parameters():
+        p.requires_grad_(False)
+    model.heads.eval()
+    logit0 = {k_: v.detach().clone() for k_, v in model.logit.items()}
+    print(f"[{time.time() - t0:.0f}s] fold {k} stage 1 val R@1: {json.dumps({a: round(b, 3) for a, b in stage1.items()})}",
+          flush=True)
+
+    # ------------------------------------------------ stage 2 (= 51_ joint, mode 'ad', adapter 'cat'), NSEED seeds
+    def fuse_n(mod, encs, X):  # inputs already normalised
+        return nz(sum(nz(model.h(f"{mod}:{e}")(X[e])) for e in encs))
+
+    def val_score():
+        model.eval()
+        with torch.no_grad():
+            v = {"mif": r1(nz(model.he(*VAL["mif"][0])), nz(model.h("mif")(VAL["mif"][1]["mif"]))),
+                 "rna": r1(nz(model.he(*VAL["rna"][0])), fuse_n("rna", RNA_E, VAL["rna"][1])),
+                 "text": r1(nz(model.he(*VAL["text"][0])), fuse_n("text", TXT_E, VAL["text"][1]))}
+        return float(np.exp(np.mean(np.log([max(x, 1e-4) for x in v.values()])))), v
+
+    def snap():
+        return ({k_: x.detach().clone() for k_, x in model.ah[0].state_dict().items()},
+                {k_: x.detach().clone() for k_, x in model.logit.items()})
+
+    def stage2(seed):
+        torch.manual_seed(seed); rng = np.random.default_rng(seed)
+        model.ah = torch.nn.ModuleList([model.new_adapter()]).to(dev)
+        with torch.no_grad():
+            for k_, v in model.logit.items():
+                v.copy_(logit0[k_])
+        opt = torch.optim.AdamW(list(model.ah.parameters()) + list(model.logit.parameters()), lr=1e-4, weight_decay=0.05)
+        sch = torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=1e-4, total_steps=1000, pct_start=0.05)
+        best, es, curve = -1, None, []
+        for step in range(1, 1001):
+            model.ah.train()
+            loss = 0
+            for task in ("mif", "rna", "text"):
+                (u, mv), X, pos = batch(task, rng)
+                za = nz(model.he(u, mv))
+                if task == "mif":
+                    loss = loss + clip_loss(model.lg("mif") * za @ nz(model.h("mif")(X["mif"])).T, pos)
+                else:
+                    encs = RNA_E if task == "rna" else TXT_E
+                    loss = loss + sum(clip_loss(model.lg(f"{task}:{e}") * za @ nz(model.h(f"{task}:{e}")(X[e])).T, pos)
+                                      for e in encs) / len(encs)
+                un = nz(u)
+                loss = loss + F.mse_loss(za @ za.T, un @ un.T)
+            opt.zero_grad(set_to_none=True); loss.backward(); opt.step(); sch.step()
+            if step % EVAL_EVERY == 0 or step == 1000:
+                sc, v = val_score()
+                curve.append({"seed": seed, "step": step, "geo": sc, **v})
+                if sc > best:
+                    best, es = sc, (step, *snap())
+        return es, (1000, *snap()), curve
+
+    seeds = [k] + [1000 * i + k for i in range(1, NSEED)]  # the first = 51_'s seed
+    runs = [stage2(sd) for sd in seeds]
+    curve = pd.DataFrame([c for r in runs for c in r[2]])
+    sel = {"es": [runs[0][0]], "last": [runs[0][1]], "ens-es": [r[0] for r in runs], "ens-last": [r[1] for r in runs],
+           **{f"es@seed{i}": [r[0]] for i, r in enumerate(runs)}, **{f"last@seed{i}": [r[1]] for i, r in enumerate(runs)}}
+    print(f"[{time.time() - t0:.0f}s] fold {k} stage 2: early-stopping step per seed {[r[0][0] for r in runs]}", flush=True)
+
+    def use(rule):
+        """set the model's adapter(s) to a selection rule (temperatures: those of the first member; unused at inference)"""
+        st = sel[rule]
+        model.ah = torch.nn.ModuleList([model.new_adapter() for _ in st]).to(dev)
+        for a, (_, sd_, _) in zip(model.ah, st):
+            a.load_state_dict(sd_)
+        with torch.no_grad():
+            for k_, v in model.logit.items():
+                v.copy_(st[0][2][k_])
+        model.eval()
+
+    res = {}
+    if test:
+        ii, tt = (torch.as_tensor(x, device=dev) for x in one_text_per_image(ts["test"], np.random.default_rng(k)))
+        for rule in ["stage1"] + list(sel):
+            if rule == "stage1":
+                model.ah = torch.nn.ModuleList([model.new_adapter()]).to(dev)  # zero output: H&E = UNI2 (= E of 51_)
+            else:
+                use(rule)
+            with torch.no_grad():
+                R = lambda st, idx: fuse_n("rna", RNA_E, {e: Xr[e][st][idx] for e in RNA_E})
+                Tx = lambda st, idx: fuse_n("text", TXT_E, {e: Xt[e][st][idx] for e in TXT_E})
+                zm = nz(model.h("mif")(Xm[o["test"], 0]))
+                zt = Tx("alltxt", tt)
+                res[rule] = {"H&E<->mIF (ORION)": g100(nz(model.he(*anc("o", o["test"]))), zm),
+                             "H&E<->RNA (HEST)": g100(nz(model.he(*anc("h", hs["test"]))), R("hest", hs["test"])),
+                             "H&E<->text (TCGA)": g100(nz(model.he(*anc("t", ii))), zt),
+                             "mIF<->RNA(virtual) (ORION)": g100(zm, R("orion_v", o["test"])),
+                             "mIF<->text(synthetic) (ORION)": g100(zm, Tx("orion_gentext", o["test"])),
+                             "text<->RNA(virtual) (TCGA)": g100(zt, R("tcga_v", ii)),
+                             "text<->mIF(MIPHEI) (TCGA)": g100(zt, nz(model.h("mif")(model.norm("mif", IF_V_T[ii]))))}
+        res["_items"] = dict(orion=o["test"], hest=hs["test"], tcga_img=ii.cpu().numpy(), tcga_txt=tt.cpu().numpy())
+    info = {"sizes": sizes, "stage1_val_R@1": stage1, "es_steps": [r[0][0] for r in runs], "seeds": seeds}
+    return model, use, res, curve, info
+
+
+if MODE == "cv":
+    TAG = f"_{sys.argv[sys.argv.index('--out') + 1]}" if "--out" in sys.argv else ""  # results/final_cv_<x><TAG>.csv
+    rows = []
+    for k in range(5):
+        _, _, res, curve, info = run(k)
+        res.pop("_items")
+        for rule, rr in res.items():
+            for pair, r in rr.items():
+                rows.append({"rule": rule, "fold": k, "pair": pair, **r})
+        D = pd.DataFrame(rows)
+        D.to_csv(os.path.join(ROOT, "results", f"final_cv_long{TAG}.csv"), index=False)
+        curve.assign(fold=k).to_csv(os.path.join(ROOT, "results", f"final_cv_curves{TAG}.csv"), mode="w" if k == 0 else "a",
+                                    header=k == 0, index=False)
+        print(D[D.fold == k].pivot_table(index="rule", columns="pair", values="R@1", sort=False).round(3).to_string(), flush=True)
+    D = pd.DataFrame(rows)
+    S = D.groupby(["rule", "pair"], sort=False)[["R@1", "pair_acc"]].agg(["mean", "std"]).round(4)
+    S.to_csv(os.path.join(ROOT, "results", f"final_cv_summary{TAG}.csv"))
+    pd.set_option("display.width", 250)
+    print(D.pivot_table(index="rule", columns="pair", values="R@1", sort=False).round(3).to_string())
+    for r in ["es", "last"]:  # seed-to-seed spread of the single-seed rules (std over seeds, mean over folds)
+        sub = D[D.rule.str.startswith(f"{r}@seed")]
+        print(r, "seed std of R@1 (mean over folds):",
+              sub.groupby(["pair", "fold"])["R@1"].std().groupby("pair").mean().round(4).to_dict())
+    sys.exit()
+
+# ---------------------------------------------------------------- fold0 / release: build, save
+release = MODE == "release"
+OUT = os.path.join(ROOT, "models", sys.argv[sys.argv.index("--out") + 1] if "--out" in sys.argv else
+                   "ej_ad_mv" if release else "ej_ad_mv_fold0")
+os.makedirs(OUT, exist_ok=True)
+model, use, res, curve, info = run(0, release=release, test=not release)
+use(RULE)
+curve.round(4).to_csv(os.path.join(OUT, "val_curve.csv"), index=False)
+torch.save({k_: v.cpu() for k_, v in model.state_dict().items()}, os.path.join(OUT, "model.pt"))
+meta = {"config": "EJ-ad+MV", "stage2_rule": RULE, "n_adapters": len(model.ah), "mode": MODE,
+        "created": time.strftime("%Y-%m-%d %H:%M"), "dims": DIMS, "adapter": "cat", "raw_encoders": sorted(RAW),
+        "hvg_genes": hvg.tolist(), "c_mv": float(model.c_mv),
+        "split": "fold0 of splits/patient_5fold (ORION), splits/final/hest_fold0, tcga_fold0"
+                 + ("; train = train + test, val = val (early stopping)" if release else "; test held out"),
+        **info, "anchor": "UNI2-h CLS (1536-d), frozen", "image_extra": "MedVTok feats (1152-d), frozen",
+        "mif_markers": "embed.MIF_MARKERS", "rna_encoders": RNA_E, "text_encoders": TXT_E}
+json.dump(meta, open(os.path.join(OUT, "meta.json"), "w"), indent=1)
+print(f"saved {OUT} (rule {RULE}, {len(model.ah)} adapter(s))", flush=True)
+if not release:
+    items = res.pop("_items")
+    np.savez(os.path.join(OUT, "test_items.npz"), **items)
+    ref = pd.read_csv(os.path.join(ROOT, "results", "medvtok_long.csv"))
+    ref = ref[(ref.config == "EJ-ad+MV") & (ref.fold == 0)].set_index("pair")
+    D = pd.DataFrame([{"pair": p, "R@1": r["R@1"], "pair_acc": r["pair_acc"], "R@1_51_fold0": ref.loc[p, "R@1"]}
+                      for p, r in res[RULE].items()])
+    D.to_csv(os.path.join(OUT, "test.csv"), index=False)
+    print(D.round(3).to_string(), flush=True)

53_embed_check.py

@@ -0,0 +1,151 @@
+"""Check embed.py from RAW inputs (pixels, strings, counts) against the cached training features and the fold-0 test.
+
+    .venv/bin/python -I 53_embed_check.py [n]        (n = items per raw check, default 300)
+
+1. feature reproduction: embed.py raw-input encoders vs the cached features the model was trained / tested on
+   (per-item cosine of the raw encoder features), for every encoder:
+     H&E  UNI2 + MedVTok   ORION tiles (PNG), TCGA/PathGen patches (PNG), HEST spot crops (cache/hest/he_img.npy)
+     text KEEP / BiomedCLIP / Qwen3   TCGA real captions (alltxt)
+     RNA  hvg / Tahoe-x1 / SE-600M / AIDO.Cell   HEST spots (rna_sets.load('hest') rows)
+     mIF  VirTues (all 3187 ORION tiles: standardisation statistics are per slide / per dataset)
+2. end-to-end: models/ej_ad_mv_fold0 on held-out fold-0 test items, embeddings from raw inputs vs from cached
+   features (same items, g100 R@1)
+Output results/embed_check.csv
+"""
+import os
+import sys
+import time
+
+import numpy as np
+import pandas as pd
+import torch
+
+ROOT = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, ROOT)
+from common import CACHE, ORION, load_all_tiles  # noqa: E402
+from embed import MIF_MARKERS, RNA_E, TXT_E, Embedder  # noqa: E402
+
+N = int(sys.argv[1]) if len(sys.argv) > 1 else 300
+MD = os.path.join(ROOT, "models", "ej_ad_mv_fold0")
+E = Embedder(MD)
+TI = np.load(os.path.join(MD, "test_items.npz"))
+rng = np.random.default_rng(0)
+rows, t0 = [], time.time()
+f32 = lambda a: np.asarray(a, np.float32)
+
+
+def cos(a, b):
+    a, b = torch.as_tensor(f32(a)), torch.as_tensor(f32(b))
+    return torch.nn.functional.cosine_similarity(a, b, dim=-1).numpy()
+
+
+def report(mod, enc, src, c):
+    rows.append({"check": "features", "modality": mod, "encoder": enc, "source": src, "n": len(c),
+                 "cos_median": float(np.median(c)), "cos_min": float(c.min())})
+    print(f"[{time.time() - t0:.0f}s] {mod:5s} {enc:11s} {src:14s} n={len(c)} cos median {np.median(c):.4f} min {c.min():.4f}",
+          flush=True)
+
+
+def r1_g100(a, b, seed=0):
+    a, b = torch.nn.functional.normalize(a, dim=-1), torch.nn.functional.normalize(b, dim=-1)
+    g = np.random.default_rng(seed)
+    out = []
+    for _ in range(50):
+        s = torch.from_numpy(g.choice(len(a), min(100, len(a)), replace=False))
+        S = a[s] @ b[s].T; d = S.diag()
+        out.append((((S > d[:, None]).sum(1) == 0).float().mean() + ((S.T > d[:, None]).sum(1) == 0).float().mean()).item() / 2)
+    return float(np.mean(out))
+
+
+# ------------------------------------------------------------------ H&E
+tiles = load_all_tiles()
+ot = rng.choice(TI["orion"], min(N, len(TI["orion"])), replace=False)
+U, M = E.he_feats([os.path.join(ORION, p) for p in tiles.image_path.values[ot]])
+report("H&E", "uni2", "ORION", cos(U, np.load(os.path.join(CACHE, "he_uni2.npy"), mmap_mode="r")[ot, 0]))
+report("H&E", "medvtok", "ORION", cos(M, np.load(os.path.join(CACHE, "he_medvtok.npy"), mmap_mode="r")[ot, 0]))
+he_raw = {"orion": (ot, U, M)}
+I = pd.read_csv(os.path.join(CACHE, "allimg_meta.csv"))
+tt_img = rng.choice(len(TI["tcga_img"]), min(N, len(TI["tcga_img"])), replace=False)
+ti = TI["tcga_img"][tt_img]
+U, M = E.he_feats(I.image_path.values[ti])
+report("H&E", "uni2", "TCGA/PathGen", cos(U, np.load(os.path.join(CACHE, "allimg_he_uni2.npy"), mmap_mode="r")[ti, 0]))
+report("H&E", "medvtok", "TCGA/PathGen", cos(M, np.load(os.path.join(CACHE, "allimg_he_medvtok.npy"), mmap_mode="r")[ti, 0]))
+he_raw["tcga"] = (tt_img, U, M)
+hsp = rng.choice(TI["hest"], min(N, len(TI["hest"])), replace=False)
+U, M = E.he_feats(np.load(os.path.join(CACHE, "hest", "he_img.npy"), mmap_mode="r")[hsp])
+report("H&E", "uni2", "HEST", cos(U, np.load(os.path.join(CACHE, "hest", "he_uni2.npy"), mmap_mode="r")[hsp, 0]))
+report("H&E", "medvtok", "HEST", cos(M, np.load(os.path.join(CACHE, "hest", "he_medvtok.npy"), mmap_mode="r")[hsp, 0]))
+he_raw["hest"] = (hsp, U, M)
+E.release()
+
+# ------------------------------------------------------------------ text
+T = pd.read_csv(os.path.join(CACHE, "alltxt_meta.csv"))
+txt_ids = TI["tcga_txt"][tt_img]  # the captions paired with the TCGA test images above
+Ft = E.text_feats(T.caption.fillna("").astype(str).values[txt_ids].tolist())
+for e in TXT_E:
+    report("text", e, "TCGA captions", cos(Ft[e], np.load(os.path.join(CACHE, f"alltxt_text_{e}.npy"), mmap_mode="r")[txt_ids, 0]))
+E.release()
+
+# ------------------------------------------------------------------ RNA
+from rna_sets import load  # noqa: E402
+
+Xh, genes = load("hest")
+Fr = E.rna_feats(Xh[hsp], genes)
+for e in RNA_E:
+    ref = (np.load(os.path.join(CACHE, "hest", "hvg.npy"), mmap_mode="r") if e == "hvg" else
+           np.load(os.path.join(CACHE, "rnafm", f"{e}_hest.npy"), mmap_mode="r"))[hsp]
+    report("RNA", e, "HEST", cos(Fr[e], ref))
+E.release()
+
+
+# ------------------------------------------------------------------ mIF (whole ORION set: dataset-level standardisation)
+class LazyIF:
+    """(N, 16, 333, 333) ORION mIF tiles read on demand (uint8 tifs, 17 channels -> MIF_MARKERS)"""
+
+    def __init__(self, df):
+        import tifffile
+        self.paths, self.rd = [os.path.join(ORION, p) for p in df.target_path], tifffile.imread
+        V = E._get("virtues")[0]
+        names = [m for m, _ in V.MARKERS]
+        self.ch = [names.index(m) for m in MIF_MARKERS]
+
+    def __len__(self):
+        return len(self.paths)
+
+    def __getitem__(self, s):
+        return np.stack([self.rd(p)[..., self.ch].transpose(2, 0, 1) for p in self.paths[s]])
+
+
+Fm = E.mif_feats(LazyIF(tiles), tiles.in_slide_name.values)
+report("mIF", "virtues", "ORION (all)", cos(Fm, np.load(os.path.join(CACHE, "if_virtues_m16.npy"), mmap_mode="r")[:, 0]))
+E.release()
+
+# ------------------------------------------------------------------ end-to-end on held-out fold-0 test items
+UO, MO = np.load(os.path.join(CACHE, "he_uni2.npy"), mmap_mode="r"), np.load(os.path.join(CACHE, "he_medvtok.npy"), mmap_mode="r")
+IFc = np.load(os.path.join(CACHE, "if_virtues_m16.npy"), mmap_mode="r")
+ot, U, M = he_raw["orion"]
+e2e = {"H&E<->mIF (ORION)": ((E.he_from_feats(U, M), E.mif_from_feats(Fm[ot])),
+                             (E.he_from_feats(UO[ot, 0], MO[ot, 0]), E.mif_from_feats(IFc[ot, 0])))}
+hsp, U, M = he_raw["hest"]
+ref_r = {e: (np.load(os.path.join(CACHE, "hest", "hvg.npy"), mmap_mode="r") if e == "hvg" else
+             np.load(os.path.join(CACHE, "rnafm", f"{e}_hest.npy"), mmap_mode="r"))[hsp] for e in RNA_E}
+e2e["H&E<->RNA (HEST)"] = ((E.he_from_feats(U, M), E.rna_from_feats(Fr)),
+                           (E.he_from_feats(np.load(os.path.join(CACHE, "hest", "he_uni2.npy"), mmap_mode="r")[hsp, 0],
+                                            np.load(os.path.join(CACHE, "hest", "he_medvtok.npy"), mmap_mode="r")[hsp, 0]),
+                            E.rna_from_feats(ref_r)))
+_, U, M = he_raw["tcga"]
+ref_t = {e: np.load(os.path.join(CACHE, f"alltxt_text_{e}.npy"), mmap_mode="r")[txt_ids, 0] for e in TXT_E}
+e2e["H&E<->text (TCGA)"] = ((E.he_from_feats(U, M), E.text_from_feats(Ft)),
+                            (E.he_from_feats(np.load(os.path.join(CACHE, "allimg_he_uni2.npy"), mmap_mode="r")[ti, 0],
+                                             np.load(os.path.join(CACHE, "allimg_he_medvtok.npy"), mmap_mode="r")[ti, 0]),
+                             E.text_from_feats(ref_t)))
+for pair, v in e2e.items():
+    (a_raw, b_raw), (a_c, b_c) = v
+    rows.append({"check": "end-to-end", "modality": pair, "n": len(a_raw), "R@1_raw": r1_g100(a_raw, b_raw),
+                 "R@1_cached": r1_g100(a_c, b_c), "emb_cos_median_a": float(np.median(cos(a_raw, a_c))),
+                 "emb_cos_median_b": float(np.median(cos(b_raw, b_c)))})
+    print(f"{pair:20s} n={len(a_raw)} g100 R@1 raw-input {rows[-1]['R@1_raw']:.3f} | cached features "
+          f"{rows[-1]['R@1_cached']:.3f}; embedding cos median {rows[-1]['emb_cos_median_a']:.4f} / "
+          f"{rows[-1]['emb_cos_median_b']:.4f}", flush=True)
+pd.DataFrame(rows).to_csv(os.path.join(ROOT, "results", "embed_check.csv"), index=False)
+print(f"done [{time.time() - t0:.0f}s]")

54_make_examples.py

@@ -0,0 +1,77 @@
+"""Build examples/: a small raw-input sample (fold-0 TEST items) + the cached training features of the same items, so
+that example_check.py can test embed.py end to end without the full datasets.
+
+    .venv/bin/python -I 54_make_examples.py [n=64]
+
+examples/
+  orion_he/*.png, orion_mif.npz (tiles uint8 (n, 16, 333, 333) in embed.MIF_MARKERS order, slide)   ORION H&E + mIF
+  hest_he.npy (n, 224, 224, 3) uint8, hest_counts.npz (raw counts, spots x genes), hest_genes.csv          HEST H&E + RNA
+  tcga_he/*.png, tcga_captions.csv                                                                    TCGA/PathGen H&E + text
+  ref_feats.npz   cached encoder features (view 0) of the same items: the inputs the model was trained / tested on
+  items.csv       which items (global indices into cache/)
+"""
+import os
+import shutil
+import sys
+
+import numpy as np
+import pandas as pd
+import scipy.sparse as sp
+
+ROOT = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, ROOT)
+from common import CACHE, ORION, load_all_tiles  # noqa: E402
+from embed import MIF_MARKERS, RNA_E, TXT_E, _load_script  # noqa: E402
+from rna_sets import load  # noqa: E402
+
+N = int(sys.argv[1]) if len(sys.argv) > 1 else 64
+OUT = os.path.join(ROOT, "examples")
+shutil.rmtree(OUT, ignore_errors=True)
+os.makedirs(os.path.join(OUT, "orion_he")); os.makedirs(os.path.join(OUT, "tcga_he"))
+TI = np.load(os.path.join(ROOT, "models", "ej_ad_mv_fold0", "test_items.npz"))
+rng = np.random.default_rng(0)
+ld = lambda *p: np.load(os.path.join(*p), mmap_mode="r")
+ref, items = {}, []
+
+# ORION: H&E PNG + 16-marker mIF
+import tifffile  # noqa: E402
+
+tiles = load_all_tiles()
+ot = np.sort(rng.choice(TI["orion"], N, replace=False))
+names = [n for n, _ in _load_script("14_extract_if_fm.py", [sys.argv[0], "virtues", "--no_probe"]).MARKERS]
+ch = [names.index(m) for m in MIF_MARKERS]
+mif = np.stack([tifffile.imread(os.path.join(ORION, p))[..., ch].transpose(2, 0, 1) for p in tiles.target_path.values[ot]])
+for i, p in zip(ot, tiles.image_path.values[ot]):
+    shutil.copy(os.path.join(ORION, p), os.path.join(OUT, "orion_he", f"{i:05d}.png"))
+np.savez_compressed(os.path.join(OUT, "orion_mif.npz"), tiles=mif.astype(np.uint8), slide=tiles.in_slide_name.values[ot])
+ref |= {"orion_uni2": ld(CACHE, "he_uni2.npy")[ot, 0], "orion_medvtok": ld(CACHE, "he_medvtok.npy")[ot, 0],
+        "orion_virtues": ld(CACHE, "if_virtues_m16.npy")[ot, 0]}
+items += [{"set": "orion", "idx": int(i)} for i in ot]
+
+# HEST: spot crops + raw counts
+hs = np.sort(rng.choice(TI["hest"], N, replace=False))
+np.save(os.path.join(OUT, "hest_he.npy"), ld(CACHE, "hest", "he_img.npy")[hs])
+X, genes = load("hest")
+sp.save_npz(os.path.join(OUT, "hest_counts.npz"), sp.csr_matrix(X[hs]))
+pd.DataFrame({"gene": genes}).to_csv(os.path.join(OUT, "hest_genes.csv"), index=False)
+ref |= {"hest_uni2": ld(CACHE, "hest", "he_uni2.npy")[hs, 0], "hest_medvtok": ld(CACHE, "hest", "he_medvtok.npy")[hs, 0],
+        **{f"hest_{e}": (ld(CACHE, "hest", "hvg.npy") if e == "hvg" else ld(CACHE, "rnafm", f"{e}_hest.npy"))[hs]
+           for e in RNA_E}}
+items += [{"set": "hest", "idx": int(i)} for i in hs]
+
+# TCGA / PathGen: patch PNG + its paired caption
+k = np.sort(rng.choice(len(TI["tcga_img"]), N, replace=False))
+ti, tt = TI["tcga_img"][k], TI["tcga_txt"][k]
+I = pd.read_csv(os.path.join(CACHE, "allimg_meta.csv"))
+T = pd.read_csv(os.path.join(CACHE, "alltxt_meta.csv"))
+for i in ti:
+    shutil.copy(I.image_path.values[i], os.path.join(OUT, "tcga_he", f"{i:05d}.png"))
+pd.DataFrame({"img_idx": ti, "txt_idx": tt, "file": [f"{i:05d}.png" for i in ti],
+              "caption": T.caption.fillna("").astype(str).values[tt]}).to_csv(os.path.join(OUT, "tcga_captions.csv"), index=False)
+ref |= {"tcga_uni2": ld(CACHE, "allimg_he_uni2.npy")[ti, 0], "tcga_medvtok": ld(CACHE, "allimg_he_medvtok.npy")[ti, 0],
+        **{f"tcga_{e}": ld(CACHE, f"alltxt_text_{e}.npy")[tt, 0] for e in TXT_E}}
+items += [{"set": "tcga", "idx": int(i), "txt_idx": int(j)} for i, j in zip(ti, tt)]
+
+np.savez_compressed(os.path.join(OUT, "ref_feats.npz"), **{k_: np.asarray(v, np.float32) for k_, v in ref.items()})
+pd.DataFrame(items).to_csv(os.path.join(OUT, "items.csv"), index=False)
+os.system(f"du -sh {OUT}")

LICENSE.md

@@ -0,0 +1,14 @@
+# Licence
+
+The trained weights in `models/` and the code in this repository are released for **non-commercial research use only**.
+
+The weights were trained on features of frozen third-party encoders and on third-party datasets; any use must also
+comply with their terms:
+
+- Encoders: UNI2-h (MahmoodLab, CC-BY-NC-ND 4.0, gated), MedVTok (MIT; aligned to CONCH v1.5, CC-BY-NC-ND 4.0),
+  SE-600M (Arc Institute, non-commercial), AIDO.Cell (GenBio AI, non-commercial), Tahoe-x1 (Apache-2.0),
+  VirTues (MIT), KEEP (MIT), BiomedCLIP (MIT), Qwen3-Embedding (Apache-2.0).
+- Data: ORION CRC, HEST-1k (CC-BY-NC-SA 4.0), TCGA, PathGen-1.6M, spatialLIBD (DLPFC); virtual data from
+  STPath and MIPHEI (non-commercial academic) and captions generated with an Azure OpenAI model (evaluation only).
+
+No warranty. Not for clinical use.

README.md

@@ -0,0 +1,205 @@
+---
+license: other
+license_name: non-commercial-research
+license_link: LICENSE.md
+library_name: pytorch
+pipeline_tag: feature-extraction
+tags:
+  - pathology
+  - multimodal
+  - imagebind
+  - spatial-transcriptomics
+  - multiplex-immunofluorescence
+  - contrastive-learning
+---
+
+# 四模态病理嵌入模型（H&E · mIF · RNA · 文本）
+
+这个模型把四种模态嵌入到同一个 1536 维空间里：
+- **H&E** 图像；
+- **mIF**：多重免疫荧光，即蛋白；
+- **RNA**：空间转录组 spot 表达；
+- **病理文本**。
+
+做法和 ImageBind 一样。冻结的 UNI2-h 图像空间是锚点，其余每种模态只和 H&E 配对训练。mIF↔RNA、mIF↔文本、文本↔RNA 之间的对齐是"涌现"出来的，没有直接训练过。
+
+- **训练数据：** 只用真实的配对数据。
+- **评估数据：** 虚拟数据和合成数据只用于评估，从不参与训练。
+
+| 内容 | 位置 |
+|---|---|
+| 发布版模型 | `models/ej_ad_mv/` |
+| fold-0 模型（带留出测试集） | `models/ej_ad_mv_fold0/` |
+| 推理接口 | `embed.py` |
+| 训练 | `52_final_model.py` |
+| 训练用的编码器特征和数据划分 | 私有数据集仓库 [`Pathstudy/fourmodal-pathology-features`](https://huggingface.co/datasets/Pathstudy/fourmodal-pathology-features) |
+| 图文报告 | `report/fourmodal_report.html` |
+| 全部实验记录 | `RESULTS.md` |
+
+## 1. 数据
+
+没有任何数据集同时测量了全部四种模态，所以每种模态各用一个和 H&E 配对的真实数据集。
+
+| 数据集 | 配对 | 规模 | 用途 |
+|---|---|---|---|
+| ORION CRC（1% 图块子集） | H&E – mIF（17 通道，使用 16 个标记） | 3,187 个 333×333 图块，约 40 名病人 | 训练 + 测试 |
+| HEST-1k Visium | H&E – RNA（14,802 基因） | 9,484 spots，57 项研究，16 个器官 | 训练 + 测试 |
+| TCGA / PathGen-1.6M | H&E – 文本 | 6,144 张图像，6,115 条清洗后的描述 | 训练 + 测试 |
+| STPath 虚拟 RNA / MIPHEI 虚拟 mIF / GPT 合成描述 | 给没有真实配对的模态对构建测试集 | ORION、TCGA 测试图块 | **只用于测试** |
+| DLPFC（spatialLIBD，脑） | H&E – RNA | 482 spots，3 名供体 | 外部测试 |
+
+- **划分：** 按病人划分 train/val/test。HEST 按研究划分，病人分组见 `splits/`。做 5 折交叉验证。
+- **验证集：** val 只用于早停，所有报告的数字都是 test 上的。
+- **虚拟模态的校准：** 每种虚拟模态先在同一批有真实数据的样本上校准，即用虚拟数据替换真实数据，看配对结果能否接近。
+
+| 虚拟模态 | 校准结果（真实 → 虚拟） | 结论 |
+|---|---|---|
+| STPath 虚拟 RNA | 0.148 → 0.126 | 可用 |
+| GPT 合成描述 | 0.219 → 0.162 | 可用，偏保守 |
+| MIPHEI 虚拟 mIF | 配对准确率 0.92 → 0.55 | 可用 |
+| GigaTIME 虚拟 mIF | 0.644 → 0.038 | 不可用，已弃用 |
+
+## 2. 最佳架构：EJ-ad+MV（两阶段，3 种子适配器平均）
+
+```
+H&E ──► UNI2-h (frozen, 1536) ───────────────┬──────────────► (+) ──► z_HE
+     └► MedVTok (frozen, 1152) ─► Adapter×3 ◄┘ (zero-init, mean of 3 seeds)  ┐
+mIF  ──► VirTues (16 markers, 512) ─► MLP head ─────────────────────► z_mIF   │ 1536-d
+RNA  ──► hvg · Tahoe-x1 · SE-600M · AIDO.Cell ─► 4 linear heads ─► fuse ─► z_RNA  │ shared
+text ──► KEEP · BiomedCLIP · Qwen3-Emb-0.6B ─► 3 linear heads ─► fuse ─► z_text  ┘ space
+```
+
+**第一阶段：每个编码器一个映射头。**
+- 每个映射头用 InfoNCE 对齐到冻结的 UNI2 CLS 特征。
+- 文本用多正样本 InfoNCE：同一张图像的多条文本都算正样本。
+- RNA 和文本是后期融合：`z = normalise(Σ_e normalise(head_e(x_e)))`。
+
+**第二阶段：只训练 H&E 一侧。**
+- 冻结所有映射头，只训练一个零初始化的残差适配器：`H&E = UNI2 + mean_s A_s([UNI2, c·MedVTok])`。
+- 损失：三种配对的 InfoNCE 之和，加上对 UNI2 相似度结构的蒸馏（λ=1）。
+- 优化：AdamW，学习率 1e-4，1000 步，OneCycle 学习率调度。
+- 选择规则：每个种子按验证集上三种 R@1 的几何平均早停，再把 3 个种子的适配器残差取平均。
+
+**为什么这样设计：** mIF、RNA、文本的嵌入和第一阶段完全一样，所以涌现配对不受第二阶段影响，只有 H&E 这一侧变好。以下结论都来自 5 折配对检验，详见 `RESULTS.md`：
+
+| 设计选择 | 依据 |
+|---|---|
+| UNI2-h 做锚点 | 在任何训练配对上都没有被显著超过。GigaPath 在 RNA 上低 0.035；CONCH 在 H&E↔mIF 上低 0.155；MedVTok 当锚点在 H&E↔RNA 上只有 0.131（UNI2 是 0.192）。 |
+| 多编码器后期融合 | 最大的单项提升：H&E↔RNA +0.037，H&E↔文本 +0.062，4 个涌现配对全部提升（5/5 折）。拼接（早期融合）和单编码器多种子平均都达��到这个效果。 |
+| 只动 H&E 侧的联合训练 | 共享 H&E 头会让涌现配对损失 26–74%；只训练残差适配器时，H&E↔mIF 提升 0.10，涌现配对完全不变。 |
+| MedVTok 作为额外输入 | H&E↔mIF +0.015，H&E↔文本 +0.016。换成同尺寸的 UNI2 随机投影没有提升。 |
+| 3 种子平均 | 单次早停的步数在 100–900 步之间波动，同一设置重跑 H&E↔mIF 可以差 0.05。3 个种子平均后，H&E↔文本再提高 0.015（5/5 折，p=0.02）。 |
+| RNA 和文本用线性头 | 训练配对持平，涌现配对 +5%。mIF 保留 MLP 头。 |
+| 编码器挑选 | RNA：4 个编码器融合（+0.037）接近 10 个融合（+0.044）。文本：3 个融合（0.299）接近 5 个融合（0.303）。mIF：VirTues 的跨模态迁移比从头训练的 CNN 好得多。 |
+
+## 3. 结果（5 折测试集均值，100 个样本的检索库，随机水平 R@1 = 0.01）
+
+| 配对 | 类型 | R@1 | R@10 | 配对准确率 |
+|---|---|---|---|---|
+| H&E ↔ mIF（ORION） | 训练 | **0.801** ± 0.043 | 0.991 | 0.942 |
+| H&E ↔ RNA（HEST） | 训练 | **0.203** ± 0.033 | 0.625 | 0.258 |
+| H&E ↔ 文本（TCGA） | 训练 | **0.323** ± 0.007 | 0.813 | 0.347 |
+| mIF ↔ RNA（虚拟） | 涌现 | 0.141 ± 0.007 | 0.657 | 0.194 |
+| mIF ↔ 文本（合成） | 涌现 | 0.089 ± 0.008 | 0.491 | 0.104 |
+| 文本 ↔ RNA（虚拟） | 涌现 | 0.095 ± 0.006 | 0.436 | 0.100 |
+| 文本 ↔ mIF（MIPHEI 虚拟） | 涌现 | 0.046 ± 0.005 | 0.271 | 0.051 |
+| H&E ↔ RNA（DLPFC 脑，外部） | 外部 | 0.012 | 0.112 | 0.012 |
+
+各阶段的提升见下表（训练配对 R@1，5 折均值）。逐折结果在 `results/final_cv_long.csv`，各阶段的完整比较在 `results/medvtok_*.csv`、`results/joint_ens_*.csv`。
+
+| 阶段 | H&E↔mIF | H&E↔RNA | H&E↔文本 |
+|---|---|---|---|
+| 单编码器（S4） | 0.675 | 0.155 | 0.236 |
+| 多编码器融合（E，第一阶段） | 0.675 | 0.192 | 0.299 |
+| + H&E 适配器 | 0.776 | 0.201 | 0.299 |
+| + MedVTok | 0.792 | 0.200 | 0.315 |
+| **最终（3 种子平均）** | **0.801** | **0.203** | **0.323** |
+
+## 4. 使用方法
+
+环境：Python 3.11 和一块 CUDA GPU。依赖见 `requirements.txt`。所有脚本都用 `python -I` 运行。
+
+```bash
+pip install -r requirements.txt
+```
+
+**用预先算好的编码器特征**（不需要任何冻结编码器）：
+
+```python
+from embed import Embedder
+E = Embedder("models/ej_ad_mv")
+z_he   = E.he_from_feats(uni2_feats, medvtok_feats)      # (N,1536) / (N,1152)
+z_mif  = E.mif_from_feats(virtues_feats)                 # (N,512)
+z_rna  = E.rna_from_feats({"hvg": X, "tahoex1": X, "se600m": X, "aidocell": X})
+z_text = E.text_from_feats({"keep": X, "biomedclip": X, "qwen3-0.6b": X})
+sim = z_he @ z_mif.T          # 输出都是 L2 归一化的 (N, 1536) CPU tensor
+```
+
+**用原始输入**。需要先运行 `bash setup_encoders.sh` 下载第三方代码和权重。UNI2-h 是受限仓库，需要账号有访问权限，或者设置 `UNI2_PATH`。
+
+```python
+z = E.he(images)                       # PIL 图像列表或 (N,H,W,3) uint8；约 0.5 µm/px，视野 112–166 µm
+z = E.text(["Tumor cells with ...", ...])
+z = E.rna(counts, genes)               # 原始计数（spots × genes），HGNC 基因名
+z = E.mif(tiles, slide)                # (N,16,H,W) 8 位强度，通道顺序为 embed.MIF_MARKERS，每个图块附 slide id
+```
+
+注意：
+- **mIF 标准化：** 统计量是在一次调用给定的全部图块上计算的，所以请一次传入整张切片或整个数据集。
+- **新平台的 RNA 数据：** 建议用 `zscore="self"`，并且一次至少传入几百个 spots。
+
+## 5. 复现（已验证：从 HF 下载到全新目录 + 全新虚拟环境）
+
+```bash
+PYTHON=python bash reproduce.sh          # 约 20 分钟（单卡 RTX A5000）
+RAW=1 PYTHON=python bash reproduce.sh    # 再加上原始输入检查（需要先运行 setup_encoders.sh）
+```
+
+`reproduce.sh` 依次执行：
+1. 下载特征仓库，得到 `cache/` 和 `splits/`，共约 1.1 GB。
+2. `evaluate_saved.py`：用已发布的 fold-0 模型在留出测试集上重算结果，必须和 `models/ej_ad_mv_fold0/test.csv` 完全一致。
+3. 从特征重新训练 fold 0 和 5 折交叉验证，然后用 `compare_reproduction.py` 和已发布的数字对比。
+   - 在相同硬件和相同依赖版本下，结果逐位一致。2026-10-28 在 RTX A5000 上用全新的 `requirements.txt` 环境验证：fold 0 和 5 折的全部配对差值都是 0。
+   - 换 GPU 或库版本时，第二阶段（H&E 适配器）可能在种子波动范围内变化，约 ±0.01；`compare_reproduction.py` 的容差就是按这个设的。
+4. 重新训练发布版模型，输出到 `models/repro_release/`。
+5. 加 `RAW=1` 时运行 `example_check.py`：把 `examples/`（每个数据集 64 个 fold-0 测试样本）的原始输入送进 `embed.py`，检查编码器特征和缓存特征是否一致，再检查检索结果。
+
+已发布的结果不会被覆盖：复现的输出写到 `repro_*` 名下。
+
+## 6. 局限
+
+- **外部泛化差：** 脑组织 DLPFC 上处于随机水平。训练用的 HEST 里脑组织也只有 R@1 0.021。所以模型只适用于训练数据覆盖过的组织类型。
+- **H&E↔文本的 MedVTok 增益可能偏高：** MedVTok 预训练时见过 PathGen 的图像，包括我们 TCGA 测试集的图像（只有图像，没有文本）。ORION 上的 mIF 增益不受影响。
+- **涌现配对的数值偏保守：** 涌现配对的一侧是虚拟或合成数据，所以这些数值是保守估计。
+- **实例级涌现对齐仍然弱：** mIF↔RNA 只有 0.141。概念级的对齐（AUROC 0.6–0.7）更可靠。
+
+## 7. 许可证
+
+**只能用于非商业研究。** 本仓库的权重是在以下冻结编码器的特征上训练的，使用时需要分别遵守它们的许可：
+
+| 编码器 | 许可 |
+|---|---|
+| UNI2-h | CC-BY-NC-ND，受限访问 |
+| MedVTok | MIT；它对齐的 CONCH v1.5 是 CC-BY-NC-ND |
+| SE-600M、AIDO.Cell | 非商业 |
+| Tahoe-x1 | Apache-2.0 |
+| VirTues、KEEP、BiomedCLIP | MIT |
+| Qwen3-Embedding | Apache-2.0 |
+
+数据本身也有各自的条款，见各数据集的原始发布：ORION、HEST-1k（CC-BY-NC-SA）、TCGA、PathGen-1.6M、spatialLIBD。
+
+## 8. 文件
+
+```
+embed.py                 推理接口 (Bind4 模型 + Embedder)
+52_final_model.py        训练 (cv / fold0 / release)
+evaluate_saved.py        特征级评估 (fold-0 测试, 与 test.csv 完全一致)
+example_check.py         原始输入端到端检查 (examples/)
+reproduce.sh, compare_reproduction.py, setup_encoders.sh, requirements.txt
+14_ 43_ 45_ 45b_ 50_ *.py, biomedclip_text.py, common.py, rna_sets.py   冻结编码器的特征提取 (embed.py 调用)
+53_embed_check.py, 54_make_examples.py   在完整原始数据上做的检查 / 生成 examples/ (需要原始数据集)
+models/                  ej_ad_mv (发布版), ej_ad_mv_fold0 (+ test.csv, test_items.npz)
+results/                 所有实验的 csv 结果
+experiments/             01_–51_ 全部实验脚本 (记录用; 依赖原始数据, 不在本仓库)
+RESULTS.md               完整实验记录;  report/fourmodal_report.html  图文报告
+```

RESULTS.md

@@ -0,0 +1,778 @@
+# Results — ImageBind-style multimodal binding on ORION (1% subsample)
+
+Metric: **R@1 on random 100-tile galleries** (chance = 0.01), mean over the two retrieval directions unless noted;
+patient 5-fold reported as mean ± std over folds. "Pairing acc" = one-to-one Hungarian matching on the full test set.
+Your split (`orion_official`) has 109 test tiles from 2 slides → single, noisy estimate.
+
+## 1. Base experiment (H&E / IF / nuclei / markers, ImageBind-Huge H&E anchor) — 22 runs
+Your split, R@1 (full 109-tile gallery):
+
+| pair | anchor loss (ImageBind) | all-pairs loss |
+|---|---|---|
+| H&E→IF | **0.49** (pair acc 0.73) | 0.38 |
+| H&E→nuclei | 0.36 | 0.28 |
+| H&E→markers | 0.16 | 0.15 |
+| IF→nuclei (emergent) | 0.41 | **0.54** |
+| IF→markers (emergent) | 0.26 | **0.70** |
+| nuclei→markers (emergent) | 0.15 | 0.29 |
+
+- Emergent pairs (never trained together) match far above chance.
+- Training the non-H&E pairs directly (all-pairs) boosts them but costs H&E→IF.
+- Patient 5-fold ≈ leave-one-IF-batch-out → small batch effect; random tile split is optimistic.
+
+## 2. H&E encoder ablation (anchor loss) — 33 runs
+Mean R@1 (g100) over H&E pairs / emergent pairs:
+
+| H&E encoder | your split | patient 5-fold |
+|---|---|---|
+| ImageBind-Huge | 0.29 / 0.25 | 0.33 ± 0.01 / 0.30 ± 0.00 |
+| PLIP (stand-in for CONCH) | 0.35 / 0.29 | 0.37 ± 0.01 / 0.31 ± 0.01 |
+| Phikon-v2 (stand-in for GigaPath) | **0.41** / 0.31 | 0.45 ± 0.02 / 0.36 ± 0.01 |
+| **UNI2-h** | 0.40 / **0.34** | **0.48 ± 0.02 / 0.37 ± 0.01** |
+
+CONCH and GigaPath: HF access refused (GatedRepoError); code ready (`04b_extract_anchor_fm.py`).
+
+## 3. Encoder benchmark (H&E = UNI2 frozen; mIF / markers / text) — 18 runs
+Text = templated per-tile descriptions generated from marker counts + nuclei count (no real text exists for ORION).
+
+### Trained pairs (H&E ↔ modality), R@1 g100
+| modality | encoder | your split | patient 5-fold |
+|---|---|---|---|
+| mIF | ResNet18 (17-ch stem) | 0.59 | 0.64 ± 0.01 |
+| mIF | **ConvNeXt-Tiny** (17-ch stem) | **0.73** (pair acc 0.95) | **0.78 ± 0.02** |
+| mIF | per-channel encoder (shared CNN + channel transformer) | 0.51 | 0.62 ± 0.01 |
+| markers | linear | 0.18 | 0.22 ± 0.01 |
+| markers | **MLP** | **0.31** | **0.39 ± 0.03** |
+| markers | FT-Transformer | 0.21 | 0.30 ± 0.02 |
+| text | PLIP text | 0.17 | 0.20 ± 0.02 |
+| text | BiomedCLIP text | 0.16 | 0.19 ± 0.02 |
+| text | ImageBind text | **0.22** | 0.21 ± 0.02 |
+| text | PLIP / ImageBind **zero-shot** (no training) | ≈0.01–0.02 | ≈0.01–0.02 (chance) |
+
+### Cross pairs (never trained together), best combos, patient 5-fold R@1 g100
+| pair | best combo | R@1 |
+|---|---|---|
+| mIF ↔ markers | ConvNeXt-T + MLP / ResNet18 + MLP | 0.48 ± 0.03 |
+| markers ↔ text | MLP + PLIP / MLP + ImageBind | 0.43 ± 0.07 / 0.42 ± 0.03 |
+| mIF ↔ text | ResNet18/ConvNeXt-T + ImageBind | 0.25 ± 0.01 |
+
+### Takeaways
+- **mIF**: ConvNeXt-Tiny clearly best (+0.14 over ResNet18). The channel-agnostic per-channel encoder is not better here
+  (fixed 17-marker panel; it would matter for transfer across panels).
+- **Markers**: the plain MLP wins; FT-Transformer < MLP, linear worst. MLP markers also give the best cross-modal pairs.
+- **Text**: the three text towers are within noise (0.19–0.21); ImageBind text slightly ahead. Zero-shot pathology CLIP
+  (PLIP) cannot match these synthetic marker-based descriptions to H&E (≈ chance) → training a projection is required.
+- Text is derived from markers, so markers↔text is partly by construction; H&E↔text measures how well H&E predicts
+  coarse cell composition. Real paired text (PathGen images or ORION reports) is needed for a true text evaluation.
+- Gene expression: none in ORION; the marker-encoder benchmark would be repeated on HEST/DLPFC for RNA.
+
+Files: `results/bench_summary.csv`, `results/bench_long.csv`, `results/anchor_ablation_*.csv`, `results/summary_*.csv`.
+
+---
+# v2 / v3 — re-sampled ORION (rsync 2026-10-11) + real text + modality-specific foundation models
+v1 sections above used the old ORION 1% sample (archived in `v1_old_orion_sample/`). Same slide split, new tiles.
+H&E anchor = UNI2 (frozen) throughout. R@1 on 100-tile galleries (chance 0.01); patient 5-fold mean ± std.
+
+## Text bound to H&E with real captions (PathGen-1.6M, patient split, 255 test pairs) — `12_train_text_pathgen.py`
+| text tower (frozen) + MLP head | H&E→text R@1 (full 255) | g100 | zero-shot (own image tower) |
+|---|---|---|---|
+| BiomedCLIP | **0.17** | **0.29** | – |
+| PLIP | 0.13 | 0.23 | 0.035 |
+| ImageBind | 0.11 | 0.22 | 0.024 |
+| PathGen-CLIP-L | 0.13 | 0.24 | 0.19* |
+*PathGen-CLIP-L was trained on PathGen-1.6M → its PathGen numbers are not held-out.
+
+## mIF / markers: from-scratch vs modality-specific foundation models — `17_bench_fm.py`, `18_eval_fm.py`
+| pair | encoder | your split (pair acc) | patient 5-fold |
+|---|---|---|---|
+| H&E↔mIF | ConvNeXt-T (scratch) | **0.73** (0.96) | **0.76 ± 0.02** |
+| H&E↔mIF | **VirTues** sp32 (frozen) + MLP | 0.67 (0.95) | 0.73 ± 0.03 |
+| H&E↔markers | MLP on raw markers | **0.28** | **0.38 ± 0.03** |
+| H&E↔markers | Geneformer V2 (frozen) + MLP | 0.12 | 0.17 ± 0.01 |
+| H&E↔markers | scGPT human (frozen) + MLP | 0.11 | 0.18 ± 0.01 |
+| mIF↔markers (emergent) | ConvNeXt-T + MLP | 0.37 | 0.48 ± 0.02 |
+| mIF↔markers (emergent) | **VirTues + MLP** | **0.51 (0.93)** | **0.66 ± 0.02** |
+| mIF↔markers (emergent) | VirTues + scGPT / + Geneformer | 0.22 / 0.19 | 0.29 / 0.26 |
+
+## Emergent text ↔ mIF / markers (text head trained only on PathGen; never sees ORION/IF/markers) — patient 5-fold AUROC (Spearman), 7 concept prompts
+| text | H&E (UNI2) | mIF ConvNeXt | mIF VirTues | markers MLP | markers scGPT | markers Geneformer |
+|---|---|---|---|---|---|---|
+| **PLIP** | 0.69 (0.32) | 0.69 (0.31) | 0.71 (0.34) | 0.76 (0.46) | **0.79 (0.49)** | 0.74 (0.41) |
+| ImageBind | 0.67 (0.32) | 0.65 (0.27) | 0.67 (0.31) | 0.73 (0.41) | 0.75 (0.45) | 0.70 (0.35) |
+| BiomedCLIP | 0.66 (0.25) | 0.66 (0.24) | 0.68 (0.26) | 0.71 (0.29) | 0.72 (0.32) | 0.68 (0.27) |
+| PathGen-CLIP-L | 0.59 (0.10) | 0.62 (0.13) | 0.63 (0.16) | 0.66 (0.18) | 0.69 (0.25) | 0.64 (0.18) |
+Zero-shot H&E (own image tower, no binding): PathGen-CLIP-L 0.75 (0.46), PLIP 0.73 (0.41), ImageBind 0.65 (0.28).
+
+### Takeaways
+- **mIF**: frozen VirTues ≈ ConvNeXt from scratch on H&E↔mIF (−0.03, within ~1 std), but its embedding transfers much better
+  to markers through the shared space (emergent mIF↔markers 0.66 vs 0.48) — the pretrained, marker-aware mIF model is the
+  better choice for a multi-modal space.
+- **markers**: Geneformer / scGPT are clearly worse than an MLP on raw values for retrieval: both models only see the rank
+  order of the 16 mapped genes in a tile (abundance is lost; ridge recovers marker props at ρ≈0.69 vs 0.998 raw).
+  scGPT markers nevertheless give the best emergent text↔markers probe (PLIP 0.79). Single-cell RNA models are not a good fit
+  for tile-level protein proportions; they should be tested on real expression (HEST/DLPFC).
+- **text**: PLIP text head transfers best to ORION concepts; BiomedCLIP best on PathGen retrieval. PathGen-CLIP-L is the
+  strongest zero-shot model, but its text features bound to UNI2 transfer worst (likely memorised PathGen captions → head
+  overfits). Recommended: UNI2 (H&E) + VirTues (mIF) + MLP (markers) + PLIP text head.
+- Data note: `PD-1_prop` ≈ 1 in every tile (doesn't track PD-1_count) — the column looks corrupted.
+
+Files: `results/fm_retrieval.csv`, `results/fm_text_probes.csv`, `runs/bench_fm/`, `runs/text_pathgen/`, logs in `logs/v2`, `logs/v3`.
+
+## Text ↔ H&E image pairing on ALL image-text pairs — `19_pairsets_prep.py`, `20_pairing.py`
+Unified pool of every text-image source in PathGen-1.6M/ (12 sources; pathor1 skipped, images absent): 6144 unique
+TCGA patches (4088 patients), 6480 pairs — caption 2697, qa_open 1799, qa_close 1391 (stem + correct option), summary 593.
+Patient 5-fold CV; one text per image at test; test gallery ≈1229 pairs ("all"), 540 caption, 330 qa_open, 278 qa_close,
+119 summary. Metric: R@1 on 100-pair galleries (chance 0.01) / Hungarian pair acc on the full fold; mean of both directions.
+
+| image enc → / text enc ↓ (text_head, eval=all, R@1 g100) | BiomedCLIP | PLIP | ImageBind | PathGen-CLIP-L* |
+|---|---|---|---|---|
+| UNI2 | **0.231** | 0.200 | 0.199 | 0.248 |
+| Phikon-v2 | **0.234** | 0.204 | 0.197 | 0.254 |
+| PLIP (image) | 0.170 | 0.147 | 0.139 | 0.180 |
+| BiomedCLIP (image) | 0.163 | 0.138 | 0.136 | 0.168 |
+| ImageBind (image) | 0.132 | 0.108 | 0.098 | 0.139 |
+| PathGen-CLIP-L (image)* | 0.282 | 0.252 | 0.243 | 0.303 |
+Zero-shot (same CLIP model): PLIP 0.090, BiomedCLIP 0.111, ImageBind 0.029, PathGen-CLIP-L* 0.188.
+Mode means (excluding PathGen-CLIP-L): zero-shot 0.077 < two_heads 0.165 ≈ text_head 0.166.
+
+By text type, UNI2 + BiomedCLIP text head (g100 / pair acc): caption 0.28 / 0.11, summary 0.29 / 0.29, qa_open 0.22 / 0.09,
+qa_close 0.13 / 0.06 (short multiple-choice stems carry little patch-specific information).
+*PathGen-CLIP-L was trained on PathGen-1.6M (most of this pool) → not held-out; shown for reference only.
+Takeaways: pathology SSL image encoders (UNI2, Phikon-v2) + BiomedCLIP text (256-token context) are the best clean
+combination; binding to a frozen image anchor (ImageBind recipe) is as good as training both projections; every trained
+head beats zero-shot CLIP by ~2×.
+Files: `results/pairing_long.csv`, `results/pairing_summary.csv`.
+
+## Four real modalities: H&E = image, text = text, mIF = protein, gene expression = RNA (HEST) — `21_`–`24_`
+The former "markers" modality was computed from the mIF image (protein), so it is retired. Gene expression is now real
+spatial transcriptomics: HEST-1k human Visium, 9484 spots (415 samples, 57 studies, 16 organs, 2046 bowel), 14,802
+common genes, precomputed UNI2-h / GigaPath spot features. One frozen UNI2 space holds every modality:
+H&E–mIF (ORION), H&E–text (PathGen), H&E–RNA (HEST) are trained; protein↔RNA, text↔RNA, text↔protein are emergent
+(no dataset measures them together, so no 1:1 pairing accuracy is possible for them).
+
+### H&E ↔ RNA (trained), HEST study+patient 5-fold, test gallery ≈1897 spots; R@1 on 100-spot galleries (chance 0.01)
+| RNA encoder | UNI2 anchor | GigaPath anchor |
+|---|---|---|
+| MLP on 2000 HVGs | 0.148 ± 0.026 | 0.113 ± 0.018 |
+| scGPT-human (frozen) + MLP | 0.154 ± 0.028 | 0.118 ± 0.017 |
+| Geneformer-V2 (frozen) + MLP | **0.159 ± 0.027** | 0.124 ± 0.010 |
+On real expression, the single-cell foundation models slightly beat the MLP. On the derived markers it was the other way round.
+
+### Protein ↔ RNA (emergent, cross-dataset neighbour transfer; Spearman, shuffled-label chance ≈ 0.00)
+ORION tile → k=20 nearest HEST spots. The predicted RNA of the matching gene is compared with the measured protein (marker⁺ cells / nuclei).
+
+| query → gallery | all spots | bowel spots |
+|---|---|---|
+| **protein (VirTues mIF) → RNA (MLP / Geneformer)** | **0.45 / 0.44** | **0.42 / 0.42** |
+| H&E (UNI2) → RNA head | 0.38 | 0.33 |
+| H&E → H&E (image similarity only) | 0.36 | 0.33 |
+| RNA → protein (reverse) | 0.18–0.22 | 0.20–0.23 |
+
+Per marker (bowel, protein→RNA vs H&E→H&E):
+- Pan-CK 0.81 vs 0.81; E-cad 0.81 vs 0.77
+- Ki67 0.63 vs 0.49
+- SMA 0.60 vs 0.49
+- CD45 0.52 vs 0.35
+- CD3e 0.38 vs 0.32
+- CD68 0.32 vs 0.21
+- CD4 0.29 vs 0.09
+- CD20 0.29 vs 0.10
+- PD-1 / PD-L1 < 0.2
+
+### Text ↔ RNA (emergent, PathGen text head; concept prompts vs RNA of the concept gene; AUROC top quartile)
+| text encoder | RNA emb (best of 3), all / bowel | H&E (UNI2) of the same spots, all / bowel |
+|---|---|---|
+| PLIP | 0.64 / 0.65 | 0.58 / 0.54 |
+| ImageBind | 0.64 / 0.63 | 0.60 / 0.56 |
+| BiomedCLIP | 0.61 / 0.63 | 0.57 / 0.54 |
+| PathGen-CLIP-L | 0.59 / 0.62 | 0.55 / 0.56 |
+Bowel AUROC per concept (RNA/MLP): stroma 0.74, immune 0.69, tumour epithelium 0.66, proliferation 0.59, macrophage 0.52, vessels 0.51.
+Files: `cache/hest/`, `runs/gene/`, `results/gene_retrieval.csv`, `gene_emergent_protein.csv`, `gene_emergent_text.csv`, logs in `logs/gene/`.
+
+## Virtual modalities predicted from H&E (to fill missing pairs; never used as test truth)
+
+### GigaTIME (H&E → 21-channel virtual mIF) — `25_gigatime_infer.py`, `cache/gigatime/`
+Model: prov-gigatime/GigaTIME (code Apache-2.0; weights research-only). Run at native tile resolution (2× upsampling was worse).
+Sets: ORION 3,187 tiles, TCGA text patches 6,144, gene-expression patches 1,640; 0 failures.
+
+ORION validation against the real mIF of the same tile (Spearman of tile-mean intensity, n = 3,187):
+
+| Channel (GigaTIME ~ ORION) | Spearman |
+|---|---|
+| DAPI ~ Hoechst | 0.91 |
+| CK ~ Pan-CK | 0.74 |
+| Ki67 | 0.73 |
+| PD-1 | 0.73 (the ORION PD-1 proportion is corrupt, ≈1 everywhere) |
+| CD3 ~ CD3e | 0.65 |
+| CD4 | 0.58 |
+| CD68 | 0.47 |
+| CD8 ~ CD8a | 0.37 |
+| CD20 | 0.36 |
+| PD-L1 | 0.34 |
+| Actin-D / Transgelin ~ SMA | 0.18 / 0.23 |
+
+Use the per-tile means only. At the pixel level, the per-tile median Pearson is ≈0.1 for every channel except DAPI (0.50), so the virtual maps are not spatially reliable.
+
+### STPath (H&E → whole-transcriptome virtual RNA) — `26_stpath_infer.py`, `cache/stpath/`
+Model: tlhuang/STPath (**no licence declared** — ask the authors before redistributing). Input: GigaPath tile features + coordinates + organ/tech tokens. Output: log1p counts for 14,791 genes.
+Sets: HEST 9,484 spots, ORION 3,187 tiles, TCGA text patches 6,144.
+
+HEST check (top-50 HVG Pearson):
+- All spots: 0.40. This number is optimistic because STPath was trained on most of HEST.
+- HEST-bench samples held out from STPath: 0.21.
+- Within-sample: ≈0.09.
+
+ORION check: predicted RNA vs the real protein of the same tile (Spearman, tiles with ≥20 nuclei; pooled / within-slide):
+
+| Gene ~ protein | Pooled | Within slide |
+|---|---|---|
+| CDH1 ~ E-cad | 0.77 | 0.77 |
+| EPCAM ~ Pan-CK | 0.76 | 0.74 |
+| MKI67 ~ Ki67 | 0.68 | 0.64 |
+| CD3E ~ CD3e | 0.45 | 0.42 |
+| CD8A ~ CD8a | 0.44 | 0.40 |
+| MS4A1 ~ CD20 | 0.42 | 0.34 |
+| PTPRC ~ CD45 | 0.37 | 0.34 |
+| ACTA2 ~ SMA | 0.34 | 0.34 |
+| CD68 ~ CD68 | 0.25 | 0.22 |
+| CD163 ~ CD163 | 0.07 | 0.15 |
+
+TCGA caveat: 2,540 of 4,088 slides contribute a single patch, so most TCGA predictions have no spatial context and are less reliable than the ORION ones.
+
+Coverage after completion: TCGA patches have H&E + real text + virtual protein + virtual RNA. ORION has H&E + real protein + virtual RNA, and synthetic text is pending (Azure). HEST has H&E features + real RNA.
+
+## H&E foundation-model ablation: UNI2 vs GigaPath vs CONCH as the frozen anchor — `run_v4.sh`, `28_eval_anchor.py`
+All other settings are identical. CONCH v1 vision tower: 448 px, unprojected 512-d (the authors' recommended tile feature). `conchclip` = CONCH projected into its own image-text space.
+
+### H&E ↔ mIF (ORION, trained pair; mIF = frozen VirTues + MLP head; both directions averaged)
+| Anchor | Patient 5-fold R@1 g100 | Patient 5-fold pair acc | ORION official R@1 (109 tiles) | ORION official pair acc |
+|---|---|---|---|---|
+| **UNI2** | **0.727 ± 0.030** | **0.751 ± 0.045** | 0.665 | **0.945** |
+| GigaPath | 0.717 ± 0.025 | 0.690 ± 0.046 | **0.702** | 0.936 |
+| CONCH | 0.571 ± 0.035 | 0.379 ± 0.037 | 0.529 | 0.651 |
+
+Same with a from-scratch ConvNeXt-T mIF encoder:
+
+| Anchor | Patient 5-fold R@1 g100 | Patient 5-fold pair acc | ORION official R@1 | ORION official pair acc |
+|---|---|---|---|---|
+| **UNI2** | **0.753 ± 0.019** | **0.744 ± 0.022** | **0.738** | **0.945** |
+| GigaPath | 0.740 ± 0.022 | 0.691 ± 0.018 | 0.707 | 0.927 |
+| CONCH | 0.639 ± 0.021 | 0.481 ± 0.058 | 0.608 | 0.780 |
+
+The ranking UNI2 ≥ GigaPath ≫ CONCH holds for both mIF encoders.
+
+### H&E ↔ text (all 6,144 TCGA patches / 6,480 real pairs, patient 5-fold, R@1 on 100-pair galleries, "all" text types)
+| Image | Text | Text head only (ImageBind) | Two heads (CLIP-style) |
+|---|---|---|---|
+| UNI2 | BiomedCLIP | **0.231 ± 0.008** | 0.210 ± 0.012 |
+| CONCH | BiomedCLIP | 0.230 ± 0.007 | **0.229 ± 0.007** |
+| CONCH-CLIP | BiomedCLIP | 0.226 ± 0.007 | 0.226 ± 0.012 |
+| GigaPath | BiomedCLIP | 0.218 ± 0.007 | 0.200 ± 0.011 |
+| CONCH | CONCH text | 0.216 ± 0.009 | 0.221 ± 0.007 |
+| UNI2 / GigaPath | CONCH text | 0.209 / 0.192 | 0.193 / 0.179 |
+| CONCH-CLIP | CONCH text, **zero-shot** | 0.170 ± 0.010 | — |
+
+On the PathGen-1.6M caption split (`12_train_text_pathgen.py`, BiomedCLIP, 255 test pairs, g100 he→text / text→he):
+- UNI2: 0.29 / 0.27
+- GigaPath: 0.26 / 0.22
+- CONCH: 0.26 / 0.24
+
+### Emergent text ↔ mIF (PathGen-trained text head per anchor, BiomedCLIP; ORION patient 5-fold AUROC, mean of 7 concepts)
+| Anchor | mIF embedding | H&E embedding |
+|---|---|---|
+| UNI2 | 0.674 | 0.665 |
+| GigaPath | 0.686 | 0.678 |
+| **CONCH** | **0.719** | **0.701** |
+
+CONCH gains most on tumour epithelium (0.89 vs UNI2 0.76), macrophage (0.65 vs 0.60) and vessels (0.57 vs 0.44).
+
+### Takeaways
+- **Trade-off between instance-level pairing and semantics.**
+  - CONCH has the weakest H&E↔mIF pairing: pair accuracy is half of UNI2's.
+  - CONCH transfers text concepts to mIF best, emergently. Its features were trained against captions, so they encode coarse semantics more than fine tile identity.
+- **UNI2 is the best single anchor.** It is the strongest or tied on every trained pair: mIF, text, and RNA (UNI2 > GigaPath on HEST, see above).
+- **The CONCH text tower is worse than BiomedCLIP** on these long PathGen texts, even when paired with CONCH images. Zero-shot CONCH (0.17) is well below any trained head (≥0.18).
+- **HEST RNA has no CONCH anchor** because only UNI2 and GigaPath features are distributed, not the images.
+
+## Virtual modality completion: train text directly on virtual pairs — `29_virtual_completion.py`
+**Question:** does training text↔RNA or text↔protein directly on virtual pairs beat the emergent route through the H&E anchor?
+- RNA pairs: TCGA real text ↔ STPath virtual RNA.
+- Protein pairs: TCGA real text ↔ GigaTIME virtual mIF (11 channels shared with ORION).
+- Model: BiomedCLIP head + MLP head on the modality vector, patient 5-fold over TCGA.
+- Features are rank-gaussianised within each dataset.
+- Testing uses only **real** measurements, with the same prompts, labels and protocol as the emergent probes.
+
+| Text ↔ | Test set | Direct on virtual pairs | Emergent via H&E anchor (best encoder) |
+|---|---|---|---|
+| RNA | HEST real RNA, all spots | 0.595 | 0.59–0.61 (MLP / scGPT / Geneformer) |
+| RNA | HEST real RNA, bowel | 0.552 | 0.61–0.62 |
+| protein | ORION real mIF means | 0.596 | **0.678** (VirTues mIF), 0.663 (H&E) |
+| protein | ORION *virtual* GigaTIME means (same tiles) | 0.686 | — |
+
+(AUROC, top-quartile label, mean over concepts.)
+
+- **In-domain retrieval** (held-out TCGA, text ↔ virtual modality, R@1 g100):
+  - RNA: 0.196 ± 0.010
+  - protein, 11 channel means: 0.048 ± 0.007
+- **Virtual pairs do not beat the emergent route.** RNA ties on all spots and is worse on bowel. Protein is clearly worse.
+- **The bottleneck is the virtual→real gap.** The same protein model scores 0.686 on ORION's virtual means but 0.596 on the real means of the same tiles.
+- **The single-channel reference** (the matched real channel used as the score) gives 0.79–0.98. The probes are far from saturated, and an 11-number protein summary carries much less than the VirTues image embedding.
+- **Recommendation:** use virtual modalities only as auxiliary or augmentation data, never as replacements for, or test truth instead of, real pairs.
+
+## Four-modality validation with virtual / synthetic data (validation only, never training) — `31_virtual_validation.py`
+- **Training:** every encoder is trained on real pairs only (H&E–mIF ORION, H&E–text PathGen, H&E–RNA HEST; frozen UNI2 anchor).
+- **Virtual / synthetic data:** used only to build quadruplet test sets, so that modality pairs never measured together get a pairing accuracy:
+  - virtual RNA: STPath
+  - virtual mIF: GigaTIME
+  - synthetic text: Azure `gpt-5.6-sol`, fixed 3-shot PathGen-style prompt, `27_azure_caption.py`; 9,031 captions, 0 failures, USD 38.11
+- **Metrics:** 100-item galleries, R@1 (both directions) / one-to-one Hungarian pairing accuracy; chance 0.01.
+- **TCGA evaluation:** 2,868 patches of 2,158 patients that the text head never saw.
+
+### Calibration: is each virtual modality a usable stand-in for the real one?
+| Virtual modality | Real pair (same items) | Same pair with the virtual modality | Verdict |
+|---|---|---|---|
+| RNA (STPath), HEST all | H&E↔RNA 0.148 / 0.166 | 0.126 / 0.182 | **usable** |
+| RNA (STPath), HEST held out from STPath | 0.086 / 0.110 | 0.080 / 0.136 | **usable** |
+| text (synthetic), 1,000 TCGA patches with real text | H&E↔text 0.184 / 0.165 | 0.134 / 0.140 | **usable, conservative (≈0.75×)** |
+| text, same 1,000 patches | text real↔RNA virtual 0.039 / 0.038 | text synthetic↔RNA virtual 0.044 / 0.049 | consistent |
+| mIF (GigaTIME, 10-ch VirTues), ORION | H&E↔mIF 0.644 / 0.863 | 0.038 / 0.046 | **not usable** |
+| mIF (GigaTIME, 10 channel means), ORION | 0.279 / 0.357 | 0.050 / 0.053 | **not usable** |
+
+GigaTIME's tile means correlate with the real mIF at the population level (Spearman ≈0.6), but they cannot tell one tile from 99 others. Pixel maps are worse. Virtual mIF is therefore only good for coarse concept-level checks, not pairing accuracy.
+
+### Pairing accuracy for every modality pair (R@1 / pair acc on 100-galleries; ⚠ = circular)
+"Circular" means both sides come from the same H&E image, so the number is an optimistic upper bound.
+
+**ORION** (H&E real, mIF real (VirTues 17 ch), RNA virtual, text synthetic), patient 5-fold:
+
+| | mIF (real) | RNA (virtual) | text (synthetic) |
+|---|---|---|---|
+| **H&E** | **0.726 / 0.935** (trained) | 0.103 / 0.192 ⚠ | 0.066 / 0.091 ⚠ |
+| **mIF (real)** | | **0.108 / 0.152** (emergent) | **0.068 / 0.069** (emergent) |
+| **RNA (virtual)** | | | 0.036 / 0.050 ⚠ |
+
+**TCGA** (H&E real, text real, RNA virtual):
+
+| | text (real) | RNA (virtual) |
+|---|---|---|
+| **H&E** | **0.178 / 0.170** (trained) | 0.121 / 0.175 ⚠ |
+| **text (real)** | | **0.042 / 0.041** (emergent) |
+
+Virtual mIF on TCGA (text↔mIF 0.012–0.021) fails calibration and is not reported as a result.
+
+### Takeaways
+- **Protein ↔ RNA**, never trained together: 0.108 R@1 / 0.152 pairing accuracy on ORION, 10–15× chance. This is real protein against virtual RNA, and virtual RNA scores close to real RNA in the HEST calibration.
+- **Text ↔ protein**, never trained together: 0.068, about 7× chance. This uses synthetic text, which scores about 0.75× real text in the calibration, so the real-text value is likely ≈0.09.
+  - ORION is a single-organ cohort (colon), so its galleries are harder than TCGA's 30-cancer galleries. H&E↔synthetic text is 0.066 on ORION vs 0.134 on TCGA.
+- **Text ↔ RNA**, never trained together: 0.042 with real text and 0.044 with synthetic text on the same patches, about 4× chance.
+- Emergent pairs are clearly above chance but an order of magnitude below the trained H&E↔mIF pair. Instance-level emergent alignment is weak; concept-level alignment (AUROC 0.6–0.7, earlier sections) is moderate.
+
+## Final protocol: real-only training, explicit train/val/test, virtual = test-time only (`33_final_eval.py`)
+Every head is trained on REAL pairs only; val is used only for early stopping; all numbers are on the TEST split
+(5 folds, mean ± std; patients/groups never cross splits, asserted). Virtual modalities (STPath RNA, GigaTIME mIF,
+GPT captions) are only fed through the frozen real-trained heads at test time. Metrics: 100-item galleries × 50 draws,
+both directions; chance R@1 = pair acc = 0.01, R@10 = 0.10. Split files: `splits/patient_5fold`, `splits/orion_official`,
+`splits/final/{hest,tcga}_fold<k>.csv`; sizes `results/final_splits.csv`; numbers `results/final_{long,summary}.csv`.
+TCGA text heads are now trained on ALL image-text pairs per fold (`runs/text_all/uni2_biomedclip_fold<k>`).
+
+| dataset | unit | train | val | test (per fold) |
+|---|---|---|---|---|
+| ORION patient 5-fold | patient | 2291 tiles / 29 pts | 258 / 3 | 637 / 8 |
+| ORION official (user split) | patient | 2954 / 36 | 124 / 2 | 109 / 2 |
+| HEST | study\|patient group | 7141 spots / 91 grp | 447 / 10 | 1897 / 25 |
+| TCGA all image-text pairs | patient | 4426 images / 2943 pts | 489 / 327 | 1229 / 818 |
+
+**(1) Real pairs (test split)**
+
+| pair | dataset | R@1 | R@10 | MRR | pair acc |
+|---|---|---|---|---|---|
+| H&E↔mIF | ORION 5-fold | 0.726±0.031 | 0.989 | 0.828 | 0.935±0.030 |
+| H&E↔mIF | ORION official | 0.665 | 1.000 | 0.791 | 0.948 |
+| H&E↔RNA | HEST | 0.148±0.026 | 0.547 | 0.277 | 0.166±0.032 |
+| H&E↔RNA | HEST, STPath-unseen spots | 0.086±0.038 | 0.509 | 0.214 | 0.110±0.038 |
+| H&E↔text | TCGA | 0.229±0.011 | 0.700 | 0.383 | 0.238±0.011 |
+
+**(2) Virtual pairs (same test items; ⚠ = both sides derived from the same H&E → optimistic)**
+
+| pair | dataset | R@1 | R@10 | pair acc | type |
+|---|---|---|---|---|---|
+| mIF↔RNA(virtual) | ORION 5-fold | 0.108±0.006 | 0.516 | 0.152±0.020 | emergent, real×virtual |
+| mIF↔text(synthetic) | ORION 5-fold | 0.073±0.011 | 0.446 | 0.077±0.012 | emergent, real×virtual |
+| text↔RNA(virtual) | TCGA | 0.055±0.009 | 0.318 | 0.058±0.010 | emergent, real×virtual |
+| RNA(virtual)↔text(synthetic) | ORION 5-fold | 0.047±0.008 | 0.343 | 0.058±0.009 | ⚠ virtual×virtual |
+| H&E↔RNA(virtual) | ORION / TCGA | 0.103 / 0.120 | 0.52 / 0.45 | 0.19 / 0.18 | ⚠ |
+| H&E↔text(synthetic) | ORION 5-fold | 0.075±0.006 | 0.439 | 0.101±0.013 | ⚠ |
+| text↔mIF(virtual) | TCGA | 0.015±0.002 | 0.133 | 0.014 | GigaTIME not usable |
+
+Calibration (virtual replaces real on the same test items): HEST H&E↔RNA 0.148 → 0.126 (STPath-unseen 0.086 → 0.080);
+TCGA H&E↔text 0.197 → 0.166 (synthetic = 0.84× real); ORION H&E↔mIF 0.644 → 0.038 (GigaTIME fails at instance level).
+ORION official numbers (109 test tiles, 2 patients) match the 5-fold ones (mIF↔RNA(virtual) 0.108 / 0.152).
+
+### Why GigaTIME virtual mIF fails at pairing (`34_gigatime_diag.py`, `results/gigatime_diag*.csv`)
+- Per-channel tile-mean Spearman vs real ORION mIF: GigaTIME mean 0.54 pooled / 0.50 within-slide (not a slide-level
+  artefact), vs 0.79 / 0.75 for an in-domain reference (ridge UNI2 → real means, trained on ORION train, scored on test).
+  Weakest: SMA 0.20 (no SMA channel; Actin-D/Transgelin proxy), PD-L1 0.34, CD20 0.36, CD8 0.37.
+- Retrieval needs every channel to be right at once: real means ↔ GigaTIME means R@1 0.035 (chance 0.01) vs
+  real ↔ UNI2-ridge 0.261 — same 10 channels, no learned head.
+- Pixel maps are sigmoid probabilities, mostly < 0.05 (58–96% of pixels per channel) with ~0 confident positives except
+  DAPI/CK; pixel Pearson ≈0.1. The real-trained VirTues head sees out-of-distribution input → 0.038.
+- Conclusion: H&E does contain this information (the in-domain reference recovers it); GigaTIME, trained on another
+  cohort/panel/scanner, does not transfer to ORION at tile resolution.
+
+### Unified mIF channels (real and virtual mIF both = the 10-channel panel shared with GigaTIME)
+`33_final_eval.py` now uses the 10-ch head (real ORION, fold-k train/val) for every mIF row; 17-ch only as reference.
+`[qm]` = GigaTIME pixels quantile-mapped per channel onto the real fold-k TRAIN pixel distribution before the same
+preprocessing (`30_virtues_shared.py virtual_qm`). ORION patient 5-fold test, R@1 / R@10 / pair acc:
+
+| pair | raw virtual | [qm] virtual |
+|---|---|---|
+| H&E↔mIF (real, 10 ch) | 0.644 / 0.979 / 0.863 | — |
+| H&E↔mIF (real, 17 ch, reference) | 0.726 / 0.989 / 0.935 | — |
+| mIF(10 ch)↔RNA(virtual) | 0.098 / 0.495 / 0.131 | — |
+| mIF(10 ch)↔text(synthetic) | 0.074 / 0.434 / 0.071 | — |
+| H&E↔mIF(virtual) | 0.038 / 0.264 / 0.046 | 0.042 / 0.278 / 0.047 |
+| mIF↔mIF(virtual) | 0.045 / 0.248 / 0.054 | 0.052 / 0.301 / 0.060 |
+| mIF(virtual)↔RNA(virtual) | 0.021 / 0.174 / 0.019 | 0.017 / 0.167 / 0.016 |
+| mIF(virtual)↔text(synthetic) | 0.024 / 0.190 / 0.025 | 0.022 / 0.182 / 0.021 |
+| TCGA text↔mIF(virtual) | 0.015 / 0.133 / 0.014 | 0.016 / 0.138 / 0.016 |
+
+Unifying channels costs the real pairs ~0.08 R@1 (17 → 10 ch); matching the pixel distribution barely helps virtual mIF
+(+0.004–0.007 R@1). The bottleneck is GigaTIME's per-tile accuracy, not channel naming or intensity scale.
+
+## MIPHEI-ViT virtual mIF (replaces GigaTIME) — `35_miphei_infer.py`, `37_miphei_diag.py`
+Model: Sanofi MIPHEI-ViT v1.0.0 (H-optimus-0 + LoRA + conv decoder; non-commercial academic licence), 16 ORION
+channels (all but PD-1). It was trained on this ORION tile set with the official split (37/2/2 slides = our
+`orion_official`), so only the official TEST tiles (CRC02, CRC11; 109 tiles) are clean for ORION; TCGA is fully clean.
+Input 0.5 µm/px tiles reflect-padded to 512, output mapped back to the 8-bit ORION scale.
+
+Tile-mean Spearman vs the real mIF (mean over channels):
+
+| set | MIPHEI (16 ch) | MIPHEI (10 GigaTIME ch) | GigaTIME (10 ch) |
+|---|---|---|---|
+| clean (official test), pooled / within-slide | 0.697 / 0.654 | 0.690 / 0.645 | 0.249 / 0.301 |
+| seen (official train) | 0.840 / 0.791 | 0.840 / 0.787 | 0.545 / 0.505 |
+
+Direct real-mean ↔ virtual-mean retrieval (no head), R@1 in 100-galleries, clean test: MIPHEI 16 ch 0.272,
+MIPHEI 10 ch 0.239, GigaTIME 10 ch 0.017 (seen tiles: 0.538 / 0.433 / 0.036).
+
+### MIPHEI virtual mIF in the final protocol (`33_final_eval.py`, 16-ch heads trained on REAL ORION)
+Clean = ORION official test (109 tiles, MIPHEI never saw them) and TCGA. R@1 / R@10 / pair acc:
+
+| pair | ORION official test (clean) | ORION 5-fold (MIPHEI-seen, optimistic) | TCGA (clean) |
+|---|---|---|---|
+| H&E↔mIF real (16 ch) | 0.591 / 0.991 / 0.921 | 0.689 / 0.984 / 0.901 | — |
+| H&E↔mIF(MIPHEI) | 0.189 / 0.784 / 0.553 | 0.255 / 0.722 / 0.509 | 0.133 / 0.455 / 0.213 |
+| mIF real↔mIF(MIPHEI) | 0.297 / 0.800 / 0.586 | 0.379 / 0.762 / 0.669 | — |
+| text↔mIF(MIPHEI) | — | — | 0.033 / 0.231 / 0.036 |
+| RNA(virtual)↔mIF(MIPHEI) | 0.041 / 0.311 / 0.078 | 0.057 / 0.352 / 0.093 | 0.035 / 0.218 / 0.040 |
+| mIF(MIPHEI)↔text(synthetic) | 0.050 / 0.396 / 0.051 | 0.049 / 0.314 / 0.058 | — |
+| (real) mIF 16 ch↔RNA(virtual) | 0.101 / 0.466 / 0.123 | 0.108 / 0.512 / 0.149 | — |
+
+GigaTIME on the same pairs: H&E↔mIF(virtual) 0.012–0.038 R@1, text↔mIF(virtual) 0.015 → MIPHEI is 5–15× better and
+usable for validation (calibration: pair acc 0.55 vs 0.92 real on clean tiles ≈ 0.6×).
+
+## External test: DLPFC (`36_external_prep.py`, `38_external_eval.py`, `39_external_check.py`, `results/external_*.csv`)
+IDC results were removed from this report on 2026-10-18 at the user's request (the prepared data stays in cache/external).
+- DLPFC: spatialLIBD Visium brain, 482 spots / 12 sections / 3 donors; gene names recovered exactly from the public h5.
+  All 12 sections are in HEST (MISC1–12) -> the whole spatialLIBD study is REMOVED from HEST training for this test.
+- Heads: H&E(UNI2)↔RNA trained on real HEST only (train/val by group; 5 heads), RNA z-scored per dataset.
+  Virtual mIF: MIPHEI of the DLPFC H&E through the 16-ch heads trained on real ORION.
+  Synthetic text: Azure `gpt-5.6-sol` captions of the same patches (same 3-shot prompt) through the TCGA real-pair text
+  heads of 33_ (clean-text heads, runs/text_clean). Nothing is trained on external or virtual data.
+
+5 heads, R@1 ± std / R@10 / pair acc; mixed galleries G=100 (chance 0.01) and within-section galleries G=35 (chance 0.029):
+
+| pair | kind | mixed G=100 | within section G=35 |
+|---|---|---|---|
+| H&E↔RNA | real, trained pair | 0.010±0.003 / 0.107 / 0.013 | 0.031 |
+| RNA↔mIF(MIPHEI) | emergent, real×virtual | 0.008 / 0.099 / 0.009 | 0.027 |
+| RNA↔text(synthetic) | emergent, real×virtual | 0.010±0.002 / 0.117 / 0.011 | 0.032 |
+| H&E↔mIF(MIPHEI) | ⚠ circular | 0.017 / 0.136 / 0.015 | 0.040 |
+| H&E↔text(synthetic) | ⚠ circular | 0.022±0.002 / 0.179 / 0.025 | 0.054 |
+| mIF(MIPHEI)↔text(synthetic) | ⚠ virtual×virtual | 0.014 / 0.144 / 0.017 | 0.038 |
+
+- Every non-circular pair is at chance on DLPFC. Reference: HEST internal test restricted to brain = 0.021 (mixed organs 0.148).
+- Caption audit: 59% of DLPFC captions say "lymphoid", only 6% recognise CNS/neuropil, 13% "no identifiable tissue"
+  (blurry 112-µm crops) -> GPT cannot read these patches either.
+- Checks: captions aligned row by row (482/482); shuffled nulls ≈ 1/G; rerun identical. Protocol check: our UNI2
+  features of 3 DLPFC spots that also exist in the HEST cache have cosine 0.52–0.83 to the cached ones (random 0.12).
+
+## Text cleaning + retrain on clean real text (v10) — `40_clean_text.py`, `33_`, `run_v10.sh`
+Supersedes every TCGA-text number above (old results kept in results/before_text_clean/; old heads runs/text_all).
+- Junk texts removed from train / val / test: 365 of 6,480 (353 bare answer letters "A A B B", 4 "error error",
+  8 answers < 5 words). 256 of 6,144 TCGA images were left without real text; they keep their split, get a few-shot GPT
+  caption (27_ --sets tcga_notext, USD 0.81; cumulative USD 65.09 of 100) and are used for evaluation only.
+- Text heads retrained on clean real pairs only (runs/text_clean/uni2_biomedclip_fold<k>; ~4,400 train pairs / fold).
+
+R@1 / R@10 / pair acc, test split, mean of 5 folds (before → after cleaning):
+
+| pair | before | after |
+|---|---|---|
+| TCGA H&E↔text (real) | 0.229 / 0.700 / 0.238 | **0.237±0.012 / 0.730 / 0.248** |
+| TCGA text↔RNA(virtual) | 0.055 / 0.318 / 0.058 | 0.056 / 0.319 / 0.057 |
+| TCGA text↔mIF(MIPHEI) | 0.033 / 0.231 / 0.036 | 0.036 / 0.237 / 0.038 |
+| calibration set: H&E↔text real vs synthetic | 0.197 vs 0.166 | 0.219 vs 0.162 |
+| ORION 5-fold mIF↔text(synthetic) | 0.074 / 0.434 / 0.071 | 0.074 / 0.437 / 0.072 |
+| ORION 5-fold H&E↔text(synthetic) | 0.075 / 0.439 / 0.101 | 0.076 / 0.441 / 0.102 |
+
+Part of the gain on TCGA comes from removing junk queries from the test split (n test pairs 1229 → 1178).
+Images without real text (256, pooled over test folds, each image through its own test-fold head):
+H&E↔text(synthetic) 0.150 / 0.557 / 0.160 (⚠ circular; consistent with 0.162 on the calibration set);
+text(synthetic)↔RNA(virtual) 0.051; text(synthetic)↔mIF(MIPHEI) 0.036 (⚠ virtual×virtual).
+
+## Best encoder per modality — re-check under the final protocol (`41_text_enc_clean.py`, paired per-fold tests)
+mIF (17_bench_fm / 28_, ORION patient_5fold) and RNA (23_, HEST group 5-fold) comparisons already used the final-protocol
+splits -> re-analysed per fold. Text was re-run on CLEAN real text with the 33_ recipe and splits (UNI2+BiomedCLIP
+reproduces 33_ fold by fold: 0.234 / 0.236 / 0.248 / 0.219 / 0.248). R@1 g100 mean of 5 folds; Δ = best − other,
+folds won, paired t-test (5 folds; 5/5 wins = sign-test p 0.06).
+
+| modality | best | runner-up (Δ, folds won, p) | verdict |
+|---|---|---|---|
+| text tower (H&E↔text, clean) | **BiomedCLIP** 0.237 (UNI2 image) | CONCH text 0.216 (+0.021, 5/5, p=0.03); PLIP 0.211 (+0.026, 5/5, p=0.007); ImageBind 0.201 (+0.036, 5/5, p=0.002) | clear winner for every image anchor |
+| H&E image for text | Phikon-v2 0.241 ≈ CONCH 0.241 ≈ UNI2 0.237 | GigaPath 0.223 (UNI2 +0.014, 4/5, p=0.05) | 3-way tie, GigaPath worse |
+| H&E image for mIF | **UNI2** 0.727 (VirTues) / 0.753 (ConvNeXt) | GigaPath (+0.009 / +0.013, ns); CONCH (+0.155 / +0.114, 5/5, p<0.001) | UNI2 ≈ GigaPath ≫ CONCH |
+| H&E image for RNA | **UNI2** 0.148 | GigaPath 0.113 (+0.035, 4/5, p=0.05) | UNI2 |
+| mIF encoder (H&E↔mIF) | **ConvNeXt-T** scratch 0.758 | VirTues 0.728 (+0.030, 5/5, p=0.07) | ConvNeXt-T consistently ahead; VirTues kept for cross-modal transfer / channel flexibility |
+| RNA encoder (H&E↔RNA) | Geneformer-V2 0.159 | scGPT 0.154 (+0.005, 3/5, p=0.36); MLP 0.148 (+0.011, 4/5, p=0.05) | tie |
+
+UNI2 is the only H&E anchor never significantly beaten on any pair (Phikon-v2 has no mIF/RNA runs: HEST ships only
+UNI2/GigaPath features). Files: results/text_enc_clean_{long,summary}.csv.
+
+## Training regimes: separate vs joint (`42_train_regimes.py`, results/regimes_{long,summary}.csv)
+Real pairs only, final-protocol splits (ORION patient 5-fold with the 16-ch VirTues panel, HEST group 5-fold, TCGA
+patient 5-fold on clean text). Same heads, loss and optimiser for every regime; virtual data only at test time.
+R@1 g100, mean of 5 folds; "rel" = mean of R@1 / S1's R@1 over the trained (3) or emergent (4) pairs.
+
+| regime | H&E↔mIF | H&E↔RNA | H&E↔text | mIF↔RNA(v) | mIF↔text(s) | text↔RNA(v) | text↔mIF(MIPHEI) | trained rel | emergent rel |
+|---|---|---|---|---|---|---|---|---|---|
+| S1 separate, frozen anchor (current) | 0.680 | 0.148 | 0.234 | 0.108 | 0.076 | 0.060 | 0.038 | 1.00 | 1.00 |
+| S2 separate, two-tower (CLIP) | **0.794** | 0.130 | 0.195 | 0.019 | 0.022 | 0.014 | 0.013 | 0.96 | 0.26 |
+| S3 separate, frozen, linear heads | 0.642 | 0.156 | 0.237 | **0.123** | 0.070 | 0.069 | 0.038 | 1.00 | 1.05 |
+| S4 separate, frozen, MLP mIF + linear RNA/text | 0.680 | 0.152 | 0.234 | 0.117 | 0.074 | **0.071** | 0.037 | 1.01 | 1.05 |
+| J1 joint, frozen anchor | 0.706 | 0.148 | **0.237** | 0.094 | 0.065 | 0.056 | 0.031 | 1.02 | 0.87 |
+| J2 joint, shared trainable H&E head | 0.761 | 0.147 | 0.204 | 0.061 | 0.060 | 0.044 | 0.031 | 0.99 | 0.72 |
+| J3 J2, size-proportional sampling | 0.740 | 0.138 | 0.187 | 0.063 | 0.060 | 0.039 | 0.031 | 0.94 | 0.70 |
+| J4 J2 + anchor distillation λ=1 | 0.765 | 0.144 | 0.202 | 0.062 | 0.058 | 0.043 | 0.030 | 0.99 | 0.71 |
+| J4b J2 + anchor distillation λ=10 | 0.768 | 0.149 | 0.209 | 0.067 | 0.058 | 0.047 | 0.030 | 1.01 | 0.74 |
+| J5 joint, residual adapter on UNI2 | 0.762 | 0.142 | 0.201 | 0.058 | 0.059 | 0.044 | 0.029 | 0.98 | 0.71 |
+| J6 J2 + uncertainty weighting | 0.765 | 0.136 | 0.195 | 0.057 | 0.055 | 0.041 | 0.027 | 0.96 | 0.66 |
+| J7 two-stage: S1 → joint residual fine-tune | 0.776 | 0.150 | 0.234 | 0.100 | 0.076 | 0.055 | 0.035 | 1.05 | 0.94 |
+| **J8 two-stage: S4 → joint residual fine-tune** | 0.762 | **0.164** | 0.232 | 0.112 | 0.073 | 0.068 | 0.035 | **1.07** | **1.01** |
+
+Pair accuracy (H&E↔mIF / RNA / text): S1 0.897 / 0.172 / 0.242 → J8 0.929 / 0.198 / 0.237.
+J8 vs S1 (paired, 5 folds): H&E↔mIF +0.083 (5/5, p=0.01), H&E↔RNA +0.015 (5/5, p=0.03), text↔RNA(v) +0.008
+(5/5, p=0.02), H&E↔text −0.002, mIF↔RNA(v) +0.004, mIF↔text(s) −0.004, text↔mIF −0.003 (all ns).
+- Training the H&E side from scratch (S2, J2–J6) helps H&E↔mIF (+0.08–0.11) but costs H&E↔text (−0.03 to −0.05) and
+  26–74% of the emergent performance: the shared H&E head is pulled towards the easiest / strongest pair (mIF).
+  Size-proportional sampling and uncertainty weighting make this worse; distillation to the UNI2 geometry helps a little.
+- Joint training with a frozen anchor (J1) only changes the step budget: +0.026 mIF, −13% emergent.
+- Linear heads for RNA / text generalise better across modalities than MLP heads (S3/S4: emergent +5%).
+- Best: **J8 = first train every head separately on the frozen UNI2 anchor (MLP for mIF, linear for RNA and text), then
+  fine-tune all heads jointly with a shared zero-initialised residual adapter on UNI2** (lr 1e-4, 1000 steps, sum of
+  the three losses, similarity distillation λ=1, joint early stopping on the geometric mean of the val R@1).
+
+## More RNA / text encoders + multi-encoder late fusion (`43_*`, `44_rnafm_eval.py`, `45_*`, `46_text_enc_eval.py`, `47_final_ens.py`)
+All encoders frozen; one head per encoder into the frozen UNI2 space (S4 recipe of 42_: linear for RNA / text), same
+splits and test items as 42_. Late fusion = mean of the L2-normalised head outputs of several encoders of one modality
+(possible because every head lands in the same frozen space). R@1 g100, mean of 5 folds; virtual / synthetic = test only.
+RNA inputs: cache/rnafm/<model>_<set>.npy (rna_sets.py: hest real, STPath virtual = round(expm1(pred)), DLPFC real);
+FM inputs z-scored with HEST-train stats (DLPFC: per dataset, heads retrained without spatialLIBD).
+
+RNA encoders (linear head):
+
+| RNA encoder | H&E↔RNA | mIF↔RNA(v) | text↔RNA(v) | H&E↔RNA(v) (HEST, calib) | DLPFC H&E↔RNA |
+|---|---|---|---|---|---|
+| hvg 2000 (current) | 0.155 | 0.124 | 0.069 | 0.198 | 0.009 |
+| AIDO.Cell-100M (MLP head 0.171) | 0.169 | 0.094 | 0.049 | 0.083 | 0.010 |
+| Tahoe-x1 70M | 0.166 | 0.087 | 0.039 | 0.070 | 0.011 |
+| Geneformer-V2-104M | 0.164 | 0.071 | 0.037 | 0.064 | 0.010 |
+| scGPT-human | 0.155 | 0.087 | 0.050 | 0.096 | 0.013 |
+| C2S-Scale-Gemma-2-2B (mean / last token) | 0.155 / 0.142 | 0.101 / 0.101 | 0.055 / 0.051 | 0.111 / 0.097 | 0.011 |
+| Nicheformer | 0.152 | 0.104 | 0.056 | 0.097 | 0.006 |
+| SE-600M (STATE) | 0.150 | 0.117 | 0.062 | 0.143 | 0.011 |
+| GenePT-w (bge / BiomedCLIP gene-summary embeddings) | 0.141 / 0.130 | 0.093 / 0.075 | 0.056 / 0.052 | 0.116 / 0.086 | 0.009 |
+| seed control: hvg x 5 seeds, averaged | 0.161 | 0.132 | 0.073 | 0.208 | 0.008 |
+| fusion hvg + Tahoe-x1 + SE-600M + AIDO.Cell | 0.192 | 0.142 | 0.075 | 0.174 | 0.012 |
+| fusion all 10 | 0.199 | 0.148 | 0.078 | 0.163 | 0.012 |
+| early fusion (concat) hvg + Tahoe-x1 | 0.169 | 0.114 | 0.059 | 0.131 | 0.009 |
+
+- Single FMs gain up to +0.016 on real H&E↔RNA but lose on every virtual-RNA pair; the calibration column shows why:
+  the FMs embed STPath virtual RNA far from the real RNA of the same spot (H&E↔RNA(v) 0.06–0.14 vs 0.198), so the
+  emergent drop reflects the virtual test data more than the encoders.
+- Late fusion beats early fusion (concat) and the seed-ensemble control by a wide margin -> the encoders are complementary.
+
+Text encoders (TCGA clean real text, linear head; 45_ / 45b_; KEEP and Quilt-1M corpora contain no PathGen / TCGA text):
+
+| text encoder | H&E↔text | mIF↔text(s) | text↔RNA(v) | text↔mIF(MIPHEI) |
+|---|---|---|---|---|
+| BiomedCLIP (current; z-scored control 0.235) | 0.236 | 0.077 | 0.069 | 0.038 |
+| KEEP (MLP head 0.267) | 0.278 | 0.077 | 0.084 | 0.041 |
+| Qwen3-Embedding-0.6B / 4B | 0.264 / 0.258 | 0.062 / 0.062 | 0.075 / 0.067 | 0.038 / 0.033 |
+| PubMedBERT-emb / bge-large / BioLORD-2023 / MedCPT-Article / MedEmbed-large | 0.262 / 0.259 / 0.258 / 0.257 / 0.252 | 0.054–0.063 | 0.068–0.076 | 0.034–0.037 |
+| QuiltNet-B-32 (QuickGELU) | 0.218 | 0.058 | 0.069 | 0.035 |
+| seed control: KEEP x 5 seeds | 0.284 | 0.082 | 0.085 | 0.045 |
+| fusion KEEP + BiomedCLIP + Qwen3-0.6B | 0.299 | 0.091 | 0.088 | 0.047 |
+| fusion + PubMedBERT-emb + MedCPT | 0.303 | 0.088 | 0.091 | 0.047 |
+
+Final combination (47_final_ens.py; mIF = VirTues 16-ch MLP head in all; paired vs S4 over 5 folds):
+
+| pair | S4 | AIDO.Cell + KEEP | ens-small (4 RNA + 3 text) | ens-large (10 RNA + 5 text) | ens-small vs S4 |
+|---|---|---|---|---|---|
+| H&E↔mIF | 0.677 | 0.677 | 0.677 | 0.677 | unchanged |
+| H&E↔RNA | 0.155 | 0.169 | 0.192 | 0.199 | +0.037 (5/5, p=0.0006) |
+| H&E↔text | 0.236 | 0.278 | 0.299 | 0.303 | +0.062 (5/5, p<0.0001) |
+| mIF↔RNA(v) | 0.124 | 0.094 | 0.142 | 0.148 | +0.018 (5/5, p=0.0002) |
+| mIF↔text(s) | 0.077 | 0.077 | 0.091 | 0.088 | +0.014 (5/5, p=0.011) |
+| text↔RNA(v) | 0.069 | 0.052 | 0.095 | 0.098 | +0.027 (5/5, p=0.0009) |
+| text↔mIF(MIPHEI) | 0.038 | 0.041 | 0.047 | 0.047 | +0.010 (5/5, p=0.008) |
+| DLPFC H&E↔RNA / RNA↔text(s) / RNA↔mIF(v) | 0.009 / 0.012 / 0.011 | 0.010 / 0.010 / 0.008 | 0.012 / 0.015 / 0.009 | 0.012 / 0.011 / 0.006 | still chance |
+
+Pair accuracy ens-small vs S4: H&E↔RNA 0.182→0.246, H&E↔text 0.245→0.324, mIF↔RNA(v) 0.191→0.198, text↔RNA(v) 0.077→0.100.
+Recommendation: ens-small (nearly all of ens-large's gain at 4 + 3 heads). Licences: AIDO.Cell (GenBio community licence,
+non-commercial; restricts using its embeddings to train "similar" models) and SE-600M (Arc non-commercial) are
+research-only; KEEP / QuiltNet / Nicheformer MIT, Tahoe-x1 / Qwen3-Embedding Apache-2.0, C2S-Scale CC-BY-4.0.
+Not yet tested: fusion inside the J8 two-stage regime (J8's mIF gain should stack, unverified).
+
+## Joint training of the fused model (`48_joint_ens.py`, results/joint_ens_{long,summary,tests}.csv)
+Does the J8 two-stage gain stack with multi-encoder fusion? Same splits / test items; all configs of a fold start from
+the SAME stage-1 heads (separate, frozen UNI2). Stage 2 = J8 recipe (shared zero-init residual adapter on UNI2, sum of
+the 3 pair losses, similarity distillation λ=1, lr 1e-4, 1000 steps, joint early stopping). E = ens-small of 47_.
+EJ-fused: loss on the fused modality embedding; EJ-each: each encoder head keeps its own loss (fusion only at test);
+EJ-frz: EJ-each with RNA / text heads frozen (only adapter + mIF head + temperatures train). R@1 g100, 5 folds.
+
+| config | H&E↔mIF | H&E↔RNA | H&E↔text | mIF↔RNA(v) | mIF↔text(s) | text↔RNA(v) | text↔mIF(MIPHEI) |
+|---|---|---|---|---|---|---|---|
+| S4 separate, single encoders | 0.675 | 0.155 | 0.236 | 0.122 | 0.076 | 0.069 | 0.038 |
+| J8 joint, single encoders | 0.776 | 0.159 | 0.238 | 0.122 | 0.073 | 0.065 | 0.035 |
+| E separate, fused encoders | 0.675 | 0.192 | 0.299 | **0.141** | **0.089** | **0.095** | **0.046** |
+| EJ-fused | 0.792 | 0.178 | 0.292 | 0.118 | 0.080 | 0.060 | 0.035 |
+| EJ-each | 0.790 | **0.202** | **0.304** | 0.128 | 0.086 | 0.086 | 0.039 |
+| **EJ-frz** | **0.793** | 0.201 | **0.304** | 0.128 | 0.088 | **0.095** | 0.040 |
+
+Pair accuracy H&E↔mIF / RNA / text: S4 0.899 / 0.182 / 0.245, E 0.899 / 0.246 / 0.324, EJ-frz 0.936 / 0.255 / 0.320.
+- Joint stage-2 gains stack on the trained pairs: EJ-frz vs E H&E↔mIF +0.118 (5/5, p<0.001), H&E↔RNA +0.009 (5/5,
+  p=0.01), H&E↔text +0.005 (ns); vs S4 all three trained pairs +0.05–0.12 (5/5).
+- Price: the pairs that involve the fine-tuned mIF head lose vs E: mIF↔RNA(v) −0.013 (0/5, p=0.04), text↔mIF −0.007
+  (0/5, p=0.02); mIF↔text(s) −0.001 (ns). text↔RNA(v) unchanged (both heads frozen).
+- Loss on the fused embedding (EJ-fused) is clearly worse than per-encoder losses (EJ-each: 5/5 better on 6 pairs):
+  the fused loss lets single heads specialise, which breaks the cross-modal geometry (text↔RNA 0.095 → 0.060).
+- Freezing RNA / text heads (EJ-frz) costs nothing on the trained pairs and keeps text↔RNA at 0.095 (vs EJ-each 0.086,
+  5/5, p=0.03): the adapter alone does the H&E-side work.
+- Recommendation: EJ-frz if H&E↔mIF matters (+0.12) and a −0.01 on the virtual mIF emergent pairs is acceptable;
+  E (pure separate) if the emergent mIF pairs are the priority.
+
+## MedVTok as H&E image information + adapter-only joint training (`49_`–`51_`, results/medvtok_{long,summary,tests}.csv)
+MedVTok (arXiv 2610.35842; github Masaaki-75/medvtok @a558c24, HF massaki75/medvtok, MIT): general medical visual
+tokenizer, SigLIP2-So400m-initialised encoder (1152-d), aligned (MERA) to frozen expert encoders, CONCH v1.5 for
+histopathology; no image-text pairs. Pathology training images: HISTAI 4.0M, **PathGen 1.61M**, bone-marrow
+cytomorphology, LYSTO. => it has seen the TCGA/PathGen test IMAGES (not their captions); ORION / HEST / DLPFC not listed.
+Loaded with weights_only=True + strict=True; reconstruction PSNR on ORION H&E 32–42 dB; feature = mean-pooled tokens ->
+LN -> Linear ("feats", 1152-d), 256 px, [-1, 1]. HEST images: 224-px level-0 crops from the HEST WSIs (`49_`, HTTP range
+reads of only the needed TIFF tiles); re-encoding them with UNI2-h reproduces cache/hest/he_uni2.npy (cos ≥ 0.9994 for
+all 9,484 spots), i.e. MedVTok sees exactly the anchor's images. DLPFC: the 0.5-µm crops behind dlpfc_he_uni2 (cos 1.000).
+Configs (RNA / text ens-small, mIF VirTues MLP; same splits / test items; R@1 g100, 5 folds):
+E separate UNI2 anchor; EJ-frz (= 48_); EJ-ad = EJ-frz with the mIF head frozen too (only the residual H&E adapter and
+the temperatures train); MV-E = MedVTok instead of UNI2 as the frozen anchor; +MV = adapter input [UNI2, c·MedVTok];
++RP = control, adapter input [UNI2, random projection of UNI2 to 1152-d at the same norm] (wider adapter, no new info).
+
+| config | H&E↔mIF | H&E↔RNA | H&E↔text | mIF↔RNA(v) | mIF↔text(s) | text↔RNA(v) | text↔mIF | DLPFC H&E↔RNA |
+|---|---|---|---|---|---|---|---|---|
+| E | 0.675 | 0.192 | 0.299 | 0.141 | 0.089 | 0.095 | 0.046 | 0.012 |
+| EJ-frz | 0.786 | 0.202 | 0.305 | 0.134 | 0.087 | 0.095 | 0.042 | |
+| EJ-ad | 0.776 | 0.201 | 0.299 | 0.141 | 0.089 | 0.095 | 0.046 | |
+| MV-E (MedVTok anchor) | 0.658 | 0.131 | 0.293 | 0.081 | **0.113** | 0.056 | **0.052** | 0.012 |
+| EJ-frz+MV | **0.811** | 0.200 | 0.307 | 0.125 | 0.084 | 0.095 | 0.041 | |
+| **EJ-ad+MV** | 0.792 | 0.200 | **0.315** | 0.141 | 0.089 | 0.095 | 0.046 | |
+| EJ-ad+RP (control) | 0.778 | 0.202 | 0.298 | 0.141 | 0.089 | 0.095 | 0.046 | |
+
+Pair accuracy H&E↔mIF / RNA / text: E 0.899 / 0.246 / 0.324, EJ-ad 0.929 / 0.256 / 0.313, EJ-ad+MV 0.937 / 0.252 / 0.338.
+- EJ-ad (only the H&E adapter moves): every emergent pair is identical to E by construction (all modality heads frozen)
+  while H&E↔mIF +0.101 (5/5, p<0.001) and H&E↔RNA +0.009 (5/5, p=0.01). vs EJ-frz: H&E↔mIF −0.010 (ns), H&E↔text −0.006
+  (0/5, p=0.04), but mIF↔RNA(v) +0.007 (5/5, p=0.045) and text↔mIF +0.005 (ns) -> the emergent losses of EJ-frz came
+  entirely from moving the mIF head.
+- MedVTok as the anchor (MV-E) is worse: H&E↔RNA −0.061 (0/5), RNA emergent pairs −0.04/−0.06 (0/5); only the pairs
+  with synthetic GPT text / virtual mIF gain (mIF↔text(s) +0.024, text↔mIF +0.006, 5/5) - CONCH-v1.5 semantics.
+  DLPFC stays at chance (0.012) with either anchor.
+- MedVTok as extra adapter input: EJ-ad+MV vs EJ-ad H&E↔mIF +0.015 (4/5, p=0.04), H&E↔text +0.016 (5/5, p=0.02),
+  H&E↔RNA −0.002 (ns), emergent unchanged. The control shows it is MedVTok's information, not adapter width:
+  EJ-ad+RP = EJ-ad (all |Δ| ≤ 0.002, ns); EJ-ad+MV vs EJ-ad+RP: H&E↔mIF +0.014 (5/5, p=0.03), H&E↔text +0.017 (5/5, p=0.01).
+  Caveat: the H&E↔text test images are PathGen images MedVTok was pretrained on (image-only, aligned to CONCH v1.5);
+  the ORION H&E↔mIF gain is free of that.
+- EJ-frz+MV has the best H&E↔mIF (0.811) but again loses the mIF emergent pairs (mIF↔RNA(v) −0.017 vs E, 0/5).
+- Recommendation: **EJ-ad+MV** (stage 1 separate heads on frozen UNI2; stage 2 trains only a residual H&E adapter that
+  reads UNI2 + MedVTok). Every non-H&E embedding is unchanged; H&E↔mIF / text improve. Cost: a 2nd image encoder
+  (MedVTok encoder 417M params) at H&E inference. EJ-ad (no MedVTok) if a single image encoder is required.
+
+## Final model: training, stage-2 selection rule, inference API (`52_final_model.py`, `53_embed_check.py`, `embed.py`)
+Configuration EJ-ad+MV (previous section). Stage 1 is deterministic (52_ reproduces 51_'s stage-1 heads exactly: the
+four emergent pairs of fold 0 are identical to 51_). Stage 2 is not: two identical fold-0 runs gave H&E↔mIF 0.791 and
+0.741, because the val geo-mean is flat after ~100 steps and the argmax picks very different steps (100–700).
+`52_ cv` (5 folds × 3 stage-2 seeds, same stage-1 heads, same test items; results/final_cv_{long,summary,tests,curves}.csv):
+
+| stage-2 rule | H&E↔mIF | H&E↔RNA | H&E↔text | geo-mean | pair acc mIF / RNA / text | seed std mIF / text |
+|---|---|---|---|---|---|---|
+| stage 1 only (= E) | 0.675 | 0.192 | 0.299 | 0.337 | 0.899 / 0.246 / 0.324 | – |
+| es (51_ rule, 1 seed) | 0.794 | 0.201 | 0.309 | 0.366 | 0.937 / 0.253 / 0.329 | 0.009 / 0.011 |
+| last (1000 steps, 1 seed) | 0.802 | 0.202 | 0.301 | 0.364 | 0.939 / 0.254 / 0.314 | 0.002 / 0.003 |
+| **ens-es (mean of 3 early-stopped adapters)** | 0.801 | 0.203 | **0.323** | **0.373** | **0.942 / 0.258 / 0.347** | ensemble |
+| ens-last | **0.807** | **0.204** | 0.309 | 0.370 | 0.941 / 0.255 / 0.322 | ensemble |
+
+ens-es vs es: H&E↔text +0.015 (5/5, p=0.02), mIF +0.007, RNA +0.002 (ns); ens-es vs ens-last: text +0.014 (5/5,
+p=0.004), mIF −0.006 (0/5, p=0.03). Emergent pairs are identical for every rule (heads frozen): mIF↔RNA(v) 0.141,
+mIF↔text(s) 0.089, text↔RNA(v) 0.095, text↔mIF 0.046. -> **ens-es** is the final rule (3 adapters, H&E = UNI2 +
+mean_s A_s([UNI2, c·MedVTok]); each adapter 2.4M parameters).
+Final 5-fold test performance (R@1 / R@10 / pair acc): H&E↔mIF 0.801±0.043 / 0.991 / 0.942, H&E↔RNA 0.203±0.033 /
+0.625 / 0.258, H&E↔text 0.323±0.007 / 0.813 / 0.347; emergent as above (chance 0.01 / 0.10 / 0.01).
+
+Saved models (`52_ fold0` / `52_ release`, rule ens-es):
+- `models/ej_ad_mv_fold0/` fold-0 train / val, test held out: test R@1 mIF 0.783, RNA 0.192, text 0.320 (51_ fold 0:
+  0.768 / 0.187 / 0.316; emergent identical).
+- `models/ej_ad_mv/` **release**: train = fold-0 train + test (ORION 3000 tiles, HEST 8938 spots, TCGA 5615 texts),
+  val = fold-0 val (187 / 546 / 500) for early stopping. No own test split: expected performance = the 5-fold CV above.
+  model.pt (heads, 3 adapters, feature-normalisation statistics) + meta.json (dims, HVG list, split sizes, val scores).
+
+`embed.py` (Embedder): raw-input embedding of all four modalities (H&E images → UNI2-h + MedVTok; text → KEEP +
+BiomedCLIP + Qwen3-Emb-0.6B; RNA counts + gene symbols → hvg + Tahoe-x1 + SE-600M + AIDO.Cell; mIF tiles (16 ORION
+markers) + slide ids → VirTues), or from precomputed features; output L2-normalised 1536-d vectors. The raw-input
+pre-processing reuses the extraction code of 04b_/14_/43_/45_/45b_/50_. `53_embed_check.py` (results/embed_check.csv):
+raw-input features vs cached training features, per-item cosine median (min): UNI2 1.000 (1.000) on ORION / PathGen /
+HEST; MedVTok 1.000 (≥0.9997); KEEP 1.000, BiomedCLIP 1.000 (0.929, bf16 batch effects), Qwen3 0.9998; hvg 1.000,
+AIDO.Cell 1.000, Tahoe-x1 0.991 (0.929) and SE-600M 0.994 (0.939) (both sample / tie-break genes at random, as in their
+official collators); VirTues 1.000 (all 3187 ORION tiles). End-to-end with models/ej_ad_mv_fold0 on held-out fold-0
+test items (300 each), g100 R@1 raw-input vs cached features: H&E↔mIF 0.792 / 0.792, H&E↔RNA 0.194 / 0.194,
+H&E↔text 0.335 / 0.336.
+
+## Release on the Hugging Face Hub and reproduction check (2026-10-28)
+
+All three repos are private:
+- Model, code and results: `Pathstudy/fourmodal-pathology-bind`. The README is the model card in Chinese: data, architecture, results and usage.
+- Features and splits (966 MB): `datasets/Pathstudy/fourmodal-pathology-features`.
+- Report: the Space `Pathstudy/imagebind-pathology-report`. Its `index.html` is the final report; the earlier report is kept as `previous_report.html`.
+
+New scripts:
+- `evaluate_saved.py`: feature-level re-evaluation of a saved checkpoint.
+- `example_check.py` + `examples/`: a raw-input check on 64 fold-0 test items per dataset, built by `54_make_examples.py`.
+- `reproduce.sh` and `compare_reproduction.py`: retrain from the features and compare with the shipped numbers.
+- `setup_encoders.sh`: clones VirTues, ModelGenerator and MedVTok at the pinned commits and fetches the MedVTok weights.
+- `requirements.txt`.
+
+Code changes:
+- `52_final_model.py --out NAME`, so a retraining does not overwrite the shipped checkpoints or results.
+- `embed.py`: `$UNI2_PATH`, falling back to the gated HF repo.
+- `43_rnafm_tahoex1.py`: the Geneformer name→ID dictionary is now loaded with `hf_hub_download` at a pinned revision. It used to be globbed from the local HF cache.
+
+Check run in a fresh directory, downloaded from the Hub, with a new uv virtualenv (Python 3.11, torch 2.7.1+cu126) on an RTX A5000:
+- **`evaluate_saved.py`:** equal to `test.csv` (max |diff| 8e-17).
+- **Fold-0 and 5-fold retraining:** every pair is identical to the shipped numbers, stage 1 and ens-es (|diff| 0).
+- **Release retraining:** the same early-stopping steps, [200, 200, 900].
+- **`example_check.py`:** all encoders reproduce the cached features. Cosine is 1.000 except Tahoe-x1 .991, SE-600M .995 and VirTues .999 (64-tile standardisation).
+- **R@1 within 64 items, raw inputs vs cached features:**
+
+  | Pair | Raw inputs | Cached features |
+  |---|---|---|
+  | H&E↔RNA | .227 | .227 |
+  | H&E↔text | .398 | .398 |
+  | H&E↔mIF | .828 | .875 |
+
+- **Bug found by the fresh run and fixed:** Tahoe-x1 raw input needed `cache/external/dlpfc_genes_full.csv`. That file now ships with the model repo.
+- **Not covered:** the frozen-encoder weights came from the existing HF cache on this machine, and UNI2-h from its local copy. A new machine downloads them from the Hub at the pinned revisions, and UNI2-h needs gated access.

biomedclip_text.py

@@ -0,0 +1,27 @@
+"""BiomedCLIP text tower without open_clip (not installed in this env): BiomedBERT + 2-layer MLP projection, CLS pooling,
+256 tokens, attention mask = token != pad (= open_clip HFTextEncoder with hf_proj_type mlp, cls_last_hidden_state_pooler).
+Weights: text.* of microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224/open_clip_pytorch_model.bin, strict load."""
+import torch
+import torch.nn as nn
+from huggingface_hub import hf_hub_download
+from transformers import AutoConfig, AutoTokenizer, BertModel
+
+REPO = "microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224"
+BERT = "microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract"
+
+
+class BiomedCLIPText(nn.Module):
+    def __init__(self):
+        super().__init__()
+        self.transformer = BertModel(AutoConfig.from_pretrained(BERT), add_pooling_layer=False)
+        self.proj = nn.Sequential(nn.Linear(768, 640, bias=False), nn.GELU(), nn.Linear(640, 512, bias=False))
+        sd = torch.load(hf_hub_download(REPO, "open_clip_pytorch_model.bin"), map_location="cpu", weights_only=True)
+        sd = {k[5:]: v for k, v in sd.items() if k.startswith("text.") and not k.endswith("position_ids")}
+        self.load_state_dict(sd, strict=True)
+        self.tok = AutoTokenizer.from_pretrained(BERT)
+
+    def forward(self, texts):
+        dev = next(self.parameters()).device
+        ids = self.tok(texts, return_tensors="pt", max_length=256, padding="max_length", truncation=True).input_ids.to(dev)
+        h = self.transformer(input_ids=ids, attention_mask=(ids != self.tok.pad_token_id).long()).last_hidden_state
+        return self.proj(h[:, 0])

common.py

@@ -0,0 +1,105 @@
+"""Shared paths / helpers for the ORION ImageBind-style 4-modality experiments.
+
+Modalities (all spatially paired per 333x333 tile at 20x):
+  he      : H&E RGB tile                      -> anchor, frozen ImageBind-Huge vision encoder
+  if      : 17-channel multiplex IF (uint8)   -> trainable encoder
+  nuclei  : nuclei instance mask (int32)      -> trainable encoder (binary mask + boundary channels)
+  markers : per-tile marker vector (17 *_prop + 16 *_count from the dataframes) -> trainable MLP
+"""
+import os
+import sys
+import types
+
+import numpy as np
+import pandas as pd
+
+ROOT = os.path.dirname(os.path.abspath(__file__))
+ORION = os.path.join(ROOT, "..", "ORION", "ORIONCRC_dataset_tile_20x")
+CACHE = os.path.join(ROOT, "cache")
+SPLITS = os.path.join(ROOT, "splits")
+RUNS = os.path.join(ROOT, "runs")
+IB_WEIGHTS = os.path.join(ROOT, "weights", "imagebind_huge.pth")
+IMG = 224
+
+MODALITIES = ["he", "if", "nuclei", "markers"]
+
+
+def load_all_tiles():
+    """All tiles across the official train/val/test csvs, with a stable global index."""
+    dfs = []
+    for s in ["train", "val", "test"]:
+        d = pd.read_csv(os.path.join(ORION, f"{s}_dataframe.csv"))
+        d["orion_split"] = s
+        dfs.append(d)
+    df = pd.concat(dfs, ignore_index=True)
+    slides = pd.read_csv(os.path.join(ORION, "slide_dataframe.csv"))
+    df = df.merge(slides[["in_slide_name", "orion_slide_id", "targ_slide_path"]], on="in_slide_name", how="left")
+    df["patient"] = df.orion_slide_id.str.split("_").str[0]          # CRC33_01 / CRC33_02 -> CRC33
+    df["if_batch"] = df.targ_slide_path.str.extract(r"_(C59[a-z]X)_")[0]  # IF acquisition batch
+    df["tile_id"] = np.arange(len(df))
+    return df
+
+
+def marker_columns(df):
+    prop = [c for c in df.columns if c.endswith("_prop")]
+    cnt = [c for c in df.columns if c.endswith("_count")]
+    return prop, cnt
+
+
+def marker_matrix(df):
+    prop, cnt = marker_columns(df)
+    return np.concatenate([df[prop].values, np.log1p(df[cnt].values)], 1).astype(np.float32)
+
+
+def dihedral(x, d):
+    """Apply one of 8 dihedral transforms to the last two (spatial) dims of a tensor/array."""
+    import torch
+    k, flip = d % 4, d // 4
+    if flip:
+        x = torch.flip(x, dims=(-1,))
+    return torch.rot90(x, k, dims=(-2, -1))
+
+
+def load_imagebind_vision(device="cuda"):
+    """Build ImageBind-Huge and load the official checkpoint without importing imagebind.data
+    (which needs torchaudio/pytorchvideo)."""
+    src = os.path.join(ROOT, "ImageBind_src", "imagebind")
+    if "imagebind" not in sys.modules:
+        pkg = types.ModuleType("imagebind")
+        pkg.__path__ = [src]
+        sys.modules["imagebind"] = pkg
+    import torch
+    from imagebind.models import imagebind_model
+    from imagebind.models.imagebind_model import ModalityType
+
+    model = imagebind_model.imagebind_huge(pretrained=False)
+    sd = torch.load(IB_WEIGHTS, map_location="cpu")
+    missing, unexpected = model.load_state_dict(sd, strict=True), None
+    model.eval().to(device)
+    return model, ModalityType
+
+
+def metrics(zq, zg, g100=True, eps=1e-6):
+    """Tie-aware retrieval metrics (expected rank under random tie-breaking; template texts can tie)."""
+    import torch
+    from scipy.optimize import linear_sum_assignment
+    sim = (zq @ zg.T).numpy()
+    n = len(sim)
+    d = sim[np.arange(n), np.arange(n)][:, None]
+    gt = (sim > d + eps).sum(1)
+    tie = (np.abs(sim - d) <= eps).sum(1) - 1  # other candidates tied with the true match
+
+    def rk(k):  # P(true match within top-k) with random tie-breaking
+        return np.clip((k - gt) / (tie + 1), 0, 1).mean()
+
+    r, c = linear_sum_assignment(-sim)
+    res = {"R@1": rk(1), "R@5": rk(5), "R@10": rk(10), "MRR": float(np.mean(1 / (gt + tie / 2 + 1))),
+           "pair_acc": float((c[np.argsort(r)] == np.arange(n)).mean()), "N": n}
+    if g100:
+        rng = np.random.default_rng(0)
+        if n <= 100:
+            res["R@1_g100"] = res["R@1"]
+        else:
+            res["R@1_g100"] = float(np.mean([metrics(zq[s], zg[s], g100=False)["R@1"]
+                                             for s in (torch.from_numpy(rng.choice(n, 100, replace=False)) for _ in range(50))]))
+    return res

compare_reproduction.py

@@ -0,0 +1,45 @@
+"""Compare a retraining (reproduce.sh) with the shipped results.
+
+    python -I compare_reproduction.py [TAG=repro]
+
+fold 0 : models/<TAG>_fold0/test.csv  vs models/ej_ad_mv_fold0/test.csv
+5-fold : results/final_cv_long_<TAG>.csv vs results/final_cv_long.csv (rule ens-es and stage 1, mean over folds)
+Stage 1 (mIF / RNA / text heads) is seeded and deterministic on the same GPU / library stack, so the emergent pairs
+should agree to ~1e-3; stage 2 (H&E adapter, 3 early-stopped seeds) is not bit-deterministic on GPU, so the trained
+H&E pairs agree within the run-to-run spread (5-fold seed std of a single adapter: mIF 0.009, text 0.011).
+Tolerances: |diff| <= 0.01 (emergent / stage 1), <= 0.03 (fold-0 H&E pairs), <= 0.015 (5-fold means of H&E pairs).
+"""
+import os
+import sys
+
+import pandas as pd
+
+ROOT = os.path.dirname(os.path.abspath(__file__))
+TAG = sys.argv[1] if len(sys.argv) > 1 else "repro"
+pd.set_option("display.width", 200)
+ok = True
+
+
+def show(D, tol_trained, title):
+    global ok
+    D["abs_diff"] = (D.reproduced - D.shipped).abs()
+    D["tol"] = [tol_trained if p.startswith("H&E") and r != "stage1" else 0.01 for p, r in zip(D.pair, D.get("rule", [""] * len(D)))]
+    D["ok"] = D.abs_diff <= D.tol
+    ok &= bool(D.ok.all())
+    print(f"\n{title}\n{D.round(4).to_string(index=False)}")
+
+
+f0 = os.path.join(ROOT, "models", f"{TAG}_fold0", "test.csv")
+if os.path.exists(f0):
+    a = pd.read_csv(f0).set_index("pair")["R@1"]
+    b = pd.read_csv(os.path.join(ROOT, "models", "ej_ad_mv_fold0", "test.csv")).set_index("pair")["R@1"]
+    show(pd.DataFrame({"pair": a.index, "reproduced": a.values, "shipped": b.loc[a.index].values}), 0.03,
+         "fold 0, held-out test R@1 (g100)")
+cv = os.path.join(ROOT, "results", f"final_cv_long_{TAG}.csv")
+if os.path.exists(cv):
+    m = lambda f: pd.read_csv(f).query("rule in ['stage1', 'ens-es']").groupby(["rule", "pair"], sort=False)["R@1"].mean()
+    a, b = m(cv), m(os.path.join(ROOT, "results", "final_cv_long.csv"))
+    show(pd.DataFrame({"rule": [r for r, _ in a.index], "pair": [p for _, p in a.index], "reproduced": a.values,
+                       "shipped": b.loc[a.index].values}), 0.015, "5-fold mean test R@1 (g100)")
+print("\nREPRODUCED within tolerance" if ok else "\nNOT reproduced within tolerance (see ok=False rows)")
+sys.exit(0 if ok else 1)

embed.py

@@ -0,0 +1,347 @@
+"""Four-modality embedding model: H&E, mIF (protein), RNA (spot expression) and text in ONE 1536-d space.
+
+Configuration EJ-ad+MV (RESULTS.md, "MedVTok as H&E image information + adapter-only joint training"):
+  stage 1  every modality encoder gets its own head into the frozen UNI2-h space (InfoNCE against UNI2 on REAL pairs:
+           ORION H&E-mIF, HEST H&E-RNA, TCGA/PathGen H&E-text)
+             mIF  : VirTues (16-marker ORION panel) -> MLP
+             RNA  : hvg (log1p CP10k of 2000 HEST HVGs), Tahoe-x1 70M, SE-600M, AIDO.Cell-100M -> one linear head each
+             text : KEEP, BiomedCLIP (text tower), Qwen3-Embedding-0.6B -> one linear head each
+           RNA / text = late fusion: z = normalise(sum_e normalise(head_e(x_e)))
+  stage 2  all mIF / RNA / text heads frozen; only a zero-initialised residual H&E adapter (+ temperatures) trains:
+             H&E = UNI2 + A([UNI2, c * MedVTok])
+           so mIF / RNA / text embeddings (and every emergent mIF<->RNA<->text pairing) are exactly the stage-1 ones.
+Checkpoints are written by 52_final_model.py (models/<name>/model.pt + meta.json).
+
+Usage (python: .venv/bin/python -I, run from anywhere; GPU):
+    from embed import Embedder
+    E = Embedder("models/ej_ad_mv")                       # release model
+    z = E.he(images)          # H&E tiles: list of PIL images / (N, H, W, 3) uint8; ~0.5 um/px, 112-166 um field
+    z = E.text(["Tumor cells with ...", ...])
+    z = E.rna(counts, genes, zscore="train")  # raw counts (spots x genes; numpy or scipy.sparse), HGNC symbols
+    z = E.mif(tiles, slide)   # (N, 16, H, W) 8-bit-scale mIF tiles, channels = MIF_MARKERS, + slide id per tile
+    sim = z_a @ z_b.T         # every output: L2-normalised (N, 1536) float32 torch tensor on CPU
+Feature-level entry points (precomputed encoder features, as in cache/):
+    E.he_from_feats(uni2, medvtok), E.mif_from_feats(virtues), E.rna_from_feats({enc: X}), E.text_from_feats({enc: X})
+Raw-input pre-processing is exactly the one used to build the training features (functions imported from
+04b_ / 14_ / 43_ / 45_ / 45b_ / 50_ logic), see each method. Notes:
+  * RNA zscore="train" uses the HEST-train feature statistics; zscore="self" z-scores each FM feature over the given
+    spots (what the external DLPFC test does; use for a new dataset of >= a few hundred spots).
+  * mIF standardisation (per-slide 0.99-quantile clip, log1p, mean / std, blur) is computed over the tiles given in
+    one call, so embed a whole dataset (or at least whole slides) at once.
+  * Licences of the frozen encoders: UNI2-h (MahmoodLab, research), MedVTok (MIT; aligned to CONCH v1.5, CC-BY-NC-ND),
+    SE-600M and AIDO.Cell (non-commercial), Tahoe-x1 (Apache-2.0), VirTues, KEEP, BiomedCLIP (MIT), Qwen3 (Apache-2.0).
+"""
+import importlib.util
+import os
+import sys
+
+import numpy as np
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+ROOT = os.path.dirname(os.path.abspath(__file__))
+if ROOT not in sys.path:
+    sys.path.insert(0, ROOT)
+
+DIM, MV_DIM = 1536, 1152
+RNA_E, TXT_E = ["hvg", "tahoex1", "se600m", "aidocell"], ["keep", "biomedclip", "qwen3-0.6b"]
+MIF_MARKERS = ["Hoechst", "CD31", "CD45", "CD68", "CD4", "FOXP3", "CD8a", "CD45RO", "CD20", "PD-L1", "CD3e", "CD163",
+               "E-cadherin", "Ki67", "Pan-CK", "SMA"]  # ORION panel without PD-1 (= 30_virtues_shared.py --m16)
+UNI2_LOCAL = "/diskarray/home/tl688/reg2026/model_alldata/uni2-h.bin"  # override with $UNI2_PATH
+
+
+def uni2_weights():
+    """UNI2-h state dict path: $UNI2_PATH, else the local copy, else the gated HF repo MahmoodLab/UNI2-h"""
+    p = os.environ.get("UNI2_PATH", UNI2_LOCAL)
+    if os.path.exists(p):
+        return p
+    from huggingface_hub import hf_hub_download
+    return hf_hub_download("MahmoodLab/UNI2-h", "pytorch_model.bin")
+nz = lambda x: F.normalize(x, dim=-1)
+
+
+def mlp(din, dout=DIM, hid=512):  # = 42_ / 47_ / 48_ / 51_
+    return nn.Sequential(nn.Linear(din, hid), nn.GELU(), nn.Dropout(0.1), nn.Linear(hid, hid), nn.GELU(),
+                         nn.LayerNorm(hid), nn.Linear(hid, dout))
+
+
+class Bind4(nn.Module):
+    """Heads + H&E adapter. Inputs are raw encoder features; normalisation statistics live in buffers."""
+
+    def __init__(self, dims, adapter="cat", n_adapters=1):
+        """dims: {'mif': 512, 'rna:hvg': 2000, ..., 'text:keep': 768, ...}; adapter None / 'u' / 'cat';
+        n_adapters > 1: seed ensemble, H&E = UNI2 + mean_s A_s(.)"""
+        super().__init__()
+        self.keys = list(dims)
+        self.heads = nn.ModuleDict({self._k(k): (mlp(d) if k == "mif" else nn.Linear(d, DIM)) for k, d in dims.items()})
+        self.logit = nn.ParameterDict({self._k(k): nn.Parameter(torch.tensor(np.log(1 / 0.07), dtype=torch.float32))
+                                       for k in dims})
+        for k, d in dims.items():  # feature standardisation (identity where the encoder is used raw)
+            self.register_buffer(f"mu_{self._k(k)}", torch.zeros(d))
+            self.register_buffer(f"sd_{self._k(k)}", torch.ones(d))
+            self.register_buffer(f"zs_{self._k(k)}", torch.tensor(False))  # True once statistics are set
+        self.adapter = adapter
+        self.register_buffer("c_mv", torch.tensor(1.0))
+        self.ah = None
+        if adapter:
+            self.ah = nn.ModuleList([self.new_adapter() for _ in range(n_adapters)])
+
+    def new_adapter(self):
+        """zero-initialised output -> starts exactly at H&E = UNI2"""
+        a = nn.Sequential(nn.Linear(DIM + (MV_DIM if self.adapter == "cat" else 0), 512), *mlp(DIM)[1:])
+        nn.init.zeros_(a[-1].weight); nn.init.zeros_(a[-1].bias)
+        return a
+
+    @staticmethod
+    def _k(k):
+        return k.replace(":", "__").replace(".", "_").replace("-", "_")
+
+    def h(self, k):
+        return self.heads[self._k(k)]
+
+    def lg(self, k):
+        return self.logit[self._k(k)].exp().clamp(max=100)
+
+    def norm(self, k, x):
+        return (x - getattr(self, f"mu_{self._k(k)}")) / getattr(self, f"sd_{self._k(k)}")
+
+    def set_norm(self, k, mu, sd):
+        """x -> (x - mu) / sd; sd already includes the epsilon of the training code"""
+        getattr(self, f"mu_{self._k(k)}").copy_(mu); getattr(self, f"sd_{self._k(k)}").copy_(sd)
+        getattr(self, f"zs_{self._k(k)}").fill_(True)
+
+    # ---- embeddings from (raw, un-normalised) encoder features; return un-normalised vectors
+    def he(self, uni2, medvtok=None):
+        if self.ah is None:
+            return uni2
+        inp = uni2 if self.adapter == "u" else torch.cat([uni2, self.c_mv * medvtok], -1)
+        return uni2 + sum(a(inp) for a in self.ah) / len(self.ah)
+
+    def mif(self, x):
+        return self.h("mif")(self.norm("mif", x))
+
+    def fuse(self, mod, X, zscore="train"):
+        """X: {enc: raw features}; zscore 'train' (stored stats) / 'self' (stats of X itself, raw encoders untouched)"""
+        z = {}
+        for e, x in X.items():
+            k = f"{mod}:{e}"
+            raw = not bool(getattr(self, f"zs_{self._k(k)}"))
+            x = x if raw else (x - x.mean(0)) / (x.std(0) + 1e-6) if zscore == "self" else self.norm(k, x)
+            z[e] = nz(self.h(k)(x))
+        return nz(sum(z.values())), z
+
+
+def _load_script(fname, argv=None):
+    """import one of the numbered project scripts as a module (their CLI code is under __main__ / argv-guarded)"""
+    old = sys.argv
+    if argv is not None:
+        sys.argv = argv
+    try:
+        spec = importlib.util.spec_from_file_location("m_" + fname.split(".")[0], os.path.join(ROOT, fname))
+        m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m)
+    finally:
+        sys.argv = old
+    return m
+
+
+def _as_uint8_list(images):
+    from PIL import Image, PngImagePlugin
+    PngImagePlugin.MAX_TEXT_CHUNK = max(PngImagePlugin.MAX_TEXT_CHUNK, 64 * 1024 * 1024)  # PathGen PNGs: huge ICC profiles
+    out = []
+    for im in images:
+        if isinstance(im, (str, os.PathLike)):
+            im = Image.open(im)
+        out.append(im.convert("RGB") if isinstance(im, Image.Image) else Image.fromarray(np.asarray(im, np.uint8)))
+    return out
+
+
+class Embedder:
+    def __init__(self, path=os.path.join(ROOT, "models", "ej_ad_mv"), device="cuda"):
+        import json
+        self.dev = device
+        ck = torch.load(os.path.join(path, "model.pt"), map_location="cpu", weights_only=True)
+        self.meta = json.load(open(os.path.join(path, "meta.json")))
+        self.model = Bind4(self.meta["dims"], self.meta["adapter"], self.meta.get("n_adapters", 1))
+        self.model.load_state_dict(ck, strict=True)
+        self.model.eval().to(device)
+        self.hvg = np.asarray(self.meta["hvg_genes"], dtype=str)
+        self._enc = {}
+
+    # ------------------------------------------------------------------ feature-level
+    @torch.no_grad()
+    def he_from_feats(self, uni2, medvtok=None):
+        t = lambda a: None if a is None else torch.as_tensor(np.asarray(a, np.float32)).to(self.dev)
+        return nz(self.model.he(t(uni2), t(medvtok))).cpu()
+
+    @torch.no_grad()
+    def mif_from_feats(self, virtues):
+        return nz(self.model.mif(torch.as_tensor(np.asarray(virtues, np.float32)).to(self.dev))).cpu()
+
+    @torch.no_grad()
+    def rna_from_feats(self, feats, zscore="train"):
+        X = {e: torch.as_tensor(np.asarray(x, np.float32)).to(self.dev) for e, x in feats.items()}
+        return self.model.fuse("rna", X, zscore)[0].cpu()
+
+    @torch.no_grad()
+    def text_from_feats(self, feats):
+        X = {e: torch.as_tensor(np.asarray(x, np.float32)).to(self.dev) for e, x in feats.items()}
+        return self.model.fuse("text", X)[0].cpu()
+
+    # ------------------------------------------------------------------ frozen encoders (lazy)
+    def _get(self, name):
+        if name in self._enc:
+            return self._enc[name]
+        if name == "uni2":  # = 04b_extract_anchor_fm.py
+            import timm
+            m = timm.create_model(
+                "vit_giant_patch14_224", pretrained=False, img_size=224, patch_size=14, depth=24, num_heads=24,
+                init_values=1e-5, embed_dim=1536, mlp_ratio=2.66667 * 2, num_classes=0, no_embed_class=True,
+                mlp_layer=timm.layers.SwiGLUPacked, act_layer=torch.nn.SiLU, reg_tokens=8, dynamic_img_size=True)
+            m.load_state_dict(torch.load(uni2_weights(), map_location="cpu", weights_only=True), strict=True)
+            obj = m.eval().to(self.dev)
+        elif name == "medvtok":  # = 50_medvtok_feats.py
+            import json
+            sys.path.insert(0, os.path.join(ROOT, "ext", "medvtok", "repo"))
+            from models.medvtok import MedVTok
+            w = os.path.join(ROOT, "ext", "medvtok", "weights")
+            cfg = json.load(open(os.path.join(w, "config.json")))
+            m = MedVTok(**cfg["model_kwargs"])
+            m.load_state_dict(torch.load(os.path.join(w, "ckpt.pth"), map_location="cpu", weights_only=True), strict=True)
+            del m.decoder
+            obj = m.eval().to(self.dev)
+        elif name == "keep":
+            obj = _load_script("45b_text_feats_pathvlm.py").KEEPText().to(self.dev).eval()
+        elif name in ("biomedclip", "qwen3-0.6b"):
+            obj = _load_script("45_text_feats.py", [sys.argv[0]]).encoder(name)[0]
+        elif name == "tahoex1":
+            T = _load_script("43_rnafm_tahoex1.py")
+            obj = (T, *T.load_model())
+        elif name == "se600m":
+            S = _load_script("43_rnafm_se600m.py")
+            obj = (S, *S.build())
+        elif name == "aidocell":
+            A = _load_script("43_rnafm_aidocell.py")
+            obj = (A, *A.load_model())
+        elif name == "virtues":
+            V = _load_script("14_extract_if_fm.py", [sys.argv[0], "virtues", "--no_probe"])
+            uni = dict(V.MARKERS)
+            model, midx = V.load_virtues([uni[m] for m in MIF_MARKERS])
+            obj = (V, model, midx.to(self.dev))
+        else:
+            raise KeyError(name)
+        self._enc[name] = obj
+        return obj
+
+    def release(self):
+        """free the frozen encoders (GPU memory)"""
+        self._enc.clear(); torch.cuda.empty_cache()
+
+    # ------------------------------------------------------------------ raw-input embedding
+    @torch.no_grad()
+    def he_feats(self, images, bs=32):
+        """UNI2-h CLS (224 px PIL bicubic, ImageNet norm, fp16) + MedVTok feats (256 px PIL bicubic, [-1, 1], bf16)"""
+        ims = _as_uint8_list(images)
+        uni, mv = self._get("uni2"), self._get("medvtok")
+        mean = torch.tensor([0.485, 0.456, 0.406], device=self.dev).view(1, 3, 1, 1)
+        std = torch.tensor([0.229, 0.224, 0.225], device=self.dev).view(1, 3, 1, 1)
+        U, M = [], []
+        for s in range(0, len(ims), bs):
+            b = ims[s:s + bs]
+            x = torch.from_numpy(np.stack([np.asarray(i.resize((224, 224), 3)) for i in b])).to(self.dev)
+            x = (x.permute(0, 3, 1, 2).float() / 255 - mean) / std
+            with torch.autocast("cuda", dtype=torch.float16):
+                U.append(uni(x).float().cpu())
+            x = torch.from_numpy(np.stack([np.asarray(i.resize((256, 256), 3)) for i in b])).to(self.dev)
+            x = x.permute(0, 3, 1, 2).float() / 255 * 2 - 1
+            with torch.autocast("cuda", dtype=torch.bfloat16):
+                M.append(mv([i for i in x], sample_posterior=False, decode=False, return_batched_tensor=False)["feats"].float().cpu())
+        return torch.cat(U), torch.cat(M)
+
+    def he(self, images, bs=32):
+        return self.he_from_feats(*self.he_feats(images, bs))
+
+    @torch.no_grad()
+    def text_feats(self, texts, bs=64):
+        texts = ["" if t is None else str(t) for t in texts]
+        order = np.argsort([len(t) for t in texts])  # length-sorted batches as in 45_ / 45b_
+        out = {}
+        for e in TXT_E:
+            f = self._get(e)
+            E = None
+            for i in range(0, len(texts), bs):
+                idx = order[i:i + bs]
+                if e == "keep":  # 45b_: fp32
+                    z = f([texts[j] for j in idx]).float().cpu().numpy()
+                else:  # 45_: bf16 autocast
+                    with torch.autocast("cuda", dtype=torch.bfloat16):
+                        z = f([texts[j] for j in idx]).float().cpu().numpy()
+                if E is None:
+                    E = np.zeros((len(texts), z.shape[1]), np.float32)
+                E[idx] = z
+            out[e] = E.astype(np.float16).astype(np.float32)  # the training features were stored as float16
+        return out
+
+    def text(self, texts, bs=64):
+        return self.text_from_feats(self.text_feats(texts, bs))
+
+    def rna_feats(self, counts, genes, encoders=RNA_E):
+        import scipy.sparse as sp
+        X = sp.csr_matrix(counts, dtype=np.float32)
+        X.eliminate_zeros()
+        genes = np.asarray(genes, dtype=str)
+        out = {}
+        for e in encoders:
+            if e == "hvg":  # 21_hest_prep.py: log1p CP10k over the given genes, then the 2000 HEST HVGs (absent -> 0)
+                tot = np.asarray(X.sum(1)).ravel()
+                L = (sp.diags(1e4 / np.clip(tot, 1, None)) @ X).tocsc()
+                L.data = np.log1p(L.data)
+                pos = {}
+                for j, g in enumerate(genes):
+                    pos.setdefault(g, j)
+                cols = [pos.get(g, -1) for g in self.hvg]
+                H = np.zeros((X.shape[0], len(self.hvg)), np.float32)
+                ok = [i for i, c in enumerate(cols) if c >= 0]
+                H[:, ok] = L[:, [cols[i] for i in ok]].toarray()
+                out[e] = H
+            elif e == "tahoex1":
+                T, m, _, ccfg, vocab = self._get(e)
+                seqs, _ = T.make_seqs(X, genes, "user", vocab, ccfg, np.random.default_rng(0))
+                out[e] = T.embed(m, seqs)
+            elif e == "se600m":
+                S, m, cfg, g2i = self._get(e)
+                Tk, C, _ = S.sentences(X, genes, g2i, cfg["dataset"]["pad_length"], cfg["dataset"]["cls_token_idx"], seed=0)
+                out[e] = S.embed(m, Tk, C)
+            elif e == "aidocell":
+                A, m, _ = self._get(e)
+                Xa, present = A.align(X, genes, "user")
+                out[e] = A.embed(m, Xa, present)
+            out[e] = np.asarray(out[e], np.float32).astype(np.float16).astype(np.float32)  # stored as float16
+        return out
+
+    def rna(self, counts, genes, zscore="train", encoders=RNA_E):
+        return self.rna_from_feats(self.rna_feats(counts, genes, encoders), zscore)
+
+    @torch.no_grad()
+    def mif_feats(self, tiles, slide, size=168, bs=32):
+        """tiles (N, 16, H, W) in MIF_MARKERS order, 8-bit intensity scale (0-255); 14_ preprocessing: antialiased
+        bilinear resize to `size` px (~1 um/px for a 166 um tile), per-slide 0.99-quantile clip + log1p, mean / std
+        over the given tiles, 3x3 Gaussian blur; VirTues mean patch-summary token (fp16)."""
+        V, model, midx = self._get("virtues")
+        N = len(tiles)
+        X = np.empty((N, len(MIF_MARKERS), size, size), np.float16)
+        for s in range(0, N, 64):
+            x = torch.as_tensor(np.asarray(tiles[s:s + 64], np.float32)).to(self.dev)
+            X[s:s + 64] = F.interpolate(x, size=(size, size), mode="bilinear", antialias=True,
+                                        align_corners=False).clamp(min=0).half().cpu().numpy()
+        X = V.standardize(X, np.asarray(slide))[0]
+        out = np.zeros((N, 512), np.float32)
+        with torch.inference_mode(), torch.autocast("cuda", dtype=torch.float16):
+            for s in range(0, N, bs):
+                x = torch.from_numpy(X[s:s + bs]).to(self.dev).float()
+                ps = model.encoder.forward_list([c for c in x], [midx] * len(x)).patch_summary_tokens
+                out[s:s + bs] = torch.stack([p.float().mean((0, 1)) for p in ps]).cpu().numpy()
+        return out.astype(np.float16).astype(np.float32)
+
+    def mif(self, tiles, slide, size=168):
+        return self.mif_from_feats(self.mif_feats(tiles, slide, size))

evaluate_saved.py

@@ -0,0 +1,84 @@
+"""Evaluate a saved checkpoint from the cached encoder features through the public embed.py API (no frozen encoders,
+no raw data; needs cache/ + splits/ from the features repo).
+
+    python -I evaluate_saved.py [models/ej_ad_mv_fold0]
+
+Uses the held-out fold-0 test items stored with the checkpoint (test_items.npz, written by 52_final_model.py fold0)
+and the same g100 protocol as 52_ (100-item galleries, 50 draws, seed 0, both directions; Hungarian pair accuracy).
+If the checkpoint has a test.csv, the numbers must match it (pure inference: deterministic up to float rounding).
+Output: <model>/test_recomputed.csv
+"""
+import os
+import sys
+
+import numpy as np
+import pandas as pd
+import torch
+
+ROOT = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, ROOT)
+from embed import RNA_E, TXT_E, Embedder  # noqa: E402
+
+MD = sys.argv[1] if len(sys.argv) > 1 else os.path.join(ROOT, "models", "ej_ad_mv_fold0")
+CACHE = os.path.join(ROOT, "cache")
+H, RF = os.path.join(CACHE, "hest"), os.path.join(CACHE, "rnafm")
+ld = lambda *p: np.load(os.path.join(*p), mmap_mode="r")
+E = Embedder(MD)
+TI = np.load(os.path.join(MD, "test_items.npz"))
+o, hs, ii, tt = TI["orion"], TI["hest"], TI["tcga_img"], TI["tcga_txt"]
+
+
+def g100(a, b, seed=0, G=100, draws=50):  # = 52_final_model.g100
+    from scipy.optimize import linear_sum_assignment
+    rng = np.random.default_rng(seed)
+    out = {"R@1": [], "R@10": [], "pair_acc": []}
+    for _ in range(draws):
+        s = torch.from_numpy(rng.choice(len(a), min(G, len(a)), replace=False))
+        sim = (a[s] @ b[s].T).numpy(); d = np.diag(sim)
+        for S in (sim, sim.T):
+            rank = (S > d[:, None]).sum(1)
+            out["R@1"].append((rank < 1).mean() / 2); out["R@10"].append((rank < 10).mean() / 2)
+        ri, ci = linear_sum_assignment(-sim)
+        out["pair_acc"].append((ci[np.argsort(ri)] == np.arange(len(s))).mean())
+    return {m: float(np.sum(v) / draws) for m, v in out.items()}
+
+
+def virt_hvg(st):  # STPath virtual RNA -> log1p CP10k on the model's HVG list (= 52_)
+    g = pd.read_csv(os.path.join(CACHE, "stpath", f"{st}_genes.csv")).gene.values
+    c = np.expm1(np.load(os.path.join(CACHE, "stpath", f"{st}_pred.npy")).astype(np.float32)).clip(0)
+    c = np.log1p(c / c.sum(1, keepdims=True) * 1e4)
+    return c[:, pd.Series(range(len(g)), index=g).loc[E.hvg].values]
+
+
+def rna(st, idx):
+    if st == "hest":
+        F = {e: (ld(H, "hvg.npy") if e == "hvg" else ld(RF, f"{e}_hest.npy"))[idx] for e in RNA_E}
+    else:
+        F = {e: (virt_hvg({"orion_v": "orion", "tcga_v": "tcga_text"}[st]) if e == "hvg" else ld(RF, f"{e}_{st}.npy"))[idx]
+             for e in RNA_E}
+    return E.rna_from_feats(F)
+
+
+text = lambda st, idx: E.text_from_feats({e: ld(CACHE, f"{st}_text_{e}.npy")[idx, 0] for e in TXT_E})
+zm = E.mif_from_feats(ld(CACHE, "if_virtues_m16.npy")[o, 0])
+zt = text("alltxt", tt)
+res = {"H&E<->mIF (ORION)": g100(E.he_from_feats(ld(CACHE, "he_uni2.npy")[o, 0], ld(CACHE, "he_medvtok.npy")[o, 0]), zm),
+       "H&E<->RNA (HEST)": g100(E.he_from_feats(ld(H, "he_uni2.npy")[hs, 0], ld(H, "he_medvtok.npy")[hs, 0]), rna("hest", hs)),
+       "H&E<->text (TCGA)": g100(E.he_from_feats(ld(CACHE, "allimg_he_uni2.npy")[ii, 0],
+                                                 ld(CACHE, "allimg_he_medvtok.npy")[ii, 0]), zt),
+       "mIF<->RNA(virtual) (ORION)": g100(zm, rna("orion_v", o)),
+       "mIF<->text(synthetic) (ORION)": g100(zm, text("orion_gentext", o)),
+       "text<->RNA(virtual) (TCGA)": g100(zt, rna("tcga_v", ii)),
+       "text<->mIF(MIPHEI) (TCGA)": g100(zt, E.mif_from_feats(ld(CACHE, "if_virtues_m16_virtual_miphei_tcga_text.npy")[ii, 0]))}
+D = pd.DataFrame([{"pair": p, **r} for p, r in res.items()])
+ref = os.path.join(MD, "test.csv")
+if os.path.exists(ref):
+    D["R@1_saved"] = pd.read_csv(ref).set_index("pair").loc[D.pair, "R@1"].values
+    D["abs_diff"] = (D["R@1"] - D["R@1_saved"]).abs()
+D.to_csv(os.path.join(MD, "test_recomputed.csv"), index=False)
+pd.set_option("display.width", 200)
+print(f"{MD}: fold-0 test, g100 (chance R@1 0.01)\n{D.round(4).to_string(index=False)}")
+if "abs_diff" in D:
+    ok = D.abs_diff.max() < 2e-3
+    print("MATCHES the saved test.csv" if ok else "DOES NOT MATCH the saved test.csv", f"(max |diff| {D.abs_diff.max():.2e})")
+    sys.exit(0 if ok else 1)

example_check.py

@@ -0,0 +1,99 @@
+"""End-to-end check of embed.py from RAW inputs on examples/ (64 held-out fold-0 test items per dataset; needs the
+frozen encoders, see setup_encoders.sh; GPU).
+
+    python -I example_check.py [models/ej_ad_mv_fold0]
+
+1. raw input -> encoder features, compared with the cached features the model was trained / tested on
+   (examples/ref_feats.npz): per-item cosine. Expected ~1.000; Tahoe-x1 / SE-600M ~0.99 (their official input
+   pipelines sample genes at random); VirTues lower here because mIF standardisation statistics are computed over the
+   tiles given in one call (64 tiles here vs the whole ORION set for the cache).
+2. raw input -> 1536-d embeddings -> retrieval inside the 64 items (R@1, both directions; chance 1/64), next to the
+   same retrieval from the cached features.
+Output: examples/example_check.csv; exit code 1 if a deterministic encoder does not reproduce its cached features.
+"""
+import os
+import sys
+import time
+
+import numpy as np
+import pandas as pd
+import scipy.sparse as sp
+import torch
+
+ROOT = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(0, ROOT)
+from embed import RNA_E, TXT_E, Embedder  # noqa: E402
+
+MD = sys.argv[1] if len(sys.argv) > 1 else os.path.join(ROOT, "models", "ej_ad_mv_fold0")
+EX = os.path.join(ROOT, "examples")
+E = Embedder(MD)
+R = dict(np.load(os.path.join(EX, "ref_feats.npz")))
+rows, t0 = [], time.time()
+MIN_COS = {"uni2": .999, "medvtok": .999, "keep": .999, "biomedclip": .99, "qwen3-0.6b": .999, "hvg": .999,
+           "aidocell": .999, "tahoex1": .97, "se600m": .97, "virtues": 0.0}  # median cosine required (virtues: report only)
+
+
+def cos(a, b):
+    a, b = torch.as_tensor(np.asarray(a, np.float32)), torch.as_tensor(np.asarray(b, np.float32))
+    return torch.nn.functional.cosine_similarity(a, b, dim=-1).numpy()
+
+
+def feat(ds, enc, raw):
+    c = cos(raw, R[f"{ds}_{enc}"])
+    rows.append({"check": "features", "data": ds, "encoder": enc, "n": len(c), "cos_median": float(np.median(c)),
+                 "cos_min": float(c.min()), "ok": bool(np.median(c) >= MIN_COS[enc])})
+    print(f"[{time.time() - t0:4.0f}s] {ds:5s} {enc:11s} cos median {np.median(c):.4f} min {c.min():.4f}"
+          f"{'' if rows[-1]['ok'] else '   <-- FAIL'}", flush=True)
+
+
+def r1(a, b):
+    s = a @ b.T; d = s.diag()
+    return float((((s > d[:, None]).sum(1) == 0).float().mean() + ((s.T > d[:, None]).sum(1) == 0).float().mean()) / 2)
+
+
+# raw H&E (PNG files / uint8 arrays) -> UNI2 + MedVTok
+U, M = {}, {}
+U["orion"], M["orion"] = E.he_feats(sorted(os.path.join(EX, "orion_he", f) for f in os.listdir(os.path.join(EX, "orion_he"))))
+U["hest"], M["hest"] = E.he_feats(np.load(os.path.join(EX, "hest_he.npy")))
+C = pd.read_csv(os.path.join(EX, "tcga_captions.csv"))
+U["tcga"], M["tcga"] = E.he_feats([os.path.join(EX, "tcga_he", f) for f in C.file])
+for ds in U:
+    feat(ds, "uni2", U[ds]); feat(ds, "medvtok", M[ds])
+E.release()
+# raw text -> KEEP / BiomedCLIP / Qwen3
+Ft = E.text_feats(C.caption.fillna("").astype(str).tolist())
+for e in TXT_E:
+    feat("tcga", e, Ft[e])
+E.release()
+# raw counts -> hvg / Tahoe-x1 / SE-600M / AIDO.Cell
+genes = pd.read_csv(os.path.join(EX, "hest_genes.csv")).gene.values
+Fr = E.rna_feats(sp.load_npz(os.path.join(EX, "hest_counts.npz")), genes)
+for e in RNA_E:
+    feat("hest", e, Fr[e])
+E.release()
+# raw mIF tiles -> VirTues
+m = np.load(os.path.join(EX, "orion_mif.npz"), allow_pickle=True)
+Fm = E.mif_feats(m["tiles"], m["slide"])
+feat("orion", "virtues", Fm)
+E.release()
+
+# embeddings + retrieval inside the 64 items
+pairs = {"H&E<->mIF (ORION)": ((E.he_from_feats(U["orion"], M["orion"]), E.mif_from_feats(Fm)),
+                               (E.he_from_feats(R["orion_uni2"], R["orion_medvtok"]), E.mif_from_feats(R["orion_virtues"]))),
+         "H&E<->RNA (HEST)": ((E.he_from_feats(U["hest"], M["hest"]), E.rna_from_feats(Fr)),
+                              (E.he_from_feats(R["hest_uni2"], R["hest_medvtok"]),
+                               E.rna_from_feats({e: R[f"hest_{e}"] for e in RNA_E}))),
+         "H&E<->text (TCGA)": ((E.he_from_feats(U["tcga"], M["tcga"]), E.text_from_feats(Ft)),
+                               (E.he_from_feats(R["tcga_uni2"], R["tcga_medvtok"]),
+                                E.text_from_feats({e: R[f"tcga_{e}"] for e in TXT_E})))}
+for p, ((a, b), (ac, bc)) in pairs.items():
+    rows.append({"check": "retrieval", "data": p, "n": len(a), "R@1_raw": r1(a, b), "R@1_cached": r1(ac, bc),
+                 "chance": 1 / len(a)})
+    print(f"{p:18s} R@1 within {len(a)}: raw inputs {rows[-1]['R@1_raw']:.3f} | cached features "
+          f"{rows[-1]['R@1_cached']:.3f} (chance {1 / len(a):.3f})", flush=True)
+D = pd.DataFrame(rows)
+D.to_csv(os.path.join(EX, "example_check.csv"), index=False)
+bad = D[(D.check == "features") & ~D.ok.astype(bool)]
+print(f"done [{time.time() - t0:.0f}s]:", "all encoders reproduce the cached features" if bad.empty else
+      f"FAILED {bad[['data', 'encoder']].values.tolist()}")
+sys.exit(0 if bad.empty else 1)

examples/hest_counts.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:78a1f9d3b380027a1d95190409851f4b5b966daf47a7602100786362f96569eb
+size 492119

examples/hest_he.npy

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d91cd94cfa57a713ecf10b2be3949c118ec6836fcf754e0e7e2192ab94efd92
+size 9633920

examples/items.csv

@@ -0,0 +1,193 @@
+set,idx,txt_idx
+orion,94,
+orion,96,
+orion,98,
+orion,103,
+orion,107,
+orion,111,
+orion,115,
+orion,118,
+orion,140,
+orion,146,
+orion,152,
+orion,176,
+orion,204,
+orion,208,
+orion,263,
+orion,267,
+orion,272,
+orion,287,
+orion,292,
+orion,596,
+orion,598,
+orion,599,
+orion,608,
+orion,621,
+orion,652,
+orion,659,
+orion,661,
+orion,1602,
+orion,1605,
+orion,1610,
+orion,1613,
+orion,1614,
+orion,1640,
+orion,1652,
+orion,1658,
+orion,1665,
+orion,1668,
+orion,1687,
+orion,1688,
+orion,1698,
+orion,1703,
+orion,1720,
+orion,1721,
+orion,1726,
+orion,1731,
+orion,1737,
+orion,1753,
+orion,1901,
+orion,1913,
+orion,1918,
+orion,1932,
+orion,1940,
+orion,1943,
+orion,1954,
+orion,1961,
+orion,1967,
+orion,1988,
+orion,2426,
+orion,2442,
+orion,2454,
+orion,2472,
+orion,2480,
+orion,2482,
+orion,2493,
+hest,1096,
+hest,1099,
+hest,1158,
+hest,1161,
+hest,1214,
+hest,1224,
+hest,1263,
+hest,1285,
+hest,1294,
+hest,1318,
+hest,1416,
+hest,1431,
+hest,1435,
+hest,1500,
+hest,1597,
+hest,1617,
+hest,1719,
+hest,1741,
+hest,1827,
+hest,1830,
+hest,1846,
+hest,1854,
+hest,1858,
+hest,1884,
+hest,1926,
+hest,1947,
+hest,1996,
+hest,1998,
+hest,2010,
+hest,2059,
+hest,2132,
+hest,2193,
+hest,2221,
+hest,2230,
+hest,2242,
+hest,2636,
+hest,2814,
+hest,2936,
+hest,3387,
+hest,3391,
+hest,3449,
+hest,3461,
+hest,3465,
+hest,3468,
+hest,3525,
+hest,5684,
+hest,5694,
+hest,6670,
+hest,6672,
+hest,6682,
+hest,6684,
+hest,7312,
+hest,7596,
+hest,7616,
+hest,7677,
+hest,8096,
+hest,8099,
+hest,8102,
+hest,8113,
+hest,8114,
+hest,8174,
+hest,8176,
+hest,8639,
+hest,8647,
+tcga,39,41.0
+tcga,113,117.0
+tcga,141,146.0
+tcga,188,193.0
+tcga,191,196.0
+tcga,268,279.0
+tcga,360,374.0
+tcga,369,383.0
+tcga,556,581.0
+tcga,601,628.0
+tcga,654,684.0
+tcga,686,717.0
+tcga,689,720.0
+tcga,712,745.0
+tcga,796,837.0
+tcga,1160,1220.0
+tcga,1210,1273.0
+tcga,1265,1332.0
+tcga,1421,1491.0
+tcga,1533,1609.0
+tcga,1554,1632.0
+tcga,1565,1643.0
+tcga,2094,2200.0
+tcga,2110,2217.0
+tcga,2287,2404.0
+tcga,2379,2500.0
+tcga,2513,2640.0
+tcga,2526,2653.0
+tcga,2725,2864.0
+tcga,3015,3172.0
+tcga,3104,3264.0
+tcga,3161,3330.0
+tcga,3167,3336.0
+tcga,3292,3467.0
+tcga,3413,3595.0
+tcga,3441,3626.0
+tcga,3584,3776.0
+tcga,3656,3850.0
+tcga,3726,3922.0
+tcga,3845,4047.0
+tcga,4124,4341.0
+tcga,4207,4428.0
+tcga,4370,4598.0
+tcga,4425,4653.0
+tcga,4609,4851.0
+tcga,4689,4933.0
+tcga,4699,4943.0
+tcga,4783,5030.0
+tcga,4807,5057.0
+tcga,4869,5126.0
+tcga,4902,5160.0
+tcga,4935,5195.0
+tcga,4982,5244.0
+tcga,5005,5269.0
+tcga,5028,5293.0
+tcga,5162,5438.0
+tcga,5256,5538.0
+tcga,5403,5691.0
+tcga,5631,5934.0
+tcga,5632,5935.0
+tcga,5660,5964.0
+tcga,5757,6070.0
+tcga,5776,6090.0
+tcga,6007,6335.0