Add ELF slide encoder weights and model card

README.md

@@ -0,0 +1,116 @@
+---
+license: gpl-3.0
+library_name: pytorch
+pipeline_tag: feature-extraction
+tags:
+  - pathology
+  - histopathology
+  - computational-pathology
+  - slide-encoder
+  - abmil
+  - foundation-model
+  - ensemble
+---
+
+# ELF slide encoder
+
+**ELF** (Ensemble Learning of Foundation models) is a slide-level pathology encoder from the Li Lab at Stanford. It aggregates tile embeddings from five public foundation models into a unified whole-slide representation.
+
+Paper: [Ensemble learning of pathology foundation models for precision oncology](https://arxiv.org/abs/2508.16085)  
+Code: [github.com/lilab-stanford/ELF](https://github.com/lilab-stanford/ELF)  
+Weights: [huggingface.co/luoxd96/ELF](https://huggingface.co/luoxd96/ELF)
+
+This Hub repo contains the **inference-only** slide encoder: interpolate → LayerNorm → 8-head gated ABMIL (~151k parameters, ~0.6 MB). It is the trained `momentum_enc` forward used in the paper, not the MoCo training snapshot.
+
+## Model details
+
+| | |
+|---|---|
+| Developed by | Li Lab, Stanford University |
+| Model type | Slide-level ABMIL aggregator |
+| Inputs | Patch features \(X \in \mathbb{R}^{N \times C}\), \(C \in \{768, 1024, 1280, 1536\}\) |
+| Outputs | `features_dim` \([1, C]\), `features` \([1, 768]\), `attention_weights` \([1, 1, N]\) |
+| Pretraining | 53,699 WSIs, 20 anatomical sites |
+| Tile encoders | UNI, CONCH v1.5, Prov-GigaPath, Virchow2, H-optimus-0 |
+| Magnification | 10× patches (CLAM) |
+| License | GPLv3, non-commercial academic use |
+
+The paper ensemble is the **concatenation** of the five `features_dim` vectors (one per tile foundation model), not an average.
+
+| Tile model | `--fm` | native `C` | `features_dim` |
+|---|---|---|---|
+| UNI | `uni` | 1024 | 1024 |
+| CONCH v1.5 | `conch_v1_5` | 768 | 768 |
+| Prov-GigaPath | `gigapath` | 1536 | 1536 |
+| H-optimus-0 | `h0` | 1536 | 1536 |
+| Virchow2 | `virchow2` | 2560 → 1280 (CLS+mean) | 1280 |
+
+## How to use
+
+```bash
+pip install torch huggingface_hub h5py numpy
+```
+
+Download this file from the Hub (`modeling_elf.py`) or clone the [GitHub repo](https://github.com/lilab-stanford/ELF) and use `inference/model.py`.
+
+```python
+import torch
+from huggingface_hub import hf_hub_download
+
+# modeling_elf.py from this repo
+from modeling_elf import ELFSlideEncoder, preprocess_patch_features
+
+model = ELFSlideEncoder.from_pretrained("luoxd96/ELF", device="cuda")
+
+# patches: [N, C] tile features for one slide and one foundation model
+patches = torch.from_numpy(features).float()
+patches = preprocess_patch_features(patches, foundation_model="uni")  # virchow2 averages CLS+mean
+
+with torch.inference_mode():
+    x = patches.unsqueeze(0).cuda()
+    features_dim, features_768, attn = model(x)
+
+# features_dim: [1, C]  — use this for the paper ensemble
+# features_768: [1, 768]
+# attn:         [1, 1, N]
+```
+
+Batch extract from CLAM-style `h5` files (`dataset "features"`):
+
+```bash
+git clone https://github.com/lilab-stanford/ELF.git
+cd ELF
+
+python inference/extract_multiple_model_slide_embedding.py \
+  --checkpoint $(python -c "from huggingface_hub import hf_hub_download; print(hf_hub_download('luoxd96/ELF','elf_slide_encoder.pth'))") \
+  --input-dir /path/to/uni/<dataset>/h5_files \
+  --output-path /path/to/uni_elf \
+  --feature-models uni \
+  --gpu 0
+```
+
+Repeat for `conch_v1_5`, `gigapath`, `virchow2`, `h0`. Concatenate the five `features_dim` vectors for downstream linear probing, as in `evaluation/`.
+
+## Intended use
+
+Research feature extraction for computational pathology (classification, biomarker prediction, therapy-response studies). This is **not** a diagnostic device and should not be used for clinical decision-making without independent validation.
+
+## Limitations
+
+- Requires pre-extracted tile embeddings from the five foundation models above; it does not encode RGB tiles.
+- Virchow2 inputs of dimension ≥ 2560 are averaged as CLS + mean → 1280, matching the paper.
+- Licensed for non-commercial academic use.
+
+## Citation
+
+```bibtex
+@misc{luo2026ensemblelearningpathologyfoundation,
+  title={Ensemble learning of pathology foundation models for precision oncology},
+  author={Xiangde Luo and Xiyue Wang and Feyisope Eweje and Xiaoming Zhang and Juan Luis Gomez Marti and Sarah Cascarino and Sen Yang and Yuchen Li and Ryan Quinton and Jinxi Xiang and Yuanfeng Ji and Zhe Li and Yijiang Chen and Colin Bergstrom and Ted Kim and Francesca Maria Olguin and Kelley Yuan and Matthew Abikenari and Andrew Heider and Sierra Willens and Sanjeeth Rajaram and Robert West and Joel Neal and Adam Schoenfeld and Maximilian Diehn and Chad Vanderbilt and Ruijiang Li},
+  year={2026},
+  eprint={2508.16085},
+  archivePrefix={arXiv},
+  primaryClass={cs.CV},
+  url={https://arxiv.org/abs/2508.16085},
+}
+```

config.json

@@ -0,0 +1,12 @@
+{
+  "model_type": "elf_slide_encoder",
+  "architectures": ["ELFSlideEncoder"],
+  "embed_dim": 768,
+  "num_heads": 8,
+  "n_params": 151304,
+  "encoder": "momentum_enc.norm+attn",
+  "weights": "elf_slide_encoder.pth",
+  "paper": "https://arxiv.org/abs/2508.16085",
+  "code": "https://github.com/lilab-stanford/ELF",
+  "hub": "https://huggingface.co/luoxd96/ELF"
+}

elf_slide_encoder.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:84464b4c83834f4b82df8e12b569c51af77e52a3d5010b8b4e2310431d5da23d
+size 621629

modeling_elf.py

@@ -0,0 +1,131 @@
+"""ELF slide encoder: interpolate → LayerNorm → 8-head ABMIL."""
+from __future__ import annotations
+
+from collections import OrderedDict
+from pathlib import Path
+from typing import Optional, Union
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+DEFAULT_REPO_ID = "luoxd96/ELF"
+WEIGHTS_FILE = "elf_slide_encoder.pth"
+
+
+class BatchedABMIL(nn.Module):
+    def __init__(self, dim: int):
+        super().__init__()
+        self.attention_a = nn.Sequential(nn.Linear(dim, dim), nn.Tanh())
+        self.attention_b = nn.Sequential(nn.Linear(dim, dim), nn.Sigmoid())
+        self.attention_c = nn.Linear(dim, 1)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        return self.attention_c(self.attention_a(x) * self.attention_b(x))
+
+
+class ELFSlideEncoder(nn.Module):
+    def __init__(self, embed_dim: int = 768, num_heads: int = 8):
+        super().__init__()
+        if embed_dim % num_heads != 0:
+            raise ValueError(f"embed_dim ({embed_dim}) must be divisible by num_heads ({num_heads})")
+        self.embed_dim = embed_dim
+        self.num_heads = num_heads
+        self.norm = nn.LayerNorm(embed_dim)
+        self.attn = nn.ModuleList(
+            [BatchedABMIL(embed_dim // num_heads) for _ in range(num_heads)]
+        )
+
+    def forward(self, x: torch.Tensor, lens: Optional[torch.Tensor] = None):
+        """
+        Args:
+            x: ``[B, N, C]`` patch features (``C`` in {768, 1024, 1280, 1536}).
+            lens: ``[B]`` native ``C`` per item; defaults to ``x.shape[-1]``.
+
+        Returns:
+            features_dim: ``[B, C]`` — ``softmax(ᾱ)ᵀ X``
+            features: ``[B, 768]`` — ``softmax(ᾱ)ᵀ X_768``
+            attention: ``[B, 1, N]``
+        """
+        if x.ndim != 3:
+            raise ValueError(f"expected [B, N, C], got {tuple(x.shape)}")
+        batch, n_tiles, feat_dim = x.shape
+        if lens is None:
+            lens = torch.full((batch,), feat_dim, dtype=torch.long, device=x.device)
+
+        x768 = []
+        for i in range(batch):
+            c = int(lens[i].item())
+            x768.append(
+                F.interpolate(x[i, :, :c].unsqueeze(0), size=self.embed_dim, mode="linear", align_corners=True).squeeze(0)
+            )
+        x768 = self.norm(torch.stack(x768, dim=0))
+
+        head_dim = self.embed_dim // self.num_heads
+        heads = x768.view(batch, n_tiles, head_dim, self.num_heads)
+        logits = torch.stack([self.attn[h](heads[:, :, :, h]) for h in range(self.num_heads)], dim=-1)
+        attn = F.softmax(logits.mean(dim=-1).transpose(1, 2), dim=-1)
+
+        feat_768 = torch.bmm(attn, x768)[:, 0]
+        feat_native = torch.stack(
+            [torch.bmm(attn[i : i + 1], x[i : i + 1, :, : int(lens[i].item())])[0, 0] for i in range(batch)]
+        )
+        return feat_native, feat_768, attn
+
+    @classmethod
+    def from_pretrained(
+        cls,
+        repo_id: str = DEFAULT_REPO_ID,
+        filename: str = WEIGHTS_FILE,
+        device: Union[str, torch.device] = "cpu",
+        embed_dim: int = 768,
+        num_heads: int = 8,
+    ) -> "ELFSlideEncoder":
+        from huggingface_hub import hf_hub_download
+
+        path = hf_hub_download(repo_id=repo_id, filename=filename)
+        return load_encoder(path, device=device, embed_dim=embed_dim, num_heads=num_heads)
+
+
+def preprocess_patch_features(features: torch.Tensor, foundation_model: Optional[str] = None) -> torch.Tensor:
+    x = features.float()
+    if (foundation_model or "").lower() == "virchow2" and x.shape[-1] >= 2560:
+        x = 0.5 * (x[..., :1280] + x[..., 1280:2560])
+    return x
+
+
+def _unwrap_state_dict(raw) -> OrderedDict:
+    if isinstance(raw, dict) and "state_dict" in raw:
+        raw = raw["state_dict"]
+    return OrderedDict((k[7:] if k.startswith("module.") else k, v) for k, v in raw.items())
+
+
+def extract_inference_weights(state_dict: dict) -> OrderedDict:
+    keys = list(state_dict.keys())
+    prefix = ""
+    if any(k.startswith("momentum_enc.") for k in keys):
+        prefix = "momentum_enc."
+    keep = ("norm.", "attn.")
+    out = OrderedDict(
+        (k[len(prefix) :], v)
+        for k, v in state_dict.items()
+        if k.startswith(prefix) and k[len(prefix) :].startswith(keep)
+    )
+    if not out:
+        raise KeyError(f"no norm/attn weights found; prefixes={sorted({k.split('.')[0] for k in keys})[:12]}")
+    return out
+
+
+def load_encoder(
+    checkpoint: Union[str, Path],
+    device: Union[str, torch.device] = "cpu",
+    embed_dim: int = 768,
+    num_heads: int = 8,
+) -> ELFSlideEncoder:
+    ckpt = torch.load(str(checkpoint), map_location="cpu", weights_only=False)
+    weights = extract_inference_weights(_unwrap_state_dict(ckpt))
+    model = ELFSlideEncoder(embed_dim=embed_dim, num_heads=num_heads)
+    missing, unexpected = model.load_state_dict(weights, strict=True)
+    if missing or unexpected:
+        raise RuntimeError(f"load mismatch missing={missing} unexpected={unexpected}")
+    return model.to(device).eval()

requirements.txt

@@ -0,0 +1,4 @@
+torch>=2.0
+huggingface_hub
+h5py
+numpy