Release SEPIA model weights and model card
Browse files
README.md
ADDED
|
@@ -0,0 +1,44 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language: en
|
| 3 |
+
tags:
|
| 4 |
+
- immunohistochemistry
|
| 5 |
+
- histopathology
|
| 6 |
+
- image-feature-extraction
|
| 7 |
+
- vision-transformer
|
| 8 |
+
---
|
| 9 |
+
# SEPIA: A Marker-Aware Foundation Model for Immunohistochemistry
|
| 10 |
+
|
| 11 |
+
SEPIA learns marker-aware representations of immunohistochemistry (IHC) images. It combines hematoxylin and DAB image representations with protein embeddings that encode the identity of the stained marker.
|
| 12 |
+
|
| 13 |
+
SEPIA is initialized from Hibou-L and trained with H/D inpainting. This repository contains the model weights.
|
| 14 |
+
|
| 15 |
+
## Model weights
|
| 16 |
+
|
| 17 |
+
| Model | Backbone | Embedding dimension | Weights |
|
| 18 |
+
|---|---|---|---|
|
| 19 |
+
| SEPIA | Hibou-L (ViT-L/14) | 1024 | sepia.pth |
|
| 20 |
+
|
| 21 |
+
## Model overview
|
| 22 |
+
|
| 23 |
+
SEPIA takes an RGB IHC tile and its marker identity as inputs. The standard tile size is 224×224 pixels. Images are separated into hematoxylin-only and DAB-only renderings, then normalized using the checkpoint's image statistics.
|
| 24 |
+
|
| 25 |
+
The two renderings share a patch projection and produce paired H/D tokens. A projected ESM protein embedding conditions the DAB tokens on the stained marker. The encoder produces a 1024-dimensional image representation and separate H and D patch representations.
|
| 26 |
+
|
| 27 |
+
## Benchmarking
|
| 28 |
+
|
| 29 |
+
Test-set AUROC (%) reported as mean ± standard error over five repeats on seven IHC tasks, alongside the mean rank over all 44 IHC-relevant tasks. The pan-cancer DAB localization task uses 13 held-out markers. Best mean values are bold. Lower mean rank is better.
|
| 30 |
+
|
| 31 |
+
| Model | Breast cancer<br>Histological subtype | Breast cancer<br>Luminal vs non-luminal | DLBCL<br>HANS subtype | DLBCL<br>Double expressor | NSCLC<br>Intratumoral CD8 | Type-2 diabetes<br>Diabetic status | Pan-cancer<br>DAB location (held out) | Overall<br>Mean rank ↓ (44 tasks) |
|
| 32 |
+
|---|---:|---:|---:|---:|---:|---:|---:|---:|
|
| 33 |
+
| DINOv2 | 54.9 ± 0.4 | 81.1 ± 0.7 | 73.2 ± 0.8 | 64.9 ± 1.5 | 71.0 ± 1.4 | 73.6 ± 0.8 | 77.4 ± 0.3 | 7.30 ± 0.19 |
|
| 34 |
+
| ResNet-50 | 56.0 ± 0.7 | 86.3 ± 0.3 | 77.6 ± 0.6 | 76.6 ± 0.9 | 75.2 ± 0.2 | 72.2 ± 1.1 | 81.4 ± 0.8 | 6.64 ± 0.20 |
|
| 35 |
+
| PathoDuet | 50.9 ± 1.7 | 85.3 ± 0.5 | 82.8 ± 0.7 | 83.0 ± 0.9 | 77.7 ± 0.6 | 74.9 ± 0.8 | 88.2 ± 0.6 | 5.68 ± 0.25 |
|
| 36 |
+
| StainNet | 54.9 ± 1.6 | 88.4 ± 0.5 | 87.5 ± 0.9 | 90.2 ± 0.7 | 80.5 ± 0.5 | 79.6 ± 1.2 | 88.4 ± 0.7 | 3.70 ± 0.26 |
|
| 37 |
+
| UNI | 56.2 ± 1.9 | 87.2 ± 0.7 | 84.9 ± 0.8 | 85.2 ± 0.6 | 82.6 ± 0.4 | 76.1 ± 0.7 | 88.2 ± 0.4 | 3.50 ± 0.18 |
|
| 38 |
+
| Hibou | 57.0 ± 1.4 | 87.6 ± 0.4 | 86.7 ± 0.4 | 89.1 ± 0.8 | 82.3 ± 0.3 | 73.1 ± 0.7 | 91.1 ± 0.5 | 3.43 ± 0.24 |
|
| 39 |
+
| UNI2-h | 57.2 ± 0.5 | 87.2 ± 0.6 | 86.3 ± 0.8 | 81.2 ± 1.2 | **83.7** ± 0.3 | 74.0 ± 0.6 | 83.6 ± 0.8 | 3.09 ± 0.32 |
|
| 40 |
+
| SEPIA | **61.6** ± 1.7 | **89.4** ± 0.3 | **90.9** ± 0.4 | **91.9** ± 0.4 | 83.6 ± 0.2 | **89.3** ± 0.4 | **96.4** ± 0.4 | **2.66** ± 0.27 |
|
| 41 |
+
|
| 42 |
+
## Acknowledgements
|
| 43 |
+
|
| 44 |
+
We thank the teams behind Hibou, DINOv2, and ESM for their publicly available models and implementations.
|
sepia.pth
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0e72e96fe3e6f4b3dfd7b2a83751529f512075c84a0632a49b76bee3a22ac39e
|
| 3 |
+
size 1217388571
|