README.md

---
license: other
license_name: sanofi-non-commercial
license_link: LICENSE
language:
- en
- fr
base_model:
- bioptimus/H-optimus-0
library_name: timm
tags:
- histopathology
- image-to-image
- vit
- cell-classification
- image-translation
pipeline_tag: image-to-image
---
# Model card for MIPHEI-ViT

<p align="center">
  <img src="logo.svg" alt="MIPHEI-ViT Logo" style="height:250px;">
</p>

<p align="center">
  <a href="https://arxiv.org/abs/2505.10294" target="_blank" rel="noopener noreferrer" style="display: inline-block; vertical-align: middle;">
    <img src="https://img.shields.io/badge/arXiv-Paper-red?logo=arxiv" height="25">
  </a>
  <a href="https://github.com/Sanofi-Public/MIPHEI-ViT" target="_blank" rel="noopener noreferrer" style="display: inline-block; vertical-align: middle;">
    <img src="https://img.shields.io/badge/GitHub-Code-black?logo=github" height="25">
  </a>
  <a href="https://huggingface.co/spaces/Estabousi/MIPHEI-vit-demo" target="_blank" rel="noopener noreferrer" style="display: inline-block; vertical-align: middle;">
    <img src="https://img.shields.io/badge/Gradio-Demo-yellow?logo=gradio" height="25">
  </a>
  <a href="https://zenodo.org/records/15340874" target="_blank" rel="noopener noreferrer" style="display: inline-block; vertical-align: middle;">
    <img src="https://img.shields.io/badge/Zenodo-Dataset-blue?logo=zenodo" height="25">
  </a>
</p>

**MIPHEI-ViT** is a deep learning model that predicts 16-channel **multiplex immunofluorescence (mIF)** images from standard **H&E-stained histology images**. It uses a **U-Net-style architecture** with a **ViT foundation model (H-Optimus-0)** as the encoder, inspired by the ViTMatte model.

This work is described in our paper:

**“MIPHEI-vit: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models.”**

Please see the publication for full results and details.

The model was trained on a processed version of the ORION-CRC dataset, available here: [🔗 MIPHEI-ViT Dataset on Zenodo](https://zenodo.org/records/15340874)

It takes H&E image tiles as input and outputs **16-channel mIF predictions** for the following markers: **Hoechst, CD31, CD45, CD68, CD4, FOXP3, CD8a, CD45RO, CD20, PD-L1, CD3e, CD163, E-cadherin, Ki67, Pan-CK, SMA**

For optimal performances, input H&E images should come from **colon tissue** and be scanned at **0.5 µm/pixel (20x magnification)**. However, because the model is built on a large ViT foundation (H-Optimus-0), so you may try applying it to other tissue type as well.

<p align="center">
  <img src="https://github.com/Sanofi-Public/MIPHEI-ViT/blob/main/figures/figure_architecture.png?raw=true" alt="MIPHEI-ViT Architecture" style="height:300px;">
</p>
<p align="center">
  <em>Figure: Overview of the MIPHEI-ViT architecture.</em>
</p>

This model was developed as part of research funded by **Sanofi** and **ANRT**.

---

> **⚠️ Important — Weights Not Stored on HuggingFace**
> For legal and compliance reasons, **no model weights are hosted on this HuggingFace model repository**.
> 
> All model files (weights, config, minimal inference code) are provided **exclusively through the official GitHub Release**:
> 
> 🔗 **https://github.com/Sanofi-Public/MIPHEI-ViT/releases**
> 
> Please follow the instructions above (`download_weights.py`) to retrieve the necessary files.

## 🚀 Demo

You can try the model directly in your browser and upload your own H&E images:

[<img src="https://img.shields.io/badge/Gradio-MIPHEI--ViT--Demo-blue?logo=gradio" alt="MIPHEI-ViT Demo" height="25">](https://huggingface.co/spaces/Estabousi/MIPHEI-vit-demo)

---

## 🔍 Model Usage

### Clone the model repository

This brings the code and files (including `model.py`, weights, config, etc.) to your machine:

```bash
git lfs install  # only needed once, if not already done
git clone https://huggingface.co/Estabousi/MIPHEI-vit
python download_weights.py
cd MIPHEI-vit
pip install -r requirements.txt # torch, timm, safetensors, numpy, Pillow, huggingface_hub
```

### Load the model

```python
import torch
from model import MIPHEIViT
device = "cuda" if torch.cuda.is_available() else "cpu"
model = MIPHEIViT.from_pretrained_hf(repo_path=".")
model.set_input_size((width, height)) # width, height power of 2 and at least 128
model.eval().to(device).half() # faster in half precision
```

### Run inference on a H&E tile

```python
from PIL import Image
import torchvision.transforms as T

# Load and preprocess your tile
img = Image.open("tile.jpg").convert("RGB")

transform = T.Compose([
    T.Resize((width, height)),
    T.ToTensor(),  # Converts to shape [3, H, W], range [0,1]
    T.Normalize(
	    mean=(0.485, 0.456, 0.406),
	    std=(0.229, 0.224, 0.225)
    ),  # H-optimus-0 normalization
])
tile_tensor = transform(img).unsqueeze(0)  # Add batch dim: [1, 3, width, height]

# Predict mIF channels
with torch.inference_mode():
    mif_pred = model(tile_tensor.to(device).half()).squeeze()  # Output: [16, width, height]
    mif_pred = (mif_pred.clamp(-0.9, 0.9) + 0.9) / 1.8  # [-0.9, 0.9] -> [0., 1.]
    mif_pred = (mif_pred * 255).to(torch.uint8)
    mif_pred = mif_pred.permute((1, 2, 0)).cpu()  # Output: [width, height, 16]
```

Output corresponds to the following 16 markers:

```
['Hoechst', 'CD31', 'CD45', 'CD68', 'CD4', 'FOXP3', 'CD8a', 'CD45RO',
 'CD20', 'PD-L1', 'CD3e', 'CD163', 'E-cadherin', 'Ki67', 'Pan-CK', 'SMA']

```

You can also try our model in colab: [<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" height="25">](https://colab.research.google.com/github/Sanofi-Public/MIPHEI-ViT/blob/main/notebooks/colab_inference.ipynb)

## 📁 Files Included (in associated GitHub release)

- `model.py`: model architecture
- `model.safetensors`: pretrained weights
- `logreg.pth`: pretrained cell type linear classifier
- `config_hf.json`: inference configuration used by huggingface
- `config.yaml`: training configuration parameters
- `requirements.txt`: requirements for installing necessary pip packages

---

## 📖 Citation

If you use this work, please cite:

> G. Balezo, R. Trullo, A. Pla Planas, E. Decenciere, and T. Walter, “MIPHEI-ViT: Multiplex Immunofluorescence Prediction from H&E Images using ViT Foundation Models,” arXiv preprint arXiv:2505.10294, 2025.
> 

---

## 🧪 More Details

For full training, preprocessing, visualizations, and evaluations, visit the [<img src="https://img.shields.io/badge/GitHub-Repository-black?logo=github" alt="GitHub Repository" height="25">](https://github.com/Sanofi-Public/MIPHEI-ViT)


---

## 📄 License

Released by **Sanofi** under specific license conditions, including a limitation to **non-commercial use only**. 
See the LICENSE file for details.