README.md

---
license: cc-by-nc-4.0
library_name: timm
pipeline_tag: image-feature-extraction
tags:
  - computational-pathology
  - knowledge-distillation
  - vision-transformer
  - medical-imaging
datasets:
  - TCGA
---

# DistillPath-IS16-UNI2h

A 22M ViT-S/16 pathology tile encoder distilled from [UNI2-h](https://huggingface.co/MahmoodLab/UNI2-h) (681M ViT-H/14) into an [ImageNet-21k pretrained ViT-S/16](https://huggingface.co/timm/vit_small_patch16_224.augreg_in21k) student using backbone-token distillation on 6,000 public TCGA slides.

This is the ImageNet-initialized variant. For the kaiko-initialized variant, see [DistillPath-KS16-UNI2h](https://huggingface.co/RamonK/DistillPath-KS16-UNI2h).

**Paper:** [DistillPath: An Efficient 22M Distilled Pathology Encoder Approaching Large Foundation Model Performance](https://arxiv.org/abs/2608.17872)
Ramon Kaspar, Andrey Ignatov, Valentina Boeva. ETH Zurich.
Published at the ECCV 2026 Workshop on Medical Foundation Models and Benchmarks (MedFM-Bench).

## Model details

| Property | Value |
|---|---|
| Architecture | ViT-S/16 (`vit_small_patch16_224` in timm) |
| Parameters | 21.7M |
| Feature dimension | 384 |
| Input size | 224 x 224 |
| Normalization | mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225) |
| Student initialization | ImageNet-21k ViT-S/16 |
| Teacher | UNI2-h (681M, ViT-H/14, CC BY-NC-ND 4.0) |
| Training data | 6,000 TCGA H&E whole-slide images, 32 cohorts |
| Training steps | 50,000 (batch size 256) |

## Benchmark results

| Benchmark | **DistillPath-IS16-UNI2h** | IN21K baseline | UNI2-h teacher |
|---|---|---|---|
| EVA mean (7 tasks) | **0.765** | 0.729 | 0.806 |
| HEST mean (9 tasks) | **0.364** | 0.311 | 0.414 |
| PLISM score | **0.561** | 0.383 | 0.333 |

See the paper for per-task results.

## Usage

Load directly from the Hub with timm:

```python
import timm

model = timm.create_model(
    "hf_hub:RamonK/DistillPath-IS16-UNI2h",
    pretrained=True,
    num_classes=0,
)
model.eval()
```

Or load manually:

```python
import timm
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

model = timm.create_model("vit_small_patch16_224", pretrained=False, num_classes=0)
path = hf_hub_download("RamonK/DistillPath-IS16-UNI2h", "model.safetensors")
state_dict = load_file(path)
model.load_state_dict(state_dict, strict=True)
model.eval()
```

This model uses ImageNet normalization: mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225).

```python
from torchvision import transforms

transform = transforms.Compose([
    transforms.Resize(224),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
```

## Distillation recipe

The recipe reads only the teacher's final class and patch tokens (no teacher pretraining heads required):
- **Class-token loss:** cosine distance + RKD (relational knowledge distillation)
- **Patch-token loss:** cosine distance after bicubic grid resizing (teacher 16x16 to student 14x14)
- **Optimizer:** AdamW, lr=1e-4, weight decay 0.05, cosine decay, 500 warmup steps
- **Projector:** DINO-style MLP (384 to 2048 to 2048 to 256 to d_t), discarded after training

Full details in the paper and the [DistillPath repository](https://github.com/RamonKaspar/DistillPath).

## License

This model is released under the [Creative Commons Attribution-NonCommercial 4.0 International License](LICENSE). The [UNI2-h](https://huggingface.co/MahmoodLab/UNI2-h) teacher is released under CC BY-NC-ND 4.0. The distillation process used UNI2-h only to generate supervisory outputs; the released student contains no UNI2-h weights. This model is intended solely for non-commercial academic research.

## Citation

```bibtex
@inproceedings{kaspar2026distillpath,
  title     = {DistillPath: An Efficient 22M Distilled Pathology Encoder Approaching Large Foundation Model Performance},
  author    = {Kaspar, Ramon and Ignatov, Andrey and Boeva, Valentina},
  booktitle = {Medical Foundation Models and Benchmarks (MedFM-Bench), ECCV 2026},
  year      = {2026}
}
```