README.md

---
license: other
license_name: kaiko-non-commercial-public-license
license_link: LICENSE
library_name: timm
pipeline_tag: image-feature-extraction
tags:
  - computational-pathology
  - knowledge-distillation
  - vision-transformer
  - medical-imaging
datasets:
  - TCGA
---

# DistillPath-KS16-Virchow2

A 22M ViT-S/16 pathology tile encoder distilled from [Virchow2](https://huggingface.co/paige-ai/Virchow2) (632M ViT-H/14) into the [kaiko ViT-S/16](https://huggingface.co/1aurent/vit_small_patch16_224.kaiko_ai_towards_large_pathology_fms) student using backbone-token distillation on 6,000 public TCGA slides.

For the ImageNet-initialized variant, see [DistillPath-IS16-Virchow2](https://huggingface.co/RamonK/DistillPath-IS16-Virchow2).

**Paper:** [DistillPath: An Efficient 22M Distilled Pathology Encoder Approaching Large Foundation Model Performance](https://arxiv.org/abs/2608.17872)
Ramon Kaspar, Andrey Ignatov, Valentina Boeva. ETH Zurich.
Published at the ECCV 2026 Workshop on Medical Foundation Models and Benchmarks (MedFM-Bench).

## Model details

| Property | Value |
|---|---|
| Architecture | ViT-S/16 (`vit_small_patch16_224` in timm) |
| Parameters | 21.7M |
| Feature dimension | 384 |
| Input size | 224 x 224 |
| Normalization | mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5) |
| Student initialization | kaiko ViT-S/16 (pathology-pretrained) |
| Teacher | Virchow2 (632M, ViT-H/14, CC BY-NC-ND 4.0) |
| Training data | 6,000 TCGA H&E whole-slide images, 32 cohorts |
| Training steps | 50,000 (batch size 256, approx. 24-29 GPU-hours on 1x RTX 4090) |

## Benchmark results

| Benchmark | **DistillPath-KS16-Virchow2** | kaiko baseline | Virchow2 teacher |
|---|---|---|---|
| EVA mean (7 tasks) | **0.795** | 0.764 | 0.810 |
| HEST mean (9 tasks) | **0.371** | 0.349 | 0.398 |
| PLISM score | **0.447** | 0.307 | 0.447 |

See the paper for per-task results and comparisons with all four DistillPath variants.

## Usage

Load directly from the Hub with timm:

```python
import timm

model = timm.create_model(
    "hf_hub:RamonK/DistillPath-KS16-Virchow2",
    pretrained=True,
    num_classes=0,
)
model.eval()
```

Or load manually:

```python
import timm
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file

model = timm.create_model("vit_small_patch16_224", pretrained=False, num_classes=0)
path = hf_hub_download("RamonK/DistillPath-KS16-Virchow2", "model.safetensors")
state_dict = load_file(path)
model.load_state_dict(state_dict, strict=True)
model.eval()
```

This model uses normalization mean=(0.5, 0.5, 0.5) and std=(0.5, 0.5, 0.5), inherited from the kaiko student. Apply this normalization to input tiles before inference:

```python
from torchvision import transforms

transform = transforms.Compose([
    transforms.Resize(224),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]),
])
```

## Distillation recipe

The recipe reads only the teacher's final class and patch tokens (no teacher pretraining heads required):
- **Class-token loss:** cosine distance + RKD (relational knowledge distillation)
- **Patch-token loss:** cosine distance after bicubic grid resizing (teacher 16x16 to student 14x14)
- **Optimizer:** AdamW, lr=1e-4, weight decay 0.05, cosine decay, 500 warmup steps
- **Projector:** DINO-style MLP (384 to 2048 to 2048 to 256 to d_t), discarded after training

Full details in the paper and the [DistillPath repository](https://github.com/RamonKaspar/DistillPath).

## License

This model is released under the [Kaiko Non-Commercial Public License](LICENSE), inherited from the kaiko ViT-S/16 student weights. The [Virchow2](https://huggingface.co/paige-ai/Virchow2) teacher is released under CC BY-NC-ND 4.0. The distillation process used Virchow2 only to generate supervisory outputs; the released student contains no Virchow2 weights. This model is intended solely for non-commercial academic research.

## Citation

```bibtex
@inproceedings{kaspar2026distillpath,
  title     = {DistillPath: An Efficient 22M Distilled Pathology Encoder Approaching Large Foundation Model Performance},
  author    = {Kaspar, Ramon and Ignatov, Andrey and Boeva, Valentina},
  booktitle = {Medical Foundation Models and Benchmarks (MedFM-Bench), ECCV 2026},
  year      = {2026}
}
```