README.md

---
license: cc-by-nc-4.0
library_name: timm
pipeline_tag: image-feature-extraction
tags:
  - pathology
  - histopathology
  - her2
  - breast-cancer
  - h-and-e
  - vision-transformer
  - multiple-instance-learning
  - safetensors
---

# HER2 reader: a small ViT-S/16 encoder and HER2 slide classifier for routine H&E

Model weights and output data behind the paper **"A small HER2 reader for routine H&E slides: real-world deployment"**
(Thang Tran, Lan Dang, 2026; arXiv link to follow). We publish them so that anyone can check our numbers, rerun our
evaluation and build on our encoder.

**Research use only.** This is not a medical device and must not be used for clinical decisions. Our reader proposes
a probability that a breast cancer is HER2-positive from its H&E slide, for a pathologist who reads it beside IHC and
ISH, never instead of them.

## What is in this repository

| Path | What it holds |
|---|---|
| `encoders/` | ViT-S/16 tile encoders (22M parameters) at every training stage, with timm parameter names |
| `checkpoints/` | the same models exactly as trained, in our original layout (including the self-distillation heads), for audit |
| `reader/her2-jepa2-t2y-s4/` | the deployed reader: encoder, five CLAM-SB fold heads, and a manifest with architecture, tile recipe, operating point and validation |
| `data/predictions/` | per-slide predictions, by seed: TCGA-BRCA out of fold, Yale-HER2 external, TCGA-BRCA + CPTAC-BRCA under the published protocol |
| `data/paper/` | data behind the paper's tables and figures |
| `CHECKSUMS.sha256` | SHA-256 of every file |

### Encoders

| File (`encoders/`) | Stage | TCGA-to-Yale AUC* |
|---|---|---|
| `stage1_ssl-vits16-final.safetensors` | self-supervised pretraining (DINOv2 + iBOT + KoLeo) on 98.6M TCGA tiles | 0.589 |
| `stage2_distilled-hoptimus0-step25000.safetensors` | distilled from H-optimus-0 (CLS + 4x4 patch targets), step 25,000 | 0.762 |
| `stage3a_jepa-round1.safetensors` | one JEPA refinement round | 0.798 |
| `stage3b_jepa-round2_deployed-encoder.safetensors` | second JEPA round: **the deployed encoder** | **0.812** |
| `r12a_distilled-1um-step25000.safetensors` | stage 3b distilled again at 1.0 um/px (not adopted) | 0.813** |
| `r12b_distilled-1um-jepa6000.safetensors` | r12a plus one JEPA round (not adopted) | 0.812** |

\* CLAM-SB reader trained on 490 TCGA-BRCA slides, tested on 192 Yale-HER2 slides, mean of 10 seeds (our protocol).
\*\* Under the published reader's protocol (TCGA + CPTAC training, all tiles, 40 attention-MIL models); the deployed
encoder scores 0.809 there.

## Load an encoder (timm)

```python
import timm
from safetensors.torch import load_file

enc = timm.models.vision_transformer.VisionTransformer(
    img_size=224, patch_size=16, embed_dim=384, depth=12, num_heads=6, mlp_ratio=4,
    qkv_bias=True, init_values=1.0, class_token=True, no_embed_class=False, reg_tokens=0,
    global_pool="token", fc_norm=False, num_classes=0)
enc.load_state_dict(load_file("encoders/stage3b_jepa-round2_deployed-encoder.safetensors"), strict=True)
enc.eval()
# input: float32 [N, 3, 224, 224], RGB, ImageNet mean (0.485, 0.456, 0.406) and std (0.229, 0.224, 0.225)
# output: [N, 384], the class token after the final LayerNorm
```

Tiles for our reader: 128 um squares read at 0.5 um/px (the coarsest pyramid level at or below 0.5 um/px), resized to
224 px by nearest neighbour, tissue only; up to 400 tiles per slide, taken at an even stride across the section. The
manifest in `reader/` gives every step.

## The deployed reader

Five CLAM-SB heads (5-fold cross-validation on 490 TCGA-BRCA slides, seed 4) read one bag of 400 tile vectors. The
reader's probability is the mean of the five heads' softmax probabilities; at or above **0.27228** it proposes
"positive". That threshold comes from TCGA-BRCA's out-of-fold predictions, and a laboratory should refit it on its own
slides, because scores shift between hospitals.

| Test set (never trained on) | AUC |
|---|---|
| Yale-HER2, 192 slides (5-model ensemble, seed 4) | 0.815 [0.753, 0.876] |
| BCNB, 1,058 core biopsies (mean of 10 seeds) | 0.628 |
| HEROHE test split, 150 slides (mean of 10 seeds) | 0.746 |
| HEROHE IHC 2+ cases, 85 slides (amplified vs not) | 0.761 |

At its threshold on Yale-HER2: sensitivity 0.72, specificity 0.79. On a 12-core CPU server it reads a slide in
5.2 to 5.6 s.

## Limitations

- Its 490 TCGA-BRCA training slides were size-biased: negatives carry less tissue than positives. This inflates the
  Yale AUC by about 0.04; on representative training sets our readers reach about 0.81.
- Probabilities run low at hospitals other than TCGA's; set the threshold per laboratory.
- On equivocal IHC 2+ cases (AUC 0.76 on HEROHE) it can help order and prioritise ISH. It does not replace ISH.
- The best published reader (H-optimus-0, 1.1B parameters, Yale AUC 0.907) remains well ahead. The gap lies in the
  encoder.

## Training data and attributions

- **Encoder:** TCGA diagnostic slides (NCI Genomic Data Commons, open access). Distillation teacher: H-optimus-0
  (Bioptimus), Apache-2.0. See `NOTICE`.
- **Reader heads:** 490 TCGA-BRCA slides with HER2 status from GDC clinical data.
- **Evaluation cohorts:** Yale-HER2 (TCIA HER2-TUMOR-ROIS, CC BY 4.0), CPTAC-BRCA (open), HEROHE (CC BY-NC-ND 3.0,
  evaluation only) and BCNB (non-commercial, evaluation only). Per-slide data for HEROHE and BCNB are not published
  here; only aggregate numbers appear in the paper.

## Licences

- Model weights (`encoders/`, `checkpoints/`, `reader/`): **CC BY-NC 4.0** (`LICENSE-weights.md`).
- Data (`data/`): **CC BY 4.0** (`LICENSE-data.md`).
- The distillation teacher's Apache-2.0 notice: `NOTICE`.

## Citation

```bibtex
@misc{tran2026her2reader,
  title  = {A small HER2 reader for routine H\&E slides: real-world deployment},
  author = {Tran, Thang and Dang, Lan},
  year   = {2026},
  note   = {Model weights and data: https://huggingface.co/cloudkites/her2-reader}
}
```