File size: 7,134 Bytes
fa4163f 9baf699 afbfba6 fa4163f 9baf699 8a8225e 9baf699 afbfba6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 | ---
license: cc-by-nc-sa-4.0
tags:
- pathology
- computational-pathology
- histopathology
- multiple-instance-learning
- whole-slide-images
- prostate-cancer
- medical-imaging
- pytorch
pipeline_tag: image-classification
---
# Artifact-aware MIL heads for prostate biopsy ISUP grading
Trained multiple instance learning (MIL) heads from the paper *Aggregation,
Ordinal Training, and Artifact Gating in Multiple Instance Learning for
Prostate Biopsy Grading*. Each model maps a bag of frozen Virchow2 patch
embeddings from one prostate biopsy whole-slide image to an ISUP grade group
(0–5).
Code, data split and result files: <https://github.com/PathOS-Code/artifact-aware-mil-prostate>
## Contents
30 models, the members of every ensemble reported in the paper. Each folder
holds `best_model.safetensors` (weights only), the run's recorded
`config.json`, and `release_info.json` (best epoch, validation QWK, SHA-256).
| Folder | Members | Aggregator | Loss | Gate | Protocol |
| --- | --- | --- | --- | --- | --- |
| `virchow2_transmil_focal_nogate_5seed/` | `seed0`–`seed4` | TransMIL-style, 2 layers | focal | none | fixed split, 5 seeds |
| `virchow2_transmil_focal_maskgate_5seed/` | `seed0`–`seed4` | TransMIL-style, 2 layers | focal | mask-only | fixed split, 5 seeds |
| `virchow2_transmil_focal_fullgate_5seed/` | `seed0`–`seed4` | TransMIL-style, 2 layers | focal | full robust | fixed split, 5 seeds |
| `virchow2_abmil_focal_maskgate_5seed/` | `seed0`–`seed4` | ABMIL | focal | mask-only | fixed split, 5 seeds |
| `virchow2_transmil_coral_maskgate_5fold/` | `fold_1`–`fold_5` | TransMIL-style, 2 layers | CORAL | mask-only | 5-fold CV |
| `virchow2_transmil_coral_fullgate_5fold/` | `fold_1`–`fold_5` | TransMIL-style, 2 layers | CORAL | full robust | 5-fold CV |
The mask-only and full robust gate variants implement the same model: flagged
tokens are excluded from attention in both, so the attenuation factor α does
not change the output. Their differences are run-to-run (replicate)
differences. α was not logged for any run; the evaluation code uses the
default α = 1e-4.
Total size about 4.1 GB (TransMIL-style members 163 MB each, ABMIL members
6 MB each). The weights were exported from the training checkpoints with
`scripts/training/export_weights.py` in the code repository; for every model
the exported weights give identical logits to the original checkpoint.
## Input
The models do **not** contain an image encoder. Each slide must be converted
to a bag saved as a `.pt` file with:
| Key | Shape | Content |
| --- | --- | --- |
| `embeddings` | `[N, 1280]` | Virchow2 CLS token of each tissue patch |
| `spatial_variances` | `[N]` | variance of the Laplacian of the grayscale patch, used by the gate |
| `coords` | `[N, 2]` | patch level-0 coordinates (used only for heatmaps) |
Patches are 256 × 256 pixels at level 0 (nominal 20×), Macenko stain
normalization with a fixed reference, resized to 224 × 224 for Virchow2. The
code repository's `scripts/preprocessing/preprocess.py` and
`scripts/feature_extraction/extract_features.py` produce these bags.
**Virchow2 weights are not included.** Request access to
[`paige-ai/Virchow2`](https://huggingface.co/paige-ai/Virchow2) and use it
under its own license terms.
## Output
- Focal-loss models: 6 logits (ISUP 0–5); class probabilities by softmax.
- CORAL models: 5 ordinal threshold logits; class probabilities by
differencing the cumulative sigmoids, clipping at 0 and renormalizing.
Ensembles average the class probabilities of their members.
## Usage
```bash
git clone https://github.com/PathOS-Code/artifact-aware-mil-prostate.git
cd artifact-aware-mil-prostate
uv sync
hf download PathOS-Code/artifact-aware-mil-prostate --local-dir weights
uv run python scripts/evaluation/ensemble_eval.py \
--exp-dirs weights/virchow2_transmil_focal_fullgate_5seed/seed{0,1,2,3,4} \
--embeddings-dir outputs/embeddings_virchow2 \
--labels-csv data/panda_labels.csv \
--split-csv splits/panda_fixed_split.csv \
--out outputs/ensembles/panda_focal_fullgate_5seed.json
```
For an external cohort, pass its embeddings and labels CSV with
`--val-ratio 0 --test-ratio 0` to evaluate every slide. See `REPRODUCE.md` in
the code repository for all experiments.
## Results
From the paper and the result files in the code repository (`results/`).
QWK = quadratic weighted kappa.
PANDA held-out test split (1,061 slides; 10,614 slides overall from Radboud
and Karolinska):
| Ensemble | Test QWK |
| --- | --- |
| TransMIL-style, focal, no gate, 5 seeds | 0.9063 |
| TransMIL-style, focal, mask-only, 5 seeds | 0.9104 |
| TransMIL-style, focal, full robust gate, 5 seeds | 0.9134 (95% CI 0.8856–0.9368) |
| ABMIL, focal, mask-only, 5 seeds | 0.8919 |
PANDA five-fold cross-validation (out-of-fold QWK, optimistic): CORAL
mask-only 0.9226, CORAL full robust gate 0.9227.
External cohorts, direct transfer without fine-tuning:
| Ensemble | SICAPv2 (155 biopsies) | CrowdGleason (232 slide identifiers) |
| --- | --- | --- |
| TransMIL-style, focal, no gate, 5 seeds | 0.8664 | 0.8076 |
| TransMIL-style, focal, mask-only, 5 seeds | 0.8633 | 0.8028 |
| TransMIL-style, focal, full robust gate, 5 seeds | 0.8482 | 0.8076 |
| TransMIL-style, CORAL, mask-only, 5 folds | 0.8760 | 0.8109 |
| TransMIL-style, CORAL, full robust gate, 5 folds | 0.8772 (95% CI 0.803–0.933) | 0.8080 (95% CI 0.733–0.871) |
Caveats, as stated in the paper:
- The PANDA test ensemble was the best of three ensembles, selected post hoc
after exploratory runs that included the test slides.
- None of the paired gated-versus-ungated QWK intervals excluded zero; no
statistically detectable gating effect was found, and gate firing was not
measured.
- All intervals treat slides as independent and ignore patient clustering.
- CrowdGleason slide labels are surrogates derived from patch labels.
- Aggregator and ordinal-loss contrasts are confounded and descriptive.
- Training is not bit-wise reproducible; retrained models differ slightly.
## Limitations
- Developed on PANDA only (two institutions, routine-practice labels);
external evaluation covers two small public cohorts.
- Performance under other scanners, stains, tissue preparation or patient
populations is unknown.
- Each model predicts one ISUP grade group per slide. Attention heatmaps
show which patches the model weighted, not where each Gleason pattern is;
pattern-level segmentation is not part of this release.
- Depends on Virchow2 embeddings; other encoders will not work with these
weights.
## License
The weights are released under
[CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)
because they were trained on PANDA (CC BY-NC-SA 4.0): noncommercial use,
attribution required, derivatives under the same license. The code in the
GitHub repository is under the PolyForm Noncommercial License 1.0.0.
## Citation
```bibtex
@article{TODO,
title = {Aggregation, Ordinal Training, and Artifact Gating in Multiple Instance Learning for Prostate Biopsy Grading},
author = {TODO},
journal = {TODO},
year = {TODO}
}
``` |