akramoo's picture
Update README.md
8a8225e verified
|
Raw History Blame Contribute Delete
7.13 kB
---
license: cc-by-nc-sa-4.0
tags:
- pathology
- computational-pathology
- histopathology
- multiple-instance-learning
- whole-slide-images
- prostate-cancer
- medical-imaging
- pytorch
pipeline_tag: image-classification
---
# Artifact-aware MIL heads for prostate biopsy ISUP grading
Trained multiple instance learning (MIL) heads from the paper *Aggregation,
Ordinal Training, and Artifact Gating in Multiple Instance Learning for
Prostate Biopsy Grading*. Each model maps a bag of frozen Virchow2 patch
embeddings from one prostate biopsy whole-slide image to an ISUP grade group
(0–5).
Code, data split and result files: <https://github.com/PathOS-Code/artifact-aware-mil-prostate>
## Contents
30 models, the members of every ensemble reported in the paper. Each folder
holds `best_model.safetensors` (weights only), the run's recorded
`config.json`, and `release_info.json` (best epoch, validation QWK, SHA-256).
| Folder | Members | Aggregator | Loss | Gate | Protocol |
| --- | --- | --- | --- | --- | --- |
| `virchow2_transmil_focal_nogate_5seed/` | `seed0`–`seed4` | TransMIL-style, 2 layers | focal | none | fixed split, 5 seeds |
| `virchow2_transmil_focal_maskgate_5seed/` | `seed0`–`seed4` | TransMIL-style, 2 layers | focal | mask-only | fixed split, 5 seeds |
| `virchow2_transmil_focal_fullgate_5seed/` | `seed0`–`seed4` | TransMIL-style, 2 layers | focal | full robust | fixed split, 5 seeds |
| `virchow2_abmil_focal_maskgate_5seed/` | `seed0`–`seed4` | ABMIL | focal | mask-only | fixed split, 5 seeds |
| `virchow2_transmil_coral_maskgate_5fold/` | `fold_1`–`fold_5` | TransMIL-style, 2 layers | CORAL | mask-only | 5-fold CV |
| `virchow2_transmil_coral_fullgate_5fold/` | `fold_1`–`fold_5` | TransMIL-style, 2 layers | CORAL | full robust | 5-fold CV |
The mask-only and full robust gate variants implement the same model: flagged
tokens are excluded from attention in both, so the attenuation factor α does
not change the output. Their differences are run-to-run (replicate)
differences. α was not logged for any run; the evaluation code uses the
default α = 1e-4.
Total size about 4.1 GB (TransMIL-style members 163 MB each, ABMIL members
6 MB each). The weights were exported from the training checkpoints with
`scripts/training/export_weights.py` in the code repository; for every model
the exported weights give identical logits to the original checkpoint.
## Input
The models do **not** contain an image encoder. Each slide must be converted
to a bag saved as a `.pt` file with:
| Key | Shape | Content |
| --- | --- | --- |
| `embeddings` | `[N, 1280]` | Virchow2 CLS token of each tissue patch |
| `spatial_variances` | `[N]` | variance of the Laplacian of the grayscale patch, used by the gate |
| `coords` | `[N, 2]` | patch level-0 coordinates (used only for heatmaps) |
Patches are 256 × 256 pixels at level 0 (nominal 20×), Macenko stain
normalization with a fixed reference, resized to 224 × 224 for Virchow2. The
code repository's `scripts/preprocessing/preprocess.py` and
`scripts/feature_extraction/extract_features.py` produce these bags.
**Virchow2 weights are not included.** Request access to
[`paige-ai/Virchow2`](https://huggingface.co/paige-ai/Virchow2) and use it
under its own license terms.
## Output
- Focal-loss models: 6 logits (ISUP 0–5); class probabilities by softmax.
- CORAL models: 5 ordinal threshold logits; class probabilities by
differencing the cumulative sigmoids, clipping at 0 and renormalizing.
Ensembles average the class probabilities of their members.
## Usage
```bash
git clone https://github.com/PathOS-Code/artifact-aware-mil-prostate.git
cd artifact-aware-mil-prostate
uv sync
hf download PathOS-Code/artifact-aware-mil-prostate --local-dir weights
uv run python scripts/evaluation/ensemble_eval.py \
--exp-dirs weights/virchow2_transmil_focal_fullgate_5seed/seed{0,1,2,3,4} \
--embeddings-dir outputs/embeddings_virchow2 \
--labels-csv data/panda_labels.csv \
--split-csv splits/panda_fixed_split.csv \
--out outputs/ensembles/panda_focal_fullgate_5seed.json
```
For an external cohort, pass its embeddings and labels CSV with
`--val-ratio 0 --test-ratio 0` to evaluate every slide. See `REPRODUCE.md` in
the code repository for all experiments.
## Results
From the paper and the result files in the code repository (`results/`).
QWK = quadratic weighted kappa.
PANDA held-out test split (1,061 slides; 10,614 slides overall from Radboud
and Karolinska):
| Ensemble | Test QWK |
| --- | --- |
| TransMIL-style, focal, no gate, 5 seeds | 0.9063 |
| TransMIL-style, focal, mask-only, 5 seeds | 0.9104 |
| TransMIL-style, focal, full robust gate, 5 seeds | 0.9134 (95% CI 0.8856–0.9368) |
| ABMIL, focal, mask-only, 5 seeds | 0.8919 |
PANDA five-fold cross-validation (out-of-fold QWK, optimistic): CORAL
mask-only 0.9226, CORAL full robust gate 0.9227.
External cohorts, direct transfer without fine-tuning:
| Ensemble | SICAPv2 (155 biopsies) | CrowdGleason (232 slide identifiers) |
| --- | --- | --- |
| TransMIL-style, focal, no gate, 5 seeds | 0.8664 | 0.8076 |
| TransMIL-style, focal, mask-only, 5 seeds | 0.8633 | 0.8028 |
| TransMIL-style, focal, full robust gate, 5 seeds | 0.8482 | 0.8076 |
| TransMIL-style, CORAL, mask-only, 5 folds | 0.8760 | 0.8109 |
| TransMIL-style, CORAL, full robust gate, 5 folds | 0.8772 (95% CI 0.803–0.933) | 0.8080 (95% CI 0.733–0.871) |
Caveats, as stated in the paper:
- The PANDA test ensemble was the best of three ensembles, selected post hoc
after exploratory runs that included the test slides.
- None of the paired gated-versus-ungated QWK intervals excluded zero; no
statistically detectable gating effect was found, and gate firing was not
measured.
- All intervals treat slides as independent and ignore patient clustering.
- CrowdGleason slide labels are surrogates derived from patch labels.
- Aggregator and ordinal-loss contrasts are confounded and descriptive.
- Training is not bit-wise reproducible; retrained models differ slightly.
## Limitations
- Developed on PANDA only (two institutions, routine-practice labels);
external evaluation covers two small public cohorts.
- Performance under other scanners, stains, tissue preparation or patient
populations is unknown.
- Each model predicts one ISUP grade group per slide. Attention heatmaps
show which patches the model weighted, not where each Gleason pattern is;
pattern-level segmentation is not part of this release.
- Depends on Virchow2 embeddings; other encoders will not work with these
weights.
## License
The weights are released under
[CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)
because they were trained on PANDA (CC BY-NC-SA 4.0): noncommercial use,
attribution required, derivatives under the same license. The code in the
GitHub repository is under the PolyForm Noncommercial License 1.0.0.
## Citation
```bibtex
@article{TODO,
title = {Aggregation, Ordinal Training, and Artifact Gating in Multiple Instance Learning for Prostate Biopsy Grading},
author = {TODO},
journal = {TODO},
year = {TODO}
}
```