--- license: cc-by-nc-sa-4.0 tags: - pathology - computational-pathology - histopathology - multiple-instance-learning - whole-slide-images - prostate-cancer - medical-imaging - pytorch pipeline_tag: image-classification --- # Artifact-aware MIL heads for prostate biopsy ISUP grading Trained multiple instance learning (MIL) heads from the paper *Aggregation, Ordinal Training, and Artifact Gating in Multiple Instance Learning for Prostate Biopsy Grading*. Each model maps a bag of frozen Virchow2 patch embeddings from one prostate biopsy whole-slide image to an ISUP grade group (0–5). Code, data split and result files: ## Contents 30 models, the members of every ensemble reported in the paper. Each folder holds `best_model.safetensors` (weights only), the run's recorded `config.json`, and `release_info.json` (best epoch, validation QWK, SHA-256). | Folder | Members | Aggregator | Loss | Gate | Protocol | | --- | --- | --- | --- | --- | --- | | `virchow2_transmil_focal_nogate_5seed/` | `seed0`–`seed4` | TransMIL-style, 2 layers | focal | none | fixed split, 5 seeds | | `virchow2_transmil_focal_maskgate_5seed/` | `seed0`–`seed4` | TransMIL-style, 2 layers | focal | mask-only | fixed split, 5 seeds | | `virchow2_transmil_focal_fullgate_5seed/` | `seed0`–`seed4` | TransMIL-style, 2 layers | focal | full robust | fixed split, 5 seeds | | `virchow2_abmil_focal_maskgate_5seed/` | `seed0`–`seed4` | ABMIL | focal | mask-only | fixed split, 5 seeds | | `virchow2_transmil_coral_maskgate_5fold/` | `fold_1`–`fold_5` | TransMIL-style, 2 layers | CORAL | mask-only | 5-fold CV | | `virchow2_transmil_coral_fullgate_5fold/` | `fold_1`–`fold_5` | TransMIL-style, 2 layers | CORAL | full robust | 5-fold CV | The mask-only and full robust gate variants implement the same model: flagged tokens are excluded from attention in both, so the attenuation factor α does not change the output. Their differences are run-to-run (replicate) differences. α was not logged for any run; the evaluation code uses the default α = 1e-4. Total size about 4.1 GB (TransMIL-style members 163 MB each, ABMIL members 6 MB each). The weights were exported from the training checkpoints with `scripts/training/export_weights.py` in the code repository; for every model the exported weights give identical logits to the original checkpoint. ## Input The models do **not** contain an image encoder. Each slide must be converted to a bag saved as a `.pt` file with: | Key | Shape | Content | | --- | --- | --- | | `embeddings` | `[N, 1280]` | Virchow2 CLS token of each tissue patch | | `spatial_variances` | `[N]` | variance of the Laplacian of the grayscale patch, used by the gate | | `coords` | `[N, 2]` | patch level-0 coordinates (used only for heatmaps) | Patches are 256 × 256 pixels at level 0 (nominal 20×), Macenko stain normalization with a fixed reference, resized to 224 × 224 for Virchow2. The code repository's `scripts/preprocessing/preprocess.py` and `scripts/feature_extraction/extract_features.py` produce these bags. **Virchow2 weights are not included.** Request access to [`paige-ai/Virchow2`](https://huggingface.co/paige-ai/Virchow2) and use it under its own license terms. ## Output - Focal-loss models: 6 logits (ISUP 0–5); class probabilities by softmax. - CORAL models: 5 ordinal threshold logits; class probabilities by differencing the cumulative sigmoids, clipping at 0 and renormalizing. Ensembles average the class probabilities of their members. ## Usage ```bash git clone https://github.com/PathOS-Code/artifact-aware-mil-prostate.git cd artifact-aware-mil-prostate uv sync hf download PathOS-Code/artifact-aware-mil-prostate --local-dir weights uv run python scripts/evaluation/ensemble_eval.py \ --exp-dirs weights/virchow2_transmil_focal_fullgate_5seed/seed{0,1,2,3,4} \ --embeddings-dir outputs/embeddings_virchow2 \ --labels-csv data/panda_labels.csv \ --split-csv splits/panda_fixed_split.csv \ --out outputs/ensembles/panda_focal_fullgate_5seed.json ``` For an external cohort, pass its embeddings and labels CSV with `--val-ratio 0 --test-ratio 0` to evaluate every slide. See `REPRODUCE.md` in the code repository for all experiments. ## Results From the paper and the result files in the code repository (`results/`). QWK = quadratic weighted kappa. PANDA held-out test split (1,061 slides; 10,614 slides overall from Radboud and Karolinska): | Ensemble | Test QWK | | --- | --- | | TransMIL-style, focal, no gate, 5 seeds | 0.9063 | | TransMIL-style, focal, mask-only, 5 seeds | 0.9104 | | TransMIL-style, focal, full robust gate, 5 seeds | 0.9134 (95% CI 0.8856–0.9368) | | ABMIL, focal, mask-only, 5 seeds | 0.8919 | PANDA five-fold cross-validation (out-of-fold QWK, optimistic): CORAL mask-only 0.9226, CORAL full robust gate 0.9227. External cohorts, direct transfer without fine-tuning: | Ensemble | SICAPv2 (155 biopsies) | CrowdGleason (232 slide identifiers) | | --- | --- | --- | | TransMIL-style, focal, no gate, 5 seeds | 0.8664 | 0.8076 | | TransMIL-style, focal, mask-only, 5 seeds | 0.8633 | 0.8028 | | TransMIL-style, focal, full robust gate, 5 seeds | 0.8482 | 0.8076 | | TransMIL-style, CORAL, mask-only, 5 folds | 0.8760 | 0.8109 | | TransMIL-style, CORAL, full robust gate, 5 folds | 0.8772 (95% CI 0.803–0.933) | 0.8080 (95% CI 0.733–0.871) | Caveats, as stated in the paper: - The PANDA test ensemble was the best of three ensembles, selected post hoc after exploratory runs that included the test slides. - None of the paired gated-versus-ungated QWK intervals excluded zero; no statistically detectable gating effect was found, and gate firing was not measured. - All intervals treat slides as independent and ignore patient clustering. - CrowdGleason slide labels are surrogates derived from patch labels. - Aggregator and ordinal-loss contrasts are confounded and descriptive. - Training is not bit-wise reproducible; retrained models differ slightly. ## Limitations - Developed on PANDA only (two institutions, routine-practice labels); external evaluation covers two small public cohorts. - Performance under other scanners, stains, tissue preparation or patient populations is unknown. - Each model predicts one ISUP grade group per slide. Attention heatmaps show which patches the model weighted, not where each Gleason pattern is; pattern-level segmentation is not part of this release. - Depends on Virchow2 embeddings; other encoders will not work with these weights. ## License The weights are released under [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/) because they were trained on PANDA (CC BY-NC-SA 4.0): noncommercial use, attribution required, derivatives under the same license. The code in the GitHub repository is under the PolyForm Noncommercial License 1.0.0. ## Citation ```bibtex @article{TODO, title = {Aggregation, Ordinal Training, and Artifact Gating in Multiple Instance Learning for Prostate Biopsy Grading}, author = {TODO}, journal = {TODO}, year = {TODO} } ```