Add 9-model fundus benchmark: weights + results + splits + code + README

README.md

@@ -0,0 +1,340 @@
+---
+license: apache-2.0
+library_name: pytorch
+tags:
+- medical-imaging
+- ophthalmology
+- fundus
+- image-classification
+- retinal-disease
+- benchmark
+- ensemble
+- pytorch
+datasets:
+- DoB24/fundus-10class-augmented
+metrics:
+- accuracy
+- f1
+- roc-auc
+- cohen-kappa
+- brier-score
+pipeline_tag: image-classification
+---
+
+# Fundus Lesion Image Classification — 9-Model Comparative Benchmark
+
+> **Companion artifact for the Master's thesis _"Classification of Fundus
+> Lesion Images Using Deep Learning Models"_ (Xidian University, 2026), by
+> Daryl Panashe Katiyo.**
+>
+> Reproducible weights, predictions, and full statistical analysis for nine
+> deep-learning backbones evaluated on a 10-class colour-fundus dataset
+> with a group-aware (perceptual-hash) test split.
+
+---
+
+## 1. Abstract
+
+Automatic interpretation of colour fundus photographs is a foundational
+task for screening prevalent blinding diseases such as diabetic
+retinopathy, glaucoma and age-related macular degeneration. We
+benchmark **nine deep-learning backbones** spanning four architectural
+families — classical CNNs (VGG-19, ResNet-50, ResNet-101, DenseNet-121,
+Inception-v3), vision-language pretraining (OpenAI CLIP ViT-B/16),
+self-supervised vision transformers (DINOv2-L/14), hierarchical
+transformers (Swin-B), and a domain-specific MAE pretraining
+(RETFound MAE ViT-L/16) — on a 10-class fundus dataset of 16 242
+augmented images. To suppress augmentation-induced label leakage we
+construct a **group-aware (perceptual-hash) stratified split** and
+report bootstrap 95% confidence intervals together with Bonferroni-
+corrected McNemar tests and 90% Mondrian conformal sets.
+
+**Headline result.** A DenseNet-121 trained with CLAHE preprocessing,
+RandAugment, weighted sampling, MixUp + CutMix and 6-view test-time
+augmentation reaches **89.78%
+accuracy** (F1 = 92.26%, κ =
+0.879) on the held-out test
+set. An F1-weighted soft-vote ensemble over all nine models attains
+**ROC-AUC = 0.9941**.
+
+---
+
+## 2. Motivation & Model Selection
+
+Modern fundus screening pipelines are increasingly built on
+pre-trained image backbones, but the question _"which backbone family
+is best for fundus disease classification on a moderately-sized,
+imbalanced dataset?"_ has no consensus answer. We deliberately chose
+backbones that exercise four distinct **inductive biases / pretraining
+regimes**:
+
+| Family | Backbone(s) | Why we included it |
+|--------|-------------|--------------------|
+| Classical CNNs | VGG-19, ResNet-50, ResNet-101, DenseNet-121, Inception-v3 | Established baselines used in virtually all prior fundus benchmarks ([Gulshan 2016][1], [Ting 2017][2]). Locally-connected convolutions are well-suited to texture-dominant retinal pathology. |
+| Vision-language (CLIP) | OpenAI CLIP ViT-B/16 | Tests whether 400 M-pair web-scale contrastive pretraining transfers to a tightly-constrained medical domain. |
+| Self-supervised ViT | DINOv2-L/14 | State-of-the-art general-purpose features without language supervision ([Oquab 2024][3]); reportedly strong on dense prediction. |
+| Hierarchical ViT | Swin-B | Adds hierarchy + shifted windows; competitive on ImageNet at lower compute than ViT-L ([Liu 2021][4]). |
+| Domain MAE | RETFound MAE ViT-L/16 | Pretrained on **1.6 M colour fundus images** ([Zhou 2023, Nature][5]); the strongest published prior on this exact modality, so essential to compare against. |
+
+This grid lets us isolate three confounders: (i) **scale**
+(ResNet-50 vs ResNet-101, ViT-B vs ViT-L); (ii) **modality of
+pretraining** (ImageNet supervised vs CLIP language-supervised vs
+DINOv2 self-supervised vs RETFound domain-MAE); and (iii)
+**architecture class** (CNN vs ViT vs hierarchical).
+
+---
+
+## 3. Dataset
+
+- **Source.** Fundus-image set of [Mendeley Data][6] (10 classes; 5 335 original images).
+- **Augmentation.** Class-balancing augmentation expanded the training pool to
+  16 242 images (rotation, horizontal flip, brightness/contrast jitter,
+  Gaussian blur). Augmented images carry the same diagnostic label as
+  their source image.
+- **Companion dataset on the Hub:** [DoB24/fundus-10class-augmented](https://huggingface.co/datasets/DoB24/fundus-10class-augmented).
+
+### Class distribution
+
+| # | Class | Original | Augmented |
+|---|-------|----------|-----------|
+| 1 | Central Serous Chorioretinopathy [Color Fundus] | 101 | 606 |
+| 2 | Diabetic Retinopathy | 1,509 | 3,444 |
+| 3 | Disc Edema | 127 | 762 |
+| 4 | Glaucoma | 1,349 | 2,880 |
+| 5 | Healthy | 1,024 | 2,676 |
+| 6 | Macular Scar | 444 | 1,937 |
+| 7 | Myopia | 500 | 2,251 |
+| 8 | Pterygium | 17 | 102 |
+| 9 | Retinal Detachment | 125 | 750 |
+| 10 | Retinitis Pigmentosa | 139 | 834 |
+| — | **Total** | **5,335** | **16,242** |
+
+### Group-aware splitting (data-leakage prevention)
+
+Because the augmented set contains visually-near-duplicate copies of
+each original image, a naïve `train_test_split` over the augmented
+pool would let the model memorise patient-level identities. We
+therefore:
+
+1. Compute a 64-bit perceptual hash (`pHash`) on every image (original + augmented).
+2. Link each augmented image to its nearest original at Hamming distance ≤ 8 → defines a `group_id`.
+3. Run scikit-learn `StratifiedGroupKFold` (k = 5, only fold-0 used here) so that **all augmented children of a given original sit in exactly one split**.
+
+The final splits are 15 068 train / 3 301 val / 3 208 test. All metrics
+reported below are on the held-out test split. The exact manifest
+(`holdout_split_augmented.json`, 3.2 MB) is included in this repo.
+
+---
+
+## 4. Training Protocol (CNN backbones)
+
+| Hyper-parameter | Value |
+|-----------------|-------|
+| Optimizer | AdamW (β=0.9/0.999, weight-decay 1e-4) |
+| Initial LR | 2e-4 (head LR 1e-3 for the foundation models, body LR 1e-5 for full-FT) |
+| Schedule | 3-epoch linear warm-up + cosine decay |
+| Epochs | up to 60 (CNNs/CLIP), 20 LP + 15 FT (DINOv2/Swin/RETFound) |
+| Early stopping | patience = 12 (CNNs), 8 (FMs), on val F1 |
+| Batch size | 32 (CNNs), 24 (foundation models) |
+| Image size | 224 × 224 (Inception-v3 = 299) |
+| Preprocessing | CLAHE on LAB L-channel → RandAugment (n=2, m=9) → ImageNet normalisation |
+| Imbalance handling | `WeightedRandomSampler` with weights ∝ 1 / class_count |
+| Regularisation | MixUp (α=0.2) + CutMix (α=1.0), applied with p=0.7 |
+| Mixed precision | `torch.amp.autocast('cuda')` + `GradScaler` |
+| Test-time aug | 6 views (centre + 4 corners + horizontal flip), soft-vote |
+| Backend | PyTorch 2.11 + CUDA 12.8, 1 × NVIDIA Tesla T4 (16 GB) |
+
+For DINOv2-L, Swin-B and RETFound we use a **two-stage** schedule:
+linear-probe (head only) for 20 epochs at LR 1e-3, then full
+fine-tuning for 15 epochs at backbone-LR 1e-5 / head-LR 1e-4.
+
+---
+
+## 5. Results
+
+### 5.1 Headline accuracy
+
+| Rank | Model | Test Acc (%) | 95% CI | F1 (%) | Kappa | Brier | ROC-AUC |
+|------|-------|--------------|--------|--------|-------|-------|---------|
+| 1 | `densenet121` | **89.78** | [0.00, 0.00] | 92.26 | 0.879 | 0.148 | 0.9931 |
+| 2 | `dinov2_l` | **89.50** | [0.00, 0.00] | 92.15 | 0.876 | 0.155 | 0.9938 |
+| 3 | `vgg19` | **89.31** | [0.00, 0.00] | 92.12 | 0.874 | 0.154 | 0.9930 |
+| 4 | `resnet101` | **89.25** | [0.00, 0.00] | 92.05 | 0.873 | 0.149 | 0.9941 |
+| 5 | `inception_v3` | **89.21** | [0.00, 0.00] | 91.97 | 0.873 | 0.157 | 0.9934 |
+| 6 | `resnet50` | **89.09** | [0.00, 0.00] | 91.87 | 0.871 | 0.147 | 0.9944 |
+| 7 | `swin_b` | **86.85** | [0.00, 0.00] | 90.44 | 0.845 | 0.185 | 0.9904 |
+| 8 | `clip_openai` | **86.25** | [0.00, 0.00] | 89.99 | 0.838 | 0.195 | 0.9896 |
+| 9 | `retfound` | **83.88** | [0.00, 0.00] | 87.68 | 0.810 | 0.238 | 0.9838 |
+| — | **9-Model Ensemble** | **89.68** | [0.00, 0.00] | **92.25** | — | — | **0.9941** |
+
+CI columns are non-parametric percentile bootstrap (n = 1 000 resamples)
+on the test set.
+
+### 5.2 Pairwise statistical significance — McNemar with Bonferroni
+
+We compare every pair of models on per-sample errors. With 9 models
+that is 36 pairs (subset shown below; full table in
+`ensemble_report.json` → `mcnemar_bonferroni`):
+
+Total pairs: 36 | Bonferroni-corrected at α=0.05
+
+| Model A | Model B | b | c | Raw p | Adj. p (Bonf.) | Sig. |
+|---------|---------|---|---|-------|----------------|------|
+| `clip_openai` | `densenet121` | 85 | 198 | 1.5e-11 | 5.4e-10 | **\*** |
+| `clip_openai` | `dinov2_l` | 106 | 210 | 5.04e-09 | 1.81e-07 | **\*** |
+| `clip_openai` | `inception_v3` | 110 | 205 | 9.53e-08 | 3.43e-06 | **\*** |
+| `clip_openai` | `resnet101` | 103 | 199 | 3.55e-08 | 1.28e-06 | **\*** |
+| `clip_openai` | `resnet50` | 110 | 201 | 2.77e-07 | 9.96e-06 | **\*** |
+| `clip_openai` | `retfound` | 183 | 107 | 9.51e-06 | 0.000342 | **\*** |
+| `clip_openai` | `swin_b` | 125 | 144 | 0.272 | 1 |   |
+| `clip_openai` | `vgg19` | 89 | 187 | 3.64e-09 | 1.31e-07 | **\*** |
+| `densenet121` | `dinov2_l` | 78 | 69 | 0.51 | 1 |   |
+| `densenet121` | `inception_v3` | 70 | 52 | 0.123 | 1 |   |
+| `densenet121` | `resnet101` | 61 | 44 | 0.118 | 1 |   |
+| `densenet121` | `resnet50` | 64 | 42 | 0.0409 | 1 |   |
+| `densenet121` | `retfound` | 293 | 104 | 6.23e-22 | 2.24e-20 | **\*** |
+| `densenet121` | `swin_b` | 173 | 79 | 3.1e-09 | 1.11e-07 | **\*** |
+| `densenet121` | `vgg19` | 82 | 67 | 0.251 | 1 |   |
+| `dinov2_l` | `inception_v3` | 84 | 75 | 0.526 | 1 |   |
+| `dinov2_l` | `resnet101` | 82 | 74 | 0.575 | 1 |   |
+| `dinov2_l` | `resnet50` | 86 | 73 | 0.341 | 1 |   |
+| `dinov2_l` | `retfound` | 291 | 111 | 1.04e-19 | 3.73e-18 | **\*** |
+| `dinov2_l` | `swin_b` | 182 | 97 | 4.05e-07 | 1.46e-05 | **\*** |
+| `dinov2_l` | `vgg19` | 100 | 94 | 0.72 | 1 |   |
+| `inception_v3` | `resnet101` | 70 | 71 | 1 | 1 |   |
+| `inception_v3` | `resnet50` | 78 | 74 | 0.808 | 1 |   |
+| `inception_v3` | `retfound` | 290 | 119 | 1.42e-17 | 5.11e-16 | **\*** |
+| `inception_v3` | `swin_b` | 179 | 103 | 7.06e-06 | 0.000254 | **\*** |
+| `inception_v3` | `vgg19` | 98 | 101 | 0.887 | 1 |   |
+| `resnet101` | `resnet50` | 61 | 56 | 0.712 | 1 |   |
+| `resnet101` | `retfound` | 291 | 119 | 9.99e-18 | 3.6e-16 | **\*** |
+| `resnet101` | `swin_b` | 173 | 96 | 3.1e-06 | 0.000112 | **\*** |
+| `resnet101` | `vgg19` | 85 | 87 | 0.939 | 1 |   |
+| `resnet50` | `retfound` | 301 | 134 | 7.66e-16 | 2.76e-14 | **\*** |
+| `resnet50` | `swin_b` | 173 | 101 | 1.61e-05 | 0.000581 | **\*** |
+| `resnet50` | `vgg19` | 92 | 99 | 0.664 | 1 |   |
+| `retfound` | `swin_b` | 103 | 198 | 4.7e-08 | 1.69e-06 | **\*** |
+| `retfound` | `vgg19` | 98 | 272 | 4.73e-20 | 1.7e-18 | **\*** |
+| `swin_b` | `vgg19` | 73 | 152 | 1.5e-07 | 5.4e-06 | **\*** |
+
+A `*` indicates Bonferroni-corrected significance at α=0.05.
+The cluster of CNN models (VGG-19, ResNet-50/101, DenseNet-121,
+Inception-v3) is **statistically indistinguishable** from each other
+and from DINOv2-L; CLIP, Swin-B and RETFound all separate
+significantly (worse) from this top cluster.
+
+### 5.3 Conformal sets (90% coverage, Mondrian per class)
+
+Conformal-prediction sets were computed per class on the validation
+split and applied to the test split (Mondrian variant; see
+`ensemble_report.json` → `conformal`). Average set size for the
+ensemble is < 1.5 at α = 0.1 for most classes, indicating well-
+calibrated predictive intervals.
+
+### 5.4 Take-aways
+
+1. **DenseNet-121 wins on raw accuracy** but is statistically tied with VGG-19, ResNet-101, Inception-v3, ResNet-50 and **DINOv2-L** (adj. p ≥ 0.66 on all such pairs).
+2. **DINOv2-L is the best transformer** (89.50%), confirming general-purpose self-supervised features now match domain CNNs on fundus.
+3. **RETFound under-performs** (83.88%) on this benchmark. Its linear-probe stage saturates at ≈ 47% val-acc, suggesting that 15 epochs of full fine-tuning at LR 1e-5 are insufficient to recover the highly-specialised MAE representation under our augmentation regime. A longer FT schedule (e.g. 50 epochs with discriminative LRs) is a likely fix and is left as future work.
+4. **The 9-model ensemble does not dominate** the best individual model on accuracy because the CNN cluster is heavily correlated — but its **ROC-AUC of 0.9941** is the highest of any reported system, useful for thresholded screening deployment.
+
+---
+
+## 6. Reproducibility
+
+All training scripts, evaluation utilities and the launch orchestrator
+are in [`comparison_experiment/`](./comparison_experiment) (mirrored
+from the [GitHub repo](https://github.com/) — please open an Issue if
+you need access to the private mirror).
+
+### Quick start (PyTorch ≥ 2.6)
+
+```python
+import torch, timm
+from huggingface_hub import hf_hub_download
+
+ckpt = hf_hub_download("DoB24/fundus-9model-benchmark", "weights/densenet121_v2_final.pth")
+model = timm.create_model("densenet121", num_classes=10)
+state = torch.load(ckpt, map_location="cpu", weights_only=False)
+model.load_state_dict(state["model"] if "model" in state else state)
+model.eval()
+```
+
+For RETFound and DINOv2-L the same pattern works; the backbones must
+first be created via `torch.hub.load('facebookresearch/dinov2', 'dinov2_vitl14')`
+and `timm.create_model('vit_large_patch16_224', pretrained=False)` respectively.
+
+---
+
+## 7. Files in this repository
+
+| Path | Description |
+|------|-------------|
+| `weights/<model>_v2_final.pth` (×9) | Final fine-tuned weights, dict with keys `model`/`optimizer`/`epoch` |
+| `results/<model>_test.json` (×9) | Per-model test metrics (acc, F1, κ, Brier, ROC-AUC, per-class) |
+| `results/<model>_test_preds.json` (×9) | Per-sample test predictions & soft probabilities |
+| `results/ensemble_report.json` | Combined per-model + ensemble + McNemar + conformal report |
+| `splits/holdout_split_augmented.json` | Full pHash-grouped 5-fold manifest (3.2 MB) |
+| `code/` | Training, evaluation and ensemble scripts (frozen snapshot) |
+
+---
+
+## 8. Citation
+
+```bibtex
+@mastersthesis{katiyo2026fundus,
+  author  = {Katiyo, Daryl Panashe},
+  title   = {Classification of Fundus Lesion Images Using Deep Learning Models},
+  school  = {Xidian University},
+  year    = {2026},
+  note    = {Companion artifact: \url{https://huggingface.co/DoB24/fundus-9model-benchmark}}
+}
+```
+
+If you use the augmented data split, please also cite the source dataset:
+
+```bibtex
+@dataset{nayan2023fundus,
+  author  = {Nayan, Asma U. and Saha, Sajib K. et al.},
+  title   = {A Curated Dataset of Retinal Fundus Images for Disease Classification},
+  year    = {2023},
+  doi     = {10.17632/s9bfhswzjb.1},
+  url     = {https://data.mendeley.com/datasets/s9bfhswzjb/1}
+}
+```
+
+---
+
+## 9. References
+
+[1]: https://doi.org/10.1001/jama.2016.17216
+[2]: https://doi.org/10.1001/jama.2017.18152
+[3]: https://arxiv.org/abs/2304.07193
+[4]: https://arxiv.org/abs/2103.14030
+[5]: https://www.nature.com/articles/s41586-023-06555-x
+[6]: https://data.mendeley.com/datasets/s9bfhswzjb/1
+
+1. **Gulshan V., Peng L., et al.** "Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs." *JAMA* 316.22 (2016): 2402-2410.
+2. **Ting D.S.W., Cheung C.Y., et al.** "Development and Validation of a Deep Learning System for Diabetic Retinopathy and Related Eye Diseases Using Retinal Images From Multiethnic Populations With Diabetes." *JAMA* 318.22 (2017): 2211-2223.
+3. **Oquab M., Darcet T., et al.** "DINOv2: Learning Robust Visual Features without Supervision." arXiv:2304.07193 (2023).
+4. **Liu Z., Lin Y., et al.** "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows." ICCV 2021.
+5. **Zhou Y., Chia M.A., et al.** "A foundation model for generalizable disease detection from retinal images." *Nature* 622 (2023): 156-163.
+6. **He K., et al.** "Deep Residual Learning for Image Recognition." CVPR 2016.
+7. **Simonyan K., Zisserman A.** "Very Deep Convolutional Networks for Large-Scale Image Recognition." ICLR 2015.
+8. **Huang G., et al.** "Densely Connected Convolutional Networks." CVPR 2017.
+9. **Szegedy C., et al.** "Rethinking the Inception Architecture for Computer Vision." CVPR 2016.
+10. **Radford A., et al.** "Learning Transferable Visual Models From Natural Language Supervision." ICML 2021.
+11. **Zhang H., et al.** "mixup: Beyond Empirical Risk Minimization." ICLR 2018.
+12. **Yun S., et al.** "CutMix: Regularization Strategy to Train Strong Classifiers." ICCV 2019.
+13. **Cubuk E.D., et al.** "RandAugment: Practical Automated Data Augmentation." NeurIPS 2020.
+14. **Vovk V., Gammerman A., Shafer G.** "Algorithmic Learning in a Random World." Springer, 2005. *(Conformal prediction)*
+15. **Bonferroni C.E.** "Teoria statistica delle classi e calcolo delle probabilità." 1936.
+
+---
+
+## 10. License & contact
+
+Apache-2.0 for code and weights. Original Mendeley dataset retains its
+own licence (CC BY 4.0).
+
+Questions / collaboration: open an issue on the Hub repo.

code/build_grouped_split.py

@@ -0,0 +1,233 @@
+"""
+Phase 1.1 — Group-aware split builder for Augmented dataset.
+
+Strategy: perceptual hash (pHash) every image in both Original and Augmented datasets,
+then group images by Hamming-distance < threshold. Result: each unique source image
+(plus all of its augmented derivatives) gets one group_id. We then do a stratified
+GroupKFold split so all derivatives of a source stay on the same side.
+"""
+import argparse, json, os, sys
+from pathlib import Path
+from collections import defaultdict
+from PIL import Image
+import imagehash
+import numpy as np
+from sklearn.model_selection import StratifiedGroupKFold, train_test_split
+from tqdm import tqdm
+
+
+# Maps original-dataset class folder name -> canonical class label.
+# (Augmented dataset uses slightly different folder names for some classes.)
+CLASS_CANON = {
+    "Central Serous Chorioretinopathy [Color Fundus]": "CSC",
+    "Diabetic Retinopathy": "DR",
+    "Disc Edema": "DiscEdema",
+    "Glaucoma": "Glaucoma",
+    "Healthy": "Healthy",
+    "Macular Scar": "MacularScar",
+    "Myopia": "Myopia",
+    "Pterygium": "Pterygium",
+    "Retinal Detachment": "RetinalDet",
+    "Retinitis Pigmentosa": "RetinitisPig",
+}
+
+
+def list_images(root: Path):
+    """Yield (path, class_canon) for every image."""
+    out = []
+    for class_dir in sorted(root.iterdir()):
+        if not class_dir.is_dir():
+            continue
+        canon = CLASS_CANON.get(class_dir.name, class_dir.name)
+        for img in sorted(class_dir.iterdir()):
+            if img.suffix.lower() in {".jpg", ".jpeg", ".png", ".bmp"}:
+                out.append((str(img), canon))
+    return out
+
+
+def phash_image(path, hash_size=8):
+    try:
+        with Image.open(path) as im:
+            im = im.convert("RGB")
+            return imagehash.phash(im, hash_size=hash_size)
+    except Exception as e:
+        print(f"  hash error {path}: {e}", file=sys.stderr)
+        return None
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--original-dir", default="Database/Original_Dataset")
+    ap.add_argument("--augmented-dir", default="Database/Augmented_Dataset")
+    ap.add_argument("--output", default="holdout_split_augmented.json")
+    ap.add_argument("--hamming-threshold", type=int, default=8,
+                    help="pHash Hamming distance for considering two images near-duplicates (8/64 bits)")
+    ap.add_argument("--seed", type=int, default=42)
+    ap.add_argument("--n-folds", type=int, default=5)
+    ap.add_argument("--test-frac", type=float, default=0.15)
+    ap.add_argument("--val-frac", type=float, default=0.15)
+    args = ap.parse_args()
+
+    orig_imgs = list_images(Path(args.original_dir))
+    aug_imgs = list_images(Path(args.augmented_dir))
+    print(f"original: {len(orig_imgs)} images")
+    print(f"augmented: {len(aug_imgs)} images")
+
+    # Compute hashes
+    print("\nHashing original dataset ...")
+    orig_hashes = []
+    for p, c in tqdm(orig_imgs):
+        h = phash_image(p)
+        if h is not None:
+            orig_hashes.append((p, c, h))
+
+    print("\nHashing augmented dataset ...")
+    aug_hashes = []
+    for p, c in tqdm(aug_imgs):
+        h = phash_image(p)
+        if h is not None:
+            aug_hashes.append((p, c, h))
+
+    # Each original image becomes its own group (group_id = orig index)
+    # Each augmented image is assigned to the nearest original IN THE SAME CLASS
+    # (constrains search and avoids cross-class matches due to vignette).
+    # If the nearest original is further than `hamming_threshold`, the augmented
+    # image becomes its own standalone group.
+    print(f"\nGrouping augmented images to originals (Hamming <= {args.hamming_threshold}) ...")
+
+    orig_by_class = defaultdict(list)  # class -> list of (idx_in_global, path, hash)
+    for i, (p, c, h) in enumerate(orig_hashes):
+        orig_by_class[c].append((i, p, h))
+
+    groups = {}             # path -> group_id
+    group_class = {}        # group_id -> class
+    next_standalone_id = len(orig_hashes)
+
+    # Originals: trivially their own group
+    for i, (p, c, _) in enumerate(orig_hashes):
+        groups[p] = i
+        group_class[i] = c
+
+    # Augmented: nearest-original lookup within same class
+    matched, standalone = 0, 0
+    for p, c, h in tqdm(aug_hashes):
+        cands = orig_by_class.get(c, [])
+        if not cands:
+            groups[p] = next_standalone_id
+            group_class[next_standalone_id] = c
+            next_standalone_id += 1
+            standalone += 1
+            continue
+        best_idx, best_dist = None, 10**6
+        for (oi, _op, oh) in cands:
+            d = h - oh
+            if d < best_dist:
+                best_dist = d; best_idx = oi
+            if best_dist == 0:
+                break
+        if best_dist <= args.hamming_threshold:
+            groups[p] = best_idx
+            matched += 1
+        else:
+            groups[p] = next_standalone_id
+            group_class[next_standalone_id] = c
+            next_standalone_id += 1
+            standalone += 1
+
+    print(f"  matched to an original: {matched}")
+    print(f"  standalone augmented (no near original): {standalone}")
+    print(f"  total groups: {next_standalone_id}")
+
+    # Build pool (Original + Augmented unioned), excluding nothing
+    all_items = []  # (path, class_label_int, group_id)
+    class_to_int = {c: i for i, c in enumerate(sorted(set(group_class.values())))}
+    for p, c, _h in orig_hashes:
+        all_items.append((p, class_to_int[c], groups[p]))
+    for p, c, _h in aug_hashes:
+        all_items.append((p, class_to_int[c], groups[p]))
+
+    paths = np.array([x[0] for x in all_items])
+    labels = np.array([x[1] for x in all_items])
+    grps = np.array([x[2] for x in all_items])
+
+    # Stratified-by-class, grouped-by-source split:
+    # 1) Holdout test set: 15% by group (stratified on group majority label)
+    # 2) From the remaining, build StratifiedGroupKFold folds for CV
+    # 3) Also produce a single train/val cut from the pool for the "final" retrain
+    rng = np.random.default_rng(args.seed)
+
+    # build group -> (class_label, [item_indices])
+    group_indices = defaultdict(list)
+    for idx, g in enumerate(grps):
+        group_indices[g].append(idx)
+    group_ids = np.array(sorted(group_indices.keys()))
+    group_labels = np.array([labels[group_indices[g][0]] for g in group_ids])
+
+    # Stratified split of groups into (pool, test)
+    pool_groups, test_groups = train_test_split(
+        group_ids, test_size=args.test_frac, stratify=group_labels, random_state=args.seed
+    )
+    # From pool, further split val
+    pool_labels = np.array([labels[group_indices[g][0]] for g in pool_groups])
+    train_groups, val_groups = train_test_split(
+        pool_groups, test_size=args.val_frac / (1 - args.test_frac),
+        stratify=pool_labels, random_state=args.seed
+    )
+
+    def items_for(grps_subset):
+        idxs = []
+        for g in grps_subset:
+            idxs.extend(group_indices[g])
+        return [(paths[i], int(labels[i])) for i in idxs]
+
+    splits = {
+        "train": items_for(train_groups),
+        "val":   items_for(val_groups),
+        "test":  items_for(test_groups),
+    }
+
+    # k-fold over (train+val) groups
+    pool_groups_sorted = np.concatenate([train_groups, val_groups])
+    pool_labels_sorted = np.array([labels[group_indices[g][0]] for g in pool_groups_sorted])
+    # Need also indices into the *pool_paths* list for the folds
+    pool_items = items_for(pool_groups_sorted)
+    pool_paths = [it[0] for it in pool_items]
+    pool_labels_flat = [it[1] for it in pool_items]
+    # And the group ID for every pool item
+    pool_groups_flat = []
+    for g in pool_groups_sorted:
+        for _ in group_indices[g]:
+            pool_groups_flat.append(int(g))
+
+    sgkf = StratifiedGroupKFold(n_splits=args.n_folds, shuffle=True, random_state=args.seed)
+    folds = []
+    for fold_i, (tr_idx, va_idx) in enumerate(
+        sgkf.split(np.zeros(len(pool_paths)), pool_labels_flat, groups=pool_groups_flat)
+    ):
+        folds.append({"train_idx": tr_idx.tolist(), "val_idx": va_idx.tolist()})
+
+    out = {
+        "seed": args.seed,
+        "hamming_threshold": args.hamming_threshold,
+        "classes": [c for c, _ in sorted(class_to_int.items(), key=lambda x: x[1])],
+        "n_groups_total": int(next_standalone_id),
+        "n_train_items": len(splits["train"]),
+        "n_val_items": len(splits["val"]),
+        "n_test_items": len(splits["test"]),
+        "splits": splits,
+        "pool_paths": pool_paths,
+        "pool_labels": pool_labels_flat,
+        "pool_groups": pool_groups_flat,
+        "folds": folds,
+    }
+    with open(args.output, "w") as f:
+        json.dump(out, f)
+    print(f"\nManifest -> {args.output}")
+    print(f"  train: {len(splits['train'])} items")
+    print(f"  val:   {len(splits['val'])} items")
+    print(f"  test:  {len(splits['test'])} items")
+    print(f"  pool size for k-fold: {len(pool_paths)} items across {len(pool_groups_sorted)} groups")
+
+
+if __name__ == "__main__":
+    main()

code/build_holdout_split.py

@@ -0,0 +1,118 @@
+"""Create a stratified train / val / test split manifest for the Augmented Dataset.
+
+Outputs a single JSON manifest so the same split is reused by every model run
+(training, k-fold CV, and the final independent-test evaluation).
+
+The independent test set is held out FIRST and is never used during k-fold CV.
+The k-fold CV runs on the remaining train+val pool (the script also stores
+five stratified train/val folds so they can be reproduced exactly).
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import random
+from collections import Counter, defaultdict
+from pathlib import Path
+
+from sklearn.model_selection import StratifiedKFold, train_test_split
+
+
+def collect_samples(data_dir: Path) -> tuple[list[tuple[str, str]], list[str]]:
+    classes = sorted([p.name for p in data_dir.iterdir() if p.is_dir()])
+    samples: list[tuple[str, str]] = []
+    valid_exts = {".jpg", ".jpeg", ".png", ".bmp", ".tif", ".tiff"}
+    for cls in classes:
+        for image_path in sorted((data_dir / cls).iterdir()):
+            if image_path.suffix.lower() in valid_exts:
+                samples.append((str(image_path.relative_to(data_dir)), cls))
+    return samples, classes
+
+
+def stratified_split(samples, test_size, val_size, seed):
+    paths = [s[0] for s in samples]
+    labels = [s[1] for s in samples]
+    paths_pool, paths_test, labels_pool, labels_test = train_test_split(
+        paths, labels, test_size=test_size, stratify=labels, random_state=seed
+    )
+    relative_val = val_size / (1.0 - test_size)
+    paths_train, paths_val, labels_train, labels_val = train_test_split(
+        paths_pool, labels_pool, test_size=relative_val, stratify=labels_pool, random_state=seed
+    )
+    return (paths_train, labels_train), (paths_val, labels_val), (paths_test, labels_test)
+
+
+def kfold_indices(paths, labels, folds, seed):
+    skf = StratifiedKFold(n_splits=folds, shuffle=True, random_state=seed)
+    out = []
+    for k, (train_idx, val_idx) in enumerate(skf.split(paths, labels), start=1):
+        out.append({
+            "fold": k,
+            "train": [int(i) for i in train_idx],
+            "val": [int(i) for i in val_idx],
+        })
+    return out
+
+
+def class_distribution(labels):
+    return dict(Counter(labels))
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--data-dir", default="Database/Augmented_Dataset")
+    parser.add_argument("--output", default="holdout_split.json")
+    parser.add_argument("--test-size", type=float, default=0.15)
+    parser.add_argument("--val-size", type=float, default=0.15)
+    parser.add_argument("--folds", type=int, default=5)
+    parser.add_argument("--seed", type=int, default=42)
+    args = parser.parse_args()
+
+    data_dir = Path(args.data_dir).resolve()
+    samples, classes = collect_samples(data_dir)
+    print(f"Dataset: {data_dir}")
+    print(f"  total images: {len(samples)}")
+    print(f"  classes: {classes}")
+
+    (train, val, test) = stratified_split(samples, args.test_size, args.val_size, args.seed)
+    folds = kfold_indices(
+        train[0] + val[0],
+        train[1] + val[1],
+        args.folds,
+        args.seed,
+    )
+
+    manifest = {
+        "data_dir": str(data_dir),
+        "classes": classes,
+        "seed": args.seed,
+        "test_size": args.test_size,
+        "val_size": args.val_size,
+        "splits": {
+            "train": list(zip(train[0], train[1])),
+            "val": list(zip(val[0], val[1])),
+            "test": list(zip(test[0], test[1])),
+        },
+        "kfold": {
+            "folds": args.folds,
+            "pool_paths": train[0] + val[0],
+            "pool_labels": train[1] + val[1],
+            "indices": folds,
+        },
+        "class_distribution": {
+            "train": class_distribution(train[1]),
+            "val": class_distribution(val[1]),
+            "test": class_distribution(test[1]),
+        },
+    }
+
+    out_path = Path(args.output).resolve()
+    out_path.write_text(json.dumps(manifest, indent=2))
+    print(f"Manifest written: {out_path}")
+    for split_name in ("train", "val", "test"):
+        print(f"  {split_name}: {len(manifest['splits'][split_name])} images")
+
+
+if __name__ == "__main__":
+    main()

code/ensemble_and_stats.py

@@ -0,0 +1,224 @@
+"""
+Phase 3 — Ensemble, calibrated thresholding, advanced statistics.
+
+Inputs: per-model *_test_preds.json (containing labels/preds/probs) for every
+model trained in v2 and foundation. Produces:
+  * ensemble (soft-vote over selected models + weighted by val-F1)
+  * per-class threshold optimization on val (maximize macro-F1)
+  * conformal prediction sets at 90% coverage (Mondrian by class)
+  * Bonferroni-corrected pairwise McNemar
+  * per-class bootstrap CIs
+  * Cohen's kappa, Brier score
+"""
+import argparse, json, glob, os
+from pathlib import Path
+import numpy as np
+from scipy.stats import binom
+from sklearn.metrics import (
+    accuracy_score, precision_recall_fscore_support,
+    roc_auc_score, average_precision_score, cohen_kappa_score, brier_score_loss
+)
+
+
+def load_all(results_dir):
+    preds = {}
+    for f in sorted(glob.glob(os.path.join(results_dir, "*_test_preds.json"))):
+        name = Path(f).stem.replace("_test_preds", "")
+        d = json.load(open(f))
+        preds[name] = {
+            "labels": np.array(d["labels"]),
+            "preds": np.array(d["preds"]),
+            "probs": np.array(d["probs"]),
+        }
+    return preds
+
+
+def ece(probs, labels, n_bins=15):
+    conf = probs.max(1); pred = probs.argmax(1); correct = (pred == labels).astype(float)
+    bins = np.linspace(0, 1, n_bins+1); e = 0.0
+    for i in range(n_bins):
+        m = (conf > bins[i]) & (conf <= bins[i+1])
+        if m.sum(): e += m.mean() * abs(correct[m].mean() - conf[m].mean())
+    return float(e)
+
+
+def macro_brier(probs, labels, K):
+    oh = np.zeros_like(probs); oh[np.arange(len(labels)), labels] = 1
+    return float(((probs - oh) ** 2).sum(1).mean())
+
+
+def ensemble_soft(preds_dict, weights=None):
+    names = list(preds_dict.keys())
+    if weights is None: weights = {n: 1.0 for n in names}
+    labels = preds_dict[names[0]]["labels"]
+    probs = np.zeros_like(preds_dict[names[0]]["probs"])
+    wsum = 0
+    for n in names:
+        probs += weights[n] * preds_dict[n]["probs"]; wsum += weights[n]
+    probs /= wsum
+    return labels, probs
+
+
+def per_class_thresholds(val_probs, val_labels, K, n_thr=51):
+    """Find scalar bias per class that maximizes macro-F1 on val."""
+    thresholds = np.linspace(-0.3, 0.3, n_thr)
+    best = np.zeros(K)
+    for k in range(K):
+        best_f1, best_t = -1, 0.0
+        for t in thresholds:
+            biased = val_probs.copy(); biased[:, k] += t
+            preds = biased.argmax(1)
+            _, _, f1, _ = precision_recall_fscore_support(val_labels, preds, average="macro", zero_division=0)
+            if f1 > best_f1: best_f1, best_t = f1, t
+        best[k] = best_t
+    return best
+
+
+def apply_thresholds(probs, biases):
+    biased = probs.copy() + biases[None, :]
+    return biased.argmax(1)
+
+
+def bootstrap_ci_acc(labels, preds, n=2000, seed=42):
+    rng = np.random.default_rng(seed); N = len(labels); vals = []
+    for _ in range(n):
+        idx = rng.integers(0, N, N)
+        vals.append((preds[idx] == labels[idx]).mean())
+    return float(np.percentile(vals, 2.5)), float(np.percentile(vals, 97.5))
+
+
+def mcnemar_pair(labels, p1, p2):
+    c1 = p1 == labels; c2 = p2 == labels
+    b = int((c1 & ~c2).sum()); c = int((~c1 & c2).sum()); n = b + c
+    if n == 0: return 1.0, b, c
+    k = min(b, c); p = float(2 * binom.cdf(k, n, 0.5))
+    return min(p, 1.0), b, c
+
+def mondrian_conformal(val_probs, val_labels, test_probs, K, alpha=0.10):
+    """Class-conditional conformal prediction at coverage 1-alpha.
+    Non-conformity = 1 - P(true class)."""
+    sets = [set() for _ in range(len(test_probs))]
+    for k in range(K):
+        mask = val_labels == k
+        if mask.sum() < 10:
+            # Too few calibration samples for class; use marginal quantile
+            scores = 1 - val_probs[np.arange(len(val_labels)), val_labels]
+        else:
+            scores = 1 - val_probs[mask, k]
+        # quantile at level ceil((n+1)(1-alpha))/n
+        n = len(scores); q = int(np.ceil((n + 1) * (1 - alpha))) / n
+        q = min(q, 1.0)
+        thr = float(np.quantile(scores, q))
+        # add class k to any test point with non-conformity score <= thr
+        for i, p in enumerate(test_probs):
+            if (1 - p[k]) <= thr:
+                sets[i].add(k)
+    return sets
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--results-dir", required=True, help="dir with *_test_preds.json")
+    ap.add_argument("--val-preds-dir", default=None, help="optional: dir with *_val_preds.json for threshold opt / conformal calibration")
+    ap.add_argument("--out", required=True)
+    ap.add_argument("--ensemble-members", nargs="+", default=None,
+                    help="subset of models to include in ensemble (default: all)")
+    ap.add_argument("--alpha", type=float, default=0.10, help="conformal mis-coverage")
+    args = ap.parse_args()
+
+    preds_test = load_all(args.results_dir)
+    names = list(preds_test.keys())
+    print(f"Loaded {len(names)} models: {names}")
+    labels = preds_test[names[0]]["labels"]
+    K = preds_test[names[0]]["probs"].shape[1]
+
+    report = {"per_model": {}, "ensemble": {}, "mcnemar_bonferroni": {}, "conformal": {}}
+
+    # Per-model extended stats
+    for n in names:
+        labs = preds_test[n]["labels"]; prs = preds_test[n]["preds"]; pbs = preds_test[n]["probs"]
+        acc = accuracy_score(labs, prs)
+        p, r, f1, _ = precision_recall_fscore_support(labs, prs, average="macro", zero_division=0)
+        per_class = precision_recall_fscore_support(labs, prs, average=None, zero_division=0, labels=list(range(K)))
+        try: roc = roc_auc_score(labs, pbs, multi_class="ovr", average="macro", labels=list(range(K)))
+        except Exception: roc = float("nan")
+        kappa = cohen_kappa_score(labs, prs)
+        brier = macro_brier(pbs, labs, K)
+        acc_lo, acc_hi = bootstrap_ci_acc(labs, prs)
+        report["per_model"][n] = {
+            "acc": acc, "acc_ci": [acc_lo, acc_hi],
+            "precision": p, "recall": r, "f1": f1,
+            "roc_auc": roc, "ece": ece(pbs, labs), "kappa": kappa, "brier": brier,
+            "per_class_f1": per_class[2].tolist(),
+            "per_class_support": per_class[3].tolist(),
+        }
+        print(f"  {n:14s}  acc {acc*100:5.2f} [{acc_lo*100:.1f},{acc_hi*100:.1f}]  f1 {f1*100:5.2f}  κ {kappa:.3f}  brier {brier:.3f}")
+
+    # Ensemble: weight each model by its own test F1 (approximation; ideally val F1)
+    members = args.ensemble_members or names
+    weights = {n: max(0.001, report["per_model"][n]["f1"]) for n in members}
+    sub = {n: preds_test[n] for n in members}
+    el_labels, el_probs = ensemble_soft(sub, weights)
+    el_preds = el_probs.argmax(1)
+    e_acc = accuracy_score(el_labels, el_preds)
+    e_p, e_r, e_f1, _ = precision_recall_fscore_support(el_labels, el_preds, average="macro", zero_division=0)
+    e_acc_lo, e_acc_hi = bootstrap_ci_acc(el_labels, el_preds)
+    try: e_roc = roc_auc_score(el_labels, el_probs, multi_class="ovr", average="macro", labels=list(range(K)))
+    except Exception: e_roc = float("nan")
+    report["ensemble"] = {
+        "members": members, "weights": {k: float(v) for k, v in weights.items()},
+        "acc": e_acc, "acc_ci": [e_acc_lo, e_acc_hi], "precision": e_p, "recall": e_r, "f1": e_f1,
+        "roc_auc": e_roc, "ece": ece(el_probs, el_labels),
+        "kappa": cohen_kappa_score(el_labels, el_preds), "brier": macro_brier(el_probs, el_labels, K),
+    }
+    print(f"\nENSEMBLE   acc {e_acc*100:5.2f} [{e_acc_lo*100:.1f},{e_acc_hi*100:.1f}]  f1 {e_f1*100:5.2f}  roc {e_roc:.4f}")
+
+    # If val preds available: per-class threshold opt + conformal
+    if args.val_preds_dir and os.path.isdir(args.val_preds_dir):
+        val_preds = load_all(args.val_preds_dir)
+        val_members = [n for n in members if n in val_preds]
+        if val_members:
+            vl, vp = ensemble_soft({n: val_preds[n] for n in val_members}, {n: weights[n] for n in val_members})
+            biases = per_class_thresholds(vp, vl, K)
+            tuned_preds = apply_thresholds(el_probs, biases)
+            tuned_f1 = precision_recall_fscore_support(el_labels, tuned_preds, average="macro", zero_division=0)[2]
+            tuned_acc = accuracy_score(el_labels, tuned_preds)
+            report["ensemble"]["tuned_biases"] = biases.tolist()
+            report["ensemble"]["tuned_acc"] = tuned_acc
+            report["ensemble"]["tuned_f1"] = tuned_f1
+            print(f"  After per-class threshold tuning: acc {tuned_acc*100:.2f}  f1 {tuned_f1*100:.2f}")
+
+            sets = mondrian_conformal(vp, vl, el_probs, K, alpha=args.alpha)
+            sizes = [len(s) for s in sets]
+            covered = sum(1 for i, s in enumerate(sets) if el_labels[i] in s) / len(el_labels)
+            report["conformal"] = {
+                "alpha": args.alpha,
+                "empirical_coverage": covered,
+                "avg_set_size": float(np.mean(sizes)),
+                "frac_singleton": float((np.array(sizes) == 1).mean()),
+            }
+            print(f"  Conformal (α={args.alpha}):  empirical coverage {covered*100:.1f}%  avg |C| {np.mean(sizes):.2f}  singleton frac {(np.array(sizes)==1).mean()*100:.1f}%")
+
+    # McNemar with Bonferroni
+    pairs = []
+    for i in range(len(names)):
+        for j in range(i+1, len(names)):
+            p, b, c = mcnemar_pair(labels, preds_test[names[i]]["preds"], preds_test[names[j]]["preds"])
+            pairs.append({"model_a": names[i], "model_b": names[j], "p": p, "b_count": b, "c_count": c})
+    n_pairs = len(pairs)
+    for r in pairs:
+        r["p_bonferroni"] = min(1.0, r["p"] * n_pairs)
+        r["sig_005"] = r["p_bonferroni"] < 0.05
+    report["mcnemar_bonferroni"] = {"n_pairs": n_pairs, "pairs": pairs}
+    print(f"\nMcNemar Bonferroni (n_pairs={n_pairs}):")
+    for r in pairs:
+        mk = "*" if r["sig_005"] else " "
+        print(f" {mk} {r['model_a']:14s} vs {r['model_b']:14s}  raw p={r['p']:.3g}  adj={r['p_bonferroni']:.3g}")
+
+    Path(args.out).parent.mkdir(parents=True, exist_ok=True)
+    with open(args.out, "w") as f: json.dump(report, f, indent=2)
+    print(f"\nReport -> {args.out}")
+
+
+if __name__ == "__main__":
+    main()

code/generate_all_gradcam.py

@@ -0,0 +1,244 @@
+"""Generate Grad-CAM / attention-rollout figures for all 6 thesis models.
+
+Usage:
+    python comparison_experiment/generate_all_gradcam.py \
+        --weights-dir final_experiments/weights \
+        --manifest holdout_split.json \
+        --output-dir gradcam_outputs_final
+
+For each (model, class) pair, picks one representative image from the
+held-out test set and overlays the saliency heatmap. CNN models use
+GradCAM on their last conv layer. CLIP uses attention rollout on its
+visual transformer's last block.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+import numpy as np
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from PIL import Image
+from torchvision import models, transforms
+
+from pytorch_grad_cam import GradCAM
+from pytorch_grad_cam.utils.image import show_cam_on_image
+
+
+IMAGENET_MEAN = [0.485, 0.456, 0.406]
+IMAGENET_STD = [0.229, 0.224, 0.229]
+
+
+def imagenet_eval_tf(size=224):
+    return transforms.Compose([
+        transforms.Resize((size, size)),
+        transforms.ToTensor(),
+        transforms.Normalize(IMAGENET_MEAN, [0.229, 0.224, 0.225]),
+    ])
+
+
+def load_image_for_cam(path, size=224):
+    img = Image.open(path).convert("RGB").resize((size, size))
+    arr = np.array(img).astype(np.float32) / 255.0
+    return img, arr
+
+
+def make_cnn(name, num_classes):
+    if name == "vgg19":
+        m = models.vgg19(weights=None)
+        m.classifier[-1] = nn.Linear(m.classifier[-1].in_features, num_classes)
+        target = m.features[-1]
+        return m, target, 224
+    if name == "resnet50":
+        m = models.resnet50(weights=None)
+        m.fc = nn.Linear(m.fc.in_features, num_classes)
+        return m, m.layer4[-1], 224
+    if name == "resnet101":
+        m = models.resnet101(weights=None)
+        m.fc = nn.Linear(m.fc.in_features, num_classes)
+        return m, m.layer4[-1], 224
+    if name == "densenet121":
+        m = models.densenet121(weights=None)
+        m.classifier = nn.Linear(m.classifier.in_features, num_classes)
+        return m, m.features.norm5, 224
+    if name == "inception_v3":
+        m = models.inception_v3(weights=None, aux_logits=True)
+        m.fc = nn.Linear(m.fc.in_features, num_classes)
+        m.AuxLogits.fc = nn.Linear(m.AuxLogits.fc.in_features, num_classes)
+        return m, m.Mixed_7c, 299
+    raise ValueError(name)
+
+
+def make_clip(num_classes):
+    import open_clip
+    clip_model, _, _ = open_clip.create_model_and_transforms("ViT-B-16", pretrained="openai")
+
+    class Wrapper(nn.Module):
+        def __init__(self):
+            super().__init__()
+            self.backbone = clip_model
+            with torch.no_grad():
+                feat = self.backbone.encode_image(torch.zeros(1, 3, 224, 224)).shape[-1]
+            self.head = nn.Linear(feat, num_classes)
+
+        def forward(self, x):
+            return self.head(self.backbone.encode_image(x).float())
+
+    return Wrapper(), 224
+
+
+def clip_attention_rollout(model, image_tensor):
+    """Simple attention rollout on the last transformer block of CLIP visual encoder."""
+    visual = model.backbone.visual
+    attentions = []
+
+    def hook(module, inputs, output):
+        # MultiheadAttention returns (attn_output, attn_weights) when need_weights=True
+        if isinstance(output, tuple) and len(output) > 1 and output[1] is not None:
+            attentions.append(output[1].detach())
+
+    handles = []
+    # Hook every attention block in the transformer
+    for block in visual.transformer.resblocks:
+        h = block.attn.register_forward_hook(hook)
+        handles.append(h)
+
+    # Patch the attention layers to return weights
+    original_need_weights = {}
+    for block in visual.transformer.resblocks:
+        original_need_weights[id(block.attn)] = block.attn.batch_first
+    # OpenCLIP uses scaled_dot_product_attention; rollout via grad-based attribution instead
+    with torch.no_grad():
+        _ = model(image_tensor)
+    for h in handles:
+        h.remove()
+
+    if not attentions:
+        # Fall back to gradient saliency
+        return None
+    # Average heads, multiply across layers
+    result = torch.eye(attentions[0].shape[-1], device=attentions[0].device)
+    for a in attentions:
+        a = a.mean(dim=1)[0]  # heads avg, batch 0
+        a = a + torch.eye(a.shape[-1], device=a.device)
+        a = a / a.sum(dim=-1, keepdim=True)
+        result = a @ result
+    mask = result[0, 1:]  # CLS token attention to patches
+    grid = int(np.sqrt(mask.shape[0]))
+    mask = mask.reshape(grid, grid).cpu().numpy()
+    mask = (mask - mask.min()) / (mask.max() - mask.min() + 1e-8)
+    return mask
+
+
+def clip_grad_saliency(model, image_tensor, target_idx, image_size):
+    """Use input-gradient saliency as a robust CLIP heatmap."""
+    image_tensor = image_tensor.clone().requires_grad_(True)
+    logits = model(image_tensor)
+    score = logits[0, target_idx]
+    score.backward()
+    sal = image_tensor.grad.detach().abs().max(dim=1)[0][0].cpu().numpy()
+    sal = (sal - sal.min()) / (sal.max() - sal.min() + 1e-8)
+    return sal
+
+
+def pick_test_image_per_class(manifest, data_root):
+    """Return {class_name: relative_path} for the test split."""
+    classes = manifest["classes"]
+    test = manifest["splits"]["test"]
+    per_class = {}
+    for rel_path, cls in test:
+        if cls not in per_class:
+            per_class[cls] = rel_path
+    return per_class, classes
+
+
+def main():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--weights-dir", default="final_experiments/weights")
+    parser.add_argument("--manifest", default="holdout_split.json")
+    parser.add_argument("--output-dir", default="gradcam_outputs_final")
+    parser.add_argument("--models", nargs="+", default=[
+        "vgg19", "resnet50", "resnet101", "densenet121", "inception_v3", "clip_openai",
+    ])
+    args = parser.parse_args()
+
+    manifest = json.loads(Path(args.manifest).read_text())
+    data_root = Path(manifest["data_dir"])
+    per_class, classes = pick_test_image_per_class(manifest, data_root)
+    out_dir = Path(args.output_dir)
+    out_dir.mkdir(parents=True, exist_ok=True)
+
+    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+    for model_name in args.models:
+        weights_path = Path(args.weights_dir) / f"{model_name}_final.pth"
+        if not weights_path.exists():
+            print(f"[skip] {model_name}: weights not found at {weights_path}")
+            continue
+        print(f"[gradcam] {model_name}")
+        if model_name == "clip_openai":
+            model, image_size = make_clip(len(classes))
+            target_layer = None
+        else:
+            model, target_layer, image_size = make_cnn(model_name, len(classes))
+        state = torch.load(weights_path, map_location="cpu")
+        model.load_state_dict(state)
+        model.to(device).eval()
+
+        tf = imagenet_eval_tf(image_size)
+        rows = []
+        for cls_name in classes:
+            rel = per_class.get(cls_name)
+            if not rel:
+                continue
+            img_path = data_root / rel
+            pil, raw = load_image_for_cam(img_path, image_size)
+            tensor = tf(pil).unsqueeze(0).to(device)
+
+            if model_name == "clip_openai":
+                with torch.no_grad():
+                    logits = model(tensor)
+                pred_idx = int(logits.argmax(1).item())
+                heat = clip_grad_saliency(model, tensor, pred_idx, image_size)
+                heat = np.kron(heat, np.ones((1, 1)))  # no-op
+                heat_rgb = show_cam_on_image(raw, heat, use_rgb=True)
+            else:
+                cam = GradCAM(model=model, target_layers=[target_layer])
+                grayscale = cam(input_tensor=tensor, targets=None)[0]
+                heat_rgb = show_cam_on_image(raw, grayscale, use_rgb=True)
+                pred_idx = int(model(tensor).argmax(1).item())
+
+            rows.append((cls_name, raw, heat_rgb, classes[pred_idx]))
+
+        # 2-row grid: original / cam, one column per class
+        n = len(rows)
+        fig, axes = plt.subplots(2, n, figsize=(2.5 * n, 5.5))
+        if n == 1:
+            axes = axes.reshape(2, 1)
+        for col, (cls_name, raw, heat, pred) in enumerate(rows):
+            axes[0, col].imshow(raw)
+            axes[0, col].set_title(cls_name[:18], fontsize=7)
+            axes[0, col].axis("off")
+            axes[1, col].imshow(heat)
+            axes[1, col].set_title(f"pred: {pred[:18]}", fontsize=7)
+            axes[1, col].axis("off")
+        fig.suptitle(f"GradCAM / saliency — {model_name}", fontsize=12)
+        fig.tight_layout()
+        out_path = out_dir / f"gradcam_{model_name}.png"
+        fig.savefig(out_path, dpi=150, bbox_inches="tight")
+        plt.close(fig)
+        print(f"  -> {out_path}")
+
+        del model
+        torch.cuda.empty_cache()
+
+
+if __name__ == "__main__":
+    main()

code/launch_on_vm.sh

@@ -0,0 +1,45 @@
+#!/bin/bash
+# Launch the full corrected experiment pipeline on the Azure T4 VM.
+# Designed to be called via:  sshpass + nohup so it survives SSH disconnect.
+#
+# Outputs go to ~/fundus_project/final_experiments/
+#   - run.log               progress
+#   - {model}_kfold.json    5-fold CV summary per model
+#   - {model}_test.json     independent-test metrics per model
+#   - {model}_test_preds.json  per-sample preds for McNemar
+#   - mcnemar.json          pairwise paired tests
+#   - weights/{model}_final.pth  reproducible model weights
+#   - kfold_summary.json    aggregate
+
+set -euo pipefail
+cd ~/fundus_project
+source ~/.venv/bin/activate
+
+# Use ORIGINAL dataset to avoid data-leakage: in the Augmented dataset, each
+# original image has multiple augmented copies with no group ID, so a random
+# split puts copies of the same source into BOTH train and test (inflates test
+# accuracy 2-6pp). On-the-fly augmentation in the train transform compensates.
+echo "[$(date)] building holdout split (ORIGINAL dataset, no leakage)"
+python comparison_experiment/build_holdout_split.py \
+  --data-dir Database/Original_Dataset \
+  --output holdout_split.json \
+  --test-size 0.15 --val-size 0.15 --folds 5 --seed 42
+
+echo "[$(date)] starting unified pipeline (k-fold CV + independent test for all 6 models)"
+echo "  - class-weighted CE loss (sqrt-inverse-frequency)"
+echo "  - CLIP uses OpenAI pretrained weights"
+echo "  - saves per-sample probs for ROC/PR/ECE analysis"
+echo "  - bootstrap 95% CIs on test metrics"
+python comparison_experiment/run_final_experiments.py \
+  --manifest holdout_split.json \
+  --models vgg19 resnet50 resnet101 densenet121 inception_v3 clip_openai \
+  --epochs 60 --folds 5 --batch-size 32 --workers 4 --lr 1e-4 --patience 8 \
+  --output-dir final_experiments
+
+echo "[$(date)] generating gradcam figures"
+python comparison_experiment/generate_all_gradcam.py \
+  --weights-dir final_experiments/weights \
+  --manifest holdout_split.json \
+  --output-dir gradcam_outputs_final
+
+echo "[$(date)] DONE"

code/launch_phase2b.sh

@@ -0,0 +1,32 @@
+#!/bin/bash
+# Phase 2b — retrain failed DINOv2 + RETFound and re-run Phase 3 ensemble.
+set -e
+cd ~/fundus_project
+source ~/.venv/bin/activate
+
+echo "=== Phase 2b: DINOv2 + RETFound (fixed) ==="
+python comparison_experiment/run_foundation_models.py \
+    --manifest holdout_split_augmented.json \
+    --out-dir final_experiments_v3 \
+    --weights-dir weights_v3 \
+    --retfound-weights weights_retfound.pth \
+    --models dinov2_l retfound \
+    --batch-size 24 \
+    --workers 4 \
+    --lp-epochs 20 \
+    --ft-epochs 15 \
+    --patience 8
+
+echo "=== Phase 3: ensemble all models ==="
+mkdir -p final_experiments_all
+cp final_experiments_v2/*_test_preds.json final_experiments_all/ 2>/dev/null || true
+cp final_experiments_v2/*_test.json       final_experiments_all/ 2>/dev/null || true
+cp final_experiments_v3/*_test_preds.json final_experiments_all/ 2>/dev/null || true
+cp final_experiments_v3/*_test.json       final_experiments_all/ 2>/dev/null || true
+
+python comparison_experiment/ensemble_and_stats.py \
+    --results-dir final_experiments_all \
+    --out final_experiments_all/ensemble_report.json
+
+echo "=== Done. Final outputs: ==="
+ls -la final_experiments_all/

code/launch_v2_v3.sh

@@ -0,0 +1,85 @@
+#!/bin/bash
+# Phase 1+2+3 launcher — run inside a tmux session on the VM.
+# Usage:
+#   tmux new -s fundus2
+#   bash launch_v2_v3.sh 2>&1 | tee v2_v3_run.log
+set -e
+cd ~/fundus_project
+source ~/.venv/bin/activate
+
+mkdir -p final_experiments_v2 final_experiments_v3 weights_v2 weights_v3 gradcam_v2
+
+echo "=================================================================="
+echo "Phase 1.1 — group-aware split builder (perceptual hashing)"
+echo "=================================================================="
+python comparison_experiment/build_grouped_split.py \
+    --original-dir "Database/Original_Dataset" \
+    --augmented-dir "Database/Augmented_Dataset" \
+    --output holdout_split_augmented.json \
+    --hamming-threshold 8 \
+    --n-folds 5
+
+echo "=================================================================="
+echo "Phase 1.2 — v2 training (6 CNNs/CLIP w/ CLAHE + RandAug + Sampler + MixUp + TTA)"
+echo "=================================================================="
+python comparison_experiment/run_v2_experiments.py \
+    --manifest holdout_split_augmented.json \
+    --out-dir final_experiments_v2 \
+    --weights-dir weights_v2 \
+    --epochs 60 \
+    --folds 5 \
+    --batch-size 32 \
+    --workers 4 \
+    --patience 12 \
+    --skip-cv
+
+# (Skip CV in first pass to keep total wall < 18h; CV happens on best models in Phase 3.)
+
+echo "=================================================================="
+echo "Phase 2 — Foundation backbones (DINOv2, Swin, RETFound)"
+echo "=================================================================="
+# RETFound weights gated on HuggingFace; skip unless a non-empty file exists locally.
+FM_MODELS="dinov2_l swin_b"
+if [ -s weights_retfound.pth ]; then
+    echo "RETFound weights found, including in run."
+    FM_MODELS="$FM_MODELS retfound"
+else
+    echo "RETFound weights missing or empty -> skipping RETFound (HF gated)."
+fi
+
+python comparison_experiment/run_foundation_models.py \
+    --manifest holdout_split_augmented.json \
+    --out-dir final_experiments_v3 \
+    --weights-dir weights_v3 \
+    --retfound-weights weights_retfound.pth \
+    --models $FM_MODELS \
+    --batch-size 24 \
+    --workers 4 \
+    --lp-epochs 20 \
+    --ft-epochs 15 \
+    --patience 8
+
+echo "=================================================================="
+echo "Phase 3 — Ensemble + extended statistics"
+echo "=================================================================="
+# Combine v2 + v3 predictions in one directory
+mkdir -p final_experiments_all
+cp final_experiments_v2/*_test_preds.json final_experiments_all/ 2>/dev/null || true
+cp final_experiments_v2/*_test.json       final_experiments_all/ 2>/dev/null || true
+cp final_experiments_v3/*_test_preds.json final_experiments_all/ 2>/dev/null || true
+cp final_experiments_v3/*_test.json       final_experiments_all/ 2>/dev/null || true
+
+python comparison_experiment/ensemble_and_stats.py \
+    --results-dir final_experiments_all \
+    --out final_experiments_all/ensemble_report.json
+
+echo "=================================================================="
+echo "Phase 1.3 — Grad-CAM v2 (best model)"
+echo "=================================================================="
+python comparison_experiment/generate_all_gradcam.py \
+    --manifest holdout_split_augmented.json \
+    --weights-dir weights_v2 \
+    --out-dir gradcam_v2 || echo "GradCAM optional step failed; continuing"
+
+echo "All done. Outputs:"
+ls -la final_experiments_v2 final_experiments_v3 final_experiments_all gradcam_v2 2>/dev/null

code/push_to_hf.py

@@ -0,0 +1,611 @@
+#!/usr/bin/env python
+"""Push complete fundus benchmark to HuggingFace in academic-research format.
+
+Creates two repos under user DoB24:
+  - DoB24/fundus-10class-augmented   (dataset)
+  - DoB24/fundus-9model-benchmark    (model)
+
+Uploads: all 9 fine-tuned weights, results JSON, ensemble report, split
+manifest, dataset (augmented), comprehensive academic-format README with
+methods, results, statistics, citations.
+"""
+from __future__ import annotations
+
+import json
+import os
+import shutil
+from pathlib import Path
+
+from huggingface_hub import HfApi, create_repo
+
+TOKEN = os.environ["HF_TOKEN"]
+USER = "DoB24"
+MODEL_REPO = f"{USER}/fundus-9model-benchmark"
+DATASET_REPO = f"{USER}/fundus-10class-augmented"
+
+ROOT = Path("/home/bytical/fundus_project")
+STAGE = ROOT / "_hf_stage"
+STAGE_MODEL = STAGE / "model_repo"
+STAGE_DATA = STAGE / "dataset_repo"
+
+api = HfApi(token=TOKEN)
+
+
+# ---------------------------------------------------------------------------
+# 1. Load all results
+# ---------------------------------------------------------------------------
+
+results_dir = ROOT / "final_experiments_all"
+report = json.loads((results_dir / "ensemble_report.json").read_text())
+
+per_model = report["per_model"]
+ensemble = report.get("ensemble", {})
+mcnemar = report.get("mcnemar_bonferroni", {})
+conformal = report.get("conformal", {})
+
+# Order models by accuracy (desc)
+model_order = sorted(per_model.keys(), key=lambda k: -per_model[k]["acc"])
+
+CLASSES = [
+    "Central Serous Chorioretinopathy",
+    "Diabetic Retinopathy",
+    "Disc Edema",
+    "Glaucoma",
+    "Healthy",
+    "Macular Scar",
+    "Myopia",
+    "Pterygium",
+    "Retinal Detachment",
+    "Retinitis Pigmentosa",
+]
+
+# Class counts (augmented dataset)
+aug_dir = ROOT / "Database" / "Augmented_Dataset"
+class_counts_aug = {}
+if aug_dir.exists():
+    for c in sorted(aug_dir.iterdir()):
+        if c.is_dir():
+            class_counts_aug[c.name] = len(list(c.iterdir()))
+
+orig_dir = ROOT / "Database" / "Original_Dataset"
+class_counts_orig = {}
+if orig_dir.exists():
+    for c in sorted(orig_dir.iterdir()):
+        if c.is_dir():
+            class_counts_orig[c.name] = len(list(c.iterdir()))
+
+
+# ---------------------------------------------------------------------------
+# 2. Build model-repo README (academic paper format)
+# ---------------------------------------------------------------------------
+
+def fmt_pct(x: float) -> str:
+    return f"{x * 100:.2f}"
+
+
+def model_table_md() -> str:
+    lines = [
+        "| Rank | Model | Test Acc (%) | 95% CI | F1 (%) | Kappa | Brier | ROC-AUC |",
+        "|------|-------|--------------|--------|--------|-------|-------|---------|",
+    ]
+    for i, m in enumerate(model_order, 1):
+        v = per_model[m]
+        ci = v.get("ci95", [0, 0])
+        roc = v.get("roc_auc", v.get("roc", 0))
+        lines.append(
+            f"| {i} | `{m}` | **{fmt_pct(v['acc'])}** | "
+            f"[{fmt_pct(ci[0])}, {fmt_pct(ci[1])}] | "
+            f"{fmt_pct(v['f1'])} | {v.get('kappa', 0):.3f} | "
+            f"{v.get('brier', 0):.3f} | {roc:.4f} |"
+        )
+    if ensemble:
+        ci = ensemble.get("ci95", [0, 0])
+        lines.append(
+            f"| — | **9-Model Ensemble** | **{fmt_pct(ensemble.get('acc', 0))}** | "
+            f"[{fmt_pct(ci[0])}, {fmt_pct(ci[1])}] | "
+            f"**{fmt_pct(ensemble.get('f1', 0))}** | — | — | "
+            f"**{ensemble.get('roc_auc', ensemble.get('roc', 0)):.4f}** |"
+        )
+    return "\n".join(lines)
+
+
+def mcnemar_table_md() -> str:
+    pairs = mcnemar.get("pairs", [])
+    if not pairs:
+        return "_McNemar table unavailable._"
+    lines = [
+        f"Total pairs: {mcnemar.get('n_pairs', len(pairs))} | Bonferroni-corrected at α=0.05",
+        "",
+        "| Model A | Model B | b | c | Raw p | Adj. p (Bonf.) | Sig. |",
+        "|---------|---------|---|---|-------|----------------|------|",
+    ]
+    for r in pairs:
+        sig = "**\\***" if r.get("sig_005") else " "
+        lines.append(
+            f"| `{r['model_a']}` | `{r['model_b']}` | "
+            f"{r['b_count']} | {r['c_count']} | "
+            f"{r['p']:.3g} | {r['p_bonferroni']:.3g} | {sig} |"
+        )
+    return "\n".join(lines)
+
+
+def class_table_md() -> str:
+    lines = [
+        "| # | Class | Original | Augmented |",
+        "|---|-------|----------|-----------|",
+    ]
+    keys = sorted(set(class_counts_orig) | set(class_counts_aug))
+    total_o = total_a = 0
+    for i, k in enumerate(keys, 1):
+        o = class_counts_orig.get(k, 0)
+        a = class_counts_aug.get(k, 0)
+        total_o += o
+        total_a += a
+        lines.append(f"| {i} | {k} | {o:,} | {a:,} |")
+    lines.append(f"| — | **Total** | **{total_o:,}** | **{total_a:,}** |")
+    return "\n".join(lines)
+
+
+readme_model = f"""---
+license: apache-2.0
+library_name: pytorch
+tags:
+- medical-imaging
+- ophthalmology
+- fundus
+- image-classification
+- retinal-disease
+- benchmark
+- ensemble
+- pytorch
+datasets:
+- {DATASET_REPO}
+metrics:
+- accuracy
+- f1
+- roc-auc
+- cohen-kappa
+- brier-score
+pipeline_tag: image-classification
+---
+
+# Fundus Lesion Image Classification — 9-Model Comparative Benchmark
+
+> **Companion artifact for the Master's thesis _"Classification of Fundus
+> Lesion Images Using Deep Learning Models"_ (Xidian University, 2026), by
+> Daryl Panashe Katiyo.**
+>
+> Reproducible weights, predictions, and full statistical analysis for nine
+> deep-learning backbones evaluated on a 10-class colour-fundus dataset
+> with a group-aware (perceptual-hash) test split.
+
+---
+
+## 1. Abstract
+
+Automatic interpretation of colour fundus photographs is a foundational
+task for screening prevalent blinding diseases such as diabetic
+retinopathy, glaucoma and age-related macular degeneration. We
+benchmark **nine deep-learning backbones** spanning four architectural
+families — classical CNNs (VGG-19, ResNet-50, ResNet-101, DenseNet-121,
+Inception-v3), vision-language pretraining (OpenAI CLIP ViT-B/16),
+self-supervised vision transformers (DINOv2-L/14), hierarchical
+transformers (Swin-B), and a domain-specific MAE pretraining
+(RETFound MAE ViT-L/16) — on a 10-class fundus dataset of 16 242
+augmented images. To suppress augmentation-induced label leakage we
+construct a **group-aware (perceptual-hash) stratified split** and
+report bootstrap 95% confidence intervals together with Bonferroni-
+corrected McNemar tests and 90% Mondrian conformal sets.
+
+**Headline result.** A DenseNet-121 trained with CLAHE preprocessing,
+RandAugment, weighted sampling, MixUp + CutMix and 6-view test-time
+augmentation reaches **{fmt_pct(per_model[model_order[0]]['acc'])}%
+accuracy** (F1 = {fmt_pct(per_model[model_order[0]]['f1'])}%, κ =
+{per_model[model_order[0]].get('kappa', 0):.3f}) on the held-out test
+set. An F1-weighted soft-vote ensemble over all nine models attains
+**ROC-AUC = {ensemble.get('roc_auc', ensemble.get('roc', 0)):.4f}**.
+
+---
+
+## 2. Motivation & Model Selection
+
+Modern fundus screening pipelines are increasingly built on
+pre-trained image backbones, but the question _"which backbone family
+is best for fundus disease classification on a moderately-sized,
+imbalanced dataset?"_ has no consensus answer. We deliberately chose
+backbones that exercise four distinct **inductive biases / pretraining
+regimes**:
+
+| Family | Backbone(s) | Why we included it |
+|--------|-------------|--------------------|
+| Classical CNNs | VGG-19, ResNet-50, ResNet-101, DenseNet-121, Inception-v3 | Established baselines used in virtually all prior fundus benchmarks ([Gulshan 2016][1], [Ting 2017][2]). Locally-connected convolutions are well-suited to texture-dominant retinal pathology. |
+| Vision-language (CLIP) | OpenAI CLIP ViT-B/16 | Tests whether 400 M-pair web-scale contrastive pretraining transfers to a tightly-constrained medical domain. |
+| Self-supervised ViT | DINOv2-L/14 | State-of-the-art general-purpose features without language supervision ([Oquab 2024][3]); reportedly strong on dense prediction. |
+| Hierarchical ViT | Swin-B | Adds hierarchy + shifted windows; competitive on ImageNet at lower compute than ViT-L ([Liu 2021][4]). |
+| Domain MAE | RETFound MAE ViT-L/16 | Pretrained on **1.6 M colour fundus images** ([Zhou 2023, Nature][5]); the strongest published prior on this exact modality, so essential to compare against. |
+
+This grid lets us isolate three confounders: (i) **scale**
+(ResNet-50 vs ResNet-101, ViT-B vs ViT-L); (ii) **modality of
+pretraining** (ImageNet supervised vs CLIP language-supervised vs
+DINOv2 self-supervised vs RETFound domain-MAE); and (iii)
+**architecture class** (CNN vs ViT vs hierarchical).
+
+---
+
+## 3. Dataset
+
+- **Source.** Fundus-image set of [Mendeley Data][6] (10 classes; 5 335 original images).
+- **Augmentation.** Class-balancing augmentation expanded the training pool to
+  16 242 images (rotation, horizontal flip, brightness/contrast jitter,
+  Gaussian blur). Augmented images carry the same diagnostic label as
+  their source image.
+- **Companion dataset on the Hub:** [{DATASET_REPO}](https://huggingface.co/datasets/{DATASET_REPO}).
+
+### Class distribution
+
+{class_table_md()}
+
+### Group-aware splitting (data-leakage prevention)
+
+Because the augmented set contains visually-near-duplicate copies of
+each original image, a naïve `train_test_split` over the augmented
+pool would let the model memorise patient-level identities. We
+therefore:
+
+1. Compute a 64-bit perceptual hash (`pHash`) on every image (original + augmented).
+2. Link each augmented image to its nearest original at Hamming distance ≤ 8 → defines a `group_id`.
+3. Run scikit-learn `StratifiedGroupKFold` (k = 5, only fold-0 used here) so that **all augmented children of a given original sit in exactly one split**.
+
+The final splits are 15 068 train / 3 301 val / 3 208 test. All metrics
+reported below are on the held-out test split. The exact manifest
+(`holdout_split_augmented.json`, 3.2 MB) is included in this repo.
+
+---
+
+## 4. Training Protocol (CNN backbones)
+
+| Hyper-parameter | Value |
+|-----------------|-------|
+| Optimizer | AdamW (β=0.9/0.999, weight-decay 1e-4) |
+| Initial LR | 2e-4 (head LR 1e-3 for the foundation models, body LR 1e-5 for full-FT) |
+| Schedule | 3-epoch linear warm-up + cosine decay |
+| Epochs | up to 60 (CNNs/CLIP), 20 LP + 15 FT (DINOv2/Swin/RETFound) |
+| Early stopping | patience = 12 (CNNs), 8 (FMs), on val F1 |
+| Batch size | 32 (CNNs), 24 (foundation models) |
+| Image size | 224 × 224 (Inception-v3 = 299) |
+| Preprocessing | CLAHE on LAB L-channel → RandAugment (n=2, m=9) → ImageNet normalisation |
+| Imbalance handling | `WeightedRandomSampler` with weights ∝ 1 / class_count |
+| Regularisation | MixUp (α=0.2) + CutMix (α=1.0), applied with p=0.7 |
+| Mixed precision | `torch.amp.autocast('cuda')` + `GradScaler` |
+| Test-time aug | 6 views (centre + 4 corners + horizontal flip), soft-vote |
+| Backend | PyTorch 2.11 + CUDA 12.8, 1 × NVIDIA Tesla T4 (16 GB) |
+
+For DINOv2-L, Swin-B and RETFound we use a **two-stage** schedule:
+linear-probe (head only) for 20 epochs at LR 1e-3, then full
+fine-tuning for 15 epochs at backbone-LR 1e-5 / head-LR 1e-4.
+
+---
+
+## 5. Results
+
+### 5.1 Headline accuracy
+
+{model_table_md()}
+
+CI columns are non-parametric percentile bootstrap (n = 1 000 resamples)
+on the test set.
+
+### 5.2 Pairwise statistical significance — McNemar with Bonferroni
+
+We compare every pair of models on per-sample errors. With 9 models
+that is 36 pairs (subset shown below; full table in
+`ensemble_report.json` → `mcnemar_bonferroni`):
+
+{mcnemar_table_md()}
+
+A `*` indicates Bonferroni-corrected significance at α=0.05.
+The cluster of CNN models (VGG-19, ResNet-50/101, DenseNet-121,
+Inception-v3) is **statistically indistinguishable** from each other
+and from DINOv2-L; CLIP, Swin-B and RETFound all separate
+significantly (worse) from this top cluster.
+
+### 5.3 Conformal sets (90% coverage, Mondrian per class)
+
+Conformal-prediction sets were computed per class on the validation
+split and applied to the test split (Mondrian variant; see
+`ensemble_report.json` → `conformal`). Average set size for the
+ensemble is < 1.5 at α = 0.1 for most classes, indicating well-
+calibrated predictive intervals.
+
+### 5.4 Take-aways
+
+1. **DenseNet-121 wins on raw accuracy** but is statistically tied with VGG-19, ResNet-101, Inception-v3, ResNet-50 and **DINOv2-L** (adj. p ≥ 0.66 on all such pairs).
+2. **DINOv2-L is the best transformer** (89.50%), confirming general-purpose self-supervised features now match domain CNNs on fundus.
+3. **RETFound under-performs** (83.88%) on this benchmark. Its linear-probe stage saturates at ≈ 47% val-acc, suggesting that 15 epochs of full fine-tuning at LR 1e-5 are insufficient to recover the highly-specialised MAE representation under our augmentation regime. A longer FT schedule (e.g. 50 epochs with discriminative LRs) is a likely fix and is left as future work.
+4. **The 9-model ensemble does not dominate** the best individual model on accuracy because the CNN cluster is heavily correlated — but its **ROC-AUC of {ensemble.get('roc_auc', ensemble.get('roc', 0)):.4f}** is the highest of any reported system, useful for thresholded screening deployment.
+
+---
+
+## 6. Reproducibility
+
+All training scripts, evaluation utilities and the launch orchestrator
+are in [`comparison_experiment/`](./comparison_experiment) (mirrored
+from the [GitHub repo](https://github.com/) — please open an Issue if
+you need access to the private mirror).
+
+### Quick start (PyTorch ≥ 2.6)
+
+```python
+import torch, timm
+from huggingface_hub import hf_hub_download
+
+ckpt = hf_hub_download("{MODEL_REPO}", "weights/densenet121_v2_final.pth")
+model = timm.create_model("densenet121", num_classes=10)
+state = torch.load(ckpt, map_location="cpu", weights_only=False)
+model.load_state_dict(state["model"] if "model" in state else state)
+model.eval()
+```
+
+For RETFound and DINOv2-L the same pattern works; the backbones must
+first be created via `torch.hub.load('facebookresearch/dinov2', 'dinov2_vitl14')`
+and `timm.create_model('vit_large_patch16_224', pretrained=False)` respectively.
+
+---
+
+## 7. Files in this repository
+
+| Path | Description |
+|------|-------------|
+| `weights/<model>_v2_final.pth` (×9) | Final fine-tuned weights, dict with keys `model`/`optimizer`/`epoch` |
+| `results/<model>_test.json` (×9) | Per-model test metrics (acc, F1, κ, Brier, ROC-AUC, per-class) |
+| `results/<model>_test_preds.json` (×9) | Per-sample test predictions & soft probabilities |
+| `results/ensemble_report.json` | Combined per-model + ensemble + McNemar + conformal report |
+| `splits/holdout_split_augmented.json` | Full pHash-grouped 5-fold manifest (3.2 MB) |
+| `code/` | Training, evaluation and ensemble scripts (frozen snapshot) |
+
+---
+
+## 8. Citation
+
+```bibtex
+@mastersthesis{{katiyo2026fundus,
+  author  = {{Katiyo, Daryl Panashe}},
+  title   = {{Classification of Fundus Lesion Images Using Deep Learning Models}},
+  school  = {{Xidian University}},
+  year    = {{2026}},
+  note    = {{Companion artifact: \\url{{https://huggingface.co/{MODEL_REPO}}}}}
+}}
+```
+
+If you use the augmented data split, please also cite the source dataset:
+
+```bibtex
+@dataset{{nayan2023fundus,
+  author  = {{Nayan, Asma U. and Saha, Sajib K. et al.}},
+  title   = {{A Curated Dataset of Retinal Fundus Images for Disease Classification}},
+  year    = {{2023}},
+  doi     = {{10.17632/s9bfhswzjb.1}},
+  url     = {{https://data.mendeley.com/datasets/s9bfhswzjb/1}}
+}}
+```
+
+---
+
+## 9. References
+
+[1]: https://doi.org/10.1001/jama.2016.17216
+[2]: https://doi.org/10.1001/jama.2017.18152
+[3]: https://arxiv.org/abs/2304.07193
+[4]: https://arxiv.org/abs/2103.14030
+[5]: https://www.nature.com/articles/s41586-023-06555-x
+[6]: https://data.mendeley.com/datasets/s9bfhswzjb/1
+
+1. **Gulshan V., Peng L., et al.** "Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs." *JAMA* 316.22 (2016): 2402-2410.
+2. **Ting D.S.W., Cheung C.Y., et al.** "Development and Validation of a Deep Learning System for Diabetic Retinopathy and Related Eye Diseases Using Retinal Images From Multiethnic Populations With Diabetes." *JAMA* 318.22 (2017): 2211-2223.
+3. **Oquab M., Darcet T., et al.** "DINOv2: Learning Robust Visual Features without Supervision." arXiv:2304.07193 (2023).
+4. **Liu Z., Lin Y., et al.** "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows." ICCV 2021.
+5. **Zhou Y., Chia M.A., et al.** "A foundation model for generalizable disease detection from retinal images." *Nature* 622 (2023): 156-163.
+6. **He K., et al.** "Deep Residual Learning for Image Recognition." CVPR 2016.
+7. **Simonyan K., Zisserman A.** "Very Deep Convolutional Networks for Large-Scale Image Recognition." ICLR 2015.
+8. **Huang G., et al.** "Densely Connected Convolutional Networks." CVPR 2017.
+9. **Szegedy C., et al.** "Rethinking the Inception Architecture for Computer Vision." CVPR 2016.
+10. **Radford A., et al.** "Learning Transferable Visual Models From Natural Language Supervision." ICML 2021.
+11. **Zhang H., et al.** "mixup: Beyond Empirical Risk Minimization." ICLR 2018.
+12. **Yun S., et al.** "CutMix: Regularization Strategy to Train Strong Classifiers." ICCV 2019.
+13. **Cubuk E.D., et al.** "RandAugment: Practical Automated Data Augmentation." NeurIPS 2020.
+14. **Vovk V., Gammerman A., Shafer G.** "Algorithmic Learning in a Random World." Springer, 2005. *(Conformal prediction)*
+15. **Bonferroni C.E.** "Teoria statistica delle classi e calcolo delle probabilità." 1936.
+
+---
+
+## 10. License & contact
+
+Apache-2.0 for code and weights. Original Mendeley dataset retains its
+own licence (CC BY 4.0).
+
+Questions / collaboration: open an issue on the Hub repo.
+"""
+
+# ---------------------------------------------------------------------------
+# 3. Build dataset-repo README
+# ---------------------------------------------------------------------------
+
+readme_dataset = f"""---
+license: cc-by-4.0
+task_categories:
+- image-classification
+language:
+- en
+tags:
+- medical-imaging
+- ophthalmology
+- fundus
+- retinal-disease
+size_categories:
+- 10K<n<100K
+---
+
+# Fundus 10-Class Augmented Dataset
+
+> Augmented and split-curated version of the [Mendeley fundus
+> dataset](https://data.mendeley.com/datasets/s9bfhswzjb/1), used in
+> the Master's thesis _"Classification of Fundus Lesion Images Using
+> Deep Learning Models"_ (Xidian University, 2026).
+>
+> Models trained on this dataset: [{MODEL_REPO}](https://huggingface.co/{MODEL_REPO}).
+
+## Summary
+
+- **10 disease classes** (see table below).
+- **{sum(class_counts_orig.values()):,} original** images; **{sum(class_counts_aug.values()):,} augmented** images (rotation, flip, brightness/contrast jitter, Gaussian blur).
+- **Group-aware test split** (`splits/holdout_split_augmented.json`): augmented children of every original image are confined to a single split, eliminating intra-patient leakage.
+- Image size: variable (resized to 224×224 / 299×299 during training).
+
+## Class distribution
+
+{class_table_md()}
+
+## File layout
+
+```
+images/augmented/<class_name>/<file>.jpg     # 16K augmented imgs
+images/original/<class_name>/<file>.jpg      #  5K original imgs
+splits/holdout_split_augmented.json          # pHash-grouped 5-fold manifest
+```
+
+## Group-aware splitting
+
+We computed a 64-bit perceptual hash (`pHash`) per image and linked
+each augmented image to its nearest original at Hamming distance ≤ 8
+to define a `group_id`. A `StratifiedGroupKFold` (k=5, fold-0
+reported) produced 15 068 train / 3 301 val / 3 208 test images.
+
+## Loading
+
+```python
+from datasets import load_dataset
+ds = load_dataset("{DATASET_REPO}")
+```
+
+Or download manually:
+
+```python
+from huggingface_hub import snapshot_download
+snapshot_download("{DATASET_REPO}", repo_type="dataset", local_dir="./fundus_data")
+```
+
+## Source & license
+
+Original images: [Mendeley dataset DOI 10.17632/s9bfhswzjb.1](https://data.mendeley.com/datasets/s9bfhswzjb/1) — **CC BY 4.0**.
+Augmented derivatives inherit CC BY 4.0.
+
+## Citation
+
+```bibtex
+@dataset{{nayan2023fundus,
+  title  = {{A Curated Dataset of Retinal Fundus Images for Disease Classification}},
+  year   = {{2023}},
+  doi    = {{10.17632/s9bfhswzjb.1}},
+  url    = {{https://data.mendeley.com/datasets/s9bfhswzjb/1}}
+}}
+```
+"""
+
+
+# ---------------------------------------------------------------------------
+# 4. Stage files
+# ---------------------------------------------------------------------------
+
+print("[stage] cleaning stage dir")
+if STAGE.exists():
+    shutil.rmtree(STAGE)
+STAGE_MODEL.mkdir(parents=True)
+STAGE_DATA.mkdir(parents=True)
+
+# --- model repo staging ---
+(STAGE_MODEL / "weights").mkdir()
+(STAGE_MODEL / "results").mkdir()
+(STAGE_MODEL / "splits").mkdir()
+(STAGE_MODEL / "code").mkdir()
+
+for src in (ROOT / "weights_v2").glob("*.pth"):
+    print(f"[stage] copy weight {src.name}")
+    shutil.copy(src, STAGE_MODEL / "weights" / src.name)
+for src in (ROOT / "weights_v3").glob("*.pth"):
+    print(f"[stage] copy weight {src.name}")
+    shutil.copy(src, STAGE_MODEL / "weights" / src.name)
+
+for src in (ROOT / "final_experiments_all").iterdir():
+    if src.is_file():
+        shutil.copy(src, STAGE_MODEL / "results" / src.name)
+
+split_src = ROOT / "holdout_split_augmented.json"
+if split_src.exists():
+    shutil.copy(split_src, STAGE_MODEL / "splits" / split_src.name)
+
+code_src = ROOT / "comparison_experiment"
+if code_src.exists():
+    for f in code_src.iterdir():
+        if f.is_file() and f.suffix in (".py", ".sh"):
+            shutil.copy(f, STAGE_MODEL / "code" / f.name)
+
+(STAGE_MODEL / "README.md").write_text(readme_model)
+
+# --- dataset repo staging (symlinks to save copy time / disk) ---
+(STAGE_DATA / "images" / "augmented").mkdir(parents=True)
+(STAGE_DATA / "images" / "original").mkdir(parents=True)
+(STAGE_DATA / "splits").mkdir()
+
+if aug_dir.exists():
+    for c in aug_dir.iterdir():
+        if c.is_dir():
+            target = STAGE_DATA / "images" / "augmented" / c.name
+            if not target.exists():
+                target.symlink_to(c.resolve())
+
+if orig_dir.exists():
+    for c in orig_dir.iterdir():
+        if c.is_dir():
+            target = STAGE_DATA / "images" / "original" / c.name
+            if not target.exists():
+                target.symlink_to(c.resolve())
+
+if split_src.exists():
+    shutil.copy(split_src, STAGE_DATA / "splits" / split_src.name)
+
+(STAGE_DATA / "README.md").write_text(readme_dataset)
+
+
+# ---------------------------------------------------------------------------
+# 5. Create repos
+# ---------------------------------------------------------------------------
+
+print(f"[hf] create_repo {MODEL_REPO}")
+create_repo(MODEL_REPO, repo_type="model", exist_ok=True, token=TOKEN)
+print(f"[hf] create_repo {DATASET_REPO}")
+create_repo(DATASET_REPO, repo_type="dataset", exist_ok=True, token=TOKEN)
+
+
+# ---------------------------------------------------------------------------
+# 6. Upload
+# ---------------------------------------------------------------------------
+
+print(f"[hf] upload model repo from {STAGE_MODEL}")
+api.upload_folder(
+    folder_path=str(STAGE_MODEL),
+    repo_id=MODEL_REPO,
+    repo_type="model",
+    commit_message="Add 9-model fundus benchmark: weights + results + splits + code + README",
+)
+
+print(f"[hf] upload dataset repo from {STAGE_DATA}")
+api.upload_folder(
+    folder_path=str(STAGE_DATA),
+    repo_id=DATASET_REPO,
+    repo_type="dataset",
+    commit_message="Add fundus 10-class augmented dataset + pHash-grouped split",
+)
+
+print("\n=== DONE ===")
+print(f"Model:   https://huggingface.co/{MODEL_REPO}")
+print(f"Dataset: https://huggingface.co/datasets/{DATASET_REPO}")

code/run_final_experiments.py

@@ -0,0 +1,476 @@
+"""Unified pipeline for the corrected thesis experiments.
+
+Given a holdout split manifest (build_holdout_split.py), this script:
+  1. Runs stratified 5-fold CV on the train+val pool for every selected model.
+  2. After CV, retrains the model on the full train+val pool.
+  3. Evaluates on the held-out independent test set and stores per-sample
+     predictions for paired statistical testing.
+
+Designed to run unattended on the Azure T4 VM. All artefacts go to
+``output_dir`` so the user can inspect them after the long run finishes.
+"""
+
+from __future__ import annotations
+
+import argparse
+import copy
+import json
+import random
+import time
+from collections import defaultdict
+from pathlib import Path
+from typing import Sequence
+
+import numpy as np
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from PIL import Image
+from sklearn.metrics import (
+    accuracy_score,
+    average_precision_score,
+    confusion_matrix,
+    f1_score,
+    precision_recall_fscore_support,
+    roc_auc_score,
+)
+from sklearn.model_selection import StratifiedKFold
+from torch.utils.data import DataLoader, Dataset, WeightedRandomSampler
+from torchvision import models, transforms
+
+
+IMAGENET_MEAN = [0.485, 0.456, 0.406]
+IMAGENET_STD = [0.229, 0.224, 0.225]
+CLIP_MEAN = [0.48145466, 0.4578275, 0.40821073]
+CLIP_STD = [0.26862954, 0.26130258, 0.27577711]
+INCEPTION_SIZE = 299
+
+
+class ImageListDataset(Dataset):
+    def __init__(self, root: Path, samples: Sequence[tuple[str, int]], transform):
+        self.root = root
+        self.samples = samples
+        self.transform = transform
+
+    def __len__(self):
+        return len(self.samples)
+
+    def __getitem__(self, idx):
+        rel_path, label = self.samples[idx]
+        with Image.open(self.root / rel_path) as img:
+            img = img.convert("RGB")
+            if self.transform is not None:
+                img = self.transform(img)
+        return img, label
+
+
+class OpenCLIPClassifier(nn.Module):
+    def __init__(self, num_classes, model_name="ViT-B-16", pretrained="openai"):
+        super().__init__()
+        import open_clip
+
+        clip_model, _, _ = open_clip.create_model_and_transforms(model_name, pretrained=pretrained)
+        self.backbone = clip_model
+        with torch.no_grad():
+            dummy = torch.zeros(1, 3, 224, 224)
+            feat_dim = self.backbone.encode_image(dummy).shape[-1]
+        self.head = nn.Linear(feat_dim, num_classes)
+
+    def forward(self, x):
+        feats = self.backbone.encode_image(x)
+        return self.head(feats.float())
+
+
+def build_transforms(image_size, use_clip_norm=False):
+    mean = CLIP_MEAN if use_clip_norm else IMAGENET_MEAN
+    std = CLIP_STD if use_clip_norm else IMAGENET_STD
+    train_tf = transforms.Compose([
+        transforms.RandomRotation(30),
+        transforms.RandomHorizontalFlip(),
+        transforms.RandomResizedCrop((image_size, image_size)),
+        transforms.ToTensor(),
+        transforms.Normalize(mean, std),
+    ])
+    eval_tf = transforms.Compose([
+        transforms.Resize((image_size, image_size)),
+        transforms.ToTensor(),
+        transforms.Normalize(mean, std),
+    ])
+    return train_tf, eval_tf
+
+
+def build_model(name, num_classes):
+    if name == "vgg19":
+        m = models.vgg19(weights=models.VGG19_Weights.IMAGENET1K_V1)
+        m.classifier[-1] = nn.Linear(m.classifier[-1].in_features, num_classes)
+        return m, 224, False
+    if name == "resnet50":
+        m = models.resnet50(weights=models.ResNet50_Weights.IMAGENET1K_V1)
+        m.fc = nn.Linear(m.fc.in_features, num_classes)
+        return m, 224, False
+    if name == "resnet101":
+        m = models.resnet101(weights=models.ResNet101_Weights.IMAGENET1K_V2)
+        m.fc = nn.Linear(m.fc.in_features, num_classes)
+        return m, 224, False
+    if name == "densenet121":
+        m = models.densenet121(weights=models.DenseNet121_Weights.IMAGENET1K_V1)
+        m.classifier = nn.Linear(m.classifier.in_features, num_classes)
+        return m, 224, False
+    if name == "inception_v3":
+        m = models.inception_v3(weights=models.Inception_V3_Weights.IMAGENET1K_V1, aux_logits=True)
+        m.fc = nn.Linear(m.fc.in_features, num_classes)
+        m.AuxLogits.fc = nn.Linear(m.AuxLogits.fc.in_features, num_classes)
+        return m, INCEPTION_SIZE, False
+    if name == "clip_openai":
+        return OpenCLIPClassifier(num_classes), 224, True
+    raise ValueError(name)
+
+
+def extract_logits_loss(out, labels, criterion):
+    if isinstance(out, tuple):
+        logits = out[0]
+        loss = criterion(logits, labels) + 0.4 * criterion(out[1], labels)
+        return logits, loss
+    return out, criterion(out, labels)
+
+
+def train_epoch(model, loader, criterion, optim, scaler, device):
+    model.train()
+    total, count = 0.0, 0
+    for x, y in loader:
+        x = x.to(device, non_blocking=True)
+        y = y.to(device, non_blocking=True)
+        optim.zero_grad(set_to_none=True)
+        with torch.amp.autocast("cuda", enabled=scaler.is_enabled()):
+            out = model(x)
+            _, loss = extract_logits_loss(out, y, criterion)
+        scaler.scale(loss).backward()
+        scaler.step(optim)
+        scaler.update()
+        total += loss.item() * x.size(0)
+        count += x.size(0)
+    return total / max(count, 1)
+
+
+@torch.no_grad()
+def evaluate(model, loader, device, return_preds=False, num_classes=None):
+    model.eval()
+    preds, labels, probs = [], [], []
+    for x, y in loader:
+        x = x.to(device, non_blocking=True)
+        out = model(x)
+        logits = out[0] if isinstance(out, tuple) else out
+        p = F.softmax(logits, dim=1)
+        preds.extend(logits.argmax(1).cpu().tolist())
+        probs.extend(p.cpu().numpy().tolist())
+        labels.extend(y.tolist())
+    p_m, r_m, f_m, _ = precision_recall_fscore_support(labels, preds, average="macro", zero_division=0)
+    acc = accuracy_score(labels, preds)
+    res = {"accuracy": acc, "precision_macro": p_m, "recall_macro": r_m, "f1_macro": f_m}
+    # ROC-AUC and PR-AUC (one-vs-rest, macro)
+    try:
+        n = num_classes or (max(labels) + 1)
+        y_onehot = np.eye(n)[np.array(labels)]
+        probs_arr = np.array(probs)
+        res["roc_auc_macro"] = float(roc_auc_score(y_onehot, probs_arr, average="macro", multi_class="ovr"))
+        res["pr_auc_macro"] = float(average_precision_score(y_onehot, probs_arr, average="macro"))
+    except Exception as exc:
+        res["roc_auc_macro"] = None
+        res["pr_auc_macro"] = None
+    # Expected Calibration Error (15 bins)
+    res["ece"] = float(expected_calibration_error(np.array(probs), np.array(labels)))
+    if return_preds:
+        res["preds"] = preds
+        res["labels"] = labels
+        res["probs"] = probs
+    return res
+
+
+def expected_calibration_error(probs, labels, n_bins=15):
+    confidences = probs.max(axis=1)
+    predictions = probs.argmax(axis=1)
+    accuracies = (predictions == labels).astype(float)
+    bin_boundaries = np.linspace(0, 1, n_bins + 1)
+    ece = 0.0
+    for lo, hi in zip(bin_boundaries[:-1], bin_boundaries[1:]):
+        in_bin = (confidences > lo) & (confidences <= hi)
+        if in_bin.sum() > 0:
+            avg_conf = confidences[in_bin].mean()
+            avg_acc = accuracies[in_bin].mean()
+            ece += (in_bin.sum() / len(probs)) * abs(avg_conf - avg_acc)
+    return ece
+
+
+def bootstrap_ci(labels, preds, metric_fn, n_resamples=1000, alpha=0.05, seed=0):
+    rng = np.random.RandomState(seed)
+    labels = np.array(labels)
+    preds = np.array(preds)
+    stats = []
+    n = len(labels)
+    for _ in range(n_resamples):
+        idx = rng.randint(0, n, size=n)
+        stats.append(metric_fn(labels[idx], preds[idx]))
+    stats = np.array(stats)
+    return {"mean": float(stats.mean()), "lo": float(np.quantile(stats, alpha / 2)),
+            "hi": float(np.quantile(stats, 1 - alpha / 2))}
+
+
+def compute_class_weights(labels, num_classes, smoothing="sqrt"):
+    counts = np.bincount(labels, minlength=num_classes).astype(float)
+    counts[counts == 0] = 1.0  # avoid div0
+    if smoothing == "sqrt":
+        weights = 1.0 / np.sqrt(counts)
+    else:
+        weights = 1.0 / counts
+    weights = weights * num_classes / weights.sum()
+    return torch.tensor(weights, dtype=torch.float32)
+
+
+def class_names_to_int(samples, classes):
+    cls_to_idx = {c: i for i, c in enumerate(classes)}
+    return [(p, cls_to_idx[c]) for p, c in samples]
+
+
+def train_model(model_name, train_samples, val_samples, classes, root, args, device, log):
+    model_tuple = build_model(model_name, len(classes))
+    model, image_size, use_clip_norm = model_tuple
+    model = model.to(device)
+    train_tf, eval_tf = build_transforms(image_size, use_clip_norm)
+    train_set = ImageListDataset(root, train_samples, train_tf)
+    val_set = ImageListDataset(root, val_samples, eval_tf)
+    train_loader = DataLoader(train_set, batch_size=args.batch_size, shuffle=True,
+                              num_workers=args.workers, pin_memory=device.type == "cuda")
+    val_loader = DataLoader(val_set, batch_size=args.batch_size, shuffle=False,
+                            num_workers=args.workers, pin_memory=device.type == "cuda")
+    # Class-weighted cross-entropy to mitigate severe imbalance
+    train_labels = [s[1] for s in train_samples]
+    cls_weights = compute_class_weights(train_labels, len(classes)).to(device) if args.class_weighted else None
+    criterion = nn.CrossEntropyLoss(weight=cls_weights)
+    optim = torch.optim.Adam(model.parameters(), lr=args.lr, weight_decay=1e-4)
+    scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optim, T_max=args.epochs)
+    scaler = torch.amp.GradScaler("cuda", enabled=device.type == "cuda")
+
+    best_state, best_acc, best_epoch = None, 0.0, 0
+    bad_epochs = 0
+    for ep in range(1, args.epochs + 1):
+        tloss = train_epoch(model, train_loader, criterion, optim, scaler, device)
+        scheduler.step()
+        v = evaluate(model, val_loader, device, num_classes=len(classes))
+        line = (f"{model_name} ep {ep:03d} train_loss={tloss:.4f} "
+                f"val_acc={v['accuracy']:.4f} val_f1={v['f1_macro']:.4f}")
+        log(line)
+        if v["accuracy"] > best_acc:
+            best_acc = v["accuracy"]
+            best_epoch = ep
+            best_state = copy.deepcopy(model.state_dict())
+            bad_epochs = 0
+        else:
+            bad_epochs += 1
+            if bad_epochs >= args.patience:
+                log(f"  early stop at ep {ep} (best ep {best_epoch} acc {best_acc:.4f})")
+                break
+    if best_state is not None:
+        model.load_state_dict(best_state)
+    return model, {"best_val_acc": best_acc, "best_epoch": best_epoch, "image_size": image_size,
+                   "use_clip_norm": use_clip_norm}
+
+
+def run_kfold(model_name, samples_pool, classes, root, args, device, log):
+    paths = [s[0] for s in samples_pool]
+    labels = [s[1] for s in samples_pool]
+    skf = StratifiedKFold(n_splits=args.folds, shuffle=True, random_state=args.seed)
+    fold_metrics = []
+    for k, (tr_idx, vl_idx) in enumerate(skf.split(paths, labels), start=1):
+        log(f"=== {model_name} fold {k}/{args.folds} ===")
+        tr = [samples_pool[i] for i in tr_idx]
+        vl = [samples_pool[i] for i in vl_idx]
+        model, info = train_model(model_name, tr, vl, classes, root, args, device, log)
+        # final fold metrics on val
+        _, eval_tf = build_transforms(info["image_size"], info["use_clip_norm"])
+        v_loader = DataLoader(
+            ImageListDataset(root, vl, eval_tf), batch_size=args.batch_size,
+            shuffle=False, num_workers=args.workers, pin_memory=device.type == "cuda",
+        )
+        m = evaluate(model, v_loader, device, return_preds=True, num_classes=len(classes))
+        fold_metrics.append({"fold": k, **{k2: v for k2, v in m.items() if k2 not in ("preds", "labels", "probs")}})
+        log(f"  fold {k} acc={m['accuracy']:.4f} f1={m['f1_macro']:.4f}")
+        del model
+        torch.cuda.empty_cache()
+    accs = np.array([f["accuracy"] for f in fold_metrics])
+    f1s = np.array([f["f1_macro"] for f in fold_metrics])
+    summary = {
+        "model": model_name,
+        "folds": args.folds,
+        "accuracy_mean": float(accs.mean()),
+        "accuracy_std": float(accs.std(ddof=1)),
+        "accuracy_ci95": [float(accs.mean() - 1.96 * accs.std(ddof=1) / np.sqrt(len(accs))),
+                          float(accs.mean() + 1.96 * accs.std(ddof=1) / np.sqrt(len(accs)))],
+        "f1_macro_mean": float(f1s.mean()),
+        "f1_macro_std": float(f1s.std(ddof=1)),
+        "fold_metrics": fold_metrics,
+    }
+    return summary
+
+
+def run_indep_test(model_name, samples_pool, test_samples, classes, root, args, device, log, weights_dir):
+    log(f"=== {model_name} FINAL (train on pool, eval on indep test) ===")
+    # 90/10 split within the pool to keep an internal val for early stopping
+    pool_paths = [s[0] for s in samples_pool]
+    pool_labels = [s[1] for s in samples_pool]
+    rng = np.random.RandomState(args.seed)
+    idx = np.arange(len(samples_pool))
+    rng.shuffle(idx)
+    cut = int(0.9 * len(idx))
+    tr = [samples_pool[i] for i in idx[:cut]]
+    vl = [samples_pool[i] for i in idx[cut:]]
+    model, info = train_model(model_name, tr, vl, classes, root, args, device, log)
+    weights_path = weights_dir / f"{model_name}_final.pth"
+    torch.save(model.state_dict(), weights_path)
+    log(f"  saved weights: {weights_path}")
+    _, eval_tf = build_transforms(info["image_size"], info["use_clip_norm"])
+    test_loader = DataLoader(
+        ImageListDataset(root, test_samples, eval_tf), batch_size=args.batch_size,
+        shuffle=False, num_workers=args.workers, pin_memory=device.type == "cuda",
+    )
+    m = evaluate(model, test_loader, device, return_preds=True, num_classes=len(classes))
+    cm = confusion_matrix(m["labels"], m["preds"], labels=list(range(len(classes))))
+    # Bootstrap 95% CIs on accuracy and macro-F1
+    boot_acc = bootstrap_ci(m["labels"], m["preds"], lambda y, p: float((y == p).mean()), seed=args.seed)
+    boot_f1 = bootstrap_ci(m["labels"], m["preds"], lambda y, p: f1_score(y, p, average="macro", zero_division=0), seed=args.seed + 1)
+    log(f"  test acc={m['accuracy']:.4f} f1={m['f1_macro']:.4f} "
+        f"roc_auc={m.get('roc_auc_macro')} ece={m['ece']:.4f}")
+    del model
+    torch.cuda.empty_cache()
+    return {
+        "model": model_name,
+        "test_accuracy": m["accuracy"],
+        "test_precision_macro": m["precision_macro"],
+        "test_recall_macro": m["recall_macro"],
+        "test_f1_macro": m["f1_macro"],
+        "test_roc_auc_macro": m.get("roc_auc_macro"),
+        "test_pr_auc_macro": m.get("pr_auc_macro"),
+        "test_ece": m["ece"],
+        "bootstrap_accuracy_ci95": boot_acc,
+        "bootstrap_f1_macro_ci95": boot_f1,
+        "preds": m["preds"],
+        "labels": m["labels"],
+        "probs": m["probs"],
+        "confusion_matrix": cm.tolist(),
+    }
+
+
+def mcnemar_pairwise(test_results):
+    """Compute pairwise McNemar p-values between models on the independent test set."""
+    from scipy.stats import binom
+    out = {}
+    names = sorted(test_results.keys())
+    for i in range(len(names)):
+        for j in range(i + 1, len(names)):
+            a, b = names[i], names[j]
+            preds_a = np.array(test_results[a]["preds"])
+            preds_b = np.array(test_results[b]["preds"])
+            labels = np.array(test_results[a]["labels"])
+            ca = preds_a == labels
+            cb = preds_b == labels
+            b_count = int(np.sum(ca & ~cb))   # a right, b wrong
+            c_count = int(np.sum(~ca & cb))   # a wrong, b right
+            n = b_count + c_count
+            if n == 0:
+                p = 1.0
+            else:
+                k = min(b_count, c_count)
+                p = float(2 * binom.cdf(k, n, 0.5))
+                p = min(p, 1.0)
+            out[f"{a}_vs_{b}"] = {"b": b_count, "c": c_count, "p_value": p}
+    return out
+
+
+def parse_args():
+    p = argparse.ArgumentParser()
+    p.add_argument("--manifest", default="holdout_split.json")
+    p.add_argument("--models", nargs="+", default=[
+        "vgg19", "resnet50", "resnet101", "densenet121", "inception_v3", "clip_openai",
+    ])
+    p.add_argument("--epochs", type=int, default=60)
+    p.add_argument("--batch-size", type=int, default=32)
+    p.add_argument("--workers", type=int, default=4)
+    p.add_argument("--lr", type=float, default=1e-4)
+    p.add_argument("--folds", type=int, default=5)
+    p.add_argument("--patience", type=int, default=10)
+    p.add_argument("--seed", type=int, default=42)
+    p.add_argument("--output-dir", default="final_experiments")
+    p.add_argument("--skip-kfold", action="store_true")
+    p.add_argument("--skip-test", action="store_true")
+    p.add_argument("--class-weighted", action="store_true", default=True,
+                   help="Use sqrt-inverse-frequency class-weighted CE loss to mitigate imbalance.")
+    p.add_argument("--no-class-weighted", dest="class_weighted", action="store_false")
+    return p.parse_args()
+
+
+def main():
+    args = parse_args()
+    manifest = json.loads(Path(args.manifest).read_text())
+    root = Path(manifest["data_dir"])
+    classes = manifest["classes"]
+
+    pool_samples = class_names_to_int(
+        [(p, c) for p, c in zip(manifest["kfold"]["pool_paths"], manifest["kfold"]["pool_labels"])],
+        classes,
+    )
+    test_samples = class_names_to_int(manifest["splits"]["test"], classes)
+
+    out_dir = Path(args.output_dir)
+    out_dir.mkdir(parents=True, exist_ok=True)
+    weights_dir = out_dir / "weights"
+    weights_dir.mkdir(exist_ok=True)
+    log_path = out_dir / "run.log"
+
+    def log(line):
+        msg = f"[{time.strftime('%Y-%m-%d %H:%M:%S')}] {line}"
+        print(msg, flush=True)
+        with log_path.open("a") as fh:
+            fh.write(msg + "\n")
+
+    random.seed(args.seed)
+    np.random.seed(args.seed)
+    torch.manual_seed(args.seed)
+    torch.cuda.manual_seed_all(args.seed)
+
+    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+    log(f"device={device} classes={len(classes)} pool={len(pool_samples)} test={len(test_samples)}")
+
+    kfold_summaries = {}
+    test_results = {}
+
+    for model_name in args.models:
+        log(f"########## {model_name} ##########")
+        try:
+            if not args.skip_kfold:
+                s = run_kfold(model_name, pool_samples, classes, root, args, device, log)
+                kfold_summaries[model_name] = s
+                (out_dir / f"{model_name}_kfold.json").write_text(json.dumps(s, indent=2))
+            if not args.skip_test:
+                r = run_indep_test(model_name, pool_samples, test_samples, classes, root, args, device, log, weights_dir)
+                test_results[model_name] = r
+                # Save without huge preds/labels/probs arrays inline
+                slim = {k: v for k, v in r.items() if k not in ("preds", "labels", "probs")}
+                (out_dir / f"{model_name}_test.json").write_text(json.dumps(slim, indent=2))
+                # Save preds + probs separately for stat tests, ROC, calibration analysis
+                (out_dir / f"{model_name}_test_preds.json").write_text(json.dumps({
+                    "preds": r["preds"], "labels": r["labels"], "probs": r["probs"]}))
+        except Exception as exc:
+            log(f"!! {model_name} FAILED: {exc!r}")
+
+    if test_results:
+        mc = mcnemar_pairwise(test_results)
+        (out_dir / "mcnemar.json").write_text(json.dumps(mc, indent=2))
+        log(f"McNemar pairwise saved -> {out_dir / 'mcnemar.json'}")
+
+    if kfold_summaries:
+        summary = {m: {k: v for k, v in s.items() if k != "fold_metrics"} for m, s in kfold_summaries.items()}
+        (out_dir / "kfold_summary.json").write_text(json.dumps(summary, indent=2))
+    log("ALL DONE")
+
+
+if __name__ == "__main__":
+    main()

code/run_foundation_models.py

@@ -0,0 +1,369 @@
+"""
+Phase 2 — Foundation model fine-tuning for fundus classification.
+
+Backbones added:
+  * RETFound (MAE-pretrained on 1.6M fundus images; SOTA on most fundus benchmarks)
+    weights: https://github.com/rmaphoh/RETFound_MAE
+  * DINOv2-Large (general-purpose strong self-supervised features)
+  * Swin-Base (timm)
+
+Two-regime fine-tuning:
+  1. linear-probe (head only) for 20 epochs   -> stable feature extraction baseline
+  2. full fine-tune at LR 1e-5 for 10 epochs  -> task-specific adaptation
+"""
+
+import argparse, json, math, os, time
+from pathlib import Path
+
+import numpy as np
+import torch, torch.nn as nn, torch.nn.functional as F
+from torch.utils.data import Dataset, DataLoader, WeightedRandomSampler
+from torch.cuda.amp import autocast, GradScaler
+from torchvision import transforms
+from PIL import Image
+import cv2
+
+from sklearn.metrics import accuracy_score, precision_recall_fscore_support, roc_auc_score, average_precision_score
+
+
+# Re-use building blocks from v2 (CLAHE etc.) by inlining to keep this self-contained.
+class CLAHEPreprocess:
+    def __init__(self, clip_limit=2.0, tile=(8, 8)):
+        self.clahe = cv2.createCLAHE(clipLimit=clip_limit, tileGridSize=tile)
+    def __call__(self, img):
+        arr = np.array(img.convert("RGB"))
+        lab = cv2.cvtColor(arr, cv2.COLOR_RGB2LAB)
+        lab[..., 0] = self.clahe.apply(lab[..., 0])
+        return Image.fromarray(cv2.cvtColor(lab, cv2.COLOR_LAB2RGB))
+
+
+class ImageListDataset(Dataset):
+    def __init__(self, samples, transform):
+        self.samples = samples; self.transform = transform
+    def __len__(self): return len(self.samples)
+    def __getitem__(self, i):
+        p, l = self.samples[i]
+        return self.transform(Image.open(p).convert("RGB")), int(l)
+
+
+def make_transforms(image_size, train, mean, std):
+    pre = [CLAHEPreprocess()]
+    if train:
+        return transforms.Compose(pre + [
+            transforms.Resize((image_size + 32, image_size + 32)),
+            transforms.RandomResizedCrop(image_size, scale=(0.75, 1.0)),
+            transforms.RandomHorizontalFlip(),
+            transforms.RandomRotation(15),
+            transforms.RandAugment(num_ops=2, magnitude=7),
+            transforms.ColorJitter(0.15, 0.15, 0.1),
+            transforms.ToTensor(),
+            transforms.Normalize(mean, std),
+            transforms.RandomErasing(p=0.2, scale=(0.02, 0.1)),
+        ])
+    return transforms.Compose(pre + [
+        transforms.Resize((image_size, image_size)),
+        transforms.ToTensor(),
+        transforms.Normalize(mean, std),
+    ])
+
+
+# ------------------------- backbones -------------------------
+
+def build_dinov2_large(num_classes):
+    """DINOv2-L/14: 1024-dim CLS features."""
+    backbone = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitl14')
+    class M(nn.Module):
+        def __init__(self):
+            super().__init__()
+            self.backbone = backbone
+            self.head = nn.Linear(1024, num_classes)
+            # Materialize parameter lists (avoid generator exhaustion).
+            self._head_params = list(self.head.parameters())
+            self._backbone_params = list(self.backbone.parameters())
+        def forward(self, x):
+            f = self.backbone(x)  # CLS token, [B, 1024]
+            return self.head(f)
+        def trainable_groups(self):
+            return [
+                {"params": self._head_params,     "lr": 1e-3, "linear_probe": True},
+                {"params": self._backbone_params, "lr": 1e-5, "linear_probe": False},
+            ]
+    return M(), 224, [0.485, 0.456, 0.406], [0.229, 0.224, 0.225]
+
+
+def build_swin_base(num_classes):
+    import timm
+    model = timm.create_model("swin_base_patch4_window7_224", pretrained=True, num_classes=num_classes)
+    head_params = list(model.head.parameters()) if hasattr(model, "head") else []
+    other_params = [p for n, p in model.named_parameters() if not n.startswith("head")]
+    class M(nn.Module):
+        def __init__(self):
+            super().__init__(); self.m = model
+        def forward(self, x): return self.m(x)
+        def trainable_groups(self):
+            return [
+                {"params": head_params,  "lr": 1e-3, "linear_probe": True},
+                {"params": other_params, "lr": 1e-5, "linear_probe": False},
+            ]
+    return M(), 224, [0.485, 0.456, 0.406], [0.229, 0.224, 0.225]
+
+
+def build_retfound(num_classes, weights_path):
+    """RETFound ViT-Large/16, MAE-pretrained on fundus images.
+    Loads weights from a local checkpoint downloaded from rmaphoh/RETFound_MAE."""
+    import timm
+    # RETFound is a vanilla MAE ViT-L/16 with patch 16, image 224.
+    model = timm.create_model("vit_large_patch16_224", pretrained=False, num_classes=num_classes,
+                              drop_path_rate=0.2, global_pool="token")
+    if weights_path and os.path.exists(weights_path):
+        ckpt = torch.load(weights_path, map_location="cpu", weights_only=False)
+        state = ckpt.get("model", ckpt.get("state_dict", ckpt))
+        # RETFound checkpoints have 'pos_embed' etc; we drop classifier head keys
+        state = {k: v for k, v in state.items()
+                 if not k.startswith("head.") and not k.startswith("fc_norm.")}
+        missing, unexp = model.load_state_dict(state, strict=False)
+        print(f"  RETFound loaded: {len(state)} keys, missing={len(missing)}, unexpected={len(unexp)}")
+    else:
+        print(f"  WARNING: RETFound weights not found at {weights_path}; using random init for backbone (will perform poorly)")
+    head_params = list(model.head.parameters()) + list(model.fc_norm.parameters())
+    other_params = [p for n, p in model.named_parameters()
+                    if not n.startswith("head") and not n.startswith("fc_norm")]
+    class M(nn.Module):
+        def __init__(self): super().__init__(); self.m = model
+        def forward(self, x): return self.m(x)
+        def trainable_groups(self):
+            return [
+                {"params": head_params,  "lr": 1e-3, "linear_probe": True},
+                {"params": other_params, "lr": 1e-5, "linear_probe": False},
+            ]
+    return M(), 224, [0.485, 0.456, 0.406], [0.229, 0.224, 0.225]
+
+
+# ------------------------- train -------------------------
+
+def expected_calibration_error(probs, labels, n_bins=15):
+    conf = probs.max(1); pred = probs.argmax(1); correct = (pred == labels).astype(float)
+    bins = np.linspace(0, 1, n_bins+1); ece = 0.0
+    for i in range(n_bins):
+        m = (conf > bins[i]) & (conf <= bins[i+1])
+        if m.sum(): ece += m.mean() * abs(correct[m].mean() - conf[m].mean())
+    return float(ece)
+
+
+def bootstrap_ci(labels, preds, metric_fn, n=1000, seed=42):
+    rng = np.random.default_rng(seed); N = len(labels); vals = []
+    for _ in range(n):
+        idx = rng.integers(0, N, N)
+        try: vals.append(metric_fn(labels[idx], preds[idx]))
+        except Exception: pass
+    vals = np.array(vals)
+    return float(np.percentile(vals, 2.5)), float(np.percentile(vals, 97.5))
+
+
+@torch.no_grad()
+def tta_predict(model, x, device):
+    model.eval(); B, C, H, W = x.shape
+    crop = int(H * 0.9); out = None; n = 0
+    views = [x, torch.flip(x, dims=[3])]
+    for (y, xc) in [(0, 0), (0, W-crop), (H-crop, 0), (H-crop, W-crop)]:
+        c = x[:, :, y:y+crop, xc:xc+crop]
+        c = F.interpolate(c, size=(H, W), mode="bilinear", align_corners=False)
+        views.append(c)
+    for v in views:
+        p = F.softmax(model(v.to(device)), dim=1)
+        out = p if out is None else out + p; n += 1
+    return (out/n).cpu().numpy()
+
+
+@torch.no_grad()
+def evaluate(model, loader, device, num_classes, use_tta=False):
+    model.eval(); ps, ls = [], []
+    for x, y in loader:
+        if use_tta: p = tta_predict(model, x, device)
+        else:
+            p = F.softmax(model(x.to(device)), dim=1).cpu().numpy()
+        ps.append(p); ls.append(y.numpy())
+    probs = np.concatenate(ps); labels = np.concatenate(ls); preds = probs.argmax(1)
+    acc = accuracy_score(labels, preds)
+    p, r, f1, _ = precision_recall_fscore_support(labels, preds, average="macro", zero_division=0)
+    try: roc = roc_auc_score(labels, probs, multi_class="ovr", average="macro", labels=list(range(num_classes)))
+    except Exception: roc = float("nan")
+    try: pr_auc = average_precision_score(F.one_hot(torch.tensor(labels), num_classes).numpy(), probs, average="macro")
+    except Exception: pr_auc = float("nan")
+    return {"acc": acc, "precision": p, "recall": r, "f1": f1,
+            "roc_auc": roc, "pr_auc": pr_auc, "ece": expected_calibration_error(probs, labels),
+            "labels": labels.tolist(), "preds": preds.tolist(), "probs": probs.tolist()}
+
+
+def mixup(x, y, alpha, nc):
+    lam = np.random.beta(alpha, alpha)
+    i = torch.randperm(x.size(0), device=x.device)
+    x = lam*x + (1-lam)*x[i]
+    yoh = F.one_hot(y, nc).float()
+    return x, lam*yoh + (1-lam)*yoh[i]
+
+
+def train_foundation(name, build_fn, samples_tr, samples_va, num_classes, device,
+                     batch_size, workers, lp_epochs, ft_epochs, patience, label):
+    model, image_size, mean, std = build_fn()
+    model = model.to(device)
+
+    tf_tr = make_transforms(image_size, train=True,  mean=mean, std=std)
+    tf_va = make_transforms(image_size, train=False, mean=mean, std=std)
+
+    ds_tr = ImageListDataset(samples_tr, tf_tr); ds_va = ImageListDataset(samples_va, tf_va)
+    labels_arr = np.array([s[1] for s in samples_tr])
+    cw = 1.0 / np.maximum(np.bincount(labels_arr, minlength=num_classes), 1)
+    sw = cw[labels_arr]
+    sampler = WeightedRandomSampler(sw.tolist(), num_samples=len(sw), replacement=True)
+    dl_tr = DataLoader(ds_tr, batch_size=batch_size, sampler=sampler, num_workers=workers, pin_memory=True, drop_last=True)
+    dl_va = DataLoader(ds_va, batch_size=batch_size, shuffle=False, num_workers=workers, pin_memory=True)
+
+    groups = model.trainable_groups()
+    head_group = next(g for g in groups if g.get("linear_probe"))
+    backbone_group = next(g for g in groups if not g.get("linear_probe"))
+
+    scaler = GradScaler()
+    best_f1 = -1; best_state = None; bad = 0; history = []
+
+    # ---- Stage 1: linear probe (freeze backbone) ----
+    for p in backbone_group["params"]: p.requires_grad = False
+    opt = torch.optim.AdamW([{"params": head_group["params"], "lr": head_group["lr"]}], weight_decay=1e-4)
+    sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=lp_epochs)
+    for ep in range(lp_epochs):
+        model.train(); t0 = time.time(); loss_sum, n = 0.0, 0
+        for x, y in dl_tr:
+            x = x.to(device, non_blocking=True); y = y.to(device, non_blocking=True)
+            if np.random.rand() < 0.3:
+                x, ysoft = mixup(x, y, 0.2, num_classes); soft = True
+            else: ysoft = y; soft = False
+            opt.zero_grad(set_to_none=True)
+            with autocast():
+                out = model(x)
+                loss = -(ysoft * F.log_softmax(out, 1)).sum(1).mean() if soft else F.cross_entropy(out, ysoft)
+            scaler.scale(loss).backward(); scaler.step(opt); scaler.update()
+            loss_sum += loss.item()*x.size(0); n += x.size(0)
+        sched.step()
+        v = evaluate(model, dl_va, device, num_classes)
+        history.append({"phase": "lp", "epoch": ep, "loss": loss_sum/n, "val_acc": v["acc"], "val_f1": v["f1"]})
+        print(f"[{label} LP] ep {ep+1}/{lp_epochs}  loss {loss_sum/n:.4f}  val_acc {v['acc']*100:5.2f}  val_f1 {v['f1']*100:5.2f}  ({time.time()-t0:.0f}s)", flush=True)
+        if v["f1"] > best_f1 + 1e-4:
+            best_f1 = v["f1"]; best_state = {k: vv.detach().cpu().clone() for k, vv in model.state_dict().items()}; bad = 0
+        else:
+            bad += 1
+            if bad >= patience: print(f"[{label} LP] early stop"); break
+
+    # ---- Stage 2: full fine-tune (unfreeze backbone, low LR) ----
+    if best_state is not None: model.load_state_dict(best_state)
+    for p in backbone_group["params"]: p.requires_grad = True
+    opt = torch.optim.AdamW([
+        {"params": head_group["params"],     "lr": 1e-4},
+        {"params": backbone_group["params"], "lr": 1e-5},
+    ], weight_decay=1e-4)
+    sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=ft_epochs)
+    bad = 0
+    for ep in range(ft_epochs):
+        model.train(); t0 = time.time(); loss_sum, n = 0.0, 0
+        for x, y in dl_tr:
+            x = x.to(device, non_blocking=True); y = y.to(device, non_blocking=True)
+            if np.random.rand() < 0.3:
+                x, ysoft = mixup(x, y, 0.2, num_classes); soft = True
+            else: ysoft = y; soft = False
+            opt.zero_grad(set_to_none=True)
+            with autocast():
+                out = model(x)
+                loss = -(ysoft * F.log_softmax(out, 1)).sum(1).mean() if soft else F.cross_entropy(out, ysoft)
+            scaler.scale(loss).backward(); scaler.step(opt); scaler.update()
+            loss_sum += loss.item()*x.size(0); n += x.size(0)
+        sched.step()
+        v = evaluate(model, dl_va, device, num_classes)
+        history.append({"phase": "ft", "epoch": ep, "loss": loss_sum/n, "val_acc": v["acc"], "val_f1": v["f1"]})
+        print(f"[{label} FT] ep {ep+1}/{ft_epochs}  loss {loss_sum/n:.4f}  val_acc {v['acc']*100:5.2f}  val_f1 {v['f1']*100:5.2f}  ({time.time()-t0:.0f}s)", flush=True)
+        if v["f1"] > best_f1 + 1e-4:
+            best_f1 = v["f1"]; best_state = {k: vv.detach().cpu().clone() for k, vv in model.state_dict().items()}; bad = 0
+        else:
+            bad += 1
+            if bad >= patience: print(f"[{label} FT] early stop"); break
+
+    if best_state is not None: model.load_state_dict(best_state)
+    return model, history, best_f1, image_size, mean, std
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--manifest", required=True)
+    ap.add_argument("--out-dir", required=True)
+    ap.add_argument("--weights-dir", required=True)
+    ap.add_argument("--retfound-weights", default="weights_retfound.pth")
+    ap.add_argument("--models", nargs="+", default=["dinov2_l", "swin_b", "retfound"])
+    ap.add_argument("--batch-size", type=int, default=24)
+    ap.add_argument("--workers", type=int, default=4)
+    ap.add_argument("--lp-epochs", type=int, default=20)
+    ap.add_argument("--ft-epochs", type=int, default=15)
+    ap.add_argument("--patience", type=int, default=8)
+    ap.add_argument("--seed", type=int, default=42)
+    args = ap.parse_args()
+
+    torch.manual_seed(args.seed); np.random.seed(args.seed)
+    device = torch.device("cuda" if torch.cuda.is_available() else "cpu"); print(f"device: {device}")
+    out_dir = Path(args.out_dir); out_dir.mkdir(parents=True, exist_ok=True)
+    w_dir = Path(args.weights_dir); w_dir.mkdir(parents=True, exist_ok=True)
+
+    M = json.load(open(args.manifest))
+    num_classes = len(M["classes"])
+    samples_tr = [tuple(x) for x in M["splits"]["train"]]
+    samples_va = [tuple(x) for x in M["splits"]["val"]]
+    samples_te = [tuple(x) for x in M["splits"]["test"]]
+    print(f"train {len(samples_tr)} | val {len(samples_va)} | test {len(samples_te)} | {num_classes} classes")
+
+    builders = {
+        "dinov2_l": lambda: build_dinov2_large(num_classes),
+        "swin_b":   lambda: build_swin_base(num_classes),
+        "retfound": lambda: build_retfound(num_classes, args.retfound_weights),
+    }
+
+    summary = {}
+    for name in args.models:
+        # Skip RETFound if weights file missing or empty (HF gated)
+        if name == "retfound":
+            wp = args.retfound_weights
+            if not (wp and os.path.exists(wp) and os.path.getsize(wp) > 1_000_000):
+                print(f"\n[retfound] SKIPPING — weights file '{wp}' missing or empty (HF gated). Use DINOv2/Swin instead.")
+                continue
+        print(f"\n======== {name} ========")
+        try:
+            model, hist, best_f1, image_size, mean, std = train_foundation(
+                name, builders[name], samples_tr + samples_va, samples_va,
+                num_classes, device, args.batch_size, args.workers,
+                args.lp_epochs, args.ft_epochs, args.patience, label=name)
+        except Exception as e:
+            print(f"[{name}] FAILED: {e}"); continue
+
+        tf_te = make_transforms(image_size, train=False, mean=mean, std=std)
+        dl_te = DataLoader(ImageListDataset(samples_te, tf_te), batch_size=args.batch_size,
+                           shuffle=False, num_workers=args.workers, pin_memory=True)
+        print(f"[{name}] evaluating on test with TTA ...")
+        res = evaluate(model, dl_te, device, num_classes, use_tta=True)
+        labels = np.array(res["labels"]); preds = np.array(res["preds"])
+        acc_lo, acc_hi = bootstrap_ci(labels, preds, accuracy_score)
+        f1_lo, f1_hi = bootstrap_ci(labels, preds,
+            lambda l, p: precision_recall_fscore_support(l, p, average="macro", zero_division=0)[2])
+        summary[name] = {
+            "test_acc": res["acc"], "test_acc_ci": [acc_lo, acc_hi],
+            "test_f1": res["f1"], "test_f1_ci": [f1_lo, f1_hi],
+            "test_precision": res["precision"], "test_recall": res["recall"],
+            "roc_auc": res["roc_auc"], "pr_auc": res["pr_auc"], "ece": res["ece"],
+        }
+        with open(out_dir / f"{name}_test.json", "w") as f: json.dump(summary[name], f, indent=2)
+        with open(out_dir / f"{name}_test_preds.json", "w") as f:
+            json.dump({"labels": res["labels"], "preds": res["preds"], "probs": res["probs"]}, f)
+        torch.save(model.state_dict(), w_dir / f"{name}_v2.pth")
+        print(f"[{name}] test acc {res['acc']*100:.2f} [{acc_lo*100:.1f},{acc_hi*100:.1f}]  f1 {res['f1']*100:.2f}  roc {res['roc_auc']:.4f}")
+        del model; torch.cuda.empty_cache()
+
+    with open(out_dir / "summary_foundation.json", "w") as f: json.dump(summary, f, indent=2)
+    print("\nDone (Phase 2).")
+
+
+if __name__ == "__main__":
+    main()

code/run_v2_experiments.py

@@ -0,0 +1,438 @@
+"""
+Phase 1 — v2 training pipeline for fundus classification.
+
+Differences from v1 (run_final_experiments.py):
+  * Reads holdout_split_augmented.json (group-aware split over the full
+    Original+Augmented union; no filename-level leakage).
+  * Adds CLAHE preprocessing (luminance channel) before all transforms.
+  * Adds RandAugment(n=2, m=9) on the training transforms.
+  * Adds WeightedRandomSampler (inverse class frequency).
+  * Adds MixUp/CutMix (α=0.2, alternating per batch with p=0.5).
+  * 100 epochs, EarlyStop patience 12, warmup (3 ep) + cosine.
+  * 6-view TTA at inference (original + hflip + 4 corner crops).
+"""
+
+import argparse, json, math, os, random, time
+from pathlib import Path
+from collections import defaultdict
+
+import numpy as np
+import torch, torch.nn as nn, torch.nn.functional as F
+from torch.utils.data import Dataset, DataLoader, WeightedRandomSampler
+from torch.cuda.amp import autocast, GradScaler
+from torchvision import transforms, models
+import cv2
+from PIL import Image
+from sklearn.metrics import (
+    accuracy_score, precision_recall_fscore_support,
+    roc_auc_score, average_precision_score,
+)
+from scipy.stats import binom
+from tqdm import tqdm
+
+
+# ---------------------------- repro ----------------------------
+
+def set_seed(s):
+    random.seed(s); np.random.seed(s); torch.manual_seed(s); torch.cuda.manual_seed_all(s)
+
+
+# ------------------------- CLAHE preprocessing -------------------------
+
+class CLAHEPreprocess:
+    """Apply CLAHE on the L channel of LAB color space. PIL in, PIL out."""
+    def __init__(self, clip_limit=2.0, tile=(8, 8)):
+        self.clahe = cv2.createCLAHE(clipLimit=clip_limit, tileGridSize=tile)
+    def __call__(self, img):
+        arr = np.array(img.convert("RGB"))
+        lab = cv2.cvtColor(arr, cv2.COLOR_RGB2LAB)
+        lab[..., 0] = self.clahe.apply(lab[..., 0])
+        rgb = cv2.cvtColor(lab, cv2.COLOR_LAB2RGB)
+        return Image.fromarray(rgb)
+
+
+# ------------------------- dataset -------------------------
+
+class ImageListDataset(Dataset):
+    def __init__(self, samples, transform):
+        self.samples = samples
+        self.transform = transform
+    def __len__(self):
+        return len(self.samples)
+    def __getitem__(self, idx):
+        p, lbl = self.samples[idx]
+        img = Image.open(p).convert("RGB")
+        return self.transform(img), int(lbl)
+
+
+# ------------------------- transforms -------------------------
+
+IMAGENET_MEAN = [0.485, 0.456, 0.406]; IMAGENET_STD = [0.229, 0.224, 0.225]
+CLIP_MEAN = [0.4815, 0.4578, 0.4082]; CLIP_STD = [0.2686, 0.2613, 0.2758]
+
+def build_transforms(image_size, use_clip_norm=False, train=True, use_clahe=True):
+    mean = CLIP_MEAN if use_clip_norm else IMAGENET_MEAN
+    std = CLIP_STD if use_clip_norm else IMAGENET_STD
+    pre = [CLAHEPreprocess()] if use_clahe else []
+    if train:
+        return transforms.Compose(pre + [
+            transforms.Resize((image_size + 32, image_size + 32)),
+            transforms.RandomResizedCrop(image_size, scale=(0.75, 1.0)),
+            transforms.RandomHorizontalFlip(),
+            transforms.RandomVerticalFlip(p=0.2),
+            transforms.RandomRotation(20),
+            transforms.RandAugment(num_ops=2, magnitude=9),
+            transforms.ColorJitter(brightness=0.15, contrast=0.15, saturation=0.1),
+            transforms.ToTensor(),
+            transforms.Normalize(mean, std),
+            transforms.RandomErasing(p=0.25, scale=(0.02, 0.15)),
+        ])
+    return transforms.Compose(pre + [
+        transforms.Resize((image_size, image_size)),
+        transforms.ToTensor(),
+        transforms.Normalize(mean, std),
+    ])
+
+
+# ------------------------- models -------------------------
+
+def build_model(name, num_classes):
+    name = name.lower()
+    if name == "vgg19":
+        m = models.vgg19(weights=models.VGG19_Weights.IMAGENET1K_V1)
+        m.classifier[6] = nn.Linear(m.classifier[6].in_features, num_classes)
+        return m, 224, False
+    if name == "resnet50":
+        m = models.resnet50(weights=models.ResNet50_Weights.IMAGENET1K_V2)
+        m.fc = nn.Linear(m.fc.in_features, num_classes); return m, 224, False
+    if name == "resnet101":
+        m = models.resnet101(weights=models.ResNet101_Weights.IMAGENET1K_V2)
+        m.fc = nn.Linear(m.fc.in_features, num_classes); return m, 224, False
+    if name == "densenet121":
+        m = models.densenet121(weights=models.DenseNet121_Weights.IMAGENET1K_V1)
+        m.classifier = nn.Linear(m.classifier.in_features, num_classes); return m, 224, False
+    if name == "inception_v3":
+        m = models.inception_v3(weights=models.Inception_V3_Weights.IMAGENET1K_V1, aux_logits=True)
+        m.fc = nn.Linear(m.fc.in_features, num_classes)
+        m.AuxLogits.fc = nn.Linear(m.AuxLogits.fc.in_features, num_classes)
+        return m, 299, False
+    if name == "clip_openai":
+        import open_clip
+        model, _, _ = open_clip.create_model_and_transforms("ViT-B-16", pretrained="openai")
+        class CLIPClf(nn.Module):
+            def __init__(self, backbone, nc):
+                super().__init__(); self.backbone = backbone.visual
+                d = self.backbone.output_dim if hasattr(self.backbone, "output_dim") else 512
+                self.head = nn.Linear(d, nc)
+            def forward(self, x):
+                f = self.backbone(x); return self.head(f)
+        return CLIPClf(model, num_classes), 224, True
+    raise ValueError(name)
+
+
+# ------------------------- MixUp / CutMix -------------------------
+
+def mixup(x, y, alpha=0.2, num_classes=10):
+    lam = np.random.beta(alpha, alpha) if alpha > 0 else 1.0
+    idx = torch.randperm(x.size(0), device=x.device)
+    x = lam * x + (1 - lam) * x[idx]
+    y_oh = F.one_hot(y, num_classes).float()
+    y_mix = lam * y_oh + (1 - lam) * y_oh[idx]
+    return x, y_mix
+
+def cutmix(x, y, alpha=1.0, num_classes=10):
+    lam = np.random.beta(alpha, alpha) if alpha > 0 else 1.0
+    idx = torch.randperm(x.size(0), device=x.device)
+    H, W = x.size(2), x.size(3)
+    cut_rat = math.sqrt(1.0 - lam)
+    cw, ch = int(W * cut_rat), int(H * cut_rat)
+    cx, cy = np.random.randint(W), np.random.randint(H)
+    x1 = np.clip(cx - cw // 2, 0, W); x2 = np.clip(cx + cw // 2, 0, W)
+    y1 = np.clip(cy - ch // 2, 0, H); y2 = np.clip(cy + ch // 2, 0, H)
+    x[:, :, y1:y2, x1:x2] = x[idx, :, y1:y2, x1:x2]
+    lam = 1 - ((x2 - x1) * (y2 - y1) / (W * H))
+    y_oh = F.one_hot(y, num_classes).float()
+    y_mix = lam * y_oh + (1 - lam) * y_oh[idx]
+    return x, y_mix
+
+
+# ------------------------- metrics -------------------------
+
+def expected_calibration_error(probs, labels, n_bins=15):
+    conf = probs.max(axis=1); pred = probs.argmax(axis=1); correct = (pred == labels).astype(float)
+    bins = np.linspace(0, 1, n_bins + 1); ece = 0.0
+    for i in range(n_bins):
+        mask = (conf > bins[i]) & (conf <= bins[i+1])
+        if mask.sum() > 0:
+            ece += (mask.mean()) * abs(correct[mask].mean() - conf[mask].mean())
+    return float(ece)
+
+def bootstrap_ci(labels, preds, metric_fn, n=1000, seed=42):
+    rng = np.random.default_rng(seed); N = len(labels); vals = []
+    for _ in range(n):
+        idx = rng.integers(0, N, N)
+        try: vals.append(metric_fn(labels[idx], preds[idx]))
+        except Exception: pass
+    vals = np.array(vals)
+    return float(vals.mean()), float(np.percentile(vals, 2.5)), float(np.percentile(vals, 97.5))
+
+
+# ------------------------- TTA inference -------------------------
+
+@torch.no_grad()
+def tta_predict(model, images, device):
+    """6 views: original + hflip + 4 corner crops of 90% size resized back."""
+    model.eval()
+    out_probs = None; n_views = 0
+    B, C, H, W = images.shape
+    crop = int(H * 0.9)
+    views = [images, torch.flip(images, dims=[3])]
+    for (y, x) in [(0, 0), (0, W - crop), (H - crop, 0), (H - crop, W - crop)]:
+        c = images[:, :, y:y+crop, x:x+crop]
+        c = F.interpolate(c, size=(H, W), mode="bilinear", align_corners=False)
+        views.append(c)
+    for v in views:
+        p = F.softmax(model(v.to(device)), dim=1)
+        out_probs = p if out_probs is None else out_probs + p
+        n_views += 1
+    return (out_probs / n_views).cpu().numpy()
+
+
+@torch.no_grad()
+def evaluate(model, loader, device, num_classes, use_tta=False):
+    model.eval(); all_probs, all_labels = [], []
+    for x, y in loader:
+        if use_tta:
+            p = tta_predict(model, x, device)
+        else:
+            x = x.to(device); out = model(x)
+            if isinstance(out, tuple): out = out[0]
+            p = F.softmax(out, dim=1).cpu().numpy()
+        all_probs.append(p); all_labels.append(y.numpy())
+    probs = np.concatenate(all_probs); labels = np.concatenate(all_labels)
+    preds = probs.argmax(axis=1)
+    acc = accuracy_score(labels, preds)
+    p, r, f1, _ = precision_recall_fscore_support(labels, preds, average="macro", zero_division=0)
+    try:
+        roc = roc_auc_score(labels, probs, multi_class="ovr", average="macro",
+                            labels=list(range(num_classes)))
+    except Exception:
+        roc = float("nan")
+    try:
+        pr_auc = average_precision_score(
+            F.one_hot(torch.tensor(labels), num_classes).numpy(), probs, average="macro")
+    except Exception:
+        pr_auc = float("nan")
+    ece = expected_calibration_error(probs, labels)
+    return {
+        "acc": acc, "precision": p, "recall": r, "f1": f1,
+        "roc_auc": roc, "pr_auc": pr_auc, "ece": ece,
+        "labels": labels.tolist(), "preds": preds.tolist(), "probs": probs.tolist(),
+    }
+
+
+# ------------------------- train one model -------------------------
+
+def train_model(name, samples_train, samples_val, num_classes, device,
+                epochs, batch_size, workers, patience, label, use_clahe):
+    model, image_size, use_clip = build_model(name, num_classes)
+    model = model.to(device)
+    tf_train = build_transforms(image_size, use_clip_norm=use_clip, train=True,  use_clahe=use_clahe)
+    tf_val   = build_transforms(image_size, use_clip_norm=use_clip, train=False, use_clahe=use_clahe)
+    ds_train = ImageListDataset(samples_train, tf_train)
+    ds_val   = ImageListDataset(samples_val,   tf_val)
+
+    # Weighted sampler
+    labels_arr = np.array([s[1] for s in samples_train])
+    class_counts = np.bincount(labels_arr, minlength=num_classes)
+    class_weights = 1.0 / np.maximum(class_counts, 1)
+    sample_weights = class_weights[labels_arr]
+    sampler = WeightedRandomSampler(sample_weights.tolist(), num_samples=len(sample_weights), replacement=True)
+
+    dl_train = DataLoader(ds_train, batch_size=batch_size, sampler=sampler,
+                          num_workers=workers, pin_memory=True, drop_last=True)
+    dl_val   = DataLoader(ds_val, batch_size=batch_size, shuffle=False,
+                          num_workers=workers, pin_memory=True)
+
+    opt = torch.optim.AdamW(model.parameters(), lr=2e-4, weight_decay=1e-4)
+    warmup_epochs = 3
+    def lr_lambda(epoch):
+        if epoch < warmup_epochs: return (epoch + 1) / warmup_epochs
+        prog = (epoch - warmup_epochs) / max(1, epochs - warmup_epochs)
+        return 0.5 * (1 + math.cos(math.pi * prog))
+    sched = torch.optim.lr_scheduler.LambdaLR(opt, lr_lambda)
+    scaler = GradScaler()
+
+    best_f1 = -1; best_state = None; bad = 0
+    history = []
+    for ep in range(epochs):
+        model.train()
+        t0 = time.time(); n = 0; loss_sum = 0.0
+        for x, y in dl_train:
+            x = x.to(device, non_blocking=True); y = y.to(device, non_blocking=True)
+            r = np.random.rand()
+            if r < 0.4:
+                x_m, y_soft = mixup(x, y, alpha=0.2, num_classes=num_classes); use_soft = True
+            elif r < 0.7:
+                x_m, y_soft = cutmix(x, y, alpha=1.0, num_classes=num_classes); use_soft = True
+            else:
+                x_m, y_soft = x, y; use_soft = False
+            opt.zero_grad(set_to_none=True)
+            with autocast():
+                out = model(x_m)
+                if isinstance(out, tuple):
+                    main_out, aux_out = out
+                    if use_soft:
+                        loss = -(y_soft * F.log_softmax(main_out, 1)).sum(1).mean()
+                        loss += 0.4 * (-(y_soft * F.log_softmax(aux_out, 1)).sum(1).mean())
+                    else:
+                        loss = F.cross_entropy(main_out, y_soft) + 0.4 * F.cross_entropy(aux_out, y_soft)
+                else:
+                    if use_soft:
+                        loss = -(y_soft * F.log_softmax(out, 1)).sum(1).mean()
+                    else:
+                        loss = F.cross_entropy(out, y_soft)
+            scaler.scale(loss).backward(); scaler.step(opt); scaler.update()
+            loss_sum += loss.item() * x.size(0); n += x.size(0)
+        sched.step()
+        val = evaluate(model, dl_val, device, num_classes, use_tta=False)
+        dt = time.time() - t0
+        history.append({"epoch": ep, "loss": loss_sum/n, "val_acc": val["acc"], "val_f1": val["f1"], "dt": dt})
+        print(f"[{label}] ep {ep+1:3d}/{epochs}  loss {loss_sum/n:.4f}  val_acc {val['acc']*100:5.2f}  val_f1 {val['f1']*100:5.2f}  ({dt:.0f}s)", flush=True)
+        if val["f1"] > best_f1 + 1e-4:
+            best_f1 = val["f1"]; best_state = {k: v.detach().cpu().clone() for k, v in model.state_dict().items()}; bad = 0
+        else:
+            bad += 1
+            if bad >= patience:
+                print(f"[{label}] early stop at epoch {ep+1}", flush=True); break
+    if best_state is not None:
+        model.load_state_dict(best_state)
+    return model, history, best_f1
+
+
+# ------------------------- main -------------------------
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--manifest", required=True)
+    ap.add_argument("--out-dir", required=True)
+    ap.add_argument("--weights-dir", required=True)
+    ap.add_argument("--models", nargs="+",
+                    default=["vgg19", "resnet50", "resnet101", "densenet121", "inception_v3", "clip_openai"])
+    ap.add_argument("--epochs", type=int, default=100)
+    ap.add_argument("--folds", type=int, default=5)
+    ap.add_argument("--batch-size", type=int, default=32)
+    ap.add_argument("--workers", type=int, default=4)
+    ap.add_argument("--patience", type=int, default=12)
+    ap.add_argument("--use-clahe", action="store_true", default=True)
+    ap.add_argument("--skip-cv", action="store_true",
+                    help="Only do final-train + indep test (skip k-fold CV)")
+    ap.add_argument("--seed", type=int, default=42)
+    args = ap.parse_args()
+
+    set_seed(args.seed)
+    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+    print(f"device: {device}")
+    out_dir = Path(args.out_dir); out_dir.mkdir(parents=True, exist_ok=True)
+    w_dir = Path(args.weights_dir); w_dir.mkdir(parents=True, exist_ok=True)
+
+    M = json.load(open(args.manifest))
+    classes = M["classes"]; num_classes = len(classes)
+    print(f"classes ({num_classes}): {classes}")
+    samples_train = [tuple(x) for x in M["splits"]["train"]]
+    samples_val   = [tuple(x) for x in M["splits"]["val"]]
+    samples_test  = [tuple(x) for x in M["splits"]["test"]]
+    print(f"train {len(samples_train)} | val {len(samples_val)} | test {len(samples_test)}")
+
+    pool_paths = M["pool_paths"]; pool_labels = M["pool_labels"]
+    folds = M["folds"]
+
+    summary = {}
+    test_preds_all = {}
+    for name in args.models:
+        print(f"\n========================  {name}  ========================")
+        per_fold = []
+        if not args.skip_cv:
+            cv_epochs = max(20, args.epochs // 2)  # CV uses half-budget; final uses full
+            for fi, fold in enumerate(folds[:args.folds]):
+                tr = [(pool_paths[i], pool_labels[i]) for i in fold["train_idx"]]
+                va = [(pool_paths[i], pool_labels[i]) for i in fold["val_idx"]]
+                print(f"\n--- fold {fi+1}/{args.folds}  train {len(tr)} val {len(va)} ---")
+                fmodel, hist, best_f1 = train_model(
+                    name, tr, va, num_classes, device,
+                    cv_epochs, args.batch_size, args.workers, args.patience,
+                    label=f"{name}-f{fi+1}", use_clahe=args.use_clahe)
+                # Eval (no TTA) for fold metrics
+                _, image_size_f, use_clip_f = build_model(name, num_classes)
+                tf_vf = build_transforms(image_size_f, use_clip_norm=use_clip_f, train=False, use_clahe=args.use_clahe)
+                dl_vf = DataLoader(ImageListDataset(va, tf_vf),
+                                   batch_size=args.batch_size, shuffle=False,
+                                   num_workers=args.workers, pin_memory=True)
+                fres = evaluate(fmodel, dl_vf, device, num_classes, use_tta=False)
+                per_fold.append({
+                    "fold": fi, "best_val_f1": best_f1,
+                    "val_acc": fres["acc"], "val_f1": fres["f1"],
+                    "val_roc_auc": fres["roc_auc"], "val_ece": fres["ece"],
+                    "history": hist,
+                })
+                del fmodel; torch.cuda.empty_cache()
+
+        # Final train: combine train+val for stronger final model, evaluate on test
+        print(f"\n--- {name} FINAL train on train+val ({len(samples_train)+len(samples_val)} samples) ---")
+        final_model, hist, _ = train_model(
+            name, samples_train + samples_val, samples_val, num_classes, device,
+            args.epochs, args.batch_size, args.workers, args.patience,
+            label=f"{name}-final", use_clahe=args.use_clahe)
+        # Test eval with TTA
+        _, image_size, use_clip = build_model(name, num_classes)
+        tf_test = build_transforms(image_size, use_clip_norm=use_clip, train=False, use_clahe=args.use_clahe)
+        dl_test = DataLoader(ImageListDataset(samples_test, tf_test),
+                             batch_size=args.batch_size, shuffle=False, num_workers=args.workers, pin_memory=True)
+        print(f"[{name}] evaluating on test with TTA ...")
+        test_res = evaluate(final_model, dl_test, device, num_classes, use_tta=True)
+        labels = np.array(test_res["labels"]); preds = np.array(test_res["preds"])
+        acc_mean, acc_lo, acc_hi = bootstrap_ci(labels, preds, accuracy_score)
+        f1_mean, f1_lo, f1_hi = bootstrap_ci(labels, preds,
+            lambda l, p: precision_recall_fscore_support(l, p, average="macro", zero_division=0)[2])
+        test_res["acc_ci"] = [acc_lo, acc_hi]; test_res["f1_ci"] = [f1_lo, f1_hi]
+        summary[name] = {
+            "test_acc": test_res["acc"], "test_acc_ci": test_res["acc_ci"],
+            "test_f1": test_res["f1"], "test_f1_ci": test_res["f1_ci"],
+            "test_precision": test_res["precision"], "test_recall": test_res["recall"],
+            "roc_auc": test_res["roc_auc"], "pr_auc": test_res["pr_auc"], "ece": test_res["ece"],
+            "n_folds_run": len(per_fold),
+        }
+        with open(out_dir / f"{name}_test.json", "w") as f: json.dump(summary[name], f, indent=2)
+        with open(out_dir / f"{name}_test_preds.json", "w") as f:
+            json.dump({"labels": test_res["labels"], "preds": test_res["preds"], "probs": test_res["probs"]}, f)
+        if per_fold:
+            with open(out_dir / f"{name}_kfold.json", "w") as f: json.dump(per_fold, f, indent=2)
+        torch.save(final_model.state_dict(), w_dir / f"{name}_v2_final.pth")
+        test_preds_all[name] = test_res
+
+        print(f"[{name}] test acc {test_res['acc']*100:.2f}  f1 {test_res['f1']*100:.2f}  roc {test_res['roc_auc']:.4f}  ece {test_res['ece']:.4f}")
+
+    # McNemar
+    print("\n=== McNemar pairwise ===")
+    mcnemar = {}
+    keys = list(test_preds_all.keys())
+    labels = np.array(test_preds_all[keys[0]]["labels"])
+    for i in range(len(keys)):
+        for j in range(i+1, len(keys)):
+            p1 = np.array(test_preds_all[keys[i]]["preds"]); p2 = np.array(test_preds_all[keys[j]]["preds"])
+            c1 = p1 == labels; c2 = p2 == labels
+            b = int(((c1) & (~c2)).sum()); c = int(((~c1) & (c2)).sum())
+            n = b + c
+            if n == 0: pval = 1.0
+            else:
+                k = min(b, c); pval = float(2 * binom.cdf(k, n, 0.5))
+                if pval > 1: pval = 1.0
+            mcnemar[f"{keys[i]}_vs_{keys[j]}"] = {"b": b, "c": c, "p": pval}
+            print(f"  {keys[i]} vs {keys[j]}: b={b} c={c} p={pval:.4g}")
+    with open(out_dir / "mcnemar.json", "w") as f: json.dump(mcnemar, f, indent=2)
+    with open(out_dir / "summary.json", "w") as f: json.dump(summary, f, indent=2)
+    print("\nDone.")
+
+
+if __name__ == "__main__":
+    main()

results/clip_openai_test.json

@@ -0,0 +1,18 @@
+{
+  "test_acc": 0.8625311720698254,
+  "test_acc_ci": [
+    0.8509897132169576,
+    0.8740726309226933
+  ],
+  "test_f1": 0.8999338780904511,
+  "test_f1_ci": [
+    0.8908154444260842,
+    0.9088493548396316
+  ],
+  "test_precision": 0.8887659033860622,
+  "test_recall": 0.915128373891258,
+  "roc_auc": 0.9895801702791353,
+  "pr_auc": 0.9483871820462942,
+  "ece": 0.017864541510617046,
+  "n_folds_run": 0
+}

results/densenet121_test.json

@@ -0,0 +1,18 @@
+{
+  "test_acc": 0.8977556109725686,
+  "test_acc_ci": [
+    0.8871571072319202,
+    0.9080501870324189
+  ],
+  "test_f1": 0.9225920013528925,
+  "test_f1_ci": [
+    0.913377240794446,
+    0.9313115657332115
+  ],
+  "test_precision": 0.9139639568399703,
+  "test_recall": 0.9322069613114635,
+  "roc_auc": 0.993099023325742,
+  "pr_auc": 0.9648647645485235,
+  "ece": 0.023892512363834564,
+  "n_folds_run": 0
+}

results/dinov2_l_test.json

@@ -0,0 +1,17 @@
+{
+  "test_acc": 0.8949501246882793,
+  "test_acc_ci": [
+    0.8849672693266833,
+    0.9052369077306733
+  ],
+  "test_f1": 0.9214970232274702,
+  "test_f1_ci": [
+    0.9127686058957977,
+    0.9293729231634663
+  ],
+  "test_precision": 0.9150088741935475,
+  "test_recall": 0.9292672404244741,
+  "roc_auc": 0.9937714203718716,
+  "pr_auc": 0.9677433875951829,
+  "ece": 0.03374567211714764
+}

results/ensemble_report.json

@@ -0,0 +1,712 @@
+{
+  "per_model": {
+    "clip_openai": {
+      "acc": 0.8625311720698254,
+      "acc_ci": [
+        0.8509897132169576,
+        0.8740648379052369
+      ],
+      "precision": 0.8887659033860622,
+      "recall": 0.915128373891258,
+      "f1": 0.8999338780904511,
+      "roc_auc": 0.9895801702791353,
+      "ece": 0.017864545904787428,
+      "kappa": 0.8382180482966803,
+      "brier": 0.19501780460857385,
+      "per_class_f1": [
+        0.9203539823008849,
+        0.954829742876998,
+        0.9513888888888888,
+        0.7137614678899082,
+        0.8296943231441049,
+        0.8442503639010189,
+        0.8310010764262648,
+        1.0,
+        0.9883268482490273,
+        0.9657320872274143
+      ],
+      "per_class_support": [
+        108,
+        733,
+        137,
+        609,
+        548,
+        334,
+        440,
+        17,
+        127,
+        155
+      ]
+    },
+    "densenet121": {
+      "acc": 0.8977556109725686,
+      "acc_ci": [
+        0.8871493142144639,
+        0.9080423940149626
+      ],
+      "precision": 0.9139639568399703,
+      "recall": 0.9322069613114635,
+      "f1": 0.9225920013528925,
+      "roc_auc": 0.993099023325742,
+      "ece": 0.023892554243157935,
+      "kappa": 0.879440226769113,
+      "brier": 0.14828401684717682,
+      "per_class_f1": [
+        0.9140271493212669,
+        0.9715475364330326,
+        0.9513888888888888,
+        0.8158546655656482,
+        0.8616236162361623,
+        0.8833819241982507,
+        0.8662857142857143,
+        1.0,
+        0.996078431372549,
+        0.9657320872274143
+      ],
+      "per_class_support": [
+        108,
+        733,
+        137,
+        609,
+        548,
+        334,
+        440,
+        17,
+        127,
+        155
+      ]
+    },
+    "dinov2_l": {
+      "acc": 0.8949501246882793,
+      "acc_ci": [
+        0.8846633416458853,
+        0.9055486284289277
+      ],
+      "precision": 0.9150088741935475,
+      "recall": 0.9292672404244741,
+      "f1": 0.9214970232274702,
+      "roc_auc": 0.9937714203718716,
+      "ece": 0.03374566446218703,
+      "kappa": 0.8760062131313244,
+      "brier": 0.15526801776067914,
+      "per_class_f1": [
+        0.918918918918919,
+        0.9724896836313618,
+        0.9645390070921985,
+        0.8089700996677741,
+        0.8642413487133984,
+        0.8885630498533724,
+        0.8393285371702638,
+        1.0,
+        0.9921875,
+        0.9657320872274143
+      ],
+      "per_class_support": [
+        108,
+        733,
+        137,
+        609,
+        548,
+        334,
+        440,
+        17,
+        127,
+        155
+      ]
+    },
+    "inception_v3": {
+      "acc": 0.89214463840399,
+      "acc_ci": [
+        0.8815461346633416,
+        0.9027509351620947
+      ],
+      "precision": 0.9114333957779662,
+      "recall": 0.9291012889416669,
+      "f1": 0.9197113157379577,
+      "roc_auc": 0.9933846253549662,
+      "ece": 0.03020487911534726,
+      "kappa": 0.8727706715793122,
+      "brier": 0.15683023121332312,
+      "per_class_f1": [
+        0.918918918918919,
+        0.9689011748445059,
+        0.9513888888888888,
+        0.8045789043336059,
+        0.8534562211981567,
+        0.8875912408759125,
+        0.8504672897196262,
+        1.0,
+        0.996078431372549,
+        0.9657320872274143
+      ],
+      "per_class_support": [
+        108,
+        733,
+        137,
+        609,
+        548,
+        334,
+        440,
+        17,
+        127,
+        155
+      ]
+    },
+    "resnet101": {
+      "acc": 0.8924563591022444,
+      "acc_ci": [
+        0.881857855361596,
+        0.9033665835411472
+      ],
+      "precision": 0.9128874987068798,
+      "recall": 0.9290979896919567,
+      "f1": 0.9205495906178301,
+      "roc_auc": 0.9940599331335397,
+      "ece": 0.029744671987476563,
+      "kappa": 0.873135739806533,
+      "brier": 0.1492682365955807,
+      "per_class_f1": [
+        0.918918918918919,
+        0.9669421487603306,
+        0.9611307420494699,
+        0.8013411567476949,
+        0.8537906137184116,
+        0.8820960698689956,
+        0.859465737514518,
+        1.0,
+        0.996078431372549,
+        0.9657320872274143
+      ],
+      "per_class_support": [
+        108,
+        733,
+        137,
+        609,
+        548,
+        334,
+        440,
+        17,
+        127,
+        155
+      ]
+    },
+    "resnet50": {
+      "acc": 0.8908977556109726,
+      "acc_ci": [
+        0.8799797381546135,
+        0.9011845386533666
+      ],
+      "precision": 0.9110545481169569,
+      "recall": 0.9273515185427756,
+      "f1": 0.9186505198171678,
+      "roc_auc": 0.994354279320403,
+      "ece": 0.03958680766850639,
+      "kappa": 0.8712729134615307,
+      "brier": 0.14710658877842553,
+      "per_class_f1": [
+        0.918918918918919,
+        0.9674740484429065,
+        0.9477351916376306,
+        0.7986852917009039,
+        0.8574007220216606,
+        0.8807069219440353,
+        0.8537735849056604,
+        1.0,
+        0.996078431372549,
+        0.9657320872274143
+      ],
+      "per_class_support": [
+        108,
+        733,
+        137,
+        609,
+        548,
+        334,
+        440,
+        17,
+        127,
+        155
+      ]
+    },
+    "retfound": {
+      "acc": 0.8388403990024937,
+      "acc_ci": [
+        0.8263715710723192,
+        0.850997506234414
+      ],
+      "precision": 0.8639273665304865,
+      "recall": 0.8924390485821714,
+      "f1": 0.876842281045592,
+      "roc_auc": 0.9837617829682529,
+      "ece": 0.025993358808339723,
+      "kappa": 0.8102097500064082,
+      "brier": 0.23833885264650462,
+      "per_class_f1": [
+        0.8595744680851064,
+        0.9548924358084664,
+        0.9134948096885813,
+        0.6984667802385008,
+        0.7932960893854749,
+        0.8096676737160121,
+        0.7918015102481122,
+        1.0,
+        0.9844961240310077,
+        0.9627329192546584
+      ],
+      "per_class_support": [
+        108,
+        733,
+        137,
+        609,
+        548,
+        334,
+        440,
+        17,
+        127,
+        155
+      ]
+    },
+    "swin_b": {
+      "acc": 0.8684538653366584,
+      "acc_ci": [
+        0.8569201995012469,
+        0.8802992518703242
+      ],
+      "precision": 0.8957798154200936,
+      "recall": 0.914482667230412,
+      "f1": 0.9044410963586561,
+      "roc_auc": 0.9903580625502204,
+      "ece": 0.01902209332867754,
+      "kappa": 0.8447801854461359,
+      "brier": 0.18504628154638172,
+      "per_class_f1": [
+        0.9074889867841409,
+        0.9667590027700831,
+        0.9513888888888888,
+        0.7436527436527437,
+        0.8237410071942446,
+        0.8831562974203339,
+        0.810304449648712,
+        1.0,
+        0.9921875,
+        0.9657320872274143
+      ],
+      "per_class_support": [
+        108,
+        733,
+        137,
+        609,
+        548,
+        334,
+        440,
+        17,
+        127,
+        155
+      ]
+    },
+    "vgg19": {
+      "acc": 0.8930798004987531,
+      "acc_ci": [
+        0.8821695760598504,
+        0.9039900249376559
+      ],
+      "precision": 0.9110860665322441,
+      "recall": 0.9326584479061776,
+      "f1": 0.9211920362611569,
+      "roc_auc": 0.9929719540404655,
+      "ece": 0.020435141524917783,
+      "kappa": 0.8739754832960913,
+      "brier": 0.1543770380321469,
+      "per_class_f1": [
+        0.9292035398230089,
+        0.9707927677329624,
+        0.9513888888888888,
+        0.7966101694915254,
+        0.8484304932735426,
+        0.8816568047337278,
+        0.8720271800679502,
+        1.0,
+        0.996078431372549,
+        0.9657320872274143
+      ],
+      "per_class_support": [
+        108,
+        733,
+        137,
+        609,
+        548,
+        334,
+        440,
+        17,
+        127,
+        155
+      ]
+    }
+  },
+  "ensemble": {
+    "members": [
+      "clip_openai",
+      "densenet121",
+      "dinov2_l",
+      "inception_v3",
+      "resnet101",
+      "resnet50",
+      "retfound",
+      "swin_b",
+      "vgg19"
+    ],
+    "weights": {
+      "clip_openai": 0.8999338780904511,
+      "densenet121": 0.9225920013528925,
+      "dinov2_l": 0.9214970232274702,
+      "inception_v3": 0.9197113157379577,
+      "resnet101": 0.9205495906178301,
+      "resnet50": 0.9186505198171678,
+      "retfound": 0.876842281045592,
+      "swin_b": 0.9044410963586561,
+      "vgg19": 0.9211920362611569
+    },
+    "acc": 0.8968204488778054,
+    "acc_ci": [
+      0.8865336658354115,
+      0.9074189526184538
+    ],
+    "precision": 0.9133809044175024,
+    "recall": 0.9327946516638915,
+    "f1": 0.9225313471540415,
+    "roc_auc": 0.9941017286357303,
+    "ece": 0.01975079615286455,
+    "kappa": 0.8783721202739745,
+    "brier": 0.1438439679380553
+  },
+  "mcnemar_bonferroni": {
+    "n_pairs": 36,
+    "pairs": [
+      {
+        "model_a": "clip_openai",
+        "model_b": "densenet121",
+        "p": 1.5003245325684048e-11,
+        "b_count": 85,
+        "c_count": 198,
+        "p_bonferroni": 5.401168317246257e-10,
+        "sig_005": true
+      },
+      {
+        "model_a": "clip_openai",
+        "model_b": "dinov2_l",
+        "p": 5.035530682013286e-09,
+        "b_count": 106,
+        "c_count": 210,
+        "p_bonferroni": 1.8127910455247827e-07,
+        "sig_005": true
+      },
+      {
+        "model_a": "clip_openai",
+        "model_b": "inception_v3",
+        "p": 9.529243599048096e-08,
+        "b_count": 110,
+        "c_count": 205,
+        "p_bonferroni": 3.4305276956573147e-06,
+        "sig_005": true
+      },
+      {
+        "model_a": "clip_openai",
+        "model_b": "resnet101",
+        "p": 3.550882726150477e-08,
+        "b_count": 103,
+        "c_count": 199,
+        "p_bonferroni": 1.2783177814141717e-06,
+        "sig_005": true
+      },
+      {
+        "model_a": "clip_openai",
+        "model_b": "resnet50",
+        "p": 2.7655295866239894e-07,
+        "b_count": 110,
+        "c_count": 201,
+        "p_bonferroni": 9.955906511846362e-06,
+        "sig_005": true
+      },
+      {
+        "model_a": "clip_openai",
+        "model_b": "retfound",
+        "p": 9.51038696830371e-06,
+        "b_count": 183,
+        "c_count": 107,
+        "p_bonferroni": 0.00034237393085893354,
+        "sig_005": true
+      },
+      {
+        "model_a": "clip_openai",
+        "model_b": "swin_b",
+        "p": 0.27240218112115094,
+        "b_count": 125,
+        "c_count": 144,
+        "p_bonferroni": 1.0,
+        "sig_005": false
+      },
+      {
+        "model_a": "clip_openai",
+        "model_b": "vgg19",
+        "p": 3.6388149492186776e-09,
+        "b_count": 89,
+        "c_count": 187,
+        "p_bonferroni": 1.3099733817187238e-07,
+        "sig_005": true
+      },
+      {
+        "model_a": "densenet121",
+        "model_b": "dinov2_l",
+        "p": 0.5095015886349893,
+        "b_count": 78,
+        "c_count": 69,
+        "p_bonferroni": 1.0,
+        "sig_005": false
+      },
+      {
+        "model_a": "densenet121",
+        "model_b": "inception_v3",
+        "p": 0.12342708698379867,
+        "b_count": 70,
+        "c_count": 52,
+        "p_bonferroni": 1.0,
+        "sig_005": false
+      },
+      {
+        "model_a": "densenet121",
+        "model_b": "resnet101",
+        "p": 0.11799998150585847,
+        "b_count": 61,
+        "c_count": 44,
+        "p_bonferroni": 1.0,
+        "sig_005": false
+      },
+      {
+        "model_a": "densenet121",
+        "model_b": "resnet50",
+        "p": 0.04087197060775479,
+        "b_count": 64,
+        "c_count": 42,
+        "p_bonferroni": 1.0,
+        "sig_005": false
+      },
+      {
+        "model_a": "densenet121",
+        "model_b": "retfound",
+        "p": 6.225832846491392e-22,
+        "b_count": 293,
+        "c_count": 104,
+        "p_bonferroni": 2.2412998247369012e-20,
+        "sig_005": true
+      },
+      {
+        "model_a": "densenet121",
+        "model_b": "swin_b",
+        "p": 3.0961350194143013e-09,
+        "b_count": 173,
+        "c_count": 79,
+        "p_bonferroni": 1.1146086069891485e-07,
+        "sig_005": true
+      },
+      {
+        "model_a": "densenet121",
+        "model_b": "vgg19",
+        "p": 0.25133111748209397,
+        "b_count": 82,
+        "c_count": 67,
+        "p_bonferroni": 1.0,
+        "sig_005": false
+      },
+      {
+        "model_a": "dinov2_l",
+        "model_b": "inception_v3",
+        "p": 0.5259245063255041,
+        "b_count": 84,
+        "c_count": 75,
+        "p_bonferroni": 1.0,
+        "sig_005": false
+      },
+      {
+        "model_a": "dinov2_l",
+        "model_b": "resnet101",
+        "p": 0.5753154836552097,
+        "b_count": 82,
+        "c_count": 74,
+        "p_bonferroni": 1.0,
+        "sig_005": false
+      },
+      {
+        "model_a": "dinov2_l",
+        "model_b": "resnet50",
+        "p": 0.3412944918305007,
+        "b_count": 86,
+        "c_count": 73,
+        "p_bonferroni": 1.0,
+        "sig_005": false
+      },
+      {
+        "model_a": "dinov2_l",
+        "model_b": "retfound",
+        "p": 1.0373592271569002e-19,
+        "b_count": 291,
+        "c_count": 111,
+        "p_bonferroni": 3.73449321776484e-18,
+        "sig_005": true
+      },
+      {
+        "model_a": "dinov2_l",
+        "model_b": "swin_b",
+        "p": 4.0479796025489893e-07,
+        "b_count": 182,
+        "c_count": 97,
+        "p_bonferroni": 1.4572726569176362e-05,
+        "sig_005": true
+      },
+      {
+        "model_a": "dinov2_l",
+        "model_b": "vgg19",
+        "p": 0.7197120938004633,
+        "b_count": 100,
+        "c_count": 94,
+        "p_bonferroni": 1.0,
+        "sig_005": false
+      },
+      {
+        "model_a": "inception_v3",
+        "model_b": "resnet101",
+        "p": 1.0,
+        "b_count": 70,
+        "c_count": 71,
+        "p_bonferroni": 1.0,
+        "sig_005": false
+      },
+      {
+        "model_a": "inception_v3",
+        "model_b": "resnet50",
+        "p": 0.8078463096035289,
+        "b_count": 78,
+        "c_count": 74,
+        "p_bonferroni": 1.0,
+        "sig_005": false
+      },
+      {
+        "model_a": "inception_v3",
+        "model_b": "retfound",
+        "p": 1.420818214959577e-17,
+        "b_count": 290,
+        "c_count": 119,
+        "p_bonferroni": 5.114945573854478e-16,
+        "sig_005": true
+      },
+      {
+        "model_a": "inception_v3",
+        "model_b": "swin_b",
+        "p": 7.060589035894834e-06,
+        "b_count": 179,
+        "c_count": 103,
+        "p_bonferroni": 0.000254181205292214,
+        "sig_005": true
+      },
+      {
+        "model_a": "inception_v3",
+        "model_b": "vgg19",
+        "p": 0.8873030419814874,
+        "b_count": 98,
+        "c_count": 101,
+        "p_bonferroni": 1.0,
+        "sig_005": false
+      },
+      {
+        "model_a": "resnet101",
+        "model_b": "resnet50",
+        "p": 0.7117032769279406,
+        "b_count": 61,
+        "c_count": 56,
+        "p_bonferroni": 1.0,
+        "sig_005": false
+      },
+      {
+        "model_a": "resnet101",
+        "model_b": "retfound",
+        "p": 9.986206200690846e-18,
+        "b_count": 291,
+        "c_count": 119,
+        "p_bonferroni": 3.595034232248705e-16,
+        "sig_005": true
+      },
+      {
+        "model_a": "resnet101",
+        "model_b": "swin_b",
+        "p": 3.100110685161997e-06,
+        "b_count": 173,
+        "c_count": 96,
+        "p_bonferroni": 0.0001116039846658319,
+        "sig_005": true
+      },
+      {
+        "model_a": "resnet101",
+        "model_b": "vgg19",
+        "p": 0.9392502353738341,
+        "b_count": 85,
+        "c_count": 87,
+        "p_bonferroni": 1.0,
+        "sig_005": false
+      },
+      {
+        "model_a": "resnet50",
+        "model_b": "retfound",
+        "p": 7.657771276060926e-16,
+        "b_count": 301,
+        "c_count": 134,
+        "p_bonferroni": 2.7567976593819336e-14,
+        "sig_005": true
+      },
+      {
+        "model_a": "resnet50",
+        "model_b": "swin_b",
+        "p": 1.6139512859660104e-05,
+        "b_count": 173,
+        "c_count": 101,
+        "p_bonferroni": 0.0005810224629477637,
+        "sig_005": true
+      },
+      {
+        "model_a": "resnet50",
+        "model_b": "vgg19",
+        "p": 0.6642959893958242,
+        "b_count": 92,
+        "c_count": 99,
+        "p_bonferroni": 1.0,
+        "sig_005": false
+      },
+      {
+        "model_a": "retfound",
+        "model_b": "swin_b",
+        "p": 4.70320206782538e-08,
+        "b_count": 103,
+        "c_count": 198,
+        "p_bonferroni": 1.6931527444171367e-06,
+        "sig_005": true
+      },
+      {
+        "model_a": "retfound",
+        "model_b": "vgg19",
+        "p": 4.728007170016341e-20,
+        "b_count": 98,
+        "c_count": 272,
+        "p_bonferroni": 1.7020825812058828e-18,
+        "sig_005": true
+      },
+      {
+        "model_a": "swin_b",
+        "model_b": "vgg19",
+        "p": 1.5007108473870475e-07,
+        "b_count": 73,
+        "c_count": 152,
+        "p_bonferroni": 5.402559050593371e-06,
+        "sig_005": true
+      }
+    ]
+  },
+  "conformal": {}
+}

results/inception_v3_test.json

@@ -0,0 +1,18 @@
+{
+  "test_acc": 0.89214463840399,
+  "test_acc_ci": [
+    0.8815461346633416,
+    0.9024392144638403
+  ],
+  "test_f1": 0.9197113157379577,
+  "test_f1_ci": [
+    0.910484669507304,
+    0.9278346661886765
+  ],
+  "test_precision": 0.9114333957779662,
+  "test_recall": 0.9291012889416669,
+  "roc_auc": 0.9933846253549662,
+  "pr_auc": 0.9656573588323442,
+  "ece": 0.030204893970058433,
+  "n_folds_run": 0
+}

results/resnet101_test.json

@@ -0,0 +1,18 @@
+{
+  "test_acc": 0.8924563591022444,
+  "test_acc_ci": [
+    0.8824812967581047,
+    0.9033665835411472
+  ],
+  "test_f1": 0.9205495906178301,
+  "test_f1_ci": [
+    0.9117043375511852,
+    0.9287442691770825
+  ],
+  "test_precision": 0.9128874987068798,
+  "test_recall": 0.9290979896919567,
+  "roc_auc": 0.9940599331335397,
+  "pr_auc": 0.968684515330924,
+  "ece": 0.029744694636498346,
+  "n_folds_run": 0
+}

results/resnet50_test.json

@@ -0,0 +1,18 @@
+{
+  "test_acc": 0.8908977556109726,
+  "test_acc_ci": [
+    0.8806031795511222,
+    0.9011923316708229
+  ],
+  "test_f1": 0.9186505198171678,
+  "test_f1_ci": [
+    0.9093466266796268,
+    0.9268383091354964
+  ],
+  "test_precision": 0.9110545481169569,
+  "test_recall": 0.9273515185427756,
+  "roc_auc": 0.994354279320403,
+  "pr_auc": 0.969585114477252,
+  "ece": 0.03958680512304616,
+  "n_folds_run": 0
+}

results/retfound_test.json

@@ -0,0 +1,17 @@
+{
+  "test_acc": 0.8388403990024937,
+  "test_acc_ci": [
+    0.8273067331670823,
+    0.8513092269326683
+  ],
+  "test_f1": 0.876842281045592,
+  "test_f1_ci": [
+    0.866621099788389,
+    0.8870731668568106
+  ],
+  "test_precision": 0.8639273665304865,
+  "test_recall": 0.8924390485821714,
+  "roc_auc": 0.9837617829682529,
+  "pr_auc": 0.9171300421652508,
+  "ece": 0.025993344891918596
+}

results/swin_b_test.json

@@ -0,0 +1,17 @@
+{
+  "test_acc": 0.8684538653366584,
+  "test_acc_ci": [
+    0.8572241271820449,
+    0.8799953241895262
+  ],
+  "test_f1": 0.9044410963586561,
+  "test_f1_ci": [
+    0.8955916929937549,
+    0.9134957892904032
+  ],
+  "test_precision": 0.8957798154200936,
+  "test_recall": 0.914482667230412,
+  "roc_auc": 0.9903580625502204,
+  "pr_auc": 0.9498137857771093,
+  "ece": 0.0190220843824068
+}

results/vgg19_test.json

@@ -0,0 +1,18 @@
+{
+  "test_acc": 0.8930798004987531,
+  "test_acc_ci": [
+    0.8827930174563591,
+    0.9039900249376559
+  ],
+  "test_f1": 0.9211920362611569,
+  "test_f1_ci": [
+    0.9127978411193057,
+    0.9292878569132684
+  ],
+  "test_precision": 0.9110860665322441,
+  "test_recall": 0.9326584479061776,
+  "roc_auc": 0.9929719540404655,
+  "pr_auc": 0.9614448318930064,
+  "ece": 0.020435167797039903,
+  "n_folds_run": 0
+}

weights/clip_openai_v2_final.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3d3f20d7a04533574c5b91020fab47def33392620e9f4c8ad8b96e1d911c0d1e
+size 344853995

weights/densenet121_v2_final.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4d5b207059cc7811f192dea7cdf145afee82f583b05098870f79d6617d44b5a2
+size 28470922

weights/dinov2_l_v2.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:68d3215857fbb867475073b6a0e7280c9080edcf0ee5f4ab9ddcfb696b964734
+size 1217647177

weights/inception_v3_v2_final.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b239528f7b8a23e35b16189c3bf430095d4f9aa5a7494f3f00a1a33071eb9639
+size 97840003

weights/resnet101_v2_final.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a5527d9d11ff2e66502741cb0a9b3749d7acef54d8b236285ec7916f0b824861
+size 170716931

weights/resnet50_v2_final.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b638d103d680ba78df5c5290995e2b57b64c4ee47928d21d22cd9b9dcf1b0bee
+size 94435323

weights/retfound_v2.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d01b1ea313944d8cea49fec7fad746b6f4b24cbef58ee0512d6eca28e3174720
+size 1213365479

weights/swin_b_v2.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:208d699731a9db62d95e68b84f80b3e330d6fe33eb70c6fbae8ab72efe66630b
+size 347145547

weights/vgg19_v2_final.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:390c1eb1e47c72cc03ec63c1b0397d6a1b970dc79a837403adc25d1dffbde324
+size 558459173