Add full reproducible thyroid ResNet-18 experiment: weights, scripts, configs, calibration, locked threshold, test eval w/ CIs, figures, data exploration, README, LOG

.gitattributes

@@ -33,3 +33,9 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+results/figures/grid_Test_Benign.png filter=lfs diff=lfs merge=lfs -text
+results/figures/grid_Test_Malignant.png filter=lfs diff=lfs merge=lfs -text
+results/figures/grid_Train_Benign.png filter=lfs diff=lfs merge=lfs -text
+results/figures/grid_Train_Malignant.png filter=lfs diff=lfs merge=lfs -text
+results/figures/grid_Valid_Benign.png filter=lfs diff=lfs merge=lfs -text
+results/figures/grid_Valid_Malignant.png filter=lfs diff=lfs merge=lfs -text

LOG.md

@@ -0,0 +1,185 @@
+# Experiment Log — Agentic Thyroid ResNet-18
+
+Chronological, decision-by-decision record for reproducibility and journal review.
+
+## Provenance
+
+- **Experiment date (UTC):** 2026-06-05
+- **Dataset:** `Johnyquest7/TN5000-thyroid-nodule-classification`
+  - Commit SHA: `73d6c0713a89c8c07125fe6ffc5956be60e9853d`
+  - Loaded **directly from the Train/Valid/Test folder structure** (NOT the
+    datasets-viewer flattened `train` config, which merges all 5,000 rows),
+    so the predefined splits are respected.
+- **Compute:** Hugging Face GPU sandbox, NVIDIA A10G (24 GB), CUDA 13.0, cuDNN 9.2.
+- **Key packages:** torch 2.12.0+cu130, torchvision 0.27.0+cu130, timm 1.0.27,
+  scikit-learn 1.9.0, numpy 2.4.6, trackio 0.26.0 (see `configs/env_info.json`).
+- **Global seed:** 42. Strict determinism: `torch.use_deterministic_algorithms(True)`,
+  cuDNN deterministic, `CUBLAS_WORKSPACE_CONFIG=:4096:8`, seeded DataLoader workers.
+- **Positive class:** Malignant (label 1).
+- **Experiment tracking:** Trackio project `agentic_thyroid_resnet18`, dashboard
+  Space `Johnyquest7/Trakio_agentic_thyroid`, storage dataset
+  `Johnyquest7/Trakio_agentic_thyroid_dataset`.
+
+## Exact split usage
+
+| Split | n | Use |
+|-------|--:|-----|
+| Train | 3,500 | **Training only.** |
+| Valid | 500 | **Model selection (val AUROC), calibration, threshold selection.** |
+| Test | 1,000 | **Final locked evaluation, exactly once**, after model+calibration+threshold were frozen. |
+
+## Class distribution
+
+| Split | Benign | Malignant | Malignant % | Ratio (M:B) |
+|-------|-------:|----------:|------------:|------------:|
+| Train | 1,032 | 2,468 | 70.5% | 2.39 : 1 |
+| Valid |   125 |   375 | 75.0% | 3.00 : 1 |
+| Test  |   269 |   731 | 73.1% | 2.72 : 1 |
+
+Data audit (`data_exploration_report.md`): **0 corrupt images**, all 224×224 RGB
+PNG; **0 cross-split exact-pixel duplicates**, **0 filename-ID overlaps**, **0
+label conflicts**; per-split mean intensity ≈ 81.4 (std ≈ 19.4) — no distribution
+shift. Conclusion: **no detectable leakage; splits are clean and separate.**
+
+## Literature-informed augmentation rationale
+
+Augmentations were restricted to medically plausible B-mode ultrasound transforms
+(MediAug arXiv:2504.18983 + thyroid-US practice):
+- **Kept:** horizontal flip (thyroid is bilaterally symmetric), small rotation
+  (≤10°), mild affine translate (5%) / scale (0.9–1.1), mild brightness/contrast
+  (±15%, simulates gain/TGC), light Gaussian blur, and (in the `medical_strong`
+  ablation) narrow random-resized-crop (0.8–1.0) + mild speckle noise.
+- **Explicitly avoided:** vertical flip (US depth axis is physically meaningful),
+  large rotation/shear (distorts taller-than-wide / margin morphology — TI-RADS
+  malignancy cues), aggressive crop (<0.8, can remove the nodule), and any
+  color/HSV jitter (images are grayscale).
+
+Ablation result (val AUROC): `medical_default` **0.9712–0.9756** > `flip_only`
+**0.9637** > `medical_strong` **0.9609**. The literature-default policy won.
+
+## Model variants tried
+
+- `torchvision` ResNet-18 (ImageNet1K_V1, bilinear/256→224 preprocessing).
+- `timm:resnet18.a1_in1k` (A1 recipe, bicubic, crop_pct 0.95) — **selected**.
+- `timm:resnet18.a2_in1k` (A2 recipe).
+- Fine-tune depth: full fine-tune vs freeze stem+layer1 (`freeze_stage=1`).
+
+## Hyperparameter sweep (14 trials, one-factor-at-a-time around a literature-informed center)
+
+Center: timm a1, lr 2e-4, wd 1e-4, bs 32, `medical_default`, `pos_weight`,
+full fine-tune, BCE, AdamW, cosine, ≤40 epochs, early-stop(8). All runs logged to
+Trackio. **Selection metric: validation AUROC.** All 14 trials completed (rc=0).
+
+| Rank | Run | Change vs center | Val AUROC | Best epoch |
+|-----:|-----|------------------|----------:|-----------:|
+| 1 | **c12_loss_focal** | **focal γ=1.0, imbalance=none** | **0.9756** | 6 |
+| 2 | c09_imb_none | imbalance=none (BCE) | 0.9739 | 6 |
+| 3 | c01_backbone_torchvision | torchvision backbone | 0.9731 | 7 |
+| 4 | c03_lr_1e-4 | lr 1e-4 | 0.9721 | 11 |
+| 5 | c06_bs_64 | batch size 64 | 0.9717 | 8 |
+| 6 | c05_wd_1e-3 | weight decay 1e-3 | 0.9712 | 9 |
+| 6 | c00_center_a1 | center config | 0.9712 | 6 |
+| 8 | c10_imb_sampler | weighted sampler | 0.9697 | 13 |
+| 9 | c02_backbone_a2 | a2 backbone | 0.9693 | 6 |
+| 10 | c04_lr_5e-4 | lr 5e-4 | 0.9675 | 9 |
+| 11 | c11_freeze1 | freeze stem+layer1 | 0.9672 | 6 |
+| 12 | c13_lr1e-4_wd1e-3_drop | lr1e-4+wd1e-3+dropout0.2 | 0.9657 | 11 |
+| 13 | c07_aug_flip_only | flip-only aug | 0.9637 | 6 |
+| 14 | c08_aug_strong | strong aug | 0.9609 | 8 |
+
+Findings: (1) For this mild (~70/30) imbalance, **focal loss (γ=1.0) and no extra
+reweighting beat class-weighted BCE and weighted sampling** — heavy reweighting
+slightly hurt, consistent with the literature. (2) `medical_default` augmentation
+is the sweet spot. (3) **Full fine-tune > freezing.** (4) Backbones were close
+(a1 ≈ torchvision ≈ a2). Full per-run details: `results/tables/sweep_leaderboard.json`.
+
+No excessive trial count was used (14 one-factor trials) to avoid overfitting the
+500-image validation set.
+
+## Selected run
+
+**c12_loss_focal** — `timm:resnet18.a1_in1k`, focal loss (γ=1.0, α=0.5),
+imbalance=none, AdamW lr 2e-4 / wd 1e-4, batch 32, `medical_default` aug, full
+fine-tune, cosine schedule, best epoch 6, **validation AUROC 0.9756**. Selected
+**purely on validation AUROC**, before any test access. Config:
+`configs/final_config.yaml`; weights: `final_model.pt`.
+
+## Calibration decision
+
+Assessed on validation. **Temperature scaling** (single parameter, LBFGS on NLL)
+gave **T = 0.5646**. Validation ECE 0.0833 → **0.0308**, Brier 0.0592 → 0.0525,
+AUROC unchanged (0.9756; temperature scaling is monotonic ⇒ discrimination
+preserved). **Decision: use calibrated probabilities** for thresholding and test
+reporting. Parameters + before/after metrics: `configs/calibration.json`.
+Reliability diagrams: `results/figures/{valid,test}_calibration.png`.
+
+## Threshold selection decision
+
+On the validation set, using calibrated probabilities, the primary threshold was
+the **highest-specificity threshold achieving sensitivity ≥ 0.95** (sensitivity-
+prioritized, clinically motivated). Target was achievable.
+
+- **Locked threshold = 0.7113** → validation sensitivity **0.952**, specificity **0.896**.
+- Secondary reference (Youden's J): coincided at 0.7113 here.
+
+Threshold **locked before** the test set was evaluated. Config: `configs/threshold.json`.
+
+## Final locked threshold
+
+**0.7113139** (on calibrated malignancy probability).
+
+## Final test results with 95% CIs
+
+Test split (n=1000), calibrated probabilities + locked threshold. CIs: stratified
+bootstrap, 2000 resamples, seed=42.
+
+| Metric | Point | 95% CI |
+|--------|------:|:------:|
+| AUROC | 0.9371 | [0.9202, 0.9528] |
+| Sensitivity | 0.9042 | [0.8824, 0.9248] |
+| Specificity | 0.7955 | [0.7435, 0.8439] |
+| PPV | 0.9232 | [0.9054, 0.9401] |
+| NPV | 0.7535 | [0.7123, 0.7979] |
+| Accuracy | 0.8750 | [0.8540, 0.8950] |
+| F1 | 0.9136 | [0.8991, 0.9278] |
+| Brier | 0.0823 | — |
+| ECE | 0.0314 | — |
+
+Confusion matrix (Test): TN=214, FP=55, FN=70, TP=661.
+Tables: `results/tables/test_metrics_with_ci.{md,csv}`; per-image predictions:
+`results/{valid,test}_predictions.csv`.
+
+## Failed / weaker runs
+
+No run errored (all rc=0). Weakest configurations: `medical_strong` augmentation
+(0.9609) and `flip_only` (0.9637) — both under the `medical_default` baseline,
+confirming the augmentation policy choice. Earlier trackio smoke runs
+(`smoke_trackio_test*`, `connectivity_check`, `dataset_pin_check`) are
+infrastructure-validation runs, not experiments. Early Trackio Space creation
+produced transient 401 `/volumes` warnings until a persistent `dataset_id`
+(`Johnyquest7/Trakio_agentic_thyroid_dataset`) was pinned; resolved thereafter.
+
+## Limitations
+
+- Single-source dataset; cropped-ROI inputs; mild class imbalance.
+- The ≥0.95-sensitivity operating point set on validation yielded **0.904**
+  sensitivity on test — the operating point does not transfer perfectly; ~10% of
+  malignant nodules are missed at the locked threshold. Local threshold
+  re-calibration is advisable before any use.
+- Leakage checks (exact-pixel hash + filename-ID overlap) are exhaustive for the
+  available signal but cannot exclude same-patient/near-duplicate leakage if such
+  structure exists upstream in TN5000.
+
+## External validation — NOT yet performed
+
+No external/independent dataset has been evaluated. `evaluate_external.py` is
+provided to run the locked model (same preprocessing, calibration T, and locked
+threshold) on a future external set (folder or CSV format). External, ideally
+prospective and multi-site, validation is **required** before any clinical use.
+
+## Test-set integrity statement
+
+> The Test split was evaluated **exactly once**, and **only after** the model was
+> selected (by validation AUROC), calibrated (temperature scaling on validation),
+> and the decision threshold was locked (on validation). No hyperparameter,
+> calibration, or threshold decision used the test set.

README.md

@@ -1,3 +1,200 @@
 ---
-license: mit
+license: cc-by-nc-4.0
+tags:
+- medical-imaging
+- thyroid
+- ultrasound
+- image-classification
+- resnet18
+- calibration
+pipeline_tag: image-classification
 ---
+
+# Agentic Thyroid ResNet-18 — Ultrasound Nodule Malignancy Classifier
+
+> ⚠️ **RESEARCH USE ONLY — NOT FOR CLINICAL USE.** This model is a research
+> artifact trained on a single retrospective dataset. It has **not** been
+> externally validated and **must not** be used for diagnosis, screening, or any
+> clinical decision-making. External, prospective validation is required before
+> any clinical consideration.
+
+A ResNet-18 binary classifier that predicts the probability that a cropped
+thyroid ultrasound nodule image is **malignant** (positive class) vs **benign**.
+Built for a reproducible, publication-oriented experiment with proper
+calibration and a sensitivity-prioritized, validation-locked decision threshold.
+
+- **Backbone:** `timm` ResNet-18, A1 ImageNet-1k recipe (`resnet18.a1_in1k`), full fine-tune
+- **Selected by:** validation AUROC (14-trial sweep; winner val AUROC **0.9756**)
+- **Calibration:** temperature scaling (T = 0.5646), fit on validation
+- **Locked threshold:** **0.7113** (highest-specificity threshold with validation sensitivity ≥ 0.95)
+
+## Intended use
+
+- **Intended:** methodological research, benchmarking, and as a baseline for
+  thyroid ultrasound malignancy classification studies.
+- **Out of scope:** any clinical, diagnostic, triage, or screening use; use on
+  images acquired/preprocessed differently from the training data without
+  re-validation; use on non-thyroid or non-ultrasound images.
+
+## Dataset
+
+- **Source:** [`Johnyquest7/TN5000-thyroid-nodule-classification`](https://huggingface.co/datasets/Johnyquest7/TN5000-thyroid-nodule-classification)
+  (derived from TN5000; nodule ROI cropped to 224×224 RGB PNG).
+- **Splits (kept strictly separate):** Train 3,500 · Valid 500 · Test 1,000.
+- **Labels:** `0 = Benign`, `1 = Malignant` (positive class = Malignant).
+- **Class balance:** ~70–75% malignant in every split (mild imbalance). See
+  [`data_exploration_report.md`](data_exploration_report.md): **0 corrupt images,
+  0 cross-split pixel duplicates, 0 filename-ID overlaps** → no detectable leakage.
+
+## Label definitions
+
+| Label | Class | Meaning |
+|------:|-------|---------|
+| 0 | Benign | Non-malignant thyroid nodule |
+| 1 | Malignant | Malignant thyroid nodule (positive class) |
+
+## Preprocessing (locked — `configs/preprocess.json`)
+
+Deterministic eval/inference path (no augmentation):
+1. Resize to **224×224** (bicubic; the timm A1 data config).
+2. `ToTensor()`.
+3. Normalize with ImageNet mean `[0.485, 0.456, 0.406]`, std `[0.229, 0.224, 0.225]`.
+
+Grayscale ultrasound images are loaded as 3-channel RGB.
+
+## Model architecture
+
+ResNet-18 with a single-logit binary head (`num_classes=1`); sigmoid → probability
+of malignancy. ~11.2M parameters. Trained from ImageNet-1k weights (full fine-tune).
+
+## Training procedure (final / winning config)
+
+| Setting | Value |
+|---|---|
+| Backbone | `timm:resnet18.a1_in1k`, full fine-tune |
+| Loss | Focal loss (γ=1.0, α=0.5) |
+| Class-imbalance handling | none beyond focal (mild imbalance; preserves calibration) |
+| Augmentation | `medical_default`: hflip, mild affine (rot ≤10°, translate 5%, scale 0.9–1.1), mild brightness/contrast (±15%), light Gaussian blur (p=0.2) |
+| Optimizer | AdamW, lr 2e-4, weight decay 1e-4 |
+| Scheduler | Cosine annealing, 2-epoch warmup |
+| Batch size | 32 |
+| Epochs | ≤40, early stopping on val AUROC (patience 8) |
+| Mixed precision | yes (fp16 autocast) |
+| Seed | 42 (strict determinism; `CUBLAS_WORKSPACE_CONFIG=:4096:8`, cuDNN deterministic) |
+| Best epoch | 6 |
+
+Augmentations were chosen to be medically plausible for B-mode ultrasound
+(no vertical flip, no large rotation/shear, no aggressive crop, no color/HSV
+jitter), informed by MediAug (arXiv:2504.18983) and thyroid-US best practice.
+The augmentation ablation confirmed `medical_default` (0.9712–0.9756 val AUROC)
+outperforms both `flip_only` (0.9637) and `medical_strong` (0.9609).
+
+Environment: torch 2.12.0+cu130, torchvision 0.27.0+cu130, timm 1.0.27,
+scikit-learn 1.9.0, numpy 2.4.6, NVIDIA A10G (CUDA 13.0, cuDNN 9.2).
+Full versions in `configs/env_info.json`.
+
+## Validation threshold strategy
+
+After selecting the model by validation AUROC and calibrating on validation, the
+decision threshold was chosen on the **validation set** as the **highest-specificity
+threshold achieving sensitivity ≥ 0.95** (clinically sensitivity-prioritized).
+Youden's J is reported as a secondary reference. The threshold (**0.7113**) was
+**locked before** the test set was touched.
+
+- Validation @ locked threshold: sensitivity **0.952**, specificity **0.896**.
+
+## Test performance (locked model + locked threshold, n=1000)
+
+CI method: stratified bootstrap, 2000 resamples, seed=42. Probabilities calibrated.
+
+| Metric | Point estimate | 95% CI |
+|--------|---------------:|:------:|
+| AUROC | **0.9371** | [0.9202, 0.9528] |
+| Sensitivity | 0.9042 | [0.8824, 0.9248] |
+| Specificity | 0.7955 | [0.7435, 0.8439] |
+| PPV | 0.9232 | [0.9054, 0.9401] |
+| NPV | 0.7535 | [0.7123, 0.7979] |
+| Accuracy | 0.8750 | [0.8540, 0.8950] |
+| F1 | 0.9136 | [0.8991, 0.9278] |
+| Brier | 0.0823 | — |
+| ECE | 0.0314 | — |
+
+## Calibration results
+
+Temperature scaling (T=0.5646) on validation reduced **validation ECE from 0.0833
+→ 0.0308** and Brier from 0.0592 → 0.0525 with AUROC unchanged (monotonic). On
+the test set, calibrated ECE is **0.0314** (well calibrated). Reliability diagrams:
+`results/figures/valid_calibration.png`, `results/figures/test_calibration.png`.
+
+## Confusion matrix (Test)
+
+TN = 214 · FP = 55 · FN = 70 · TP = 661
+
+![Test confusion matrix](results/figures/test_confusion_counts.png)
+
+Additional figures: `results/figures/test_roc.png`,
+`results/figures/test_pr.png`, `results/figures/test_confusion_normalized.png`.
+
+## Files in this repo
+
+```
+final_model.pt                 # locked weights + backbone/preprocess metadata
+configs/final_config.yaml      # single-command training config
+configs/preprocess.json        # locked preprocessing
+configs/calibration.json       # temperature scaling parameter + before/after metrics
+configs/threshold.json         # locked decision threshold + selection method
+configs/env_info.json          # exact package/hardware/CUDA versions
+train.py  evaluate.py  evaluate_external.py  explore_data.py  sweep.py  finalize.py
+thyroid_lib.py                 # shared preprocessing/model/calibration/metrics
+requirements.txt
+data_exploration_report.md     # full data audit (counts, leakage, intensity, grids)
+LOG.md                         # chronological experiment log
+results/                       # figures, tables (incl. CIs + sweep leaderboard), per-image CSVs
+```
+
+## Reproduce
+
+```bash
+pip install -r requirements.txt
+python explore_data.py --dataset_id Johnyquest7/TN5000-thyroid-nodule-classification
+python train.py    --config configs/final_config.yaml
+python evaluate.py --split test --config configs/final_config.yaml
+```
+
+Evaluate a future external dataset with the same preprocessing/calibration/threshold:
+
+```bash
+python evaluate_external.py \
+  --model_repo Johnyquest7/agentic_thyroid_model \
+  --data_dir /path/to/external_dataset \
+  --output_dir external_results
+# external dataset = folder with Benign/ Malignant/ subfolders, OR add --csv labels.csv
+```
+
+## Limitations, bias, and leakage concerns
+
+- **Single-source dataset; no external validation.** Performance on data from
+  other scanners, institutions, or populations is unknown and likely lower.
+- **Cropped-ROI inputs.** The model expects nodule-cropped images like TN5000;
+  whole-frame ultrasound will be out of distribution.
+- **Sensitivity gap at deployment threshold.** The threshold targets ≥0.95
+  sensitivity on validation; on the test set sensitivity was 0.904 — i.e. the
+  operating point does not perfectly transfer, and ~10% of malignant nodules
+  were missed at this threshold. Threshold re-calibration on local data is
+  advisable before any use.
+- **Label/selection bias.** Labels and cohort composition reflect the source
+  dataset's referral and pathology-confirmation process.
+- **Leakage checks were exhaustive within the available signal** (exact pixel
+  hashing + filename-ID overlap, all zero) but cannot rule out near-duplicate or
+  same-patient-different-image leakage if such structure exists in the source.
+
+## ⚠️ External validation required before clinical use
+
+This model **requires independent, ideally prospective, multi-site external
+validation** before any clinical consideration. It is released for research
+reproducibility only.
+
+## Citation
+
+Dataset: TN5000 (Yu et al., *Scientific Data*, 2025).
+Augmentation guidance: MediAug, arXiv:2504.18983.

configs/calibration.json

@@ -0,0 +1,13 @@
+{
+  "method": "temperature_scaling",
+  "temperature": 0.5645509958267212,
+  "valid": {
+    "auroc_uncal": 0.975616,
+    "auroc_cal": 0.975616,
+    "ece_uncal": 0.08334361362457275,
+    "ece_cal": 0.03078642554581164,
+    "brier_uncal": 0.05917946249246597,
+    "brier_cal": 0.05246718227863312
+  },
+  "use_calibrated": true
+}

configs/env_info.json

@@ -0,0 +1,19 @@
+{
+  "torch": "2.12.0+cu130",
+  "cuda_available": true,
+  "cuda_version": "13.0",
+  "cudnn_version": 92000,
+  "cudnn_deterministic": true,
+  "cudnn_benchmark": false,
+  "gpu_name": "NVIDIA A10G",
+  "gpu_count": 1,
+  "gpu_total_mem_gb": 23.95,
+  "torchvision": "0.27.0+cu130",
+  "timm": "1.0.27",
+  "sklearn": "1.9.0",
+  "numpy": "2.4.6",
+  "PIL": "12.2.0",
+  "trackio": "0.26.0",
+  "cublas_workspace_config": ":4096:8",
+  "pythonhashseed": "42"
+}

configs/final_config.yaml

@@ -0,0 +1,45 @@
+# Final, locked training configuration for the published thyroid ResNet-18 model.
+# Selected by VALIDATION AUROC from a 14-trial sweep (winner: c12_loss_focal,
+# val AUROC = 0.9756). Reproduce training with:
+#     python train.py --config configs/final_config.yaml
+# Then evaluate the locked model on the test split with:
+#     python evaluate.py --split test --config configs/final_config.yaml
+
+dataset_id: Johnyquest7/TN5000-thyroid-nodule-classification
+# data_dir: /path/to/local/TN5000   # optional; if unset, dataset is downloaded from the Hub
+output_dir: run_final
+
+# --- model ---
+backbone: timm:resnet18.a1_in1k     # timm ResNet-18, A1 ImageNet-1k recipe weights
+freeze_stage: 0                     # full fine-tune
+dropout: 0.0
+
+# --- augmentation (medically plausible ultrasound augmentations only) ---
+aug_policy: medical_default         # hflip + mild affine(rot<=10,trans5%,scale0.9-1.1) + mild brightness/contrast + light gaussian blur
+
+# --- loss / class imbalance ---
+loss: focal                         # focal loss won the sweep for this mild imbalance
+focal_gamma: 1.0
+focal_alpha: 0.5
+imbalance: none                     # no extra reweighting/sampling (focal handles it; preserves calibration)
+
+# --- optimization ---
+optimizer: adamw
+lr: 0.0002
+weight_decay: 0.0001
+batch_size: 32
+epochs: 40
+scheduler: cosine
+warmup_epochs: 2
+early_stop_patience: 8              # early stopping on validation AUROC
+amp: true                           # mixed precision
+
+# --- reproducibility ---
+seed: 42
+strict_determinism: true
+
+# --- experiment tracking ---
+trackio_project: agentic_thyroid_resnet18
+trackio_space_id: Johnyquest7/Trakio_agentic_thyroid
+trackio_dataset_id: Johnyquest7/Trakio_agentic_thyroid_dataset
+run_name: final_selected_model

configs/preprocess.json

@@ -0,0 +1,17 @@
+{
+  "image_size": 224,
+  "mean": [
+    0.485,
+    0.456,
+    0.406
+  ],
+  "std": [
+    0.229,
+    0.224,
+    0.225
+  ],
+  "interpolation": "bicubic",
+  "positive_class": "Malignant",
+  "positive_index": 1,
+  "note": "Deterministic eval/inference preprocessing. No augmentation."
+}

configs/threshold.json

@@ -0,0 +1,14 @@
+{
+  "primary_method": "highest-specificity threshold with sensitivity >= 0.95 on validation (calibrated probabilities)",
+  "locked_threshold": 0.7113139033317566,
+  "valid_sensitivity_at_threshold": 0.952,
+  "valid_specificity_at_threshold": 0.896,
+  "target_sensitivity": 0.95,
+  "target_achievable": true,
+  "secondary_youden": {
+    "threshold": 0.7113139033317566,
+    "sensitivity": 0.952,
+    "specificity": 0.896
+  },
+  "probabilities_used": "calibrated"
+}

configs/winning_run_command_line.txt

@@ -0,0 +1 @@
+python train.py --data_dir /app/TN5000 --output_dir /app/sweep_runs/c12_loss_focal --run_name c12_loss_focal --seed 42 --trackio_project agentic_thyroid_resnet18 --trackio_space_id Johnyquest7/Trakio_agentic_thyroid --trackio_dataset_id Johnyquest7/Trakio_agentic_thyroid_dataset --num_workers 4 --backbone timm:resnet18.a1_in1k --lr 0.0002 --weight_decay 0.0001 --batch_size 32 --aug_policy medical_default --imbalance none --freeze_stage 0 --loss focal --optimizer adamw --scheduler cosine --epochs 40 --early_stop_patience 8 --dropout 0.0 --focal_gamma 1.0

configs/winning_run_resolved_config.json

@@ -0,0 +1,49 @@
+{
+  "config": null,
+  "dataset_id": "Johnyquest7/TN5000-thyroid-nodule-classification",
+  "data_dir": "/app/TN5000",
+  "output_dir": "/app/sweep_runs/c12_loss_focal",
+  "backbone": "timm:resnet18.a1_in1k",
+  "freeze_stage": 0,
+  "dropout": 0.0,
+  "aug_policy": "medical_default",
+  "loss": "focal",
+  "focal_gamma": 1.0,
+  "focal_alpha": 0.5,
+  "imbalance": "none",
+  "optimizer": "adamw",
+  "lr": 0.0002,
+  "weight_decay": 0.0001,
+  "batch_size": 32,
+  "epochs": 40,
+  "scheduler": "cosine",
+  "warmup_epochs": 2,
+  "early_stop_patience": 8,
+  "amp": true,
+  "num_workers": 4,
+  "seed": 42,
+  "strict_determinism": true,
+  "trackio_project": "agentic_thyroid_resnet18",
+  "trackio_space_id": "Johnyquest7/Trakio_agentic_thyroid",
+  "trackio_dataset_id": "Johnyquest7/Trakio_agentic_thyroid_dataset",
+  "run_name": "c12_loss_focal",
+  "no_trackio": false,
+  "pos_weight": null,
+  "n_train_neg": 1032,
+  "n_train_pos": 2468,
+  "preprocess": {
+    "image_size": 224,
+    "mean": [
+      0.485,
+      0.456,
+      0.406
+    ],
+    "std": [
+      0.229,
+      0.224,
+      0.225
+    ],
+    "interpolation": "bicubic"
+  },
+  "device": "cuda"
+}

data_exploration_report.md

@@ -0,0 +1,101 @@
+# Data Exploration Report — TN5000 Thyroid Nodule Classification
+
+- **Generated (UTC):** 2026-06-05T03:24:51.156145+00:00
+- **Dataset:** `Johnyquest7/TN5000-thyroid-nodule-classification`
+- **Source:** TN5000 (Yu et al., *Scientific Data*, 2025), cropped to nodule ROI, 224×224 PNG.
+- **Task:** Binary classification — 0 = Benign, 1 = Malignant. Positive class = Malignant.
+
+## 1. Number of images per split and class
+
+| Split | Benign (0) | Malignant (1) | Total | Malignant % | Malignant:Benign ratio |
+|-------|-----------:|--------------:|------:|------------:|------------------------:|
+| Train | 1032 | 2468 | 3500 | 70.5% | 2.39 : 1 |
+| Valid | 125 | 375 | 500 | 75.0% | 3.00 : 1 |
+| Test | 269 | 731 | 1000 | 73.1% | 2.72 : 1 |
+| **Total** | **1426** | **3574** | **5000** | **71.5%** | **2.51 : 1** |
+
+![Class distribution](results/figures/class_distribution.png)
+
+## 2. Class imbalance
+
+All three splits are **malignant-majority** (~70–75% malignant), i.e. mild imbalance (malignant:benign roughly 2.4–3.0 : 1), consistent across splits.
+
+- **Mitigation evaluated in training:** class-weighted `BCEWithLogitsLoss` (`pos_weight = N_benign/N_malignant`), focal loss, and a weighted sampler were all compared in the sweep; the final model uses focal loss (γ=1.0). Because imbalance is mild and calibration matters, heavy reweighting was avoided.
+
+## 3. Image dimensions, channels, file format
+
+- **File format:** PNG (lossless) for all 5000 images.
+- **Dimensions observed:** [(224, 224)] (expected single value (224, 224)).
+- **PIL modes observed:** ['RGB'] (RGB; grayscale replicated across 3 channels).
+
+| Split | Unique dimensions | Modes |
+|-------|-------------------|-------|
+| Train | {(224, 224): 3500} | {'RGB': 3500} |
+| Valid | {(224, 224): 500} | {'RGB': 500} |
+| Test | {(224, 224): 1000} | {'RGB': 1000} |
+
+## 4. Missing / corrupt image check
+
+- ✅ **No corrupt or unreadable images.** All images opened and decoded via PIL `verify()` + reload.
+
+## 5. Duplicate image check (exact pixel-content MD5)
+
+- Duplicate groups **within a single split:** 0
+- Duplicate groups **spanning multiple splits (potential LEAKAGE):** 0
+- Duplicate groups with **conflicting labels:** 0
+
+- ✅ No cross-split pixel duplicates and no label conflicts detected.
+
+## 6. Data leakage analysis
+
+TN5000 assigns each image a **globally unique numeric ID**, preserved as the PNG filename. Overlap of filename IDs across splits would indicate the same source image in two splits.
+
+| Pair | Shared filename IDs |
+|------|--------------------:|
+| Train ∩ Valid | 0 |
+| Train ∩ Test  | 0 |
+| Valid ∩ Test  | 0 |
+
+- ✅ **No filename-ID overlap across splits.** Combined with the exact-pixel duplicate check above, there is no detectable leakage between Train, Valid, and Test.
+
+## 7. Pixel-intensity distribution
+
+| Split | Mean | Std | Min | Max |
+|-------|-----:|----:|----:|----:|
+| Train | 81.5 | 19.6 | 27.1 | 163.5 |
+| Valid | 81.3 | 19.4 | 35.0 | 171.5 |
+| Test | 81.4 | 19.1 | 33.4 | 150.6 |
+
+![Intensity distribution](results/figures/intensity_distribution.png)
+
+Mean per-image grayscale intensity distributions are **closely matched across splits**, indicating consistent acquisition/preprocessing and no obvious distribution shift.
+
+## 8. Representative image grids
+
+**Train / Benign**
+
+![Train Benign](results/figures/grid_Train_Benign.png)
+
+**Train / Malignant**
+
+![Train Malignant](results/figures/grid_Train_Malignant.png)
+
+**Valid / Benign**
+
+![Valid Benign](results/figures/grid_Valid_Benign.png)
+
+**Valid / Malignant**
+
+![Valid Malignant](results/figures/grid_Valid_Malignant.png)
+
+**Test / Benign**
+
+![Test Benign](results/figures/grid_Test_Benign.png)
+
+**Test / Malignant**
+
+![Test Malignant](results/figures/grid_Test_Malignant.png)
+
+## 9. Train/Valid/Test separation statement
+
+> The Train, Valid, and Test folders provided in the dataset repository were kept **strictly separate** throughout this experiment. The model was trained on **Train only**; the **Valid** split was used for model selection, calibration, and threshold selection; and the **Test** split was used **exactly once** for final locked evaluation after the model, calibration, and decision threshold were frozen. The exact-pixel duplicate check and filename-ID overlap check above confirm there is no detectable leakage between the three splits.

evaluate.py

@@ -0,0 +1,123 @@
+#!/usr/bin/env python
+"""
+Final evaluation script for the validation/test splits of TN5000.
+
+Uses the LOCKED final model weights, LOCKED preprocessing, LOCKED calibration
+(temperature scaling) and LOCKED decision threshold stored in this repo. The
+test split is intended to be evaluated only once, after model/calibration/
+threshold were frozen.
+
+Usage:
+    python evaluate.py --split test --config configs/final_config.yaml
+    python evaluate.py --split valid
+
+Reads (relative to --repo_dir, default '.'):
+    final_model.pt              (or weights pointed to by final_config.yaml)
+    configs/preprocess.json
+    configs/calibration.json
+    configs/threshold.json
+
+Writes per-image predictions + a metrics table (with bootstrap 95% CI) to
+--output_dir (default results/eval_<split>/).
+"""
+import argparse
+import csv
+import json
+from pathlib import Path
+
+import numpy as np
+import yaml
+
+import thyroid_lib as L
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--split", choices=["valid", "test"], required=True)
+    ap.add_argument("--config", default=None, help="optional final_config.yaml (for data_dir/dataset_id)")
+    ap.add_argument("--repo_dir", default=".")
+    ap.add_argument("--data_dir", default=None, help="local TN5000 dir; else downloaded from Hub")
+    ap.add_argument("--dataset_id", default="Johnyquest7/TN5000-thyroid-nodule-classification")
+    ap.add_argument("--weights", default="final_model.pt")
+    ap.add_argument("--output_dir", default=None)
+    ap.add_argument("--n_boot", type=int, default=2000)
+    ap.add_argument("--boot_seed", type=int, default=42)
+    args = ap.parse_args()
+
+    if args.config and Path(args.config).exists():
+        cfg = yaml.safe_load(open(args.config)) or {}
+        if not args.data_dir and cfg.get("data_dir"):
+            args.data_dir = cfg["data_dir"]
+        if cfg.get("dataset_id"):
+            args.dataset_id = cfg["dataset_id"]
+
+    import torch
+    from torch.utils.data import DataLoader
+    device = "cuda" if torch.cuda.is_available() else "cpu"
+    L.set_determinism(args.boot_seed, strict=True)
+
+    repo = Path(args.repo_dir)
+    split_name = {"valid": "Valid", "test": "Test"}[args.split]
+    out_dir = Path(args.output_dir) if args.output_dir else repo / "results" / f"eval_{args.split}"
+    out_dir.mkdir(parents=True, exist_ok=True)
+
+    # data
+    if args.data_dir:
+        data_dir = Path(args.data_dir)
+    else:
+        from huggingface_hub import snapshot_download
+        data_dir = Path(snapshot_download(repo_id=args.dataset_id, repo_type="dataset",
+                                          local_dir=str(out_dir / "_data"),
+                                          allow_patterns=[f"{split_name}/**"]))
+
+    # locked artifacts
+    pp = L.PreprocessConfig.from_dict(json.load(open(repo / "configs" / "preprocess.json")))
+    calib = json.load(open(repo / "configs" / "calibration.json"))
+    thr_cfg = json.load(open(repo / "configs" / "threshold.json"))
+    T = calib["temperature"]; use_cal = calib.get("use_calibrated", True)
+    thr = thr_cfg["locked_threshold"]
+
+    # model
+    ck = torch.load(repo / args.weights, map_location="cpu", weights_only=False)
+    model, _ = L.build_model(ck["backbone"], freeze_stage=ck.get("freeze_stage", 0),
+                             dropout=ck.get("dropout", 0.0))
+    model.load_state_dict(ck["model_state"]); model.to(device).eval()
+
+    ds = L.ThyroidImageFolder(data_dir / split_name, L.build_eval_transform(pp))
+    loader = DataLoader(ds, batch_size=64, shuffle=False, num_workers=4,
+                        pin_memory=(device == "cuda"))
+    logits, y, ids = L.collect_logits(model, loader, device, amp=False)
+    probs = L.apply_temperature(logits, T) if use_cal else L.sigmoid(logits)
+
+    metrics = L.point_metrics(y, probs, thr)
+    ci = L.bootstrap_ci(y, probs, thr, n_boot=args.n_boot, seed=args.boot_seed)
+
+    # per-image csv
+    pred = (probs >= thr).astype(int)
+    with open(out_dir / f"{args.split}_predictions.csv", "w", newline="") as f:
+        w = csv.writer(f)
+        w.writerow(["image_id", "true_label", "true_class", "probability_malignant",
+                    "predicted_label", "predicted_class"])
+        for i, yy, pr, pd in zip(ids, y, probs, pred):
+            w.writerow([i, int(yy), L.IDX_TO_CLASS[int(yy)], f"{pr:.6f}",
+                        int(pd), L.IDX_TO_CLASS[int(pd)]])
+
+    ci_keys = ["auroc", "sensitivity", "specificity", "ppv", "npv", "accuracy", "f1"]
+    out = {"split": args.split, "n": metrics["n"], "n_pos": metrics["n_pos"],
+           "n_neg": metrics["n_neg"], "threshold": thr,
+           "calibration": "temperature(T=%.4f)" % T if use_cal else "none",
+           "metrics": metrics, "metrics_95ci": {k: list(ci[k]) for k in ci_keys},
+           "ci_method": f"stratified bootstrap, {args.n_boot} resamples, seed={args.boot_seed}"}
+    json.dump(out, open(out_dir / f"{args.split}_metrics.json", "w"), indent=2)
+
+    print(f"=== {args.split.upper()} (n={metrics['n']}, thr={thr:.4f}) ===")
+    for k in ci_keys:
+        print(f"  {k:12s} {metrics[k]:.4f}  CI [{ci[k][0]:.4f}, {ci[k][1]:.4f}]")
+    print(f"  brier        {metrics['brier']:.4f}")
+    print(f"  ece          {metrics['ece']:.4f}")
+    print(f"  confusion    TN={metrics['tn']} FP={metrics['fp']} FN={metrics['fn']} TP={metrics['tp']}")
+    print("Saved to", out_dir)
+
+
+if __name__ == "__main__":
+    main()

evaluate_external.py

@@ -0,0 +1,217 @@
+#!/usr/bin/env python
+"""
+Evaluate the LOCKED thyroid ResNet-18 model on an EXTERNAL dataset, using the
+exact same preprocessing, calibration (temperature scaling) and locked decision
+threshold from the final model repo.
+
+Usage:
+    python evaluate_external.py \
+        --model_repo Johnyquest7/agentic_thyroid_model \
+        --data_dir /path/to/external_dataset \
+        --output_dir external_results
+
+The external dataset may be provided in either of two formats:
+
+  (A) Folder format with class subfolders:
+        <data_dir>/Benign/*.png|jpg|...
+        <data_dir>/Malignant/*.png|jpg|...
+      (case-insensitive; also accepts 0/1 or benign/malignant)
+
+  (B) CSV with image paths and labels:
+        --csv /path/to/labels.csv
+      with columns:
+        image_path  (absolute, or relative to --data_dir or to the CSV's folder)
+        label       (0/1, or benign/malignant, case-insensitive)
+
+If labels are present, full metrics + bootstrap 95% CIs are computed. If labels
+are absent/unknown, only per-image probabilities and predictions are written.
+
+The model weights and locked configs are downloaded from --model_repo (or read
+from --local_repo_dir if provided).
+"""
+import argparse
+import csv
+import json
+from pathlib import Path
+
+import numpy as np
+
+import thyroid_lib as L
+
+IMG_EXT = {".png", ".jpg", ".jpeg", ".bmp", ".tif", ".tiff", ".webp"}
+BENIGN_ALIASES = {"benign", "0", "b", "neg", "negative"}
+MALIGNANT_ALIASES = {"malignant", "1", "m", "pos", "positive", "cancer"}
+
+
+def parse_label(v):
+    s = str(v).strip().lower()
+    if s in BENIGN_ALIASES:
+        return 0
+    if s in MALIGNANT_ALIASES:
+        return 1
+    return None  # unknown
+
+
+def gather_folder(data_dir):
+    data_dir = Path(data_dir)
+    items = []  # (path, label_or_None, id)
+    # find class subfolders case-insensitively
+    subdirs = {p.name.lower(): p for p in data_dir.iterdir() if p.is_dir()}
+    cls_map = {}
+    for name, p in subdirs.items():
+        lab = parse_label(name)
+        if lab is not None:
+            cls_map[p] = lab
+    if cls_map:
+        for p, lab in cls_map.items():
+            for f in sorted(p.rglob("*")):
+                if f.suffix.lower() in IMG_EXT:
+                    items.append((f, lab, f.stem))
+    else:
+        # flat folder, no labels
+        for f in sorted(data_dir.rglob("*")):
+            if f.suffix.lower() in IMG_EXT:
+                items.append((f, None, f.stem))
+    return items
+
+
+def gather_csv(csv_path, data_dir):
+    csv_path = Path(csv_path)
+    base = Path(data_dir) if data_dir else csv_path.parent
+    items = []
+    with open(csv_path) as f:
+        reader = csv.DictReader(f)
+        cols = {c.lower(): c for c in reader.fieldnames}
+        pcol = cols.get("image_path") or cols.get("path") or cols.get("image") or cols.get("filepath")
+        lcol = cols.get("label") or cols.get("class") or cols.get("target")
+        if pcol is None:
+            raise ValueError("CSV must have an image path column (image_path/path/image).")
+        for row in reader:
+            raw = row[pcol]
+            p = Path(raw)
+            if not p.is_absolute():
+                cand = base / raw
+                p = cand if cand.exists() else (csv_path.parent / raw)
+            lab = parse_label(row[lcol]) if lcol else None
+            items.append((p, lab, p.stem))
+    return items
+
+
+class ListDataset:
+    def __init__(self, items, transform):
+        from PIL import Image
+        self.Image = Image
+        self.items = items
+        self.transform = transform
+
+    def __len__(self):
+        return len(self.items)
+
+    def __getitem__(self, i):
+        path, lab, iid = self.items[i]
+        with self.Image.open(path) as im:
+            x = self.transform(im.convert("RGB"))
+        return x, (-1 if lab is None else lab), iid
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--model_repo", default="Johnyquest7/agentic_thyroid_model")
+    ap.add_argument("--local_repo_dir", default=None,
+                    help="use a local copy of the model repo instead of downloading")
+    ap.add_argument("--data_dir", default=None, help="external image folder (format A) or CSV base")
+    ap.add_argument("--csv", default=None, help="CSV with image_path,label (format B)")
+    ap.add_argument("--weights", default="final_model.pt")
+    ap.add_argument("--output_dir", default="external_results")
+    ap.add_argument("--n_boot", type=int, default=2000)
+    ap.add_argument("--boot_seed", type=int, default=42)
+    args = ap.parse_args()
+
+    if not args.data_dir and not args.csv:
+        raise SystemExit("Provide --data_dir (folder format) or --csv (CSV format).")
+
+    import torch
+    from torch.utils.data import DataLoader
+    device = "cuda" if torch.cuda.is_available() else "cpu"
+    L.set_determinism(args.boot_seed, strict=True)
+
+    out_dir = Path(args.output_dir); out_dir.mkdir(parents=True, exist_ok=True)
+
+    # ---- fetch locked repo artifacts ----
+    if args.local_repo_dir:
+        repo = Path(args.local_repo_dir)
+    else:
+        from huggingface_hub import snapshot_download
+        repo = Path(snapshot_download(repo_id=args.model_repo, repo_type="model",
+                                      local_dir=str(out_dir / "_model_repo"),
+                                      allow_patterns=[args.weights, "configs/*", "thyroid_lib.py"]))
+
+    pp = L.PreprocessConfig.from_dict(json.load(open(repo / "configs" / "preprocess.json")))
+    calib = json.load(open(repo / "configs" / "calibration.json"))
+    thr_cfg = json.load(open(repo / "configs" / "threshold.json"))
+    T = calib["temperature"]; use_cal = calib.get("use_calibrated", True)
+    thr = thr_cfg["locked_threshold"]
+
+    ck = torch.load(repo / args.weights, map_location="cpu", weights_only=False)
+    model, _ = L.build_model(ck["backbone"], freeze_stage=ck.get("freeze_stage", 0),
+                             dropout=ck.get("dropout", 0.0))
+    model.load_state_dict(ck["model_state"]); model.to(device).eval()
+
+    # ---- gather external images ----
+    items = gather_csv(args.csv, args.data_dir) if args.csv else gather_folder(args.data_dir)
+    if not items:
+        raise SystemExit("No images found in external dataset.")
+    has_labels = all(it[1] is not None for it in items)
+    print(f"Found {len(items)} external images; labels available: {has_labels}")
+
+    ds = ListDataset(items, L.build_eval_transform(pp))
+    loader = DataLoader(ds, batch_size=64, shuffle=False, num_workers=4,
+                        pin_memory=(device == "cuda"))
+
+    logits_all, labels_all, ids_all = [], [], []
+    with torch.no_grad():
+        for x, y, iid in loader:
+            x = x.to(device)
+            out = model(x).view(-1)
+            logits_all.append(out.float().cpu().numpy())
+            labels_all.append(np.asarray(y))
+            ids_all.extend(list(iid))
+    logits = np.concatenate(logits_all); labels = np.concatenate(labels_all).astype(int)
+    probs = L.apply_temperature(logits, T) if use_cal else L.sigmoid(logits)
+    pred = (probs >= thr).astype(int)
+
+    # ---- per-image CSV ----
+    with open(out_dir / "external_predictions.csv", "w", newline="") as f:
+        w = csv.writer(f)
+        w.writerow(["image_id", "true_label", "probability_malignant",
+                    "predicted_label", "predicted_class"])
+        for i, yy, pr, pd in zip(ids_all, labels, probs, pred):
+            w.writerow([i, ("" if yy < 0 else int(yy)), f"{pr:.6f}",
+                        int(pd), L.IDX_TO_CLASS[int(pd)]])
+
+    result = {"n": len(items), "threshold": thr,
+              "calibration": "temperature(T=%.4f)" % T if use_cal else "none",
+              "labels_available": bool(has_labels)}
+
+    if has_labels:
+        metrics = L.point_metrics(labels, probs, thr)
+        ci = L.bootstrap_ci(labels, probs, thr, n_boot=args.n_boot, seed=args.boot_seed)
+        ci_keys = ["auroc", "sensitivity", "specificity", "ppv", "npv", "accuracy", "f1"]
+        result["metrics"] = metrics
+        result["metrics_95ci"] = {k: list(ci[k]) for k in ci_keys}
+        result["ci_method"] = f"stratified bootstrap, {args.n_boot} resamples, seed={args.boot_seed}"
+        print("=== EXTERNAL METRICS ===")
+        for k in ci_keys:
+            print(f"  {k:12s} {metrics[k]:.4f}  CI [{ci[k][0]:.4f}, {ci[k][1]:.4f}]")
+        print(f"  brier        {metrics['brier']:.4f}")
+        print(f"  ece          {metrics['ece']:.4f}")
+        print(f"  confusion    TN={metrics['tn']} FP={metrics['fp']} FN={metrics['fn']} TP={metrics['tp']}")
+    else:
+        print("No labels provided — wrote probabilities and predictions only.")
+
+    json.dump(result, open(out_dir / "external_metrics.json", "w"), indent=2)
+    print("Saved to", out_dir)
+
+
+if __name__ == "__main__":
+    main()

explore_data.py

@@ -0,0 +1,313 @@
+#!/usr/bin/env python
+"""
+Data exploration for the TN5000 thyroid-nodule classification dataset.
+
+Generates a publication-ready `data_exploration_report.md` plus figures in
+`results/figures/` covering: split/class counts, imbalance ratios, image
+dimension/channel/format summary, corrupt-image check, duplicate-image check
+(exact pixel hash, within and across splits), pixel-intensity distribution,
+representative benign/malignant image grids per split, and leakage analysis.
+
+Usage:
+    python explore_data.py --dataset_id Johnyquest7/TN5000-thyroid-nodule-classification \
+        --output_dir . [--data_dir /path/to/already/downloaded/TN5000]
+
+If --data_dir is not given, the dataset is downloaded from the Hub.
+The expected layout is <data_dir>/<Split>/<Class>/<id>.png with
+Split in {Train, Valid, Test} and Class in {Benign, Malignant}.
+"""
+import argparse
+import hashlib
+import json
+import os
+from collections import Counter
+from pathlib import Path
+
+import numpy as np
+from PIL import Image
+
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+
+SPLITS = ["Train", "Valid", "Test"]
+CLASSES = ["Benign", "Malignant"]  # index 0 = Benign, 1 = Malignant
+
+
+def get_data_dir(args):
+    if args.data_dir:
+        return Path(args.data_dir)
+    from huggingface_hub import snapshot_download
+    p = snapshot_download(
+        repo_id=args.dataset_id, repo_type="dataset",
+        local_dir=os.path.join(args.output_dir, "_tn5000_data"),
+        allow_patterns=["Train/**", "Valid/**", "Test/**"],
+    )
+    return Path(p)
+
+
+def list_images(data_dir):
+    out = {s: {c: [] for c in CLASSES} for s in SPLITS}
+    for s in SPLITS:
+        for c in CLASSES:
+            d = data_dir / s / c
+            if d.is_dir():
+                out[s][c] = sorted(d.glob("*.png"))
+    return out
+
+
+def md5_of_pixels(path):
+    with Image.open(path) as im:
+        arr = np.asarray(im.convert("RGB"))
+    return hashlib.md5(arr.tobytes()).hexdigest()
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--dataset_id", default="Johnyquest7/TN5000-thyroid-nodule-classification")
+    ap.add_argument("--data_dir", default=None)
+    ap.add_argument("--output_dir", default=".")
+    args = ap.parse_args()
+
+    out_dir = Path(args.output_dir)
+    fig_dir = out_dir / "results" / "figures"
+    tab_dir = out_dir / "results" / "tables"
+    fig_dir.mkdir(parents=True, exist_ok=True)
+    tab_dir.mkdir(parents=True, exist_ok=True)
+
+    data_dir = get_data_dir(args)
+    print("Data dir:", data_dir)
+    images = list_images(data_dir)
+
+    counts = {s: {c: len(images[s][c]) for c in CLASSES} for s in SPLITS}
+    rows = []
+    for s in SPLITS:
+        b, m = counts[s]["Benign"], counts[s]["Malignant"]
+        tot = b + m
+        mal_pct = 100.0 * m / tot if tot else 0.0
+        imb = m / b if b else float("inf")
+        rows.append((s, b, m, tot, mal_pct, imb))
+    tot_b = sum(counts[s]["Benign"] for s in SPLITS)
+    tot_m = sum(counts[s]["Malignant"] for s in SPLITS)
+    tot_all = tot_b + tot_m
+
+    corrupt = []
+    intensity = {s: [] for s in SPLITS}
+    all_dims = []
+    dim_summary, mode_summary = {}, {}
+    for s in SPLITS:
+        dims, modes = Counter(), Counter()
+        for c in CLASSES:
+            for p in images[s][c]:
+                try:
+                    with Image.open(p) as im:
+                        im.verify()
+                    with Image.open(p) as im:
+                        w, h = im.size
+                        modes[im.mode] += 1
+                        dims[(w, h)] += 1
+                        all_dims.append((w, h))
+                        g = np.asarray(im.convert("L"), dtype=np.float32)
+                        intensity[s].append(float(g.mean()))
+                except Exception as e:
+                    corrupt.append((s, c, str(p), repr(e)))
+        dim_summary[s] = dims
+        mode_summary[s] = modes
+
+    pixel_hash = {}
+    for s in SPLITS:
+        for c in CLASSES:
+            for p in images[s][c]:
+                try:
+                    hh = md5_of_pixels(p)
+                except Exception:
+                    continue
+                pixel_hash.setdefault(hh, []).append((s, c, p.name))
+    dup_within, dup_across, dup_labelconf = [], [], []
+    for hh, locs in pixel_hash.items():
+        if len(locs) > 1:
+            splits_involved = set(l[0] for l in locs)
+            classes_involved = set(l[1] for l in locs)
+            (dup_across if len(splits_involved) > 1 else dup_within).append((hh, locs))
+            if len(classes_involved) > 1:
+                dup_labelconf.append((hh, locs))
+
+    ids = {s: set() for s in SPLITS}
+    for s in SPLITS:
+        for c in CLASSES:
+            for p in images[s][c]:
+                ids[s].add(p.stem)
+    id_tr_va = ids["Train"] & ids["Valid"]
+    id_tr_te = ids["Train"] & ids["Test"]
+    id_va_te = ids["Valid"] & ids["Test"]
+
+    # figures
+    fig, ax = plt.subplots(figsize=(7, 4.2))
+    x = np.arange(len(SPLITS)); w = 0.38
+    ben = [counts[s]["Benign"] for s in SPLITS]; mal = [counts[s]["Malignant"] for s in SPLITS]
+    ax.bar(x - w / 2, ben, w, label="Benign (0)", color="#4C72B0")
+    ax.bar(x + w / 2, mal, w, label="Malignant (1)", color="#C44E52")
+    for i, (bb, mm) in enumerate(zip(ben, mal)):
+        ax.text(i - w / 2, bb + 5, str(bb), ha="center", va="bottom", fontsize=9)
+        ax.text(i + w / 2, mm + 5, str(mm), ha="center", va="bottom", fontsize=9)
+    ax.set_xticks(x); ax.set_xticklabels(SPLITS); ax.set_ylabel("Number of images")
+    ax.set_title("Class distribution by split"); ax.legend()
+    fig.tight_layout(); fig.savefig(fig_dir / "class_distribution.png", dpi=150); plt.close(fig)
+
+    fig, ax = plt.subplots(figsize=(7, 4.2))
+    for s in SPLITS:
+        ax.hist(intensity[s], bins=50, alpha=0.5, label=f"{s} (n={len(intensity[s])})", density=True)
+    ax.set_xlabel("Mean grayscale intensity (0-255)"); ax.set_ylabel("Density")
+    ax.set_title("Per-image mean pixel-intensity distribution"); ax.legend()
+    fig.tight_layout(); fig.savefig(fig_dir / "intensity_distribution.png", dpi=150); plt.close(fig)
+
+    rng = np.random.default_rng(42)
+    for s in SPLITS:
+        for c in CLASSES:
+            paths = images[s][c]
+            if not paths:
+                continue
+            sel = rng.choice(len(paths), size=min(8, len(paths)), replace=False)
+            ncol = 4; nrow = int(np.ceil(len(sel) / ncol))
+            fig, axes = plt.subplots(nrow, ncol, figsize=(2.2 * ncol, 2.2 * nrow))
+            axes = np.array(axes).reshape(-1)
+            for ax in axes:
+                ax.axis("off")
+            for ax, idx in zip(axes, sel):
+                with Image.open(paths[idx]) as im:
+                    ax.imshow(np.asarray(im.convert("RGB")))
+                ax.set_title(paths[idx].name, fontsize=7); ax.axis("off")
+            fig.suptitle(f"{s} / {c} (representative)", fontsize=11)
+            fig.tight_layout(); fig.savefig(fig_dir / f"grid_{s}_{c}.png", dpi=130); plt.close(fig)
+
+    import csv
+    with open(tab_dir / "class_distribution.csv", "w", newline="") as f:
+        wri = csv.writer(f)
+        wri.writerow(["split", "benign", "malignant", "total", "malignant_pct", "malignant_to_benign_ratio"])
+        for (s, b, m, tot, mal_pct, imb) in rows:
+            wri.writerow([s, b, m, tot, f"{mal_pct:.2f}", f"{imb:.3f}"])
+        wri.writerow(["Total", tot_b, tot_m, tot_all, f"{100.0*tot_m/tot_all:.2f}", f"{tot_m/tot_b:.3f}"])
+
+    intensity_stats = {s: (float(np.mean(intensity[s])), float(np.std(intensity[s])),
+                           float(np.min(intensity[s])), float(np.max(intensity[s])))
+                       for s in SPLITS}
+    all_dims_set = set(all_dims)
+    all_modes = set()
+    for s in SPLITS:
+        all_modes |= set(mode_summary[s].keys())
+
+    import datetime
+    now = datetime.datetime.now(datetime.timezone.utc).isoformat()
+    L = []
+    L.append("# Data Exploration Report — TN5000 Thyroid Nodule Classification\n")
+    L.append(f"- **Generated (UTC):** {now}")
+    L.append(f"- **Dataset:** `{args.dataset_id}`")
+    L.append("- **Source:** TN5000 (Yu et al., *Scientific Data*, 2025), cropped to nodule ROI, 224×224 PNG.")
+    L.append("- **Task:** Binary classification — 0 = Benign, 1 = Malignant. Positive class = Malignant.\n")
+
+    L.append("## 1. Number of images per split and class\n")
+    L.append("| Split | Benign (0) | Malignant (1) | Total | Malignant % | Malignant:Benign ratio |")
+    L.append("|-------|-----------:|--------------:|------:|------------:|------------------------:|")
+    for (s, b, m, tot, mal_pct, imb) in rows:
+        L.append(f"| {s} | {b} | {m} | {tot} | {mal_pct:.1f}% | {imb:.2f} : 1 |")
+    L.append(f"| **Total** | **{tot_b}** | **{tot_m}** | **{tot_all}** | **{100.0*tot_m/tot_all:.1f}%** | **{tot_m/tot_b:.2f} : 1** |\n")
+    L.append("![Class distribution](results/figures/class_distribution.png)\n")
+
+    L.append("## 2. Class imbalance\n")
+    L.append("All three splits are **malignant-majority** (~70–75% malignant), i.e. mild imbalance "
+             "(malignant:benign roughly 2.4–3.0 : 1), consistent across splits.\n")
+    L.append("- **Mitigation evaluated in training:** class-weighted `BCEWithLogitsLoss` "
+             "(`pos_weight = N_benign/N_malignant`), focal loss, and a weighted sampler "
+             "were all compared in the sweep; the final model uses focal loss (γ=1.0). "
+             "Because imbalance is mild and calibration matters, heavy reweighting was avoided.\n")
+
+    L.append("## 3. Image dimensions, channels, file format\n")
+    L.append(f"- **File format:** PNG (lossless) for all {tot_all} images.")
+    L.append(f"- **Dimensions observed:** {sorted(all_dims_set)} (expected single value (224, 224)).")
+    L.append(f"- **PIL modes observed:** {sorted(all_modes)} (RGB; grayscale replicated across 3 channels).")
+    L.append("\n| Split | Unique dimensions | Modes |")
+    L.append("|-------|-------------------|-------|")
+    for s in SPLITS:
+        L.append(f"| {s} | {dict(dim_summary[s])} | {dict(mode_summary[s])} |")
+    L.append("")
+
+    L.append("## 4. Missing / corrupt image check\n")
+    if corrupt:
+        L.append(f"- **{len(corrupt)} corrupt/unreadable images found:**")
+        for (s, c, p, e) in corrupt[:50]:
+            L.append(f"  - `{s}/{c}/{Path(p).name}` — {e}")
+    else:
+        L.append("- ✅ **No corrupt or unreadable images.** All images opened and decoded via PIL `verify()` + reload.")
+    L.append("")
+
+    L.append("## 5. Duplicate image check (exact pixel-content MD5)\n")
+    L.append(f"- Duplicate groups **within a single split:** {len(dup_within)}")
+    L.append(f"- Duplicate groups **spanning multiple splits (potential LEAKAGE):** {len(dup_across)}")
+    L.append(f"- Duplicate groups with **conflicting labels:** {len(dup_labelconf)}")
+    if dup_across:
+        L.append("\n  **Cross-split duplicate groups (first 50):**")
+        for hh, locs in dup_across[:50]:
+            L.append("  - " + ", ".join(f"{s}/{c}/{n}" for (s, c, n) in locs))
+    if not dup_across and not dup_labelconf:
+        L.append("\n- ✅ No cross-split pixel duplicates and no label conflicts detected.")
+    L.append("")
+
+    L.append("## 6. Data leakage analysis\n")
+    L.append("TN5000 assigns each image a **globally unique numeric ID**, preserved as the PNG filename. "
+             "Overlap of filename IDs across splits would indicate the same source image in two splits.\n")
+    L.append("| Pair | Shared filename IDs |")
+    L.append("|------|--------------------:|")
+    L.append(f"| Train ∩ Valid | {len(id_tr_va)} |")
+    L.append(f"| Train ∩ Test  | {len(id_tr_te)} |")
+    L.append(f"| Valid ∩ Test  | {len(id_va_te)} |")
+    if id_tr_va or id_tr_te or id_va_te:
+        L.append("\n- ⚠️ **Filename overlap detected** — review listed IDs.")
+    else:
+        L.append("\n- ✅ **No filename-ID overlap across splits.** Combined with the exact-pixel duplicate "
+                 "check above, there is no detectable leakage between Train, Valid, and Test.")
+    L.append("")
+
+    L.append("## 7. Pixel-intensity distribution\n")
+    L.append("| Split | Mean | Std | Min | Max |")
+    L.append("|-------|-----:|----:|----:|----:|")
+    for s in SPLITS:
+        mu, sd, mn, mx = intensity_stats[s]
+        L.append(f"| {s} | {mu:.1f} | {sd:.1f} | {mn:.1f} | {mx:.1f} |")
+    L.append("\n![Intensity distribution](results/figures/intensity_distribution.png)\n")
+    L.append("Mean per-image grayscale intensity distributions are **closely matched across splits**, "
+             "indicating consistent acquisition/preprocessing and no obvious distribution shift.\n")
+
+    L.append("## 8. Representative image grids\n")
+    for s in SPLITS:
+        for c in CLASSES:
+            L.append(f"**{s} / {c}**\n")
+            L.append(f"![{s} {c}](results/figures/grid_{s}_{c}.png)\n")
+
+    L.append("## 9. Train/Valid/Test separation statement\n")
+    L.append("> The Train, Valid, and Test folders provided in the dataset repository were kept "
+             "**strictly separate** throughout this experiment. The model was trained on **Train only**; "
+             "the **Valid** split was used for model selection, calibration, and threshold selection; "
+             "and the **Test** split was used **exactly once** for final locked evaluation after the model, "
+             "calibration, and decision threshold were frozen. The exact-pixel duplicate check and "
+             "filename-ID overlap check above confirm there is no detectable leakage between the three splits.\n")
+
+    (out_dir / "data_exploration_report.md").write_text("\n".join(L))
+
+    summary = {"generated_utc": now, "counts": counts,
+               "totals": {"benign": tot_b, "malignant": tot_m, "all": tot_all},
+               "corrupt_count": len(corrupt), "dup_within_groups": len(dup_within),
+               "dup_across_groups": len(dup_across), "dup_labelconflict_groups": len(dup_labelconf),
+               "filename_overlap": {"train_valid": len(id_tr_va), "train_test": len(id_tr_te),
+                                    "valid_test": len(id_va_te)},
+               "dims_observed": sorted([list(d) for d in all_dims_set]),
+               "modes_observed": sorted(list(all_modes)),
+               "intensity_stats": {s: {"mean": intensity_stats[s][0], "std": intensity_stats[s][1],
+                                       "min": intensity_stats[s][2], "max": intensity_stats[s][3]}
+                                   for s in SPLITS}}
+    (tab_dir / "data_exploration_summary.json").write_text(json.dumps(summary, indent=2))
+    print(json.dumps(summary, indent=2))
+    print("Report written to", out_dir / "data_exploration_report.md")
+
+
+if __name__ == "__main__":
+    main()

final_model.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a419778170c029203994df546d9ba69bfe20046cd4b3fcdcced5d4e56dc2338c
+size 44789067

finalize.py

@@ -0,0 +1,277 @@
+#!/usr/bin/env python
+"""
+Finalization pipeline (run ONCE after the sweep selects the best model by
+validation AUROC):
+
+  1. Load the best checkpoint (selected purely on val AUROC).
+  2. Recompute Valid + Test logits with deterministic eval preprocessing.
+  3. Calibrate on VALIDATION via temperature scaling; compare uncalibrated vs
+     calibrated (ECE, Brier, AUROC). Keep calibration only if it does not harm
+     discrimination (AUROC unchanged — temperature scaling is monotonic) and
+     improves/maintains calibration.
+  4. Select a LOCKED threshold on VALIDATION: highest-specificity threshold with
+     sensitivity >= 0.95 (primary); Youden's J reported as secondary.
+  5. Evaluate ONCE on TEST with calibrated probs + locked threshold.
+  6. Bootstrap 95% CIs (stratified, 2000 resamples, seed=42).
+  7. Save all figures (ROC, PR, calibration/reliability, confusion matrices),
+     tables (markdown + CSV), per-image prediction CSVs (valid + test),
+     calibration config, threshold config, preprocessing config.
+
+Outputs go to --output_dir (default model_repo/).
+"""
+import argparse
+import csv
+import json
+from pathlib import Path
+
+import numpy as np
+import matplotlib
+matplotlib.use("Agg")
+import matplotlib.pyplot as plt
+
+import thyroid_lib as L
+
+TARGET_SENS = 0.95
+N_BOOT = 2000
+BOOT_SEED = 42
+
+
+def load_model(ckpt_path, device):
+    import torch
+    ck = torch.load(ckpt_path, map_location="cpu", weights_only=False)
+    model, pp = L.build_model(ck["backbone"], freeze_stage=ck.get("freeze_stage", 0),
+                              dropout=ck.get("dropout", 0.0))
+    model.load_state_dict(ck["model_state"])
+    model.to(device).eval()
+    pp = L.PreprocessConfig.from_dict(ck["preprocess"])
+    return model, pp, ck
+
+
+def get_logits(model, data_dir, split, pp, device):
+    import torch
+    from torch.utils.data import DataLoader
+    ds = L.ThyroidImageFolder(Path(data_dir) / split, L.build_eval_transform(pp))
+    loader = DataLoader(ds, batch_size=64, shuffle=False, num_workers=4,
+                        pin_memory=(device == "cuda"))
+    logits, labels, ids = L.collect_logits(model, loader, device, amp=False)
+    return logits, labels, ids
+
+
+def save_per_image_csv(path, ids, labels, probs, thr):
+    pred = (np.asarray(probs) >= thr).astype(int)
+    with open(path, "w", newline="") as f:
+        w = csv.writer(f)
+        w.writerow(["image_id", "true_label", "true_class", "probability_malignant",
+                    "predicted_label", "predicted_class"])
+        for i, y, p, pr in zip(ids, labels, probs, pred):
+            w.writerow([i, int(y), L.IDX_TO_CLASS[int(y)], f"{p:.6f}",
+                        int(pr), L.IDX_TO_CLASS[int(pr)]])
+
+
+def plot_roc(y, p, path, title):
+    from sklearn.metrics import roc_curve, roc_auc_score
+    fpr, tpr, _ = roc_curve(y, p)
+    auc = roc_auc_score(y, p)
+    fig, ax = plt.subplots(figsize=(5, 5))
+    ax.plot(fpr, tpr, label=f"AUROC = {auc:.3f}", color="#C44E52")
+    ax.plot([0, 1], [0, 1], "--", color="gray")
+    ax.set_xlabel("1 - Specificity (FPR)"); ax.set_ylabel("Sensitivity (TPR)")
+    ax.set_title(title); ax.legend(loc="lower right")
+    fig.tight_layout(); fig.savefig(path, dpi=150); plt.close(fig)
+
+
+def plot_pr(y, p, path, title):
+    from sklearn.metrics import precision_recall_curve, average_precision_score
+    prec, rec, _ = precision_recall_curve(y, p)
+    ap = average_precision_score(y, p)
+    fig, ax = plt.subplots(figsize=(5, 5))
+    ax.plot(rec, prec, label=f"AP = {ap:.3f}", color="#4C72B0")
+    ax.set_xlabel("Recall (Sensitivity)"); ax.set_ylabel("Precision (PPV)")
+    ax.set_title(title); ax.legend(loc="lower left")
+    fig.tight_layout(); fig.savefig(path, dpi=150); plt.close(fig)
+
+
+def plot_reliability(y, p_uncal, p_cal, path, title):
+    from sklearn.calibration import calibration_curve
+    fig, ax = plt.subplots(figsize=(5.5, 5.5))
+    ax.plot([0, 1], [0, 1], "--", color="gray", label="Perfect calibration")
+    for p, lab, col in [(p_uncal, "Uncalibrated", "#888888"), (p_cal, "Temperature-scaled", "#C44E52")]:
+        fpos, mpred = calibration_curve(y, p, n_bins=10, strategy="uniform")
+        ece = L.expected_calibration_error(y, p)
+        br = L.brier(y, p)
+        ax.plot(mpred, fpos, "o-", color=col, label=f"{lab} (ECE={ece:.3f}, Brier={br:.3f})")
+    ax.set_xlabel("Mean predicted probability"); ax.set_ylabel("Observed frequency")
+    ax.set_title(title); ax.legend(loc="upper left", fontsize=8)
+    fig.tight_layout(); fig.savefig(path, dpi=150); plt.close(fig)
+
+
+def plot_confusion(cm, path, title, normalize=False):
+    cm = np.asarray(cm, dtype=float)
+    disp = cm.copy()
+    if normalize:
+        disp = cm / cm.sum(axis=1, keepdims=True).clip(min=1e-9)
+    fig, ax = plt.subplots(figsize=(4.8, 4.4))
+    im = ax.imshow(disp, cmap="Blues", vmin=0, vmax=disp.max())
+    ax.set_xticks([0, 1]); ax.set_yticks([0, 1])
+    ax.set_xticklabels(["Predicted benign", "Predicted malignant"])
+    ax.set_yticklabels(["True benign", "True malignant"])
+    for i in range(2):
+        for j in range(2):
+            txt = f"{int(cm[i,j])}" + (f"\n({disp[i,j]*100:.1f}%)" if normalize else "")
+            ax.text(j, i, txt, ha="center", va="center",
+                    color="white" if disp[i, j] > disp.max() / 2 else "black", fontsize=11)
+    ax.set_title(title)
+    fig.tight_layout(); fig.savefig(path, dpi=150); plt.close(fig)
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--ckpt", required=True)
+    ap.add_argument("--data_dir", default="/app/TN5000")
+    ap.add_argument("--output_dir", default="/app/model_repo")
+    ap.add_argument("--best_run_name", default="")
+    ap.add_argument("--best_val_auroc", default="")
+    args = ap.parse_args()
+
+    import torch
+    device = "cuda" if torch.cuda.is_available() else "cpu"
+    L.set_determinism(42, strict=True)
+
+    out = Path(args.output_dir)
+    res = out / "results"; figs = res / "figures"; tabs = res / "tables"
+    for d in [out / "configs", figs, tabs]:
+        d.mkdir(parents=True, exist_ok=True)
+
+    model, pp, ck = load_model(args.ckpt, device)
+
+    # ---- logits ----
+    val_logits, val_y, val_ids = get_logits(model, args.data_dir, "Valid", pp, device)
+    test_logits, test_y, test_ids = get_logits(model, args.data_dir, "Test", pp, device)
+
+    val_p_uncal = L.sigmoid(val_logits)
+    test_p_uncal = L.sigmoid(test_logits)
+
+    # ---- calibration: temperature scaling on VALIDATION ----
+    T = L.fit_temperature(val_logits, val_y)
+    val_p_cal = L.apply_temperature(val_logits, T)
+    test_p_cal = L.apply_temperature(test_logits, T)
+
+    from sklearn.metrics import roc_auc_score
+    cal_report = {
+        "method": "temperature_scaling",
+        "temperature": T,
+        "valid": {
+            "auroc_uncal": float(roc_auc_score(val_y, val_p_uncal)),
+            "auroc_cal": float(roc_auc_score(val_y, val_p_cal)),
+            "ece_uncal": L.expected_calibration_error(val_y, val_p_uncal),
+            "ece_cal": L.expected_calibration_error(val_y, val_p_cal),
+            "brier_uncal": L.brier(val_y, val_p_uncal),
+            "brier_cal": L.brier(val_y, val_p_cal),
+        },
+    }
+    # Decision: temperature scaling is monotonic -> AUROC unchanged. Use calibrated
+    # if ECE improves or is within tolerance; else fall back to uncalibrated.
+    use_calibrated = cal_report["valid"]["ece_cal"] <= cal_report["valid"]["ece_uncal"] + 1e-6
+    cal_report["use_calibrated"] = bool(use_calibrated)
+    val_p = val_p_cal if use_calibrated else val_p_uncal
+    test_p = test_p_cal if use_calibrated else test_p_uncal
+    L.save_json(cal_report, out / "configs" / "calibration.json")
+
+    # ---- threshold selection on VALIDATION (calibrated probs) ----
+    thr, sens_v, spec_v, achievable = L.threshold_for_sensitivity(val_y, val_p, TARGET_SENS)
+    yj_thr, yj_sens, yj_spec = L.youden_threshold(val_y, val_p)
+    thr_report = {
+        "primary_method": f"highest-specificity threshold with sensitivity >= {TARGET_SENS} on validation (calibrated probabilities)",
+        "locked_threshold": thr,
+        "valid_sensitivity_at_threshold": sens_v,
+        "valid_specificity_at_threshold": spec_v,
+        "target_sensitivity": TARGET_SENS,
+        "target_achievable": bool(achievable),
+        "secondary_youden": {"threshold": yj_thr, "sensitivity": yj_sens, "specificity": yj_spec},
+        "probabilities_used": "calibrated" if use_calibrated else "uncalibrated",
+    }
+    L.save_json(thr_report, out / "configs" / "threshold.json")
+
+    # ---- preprocessing config (locked) ----
+    L.save_json({**pp.to_dict(), "positive_class": "Malignant", "positive_index": 1,
+                 "note": "Deterministic eval/inference preprocessing. No augmentation."},
+                out / "configs" / "preprocess.json")
+
+    # ---- VALIDATION metrics at locked threshold ----
+    val_metrics = L.point_metrics(val_y, val_p, thr)
+
+    # ---- TEST metrics (locked) ----
+    test_metrics = L.point_metrics(test_y, test_p, thr)
+    test_ci = L.bootstrap_ci(test_y, test_p, thr, n_boot=N_BOOT, seed=BOOT_SEED)
+
+    # ---- per-image CSVs ----
+    save_per_image_csv(res / "valid_predictions.csv", val_ids, val_y, val_p, thr)
+    save_per_image_csv(res / "test_predictions.csv", test_ids, test_y, test_p, thr)
+
+    # ---- figures ----
+    plot_roc(test_y, test_p, figs / "test_roc.png", "ROC — Test set")
+    plot_pr(test_y, test_p, figs / "test_pr.png", "Precision-Recall — Test set")
+    plot_reliability(val_y, val_p_uncal, val_p_cal, figs / "valid_calibration.png",
+                     "Reliability diagram — Validation")
+    plot_reliability(test_y, test_p_uncal, test_p_cal, figs / "test_calibration.png",
+                     "Reliability diagram — Test")
+    cm = np.array([[test_metrics["tn"], test_metrics["fp"]],
+                   [test_metrics["fn"], test_metrics["tp"]]])
+    plot_confusion(cm, figs / "test_confusion_counts.png", "Confusion matrix (counts) — Test")
+    plot_confusion(cm, figs / "test_confusion_normalized.png",
+                   "Confusion matrix (row-normalized) — Test", normalize=True)
+
+    # ---- metrics table (markdown + csv) with CIs ----
+    ci_keys = ["auroc", "sensitivity", "specificity", "ppv", "npv", "accuracy", "f1"]
+    md = ["# Final Test Metrics (locked model + locked threshold)\n",
+          f"- Selected run: {args.best_run_name}  |  selection val AUROC: {args.best_val_auroc}",
+          f"- Backbone: {ck['backbone']}  |  Calibration: temperature scaling (T={T:.4f}, "
+          f"{'used' if use_calibrated else 'not used'})",
+          f"- Locked threshold (val sens>={TARGET_SENS}): {thr:.4f}  |  probabilities: "
+          f"{'calibrated' if use_calibrated else 'uncalibrated'}",
+          f"- CI method: stratified bootstrap, {N_BOOT} resamples, seed={BOOT_SEED}\n",
+          "| Metric | Point estimate | 95% CI |",
+          "|--------|---------------:|:------:|"]
+    rows_csv = [["metric", "point_estimate", "ci_low", "ci_high"]]
+    for k in ci_keys:
+        pe = test_metrics[k]; lo, hi = test_ci[k]
+        md.append(f"| {k.upper()} | {pe:.4f} | [{lo:.4f}, {hi:.4f}] |")
+        rows_csv.append([k, f"{pe:.6f}", f"{lo:.6f}", f"{hi:.6f}"])
+    for k in ["brier", "ece"]:
+        md.append(f"| {k.upper()} | {test_metrics[k]:.4f} | — |")
+        rows_csv.append([k, f"{test_metrics[k]:.6f}", "", ""])
+    md.append(f"\n**Confusion matrix (Test):** TN={test_metrics['tn']}, FP={test_metrics['fp']}, "
+              f"FN={test_metrics['fn']}, TP={test_metrics['tp']}\n")
+    (tabs / "test_metrics_with_ci.md").write_text("\n".join(md))
+    with open(tabs / "test_metrics_with_ci.csv", "w", newline="") as f:
+        csv.writer(f).writerows(rows_csv)
+
+    # ---- consolidated results json ----
+    final = {
+        "selected_run": args.best_run_name,
+        "selection_val_auroc": args.best_val_auroc,
+        "backbone": ck["backbone"],
+        "preprocess": pp.to_dict(),
+        "calibration": cal_report,
+        "threshold": thr_report,
+        "valid_metrics_at_locked_threshold": val_metrics,
+        "test_metrics_at_locked_threshold": test_metrics,
+        "test_metrics_95ci": {k: list(test_ci[k]) for k in ci_keys},
+        "ci_method": f"stratified bootstrap, {N_BOOT} resamples, seed={BOOT_SEED}",
+    }
+    L.save_json(final, res / "final_results.json")
+
+    print("=== CALIBRATION ===")
+    print(json.dumps(cal_report, indent=2))
+    print("=== THRESHOLD ===")
+    print(json.dumps(thr_report, indent=2))
+    print("=== TEST METRICS ===")
+    for k in ci_keys:
+        print(f"  {k:12s} {test_metrics[k]:.4f}  CI [{test_ci[k][0]:.4f}, {test_ci[k][1]:.4f}]")
+    print(f"  brier        {test_metrics['brier']:.4f}")
+    print(f"  ece          {test_metrics['ece']:.4f}")
+    print("Saved to", out)
+
+
+if __name__ == "__main__":
+    main()

requirements.txt

@@ -0,0 +1,15 @@
+# Pinned to the exact versions used to produce the published results
+# (NVIDIA A10G, CUDA 13.0, Python 3.12). torch/torchvision built with +cu130;
+# install the CUDA build matching your system from https://pytorch.org if needed.
+torch==2.12.0
+torchvision==0.27.0
+timm==1.0.27
+scikit-learn==1.9.0
+numpy==2.4.6
+pillow==12.2.0
+matplotlib==3.10.9
+pyyaml==6.0.3
+trackio==0.26.0
+huggingface_hub==1.17.0
+# optional: only needed if you enable the 'clahe' augmentation ablation
+opencv-python-headless>=4.8

results/final_results.json

@@ -0,0 +1,115 @@
+{
+  "selected_run": "c12_loss_focal",
+  "selection_val_auroc": "0.9756",
+  "backbone": "timm:resnet18.a1_in1k",
+  "preprocess": {
+    "image_size": 224,
+    "mean": [
+      0.485,
+      0.456,
+      0.406
+    ],
+    "std": [
+      0.229,
+      0.224,
+      0.225
+    ],
+    "interpolation": "bicubic"
+  },
+  "calibration": {
+    "method": "temperature_scaling",
+    "temperature": 0.5645509958267212,
+    "valid": {
+      "auroc_uncal": 0.975616,
+      "auroc_cal": 0.975616,
+      "ece_uncal": 0.08334361362457275,
+      "ece_cal": 0.03078642554581164,
+      "brier_uncal": 0.05917946249246597,
+      "brier_cal": 0.05246718227863312
+    },
+    "use_calibrated": true
+  },
+  "threshold": {
+    "primary_method": "highest-specificity threshold with sensitivity >= 0.95 on validation (calibrated probabilities)",
+    "locked_threshold": 0.7113139033317566,
+    "valid_sensitivity_at_threshold": 0.952,
+    "valid_specificity_at_threshold": 0.896,
+    "target_sensitivity": 0.95,
+    "target_achievable": true,
+    "secondary_youden": {
+      "threshold": 0.7113139033317566,
+      "sensitivity": 0.952,
+      "specificity": 0.896
+    },
+    "probabilities_used": "calibrated"
+  },
+  "valid_metrics_at_locked_threshold": {
+    "auroc": 0.975616,
+    "accuracy": 0.938,
+    "sensitivity": 0.952,
+    "specificity": 0.896,
+    "ppv": 0.9648648648648649,
+    "npv": 0.8615384615384616,
+    "f1": 0.9583892617449664,
+    "brier": 0.05246718227863312,
+    "ece": 0.03078642554581164,
+    "tp": 357,
+    "tn": 112,
+    "fp": 13,
+    "fn": 18,
+    "threshold": 0.7113139033317566,
+    "n": 500,
+    "n_pos": 375,
+    "n_neg": 125
+  },
+  "test_metrics_at_locked_threshold": {
+    "auroc": 0.9371436998764233,
+    "accuracy": 0.875,
+    "sensitivity": 0.9042407660738714,
+    "specificity": 0.7955390334572491,
+    "ppv": 0.9231843575418994,
+    "npv": 0.7535211267605634,
+    "f1": 0.9136143745680718,
+    "brier": 0.08228471130132675,
+    "ece": 0.03143832388520238,
+    "tp": 661,
+    "tn": 214,
+    "fp": 55,
+    "fn": 70,
+    "threshold": 0.7113139033317566,
+    "n": 1000,
+    "n_pos": 731,
+    "n_neg": 269
+  },
+  "test_metrics_95ci": {
+    "auroc": [
+      0.9202044101119309,
+      0.9528172183544465
+    ],
+    "sensitivity": [
+      0.8823529411764706,
+      0.9247606019151847
+    ],
+    "specificity": [
+      0.7434944237918215,
+      0.8438661710037175
+    ],
+    "ppv": [
+      0.9054223401271771,
+      0.9400883200831714
+    ],
+    "npv": [
+      0.7122702943800179,
+      0.797906592520954
+    ],
+    "accuracy": [
+      0.854,
+      0.895
+    ],
+    "f1": [
+      0.8990953375086986,
+      0.9277781562216167
+    ]
+  },
+  "ci_method": "stratified bootstrap, 2000 resamples, seed=42"
+}

results/tables/class_distribution.csv

@@ -0,0 +1,5 @@
+split,benign,malignant,total,malignant_pct,malignant_to_benign_ratio
+Train,1032,2468,3500,70.51,2.391
+Valid,125,375,500,75.00,3.000
+Test,269,731,1000,73.10,2.717
+Total,1426,3574,5000,71.48,2.506

results/tables/data_exploration_summary.json

@@ -0,0 +1,60 @@
+{
+  "generated_utc": "2026-06-05T03:24:51.156145+00:00",
+  "counts": {
+    "Train": {
+      "Benign": 1032,
+      "Malignant": 2468
+    },
+    "Valid": {
+      "Benign": 125,
+      "Malignant": 375
+    },
+    "Test": {
+      "Benign": 269,
+      "Malignant": 731
+    }
+  },
+  "totals": {
+    "benign": 1426,
+    "malignant": 3574,
+    "all": 5000
+  },
+  "corrupt_count": 0,
+  "dup_within_groups": 0,
+  "dup_across_groups": 0,
+  "dup_labelconflict_groups": 0,
+  "filename_overlap": {
+    "train_valid": 0,
+    "train_test": 0,
+    "valid_test": 0
+  },
+  "dims_observed": [
+    [
+      224,
+      224
+    ]
+  ],
+  "modes_observed": [
+    "RGB"
+  ],
+  "intensity_stats": {
+    "Train": {
+      "mean": 81.51711741311209,
+      "std": 19.637604872156206,
+      "min": 27.07162857055664,
+      "max": 163.4593963623047
+    },
+    "Valid": {
+      "mean": 81.32348302459717,
+      "std": 19.410170086309197,
+      "min": 35.048988342285156,
+      "max": 171.485107421875
+    },
+    "Test": {
+      "mean": 81.36260342407226,
+      "std": 19.13009627746057,
+      "min": 33.38518524169922,
+      "max": 150.60565185546875
+    }
+  }
+}

results/tables/sweep_leaderboard.json

@@ -0,0 +1,311 @@
+[
+  {
+    "name": "c12_loss_focal",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0002,
+      "weight_decay": 0.0001,
+      "batch_size": 32,
+      "aug_policy": "medical_default",
+      "imbalance": "none",
+      "freeze_stage": 0,
+      "loss": "focal",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0,
+      "focal_gamma": 1.0
+    },
+    "best_val_auroc": 0.9756053333333333,
+    "best_epoch": 6,
+    "out_dir": "/app/sweep_runs/c12_loss_focal",
+    "returncode": 0
+  },
+  {
+    "name": "c09_imb_none",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0002,
+      "weight_decay": 0.0001,
+      "batch_size": 32,
+      "aug_policy": "medical_default",
+      "imbalance": "none",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9739093333333334,
+    "best_epoch": 6,
+    "out_dir": "/app/sweep_runs/c09_imb_none",
+    "returncode": 0
+  },
+  {
+    "name": "c01_backbone_torchvision",
+    "config": {
+      "backbone": "torchvision",
+      "lr": 0.0002,
+      "weight_decay": 0.0001,
+      "batch_size": 32,
+      "aug_policy": "medical_default",
+      "imbalance": "pos_weight",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9730986666666667,
+    "best_epoch": 7,
+    "out_dir": "/app/sweep_runs/c01_backbone_torchvision",
+    "returncode": 0
+  },
+  {
+    "name": "c03_lr_1e-4",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0001,
+      "weight_decay": 0.0001,
+      "batch_size": 32,
+      "aug_policy": "medical_default",
+      "imbalance": "pos_weight",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9720533333333334,
+    "best_epoch": 11,
+    "out_dir": "/app/sweep_runs/c03_lr_1e-4",
+    "returncode": 0
+  },
+  {
+    "name": "c06_bs_64",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0002,
+      "weight_decay": 0.0001,
+      "batch_size": 64,
+      "aug_policy": "medical_default",
+      "imbalance": "pos_weight",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9717439999999999,
+    "best_epoch": 8,
+    "out_dir": "/app/sweep_runs/c06_bs_64",
+    "returncode": 0
+  },
+  {
+    "name": "c05_wd_1e-3",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0002,
+      "weight_decay": 0.001,
+      "batch_size": 32,
+      "aug_policy": "medical_default",
+      "imbalance": "pos_weight",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9712106666666667,
+    "best_epoch": 9,
+    "out_dir": "/app/sweep_runs/c05_wd_1e-3",
+    "returncode": 0
+  },
+  {
+    "name": "c00_center_a1",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0002,
+      "weight_decay": 0.0001,
+      "batch_size": 32,
+      "aug_policy": "medical_default",
+      "imbalance": "pos_weight",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9711786666666667,
+    "best_epoch": 6,
+    "out_dir": "/app/sweep_runs/c00_center_a1",
+    "returncode": 0
+  },
+  {
+    "name": "c10_imb_sampler",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0002,
+      "weight_decay": 0.0001,
+      "batch_size": 32,
+      "aug_policy": "medical_default",
+      "imbalance": "sampler",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9696640000000001,
+    "best_epoch": 13,
+    "out_dir": "/app/sweep_runs/c10_imb_sampler",
+    "returncode": 0
+  },
+  {
+    "name": "c02_backbone_a2",
+    "config": {
+      "backbone": "timm:resnet18.a2_in1k",
+      "lr": 0.0002,
+      "weight_decay": 0.0001,
+      "batch_size": 32,
+      "aug_policy": "medical_default",
+      "imbalance": "pos_weight",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9693440000000001,
+    "best_epoch": 6,
+    "out_dir": "/app/sweep_runs/c02_backbone_a2",
+    "returncode": 0
+  },
+  {
+    "name": "c04_lr_5e-4",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0005,
+      "weight_decay": 0.0001,
+      "batch_size": 32,
+      "aug_policy": "medical_default",
+      "imbalance": "pos_weight",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9675306666666666,
+    "best_epoch": 9,
+    "out_dir": "/app/sweep_runs/c04_lr_5e-4",
+    "returncode": 0
+  },
+  {
+    "name": "c11_freeze1",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0002,
+      "weight_decay": 0.0001,
+      "batch_size": 32,
+      "aug_policy": "medical_default",
+      "imbalance": "pos_weight",
+      "freeze_stage": 1,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9671786666666665,
+    "best_epoch": 6,
+    "out_dir": "/app/sweep_runs/c11_freeze1",
+    "returncode": 0
+  },
+  {
+    "name": "c13_lr1e-4_wd1e-3_drop",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0001,
+      "weight_decay": 0.001,
+      "batch_size": 32,
+      "aug_policy": "medical_default",
+      "imbalance": "pos_weight",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.2
+    },
+    "best_val_auroc": 0.965696,
+    "best_epoch": 11,
+    "out_dir": "/app/sweep_runs/c13_lr1e-4_wd1e-3_drop",
+    "returncode": 0
+  },
+  {
+    "name": "c07_aug_flip_only",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0002,
+      "weight_decay": 0.0001,
+      "batch_size": 32,
+      "aug_policy": "flip_only",
+      "imbalance": "pos_weight",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9637226666666667,
+    "best_epoch": 6,
+    "out_dir": "/app/sweep_runs/c07_aug_flip_only",
+    "returncode": 0
+  },
+  {
+    "name": "c08_aug_strong",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0002,
+      "weight_decay": 0.0001,
+      "batch_size": 32,
+      "aug_policy": "medical_strong",
+      "imbalance": "pos_weight",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9609386666666667,
+    "best_epoch": 8,
+    "out_dir": "/app/sweep_runs/c08_aug_strong",
+    "returncode": 0
+  }
+]

results/tables/sweep_results.json

@@ -0,0 +1,311 @@
+[
+  {
+    "name": "c00_center_a1",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0002,
+      "weight_decay": 0.0001,
+      "batch_size": 32,
+      "aug_policy": "medical_default",
+      "imbalance": "pos_weight",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9711786666666667,
+    "best_epoch": 6,
+    "out_dir": "/app/sweep_runs/c00_center_a1",
+    "returncode": 0
+  },
+  {
+    "name": "c01_backbone_torchvision",
+    "config": {
+      "backbone": "torchvision",
+      "lr": 0.0002,
+      "weight_decay": 0.0001,
+      "batch_size": 32,
+      "aug_policy": "medical_default",
+      "imbalance": "pos_weight",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9730986666666667,
+    "best_epoch": 7,
+    "out_dir": "/app/sweep_runs/c01_backbone_torchvision",
+    "returncode": 0
+  },
+  {
+    "name": "c02_backbone_a2",
+    "config": {
+      "backbone": "timm:resnet18.a2_in1k",
+      "lr": 0.0002,
+      "weight_decay": 0.0001,
+      "batch_size": 32,
+      "aug_policy": "medical_default",
+      "imbalance": "pos_weight",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9693440000000001,
+    "best_epoch": 6,
+    "out_dir": "/app/sweep_runs/c02_backbone_a2",
+    "returncode": 0
+  },
+  {
+    "name": "c03_lr_1e-4",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0001,
+      "weight_decay": 0.0001,
+      "batch_size": 32,
+      "aug_policy": "medical_default",
+      "imbalance": "pos_weight",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9720533333333334,
+    "best_epoch": 11,
+    "out_dir": "/app/sweep_runs/c03_lr_1e-4",
+    "returncode": 0
+  },
+  {
+    "name": "c04_lr_5e-4",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0005,
+      "weight_decay": 0.0001,
+      "batch_size": 32,
+      "aug_policy": "medical_default",
+      "imbalance": "pos_weight",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9675306666666666,
+    "best_epoch": 9,
+    "out_dir": "/app/sweep_runs/c04_lr_5e-4",
+    "returncode": 0
+  },
+  {
+    "name": "c05_wd_1e-3",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0002,
+      "weight_decay": 0.001,
+      "batch_size": 32,
+      "aug_policy": "medical_default",
+      "imbalance": "pos_weight",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9712106666666667,
+    "best_epoch": 9,
+    "out_dir": "/app/sweep_runs/c05_wd_1e-3",
+    "returncode": 0
+  },
+  {
+    "name": "c06_bs_64",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0002,
+      "weight_decay": 0.0001,
+      "batch_size": 64,
+      "aug_policy": "medical_default",
+      "imbalance": "pos_weight",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9717439999999999,
+    "best_epoch": 8,
+    "out_dir": "/app/sweep_runs/c06_bs_64",
+    "returncode": 0
+  },
+  {
+    "name": "c07_aug_flip_only",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0002,
+      "weight_decay": 0.0001,
+      "batch_size": 32,
+      "aug_policy": "flip_only",
+      "imbalance": "pos_weight",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9637226666666667,
+    "best_epoch": 6,
+    "out_dir": "/app/sweep_runs/c07_aug_flip_only",
+    "returncode": 0
+  },
+  {
+    "name": "c08_aug_strong",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0002,
+      "weight_decay": 0.0001,
+      "batch_size": 32,
+      "aug_policy": "medical_strong",
+      "imbalance": "pos_weight",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9609386666666667,
+    "best_epoch": 8,
+    "out_dir": "/app/sweep_runs/c08_aug_strong",
+    "returncode": 0
+  },
+  {
+    "name": "c09_imb_none",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0002,
+      "weight_decay": 0.0001,
+      "batch_size": 32,
+      "aug_policy": "medical_default",
+      "imbalance": "none",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9739093333333334,
+    "best_epoch": 6,
+    "out_dir": "/app/sweep_runs/c09_imb_none",
+    "returncode": 0
+  },
+  {
+    "name": "c10_imb_sampler",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0002,
+      "weight_decay": 0.0001,
+      "batch_size": 32,
+      "aug_policy": "medical_default",
+      "imbalance": "sampler",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9696640000000001,
+    "best_epoch": 13,
+    "out_dir": "/app/sweep_runs/c10_imb_sampler",
+    "returncode": 0
+  },
+  {
+    "name": "c11_freeze1",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0002,
+      "weight_decay": 0.0001,
+      "batch_size": 32,
+      "aug_policy": "medical_default",
+      "imbalance": "pos_weight",
+      "freeze_stage": 1,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0
+    },
+    "best_val_auroc": 0.9671786666666665,
+    "best_epoch": 6,
+    "out_dir": "/app/sweep_runs/c11_freeze1",
+    "returncode": 0
+  },
+  {
+    "name": "c12_loss_focal",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0002,
+      "weight_decay": 0.0001,
+      "batch_size": 32,
+      "aug_policy": "medical_default",
+      "imbalance": "none",
+      "freeze_stage": 0,
+      "loss": "focal",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.0,
+      "focal_gamma": 1.0
+    },
+    "best_val_auroc": 0.9756053333333333,
+    "best_epoch": 6,
+    "out_dir": "/app/sweep_runs/c12_loss_focal",
+    "returncode": 0
+  },
+  {
+    "name": "c13_lr1e-4_wd1e-3_drop",
+    "config": {
+      "backbone": "timm:resnet18.a1_in1k",
+      "lr": 0.0001,
+      "weight_decay": 0.001,
+      "batch_size": 32,
+      "aug_policy": "medical_default",
+      "imbalance": "pos_weight",
+      "freeze_stage": 0,
+      "loss": "bce",
+      "optimizer": "adamw",
+      "scheduler": "cosine",
+      "epochs": 40,
+      "early_stop_patience": 8,
+      "dropout": 0.2
+    },
+    "best_val_auroc": 0.965696,
+    "best_epoch": 11,
+    "out_dir": "/app/sweep_runs/c13_lr1e-4_wd1e-3_drop",
+    "returncode": 0
+  }
+]

results/tables/test_metrics_with_ci.csv

@@ -0,0 +1,10 @@
+metric,point_estimate,ci_low,ci_high
+auroc,0.937144,0.920204,0.952817
+sensitivity,0.904241,0.882353,0.924761
+specificity,0.795539,0.743494,0.843866
+ppv,0.923184,0.905422,0.940088
+npv,0.753521,0.712270,0.797907
+accuracy,0.875000,0.854000,0.895000
+f1,0.913614,0.899095,0.927778
+brier,0.082285,,
+ece,0.031438,,

results/tables/test_metrics_with_ci.md

@@ -0,0 +1,20 @@
+# Final Test Metrics (locked model + locked threshold)
+
+- Selected run: c12_loss_focal  |  selection val AUROC: 0.9756
+- Backbone: timm:resnet18.a1_in1k  |  Calibration: temperature scaling (T=0.5646, used)
+- Locked threshold (val sens>=0.95): 0.7113  |  probabilities: calibrated
+- CI method: stratified bootstrap, 2000 resamples, seed=42
+
+| Metric | Point estimate | 95% CI |
+|--------|---------------:|:------:|
+| AUROC | 0.9371 | [0.9202, 0.9528] |
+| SENSITIVITY | 0.9042 | [0.8824, 0.9248] |
+| SPECIFICITY | 0.7955 | [0.7435, 0.8439] |
+| PPV | 0.9232 | [0.9054, 0.9401] |
+| NPV | 0.7535 | [0.7123, 0.7979] |
+| ACCURACY | 0.8750 | [0.8540, 0.8950] |
+| F1 | 0.9136 | [0.8991, 0.9278] |
+| BRIER | 0.0823 | — |
+| ECE | 0.0314 | — |
+
+**Confusion matrix (Test):** TN=214, FP=55, FN=70, TP=661

results/test_predictions.csv

@@ -0,0 +1,1001 @@
+image_id,true_label,true_class,probability_malignant,predicted_label,predicted_class
+000001,0,Benign,0.376765,0,Benign
+000006,0,Benign,0.018830,0,Benign
+000009,0,Benign,0.516381,0,Benign
+000011,0,Benign,0.011112,0,Benign
+000016,0,Benign,0.016702,0,Benign
+000025,0,Benign,0.572745,0,Benign
+000035,0,Benign,0.045335,0,Benign
+000040,0,Benign,0.387496,0,Benign
+000044,0,Benign,0.532899,0,Benign
+000046,0,Benign,0.932031,1,Malignant
+000048,0,Benign,0.997977,1,Malignant
+000054,0,Benign,0.960895,1,Malignant
+000060,0,Benign,0.025679,0,Benign
+000083,0,Benign,0.948330,1,Malignant
+000086,0,Benign,0.003489,0,Benign
+000093,0,Benign,0.135352,0,Benign
+000105,0,Benign,0.092530,0,Benign
+000120,0,Benign,0.103708,0,Benign
+000123,0,Benign,0.068463,0,Benign
+000126,0,Benign,0.251543,0,Benign
+000131,0,Benign,0.780844,1,Malignant
+000137,0,Benign,0.925188,1,Malignant
+000149,0,Benign,0.019343,0,Benign
+000153,0,Benign,0.062035,0,Benign
+000154,0,Benign,0.090431,0,Benign
+000161,0,Benign,0.834368,1,Malignant
+000169,0,Benign,0.386561,0,Benign
+000174,0,Benign,0.879822,1,Malignant
+000177,0,Benign,0.786450,1,Malignant
+000184,0,Benign,0.034857,0,Benign
+000190,0,Benign,0.463699,0,Benign
+000197,0,Benign,0.021787,0,Benign
+000202,0,Benign,0.902157,1,Malignant
+000212,0,Benign,0.056520,0,Benign
+000222,0,Benign,0.439280,0,Benign
+000223,0,Benign,0.586564,0,Benign
+000224,0,Benign,0.935356,1,Malignant
+000225,0,Benign,0.972133,1,Malignant
+000228,0,Benign,0.506554,0,Benign
+000244,0,Benign,0.970782,1,Malignant
+000246,0,Benign,0.079815,0,Benign
+000276,0,Benign,0.007983,0,Benign
+000300,0,Benign,0.402881,0,Benign
+000304,0,Benign,0.050980,0,Benign
+000307,0,Benign,0.103203,0,Benign
+000318,0,Benign,0.750014,1,Malignant
+000323,0,Benign,0.100095,0,Benign
+000337,0,Benign,0.706239,0,Benign
+000339,0,Benign,0.869558,1,Malignant
+000342,0,Benign,0.042517,0,Benign
+000349,0,Benign,0.835838,1,Malignant
+000361,0,Benign,0.180529,0,Benign
+000363,0,Benign,0.101132,0,Benign
+000366,0,Benign,0.556296,0,Benign
+000371,0,Benign,0.993220,1,Malignant
+000379,0,Benign,0.003288,0,Benign
+000382,0,Benign,0.885058,1,Malignant
+000385,0,Benign,0.916005,1,Malignant
+000387,0,Benign,0.148785,0,Benign
+000400,0,Benign,0.691709,0,Benign
+000405,0,Benign,0.366323,0,Benign
+000410,0,Benign,0.004203,0,Benign
+000413,0,Benign,0.156769,0,Benign
+000415,0,Benign,0.502721,0,Benign
+000421,0,Benign,0.000583,0,Benign
+000430,0,Benign,0.010474,0,Benign
+000432,0,Benign,0.203537,0,Benign
+000433,0,Benign,0.081840,0,Benign
+000438,0,Benign,0.364286,0,Benign
+000445,0,Benign,0.799353,1,Malignant
+000451,0,Benign,0.647562,0,Benign
+000455,0,Benign,0.154093,0,Benign
+000459,0,Benign,0.011427,0,Benign
+000460,0,Benign,0.008466,0,Benign
+000461,0,Benign,0.014595,0,Benign
+000468,0,Benign,0.125134,0,Benign
+000487,0,Benign,0.008266,0,Benign
+000496,0,Benign,0.022297,0,Benign
+000519,0,Benign,0.108617,0,Benign
+000537,0,Benign,0.018078,0,Benign
+000541,0,Benign,0.017684,0,Benign
+000545,0,Benign,0.007944,0,Benign
+000546,0,Benign,0.078626,0,Benign
+000554,0,Benign,0.012844,0,Benign
+000566,0,Benign,0.013038,0,Benign
+000573,0,Benign,0.060264,0,Benign
+000576,0,Benign,0.128162,0,Benign
+000579,0,Benign,0.001836,0,Benign
+000582,0,Benign,0.165779,0,Benign
+000601,0,Benign,0.046323,0,Benign
+000605,0,Benign,0.930099,1,Malignant
+000606,0,Benign,0.978089,1,Malignant
+000607,0,Benign,0.797922,1,Malignant
+000623,0,Benign,0.892158,1,Malignant
+000630,0,Benign,0.219395,0,Benign
+000634,0,Benign,0.897862,1,Malignant
+000635,0,Benign,0.832107,1,Malignant
+000637,0,Benign,0.829636,1,Malignant
+000641,0,Benign,0.029093,0,Benign
+000649,0,Benign,0.095892,0,Benign
+000651,0,Benign,0.558030,0,Benign
+000652,0,Benign,0.074146,0,Benign
+000669,0,Benign,0.007034,0,Benign
+001729,0,Benign,0.034281,0,Benign
+001732,0,Benign,0.721947,1,Malignant
+001750,0,Benign,0.070680,0,Benign
+001751,0,Benign,0.048564,0,Benign
+001762,0,Benign,0.156416,0,Benign
+001763,0,Benign,0.029954,0,Benign
+001764,0,Benign,0.151097,0,Benign
+001765,0,Benign,0.010756,0,Benign
+001766,0,Benign,0.900002,1,Malignant
+001769,0,Benign,0.142776,0,Benign
+001770,0,Benign,0.628850,0,Benign
+001771,0,Benign,0.003992,0,Benign
+001773,0,Benign,0.336814,0,Benign
+001775,0,Benign,0.001492,0,Benign
+001776,0,Benign,0.005223,0,Benign
+001781,0,Benign,0.186179,0,Benign
+001785,0,Benign,0.025104,0,Benign
+001786,0,Benign,0.749363,1,Malignant
+001790,0,Benign,0.640222,0,Benign
+001795,0,Benign,0.008402,0,Benign
+001807,0,Benign,0.013915,0,Benign
+001812,0,Benign,0.032009,0,Benign
+001819,0,Benign,0.408997,0,Benign
+001823,0,Benign,0.015756,0,Benign
+001831,0,Benign,0.019138,0,Benign
+001834,0,Benign,0.013611,0,Benign
+001847,0,Benign,0.004663,0,Benign
+001858,0,Benign,0.066907,0,Benign
+001871,0,Benign,0.405174,0,Benign
+001874,0,Benign,0.780143,1,Malignant
+001875,0,Benign,0.089945,0,Benign
+001876,0,Benign,0.982950,1,Malignant
+001878,0,Benign,0.078918,0,Benign
+001879,0,Benign,0.068803,0,Benign
+001885,0,Benign,0.059237,0,Benign
+001886,0,Benign,0.161767,0,Benign
+001891,0,Benign,0.864839,1,Malignant
+001892,0,Benign,0.023491,0,Benign
+001898,0,Benign,0.002351,0,Benign
+001908,0,Benign,0.139884,0,Benign
+001918,0,Benign,0.958023,1,Malignant
+001919,0,Benign,0.115347,0,Benign
+001920,0,Benign,0.080368,0,Benign
+001928,0,Benign,0.000542,0,Benign
+001937,0,Benign,0.183525,0,Benign
+001941,0,Benign,0.004168,0,Benign
+001942,0,Benign,0.057188,0,Benign
+001944,0,Benign,0.000413,0,Benign
+002339,0,Benign,0.122457,0,Benign
+002340,0,Benign,0.001082,0,Benign
+002343,0,Benign,0.053878,0,Benign
+002356,0,Benign,0.003285,0,Benign
+002359,0,Benign,0.309704,0,Benign
+002362,0,Benign,0.871511,1,Malignant
+002364,0,Benign,0.273575,0,Benign
+002365,0,Benign,0.163321,0,Benign
+002366,0,Benign,0.402194,0,Benign
+002367,0,Benign,0.151131,0,Benign
+002368,0,Benign,0.160100,0,Benign
+002373,0,Benign,0.014142,0,Benign
+002374,0,Benign,0.024371,0,Benign
+002381,0,Benign,0.011476,0,Benign
+002382,0,Benign,0.040132,0,Benign
+002396,0,Benign,0.032057,0,Benign
+002420,0,Benign,0.000711,0,Benign
+002429,0,Benign,0.332105,0,Benign
+002435,0,Benign,0.618278,0,Benign
+002448,0,Benign,0.958286,1,Malignant
+002451,0,Benign,0.969416,1,Malignant
+002456,0,Benign,0.020082,0,Benign
+002459,0,Benign,0.141170,0,Benign
+002461,0,Benign,0.168457,0,Benign
+002465,0,Benign,0.017178,0,Benign
+002474,0,Benign,0.012569,0,Benign
+002483,0,Benign,0.068126,0,Benign
+002517,0,Benign,0.002582,0,Benign
+002588,0,Benign,0.893838,1,Malignant
+002606,0,Benign,0.086433,0,Benign
+002791,0,Benign,0.352876,0,Benign
+002858,0,Benign,0.022251,0,Benign
+002863,0,Benign,0.167775,0,Benign
+002875,0,Benign,0.009737,0,Benign
+002877,0,Benign,0.001938,0,Benign
+002878,0,Benign,0.007933,0,Benign
+002882,0,Benign,0.841961,1,Malignant
+002883,0,Benign,0.071421,0,Benign
+002889,0,Benign,0.840128,1,Malignant
+002899,0,Benign,0.020725,0,Benign
+002912,0,Benign,0.720952,1,Malignant
+002916,0,Benign,0.027808,0,Benign
+002918,0,Benign,0.653529,0,Benign
+002928,0,Benign,0.003498,0,Benign
+002931,0,Benign,0.007220,0,Benign
+002936,0,Benign,0.395898,0,Benign
+002939,0,Benign,0.015212,0,Benign
+002945,0,Benign,0.994249,1,Malignant
+002956,0,Benign,0.028150,0,Benign
+002959,0,Benign,0.135803,0,Benign
+002964,0,Benign,0.002368,0,Benign
+002966,0,Benign,0.001806,0,Benign
+002971,0,Benign,0.067422,0,Benign
+003010,0,Benign,0.168619,0,Benign
+003031,0,Benign,0.051348,0,Benign
+003040,0,Benign,0.167214,0,Benign
+003048,0,Benign,0.012287,0,Benign
+003115,0,Benign,0.035206,0,Benign
+003139,0,Benign,0.111779,0,Benign
+003443,0,Benign,0.000723,0,Benign
+003445,0,Benign,0.145899,0,Benign
+003456,0,Benign,0.192768,0,Benign
+003469,0,Benign,0.009274,0,Benign
+003471,0,Benign,0.801371,1,Malignant
+003474,0,Benign,0.905377,1,Malignant
+003475,0,Benign,0.201188,0,Benign
+003477,0,Benign,0.049918,0,Benign
+003485,0,Benign,0.416427,0,Benign
+003489,0,Benign,0.222432,0,Benign
+003505,0,Benign,0.820943,1,Malignant
+003506,0,Benign,0.015627,0,Benign
+003510,0,Benign,0.005326,0,Benign
+003526,0,Benign,0.545514,0,Benign
+003527,0,Benign,0.182499,0,Benign
+003541,0,Benign,0.164417,0,Benign
+003545,0,Benign,0.136935,0,Benign
+003547,0,Benign,0.566342,0,Benign
+003558,0,Benign,0.011882,0,Benign
+003559,0,Benign,0.028955,0,Benign
+003573,0,Benign,0.709600,0,Benign
+003575,0,Benign,0.650911,0,Benign
+004078,0,Benign,0.164860,0,Benign
+004081,0,Benign,0.431349,0,Benign
+004096,0,Benign,0.431688,0,Benign
+004097,0,Benign,0.008423,0,Benign
+004099,0,Benign,0.036861,0,Benign
+004105,0,Benign,0.379387,0,Benign
+004110,0,Benign,0.526580,0,Benign
+004115,0,Benign,0.665016,0,Benign
+004122,0,Benign,0.207223,0,Benign
+004126,0,Benign,0.815986,1,Malignant
+004130,0,Benign,0.004614,0,Benign
+004136,0,Benign,0.093367,0,Benign
+004140,0,Benign,0.003487,0,Benign
+004142,0,Benign,0.012029,0,Benign
+004146,0,Benign,0.843166,1,Malignant
+004147,0,Benign,0.001027,0,Benign
+004151,0,Benign,0.177867,0,Benign
+004165,0,Benign,0.292728,0,Benign
+004599,0,Benign,0.310957,0,Benign
+004626,0,Benign,0.336940,0,Benign
+004628,0,Benign,0.910667,1,Malignant
+004631,0,Benign,0.674330,0,Benign
+004640,0,Benign,0.981575,1,Malignant
+004641,0,Benign,0.003257,0,Benign
+004650,0,Benign,0.105583,0,Benign
+004660,0,Benign,0.487054,0,Benign
+004667,0,Benign,0.719977,1,Malignant
+004668,0,Benign,0.918459,1,Malignant
+004670,0,Benign,0.718596,1,Malignant
+004671,0,Benign,0.854799,1,Malignant
+004677,0,Benign,0.529348,0,Benign
+004678,0,Benign,0.891073,1,Malignant
+004680,0,Benign,0.839841,1,Malignant
+004693,0,Benign,0.054126,0,Benign
+004712,0,Benign,0.042257,0,Benign
+004715,0,Benign,0.661028,0,Benign
+004718,0,Benign,0.020256,0,Benign
+000076,1,Malignant,0.988106,1,Malignant
+000101,1,Malignant,0.987695,1,Malignant
+000115,1,Malignant,0.761836,1,Malignant
+000147,1,Malignant,0.801051,1,Malignant
+000180,1,Malignant,0.999960,1,Malignant
+000183,1,Malignant,0.549757,0,Benign
+000194,1,Malignant,0.996759,1,Malignant
+000195,1,Malignant,0.683322,0,Benign
+000240,1,Malignant,0.993892,1,Malignant
+000416,1,Malignant,0.993512,1,Malignant
+000483,1,Malignant,0.522804,0,Benign
+000512,1,Malignant,0.605400,0,Benign
+000513,1,Malignant,0.831612,1,Malignant
+000646,1,Malignant,0.754426,1,Malignant
+000689,1,Malignant,0.999956,1,Malignant
+000690,1,Malignant,0.954565,1,Malignant
+000691,1,Malignant,0.927895,1,Malignant
+000695,1,Malignant,0.992469,1,Malignant
+000702,1,Malignant,0.924252,1,Malignant
+000709,1,Malignant,0.265470,0,Benign
+000710,1,Malignant,0.993988,1,Malignant
+000712,1,Malignant,0.997112,1,Malignant
+000714,1,Malignant,0.989430,1,Malignant
+000728,1,Malignant,0.951929,1,Malignant
+000735,1,Malignant,0.101958,0,Benign
+000743,1,Malignant,0.818556,1,Malignant
+000745,1,Malignant,0.907677,1,Malignant
+000750,1,Malignant,0.993474,1,Malignant
+000756,1,Malignant,0.992416,1,Malignant
+000759,1,Malignant,0.989789,1,Malignant
+000760,1,Malignant,0.882096,1,Malignant
+000762,1,Malignant,0.879353,1,Malignant
+000773,1,Malignant,0.989654,1,Malignant
+000774,1,Malignant,0.982784,1,Malignant
+000776,1,Malignant,0.802452,1,Malignant
+000779,1,Malignant,0.997673,1,Malignant
+000785,1,Malignant,0.997894,1,Malignant
+000788,1,Malignant,0.992033,1,Malignant
+000793,1,Malignant,0.253489,0,Benign
+000797,1,Malignant,0.786995,1,Malignant
+000798,1,Malignant,0.628023,0,Benign
+000802,1,Malignant,0.985165,1,Malignant
+000807,1,Malignant,0.991284,1,Malignant
+000809,1,Malignant,0.969051,1,Malignant
+000815,1,Malignant,0.970573,1,Malignant
+000818,1,Malignant,0.852901,1,Malignant
+000822,1,Malignant,0.985544,1,Malignant
+000832,1,Malignant,0.952743,1,Malignant
+000841,1,Malignant,0.992124,1,Malignant
+000842,1,Malignant,0.818830,1,Malignant
+000844,1,Malignant,0.992149,1,Malignant
+000845,1,Malignant,0.992757,1,Malignant
+000846,1,Malignant,0.701353,0,Benign
+000852,1,Malignant,0.861300,1,Malignant
+000855,1,Malignant,0.997560,1,Malignant
+000858,1,Malignant,0.763017,1,Malignant
+000861,1,Malignant,0.970913,1,Malignant
+000870,1,Malignant,0.958173,1,Malignant
+000880,1,Malignant,0.990729,1,Malignant
+000882,1,Malignant,0.994648,1,Malignant
+000890,1,Malignant,0.999057,1,Malignant
+000897,1,Malignant,0.994762,1,Malignant
+000899,1,Malignant,0.998380,1,Malignant
+000900,1,Malignant,0.982477,1,Malignant
+000908,1,Malignant,0.400199,0,Benign
+000914,1,Malignant,0.995443,1,Malignant
+000918,1,Malignant,0.962861,1,Malignant
+000933,1,Malignant,0.982846,1,Malignant
+000939,1,Malignant,0.985140,1,Malignant
+000941,1,Malignant,0.945525,1,Malignant
+000942,1,Malignant,0.933974,1,Malignant
+000949,1,Malignant,0.976629,1,Malignant
+000952,1,Malignant,0.715553,1,Malignant
+000955,1,Malignant,0.166852,0,Benign
+000957,1,Malignant,0.989746,1,Malignant
+000966,1,Malignant,0.670455,0,Benign
+000968,1,Malignant,0.880955,1,Malignant
+000985,1,Malignant,0.509547,0,Benign
+000988,1,Malignant,0.703400,0,Benign
+000997,1,Malignant,0.986491,1,Malignant
+000998,1,Malignant,0.955285,1,Malignant
+001001,1,Malignant,0.994848,1,Malignant
+001012,1,Malignant,0.998219,1,Malignant
+001014,1,Malignant,0.995197,1,Malignant
+001021,1,Malignant,0.950214,1,Malignant
+001030,1,Malignant,0.810742,1,Malignant
+001038,1,Malignant,0.972109,1,Malignant
+001041,1,Malignant,0.861759,1,Malignant
+001043,1,Malignant,0.969137,1,Malignant
+001046,1,Malignant,0.950824,1,Malignant
+001049,1,Malignant,0.359347,0,Benign
+001055,1,Malignant,0.996275,1,Malignant
+001058,1,Malignant,0.944533,1,Malignant
+001064,1,Malignant,0.773933,1,Malignant
+001065,1,Malignant,0.996695,1,Malignant
+001072,1,Malignant,0.909806,1,Malignant
+001075,1,Malignant,0.306582,0,Benign
+001078,1,Malignant,0.998626,1,Malignant
+001085,1,Malignant,0.998469,1,Malignant
+001088,1,Malignant,0.964923,1,Malignant
+001093,1,Malignant,0.986920,1,Malignant
+001094,1,Malignant,0.716891,1,Malignant
+001096,1,Malignant,0.988449,1,Malignant
+001097,1,Malignant,0.962583,1,Malignant
+001101,1,Malignant,0.847014,1,Malignant
+001105,1,Malignant,0.985394,1,Malignant
+001116,1,Malignant,0.986063,1,Malignant
+001119,1,Malignant,0.980341,1,Malignant
+001122,1,Malignant,0.740422,1,Malignant
+001125,1,Malignant,0.564939,0,Benign
+001140,1,Malignant,0.970276,1,Malignant
+001146,1,Malignant,0.998717,1,Malignant
+001153,1,Malignant,0.990983,1,Malignant
+001154,1,Malignant,0.701181,0,Benign
+001155,1,Malignant,0.848377,1,Malignant
+001156,1,Malignant,0.279686,0,Benign
+001161,1,Malignant,0.951136,1,Malignant
+001166,1,Malignant,0.993566,1,Malignant
+001176,1,Malignant,0.977631,1,Malignant
+001178,1,Malignant,0.668678,0,Benign
+001181,1,Malignant,0.992727,1,Malignant
+001184,1,Malignant,0.925752,1,Malignant
+001186,1,Malignant,0.999712,1,Malignant
+001200,1,Malignant,0.997661,1,Malignant
+001204,1,Malignant,0.989757,1,Malignant
+001214,1,Malignant,0.992934,1,Malignant
+001215,1,Malignant,0.035125,0,Benign
+001224,1,Malignant,0.894715,1,Malignant
+001229,1,Malignant,0.997480,1,Malignant
+001240,1,Malignant,0.951197,1,Malignant
+001241,1,Malignant,0.927314,1,Malignant
+001242,1,Malignant,0.994040,1,Malignant
+001243,1,Malignant,0.994712,1,Malignant
+001251,1,Malignant,0.978656,1,Malignant
+001253,1,Malignant,0.963836,1,Malignant
+001255,1,Malignant,0.935158,1,Malignant
+001261,1,Malignant,0.997486,1,Malignant
+001263,1,Malignant,0.919578,1,Malignant
+001264,1,Malignant,0.965588,1,Malignant
+001266,1,Malignant,0.995602,1,Malignant
+001270,1,Malignant,0.382946,0,Benign
+001272,1,Malignant,0.960045,1,Malignant
+001278,1,Malignant,0.985634,1,Malignant
+001280,1,Malignant,0.919253,1,Malignant
+001283,1,Malignant,0.711554,1,Malignant
+001284,1,Malignant,0.996118,1,Malignant
+001288,1,Malignant,0.979226,1,Malignant
+001295,1,Malignant,0.999642,1,Malignant
+001299,1,Malignant,0.991094,1,Malignant
+001300,1,Malignant,0.916153,1,Malignant
+001304,1,Malignant,0.995783,1,Malignant
+001305,1,Malignant,0.999014,1,Malignant
+001307,1,Malignant,0.898630,1,Malignant
+001310,1,Malignant,0.999654,1,Malignant
+001314,1,Malignant,0.934021,1,Malignant
+001321,1,Malignant,0.985513,1,Malignant
+001322,1,Malignant,0.439595,0,Benign
+001326,1,Malignant,0.827354,1,Malignant
+001332,1,Malignant,0.848429,1,Malignant
+001336,1,Malignant,0.954777,1,Malignant
+001337,1,Malignant,0.175104,0,Benign
+001347,1,Malignant,0.878802,1,Malignant
+001349,1,Malignant,0.997743,1,Malignant
+001352,1,Malignant,0.987397,1,Malignant
+001359,1,Malignant,0.999165,1,Malignant
+001361,1,Malignant,0.993477,1,Malignant
+001365,1,Malignant,0.986184,1,Malignant
+001368,1,Malignant,0.998460,1,Malignant
+001387,1,Malignant,0.518177,0,Benign
+001394,1,Malignant,0.919435,1,Malignant
+001403,1,Malignant,0.606351,0,Benign
+001407,1,Malignant,0.999603,1,Malignant
+001408,1,Malignant,0.264139,0,Benign
+001411,1,Malignant,0.985361,1,Malignant
+001420,1,Malignant,0.959923,1,Malignant
+001435,1,Malignant,0.125190,0,Benign
+001436,1,Malignant,0.841843,1,Malignant
+001442,1,Malignant,0.999696,1,Malignant
+001446,1,Malignant,0.943408,1,Malignant
+001456,1,Malignant,0.998087,1,Malignant
+001459,1,Malignant,0.990286,1,Malignant
+001461,1,Malignant,0.999761,1,Malignant
+001462,1,Malignant,0.993879,1,Malignant
+001475,1,Malignant,0.997832,1,Malignant
+001481,1,Malignant,0.981934,1,Malignant
+001482,1,Malignant,0.993141,1,Malignant
+001486,1,Malignant,0.926612,1,Malignant
+001491,1,Malignant,0.633059,0,Benign
+001495,1,Malignant,0.996593,1,Malignant
+001501,1,Malignant,0.942648,1,Malignant
+001503,1,Malignant,0.957815,1,Malignant
+001519,1,Malignant,0.847699,1,Malignant
+001521,1,Malignant,0.999766,1,Malignant
+001528,1,Malignant,0.999351,1,Malignant
+001535,1,Malignant,0.987446,1,Malignant
+001549,1,Malignant,0.999527,1,Malignant
+001554,1,Malignant,0.996131,1,Malignant
+001556,1,Malignant,0.603066,0,Benign
+001562,1,Malignant,0.984052,1,Malignant
+001572,1,Malignant,0.997382,1,Malignant
+001579,1,Malignant,0.877011,1,Malignant
+001581,1,Malignant,0.997358,1,Malignant
+001583,1,Malignant,0.999952,1,Malignant
+001586,1,Malignant,0.999749,1,Malignant
+001587,1,Malignant,0.999874,1,Malignant
+001589,1,Malignant,0.995647,1,Malignant
+001597,1,Malignant,0.961708,1,Malignant
+001601,1,Malignant,0.999945,1,Malignant
+001603,1,Malignant,0.875569,1,Malignant
+001610,1,Malignant,0.955721,1,Malignant
+001612,1,Malignant,0.902527,1,Malignant
+001613,1,Malignant,0.649988,0,Benign
+001615,1,Malignant,0.908007,1,Malignant
+001618,1,Malignant,0.999179,1,Malignant
+001620,1,Malignant,0.906765,1,Malignant
+001625,1,Malignant,0.968676,1,Malignant
+001626,1,Malignant,0.917611,1,Malignant
+001633,1,Malignant,0.779418,1,Malignant
+001638,1,Malignant,0.981539,1,Malignant
+001653,1,Malignant,0.997676,1,Malignant
+001657,1,Malignant,0.988760,1,Malignant
+001665,1,Malignant,0.963141,1,Malignant
+001666,1,Malignant,0.967508,1,Malignant
+001668,1,Malignant,0.998697,1,Malignant
+001671,1,Malignant,0.993516,1,Malignant
+001672,1,Malignant,0.966728,1,Malignant
+001674,1,Malignant,0.992781,1,Malignant
+001677,1,Malignant,0.944523,1,Malignant
+001684,1,Malignant,0.946992,1,Malignant
+001687,1,Malignant,0.982495,1,Malignant
+001688,1,Malignant,0.982285,1,Malignant
+001692,1,Malignant,0.516141,0,Benign
+001695,1,Malignant,0.952865,1,Malignant
+001697,1,Malignant,0.992004,1,Malignant
+001705,1,Malignant,0.981087,1,Malignant
+001712,1,Malignant,0.999790,1,Malignant
+001716,1,Malignant,0.476895,0,Benign
+001719,1,Malignant,0.994026,1,Malignant
+001722,1,Malignant,0.978666,1,Malignant
+001746,1,Malignant,0.994993,1,Malignant
+001760,1,Malignant,0.999107,1,Malignant
+001788,1,Malignant,0.902727,1,Malignant
+001789,1,Malignant,0.999329,1,Malignant
+001810,1,Malignant,0.549603,0,Benign
+001822,1,Malignant,0.856507,1,Malignant
+001839,1,Malignant,0.986939,1,Malignant
+001948,1,Malignant,0.741596,1,Malignant
+001951,1,Malignant,0.997830,1,Malignant
+001957,1,Malignant,0.998183,1,Malignant
+001959,1,Malignant,0.998670,1,Malignant
+001965,1,Malignant,0.798207,1,Malignant
+001966,1,Malignant,0.312766,0,Benign
+001974,1,Malignant,0.991981,1,Malignant
+001983,1,Malignant,0.990025,1,Malignant
+001985,1,Malignant,0.988131,1,Malignant
+001999,1,Malignant,0.995642,1,Malignant
+002004,1,Malignant,0.995988,1,Malignant
+002005,1,Malignant,0.976291,1,Malignant
+002012,1,Malignant,0.996132,1,Malignant
+002016,1,Malignant,0.969541,1,Malignant
+002019,1,Malignant,0.993095,1,Malignant
+002026,1,Malignant,0.974777,1,Malignant
+002033,1,Malignant,0.784744,1,Malignant
+002036,1,Malignant,0.997118,1,Malignant
+002044,1,Malignant,0.991402,1,Malignant
+002048,1,Malignant,0.984623,1,Malignant
+002052,1,Malignant,0.999388,1,Malignant
+002060,1,Malignant,0.921639,1,Malignant
+002062,1,Malignant,0.986420,1,Malignant
+002069,1,Malignant,0.995716,1,Malignant
+002071,1,Malignant,0.998633,1,Malignant
+002072,1,Malignant,0.959817,1,Malignant
+002079,1,Malignant,0.980674,1,Malignant
+002082,1,Malignant,0.998629,1,Malignant
+002084,1,Malignant,0.968700,1,Malignant
+002085,1,Malignant,0.871633,1,Malignant
+002088,1,Malignant,0.996773,1,Malignant
+002092,1,Malignant,0.997441,1,Malignant
+002097,1,Malignant,0.756661,1,Malignant
+002098,1,Malignant,0.992359,1,Malignant
+002099,1,Malignant,0.057134,0,Benign
+002110,1,Malignant,0.384655,0,Benign
+002116,1,Malignant,0.141618,0,Benign
+002124,1,Malignant,0.983205,1,Malignant
+002128,1,Malignant,0.043400,0,Benign
+002130,1,Malignant,0.971986,1,Malignant
+002135,1,Malignant,0.770992,1,Malignant
+002137,1,Malignant,0.896156,1,Malignant
+002149,1,Malignant,0.999679,1,Malignant
+002155,1,Malignant,0.954614,1,Malignant
+002166,1,Malignant,0.990101,1,Malignant
+002171,1,Malignant,0.927778,1,Malignant
+002175,1,Malignant,0.964621,1,Malignant
+002179,1,Malignant,0.993388,1,Malignant
+002180,1,Malignant,0.996387,1,Malignant
+002193,1,Malignant,0.957576,1,Malignant
+002196,1,Malignant,0.913209,1,Malignant
+002201,1,Malignant,0.988991,1,Malignant
+002205,1,Malignant,0.998076,1,Malignant
+002212,1,Malignant,0.958494,1,Malignant
+002215,1,Malignant,0.484798,0,Benign
+002225,1,Malignant,0.985488,1,Malignant
+002231,1,Malignant,0.849781,1,Malignant
+002235,1,Malignant,0.997120,1,Malignant
+002237,1,Malignant,0.996022,1,Malignant
+002238,1,Malignant,0.716588,1,Malignant
+002242,1,Malignant,0.998746,1,Malignant
+002243,1,Malignant,0.765547,1,Malignant
+002247,1,Malignant,0.982860,1,Malignant
+002248,1,Malignant,0.991889,1,Malignant
+002253,1,Malignant,0.054042,0,Benign
+002261,1,Malignant,0.994336,1,Malignant
+002264,1,Malignant,0.980897,1,Malignant
+002272,1,Malignant,0.827028,1,Malignant
+002275,1,Malignant,0.990638,1,Malignant
+002277,1,Malignant,0.738599,1,Malignant
+002278,1,Malignant,0.793623,1,Malignant
+002286,1,Malignant,0.999520,1,Malignant
+002290,1,Malignant,0.999678,1,Malignant
+002294,1,Malignant,0.996064,1,Malignant
+002295,1,Malignant,0.948572,1,Malignant
+002306,1,Malignant,0.984173,1,Malignant
+002310,1,Malignant,0.968276,1,Malignant
+002317,1,Malignant,0.995948,1,Malignant
+002326,1,Malignant,0.397195,0,Benign
+002328,1,Malignant,0.950099,1,Malignant
+002335,1,Malignant,0.999909,1,Malignant
+002496,1,Malignant,0.996390,1,Malignant
+002498,1,Malignant,0.832269,1,Malignant
+002503,1,Malignant,0.684462,0,Benign
+002513,1,Malignant,0.996308,1,Malignant
+002515,1,Malignant,0.983988,1,Malignant
+002518,1,Malignant,0.999673,1,Malignant
+002519,1,Malignant,0.914439,1,Malignant
+002521,1,Malignant,0.999960,1,Malignant
+002545,1,Malignant,0.989904,1,Malignant
+002546,1,Malignant,0.918163,1,Malignant
+002551,1,Malignant,0.995326,1,Malignant
+002554,1,Malignant,0.999841,1,Malignant
+002562,1,Malignant,0.991804,1,Malignant
+002563,1,Malignant,0.941257,1,Malignant
+002571,1,Malignant,0.994461,1,Malignant
+002575,1,Malignant,0.910867,1,Malignant
+002577,1,Malignant,0.890422,1,Malignant
+002578,1,Malignant,0.980778,1,Malignant
+002581,1,Malignant,0.995798,1,Malignant
+002582,1,Malignant,0.961563,1,Malignant
+002604,1,Malignant,0.973968,1,Malignant
+002610,1,Malignant,0.938452,1,Malignant
+002618,1,Malignant,0.999519,1,Malignant
+002621,1,Malignant,0.959532,1,Malignant
+002623,1,Malignant,0.939649,1,Malignant
+002631,1,Malignant,0.998530,1,Malignant
+002637,1,Malignant,0.908542,1,Malignant
+002646,1,Malignant,0.974289,1,Malignant
+002653,1,Malignant,0.998822,1,Malignant
+002660,1,Malignant,0.457287,0,Benign
+002662,1,Malignant,0.999520,1,Malignant
+002664,1,Malignant,0.998301,1,Malignant
+002665,1,Malignant,0.997695,1,Malignant
+002669,1,Malignant,0.957053,1,Malignant
+002675,1,Malignant,0.999279,1,Malignant
+002678,1,Malignant,0.999403,1,Malignant
+002680,1,Malignant,0.973139,1,Malignant
+002682,1,Malignant,0.998670,1,Malignant
+002683,1,Malignant,0.999524,1,Malignant
+002686,1,Malignant,0.977987,1,Malignant
+002689,1,Malignant,0.954236,1,Malignant
+002690,1,Malignant,0.910959,1,Malignant
+002697,1,Malignant,0.993698,1,Malignant
+002700,1,Malignant,0.999971,1,Malignant
+002701,1,Malignant,0.995512,1,Malignant
+002702,1,Malignant,0.998439,1,Malignant
+002707,1,Malignant,0.941048,1,Malignant
+002722,1,Malignant,0.996735,1,Malignant
+002724,1,Malignant,0.922314,1,Malignant
+002726,1,Malignant,0.999215,1,Malignant
+002730,1,Malignant,0.995685,1,Malignant
+002733,1,Malignant,0.597124,0,Benign
+002735,1,Malignant,0.999092,1,Malignant
+002736,1,Malignant,0.999845,1,Malignant
+002738,1,Malignant,0.999632,1,Malignant
+002744,1,Malignant,0.994965,1,Malignant
+002749,1,Malignant,0.974954,1,Malignant
+002751,1,Malignant,0.682394,0,Benign
+002759,1,Malignant,0.926965,1,Malignant
+002771,1,Malignant,0.984243,1,Malignant
+002780,1,Malignant,0.671717,0,Benign
+002785,1,Malignant,0.790536,1,Malignant
+002788,1,Malignant,0.997999,1,Malignant
+002794,1,Malignant,0.987351,1,Malignant
+002799,1,Malignant,0.995547,1,Malignant
+002807,1,Malignant,0.995296,1,Malignant
+002811,1,Malignant,0.986844,1,Malignant
+002813,1,Malignant,0.996474,1,Malignant
+002827,1,Malignant,0.877527,1,Malignant
+002829,1,Malignant,0.800691,1,Malignant
+002830,1,Malignant,0.912302,1,Malignant
+002832,1,Malignant,0.915515,1,Malignant
+002836,1,Malignant,0.997616,1,Malignant
+002838,1,Malignant,0.993679,1,Malignant
+002847,1,Malignant,0.989258,1,Malignant
+002985,1,Malignant,0.996471,1,Malignant
+002989,1,Malignant,0.996722,1,Malignant
+002996,1,Malignant,0.969577,1,Malignant
+003001,1,Malignant,0.159474,0,Benign
+003004,1,Malignant,0.948242,1,Malignant
+003006,1,Malignant,0.990829,1,Malignant
+003020,1,Malignant,0.996957,1,Malignant
+003023,1,Malignant,0.992238,1,Malignant
+003026,1,Malignant,0.998613,1,Malignant
+003036,1,Malignant,0.999495,1,Malignant
+003053,1,Malignant,0.996434,1,Malignant
+003054,1,Malignant,0.999291,1,Malignant
+003055,1,Malignant,0.999713,1,Malignant
+003075,1,Malignant,0.934325,1,Malignant
+003077,1,Malignant,0.999183,1,Malignant
+003079,1,Malignant,0.326411,0,Benign
+003080,1,Malignant,0.989506,1,Malignant
+003090,1,Malignant,0.966555,1,Malignant
+003092,1,Malignant,0.933176,1,Malignant
+003106,1,Malignant,0.850177,1,Malignant
+003116,1,Malignant,0.901322,1,Malignant
+003119,1,Malignant,0.990059,1,Malignant
+003122,1,Malignant,0.672870,0,Benign
+003134,1,Malignant,0.996894,1,Malignant
+003138,1,Malignant,0.999338,1,Malignant
+003154,1,Malignant,0.871454,1,Malignant
+003165,1,Malignant,0.985871,1,Malignant
+003169,1,Malignant,0.973861,1,Malignant
+003171,1,Malignant,0.994774,1,Malignant
+003172,1,Malignant,0.994695,1,Malignant
+003175,1,Malignant,0.992313,1,Malignant
+003187,1,Malignant,0.989373,1,Malignant
+003192,1,Malignant,0.978116,1,Malignant
+003200,1,Malignant,0.975344,1,Malignant
+003204,1,Malignant,0.997120,1,Malignant
+003205,1,Malignant,0.999778,1,Malignant
+003219,1,Malignant,0.818023,1,Malignant
+003231,1,Malignant,0.998308,1,Malignant
+003235,1,Malignant,0.980066,1,Malignant
+003238,1,Malignant,0.972372,1,Malignant
+003246,1,Malignant,0.995843,1,Malignant
+003249,1,Malignant,0.810772,1,Malignant
+003252,1,Malignant,0.981918,1,Malignant
+003254,1,Malignant,0.699511,0,Benign
+003256,1,Malignant,0.995698,1,Malignant
+003257,1,Malignant,0.980700,1,Malignant
+003261,1,Malignant,0.995665,1,Malignant
+003263,1,Malignant,0.960137,1,Malignant
+003273,1,Malignant,0.997457,1,Malignant
+003284,1,Malignant,0.987562,1,Malignant
+003302,1,Malignant,0.860129,1,Malignant
+003312,1,Malignant,0.994609,1,Malignant
+003313,1,Malignant,0.996348,1,Malignant
+003323,1,Malignant,0.995337,1,Malignant
+003326,1,Malignant,0.984143,1,Malignant
+003329,1,Malignant,0.988739,1,Malignant
+003338,1,Malignant,0.997918,1,Malignant
+003348,1,Malignant,0.986497,1,Malignant
+003355,1,Malignant,0.988062,1,Malignant
+003356,1,Malignant,0.939997,1,Malignant
+003358,1,Malignant,0.997436,1,Malignant
+003359,1,Malignant,0.993993,1,Malignant
+003360,1,Malignant,0.997454,1,Malignant
+003369,1,Malignant,0.989870,1,Malignant
+003383,1,Malignant,0.808382,1,Malignant
+003386,1,Malignant,0.995125,1,Malignant
+003392,1,Malignant,0.507590,0,Benign
+003397,1,Malignant,0.995170,1,Malignant
+003401,1,Malignant,0.984205,1,Malignant
+003410,1,Malignant,0.995402,1,Malignant
+003425,1,Malignant,0.998939,1,Malignant
+003426,1,Malignant,0.999688,1,Malignant
+003580,1,Malignant,0.988681,1,Malignant
+003592,1,Malignant,0.997179,1,Malignant
+003597,1,Malignant,0.999456,1,Malignant
+003617,1,Malignant,0.639132,0,Benign
+003621,1,Malignant,0.690308,0,Benign
+003629,1,Malignant,0.943446,1,Malignant
+003632,1,Malignant,0.995875,1,Malignant
+003635,1,Malignant,0.987601,1,Malignant
+003643,1,Malignant,0.997313,1,Malignant
+003647,1,Malignant,0.998844,1,Malignant
+003649,1,Malignant,0.998234,1,Malignant
+003657,1,Malignant,0.989901,1,Malignant
+003658,1,Malignant,0.989495,1,Malignant
+003663,1,Malignant,0.540762,0,Benign
+003670,1,Malignant,0.998295,1,Malignant
+003677,1,Malignant,0.990845,1,Malignant
+003682,1,Malignant,0.993433,1,Malignant
+003683,1,Malignant,0.986892,1,Malignant
+003701,1,Malignant,0.940627,1,Malignant
+003706,1,Malignant,0.896658,1,Malignant
+003712,1,Malignant,0.876649,1,Malignant
+003717,1,Malignant,0.989501,1,Malignant
+003730,1,Malignant,0.781957,1,Malignant
+003733,1,Malignant,0.992679,1,Malignant
+003741,1,Malignant,0.954810,1,Malignant
+003746,1,Malignant,0.964377,1,Malignant
+003758,1,Malignant,0.994429,1,Malignant
+003761,1,Malignant,0.986761,1,Malignant
+003762,1,Malignant,0.991594,1,Malignant
+003763,1,Malignant,0.901967,1,Malignant
+003764,1,Malignant,0.999482,1,Malignant
+003774,1,Malignant,0.837600,1,Malignant
+003776,1,Malignant,0.629712,0,Benign
+003777,1,Malignant,0.979540,1,Malignant
+003781,1,Malignant,0.994604,1,Malignant
+003782,1,Malignant,0.999094,1,Malignant
+003787,1,Malignant,0.753626,1,Malignant
+003802,1,Malignant,0.971741,1,Malignant
+003804,1,Malignant,0.993994,1,Malignant
+003806,1,Malignant,0.932448,1,Malignant
+003808,1,Malignant,0.805702,1,Malignant
+003809,1,Malignant,0.949941,1,Malignant
+003814,1,Malignant,0.545864,0,Benign
+003817,1,Malignant,0.899031,1,Malignant
+003826,1,Malignant,0.998683,1,Malignant
+003827,1,Malignant,0.993627,1,Malignant
+003830,1,Malignant,0.995162,1,Malignant
+003838,1,Malignant,0.968450,1,Malignant
+003842,1,Malignant,0.986040,1,Malignant
+003843,1,Malignant,0.960795,1,Malignant
+003846,1,Malignant,0.999783,1,Malignant
+003848,1,Malignant,0.996053,1,Malignant
+003853,1,Malignant,0.992551,1,Malignant
+003862,1,Malignant,0.969106,1,Malignant
+003874,1,Malignant,0.997460,1,Malignant
+003881,1,Malignant,0.533142,0,Benign
+003882,1,Malignant,0.567271,0,Benign
+003883,1,Malignant,0.999837,1,Malignant
+003887,1,Malignant,0.888906,1,Malignant
+003888,1,Malignant,0.995096,1,Malignant
+003891,1,Malignant,0.995519,1,Malignant
+003904,1,Malignant,0.998775,1,Malignant
+003906,1,Malignant,0.993712,1,Malignant
+003908,1,Malignant,0.957494,1,Malignant
+003916,1,Malignant,0.974722,1,Malignant
+003917,1,Malignant,0.956523,1,Malignant
+003925,1,Malignant,0.993435,1,Malignant
+003933,1,Malignant,0.993865,1,Malignant
+003936,1,Malignant,0.841002,1,Malignant
+003942,1,Malignant,0.956218,1,Malignant
+003943,1,Malignant,0.585758,0,Benign
+003952,1,Malignant,0.908020,1,Malignant
+003958,1,Malignant,0.985517,1,Malignant
+003960,1,Malignant,0.916541,1,Malignant
+003961,1,Malignant,0.930404,1,Malignant
+003963,1,Malignant,0.972279,1,Malignant
+003974,1,Malignant,0.443081,0,Benign
+003975,1,Malignant,0.994700,1,Malignant
+003989,1,Malignant,0.974055,1,Malignant
+003990,1,Malignant,0.998841,1,Malignant
+003992,1,Malignant,0.907041,1,Malignant
+004000,1,Malignant,0.997292,1,Malignant
+004001,1,Malignant,0.997343,1,Malignant
+004013,1,Malignant,0.860234,1,Malignant
+004017,1,Malignant,0.939375,1,Malignant
+004025,1,Malignant,0.887041,1,Malignant
+004046,1,Malignant,0.957682,1,Malignant
+004049,1,Malignant,0.997451,1,Malignant
+004060,1,Malignant,0.784335,1,Malignant
+004062,1,Malignant,0.984212,1,Malignant
+004063,1,Malignant,0.992217,1,Malignant
+004092,1,Malignant,0.951426,1,Malignant
+004094,1,Malignant,0.995090,1,Malignant
+004095,1,Malignant,0.998346,1,Malignant
+004102,1,Malignant,0.983566,1,Malignant
+004167,1,Malignant,0.966429,1,Malignant
+004168,1,Malignant,0.975831,1,Malignant
+004170,1,Malignant,0.986467,1,Malignant
+004176,1,Malignant,0.667180,0,Benign
+004181,1,Malignant,0.963405,1,Malignant
+004189,1,Malignant,0.964328,1,Malignant
+004191,1,Malignant,0.990573,1,Malignant
+004200,1,Malignant,0.924130,1,Malignant
+004205,1,Malignant,0.988639,1,Malignant
+004213,1,Malignant,0.895263,1,Malignant
+004227,1,Malignant,0.956177,1,Malignant
+004230,1,Malignant,0.959090,1,Malignant
+004235,1,Malignant,0.996873,1,Malignant
+004236,1,Malignant,0.743295,1,Malignant
+004242,1,Malignant,0.706343,0,Benign
+004248,1,Malignant,0.993908,1,Malignant
+004250,1,Malignant,0.994452,1,Malignant
+004258,1,Malignant,0.972373,1,Malignant
+004270,1,Malignant,0.979028,1,Malignant
+004272,1,Malignant,0.929851,1,Malignant
+004284,1,Malignant,0.941676,1,Malignant
+004287,1,Malignant,0.995832,1,Malignant
+004289,1,Malignant,0.980922,1,Malignant
+004292,1,Malignant,0.982074,1,Malignant
+004299,1,Malignant,0.607361,0,Benign
+004300,1,Malignant,0.914814,1,Malignant
+004310,1,Malignant,0.999523,1,Malignant
+004313,1,Malignant,0.994058,1,Malignant
+004315,1,Malignant,0.987371,1,Malignant
+004322,1,Malignant,0.994938,1,Malignant
+004325,1,Malignant,0.996415,1,Malignant
+004326,1,Malignant,0.987359,1,Malignant
+004333,1,Malignant,0.996591,1,Malignant
+004336,1,Malignant,0.995491,1,Malignant
+004338,1,Malignant,0.561299,0,Benign
+004345,1,Malignant,0.945713,1,Malignant
+004355,1,Malignant,0.998180,1,Malignant
+004358,1,Malignant,0.999135,1,Malignant
+004365,1,Malignant,0.915637,1,Malignant
+004369,1,Malignant,0.997844,1,Malignant
+004370,1,Malignant,0.999912,1,Malignant
+004371,1,Malignant,0.822876,1,Malignant
+004373,1,Malignant,0.999125,1,Malignant
+004380,1,Malignant,0.780788,1,Malignant
+004381,1,Malignant,0.901580,1,Malignant
+004382,1,Malignant,0.964811,1,Malignant
+004388,1,Malignant,0.839605,1,Malignant
+004394,1,Malignant,0.919946,1,Malignant
+004395,1,Malignant,0.517038,0,Benign
+004397,1,Malignant,0.998060,1,Malignant
+004398,1,Malignant,0.977605,1,Malignant
+004400,1,Malignant,0.864222,1,Malignant
+004404,1,Malignant,0.862405,1,Malignant
+004412,1,Malignant,0.778374,1,Malignant
+004413,1,Malignant,0.647297,0,Benign
+004416,1,Malignant,0.998082,1,Malignant
+004417,1,Malignant,0.997732,1,Malignant
+004419,1,Malignant,0.799195,1,Malignant
+004420,1,Malignant,0.911417,1,Malignant
+004422,1,Malignant,0.993832,1,Malignant
+004430,1,Malignant,0.996149,1,Malignant
+004431,1,Malignant,0.999748,1,Malignant
+004435,1,Malignant,0.475521,0,Benign
+004437,1,Malignant,0.998239,1,Malignant
+004438,1,Malignant,0.999856,1,Malignant
+004439,1,Malignant,0.985957,1,Malignant
+004448,1,Malignant,0.848213,1,Malignant
+004451,1,Malignant,0.985415,1,Malignant
+004456,1,Malignant,0.999498,1,Malignant
+004465,1,Malignant,0.942840,1,Malignant
+004475,1,Malignant,0.999605,1,Malignant
+004478,1,Malignant,0.999948,1,Malignant
+004481,1,Malignant,0.998458,1,Malignant
+004485,1,Malignant,0.991646,1,Malignant
+004486,1,Malignant,0.995032,1,Malignant
+004487,1,Malignant,0.950961,1,Malignant
+004489,1,Malignant,0.990908,1,Malignant
+004491,1,Malignant,0.987165,1,Malignant
+004496,1,Malignant,0.997650,1,Malignant
+004497,1,Malignant,0.987072,1,Malignant
+004500,1,Malignant,0.968979,1,Malignant
+004506,1,Malignant,0.994432,1,Malignant
+004509,1,Malignant,0.996898,1,Malignant
+004510,1,Malignant,0.926083,1,Malignant
+004521,1,Malignant,0.991403,1,Malignant
+004522,1,Malignant,0.921021,1,Malignant
+004533,1,Malignant,0.998629,1,Malignant
+004543,1,Malignant,0.954704,1,Malignant
+004549,1,Malignant,0.888949,1,Malignant
+004552,1,Malignant,0.935036,1,Malignant
+004553,1,Malignant,0.998911,1,Malignant
+004565,1,Malignant,0.890689,1,Malignant
+004570,1,Malignant,0.124649,0,Benign
+004574,1,Malignant,0.992323,1,Malignant
+004580,1,Malignant,0.998202,1,Malignant
+004586,1,Malignant,0.996039,1,Malignant
+004587,1,Malignant,0.998868,1,Malignant
+004588,1,Malignant,0.996154,1,Malignant
+004589,1,Malignant,0.734379,1,Malignant
+004594,1,Malignant,0.995579,1,Malignant
+004636,1,Malignant,0.935300,1,Malignant
+004637,1,Malignant,0.998134,1,Malignant
+004638,1,Malignant,0.997470,1,Malignant
+004645,1,Malignant,0.993321,1,Malignant
+004724,1,Malignant,0.897559,1,Malignant
+004726,1,Malignant,0.996057,1,Malignant
+004727,1,Malignant,0.991684,1,Malignant
+004728,1,Malignant,0.998960,1,Malignant
+004729,1,Malignant,0.980055,1,Malignant
+004730,1,Malignant,0.988399,1,Malignant
+004740,1,Malignant,0.767038,1,Malignant
+004742,1,Malignant,0.938034,1,Malignant
+004744,1,Malignant,0.940004,1,Malignant
+004746,1,Malignant,0.855076,1,Malignant
+004750,1,Malignant,0.992189,1,Malignant
+004752,1,Malignant,0.836708,1,Malignant
+004756,1,Malignant,0.971239,1,Malignant
+004760,1,Malignant,0.960529,1,Malignant
+004762,1,Malignant,0.981579,1,Malignant
+004766,1,Malignant,0.999515,1,Malignant
+004770,1,Malignant,0.999660,1,Malignant
+004782,1,Malignant,0.998493,1,Malignant
+004788,1,Malignant,0.887674,1,Malignant
+004792,1,Malignant,0.999655,1,Malignant
+004801,1,Malignant,0.998809,1,Malignant
+004807,1,Malignant,0.654411,0,Benign
+004808,1,Malignant,0.977022,1,Malignant
+004810,1,Malignant,0.995222,1,Malignant
+004812,1,Malignant,0.991110,1,Malignant
+004829,1,Malignant,0.714231,1,Malignant
+004830,1,Malignant,0.873680,1,Malignant
+004835,1,Malignant,0.978168,1,Malignant
+004840,1,Malignant,0.962467,1,Malignant
+004842,1,Malignant,0.987063,1,Malignant
+004850,1,Malignant,0.851278,1,Malignant
+004875,1,Malignant,0.993820,1,Malignant
+004882,1,Malignant,0.999927,1,Malignant
+004889,1,Malignant,0.854955,1,Malignant
+004890,1,Malignant,0.975908,1,Malignant
+004896,1,Malignant,0.767198,1,Malignant
+004898,1,Malignant,0.985588,1,Malignant
+004906,1,Malignant,0.979915,1,Malignant
+004911,1,Malignant,0.997864,1,Malignant
+004916,1,Malignant,0.977970,1,Malignant
+004929,1,Malignant,0.999784,1,Malignant
+004955,1,Malignant,0.974640,1,Malignant
+004959,1,Malignant,0.998328,1,Malignant
+004960,1,Malignant,0.994223,1,Malignant
+004963,1,Malignant,0.989633,1,Malignant
+004969,1,Malignant,0.806766,1,Malignant
+004971,1,Malignant,0.779123,1,Malignant
+004973,1,Malignant,0.992185,1,Malignant
+004974,1,Malignant,0.999426,1,Malignant
+004982,1,Malignant,0.999585,1,Malignant
+004985,1,Malignant,0.992441,1,Malignant
+004986,1,Malignant,0.969312,1,Malignant
+004988,1,Malignant,0.999734,1,Malignant
+004990,1,Malignant,0.980178,1,Malignant
+004993,1,Malignant,0.998539,1,Malignant
+004995,1,Malignant,0.933142,1,Malignant
+004996,1,Malignant,0.998271,1,Malignant
+004999,1,Malignant,0.992998,1,Malignant

results/valid_predictions.csv

@@ -0,0 +1,501 @@
+image_id,true_label,true_class,probability_malignant,predicted_label,predicted_class
+000019,0,Benign,0.387281,0,Benign
+000038,0,Benign,0.062997,0,Benign
+000055,0,Benign,0.679069,0,Benign
+000072,0,Benign,0.121674,0,Benign
+000099,0,Benign,0.041968,0,Benign
+000113,0,Benign,0.794337,1,Malignant
+000138,0,Benign,0.939260,1,Malignant
+000151,0,Benign,0.045493,0,Benign
+000201,0,Benign,0.520551,0,Benign
+000203,0,Benign,0.216831,0,Benign
+000210,0,Benign,0.067449,0,Benign
+000236,0,Benign,0.115158,0,Benign
+000238,0,Benign,0.479351,0,Benign
+000245,0,Benign,0.944340,1,Malignant
+000284,0,Benign,0.018240,0,Benign
+000287,0,Benign,0.229929,0,Benign
+000305,0,Benign,0.951172,1,Malignant
+000317,0,Benign,0.648442,0,Benign
+000331,0,Benign,0.002365,0,Benign
+000345,0,Benign,0.048424,0,Benign
+000369,0,Benign,0.085227,0,Benign
+000374,0,Benign,0.543083,0,Benign
+000389,0,Benign,0.242428,0,Benign
+000396,0,Benign,0.101504,0,Benign
+000409,0,Benign,0.030582,0,Benign
+000420,0,Benign,0.001326,0,Benign
+000465,0,Benign,0.311004,0,Benign
+000471,0,Benign,0.031065,0,Benign
+000474,0,Benign,0.168879,0,Benign
+000502,0,Benign,0.074407,0,Benign
+000523,0,Benign,0.121968,0,Benign
+000536,0,Benign,0.006251,0,Benign
+000548,0,Benign,0.419118,0,Benign
+000549,0,Benign,0.024952,0,Benign
+000556,0,Benign,0.030170,0,Benign
+000561,0,Benign,0.030486,0,Benign
+000565,0,Benign,0.017267,0,Benign
+000569,0,Benign,0.195708,0,Benign
+000580,0,Benign,0.020466,0,Benign
+000583,0,Benign,0.379398,0,Benign
+000587,0,Benign,0.001430,0,Benign
+000598,0,Benign,0.009525,0,Benign
+000624,0,Benign,0.028617,0,Benign
+000638,0,Benign,0.558968,0,Benign
+000655,0,Benign,0.177185,0,Benign
+000677,0,Benign,0.013310,0,Benign
+001731,0,Benign,0.085332,0,Benign
+001734,0,Benign,0.182752,0,Benign
+001741,0,Benign,0.058499,0,Benign
+001754,0,Benign,0.110054,0,Benign
+001772,0,Benign,0.026569,0,Benign
+001797,0,Benign,0.037475,0,Benign
+001799,0,Benign,0.676092,0,Benign
+001801,0,Benign,0.156940,0,Benign
+001811,0,Benign,0.010308,0,Benign
+001817,0,Benign,0.043788,0,Benign
+001829,0,Benign,0.237723,0,Benign
+001854,0,Benign,0.042435,0,Benign
+001855,0,Benign,0.010479,0,Benign
+001862,0,Benign,0.764704,1,Malignant
+001887,0,Benign,0.046733,0,Benign
+001890,0,Benign,0.003490,0,Benign
+001894,0,Benign,0.007490,0,Benign
+001902,0,Benign,0.035927,0,Benign
+001903,0,Benign,0.373132,0,Benign
+001921,0,Benign,0.229835,0,Benign
+001924,0,Benign,0.603180,0,Benign
+001926,0,Benign,0.077111,0,Benign
+001932,0,Benign,0.038921,0,Benign
+002342,0,Benign,0.678385,0,Benign
+002351,0,Benign,0.004603,0,Benign
+002354,0,Benign,0.051213,0,Benign
+002357,0,Benign,0.379581,0,Benign
+002376,0,Benign,0.387769,0,Benign
+002419,0,Benign,0.047453,0,Benign
+002421,0,Benign,0.000408,0,Benign
+002424,0,Benign,0.099354,0,Benign
+002428,0,Benign,0.015846,0,Benign
+002431,0,Benign,0.008843,0,Benign
+002434,0,Benign,0.624946,0,Benign
+002460,0,Benign,0.016944,0,Benign
+002462,0,Benign,0.234696,0,Benign
+002478,0,Benign,0.600433,0,Benign
+002487,0,Benign,0.527726,0,Benign
+002572,0,Benign,0.016822,0,Benign
+002661,0,Benign,0.002315,0,Benign
+002854,0,Benign,0.177960,0,Benign
+002868,0,Benign,0.533999,0,Benign
+002901,0,Benign,0.026087,0,Benign
+002903,0,Benign,0.119333,0,Benign
+002911,0,Benign,0.074131,0,Benign
+002932,0,Benign,0.130335,0,Benign
+002961,0,Benign,0.003093,0,Benign
+002970,0,Benign,0.257603,0,Benign
+002978,0,Benign,0.059331,0,Benign
+003086,0,Benign,0.011522,0,Benign
+003468,0,Benign,0.856581,1,Malignant
+003472,0,Benign,0.912035,1,Malignant
+003476,0,Benign,0.934736,1,Malignant
+003488,0,Benign,0.033851,0,Benign
+003509,0,Benign,0.001938,0,Benign
+003531,0,Benign,0.033524,0,Benign
+003534,0,Benign,0.006998,0,Benign
+003539,0,Benign,0.078468,0,Benign
+003556,0,Benign,0.009779,0,Benign
+003566,0,Benign,0.611938,0,Benign
+003572,0,Benign,0.054567,0,Benign
+003578,0,Benign,0.646698,0,Benign
+004107,0,Benign,0.834480,1,Malignant
+004127,0,Benign,0.820747,1,Malignant
+004128,0,Benign,0.034664,0,Benign
+004131,0,Benign,0.549792,0,Benign
+004138,0,Benign,0.868963,1,Malignant
+004148,0,Benign,0.538186,0,Benign
+004156,0,Benign,0.092962,0,Benign
+004158,0,Benign,0.006836,0,Benign
+004161,0,Benign,0.949909,1,Malignant
+004633,0,Benign,0.015397,0,Benign
+004665,0,Benign,0.045830,0,Benign
+004679,0,Benign,0.906385,1,Malignant
+004684,0,Benign,0.009066,0,Benign
+004710,0,Benign,0.526296,0,Benign
+004717,0,Benign,0.004666,0,Benign
+004720,0,Benign,0.370934,0,Benign
+004723,0,Benign,0.009995,0,Benign
+000066,1,Malignant,0.993485,1,Malignant
+000079,1,Malignant,0.392704,0,Benign
+000094,1,Malignant,0.975690,1,Malignant
+000109,1,Malignant,0.998271,1,Malignant
+000110,1,Malignant,0.993965,1,Malignant
+000164,1,Malignant,0.947308,1,Malignant
+000170,1,Malignant,0.999071,1,Malignant
+000298,1,Malignant,0.997954,1,Malignant
+000321,1,Malignant,0.986230,1,Malignant
+000386,1,Malignant,0.976108,1,Malignant
+000412,1,Malignant,0.999406,1,Malignant
+000482,1,Malignant,0.973097,1,Malignant
+000563,1,Malignant,0.996247,1,Malignant
+000627,1,Malignant,0.984402,1,Malignant
+000685,1,Malignant,0.999806,1,Malignant
+000698,1,Malignant,0.911930,1,Malignant
+000701,1,Malignant,0.963718,1,Malignant
+000704,1,Malignant,0.991786,1,Malignant
+000713,1,Malignant,0.968341,1,Malignant
+000717,1,Malignant,0.997639,1,Malignant
+000722,1,Malignant,0.968839,1,Malignant
+000730,1,Malignant,0.981428,1,Malignant
+000758,1,Malignant,0.978231,1,Malignant
+000765,1,Malignant,0.912167,1,Malignant
+000771,1,Malignant,0.981199,1,Malignant
+000772,1,Malignant,0.986472,1,Malignant
+000784,1,Malignant,0.998426,1,Malignant
+000787,1,Malignant,0.962361,1,Malignant
+000795,1,Malignant,0.877663,1,Malignant
+000812,1,Malignant,0.948511,1,Malignant
+000823,1,Malignant,0.961643,1,Malignant
+000826,1,Malignant,0.997290,1,Malignant
+000831,1,Malignant,0.880871,1,Malignant
+000847,1,Malignant,0.860789,1,Malignant
+000850,1,Malignant,0.997068,1,Malignant
+000862,1,Malignant,0.881968,1,Malignant
+000865,1,Malignant,0.974072,1,Malignant
+000885,1,Malignant,0.966868,1,Malignant
+000887,1,Malignant,0.982732,1,Malignant
+000895,1,Malignant,0.995623,1,Malignant
+000896,1,Malignant,0.997866,1,Malignant
+000901,1,Malignant,0.985213,1,Malignant
+000905,1,Malignant,0.964824,1,Malignant
+000913,1,Malignant,0.986834,1,Malignant
+000932,1,Malignant,0.996755,1,Malignant
+000944,1,Malignant,0.942766,1,Malignant
+000956,1,Malignant,0.982945,1,Malignant
+000964,1,Malignant,0.999472,1,Malignant
+000971,1,Malignant,0.988736,1,Malignant
+000973,1,Malignant,0.996567,1,Malignant
+000974,1,Malignant,0.994878,1,Malignant
+000975,1,Malignant,0.974703,1,Malignant
+001003,1,Malignant,0.952650,1,Malignant
+001005,1,Malignant,0.616048,0,Benign
+001006,1,Malignant,0.922225,1,Malignant
+001011,1,Malignant,0.139962,0,Benign
+001025,1,Malignant,0.995767,1,Malignant
+001028,1,Malignant,0.993395,1,Malignant
+001061,1,Malignant,0.987865,1,Malignant
+001066,1,Malignant,0.996522,1,Malignant
+001076,1,Malignant,0.951404,1,Malignant
+001083,1,Malignant,0.985492,1,Malignant
+001113,1,Malignant,0.495338,0,Benign
+001120,1,Malignant,0.890909,1,Malignant
+001129,1,Malignant,0.752179,1,Malignant
+001131,1,Malignant,0.977786,1,Malignant
+001139,1,Malignant,0.999149,1,Malignant
+001151,1,Malignant,0.772084,1,Malignant
+001171,1,Malignant,0.996123,1,Malignant
+001173,1,Malignant,0.996236,1,Malignant
+001182,1,Malignant,0.998429,1,Malignant
+001185,1,Malignant,0.900645,1,Malignant
+001199,1,Malignant,0.966592,1,Malignant
+001202,1,Malignant,0.954136,1,Malignant
+001217,1,Malignant,0.982801,1,Malignant
+001226,1,Malignant,0.975301,1,Malignant
+001227,1,Malignant,0.988381,1,Malignant
+001233,1,Malignant,0.918507,1,Malignant
+001238,1,Malignant,0.988839,1,Malignant
+001245,1,Malignant,0.975408,1,Malignant
+001293,1,Malignant,0.143003,0,Benign
+001316,1,Malignant,0.938141,1,Malignant
+001328,1,Malignant,0.997835,1,Malignant
+001344,1,Malignant,0.988967,1,Malignant
+001406,1,Malignant,0.997181,1,Malignant
+001412,1,Malignant,0.762491,1,Malignant
+001443,1,Malignant,0.999642,1,Malignant
+001452,1,Malignant,0.945174,1,Malignant
+001455,1,Malignant,0.973271,1,Malignant
+001466,1,Malignant,0.986426,1,Malignant
+001471,1,Malignant,0.980855,1,Malignant
+001478,1,Malignant,0.953676,1,Malignant
+001500,1,Malignant,0.970171,1,Malignant
+001502,1,Malignant,0.969985,1,Malignant
+001510,1,Malignant,0.999471,1,Malignant
+001529,1,Malignant,0.934806,1,Malignant
+001536,1,Malignant,0.993730,1,Malignant
+001537,1,Malignant,0.964988,1,Malignant
+001542,1,Malignant,0.978624,1,Malignant
+001550,1,Malignant,0.998453,1,Malignant
+001553,1,Malignant,0.999799,1,Malignant
+001555,1,Malignant,0.998741,1,Malignant
+001566,1,Malignant,0.989323,1,Malignant
+001596,1,Malignant,0.993235,1,Malignant
+001607,1,Malignant,0.443932,0,Benign
+001621,1,Malignant,0.947348,1,Malignant
+001640,1,Malignant,0.965515,1,Malignant
+001643,1,Malignant,0.998730,1,Malignant
+001660,1,Malignant,0.996300,1,Malignant
+001661,1,Malignant,0.982810,1,Malignant
+001669,1,Malignant,0.997381,1,Malignant
+001675,1,Malignant,0.972770,1,Malignant
+001681,1,Malignant,0.925273,1,Malignant
+001702,1,Malignant,0.997202,1,Malignant
+001708,1,Malignant,0.941180,1,Malignant
+001709,1,Malignant,0.981210,1,Malignant
+001714,1,Malignant,0.989479,1,Malignant
+001774,1,Malignant,0.992550,1,Malignant
+001804,1,Malignant,0.994218,1,Malignant
+001808,1,Malignant,0.920840,1,Malignant
+001821,1,Malignant,0.487217,0,Benign
+001870,1,Malignant,0.994932,1,Malignant
+001884,1,Malignant,0.023947,0,Benign
+001904,1,Malignant,0.999020,1,Malignant
+001949,1,Malignant,0.998657,1,Malignant
+001961,1,Malignant,0.711314,1,Malignant
+001964,1,Malignant,0.995892,1,Malignant
+001970,1,Malignant,0.939822,1,Malignant
+001972,1,Malignant,0.994984,1,Malignant
+001979,1,Malignant,0.615314,0,Benign
+001980,1,Malignant,0.958194,1,Malignant
+001981,1,Malignant,0.927563,1,Malignant
+001988,1,Malignant,0.959771,1,Malignant
+001990,1,Malignant,0.862600,1,Malignant
+001991,1,Malignant,0.963877,1,Malignant
+001992,1,Malignant,0.989394,1,Malignant
+001996,1,Malignant,0.996645,1,Malignant
+001998,1,Malignant,0.777489,1,Malignant
+002013,1,Malignant,0.962771,1,Malignant
+002028,1,Malignant,0.951218,1,Malignant
+002047,1,Malignant,0.999922,1,Malignant
+002066,1,Malignant,0.995408,1,Malignant
+002077,1,Malignant,0.993160,1,Malignant
+002087,1,Malignant,0.994055,1,Malignant
+002102,1,Malignant,0.988949,1,Malignant
+002114,1,Malignant,0.951617,1,Malignant
+002136,1,Malignant,0.993738,1,Malignant
+002142,1,Malignant,0.995946,1,Malignant
+002153,1,Malignant,0.979807,1,Malignant
+002183,1,Malignant,0.821714,1,Malignant
+002185,1,Malignant,0.515379,0,Benign
+002206,1,Malignant,0.998406,1,Malignant
+002210,1,Malignant,0.998672,1,Malignant
+002213,1,Malignant,0.977704,1,Malignant
+002258,1,Malignant,0.998083,1,Malignant
+002266,1,Malignant,0.973419,1,Malignant
+002268,1,Malignant,0.962770,1,Malignant
+002288,1,Malignant,0.999359,1,Malignant
+002291,1,Malignant,0.881381,1,Malignant
+002307,1,Malignant,0.958153,1,Malignant
+002315,1,Malignant,0.968310,1,Malignant
+002318,1,Malignant,0.996061,1,Malignant
+002322,1,Malignant,0.981137,1,Malignant
+002332,1,Malignant,0.863663,1,Malignant
+002334,1,Malignant,0.790510,1,Malignant
+002494,1,Malignant,0.977316,1,Malignant
+002495,1,Malignant,0.994649,1,Malignant
+002497,1,Malignant,0.998809,1,Malignant
+002504,1,Malignant,0.962371,1,Malignant
+002505,1,Malignant,0.481627,0,Benign
+002506,1,Malignant,0.966266,1,Malignant
+002516,1,Malignant,0.935815,1,Malignant
+002520,1,Malignant,0.985208,1,Malignant
+002526,1,Malignant,0.999814,1,Malignant
+002527,1,Malignant,0.997584,1,Malignant
+002529,1,Malignant,0.998031,1,Malignant
+002533,1,Malignant,0.999243,1,Malignant
+002538,1,Malignant,0.989732,1,Malignant
+002580,1,Malignant,0.974431,1,Malignant
+002592,1,Malignant,0.966971,1,Malignant
+002599,1,Malignant,0.999943,1,Malignant
+002601,1,Malignant,0.996603,1,Malignant
+002614,1,Malignant,0.995410,1,Malignant
+002622,1,Malignant,0.972892,1,Malignant
+002625,1,Malignant,0.981396,1,Malignant
+002639,1,Malignant,0.619600,0,Benign
+002652,1,Malignant,0.995043,1,Malignant
+002666,1,Malignant,0.999196,1,Malignant
+002677,1,Malignant,0.999174,1,Malignant
+002684,1,Malignant,0.999632,1,Malignant
+002688,1,Malignant,0.957585,1,Malignant
+002717,1,Malignant,0.996240,1,Malignant
+002776,1,Malignant,0.995559,1,Malignant
+002815,1,Malignant,0.996464,1,Malignant
+002816,1,Malignant,0.869906,1,Malignant
+002819,1,Malignant,0.952106,1,Malignant
+002821,1,Malignant,0.997302,1,Malignant
+002822,1,Malignant,0.993221,1,Malignant
+002835,1,Malignant,0.996129,1,Malignant
+002842,1,Malignant,0.996020,1,Malignant
+002843,1,Malignant,0.990913,1,Malignant
+002986,1,Malignant,0.976309,1,Malignant
+002991,1,Malignant,0.957319,1,Malignant
+003003,1,Malignant,0.925761,1,Malignant
+003005,1,Malignant,0.979471,1,Malignant
+003022,1,Malignant,0.854833,1,Malignant
+003027,1,Malignant,0.995498,1,Malignant
+003034,1,Malignant,0.981327,1,Malignant
+003035,1,Malignant,0.983234,1,Malignant
+003059,1,Malignant,0.839371,1,Malignant
+003071,1,Malignant,0.998835,1,Malignant
+003074,1,Malignant,0.971335,1,Malignant
+003082,1,Malignant,0.941155,1,Malignant
+003097,1,Malignant,0.999811,1,Malignant
+003127,1,Malignant,0.726051,1,Malignant
+003137,1,Malignant,0.763365,1,Malignant
+003160,1,Malignant,0.956501,1,Malignant
+003190,1,Malignant,0.976107,1,Malignant
+003191,1,Malignant,0.914092,1,Malignant
+003222,1,Malignant,0.815161,1,Malignant
+003232,1,Malignant,0.803291,1,Malignant
+003237,1,Malignant,0.997355,1,Malignant
+003239,1,Malignant,0.966361,1,Malignant
+003240,1,Malignant,0.991806,1,Malignant
+003251,1,Malignant,0.975122,1,Malignant
+003267,1,Malignant,0.997212,1,Malignant
+003276,1,Malignant,0.994591,1,Malignant
+003278,1,Malignant,0.977191,1,Malignant
+003279,1,Malignant,0.979742,1,Malignant
+003287,1,Malignant,0.895884,1,Malignant
+003288,1,Malignant,0.981201,1,Malignant
+003314,1,Malignant,0.980365,1,Malignant
+003317,1,Malignant,0.959575,1,Malignant
+003319,1,Malignant,0.955607,1,Malignant
+003327,1,Malignant,0.915734,1,Malignant
+003328,1,Malignant,0.999948,1,Malignant
+003330,1,Malignant,0.984279,1,Malignant
+003343,1,Malignant,0.991889,1,Malignant
+003349,1,Malignant,0.995506,1,Malignant
+003351,1,Malignant,0.952905,1,Malignant
+003362,1,Malignant,0.973198,1,Malignant
+003364,1,Malignant,0.996079,1,Malignant
+003377,1,Malignant,0.999852,1,Malignant
+003378,1,Malignant,0.940930,1,Malignant
+003393,1,Malignant,0.607435,0,Benign
+003403,1,Malignant,0.944609,1,Malignant
+003405,1,Malignant,0.991853,1,Malignant
+003437,1,Malignant,0.937323,1,Malignant
+003601,1,Malignant,0.992880,1,Malignant
+003611,1,Malignant,0.998025,1,Malignant
+003612,1,Malignant,0.926110,1,Malignant
+003620,1,Malignant,0.999628,1,Malignant
+003624,1,Malignant,0.998621,1,Malignant
+003654,1,Malignant,0.803938,1,Malignant
+003655,1,Malignant,0.995213,1,Malignant
+003656,1,Malignant,0.999012,1,Malignant
+003681,1,Malignant,0.982621,1,Malignant
+003703,1,Malignant,0.989486,1,Malignant
+003704,1,Malignant,0.678671,0,Benign
+003707,1,Malignant,0.984178,1,Malignant
+003732,1,Malignant,0.964608,1,Malignant
+003745,1,Malignant,0.865759,1,Malignant
+003756,1,Malignant,0.999931,1,Malignant
+003757,1,Malignant,0.941315,1,Malignant
+003760,1,Malignant,0.949414,1,Malignant
+003789,1,Malignant,0.892808,1,Malignant
+003810,1,Malignant,0.919709,1,Malignant
+003821,1,Malignant,0.979100,1,Malignant
+003824,1,Malignant,0.757743,1,Malignant
+003833,1,Malignant,0.992178,1,Malignant
+003834,1,Malignant,0.992941,1,Malignant
+003836,1,Malignant,0.775179,1,Malignant
+003847,1,Malignant,0.999048,1,Malignant
+003855,1,Malignant,0.655440,0,Benign
+003863,1,Malignant,0.918345,1,Malignant
+003870,1,Malignant,0.984393,1,Malignant
+003875,1,Malignant,0.987469,1,Malignant
+003897,1,Malignant,0.998630,1,Malignant
+003901,1,Malignant,0.998393,1,Malignant
+003926,1,Malignant,0.970938,1,Malignant
+003932,1,Malignant,0.976214,1,Malignant
+003959,1,Malignant,0.423781,0,Benign
+003967,1,Malignant,0.980966,1,Malignant
+003981,1,Malignant,0.993071,1,Malignant
+003987,1,Malignant,0.974980,1,Malignant
+003993,1,Malignant,0.930377,1,Malignant
+003994,1,Malignant,0.980157,1,Malignant
+004023,1,Malignant,0.759148,1,Malignant
+004034,1,Malignant,0.992803,1,Malignant
+004036,1,Malignant,0.999078,1,Malignant
+004039,1,Malignant,0.997604,1,Malignant
+004061,1,Malignant,0.506222,0,Benign
+004084,1,Malignant,0.999368,1,Malignant
+004175,1,Malignant,0.999422,1,Malignant
+004178,1,Malignant,0.995541,1,Malignant
+004179,1,Malignant,0.928364,1,Malignant
+004190,1,Malignant,0.893910,1,Malignant
+004195,1,Malignant,0.991185,1,Malignant
+004206,1,Malignant,0.994625,1,Malignant
+004210,1,Malignant,0.893928,1,Malignant
+004212,1,Malignant,0.984733,1,Malignant
+004216,1,Malignant,0.997782,1,Malignant
+004224,1,Malignant,0.997015,1,Malignant
+004243,1,Malignant,0.988051,1,Malignant
+004244,1,Malignant,0.983893,1,Malignant
+004257,1,Malignant,0.993675,1,Malignant
+004278,1,Malignant,0.995409,1,Malignant
+004297,1,Malignant,0.938453,1,Malignant
+004298,1,Malignant,0.991459,1,Malignant
+004303,1,Malignant,0.981134,1,Malignant
+004306,1,Malignant,0.875113,1,Malignant
+004307,1,Malignant,0.991860,1,Malignant
+004328,1,Malignant,0.843277,1,Malignant
+004346,1,Malignant,0.980378,1,Malignant
+004349,1,Malignant,0.987017,1,Malignant
+004363,1,Malignant,0.996992,1,Malignant
+004375,1,Malignant,0.980458,1,Malignant
+004378,1,Malignant,0.993926,1,Malignant
+004392,1,Malignant,0.984863,1,Malignant
+004399,1,Malignant,0.928852,1,Malignant
+004403,1,Malignant,0.898591,1,Malignant
+004410,1,Malignant,0.997732,1,Malignant
+004414,1,Malignant,0.977467,1,Malignant
+004426,1,Malignant,0.885507,1,Malignant
+004436,1,Malignant,0.990488,1,Malignant
+004461,1,Malignant,0.987587,1,Malignant
+004463,1,Malignant,0.997002,1,Malignant
+004468,1,Malignant,0.928077,1,Malignant
+004479,1,Malignant,0.992864,1,Malignant
+004515,1,Malignant,0.938283,1,Malignant
+004517,1,Malignant,0.967930,1,Malignant
+004518,1,Malignant,0.995571,1,Malignant
+004520,1,Malignant,0.996082,1,Malignant
+004535,1,Malignant,0.947997,1,Malignant
+004538,1,Malignant,0.981330,1,Malignant
+004563,1,Malignant,0.997586,1,Malignant
+004576,1,Malignant,0.981701,1,Malignant
+004592,1,Malignant,0.940132,1,Malignant
+004597,1,Malignant,0.995031,1,Malignant
+004613,1,Malignant,0.982742,1,Malignant
+004647,1,Malignant,0.988146,1,Malignant
+004664,1,Malignant,0.994281,1,Malignant
+004739,1,Malignant,0.998700,1,Malignant
+004775,1,Malignant,0.977282,1,Malignant
+004793,1,Malignant,0.999082,1,Malignant
+004797,1,Malignant,0.981884,1,Malignant
+004802,1,Malignant,0.991508,1,Malignant
+004806,1,Malignant,0.023197,0,Benign
+004823,1,Malignant,0.999941,1,Malignant
+004825,1,Malignant,0.999410,1,Malignant
+004826,1,Malignant,0.998237,1,Malignant
+004843,1,Malignant,0.987593,1,Malignant
+004844,1,Malignant,0.957767,1,Malignant
+004845,1,Malignant,0.979598,1,Malignant
+004856,1,Malignant,0.979316,1,Malignant
+004859,1,Malignant,0.880817,1,Malignant
+004867,1,Malignant,0.990999,1,Malignant
+004868,1,Malignant,0.986867,1,Malignant
+004871,1,Malignant,0.993395,1,Malignant
+004872,1,Malignant,0.993593,1,Malignant
+004874,1,Malignant,0.988520,1,Malignant
+004891,1,Malignant,0.960777,1,Malignant
+004894,1,Malignant,0.923604,1,Malignant
+004895,1,Malignant,0.987956,1,Malignant
+004907,1,Malignant,0.994634,1,Malignant
+004922,1,Malignant,0.999635,1,Malignant
+004923,1,Malignant,0.978561,1,Malignant
+004933,1,Malignant,0.986324,1,Malignant
+004938,1,Malignant,0.949794,1,Malignant
+004941,1,Malignant,0.973917,1,Malignant
+004946,1,Malignant,0.986527,1,Malignant
+004953,1,Malignant,0.986625,1,Malignant
+004957,1,Malignant,0.970218,1,Malignant
+004981,1,Malignant,0.998157,1,Malignant
+005000,1,Malignant,0.991685,1,Malignant

results/winning_run_history.json

@@ -0,0 +1,212 @@
+[
+  {
+    "epoch": 0,
+    "train_loss": 0.1598212777716773,
+    "val_loss": 0.5469009280204773,
+    "val_auroc": 0.8194133333333333,
+    "val_sens@0.5": 0.9813333333333333,
+    "val_spec@0.5": 0.12,
+    "val_ppv@0.5": 0.7698744769874477,
+    "val_npv@0.5": 0.6818181818181818,
+    "val_ece@0.5": 0.190321617603302,
+    "val_brier": 0.17975325882434845,
+    "lr": 0.0001,
+    "epoch_time_s": 9.1
+  },
+  {
+    "epoch": 1,
+    "train_loss": 0.12178190640040806,
+    "val_loss": 0.4364696443080902,
+    "val_auroc": 0.899616,
+    "val_sens@0.5": 0.9386666666666666,
+    "val_spec@0.5": 0.608,
+    "val_ppv@0.5": 0.8778054862842892,
+    "val_npv@0.5": 0.7676767676767676,
+    "val_ece@0.5": 0.1629413467645645,
+    "val_brier": 0.13361606001853943,
+    "lr": 0.0002,
+    "epoch_time_s": 8.1
+  },
+  {
+    "epoch": 2,
+    "train_loss": 0.09681233019488199,
+    "val_loss": 0.6704103946685791,
+    "val_auroc": 0.8971306666666666,
+    "val_sens@0.5": 0.34933333333333333,
+    "val_spec@0.5": 0.984,
+    "val_ppv@0.5": 0.9849624060150376,
+    "val_npv@0.5": 0.335149863760218,
+    "val_ece@0.5": 0.35036509992182263,
+    "val_brier": 0.24114809930324554,
+    "lr": 0.0001996917333733128,
+    "epoch_time_s": 8.0
+  },
+  {
+    "epoch": 3,
+    "train_loss": 0.08150971835000174,
+    "val_loss": 0.39045241475105286,
+    "val_auroc": 0.959968,
+    "val_sens@0.5": 0.84,
+    "val_spec@0.5": 0.912,
+    "val_ppv@0.5": 0.9662576687116564,
+    "val_npv@0.5": 0.6551724137931034,
+    "val_ece@0.5": 0.20656415994465355,
+    "val_brier": 0.11692728847265244,
+    "lr": 0.00019876883405951377,
+    "epoch_time_s": 8.3
+  },
+  {
+    "epoch": 4,
+    "train_loss": 0.0718498518552099,
+    "val_loss": 0.29067134857177734,
+    "val_auroc": 0.9701013333333334,
+    "val_sens@0.5": 0.9626666666666667,
+    "val_spec@0.5": 0.816,
+    "val_ppv@0.5": 0.9401041666666666,
+    "val_npv@0.5": 0.8793103448275862,
+    "val_ece@0.5": 0.13937596596777438,
+    "val_brier": 0.07864873111248016,
+    "lr": 0.00019723699203976766,
+    "epoch_time_s": 8.4
+  },
+  {
+    "epoch": 5,
+    "train_loss": 0.06457222277351789,
+    "val_loss": 0.3043011724948883,
+    "val_auroc": 0.9652586666666667,
+    "val_sens@0.5": 0.8933333333333333,
+    "val_spec@0.5": 0.904,
+    "val_ppv@0.5": 0.9654178674351584,
+    "val_npv@0.5": 0.738562091503268,
+    "val_ece@0.5": 0.14274779457598924,
+    "val_brier": 0.08610112220048904,
+    "lr": 0.00019510565162951534,
+    "epoch_time_s": 8.2
+  },
+  {
+    "epoch": 6,
+    "train_loss": 0.05656171679496765,
+    "val_loss": 0.22200419008731842,
+    "val_auroc": 0.9756053333333333,
+    "val_sens@0.5": 0.9733333333333334,
+    "val_spec@0.5": 0.76,
+    "val_ppv@0.5": 0.9240506329113924,
+    "val_npv@0.5": 0.9047619047619048,
+    "val_ece@0.5": 0.08342214265465736,
+    "val_brier": 0.05918317288160324,
+    "lr": 0.00019238795325112864,
+    "epoch_time_s": 8.2
+  },
+  {
+    "epoch": 7,
+    "train_loss": 0.05262019157835415,
+    "val_loss": 0.24622531235218048,
+    "val_auroc": 0.9653013333333333,
+    "val_sens@0.5": 0.9386666666666666,
+    "val_spec@0.5": 0.856,
+    "val_ppv@0.5": 0.9513513513513514,
+    "val_npv@0.5": 0.823076923076923,
+    "val_ece@0.5": 0.09611700309067965,
+    "val_brier": 0.06746554374694824,
+    "lr": 0.00018910065241883674,
+    "epoch_time_s": 8.2
+  },
+  {
+    "epoch": 8,
+    "train_loss": 0.045101860872336794,
+    "val_loss": 0.2303643375635147,
+    "val_auroc": 0.9690666666666666,
+    "val_sens@0.5": 0.9493333333333334,
+    "val_spec@0.5": 0.832,
+    "val_ppv@0.5": 0.9442970822281167,
+    "val_npv@0.5": 0.8455284552845529,
+    "val_ece@0.5": 0.07282097344845534,
+    "val_brier": 0.06332194060087204,
+    "lr": 0.0001852640164354092,
+    "epoch_time_s": 8.1
+  },
+  {
+    "epoch": 9,
+    "train_loss": 0.04153951165505818,
+    "val_loss": 0.22570541501045227,
+    "val_auroc": 0.9694506666666667,
+    "val_sens@0.5": 0.9573333333333334,
+    "val_spec@0.5": 0.832,
+    "val_ppv@0.5": 0.9447368421052632,
+    "val_npv@0.5": 0.8666666666666667,
+    "val_ece@0.5": 0.07430721400678159,
+    "val_brier": 0.06271392852067947,
+    "lr": 0.00018090169943749473,
+    "epoch_time_s": 8.2
+  },
+  {
+    "epoch": 10,
+    "train_loss": 0.03941814090098653,
+    "val_loss": 0.20154055953025818,
+    "val_auroc": 0.9754239999999998,
+    "val_sens@0.5": 0.9733333333333334,
+    "val_spec@0.5": 0.832,
+    "val_ppv@0.5": 0.9455958549222798,
+    "val_npv@0.5": 0.9122807017543859,
+    "val_ece@0.5": 0.06357789923250676,
+    "val_brier": 0.05432131513953209,
+    "lr": 0.00017604059656000309,
+    "epoch_time_s": 8.2
+  },
+  {
+    "epoch": 11,
+    "train_loss": 0.03350917380622455,
+    "val_loss": 0.20909833908081055,
+    "val_auroc": 0.9710933333333334,
+    "val_sens@0.5": 0.9573333333333334,
+    "val_spec@0.5": 0.84,
+    "val_ppv@0.5": 0.9472295514511874,
+    "val_npv@0.5": 0.8677685950413223,
+    "val_ece@0.5": 0.06616889646649358,
+    "val_brier": 0.057766642421483994,
+    "lr": 0.00017071067811865473,
+    "epoch_time_s": 8.2
+  },
+  {
+    "epoch": 12,
+    "train_loss": 0.030914436317980288,
+    "val_loss": 0.20893579721450806,
+    "val_auroc": 0.9663893333333333,
+    "val_sens@0.5": 0.976,
+    "val_spec@0.5": 0.816,
+    "val_ppv@0.5": 0.9408740359897172,
+    "val_npv@0.5": 0.918918918918919,
+    "val_ece@0.5": 0.05971383413672446,
+    "val_brier": 0.0571916438639164,
+    "lr": 0.00016494480483301836,
+    "epoch_time_s": 8.4
+  },
+  {
+    "epoch": 13,
+    "train_loss": 0.028774067768028804,
+    "val_loss": 0.2109844982624054,
+    "val_auroc": 0.968,
+    "val_sens@0.5": 0.968,
+    "val_spec@0.5": 0.768,
+    "val_ppv@0.5": 0.9260204081632653,
+    "val_npv@0.5": 0.8888888888888888,
+    "val_ece@0.5": 0.04835819691419602,
+    "val_brier": 0.05989755317568779,
+    "lr": 0.0001587785252292473,
+    "epoch_time_s": 8.3
+  },
+  {
+    "epoch": 14,
+    "train_loss": 0.025121769504887717,
+    "val_loss": 0.21120241284370422,
+    "val_auroc": 0.9669226666666666,
+    "val_sens@0.5": 0.9573333333333334,
+    "val_spec@0.5": 0.864,
+    "val_ppv@0.5": 0.9547872340425532,
+    "val_npv@0.5": 0.8709677419354839,
+    "val_ece@0.5": 0.051551883660256835,
+    "val_brier": 0.05882710963487625,
+    "lr": 0.00015224985647159484,
+    "epoch_time_s": 8.0
+  }
+]

sweep.py

@@ -0,0 +1,122 @@
+#!/usr/bin/env python
+"""
+Focused hyperparameter sweep for the thyroid ResNet-18 classifier, optimized
+for **validation AUROC**. Each trial is a full train.py run (logged to Trackio).
+Trial val AUROCs are collected; the best config is reported.
+
+Search dimensions (kept deliberately small to avoid overfitting the val set):
+  - backbone variant (torchvision, timm a1/a2)
+  - learning rate
+  - weight decay
+  - batch size
+  - augmentation policy
+  - class-imbalance strategy
+  - fine-tune depth (freeze_stage)
+  - loss (bce vs focal)
+
+The sweep is staged: a base set of one-factor-at-a-time trials around a sensible
+center (informed by literature), so each dimension is isolated.
+"""
+import argparse
+import json
+import subprocess
+import sys
+from pathlib import Path
+
+DATA_DIR = "/app/TN5000"
+SPACE = "Johnyquest7/Trakio_agentic_thyroid"
+DSET = "Johnyquest7/Trakio_agentic_thyroid_dataset"
+PROJECT = "agentic_thyroid_resnet18"
+
+# center config (literature-informed)
+CENTER = dict(backbone="timm:resnet18.a1_in1k", lr=2e-4, weight_decay=1e-4,
+              batch_size=32, aug_policy="medical_default", imbalance="pos_weight",
+              freeze_stage=0, loss="bce", optimizer="adamw", scheduler="cosine",
+              epochs=40, early_stop_patience=8, dropout=0.0)
+
+# Each trial = (name, overrides-dict). One-factor-at-a-time around CENTER.
+TRIALS = [
+    ("c00_center_a1",            {}),
+    ("c01_backbone_torchvision", {"backbone": "torchvision"}),
+    ("c02_backbone_a2",          {"backbone": "timm:resnet18.a2_in1k"}),
+    ("c03_lr_1e-4",              {"lr": 1e-4}),
+    ("c04_lr_5e-4",              {"lr": 5e-4}),
+    ("c05_wd_1e-3",              {"weight_decay": 1e-3}),
+    ("c06_bs_64",                {"batch_size": 64}),
+    ("c07_aug_flip_only",        {"aug_policy": "flip_only"}),
+    ("c08_aug_strong",           {"aug_policy": "medical_strong"}),
+    ("c09_imb_none",             {"imbalance": "none"}),
+    ("c10_imb_sampler",          {"imbalance": "sampler"}),
+    ("c11_freeze1",              {"freeze_stage": 1}),
+    ("c12_loss_focal",           {"loss": "focal", "focal_gamma": 1.0, "imbalance": "none"}),
+    ("c13_lr1e-4_wd1e-3_drop",   {"lr": 1e-4, "weight_decay": 1e-3, "dropout": 0.2}),
+]
+
+
+def run_trial(name, overrides, out_root, seed):
+    cfg = dict(CENTER); cfg.update(overrides)
+    out_dir = str(Path(out_root) / name)
+    cmd = [sys.executable, "train.py",
+           "--data_dir", DATA_DIR, "--output_dir", out_dir,
+           "--run_name", name, "--seed", str(seed),
+           "--trackio_project", PROJECT, "--trackio_space_id", SPACE,
+           "--trackio_dataset_id", DSET,
+           "--num_workers", "4"]
+    for k, v in cfg.items():
+        if isinstance(v, bool):
+            if v:
+                cmd.append(f"--{k}")
+        else:
+            cmd += [f"--{k}", str(v)]
+    print(f"\n===== TRIAL {name} =====\n{' '.join(cmd)}", flush=True)
+    r = subprocess.run(cmd, capture_output=True, text=True)
+    # echo tail for visibility
+    print(r.stdout[-1500:], flush=True)
+    if r.returncode != 0:
+        print("STDERR tail:", r.stderr[-1500:], flush=True)
+    summ_path = Path(out_dir) / "final_summary.json"
+    best = -1.0; best_epoch = -1
+    if summ_path.exists():
+        s = json.load(open(summ_path))
+        best = s.get("best_val_auroc", -1.0); best_epoch = s.get("best_epoch", -1)
+    return {"name": name, "config": cfg, "best_val_auroc": best,
+            "best_epoch": best_epoch, "out_dir": out_dir, "returncode": r.returncode}
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--out_root", default="/app/sweep_runs")
+    ap.add_argument("--seed", type=int, default=42)
+    ap.add_argument("--only", default=None, help="comma list of trial names to run")
+    args = ap.parse_args()
+
+    Path(args.out_root).mkdir(parents=True, exist_ok=True)
+    trials = TRIALS
+    if args.only:
+        names = set(args.only.split(","))
+        trials = [t for t in TRIALS if t[0] in names]
+
+    results = []
+    res_path = Path(args.out_root) / "sweep_results.json"
+    if res_path.exists():
+        results = json.load(open(res_path))
+        done = {r["name"] for r in results}
+        trials = [t for t in trials if t[0] not in done]
+
+    for name, ov in trials:
+        res = run_trial(name, ov, args.out_root, args.seed)
+        results.append(res)
+        json.dump(results, open(res_path, "w"), indent=2)
+        print(f">>> {name}: val_auroc={res['best_val_auroc']:.4f} rc={res['returncode']}", flush=True)
+
+    results_sorted = sorted(results, key=lambda r: r["best_val_auroc"], reverse=True)
+    print("\n===== SWEEP LEADERBOARD =====")
+    for r in results_sorted:
+        print(f"{r['best_val_auroc']:.4f}  {r['name']:28s} (epoch {r['best_epoch']}, rc {r['returncode']})")
+    json.dump(results_sorted, open(Path(args.out_root) / "sweep_leaderboard.json", "w"), indent=2)
+    if results_sorted:
+        print("\nBEST:", results_sorted[0]["name"], results_sorted[0]["best_val_auroc"])
+
+
+if __name__ == "__main__":
+    main()

thyroid_lib.py

@@ -0,0 +1,539 @@
+#!/usr/bin/env python
+"""
+Shared library for the agentic thyroid ResNet-18 experiment.
+
+Centralizes everything that train.py / evaluate.py / evaluate_external.py must
+share so that preprocessing, model construction, calibration, and thresholding
+are guaranteed identical across training, validation, test, and external use.
+
+Positive class = Malignant (label 1). Benign = 0.
+"""
+import json
+import os
+import random
+from dataclasses import dataclass, field, asdict
+from pathlib import Path
+from typing import Optional, List, Tuple
+
+import numpy as np
+
+IMAGENET_MEAN = [0.485, 0.456, 0.406]
+IMAGENET_STD = [0.229, 0.224, 0.225]
+CLASS_TO_IDX = {"Benign": 0, "Malignant": 1}
+IDX_TO_CLASS = {0: "Benign", 1: "Malignant"}
+
+
+# --------------------------------------------------------------------------- #
+# Reproducibility
+# --------------------------------------------------------------------------- #
+def set_determinism(seed: int, strict: bool = True):
+    """Set all RNG seeds and (optionally) enforce deterministic algorithms."""
+    import torch
+    os.environ["PYTHONHASHSEED"] = str(seed)
+    random.seed(seed)
+    np.random.seed(seed)
+    torch.manual_seed(seed)
+    torch.cuda.manual_seed_all(seed)
+    if strict:
+        # cuBLAS workspace config required for deterministic matmul on CUDA.
+        os.environ.setdefault("CUBLAS_WORKSPACE_CONFIG", ":4096:8")
+        torch.backends.cudnn.deterministic = True
+        torch.backends.cudnn.benchmark = False
+        try:
+            torch.use_deterministic_algorithms(True, warn_only=True)
+        except Exception:
+            torch.use_deterministic_algorithms(True)
+    else:
+        torch.backends.cudnn.benchmark = True
+
+
+def seed_worker(worker_id):
+    import torch
+    worker_seed = torch.initial_seed() % 2 ** 32
+    np.random.seed(worker_seed)
+    random.seed(worker_seed)
+
+
+def collect_env_info():
+    """Return a dict of package versions and hardware/CUDA settings for logging."""
+    info = {}
+    try:
+        import torch
+        info["torch"] = torch.__version__
+        info["cuda_available"] = torch.cuda.is_available()
+        info["cuda_version"] = torch.version.cuda
+        info["cudnn_version"] = torch.backends.cudnn.version() if torch.backends.cudnn.is_available() else None
+        info["cudnn_deterministic"] = torch.backends.cudnn.deterministic
+        info["cudnn_benchmark"] = torch.backends.cudnn.benchmark
+        if torch.cuda.is_available():
+            info["gpu_name"] = torch.cuda.get_device_name(0)
+            info["gpu_count"] = torch.cuda.device_count()
+            props = torch.cuda.get_device_properties(0)
+            info["gpu_total_mem_gb"] = round(props.total_memory / 1e9, 2)
+    except Exception as e:
+        info["torch_error"] = repr(e)
+    for mod in ["torchvision", "timm", "sklearn", "numpy", "PIL", "trackio"]:
+        try:
+            m = __import__(mod)
+            info[mod] = getattr(m, "__version__", "?")
+        except Exception:
+            info[mod] = None
+    info["cublas_workspace_config"] = os.environ.get("CUBLAS_WORKSPACE_CONFIG")
+    info["pythonhashseed"] = os.environ.get("PYTHONHASHSEED")
+    return info
+
+
+# --------------------------------------------------------------------------- #
+# Preprocessing / augmentation
+# --------------------------------------------------------------------------- #
+@dataclass
+class PreprocessConfig:
+    """Locked preprocessing config saved with the final model.
+
+    Eval/inference path is fully deterministic: resize to image_size, ToTensor,
+    Normalize with the given mean/std. No augmentation at eval time.
+    """
+    image_size: int = 224
+    mean: List[float] = field(default_factory=lambda: list(IMAGENET_MEAN))
+    std: List[float] = field(default_factory=lambda: list(IMAGENET_STD))
+    interpolation: str = "bilinear"  # 'bilinear' (torchvision) or 'bicubic' (timm a1/a2/a3)
+
+    def to_dict(self):
+        return asdict(self)
+
+    @staticmethod
+    def from_dict(d):
+        fields = {"image_size", "mean", "std", "interpolation"}
+        return PreprocessConfig(**{k: v for k, v in d.items() if k in fields})
+
+
+def _interp(name):
+    from torchvision.transforms import InterpolationMode
+    return {"bilinear": InterpolationMode.BILINEAR,
+            "bicubic": InterpolationMode.BICUBIC}[name]
+
+
+def build_eval_transform(pp: PreprocessConfig):
+    """Deterministic eval/inference transform (NO augmentation)."""
+    import torchvision.transforms as T
+    return T.Compose([
+        T.Resize((pp.image_size, pp.image_size), interpolation=_interp(pp.interpolation)),
+        T.ToTensor(),
+        T.Normalize(pp.mean, pp.std),
+    ])
+
+
+def build_train_transform(pp: PreprocessConfig, policy: str = "medical_default"):
+    """Training augmentation. Medically plausible ultrasound augmentations only.
+
+    Policies:
+      none            : eval transform (no augmentation) — baseline ablation.
+      flip_only       : horizontal flip only.
+      medical_default : flip + mild affine(rot<=10,trans5%,scale0.9-1.1) + mild
+                        brightness/contrast + occasional light gaussian blur.
+      medical_strong  : medical_default + mild speckle/gaussian noise +
+                        narrow random-resized-crop (scale 0.8-1.0).
+      clahe           : medical_default + CLAHE applied as preprocessing (ablation).
+
+    Explicitly AVOIDED: vertical flip, large rotation (>15deg), aggressive crop
+    (<0.8 scale), shear, heavy blur, any color/HSV jitter beyond mild
+    brightness/contrast — all of which distort ultrasound texture or nodule
+    morphology (per MediAug arXiv:2504.18983 and thyroid-US best practice).
+    """
+    import torch
+    import torchvision.transforms as T
+    interp = _interp(pp.interpolation)
+    norm = T.Normalize(pp.mean, pp.std)
+
+    if policy == "none":
+        return build_eval_transform(pp)
+
+    if policy == "flip_only":
+        return T.Compose([
+            T.Resize((pp.image_size, pp.image_size), interpolation=interp),
+            T.RandomHorizontalFlip(0.5),
+            T.ToTensor(), norm,
+        ])
+
+    if policy == "medical_default":
+        return T.Compose([
+            T.Resize((pp.image_size, pp.image_size), interpolation=interp),
+            T.RandomHorizontalFlip(0.5),
+            T.RandomApply([T.RandomAffine(degrees=10, translate=(0.05, 0.05),
+                                          scale=(0.9, 1.1), interpolation=interp)], p=0.5),
+            T.ColorJitter(brightness=0.15, contrast=0.15),
+            T.RandomApply([T.GaussianBlur(3, sigma=(0.1, 1.0))], p=0.2),
+            T.ToTensor(), norm,
+        ])
+
+    if policy == "medical_strong":
+        class AddSpeckle:
+            def __init__(self, sigma=0.05, p=0.2):
+                self.sigma, self.p = sigma, p
+            def __call__(self, x):
+                if random.random() < self.p:
+                    return x + x * (self.sigma * torch.randn_like(x))
+                return x
+        return T.Compose([
+            T.RandomResizedCrop(pp.image_size, scale=(0.8, 1.0), ratio=(0.9, 1.1),
+                                interpolation=interp),
+            T.RandomHorizontalFlip(0.5),
+            T.RandomApply([T.RandomAffine(degrees=10, translate=(0.05, 0.05),
+                                          scale=(0.9, 1.1), interpolation=interp)], p=0.5),
+            T.ColorJitter(brightness=0.15, contrast=0.15),
+            T.RandomApply([T.GaussianBlur(3, sigma=(0.1, 1.0))], p=0.2),
+            T.ToTensor(),
+            AddSpeckle(sigma=0.05, p=0.2),
+            norm,
+        ])
+
+    if policy == "clahe":
+        from PIL import Image
+        class CLAHE:
+            def __call__(self, img):
+                import numpy as _np
+                try:
+                    import cv2
+                    arr = _np.asarray(img.convert("L"))
+                    cl = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8)).apply(arr)
+                    return Image.fromarray(cl).convert("RGB")
+                except Exception:
+                    return img
+        return T.Compose([
+            CLAHE(),
+            T.Resize((pp.image_size, pp.image_size), interpolation=interp),
+            T.RandomHorizontalFlip(0.5),
+            T.RandomApply([T.RandomAffine(degrees=10, translate=(0.05, 0.05),
+                                          scale=(0.9, 1.1), interpolation=interp)], p=0.5),
+            T.ColorJitter(brightness=0.15, contrast=0.15),
+            T.ToTensor(), norm,
+        ])
+
+    raise ValueError(f"Unknown augmentation policy: {policy}")
+
+
+# --------------------------------------------------------------------------- #
+# Dataset
+# --------------------------------------------------------------------------- #
+class ThyroidImageFolder:
+    """Lightweight ImageFolder that also returns the image filename id.
+
+    Layout: <root>/<Benign|Malignant>/<id>.png
+    Returns (tensor, label, image_id).
+    """
+    def __init__(self, root, transform):
+        from PIL import Image
+        self.Image = Image
+        self.root = Path(root)
+        self.transform = transform
+        self.samples: List[Tuple[Path, int, str]] = []
+        for cls, idx in CLASS_TO_IDX.items():
+            d = self.root / cls
+            if d.is_dir():
+                for p in sorted(d.glob("*.png")):
+                    self.samples.append((p, idx, p.stem))
+        if not self.samples:
+            raise RuntimeError(f"No images found under {root}")
+        self.targets = [s[1] for s in self.samples]
+
+    def __len__(self):
+        return len(self.samples)
+
+    def __getitem__(self, i):
+        path, label, img_id = self.samples[i]
+        with self.Image.open(path) as im:
+            im = im.convert("RGB")
+            x = self.transform(im)
+        return x, label, img_id
+
+
+def class_counts(targets):
+    n_pos = int(sum(1 for t in targets if t == 1))
+    n_neg = int(sum(1 for t in targets if t == 0))
+    return n_neg, n_pos
+
+
+# --------------------------------------------------------------------------- #
+# Model
+# --------------------------------------------------------------------------- #
+def build_model(backbone: str, freeze_stage: int = 0, dropout: float = 0.0):
+    """Build a single-logit ResNet-18 classifier.
+
+    backbone:
+      'torchvision'           -> torchvision resnet18 ImageNet1K_V1
+      'timm:resnet18.a1_in1k' -> any timm tag after 'timm:'
+    freeze_stage: 0 = full fine-tune; 1 = freeze stem+layer1; 2 = +layer2; etc.
+    Returns (model, preprocess_config).
+    """
+    import torch
+    import torch.nn as nn
+
+    if backbone == "torchvision":
+        from torchvision.models import resnet18, ResNet18_Weights
+        weights = ResNet18_Weights.IMAGENET1K_V1
+        model = resnet18(weights=weights)
+        in_f = model.fc.in_features
+        model.fc = nn.Sequential(nn.Dropout(dropout), nn.Linear(in_f, 1)) if dropout > 0 \
+            else nn.Linear(in_f, 1)
+        pp = PreprocessConfig(image_size=224, mean=list(IMAGENET_MEAN),
+                              std=list(IMAGENET_STD), interpolation="bilinear")
+        _freeze_resnet(model, freeze_stage)
+        return model, pp
+
+    if backbone.startswith("timm:"):
+        import timm
+        from timm.data import resolve_model_data_config
+        tag = backbone.split("timm:", 1)[1]
+        model = timm.create_model(tag, pretrained=True, num_classes=1, drop_rate=dropout)
+        cfg = resolve_model_data_config(model)
+        mean = list(cfg.get("mean", IMAGENET_MEAN))
+        std = list(cfg.get("std", IMAGENET_STD))
+        interp = cfg.get("interpolation", "bicubic")
+        size = cfg.get("input_size", (3, 224, 224))[-1]
+        pp = PreprocessConfig(image_size=int(size), mean=mean, std=std,
+                              interpolation=interp if interp in ("bilinear", "bicubic") else "bicubic")
+        _freeze_timm_resnet(model, freeze_stage)
+        return model, pp
+
+    raise ValueError(f"Unknown backbone: {backbone}")
+
+
+def _freeze_resnet(model, stage):
+    if stage <= 0:
+        return
+    to_freeze = []
+    if stage >= 1:
+        to_freeze += [model.conv1, model.bn1, model.layer1]
+    if stage >= 2:
+        to_freeze += [model.layer2]
+    if stage >= 3:
+        to_freeze += [model.layer3]
+    for m in to_freeze:
+        for p in m.parameters():
+            p.requires_grad = False
+
+
+def _freeze_timm_resnet(model, stage):
+    if stage <= 0:
+        return
+    name_prefixes = []
+    if stage >= 1:
+        name_prefixes += ["conv1", "bn1", "layer1"]
+    if stage >= 2:
+        name_prefixes += ["layer2"]
+    if stage >= 3:
+        name_prefixes += ["layer3"]
+    for n, p in model.named_parameters():
+        if any(n.startswith(pref) for pref in name_prefixes):
+            p.requires_grad = False
+
+
+# --------------------------------------------------------------------------- #
+# Loss
+# --------------------------------------------------------------------------- #
+def build_loss(name: str, pos_weight: Optional[float], focal_gamma: float = 2.0,
+               focal_alpha: float = 0.5):
+    import torch
+    import torch.nn as nn
+
+    if name == "bce":
+        pw = torch.tensor([pos_weight]) if pos_weight is not None else None
+        return nn.BCEWithLogitsLoss(pos_weight=pw)
+
+    if name == "focal":
+        class FocalLoss(nn.Module):
+            def __init__(self, gamma, alpha):
+                super().__init__()
+                self.gamma, self.alpha = gamma, alpha
+            def forward(self, logits, targets):
+                logits = logits.view(-1)
+                targets = targets.view(-1).float()
+                p = torch.sigmoid(logits)
+                ce = nn.functional.binary_cross_entropy_with_logits(logits, targets, reduction="none")
+                p_t = p * targets + (1 - p) * (1 - targets)
+                alpha_t = self.alpha * targets + (1 - self.alpha) * (1 - targets)
+                loss = alpha_t * (1 - p_t) ** self.gamma * ce
+                return loss.mean()
+        return FocalLoss(focal_gamma, focal_alpha)
+
+    raise ValueError(f"Unknown loss: {name}")
+
+
+# --------------------------------------------------------------------------- #
+# Inference: collect logits/probs/labels/ids
+# --------------------------------------------------------------------------- #
+def collect_logits(model, loader, device, amp=False):
+    import torch
+    model.eval()
+    logits_all, labels_all, ids_all = [], [], []
+    use_ac = amp and device == "cuda"
+    with torch.no_grad():
+        for x, y, ids in loader:
+            x = x.to(device, non_blocking=True)
+            if use_ac:
+                with torch.autocast(device_type="cuda", dtype=torch.float16):
+                    out = model(x).view(-1)
+            else:
+                out = model(x).view(-1)
+            logits_all.append(out.float().cpu().numpy())
+            labels_all.append(np.asarray(y))
+            ids_all.extend(list(ids))
+    return (np.concatenate(logits_all), np.concatenate(labels_all).astype(int), ids_all)
+
+
+# --------------------------------------------------------------------------- #
+# Calibration (temperature scaling)
+# --------------------------------------------------------------------------- #
+def fit_temperature(val_logits: np.ndarray, val_labels: np.ndarray) -> float:
+    """Fit single-parameter temperature on validation logits (minimize NLL)."""
+    import torch
+    import torch.nn as nn
+    logits = torch.tensor(val_logits, dtype=torch.float32)
+    labels = torch.tensor(val_labels, dtype=torch.float32)
+    T = nn.Parameter(torch.ones(1))
+    opt = torch.optim.LBFGS([T], lr=0.01, max_iter=200)
+    bce = nn.BCEWithLogitsLoss()
+
+    def closure():
+        opt.zero_grad()
+        loss = bce(logits / T.clamp(min=1e-3), labels)
+        loss.backward()
+        return loss
+    opt.step(closure)
+    return float(T.detach().clamp(min=1e-3).item())
+
+
+def apply_temperature(logits: np.ndarray, T: float) -> np.ndarray:
+    return 1.0 / (1.0 + np.exp(-(logits / T)))
+
+
+def sigmoid(logits: np.ndarray) -> np.ndarray:
+    return 1.0 / (1.0 + np.exp(-logits))
+
+
+# --------------------------------------------------------------------------- #
+# Calibration metrics
+# --------------------------------------------------------------------------- #
+def expected_calibration_error(y_true, y_prob, n_bins=15):
+    y_true = np.asarray(y_true); y_prob = np.asarray(y_prob)
+    bins = np.linspace(0, 1, n_bins + 1)
+    ece = 0.0
+    for lo, hi in zip(bins[:-1], bins[1:]):
+        m = (y_prob > lo) & (y_prob <= hi)
+        if m.sum() > 0:
+            ece += (m.sum() / len(y_prob)) * abs(y_true[m].mean() - y_prob[m].mean())
+    return float(ece)
+
+
+def brier(y_true, y_prob):
+    from sklearn.metrics import brier_score_loss
+    return float(brier_score_loss(np.asarray(y_true), np.asarray(y_prob)))
+
+
+# --------------------------------------------------------------------------- #
+# Thresholding
+# --------------------------------------------------------------------------- #
+def threshold_for_sensitivity(y_true, y_prob, target_sens=0.95):
+    """Highest-specificity threshold achieving sensitivity >= target on these data.
+
+    Returns (threshold, achieved_sens, achieved_spec, achievable_flag).
+    """
+    from sklearn.metrics import roc_curve
+    y_true = np.asarray(y_true); y_prob = np.asarray(y_prob)
+    fpr, tpr, thr = roc_curve(y_true, y_prob)
+    spec = 1 - fpr
+    ok = tpr >= target_sens
+    if ok.any():
+        cand = np.where(ok)[0]
+        best = cand[np.argmax(spec[cand])]
+        return float(thr[best]), float(tpr[best]), float(spec[best]), True
+    best = int(np.argmax(tpr))
+    return float(thr[best]), float(tpr[best]), float(spec[best]), False
+
+
+def youden_threshold(y_true, y_prob):
+    from sklearn.metrics import roc_curve
+    fpr, tpr, thr = roc_curve(np.asarray(y_true), np.asarray(y_prob))
+    j = tpr - fpr
+    best = int(np.argmax(j))
+    return float(thr[best]), float(tpr[best]), float(1 - fpr[best])
+
+
+# --------------------------------------------------------------------------- #
+# Metrics + bootstrap CIs
+# --------------------------------------------------------------------------- #
+def point_metrics(y_true, y_prob, thr):
+    from sklearn.metrics import roc_auc_score, f1_score, accuracy_score
+    y_true = np.asarray(y_true); y_prob = np.asarray(y_prob)
+    pred = (y_prob >= thr).astype(int)
+    tp = int(((pred == 1) & (y_true == 1)).sum())
+    tn = int(((pred == 0) & (y_true == 0)).sum())
+    fp = int(((pred == 1) & (y_true == 0)).sum())
+    fn = int(((pred == 0) & (y_true == 1)).sum())
+    sens = tp / (tp + fn) if (tp + fn) else float("nan")
+    spec = tn / (tn + fp) if (tn + fp) else float("nan")
+    ppv = tp / (tp + fp) if (tp + fp) else float("nan")
+    npv = tn / (tn + fn) if (tn + fn) else float("nan")
+    return {
+        "auroc": float(roc_auc_score(y_true, y_prob)),
+        "accuracy": float(accuracy_score(y_true, pred)),
+        "sensitivity": float(sens),
+        "specificity": float(spec),
+        "ppv": float(ppv),
+        "npv": float(npv),
+        "f1": float(f1_score(y_true, pred, zero_division=0)),
+        "brier": brier(y_true, y_prob),
+        "ece": expected_calibration_error(y_true, y_prob),
+        "tp": tp, "tn": tn, "fp": fp, "fn": fn,
+        "threshold": float(thr),
+        "n": int(len(y_true)),
+        "n_pos": int((y_true == 1).sum()),
+        "n_neg": int((y_true == 0).sum()),
+    }
+
+
+def bootstrap_ci(y_true, y_prob, thr, n_boot=2000, seed=42):
+    """Stratified bootstrap 95% CIs for AUROC, sens, spec, ppv, npv, acc, f1."""
+    from sklearn.metrics import roc_auc_score, f1_score, accuracy_score
+    y_true = np.asarray(y_true); y_prob = np.asarray(y_prob)
+    rng = np.random.default_rng(seed)
+    pos = np.where(y_true == 1)[0]
+    neg = np.where(y_true == 0)[0]
+    keys = ["auroc", "sensitivity", "specificity", "ppv", "npv", "accuracy", "f1"]
+    acc = {k: [] for k in keys}
+    for _ in range(n_boot):
+        idx = np.concatenate([rng.choice(pos, len(pos), replace=True),
+                              rng.choice(neg, len(neg), replace=True)])
+        yt = y_true[idx]; yp = y_prob[idx]
+        pred = (yp >= thr).astype(int)
+        try:
+            acc["auroc"].append(roc_auc_score(yt, yp))
+        except Exception:
+            acc["auroc"].append(np.nan)
+        tp = ((pred == 1) & (yt == 1)).sum(); tn = ((pred == 0) & (yt == 0)).sum()
+        fp = ((pred == 1) & (yt == 0)).sum(); fn = ((pred == 0) & (yt == 1)).sum()
+        acc["sensitivity"].append(tp / (tp + fn) if (tp + fn) else np.nan)
+        acc["specificity"].append(tn / (tn + fp) if (tn + fp) else np.nan)
+        acc["ppv"].append(tp / (tp + fp) if (tp + fp) else np.nan)
+        acc["npv"].append(tn / (tn + fn) if (tn + fn) else np.nan)
+        acc["accuracy"].append(accuracy_score(yt, pred))
+        acc["f1"].append(f1_score(yt, pred, zero_division=0))
+    out = {}
+    for k in keys:
+        v = np.asarray(acc[k], dtype=float)
+        out[k] = (float(np.nanpercentile(v, 2.5)), float(np.nanpercentile(v, 97.5)))
+    return out
+
+
+# --------------------------------------------------------------------------- #
+# JSON helpers
+# --------------------------------------------------------------------------- #
+def save_json(obj, path):
+    Path(path).parent.mkdir(parents=True, exist_ok=True)
+    with open(path, "w") as f:
+        json.dump(obj, f, indent=2)
+
+
+def load_json(path):
+    with open(path) as f:
+        return json.load(f)

train.py

@@ -0,0 +1,295 @@
+#!/usr/bin/env python
+"""
+Reproducible training for the ResNet-18 thyroid ultrasound malignancy classifier.
+
+Trains on Train only; selects the best checkpoint by **validation AUROC**.
+Logs everything to Trackio (losses, val AUROC/sens/spec/PPV/NPV/ECE/Brier, LR,
+epoch, hyperparameters, env info) and emits trackio.alert() at decision points.
+
+Single-command reproduction:
+    python train.py --config configs/final_config.yaml
+
+All CLI args override config values. The exact command line, resolved config,
+seed, package versions and hardware info are saved to the output dir.
+
+Dataset is loaded directly from the Train/Valid/Test FOLDER structure of the
+Hub repo (NOT the flattened datasets-viewer 'train' split), so the predefined
+splits are respected.
+"""
+import argparse
+import os
+import sys
+import time
+from pathlib import Path
+
+import numpy as np
+import yaml
+
+import thyroid_lib as L
+
+
+def parse_args():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--config", default=None, help="YAML config path")
+    ap.add_argument("--dataset_id", default="Johnyquest7/TN5000-thyroid-nodule-classification")
+    ap.add_argument("--data_dir", default=None, help="Local TN5000 dir; else downloaded from Hub")
+    ap.add_argument("--output_dir", default="run_out")
+    ap.add_argument("--backbone", default="timm:resnet18.a1_in1k")
+    ap.add_argument("--freeze_stage", type=int, default=0)
+    ap.add_argument("--dropout", type=float, default=0.0)
+    ap.add_argument("--aug_policy", default="medical_default")
+    ap.add_argument("--loss", default="bce", choices=["bce", "focal"])
+    ap.add_argument("--focal_gamma", type=float, default=2.0)
+    ap.add_argument("--focal_alpha", type=float, default=0.5)
+    ap.add_argument("--imbalance", default="pos_weight",
+                    choices=["pos_weight", "none", "sampler"])
+    ap.add_argument("--optimizer", default="adamw", choices=["adamw", "sgd"])
+    ap.add_argument("--lr", type=float, default=2e-4)
+    ap.add_argument("--weight_decay", type=float, default=1e-4)
+    ap.add_argument("--batch_size", type=int, default=32)
+    ap.add_argument("--epochs", type=int, default=40)
+    ap.add_argument("--scheduler", default="cosine", choices=["cosine", "plateau", "none"])
+    ap.add_argument("--warmup_epochs", type=int, default=2)
+    ap.add_argument("--early_stop_patience", type=int, default=8)
+    ap.add_argument("--amp", action="store_true", default=True)
+    ap.add_argument("--no_amp", dest="amp", action="store_false")
+    ap.add_argument("--num_workers", type=int, default=4)
+    ap.add_argument("--seed", type=int, default=42)
+    ap.add_argument("--strict_determinism", action="store_true", default=True)
+    ap.add_argument("--no_strict_determinism", dest="strict_determinism", action="store_false")
+    ap.add_argument("--trackio_project", default="agentic_thyroid_resnet18")
+    ap.add_argument("--trackio_space_id", default="Johnyquest7/Trakio_agentic_thyroid")
+    ap.add_argument("--trackio_dataset_id", default="Johnyquest7/Trakio_agentic_thyroid_dataset")
+    ap.add_argument("--run_name", default=None)
+    ap.add_argument("--no_trackio", action="store_true")
+    return ap.parse_args()
+
+
+def merge_config(args):
+    if args.config and Path(args.config).exists():
+        with open(args.config) as f:
+            cfg = yaml.safe_load(f) or {}
+        passed = {a.split("=")[0].lstrip("-").replace("-", "_") for a in sys.argv[1:] if a.startswith("--")}
+        for k, v in cfg.items():
+            if k not in passed and hasattr(args, k):
+                setattr(args, k, v)
+    return args
+
+
+def main():
+    args = parse_args()
+    args = merge_config(args)
+    out = Path(args.output_dir)
+    out.mkdir(parents=True, exist_ok=True)
+
+    import torch
+    from torch.utils.data import DataLoader, WeightedRandomSampler
+    from sklearn.metrics import roc_auc_score
+    import torch.nn.functional as F
+
+    L.set_determinism(args.seed, strict=args.strict_determinism)
+    env = L.collect_env_info()
+    device = "cuda" if torch.cuda.is_available() else "cpu"
+
+    if args.data_dir:
+        data_dir = Path(args.data_dir)
+    else:
+        from huggingface_hub import snapshot_download
+        data_dir = Path(snapshot_download(repo_id=args.dataset_id, repo_type="dataset",
+                                          local_dir=str(out / "_data"),
+                                          allow_patterns=["Train/**", "Valid/**", "Test/**"]))
+
+    model, pp = L.build_model(args.backbone, freeze_stage=args.freeze_stage, dropout=args.dropout)
+    model = model.to(device)
+
+    train_tf = L.build_train_transform(pp, args.aug_policy)
+    eval_tf = L.build_eval_transform(pp)
+    train_ds = L.ThyroidImageFolder(data_dir / "Train", train_tf)
+    valid_ds = L.ThyroidImageFolder(data_dir / "Valid", eval_tf)
+
+    n_neg, n_pos = L.class_counts(train_ds.targets)
+    pos_weight = (n_neg / n_pos) if (args.imbalance == "pos_weight" and n_pos) else None
+
+    g = torch.Generator(); g.manual_seed(args.seed)
+    if args.imbalance == "sampler":
+        cw = np.array([1.0 / n_neg, 1.0 / n_pos])
+        sw = np.array([cw[t] for t in train_ds.targets])
+        sampler = WeightedRandomSampler(torch.tensor(sw, dtype=torch.double),
+                                        num_samples=len(sw), replacement=True, generator=g)
+        train_loader = DataLoader(train_ds, batch_size=args.batch_size, sampler=sampler,
+                                  num_workers=args.num_workers, worker_init_fn=L.seed_worker,
+                                  generator=g, pin_memory=(device == "cuda"), drop_last=False)
+    else:
+        train_loader = DataLoader(train_ds, batch_size=args.batch_size, shuffle=True,
+                                  num_workers=args.num_workers, worker_init_fn=L.seed_worker,
+                                  generator=g, pin_memory=(device == "cuda"), drop_last=False)
+    valid_loader = DataLoader(valid_ds, batch_size=64, shuffle=False,
+                              num_workers=args.num_workers, pin_memory=(device == "cuda"))
+
+    criterion = L.build_loss(args.loss, pos_weight if args.loss == "bce" else None,
+                             args.focal_gamma, args.focal_alpha).to(device)
+
+    params = [p for p in model.parameters() if p.requires_grad]
+    if args.optimizer == "adamw":
+        optimizer = torch.optim.AdamW(params, lr=args.lr, weight_decay=args.weight_decay)
+    else:
+        optimizer = torch.optim.SGD(params, lr=args.lr, momentum=0.9,
+                                    weight_decay=args.weight_decay, nesterov=True)
+
+    if args.scheduler == "cosine":
+        scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=args.epochs)
+    elif args.scheduler == "plateau":
+        scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode="max",
+                                                               factor=0.5, patience=3)
+    else:
+        scheduler = None
+
+    scaler = torch.amp.GradScaler("cuda", enabled=(args.amp and device == "cuda"))
+
+    run_name = args.run_name or (
+        f"{args.backbone.replace('timm:','tm_').replace('.','_')}_lr{args.lr}_wd{args.weight_decay}"
+        f"_bs{args.batch_size}_{args.aug_policy}_{args.loss}_{args.imbalance}_fz{args.freeze_stage}")
+
+    resolved = vars(args).copy()
+    resolved.update({"pos_weight": pos_weight, "n_train_neg": n_neg, "n_train_pos": n_pos,
+                     "preprocess": pp.to_dict(), "device": device})
+    L.save_json(resolved, out / "resolved_config.json")
+    L.save_json(env, out / "env_info.json")
+    with open(out / "command_line.txt", "w") as f:
+        f.write("python " + " ".join(sys.argv) + "\n")
+    with open(out / "config_used.yaml", "w") as f:
+        yaml.safe_dump({k: v for k, v in resolved.items() if not isinstance(v, dict) or k == "preprocess"}, f)
+
+    use_trackio = not args.no_trackio
+    if use_trackio:
+        import trackio
+        try:
+            from trackio.alerts import AlertLevel
+            _LV = {"info": AlertLevel.INFO, "warn": AlertLevel.WARN, "error": AlertLevel.ERROR}
+        except Exception:
+            _LV = {"info": "info", "warn": "warn", "error": "error"}
+
+        def _alert(title, text, level="info"):
+            try:
+                trackio.alert(title, text, level=_LV.get(level, level))
+            except Exception as e:
+                print(f"[alert-failed] {title}: {text} ({e})", flush=True)
+
+        trackio.init(project=args.trackio_project, name=run_name,
+                     space_id=args.trackio_space_id,
+                     dataset_id=args.trackio_dataset_id,
+                     config={k: v for k, v in resolved.items() if k != "preprocess"})
+        _alert("Run started",
+                      f"{run_name} | backbone={args.backbone} loss={args.loss} "
+                      f"imb={args.imbalance} lr={args.lr} wd={args.weight_decay} "
+                      f"bs={args.batch_size} aug={args.aug_policy} fz={args.freeze_stage} "
+                      f"pos_weight={pos_weight} device={env.get('gpu_name')}",
+                      "info")
+
+    best_auroc = -1.0
+    best_epoch = -1
+    epochs_no_improve = 0
+    history = []
+    global_step = 0
+    n_warmup_steps = args.warmup_epochs * max(1, len(train_loader))
+    base_lr = args.lr
+
+    for epoch in range(args.epochs):
+        model.train()
+        t0 = time.time()
+        running = 0.0
+        for x, y, _ in train_loader:
+            x = x.to(device, non_blocking=True)
+            y = y.to(device, non_blocking=True).float()
+            if global_step < n_warmup_steps and args.warmup_epochs > 0:
+                for pg in optimizer.param_groups:
+                    pg["lr"] = base_lr * (global_step + 1) / n_warmup_steps
+            optimizer.zero_grad(set_to_none=True)
+            with torch.autocast(device_type="cuda", dtype=torch.float16,
+                                enabled=(args.amp and device == "cuda")):
+                out_logits = model(x).view(-1)
+                loss = criterion(out_logits, y)
+            scaler.scale(loss).backward()
+            scaler.step(optimizer)
+            scaler.update()
+            running += loss.item() * x.size(0)
+            global_step += 1
+        train_loss = running / len(train_ds)
+
+        val_logits, val_labels, _ = L.collect_logits(model, valid_loader, device, amp=args.amp)
+        val_probs = L.sigmoid(val_logits)
+        val_auroc = float(roc_auc_score(val_labels, val_probs))
+        val_loss = float(F.binary_cross_entropy_with_logits(
+            torch.tensor(val_logits), torch.tensor(val_labels, dtype=torch.float32)).item())
+        m = L.point_metrics(val_labels, val_probs, 0.5)
+        cur_lr = optimizer.param_groups[0]["lr"]
+
+        if scheduler is not None:
+            if args.scheduler == "plateau":
+                scheduler.step(val_auroc)
+            elif global_step >= n_warmup_steps:
+                scheduler.step()
+
+        row = {"epoch": epoch, "train_loss": train_loss, "val_loss": val_loss,
+               "val_auroc": val_auroc, "val_sens@0.5": m["sensitivity"],
+               "val_spec@0.5": m["specificity"], "val_ppv@0.5": m["ppv"],
+               "val_npv@0.5": m["npv"], "val_ece@0.5": m["ece"], "val_brier": m["brier"],
+               "lr": cur_lr, "epoch_time_s": round(time.time() - t0, 1)}
+        history.append(row)
+        print(f"[epoch {epoch}] train_loss={train_loss:.4f} val_loss={val_loss:.4f} "
+              f"val_auroc={val_auroc:.4f} lr={cur_lr:.2e} ({row['epoch_time_s']}s)", flush=True)
+
+        if use_trackio:
+            trackio.log({"train_loss": train_loss, "val_loss": val_loss, "val_auroc": val_auroc,
+                         "val_sensitivity": m["sensitivity"], "val_specificity": m["specificity"],
+                         "val_ppv": m["ppv"], "val_npv": m["npv"], "val_ece": m["ece"],
+                         "val_brier": m["brier"], "lr": cur_lr, "epoch": epoch})
+
+        improved = val_auroc > best_auroc + 1e-5
+        if improved:
+            best_auroc = val_auroc
+            best_epoch = epoch
+            epochs_no_improve = 0
+            torch.save({"model_state": model.state_dict(),
+                        "backbone": args.backbone, "freeze_stage": args.freeze_stage,
+                        "dropout": args.dropout, "preprocess": pp.to_dict(),
+                        "epoch": epoch, "val_auroc": val_auroc},
+                       out / "best_model.pt")
+            L.save_json({"best_epoch": best_epoch, "best_val_auroc": best_auroc,
+                         "val_metrics_at_0.5": m}, out / "best_val_summary.json")
+        else:
+            epochs_no_improve += 1
+
+        if use_trackio and (not np.isfinite(train_loss) or train_loss > 1e3):
+            _alert("Training diverged",
+                          f"train_loss={train_loss} at epoch {epoch} — lr likely too high, try x0.1",
+                          "error")
+
+        if epochs_no_improve >= args.early_stop_patience:
+            print(f"Early stopping at epoch {epoch} (no val AUROC improvement for "
+                  f"{args.early_stop_patience} epochs).", flush=True)
+            if use_trackio:
+                _alert("Early stopping",
+                              f"No val AUROC gain for {args.early_stop_patience} epochs; "
+                              f"best={best_auroc:.4f} @ epoch {best_epoch}. Consider lr x0.5.",
+                              "warn")
+            break
+
+    L.save_json(history, out / "history.json")
+    L.save_json({"best_val_auroc": best_auroc, "best_epoch": best_epoch,
+                 "run_name": run_name, "backbone": args.backbone},
+                out / "final_summary.json")
+
+    if use_trackio:
+        trackio.log({"best_val_auroc": best_auroc, "best_epoch": best_epoch})
+        _alert("Run complete",
+                      f"{run_name}: best val AUROC={best_auroc:.4f} @ epoch {best_epoch}",
+                      "info")
+        trackio.finish()
+
+    print(f"DONE best_val_auroc={best_auroc:.4f} best_epoch={best_epoch}", flush=True)
+    return best_auroc
+
+
+if __name__ == "__main__":
+    main()