LOG.md

# Experiment Log — Agentic Thyroid ResNet-18

Chronological, decision-by-decision record for reproducibility and journal review.

## Provenance

- **Experiment date (UTC):** 2026-06-05
- **Dataset:** `Johnyquest7/TN5000-thyroid-nodule-classification`
  - Commit SHA: `73d6c0713a89c8c07125fe6ffc5956be60e9853d`
  - Loaded **directly from the Train/Valid/Test folder structure** (NOT the
    datasets-viewer flattened `train` config, which merges all 5,000 rows),
    so the predefined splits are respected.
- **Compute:** Hugging Face GPU sandbox, NVIDIA A10G (24 GB), CUDA 13.0, cuDNN 9.2.
- **Key packages:** torch 2.12.0+cu130, torchvision 0.27.0+cu130, timm 1.0.27,
  scikit-learn 1.9.0, numpy 2.4.6, trackio 0.26.0 (see `configs/env_info.json`).
- **Global seed:** 42. Strict determinism: `torch.use_deterministic_algorithms(True)`,
  cuDNN deterministic, `CUBLAS_WORKSPACE_CONFIG=:4096:8`, seeded DataLoader workers.
- **Positive class:** Malignant (label 1).
- **Experiment tracking:** Trackio project `agentic_thyroid_resnet18`, dashboard
  Space `Johnyquest7/Trakio_agentic_thyroid`, storage dataset
  `Johnyquest7/Trakio_agentic_thyroid_dataset`.

## Exact split usage

| Split | n | Use |
|-------|--:|-----|
| Train | 3,500 | **Training only.** |
| Valid | 500 | **Model selection (val AUROC), calibration, threshold selection.** |
| Test | 1,000 | **Final locked evaluation, exactly once**, after model+calibration+threshold were frozen. |

## Class distribution

| Split | Benign | Malignant | Malignant % | Ratio (M:B) |
|-------|-------:|----------:|------------:|------------:|
| Train | 1,032 | 2,468 | 70.5% | 2.39 : 1 |
| Valid |   125 |   375 | 75.0% | 3.00 : 1 |
| Test  |   269 |   731 | 73.1% | 2.72 : 1 |

Data audit (`data_exploration_report.md`): **0 corrupt images**, all 224×224 RGB
PNG; **0 cross-split exact-pixel duplicates**, **0 filename-ID overlaps**, **0
label conflicts**; per-split mean intensity ≈ 81.4 (std ≈ 19.4) — no distribution
shift. Conclusion: **no detectable leakage; splits are clean and separate.**

## Literature-informed augmentation rationale

Augmentations were restricted to medically plausible B-mode ultrasound transforms
(MediAug arXiv:2504.18983 + thyroid-US practice):
- **Kept:** horizontal flip (thyroid is bilaterally symmetric), small rotation
  (≤10°), mild affine translate (5%) / scale (0.9–1.1), mild brightness/contrast
  (±15%, simulates gain/TGC), light Gaussian blur, and (in the `medical_strong`
  ablation) narrow random-resized-crop (0.8–1.0) + mild speckle noise.
- **Explicitly avoided:** vertical flip (US depth axis is physically meaningful),
  large rotation/shear (distorts taller-than-wide / margin morphology — TI-RADS
  malignancy cues), aggressive crop (<0.8, can remove the nodule), and any
  color/HSV jitter (images are grayscale).

Ablation result (val AUROC): `medical_default` **0.9712–0.9756** > `flip_only`
**0.9637** > `medical_strong` **0.9609**. The literature-default policy won.

## Model variants tried

- `torchvision` ResNet-18 (ImageNet1K_V1, bilinear/256→224 preprocessing).
- `timm:resnet18.a1_in1k` (A1 recipe, bicubic, crop_pct 0.95) — **selected**.
- `timm:resnet18.a2_in1k` (A2 recipe).
- Fine-tune depth: full fine-tune vs freeze stem+layer1 (`freeze_stage=1`).

## Hyperparameter sweep (14 trials, one-factor-at-a-time around a literature-informed center)

Center: timm a1, lr 2e-4, wd 1e-4, bs 32, `medical_default`, `pos_weight`,
full fine-tune, BCE, AdamW, cosine, ≤40 epochs, early-stop(8). All runs logged to
Trackio. **Selection metric: validation AUROC.** All 14 trials completed (rc=0).

| Rank | Run | Change vs center | Val AUROC | Best epoch |
|-----:|-----|------------------|----------:|-----------:|
| 1 | **c12_loss_focal** | **focal γ=1.0, imbalance=none** | **0.9756** | 6 |
| 2 | c09_imb_none | imbalance=none (BCE) | 0.9739 | 6 |
| 3 | c01_backbone_torchvision | torchvision backbone | 0.9731 | 7 |
| 4 | c03_lr_1e-4 | lr 1e-4 | 0.9721 | 11 |
| 5 | c06_bs_64 | batch size 64 | 0.9717 | 8 |
| 6 | c05_wd_1e-3 | weight decay 1e-3 | 0.9712 | 9 |
| 6 | c00_center_a1 | center config | 0.9712 | 6 |
| 8 | c10_imb_sampler | weighted sampler | 0.9697 | 13 |
| 9 | c02_backbone_a2 | a2 backbone | 0.9693 | 6 |
| 10 | c04_lr_5e-4 | lr 5e-4 | 0.9675 | 9 |
| 11 | c11_freeze1 | freeze stem+layer1 | 0.9672 | 6 |
| 12 | c13_lr1e-4_wd1e-3_drop | lr1e-4+wd1e-3+dropout0.2 | 0.9657 | 11 |
| 13 | c07_aug_flip_only | flip-only aug | 0.9637 | 6 |
| 14 | c08_aug_strong | strong aug | 0.9609 | 8 |

Findings: (1) For this mild (~70/30) imbalance, **focal loss (γ=1.0) and no extra
reweighting beat class-weighted BCE and weighted sampling** — heavy reweighting
slightly hurt, consistent with the literature. (2) `medical_default` augmentation
is the sweet spot. (3) **Full fine-tune > freezing.** (4) Backbones were close
(a1 ≈ torchvision ≈ a2). Full per-run details: `results/tables/sweep_leaderboard.json`.

No excessive trial count was used (14 one-factor trials) to avoid overfitting the
500-image validation set.

## Selected run

**c12_loss_focal** — `timm:resnet18.a1_in1k`, focal loss (γ=1.0, α=0.5),
imbalance=none, AdamW lr 2e-4 / wd 1e-4, batch 32, `medical_default` aug, full
fine-tune, cosine schedule, best epoch 6, **validation AUROC 0.9756**. Selected
**purely on validation AUROC**, before any test access. Config:
`configs/final_config.yaml`; weights: `final_model.pt`.

## Calibration decision

Assessed on validation. **Temperature scaling** (single parameter, LBFGS on NLL)
gave **T = 0.5646**. Validation ECE 0.0833 → **0.0308**, Brier 0.0592 → 0.0525,
AUROC unchanged (0.9756; temperature scaling is monotonic ⇒ discrimination
preserved). **Decision: use calibrated probabilities** for thresholding and test
reporting. Parameters + before/after metrics: `configs/calibration.json`.
Reliability diagrams: `results/figures/{valid,test}_calibration.png`.

## Threshold selection decision

On the validation set, using calibrated probabilities, the primary threshold was
the **highest-specificity threshold achieving sensitivity ≥ 0.95** (sensitivity-
prioritized, clinically motivated). Target was achievable.

- **Locked threshold = 0.7113** → validation sensitivity **0.952**, specificity **0.896**.
- Secondary reference (Youden's J): coincided at 0.7113 here.

Threshold **locked before** the test set was evaluated. Config: `configs/threshold.json`.

## Final locked threshold

**0.7113139** (on calibrated malignancy probability).

## Final test results with 95% CIs

Test split (n=1000), calibrated probabilities + locked threshold. CIs: stratified
bootstrap, 2000 resamples, seed=42.

| Metric | Point | 95% CI |
|--------|------:|:------:|
| AUROC | 0.9371 | [0.9202, 0.9528] |
| Sensitivity | 0.9042 | [0.8824, 0.9248] |
| Specificity | 0.7955 | [0.7435, 0.8439] |
| PPV | 0.9232 | [0.9054, 0.9401] |
| NPV | 0.7535 | [0.7123, 0.7979] |
| Accuracy | 0.8750 | [0.8540, 0.8950] |
| F1 | 0.9136 | [0.8991, 0.9278] |
| Brier | 0.0823 | — |
| ECE | 0.0314 | — |

Confusion matrix (Test): TN=214, FP=55, FN=70, TP=661.
Tables: `results/tables/test_metrics_with_ci.{md,csv}`; per-image predictions:
`results/{valid,test}_predictions.csv`.

## Failed / weaker runs

No run errored (all rc=0). Weakest configurations: `medical_strong` augmentation
(0.9609) and `flip_only` (0.9637) — both under the `medical_default` baseline,
confirming the augmentation policy choice. Earlier trackio smoke runs
(`smoke_trackio_test*`, `connectivity_check`, `dataset_pin_check`) are
infrastructure-validation runs, not experiments. Early Trackio Space creation
produced transient 401 `/volumes` warnings until a persistent `dataset_id`
(`Johnyquest7/Trakio_agentic_thyroid_dataset`) was pinned; resolved thereafter.

## Limitations

- Single-source dataset; cropped-ROI inputs; mild class imbalance.
- The ≥0.95-sensitivity operating point set on validation yielded **0.904**
  sensitivity on test — the operating point does not transfer perfectly; ~10% of
  malignant nodules are missed at the locked threshold. Local threshold
  re-calibration is advisable before any use.
- Leakage checks (exact-pixel hash + filename-ID overlap) are exhaustive for the
  available signal but cannot exclude same-patient/near-duplicate leakage if such
  structure exists upstream in TN5000.

## External validation — NOT yet performed

No external/independent dataset has been evaluated. `evaluate_external.py` is
provided to run the locked model (same preprocessing, calibration T, and locked
threshold) on a future external set (folder or CSV format). External, ideally
prospective and multi-site, validation is **required** before any clinical use.

## Test-set integrity statement

> The Test split was evaluated **exactly once**, and **only after** the model was
> selected (by validation AUROC), calibrated (temperature scaling on validation),
> and the decision threshold was locked (on validation). No hyperparameter,
> calibration, or threshold decision used the test set.