data_exploration_report.md

# Data Exploration Report — TN5000 Thyroid Nodule Classification

- **Generated (UTC):** 2026-06-05T03:24:51.156145+00:00
- **Dataset:** `Johnyquest7/TN5000-thyroid-nodule-classification`
- **Source:** TN5000 (Yu et al., *Scientific Data*, 2025), cropped to nodule ROI, 224×224 PNG.
- **Task:** Binary classification — 0 = Benign, 1 = Malignant. Positive class = Malignant.

## 1. Number of images per split and class

| Split | Benign (0) | Malignant (1) | Total | Malignant % | Malignant:Benign ratio |
|-------|-----------:|--------------:|------:|------------:|------------------------:|
| Train | 1032 | 2468 | 3500 | 70.5% | 2.39 : 1 |
| Valid | 125 | 375 | 500 | 75.0% | 3.00 : 1 |
| Test | 269 | 731 | 1000 | 73.1% | 2.72 : 1 |
| **Total** | **1426** | **3574** | **5000** | **71.5%** | **2.51 : 1** |

![Class distribution](results/figures/class_distribution.png)

## 2. Class imbalance

All three splits are **malignant-majority** (~70–75% malignant), i.e. mild imbalance (malignant:benign roughly 2.4–3.0 : 1), consistent across splits.

- **Mitigation evaluated in training:** class-weighted `BCEWithLogitsLoss` (`pos_weight = N_benign/N_malignant`), focal loss, and a weighted sampler were all compared in the sweep; the final model uses focal loss (γ=1.0). Because imbalance is mild and calibration matters, heavy reweighting was avoided.

## 3. Image dimensions, channels, file format

- **File format:** PNG (lossless) for all 5000 images.
- **Dimensions observed:** [(224, 224)] (expected single value (224, 224)).
- **PIL modes observed:** ['RGB'] (RGB; grayscale replicated across 3 channels).

| Split | Unique dimensions | Modes |
|-------|-------------------|-------|
| Train | {(224, 224): 3500} | {'RGB': 3500} |
| Valid | {(224, 224): 500} | {'RGB': 500} |
| Test | {(224, 224): 1000} | {'RGB': 1000} |

## 4. Missing / corrupt image check

- ✅ **No corrupt or unreadable images.** All images opened and decoded via PIL `verify()` + reload.

## 5. Duplicate image check (exact pixel-content MD5)

- Duplicate groups **within a single split:** 0
- Duplicate groups **spanning multiple splits (potential LEAKAGE):** 0
- Duplicate groups with **conflicting labels:** 0

- ✅ No cross-split pixel duplicates and no label conflicts detected.

## 6. Data leakage analysis

TN5000 assigns each image a **globally unique numeric ID**, preserved as the PNG filename. Overlap of filename IDs across splits would indicate the same source image in two splits.

| Pair | Shared filename IDs |
|------|--------------------:|
| Train ∩ Valid | 0 |
| Train ∩ Test  | 0 |
| Valid ∩ Test  | 0 |

- ✅ **No filename-ID overlap across splits.** Combined with the exact-pixel duplicate check above, there is no detectable leakage between Train, Valid, and Test.

## 7. Pixel-intensity distribution

| Split | Mean | Std | Min | Max |
|-------|-----:|----:|----:|----:|
| Train | 81.5 | 19.6 | 27.1 | 163.5 |
| Valid | 81.3 | 19.4 | 35.0 | 171.5 |
| Test | 81.4 | 19.1 | 33.4 | 150.6 |

![Intensity distribution](results/figures/intensity_distribution.png)

Mean per-image grayscale intensity distributions are **closely matched across splits**, indicating consistent acquisition/preprocessing and no obvious distribution shift.

## 8. Representative image grids

**Train / Benign**

![Train Benign](results/figures/grid_Train_Benign.png)

**Train / Malignant**

![Train Malignant](results/figures/grid_Train_Malignant.png)

**Valid / Benign**

![Valid Benign](results/figures/grid_Valid_Benign.png)

**Valid / Malignant**

![Valid Malignant](results/figures/grid_Valid_Malignant.png)

**Test / Benign**

![Test Benign](results/figures/grid_Test_Benign.png)

**Test / Malignant**

![Test Malignant](results/figures/grid_Test_Malignant.png)

## 9. Train/Valid/Test separation statement

> The Train, Valid, and Test folders provided in the dataset repository were kept **strictly separate** throughout this experiment. The model was trained on **Train only**; the **Valid** split was used for model selection, calibration, and threshold selection; and the **Test** split was used **exactly once** for final locked evaluation after the model, calibration, and decision threshold were frozen. The exact-pixel duplicate check and filename-ID overlap check above confirm there is no detectable leakage between the three splits.