Data Exploration Report — TN5000 Thyroid Nodule Classification
- Generated (UTC): 2026-06-05T03:24:51.156145+00:00
- Dataset:
Johnyquest7/TN5000-thyroid-nodule-classification - Source: TN5000 (Yu et al., Scientific Data, 2025), cropped to nodule ROI, 224×224 PNG.
- Task: Binary classification — 0 = Benign, 1 = Malignant. Positive class = Malignant.
1. Number of images per split and class
| Split | Benign (0) | Malignant (1) | Total | Malignant % | Malignant:Benign ratio |
|---|---|---|---|---|---|
| Train | 1032 | 2468 | 3500 | 70.5% | 2.39 : 1 |
| Valid | 125 | 375 | 500 | 75.0% | 3.00 : 1 |
| Test | 269 | 731 | 1000 | 73.1% | 2.72 : 1 |
| Total | 1426 | 3574 | 5000 | 71.5% | 2.51 : 1 |
2. Class imbalance
All three splits are malignant-majority (~70–75% malignant), i.e. mild imbalance (malignant:benign roughly 2.4–3.0 : 1), consistent across splits.
- Mitigation evaluated in training: class-weighted
BCEWithLogitsLoss(pos_weight = N_benign/N_malignant), focal loss, and a weighted sampler were all compared in the sweep; the final model uses focal loss (γ=1.0). Because imbalance is mild and calibration matters, heavy reweighting was avoided.
3. Image dimensions, channels, file format
- File format: PNG (lossless) for all 5000 images.
- Dimensions observed: [(224, 224)] (expected single value (224, 224)).
- PIL modes observed: ['RGB'] (RGB; grayscale replicated across 3 channels).
| Split | Unique dimensions | Modes |
|---|---|---|
| Train | {(224, 224): 3500} | {'RGB': 3500} |
| Valid | {(224, 224): 500} | {'RGB': 500} |
| Test | {(224, 224): 1000} | {'RGB': 1000} |
4. Missing / corrupt image check
- ✅ No corrupt or unreadable images. All images opened and decoded via PIL
verify()+ reload.
5. Duplicate image check (exact pixel-content MD5)
Duplicate groups within a single split: 0
Duplicate groups spanning multiple splits (potential LEAKAGE): 0
Duplicate groups with conflicting labels: 0
✅ No cross-split pixel duplicates and no label conflicts detected.
6. Data leakage analysis
TN5000 assigns each image a globally unique numeric ID, preserved as the PNG filename. Overlap of filename IDs across splits would indicate the same source image in two splits.
| Pair | Shared filename IDs |
|---|---|
| Train ∩ Valid | 0 |
| Train ∩ Test | 0 |
| Valid ∩ Test | 0 |
- ✅ No filename-ID overlap across splits. Combined with the exact-pixel duplicate check above, there is no detectable leakage between Train, Valid, and Test.
7. Pixel-intensity distribution
| Split | Mean | Std | Min | Max |
|---|---|---|---|---|
| Train | 81.5 | 19.6 | 27.1 | 163.5 |
| Valid | 81.3 | 19.4 | 35.0 | 171.5 |
| Test | 81.4 | 19.1 | 33.4 | 150.6 |
Mean per-image grayscale intensity distributions are closely matched across splits, indicating consistent acquisition/preprocessing and no obvious distribution shift.
8. Representative image grids
Train / Benign
Train / Malignant
Valid / Benign
Valid / Malignant
Test / Benign
Test / Malignant
9. Train/Valid/Test separation statement
The Train, Valid, and Test folders provided in the dataset repository were kept strictly separate throughout this experiment. The model was trained on Train only; the Valid split was used for model selection, calibration, and threshold selection; and the Test split was used exactly once for final locked evaluation after the model, calibration, and decision threshold were frozen. The exact-pixel duplicate check and filename-ID overlap check above confirm there is no detectable leakage between the three splits.