ARotting/uncertainty-lens

Uncertainty Lens

Uncertainty Lens trains five compact CNNs from independent initializations and turns their disagreement into a probabilistic diagnostic.

The project measures:

• single-model versus ensemble accuracy;
• negative log-likelihood, Brier score, and expected calibration error;
• validation-only temperature scaling;
• predictive entropy and mutual information on held-out digits versus uniform-noise and pixel-scrambled OOD inputs.

The OOD sets are synthetic stress tests, not evidence of real-world deployment safety.

Reproduce

uv run python projects/tiny-vision-foundry/prepare_data.py
uv run python projects/uncertainty-lens/train.py

Verified results

Metric	Single member	Five-member ensemble
Test accuracy	98.15%	99.26%
Negative log-likelihood	0.0551	0.0570
Brier score	0.02784	0.02498
Expected calibration error	0.02185	0.02547

Validation selected temperature 1.0, meaning temperature scaling provided no improvement and left ensemble metrics unchanged.

For 270 uniform-noise and 270 pixel-scrambled inputs, predictive entropy achieved 0.9771 ROC-AUC against clean held-out digits. Mean entropy rose from 0.105 on clean data to 1.113 on OOD data; mean mutual information rose from 0.018 to 0.462.