ml_training/cards/modality-classifier.md

Model card — modality classifier

Model. EfficientNet-B0 (timm, ImageNet init, standard 3-channel stem; grayscale inputs replicated 1→3ch in the transform), 3-class head: ct / mri / xray. Weights: backend/weights/modality_efficientnet_b0.pt + modality_config.json (classes, input size, normalization, fitted temperature). Served by backend/app/ml/imaging/real.py under MODEL_BACKEND=real.

Intended use

Routing and cross-checking only: classify the modality of a claimant-uploaded medical image so the diagnostic-report draft addresses the right study type, and flag disagreement with the DICOM Modality tag as a fraud signal. The confidence that gates mandatory human review is the temperature-scaled probability.

Not a diagnostic device. No clinical claim is made or implied; every output is reviewed by an imaging specialist before anything downstream happens.

Training data

15,000 images from ROCOv2 (eltorio/ROCOv2-radiology), 5,000 per class, labels derived from UMLS CUI tags; only rows matching exactly one modality CUI are kept (ml_training/datasets/build_datasets.py). Single-source by design: drawing all three classes from one corpus avoids the per-class source confound (resolution, scanner, compression signatures) that turns stitched-together datasets into accidental source detectors. Train recipe adds random JPEG-quality re-encoding (q 60–95) and occasional Gaussian blur on top of geometric augs to kill the remaining compression/sharpness shortcuts. Split 80/10/10, seed 42.

Metrics (held-out test split)

Metric	Value
Accuracy	0.942 (n=1,543)
Macro-F1	0.943
Per-class precision/recall	ct 0.92/0.93 · mri 0.91/0.92 · xray 1.00/0.98
ECE before → after temperature scaling	0.030 → 0.016
Fitted temperature	1.90

Most confusion is ct↔mri (76 of the 90 errors); xray is near-perfect. Trained 2026-06-10, 12 epochs, seed 42, best epoch selected by val macro-F1 (0.938 at epoch 8).

Reproduce: uv run python -m ml_training.evaluate --weights-dir weights/ --data-dir ml_training/data --report (full report at ml_training/data/eval_report.json, not committed).

Caveats and limitations

• Domain shift is unmeasured. ROCOv2 is publication-figure radiology; real claimant uploads (phone photos of films, portal exports, cropped screenshots) look different. The confound-killer augs narrow this gap but do not close it.
• Three classes only; ultrasound, PET, mammography and anything else will be forced into the nearest of ct/mri/xray — the calibrated confidence (and the mandatory-review gate it feeds) is the only guardrail.
• Calibration was fitted on the validation split of the same corpus; the temperature is not guaranteed to transfer under domain shift.
• CUI-derived labels are weak labels; ROCOv2 label noise propagates.