odurugol/vlm3d-beyond-normals

VLM3D winning models of team Beyond Normals

Team members: Ömer Faruk Durugöl¹, Franziska Weber¹, Lukas Buess¹, Luca Hagen¹, Xiaoliang Wang¹, Harsh Khandelwal¹, James Myles³, Johanna Paula Müller^1,2, Hadrien Reynaud¹, Bernhard Kainz^1,3, Andreas Maier<sup>1

1</sup>Friedrich-Alexander-Universität Erlangen-Nürnberg   ²King's College London   ³Imperial College London

Weights and code for our entries in four tasks of the VLM3D 2026 challenge: abnormality classification and report generation for chest CT and brain MRI.

📄 Paper: the challenge paper is in preparation. We will link it here.

💻 Code: everything needed to run and train the models is in code/.

Task	Rank	Leaderboard result
CT abnormality classification	1	AUROC 0.8764, F1 0.6487, CRG 0.5715
CT report generation	1	CRG 0.5709, F1 0.6478, BLEU 0.2627
MR abnormality classification	2	AUROC 0.8901, F1 0.3934, precision 0.5163, recall 0.5917
MR report generation	1	CRG 0.5593, F1 0.3864, BLEU 0.2715

Files

config.json                         which folders each task uses
ct_classification/
  z1p5_mhead.safetensors            model 1: 1.5 mm slices
  z2_avg.safetensors                model 2: 2.0 mm slices, average of the epoch 0 and epoch 1 weights
  z1p5_gated_maxtoken.safetensors   model 3: 1.5 mm slices, gated max-token head
  streams.json                      architecture and preprocessing of each model
  fusion_policy.json                how the three models are combined
ct_report_generation/
  report_writer.json                label combination -> report text
mr_classification/
  member_1..8.safetensors           eight 3D ResNets
  config.json                       labels, architecture and input geometry of each model
mr_report_generation/
  model.safetensors                 one 3D ResNet-18
  config.json                       labels, architecture, input geometry
  report_config.json                template, bullet wording and thresholds
code/
  ct/                               CT inference and training
  mr/                               MR inference and training
  requirements.txt

Quick start

hf download odurugol/vlm3d-beyond-normals --local-dir vlm3d
cd vlm3d/code
pip install -r requirements.txt

CT classification and reports. The input is a folder of .nii.gz volumes in Hounsfield units:

cd ct
python predict_classification.py --weights ../../ct_classification --input scans/ --output results.json
python predict_report.py --weights ../../ct_classification \
    --writer ../../ct_report_generation/report_writer.json --input scans/ --output reports.json

MR classification and reports. The input is a folder of MR-RATE study zips, or of extracted <study>/atlas/*.nii.gz folders:

cd mr
python predict_classification.py --weights ../../mr_classification --input studies/ --output results.json
python predict_report.py --weights ../../mr_report_generation --input studies/ --output reports.json

The output format is the written file: results.json for classification, reports.json for reports. The CT models need the FlexiCT-2D model code, which transformers downloads on first use (trust_remote_code).

Data

• CT: CT-RATE
• MR: MR-RATE

CT abnormality classification

We fine-tuned FlexiCT-2D (revision fcefac1e) on CT-RATE with LoRA (r=8). FlexiCT-2D is a vision transformer that encodes one slice at a time. Each weights file holds the whole fine-tuned model.

Preprocessing

1. Reorient to RAS.
2. Resample to 0.75 mm in-plane and 1.5 mm (models 1 and 3) or 2.0 mm (model 2) along z.
3. Crop to the body: voxels above −500 HU, plus a 4-voxel margin.
4. Clip to [−1000, 1000] HU.
5. Keep 240 slices at 1.5 mm or 180 at 2.0 mm, centred on the slices that contain tissue. Shorter scans are padded with air.
6. Normalise each slice to zero mean and unit variance, then pad or crop to 480×480 px.

Model. We raised the patch size from 8 to 16 px and added a small convolution branch next to the patch projection. Each of the 18 abnormalities has a learned query that attends over the tokens of all slices. One self-attention layer then lets the 18 pooled vectors exchange information, and each abnormality has its own classifier. Model 3 also adds the single highest-scoring token per abnormality, scaled by a learned tanh gate, so that a finding seen on only a few slices is not averaged away.

Combining the three models

1. Replace each model's score by its rank among development-set scores for that abnormality, and convert the rank to a normal score.
2. Average the three scores with weights 0.375, 0.375 and 0.25.
3. Apply Platt scaling per abnormality.
4. Rescale each probability so that the class threshold t sits at 0.5: q = p(1−t) / (p(1−t) + (1−p)t). This keeps the order of cases, so AUROC does not change.

These steps were fitted on 2,398 scans from the public CT-RATE validation split, none of which were used for training.

CT report generation

The challenge scores a CT report by running a RadBERT label extractor on it and comparing the 18 labels it finds with the truth. Each of our reports states exactly the labels our classifier predicted, as checked with this extractor.

The classifier's 18 yes/no decisions form a key, and report_writer.json maps each key to a finished report. For every label combination in the CT-RATE training set, the table holds the first of these that the extractor reads back exactly:

1. one sentence per positive finding, followed by 12 normal-anatomy sentences chosen from frequent training sentences for BLEU;
2. the most common training report with that combination;
3. the finding sentences followed by that training report.

The table also covers the 683 combinations our classifier predicted on development scans. We reworded entries the extractor misread, but 23 of these 683 still do not read back exactly. These entries are also cut at a sentence end near 240 words, if the cut does not change the extracted labels, because long reports lowered BLEU. A key that is not in the table gets the report of the nearest key, with its own finding sentences put in front.

Many entries are verbatim CT-RATE training reports, so they can describe sizes, locations and findings of other patients.

MR abnormality classification

Eight 3D ResNets trained from scratch on MR-RATE: ResNet-18 at width 48, ResNet-18 at width 64 and ResNet-26 at width 64, all with GroupNorm, SiLU and global average pooling. They predict 74 diagnoses, of which the challenge scores 32. The training labels come from running the organisers' Gemma extraction prompt on all MR-RATE reports.

Input. MR-RATE studies come registered to the MNI152 atlas at 1 mm. For each MR type (T1w, T2w, FLAIR, SWI), every series is scaled to [0, 1] between its 1st and 99.5th percentile of non-zero voxels, and up to four series are averaged into one channel. A missing type gives an all-zero channel. Four presence channels follow, set to 1 if the study has that type. Without them the network cannot tell a missing series from an empty image. The head is cropped and resampled to 112×128×104 voxels at 1.5 mm; one model uses 96×112×96 voxels at 2 mm.

Inference. Each model averages its logits for the study and its left-right mirror image. The ensemble output is the mean of the eight probabilities. The threshold for each diagnosis is set so that a fixed share of studies is positive. These shares (positive_rates in mr_classification/config.json) were fitted on the MR-RATE validation set.

MR report generation

One ResNet-18 trained with the same recipe and input as the classification models, in a separate run.

Each report has two parts. "FINDINGS:" is followed by 8 fixed lines that describe a normal brain, picked from frequent training-report lines for BLEU. "IMPRESSION:" has one fixed bullet for each diagnosis above its threshold, up to 10. A study with no such diagnosis gets "Cranial MRI examination within normal limits." The thresholds (thresholds in mr_report_generation/report_config.json) were fitted on the MR-RATE validation set.

The Gemma extractor that reads the reports misses more diagnoses in longer lists, so we keep the list short:

• Some diagnoses imply others. A report that says glioma is also read as brain tumor, so only glioma is written. There are 11 such links, each checked with the extractor.
• Studies with 7 or more bullets keep only 5 of the 8 FINDINGS lines. The 3 dropped lines contradict common diagnoses, for example "the ventricles and sulci are normal" and ventriculomegaly.

Training

Script	What it does
ct/train.py	fine-tunes one FlexiCT-2D model with LoRA (--pool mhead, or hybrid for the gated max-token head; --z-spacing, --max-slices, --head-dim)
ct/average_checkpoints.py	averages the weights of two epochs (model 2)
ct/fit_fusion_policy.py	fits fusion_policy.json from development-set scores
ct/report_compile/build_report_writer.py	builds report_writer.json from training reports with the RadBERT extractor
mr/build_cache.py	preprocesses one MR-RATE batch into a uint8 shard
mr/train.py	trains one 3D ResNet (one GPU or torchrun)
mr/eval_run.py	validation and test probabilities of a run, in study order
mr/template_search.py	picks the FINDINGS lines with the highest BLEU

The training scripts are cleaned-up versions of our research code. We ran them on small subsets only, so a full run follows the same recipe but will not reproduce the released weights bit for bit. mr/build_cache.py has not been run in this form.

Limitations

These are challenge models, not clinical tools. Each was trained and tested on a single public dataset, CT-RATE or MR-RATE, with labels extracted automatically from radiology reports. They inherit the labellers' errors and have not been tested on data from other hospitals or scanners.

License

CC BY-NC-SA 4.0, because the CT report table contains CT-RATE report text. The terms of CT-RATE, MR-RATE and FlexiCT-2D also apply.

Citation

The challenge paper is in preparation. Until then, please cite this repository:

Team Beyond Normals. VLM3D winning models of team Beyond Normals. Hugging Face, 2026. https://huggingface.co/odurugol/vlm3d-beyond-normals

References

• CT-RATE: I. E. Hamamci, S. Er, C. Wang, et al. Generalist foundation models from a multimodal dataset for 3D computed tomography. Nature Biomedical Engineering, 2026. Dataset: https://huggingface.co/datasets/ibrahimhamamci/CT-RATE
• MR-RATE: Forithmus, University of Zurich. MR-RATE dataset, 2026 (paper forthcoming). https://huggingface.co/datasets/Forithmus/MR-RATE
• RadBERT: A. Yan, J. McAuley, X. Lu, J. Du, E. Y. Chang, A. Gentili, C.-N. Hsu. RadBERT: Adapting transformer-based language models to radiology. Radiology: Artificial Intelligence 4(4):e210258, 2022. https://doi.org/10.1148/ryai.210258

Acknowledgements

The authors gratefully acknowledge the scientific support and HPC resources provided by the Erlangen National High Performance Computing Center (NHR@FAU) of the Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU).