Image-to-Text
uvlm
medical-imaging
ct
report-generation
3d
vision-language

U-VLM: Hierarchical Vision Language Modeling for Report Generation

Paper GitHub HuggingFace

Released U-VLM checkpoints for 3D CT radiology report generation on CT-RATE and AbdomenAtlas 3.0. U-VLM pairs a residual U-Net encoder pretrained through segmentation and classification with a 0.1B language decoder trained from scratch. Encoder features are injected into multiple decoder layers.

Released Checkpoints

Dataset F1 BLEU-mean
CT-RATE 0.4506 0.3404
AbdomenAtlas 3.0 (structured) 0.5791 0.4337

Both entries use checkpoint_final.pth. Metrics are recomputed on the full CT-RATE validation split (3,039 cases) and the AbdomenAtlas test split (926 cases). CT-RATE F1 is RadBERT macro F1; AbdomenAtlas F1 uses DeepSeek label extraction. BLEU-mean is computed with pycocoevalcap.

Retraining Note

The released checkpoints are final-epoch models retrained after the paper. Their recomputed metrics differ from the paper table:

Dataset Paper F1 Released F1 Paper BLEU-mean Released BLEU-mean
CT-RATE 0.414 0.4506 0.349 0.3404
AbdomenAtlas 3.0 0.624 0.5791 0.437 0.4337

The CT-RATE checkpoint is a narrative-report model. The AbdomenAtlas checkpoint is a structured-report model (report_structured).

Parameters

Module Approx. parameters
Residual U-Net encoder 90.3M
Token embedding + tied LM head 77.7M
Transformer decoder 18.9M
Vision projection 4.3M
Norms 9.4K
Total ~191M

The 0.1B language decoder is about 96.6M parameters. The checkpoint is about 765 MB in FP32.

Layout

.
β”œβ”€β”€ config.json
β”œβ”€β”€ Dataset1131_CT-RATE/
β”‚   └── nnUNetTrainer_UVLM__CTRATE_report_192x256x256_clsseg_bal_aug_ds_fr_narr__3d_fullres/
β”‚       β”œβ”€β”€ dataset.json
β”‚       β”œβ”€β”€ plans.json
β”‚       β”œβ”€β”€ dataset_fingerprint.json
β”‚       β”œβ”€β”€ metrics.json
β”‚       └── fold_0/checkpoint_final.pth
β”œβ”€β”€ Dataset1141_AbdomenAtlas3.0Mini/
β”‚   └── nnUNetTrainer_UVLM__ABDOMEN_report_192x256x256_clsseg_seg_coarse_lesion_bal_aug_ds_fr_struct__3d_fullres/
β”‚       β”œβ”€β”€ dataset.json
β”‚       β”œβ”€β”€ plans.json
β”‚       β”œβ”€β”€ dataset_fingerprint.json
β”‚       β”œβ”€β”€ metrics.json
β”‚       └── fold_0/checkpoint_final.pth
β”œβ”€β”€ inference_results/
β”‚   β”œβ”€β”€ ct_rate_validation.csv
β”‚   β”œβ”€β”€ abdomenatlas3_test.csv
β”‚   └── metrics.json
└── tokenizer_qwen3_4b/

Only inference weights, JSON configs, metrics, and prediction CSVs are included. Training logs, optimizer states, images, and patient identifiers are not included.

Method

U-VLM progressively trains a shared residual U-Net encoder:

  1. Segmentation pretraining β€” dense per-voxel supervision.
  2. Classification pretraining β€” 18-class chest or 3-class abdominal labels.
  3. Report generation β€” image-report pairs.

For report generation, encoder features from six stages are routed to six decoder layers. The decoder has 512 hidden dimensions, 8 heads, and is trained from scratch. The released models use the Qwen3-4B tokenizer; the Qwen3-4B language-model weights are not used.

Requirements

Use the code repository's environment:

git clone https://github.com/yinghemedical/U-VLM.git
cd U-VLM
bash uvlm/scripts/setup_env.sh

Python 3.10, PyTorch, nnU-Net v2, blosc2, and transformers are required. See the code repository for the full installation instructions.

Inference

Download the weights:

hf download yinghemedical/U-VLM --repo-type model --local-dir /path/to/uvlm_weights

The input CSV must contain series_id and either a preprocessed Blosc2 path (blosc2_path) or a NIfTI path (image_path / nifti_path). NIfTI volumes are normalized and resized on the fly; volumes already at the checkpoint shape are used as-is. Run report generation with the code repository:

# CT-RATE, narrative report
uvlm_inference report \
    --csv-path /path/to/ct_rate_test.csv \
    --model-dir /path/to/uvlm_weights/Dataset1131_CT-RATE/nnUNetTrainer_UVLM__CTRATE_report_192x256x256_clsseg_bal_aug_ds_fr_narr__3d_fullres \
    --output-dir /path/to/output/ct_rate \
    --gpu-config "0:1"

# AbdomenAtlas 3.0, structured report
uvlm_inference report \
    --csv-path /path/to/abdomen_test.csv \
    --model-dir /path/to/uvlm_weights/Dataset1141_AbdomenAtlas3.0Mini/nnUNetTrainer_UVLM__ABDOMEN_report_192x256x256_clsseg_seg_coarse_lesion_bal_aug_ds_fr_struct__3d_fullres \
    --output-dir /path/to/output/abdomen_atlas \
    --gpu-config "0:1"

Output is output/.../results.csv with a generated_report column.

The tokenizer is included in this repository and in the code repository. Set UVLM_TOKENIZER_PATH to override the default:

export UVLM_TOKENIZER_PATH=/path/to/uvlm_weights/tokenizer_qwen3_4b

Inference Outputs

Full inference outputs from the released checkpoints are included as four-column CSVs:

Each file has case, resized_image_shape, ground_truth_report, and predicted_report. Reports were generated with max_new_tokens=512, matching the evaluation protocol above. Ground truth is the CT-RATE narrative report (Findings + Impressions) or the AbdomenAtlas report_structured field.

Evaluation Details

See the code repository's docs/RELEASED_CHECKPOINTS.md for the full metric breakdown and protocol.

Dataset F1 BLEU-1 BLEU-2 BLEU-3 BLEU-4 BLEU-mean
CT-RATE 0.4506 0.4818 0.3595 0.2845 0.2358 0.3404
AbdomenAtlas 3.0 (structured) 0.5791 0.5562 0.4629 0.3916 0.3240 0.4337

These are local evaluation results, not official challenge scores. Metrics are reported on the full CT-RATE validation split (3,039 cases) and the AbdomenAtlas test split (926 cases), respectively.

Data and License

CT-RATE and AbdomenAtlas 3.0 images and reports are not redistributed. Users must obtain the datasets under their upstream terms.

  • Code: Apache-2.0.
  • Model weights in this repository: CC BY-NC 4.0 (non-commercial use only).
  • Qwen3-4B tokenizer and upstream components retain their own licenses.

Citation

@article{shi2026u,
  title={U-VLM: Hierarchical Vision Language Modeling for Report Generation},
  author={Shi, Pengcheng and Zhang, Minghui and Song, Kehan and Liu, Jiaqi and Gu, Yun and Zhang, Xinglin},
  journal={arXiv preprint arXiv:2603.00479},
  year={2026}
}

Acknowledgements

U-VLM builds on nnU-Net and the Qwen3 tokenizer. We thank the CT-RATE and AbdomenAtlas teams for their datasets. U-VLM is developed and maintained by Medical Image Insights and Shanghai Jiao Tong University.

Downloads last month
182
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Datasets used to train yinghemedical/U-VLM

Paper for yinghemedical/U-VLM