ORGANIC-3D encoder (enc4_organic3d)

A 3D CT image–text encoder that scores any sentence against a CT volume and follows the image evidence for each finding: its score drops when the finding's organ is erased, stays when another organ is erased, and prefers the sentence that names the correct side. Paper: ORGANIC-3D: Grounding 3D CT Vision Language Models in Organ-Level Image–Text Coupling.

· Code: https://github.com/lzl199704/organic3d

· Benchmark: https://huggingface.co/datasets/zllhhhh/organic3d-benchmark

· VLM: https://huggingface.co/zllhhhh/organic3d-vlm

Research use only. Not for clinical use.

Usage

pip install -e <organic3d repository>        # the `organic` package and its requirements
python organic3d.py encoder --ct scan.nii.gz --region chest --totalseg \
    --sentence "There is a nodule in the right upper lobe."

The script downloads this repository, resamples the CT to 1 mm RAS, centres a 384 mm crop with TotalSegmentator (or an estimate without it) and prints the score of every finding of the region (Δf = S(V, "There is f.") − S(V, "No f.")) and of each --sentence. From Python: from organic3d import Encoder.

Model

Image tower COLIPRI 3D ViT (Primus-M), 192³ input at 2 mm, fine-tuned
Anatomy tokens 144 learned queries (16 organ groups × 8, left and right lung and kidney separate, + 16 global), 2 cross-attention layers
Score each sentence is routed to the tokens of the organs it names and pools them with text-conditioned weights; cosine to the CXR-BERT sentence embedding / temperature
Files model.safetensors (272 M parameters, fp32), config.json, organic3d.py

Training

Public data only: the CT-RATE and Merlin training splits (3 : 2). Contrastive loss, classification loss and the four ORGANIC losses (organ-erasure with a control organ, mirror, matched-pair, anatomy-attention), 12,000 steps on 4 × H100. Config: configs/encoder/enc4_organic3d.yaml in the code repository.

Results (paper Table 1, %)

CT-RATE RAD-ChestCT Merlin
MLC-AUROC 87.7 79.7 87.7
Matched-pair AUROC (MP) 82.5 78.2 86.5
Organ-erasure ΔAUROC (OE) 28.5 18.4 21.3
Laterality accuracy (Side) 76.9 – 63.8

Lung lobe erasure hit rate 74.7 (CT-RATE); lesion AUROC 79.5 and lesion-erasure hit rate 74.9 (ReXGroundingCT); zero-shot tumour detection 75.0 and lesion-erasure hit rate 70.7 (AbdomenAtlas 3.0).

Limitations

Trained on chest CT from one centre (CT-RATE) and abdominal CT from one centre (Merlin) with report-derived labels; performance elsewhere is unknown. Laterality is far from perfect (Side 76.9 / 63.8). Licence terms of COLIPRI, CT-RATE and Merlin apply.

Citation

@article{liu2026organic3d,
  title   = {ORGANIC3D: Grounding 3D CT Vision Language Models in Organ Level Image Text Coupling},
  author  = {Liu, Zelong and Ahmed, Hamza and Lee, Wynn Wingyi and Torjman, Mickael and Dimitrakakis, Alexander and
             Lu, Jiahua and Deyer, Timothy and Taouli, Bachir and Fayad, Zahi A. and Mei, Xueyan},
  journal = {arXiv preprint arXiv:2611.23529},
  year    = {2026}
}
Downloads last month
5
Safetensors
Model size
0.3B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train zllhhhh/organic3d-encoder

Paper for zllhhhh/organic3d-encoder