Instructions to use zllhhhh/organic3d-vlm with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use zllhhhh/organic3d-vlm with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
Configuration Parsing Warning:In adapter_config.json: "peft.task_type" must be a string
ORGANIC-3D VLM (vlm4_organic3d)
A 3D CT vision–language model that answers yes/no finding questions and writes findings reports, and whose answers depend on the image: it withdraws 96–99% of correctly stated findings when their organ is erased. The frozen ORGANIC-3D encoder (https://huggingface.co/zllhhhh/organic3d-encoder) feeds its 144 anatomy tokens, in a fixed organ order, through an MLP into Qwen3.5-4B with LoRA. Paper: ORGANIC-3D: Grounding 3D CT Vision Language Models in Organ-Level Image–Text Coupling.
· Code: https://github.com/lzl199704/organic3d
· Benchmark:https://huggingface.co/datasets/zllhhhh/organic3d-benchmark
Research use only. Not for clinical use. Generated reports can contain findings that are not in the image and miss findings that are.
Usage
pip install -e <organic3d repository>
python organic3d.py vlm --ct scan.nii.gz --region abdomen --totalseg --report \
--question "Is there a kidney mass in this CT scan? Answer Yes or No."
The script downloads this repository, the encoder and Qwen/Qwen3.5-4B (pinned revision), and prints P(Yes) for every
finding of the region, any --question, and with --report an answer-first report (the findings it answers Yes to,
then the report, started with "Trachea" on chest and "Lower thorax:" on abdominal CT).
Files
adapter_model.safetensors / adapter_config.json (LoRA, r 32, α 64; 244 MB), projector.safetensors (MLP,
34 MB), config.json, organic3d.py. The base model is not included.
Training
Public data only: CT-RATE and Merlin training splits. Language-modelling loss on yes/no answers and answer-first
reports, the dependence margin (the correct answer must be more likely on the patient's own scan than on a matched
partner's scan or on the scan with the finding's organ erased, and unchanged when a control organ is erased), the
answer-attention loss and a report margin. 10,000 steps on 4 × B200, encoder frozen. Config:
configs/vlm/vlm4_organic3d.yaml in the code repository.
Results (paper Tables 2 and 3, %)
| CT-RATE | Merlin | ReXGroundingCT | |
|---|---|---|---|
| Answer AUROC | 86.3 | 87.6 | 79.5 |
| Balanced accuracy / F1 | 78.9 / 58.7 | 80.0 / 40.6 | 72.1 / 49.0 |
| Matched-pair AUROC (MP) | 82.8 | 86.0 | 74.1 |
| Organ-erasure hit rate (OEH) | 96.8 | 96.2 | 98.9 |
| Free-text report F1 / CRG | 56.8 / 52.9 | 37.7 / 49.4 | – |
| Finding-based report F1 / IA-F1 | 59.6 / 37.6 | 48.2 / 37.4 | – |
Licence terms of Qwen3.5 (Apache-2.0), COLIPRI, CT-RATE and Merlin apply.
Citation
@article{liu2026organic3d,
title = {ORGANIC3D: Grounding 3D CT Vision Language Models in Organ Level Image Text Coupling},
author = {Liu, Zelong and Ahmed, Hamza and Lee, Wynn Wingyi and Torjman, Mickael and Dimitrakakis, Alexander and
Lu, Jiahua and Deyer, Timothy and Taouli, Bachir and Fayad, Zahi A. and Mei, Xueyan},
journal = {arXiv preprint arXiv:2611.23529},
year = {2026}
}
- Downloads last month
- 4