File size: 7,055 Bytes
6064a2a c6be546 6064a2a c6be546 fcda449 c6be546 fcda449 c6be546 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 | ---
license: cc-by-nc-sa-4.0
library_name: transformers
pipeline_tag: image-segmentation
base_model: OpenGVLab/InternVL2_5-1B
tags:
- medical
- radiology
- ct
- 3d
- tumor-segmentation
- referring-segmentation
- report-generation
- visual-question-answering
- custom_code
---
# RepConTUR-1B
**RepConTUR** is a report-conditioned volumetric multimodal model for abdominal tumor segmentation. It reads a whole
abdominal CT volume and either a short referring phrase or the full narrative radiology report, and segments liver,
kidney and pancreas tumors natively in 3D. The same backbone also generates reports and answers questions about the
scan.
- **Code:** <https://github.com/Oncosoft-Research/RepConTUR>
- **Data:** <https://huggingface.co/datasets/Oncosoft-Research/RepConTUR>
- **Paper:** <https://arxiv.org/abs/2610.40094>
## Model
| Component | Detail |
|---|---|
| 3D CT encoder | CT-ViT from CT-CLIP, pretrained with a CLIP objective on CT-RATE chest and Merlin abdominal image–report pairs; frozen |
| Visual tokens | Attentional pooler with 256 learnable queries, projected to the LLM width |
| Language model | InternVL2.5-1B, initialized from UniBioMed, adapted with LoRA (merged into this checkpoint) |
| Segmentation module | VoxTell decoder, prompted by the hidden state of the `[SEG]` token through a two-layer projection |
| Input grid | CT-ViT: 140 × 240 × 240 at (3.0, 1.5, 1.5) mm, HU clipped to [−1000, 1000]; VoxTell: RAS, 1 mm, z-scored |
## Tasks
| Task | Input | Output |
|---|---|---|
| Referring segmentation (RES) | CT + phrase, e.g. "Please segment the tumor in hepatic segment 7." | `[SEG]` + 3D mask |
| Report-grounded segmentation (GSEG) | CT + narrative report + phrase | `[SEG]` + 3D mask |
| Radiology report generation (RRG) | CT | narrative report |
| Visual question answering (VQA) | CT + question | yes/no, short answer or multiple choice |
The model was trained jointly on all four tasks.
## Training data
AbdomenAtlas 3.0 Mini, restricted to the 3,798 CT volumes with a liver, kidney or pancreas tumor (3,425 train,
373 test). Tumor masks were decomposed into suborgan-level targets such as liver Couinaud segments using the
structured reports. VQA pairs were generated from the structured and narrative reports and checked by a second
language model. The reports in AbdomenAtlas 3.0 are synthetic, produced by the RadGPT pipeline, not written by
radiologists. The annotations are released at [Oncosoft-Research/RepConTUR](https://huggingface.co/datasets/Oncosoft-Research/RepConTUR); the CT volumes
and masks come from AbdomenAtlas 3.0 Mini.
## Results
In-domain test split; DSC is image-wise Dice, and R/P/F1 are instance-wise recall, precision and F1 at IoU ≥ 0.10.
Means are macro averages over the three organs, with 463 liver, 251 pancreas and 524 kidney GT-positive cases.
| Organ | Mode | DSC | R | P | F1 |
|---|---|---|---|---|---|
| Liver | RES | 0.522 | 0.616 | 0.369 | 0.461 |
| | GSEG | 0.542 | 0.611 | 0.424 | 0.500 |
| Pancreas | RES | 0.386 | 0.624 | 0.336 | 0.437 |
| | GSEG | 0.399 | 0.644 | 0.376 | 0.475 |
| Kidney | RES | 0.621 | 0.781 | 0.487 | 0.600 |
| | GSEG | 0.627 | 0.775 | 0.501 | 0.609 |
| **Mean** | RES | 0.510 | 0.674 | 0.397 | 0.499 |
| | GSEG | 0.523 | 0.677 | 0.433 | 0.528 |
On the language tasks over the same 373 test volumes, overall VQA exact-match accuracy is 0.732, and RRG
reaches BERTScore-F1 0.593 and ROUGE-L 0.377.
## Usage
The model ships its own modeling code, so it loads with `trust_remote_code=True`. The CT preprocessing and prompt
templates live in the GitHub repository, so set up the environment there first:
```bash
git clone https://github.com/Oncosoft-Research/RepConTUR.git && cd RepConTUR
bash setup_env.sh # uv environment under projects/repcontur
source projects/repcontur/.venv/bin/activate
```
Interactive demo:
```bash
python -m projects.repcontur.gradio.app Oncosoft-Research/RepConTUR
```
Python, run from the repository root:
```python
from projects.repcontur.gradio.app import build_runtime, load_case, strip_response
from projects.repcontur.gradio.task_prompts import Task, build_task_prompt
runtime = build_runtime('Oncosoft-Research/RepConTUR') # model, tokenizer and both CT preprocessors
case = load_case(runtime, 'ct.nii.gz') # any 3D abdominal CT in NIfTI
report_text = open('report.txt').read() # the narrative report for GSEG
prompt = runtime.image_token_string + '\n' + build_task_prompt(
Task.GSEG, 'tumor in the right kidney', report_text)
input_ids = runtime.tokenizer.encode(
f'<|user|>\n{prompt}<|end|>\n<|assistant|>\n', add_special_tokens=False)
output = runtime.model.predict_volume(
pixel_values=case.pixel_values.cuda(), input_ids=input_ids, g_pixel_values=case.grounding_volume)
text = strip_response(output['prediction'], runtime.model.template) # '[SEG]' for segmentation tasks
mask = output['prediction_mask'] # (D, H, W) uint8 on the 1 mm RAS grid
```
Use `Task.RES` with an empty report for phrase-only segmentation, `Task.VQA` with a question, or `Task.RRG` for report
generation. A single 24 GB GPU is enough for inference.
## Fine-tuning
The repository's `projects/repcontur/configs/repcontur_in25_1b_finetune.py` starts from this model. It builds the
InternVL2.5-1B backbone, loads all weights from this model's `pytorch_model.bin` (expected in `pretrained/RepConTUR-1B`),
and attaches a fresh LoRA on top. The VoxTell decoder is built from this model's `plans.json`. Download the model
first:
```bash
hf download Oncosoft-Research/RepConTUR --local-dir pretrained/RepConTUR-1B
bash tools/dist.sh train projects/repcontur/configs/repcontur_in25_1b_finetune.py 4
```
## Limitations
- Research use only. The model is not a medical device and has not been validated for clinical decisions.
- Trained on a single public dataset with synthetic reports. Performance on other scanners, protocols, populations or
real radiologist reports may differ.
- Only liver, kidney and pancreas tumors are supported.
- Generated reports and VQA answers can be wrong even when fluent; check them against the image.
## License
Released under **CC BY-NC-SA 4.0**, the license of the AbdomenAtlas 3.0 data the model was trained on: attribution,
non-commercial use only, and adaptations under the same license.
## Citation
```bibtex
@article{lee2026repcontur,
title = {RepConTUR: A Report-Conditioned Volumetric Multimodal Model for Abdominal Tumor Segmentation},
author = {Lee, Seunghwan and Park, Sangjoon and Lee, Jongmin and Kim, Yoon Jo and Cho, Wonyoung and Kim, Jin Sung and Chae, Han Joo},
journal = {arXiv preprint arXiv:2610.40094},
year = {2026}
}
```
## Acknowledgements
Built on [Sa2VA](https://github.com/bytedance/Sa2VA), [VoxTell](https://github.com/MIC-DKFZ/VoxTell),
[CT-CLIP / CT-RATE](https://github.com/ibrahimethemhamamci/CT-CLIP), [InternVL](https://github.com/OpenGVLab/InternVL),
UniBioMed and AbdomenAtlas 3.0 / RadGPT.
|