Link arXiv paper 2610.40094

fcda449 verified 2 days ago

7.06 kB

---
license: cc-by-nc-sa-4.0
library_name: transformers
pipeline_tag: image-segmentation
base_model: OpenGVLab/InternVL2_5-1B
tags:
  - medical
  - radiology
  - ct
  - 3d
  - tumor-segmentation
  - referring-segmentation
  - report-generation
  - visual-question-answering
  - custom_code
---
# RepConTUR-1B

**RepConTUR** is a report-conditioned volumetric multimodal model for abdominal tumor segmentation. It reads a whole
abdominal CT volume and either a short referring phrase or the full narrative radiology report, and segments liver,
kidney and pancreas tumors natively in 3D. The same backbone also generates reports and answers questions about the
scan.

- **Code:** <https://github.com/Oncosoft-Research/RepConTUR>
- **Data:** <https://huggingface.co/datasets/Oncosoft-Research/RepConTUR>
- **Paper:** <https://arxiv.org/abs/2610.40094>

## Model

| Component | Detail |
|---|---|
| 3D CT encoder | CT-ViT from CT-CLIP, pretrained with a CLIP objective on CT-RATE chest and Merlin abdominal image–report pairs; frozen |
| Visual tokens | Attentional pooler with 256 learnable queries, projected to the LLM width |
| Language model | InternVL2.5-1B, initialized from UniBioMed, adapted with LoRA (merged into this checkpoint) |
| Segmentation module | VoxTell decoder, prompted by the hidden state of the `[SEG]` token through a two-layer projection |
| Input grid | CT-ViT: 140 × 240 × 240 at (3.0, 1.5, 1.5) mm, HU clipped to [−1000, 1000]; VoxTell: RAS, 1 mm, z-scored |

## Tasks

| Task | Input | Output |
|---|---|---|
| Referring segmentation (RES) | CT + phrase, e.g. "Please segment the tumor in hepatic segment 7." | `[SEG]` + 3D mask |
| Report-grounded segmentation (GSEG) | CT + narrative report + phrase | `[SEG]` + 3D mask |
| Radiology report generation (RRG) | CT | narrative report |
| Visual question answering (VQA) | CT + question | yes/no, short answer or multiple choice |

The model was trained jointly on all four tasks.

## Training data

AbdomenAtlas 3.0 Mini, restricted to the 3,798 CT volumes with a liver, kidney or pancreas tumor (3,425 train,
373 test). Tumor masks were decomposed into suborgan-level targets such as liver Couinaud segments using the
structured reports. VQA pairs were generated from the structured and narrative reports and checked by a second
language model. The reports in AbdomenAtlas 3.0 are synthetic, produced by the RadGPT pipeline, not written by
radiologists. The annotations are released at [Oncosoft-Research/RepConTUR](https://huggingface.co/datasets/Oncosoft-Research/RepConTUR); the CT volumes
and masks come from AbdomenAtlas 3.0 Mini.

## Results

In-domain test split; DSC is image-wise Dice, and R/P/F1 are instance-wise recall, precision and F1 at IoU ≥ 0.10.
Means are macro averages over the three organs, with 463 liver, 251 pancreas and 524 kidney GT-positive cases.

| Organ | Mode | DSC | R | P | F1 |
|---|---|---|---|---|---|
| Liver | RES | 0.522 | 0.616 | 0.369 | 0.461 |
| | GSEG | 0.542 | 0.611 | 0.424 | 0.500 |
| Pancreas | RES | 0.386 | 0.624 | 0.336 | 0.437 |
| | GSEG | 0.399 | 0.644 | 0.376 | 0.475 |
| Kidney | RES | 0.621 | 0.781 | 0.487 | 0.600 |
| | GSEG | 0.627 | 0.775 | 0.501 | 0.609 |
| **Mean** | RES | 0.510 | 0.674 | 0.397 | 0.499 |
| | GSEG | 0.523 | 0.677 | 0.433 | 0.528 |

On the language tasks over the same 373 test volumes, overall VQA exact-match accuracy is 0.732, and RRG
reaches BERTScore-F1 0.593 and ROUGE-L 0.377.

## Usage

The model ships its own modeling code, so it loads with `trust_remote_code=True`. The CT preprocessing and prompt
templates live in the GitHub repository, so set up the environment there first:

```bash
git clone https://github.com/Oncosoft-Research/RepConTUR.git && cd RepConTUR
bash setup_env.sh                      # uv environment under projects/repcontur
source projects/repcontur/.venv/bin/activate
```

Interactive demo:

```bash
python -m projects.repcontur.gradio.app Oncosoft-Research/RepConTUR
```

Python, run from the repository root:

```python
from projects.repcontur.gradio.app import build_runtime, load_case, strip_response
from projects.repcontur.gradio.task_prompts import Task, build_task_prompt
runtime = build_runtime('Oncosoft-Research/RepConTUR')         # model, tokenizer and both CT preprocessors
case = load_case(runtime, 'ct.nii.gz')                 # any 3D abdominal CT in NIfTI
report_text = open('report.txt').read()                # the narrative report for GSEG
prompt = runtime.image_token_string + '\n' + build_task_prompt(
    Task.GSEG, 'tumor in the right kidney', report_text)
input_ids = runtime.tokenizer.encode(
    f'<|user|>\n{prompt}<|end|>\n<|assistant|>\n', add_special_tokens=False)
output = runtime.model.predict_volume(
    pixel_values=case.pixel_values.cuda(), input_ids=input_ids, g_pixel_values=case.grounding_volume)
text = strip_response(output['prediction'], runtime.model.template)   # '[SEG]' for segmentation tasks
mask = output['prediction_mask']                                       # (D, H, W) uint8 on the 1 mm RAS grid
```

Use `Task.RES` with an empty report for phrase-only segmentation, `Task.VQA` with a question, or `Task.RRG` for report
generation. A single 24 GB GPU is enough for inference.

## Fine-tuning

The repository's `projects/repcontur/configs/repcontur_in25_1b_finetune.py` starts from this model. It builds the
InternVL2.5-1B backbone, loads all weights from this model's `pytorch_model.bin` (expected in `pretrained/RepConTUR-1B`),
and attaches a fresh LoRA on top. The VoxTell decoder is built from this model's `plans.json`. Download the model
first:

```bash
hf download Oncosoft-Research/RepConTUR --local-dir pretrained/RepConTUR-1B
bash tools/dist.sh train projects/repcontur/configs/repcontur_in25_1b_finetune.py 4
```

## Limitations

- Research use only. The model is not a medical device and has not been validated for clinical decisions.
- Trained on a single public dataset with synthetic reports. Performance on other scanners, protocols, populations or
  real radiologist reports may differ.
- Only liver, kidney and pancreas tumors are supported.
- Generated reports and VQA answers can be wrong even when fluent; check them against the image.

## License

Released under **CC BY-NC-SA 4.0**, the license of the AbdomenAtlas 3.0 data the model was trained on: attribution,
non-commercial use only, and adaptations under the same license.

## Citation

```bibtex
@article{lee2026repcontur,
  title  = {RepConTUR: A Report-Conditioned Volumetric Multimodal Model for Abdominal Tumor Segmentation},
  author = {Lee, Seunghwan and Park, Sangjoon and Lee, Jongmin and Kim, Yoon Jo and Cho, Wonyoung and Kim, Jin Sung and Chae, Han Joo},
  journal = {arXiv preprint arXiv:2610.40094},
  year   = {2026}
}
```

## Acknowledgements

Built on [Sa2VA](https://github.com/bytedance/Sa2VA), [VoxTell](https://github.com/MIC-DKFZ/VoxTell),
[CT-CLIP / CT-RATE](https://github.com/ibrahimethemhamamci/CT-CLIP), [InternVL](https://github.com/OpenGVLab/InternVL),
UniBioMed and AbdomenAtlas 3.0 / RadGPT.

	---
	license: cc-by-nc-sa-4.0
	library_name: transformers
	pipeline_tag: image-segmentation
	base_model: OpenGVLab/InternVL2_5-1B
	tags:
	- medical
	- radiology
	- ct
	- 3d
	- tumor-segmentation
	- referring-segmentation
	- report-generation
	- visual-question-answering
	- custom_code
	---

	# RepConTUR-1B

	RepConTUR is a report-conditioned volumetric multimodal model for abdominal tumor segmentation. It reads a whole
	abdominal CT volume and either a short referring phrase or the full narrative radiology report, and segments liver,
	kidney and pancreas tumors natively in 3D. The same backbone also generates reports and answers questions about the
	scan.

	- Code: <https://github.com/Oncosoft-Research/RepConTUR>
	- Data: <https://huggingface.co/datasets/Oncosoft-Research/RepConTUR>
	- Paper: <https://arxiv.org/abs/2610.40094>

	## Model

	\| Component \| Detail \|
	\|---\|---\|
	\| 3D CT encoder \| CT-ViT from CT-CLIP, pretrained with a CLIP objective on CT-RATE chest and Merlin abdominal image–report pairs; frozen \|
	\| Visual tokens \| Attentional pooler with 256 learnable queries, projected to the LLM width \|
	\| Language model \| InternVL2.5-1B, initialized from UniBioMed, adapted with LoRA (merged into this checkpoint) \|
	\| Segmentation module \| VoxTell decoder, prompted by the hidden state of the `[SEG]` token through a two-layer projection \|
	\| Input grid \| CT-ViT: 140 × 240 × 240 at (3.0, 1.5, 1.5) mm, HU clipped to [−1000, 1000]; VoxTell: RAS, 1 mm, z-scored \|

	## Tasks

	\| Task \| Input \| Output \|
	\|---\|---\|---\|
	\| Referring segmentation (RES) \| CT + phrase, e.g. "Please segment the tumor in hepatic segment 7." \| `[SEG]` + 3D mask \|
	\| Report-grounded segmentation (GSEG) \| CT + narrative report + phrase \| `[SEG]` + 3D mask \|
	\| Radiology report generation (RRG) \| CT \| narrative report \|
	\| Visual question answering (VQA) \| CT + question \| yes/no, short answer or multiple choice \|

	The model was trained jointly on all four tasks.

	## Training data

	AbdomenAtlas 3.0 Mini, restricted to the 3,798 CT volumes with a liver, kidney or pancreas tumor (3,425 train,
	373 test). Tumor masks were decomposed into suborgan-level targets such as liver Couinaud segments using the
	structured reports. VQA pairs were generated from the structured and narrative reports and checked by a second
	language model. The reports in AbdomenAtlas 3.0 are synthetic, produced by the RadGPT pipeline, not written by
	radiologists. The annotations are released at [Oncosoft-Research/RepConTUR](https://huggingface.co/datasets/Oncosoft-Research/RepConTUR); the CT volumes
	and masks come from AbdomenAtlas 3.0 Mini.

	## Results

	In-domain test split; DSC is image-wise Dice, and R/P/F1 are instance-wise recall, precision and F1 at IoU ≥ 0.10.
	Means are macro averages over the three organs, with 463 liver, 251 pancreas and 524 kidney GT-positive cases.

	\| Organ \| Mode \| DSC \| R \| P \| F1 \|
	\|---\|---\|---\|---\|---\|---\|
	\| Liver \| RES \| 0.522 \| 0.616 \| 0.369 \| 0.461 \|
	\| \| GSEG \| 0.542 \| 0.611 \| 0.424 \| 0.500 \|
	\| Pancreas \| RES \| 0.386 \| 0.624 \| 0.336 \| 0.437 \|
	\| \| GSEG \| 0.399 \| 0.644 \| 0.376 \| 0.475 \|
	\| Kidney \| RES \| 0.621 \| 0.781 \| 0.487 \| 0.600 \|
	\| \| GSEG \| 0.627 \| 0.775 \| 0.501 \| 0.609 \|
	\| Mean \| RES \| 0.510 \| 0.674 \| 0.397 \| 0.499 \|
	\| \| GSEG \| 0.523 \| 0.677 \| 0.433 \| 0.528 \|

	On the language tasks over the same 373 test volumes, overall VQA exact-match accuracy is 0.732, and RRG
	reaches BERTScore-F1 0.593 and ROUGE-L 0.377.

	## Usage

	The model ships its own modeling code, so it loads with `trust_remote_code=True`. The CT preprocessing and prompt
	templates live in the GitHub repository, so set up the environment there first:

	```bash
	git clone https://github.com/Oncosoft-Research/RepConTUR.git && cd RepConTUR
	bash setup_env.sh # uv environment under projects/repcontur
	source projects/repcontur/.venv/bin/activate
	```

	Interactive demo:

	```bash
	python -m projects.repcontur.gradio.app Oncosoft-Research/RepConTUR
	```

	Python, run from the repository root:

	```python
	from projects.repcontur.gradio.app import build_runtime, load_case, strip_response
	from projects.repcontur.gradio.task_prompts import Task, build_task_prompt

	runtime = build_runtime('Oncosoft-Research/RepConTUR') # model, tokenizer and both CT preprocessors
	case = load_case(runtime, 'ct.nii.gz') # any 3D abdominal CT in NIfTI
	report_text = open('report.txt').read() # the narrative report for GSEG

	prompt = runtime.image_token_string + '\n' + build_task_prompt(
	Task.GSEG, 'tumor in the right kidney', report_text)
	input_ids = runtime.tokenizer.encode(
	f'<\|user\|>\n{prompt}<\|end\|>\n<\|assistant\|>\n', add_special_tokens=False)

	output = runtime.model.predict_volume(
	pixel_values=case.pixel_values.cuda(), input_ids=input_ids, g_pixel_values=case.grounding_volume)
	text = strip_response(output['prediction'], runtime.model.template) # '[SEG]' for segmentation tasks
	mask = output['prediction_mask'] # (D, H, W) uint8 on the 1 mm RAS grid
	```

	Use `Task.RES` with an empty report for phrase-only segmentation, `Task.VQA` with a question, or `Task.RRG` for report
	generation. A single 24 GB GPU is enough for inference.

	## Fine-tuning

	The repository's `projects/repcontur/configs/repcontur_in25_1b_finetune.py` starts from this model. It builds the
	InternVL2.5-1B backbone, loads all weights from this model's `pytorch_model.bin` (expected in `pretrained/RepConTUR-1B`),
	and attaches a fresh LoRA on top. The VoxTell decoder is built from this model's `plans.json`. Download the model
	first:

	```bash
	hf download Oncosoft-Research/RepConTUR --local-dir pretrained/RepConTUR-1B
	bash tools/dist.sh train projects/repcontur/configs/repcontur_in25_1b_finetune.py 4
	```

	## Limitations

	- Research use only. The model is not a medical device and has not been validated for clinical decisions.
	- Trained on a single public dataset with synthetic reports. Performance on other scanners, protocols, populations or
	real radiologist reports may differ.
	- Only liver, kidney and pancreas tumors are supported.
	- Generated reports and VQA answers can be wrong even when fluent; check them against the image.

	## License

	Released under CC BY-NC-SA 4.0, the license of the AbdomenAtlas 3.0 data the model was trained on: attribution,
	non-commercial use only, and adaptations under the same license.

	## Citation

	```bibtex
	@article{lee2026repcontur,
	title = {RepConTUR: A Report-Conditioned Volumetric Multimodal Model for Abdominal Tumor Segmentation},
	author = {Lee, Seunghwan and Park, Sangjoon and Lee, Jongmin and Kim, Yoon Jo and Cho, Wonyoung and Kim, Jin Sung and Chae, Han Joo},
	journal = {arXiv preprint arXiv:2610.40094},
	year = {2026}
	}
	```

	## Acknowledgements

	Built on [Sa2VA](https://github.com/bytedance/Sa2VA), [VoxTell](https://github.com/MIC-DKFZ/VoxTell),
	[CT-CLIP / CT-RATE](https://github.com/ibrahimethemhamamci/CT-CLIP), [InternVL](https://github.com/OpenGVLab/InternVL),
	UniBioMed and AbdomenAtlas 3.0 / RadGPT.