--- license: cc-by-nc-sa-4.0 library_name: transformers pipeline_tag: image-segmentation base_model: OpenGVLab/InternVL2_5-1B tags: - medical - radiology - ct - 3d - tumor-segmentation - referring-segmentation - report-generation - visual-question-answering - custom_code --- # RepConTUR-1B **RepConTUR** is a report-conditioned volumetric multimodal model for abdominal tumor segmentation. It reads a whole abdominal CT volume and either a short referring phrase or the full narrative radiology report, and segments liver, kidney and pancreas tumors natively in 3D. The same backbone also generates reports and answers questions about the scan. - **Code:** - **Data:** - **Paper:** ## Model | Component | Detail | |---|---| | 3D CT encoder | CT-ViT from CT-CLIP, pretrained with a CLIP objective on CT-RATE chest and Merlin abdominal image–report pairs; frozen | | Visual tokens | Attentional pooler with 256 learnable queries, projected to the LLM width | | Language model | InternVL2.5-1B, initialized from UniBioMed, adapted with LoRA (merged into this checkpoint) | | Segmentation module | VoxTell decoder, prompted by the hidden state of the `[SEG]` token through a two-layer projection | | Input grid | CT-ViT: 140 × 240 × 240 at (3.0, 1.5, 1.5) mm, HU clipped to [−1000, 1000]; VoxTell: RAS, 1 mm, z-scored | ## Tasks | Task | Input | Output | |---|---|---| | Referring segmentation (RES) | CT + phrase, e.g. "Please segment the tumor in hepatic segment 7." | `[SEG]` + 3D mask | | Report-grounded segmentation (GSEG) | CT + narrative report + phrase | `[SEG]` + 3D mask | | Radiology report generation (RRG) | CT | narrative report | | Visual question answering (VQA) | CT + question | yes/no, short answer or multiple choice | The model was trained jointly on all four tasks. ## Training data AbdomenAtlas 3.0 Mini, restricted to the 3,798 CT volumes with a liver, kidney or pancreas tumor (3,425 train, 373 test). Tumor masks were decomposed into suborgan-level targets such as liver Couinaud segments using the structured reports. VQA pairs were generated from the structured and narrative reports and checked by a second language model. The reports in AbdomenAtlas 3.0 are synthetic, produced by the RadGPT pipeline, not written by radiologists. The annotations are released at [Oncosoft-Research/RepConTUR](https://huggingface.co/datasets/Oncosoft-Research/RepConTUR); the CT volumes and masks come from AbdomenAtlas 3.0 Mini. ## Results In-domain test split; DSC is image-wise Dice, and R/P/F1 are instance-wise recall, precision and F1 at IoU ≥ 0.10. Means are macro averages over the three organs, with 463 liver, 251 pancreas and 524 kidney GT-positive cases. | Organ | Mode | DSC | R | P | F1 | |---|---|---|---|---|---| | Liver | RES | 0.522 | 0.616 | 0.369 | 0.461 | | | GSEG | 0.542 | 0.611 | 0.424 | 0.500 | | Pancreas | RES | 0.386 | 0.624 | 0.336 | 0.437 | | | GSEG | 0.399 | 0.644 | 0.376 | 0.475 | | Kidney | RES | 0.621 | 0.781 | 0.487 | 0.600 | | | GSEG | 0.627 | 0.775 | 0.501 | 0.609 | | **Mean** | RES | 0.510 | 0.674 | 0.397 | 0.499 | | | GSEG | 0.523 | 0.677 | 0.433 | 0.528 | On the language tasks over the same 373 test volumes, overall VQA exact-match accuracy is 0.732, and RRG reaches BERTScore-F1 0.593 and ROUGE-L 0.377. ## Usage The model ships its own modeling code, so it loads with `trust_remote_code=True`. The CT preprocessing and prompt templates live in the GitHub repository, so set up the environment there first: ```bash git clone https://github.com/Oncosoft-Research/RepConTUR.git && cd RepConTUR bash setup_env.sh # uv environment under projects/repcontur source projects/repcontur/.venv/bin/activate ``` Interactive demo: ```bash python -m projects.repcontur.gradio.app Oncosoft-Research/RepConTUR ``` Python, run from the repository root: ```python from projects.repcontur.gradio.app import build_runtime, load_case, strip_response from projects.repcontur.gradio.task_prompts import Task, build_task_prompt runtime = build_runtime('Oncosoft-Research/RepConTUR') # model, tokenizer and both CT preprocessors case = load_case(runtime, 'ct.nii.gz') # any 3D abdominal CT in NIfTI report_text = open('report.txt').read() # the narrative report for GSEG prompt = runtime.image_token_string + '\n' + build_task_prompt( Task.GSEG, 'tumor in the right kidney', report_text) input_ids = runtime.tokenizer.encode( f'<|user|>\n{prompt}<|end|>\n<|assistant|>\n', add_special_tokens=False) output = runtime.model.predict_volume( pixel_values=case.pixel_values.cuda(), input_ids=input_ids, g_pixel_values=case.grounding_volume) text = strip_response(output['prediction'], runtime.model.template) # '[SEG]' for segmentation tasks mask = output['prediction_mask'] # (D, H, W) uint8 on the 1 mm RAS grid ``` Use `Task.RES` with an empty report for phrase-only segmentation, `Task.VQA` with a question, or `Task.RRG` for report generation. A single 24 GB GPU is enough for inference. ## Fine-tuning The repository's `projects/repcontur/configs/repcontur_in25_1b_finetune.py` starts from this model. It builds the InternVL2.5-1B backbone, loads all weights from this model's `pytorch_model.bin` (expected in `pretrained/RepConTUR-1B`), and attaches a fresh LoRA on top. The VoxTell decoder is built from this model's `plans.json`. Download the model first: ```bash hf download Oncosoft-Research/RepConTUR --local-dir pretrained/RepConTUR-1B bash tools/dist.sh train projects/repcontur/configs/repcontur_in25_1b_finetune.py 4 ``` ## Limitations - Research use only. The model is not a medical device and has not been validated for clinical decisions. - Trained on a single public dataset with synthetic reports. Performance on other scanners, protocols, populations or real radiologist reports may differ. - Only liver, kidney and pancreas tumors are supported. - Generated reports and VQA answers can be wrong even when fluent; check them against the image. ## License Released under **CC BY-NC-SA 4.0**, the license of the AbdomenAtlas 3.0 data the model was trained on: attribution, non-commercial use only, and adaptations under the same license. ## Citation ```bibtex @article{lee2026repcontur, title = {RepConTUR: A Report-Conditioned Volumetric Multimodal Model for Abdominal Tumor Segmentation}, author = {Lee, Seunghwan and Park, Sangjoon and Lee, Jongmin and Kim, Yoon Jo and Cho, Wonyoung and Kim, Jin Sung and Chae, Han Joo}, journal = {arXiv preprint arXiv:2610.40094}, year = {2026} } ``` ## Acknowledgements Built on [Sa2VA](https://github.com/bytedance/Sa2VA), [VoxTell](https://github.com/MIC-DKFZ/VoxTell), [CT-CLIP / CT-RATE](https://github.com/ibrahimethemhamamci/CT-CLIP), [InternVL](https://github.com/OpenGVLab/InternVL), UniBioMed and AbdomenAtlas 3.0 / RadGPT.