Hugging Face's logo

ai4colonoscopy
/
ColonR1

Image-Text-to-Text
Transformers
Safetensors
qwen2_5_vl
image-to-text
conversational
text-generation-inference
Model card Files
xet
Community
1
ColonR1
File size: 3,808 Bytes 
4387770
 
96bc3b8
 
4387770
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
96bc3b8
 
4387770
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
96bc3b8
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
 
---
license: apache-2.0
library_name: transformers
pipeline_tag: image-text-to-text
---

# The first R1-styled model (ColonR1) tailored for reasoning in colonoscopy tasks

📖 [Colon-X: Advancing Intelligent Colonoscopy from Multimodal Understanding to Clinical Reasoning](https://arxiv.org/abs/2512.03667) 

🏠 More details refer to our project page: [https://github.com/ai4colonoscopy/Colon-X](https://github.com/ai4colonoscopy/Colon-X) 


<p align="center">
    <img src="./assets/ColonR1.jpg"/> <br />
    <em> 
    Figure 1: Details of our colonoscopy-specific reasoning model, ColonR1.
    </em>
</p>


# Quick start

Below is a code snippet to help you quickly try out our ColonR1 model using Hugging Face Transformers. For convenience, we manually combined some configuration and code files. Please note that this is a quick code, we recommend you using a source code to explore more.

- Before running the snippet, you need to install the following minimum dependencies.

	```shell
	conda create -n quickstart python=3.10
	conda activate quickstart
	pip install torch transformers accelerate pillow
	```

- Then you can use `python ColonR1/quickstart.py` to run it, as shown in the following code.

	```python
	import torch
	from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
	from PIL import Image
	import warnings
	import os

	warnings.filterwarnings('ignore')
	device = "cuda" if torch.cuda.is_available() else "cpu"

	MODEL_PATH = "ai4colonoscopy/ColonR1"
	IMAGE_PATH = "assets/example.jpg"
	Question = "Does the image contain a polyp? Answer me with Yes or No."

	print(f"[Info] Loading model from {MODEL_PATH}...")
	model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
		MODEL_PATH,
		torch_dtype=torch.bfloat16,
		attn_implementation="flash_attention_2",
		device_map="auto"
	)
	model.eval()

	processor = AutoProcessor.from_pretrained(MODEL_PATH)

	if not os.path.exists(IMAGE_PATH):
		raise FileNotFoundError(f"Image not found at {IMAGE_PATH}. Please provide a valid image path.")

	image = Image.open(IMAGE_PATH).convert("RGB")

	TASK_SUFFIX = (
		"Your task: 1. First, Think through the question step by step, enclose your reasoning process "
		"in <think>...</think> tags. 2. Then provide the correct answer inside <answer>...</answer> tags. "
		"3. No extra information or text outside of these tags."
	)

	final_question = f"{Question}
{TASK_SUFFIX}"

	messages = [
		{
			"role": "user",
			"content": [
				{"type": "image", "image": IMAGE_PATH},
				{"type": "text", "text": final_question},
			],
		}
	]

	print("[Info] Processing inputs...")
	text_prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

	inputs = processor(
		text=[text_prompt],
		images=[image],
		padding=True,
		return_tensors="pt",
	).to(device)


	print("[Info] Generating response...")
	with torch.no_grad():
		generated_ids = model.generate(
			**inputs,
			max_new_tokens=1024,
			do_sample=False
		)

	generated_ids_trimmed = generated_ids[:, inputs.input_ids.shape[1]:]
	output_text = processor.batch_decode(generated_ids_trimmed, skip_special_tokens=True)[0]

	print(output_text)
	```

# Reference

Feel free to cite if you find the Colon-X Project useful for your work:

```
@article{ji2025colonx,
    title={Colon-X: Advancing Intelligent Colonoscopy from Multimodal Understanding to Clinical Reasoning},
    author={Ji, Ge-Peng and Liu, Jingyi and Fan, Deng-Ping and Barnes, Nick},
    journal={arXiv preprint arXiv:2512.03667},
    year={2025}
}
```

# License

This project utilizes certain datasets and checkpoints that are subject to their respective original licenses. Users must comply with all terms and conditions of these original licenses. The content of this project itself is licensed under the Apache license 2.0.