THP2903/finetuning_medical_MiniCPM-V-2_6_8B_finding

MiniCPM-V-2.6-8B – Stage 1 (Findings)

1. Model Overview

This model is part of a Vision-Language AI system designed for chest X-ray analysis in Vietnamese clinical settings.

The full pipeline consists of 3 stages:

• Stage 1: Findings generation (image → radiology findings)
• Stage 2: Impression generation (image → clinical impression)
• Stage 3: Multi-turn conversation (findings + impression + dialogue)

This repository corresponds to:

• Stage: 1 (Findings)
• Task: Generate radiology findings from chest X-ray images
• Domain: Vietnamese medical imaging (Chest X-ray)

The model is fine-tuned from MiniCPM-V and evaluated against multiple architectures (InternVL, Vintern, Qwen2-VL, MiniCPM-V, LaVy).

Among all models, Qwen2-VL-7B achieved the best performance, but this model is provided for benchmarking and comparison.

---

2. Installation

pip install Pillow==10.1.0 torch==2.1.2 torchvision==0.16.2 transformers==4.40.0 sentencepiece==0.1.99 decord

---

3. Inference

GPU with bfloat16 is recommended.

import torch
from PIL import Image
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained(
    "THP2903/finetuning_medical_MiniCPM-V-2_6_8B_finding",
    trust_remote_code=True,
    attn_implementation="sdpa",
    torch_dtype=torch.bfloat16
)

model = model.eval().cuda()

tokenizer = AutoTokenizer.from_pretrained(
    "THP2903/finetuning_medical_MiniCPM-V-2_6_8B_finding",
    trust_remote_code=True
)

image = Image.open("your_image.jpg").convert("RGB")

question = "Ảnh chụp xray benh nhân nam, 48 tuổi PA có dấu hieu gi?"

msgs = [
    {
        "role": "user",
        "content": [image, question]
    }
]

res = model.chat(
    image=None,
    msgs=msgs,
    tokenizer=tokenizer
)

print(res)

# Streaming mode (optional)
res = model.chat(
    image=None,
    msgs=msgs,
    tokenizer=tokenizer,
    sampling=True,
    stream=True
)

generated_text = ""
for new_text in res:
    generated_text += new_text
    print(new_text, flush=True, end="")

---

4. Notes

• Input must be a chest X-ray image
• Output is radiology findings (not final diagnosis)
• This model follows the original MiniCPM-V inference pipeline without modification
• For best performance, consider using Qwen2-VL-7B