Instructions to use Siddartha01/blip-medical-captioning-roco with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Siddartha01/blip-medical-captioning-roco with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="Siddartha01/blip-medical-captioning-roco")

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("Siddartha01/blip-medical-captioning-roco")
model = AutoModelForMultimodalLM.from_pretrained("Siddartha01/blip-medical-captioning-roco")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Siddartha01/blip-medical-captioning-roco with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Siddartha01/blip-medical-captioning-roco"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Siddartha01/blip-medical-captioning-roco",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/Siddartha01/blip-medical-captioning-roco

SGLang

How to use Siddartha01/blip-medical-captioning-roco with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Siddartha01/blip-medical-captioning-roco" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Siddartha01/blip-medical-captioning-roco",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Siddartha01/blip-medical-captioning-roco" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Siddartha01/blip-medical-captioning-roco",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use Siddartha01/blip-medical-captioning-roco with Docker Model Runner:
```
docker model run hf.co/Siddartha01/blip-medical-captioning-roco
```

Medical Report Generator (BLIP fine-tuned on ROCO)

Fine-tuned BLIP model for generating radiology reports from medical images, developed as part of a capstone project in medical image analysis.

This model generates descriptive captions for radiology images (e.g., X-rays, CT scans) using the Salesforce/blip-image-captioning-base as the base model, fine-tuned on the eltorio/ROCOv2-radiology dataset.

Model Details

Model Description

Developed by a Computer Science student focusing on ML for medical imaging.
Funded by: optional (e.g., academic project).
Shared by: Siddartha01.
Model type: BLIP (vision-language).
Languages: English (NLP for captions).
License: apache-2.0 (or specify your choice).
Finetuned from model: Salesforce/blip-image-captioning-base.

Model Sources

Repository: https://huggingface.co/Siddartha01/blip-medical-captioning-roco
Dataset: eltorio/ROCOv2-radiology.
Demo: Siddartha01/medical-report-generator.

Uses

Direct Use

This model generates radiology-style captions from medical images without further fine-tuning. Suitable for research, education, and prototyping medical report generators. Example: Input chest X-ray → Output: "Chest X-ray shows normal lung fields."

Downstream Use

Integrate into web apps with Gradio/Hugging Face Spaces, RAG systems for reports, or LLMs for detailed analysis.

Out-of-Scope Use

Not for clinical diagnosis, real-world medical decisions, or non-radiology images. Misuse may lead to inaccurate reports.

Bias, Risks, and Limitations

Medical datasets like ROCO have biases toward common pathologies and English-language captions from publications. Model may underperform on rare conditions, diverse demographics, or low-quality images. Risks include over-reliance leading to misdiagnosis—use only for research.

Recommendations

Users should validate outputs with experts, disclose model limitations in deployments, and fine-tune further on domain-specific data. Monitor for hallucinations in generated reports.

How to Get Started with the Model

from transformers import BlipProcessor, BlipForConditionalGeneration
from PIL import Image
import requests

model = BlipForConditionalGeneration.from_pretrained("[YOUR HF USERNAME]/[YOUR REPO NAME]")
processor = BlipProcessor.from_pretrained("[YOUR HF USERNAME]/[YOUR REPO NAME]")

url = "https://example.com/radiology-image.jpg"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")

inputs = processor(images=image, return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
caption = processor.decode(outputs, skip_special_tokens=True)

print("Generated Report:", caption)

Downloads last month: 45

Safetensors

Model size

0.2B params

Tensor type

F32

Inference Providers NEW

Image-Text-to-Text

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Siddartha01
/

blip-medical-captioning-roco