|
|
--- |
|
|
language: en |
|
|
tags: |
|
|
- clip |
|
|
- medical-imaging |
|
|
- radiology |
|
|
- roco |
|
|
- vision-language |
|
|
base_model: openai/clip-vit-base-patch32 |
|
|
datasets: |
|
|
- eltorio/ROCO-radiology |
|
|
metrics: |
|
|
- recall |
|
|
license: mit |
|
|
--- |
|
|
|
|
|
# ROCO-Radiology-CLIP (ViT-B/32) |
|
|
|
|
|
> **A specialized vision-language model for radiology, fine-tuned on the ROCO dataset.** |
|
|
|
|
|
This model aligns medical images (X-rays, CTs, MRIs) with their textual descriptions, enabling **zero-shot classification** and **semantic search** for radiology concepts. |
|
|
|
|
|
## Performance (Test Set) |
|
|
- **Batch-wise Recall@1:** 70.83% (State-of-the-art for T4 fine-tuning) |
|
|
- **Batch-wise Recall@5:** 96.99% |
|
|
- **Global Retrieval Recall@1:** ~6% (500x better than random chance) |
|
|
- **Global Retrieval Recall@5:** ~16% |
|
|
Though a lot of work need to be done on this as the recall is still quite low. It will be updated with newer version |
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
from transformers import CLIPProcessor, CLIPModel |
|
|
from PIL import Image |
|
|
|
|
|
model = CLIPModel.from_pretrained("spicy03/CLIP-ROCO-v1") |
|
|
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32") |
|
|
|
|
|
# Predict |
|
|
image = Image.open("chest_xray.jpg") |
|
|
labels = ["Pneumonia", "Normal", "Edema"] |
|
|
inputs = processor(text=labels, images=image, return_tensors="pt", padding=True) |
|
|
outputs = model(**inputs) |
|
|
probs = outputs.logits_per_image.softmax(dim=1) |
|
|
print(probs) |
