---
license: apache-2.0
language:
- en
base_model:
- openai/clip-vit-large-patch14
pipeline_tag: image-to-text
tags:
- medical
- retinal
---


# RetinalGPT: Large Language-and-Vision Assistant for Retinal Health 👁️


**RetinalGPT** is a specialized multimodal vision-language model (VLM) based on the **LLaVA-v1.5** architecture. It is specifically engineered for the high-precision domain of **ophthalmology**, with a focus on interpreting retinal fundus photography and Optical Coherence Tomography (OCT) scans.

---

## 📌 Model Summary

RetinalGPT bridges the gap between general-purpose VLMs and specialized ophthalmic diagnostics. By fine-tuning on a curated corpus of retinal image-text pairs, the model demonstrates advanced capabilities in identifying pathologies such as Diabetic Retinopathy (DR), Glaucoma, and Age-related Macular Degeneration (AMD).

- **Base LLM:** Llama-7b 
- **Vision Tower:** CLIP-ViT-L-14-336px
- **Connector:** MLP Projection Layer
- **Domain:** Ophthalmology / Retinal Imaging

---

## 🚀 Key Capabilities

RetinalGPT is trained to perform complex visual reasoning tasks including:
* **Automated Screening:** Grading Diabetic Retinopathy severity (Stage 0-4).
* **Lesion Characterization:** Identifying and describing microaneurysms, hemorrhages, and exudates.
* **Anatomical Mapping:** Precise description of the optic disc, cup-to-disc ratio, and foveal reflex.
* **Clinical QA:** Engaging in multi-turn dialogues about specific clinical findings in a retinal scan.

---

## 💻 How to Use

RetinalGPT follows the standard LLaVA inference pipeline. You will need the `llava` library installed.

### Usage

```Python
from llava.model.builder import load_pretrained_model
from llava.mm_utils import process_images, tokenizer_image_token, get_model_name_from_path
from llava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN, DEFAULT_IM_START_TOKEN, DEFAULT_IM_END_TOKEN
from PIL import Image
import torch

model_path = "your-username/retinalgpt"
model_name = get_model_name_from_path(model_path)

tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path=model_path, 
    model_base=None, 
    model_name=model_name
)

# Prepare Image
image = Image.open("fundus_sample.jpg")
image_tensor = image_processor.preprocess(image, return_tensors='pt')['images'].half().cuda()

prompt = "Can you describe this image?"

input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors='pt').unsqueeze(0).cuda()

# Generate Response
with torch.inference_mode():
    output_ids = model.generate(
        input_ids,
        images=image_tensor,
        do_sample=True,
        temperature=0.2,
        max_new_tokens=512,
        use_cache=True
    )
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
```