metadata
base_model: microsoft/Florence-2-base
library_name: peft
tags:
- medical
- vision-language-model
- vqa
- radiology
Generative AI Radiology VLM (Florence-2)
This model is a Parameter-Efficient Fine-Tuned (PEFT/LoRA) version of Microsoft's Florence-2-base. It has been specifically trained on the VQA-RAD dataset to act as a Generative AI Vision-Language Model capable of answering free-form textual questions about medical X-Rays.
Model Details
- Architecture: Vision Encoder + Text Decoder (Florence-2)
- Task: Medical Visual Question Answering (VQA)
- Fine-Tuning Technique: Low-Rank Adaptation (LoRA)
- Target Modules:
q_proj,v_proj,o_proj
Training Results
The model was fine-tuned for 3 epochs on an NVIDIA A100-40GB GPU using mixed precision (fp16). The training loss steadily decreased, demonstrating strong anatomical and vocabulary convergence.
Local Web UI (Gradio)
The repository includes a local app.py script that loads these LoRA adapters and spins up a local web UI for inference.
Framework versions
- PEFT 0.11.1
- Transformers 4.42.4