metadata
license: bsd-3-clause
language:
- en
metrics:
- accuracy
- meteor
- rouge
base_model:
- Vision-CAIR/MiniGPT-4
tags:
- medical
PeFoMed
This is the official implementation of PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imaging.
Datasets
The configuration of all datasets needs to be set in the corresponding dataset configuration file in the pefomed/configs/datasets/medical
Stage 1 finetune datasets: ROCO, CLEF2022, MEDICAT, and MIMIC-CXR
Stage 2 finetune medical VQA datasets: VQA-RAD, PathVQA and Slake.
Stage 2 finetune MRG dataset: IU-Xray
Acknowledgement
If you're using PeFoMed in your research or applications, please cite using this BibTeX:
@misc{liu2024pefomedparameterefficientfinetuning,
title={PeFoMed: Parameter Efficient Fine-tuning of Multimodal Large Language Models for Medical Imaging},
author={ Jinlong He and Gang Liu and Pengfei Li and Genrong He and Zhaolin Chen and Shenjun Zhong},
year={2024},
eprint={2401.02797},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2401.02797},
}
License
This repository is under BSD 3-Clause License.
Many codes are based on Lavis and MiniGPT-v2