Improve model card: add pipeline tag, library, paper & code links, Gradio demo usage

README.md

@@ -1,3 +1,111 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+pipeline_tag: any-to-any
+library_name: transformers
+---
+
+# MedPLIB: Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine
+
+This repository contains the official implementation of the paper: [**Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine**](https://huggingface.co/papers/2412.09278).
+
+<p align="center">
+    <img src="https://github.com/ShawnHuang497/MedPLIB/raw/main/assets/logo.png" width="150" style="margin-bottom: 0.2;"/>
+</p>
+
+<p align="center">
+    <a href="https://github.com/ShawnHuang497/MedPLIB"><img src="https://img.shields.io/badge/GitHub-Code-blue.svg?logo=github&style=flat-square" alt="GitHub Code"></a>
+</p>
+
+## Abstract
+Multimodal Large Language Models (MLLMs) have shown promising progress in understanding and analyzing video content. However, processing long videos remains a significant challenge constrained by LLM's context size. To address this limitation, we propose LongVU, a spatiotemporal adaptive compression mechanism thats reduces the number of video tokens while preserving visual details of long videos. Our idea is based on leveraging cross-modal query and inter-frame dependencies to adaptively reduce temporal and spatial redundancy in videos. Specifically, we leverage DINOv2 features to remove redundant frames that exhibit high similarity. Then we utilize text-guided cross-modal query for selective frame feature reduction. Further, we perform spatial token reduction across frames based on their temporal dependencies. Our adaptive compression strategy effectively processes a large number of frames with little visual information loss within given context length. Our LongVU consistently surpass existing methods across a variety of video understanding benchmarks, especially on hour-long video understanding tasks such as VideoMME and MLVU. Given a light-weight LLM, our LongVU also scales effectively into a smaller size with state-of-the-art video understanding performance.
+
+## Highlights
+MedPLIB shows excellent performance in pixel-level understanding in biomedical field.
+
+- ✨ MedPLIB is a biomedical MLLM with a huge breadth of abilities and supports multiple imaging modalities. Not only can it perform image-level visual language tasks like VQA, but it also facilitates question answering at the pixel level.
+
+<p align="center">
+    <img src="https://github.com/ShawnHuang497/MedPLIB/raw/main/assets/capa.png"  style="margin-bottom: 0.2;"/>
+</p>
+
+- ✨ We constructe MeCoVQA Dataset. It comprises an array of 8 modalities with a total of 310k pairs for complex medical imaging question answering and image region understanding.
+<p align="center">
+    <img src="https://github.com/ShawnHuang497/MedPLIB/raw/main/assets/data.png"  style="margin-bottom: 0.2;"/>
+</p>
+
+## Installation
+For detailed instructions, please refer to the [GitHub repository](https://github.com/ShawnHuang497/MedPLIB).
+
+1.  **Clone this repository and navigate to MedPLIB folder**
+    ```bash
+    git clone https://github.com/ShawnHuang497/MedPLIB.git
+    cd MedPLIB
+    ```
+2.  **Install Package**
+    ```Shell
+    conda create -n medplib python=3.10 -y
+    conda activate medplib
+    pip install --upgrade pip 
+    pip install -r requirements.txt
+    ```
+3.  **Install additional packages for training cases**
+    ```Shell
+    pip install ninja==1.11.1.1
+    pip install flash-attn==2.5.2 --no-build-isolation
+    ```
+
+## Sample Usage: Gradio Web UI
+We recommend trying our web demo, which includes all the features currently supported by MedPLIB. To run our demo, you need to download or train MedPLIB to make the checkpoints locally. Please run the following commands one by one.
+
+```bash
+# launch the server controller
+python -m model.serve.controller --host 0.0.0.0 --port 64000
+```
+
+```bash
+# launch the web server
+python -m model.serve.gradio_web_server --controller http://localhost:64000 --model-list-mode reload --add_region_feature --port 64001 
+```
+
+```bash
+# launch the model worker
+CUDA_VISIBLE_DEVICES=0 python -m model.serve.model_worker --host localhost --controller http://localhost:64000 --port 64002 --worker http://localhost:64002 --model-path /path/to/the/medplib_checkpoints --add_region_feature --device_map cuda --vision_pretrained /path/to/the/sam-med2d_b.pth
+```
+
+- **Pixel grounding:** 
+<p align="center">
+    <img src="https://github.com/ShawnHuang497/MedPLIB/raw/main/assets/seg.gif"  style="width: 70%;"/>
+</p>
+
+- **Region VQA:**
+<p align="center">
+    <img src="https://github.com/ShawnHuang497/MedPLIB/raw/main/assets/rqa.gif"  style="width: 70%;"/>
+</p>
+
+- **VQA:**
+<p align="center">
+    <img src="https://github.com/ShawnHuang497/MedPLIB/raw/main/assets/vqa.gif"  style="width: 70%;"/>
+</p>
+
+## Acknowledgement
+We thank the following works for giving us the inspiration and part of the code: [LISA](https://github.com/dvlab-research/LISA), [MoE-LLaVA](https://github.com/PKU-YuanGroup/MoE-LLaVA), [LLaVA](https://github.com/haotian-liu/LLaVA), [SAM-Med2D](https://github.com/OpenGVLab/SAM-Med2D), [SAM](https://github.com/facebookresearch/segment-anything) and [SEEM](https://github.com/UX-Decoder/Segment-Everything-Everywhere-All-At-Once).
+
+## Model Use
+### Intended Use
+The data, code, and model checkpoints are intended to be used solely for (I) future research on visual-language processing and (II) reproducibility of the experimental results reported in the reference paper. The data, code, and model checkpoints are not intended to be used in clinical care or for any clinical decision making purposes.
+### Primary Intended Use
+The primary intended use is to support AI researchers reproducing and building on top of this work. MedPLIB and its associated models should be helpful for exploring various biomedical pixel grunding and vision question answering (VQA) research questions.
+### Out-of-Scope Use
+Any deployed use case of the model --- commercial or otherwise --- is out of scope. Although we evaluated the models using a broad set of publicly-available research benchmarks, the models and evaluations are intended for research use only and not intended for deployed use cases. 
+
+## Citation
+If you find our paper and code useful in your research, please consider giving a star and citation.
+
+```bibtex
+@article{huang2024towards,
+  title={Towards a Multimodal Large Language Model with Pixel-Level Insight for Biomedicine},
+  author={Huang, Xiaoshuang and Shen, Lingdong and Liu, Jia and Shang, Fangxin and Li, Hongxiang and Huang, Haifeng and Yang, Yehui},
+  journal={arXiv preprint arXiv:2412.09278},
+  year={2024}
+}
+```