Create README.md

README.md

@@ -0,0 +1,187 @@
+
+# AlignGPT: Multi-modal Large Language Models with Adaptive Alignment Capability
+[[Project Page](https://aligngpt-vl.github.io/)] [[Paper](https://arxiv.org/abs/2405.14129)] [[Demo](http://47.116.173.89:7870/)] [[Model](https://huggingface.co/nlpzhaof)]
+
+
+
+Authors: [Fei Zhao*](https://scholar.google.com/citations?user=V01xzWQAAAAJ&hl=zh-CN), Taotian Pang*, Chunhui Li, [Zhen Wu](https://scholar.google.com/citations?user=IoGlgtoAAAAJ&hl=zh-CN), Junjie Guo, Shangyu Xing, [Xinyu Dai](https://scholar.google.com/citations?user=zpWB1CgAAAAJ&hl=zh-CN)
+
+<div align="center">
+<img src="./assert/architecture.png" width="800px">
+</div>
+
+<!-- ![architecture](./assert/architecture.png) -->
+
+## News and Updates
+- [5/24] 🔥 We released **AlignGPT: Multi-modal Large Language Models with Adaptive Alignment Capability**.  Checkout the [paper](https://arxiv.org/abs/2405.14129) and [demo](http://47.116.173.89:7870/).
+- [5/24] 🔥 The data is not ready yet. We will upload it within a week.
+
+
+## Contents
+- [Install](#install)
+- [Model Zoo](#model-zoo)
+- [Demo](#demo)
+- [Training](#training)
+- [Evaluation](#evaluation)
+- [Performance](#performance)
+
+## Install
+
+### Docker
+
+We recommend to use docker to prepare the environment.
+
+1. Clone this repository and navigate to AlignGPT folder
+
+```bash
+git clone https://github.com/AlignGPT-VL/AlignGPT.git
+cd AlignGPT
+```
+
+2. Build the docker image
+
+```bash
+cd deploy
+docker build -t aligngpt:1.0 .
+```
+
+If your machine cannot connect to github to download the flash attention pip wheel, you can download it manually on https://github.com/Dao-AILab/flash-attention/releases/download/v2.5.5/flash_attn-2.5.5+cu118torch2.1cxx11abiFALSE-cp310-cp310-linux_x86_64.whl and put it to `deploy/flash_attn-2.5.5+cu118torch2.1cxx11abiFALSE-cp310-cp310-linux_x86_64.whl`.
+
+3. To start the container, run the following command in the project root directory
+
+```bash
+docker run --gpus all --ipc=host --network=host --rm -it -v .:/workspace aligngpt:1.0
+```
+
+More `-v` options can be added to mount the data and output directories.
+
+### Conda
+
+1. Clone this repository and navigate to AlignGPT folder
+
+```bash
+git clone https://github.com/AlignGPT-VL/AlignGPT.git
+cd AlignGPT
+```
+
+2. Install Package
+
+```Shell
+conda create -n aligngpt python=3.10 -y
+conda activate aligngpt
+pip install --upgrade pip  # enable PEP 660 support
+pip install torch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 --index-url https://download.pytorch.org/whl/cu118
+pip install -r deploy/requirements.txt
+```
+
+Finally, you need to install flash-attention manually before running the model.
+
+## Model Zoo
+
+Please download the weights for LLM, Vision Backbone and place them in the `./playground/model` folder, we also provide all the weights for the AlignGPT checkpoint.
+
+| Model | LLM | Vision Backbone | Pre-training | Instruct-tuning | 
+|----------|----------|-----------|---|---|
+| AlignGPT-7B | [Vicuna 7B](https://huggingface.co/lmsys/vicuna-7b-v1.5) | [CLIP ViT-L/14](https://huggingface.co/openai/clip-vit-large-patch14-336) |[aligngpt-7b-pretrain](https://huggingface.co/nlpzhaof/aligngpt-7b-pretrain/tree/main)| [aligngpt-7b](https://huggingface.co/nlpzhaof/aligngpt-7b/tree/main)|
+| AlignGPT-13B | [Vicuna 13B](https://huggingface.co/lmsys/vicuna-13b-v1.5) | [CLIP ViT-L/14](https://huggingface.co/openai/clip-vit-large-patch14-336) |[aligngpt-13b-pretrain](https://huggingface.co/nlpzhaof/aligngpt-13b-pretrain/tree/main)| [aligngpt-13b](https://huggingface.co/nlpzhaof/aligngpt-13b/tree/main)|
+| AlignGPT-LLaMA2 | [LLaMA-2-7B-Chat](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) | [CLIP ViT-L/14](https://huggingface.co/openai/clip-vit-large-patch14-336) |To be released| To be released|
+| AlignGPT-LLaMA3 | [LLaMA-3-8B-Base](https://huggingface.co/meta-llama/Meta-Llama-3-8B) | [CLIP ViT-L/14](https://huggingface.co/openai/clip-vit-large-patch14-336) |To be released|To be released|
+
+## Demo
+
+### Start Gradio UI
+You can start gradio service with the following command:
+
+```
+cd AlignGPT
+bash start_api.sh
+```
+This script will launch three processes: the controller, the Gradio web server, and the model worker, all of which will run in the background. You can view logs of these processes in folder `log/`, and view process status with command `ps -ef | grep src.serve`.
+
+### CLI Inference
+Chat about images using AlignGPT without the need of Gradio interface. 
+```
+python -m src.serve.cli \
+    --model-path playground/model/aligngpt-13b \
+    --image-file "image folder/image.jpg" \
+```
+
+## Training
+
+We place all training data in the `./playground/data` folder. Please download [aligngpt_pretrain_data]() from HuggingFace and place it in `./playground/data`. The details are introduced below.
+
+### Pre-training
+* **Dataset**: We use the 558K image-text pairs in the pre-training phase. Organize them in `./playground/data` as follows:
+
+```
+├── LLaVA-Pretrain
+│   └── blip_laion_cc_sbu_558k_with_similarity_number.json
+│   └── images
+```
+
+* **Run**: You can launch the pre-training phase using the following command:
+    ```
+    bash scripts/pretrain.sh
+    ```
+Before running the script of pretraining, you should set the arguments related to **directories** of model checkpoints, data and outputs, *i.e.*, `model_name_or_path`, `data_path`, `image_folder`, `vision_tower` and `output_dir`.
+
+### Instruction-tuning
+* **Dataset**: We used 665K image-text pairs/text data in the instruction-tuning phase. The images corresponding to these data include: `COCO`, `GQA`, `OCR-VQA`, `TextVQA`, and `VisualGenome`. Organize them in `./playground/data` as follows:
+
+```
+├── llava_v1_5_mix665k.json
+├── coco
+│   └── train2017
+├── gqa
+│   └── images
+├── ocr_vqa
+│   └── images
+├── textvqa
+│   └── train_images
+└── vg
+    ├── VG_100K
+    └── VG_100K_2
+```
+
+* **Run**: You can launch the instruction-tuning stage using the following command:
+    ```
+    bash scripts/finetune.sh
+    ```
+Before running the script of instruction tuning, you should set the argument `pretrain_mm_mlp_align`, which is the path where you store the weights of the pre-training phase.
+
+## Evaluation
+
+We conduct evaluation on 12 benchmarks. The dataset to be evaluated is placed in `./playground/data/eval`. Please download [aligngpt_eval_data]() from HuggingFace and place it in `./playground/data/eval`. It contains custom annotations, scripts, and prediction files for AlignGPT. Here, we demonstrate how to evaluate the performance of our model on `MME` dataset. We use the following command to run the evaluation stage:
+```
+CUDA_VISIBLE_DEVICES=0 bash scripts/eval/mme.sh
+```
+You should set the directories of the model checkpoints and datasets in the scripts before running it. The evaluation of other datasets can be found in [Evaluation.md](docs/Evaluation.md).
+
+## Performance
+| Model | VQAv2 | GQA | VizWiz | SQA | T-VQA | POPE | MME | MM-Bench | MM-Bench-CN | SEED | LLaVA-Bench-Wild | MM-Vet |
+|----------|---|---|---|---|---|---|---|---|---|---|---|---|
+| AlignGPT-7B | 79.1 | 62.9 | 54.2 | 68.5 | 58.4 | 86.0 | 1527.4 | 67.3 | 59.9 | 66.5 | 68.4 | 30.8 |
+| AlignGPT-13B | 80.0 | 63.6 | 56.4 | 70.3 | 60.2 | 86.2 | 1572.0 | 69.5 | 63.7 | 67.8 | 75.2 | 35.6 |
+
+## Citation
+If you find AlignGPT useful for your research and applications, please cite using this BibTeX:
+```
+@misc{zhao2024aligngpt,
+      title={AlignGPT: Multi-modal Large Language Models with Adaptive Alignment Capability}, 
+      author={Fei Zhao and Taotian Pang and Chunhui Li and Zhen Wu and Junjie Guo and Shangyu Xing and Xinyu Dai},
+      year={2024},
+      eprint={2405.14129},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```
+
+## Acknowledgement
+We build our project based on [LLaVA: Large Language and Vision Assistant](https://github.com/haotian-liu/LLaVA).
+
+## License
+
+[![Code License](https://img.shields.io/badge/Code%20License-Apache_2.0-green.svg)](https://github.com/tatsu-lab/stanford_alpaca/blob/main/LICENSE)
+[![Data License](https://img.shields.io/badge/Data%20License-CC%20By%20NC%204.0-red.svg)](https://github.com/tatsu-lab/stanford_alpaca/blob/main/DATA_LICENSE)
+
+The data and checkpoint is intended and licensed for research use only. They are also restricted to uses that follow the license agreement of LLaMA, Vicuna and GPT-4. The dataset is CC BY NC 4.0 (allowing only non-commercial use) and models trained using the dataset should not be used outside of research purposes.