Upload README.md

README.md

@@ -1,3 +1,101 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+datasets:
+- TIGER-Lab/MMEB-train
+language:
+- en
+base_model:
+- Qwen/Qwen2-VL-7B-Instruct
+library_name: transformers
+---
+
+A new checkpoint trained using [Qwen/Qwen2-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct) with an enhanced training setup (LoRA tuning, batch size of 2048, maximum sub-dataset size of 100k). This model has shown significantly improved performance on MMEB & Flickr30K compared to the previous models using Phi-3.5 and llava-v1.6-mistral as backbone.
+
+This repo contains the code and data for [VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks](https://arxiv.org/abs/2410.05160). In this paper, we focus on building a unified multimodal embedding model suitable for a wide range of tasks. Our approach is based on transforming an existing, well-trained Vision-Language Model (VLM) into an embedding model.
+
+## Github
+ - [Github](https://github.com/TIGER-AI-Lab/VLM2Vec)
+
+
+## Data
+
+Our model is being trained on MMEB-train and evaluated on MMEB-eval with contrastive learning. We only use in-batch negatives for training. 
+
+ - Train data: https://huggingface.co/datasets/TIGER-Lab/MMEB-train
+ - Eval data: https://huggingface.co/datasets/TIGER-Lab/MMEB-eval
+
+
+## Performance
+This model outperforms the baselines and previous version of VLM2Vec by a large margin.
+
+![image/png](https://github.com/TIGER-AI-Lab/VLM2Vec/blob/main/figures/vlm2vec_results.png?raw=true)
+
+
+## How to use VLM2Vec
+(More details please refer to our Github repo, here is just a simple demo.)
+
+First you can clone our github
+```bash
+git clone https://github.com/TIGER-AI-Lab/VLM2Vec.git
+pip -r requirements.txt
+```
+
+```python
+from src.model import MMEBModel
+from src.arguments import ModelArguments
+from src.utils import load_processor
+
+import torch
+from transformers import HfArgumentParser, AutoProcessor
+from PIL import Image
+import numpy as np
+
+
+model_args = (
+    model_name='TIGER-Lab/VLM2Vec-Qwen2VL',
+    pooling='last',
+    normalize=True,
+    model_backbone='qwen2_vl')
+
+processor = load_processor(model_args)
+
+model = MMEBModel.load(model_args)
+model.eval()
+model = model.to('cuda', dtype=torch.bfloat16)
+
+# Image + Text -> Text
+inputs = processor(text='<image> Represent the given image with the following question: What is in the image',
+                   images=Image.open('figures/example.jpg'),
+                   return_tensors="pt")
+inputs = {key: value.to('cuda') for key, value in inputs.items()}
+qry_output = model(qry=inputs)["qry_reps"]
+
+string = 'A cat and a dog'
+inputs = processor(text=string,
+                   images=None,
+                   return_tensors="pt")
+inputs = {key: value.to('cuda') for key, value in inputs.items()}
+tgt_output = model(tgt=inputs)["tgt_reps"]
+print(string, '=', model.compute_similarity(qry_output, tgt_output))
+## A cat and a dog = tensor([[0.4414]], device='cuda:0', dtype=torch.bfloat16)
+
+string = 'A cat and a tiger'
+inputs = processor(text=string,
+                   images=None,
+                   return_tensors="pt")
+inputs = {key: value.to('cuda') for key, value in inputs.items()}
+tgt_output = model(tgt=inputs)["tgt_reps"]
+print(string, '=', model.compute_similarity(qry_output, tgt_output))
+## A cat and a tiger = tensor([[0.3555]], device='cuda:0', dtype=torch.bfloat16)
+
+```
+
+
+## Citation
+```
+@article{jiang2024vlm2vec,
+  title={VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks},
+  author={Jiang, Ziyan and Meng, Rui and Yang, Xinyi and Yavuz, Semih and Zhou, Yingbo and Chen, Wenhu},
+  journal={arXiv preprint arXiv:2410.05160},
+  year={2024}
+}