Update README.md

README.md

@@ -1,3 +1,24 @@
+## About `GME-VARCO-VISION-Embedding`
+`GME-VARCO-VISION-Embedding` is a multimodal embedding model that computes semantic similarity between text, images, and videos in a high-dimensional embedding space. In particular, the model focuses on video retrieval, which demands greater complexity and contextual understanding compared to image retrieval. It achieves high retrieval accuracy and strong generalization performance across various scenarios, such as scene-based search, description-based search, and question-answering-based search.
+
+
+### Model Architecture and Training Method
+`GME-VARCO-VISION-Embedding` is based on [`Qwen/Qwen2-VL-7B-Instruct`](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct), and uses the parameters of [`Alibaba-NLP/gme-Qwen2-VL-7B-Instruct`](https://huggingface.co/Alibaba-NLP/gme-Qwen2-VL-7B-Instruct) to improve the model's general retrieval ability.
+
+#### 1. Fine-tuning (Contrastive Learning) on video preference dataset
+To efficiently fine-tune the model, we utilize [ShareGPTVideo’s 17𝑘 video preference dataset](https://huggingface.co/datasets/ShareGPTVideo/train_video_and_instruction), which includes prompts, videos, gold answers, and chosen-rejected text pairs. We treat the prompts and videos as queries, and the rejected responses
+as hard-negatives for the gold answers. . Each query is trained with in-batch negatives as well as one hard negative using the InfoNCE loss. The model is fully fine-tuned for two epochs on 8 A100 GPUs with a batch size of 8, requiring only a few hours for training.
+
+#### 2. Adding Retrieval Vector
+To compensate for the insufficiency of training instances and enhance the generalization ability of the fine-tuned model, we compute a retrieval vector 𝜏 by subtracting the weights of the original [`Qwen/Qwen2-VL-7B-Instruct`](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct) model from those of [`Alibaba-NLP/gme-Qwen2-VL-7B-Instruct`](https://huggingface.co/Alibaba-NLP/gme-Qwen2-VL-7B-Instruct), a Qwen2-VL based image-text retrieval model. This approach is inspired by Chat Vector, which is a method to equip pre-trained language models with chat capabilities in new languages by adding a vector obtained from the weight difference between a base model and its chat-optimized counterpart.
+
+<br>
+
+### Performance
+Our model achieves **state-of-the-art (SOTA) zero-shot performance** on the MultiVENT2.0 dataset as of July 2025. See the [official leaderboard](https://eval.ai/web/challenges/challenge-page/2507/leaderboard/6262) for detailed results.
+
+<br>
+
 ## Demo Video
 Check out our demo videos showcasing our multimodal embedding model in action:
 - [English Demo Video](https://www.youtube.com/watch?v=kCvz82Y1BQg)
@@ -5,17 +26,172 @@ Check out our demo videos showcasing our multimodal embedding model in action:
 
 The demo demonstrates how our embedding model works together with an AI agent to search for relevant videos based on user queries and generate responses using the retrieved video content.
 
+<br>
+
+## Code Examples
+`GME-VARCO-VISION-Embedding` adopts the inference pipeline of [`Qwen/Qwen2-VL-7B-Instruct`](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct).
+
+### Image-Text Retrieval
+
+```python
+import torch
+import requests
+from PIL import Image
+from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
+
+model_name = "NCSOFT/GME-VARCO-VISION-Embedding"
+
+model = Qwen2VLForConditionalGeneration.from_pretrained(
+    model_name, 
+    torch_dtype=torch.bfloat16,
+    attn_implementation="flash_attention_2",
+    device_map="auto",
+)
+
+tokenizer = AutoTokenizer.from_pretrained(
+        model_name,
+        padding_side="left",
+    )
+
+
+processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct")
+processor.tokenizer = tokenizer
+device = model.device
+
+
+qry_msg = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "text", "text": "Find a photo of a cat."},
+        ],
+    },
+]
+
+qry_txt = processor.apply_chat_template(
+    qry_msg, tokenize=False, add_generation_prompt=True
+) + tokenizer.eos_token
 
-## Performance
-Our model achieves **state-of-the-art (SOTA) performance** on the MultiVENT2.0 dataset in zero-shot settings. See the [official leaderboard](https://eval.ai/web/challenges/challenge-page/2507/leaderboard/6262) for detailed results.
+qry_input = processor(
+            text=[qry_txt],
+            padding=True,
+            return_tensors="pt",
+        ).to("cuda")  
 
 
+img_msg = [
+    {
+        "role": "user",
+        "content": [{
+            "type": "image",
+            "image": "image"
+        }]
+    }
+]
 
-## Release Schedule
-**Coming Soon**
-- Model checkpoints and agent prompts will be released shortly
-- Research paper will be published soon
+img_txt = processor.apply_chat_template(
+    img_msg, tokenize=False, add_generation_prompt=True
+) + tokenizer.eos_token
+
+
+candidate_imgs= [
+        # Photo of two cats
+        {
+            "role": "user",
+            "content": [{
+                "type": "image",
+                "image": "http://images.cocodataset.org/val2017/000000039769.jpg"}]
+        },
+        # Photo of two dogs
+        {
+            "role": "user",
+            "content": [{
+                "type": "image",
+                "image": "https://farm1.staticflickr.com/116/290755713_a5de6c1079_z.jpg"}]
+        },
+        # photo of two people playing baseball
+        {
+            "role": "user",
+            "content": [{
+                "type": "image",
+                "image": "http://farm3.staticflickr.com/2418/2193688811_d9f5e23bbd_z.jpg"}]
+        },
+        # Photo of a large crowd in a busy city street
+        {
+            "role": "user",
+            "content": [{
+                "type": "image",
+                "image":"http://farm7.staticflickr.com/6049/6329686751_997c68fff9_z.jpg"}]
+        },
+    ]
+
+candidate_images, _ = process_vision_info(candidate_imgs)
+
+image_inputs = processor(
+        text=[img_txt] * len(candidate_images),
+        images=candidate_images,
+        # videos=,
+        padding=True,
+        return_tensors="pt",
+    ).to("cuda")
+
+with torch.inference_mode():
+    qry_emb = model(
+        **qry_input, output_hidden_states=True, return_dict=True
+    ).hidden_states[-1][:, -1, :]
+
+    img_emb = model(
+        **image_inputs, output_hidden_states=True, return_dict=True
+    ).hidden_states[-1][:, -1, :]
+
+qry_emb = F.normalize(qry_emb, dim=-1)    
+img_emb = F.normalize(img_emb, dim=-1)
+
+score = qry_emb @ img_emb.t()
+# tensor([[0.2432, 0.0962, 0.0747, 0.0898]], device='cuda:0', dtype=torch.bfloat16)
+# corresponding to the score of photos (cat, dog, baseball, crowd) 
+```
+<br>
+
+### Video Embedding
+```Python
+vid_message = [
+        {
+            "role": "user",
+            "content": [{
+                "type": "video",
+                "video": video_path,
+                "max_pixels": 262144,
+                "fps": 2.0,}]
+        }
+    ]
+
+video_text = processor.apply_chat_template(
+            vid_message, tokenize=False, add_generation_prompt=True
+        ) + tokenizer.eos_token
+
+image_input, video_input = process_vision_info(vid_message)
+
+video_input = processor(
+        text=[video_text],
+        images=image_input,
+        videos=video_input,
+        padding=True,
+        return_tensors="pt",
+    ).to("cuda")
+
+with torch.inference_mode():
+    video_emb = model(
+        **video_input, output_hidden_states=True, return_dict=True
+    ).hidden_states[-1][:, -1, :]
+
+video_emb = F.normalize(video_emb, dim=-1)
+
+```
+
+
+<br>
 
 ---
 license: cc-by-nc-4.0
----
+---