README.md

---
license: apache-2.0
tags:
- embedding
- multimodal
- vision-language
- feature-extraction
- mmeb
- mveb
base_model: Qwen/Qwen2.5-VL-7B-Instruct
library_name: transformers
---

# VisME (Qwen2.5-VL-7B)

English | [简体中文](README-zh.md)

**VisME** is a universal multimodal embedding model from our CVPR 2026 paper [*Illuminating Visual Identity in Universal Multimodal Embeddings*](https://openaccess.thecvf.com/content/CVPR2026/html/Cao_Illuminating_Visual_Identity_in_Universal_Multimodal_Embeddings_CVPR_2026_paper.html). Built on [Qwen2.5-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct), VisME produces dense embeddings for images, text, and image–text pairs, with strong performance on both general multimodal benchmarks (**MMEB**) and identity-centric retrieval (**MVEB**).

<p align="center">
  <img src="assets/teaser.jpg" alt="VisME teaser" width="95%">
</p>

## Highlights

- **Identity-aware retrieval** — trained with visual-identity supervision via the [MVEB](https://chrisclear3.github.io/MVEB/) benchmark (4 meta-tasks, 28 test subsets).
- **Strong general capability** — jointly trained on **MMEB-v1** and **MVEB**, maintaining competitive performance on standard MMEB tasks.

## Performance

Results on **MMEB** (Cls / VQA / Ret / Grd) and **MVEB** (ID-Rec / Re-ID / ID-Grd / ID-Edit). Metrics are reported as in the paper (average Precision@1 across datasets within each task group).

<p align="center">
  <img src="assets/performance.jpg" alt="VisME performance on MMEB and MVEB" width="95%">
</p>

## Training Data

VisME is trained on a mixture of:

| Benchmark | Role |
|-----------|------|
| [**MMEB-v1**](https://github.com/TIGER-AI-Lab/VLM2Vec) | General multimodal embedding tasks (classification, VQA, retrieval, grounding) |
| [**MVEB**](https://chrisclear3.github.io/MVEB/) | Visual-identity retrieval across recognition, re-ID, grounding, and editing |

The **MVEB** datasets are now open-sourced. Training code will be released soon.

## Usage

```python
import torch
import torch.nn.functional as F
from io import BytesIO
from urllib.request import Request, urlopen
from PIL import Image
from visme import VisME

model = VisME("path/to/this/repo")
model = model.cuda().eval()

instruction = "Represent the face with the following text."
text = "Retrieve all images with the same cartoon character."

samples = {
    "SpiderMan_comic_E616": "https://static.wikia.nocookie.net/spiderman/images/a/ad/Peter_Parker_%28Earth-616%29_017.png/revision/latest?cb=20210807043502",
    "SpiderMan_promo_E199999": "https://static.wikia.nocookie.net/marveldatabase/images/2/28/Peter_Parker_%28Earth-199999%29_from_Spider-Man_No_Way_Home_promotional_art_002.jpg/revision/latest/scale-to-width-down/1000?cb=20230730084204",
    "Toxin": "https://static.wikia.nocookie.net/superheroes/images/5/53/Toxin.jpg/revision/latest?cb=20240813171932",
}

def load_image(url: str) -> Image.Image:
    req = Request(url, headers={"User-Agent": "Mozilla/5.0"})
    with urlopen(req, timeout=30) as resp:
        return Image.open(BytesIO(resp.read())).convert("RGB")

names = list(samples.keys())
batch = [
    {"image": load_image(samples[name]), "text": text, "instruction": instruction}
    for name in names
]

with torch.no_grad():
    embeddings = model.encode_input(batch)  # shape: (3, 3584), L2-normalized

# Pairwise cosine similarity
# SpiderMan_comic_E616  <-> SpiderMan_promo_E199999
# SpiderMan_comic_E616  <-> Toxin
# SpiderMan_promo_E199999 <-> Toxin
for i in range(len(names)):
    for j in range(i + 1, len(names)):
        sim = F.cosine_similarity(embeddings[i : i + 1], embeddings[j : j + 1]).item()
        print(f"cosine_similarity({names[i]}, {names[j]}) = {sim:.4f}")
```

## Evaluation

Benchmark evaluation code for MMEB and MVEB is **coming soon**. In the meantime, please refer to the paper for full experimental settings and baselines.

## Citation

If you find VisME or MVEB useful, please cite:

```bibtex
@inproceedings{cao2026illuminating,
  title={Illuminating Visual Identity in Universal Multimodal Embeddings},
  author={Cao, Jiawei and Feng, Junyi and Hua, Jiashen and Huang, Ziheng and Deng, Bing and Wu, Kaijie and Gu, Chaochen and Ye, Jieping},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={8737--8748},
  year={2026}
}
```

## License

This model is built upon [Qwen2.5-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct). Please follow the license terms of the base model and the respective datasets used during training.