File size: 10,642 Bytes

---

license: cc-by-nc-4.0
datasets:
- TIGER-Lab/MMEB-train
- Sony/SCaR-Train
language:
- en
metrics:
- accuracy
base_model:
- Qwen/Qwen2-VL-7B-Instruct
library_name: transformers
tags:
- Embedding
---


# VIRTUE Model Card

[VIRTUE-2B-SCaR](https://huggingface.co/Sony/VIRTUE-2B-SCaR) and [VIRTUE-7B-SCaR](https://huggingface.co/Sony/VIRTUE-7B-SCaR) are the official checkpoints for the paper "[VIRTUE: Visual-Interactive Text-Image Universal Embedder](https://arxiv.org/abs/2510.00523)" that are trained with MMEB-Train and SCaR-Train.
VIRTUE is a visual-interactive text-image universal embedder consisting of a VLM as well as a segmentation model to enable the visual interaction modality for human interactions.
In addition, we introduce the SCaR benchmark ([train](https://huggingface.co/datasets/Sony/SCaR-Train), [eval](https://huggingface.co/datasets/Sony/SCaR-Eval)), composed of 1M samples for visual-interactive image-to-text retrieval, to evaluate visual-interactive embedding capabilities.
SCaR enables evaluation of advanced reasoning and compositional tasks in multimodal, visual-interaction-aware embedding scenarios that remain unexplored.

## Model Checkpoints

- [VIRTUE-2B-SCaR](https://huggingface.co/Sony/VIRTUE-2B-SCaR)
- [VIRTUE-7B-SCaR](https://huggingface.co/Sony/VIRTUE-7B-SCaR)

## SCaR Dataset

- [SCaR-Train](https://huggingface.co/datasets/Sony/SCaR-Train)
- [SCaR-Eval](https://huggingface.co/datasets/Sony/SCaR-Eval)

## Experimental Results

### MMEB
- Without SCaR-Train: 

  ![MMEB Results](images/MMEB-results.png)
- With SCaR-Train

  ![MMEB Results with SCaR-Train](images/MMEB-results-with-SCaR.png)


### SCaR
![SCaR Results](images/SCaR-results.png)

## Resources
- [Paper](https://arxiv.org/abs/2510.00523)
- [Webpage](https://sony.github.io/virtue/)
- [Repository](https://github.com/sony/virtue)

## How to Use
```=python

import os

import sys

import torch

import numpy as np

import json

import hydra

from hydra.core.global_hydra import GlobalHydra

from PIL import Image



# Add parent directory to path for src imports

sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))



from src.arguments import ModelArguments, DataArguments, TrainingArguments

from src.model.model import MMEBModel

from src.model.processor import load_processor, VLM_IMAGE_TOKENS, get_backbone_name, process_vlm_inputs_fns

from transformers import AutoConfig





# Initialize Hydra for SAM2 loading

if not GlobalHydra().is_initialized():

    hydra.initialize(config_path="./configs", version_base=None)



# Determinism

torch.manual_seed(42)

torch.cuda.manual_seed_all(42)

torch.backends.cudnn.deterministic = True

torch.backends.cudnn.benchmark = False

np.random.seed(42)



model_dir = 'Sony/VIRTUE-2B-SCaR'

device = 'cuda' if torch.cuda.is_available() else 'cpu'



config = AutoConfig.from_pretrained(model_dir, trust_remote_code=True, token=True)



# Build arguments directly (no YAML required)

model_args = ModelArguments(

    model_name=model_dir,

    checkpoint_path=None,

    pooling="last",

    normalize=True,

    lora=False,

    model_backbone='qwen2_vl',

)

persisted_sam = config.virtue_sam



model_args.sam = True

model_args.sam_config = {

    "config_path": persisted_sam.get('config_path') if persisted_sam else None,

    "checkpoint": persisted_sam.get('checkpoint') if persisted_sam else None,

    "points_per_side": (persisted_sam.get('points_per_side') if persisted_sam else 16),

    "feature_levels": (persisted_sam.get('feature_levels') if persisted_sam else 3),

}



data_args = DataArguments()

training_args = TrainingArguments()



processor = load_processor(model_args, data_args)

model = MMEBModel.load(model_args, is_trainable=False, processor=processor)

model.eval()

model = model.to(device, dtype=torch.bfloat16)



# Get model backbone and image token

model_backbone = get_backbone_name(hf_config=config)

image_token = VLM_IMAGE_TOKENS[model_backbone]



# Image + Text -> Text

image_path = '../assets/example.jpg'

image = Image.open(image_path).convert('RGB')



model_inputs = {

    'text': [f"{image_token}\nRepresent the given image with the following question: What is in the image"],

    'images': [image]

}



process_fn = process_vlm_inputs_fns[model_backbone]

inputs = process_fn(model_inputs, processor=processor, max_length=512)

device = next(model.parameters()).device

inputs = {k: v.to(device) if torch.is_tensor(v) else v for k, v in inputs.items()}



with torch.no_grad():

    with torch.autocast(enabled=True, dtype=torch.bfloat16, device_type="cuda"):

        qry_output = model(qry=inputs)["qry_reps"]



# Candidates for all scenarios

test_strings = ['A cat', 'A dog', 'A tiger']



# Scenario 1: No visual prompts (image only)

print("\n--- Similarities (no visual prompts) ---")

for string in test_strings:

    cand_inputs = process_fn({'text': [string], 'images': [None]}, processor=processor)

    cand_inputs = {k: v.to(device) if torch.is_tensor(v) else v for k, v in cand_inputs.items()}

    with torch.no_grad():

        with torch.autocast(enabled=True, dtype=torch.bfloat16, device_type="cuda"):

            tgt_output = model(tgt=cand_inputs)["tgt_reps"]

    sim = model.compute_similarity(qry_output, tgt_output)

    print(f"no-prompt | {string} = {sim}")



'''

--- Similarities (no visual prompts) ---

no-prompt | A cat = tensor([[0.3030]], device='cuda:0')

no-prompt | A dog = tensor([[0.2453]], device='cuda:0')

no-prompt | A tiger = tensor([[0.1714]], device='cuda:0')

'''



# Scenario 2: Point prompts — two examples (left/right)

print("\n--- Similarities (point prompts) ---")

sam_size = 1024  # SAM2Transforms output size

point_examples = [(0.25, 0.5), (0.75, 0.5)]

for (px, py) in point_examples:

    point_text = f"{image_token}\nFind the caption that best describes the segmented object, considering both local details and global context in the given image.\nReferring object point: ({int(px*image.size[0])}, {int(py*image.size[1])})"

    q_inputs = process_fn({'text': [point_text], 'images': [image]}, processor=processor)

    q_inputs['point'] = [px * sam_size, py * sam_size]

    q_inputs = {k: v.to(device) if torch.is_tensor(v) else v for k, v in q_inputs.items()}

    with torch.no_grad():

        with torch.autocast(enabled=True, dtype=torch.bfloat16, device_type="cuda"):

            point_qry = model(qry=q_inputs)["qry_reps"]

    for string in test_strings:

        cand_inputs = process_fn({'text': [string], 'images': [None]}, processor=processor)

        cand_inputs = {k: v.to(device) if torch.is_tensor(v) else v for k, v in cand_inputs.items()}

        with torch.no_grad():

            with torch.autocast(enabled=True, dtype=torch.bfloat16, device_type="cuda"):

                tgt_output = model(tgt=cand_inputs)["tgt_reps"]

        sim = model.compute_similarity(point_qry, tgt_output)

        print(f"point ({px:.2f},{py:.2f}) | {string} = {sim}")



'''

--- Similarities (point prompts) ---

point (0.25,0.50) | A cat = tensor([[0.1793]], device='cuda:0')

point (0.25,0.50) | A dog = tensor([[0.1339]], device='cuda:0')

point (0.25,0.50) | A tiger = tensor([[0.1314]], device='cuda:0')

point (0.75,0.50) | A cat = tensor([[0.2232]], device='cuda:0')

point (0.75,0.50) | A dog = tensor([[0.1742]], device='cuda:0')

point (0.75,0.50) | A tiger = tensor([[0.1692]], device='cuda:0')

'''



# Scenario 3: BBox prompts — two examples (left/right)

print("\n--- Similarities (bbox prompts) ---")

bbox_examples = [

    (0.05, 0.20, 0.45, 0.80),  # left

    (0.55, 0.20, 0.95, 0.80),  # right

]

for (x1, y1, x2, y2) in bbox_examples:

    bbox_text = f"{image_token}\nFind the caption that best describes the object in the bounding box, considering both local details and global context in the given image.\nReferring object bbox: ({int(x1*image.size[0])}, {int(y1*image.size[1])}, {int(x2*image.size[0])}, {int(y2*image.size[1])})"

    q_inputs = process_fn({'text': [bbox_text], 'images': [image]}, processor=processor)

    q_inputs['bbox'] = [x1 * sam_size, y1 * sam_size, x2 * sam_size, y2 * sam_size]

    q_inputs = {k: v.to(device) if torch.is_tensor(v) else v for k, v in q_inputs.items()}

    with torch.no_grad():

        with torch.autocast(enabled=True, dtype=torch.bfloat16, device_type="cuda"):

            bbox_qry = model(qry=q_inputs)["qry_reps"]

    for string in test_strings:

        cand_inputs = process_fn({'text': [string], 'images': [None]}, processor=processor)

        cand_inputs = {k: v.to(device) if torch.is_tensor(v) else v for k, v in cand_inputs.items()}

        with torch.no_grad():

            with torch.autocast(enabled=True, dtype=torch.bfloat16, device_type="cuda"):

                tgt_output = model(tgt=cand_inputs)["tgt_reps"]

        sim = model.compute_similarity(bbox_qry, tgt_output)

        print(f"bbox ({x1:.2f},{y1:.2f},{x2:.2f},{y2:.2f}) | {string} = {sim}")



'''

--- Similarities (bbox prompts) ---

bbox (0.05,0.20,0.45,0.80) | A cat = tensor([[0.2100]], device='cuda:0')

bbox (0.05,0.20,0.45,0.80) | A dog = tensor([[0.1512]], device='cuda:0')

bbox (0.05,0.20,0.45,0.80) | A tiger = tensor([[0.1719]], device='cuda:0')

bbox (0.55,0.20,0.95,0.80) | A cat = tensor([[0.1583]], device='cuda:0')

bbox (0.55,0.20,0.95,0.80) | A dog = tensor([[0.1953]], device='cuda:0')

bbox (0.55,0.20,0.95,0.80) | A tiger = tensor([[0.1225]], device='cuda:0')

'''

```

## Ethical Considerations
_Note: This section is mainly taken from the [AKI](https://huggingface.co/Sony/AKI-4B-phi-3.5-mini) models_.

This release is for research purposes only in support of an academic paper. Our models, datasets, and code are not specifically designed or evaluated for all downstream purposes. We strongly recommend users evaluate and address potential concerns related to accuracy, safety, and fairness before deploying this model. We encourage users to consider the common limitations of AI, comply with applicable laws, and leverage best practices when selecting use cases, particularly for high-risk scenarios where errors or misuse could significantly impact people’s lives, rights, or safety.


## Citation
```

@article{wangICLR2026virtue,

  author       = {Wei-Yao Wang and

                  Kazuya Tateishi and

                  Qiyu Wu and

                  Shusuke Takahashi and

                  Yuki Mitsufuji},

  title        = {VIRTUE: Visual-Interactive Text-Image Universal Embedder},

  journal      = {arXiv preprint arXiv:2510.00523},

  year         = {2025}

}

```