IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models

Paper · Project page · Hugging Face · ModelScope

Overview

IoU-PD is a training method for coordinate-generating multimodal large language models. It uses ground-truth boxes as coordinate targets and as privileged visual guidance for a frozen teacher. The student receives the original image and referring-expression prompt. The teacher receives the image with the target box marked and a short location hint.

The student is trained with a supervised fine-tuning anchor and privileged distillation. Token weights account for geometric importance and teacher reliability through sample IoU, coordinate error, digit significance, teacher-student agreement, and teacher confidence. Inference uses only the student with the original image and prompt.

This model card is shared between the Hugging Face and ModelScope releases.

Model details

Property	Description
Task	Referring-expression visual grounding
Input	An original image and a referring expression
Output	A bounding box expressed as four coordinates
Architecture	Qwen3-VL
Model class	Qwen3VLForConditionalGeneration
Weight format	Safetensors, bfloat16, two shards
Saved Transformers version	4.57.1
Main paper setting	Qwen3-VL-4B; 300k grounding examples; 3 training epochs
Coordinate order	[x_min, y_min, x_max, y_max]
Coordinate convention	Normalized 1,000-scale coordinates
Inference	Student only; no box overlay, privileged hint, or teacher branch

Download

Choose either hosting platform. The local checkpoint can be used with the same inference interface.

Hugging Face:

from huggingface_hub import snapshot_download

model_dir = snapshot_download(
    repo_id="xyzzzh/IoU-PD",
    local_dir="./IoU-PD",
)

ModelScope:

from modelscope import snapshot_download

model_dir = snapshot_download(
    "xyzzzh/IoU-PD",
    local_dir="./IoU-PD",
)

Inference

Install PyTorch and a Transformers version supporting Qwen3-VL, together with accelerate and pillow. Load both the model and processor from the released checkpoint.

The grounding prompt used in the paper is:

Please provide the bounding box coordinate of the region this sentence describes: <expr>.

Replace <expr> with the referring expression. Supply the original image. The box overlay and location hint belong to the training teacher and are not inference inputs.

Parse the response as [x_min, y_min, x_max, y_max] in the normalized coordinate convention. Apply the same coordinate normalization and box canonicalization when comparing predictions with evaluation annotations.

The following example uses the local checkpoint downloaded above and an image saved as image.jpg. A CUDA GPU with enough memory for the model and image tokens is recommended.

import torch
from PIL import Image
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

model_dir = "./IoU-PD"
processor = AutoProcessor.from_pretrained(model_dir)
model = Qwen3VLForConditionalGeneration.from_pretrained(
    model_dir,
    torch_dtype="auto",
    device_map="auto",
).eval()

image = Image.open("image.jpg").convert("RGB")
expression = "the person on the left"
messages = [{
    "role": "user",
    "content": [
        {"type": "image"},
        {
            "type": "text",
            "text": (
                "Please provide the bounding box coordinate of the region "
                f"this sentence describes: {expression}."
            ),
        },
    ],
}]
prompt = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
inputs = processor(
    text=[prompt], images=[image], padding=True, return_tensors="pt"
).to(model.device)

with torch.inference_mode():
    generated = model.generate(
        **inputs, max_new_tokens=128, do_sample=False, use_cache=True
    )
response_ids = generated[:, inputs.input_ids.shape[1]:]
response = processor.batch_decode(
    response_ids,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False,
)[0]
print(response)

Results reported in the paper

The following results compare the Qwen3-VL-4B base model with the main IoU-PD setting. Overall metrics pool all examples from the five held-out splits. Values are percentages.

Evaluation split	Base mIoU	IoU-PD mIoU	Base Acc@0.5	IoU-PD Acc@0.5	Base Acc@0.7	IoU-PD Acc@0.7
Overall	81.74	85.78	88.58	91.56	82.51	86.76
RefCOCO testA	85.90	88.45	93.25	95.19	88.56	91.44
RefCOCO testB	81.45	84.20	88.85	90.95	81.33	84.14
RefCOCOg test	81.85	87.23	88.31	91.45	82.18	87.34
RefCOCO+ testA	83.79	87.14	90.85	93.59	85.98	89.91
RefCOCO+ testB	74.64	79.88	80.73	85.80	73.33	79.28

The paper also reports improvements across object sizes and IoU thresholds from 0.5 to 0.95, with component and scaling ablations.

Intended use

The model supports visual grounding research and applications that locate image regions from natural language. Predictions should be evaluated with a consistent coordinate parser and normalization rule. The model can make localization errors and should be validated on the intended image and language distribution.

Use of the checkpoint is subject to the applicable base-model and dataset terms. Public availability does not replace those terms.

Citation

@article{zhu2026ioupd,
  title   = {IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models},
  author  = {Zhu, Xiuyuan and Lu, Ke and Wu, Hao and Jiao, Siwen and Du, Zijin and Zhang, Dongming and Xue, Jian},
  journal = {arXiv preprint arXiv:2607.15732},
  year    = {2026}
}

Contact

For questions about the paper or checkpoint: zhuxiuyuan22@mails.ucas.edu.cn.