README.md

---
license: apache-2.0
base_model: Qwen/Qwen3-VL-8B-Instruct
library_name: transformers
pipeline_tag: image-text-to-text
language:
  - en
tags:
  - video
  - multi-image
  - video-captioning
  - reference-grounding
  - qwen3-vl
  - grpo
---

# RefCaptioner

RefCaptioner is an 8B vision-language model for **multi-reference image-grounded video captioning**. Given a video and an ordered set of reference images, it writes a fluent English description and places tags such as `<Image_1>` directly after the visual phrases grounded by those references. Reference images may include distractors; the model is expected to omit tags that cannot be grounded in the video.

The released weights are a merged Hugging Face checkpoint based on [Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct). No PEFT adapter is required at inference time.

## Task overview

![Multi-reference image-grounded video captioning](teaser.png)

RefCaptioner grounds relevant reference images to local phrases in a video caption while rejecting distractor references.

## Model details

| Item | Value |
|---|---|
| Base model | `Qwen/Qwen3-VL-8B-Instruct` |
| Release checkpoint | `RefCaptioner1.0` |
| Input | Ordered reference images plus one video |
| Output | English caption with local `<Image_n>` bindings |
| Thinking mode | Disabled |
| Weight format | Merged `safetensors` |

## Method overview

![RefCaptioner training framework](framework.png)
Training Framework

## Installation

The tested environment for loading RefCaptioner and running inference is provided in `requirement.txt`. Use Python 3.10 and install it with:

```bash
pip install -r requirement.txt
```



## Quick start

```bash
python inference.py \
  --model TengfeiLiuCoder/RefCaptioner \
  --video /path/to/video.mp4 \
  --images /path/to/ref_1.jpg /path/to/ref_2.jpg /path/to/ref_3.png
```

The image order defines the tag mapping: the first image is `<Image_1>`, the second is `<Image_2>`, and so on. The script uses the exact prompt structure and deterministic decoding configuration used for the reported test-set inference.

## Reproduction configuration

The released benchmark captions were generated with the following settings:

| Parameter | Value |
|---|---:|
| Prompt protocol | `Prompt_1.0` |
| Maximum input length | 18,000 tokens |
| Maximum new tokens | 512 |
| Video sampling rate | 2 FPS |
| Minimum video frames | 4 |
| Reference-image maximum pixels | 602,112 |
| Video-frame maximum pixels | 602,112 |
| Decoding | Greedy (`do_sample=False`) |
| Temperature | 0.0 |
| Top-p / top-k | 1.0 / 0 |
| Inference dtype | `bfloat16` |
| Thinking | Disabled |

The same values are machine-readable in `inference_config.json`. The eight GPUs used for the full benchmark only provided sample-level parallelism; eight GPUs are not a model requirement. A single device can run the model when it has enough memory, while `device_map="auto"` can distribute the model across available devices.

## Input and output contract

The prompt labels each ordered reference before presenting the associated image, then presents the video. The model must:

1. describe visible video content in one English paragraph;
2. place every used `<Image_n>` immediately after a concrete grounded phrase;
3. group tags only when they ground the same visual unit; and
4. omit distractor or uncertain references rather than forcing all tags into the caption.

Example output:

![RefCaptioner example output](captioncase_01.png)

## MRVBench Results

The following scores report RefCaptioner on MRVBench. `Eff-Bind = Ref-Tag-R × Ref-Bind`.

| Model | KP-Cov ↑ | VQA ↑ | VQA-Cov ↑ | Ref-Tag-P ↑ | Ref-Tag-R ↑ | Ref-Bind ↑ | Eff-Bind ↑ | Dist-Rej ↑ | FalseRef-Any ↓ | Subj-R ↑ | Subj-F1 ↑ | MRVScore ↑ |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| **RefCaptioner** | 0.882 | 0.686 | 0.837 | 0.994 | 0.943 | 0.967 | 0.912 | 0.985 | 0.100 | 0.817 | 0.869 | 0.888 |

These values are tied to the prompt and preprocessing configuration above; changing frame sampling, pixel limits, prompt wording, or decoding can change the results.

## Intended use

The model is intended for research on multi-reference video understanding, grounded caption generation, reference selection, and local phrase-to-image binding. It is not designed to identify people, infer sensitive attributes, or establish the provenance or ownership of visual content.

## Limitations


- Results depend on frame sampling and may miss short events between sampled frames.
- The output is English and follows a task-specific caption style.
- Reference tags express visual grounding, not identity verification.

Users should verify generated descriptions before applying them in consequential settings.

## License

The model weights are released under the Apache 2.0 license, following the base model. The companion benchmark contains separate media and annotation considerations; consult its dataset card before use or redistribution.

## Citation

```bibtex
@misc{liu2026refcaptionermultireferenceimagegroundedvideo,
      title={RefCaptioner: Multi-Reference Image-Grounded Video Captioning},
      author={Tengfei Liu and Yang Shi and Yuran Wang and Xiaohan Zhang and Yuqing Wen and Yuqi Tang and Qixun Wang and Zhuoran Zhang and Xuanyu Zhu and Weihong Lin and Xinlei Yu and Yujie Wei and Xinwei Long and Fengxiang Wang and Xinlong Chen and Yue Ding and Jialu Chen and Haotian Wang and Yuanxing Zhang},
      year={2026},
      eprint={2607.28509},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.28509}
}
```