RefCaptioner

RefCaptioner is an 8B vision-language model for multi-reference image-grounded video captioning. Given a video and an ordered set of reference images, it writes a fluent English description and places tags such as <Image_1> directly after the visual phrases grounded by those references. Reference images may include distractors; the model is expected to omit tags that cannot be grounded in the video.

The released weights are a merged Hugging Face checkpoint based on Qwen3-VL-8B-Instruct. No PEFT adapter is required at inference time.

Model details

Item	Value
Base model	Qwen/Qwen3-VL-8B-Instruct
Release checkpoint	RefCaptioner1.0
Input	Ordered reference images plus one video
Output	English caption with local <Image_n> bindings
Thinking mode	Disabled
Weight format	Merged safetensors

Installation

pip install -r requirements.txt

RefCaptioner uses the standard Qwen3-VL inference stack and does not require a task-specific runtime. Any environment that can run Qwen3-VL with a compatible Transformers release can be used directly.

The benchmark environment used torch==2.9.0, transformers==4.57.6, accelerate==1.14.0, qwen-vl-utils==0.0.14, and av==17.1.0.

Quick start

python inference.py \
  --model NOVAglow646/RefCaptioner \
  --video /path/to/video.mp4 \
  --images /path/to/ref_1.jpg /path/to/ref_2.jpg /path/to/ref_3.png

The image order defines the tag mapping: the first image is <Image_1>, the second is <Image_2>, and so on. The script uses the exact prompt structure and deterministic decoding configuration used for the reported test-set inference.

Reproduction configuration

The released benchmark captions were generated with the following settings:

Parameter	Value
Prompt protocol	Prompt_1.0
Maximum input length	18,000 tokens
Maximum new tokens	512
Video sampling rate	2 FPS
Minimum video frames	4
Reference-image maximum pixels	602,112
Video-frame maximum pixels	602,112
Decoding	Greedy (do_sample=False)
Temperature	0.0
Top-p / top-k	1.0 / 0
Inference dtype	bfloat16
Thinking	Disabled

The same values are machine-readable in inference_config.json. The eight GPUs used for the full benchmark only provided sample-level parallelism; eight GPUs are not a model requirement. A single device can run the model when it has enough memory, while device_map="auto" can distribute the model across available devices.

Input and output contract

The prompt labels each ordered reference before presenting the associated image, then presents the video. The model must:

1. describe visible video content in one English paragraph;
2. place every used <Image_n> immediately after a concrete grounded phrase;
3. group tags only when they ground the same visual unit; and
4. omit distractor or uncertain references rather than forcing all tags into the caption.

Example output:

MRVBench Results

The following scores report RefCaptioner on MRVBench. Eff-Bind = Ref-Tag-R × Ref-Bind.

Model	KP-Cov ↑	VQA ↑	VQA-Cov ↑	Ref-Tag-P ↑	Ref-Tag-R ↑	Ref-Bind ↑	Eff-Bind ↑	Dist-Rej ↑	FalseRef-Any ↓	Subj-R ↑	Subj-F1 ↑	MRVScore ↑
RefCaptioner	0.882	0.686	0.837	0.994	0.943	0.967	0.912	0.985	0.100	0.817	0.869	0.888

These values are tied to the prompt and preprocessing configuration above; changing frame sampling, pixel limits, prompt wording, or decoding can change the results.

Intended use

The model is intended for research on multi-reference video understanding, grounded caption generation, reference selection, and local phrase-to-image binding. It is not designed to identify people, infer sensitive attributes, or establish the provenance or ownership of visual content.

Limitations

• Results depend on frame sampling and may miss short events between sampled frames.
• The output is English and follows a task-specific caption style.
• Reference tags express visual grounding, not identity verification.

Users should verify generated descriptions before applying them in consequential settings.

License

The model weights are released under the Apache 2.0 license, following the base model. The companion benchmark contains separate media and annotation considerations; consult its dataset card before use or redistribution.

Citation

@misc{liu2026refcaptionermultireferenceimagegroundedvideo,
      title={RefCaptioner: Multi-Reference Image-Grounded Video Captioning},
      author={Tengfei Liu and Yang Shi and Yuran Wang and Xiaohan Zhang and Yuqing Wen and Yuqi Tang and Qixun Wang and Zhuoran Zhang and Xuanyu Zhu and Weihong Lin and Xinlei Yu and Yujie Wei and Xinwei Long and Fengxiang Wang and Xinlong Chen and Yue Ding and Jialu Chen and Haotian Wang and Yuanxing Zhang},
      year={2026},
      eprint={2607.28509},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.28509}
}