arxiv:2608.02589

CAPEval: A Decoupled Caption Evaluation across Understanding and Generation

Published on Aug 3

· Submitted by

Galaxy on Aug 5

Upvote

Authors:

Zhipeng Liu ,

Abstract

Captions serve as a primary supervision signal for both multimodal understanding and text-to-image generation. However, previous evaluations treat the caption quality as a single scalar objective, which conflates two distinct properties: (1) how much visual information a caption covers and (2) how reliably the image supports its stated claims. To this end, we design a decoupled caption evaluation benchmark, CAPEval (Coverage And Precision Evaluation), with human-written ground-truth captions and human-verified atomic checklist items. Specifically, CAPEval decomposes caption quality into Coverage and Precision. The former quantifies how thoroughly a caption covers ground-truth factual content, while the latter reflects the factual correctness rate of all claims expressed in the caption. We select 10 captioners and further conduct controlled downstream end-to-end experiments with them from four model families, where the caption source is the only variable. Empirically, we find a consistent task-dependent dissociation: Coverage serves as the stronger correlate for understanding performance, whereas Precision acts as the dominant predictor for generation performance. This decoupled evaluation paradigm not only delivers a more fine-grained diagnosis of caption quality, but also offers actionable guidance for selecting and optimizing captioners tailored to different downstream tasks.

View arXiv page View PDF Project page GitHub 9 Add to collection

Community

LiuzhipengUCAS

Paper author Paper submitter about 14 hours ago

Code: https://github.com/liuzhipenggg/CAPEval

Suppose an image contains 10 factual claims: Caption A references 9 of them but misstates 2, while Caption B only mentions 5, all of which are factually correct. Which caption is higher‑quality?

You may find yourself stumped.

This question is difficult to resolve with a single aggregated score, since each has its own merits. Caption A’s strength lies in its comprehensiveness, whereas Caption B excels in factual accuracy.

Over the past several years, a growing body of work has used vision‑language models to‑recaption training images, aiming to improve dataset quality with richer and more accurate captions. The importance of captions in multimodal model training is now widely recognized.

Yet, as the opening example demonstrates, a more fundamental question remains poorly understood: what kinds of captions genuinely benefit downstream model training?

To tackle this problem, we have proposed a new caption evaluation framework: CAPEval (Coverage And Precision Evaluation). Unlike previous benchmarks that evaluate captions in isolation, we further train vision‑language and text‑to‑image models on captions produced by different captioning models. Through controlled‑variable experiments, we examine how different quality attributes of captions ultimately influence downstream model performance.

Our paper identifies a clear task‑dependent dissociation: Coverage serves as the stronger correlate for understanding performance, whereas Precision acts as the dominant predictor for generation performance.