Papers
arxiv:2611.21644

MInfoVisQA: A Multilingual Benchmark for Information-Rich Visual Question Answering

Published on Oct 14
· Submitted by
Changjiang Jiang
on Nov 18
Authors:
,
,
,
,
,

Abstract

MInfoVisQA, a 24-language parallel benchmark, reveals substantial cross-language robustness gaps in multimodal large language models for information-rich visual question answering.

Multimodal large language models (MLLMs) excel in vision-language tasks; yet, whether they maintain consistent understanding of information-rich visuals (InfoVis) when the surface language changes remains unclear. To quantify this capability, we introduce MInfoVisQA, a parallel benchmark with 8,960 QA configurations derived from 128 seed instances across 24 languages. We construct seed-matched, language-parallel visuals and QA pairs by disentangling numerical-structural data from text and programmatically synchronizing translated labels. We evaluate 25 MLLMs under two settings: Language-Aligned QA (LQA, matching visual-text languages) and Cross-Language QA (XQA, fixed QA language with varying visual languages). The results show that aggregate accuracy masks substantial robustness gaps: two models with comparable mean LQA accuracy (80.4\%) exhibit performance spreads of 7.8 and 20.3 percentage points across languages. Moreover, all 25 models score lower with non-English visuals under fixed English QA, underscoring the need for cross-language sensitivity metrics.

Community

Paper submitter

We built MInfoVisQA to test this across 24 languages. We render the same charts and tables in each language, preserving their numbers and structure. This lets us ask the same question in English about each version and see what happens when only the language in the image changes. Even with the question and answer language fixed to English, we still see accuracy gaps across languages.

We’d love to hear what you find when you try it with your own models! 🤗

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.21644
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.21644 in a model README.md to link it from this page.

Datasets citing this paper 1

Spaces citing this paper 1

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.