Papers
arxiv:2611.18765

Mutual Colormap Dependence: How Recoloring Biomedical Images Changes Vision-Language Model Answers

Published on Nov 11
Authors:
,

Abstract

Vision-language models (VLMs) are being built to answer questions about biomedical images, yet they never see the measured intensities, only the colors that a colormap has assigned to them. For questions about image content, the colormap should not change the answer. We call the extent to which it does Mutual Colormap Dependence (MCD), the mutual information between the colormap and a model's answer, which is zero exactly when every colormap yields the same answer distribution. Because VLMs sample their answers, we compare recoloring with a control that uses the same number of calls to ask again about the unchanged gray image, so that ordinary resampling is not mistaken for an effect of the colormap. We test GPT-5.4, Gemini 3.1 Pro, Claude Sonnet 4.6, and Qwen3-VL-235B under six luminance-monotonic colormaps (gray, viridis, plasma, inferno, magma, and cividis) on 50 medical visual-question-answering items from 16 SLAKE images, 64 ChestX-ray14 chest radiographs, and 200 BBBC039 fluorescence-microscopy images for exact nucleus counting. Every prompt tells the model to ignore the colormap. Even so, with three answers per colormap on the chest radiographs, the answers of all four models depend on the colormap (all Holm-adjusted p leq 0.002). For Claude, some colored map overturns the majority answer on 29 of the 64 radiographs, where resampling gray alone overturns about 4, and Qwen's accuracy falls from 0.54 under gray to between 0.32 and 0.41 under the colored maps, mostly because it abstains. With one answer per colormap, recoloring adds disagreement beyond repeated sampling in 5 of 12 model-panel cells under per-cell 95% intervals, and repeating the instruction to ignore color does not reliably reduce it. The colormap is therefore part of what a VLM sees, and it can change the answer.

Community

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.18765 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.18765 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.18765 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.