Papers
arxiv:2608.30725

Where Do Multilingual Vision-Language Encoders Fail on Low-Resource Languages?

Published on Aug 31
Authors:
,
,
,
,

Abstract

In multilingual vision-language encoders, the low-resource retrieval gap is not caused by a linear language direction at the output but by language-specific divergence of the EOS hidden state with depth, and correcting this divergence improves low-resource retrieval.

Recent multilingual vision--language encoders cover hundreds of languages in a single model, yet on two state-of-the-art instances retrieval on low-resource languages (LRL; e.g. Swahili) trails high-resource ones (HRL; e.g. English) by 30^+\,pp. We ask where in the trained encoder this gap is located. Prior modality-gap and cross-lingual subspace work suggests a linear language direction at the output crowds out alignment-relevant geometry. We falsify this: LEACE drives the linear language classifier from >99% to near chance and iterated INLP to 37--50% while LRL retrieval moves within pm 1.5\,pp and all tier means within 2.2\,pp, tracking random controls. The linear bias is a symptom, not the cause. Instead, the alignment-causal factor lies along the encoder's forward path: the EOS (end-of-sequence) hidden state's per-language trajectory diverges with depth. Substituting the EOS with its parallel English value three blocks before the projector lifts Swahili from 22.1% to 69.1% on one encoder (and reproduces on the other); three controls rule out pooled-position tautology and English specificity. A front-layer trunk that pulls each language's projection toward the parallel-content centroid corroborates the diagnosis at training time, recovering +9.6 / +17.1\,pp on LRL XM3600 retrieval (1{,}000-image subset), with consistent gains across three further benchmarks while preserving HRL performance.

Community

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2608.30725
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2608.30725 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2608.30725 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2608.30725 in a Space README.md to link it from this page.

Collections including this paper 1