mSigLIP base — trunk-calibrated

Research artifact for "Where Do Multilingual Vision-Language Encoders Fail on Low-Resource Languages?"

This is google/siglip-base-patch16-256-multilingual with its front 4 text-encoder blocks replaced by trunk-calibrated weights. Everything else — the vision tower, the remaining 8 text blocks, and the projection head — is byte-identical to the frozen public model.

What this changes

Multilingual VL encoders trail English by 30+ pp on low-resource-language retrieval. The paper localises that gap: it is not a linear language direction at the output (erasing it with LEACE or INLP leaves retrieval within ~1.5 pp), but the pooled EOS hidden state's per-language trajectory drift with depth. Calibrating only the first 4 blocks — pulling each language's projection toward the parallel-content centroid — closes much of the gap without touching the frozen back-half.

Measured results

XM3600 (Crossmodal-3600), pinned 1,000-image subset, image-to-text R@1, bfloat16. frozen is the unmodified base model measured in the same harness.

Language	Tier	Frozen	This model	Δ
English (`en`)	HRL	68.3	69.0	+0.7
French (`fr`)	HRL	81.0	82.5	+1.5
German (`de`)	HRL	86.2	86.3	+0.1
Spanish (`es`)	HRL	75.6	76.6	+1.0
Chinese (`zh`)	HRL	58.5	70.1	+11.6
Korean (`ko`)	HRL	71.0	73.2	+2.2
Bengali (`bn`)	LRL	45.5	58.3	+12.8
Filipino (`fil`)	LRL	40.7	54.5	+13.8
Hindi (`hi`)	LRL	37.1	46.5	+9.4
Swahili (`sw`)	LRL	25.4	50.5	+25.1
Telugu (`te`)	LRL	11.9	41.1	+29.2
HRL-6 mean		73.43	76.28	+2.85
LRL-5 mean		32.12	50.18	+18.06

High-resource performance is preserved; the gain concentrates on low-resource languages.

Usage

import torch
from transformers import AutoModel, AutoProcessor

model = AutoModel.from_pretrained("dnotitia/geometric-bottleneck-msiglip", dtype=torch.bfloat16).cuda().eval()
processor = AutoProcessor.from_pretrained("dnotitia/geometric-bottleneck-msiglip")

texts = ["a dog on a beach", "mbwa ufukweni"]          # English, Swahili
inputs = processor(text=texts, padding="max_length", truncation=True,
                   max_length=64, return_tensors="pt").to("cuda")
with torch.no_grad():
    z = torch.nn.functional.normalize(model.get_text_features(**inputs).float(), dim=-1)

Image encoding is unchanged from the base model — the vision tower was never trained.

Training


Calibrated blocks	front 4 of 12 text blocks
Text tower	bidirectional, last token (sticky EOS) pooling, max 64 tokens
Objective	InfoNCE + λ·(1 − cosine to parallel-content centroid)
λ_align	2.0
Learning rate	0.00012
Temperature	0.02
Steps	12001 of 20000 (warmup 500)
Weight decay	0.05
Languages	11: en, fr, de, es, zh, ko, bn, fil, hi, sw, te
Training data	CC12M caption subset, machine-translated into the 11-language pool
Precision	bfloat16

Limitations

Language-specific. Calibration is trained on these 11 languages. It does not transfer to unseen languages, and can regress on them.
Visual-vocabulary scoped. Gains are measured on image captions. Abstract prose benefits far less.
Does not reach parity. The LRL tier improves substantially but does not match HRL.
Single run. No seed variance is reported for this checkpoint.

License

These weights contain the base model's parameters, so the base model's license governs use.


Base model	`google/siglip-base-patch16-256-multilingual`
Pinned revision	`8952a4eafcde3cb7ab46b1dd629b33f8784ca9c6`
License	apache-2.0

The calibrated front-4 blocks are released as a research artifact accompanying the paper.

Citation

@inproceedings{han2026geometric,
  title     = {Where Do Multilingual Vision-Language Encoders Fail on Low-Resource Languages?},
  author    = {Han, Donghoon and Moon, SungHyun and Zhakatayev, Aidyn and Cha, Junghun and Lee, SeungJae},
  booktitle = {Proceedings of EMNLP 2026},
  year      = {2026}
}

Code: https://github.com/dnotitia/geometric-bottleneck