SigLIP-2 SO400M — trunk-calibrated

Research artifact for "Where Do Multilingual Vision-Language Encoders Fail on Low-Resource Languages?"

This is google/siglip2-so400m-patch16-256 with its front 3 text-encoder blocks replaced by trunk-calibrated weights. Everything else — the vision tower, the remaining 24 text blocks, and the projection head — is byte-identical to the frozen public model.

What this changes

Multilingual VL encoders trail English by 30+ pp on low-resource-language retrieval. The paper localises that gap: it is not a linear language direction at the output (erasing it with LEACE or INLP leaves retrieval within ~1.5 pp), but the pooled EOS hidden state's per-language trajectory drift with depth. Calibrating only the first 3 blocks — pulling each language's projection toward the parallel-content centroid — closes much of the gap without touching the frozen back-half.

Measured results

XM3600 (Crossmodal-3600), pinned 1,000-image subset, image-to-text R@1, bfloat16. frozen is the unmodified base model measured in the same harness.

Language	Tier	Frozen	This model	Δ
English (`en`)	HRL	66.1	70.4	+4.3
French (`fr`)	HRL	78.2	82.6	+4.4
German (`de`)	HRL	33.7	43.8	+10.1
Spanish (`es`)	HRL	64.1	67.4	+3.3
Chinese (`zh`)	HRL	63.8	70.5	+6.7
Korean (`ko`)	HRL	74.0	77.0	+3.0
Bengali (`bn`)	LRL	30.2	42.7	+12.5
Filipino (`fil`)	LRL	26.9	44.9	+18.0
Hindi (`hi`)	LRL	33.5	38.4	+4.9
Swahili (`sw`)	LRL	12.4	35.7	+23.3
Telugu (`te`)	LRL	5.2	31.7	+26.5
HRL-6 mean		63.32	68.62	+5.30
LRL-5 mean		21.64	38.68	+17.04

High-resource performance is preserved; the gain concentrates on low-resource languages.

Usage

import torch
from transformers import AutoModel, AutoProcessor

model = AutoModel.from_pretrained("dnotitia/geometric-bottleneck-siglip2", dtype=torch.bfloat16).cuda().eval()
processor = AutoProcessor.from_pretrained("dnotitia/geometric-bottleneck-siglip2")

texts = ["a dog on a beach", "mbwa ufukweni"]          # English, Swahili
inputs = processor(text=texts, padding="max_length", truncation=True,
                   max_length=64, return_tensors="pt").to("cuda")
with torch.no_grad():
    z = torch.nn.functional.normalize(model.get_text_features(**inputs).float(), dim=-1)

Image encoding is unchanged from the base model — the vision tower was never trained.

Training


Calibrated blocks	front 3 of 27 text blocks
Text tower	bidirectional, last token (sticky EOS) pooling, max 64 tokens
Objective	InfoNCE + λ·(1 − cosine to parallel-content centroid)
λ_align	1.0
Learning rate	0.00024
Temperature	0.02
Steps	2000 of 2000 (warmup 200)
Weight decay	0.05
Languages	11: en, fr, de, es, zh, ko, bn, fil, hi, sw, te
Training data	CC12M caption subset, machine-translated into the 11-language pool
Precision	bfloat16

Limitations

Language-specific. Calibration is trained on these 11 languages. It does not transfer to unseen languages, and can regress on them.
Visual-vocabulary scoped. Gains are measured on image captions. Abstract prose benefits far less.
Does not reach parity. The LRL tier improves substantially but does not match HRL.
Single run. No seed variance is reported for this checkpoint.

License

These weights contain the base model's parameters, so the base model's license governs use.


Base model	`google/siglip2-so400m-patch16-256`
Pinned revision	`e8708ab72d125807e45b36fb7d4e0aacbb59f379`
License	apache-2.0

The calibrated front-3 blocks are released as a research artifact accompanying the paper.

Citation

@inproceedings{han2026geometric,
  title     = {Where Do Multilingual Vision-Language Encoders Fail on Low-Resource Languages?},
  author    = {Han, Donghoon and Moon, SungHyun and Zhakatayev, Aidyn and Cha, Junghun and Lee, SeungJae},
  booktitle = {Proceedings of EMNLP 2026},
  year      = {2026}
}

Code: https://github.com/dnotitia/geometric-bottleneck