AltCLIP — trunk-calibrated

Research artifact for "Where Do Multilingual Vision-Language Encoders Fail on Low-Resource Languages?"

This is BAAI/AltCLIP with its front 5 text-encoder blocks replaced by trunk-calibrated weights. Everything else — the vision tower, the remaining 19 text blocks, and the projection head — is byte-identical to the frozen public model.

What this changes

Multilingual VL encoders trail English by 30+ pp on low-resource-language retrieval. The paper localises that gap: it is not a linear language direction at the output (erasing it with LEACE or INLP leaves retrieval within ~1.5 pp), but the pooled EOS hidden state's per-language trajectory drift with depth. Calibrating only the first 5 blocks — pulling each language's projection toward the parallel-content centroid — closes much of the gap without touching the frozen back-half.

Measured results

XM3600 (Crossmodal-3600), pinned 1,000-image subset, image-to-text R@1, bfloat16. frozen is the unmodified base model measured in the same harness.

Language	Tier	Frozen	This model	Δ
English (`en`)	HRL	63.8	62.4	-1.4
French (`fr`)	HRL	58.9	69.4	+10.5
German (`de`)	HRL	44.5	65.6	+21.1
Spanish (`es`)	HRL	53.4	62.2	+8.8
Chinese (`zh`)	HRL	70.7	70.3	-0.4
Korean (`ko`)	HRL	3.6	57.8	+54.2
Bengali (`bn`)	LRL	1.9	50.1	+48.2
Filipino (`fil`)	LRL	16.2	47.7	+31.5
Hindi (`hi`)	LRL	4.8	39.0	+34.2
Swahili (`sw`)	LRL	6.4	41.3	+34.9
Telugu (`te`)	LRL	3.4	41.3	+37.9
HRL-6 mean		49.15	64.62	+15.47
LRL-5 mean		6.54	43.88	+37.34

High-resource performance is preserved; the gain concentrates on low-resource languages.

Usage

import torch
from transformers import AutoModel, AutoProcessor

model = AutoModel.from_pretrained("dnotitia/geometric-bottleneck-altclip", dtype=torch.bfloat16).cuda().eval()
processor = AutoProcessor.from_pretrained("dnotitia/geometric-bottleneck-altclip")

texts = ["a dog on a beach", "mbwa ufukweni"]          # English, Swahili
inputs = processor(text=texts, padding="max_length", truncation=True,
                   max_length=77, return_tensors="pt").to("cuda")
with torch.no_grad():
    z = torch.nn.functional.normalize(model.get_text_features(**inputs).float(), dim=-1)

Image encoding is unchanged from the base model — the vision tower was never trained.

Training


Calibrated blocks	front 5 of 24 text blocks
Text tower	bidirectional (XLM-R), CLS pooling, max 77 tokens
Objective	InfoNCE + λ·(1 − cosine to parallel-content centroid)
λ_align	2.0
Learning rate	0.00012
Temperature	0.02
Steps	14001 of 20000 (warmup 500)
Weight decay	0.05
Languages	11: en, fr, de, es, zh, ko, bn, fil, hi, sw, te
Training data	CC12M caption subset, machine-translated into the 11-language pool
Precision	bfloat16

Limitations

Language-specific. Calibration is trained on these 11 languages. It does not transfer to unseen languages, and can regress on them.
Visual-vocabulary scoped. Gains are measured on image captions. Abstract prose benefits far less.
Does not reach parity. The LRL tier improves substantially but does not match HRL.
Single run. No seed variance is reported for this checkpoint.

License

These weights contain the base model's parameters, so the base model's license governs use.


Base model	`BAAI/AltCLIP`
Pinned revision	`17788d7d45af0e41d0f10682fe91d655ac96435c`
License	creativeml-openrail-m

The calibrated front-5 blocks are released as a research artifact accompanying the paper.

Citation

@inproceedings{han2026geometric,
  title     = {Where Do Multilingual Vision-Language Encoders Fail on Low-Resource Languages?},
  author    = {Han, Donghoon and Moon, SungHyun and Zhakatayev, Aidyn and Cha, Junghun and Lee, SeungJae},
  booktitle = {Proceedings of EMNLP 2026},
  year      = {2026}
}

Code: https://github.com/dnotitia/geometric-bottleneck