VLM-MCQA: Tuned Lens translators

Trained Tuned Lens translators for four vision-language models, from the project Understanding How VLMs Answer Multiple-Choice Questions in Multimodal Settings. Code: https://github.com/DivyG007/vlm-mcqa

Folder	Base model (revision)	Stages	Hidden size
qwen25vl_7b/	Qwen/Qwen2.5-VL-7B-Instruct (cc59489)	28	3584
internvl35_8b/	OpenGVLab/InternVL3_5-8B-HF (741a7d0)	36	4096
llava_ov_7b_si/	llava-hf/llava-onevision-qwen2-7b-si-hf (cae7503)	28	3584
paligemma2_10b_mix/	google/paligemma2-10b-mix-448 (b26d16f)	42	3584

Files (per model)

• translators/translator_stage_XX.pt: one translator per residual stage (the state entering decoder layer XX). Each file is a dict with state (weight d x d and bias d, float32, for nn.Linear(d, d)) and summary (validation KL/top-1 against the Logit Lens and the training history).
• translators/train_summary.json: training config, per-stage metrics, file SHA-256 hashes and the W&B run.
• eval/: Logit Lens versus Tuned Lens evaluation on 3,072 behaviour variants (comparison_behavior.json, calibration.json, tables/, figures/, per-variant lens_behavior_{logit,tuned}.jsonl).
• calibration/, run.log: run calibration and log.

Using a translator

The translator is applied residually, then decoded with the model's own final norm and unembedding:

import torch
payload = torch.load("qwen25vl_7b/translators/translator_stage_20.pt", map_location="cpu", weights_only=True)
W, b = payload["state"]["weight"], payload["state"]["bias"]
h = hidden_state_entering_layer_20.float()   # [..., d]
h_tuned = h + h @ W.T + b                     # zero W, b reproduces the Logit Lens
logits = lm_head(final_norm(h_tuned))

The project's loader is vlm_mcqa.tuned_lens.translator.load_lens in the code repository.

Training

• Translators are zero-initialized, so an untrained translator is exactly the Logit Lens. The VLM stays frozen.
• Loss: KL(final output distribution || lens distribution) at prompt text-token positions; images remain part of the model input.
• Data: CLEVR-MCQ-4 screening and calibration scenes (360 training, 60 validation scenes), disjoint from the evaluation scenes.
• Optimizer: SGD with Nesterov momentum 0.9, lr 0.1 with linear decay, weight decay 1e-3, gradient clip 1.0, 2^14 tokens per step, 250 steps; best validation-KL checkpoint (the final step for every translator). Seed 20261027.

Results (last translated stage, KL to the final distribution)

Model	Logit Lens	Tuned Lens
Qwen2.5-VL-7B	7.64	0.050
InternVL3.5-8B	0.89	0.10
LLaVA-OneVision-7B-SI	1.10	0.027
PaliGemma2-10B	2.06	0.16

PaliGemma2's translators fit less well in the middle layers (KL about 0.76), so treat its readouts as provisional.

W&B runs: Qwen, InternVL, LLaVA, PaliGemma2.

Citation

If you use these translators, please also cite the Tuned Lens paper: Belrose et al., 2023, Eliciting Latent Predictions from Transformers with the Tuned Lens (arXiv:2303.08112).