README.md

---
license: mit
tags:
  - vision-language-model
  - spatial-reasoning
  - parameter-efficient
  - adapter
library_name: pytorch
---

# Lightweight Visual-Token Spatial Adapters for Frozen VLMs

Tiny adapters that improve spatial reasoning in **frozen** vision–language models by injecting at
the **visual-token interface** (projector/merger output + early decoder layers). The base VLM is
never fine-tuned; only these adapters (0.003–0.08% of the model) are trained, on VSR.

They accompany our analysis that VLM spatial failure is **mis-routing / a decision-stage language
prior**, not missing perception: the spatial relation is already decodable in the visual tokens
from layer 0, but a language prior over relation words buries the correct answer (e.g. "under" and
"behind" are emitted 0% of the time). Injecting a small learned signal at the visual-token
interface — the site we localize causally with activation patching — unlocks it.

## Checkpoints

Each `*.pt` is a `state_dict` of a bottleneck adapter `Δ(x) = W_up · GELU(W_down · x)` added to the
visual tokens (and, for the Qwen files, a few early decoder layers at image positions).

| file | base model | trainable params | notes |
|---|---|---|---|
| `qwen3vl-8b_adapter_rank8.pt`  | Qwen/Qwen3-VL-8B-Instruct | 279K (0.003%) | best efficiency; VSR +3.2, CV-Bench transfer +9.9 |
| `qwen3vl-8b_adapter_rank96.pt` | Qwen/Qwen3-VL-8B-Instruct | 3.16M (0.04%) | VSR +2.8, CV-Bench transfer +10.7 |
| `qwen25-3b_adapter.pt`   | Qwen/Qwen2.5-VL-3B-Instruct | 3.2M | VSR +3.9 |
| `internvl3-2b_adapter.pt`| OpenGVLab/InternVL3-2B-hf   | 2.4M | VSR +4.2 |
| `smolvlm_adapter.pt`     | HuggingFaceTB/SmolVLM2-2.2B-Instruct | 3.2M | VSR +1.3 |
| `llava_adapter.pt`       | llava-hf/llava-1.5-7b-hf    | 6.3M | ~0 (base at chance on VSR) |
| `idefics2_adapter.pt`    | HuggingFaceM4/idefics2-8b   | 6.3M | −2.0 (Perceiver connector; included for completeness) |

All numbers are VSR test accuracy gains with the base VLM frozen. The adapter beats a well-tuned
LoRA (attention weights) at 11–55× fewer parameters, and matches its cross-benchmark transfer.

## Boundaries (reported honestly)

- The gain is largely a learned **de-biasing steer** at the visual tokens; on visually homogeneous
  data a *constant, content-free* offset reproduces most of it. A **training-free alternative**
  (prior-calibrated / contrastive decoding) recovers the same failing poles with **zero
  parameters** — use that first.
- Transfer is **relation-type-specific**: helps location/proximity, can *hurt* orientation/viewpoint.
- The interface adapter does **not** work on Perceiver-resampler connectors (Idefics2).

## Usage sketch

```python
import torch, torch.nn.functional as F
from torch import nn

class Bottleneck(nn.Module):
    def __init__(self, d, r):
        super().__init__()
        self.down = nn.Linear(d, r); self.up = nn.Linear(r, d)
    def forward(self, x):
        return self.up(F.gelu(self.down(x)))

sd = torch.load("qwen3vl-8b_adapter_rank8.pt")   # {"merger": ..., "L2": ..., "L6": ..., "L10": ...}
# register forward hooks that add Bottleneck(x) to the projector output (key "merger")
# and to image-token positions at decoder layers {2,6,10} (keys "L2","L6","L10").
```

See the project repository for the exact hook/registration code and evaluation scripts.