README.md

metadata

license: other
license_name: base-model-terms
pipeline_tag: image-text-to-text
base_model:
  - Qwen/Qwen2.5-VL-3B-Instruct
  - google/paligemma2-3b-mix-448
  - lmsys/vicuna-7b-v1.5
  - openai/clip-vit-large-patch14-336
datasets:
  - Lyy0725/VisRO
tags:
  - vision-language-model
  - visual-reasoning
  - adaloop

AdaLoop: Thinking with the Vision Encoder

     

These are the checkpoints for the paper AdaLoop: Thinking with the Vision Encoder, for Qwen2.5-VL-3B, PaliGemma2-3B and LLaVA-1.5-7B. They are loaded with the AdaLoop code, which also has the training and evaluation recipes. The training and evaluation data is VisRO.

Looking once versus repeatedly. AdaLoop turns a span of the frozen vision encoder into a question-conditioned perception loop that updates its visual state over T iterations; the language model answers from the resulting sequence.

Checkpoints

Scores are macro accuracy on VisRO-Core6 (3,600 items per split), as measured from this checkpoint / reported in the paper. Variants A, B and C use p=1 stopping (stop once two consecutive answers agree) with at most 3 iterations.

Backbone	Setting	Path	ID	OOD
Qwen2.5-VL-3B	SFT	qwen/sft	49.9 / 49.9	23.6 / 23.9
Qwen2.5-VL-3B	AdaLN	qwen/adaln	59.9 / 60.0	30.2 / 30.2
Qwen2.5-VL-3B	AdaLoop-A	qwen/variant_a	64.6 / 64.6	32.5 / 32.4
Qwen2.5-VL-3B	AdaLoop-B	qwen/variant_b	62.1 / 62.1	31.1 / 31.1
Qwen2.5-VL-3B	AdaLoop-C	qwen/variant_c	63.0 / 63.0	32.0 / 32.0
PaliGemma2-3B	SFT	paligemma/sft	62.7 / 62.6	31.5 / 31.5
PaliGemma2-3B	AdaLN	paligemma/adaln	72.4 / 72.4	40.0 / 40.0
PaliGemma2-3B	AdaLoop-A	paligemma/variant_a	75.2 / 75.2	43.9 / 43.9
PaliGemma2-3B	AdaLoop-B	paligemma/variant_b	74.0 / –	43.7 / –
PaliGemma2-3B	AdaLoop-C	paligemma/variant_c	74.1 / –	41.1 / –
LLaVA-1.5-7B	SFT	llava/sft	50.4 / 50.4	24.8 / 24.8
LLaVA-1.5-7B	AdaLN	llava/adaln	57.7 / 57.7	29.6 / 29.6
LLaVA-1.5-7B	AdaLoop-A	llava/variant_a	61.9 / 61.8	31.2 / 31.1
LLaVA-1.5-7B	AdaLoop-B	llava/variant_b	59.3 / –	29.7 / –
LLaVA-1.5-7B	AdaLoop-C	llava/variant_c	60.0 / –	30.6 / –

Slight differences may arise from different hardware and numeric precision. The Qwen SFT checkpoint and the PaliGemma and LLaVA Variant B and C checkpoints were trained by us with the same recipes (the paper reports Variants B and C for Qwen only); all others are the paper's runs.

Download

Download one checkpoint, or everything:

hf download Lyy0725/AdaLoop --include "qwen/variant_a/*" --local-dir checkpoints
hf download Lyy0725/AdaLoop --local-dir checkpoints

Then evaluate it on VisRO with the AdaLoop code, for example:

adaloop-qwen-eval --checkpoint checkpoints/qwen/variant_a \
    --visro-root data/VisRO --tasks core6 --splits eval,ood \
    --output-dir outputs/eval/qwen_variant_a

Base Models

Each checkpoint holds only the trained weights (LoRA and the AdaLoop modules) and is loaded on top of its base model, which its adaloop_manifest.json pins:

Backbone	Base model
Qwen2.5-VL-3B	Qwen/Qwen2.5-VL-3B-Instruct, fetched from the Hub at the pinned revision
PaliGemma2-3B	the Big Vision weights paligemma2-3b-mix-448.b16.npz from the gated google/paligemma2-3b-mix-448-jax, and the PaliGemma SentencePiece tokenizer
LLaVA-1.5-7B	our stage-2 LLaVA-1.5-7B at 448 px, llava/base/llava-v1.5-7b-448-s2 in this repository (Vicuna-7B-v1.5 trained with the LLaVA-1.5 recipe at 448 px), and the CLIP tower openai/clip-vit-large-patch14-336

Files

Every checkpoint directory holds adaloop_config.yaml (the training recipe), adaloop_manifest.json (the base model and every file, pinned by size and SHA-256), adaloop_runtime.json and SHA256SUMS, plus its weights:

• Qwen: adapter_config.json, adapter_model.safetensors (LoRA), non_lora_trainables.safetensors (merger and AdaLoop modules) and processor/;
• PaliGemma: trainable.params.msgpack;
• LLaVA: adapter_config.json, adapter_model.safetensors (LoRA), non_lora_trainables.safetensors (projector and AdaLoop modules) and config.json.

The loaders check every file against the manifest before use. SHA256SUMS at the root lists every checkpoint file.

License

The checkpoints are derived from Qwen2.5-VL, PaliGemma 2 and LLaVA-1.5 (Vicuna, Llama 2, CLIP), and each is subject to the terms of its base model: the Qwen Research License of Qwen2.5-VL-3B-Instruct, the Gemma Terms of Use, and the Llama 2 Community License. The AdaLoop code is under the Apache License 2.0, and VisRO under CC BY-SA 4.0.

Citation

If you find this work useful in your research, please consider citing our paper :)

@article{adaloop2026,
  title={AdaLoop: Thinking with the Vision Encoder},
  author={Han, Xinran Nicole and Lu, Yiyang and Gan, Katelyn and Zickler, Todd},
  journal={arXiv preprint arXiv:2610.21323},
  year={2026}
}