File size: 7,104 Bytes

006b0d5

---
license: other
license_name: base-model-terms
pipeline_tag: image-text-to-text
base_model:
- Qwen/Qwen2.5-VL-3B-Instruct
- google/paligemma2-3b-mix-448
- lmsys/vicuna-7b-v1.5
- openai/clip-vit-large-patch14-336
datasets:
- Lyy0725/VisRO
tags:
- vision-language-model
- visual-reasoning
- adaloop
---

# AdaLoop: Thinking with the Vision Encoder

[![arXiv](https://img.shields.io/badge/arXiv-2610.21323-b31b1b.svg)](https://arxiv.org/abs/2610.21323)&nbsp;
[![Code](https://img.shields.io/badge/GitHub-AdaLoop-black.svg)](https://github.com/Lyy-iiis/AdaLoop)&nbsp;
[![Dataset](https://img.shields.io/badge/Hugging%20Face-VisRO-yellow.svg)](https://huggingface.co/datasets/Lyy0725/VisRO)&nbsp;

These are the checkpoints for the paper [AdaLoop: Thinking with the Vision Encoder](https://arxiv.org/abs/2610.21323), for Qwen2.5-VL-3B, PaliGemma2-3B and LLaVA-1.5-7B. They are loaded with the [AdaLoop code](https://github.com/Lyy-iiis/AdaLoop), which also has the training and evaluation recipes. The training and evaluation data is [VisRO](https://huggingface.co/datasets/Lyy0725/VisRO).

<p align="center">
  <img src="assets/teaser.png" alt="A standard VLM encodes the image once; AdaLoop iterates a question-conditioned span of the vision encoder" width="100%"/>
</p>
<p align="left">
  <em><strong>Looking once versus repeatedly.</strong> AdaLoop turns a span of the frozen vision encoder into a question-conditioned perception loop that updates its visual state over T iterations; the language model answers from the resulting sequence.</em>
</p>

## Checkpoints

Scores are macro accuracy on VisRO-Core6 (3,600 items per split), as *measured from this checkpoint / reported in the paper*. Variants A, B and C use `p=1` stopping (stop once two consecutive answers agree) with at most 3 iterations.

| Backbone | Setting | Path | ID | OOD |
| --- | --- | --- | --- | --- |
| Qwen2.5-VL-3B | SFT | [`qwen/sft`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/qwen/sft) | 49.9 / 49.9 | 23.6 / 23.9 |
| Qwen2.5-VL-3B | AdaLN | [`qwen/adaln`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/qwen/adaln) | 59.9 / 60.0 | 30.2 / 30.2 |
| Qwen2.5-VL-3B | AdaLoop-A | [`qwen/variant_a`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/qwen/variant_a) | 64.6 / 64.6 | 32.5 / 32.4 |
| Qwen2.5-VL-3B | AdaLoop-B | [`qwen/variant_b`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/qwen/variant_b) | 62.1 / 62.1 | 31.1 / 31.1 |
| Qwen2.5-VL-3B | AdaLoop-C | [`qwen/variant_c`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/qwen/variant_c) | 63.0 / 63.0 | 32.0 / 32.0 |
| PaliGemma2-3B | SFT | [`paligemma/sft`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/paligemma/sft) | 62.7 / 62.6 | 31.5 / 31.5 |
| PaliGemma2-3B | AdaLN | [`paligemma/adaln`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/paligemma/adaln) | 72.4 / 72.4 | 40.0 / 40.0 |
| PaliGemma2-3B | AdaLoop-A | [`paligemma/variant_a`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/paligemma/variant_a) | 75.2 / 75.2 | 43.9 / 43.9 |
| PaliGemma2-3B | AdaLoop-B | [`paligemma/variant_b`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/paligemma/variant_b) | 74.0 / – | 43.7 / – |
| PaliGemma2-3B | AdaLoop-C | [`paligemma/variant_c`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/paligemma/variant_c) | 74.1 / – | 41.1 / – |
| LLaVA-1.5-7B | SFT | [`llava/sft`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/llava/sft) | 50.4 / 50.4 | 24.8 / 24.8 |
| LLaVA-1.5-7B | AdaLN | [`llava/adaln`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/llava/adaln) | 57.7 / 57.7 | 29.6 / 29.6 |
| LLaVA-1.5-7B | AdaLoop-A | [`llava/variant_a`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/llava/variant_a) | 61.9 / 61.8 | 31.2 / 31.1 |
| LLaVA-1.5-7B | AdaLoop-B | [`llava/variant_b`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/llava/variant_b) | 59.3 / – | 29.7 / – |
| LLaVA-1.5-7B | AdaLoop-C | [`llava/variant_c`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/llava/variant_c) | 60.0 / – | 30.6 / – |

Slight differences may arise from different hardware and numeric precision. The Qwen SFT checkpoint and the PaliGemma and LLaVA Variant B and C checkpoints were trained by us with the same recipes (the paper reports Variants B and C for Qwen only); all others are the paper's runs.

## Download

Download one checkpoint, or everything:

```bash
hf download Lyy0725/AdaLoop --include "qwen/variant_a/*" --local-dir checkpoints
hf download Lyy0725/AdaLoop --local-dir checkpoints
```

Then evaluate it on VisRO with the AdaLoop code, for example:

```bash
adaloop-qwen-eval --checkpoint checkpoints/qwen/variant_a \
    --visro-root data/VisRO --tasks core6 --splits eval,ood \
    --output-dir outputs/eval/qwen_variant_a
```

## Base Models

Each checkpoint holds only the trained weights (LoRA and the AdaLoop modules) and is loaded on top of its base model, which its `adaloop_manifest.json` pins:

| Backbone | Base model |
| --- | --- |
| Qwen2.5-VL-3B | [`Qwen/Qwen2.5-VL-3B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct), fetched from the Hub at the pinned revision |
| PaliGemma2-3B | the Big Vision weights `paligemma2-3b-mix-448.b16.npz` from the gated [`google/paligemma2-3b-mix-448-jax`](https://huggingface.co/google/paligemma2-3b-mix-448-jax), and the PaliGemma SentencePiece tokenizer |
| LLaVA-1.5-7B | our stage-2 LLaVA-1.5-7B at 448 px, [`llava/base/llava-v1.5-7b-448-s2`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/llava/base) in this repository (Vicuna-7B-v1.5 trained with the LLaVA-1.5 recipe at 448 px), and the CLIP tower [`openai/clip-vit-large-patch14-336`](https://huggingface.co/openai/clip-vit-large-patch14-336) |

## Files

Every checkpoint directory holds `adaloop_config.yaml` (the training recipe), `adaloop_manifest.json` (the base model and every file, pinned by size and SHA-256), `adaloop_runtime.json` and `SHA256SUMS`, plus its weights:

- Qwen: `adapter_config.json`, `adapter_model.safetensors` (LoRA), `non_lora_trainables.safetensors` (merger and AdaLoop modules) and `processor/`;
- PaliGemma: `trainable.params.msgpack`;
- LLaVA: `adapter_config.json`, `adapter_model.safetensors` (LoRA), `non_lora_trainables.safetensors` (projector and AdaLoop modules) and `config.json`.

The loaders check every file against the manifest before use. `SHA256SUMS` at the root lists every checkpoint file.

## License

The checkpoints are derived from Qwen2.5-VL, PaliGemma 2 and LLaVA-1.5 (Vicuna, Llama 2, CLIP), and each is subject to the terms of its base model: the [Qwen Research License](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct/blob/main/LICENSE) of Qwen2.5-VL-3B-Instruct, the Gemma Terms of Use, and the Llama 2 Community License. The AdaLoop code is under the Apache License 2.0, and VisRO under CC BY-SA 4.0.

## Citation

If you find this work useful in your research, please consider citing our paper :)

```bib
@article{adaloop2026,
  title={AdaLoop: Thinking with the Vision Encoder},
  author={Han, Xinran Nicole and Lu, Yiyang and Gan, Katelyn and Zickler, Todd},
  journal={arXiv preprint arXiv:2610.21323},
  year={2026}
}
```