| --- |
| license: other |
| license_name: base-model-terms |
| pipeline_tag: image-text-to-text |
| base_model: |
| - Qwen/Qwen2.5-VL-3B-Instruct |
| - google/paligemma2-3b-mix-448 |
| - lmsys/vicuna-7b-v1.5 |
| - openai/clip-vit-large-patch14-336 |
| datasets: |
| - Lyy0725/VisRO |
| tags: |
| - vision-language-model |
| - visual-reasoning |
| - adaloop |
| --- |
| |
| # AdaLoop: Thinking with the Vision Encoder |
|
|
| [](https://arxiv.org/abs/2610.21323) |
| [](https://github.com/Lyy-iiis/AdaLoop) |
| [](https://huggingface.co/datasets/Lyy0725/VisRO) |
|
|
| These are the checkpoints for the paper [AdaLoop: Thinking with the Vision Encoder](https://arxiv.org/abs/2610.21323), for Qwen2.5-VL-3B, PaliGemma2-3B and LLaVA-1.5-7B. They are loaded with the [AdaLoop code](https://github.com/Lyy-iiis/AdaLoop), which also has the training and evaluation recipes. The training and evaluation data is [VisRO](https://huggingface.co/datasets/Lyy0725/VisRO). |
|
|
| <p align="center"> |
| <img src="assets/teaser.png" alt="A standard VLM encodes the image once; AdaLoop iterates a question-conditioned span of the vision encoder" width="100%"/> |
| </p> |
| <p align="left"> |
| <em><strong>Looking once versus repeatedly.</strong> AdaLoop turns a span of the frozen vision encoder into a question-conditioned perception loop that updates its visual state over T iterations; the language model answers from the resulting sequence.</em> |
| </p> |
|
|
| ## Checkpoints |
|
|
| Scores are macro accuracy on VisRO-Core6 (3,600 items per split), as *measured from this checkpoint / reported in the paper*. Variants A, B and C use `p=1` stopping (stop once two consecutive answers agree) with at most 3 iterations. |
|
|
| | Backbone | Setting | Path | ID | OOD | |
| | --- | --- | --- | --- | --- | |
| | Qwen2.5-VL-3B | SFT | [`qwen/sft`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/qwen/sft) | 49.9 / 49.9 | 23.6 / 23.9 | |
| | Qwen2.5-VL-3B | AdaLN | [`qwen/adaln`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/qwen/adaln) | 59.9 / 60.0 | 30.2 / 30.2 | |
| | Qwen2.5-VL-3B | AdaLoop-A | [`qwen/variant_a`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/qwen/variant_a) | 64.6 / 64.6 | 32.5 / 32.4 | |
| | Qwen2.5-VL-3B | AdaLoop-B | [`qwen/variant_b`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/qwen/variant_b) | 62.1 / 62.1 | 31.1 / 31.1 | |
| | Qwen2.5-VL-3B | AdaLoop-C | [`qwen/variant_c`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/qwen/variant_c) | 63.0 / 63.0 | 32.0 / 32.0 | |
| | PaliGemma2-3B | SFT | [`paligemma/sft`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/paligemma/sft) | 62.7 / 62.6 | 31.5 / 31.5 | |
| | PaliGemma2-3B | AdaLN | [`paligemma/adaln`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/paligemma/adaln) | 72.4 / 72.4 | 40.0 / 40.0 | |
| | PaliGemma2-3B | AdaLoop-A | [`paligemma/variant_a`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/paligemma/variant_a) | 75.2 / 75.2 | 43.9 / 43.9 | |
| | PaliGemma2-3B | AdaLoop-B | [`paligemma/variant_b`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/paligemma/variant_b) | 74.0 / – | 43.7 / – | |
| | PaliGemma2-3B | AdaLoop-C | [`paligemma/variant_c`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/paligemma/variant_c) | 74.1 / – | 41.1 / – | |
| | LLaVA-1.5-7B | SFT | [`llava/sft`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/llava/sft) | 50.4 / 50.4 | 24.8 / 24.8 | |
| | LLaVA-1.5-7B | AdaLN | [`llava/adaln`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/llava/adaln) | 57.7 / 57.7 | 29.6 / 29.6 | |
| | LLaVA-1.5-7B | AdaLoop-A | [`llava/variant_a`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/llava/variant_a) | 61.9 / 61.8 | 31.2 / 31.1 | |
| | LLaVA-1.5-7B | AdaLoop-B | [`llava/variant_b`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/llava/variant_b) | 59.3 / – | 29.7 / – | |
| | LLaVA-1.5-7B | AdaLoop-C | [`llava/variant_c`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/llava/variant_c) | 60.0 / – | 30.6 / – | |
|
|
| Slight differences may arise from different hardware and numeric precision. The Qwen SFT checkpoint and the PaliGemma and LLaVA Variant B and C checkpoints were trained by us with the same recipes (the paper reports Variants B and C for Qwen only); all others are the paper's runs. |
|
|
| ## Download |
|
|
| Download one checkpoint, or everything: |
|
|
| ```bash |
| hf download Lyy0725/AdaLoop --include "qwen/variant_a/*" --local-dir checkpoints |
| hf download Lyy0725/AdaLoop --local-dir checkpoints |
| ``` |
|
|
| Then evaluate it on VisRO with the AdaLoop code, for example: |
|
|
| ```bash |
| adaloop-qwen-eval --checkpoint checkpoints/qwen/variant_a \ |
| --visro-root data/VisRO --tasks core6 --splits eval,ood \ |
| --output-dir outputs/eval/qwen_variant_a |
| ``` |
|
|
| ## Base Models |
|
|
| Each checkpoint holds only the trained weights (LoRA and the AdaLoop modules) and is loaded on top of its base model, which its `adaloop_manifest.json` pins: |
|
|
| | Backbone | Base model | |
| | --- | --- | |
| | Qwen2.5-VL-3B | [`Qwen/Qwen2.5-VL-3B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct), fetched from the Hub at the pinned revision | |
| | PaliGemma2-3B | the Big Vision weights `paligemma2-3b-mix-448.b16.npz` from the gated [`google/paligemma2-3b-mix-448-jax`](https://huggingface.co/google/paligemma2-3b-mix-448-jax), and the PaliGemma SentencePiece tokenizer | |
| | LLaVA-1.5-7B | our stage-2 LLaVA-1.5-7B at 448 px, [`llava/base/llava-v1.5-7b-448-s2`](https://huggingface.co/Lyy0725/AdaLoop/tree/main/llava/base) in this repository (Vicuna-7B-v1.5 trained with the LLaVA-1.5 recipe at 448 px), and the CLIP tower [`openai/clip-vit-large-patch14-336`](https://huggingface.co/openai/clip-vit-large-patch14-336) | |
|
|
| ## Files |
|
|
| Every checkpoint directory holds `adaloop_config.yaml` (the training recipe), `adaloop_manifest.json` (the base model and every file, pinned by size and SHA-256), `adaloop_runtime.json` and `SHA256SUMS`, plus its weights: |
|
|
| - Qwen: `adapter_config.json`, `adapter_model.safetensors` (LoRA), `non_lora_trainables.safetensors` (merger and AdaLoop modules) and `processor/`; |
| - PaliGemma: `trainable.params.msgpack`; |
| - LLaVA: `adapter_config.json`, `adapter_model.safetensors` (LoRA), `non_lora_trainables.safetensors` (projector and AdaLoop modules) and `config.json`. |
|
|
| The loaders check every file against the manifest before use. `SHA256SUMS` at the root lists every checkpoint file. |
|
|
| ## License |
|
|
| The checkpoints are derived from Qwen2.5-VL, PaliGemma 2 and LLaVA-1.5 (Vicuna, Llama 2, CLIP), and each is subject to the terms of its base model: the [Qwen Research License](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct/blob/main/LICENSE) of Qwen2.5-VL-3B-Instruct, the Gemma Terms of Use, and the Llama 2 Community License. The AdaLoop code is under the Apache License 2.0, and VisRO under CC BY-SA 4.0. |
|
|
| ## Citation |
|
|
| If you find this work useful in your research, please consider citing our paper :) |
|
|
| ```bib |
| @article{adaloop2026, |
| title={AdaLoop: Thinking with the Vision Encoder}, |
| author={Han, Xinran Nicole and Lu, Yiyang and Gan, Katelyn and Zickler, Todd}, |
| journal={arXiv preprint arXiv:2610.21323}, |
| year={2026} |
| } |
| ``` |
|
|
