README.md

---
license: apache-2.0
language:
- en
base_model: Qwen/Qwen3-VL-8B-Instruct
library_name: transformers
pipeline_tag: image-text-to-text
tags:
- qwen3-vl
- vision-language
- document-vqa
- long-document-understanding
- tool-use
- agentic-ai
- reinforcement-learning
- visual-question-answering
datasets:
- m-Just/InSight-doc-SFT-18k
- m-Just/InSight-doc-RL-19k
---

# InSight-doc-8B

<p align="center">
  <img alt="InSight-doc logo" src="assets/insight_doc_logo.png" width="700" style="max-width: 100%;">
</p>

<h3 align="center">Agentic Visual Perception for Long-Document Understanding</h3>

<div align="center">

📄 **[Paper](https://arxiv.org/abs/2608.10628)** |
💻 **[Code](https://github.com/m-Just/InSight-doc)** |
🧩 **[SFT Data](https://huggingface.co/datasets/m-Just/InSight-doc-SFT-18k)** |
🎯 **[RL Data](https://huggingface.co/datasets/m-Just/InSight-doc-RL-19k)** |
🎬 **[Replay Demo](https://vaynexie.github.io/insight-doc-demo-display/demo_display.html)** |
🚀 **[Live Demo](https://huggingface.co/spaces/leoyu112211/insight-doc-online-demo)**

</div>

<p align="center">
  <i>Understand the big picture.&nbsp; Focus on the right details.&nbsp; Answer from the evidence.</i>
</p>

## Overview

InSight-doc-8B is an 8B-class vision-language agent for long-document
understanding. It is initialized from
[Qwen/Qwen3-VL-8B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct)
and trained to use visual resolution as an adaptive inference-time resource:
start from low-resolution page views, call a zoom-in tool on selected regions,
then answer from the returned high-resolution evidence.

<p align="center">
  <img alt="InSight-doc coarse-to-fine visual perception" src="assets/teaser.png" width="850" style="max-width: 100%;">
</p>

This checkpoint is a Hugging Face-compatible Qwen3-VL model. To reproduce the
reported agentic behavior, use it with the InSight-doc agent loop and
`image_zoom_in_tool` rather than plain single-turn inference.

## Model Details

| Item | Value |
|---|---|
| Model | InSight-doc-8B |
| Base model | Qwen/Qwen3-VL-8B-Instruct |
| Architecture | Qwen3-VL vision-language model with tool-use SFT and RL |
| Primary task | Long-document visual question answering |
| Agent tool | `image_zoom_in_tool` for region-level visual evidence acquisition |
| Training data | InSight-doc-SFT-18k and InSight-doc-RL-19k |
| License | Apache-2.0 |

## Training Summary

InSight-doc-8B is trained in two stages:

1. **Supervised fine-tuning.** Full-parameter SFT from Qwen3-VL-8B-Instruct on
   17,913 curated zoom-in trajectories. The released recipe freezes the vision
   tower, uses max sequence length 65,536, sequence parallel size 4, global
   batch size 32, cosine LR `5e-6 -> 5e-7`, and two epochs.
2. **Reinforcement learning.** RL starts from the SFT checkpoint and trains on
   19,236 hard prompts with the same zoom-in agent interface. The released RL
   recipe uses weighted refill source sampling, temperature 0.7, top-p 0.8,
   top-k 20, presence penalty 1.5, and 2,000 total RL steps.

The SFT data contains 14,216 answerable and 3,697 unanswerable trajectories.
The RL data contains 10,579 answerable and 8,657 unanswerable prompts, with the
released sampler targeting an effective 86%/14% answerable/unanswerable mix.

## Evaluation Highlights

Across DUDE, MP-DocVQA, MMLongBench-Doc, and LongDocURL, InSight-doc-8B
improves over Qwen3-VL-8B by **4.3-16.4 accuracy points** under medium-to-low
input resolution settings. On MMLongBench-Doc and LongDocURL, it reduces
hallucination on unanswerable questions by **40%+** and lowers latency by
**41%-68%** (**1.7x-3.1x speedup**) while maintaining an accuracy lead.

<p align="center">
  <img alt="Long-document efficiency comparison" src="assets/longdoc_efficiency.png" width="450" style="max-width: 100%;">
</p>

Across the four document VQA benchmarks, InSight-doc shifts the accuracy-efficiency
Pareto frontier upward and leftward, achieving higher accuracy with shorter
sequences and lower latency. Darker points indicate higher initial input DPI.

<p align="center">
  <img alt="Accuracy-efficiency Pareto frontier" src="assets/accuracy_efficiency_pareto.png" width="850" style="max-width: 100%;">
</p>

See the paper and release repository for complete benchmark tables and the
exact evaluation configuration.

## Demos

- [Replay demo](https://vaynexie.github.io/insight-doc-demo-display/demo_display.html):
  compares InSight-doc-8B and Qwen3-VL-8B side by side on replayed examples.
- [Live demo](https://huggingface.co/spaces/leoyu112211/insight-doc-online-demo):
  runs an interactive InSight-doc-8B demo.

## Usage

### Agentic Evaluation

The recommended path is the release evaluation script, which serves the model
with vLLM and runs the shared InSight-doc agent/evaluator:

```sh
git clone --recurse-submodules https://github.com/m-Just/InSight-doc.git
cd InSight-doc

pip install -e .
pip install -e ./verl

export MODEL_PATH=InSight-doc/InSight-doc-8B
export VAL_FILES='/path/to/longdocurl.parquet,/path/to/mmlongbench.parquet'
export RESCALES='0.25 0.35 0.5'
export EVAL_CUDA_VISIBLE_DEVICES=0,1,2,3
export OPENAI_API_KEY=...
export OPENAI_BASE_URL=https://.../v1

bash scripts/evaluate_insight_doc.sh
```

Important vLLM defaults in the release config are 4 replicas, 1 GPU per
replica, `max_model_len=262144`, `max_num_seqs=64`, chunked prefill, prefix
caching, and the same sampling settings used by RL validation.

### Plain Transformers Loading

The checkpoint can also be loaded as a regular Qwen3-VL model for basic
experimentation, but this does not execute the zoom-in tool loop:

```python
import torch
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

model_id = "InSight-doc/InSight-doc-8B"

processor = AutoProcessor.from_pretrained(model_id)
model = Qwen3VLForConditionalGeneration.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)
```

Use explicit bf16 loading for inference; the uploaded config currently records
`dtype: float32`, so `dtype="auto"` may load fp32 weights in plain Transformers.

## Input Format

The agentic policy expects multimodal chat messages containing low-resolution
page images and a tool schema for `image_zoom_in_tool`. During generation, the
model may emit structured tool calls that specify an image index and a 2D
bounding box. The runtime crops the requested region from the original page and
returns it as a new visual observation.

The released SFT dataset includes the tool schema in a `tools` column. For RL
and evaluation, the same schema is loaded from the release configuration at
[`recipe/vsearch/config/image_zoom_in_tool_config.yaml`](https://github.com/m-Just/InSight-doc/blob/main/recipe/vsearch/config/image_zoom_in_tool_config.yaml).

## License

The model weights are released under the Apache License 2.0. The checkpoint is
derived from Qwen/Qwen3-VL-8B-Instruct, which is also released under
Apache-2.0.

## Citation

```bibtex
@article{li2026insightdoc,
  title={InSight-doc: Agentic Visual Perception for Long-Document Understanding},
  author={Li, Kaican and Xie, Weiyan and Yao, Lewei and Wu, Jiannan and Hong, Lanqing and Huang, Yongxiang and Zhang, Nevin L.},
  journal={arXiv preprint arXiv:2608.10628},
  year={2026}
}
```