README.md

metadata

license: apache-2.0
language:
  - en
pipeline_tag: image-text-to-text
library_name: transformers
tags:
  - multimodal
  - document-understanding

DocPilot-4B

DocPilot-4B is a 4B action-tag model for long visual document understanding. Weights are in the DocPilot-4B folder.

• Paper: https://arxiv.org/abs/2610.21327
• Code: https://github.com/HKUSTDial/DocPilot

Requirements

pip install vllm transformers

Quickstart

from transformers import AutoProcessor, AutoModelForImageTextToText

model = AutoModelForImageTextToText.from_pretrained(
    "SetonLiang2/DocPilot",
    subfolder="DocPilot-4B",
    torch_dtype="auto",
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(
    "SetonLiang2/DocPilot",
    subfolder="DocPilot-4B",
)

Deployment

vLLM reads a local folder directly. Download the subfolder, then serve it:

huggingface-cli download SetonLiang2/DocPilot --include "DocPilot-4B/*" --local-dir ./DocPilot
vllm serve ./DocPilot/DocPilot-4B --host 0.0.0.0 --port 8088 --dtype bfloat16

Point DocPilot at this server with protocol: vllm_tags:

llm:
  provider: openai
  protocol: vllm_tags
  model: ./DocPilot/DocPilot-4B
  base_url: http://127.0.0.1:8088/v1
  api_key: EMPTY

Citation

@article{liang2026docpilot,
  title={DocPilot: Interactive Agent Orchestration and Training for Long Visual Document Understanding},
  author={Liang, Zhuowen and Lin, Xiaotian and Luo, Yuyu and Wang, Haixun and Tang, Nan},
  journal={arXiv preprint arXiv:2610.21327},
  year={2026}
}