| --- |
| license: apache-2.0 |
| language: |
| - en |
| pipeline_tag: image-text-to-text |
| library_name: transformers |
| tags: |
| - multimodal |
| - document-understanding |
| --- |
| |
| # DocPilot-4B |
|
|
| DocPilot-4B is a 4B action-tag model for long visual document understanding. |
| Weights are in the `DocPilot-4B` folder. |
|
|
| - Paper: https://arxiv.org/abs/2610.21327 |
| - Code: https://github.com/HKUSTDial/DocPilot |
|
|
| ## Requirements |
|
|
| ```bash |
| pip install vllm transformers |
| ``` |
|
|
| ## Quickstart |
|
|
| ```python |
| from transformers import AutoProcessor, AutoModelForImageTextToText |
| |
| model = AutoModelForImageTextToText.from_pretrained( |
| "SetonLiang2/DocPilot", |
| subfolder="DocPilot-4B", |
| torch_dtype="auto", |
| device_map="auto", |
| ) |
| processor = AutoProcessor.from_pretrained( |
| "SetonLiang2/DocPilot", |
| subfolder="DocPilot-4B", |
| ) |
| ``` |
|
|
| ## Deployment |
|
|
| vLLM reads a local folder directly. Download the subfolder, then serve it: |
|
|
| ```bash |
| huggingface-cli download SetonLiang2/DocPilot --include "DocPilot-4B/*" --local-dir ./DocPilot |
| vllm serve ./DocPilot/DocPilot-4B --host 0.0.0.0 --port 8088 --dtype bfloat16 |
| ``` |
|
|
| Point DocPilot at this server with `protocol: vllm_tags`: |
|
|
| ```yaml |
| llm: |
| provider: openai |
| protocol: vllm_tags |
| model: ./DocPilot/DocPilot-4B |
| base_url: http://127.0.0.1:8088/v1 |
| api_key: EMPTY |
| ``` |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{liang2026docpilot, |
| title={DocPilot: Interactive Agent Orchestration and Training for Long Visual Document Understanding}, |
| author={Liang, Zhuowen and Lin, Xiaotian and Luo, Yuyu and Wang, Haixun and Tang, Nan}, |
| journal={arXiv preprint arXiv:2610.21327}, |
| year={2026} |
| } |
| ``` |
|
|
