File size: 1,583 Bytes
f93cc9b 18b2011 f93cc9b 18b2011 f93cc9b 18b2011 f93cc9b 18b2011 f93cc9b 18b2011 f93cc9b 18b2011 f93cc9b 18b2011 f93cc9b 18b2011 f93cc9b 18b2011 f93cc9b 18b2011 f93cc9b 18b2011 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 | ---
license: apache-2.0
language:
- en
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- multimodal
- document-understanding
---
# DocPilot-4B
DocPilot-4B is a 4B action-tag model for long visual document understanding.
Weights are in the `DocPilot-4B` folder.
- Paper: https://arxiv.org/abs/2610.21327
- Code: https://github.com/HKUSTDial/DocPilot
## Requirements
```bash
pip install vllm transformers
```
## Quickstart
```python
from transformers import AutoProcessor, AutoModelForImageTextToText
model = AutoModelForImageTextToText.from_pretrained(
"SetonLiang2/DocPilot",
subfolder="DocPilot-4B",
torch_dtype="auto",
device_map="auto",
)
processor = AutoProcessor.from_pretrained(
"SetonLiang2/DocPilot",
subfolder="DocPilot-4B",
)
```
## Deployment
vLLM reads a local folder directly. Download the subfolder, then serve it:
```bash
huggingface-cli download SetonLiang2/DocPilot --include "DocPilot-4B/*" --local-dir ./DocPilot
vllm serve ./DocPilot/DocPilot-4B --host 0.0.0.0 --port 8088 --dtype bfloat16
```
Point DocPilot at this server with `protocol: vllm_tags`:
```yaml
llm:
provider: openai
protocol: vllm_tags
model: ./DocPilot/DocPilot-4B
base_url: http://127.0.0.1:8088/v1
api_key: EMPTY
```
## Citation
```bibtex
@article{liang2026docpilot,
title={DocPilot: Interactive Agent Orchestration and Training for Long Visual Document Understanding},
author={Liang, Zhuowen and Lin, Xiaotian and Luo, Yuyu and Wang, Haixun and Tang, Nan},
journal={arXiv preprint arXiv:2610.21327},
year={2026}
}
```
|