README.md

---
license: apache-2.0
pipeline_tag: image-text-to-text
tags:
- image-text-to-text
- transformers
- safetensors
- visual-language
- paddleocr
- document-parse
- HPD-Parsing
- speculative-decoding
- P-MTP
- eval results
base_model: OpenGVLab/InternVL3_5-1B
language:
- en
- zh
library_name: transformers
---
<div align="center">

<h1 align="center">

HPD-Parsing: Hierarchical Parallel Document Parsing

</h1>

[![PaddleOCR](https://img.shields.io/github/stars/PaddlePaddle/PaddleOCR?color=ccf)](https://github.com/PaddlePaddle/PaddleOCR)
[![vLLM](https://img.shields.io/badge/Serving-vLLM-6080F0)](https://github.com/vllm-project/vllm)
[![arxiv](https://img.shields.io/badge/arXiv-b31b1b?logo=arxiv&logoColor=white)](https://arxiv.org/abs/2607.18839)
[![X](https://img.shields.io/badge/X-PaddlePaddle-6080F0)](https://x.com/PaddlePaddle)
[![License](https://img.shields.io/badge/license-Apache_2.0-green)](https://www.apache.org/licenses/LICENSE-2.0)

</div>

<div align="center">

<img src="https://raw.githubusercontent.com/weishu20/HPD-Parsing-images/master/images/teaser.svg" width="800"/>

</div>



## News

* ```2026.07``` 🎉 A live demo is available at [hugging-apps/hpd-parsing](https://huggingface.co/spaces/hugging-apps/hpd-parsing). Thanks to [multimodalart](https://huggingface.co/multimodalart) for building it!
* ```2026.07``` 🚀 We release **HPD-Parsing**, a 1B hierarchical parallel document parser. It achieves 94.91% overall on OmniDocBench v1.6 and 4,752 TPS peak throughput.


## Introduction

<div align="center">
<img src="https://raw.githubusercontent.com/weishu20/HPD-Parsing-images/master/images/hierarchical_decoding.gif " width="800"/>
</div>

We introduce **HPD-Parsing**, a lightweight (1B) and high-throughput document parsing model built on a Hierarchical Parallel Decoding paradigm. Unified VLM-based parsers process an entire page jointly but generate the output through a single token-by-token autoregressive trajectory, creating a sequential bottleneck that grows with document length. HPD-Parsing is motivated by a key property of document parsing: **page structure requires global coordination, whereas content generation is largely localized within individual regions.** Based on this observation, a main layout branch coordinates the global document structure and dynamically dispatches localized content generation to concurrent branches, while Progressive Multi-Token Prediction (P-MTP) further reduces the decoding steps within each branch. **HPD-Parsing achieves an overall score of 94.91% on OmniDocBench v1.6 — a new state of the art among end-to-end unified parsers — while reaching a peak throughput of 4,752 TPS, 2.62× the fastest existing document parser and 3.06× its own autoregressive baseline.**





### **Key Capabilities of HPD-Parsing**

**🚀 Hierarchical Parallel Decoding for High-Throughput Document Parsing**: We introduce Hierarchical Parallel Decoding (HPD), a new decoding paradigm that restructures full-page autoregressive generation into globally coordinated, localized parallel decoding. A main layout branch performs global coordination and dynamically decomposes the conventional single decoding trajectory into concurrent content branches, each responsible for a localized document region. Within each branch, P-MTP further reduces the number of decoding steps by predicting multiple future tokens at each iteration. Together with shared-prefix KV cache reuse, HPD substantially shortens the effective sequential decoding path along both branch and token dimensions.

**🔄 Staged Adaptation with Automated Difficulty-Aware Data Curation**: We develop a staged adaptation strategy that transfers conventional autoregressive document parsing capabilities to the proposed hierarchical parallel decoding paradigm while preserving parsing accuracy. The strategy is supported by an automated difficulty-aware data curation pipeline that integrates large-scale data collection, model-assisted annotation, difficulty estimation, and balanced sampling. By progressively adapting the model and emphasizing challenging samples, the training framework mitigates the accuracy degradation caused by the transition to parallel decoding with minimal manual annotation effort.


**⚡ State-of-the-Art Throughput with Competitive Parsing Accuracy**: HPD-Parsing achieves state-of-the-art inference efficiency on OmniDocBench v1.6, reaching a peak throughput of 4,752 Tokens Per Second (TPS). It delivers 1.62× the throughput of the fastest existing document parsing model and more than 3.06× that of its autoregressive baseline, while maintaining competitive parsing accuracy. These results demonstrate that document parsing can be effectively executed through global layout coordination and localized parallel decoding rather than a single sequential generation trajectory.


### **HPD-Parsing Architecture**

HPD-Parsing adopts **InternVL3.5-1B** as its backbone, applies dynamic tile-based cropping (up to 24 tiles of 448×448) to preserve high-resolution details. Its primary departure from conventional unified parsers is the decoding paradigm: instead of generating the entire page along a single autoregressive trajectory, HPD-Parsing employs a main layout branch to coordinate the global structure and spawns localized content branches for concurrent region-level decoding, with P-MTP integrated into each branch.

<div align="center">
<img src="https://raw.githubusercontent.com/weishu20/HPD-Parsing-images/master/images/method.svg" width="800"/>
</div>


## Inference with vLLM

HPD-Parsing runs on a **customized build of vLLM** (based on vLLM v0.17.1) that implements the dynamic request forking required by hierarchical parallel decoding and adapts P-MTP speculative decoding.


### 1. Using Docker

The Docker image ships the customized vLLM build and all dependencies, which is the easiest way to get started. Its default entrypoint runs the Python API example below out of the box.

**With the online Docker image**, start the container directly. It downloads the model and starts the inference server, which listens on port **8118** by default:

```shell
docker run \
    -it \
    --rm \
    --gpus all \
    --network host \
    ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlepaddle/hpd-parsing-vllm:latest-nvidia-gpu
```

### 2. vLLM Python API

Without Docker, install the customized vLLM prebuilt package (Python 3.10–3.13, NVIDIA driver with CUDA 12.8+) in a virtual environment to avoid dependency conflicts:

```shell
python -m venv .venv_hpd_parsing
source .venv_hpd_parsing/bin/activate
python -m pip install https://paddle-model-ecology.bj.bcebos.com/paddlex/PaddleX3.0/deploy/hpd_parsing/vllm-0.17.1+hpdparsing-cp38-abi3-manylinux_2_31_x86_64.whl
```

Then run inference with the vLLM Python API:

```python
# Set the environment variable before running: export MAX_PATCHES_WITH_RESIZE=true
import base64
from vllm import LLM, SamplingParams

llm = LLM(
    model="PaddlePaddle/HPD-Parsing",
    trust_remote_code=True,
    max_model_len=16384,
    limit_mm_per_prompt={"image": 1},
    gpu_memory_utilization=0.9,
    attention_backend="FLASHINFER",
    enable_prefix_caching=True,
    speculative_config={
        "method": "medusa",
        "model": "PaddlePaddle/HPD-Parsing/P-MTP",
        "num_speculative_tokens": 6,
    },
)
sampling_params = SamplingParams(temperature=0, max_tokens=8000)

with open("demo.png", "rb") as f:
    image_base64 = base64.b64encode(f.read()).decode("utf-8")

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_base64}"}},
            {"type": "text", "text": "document parsing with fork."},
        ],
    }
]

outputs = llm.chat(messages=messages, sampling_params=sampling_params)
print(outputs[0].outputs[0].text)
```



## Inference with transformers

`model.generate_hpd(...)` reproduces the vLLM decoding paradigm in plain `transformers` (single image, batch size 1): the parent **layout** branch is decoded greedily and every `<FORK>` token spawns a **content** child branch that inherits the parent KV cache (shared-prefix reuse); the branches are then spliced back into one sequence. The `P-MTP/` head is used for speculative decoding when `use_mtp=True`.

The `P-MTP` weights ship inside the main checkpoint (keys `language_model.mtp.*`) and are loaded automatically by `from_pretrained`; `load_mtp_weights()` simply enables them. To load them from the standalone `P-MTP/` directory instead, pass its path.

Image preprocessing (dynamic tiling that mirrors vLLM's InternVL path with `MAX_PATCHES_WITH_RESIZE=true`) lives in [`image_preprocess.py`](./image_preprocess.py); `load_image` returns the stacked `pixel_values` tensor consumed by `generate_hpd`.

```python
import torch
from transformers import AutoModel, AutoTokenizer

from image_preprocess import load_image

MODEL = "PaddlePaddle/HPD-Parsing"

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
model = AutoModel.from_pretrained(
    MODEL, torch_dtype=torch.bfloat16, trust_remote_code=True,
).eval().to(DEVICE)
tokenizer = AutoTokenizer.from_pretrained(MODEL, trust_remote_code=True, use_fast=False)

model.load_mtp_weights()                       # or model.load_mtp_weights(f"{MODEL}/P-MTP")

pixel_values = load_image("test.png").to(torch.bfloat16).to(DEVICE)

response = model.generate_hpd(
    tokenizer,
    pixel_values,
    "document parsing with fork.",             # `document parsing.` for non-fork layout
    dict(max_new_tokens=8000),
    use_mtp=True,                              # P-MTP speculative decoding (parent branch)
    num_speculative_tokens=6,
    batch_children=False,                      # see note below
)
print(response)
```

Key arguments:

- `use_mtp` (default `False`) / `num_speculative_tokens` (default `6`): enable P-MTP speculative decoding with the given draft length; requires `load_mtp_weights(...)`. Greedy verification keeps output identical to plain autoregressive decoding.
- `batch_children` (default `False`): serial per-child decoding that reuses the parent KV cache in place — lowest memory and fastest in practice. Set `True` to decode all children concurrently in one left-padded batch (same output, but heavier since `transformers` duplicates the shared prefix KV per child).
- `return_token_ids` (default `False`): also return the final token id list.

> Reference implementation only. Use the vLLM path for production throughput — its paged KV cache enables true concurrent branches with zero-copy prefix sharing.

## Performance

#### 1. OmniDocBench v1.6 — Accuracy

With only 1B parameters, HPD-Parsing establishes a new state of the art among end-to-end **unified** parsers. Best results in each column are in **bold**.

<div align="center">
<img src="https://raw.githubusercontent.com/weishu20/HPD-Parsing-images/master/images/Accuracy.png" width="800"/>
</div>



#### 2. OmniDocBench v1.6 — Efficiency
Throughput under batch size 512 on NVIDIA A800 80GB with vLLM, HPD-Parsing increases throughput from 1.02 to 2.68 PPS and from 1,554.8 to 4,752.1 TPS, corresponding to improvements 2.62× and 3.06×, respectively. Despite processing approximately 4,800 input tokens per page, over four times that of DeepSeek-OCR-2, HPD-Parsing still achieves 1.31× higher PPS and 1.62× higher TPS, demonstrating strong inference efficiency under a considerably larger input-token budget.Its acceleration advantage grows with document length, reaching up to 18.04× fewer decoding steps, 3.67× higher request throughput, and 5.80× lower single-request latency in the longest output-length bucket.

<div align="center">
<img src="https://raw.githubusercontent.com/weishu20/HPD-Parsing-images/master/images/Efficiency.png" width="800"/>
</div>


## Evaluation & Benchmark

Scripts under [`eval/`](./eval) reproduce both the throughput (TPS) and the OmniDocBench v1.6 accuracy numbers via a decoupled **infer -> convert -> evaluate** pipeline:

- [`eval/benchmark_tps.py`](./eval/benchmark_tps.py) — batched vLLM inference that reports TPS metrics and dumps the raw predictions in one run.
- [`eval/hpd_to_markdown.py`](./eval/hpd_to_markdown.py) — converts the `<BLOCK>...<CHILD>...` predictions into per-page markdown for OmniDocBench's end2end evaluation.

See [`eval/README.md`](./eval/README.md) for the full workflow, including how to run the official [OmniDocBench](https://github.com/opendatalab/OmniDocBench) evaluation on the generated markdown.

## Acknowledgments

We would like to thank [InternVL](https://github.com/OpenGVLab/InternVL) and [Qwen3](https://github.com/QwenLM/Qwen3) for the backbone, [vLLM](https://github.com/vllm-project/vllm) for the serving framework, and [PaddleOCR-VL](https://github.com/PaddlePaddle/PaddleOCR), [MinerU](https://github.com/opendatalab/MinerU) and [OmniDocBench](https://github.com/opendatalab/OmniDocBench) for providing valuable data, model weights and benchmarks. We also appreciate everyone's contribution to this open-source project!

## Citation

If you find HPD-Parsing helpful, feel free to give us a star and citation.

```bibtex
@misc{wei2026hpdparsinghierarchicalparalleldocument,
      title={HPD-Parsing: Hierarchical Parallel Document Parsing}, 
      author={Shu Wei and Jingjing Wu and Lingshu Zhang and Qunyi Xie and Hao Zou and Le Xiang and Xu Fan and Yangliu Xu and Manhui Lin and Xiaolong Ma and Cheng Cui and Tengyu Du and YY},
      year={2026},
      eprint={2607.18839},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2607.18839}, 
}
```