eval/README.md

HPD-Parsing Evaluation & Benchmark

Scripts to reproduce the HPD-Parsing throughput (TPS) numbers and the OmniDocBench v1.6 accuracy, using the customized vLLM build (<FORK>/<CHILD> hierarchical parallel decoding + P-MTP speculative decoding).

eval/
├── benchmark_tps.py     # vLLM batch inference: measures TPS and dumps raw predictions
└── hpd_to_markdown.py   # converts <BLOCK>...<CHILD>... predictions to per-page markdown

The pipeline is decoupled into three steps: infer → convert → evaluate. A single inference run produces both the speed metrics and the prediction file that feeds the accuracy evaluation.

Prerequisites

• The customized vLLM build that supports <FORK>/<CHILD> decoding and RepetitionDetectionParams (see the main README install section).
• The model weights PaddlePaddle/HPD-Parsing (including the P-MTP/ speculative head).
• The OmniDocBench evaluation dataset (OmniDocBench.json + images/) and its evaluation suite (for step 3).

Step 1 — Throughput benchmark + dump predictions

benchmark_tps.py runs batched inference over an image folder, times the whole batch loop with time.perf_counter(), and reports TPS metrics. It also writes the raw predictions used by the accuracy evaluation, so you only run the model once.

MAX_PATCHES_WITH_RESIZE=true python eval/benchmark_tps.py

Edit the variables at the top of __main__ to match your setup:

• model_path / model_path_medusa — model dir and its P-MTP/ head (default PaddlePaddle/HPD-Parsing/).
• root — image folder (default OmniDocBench_1_6/images/).
• prompt — document parsing with fork. enables hierarchical parallel decoding; use document parsing. for standard full-page parsing.
• batch_size (default 512), max_model_len, max_num_seqs, and the speculative_config (num_speculative_tokens=6 for P-MTP). To measure the autoregressive baseline, remove speculative_config.

Outputs:

• Predictions -> batch_512_pred_HPD-Parsing.json (a list of {index, img_path, pred}), consumed by step 2.
• Metrics -> records/<ckpt>.txt: Total Time, Throughput (Requests/s), Input/Output/Total Tokens/s, and average tokens per request.

Step 2 — Convert predictions to markdown

hpd_to_markdown.py parses each <BLOCK> <type> [bbox] <CHILD> <content> prediction into a reading-order markdown file named after the source image (<image_stem>.md), which is the input format OmniDocBench's end2end evaluation expects.

python eval/hpd_to_markdown.py \
    --input batch_512_pred_HPD-Parsing.json \
    --out-md pred_md/HPD-Parsing/

Step 3 — Run OmniDocBench end2end evaluation

Use the official OmniDocBench suite. Point its end2end config at the markdown folder from step 2 and the ground-truth OmniDocBench.json, then run its evaluator to get the overall score and per-metric breakdown (text / formula / table / reading order).

git clone https://github.com/opendatalab/OmniDocBench.git
# set the prediction folder to pred_md/HPD-Parsing/ and gt to OmniDocBench.json
# in the end2end config, then run the OmniDocBench evaluation entry point

The OmniDocBench evaluation code is a separate project with its own license and is intentionally not vendored here.