File size: 3,406 Bytes
7325252 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | # HPD-Parsing Evaluation & Benchmark
Scripts to reproduce the HPD-Parsing **throughput (TPS)** numbers and the **OmniDocBench v1.6** accuracy, using the customized vLLM build (`<FORK>`/`<CHILD>` hierarchical parallel decoding + P-MTP speculative decoding).
```
eval/
├── benchmark_tps.py # vLLM batch inference: measures TPS and dumps raw predictions
└── hpd_to_markdown.py # converts <BLOCK>...<CHILD>... predictions to per-page markdown
```
The pipeline is decoupled into three steps: **infer → convert → evaluate**. A single inference run produces both the speed metrics and the prediction file that feeds the accuracy evaluation.
## Prerequisites
- The customized vLLM build that supports `<FORK>`/`<CHILD>` decoding and `RepetitionDetectionParams` (see the main [README](../README.md) install section).
- The model weights `PaddlePaddle/HPD-Parsing` (including the `P-MTP/` speculative head).
- The [OmniDocBench](https://github.com/opendatalab/OmniDocBench) evaluation dataset (`OmniDocBench.json` + `images/`) and its evaluation suite (for step 3).
## Step 1 — Throughput benchmark + dump predictions
`benchmark_tps.py` runs batched inference over an image folder, times the whole batch loop with `time.perf_counter()`, and reports TPS metrics. It also writes the raw predictions used by the accuracy evaluation, so you only run the model once.
```bash
MAX_PATCHES_WITH_RESIZE=true python eval/benchmark_tps.py
```
Edit the variables at the top of `__main__` to match your setup:
- `model_path` / `model_path_medusa` — model dir and its `P-MTP/` head (default `PaddlePaddle/HPD-Parsing/`).
- `root` — image folder (default `OmniDocBench_1_6/images/`).
- `prompt` — `document parsing with fork.` enables hierarchical parallel decoding; use `document parsing.` for standard full-page parsing.
- `batch_size` (default `512`), `max_model_len`, `max_num_seqs`, and the `speculative_config` (`num_speculative_tokens=6` for P-MTP). To measure the autoregressive baseline, remove `speculative_config`.
Outputs:
- Predictions -> `batch_512_pred_HPD-Parsing.json` (a list of `{index, img_path, pred}`), consumed by step 2.
- Metrics -> `records/<ckpt>.txt`: Total Time, Throughput (Requests/s), Input/Output/Total Tokens/s, and average tokens per request.
## Step 2 — Convert predictions to markdown
`hpd_to_markdown.py` parses each `<BLOCK> <type> [bbox] <CHILD> <content>` prediction into a reading-order markdown file named after the source image (`<image_stem>.md`), which is the input format OmniDocBench's end2end evaluation expects.
```bash
python eval/hpd_to_markdown.py \
--input batch_512_pred_HPD-Parsing.json \
--out-md pred_md/HPD-Parsing/
```
## Step 3 — Run OmniDocBench end2end evaluation
Use the official [OmniDocBench](https://github.com/opendatalab/OmniDocBench) suite. Point its end2end config at the markdown folder from step 2 and the ground-truth `OmniDocBench.json`, then run its evaluator to get the overall score and per-metric breakdown (text / formula / table / reading order).
```bash
git clone https://github.com/opendatalab/OmniDocBench.git
# set the prediction folder to pred_md/HPD-Parsing/ and gt to OmniDocBench.json
# in the end2end config, then run the OmniDocBench evaluation entry point
```
> The OmniDocBench evaluation code is a separate project with its own license and is intentionally not vendored here.
|