| # HPD-Parsing Evaluation & Benchmark |
|
|
| Scripts to reproduce the HPD-Parsing **throughput (TPS)** numbers and the **OmniDocBench v1.6** accuracy, using the customized vLLM build (`<FORK>`/`<CHILD>` hierarchical parallel decoding + P-MTP speculative decoding). |
|
|
| ``` |
| eval/ |
| ├── benchmark_tps.py # vLLM batch inference: measures TPS and dumps raw predictions |
| └── hpd_to_markdown.py # converts <BLOCK>...<CHILD>... predictions to per-page markdown |
| ``` |
|
|
| The pipeline is decoupled into three steps: **infer → convert → evaluate**. A single inference run produces both the speed metrics and the prediction file that feeds the accuracy evaluation. |
|
|
| ## Prerequisites |
|
|
| - The customized vLLM build that supports `<FORK>`/`<CHILD>` decoding and `RepetitionDetectionParams` (see the main [README](../README.md) install section). |
| - The model weights `PaddlePaddle/HPD-Parsing` (including the `P-MTP/` speculative head). |
| - The [OmniDocBench](https://github.com/opendatalab/OmniDocBench) evaluation dataset (`OmniDocBench.json` + `images/`) and its evaluation suite (for step 3). |
|
|
| ## Step 1 — Throughput benchmark + dump predictions |
|
|
| `benchmark_tps.py` runs batched inference over an image folder, times the whole batch loop with `time.perf_counter()`, and reports TPS metrics. It also writes the raw predictions used by the accuracy evaluation, so you only run the model once. |
|
|
| ```bash |
| MAX_PATCHES_WITH_RESIZE=true python eval/benchmark_tps.py |
| ``` |
|
|
| Edit the variables at the top of `__main__` to match your setup: |
|
|
| - `model_path` / `model_path_medusa` — model dir and its `P-MTP/` head (default `PaddlePaddle/HPD-Parsing/`). |
| - `root` — image folder (default `OmniDocBench_1_6/images/`). |
| - `prompt` — `document parsing with fork.` enables hierarchical parallel decoding; use `document parsing.` for standard full-page parsing. |
| - `batch_size` (default `512`), `max_model_len`, `max_num_seqs`, and the `speculative_config` (`num_speculative_tokens=6` for P-MTP). To measure the autoregressive baseline, remove `speculative_config`. |
|
|
| Outputs: |
|
|
| - Predictions -> `batch_512_pred_HPD-Parsing.json` (a list of `{index, img_path, pred}`), consumed by step 2. |
| - Metrics -> `records/<ckpt>.txt`: Total Time, Throughput (Requests/s), Input/Output/Total Tokens/s, and average tokens per request. |
|
|
| ## Step 2 — Convert predictions to markdown |
|
|
| `hpd_to_markdown.py` parses each `<BLOCK> <type> [bbox] <CHILD> <content>` prediction into a reading-order markdown file named after the source image (`<image_stem>.md`), which is the input format OmniDocBench's end2end evaluation expects. |
|
|
| ```bash |
| python eval/hpd_to_markdown.py \ |
| --input batch_512_pred_HPD-Parsing.json \ |
| --out-md pred_md/HPD-Parsing/ |
| ``` |
|
|
| ## Step 3 — Run OmniDocBench end2end evaluation |
|
|
| Use the official [OmniDocBench](https://github.com/opendatalab/OmniDocBench) suite. Point its end2end config at the markdown folder from step 2 and the ground-truth `OmniDocBench.json`, then run its evaluator to get the overall score and per-metric breakdown (text / formula / table / reading order). |
|
|
| ```bash |
| git clone https://github.com/opendatalab/OmniDocBench.git |
| # set the prediction folder to pred_md/HPD-Parsing/ and gt to OmniDocBench.json |
| # in the end2end config, then run the OmniDocBench evaluation entry point |
| ``` |
|
|
| > The OmniDocBench evaluation code is a separate project with its own license and is intentionally not vendored here. |
|
|
