eval/README.md

# HPD-Parsing Evaluation & Benchmark

Scripts to reproduce the HPD-Parsing **throughput (TPS)** numbers and the **OmniDocBench v1.6** accuracy, using the customized vLLM build (`<FORK>`/`<CHILD>` hierarchical parallel decoding + P-MTP speculative decoding).

```
eval/
├── benchmark_tps.py     # vLLM batch inference: measures TPS and dumps raw predictions
└── hpd_to_markdown.py   # converts <BLOCK>...<CHILD>... predictions to per-page markdown
```

The pipeline is decoupled into three steps: **infer → convert → evaluate**. A single inference run produces both the speed metrics and the prediction file that feeds the accuracy evaluation.

## Prerequisites

- The customized vLLM build that supports `<FORK>`/`<CHILD>` decoding and `RepetitionDetectionParams` (see the main [README](../README.md) install section).
- The model weights `PaddlePaddle/HPD-Parsing` (including the `P-MTP/` speculative head).
- The [OmniDocBench](https://github.com/opendatalab/OmniDocBench) evaluation dataset (`OmniDocBench.json` + `images/`) and its evaluation suite (for step 3).

## Step 1 — Throughput benchmark + dump predictions

`benchmark_tps.py` runs batched inference over an image folder, times the whole batch loop with `time.perf_counter()`, and reports TPS metrics. It also writes the raw predictions used by the accuracy evaluation, so you only run the model once.

```bash
MAX_PATCHES_WITH_RESIZE=true python eval/benchmark_tps.py
```

Edit the variables at the top of `__main__` to match your setup:

- `model_path` / `model_path_medusa` — model dir and its `P-MTP/` head (default `PaddlePaddle/HPD-Parsing/`).
- `root` — image folder (default `OmniDocBench_1_6/images/`).
- `prompt` — `document parsing with fork.` enables hierarchical parallel decoding; use `document parsing.` for standard full-page parsing.
- `batch_size` (default `512`), `max_model_len`, `max_num_seqs`, and the `speculative_config` (`num_speculative_tokens=6` for P-MTP). To measure the autoregressive baseline, remove `speculative_config`.

Outputs:

- Predictions -> `batch_512_pred_HPD-Parsing.json` (a list of `{index, img_path, pred}`), consumed by step 2.
- Metrics -> `records/<ckpt>.txt`: Total Time, Throughput (Requests/s), Input/Output/Total Tokens/s, and average tokens per request.

## Step 2 — Convert predictions to markdown

`hpd_to_markdown.py` parses each `<BLOCK> <type> [bbox] <CHILD> <content>` prediction into a reading-order markdown file named after the source image (`<image_stem>.md`), which is the input format OmniDocBench's end2end evaluation expects.

```bash
python eval/hpd_to_markdown.py \
    --input batch_512_pred_HPD-Parsing.json \
    --out-md pred_md/HPD-Parsing/
```

## Step 3 — Run OmniDocBench end2end evaluation

Use the official [OmniDocBench](https://github.com/opendatalab/OmniDocBench) suite. Point its end2end config at the markdown folder from step 2 and the ground-truth `OmniDocBench.json`, then run its evaluator to get the overall score and per-metric breakdown (text / formula / table / reading order).

```bash
git clone https://github.com/opendatalab/OmniDocBench.git
# set the prediction folder to pred_md/HPD-Parsing/ and gt to OmniDocBench.json
# in the end2end config, then run the OmniDocBench evaluation entry point
```

> The OmniDocBench evaluation code is a separate project with its own license and is intentionally not vendored here.