# bench/ — PDF processing pipeline evaluation set

This directory is the **canonical test set** for evaluating the end-to-end PDF
processing pipeline (layout → OCR → markdown / structured text). It bundles
two complementary, pre-sampled subsets so that runs are reproducible and
cheap to iterate on.

| Subset | PDFs | Source benchmark | Focus |
|---|---:|---|---|
| [`olmocr_bench_50/`](./olmocr_bench_50) | 50 | [olmOCR-bench](https://huggingface.co/datasets/allenai/olmOCR-bench) | Fine-grained unit tests on text presence / absence, reading order, tables, math |
| [`omnidocbench_100/`](./omnidocbench_100) | 100 | [OmniDocBench](https://github.com/opendatalab/OmniDocBench) | Holistic document-level eval with layout / language / special-issue coverage |

Total footprint: ~108 MB, 150 PDFs.

## Subset details

### `olmocr_bench_50/`
Stratified sample drawn from the 1,403-PDF olmOCR-bench with the script
`scripts/sample_olmocr_subset.py` (seed `20260411`). Covers all 7 document
sources with a minimum floor of 3 PDFs per category plus largest-remainder
proportional allocation, and diversifies by source document inside each
category (at most one page per arXiv paper / scan ID before any repeat).

```
olmocr_bench_50/
├── pdfs/
│   ├── arxiv_math/         (14)
│   ├── headers_footers/    (8)
│   ├── long_tiny_text/     (4)
│   ├── multi_column/       (8)
│   ├── old_scans/          (5)
│   ├── old_scans_math/     (4)
│   └── tables/             (7)
├── subset_tests.jsonl      # 283 olmOCR-bench unit tests for these 50 PDFs
└── subset_manifest.json    # seed, quotas, selected file list, source bench_dir
```

The `subset_tests.jsonl` file is a filtered copy of the original per-category
`*.jsonl` test files merged into one; each row keeps the exact schema used by
the upstream olmOCR-bench evaluator (`pdf`, `type`, `max_diffs`, `checked`,
and type-specific fields like `math`, `cell`, `before`/`after`, …).

Regenerate or resize:
```bash
python3 scripts/sample_olmocr_subset.py --target 50             # default → bench/olmocr_bench_50
python3 scripts/sample_olmocr_subset.py --target 100 --seed 42  # alt subset
python3 scripts/sample_olmocr_subset.py --dry-run               # plan only
```

### `omnidocbench_100/`
Pre-built 100-PDF subset of OmniDocBench v2 with full stratified coverage
across every categorical axis in the upstream dataset.

```
omnidocbench_100/
├── pdfs/                   # 100 single-page PDFs
├── img/                    # matching rendered JPGs (1 per PDF)
├── subset_100.json         # full OmniDocBench annotations for the 100 samples
├── subset_100_stats.json   # coverage & distribution stats vs. full 981-doc set
├── subset_100_pdfs.txt     # flat list of selected PDF filenames
└── subset_100_images.txt   # flat list of selected image filenames
```

Coverage (from `subset_100_stats.json`) — every bucket of every axis is hit:
- **data_source** 9/9 · **language** 3/3 · **layout** 5/5
- **special_issue** 13/13 · **stratum** 67/67

## Using the bench

These two subsets are intended to be run as a pair — olmOCR-bench gives you
sharp per-feature pass/fail signals and OmniDocBench gives you an aggregate
quality score across real-world document types. For each new pipeline
version, run both subsets, record per-subset metrics, and diff against the
previous run.

Common entry points (to be wired up by the pipeline evaluator):

```text
bench/olmocr_bench_50/pdfs/**/*.pdf      # inputs
bench/olmocr_bench_50/subset_tests.jsonl # ground truth unit tests

bench/omnidocbench_100/pdfs/*.pdf        # inputs
bench/omnidocbench_100/subset_100.json   # ground truth annotations
```

Do **not** manually edit files under `bench/`. Regenerate with the sampling
script (for olmocr) or re-export from the upstream builder (for omnidoc) so
results stay reproducible.