| # TrueDoc on olmOCR-bench: run 97 |
|
|
| **Overall 86.8** (macro-average of the eight sections; 95% bootstrap CI 86.0-87.7), scored on |
| 2026-09-18 with the benchmark's official scorer (`olmocr` 0.4.27's `benchmark.py`) at dataset revision |
| `54a96a6fb6a2bd3b297e59869491db4d3625b711`, over all 1,403 pages and all 7,019 checks plus the 1,394 baseline checks. |
|
|
| | section | score | checks passed | |
| |---|---|---| |
| | arxiv_math | 88.6 | 2594 / 2927 | |
| | old_scans_math | 86.7 | 397 / 458 | |
| | table_tests | 89.3 | 913 / 1022 | |
| | old_scans | 58.4 | 307 / 526 | |
| | headers_footers | 96.7 | 735 / 760 | |
| | multi_column | 83.6 | 739 / 884 | |
| | long_tiny_text | 91.4 | 404 / 442 | |
| | baseline | 99.9 | 1392 / 1394 | |
| |
| ## What produced these pages |
| |
| TrueDoc is a PDF-to-markdown conversion pipeline, not a model. For this run: |
| |
| - **1,122 pages with a digital text layer** were read from the PDF's own text on CPU, with a layout model |
| (docling-layout-heron) for structure. No vision model read those pages. |
| - **281 pages with no text layer** (scans, photographs of paper) were read by **Infinity-Parser2-Flash** |
| (2.2B, Apache-2.0). The **102** of those that the pipeline's own OCR could not read (its word-likeness gate) |
| were read by **Infinity-Parser2-Pro** (35B, Apache-2.0) instead. |
| - **92 picture regions on digital pages** (pictures that hold text: a scanned table, a screenshot) were read by |
| **olmOCR-2-7B-1025** (Apache-2.0). |
| - The model readings were recorded on a rented GPU on 17 September 2026 (Infinity-Parser2 through vLLM 0.17.1 with |
| the settings of its authors' olmOCR-bench guide, raw markdown, none of their post-processing) and on 8 September |
| 2026 (olmOCR 2), and replayed from disk during the run. The readings are in the code repository. |
| - No post-processing keyed on the benchmark's category folder names. Every page was converted the same way. |
| |
| The pipeline's code is at https://github.com/islandtimer/TrueDoc, tag `run-97` (commit `9f9b77b`); its README says how to reproduce this run |
| from the committed readings, and `docs/BENCHMARKS.md` there holds every run before this one. |
| |
| ## What is in this repository |
| |
| - `outputs/<section>/<page>_pg1_repeat1.md`: the 1,403 page outputs exactly as scored, in the scorer's layout. |
| Put the folder beside the dataset's `bench_data/pdfs` and run the scorer to re-score in minutes. |
| - `scoring/summary.json`, `scoring/score_output.txt`, `scoring/failed_tests.jsonl`: the scorer's own output, the |
| bootstrap interval, and every failed check. |
| - `.eval_results/olmocrbench.yaml`: the entry the dataset's leaderboard reads. |
|
|
| ## How to read the number |
|
|
| - It is a self-reported run of the official scorer, not an independent one. Everything needed to check it is |
| here. |
| - A second scoring of the same pages moves a section by about three checks, so the tenth is not significant. |
| - A fifth of the benchmark's pages (`bench/holdout.txt` in the code repository) was never looked at while the |
| pipeline's rules were built; that fifth scores 86.0 in this run, the rest 84.8. |
| - As of 2026-09-18 this overall would rank second among the entries on the leaderboard by point estimate; the leading |
| entry's 87.6 sits inside this run's interval. A live integration of the served readers (rather than replayed |
| readings) is pending. |
|
|
