Add 1000 files (ocr-tech/tests/test_audit_limbu_book_missed_line_review_coverage.py ...)
d82db76 verified |
Download ocr-tech-mps-integration/docs/EVAL_PACK.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 7.13 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech-mps-integration/docs/EVAL_PACK.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech-mps-integration/docs/EVAL_PACK.md
-
curl -L -o EVAL_PACK.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech-mps-integration/docs/EVAL_PACK.md
7.13 kB
| # Eval Pack Workflow | |
| The eval pack is the internal held-out document set used to test the exact target domain: | |
| - Devanagari/Nepali OCR | |
| - English OCR | |
| - mixed-script lines | |
| - multi-page mixed-script documents | |
| - tables | |
| - forms | |
| - figure captions | |
| - reading order | |
| - noisy scanned variants | |
| This does not replace public benchmarks. It complements public suites with the Nepali + English documents we actually care about. | |
| ## Smoke Pack | |
| A text pack exercises document reconstruction and scoring without needing OCR engines: | |
| ```bash | |
| ocrtech create-eval-pack \ | |
| --out data/eval-pack-smoke \ | |
| --input-format text | |
| ocrtech audit-eval-pack \ | |
| data/eval-pack-smoke/manifests/ocrtech-eval-pack.jsonl \ | |
| --out outputs/eval-pack-smoke-audit | |
| ``` | |
| Text packs are not OCR evidence. They are useful for validating Markdown/JSON/table/reading-order scoring. | |
| ## Claim Pack | |
| An OCR claim pack must use rendered image inputs: | |
| ```bash | |
| ocrtech create-eval-pack \ | |
| --out data/eval-pack \ | |
| --input-format image \ | |
| --degradation clean \ | |
| --degradation scan \ | |
| --degradation blur \ | |
| --degradation low_contrast \ | |
| --degradation skew \ | |
| --count-per-template 25 | |
| ocrtech audit-eval-pack \ | |
| data/eval-pack/manifests/ocrtech-eval-pack.jsonl \ | |
| --out outputs/eval-pack-audit \ | |
| --require-claim-ready | |
| ocrtech audit-references \ | |
| data/eval-pack/manifests/ocrtech-eval-pack.jsonl \ | |
| --out outputs/eval-pack-reference-audit | |
| ``` | |
| Image packs require Pillow: | |
| ```bash | |
| python -m pip install -e '.[eval]' | |
| ``` | |
| For Devanagari rendering quality, pass a real font if auto-discovery is not good enough: | |
| ```bash | |
| ocrtech create-eval-pack \ | |
| --out data/eval-pack \ | |
| --input-format image \ | |
| --font-path /path/to/NotoSansDevanagari-Regular.ttf | |
| ``` | |
| ## Targeted Development Packs | |
| Use template selection for model-development packs that target a known failure pocket. These packs are useful for iteration, but they are not broad claim evidence unless they also satisfy the claim-pack coverage requirements. | |
| ```bash | |
| ocrtech create-eval-pack \ | |
| --out data/eval-pack-table-figure-dev-v1 \ | |
| --input-format image \ | |
| --template table-ledger \ | |
| --template figure-caption \ | |
| --count-per-template 8 \ | |
| --variant-offset 100 \ | |
| --degradation clean \ | |
| --degradation scan \ | |
| --degradation blur \ | |
| --degradation low_contrast \ | |
| --degradation skew | |
| ``` | |
| Use `--variant-offset` when creating a development pack after a held-out pack, so sample IDs and visible reference text do not collide with the held-out evidence. Still check leakage explicitly before using a generated pack for training. | |
| For span-aware table validation, use the `complex-table-ledger` template: | |
| ```bash | |
| ocrtech create-eval-pack \ | |
| --out data/eval-pack-complex-table-dev-v1 \ | |
| --input-format image \ | |
| --template complex-table-ledger \ | |
| --count-per-template 8 \ | |
| --variant-offset 500 \ | |
| --degradation clean \ | |
| --degradation scan \ | |
| --degradation blur \ | |
| --degradation low_contrast | |
| ``` | |
| This template renders a visible rectangular table but writes structured reference cells with a merged header (`colspan=2`). It is useful for validating that scorers and candidate outputs preserve spans. It is still generated data, not a substitute for real scanned complex tables. | |
| ## Artifacts | |
| Generated structure: | |
| ```text | |
| data/eval-pack/ | |
| inputs/ | |
| references/ | |
| manifests/ocrtech-eval-pack.jsonl | |
| eval-pack.json | |
| eval-pack.md | |
| ``` | |
| Each manifest entry includes: | |
| - `sample_id` | |
| - `image_path` | |
| - `sha256` | |
| - `metadata.slices` | |
| - `metadata.document_type` | |
| - `metadata.degradation` | |
| - `metadata.input_format` | |
| - `metadata.reference_path` | |
| - `metadata.claim_evidence_eligible` | |
| For single-page samples, `image_path` points to one file. For multi-page samples, `image_path` points to a page-bundle directory such as: | |
| ```text | |
| inputs/eval-multi-page-brief-000-clean/ | |
| page-0001.png | |
| page-0002.png | |
| ``` | |
| Text smoke packs use the same shape for multi-page samples, but with a `document.ocr.json` sidecar inside the bundle directory so the deterministic sidecar engine can reconstruct the full document. | |
| Structured references contain: | |
| - `text` | |
| - `reading_order` | |
| - `tables` as either nested grids or structured cell objects with `row`, `col`, `rowspan`, and `colspan` | |
| - `figures` | |
| - metadata | |
| Before benchmarking, run `ocrtech audit-references` against the manifest. This | |
| uses the same reference parser as scoring, so malformed `reading_order`, table, | |
| or figure references fail before benchmark time. For manually verified | |
| Gorkhapatra/Naya Nepal pages, add `--require-claim-ready` so every reference | |
| must have `metadata.reference_status=verified` and | |
| `metadata.claim_evidence_eligible=true`. | |
| `ocrtech audit-eval-pack` also parses every reference and reports reference | |
| coverage counts for text, Markdown, reading order, tables, and figures. Use the | |
| standalone `audit-references` command when you need a reusable reference-only | |
| artifact outside the generated eval-pack workflow. | |
| For image packs generated after the crop-extraction update, reference metadata also contains renderer layout metadata: | |
| - `metadata.layout.pages[].lines[]` | |
| - `metadata.layout.pages[].table_cells[]` | |
| - validated pixel bboxes for non-skew degradations | |
| Skewed samples deliberately do not expose pre-rotation bboxes. Crop extraction skips those unavailable pages instead of using invalid coordinates. | |
| ## Crop Manifests | |
| The failed full-page TrOCR table/figure branch showed that treating a whole page as one recognition target is the wrong training shape. Use crop manifests for the next recognizer branch: | |
| ```bash | |
| ocrtech extract-crops \ | |
| data/eval-pack-table-figure-dev-v1/manifests/ocrtech-eval-pack.jsonl \ | |
| --out data/crops-table-figure-dev-v1 \ | |
| --crop-type line \ | |
| --crop-type table-cell \ | |
| --crop-type figure-caption | |
| ``` | |
| This writes: | |
| ```text | |
| data/crops-table-figure-dev-v1/ | |
| images/ | |
| manifests/crops.jsonl | |
| crop-manifest-summary.json | |
| crop-manifest-summary.md | |
| ``` | |
| Each crop manifest row is a normal OCR training row with: | |
| - crop image path | |
| - crop text label | |
| - source sample id | |
| - source image path | |
| - source reference path | |
| - page index | |
| - source bbox | |
| - crop type | |
| - `text_sha256` | |
| - `sample_sha256` | |
| If a reference lacks renderer layout metadata, `extract-crops` fails. That is intentional; do not infer crop coordinates from template names or visual guesses. | |
| ## Benchmark | |
| Run the same inputs against every baseline: | |
| ```bash | |
| inputs=$(python - <<'PY' | |
| from ocrtech.manifest import load_manifest | |
| for entry in load_manifest("data/eval-pack/manifests/ocrtech-eval-pack.jsonl"): | |
| print(entry.image_path) | |
| PY | |
| ) | |
| ocrtech benchmark $inputs \ | |
| --out outputs/benchmark \ | |
| --eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \ | |
| --references-dir data/eval-pack/references \ | |
| --baselines ours,tesseract,stock-paddle,glm-ocr,paddleocr-vl | |
| ``` | |
| Then validate the claim: | |
| ```bash | |
| ocrtech validate-claim \ | |
| --benchmark-report outputs/benchmark/report.json \ | |
| --eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \ | |
| --train-manifest data/manifests/train.jsonl \ | |
| --config validation.json \ | |
| --out outputs/validation | |
| ``` | |
| Do not use `--engine sidecar` or OCR sidecars for claim benchmarks. | |