# Eval Pack Workflow The eval pack is the internal held-out document set used to test the exact target domain: - Devanagari/Nepali OCR - English OCR - mixed-script lines - multi-page mixed-script documents - tables - forms - figure captions - reading order - noisy scanned variants This does not replace public benchmarks. It complements public suites with the Nepali + English documents we actually care about. ## Smoke Pack A text pack exercises document reconstruction and scoring without needing OCR engines: ```bash ocrtech create-eval-pack \ --out data/eval-pack-smoke \ --input-format text ocrtech audit-eval-pack \ data/eval-pack-smoke/manifests/ocrtech-eval-pack.jsonl \ --out outputs/eval-pack-smoke-audit ``` Text packs are not OCR evidence. They are useful for validating Markdown/JSON/table/reading-order scoring. ## Claim Pack An OCR claim pack must use rendered image inputs: ```bash ocrtech create-eval-pack \ --out data/eval-pack \ --input-format image \ --degradation clean \ --degradation scan \ --degradation blur \ --degradation low_contrast \ --degradation skew \ --count-per-template 25 ocrtech audit-eval-pack \ data/eval-pack/manifests/ocrtech-eval-pack.jsonl \ --out outputs/eval-pack-audit \ --require-claim-ready ocrtech audit-references \ data/eval-pack/manifests/ocrtech-eval-pack.jsonl \ --out outputs/eval-pack-reference-audit ``` Image packs require Pillow: ```bash python -m pip install -e '.[eval]' ``` For Devanagari rendering quality, pass a real font if auto-discovery is not good enough: ```bash ocrtech create-eval-pack \ --out data/eval-pack \ --input-format image \ --font-path /path/to/NotoSansDevanagari-Regular.ttf ``` ## Targeted Development Packs Use template selection for model-development packs that target a known failure pocket. These packs are useful for iteration, but they are not broad claim evidence unless they also satisfy the claim-pack coverage requirements. ```bash ocrtech create-eval-pack \ --out data/eval-pack-table-figure-dev-v1 \ --input-format image \ --template table-ledger \ --template figure-caption \ --count-per-template 8 \ --variant-offset 100 \ --degradation clean \ --degradation scan \ --degradation blur \ --degradation low_contrast \ --degradation skew ``` Use `--variant-offset` when creating a development pack after a held-out pack, so sample IDs and visible reference text do not collide with the held-out evidence. Still check leakage explicitly before using a generated pack for training. For span-aware table validation, use the `complex-table-ledger` template: ```bash ocrtech create-eval-pack \ --out data/eval-pack-complex-table-dev-v1 \ --input-format image \ --template complex-table-ledger \ --count-per-template 8 \ --variant-offset 500 \ --degradation clean \ --degradation scan \ --degradation blur \ --degradation low_contrast ``` This template renders a visible rectangular table but writes structured reference cells with a merged header (`colspan=2`). It is useful for validating that scorers and candidate outputs preserve spans. It is still generated data, not a substitute for real scanned complex tables. ## Artifacts Generated structure: ```text data/eval-pack/ inputs/ references/ manifests/ocrtech-eval-pack.jsonl eval-pack.json eval-pack.md ``` Each manifest entry includes: - `sample_id` - `image_path` - `sha256` - `metadata.slices` - `metadata.document_type` - `metadata.degradation` - `metadata.input_format` - `metadata.reference_path` - `metadata.claim_evidence_eligible` For single-page samples, `image_path` points to one file. For multi-page samples, `image_path` points to a page-bundle directory such as: ```text inputs/eval-multi-page-brief-000-clean/ page-0001.png page-0002.png ``` Text smoke packs use the same shape for multi-page samples, but with a `document.ocr.json` sidecar inside the bundle directory so the deterministic sidecar engine can reconstruct the full document. Structured references contain: - `text` - `reading_order` - `tables` as either nested grids or structured cell objects with `row`, `col`, `rowspan`, and `colspan` - `figures` - metadata Before benchmarking, run `ocrtech audit-references` against the manifest. This uses the same reference parser as scoring, so malformed `reading_order`, table, or figure references fail before benchmark time. For manually verified Gorkhapatra/Naya Nepal pages, add `--require-claim-ready` so every reference must have `metadata.reference_status=verified` and `metadata.claim_evidence_eligible=true`. `ocrtech audit-eval-pack` also parses every reference and reports reference coverage counts for text, Markdown, reading order, tables, and figures. Use the standalone `audit-references` command when you need a reusable reference-only artifact outside the generated eval-pack workflow. For image packs generated after the crop-extraction update, reference metadata also contains renderer layout metadata: - `metadata.layout.pages[].lines[]` - `metadata.layout.pages[].table_cells[]` - validated pixel bboxes for non-skew degradations Skewed samples deliberately do not expose pre-rotation bboxes. Crop extraction skips those unavailable pages instead of using invalid coordinates. ## Crop Manifests The failed full-page TrOCR table/figure branch showed that treating a whole page as one recognition target is the wrong training shape. Use crop manifests for the next recognizer branch: ```bash ocrtech extract-crops \ data/eval-pack-table-figure-dev-v1/manifests/ocrtech-eval-pack.jsonl \ --out data/crops-table-figure-dev-v1 \ --crop-type line \ --crop-type table-cell \ --crop-type figure-caption ``` This writes: ```text data/crops-table-figure-dev-v1/ images/ manifests/crops.jsonl crop-manifest-summary.json crop-manifest-summary.md ``` Each crop manifest row is a normal OCR training row with: - crop image path - crop text label - source sample id - source image path - source reference path - page index - source bbox - crop type - `text_sha256` - `sample_sha256` If a reference lacks renderer layout metadata, `extract-crops` fails. That is intentional; do not infer crop coordinates from template names or visual guesses. ## Benchmark Run the same inputs against every baseline: ```bash inputs=$(python - <<'PY' from ocrtech.manifest import load_manifest for entry in load_manifest("data/eval-pack/manifests/ocrtech-eval-pack.jsonl"): print(entry.image_path) PY ) ocrtech benchmark $inputs \ --out outputs/benchmark \ --eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \ --references-dir data/eval-pack/references \ --baselines ours,tesseract,stock-paddle,glm-ocr,paddleocr-vl ``` Then validate the claim: ```bash ocrtech validate-claim \ --benchmark-report outputs/benchmark/report.json \ --eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \ --train-manifest data/manifests/train.jsonl \ --config validation.json \ --out outputs/validation ``` Do not use `--engine sidecar` or OCR sidecars for claim benchmarks.