prosperious's picture
Add 1000 files (ocr-tech/tests/test_audit_limbu_book_missed_line_review_coverage.py ...)
d82db76 verified
|
Raw History Blame Contribute Delete
7.13 kB

Eval Pack Workflow

The eval pack is the internal held-out document set used to test the exact target domain:

  • Devanagari/Nepali OCR
  • English OCR
  • mixed-script lines
  • multi-page mixed-script documents
  • tables
  • forms
  • figure captions
  • reading order
  • noisy scanned variants

This does not replace public benchmarks. It complements public suites with the Nepali + English documents we actually care about.

Smoke Pack

A text pack exercises document reconstruction and scoring without needing OCR engines:

ocrtech create-eval-pack \
  --out data/eval-pack-smoke \
  --input-format text

ocrtech audit-eval-pack \
  data/eval-pack-smoke/manifests/ocrtech-eval-pack.jsonl \
  --out outputs/eval-pack-smoke-audit

Text packs are not OCR evidence. They are useful for validating Markdown/JSON/table/reading-order scoring.

Claim Pack

An OCR claim pack must use rendered image inputs:

ocrtech create-eval-pack \
  --out data/eval-pack \
  --input-format image \
  --degradation clean \
  --degradation scan \
  --degradation blur \
  --degradation low_contrast \
  --degradation skew \
  --count-per-template 25

ocrtech audit-eval-pack \
  data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
  --out outputs/eval-pack-audit \
  --require-claim-ready

ocrtech audit-references \
  data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
  --out outputs/eval-pack-reference-audit

Image packs require Pillow:

python -m pip install -e '.[eval]'

For Devanagari rendering quality, pass a real font if auto-discovery is not good enough:

ocrtech create-eval-pack \
  --out data/eval-pack \
  --input-format image \
  --font-path /path/to/NotoSansDevanagari-Regular.ttf

Targeted Development Packs

Use template selection for model-development packs that target a known failure pocket. These packs are useful for iteration, but they are not broad claim evidence unless they also satisfy the claim-pack coverage requirements.

ocrtech create-eval-pack \
  --out data/eval-pack-table-figure-dev-v1 \
  --input-format image \
  --template table-ledger \
  --template figure-caption \
  --count-per-template 8 \
  --variant-offset 100 \
  --degradation clean \
  --degradation scan \
  --degradation blur \
  --degradation low_contrast \
  --degradation skew

Use --variant-offset when creating a development pack after a held-out pack, so sample IDs and visible reference text do not collide with the held-out evidence. Still check leakage explicitly before using a generated pack for training.

For span-aware table validation, use the complex-table-ledger template:

ocrtech create-eval-pack \
  --out data/eval-pack-complex-table-dev-v1 \
  --input-format image \
  --template complex-table-ledger \
  --count-per-template 8 \
  --variant-offset 500 \
  --degradation clean \
  --degradation scan \
  --degradation blur \
  --degradation low_contrast

This template renders a visible rectangular table but writes structured reference cells with a merged header (colspan=2). It is useful for validating that scorers and candidate outputs preserve spans. It is still generated data, not a substitute for real scanned complex tables.

Artifacts

Generated structure:

data/eval-pack/
  inputs/
  references/
  manifests/ocrtech-eval-pack.jsonl
  eval-pack.json
  eval-pack.md

Each manifest entry includes:

  • sample_id
  • image_path
  • sha256
  • metadata.slices
  • metadata.document_type
  • metadata.degradation
  • metadata.input_format
  • metadata.reference_path
  • metadata.claim_evidence_eligible

For single-page samples, image_path points to one file. For multi-page samples, image_path points to a page-bundle directory such as:

inputs/eval-multi-page-brief-000-clean/
  page-0001.png
  page-0002.png

Text smoke packs use the same shape for multi-page samples, but with a document.ocr.json sidecar inside the bundle directory so the deterministic sidecar engine can reconstruct the full document.

Structured references contain:

  • text
  • reading_order
  • tables as either nested grids or structured cell objects with row, col, rowspan, and colspan
  • figures
  • metadata

Before benchmarking, run ocrtech audit-references against the manifest. This uses the same reference parser as scoring, so malformed reading_order, table, or figure references fail before benchmark time. For manually verified Gorkhapatra/Naya Nepal pages, add --require-claim-ready so every reference must have metadata.reference_status=verified and metadata.claim_evidence_eligible=true.

ocrtech audit-eval-pack also parses every reference and reports reference coverage counts for text, Markdown, reading order, tables, and figures. Use the standalone audit-references command when you need a reusable reference-only artifact outside the generated eval-pack workflow.

For image packs generated after the crop-extraction update, reference metadata also contains renderer layout metadata:

  • metadata.layout.pages[].lines[]
  • metadata.layout.pages[].table_cells[]
  • validated pixel bboxes for non-skew degradations

Skewed samples deliberately do not expose pre-rotation bboxes. Crop extraction skips those unavailable pages instead of using invalid coordinates.

Crop Manifests

The failed full-page TrOCR table/figure branch showed that treating a whole page as one recognition target is the wrong training shape. Use crop manifests for the next recognizer branch:

ocrtech extract-crops \
  data/eval-pack-table-figure-dev-v1/manifests/ocrtech-eval-pack.jsonl \
  --out data/crops-table-figure-dev-v1 \
  --crop-type line \
  --crop-type table-cell \
  --crop-type figure-caption

This writes:

data/crops-table-figure-dev-v1/
  images/
  manifests/crops.jsonl
  crop-manifest-summary.json
  crop-manifest-summary.md

Each crop manifest row is a normal OCR training row with:

  • crop image path
  • crop text label
  • source sample id
  • source image path
  • source reference path
  • page index
  • source bbox
  • crop type
  • text_sha256
  • sample_sha256

If a reference lacks renderer layout metadata, extract-crops fails. That is intentional; do not infer crop coordinates from template names or visual guesses.

Benchmark

Run the same inputs against every baseline:

inputs=$(python - <<'PY'
from ocrtech.manifest import load_manifest
for entry in load_manifest("data/eval-pack/manifests/ocrtech-eval-pack.jsonl"):
    print(entry.image_path)
PY
)

ocrtech benchmark $inputs \
  --out outputs/benchmark \
  --eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
  --references-dir data/eval-pack/references \
  --baselines ours,tesseract,stock-paddle,glm-ocr,paddleocr-vl

Then validate the claim:

ocrtech validate-claim \
  --benchmark-report outputs/benchmark/report.json \
  --eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
  --train-manifest data/manifests/train.jsonl \
  --config validation.json \
  --out outputs/validation

Do not use --engine sidecar or OCR sidecars for claim benchmarks.