Download ocr-tech-mps-integration/docs/EVAL_PACK.md from prosperious/ocr-projects-archive: direct link, hf CLI and curl.
- Browser
- Download file 7.13 kB
-
https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech-mps-integration/docs/EVAL_PACK.md
- Command line
-
hf download hf://prosperious/ocr-projects-archive/ocr-tech-mps-integration/docs/EVAL_PACK.md
-
curl -L -o EVAL_PACK.md https://huggingface.co/prosperious/ocr-projects-archive/resolve/main/ocr-tech-mps-integration/docs/EVAL_PACK.md
Eval Pack Workflow
The eval pack is the internal held-out document set used to test the exact target domain:
- Devanagari/Nepali OCR
- English OCR
- mixed-script lines
- multi-page mixed-script documents
- tables
- forms
- figure captions
- reading order
- noisy scanned variants
This does not replace public benchmarks. It complements public suites with the Nepali + English documents we actually care about.
Smoke Pack
A text pack exercises document reconstruction and scoring without needing OCR engines:
ocrtech create-eval-pack \
--out data/eval-pack-smoke \
--input-format text
ocrtech audit-eval-pack \
data/eval-pack-smoke/manifests/ocrtech-eval-pack.jsonl \
--out outputs/eval-pack-smoke-audit
Text packs are not OCR evidence. They are useful for validating Markdown/JSON/table/reading-order scoring.
Claim Pack
An OCR claim pack must use rendered image inputs:
ocrtech create-eval-pack \
--out data/eval-pack \
--input-format image \
--degradation clean \
--degradation scan \
--degradation blur \
--degradation low_contrast \
--degradation skew \
--count-per-template 25
ocrtech audit-eval-pack \
data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
--out outputs/eval-pack-audit \
--require-claim-ready
ocrtech audit-references \
data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
--out outputs/eval-pack-reference-audit
Image packs require Pillow:
python -m pip install -e '.[eval]'
For Devanagari rendering quality, pass a real font if auto-discovery is not good enough:
ocrtech create-eval-pack \
--out data/eval-pack \
--input-format image \
--font-path /path/to/NotoSansDevanagari-Regular.ttf
Targeted Development Packs
Use template selection for model-development packs that target a known failure pocket. These packs are useful for iteration, but they are not broad claim evidence unless they also satisfy the claim-pack coverage requirements.
ocrtech create-eval-pack \
--out data/eval-pack-table-figure-dev-v1 \
--input-format image \
--template table-ledger \
--template figure-caption \
--count-per-template 8 \
--variant-offset 100 \
--degradation clean \
--degradation scan \
--degradation blur \
--degradation low_contrast \
--degradation skew
Use --variant-offset when creating a development pack after a held-out pack, so sample IDs and visible reference text do not collide with the held-out evidence. Still check leakage explicitly before using a generated pack for training.
For span-aware table validation, use the complex-table-ledger template:
ocrtech create-eval-pack \
--out data/eval-pack-complex-table-dev-v1 \
--input-format image \
--template complex-table-ledger \
--count-per-template 8 \
--variant-offset 500 \
--degradation clean \
--degradation scan \
--degradation blur \
--degradation low_contrast
This template renders a visible rectangular table but writes structured reference cells with a merged header (colspan=2). It is useful for validating that scorers and candidate outputs preserve spans. It is still generated data, not a substitute for real scanned complex tables.
Artifacts
Generated structure:
data/eval-pack/
inputs/
references/
manifests/ocrtech-eval-pack.jsonl
eval-pack.json
eval-pack.md
Each manifest entry includes:
sample_idimage_pathsha256metadata.slicesmetadata.document_typemetadata.degradationmetadata.input_formatmetadata.reference_pathmetadata.claim_evidence_eligible
For single-page samples, image_path points to one file. For multi-page samples, image_path points to a page-bundle directory such as:
inputs/eval-multi-page-brief-000-clean/
page-0001.png
page-0002.png
Text smoke packs use the same shape for multi-page samples, but with a document.ocr.json sidecar inside the bundle directory so the deterministic sidecar engine can reconstruct the full document.
Structured references contain:
textreading_ordertablesas either nested grids or structured cell objects withrow,col,rowspan, andcolspanfigures- metadata
Before benchmarking, run ocrtech audit-references against the manifest. This
uses the same reference parser as scoring, so malformed reading_order, table,
or figure references fail before benchmark time. For manually verified
Gorkhapatra/Naya Nepal pages, add --require-claim-ready so every reference
must have metadata.reference_status=verified and
metadata.claim_evidence_eligible=true.
ocrtech audit-eval-pack also parses every reference and reports reference
coverage counts for text, Markdown, reading order, tables, and figures. Use the
standalone audit-references command when you need a reusable reference-only
artifact outside the generated eval-pack workflow.
For image packs generated after the crop-extraction update, reference metadata also contains renderer layout metadata:
metadata.layout.pages[].lines[]metadata.layout.pages[].table_cells[]- validated pixel bboxes for non-skew degradations
Skewed samples deliberately do not expose pre-rotation bboxes. Crop extraction skips those unavailable pages instead of using invalid coordinates.
Crop Manifests
The failed full-page TrOCR table/figure branch showed that treating a whole page as one recognition target is the wrong training shape. Use crop manifests for the next recognizer branch:
ocrtech extract-crops \
data/eval-pack-table-figure-dev-v1/manifests/ocrtech-eval-pack.jsonl \
--out data/crops-table-figure-dev-v1 \
--crop-type line \
--crop-type table-cell \
--crop-type figure-caption
This writes:
data/crops-table-figure-dev-v1/
images/
manifests/crops.jsonl
crop-manifest-summary.json
crop-manifest-summary.md
Each crop manifest row is a normal OCR training row with:
- crop image path
- crop text label
- source sample id
- source image path
- source reference path
- page index
- source bbox
- crop type
text_sha256sample_sha256
If a reference lacks renderer layout metadata, extract-crops fails. That is intentional; do not infer crop coordinates from template names or visual guesses.
Benchmark
Run the same inputs against every baseline:
inputs=$(python - <<'PY'
from ocrtech.manifest import load_manifest
for entry in load_manifest("data/eval-pack/manifests/ocrtech-eval-pack.jsonl"):
print(entry.image_path)
PY
)
ocrtech benchmark $inputs \
--out outputs/benchmark \
--eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
--references-dir data/eval-pack/references \
--baselines ours,tesseract,stock-paddle,glm-ocr,paddleocr-vl
Then validate the claim:
ocrtech validate-claim \
--benchmark-report outputs/benchmark/report.json \
--eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
--train-manifest data/manifests/train.jsonl \
--config validation.json \
--out outputs/validation
Do not use --engine sidecar or OCR sidecars for claim benchmarks.