ocr-tech-k2 / docs /EVAL_PACK.md
prosperious's picture
Add 51 files (.gitignore ...)
1c43956 verified
|
Raw History Blame Contribute Delete
6.32 kB
# Eval Pack Workflow
The eval pack is the internal held-out document set used to test the exact target domain:
- Devanagari/Nepali OCR
- English OCR
- mixed-script lines
- multi-page mixed-script documents
- tables
- forms
- figure captions
- reading order
- noisy scanned variants
This does not replace public benchmarks. It complements public suites with the Nepali + English documents we actually care about.
## Smoke Pack
A text pack exercises document reconstruction and scoring without needing OCR engines:
```bash
ocrtech create-eval-pack \
--out data/eval-pack-smoke \
--input-format text
ocrtech audit-eval-pack \
data/eval-pack-smoke/manifests/ocrtech-eval-pack.jsonl \
--out outputs/eval-pack-smoke-audit
```
Text packs are not OCR evidence. They are useful for validating Markdown/JSON/table/reading-order scoring.
## Claim Pack
An OCR claim pack must use rendered image inputs:
```bash
ocrtech create-eval-pack \
--out data/eval-pack \
--input-format image \
--degradation clean \
--degradation scan \
--degradation blur \
--degradation low_contrast \
--degradation skew \
--count-per-template 25
ocrtech audit-eval-pack \
data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
--out outputs/eval-pack-audit \
--require-claim-ready
```
Image packs require Pillow:
```bash
python -m pip install -e '.[eval]'
```
For Devanagari rendering quality, pass a real font if auto-discovery is not good enough:
```bash
ocrtech create-eval-pack \
--out data/eval-pack \
--input-format image \
--font-path /path/to/NotoSansDevanagari-Regular.ttf
```
## Targeted Development Packs
Use template selection for model-development packs that target a known failure pocket. These packs are useful for iteration, but they are not broad claim evidence unless they also satisfy the claim-pack coverage requirements.
```bash
ocrtech create-eval-pack \
--out data/eval-pack-table-figure-dev-v1 \
--input-format image \
--template table-ledger \
--template figure-caption \
--count-per-template 8 \
--variant-offset 100 \
--degradation clean \
--degradation scan \
--degradation blur \
--degradation low_contrast \
--degradation skew
```
Use `--variant-offset` when creating a development pack after a held-out pack, so sample IDs and visible reference text do not collide with the held-out evidence. Still check leakage explicitly before using a generated pack for training.
For span-aware table validation, use the `complex-table-ledger` template:
```bash
ocrtech create-eval-pack \
--out data/eval-pack-complex-table-dev-v1 \
--input-format image \
--template complex-table-ledger \
--count-per-template 8 \
--variant-offset 500 \
--degradation clean \
--degradation scan \
--degradation blur \
--degradation low_contrast
```
This template renders a visible rectangular table but writes structured reference cells with a merged header (`colspan=2`). It is useful for validating that scorers and candidate outputs preserve spans. It is still generated data, not a substitute for real scanned complex tables.
## Artifacts
Generated structure:
```text
data/eval-pack/
inputs/
references/
manifests/ocrtech-eval-pack.jsonl
eval-pack.json
eval-pack.md
```
Each manifest entry includes:
- `sample_id`
- `image_path`
- `sha256`
- `metadata.slices`
- `metadata.document_type`
- `metadata.degradation`
- `metadata.input_format`
- `metadata.reference_path`
- `metadata.claim_evidence_eligible`
For single-page samples, `image_path` points to one file. For multi-page samples, `image_path` points to a page-bundle directory such as:
```text
inputs/eval-multi-page-brief-000-clean/
page-0001.png
page-0002.png
```
Text smoke packs use the same shape for multi-page samples, but with a `document.ocr.json` sidecar inside the bundle directory so the deterministic sidecar engine can reconstruct the full document.
Structured references contain:
- `text`
- `reading_order`
- `tables` as either nested grids or structured cell objects with `row`, `col`, `rowspan`, and `colspan`
- `figures`
- metadata
For image packs generated after the crop-extraction update, reference metadata also contains renderer layout metadata:
- `metadata.layout.pages[].lines[]`
- `metadata.layout.pages[].table_cells[]`
- validated pixel bboxes for non-skew degradations
Skewed samples deliberately do not expose pre-rotation bboxes. Crop extraction skips those unavailable pages instead of using invalid coordinates.
## Crop Manifests
The failed full-page TrOCR table/figure branch showed that treating a whole page as one recognition target is the wrong training shape. Use crop manifests for the next recognizer branch:
```bash
ocrtech extract-crops \
data/eval-pack-table-figure-dev-v1/manifests/ocrtech-eval-pack.jsonl \
--out data/crops-table-figure-dev-v1 \
--crop-type line \
--crop-type table-cell \
--crop-type figure-caption
```
This writes:
```text
data/crops-table-figure-dev-v1/
images/
manifests/crops.jsonl
crop-manifest-summary.json
crop-manifest-summary.md
```
Each crop manifest row is a normal OCR training row with:
- crop image path
- crop text label
- source sample id
- source image path
- source reference path
- page index
- source bbox
- crop type
- `text_sha256`
- `sample_sha256`
If a reference lacks renderer layout metadata, `extract-crops` fails. That is intentional; do not infer crop coordinates from template names or visual guesses.
## Benchmark
Run the same inputs against every baseline:
```bash
inputs=$(python - <<'PY'
from ocrtech.manifest import load_manifest
for entry in load_manifest("data/eval-pack/manifests/ocrtech-eval-pack.jsonl"):
print(entry.image_path)
PY
)
ocrtech benchmark $inputs \
--out outputs/benchmark \
--eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
--references-dir data/eval-pack/references \
--baselines ours,tesseract,stock-paddle,glm-ocr,paddleocr-vl
```
Then validate the claim:
```bash
ocrtech validate-claim \
--benchmark-report outputs/benchmark/report.json \
--eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
--train-manifest data/manifests/train.jsonl \
--config validation.json \
--out outputs/validation
```
Do not use `--engine sidecar` or OCR sidecars for claim benchmarks.