File size: 7,127 Bytes
d82db76 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 | # Eval Pack Workflow
The eval pack is the internal held-out document set used to test the exact target domain:
- Devanagari/Nepali OCR
- English OCR
- mixed-script lines
- multi-page mixed-script documents
- tables
- forms
- figure captions
- reading order
- noisy scanned variants
This does not replace public benchmarks. It complements public suites with the Nepali + English documents we actually care about.
## Smoke Pack
A text pack exercises document reconstruction and scoring without needing OCR engines:
```bash
ocrtech create-eval-pack \
--out data/eval-pack-smoke \
--input-format text
ocrtech audit-eval-pack \
data/eval-pack-smoke/manifests/ocrtech-eval-pack.jsonl \
--out outputs/eval-pack-smoke-audit
```
Text packs are not OCR evidence. They are useful for validating Markdown/JSON/table/reading-order scoring.
## Claim Pack
An OCR claim pack must use rendered image inputs:
```bash
ocrtech create-eval-pack \
--out data/eval-pack \
--input-format image \
--degradation clean \
--degradation scan \
--degradation blur \
--degradation low_contrast \
--degradation skew \
--count-per-template 25
ocrtech audit-eval-pack \
data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
--out outputs/eval-pack-audit \
--require-claim-ready
ocrtech audit-references \
data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
--out outputs/eval-pack-reference-audit
```
Image packs require Pillow:
```bash
python -m pip install -e '.[eval]'
```
For Devanagari rendering quality, pass a real font if auto-discovery is not good enough:
```bash
ocrtech create-eval-pack \
--out data/eval-pack \
--input-format image \
--font-path /path/to/NotoSansDevanagari-Regular.ttf
```
## Targeted Development Packs
Use template selection for model-development packs that target a known failure pocket. These packs are useful for iteration, but they are not broad claim evidence unless they also satisfy the claim-pack coverage requirements.
```bash
ocrtech create-eval-pack \
--out data/eval-pack-table-figure-dev-v1 \
--input-format image \
--template table-ledger \
--template figure-caption \
--count-per-template 8 \
--variant-offset 100 \
--degradation clean \
--degradation scan \
--degradation blur \
--degradation low_contrast \
--degradation skew
```
Use `--variant-offset` when creating a development pack after a held-out pack, so sample IDs and visible reference text do not collide with the held-out evidence. Still check leakage explicitly before using a generated pack for training.
For span-aware table validation, use the `complex-table-ledger` template:
```bash
ocrtech create-eval-pack \
--out data/eval-pack-complex-table-dev-v1 \
--input-format image \
--template complex-table-ledger \
--count-per-template 8 \
--variant-offset 500 \
--degradation clean \
--degradation scan \
--degradation blur \
--degradation low_contrast
```
This template renders a visible rectangular table but writes structured reference cells with a merged header (`colspan=2`). It is useful for validating that scorers and candidate outputs preserve spans. It is still generated data, not a substitute for real scanned complex tables.
## Artifacts
Generated structure:
```text
data/eval-pack/
inputs/
references/
manifests/ocrtech-eval-pack.jsonl
eval-pack.json
eval-pack.md
```
Each manifest entry includes:
- `sample_id`
- `image_path`
- `sha256`
- `metadata.slices`
- `metadata.document_type`
- `metadata.degradation`
- `metadata.input_format`
- `metadata.reference_path`
- `metadata.claim_evidence_eligible`
For single-page samples, `image_path` points to one file. For multi-page samples, `image_path` points to a page-bundle directory such as:
```text
inputs/eval-multi-page-brief-000-clean/
page-0001.png
page-0002.png
```
Text smoke packs use the same shape for multi-page samples, but with a `document.ocr.json` sidecar inside the bundle directory so the deterministic sidecar engine can reconstruct the full document.
Structured references contain:
- `text`
- `reading_order`
- `tables` as either nested grids or structured cell objects with `row`, `col`, `rowspan`, and `colspan`
- `figures`
- metadata
Before benchmarking, run `ocrtech audit-references` against the manifest. This
uses the same reference parser as scoring, so malformed `reading_order`, table,
or figure references fail before benchmark time. For manually verified
Gorkhapatra/Naya Nepal pages, add `--require-claim-ready` so every reference
must have `metadata.reference_status=verified` and
`metadata.claim_evidence_eligible=true`.
`ocrtech audit-eval-pack` also parses every reference and reports reference
coverage counts for text, Markdown, reading order, tables, and figures. Use the
standalone `audit-references` command when you need a reusable reference-only
artifact outside the generated eval-pack workflow.
For image packs generated after the crop-extraction update, reference metadata also contains renderer layout metadata:
- `metadata.layout.pages[].lines[]`
- `metadata.layout.pages[].table_cells[]`
- validated pixel bboxes for non-skew degradations
Skewed samples deliberately do not expose pre-rotation bboxes. Crop extraction skips those unavailable pages instead of using invalid coordinates.
## Crop Manifests
The failed full-page TrOCR table/figure branch showed that treating a whole page as one recognition target is the wrong training shape. Use crop manifests for the next recognizer branch:
```bash
ocrtech extract-crops \
data/eval-pack-table-figure-dev-v1/manifests/ocrtech-eval-pack.jsonl \
--out data/crops-table-figure-dev-v1 \
--crop-type line \
--crop-type table-cell \
--crop-type figure-caption
```
This writes:
```text
data/crops-table-figure-dev-v1/
images/
manifests/crops.jsonl
crop-manifest-summary.json
crop-manifest-summary.md
```
Each crop manifest row is a normal OCR training row with:
- crop image path
- crop text label
- source sample id
- source image path
- source reference path
- page index
- source bbox
- crop type
- `text_sha256`
- `sample_sha256`
If a reference lacks renderer layout metadata, `extract-crops` fails. That is intentional; do not infer crop coordinates from template names or visual guesses.
## Benchmark
Run the same inputs against every baseline:
```bash
inputs=$(python - <<'PY'
from ocrtech.manifest import load_manifest
for entry in load_manifest("data/eval-pack/manifests/ocrtech-eval-pack.jsonl"):
print(entry.image_path)
PY
)
ocrtech benchmark $inputs \
--out outputs/benchmark \
--eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
--references-dir data/eval-pack/references \
--baselines ours,tesseract,stock-paddle,glm-ocr,paddleocr-vl
```
Then validate the claim:
```bash
ocrtech validate-claim \
--benchmark-report outputs/benchmark/report.json \
--eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
--train-manifest data/manifests/train.jsonl \
--config validation.json \
--out outputs/validation
```
Do not use `--engine sidecar` or OCR sidecars for claim benchmarks.
|