File size: 7,127 Bytes
f9bc0c1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
# Eval Pack Workflow

The eval pack is the internal held-out document set used to test the exact target domain:

- Devanagari/Nepali OCR
- English OCR
- mixed-script lines
- multi-page mixed-script documents
- tables
- forms
- figure captions
- reading order
- noisy scanned variants

This does not replace public benchmarks. It complements public suites with the Nepali + English documents we actually care about.

## Smoke Pack

A text pack exercises document reconstruction and scoring without needing OCR engines:

```bash
ocrtech create-eval-pack \
  --out data/eval-pack-smoke \
  --input-format text

ocrtech audit-eval-pack \
  data/eval-pack-smoke/manifests/ocrtech-eval-pack.jsonl \
  --out outputs/eval-pack-smoke-audit
```

Text packs are not OCR evidence. They are useful for validating Markdown/JSON/table/reading-order scoring.

## Claim Pack

An OCR claim pack must use rendered image inputs:

```bash
ocrtech create-eval-pack \
  --out data/eval-pack \
  --input-format image \
  --degradation clean \
  --degradation scan \
  --degradation blur \
  --degradation low_contrast \
  --degradation skew \
  --count-per-template 25

ocrtech audit-eval-pack \
  data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
  --out outputs/eval-pack-audit \
  --require-claim-ready

ocrtech audit-references \
  data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
  --out outputs/eval-pack-reference-audit
```

Image packs require Pillow:

```bash
python -m pip install -e '.[eval]'
```

For Devanagari rendering quality, pass a real font if auto-discovery is not good enough:

```bash
ocrtech create-eval-pack \
  --out data/eval-pack \
  --input-format image \
  --font-path /path/to/NotoSansDevanagari-Regular.ttf
```

## Targeted Development Packs

Use template selection for model-development packs that target a known failure pocket. These packs are useful for iteration, but they are not broad claim evidence unless they also satisfy the claim-pack coverage requirements.

```bash
ocrtech create-eval-pack \
  --out data/eval-pack-table-figure-dev-v1 \
  --input-format image \
  --template table-ledger \
  --template figure-caption \
  --count-per-template 8 \
  --variant-offset 100 \
  --degradation clean \
  --degradation scan \
  --degradation blur \
  --degradation low_contrast \
  --degradation skew
```

Use `--variant-offset` when creating a development pack after a held-out pack, so sample IDs and visible reference text do not collide with the held-out evidence. Still check leakage explicitly before using a generated pack for training.

For span-aware table validation, use the `complex-table-ledger` template:

```bash
ocrtech create-eval-pack \
  --out data/eval-pack-complex-table-dev-v1 \
  --input-format image \
  --template complex-table-ledger \
  --count-per-template 8 \
  --variant-offset 500 \
  --degradation clean \
  --degradation scan \
  --degradation blur \
  --degradation low_contrast
```

This template renders a visible rectangular table but writes structured reference cells with a merged header (`colspan=2`). It is useful for validating that scorers and candidate outputs preserve spans. It is still generated data, not a substitute for real scanned complex tables.

## Artifacts

Generated structure:

```text
data/eval-pack/
  inputs/
  references/
  manifests/ocrtech-eval-pack.jsonl
  eval-pack.json
  eval-pack.md
```

Each manifest entry includes:

- `sample_id`
- `image_path`
- `sha256`
- `metadata.slices`
- `metadata.document_type`
- `metadata.degradation`
- `metadata.input_format`
- `metadata.reference_path`
- `metadata.claim_evidence_eligible`

For single-page samples, `image_path` points to one file. For multi-page samples, `image_path` points to a page-bundle directory such as:

```text
inputs/eval-multi-page-brief-000-clean/
  page-0001.png
  page-0002.png
```

Text smoke packs use the same shape for multi-page samples, but with a `document.ocr.json` sidecar inside the bundle directory so the deterministic sidecar engine can reconstruct the full document.

Structured references contain:

- `text`
- `reading_order`
- `tables` as either nested grids or structured cell objects with `row`, `col`, `rowspan`, and `colspan`
- `figures`
- metadata

Before benchmarking, run `ocrtech audit-references` against the manifest. This
uses the same reference parser as scoring, so malformed `reading_order`, table,
or figure references fail before benchmark time. For manually verified
Gorkhapatra/Naya Nepal pages, add `--require-claim-ready` so every reference
must have `metadata.reference_status=verified` and
`metadata.claim_evidence_eligible=true`.

`ocrtech audit-eval-pack` also parses every reference and reports reference
coverage counts for text, Markdown, reading order, tables, and figures. Use the
standalone `audit-references` command when you need a reusable reference-only
artifact outside the generated eval-pack workflow.

For image packs generated after the crop-extraction update, reference metadata also contains renderer layout metadata:

- `metadata.layout.pages[].lines[]`
- `metadata.layout.pages[].table_cells[]`
- validated pixel bboxes for non-skew degradations

Skewed samples deliberately do not expose pre-rotation bboxes. Crop extraction skips those unavailable pages instead of using invalid coordinates.

## Crop Manifests

The failed full-page TrOCR table/figure branch showed that treating a whole page as one recognition target is the wrong training shape. Use crop manifests for the next recognizer branch:

```bash
ocrtech extract-crops \
  data/eval-pack-table-figure-dev-v1/manifests/ocrtech-eval-pack.jsonl \
  --out data/crops-table-figure-dev-v1 \
  --crop-type line \
  --crop-type table-cell \
  --crop-type figure-caption
```

This writes:

```text
data/crops-table-figure-dev-v1/
  images/
  manifests/crops.jsonl
  crop-manifest-summary.json
  crop-manifest-summary.md
```

Each crop manifest row is a normal OCR training row with:

- crop image path
- crop text label
- source sample id
- source image path
- source reference path
- page index
- source bbox
- crop type
- `text_sha256`
- `sample_sha256`

If a reference lacks renderer layout metadata, `extract-crops` fails. That is intentional; do not infer crop coordinates from template names or visual guesses.

## Benchmark

Run the same inputs against every baseline:

```bash
inputs=$(python - <<'PY'
from ocrtech.manifest import load_manifest
for entry in load_manifest("data/eval-pack/manifests/ocrtech-eval-pack.jsonl"):
    print(entry.image_path)
PY
)

ocrtech benchmark $inputs \
  --out outputs/benchmark \
  --eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
  --references-dir data/eval-pack/references \
  --baselines ours,tesseract,stock-paddle,glm-ocr,paddleocr-vl
```

Then validate the claim:

```bash
ocrtech validate-claim \
  --benchmark-report outputs/benchmark/report.json \
  --eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
  --train-manifest data/manifests/train.jsonl \
  --config validation.json \
  --out outputs/validation
```

Do not use `--engine sidecar` or OCR sidecars for claim benchmarks.