File size: 31,787 Bytes
f9bc0c1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
# Validation Design

The claim target is not "we ran OCR and it looked good." The claim target is:

```text
ocrtech is better than named baselines on held-out Nepali + English OCR and document-structure extraction, under measured gates.
```

Until the validation report says `validated`, do not claim SOTA.

Claim-grade `validation.json` files record provenance for the benchmark report,
validation config, eval manifest, train manifests, candidate model card, and
fallback model card used by `validate-claim`. `review-claim` cross-checks
those hashes against `experiment.json`, so a validation artifact from a
different benchmark, eval split, validation config, model card, fallback card,
or training-manifest set cannot be reused as portfolio evidence.

The SOTA review profile is paired with these claim-grade validation configs:

- `configs/validation-internal-sota-v1.json`
- `configs/validation-public-omnidocbench-sota-v1.json`
- `configs/validation-public-real5-sota-v1.json`

Each one covers the `required_validation_gates` declared in
`configs/claim-review-sota-v1.json`. Use the slot-specific config when running
`run-experiment` or `validate-claim`; otherwise portfolio review may have
metrics and baselines but still lack the passed gates required for the paper
claim.

The built-in fallback validation config is intentionally broader than a plain
OCR smoke check. If `validate-claim` is called without `--config`, it now gates
Nepali CER/WER, English non-regression, table cell and TEDS-text quality,
reading order, document reconstruction, and figure detection/caption quality.
That default remains smaller than the SOTA configs above, but it should not
allow a "document OCR" claim to pass on text-only evidence.

## Evidence Required

A defensible run needs all of this:

- A claim-ready image eval pack from `ocrtech create-eval-pack`, plus public benchmark runs.
- A candidate model card from `ocrtech create-model-card` and a passing `ocrtech audit-model` report.
- Training data prepared with `audit-manifest` and `split-manifest`.
- Held-out eval manifest that was never used for recognizer, corrector, threshold, or prompt tuning.
- Image/PDF eval inputs normalized to OCR-supported file suffixes before any
  claim-grade PaddleOCR/Tesseract/Surya/VLM comparison.
- For routed or thresholded candidates, a separate calibration experiment artifact that records how the threshold was chosen.
- Train manifests for leakage checks.
- Paired benchmark outputs for every baseline on the same input files.
- Per-sample metrics, not only aggregate averages.
- For `glm-ocr` and `paddleocr-vl`, a real runnable command hook that emits Markdown or `document.md`, not a placeholder env var.
- Slice coverage for Nepali, English, multi-page documents, tables, reading order, figures/charts, forms, scans, photos, skew, blur, compression, and low contrast.
- Gates that name the baseline, metric, slice, direction, required margin, and confidence requirement.
- Paired win/loss evidence, a minimum paired win-rate requirement when the
  claim needs broad per-document improvement, and a nonparametric significance
  check when the claim requires it.
- Failure-case report, not just the winning average.

## Public Benchmark Gate

Generated eval packs are development evidence. They are not enough for SOTA
wording. Before a paper claim can say "state of the art," at least one public
document parsing benchmark must be converted into an auditable manifest and run
as a normal `ocrtech run-experiment`.

Known public benchmark registry:

```bash
ocrtech list-public-benchmarks
```

Current registry entries:

- `opendatalab/OmniDocBench`: real-world document parsing benchmark, dataset page `https://huggingface.co/datasets/opendatalab/OmniDocBench`
- `allenai/olmOCR-bench`: PDF-to-Markdown benchmark with unit-test style assertions, dataset page `https://huggingface.co/datasets/allenai/olmOCR-bench`
- `PaddlePaddle/Real5-OmniDocBench`: real-world physical reconstruction benchmark with scanning, warping, screen-photo, illumination, and skew scenarios, dataset page `https://huggingface.co/datasets/PaddlePaddle/Real5-OmniDocBench`

The first implemented public converter is:

```bash
ocrtech prepare-public-benchmark opendatalab/OmniDocBench \
  --out data/public/omnidocbench \
  --limit 100 \
  --language english \
  --require-tables
```

The converter was written only after checking the actual dataset structure:
`OmniDocBench.json` contains `page_info`, `page_info.page_attribute`, and
`layout_dets`; image files resolve as `images/{page_info.image_path}`; table
ground truth is stored in table-block `html`.

Every public benchmark manifest row must include:

- `metadata.public_benchmark = true`
- `metadata.benchmark_name`
- `metadata.source_url`
- `metadata.reference_path`, unless the benchmark has a separate official scorer integration
- `metadata.slices` including `public` plus benchmark-specific slices
- an existing `image_path`
- a matching image `sha256`, when the manifest records one
- a matching `metadata.text_sha256`, when the manifest records one

Audit it before running:

```bash
ocrtech audit-public-benchmark data/public/omnidocbench/manifests/omnidocbench-eval.jsonl \
  --benchmark opendatalab/OmniDocBench \
  --out outputs/public-omnidocbench-audit
```

This audit intentionally fails if provenance, references, images, or recorded
hashes are missing or stale. Do not use `--allow-missing-reference-paths` for
claim runs unless an official external scorer is wired and documented in the
experiment report.

Portfolio review for SOTA claims should use:

```bash
ocrtech review-claim \
  --experiment outputs/internal-experiment/experiment.json \
  --experiment outputs/public-omnidocbench/experiment.json \
  --config configs/claim-review-sota-v1.json \
  --model-config outputs/figure-caption-table-lexicon-v1/model-card.json \
  --out outputs/claim-review-sota-v1
```

The config requires internal Nepali-English evidence plus public benchmark
evidence. It should fail until those public experiments actually exist.

Claim-grade validation configs can also set `min_pair_win_rate` on a gate. This
is intentionally different from the reported `win_rate`: `win_rate` is computed
over non-tied pairs, while `min_pair_win_rate` is checked as wins divided by all
paired samples. That distinction matters for document OCR claims. A result with
one win and many ties can have `win_rate = 1.0`, but it has low pair coverage
and should not satisfy a broad "better than baseline" claim unless the gate is
explicitly a no-regression/tie-acceptable gate.

The public OmniDocBench-vs-Surya validation gate is:

```bash
ocrtech validate-claim \
  --benchmark-report outputs/experiment-public-omnidocbench-smoke-v1/benchmark/report.json \
  --config configs/validation-public-omnidocbench-surya.json \
  --eval-manifest data/public/omnidocbench-smoke/manifests/omnidocbench-eval.jsonl \
  --candidate-model-config runs/recognizer/package/model-card.json \
  --fallback-model-config runs/vlm-fallback/package/model-card.json \
  --out outputs/validation-public-omnidocbench-smoke-v1
```

It requires at least `100` public document-parsing samples and at least `40`
table samples, then gates candidate-vs-Surya CER, table cell F1, structure-only
TEDS, and text-sensitive TEDS. A 5-page smoke run is allowed as engineering
evidence, but it must remain `not_validated`.

The first audited manifest sized for this gate is:

```text
data/public/omnidocbench-table100/manifests/omnidocbench-eval.jsonl
```

Its audit artifact is
`outputs/public-omnidocbench-table100-audit/public-benchmark-audit.json`; the
audit passed with `100` public/document/table English samples. The corresponding
long benchmark run is tracked under
`outputs/experiment-public-omnidocbench-table100-width-guard-v4/`. Treat that
run as incomplete until `benchmark/report.json`, `summary/summary.json`,
`validation/validation.json`, and `table-cell-analysis/table-cell-analysis.json`
exist.

The completed width-guard-v4 Table100 run is an instructive non-passing result:

- rows: `200/200`, with one shared `300s` timeout for candidate and Surya
- paired scored samples: `99`
- candidate improves public table metrics over Surya:
  `table_cell_f1` `0.484364` vs `0.406694`,
  `table_teds_structure` `0.720160` vs `0.687550`,
  `table_teds_text` `0.565284` vs `0.543987`
- candidate does not improve OCR CER:
  `0.613201` vs Surya `0.612623`
- validation status: `not_validated`

For long-run status files, distinguish package readiness from claim readiness.
`evidence_package_ready` means the expected rows and final report artifacts are
present. `paper_claim_ready` must only be true when the validation report itself
passes. A completed benchmark package with `validation.claim_status =
not_validated` is useful evidence, but not a publishable SOTA claim.

Long public runs should use benchmark checkpointing and per-page timeouts:

```bash
ocrtech benchmark ... \
  --inputs-from-manifest data/eval/manifest-with-refs.jsonl \
  --resume-existing \
  --sample-timeout-seconds 300
```

`--inputs-from-manifest` loads authoritative image paths from the manifest and
defaults `--eval-manifest` to the same file when it is not supplied. This avoids
shell-generated input lists drifting away from the manifest that contains
reference paths and slice labels. `--resume-existing` scores existing
`document.json` outputs and only runs OCR for missing pages.
`--sample-timeout-seconds` runs each OCR call behind a hard process watchdog on
platforms that support `fork`; a stalled backend is terminated and recorded as a
page-level `TimeoutError`, then the benchmark continues. Platforms without
`fork` fall back to the Python signal timer. A timeout is still a
candidate/baseline failure for validation; it is not a pass-through.
For timeout runs, the benchmark now constructs the OCR engine inside the
watchdog child process for each page. This avoids inheriting already-initialized
PaddleOCR/PaddlePaddle C++ runtime state across `fork`, which produced
`exit_code=-11` worker crashes in the first line-level Paddle composite run.

Table scoring matches each reference table to the best unused predicted table
before averaging table metrics. This matters on public document pages where an
engine may emit extra table fragments before the main table. Positional
table-index matching is too brittle for claim evidence.

When the evaluator changes, do not rerun slow OCR just to update metrics. Use:

```bash
ocrtech rescore-benchmark \
  --benchmark-report outputs/experiment-public-omnidocbench-smoke-v1/benchmark/report.json \
  --eval-manifest data/public/omnidocbench-smoke/manifests/omnidocbench-eval.jsonl \
  --out outputs/experiment-public-omnidocbench-smoke-v2-rescored/benchmark
```

The rescore command copies the existing `document.json`, Markdown, table, and
figure outputs into the new benchmark directory so downstream diagnostics such
as `analyze-table-cells` remain usable.

When evaluating table postprocessors, do not rely only on separate candidate and
baseline OCR runs. Independent OCR/layout calls can introduce run-to-run variance
that makes a small table postprocessor look better or worse than it is. Use a
fixed-source derived comparison to isolate the postprocessor:

```bash
ocrtech derive-table-postprocess-benchmark \
  --benchmark-report outputs/experiment-public-omnidocbench-table40-normalize-rows-v3/benchmark/report.json \
  --source-baseline surya \
  --candidate-baseline candidate \
  --eval-manifest data/public/omnidocbench-table40/manifests/omnidocbench-eval.jsonl \
  --out outputs/experiment-public-omnidocbench-table40-derived-postprocess-v1/benchmark
```

This command copies the chosen source baseline outputs, applies the current
reference-free table postprocessors to the copied candidate `document.json`, and
rescored both systems against the same references. The result is valid evidence
for the postprocessor contribution only. It is not an end-to-end OCR model card
and cannot support a broad OCR SOTA claim by itself.

The current normalized public candidate is
`ocrtech-surya-tesseract-figure-table-lexicon-normalize-v2`. Its table-grid
normalization is runtime-only and reference-free: empty columns, separator-only
columns, and exact duplicate adjacent columns are removed. On the 5-page
OmniDocBench smoke it improves table metrics over Surya, but the run remains
`not_validated` because the public sample count, table sample count, CER margin,
and bootstrap confidence requirements are not met.

## Reference Format

Text-only reference:

```text
page.png.ref.txt
```

Structured reference:

```json
{
  "text": "नेपाल सरकार",
  "reading_order": ["p0-l0", "p0-l1", "p0-l2"],
  "tables": [
    [
      ["नाम", "उमेर", "जिल्ला"],
      ["राम", "३२", "काठमाडौं"]
    ]
  ],
  "metadata": {
    "source": "hand_labeled"
  }
}
```

For manually reviewed Gorkhapatra language pages, `reading_order` may also use
structured objects. The scorer uses each object's `id` for ordering and keeps
the `text` as reviewer-visible reference context:

```json
{
  "reading_order": [
    {"id": "p0-l0", "type": "heading", "text": "मगर भाषा पृष्ठ"},
    {"id": "p0-l1", "type": "paragraph", "text": "मुख्य समाचार"}
  ]
}
```

Complex table references can also use cell objects with spans:

```json
{
  "tables": [
    {
      "cells": [
        {"row": 0, "col": 0, "text": "नाम", "rowspan": 1, "colspan": 2},
        {"row": 1, "col": 0, "text": "राम"},
        {"row": 1, "col": 1, "text": "१२"}
      ]
    }
  ]
}
```

The nested-list grid form remains supported. Span-aware references make `table_exact_match` and `table_structure_similarity` sensitive to missing rowspans/colspans even when flattened table text is correct.

The candidate parser can apply a conservative repeated-header span postprocessor:
adjacent identical cells in the first header rows are collapsed into a single
`colspan` cell only when the following row has distinct child columns. This lets
`document.json` and exported table HTML represent the simple merged-header
pattern in `complex-table-ledger`. Treat this as a heuristic parser repair until
it is validated on real scanned tables with varied merged cells and rowspans.

Supported structured metrics:

- `quality_score`
- `cer`
- `wer`
- `text_cer`
- `text_wer`
- `document_reconstruction_cer`
- `document_reconstruction_wer`
- `document_reconstruction_text_similarity`
- `document_reconstruction_similarity`
- `figure_detection_precision`
- `figure_detection_recall`
- `figure_detection_f1`
- `figure_caption_cer`
- `reading_order_pair_accuracy`
- `table_cell_precision`
- `table_cell_recall`
- `table_cell_f1`
- `table_exact_match`
- `table_structure_similarity`
- `table_teds_structure`
- `table_teds_text`

`document_reconstruction_text_similarity` is `1 - CER` over rendered Markdown
only. `document_reconstruction_similarity` is stricter: when the reference also
declares reading order, tables, or figures, the reconstruction score is capped
by those structure metrics. This prevents a parser from receiving full document
reconstruction credit by emitting Markdown-shaped text while failing to populate
structured `document.json` tables or figures.

## Validation Config

Example:

```json
{
  "candidate": "ours",
  "min_samples": 300,
  "required_slices": {
    "nepali": 100,
    "english": 100,
    "multi_page": 25,
    "table": 50,
    "reading_order": 50,
    "figure": 30,
    "scan": 50,
    "phone_photo": 50
  },
  "bootstrap_iterations": 1000,
  "bootstrap_confidence": 0.95,
  "require_full_eval_manifest_coverage": true,
  "gates": [
    {
      "baseline": "tesseract",
      "metric": "cer",
      "direction": "lower",
      "slice": "nepali",
      "min_pairs": 50,
      "min_relative_improvement": 0.05,
      "max_p_value": 0.05
    },
    {
      "baseline": "stock-paddle",
      "metric": "cer",
      "direction": "lower",
      "slice": "english",
      "min_absolute_margin": 0.0
    },
    {
      "baseline": "glm-ocr",
      "metric": "table_cell_f1",
      "direction": "higher",
      "slice": "table",
      "min_absolute_margin": 0.01
    },
    {
      "baseline": "surya",
      "metric": "cer",
      "direction": "lower",
      "slices": ["hard_eval", "table"],
      "min_pairs": 8,
      "min_absolute_margin": 0.02
    },
    {
      "baseline": "paddleocr-vl",
      "metric": "reading_order_pair_accuracy",
      "direction": "higher",
      "slice": "reading_order",
      "min_absolute_margin": 0.01
    }
  ]
}
```

Use `slice` for a single required slice. Use `slices` when a gate must apply to the intersection of multiple labels, for example `["hard_eval", "table"]` for table samples inside a hard-eval pocket. Do not approximate this with separate `hard_eval` and `table` gates; that allows unrelated samples to satisfy the table gate.

For claim-grade runs, set `require_full_eval_manifest_coverage` to `true`.
When enabled, `validate-claim` fails unless the candidate and every baseline
named by the configured gates have a successful row for every sample in
`--eval-manifest`. This prevents a validation report from passing on an easy
subset while silently omitting hard pages from the held-out manifest.

Run:

```bash
ocrtech preflight-claim \
  --eval-manifest data/manifests/eval.jsonl \
  --references-dir data/references \
  --baselines candidate,tesseract,stock-paddle,glm-ocr,paddleocr-vl \
  --candidate-model-config runs/recognizer/package/model-card.json \
  --validation-config validation.json \
  --train-manifest data/manifests/train.jsonl \
  --out outputs/preflight

ocrtech benchmark \
  --eval-manifest data/manifests/eval.jsonl \
  --references-dir data/references \
  --baselines ours,tesseract,stock-paddle,glm-ocr,paddleocr-vl \
  --out outputs/benchmark \
  $(cut -f4 data/manifests/eval.inputs)

ocrtech validate-claim \
  --benchmark-report outputs/benchmark/report.json \
  --config validation.json \
  --eval-manifest data/manifests/eval.jsonl \
  --train-manifest data/manifests/train.jsonl \
  --out outputs/validation
```

`preflight-claim` should be the first command in a claim-grade workflow. If it fails, the benchmark run is not ready.

For explicit `--validation-config` runs, preflight now checks the config against
the requested baselines and eval manifest before any OCR starts. It fails when
the candidate label is absent from `--baselines`, a gate baseline is not
scheduled, `min_samples` exceeds the manifest size, required slices are missing,
or a gate's `min_pairs` exceeds the number of manifest rows matching the gate
slice intersection. Omitting `--validation-config` is only a warning; it is not
claim-grade evidence.

For hybrid runs with `--fallback-model-config`, preflight also checks that the
candidate model card declares the same fallback under
`provenance.fallback_model_card`. This catches a wrong runtime fallback flag
before a benchmark produces evidence for a different composite engine than the
one described by the candidate model card.

If the candidate model uses `quality_select_composite`, `preflight-claim` now checks that the model card contains calibration provenance and that the calibration eval manifest hash is different from the final claim eval manifest hash. Tuning on the same eval pack invalidates the claim path.

`validate-claim` exits with `0` only when the configured claim is validated. It exits with `1` when evidence is missing or a gate fails.

Each gate now reports:

- paired sample count
- win / loss / tie counts
- win rate on non-tied pairs
- bootstrap CI for mean paired improvement
- optional one-sided sign-test p-value

For claim-grade runs, use `min_pairs` and `max_p_value` instead of relying only on a favorable mean. A narrow win on a handful of pages is not SOTA evidence.

For a single reproducible command that runs benchmark and validation together, use `run-experiment`; see [EXPERIMENTS.md](EXPERIMENTS.md).

For the final portfolio-level claim review across the internal pack and public suites, use `review-claim`; see [CLAIM_REVIEW.md](CLAIM_REVIEW.md).

For model artifact provenance, see [MODELS.md](MODELS.md). Do not use an implicit `auto` engine for SOTA claims.

## Table Cell Analysis

Aggregate `table_cell_f1` is not enough to explain whether a table-oriented candidate is actually improving table OCR. Use `analyze-table-cells` after a benchmark has exported per-system `document.json` files:

```bash
ocrtech analyze-table-cells \
  --benchmark-report outputs/experiment-table-cell-refine-empty-hard15-v1/benchmark/report.json \
  --eval-manifest data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl \
  --out outputs/table-cell-analysis-refine-empty-hard15-v1 \
  --candidate candidate \
  --baseline surya \
  --baseline tesseract
```

The command requires:

- A benchmark report from `benchmark` or `run-experiment`.
- Exported benchmark documents at `benchmark/<system>/<input_stem>/document.json`.
- An eval manifest whose references contain structured `tables`.

It writes:

- `table-cell-analysis.json`
- `table-cell-analysis.md`

The report includes reference table count, predicted table count, duplicate overlapping table count, reference cell count, exact cell count, exact-cell rate, missing-cell count, mean cell CER, paired candidate-vs-baseline wins/losses/ties, and the highest-error candidate cells. This is the right artifact for identifying whether a table refiner is adding headroom, damaging already-correct cells, or producing structurally duplicated tables.

Interpret raw full-page Tesseract table results carefully. In the current v1 engine, Tesseract can be strong on exact image crops but still report zero structured table cells in full-page document parsing because it does not emit the table structure expected by `document.json`.

For the rendered hard15 table pages, reference `text` is the page title, while table contents live in structured `tables`. Do not mix these contracts: page-level `cer` answers whether the surrounding page text/title was recognized, and `table_cell_f1` plus cell exact analysis answers whether the table content was extracted. A benchmark change that inserts table grids into `Document.text` without also regenerating references changes the metric definition and is invalid for this pack.

The current `validation-hard15-surya.json` table-cell gate can fail even with a higher mean table-cell F1 when the bootstrap confidence interval includes zero. The figure-caption/table-lexicon v3 run is the concrete example: candidate `1.000000` vs Surya `0.979167`, but only 2 table pages improve and 6 tie, so the CI low is `0.000000`. Treat that as evidence that the table slice is too small for a strong claim. Do not remove the CI gate after observing the result; build a larger held-out table pack.

`configs/validation-table-ledger-surya.json` is the first targeted table-only validation config. It requires at least 40 table samples, gates `table_cell_f1` and `table_exact_match` against Surya with bootstrap CI above zero, and guards `table_structure_similarity` for non-regression. The v2 table-ledger pack (`60` samples, `720` cells) passes this config:

- `table_cell_f1`: candidate `1.000000`, Surya `0.979167`, CI low `0.012500`
- `table_exact_match`: candidate `1.000000`, Surya `0.750000`, CI low `0.150000`
- `table_structure_similarity`: candidate `1.000000`, Surya `1.000000`, non-regression guard passes

`table_structure_similarity` is a deterministic table-structure score over row count, column count, occupied cell coordinates, and rowspans/colspans when structured cell references are present. It is still not full HTML tree-edit TEDS. In the table-ledger v2 run it correctly shows no structure improvement over Surya because both systems already recover the simple grid structure perfectly. The candidate's validated gain is table cell-content accuracy with structure non-regression.

`table_teds_structure` and `table_teds_text` are deterministic TEDS-style ordered tree-edit scores over a table DOM made of `table`, `tr`, and `td` nodes. Structure mode compares row/cell tree shape plus `rowspan`/`colspan`; text mode also includes normalized cell text in `td` node labels. These are closer to paper-grade table metrics than the older token score, but they are still internal deterministic metrics, not a substitute for running an accepted public benchmark implementation when making broad SOTA claims.

After the span-inference postprocessor and tree metrics, the generated complex-table pack shows a structure gain over Surya (`table_structure_similarity` `1.000000` vs `0.875000`, `table_teds_structure` `1.000000` vs `0.904762`, `table_teds_text` `0.996250` vs `0.891667`, `table_exact_match` `0.925000` vs `0.000000`). This is evidence for the generated repeated-header pattern only. It does not replace a public or real scanned-table benchmark.

The public OmniDocBench smoke now also has a row-fragment normalization
candidate. It improves public table metrics over Surya on the same 5-page smoke
(`table_cell_f1` `0.564335` vs `0.296264`, `table_teds_structure` `0.942836`
vs `0.740128`, `table_teds_text` `0.675633` vs `0.526327`) and fixes one
`16x5` fragmented table into the expected `3x5` shape. It is still
`not_validated`: the sample count is far below the configured public gates,
bootstrap CI lower bounds are `0.000000` on table metrics, and text CER is
slightly worse than Surya on the smoke (`0.481629` vs `0.480531`).
The same sparse-row normalizer now recognizes numbered list markers such as
`1)`, `2.`, `(a)`, and Devanagari-digit variants. This is still guarded by the
same single-cell continuation-row rule and should be treated as a parser
postprocessor until rerun on the public table gate.

The larger public OmniDocBench `table40` run is more important. It reaches the
configured table-pair count (`40` public table pages) but still fails validation:
`table_cell_f1` improves in mean (`0.323122` vs `0.280062`) while losing more
pages than it wins (`7` wins, `9` losses, `24` ties), and the bootstrap CI low is
negative (`-0.019152`). `table_teds_structure` also has a positive mean
(`0.520959` vs `0.503603`) but CI low is negative (`-0.038381`). `table_teds_text`
does not clear the margin (`0.003609` improvement). The document CER gate is
also not met: only `40/100` required public document pages, and CER improvement
is only `0.000634` against a `0.005` margin.

The width-guarded v4 rerun improves the public table40 evidence but still does
not validate the full claim. Skipping table-grid normalization on tables wider
than `12` columns makes the table-cell F1 gate pass (`0.339319` vs `0.280062`,
CI low `0.001433`) and the table TEDS-structure gate pass (`0.545375` vs
`0.503603`, CI low `0.004790`). The text-sensitive table gate still fails:
`table_teds_text` improves in mean (`0.386013` vs `0.358057`) but CI low is
`-0.000220`. The public document CER gate also still fails because the slice has
only `40/100` required public document pages, despite a mean CER improvement of
`0.006096`.

On `2026-06-14`, the `table100` public run was launched on `k2` using
`scripts/run_public_table100_claim.py` and the v4 width-guard model card. The
first all-or-nothing launch stalled after three completed candidate rows. We
restarted with an isolated per-page runner, `--resume-existing`, and
`--sample-timeout-seconds 300`, preserving completed outputs and forcing future
stalls to become explicit error rows. The first measured timeout row was
`omnidocbench-00216-page-dca64e05-1ce6-49ee-857a-9b5d05a87357`. Do not cite this
run as a result until the validation artifact exists and has been inspected.

This validates a targeted table-ledger extraction claim, not full document SOTA. Full SOTA validation still needs broad document coverage, real scanned tables, merged-cell/header cases, true TEDS/TEDS-IOU, and public benchmark evidence.

## Recognizer Hard-Failure Gate

The failed AWS G6 PaddleOCR recognizer produced a normalized `50`-sample
hard-eval slice:

```text
data/splits-mixed-v1/eval-paddle-mixed-v2-cuda-hard50-normalized.jsonl
```

Use this slice as a regression gate for the next trained recognizer branch. It
must remain held out; do not train on it.

Validation config:

```text
configs/validation-paddle-hard50-tesseract.json
```

Run it after benchmarking a candidate recognizer against Tesseract:

```bash
ocrtech validate-claim \
  --benchmark-report outputs/next-recognizer-hard50/benchmark/report.json \
  --config configs/validation-paddle-hard50-tesseract.json \
  --eval-manifest data/splits-mixed-v1/eval-paddle-mixed-v2-cuda-hard50-normalized.jsonl \
  --out outputs/validation-next-recognizer-hard50
```

This gate is intentionally framed as failure recovery:

- all `50` hard rows must be present
- the `38` English synthetic line-crop failures must not regress versus
  Tesseract on CER or WER
- the `12` Nepali hard failures must improve WER over Tesseract by at least
  `0.05`

Passing this gate alone is not a SOTA claim. It is a recognizer branch admission
test before rerunning the full mixed OCR eval, rendered document pack, and public
benchmark gates.

The first line-level Tesseract/Paddle composite run exposed a benchmark-lifecycle
bug, not a model result: every candidate row crashed with `exit_code=-11` because
PaddleOCR was initialized in the parent process and then inherited by forked
timeout workers. After changing timeout mode to initialize engines inside the
watchdog worker, the rerun completed all `100` rows:

```text
outputs/experiment-line-script-paddle-hard50-v2/report.json
outputs/summary-line-script-paddle-hard50-v2/summary.md
outputs/validation-line-script-paddle-hard50-v2/validation.md
```

Validation status: `not_validated`.

- hard-50 CER: candidate `0.276489`, Tesseract `0.120106`
- English line-crop CER: candidate `0.060429`, Tesseract `0.060429`
- English line-crop WER: candidate `0.132259`, Tesseract `0.132259`
- Nepali hard CER: candidate `0.960679`, Tesseract `0.309083`
- Nepali hard WER: candidate `1.000000`, Tesseract `0.711607`

Interpretation: line-level routing can preserve English by leaving English lines
on the Tesseract primary path, but the trained PaddleOCR secondary recognizer is
not admissible for Devanagari overrides.

## Leakage Rules

Fatal leakage:

- same `sample_id`
- same image SHA-256
- same image path
- same image+text sample signature

Warning only:

- same text SHA-256

Text-only overlap is not fatal because short OCR labels repeat naturally across real corpora. It is still reported because repeated long labels may indicate bad dataset splitting.

## External Benchmarks

Use public suites for broad claims and the internal Nepali/English pack for the actual target domain:

- OmniDocBench for general document parsing.
- olmOCR-bench for PDF-to-Markdown structure preservation.
- OCRBench / OCRBench v2 for VLM OCR coverage.
- Internal Nepali + English pages for Devanagari OCR, mixed-script pages, government forms, tables, captions, and noisy scans.

For table claims, cell exact match, `table_cell_f1`, and span-aware `table_structure_similarity` are v1 metrics. Add true TEDS/TEDS-IOU before claiming leadership on complex merged-cell tables.

## Internal Eval Pack

See [EVAL_PACK.md](EVAL_PACK.md). The internal pack is necessary because public OCR/document benchmarks do not sufficiently cover Nepali, mixed Devanagari-English pages, and local document layouts.

## Dataset Splitting

See [DATASETS.md](DATASETS.md). Do not manually split data when making benchmark claims; use `split-manifest` so duplicate sample signatures cannot cross train/eval boundaries.