File size: 61,515 Bytes
f9bc0c1
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
# Experiment Workflow

If you are writing a paper from these runs, read [PAPER_NOTES.md](PAPER_NOTES.md) alongside this file. This document is the raw chronology; `PAPER_NOTES.md` is the stricter claim ledger.

`run-experiment` is the reproducible wrapper around benchmark and validation. It records the manifest hash, train manifest hashes, environment metadata, baseline list, benchmark report path, and validation status.

## Smoke Experiment

```bash
ocrtech create-eval-pack --out data/eval-pack-smoke --input-format text

ocrtech run-experiment \
  --eval-manifest data/eval-pack-smoke/manifests/ocrtech-eval-pack.jsonl \
  --references-dir data/eval-pack-smoke/references \
  --baselines ours \
  --skip-validation \
  --out outputs/smoke-experiment
```

This proves the pipeline and scoring path works. It is not OCR evidence.

## Claim Experiment

Run preflight first:

```bash
ocrtech preflight-claim \
  --eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
  --references-dir data/eval-pack/references \
  --baselines candidate,tesseract,stock-paddle,glm-ocr,paddleocr-vl \
  --candidate-model-config runs/recognizer/package/model-card.json \
  --validation-config validation.json \
  --train-manifest data/splits/train.jsonl \
  --out outputs/preflight
```

If the candidate is a routed composite, do not under-specify the baseline list. A claim-grade `script_select_composite` run must include its raw components in the same experiment, for example `candidate,tesseract,surya`, otherwise `preflight-claim` now fails `candidate-components`.

```bash
ocrtech run-experiment \
  --eval-manifest data/eval-pack/manifests/ocrtech-eval-pack.jsonl \
  --references-dir data/eval-pack/references \
  --baselines candidate,tesseract,stock-paddle,glm-ocr,paddleocr-vl \
  --candidate-model-config runs/recognizer/package/model-card.json \
  --validation-config validation.json \
  --train-manifest data/splits/train.jsonl \
  --preflight \
  --require-trained-recognizer \
  --require-model-admission \
  --out outputs/claim-experiment \
  --require-validated
```

`--preflight` runs the same claim-readiness checks inside `run-experiment` and
stops before benchmark output is produced if the setup cannot satisfy the
configured gates. `--require-validated` returns a non-zero exit code unless
validation status is `validated`.

After the internal claim experiment and the public suites are all complete, run `review-claim`; see [CLAIM_REVIEW.md](CLAIM_REVIEW.md).

Use `summarize-benchmark` when you need aggregate and paired slice comparisons from a raw `benchmark/report.json`:

```bash
ocrtech summarize-benchmark \
  --benchmark-report outputs/claim-experiment/benchmark/report.json \
  --out outputs/claim-experiment/benchmark-summary \
  --candidate candidate \
  --baseline tesseract \
  --metric cer
```

Use `paper-benchmark-table` when you need one paper-facing table merged from multiple `summary.json` artifacts, for example separate candidate-vs-Tesseract, candidate-vs-Surya, and candidate-vs-stock-Paddle summaries from the same pack:

```bash
ocrtech paper-benchmark-table \
  --summary outputs/paper-claimpack-summary-tesseract/summary.json \
  --summary outputs/paper-claimpack-summary-surya/summary.json \
  --summary outputs/paper-claimpack-summary-stock-paddle/summary.json \
  --out outputs/paper-claimpack-table \
  --system candidate \
  --system surya \
  --system tesseract \
  --system stock-paddle \
  --label 'candidate=Candidate' \
  --label 'surya=Surya' \
  --label 'tesseract=Tesseract' \
  --label 'stock-paddle=Stock Paddle' \
  --source-preference 'candidate=outputs/paper-claimpack-summary-tesseract/summary.json' \
  --source-preference 'tesseract=outputs/paper-claimpack-summary-tesseract/summary.json' \
  --source-preference 'surya=outputs/paper-claimpack-summary-surya/summary.json' \
  --source-preference 'stock-paddle=outputs/paper-claimpack-summary-stock-paddle/summary.json' \
  --metric cer \
  --metric table_cell_f1 \
  --metric reading_order_pair_accuracy \
  --metric latency_seconds
```

The `--source-preference` pins each system to one summary artifact. That matters when the same candidate was rerun in separate comparisons and latency differs across those reruns.

## Outputs

```text
outputs/claim-experiment/
  benchmark/report.json
  benchmark/report.md
  validation/validation.json
  validation/validation.md
  experiment.json
  experiment.md
```

`experiment.json` is the artifact to attach to a claim review. It does not replace the full benchmark and validation reports.

For candidate model-card requirements, see [MODELS.md](MODELS.md).

## Verified k2 HF Runs

As of June 14, 2026, the most useful live experiment series on `k2` is a candidate-only TrOCR comparison over one held-out mixed Nepali/English eval manifest with `180` samples. This is workflow evidence, not claim evidence, because it does not yet include Tesseract, stock Paddle, GLM-OCR, PaddleOCR-VL, or public benchmark suites.

Corpus construction used:

- `97` valid `gauravgiri` Nepali real rows from the first `100`
- `100` `prashant0919` Nepali synthetic rows
- `1000` NVIDIA English synthetic line crops from `nvidia/OCR-Synthetic-Multilingual-v1 --hf-subset en --split test`
- deterministic split: `1017` train / `180` eval
- materialized references from the eval manifest
- rebalanced training variant: `english=850`, `nepali=850`

Artifacts pulled from `k2` into this workspace:

- `outputs/live-k2-baseline-bundle/outputs/experiment-hf-smoke-on-mixed-v1/benchmark/report.json`
- `outputs/live-k2-baseline-bundle/outputs/experiment-hf-mixed-v1/benchmark/report.json`
- `outputs/live-k2-rebalanced-bundle/outputs/experiment-hf-mixed-v1-rebalanced/benchmark/report.json`
- `outputs/live-k2-rebalanced-bundle/runs/hf-recognizer-mixed-v1-rebalanced/training_metrics.json`
- `outputs/live-k2-rebalanced-bundle/data/splits-mixed-v1/train-rebalanced-nep-eng-rebalance.json`
- `outputs/live-k2-vs-tesseract-bundle/outputs/experiment-hf-mixed-v1-rebalanced-vs-tesseract/benchmark/report.json`

Benchmark means on that same eval split:

| Run | Overall CER | Overall WER | English CER | Nepali CER | Real CER |
| --- | ---: | ---: | ---: | ---: | ---: |
| smoke | 11.613341 | 2.637120 | 9.191231 | 23.723889 | 19.593537 |
| mixed | 1.628157 | 2.916153 | 0.379729 | 7.870295 | 12.702701 |
| rebalanced | 0.586139 | 0.801844 | 0.300454 | 2.014560 | 1.770016 |

The rebalanced run is the first one here that materially improves both English and Nepali on the same held-out split instead of trading one off against the other. It is still not a SOTA claim because:

- it is candidate-only
- it uses one internal eval manifest, not the full internal document pack plus public suites
- it has no paired validation against named baselines yet
- it does not measure document-structure leadership, only OCR metrics from the current report path

That last caveat is no longer theoretical. The first named baseline check on the same eval split is already negative for the project claim:

| Baseline | Overall CER | Overall WER | English CER | Nepali CER | Real CER |
| --- | ---: | ---: | ---: | ---: | ---: |
| candidate | 0.586139 | 0.801844 | 0.300454 | 2.014560 | 1.770016 |
| tesseract | 0.329012 | 0.486485 | 0.243379 | 0.757175 | 1.162917 |

So the current state is precise:

- we fixed our own mixed-script HF candidate substantially
- we have a reproducible training and benchmark path on `k2`
- we still do not beat Tesseract on this held-out eval

We also tested a mined-correction fallback instead of pretending post-OCR cleanup would obviously help. A ByT5 corrector was trained on `1700` Tesseract noisy/clean pairs mined from the rebalanced training manifest, then packaged as a `text_correction_composite` candidate with raw Tesseract as the base engine. That result was much worse than raw Tesseract:

| Baseline | Overall CER | Overall WER | English CER | Nepali CER | Real CER |
| --- | ---: | ---: | ---: | ---: | ---: |
| Tesseract + ByT5 corrector | 6.912065 | 3.312896 | 6.578699 | 8.578897 | 15.254935 |
| raw tesseract | 0.329012 | 0.486485 | 0.243379 | 0.757175 | 1.162917 |

Paired CER on the same `180` held-out samples was decisive: `0` wins, `170` losses, `10` ties. The corrector trainer metrics were already warning us that this was not a good model (`eval_cer=7.067461`, `eval_wer=3.334917`, `eval_exact_line_accuracy=0.0`), and the benchmark confirmed it.

Artifacts recorded locally:

- `outputs/live-k2-corrector-vs-tesseract-summary/summary.json`
- `outputs/live-k2-corrector-vs-tesseract-summary/summary.md`

We also tested whether Tesseract itself had enough headroom to matter before touching the recognizer again. Three direct variant checks on the same held-out `180`-sample eval all lost overall to raw default Tesseract (`psm=6`):

| Candidate | Overall CER | English CER | Nepali CER | Real CER |
| --- | ---: | ---: | ---: | ---: |
| raw tesseract (`psm=6`) | 0.329012 | 0.243379 | 0.757175 | 1.162917 |
| tesseract (`psm=7`) | 0.353708 | 0.242640 | 0.909049 | 1.104155 |
| tesseract (`psm=11`) | 0.344319 | 0.258868 | 0.771572 | 1.073521 |
| tesseract (`psm=13`) | 0.364878 | 0.243850 | 0.970017 | 1.004421 |

That said, `psm=13` was not uniformly useless. Using the existing `quality_score` as a routing heuristic, a `quality_select_composite` candidate that keeps default Tesseract unless `psm=13` is better by more than `0.02` did improve the same eval pack slightly:

| Baseline | Overall CER | English CER | Nepali CER | Real CER |
| --- | ---: | ---: | ---: | ---: |
| raw tesseract (`psm=6`) | 0.329012 | 0.243379 | 0.757175 | 1.162917 |
| routed tesseract (`psm=6` vs `psm=13`, margin `0.02`) | 0.324000 | 0.242003 | 0.733983 | 1.116534 |

Paired CER on that same pack was `5` wins, `3` losses, and `172` ties against raw default Tesseract. This is worth keeping as a baseline candidate backend, but it is not strong enough to confuse with model leadership, and the routing margin was selected on the same held-out pack.

We then used the new calibration workflow to stop cheating on that threshold. The original `180`-sample mixed eval was split into a calibration half (`89` samples) and untouched holdout half (`91` samples). `psm=13` was benchmarked against raw default Tesseract on the calibration half, `calibrate-quality-router` selected `secondary_quality_margin=0.03`, and that routed candidate was evaluated on the untouched holdout:

| Baseline | Overall CER | English CER | Nepali CER | Real CER |
| --- | ---: | ---: | ---: | ---: |
| raw tesseract (`psm=6`) | 0.342234 | 0.236933 | 0.835834 | 1.269163 |
| routed tesseract (`psm=6` vs `psm=13`, calibrated on separate half) | 0.340608 | 0.235094 | 0.835207 | 1.267909 |

Paired CER on the untouched holdout was `2` wins, `1` loss, and `88` ties. That is a real out-of-sample gain, but the size is too small to matter for the main project objective. It tells us the routing idea is not pure leakage, not that we have a strong OCR engine.

We also tested a richer calibrated ensemble over `psm=6/7/11/13`. The calibration half liked it and selected biases:

- `psm7=-0.12`
- `psm11=0.02`
- `psm13=-0.02`

But the untouched holdout rejected it:

| Baseline | Overall CER | English CER | Nepali CER | Real CER |
| --- | ---: | ---: | ---: | ---: |
| raw tesseract (`psm=6`) | 0.342234 | 0.236933 | 0.835834 | 1.269163 |
| calibrated `psm=6/7/11/13` ensemble | 0.358352 | 0.246311 | 0.883548 | 1.375632 |

Surya by itself was not competitive on the same `91`-sample holdout. As a raw baseline it landed at overall CER `2.198362` versus raw Tesseract `0.342234`, mainly because it was disastrous on the English-heavy line-crop slice. But Surya was materially better on the Nepali slice, so we tested a calibrated script-aware route instead of discarding it outright.

That routed candidate uses raw Tesseract output as the primary signal: if the Tesseract text looks sufficiently Devanagari-heavy, switch to Surya; otherwise keep Tesseract. The threshold was calibrated on the separate `89`-sample half with `ocrtech calibrate-script-router`, which selected `primary_script_threshold=0.4`. On the untouched `91`-sample holdout:

| Baseline | Overall CER | English CER | Nepali CER | Real CER |
| --- | ---: | ---: | ---: | ---: |
| raw tesseract (`psm=6`) | 0.342234 | 0.236933 | 0.835834 | 1.269163 |
| calibrated script router (`tesseract` -> `surya`) | 0.313084 | 0.230513 | 0.700133 | 1.172699 |

Paired CER on that holdout was `9` wins, `0` losses, and `82` ties against raw Tesseract. This is the strongest out-of-sample baseline improvement in the repo so far. It is still not claim-grade evidence, because it is one internal line-level split rather than the full internal document pack plus public suites, but it is a real improvement and a better direction than the TrOCR branch.

We also ran an exploratory `routing_granularity=page` variant of the same Tesseract-to-Surya router on that holdout, reusing the calibrated threshold `0.4`. It produced the same summary numbers as the document-level router. That is not a hidden win or loss; it just means this holdout is mostly single-page line-crop material, so page routing has no additional room to differentiate behavior yet.

After that, the internal eval pack was upgraded so image-mode tables are rendered as actual grids and the pack now includes a real `multi_page` slice as bundled page images. On that rendered `5`-sample smoke pack, the plain script router reached `table_cell_f1 = 1.0` on the dedicated table page but still missed the table inside the multi-page report when it kept Tesseract text for the document. A follow-up exploratory candidate with `secondary_structure_backfill=true` fixed that exact gap: it kept the same CER while backfilling Surya table structure on the multi-page report, moving aggregate `table_cell_f1` from `0.5` to `1.0` on the rendered smoke pack. This is still smoke evidence, not claim evidence, because the pack is tiny and only covers the `clean` slice, but it is the first concrete sign that composition can improve document structure without paying a text penalty.

The next run made that path less hand-wavy. A broader rendered image pack was generated with `25` samples across `clean`, `scan`, `blur`, `low_contrast`, and `skew`, and `audit-eval-pack --require-claim-ready` passed on it. The backfill router was then calibrated through `ocrtech calibrate-script-router` with `secondary_structure_backfill=true`, producing `ocrtech-tesseract-surya-script-router-backfill-calibrated`, and `preflight-claim` passed on the final experiment pack.

Pulled artifacts for that run:

- `outputs/live-k2-claimpack-rendered-backfill/eval-pack-audit.json`
- `outputs/live-k2-claimpack-rendered-backfill/eval-pack-audit.md`

The next useful tightening step was not “run everything on all `100` rendered samples and hope.” That is too expensive for the current GLM and PaddleOCR-VL paths on `k2`. Instead, the repo now has `ocrtech filter-manifest`, which was used to create a reproducible `5`-sample blur-only subset covering:

- paragraph
- table
- form
- figure
- multi-page report

That subset was benchmarked on `k2` across the real baseline set:

- `candidate` using `outputs/script-router-backfill-calibration/model-card.json`
- `tesseract`
- `surya`
- `stock-paddle`
- `glm-ocr` via the isolated `.venv-glm` hook
- `paddleocr-vl`

All `30` runs completed successfully. The pulled local artifacts are:

- `outputs/live-k2-rendered-blur5/report.json`
- `outputs/live-k2-rendered-blur5/summary.json`

Aggregate OCR/text result on that subset:

- candidate CER: `0.223273`
- Surya CER: `0.197601`
- Tesseract CER: `0.732364`
- stock-Paddle CER: `2.810426`
- GLM-OCR CER: `2.820269`
- PaddleOCR-VL CER: `18.470663`

Aggregate structure result on that subset:

- candidate `table_cell_f1`: `1.0`
- Surya `table_cell_f1`: `1.0`
- Tesseract / stock-Paddle / GLM-OCR / PaddleOCR-VL `table_cell_f1`: `0.0`

Aggregate reading-order result on that subset:

- candidate: `0.783636`
- Tesseract: `0.823636`
- Surya: `0.750909`
- stock-Paddle: `1.0`
- GLM-OCR: `1.0`
- PaddleOCR-VL: `0.750909`

That subset is exploratory, not claim-grade. But it is useful because it removes more guesswork:

- the current candidate is much better than Tesseract, stock-Paddle, GLM-OCR, and PaddleOCR-VL on blur-heavy rendered OCR CER
- the current candidate still does not beat raw Surya on text CER
- the current candidate still trades away some reading-order quality relative to Tesseract and the layout-heavy baselines

For paper use, the right interpretation is narrower than "our model wins." This subset supports a more defensible statement: the current backfill candidate is a strong internal engineering baseline on degraded rendered documents, especially for table recovery, but raw Surya is still the stronger text OCR baseline on the same subset.

Additional pulled artifacts for that `25`-sample backfill run:

- `outputs/live-k2-claimpack-rendered-backfill/calibration.json`
- `outputs/live-k2-claimpack-rendered-backfill/model-card.json`
- `outputs/live-k2-claimpack-rendered-backfill/experiment.json`
- `outputs/live-k2-claimpack-rendered-backfill/report.json`
- `outputs/live-k2-claimpack-rendered-backfill/summary.json`
- `outputs/live-k2-claimpack-rendered-backfill/summary.md`

Key numbers versus raw Tesseract on that rendered claim-ready pack:

| Slice | Candidate CER | Tesseract CER | Notes |
| --- | ---: | ---: | --- |
| all | 0.248654 | 0.689214 | paired CER `9` wins, `1` loss, `15` ties |
| clean | 0.243224 | 0.540584 | same win pattern as smoke pack |
| blur | 0.260039 | 0.991613 | strongest degraded-slice text gain |
| scan | 0.237424 | 0.593631 | clear text win |
| low_contrast | 0.263265 | 0.554174 | smaller but still positive |
| skew | 0.239317 | 0.766067 | one loss, two wins, two ties |
| multi_page | 0.315789 | 0.315789 | CER tie |
| table | 0.339713 | 1.421531 | large text gain on table-heavy pages |

Structure metrics on the same pack matter more than the CER win:

- candidate aggregate `table_cell_f1`: `0.955556`
- Tesseract aggregate `table_cell_f1`: `0.0`
- candidate aggregate `table_exact_match`: `0.8`
- Tesseract aggregate `table_exact_match`: `0.0`
- candidate `multi_page` `table_cell_f1`: `0.911111`
- Tesseract `multi_page` `table_cell_f1`: `0.0`

This is better evidence than the tiny smoke pack, but it still is not a clean claim story. Two real weaknesses remain on this rendered pack:

- `form` CER regressed slightly: `0.231933` vs `0.230010`
- aggregate `reading_order_pair_accuracy` is lower than Tesseract: `0.764` vs `0.816`

So the honest reading is narrow: the calibrated Tesseract-plus-Surya backfill router is a strong internal rendered-pack baseline for OCR robustness and table extraction, but it is not yet a broad document-understanding winner.

We then ran two more comparisons on that same rendered claim-ready pack to stop overfitting the story to Tesseract.

First, raw Surya on the rendered pack was not the disaster it had been on the older line-crop holdout. On the `25`-sample rendered document pack it beat the calibrated backfill router on text CER across every checked slice:

| Slice | Candidate CER | Raw Surya CER |
| --- | ---: | ---: |
| all | 0.248654 | 0.195260 |
| blur | 0.260039 | 0.202626 |
| scan | 0.237424 | 0.191862 |
| low_contrast | 0.263265 | 0.191862 |
| skew | 0.239317 | 0.194733 |
| form | 0.231933 | 0.138462 |
| multi_page | 0.315789 | 0.200000 |
| table | 0.339713 | 0.281818 |

Paired CER was not close: `0` wins, `15` losses, `10` ties for the candidate on `all`.

That does not mean the router is useless. Structure on the same pack is more mixed:

- aggregate `table_cell_f1`: candidate `0.955556`, Surya `0.955556`
- aggregate `reading_order_pair_accuracy`: candidate `0.764000`, Surya `0.750909`
- paired reading-order wins on `all`: candidate `2`, losses `0`, ties `23`

So the rendered document-pack conclusion is stricter than the earlier line-crop conclusion: the current routed candidate is still behind raw Surya on text OCR, while only slightly ahead on reading-order behavior and roughly tied on table structure.

Second, `stock-paddle` stopped being a fake baseline. The original `k2` benchmark result showing `1.0` CER everywhere was an adapter bug in this repo, not a real PaddleOCR result: `paddleocr 3.7.0` on `k2` returns v3 `OCRResult` objects, and the old parser only understood the older tuple/list shape. After fixing that adapter and installing `paddlepaddle 3.3.1`, the stock-Paddle baseline became runnable on `k2`.

With that fix in place, the same rendered claim-ready pack produced:

| Slice | Candidate CER | Stock-Paddle CER |
| --- | ---: | ---: |
| all | 0.248654 | 1.749081 |
| blur | 0.260039 | 1.549743 |
| scan | 0.237424 | 1.896558 |
| low_contrast | 0.263265 | 1.854006 |
| skew | 0.239317 | 1.568643 |
| form | 0.231933 | 0.470694 |
| multi_page | 0.315789 | 0.391388 |
| table | 0.339713 | 3.395694 |

Paired CER on `all` was `21` wins, `4` losses, `0` ties for the candidate.

Structure was lopsided in the other direction:

- candidate aggregate `table_cell_f1`: `0.955556`
- stock-Paddle aggregate `table_cell_f1`: `0.0`
- candidate aggregate `reading_order_pair_accuracy`: `0.764000`
- stock-Paddle aggregate `reading_order_pair_accuracy`: `1.0`

So stock-Paddle, as currently configured here with `PaddleOCR(lang='ne')`, preserves reading order very aggressively but is not competitive on text accuracy or table extraction for this mixed rendered pack.

The stricter rendered-document result is now also explicit. On `k2`, a larger `claimpack-rendered-v2` split was calibrated separately and evaluated on an untouched `50`-sample holdout. The routed script candidate selected `primary_script_threshold=0.0` on calibration, which means it collapsed to "always pick Surya" there. On the holdout:

| Baseline | Overall CER | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: |
| candidate | 0.180893 | 0.947500 | 0.750909 |
| raw Surya | 0.180893 | 0.970833 | 0.750909 |
| raw Tesseract | 0.714207 | 0.000000 | 0.787818 |

Pulled local artifacts:

- `outputs/live-k2-claimpack-v2-script-router/calibration.json`
- `outputs/live-k2-claimpack-v2-script-router/model-card.json`
- `outputs/live-k2-claimpack-v2-script-router/experiment.json`
- `outputs/live-k2-claimpack-v2-script-router/summary.json`
- `outputs/live-k2-claimpack-v2-script-router/preflight.json`
- `outputs/paper-claimpack-v2-table/paper-table.md`

This is not a routed-model win. It is a branch ceiling. The candidate does not beat Surya on text, ties it exactly on CER, and is slightly worse on table structure. The new claim guardrail now rejects this case directly: `preflight-claim` fails `candidate-calibration` when a routed candidate collapses to one engine on every calibration sample.

We also tested the obvious opposite branch after adding Latin-script routing support to the repo: a Surya-primary page router with Tesseract fallback for English-heavy pages. The rendered-v2 calibration on `k2` selected `primary_script_threshold=0.7`, and the holdout result was unambiguously bad:

| Baseline | Overall CER | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: |
| candidate | 0.416854 | 0.337500 | 0.761818 |
| raw Surya | 0.180893 | 0.970833 | 0.750909 |
| raw Tesseract | 0.714207 | 0.000000 | 0.787818 |

Pulled local artifacts:

- `outputs/live-k2-claimpack-v2-surya-latin-router/calibration.json`
- `outputs/live-k2-claimpack-v2-surya-latin-router/experiment.json`
- `outputs/live-k2-claimpack-v2-surya-latin-router/report.json`
- `outputs/live-k2-claimpack-v2-surya-latin-router/summary.json`
- `outputs/paper-claimpack-v2-surya-latin-table/paper-table.md`

Paired CER versus raw Surya on that holdout was `0` wins, `12` losses, `38` ties. So the inverted script router is not a refinement. It is a disproved branch.

We then tested the narrower quality-based version of the same idea: keep Tesseract and Surya as components, but let the router pick based on the local `quality_score` signal instead of script ratio. The rendered-v2 calibration selected `selected_threshold=0.0`, which already tells you the calibration split did not find a real decision boundary beyond "prefer Surya whenever it is not clearly worse." On the untouched holdout:

| Baseline | Overall CER | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: |
| candidate | 0.181128 | 0.947500 | 0.750909 |
| raw Surya | 0.180893 | 0.947500 | 0.750909 |
| raw Tesseract | 0.714207 | 0.000000 | 0.787818 |

Pulled local artifacts:

- `outputs/live-k2-claimpack-v2-surya-quality-router/calibration.json`
- `outputs/live-k2-claimpack-v2-surya-quality-router/experiment.json`
- `outputs/live-k2-claimpack-v2-surya-quality-router/report.json`
- `outputs/live-k2-claimpack-v2-surya-quality-router/summary.json`
- `outputs/paper-claimpack-v2-surya-quality-router-table/paper-table.md`

Paired CER versus raw Surya on that holdout was `0` wins, `1` loss, `49` ties. So this is not the missing refinement either. It is nearly indistinguishable from Surya and still not better.

We also tested a non-routing composition on the same rendered-v2 holdout: `line_align_composite`. The idea was simple enough to be worth disproving. Keep Surya as the text engine, reuse Tesseract line scaffolding, and see whether better ordering helps enough to justify the merge. It did not:

| Baseline | Overall CER | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: |
| candidate | 0.310549 | 0.897500 | 0.792727 |
| raw Surya | 0.180893 | 0.970833 | 0.750909 |
| raw Tesseract | 0.714207 | 0.000000 | 0.787818 |

Pulled local artifacts:

- `outputs/live-k2-claimpack-v2-line-align-summary/summary.json`
- `outputs/paper-claimpack-v2-line-align-table/paper-table.md`
- `outputs/live-k2-claimpack-v2-line-align-model-card.json`

Paired CER versus raw Surya on that holdout was `0` wins, `7` losses, `43` ties. The candidate did improve reading-order accuracy relative to raw Surya, and even edged past Tesseract there, but the text and table regressions are too large to defend. So line alignment is another useful negative result, not a candidate answer.

Finally, we measured the sample-level upper bound for the same component family using `benchmark-oracle`. This is not a deployable candidate. It chooses the better of raw Surya and raw Tesseract per sample using ground-truth CER. That is the right diagnostic question before wasting more time on routing heuristics:

| System | Overall CER | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: |
| Surya/Tesseract oracle | 0.180529 | 0.970833 | 0.750909 |
| raw Surya | 0.180893 | 0.970833 | 0.750909 |
| raw Tesseract | 0.714207 | 0.000000 | 0.787818 |

Pulled/generated local artifacts:

- `outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-analysis.json`
- `outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-choices.json`
- `outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-summary/summary.json`
- `outputs/paper-claimpack-v2-oracle-table/paper-table.md`

The oracle picked raw Surya for `49` samples and raw Tesseract for `1`. Those per-sample source choices are stored in `oracle-choices.json` as provenance, not mixed into benchmark metrics. Paired CER versus raw Surya was `1` win, `0` losses, `49` ties. That means the current Surya/Tesseract routing space is effectively exhausted for text OCR on rendered-v2. The next useful model-side work needs a stronger recognizer or document model, not another selector over these two engines.

We then added `benchmark-error-analysis` to convert that failure into a training target instead of another routing guess. On the same rendered-v2 benchmark report, raw Surya's highest absolute CER slices are `figure` (`0.253157`) and `table` (`0.252781`). The hardest individual samples are dominated by table-ledger and figure-caption pages. The Surya-vs-oracle diagnostic confirms that Tesseract only improves one sample, `eval-form-notice-003-blur`, by `0.018182` CER. So the next serious model branch should target table/figure-heavy mixed-script recognition and document context, while treating Surya/Tesseract routing as exhausted.

We also exported those ranked failures back into a real manifest instead of a hand-maintained list. `data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl` is a `15`-sample `hard_eval` manifest built from the top Surya CER failures. It keeps the original reference metadata and adds `metadata.failure_analysis` with rank, metric, value, and comparison details. The local checkout does not include the rendered image assets, so the local audit reports missing image paths; the same manifest passes full file/reference audit on `k2` at `outputs/audit-claimpack-v2-surya-hard15/manifest-audit.json`, pulled locally to `outputs/audit-claimpack-v2-surya-hard15-k2/manifest-audit.json`.

The hard pocket now has an explicit validation config: `configs/validation-hard15-surya.json`. This uses conjunctive gate slices such as `["hard_eval", "table"]` and `["hard_eval", "figure"]`, so the table and figure gates cannot be satisfied by easier non-hard samples. As a negative control, the current line-align candidate fails this config on `outputs/live-k2-claimpack-v2-line-align/report.json`: hard-eval CER is `0.482039` versus Surya `0.348706`, hard-table CER is `0.613636` versus Surya `0.363636`, and the figure/table structure gates do not improve over Surya.

For model iteration, we generated a separate table/figure development pack on `k2`:

- manifest: `data/eval-pack-table-figure-dev-v1/manifests/ocrtech-eval-pack.jsonl`
- audit: `outputs/audit-eval-pack-table-figure-dev-v1-k2/eval-pack-audit.json`
- samples: `80`
- slices: `40` table, `40` figure, `80` Nepali/English/mixed-script/image/reading_order
- degradations: `clean`, `scan`, `blur`, `low_contrast`, `skew`
- hard15 overlap: `0` sample IDs and `0` text hashes

This pack was generated with `--template table-ledger --template figure-caption --variant-offset 100`, after updating the eval-pack generator so table OCR references include visible table rows and figure variants include copy-specific visible text. Treat this as a development/training target, not as held-out claim evidence.

The pack was then split with text-level grouping so degraded variants of the same target do not leak across train/eval:

```bash
ocrtech split-manifest data/eval-pack-table-figure-dev-v1/manifests/ocrtech-eval-pack.jsonl \
  --out data/splits-table-figure-dev-v1 \
  --eval-ratio 0.2 \
  --seed 17 \
  --stratify-by document_type \
  --group-by text_sha256
```

Verified split:

- train: `60` samples
- eval: `20` samples
- text groups: `16`
- leakage check: `pass`
- train audit on `k2`: `pass`
- eval audit on `k2`: `pass`

Additional leakage checks on `k2` found no sample-id or text-hash overlap with `hard15` or rendered-v2 eval. That makes this a legitimate development pack for the next model branch, but it is still not claim evidence.

The first model-side attempt on this branch was a minimal Hugging Face VisionEncoderDecoder fine-tune using `microsoft/trocr-small-printed`:

```bash
ocrtech train-recognizer \
  --backend hf-vision-encoder-decoder \
  --base-model microsoft/trocr-small-printed \
  --train-manifest data/splits-table-figure-dev-v1/train.jsonl \
  --eval-manifest data/splits-table-figure-dev-v1/eval.jsonl \
  --out runs/hf-table-figure-dev-v1 \
  --epochs 1 \
  --train-batch-size 4 \
  --eval-batch-size 4 \
  --learning-rate 5e-5 \
  --run
```

Training completed and the package audited cleanly, but the result was a decisive negative:

| metric | value |
| --- | ---: |
| eval_loss | 7.884517 |
| eval_cer | 3.062119 |
| eval_wer | 3.562689 |
| eval_exact_line_accuracy | 0.000000 |

Benchmarking the packaged candidate against Surya and Tesseract on the `20`-sample dev eval confirmed the branch is not viable:

| system | CER | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: |
| HF table/figure candidate | 2.422001 | 0.000000 | 0.000000 |
| Surya | 0.563399 | 0.858333 | 0.550000 |
| Tesseract | 0.546227 | 0.000000 | 0.695000 |

Paired CER was `0` wins, `20` losses, and `0` ties against both Surya and Tesseract. The failure mode is not subtle: naive full-page TrOCR fine-tuning does not learn table/figure document recognition from this small pack. The next credible recognizer branch should train on line/cell crops or use a page/document model with structure-aware supervision.

We then implemented that crop-level path and tested it instead of leaving it as advice. Newly generated image-pack references now carry renderer layout metadata for non-skew pages, and `ocrtech extract-crops` turns those verified bboxes into ordinary recognizer manifests. Skew pages are skipped because pre-rotation bboxes are invalid.

Full crop development pack on `k2`:

- source pack: `data/eval-pack-table-figure-crops-dev-v1/manifests/ocrtech-eval-pack.jsonl`
- crop manifest: `data/crops-table-figure-dev-v1/manifests/crops.jsonl`
- source samples: `80`
- crops: `544`
- crop types: `384` table cells, `128` lines, `32` figure captions
- skipped candidates: `32` skew pages
- crop audit: `outputs/audit-crops-table-figure-dev-v1/manifest-audit.md`
- split: `480` train / `64` eval
- split grouping: `text_sha256`
- split leakage: pass

The first crop-level HF model used the same `microsoft/trocr-small-printed` backend for one epoch, this time on `480` crop images instead of full pages. Trainer metrics improved relative to the full-page branch, but still were not competitive:

| metric | value |
| --- | ---: |
| eval_loss | 3.087561 |
| eval_cer | 0.959831 |
| eval_wer | 1.363715 |
| eval_exact_line_accuracy | 0.078125 |

Packaged crop model audit passed:

- `runs/hf-crops-table-figure-dev-v1/training_metrics.json`
- `runs/hf-crops-table-figure-dev-v1/package/model-card.json`
- `outputs/audit-hf-crops-table-figure-dev-v1-model/model-audit.md`

Benchmark result on the `64`-sample crop eval:

| system | CER |
| --- | ---: |
| HF crop candidate | 1.043369 |
| Surya | 0.475309 |
| Tesseract | 0.077453 |

Paired CER:

- versus Surya: `7` wins, `50` losses, `7` ties
- versus Tesseract: `0` wins, `64` losses, `0` ties

This means crop-level training is the right data shape, but this tiny HF recognizer is still the wrong recognizer. The immediate technical takeaway is that Tesseract is extremely strong on these generated crop images; the next branch should use strong crop OCR behind better layout/cell/caption detection, or train a more suitable recognizer with substantially more real crop data.

The next composition test used that crop-OCR signal more directly: `table_cell_refine_composite` keeps Surya as the document/table structure engine and rereads detected table cells with Tesseract crops.

Two hard15 variants were tested:

| variant | hard_eval CER | table_cell_f1 | reading_order_pair_accuracy | verdict |
| --- | ---: | ---: | ---: | --- |
| Surya baseline | 0.348706 | 0.979167 | 0.520000 | current hard15 target |
| `replacement_policy=always` | 0.348706 | 0.593750 | 0.520000 | unsafe; damages good Surya cells |
| `replacement_policy=empty` | 0.348706 | 0.979167 | 0.520000 | safe tie; no improvement |
| `replacement_policy=empty`, table de-dup | 0.348706 | 0.979167 | 0.520000 | parser hygiene fix; still no improvement |

The safe policy also failed the hard15 validation gate with status `not_validated`. This is another useful negative result: Tesseract crop OCR is strong when the crop is exact, but naively applying it to Surya-inferred table cells does not improve Surya on this hard pocket.

Cell-level analysis explains why:

| system | reference_cells | exact_cells | exact_rate | mean_cer |
| --- | ---: | ---: | ---: | ---: |
| Surya | 96 | 94 | 0.979167 | 0.010417 |
| table-cell refine, `always` | 96 | 57 | 0.593750 | 0.284309 |
| table-cell refine, `empty` | 96 | 94 | 0.979167 | 0.010417 |

Against Surya, the unsafe `always` policy improved `0` cells, worsened `37`, and tied `59`. The safe `empty` policy tied all `96` cells because Surya had no missing cells to fill in this hard pocket. The remaining Surya cell errors are only two occurrences of `किरण` predicted as `करिण`, both on the same table-ledger variant. The next table branch needs a targeted confidence/replacement signal for rare wrong-but-nonempty cells; blind crop rerecognition is the wrong direction.

The table de-duplication branch fixed a separate parser hygiene issue: line-based table inference no longer creates a second table when an engine already supplied an overlapping structured table. On the rerun artifact, both candidate and Surya now report `8` predicted tables for `8` reference tables and `0` duplicate overlapping tables. This is useful for cleaner exports, but because the fix applies at parser assembly time it also benefits the Surya baseline; hard15 validation remains `not_validated`.

Generated diagnostics:

- `outputs/paper-claimpack-v2-surya-error-analysis/error-analysis.md`
- `outputs/paper-claimpack-v2-surya-error-analysis/top-failures.json`
- `outputs/paper-claimpack-v2-surya-vs-tesseract-error-analysis/error-analysis.md`
- `outputs/paper-claimpack-v2-surya-vs-oracle-error-analysis/error-analysis.md`
- `data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl`
- `configs/validation-hard15-surya.json`
- `outputs/validation-claimpack-v2-line-align-hard15/validation.json`
- `outputs/audit-claimpack-v2-surya-hard15-k2/manifest-audit.json`
- `data/eval-pack-table-figure-dev-v1/manifests/ocrtech-eval-pack.jsonl`
- `outputs/audit-eval-pack-table-figure-dev-v1-k2/eval-pack-audit.json`
- `data/splits-table-figure-dev-v1/split-summary.md`
- `outputs/audit-table-figure-dev-v1-train-k2/manifest-audit.md`
- `outputs/audit-table-figure-dev-v1-eval-k2/manifest-audit.md`
- `runs/hf-table-figure-dev-v1/training_metrics.json`
- `runs/hf-table-figure-dev-v1/package/model-card.json`
- `outputs/audit-hf-table-figure-dev-v1-model/model-audit.md`
- `outputs/summary-hf-table-figure-dev-v1/summary.md`
- `outputs/experiment-hf-table-figure-dev-v1/benchmark/report.json`
- `outputs/crops-table-figure-dev-v1-k2/crop-manifest-summary.md`
- `outputs/crops-table-figure-dev-v1-k2/manifest-audit.md`
- `data/splits-crops-table-figure-dev-v1/split-summary.md`
- `runs/hf-crops-table-figure-dev-v1/training_metrics.json`
- `runs/hf-crops-table-figure-dev-v1/package/model-card.json`
- `outputs/audit-hf-crops-table-figure-dev-v1-model/model-audit.md`
- `outputs/summary-hf-crops-table-figure-dev-v1/summary.md`
- `outputs/experiment-hf-crops-table-figure-dev-v1/benchmark/report.json`
- `outputs/table-cell-refine-v1/model-card.json`
- `outputs/summary-table-cell-refine-hard15-v1/summary.md`
- `outputs/table-cell-refine-empty-v1/model-card.json`
- `outputs/summary-table-cell-refine-empty-hard15-v1/summary.md`
- `outputs/validation-table-cell-refine-empty-hard15-v1/validation.md`
- `outputs/table-cell-analysis-refine-always-hard15-v1/table-cell-analysis.md`
- `outputs/table-cell-analysis-refine-empty-hard15-v1/table-cell-analysis.md`
- `outputs/table-cell-refine-empty-dedup-v1/model-card.json`
- `outputs/summary-table-cell-refine-empty-dedup-hard15-v3/summary.md`
- `outputs/table-cell-analysis-refine-empty-dedup-hard15-v3/table-cell-analysis.md`
- `outputs/validation-table-cell-refine-empty-dedup-hard15-v3/validation.md`

When repeating this workflow, keep the comparison honest:

```bash
ocrtech prepare-data gauravgiri/nepali-ocr-dataset --hf --split train --limit 100 --skip-invalid --out data/manifests
ocrtech prepare-data prashant0919/nepali-synthetic-ocr-lines --hf --split train --limit 100 --out data/manifests
ocrtech prepare-data nvidia/OCR-Synthetic-Multilingual-v1 --hf --hf-subset en --split test --limit 1000 --out data/manifests-english-test
ocrtech merge-manifests \
  data/manifests/gauravgiri__nepali-ocr-dataset-train.jsonl \
  data/manifests/prashant0919__nepali-synthetic-ocr-lines-train.jsonl \
  data/manifests-english-test/nvidia__OCR-Synthetic-Multilingual-v1__en-test.jsonl \
  --out data/manifests/nep-eng-mixed-v1.jsonl
ocrtech split-manifest data/manifests/nep-eng-mixed-v1.jsonl --out data/splits-mixed-v1 --eval-ratio 0.15 --seed 13
ocrtech materialize-references data/splits-mixed-v1/eval.jsonl --out data/references-mixed-v1 --rewritten-manifest data/splits-mixed-v1/eval-with-refs.jsonl
ocrtech rebalance-manifest data/splits-mixed-v1/train.jsonl --out data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl --slice english --slice nepali --seed 13
ocrtech train-recognizer --backend hf-vision-encoder-decoder --base-model microsoft/trocr-small-printed --train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl --eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl --out runs/hf-recognizer-mixed-v1-rebalanced --train-batch-size 4 --eval-batch-size 4 --epochs 0.5 --run
ocrtech package-hf-recognizer --model-dir runs/hf-recognizer-mixed-v1-rebalanced/final-model --out runs/hf-recognizer-mixed-v1-rebalanced/package --model-id ocrtech-hf-mixed-v1-rebalanced --base-model microsoft/trocr-small-printed --train-manifest data/splits-mixed-v1/train-rebalanced-nep-eng.jsonl --eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl --training-summary runs/hf-recognizer-mixed-v1-rebalanced/recognizer-training-summary.json --metrics-report runs/hf-recognizer-mixed-v1-rebalanced/training_metrics.json
ocrtech run-experiment --eval-manifest data/splits-mixed-v1/eval-with-refs.jsonl --out outputs/experiment-hf-mixed-v1-rebalanced --baselines candidate --candidate-model-config runs/hf-recognizer-mixed-v1-rebalanced/package/model-card.json --references-dir data/references-mixed-v1 --skip-validation
```

## Figure Caption Refinement Composite

The figure-caption hard15 slice exposed a parser problem first: after `[Figure: ...]`, the actual caption line was being rejected when OCR damaged the Nepali caption keyword (`चित्र` became variants such as `चतुर्थ` or `चत्रि`). The parser now accepts a nearby line after a figure marker as the caption if it is spatially close and not another marker/table line. That reduced Surya/candidate figure-caption CER on hard15 from about `0.737609` to `0.092323`.

The next branch used that cleaner structure and tried a narrow composite: keep Surya as the primary document engine, but replace only overlapping figure-caption lines with Tesseract when the Tesseract line has a better caption-prefix score. The replacement keeps Surya's layout bbox so skewed Tesseract boxes do not break caption attachment.

Hard15 v2 result:

| system | CER | figure_caption_cer | figure_detection_f1 | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: | ---: | ---: |
| Surya | 0.236839 | 0.092323 | 1.000000 | 0.979167 | 0.520000 |
| Tesseract | 1.266364 | 0.051020 | 0.857143 | 0.000000 | 0.613333 |
| figure-caption refine | 0.228008 | 0.052478 | 1.000000 | 0.979167 | 0.520000 |

Validation against the hard15 Surya config is still `not_validated`, but one gate now passes:

- `hard_eval,figure` `figure_caption_cer`: pass, `6` wins / `0` losses / `1` tie, candidate `0.052478`, Surya `0.092323`, improvement `0.039845`
- overall hard_eval CER: fail, improvement `0.008830 < 0.020000`
- hard_eval figure CER: fail, improvement `0.018922 < 0.020000`
- table CER/F1 gates: fail, ties only

Artifacts:

- `outputs/figure-caption-refine-v1/model-card.json`
- `outputs/audit-figure-caption-refine-v1-model/model-audit.md`
- `outputs/summary-figure-caption-refine-hard15-v2/summary.md`
- `outputs/experiment-figure-caption-refine-hard15-v2/benchmark/report.json`
- `outputs/validation-figure-caption-refine-hard15-v2/validation.md`

This is the first branch in this sequence that beats Surya on a configured gate, but it is a slice win, not a SOTA claim.

The stronger opt-in structural-label branch adds edit-distance guarded correction for document-role labels (`वार्षिक प्रतिवेदन`, `चित्र`, `निष्कर्ष`) after caption replacement:

| system | CER | figure_caption_cer | figure_detection_f1 | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: | ---: | ---: |
| Surya | 0.236839 | 0.092323 | 1.000000 | 0.979167 | 0.520000 |
| figure-caption refine + labels | 0.204610 | 0.008746 | 1.000000 | 0.979167 | 0.520000 |

Validation still reports `not_validated`, but the pass/fail split is materially better:

- overall hard_eval CER: pass, `7` wins / `0` losses / `8` ties, candidate `0.204610`, Surya `0.236839`, improvement `0.032228`
- `hard_eval,figure` CER: pass, `7` wins / `0` losses / `0` ties, candidate `0.022866`, Surya `0.091927`, improvement `0.069061`
- `hard_eval,figure` `figure_caption_cer`: pass, `7` wins / `0` losses / `0` ties, candidate `0.008746`, Surya `0.092323`, improvement `0.083576`
- table CER/F1 gates: fail, ties only

Artifacts:

- `outputs/figure-caption-refine-labels-v1/model-card.json`
- `outputs/audit-figure-caption-refine-labels-v1-model/model-audit.md`
- `outputs/summary-figure-caption-refine-labels-hard15-v1/summary.md`
- `outputs/experiment-figure-caption-refine-labels-hard15-v1/benchmark/report.json`
- `outputs/validation-figure-caption-refine-labels-hard15-v1/validation.md`

The next hard15 candidate added two explicitly auditable pieces on top of that branch:

- a small curated table-name lexicon, `configs/nepali-table-name-lexicon-v1.txt`, used only for cells under a header close to `नाम`
- a guarded structural-title correction for the recurring table title `दैनिक हिसाब`

This v3 run is the strongest internal hard15 result so far:

| system | CER | figure_caption_cer | figure_detection_f1 | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: | ---: | ---: |
| Surya | 0.236839 | 0.092323 | 1.000000 | 0.979167 | 0.520000 |
| Tesseract | 1.266364 | 0.051020 | 0.857143 | 0.000000 | 0.613333 |
| figure-caption + labels + table lexicon | 0.010671 | 0.008746 | 1.000000 | 1.000000 | 0.520000 |

Validation remains `not_validated`, but only one configured gate now fails:

- overall hard_eval CER: pass, `15` wins / `0` losses / `0` ties, candidate `0.010671`, Surya `0.236839`
- `hard_eval,table` CER: pass, `8` wins / `0` losses / `0` ties, candidate `0.000000`, Surya `0.363636`
- `hard_eval,figure` CER: pass, `7` wins / `0` losses / `0` ties, candidate `0.022866`, Surya `0.091927`
- `hard_eval,figure` `figure_caption_cer`: pass, `7` wins / `0` losses / `0` ties, candidate `0.008746`, Surya `0.092323`
- `hard_eval,table` `table_cell_f1`: fail, candidate `1.000000`, Surya `0.979167`, improvement `0.020833`, but bootstrap CI low is `0.000000` because the 8-page table slice has only 2 improving table pages and 6 ties

Cell-level analysis is still useful evidence:

- candidate: `96/96` exact cells, mean cell CER `0.000000`
- Surya: `94/96` exact cells, mean cell CER `0.010417`
- Tesseract full-page parser: `0/96` structured cells
- paired candidate-vs-Surya cells: `2` candidate-better, `0` Surya-better, `94` ties

Artifacts:

- `outputs/figure-caption-table-lexicon-v1/model-card.json`
- `outputs/audit-figure-caption-table-lexicon-v1-model/model-audit.md`
- `outputs/experiment-figure-caption-table-lexicon-hard15-v3/benchmark/report.json`
- `outputs/summary-figure-caption-table-lexicon-hard15-v3/summary.md`
- `outputs/table-cell-analysis-figure-caption-table-lexicon-hard15-v3/table-cell-analysis.md`
- `outputs/validation-figure-caption-table-lexicon-hard15-v3/validation.md`

Important interpretation: the hard15 table-page reference `text` is the page title (`दैनिक हिसाब`), while table contents are evaluated through structured table references. A temporary v2 attempt to include table grids in `Document.text` was rejected after verifying the actual reference schema. We did not weaken validation gates after observing the result.

## Table-Ledger Claim Pack V2

To test whether the table-cell gain survives a larger table-only pack, we generated an independent rendered image pack:

```bash
ocrtech create-eval-pack \
  --out data/eval-pack-table-ledger-claim-v1 \
  --input-format image \
  --template table-ledger \
  --count-per-template 12 \
  --variant-offset 300 \
  --degradation clean \
  --degradation scan \
  --degradation blur \
  --degradation low_contrast \
  --degradation skew
```

This produced `60` table-ledger image samples. The pack audited cleanly but is intentionally not broad claim-ready because it covers one document type.

Validation config: `configs/validation-table-ledger-surya.json`.

Validation result: `validated` after adding a grid-structure non-regression gate.

| gate | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| table_cell_f1 | 60 | 15 | 0 | 45 | 1.000000 | 0.979167 | 0.020833 | 0.012500 |
| table_exact_match | 60 | 15 | 0 | 45 | 1.000000 | 0.750000 | 0.250000 | 0.150000 |
| table_structure_similarity | 60 | 0 | 0 | 60 | 1.000000 | 1.000000 | 0.000000 | 0.000000 |

Cell-level analysis:

| system | reference_tables | reference_cells | exact_cells | exact_rate | missing_cells | mean_cer |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| candidate | 60 | 720 | 720 | 1.000000 | 0 | 0.000000 |
| Surya | 60 | 720 | 705 | 0.979167 | 0 | 0.005208 |
| Tesseract | 60 | 720 | 0 | 0.000000 | 715 | 0.999868 |

Runtime on k2:

| system | mean latency seconds | median latency seconds | max latency seconds |
| --- | ---: | ---: | ---: |
| candidate | 3.017929 | 2.967943 | 5.104267 |
| Surya | 2.478481 | 2.453125 | 3.175586 |
| Tesseract | 0.146866 | 0.134735 | 0.301695 |

Artifacts:

- `data/eval-pack-table-ledger-claim-v1/manifests/ocrtech-eval-pack.jsonl`
- `outputs/audit-table-ledger-claim-v1/eval-pack-audit.md`
- `outputs/experiment-table-ledger-claim-v2/benchmark/report.json`
- `outputs/summary-table-ledger-claim-v2/summary.md`
- `outputs/table-cell-analysis-table-ledger-claim-v2/table-cell-analysis.md`
- `outputs/validation-table-ledger-claim-v2b/validation.md`

Interpretation:

- This is now statistically validated evidence that the lexicon-assisted candidate improves table cell-content extraction over Surya on generated table-ledger pages while preserving Surya's already-perfect simple grid structure.
- It is not a broad OCR claim. Page CER on this targeted pack is diagnostic only because the parser does not currently render structured table blocks into `Document.text`, while the generated references include visible table rows in `text`.
- The candidate remains lexicon-assisted; the paper must state that the table-name correction uses a small curated lexicon and suffix-preserving matching for generated IDs such as `किरण-302`.
- `table_structure_similarity` can score rowspans/colspans when references use structured cells, but this table-ledger pack is simple-grid only and the metric is still not true TEDS over merged-cell HTML trees.

The eval-pack generator now includes a `complex-table-ledger` template that writes span-aware references with a merged header (`colspan=2`). This makes the validation path capable of catching a model that gets flattened cell text right but loses table span structure. We have not yet run a k2 claim experiment on this template; that should be the next table-specific benchmark before discussing complex-table behavior in the paper.

## Complex-Table Pack V1

We generated and benchmarked a targeted complex-table pack:

```bash
ocrtech create-eval-pack \
  --out data/eval-pack-complex-table-v1 \
  --input-format image \
  --template complex-table-ledger \
  --count-per-template 10 \
  --variant-offset 500 \
  --degradation clean \
  --degradation scan \
  --degradation blur \
  --degradation low_contrast
```

The pack has `40` samples and span-aware references with a merged header. It audited cleanly but is targeted development evidence, not broad claim-ready coverage.

Validation config: `configs/validation-complex-table-surya.json`.

Validation result: `validated` under a deliberately modest contract: improve cell-content F1 and do not regress exact/span structure against Surya.

| gate | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| table_cell_f1 | 40 | 8 | 0 | 32 | 0.928333 | 0.915000 | 0.013333 | 0.005000 |
| table_exact_match | 40 | 0 | 0 | 40 | 0.000000 | 0.000000 | 0.000000 | 0.000000 |
| table_structure_similarity | 40 | 0 | 0 | 40 | 0.875000 | 0.875000 | 0.000000 | 0.000000 |

Cell analysis:

| system | reference_cells | exact_cells | exact_rate | mean_cer |
| --- | ---: | ---: | ---: | ---: |
| candidate | 600 | 557 | 0.928333 | 0.067667 |
| Surya | 600 | 549 | 0.915000 | 0.071000 |
| Tesseract | 600 | 40 | 0.066667 | 0.933333 |

Artifacts:

- `data/eval-pack-complex-table-v1/manifests/ocrtech-eval-pack.jsonl`
- `outputs/audit-complex-table-v1/eval-pack-audit.md`
- `outputs/experiment-complex-table-v1/benchmark/report.json`
- `outputs/summary-complex-table-v1/summary.md`
- `outputs/table-cell-analysis-complex-table-v1/table-cell-analysis.md`
- `outputs/validation-complex-table-v1/validation.md`

Interpretation:

- The candidate improves some table cell text over Surya because the lexicon correction now works below grouped headers.
- Neither candidate nor Surya reconstructs the merged-header span; both have `table_exact_match = 0` and `table_structure_similarity = 0.875`.
- This is not a complex-table SOTA result. It is a concrete failure analysis result showing the next engineering target: preserve or infer rowspans/colspans in parser output.

## Complex-Table Span-Inference V1

After adding repeated-header colspan inference to the candidate parser, we reran the same complex-table pack:

```bash
ocrtech run-experiment \
  --eval-manifest data/eval-pack-complex-table-v1/manifests/ocrtech-eval-pack.jsonl \
  --out outputs/experiment-complex-table-span-infer-v1 \
  --baselines candidate,surya,tesseract \
  --candidate-model-config outputs/figure-caption-table-lexicon-v1/model-card.json \
  --validation-config configs/validation-complex-table-surya.json
```

Then:

```bash
ocrtech summarize-benchmark \
  --benchmark-report outputs/experiment-complex-table-span-infer-v1/benchmark/report.json \
  --out outputs/summary-complex-table-span-infer-v1 \
  --baseline surya \
  --baseline tesseract \
  --metric table_cell_f1 \
  --metric table_exact_match \
  --metric table_structure_similarity

ocrtech analyze-table-cells \
  --benchmark-report outputs/experiment-complex-table-span-infer-v1/benchmark/report.json \
  --eval-manifest data/eval-pack-complex-table-v1/manifests/ocrtech-eval-pack.jsonl \
  --out outputs/table-cell-analysis-complex-table-span-infer-v1 \
  --baseline surya \
  --baseline tesseract
```

Validation result: `validated`.

| gate | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| table_cell_f1 | 40 | 40 | 0 | 0 | 0.995000 | 0.915000 | 0.080000 | 0.071667 |
| table_exact_match | 40 | 37 | 0 | 3 | 0.925000 | 0.000000 | 0.925000 | 0.825000 |
| table_structure_similarity | 40 | 40 | 0 | 0 | 1.000000 | 0.875000 | 0.125000 | 0.125000 |

Cell analysis:

| system | reference_cells | exact_cells | exact_rate | mean_cer |
| --- | ---: | ---: | ---: | ---: |
| candidate | 600 | 597 | 0.995000 | 0.001000 |
| Surya | 600 | 549 | 0.915000 | 0.071000 |
| Tesseract | 600 | 40 | 0.066667 | 0.933333 |

Artifacts:

- `outputs/experiment-complex-table-span-infer-v1/benchmark/report.json`
- `outputs/experiment-complex-table-span-infer-v1/validation/validation.md`
- `outputs/summary-complex-table-span-infer-v1/summary.md`
- `outputs/table-cell-analysis-complex-table-span-infer-v1/table-cell-analysis.md`

Interpretation:

- The candidate now reconstructs the generated merged header pattern via a narrow repeated-header span inference rule.
- This is a parser/postprocessing improvement, not a trained model improvement.
- It does not solve broad table understanding. We still need real scanned table benchmarks, varied spans/rowspans, and true HTML/TEDS scoring.

## Complex-Table TEDS-Style Metrics V1

We added two deterministic ordered table-tree edit metrics:

- `table_teds_structure`: compares `table/tr/td` tree shape plus `rowspan`/`colspan`
- `table_teds_text`: also includes normalized cell text in `td` node labels

Then we reran the same complex-table pack with stricter validation gates:

```bash
ocrtech run-experiment \
  --eval-manifest data/eval-pack-complex-table-v1/manifests/ocrtech-eval-pack.jsonl \
  --out outputs/experiment-complex-table-teds-v1 \
  --baselines candidate,surya,tesseract \
  --candidate-model-config outputs/figure-caption-table-lexicon-v1/model-card.json \
  --validation-config configs/validation-complex-table-surya.json
```

Validation result: `validated`.

| gate | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| table_cell_f1 | 40 | 40 | 0 | 0 | 0.995000 | 0.915000 | 0.080000 | 0.071667 |
| table_exact_match | 40 | 37 | 0 | 3 | 0.925000 | 0.000000 | 0.925000 | 0.825000 |
| table_structure_similarity | 40 | 40 | 0 | 0 | 1.000000 | 0.875000 | 0.125000 | 0.125000 |
| table_teds_structure | 40 | 40 | 0 | 0 | 1.000000 | 0.904762 | 0.095238 | 0.095238 |
| table_teds_text | 40 | 40 | 0 | 0 | 0.996250 | 0.891667 | 0.104583 | 0.098690 |

Cell analysis is unchanged from the span-inference run:

| system | reference_cells | exact_cells | exact_rate | mean_cer |
| --- | ---: | ---: | ---: | ---: |
| candidate | 600 | 597 | 0.995000 | 0.001000 |
| Surya | 600 | 549 | 0.915000 | 0.071000 |
| Tesseract | 600 | 40 | 0.066667 | 0.933333 |

Artifacts:

- `outputs/experiment-complex-table-teds-v1/benchmark/report.json`
- `outputs/experiment-complex-table-teds-v1/validation/validation.md`
- `outputs/summary-complex-table-teds-v1/summary.md`
- `outputs/table-cell-analysis-complex-table-teds-v1/table-cell-analysis.md`

Interpretation:

- This is stronger than the old span-token score because it evaluates ordered table tree structure.
- It is still an internal deterministic metric over a generated pack. A broad paper claim still needs public benchmark scoring and real scanned tables with diverse merged-cell patterns.

## Quality Report

Every `parse-doc` output now includes:

```text
quality.json
```

The quality report tracks:

- line/block/table counts
- average and minimum OCR confidence
- missing confidence rates
- low confidence rates
- malformed table/block warnings
- `quality_score`
- fallback trigger/success/failure diagnostics when `metadata.fallback` exists

`quality_score` is useful as a diagnostic gate. It is not a substitute for CER/WER, table metrics, reading order metrics, or public benchmark results.

Figure/caption extraction is scored separately with `figure_detection_f1` and `figure_caption_cer` when structured references include `figures`.

Figure artifacts are exported explicitly. If an engine returns `figure.image_path`,
the parser copies that image into `figures/`; if only a figure bbox is available
and the source is a regular image or page-bundle directory, it crops
`figures/{figure_id}.png` when Pillow is installed. For bundles, it uses
`figure.page_index` to select the page image. When an artifact cannot be
produced, the figure metadata gets `metadata.export_warnings`. This makes figure
detections auditable as visual artifacts instead of only JSON boxes.

Selective fallback usage is exposed in benchmark reports as diagnostic metrics:

- `fallback_triggered`
- `fallback_succeeded`
- `fallback_failed`
- `fallback_primary_average_confidence`
- `fallback_threshold`
- `fallback_primary_quality_score`
- `fallback_min_quality_score`

These are not quality scores by themselves. They are audit fields that make it
possible to report how much of a claimed system came from the local engine and
how much came from a fallback such as PaddleOCR-VL.

Claim review can also cap `fallback_triggered` by slice with
`max_candidate_fallback_trigger_rate_by_slice`. Use this for SOTA runs to keep
the VLM fallback selective: high fallback use on figure or hard document slices
is a different claim from high fallback use on ordinary English/Nepali text
pages.

Benchmark rows now also include an `output_artifacts` object for successful
runs. It records the path, SHA-256, and byte size for exported files such as
`document.json`, `document.md`, `quality.json`, `tables/*.csv`,
`tables/*.html`, `figures/metadata.json`, and figure image crops. This does not
change any metric, but it makes every scored row point at the concrete
structured artifacts behind the score.

Benchmark rows also include `resource_metrics` when the host exposes process
resource usage. The current fields are `process_max_rss_mb` and
`process_max_rss_delta_mb`; `summarize-benchmark` aggregates them so paper
tables can report memory alongside `latency_seconds`. This is process RSS
telemetry, not GPU memory telemetry.

For CUDA or VLM benchmark hosts, pass `--capture-gpu-metrics` to sample
`nvidia-smi` before and after each row. When available, rows also record
`gpu_memory_used_mb`, `gpu_memory_total_mb`, `gpu_memory_used_delta_mb`, and
`gpu_utilization_percent`.