File size: 54,321 Bytes
d82db76
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
# Paper Results

This file is the shortest path from repo artifacts to a paper draft.

Use [PAPER_NOTES.md](PAPER_NOTES.md) for claim boundaries and framing.
Use this file for the concrete result tables and the branch chronology.

## Current Comparison Tables

## Engine Guardrails Added After These Tables

### 2026-06-16 - Preserve Explicit Table Spans During Grid Cleanup

The `figure_caption_refine_composite` branch normalizes noisy flat table grids
by removing separator columns, duplicate columns, and sparse continuation rows.
That cleanup is useful for simple OCR table noise, but it is unsafe once a table
module has already emitted real `rowspan` or `colspan` structure. Rebuilding
from a plain grid would erase the very merged-header structure that the paper
metrics now score.

Implementation change:

- `normalize_table_grid_structure` now skips tables with explicit row/column
  spans and records `normalized_table_grid_skipped_reason="explicit_spans"`.
- The table helper and candidate composite tests now verify that `colspan`
  survives when grid normalization is enabled.

This is an engine guardrail, not a new accuracy result. It protects future
complex-table benchmarks from a known postprocessing failure mode.

### HF Recognizer Next V1 Admission

Generated artifact:

- [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/k2-hf-recognizer-next-v1-combined-full/summary/summary.md)
- [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/k2-hf-recognizer-next-v1-combined-full/summary/summary.json)
- [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-hf-recognizer-next-v1-admission/validation.md)
- [validation config](/Users/cdjk/github/llm/ocr-tech/configs/validation-hf-recognizer-next-v1-admission.json)

Coverage:

- `244` held-out eval samples
- candidate, Tesseract, and stock PaddleOCR completed `244/244`
- Surya completed `243/244`; one English synthetic line crop timed out at `240s`
- admission validation status: `validated`

Headline paired CER table:

| baseline | slice | pairs | candidate CER | baseline CER | abs improvement | wins | losses | ties |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| Tesseract | all | 180 | 0.226628 | 0.329012 | 0.102384 | 74 | 54 | 52 |
| Tesseract | Nepali | 30 | 0.633814 | 0.757175 | 0.123360 | 18 | 12 | 0 |
| stock PaddleOCR | all | 180 | 0.226628 | 0.656675 | 0.430048 | 159 | 10 | 11 |
| stock PaddleOCR | Nepali | 30 | 0.633814 | 0.945114 | 0.311300 | 24 | 2 | 4 |
| Surya | all | 179 | 0.223364 | 3.307235 | 3.083871 | 119 | 42 | 18 |
| Surya | Nepali | 30 | 0.633814 | 0.930927 | 0.297113 | 10 | 20 | 0 |

Interpretation:

- This is the first trained recognizer branch in the repo that clears a held-out OCR admission gate against Tesseract and stock PaddleOCR.
- Treat the Surya comparison as adapter-specific evidence. The English line-crop Surya CER is suspiciously high, and the Nepali slice has lower mean CER for the candidate but worse paired win count against Surya.
- This result supports "promising recognizer component"; it does not support "complete document-understanding engine" or SOTA.

### HF Recognizer Next V1 Table-Cell Refinement Smoke

Generated artifact:

- [model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/hf-table-cell-refine-next-v1/model-card.json)
- [model-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-hf-table-cell-refine-next-v1-model/model-audit.md)
- [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-hf-table-cell-refine-next-v1-smoke/summary.md)
- [benchmark report.json](/Users/cdjk/github/llm/ocr-tech/outputs/experiment-hf-table-cell-refine-next-v1-smoke/report.json)

One-sample hard-table smoke:

| system | CER | table_cell_f1 | table_teds_text | reading_order_pair_accuracy | latency_seconds |
| --- | ---: | ---: | ---: | ---: | ---: |
| HF table-cell refine candidate | 0.363636 | 0.000000 | 0.294118 | 0.100000 | 9.888044 |
| Surya | 0.363636 | 1.000000 | 1.000000 | 0.100000 | 5.529877 |

Interpretation:

- The trained HF recognizer now runs as a leaf OCR engine inside the document pipeline.
- Unconditional HF replacement is not viable: on `eval-table-ledger-000-clean`, it replaced all `12` table cells and destroyed an already-correct Surya table.
- Do not scale this branch as-is. Any future HF table refinement needs a guarded replacement policy or a crop-specific recognizer, not forced replacement.

### Rendered Claim Pack

Generated artifact:

- [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-table/paper-table.md)
- [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-table/paper-table.json)

Headline table:

| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
| --- | ---: | ---: | ---: | ---: | ---: |
| Candidate | 25 | 0.248654 | 0.955556 | 0.764000 | 3.268937 |
| Surya | 25 | 0.195260 | 0.955556 | 0.750909 | 2.618872 |
| Tesseract | 25 | 0.689214 | 0.000000 | 0.816000 | 0.240963 |
| Stock Paddle | 25 | 1.749081 | 0.000000 | 1.000000 | 7.732034 |

Interpretation:

- Candidate beats Tesseract and stock-Paddle badly on text CER and table extraction.
- Candidate ties Surya on aggregate `table_cell_f1`.
- Candidate is slightly better than Surya on aggregate reading-order accuracy.
- Candidate still loses to raw Surya on text CER.
- Candidate is also slower than Tesseract and somewhat slower than Surya on this pack.

Latency caveat:

- these latency means come from separate benchmark reports on the same host and pack, then get pinned per system during table export
- treat them as comparable operational signals, not as a single synchronized runtime shootout

### Rendered Claim Pack V2 Holdout

Generated artifact:

- [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-table/paper-table.md)
- [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-table/paper-table.json)
- [preflight.md](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-script-router/preflight.md)

Headline table:

| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: | ---: |
| Candidate | 50 | 0.180893 | 0.947500 | 0.750909 |
| Surya | 50 | 0.180893 | 0.970833 | 0.750909 |
| Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 |

Interpretation:

- This is the stronger rendered-document holdout because calibration and holdout were separated at `50/50`.
- The routed candidate does not beat Surya here. It ties Surya exactly on text CER and reading-order accuracy.
- The routed candidate is slightly worse than raw Surya on aggregate `table_cell_f1`.
- The calibration artifact selected `primary_script_threshold=0.0`, which means the routed candidate collapsed to "always pick Surya" on the calibration split.
- That is why claim preflight now rejects this candidate as a routed model fit for a broad "better engine" claim.

### Rendered Claim Pack V2 Holdout - Surya Primary Latin Router

Generated artifact:

- [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-latin-table/paper-table.md)
- [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-latin-table/paper-table.json)

Headline table:

| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
| --- | ---: | ---: | ---: | ---: | ---: |
| Candidate | 50 | 0.416854 | 0.337500 | 0.761818 | 6.067455 |
| Surya | 50 | 0.180893 | 0.970833 | 0.750909 | 6.699851 |
| Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 | 0.238464 |

Interpretation:

- This candidate was a Surya-primary, Latin-triggered page router with Tesseract fallback.
- It was meant to preserve Surya's text strength while using Tesseract for English-heavy pages.
- It failed badly on the rendered-v2 holdout: worse than raw Surya on text CER and much worse on table structure.
- Paired CER against Surya was decisive enough to rule the branch out: `0` wins, `12` losses, `38` ties.

### Rendered Claim Pack V2 Holdout - Surya Quality Router

Generated artifact:

- [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-quality-router-table/paper-table.md)
- [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-quality-router-table/paper-table.json)
- [calibration.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-surya-quality-router/calibration.json)

Headline table:

| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
| --- | ---: | ---: | ---: | ---: | ---: |
| Candidate | 50 | 0.181128 | 0.947500 | 0.750909 | 4.620400 |
| Surya | 50 | 0.180893 | 0.947500 | 0.750909 | 5.603801 |
| Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 | 0.240773 |

Interpretation:

- This candidate was a Surya-vs-Tesseract quality selector calibrated on the rendered-v2 calibration split.
- Calibration selected `selected_threshold=0.0`, which means the quality gate kept Surya whenever it was not scored worse than Tesseract.
- On the untouched holdout, the candidate still did not beat raw Surya on text CER.
- Paired CER against Surya was effectively a tie with one bad miss: `0` wins, `1` loss, `49` ties.
- This is operationally closer to "Surya with a guardrail" than to a meaningfully new OCR engine.

### Rendered Claim Pack V2 Holdout - Line Align Composite

Generated artifact:

- [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-line-align-table/paper-table.md)
- [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-line-align-table/paper-table.json)
- [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-line-align-summary/summary.json)

Headline table:

| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
| --- | ---: | ---: | ---: | ---: | ---: |
| Candidate | 50 | 0.310549 | 0.897500 | 0.792727 | 4.142242 |
| Surya | 50 | 0.180893 | 0.970833 | 0.750909 | 3.223913 |
| Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 | 0.235074 |

Interpretation:

- This candidate kept Surya as the text engine and reused Tesseract line scaffolding through `line_align_composite`.
- It did improve reading-order accuracy relative to raw Surya, and even edged past raw Tesseract on this holdout.
- That gain is not enough. CER regressed badly versus Surya, and aggregate `table_cell_f1` also regressed.
- Paired CER against Surya was one-sided enough to kill the branch: `0` wins, `7` losses, `43` ties.
- This is a useful negative result for the paper because it shows that better line ordering alone does not rescue the Surya/Tesseract composite family.

### Rendered Claim Pack V2 Holdout - Sample-Level Oracle

Generated artifact:

- [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-oracle-table/paper-table.md)
- [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-oracle-table/paper-table.json)
- [oracle-analysis.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-analysis.json)
- [oracle-choices.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-choices.json)

Headline table:

| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
| --- | ---: | ---: | ---: | ---: | ---: |
| Oracle Surya/Tesseract | 50 | 0.180529 | 0.970833 | 0.750909 | 3.193115 |
| Surya | 50 | 0.180893 | 0.970833 | 0.750909 | 3.223913 |
| Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 | 0.235074 |

Interpretation:

- This is not a deployable system. It is an upper-bound analysis that picks the better of raw Surya and raw Tesseract per sample using ground-truth CER.
- The oracle picked Surya for `49` of `50` holdout samples and Tesseract for only `1`.
- Per-sample oracle selections are recorded separately in `oracle-choices.json`; they are provenance, not benchmark metrics.
- The absolute CER gain over raw Surya is only `0.000364`.
- That means the current Surya/Tesseract routing space has almost no text-OCR headroom left on rendered-v2.
- Future work should move to a stronger recognizer/document model, not more Surya/Tesseract routing heuristics.

### Rendered Claim Pack V2 Holdout - Surya Error Pockets

Generated artifact:

- [error-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-error-analysis/error-analysis.md)
- [top-failures.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-error-analysis/top-failures.json)
- [surya-vs-tesseract error-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-vs-tesseract-error-analysis/error-analysis.md)
- [surya-vs-oracle error-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-vs-oracle-error-analysis/error-analysis.md)
- [eval-surya-hard15.jsonl](/Users/cdjk/github/llm/ocr-tech/data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl)
- [validation-hard15-surya.json](/Users/cdjk/github/llm/ocr-tech/configs/validation-hard15-surya.json)
- [line-align hard15 validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-claimpack-v2-line-align-hard15/validation.md)
- [k2 manifest-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-claimpack-v2-surya-hard15-k2/manifest-audit.md)
- [table/figure dev pack manifest](/Users/cdjk/github/llm/ocr-tech/data/eval-pack-table-figure-dev-v1/manifests/ocrtech-eval-pack.jsonl)
- [table/figure dev pack audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-eval-pack-table-figure-dev-v1-k2/eval-pack-audit.md)

Diagnostic summary:

| slice | sample_count | Surya CER |
| --- | ---: | ---: |
| figure | 20 | 0.253157 |
| table | 20 | 0.252781 |
| multi_page | 10 | 0.196472 |
| scan | 10 | 0.194861 |
| low_contrast | 10 | 0.193317 |
| blur | 10 | 0.190940 |
| clean | 10 | 0.182256 |

Interpretation:

- Surya's hardest absolute CER pockets on rendered-v2 are table-ledger and figure-caption samples, not generic blur/skew alone.
- Tesseract is worse than Surya on `49` of `50` samples and better on only `eval-form-notice-003-blur`; that single oracle win is too small to justify more Surya/Tesseract routing work.
- A ranked `15`-sample `hard_eval` manifest now exists for the next model branch; it contains `8` table samples and `7` figure samples, and its full file/reference audit passes on `k2`.
- `configs/validation-hard15-surya.json` defines the pocket gate a future candidate must clear against Surya, including intersection gates for `hard_eval,table` and `hard_eval,figure`.
- The current line-align candidate fails the hard15 gate: hard-eval CER is `0.482039` versus Surya `0.348706`, and hard table CER is `0.613636` versus Surya `0.363636`.
- A separate `80`-sample image development pack now exists for table/figure work: `40` table samples, `40` figure samples, all five degradations. It has zero sample-id overlap and zero text-hash overlap with hard15, and its `k2` audit passes.
- That development pack is not claim evidence; it is for model iteration before returning to the held-out hard15 and rendered-v2 gates.
- The next credible recognizer branch should target table/figure-heavy mixed-script pages and preserve Surya's already-strong behavior on the other `49` samples.

### Table/Figure Development Pack - HF TrOCR Negative Result

Generated artifact:

- [split-summary.md](/Users/cdjk/github/llm/ocr-tech/data/splits-table-figure-dev-v1/split-summary.md)
- [training_metrics.json](/Users/cdjk/github/llm/ocr-tech/runs/hf-table-figure-dev-v1/training_metrics.json)
- [model-card.json](/Users/cdjk/github/llm/ocr-tech/runs/hf-table-figure-dev-v1/package/model-card.json)
- [model-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-hf-table-figure-dev-v1-model/model-audit.md)
- [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-hf-table-figure-dev-v1/summary.md)
- [benchmark report.json](/Users/cdjk/github/llm/ocr-tech/outputs/experiment-hf-table-figure-dev-v1/benchmark/report.json)
- [paddle_recognizer.yml](/Users/cdjk/github/llm/ocr-tech/runs/paddle-table-figure-dev-v1/paddle_recognizer.yml)

Pack and split:

- `80` image samples: `40` table and `40` figure
- degradations: `clean`, `scan`, `blur`, `low_contrast`, `skew`
- split: `60` train / `20` eval
- grouping: `text_sha256`
- text groups: `16`
- split leakage: pass
- train/eval manifest audits on `k2`: pass
- hard15 overlap: `0` sample IDs and `0` text hashes

Training result for `microsoft/trocr-small-printed` after one epoch:

| metric | value |
| --- | ---: |
| eval_loss | 7.884517 |
| eval_cer | 3.062119 |
| eval_wer | 3.562689 |
| eval_exact_line_accuracy | 0.000000 |

Benchmark result on the `20`-sample dev eval:

| system | CER | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: |
| HF table/figure candidate | 2.422001 | 0.000000 | 0.000000 |
| Surya | 0.563399 | 0.858333 | 0.550000 |
| Tesseract | 0.546227 | 0.000000 | 0.695000 |

Interpretation:

- The model package audits cleanly, so this is a quality failure rather than a packaging failure.
- Paired CER was `0` wins, `20` losses, and `0` ties against Surya.
- Paired CER was also `0` wins, `20` losses, and `0` ties against Tesseract.
- Naive full-page TrOCR fine-tuning is not a serious next step for table/figure-heavy document pages.
- The next model branch should use line/cell crops or a structure-aware page/document model rather than treating full document images as single recognition targets.

### Table/Figure Crop Development Pack - HF TrOCR Negative Result

Generated artifact:

- [crop-manifest-summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/crops-table-figure-dev-v1-k2/crop-manifest-summary.md)
- [crop manifest audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/crops-table-figure-dev-v1-k2/manifest-audit.md)
- [crop split-summary.md](/Users/cdjk/github/llm/ocr-tech/data/splits-crops-table-figure-dev-v1/split-summary.md)
- [training_metrics.json](/Users/cdjk/github/llm/ocr-tech/runs/hf-crops-table-figure-dev-v1/training_metrics.json)
- [model-card.json](/Users/cdjk/github/llm/ocr-tech/runs/hf-crops-table-figure-dev-v1/package/model-card.json)
- [model-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-hf-crops-table-figure-dev-v1-model/model-audit.md)
- [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-hf-crops-table-figure-dev-v1/summary.md)
- [benchmark report.json](/Users/cdjk/github/llm/ocr-tech/outputs/experiment-hf-crops-table-figure-dev-v1/benchmark/report.json)

Pack and split:

- source rendered samples: `80`
- extracted crops: `544`
- crop types: `384` table cells, `128` lines, `32` figure captions
- skipped candidates: `32` skew pages because pre-rotation bboxes are invalid
- split: `480` train / `64` eval
- grouping: `text_sha256`
- text groups: `68`
- crop manifest audit on `k2`: pass
- split leakage: pass

Training result for `microsoft/trocr-small-printed` after one crop-level epoch:

| metric | value |
| --- | ---: |
| eval_loss | 3.087561 |
| eval_cer | 0.959831 |
| eval_wer | 1.363715 |
| eval_exact_line_accuracy | 0.078125 |

Benchmark result on the `64`-sample crop eval:

| system | CER |
| --- | ---: |
| HF crop candidate | 1.043369 |
| Surya | 0.475309 |
| Tesseract | 0.077453 |

Interpretation:

- Crop-level training is a better problem shape than full-page recognition, but this recognizer is still not competitive.
- Paired CER against Surya was `7` wins, `50` losses, and `7` ties.
- Paired CER against Tesseract was `0` wins, `64` losses, and `0` ties.
- Tesseract is extremely strong on these rendered crop images, so near-term gains are more likely from better crop/layout detection plus strong crop OCR than from this small synthetic TrOCR fine-tune.

### Hard15 - Table Cell Refinement Composite

Generated artifact:

- [unsafe model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-refine-v1/model-card.json)
- [unsafe summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-table-cell-refine-hard15-v1/summary.md)
- [unsafe cell-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-analysis-refine-always-hard15-v1/table-cell-analysis.md)
- [safe model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-refine-empty-v1/model-card.json)
- [safe summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-table-cell-refine-empty-hard15-v1/summary.md)
- [safe cell-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-analysis-refine-empty-hard15-v1/table-cell-analysis.md)
- [safe validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-table-cell-refine-empty-hard15-v1/validation.md)
- [safe+dedup model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-refine-empty-dedup-v1/model-card.json)
- [safe+dedup summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-table-cell-refine-empty-dedup-hard15-v3/summary.md)
- [safe+dedup cell-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-analysis-refine-empty-dedup-hard15-v3/table-cell-analysis.md)
- [safe+dedup validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-table-cell-refine-empty-dedup-hard15-v3/validation.md)

Headline hard15 comparison:

| system | hard_eval CER | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: |
| Surya baseline | 0.348706 | 0.979167 | 0.520000 |
| Table-cell refine, `always` | 0.348706 | 0.593750 | 0.520000 |
| Table-cell refine, `empty` | 0.348706 | 0.979167 | 0.520000 |
| Table-cell refine, `empty` + parser de-dup | 0.348706 | 0.979167 | 0.520000 |

Interpretation:

- The unsafe table-cell crop replacement policy damaged Surya's already-good table cells.
- The conservative empty-cell-only policy avoided regression but did not improve any hard15 metric.
- The safe policy's hard15 validation status is `not_validated`.
- The parser de-dup branch removes duplicate overlapping table exports for both candidate and Surya, so it improves artifact cleanliness but does not create a candidate advantage.
- Exact crop OCR being strong is not enough; the deployment problem is reliable cell detection and knowing when a replacement is safer than the structure engine's existing cell text.

Cell-level result:

- Surya is already exact on `94/96` hard15 reference table cells.
- Unsafe replacement is exact on only `57/96` cells.
- Unsafe replacement improves `0` cells over Surya and worsens `37`.
- Safe+dedup is exact on `94/96` cells, with `8` predicted tables for `8` reference tables and `0` duplicate overlapping tables on the v3 rerun.

### Hard15 - Figure Caption Refinement Composite

Generated artifact:

- [model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/figure-caption-refine-v1/model-card.json)
- [model-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-figure-caption-refine-v1-model/model-audit.md)
- [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-figure-caption-refine-hard15-v2/summary.md)
- [benchmark report.json](/Users/cdjk/github/llm/ocr-tech/outputs/experiment-figure-caption-refine-hard15-v2/benchmark/report.json)
- [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-figure-caption-refine-hard15-v2/validation.md)
- [labels model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/figure-caption-refine-labels-v1/model-card.json)
- [labels summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-figure-caption-refine-labels-hard15-v1/summary.md)
- [labels validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-figure-caption-refine-labels-hard15-v1/validation.md)

Headline hard15 comparison after the parser caption-attachment fix:

| system | CER | figure_caption_cer | figure_detection_f1 | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: | ---: | ---: |
| Surya | 0.236839 | 0.092323 | 1.000000 | 0.979167 | 0.520000 |
| Tesseract | 1.266364 | 0.051020 | 0.857143 | 0.000000 | 0.613333 |
| Figure-caption refine | 0.228008 | 0.052478 | 1.000000 | 0.979167 | 0.520000 |
| Figure-caption refine + labels | 0.204610 | 0.008746 | 1.000000 | 0.979167 | 0.520000 |

Validation result:

- basic branch overall status: `not_validated`
- basic branch figure-caption gate: pass, `6` wins / `0` losses / `1` tie against Surya, improvement `0.039845`
- label branch overall status: `not_validated`
- label branch overall hard-eval CER gate: pass, improvement `0.032228`
- label branch figure CER gate: pass, improvement `0.069061`
- label branch figure-caption gate: pass, `7` wins / `0` losses / `0` ties against Surya, improvement `0.083576`
- label branch table gates: fail, ties against Surya

Interpretation:

- This is a real slice win over Surya on figure-caption extraction.
- It is not broad enough for a SOTA claim.
- Tesseract has better caption text but worse figure detection/table structure; the composite keeps Surya's structure and selectively borrows caption text.
- Safe empty-only replacement ties all `96` cells with Surya because there are no missing Surya cells in this pocket.
- The only remaining Surya cell errors are `किरण` predicted as `करिण` on `eval-table-ledger-002-clean` and `eval-table-ledger-002-scan`.

### Rendered Blur-Only Expensive-Baseline Subset

Generated artifact:

- [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-blur5-table/paper-table.md)
- [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-blur5-table/paper-table.json)

Headline table:

| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
| --- | ---: | ---: | ---: | ---: | ---: |
| Candidate | 5 | 0.223273 | 1.000000 | 0.783636 | 3.784279 |
| Surya | 5 | 0.197601 | 1.000000 | 0.750909 | 2.943875 |
| Tesseract | 5 | 0.732364 | 0.000000 | 0.823636 | 0.220515 |
| Stock Paddle | 5 | 2.810426 | 0.000000 | 1.000000 | 7.440482 |
| GLM-OCR | 5 | 2.820269 | 0.000000 | 1.000000 | 16.846619 |
| PaddleOCR-VL | 5 | 18.470663 | 0.000000 | 0.750909 | 32.793166 |

Interpretation:

- Candidate is materially stronger than Tesseract, stock-Paddle, GLM-OCR, and PaddleOCR-VL on this degraded rendered subset.
- Candidate still does not beat raw Surya on text CER.
- The external VLM-style baselines are also much slower than the current candidate here.

## Branch Chronology

This is the shortest honest story of the project so far.

| branch | eval setting | candidate CER | baseline CER | paired result | verdict | evidence |
| --- | --- | ---: | ---: | --- | --- | --- |
| HF TrOCR recognizer | mixed Nepali/English line-style holdout, 180 samples | 0.586139 | 0.329012 vs Tesseract | 36 wins, 97 losses, 47 ties | runnable but not competitive | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-hf-vs-tesseract-summary/summary.json) |
| Tesseract + ByT5 corrector | same 180-sample mixed holdout | 6.912065 | 0.329012 vs Tesseract | 0 wins, 170 losses, 10 ties | failed badly | [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-corrector-vs-tesseract-summary/summary.md) |
| quality router (`psm=6` vs `psm=13`) | 91-sample mixed holdout | 0.340608 | 0.342234 vs Tesseract | 2 wins, 1 loss, 88 ties | tiny gain, plausible baseline only | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-quality-router-holdout/summary.json) |
| ranked PSM ensemble (`6/7/11/13`) | same 91-sample mixed holdout | 0.358352 | 0.342234 vs Tesseract | 9 wins, 16 losses, 66 ties | overfit, worse than baseline | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-psm-ensemble-holdout/summary.json) |
| script router (`tesseract -> surya`) | same 91-sample mixed holdout | 0.313084 | 0.342234 vs Tesseract | 9 wins, 0 losses, 82 ties | strongest older line-style result | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-script-router-holdout/summary.json) |
| script router + structure backfill | rendered claim-ready document pack, 25 samples | 0.248654 | 0.689214 vs Tesseract | 9 wins, 1 loss, 15 ties | strong document-pack improvement over Tesseract | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-rendered-backfill/summary.json) |
| raw Surya on rendered claim pack | rendered claim-ready document pack, 25 samples | 0.195260 | 0.248654 vs Candidate | candidate loses on text CER | current text leader on rendered pack | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-surya-compare/summary.json) |
| rendered-v2 script router | rendered claim-ready document holdout, 50 samples | 0.180893 | 0.180893 vs Surya | exact CER tie, worse table structure | routed candidate collapses to Surya on calibration; not a meaningful new engine | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-script-router/summary.json) |
| rendered-v2 Surya-primary Latin router | rendered claim-ready document holdout, 50 samples | 0.416854 | 0.180893 vs Surya | 0 wins, 12 losses, 38 ties on CER | failed badly; inverted script routing is not a viable answer | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-surya-latin-router/summary.json) |
| rendered-v2 Surya quality router | rendered claim-ready document holdout, 50 samples | 0.181128 | 0.180893 vs Surya | 0 wins, 1 loss, 49 ties on CER | nearly indistinguishable from Surya and still not better | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-surya-quality-router/summary.json) |
| rendered-v2 line-align composite | rendered claim-ready document holdout, 50 samples | 0.310549 | 0.180893 vs Surya | 0 wins, 7 losses, 43 ties on CER | better reading order, but text and table regressions are too large | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-line-align-summary/summary.json) |
| rendered-v2 Surya/Tesseract sample oracle | rendered claim-ready document holdout, 50 samples | 0.180529 | 0.180893 vs Surya | oracle picks Tesseract only 1/50 samples | upper bound shows routing headroom is essentially exhausted | [oracle-analysis.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-analysis.json) |
| HF TrOCR table/figure dev | table/figure development eval, 20 samples | 2.422001 | 0.563399 vs Surya; 0.546227 vs Tesseract | 0 wins, 20 losses, 0 ties vs both | failed badly; full-page TrOCR fine-tuning is not viable here | [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-hf-table-figure-dev-v1/summary.md) |
| HF TrOCR crop dev | table/figure crop eval, 64 samples | 1.043369 | 0.475309 vs Surya; 0.077453 vs Tesseract | 7 wins, 50 losses, 7 ties vs Surya; 0 wins, 64 losses vs Tesseract | crop shape helps but this recognizer still fails | [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-hf-crops-table-figure-dev-v1/summary.md) |
| table-cell refine, unsafe | hard15 rendered-v2 pocket, 15 samples | 0.348706 | 0.348706 vs Surya | text tie, table F1 regresses from 0.979167 to 0.593750 | replacement policy is unsafe | [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-table-cell-refine-hard15-v1/summary.md) |
| table-cell refine, empty-only | hard15 rendered-v2 pocket, 15 samples | 0.348706 | 0.348706 vs Surya | exact tie across CER/table/reading order; validation not_validated | safe but no improvement | [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-table-cell-refine-empty-hard15-v1/validation.md) |
| normalized script router, hard-timeout rerun | normalized mixed Nepali/English line-style eval, 180 samples | 0.354840 | 0.329012 vs Tesseract; 3.157317 vs Surya on 178 scored Surya rows | 17 wins, 4 losses, 159 ties on CER vs Tesseract but worse aggregate mean; 2 Surya rows timed out at 45s | not a better engine; benchmark watchdog works and routing still loses overall CER | [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/k2-mixed-v1-normalized-script-router-hard-timeout/outputs/summary-mixed-v1-normalized-script-router-hard-timeout/summary.md) |
| PaddleOCR recognizer hard-50 failure slice | top 50 failures from AWS G6 trained PaddleOCR recognizer, remapped to normalized k2 eval paths | n/a | n/a | 50 failures: 38 English line-crop synthetic, 12 Nepali synthetic; k2 manifest audit passed | next recognizer gate; do not train on it | [manifest-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/k2-paddle-mixed-v2-hard50-normalized/outputs/manifest-audit-paddle-mixed-v2-cuda-hard50-normalized/manifest-audit.md) |
| PaddleOCR hard-50 gate negative control | failed AWS G6 trained PaddleOCR recognizer evaluated on the hard-50 gate | 0.887557 | 0.080249 vs Tesseract | fails all four hard gates; English line-crop CER `0.852048` vs `0.031244` | validates the gate catches the known recognizer collapse | [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-paddle-mixed-v2-cuda-hard50-tesseract-negative/validation.md) |
| line-level Tesseract/Paddle composite | hard-50 failure slice, Tesseract primary with PaddleOCR secondary only for Devanagari-looking lines | 0.276489 | 0.120106 vs Tesseract | English line-crop ties exactly (`0.060429` CER); Nepali collapses (`0.960679` vs `0.309083` CER, `1.000000` vs `0.711607` WER) | safe English preservation but failed Devanagari override; do not promote this recognizer | [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-line-script-paddle-hard50-v2/validation.md) |
| PaddleOCR v3 CUDA recognizer | G6-trained lower-LR PaddleOCR branch on hard-50 admission gate | 0.921416 | 0.080249 vs Tesseract | 0 wins, 50 losses on CER; English line-crop CER `0.917511` vs `0.031244`; Nepali WER `1.055556` vs `0.716369` | failed worse than v2; stop branch before full mixed eval | [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/aws-g6-paddle-mixed-v3-cuda/outputs/validation-paddle-mixed-v3-cuda-hard50-rescored/validation.md) |
| PaddleOCR v4 CUDA recognizer | G6-trained Devanagari PP-OCRv3 branch, LR `0.00005`, 20 epochs, evaluated on hard-50 admission gate | 0.940500 | 0.080249 vs Tesseract | 0 wins, 50 losses on CER; English line-crop CER `0.950730` vs `0.031244`; Nepali CER `0.908108` vs `0.235433` | failed worse than v2/v3; the same PaddleOCR mixed-script fine-tune branch should not be the engine core | [artifact bundle](/Users/cdjk/github/llm/ocr-tech/outputs/aws-g6-paddle-mixed-v4-cuda/ocrtech-paddle-mixed-v4-cuda-artifacts.tgz) |

## Current Best-Evidence Statement

The strongest evidence-backed statement available today is:

1. The current Tesseract-plus-Surya backfill candidate is a real document-OCR improvement over Tesseract and several weaker baselines on internal rendered Nepali/English document packs.
2. On the larger rendered-v2 holdout, every Tesseract/Surya composite tested so far either collapses to Surya, trails Surya, becomes effectively indistinguishable from Surya, or trades too much OCR accuracy for modest ordering gains.
3. On the cleaned 180-sample mixed line-style eval with hard per-sample timeouts, the older script-router candidate also loses to Tesseract overall (`0.354840` vs `0.329012` CER), despite a small overall WER improvement (`0.469944` vs `0.486485`) and a Nepali WER improvement.
4. The trained PaddleOCR recognizer failure profile is now materialized as a normalized hard-50 eval gate: all top failures are synthetic line crops, mostly English (`38/50`), so the next recognizer branch must prove English preservation instead of only improving Nepali.
5. The line-level Tesseract/Paddle composite proves the candidate shape can preserve English while routing Devanagari-looking lines, but the available trained PaddleOCR recognizer is too weak on the Nepali hard slice to use as an override (`0.960679` vs Tesseract `0.309083` CER).
6. The second AWS G6 PaddleOCR attempt fixed recipe/runtime issues but failed the hard-50 gate even worse than the first (`0.921416` vs Tesseract `0.080249` CER), so this PP-OCRv3 Devanagari recipe family is not the engine path.
7. A sample-level oracle over raw Surya and raw Tesseract has almost no text-OCR headroom over Surya (`0.180529` vs `0.180893` CER), so more routing in this component family is not a serious path to SOTA.
8. The repo now has enough validation discipline and artifact traceability to support a serious comparative paper, but not a SOTA claim.

## Regeneration Workflow

Rebuild the generated comparison tables with:

```bash
ocrtech summarize-benchmark \
  --benchmark-report outputs/live-k2-claimpack-rendered-backfill/report.json \
  --out outputs/paper-claimpack-summary-tesseract \
  --candidate candidate \
  --baseline tesseract \
  --metric cer
```

```bash
ocrtech summarize-benchmark \
  --benchmark-report outputs/live-k2-claimpack-surya-compare/report.json \
  --out outputs/paper-claimpack-summary-surya \
  --candidate candidate \
  --baseline surya \
  --metric cer
```

```bash
ocrtech summarize-benchmark \
  --benchmark-report outputs/live-k2-claimpack-stock-paddle-fixed/report.json \
  --out outputs/paper-claimpack-summary-stock-paddle \
  --candidate candidate \
  --baseline stock-paddle \
  --metric cer
```

```bash
ocrtech paper-benchmark-table \
  --summary outputs/paper-claimpack-summary-tesseract/summary.json \
  --summary outputs/paper-claimpack-summary-surya/summary.json \
  --summary outputs/paper-claimpack-summary-stock-paddle/summary.json \
  --out outputs/paper-claimpack-table \
  --system candidate \
  --system surya \
  --system tesseract \
  --system stock-paddle \
  --label 'candidate=Candidate' \
  --label 'surya=Surya' \
  --label 'tesseract=Tesseract' \
  --label 'stock-paddle=Stock Paddle' \
  --source-preference 'candidate=outputs/paper-claimpack-summary-tesseract/summary.json' \
  --source-preference 'tesseract=outputs/paper-claimpack-summary-tesseract/summary.json' \
  --source-preference 'surya=outputs/paper-claimpack-summary-surya/summary.json' \
  --source-preference 'stock-paddle=outputs/paper-claimpack-summary-stock-paddle/summary.json' \
  --metric cer \
  --metric table_cell_f1 \
  --metric reading_order_pair_accuracy \
  --metric latency_seconds
```

```bash
ocrtech summarize-benchmark \
  --benchmark-report outputs/live-k2-rendered-blur5/report.json \
  --out outputs/paper-blur5-summary \
  --candidate candidate \
  --baseline tesseract \
  --baseline surya \
  --baseline stock-paddle \
  --baseline glm-ocr \
  --baseline paddleocr-vl \
  --metric cer
```

```bash
ocrtech paper-benchmark-table \
  --summary outputs/paper-blur5-summary/summary.json \
  --out outputs/paper-blur5-table \
  --system candidate \
  --system surya \
  --system tesseract \
  --system stock-paddle \
  --system glm-ocr \
  --system paddleocr-vl \
  --label 'candidate=Candidate' \
  --label 'surya=Surya' \
  --label 'tesseract=Tesseract' \
  --label 'stock-paddle=Stock Paddle' \
  --label 'glm-ocr=GLM-OCR' \
  --label 'paddleocr-vl=PaddleOCR-VL' \
  --metric cer \
  --metric table_cell_f1 \
  --metric reading_order_pair_accuracy \
  --metric latency_seconds
```

```bash
ocrtech paper-benchmark-table \
  --summary outputs/live-k2-claimpack-v2-script-router/summary.json \
  --out outputs/paper-claimpack-v2-table \
  --system candidate \
  --system surya \
  --system tesseract \
  --label 'candidate=Candidate' \
  --label 'surya=Surya' \
  --label 'tesseract=Tesseract' \
  --metric cer \
  --metric table_cell_f1 \
  --metric reading_order_pair_accuracy \
  --title 'Rendered Claim Pack V2 Holdout Comparison'
```

```bash
ocrtech paper-benchmark-table \
  --summary outputs/live-k2-claimpack-v2-surya-quality-router/summary.json \
  --out outputs/paper-claimpack-v2-surya-quality-router-table \
  --system candidate \
  --system surya \
  --system tesseract \
  --label 'candidate=Candidate' \
  --label 'surya=Surya' \
  --label 'tesseract=Tesseract' \
  --metric cer \
  --metric table_cell_f1 \
  --metric reading_order_pair_accuracy \
  --metric latency_seconds \
  --title 'Rendered Claim Pack V2 Holdout - Surya Quality Router'
```

```bash
ocrtech paper-benchmark-table \
  --summary outputs/live-k2-claimpack-v2-line-align-summary/summary.json \
  --out outputs/paper-claimpack-v2-line-align-table \
  --system candidate \
  --system surya \
  --system tesseract \
  --label 'candidate=Candidate' \
  --label 'surya=Surya' \
  --label 'tesseract=Tesseract' \
  --metric cer \
  --metric table_cell_f1 \
  --metric reading_order_pair_accuracy \
  --metric latency_seconds \
  --title 'Rendered Claim Pack V2 Holdout - Line Align Composite'
```

```bash
ocrtech benchmark-oracle \
  --benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \
  --out outputs/paper-claimpack-v2-surya-tesseract-oracle \
  --system surya \
  --system tesseract \
  --metric cer \
  --oracle-name oracle-surya-tesseract

ocrtech paper-benchmark-table \
  --summary outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-summary/summary.json \
  --out outputs/paper-claimpack-v2-oracle-table \
  --system oracle-surya-tesseract \
  --system surya \
  --system tesseract \
  --label 'oracle-surya-tesseract=Oracle Surya/Tesseract' \
  --label 'surya=Surya' \
  --label 'tesseract=Tesseract' \
  --metric cer \
  --metric table_cell_f1 \
  --metric reading_order_pair_accuracy \
  --metric latency_seconds \
  --title 'Rendered Claim Pack V2 Holdout - Sample-Level Oracle'
```

```bash
ocrtech benchmark-error-analysis \
  --benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \
  --out outputs/paper-claimpack-v2-surya-error-analysis \
  --system surya \
  --metric cer \
  --top-n 15

ocrtech benchmark-error-analysis \
  --benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \
  --out outputs/paper-claimpack-v2-surya-vs-tesseract-error-analysis \
  --system surya \
  --baseline tesseract \
  --metric cer \
  --top-n 15

ocrtech benchmark-error-analysis \
  --benchmark-report outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-report.json \
  --out outputs/paper-claimpack-v2-surya-vs-oracle-error-analysis \
  --system surya \
  --baseline oracle-surya-tesseract \
  --metric cer \
  --top-n 10

ocrtech export-failure-manifest \
  --source-manifest data/splits-claimpack-rendered-v2/eval.jsonl \
  --top-failures outputs/paper-claimpack-v2-surya-error-analysis/top-failures.json \
  --out data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl \
  --split hard_eval

ocrtech validate-claim \
  --benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \
  --config configs/validation-hard15-surya.json \
  --eval-manifest data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl \
  --out outputs/validation-claimpack-v2-line-align-hard15

ocrtech create-eval-pack \
  --out data/eval-pack-table-figure-dev-v1 \
  --input-format image \
  --template table-ledger \
  --template figure-caption \
  --count-per-template 8 \
  --variant-offset 100 \
  --degradation clean \
  --degradation scan \
  --degradation blur \
  --degradation low_contrast \
  --degradation skew
```

Memory still is not part of these generated tables. The current benchmark artifact format does not record it.

## Hard15 Figure-Caption + Table-Lexicon Candidate

Latest artifact set:

- `outputs/figure-caption-table-lexicon-v1/model-card.json`
- `outputs/experiment-figure-caption-table-lexicon-hard15-v3/benchmark/report.json`
- `outputs/summary-figure-caption-table-lexicon-hard15-v3/summary.md`
- `outputs/table-cell-analysis-figure-caption-table-lexicon-hard15-v3/table-cell-analysis.md`
- `outputs/validation-figure-caption-table-lexicon-hard15-v3/validation.md`

Headline table for internal hard15 only:

| system | CER | figure_caption_cer | figure_detection_f1 | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: | ---: | ---: |
| candidate | 0.010671 | 0.008746 | 1.000000 | 1.000000 | 0.520000 |
| Surya | 0.236839 | 0.092323 | 1.000000 | 0.979167 | 0.520000 |
| Tesseract | 1.266364 | 0.051020 | 0.857143 | 0.000000 | 0.613333 |

Validation status is still `not_validated`.

Passed configured Surya gates:

- overall hard_eval CER: `15` wins / `0` losses / `0` ties
- table-slice CER: `8` wins / `0` losses / `0` ties
- figure-slice CER: `7` wins / `0` losses / `0` ties
- figure-caption CER: `7` wins / `0` losses / `0` ties

Remaining configured failure:

- table-cell F1: candidate `1.000000`, Surya `0.979167`, but bootstrap CI low is `0.000000` because there are only 2 improving table pages and 6 ties

Cell-level table evidence:

| system | reference_cells | exact_cells | exact_rate | mean_cer |
| --- | ---: | ---: | ---: | ---: |
| candidate | 96 | 96 | 1.000000 | 0.000000 |
| Surya | 96 | 94 | 0.979167 | 0.010417 |
| Tesseract | 96 | 0 | 0.000000 | 1.000000 |

Use this result as an internal milestone, not a paper claim of broad SOTA. The candidate is lexicon-assisted for table names and uses guarded structural-title corrections (`वार्षिक प्रतिवेदन`, `दैनिक हिसाब`, `चित्र`, `निष्कर्ष`). A publishable table claim needs a larger independent table pack so confidence intervals are meaningful.

## Targeted Table-Ledger Validation Pack V2

Artifacts:

- `data/eval-pack-table-ledger-claim-v1/manifests/ocrtech-eval-pack.jsonl`
- `outputs/audit-table-ledger-claim-v1/eval-pack-audit.md`
- `outputs/experiment-table-ledger-claim-v2/benchmark/report.json`
- `outputs/summary-table-ledger-claim-v2/summary.md`
- `outputs/table-cell-analysis-table-ledger-claim-v2/table-cell-analysis.md`
- `outputs/validation-table-ledger-claim-v2b/validation.md`
- `configs/validation-table-ledger-surya.json`

Pack:

- template: `table-ledger`
- samples: `60`
- cells: `720`
- degradations: clean, scan, blur, low_contrast, skew
- variant offset: `300`
- claim-ready broad coverage: no, targeted table-only pack

Validation status: `validated`.

| metric | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| table_cell_f1 | 60 | 15 | 0 | 45 | 1.000000 | 0.979167 | 0.020833 | 0.012500 |
| table_exact_match | 60 | 15 | 0 | 45 | 1.000000 | 0.750000 | 0.250000 | 0.150000 |
| table_structure_similarity | 60 | 0 | 0 | 60 | 1.000000 | 1.000000 | 0.000000 | 0.000000 |

Cell exactness:

| system | reference_cells | exact_cells | exact_rate | mean_cell_cer |
| --- | ---: | ---: | ---: | ---: |
| candidate | 720 | 720 | 1.000000 | 0.000000 |
| Surya | 720 | 705 | 0.979167 | 0.005208 |
| Tesseract | 720 | 0 | 0.000000 | 0.999868 |

Runtime on k2:

| system | mean latency seconds | median latency seconds |
| --- | ---: | ---: |
| candidate | 3.017929 | 2.967943 |
| Surya | 2.478481 | 2.453125 |
| Tesseract | 0.146866 | 0.134735 |

Paper-safe claim text:

> On a targeted 60-page generated Nepali-English table-ledger validation pack, the lexicon-assisted candidate improved structured table extraction over Surya, reaching 720/720 exact cells versus Surya's 705/720, and passed bootstrap-gated validation for both table-cell F1 and table exact match.

More precise paper-safe wording:

> On a targeted 60-page generated Nepali-English table-ledger validation pack, the lexicon-assisted candidate improved table cell-content extraction over Surya, reaching 720/720 exact cells versus Surya's 705/720, while matching Surya's perfect grid-structure score.

Do not generalize this to all tables. The pack has no merged cells, no nested headers, and no real scanned government tables. The scoring path now supports span-aware structured references, but this pack does not exercise that capability and the metric is still not true HTML-tree TEDS.

## Targeted Complex-Table Validation Pack V1

Artifacts:

- `data/eval-pack-complex-table-v1/manifests/ocrtech-eval-pack.jsonl`
- `outputs/audit-complex-table-v1/eval-pack-audit.md`
- `outputs/experiment-complex-table-v1/benchmark/report.json`
- `outputs/summary-complex-table-v1/summary.md`
- `outputs/table-cell-analysis-complex-table-v1/table-cell-analysis.md`
- `outputs/validation-complex-table-v1/validation.md`
- `configs/validation-complex-table-surya.json`

Pack:

- template: `complex-table-ledger`
- samples: `40`
- reference cells: `600`
- span reference: merged header with `colspan=2`
- degradations: clean, scan, blur, low_contrast
- broad claim-ready coverage: no

Validation status: `validated` for cell-content improvement plus structure non-regression.

| metric | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| table_cell_f1 | 40 | 8 | 0 | 32 | 0.928333 | 0.915000 | 0.013333 | 0.005000 |
| table_exact_match | 40 | 0 | 0 | 40 | 0.000000 | 0.000000 | 0.000000 | 0.000000 |
| table_structure_similarity | 40 | 0 | 0 | 40 | 0.875000 | 0.875000 | 0.000000 | 0.000000 |

Cell exactness:

| system | reference_cells | exact_cells | exact_rate | mean_cell_cer |
| --- | ---: | ---: | ---: | ---: |
| candidate | 600 | 557 | 0.928333 | 0.067667 |
| Surya | 600 | 549 | 0.915000 | 0.071000 |
| Tesseract | 600 | 40 | 0.066667 | 0.933333 |

Paper-safe claim text:

> On a targeted generated complex-table pack with span-aware references, the lexicon-assisted candidate improved table cell-content F1 over Surya but did not improve merged-header reconstruction; both systems failed exact table match and tied on span-aware structure similarity.

This is a negative result for complex table structure. The next model/parser branch must preserve or infer rowspans/colspans before we can claim complex-table document understanding.

## Targeted Complex-Table Span-Inference Pack V1

After adding conservative repeated-header colspan inference, we reran the same `40`-sample generated complex-table pack on k2.

Artifacts:

- `outputs/experiment-complex-table-span-infer-v1/benchmark/report.json`
- `outputs/experiment-complex-table-span-infer-v1/validation/validation.md`
- `outputs/summary-complex-table-span-infer-v1/summary.md`
- `outputs/table-cell-analysis-complex-table-span-infer-v1/table-cell-analysis.md`

Validation status: `validated`.

| metric | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| table_cell_f1 | 40 | 40 | 0 | 0 | 0.995000 | 0.915000 | 0.080000 | 0.071667 |
| table_exact_match | 40 | 37 | 0 | 3 | 0.925000 | 0.000000 | 0.925000 | 0.825000 |
| table_structure_similarity | 40 | 40 | 0 | 0 | 1.000000 | 0.875000 | 0.125000 | 0.125000 |

Cell exactness:

| system | reference_cells | exact_cells | exact_rate | mean_cell_cer |
| --- | ---: | ---: | ---: | ---: |
| candidate | 600 | 597 | 0.995000 | 0.001000 |
| Surya | 600 | 549 | 0.915000 | 0.071000 |
| Tesseract | 600 | 40 | 0.066667 | 0.933333 |

Paper-safe claim text:

> On a targeted generated complex-table pack with one repeated merged-header pattern, a conservative span-inference postprocessor improved table structure extraction over Surya, raising span-aware structure similarity from 0.875 to 1.000 and table exact match from 0.000 to 0.925.

Do not generalize this to all complex tables. This is generated data with a single span pattern and a heuristic parser repair. The next paper-grade check needs real scanned tables with varied merged cells, rowspans, nested headers, and a true HTML/TEDS scorer.

## Targeted Complex-Table TEDS-Style Metrics V1

We added deterministic ordered table-tree edit metrics and reran the same complex-table pack with these gates included.

Artifacts:

- `outputs/experiment-complex-table-teds-v1/benchmark/report.json`
- `outputs/experiment-complex-table-teds-v1/validation/validation.md`
- `outputs/summary-complex-table-teds-v1/summary.md`
- `outputs/table-cell-analysis-complex-table-teds-v1/table-cell-analysis.md`

Validation status: `validated`.

| metric | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| table_cell_f1 | 40 | 40 | 0 | 0 | 0.995000 | 0.915000 | 0.080000 | 0.071667 |
| table_exact_match | 40 | 37 | 0 | 3 | 0.925000 | 0.000000 | 0.925000 | 0.825000 |
| table_structure_similarity | 40 | 40 | 0 | 0 | 1.000000 | 0.875000 | 0.125000 | 0.125000 |
| table_teds_structure | 40 | 40 | 0 | 0 | 1.000000 | 0.904762 | 0.095238 | 0.095238 |
| table_teds_text | 40 | 40 | 0 | 0 | 0.996250 | 0.891667 | 0.104583 | 0.098690 |

Paper-safe claim text:

> On a targeted generated complex-table pack, the candidate improved ordered table-tree metrics over Surya, reaching 1.000 structure-only TEDS-style similarity and 0.996 text-sensitive TEDS-style similarity, compared with Surya's 0.905 and 0.892 respectively.

This is still not a public-benchmark SOTA claim. The metrics are deterministic and closer to table-structure scoring, but the pack is generated and covers one repeated-header merged-cell pattern.

## HF Recognizer Next V1 Admission Benchmark

The AWS G6 trained HF recognizer was evaluated on the `recognizer-next-v1`
held-out eval manifest against Tesseract on `k2`.

Artifacts:

- `outputs/k2-hf-recognizer-next-v1-full-cpu/benchmark/report.json`
- `outputs/k2-hf-recognizer-next-v1-full-cpu/summary/summary.md`
- candidate model card:
  `outputs/aws-g6-hf-recognizer-next-v1/package/model-card-cpu.json`

Run status:

- candidate rows: `244` ok, `0` errors
- Tesseract rows: `244` ok, `0` errors
- paired CER rows with text references: `180`

| slice | pairs | candidate CER | Tesseract CER | improvement | wins | losses | ties |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| all | 180 | 0.226628 | 0.329012 | 0.102384 | 74 | 54 | 52 |
| english/en/line_crop | 150 | 0.145190 | 0.243379 | 0.098189 | 56 | 42 | 52 |
| nepali | 30 | 0.633814 | 0.757175 | 0.123360 | 18 | 12 | 0 |
| real | 15 | 0.832255 | 1.162917 | 0.330662 | 12 | 3 | 0 |
| synthetic | 165 | 0.171571 | 0.253202 | 0.081631 | 62 | 51 | 52 |

Paper-safe claim text:

> On the held-out `recognizer-next-v1` text-reference slices, the trained HF
> recognizer improves paired CER over Tesseract, reducing aggregate CER from
> 0.329 to 0.227 across 180 paired samples.

Do not call this SOTA. The result is a recognizer admission benchmark against
Tesseract only; the branch still needs Surya, stock PaddleOCR, GLM-OCR, public
document benchmarks, table structure metrics, reading-order validation, and
document reconstruction gates.