File size: 54,321 Bytes
f9bc0c1 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 | # Paper Results
This file is the shortest path from repo artifacts to a paper draft.
Use [PAPER_NOTES.md](PAPER_NOTES.md) for claim boundaries and framing.
Use this file for the concrete result tables and the branch chronology.
## Current Comparison Tables
## Engine Guardrails Added After These Tables
### 2026-06-16 - Preserve Explicit Table Spans During Grid Cleanup
The `figure_caption_refine_composite` branch normalizes noisy flat table grids
by removing separator columns, duplicate columns, and sparse continuation rows.
That cleanup is useful for simple OCR table noise, but it is unsafe once a table
module has already emitted real `rowspan` or `colspan` structure. Rebuilding
from a plain grid would erase the very merged-header structure that the paper
metrics now score.
Implementation change:
- `normalize_table_grid_structure` now skips tables with explicit row/column
spans and records `normalized_table_grid_skipped_reason="explicit_spans"`.
- The table helper and candidate composite tests now verify that `colspan`
survives when grid normalization is enabled.
This is an engine guardrail, not a new accuracy result. It protects future
complex-table benchmarks from a known postprocessing failure mode.
### HF Recognizer Next V1 Admission
Generated artifact:
- [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/k2-hf-recognizer-next-v1-combined-full/summary/summary.md)
- [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/k2-hf-recognizer-next-v1-combined-full/summary/summary.json)
- [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-hf-recognizer-next-v1-admission/validation.md)
- [validation config](/Users/cdjk/github/llm/ocr-tech/configs/validation-hf-recognizer-next-v1-admission.json)
Coverage:
- `244` held-out eval samples
- candidate, Tesseract, and stock PaddleOCR completed `244/244`
- Surya completed `243/244`; one English synthetic line crop timed out at `240s`
- admission validation status: `validated`
Headline paired CER table:
| baseline | slice | pairs | candidate CER | baseline CER | abs improvement | wins | losses | ties |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| Tesseract | all | 180 | 0.226628 | 0.329012 | 0.102384 | 74 | 54 | 52 |
| Tesseract | Nepali | 30 | 0.633814 | 0.757175 | 0.123360 | 18 | 12 | 0 |
| stock PaddleOCR | all | 180 | 0.226628 | 0.656675 | 0.430048 | 159 | 10 | 11 |
| stock PaddleOCR | Nepali | 30 | 0.633814 | 0.945114 | 0.311300 | 24 | 2 | 4 |
| Surya | all | 179 | 0.223364 | 3.307235 | 3.083871 | 119 | 42 | 18 |
| Surya | Nepali | 30 | 0.633814 | 0.930927 | 0.297113 | 10 | 20 | 0 |
Interpretation:
- This is the first trained recognizer branch in the repo that clears a held-out OCR admission gate against Tesseract and stock PaddleOCR.
- Treat the Surya comparison as adapter-specific evidence. The English line-crop Surya CER is suspiciously high, and the Nepali slice has lower mean CER for the candidate but worse paired win count against Surya.
- This result supports "promising recognizer component"; it does not support "complete document-understanding engine" or SOTA.
### HF Recognizer Next V1 Table-Cell Refinement Smoke
Generated artifact:
- [model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/hf-table-cell-refine-next-v1/model-card.json)
- [model-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-hf-table-cell-refine-next-v1-model/model-audit.md)
- [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-hf-table-cell-refine-next-v1-smoke/summary.md)
- [benchmark report.json](/Users/cdjk/github/llm/ocr-tech/outputs/experiment-hf-table-cell-refine-next-v1-smoke/report.json)
One-sample hard-table smoke:
| system | CER | table_cell_f1 | table_teds_text | reading_order_pair_accuracy | latency_seconds |
| --- | ---: | ---: | ---: | ---: | ---: |
| HF table-cell refine candidate | 0.363636 | 0.000000 | 0.294118 | 0.100000 | 9.888044 |
| Surya | 0.363636 | 1.000000 | 1.000000 | 0.100000 | 5.529877 |
Interpretation:
- The trained HF recognizer now runs as a leaf OCR engine inside the document pipeline.
- Unconditional HF replacement is not viable: on `eval-table-ledger-000-clean`, it replaced all `12` table cells and destroyed an already-correct Surya table.
- Do not scale this branch as-is. Any future HF table refinement needs a guarded replacement policy or a crop-specific recognizer, not forced replacement.
### Rendered Claim Pack
Generated artifact:
- [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-table/paper-table.md)
- [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-table/paper-table.json)
Headline table:
| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
| --- | ---: | ---: | ---: | ---: | ---: |
| Candidate | 25 | 0.248654 | 0.955556 | 0.764000 | 3.268937 |
| Surya | 25 | 0.195260 | 0.955556 | 0.750909 | 2.618872 |
| Tesseract | 25 | 0.689214 | 0.000000 | 0.816000 | 0.240963 |
| Stock Paddle | 25 | 1.749081 | 0.000000 | 1.000000 | 7.732034 |
Interpretation:
- Candidate beats Tesseract and stock-Paddle badly on text CER and table extraction.
- Candidate ties Surya on aggregate `table_cell_f1`.
- Candidate is slightly better than Surya on aggregate reading-order accuracy.
- Candidate still loses to raw Surya on text CER.
- Candidate is also slower than Tesseract and somewhat slower than Surya on this pack.
Latency caveat:
- these latency means come from separate benchmark reports on the same host and pack, then get pinned per system during table export
- treat them as comparable operational signals, not as a single synchronized runtime shootout
### Rendered Claim Pack V2 Holdout
Generated artifact:
- [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-table/paper-table.md)
- [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-table/paper-table.json)
- [preflight.md](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-script-router/preflight.md)
Headline table:
| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: | ---: |
| Candidate | 50 | 0.180893 | 0.947500 | 0.750909 |
| Surya | 50 | 0.180893 | 0.970833 | 0.750909 |
| Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 |
Interpretation:
- This is the stronger rendered-document holdout because calibration and holdout were separated at `50/50`.
- The routed candidate does not beat Surya here. It ties Surya exactly on text CER and reading-order accuracy.
- The routed candidate is slightly worse than raw Surya on aggregate `table_cell_f1`.
- The calibration artifact selected `primary_script_threshold=0.0`, which means the routed candidate collapsed to "always pick Surya" on the calibration split.
- That is why claim preflight now rejects this candidate as a routed model fit for a broad "better engine" claim.
### Rendered Claim Pack V2 Holdout - Surya Primary Latin Router
Generated artifact:
- [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-latin-table/paper-table.md)
- [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-latin-table/paper-table.json)
Headline table:
| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
| --- | ---: | ---: | ---: | ---: | ---: |
| Candidate | 50 | 0.416854 | 0.337500 | 0.761818 | 6.067455 |
| Surya | 50 | 0.180893 | 0.970833 | 0.750909 | 6.699851 |
| Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 | 0.238464 |
Interpretation:
- This candidate was a Surya-primary, Latin-triggered page router with Tesseract fallback.
- It was meant to preserve Surya's text strength while using Tesseract for English-heavy pages.
- It failed badly on the rendered-v2 holdout: worse than raw Surya on text CER and much worse on table structure.
- Paired CER against Surya was decisive enough to rule the branch out: `0` wins, `12` losses, `38` ties.
### Rendered Claim Pack V2 Holdout - Surya Quality Router
Generated artifact:
- [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-quality-router-table/paper-table.md)
- [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-quality-router-table/paper-table.json)
- [calibration.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-surya-quality-router/calibration.json)
Headline table:
| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
| --- | ---: | ---: | ---: | ---: | ---: |
| Candidate | 50 | 0.181128 | 0.947500 | 0.750909 | 4.620400 |
| Surya | 50 | 0.180893 | 0.947500 | 0.750909 | 5.603801 |
| Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 | 0.240773 |
Interpretation:
- This candidate was a Surya-vs-Tesseract quality selector calibrated on the rendered-v2 calibration split.
- Calibration selected `selected_threshold=0.0`, which means the quality gate kept Surya whenever it was not scored worse than Tesseract.
- On the untouched holdout, the candidate still did not beat raw Surya on text CER.
- Paired CER against Surya was effectively a tie with one bad miss: `0` wins, `1` loss, `49` ties.
- This is operationally closer to "Surya with a guardrail" than to a meaningfully new OCR engine.
### Rendered Claim Pack V2 Holdout - Line Align Composite
Generated artifact:
- [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-line-align-table/paper-table.md)
- [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-line-align-table/paper-table.json)
- [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-line-align-summary/summary.json)
Headline table:
| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
| --- | ---: | ---: | ---: | ---: | ---: |
| Candidate | 50 | 0.310549 | 0.897500 | 0.792727 | 4.142242 |
| Surya | 50 | 0.180893 | 0.970833 | 0.750909 | 3.223913 |
| Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 | 0.235074 |
Interpretation:
- This candidate kept Surya as the text engine and reused Tesseract line scaffolding through `line_align_composite`.
- It did improve reading-order accuracy relative to raw Surya, and even edged past raw Tesseract on this holdout.
- That gain is not enough. CER regressed badly versus Surya, and aggregate `table_cell_f1` also regressed.
- Paired CER against Surya was one-sided enough to kill the branch: `0` wins, `7` losses, `43` ties.
- This is a useful negative result for the paper because it shows that better line ordering alone does not rescue the Surya/Tesseract composite family.
### Rendered Claim Pack V2 Holdout - Sample-Level Oracle
Generated artifact:
- [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-oracle-table/paper-table.md)
- [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-oracle-table/paper-table.json)
- [oracle-analysis.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-analysis.json)
- [oracle-choices.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-choices.json)
Headline table:
| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
| --- | ---: | ---: | ---: | ---: | ---: |
| Oracle Surya/Tesseract | 50 | 0.180529 | 0.970833 | 0.750909 | 3.193115 |
| Surya | 50 | 0.180893 | 0.970833 | 0.750909 | 3.223913 |
| Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 | 0.235074 |
Interpretation:
- This is not a deployable system. It is an upper-bound analysis that picks the better of raw Surya and raw Tesseract per sample using ground-truth CER.
- The oracle picked Surya for `49` of `50` holdout samples and Tesseract for only `1`.
- Per-sample oracle selections are recorded separately in `oracle-choices.json`; they are provenance, not benchmark metrics.
- The absolute CER gain over raw Surya is only `0.000364`.
- That means the current Surya/Tesseract routing space has almost no text-OCR headroom left on rendered-v2.
- Future work should move to a stronger recognizer/document model, not more Surya/Tesseract routing heuristics.
### Rendered Claim Pack V2 Holdout - Surya Error Pockets
Generated artifact:
- [error-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-error-analysis/error-analysis.md)
- [top-failures.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-error-analysis/top-failures.json)
- [surya-vs-tesseract error-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-vs-tesseract-error-analysis/error-analysis.md)
- [surya-vs-oracle error-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-vs-oracle-error-analysis/error-analysis.md)
- [eval-surya-hard15.jsonl](/Users/cdjk/github/llm/ocr-tech/data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl)
- [validation-hard15-surya.json](/Users/cdjk/github/llm/ocr-tech/configs/validation-hard15-surya.json)
- [line-align hard15 validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-claimpack-v2-line-align-hard15/validation.md)
- [k2 manifest-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-claimpack-v2-surya-hard15-k2/manifest-audit.md)
- [table/figure dev pack manifest](/Users/cdjk/github/llm/ocr-tech/data/eval-pack-table-figure-dev-v1/manifests/ocrtech-eval-pack.jsonl)
- [table/figure dev pack audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-eval-pack-table-figure-dev-v1-k2/eval-pack-audit.md)
Diagnostic summary:
| slice | sample_count | Surya CER |
| --- | ---: | ---: |
| figure | 20 | 0.253157 |
| table | 20 | 0.252781 |
| multi_page | 10 | 0.196472 |
| scan | 10 | 0.194861 |
| low_contrast | 10 | 0.193317 |
| blur | 10 | 0.190940 |
| clean | 10 | 0.182256 |
Interpretation:
- Surya's hardest absolute CER pockets on rendered-v2 are table-ledger and figure-caption samples, not generic blur/skew alone.
- Tesseract is worse than Surya on `49` of `50` samples and better on only `eval-form-notice-003-blur`; that single oracle win is too small to justify more Surya/Tesseract routing work.
- A ranked `15`-sample `hard_eval` manifest now exists for the next model branch; it contains `8` table samples and `7` figure samples, and its full file/reference audit passes on `k2`.
- `configs/validation-hard15-surya.json` defines the pocket gate a future candidate must clear against Surya, including intersection gates for `hard_eval,table` and `hard_eval,figure`.
- The current line-align candidate fails the hard15 gate: hard-eval CER is `0.482039` versus Surya `0.348706`, and hard table CER is `0.613636` versus Surya `0.363636`.
- A separate `80`-sample image development pack now exists for table/figure work: `40` table samples, `40` figure samples, all five degradations. It has zero sample-id overlap and zero text-hash overlap with hard15, and its `k2` audit passes.
- That development pack is not claim evidence; it is for model iteration before returning to the held-out hard15 and rendered-v2 gates.
- The next credible recognizer branch should target table/figure-heavy mixed-script pages and preserve Surya's already-strong behavior on the other `49` samples.
### Table/Figure Development Pack - HF TrOCR Negative Result
Generated artifact:
- [split-summary.md](/Users/cdjk/github/llm/ocr-tech/data/splits-table-figure-dev-v1/split-summary.md)
- [training_metrics.json](/Users/cdjk/github/llm/ocr-tech/runs/hf-table-figure-dev-v1/training_metrics.json)
- [model-card.json](/Users/cdjk/github/llm/ocr-tech/runs/hf-table-figure-dev-v1/package/model-card.json)
- [model-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-hf-table-figure-dev-v1-model/model-audit.md)
- [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-hf-table-figure-dev-v1/summary.md)
- [benchmark report.json](/Users/cdjk/github/llm/ocr-tech/outputs/experiment-hf-table-figure-dev-v1/benchmark/report.json)
- [paddle_recognizer.yml](/Users/cdjk/github/llm/ocr-tech/runs/paddle-table-figure-dev-v1/paddle_recognizer.yml)
Pack and split:
- `80` image samples: `40` table and `40` figure
- degradations: `clean`, `scan`, `blur`, `low_contrast`, `skew`
- split: `60` train / `20` eval
- grouping: `text_sha256`
- text groups: `16`
- split leakage: pass
- train/eval manifest audits on `k2`: pass
- hard15 overlap: `0` sample IDs and `0` text hashes
Training result for `microsoft/trocr-small-printed` after one epoch:
| metric | value |
| --- | ---: |
| eval_loss | 7.884517 |
| eval_cer | 3.062119 |
| eval_wer | 3.562689 |
| eval_exact_line_accuracy | 0.000000 |
Benchmark result on the `20`-sample dev eval:
| system | CER | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: |
| HF table/figure candidate | 2.422001 | 0.000000 | 0.000000 |
| Surya | 0.563399 | 0.858333 | 0.550000 |
| Tesseract | 0.546227 | 0.000000 | 0.695000 |
Interpretation:
- The model package audits cleanly, so this is a quality failure rather than a packaging failure.
- Paired CER was `0` wins, `20` losses, and `0` ties against Surya.
- Paired CER was also `0` wins, `20` losses, and `0` ties against Tesseract.
- Naive full-page TrOCR fine-tuning is not a serious next step for table/figure-heavy document pages.
- The next model branch should use line/cell crops or a structure-aware page/document model rather than treating full document images as single recognition targets.
### Table/Figure Crop Development Pack - HF TrOCR Negative Result
Generated artifact:
- [crop-manifest-summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/crops-table-figure-dev-v1-k2/crop-manifest-summary.md)
- [crop manifest audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/crops-table-figure-dev-v1-k2/manifest-audit.md)
- [crop split-summary.md](/Users/cdjk/github/llm/ocr-tech/data/splits-crops-table-figure-dev-v1/split-summary.md)
- [training_metrics.json](/Users/cdjk/github/llm/ocr-tech/runs/hf-crops-table-figure-dev-v1/training_metrics.json)
- [model-card.json](/Users/cdjk/github/llm/ocr-tech/runs/hf-crops-table-figure-dev-v1/package/model-card.json)
- [model-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-hf-crops-table-figure-dev-v1-model/model-audit.md)
- [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-hf-crops-table-figure-dev-v1/summary.md)
- [benchmark report.json](/Users/cdjk/github/llm/ocr-tech/outputs/experiment-hf-crops-table-figure-dev-v1/benchmark/report.json)
Pack and split:
- source rendered samples: `80`
- extracted crops: `544`
- crop types: `384` table cells, `128` lines, `32` figure captions
- skipped candidates: `32` skew pages because pre-rotation bboxes are invalid
- split: `480` train / `64` eval
- grouping: `text_sha256`
- text groups: `68`
- crop manifest audit on `k2`: pass
- split leakage: pass
Training result for `microsoft/trocr-small-printed` after one crop-level epoch:
| metric | value |
| --- | ---: |
| eval_loss | 3.087561 |
| eval_cer | 0.959831 |
| eval_wer | 1.363715 |
| eval_exact_line_accuracy | 0.078125 |
Benchmark result on the `64`-sample crop eval:
| system | CER |
| --- | ---: |
| HF crop candidate | 1.043369 |
| Surya | 0.475309 |
| Tesseract | 0.077453 |
Interpretation:
- Crop-level training is a better problem shape than full-page recognition, but this recognizer is still not competitive.
- Paired CER against Surya was `7` wins, `50` losses, and `7` ties.
- Paired CER against Tesseract was `0` wins, `64` losses, and `0` ties.
- Tesseract is extremely strong on these rendered crop images, so near-term gains are more likely from better crop/layout detection plus strong crop OCR than from this small synthetic TrOCR fine-tune.
### Hard15 - Table Cell Refinement Composite
Generated artifact:
- [unsafe model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-refine-v1/model-card.json)
- [unsafe summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-table-cell-refine-hard15-v1/summary.md)
- [unsafe cell-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-analysis-refine-always-hard15-v1/table-cell-analysis.md)
- [safe model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-refine-empty-v1/model-card.json)
- [safe summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-table-cell-refine-empty-hard15-v1/summary.md)
- [safe cell-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-analysis-refine-empty-hard15-v1/table-cell-analysis.md)
- [safe validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-table-cell-refine-empty-hard15-v1/validation.md)
- [safe+dedup model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-refine-empty-dedup-v1/model-card.json)
- [safe+dedup summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-table-cell-refine-empty-dedup-hard15-v3/summary.md)
- [safe+dedup cell-analysis.md](/Users/cdjk/github/llm/ocr-tech/outputs/table-cell-analysis-refine-empty-dedup-hard15-v3/table-cell-analysis.md)
- [safe+dedup validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-table-cell-refine-empty-dedup-hard15-v3/validation.md)
Headline hard15 comparison:
| system | hard_eval CER | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: |
| Surya baseline | 0.348706 | 0.979167 | 0.520000 |
| Table-cell refine, `always` | 0.348706 | 0.593750 | 0.520000 |
| Table-cell refine, `empty` | 0.348706 | 0.979167 | 0.520000 |
| Table-cell refine, `empty` + parser de-dup | 0.348706 | 0.979167 | 0.520000 |
Interpretation:
- The unsafe table-cell crop replacement policy damaged Surya's already-good table cells.
- The conservative empty-cell-only policy avoided regression but did not improve any hard15 metric.
- The safe policy's hard15 validation status is `not_validated`.
- The parser de-dup branch removes duplicate overlapping table exports for both candidate and Surya, so it improves artifact cleanliness but does not create a candidate advantage.
- Exact crop OCR being strong is not enough; the deployment problem is reliable cell detection and knowing when a replacement is safer than the structure engine's existing cell text.
Cell-level result:
- Surya is already exact on `94/96` hard15 reference table cells.
- Unsafe replacement is exact on only `57/96` cells.
- Unsafe replacement improves `0` cells over Surya and worsens `37`.
- Safe+dedup is exact on `94/96` cells, with `8` predicted tables for `8` reference tables and `0` duplicate overlapping tables on the v3 rerun.
### Hard15 - Figure Caption Refinement Composite
Generated artifact:
- [model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/figure-caption-refine-v1/model-card.json)
- [model-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/audit-figure-caption-refine-v1-model/model-audit.md)
- [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-figure-caption-refine-hard15-v2/summary.md)
- [benchmark report.json](/Users/cdjk/github/llm/ocr-tech/outputs/experiment-figure-caption-refine-hard15-v2/benchmark/report.json)
- [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-figure-caption-refine-hard15-v2/validation.md)
- [labels model-card.json](/Users/cdjk/github/llm/ocr-tech/outputs/figure-caption-refine-labels-v1/model-card.json)
- [labels summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-figure-caption-refine-labels-hard15-v1/summary.md)
- [labels validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-figure-caption-refine-labels-hard15-v1/validation.md)
Headline hard15 comparison after the parser caption-attachment fix:
| system | CER | figure_caption_cer | figure_detection_f1 | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: | ---: | ---: |
| Surya | 0.236839 | 0.092323 | 1.000000 | 0.979167 | 0.520000 |
| Tesseract | 1.266364 | 0.051020 | 0.857143 | 0.000000 | 0.613333 |
| Figure-caption refine | 0.228008 | 0.052478 | 1.000000 | 0.979167 | 0.520000 |
| Figure-caption refine + labels | 0.204610 | 0.008746 | 1.000000 | 0.979167 | 0.520000 |
Validation result:
- basic branch overall status: `not_validated`
- basic branch figure-caption gate: pass, `6` wins / `0` losses / `1` tie against Surya, improvement `0.039845`
- label branch overall status: `not_validated`
- label branch overall hard-eval CER gate: pass, improvement `0.032228`
- label branch figure CER gate: pass, improvement `0.069061`
- label branch figure-caption gate: pass, `7` wins / `0` losses / `0` ties against Surya, improvement `0.083576`
- label branch table gates: fail, ties against Surya
Interpretation:
- This is a real slice win over Surya on figure-caption extraction.
- It is not broad enough for a SOTA claim.
- Tesseract has better caption text but worse figure detection/table structure; the composite keeps Surya's structure and selectively borrows caption text.
- Safe empty-only replacement ties all `96` cells with Surya because there are no missing Surya cells in this pocket.
- The only remaining Surya cell errors are `किरण` predicted as `करिण` on `eval-table-ledger-002-clean` and `eval-table-ledger-002-scan`.
### Rendered Blur-Only Expensive-Baseline Subset
Generated artifact:
- [paper-table.md](/Users/cdjk/github/llm/ocr-tech/outputs/paper-blur5-table/paper-table.md)
- [paper-table.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-blur5-table/paper-table.json)
Headline table:
| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
| --- | ---: | ---: | ---: | ---: | ---: |
| Candidate | 5 | 0.223273 | 1.000000 | 0.783636 | 3.784279 |
| Surya | 5 | 0.197601 | 1.000000 | 0.750909 | 2.943875 |
| Tesseract | 5 | 0.732364 | 0.000000 | 0.823636 | 0.220515 |
| Stock Paddle | 5 | 2.810426 | 0.000000 | 1.000000 | 7.440482 |
| GLM-OCR | 5 | 2.820269 | 0.000000 | 1.000000 | 16.846619 |
| PaddleOCR-VL | 5 | 18.470663 | 0.000000 | 0.750909 | 32.793166 |
Interpretation:
- Candidate is materially stronger than Tesseract, stock-Paddle, GLM-OCR, and PaddleOCR-VL on this degraded rendered subset.
- Candidate still does not beat raw Surya on text CER.
- The external VLM-style baselines are also much slower than the current candidate here.
## Branch Chronology
This is the shortest honest story of the project so far.
| branch | eval setting | candidate CER | baseline CER | paired result | verdict | evidence |
| --- | --- | ---: | ---: | --- | --- | --- |
| HF TrOCR recognizer | mixed Nepali/English line-style holdout, 180 samples | 0.586139 | 0.329012 vs Tesseract | 36 wins, 97 losses, 47 ties | runnable but not competitive | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-hf-vs-tesseract-summary/summary.json) |
| Tesseract + ByT5 corrector | same 180-sample mixed holdout | 6.912065 | 0.329012 vs Tesseract | 0 wins, 170 losses, 10 ties | failed badly | [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-corrector-vs-tesseract-summary/summary.md) |
| quality router (`psm=6` vs `psm=13`) | 91-sample mixed holdout | 0.340608 | 0.342234 vs Tesseract | 2 wins, 1 loss, 88 ties | tiny gain, plausible baseline only | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-quality-router-holdout/summary.json) |
| ranked PSM ensemble (`6/7/11/13`) | same 91-sample mixed holdout | 0.358352 | 0.342234 vs Tesseract | 9 wins, 16 losses, 66 ties | overfit, worse than baseline | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-psm-ensemble-holdout/summary.json) |
| script router (`tesseract -> surya`) | same 91-sample mixed holdout | 0.313084 | 0.342234 vs Tesseract | 9 wins, 0 losses, 82 ties | strongest older line-style result | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-script-router-holdout/summary.json) |
| script router + structure backfill | rendered claim-ready document pack, 25 samples | 0.248654 | 0.689214 vs Tesseract | 9 wins, 1 loss, 15 ties | strong document-pack improvement over Tesseract | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-rendered-backfill/summary.json) |
| raw Surya on rendered claim pack | rendered claim-ready document pack, 25 samples | 0.195260 | 0.248654 vs Candidate | candidate loses on text CER | current text leader on rendered pack | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-surya-compare/summary.json) |
| rendered-v2 script router | rendered claim-ready document holdout, 50 samples | 0.180893 | 0.180893 vs Surya | exact CER tie, worse table structure | routed candidate collapses to Surya on calibration; not a meaningful new engine | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-script-router/summary.json) |
| rendered-v2 Surya-primary Latin router | rendered claim-ready document holdout, 50 samples | 0.416854 | 0.180893 vs Surya | 0 wins, 12 losses, 38 ties on CER | failed badly; inverted script routing is not a viable answer | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-surya-latin-router/summary.json) |
| rendered-v2 Surya quality router | rendered claim-ready document holdout, 50 samples | 0.181128 | 0.180893 vs Surya | 0 wins, 1 loss, 49 ties on CER | nearly indistinguishable from Surya and still not better | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-surya-quality-router/summary.json) |
| rendered-v2 line-align composite | rendered claim-ready document holdout, 50 samples | 0.310549 | 0.180893 vs Surya | 0 wins, 7 losses, 43 ties on CER | better reading order, but text and table regressions are too large | [summary.json](/Users/cdjk/github/llm/ocr-tech/outputs/live-k2-claimpack-v2-line-align-summary/summary.json) |
| rendered-v2 Surya/Tesseract sample oracle | rendered claim-ready document holdout, 50 samples | 0.180529 | 0.180893 vs Surya | oracle picks Tesseract only 1/50 samples | upper bound shows routing headroom is essentially exhausted | [oracle-analysis.json](/Users/cdjk/github/llm/ocr-tech/outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-analysis.json) |
| HF TrOCR table/figure dev | table/figure development eval, 20 samples | 2.422001 | 0.563399 vs Surya; 0.546227 vs Tesseract | 0 wins, 20 losses, 0 ties vs both | failed badly; full-page TrOCR fine-tuning is not viable here | [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-hf-table-figure-dev-v1/summary.md) |
| HF TrOCR crop dev | table/figure crop eval, 64 samples | 1.043369 | 0.475309 vs Surya; 0.077453 vs Tesseract | 7 wins, 50 losses, 7 ties vs Surya; 0 wins, 64 losses vs Tesseract | crop shape helps but this recognizer still fails | [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-hf-crops-table-figure-dev-v1/summary.md) |
| table-cell refine, unsafe | hard15 rendered-v2 pocket, 15 samples | 0.348706 | 0.348706 vs Surya | text tie, table F1 regresses from 0.979167 to 0.593750 | replacement policy is unsafe | [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/summary-table-cell-refine-hard15-v1/summary.md) |
| table-cell refine, empty-only | hard15 rendered-v2 pocket, 15 samples | 0.348706 | 0.348706 vs Surya | exact tie across CER/table/reading order; validation not_validated | safe but no improvement | [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-table-cell-refine-empty-hard15-v1/validation.md) |
| normalized script router, hard-timeout rerun | normalized mixed Nepali/English line-style eval, 180 samples | 0.354840 | 0.329012 vs Tesseract; 3.157317 vs Surya on 178 scored Surya rows | 17 wins, 4 losses, 159 ties on CER vs Tesseract but worse aggregate mean; 2 Surya rows timed out at 45s | not a better engine; benchmark watchdog works and routing still loses overall CER | [summary.md](/Users/cdjk/github/llm/ocr-tech/outputs/k2-mixed-v1-normalized-script-router-hard-timeout/outputs/summary-mixed-v1-normalized-script-router-hard-timeout/summary.md) |
| PaddleOCR recognizer hard-50 failure slice | top 50 failures from AWS G6 trained PaddleOCR recognizer, remapped to normalized k2 eval paths | n/a | n/a | 50 failures: 38 English line-crop synthetic, 12 Nepali synthetic; k2 manifest audit passed | next recognizer gate; do not train on it | [manifest-audit.md](/Users/cdjk/github/llm/ocr-tech/outputs/k2-paddle-mixed-v2-hard50-normalized/outputs/manifest-audit-paddle-mixed-v2-cuda-hard50-normalized/manifest-audit.md) |
| PaddleOCR hard-50 gate negative control | failed AWS G6 trained PaddleOCR recognizer evaluated on the hard-50 gate | 0.887557 | 0.080249 vs Tesseract | fails all four hard gates; English line-crop CER `0.852048` vs `0.031244` | validates the gate catches the known recognizer collapse | [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-paddle-mixed-v2-cuda-hard50-tesseract-negative/validation.md) |
| line-level Tesseract/Paddle composite | hard-50 failure slice, Tesseract primary with PaddleOCR secondary only for Devanagari-looking lines | 0.276489 | 0.120106 vs Tesseract | English line-crop ties exactly (`0.060429` CER); Nepali collapses (`0.960679` vs `0.309083` CER, `1.000000` vs `0.711607` WER) | safe English preservation but failed Devanagari override; do not promote this recognizer | [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/validation-line-script-paddle-hard50-v2/validation.md) |
| PaddleOCR v3 CUDA recognizer | G6-trained lower-LR PaddleOCR branch on hard-50 admission gate | 0.921416 | 0.080249 vs Tesseract | 0 wins, 50 losses on CER; English line-crop CER `0.917511` vs `0.031244`; Nepali WER `1.055556` vs `0.716369` | failed worse than v2; stop branch before full mixed eval | [validation.md](/Users/cdjk/github/llm/ocr-tech/outputs/aws-g6-paddle-mixed-v3-cuda/outputs/validation-paddle-mixed-v3-cuda-hard50-rescored/validation.md) |
| PaddleOCR v4 CUDA recognizer | G6-trained Devanagari PP-OCRv3 branch, LR `0.00005`, 20 epochs, evaluated on hard-50 admission gate | 0.940500 | 0.080249 vs Tesseract | 0 wins, 50 losses on CER; English line-crop CER `0.950730` vs `0.031244`; Nepali CER `0.908108` vs `0.235433` | failed worse than v2/v3; the same PaddleOCR mixed-script fine-tune branch should not be the engine core | [artifact bundle](/Users/cdjk/github/llm/ocr-tech/outputs/aws-g6-paddle-mixed-v4-cuda/ocrtech-paddle-mixed-v4-cuda-artifacts.tgz) |
## Current Best-Evidence Statement
The strongest evidence-backed statement available today is:
1. The current Tesseract-plus-Surya backfill candidate is a real document-OCR improvement over Tesseract and several weaker baselines on internal rendered Nepali/English document packs.
2. On the larger rendered-v2 holdout, every Tesseract/Surya composite tested so far either collapses to Surya, trails Surya, becomes effectively indistinguishable from Surya, or trades too much OCR accuracy for modest ordering gains.
3. On the cleaned 180-sample mixed line-style eval with hard per-sample timeouts, the older script-router candidate also loses to Tesseract overall (`0.354840` vs `0.329012` CER), despite a small overall WER improvement (`0.469944` vs `0.486485`) and a Nepali WER improvement.
4. The trained PaddleOCR recognizer failure profile is now materialized as a normalized hard-50 eval gate: all top failures are synthetic line crops, mostly English (`38/50`), so the next recognizer branch must prove English preservation instead of only improving Nepali.
5. The line-level Tesseract/Paddle composite proves the candidate shape can preserve English while routing Devanagari-looking lines, but the available trained PaddleOCR recognizer is too weak on the Nepali hard slice to use as an override (`0.960679` vs Tesseract `0.309083` CER).
6. The second AWS G6 PaddleOCR attempt fixed recipe/runtime issues but failed the hard-50 gate even worse than the first (`0.921416` vs Tesseract `0.080249` CER), so this PP-OCRv3 Devanagari recipe family is not the engine path.
7. A sample-level oracle over raw Surya and raw Tesseract has almost no text-OCR headroom over Surya (`0.180529` vs `0.180893` CER), so more routing in this component family is not a serious path to SOTA.
8. The repo now has enough validation discipline and artifact traceability to support a serious comparative paper, but not a SOTA claim.
## Regeneration Workflow
Rebuild the generated comparison tables with:
```bash
ocrtech summarize-benchmark \
--benchmark-report outputs/live-k2-claimpack-rendered-backfill/report.json \
--out outputs/paper-claimpack-summary-tesseract \
--candidate candidate \
--baseline tesseract \
--metric cer
```
```bash
ocrtech summarize-benchmark \
--benchmark-report outputs/live-k2-claimpack-surya-compare/report.json \
--out outputs/paper-claimpack-summary-surya \
--candidate candidate \
--baseline surya \
--metric cer
```
```bash
ocrtech summarize-benchmark \
--benchmark-report outputs/live-k2-claimpack-stock-paddle-fixed/report.json \
--out outputs/paper-claimpack-summary-stock-paddle \
--candidate candidate \
--baseline stock-paddle \
--metric cer
```
```bash
ocrtech paper-benchmark-table \
--summary outputs/paper-claimpack-summary-tesseract/summary.json \
--summary outputs/paper-claimpack-summary-surya/summary.json \
--summary outputs/paper-claimpack-summary-stock-paddle/summary.json \
--out outputs/paper-claimpack-table \
--system candidate \
--system surya \
--system tesseract \
--system stock-paddle \
--label 'candidate=Candidate' \
--label 'surya=Surya' \
--label 'tesseract=Tesseract' \
--label 'stock-paddle=Stock Paddle' \
--source-preference 'candidate=outputs/paper-claimpack-summary-tesseract/summary.json' \
--source-preference 'tesseract=outputs/paper-claimpack-summary-tesseract/summary.json' \
--source-preference 'surya=outputs/paper-claimpack-summary-surya/summary.json' \
--source-preference 'stock-paddle=outputs/paper-claimpack-summary-stock-paddle/summary.json' \
--metric cer \
--metric table_cell_f1 \
--metric reading_order_pair_accuracy \
--metric latency_seconds
```
```bash
ocrtech summarize-benchmark \
--benchmark-report outputs/live-k2-rendered-blur5/report.json \
--out outputs/paper-blur5-summary \
--candidate candidate \
--baseline tesseract \
--baseline surya \
--baseline stock-paddle \
--baseline glm-ocr \
--baseline paddleocr-vl \
--metric cer
```
```bash
ocrtech paper-benchmark-table \
--summary outputs/paper-blur5-summary/summary.json \
--out outputs/paper-blur5-table \
--system candidate \
--system surya \
--system tesseract \
--system stock-paddle \
--system glm-ocr \
--system paddleocr-vl \
--label 'candidate=Candidate' \
--label 'surya=Surya' \
--label 'tesseract=Tesseract' \
--label 'stock-paddle=Stock Paddle' \
--label 'glm-ocr=GLM-OCR' \
--label 'paddleocr-vl=PaddleOCR-VL' \
--metric cer \
--metric table_cell_f1 \
--metric reading_order_pair_accuracy \
--metric latency_seconds
```
```bash
ocrtech paper-benchmark-table \
--summary outputs/live-k2-claimpack-v2-script-router/summary.json \
--out outputs/paper-claimpack-v2-table \
--system candidate \
--system surya \
--system tesseract \
--label 'candidate=Candidate' \
--label 'surya=Surya' \
--label 'tesseract=Tesseract' \
--metric cer \
--metric table_cell_f1 \
--metric reading_order_pair_accuracy \
--title 'Rendered Claim Pack V2 Holdout Comparison'
```
```bash
ocrtech paper-benchmark-table \
--summary outputs/live-k2-claimpack-v2-surya-quality-router/summary.json \
--out outputs/paper-claimpack-v2-surya-quality-router-table \
--system candidate \
--system surya \
--system tesseract \
--label 'candidate=Candidate' \
--label 'surya=Surya' \
--label 'tesseract=Tesseract' \
--metric cer \
--metric table_cell_f1 \
--metric reading_order_pair_accuracy \
--metric latency_seconds \
--title 'Rendered Claim Pack V2 Holdout - Surya Quality Router'
```
```bash
ocrtech paper-benchmark-table \
--summary outputs/live-k2-claimpack-v2-line-align-summary/summary.json \
--out outputs/paper-claimpack-v2-line-align-table \
--system candidate \
--system surya \
--system tesseract \
--label 'candidate=Candidate' \
--label 'surya=Surya' \
--label 'tesseract=Tesseract' \
--metric cer \
--metric table_cell_f1 \
--metric reading_order_pair_accuracy \
--metric latency_seconds \
--title 'Rendered Claim Pack V2 Holdout - Line Align Composite'
```
```bash
ocrtech benchmark-oracle \
--benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \
--out outputs/paper-claimpack-v2-surya-tesseract-oracle \
--system surya \
--system tesseract \
--metric cer \
--oracle-name oracle-surya-tesseract
ocrtech paper-benchmark-table \
--summary outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-summary/summary.json \
--out outputs/paper-claimpack-v2-oracle-table \
--system oracle-surya-tesseract \
--system surya \
--system tesseract \
--label 'oracle-surya-tesseract=Oracle Surya/Tesseract' \
--label 'surya=Surya' \
--label 'tesseract=Tesseract' \
--metric cer \
--metric table_cell_f1 \
--metric reading_order_pair_accuracy \
--metric latency_seconds \
--title 'Rendered Claim Pack V2 Holdout - Sample-Level Oracle'
```
```bash
ocrtech benchmark-error-analysis \
--benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \
--out outputs/paper-claimpack-v2-surya-error-analysis \
--system surya \
--metric cer \
--top-n 15
ocrtech benchmark-error-analysis \
--benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \
--out outputs/paper-claimpack-v2-surya-vs-tesseract-error-analysis \
--system surya \
--baseline tesseract \
--metric cer \
--top-n 15
ocrtech benchmark-error-analysis \
--benchmark-report outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-report.json \
--out outputs/paper-claimpack-v2-surya-vs-oracle-error-analysis \
--system surya \
--baseline oracle-surya-tesseract \
--metric cer \
--top-n 10
ocrtech export-failure-manifest \
--source-manifest data/splits-claimpack-rendered-v2/eval.jsonl \
--top-failures outputs/paper-claimpack-v2-surya-error-analysis/top-failures.json \
--out data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl \
--split hard_eval
ocrtech validate-claim \
--benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \
--config configs/validation-hard15-surya.json \
--eval-manifest data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl \
--out outputs/validation-claimpack-v2-line-align-hard15
ocrtech create-eval-pack \
--out data/eval-pack-table-figure-dev-v1 \
--input-format image \
--template table-ledger \
--template figure-caption \
--count-per-template 8 \
--variant-offset 100 \
--degradation clean \
--degradation scan \
--degradation blur \
--degradation low_contrast \
--degradation skew
```
Memory still is not part of these generated tables. The current benchmark artifact format does not record it.
## Hard15 Figure-Caption + Table-Lexicon Candidate
Latest artifact set:
- `outputs/figure-caption-table-lexicon-v1/model-card.json`
- `outputs/experiment-figure-caption-table-lexicon-hard15-v3/benchmark/report.json`
- `outputs/summary-figure-caption-table-lexicon-hard15-v3/summary.md`
- `outputs/table-cell-analysis-figure-caption-table-lexicon-hard15-v3/table-cell-analysis.md`
- `outputs/validation-figure-caption-table-lexicon-hard15-v3/validation.md`
Headline table for internal hard15 only:
| system | CER | figure_caption_cer | figure_detection_f1 | table_cell_f1 | reading_order_pair_accuracy |
| --- | ---: | ---: | ---: | ---: | ---: |
| candidate | 0.010671 | 0.008746 | 1.000000 | 1.000000 | 0.520000 |
| Surya | 0.236839 | 0.092323 | 1.000000 | 0.979167 | 0.520000 |
| Tesseract | 1.266364 | 0.051020 | 0.857143 | 0.000000 | 0.613333 |
Validation status is still `not_validated`.
Passed configured Surya gates:
- overall hard_eval CER: `15` wins / `0` losses / `0` ties
- table-slice CER: `8` wins / `0` losses / `0` ties
- figure-slice CER: `7` wins / `0` losses / `0` ties
- figure-caption CER: `7` wins / `0` losses / `0` ties
Remaining configured failure:
- table-cell F1: candidate `1.000000`, Surya `0.979167`, but bootstrap CI low is `0.000000` because there are only 2 improving table pages and 6 ties
Cell-level table evidence:
| system | reference_cells | exact_cells | exact_rate | mean_cer |
| --- | ---: | ---: | ---: | ---: |
| candidate | 96 | 96 | 1.000000 | 0.000000 |
| Surya | 96 | 94 | 0.979167 | 0.010417 |
| Tesseract | 96 | 0 | 0.000000 | 1.000000 |
Use this result as an internal milestone, not a paper claim of broad SOTA. The candidate is lexicon-assisted for table names and uses guarded structural-title corrections (`वार्षिक प्रतिवेदन`, `दैनिक हिसाब`, `चित्र`, `निष्कर्ष`). A publishable table claim needs a larger independent table pack so confidence intervals are meaningful.
## Targeted Table-Ledger Validation Pack V2
Artifacts:
- `data/eval-pack-table-ledger-claim-v1/manifests/ocrtech-eval-pack.jsonl`
- `outputs/audit-table-ledger-claim-v1/eval-pack-audit.md`
- `outputs/experiment-table-ledger-claim-v2/benchmark/report.json`
- `outputs/summary-table-ledger-claim-v2/summary.md`
- `outputs/table-cell-analysis-table-ledger-claim-v2/table-cell-analysis.md`
- `outputs/validation-table-ledger-claim-v2b/validation.md`
- `configs/validation-table-ledger-surya.json`
Pack:
- template: `table-ledger`
- samples: `60`
- cells: `720`
- degradations: clean, scan, blur, low_contrast, skew
- variant offset: `300`
- claim-ready broad coverage: no, targeted table-only pack
Validation status: `validated`.
| metric | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| table_cell_f1 | 60 | 15 | 0 | 45 | 1.000000 | 0.979167 | 0.020833 | 0.012500 |
| table_exact_match | 60 | 15 | 0 | 45 | 1.000000 | 0.750000 | 0.250000 | 0.150000 |
| table_structure_similarity | 60 | 0 | 0 | 60 | 1.000000 | 1.000000 | 0.000000 | 0.000000 |
Cell exactness:
| system | reference_cells | exact_cells | exact_rate | mean_cell_cer |
| --- | ---: | ---: | ---: | ---: |
| candidate | 720 | 720 | 1.000000 | 0.000000 |
| Surya | 720 | 705 | 0.979167 | 0.005208 |
| Tesseract | 720 | 0 | 0.000000 | 0.999868 |
Runtime on k2:
| system | mean latency seconds | median latency seconds |
| --- | ---: | ---: |
| candidate | 3.017929 | 2.967943 |
| Surya | 2.478481 | 2.453125 |
| Tesseract | 0.146866 | 0.134735 |
Paper-safe claim text:
> On a targeted 60-page generated Nepali-English table-ledger validation pack, the lexicon-assisted candidate improved structured table extraction over Surya, reaching 720/720 exact cells versus Surya's 705/720, and passed bootstrap-gated validation for both table-cell F1 and table exact match.
More precise paper-safe wording:
> On a targeted 60-page generated Nepali-English table-ledger validation pack, the lexicon-assisted candidate improved table cell-content extraction over Surya, reaching 720/720 exact cells versus Surya's 705/720, while matching Surya's perfect grid-structure score.
Do not generalize this to all tables. The pack has no merged cells, no nested headers, and no real scanned government tables. The scoring path now supports span-aware structured references, but this pack does not exercise that capability and the metric is still not true HTML-tree TEDS.
## Targeted Complex-Table Validation Pack V1
Artifacts:
- `data/eval-pack-complex-table-v1/manifests/ocrtech-eval-pack.jsonl`
- `outputs/audit-complex-table-v1/eval-pack-audit.md`
- `outputs/experiment-complex-table-v1/benchmark/report.json`
- `outputs/summary-complex-table-v1/summary.md`
- `outputs/table-cell-analysis-complex-table-v1/table-cell-analysis.md`
- `outputs/validation-complex-table-v1/validation.md`
- `configs/validation-complex-table-surya.json`
Pack:
- template: `complex-table-ledger`
- samples: `40`
- reference cells: `600`
- span reference: merged header with `colspan=2`
- degradations: clean, scan, blur, low_contrast
- broad claim-ready coverage: no
Validation status: `validated` for cell-content improvement plus structure non-regression.
| metric | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| table_cell_f1 | 40 | 8 | 0 | 32 | 0.928333 | 0.915000 | 0.013333 | 0.005000 |
| table_exact_match | 40 | 0 | 0 | 40 | 0.000000 | 0.000000 | 0.000000 | 0.000000 |
| table_structure_similarity | 40 | 0 | 0 | 40 | 0.875000 | 0.875000 | 0.000000 | 0.000000 |
Cell exactness:
| system | reference_cells | exact_cells | exact_rate | mean_cell_cer |
| --- | ---: | ---: | ---: | ---: |
| candidate | 600 | 557 | 0.928333 | 0.067667 |
| Surya | 600 | 549 | 0.915000 | 0.071000 |
| Tesseract | 600 | 40 | 0.066667 | 0.933333 |
Paper-safe claim text:
> On a targeted generated complex-table pack with span-aware references, the lexicon-assisted candidate improved table cell-content F1 over Surya but did not improve merged-header reconstruction; both systems failed exact table match and tied on span-aware structure similarity.
This is a negative result for complex table structure. The next model/parser branch must preserve or infer rowspans/colspans before we can claim complex-table document understanding.
## Targeted Complex-Table Span-Inference Pack V1
After adding conservative repeated-header colspan inference, we reran the same `40`-sample generated complex-table pack on k2.
Artifacts:
- `outputs/experiment-complex-table-span-infer-v1/benchmark/report.json`
- `outputs/experiment-complex-table-span-infer-v1/validation/validation.md`
- `outputs/summary-complex-table-span-infer-v1/summary.md`
- `outputs/table-cell-analysis-complex-table-span-infer-v1/table-cell-analysis.md`
Validation status: `validated`.
| metric | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| table_cell_f1 | 40 | 40 | 0 | 0 | 0.995000 | 0.915000 | 0.080000 | 0.071667 |
| table_exact_match | 40 | 37 | 0 | 3 | 0.925000 | 0.000000 | 0.925000 | 0.825000 |
| table_structure_similarity | 40 | 40 | 0 | 0 | 1.000000 | 0.875000 | 0.125000 | 0.125000 |
Cell exactness:
| system | reference_cells | exact_cells | exact_rate | mean_cell_cer |
| --- | ---: | ---: | ---: | ---: |
| candidate | 600 | 597 | 0.995000 | 0.001000 |
| Surya | 600 | 549 | 0.915000 | 0.071000 |
| Tesseract | 600 | 40 | 0.066667 | 0.933333 |
Paper-safe claim text:
> On a targeted generated complex-table pack with one repeated merged-header pattern, a conservative span-inference postprocessor improved table structure extraction over Surya, raising span-aware structure similarity from 0.875 to 1.000 and table exact match from 0.000 to 0.925.
Do not generalize this to all complex tables. This is generated data with a single span pattern and a heuristic parser repair. The next paper-grade check needs real scanned tables with varied merged cells, rowspans, nested headers, and a true HTML/TEDS scorer.
## Targeted Complex-Table TEDS-Style Metrics V1
We added deterministic ordered table-tree edit metrics and reran the same complex-table pack with these gates included.
Artifacts:
- `outputs/experiment-complex-table-teds-v1/benchmark/report.json`
- `outputs/experiment-complex-table-teds-v1/validation/validation.md`
- `outputs/summary-complex-table-teds-v1/summary.md`
- `outputs/table-cell-analysis-complex-table-teds-v1/table-cell-analysis.md`
Validation status: `validated`.
| metric | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| table_cell_f1 | 40 | 40 | 0 | 0 | 0.995000 | 0.915000 | 0.080000 | 0.071667 |
| table_exact_match | 40 | 37 | 0 | 3 | 0.925000 | 0.000000 | 0.925000 | 0.825000 |
| table_structure_similarity | 40 | 40 | 0 | 0 | 1.000000 | 0.875000 | 0.125000 | 0.125000 |
| table_teds_structure | 40 | 40 | 0 | 0 | 1.000000 | 0.904762 | 0.095238 | 0.095238 |
| table_teds_text | 40 | 40 | 0 | 0 | 0.996250 | 0.891667 | 0.104583 | 0.098690 |
Paper-safe claim text:
> On a targeted generated complex-table pack, the candidate improved ordered table-tree metrics over Surya, reaching 1.000 structure-only TEDS-style similarity and 0.996 text-sensitive TEDS-style similarity, compared with Surya's 0.905 and 0.892 respectively.
This is still not a public-benchmark SOTA claim. The metrics are deterministic and closer to table-structure scoring, but the pack is generated and covers one repeated-header merged-cell pattern.
## HF Recognizer Next V1 Admission Benchmark
The AWS G6 trained HF recognizer was evaluated on the `recognizer-next-v1`
held-out eval manifest against Tesseract on `k2`.
Artifacts:
- `outputs/k2-hf-recognizer-next-v1-full-cpu/benchmark/report.json`
- `outputs/k2-hf-recognizer-next-v1-full-cpu/summary/summary.md`
- candidate model card:
`outputs/aws-g6-hf-recognizer-next-v1/package/model-card-cpu.json`
Run status:
- candidate rows: `244` ok, `0` errors
- Tesseract rows: `244` ok, `0` errors
- paired CER rows with text references: `180`
| slice | pairs | candidate CER | Tesseract CER | improvement | wins | losses | ties |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| all | 180 | 0.226628 | 0.329012 | 0.102384 | 74 | 54 | 52 |
| english/en/line_crop | 150 | 0.145190 | 0.243379 | 0.098189 | 56 | 42 | 52 |
| nepali | 30 | 0.633814 | 0.757175 | 0.123360 | 18 | 12 | 0 |
| real | 15 | 0.832255 | 1.162917 | 0.330662 | 12 | 3 | 0 |
| synthetic | 165 | 0.171571 | 0.253202 | 0.081631 | 62 | 51 | 52 |
Paper-safe claim text:
> On the held-out `recognizer-next-v1` text-reference slices, the trained HF
> recognizer improves paired CER over Tesseract, reducing aggregate CER from
> 0.329 to 0.227 across 180 paired samples.
Do not call this SOTA. The result is a recognizer admission benchmark against
Tesseract only; the branch still needs Surya, stock PaddleOCR, GLM-OCR, public
document benchmarks, table structure metrics, reading-order validation, and
document reconstruction gates.
|