ocr-projects-archive / ocr-tech /docs /PAPER_RESULTS.md
prosperious's picture
Add 1000 files (OCR_INDRA/.gitignore ...)
f9bc0c1 verified
|
Raw History Blame Contribute Delete
54.3 kB

Paper Results

This file is the shortest path from repo artifacts to a paper draft.

Use PAPER_NOTES.md for claim boundaries and framing. Use this file for the concrete result tables and the branch chronology.

Current Comparison Tables

Engine Guardrails Added After These Tables

2026-06-16 - Preserve Explicit Table Spans During Grid Cleanup

The figure_caption_refine_composite branch normalizes noisy flat table grids by removing separator columns, duplicate columns, and sparse continuation rows. That cleanup is useful for simple OCR table noise, but it is unsafe once a table module has already emitted real rowspan or colspan structure. Rebuilding from a plain grid would erase the very merged-header structure that the paper metrics now score.

Implementation change:

  • normalize_table_grid_structure now skips tables with explicit row/column spans and records normalized_table_grid_skipped_reason="explicit_spans".
  • The table helper and candidate composite tests now verify that colspan survives when grid normalization is enabled.

This is an engine guardrail, not a new accuracy result. It protects future complex-table benchmarks from a known postprocessing failure mode.

HF Recognizer Next V1 Admission

Generated artifact:

Coverage:

  • 244 held-out eval samples
  • candidate, Tesseract, and stock PaddleOCR completed 244/244
  • Surya completed 243/244; one English synthetic line crop timed out at 240s
  • admission validation status: validated

Headline paired CER table:

baseline slice pairs candidate CER baseline CER abs improvement wins losses ties
Tesseract all 180 0.226628 0.329012 0.102384 74 54 52
Tesseract Nepali 30 0.633814 0.757175 0.123360 18 12 0
stock PaddleOCR all 180 0.226628 0.656675 0.430048 159 10 11
stock PaddleOCR Nepali 30 0.633814 0.945114 0.311300 24 2 4
Surya all 179 0.223364 3.307235 3.083871 119 42 18
Surya Nepali 30 0.633814 0.930927 0.297113 10 20 0

Interpretation:

  • This is the first trained recognizer branch in the repo that clears a held-out OCR admission gate against Tesseract and stock PaddleOCR.
  • Treat the Surya comparison as adapter-specific evidence. The English line-crop Surya CER is suspiciously high, and the Nepali slice has lower mean CER for the candidate but worse paired win count against Surya.
  • This result supports "promising recognizer component"; it does not support "complete document-understanding engine" or SOTA.

HF Recognizer Next V1 Table-Cell Refinement Smoke

Generated artifact:

One-sample hard-table smoke:

system CER table_cell_f1 table_teds_text reading_order_pair_accuracy latency_seconds
HF table-cell refine candidate 0.363636 0.000000 0.294118 0.100000 9.888044
Surya 0.363636 1.000000 1.000000 0.100000 5.529877

Interpretation:

  • The trained HF recognizer now runs as a leaf OCR engine inside the document pipeline.
  • Unconditional HF replacement is not viable: on eval-table-ledger-000-clean, it replaced all 12 table cells and destroyed an already-correct Surya table.
  • Do not scale this branch as-is. Any future HF table refinement needs a guarded replacement policy or a crop-specific recognizer, not forced replacement.

Rendered Claim Pack

Generated artifact:

Headline table:

system sample_count cer table_cell_f1 reading_order_pair_accuracy latency_seconds
Candidate 25 0.248654 0.955556 0.764000 3.268937
Surya 25 0.195260 0.955556 0.750909 2.618872
Tesseract 25 0.689214 0.000000 0.816000 0.240963
Stock Paddle 25 1.749081 0.000000 1.000000 7.732034

Interpretation:

  • Candidate beats Tesseract and stock-Paddle badly on text CER and table extraction.
  • Candidate ties Surya on aggregate table_cell_f1.
  • Candidate is slightly better than Surya on aggregate reading-order accuracy.
  • Candidate still loses to raw Surya on text CER.
  • Candidate is also slower than Tesseract and somewhat slower than Surya on this pack.

Latency caveat:

  • these latency means come from separate benchmark reports on the same host and pack, then get pinned per system during table export
  • treat them as comparable operational signals, not as a single synchronized runtime shootout

Rendered Claim Pack V2 Holdout

Generated artifact:

Headline table:

system sample_count cer table_cell_f1 reading_order_pair_accuracy
Candidate 50 0.180893 0.947500 0.750909
Surya 50 0.180893 0.970833 0.750909
Tesseract 50 0.714207 0.000000 0.787818

Interpretation:

  • This is the stronger rendered-document holdout because calibration and holdout were separated at 50/50.
  • The routed candidate does not beat Surya here. It ties Surya exactly on text CER and reading-order accuracy.
  • The routed candidate is slightly worse than raw Surya on aggregate table_cell_f1.
  • The calibration artifact selected primary_script_threshold=0.0, which means the routed candidate collapsed to "always pick Surya" on the calibration split.
  • That is why claim preflight now rejects this candidate as a routed model fit for a broad "better engine" claim.

Rendered Claim Pack V2 Holdout - Surya Primary Latin Router

Generated artifact:

Headline table:

system sample_count cer table_cell_f1 reading_order_pair_accuracy latency_seconds
Candidate 50 0.416854 0.337500 0.761818 6.067455
Surya 50 0.180893 0.970833 0.750909 6.699851
Tesseract 50 0.714207 0.000000 0.787818 0.238464

Interpretation:

  • This candidate was a Surya-primary, Latin-triggered page router with Tesseract fallback.
  • It was meant to preserve Surya's text strength while using Tesseract for English-heavy pages.
  • It failed badly on the rendered-v2 holdout: worse than raw Surya on text CER and much worse on table structure.
  • Paired CER against Surya was decisive enough to rule the branch out: 0 wins, 12 losses, 38 ties.

Rendered Claim Pack V2 Holdout - Surya Quality Router

Generated artifact:

Headline table:

system sample_count cer table_cell_f1 reading_order_pair_accuracy latency_seconds
Candidate 50 0.181128 0.947500 0.750909 4.620400
Surya 50 0.180893 0.947500 0.750909 5.603801
Tesseract 50 0.714207 0.000000 0.787818 0.240773

Interpretation:

  • This candidate was a Surya-vs-Tesseract quality selector calibrated on the rendered-v2 calibration split.
  • Calibration selected selected_threshold=0.0, which means the quality gate kept Surya whenever it was not scored worse than Tesseract.
  • On the untouched holdout, the candidate still did not beat raw Surya on text CER.
  • Paired CER against Surya was effectively a tie with one bad miss: 0 wins, 1 loss, 49 ties.
  • This is operationally closer to "Surya with a guardrail" than to a meaningfully new OCR engine.

Rendered Claim Pack V2 Holdout - Line Align Composite

Generated artifact:

Headline table:

system sample_count cer table_cell_f1 reading_order_pair_accuracy latency_seconds
Candidate 50 0.310549 0.897500 0.792727 4.142242
Surya 50 0.180893 0.970833 0.750909 3.223913
Tesseract 50 0.714207 0.000000 0.787818 0.235074

Interpretation:

  • This candidate kept Surya as the text engine and reused Tesseract line scaffolding through line_align_composite.
  • It did improve reading-order accuracy relative to raw Surya, and even edged past raw Tesseract on this holdout.
  • That gain is not enough. CER regressed badly versus Surya, and aggregate table_cell_f1 also regressed.
  • Paired CER against Surya was one-sided enough to kill the branch: 0 wins, 7 losses, 43 ties.
  • This is a useful negative result for the paper because it shows that better line ordering alone does not rescue the Surya/Tesseract composite family.

Rendered Claim Pack V2 Holdout - Sample-Level Oracle

Generated artifact:

Headline table:

system sample_count cer table_cell_f1 reading_order_pair_accuracy latency_seconds
Oracle Surya/Tesseract 50 0.180529 0.970833 0.750909 3.193115
Surya 50 0.180893 0.970833 0.750909 3.223913
Tesseract 50 0.714207 0.000000 0.787818 0.235074

Interpretation:

  • This is not a deployable system. It is an upper-bound analysis that picks the better of raw Surya and raw Tesseract per sample using ground-truth CER.
  • The oracle picked Surya for 49 of 50 holdout samples and Tesseract for only 1.
  • Per-sample oracle selections are recorded separately in oracle-choices.json; they are provenance, not benchmark metrics.
  • The absolute CER gain over raw Surya is only 0.000364.
  • That means the current Surya/Tesseract routing space has almost no text-OCR headroom left on rendered-v2.
  • Future work should move to a stronger recognizer/document model, not more Surya/Tesseract routing heuristics.

Rendered Claim Pack V2 Holdout - Surya Error Pockets

Generated artifact:

Diagnostic summary:

slice sample_count Surya CER
figure 20 0.253157
table 20 0.252781
multi_page 10 0.196472
scan 10 0.194861
low_contrast 10 0.193317
blur 10 0.190940
clean 10 0.182256

Interpretation:

  • Surya's hardest absolute CER pockets on rendered-v2 are table-ledger and figure-caption samples, not generic blur/skew alone.
  • Tesseract is worse than Surya on 49 of 50 samples and better on only eval-form-notice-003-blur; that single oracle win is too small to justify more Surya/Tesseract routing work.
  • A ranked 15-sample hard_eval manifest now exists for the next model branch; it contains 8 table samples and 7 figure samples, and its full file/reference audit passes on k2.
  • configs/validation-hard15-surya.json defines the pocket gate a future candidate must clear against Surya, including intersection gates for hard_eval,table and hard_eval,figure.
  • The current line-align candidate fails the hard15 gate: hard-eval CER is 0.482039 versus Surya 0.348706, and hard table CER is 0.613636 versus Surya 0.363636.
  • A separate 80-sample image development pack now exists for table/figure work: 40 table samples, 40 figure samples, all five degradations. It has zero sample-id overlap and zero text-hash overlap with hard15, and its k2 audit passes.
  • That development pack is not claim evidence; it is for model iteration before returning to the held-out hard15 and rendered-v2 gates.
  • The next credible recognizer branch should target table/figure-heavy mixed-script pages and preserve Surya's already-strong behavior on the other 49 samples.

Table/Figure Development Pack - HF TrOCR Negative Result

Generated artifact:

Pack and split:

  • 80 image samples: 40 table and 40 figure
  • degradations: clean, scan, blur, low_contrast, skew
  • split: 60 train / 20 eval
  • grouping: text_sha256
  • text groups: 16
  • split leakage: pass
  • train/eval manifest audits on k2: pass
  • hard15 overlap: 0 sample IDs and 0 text hashes

Training result for microsoft/trocr-small-printed after one epoch:

metric value
eval_loss 7.884517
eval_cer 3.062119
eval_wer 3.562689
eval_exact_line_accuracy 0.000000

Benchmark result on the 20-sample dev eval:

system CER table_cell_f1 reading_order_pair_accuracy
HF table/figure candidate 2.422001 0.000000 0.000000
Surya 0.563399 0.858333 0.550000
Tesseract 0.546227 0.000000 0.695000

Interpretation:

  • The model package audits cleanly, so this is a quality failure rather than a packaging failure.
  • Paired CER was 0 wins, 20 losses, and 0 ties against Surya.
  • Paired CER was also 0 wins, 20 losses, and 0 ties against Tesseract.
  • Naive full-page TrOCR fine-tuning is not a serious next step for table/figure-heavy document pages.
  • The next model branch should use line/cell crops or a structure-aware page/document model rather than treating full document images as single recognition targets.

Table/Figure Crop Development Pack - HF TrOCR Negative Result

Generated artifact:

Pack and split:

  • source rendered samples: 80
  • extracted crops: 544
  • crop types: 384 table cells, 128 lines, 32 figure captions
  • skipped candidates: 32 skew pages because pre-rotation bboxes are invalid
  • split: 480 train / 64 eval
  • grouping: text_sha256
  • text groups: 68
  • crop manifest audit on k2: pass
  • split leakage: pass

Training result for microsoft/trocr-small-printed after one crop-level epoch:

metric value
eval_loss 3.087561
eval_cer 0.959831
eval_wer 1.363715
eval_exact_line_accuracy 0.078125

Benchmark result on the 64-sample crop eval:

system CER
HF crop candidate 1.043369
Surya 0.475309
Tesseract 0.077453

Interpretation:

  • Crop-level training is a better problem shape than full-page recognition, but this recognizer is still not competitive.
  • Paired CER against Surya was 7 wins, 50 losses, and 7 ties.
  • Paired CER against Tesseract was 0 wins, 64 losses, and 0 ties.
  • Tesseract is extremely strong on these rendered crop images, so near-term gains are more likely from better crop/layout detection plus strong crop OCR than from this small synthetic TrOCR fine-tune.

Hard15 - Table Cell Refinement Composite

Generated artifact:

Headline hard15 comparison:

system hard_eval CER table_cell_f1 reading_order_pair_accuracy
Surya baseline 0.348706 0.979167 0.520000
Table-cell refine, always 0.348706 0.593750 0.520000
Table-cell refine, empty 0.348706 0.979167 0.520000
Table-cell refine, empty + parser de-dup 0.348706 0.979167 0.520000

Interpretation:

  • The unsafe table-cell crop replacement policy damaged Surya's already-good table cells.
  • The conservative empty-cell-only policy avoided regression but did not improve any hard15 metric.
  • The safe policy's hard15 validation status is not_validated.
  • The parser de-dup branch removes duplicate overlapping table exports for both candidate and Surya, so it improves artifact cleanliness but does not create a candidate advantage.
  • Exact crop OCR being strong is not enough; the deployment problem is reliable cell detection and knowing when a replacement is safer than the structure engine's existing cell text.

Cell-level result:

  • Surya is already exact on 94/96 hard15 reference table cells.
  • Unsafe replacement is exact on only 57/96 cells.
  • Unsafe replacement improves 0 cells over Surya and worsens 37.
  • Safe+dedup is exact on 94/96 cells, with 8 predicted tables for 8 reference tables and 0 duplicate overlapping tables on the v3 rerun.

Hard15 - Figure Caption Refinement Composite

Generated artifact:

Headline hard15 comparison after the parser caption-attachment fix:

system CER figure_caption_cer figure_detection_f1 table_cell_f1 reading_order_pair_accuracy
Surya 0.236839 0.092323 1.000000 0.979167 0.520000
Tesseract 1.266364 0.051020 0.857143 0.000000 0.613333
Figure-caption refine 0.228008 0.052478 1.000000 0.979167 0.520000
Figure-caption refine + labels 0.204610 0.008746 1.000000 0.979167 0.520000

Validation result:

  • basic branch overall status: not_validated
  • basic branch figure-caption gate: pass, 6 wins / 0 losses / 1 tie against Surya, improvement 0.039845
  • label branch overall status: not_validated
  • label branch overall hard-eval CER gate: pass, improvement 0.032228
  • label branch figure CER gate: pass, improvement 0.069061
  • label branch figure-caption gate: pass, 7 wins / 0 losses / 0 ties against Surya, improvement 0.083576
  • label branch table gates: fail, ties against Surya

Interpretation:

  • This is a real slice win over Surya on figure-caption extraction.
  • It is not broad enough for a SOTA claim.
  • Tesseract has better caption text but worse figure detection/table structure; the composite keeps Surya's structure and selectively borrows caption text.
  • Safe empty-only replacement ties all 96 cells with Surya because there are no missing Surya cells in this pocket.
  • The only remaining Surya cell errors are किरण predicted as करिण on eval-table-ledger-002-clean and eval-table-ledger-002-scan.

Rendered Blur-Only Expensive-Baseline Subset

Generated artifact:

Headline table:

system sample_count cer table_cell_f1 reading_order_pair_accuracy latency_seconds
Candidate 5 0.223273 1.000000 0.783636 3.784279
Surya 5 0.197601 1.000000 0.750909 2.943875
Tesseract 5 0.732364 0.000000 0.823636 0.220515
Stock Paddle 5 2.810426 0.000000 1.000000 7.440482
GLM-OCR 5 2.820269 0.000000 1.000000 16.846619
PaddleOCR-VL 5 18.470663 0.000000 0.750909 32.793166

Interpretation:

  • Candidate is materially stronger than Tesseract, stock-Paddle, GLM-OCR, and PaddleOCR-VL on this degraded rendered subset.
  • Candidate still does not beat raw Surya on text CER.
  • The external VLM-style baselines are also much slower than the current candidate here.

Branch Chronology

This is the shortest honest story of the project so far.

branch eval setting candidate CER baseline CER paired result verdict evidence
HF TrOCR recognizer mixed Nepali/English line-style holdout, 180 samples 0.586139 0.329012 vs Tesseract 36 wins, 97 losses, 47 ties runnable but not competitive summary.json
Tesseract + ByT5 corrector same 180-sample mixed holdout 6.912065 0.329012 vs Tesseract 0 wins, 170 losses, 10 ties failed badly summary.md
quality router (psm=6 vs psm=13) 91-sample mixed holdout 0.340608 0.342234 vs Tesseract 2 wins, 1 loss, 88 ties tiny gain, plausible baseline only summary.json
ranked PSM ensemble (6/7/11/13) same 91-sample mixed holdout 0.358352 0.342234 vs Tesseract 9 wins, 16 losses, 66 ties overfit, worse than baseline summary.json
script router (tesseract -> surya) same 91-sample mixed holdout 0.313084 0.342234 vs Tesseract 9 wins, 0 losses, 82 ties strongest older line-style result summary.json
script router + structure backfill rendered claim-ready document pack, 25 samples 0.248654 0.689214 vs Tesseract 9 wins, 1 loss, 15 ties strong document-pack improvement over Tesseract summary.json
raw Surya on rendered claim pack rendered claim-ready document pack, 25 samples 0.195260 0.248654 vs Candidate candidate loses on text CER current text leader on rendered pack summary.json
rendered-v2 script router rendered claim-ready document holdout, 50 samples 0.180893 0.180893 vs Surya exact CER tie, worse table structure routed candidate collapses to Surya on calibration; not a meaningful new engine summary.json
rendered-v2 Surya-primary Latin router rendered claim-ready document holdout, 50 samples 0.416854 0.180893 vs Surya 0 wins, 12 losses, 38 ties on CER failed badly; inverted script routing is not a viable answer summary.json
rendered-v2 Surya quality router rendered claim-ready document holdout, 50 samples 0.181128 0.180893 vs Surya 0 wins, 1 loss, 49 ties on CER nearly indistinguishable from Surya and still not better summary.json
rendered-v2 line-align composite rendered claim-ready document holdout, 50 samples 0.310549 0.180893 vs Surya 0 wins, 7 losses, 43 ties on CER better reading order, but text and table regressions are too large summary.json
rendered-v2 Surya/Tesseract sample oracle rendered claim-ready document holdout, 50 samples 0.180529 0.180893 vs Surya oracle picks Tesseract only 1/50 samples upper bound shows routing headroom is essentially exhausted oracle-analysis.json
HF TrOCR table/figure dev table/figure development eval, 20 samples 2.422001 0.563399 vs Surya; 0.546227 vs Tesseract 0 wins, 20 losses, 0 ties vs both failed badly; full-page TrOCR fine-tuning is not viable here summary.md
HF TrOCR crop dev table/figure crop eval, 64 samples 1.043369 0.475309 vs Surya; 0.077453 vs Tesseract 7 wins, 50 losses, 7 ties vs Surya; 0 wins, 64 losses vs Tesseract crop shape helps but this recognizer still fails summary.md
table-cell refine, unsafe hard15 rendered-v2 pocket, 15 samples 0.348706 0.348706 vs Surya text tie, table F1 regresses from 0.979167 to 0.593750 replacement policy is unsafe summary.md
table-cell refine, empty-only hard15 rendered-v2 pocket, 15 samples 0.348706 0.348706 vs Surya exact tie across CER/table/reading order; validation not_validated safe but no improvement validation.md
normalized script router, hard-timeout rerun normalized mixed Nepali/English line-style eval, 180 samples 0.354840 0.329012 vs Tesseract; 3.157317 vs Surya on 178 scored Surya rows 17 wins, 4 losses, 159 ties on CER vs Tesseract but worse aggregate mean; 2 Surya rows timed out at 45s not a better engine; benchmark watchdog works and routing still loses overall CER summary.md
PaddleOCR recognizer hard-50 failure slice top 50 failures from AWS G6 trained PaddleOCR recognizer, remapped to normalized k2 eval paths n/a n/a 50 failures: 38 English line-crop synthetic, 12 Nepali synthetic; k2 manifest audit passed next recognizer gate; do not train on it manifest-audit.md
PaddleOCR hard-50 gate negative control failed AWS G6 trained PaddleOCR recognizer evaluated on the hard-50 gate 0.887557 0.080249 vs Tesseract fails all four hard gates; English line-crop CER 0.852048 vs 0.031244 validates the gate catches the known recognizer collapse validation.md
line-level Tesseract/Paddle composite hard-50 failure slice, Tesseract primary with PaddleOCR secondary only for Devanagari-looking lines 0.276489 0.120106 vs Tesseract English line-crop ties exactly (0.060429 CER); Nepali collapses (0.960679 vs 0.309083 CER, 1.000000 vs 0.711607 WER) safe English preservation but failed Devanagari override; do not promote this recognizer validation.md
PaddleOCR v3 CUDA recognizer G6-trained lower-LR PaddleOCR branch on hard-50 admission gate 0.921416 0.080249 vs Tesseract 0 wins, 50 losses on CER; English line-crop CER 0.917511 vs 0.031244; Nepali WER 1.055556 vs 0.716369 failed worse than v2; stop branch before full mixed eval validation.md
PaddleOCR v4 CUDA recognizer G6-trained Devanagari PP-OCRv3 branch, LR 0.00005, 20 epochs, evaluated on hard-50 admission gate 0.940500 0.080249 vs Tesseract 0 wins, 50 losses on CER; English line-crop CER 0.950730 vs 0.031244; Nepali CER 0.908108 vs 0.235433 failed worse than v2/v3; the same PaddleOCR mixed-script fine-tune branch should not be the engine core artifact bundle

Current Best-Evidence Statement

The strongest evidence-backed statement available today is:

  1. The current Tesseract-plus-Surya backfill candidate is a real document-OCR improvement over Tesseract and several weaker baselines on internal rendered Nepali/English document packs.
  2. On the larger rendered-v2 holdout, every Tesseract/Surya composite tested so far either collapses to Surya, trails Surya, becomes effectively indistinguishable from Surya, or trades too much OCR accuracy for modest ordering gains.
  3. On the cleaned 180-sample mixed line-style eval with hard per-sample timeouts, the older script-router candidate also loses to Tesseract overall (0.354840 vs 0.329012 CER), despite a small overall WER improvement (0.469944 vs 0.486485) and a Nepali WER improvement.
  4. The trained PaddleOCR recognizer failure profile is now materialized as a normalized hard-50 eval gate: all top failures are synthetic line crops, mostly English (38/50), so the next recognizer branch must prove English preservation instead of only improving Nepali.
  5. The line-level Tesseract/Paddle composite proves the candidate shape can preserve English while routing Devanagari-looking lines, but the available trained PaddleOCR recognizer is too weak on the Nepali hard slice to use as an override (0.960679 vs Tesseract 0.309083 CER).
  6. The second AWS G6 PaddleOCR attempt fixed recipe/runtime issues but failed the hard-50 gate even worse than the first (0.921416 vs Tesseract 0.080249 CER), so this PP-OCRv3 Devanagari recipe family is not the engine path.
  7. A sample-level oracle over raw Surya and raw Tesseract has almost no text-OCR headroom over Surya (0.180529 vs 0.180893 CER), so more routing in this component family is not a serious path to SOTA.
  8. The repo now has enough validation discipline and artifact traceability to support a serious comparative paper, but not a SOTA claim.

Regeneration Workflow

Rebuild the generated comparison tables with:

ocrtech summarize-benchmark \
  --benchmark-report outputs/live-k2-claimpack-rendered-backfill/report.json \
  --out outputs/paper-claimpack-summary-tesseract \
  --candidate candidate \
  --baseline tesseract \
  --metric cer
ocrtech summarize-benchmark \
  --benchmark-report outputs/live-k2-claimpack-surya-compare/report.json \
  --out outputs/paper-claimpack-summary-surya \
  --candidate candidate \
  --baseline surya \
  --metric cer
ocrtech summarize-benchmark \
  --benchmark-report outputs/live-k2-claimpack-stock-paddle-fixed/report.json \
  --out outputs/paper-claimpack-summary-stock-paddle \
  --candidate candidate \
  --baseline stock-paddle \
  --metric cer
ocrtech paper-benchmark-table \
  --summary outputs/paper-claimpack-summary-tesseract/summary.json \
  --summary outputs/paper-claimpack-summary-surya/summary.json \
  --summary outputs/paper-claimpack-summary-stock-paddle/summary.json \
  --out outputs/paper-claimpack-table \
  --system candidate \
  --system surya \
  --system tesseract \
  --system stock-paddle \
  --label 'candidate=Candidate' \
  --label 'surya=Surya' \
  --label 'tesseract=Tesseract' \
  --label 'stock-paddle=Stock Paddle' \
  --source-preference 'candidate=outputs/paper-claimpack-summary-tesseract/summary.json' \
  --source-preference 'tesseract=outputs/paper-claimpack-summary-tesseract/summary.json' \
  --source-preference 'surya=outputs/paper-claimpack-summary-surya/summary.json' \
  --source-preference 'stock-paddle=outputs/paper-claimpack-summary-stock-paddle/summary.json' \
  --metric cer \
  --metric table_cell_f1 \
  --metric reading_order_pair_accuracy \
  --metric latency_seconds
ocrtech summarize-benchmark \
  --benchmark-report outputs/live-k2-rendered-blur5/report.json \
  --out outputs/paper-blur5-summary \
  --candidate candidate \
  --baseline tesseract \
  --baseline surya \
  --baseline stock-paddle \
  --baseline glm-ocr \
  --baseline paddleocr-vl \
  --metric cer
ocrtech paper-benchmark-table \
  --summary outputs/paper-blur5-summary/summary.json \
  --out outputs/paper-blur5-table \
  --system candidate \
  --system surya \
  --system tesseract \
  --system stock-paddle \
  --system glm-ocr \
  --system paddleocr-vl \
  --label 'candidate=Candidate' \
  --label 'surya=Surya' \
  --label 'tesseract=Tesseract' \
  --label 'stock-paddle=Stock Paddle' \
  --label 'glm-ocr=GLM-OCR' \
  --label 'paddleocr-vl=PaddleOCR-VL' \
  --metric cer \
  --metric table_cell_f1 \
  --metric reading_order_pair_accuracy \
  --metric latency_seconds
ocrtech paper-benchmark-table \
  --summary outputs/live-k2-claimpack-v2-script-router/summary.json \
  --out outputs/paper-claimpack-v2-table \
  --system candidate \
  --system surya \
  --system tesseract \
  --label 'candidate=Candidate' \
  --label 'surya=Surya' \
  --label 'tesseract=Tesseract' \
  --metric cer \
  --metric table_cell_f1 \
  --metric reading_order_pair_accuracy \
  --title 'Rendered Claim Pack V2 Holdout Comparison'
ocrtech paper-benchmark-table \
  --summary outputs/live-k2-claimpack-v2-surya-quality-router/summary.json \
  --out outputs/paper-claimpack-v2-surya-quality-router-table \
  --system candidate \
  --system surya \
  --system tesseract \
  --label 'candidate=Candidate' \
  --label 'surya=Surya' \
  --label 'tesseract=Tesseract' \
  --metric cer \
  --metric table_cell_f1 \
  --metric reading_order_pair_accuracy \
  --metric latency_seconds \
  --title 'Rendered Claim Pack V2 Holdout - Surya Quality Router'
ocrtech paper-benchmark-table \
  --summary outputs/live-k2-claimpack-v2-line-align-summary/summary.json \
  --out outputs/paper-claimpack-v2-line-align-table \
  --system candidate \
  --system surya \
  --system tesseract \
  --label 'candidate=Candidate' \
  --label 'surya=Surya' \
  --label 'tesseract=Tesseract' \
  --metric cer \
  --metric table_cell_f1 \
  --metric reading_order_pair_accuracy \
  --metric latency_seconds \
  --title 'Rendered Claim Pack V2 Holdout - Line Align Composite'
ocrtech benchmark-oracle \
  --benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \
  --out outputs/paper-claimpack-v2-surya-tesseract-oracle \
  --system surya \
  --system tesseract \
  --metric cer \
  --oracle-name oracle-surya-tesseract

ocrtech paper-benchmark-table \
  --summary outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-summary/summary.json \
  --out outputs/paper-claimpack-v2-oracle-table \
  --system oracle-surya-tesseract \
  --system surya \
  --system tesseract \
  --label 'oracle-surya-tesseract=Oracle Surya/Tesseract' \
  --label 'surya=Surya' \
  --label 'tesseract=Tesseract' \
  --metric cer \
  --metric table_cell_f1 \
  --metric reading_order_pair_accuracy \
  --metric latency_seconds \
  --title 'Rendered Claim Pack V2 Holdout - Sample-Level Oracle'
ocrtech benchmark-error-analysis \
  --benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \
  --out outputs/paper-claimpack-v2-surya-error-analysis \
  --system surya \
  --metric cer \
  --top-n 15

ocrtech benchmark-error-analysis \
  --benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \
  --out outputs/paper-claimpack-v2-surya-vs-tesseract-error-analysis \
  --system surya \
  --baseline tesseract \
  --metric cer \
  --top-n 15

ocrtech benchmark-error-analysis \
  --benchmark-report outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-report.json \
  --out outputs/paper-claimpack-v2-surya-vs-oracle-error-analysis \
  --system surya \
  --baseline oracle-surya-tesseract \
  --metric cer \
  --top-n 10

ocrtech export-failure-manifest \
  --source-manifest data/splits-claimpack-rendered-v2/eval.jsonl \
  --top-failures outputs/paper-claimpack-v2-surya-error-analysis/top-failures.json \
  --out data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl \
  --split hard_eval

ocrtech validate-claim \
  --benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \
  --config configs/validation-hard15-surya.json \
  --eval-manifest data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl \
  --out outputs/validation-claimpack-v2-line-align-hard15

ocrtech create-eval-pack \
  --out data/eval-pack-table-figure-dev-v1 \
  --input-format image \
  --template table-ledger \
  --template figure-caption \
  --count-per-template 8 \
  --variant-offset 100 \
  --degradation clean \
  --degradation scan \
  --degradation blur \
  --degradation low_contrast \
  --degradation skew

Memory still is not part of these generated tables. The current benchmark artifact format does not record it.

Hard15 Figure-Caption + Table-Lexicon Candidate

Latest artifact set:

  • outputs/figure-caption-table-lexicon-v1/model-card.json
  • outputs/experiment-figure-caption-table-lexicon-hard15-v3/benchmark/report.json
  • outputs/summary-figure-caption-table-lexicon-hard15-v3/summary.md
  • outputs/table-cell-analysis-figure-caption-table-lexicon-hard15-v3/table-cell-analysis.md
  • outputs/validation-figure-caption-table-lexicon-hard15-v3/validation.md

Headline table for internal hard15 only:

system CER figure_caption_cer figure_detection_f1 table_cell_f1 reading_order_pair_accuracy
candidate 0.010671 0.008746 1.000000 1.000000 0.520000
Surya 0.236839 0.092323 1.000000 0.979167 0.520000
Tesseract 1.266364 0.051020 0.857143 0.000000 0.613333

Validation status is still not_validated.

Passed configured Surya gates:

  • overall hard_eval CER: 15 wins / 0 losses / 0 ties
  • table-slice CER: 8 wins / 0 losses / 0 ties
  • figure-slice CER: 7 wins / 0 losses / 0 ties
  • figure-caption CER: 7 wins / 0 losses / 0 ties

Remaining configured failure:

  • table-cell F1: candidate 1.000000, Surya 0.979167, but bootstrap CI low is 0.000000 because there are only 2 improving table pages and 6 ties

Cell-level table evidence:

system reference_cells exact_cells exact_rate mean_cer
candidate 96 96 1.000000 0.000000
Surya 96 94 0.979167 0.010417
Tesseract 96 0 0.000000 1.000000

Use this result as an internal milestone, not a paper claim of broad SOTA. The candidate is lexicon-assisted for table names and uses guarded structural-title corrections (वार्षिक प्रतिवेदन, दैनिक हिसाब, चित्र, निष्कर्ष). A publishable table claim needs a larger independent table pack so confidence intervals are meaningful.

Targeted Table-Ledger Validation Pack V2

Artifacts:

  • data/eval-pack-table-ledger-claim-v1/manifests/ocrtech-eval-pack.jsonl
  • outputs/audit-table-ledger-claim-v1/eval-pack-audit.md
  • outputs/experiment-table-ledger-claim-v2/benchmark/report.json
  • outputs/summary-table-ledger-claim-v2/summary.md
  • outputs/table-cell-analysis-table-ledger-claim-v2/table-cell-analysis.md
  • outputs/validation-table-ledger-claim-v2b/validation.md
  • configs/validation-table-ledger-surya.json

Pack:

  • template: table-ledger
  • samples: 60
  • cells: 720
  • degradations: clean, scan, blur, low_contrast, skew
  • variant offset: 300
  • claim-ready broad coverage: no, targeted table-only pack

Validation status: validated.

metric pairs wins losses ties candidate Surya improvement ci_low
table_cell_f1 60 15 0 45 1.000000 0.979167 0.020833 0.012500
table_exact_match 60 15 0 45 1.000000 0.750000 0.250000 0.150000
table_structure_similarity 60 0 0 60 1.000000 1.000000 0.000000 0.000000

Cell exactness:

system reference_cells exact_cells exact_rate mean_cell_cer
candidate 720 720 1.000000 0.000000
Surya 720 705 0.979167 0.005208
Tesseract 720 0 0.000000 0.999868

Runtime on k2:

system mean latency seconds median latency seconds
candidate 3.017929 2.967943
Surya 2.478481 2.453125
Tesseract 0.146866 0.134735

Paper-safe claim text:

On a targeted 60-page generated Nepali-English table-ledger validation pack, the lexicon-assisted candidate improved structured table extraction over Surya, reaching 720/720 exact cells versus Surya's 705/720, and passed bootstrap-gated validation for both table-cell F1 and table exact match.

More precise paper-safe wording:

On a targeted 60-page generated Nepali-English table-ledger validation pack, the lexicon-assisted candidate improved table cell-content extraction over Surya, reaching 720/720 exact cells versus Surya's 705/720, while matching Surya's perfect grid-structure score.

Do not generalize this to all tables. The pack has no merged cells, no nested headers, and no real scanned government tables. The scoring path now supports span-aware structured references, but this pack does not exercise that capability and the metric is still not true HTML-tree TEDS.

Targeted Complex-Table Validation Pack V1

Artifacts:

  • data/eval-pack-complex-table-v1/manifests/ocrtech-eval-pack.jsonl
  • outputs/audit-complex-table-v1/eval-pack-audit.md
  • outputs/experiment-complex-table-v1/benchmark/report.json
  • outputs/summary-complex-table-v1/summary.md
  • outputs/table-cell-analysis-complex-table-v1/table-cell-analysis.md
  • outputs/validation-complex-table-v1/validation.md
  • configs/validation-complex-table-surya.json

Pack:

  • template: complex-table-ledger
  • samples: 40
  • reference cells: 600
  • span reference: merged header with colspan=2
  • degradations: clean, scan, blur, low_contrast
  • broad claim-ready coverage: no

Validation status: validated for cell-content improvement plus structure non-regression.

metric pairs wins losses ties candidate Surya improvement ci_low
table_cell_f1 40 8 0 32 0.928333 0.915000 0.013333 0.005000
table_exact_match 40 0 0 40 0.000000 0.000000 0.000000 0.000000
table_structure_similarity 40 0 0 40 0.875000 0.875000 0.000000 0.000000

Cell exactness:

system reference_cells exact_cells exact_rate mean_cell_cer
candidate 600 557 0.928333 0.067667
Surya 600 549 0.915000 0.071000
Tesseract 600 40 0.066667 0.933333

Paper-safe claim text:

On a targeted generated complex-table pack with span-aware references, the lexicon-assisted candidate improved table cell-content F1 over Surya but did not improve merged-header reconstruction; both systems failed exact table match and tied on span-aware structure similarity.

This is a negative result for complex table structure. The next model/parser branch must preserve or infer rowspans/colspans before we can claim complex-table document understanding.

Targeted Complex-Table Span-Inference Pack V1

After adding conservative repeated-header colspan inference, we reran the same 40-sample generated complex-table pack on k2.

Artifacts:

  • outputs/experiment-complex-table-span-infer-v1/benchmark/report.json
  • outputs/experiment-complex-table-span-infer-v1/validation/validation.md
  • outputs/summary-complex-table-span-infer-v1/summary.md
  • outputs/table-cell-analysis-complex-table-span-infer-v1/table-cell-analysis.md

Validation status: validated.

metric pairs wins losses ties candidate Surya improvement ci_low
table_cell_f1 40 40 0 0 0.995000 0.915000 0.080000 0.071667
table_exact_match 40 37 0 3 0.925000 0.000000 0.925000 0.825000
table_structure_similarity 40 40 0 0 1.000000 0.875000 0.125000 0.125000

Cell exactness:

system reference_cells exact_cells exact_rate mean_cell_cer
candidate 600 597 0.995000 0.001000
Surya 600 549 0.915000 0.071000
Tesseract 600 40 0.066667 0.933333

Paper-safe claim text:

On a targeted generated complex-table pack with one repeated merged-header pattern, a conservative span-inference postprocessor improved table structure extraction over Surya, raising span-aware structure similarity from 0.875 to 1.000 and table exact match from 0.000 to 0.925.

Do not generalize this to all complex tables. This is generated data with a single span pattern and a heuristic parser repair. The next paper-grade check needs real scanned tables with varied merged cells, rowspans, nested headers, and a true HTML/TEDS scorer.

Targeted Complex-Table TEDS-Style Metrics V1

We added deterministic ordered table-tree edit metrics and reran the same complex-table pack with these gates included.

Artifacts:

  • outputs/experiment-complex-table-teds-v1/benchmark/report.json
  • outputs/experiment-complex-table-teds-v1/validation/validation.md
  • outputs/summary-complex-table-teds-v1/summary.md
  • outputs/table-cell-analysis-complex-table-teds-v1/table-cell-analysis.md

Validation status: validated.

metric pairs wins losses ties candidate Surya improvement ci_low
table_cell_f1 40 40 0 0 0.995000 0.915000 0.080000 0.071667
table_exact_match 40 37 0 3 0.925000 0.000000 0.925000 0.825000
table_structure_similarity 40 40 0 0 1.000000 0.875000 0.125000 0.125000
table_teds_structure 40 40 0 0 1.000000 0.904762 0.095238 0.095238
table_teds_text 40 40 0 0 0.996250 0.891667 0.104583 0.098690

Paper-safe claim text:

On a targeted generated complex-table pack, the candidate improved ordered table-tree metrics over Surya, reaching 1.000 structure-only TEDS-style similarity and 0.996 text-sensitive TEDS-style similarity, compared with Surya's 0.905 and 0.892 respectively.

This is still not a public-benchmark SOTA claim. The metrics are deterministic and closer to table-structure scoring, but the pack is generated and covers one repeated-header merged-cell pattern.

HF Recognizer Next V1 Admission Benchmark

The AWS G6 trained HF recognizer was evaluated on the recognizer-next-v1 held-out eval manifest against Tesseract on k2.

Artifacts:

  • outputs/k2-hf-recognizer-next-v1-full-cpu/benchmark/report.json
  • outputs/k2-hf-recognizer-next-v1-full-cpu/summary/summary.md
  • candidate model card: outputs/aws-g6-hf-recognizer-next-v1/package/model-card-cpu.json

Run status:

  • candidate rows: 244 ok, 0 errors
  • Tesseract rows: 244 ok, 0 errors
  • paired CER rows with text references: 180
slice pairs candidate CER Tesseract CER improvement wins losses ties
all 180 0.226628 0.329012 0.102384 74 54 52
english/en/line_crop 150 0.145190 0.243379 0.098189 56 42 52
nepali 30 0.633814 0.757175 0.123360 18 12 0
real 15 0.832255 1.162917 0.330662 12 3 0
synthetic 165 0.171571 0.253202 0.081631 62 51 52

Paper-safe claim text:

On the held-out recognizer-next-v1 text-reference slices, the trained HF recognizer improves paired CER over Tesseract, reducing aggregate CER from 0.329 to 0.227 across 180 paired samples.

Do not call this SOTA. The result is a recognizer admission benchmark against Tesseract only; the branch still needs Surya, stock PaddleOCR, GLM-OCR, public document benchmarks, table structure metrics, reading-order validation, and document reconstruction gates.