Download docs/PAPER_RESULTS.md from prosperious/ocr-tech-k2: direct link, hf CLI and curl.
- Browser
- Download file 48.7 kB
-
https://huggingface.co/prosperious/ocr-tech-k2/resolve/main/docs/PAPER_RESULTS.md
- Command line
-
hf download hf://prosperious/ocr-tech-k2/docs/PAPER_RESULTS.md
-
curl -L -o PAPER_RESULTS.md https://huggingface.co/prosperious/ocr-tech-k2/resolve/main/docs/PAPER_RESULTS.md
Paper Results
This file is the shortest path from repo artifacts to a paper draft.
Use PAPER_NOTES.md for claim boundaries and framing. Use this file for the concrete result tables and the branch chronology.
Current Comparison Tables
Rendered Claim Pack
Generated artifact:
Headline table:
| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
|---|---|---|---|---|---|
| Candidate | 25 | 0.248654 | 0.955556 | 0.764000 | 3.268937 |
| Surya | 25 | 0.195260 | 0.955556 | 0.750909 | 2.618872 |
| Tesseract | 25 | 0.689214 | 0.000000 | 0.816000 | 0.240963 |
| Stock Paddle | 25 | 1.749081 | 0.000000 | 1.000000 | 7.732034 |
Interpretation:
- Candidate beats Tesseract and stock-Paddle badly on text CER and table extraction.
- Candidate ties Surya on aggregate
table_cell_f1. - Candidate is slightly better than Surya on aggregate reading-order accuracy.
- Candidate still loses to raw Surya on text CER.
- Candidate is also slower than Tesseract and somewhat slower than Surya on this pack.
Latency caveat:
- these latency means come from separate benchmark reports on the same host and pack, then get pinned per system during table export
- treat them as comparable operational signals, not as a single synchronized runtime shootout
Rendered Claim Pack V2 Holdout
Generated artifact:
Headline table:
| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy |
|---|---|---|---|---|
| Candidate | 50 | 0.180893 | 0.947500 | 0.750909 |
| Surya | 50 | 0.180893 | 0.970833 | 0.750909 |
| Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 |
Interpretation:
- This is the stronger rendered-document holdout because calibration and holdout were separated at
50/50. - The routed candidate does not beat Surya here. It ties Surya exactly on text CER and reading-order accuracy.
- The routed candidate is slightly worse than raw Surya on aggregate
table_cell_f1. - The calibration artifact selected
primary_script_threshold=0.0, which means the routed candidate collapsed to "always pick Surya" on the calibration split. - That is why claim preflight now rejects this candidate as a routed model fit for a broad "better engine" claim.
Rendered Claim Pack V2 Holdout - Surya Primary Latin Router
Generated artifact:
Headline table:
| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
|---|---|---|---|---|---|
| Candidate | 50 | 0.416854 | 0.337500 | 0.761818 | 6.067455 |
| Surya | 50 | 0.180893 | 0.970833 | 0.750909 | 6.699851 |
| Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 | 0.238464 |
Interpretation:
- This candidate was a Surya-primary, Latin-triggered page router with Tesseract fallback.
- It was meant to preserve Surya's text strength while using Tesseract for English-heavy pages.
- It failed badly on the rendered-v2 holdout: worse than raw Surya on text CER and much worse on table structure.
- Paired CER against Surya was decisive enough to rule the branch out:
0wins,12losses,38ties.
Rendered Claim Pack V2 Holdout - Surya Quality Router
Generated artifact:
Headline table:
| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
|---|---|---|---|---|---|
| Candidate | 50 | 0.181128 | 0.947500 | 0.750909 | 4.620400 |
| Surya | 50 | 0.180893 | 0.947500 | 0.750909 | 5.603801 |
| Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 | 0.240773 |
Interpretation:
- This candidate was a Surya-vs-Tesseract quality selector calibrated on the rendered-v2 calibration split.
- Calibration selected
selected_threshold=0.0, which means the quality gate kept Surya whenever it was not scored worse than Tesseract. - On the untouched holdout, the candidate still did not beat raw Surya on text CER.
- Paired CER against Surya was effectively a tie with one bad miss:
0wins,1loss,49ties. - This is operationally closer to "Surya with a guardrail" than to a meaningfully new OCR engine.
Rendered Claim Pack V2 Holdout - Line Align Composite
Generated artifact:
Headline table:
| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
|---|---|---|---|---|---|
| Candidate | 50 | 0.310549 | 0.897500 | 0.792727 | 4.142242 |
| Surya | 50 | 0.180893 | 0.970833 | 0.750909 | 3.223913 |
| Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 | 0.235074 |
Interpretation:
- This candidate kept Surya as the text engine and reused Tesseract line scaffolding through
line_align_composite. - It did improve reading-order accuracy relative to raw Surya, and even edged past raw Tesseract on this holdout.
- That gain is not enough. CER regressed badly versus Surya, and aggregate
table_cell_f1also regressed. - Paired CER against Surya was one-sided enough to kill the branch:
0wins,7losses,43ties. - This is a useful negative result for the paper because it shows that better line ordering alone does not rescue the Surya/Tesseract composite family.
Rendered Claim Pack V2 Holdout - Sample-Level Oracle
Generated artifact:
Headline table:
| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
|---|---|---|---|---|---|
| Oracle Surya/Tesseract | 50 | 0.180529 | 0.970833 | 0.750909 | 3.193115 |
| Surya | 50 | 0.180893 | 0.970833 | 0.750909 | 3.223913 |
| Tesseract | 50 | 0.714207 | 0.000000 | 0.787818 | 0.235074 |
Interpretation:
- This is not a deployable system. It is an upper-bound analysis that picks the better of raw Surya and raw Tesseract per sample using ground-truth CER.
- The oracle picked Surya for
49of50holdout samples and Tesseract for only1. - Per-sample oracle selections are recorded separately in
oracle-choices.json; they are provenance, not benchmark metrics. - The absolute CER gain over raw Surya is only
0.000364. - That means the current Surya/Tesseract routing space has almost no text-OCR headroom left on rendered-v2.
- Future work should move to a stronger recognizer/document model, not more Surya/Tesseract routing heuristics.
Rendered Claim Pack V2 Holdout - Surya Error Pockets
Generated artifact:
- error-analysis.md
- top-failures.json
- surya-vs-tesseract error-analysis.md
- surya-vs-oracle error-analysis.md
- eval-surya-hard15.jsonl
- validation-hard15-surya.json
- line-align hard15 validation.md
- k2 manifest-audit.md
- table/figure dev pack manifest
- table/figure dev pack audit.md
Diagnostic summary:
| slice | sample_count | Surya CER |
|---|---|---|
| figure | 20 | 0.253157 |
| table | 20 | 0.252781 |
| multi_page | 10 | 0.196472 |
| scan | 10 | 0.194861 |
| low_contrast | 10 | 0.193317 |
| blur | 10 | 0.190940 |
| clean | 10 | 0.182256 |
Interpretation:
- Surya's hardest absolute CER pockets on rendered-v2 are table-ledger and figure-caption samples, not generic blur/skew alone.
- Tesseract is worse than Surya on
49of50samples and better on onlyeval-form-notice-003-blur; that single oracle win is too small to justify more Surya/Tesseract routing work. - A ranked
15-samplehard_evalmanifest now exists for the next model branch; it contains8table samples and7figure samples, and its full file/reference audit passes onk2. configs/validation-hard15-surya.jsondefines the pocket gate a future candidate must clear against Surya, including intersection gates forhard_eval,tableandhard_eval,figure.- The current line-align candidate fails the hard15 gate: hard-eval CER is
0.482039versus Surya0.348706, and hard table CER is0.613636versus Surya0.363636. - A separate
80-sample image development pack now exists for table/figure work:40table samples,40figure samples, all five degradations. It has zero sample-id overlap and zero text-hash overlap with hard15, and itsk2audit passes. - That development pack is not claim evidence; it is for model iteration before returning to the held-out hard15 and rendered-v2 gates.
- The next credible recognizer branch should target table/figure-heavy mixed-script pages and preserve Surya's already-strong behavior on the other
49samples.
Table/Figure Development Pack - HF TrOCR Negative Result
Generated artifact:
- split-summary.md
- training_metrics.json
- model-card.json
- model-audit.md
- summary.md
- benchmark report.json
- paddle_recognizer.yml
Pack and split:
80image samples:40table and40figure- degradations:
clean,scan,blur,low_contrast,skew - split:
60train /20eval - grouping:
text_sha256 - text groups:
16 - split leakage: pass
- train/eval manifest audits on
k2: pass - hard15 overlap:
0sample IDs and0text hashes
Training result for microsoft/trocr-small-printed after one epoch:
| metric | value |
|---|---|
| eval_loss | 7.884517 |
| eval_cer | 3.062119 |
| eval_wer | 3.562689 |
| eval_exact_line_accuracy | 0.000000 |
Benchmark result on the 20-sample dev eval:
| system | CER | table_cell_f1 | reading_order_pair_accuracy |
|---|---|---|---|
| HF table/figure candidate | 2.422001 | 0.000000 | 0.000000 |
| Surya | 0.563399 | 0.858333 | 0.550000 |
| Tesseract | 0.546227 | 0.000000 | 0.695000 |
Interpretation:
- The model package audits cleanly, so this is a quality failure rather than a packaging failure.
- Paired CER was
0wins,20losses, and0ties against Surya. - Paired CER was also
0wins,20losses, and0ties against Tesseract. - Naive full-page TrOCR fine-tuning is not a serious next step for table/figure-heavy document pages.
- The next model branch should use line/cell crops or a structure-aware page/document model rather than treating full document images as single recognition targets.
Table/Figure Crop Development Pack - HF TrOCR Negative Result
Generated artifact:
- crop-manifest-summary.md
- crop manifest audit.md
- crop split-summary.md
- training_metrics.json
- model-card.json
- model-audit.md
- summary.md
- benchmark report.json
Pack and split:
- source rendered samples:
80 - extracted crops:
544 - crop types:
384table cells,128lines,32figure captions - skipped candidates:
32skew pages because pre-rotation bboxes are invalid - split:
480train /64eval - grouping:
text_sha256 - text groups:
68 - crop manifest audit on
k2: pass - split leakage: pass
Training result for microsoft/trocr-small-printed after one crop-level epoch:
| metric | value |
|---|---|
| eval_loss | 3.087561 |
| eval_cer | 0.959831 |
| eval_wer | 1.363715 |
| eval_exact_line_accuracy | 0.078125 |
Benchmark result on the 64-sample crop eval:
| system | CER |
|---|---|
| HF crop candidate | 1.043369 |
| Surya | 0.475309 |
| Tesseract | 0.077453 |
Interpretation:
- Crop-level training is a better problem shape than full-page recognition, but this recognizer is still not competitive.
- Paired CER against Surya was
7wins,50losses, and7ties. - Paired CER against Tesseract was
0wins,64losses, and0ties. - Tesseract is extremely strong on these rendered crop images, so near-term gains are more likely from better crop/layout detection plus strong crop OCR than from this small synthetic TrOCR fine-tune.
Hard15 - Table Cell Refinement Composite
Generated artifact:
- unsafe model-card.json
- unsafe summary.md
- unsafe cell-analysis.md
- safe model-card.json
- safe summary.md
- safe cell-analysis.md
- safe validation.md
- safe+dedup model-card.json
- safe+dedup summary.md
- safe+dedup cell-analysis.md
- safe+dedup validation.md
Headline hard15 comparison:
| system | hard_eval CER | table_cell_f1 | reading_order_pair_accuracy |
|---|---|---|---|
| Surya baseline | 0.348706 | 0.979167 | 0.520000 |
Table-cell refine, always |
0.348706 | 0.593750 | 0.520000 |
Table-cell refine, empty |
0.348706 | 0.979167 | 0.520000 |
Table-cell refine, empty + parser de-dup |
0.348706 | 0.979167 | 0.520000 |
Interpretation:
- The unsafe table-cell crop replacement policy damaged Surya's already-good table cells.
- The conservative empty-cell-only policy avoided regression but did not improve any hard15 metric.
- The safe policy's hard15 validation status is
not_validated. - The parser de-dup branch removes duplicate overlapping table exports for both candidate and Surya, so it improves artifact cleanliness but does not create a candidate advantage.
- Exact crop OCR being strong is not enough; the deployment problem is reliable cell detection and knowing when a replacement is safer than the structure engine's existing cell text.
Cell-level result:
- Surya is already exact on
94/96hard15 reference table cells. - Unsafe replacement is exact on only
57/96cells. - Unsafe replacement improves
0cells over Surya and worsens37. - Safe+dedup is exact on
94/96cells, with8predicted tables for8reference tables and0duplicate overlapping tables on the v3 rerun.
Hard15 - Figure Caption Refinement Composite
Generated artifact:
- model-card.json
- model-audit.md
- summary.md
- benchmark report.json
- validation.md
- labels model-card.json
- labels summary.md
- labels validation.md
Headline hard15 comparison after the parser caption-attachment fix:
| system | CER | figure_caption_cer | figure_detection_f1 | table_cell_f1 | reading_order_pair_accuracy |
|---|---|---|---|---|---|
| Surya | 0.236839 | 0.092323 | 1.000000 | 0.979167 | 0.520000 |
| Tesseract | 1.266364 | 0.051020 | 0.857143 | 0.000000 | 0.613333 |
| Figure-caption refine | 0.228008 | 0.052478 | 1.000000 | 0.979167 | 0.520000 |
| Figure-caption refine + labels | 0.204610 | 0.008746 | 1.000000 | 0.979167 | 0.520000 |
Validation result:
- basic branch overall status:
not_validated - basic branch figure-caption gate: pass,
6wins /0losses /1tie against Surya, improvement0.039845 - label branch overall status:
not_validated - label branch overall hard-eval CER gate: pass, improvement
0.032228 - label branch figure CER gate: pass, improvement
0.069061 - label branch figure-caption gate: pass,
7wins /0losses /0ties against Surya, improvement0.083576 - label branch table gates: fail, ties against Surya
Interpretation:
- This is a real slice win over Surya on figure-caption extraction.
- It is not broad enough for a SOTA claim.
- Tesseract has better caption text but worse figure detection/table structure; the composite keeps Surya's structure and selectively borrows caption text.
- Safe empty-only replacement ties all
96cells with Surya because there are no missing Surya cells in this pocket. - The only remaining Surya cell errors are
किरणpredicted asकरिणoneval-table-ledger-002-cleanandeval-table-ledger-002-scan.
Rendered Blur-Only Expensive-Baseline Subset
Generated artifact:
Headline table:
| system | sample_count | cer | table_cell_f1 | reading_order_pair_accuracy | latency_seconds |
|---|---|---|---|---|---|
| Candidate | 5 | 0.223273 | 1.000000 | 0.783636 | 3.784279 |
| Surya | 5 | 0.197601 | 1.000000 | 0.750909 | 2.943875 |
| Tesseract | 5 | 0.732364 | 0.000000 | 0.823636 | 0.220515 |
| Stock Paddle | 5 | 2.810426 | 0.000000 | 1.000000 | 7.440482 |
| GLM-OCR | 5 | 2.820269 | 0.000000 | 1.000000 | 16.846619 |
| PaddleOCR-VL | 5 | 18.470663 | 0.000000 | 0.750909 | 32.793166 |
Interpretation:
- Candidate is materially stronger than Tesseract, stock-Paddle, GLM-OCR, and PaddleOCR-VL on this degraded rendered subset.
- Candidate still does not beat raw Surya on text CER.
- The external VLM-style baselines are also much slower than the current candidate here.
Branch Chronology
This is the shortest honest story of the project so far.
| branch | eval setting | candidate CER | baseline CER | paired result | verdict | evidence |
|---|---|---|---|---|---|---|
| HF TrOCR recognizer | mixed Nepali/English line-style holdout, 180 samples | 0.586139 | 0.329012 vs Tesseract | 36 wins, 97 losses, 47 ties | runnable but not competitive | summary.json |
| Tesseract + ByT5 corrector | same 180-sample mixed holdout | 6.912065 | 0.329012 vs Tesseract | 0 wins, 170 losses, 10 ties | failed badly | summary.md |
quality router (psm=6 vs psm=13) |
91-sample mixed holdout | 0.340608 | 0.342234 vs Tesseract | 2 wins, 1 loss, 88 ties | tiny gain, plausible baseline only | summary.json |
ranked PSM ensemble (6/7/11/13) |
same 91-sample mixed holdout | 0.358352 | 0.342234 vs Tesseract | 9 wins, 16 losses, 66 ties | overfit, worse than baseline | summary.json |
script router (tesseract -> surya) |
same 91-sample mixed holdout | 0.313084 | 0.342234 vs Tesseract | 9 wins, 0 losses, 82 ties | strongest older line-style result | summary.json |
| script router + structure backfill | rendered claim-ready document pack, 25 samples | 0.248654 | 0.689214 vs Tesseract | 9 wins, 1 loss, 15 ties | strong document-pack improvement over Tesseract | summary.json |
| raw Surya on rendered claim pack | rendered claim-ready document pack, 25 samples | 0.195260 | 0.248654 vs Candidate | candidate loses on text CER | current text leader on rendered pack | summary.json |
| rendered-v2 script router | rendered claim-ready document holdout, 50 samples | 0.180893 | 0.180893 vs Surya | exact CER tie, worse table structure | routed candidate collapses to Surya on calibration; not a meaningful new engine | summary.json |
| rendered-v2 Surya-primary Latin router | rendered claim-ready document holdout, 50 samples | 0.416854 | 0.180893 vs Surya | 0 wins, 12 losses, 38 ties on CER | failed badly; inverted script routing is not a viable answer | summary.json |
| rendered-v2 Surya quality router | rendered claim-ready document holdout, 50 samples | 0.181128 | 0.180893 vs Surya | 0 wins, 1 loss, 49 ties on CER | nearly indistinguishable from Surya and still not better | summary.json |
| rendered-v2 line-align composite | rendered claim-ready document holdout, 50 samples | 0.310549 | 0.180893 vs Surya | 0 wins, 7 losses, 43 ties on CER | better reading order, but text and table regressions are too large | summary.json |
| rendered-v2 Surya/Tesseract sample oracle | rendered claim-ready document holdout, 50 samples | 0.180529 | 0.180893 vs Surya | oracle picks Tesseract only 1/50 samples | upper bound shows routing headroom is essentially exhausted | oracle-analysis.json |
| HF TrOCR table/figure dev | table/figure development eval, 20 samples | 2.422001 | 0.563399 vs Surya; 0.546227 vs Tesseract | 0 wins, 20 losses, 0 ties vs both | failed badly; full-page TrOCR fine-tuning is not viable here | summary.md |
| HF TrOCR crop dev | table/figure crop eval, 64 samples | 1.043369 | 0.475309 vs Surya; 0.077453 vs Tesseract | 7 wins, 50 losses, 7 ties vs Surya; 0 wins, 64 losses vs Tesseract | crop shape helps but this recognizer still fails | summary.md |
| table-cell refine, unsafe | hard15 rendered-v2 pocket, 15 samples | 0.348706 | 0.348706 vs Surya | text tie, table F1 regresses from 0.979167 to 0.593750 | replacement policy is unsafe | summary.md |
| table-cell refine, empty-only | hard15 rendered-v2 pocket, 15 samples | 0.348706 | 0.348706 vs Surya | exact tie across CER/table/reading order; validation not_validated | safe but no improvement | validation.md |
| normalized script router, hard-timeout rerun | normalized mixed Nepali/English line-style eval, 180 samples | 0.354840 | 0.329012 vs Tesseract; 3.157317 vs Surya on 178 scored Surya rows | 17 wins, 4 losses, 159 ties on CER vs Tesseract but worse aggregate mean; 2 Surya rows timed out at 45s | not a better engine; benchmark watchdog works and routing still loses overall CER | summary.md |
| PaddleOCR recognizer hard-50 failure slice | top 50 failures from AWS G6 trained PaddleOCR recognizer, remapped to normalized k2 eval paths | n/a | n/a | 50 failures: 38 English line-crop synthetic, 12 Nepali synthetic; k2 manifest audit passed | next recognizer gate; do not train on it | manifest-audit.md |
| PaddleOCR hard-50 gate negative control | failed AWS G6 trained PaddleOCR recognizer evaluated on the hard-50 gate | 0.887557 | 0.080249 vs Tesseract | fails all four hard gates; English line-crop CER 0.852048 vs 0.031244 |
validates the gate catches the known recognizer collapse | validation.md |
| line-level Tesseract/Paddle composite | hard-50 failure slice, Tesseract primary with PaddleOCR secondary only for Devanagari-looking lines | 0.276489 | 0.120106 vs Tesseract | English line-crop ties exactly (0.060429 CER); Nepali collapses (0.960679 vs 0.309083 CER, 1.000000 vs 0.711607 WER) |
safe English preservation but failed Devanagari override; do not promote this recognizer | validation.md |
| PaddleOCR v3 CUDA recognizer | G6-trained lower-LR PaddleOCR branch on hard-50 admission gate | 0.921416 | 0.080249 vs Tesseract | 0 wins, 50 losses on CER; English line-crop CER 0.917511 vs 0.031244; Nepali WER 1.055556 vs 0.716369 |
failed worse than v2; stop branch before full mixed eval | validation.md |
| PaddleOCR v4 CUDA recognizer | G6-trained Devanagari PP-OCRv3 branch, LR 0.00005, 20 epochs, evaluated on hard-50 admission gate |
0.940500 | 0.080249 vs Tesseract | 0 wins, 50 losses on CER; English line-crop CER 0.950730 vs 0.031244; Nepali CER 0.908108 vs 0.235433 |
failed worse than v2/v3; the same PaddleOCR mixed-script fine-tune branch should not be the engine core | artifact bundle |
Current Best-Evidence Statement
The strongest evidence-backed statement available today is:
- The current Tesseract-plus-Surya backfill candidate is a real document-OCR improvement over Tesseract and several weaker baselines on internal rendered Nepali/English document packs.
- On the larger rendered-v2 holdout, every Tesseract/Surya composite tested so far either collapses to Surya, trails Surya, becomes effectively indistinguishable from Surya, or trades too much OCR accuracy for modest ordering gains.
- On the cleaned 180-sample mixed line-style eval with hard per-sample timeouts, the older script-router candidate also loses to Tesseract overall (
0.354840vs0.329012CER), despite a small overall WER improvement (0.469944vs0.486485) and a Nepali WER improvement. - The trained PaddleOCR recognizer failure profile is now materialized as a normalized hard-50 eval gate: all top failures are synthetic line crops, mostly English (
38/50), so the next recognizer branch must prove English preservation instead of only improving Nepali. - The line-level Tesseract/Paddle composite proves the candidate shape can preserve English while routing Devanagari-looking lines, but the available trained PaddleOCR recognizer is too weak on the Nepali hard slice to use as an override (
0.960679vs Tesseract0.309083CER). - The second AWS G6 PaddleOCR attempt fixed recipe/runtime issues but failed the hard-50 gate even worse than the first (
0.921416vs Tesseract0.080249CER), so this PP-OCRv3 Devanagari recipe family is not the engine path. - A sample-level oracle over raw Surya and raw Tesseract has almost no text-OCR headroom over Surya (
0.180529vs0.180893CER), so more routing in this component family is not a serious path to SOTA. - The repo now has enough validation discipline and artifact traceability to support a serious comparative paper, but not a SOTA claim.
Regeneration Workflow
Rebuild the generated comparison tables with:
ocrtech summarize-benchmark \
--benchmark-report outputs/live-k2-claimpack-rendered-backfill/report.json \
--out outputs/paper-claimpack-summary-tesseract \
--candidate candidate \
--baseline tesseract \
--metric cer
ocrtech summarize-benchmark \
--benchmark-report outputs/live-k2-claimpack-surya-compare/report.json \
--out outputs/paper-claimpack-summary-surya \
--candidate candidate \
--baseline surya \
--metric cer
ocrtech summarize-benchmark \
--benchmark-report outputs/live-k2-claimpack-stock-paddle-fixed/report.json \
--out outputs/paper-claimpack-summary-stock-paddle \
--candidate candidate \
--baseline stock-paddle \
--metric cer
ocrtech paper-benchmark-table \
--summary outputs/paper-claimpack-summary-tesseract/summary.json \
--summary outputs/paper-claimpack-summary-surya/summary.json \
--summary outputs/paper-claimpack-summary-stock-paddle/summary.json \
--out outputs/paper-claimpack-table \
--system candidate \
--system surya \
--system tesseract \
--system stock-paddle \
--label 'candidate=Candidate' \
--label 'surya=Surya' \
--label 'tesseract=Tesseract' \
--label 'stock-paddle=Stock Paddle' \
--source-preference 'candidate=outputs/paper-claimpack-summary-tesseract/summary.json' \
--source-preference 'tesseract=outputs/paper-claimpack-summary-tesseract/summary.json' \
--source-preference 'surya=outputs/paper-claimpack-summary-surya/summary.json' \
--source-preference 'stock-paddle=outputs/paper-claimpack-summary-stock-paddle/summary.json' \
--metric cer \
--metric table_cell_f1 \
--metric reading_order_pair_accuracy \
--metric latency_seconds
ocrtech summarize-benchmark \
--benchmark-report outputs/live-k2-rendered-blur5/report.json \
--out outputs/paper-blur5-summary \
--candidate candidate \
--baseline tesseract \
--baseline surya \
--baseline stock-paddle \
--baseline glm-ocr \
--baseline paddleocr-vl \
--metric cer
ocrtech paper-benchmark-table \
--summary outputs/paper-blur5-summary/summary.json \
--out outputs/paper-blur5-table \
--system candidate \
--system surya \
--system tesseract \
--system stock-paddle \
--system glm-ocr \
--system paddleocr-vl \
--label 'candidate=Candidate' \
--label 'surya=Surya' \
--label 'tesseract=Tesseract' \
--label 'stock-paddle=Stock Paddle' \
--label 'glm-ocr=GLM-OCR' \
--label 'paddleocr-vl=PaddleOCR-VL' \
--metric cer \
--metric table_cell_f1 \
--metric reading_order_pair_accuracy \
--metric latency_seconds
ocrtech paper-benchmark-table \
--summary outputs/live-k2-claimpack-v2-script-router/summary.json \
--out outputs/paper-claimpack-v2-table \
--system candidate \
--system surya \
--system tesseract \
--label 'candidate=Candidate' \
--label 'surya=Surya' \
--label 'tesseract=Tesseract' \
--metric cer \
--metric table_cell_f1 \
--metric reading_order_pair_accuracy \
--title 'Rendered Claim Pack V2 Holdout Comparison'
ocrtech paper-benchmark-table \
--summary outputs/live-k2-claimpack-v2-surya-quality-router/summary.json \
--out outputs/paper-claimpack-v2-surya-quality-router-table \
--system candidate \
--system surya \
--system tesseract \
--label 'candidate=Candidate' \
--label 'surya=Surya' \
--label 'tesseract=Tesseract' \
--metric cer \
--metric table_cell_f1 \
--metric reading_order_pair_accuracy \
--metric latency_seconds \
--title 'Rendered Claim Pack V2 Holdout - Surya Quality Router'
ocrtech paper-benchmark-table \
--summary outputs/live-k2-claimpack-v2-line-align-summary/summary.json \
--out outputs/paper-claimpack-v2-line-align-table \
--system candidate \
--system surya \
--system tesseract \
--label 'candidate=Candidate' \
--label 'surya=Surya' \
--label 'tesseract=Tesseract' \
--metric cer \
--metric table_cell_f1 \
--metric reading_order_pair_accuracy \
--metric latency_seconds \
--title 'Rendered Claim Pack V2 Holdout - Line Align Composite'
ocrtech benchmark-oracle \
--benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \
--out outputs/paper-claimpack-v2-surya-tesseract-oracle \
--system surya \
--system tesseract \
--metric cer \
--oracle-name oracle-surya-tesseract
ocrtech paper-benchmark-table \
--summary outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-summary/summary.json \
--out outputs/paper-claimpack-v2-oracle-table \
--system oracle-surya-tesseract \
--system surya \
--system tesseract \
--label 'oracle-surya-tesseract=Oracle Surya/Tesseract' \
--label 'surya=Surya' \
--label 'tesseract=Tesseract' \
--metric cer \
--metric table_cell_f1 \
--metric reading_order_pair_accuracy \
--metric latency_seconds \
--title 'Rendered Claim Pack V2 Holdout - Sample-Level Oracle'
ocrtech benchmark-error-analysis \
--benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \
--out outputs/paper-claimpack-v2-surya-error-analysis \
--system surya \
--metric cer \
--top-n 15
ocrtech benchmark-error-analysis \
--benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \
--out outputs/paper-claimpack-v2-surya-vs-tesseract-error-analysis \
--system surya \
--baseline tesseract \
--metric cer \
--top-n 15
ocrtech benchmark-error-analysis \
--benchmark-report outputs/paper-claimpack-v2-surya-tesseract-oracle/oracle-report.json \
--out outputs/paper-claimpack-v2-surya-vs-oracle-error-analysis \
--system surya \
--baseline oracle-surya-tesseract \
--metric cer \
--top-n 10
ocrtech export-failure-manifest \
--source-manifest data/splits-claimpack-rendered-v2/eval.jsonl \
--top-failures outputs/paper-claimpack-v2-surya-error-analysis/top-failures.json \
--out data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl \
--split hard_eval
ocrtech validate-claim \
--benchmark-report outputs/live-k2-claimpack-v2-line-align/report.json \
--config configs/validation-hard15-surya.json \
--eval-manifest data/splits-claimpack-rendered-v2/eval-surya-hard15.jsonl \
--out outputs/validation-claimpack-v2-line-align-hard15
ocrtech create-eval-pack \
--out data/eval-pack-table-figure-dev-v1 \
--input-format image \
--template table-ledger \
--template figure-caption \
--count-per-template 8 \
--variant-offset 100 \
--degradation clean \
--degradation scan \
--degradation blur \
--degradation low_contrast \
--degradation skew
Memory still is not part of these generated tables. The current benchmark artifact format does not record it.
Hard15 Figure-Caption + Table-Lexicon Candidate
Latest artifact set:
outputs/figure-caption-table-lexicon-v1/model-card.jsonoutputs/experiment-figure-caption-table-lexicon-hard15-v3/benchmark/report.jsonoutputs/summary-figure-caption-table-lexicon-hard15-v3/summary.mdoutputs/table-cell-analysis-figure-caption-table-lexicon-hard15-v3/table-cell-analysis.mdoutputs/validation-figure-caption-table-lexicon-hard15-v3/validation.md
Headline table for internal hard15 only:
| system | CER | figure_caption_cer | figure_detection_f1 | table_cell_f1 | reading_order_pair_accuracy |
|---|---|---|---|---|---|
| candidate | 0.010671 | 0.008746 | 1.000000 | 1.000000 | 0.520000 |
| Surya | 0.236839 | 0.092323 | 1.000000 | 0.979167 | 0.520000 |
| Tesseract | 1.266364 | 0.051020 | 0.857143 | 0.000000 | 0.613333 |
Validation status is still not_validated.
Passed configured Surya gates:
- overall hard_eval CER:
15wins /0losses /0ties - table-slice CER:
8wins /0losses /0ties - figure-slice CER:
7wins /0losses /0ties - figure-caption CER:
7wins /0losses /0ties
Remaining configured failure:
- table-cell F1: candidate
1.000000, Surya0.979167, but bootstrap CI low is0.000000because there are only 2 improving table pages and 6 ties
Cell-level table evidence:
| system | reference_cells | exact_cells | exact_rate | mean_cer |
|---|---|---|---|---|
| candidate | 96 | 96 | 1.000000 | 0.000000 |
| Surya | 96 | 94 | 0.979167 | 0.010417 |
| Tesseract | 96 | 0 | 0.000000 | 1.000000 |
Use this result as an internal milestone, not a paper claim of broad SOTA. The candidate is lexicon-assisted for table names and uses guarded structural-title corrections (वार्षिक प्रतिवेदन, दैनिक हिसाब, चित्र, निष्कर्ष). A publishable table claim needs a larger independent table pack so confidence intervals are meaningful.
Targeted Table-Ledger Validation Pack V2
Artifacts:
data/eval-pack-table-ledger-claim-v1/manifests/ocrtech-eval-pack.jsonloutputs/audit-table-ledger-claim-v1/eval-pack-audit.mdoutputs/experiment-table-ledger-claim-v2/benchmark/report.jsonoutputs/summary-table-ledger-claim-v2/summary.mdoutputs/table-cell-analysis-table-ledger-claim-v2/table-cell-analysis.mdoutputs/validation-table-ledger-claim-v2b/validation.mdconfigs/validation-table-ledger-surya.json
Pack:
- template:
table-ledger - samples:
60 - cells:
720 - degradations: clean, scan, blur, low_contrast, skew
- variant offset:
300 - claim-ready broad coverage: no, targeted table-only pack
Validation status: validated.
| metric | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
|---|---|---|---|---|---|---|---|---|
| table_cell_f1 | 60 | 15 | 0 | 45 | 1.000000 | 0.979167 | 0.020833 | 0.012500 |
| table_exact_match | 60 | 15 | 0 | 45 | 1.000000 | 0.750000 | 0.250000 | 0.150000 |
| table_structure_similarity | 60 | 0 | 0 | 60 | 1.000000 | 1.000000 | 0.000000 | 0.000000 |
Cell exactness:
| system | reference_cells | exact_cells | exact_rate | mean_cell_cer |
|---|---|---|---|---|
| candidate | 720 | 720 | 1.000000 | 0.000000 |
| Surya | 720 | 705 | 0.979167 | 0.005208 |
| Tesseract | 720 | 0 | 0.000000 | 0.999868 |
Runtime on k2:
| system | mean latency seconds | median latency seconds |
|---|---|---|
| candidate | 3.017929 | 2.967943 |
| Surya | 2.478481 | 2.453125 |
| Tesseract | 0.146866 | 0.134735 |
Paper-safe claim text:
On a targeted 60-page generated Nepali-English table-ledger validation pack, the lexicon-assisted candidate improved structured table extraction over Surya, reaching 720/720 exact cells versus Surya's 705/720, and passed bootstrap-gated validation for both table-cell F1 and table exact match.
More precise paper-safe wording:
On a targeted 60-page generated Nepali-English table-ledger validation pack, the lexicon-assisted candidate improved table cell-content extraction over Surya, reaching 720/720 exact cells versus Surya's 705/720, while matching Surya's perfect grid-structure score.
Do not generalize this to all tables. The pack has no merged cells, no nested headers, and no real scanned government tables. The scoring path now supports span-aware structured references, but this pack does not exercise that capability and the metric is still not true HTML-tree TEDS.
Targeted Complex-Table Validation Pack V1
Artifacts:
data/eval-pack-complex-table-v1/manifests/ocrtech-eval-pack.jsonloutputs/audit-complex-table-v1/eval-pack-audit.mdoutputs/experiment-complex-table-v1/benchmark/report.jsonoutputs/summary-complex-table-v1/summary.mdoutputs/table-cell-analysis-complex-table-v1/table-cell-analysis.mdoutputs/validation-complex-table-v1/validation.mdconfigs/validation-complex-table-surya.json
Pack:
- template:
complex-table-ledger - samples:
40 - reference cells:
600 - span reference: merged header with
colspan=2 - degradations: clean, scan, blur, low_contrast
- broad claim-ready coverage: no
Validation status: validated for cell-content improvement plus structure non-regression.
| metric | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
|---|---|---|---|---|---|---|---|---|
| table_cell_f1 | 40 | 8 | 0 | 32 | 0.928333 | 0.915000 | 0.013333 | 0.005000 |
| table_exact_match | 40 | 0 | 0 | 40 | 0.000000 | 0.000000 | 0.000000 | 0.000000 |
| table_structure_similarity | 40 | 0 | 0 | 40 | 0.875000 | 0.875000 | 0.000000 | 0.000000 |
Cell exactness:
| system | reference_cells | exact_cells | exact_rate | mean_cell_cer |
|---|---|---|---|---|
| candidate | 600 | 557 | 0.928333 | 0.067667 |
| Surya | 600 | 549 | 0.915000 | 0.071000 |
| Tesseract | 600 | 40 | 0.066667 | 0.933333 |
Paper-safe claim text:
On a targeted generated complex-table pack with span-aware references, the lexicon-assisted candidate improved table cell-content F1 over Surya but did not improve merged-header reconstruction; both systems failed exact table match and tied on span-aware structure similarity.
This is a negative result for complex table structure. The next model/parser branch must preserve or infer rowspans/colspans before we can claim complex-table document understanding.
Targeted Complex-Table Span-Inference Pack V1
After adding conservative repeated-header colspan inference, we reran the same 40-sample generated complex-table pack on k2.
Artifacts:
outputs/experiment-complex-table-span-infer-v1/benchmark/report.jsonoutputs/experiment-complex-table-span-infer-v1/validation/validation.mdoutputs/summary-complex-table-span-infer-v1/summary.mdoutputs/table-cell-analysis-complex-table-span-infer-v1/table-cell-analysis.md
Validation status: validated.
| metric | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
|---|---|---|---|---|---|---|---|---|
| table_cell_f1 | 40 | 40 | 0 | 0 | 0.995000 | 0.915000 | 0.080000 | 0.071667 |
| table_exact_match | 40 | 37 | 0 | 3 | 0.925000 | 0.000000 | 0.925000 | 0.825000 |
| table_structure_similarity | 40 | 40 | 0 | 0 | 1.000000 | 0.875000 | 0.125000 | 0.125000 |
Cell exactness:
| system | reference_cells | exact_cells | exact_rate | mean_cell_cer |
|---|---|---|---|---|
| candidate | 600 | 597 | 0.995000 | 0.001000 |
| Surya | 600 | 549 | 0.915000 | 0.071000 |
| Tesseract | 600 | 40 | 0.066667 | 0.933333 |
Paper-safe claim text:
On a targeted generated complex-table pack with one repeated merged-header pattern, a conservative span-inference postprocessor improved table structure extraction over Surya, raising span-aware structure similarity from 0.875 to 1.000 and table exact match from 0.000 to 0.925.
Do not generalize this to all complex tables. This is generated data with a single span pattern and a heuristic parser repair. The next paper-grade check needs real scanned tables with varied merged cells, rowspans, nested headers, and a true HTML/TEDS scorer.
Targeted Complex-Table TEDS-Style Metrics V1
We added deterministic ordered table-tree edit metrics and reran the same complex-table pack with these gates included.
Artifacts:
outputs/experiment-complex-table-teds-v1/benchmark/report.jsonoutputs/experiment-complex-table-teds-v1/validation/validation.mdoutputs/summary-complex-table-teds-v1/summary.mdoutputs/table-cell-analysis-complex-table-teds-v1/table-cell-analysis.md
Validation status: validated.
| metric | pairs | wins | losses | ties | candidate | Surya | improvement | ci_low |
|---|---|---|---|---|---|---|---|---|
| table_cell_f1 | 40 | 40 | 0 | 0 | 0.995000 | 0.915000 | 0.080000 | 0.071667 |
| table_exact_match | 40 | 37 | 0 | 3 | 0.925000 | 0.000000 | 0.925000 | 0.825000 |
| table_structure_similarity | 40 | 40 | 0 | 0 | 1.000000 | 0.875000 | 0.125000 | 0.125000 |
| table_teds_structure | 40 | 40 | 0 | 0 | 1.000000 | 0.904762 | 0.095238 | 0.095238 |
| table_teds_text | 40 | 40 | 0 | 0 | 0.996250 | 0.891667 | 0.104583 | 0.098690 |
Paper-safe claim text:
On a targeted generated complex-table pack, the candidate improved ordered table-tree metrics over Surya, reaching 1.000 structure-only TEDS-style similarity and 0.996 text-sensitive TEDS-style similarity, compared with Surya's 0.905 and 0.892 respectively.
This is still not a public-benchmark SOTA claim. The metrics are deterministic and closer to table-structure scoring, but the pack is generated and covers one repeated-header merged-cell pattern.